Six decibels a bit, and the half step blamed on it

The previous essay used q/√12 as a floor and did not ask where it comes from or whether a real converter reaches it. This one does both, and the answer contains a small surprise of a kind this site has met before: the famous formula is exactly right, the famous shortfall is real, and the shortfall is not the formula’s.

Where the formula comes from, and what it assumes #

A quantiser rounds to the nearest of a set of levels q apart, so its error lies between −q/2 and +q/2. If that error is treated as a random variable uniform over the step, its mean square is q²/12 and its root mean square is q/√12. A full-scale sinusoid on a span of 2A has amplitude A and root mean square A/√2, so the ratio of the two is

$$\frac{A/\sqrt2}{q/\sqrt{12}} = \frac{2^{N}}{\sqrt{2}}\cdot\frac{\sqrt{12}}{2} = 2^{N}\sqrt{1.5}$$

which in decibels is 6.02N + 1.76.

Three assumptions went into that and it is worth listing them, because two are checked in this essay and one is the subject of the next.

That the error fills the step uniformly. True when the signal wanders across many levels between samples and false when it does not, which is the boundary the next essay measures.

That the error is uncorrelated with the signal. The same condition, stated the other way round, and the same essay.

That the signal is a full-scale sinusoid. This is the assumption that produces the shortfall, because “full scale” turns out to be half a step further than a mid-tread converter can go.

Measuring it rather than believing it #

The measurement takes the error sequence — the quantised samples minus the exact ones, sample by sample — and takes its root mean square directly. Nothing about the formula enters. The signal’s own root mean square is A/√2 by construction, so the ratio is one division.

The subtlety is entirely in the choice of input frequency, and it is the sort of subtlety that produces a beautiful wrong figure rather than an obviously broken one. If the ratio of input frequency to sample rate is simple, the quantiser visits the same few points of the waveform over and over; the error sequence becomes periodic with a short period, its energy collects into a handful of lines, and the measured ratio can be anything from three decibels pessimistic to fifteen optimistic — stably and repeatably, at every record length, so that nothing about the number invites suspicion.

coherentCycles therefore chooses a cycle count that is an integer (so the record holds whole cycles and the transform needs no window), odd, and coprime with the record length (so the sample points do not repeat). The gate checks all three properties rather than trusting the construction.

With that in place, one step inside full scale:

The formula is not an approximation to anything at this level. The worst departure across three record lengths and five bit counts — fifteen measurements — is 0.29 dB, and it is scatter rather than bias: the sign changes.

The half step #

Now drive the same converters to exactly full scale, which is what a bench measurement does and what the ratio is nominally defined at.

The last column is the explanation and it is a count rather than an inference. A mid-tread converter has levels running from $-2^{N-1}q$ to $(2^{N-1}-1)q$: symmetric about zero in the middle and one step short at the top. A sinusoid whose peak reaches full scale therefore spends a little time above the highest available code, and the samples taken there are clipped rather than rounded.

Clipping near a peak is the worst possible place for it. The sinusoid is flat there, so many samples are affected; and the error is one-sided and correlated with the signal, so it appears as harmonics rather than as noise. Both properties make it far more damaging per sample than an ordinary rounding error.

The count halves with every bit — 327, 163, 81, 41, 21 — because the region above the top code is half a step deep and a step halves with every bit, so the fraction of the cycle spent in it halves too. That halving is what identifies the mechanism, and the gate holds it rather than holding the decibel figures: a check on 0.83 dB would pass for several wrong reasons, and a check on a count that halves five times would not.

By sixteen bits the loss has fallen to −0.13 dB, which is to say the full-scale drive now measures better — the extra amplitude is worth more than twenty-one clipped samples cost. A first version of the figure asserted that the full-scale drive is worse at every bit count and failed at sixteen, which is the right way round for a gate to fail.

Why the record length does not matter and the frequency does #

Fifteen measurements went into the agreement above: five bit counts at three record lengths — 4,096, 8,192 and 16,384 samples. The record length is varied deliberately and it is worth saying what it is testing, because it is not what a reader might assume.

It is not testing convergence. The error’s root mean square is taken directly from the sequence, not estimated from a spectrum, so a longer record does not buy a better estimate of a fixed quantity in the way a longer noise run does. What varies with the record length is the cycle count, since coherentCycles picks a count near a third of the record — so three record lengths are three different input frequencies, chosen by a rule rather than by hand.

That is the property being tested. If the measured ratio depended on the input frequency, three records would disagree, and the disagreement would say that the error is not behaving as the formula assumes. They agree to 0.29 dB, which is the evidence that the treatment holds — at full amplitude.

The complementary experiment is the one this essay does not run and the next one does: hold the frequency and lower the amplitude, and the treatment fails. The error’s harmonic share climbs from 0.29% at 230 levels to 76% at 1.3, and at that point the ratio measured here would still come out close to the formula while describing something that is not noise at all.

There is one more property of the error worth recording, because it is easy to assume and is measured: the harmonic share is a function of the levels the input uses and nothing about the converter. A twelve-bit part and a sixteen-bit part sweeping the same range of levels agree to the floor of the arithmetic. That is why the next essay’s sweep is drawn against levels rather than against volts, and it is the property that lets one figure stand for every converter.

The same mistake this site has made once before #

The shape of this is familiar and the collection has a worked example of it in the semiconductor field.

The bias-point essay once quoted a diode’s drop as moving 60.4 mV per decade of current over one span and 62 mV over another, against the 59.5 mV its own algebra derives, and explained the difference as the load line’s share of the supply moving over the range. There was no difference to explain: the measurement is 59.53 mV per decade at every supply and every resistance, 28 solves, agreeing with the closed form to a part in 10⁵. A departure had been observed, an explanation had been supplied, and the departure was an artefact of how it was measured.

This is the same thing with the roles reversed. Here the departure is real — 0.83 dB, reproducibly — and the explanation usually attached to it is wrong. In both cases the repair is the same: measure the two things separately instead of measuring the sum and reasoning about it.

That is also why this essay’s figure draws three curves rather than two. Two curves would show a measurement and a formula disagreeing, and a reader would be entitled to conclude whichever they were already inclined to. Three curves show a measurement agreeing with the formula and a different measurement disagreeing, and the difference between the two measurements is one sentence long.

What the error actually looks like at full amplitude #

One check remains before the treatment can be called sound, and it is the one that separates “the number came out right” from “the error is what the formula says it is”.

The formula’s hypothesis is that the error is spread across the band. That can be measured directly: transform the error sequence, and read the power sitting at harmonics of the input. At 230 levels the harmonics of the input hold 0.29% of the error’s power, so 99.7% of it is somewhere else — spread, which is what the hypothesis says.

Reading that number correctly took a repair, and the repair is worth recording because it is the kind of measurement error that produces a convincing answer.

A record eight thousand samples long holds a hundred and twenty-nine cycles of the input, so the input is line 129 of the transform and its third harmonic is line 387. A first version of the measurement asked for the first twelve lines of the record — lines 1 to 12 — which is where nothing in this signal lives. The number that came back was small, stable, and fell smoothly as the amplitude fell, which is the opposite of the effect it was looking for and which nothing about its shape suggested was wrong.

It was found by asking what a quantiser producing something close to a square wave could possibly be putting into bin 3. binPower now evaluates the transform at named bins, the gate checks it on a record whose two lines are known — 1.000000 and 0.250000 against 1 and 0.25 — and it also checks that the first three bins of that record hold 3.4 × 10⁻³² of power, which is what the first version was measuring.

That is the third time this collection has caught a measurement of the wrong thing by asking what the number ought to be rather than whether it looked reasonable: the trapezoidal ringing that settled at −1.5 × 10⁻⁸ for a divider settling at 0.1, the dead slider that reported the same 12.57 dB at every one of its positions, and now twelve bins holding nothing.

What “full scale” should mean #

The practical consequence is a definition rather than a design change, and it is worth stating plainly because data sheets differ on it.

A converter’s ratio should be quoted at an amplitude that its codes can represent — one step, or conventionally 0.5 dB, inside full scale — and the amplitude should be stated. Quoting at exactly full scale measures an overload; quoting well inside it wastes signal-to-noise that a fair comparison would have. The half-decibel convention exists precisely because the ambiguity above is real, and a part characterised at −0.5 dBFS and one characterised at 0 dBFS are not comparable at eight or ten bits and are nearly comparable at sixteen.

Two boundaries confused for one, again #

The pattern this essay is an instance of appeared first in the semiconductor field and is worth naming as a pattern, because the digital field has now produced its own case.

How small is small signal measured two boundaries on one device: the drive at which distortion reaches one per cent, and the drive at which the gain is one per cent low. They are 1.03 mV and 7.30 mV — 7.06 times apart — and the ratio moves by √10 for every decade of tolerance, exactly as a first-order effect against a second-order one predicts. The differential pair reverses them, and the reversal is a fact about which harmonic the device is able to produce rather than about how good it is.

A converter has the same structure. Its quantisation boundary is a floor in amplitude, at q/√12, and it is second order in nothing — it does not move with the drive at all. Its overload boundary is a ceiling at full scale, and it is abrupt. Between them the ratio is exactly what the formula says. The failure mode this essay describes is what happens when a measurement is taken at the ceiling and reported as a measurement of the floor.

Two intermediate drives fill in the axis between those two pictures, and they are worth having because the transition is not the shape a reader expects. The number of clipped samples falls smoothly from 327 to zero as the drive backs off; the cost of those samples falls much faster, and disappears into the measurement’s own scatter while a fifth of the record is still clipping.

The general instruction the site draws from this is short and it has now been earned three times. When a measurement disagrees with a closed form, measure the closed form’s hypothesis before explaining the disagreement. Here the hypothesis was “a full-scale sinusoid”, the hypothesis was not met, and the explanation that stood in its place — that the formula is an approximation — was sitting on a defect nobody had looked for because the number it produced was small and plausible.

One number this essay leaves stated rather than measured #

The 1.76 dB is a constant and has been carried through as one. It is worth saying where it comes from and why nothing here tests it separately, since the site’s habit is to be suspicious of constants.

It is 10 log₁₀(1.5), and the 1.5 is the ratio of a sinusoid’s mean square to the mean square of a uniform distribution over the same peak-to-peak span: a sinusoid puts half its squared amplitude into its mean square and a uniform distribution puts a third. So the term is not about converters at all — it is about the waveform being converted, and it changes if the waveform does.

That is not a small caveat and it is measurable. A full-scale triangle wave, whose amplitude distribution genuinely is uniform, has a mean square of A²/3 rather than A²/2, so its ratio is 6.02N dB exactly — 1.76 dB worse than a sinusoid’s at every bit count, for a signal using exactly the same range of codes. A square wave, whose root mean square is its amplitude, comes out at 6.02N + 4.77 dB, which is 3.01 dB better than the sinusoid for the same reason read the other way.

So the number on a data sheet is a number about a sinusoid, and comparing it with the performance of a system carrying anything else needs the crest factor of the actual signal. This site has made that observation once already in a completely different field: the power factor of a rectifier’s current is 0.780 while its displacement factor is exactly 1.000000, because a phasor calculation describes a sinusoid and the current is not one. Both are cases of a figure of merit defined on a waveform and quoted about a system.

What this essay measured, then, is narrower than the formula and firmer: for a coherent sinusoid one step inside full scale, at five bit counts and three record lengths, 6.02N + 1.76 dB is what a quantiser does, to within the scatter of the measurement — and the shortfall everybody sees is half a step of overload that happens somewhere else.

The narrowness matters because three later results in this field are stated against this number and each of them widens the conditions differently. When a floor stops being a floor removes the amplitude condition and finds the formula’s premise failing rather than the formula: at 1.3 levels of amplitude 76 per cent of the error’s power sits in harmonics of the input, with the total exactly where it was. A picosecond, read as bits removes the assumption that the sampling instants are the right ones, and adds a term that depends on input frequency and on nothing else about the part. And the floor a converter sets removes the assumption that the quantiser is the only thing contributing, crossing it against the Johnson noise of the source at 18.80 bits.

Read together, those say that the shortfall everybody attributes to the formula being approximate has at least four separate causes, three of which are real and none of which is the formula. Which is the argument for measuring the clean case first and exactly: an expression that is right to 0.15 dB under stated conditions is a usable reference to subtract from, and one believed to be approximate is not a reference at all.