Demod 2400x

AI-DISCLOSURE.md

@@ -0,0 +1,39 @@
+# Use of generative AI on this branch
+
+This branch (`demod-2400x`) contains code and text written with Claude Code
+(Anthropic, Claude Opus models), working under my direction.
+
+## Who did what
+
+The DEMOD_2400X design is mine: 12 MHz polyphase upsampling of the I/Q, a
+magnitude preamble metric at every 12 MHz position, keeping I/Q in the FIFO
+for later coherent processing, and the split into starch kernels. So is the
+initial implementation.
+
+Claude Code wrote code to my specification and review, and ran tests and
+measurements I asked for. I made the design decisions along the way, for
+example integer arithmetic over floating point, generic kept as the plain
+reference implementation, equal coefficient sums across phases, the
+upsampler rounding shift, and which experiments to drop.
+
+Commits with a `Co-Authored-By: Claude` trailer were written with Claude
+Code. Some earlier commits without the trailer also contain AI-written code,
+notably the vectorised `locate_preamble`, the exact integer square root, the
+NEON `upsample12` and the DC removal in the input converters
+(`3a7afd3`, `5a51c13`, `d70f5d4`, `0f244bc`). The `sim/` tools,
+README-demod2400x.md and this file were also written with Claude Code.
+
+## How it was checked
+
+* Every SIMD kernel is checked bit-for-bit against the generic version by
+  `starch-benchmark -V` (`make test`), on AArch64, ARMv7 and x86 AVX2,
+  including full-range worst-case input and a 64-bit check of the generic
+  upsampler.
+* Decoded output was compared with stock dump1090-fa on an RTL-SDR
+  recording (FlightAware Pro Stick Plus), message by message (contents,
+  signal level and timestamps). An SDRplay recording was used to check that
+  each change gave the same or more messages than the version before.
+* Synthetic frames at known times (`sim/`) check sensitivity and MLAT
+  timestamp accuracy.
+* The build has run live on my Raspberry Pi 4 receiver feeding FlightAware,
+  with MLAT synchronized as with the stock build.

Makefile

@@ -1,11 +1,12 @@
 PROGNAME=dump1090
 
-DUMP1090_VERSION ?= unknown
+DUMP1090_VERSION ?= 11.1x
 
 CFLAGS ?= -O3 -g
-DUMP1090_CFLAGS := -std=c11 -fno-common -Wall -Wmissing-declarations -Werror -Wformat-signedness -W
+DUMP1090_CFLAGS := -std=c11 -fno-common -Wall -Wmissing-declarations -Wformat-signedness -W
 DUMP1090_CPPFLAGS := -I. -D_POSIX_C_SOURCE=200112L -DMODES_DUMP1090_VERSION=\"$(DUMP1090_VERSION)\" -DMODES_DUMP1090_VARIANT=\"dump1090-fa\"
-
+DUMP1090_CPPFLAGS += -DDEMOD_2400X -DUSE_MAGNITUDE
+
 LIBS = -lpthread -lm
 SDR_OBJ = cpu.o sdr.o fifo.o sdr_ifile.o dsp/helpers/tables.o
 
@@ -205,7 +206,7 @@ showconfig:
 %.o: %.c *.h
 	$(CC) $(ALL_CCFLAGS) -c $< -o $@
 
-dump1090: dump1090.o anet.o interactive.o mode_ac.o mode_s.o comm_b.o net_io.o crc.o demod_2400.o stats.o cpr.o icao_filter.o track.o util.o convert.o ais_charset.o adaptive.o $(SDR_OBJ) $(COMPAT) $(STARCH_OBJS)
+dump1090: dump1090.o anet.o interactive.o mode_ac.o mode_s.o comm_b.o net_io.o crc.o demod_2400.o demod_2400x.o stats.o cpr.o icao_filter.o track.o util.o convert.o ais_charset.o adaptive.o $(SDR_OBJ) $(COMPAT) $(STARCH_OBJS)
 	$(CC) -g -o $@ $^ $(LDFLAGS) $(LIBS) $(LIBS_SDR) $(LIBS_CURSES)
 
 view1090: view1090.o anet.o interactive.o mode_ac.o mode_s.o comm_b.o net_io.o crc.o stats.o cpr.o icao_filter.o track.o util.o ais_charset.o sdr_stub.o $(COMPAT)

NOTES-demod2400x.md

@@ -0,0 +1,144 @@
+# DEMOD_2400X development notes
+
+What was built, why, and what was tried and dropped. Results are in
+[README-demod2400x.md](README-demod2400x.md). Times are for the 60 s Pi 4
+RTL-SDR capture decoded on a Raspberry Pi 5 unless stated otherwise.
+
+## Test data
+
+* **SDRplay RSP1B recording**: 2.4 MHz SC16 I/Q, 17.8 s of signal. Used for
+  the early development and speed work. The stock demodulator crashes on
+  RSP1B input; that is a separate issue, not fixed on this branch.
+* **RTL-SDR recording**: 60 s at 2.4 MHz, UC8, from the live receiver: a
+  Raspberry Pi 4 with the FlightAware Pro Stick Plus (blue, internal LNA and
+  SAW filter) at gain 49.6 dB, outdoor antenna on a 3 m pole, 12 m of
+  LMR400. Used for the stock comparison, the later tuning and CPU figures.
+* **Synthetic frames** from `sim/gen_timing.c` (based on `gen_data.c`):
+  DF17 frames at known 12 MHz ticks with noise and a random carrier offset,
+  for sensitivity and MLAT timestamp accuracy.
+* **Live**: the same Pi 4 receiver feeding FlightAware, including MLAT.
+
+## Signal path
+
+1. **Input conversion** (`iqdata_conv_uc8/sc16/sc16q11`, `convert.c`).
+   Converts the SDR samples to 12-bit signed I/Q and removes DC in the same
+   pass. The DC filter holds its correction for blocks of 16 samples
+   (`IQ_DC_BLOCK`) and updates a running mean with a time constant of 4096
+   samples, so SIMD versions need no serial loop. The filter state lives in
+   `convert.c` (`converter_state`) and is passed in and out (`dc_in`,
+   `dc_out`), never static, which also keeps benchmark verification
+   repeatable. Mean level and power are computed after DC removal. The FIFO
+   holds I/Q (`magbuf_sample()`), so later stages, such as a coherent
+   demodulator, can reuse it.
+2. **Upsampling** (`upsample12`). A 5-phase, 8-tap polyphase FIR takes
+   2.4 MHz I/Q to 12 MHz, rounds by `UPSAMPLE_SHIFT` and takes the exact
+   integer magnitude, `floor(sqrt(I² + Q²))`. Output is stored as 6 rows, one
+   per 12 MHz tick within a 0.5 µs chip.
+3. **Preamble search** (`locate_preamble`). The metric
+   `512·minp / (2·minp + gap + 1)` at every 12 MHz position, from the
+   smallest of the 4 preamble pulses and the sum of the 12 gaps. Below
+   `PEAMBLE_THRES` it is set to 0 without dividing, which skips most noise.
+4. **Candidates** (`demod_2400x.c`). Peaks are scanned 8 metric bytes at a
+   time (a SWAR test against the threshold). For each peak, the first 56 bits
+   are decoded at the peak and its two neighbours, and the best quality wins
+   (`bestPhase`). The second 56 bits are always decoded too, because CRC
+   correction can change the DF field, including the bit that selects a
+   56- or 112-bit frame. A long frame other than DF17/18 is rejected if its
+   second half quality is low.
+5. **Bit decoding** (`decode_bits`). Bit = 1 if the first chip of the pair is
+   larger. Quality = 256·Σ|a−b| / Σ(a+b).
+6. **Output** (`emitFrame`). Scored and decoded by the stock code. Then:
+   * Fine arrival time: the decoded chips are correlated with the signal at
+     `pos−2 … pos+2` and the best offset is used for the MLAT timestamp.
+   * Signal level from the I/Q, with each sample's power capped at full
+     scale as the stock magnitude kernels do.
+   * Adaptive gain gets 2.4 MHz magnitudes and each decoded message, fed the
+     same way as the stock demodulator (only when adaptive gain is on).
+
+## Decisions and why
+
+| Change | Why | Effect |
+|---|---|---|
+| DC removal in the converter | One pass over the data; FIFO keeps clean I/Q for later use | Fixed an overlap write-offset bug on the way; 3325 → 3332 messages on the SDRplay recording |
+| Always decode 112 bits | CRC correction can fix the DF bits, so length is only known after scoring | No lost long frames |
+| Skip the metric divide below threshold | `floor(n/d) ≥ T ⇔ n ≥ T·d`; peaks below T are never used | ~1570 → ~1490 ms (SDRplay recording) |
+| AVX2 divide/sqrt corrected both ways | `-ffast-math` lets the compiler use reciprocal estimates; results were one too small on real Intel hardware | Validation passes on Intel |
+| Fine arrival time | Bit quality is flat over ~3 ticks on a clean pulse, so `pos` was biased (0 to +2 ticks by phase) and jittery (σ 0.88) | Exact on synthetic frames; MLAT synced with ~260 receivers live |
+| Least-squares coefficients | The windowed sinc had uneven delay and gain across phases | Weak synthetic frames 1205 → 1405 |
+| All phases sum to 128 | Less rounding error; still fits the overflow bound | Delay error 0.046 → 0.028 samples |
+| `DATA_THRES` 80 → 65 | More messages, no extra CPU or junk tracks | +0.2% messages |
+| `UPSAMPLE_SHIFT` 5 → 4 | One more bit for weak signals; power still < 2³¹ | Results unchanged within noise |
+| Mirror-symmetric upsampler (NEON, AVX2) | Phase 4−c is phase c reversed, so pairs share sums `x[k] ± x[7−k]`: half the multiplies | `upsample12` 2.48 → 1.54 ms per call |
+| `decode_bits` starch kernel | Called 5 times per candidate, ~2M candidates a minute | 82 → 26 ns per call; demod 4.8 → 4.3 s |
+| Adaptive gain fed | It was not called at all by the new demodulator | Same gain decisions as stock |
+| Signal power capped at full scale | Clipped frames reported up to +1.1 dBFS | RSSI matches stock (mean +0.01 dB, σ 0.2 dB) |
+
+### Overflow bound
+
+Input is 12-bit (|x| ≤ 2048). Each filter output is at most
+`2048·UPSAMPLE_ABS_SUM >> UPSAMPLE_SHIFT`, and the power I² + Q² must stay
+below 2³¹ for signed 32-bit arithmetic and the exact square root. With
+`UPSAMPLE_ABS_SUM` 244 (the largest Σ|tap|, printed by `sim/ls_coeffs.py`) and
+shift 4, the worst case is 1.95·10⁹. A `_Static_assert` in
+`demod2400x_coeffs.h` enforces it, and the benchmark runs worst-case input.
+
+### Bit-exactness
+
+Generic is the reference; NEON and AVX2 must match it exactly, which
+`starch-benchmark -V` checks. That means exact methods only:
+
+* Square root: float sqrt, then integer checks in both directions.
+* Metric divide: float divide, then integer checks in both directions.
+* The mirror trick is exact integer arithmetic. Generic keeps the direct form.
+
+## Tried and dropped
+
+* **2.4 MHz pre-detector gate** (skip upsampling where nothing is
+  happening): real frames occur at 0.75–1.25 times the mean magnitude, so a
+  gate cheap enough to help misses frames.
+* **Sharpened upsampler** (`coeffs.py` parameter `a`): amplifies noise more
+  than it sharpens pulses; weak frames 1205 → 616 at `a = 0.1`.
+* **Lower `PEAMBLE_THRES`**: 50 → 40 gives +1.2% messages for +64% CPU;
+  at 30, false aircraft appear.
+* **Soft-decision bit flipping** (flip the weakest bits when the CRC fails):
+  `--fix` already repairs any single bit, so this only helps 2-bit errors,
+  at a false-decode risk. Not pursued.
+
+## Testing
+
+* `make test`: `cprtests` and `starch-benchmark -V`. Each kernel is checked
+  against generic, including full-range worst-case input for `upsample12`
+  and a 64-bit check of the generic upsampler.
+* Cross-builds: ARMv7 (`ARCH=armv7l CC=arm-linux-gnueabihf-gcc
+  LDFLAGS=-static`, runs natively on AArch64) and x86
+  (`ARCH=x86_64 CC=x86_64-linux-gnu-gcc`, run with `qemu-x86_64 -cpu max`,
+  correctness only; QEMU computes reciprocal estimates exactly, so it can't
+  reproduce `-ffast-math` problems).
+* Real data: stock and DEMOD_2400X on the same capture, compared message by
+  message (contents, RSSI, timestamps).
+* Synthetic data: `sim/gen_timing.c`. Frames need a carrier offset: at
+  exactly 0 Hz the DC filter removes part of the signal.
+* Adaptive gain with `--ifile` needs a gain table; for testing, a fake R820T
+  table was patched into a scratch copy of `sdr.c` (not committed).
+
+## Gotchas
+
+* `dsp/generated/*.o` don't depend on `dsp/helpers/*.h`. After a header edit,
+  run `rm dsp/generated/*.o`, or generic keeps old code and verification
+  compares against stale results.
+* After adding a starch function or changing a wisdom file, rerun
+  `python3 dsp/starchgen.py .`; the wisdom order is compiled into
+  `dispatcher.c`.
+* After changing the version string, run `make clean`: objects don't depend
+  on the Makefile.
+* starch finds `STARCH_IMPL` by regex, so implementations can't be generated
+  by macros.
+* starch doesn't pass a kernel's return value to its verifier, so
+  `decode_bits` returns its quality through a pointer.
+
+## Possible next steps
+
+* Coherent demodulation as a fallback when magnitude demodulation fails,
+  reusing the I/Q kept in the FIFO.
+* An early DF check before decoding the second half and scoring.
+* Real x86 timings for `wisdom.x86` (`decode_bits` order is provisional).

README-demod2400x.md

@@ -0,0 +1,115 @@
+# DEMOD_2400X demodulator
+
+This branch adds a new Mode S demodulator for 2.4 MHz SDR input
+(`demod_2400x.c`, enabled with `-DDEMOD_2400X -DUSE_MAGNITUDE` in the
+Makefile; the build reports version `11.1x`). In outline:
+
+* DC removal and conversion to 12-bit I/Q in the input converter, so the
+  FIFO keeps I/Q samples for later processing.
+* A 5-phase polyphase FIR upsamples to 12 MHz (least-squares fractional-delay
+  coefficients, all integer arithmetic), then takes the magnitude.
+* A preamble metric at every 12 MHz position, and the 112 bits are decoded
+  from the best of the neighbouring positions.
+* The arrival time is refined by correlating the decoded chips with the
+  signal, so MLAT timestamps are unbiased across sub-sample phases.
+* Adaptive gain and signal level statistics behave as in the stock
+  demodulator.
+
+The heavy kernels (`iqdata_conv_*`, `upsample12`, `locate_preamble`,
+`decode_bits`) are starch kernels with generic, ARM NEON (ARMv7 and AArch64)
+and x86 AVX2 versions. All versions give bit-identical results, which
+`starch-benchmark -V` checks, including on full-range worst-case input.
+
+## Results
+
+A 60 s capture from a Raspberry Pi 4 with a FlightAware Pro Stick Plus
+(gain 49.6 dB), with an outdoor antenna on a 3 m pole and 12 m of LMR400,
+decoded by stock dump1090-fa 11.1 and by this branch with the same options
+(`--fix`, receiver position set). Development also used an SDRplay
+recording and synthetic frames; see
+[NOTES-demod2400x.md](NOTES-demod2400x.md#test-data).
+
+| | Stock | DEMOD_2400X | Change |
+|---|---|---|---|
+| Usable messages | 73,146 | 84,389 | +15% |
+| DF17 (ADS-B) | 10,352 | 11,795 | +14% |
+| Global CPR positions | 2,807 | 3,750 | +34% |
+| Aircraft | 136 | 141 | +5 |
+
+The gain is concentrated in weak signals. Messages by signal strength:
+
+| RSSI (dBFS) | Stock | DEMOD_2400X | Change |
+|---|---|---|---|
+| −30 to −27 | 131 | 302 | +131% |
+| −27 to −24 | 2,347 | 4,205 | +79% |
+| −24 to −21 | 7,111 | 9,407 | +32% |
+| −21 to −18 | 9,010 | 10,759 | +19% |
+| −18 to −15 | 12,213 | 13,703 | +12% |
+| −15 to −10 | 18,226 | 19,924 | +9% |
+| −10 to −5 | 9,952 | 10,710 | +8% |
+| above −5 | 14,151 | 15,375 | +9% |
+
+Position messages by distance from the receiver:
+
+| Distance | Stock | DEMOD_2400X | Change | Aircraft (stock / new) |
+|---|---|---|---|---|
+| 0–50 km | 643 | 792 | +23% | 12 / 13 |
+| 50–100 km | 1,225 | 1,520 | +24% | 30 / 30 |
+| 100–150 km | 746 | 957 | +28% | 33 / 33 |
+| 150–200 km | 356 | 472 | +33% | 22 / 22 |
+| 200–250 km | 93 | 130 | +40% | 8 / 13 |
+| 250–300 km | 11 | 24 | +118% | 2 / 5 |
+
+Weak synthetic frames (2,000 DF17 frames with a random ±100 kHz carrier
+offset, decoded out of 2,000): 689 (stock) against 1,404 (DEMOD_2400X).
+The generator and checker are in [sim/](sim/).
+
+## Timing (MLAT)
+
+On synthetic frames at known 12 MHz ticks, covering all 5 sub-sample
+phases:
+
+| | Stock | DEMOD_2400X |
+|---|---|---|
+| Bias across phases (clean) | 0 to +2 ticks | none |
+| Jitter, noisy frames (σ) | 0.41–0.48 ticks | < 0.2 ticks |
+
+One tick is 83.3 ns. Running live on the Pi 4, mlat-client stays synchronized
+with about 260 nearby receivers, as with stock.
+
+## CPU
+
+Demodulation time for the 60 s capture:
+
+| | Stock | DEMOD_2400X |
+|---|---|---|
+| Raspberry Pi 5 (Cortex-A76) | 2.0 s | 4.3 s (gcc 14), 4.1 s (clang 19) |
+| Raspberry Pi 4 (Cortex-A72) | 3.9 s | 10.6 s |
+
+The Pi 4 times were measured while the live receiver was also running on
+the same Pi, so both are probably slightly high. Live on the Pi 4 this is
+about 31% of one core. These results come from
+one receiver and a one-minute capture; longer side-by-side runs are in
+progress.
+
+## Compilers
+
+Built with the same options as the GitHub workflow
+(`make RTLSDR=yes BLADERF=yes HACKRF=yes LIMESDR=yes SOAPYSDR=yes all`,
+then `make test`):
+
+| Compiler | Target | Warnings | `make test` |
+|---|---|---|---|
+| gcc 14.2 | AArch64 (Pi 5) | 0 | pass (65 validations) |
+| clang 19.1 | AArch64 (Pi 5) | 0 | pass (65 validations) |
+| gcc 14.2 (cross) | ARMv7 | 0 | `starch-benchmark -V` passes, run natively on AArch64 |
+| gcc 14.2 (cross) | x86-64 AVX2 | 0 | pass under qemu-x86_64 |
+| clang 19.1 (cross) | x86-64 AVX2 | 0 | pass under qemu-x86_64 (59 validations) |
+
+The gcc and clang AArch64 builds give byte-identical decoder output,
+timestamps included. On x86, clang with `-ffast-math` compiles every AVX2
+divide and square root to a reciprocal estimate (`vrcpps`/`vrsqrtps`), so the
+kernels correct the estimate in both directions to stay exact. QEMU computes
+these estimates exactly, so x86 correctness also needs a run on real
+hardware. The AVX2 kernels passed on an Intel CPU before the mirrored
+upsampler and `decode_bits` were added; those still need an Intel run.

README.md

@@ -16,6 +16,26 @@ it can be used to contribute crowd-sourced flight tracking data to FlightAware.
 It is designed to build as a Debian package, but should also be buildable on
 many other Linux or Unix-like systems.
 
+## DEMOD_2400X demodulator (this branch)
+
+This branch adds an alternative Mode S demodulator for 2.4 MHz SDR input
+(`demod_2400x.c`, enabled with `-DDEMOD_2400X -DUSE_MAGNITUDE` in the
+Makefile; the build reports version `11.1x`). It upsamples the I/Q to
+12 MHz with a polyphase filter, finds preambles at every 12 MHz position and
+refines the arrival time for MLAT. The heavy kernels are starch functions
+with generic, NEON and AVX2 versions, checked bit-identical by
+`starch-benchmark -V`.
+
+On a 60 s Raspberry Pi 4 RTL-SDR capture it decodes about 15% more messages
+than stock dump1090-fa, mostly weak signals from distant aircraft, for about
+2 to 3 times the CPU.
+
+* [README-demod2400x.md](README-demod2400x.md): design and results
+* [NOTES-demod2400x.md](NOTES-demod2400x.md): development notes, decisions and
+  what was tried
+* [AI-DISCLOSURE.md](AI-DISCLOSURE.md): use of generative AI on this branch
+* [sim/](sim/): coefficient generator and test-signal tools
+
 ## Building under bullseye, buster, or stretch
 
 ```bash