Research · benchmarks · evidence

Benchmark evidence

Build snapshot · Not live

A decision surface for measured results, the contract each one is judged under, and the limits it inherits. The raw artefacts under benchmarks/reports/ remain the source of truth, and every card here links back to the file it was read from.

Build snapshot Not live Captured 2026-08-10T07:58:14.256Z Source benchmarks/reports Ingested 2026-08-10T07:58:14.236Z 44 JSON files seen

This page is a build snapshot, not a live reading

No exporter produces these numbers. They are files written by bench harnesses run by hand and read off disk when this page was built, so nothing on this page moves until the next build.

Captured 2026-08-10T07:58:14.256Z · ingestion pass 2026-08-10T07:58:14.236Z · source directory ~/Ulap/sidsense-run/wg3-workitem6-2026-demo/benchmarks/reports · 44 JSON files seen. Those are three different times and neither of the first two is any run's measured-at, which is read only from an instant field inside the artefact itself. A file's mtime is a fourth time and is shown only as a file fact.

Evidence summary

Indexed artefacts

Every one of the 44 JSON files found was indexed.

Governed suites

11 / 14 contracts

Every indexed artefact is claimed by a suite contract.

Measured-at present

29 / 44

15 runs carry no measurement instant and cannot be placed on a timeline.

Held-out evaluations

Dataset, split, preprocessing and generator hashes are blocking metadata here and none are present, so no ranking between them is expressible.

Over-the-air evaluations

11 of 12 raise antenna-band mismatch, which every vacant-channel and dynamic-range claim inherits. It does NOT explain a missed occupied channel: the two strongest carriers in the 2026-08-04 survey, 42 dB over the floor, were still called Noise on 20 of 20 records.

Hover any counter for the exact rule that produced it. None of these is a score: there is no aggregate pass or fail across suites on this page, because the suites do not measure the same kind of thing and a single number over them would not mean anything.

What the evidence currently supports

11 suites worst verdict first

One card per suite contract, ordered by the verdict its own gates produced. The headline number is read off a single named run, and the rule that picked that run is printed in the drawer beside it.

TVWS occupancy on real over-the-air captures

Fails safety gate

0.4 fraction record-level occupancy rate

All 3 runs fail a safety-critical gate.

Proves: Does the sensing model call a licensed, occupied multiplex OCCUPIED, on real captured RF?
Does not: Nothing here establishes anything about vacant channels: no vacant captures exist in this set.

Over the air 3 runs n=100 7 blocked claims

RF channel quality of the capture set

Fails gate

Value absent *.est_in_window_snr_db — 5 candidates

1 of 2 runs fails a blocking gate; 1 is incomplete — something needed to judge it is absent.

Proves: How much signal is actually in each captured channel, relative to a control floor?
Does not: In-window SNR figures are ESTIMATES: no control IQ was retained, only a scalar floor.

Over the air · Lab RF 2 runs 3 blocked claims

TVWS per-epoch OTA diagnostic

Fails gate

Value absent Spearman: synthetic val acc vs real OTA

All 4 runs fail a blocking gate.

Proves: Does synthetic validation accuracy track real-OTA behaviour across training epochs?
Does not: Diagnostic, not a selection criterion — n=5 channels, all occupied.

Over the air 4 runs 3 blocked claims

cuPHY LDPC decode latency

Fails gate

229.5 us cuphy_decode p50

All 2 runs fail a blocking gate.

Proves: How long does ONE cuPHY LDPC decode take, timed on the decoder’s own CUDA stream?
Does not: LDPC decode only — no MAC, no scheduler, no fronthaul, no cell, no OTA.

Hardware in loop 2 runs n=1000 3 blocked claims

cuPHY LDPC under a capped MPS AI tenant

Fails gate

Value absent *.p50 — 2 candidates

The single run fails a blocking gate.

Proves: What happens to cuPHY LDPC invocation-average latency while a capped MPS-managed AI client loads the same GPU?
Does not: CUDA_MPS_ACTIVE_THREAD_PERCENTAGE PROVISIONS but does not RESERVE: kernels from different clients may still execute on the same SM.

Hardware in loop 1 run 7 blocked claims

TVWS checkpoint selection, calibration and OOD

Incomplete

0.241781 Spearman: synthetic val acc vs real records

All 4 runs are incomplete — something needed to judge them is absent.

Proves: Which epoch shipped, how confident is it, and how does it behave off-distribution?
Does not: There are ZERO real vacant captures, so no false-positive rate on real RF can be measured at all.

Over the air 4 runs 4 blocked claims

GPU partitioning mechanism probe

Incomplete

Value absent fp16 matmul throughput

All 2 runs are incomplete — something needed to judge them is absent.

Proves: Does the partitioning mechanism itself do what its documentation says?
Does not: Concurrent non-interference was NOT demonstrated: two instances produced no output when all three ran at once.

Hardware in loop 2 runs 2 blocked claims

LDPC latency under a stepped AI load

Incomplete

Value absent *.p90 — 4 candidates

All 2 runs are incomplete — something needed to judge them is absent.

Proves: How does LDPC decode latency move as an AI tenant’s load steps up on the same GPU?
Does not: LDPC decode only: no fronthaul, no MAC, no scheduler, no cell.

Hardware in loop 2 runs 6 blocked claims

Sensing generalization across synthetic conditions

Incomplete

Value absent *.accuracy — 7 candidates

All 15 runs are incomplete — something needed to judge them is absent.

Proves: How does the served sensing model behave as synthetic conditions move away from the training distribution?
Does not: This is NOT over-the-air validation and does not substitute for it. A model can pass every condition here and still collapse on air.

Deployed service 15 runs 2 blocked claims

Synthetic held-out classification accuracy

Incomplete

0.88642 fraction overall accuracy

All 6 runs are incomplete — something needed to judge them is absent.

Proves: How accurately does the checkpoint classify a held-out split of the SAME generator?
Does not: synthetic-only; NOT OTA; NOT FCC.

Synthetic 6 runs n=1620 3 blocked claims

Training history

Incomplete

0.951111 fraction best validation accuracy

All 3 runs are incomplete — something needed to judge them is absent.

Proves: What did the training curve do, and which epoch was best on the validation split?
Does not: A validation maximum is a selection score, not a test score.

Synthetic 3 runs 3 blocked claims

Evidence quality

17 blockers

65.9%

Measured-at coverage

Metadata coverage only

15 runs carry no measurement instant and cannot be placed on a timeline.

Identity survives only in the filename — warning

A rename loses the identity; the artefact is not self-describing.
30
RF front end not matched to the band — warning

Every spectral claim inherits that mismatch.
12
Selection set scored as a test set — warning

The score is optimistically biased and is not a test result.
4
Uncontrolled tenants on the GPU — warning

The run is not an isolated measurement.
3
No producing harness — warning

It is hand-authored and cannot be regenerated.
1
Partial run — warning

Cells never measured are ABSENT, not passing. No envelope may be read.
1
No dataset hash — missing

Two runs cannot be shown to have used the same inputs.
44
No harness revision — missing

The measurement is not reproducible from the artefact alone.
43
No workload hash — missing

Two runs cannot be shown to have run the same protocol.
40
Summary statistics only — missing

No percentile beyond the ones written down can be derived.
36
No environment recorded — missing

The number cannot be attributed to any machine.
29
No measurement timestamp — missing

It cannot be placed on a timeline. File mtime is not a substitute.
15
Metrics explicitly null — missing

Those numbers are absent. Rendering them as 0 would be a fabrication.
5
An empty list that was never collected — missing

Empty does not mean the condition was absent.
2
Checkpoint not bound to an endpoint — scope note

Model identity is provenance, not proof.
30
Decode-only microbenchmark — scope note

It is a lower bound on real work, never a system latency.
5
Percentiles of invocation averages — scope note

These are percentiles of averages, and cannot support tail claims.
1

Evidence maturity

by tier

How close to the real world each artefact was collected. A tier is a fact about the collection method, not a grade.

Over the air RF channel quality of the capture set · TVWS checkpoint selection, calibration and OOD · TVWS occupancy on real over-the-air captures · TVWS per-epoch OTA diagnostic 12
Lab RF RF channel quality of the capture set 1
Hardware in loop GPU partitioning mechanism probe · LDPC latency under a stepped AI load · cuPHY LDPC decode latency · cuPHY LDPC under a capped MPS AI tenant 7
Deployed service Sensing generalization across synthetic conditions 15
Synthetic Synthetic held-out classification accuracy · Training history 9

What is deliberately not here

honesty

No aggregate pass or fail across suites. The suites measure different things under different contracts, and one number over them would mean nothing.
No trend line across dated runs. The runs differ in host, harness revision and scope, so a line through them would imply a comparison nobody made.
No ranking between runs unless the comparator proves their identity matches. Where it does not, the drawer names the field that decided it.
No percentile inferred from a summary. A p99 is offered only where the artefact retained the raw sample distribution the contract requires.

Benchmark suites

11 shown

Scope

Benchmark suites: scope, latest comparable result, decision status, run count and a control that opens the full detail for each suite.
Suite	Scope	Latest comparable result	Decision status	Runs
TVWS occupancy on real over-the-air captures tvws-ota-occupancy@1	Over the air	0.4 fraction Not comparable: 12 are absent from the artefacts.	Fails safety gate	3
RF channel quality of the capture set rf-channel-quality@1	Over the air · Lab RF	No measured-at in this suite absent	Fails gate	2
TVWS per-epoch OTA diagnostic tvws-ota-per-epoch@1	Over the air	absent Not comparable: 12 are absent from the artefacts.	Fails gate	4
cuPHY LDPC decode latency cuphy-ldpc-decode@1	Hardware in loop	229.5 us Not comparable: 1 identity field differs and 5 are absent.	Fails gate	2
cuPHY LDPC under a capped MPS AI tenant mps-cotenancy@1	Hardware in loop	absent This suite holds a single run. There is nothing to compare it with, so no comparison is offered.	Fails gate	1
TVWS checkpoint selection, calibration and OOD tvws-selection-calibration@1	Over the air	0.241781 Not comparable: 1 identity field differs and 11 are absent.	Incomplete	4
GPU partitioning mechanism probe gpu-partition-mechanism@1	Hardware in loop	No measured-at in this suite absent	Incomplete	2
LDPC latency under a stepped AI load ldpc-step-load@1	Hardware in loop	No measured-at in this suite absent	Incomplete	2
Sensing generalization across synthetic conditions sensing-generalization@1	Deployed service	absent Not comparable: 1 identity field differs and 11 are absent.	Incomplete	15
Synthetic held-out classification accuracy heldout-accuracy@1	Synthetic	No measured-at in this suite 0.88642 fraction	Incomplete	6
Training history training-history@1	Synthetic	No measured-at in this suite 0.951111 fraction	Incomplete	3

Every indexed artefact

44 files

The complete corpus, including the artefacts that normalised to nothing useful. An artefact is never dropped for being unparseable, unknown or invalid — it appears here, flagged, and stays addressable.

Every artefact file indexed from benchmarks/reports.
File	Schema	Measured at	Status	Size	Raw
ch39-618mhz-quality-2026-07-31.json	channel-quality@1	No measured-at field	Fails gate	15.1 kB	Raw JSON for ch39-618mhz-quality-2026-07-31.json
cuphy-per-slot-20260729-005456.json	cuphy-per-slot@2	2026-07-28T21:54:56.294903+00:00 (recorded_at)	Fails gate	17.4 kB	Raw JSON for cuphy-per-slot-20260729-005456.json
cuphy-per-slot-20260730-080736.json	cuphy-per-slot@3	2026-07-30T08:07:36.334607+00:00 (recorded_at)	Fails gate	17.9 kB	Raw JSON for cuphy-per-slot-20260730-080736.json
detector-scores-2026-08-03.json	detector-scores@unversioned	No measured-at field	Incomplete	609.0 kB	Raw JSON for detector-scores-2026-08-03.json
fixC-training-history-2026-07-30.json	training-history@unversioned	No measured-at field	Incomplete	5.5 kB	Raw JSON for fixC-training-history-2026-07-30.json
heldout-committed-8eca022f-2026-07-31.json	heldout@unversioned	No measured-at field	Incomplete	1.7 kB	Raw JSON for heldout-committed-8eca022f-2026-07-31.json
heldout-committed-8eca022f-IMAGE-APP-SRC-trap-2026-07-31.json	heldout@unversioned	No measured-at field	Incomplete	1.6 kB	Raw JSON for heldout-committed-8eca022f-IMAGE-APP-SRC-trap-2026-07-31.json
heldout-committed-8eca022f-prefixgen-17517e57-2026-07-31.json	heldout@unversioned	No measured-at field	Incomplete	1.6 kB	Raw JSON for heldout-committed-8eca022f-prefixgen-17517e57-2026-07-31.json
heldout-v1-828dc46e-gen-9dc4f892-2026-07-31.json	heldout@unversioned	No measured-at field	Incomplete	1.6 kB	Raw JSON for heldout-v1-828dc46e-gen-9dc4f892-2026-07-31.json
heldout-v1-828dc46e-headgen-c23711ab-2026-07-31.json	heldout@unversioned	No measured-at field	Incomplete	1.6 kB	Raw JSON for heldout-v1-828dc46e-headgen-c23711ab-2026-07-31.json
heldout-v1-828dc46e-prefixgen-17517e57-2026-07-31.json	heldout@unversioned	No measured-at field	Incomplete	1.6 kB	Raw JSON for heldout-v1-828dc46e-prefixgen-17517e57-2026-07-31.json
mig-h200-2026-08-03.json	mig-partition@unversioned	No measured-at field	Incomplete	1.8 kB	Raw JSON for mig-h200-2026-08-03.json
mps-cap-result.json	mps-cap@unversioned	No measured-at field	Incomplete	1.0 kB	Raw JSON for mps-cap-result.json
mps-sweep-2026-08-03-livebox.json	mps-sweep@3	2026-08-03T05:30:52.713622+00:00 (recorded_at)	Fails gate	5.1 kB	Raw JSON for mps-sweep-2026-08-03-livebox.json
ota-eval-fixB.json	ota-eval@1	2026-07-30T07:33:25.584102+00:00 (measured_at)	Fails safety gate	3.3 kB	Raw JSON for ota-eval-fixB.json
ota-eval-fixC-final.json	ota-eval@1	2026-07-30T07:55:11.979117+00:00 (measured_at)	Fails safety gate	3.3 kB	Raw JSON for ota-eval-fixC-final.json
ota-eval-fixC.json	ota-eval@1	2026-07-30T07:33:28.329756+00:00 (measured_at)	Fails safety gate	3.3 kB	Raw JSON for ota-eval-fixC.json
ota-per-epoch-2026-07-30.json	ota-per-epoch@1	2026-07-30T11:26:42.565497+00:00 (measured_at)	Fails gate	37.8 kB	Raw JSON for ota-per-epoch-2026-07-30.json
ota-per-epoch-packed-2026-07-30.json	ota-per-epoch@1	2026-07-30T12:16:18.498475+00:00 (measured_at)	Fails gate	38.0 kB	Raw JSON for ota-per-epoch-packed-2026-07-30.json
ota-per-epoch-packed-seed7-2026-07-30.json	ota-per-epoch@1	2026-07-30T12:25:36.296426+00:00 (measured_at)	Fails gate	37.8 kB	Raw JSON for ota-per-epoch-packed-seed7-2026-07-30.json
ota-per-epoch-seed7-2026-07-30.json	ota-per-epoch@1	2026-07-30T11:41:47.463316+00:00 (measured_at)	Fails gate	32.1 kB	Raw JSON for ota-per-epoch-seed7-2026-07-30.json
perepoch-history-seed42-2026-07-30.json	training-history@unversioned	No measured-at field	Incomplete	5.5 kB	Raw JSON for perepoch-history-seed42-2026-07-30.json
perepoch-history-seed7-2026-07-30.json	training-history@unversioned	No measured-at field	Incomplete	4.6 kB	Raw JSON for perepoch-history-seed7-2026-07-30.json
sensing-generalization-new-control-seed4242.json	sensing-generalization@2	2026-07-29T08:18:00.707587+00:00 (recorded_at)	Incomplete	19.2 kB	Raw JSON for sensing-generalization-new-control-seed4242.json
sensing-generalization-new-fixA-seed4242.json	sensing-generalization@2	2026-07-29T08:19:28.773270+00:00 (recorded_at)	Incomplete	19.2 kB	Raw JSON for sensing-generalization-new-fixA-seed4242.json
sensing-generalization-new-fixB-seed4242.json	sensing-generalization@2	2026-07-29T08:20:44.905532+00:00 (recorded_at)	Incomplete	19.2 kB	Raw JSON for sensing-generalization-new-fixB-seed4242.json
sensing-generalization-new-legacy-seed4242.json	sensing-generalization@2	2026-07-29T08:22:02.242789+00:00 (recorded_at)	Incomplete	19.2 kB	Raw JSON for sensing-generalization-new-legacy-seed4242.json
sensing-generalization-old-control-seed4242.json	sensing-generalization@2	2026-07-29T08:07:21.270412+00:00 (recorded_at)	Incomplete	19.2 kB	Raw JSON for sensing-generalization-old-control-seed4242.json
sensing-generalization-old-control-seed9137.json	sensing-generalization@2	2026-07-29T08:08:31.855985+00:00 (recorded_at)	Incomplete	19.2 kB	Raw JSON for sensing-generalization-old-control-seed9137.json
sensing-generalization-old-fixA-seed4242.json	sensing-generalization@2	2026-07-29T08:09:46.344012+00:00 (recorded_at)	Incomplete	19.2 kB	Raw JSON for sensing-generalization-old-fixA-seed4242.json
sensing-generalization-old-fixA-seed9137.json	sensing-generalization@2	2026-07-29T08:10:57.776326+00:00 (recorded_at)	Incomplete	19.2 kB	Raw JSON for sensing-generalization-old-fixA-seed9137.json
sensing-generalization-old-fixB-DEPLOYED-seed4242.json	sensing-generalization@3	2026-07-29T10:42:54.924232+00:00 (recorded_at)	Incomplete	20.8 kB	Raw JSON for sensing-generalization-old-fixB-DEPLOYED-seed4242.json
sensing-generalization-old-fixB-seed4242.json	sensing-generalization@2	2026-07-29T08:12:12.681942+00:00 (recorded_at)	Incomplete	19.2 kB	Raw JSON for sensing-generalization-old-fixB-seed4242.json
sensing-generalization-old-fixB-seed9137.json	sensing-generalization@2	2026-07-29T08:13:24.170487+00:00 (recorded_at)	Incomplete	19.2 kB	Raw JSON for sensing-generalization-old-fixB-seed9137.json
sensing-generalization-old-legacy-seed4242.json	sensing-generalization@2	2026-07-29T08:14:38.987009+00:00 (recorded_at)	Incomplete	19.2 kB	Raw JSON for sensing-generalization-old-legacy-seed4242.json
sensing-generalization-old-legacy-seed9137.json	sensing-generalization@2	2026-07-29T08:16:01.261101+00:00 (recorded_at)	Incomplete	19.2 kB	Raw JSON for sensing-generalization-old-legacy-seed9137.json
sensing-generalization-runA.json	sensing-generalization@2	2026-07-29T07:23:44.030394+00:00 (recorded_at)	Incomplete	19.2 kB	Raw JSON for sensing-generalization-runA.json
sensing-generalization-runB.json	sensing-generalization@2	2026-07-29T07:25:23.080312+00:00 (recorded_at)	Incomplete	19.2 kB	Raw JSON for sensing-generalization-runB.json
tvws-retrain-metrics-seed42-2026-08-03.json	tvws-retrain-metrics@1	2026-08-03T09:17:01.537987+00:00 (measured_at)	Incomplete	78.0 kB	Raw JSON for tvws-retrain-metrics-seed42-2026-08-03.json
tvws-retrain-metrics-seed7-2026-08-03.json	tvws-retrain-metrics@1	2026-08-03T09:23:45.176212+00:00 (measured_at)	Incomplete	67.0 kB	Raw JSON for tvws-retrain-metrics-seed7-2026-08-03.json
tvws-retrain-seed42-2026-08-03.json	tvws-retrain@1	2026-08-03T08:55:55.267552+00:00 (measured_at)	Incomplete	814.6 kB	Raw JSON for tvws-retrain-seed42-2026-08-03.json
tvws-retrain-seed7-2026-08-03.json	tvws-retrain@1	2026-08-03T09:18:57.411690+00:00 (measured_at)	Incomplete	813.9 kB	Raw JSON for tvws-retrain-seed7-2026-08-03.json
window-clean-gpu-runA.json	window-clean@bare-array	No measured-at field	Incomplete	0.8 kB	Raw JSON for window-clean-gpu-runA.json
window-clean-gpu-runB.json	window-clean@bare-array	No measured-at field	Incomplete	0.8 kB	Raw JSON for window-clean-gpu-runB.json