Testing & QA

F1 StratLab keeps the pull-request gate fast without deleting the tests that need real data, expensive measurement, or an external service. The gate runs the hermetic tiers in parallel; scheduled and local commands cover the rest.

Run the suite

# The fast PR-equivalent gate
uv run pytest -v -n 4 --dist=loadfile \
  -m "not data and not slow and not network" \
  --cov=src --cov-report=term-missing

# The complete collected suite
uv run pytest -v -n 4 --dist=loadfile \
  --cov=src --cov-report=term-missing

PITWALL real-path trace

Issue #1252 has a local trace for one measured Lusail 2025 tick. It captures one message from the live Arcade TCP server and the real /api/tick, /api/bulk, /api/live, and /api/agents responses. The JSON report records the complete wire message and each sequence. Check the rendered DATA and AGENTS pages separately in a browser.

Build the bundle, then choose unused loopback ports. In one PowerShell window, launch the no-LLM replay:

Push-Location src/pitwall/ui
npm run build
Pop-Location

$env:F1_STREAM_PORT = "55998"
$env:F1_PITWALL_BROWSER_PORT = "55999"
$env:OPENAI_API_KEY = ""
$env:HF_HUB_OFFLINE = "1"
$env:TRANSFORMERS_OFFLINE = "1"
uv run f1-arcade --viewer --year 2025 --round 23 --driver NOR --team McLaren --strategy --no-llm

Raise playback to 8x with five Up-arrow presses. In two other windows, set the same ports and start both captures with one fresh run ID. Run them at the same time so the pages can be matched to the wire sequence.

$env:F1_STREAM_PORT = "55998"
$env:F1_PITWALL_BROWSER_PORT = "55999"
uv run python scripts/trace_pitwall_real_path.py --run-id 1252-lusail-nor-l7-run1

In the third window, run the Playwright page probe:

node src/pitwall/ui/scripts/trace-live.mjs `
  http://127.0.0.1:55999 1252-lusail-nor-l7-run1 7 Lusail NOR

It saves the API responses, rendered page text, console errors, and screenshots beside the report. After both capture commands finish, verify the files together:

uv run python scripts/verify_pitwall_trace.py --run-id 1252-lusail-nor-l7-run1

The verifier checks the five sequences, strategy action, DATA values, browser errors, and screenshot files, then writes a separate _verified.json. Use a new run ID for each retry. This local run needs 2025 data and model weights. --no-llm and the empty API key prevent paid model calls. It is a local data run, not part of the PR gate.

RAG retrieval evaluation

The shared f1-eval rag command evaluates the production BGE-M3 retriever over the 30 manually verified queries in data/rag_eval/queries_v2.json:

uv run f1-eval rag

It reports conventional P@1/P@3/P@5, hit@5 for comparison with the historical N30B notebook, MRR, retrieval-level citation match, wrong-year rate, and P50/P95 latency. The command runs the season-scoped production path and an unscoped control. It loads BGE-M3 once and makes no LLM calls. The Markdown and JSON reports land in documents/eval_reports/rag.{md,json}. The complete run needs the local FIA PDFs and Qdrant index, so it stays outside the fast CI gate.

Issue #323 adds five verified 2026 questions and a separate A/B command. It compares the production article-aware 512/64 index with a temporary 1024/128 candidate using the same BGE-M3 encoder and never mutates data/rag/:

uv run python scripts/benchmark_rag_chunking.py

The candidate is adopted only when P@5, MRR, citation match, and wrong-year safety do not regress and at least one quality metric improves.

The fast gate took 56.24 seconds on the development machine used for the 2026-09-16 audit. The complete suite took 245.64 seconds with four workers, down from 276.82 seconds before this cleanup and 383.27 seconds before the tier split. The earlier serial comparison fell from 670.47 seconds to 525.48 seconds. These figures are local wall-clock measurements, so they are not a promise about every runner. On GitHub, the PR test job fell from 127 seconds to 100 seconds. The previous feature push also ran a duplicate 194-second test job; feature branches no longer trigger that push workflow, reducing the two-run cost from 321 to 100 job-seconds, a 68.8% reduction.

Markers

Markers are declared in pyproject.toml. They explain why a test is excluded, not whether the test is less important.

Marker Meaning Default gate
unit Pure hermetic logic Included
contract API, schema, serialization, or fixture boundary Included
data HF dataset, model weights, or measured tables Excluded
gpu CUDA-dependent execution Excluded
llm Live or hermetic LLM backend Excluded until a stubbed tier exists
slow Expensive measurement or real-path check Excluded
network Public service or external artefact Excluded

Pytest's -ra setting prints skipped tests. A skipped data test means that its artefacts were not present and that measurement did not run.

CI and scheduled checks

The test job in ci.yml uses four xdist workers, --dist=loadfile, coverage, and the fast marker expression. Its collection floor uses the same expression, guarding against a broken import or accidental mass skip. The push trigger covers main, dev, and test; feature branches use the pull-request trigger, so one commit does not run both workflows.

The excluded checks remain explicit:

What the efficiency pass changed

Dependency cleanup

The parent lock updates accelerate to 1.15.0 and pytorch-lightning to 2.6.6, and removes unused passlib and bcrypt. Test environments use httpx2 for Starlette's current TestClient. The latest local pip-audit run reports 37 findings in Pillow 11.3.0 and setuptools 81.0.0. Their fixes remain constrained by the Arcade and CUDA PyTorch graphs; the exact waivers stay in osv-scanner.toml.

When a production contract changes, update the matching guide under documents/dev_docs/ and this page in the same pull request. When deleting a test, state which remaining assertion protects the behaviour and why the old case was redundant.

Treat src/agents/ as a protected path. Do not edit it for routine cleanup or simplification. A feature change may touch it only after its impact is mapped, the scope is minimal, and extra regression checks cover the affected contract.

Further reading