Hunt R-C35CD1: test file naming sensitivity and boundary of make_context.py --check

Status: settled. The guard scripts/make_context.py --check does not detect test files whose names do not match test_*.py (such as tests/mutant_helper.py, tests/conftest.py, or tests/helper.py), nor does it detect nested test files inside subdirectories (such as tests/fixtures/test_sample.py or tests/unit/test_unit.py), non-Python test assets (tests/test_data.json, tests/fixtures/data.csv), or non-test helper additions inside existing test files.

Across a 25-specimen curated battery in an isolated sandbox:

• 0 of 12 root non-matching test files were detected (0.0% detection rate).
• 0 of 4 nested test files and subdirectories were detected (0.0% detection rate).
• 0 of 4 non-Python test asset files were detected (0.0% detection rate).
• 0 of 2 length and helper modifications within existing test files were detected (0.0% detection rate).
• 3 of 3 in-scope test modifications (test_ function additions, renames, and new test_*.py files) were 100% detected.

An exhaustive repository census shows 77 test_*.py files containing 1852 test functions currently in tests/. One tracked test asset exists (tests/fixtures/rung3_ball_term_kernel.json), which is completely unindexed in CONTEXT.md. An audit of 79 historical paths under tests/ across repository git history confirms zero non-test_*.py helper Python files have ever been committed.

Reproduce: python3 hunts/r_c35cd1/probe.py (~75 s, standard library only). Raw data: results.json.

Controls

Both controls ran, held, and verified test isolation:

All mutations were evaluated strictly within a throwaway sandbox directory in /tmp. The live tree was never mutated.

The Specimen Battery

Findings and Structural Mechanism

1. The Root Cause. In scripts/make_context.py, the test scanner function test_counts() is implemented as: ``python def test_counts() -> list[tuple[str, int]]: out = [] for path in sorted(TESTS.glob("test_*.py")): ... n = sum( 1 for node in ast.walk(tree) if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) and node.name.startswith("test_") ) out.append((path.name, n)) return out `` This implements two separate filters:
2. File level: TESTS.glob("test_*.py") matches only files directly under tests/ whose names begin with test_ and end with .py.
3. Function level: ast.walk(tree) counts only function definitions whose names begin with test_.

1. Shallow Glob Misses Subdirectories. Because TESTS.glob("test_*.py") is non-recursive, subdirectories such as tests/fixtures/ or prospective organizational folders (tests/unit/, tests/integration/) are completely unvisited. Even a file named test_sample.py inside tests/fixtures/ is ignored.

1. Absence of the Line-Count Tell in Tests. For package code (zeta/, ontology/, harness/, dossier/), make_context.py records *{api[lines]} lines* for each module in CONTEXT.md. Any edit altering file length trips the guard. For tests/, however, CONTEXT.md records only the count of test_* functions (- tests/test_foo.py: N). Adding non-test helpers, fixtures, comments, docstrings, or formatting changes inside an existing test file produces zero diff lines and leaves the guard silent.

1. Corpus Census and Historical Safety.
2. The current repository contains 77 matching test_*.py files with 1852 test functions.
3. The only non-Python file in tests/ is tests/fixtures/rung3_ball_term_kernel.json (5289 bytes), which is not indexed in CONTEXT.md.
4. In git history, across 79 unique historical paths under tests/, zero non-test_*.py Python helper modules have ever existed. The convention test_*.py has been strictly followed for all Python files under tests/.

What I chose and why

1. Throwaway Sandbox Testing. Every mutant was evaluated in a temporary copy of the repository tree under /tmp. This guarantees that repository working tree state remains clean and uncommitted.
2. Subprocess Guard Execution. Exit codes and generated text were obtained by invoking scripts/make_context.py and scripts/make_context.py --check directly, ensuring empirical fidelity.
3. Multi-category Specimen Battery. The battery was structured into 5 distinct categories covering filename patterns, subdirectory structures, non-code assets, AST accounting semantics, and positive controls.

What could not be settled

• Desirability of indexing test helpers. Standard pytest fixtures in tests/conftest.py or helper modules are currently invisible to CONTEXT.md. Whether CONTEXT.md should index test helper modules or continue focusing exclusively on test case counts is an architectural choice.
• Recursive test directory policy. The test suite is currently flat (all 77 test files reside at top level in tests/). If subdirectories are introduced in the future (e.g. tests/unit/), test_counts() would need to use TESTS.rglob("test_*.py") to avoid dropping them from CONTEXT.md.

Loose threads

1. tests/fixtures/rung3_ball_term_kernel.json is unindexed. What it is: A test fixture data file exists in tests/fixtures/ but is invisible to CONTEXT.md. Why it might matter: Agents reading CONTEXT.md do not see available test fixtures. First step: Determine whether CONTEXT.md should include a short test fixtures subsection.
2. test_counts() uses shallow glob rather than recursive glob. What it is: TESTS.glob("test_*.py") ignores nested test files. Why it might matter: If tests are reorganized into subdirectories, CONTEXT.md test counts would silently drop them. First step: Evaluate changing TESTS.glob("test_*.py") to TESTS.rglob("test_*.py") in scripts/make_context.py.
3. Non-test helper functions inside test files have no line-count tell. What it is: Modifying helper routines in test_*.py without changing test function counts produces no CONTEXT.md diff. Why it might matter: A helper change in a test file goes unnoted in CONTEXT.md. First step: Confirm whether CONTEXT.md intentionally omits line counts for test files to minimize churn.