Headless Codex Pilot

This page records a small exploratory extension to the main evaluation pipeline: running the same broad research-evaluation scaffold through codex exec using ChatGPT/Codex subscription access rather than metered OpenAI API calls.

This is not part of the main evidence base. The primary results in this working paper use API-based model calls with controlled inputs, recorded model identifiers, and native PDF ingestion where available. The headless Codex pilot is useful for low-cost exploration, prompt development, and deciding whether further evaluations are worth running, but it should be treated as a separate input modality and execution surface.

What we tried

We added a small runner:

Script: scripts/run_headless_codex_eval.py
Pilot run: results/codex_gpt55_headless_pilot_jul2026/
Model surface: gpt-5.5 via codex exec, authenticated through ChatGPT/Codex rather than an API key
Prompt version: v4_assessment_current_text_extraction
Input modality: pdftotext-extracted text, truncated at 60,000 characters for the first pilot
First paper: Bruers_2021.pdf

The runner deliberately marks the result as pdftotext_extracted_text, stores the source PDF hash, writes a run_config.json, and emits JSON in a compatibility shape that the existing analysis code can read. The aggregation helper now reads both older OpenAI response payloads and the newer parsed / output_text shape used by this pilot.

The one-paper pilot produced the expected artifacts:

json/Bruers_2021.response.json
metrics_long.csv
tiers_long.csv
combined_long.csv
assessment_summaries.csv

For this first run, the combined metrics were:

Metric	Midpoint	Lower	Upper
Overall	46	32	61
Claims and evidence	42	27	58
Methods	34	20	50
Advancing knowledge	57	40	72
Logic and communication	62	48	75
Open science	36	20	55
Global relevance	70	55	82
Tier should	2.3	1.5	3.2
Tier will	2.4	1.4	3.4

The model also correctly noted in its assessment that the text had been truncated and widened uncertainty accordingly. This is the behavior we want from an exploratory runner: it should surface input limitations rather than presenting the output as equivalent to a full native-PDF evaluation.

Terra → Sol concordance follow-up

We also tested a two-stage concordance workflow for comparing separately enumerated human issues with LLM-generated issues. A high-effort Terra pass checked input alignment and proposed mappings; only three ambiguous pair judgments were escalated to a high-effort Sol pass. The workflow used stable issue IDs, retained a historical overlap threshold of 30, recorded exact Codex telemetry, and kept all outputs separate from the confirmatory API results.

We fixed the candidate set as the first three records in source order before inspecting concordance. A manual title/PDF/issue screen excluded Bénabou et al. before judgment: the recorded title and human issues concerned Willful Ignorance and Moral Behavior, while the local PDF and LLM issues concerned Ends versus Means: Kantians, Utilitarians, and Moral Decisions. Acemoglu et al. and Adena–Hager passed the screen and were processed together in one Terra batch.

Result	Acemoglu et al.	Adena–Hager	Pooled eligible issues
Human-issue coverage	2/3 (66.7%)	6/7 (85.7%)	8/10 (80.0%)
LLM overlap rate	3/12 (25.0%)	6/12 (50.0%)	9/24 (37.5%)

The LLM overlap rate is the fraction of unique LLM issue IDs with at least one non-distinct mapping scoring 30 or higher. It is not an estimate of whether unmatched LLM concerns are incorrect. Sol changed one threshold-sensitive mapping out of the three escalated pairs: an Adena–Hager mapping moved from 31 to 27, lowering the pooled LLM overlap rate from 41.7% to 37.5%. Human-issue coverage was unchanged because that human issue retained another qualifying match.

The original Terra and Sol calls used 41,993 input tokens and 4,001 output tokens, including 1,229 reasoning-output tokens. An identical verification run reused both completed artifacts and made no new model calls. The compact methods note, eligibility screen, and structured final judgments are retained in the repository.

This increment supports preparing a larger preregistered-style study, but not running the full corpus yet. The remaining inputs should first receive the same alignment audit, and the protocol should freeze eligibility, compound-issue handling, aggregation, ambiguity ranking under an escalation cap, and treatment of ambiguous pairs that are not escalated.

Replicability and consistency

The main concern is real: this approach is less standardized than the API-based approach.

The API workflow is better for strict replication because it gives us a cleaner contract: explicit model identifiers, structured-output enforcement, direct request and response payloads, file IDs or native PDF inputs, usage metadata, and a provider surface designed for programmatic runs. It is also easier to re-run from a clean environment, pin defaults, and audit failures.

The headless Codex workflow is attractive for exploration because it can use subscription access and avoid metered API costs. But several details make it less directly replicable:

The execution surface is the Codex CLI, not the model API.
Subscription entitlements and rate limits can vary by account and over time.
Default behavior may change with Codex releases unless we pin command flags and record the CLI version.
The pilot currently uses extracted text, not native PDF input, so figures, tables, equations, and layout can be degraded.
The first run intentionally truncated the extracted text; this must be treated as a pilot constraint, not a production setting.
codex exec is an agentic wrapper around a model call. Even when prompted not to use tools, it is not as minimal as a direct API request.

This does not make the approach unusable. It means we should use it with a different evidential status. It is well suited for:

cheap preliminary screening;
prompt iteration before API spending;
exploratory robustness checks;
identifying candidate papers for deeper API-based evaluation;
running larger informal batches where exact comparability is not the goal.

It is less suited for:

headline quantitative comparisons with human evaluators;
claims about model-vs-model performance;
strict reproduction of the native-PDF GPT-5 Pro runs;
archival baselines where future re-runs must be as close as possible to the original call.

Practical guardrails

For any larger headless batch, we should keep the following guardrails:

Record the Codex CLI version, model flag, auth mode, run timestamp, prompt version, PDF hash, text-extraction command, input character count, and truncation status.
Keep the output in a separate results/<run_id>/ directory with input_modality clearly marked.
Do not merge headless/text-extraction results into the main model-comparison plots without a visible modality label.
Use API-based runs for confirmatory comparisons or for any result we expect others to reproduce exactly.
Avoid browser automation or ChatGPT web scraping; use documented Codex non-interactive execution instead.

The appropriate interpretation is therefore: headless Codex is a useful exploration and cost-control tool, not a drop-in replacement for the API-based evidence pipeline.

# Headless Codex Pilot This page records a small exploratory extension to the main evaluation pipeline: running the same broad research-evaluation scaffold through `codex exec` using ChatGPT/Codex subscription access rather than metered OpenAI API calls. This is not part of the main evidence base. The primary results in this working paper use API-based model calls with controlled inputs, recorded model identifiers, and native PDF ingestion where available. The headless Codex pilot is useful for low-cost exploration, prompt development, and deciding whether further evaluations are worth running, but it should be treated as a separate input modality and execution surface. ## What we tried We added a small runner: - Script: `scripts/run_headless_codex_eval.py` - Pilot run: `results/codex_gpt55_headless_pilot_jul2026/` - Model surface: `gpt-5.5` via `codex exec`, authenticated through ChatGPT/Codex rather than an API key - Prompt version: `v4_assessment_current_text_extraction` - Input modality: `pdftotext`-extracted text, truncated at 60,000 characters for the first pilot - First paper: `Bruers_2021.pdf` The runner deliberately marks the result as `pdftotext_extracted_text`, stores the source PDF hash, writes a `run_config.json`, and emits JSON in a compatibility shape that the existing analysis code can read. The aggregation helper now reads both older OpenAI response payloads and the newer `parsed` / `output_text` shape used by this pilot. The one-paper pilot produced the expected artifacts: - `json/Bruers_2021.response.json` - `metrics_long.csv` - `tiers_long.csv` - `combined_long.csv` - `assessment_summaries.csv` For this first run, the combined metrics were: | Metric | Midpoint | Lower | Upper | |---|---:|---:|---:| | Overall | 46 | 32 | 61 | | Claims and evidence | 42 | 27 | 58 | | Methods | 34 | 20 | 50 | | Advancing knowledge | 57 | 40 | 72 | | Logic and communication | 62 | 48 | 75 | | Open science | 36 | 20 | 55 | | Global relevance | 70 | 55 | 82 | | Tier should | 2.3 | 1.5 | 3.2 | | Tier will | 2.4 | 1.4 | 3.4 | The model also correctly noted in its assessment that the text had been truncated and widened uncertainty accordingly. This is the behavior we want from an exploratory runner: it should surface input limitations rather than presenting the output as equivalent to a full native-PDF evaluation. ## Terra → Sol concordance follow-up We also tested a two-stage concordance workflow for comparing separately enumerated human issues with LLM-generated issues. A high-effort Terra pass checked input alignment and proposed mappings; only three ambiguous pair judgments were escalated to a high-effort Sol pass. The workflow used stable issue IDs, retained a historical overlap threshold of 30, recorded exact Codex telemetry, and kept all outputs separate from the confirmatory API results. We fixed the candidate set as the first three records in source order before inspecting concordance. A manual title/PDF/issue screen excluded Bénabou et al. before judgment: the recorded title and human issues concerned *Willful Ignorance and Moral Behavior*, while the local PDF and LLM issues concerned *Ends versus Means: Kantians, Utilitarians, and Moral Decisions*. Acemoglu et al. and Adena–Hager passed the screen and were processed together in one Terra batch. | Result | Acemoglu et al. | Adena–Hager | Pooled eligible issues | |---|---:|---:|---:| | Human-issue coverage | 2/3 (66.7%) | 6/7 (85.7%) | 8/10 (80.0%) | | LLM overlap rate | 3/12 (25.0%) | 6/12 (50.0%) | 9/24 (37.5%) | The LLM overlap rate is the fraction of unique LLM issue IDs with at least one non-distinct mapping scoring 30 or higher. It is not an estimate of whether unmatched LLM concerns are incorrect. Sol changed one threshold-sensitive mapping out of the three escalated pairs: an Adena–Hager mapping moved from 31 to 27, lowering the pooled LLM overlap rate from 41.7% to 37.5%. Human-issue coverage was unchanged because that human issue retained another qualifying match. The original Terra and Sol calls used 41,993 input tokens and 4,001 output tokens, including 1,229 reasoning-output tokens. An identical verification run reused both completed artifacts and made no new model calls. The compact [methods note](https://github.com/valentinklotzbuecher/llm-uj-research-eval/blob/working-paper/docs/concordance_three_candidate_methods_note_jul2026.md), [eligibility screen](https://github.com/valentinklotzbuecher/llm-uj-research-eval/blob/working-paper/results/concordance_codex_three_candidate_screened_batch_jul2026/eligibility_screen.md), and [structured final judgments](https://github.com/valentinklotzbuecher/llm-uj-research-eval/blob/working-paper/results/concordance_codex_three_candidate_screened_batch_jul2026/concordance_results.json) are retained in the repository. This increment supports preparing a larger preregistered-style study, but not running the full corpus yet. The remaining inputs should first receive the same alignment audit, and the protocol should freeze eligibility, compound-issue handling, aggregation, ambiguity ranking under an escalation cap, and treatment of ambiguous pairs that are not escalated. ## Replicability and consistency The main concern is real: this approach is less standardized than the API-based approach. The API workflow is better for strict replication because it gives us a cleaner contract: explicit model identifiers, structured-output enforcement, direct request and response payloads, file IDs or native PDF inputs, usage metadata, and a provider surface designed for programmatic runs. It is also easier to re-run from a clean environment, pin defaults, and audit failures. The headless Codex workflow is attractive for exploration because it can use subscription access and avoid metered API costs. But several details make it less directly replicable: - The execution surface is the Codex CLI, not the model API. - Subscription entitlements and rate limits can vary by account and over time. - Default behavior may change with Codex releases unless we pin command flags and record the CLI version. - The pilot currently uses extracted text, not native PDF input, so figures, tables, equations, and layout can be degraded. - The first run intentionally truncated the extracted text; this must be treated as a pilot constraint, not a production setting. - `codex exec` is an agentic wrapper around a model call. Even when prompted not to use tools, it is not as minimal as a direct API request. This does not make the approach unusable. It means we should use it with a different evidential status. It is well suited for: - cheap preliminary screening; - prompt iteration before API spending; - exploratory robustness checks; - identifying candidate papers for deeper API-based evaluation; - running larger informal batches where exact comparability is not the goal. It is less suited for: - headline quantitative comparisons with human evaluators; - claims about model-vs-model performance; - strict reproduction of the native-PDF GPT-5 Pro runs; - archival baselines where future re-runs must be as close as possible to the original call. ## Practical guardrails For any larger headless batch, we should keep the following guardrails: - Record the Codex CLI version, model flag, auth mode, run timestamp, prompt version, PDF hash, text-extraction command, input character count, and truncation status. - Keep the output in a separate `results/<run_id>/` directory with `input_modality` clearly marked. - Do not merge headless/text-extraction results into the main model-comparison plots without a visible modality label. - Use API-based runs for confirmatory comparisons or for any result we expect others to reproduce exactly. - Avoid browser automation or ChatGPT web scraping; use documented Codex non-interactive execution instead. The appropriate interpretation is therefore: headless Codex is a useful exploration and cost-control tool, not a drop-in replacement for the API-based evidence pipeline.