Benchmarks

简体中文

Fresh paid evidence · 2026-10-04

96 records screened in a median 1.313 seconds, with exactly the same selected IDs. The new same-output experiment compares the available Jev API and signed-in Codex CLI screening paths. All 9/9 runs were exact and complete. This stage result is separate from the whole-agent context A/B below.

91.6–97.2% less returned tool context, with the original evidence still recoverable. This is the new whole-operation result: 24 real primary-agent runs, three fixed synthetic scenarios (code-search, triage, exec), two primary models (gpt-5.6-luna, gpt-6-astra), raw/filtered arms and two repetitions per cell. Filtering completed 12/12 operations, raw context 11/12; all 24/24 selected the exact expected ID sets. The raw Astra triage arm selected correctly but one run failed the full expected response contract; it remains in the denominator.

Same-output screening: fast selection through the measured paths

The same-output dataset uses the same 96 synthetic records, two atomic semantic conditions and expected output of 48 matching IDs. Jev performs auto-batched API filtering; signed-in Codex CLI directly screens the supplied records with medium reasoning and no tools. Arm order rotates over three repetitions; every run is fresh, without an inference-result cache. The fixture repeats eight easy bilingual templates twelve times with distinct service IDs. It covers clear current/historical and pre/post-response distinctions, not 96 independent difficult cases.

This measures execution-path screening latency, not isolated native model/API latency or a whole-agent speedup. Jev timing includes planning, transport, typed parsing and program reduction. Codex timing includes prompt serialization, process startup, the provider/agent turn and final response. Prepared fixture directories are timed separately and excluded. Native inference time and CLI startup overhead cannot be separated; request-level latency was not captured. No primary-agent continuation follows Jev. Direct primary-model APIs were not benchmarked. Three repetitions are a smoke sample, not a confidence interval or a broad workload guarantee; repeated simple templates limit the claim. Provider caching may occur even though there is no inference-result cache.

All nine runs returned the same 375-byte normalized decision and had zero false positives, missed IDs, reviews, timeouts, failed requests or unknown-usage attempts. This is equal output and source evidence, not equal serialized prompts: records alone are 12,351 bytes; Jev's typed requests total 93,226 bytes per run, while each primary prompt is 13,224 bytes plus a 196-byte output schema and the CLI's own harness. Jev used two physical requests per run. Primary paths reported one turn per run; their physical provider request count is unavailable.

Jev used more input tokens than either primary path in this same-output test. Output tokens also include Jev's typed per-record judgments rather than only the final ID list. Usage is complete; actual billed amounts remain null. The primary paths use subscription authentication, so these timings and tokens establish no invoice saving. Jev response IDs resolve to jev-1.13.0; the CLI exposes requested primary models without independently exposing provider-resolved model IDs.

Report and limitations · Separate usage receipt · Reproducible driver

Whole operations: context is the win, latency is mixed

Five of six cells were slower by 0.1–18.4%. Only Astra triage was faster (15.3%), and its raw arm contains the failed response contract. Two repetitions are a smoke comparison, not a statistical latency study. Fixture setup is excluded and recorded separately; collection, Jev inference, real primary-agent shell tool use and continuation are timed together. Tool-context bytes count what the tool returns, not the entire model prompt. Main input includes the harness and the agent's fixed instructions.

Actual token totals, summed over each model's six raw or six filtered runs:

Provider usage is complete for these runs. Codex CLI uses configured account access: actual subscription billing is unknown. Cold estimates treat main input as uncached; cache-adjusted estimates use the observed cached subset. Neither is an invoice. Luna exec becomes 12.0% more expensive under the cache-adjusted model; Astra exec has unusually more cached input in the filtered arm, so its 70.6% adjusted reduction should not be attributed solely to filtering. Pricing is a dated, caller-supplied 2026-10-04 rate snapshot. Missing usage or unavailable billing remains null, never a zero-cost success.

Per-run JSON · Summary and exact arithmetic · CSV · Driver

Batching: a faster Jev stage, with a smaller input bill

The fresh native batching test uses 96 fixed synthetic records × two predicates, four arms and three repetitions. All 12/12 operations returned the exact expected selection and complete usage. Single-record parallel uses at most 12 workers; batch-parallel needed two requests and two workers, while single-record arms made 96 requests per operation.

Batching reduces input 56.5%; batch-parallel has 2.58× filtering speed and 61.2% less elapsed time than single-record parallel (3.215 → 1.246 s) in this sample. Time includes planning, native inference and reduction, but no primary-agent continuation. This is not a whole-agent speedup. Repeated easy record templates and network variation limit generalization. Across all arms the provider reported 600,660 input / 110,964 output tokens, complete usage and no unknown attempts. This legacy driver exposes requested jev-1.13.0, not observed response IDs. Driver.

Record workflows: better selection comes with more work

The native record A/B uses fixed synthetic code, full Git before/after files and connection-phase records. It times collection → admission → real native Jev → reduction → source freshness. There is no primary-agent continuation in this experiment.

The deliberately missing lexical-overlap case is still missed in both code arms. The current recall implementation broadens lexical/symbol/caller candidates; it is not exhaustive semantic indexing. Calibration and holdout use distinct records but similar handwritten scenarios. The threshold selector fits no probability calibrator and establishes no population guarantee.

This canonical series made 20 native requests, with 35,250 input / 5,858 output tokens, returned model jev-1.13.0, complete usage and successful freshness checks. Fixed inputs and runner disclose the gold labels, policy grid, model usage by operation and the negative results.

Interfaces: real paid calls through the shipped routes

All 13 checks passed: Python CLI and a fresh registry installation of npm 0.4.0 each exercised 32-record query, code recall, eight correlated events and a six-file diff. An independent official MCP client negotiated 2025-11-25, called filter/search/triage, and recovered the exact original plus its SHA-256. Local Chromium extraction selected four in-stock USB-C lamps; code parsed the source prices and retained the three under $35. CLI survey classified 64/64 topic labels with exact aggregate counts.

These are synthetic interface checks, not open-world accuracy. The survey repeats eight simple ticket templates. 16 requests returned 71,616 input / 17,562 output tokens, complete usage. These public interfaces suppress the returned provider model ID; the result records requested jev-1.13.0 separately and keeps resolved identity unavailable. Runner preserves complete synthetic output packets while replacing machine paths and receipt locations.

Browser: strict review, a verified fix, and failed experiments

The final pinned source A/B compares the released 0.4.0 source with the scroll-evidence repair on the same jev-1.13.0, 0.55 top probability / 0.10 margin policy: eight local fixtures, three repetitions per arm, independent DOM/storage checks. Expected outcomes are 15/24 → 18/24: verified completions 12 → 15 plus three correct irreversible stops in both arms. Wrong actions and false completions are zero in this matrix.

The nested-scroll target changes from 0/3 to 3/3 completed runs. A failed baseline stops early; the completed treatment needs 6 → 21 model calls and median time 1.33 → 3.25 s across these three runs. The existing six-task subset has unchanged 12/18 expected outcomes: six ordinary goal runs still need review. Its known input grows 2.36%, request bytes 7.52%, and median time 3.58%. The fix keeps the strict thresholds; it does not make pure Jev finish every goal.

Across the final matrix, baseline/treatment use 171,320 / 202,290 input tokens and 12,012 / 12,818 output tokens, all resolved to jev-1.13.0 with complete usage. Browser startup is reported separately; fixture navigation, observation, decisions, actions and independent final checks are included in the operation timing.

Earlier experiments remain public: native-select pruning reduced expected passes 11/18 → 9/18; stronger controls and nested scroll showed no pass improvement; the scroll-evidence trial had one provider failure with unknown usage. Its total cost is null, not zero. These failed or incomplete trials are separate from the final complete matrix.

The first benchmark-only primary-planner recovery reached 5/6 expected outcomes. Its independent check rejected one premature Jev DONE after adding the right cart item but before reaching the order gate. A model's DONE alone is not usable completion. The negative first matrix stays published.

The verified continuation reference then reached 6/6: three search completions and three correct Place order stops, under the same strict policy on pinned released 0.4.0. It detects and continues one premature Jev DONE; no wrong actions, accepted false completion claims or main-model executed tools occurred. The program owns observation and enumerated action IDs, checks main-planner ID/value-reference output, rechecks freshness, executes guarded actions and independently verifies the final state. Reference flow.

This is reference integration, not a shipped automatic primary-model fallback. Whole operation time is 14.7–51.6 s, median 33.1 s, including recovery overhead. The six new runs use gpt-5.6-luna: 275,556 input / 70,400 cached-input subset / 1,155 output tokens, plus actual jev-1.13.0 180,522 input / 11,378 output tokens; usage is complete, subscription billing unknown. The result also retains the first matrix's spend separately. Reliable recovery adds work; it does not establish a cheap or fast universal browser agent.

Earlier cumulative audit usage, including failed trials

The cumulative usage ledger includes pilots, repeated validation and failed experiments, not just the successful canonical tables: known Jev 2,928,677 input / 324,772 output tokens, plus one unknown provider attempt. At the stated $0.042/million-input rate, known input corresponds to a $0.123004434 estimate lower bound; cumulative total and actual invoice remain unknown. Primary-account usage is 1,463,583 input / 563,200 cached subset / 7,315 output tokens under subscription access, not a verified API bill. Costs are not silently zeroed or summed across incompatible billing bases.

This earlier series remains unchanged. The new same-output speed experiment adds 91,086 Jev input / 27,546 output and 125,268 primary input / 0 cached / 1,280 output tokens, recorded in a separate incremental receipt.

Reproduce a new paid run

Supply a permitted TypeSafe test credential through TYPESAFE_API_KEY or TYPESAFE_API_KEY_FILE; do not put it into inputs, reports or source. Model calls are opt-in and billable. Use new output paths and keep private traces private.

python -m pip install -e '.[code,dev,docs,browser-test]'
# The independent MCP check also needs the official mcp package.
python -m pip install mcp
python -m benchmarks.run --live --records 96 --repeats 3 --parallel-workers 12 \
  --output local-results/batching-new.json
python -m benchmarks.selection_speed --live --records 96 --repeats 3 --workers 12 \
  --models gpt-5.6-luna gpt-6-astra --output local-results/selection-speed-new.json \
  --private-dir ../jev-selection-private-new
python -m benchmarks.live_records --live --repeats 2 --output local-results/records-new.json
python -m benchmarks.live_interfaces --live --output local-results/interfaces-new.json
python -m benchmarks.operations --live --models gpt-5.6-luna gpt-6-astra \
  --cases code-search triage exec --repeats 2 --jev-workers 4 \
  --rate-snapshot benchmarks/results/2026-10-04-operations-rates.json \
  --report local-results/operations-new.json --private-dir local-results/operations-traces

The workflows above answer different questions: screening-path speed, whole-primary-agent context, record decision quality, and interface correctness. The same-output speed test also requires a signed-in Codex CLI; its private directory must be fresh and outside the repository. Do not combine the experiments' denominators or turn a local stage improvement into a whole-agent saving.

Release 0.4 evidence 2026-10-04

These offline checks target the 0.4 program's contracts and interfaces. Contract A/B uses pinned source baselines; collector recall compares fixed candidate strategies. They make zero semantic model calls. Their scripted responses, labels and program policies are disclosed. Fresh paid evidence appears above; historical live results follow.

The decision cases check error/unknown-usage preservation and the application of supplied probability distributions. They do not measure whether Jev predicted the right answer. The code collector still misses the fixture with no overlapping lexical clues; final semantic precision needs its own evaluation. Browser tests run real local Chromium with a fixed program policy, excluding browser startup; they do not estimate open-web or Jev-driven success. All per-case failures, context bytes, timing, source/script hashes and zero actual model usage are retained: decisions, retrieval, browser observation.

The interface E2E runs 10 actual process checks: CLI setup and context admission, collection, stdin triage/redaction, tracked diff, survey planning, offline evaluation, MCP stdio, the real JS client and an independent official MCP client (protocol 2025-11-25; four tools; evidence roundtrip). Inputs are synthetic and commands deliberately plan or require missing context; zero model requests means this is interface proof rather than semantic quality evidence.

python -m benchmarks.decision_contracts --output decisions.json
python -m benchmarks.retrieval --repeats 5 --output retrieval.json
python -m benchmarks.browser_observation --repeats 3 --output browser.json
python -m benchmarks.release_e2e --output interfaces.json
# Independent client check: install mcp separately, then add --official-mcp.
jev-filter eval --input examples/evaluation.json --output heldout.json

For a new provider comparison, install .[code,dev,bench-llm] and the provider's System One Adapter SDK extra. The frozen plan records 32 input rows, fixed expectations and complete request bytes with no inference. python -m benchmarks.providers --live --provider PROVIDER --model MODEL --output X.json is explicitly billable and requires supplied test credentials. It measures the filtering operation, keeps retry-total known usage and unknown failed attempts, and separates native Jev probabilities from generated LLM probabilities. No fresh provider superiority, whole-agent speedup or invoice saving is established by the offline release evidence.

See integration and policy contracts for defaults, compatibility and a group-separated held-out evaluation that does not fit a probability calibrator.

Hosted execution: 2026-09-30

Scope: synthetic local fixtures only, live Jev (jev-1.13.x), three repetitions per task. A run passes only when its final status is the expected one and a program check on the final state holds: URL and storage for the browser, window text for the desktop, generated ground truth for the survey. The model's DONE is never counted as success. Irreversible fixture actions are expected to pause (needs_confirmation) and are verified not to have happened.

Per-run JSON

How to read it:

• Browser. The fixture shop has a modal cookie banner, an autocomplete combobox, a native select, a results table, a cart with Place order and Remove buttons, a long form that needs scrolling, an open shadow root and a same-origin iframe. Camofox clicks by CSS selector in the top document, so the iframe task is not offered there, and every Camofox click spends about 1.7 s inside Camofox.
• Ablation. no gated note removes the program's fixed sentence ("irreversible steps are gated by the program; keep advancing"). In isolated probes on the results page, goals ending in buy made Jev choose BLOCKED with probability 0.46–0.60 without it and 0.09–0.13 with it. In whole runs the low-confidence-terminal fallback and row-context text also rescue the task, so the end-to-end difference is small; the note is kept because it removes the failure at its source.
• Desktop. A WinForms application (text field, drop-down list, checkbox, tabs, list, a Delete all records button and a button that opens a modal MessageBox) and a Tk canvas whose buttons are drawn pixels, driven through the OCR fallback. Window start-up time is excluded.
• Survey. Seeded synthetic support tickets (five topics, three tones, churn mentions, 10% spam), English and Chinese. The records are template-generated and easy, so 100% topic accuracy here says nothing about your data; measure with --labels on a labelled sample. unpacked sends one record per request (the same questions); no-screen skips the relevance pre-pass. With only 10% irrelevant records, screening costs more than it saves: it pays off when most records are irrelevant.

Reproduce (billable, needs a Chromium; Camofox and the desktop line are optional):

pip install -e '.[dev,browser-test,desktop]'
python -m benchmarks.hosted --live --lines browser,desktop,survey --repeats 3 \
  --camofox --output /tmp/hosted.json
python -m benchmarks.hosted_report /tmp/hosted.json

Real websites and applications: 2026-09-30

The fixture results above show the mechanics. This read-only audit asks how far the same code gets on real public websites and real Windows applications. Nothing was bought, posted or signed in to, and nothing was changed in the code during the audit. Summary data: 2026-09-30-real-world.json. The observation probe is python -m benchmarks.real_world_coverage.

Setup: one Windows 11 machine on one network in mainland China, headless Chrome at 1280x900 with a zh-CN browser language, jev-1.13.0. Results depend on region, IP reputation and time, and bot checks in particular will differ elsewhere.

Observation coverage. On 40 public pages, an independent in-page script listed everything a person could click in the viewport. It then checked which of those the executor's observation offered. One page failed to launch. Seven were bot checks, error pages or a risk overlay.

The misses are mostly clickable elements without a role (58, 51 of them on Chinese sites) and native inputs made fully transparent under a styled control (23, on 7 sites, for example Wikipedia's menus). The metric only counts the viewport. Targets below the fold or inside a scrolled container are a separate gap.

Live goals. 24 read-only goals (search, open a page, set a widget) ran twice each, about 12 runs in parallel. "Reached" is a reviewer's judgment from final URLs, titles and archives, not the verifier's.

The 95% interval for 16 of 34 is roughly 31–63%. One done was likely false: a DuckDuckGo run whose URL check also matched the bot-check page. Six runs reached the goal without done, mostly because the verifier sees page text but not widget state.

Desktop. Eight Windows applications were observed, and five live goals ran once each; two passed.

What failed, by goals affected. A counterfactual check means the change was applied in memory and replayed. None of these fixes is in 0.3.0.

Historical whole-operation benchmark: 48 agent runs · 2026-09-23

Per-run JSON · CSV · Aggregates · Typed decision evidence

This historical 2026-09-23 benchmark tested the public core: two primary models, four scenarios, raw/filtered arms and three repetitions = 48 real agent operations. The primary agent invoked the same fixed collector exactly once, consumed its output and returned structured selected IDs. Each model used medium reasoning. Arm order alternated; each model lane ran serially, with two model lanes active. No Jev result cache was used.

Timing includes the tool call, any Jev inference, the primary model's continuation and the final browser freshness guard. Browser/page setup is recorded separately and excluded because the measured locator operation assumes an existing page. This compares a fixed candidate stream, not unrestricted agents choosing different tools.

Positive cost/latency changes below mean more expensive/slower. Context is returned tool-output characters, not total primary-model tokens.

Quality: all 48 runs selected the expected ID sets, with no observed false additions or omissions. Four raw Astra runs additionally requested review (one code and three log runs); all 24 filtered runs finished without review. This is reported separately from ID correctness. Fewer reviews do not prove better calibration on unseen inputs. All 48 operations were protocol-valid and reported complete model usage.

Measurement limits: two raw command events omitted aggregated_output from the Codex event stream. Their character lengths are null, not zero; no values were imputed. Context means use the available measurements (2 or 3 per arm). Timing, quality and usage remain available. Three repetitions per cell are too few for strong statistical claims. The report includes mean, median, observed min/max and sample counts; it does not present a noisy p95 as a production guarantee.

Cost: estimates use standard, short-context USD rates verified on 2026-09-23: Astra input/cached input/output $10/$1/$50 per million tokens; Luna $0.20/$0.02/$1.20; Jev $0.042 input and free output. Both cold and cache-adjusted estimates are retained. Cache-write charges, regional surcharges, service tiers and discounts are not modeled. These are API-equivalent estimates, not the owner's subscription invoice.

The Luna command scenario was 0.6% more expensive after adding Jev. Several filtered scenarios were slower, including Luna browser selection (+43.0%). Use the tool when context pressure or a measured workload justifies it; do not force it onto cheap, small or latency-sensitive tasks.

Absolute values and timing distribution

Each cell uses three runs. Arrows show raw → filtered; times are seconds and costs are API-equivalent USD per operation.

The Luna browser mean is affected by one 39.49-second sample. Its filtered median is about 17.52 seconds versus 17.35 raw; the three-run mean is not a stable production latency estimate.

Reproduce the complete workflow

Requires a compatible Codex CLI authenticated to your own account, Jev credentials, and a local Camofox server on port 9377 for the browser scenario. The harness creates only synthetic fixtures and its own browser sessions; it does not click or send messages.

python -m benchmarks.operations --live \
  --codex "$(command -v codex)" \
  --models gpt-6-astra gpt-5.6-luna --repeats 3 \
  --report local-results/operations.json \
  --private-dir /tmp/jev-filter-operations-unique

Use fresh paths. This makes 48 primary-agent calls plus the filtered Jev steps and consumes account usage. Raw agent events remain private outside the repository; only allowlisted metrics and synthetic typed answers are exported. The report includes fixture and runtime hashes. Inputs/gold construction live in scenarios.py; the full driver is operations.py.

Public package: measured 2026-09-22

Machine-readable results, fixture, runner.

96 synthetic service records, two atomic predicates each, Jev jev-1.13.0, two repetitions per arm with reversed order on repetition two. No result cache. Wall time includes planning, network, validation and result restoration, but excludes launching the Python interpreter and downstream primary-agent reasoning.

Batching reduced input tokens 56.47%. Batch + parallel was 32.37% faster than single-record parallel and 94.06% faster than single-record serial in this run. All runs returned the expected selected IDs with no unresolved records; usage was reported complete. The result file retains usage, request counts, workers and timings. These are API-reported tokens, not estimated character counts.

Two runs are a smoke benchmark, not a statistically powered study. Inputs repeat eight record patterns and are easier than open-world tasks. Network variation is significant; serial batching was faster than single-record parallel here. A cap of 30 is an upper bound, not a claim that every request used 30 workers or that 30 is always optimal.

Historical prototype: primary-agent workflow comparison

Before this standalone package was extracted, a private prototype was tested on fixed synthetic code-search, browser-control and incident-triage candidates. Two paired runs used an Astra/medium primary-agent baseline. The following numeric observations are historical, not measurements of this release. Private full traces are not shipped because they contain local environment and agent configuration. They therefore have less public reproducibility than the fresh benchmark above.

Selected IDs matched the expected answers in these fixtures; the raw triage arm also returned an extra review. The cost figures include the extra Jev step and the primary model's observed input/output under the historical cold-token price assumptions. They are API-equivalent estimates, not subscription bills or evidence of money saved on this particular account. They are not current provider price quotes.

Reproduce and price your own workload

python -m pip install -e '.[code,dev]'
python -m benchmarks.run --output local-results/offline-new.json
# Billable; uses configured Jev credentials and never caches results.
python -m benchmarks.run --live --output local-results/live-new.json

Each output path must be new so an old or failed run is not overwritten. The offline mode measures planning only; it is not evidence of model quality or live latency.

For a downstream agent A/B, keep the primary model, reasoning setting, task, candidate corpus and expected answers fixed. Alternate raw-native and filtered arms; include collection, Jev, retries, main-agent continuation and any evidence rereads in elapsed time and usage. Record incomplete/review/failed runs, exact selection or task success, false omissions, returned context bytes/tokens and input/output tokens per model. A smaller packet alone is not a quality or cost result.

Use your provider's current prices (including cached-input rates where applicable):

raw_cost = main_input * main_input_rate + main_output * main_output_rate
filtered_cost = filtered_main_input * main_input_rate
              + filtered_main_output * main_output_rate
              + jev_input * jev_input_rate + jev_output * jev_output_rate

Rates must use the same currency and token unit. Include retries and evidence rereads. If usage is incomplete, label cost as a lower bound. Report both negative and positive results. Cheap primary models, small inputs, poor context or frequent rereads can erase the saving. No fixed dollar claim is inferred from the public model-layer benchmark.

First-user integration acceptance

Sanitized observations cover the maintainer's installed source annotation, Telegram maintenance planning and SRE routing entrypoints with synthetic data and real Jev calls. All three preserved their expected decisions and withheld raw input from the compact packet. No message was sent and no production action was authorized by the test. Existing workflow scripts, identities and fallback contracts remain private and unchanged.

A separate two-repetition migration A/B used 48 synthetic DNS signals and included Python process startup: the legacy backend averaged 1.413 s, the public package 1.797 s (27.2% slower). Both returned 48/48 correct decisions on each run and identical usage (6,752 input + 1,913 output tokens per run). No raw signals were in the returned packet. This small test establishes compatibility, not a migration speedup. The isolated package boundary and transport differ; two runs cannot attribute the difference to one cause. The privacy-preserving integration bridge adds a subprocess per whole batch, not per record.

Visual summary

Charts are rendered by python scripts/render_assets.py with the docs extra. The public chart reads the checked-in JSON directly. Historical chart values are explicitly labeled prototype observations. Neither illustration claims a general accuracy guarantee or a measured whole-agent speedup for this release.

Native distribution parity

Source/native observations compare 32 synthetic DNS signals across two alternating-order runs, including process startup and network inference. All four runs returned 32/32 expected judgments with exactly 4,583 input and 1,273 output tokens per run. Returned JSON was 5,660–5,661 characters in both arms. The native executable averaged 1.838 s versus Python's 1.178 s; the individual native runs were 2.570 s and 1.105 s. This small sample does not isolate startup from network variance and does not demonstrate a native speedup.

The native bundle removes installation prerequisites while preserving decisions and model usage in this fixture. It does not reduce inference cost by itself. The Node launcher is separately tested through real tarball installation; this timing compares the underlying native executable, not the extra Node launcher or npm installation.

python -m benchmarks.distribution --live \
  --native /path/to/native/jev-filter \
  --output local-results/distribution-new.json