Benchmarks

简体中文

Whole-operation benchmark: 48 agent runs

Per-run JSON · CSV · Aggregates · Typed decision evidence

This is a fresh benchmark of the public core: two primary models, four scenarios, raw/filtered arms and three repetitions = 48 real agent operations. The primary agent invoked the same fixed collector exactly once, consumed its output and returned structured selected IDs. Each model used medium reasoning. Arm order alternated; each model lane ran serially, with two model lanes active. No Jev result cache was used.

Timing includes the tool call, any Jev inference, the primary model's continuation and the final browser freshness guard. Browser/page setup is recorded separately and excluded because the measured locator operation assumes an existing page. This compares a fixed candidate stream, not unrestricted agents choosing different tools.

Positive cost/latency changes below mean more expensive/slower. Context is returned tool-output characters, not total primary-model tokens.

Quality: all 48 runs selected the expected ID sets, with no observed false additions or omissions. Four raw Astra runs additionally requested review (one code and three log runs); all 24 filtered runs finished without review. This is reported separately from ID correctness. Fewer reviews do not prove better calibration on unseen inputs. All 48 operations were protocol-valid and reported complete model usage.

Measurement limits: two raw command events omitted aggregated_output from the Codex event stream. Their character lengths are null, not zero; no values were imputed. Context means use the available measurements (2 or 3 per arm). Timing, quality and usage remain available. Three repetitions per cell are too few for strong statistical claims. The report includes mean, median, observed min/max and sample counts; it does not present a noisy p95 as a production guarantee.

Cost: estimates use standard, short-context USD rates verified on 2026-09-23: Astra input/cached input/output $10/$1/$50 per million tokens; Luna $0.20/$0.02/$1.20; Jev $0.042 input and free output. Both cold and cache-adjusted estimates are retained. Cache-write charges, regional surcharges, service tiers and discounts are not modeled. These are API-equivalent estimates, not the owner's subscription invoice.

The Luna command scenario was 0.6% more expensive after adding Jev. Several filtered scenarios were slower, including Luna browser selection (+43.0%). Use the tool when context pressure or a measured workload justifies it; do not force it onto cheap, small or latency-sensitive tasks.

Absolute values and timing distribution

Each cell uses three runs. Arrows show raw → filtered; times are seconds and costs are API-equivalent USD per operation.

The Luna browser mean is affected by one 39.49-second sample. Its filtered median is about 17.52 seconds versus 17.35 raw; the three-run mean is not a stable production latency estimate.

Reproduce the complete workflow

Requires a compatible Codex CLI authenticated to your own account, Jev credentials, and a local Camofox server on port 9377 for the browser scenario. The harness creates only synthetic fixtures and its own browser sessions; it does not click or send messages.

python -m benchmarks.operations --live \
  --codex "$(command -v codex)" \
  --models gpt-6-astra gpt-5.6-luna --repeats 3 \
  --report local-results/operations.json \
  --private-dir /tmp/jev-filter-operations-unique

Use fresh paths. This makes 48 primary-agent calls plus the filtered Jev steps and consumes account usage. Raw agent events remain private outside the repository; only allowlisted metrics and synthetic typed answers are exported. The report includes fixture and runtime hashes. Inputs/gold construction live in scenarios.py; the full driver is operations.py.

Public package: measured 2026-09-22

Machine-readable results, fixture, runner.

96 synthetic service records, two atomic predicates each, Jev jev-1.13.0, two repetitions per arm with reversed order on repetition two. No result cache. Wall time includes planning, network, validation and result restoration, but excludes launching the Python interpreter and downstream primary-agent reasoning.

Batching reduced input tokens 56.47%. Batch + parallel was 32.37% faster than single-record parallel and 94.06% faster than single-record serial in this run. All runs returned the expected selected IDs with no unresolved records; usage was reported complete. The result file retains usage, request counts, workers and timings. These are API-reported tokens, not estimated character counts.

Two runs are a smoke benchmark, not a statistically powered study. Inputs repeat eight record patterns and are easier than open-world tasks. Network variation is significant; serial batching was faster than single-record parallel here. A cap of 30 is an upper bound, not a claim that every request used 30 workers or that 30 is always optimal.

Historical prototype: primary-agent workflow comparison

Before this standalone package was extracted, a private prototype was tested on fixed synthetic code-search, browser-control and incident-triage candidates. Two paired runs used an Astra/medium primary-agent baseline. The following numeric observations are historical, not measurements of this release. Private full traces are not shipped because they contain local environment and agent configuration. They therefore have less public reproducibility than the fresh benchmark above.

Selected IDs matched the expected answers in these fixtures; the raw triage arm also returned an extra review. The cost figures include the extra Jev step and the primary model's observed input/output under the historical cold-token price assumptions. They are API-equivalent estimates, not subscription bills or evidence of money saved on this particular account. They are not current provider price quotes.

Reproduce and price your own workload

python -m pip install -e '.[code,dev]'
python -m benchmarks.run --output local-results/offline-new.json
# Billable; uses configured Jev credentials and never caches results.
python -m benchmarks.run --live --output local-results/live-new.json

Each output path must be new so an old or failed run is not overwritten. The offline mode measures planning only; it is not evidence of model quality or live latency.

For a downstream agent A/B, keep the primary model, reasoning setting, task, candidate corpus and expected answers fixed. Alternate raw-native and filtered arms; include collection, Jev, retries, main-agent continuation and any evidence rereads in elapsed time and usage. Record incomplete/review/failed runs, exact selection or task success, false omissions, returned context bytes/tokens and input/output tokens per model. A smaller packet alone is not a quality or cost result.

Use your provider's current prices (including cached-input rates where applicable):

raw_cost = main_input * main_input_rate + main_output * main_output_rate
filtered_cost = filtered_main_input * main_input_rate
              + filtered_main_output * main_output_rate
              + jev_input * jev_input_rate + jev_output * jev_output_rate

Rates must use the same currency and token unit. Include retries and evidence rereads. If usage is incomplete, label cost as a lower bound. Report both negative and positive results. Cheap primary models, small inputs, poor context or frequent rereads can erase the saving. No fixed dollar claim is inferred from the public model-layer benchmark.

First-user integration acceptance

Sanitized observations cover the maintainer's installed source annotation, Telegram maintenance planning and SRE routing entrypoints with synthetic data and real Jev calls. All three preserved their expected decisions and withheld raw input from the compact packet. No message was sent and no production action was authorized by the test. Existing workflow scripts, identities and fallback contracts remain private and unchanged.

A separate two-repetition migration A/B used 48 synthetic DNS signals and included Python process startup: the legacy backend averaged 1.413 s, the public package 1.797 s (27.2% slower). Both returned 48/48 correct decisions on each run and identical usage (6,752 input + 1,913 output tokens per run). No raw signals were in the returned packet. This small test establishes compatibility, not a migration speedup. The isolated package boundary and transport differ; two runs cannot attribute the difference to one cause. The privacy-preserving integration bridge adds a subprocess per whole batch, not per record.

Visual summary

Charts are rendered by python scripts/render_assets.py with the docs extra. The public chart reads the checked-in JSON directly. Historical chart values are explicitly labeled prototype observations. Neither illustration claims a general accuracy guarantee or a measured whole-agent speedup for this release.

Native distribution parity

Source/native observations compare 32 synthetic DNS signals across two alternating-order runs, including process startup and network inference. All four runs returned 32/32 expected judgments with exactly 4,583 input and 1,273 output tokens per run. Returned JSON was 5,660–5,661 characters in both arms. The native executable averaged 1.838 s versus Python's 1.178 s; the individual native runs were 2.570 s and 1.105 s. This small sample does not isolate startup from network variance and does not demonstrate a native speedup.

The native bundle removes installation prerequisites while preserving decisions and model usage in this fixture. It does not reduce inference cost by itself. The Node launcher is separately tested through real tarball installation; this timing compares the underlying native executable, not the extra Node launcher or npm installation.

python -m benchmarks.distribution --live \
  --native /path/to/native/jev-filter \
  --output local-results/distribution-new.json