Hosted execution

简体中文

New in 0.4: ambiguous operation/target/value choices return needs_review before execution (default top probability 0.55, margin 0.10). Independent --verify-controls checks unique current checked/pressed/selected/value states; invalid or stale final observations cannot prove completion. The observer supports explicit roleless controls, labelled transparent toggles and guarded nested vertical scrolling. New-tab following remains unsupported. See policy and integration examples.

Jev Filter can execute, not only filter. Three commands run a program-owned loop in which Jev chooses among options the program enumerated, the program acts, and the program checks the result:

Jev never produces text, selectors, coordinates, commands or scripts. Every executed target is a node the program observed; its identity and state are re-checked immediately before input.

How one step works

flowchart LR
  O[Observe: indexed controls + visible text] --> Q[One Jev request: operation + target per operation + value + irreversible?]
  Q --> G{Program gates}
  G -->|weak DONE/BLOCKED| Q
  G -->|pay/send/delete-like| P[Pause: needs_confirmation + token]
  G -->|no value for field| S[Skip field once; needs_value if stuck]
  G -->|ok| F[Freshness guard]
  F -->|changed| O
  F -->|same| X[Execute]
  X --> L[Log step] --> O

• One request per step. The operation question (CLICK, TYPE_TEXT, SELECT, SCROLL_*, PRESS_ENTER, BACK, WAIT, DONE, BLOCKED) and a target question for each operation are asked together; only the head matching the chosen operation can execute. This design comes from browser-use/jev-ultrafast (MIT).
• Decisions are consumed once. A stale decision is discarded before any mutation; a retry cannot double-click. Executions are logged before the next observation.
• DONE is not proof. --verify-text, --verify-url and --verify-question run on a fresh observation after the loop stops; failure yields unverified.
• Budgets, stalls and cycles. 60 steps and 120 decisions by default; three consecutive unchanged observations (other than WAIT) or a page state revisited three times end the run as blocked.
• Hand-back is a normal outcome. Verification challenges (reCAPTCHA, hCaptcha, Cloudflare) are detected deterministically and never solved: the run ends as blocked with reason challenge. BLOCKED on a page with a password field reports login_required. --dry-run observes, decides one step and executes nothing.

Browser: browse

jev-filter browse --url https://shop.example/ \
  --goal 'Search for red shoes in the Shoes category, in stock only' \
  --value query='red shoes' \
  --verify-url 'results.*cat=shoes'

• Transports. The default launches a private headless Chrome, Edge or Chromium with a throw-away profile (set JEV_BROWSER_EXECUTABLE or --browser-executable). --cdp-port N attaches to a browser you started with --remote-debugging-port=N, including your logged-in profile. --transport camofox --session S drives a local Camofox server on 127.0.0.1:9377 (anti-detection Firefox). The DevTools client is standard-library only and accepts loopback ws:// endpoints only.
• What the page script observes. Common HTML and ARIA controls, including open shadow roots and same-origin frames; native <select> options become individual choices; controls covered by an overlay are excluded; visible text is joined by table row; controls inside a row carry that row's text. Password, hidden and file inputs are never observed. Fields with autocomplete values such as cc-*, one-time-code or *-password are shown only as [filled].
• Values. Jev cannot write text. Supply named values with --value key=value or --values file.json ({"card": {"value": "…", "sensitive": true}} keeps a value out of the model state and the archive). Jev chooses which named value fits the chosen field. With no fitting value the field is skipped once; if the run then cannot finish it ends as needs_value and lists the fields. Optionally, --text-model lets an OpenAI-compatible model (JEV_TEXT_BASE_URL, JEV_TEXT_API_KEY, JEV_TEXT_MODEL) write a value that the goal implies; it must return exactly {"text": …} and never invents personal data.
• Origins. Navigation is limited to the start URL's origin plus --allow-origin. Leaving it ends the run as origin_blocked. --any-origin removes the restriction.
• Dialogs. JavaScript dialogs are dismissed and recorded; --accept-dialogs accepts only alert and beforeunload, never confirm.

Irreversible actions pause

Two independent signals mark an action irreversible: a keyword rule on commit-style controls (buttons, links, menu items; for example pay, place order, send, delete and their Chinese equivalents), and Jev's probability for the question "would the best next operation commit an effect the user cannot undo". Toggles, tabs and options are never gated. A gated action ends the run with the pending action and a confirm_token:

jev-filter browse --url https://shop.example/ --goal 'Buy the cheapest red shoes in size 42' \
  --value query='red shoes' --keep-open
# → {"status":"needs_confirmation","pending":{"label":"Place order"},
#    "confirm_token":"f25e…:e2","session":{"cdp_port":58111,"target_id":"0A92…"}}

# After the user approves:
jev-filter browse --cdp-port 58111 --target-id 0A92… --confirm f25e…:e2 \
  --goal 'Buy the cheapest red shoes in size 42' --verify-text 'order has been placed' --close-browser

The token binds the page fingerprint and the action; if anything changed, the confirmation is rejected as stale. By default the resumed run executes only the confirmed action, verifies and stops (--continue-after-confirm keeps going). --allow-irreversible disables the gate for a run.

The program appends one fixed sentence to the goal: irreversible steps are gated by the program, so keep advancing. Without it, goals ending in buy or place the order made Jev choose BLOCKED early (probability 0.46–0.60 in probes; 0.09–0.13 with the sentence).

Page data: extract

jev-filter extract --url 'https://shop.example/results?q=red' \
  --task 'In-stock shoes under $70' --analysis analysis.json

Tables become header-labelled rows (Product: Red Runner | Price: $59 | …); headings, list items, paragraphs and links become records with their nearest section. The records then go through the same contract as query, so selected_ids, review_ids, context admission and output projection behave identically. Nothing is clicked.

Desktop: desktop

pip install 'jev-filter[desktop] @ git+https://github.com/apixly-ai/jev-filter.git@v0.4.1'   # Windows: UI Automation + OCR; macOS: pyobjc
jev-filter desktop --list                  # candidate windows
jev-filter desktop --window '^Invoice Tool$' \
  --goal 'Set the customer name to Ada Lovelace, choose the Pro plan and save' \
  --value name='Ada Lovelace' --verify-text 'saved Ada Lovelace'

• Windows uses UI Automation through the uiautomation package. Controls are executed through their patterns first (Invoke, Toggle, SelectionItem, Value, ExpandCollapse), which need no pointer movement or focus. A pointer click is the fallback and happens only after hit-testing that the point still belongs to the target. The observation covers the target window plus the same process's popups and dialogs (for example a combo box's drop-down list). Title-bar buttons are excluded, and password fields are never observed.
• macOS uses the Accessibility API through pyobjc (AXPress, AXPick, AXValue). Grant the terminal or app that runs jev-filter access in System Settings → Privacy & Security → Accessibility; without it the run stops with accessibility_permission_required.
• OCR fallback. When accessibility exposes fewer than two controls (canvas, games, custom-drawn UI), text lines from Windows.Media.Ocr or macOS Vision become click targets. A pixel hash of the line's rectangle must be unchanged at click time. --ocr on|off forces it.
• Launching. --launch 'command' starts the application and closes it after the run unless --keep-open.
• Modal dialogs. A UI Automation Invoke on a button whose handler opens a modal dialog does not return while the dialog is open, and every later UIA call to that application hangs (seen on a WinForms MessageBox). Main-window buttons are therefore clicked with BM_CLICK through SendMessageTimeout, which returns within 1 s and keeps the dialog observable; buttons inside dialogs use an Invoke on a separate thread.

Desktop safety gates

These deterministic checks run before any Jev signal:

• Sensitive targets are refused: terminal and console windows (typing there runs commands), password managers, credential and elevation prompts, system settings, registry and process managers. --allow-sensitive-app overrides this for one run.
• Abort at any time: create the file named by JEV_STOP_FILE (default ~/.jev-filter/STOP) or park the pointer in the top-left screen corner; the next input is refused and the run ends as error.
• Environment: a locked session or secure desktop, and (Windows) an elevated target while jev-filter is not elevated, are refused instead of silently dropping input.
• jev-filter doctor reports browser, desktop backend, OCR, lock and elevation readiness; on macOS it reports whether Accessibility and screen capture are granted.

Many records: survey

jev-filter survey --input tickets.jsonl --spec survey.json --format md

{
  "task": "What do customers contact support about, how do they feel, and who may churn?",
  "keep": ["product"],
  "screen": {"instructions": "The record is a genuine support request (not spam)."},
  "questions": {
    "topic": {"type": "choice", "instructions": "Main topic?",
              "criteria": {"billing": "…", "bug": "…", "shipping": "…", "other": "…"}},
    "sentiment": {"type": "score", "instructions": "How does the customer feel?",
                  "criteria": ["Angry", "Neutral", "Positive"]},
    "churn": {"type": "noul", "instructions": "Threatens to cancel or switch."}
  },
  "group_by": ["topic", "product"],
  "confidence_floor": 0.6
}

• Inputs: JSONL, JSON arrays, CSV and directories (.txt/.md files become records); - reads stdin. Long texts are cut and flagged. The record cap (200,000 by default) is reported, never silent.
• screen asks one noul per record first; only records that pass get the full question set.
• Questions are packed many records per request and run with up to 30 requests in flight, reusing the planner behind query.
• The report is computed by code: counts and shares per option, score means and levels, yes shares, crosstabs by question or kept field, the most confident examples per group (bounded by --budget-chars), uncertain and failed record IDs, token usage and estimated input cost. Per-record answers go to the private archive.
• Budgets. A pre-flight plan estimates requests and input tokens (UTF-8 bytes / 3, an upper bound). Runs above --max-usd (default 5) or --max-requests (default 10,000) are refused; --dry-run prints the estimate without inference.
• Calibration. --labels labels.jsonl ({"id": …, "topic": "billing", "churn": true}) adds accuracy per question, accuracy by confidence band and the lowest confidence floor that reaches 95% accuracy with at least 20 samples. Without labels the report says "calibrated": false: thresholds depend on your data and the model version.
• --propose-categories QUESTION lets the text model name categories from a fixed-seed sample; Jev then classifies every record into them plus other. Jev itself never writes summaries: use your agent's LLM to narrate the report.

Statuses and exit codes

stdout carries a compact packet (status, trace of operations and labels, usage, verification). The archive file named in archive holds the full history, per-step decision distributions and latencies.

Limits

• A valid choice can still be wrong. Verify outcomes independently for anything that matters. Treat hosted execution as a short-step executor with a single observable outcome; long open-web tasks belong to your planning agent, which can call browse per step.
• Pages with more than 250 controls are truncated for one step (omitted_actions); scroll or narrow the goal.
• Canvas, closed shadow roots, cross-origin frames, drag and drop, file uploads, CAPTCHAs and sign-in forms are not handled. Password fields are never filled: use a pre-authenticated profile (--cdp-port, Camofox) instead.
• Camofox clicks by CSS selector in the top document, so controls inside frames are not offered there. Each Camofox click also waits about 1.7 s inside Camofox.
• Desktop coverage depends on what an application exposes to accessibility APIs. Electron and Chromium windows expose page content only with renderer accessibility on (for example --force-renderer-accessibility); then they are large (hundreds of controls, about 1.5 s per observation on Windows).
• macOS support is tested with a fake accessibility tree offline and on macOS runners when Accessibility can be granted; it was not exercised on a physical Mac during development.
• Measured gaps on real sites (0.3.0, audit): clickable elements without an ARIA role and native inputs hidden with opacity: 0 are not offered, which hurts custom widgets and many Chinese sites. Only the viewport is observed, so a link below the fold or in a scrolled sidebar needs a scroll first. Links that open a new tab are not followed. Challenge pages with a localized title (for example Cloudflare's check page in a Chinese-language browser) or from other vendors end as model_blocked rather than challenge. Very link-dense pages can exceed the provider's input limit and end as error. --verify-question sees page text, not checked or pressed state.
• Desktop gaps measured on real applications: list items scrolled out of view are not offered and there is no scroll action. Qt menu-bar items need Expand rather than Invoke. Qt builds without accessibility fall back to OCR, which may merge a menu bar into one target. Chromium and Electron windows hide page content unless started with --force-renderer-accessibility.
• Page text, control labels and record text are sent to TypeSafe for inference.