Benchmarks & research: open source Jev projects

Where to start

typesafe-ai-benchmarkiammrduncan/typesafe-ai-benchmarkMaps the same tasks to Jev Choice/Noul questions and normalizes answers to a shared result format.★ 38 · Jev code linkedjev-curateAkashPriyadarshii/jev-curateRuns local prefilters, calls TypeSafe and applies probability and score thresholds to each record.★ 26 · Jev code linkedjevifyaltryne/jevifyDesigns Noul, Choice, and Score question packs for an application, with a script for API-backed case runs.★ 24 · Jev code linked

All 112 Benchmarks & research projects

Sorted by GitHub stars, which belong to the whole repository rather than its Jev integration.

1. NanoJevTianyuCodings/NanoJevOpen 0.6B Jev replica with parallel decisions, complete probability distributions, training pipeline, weights, dataset, and live demos.★ 1,988
   Python
2. jevlikevinnylarouge/jevlikeTrain a small model that chooses among a changing list of text options, one probability per option in a single pass. Includes Doom, chess, and Wikispeedia demos.★ 1,231
   Python
3. deciderMapika/deciderOne-pass typed decisions with calibrated probabilities (System One style model), fine-tuned from Qwen3.5-2B.★ 311
   Python
4. jev-visualhr98w/jev-visualAn educational Jev-like visual inference experiment on Apple Silicon: shared context, direct candidate scoring, and local visual demos.★ 240
   Python
5. convex-evalsget-convex/convex-evalsConvex's evaluation suite, including a decision track that asks Jev multiple-choice questions about the Convex platform.★ 129
   TypeScript
6. reflexkshetrajna12/reflexA small open decision model: state + typed questions -> calibrated probabilities. A Jev / System One re-creation on Qwen3.5.★ 119
   Python
7. jev-eval-agentvinilana/jev-eval-agentPersonal-assistant agent built on Vercel's eve with 100 mocked tools, measuring how many steps it takes when Jev picks the tool versus the LLM.★ 105
   HTML
8. open-jev (daseinlabs)daseinlabs/open-jevOne-pass option scoring with a local Gemma 3 4B on Apple silicon via MLX, inspired by jevlike, with a Doom demo.★ 102
   Python
9. jevbenchfstandhartinger/jevbenchJevBench v1 - a benchmark for Jev-class typed decision models: smart, cheap, fast, reliable, open.★ 80
   Python
10. WindTunnelnekuda-ai/WindTunnelWebMCP benchmark comparing browser-agent interfaces, with Jev included as one of the evaluated configurations.★ 78
    HTML
11. jev-as-a-judgedanielgshea/jev-as-a-judgeUsing Jev as an evaluator.★ 72
    Python
12. Verdict-open-jevHeman10x-NGU/Verdict-open-jevNon-autoregressive decision engine on ModernBERT (151M) with calibrated uncertainty (RLCD), TypeSafe AI Jev benchmark audit, and in-browser WebGPU playground.★ 68
    Python
13. jevmlxbnsd55/jevmlxJev-style parallel constrained decisions for any MLX model on Apple Silicon. Typed, schema-valid JSON in one forward pass.★ 57
    Python
14. open-alternative-jevikermoel/open-alternative-jevOpen alternative to Jev: typed, calibrated decisions from any open-weights LLM in one forward pass (HF + vLLM), with benchmarks.★ 50
    Python
15. mini-jevr-ms/mini-jevMini-Jev: what a Jev-style typed-decision interface looks like on a frozen Qwen3-4B — read the option letter's logits instead of generating JSON. Preregistered experiment, results, teaching bench.★ 45
    Python
16. LitJevzhengxuyu/litjevA reproduction of Jev that turns any Qwen model into a fast decision model, serving the same /v1/systemone schema (Choice, Score, Noul) with no training and no generated answer text.★ 41
    Python
17. TypeARTypeLLM/TypeLLMType-safe one-decision-per-token decoding engine for autoregressive LLMs, inspired by Jev.★ 38
    Python
18. typesafe-ai-benchmarkiammrduncan/typesafe-ai-benchmarkCompares Jev with other structured-output models on shared application tasks, recording errors, latency, Tokens, and estimated cost.★ 38
    TypeScript
19. jevfirekikoncuo/jevfireJEV-inspired parallel decisions for CUDA LLMs. One context, many decisions. vLLM API, game-agent examples, and reproducible benchmarks.★ 35
    JavaScript
20. system-one-openmithalouni/system-one-openOpen replica of TypeSafe's Jev: typed calibrated decisions in one forward pass, on Gemma 4 E2B / Gemma 3 270M (Modal).★ 32
    Python
21. Jev_appsJackZeng/Jev_apps看看 Jev 能做什么：用中英文讲清热门应用、工作原理和各自优缺点。Explore Jev apps with plain-language examples, explanations, and comparisons.★ 31
    Python
22. openjev (zhihz)zhihz/openjevLocal bilingual probability decisions from context, questions, and candidate answers. Independent research preview inspired by TypeSafe Jev.★ 31
    Python
23. system-onesgoedecke/system-oneBatched single-token choice inference for open language models, compatible with TypeSafe.★ 29
    Python
24. jev-curateAkashPriyadarshii/jev-curateA Rust dataset-filtering experiment using Jev scores to keep or reject text records.★ 26
    Rust
25. jevifyaltryne/jevifyAn Agent Skill for finding suitable Jev decision points and designing questions and comparison experiments.★ 24
    Python
26. jev-on-a-laptoprorshopping/jev-on-a-laptopUnofficial study: Jev-style parallel typed decisions on stock 1.5B-8B models on an Apple Silicon laptop. Benchmarks, research notes, and a Hugging Face Space demo.★ 23
    Python
27. jev-column-racegoodrahstar/jev-column-raceJev vs Gemini 3.8 Flash: labelling 1,000 app reviews, 4.1× faster and 7× cheaper.★ 22
    JavaScript
28. jevgptBewinxed/jevgptA chatbot built on a model that cannot generate text (TypeSafe AI's Jev, driven autoregressively).★ 21
    TypeScript
29. jev-benchmarksAbdelStark/jev-benchmarksA benchmark comparing Jev and GLiNER on text classification, probability calibration and selective automation.★ 17
    Python
30. jevbetterolanotolu/jevbetterA stronger one-pass scorer over a variable list of text options: hashed n-gram encoder, rival-aware attention, gated head, temperature scaling, benchmarked against jevlike.★ 14
    Python
31. openvonsgenai-craft/openvonsOpenvons (open-Jev): 有限選択肢に確率で答える判断層 — テキスト / 画像 / 日本語音声コマンド.★ 13
    Python
32. jev_stocksosopop/jev_stockAn experimental JEV-powered framework for forecasting short-term stock price direction from structured market data.★ 12
    Python
33. jev-harnessAntonioCoppe/jev-harnessDecision harness for TypeSafe Jev — confidence gates, shadow mode, recipes, and evals. Claude CLI 48.9s → Jev 1.3s on the same row-filter job.★ 10
    TypeScript
34. jevcalabhixhek/jevcalA toolkit for evaluating Jev probabilities on labeled data, selecting confidence thresholds and checking model drift.★ 10
    Python
35. typesafe-localaabolfazl/typesafe-localInspired by TypeSafe Ai, Ask a local LLM typed questions, get calibrated probabilities instead of text. Structured output without generation or parsing. MLX / Apple Silicon.★ 8
    Python
36. jev-rerank-benchanessbelbati/jev-rerank-benchCan a decision model beat dedicated rerankers? TypeSafe Jev vs Cohere Rerank 4 vs ZeroEntropy zerank-2 vs a chat-model baseline: 14 datasets, every raw API response, bootstrap ranges on every gap.★ 6
    Python
37. trade-jevjustinhe16/trade-jevBacktest Jev (TypeSafe) as a BUY/SELL/HOLD trader on NQ L10 order-book data.★ 6
    Python
38. jev-korean-benchmarkmahlernim/jev-korean-benchmarkReproducible early-access evaluation of Jev on Korean understanding and medical text, with runtime and cost evidence.★ 6
    Python
39. jev-benchmarkwondertwins/jev-benchmarkBenchmarks and a playground for TypeSafe's Jev (System One) model: chess, and who-is-the-player-talking-to for speech-to-text game NPCs.★ 6
    Python
40. jev-lmy0usaf/jev-lmA word-level generation experiment that asks Jev to select words or verify locally drafted continuations.★ 6
    TypeScript
41. jev-search-rerank-evalzhuyansen/jev-search-rerank-evalDoes a TypeSafe Jev rerank beat embedding search? Graded relevance eval (9,831 pairs, 164 zh/en queries) over the Agent Skills Hub catalog, with the judge-circularity bias measured.★ 6
    Python
42. daf-jevdocxology/daf-jevA Python toolkit for Jev requests, batch evaluation, calibration and MCP access.★ 5
    Python
43. LegalForecastBenchjohnhughes3/LegalForecastBenchLegalForecast-MTD benchmark alpha and official evaluation workflows.★ 5
    Python
44. jev-little-airwayslbotinelly/jev-little-airwaysAn island-airport simulator using Jev for routes, yielding, emergency broadcasts and landing order.★ 5
    HTML
45. mcts-agentlhemerly/mcts-agentDiscriminative Monte Carlo Tree Search using TypeSafe Jev System One Primitives and Gemini.★ 5
    Python
46. jev-ood-calibrationscienthoon/jev-ood-calibrationReproducible evaluation of Jev calibration on synthetic support tickets and public classification benchmarks.★ 5
    Python
47. jev-does-not-play-diceKantaHayashiAI/jev-does-not-play-diceExperiments on Jev’s probability calibration, uncertainty reporting, and forecast probability preservation.★ 4
    JavaScript
48. system-one-gemmaakash-kamat/system-one-gemmaOpen-source Jev-style System One decision model. Gemma 3 270M with a scoring head — fast, calibrated decisions in a single forward pass. No text generation. Inspired by TypeSafe.ai's Jev.★ 4
    Python
49. jev-phishing-benchanisselbd/jev-phishing-benchJev (TypeSafe) vs Claude Haiku 4.5 on 2 000 phishing emails: accuracy, calibration, latency, cost. Reproducible benchmark.★ 4
    Python
50. typed-decisionskotoba-lang/typed-decisionsJev-shaped typed-decision models with measured latency, accuracy, calibration, and training cost.★ 4
    Python
51. qwen-rlcdshamazharikh/qwen-rlcdJev-style calibrated decision model (Choice/Score/Noul) on Qwen3.5-0.8B.★ 4
    Python
52. openjev (DECRUX9812)DECRUX9812/openjevOpen, local reimplementation of a Jev-style decision layer for job postings.★ 3
    Python
53. jev-for-engineersFoadsf/jev-for-engineersEight mechanical and electrical engineering experiments using Jev for task routing, log checks and component selection.★ 3
    Python
54. jev-gateMongLong0214/jev-gateNot every coding task needs your best model. Experimental Jev-powered model routing for Claude Code — V3 prototype runs today, V4 routes at the task boundary.★ 3
    TypeScript
55. jev-behavior-studyRINNECODER/jev-behavior-studyAn independent Jev 1.13.0 behavior study recording successes and failures across question framing, input conditions and games.★ 3
    Python
56. jev-explorationSamuelSacco/jev-explorationA research repository tracking Jev claims and limitations, with calibration experiments and runnable examples.★ 3
    Python
57. jev-chatadhyaay-karnwal/jev-chatA chatbot from typed Jev decisions: hierarchical speculative decoding over System One probabilities.★ 3
    Python
58. jev-dspy-labjmanhype/jev-dspy-labReproducible calibration and selective-risk benchmarks for Jev/TypeSafe decisions in DSPy workflows★ 3
    Python
59. judgekitlexingtonhibiki/judgekitA provider-agnostic runtime for System One judge models, with YAML-defined classify, score, route and verify tasks and a cost-accuracy benchmark.★ 3
    Python
60. calibreFirasSX914/JanusCalibration and confidence-based routing measured on Banking77: 80.2% accuracy at $0.103 per 500 decisions.★ 2
    Python
61. jev-sec-benchGaurav-Gosain/jev-sec-benchBlind security benchmarks for Jev, TypeSafe's System One model: prompt injection and vulnerable code detection, built on jev-go.★ 2
    Go
62. jev-agent-failure-benchmarkTokenTrim/jev-agent-failure-benchmarkA benchmark using Jev to attribute multi-Agent failures to an Agent, step and error type.★ 2
    Python
63. jev-routing-experimentTokenTrim/jev-routing-experimentBenchmarking TypeSafe's Jev decision model as a cost-efficient LLM router on RouterArena.★ 2
    Python
64. jev-research-evaljgridifier/jev-research-evalReproducible Jev Ultrafast research-browser eval harness + field note (QC’d cases, suite runner, report generator). Not investment advice.★ 2
    HTML
65. jev-medhallu-benchmarkstperic/jev-medhallu-benchmarkTypeSafe's Jev and four fast LLMs added to Stanford MedHELM's MedHallu results: harness, preregistered run plans and every run file.★ 2
    Python
66. paper-packageCompleteDotTech/paper-packageReproducible Jev research manuscript, evidence, and paper package.★ 1
    Python
67. decision-benchHanno-Labs/decision-benchOpen benchmark runtime for document-grounded decision models.★ 1
    Python
68. jevals-dataJevals/jevals-dataIndependent benchmark data for TypeSafe's Jev (System One model) vs LLMs: accuracy, calibration, cost. Boards + per-decision logs, CC-BY-4.0★ 1
69. PocketJevNullPo-jp/PocketJevOn-device iPhone visual decision tool using MLX and Qwen3-VL direct option logits.★ 1
    Swift
70. jev-synergy-screeningPistachioAIHQ/jev-synergy-screeningA Jev title-and-abstract screening experiment compared with Cohen Abstract Triage labels for an ADHD review.★ 1
    Python
71. Jev-MedQAZtrura/Jev-MedQAExperiments running Jev on medical multiple-choice question answering.★ 1
    Python
72. decisionbridgegrishahq/decisionbridgeA Jev-inspired decision interface for existing LLMs. Explicit choices, scores, calibration, and review thresholds.★ 1
    HTML
73. RISC-jeVi2cjak/RISC-jeVI tortured Jev into being a RISC-V CPU.★ 1
    Python
74. jev-freeformkesku/jev-freeformAn observable raw-character chat experiment powered entirely by TypeSafe Jev Choice.★ 1
    JavaScript
75. jev-calculatorpc418/jev-calculatorIt's the result, correct. *probably. - A probabilistic AI calculator powered by Jev.★ 1
    TypeScript
76. kyotsu-ai-benchshibadogcap/kyotsu-ai-benchAI benchmark on Japan's 2026 Common Test: Jev vs luna-none vs luna-low (static dashboard).★ 1
    HTML
77. jev-secret-detectionteyhouse/jev-secret-detectionMeasures how well TypeSafe's RLCD-Jev model spots real secret credentials in file snippets.★ 1
    Python
78. openjev-experimentszefir1990/openjev-experimentsExperiments with openjev, an open Jev-style option-logit runner, on local models.★ 1
    Python
79. padflow-jev-evalszsavage8/padflow-jev-evalsTyped-decision benchmark from PadFlow (land development SaaS): schemas, anonymized labeled rows, and a runner for confidence-calibrated models like TypeSafe Jev.★ 1
    Python
80. Typesafe_chess_evalAliceRoselia/Typesafe_chess_evalAn evaluation of typesafe AI chess. As it turns out, the AI isn't doing really well even though chess is not a particularly open-ended game. Still, it's only a prototype and this probably wasn't optimzied for games.★ 0
    Python
81. jev-alpha-benchGaurav-Gosain/jev-alpha-benchDoes Jev predict stock returns from news? It reads the news well; there is no tradeable alpha. Three arms separate reading from recall.★ 0
    Go
82. jev-headline-benchGaurav-Gosain/jev-headline-benchCan Jev pick the winner of a real headline A/B test? 64.5% across 10,984 Upworthy randomized experiments, 74.7% when the difference was decisive.★ 0
    Go
83. jev-reportHackSing/jev-report发明 RLHF 的人，这次做了个不会说话的模型：Jev 独立研究报告。52 页 PDF + 50 条中文实测复现包 + 143 条可回溯数据表.★ 0
    Python
84. jev-labMenny1337/jev-labTypeScript experiments, evaluations, and latency benchmarks for TypeSafe's Jev model.★ 0
    TypeScript
85. jev-spam-evalbitnovus/jev-spam-evalZero-shot spam filtering with TypeSafe Jev Noul questions, compared with TF-IDF baselines.★ 0
    Jupyter Notebook
86. jev-shadcn-lint-evalblas0/jev-shadcn-lint-evalA small second eval for shadcn-ui/lint that uses TypeSafe's Jev to judge the linter's own output.★ 0
    JavaScript
87. jev-fanout-benchblowxian/jev-fanout-benchMeasured: asking TypeSafe Jev N questions in one call bills the state once. 2,976 real requests, raw data, exact billing check.★ 0
    Python
88. system-one-adapter-rustcodeitlikemiley/system-one-adapter-rustRust port of TypeSafe system-one-adapter (LLM-backed system_one evaluations).★ 0
    Rust
89. jev-deferred-crispificationdnakhoa/jev-deferred-crispificationPosition paper: the Hidden-Markov and fuzzy primitives missing from TypeSafe AI's Jev and System-One decision models. Two lemmas, one principle (Deferred Crispification), one architecture (BSF-S1).★ 0
    TeX
90. jev-playground (hegargarcia)hegargarcia/jev-playgroundBenchmarks Jev against other evaluation models in games with explicit states, legal actions, and measurable outcomes.★ 0
    TypeScript
91. jev-finance-benchmarkhifizz/jev-finance-benchmarkTypesafe.ai model jev finance benchmark.★ 0
92. jev-calibration-auditjujumilk3/jev-calibration-auditAudits Jev calibration, option-wording effects and Korean judgments through public APIs and datasets.★ 0
    Python
93. jev-transcript-screenerkeduseworku/jev-transcript-screenerDemo and evaluation harness that uses Jev to rank solver transcripts for review by a more costly judge.★ 0
    Python
94. labskiarina/labsResearch workspace containing reproducible TypeSafe Jev evaluation and safety-judgment experiments.★ 0
    Python
95. jev-anotacao-sentencaslab-dados/jev-anotacao-sentencasJev (TypeSafe) vs. Gemini 3.8 Flash vs. GPT-5.6 Luna na anotação estruturada de sentenças do TJSP: qualidade, tempo e custo.★ 0
    Python
96. jev-vs-lunamameli/jev-vs-lunaReproducible review-classification comparison measuring Jev and Luna accuracy, latency, and cost.★ 0
    Python
97. misereru-slide-jevmyokoym/misereru-slide-jevOngoing Japanese research deck on Jev and System One models, maintained as Markdown slides.★ 0
    JavaScript
98. shade-arena-jev-monitornican2018/shade-arena-jev-monitorEvaluating TypeSafe's Jev as a fast monitor and action gate for agent sabotage in SHADE-Arena, compared with Gemini 2.5 Flash/Pro.★ 0
    Python
99. jev-certifynikkoxgonzales/jev-certifyResearch toolkit applying conformal risk control and prediction-powered inference to certify Jev routing thresholds.★ 0
    Python
100. jev-evalonlyoneaman/jev-evalTypeSafe's Jev vs gpt-5.4-mini and gpt-5.6-luna on four public classification sets: cases, per-item answers, scoring, charts★ 0
     TypeScript
101. jev-regress-benchredhatpanda/jev-regress-benchWhich of an agent's approved answers actually changed after a config edit? A reproducible bench: exact match vs embeddings vs a three-tier stack vs an LLM judge vs one Jev Choice.★ 0
     TypeScript
102. jev-vs-clefrmax-ai/jev-vs-clefLaunch-window comparison harness for Jev and Cloudflare Clef decision models on synthetic decision tasks.★ 0
     Python
103. jev-jp-addresssmasato/jev-jp-addressJev (TypeSafe) 性能評価プロジェクト — 日本郵便 KEN_ALL をマスタに、AI SDK 経由の Jev が住所のあいまい一致にどこまで使えるかを検証.★ 0
     TypeScript
104. jev-verifystillmarcus24/jev-verifyCheck whether a published Jev output was actually produced by Jev. Implements the Yurin confidence identity; flags published fixtures that violate it.★ 0
     JavaScript
105. jev-trace-classifiersypherin/jev-trace-classifierApplication of TypeSafe Jev (noul judgment primitive) on the collusion.wiki corpus: agent vs human page authorship, head-to-head vs local Qwen3.8-Flash-Next.★ 0
     Python
106. FinancialPredictionJevthodoh1/FinancialPredictionJevUsing Jev to test how well it predicts financial markets(just like most llms as of september 2026, it doesnt do that good).★ 0
     Python
107. typesafe-oraclestrophee-bot/typesafe-oraclesEvaluating TypeSafe's System One primitives (Choice/Score/Noul) — where a typed oracle beats an LLM call.★ 0
     JavaScript
108. jev-pick-and-place-studytryaksh/jev-pick-and-place-studyA small reproducible MuJoCo pilot comparing Jev, Claude Haiku, and reactive rules for pick-and-place.★ 0
     Python
109. thaiexam-jev-chartsvehas/thaiexam-jev-chartsCharts: TypeSafe Jev evaluated on Thai standardized exams vs 110 other models.★ 0
     HTML
110. jev-comparewestsmith-open/jev-compareSide-by-side Jev and Gemini evaluation harness comparing typed decisions, requests, latency, cost and calibration.★ 0
     HTML
111. jev-evaluationwillkelly/jev-evaluationAdversarial Jev evaluation suite with preregistered predictions, request logs, and reproducible experiment artifacts.★ 0
     Python
112. jev-orderby-benchyodablocks/jev-orderby-benchDoes ORDER BY over a Jev probability put rows in a defensible order? Independent ranking, calibration and invariant measurements of TypeSafe AI's Jev: passes six pre-registered gates on 360 labeled rows, fails four of six on graded product relevance.★ 0
     Python

Related categories

• Community (New) 251
• Agent tooling 179
• Community additions 156
• Applications 76
• Games & simulations 75
• Browser & computer use 74
• Routing & Cost Optimization 70
• SDKs & clients 64