Is Jev a scam? What TypeSafe AI's System One model is, and what happened when we ran it
Jev is a real, hosted, closed model that returns typed decisions with a probability vector. Its 193x / 445x marketing numbers are unproven by anyone, including TypeSafe. We ran jev-1.13.0 on Swiss bookkeeping data: accurate and calibrated on 54 real accounts, confidently wrong on a 10-item cohort.
tsukumo
Short version: no. Jev is a real, hosted, closed model that takes text and returns one typed decision with a probability over every option you supplied. What's unproven is the marketing: the 193x-faster and 445x-cheaper claims come from TypeSafe's own undisclosed evals, and nobody, TypeSafe included, has published its calibration. On our run, jev-1.13.0 was accurate and calibrated on 54 real Swiss account decisions. On a 10-item cohort it was confidently wrong. The product is real. The marketing runs ahead of it.
A developer we work with put it more bluntly in a Slack channel last week: Jev is a scam, classifiers existed before LLMs, it only looks new because nobody did AI before ChatGPT. Fireship's Code Report on September 21 gave the same doubters a segment. That's the question worth answering properly, because "scam" and "overhyped" are different accusations with different evidence, and the difference decides whether you should spend an afternoon on it.
TypeSafe AI, founded in 2024 in San Francisco by Diogo Almeida, Erik Gafni and Sasha Sheng, launched Jev on September 15, 2026 with a $40M seed led by DCVC at a $200M valuation. The company calls it a "System One Model", after Kahneman: fast, intuitive, no deliberation. The pitch in their own words is a "frontier-intelligence function call: unstructured state in, typed probabilistic decisions out".
Mechanically, it's one endpoint. You POST to /v1/systemone with state (a string, a JSON object, or an array of text) and one or more questions. Each question is one of three primitives:
Choice: pick one of up to 255 options.
Score: pick a level on an ordered scale.
Noul: the probability that a statement is true.
Every question runs in parallel over the shared state, so adding questions barely changes latency. The response carries the winning candidate, the full probability vector across the options you supplied, and a separate confidence number derived from the shape of that distribution. A top option at 0.84 with a runner-up at 0.159 comes back with confidence 0.596, for example. There is no free text, and the model cannot return anything outside your candidate list.
Pricing is $0.042 per million input tokens, or as TypeSafe puts it, "$42 per billion". Output tokens are free. TypeSafe's own blog says it "cannot prove the price is unsubsidized", which is a more candid line than most launch posts manage. The version we hit in September was jev-1.13.0.
The founder story is where the hype starts, so it's worth being exact. Almeida is the fourth of twenty listed authors on the 2022 InstructGPT paper, the work that established RLHF as OpenAI's instruction-following method.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., et al. · arXiv · 2022
“Almeida is the fourth of twenty listed authors. TypeSafe's team page says he "co-invented RLHF and
InstructGPT"; the viral version, "co-inventor of ChatGPT", is a stronger framing than the paper
supports. A collective piece of work with twenty names on it is not one person's invention.”
His argument against his own earlier work is the most interesting thing TypeSafe has said, and it holds up whether or not Jev does.
“Why do all LLMs require a human in the loop? The simple answer is we literally put them in the
loop.”
— Diogo Almeida, BigGo Finance interview, August 2026
RLHF trains a model to produce output a human rater prefers. A rater prefers confident, complete, agreeable text. So the objective rewards sounding right over being right, and Almeida is explicit that this is structural: "Overpromising is a feature. This is by design." And: "No matter how wrong the models are, they will look right." His conclusion, in the same interview: "What you really want if you want automation is for it to just not give a damn about the humans."
Jev is his attempt at a different post-training target. Instead of scoring the model on human preference, RLCD scores it on whether the probability it reports matches how often it's actually right. That is the whole product: a model whose confidence you can act on programmatically. Accept above a threshold, route to a human below it.
Two more positions matter for what follows. Almeida rejects public benchmarks as "extremely gameable" and evaluates only on internal workflow tasks. And he's said on the record that "nines of reliability" are still aspirational.
Here's the strongest version of the accusation we could assemble, from Hacker News (1,973 points, 512 comments on launch day), from the Langfuse and AINews write-ups, from Fireship, and from the Slack thread that started this article. Then a verdict on each point.
1. Nobody has seen the calibration. The entire pitch is "calibrated probabilities". TypeSafe has published no reliability diagram, no expected calibration error, no Brier score. One HN commenter framed the test correctly: if the model marks 0.9 confidence on 1,000 answers, about 900 of them should be right. Nobody outside TypeSafe has published that count. Fair. This is the load-bearing claim and it's unverified.
2. The benchmarks grade themselves. TypeSafe's "193.6x faster, 444.6x cheaper" comes from four internal tasks: security triage, agent-trace review, invoice pay/hold/reject, and customer next-action. The reference labels are the averaged answers of Claude Fable 5.1 and GPT-6 Astra. The dataset is not released. The one independent measurement we found, reported via agentconn, got roughly 2.9x faster and 12x cheaper. Fair. Real numbers, chosen conditions, no way to reproduce.
3. The price may be subsidized. TypeSafe says so itself. Fair, and they admitted it first. A $40M seed can fund a lot of $42-per-billion.
4. It's just a classifier. HN called it a "Large Classification Model". AINews: "BERT-like, but with the data, compute, and training recipe of modern LLMs". Fireship's segment: some people say it's no different from the zero-shot classifiers of the past, and the company gives no credit to the people who built them over a decade ago. Several independent OpenJev projects reproduce the whole interface by prompting a frozen Qwen3-4B with the options and reading the next-token logits, no training, one forward pass, runs on a 3090. One of them reports 0.845 agreement with Jev on a 102-row subset, by its author's own count. Mostly fair. Structurally it is a classifier with a probability head. The counter-argument, that Jev is a general classifier you don't have to train per task, is a real engineering difference. It's also the difference a frozen Qwen appears to cover most of the way.
5. The output isn't stable. Reorder the same options and the probability on the same answer moves. The pijev package documents a billing decision going from 0.43 to 0.63 across six permutations of the candidate list on jev-1.13.0, and exists to average them away. Fireship: "it's not even deterministic". Fair on ordering. On repeat variance we have our own number below, and it's smaller than the critique implies.
6. The demos are cooked. The Doom demo that circulated at launch fed the model text coordinates rather than pixels. Fair. It was a demo of the interface, presented as something more.
7. The traction is self-reported. Over a trillion tokens a day within launch week, 100k on Discord, 38M video views. All from TypeSafe. Unverifiable. Which is not the same as false.
8. The halo does the work. This was the Slack developer's real point. He builds a tool that does the same job deterministically, with some adaptation per task, and it gets no attention because he isn't "the co-creator of ChatGPT". Same mission, no fame, nobody cares. Fireship made the mirror-image point: a researcher claims his paper from a year ago is the same thing as Jev. Fair, and it's the honest core of the "scam" feeling. The credential is doing marketing work the evidence hasn't done yet. The same developer also conceded the thing Jev actually did: it made a lot of people notice the value of a structured, typed answer instead of a paragraph. That's a contribution. It's also not a model architecture.
None of that adds up to a scam. A scam takes your money for something that doesn't exist. The endpoint exists, it answers in 300 milliseconds, it bills you, and the behaviour is measurable. What the eight points add up to is a launch where every quantitative claim is unverified and the qualitative claim (calibration) is the one thing nobody has checked in public. So we checked.
One use case: Swiss bookkeeping, jev-1.13.0 versus an open alternative#
Why this task: it's the middle ground the product is supposed to own. Too semantic for a regex (a vendor description that doesn't map to a rule), too variable for a lookup table, but the legal outcomes are bounded in advance. Every invoice lands in exactly one account. And it's not English, which every public Jev eval is.
On account classification, Jev picked the correct one of five to eleven plausible operating accounts on all 54 examples. Brier 0.039, expected calibration error 0.082, mean reported confidence 0.92. The exact 95% Clopper-Pearson interval on 54 of 54 is 0.93 to 1.0, and the report notes those intervals assume independent trials, which grouped vendor fixtures don't establish. Laya, on the identical manifest, got 48% with a Brier of 0.709.
VAT class, 28 paired examples: Jev 1.00, Laya 0.857. The language rows are four source descriptions translated into four languages with labels unchanged, so 16 dependent variants, not 16 examples. That ECE of 0.082 is the number point 1 above asks for, on one task, in one shop, at n=54. It's consistent with the calibration claim. It isn't proof of it.
Repeat stability, which point 5 questions: 20 repeats of the same request, zero answer flips, per-class standard deviation at or below 0.03. Laya, being a plain forward pass, is fully deterministic. So Jev is not deterministic, and on this cohort the variance is small enough that you'd never see it at a sane threshold. Option ordering is a separate problem we didn't measure; pijev did.
Ten items. The report itself flags the cohort as insufficient, and we're keeping that flag on. The confusion matrix explains the 0.30 better than the accuracy does: Jev answered "Pending" for all ten claims. Three were Pending, seven were Processed, and the seven wrong answers came back at 0.83 to 0.95 confidence. Expected calibration error on the cohort: 0.615. The model didn't weigh ten decisions and lose seven. It collapsed onto one class and reported high confidence each time, which is the exact behaviour the training objective is supposed to remove.
Category classification on the same dataset, n=12, landed at 0.50 with a Brier of 0.744. The worst miss is instructive: the line "Automatically generated expense line for creation of asset", ground truth Travel, predicted Assets at probability 1.0. Read the text. Jev's answer is the defensible one and the label is the problem. Public datasets carry noise like this, and a calibrated model will be confidently "wrong" against a wrong label. That argues for auditing your labels before you audit the model.
Two readings fit the decision cohort. Either the calibration doesn't transfer to ServiceNow's task framing, where the answer depends on process history the input doesn't contain, or ten items are noise. We can't distinguish them at this size, and neither could TypeSafe. The report's own line applies: ten calls are a smoke test, not evidence of a production error rate.
Take a case Jev gets right. Remove the correct candidate. Ask again. Across 10 such pairs the prediction flipped every time, which is inevitable. What matters is the confidence: mean absolute delta of 0.38 to 0.46, and on the single-candidate VAT variant, the forced-wrong answer still came back at 0.90 or higher, every time. That last one is structural. With one candidate the probability is 1.0 by construction. Design your candidate lists accordingly.
Now add a "none of the above" candidate to the same truncated list. Abstention rate: 10 of 10. Offered a way out, the model took it every time. Over 20 repeats the abstention probability sat between 0.47 and 0.58, which means an automation threshold at 0.5 would have flipped the routing decision 39% of the time.
Production behaviours nobody's launch post mentions#
Two things happened during the run that belong in any evaluation and in no marketing deck. One request came back with probabilities that didn't sum to one, and the SDK refused it: "Probabilities do not sum to one; refusing to normalize silently". That's the correct behaviour and it's a 500 in your pipeline. Four calls failed with a 429, "Your account is not active, please check your billing details", on an account that was active. Budget for both.
The control that matters most for the deterministic-first argument: on the 28 VAT cases where the rate was stated explicitly in the input, both Jev and Laya scored 1.00, Jev at ECE 0.011 and Laya at 0.110. A lookup would score 1.00 too, at ECE zero, for free. If vendor == "Swisscom" maps to account 6500 reliably, replacing that line with an API call buys you latency, cost, and a non-deterministic dependency, and no accuracy. The Slack developer was right about that part. The interesting operating region for Jev starts where the rules get expensive to maintain, and on our data that region was real, but it was smaller than the candidate list.
Where it's real: the interface is a genuine improvement over parsing free text and hoping. The calibration held on the one real-taxonomy, non-English task we could run at n=54. Abstention works when you supply it. Latency was 260 to 300 milliseconds at p50 from Switzerland, 640 at p99.
Where the claims fall: 193x and 445x are self-graded on a hidden dataset. The one outside measurement found single digits. Calibration is unpublished. The price is admitted to be possibly subsidized. The architecture is a secret with a name. And a frozen open model reads option probabilities off its logits well enough that the moat, if there is one, is the training recipe nobody can inspect.
The rule we're keeping: deterministic first, then a classifier you own, then Jev, and only with an abstention candidate and a cohort big enough that "30% wrong at 90% confidence" means something.
From TypeSafe: a reliability diagram on a released dataset, the paper, and a third-party rerun of the four workflow evals. From us: the LLM baseline that died on billing, a rules-engine baseline on the full Swiss fixture, and cohorts an order of magnitude larger on the expense-claim task.
This is the same question we ask before any agent touches a client's production system: what's the actual accuracy under adversarial conditions, and does the confidence it reports match the confidence it should have. A launch video doesn't answer it. A frozen manifest and a paired baseline get closer.
We run this calibration-and-failure-mode discipline on client agent stacks, with the same rule: deterministic first, and no number without its sample size.
Want your AI stack's failure modes measured before production finds them?
Jev is TypeSafe AI's "System One" model, launched September 15, 2026. You send it text state plus a fixed set of candidate answers, and it returns one candidate with a probability for every option and a separate confidence score. It runs only on TypeSafe's API, its weights and architecture are undisclosed, and it handles text only, with a 64K-token context.
Is Jev just a classifier?
Structurally, yes. Closed-label decisions with a confidence score are decades old, and several open projects reproduce Jev's interface by reading option logits off a frozen 4-billion-parameter Qwen in one forward pass. Jev's difference is a general-purpose, software-facing version of that idea, trained on modern data and compute, so a team doesn't train a classifier per decision. Whether that difference is worth paying for depends on the task.
Are Jev's benchmark numbers real?
They are real numbers from TypeSafe's own four internal tasks, with reference labels produced by averaging Claude Fable 5.1 and GPT-6 Astra, on a dataset TypeSafe has not released. No reliability diagram, expected calibration error, or Brier score has been published. The one independent measurement we found reported roughly 2.9x faster and 12x cheaper, not 193x and 445x.
Should we use Jev in production?
Only after a deterministic baseline loses. On our Swiss accounting data, every case with an explicit statutory rate was a lookup, and a rules engine ties Jev there for free. Where the decision is semantic and the outcomes are bounded, jev-1.13.0 was accurate and calibrated on our 54-item account task. Supply an abstention candidate, measure your own cohort, and keep the sample large enough to mean something.
