It's not about generating text. What kind of AI is Jev?

Overview

Even if you encounter the name “Jev,” it’s not immediately clear whether it’s an AI that answers questions in chat, a search model, or a scaled‑down version of an existing large language model. I myself built a MCP server that uses Jev, but knowing how to call its API and understanding its return values are separate issues.

Jev from TypeSafe AI is a model that, given input information, returns a judgment on predefined questions. Rather than producing free‑form explanatory text, it returns values suitable for programmatic handling—such as the probability of a “yes” answer, probabilities for each candidate, tiered ratings, and so on. For example, you feed it a query text and have it evaluate whether it concerns an invoice inquiry, a shipping inquiry, or whether it needs to be routed to a specific person. Rather than reading the answer and continuing the conversation, the usage is that the code receiving the answer decides the next action to take.

Such tasks can also be performed by an LLM. In fact, many systems ask LLMs to classify or score content. However, generating a textual answer, returning a value in a specified format, providing accurate probabilities, and deciding actions based on the returned judgment are each distinct problems. Even if the output is correctly formatted as JSON, the underlying judgment can still be wrong. Even if the response contains something like “0.9,” that number does not necessarily represent the actual correctness rate.

In this article, we first consider what challenges arise when integrating an LLM into a decision‑making process. After that, we examine the publicly documented inputs, outputs, and processing methods of Jev, and then move on to the underlying theories of probability prediction, calibration, appropriate scoring rules, decision‑making, and judgment deferral. Finally, we apply these concepts back to an inquiry‑classification system, considering what should be left to the model and what should be decided by code.

The material covered here is based on official documents and public research verified as of October 1, 2026. This article does not reproduce Jev’s network architecture, weights, or exact training recipes. It treats separately the specifications that can be inferred from publicly available information, the general theories needed to understand them, and my own example designs. The sample inquiries, probabilities, fee calculations, and evaluation tables below are illustrative examples created for explanatory purposes; they are not results obtained by sending them to Jev.

1. For those who have heard of Jev for the first time

1.1 Read the text and return an answer within a predefined scope

For example, suppose an online shop receives the following inquiry:

The product arrived, but there are two identical charges on my credit‑card statement. I would like one of them cancelled.

A human would recognize this as a billing issue and a request for cancellation. However, when trying to implement the same logic in code, the problem becomes which words to use as conditions. Even if you decide that the presence of the term “duplicate charge” should route the inquiry to the billing department, that exact phrase does not appear in the message. If you route every message containing the word “card” to billing, you might also end up mixing in questions about cards that come with the product.

In such cases, where the wording varies but the meaning needs to be judged, a model is used. However, if you only need to select the appropriate department, there’s no need to generate a full, polite response each time. What the system actually requires is information such as which of the codes billing, delivery, product, or unknown the inquiry most closely matches.

Jev is used in a format where you provide such input and a predefined answer range. You feed the document’s text or the customer’s message into state, define what needs to be judged as a question, and the model returns candidate answers and their probabilities, which the calling code then receives. Actions such as sending the inquiry, notifying the appropriate person, or requesting additional information are handled by the downstream system.

flowchart TD
    A["Inquiry text"] --> B["Prepare the state<br/>order information and required history"]
    B --> C["Ask Jev<br/>which domain is responsible"]
    C --> D["Receive candidates and their probabilities"]
    D --> E["Decide handling based on the code"]
    E --> F["Forward to the responsible party"]
    E --> G["Add information and verify"]

In this diagram, Jev does not handle the entire process of responding to an inquiry. Contacting the card company or checking the actual billing records are separate procedures that must be implemented elsewhere. Even if you can classify a message as “apparently a duplicate‑charge inquiry” just from the text, you cannot know for sure whether a double payment actually occurred without examining the transaction records.

Separating this point from the start makes it easier to understand what Jev does as a tool. It evaluates a predefined question based on the evidence you provide. Investigating facts in the external world and judging the meaning of the evidence are distinct tasks within the system.

1.2 The name and the term “System One”

TypeSafe refers to Jev as the System One model. To understand this name, we first want to explain the psychological concepts of System 1 and System 2. This is not a proprietary term unique to Jev nor a computer‑industry standard; it is a label used to describe how human thinking can be divided into fast, automatic processing and slower, effortful, attentive processing.

System 1 refers to the intuitive processing that occurs without conscious, step‑by‑step reasoning. For example, when you recognize a familiar face, you don’t enumerate each facial feature in words and compare them against a list of candidates. As a simple illustration, when asked “What is two plus two?” most people can answer “four” instantly without consciously following the calculation steps.

In contrast, System 2 denotes the kind of processing that requires attention and deliberate, step‑by‑step reasoning, retaining intermediate results as you go. For instance, when mentally calculating “17 × 24,” you might break it into “17 × 20” and “17 × 4,” then add the two products together. Similarly, reading unfamiliar contract terms and verifying exceptions and scopes also falls under this mode of thinking.

This terminology was introduced by Keith Stanovich and Richard West and later popularized by Daniel Kahneman. Kahneman’s A Perspective on Judgment and Choice also explains the characteristics of the two processes and the origins of the names. It does not mean that the brain contains separate “System 1” and “System 2” components that exist like two independent computers. Here, it should be read as a framework for distinguishing modes of thinking.

Moreover, it is not a distinction that “intuition is wrong and deliberations correct.” In familiar tasks, an experienced person can make quick, appropriate judgments. Conversely, even if you spend time thinking, the answer can still be wrong if the premises or information used are incorrect. It is better not to conflate differences in speed with differences in accuracy.

Returning to this discussion for Jev, the official explanation Official explanation derives the name “System One” from this contrast between rapid judgment and deliberation. It refers to the usage where you supply the necessary inputs and a question, and receive a limited answer—a usage that is likened to human intuitive judgment. It does not mean that the model reproduces the exact workings of the human brain.

For example, gathering a long history of inquiries, consulting the terms of service, and checking conflicting records all require investigation and multi‑step processing. Once those materials are assembled, you can place Jev in the part that quickly decides which department should handle the inquiry. Rather than judging performance solely from the name “System One,” we should view it as an explanation for understanding which portion of the system the model is intended to serve.

Who was Jevons, the person after whom Jev is named?

While “System One” derives its name from the way thinking proceeds, the model name “Jev” has roots in economics. William Stanley Jevons was a 19th‑century British economist. The relevant concept here is the “Jevons paradox,” named after him.

Jevons discussed the use of coal and industrial growth in The Coal Question, whose first edition appeared in 1865. Discussion of fuel efficiency in Chapter 7 states that improving fuel efficiency does not necessarily reduce overall consumption. Even if each machine uses less fuel, the total consumption can rise if the number of industries or machines employing that fuel increases.

Intuitively, if a steam engine can perform the same work with less coal, it seems coal usage should be reduced. That reasoning holds when the same number of machines runs for the same amount of time. However, when the cost of operation drops, machines can be applied to tasks that previously were not profitable. When products become cheaper, the quantity produced may also increase.

Consequently, even though the coal consumption per machine declines, the increase in the number of machines and the hours they operate can cause total consumption to actually rise. This “efficiency leading to higher consumption” is why the phenomenon is called a paradox. It is not that efficiency itself has failed; rather, the efficiency change has altered the scale of usage.

Let’s examine the relationship between quantities using a simple example. If the resources used per single operation drop from 10 to 1, that represents a 90 % reduction per operation. However, if the number of operations rises from 100 to 2,000, the total consumption increases from 1,000 to 2,000.

Before change: 10 per run × 100 runs = 1,000
After change: 1 per run × 2,000 runs = 2,000

These numbers are not historical measurements of coal nor performance metrics for Jev. They serve as an example to illustrate that per‑operation efficiency and total usage are distinct quantities. If the number of operations stays at 100, total usage drops to 100, and even when it rises to 500, the total becomes 500—still less than before the change. This shows that efficiency improvements do not automatically lead to higher overall consumption; we must also consider how much usage actually increases.

Jev announcement article explains that the model was named after Jevons. The underlying expectation is that if decisions can be invoked cheaply and quickly, it will not only lower the cost of the same tasks but also expand their use to jobs that previously did not employ AI.

For example, if a single call incurs a long wait time, the system might limit itself to merely classifying the incoming inquiry once. If it can be processed with a short wait, then use cases such as evaluating each candidate retrieved by a search one by one, or showing classification suggestions as the user types, become feasible. Speeding up a process while keeping the number of invocations fixed is different from increasing the number of places where it can be invoked simply because it has become faster.

Even when a classification model is introduced to handle inquiries that humans previously read one by one, it can not only process the same volume more cheaply but also potentially cover internal memos or old records that were not classified before. In that case, although each individual decision becomes cheaper, the total number of decisions the system makes overall increases.

However, this reflects the provider’s perspective embedded in the name; it does not mean that Jev’s accuracy or economic impact has been proven. You still need to verify separately how much usage will increase for your specific purpose, how total costs will change, and whether it is worthwhile to increase the number of decisions. What the name conveys is a direction toward not only reducing the cost of existing conversations but also broadly embedding decision‑making processes into software.

1.3 Relationship with Search and Text Generation

Asking Jev, “Please look up the return policy for this product,” is not the kind of use case where the system automatically gathers the necessary terms. You still need to obtain the return policy, verify the order’s date and status, and provide the information required for the decision. If you leave it to Jev, the question becomes something like, “Does this inquiry request a return?” regarding the retrieved documents.

Similarly, an embedding model can convert text into vectors to search for similar documents, but it does not guarantee that those similar documents are the correct basis for the decision. An LLM can generate explanatory text or natural responses for users. Before that, a structure that classifies the subject area handled by Jev could also be considered.

In the previous How to differentiate LLM, embedding models, Jev, and classic NLP , we considered separating tasks by type. This time, we’ll go a step further back and examine what kind of mathematical object a Jev decision is and what can be inferred from its numerical output. A simple comparison table cannot fully explain, for example, the difference between probabilities and confidence, nor the purpose of setting a threshold.

2. What Makes Using an LLM for Decision Processing Difficult

2.1 Text Read by Humans vs. Values Read by Code

When you ask an LLM, “Is this inquiry about billing, shipping, or the product?” it often returns a coherent answer. For a human reader, “I think it’s most likely a billing inquiry” might be sufficient. However, when using it from code, you must decide what value to treat as the output.

Should the classification for billing return the Japanese word “billing,” or should it return the program‑specific identifier billing? The use of English here is not because the input is being translated into English, but simply as an example of the class name that code would handle. The identifier could be in Japanese, but if the code is designed to receive billing, then “billing” would not match directly. Even though humans perceive them as the same meaning, when using it from code you must first decide which value to return.

Furthermore, should a reason be appended? What should be returned when classification is impossible? When an item spans multiple domains, should it be narrowed down to a single category? Should the response be truncated at its beginning, or parsed as JSON? If these points are ambiguous, even a semantically reasonable answer may be unusable by the program.

For example, the following two responses appear almost identical to a human.

This is an inquiry regarding billing.

The primary category is billing, but it may also involve return intent.

However, a code that checks only for exact matches with the string billing will find that the first does not match, and the second contains extra text. Conversely, if you match based on the presence of the word “billing,” you would also capture negative statements such as “This is not a billing inquiry.” When the output is designed on the assumption that a human will interpret it, additional processing is required to translate that interpretation into something the code can use.

This issue is not simply that LLMs always fail because they generate text. By using constrained generation based on JSON schemas or formal grammars, the shape of the output can be tightly restricted. Geng et al.'s research on Grammar-Constrained Decoding discusses methods for limiting possible outputs using formal grammars. Describing current LLMs as merely “models that break JSON” ignores these mitigation strategies.

Nevertheless, the mechanism that enforces format constraints and the mechanism that evaluates whether the content is correct remain separate. Even if the system can reliably return a JSON like {"department":"billing"}, there is still a chance that a shipping inquiry could be misrouted to a billing decision. Adhering to a schema is one condition for ensuring a decision’s correctness, but it is not sufficient.

2.2 Token Probabilities and Decision Probabilities

A typical autoregressive language model computes the distribution of the next token conditioned on the previous context and the tokens it has already generated. It repeats this process to construct a sentence. To put it simply, the probability of the entire sentence is the product of the conditional probabilities at each position.

P(sentence | input)
  = P(first token | input)
    × P(second token | input, first)
    × ...

Here, we want to distinguish between the LLM’s prediction of “what string to write next” and the “which classification is correct” that we are interested in. For example, suppose we examine the probability that the model will write “billing” next for a billing inquiry. Even if the model is making the same judgment, it might instead produce “This is a billing inquiry.” In that case, the first token it writes would be “This” rather than “billing.” Relying solely on the probability of starting with “billing” would not capture all instances where the model is judging the inquiry as billing.

If we restrict the response to a fixed classification label such as billing, we can reduce this phrasing issue. However, even then, the probability that the model selects that label does not automatically equal the actual proportion of cases where that label is correct. For instance, even if you collect many inquiries that the model assigns a 0.9 probability to as “billing,” you might find that only 70 % of them actually involve billing when verified. This figure is an illustrative example, but it shows that you need to compare the model’s output probabilities with the real outcomes.

In other words, what we actually need is not merely information that “the model tends to output ‘billing’,” but a probability that aligns with the true result—e.g., cases in which the model predicts a 0.9 likelihood of being billing should indeed correspond to roughly 90 % of instances that truly are billing. If you want to extract such a probability from an LLM’s output, you must first define which responses count as the same class and then evaluate whether those probabilities match the actual classification outcomes.

Of course, there are ways to limit the candidate set and read the label probabilities. You can also train a model in a supervised manner to match the desired classifications. The issue, however, is clarifying what each number represents—what probability it corresponds to. We should avoid the leap of assuming that because an LLM computes a probability, that number must reflect the real-world accuracy.

A more familiar example is when a model writes “Confidence: 90 %” into its output. That 90 % is part of the generated text. In practice, unless you collect many such responses and verify that roughly 90 % of them are correct, you cannot know whether the figure carries the probabilistic meaning users expect.

When a number is attached, the result appears more measurable than a plain textual answer. However, having a measurement‑like appearance is not the same as actually measuring the correct quantity. A value that rates an inquiry’s urgency at 0.9, a probability of “yes” of 0.9, or a concentration score of the response distribution at 0.9 each convey different meanings.

2.3 Decision Generation and Explanatory Text Tend to Be Combined

When you ask an LLM to make a decision, you naturally also want it to provide an explanation. An explanation makes it easier for humans to verify the decision and can serve as a clue for detecting misclassifications. However, we should consider whether the value the system needs and the explanatory text are always the same task.

If we classify ten thousand inquiries and have humans read only the roughly five hundred ambiguous cases, it may not be necessary to attach lengthy explanatory texts to every case. Generating explanations increases processing time, communication volume, and storage requirements. Producing explanations even for items that humans never read adds more data that later needs to be analyzed.

Moreover, when the generated explanations are well‑structured, they can give the impression that the judgments are correct. The explanation “A duplicate payment has occurred because two entries appear on the card statement” sounds natural, but in reality it may simply reflect a provisional charge alongside a finalized charge. Evidence that the customer made that claim is different from evidence that the system has confirmed a duplicate payment.

Reading the explanatory text does have value. However, that explanation should not be treated as an independent verification result. A reason string produced by the same model from the same input is not based on newly acquired payment data. Even if the amount of text increases, that does not necessarily mean the amount of evidence has increased—a point that is crucial for decision‑making systems.

2.4 The Cost of Using Long Inferences Every Time

When an LLM tackles a complex problem, multi‑step reasoning and the use of tools can be beneficial. However, not every classification requires the same depth of analysis. For questions that can be resolved briefly by looking at the input and the candidate answers, invoking a long generation process each time leads to noticeable latency as the frequency of processing rises.

For example, a single judgment may be tolerable for a user, but querying the same document sequentially on ten different points adds up the total time. While parallel invocation can sometimes reduce waiting time, one must also consider the cost of repeatedly sending the same input, the number of concurrent executions, API limits, and the need for retries on failure.

Answering ten questions in a single call reduces the number of communications. However, with typical LLMs the response itself is generated sequentially. It remains unclear how much the evaluation of individual questions can be kept independent, and how to fix the answer format. One must consider not only the number of input‑output tokens but also the overall processing architecture.

What matters here is not rejecting the use of inference time altogether. Tasks such as investigating ambiguous specifications, searching for contradictions across multiple documents, or isolating unknown bugs require thorough exploration. Treating quick judgments and tasks that require additional research to produce an answer as the same thing leads to flawed comparisons of speed and inappropriate choices of use cases.

2.5 Biases When Using an LLM as an Evaluator

Having an LLM read another model’s responses and score them—so‑called LLM‑as‑a‑Judge—is a convenient approach. It can handle far more cases than a human could compare, and it can evaluate the meaning of the text in depth. However, when used as an evaluator, it is necessary to study how factors such as the order and length of the input affect the outcome.

Zheng et al.'s research on MT-Bench and Chatbot Arena examines, alongside the usefulness of LLM‑based evaluation, issues such as the position of presented options, redundancy, and bias toward answers that are personally relevant. This does not mean that using an LLM as an evaluator is inevitably unusable; rather, it is a study aimed at determining whether the evaluation method itself contains bias.

For example, when comparing candidates A and B, if swapping their order—without changing their content—leads to a different judgment, then factors other than the content are influencing the result. If shortening an answer alone causes its evaluation to drop, we need to investigate whether length is being used as a proxy for quality. It is necessary to decide what constitutes a correct judgment before even naming the model.

This caution also applies to Jev. Adopting a different output format or training objective does not guarantee that all semantic biases disappear. In comparative experiments, one must fix not only the model being swapped but also the order of candidates, the questions, the input scope, and the evaluator’s criteria, in order to pinpoint what has changed.

2.6 There Are Studies on Probability Handling in LLMs

After reading this far, you might think that LLMs lack the ability to produce reliable probabilities. That would be an overstatement. Kadavath et al.'s 'Language Models (Mostly) Know What They Know' examines how models assess the correctness of an answer and their own confidence in answering, and studies the resulting probabilities.

Furthermore, Lin et al.'s 'Teaching Models to Express Their Uncertainty in Words' is a study that evaluates models trained to express uncertainty in natural language. Even when probabilities are output as text, appropriate training and validation for the intended purpose enable research on calibration.

Therefore, we do not want to describe the difference between LLMs and Jev as “one always produces false probabilities while the other always produces correct ones.” Instead, we should examine the differences in what output the model was trained for, the format in which it receives information, and the conditions under which it is evaluated. Jev’s distinguishing feature lies in its specification and design goals, which focus on using limited judgments and probabilities directly from software.

3. Reconstructing Jev’s Mechanism from Public Information

3.1 Three Elements Visible from the Outside

The aspects of Jev that can be explained from publicly available information can be broadly divided into three categories: the contract between input and response, the method for evaluating multiple responses, and the properties targeted during training. TypeSafe explains that it employs a new model architecture, parallel samplers, and a training method called RLCD. However, based on the official documents identified in this article, it is not possible to reproduce details such as the layer configuration or the specific optimization procedures.

The contract between input and response is easy for users to verify directly. By sending a query such as state, one receives values for each response type. The system is not primarily intended for re‑reading the generated explanatory text in order to extract options. The code that receives the return value can directly handle the probabilities assigned to each candidate.

Regarding the processing method, Announcement article explains that, unlike approaches that generate output token by token in an autoregressive manner, it uses a design that generates outputs in parallel. From the user’s perspective, this appears as a format that bundles multiple questions about the same state into a single request. Simply describing it as consolidating HTTP requests does not fully capture the differences on the model side that TypeSafe claims.

Regarding the training objectives, AI Primer describes RLCD as a reinforcement‑learning method that returns judgments together with calibrated probabilities. The specific losses and rewards used, as well as the data collection methods, are not disclosed in the materials examined here. The Brier loss and other concepts discussed later are general theories for understanding this objective.

flowchart TD
    A["Public description of Jev"] --> B["I/O contract<br/>state, question, typed answer"]
    A --> C["Evaluation method<br/>evaluate multiple outputs in parallel"]
    A --> D["Learning objective<br/>calibration of RLCD and probabilities"]
    B --> E["Verify against the API specification"]
    C --> F["Read the official documentation and measurements separately"]
    D --> G["The goal is public<br/>specific learning procedures are unknown"]

The term “mechanism” can encompass the HTTP format, the mathematical objects returned by the model, the learning algorithm, and even the execution hardware. Drawing a single internal diagram without indicating which parts are publicly disclosed can make speculation appear as if it were the actual implementation. Here, we begin the explanation from the publicly known boundaries.

3.2 What You Lose and Gain by Eliminating Text Generation

Free‑form strings are an extremely powerful output format. Explanations, code, equations, conversations, search queries, and procedures can all be expressed with the same mechanism. Users do not need to decide the answer format in advance; the model can generate the required text. When tackling unknown problems, this flexibility is a major advantage.

Conversely, for a task that merely selects a responsible department, such flexibility is unnecessary. The system does not accept nonexistent departments, and the tone of the response is irrelevant. If the set of permissible answers is already defined, that range can be explicitly communicated to the model.

Because Jev does not make free‑form text generation its primary output, tasks that require lengthy explanations or new code need alternative approaches. Instead, assuming a limited answer space, the system can be designed to return probability distributions. This involves giving up the ability to express anything arbitrarily and narrowing the portion that can be consumed programmatically.

At this point, we cannot assert categorically that “not generating text always means lighter computation.” Processing the input is still required, and the workload grows as the number of questions or candidates increases. The advantages of a processing method and its actual speed must be measured taking into account the model, input length, number of questions, network conditions, and service congestion.

Nevertheless, it becomes clear which kind of processing is being offered to users. Generating an explanation once and returning a hundred limited judgments once involve different input‑output structures and different amounts of work. Before merely listing speed figures, one should verify that the same question and the same quality are being compared.

3.3 Reading the Meaning of “Type‑Safe” Narrowly and Precisely

Jev’s announcement includes strong statements about typed output and hallucination. When considering an actual system, we need to separate out which kinds of failures it is meant to prevent. If the only candidates are billing and delivery, then preventing a case where no answer exists for super_department is simply a constraint on the answer space.

However, returning billing in a delivery response does not violate the candidate constraints. It may return an incorrect classification as a correctly typed value. Even if the candidate set is restricted, issues such as misreading the text, missing necessary information, or ambiguous questions remain.

Similarly, even if the returned probabilities lie between 0 and 1 and sum to 1, we cannot know whether they have been properly calibrated. Collecting cases where 0.99 was returned and finding a distribution in which only half are correct is still formally permissible. Code that checks only the numeric range or the total cannot detect such semantic problems.

flowchart TD
    A["Receive the answer"] --> B["Check format correctness<br/>type, candidates, numeric ranges"]
    B --> C["Assess judgment validity<br/>compare evidence with the correct label"]
    C --> D["Whether the probability has been calibrated<br/>verified across many cases"]
    D --> E["Whether this action is permissible<br/>considering cost, authority, and operational constraints"]

This diagram shows that a later condition does not automatically follow from an earlier one. It does not form a chain where “if the format is correct, the content is correct; if the content is reasonably correct, the numbers are correct; if the probabilities are good, the action can be taken.” In practice, each of these aspects must be verified by separate mechanisms.

Moreover, even with typed decisions, communication failures or API errors can still occur. Whether to treat a missing response as a “yes,” a “no,” or as a pending state is a design choice in the application. “The model adheres to its type” and “the service call always succeeds” are distinct conditions.

4. State Represents the Scope of the World the Model Observes

4.1 Even When the Question Is the Same, Different Evidence Leads to Different Answers

In the official description of State (see the link), you can pass strings, JSON objects, or arrays to state. These are the materials the model uses for its judgment. If only the inquiry text is provided, the model’s decision will primarily be based on that text’s content.

Official State documentation

From the preceding text, it might be possible to determine whether the user is requesting a cancellation. However, deciding whether there are duplicate charges or whether a refund can be issued immediately requires order and payment information. Treating these as the same question would cause the system to accept the customer’s claim as fact.

When adding the necessary data, it’s important to clarify what kind of observed information each piece represents. Customer statements, records obtained from the payment API, staff notes, and internal policies can all be expressed as strings, but their evidentiary nature differs. Rather than simply concatenating everything into one long text, it is better to label each piece with its source and role, which makes later investigation easier.

{
  "customer_message": "My card statement has two charges for the same amount.",
  "payment_observation": {
    "status": "not_checked"
  },
  "task": "Classify the type of inquiry"
}

This is an example used to illustrate the concept of state. not_checked does not mean that there is no problem with the payment; it simply indicates that the payment status has not yet been verified. Deciding not to treat unknowns as “no” is a choice made during the data design before using the model.

4.2 The Absence of Information versus the Failure of a Condition

For example, when evaluating whether a return is permissible, not knowing the purchase date is different from the fact that the return deadline has already passed. Assuming that a missing date means the deadline has been missed turns a lack of information into a negative conclusion. If this transformation occurs before the data is passed to the model, the model’s correct interpretation of the text will still lead to an incorrect result.

Even in a database, a condition like NULL, or an API response indicating that data has not been retrieved, or the customer’s silence, or a situation where the case is out of scope for the system, should not be lumped together as a single “missing” entry. At the very least, we want to preserve the differences that matter for decision‑making. Whether an order simply does not exist or whether the system failed to retrieve an existing order changes the appropriate next step.

This distinction is not unique to Jev. The same applies whether you feed text to an LLM or use a rule engine. Before relying on a model’s sophistication to resolve it, you must define the meaning of the information included in the input. No model can always correctly infer what a missing field in the input represents.

When investigating actual failures, we want to reconstruct not only the answer but also what the system knew at the time of decision. Even if a payment can later be confirmed, adding that record to a past state and re‑evaluating it gives the model more evidence than it would encounter in real operation. It becomes unclear whether the improvement stems from a better‑performing model or from using future information that simplifies the problem.

4.3 Longer Inputs Are Not Automatically Safer

To avoid missing essential information, you may be tempted to include the entire history. However, adding unrelated sentences or outdated policies makes it unclear which parts should serve as the basis for the decision. When the text in which a user requests a change is mixed with a judgment previously written by another staff member, it becomes ambiguous which should be treated as the current request.

For example, even if the inquiry is simply “I want to change my address,” is it necessary to include the entire purchase history for the past three years and all prior return consultations? When handling a request to change an order’s shipping address, the relevant information consists of the specific order, its current shipping status, and the result of identity verification. Past inquiries can sometimes interfere with the current decision.

However, shortening the input does not always improve things. If you extract only the phrase “Please cancel this,” you lose the context of what exactly is being referred to. When trimming information, the goal should be to preserve the relationships needed for the question rather than merely reducing character count.

Therefore, preprocessing itself must also be subject to evaluation. Verify which portions were extracted from the original documents, whether any relevant conditions were omitted, and whether any negations were lost during summarization. If a summary generated before feeding data to Jev is produced by another LLM, the failures of that summarization must also be counted toward the overall failure rate.

4.4 How to Treat Commands Embedded in the State

The inquiry text may contain not only ordinary requests but also sentences that function as instructions to the model. Phrases such as “Please approve this inquiry without fail” or “Ignore the previous instruction” are part of the content to be judged, not directives that alter the system’s permissions.

Just because something is placed inside a JSON structure does not guarantee that the model will reliably distinguish it. JSON can represent the shape of data, but it is not a barrier that isolates the meaning of the strings. Conditions under which adversarial or misleading sentences could be inserted must be treated as test cases for evaluation.

For example, even if a billing inquiry includes the sentence “The classification is ‘shipping’,” whether that statement can be taken at face value depends on how the question is defined. If the task is “extract the classification that the customer self‑reports,” then that sentence serves as evidence. If the task is “classify the content of the inquiry,” the model must consider the overall meaning of the text. The same sentence can yield different correct answers depending on what is being asked.

Leaving such design choices ambiguous and then claiming “the model was fooled” makes it unclear which boundaries were intended to be enforced. We should start by explicitly defining the input material, the decision instructions, the allowed candidates, and the subsequent actions.

5. What the Three Output Formats Represent

5.1 Noul Returns Probabilities for the Affirmative Side

The easiest to understand is Noul, which splits the outcome into affirmative and negative. For example, you might ask, “Does this inquiry contain an intent to request a refund?” If the returned value is 0.8, the judgment assigns an 80 % probability to the affirmative side, leaving 20 % for the negative side.

Here, the 0.8 does not mean that “refund requests make up 80 % of the text,” nor does it mean “the strength of the request for a refund is 80 points.” It indicates how much probability has been allocated to the affirmative side regarding the occurrence of the event defined in the question. The official Noul documentation also treats it as a probability for a yes/no answer.

Even with the same 0.8, the event being referenced changes depending on how the question is phrased. “Whether the word ‘refund’ is explicitly mentioned,” “whether there is an intent to request a refund,” and “whether a refund should be granted” are distinct issues. The first may simply be a matter of checking for a string. The second requires paraphrasing and understanding context. The third demands information such as contracts, payment histories, policies, and handling guidelines.

If an inquiry states, “It appears I was charged twice, so please refund one of them,” the intent to request a refund can be inferred. However, whether the double charge actually occurred cannot be determined from the text alone. Moreover, deciding whether a refund is permissible is a separate judgment. Combining these three aspects into a single question makes it ambiguous whether the model is merely reading the text or evaluating the business feasibility.

flowchart TD
    A["Inquiry body text"] --> B["Whether there is an intent to request a refund"]
    B --> C["Noul: Probability for the affirmative side"]
    C --> D["Treat as a refund consultation"]
    D --> E["Review payment history and terms of service"]
    E --> F["Determine separately whether a refund is possible"]

A value of 0.02 does not indicate low confidence. It reflects a prediction that strongly leans toward the negative side, with the affirmative possibility being low. Both 0.98 and 0.02 are biased for a binary choice, while values near 0.5 are difficult to decide either way. However, the fact that the number leans toward an extreme does not guarantee that the prediction is correct. This point will be addressed later in the discussion of calibration and evaluation.

When using Noul, it is better to narrow the question to a single meaning. Asking, “Is there a refund request, is the deadline approaching, does it comply with the policies, and is it not due to fraudulent use?” makes it unclear which condition caused the negative result. Instead, ask each condition separately and verify the determinable ones in code. Then, in the subsequent step, assemble whether the required conditions have all been satisfied.

This isn’t simply a matter of generating more questions. Turning even things that can be determined without consulting the model into questions increases the number of places where errors can be introduced. If the deadline is stored as a date‑time in the database, the time comparison should be performed in code. Noul’s role should be limited to aspects that cannot be resolved by definitive calculations, such as extracting intent from text.

5.2 Choice distributes probabilities over a set of candidates

In Choice, you select from the prepared candidates. If the responsible department for an inquiry is divided into “Billing,” “Shipping,” “Product,” and “Other,” each gets a probability, and the candidate with the highest probability becomes the result of the Choice. In the Choice specification , the probabilities for each candidate, the selected choice, and its confidence are returned.

For illustration, consider the following output. This is not a value measured from the actual API.

In this example, “Billing” would be selected. However, if you retain only the chosen string, you cannot distinguish between a selection made with 0.78 confidence and one that was barely chosen with 0.29 confidence. With four candidates, even if each has a low probability, you can still pick the one with the highest value. “Highest” is merely a relative measure compared with the other candidates.

The definition of the candidates also becomes part of the problem as seen by the model. The name “Billing” alone does not make it clear whether it covers issuing receipts, changing payment methods, duplicate charges, or refund inquiries. The boundary with “Shipping” also becomes ambiguous when the question concerns shipping fees. We want to attach explanations to each candidate and explicitly indicate which scope they should cover in the business context.

Candidates can also overlap. “Urgent” and “Billing” are not the same type of category. It is common for an inquiry to be both about billing and urgent. If these two are placed as mutually exclusive options within the same Choice, you end up forcing information that is naturally compatible into only one of the options. It is better to ask about the responsible department with a Choice and to inquire about urgency with a separate Noul or Score, which aligns with the structure of the situation.

Moreover, if none of the candidates is the correct answer, the fact that some option was selected is meaningless. If you remove categories such as returns, accounts, or legal matters from the candidate list and then force every inquiry into a four‑option choice, those cases will still be forced into one of the four buckets. Including options like “Other” or “Insufficient information” is a design choice that represents such situations. However, merely adding those candidates does not guarantee that unknown issues will be reliably identified.

It is also important to recognize that changing the set of candidates changes the question itself. A billing option that receives a probability of 0.9 in a two‑choice scenario of “billing or shipping” does not necessarily retain a 0.9 probability when the choices are expanded to five options—billing, shipping, returns, contracts, and accounts. If the newly added candidates provide more appropriate explanations, the distribution of probabilities will shift. Before labeling this as mere output instability, you need to verify what conditions the probabilities are based on.

5.3 Score aggregates a distribution assigned to ordered stages into a single numeric value

Score deals with ordered stages defined in text. For example, regarding the urgency of an inquiry, we can define the following four stages.

Score returns the weighted average of the stage indices based on the probabilities assigned to each stage. According to the Official Score description , if the stage indices are 0, 1, 2, 3 with probabilities 0.1, 0.2, 0.6, 0.1 respectively, the score would be as follows.

score = 0 × 0.1 + 1 × 0.2 + 2 × 0.6 + 3 × 0.1
      = 1.7

The value 1.7 is not a direct measurement of physical urgency. It is the result of assigning a distribution to the predefined stage indices and then collapsing that distribution into a single number. Changing the descriptions of the stages changes the meaning of 1.7. There is no guarantee that the intervals from 0 to 3 correspond to equally spaced levels of actual damage.

Looking only at the average can be misleading. Consider three stages, 0, 1, and 2. If all the probability mass is concentrated on stage 1, the average is 1; likewise, if the probability is split evenly between stage 0 and stage 2, the average is also 1. However, in the first case the system has clearly chosen an intermediate state, whereas in the second it is uncertain between the two extremes. Even though the averages are identical, the state of the decision is quite different.

Treating the latter as “normal urgency” could cause very important inquiries to be overlooked. At a minimum, the design should consider both the mean and the distribution. For example, if the probability of the highest stage exceeds a certain threshold, additional verification could be performed—a processing approach that differs from simply looking at the average. Which handling is appropriate should be decided based on the business losses it entails.

In the official description, each stage of a Score is evaluated independently, and the explanation must not rely on the stage number or on adjacent stages. Instead of merely stating “slightly stronger than the previous stage,” the description should convey what that stage means on its own. If the stage descriptions are vague, calculating the average to many decimal places does not yield a precise metric.

5.4 The three formats are not merely rephrasings of the same problem

Noul, Choice, and Score each constrain the answer space in a different way: a binary decision, mutually exclusive candidates, and ordered stages, respectively. Simply forcing everything into a Score does not guarantee a fine‑grained evaluation, nor does using a Choice automatically resolve any ambiguity.

First, the meaning of the question must align with the output format. Even a question like “Should we respond?” requires different evidence and liability depending on whether it concerns assigning it to a department, processing a refund, or contacting the police. Before deciding on the output format, it is advisable to list precisely which judgment you are trying to extract.

Rather than forcing business processes to fit the types a model can produce, choose the type based on the inherent structure of the decisions in the business. Following this order also reveals situations where using Jev would be inappropriate. If the input is unambiguous and a definitive answer can be computed directly, there is little reason to route it through a model.

6. What conditions underlie the probabilities produced by Jev

6.1 It answers based on the information provided, not the entire world

If we express the probability in a slightly mathematical way, what Jev handles can be thought of as the following entity.

P(answer | state, question, definition of candidates, model)

This is not a definitive statement about the internal calculation formula; it is a perspective for interpreting the output. It is not an “answer probability” without any conditions. It does not directly examine contract information that is not part of the state, nor future events that have not yet occurred.

For example, if you ask “Is the customer being double‑charged?” and the state contains only the customer’s complaint, the model will make its judgment based on that complaint text. When there are two payment records versus only one, the information supplied is different. Even if the readability of the text is the same, a change in the content of the evidence naturally leads to a different prediction.

If we rephrase the question to “Does the text claim a double charge?” the required evidence becomes less extensive, because the problem can now be answered solely from the text. Conversely, expanding the question to “Has the customer actually been double‑charged?” requires external fact‑checking. Before evaluating the model’s performance, we should verify that we have not conflated these two cases.

A similar situation arises when humans read a text and make a judgment. “The person says so” and “It actually happened” are distinct. When Jev produces a numeric value, it can give the impression that the latter has been objectively verified. However, the scope of the evidence supplied does not change. Converting it to a number does not create additional facts that were not observed.

6.2 Problems with closed candidates versus problems with open answers

In situations such as selecting one of four responsible departments, the set of possible outputs can be defined in advance. Such closed problems are easy to evaluate; you can record exactly where the error occurred, e.g., “a billing inquiry was misclassified as shipping.”

In contrast, a question like “How should we handle this customer?” does not have a predetermined set of answer candidates. Possible responses include follow‑up questions, refunds, providing policy information, consulting another department, and even considerations of politeness or legal phrasing. At this stage, the work of generating the response still falls to the LLM. You cannot simply use the classification produced by Jev as the customer‑facing answer.

Whether a closed set of candidates can be created depends less on the specific text being processed and more on what the system is intended to do. For the same inquiry, you might use a Choice to pick the responsible department, an LLM to compose a reply, or a parser/database to verify an order number. There is no need to feed everything into the same model merely because the input is natural language.

6.3 Probability is not a universal expression of ignorance

When “unknown” is treated as a 0.5 score, two distinct situations can become conflated: one where there is insufficient data to make a clear judgment, and another where there is enough data but the case lies near the decision boundary. Both can yield scores close to 0.5, yet the appropriate next steps differ.

If the problem is a lack of data, you can improve the situation by retrieving order histories or asking the customer additional questions. If the issue is a borderline case, you may need to define operational rules for the relevant department. “Deciding whether to route a concern about ‘high shipping costs’ to the billing side or the delivery side is not a problem of the model uncovering hidden truths; it becomes a matter of the organization defining its responsibility boundaries.

Moreover, a model can sometimes be confident about inputs it has never seen before. If only low‑confidence scores are used to identify unknown cases, you may overlook instances where the model returns a high score despite the input being truly unknown. We want to be able to flag cases for review based on factors other than the model’s numeric output—such as input format, target language, missing documentation, or intents that fall outside the candidate set.

7. Being able to evaluate in parallel is not the same as being independent

7.1 Asking multiple perspectives from the same state

Classification of an inquiry, whether a refund request is present, and its urgency can all be based on the same text. In Jev, you can provide multiple questions at once. This differs from processing where you read the text, generate an answer to one question in prose, and then feed that prose into the next question.

flowchart TD
    S["Common state: the inquiry and verified information"] --> C["Choice: responsible domain"]
    S --> N["Noul: Presence or absence of a refund request"]
    S --> G["Score: urgency of response"]
    C --> R["Combine results in subsequent code"]
    N --> R
    G --> R

With this structure, you can assess urgency without waiting for the classification result. Even as the number of questions grows, the waiting pattern changes compared to generating a free‑form explanation for each question one by one. However, the cost and constraints of adding more questions do not disappear. The actual response time also depends on factors such as the length of the state, the content of the questions, and server load, so you need to measure it with your own inputs.

7.2 It does not guarantee statistical independence

Evaluating questions as independent is different from assuming that the random variables representing the answers are statistically independent. An inquiry that includes a refund request is also likely to be classified as a billing issue. The two judgments are related through the content of the text.

For example, suppose the probability of a refund request is 0.8 and the probability of a billing classification is 0.9. To compute the “probability that both occur” simply as 0.8 × 0.9 = 0.72, you would need to assume independence. It is hard to say that this assumption naturally holds for the content of an inquiry.

In general, the probability that both occur should be evaluated using the conditional probability of one given the other.

P(A and B) = P(A) × P(B | A)

Having obtained P(A) and P(B) separately from Jev does not give you P(B | A). Grouping multiple questions into a single request does not fill this gap. If you need a combined probability, you must either define the joint event directly and evaluate it, or verify the combination method using separate evaluation data.

7.3 Logical consistency must be enforced in the code

When you ask separate questions about the same event—“Is it A?” and “Is it not A?”—the two positive probabilities do not necessarily sum to 1. This is because the wording and interpretation of the questions can differ, and there is no guarantee of a mechanism that simultaneously constrains both outputs. While a single Noul’s affirmative and negative can be treated as complements, a negative statement asked as a separate Noul is evaluated independently.

If you place mutually exclusive candidates within a single Choice, they can be treated as a probability distribution at least within that question. Conversely, you must separately verify that conditions split across multiple Nouls do not conflict. If you treat “refunded” and “not refunded” as mutually exclusive states, prioritize the database’s state transition and do not allow the model’s output to advance both states simultaneously.

Do not assume that writing extensive logical explanations for the model will automatically guarantee the system’s invariants. Conditions such as allowing a refund only once, not disclosing information to parties other than the order owner, and not treating an unsettled payment as already settled must be enforced regardless of the model. Even when using Jev, code that checks consistency is still required.

8. Models that Predict Correctly and Models with Accurate Probabilities

8.1 Accuracy Alone Does Not Reveal the Quality of Probabilities

Consider a model that solves 100 binary‑choice problems and gets 90 correct. Its accuracy is 90 %. However, the meaning of the probabilities changes depending on whether the model assigned a probability of 0.99 to every answer, 0.9, or 0.6.

If, out of 100 cases assigned a probability of 0.99, only 90 were actually correct, the probabilities are overly high. Conversely, if 90 out of 100 cases assigned a probability of 0.6 turned out to be correct, the model is under‑estimating the likelihood for that set. In either case, selecting the option with the higher probability would still yield the same 90 correct answers. Looking only at accuracy hides this distinction.

When probabilities are used as criteria for automated processing, this distinction becomes significant. For example, if the policy is to process automatically when the probability is 0.99 and to route the case to a human when it is 0.6, the amount of work handled by automation will differ even though the overall accuracy is the same. In a model whose probabilities are overly high, a larger share of erroneous decisions will be automatically processed.

A representative study that formalizes this distinction is Guo et al., On Calibration of Modern Neural Networks . Even neural networks with high classification performance do not necessarily produce output probabilities that match the actual frequency of correct answers. When considering Jev’s “calibration,” it is first necessary to grasp this distinction.

8.2 Calibration Is Verified on Groups of Similar Predictions

Let p denote the model’s predicted probability for the positive class in a binary choice, and let y be the actual outcome (1 for positive, 0 for negative). Ideal calibration can be expressed roughly as follows.

When collecting cases with predicted probability p, the proportion of affirmative cases becomes p

If you gather a sufficiently large number of cases predicted with probability 0.8 and find that about 80 % of them are actually positive, then the predictions in that range are calibrated. This does not mean that a single inquiry is “80 % correct.” Each individual case is either positive or negative, and the quality of the probability is assessed across multiple observations.

This can be counterintuitive. Even if a single case predicted with probability 0.8 turns out to be negative, that does not by itself indicate a poor probability estimate. A well‑calibrated 0.8 prediction will still be negative about 20 % of the time. Conversely, a single instance where a 0.99 prediction happens to be correct does not prove that the calibration is good.

In real data, not every prediction will be exactly 0.8. Instead, you group predictions into intervals such as 0.76–0.84 and compare the average predicted probability with the observed positive rate within each interval. The table below is a hypothetical example illustrating how to view the calculations.

The higher the predicted probability, the more it exceeds the actual positive rate. In this example, the model appears to be overestimating the likelihood of a positive outcome. However, changing the interval boundaries would alter the interpretation. In intervals with few samples, random fluctuations are larger. Simply observing a discrepancy does not allow you to conclude that the problem lies solely with the model.

flowchart TD
    A["Evaluation cases and ground‑truth labels"] --> B["Record the model’s predicted probabilities"]
    B --> C["Group cases with similar predicted probabilities"]
    C --> D["Compute the average predicted probability"]
    C --> E["Compute the actual affirmative rate"]
    D --> F["Compare the two, also checking case counts and variability"]
    E --> F

8.3 Positive‑class probability in a binary choice and the accuracy of the selected answer

When a Noul’s p is 0.1, that value indicates a low likelihood of the positive class. If you choose the negative answer, the probability assigned to that answer is 0.9. Calibration that examines the frequency of the positive class and calibration that examine the frequency of the chosen answer being correct involve different aggregated values.

For example, among cases where p is around 0.1, you can verify that the positive class occurs about 10 % of the time. Viewed from the other perspective—as the accuracy of the negative answer—this corresponds to roughly 90 % correctness. Both figures stem from the same information, but you must not compare them by saying “the calibration is poor because the probability is 0.1 yet the answer is correct 90 % of the time.” It is essential to keep the event whose probability you are assessing consistent.

In a Choice setting, you can compare the probability assigned to the selected candidate with the proportion of times that candidate was actually correct. However, this alone does not let you verify the probabilities of the candidates that were not selected. If an important category such as “refund” is consistently ranked as the second‑most likely option, aggregating only the top‑ranked candidates can mask an underestimation of that category. For classes that are critical to the business, you should also examine each class individually.

8.4 Being calibrated does not necessarily mean it is useful

Consider a task where the positive class occurs in only 10 % of cases, and a model that returns 0.1 for every instance. Overall, the predictions match the actual frequency. However, the model cannot distinguish which individual cases are more likely to be positive. What is needed is not only accurate probabilities but also the ability to differentiate between cases.

The model’s discriminative power can be thought of as the fact that cases assigned higher probabilities tend to have a higher proportion of positives, while those assigned lower probabilities have fewer positives. Moreover, calibration requires that each probability aligns with the observed frequency. Rather than returning an average probability for every case, we want predictions that differentiate cases while also matching the true frequencies.

Therefore, selecting a model based on a single calibration metric alone is risky. One must consider identification, calibration, errors at specific thresholds, and the number of cases it can handle together. It is also advisable not to overgeneralize the claim that Jev emphasizes calibration to mean that calibration alone can determine everything.

8.5 Even if it works overall, it may fail in some parts

A further difficulty arises when aggregating data causes errors to cancel each other out. Suppose the model assigns high probabilities to Japanese‑language inquiries and low probabilities to English‑language ones. When all cases are combined, the average may look correct, yet the probabilities could be inaccurate for each individual language.

The same phenomenon occurs with factors such as the difficulty of a case, the department handling it, the user’s phrasing, the length of the material, or missing inputs. If your system deals with long Japanese queries, a calibration that performed well on short English problems will not be sufficient to verify those conditions. It is essential to evaluate performance according to the distribution of the data you will actually apply it to.

However, the more finely you segment the data, the fewer cases fall into each segment. When you combine factors such as department, language, style, length, and time, a single segment may contain only a few instances. From such a small number you cannot reliably infer precise calibration. You should identify the segments that are most important for your target, and report both the number of cases and the associated uncertainty. Evaluation also cannot be reduced to a single, all‑purpose table.

9. How to turn the quality of probabilities into a score

9.1 Not just whether you were wrong, but how much probability you assigned

Accuracy measures whether the final chosen answer was correct. In contrast, when learning or evaluating probabilities, we want to assign a score to the predictive distribution itself. The tool for this is the scoring rule. Here we will treat it as a loss, so lower values are preferable.

For example, if a case where the positive class is correct receives a score of 0.9 versus 0.51, both would still be classified as positive and thus be correct. However, they represent different probability predictions. Conversely, if a case where the negative class is correct receives a score of 0.99, we want to reflect in the evaluation that the model was strongly confident yet wrong, rather than merely making a slight mistake.

If we simply think “praise the model for assigning a high probability to the correct answer,” we could devise a scoring method that makes a strategy of assigning probabilities of 0 or 1 to every case advantageous. To elicit honest probabilistic predictions, we need a scoring rule for which, on average, reporting the true probability is optimal. Such rules are called proper scoring rules, and when a unique optimal distribution exists they are referred to as strictly proper.

The foundational theory is organized in Gneiting and Raftery's Strictly Proper Scoring Rules, Prediction, and Estimation . The equations that follow are not intended to describe the exact formulas Jev uses for training; they are presented as a general tool for evaluating probabilities.

9.2 Considering the Brier score for binary outcomes

For binary outcomes, the Brier score is the squared difference between the predicted probability p and the actual result y.

Brier score = (p - y)²

p: predicted probability of affirmation
y: 1 if affirmative, 0 if negative

If the positive class is correct, the loss is 0.01 when p = 0.9 and 0.16 when p = 0.6. If the negative class is correct, the loss becomes 0.81 when p = 0.9. Predictions that are closer to the correct side incur smaller losses, while predictions that are strongly biased toward the wrong side incur larger losses.

Even this simple formula shows why reporting the true probability is preferable. If the actual probability of the positive outcome is q, the average loss can be expressed as follows.

Expected loss = q × (p − 1)² + (1 − q) × p²
         = (p - q)² + q × (1 - q)

When q is fixed, the final term does not change as p varies. The first term becomes zero when p is set to q. Therefore, the average loss is minimized when p = q. If the true probability of the positive class is 0.7, answering 0.7 yields a lower expected loss than always answering 1.

Of course, in practice we cannot know the true q directly. We evaluate it based on the observed cases and their outcomes. What this calculation shows is that the scoring method is not designed so that consistently pushing probabilities toward the extremes is always advantageous.

When extending to multi‑class settings, one uses the difference between the predicted probabilities for each class and the 0/1 vector indicating the correct class. Because the definitions differ in how the sum or average is taken, when comparing reported Brier scores you must verify whether the task is binary or multi‑class and which normalization was applied. Listing only the numbers can obscure these differences.

9.3 Log loss penalizes strongly incorrect predictions more harshly

Another representative example is log loss. For a binary case, the formula is as follows.

log loss = -[y × log(p) + (1 - y) × log(1 - p)]

When the positive class is correct, we take the negative logarithm of the predicted probability for the positive class. Assigning a probability of 0.9 to the correct class yields a small loss, whereas assigning only 0.01 yields a large loss. Using the natural logarithm, the loss is about 0.105 when the probability for the correct class is 0.9, about 0.693 when it is 0.5, and about 4.605 when it is 0.01.

The loss grows larger as the assigned probability for the correct class approaches zero. This property strongly penalizes cases where an event treated as “impossible” actually occurs. While this characteristic is useful, it also produces large values when the labels in the evaluation data are incorrect. If an excessively large loss appears, you need to examine not only the model but also the inputs and the labels.

In implementation, you must decide how to handle cases where rounding causes p to become exactly 0 or 1. Simply discarding those instances because the logarithm is undefined would eliminate the most severely mis‑predicted cases from the evaluation. If you clip the values, record the lower bound. Keep separate records of the processing performed for evaluation purposes and the predictions actually produced by the model.

The Brier score and log loss assign losses differently even for the same prediction. In particular, their strictness toward errors that lie near the extremes differs, so the ranking of two models may not be the same. Neither metric alone explains everything; it is best to consider them together according to the intended purpose.

9.4 Calibration error and proper losses are not the same

A commonly used form of expected calibration error (ECE) partitions predictions into intervals and sums, weighting by the number of cases, the difference between the average predicted probability and the actual frequency within each interval. While this is intuitively easy to understand, its value changes with the number and placement of intervals, and with finite data it also includes random fluctuations.

Moreover, minimizing ECE alone is not the same as producing good probabilistic predictions. If you output the overall positive rate for every case, the average within each interval will tend to match, but the model’s ability to distinguish individual cases remains poor. Losses such as Brier score or log loss can be used to evaluate probabilistic predictions that incorporate this discrimination.

This distinction is also important when inferring the specific learning method from the name RLCD. From the statement that they “emphasize calibration” alone, you cannot tell whether they are directly using ECE as a reward or optimizing the Brier score. The methods that can be employed in theory are not necessarily the same as those implemented in the product.

10. What is known and unknown about RLCD

10.1 What constitutes a good output for learning

RLCD is described as an abbreviation for Reinforcement Learning for Calibrated Decisions. In TypeSafe AI’s AI primer , the learning objective is framed not only toward responses that are preferred in text or that can be verified, but also toward decisions and calibrated probabilities.

This distinction becomes clear when you consider what is expected of the output. A model that returns friendly, easy‑to‑read answers to inquiries will have its performance judged by the usefulness and pleasantness of the text. A model that supplies mathematical answers can be evaluated by whether the answer can be verified as correct. In a decision‑making model, you want to assess not only which answer is chosen but also the probability assigned to it.

However, these learning objectives are not mutually exclusive. Even in learning that relies on human preferences, a response that honestly conveys uncertainty can be valued as a desirable output. Likewise, learning that uses verifiable results can be designed with considerations for errors and uncertainty. It is better not to adopt a binary view that “RLHF is always overconfident, while RLCD is always honest.”

What can be said about Jev is that the provider explicitly states that its purpose is decision‑making and calibration. Whether that purpose is actually achieved in one’s own work must be evaluated separately. An explanation of the learning objective does not serve as a proof that evaluation is unnecessary.

10.2 The reward function cannot be reconstructed from the name alone

From the publicly available documentation, we could not determine which loss function is used in actual training, how it is converted into a reward, what data distribution the model is trained on, or at which stage probabilities are calibrated. The proper scoring rules described earlier are useful for understanding the background of RLCD, but they do not explain the specifics of Jev’s implementation.

Even when the goal is the same—learning calibrated judgments—there are design choices such as using direct probabilistic loss, incorporating the loss into the reward, applying additional calibration steps, or combining multiple objectives. If it is unclear which approach was adopted, that part should be treated as unknown.

flowchart TD
    A["Publicly stated learning objectives"] --> B["Return the decision"]
    A --> C["Emphasize probability calibration"]
    D["General theory: proper scoring rule"] --> E["Background for considering the objective"]
    E -. "Does not serve as a basis for definitive implementation" .-> F["Jev’s specific reward function and learning procedure"]

Leaving this ambiguous leads to a two‑step explanation that lacks justification, such as “Jev uses the Brier score, so the probabilities are correct.” It is uncertain whether they actually use the Brier score, and even if they do, matching probabilities to an unknown target distribution is a separate issue.

10.3 When the training and operational distributions differ, the probabilities shift

If, for example, refund inquiries dominate during training while shipping inquiries dominate in production, the prior probabilities will change. Changes in wording or policy also alter the meaning of the same terms. When a new pricing scheme is introduced, judgments based on past knowledge may no longer apply.

Even a model trained with calibration in mind does not guarantee accurate probabilities across all situations beyond these changes. It is necessary to narrow the scope of cases at the entry point of operation, evaluate them within that scope, and verify that the situation remains consistent over time. In particular, an operational approach that simply decides once—“the current model is calibrated”—and then stops will miss any drift that occurs after changes.

In this respect, updating the model itself also constitutes a shift in the underlying distribution. Even when the same input is queried in the same format, a new model may return different probabilities. Even if classification accuracy improves, the number of cases handled by automated processes using existing thresholds can still change. This is why it is important to record the model name and version of the question.

11. How to interpret confidence

11.1 Summarizing the shape of a distribution into a single value

Jev’s Choice and Score include a confidence metric. According to the official documentation , this metric summarizes the concentration of the predicted probability distribution into a single value. Noul does not have a separate confidence metric.

The crucial point here is that confidence is not a direct measurement of the model’s internal state. It is presented as a value that reflects how concentrated the probabilities are across the candidate outcomes—that is, it relates to the shape of the returned distribution. A high value can be interpreted as indicating that the prediction is relatively clear between the candidates or stages.

However, a clear distribution does not guarantee that the correct answer is identified. Misreading the documentation can lead to a situation where the distribution is strongly biased toward a single candidate. If the definition of the candidates is incorrect, the model will produce a definitive answer for that erroneous problem. Cases where a high confidence leads to a wrong prediction also need to be collected and examined during evaluation.

11.2 Maximum probability, margin, and entropy are different summaries

Common ways to summarize the shape of a distribution include looking at the highest probability, examining the difference between the top two probabilities, or measuring entropy. This explanation is intended to compare these viewpoints and does not assert that Jev’s confidence is calculated by any specific formula. The official page we consulted does not provide a concrete calculation method.

Consider a four‑choice scenario with the following two distributions.

The maximum probability is 0.55 for both. However, in Distribution A the top two candidates (1 and 2) are almost evenly contested, whereas in Distribution B candidate 1 is clearly larger than the other three. If we look at the gap between the top two, A has a difference of 0.15 and B a difference of 0.40. On the other hand, when considering the overall spread of the distribution, B is more evenly distributed among the remaining candidates. The assessment changes depending on which aspect you prioritize.

Entropy, for a discrete distribution, can be expressed as follows.

H = -Σ p_i × log(p_i)

It is small when all probability mass is concentrated on a single candidate and grows larger as the distribution becomes more evenly spread. Because the maximum possible entropy changes with the number of candidates, care must be taken when comparing distributions with different numbers of candidates directly. Normalization can be used, but no normalization automatically reflects the correctness required in business contexts.

11.3 Do not confuse Noul’s proximity to the extremes with the probability of being correct

You can convert how far Noul’s p deviates from 0.5 into a single metric in your own code. For example, using the absolute value lets you treat both a strong bias toward the positive side and a strong bias toward the negative side as equally clear binary choices.

Example of proximity to the extremes = 2 × |p − 0.5|

This value becomes 0.8 whether p = 0.9 or p = 0.1. However, a value of 0.8 does not mean there is an 80 % chance of being correct. It is merely a metric that transforms the distance from 0.5. If you wish to interpret it that way, you need to evaluate the relationship between this metric and the actual accuracy rate.

In particular, a high probability of the positive class should be distinguished from a clear decision. For a question like “Is this fraudulent?” a p of 0.01 indicates a weak positive (fraud) bias and a strong negative bias. You must not write code that treats the case as fraudulent simply because the confidence is high. The predicted direction and the degree of certainty about that direction need to be handled separately.

11.4 Do not apply a single threshold to all tasks

The impact of an error differs between a process that changes the order of items displayed on a screen and a process that finalizes a refund. Even if the confidence is the same, there is no reason to use identical criteria for allowing actions. The handling should be adjusted based on whether the action can be undone, whether the user can be consulted, whether the mistake will be communicated externally, and how large the potential loss is.

You should also avoid copying the thresholds shown in the official sample code and applying them unchanged to every use case. The sample merely illustrates a control strategy; it is not a value that has been measured to stay within an acceptable error range for a specific Japanese‑language business scenario. Start by examining your own evaluation data to see how many cases would be automatically processed and how many would be misclassified if you adjust the threshold.

12. Moving from probabilities to actions requires considering loss

12.1 Choosing the correct answer versus choosing a good action

A model’s output is a prediction, while the system’s behavior is a decision. It is not necessarily appropriate to adopt a rule that says “perform the positive‑side processing because the positive outcome is most likely,” because the consequences of errors differ between the two sides.

For example, overlooking an inquiry that could indicate a serious failure would incur a large loss, whereas sending a regular inquiry to detailed verification would incur only a small loss. In such a case, even if the probability of a serious failure is below 0.5, it may still be rational to route the case to verification.

Conversely, when considering issuing a refund, mistakenly refunding a case and delaying the verification of a case that should receive a refund are not symmetric. Moreover, there are customer losses and delays in response, so the decision cannot be based solely on the merchant’s cost. How each type of loss is treated must be defined by humans.

12.2 Deriving a two‑choice threshold from a simple loss table

For illustration, consider a binary decision: whether to route a case to verification or to proceed with normal processing. Let C_FP denote the loss incurred when a non‑target case is sent to verification, and C_FN denote the loss incurred when a target case is sent to normal processing and thus missed. For correctly handled cases, we simplify the loss to zero.

Let p be the probability that verification is needed. The expected losses for each option are then as follows.

Expected loss for forwarding to confirm = (1 − p) × C_FP
Expected loss for normal processing = p × C_FN

We organize the condition under which routing to verification yields a lower expected loss.

(1 - p) × C_FP < p × C_FN

p > C_FP / (C_FP + C_FN)

If the loss from a missed case is set to 99 and the loss from an unnecessary verification to 1, the threshold becomes 0.01. Under this simple assumption, whenever the probability exceeds 1 %, sending the case to verification reduces the expected loss. Using a threshold of 0.5 is only justified when conditions such as treating false positives and false negatives as equally costly are in place.

The p used here is the probability of the event in question. You cannot simply substitute a generic confidence value. Moreover, if the probabilities have not been calibrated on your own data, the expected‑loss calculation will be off. Probability calibration and decision‑making are distinct concepts, but they intersect at this point.

12.3 A loss table cannot capture every aspect of reality

In practice, the verification staff may be busy, so increasing the number of cases sent to verification lengthens wait times. Errors in normal processing also vary depending on the amount involved or the nature of the case. Even correct handling incurs costs, and verification can still be mistaken. The two‑row table presented earlier does not represent all of these factors.

Nevertheless, explicitly defining losses is valuable. Rather than a vague rule such as “if the probability is 0.9 or higher, it’s safe,” it makes clear which kinds of errors you want to reduce. If you also take the verifier’s capability into account, you must add the residual errors after verification and the losses caused by longer wait times.

There are also constraints—such as safety, rights, or legal requirements—that cannot be easily reduced to an average cost. A small expected loss does not justify authorizing a refund without proper authority. It is necessary to separate constraints that must be observed regardless of probability from the actions chosen based on probability.

flowchart TD
    A["Model's predicted distribution"] --> B["Evaluate probabilities on the target data"]
    B --> C["Define loss and verification cost for each action"]
    C --> D["Select candidate actions"]
    D --> E{"Check whether permissions and operational constraints are satisfied"}
    E -->|Yes| F["Execute or confirm with the user"]
    E -->|No| G["Do not execute"]

12.4 “Hold” is also an action

There is not only the option of deciding yes or no, but also the option of deferring the case to gather more information. If payment history has not been obtained, acquire it. If the documentation contains contradictions, verify with the responsible person. If the customer’s intent is unclear, ask additional questions. Deferring is not a failure to take no action; it is a step to obtain information.

However, deferring also incurs costs. The waiting time for inquiries lengthens, and the number of cases handed over to humans increases. While deferring everything would prevent the model’s erroneous decisions from being acted on directly, it would diminish the purpose of deploying such a system. It is necessary to consider both error rates and processing volume when deciding how much to automate and where to intervene.

13. Design that leaves difficult cases aside, and risk coverage

13.1 Process only cases that can be decided

Instead of a classifier that must answer every input, consider a classifier that processes only inputs meeting certain criteria and defers the rest. This problem has been studied as selective classification. Geifman and El‑Yaniv's Selective Classification for Deep Neural Networks is one representative example.

Coverage is the proportion of the total cases that are taken on for a decision. Risk is defined as the error or loss incurred on the cases that are taken on. Deferring difficult cases can sometimes reduce risk, but it also lowers coverage. We need to examine under what conditions this trade‑off is operationally meaningful.

For example, if 80 out of 100 cases are automatically classified and 8 of those are misclassified, the error rate of the automatically classified set is 10 %. The fact that the remaining 20 cases were deferred, that the overall error rate is 8 %, and that the error rate of the automatically classified portion is 10 % are distinct points. If the denominator is left ambiguous, the report can make performance appear better simply by increasing the number of deferred cases.

13.2 What happens when the threshold is changed

The table below is a hypothetical evaluation result that illustrates the mechanism. It does not represent actual Jev measurements.

As you move down the table, the error rate of the automatically classified portion decreases, but the final row still requires separate handling of 900 cases. While the 3 % error rate of the automatic classification looks good, it may not actually reduce the overall workload much. We want to evaluate performance taking human processing capacity into account.

Moreover, raising the threshold does not always guarantee a lower error rate. If the model is strongly confident about a particular type of error, that error will still concentrate in the set of cases retained at higher thresholds. It is necessary to assess whether ordering cases by confidence can serve as a proxy for correctness.

13.3 Measure up to the point after handing to humans

It is important to consider not only the automated portion but also the processing that follows deferral. If the cases handed over to humans are predominantly difficult ones, verification will take longer than before. Moreover, when the model presents an incorrect classification, humans may be swayed by that classification. Simply inserting a human step does not mean the error disappears.

In practice, we want to measure the time required for verification, the final error rate, the time taken to respond to the end user, the reasons for deferral, and so on. It is also possible to conduct a comparison where the model’s automatic classification is hidden during verification. The classification of which department should handle the verification, and the quality of that verification, are separate evaluation targets.

To use deferral appropriately, a mechanism for handling the deferred cases is required. Without processes for acquiring additional information, a user interface for the responsible person, questions to the user, or a reminder after a set period, deferral merely results in an increasing number of unprocessed cases. The value of using a model that can output probabilities should be considered together with the design of this downstream handling.

14. What can be learned from studies that evaluated Jev itself

14.1 Separating the model’s decision‑making ability from its calibration

As a publicly available study that deals with Jev, there is a preprint dated September 24, 2026 Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero‑Shot Detector of AI Alignment Failures . Here we are referring to version 1 of the paper, not results obtained by reproducing the experiments ourselves. We should not treat the preprint’s findings as definitive facts established by peer review.

The target is jev‑1.13.0, a benchmark suite that deals with English AI alignment failures. It comprises 31 benchmarks that can be used with standard Noul questions, reporting a median AUROC of 0.886. In contrast, the aggregated set has an ECE of 0.047, while the median ECE across individual benchmarks is 0.168. When comparing with finite outcomes generated from perfectly calibrated probabilities, the median ECE is 0.074, and 24 of the 31 benchmarks exceed the 95th percentile of that comparison.

What we want to infer is that the ranking based on probabilities and the usability of those probabilities for each individual case are distinct. An overall aggregated evaluation can overlook discrepancies in the probabilities for specific cases. However, these results cannot be directly transferred to Japanese inquiry classification, because the model version, the questions, the inputs, and the definition of the correct answer differ.

14.2 A high AUROC does not mean you can simply split at 0.5

AUROC is a metric that summarizes the model’s discrimination performance across all possible thresholds. It can be understood as the model’s ability to rank positive instances higher than negative ones. Although a formal definition that includes handling of ties exists, for beginners it is helpful to think of it as measuring how good the ranking is.

For example, if a model returns 0.4 for positive instances and 0.2 for negative instances, the ordering is correct. However, if you split at 0.5, all instances would be classified as negative. A good ranking alone does not guarantee that the predicted probabilities match the actual frequencies, nor that 0.5 is an appropriate cutoff for the intended use.

This example does not mean that the threshold should always be set low. The appropriate threshold changes with the positive rate in production and the cost of errors. Choose the threshold on evaluation data and then validate it on a separate dataset. Skipping this step and determining automation criteria solely from favorable numbers on public benchmarks is not advisable.

The interesting aspect of the research is that it shows not only Jev’s characteristics but also the conditions under which it can be used. Rather than interpreting it as “because the model is intended for calibration, evaluation can be omitted,” we should read it as “since the model provides probabilities, we can separately measure discrimination, calibration, and the conditions for taking action,” which leads to practical implementation.

14.3 Treating ground‑truth labels as observable data

From here on, the discussion shifts from the results of a specific paper to the issues that arise when applying evaluation to one’s own use case. In natural‑language judgment tasks, the correct label may not be uniquely determined. For example, in a classification performed by a department, a question about shipping costs might be routed to the billing team in one organizational workflow and to the logistics team in another.

When multiple people assign different labels, it is not appropriate to treat all of them as model errors. First, determine whether the classification rule is ambiguous, the documentation is insufficient, or the annotators missed something. Expecting a model to achieve 100 % agreement with humans on tasks where human consensus is low is also unrealistic.

That said, it is also not correct to simply discard all cases where human labels are split from the evaluation. Such cases will appear in operational use. It is necessary to define the criteria for whether ambiguous cases should be deferred, whether additional documentation should be requested, or whether departmental priority rules should dictate the handling. The work of creating ground‑truth data is also a process of clarifying the business definitions.

15. What to measure when evaluating for your own use case

15.1 Fix the question before creating the data

Before starting evaluation, fix the question you will ask the model. If you initially labeled cases as “related to billing” and later change the question to “should be routed to the billing department,” the meaning of the correct label changes even for the same data. The question and the definition of the label should be version‑controlled together.

Also fix the way the input state is constructed. Whether the input consists only of the text, includes order information, or incorporates the responsible person’s notes changes the nature of the task. If you add the responsible person’s notes—written after the ground truth has been assigned—to the input, the answer may inadvertently leak into the documentation. When replaying past cases, restrict the information to that which was available at the time of decision.

For illustration, suppose we retain the following items in each evaluation record. The actual storage format can be adapted to the system being used.

Case ID
State available at the time of decision
Question and candidate version
Correct label and its definition
Model identifier
Returned probability distribution
Action chosen by the downstream component
Response time and type of failure

It is not acceptable to store any text that contains personal information indiscriminately. One must separate the information needed for reproducibility from the information that may be stored. Anonymization, access controls, and retention periods are also required. For documents that will be sent to external APIs, their handling should be verified in advance.

15.2 Separate data for tuning from data for final evaluation

If you keep looking at the same data while changing the phrasing of the questions, altering the candidate explanations, and adjusting the thresholds, you are essentially selecting the settings that best fit that data. Even without changing the model’s weights, you are still tuning the system based on the evaluation results.

At a minimum, you should separate the set used for adjusting questions and thresholds from the set used for final performance verification. Be careful even when nearly identical inquiries from the same customer end up in both sets. Simply splitting cases randomly one by one can still cause paraphrases of the same incident to appear in both groups.

If you are handling future cases in production, you could also consider a split where you tune the model on data from an earlier period and verify its performance on a later period. This approach evaluates changes over time, including shifts in language and workflow. However, if the underlying rules themselves changed during the period, those changes must also be recorded.

flowchart TD
    A["Gather cases for evaluation"] --> B["Check for duplicates of the same event or user"]
    B --> C["Set for tuning"]
    B --> D["Set for final evaluation"]
    C --> E["Adjust questions, candidates, and thresholds"]
    E --> F["Lock the settings"]
    F --> G["Perform the final evaluation once"]
    D --> G
    G --> H["Continue verification with new operational data"]

15.3 Look at the Types of Errors, Not Just Accuracy

When classifying inquiries, examine the confusion matrix in addition to the overall accuracy. This reveals which categories are mistaken for others. Confusing billing with shipping, or misclassifying a serious issue as “other,” requires different corrective actions.

For a binary classification, you can use precision and recall. Precision is the proportion of cases treated as positive that are actually positive. Recall is the proportion of actual positives that were correctly identified as positive. Raising recall for the tasks handed to the verification staff can reduce missed items, but if precision drops, unnecessary verifications increase.

The F1 score is one way to combine the two, but it does not directly represent business loss. Whether to treat safety oversights and unnecessary verifications as equally weighted is a separate consideration. Distinguish between metrics used for model comparison and those that determine actual actions.

For probabilities, examine the Brier score, log loss, and the calibration table broken down by interval. Additionally, adjust the automatic‑processing threshold and check the coverage and errors in the processing portion. Although this generates many numbers, it is easier to explain which aspects are strong and where problems lie than to collapse everything into a single figure.

15.4 Do Not Claim “Almost Never Wrong” Based on a Few Cases

Even if you process 100 cases in an evaluation and make no mistakes, that does not confirm an error rate of zero. The sample may lack rare inputs, or it may simply have been lucky. In particular, if you select only cases with very high confidence, the number of such cases will be small.

As a simple rule of thumb, consider performing n independent trials that all have the same error rate r and observe zero errors. The probability that all n trials are correct, given an error rate r, is:

Probability of zero errors = (1 − r)ⁿ

Setting this to 0.05 yields a one‑sided 95 % upper bound for r of 1 - 0.05^(1/n). When n is sufficiently large, this bound is roughly 3 / n, so even with zero errors out of 100 cases the upper‑bound estimate is about 3 %. This calculation assumes a binomial model, and the actual inquiries may not be independent. At the very least, you cannot claim that “because there were no errors in 100 cases, the error rate is below 0.1 %.”

Even if you collect hundreds of inquiries that follow the same pattern, the sample size grows, but it does not test rare expressions or missing documentation. It is better to keep separate sets: one that measures the typical distribution and another that targets conditions likely to cause failures. While the latter cannot be used to estimate the normal occurrence rate, it helps uncover how the system breaks.

15.5 Measure API Failures Separately from Model Errors

An incorrect answer and the absence of an answer from the API are distinct issues. Treating timeouts, authentication errors, rate limits, communication failures, and unexpected responses as “negative” outcomes mixes operational disruptions into the distribution of normal predictions.

If the refund‑request detection API fails and you record it as “no refund request,” both the business classification and the evaluation data become corrupted. Failures should be logged as failures, and you must decide whether to retry, defer, or revert to an alternative method. It is crucial not to conflate a genuine “no” prediction with a situation where no answer exists.

When looking at response times, consider not only the median but also the slower tail. Even if most queries are fast, if some take a long time, you need to provide appropriate UI feedback or hold processing during that period. Costs should include not only API calls but also human verification and handling of misclassifications. The small cost incurred by an individual model does not automatically translate into a small overall system cost.

16. Small Architecture for Routing Inquiries

16.1 First, Complete Processing That Does Not Involve a Model

Let’s apply the discussion so far to the task of routing inquiries. When an input is received, first verify authentication and permissions. If an order ID is present, validate its format and retrieve the order information within the allowed scope. These steps are not decisions to be made by Jev.

Next, construct a state from the message body and the already‑verified information. Separate the customer’s claim, the facts you have obtained, and the items that have not yet been confirmed. With that state, ask about the responsible department and whether a refund request is present. If urgency is also needed, add a separate question for it, but there is no need to ask the model about every possible aspect right from the start.

flowchart TD
    A["Receive an inquiry"] --> B["Verify authentication, permissions, and input format"]
    B --> C["Retrieve required order information"]
    C --> D["Create a state by separating claims, facts, and unverified items"]
    D --> E["Ask Jev about the responsible area and intent"]
    E --> F{"Whether the API responded successfully"}
    F -->|No| G["Record as a failure and put on hold"]
    F -->|Yes| H["Evaluate the distribution and operational rules"]
    H --> I["Routing, additional verification, and hold"]

16.2 A Request Consists of a State and a Set of Questions

In the format published by the API reference, you send the model, state, and questions to /v1/systemone. Below is an example of the raw request body I created for inquiry classification. The model identifier must be replaced with an available identifier, and this example does not represent an actual execution result.

API reference

{
  "model": "<model identifier>",
  "state": {
    "customer_message": "It looks like my card was charged twice. Can one charge be refunded?",
    "payment_check": "not_checked",
    "order_status": "shipped"
  },
  "questions": {
    "department": {
      "type": "choice",
      "instructions": "Select the team responsible for the main issue in the message. Do not determine whether a duplicate charge actually occurred.",
      "criteria": {
        "billing": "Questions about pricing, payment, billing, or refunds",
        "delivery": "Questions about shipping, delivery, or receipt",
        "product": "Questions about product specifications or usage",
        "other": "None of the above, or the main issue cannot be identified"
      }
    },
    "refund_intent": {
      "type": "noul",
      "instructions": "Does the message express an intent to request money back? Do not determine refund eligibility."
    }
  }
}

The purpose of this example is not to reach a conclusion that a refund is possible. It separates treating the inquiry as a billing consultation from detecting an intent to request a refund. The fact that payment verification has not yet been performed becomes a separate state that halts further processing.

The question keys are used by downstream code to identify which result corresponds to which question. By distinguishing keys and types so that the classification output is not treated as a refund approval, you can reduce implementation confusion. However, simply giving good names does not itself constitute a security measure; the execution side must still verify permissions and state.

16.3 Do Not Execute Immediately After Receiving the Output

First, inspect the API’s output for its type and content. Verify that answers to the required questions are present, that the candidates belong to the allowed set, that the probabilities are finite values, and that the distribution matches expectations. Even if the service provides typed responses, it remains necessary to check these at the consumer’s boundary to guard against communication issues or version changes.

Next, apply the operational criteria determined during evaluation. If the responsible department is ambiguous, route the case for additional verification. Even when the case clearly belongs to the billing domain, the refund itself should be processed only after confirming payment information. At this stage, it is crucial not to convert the decision type directly into an execution permission.

If you place a generative LLM downstream, you can limit its role to drafting guidance text. Instead of letting it automatically write something like “It has been confirmed that this is a duplicate charge,” include the verification status—i.e., that confirmation is still pending—in the output. When combining two models, avoid promoting the upstream hypothesis to a fact in the downstream stage.

16.4 Audits Require More Than Fluent Explanations

It’s not necessary to equate Jev’s lack of free‑form explanations with an inability to conduct audits. You can record which documents were fed to which questions, what distributions were returned, and what rules were used to select actions. While this does not guarantee that the exact same values can be reproduced by the model later, preserving the inputs and outputs at each point provides material for traceability.

On the other hand, numbers alone do not let you directly see which aspects of the description the model emphasized. Even if you ask another LLM for an explanation, you cannot treat that explanation as a faithful reconstruction of Jev’s internal calculations. Explanation texts can aid in interpreting the input or serve as material for human review, but they do not constitute an internal proof of the decision model.

In audits, the source of the facts, the definition of the questions, the rules applied, and the actual outcomes are all critical. If a refund for a duplicate charge was issued despite having only a single payment record, even a fluent justification does not resolve the problem. First, you need to trace at which stage unverified information was treated as definitive.

17. Tasks That Should Not Be Left to Jev

17.1 Do Not Turn Precise Calculations or State Management into Decision Problems

The total billing amount, date differences, ownership of an order, and whether a refund has already been issued should be determined from data and code whenever possible. Just because the model can understand natural language does not mean you need to express every calculation as a textual question.

For example, if the refund eligibility period is defined as 30 days from purchase and the purchase timestamp is recorded, you can make the determination using that date and the appropriate time zone. If the information is only given as “about a month ago,” you must first obtain the exact date. Filling that gap with the model’s confidence can let through conditions that have not been truly verified.

Jev returns numeric values, so it may look like a calculator. However, the fact that the output is numeric does not guarantee strict arithmetic correctness for the numbers present in the input. A score is also a value that aggregates the probabilities assigned to each class, not a representation of the actual billing amount calculation.

17.2 Also Verify Known Weaknesses That Are Publicly Documented

TypeSafe AI explains, in the Jev 1.13 jaggedness documentation, the strengths and weaknesses of that version. It does not automatically guarantee precise calculations, complex multi‑step reasoning, the omission of unnecessary information, or logical consistency across questions. Also, in the “state” documentation, it is noted that it is primarily trained on English and cautions are provided regarding its accuracy for CJK languages.

Jev 1.13 jaggedness

State documentation

This explanation should be read within the scope of the version and materials examined. It does not assert that all future Jev versions will have the same constraints, nor does it mean that it cannot be used uniformly in Japanese. You need to assess how it handles Japanese-specific elements such as omissions, honorifics, indirect requests, and industry terminology.

For example, not everyone will write “I want a refund” directly. Some may say, “Is there any way to resolve this invoice?” or “Can one of the items be cancelled?” There are also formulations that contain negations, and references to prior conversations. If you exclude such expressions from the evaluation, you will only confirm good results on simple examples.

17.3 Search and Decision Are Separate Stages

When you need to locate potentially relevant documents from a large corpus, you can narrow down candidates using full‑text search or an embedding model. While it is formally possible to feed all documents to Jev and ask for a relevance score, you must consider the communication overhead, cost, and any limits on the material.

Search is the task of “finding candidates,” while decision is the task of “evaluating whether those candidates meet the required conditions.” If a candidate is filtered out during search, the downstream decision model will never see that document. Even a highly capable Jev cannot make use of evidence that was not retrieved. Therefore, the quality of search recall and the quality of the decision must be evaluated separately.

Moreover, the closeness of embeddings does not equate to the probability that a question will be answered affirmatively. Even if a clause containing similar wording is found, the product or contract period it refers to may differ. Moving from “similar” to “applicable” still requires confirming the necessary conditions. Conversely, when the document ID or contract version is already known, it may be more reliable to retrieve the document directly rather than relying on semantic search.

18. How to Allocate Responsibilities Among LLM, Jev, Search, and Rules

18.1 Do Not Process Everything with a Single Technology

Consider at each stage of the system what remains uncertain. If you don’t know which documents to look for, perform a search. If you have the documents but need semantic judgment about the conditions, consider using Jev. If you need to generate text, use an LLM. If the format and values are fixed and can be computed by rules, handle it with code.

Jev does not replace every item in this table. It is an option to isolate the component that extracts judgments from natural‑language input, separate from the component that generates text. If you can train a classifier yourself, that is also a comparable alternative. When there are sufficient labels and the target domain is narrow and fixed, existing machine‑learning models or rule‑based systems can often achieve the required performance.

18.2 Sometimes Start with a Model and Later Replace It with Rules

In the early stages of development, you may not know the phrasing of inquiries or the categories that should be defined. Using an LLM or Jev at that point lets you organize unknown inputs while shaping the specifications. However, even after deployment has gathered examples and the conditions have become clear, there is no need to continue processing everything with the same model.

For example, with fixed inquiries that come from a specific form, the type of form may determine the responsible department. Once the pricing rules are clear, many refund conditions can be moved into code. By passing only the remaining free‑text portions to the model, you can reduce not only costs and latency but also the number of places where errors can occur.

This substitution is not a failure of the model. It means that a task that was initially vague has become a decisive one through observation and specification. It is natural to revisit the technologies used, aligning them with the uncertainty that remains at each stage.

18.3 Decide What to Verify Before Stacking Multiple Models

Designing that a second LLM will simply confirm Jev’s judgments does not work out of the box. If both models read the same insufficient documents, they may err in the same direction. Moreover, showing one model’s answer to the other can cause the latter to be biased toward that answer.

If you use a different model, you should define what it is supposed to verify anew. Whether it re‑examines the meaning of candidates, looks for missed evidence, or checks for mismatches between the question and the state, its role differs. When definitive evidence is available, it may be better to retrieve that evidence rather than stacking models.

When multiple models agree, that consensus can be used as a single observation. However, converting the number of agreeing models directly into a correctness probability requires evaluating that specific combination. The issue that errors across models are not necessarily independent remains here as well.

19. Small Checks Before Using Jev

Rather than entrusting production actions to Jev from the start, you can record its predictions alongside existing processing. By letting the system make judgments without showing them to the user and then comparing those judgments with the final, confirmed outcomes, you can observe the model’s behavior without it influencing human decisions. At this stage, erroneous outputs do not lead directly to execution.

The initial tasks you select should have relatively clear definitions of input and answer—such as classified inquiries where the candidate categories are already organized. Conversely, processes that have significant impacts—such as contractual approvals, medical decisions, or actions that affect users’ rights—should not be automated solely based on a few successful examples.

You should also establish comparison baselines: a method that always selects the majority, simple keyword rules, existing classifiers, or LLMs that output structured data. Measuring only Jev makes it unclear what exactly has improved. Even if the overall accuracy is high, it may simply reflect that many cases are inherently easy.

Next, aggregate the probabilities and observe the errors. Examine the cases that were misclassified despite high confidence scores, and categorize the failures as missing input, duplicate candidates, ambiguous questions, language differences, and so on. If you modify the question, re‑evaluate using new data. Do not report only the final numbers from the same dataset after repeated evaluations.

Even when moving to actual automation, start with a reversible process. For example, present candidate responsible parties or adjust the prioritization of search results, and observe the impact. For actions where a mistake would cause large losses, retain an additional verification step. By recording the relationship between the model’s output and the executed result, you can trace back to investigate any problems that arise.

When updating the model or the questions, verify regressions using a fixed set of representative examples. However, relying solely on fixed examples will not uncover new issues. Incorporate recent operational data into the evaluation while respecting the conditions for data retention. Even after a model is deployed, the items it is required to assess will continue to evolve.

20. Summary: Separating Judgment from Text Reveals the Design

Jev is a model that, instead of generating free‑form text from natural‑language input, returns judgments and probabilities for predefined questions. Noul handles binary choices, Choice deals with candidate options, and Score manages ordered, graded levels. The public documentation explains that its purpose is parallel evaluation and calibrated judgments using RLCD. However, the exact internal architecture and the reward function used during training have not been fully disclosed.

When using an LLM for judgment, issues can arise concerning the format of the answer, the latency of text generation, the distinction between explanations and evidence, and the interpretation of probabilities. Jev provides an option to isolate this task into input‑output specifically suited for judgment. However, having consistent formatting does not guarantee semantic correctness, and a concentrated distribution does not ensure guaranteed accuracy.

If you rely on probabilities, accuracy alone is insufficient. Verify on the target data whether the predicted probabilities match the actual frequencies. Even when automatically handling cases with high confidence scores, measure both the proportion of cases taken on and the error rate within that subset. The public research on Jev also needed to be read by separating the evaluation of its discriminative ability from the calibration performed on individual instances.

Furthermore, the step from probability to action still requires human decisions. You must determine what counts as a correct answer, which evidence is required, how to handle missed detections versus misprocessing, what should be left pending, and which actions need confirmation. Authentication, authorization, and business invariants cannot be omitted simply because the model is confident.

What I find interesting about Jev isn’t just that it returns a brief JSON; it’s the ability to treat text generation and the output of judgments as separate components. You can reconsider, as parts of the system, what you’re asking, what probabilities you receive, and how you’ll use those results. Thus, you leave the ambiguous aspects to the model while returning the certain aspects to the code. Deciding this division of labor is the user’s responsibility.

References

The links in the text point directly to the sources that support each explanation. To follow the material most effectively, first review the official input‑output specifications, then study the fundamentals of proofreading and probabilistic prediction, and finally examine the evaluation research focused on Jev.

• TypeSafe AI: Introduction : Overview of Jev and the decision model.
• TypeSafe AI: System One : Official explanation of the difference between generation and decision.
• TypeSafe AI: Noul , Choice , Score : Definitions of output formats.
• TypeSafe AI: Confidence : Relationship between distributions and confidence.
• TypeSafe AI: AI primer : Positioning of RLCD.
• Guo et al., On Calibration of Modern Neural Networks, ICML 2017 : Distinction between classification performance and calibration.
• Gneiting and Raftery, Strictly Proper Scoring Rules, Prediction, and Estimation, 2007 : Theory for scoring probabilistic forecasts.
• Geifman and El-Yaniv, Selective Classification for Deep Neural Networks, NeurIPS 2017 : Classification with abstention and risk‑coverage.
• Guo et al., Just Ask Jev, arXiv:2609.29429v1, 2026 : Preprint evaluating Jev for decision‑making purposes.

The references were verified on October 1, 2026. The example queries, probability tables, and loss calculations in this article were created for explanatory purposes, except for results explicitly noted as research findings, and they do not represent actual measurements obtained via the Jev API.