Added evaluation JSONL and production‑ready features to Jev MCP.

I updated the jev-mcp .

In the previous post, I wrote that we had created an MCP server that invokes Jev, TypeSafe AI’s decision model, from Codex. Simply connecting the four decision tools doesn’t tell us whether they can be reliably used in actual coding work.

This time, we added a small JSONL file for evaluating the decision results, an evaluation CLI, telemetry for monitoring invocation status, and handling for failures. We have also expanded the Codex‑specific Skill with additional decision recipes tailored to different use cases.

In the previous article “Created an MCP server that extracts Codex decisions into Jev” (related article), we described the four decision tools of Jev MCP and their basic usage.

Evaluation data is also required when using a decision model

The inputs given to Jev are small decisions, such as “Is this test failure related to the change?”, “Which part should be investigated next?”, or “How risky is this change?”.

Just making a single call and receiving a plausible answer doesn’t tell you what changed when you modify the model, prompt, or the MCP‑side transformation. Therefore, we first created a small evaluation set for regression testing.

eval/ja-coding-minimal.jsonl contains twelve fictional Japanese coding‑decision examples.

We included not only cases where a decision can be made unambiguously, but also situations where information is insufficient, where candidates conflict, and where unknown should be selected. In Choice questions there are cases where a single correct answer cannot be fixed, so we treat the set of acceptable candidates as the expected value.

This is not a benchmark for measuring overall performance of Japanese coding decisions. Because it consists of only a few fictional cases, it serves as a starting point for detecting obvious regressions and reviewing failure examples.

Inspecting the evaluation JSONL without calling the API

The evaluation file can first be checked for format locally.

cargo run --locked -- eval validate eval/ja-coding-minimal.jsonl

eval validate does not require an API key. It reads each line of the JSONL, checking for duplicate IDs, empty lines, mismatches between question and expected‑value types, numeric ranges for Noul and Score, and whether the candidates expected in a Choice are present among the actual options.

We have made it possible to eliminate errors in the evaluation data itself before sending it to the model. In testing, we confirmed that valid files pass while invalid files fail without an API call.

Preserve real model results in JSONL

When actually invoking Jev, a separate subcommand is used.

TYPESAFE_API_KEY="your_typesafe_api_key" \
  cargo run --locked -- eval run eval/ja-coding-minimal.jsonl \
  > eval/baselines/ja-coding-YYYY-MM-DD.jsonl

This route calls the real API, so usage fees apply. For each case, we output pass, fail, error, measurement values, the resolved model name, execution time, and the number of input/output tokens in JSONL format. At the end, we add a single summary line containing the total count, the numbers of successes, failures, and errors, the average time, and the total token count.

Even if a single request fails, the remaining cases continue to run. An API error that occurs during evaluation and a deviation from the expected value are distinct, so we do not treat both as the same “fail.”

We have not yet added baseline results measured with the real API to the repository. First, we will fix the same fixtures, save the outcomes, and set up a state where we can compare them when we change the model name or the Skill’s instructions.

Invocation statistics that do not pollute MCP’s stdout

Since MCP communicates via stdio, emitting debug logs to standard output would break the protocol. Nevertheless, because we also call other APIs along the way, we want to monitor latency, retries, and token usage.

Therefore, when JEV_TELEMETRY=1 is enabled, we output a single JSON line to standard error for each tool invocation.

We record only the tool name, the number of questions, execution time, number of attempts, the requested/resolved model, input and output token counts, and whether the result was a success or an error type. state, instructions, criteria, probability distributions, and the API key are not recorded.

The diffs and logs sent to Jev may contain code or failure information. To avoid duplicating the same content locally just for logging, we limit the statistics to those that do not include the input data.

Do not treat failures as answers

If an external API failure is interpreted as “no answer” or “low score,” it mixes communication errors with actual decision results. MCP tool errors are returned as structured errors with types such as validation, authentication, rate_limit, timeout, network, http, and invalid_response.

For example, if input validation fails, we do not forward the request to the API. Authentication errors, overload conditions such as 429 or 529, timeouts, and network failures are also distinguished. The body of an HTTP error may contain the sent state or authentication information, so we do not include it directly in the MCP result.

Retries are also limited. For 429 and 529 errors we retry up to two times, but the overall limit—including request time and waiting—is set to 60 seconds. This distinction also prevents Codex from endlessly repeating calls that fail with the same input.

Added more decision recipes to the Skill

The bundled jev-decisions Skill now includes a reference that organizes which decisions to use Jev for, according to different use cases.

• Prioritizing failure hypotheses and determining the next diagnostic point
• Assessing the sufficiency of evidence concerning causes, reproducibility, regressions, and impacts
• Checking for stalls or repetitions from execution history
• Selecting observations to retain from a long work history
• Advising on intent, target, impact, and outcome of an operation

The Skill does not execute these automatically. It reads only the recipes needed for the current task and passes to Jev the evidence gathered by Codex together with the questions that can alter the next action.

Independent judgments on the same evidence are consolidated in jev.batch. Conversely, when a previous answer indicates that additional investigation is required, we first verify the result and then create the next state. This distinction remains unchanged from the previous version.

The Skill’s main text and reference are in English, but there is no need to unify the requests to the Skill or the materials passed to Jev into English. The evaluation fixtures this time remain in Japanese.

Scope that can be verified through testing

In the current implementation, we use a mock HTTP server to verify input validation, mismatched response types, overly large responses, retries, timeouts, structured errors, and telemetry. The JSONL validation performed by the evaluation CLI, the twelve‑case composition of the Japanese fixtures, and the JSON examples in the Skill reference are also included in the test suite.

Running cargo test --locked --offline locally, 17 unit tests and 3 integration tests passed. This confirms the communication format and the local evaluation processing, but it does not measure the quality of the actual decisions made by Jev.

The results from the real model need to be saved separately as a baseline after confirming the fees and usage terms. At this stage, we have built not only the pathway to invoke decisions but also the ability to later trace what was expected and what failed.

Whether to use Jev or which result to adopt remains a decision on Codex’s side. Even if the model returns a high‑probability answer, it does not replace definitive cause identification or testing. While preserving this premise, we will gradually expand the situations where extracting decisions is worthwhile.