How to Choose Between LLM, Embedding Models, Jev, and Classical NLP

I often see people treating search, classification, comparison, and judgment as tasks that can simply be handed off to an LLM. If you ask it in natural language, you get a response, and you can keep asking follow‑up questions directly. That convenience is indeed useful, and I’m prone to entrusting everything to it. As a result, it’s easy to develop an impression that an LLM can solve any problem.

On the other hand, even when using a judgment model like Jev, you can still give it tasks that aren’t suited for it. Asking it to hunt for information before the necessary data has been gathered, or to have it formulating an overall strategy before any candidates have been identified, is asking a limited‑purpose decision‑making tool to perform investigative work as well. If you only need to locate known terms, you can simply perform a string search without invoking a model at all.

The fact that “if you ask, you get something back” is not the same as having the right tool for the job. If you need to gather similar documents, an embedding model is also a candidate, but similarity alone does not determine a document’s correctness. You should clarify what you want to entrust to an LLM, Jev, an embedding model, or classic search and text‑processing methods. This isn’t about comparing accuracy across products; it’s first about matching inputs/outputs to the type of task.

Separating Search from Judgment

Below, I’ll consider examples from the development work I frequently handle. Suppose a test fails and you want to “find the code that’s related.” At this stage, the answer is a list of candidate files or lines. In contrast, the question “Is this failure caused by the current diff?” requires a judgment that compares the diff with the failure logs. “Investigate the cause and fix it” involves additional work such as gathering more information and running experiments.

Even when you input similar sentences, the desired output differs for each case.

This table does not indicate the limits of each capability. You can also have an LLM perform classification, or build classifiers that use embeddings as features. However, the distinction is that you don’t need to reinterpret raw similarity scores as probabilities for judgment, nor invoke an LLM every time you perform a simple search.

Splitting the Work Based on the Desired Answer

For example, suppose an API test times out after a change. At first you might want to locate the relevant code using the test name or error message. Once the logs and diff are available, you’ll want to judge whether this change is the cause, and if the cause remains unclear, further investigation becomes necessary. Even for the same failure, the work changes depending on what answer you need at that moment.

In my approach, I don’t choose a tool by its name first; instead, I first ask, “What answer do I want?” and “Do I already have the material needed to produce that answer?” The steps below illustrate this flow, but they are not a procedure that requires using all four methods in order.

flowchart LR
    START{"Is this the stage of looking for related documents or code?"}
    START -->|Yes| FIND{"Can it be searched using keywords or conditions?"}
    FIND -->|Yes| SEARCH["String search, rules, BM25<br/>available locally: error name, function name, search criteria"]
    FIND -->|No| EMBED["Embedding and keyword search<br/>available locally: description of the symptom, set of documents to search"]
    START -->|No| RESOLVED{"If the locally available information is read, can the answer be determined?"}
    RESOLVED -->|Yes| DONE["End here"]
    RESOLVED -->|No| JUDGE{"With the evidence gathered, can the question be closed?"}
    JUDGE -->|Yes| JEV["Limited judgment by Jev<br/>available locally: logs, diffs, decision options"]
    JUDGE -->|No| LLM["Investigate and implement using LLM and tools<br/>available locally: problem and additional investigation methods"]

First, determine whether what you need is the discovery of candidates or a judgment/implementation based on the evidence you already have. If you’re looking for candidates and you have known terms such as an error message, function name, or test name, use a string search. If you need to rank a large collection of documents by keyword relevance, BM25 is also a viable option. If the answer can be confirmed by reading the locations found, you can stop there.

If the candidates are insufficient and you need to locate documents that describe the same issue in different wording, consider using an embedding model. However, you don’t have to rely solely on embeddings to find proper nouns or code identifiers; it’s natural to combine keyword search with embedding search and treat the retrieved similar documents as “candidates to verify.”

When you have both candidates and supporting evidence, and you can frame questions such as “Is this diff related to the failure?” or “Which of A, B, or unknown should be investigated next?” you can hand them over to Jev. Giving Jev a choice when the necessary material is missing won’t fill in the gaps; any missing information remains unresolved. Even after Jev provides an answer, you should still verify it with code or tests.

When even the cause candidates are unclear, additional observations are needed, or you want to proceed with fixes and regression testing, it’s time to bring in an LLM‑driven agent. If an identifier becomes apparent during the investigation, you revert to a search; once you have gathered the candidates you want to compare, you can pass a small question to Jev as needed. Choosing an LLM does not mean you abandon other methods.

Also, if you’re sending logs or code to an external API, you must verify separately—outside the tool‑selection flow—whether it’s acceptable to transmit that data. Even if the answer format is correct, the configuration cannot be used if it includes information that cannot be sent.

Start with Classical Text Processing and Information Retrieval

If you already know the error name, test name, or function name, you can begin with a simple search such as rg or a syntax‑based query. Questions like “Where does TimeoutError appear?” or “Which part of the code uses a particular API name?” can be answered by directly searching the code. Whether FAILED appears in the logs is not a matter for a model to decide.

When dealing with large volumes of documents, ranking methods such as BM25— which take into account term frequency and document length—are useful. The Apache Lucene implementation of BM25Similarity (source) is an example of such a system. The scores produced here are for ranking search results, not for the probability that a particular document is the cause.

For Japanese text, you can combine morphological analysis, normalization, and synonym dictionaries. However, if you are familiar with the naming conventions or log format of the target system, using that structure first makes results more reproducible. Conversely, when the same concept appears in English in the source code and in Japanese in an issue report, simple word matching alone will make it hard to retrieve them.

Here, “classical NLP” refers broadly to search and rule‑based text processing. I’m not trying to treat all of these as a single unified method. What they have in common is that if a problem can be defined mechanically, you should first apply the tool that best fits it.

Embedding Models Find Candidates

An embedding model converts text (and other data) into vectors. By encoding both the query and the documents with the same model, you can locate the most similar items using distance or cosine similarity. The OpenAI Embeddings guide (documentation) includes an example where code documentation and natural‑language queries are embedded, and functions are ranked by similarity.

For example, a log might say “the operation timed out while waiting for a response,” while past issues mention “external API latency” or “long retry waits.” Even without exact word matches, this can serve as an entry point to uncover related candidates. When you need to pull a few items out of a large document collection, you can treat search and judgment as separate steps.

However, finding a document that is textually similar does not mean it is the cause of the current failure. Even if two issues describe the same subsystem and thus have a high similarity score, a fix for one does not necessarily cause the test failure of the other. You cannot interpret a cosine similarity of 0.8 as “an 80 % probability that it is the cause.”

You also need to design how you split documents, when you re‑index updated code, and how to avoid missing items. Because proper nouns or code identifiers are often more reliably retrieved by keyword search, it’s natural to combine keyword search with embedding search. If there are only a few candidates and you already know the file names, preparing a vector search may be unnecessary.

Jev Evaluates the Provided Evidence in a Predefined Format

TypeSafe AI’s Jev (source) is a judgment model that takes state and typed questions as input and returns structured answers. In my jev‑mcp (source), I expose noul, which returns the probability of “yes” for a given condition; choice, which selects one item from a list of candidates; score, which provides a graded assessment; and batch, which aggregates independent questions about the same material.

The crucial point here is that Jev does not search the repository itself. The caller must gather the diff, failure logs, and the meaning of the candidates, and pass them to state. Jev then evaluates the predefined questions against that material.

For example, suppose you have examined the diff and the test failure, but the relationship is still ambiguous and you want to decide where to investigate next. You can ask “Is it related?” using a Noul question and “Which candidate should be examined first?” using a Choice question. However, when using Choice, you should also include unknown among the candidates and avoid forcing a cause selection when the evidence is insufficient. If you need to evaluate multiple independent questions with the same state, you can do so in Batch mode. Since the answer to one question cannot serve as a premise for the next, any additional investigation requires creating a new state.

The confidence values for Choice and Score are calculated from the concentration of the answer distribution, as described in the TypeSafe documentation (source). 0.9 is not a metric that has been validated to achieve 90 % accuracy on this repository. Noul does not have a separate confidence. The policies such as accept, verify, and reevaluate returned by jev‑mcp are also merely auxiliary information for how to handle the answers; they do not grant permission to skip testing.

Both the search score and Jev’s probability distribution are numeric values, but they serve different purposes. The former is a metric used to rank candidates, while the latter represents the model’s response to a given state and question. Even Jev’s answer does not constitute a definitive determination of the cause. If the evidence that influences the decision is missing from state, even a properly formatted answer can lead to an incorrect judgment.

Let the LLM Handle Problem Decomposition and Execution

If you only need to locate candidate causes, a simple search might suffice. When the candidates are fully enumerated and the choice set is closed, Jev can produce a single decision. However, in real bug investigations you must examine log meanings, read code, formulate hypotheses, vary reproduction conditions, and run tests. Sometimes the candidates themselves are still unknown.

This is where a coding agent that uses an LLM comes in. It isn’t just about generating explanatory text; it involves fetching the necessary documents, running tools, and re‑evaluating based on the results. Since an LLM can be instructed to return classifications in JSON, it isn’t the case that “classification is impossible for an LLM.” However, we want to separate the role of advancing the overall investigation from the role of providing a definitive answer to a single closed question.

Even if candidate A ranks highest in Jev’s results, the LLM will still read A’s code, verify the diff, and confirm it with tests. When candidates are closely matched, rather than repeatedly asking the same question, we should consider what observations can distinguish them. We do not intend to automate this part solely based on Jev’s scores.

Considering a Single Investigation

Suppose the issue is “the API test timed out after a change.” In my approach, I would consider it in the following order.

The test name and TimeoutError are known
  → Find the failed tests in rg, the timeout settings, and the changed locations

There are many past issues, and other expressions such as “response delay” also exist
  → If needed, expand candidate related documents using BM25 or embedding

Even when collecting the modified retry settings and failure logs, the relevance is subtle
  → If necessary, ask Jev whether this change is related to failures

Reproduce the timeout conditions to verify the cause and apply fixes as needed
  → Proceed with an agent that uses an LLM and the actual tools

It isn’t necessary to run all four steps every time. If the test shows an expected value of 3 but an actual value of 1, and the diff indicates that the retry limit was changed from 3 to 0, then you should first inspect the code and the test. At that point, formulating a question to ask Jev would only be a detour. Conversely, when there are many candidates and the terminology in the explanations is fragmented, you’d want to insert a search layer before feeding everything to the LLM at once.

There’s No Need to Stick with the Initial Method

The distinctions made so far are not meant to lock you into a single approach from the start. As your understanding of the problem grows, the appropriate method may change. Even if you initially used an LLM to explore because you didn’t know what to look for, later discovering a distinctive error name or configuration setting lets you narrow the search with simple string matching in subsequent investigations.

For example, when encountering a timeout on an API you’ve never seen before, the logs alone might not even produce any candidate causes. Suppose you examine the code and execution history with an LLM, reproduce the issue a few times, and determine that it occurs only when a particular retry setting is combined with a specific error code. After that, you don’t need to ask the LLM each time to detect the same condition. If the error code and configuration values are retained in structured logs, you can retrieve candidates using ordinary searches or rules. If the condition is deterministic, you can even incorporate it into tests or monitoring checks.

The same applies to Jev. Even for questions that couldn’t be resolved without comparing multiple pieces of evidence, repeated cases can clarify decision criteria—e.g., “if this field has this value and another field is empty, there is a mismatch.” If those criteria can be expressed in code, handling them with definitive rules thereafter will produce consistent results for the same input and make the reasoning for the decision easier to trace. It doesn’t mean Jev has become unnecessary; rather, some of the questions that used to be asked to Jev can now be expressed as simpler procedural steps.

Even for documents that were previously searched via embeddings, once the common paraphrases become known they can be captured with synonym dictionaries or keyword searches. Conversely, if new phrasings keep appearing, simply expanding the dictionary won’t keep up, so retaining embeddings remains justified. It’s not a matter of which technology is superior, but rather how far the currently known rules can be explicitly expressed.

However, it is risky to replace a model’s judgment with a rule solely because the same answer has been produced repeatedly. You need to verify which input conditions actually determine the conclusion and decide how to handle exceptions and unknown inputs. Even if the rule worked correctly on past cases, applying it definitively to entirely new types of failures can actually slow the investigation. Only the portions that can be resolved with rules should be processed automatically; cases that don’t fit the rules should be sent back for further investigation or limited decision‑making.

In other words, it is plausible to start by using an LLM to explore the problem, then hand off questions with sufficient evidence to Jev, and finally move any parts whose conditions have been formalized into search, rule‑based processing, or testing. However, this sequence is not mandatory, and if at any stage a human can read the relevant documents and reach a definitive answer, the process can stop there. The choice of method depends less on the problem itself and more on how much information is currently available.

We have not yet measured how much time, API costs, or frequency of selecting incorrect candidates can actually be saved by this approach. A small Japanese evaluation fixture has been added to jev‑mcp, but this is not a comparative experiment to assess the relative merits of the four methods. As noted in the evaluation JSONL article, verification of the model‑invocation pipeline and verification of decision quality in the field need to be kept separate. (related article)

Also, even when jev‑mcp runs on a local MCP server, the decision content is still sent to the TypeSafe API. If embeddings or the LLM are configured to use external APIs, the same considerations about transmitted data apply. Before relying on convenience, we should first confirm whether it is acceptable to send logs or proprietary code that contain confidential information as‑is.

For now, the boundaries I want to enforce are threefold: “candidates obtained via search,” “relevance judged from evidence,” and “the actual cause confirmed.” While it is possible to funnel everything into a single model, I prefer an architecture that keeps clear what has actually been verified.