CTGT Separates Repeatable AI Outputs From Correct Business Decisions

An AI agent can process the same business document four times and produce four different sets of obligations—even when the underlying language model responds consistently to identical requests. That is the reliability problem CTGT wants to address with Concordance, a new open-source tool designed to trace where AI applications begin to behave differently.

Released under the Apache 2.0 license, Concordance gives businesses a way to compare model requests, responses, and the intermediate stages of an AI workflow. Its central insight is important for companies moving AI from experimentation into operations: inconsistent decisions do not always originate inside the model. Application code, changing identifiers, shifting lists, and evolving internal registries can alter the information sent to the model before it makes a judgment.

For enterprises using AI to handle compliance, insurance claims, approvals, or other consequential tasks, that distinction matters. A system can produce plausible answers and still fail to apply business policy consistently. But there is another critical distinction: reproducibility is not the same as correctness. Concordance aims to help teams investigate the former; they still need independent checks to establish the latter.

CTGT Opens Its AI Reliability Testing Approach to Developers

CTGT describes itself as an applied interpretability lab studying how AI models and the software around them behave in regulated environments such as banking and insurance. The company has released Concordance's testing protocol, reference implementation, and experimental data as open-source resources.

According to CTGT, businesses can use and modify the software commercially under Apache 2.0. The company says it has no paid or hosted Concordance edition planned. Its proprietary compliance application, however, is separate from the open-source release.

The tool emerged from a problem inside CTGT's own compliance application. When the system processed a nine-page retail banking manual four times, it produced four different sets of obligations. One result omitted a requirement to escalate a serious issue to the compliance chief.

That omission illustrates the risk of deploying AI beyond drafting and summarization. When an agent's output becomes an operational record, a missing requirement can matter more than a stylistic difference in its response.

CTGT founder and CEO Cyril Gorlla argued that testing should begin at the point where AI commits a decision that someone may later need to defend. In regulated industries, that means being able to reconstruct what the system received, what it returned, and how the application turned that information into an action.

Four Runs Produced Different Results Despite Consistent Model Responses

CTGT's experiment used Qwen3.8-27B with deterministic inference enabled through SGLang. Across four runs, the application extracted 32, 31, 33, and 33 obligations from the same banking manual.

The totals were close. The actual results were not. All four outputs differed.

The most revealing finding was that the model returned consistent responses across all 74 identical requests that recurred during those runs. The application, rather than the model's response to a repeated request, was changing what it sent.

CTGT identified three causes:

• Upload-specific identifiers appeared in prompts.

• A list of 30 routing options changed order.

• A topic registry expanded between runs, altering subsequent requests.

In one comparison, just four identifier lines differed within a 12,525-character request. That small change was enough to affect the outcome: the model classified five conflict-resolution rules as background information, causing an obligation retained in another run to disappear.

CTGT then replaced variable identifiers with stable labels, sorted the routing list, and isolated each test in a fresh organizational environment. Five subsequent runs each produced the same 34 obligations.

This result demonstrates how seemingly minor implementation details can influence a complex workflow. It does not establish that every enterprise AI application has the same problem. The experiment examined one application processing one document, and the source report provides no evidence that quantifies how frequently these failures occur across the broader market.

Nevertheless, the mechanism is significant. AI reliability depends on the entire path from source data to final action, not just the model selected by an engineering team.

Concordance Finds the First Point Where an AI Workflow Diverges

Concordance is designed to examine both the software serving a model and the application using it. It captures exact requests and responses, compares outputs at successive stages, and identifies the earliest point where two runs stop matching.

That process separates two different failure modes. The model may respond differently to identical requests, or the application may change the requests themselves. Without that distinction, a developer could spend time adjusting model settings when the real problem sits in application logic.

Research from Thinking Machines Lab helps explain why the first failure mode can occur. Even at the least-random generation setting, differences in server batching can affect numerical calculations and prevent identical outputs. CTGT uses SGLang's deterministic mode, which its representatives said builds on this research.

In a separate serving test, CTGT reported that Qwen3.8-27B matched its control response across all 800 requests under the deterministic configuration. That measures exact textual agreement, not whether the answers were correct.

The compliance experiment makes the distinction even clearer. When CTGT replayed the request behind an incorrect classification, the model reproduced the same mistake eight times.

A repeatable error is still an error. Concordance can help establish whether a result is reproducible and locate where variations begin, but its reported results do not show that the tool automatically prevents policy violations. CTGT identifies measuring whether decisions remain faithful to policies despite changes in wording as a future challenge.

Decision Models Address Another Part of the Reliability Problem

Concordance is arriving alongside another approach to more controlled AI behavior: decision models. TypeSafe's Jev, for example, returns constrained choices, scores, and probabilities that software can consume instead of unrestricted text. AWS also describes a broader category that includes Jev, SemIf, and Strands Decider 2B.

The distinction between these approaches is important. A decision model constrains an individual judgment to a defined output format. Concordance examines whether the wider application reproduces its behavior and where discrepancies arise.

A structured answer can prevent a model from returning an unexpected category, but it cannot guarantee that the selected category is correct. If an earlier stage omits a compliance obligation, a downstream classifier cannot reliably enforce a requirement it never receives.

TypeSafe's documentation for Jev 1.13 also describes sensitivity to option ordering and difficulties with irrelevant context. Its self-consistency demonstration reported label changes on two of eight questions across repeated tests that included a fresh identifier in each request. Routing uncertain outcomes to human review improved agreement, but did not make the model deterministic. Because the requests changed, that demonstration does not establish instability under identical inputs.

The practical lesson is to treat these tools as complementary rather than interchangeable. Constrained models can narrow the choices available to AI. Conventional code can enforce mandatory actions. Concordance can help identify where a complete workflow becomes inconsistent. Independent evaluation must still determine whether the final decisions comply with business rules.

CTGT has not published a comparison of Concordance using Jev, so the available evidence does not establish which combination would perform better on the banking example.

Existing AI Monitoring Platforms Remain Part of the Picture

Concordance enters a market where developers already have tools for tracing, evaluating, and debugging AI applications. Gorlla positions it as an addition to those systems rather than a replacement.

LangSmith supports repeated evaluations, inspection of intermediate agent decisions, and comparison between application versions. Braintrust supports multiple trials and exposes model calls, application functions, and tool activity. Langfuse captures prompts, responses, retrieval steps, and tool usage, alongside timing and usage information. Arize Phoenix combines tracing, evaluation, and experimentation through OpenTelemetry and OpenInference instrumentation.

These capabilities overlap with the evidence needed to investigate inconsistent behavior. CTGT has not supplied a head-to-head benchmark proving that competing platforms cannot identify the same failures.

Concordance's proposed distinction is its focused comparison procedure: repeat the work, align the stages, and determine whether the divergence originated in changing inputs or changing responses. If that procedure makes investigations easier to reproduce, it could provide value alongside broader monitoring and evaluation platforms.

CTGT also cited an anecdote about an unnamed consulting firm finding overlooked false positives with its earlier LineageEval framework. That example concerns a different tool and should not be treated as evidence of customer outcomes from Concordance.

Businesses Need to Prioritize Decisions With Real Consequences

The first adoption question is not whether every AI interaction should be monitored. It is which decisions create enough operational or regulatory risk to justify repeatability testing.

Gorlla recommends prioritizing approvals, compliance obligations, insurance claims, and other decisions that can become authoritative business records. A drafting assistant whose work receives human review presents a different risk from an agent that automatically approves a transaction or records a compliance outcome.

Enterprises can begin with a practical testing process:

1. Preserve complete inputs and outputs. Record the requests and responses associated with consequential model calls so teams can reconstruct what happened.

2. Establish repeatable test cases. Use trusted examples with known expected results and rerun them when models, prompts, source documents, or application code change.

3. Separate consistency from correctness. Compare repeated outputs, then independently evaluate them against approved policies and expected outcomes.

4. Enforce mandatory rules in code. Where a business process requires escalation under a defined condition, make the escalation a deterministic workflow requirement rather than relying entirely on an agent to choose it.

5. Assign business ownership. Define who approves decision categories, sets escalation thresholds, reviews failures, and accepts the consequences of automated actions.

These practices require engineering effort even when the debugging software is free. Teams must still integrate the tool into their workflows and account for model calls, storage, testing, and maintenance.

Open Source Can Lower the Barrier Without Removing the Work

Apache 2.0 licensing makes Concordance available for commercial use and modification without requiring businesses to purchase a hosted edition from CTGT. That removes one potential purchasing barrier, although it does not eliminate deployment costs or guarantee compatibility with existing systems.

For companies building AI infrastructure, the strategic value lies in gaining a repeatable way to investigate failures before those failures become expensive incidents. Better evidence can help teams determine whether to change prompts, stabilize application inputs, adjust workflow logic, or reconsider the model itself.

The approach may be particularly relevant in regulated industries, where businesses need to explain decisions rather than simply demonstrate that an AI system usually performs well. But the source material does not establish that Concordance meets every regulatory audit requirement or that its use alone makes an AI deployment compliant.

Reproducible AI Is Becoming an Engineering Requirement

CTGT's Concordance release highlights a weakness in a common assumption about enterprise AI. Making a model more deterministic does not necessarily make the application around it deterministic. And making the application repeatable does not guarantee that its decisions are right.

The banking-manual experiment shows why the distinction matters. Small changes in identifiers, routing order, and registry contents coincided with different compliance outcomes, even though the model consistently answered identical requests. Stabilizing the inputs produced five runs with the same 34 obligations, offering a concrete demonstration of how workflow design can affect repeatability.

For businesses, the next step is not to expect a single tool to solve AI reliability. It is to combine reproducible testing, explicit business rules, controlled model outputs, and independent policy validation. Concordance offers one open-source method for investigating where a workflow begins to diverge. Its long-term value will depend on how effectively organizations can integrate that method into real systems and use it to prevent consequential errors.

The central lesson is straightforward: enterprise AI must be evaluated not only on the quality of its answers, but also on the consistency of its decisions, the integrity of its inputs, and the ability of the business to defend what the system ultimately does.

CTGT Separates Repeatable AI Outputs From Correct Business Decisions

CTGT Concordance, AI agent reliability, AI debugging tools, open source AI, AI compliance testing, deterministic AI inference, AI workflow tracing, enterprise AI governance, AI decision consistency, language model testing, AI policy enforcement, Jev decision models, AI observability, reproducible AI systems