Can AI convert COBOL?

Part I — What LLMs are genuinely good at here

This is not a strawman section, because the strengths are real. Ask a capable model to read a 2,000-line COBOL paragraph soup and describe its intent, and it will produce a plausible narrative in seconds — faster than any junior engineer, often catching the gist of a routine no one has read since the engineer who wrote it retired. For orientation, that is genuinely valuable.

Three uses hold up in practice. Reading intent: summarizing what a program appears to do, so a human knows where to look first. Drafting documentation: turning verified facts into readable prose, which is most of what documentation is. Review assistance: a second pair of eyes that never gets tired, flagging the paragraph a reviewer should re-check. We use models for exactly these jobs ourselves, and Part V explains where they sit in our own pipeline.

It is also worth being honest about why COBOL, specifically, sits near the edge of what these models do well. Production mainframe programs are long — a single program can run past 50,000 lines with copybooks fanning out behind it — and a model working through a context window reads such a system the way a tourist reads a city through a bus window: real glimpses, no map. And where the public internet holds oceans of JavaScript and Python for a model to learn from, the COBOL that matters is precisely the code that never left the building. The models are weakest exactly where your risk is highest.

Notice what the three good uses above have in common: in each case, a human verifies the output, or the output is not load-bearing. The failure modes begin the moment that stops being true.

Part II — The failure modes that matter

Four failure modes decide whether an AI-assisted conversion survives contact with production. None of them shows up in a demo, because demos are built from code the presenter understands. All of them show up in an estate, because estates are built from code nobody fully does.

Probabilistic variance: same input, different output

A language model samples from a distribution. Ask it to extract the rules from the same program three times and you get three artifacts — sibling documents, similar in shape, different in substance. Which one goes to the sign-off meeting? In our benchmark runs, three LLM passes over the same program produced three different outputs; the deterministic pipeline produced one, three times — three runs, one hash. Nobody can sign off on a moving target, and in a regulated cutover, sign-off is the product.

Silent semantic drift: the edge cases are the business

COBOL’s hard parts are not syntax; they are semantics that have no clean analogue in the target language. EBCDIC collation orders characters differently from ASCII, so a translated comparison can sort and branch differently on the same data. Packed-decimal COMP-3 fields carry precision and truncation behavior that a casual mapping to a float or an int quietly changes. PERFORM ... THRU fall-through means control flow depends on paragraph order in ways a structured rewrite silently reorganizes. REDEFINES makes two fields share the same bytes, and a translator that models them as separate variables has changed the program’s meaning without changing its apparent logic. Every one of these failures compiles. Every one of them reads well in review. They surface only when production data hits the edge — and in a system that has run for thirty years, the edges are where the business lives.

The audit problem: no provenance, no sign-off

Suppose the model’s extraction happens to be right. Can you show it? An LLM cannot tell you which source lines a claimed rule came from — not reliably, and not verifiably, because there is no mechanical chain from input to output. Your reviewers are left approving assertions, and your regulator is asked to accept “the model said so.” An unsourced answer is not an answer in that room; as we put it elsewhere, it is a liability with good grammar. Without lineage, even correct output is unusable at exactly the moment it matters.

Think about who actually signs a cutover in a bank or an insurer: an application owner, a risk officer, often an external auditor. Each of them signs against evidence, and evidence has a shape — this output, from this input, by this documented transformation, checkable by a third party who was not in the room. A pipeline that cannot produce that shape has not failed a technicality; it has failed the definition of the deliverable. The engineering conversation about model quality is interesting. The governance conversation about provenance is decisive.

The 2% that costs 100%

Accuracy percentages hide an asymmetry. A 98%-correct translation of an interest-calculation module is not a 98%-good system; it is a system that computes interest wrong for some fraction of accounts, in production, with the error compounding daily until someone’s reconciliation catches it. The cost of the miss is not proportional to its size — one wrong rounding rule can exceed the cost of the entire modernization program. If the errors announced themselves this would be manageable; the trouble with fluent output is that its failures are fluent too. This is why the operative question was never “what percent can AI convert” but which percent — and whether the pipeline can tell you.

Part III — Deterministic vs probabilistic translation

The dividing line in this market is not AI versus no AI. It is whether the load-bearing step of the pipeline is a parser or a sampler.

Two clarifications, because “deterministic” gets misread in both directions. It does not mean no AI — the next section and the methodology after it should make that plain. And it does not mean the pipeline claims to understand everything: a parser meets constructs it cannot prove, and the deterministic answer to those is a flag and a human, not a guess. The roughly 5% that Palm Ark does not extract automatically arrives as a named, located, reviewable list — which is a different universe from 5% of silent errors scattered anonymously through fluent output. Known unknowns are a work plan. Unknown unknowns are a production incident with a date you don’t get to choose.

The other nuance the table can’t hold: these two columns are not enemies. LLMs earn their keep inside a deterministic pipeline — proposing human-readable names for extracted rules, drafting the prose descriptions reviewers read, flagging the paragraph that deserves a second look. The determinism does the proving; the model does the prose. What a model should never be is the unverified step your cutover depends on.

Part IV — PalmBench: measure it, don’t argue it

PalmBench is our benchmark, and it is deliberately boring: one production-scale COBOL program of roughly 50,000 lines, one instruction — extract the business rules — and the receipts on the table. Each extracted rule is scored for correctness against a human-adjudicated reference, and for lineage: whether it arrives with the source lines it came from. In that benchmark, generic LLMs land around 30% with no lineage; Palm Ark lands around 95% with 100% lineage — and identical output across runs. Figures are from our internal benchmark; ask us for the methodology.

generic llm · single pass~30% correct · lineage 0%

palm ark · deterministic pipeline~95% correct · lineage 100%

repeatability · llm: three runs, three artifacts · ark: three runs, one hash

PalmDigitalz · PalmBench 2026 · palmdigitalz.com

Read the gap between those bars carefully: it is not primarily a statement about model quality. It is a statement about architecture — a sampler asked to do a parser’s job, versus a parser doing it. Swapping in next year’s model moves the first bar somewhat; it does not give the first bar lineage, and it does not make two runs agree.

Two things PalmBench deliberately does not claim. It is not a statement that language models are useless — Part I said the opposite, and the models keep improving. And it is not a universal score: it measures one task, rule extraction from production-scale COBOL, because that is the task a modernization program lives or dies on. A model that writes excellent Python and a model that scores 30% on this benchmark can be the same model. Task-specific evidence beats general reputation, in both directions.

You can run PalmBench against any vendor on the market — including us. The protocol fits in a paragraph: pick one production-scale program of yours, the uglier the better. Give every contender the same instruction — extract the business rules. Have your own experts score a sample of the output for correctness, and for every rule, ask for the source lines it came from. Then run the whole thing again and diff. The benchmark page carries the standing offer: we will reproduce it on your code, against any model you choose, inside your perimeter.

Part V — The Gazelle methodology

The Gazelle methodology is our name for the discipline behind those numbers: verifiable rule extraction, in four steps, each producing an artifact the next step — and your auditors — can check.

Why name it for the gazelle? Because a gazelle is fast — and it never takes a step it can’t land. Speed, in this method, is a consequence of certain footing rather than a bet against it: nothing has to be redone, because every step was provable when it was taken. Programs built on guessed foundations move quickly for a quarter and then spend a year in re-verification; programs built on receipts move at one speed the whole way, which is how they arrive first.

In practice, the methodology also changes the shape of a proof of concept. Because every artifact carries lineage, a POC does not end with a demo and a feeling — it ends with a scored extraction of your own program that your own reviewers verified line by line. Whatever you decide afterwards, the rulebook is evidence you keep. That is the calm way off the mainframe, and it is also the answer to the question in the title. Can AI convert COBOL? Yes — when the AI is held inside a pipeline that can show its work. The strategy that follows from that is the Exit Doctrine.

Frequently asked questions