OpenAI reports 97.9% accuracy on its COBOL migration test

OpenAI published a COBOL result on 19 October that reads like the end of a problem the banking industry has carried for forty years. Its GPT-6.1 Sol model scored 97.9% on byte-for-byte behavioural fidelity when translating COBOL programs, against 5.4% for the previous model, a sixteenfold jump. At low reasoning effort it still managed 86.6%, and 39% cheaper per call. The MultiLangBench results went up on GitHub for anyone to check.

“The evaluation targets edge cases in rounding, missing-record handling, and COBOL-specific numeric coercions.”

— OpenAI MultiLangBench results, published 19 October 2026, https://www.techtimes.com/articles/329306/20261021/gpt-6-cracks-cobol-979-migration-accuracy-shifts-ai-bottleneck-code-enterprise.htm

Then do the arithmetic the percentage invites. A 97.9% score means 2.1% of lines behave differently from the original. On a single 100,000-line program that is 2,100 lines that do something other than what the bank thinks they do, and finding which 2,100 is the entire job. Estimates of how much COBOL is still in production run from 250 billion lines upward; 2.1% of the lower figure is 5.25 billion lines. At the cheap setting it is 33.5 billion.

Where the benchmark stops

The benchmark also stops where mainframe migrations actually fail. It does not cover copybooks, the shared data layouts every program depends on, nor JCL job streams, nor CICS transaction environments, nor cross-file dependencies. Those are not edge cases in a bank; they are the system. Translating the COBOL is the part of a migration that was never the hard part.

Anthropic went at the same market from the other end in February, pointing Claude Code at discovery rather than translation: read the whole codebase, map the dependencies, trace execution paths, surface the couplings nobody documented. IBM sells watsonx Code Assistant to turn COBOL into Java. Three vendors, three different claims about which part of the problem is the problem.

The stakes are easy to state and hard to hedge. IBM says COBOL runs 95% of ATM transactions worldwide and 80% of in-person card payments. A migration that is 97.9% right is a migration that silently changes how 2.1% of those behave, and rounding and missing-record handling are exactly the categories the benchmark admits are hardest.

A real number answering a narrow question

We have been here before with a benchmark that answered a narrower question than the headline implied, when a 30B model beat Nvidia's flagship on Nvidia's own test. The number is real and the test is published, which is more than most vendors offer. What it does not cover is where the forty years went.

Nothing here should be taken as financial advice, just information to consider. The 97.9% is OpenAI's own benchmark on its own model, the line-count estimates are third-party and disputed, and the multiplications of one by the other are ours.