OpenAI put an impressive-sounding case study about how they helped Asana save $5.9M with a single migration.

“Asana cleared 5 years of engineering work in 2 weeks with Codex. Using OpenAI Codex, Asana replaced an outdated testing system in two weeks for about $12K.
For this project, Codex helped Asana’s engineers remove Enzyme, an outdated testing system that had made the company’s code harder to upgrade. Model and infrastructure costs came to about $12,000, compared with Asana’s roughly $6 million estimate for the previous staffing plan.
After 1.5 weeks of engineering effort spread across two calendar weeks, Enzyme was fully removed. Model and infrastructure costs totaled about $12K. For comparison: the previous plan was expected to take at least five years and estimated to cost roughly $6M. The experience changed which long-running software projects the company believes are practical to take on.”

For context, Asana migrated from Enzyme to React Testing Library, which indeed would not be a simple migration. However, their estimate of four engineers (each on circa $300K/year, according to OpenAI’s arithmetic), spending five years on the project, had me like: But thinking about this for longer raised the question: what does a migration from Enzyme to React Testing Library even look like?

Enzyme to React Testing Library migration

Let’s take a simple test and see how it looks in various testing libraries. For our test, we want to verify that a button increments a counter. Here’s our button in React:

Now, the test to verify this, in Enzyme:

And let’s rewrite this test in React Testing Library:

The two tests do the same, but they have completely different syntax! The main reason for this difference is that the two frameworks use a fundamentally different approach to testing. Whereas Enzyme is oriented towards component testing (notice how the test operates on a component instance), the React Testing Library operates on the rendered Document Object Model (a data structure representing the HTML shipped to the client) so the test sees the whole rendered page, not just the component its written for. That’s why the testing approach will differ radically between the two frameworks, especially when testing complex user journeys.

It took Airbnb 6 weeks to migrate 3,500 tests with AI

Last year, Airbnb revealed how they migrated their Enzyme test suite of 3,500 component test files within six weeks with LLMs. The estimate of doing this by hand was 1.5 engineering years. Airbnb did the LLM-aided migration in a multi-phase process.

Airbnb’s team had to build loops to keep retrying migrations; once they did, 75% of files were migrated in just four hours, and the migrations were straightforward. They then built a more sophisticated refactor pipeline for the remaining 25% of tests; after building the pipeline, the new loop migrated most of the remaining tests (97%) in total, after running over 4 days. The remaining 3% was done with LLM input, with engineers finishing it in a week.

AI makes impractical migrations doable

On the basis that it took Airbnb six weeks, I find it credible that it took Asana two weeks to migrate what is probably a similarly complex test infrastructure from Enzyme to RTL, a year later.

The time and $6M cost as quoted by OpenAI feels inflated. I assume the numbers were based on an estimate that a fulltime engineer could do a maximum of X tests migrated per day, where X was between 5 and 10. Then, calculate the number of engineering years this takes (perhaps 20 engineering years), and multiply by the cost of an engineer. You estimate a project like this when it’s an undesirable project you really don’t want to do as an engineer!

So, looking at it from this point of view: does it matter if the estimate was 1.5 years (Airbnb) or half a decade (Asana, hypothetically)? Or if the estimated cost was $1M or $5M? It would still be an impractical migration and a foolish endeavor: too long and distracting, at least, in the “traditional” way!

Pre-AI, years-long migrations were rare. In 2021, Sentry took 1.5 years to convert their frontend codebase from JavaScript to TypeScript. That was a migration of 1,100 files and 95,000 lines of code! Around 10 engineers worked on the migration, so if we assume a $300K-per-engineer cost, that’s a $2–4M cost for circa 95,000 lines of code.

Indeed, Asana’s reported $12,000 migration cost could actually cost even less if there was ruthless focus on cost optimization. What about using a model that’s 10x cheaper than OpenAI, like an open model running on inference providers? After all, why use the most expensive model, especially if a company already owns GPUs, running models, making inference practically free (except for power costs, that is.)

A few more details from inside Asana

I managed to catch up with Dan Ubilla at Asana, who leads the Developer Productivity group at the company. He helpfully clarified a few things about the Enzyme migration post on the Asana site:

• Asana started the Enzyme migration in 2024. The team had 4,000+ Enzyme files to migrate, and they took a first stab at the migration with LLMs, at the time. Beyond migrating existing tests, much of this phase focused on getting the codebase into a state where future tests could be written more idiomatically.
• The remainder of the migration became low priority. Asana prioritized migrations as critical (“must do soon”), important (“important to do soon”), and opportunistic (“nice to have”). The company has a good number of critical and important migrations, and the remainder of the Enzyme migration sat as one of the many opportunistic.
• Five years was about when the migration was projected to finish. The effort was not about “a team working for five years, nonstop on the project” – not at all! It was about the timeframe that the opportunistic Enzyme migration would have been completed, given its priority, and done, realistically. Given this migration was going on for two years, the five-year estimate was a rational one.
• The team wanted to prove that LLMs drastically speed up migrations, and demonstrated it with an opportunistic, low-priority one. The Enzyme migration was not all that important to do quickly, and would have sat around for years. So it was the perfect candidate to prove that with LLMs, long migrations can be sped up!

And finally, one addition from my end:

The $6M cost was a back-of-the envelope estimation. I asked Dan how this estimate of the migration costing $6M came together. Dan confirmed that the estimation was done the same way as most of us do these estimations:

• Estimate how long it takes an engineer to manually migrate a single Enzyme file (including rewrite + validation). Be generous with this estimation
• Multiply it by the outstanding files to migrate (the 3,000+ ones)
• Add in the time to remove any traces of the framework, monitoring, static analysis tools
• Multiply by the hourly rate of a typical engineer
• … and the number came out to $6M

Expect long-avoided migrations to finally happen

A few months ago, Uber shared that they executed a massive JUnit migration in four months with two engineers and AI: moving 600,000 unit tests spanning 15 million lines of code by moving from the unsupported JUnit 4 to JUnit 5. This type of migration used to be impractical; with AI tools, it took eight engineering months of effort, plus AI costs.

The shared characteristic of all of the above migrations is that engineers needed to plan for it, design verification loops, and be involved throughout. I always dreaded migrations, so I see it as good news that we have a new tool to use for “grunt work” like this.