Introducing the Guest: #5 Josh Wills

Josh Wills is a software and data engineer with 25 years of experience. At Google, he worked on the ad auction system and led the analytics infrastructure behind Google+. As head of data science at Cloudera, he founded the Apache Crunch project and co-authored O’Reilly’s Advanced Analytics with Spark, and he went on to build and lead Slack’s data engineering team. He’s also the original creator of the dbt-duckdb adapter.

Since 2024, Josh has been a Member of Technical Staff at DatologyAI, which helps teams train better models, faster and smaller. There he builds tools that automatically select the best data for training deep learning models, along with the distributed pipelines that process it at trillion-token pretraining scale. We go into more detail on what it takes to work with datasets that gigantic.

DuckDB Yes, but also Kubernetes Disk Abstractions

As many of you might know, Josh was the initial creator of dbt-duckdb, so I asked him if he still uses dbt and, more importantly, how much DuckDB.

His answer was that everyone uses it: “Everybody uses DuckDB in some capacity. I feel like anyone who knows about DuckDB uses DuckDB for something. Everyone has their own specific use case for it”. He is using it less for his specific pipelines at work, as they have huge datasets, text data, not classical datasets: e.g., 30 TB is tiny. He uses Spark or Ray. He does much of the processing on text and multimodal data (time series, images, etc.), not much classical relational data in his day-to-day.

The use case for DuckDB he uses a lot is reading Parquet metadata that sits on S3, as the above data pipelines “throw off enormous amounts of Parquet metadata, which is still substantial but much smaller, on the order of 100 GBs”. Information about runs, when a dataset was last used to train a model, how much he pays to store it in S3 for indefinite periods. He usually asks his agents to read that on his dev box on EC2, which he runs these commands on, to cut down on transfer times, just for speed.

“We Lost a Whole Generation” of Data Infra People, Retired 2022, AI Pulled Him Back

When talking about what has changed since he worked at Google as a staff engineer (2007–2011) and led data engineering at Slack (2015–2019), and how data engineering workloads differ today with AI, he said Snowflake is great. Still, we do not have many people anymore who know how the data is laid out on disk.

But not for his size of workload. Josh works with such big data that we normies can barely fathom, and that’s why he uses lots of Kubernetes jobs and workloads. He likes Apache Celeborn (Pronunciation: spelled Celeborn, pronounced “Keleborn”), which abstracts the disks (not needing to know where they physically come from), because in Kubernetes you still do that manually, as Kubernetes is not really cloud-abstracted, he says:

Kubernetes is fantastic in that it’s in theory abstracted away from an individual cloud, but in practice it’s not, because where does your disk come from? If you’re on AWS or GCP you can use an EBS-like service, but if you’re running bare metal, that doesn’t exist the same way.

Celeborn has something called push-based shuffle write, where mapper nodes skip local disk storage, ideal for cloud-native architectures, which lets you separate storage and compute, so

The Spark jobs don’t need to worry about where their disk comes from anymore, from a shuffle perspective. There’s the shuffle service, no matter how much data you’re going to shuffle. You have your local disk, and then you have your shuffle service for all your other disk needs.

How Much Does the Model Matter to You?

When asked about the models and how much they matter, Josh said for him it’s like the iMessage blue bubbles vs. green bubbles. Android phones are fantastic, but he wants the blue bubbles. And Josh’s prediction is that all the models will become competent, so the real question becomes:

which one you like and enjoy hanging out with

In this interview, he added that it’s also the way it communicates. An interesting insight, he said, as someone who works on building models all day:

the fact that Opus 5 is so bad, it deeply indicates to me that we literally have no idea what we’re doing at all when it comes to training these models.

How to Code with Agents These Days

Josh says:

[..] a joke that I want to start my own RL environments company where it’s just me teaching the models how not to write absolutely garbage data pipelines. Because I use agents to write all of my code, with very, very few exceptions. And yet I still find myself deeply unhappy with the data pipelines it writes.

How to Get Fast Feedback Loops with Agents?

Josh says AI wins where there’s a large training model and fast responses, because if it’s not in the training data, it’s slow and it needs to do a lot of stuff. He says: “Any sort of environment in which the models have highly reliable training data with fast feedback mechanisms, they’re just going to dominate. Absolutely going to dominate. You see this with math, you see this with front-end engineering, any RL environment where there is a fast feedback mechanism.”

And that’s exactly what’s missing in large-scale data pipelines today:

What’s tricky about data pipelines, at least at a large scale, is that you don’t actually have that fast feedback mechanism available to you, because running a giant pipeline can take a long time and a lot of computers.

Feeling the Tuples Flowing through the Pipelines, a Gut Feeling

Josh, with his years of experience, has built a gut feeling:

Through years and years of suffering and writing gigantic pipelines, I can look at a pipeline and it kind of feels like something to me. I can feel the tuples flowing through it, because I have a mental model of how this is going to go.

Josh describes an aesthetic aspect of data pipelines:

I have, weirdly, a kind of aesthetic sensibility when it comes to data pipelines. A data pipeline can be beautiful to me. It can be elegant, well designed, thoughtful. That makes me want to look, in the way other people would visit an art museum, or read math proofs because they find them beautiful.

The Cost Aspect, Literally Lighting Money on Fire

When talking about cost, Josh says that:

“the models might waste time on the most pedantic, stupid checks for errors that are simply not possible given my knowledge of the structure of the upstream data sources.” “Literally lighting money on fire”

The Job of ‘Emergency Exception Handler’

He mentions that in his job, because the models haven’t learned everything yet, he is basically like a pilot who usually flies on autopilot, but in case of emergency, he is there to fix it or take over.

What Is the Tooling He Uses & How Agents Ease His Life at Work

When asked how he works or reviews code, he said: mostly giving feedback, like a staff engineer does. His life as a senior technical IC has been trending towards that for a very long time, he says. He primarily gives feedback. That’s his primary job. Interviewing people, reviewing design documents, and less hands-on, with very few exceptions.

He likes to work on high-demand, high-urgency work that forces him to create. This is back to the ’emergency exception handler':

If there’s a broken-ass data pipeline and it has to run right now, because a customer has a training job coming up and the data has to be ready for it. That’s the primary value I provide.

What Josh Likes about Agents: No Feelings and the Caffeine Dial

Josh also revealed that he likes working with agents because agents have no feelings. He can give agents feedback on demand, force things to be created, he says:

I don’t have to be very nice to the agents.

He can just say things:

I don’t have to care about their feelings or their career development, because they’re going to cease to exist when I’m done with this coding session. I can be like: this is bad and you should feel bad. Ctrl-C, you’re dead, I’m going to start over with someone else.

No More Unit Tests!

He also likes to automate most of the boring work. Who wants to do one more database migration? Or fix “flaky unit tests”? He likes to do the higher-level strategic stuff, which is how people end up becoming investors or product managers, he says.

I have an agent whose whole job is just to watch for flaky tests and fix them. That’s it, that’s all it does. Database migrations? Absolutely not. I have an agent that does database migrations. I have automated away the shittiest, lowest-value, most unpleasant parts of my life, and I am here for it.

Decisions Made Are What Is Left, and Communicating Why Is Key

It’s really the decisions we need to communicate. The decisions build the shared understanding, the choices we make, the trade-offs. That’s still the meat of engineering work.

Is Data Engineering Automated Away with Agents?

Josh says clearly that data engineering is not going away. For the same reason it hasn’t gone away for so long: the fundamental problems won’t go away. We’re still solving the problem of materializing views and caching!

I don’t see data engineering going away in any way, shape or form, even in this world. Aside from the fact that the RL environment problem makes it unusually difficult for the agents to be good at running these very long pipelines, the fundamental problem of data engineering is the same problem it’s been my whole career, and 25 years before I started working. It’s the materialized view problem.

It’s the two hard problems in computer science, right? Cache invalidation and naming things. And the agents still suck at them because they’re legit hard.

Humility and Burnout

It helps to stay humble, as he says he is very good at it, telling his family:

I am simultaneously an idiot and also among the best in the world at what I do.

Conclusion

One main takeaway from this conversation is that data engineering is the cache invalidation problem, just done in a batch-processing context. That’s important: when done right, agents dominate because of a fast feedback loop, like SQL with its explain plans or math. Gigantic Spark and Ray pipelines that run for hours across hundreds of machines don’t have that cycle yet. That’s why Josh still ends up as the emergency exception handler, and why others can vibe code a Databricks job that costs $200k a day.

What’s interesting is how little the model itself matters to him. It’s more about design and feel, e.g., blue bubbles vs. green bubbles: pick the one you enjoy hanging out with, switch when you don’t like it anymore, or hand off the flaky tests and database migrations to an agent. What’s left is what was always the hard part of data engineering: making the decisions and communicating the trade-offs.