The social-proof machine for AI techniques and tools is circular, and it’s getting absurd. A rant.

The social proof machine

On the way to the office the other day I was catching up on AI coding videos, and I heard something I’d heard before, but this time it was the last straw. A popular agentic coding influencer said: “this skill is absolutely blowing up on social media!”.

OK… so?

We’re in this strange era where a lot of people are talking about AI coding in ways that aren’t backed up by anything except social proof. Other people are talking about it, other people are doing it, so it must be good.

In particular, there’s a flood of new frameworks, skill repositories, plugins, and whole agentic apps that are supposed to replace your entire office with a bunch of virtual employees. They have tens of thousands of GitHub stars. People get on YouTube and say, “Look, this thing has tens of thousands of GitHub stars, so this is the new hotness.” Then you start to dig into it, and you realize there’s no there there. There’s zero evidence anyone is using these tools for anything other than making breathless videos about how they tried it for a week, and it replaced a team that didn’t exist in the first place, and they felt like they were getting so much done.

The tools that do nothing (except accumulate stars)

What I see a lot of lately is tools that promise to make agentic coding more effective, or to hold down your token count, launched with very excited AI-generated copy about how well they work. Then somebody does a little digging, and it turns out they have no benefit whatsoever.

There was one tool for reducing token count and making agents more efficient. The folks at JetBrains did a rigorous analysis and found that at best it was zero gain, and from there it went downhill: it actually increased token counts and cost. Then somebody else did their own research and found the exact same thing. I’ve seen a few of these now. It’s not just that the claims are overblown; there’s no benefit at all. And yet the repositories sprang up overnight like mushrooms, collected tens of thousands of stars, and people treat those stars as a measure of legitimacy.

Stars used to mean something

They used to. A development tool could only pile up that many GitHub stars if it was genuinely in widespread use across a lot of working shops over a long period of time. Now plenty of the people promoting these things on social media and YouTube aren’t working engineers with a deep background. AI tools have opened the field to a lot of people who weren’t previously involved in software development, or only tangentially. Don’t get me wrong, I think that’s great. I’m not interested in putting walls around the ivory tower, and I love it when something brings a whole influx of new people into software.

The downside is that when a technology lets anyone look legitimate… then whoever has the best social media skills, the best production values, and the best on-camera charisma can make something look real when it isn’t.

It’s compounded by the fact that these tools are very good at making you feel busy and effective. The agents are happy to tell you how smart and correct you are and how much work they’ve done. You can have a little virtual office full of virtual employees, all looking very busy. And the agents can write voluminous, professional-looking copy about how great the tool is that you published overnight on a GitHub page. And even build you a professional-looking website for it. It’s not at all obvious that there’s no “there” there.

OK, so what?

If you’re someone using the ramp of agentic coding to get into software development, what do you do with this information? I could say you should follow and subscribe to old graybeards like me. And you should; some of us actually do have useful things to contribute. Also, I have a family to feed, so check out Graceful.Dev.

But beyond that, the key is metrics. Sit down, step back, and figure out what defines actual effectiveness and actual delivery. Hard metrics for whether you’re accomplishing things, whether what you’re trying is making the process better, whether you’re pushing your goals forward. Which metrics depends on your position:

• In a shop that already has products, it’s probably internal metrics: Is the bug count going down? Are we making our delivery goals? Are new features getting in front of customers, and are they buggy? Is the feedback good or bad? Are we gaining customers or losing them?
• If you’re building your own thing: Where are you on the line to market? Do you have a hard goal? Do you know what needs to be in place before you get there?

There are many ways to define metrics, and plenty of books about it. I think we’re in an era where anyone involved in software development has to get a lot more literate about these business-facing measurements, because we have minions to do the grunt work we didn’t have before. Now you’re the one responsible for asking whether you’re actually progressing, and whether you’re getting better at progressing.

So when a new tool sounds exciting, figure out your evaluation criteria for deciding whether it actually helps. Agents are good at helping with this. Tell one you want to set up an eval. I did this recently for a work process that could be split across many lighter agents or a few larger ones. I had an agent construct an evaluation protocol to work out which approach performs better with current models, came to some hard conclusions, and rolled them into my standing guidance.

That’s what you have to do in an era of everything from well-meaning people who aren’t as rigorous as they could be, to total hucksters with 10,000 stars on their GitHub repositories. Become responsible for your metrics, your measurements, your hard data: what’s showing you that this is helping or not helping you reach your goals? That means knowing what your goals are, and that’s more essential than ever.