An agent told me the refactor was done

I asked an agent to change a pattern in a codebase I already understood. Not a port from one language to another. One pattern into a different pattern.

It made progress. Then it stopped in the middle and told me the work was done. Half the tree had moved. Half of it had not. I did not have a map for the leftover rooms, and the agent was already proud of itself.

That is the part the replacement crowd skips.

I have a simple rule about technology. It is a tool. I have believed that about languages and frameworks for a long time. Agents and LLMs sit in the same pile. I pick them for a job. I do not build a personality around the one I happened to open this month.

The difference is the other tools are mostly deterministic. You run the compiler twice and you get the same complaint. You run the same agent prompt twice and you can get two different buildings. Sometimes both look finished.

When an agent beats a program

I keep a question in my belt: when does an AI agent make more sense than a programmatic solution?

The answer I keep landing on is when the input is unknown, unstructured, or the boundaries are loose. If I can name the inputs and the outputs, I want a program. Programs do not get bored and declare victory. If I cannot name them yet, an agent is a decent way to poke at the dark.

That is a narrower job than “do my work.”

Prototyping lives there. So does generating throwaway test data. Anything that does not need to be true yet, where I can accept speed over accuracy because I can throw the output away. I will not take that trade on a decision I have to live with.

FormBeam is the cleanest example I have. Getting from idea to a working prototype took a matter of hours. Two days if I count the wandering. That still feels like a trick. I had something I could click. I could tell if the idea was real.

Then I kept going. I leaned on the agent for choices I should have made myself. Not because the code would not compile. Because I let it keep deciding what the product was. By the time I wanted to launch, I had a lot of refactoring in front of me, and most of it was me showing up late to my own project.

The agent did not sneak that past me. I handed it the wheel and then acted surprised about the destination.

The replacement story

I have met enough people who talk about AI as the end of knowledge work. I have not seen that proven. Not for software, and not for the other jobs people wave at.

It is good at pieces of those jobs. A first spike. Fake data so I can see a screen. The kind of search I used to take to Google. None of that is a full replacement. A replacement would finish the refactor, know it was not finished, and not need me to reconstruct the plot. We are not there. I do not think we are close.

Calling agents a tool is not a humility pose. It is a way to keep the job description honest. You still pick the job. You still decide when the output is good enough. You still own the merge.

The LGTM problem

Using these things every day has hardened what I think a good engineer is.

It is very easy to watch an agent produce a pile of plausible files and hit approve. The diff is long. The summary is confident. LGTM. Merge.

That is the dangerous moment.

A good engineer used to be the person who could hold the system in their head and change it without lying to themselves. That bar did not drop because the keyboard got help. If anything it moved. You now have to catch work that looks finished. You have to notice when a decision got made in a file you did not open. You have to stop the run when the agent is solving a different problem than the one you asked.

I do not think that makes the tools worthless. It makes review the part that still counts. Anyone can kick off a run.

Greenfield is not a blank check

I trust agents more at the start of a repo than in the middle of one, and even that has a limit.

Greenfield is loose by nature. The inputs are not fully known. That matches the question I asked earlier. Give the agent a blank folder and a goal, and it will give you something you can react to. That is useful.

Give it a living codebase with no rules and it will do whatever is convenient. A new helper next to the one you already shipped. A refactor that stops at the files it felt like opening.

Clear documentation and boundaries help. Not perfectly. Better. If I write down how we name things, where new code lives, and what not to invent, the output stays in the lines more often. If I do not, I am paying myself to clean up a stranger’s taste.

That is also why I do not want another agent framework every time a systems problem shows up. Most of those problems already have boring tools. I would rather point an agent at a package that exists than watch it invent a queue mid-task. The instructions are cheap. Someone still has to install the dependency and read the diff.

Who gets credit

If an agent wrote most of a pull request, I still give the credit to the person who ran it.

They picked the tool and they were supposed to review the work. Listing the model is fine. Pretending the human was a bystander is not. If you kicked it off and you merged it, it is yours. That includes the stupid parts.

I want that to stay true in the other direction. If I ship a mess because I approved a run I did not read, that is also mine.

None of this is me pretending I write every line by hand. I use coding agents every day. Hermes, Claude Code, tools in that family. I still open a chat model the way I used to open Google. The brand names are not the point. The supervision is.

Agents will get better. The tool test does not care. When they do, I will still ask the same question: is this a job for a program, or is the input too loose? I am done treating a confident summary as proof that the work is finished.