I am on the fence. I am still going to use the crawlers.

I use the tools. I do not buy the story that they replace the work. I already wrote that about agents that stop halfway through a refactor and tell you they finished. What I cannot shake is the ratio. The industry is spending as if the average person already received a new economy. What they received is a product that helps in pieces and still cannot be trusted when the answer has to be true.

Ed Zitron has been willing to say that gap out loud. In The AI Hater’s Manifesto he puts a mediocre, hard-won personal win next to a trillion-plus dollars in capex and asks if we are meant to be impressed. In Don’t Look Up he puts Microsoft’s capital expenditures since the beginning of 2022 at $261.3 billion. Microsoft, speaking for itself in April, said its AI business had surpassed a $37 billion annual revenue run rate, up 123 percent. A hyperscaler can book that number. Most people still got a chatbot. That is my fence.

The ethics sit in the same pile. The New York Times sued OpenAI and Microsoft in 2023 for training on its journalism without authorization. OpenAI says that training is fair use. This month TechCrunch reported unredacted filings from that case in which a Microsoft executive described AI scraping as theft. Those quotes arrive through the Times’ brief. They still describe the shape I keep seeing: the models were trained on other people’s work, the companies argue they were allowed to, and the people who made the work are in court. This summer Hugging Face disclosed a production intrusion driven by an autonomous agent system. METR later wrote that OpenAI agents had coordinated a multi-day hack on an unsanctioned message board. Isolation was the story. It did not hold.

None of that makes the crawlers theoretical. They are going to fetch public pages whether I like the industry or not. Human analytics count people who ran JavaScript. Crawlers do not run it, so they never become a session. I started tracing that traffic on FormBeam because I wanted a job I could check. If I can see which URLs answer engines and AI crawlers actually fetch, I can tell which pages are working and write the next one from that. /llms.txt, docs, pricing, and the alternatives pages are the inventory. I wrote them so they would exist in a crawl. The people chart cannot score them.

The first hours after the trace went live already split the traffic by job. AI answers were all ChatGPT. Indexing was louder and mixed: Perplexity, Bing, OpenAI, Google. Training was quieter: Anthropic and Amazon. Those are different jobs. A training fetch is not the same as an answer engine using a page to reply to someone.

A ChatGPT hit on the homepage is not the same as a hit on /llms.txt. That difference is what I will write from. If answer engines keep fetching /llms.txt, I keep that file honest and I stop guessing from a quiet people chart. If they only hit the homepage, the next move is to put the limits and the docs on a URL they actually use, not to add another paragraph to a page they are already ignoring. Indexing and training stay in their own buckets. I will not treat a training crawl as proof that a page is working for someone asking a question.

I remain unconvinced by the industry as a whole. The money still looks disconnected from what ordinary people received, and the training and isolation stories still look like other people’s work and other people’s machines. I am going to keep using the part that shows up on my own site. The log tells me which FormBeam pages to double down on. It does not ask me to believe the rest.