Three months ago, Google was indexing 372 pages from my website.
Last week, that number dropped to 34.
I did not know why.
The traffic data looked like someone had unplugged a machine.
Impressions: gone. Clicks: basically zero. Google just stopped caring about my site.
The First Clue
I finally sat down with Google Search Console and looked at the Coverage report.
The numbers were ugly.
| Problem | Pages |
|---|---|
| Blocked by robots.txt | 296 |
| Discovered — not indexed | 727 |
| Crawled — not indexed | 171 |
296 pages blocked by robots.txt.
That is not a small thing. That is a signal to Google that a huge chunk of my website does not want to be crawled.
And when Google sees that, it starts questioning everything else on the site too.
The Culprit
I checked my robots.txt file.
There it was.
Disallow: /tools/majors/
Disallow: /tools/majors-cn/I had added those lines myself, a few months ago.
My logic at the time: these are programmatic pages, maybe Google does not need to crawl all of them.
My actual result: I blocked over 1,500 pages that Google had already started ranking.
One of those pages — a guide to studying Flexible Electronics in China — was sitting at position 3.72 in search results. I blocked it myself.
But it gets worse.
The Second Culprit
While digging through the server files, I found this inside a LiteSpeed cache folder:
User-agent: *
Disallow: /A hidden robots.txt file, generated automatically by the LiteSpeed caching plugin, telling every search engine to stay away from the entire website.
I do not know exactly when it appeared. I do not know how long Google had been reading it.
But the timeline fit perfectly.
July 1: 372 pages indexed. August 15: 35 pages indexed.
The collapse happened in exactly the weeks when Google would have refreshed its robots.txt cache.
The Third Problem
Two sitemaps containing 1,789 pages had never been properly submitted to Google.
Google knew the pages existed — it had found them through internal links. But every time it tried to fetch the sitemap, it got blocked. Because of the robots.txt problem.
So 1,789 pages were sitting in a “Discovered — not indexed” limbo. Google could see them. Just could not read them.
What I Fixed
In one afternoon:
- Removed the bad Disallow rules from robots.txt
- Deleted the LiteSpeed hidden robots.txt file
- Updated robots.txt to point directly to the correct sitemap index
- Added two missing sitemaps that Google had never been able to fetch
- Started a batch content upgrade on 1,019 thin pages using an AI API
The fixes took a few hours. The damage took about six weeks to happen.
What I Learned
Every time I optimized something without fully understanding the consequences, I made things worse.
The robots.txt changes felt reasonable at the time. Block the programmatic pages, keep Google focused on quality content.
The actual effect: I told Google my website was hiding something. Google responded accordingly.
There is a version of this mistake that every website owner makes eventually. You tweak something small. You forget about it. Three months later you are staring at a coverage report wondering why 91% of your pages disappeared.
What Happens Next
Google’s robots.txt cache takes 24 hours to clear.
Sitemap re-indexing takes days to weeks.
Full recovery — getting back to 372+ indexed pages — will probably take 2 to 3 months.
I will report back when it happens. Or when it does not.
That is the deal with documenting this honestly. You write about the mess while you are still in it.
Related: My Goji Berry Website Ran for 6 Years and Never Made Money | 3 AIs Rated My Website 7, 8, and 9
Leave a Reply