Google dropping pages from search index
While transitioning sites from serving content from both HTTP and HTTPS to a setup where the HTTP-site redirects the traffic to HTTPS (TLS/SSL) to prepare for HTTP Strict Transport Security (HSTS) I noticed that the Google crawled and reindexed the changes really slow or not at all. The reports in the Google Search Console were lagging behind it’s normal that the delay can be up to a couple of days at times. But in this case the delay in the reports were at times more than a week.
The delay in the reports made it difficult to know if Google had started to reindex the pages on the site as expected or if there was an error on our part. Since we weren’t able to confirm that the changes worked as we expected in the Google Search Console we started to analyze and verify the changes. We could not find any errors in our configurations and we were uncertain about what to do since we couldn’t find the reason for the changes not being processed.
A couple of days after the changes to the site configurations the article Google Is Dropping Pages Out Of The Search Index; A Bug? popped up. That was probably the culprit due to the fact that Google had a pretty hefty backlog of indexing to do. So the reindexing of our pages were delayed. After a few days I noticed Google Still Working On Reindexing Pages That Dropped Out Of The Index From Thursday.
On the 11 of April we detected an increase of Googlebot traffic and later on the same day Google acknowledged that the issue was resolved Google De-indexing Bug Fully Fixed. The delay in the reports in the Google Search Console also confirmed that the issue was being resolved as the reports started to catch up.
Things do go wrong sometimes. But it can be hard to keep track of all changes and even harder to find out that there is an ongoing issue when working in a small team. It would have been beneficial to us if Google would have notified the users within Google Search Console that there was an ongoing issue and that they were aware of it and were working on resolving it.