How to make a bulk deindex request to Google Webmaster Tool
What is it?
The Google Bulk De-indexer is a tool allowing you to submit one or more URLs to be de-indexed via Google Webmaster Tool in a single request.
Even though Google Webmaster Tool is powerful, it does not support any way to de-index multiple URLs in a single operation. I needed a way to de-indexing many URLs without having to click through GWT’s UI, one at a time.
What you need to get this up and running.
- Ruby (MRI > 1.9.X or JRuby > 1.7.X)
$ git clone https://github.com/inhang/google-bulk-deindexer.git
$ cd google-bulk-deindexer
$ [sudo] gem install bundler
$ [sudo] bundle install
NOTE: If you’re using RVM/RBenv, you don’t need sudo. Otherwise, you probably do need it.
To use this tool with GWT, you need to be able to login to GWT with admin rights to your sites.
You need to add config.json before you can run. For details, you can refer to https://github.com/inhang/google-bulk-deindexer/blob/master/README.md#pre-requisite
How do you use it?
From the command-line
$ ruby deindexer_cmdline.rb file-list-of-urls.txt
If you are not comfortable with the command-line, there is a little frontend UI you can use as well.
You can go to http://localhost:9292 to see the UI.
You can input URLs to deindex, one URL per line.
When you click submit, the job will be initiated in the background, and a notification will be sent to those you defined in config.json “mailto” email list along with the status.
Good thing about using the UI is that you can have it running in a remote server, submit a request, close your device, and go about your business. The request will be queued, executed, and you’ll be notified.
Why Firefox? Why Watir/WebDriver?
I wanted to make the tool as unobtrusive as possible to Google Webmaster Tool, mimicking what a human might perform manually. The tool should not abuse Google Webmaster Tool.
For more detail, you can refer to the project’s github page at https://github.com/inhang/google-bulk-deindexer