Migrating to Elasticsearch 5.4.0 on Heroku using Elastic Cloud: [indices:admin/delete] requires authentication.

I learned the hard way how to setup access authentication for a protected cluster after upgrading to Elasticsearch 5.4.0 on Heroku’s Elastic Cloud service. Our previous instance did not require Shield access authentication.

After porting the changes and pushing the upgrade, I hit the following error when rebuilding indices:

Elasticsearch::Transport::Transport::Errors::Unauthorized: [401] {“error”:{“root_cause”:[{“type”:”security_exception”,”reason”:”action [indices:admin/delete] requires authentication”,”header”:{“WWW-Authenticate”:”Basic realm=\”security\” charset=\”UTF-8\””}}],”type”:”security_exception”,”reason”:”action [indices:admin/delete] requires authentication”,”header”:{“WWW-Authenticate”:”Basic realm=\”security\” charset=\”UTF-8\””}},”status”:401}

To setup access authentication using a transport service:

(1) Install and configure the `elasticsearch-transport` gem (or alternative for non-Rails projects)

gem 'elasticsearch-transport', '~> 5.0.4'

(2) Push changes to Heroku server

$ git push heroku <app-name> <branch-name>

(3) Update your version on Elastic Cloud. This is accessible from the Heroku Dashboard add-on section.

Version: 5.4.0

Note: Shield is enabled by default. Save the auto-generated password that is provided for the `elastic` user.

(4) Using `elastic` credentials, log into Kibana and create an additional superuser. This is accessible from the Elastic Cloud dashboard, click on Kibana and then the management tab and elasticsearch security.

# Elasticsearch security via Kibana management tab
user: <user>
pass: <password>
role: superuser

(5) Set the access credentials on the config variable `FOUNDELASTICSEARCH_URL` like:

https://username:password@cluster_id.region.aws.found.io

This is accessible from your Heroku dashboard settings tab.

(6) Run rake commands to rebuild indices on Heroku environment.

$ heroku run rake elasticsearch:import:all

You should be good to go now.