How to search for Open Amazon s3 Buckets and their contents — https://buckets.grayhatwarfare.com

Intro

TLTR — This is the story of why I created https://buckets.grayhatwarfare.com/ a free tool that lists open s3 buckets and helps you search for interesting files.

For an intro on what Amazon open buckets is you can read this https://blog.rapid7.com/2013/03/27/open-s3-buckets/ . In essence, many files are publicly accessible, some by design, some by incompetence of the admins. These files sometimes include very sensitive data. https://github.com/nagwww/s3-leaks has a list of the biggest leaks recorded.

Since this was exposed, many projects have been created that can enumerate s3 buckets:

All these tools/projects have some common problems:

  • The real problem (at least for me) is where to find the list to bruteforce for buckets, not actually doing the bruteforce.
  • All tools/projects only scan the first page for results.
  • thebuckhacker.com includes uninteresting files and useful results tend to be lost in the noise. Also the first 1000 results of each bucket is fairly limited.
  • The process is slow and not productive. Its not very useful for pentesters to run a tool to run for days, save the exports somewhere and then grep them whenever they want to search for something. What is better is a useful tool in front of a large database.

buckets.grayhatwarfare.com

And like that I have created http://buckets.grayhatwarfare.com/. I took ideas from the tools/projects above, but I mostly rewrote them myself, and runned them on my infrastructure. The project’s features are:

  • It is a searchable database of open buckets.
  • I have included up to million results of each bucket. (In the future might be more).
  • I removed all uninteresting (in my opinion) files like images. Most images names are auto-generated.
  • I currently have ~180.000.000 files. In I included all images that number would go up to a few billion, which is a completely different system.
  • As of today, 70 000 buckets are listed (not all of them have “interesting” files)
  • Full text search with binary logic (can search for keywords and also stopwords)
  • I include the list of the buckets.
  • The user can browse the contents of the bucket.
  • Excluded a lot of other things that are not interesting like cloudwatch logs.
  • Found a solution the problem on how to generate possible names for buckets. My process reviles some hundreds new buckets per day.
  • I have mostly automated the process.

Why create this ?

Although I consider my self a software engineer, I was always fascinated with security. I have on many occasions created multiple tools that mass scan for vulnerabilities, just to see if is possible. Although I created the tools, I never publish anything it always felt like I was wasting my time. grayhatwarfare.com is my attempt to present some of my work, even if it is anonymous.

The project is currently free and running on servers paid by me. There are some limitations in place to protect resources, but otherwise pentesters can use this on their daily tasks.

Whats to come in grayhatwarfare.com

Lots of cool things, if I have the time:

  • Subdomains pointing to expired buckets which can lead to something like this: https://www.reddit.com/r/netsec/comments/8t0pb2/how_i_hacked_applecom/
  • Huge lists of exposed version control (.git) which can expose the website’s repository (Source, password files, log files etc).
  • Exposed cameras/IOT devices.
  • Huge resources like extremely large (actual) cracked password lists.