Did Google just got hacked?!

7 min readJun 3, 2019

Are you curious? is it possible? It is a potential and there is no clear statement out as we speak (2/6/19–2 PM GMT). Nonetheless, I don’t want to do another blog that slag Google… they offer a service and a bloody good one.

Think how much collaboration you get done through Gcloud, Gdocs etc… I do use it a lot and makes a brilliant writing experience across the board…

The incident

Now yesterday 02/06/2019 my google docs started acting up. I’ve rebooted half of the device in the house (turn it on and off anyone?), but nothing did seem to be working.

Around 9.45 PM GMT, I did go and checked the services and…holy, moly half of Google was down! Now at this point in time, it started getting hard working on a document with my team.

This made me realize how dependent my organization is on a number of easy to use services. Even having a solid plan for backup, and a fairly regular test and run I realize the alternatives were not as good. Will add some more comments and my 5 suggestion on the Backup and recovery options later.

Decide how much data you want to lose
That drives the decision on how frequently you back up
Back up on a different cloud provider
Make sure you know how to access and use the other cloud provider (don’t just copy google drive documents in Dropbox)
Test the backup and recovery into google cloud once every 2–6 months depending on how risky you want to be

So what happened?

YouTube, Snapchat, Gmail, Nest, Discord, and a number of other web services suffered major outages in USA and Europe region.

The root cause is still unknown and there are rumours on too much traffic….could be a nation-state hack? Could it be a misconfiguration? Could be a Hack? All is possible for now…

After 1 or 2 hours of frustration on google docs, I kick-started the backup plan (work on other providers and with another method). It was still possible to work on the text of the documents and extract them but was a painful experience.

I felt at this point in time the pain of the Google Operation Center

Speculation bingo was open and the conversation was on fire…

And John did pick me up on the pool organizing a more serious one (DDoS and cert seem to be winning)…even if I’m sceptical about the DDoS as the capacity of Google datacenter is probably massive.

Timeline

Nonetheless, there was a serious issue across the board that affected all the services. This is the timeline just for google docs.

Statement from Google on the verge: ‘ Google’s issues started at around 3 PM ET /12 PM PT, and the company resolved them after more than four hours.’

Official statement: “high levels of network congestion in the eastern USA” for the issues. “We will conduct a post mortem and make appropriate improvements to our systems to prevent this from happening again,” says a Google spokesperson in a statement to The Verge. “We sincerely apologize to those that were impacted by today’s issues.”

Now on the facts, the service kept on working with glitches on the first hour, got really bad the second (unusable) and then annoying glitches for the other two hours…

the service was completely unusable for 2 or so hours, and very spotty for the remaining two (sometimes unusable).

I’ve invoked my Disaster Recovery plan at this point as the Business Continuity (keep on using the Google cloud with other integrated tools) was not working anymore.

I have lost some work and time since the last time I’ve backed up on another provider but not too much.

The important of the plan

Those are my 5 steps to have a pain free experience

have a plan (continuity and disaster recovery)
test the plan regularly
do backup
test the backup regularly
Go back to 1 rinse and repeat, also update with the lesson learned

When using google cloud services (drive, gdoc,…) you have two options:

use the native format — this maximizes the use of collaboration services and notifications
use native Microsoft format (.docx, .ppt) and have limited functionalities but the easier method of backup

Now is all good to have Business Continuity Plans but if you haven’t actually tested than they are pointless…

The testing part is not just performing a backup of your documents in some other cloud provider (this for drive/google docs) but also making sure the backups are regular, and that you actually have a way to use those backups.

Backing up from one cloud provider is not as easy it might sound and especially if you use google docs and drive sync is not always intuitive to extract document

If you want to backup your document in a usable form you’ll have to export in Microsoft format (.pptx or .docx) and then move to another cloud provider.

This ensures that you will be able to open the document with MS-Word regardless of where you are.

Now if you want to be even more drastic and only work with Microsoft word in google cloud (option 2 mentioned above)

Is all google fault if I couldn’t deliver that document in time!!!

No, it was not, as cloud users, we shall be a bit lawyer and go through the terms and conditions of a cloud provider.

The cloud provider ensures that it will operate within certain Service Level Agreement in their Term and conditions (those lengthy legalese documents that no one reads)…

https://gsuite.google.com/intl/en/terms/sla.html

Google simply will rebound you if those Service Level are breached…nonetheless you have to reflect (as an individual or more important as a business) how much downtime of service will affect you.

The Business Continuity and Disaster Recovery plan ensure exactly this. As users of a cloud provider we are responsible for putting one in place (that might account for the cloud provider one), test verify and use it at least once per year.

this leads to another one of my pet peeves: the cloud responsibility model

With Google Gclud (docs, Gmail etc…) we are in a SaaS case. As a user of the cloud we are responsible for:

Controlling the users (granting and revoking access)
Content what goes in the provider (harmful material and backup).
Overall usage (not abusing the term and conditions).

There is no way out of those elements. you can delegate the responsibility of some of those but never the accountability as you, the customer, your revenues your brand will get affected.

For a more comprehensive overview of the division of responsibilities see below an extract of one of my upcoming talk: is the cloud secure? (you can find the others here NSC42 Press)

Conclusions

To recap what you should do to prevent those harmful events to affect you in the future?

Understand the service credits
Understand the term and conditions of the cloud providers
Have an exit strategy (how to get out of a cloud provider)
Have tested disaster recovery plans and business continuity plan.

Easy steps to have a piece of mind

Decide how much data you want to lose
That drives the decision on how frequently you back up
Back up on a different cloud provider
Make sure you know how to access and use the other cloud provider (don’t just copy google drive documents in Dropbox)
Test the backup and recovery into google cloud once every 2–6 months depending on how risky you want to be

Rinse and repeat :) and enjoy a happy cloud experience