In the week after its astounding deletion of Australian pension fund UniSuper's entire account, you might think Google Cloud would be on its very best behavior.
Nope.
At 15:22 last Friday, US Pacific Time, Google Cloud ran "maintenance automation intended to shutdown an unused network control component in a single location." Which is fair enough.
It worked! Unfortunately, it also worked in other locations - about 40 of them, by The Register's count.
The result was two hours and forty-eight minutes during which users of 33 Google Cloud Services - including biggies like the Compute Engine and Kubernetes Engine - experienced the following symptoms:
Other services that needed VMs in Google Cloud Engine or network configuration updates "were not able to successfully complete operations during this time."
The incident was over by 18:10 Pacific Time last Friday.
The Register can only imagine how US-based customers felt as they wound down for the weekend only to see their cloudy connectivity evaporate.
Google has told customers that the incident was caused by a bug in the automation it used to shut down networks, and that once the flawed component was restarted the problem went away.
The automation tool has been sin-binned "until appropriate safeguards are put in place."
And Google has told clients "There is no risk of a recurrence of this outage at the moment." Which isn't very reassuring, given the org's recent rotten track record.
The search giant's cloud limb has promised to reveal more info about the mess. If there's anything juicy in it, we'll let you know. ®
https://www.theregister.com//2024/05/20/google_cloud_network_outage/
Created by Tan KW | Jul 28, 2024
Created by Tan KW | Jul 28, 2024
Created by Tan KW | Jul 27, 2024