What unholy thing did they do that broke it across 12 different datacenters, good lord.
This is OT, but I have a droplet on DO and I'm amazed at the amount of malicious traffic it gets. Is it normal for a very private vps to receive thousands of ssh attempts per hour? I have fail2ban installed and the jail is so busy it's quite astounding. Anyone with more web hosting experience that can weigh in?
Not sure why the previous incident page got flagged. This is the new one.
It's affecting us for real. Making almost our whole service - serpapi.com - down. As we are storing database files on block storage volumes.
Isn't Digital Ocean running Ceph for their block storage?
I would wonder - as others suggested - that they may have stretched the cluster across datacenters ?!
Would be interested in the post-mortem.
Thank you Digital Ocean for once again proving that 'The Cloud' is not a backup.
This is your weekly reminder that anything you want to be reasonably “HA” should span multiple vendors in multiple DCs.
Anyone have a review of using DO k8s or DO managed DB in production?
DigitalOcean just posted a post-mortem on http://status.digitalocean.com/incidents/g76kgjxqrzxs
(the same url)
Higher latency (per status) is not end of world especially if it’s just “may experience” higher latency.
Hrm, Atlassian BitBucket is also down. Just a coincidence? Does BB use DO?
https://bitbucket.status.atlassian.com/incidents/4t1pkwrdtl8...
Their block storage is such a failure. I’m back and forth with support to automatically delete files with lifecycles for over 2 months now and it’s still not resolved.
It looks like they have just updated it as resolved and monitoring.
I was always wondering how I can get know proactively if something like this break or some service has an outage. As a result, I have built this tool( http://incidentok.com )
So that’s why bot attacks and spam traffic was lower.
This is really down for more than 2 hours!!!
DigitalOCean bad experience
Their ad was “you’ve been developing like a beast and your app is ready to go live”
DO is a nice thing to play around with and maybe launch something, but I wouldn’t run full production on it.
Personally, if DO don’t have anything new in a status post, I’d prefer seeing an update that says something like “We are continuing to work on the issue. Nothing new to report. Next update in X minutes.” That is a lot easier for me to parse than the text that someone seems to be copy/pasting in each update.