Major Service Outage
Incident Report for Bitly
Postmortem

Between 23:13 Monday and 01:14 Tuesday UTC, a networking provider that maintains a high bandwidth connection between our two primary data centers suffered a complete outage that resulted in some of Bitly’s production systems being unable to talk to one another and service requests. While we maintain multiple connections for redundancy, all connections flowed through a facility that suffered a widespread power failure.

During this time, bit.ly redirects and some Custom Domain redirects were unaffected. However, the website and API were down and Custom Domain redirects were seriously degraded. Metrics collection continued to function throughout and no data was lost.

We apologize for the inconvenience this outage has caused and understand the impact to our customers. Our engineering team is accelerating existing plans to remove our production dependency on this data center network connection in order to prevent future outages of this type.

Posted about 2 months ago. Jun 20, 2019 - 17:17 EDT

Resolved
Good news: We've completed testing our provider's networking issue and we're confident that they've implemented a fix. If you were impacted, things should now be resolved. If you’re still experiencing issues, please email support@bit.ly so we can look into the root cause of your problem.
Posted 2 months ago. Jun 17, 2019 - 21:43 EDT
Monitoring
Good news, community: if you were impacted by the earlier datacenter connectivity issue, you should now be able to access our services without any problems. Thanks for bearing with us! For more info on this incident please see our status page: status.bitly.com
Posted 2 months ago. Jun 17, 2019 - 21:16 EDT
Update
Update: The datacenter networking outage has been identified and we are continuing to work with our vendor to get service restored. Thanks so much for your patience!
Posted 2 months ago. Jun 17, 2019 - 20:32 EDT
Update
Hi all. Bit.ly redirects and custom domain redirects are beginning to recover. Our team is hard at work with our datacenter provider to get everyone up and running asap. We'll continue to update you in the meantime.
Posted 2 months ago. Jun 17, 2019 - 19:53 EDT
Identified
Update: We are working with our datacenter provider to resolve a networking issue. Thanks for hanging in there!
Posted 2 months ago. Jun 17, 2019 - 19:33 EDT
Investigating
Hi community. We’re currently investigating reports of issues with the website, API, and redirects impacting much of our user base. We apologize if this is causing trouble for you. We’ll be sure to update you as soon as we know more!
Posted 2 months ago. Jun 17, 2019 - 19:21 EDT
This incident affected: Bit.ly Redirects, Custom Domain Redirects, Website (Bitly.com), API, and Metrics.