Between 23:13 Monday and 01:14 Tuesday UTC, a networking provider that maintains a high bandwidth connection between our two primary data centers suffered a complete outage that resulted in some of Bitly’s production systems being unable to talk to one another and service requests. While we maintain multiple connections for redundancy, all connections flowed through a facility that suffered a widespread power failure.

During this time, bit.ly redirects and some Custom Domain redirects were unaffected. However, the website and API were down and Custom Domain redirects were seriously degraded. Metrics collection continued to function throughout and no data was lost.

We apologize for the inconvenience this outage has caused and understand the impact to our customers. Our engineering team is accelerating existing plans to remove our production dependency on this data center network connection in order to prevent future outages of this type.