A data center maintenance on a core router affected all inter-datacenter connectivity which caused a complete failure due to an incorrectly configured redundant link. Because Bitly was mid datacenter move, no part of the application was able to tolerate the loss of inter-datacenter connectivity. Some Bitlink redirect functionality which was recovered by engaging a Disaster Recovery plan. Service was restored when the maintenance completed. We will be reconfiguring the inter-datacenter backup connection to connect to a router in a different region as a result of this outage.
Feb 18 2019 0905 UTC: Outage detected
Feb 18 2019 0913 UTC: Execution of disaster recovery
Feb 18 2019 0920 UTC: Update status page
Feb 18 2019 1035 UTC: Services restored