Degraded API impacting some link creation
Incident Report for Bitly
Postmortem

From 22:27 UTC until 23:17 UTC on Oct 27, 2021 some API requests to create new links failed with a HTTP 500 or a HTTP 503 response. API Clients that implemented retry may have succeeded on subsequent attempts.

Root Cause: A new database ACL change was released which contained an invalid permission setting. This permission setting took priority over existing settings and blocked new database connections causing a progressive increase in application failures as applications opened new connections. The invalid configuration was identified and removed.

We understand the availability to create links is important to our customers and have planned work to prevent this impact in the future by improving how changes like this are processed and validated and by improving our ability to quickly rollback database ACL changes.

Posted Oct 29, 2021 - 09:09 EDT

Resolved
The incident has been resolved.
Posted Oct 27, 2021 - 19:21 EDT
Investigating
We are currently investigating this issue.
Posted Oct 27, 2021 - 18:47 EDT
This incident affected: API.