22:27 UTC until
23:17 UTC on Oct 27, 2021 some API requests to create new links failed with a
HTTP 500 or a
HTTP 503 response. API Clients that implemented retry may have succeeded on subsequent attempts.
Root Cause: A new database ACL change was released which contained an invalid permission setting. This permission setting took priority over existing settings and blocked new database connections causing a progressive increase in application failures as applications opened new connections. The invalid configuration was identified and removed.
We understand the availability to create links is important to our customers and have planned work to prevent this impact in the future by improving how changes like this are processed and validated and by improving our ability to quickly rollback database ACL changes.