At approximately 2:30 PM ET on October 29, alerts began to be received by ClearlyIP's Network Operations team indicating problems accessing the database cluster used by Clearly Cloud for most US-based customers. This was interfering with call routing for a portion of customers and prevented access to Call Panel and other system components. The team immediately began investigating and observed the problem related to a protective mechanism which had detected large amounts of database request traffic and proactively restarted services to prevent data loss or corruption. The source of the requests was quickly identified, stopped, and determined to be legitimate. The team then checked each server in the cluster before determining it was safe to restore normal service by approximately 2:45 PM ET.
A postmortem analysis on the source of the database traffic identified it to be related to delivering an unusual backlog of SMS and email messages from the platform. Temporary measures were taken to prevent a recurrence of the same issue while long-term improvements are scheduled.