Processed Emails
Incident Report for SMTP2GO
Postmortem

The cause of the incident was determined to be a subtle coding bug that was particularly difficult to pinpoint. Technically, 4 lines of code, each had 4 spaces too many, out of nearly 100,000 lines total. Measures now in place to prevent recurrence include a regime of brutal pressure-testing of simulation nodes prior to deployment.

Preventative measures to prevent recurrence:

  1. Setting up a better staging cluster, used solely for pre-deployment testing, including heavy pressure tests, with formal acceptance criteria.
  2. Nothing to go out to the field unless it has passed the stringent staging tests.
  3. Better scheduling and staggering of code releases.
Posted Jun 30, 2022 - 02:46 UTC

Resolved
This incident has been resolved. The underlying cause was determined to be a few lines of code which each had a few too many spaces, which ended up causing an obscure issue that was very difficult to pinpoint. We are working to put measures in place to change our testing procedures to avoid such an event from happening again. More information will follow in a Postmortem within the next few days.
Posted Jun 24, 2022 - 00:46 UTC
Update
A fix is now being pushed to live servers.
Posted Jun 23, 2022 - 22:44 UTC
Identified
We apologize for the large delay resolving this issue. A fix is being implemented and is currently being tested before being pushed to our live servers. We'll update with further information as soon as possible.
Posted Jun 23, 2022 - 21:38 UTC
Investigating
We are currently investigating this issue.
Posted Jun 23, 2022 - 14:24 UTC
Identified
Some users may experience delays. The issue has been identified and a fix is being implemented.

We are very sorry for the inconvenience this causes.
Posted Jun 23, 2022 - 12:40 UTC
This incident affected: Email sending (SMTP service).