The cause of the incident was determined to be a subtle coding bug that was particularly difficult to pinpoint. Technically, 4 lines of code, each had 4 spaces too many, out of nearly 100,000 lines total. Measures now in place to prevent recurrence include a regime of brutal pressure-testing of simulation nodes prior to deployment.
Preventative measures to prevent recurrence: