Internet Service Issue
Incident Report for ITS System Status
Postmortem

On 6-Aug-2019 at 1:21 PM EDT, two 12-strand single-mode MPO connections within the ITS Data Centre, and between the pairs of redundant switching and routing components, became dislodged during routine maintenance, causing what is known as a “split-brain” condition in the Campus Core Switching layer that delayed and impaired identification and resolution of the issue. The problem was resolved by re-seating the MPO connectors at 1:54 PM EDT.

Normally, these connectors are quite strong and do not simply pop loose with minor contact, as happened in this case while ITS Data Centre staff were working in close proximity. Staff continue to inspect neighboring connections of the same type for signs of susceptibility and will escalate the issue with the manufacturer.

We apologize for the inconvenience and appreciate your patience.

Posted Aug 06, 2019 - 14:26 EDT

Resolved
The issue has been resolved and post-mortem analysis is ongoing.
Posted Aug 06, 2019 - 14:00 EDT
Investigating
We are investigating reports of some users having Internet connectivity issues.
Please be assured that we are aware of the problem and we have technical staff looking into it.
Posted Aug 06, 2019 - 13:34 EDT
This incident affected: Enterprise Networks (Campus Backbone Network).