On 6-Aug-2019 at 1:21 PM EDT, two 12-strand single-mode MPO connections within the ITS Data Centre, and between the pairs of redundant switching and routing components, became dislodged during routine maintenance, causing what is known as a “split-brain” condition in the Campus Core Switching layer that delayed and impaired identification and resolution of the issue. The problem was resolved by re-seating the MPO connectors at 1:54 PM EDT.
Normally, these connectors are quite strong and do not simply pop loose with minor contact, as happened in this case while ITS Data Centre staff were working in close proximity. Staff continue to inspect neighboring connections of the same type for signs of susceptibility and will escalate the issue with the manufacturer.
We apologize for the inconvenience and appreciate your patience.