Network Switch failure at UC

Resolved Posted by Michael Sherman on March 01, 2022
Outage start Tuesday, March 01, 2022 4:04 p.m.
Expected end Sunday, May 01, 2022 4:04 p.m.

Update: Connectivity has been restored. Root cause was a software bug preventing the creation of a PVST instance on the switch, due to a large number of configured vlans. Using a single instance for all VLANs restored functionality.

Update: P2 connectivity has been restored as of 5:30pm 03/01/2022. Nodes P3-CPU-020 to P3-CPU-038 are still not provisionable.

The 1g switch serving out-of-band access for nodes in rack BG-41 has encountered a (so far) unrecoverable software error, preventing traffic to the out of band interface on nodes P3-CPU-020 to P3-CPU-038. We do not have an ETA for resolution at this time.

In addition, there has been a brief interruption to out of band ports for all phase 2 nodes.

Users will not be able to provision new instances on nodes P3-CPU-020 to P3-CPU-038, but running instances are unaffected.