Instance Provisioning failures at UC

Resolved Posted by Michael Sherman on May 11, 2022
Outage start Wednesday, May 11, 2022 8 a.m.
Expected end Wednesday, May 11, 2022 4:13 p.m.

Update: 4PM 05/11/2022: This issue is now resolved, provisioning and connectivity should be restored for all UC nodes.

For those who want the details, this issue was triggered by a (suspected) switch operating system bug. The switches had Per-VLAN Spanning-Tree enabled, and we provision isolated networks by configuring a range of vlans on each top of rack switch. However, once a certain threshold # of vlans is configured, the switch ceases to create more spanning-tree daemons. The observed behavior is that newly active ports no longer pass traffic, but already provisioned ports show no issues. 

Forcing the switches to use a single spanning-tree daemon (RSTP mode), restored connectivity on affected ports.


An issue is affecting the provisioning of new instances on P3 nodes at UC. Existing nodes are unaffected.