The majority of CHI@Edge devices are currently offline due to a network outage at UChicago. Devices not at UChicago are still working, but there are only a small number of those at the moment.
11/11/2026: We are working with campus IT on a resolution, but still do not have an ETA
ResolvedPosted by Bodhi Rubinstein on October 16, 2026
Outage start
Friday, October 16, 2026 2:35 p.m.
Expected end
Monday, October 19, 2026 8:20 p.m.
RESOLVED: All compute_zen5 and tharm nodes at CHI@NCAR are back in production. We are waiting for a replacement SSD for our switch, but have moved the primary links for the affected nodes to a different switch. The secondary links will be unavailable until we receive the replacement hardware. All existing instances on the affected nodes have been migrated and should be accessible. The impacted leases have been extended to account for the unplanned downtime. Please let us know if you experience any issues or have any questions at help@chameleoncloud.org. Thank you for your patience!
On August 25, Chameleon's portal will be upgraded. During the window, the service will be stopped and restarted. We expect total downtime to be less than 15 minutes.
Users will experience degraded ability to create/update leases at any site, access project management pages, and access to the help desk (except via email). No running experiments will be impacted.
ResolvedPosted by Michael Sherman on August 12, 2026
Outage start
Monday, August 17, 2026 9 a.m.
Expected end
Tuesday, August 18, 2026 5:56 p.m.
On Monday, August 17th, CHI@UC will have a maintenance window to upgrade remaining services that could not be upgraded in the prior window.
This will be visible as intermittent disruption to services and networking for several hours. Although we are holding a 9-5 window, we expect the work to take only 1-2 hours in the morning.
Please get in touch if this will cause issues for your experiments.
ResolvedPosted by Michael Sherman on August 03, 2026
Outage start
Tuesday, July 28, 2026 6 p.m.
Expected end
Thursday, July 30, 2026 6 p.m.
We observed instability in the ironic provisioning service at CHI@UC following the openstack version upgrade on Tuesday Jul 28th, lasting until early on Thursday the 30th.
Users may have observed instances moving to an "Error" state instead of provisioning successfully, failures to access the serial console, and a brief window where nodes were marked as unavailable for reservation.
ResolvedPosted by Michael Sherman on July 20, 2026
Outage start
Tuesday, July 28, 2026 9 a.m.
Expected end
Tuesday, July 28, 2026 7 p.m.
On Tuesday, July 28th, we're planning downtime at CHI@UC in order to perform an openstack version upgrade and maintenance on the controller nodes. From 9am-5pm central time on the 28th, new instances won't be able to be launched, and existing ones won't be accessible via the network.
If this will cause issues for you, please get in touch with us at help@chameleoncloud.org