Reported Outages

Update June 10th: The datacenter maintenance has concluded, and CHI@NCAR is back online.

Tonight, from 6pm - 10pm US Central time, network maintenance at ANL will cause rolling outages impacting access to CHI@UC, Trovi, and to Authentication services for all Chameleon sites.

We expect the interruptions to be in the range of minutes to <= 1 hour, sometime during this window.

Connections (e.g. over ssh) to instances that are already running at other sites will not be affected, but you may observe interuptions in the ability to launch new instances due to the authentication outage.

Resolved: 7:00 pm Monday:

Reservations are workign again, and tests to launch a container and access it via floating IP are succeeding, implying that the network underlay is also healthy once again.

Please let us know if you continue to observe issues.

Update: 5:30pm: services appear to be reachable again, we saw a downtime of ~ 10 minutes. Our aplogies for the interruption.

We're waiting for more information about whether to excpect any more interruptions.

9:00 AM 04/25/25: A fix has been deployed and we belive the issue to be resolved,  but please reach out if you observe further issues.

Currently, authenticating from jupyter to KVM@TACC, or using "CLI" auth with federated ID credentials to KVM@TACC is failing.

Update: All work associated with this outage is complete.

On April 29th, CHI@Edge will be inaccessible as we migrate the control-plane to more robust hosting and upgrade several dependent services. During the maintenance window, the dashboard, running containers, and device enrollement will be unavailable.

Resolved

To prepare for upgrades to KVM@TACC, we are scheduling maintenance starting at 9:00 AM CDT on Thursday April 10th.
During this time, Staff will Stop All Running Instances, perform upgrades, then restart those instances.
This work will ensure that VMs are compatible with following upgrades, preventing further downtime.

We're observing issues with leases for some devices in CHI@edge, and have identified some configuration inconistency on the backend that may be contributing. While debugging, there was a brief outage in the reservation api service, which manifested as a failure to load the "leases" dialog.

Thing seem to be back, but we're still investigating.

11AM March 5th:

This outage is now resolved, CHI@UC appears to be working as normal.

We don't yet have a root cause from our provider, but are working with them to prevent a repeat.