Reported Outages

We're observing intermittent provisioning failures for a small set of nodes at CHI@UC, all of which are "phase 2" nodes, mostly rtx_6000s.

The nodes currently known to be affected are listed below, and have been placed into a non-reservable maintenance mode until we can resolve the issue.

Resolved outage

At CHI@UC, when submitting a request for a new lease, users are intermittently receiving an error message about "enforcement failed". After restarting a relevant service, lease requests are now succeeding again.

We suspect this issue is related to a service token expiring and not being correctly renewed, and are investigating a proper fix.

On the morning of Tuesday, July 23rd, there will be a brief outage affecting login to all Chameleon sites and services at 9:30 AM central time.

We expect a 5-10 minute outage while we apply updates to the service that handles federated login.
This won't affect any running workloads or nodes, but you may need to refresh your browser once the outage ends.

As of 1pm, July 23rd, reservation enformenents are fixed, and we're declaring the CHI@UC outage over.
You may notice some minor changes to the horizon dashboard as we fix some remaining UI issues in the instance launch dialog, but these should not affect any functionality.

UPDATE: All services are updated and should be working as expected

On the morning of Tuesday, June 18th, there will be a brief outage affecting login to all Chameleon sites and services at 9:30 AM central time.

We expect a 5-10 minute outage while we apply updates to the service that handles federated login.
This won't affect any running workloads or nodes, but you may need to refresh your browser once the outage ends.

CHI@UC had a brief interruption in connectivity, preventing access to chi.uc.chameleoncloud.org. This issue manifested as a failure for the UC control-plane servers to contact the chameleon authentication server, triggered by what was planned to be unrelated work elsewhere in the network.

Upon discovering the issue, the other network changes were backed out, and service has been restored.

CHI@TACC was unavailable from May 25 through the morning of May 28, 2024. This effected the API services and web interface, but did not impact running instances.

We have corrected the problem, and service is restored.

UPDATE: All services are updated and should be working as expected

On the morning of Tuesday, June 4th, there will be a brief outage affecting login to all Chameleon sites and services at 9:30 AM central time.

We expect a 5-10 minute outage while we apply updates to the service that handles federated login.
This won't affect any running workloads or nodes, but you may need to refresh your browser once the outage ends.

CHI@TACC was unavailble between 7:15PM CDT Monday May 20, 2024 and 9:40AM CDT Tuesday May 21, 2024 due to an automation failure issuing an updated SSL certificate. The issue has been corrected and we do not expect further disruption.