Reported Outages

At 9:00AM on Thursday, February 19, 2026, the H100 GPU hosts for KVM@TACC will be powered off for approximately one hour in order to perform urgent maintenance on the liquid cooling system to the rack. The VMs on these nodes will be shut down prior to poweroff and will be restarted when the maintenace is complete.

At 1:07 pm while Chameleon at TACC wass undergoing maintenance to datacenter uplinks, intermittent and brief interruptions were experienced during the transition.

We apologize for any inconvenience.

As of 3:15PM, the upstream issue, and the outage has resolved.

As of 2PM central time, auth.chameleoncloud.org became unreachable, with users seeing a cloudflare message about error code 522.
This is preventing sign-in to all chameleon sites, portal, and jupyter, although active sessions may continue to work until token expiry.

The outage is caused by an issue in our upstream provider's load-balancer, they are actively working on a resolution. We will update once we know more.

The CHI@Edge maintenance has now concluded. All smoke tests are passing, and we're observing significant performance improvements for floating IP connectivity and the reliability of container launches.

Update: Work is complete.

Update: We are extending the maintenance window to accommodate staff availability. Work is expected to complete by end of day.

On November 10, 2025 the Chameleon site at TACC will be transitioning cluster connectivity to new switch hardware. During a window of approximately one hour, interruptions of less than a minute will affect traffic to services and instances at CHI@TACC, KVM@TACC, and CHI@Edge as individual links are moved.

Update 10/20/2025:

The remaining maintenance for composable hardware has now been completed.

Update 10/02/2025:

KVM@TACC and CHI@Edge are now back to normal.
CHI@TACC is back to normal, exept for the following:

• A subset of baremetal nodes that lost network configuration
• Composable hardware (Liqid and GigaIO) cannot reach their attached GPUs, pending a restart of these nodes and their shared PCIe backplane.

Thank you for your patience as we restore these!
 

Chameleon Jupyterhub is unavailable.

As of the morning of September 16th, the switch has had another issue, and is no longer passing traffic. Staff are investigating, but we do not have an ETA yet.

This is now resolved.

For those interested in the details, see https://github.com/projectcalico/calico/issues/9622

CHI@Edge containers are currently failing to send any network traffic. This is due to an unanticipated edge case during routine maintenance breaking breaking updates to the calico network database.

On the morning of July 22, we'll be upgrading the infrastructure hosting trovi. During this time, you will not be able to upload, edit, or launch trovi artifacts. Running Jupyter servers will not be affected.