Reported Outages

Update: The upgrade is complete, and Jupyter service should be as normal

---

We will be upgrading JupyterHub versions, which will take down JupyterHub and running JupyterLab instances for a short duration.

2:22 PM: Issues affecting Jupyterhub are resolved.

The root cause was a network interruption affecting availability of the storage services which Juptyerhub's infra depends on. After the network interruption was resolved, dependent services, including Jupyterhub, returned to a healthy state.

We are still investigating other issues affecting KVM@TACC, unrelated to this outage. If your work in Jupyter depends on KVM, you may be able to continue your work by using a different Chameleon site.

Dec 13: The outage is now resolved.

You should observe that I/O performance on ephemeral disks is back to normal, as they have been migrated to a different storage backend.
Note: If you attach an additonal cinder volume of type "ceph-hdd", that volume may experience unexpectedly delayed I/O as the system state settles, but this will not impact the majority of instances.

We're observing intermittent issues accessing Chameleon's Jupyterhub. There was a failure in one of the VMs underlying it, and in combination, some requests are still being routed to the failed instance in error.

In some cases, users may be able to work around this error by refreshing their browser.

Site staff are investigating to revive both the VM, and to correct the request routing.

The outage is now resolved.

Due to a combination of factors, a misconfigured node was able to cause a DoS of control plane services. The relevant traffic has been blocked, and the site is back online.

Chameleon's JupyterHub will be taken down Sunday, November 5 at 12:00 PM CT to perform an urgent infrastructure migration.

----

UPDATE: The upgrade is now complete. If you encounter any issues with Jupyter or Trovi, please let us know via the help desk . You may experience an unresponsive host while DNS updates propagate.

Resolved The switch has been restored, and running instances have been reconnected.

The network switch connecting the nodes equiped with NVIDIA M40, K80, P100, and V100 GPUs has failed. A replacement switch is expected to be installed on Oct 30, 2023. 

Existing leases for the affected nodes will be extended to prevent instances from being shut down.

This maintenance is now completed.

On the morning of Monday, October 23rd, there will be a brief outage affecting login to all Chameleon sites and services.

We expect a 5-10 minute outage while we apply updates to the service that handles federated login.
This won't affect any running workloads or nodes, but you may need to refresh your browser once the outage ends.

Resolved: the work is complete.

Starting at 9/20/2023 the Haswell+InfiniBand nodes have been unavailable while they are reconfigured in order to decommission a portion of them. This work is taking longer than anticipated to complete, but is expected to conclude by 10/06/2023.

We thank you for your patience.

Due to a water leak in the StarLight datacenter, power has been cut to the CHI@NU racks.

We're monitoring the situation, and will update when we have more information on an estimated time to resolution.