| Outage start | Thursday, April 04, 2024 10 a.m. |
| Expected end | Thursday, April 04, 2024 12:14 p.m. |
The networking issue has now been resolved
Chameleon's JupyterHub is current experiencing an unexpected outage. Operators are investigating.
| Outage start | Tuesday, April 16, 2024 9:30 a.m. |
| Expected end | Tuesday, April 16, 2024 10 a.m. |
The morning of April 16th, from 9:30-10:00am, we are scheduling a maintenance window which will affect the CHI@UC network uplink.
This is in order to debug some issues affecting the routers "upstream" from CHI@UC.
For the duration, you may encounter failures to access the CHI@UC website or API, or access to public IPs.
| Outage start | Wednesday, March 06, 2024 1:55 p.m. |
| Expected end | Thursday, March 07, 2024 1:36 p.m. |
UPDATE: The instability should be fixed, and logging into services should work as normal.
----
Some users are reporting issues authenticating to CHI@UC, CHI@TACC, and CHI@Edge dashboards. We are working on resolving this issue.
| Outage start | Wednesday, March 27, 2024 8 a.m. |
| Expected end | Wednesday, March 27, 2024 10:33 a.m. |
Resolved: Work is complete and KVM@TACC is available.
On Wednesday, March 27 2024 KVM@TACC will be unavailable between 8:00am CDT and 4:00pm CDT to perform necessary system maintenance. Access to the KVM@TACC web interface and API will be unavaiable, and network access to instances will be interrupted. Instances will continue to run, even though they will not be reachable.
| Outage start | Monday, March 11, 2024 9 a.m. |
| Expected end | Tuesday, March 12, 2024 1 p.m. |
5:00 PM We have a workaround in place that should restore connectivity to all floating IPs, however we are still investigating the root cause. There may be additonal minor interruptions while we troubleshoot, but currently all floating IPs tested respond to ping and ssh.
1:30 PM: The maintenance is done, but we are receiving reports of failure to SSH to public IPs from for some users and some instances, and are investigating. Access to the API and dashboard have been fully restored.
| Outage start | Tuesday, January 30, 2024 8:30 a.m. |
| Expected end | Wednesday, January 31, 2024 5:59 p.m. |
Update 5:45 PM Jan 31: Jupyterhub and CHI@Edge are back online.
This morning the VM cluster that hosts Portal, CHI@Edge, and JupyterHub went down. This affected the ability to create new leases at any Chameleon site, and access to the help desk.
| Outage start | Tuesday, February 06, 2024 6 a.m. |
| Expected end | Thursday, February 08, 2024 3:21 p.m. |
Update: 3pm Feb 8: Outage is resolved, all skylake and rtx_6000 nodes now have 2 10G network interfaces (up from one each)
Update: 7PM Feb 6th: The site is online, and all P3 nodes (everything except compute_skylake and gpu_rtx_6000) are accessible.
On Feb 6th, 2024, we'll be taking the CHI@UC site down for maintenance, in order to replace some failing network hardware.
| Outage start | Friday, January 26, 2024 6 a.m. |
| Expected end | Monday, January 29, 2024 6 p.m. |
CHI@NCAR will be down for maintenance from January 26th to January 29th inclusive, in order to facilitate work on the datacenter racks currently hosting it.
| Outage start | Monday, January 08, 2024 9:15 a.m. |
| Expected end | Monday, January 08, 2024 9:20 a.m. |
Update: The upgrade is complete, and Jupyter service should be as normal
---
We will be upgrading JupyterHub versions, which will take down JupyterHub and running JupyterLab instances for a short duration.
| Outage start | Thursday, December 07, 2023 12:33 p.m. |
| Expected end | Thursday, December 07, 2023 2:22 p.m. |
2:22 PM: Issues affecting Jupyterhub are resolved.
The root cause was a network interruption affecting availability of the storage services which Juptyerhub's infra depends on. After the network interruption was resolved, dependent services, including Jupyterhub, returned to a healthy state.
We are still investigating other issues affecting KVM@TACC, unrelated to this outage. If your work in Jupyter depends on KVM, you may be able to continue your work by using a different Chameleon site.