Reported Outages

CHI@UC Issues mounting shared filesystem

Resolved Posted by Michael Sherman on June 12, 2023
Outage start Monday, June 12, 2023 6 a.m.
Expected end Tuesday, June 13, 2023 12 a.m.

We're currently observing an issue where some projects fail to mount the filesystem at CHI@UC, receiving an error message about an "RBAC Quota". We are currently investigating.

This is purely an access issue with new "storage leases", existing storage leases/mounts should be unaffected.

CHI@IIT down

Resolved Posted by Michael Sherman on June 12, 2023
Outage start Saturday, June 03, 2023 6 p.m.
Expected end Tuesday, June 13, 2023 6 p.m.

CHI@IIT is currently down, we've had a persistent hardware issue with the controller node at this side.

Site staff are currently investigating.

CHI@NU down

Resolved Posted by Michael Sherman on May 30, 2023
Outage start Tuesday, May 30, 2023 2 a.m.
Expected end Wednesday, June 07, 2023 11:36 a.m.

CHI@NU was down due to a hardware failure in the controller node. This has now been replaced, and the site is back online.

Datacenter Maintenance affecting CHI@UC May 7-12!

Resolved Posted by Michael Sherman on April 19, 2023
Outage start Sunday, May 07, 2023 5 p.m.
Expected end Friday, May 12, 2023 5:55 p.m.

05/12/23: CHI@UC is back online. All systems look stable, but we'll be keeping an eye on it over the weekend to be sure. 

Pleas let us know if you encounter issues!

Authentication system outage

Resolved Posted by Michael Sherman on April 03, 2023
Outage start Monday, April 03, 2023 10:06 a.m.
Expected end Monday, April 03, 2023 11:01 a.m.

Update 11:01am: The upstream issue seems to have resolved, and sites are again accessible. We'll be monitoring the situation to see if it remains stable.

Authentication to all sites is currently down due to a failure in our central authentication system.

Existing instances should be unaffected, but users won’t be able to create or modify existing instances.

This seems to be caused by a failure in our upstream DNS provider, we are working with them to get an ETA for resolution.

TACC Network maintenance 26 March 2023

Resolved Posted by Cody Hammock on March 23, 2023
Outage start Sunday, March 26, 2023 8 a.m.
Expected end Sunday, March 26, 2023 1:12 p.m.

Update: Maintenance was completed a 1:12 PM (CDT) yesterday Sunday, 26 March 2023.

Network maintenance will be carried out between 8:00 AM and 2:00 PM (CDT) on Sunday, 26 March 2023. Access to all TACC systems will be unavailable during this time, including CHI@TACC, KVM@TACC, and the Chameleon Portal. Instances will continue to run, but users will have no access to TACC services and systems until the upgrade is complete.

Please submit any questions you may have via the TACC User Portal.

CHI@EVL maintenance window April 3rd

Resolved Posted by Michael Sherman on March 22, 2023
Outage start Monday, April 03, 2023 11 a.m.
Expected end Wednesday, April 05, 2023 4 p.m.

On April 3rd, CHI@EVL will be down while we replace the controller node. Conservatively, this should take about 4 hours before services are restored. Running instances and leases won't be modified, but will not be accessible during the outage window.

temporary outage for object store at CHI@UC

Resolved Posted by Michael Sherman on March 15, 2023
Outage start Thursday, March 16, 2023 1 p.m.
Expected end Thursday, March 16, 2023 1:30 p.m.

Update 5PM CT: Object store is back online via a workaround.
There will be a blip tomorrow at 1PM so we can test a permanent fix for this issue that triggered this outage.

CHI@IIT currently down

Resolved Posted by Michael Sherman on January 27, 2023
Outage start Friday, January 27, 2023 5:01 p.m.
Expected end Monday, January 30, 2023 5:14 p.m.

CHI@IIT is back up, but we're still waiting for the arrival of replacement hardware. Currently, bringing the site back online requires in-person actions, and so you'll observe instability until said hardware is installed. We plan for this work to be completed by the end of this week, subject to parts availability.