Tickets of the Year: 2025

It's the end of the year, so that means it's time for our annual Tickets of the Year blog! This blog covers some of our most common help desk tickets and provides solutions, so it can hopefully be a helpful reference if you run into any of these problems. If you need more help, the best places to look would be the docs, the FAQ, or submitting a new ticket to the help desk.

If you're curious about the most common tickets we've encountered in the past, you can check out our Tickets of the Year blogs from 2024, 2023, 2022, 2021, and 2020.

Tickets of the Year!

Ticket: You've successfully reserved and launched an instance with A100 GPUs, but when you try to run your CUDA code, you get an error saying "no CUDA-capable device is detected"
Solution: This is usually a driver or environment issue. First, verify that the GPU is actually visible to your instance by running nvidia-smi. If this command isn't found or shows no GPUs, you may need to install the NVIDIA drivers. For most Chameleon images, you can install drivers using our convenience script: sudo /usr/local/bin/install_nvidia_drivers.sh. After installation, reboot your instance. If you're working in a container, make sure you're using the NVIDIA container runtime and that your container has access to the GPU devices. You can verify CUDA availability with nvidia-smi inside the container. For persistent issues, check that your CUDA version is compatible with the installed driver version.

Ticket: You're trying to set up a Spark cluster or another distributed computing framework on Chameleon, and you want to make sure you're doing it securely
Solution: Great question! For distributed computing setups, security is crucial. First, never expose your cluster services directly to the public internet. Instead, use Chameleon's private network to allow communication between your cluster nodes—all instances within the same project can communicate over the shared private network (typically 192.168.X.X addresses). For accessing web interfaces (like Spark UI), use SSH tunneling rather than opening ports. One easy approach is to use sshuttle, which creates a VPN-like tunnel to your Chameleon network. You can also use a bastion host pattern to simplify access to multiple nodes. If you prefer the traditional approach, you can set up an SSH tunnel like this: ssh -L 8080:localhost:8080 cc@<floating-ip></floating-ip>. This lets you access the Spark UI at localhost:8080 on your local machine while keeping the port closed to the internet. For authentication between nodes, use SSH keys and configure your framework to use secure communication. Check out our documentation on complex appliances for more details on multi-node setups.

Ticket: You're trying to make a reservation using the CLI, but you're getting a flavor reservation error
Solution: Flavor reservations are specific to KVM@TACC, and the key detail is that you must use the flavor ID, not the flavor name. You can list available flavors and their IDs with openstack flavor list. Once you have the correct flavor ID, you can create your lease using that ID in your reservation command. If you're still getting errors, check the resource calendar to confirm that the hardware you want is actually available during your desired time window. Make sure you're targeting KVM@TACC with your CLI environment variables. For detailed instructions on creating flavor-based leases, check out our CLI reservations documentation.

Ticket: You need to open specific ports for communication between instances in your project
Solution: To allow communication between instances in the same project, you'll need to configure your firewall settings. For bare metal instances, you need to update the firewall rules on each instance using tools like iptables or firewalld. For KVM instances, you'll need to modify your security groups to allow the specific ports and protocols you need. By default, instances within the same project can communicate over the shared private network (typically 192.168.X.X addresses), but you need to explicitly allow the ports your applications use. For example, if you're running a database on port 5432, you'd need to allow incoming connections on that port. For detailed information on configuring security settings, check out our network security documentation. Remember to only open the ports you actually need, and restrict access to specific IP ranges when possible to maintain security.

Ticket: You're trying to extend your lease, but the system won't let you
Solution: There are a few common reasons this happens. First, check if the resource you're trying to extend is already reserved by another user after your current lease ends—you can verify this on the resource calendar. If the hardware is available, the next thing to check is your allocation: your project allocation might have expired or be expiring during your desired extension period. You can check your allocation status under the project dashboard. If your allocation has expired, your PI will need to request a renewal. Another possibility is that you've exhausted your Service Units (SUs)—if so, your PI can request a recharge.

Ticket: You haven't logged into Chameleon in a while, and now your old credentials aren't working, or you're having trouble logging in
Solution: If you haven't used Chameleon in some time, your account might need to be re-linked. Don't create a new account! Instead, submit a help desk ticket from the same email address associated with your old account, and our team can help reconnect it. If you want to use a different email address, you can link multiple email addresses to your account through Globus. Simply log into Globus, go to Account Settings, and add your new email under "Linked Identities." Then you can set it as your primary identity. If you're experiencing login issues specifically (like logins that hang), try clearing your browser cache and cookies, or try a different browser. Some users have found that browser extensions or VPNs can interfere with the authentication process.

Ticket: You're trying to SSH into your instance, but the connection keeps failing or timing out
Solution: SSH issues can have many causes, so here's a systematic troubleshooting approach. First, verify the basics: Did you associate a floating IP address to your instance? You can check this in the instance details page. Are you using the correct username? For most Chameleon-supported images, it's cc, not root or ubuntu. Are you specifying the correct SSH key file with the -i flag?

If those check out, verify your key file has the right permissions: chmod 600 your_key.pem. Next, check if your instance actually booted successfully—look at the web console (available in the GUI under your instance's Actions menu) to see if there were any boot errors. Sometimes instances show as "Active" but didn't complete the boot process.

For more detailed diagnosis, use verbose SSH output: ssh -v cc@your-floating-ip. This will show you exactly where the connection is failing. Common issues include: wrong key being offered, security group blocking port 22 (mainly an issue on KVM@TACC), or network connectivity problems.

If you're on a restrictive network (like a corporate or university network), try from a different network or ask your network administrator about SSH access. As a last resort, you can add your SSH public key through the web console if you can't connect—open the console and edit ~/.ssh/authorized_keys to add your key.

Happy Holidays!

As these are some of the most common issues reported on Chameleon this year, we hope this blog helps you troubleshoot these problems if you encounter them! Remember, the help desk is always here if you need assistance. Have a great holiday season, and happy experimenting in 2025!