It’s the end of the year, so that means it’s time for our annual TIckets of the Year blog! This blog encompasses some of our most common help desk tickets and outlines some answers for them, so it can hopefully be a helpful reference if you run into any of these problems. If you need more help, the best places to look would be the docs, the FAQ, or submitting a new ticket to the help desk. 

If you’re curious about the most common tickets we’ve encountered in the past, you can check out our Tickets of the Year blogs from 2021 and 2020.

Tickets of the Year!

Ticket: Your instance has started, but you can’t connect to it via SSH even though you’ve associated a floating IP address and set up your keys correctly. You go to check the web console to diagnose the issue, but that’s not working either!
Solution: You might have a bad image that failed to boot properly, even though it looks like your instance is active. This can sometimes happen if you’re using a custom image which has some weird state change occur while it is being built. It might be a good idea to try one of the officially supported Chameleon images on the same node to narrow down the issue. If that doesn’t work at all, it’s a good idea to monitor the web console as the node boots so you can see output from the kernel or our init scripts, which often yields an answer to the problem. The answer might be an internal networking/hardware issue, which you should report to the help desk along with whatever errors you observed.

Ticket: I know what resources I want, but when I go to make a lease to reserve them, it says there are “not enough resources available!”
Solution: It’s probably because the hardware isn’t available. Your best bet to confirm this would be to check the resource calendar at the appropriate site. If that hardware is not available in the time window you need it, you can check the hardware browser to see if there is any other hardware which meets your needs. For more details on how to filter for your desired specifications, check out the docs. Another option is to make a lease which starts in the future, so that you can reserve the hardware as soon as the existing reservation ends.

Ticket: I know for a fact that my hardware is available, but I still can’t make a lease!
Solution: Your project’s allocation has probably expired, or will expire during the time of your desired lease. It’s also possible that your allocation no longer has any SUs available. You can double check this here, and ask your PI to request a renewal/recharge so that you can reserve your resources.

Ticket: I haven’t logged into Chameleon in a long time, and now I can’t use my old credentials anymore!
Solution: You may have a legacy account, or perhaps an internal identifier was changed by TACC or Globus. No need to make a new account! Simply report it to the help desk using the same email address as your broken account, and we can make sure it is properly linked up to Chameleon. If the issue is that you want to use a new email address, you can link it to your existing account under Globus, and then set it as your primary identity under the "Manage Identities" menu.

Ticket: I don’t know how to SSH into my instance!
Solution: We allow users to provision SSH keys on their instances in 2 different ways: You can either generate a keypair in our web GUI, or you can import an existing keypair from your local machine. Then, you can load them onto your instance at creation time. <picture> If you accidentally start up an instance without the right keys loaded, you can always open your ~/.ssh/authorized_keys file using the web console, and paste the right public key in. Once you do this, you will be able to SSH to your instance. It is important that you never upload your private key, and never enable SSH password authentication!</picture>

Ticket: I waited a long time for my instance to spawn, and it eventually timed out.
Solution: Similar to the above ticket, it could be a bad image or other start-up errors. You should keep an eye on the web console while your image spawns and see if it reports any errors. Another issue could be that the network switch attached to your node was overloaded while your instance was spawning. Did you try to spawn many instances of the same type at the same time? It might help to limit that to 1-3 instances at the same time. It could also be the case that another user overloaded the switch if you only tried to spawn 1 instance. It never hurts to try again. If not, it could be internal hardware/networking issues, in which case, it’s best to report the issue to the help desk.

Ticket: I have a Postgres/Spark/Zookeeper/Kubernetes/some other service running on my instance which I want to expose to the open internet!
Solution: Don’t do that! There is very rarely a need to allow incoming connections to your instance under any circumstances. If you have a service which needs to be contacted via the internet, the best way to do this is within a secure, encrypted tunnel. We highly recommend an SSH tunnel, since it is the easiest to set up. We never recommend changing firewall rules, and there is no need to mess with security groups unless you are using the KVM cloud. 

Ticket: How do I store my data for a long-term experiment? How can I make it easily accessible to multiple nodes or users?
Solution: Luckily for you, we recently published another blog post which can hopefully answer all your storage questions! In this blog, we go over different ways to store your data, including ephemeral NFS shares, Object Storage, and Block Storage.

Ticket: My PI invited me to our project, but I never received my invite!
Solution: It could be that the invite went to your spam folder. In some cases, email providers (particularly Google) reject our emails outright. If this happens, open a new help ticket or send an email to help@chameleoncloud.org from the same email address that your PI invited, and we can help you out with an invite link. 

Happy Holidays!

As these are some of the most common issues reported on Chameleon, we hope this blog helps everyone on mitigating these pesky issues on Chameleon! Have a great holiday, and a happy new year!