The "Foundation of Power": Setting Up CUDA 13.0 and NVIDIA Drivers on Ubuntu 24.04 Without Losing Your Mind

1. Why the "Foundation" Matters (The Bragging Part)

Before you can run Unsloth or play with uv, you need to talk to the metal. Most people treat NVIDIA drivers like a scary ritual, but if you get this right, your GPU becomes a beast. I’m running Ubuntu 24.04 with CUDA 13.0—the bleeding edge.

Why bother with the latest? Because performance is money. CUDA 13.0 brings optimizations that make modern kernels run smoother. If your base layer is shaky, your LLM training will throw "Illegal Instruction" errors or "CUDA Out of Memory" for no reason. I spent the time to figure out the cleanest path so you don't have to. Here is how I laid the groundwork for my local AI rig.

2. Phase 1: The CUDA Toolkit (The Brains)

I prefer the local installer method for Ubuntu 24.04. It’s more controlled and avoids the "version drift" you get with standard repos.

First, we set the package priority (the "pin" file) so Ubuntu knows we mean business about using NVIDIA's official repo:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-ubuntu2404.pin
sudo mv cuda-ubuntu2404.pin /etc/apt/preferences.d/cuda-repository-pin-600

Now, we grab the massive local installer for CUDA 13.0.2. This is the heavy lifting:

wget https://developer.download.nvidia.com/compute/cuda/13.0.2/local_installers/cuda-repo-ubuntu2404-13-0-local_13.0.2-580.95.05-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu2404-13-0-local_13.0.2-580.95.05-1_amd64.deb
sudo cp /var/cuda-repo-ubuntu2404-13-0-local/cuda-*-keyring.gpg /usr/share/keyrings/
sudo apt-get update
sudo apt-get -y install cuda-toolkit-13-0

3. Phase 2: The Driver (The Muscles)

The toolkit is the software, but the driver is what actually moves the fans and the voltage. You have two choices here.

Option A: The Modern Route (Open Kernel Module)
If you are on a newer card (RTX 30-series, 40-series, or H100s), the "open" flavor is getting a lot of love lately for its integration with modern Linux kernels.

sudo apt-get install -y nvidia-open

Option B: The Battle-Tested Route (Proprietary)
If you want the classic experience that has powered AI for the last decade, go with the standard cuda-drivers:

sudo apt-get install -y cuda-drivers

Personal Note: I usually go for cuda-drivers (Proprietary) when I'm doing heavy LLM training—it’s just one less variable to worry about when things get intense.

4. The "Moment of Truth"

After a reboot, I always run the magic command:

nvidia-smi

If you see that glorious table showing your GPU name and "CUDA Version: 13.0," you are officially in the elite club. Your hardware is now ready to feed tokens into Unsloth at lightning speeds.

Summary

Don't let the "Linux Driver" memes scare you. On Ubuntu 24.04, the process is cleaner than ever. By manually pinning the repository and grabbing the 13.0 toolkit, you've built a foundation that can handle the most demanding fine-tuning jobs. Now that the metal is ready, you're cleared for takeoff with uv and Unsloth.