Drive a remote GPU & ML training box from Claude
Your GPU box lives in a closet, a colo, or a rented instance — headless, no monitor, often behind NAT.
AI Commander lets Claude operate it over a real shell: watch nvidia-smi,
diagnose a CUDA mismatch, free a wedged card, and on Pro pull the latest metrics or a checkpoint file, all from your laptop
without exposing SSH. Long training runs get their own treatment — see
detached jobs.
The job
ML rigs are the definition of headless: there's nothing to screen-share, and everything happens at the command line. The friction is reaching them — port forwarding, jump hosts, or fiddly tunnels. AI Commander removes that and adds an agent that can reason about the output:
- Keep an eye on a run in progress: periodic
nvidia-smiplus the tail of the training log. - Diagnose the classic "CUDA error: no kernel image" by checking driver/toolkit versions.
- Free a stuck GPU: find the orphaned process and kill it.
- Check disk before a checkpoint write, then tail the loss to see if it's converging.
- On Pro, bring back a checkpoint, render, or metrics file with built-in remote file transfer, up to 100 MiB; use your own object storage for larger artifacts.
What it looks like
Training runs are detached jobs
nvidia-smi,
a CUDA fix, a quick disk check — is an ordinary command that finishes while you wait. A fine-tune that
takes eleven hours is not. Ask Claude to start it and it uses remote_job_start instead, which
hands back a job ID and leaves the process running on your machine. Later, in the same chat or a new one,
Claude checks the status and reads the logs by that ID. The run keeps going after the chat ends, the laptop
sleeps, or the network drops — no tmux, no screen, no SSH session to keep alive. It also carries on when the
agent itself restarts: on a Linux rig each job started by an agent that already has this feature is launched
into its own transient systemd scope, outside the agent service's control group, so restarting
or upgrading the agent leaves it running — provided the box runs systemd and the agent runs as
root, and without both an upgrade still stops running jobs, as it does for a run that was already going
when the agent was upgraded to that version.
See how detached training runs work →
Why AI Commander for GPU work
| AI Commander | SSH tunnel | Jupyter exposed | Cloud sandbox (E2B) | |
|---|---|---|---|---|
| Your actual rig | ✓ | ✓ | ✓ | ephemeral |
| No inbound port | ✓ | ✗ | ✗ | ✓ |
| Headless / no display | ✓ | ✓ | browser | ✓ |
| AI client drives it | ✓ MCP | ✗ | manual | SDK |
Set it up
On the GPU machine (Linux), install the agent:
Follow the signed Linux installer steps and verify the installer before sudo.
Connect your AI client and quote the session code (or alias it gpu-rig). Then describe what you want done.
FAQ
nvidia-smi or read training logs by quoting the session code. The text result comes back when the command finishes, so a display is never needed — ideal for headless rigs and rented GPU instances.remote_job_start with your GPU box's session code and the training command, and gets back a job ID immediately. From then on it polls the job status and reads the job logs to follow the loss curve or catch a stack trace, and can cancel the job to kill the run and free the GPU. Ordinary commands are for the short things around the run — nvidia-smi, checking disk space, inspecting a checkpoint — not for the run itself.nvidia-smi to see which card is busy and how much VRAM is held, then queries the compute processes on that card to find what is still holding it — usually an orphaned python from a run that crashed without releasing memory — and kills that pid once you confirm. If a detached job holds the card rather than a stray process, cancel the job instead: that terminates its whole process tree and releases the card's reservation.Talk to your GPU box
Install the agent and let Claude watch, debug, and report on the machine your models train on.