Developers & cloud

Ask Claude to start the training. Come back tomorrow and ask how it went.

A fine-tune runs for ten hours. The chat you started it from will not last ten hours, and neither will your laptop's battery. AI Commander starts the run as a detached job on your own GPU box, hands back a job id, and gets out of the way — the run keeps going after the chat ends, the laptop sleeps, or the network drops.

Nothing to keep open

You already know the ritual. Open a terminal, ssh into the box, start tmux so the session survives the connection dropping, launch the run under nohup with output redirected somewhere you will remember, detach, and then hope you can find your way back to the same pane tomorrow. Half of that is ceremony for keeping a session alive that nobody wanted to keep alive.

A detached job removes the ceremony. The agent spawns the process on the machine, writes its stdout and stderr to a file there, records the exit code to disk, and returns a jobId to the caller immediately. Nothing on your side has to stay connected for the run to continue:

One honest exception, on the platform most GPU boxes run: on macOS and Windows the job also survives the AI Commander agent itself restarting. On Linux the agent runs as a systemd service and its jobs live inside that service's control group, so stopping, restarting or upgrading the agent takes running jobs down with it. Upgrade a Linux GPU box between runs, not during one.

No tmux, no screen, no SSH session to keep alive. Ask for the run, then ask later how it went.

Pick the machine, then reserve the card

Before anything starts, the agent needs to know where to put it. list_machines reports every machine on your account, and every NVIDIA card on each one — model, total and used VRAM in MiB, and current utilization. session_status reports the same for a single machine. That is enough to choose a box from data the client already has, instead of blind-probing each one with nvidia-smi:

list_machines
- gpu-box — ONLINE, last seen 2026-08-04T09:12:00Z GPU [0] NVIDIA GeForce RTX 5090 — 2104 MiB free of 32607 MiB, 97% utilized GPU [1] NVIDIA GeForce RTX 3090 — 24100 MiB free of 24576 MiB, 0% utilized

No GPU section at all means no NVIDIA card or no driver on that machine — the agent should say so rather than try. Where a card is busy, the fix is not to hope: remember the card's index and pass it as gpu_index when starting the job. That takes an exclusive lock on the card and sets CUDA_VISIBLE_DEVICES for the job, so a second job asking for the same card is refused with gpu_busy, naming the job that holds it, instead of both runs hitting CUDA OOM four hours in. The lock is released when the job ends.

The figures are a reading the machine pushes about once a minute. While a machine is offline they are its last known values and may be stale.

The five tools, and their REST twins

Five MCP tools cover the whole life of a job, and each one has an HTTP equivalent for clients that do not speak MCP. code names the machine exactly as it does for /api/v1/exec — in the JSON body for the POST, and as a query parameter everywhere else.

MCP toolDoesREST
remote_job_start(code, command, cwd?, env?, name?, gpu_index?) Spawns the job, returns its jobId immediately. POST /api/v1/jobs
remote_job_list(code, status?, include_command?) Running jobs plus finished ones still retained. GET /api/v1/jobs
remote_job_status(code, job_id, include_command?) running, exited with the exit code, or unknown. GET /api/v1/jobs/{id}
remote_job_logs(code, job_id, tail_lines?, offset_bytes?, max_bytes?) stdout and stderr interleaved; last 200 lines by default. GET /api/v1/jobs/{id}/logs
remote_job_cancel(code, job_id) Kills the whole process tree and releases any reserved GPU. DELETE /api/v1/jobs/{id}

cwd must be an absolute path that already exists, and env takes string values only. There is no elevated option for jobs — a job runs with exactly the rights remote_exec has on that machine. Full shapes and error codes are in the tool reference and the machine-readable OpenAPI spec.

What it looks like

A PyTorch fine-tune, in a uv workspace under the user's home, on the free card:

Claude — MCP
You: on gpu-box, start the LoRA fine-tune in aic-jobs/lora7b on the free card Claude: > remote_exec code=gpu-box command=/home/lu/aic-jobs/lora7b/.venv/bin/python -c "import torch; print(torch.cuda.is_available())" True > remote_job_start code=gpu-box cwd=/home/lu/aic-jobs/lora7b gpu_index=1 name=lora7b   command=./.venv/bin/python train.py --config lora.yaml   env={"HF_HOME":"/home/lu/aic-jobs/lora7b/.cache/hf"} ✓ jobId 9f2c1b7ad4e05386 — running on GPU 1 (RTX 3090) Claude: Started as job 9f2c1b7ad4e05386 on card 1, which is now reserved. You can close this — ask me about that id whenever.

Nine hours later, from the phone, in a brand new conversation:

Claude — MCP
You: how's job 9f2c1b7ad4e05386 on gpu-box going? Claude: > remote_job_status code=gpu-box job_id=9f2c1b7ad4e05386 status: running, gpuIndex: 1, logBytes: 1082944 > remote_job_logs code=gpu-box job_id=9f2c1b7ad4e05386 step 21400/24000 loss 0.612 lr 4.1e-5 0.66 it/s nextOffsetBytes: 1082944 Claude: Still training — step 21,400 of 24,000, loss down to 0.612. At 0.66 it/s the remaining 2,600 steps are about an hour. Ask me again whenever and I'll read from offset 1082944, so we only see what's new. You: (an hour later) did it finish? Claude: > remote_job_logs code=gpu-box job_id=9f2c1b7ad4e05386 offset_bytes=1082944 step 24000/24000 loss 0.588 saved adapter to out/lora7b-final > remote_job_status code=gpu-box job_id=9f2c1b7ad4e05386 ✓ exited, exitCode 0 Claude: Finished cleanly — 24,000 steps, final loss 0.588, adapter written to out/lora7b-final. Card 1 is free again.

The offset_bytes round-trip is the whole trick for following a long run: each log reply carries a nextOffsetBytes, you hand it back on the next call, and you get only what was written since. Note that nothing polls on its own: there is no scheduler and no callback, so a job is looked at exactly when someone asks about it — which is why the jobId matters more than the conversation it was started in. See the GPU jobs skill for the full procedure an agent follows — workspace layout with uv, the CUDA sanity check, and where model weights should land.

What this is not

Your own box is a real alternative to Modal or RunPod for this work — it is your hardware, your disk, and a real shell. What it is not is a managed platform, and it is worth being blunt about the difference:

Set it up

On the GPU machine (Linux), install the agent:

Follow the signed Linux installer steps and verify the installer before sudo.

Connect your AI client, save the machine under an alias like gpu-box, and describe the run you want started.

FAQ

Will my training keep running if I close the chat?
Yes. A detached job is spawned on your own machine and the call returns as soon as it has started, so nothing on your side has to stay open. The run survives the conversation ending, the laptop going to sleep, and the network dropping. On macOS and Windows it also survives the AI Commander agent itself restarting; on Linux the agent runs as a systemd service and its jobs stay inside that service's control group, so stopping, restarting or upgrading the service stops running jobs too. Keep the jobId and any client can pick the run back up later.
Can I check on a training run from my phone?
Yes. Job state lives on the machine, not in the conversation, so any signed-in client can query it — a chat app on your phone, a different laptop, or a script hitting the REST API. Ask for remote_job_status with the jobId for the state, or remote_job_logs for the recent output. Nothing needs to be resumed first.
Do I still need tmux or screen?
No. The agent writes the job's stdout and stderr to a file on the machine and records the exit code to disk, which is exactly what you were using tmux plus nohup to approximate. There is no session to keep alive and nothing to reattach to — you read the log by jobId whenever you want it.
How do I stop two runs from grabbing the same GPU?
Pass gpu_index when you start the job. The machine takes an exclusive lock on that card and sets CUDA_VISIBLE_DEVICES for the job, so a second job asking for the same card is refused with gpu_busy and told which job holds it, instead of both runs hitting CUDA OOM halfway through. The lock is released when the job ends.
Can I use my own 5090 instead of renting from RunPod?
That is the point of it. Your own box is a real alternative to a rented GPU here — it is your hardware, your disk, and a real shell, with no per-hour meter and no image to build. What it is not is a managed platform: nothing queues, schedules, or retries for you, so you or the agent decide what runs and when.
What happens if the machine reboots mid-run?
The process is gone and no exit code was recorded, so remote_job_status reports unknown. That is an honest answer, not a failure code: the outcome genuinely cannot be determined, and it should never be reported as success. Read the job's log and check your checkpoints to see how far the run actually got before restarting it.

Start the run, then close the laptop

Install the agent on your GPU box and let Claude launch, follow, and report on your training runs.