> ## Documentation Index
> Fetch the complete documentation index at: https://docs.hardlightsim.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Managed Training Quickstart

> Run a full rollout-train loop with a test trainer container image.

## Goal

Create a managed training run (`/v1/training-runs`) that:

* dispatches rollout jobs
* builds an iteration dataset manifest
* runs a trainer container
* emits `metrics.json` and chained checkpoints

## Prerequisites

* A service API key (`hls_live_*`) with training scopes.
* Docker installed locally to build your trainer image.
* API environment configured:

```bash theme={null}
export HARDSIM_API_URL=https://api-sim.hardlightsim.com
export HARDSIM_API_KEY=hls_live_your_service_key
```

## 1) Build and Push Test Trainer Image

Use the reference image in this repo:

* `examples/managed_training/smoke_trainer/Dockerfile`
* `examples/managed_training/smoke_trainer/train_smoke.py`

Build locally:

```bash theme={null}
docker build -t hardsim-trainer-smoke:0.1.1 -f examples/managed_training/smoke_trainer/Dockerfile examples/managed_training/smoke_trainer
```

Push to your registry (GHCR/ECR) and use that image URI in `trainer_spec.image_uri`.

## 2) Submit Training Run

```bash theme={null}
curl -X POST "$HARDSIM_API_URL/v1/training-runs" \
  -H "Authorization: Bearer $HARDSIM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "project_id": "proj_demo",
    "name": "managed-loop-smoke",
    "rollout_template": {
      "rollout_parallelism": 2,
      "episodes_per_iteration": 2,
      "scene_id": "franka_cabinet_oige_v1",
      "num_envs": 8,
      "steps": 128,
      "control": {"task_mode": "fr3_pick_lift_block_v1", "fix_base": true}
    },
    "trainer_spec": {
      "image_uri": "ghcr.io/acme-robotics/hardsim-trainer-smoke:0.1.1",
      "entrypoint": "python /app/train_smoke.py",
      "args_template": [
        "--dataset-manifest-path", "{dataset_manifest_path}",
        "--output-dir", "{output_dir}",
        "--checkpoint-in-uri", "{checkpoint_in_uri}"
      ],
      "env": {
        "HARDSIM_RUN_ID": "{run_id}",
        "HARDSIM_ITERATION_ID": "{iteration_id}"
      },
      "resources": {"gpu": 1, "cpu": 1, "memory_gb": 2}
    },
    "loop_policy": {
      "max_iterations": 3,
      "max_wallclock_hours": 2,
      "stop_on_iteration_failure": true,
      "rollout_retry_limit": 1,
      "trainer_retry_limit": 1
    },
    "success_criteria": {"patience_iterations": 2},
    "budget_policy": {"max_credits": 50}
  }'
```

Notes:

* Managed training is GPU-only by default (`trainer_spec.resources.gpu > 0` is required).
* If `gpu` is omitted or `0`, API returns `HTTP 400`.
* Trainer fleet scaling uses active runnable tasks and includes startup recovery + ghost-running protection.
* You can use `checkpoint_init_asset_id` instead of raw checkpoint URI.

## 3) Monitor

Use:

* `GET /v1/training-runs/{run_id}`
* `GET /v1/training-runs/{run_id}/iterations`
* `GET /v1/training-runs/{run_id}/checkpoints`
* `GET /v1/training-runs/{run_id}/events`
* `GET /v1/training-runs/{run_id}/checkpoints/{checkpoint_id}/download-url` (checkpoint download link)

SDK terminal monitoring:

```python theme={null}
import hardsim as hs

client = hs.HardsimClient.from_env()

# Manual status checks anytime:
print(client.get_training_run(run_id))
print(client.list_training_events(run_id, limit=20))

# Live watch stream (full watch if timeout is large enough):
client.watch_training_run(run_id, poll_interval_s=5.0, timeout_s=24*3600)
```

## Expected Success Contract

Each completed iteration must produce:

* trainer output `metrics.json`
* one checkpoint (`checkpoint.pt` or `checkpoint_out_uri.txt`)
* one checkpoint lineage entry in `/checkpoints`

`checkpoint_init_asset_id` is optional. Add it only when you want to start from an existing checkpoint asset.