Managed Training Quickstart

Goal

Create a managed training run (/v1/training-runs) that:

dispatches rollout jobs
builds an iteration dataset manifest
runs a trainer container
emits metrics.json and chained checkpoints

Prerequisites

A service API key (hls_live_*) with training scopes.
Docker installed locally to build your trainer image.
API environment configured:

export HARDSIM_API_URL=https://api-sim.hardlightsim.com
export HARDSIM_API_KEY=hls_live_your_service_key

1) Build and Push Test Trainer Image

Use the reference image in this repo:

examples/managed_training/smoke_trainer/Dockerfile
examples/managed_training/smoke_trainer/train_smoke.py

Build locally:

docker build -t hardsim-trainer-smoke:0.1.1 -f examples/managed_training/smoke_trainer/Dockerfile examples/managed_training/smoke_trainer

Push to your registry (GHCR/ECR) and use that image URI in trainer_spec.image_uri.

2) Submit Training Run

curl -X POST "$HARDSIM_API_URL/v1/training-runs" \
  -H "Authorization: Bearer $HARDSIM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "project_id": "proj_demo",
    "name": "managed-loop-smoke",
    "rollout_template": {
      "rollout_parallelism": 2,
      "episodes_per_iteration": 2,
      "scene_id": "franka_cabinet_oige_v1",
      "num_envs": 8,
      "steps": 128,
      "control": {"task_mode": "fr3_pick_lift_block_v1", "fix_base": true}
    },
    "trainer_spec": {
      "image_uri": "ghcr.io/acme-robotics/hardsim-trainer-smoke:0.1.1",
      "entrypoint": "python /app/train_smoke.py",
      "args_template": [
        "--dataset-manifest-path", "{dataset_manifest_path}",
        "--output-dir", "{output_dir}",
        "--checkpoint-in-uri", "{checkpoint_in_uri}"
      ],
      "env": {
        "HARDSIM_RUN_ID": "{run_id}",
        "HARDSIM_ITERATION_ID": "{iteration_id}"
      },
      "resources": {"gpu": 1, "cpu": 1, "memory_gb": 2}
    },
    "loop_policy": {
      "max_iterations": 3,
      "max_wallclock_hours": 2,
      "stop_on_iteration_failure": true,
      "rollout_retry_limit": 1,
      "trainer_retry_limit": 1
    },
    "success_criteria": {"patience_iterations": 2},
    "budget_policy": {"max_credits": 50}
  }'

Notes:

Managed training is GPU-only by default (trainer_spec.resources.gpu > 0 is required).
If gpu is omitted or 0, API returns HTTP 400.
Trainer fleet scaling uses active runnable tasks and includes startup recovery + ghost-running protection.
You can use checkpoint_init_asset_id instead of raw checkpoint URI.

3) Monitor

Use:

GET /v1/training-runs/{run_id}
GET /v1/training-runs/{run_id}/iterations
GET /v1/training-runs/{run_id}/checkpoints
GET /v1/training-runs/{run_id}/events
GET /v1/training-runs/{run_id}/checkpoints/{checkpoint_id}/download-url (checkpoint download link)

SDK terminal monitoring:

import hardsim as hs

client = hs.HardsimClient.from_env()

# Manual status checks anytime:
print(client.get_training_run(run_id))
print(client.list_training_events(run_id, limit=20))

# Live watch stream (full watch if timeout is large enough):
client.watch_training_run(run_id, poll_interval_s=5.0, timeout_s=24*3600)

Expected Success Contract

Each completed iteration must produce:

trainer output metrics.json
one checkpoint (checkpoint.pt or checkpoint_out_uri.txt)
one checkpoint lineage entry in /checkpoints

checkpoint_init_asset_id is optional. Add it only when you want to start from an existing checkpoint asset.

Start Here

Core Guides

Billing

Managed Training Quickstart

Goal

Prerequisites

1) Build and Push Test Trainer Image

2) Submit Training Run

3) Monitor

Expected Success Contract

​Goal

​Prerequisites

​1) Build and Push Test Trainer Image

​2) Submit Training Run

​3) Monitor

​Expected Success Contract

Goal

Prerequisites

1) Build and Push Test Trainer Image

2) Submit Training Run

3) Monitor

Expected Success Contract