Trials 8 tasks x 2 conditions x 100Primary configs kinrt_full / kinrt_loraSimulator checkout external

Simulation benchmark

Reproduce RoboTwin

Train KinRT on the 800-episode RoboTwin set, serve a concrete checkpoint, and evaluate all eight tasks under clean and randomized conditions without mixing sessions or output directories.

Training: 10,000 stepsGlobal batch: 32Evaluation: 100 trials per condition
01

Set the reproduction boundary

policy/ contains the single KinRT implementation, and script/ contains the RoboTwin evaluation overlay. The release does not contain the complete simulator, task assets, or the 800-episode training dataset.

Use an isolated simulator worktree.

Apply the unified overlay once, select either kinrt_full or kinrt_lora, and record that config with every checkpoint and evaluation.

RunSourceConfigInitialization
KinRT FULLpolicy/pi05kinrt_fullPI0.5 dense base
KinRT LoRApolicy/pi05kinrt_loraPI0.5 base plus LoRA
02

Prepare a compatible RoboTwin checkout

Install RoboTwin 2.0 and its simulator assets according to the upstream project. Before applying the policy overlay, verify that the checkout contains envs/, task_config/, script/, and the embodiment assets required by the selected tasks.

Apply the KinRT overlay
export ROBOTWIN_ROOT=/work/RoboTwin-kinrt
test -d "$ROBOTWIN_ROOT/envs"
test -d "$ROBOTWIN_ROOT/task_config"
cp -a /path/to/KinRT/policy/. "$ROBOTWIN_ROOT/policy/"
cp -a /path/to/KinRT/script/. "$ROBOTWIN_ROOT/script/"
cd "$ROBOTWIN_ROOT/policy/pi05"
uv sync --frozen

FULL and LoRA use this same overlay. Use separate experiment names and output directories, and preserve the upstream RoboTwin commit and simulator asset version in the experiment record.

Gate

Import the task module, load one task config, and reset one episode without loading a policy. Simulator failures must be resolved before policy debugging.

03

Verify the 800-episode dataset

The reported comparison uses 162,545 frames from 800 episodes. The active config expects three images, state, action, and prompt fields after repacking.

Model inputLeRobot field in the reported config
images.cam_highobservation.images.cam_high
images.cam_left_wristobservation.images.cam_left_wrist
images.cam_right_wristobservation.images.cam_right_wrist
stateobservation.state
actionsaction
promptTask text via prompt_from_task=True

Replace the placeholder repo_id="demo_mixed_repo" with the local LeRobot dataset identity. Verify episode count, frame count, field shapes, action units, and global index coverage.

04

Compute norm stats and train

Generate and verify the K=4 supervision using the KinRT method procedure, then point the selected config to the local dataset, labels, base weights, and writable output. Keep one config name from normalization through serving.

FULL training
cd "$ROBOTWIN_ROOT/policy/pi05"
uv run python scripts/compute_norm_stats.py --config-name kinrt_full
uv run python scripts/train.py kinrt_full --exp-name robotwin_kinrt_full
LoRA training
cd /work/RoboTwin-kinrt-lora/policy/pi05
uv run python scripts/compute_norm_stats.py --config-name kinrt_lora
uv run python scripts/train.py kinrt_lora --exp-name robotwin_kinrt_lora

The audited settings are 10,000 steps, global batch 32, EMA disabled, supervised router coefficient 0.05, and inverse-square-root balanced sampling. FULL uses one FSDP device in the retained config; LoRA uses two. Report any resource override.

05

Evaluate one immutable checkpoint

Create deploy_policy.yml from the included example. Resolve every placeholder before starting the server.

Required deployment fields
policy_name: pi05
task_name: <task-module-name>
task_config: <clean-or-random-config>
test_num: 100
train_config_name: kinrt_full
model_name: robotwin_kinrt_full
checkpoint_id: 10000
checkpoint_dir: /checkpoints
model_server_host: <reachable-server-host>
server_bind_host: 0.0.0.0
port: 8001
eval_result_root: ./eval_result
router_info_root: ./eval_result/router_info
record_router_info: true

Local evaluation

One task and condition
cd "$ROBOTWIN_ROOT/policy/pi05"
bash eval.sh <task_name> <task_config> kinrt_full \
  robotwin_kinrt_full 0 0

Remote evaluation

Model server
cd "$ROBOTWIN_ROOT/policy/pi05"
bash serve_remote_model.sh deploy_policy.yml
Simulator client
cd "$ROBOTWIN_ROOT/policy/pi05"
MODEL_SERVER_HOST=<reachable-server-host> \
  bash eval_remote.sh <task_name> 0

The server creates an independent policy session for each socket connection. LORA additionally uses an explicit eval_done acknowledgment, a 300-second client socket timeout, episode-level exception isolation, and checkpoint identifiers in responses. These are robustness differences, not changes to KinRT.

06

Complete the 16-condition task matrix

Resolve the exact task module and task-config filenames from the compatible RoboTwin checkout. The paper reports these eight task labels under both clean and randomized conditions.

RoboTwin hand block taskHand block
RoboTwin hang mug taskHang mug
RoboTwin move can taskMove can
RoboTwin open laptop taskOpen laptop
RoboTwin place shoes taskPlace shoes
RoboTwin place object on pad taskPlace on pad
RoboTwin rotate QR code taskRotate QR code
RoboTwin turn switch taskTurn switch
Clean condition

100 requested trials per task with a fixed checkpoint and recorded seeds.

Random condition

100 requested trials per task using the corresponding randomization config.

Failure accounting

Transport failure is terminal; valid simulation or inference failures count as failed episodes.

Completion

Count results on disk and do not trust a partial client exit as a completed evaluation.

07

Report enough information to audit the result

  • FULL or LoRA config name, source commit, and all config overrides.
  • Dataset identity, frame count, label digest, norm-stat asset ID, and base checkpoint.
  • Fine-tuned experiment name, checkpoint step, and parameterization.
  • RoboTwin commit, task module, task config, simulator assets, condition, and seed set.
  • Requested, completed, successful, failed, and transport-aborted episode counts.
  • Evaluation result directory and router telemetry directory.
Never combine partial runs silently.

If an evaluation is resumed, preserve the original seed/result records and document the merge rule.