Skip to content

Commit 5986888

Browse files
alexmillaneviiik-insideqianl-nvxyao-nvcvolkcvolk
authored
Merge main into develop (#615)
## Summary Merge main into develop --------- Signed-off-by: Clemens Volk <cvolk@nvidia.com> Signed-off-by: Lionel Gulich <lgulich@nvidia.com> Co-authored-by: Vikram Ramasamy <158473438+viiik-inside@users.noreply.github.com> Co-authored-by: qianlin <53278415+qianl-nv@users.noreply.github.com> Co-authored-by: Xinjie Yao <xyao@nvidia.com> Co-authored-by: Clemens Volk <cvolk@nvidia.com> Co-authored-by: lgulich <22480644+lgulich@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: isaaclab-review-bot[bot] <270793704+isaaclab-review-bot[bot]@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
1 parent 719bc59 commit 5986888

38 files changed

Lines changed: 412 additions & 515 deletions

‎docker/Dockerfile.isaaclab_arena‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ RUN /isaac-sim/python.sh -m pip install --no-deps -e ${WORKDIR}/submodules/Isaac
5050
RUN ${ISAACLAB_PATH}/isaaclab.sh -i
5151

5252
# Install Isaac Teleop Python APIs (retargeters, device I/O, OpenXR bindings)
53-
RUN /isaac-sim/python.sh -m pip install isaacteleop~=1.0 --extra-index-url https://pypi.nvidia.com
53+
RUN /isaac-sim/python.sh -m pip install isaacteleop~=1.1.0 --extra-index-url https://pypi.nvidia.com
5454

5555
# Upgrade pip ahead of the editable install
5656
RUN /isaac-sim/python.sh -m pip install --upgrade pip

‎docs/index.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -229,8 +229,8 @@ TABLE OF CONTENTS
229229

230230
pages/advanced/private_omniverse
231231
pages/advanced/assets_management
232-
pages/advanced/gr00t_server_client
233232
pages/quickstart/jupyter_notebooks
233+
pages/advanced/gr00t_server_client
234234

235235
.. toctree::
236236
:maxdepth: 1

‎docs/pages/concepts/policy/concept_evaluation_types.rst‎

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -29,10 +29,6 @@ Summary
2929
- Multiple jobs (env/policy combos) in sequence
3030
- ``eval_runner.py``
3131
- No
32-
* - Server–client
33-
- Policy runs in separate process/machine
34-
- Policy runner + remote server
35-
- Depends on client
3632

3733
1. Policy runner — single job (single GPU and multi-GPU)
3834
--------------------------------------------------------
@@ -287,5 +283,3 @@ Choosing an evaluation type
287283
use ``--object_set`` for heterogeneous objects in one run.
288284
- **Many env/policy combinations in one go**: use the **sequential batch eval
289285
runner** with a jobs JSON; use ``--object_set`` for heterogeneous objects in one run.
290-
- **Heavy model on another machine or process**: use **server–client** with the
291-
policy runner on the client and a remote policy server.

‎docs/pages/example_workflows/locomanipulation/step_2_teleoperation.rst‎

Lines changed: 52 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -6,48 +6,58 @@ This workflow covers collecting demonstrations for the G1 loco-manipulation task
66
Step 1: Start the CloudXR Runtime
77
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
88

9-
On the host machine, configure the firewall to allow CloudXR traffic. The required ports depend on the client type.
9+
#. On the host machine, configure the firewall to allow CloudXR traffic. The required ports depend on the client type.
1010

11-
.. code-block:: bash
12-
13-
sudo ufw allow 49100/tcp # Signaling
14-
sudo ufw allow 47998/udp # Media stream
15-
sudo ufw allow 48322/tcp # Proxy (HTTPS mode only)
11+
.. code-block:: bash
1612
13+
sudo ufw allow 49100/tcp # Signaling
14+
sudo ufw allow 47998/udp # Media stream
15+
sudo ufw allow 48322/tcp # Proxy (HTTPS mode only)
1716
18-
Start the CloudXR runtime from the Arena Docker container:
17+
#. Start the CloudXR runtime from the Arena Docker container:
1918

20-
:docker_run_default:
19+
:docker_run_default:
2120

22-
.. code-block:: bash
21+
.. code-block:: bash
2322
24-
python -m isaacteleop.cloudxr
23+
python -m isaacteleop.cloudxr
2524
2625
2726
Step 2: Start Arena Teleop
2827
^^^^^^^^^^^^^^^^^^^^^^^^^^
2928

30-
In another terminal, start the Arena Docker container and launch the teleop session to verify the pipeline:
29+
#. In another terminal, start the Arena Docker container:
3130

32-
:docker_run_default:
31+
:docker_run_default:
3332

34-
.. code-block:: bash
33+
#. Run the following command to activate IsaacTeleop CloudXR environment settings:
3534

36-
source ~/.cloudxr/run/cloudxr.env
37-
python isaaclab_arena/scripts/imitation_learning/teleop.py \
38-
--viz kit \
39-
--device cpu \
40-
galileo_g1_locomanip_pick_and_place \
41-
--teleop_device openxr
35+
.. code-block:: bash
4236
43-
Start the session from the **XR** tab in the application window.
37+
source ~/.cloudxr/run/cloudxr.env
4438
45-
.. figure:: ../../../images/locomanip_arena_server.png
46-
:width: 100%
47-
:alt: Arena teleop with XR running (stereoscopic view and OpenXR settings)
48-
:align: center
39+
.. important::
40+
**Order matters.** In the terminal where you will run Arena, ``source ~/.cloudxr/run/cloudxr.env`` *after* the CloudXR runtime from Step 1 is already running,
41+
and *before* you start the Arena app. The Arena app must inherit the IsaacTeleop CloudXR environment variables.
42+
43+
#. Run the teleop script:
44+
45+
.. code-block:: bash
46+
47+
python isaaclab_arena/scripts/imitation_learning/teleop.py \
48+
--viz kit \
49+
--device cpu \
50+
galileo_g1_locomanip_pick_and_place \
51+
--teleop_device openxr
52+
53+
#. In the running application, start the session from the **XR** tab in the application window.
4954

50-
Arena teleop session with XR running. Stereoscopic view (left) and OpenXR settings in the XR tab (right).
55+
.. figure:: ../../../images/locomanip_arena_server.png
56+
:width: 100%
57+
:alt: Arena teleop with XR running (stereoscopic view and OpenXR settings)
58+
:align: center
59+
60+
Arena teleop session with XR running. Stereoscopic view (left) and OpenXR settings in the XR tab (right).
5161

5262

5363
Step 3: Connect from Meta Quest 3
@@ -59,7 +69,6 @@ A strong wireless connection is essential for a high-quality streaming experienc
5969

6070
#. Open the browser on your headset and navigate to `<https://nvidia.github.io/IsaacTeleop/client>`_.
6171

62-
6372
#. Enter the IP address of your Isaac Lab host machine in the **Server IP** field.
6473

6574
#. Click the **Click https://<ip>:48322/ to accept cert** link that appears on the page.
@@ -70,9 +79,9 @@ A strong wireless connection is essential for a high-quality streaming experienc
7079

7180
#. **Teleoperation Controls**:
7281

73-
* **Left joystick**: Move the body forward/backward/left/right.
74-
* **Right joystick**: Squat (down), rotate torso (left/right).
75-
* **Controllers**: Move end-effector (EE) targets for the arms.
82+
* **Left joystick**: Move the body forward/backward/left/right.
83+
* **Right joystick**: Squat (down), rotate torso (left/right).
84+
* **Controllers**: Move end-effector (EE) targets for the arms.
7685

7786

7887
.. note::
@@ -86,11 +95,20 @@ A strong wireless connection is essential for a high-quality streaming experienc
8695

8796
Reducing render resolution from 1 (default) to 0.2.
8897

98+
Once you have verified the teleoperation pipeline, exit VR from the Quest 3 headset, and stop the Arena teleop app.
8999

90100
Step 4: Record with Quest 3
91101
^^^^^^^^^^^^^^^^^^^^^^^^^^^
92102

93-
#. **Recording**: When ready to collect data, run the recording script from the Arena container:
103+
.. note::
104+
105+
Run the following command to activate IsaacTeleop CloudXR environment settings again if you are starting the recording app from a different terminal.
106+
107+
.. code-block:: bash
108+
109+
source ~/.cloudxr/run/cloudxr.env
110+
111+
#. Run the recording script from the Arena container:
94112

95113
.. code-block:: bash
96114
@@ -107,6 +125,10 @@ Step 4: Record with Quest 3
107125
galileo_g1_locomanip_pick_and_place \
108126
--teleop_device openxr
109127
128+
#. In the running application, start the session from the XR tab in the application window.
129+
130+
#. Follow Step 3 to connect the Quest 3 headset again.
131+
110132
#. Complete the task for each demo. Reset between demos. The script saves successful runs to the HDF5 file above.
111133

112134
.. hint::

‎docs/pages/example_workflows/locomanipulation/step_4_policy_training.rst‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -110,9 +110,15 @@ Training takes approximately 4-8 hours on 8x L40s GPUs.
110110
Compute Requirements:
111111

112112
- **GPUs:** 8x with at least 48 GB VRAM each (e.g. L40s, GB200, etc.)
113-
- **System RAM:** 256 GB or more recommended — multi-GPU training with large batch sizes
113+
- **System RAM:** 512 GB or more recommended — multi-GPU training with large batch sizes
114114
and multiple dataloader workers requires substantial host memory
115115

116+
.. note::
117+
118+
If your system has less RAM or fewer GPUs, you can reduce ``global_batch_size`` and
119+
``dataloader_num_workers`` to fit your hardware. Training will still work but will take
120+
longer to converge.
121+
116122
Training Configuration:
117123

118124
- **Base Model:** GR00T-N1.6-3B (foundation model)
@@ -140,7 +146,6 @@ To post-train the policy, run the following command
140146
--tune_projector \
141147
--tune_diffusion_model \
142148
--dataloader_num_workers=16 \
143-
--use-wandb \
144149
--color_jitter_params brightness 0.3 contrast 0.4 saturation 0.5 hue 0.08 \
145150
--embodiment_tag=NEW_EMBODIMENT
146151

‎docs/pages/example_workflows/locomanipulation/step_5_evaluation.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,7 @@ Parallel evaluation of the policy in multiple parallel environments is also supp
104104
.. code-block:: bash
105105
106106
python isaaclab_arena/evaluation/policy_runner.py \
107+
--viz kit \
107108
--policy_type isaaclab_arena_gr00t.policy.gr00t_closedloop_policy.Gr00tClosedloopPolicy \
108109
--policy_config_yaml_path isaaclab_arena_gr00t/policy/config/g1_locomanip_gr00t_closedloop_config.yaml \
109110
--num_steps 1200 \

‎docs/pages/example_workflows/sequential_static_manipulation/step_2_teleoperation.rst‎

Lines changed: 60 additions & 55 deletions
Original file line numberDiff line numberDiff line change
@@ -6,98 +6,103 @@ This workflow covers collecting demonstrations using Isaac Teleop with an XR dev
66
Step 1: Start the CloudXR Runtime
77
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
88

9-
109
.. tab-set::
1110

1211
.. tab-item:: Meta Quest 3 / Pico 4 Ultra
1312
:selected:
1413

15-
On the host machine, configure the firewall to allow CloudXR traffic.
16-
17-
.. code-block:: bash
14+
#. On the host machine, configure the firewall to allow CloudXR traffic.
1815

19-
sudo ufw allow 49100/tcp # Signaling
20-
sudo ufw allow 47998/udp # Media stream
21-
sudo ufw allow 48322/tcp # Proxy (HTTPS mode only)
16+
.. code-block:: bash
2217
18+
sudo ufw allow 49100/tcp # Signaling
19+
sudo ufw allow 47998/udp # Media stream
20+
sudo ufw allow 48322/tcp # Proxy (HTTPS mode only)
2321
24-
Start the CloudXR runtime from the Arena Docker container:
22+
#. Start the CloudXR runtime from the Arena Docker container:
2523

26-
:docker_run_default:
24+
:docker_run_default:
2725

28-
Create a CloudXR config to enable hand tracking:
26+
#. Create a CloudXR config to enable hand tracking:
2927

30-
.. code-block:: bash
28+
.. code-block:: bash
3129
32-
echo "NV_DEVICE_PROFILE=auto-native" > handtracking.env
30+
echo "NV_CXR_ENABLE_PUSH_DEVICES=0" > handtracking.env
3331
32+
#. Start the CloudXR runtime with the customized config file:
3433

35-
Start the CloudXR runtime with the customized config file:
34+
.. code-block:: bash
3635
37-
.. code-block:: bash
38-
39-
python -m isaacteleop.cloudxr --cloudxr-env-config=handtracking.env
36+
python -m isaacteleop.cloudxr --cloudxr-env-config=handtracking.env
4037
4138
4239
.. tab-item:: Apple Vision Pro
4340

44-
On the host machine, configure the firewall to allow CloudXR traffic.
45-
46-
.. code-block:: bash
41+
#. On the host machine, configure the firewall to allow CloudXR traffic.
4742

48-
# Signaling (use one based on connection mode)
49-
sudo ufw allow 48010/tcp # Standard mode
50-
sudo ufw allow 48322/tcp # Secure mode
51-
# Video
52-
sudo ufw allow 47998/udp
53-
sudo ufw allow 48005/udp
54-
sudo ufw allow 48008/udp
55-
sudo ufw allow 48012/udp
56-
# Input
57-
sudo ufw allow 47999/udp
58-
# Audio
59-
sudo ufw allow 48000/udp
60-
sudo ufw allow 48002/udp
43+
.. code-block:: bash
6144
62-
Start the CloudXR runtime from the Arena Docker container:
45+
# Signaling (use one based on connection mode)
46+
sudo ufw allow 48010/tcp # Standard mode
47+
sudo ufw allow 48322/tcp # Secure mode
48+
# Video
49+
sudo ufw allow 47998/udp
50+
sudo ufw allow 48005/udp
51+
sudo ufw allow 48008/udp
52+
sudo ufw allow 48012/udp
53+
# Input
54+
sudo ufw allow 47999/udp
55+
# Audio
56+
sudo ufw allow 48000/udp
57+
sudo ufw allow 48002/udp
6358
64-
:docker_run_default:
59+
#. Start the CloudXR runtime from the Arena Docker container:
6560

66-
Create a customized config file with the following content:
61+
:docker_run_default:
6762

68-
.. code-block:: bash
63+
#. Create a customized config file with the following content:
6964

70-
printf '%s\n' 'NV_DEVICE_PROFILE=auto-native' 'NV_CXR_ENABLE_PUSH_DEVICES=0' > avp.env
65+
.. code-block:: bash
7166
67+
printf '%s\n' 'NV_DEVICE_PROFILE=auto-native' 'NV_CXR_ENABLE_PUSH_DEVICES=0' > avp.env
7268
73-
Start the CloudXR runtime with the customized config file:
69+
#. Start the CloudXR runtime with the customized config file:
7470

75-
.. code-block:: bash
71+
.. code-block:: bash
7672
77-
python -m isaacteleop.cloudxr --cloudxr-env-config=avp.env
73+
python -m isaacteleop.cloudxr --cloudxr-env-config=avp.env
7874
7975
Step 2: Start Recording
8076
^^^^^^^^^^^^^^^^^^^^^^^
8177

82-
In another terminal, start the Arena Docker container:
78+
#. In another terminal, start the Arena Docker container:
79+
80+
:docker_run_default:
81+
82+
#. Run the following command to activate IsaacTeleop CloudXR environment settings:
83+
84+
.. code-block:: bash
85+
86+
source ~/.cloudxr/run/cloudxr.env
8387
84-
:docker_run_default:
88+
.. important::
89+
**Order matters.** In the terminal where you will run Arena, ``source ~/.cloudxr/run/cloudxr.env`` *after* the CloudXR runtime from Step 1 is already running,
90+
and *before* you start the Arena app. The Arena app must inherit the IsaacTeleop CloudXR environment variables.
8591

86-
Run the recording script:
92+
#. Run the recording script:
8793

88-
.. code-block:: bash
94+
.. code-block:: bash
8995
90-
source ~/.cloudxr/run/cloudxr.env
91-
python isaaclab_arena/scripts/imitation_learning/record_demos.py \
92-
--device cpu \
93-
--viz kit \
94-
--dataset_file $DATASET_DIR/ranch_bottle_into_fridge_recorded.hdf5 \
95-
--num_demos 10 \
96-
--num_success_steps 10 \
97-
put_item_in_fridge_and_close_door \
98-
--object ranch_dressing_hope_robolab \
99-
--embodiment gr1_pink \
100-
--teleop_device openxr
96+
python isaaclab_arena/scripts/imitation_learning/record_demos.py \
97+
--device cpu \
98+
--viz kit \
99+
--dataset_file $DATASET_DIR/ranch_bottle_into_fridge_recorded.hdf5 \
100+
--num_demos 10 \
101+
--num_success_steps 10 \
102+
put_item_in_fridge_and_close_door \
103+
--object ranch_dressing_hope_robolab \
104+
--embodiment gr1_pink \
105+
--teleop_device openxr
101106
102107
103108
Step 3: Connect XR Device and Record

0 commit comments

Comments
 (0)