Turing Sim: Camera Ground Truth from One Hydra Render

Camera sensors now render RGB with pixel-aligned depth, normals, object and class masks, and boxes.

中文版在文末，点开即可阅读。

The overview camera switching between RGB, depth, normals, objects, classes and boxes.

One colour per object: each UR10 link, the table and the cube.

Metric depth along the optical axis, shown with an inverse-depth Turbo ramp.

Situation

After IMU and LiDAR, the sensor stage needed cameras: not only pictures, but the ground truth that robot learning uses, such as depth, segmentation and boxes, aligned exactly with the image. Turing Sim renders with Storm (rasterised) or Turing RT (ray traced) through Hydra. Neither exposed its depth or id buffers to Python.

Task

Give camera sensors every output from a single render, so all of them describe the same pixels, with accuracy that can be checked. They also had to run in the editor without degrading the viewport.

Action

1. Cameras are USD cameras. A camera sensor is a standard UsdGeom.Camera with turingSensor:type = "camera". Presets set the lens from a field of view and the image's aspect.
2. Read the AOVs back. A small native module reads Hydra's color, depth, primId and instanceId buffers and decodes ids to scene paths, including instance proxies. It works with Storm and with Turing RT.
3. A separate render process. Integer ids need a render without multisampling, and Hydra reads that setting once per process, so cameras render in their own worker and the editor's viewport keeps its antialiasing. Storm's eye-normal buffer turned out to clamp negative components, so normals are rebuilt from depth.
4. From buffers to ground truth. RGB is averaged over odd supersampling; ground truth uses each pixel's centre sample. Depth is converted to metres along the optical axis. An object is the nearest labelled prim, else its rigid body. Tight and amodal 2D boxes and oriented 3D boxes come with OpenCV intrinsics and the camera's pose.

Result

On a wall 4 m away, depth is within 2 mm wherever the wall shows, and back-projected pixels land on the wall plane within 2 mm. Storm and Turing RT agree pixel for pixel on the masks, with 0.01 mm median depth difference.

A 640×480 frame with every output takes 36 ms; with 3×3 antialiasing, 73 ms. The presets' rates follow these measurements.

The captures show the editor's camera preview switching between outputs on the UR10 workcell. A follow-up commit stopped box names from piling up on the robot's links. Frames are best effort in the editor; exact-rate capture belongs to the synthetic-data pipeline, which is next.