BrickSim: A Physics-Based Simulator
for Manipulating Interlocking Brick Assemblies

Interlocking brick assemblies provide a standardized yet challenging testbed for contact-rich and long-horizon robotic manipulation, but existing rigid-body simulators do not faithfully capture snap-fit mechanics. We present BrickSim, the first real-time physics-based simulator for interlocking brick assemblies. BrickSim introduces a compact force-based mechanics model for snap-fit connections and solves the resulting internal force distribution using a structured convex quadratic program. Combined with a hybrid architecture that delegates rigid-body dynamics to the underlying physics engine while handling snap-fit mechanics separately, BrickSim enables real-time, high-fidelity simulation of assembly, disassembly, and structural collapse. On 150 real-world assemblies, BrickSim achieves 100% accuracy in static stability prediction with an average solve time of 5 ms. In dynamic drop tests, it also faithfully reproduces real-world structural collapse, precisely mirroring both the occurrence of breakage and the specific breakage locations. Built on Isaac Sim, BrickSim further supports seamless integration with a wide variety of robots and existing pipelines. We demonstrate robotic construction of brick assemblies using BrickSim, highlighting its potential as a foundation for research in dexterous, long-horizon robotic manipulation.

BrickSim is a ready-to-use simulator for robotic manipulation of interlocking brick assemblies. It inherits Isaac Sim’s support for robotic platform integration, rigid-body simulation, and rendering. As shown on the left of the figure below, users can configure both robotic embodiments and brick assets, including individual bricks and a wide variety of predefined assemblies. The simulator supports direct UI interaction with bricks via drag-and-drop operations, as well as policy deployment and teleoperation.

As shown in the figure below, BrickSim augments the Isaac Sim backbone with three modules: the Brick Topology Graph (BTG), the Assembly Monitor (ASM), and the Breakage Detector (BRD). The BTG stores snap-fit connectivity, maintains consistent relative poses of bricks within each connected component, and synchronizes rigid constraints and collision filtering with the underlying PhysX physics engine. The ASM monitors contact reports and creates new connections when two valid stud-hole engagements are detected between two bricks. The BRD evaluates the current assembly under external loads, solves for the internal force distribution, and removes overloaded connections. At each simulation step, Isaac Sim advances the rigid-body state and reports contact and impulse information. The ASM and BRD then update the topology, and the updated BTG is synchronized back to PhysX for the next step.

BrickSim enables real-time, high-fidelity simulation of assembly, disassembly, and structural collapse. It faithfully reproduces real-world structural behavior, precisely mirroring both the occurrence of breakage and the specific breakage locations.

BrickSim supports seamless integration with a wide variety of robot platforms. We demonstrate robotic construction of brick assemblies using BrickSim in both single-arm and bimanual scenarios, highlighting its potential as a foundation for research in dexterous, long-horizon robotic manipulation.