[ PHYSICAL AI RESEARCH // EMBODIED MANIPULATION ]

Can Robotic Arms Build with Interlocking Bricks?

Deconstructing spatial reasoning, millimeter tolerances, and why snapping a child's toy is one of the hardest benchmarks in modern robotics.

RoboRoots Curriculum · SO-101 · Pick-and-place

RoboRoots infographic. Left: the LEGO force model, showing gravity, pressing and pulling, dragging and supporting, and horizontal normal forces on a single brick and on connected bricks. Right: a robotic arm running a pick, place, build sequence, and the base plate, bricks and target builds used for assembly. — Fig 1: The RoboRoots Lego Force Model & Automated Assembly Engine — Bridging normal force vectors, stud friction, and multi-step pick-and-place trajectories.

[ S1 // The Illusion of Simplicity ]

The Illusion of Simplicity

A four-year-old can press two bricks together without looking. They feel the studs catch the rim, rock the brick a fraction of a millimetre until it drops into register, then push until it clicks. Each of those steps happens through touch, and none of them happens in a plan.

A robot without force sensing works the other way. It receives coordinates, such as (dx, dy, dz) on an 8 mm stud grid, and drives its joints until the encoders agree that it has arrived. That is open-loop placement: the controller trusts that the brick is where the model says it is and that the gripper is holding it square. The parts are moulded to tolerances on the order of ten micrometres. Hobby servos and 3D-printed links tend to stack up errors of a millimetre or more.

15–35 N

Typical press-fit force to seat a brick, depending on stud count and wear. The clutch is pure friction: an interference fit between stud and tube, with no latch to aim for.

That is the illusion: the task looks like geometry, but it is mostly contact. You have to arrive within a fraction of a stud pitch and then deliver a controlled push through a structure that can topple or skate across the baseplate.

[ S2 // The Tabletop Manipulator ]

The Tabletop Manipulator

The SO-101 is an open-source tabletop arm from TheRobotStudio in collaboration with Hugging Face and other hardware partners like Seeed Studio. It is built from 3D-printed links and serial-bus Feetech servos. It is small and cheap, and it is honest about the limits of hobby hardware, which makes it a good benchmark platform.

The arm has six actuated axes. Strictly, five of them position and orient the tool and the sixth only grips, but the community still counts it as six-DoF. For a brick pick, the joints split cleanly into three jobs: aim, reach, and align.

The six actuated axes of the SO-101 arm
Axis	Joint	What it does in a brick pick
J1	Base pan Yaws the whole arm so its working plane points at the brick.	Yaws the whole arm so its working plane points at the brick.
J2	Shoulder lift Pitches the 220 mm upper link (L1). Carries the largest gravity load.	Pitches the 220 mm upper link (L1). Carries the largest gravity load.
J3	Elbow flex Folds the 200 mm forearm (L2). With J2, sets reach and height.	Folds the 200 mm forearm (L2). With J2, sets reach and height.
J4	Wrist flex Holds the tool vertical: θ4 = −90° − θ2 − θ3, so bricks stay level.	Holds the tool vertical: θ4 = −90° − θ2 − θ3, so bricks stay level.
J5	Wrist roll Spins the jaws onto the brick's short edge (16 mm on a 2-wide brick).	Spins the jaws onto the brick's short edge (16 mm on a 2-wide brick).
J6	Gripper Parallel clamp. Opens to 40 mm, closes to the brick width.	Parallel clamp. Opens to 40 mm, closes to the brick width.

Because J4 keeps the tool vertical, the reach problem collapses to a planar two-link chain. We can then solve it in closed form instead of iterating. With the wrist point at horizontal distance r and height v above the shoulder, the elbow follows from the law of cosines:

cos θ3 = (r² + v² − L1² − L2²) / (2 · L1 · L2)
θ2     = atan2(v, r) − atan2(L2 · sin θ3, L1 + L2 · cos θ3)
θ4     = −90° − θ2 − θ3

Inverse kinematics tells the arm how to reach one brick. Deciding which brick comes next is a planning problem. An autoregressive model writes the build one brick at a time from a text prompt, rejects invalid predictions, and rolls back to the first unstable brick whenever the finished structure fails a stability check.

[ Algorithm 1 // Autoregressive Lego Build & Rollback Stability Inference ]

ALGORITHM 1: Lego Build Inference with Rejection Sampling & Rollback
Input:  Text prompt c; Autoregressive model θ
Output: Mechanically stable LEGO design matrix B

1:  B ← empty LEGO structure;
2:  loop
3:      /* Predict next brick with rejection sampling */
4:      for k = 1, ..., max_rejections do
5:          context ← T ⊕ B.to_text_format();
6:          b ← θ.predict_tokens(context);
7:          if b is valid then break;
8:      end for
9:      B.add_brick(b);
10:     if b contains EOF then // Structure complete
11:         if B is stable or max rollbacks exceeded then return B;
12:         while B is unstable do // Rollback if unstable
13:             I ← indices of unstable bricks in B;
14:             i ← min(I); // Index of first unstable brick
15:             B ← [b_1, ..., b_{i-1}];
16:         end while
17:     end if
18: end loop

[ S3 // Interactive Sim Lab ]

Interactive Sim Lab

Type a structure or pick a preset. A planner turns the prompt into a placement matrix, and the arm, mounted 0.35 m to the left of the baseplate, works through it brick by brick. Each cycle approaches the staging tray, descends, clamps, lifts 10 cm, transfers above the target stud, presses through a 15 mm insertion stroke to seat the brick, then releases and retracts to its ready pose.

The arm is the authentic SO-101 CAD, rendered at 2× so its reach spans the whole baseplate. Watch the telemetry panel. Reaching the far edge of a tall build pushes the two-link chain past 85% of its extension. That is where small joint errors turn into the largest tip errors.

Sandbox // Prompt → Brick Matrix

Lego Assembly Lab

READY · 0/0

BlueprintPYRAMID

Drag to orbit · scroll to zoom · cyan outlines mark unplaced slots · grid = 10 cm

Transport

Arm speed

SO-101 joint telemetry

—

J1 Base pan: —
J2 Shoulder lift: —
J3 Elbow flex: —
J4 Wrist flex: —
J5 Wrist roll: —
J6 Gripper: —

TCP —

Placement matrix

1× 2x2 · 2× 2x4

012x4[-1, 0, 0]QUEUED
022x4[+1, 0, 0]QUEUED
032x2[0, +1, 0]QUEUED

[ S4 // The Force & Dexterity Problem ]

The Force & Dexterity Problem

The obvious fix for a brick that won't seat is to push harder. It is also the wrong one. An SO-101 servo stalls at roughly 1.9 N·m. Spread across a 0.42 m arm held out horizontally, that is only about 4.5 N at the fingertips before the arm lifts its own weight, which is well short of what a stubborn brick needs.

≈ 4.5 N vs 15–35 N

Fingertip force available from shoulder torque at full horizontal extension, compared with the force needed to seat a brick.

Commanding a position below the seat doesn't help either. A position-controlled servo ramps current until it reaches its stall threshold, and then the firmware's overload protection cuts torque. The arm sags, the brick tilts, and the next placement inherits the error.

Successful systems get the force from geometry instead. They fold the forearm near vertical so the push travels down the links and into the table, not through the shoulder motor. They rely on passive compliance, such as rubber fingertip pads, a slightly springy wrist, or a remote-centre-compliance mount, so the brick can self-centre on the stud chamfers. And they use small spiral or wiggle searches that trade brute force for patience. The 15 mm stroke in the simulation is that final phase: the moment the controller stops trusting coordinates and starts trusting contact.

[ Mechanical Breakdown // Fig 1 ]

Insight A
Brick Force Model (Single Brick)
- Gravity pulls the brick straight down.
- Pressing and pulling normal forces act through the top stud connection.
- Horizontal normal forces (±X, ±Y) come from the side walls of the connections.
- Dragging and supporting forces act through the bottom connection.
Insight B
Structural Force Model (Connected Bricks)
- Load transfers down through every stud-and-tube joint into the brick below.
- Adjacent bricks push on each other with horizontal normal forces.
- A structure is stable only when every brick balances under these forces.
Insight C
Automated Assembly Pipeline
- 1. Pick — grasp the brick from the staging tray.
- 2. Place — align over the target stud and press to seat.
- 3. Build — repeat across the standard baseplate until the verified target geometry is complete.

[ S5 // From Toy Bricks to Autonomous Construction ]

From Toy Bricks to Autonomous Construction

Interlocking bricks are a laboratory stand-in for a serious idea: build big things from many identical, self-aligning parts. Each piece carries its own registration features, so a robot only has to get close. The geometry of the joint handles the last millimetre, and errors stop accumulating as the structure grows.

The same logic drives research into robots that assemble lattice structures from modular building blocks, including NASA work on reconfigurable space structures and "digital materials" built from discrete voxels. It drives factory-built modular housing, where robots stack standardised components. It also drives habitat concepts for the Moon and Mars, where bricks made from local regolith would be laid by machines working far beyond a teleoperator's reaction time.

In every case the skill you practised above is the core primitive: perceive a slot, plan a reachable pose, and finish with compliant contact instead of blind force. A robot that can reliably snap a toy brick has solved a small version of autonomous construction.

[ PHYSICAL AI ECOSYSTEM PARTNERS ]

Special thanks to Seeed Studio and Hugging Face for supporting RoboRoots and powering our community builders with open-source physical AI hardware, tabletop manipulators, and edge robotics infrastructure.