Can Your AI Engineer a Robot? A 48-Task Test Says Not Yet

TL;DR: AI Can Write Robot Code, But Physics Still Trips It Up

TL;DR: On September 17, 2026, Harvard and Georgia Tech released RLE-Bench, an open-source test of 48 engineering tasks that asks general-purpose coding agents to build and operate physical robots.

Researchers at the Harvard John A. Paulson School of Engineering and Applied Sciences (SEAS) and the Georgia Tech School of Computational Science and Engineering developed the evaluation tool to measure how well AI coding agents can perform the work of engineering an actual, physical robot, according to the Harvard SEAS report titled Can Your AI Engineer a Robot?. The team was led by Na Li, the Winokur Family Professor of Electrical Engineering and Applied Mathematics at Harvard SEAS, and Bo Dai, assistant professor at Georgia Tech, and included Harvard graduate student Haitong Ma and Chenxiao Gao and Rushi Qiang at Georgia Tech.

The benchmark, known as RLE-Bench, functions as a standardized test for an AI robot learning engineer, a role that combines machine learning, robotics, control, software, and hardware. The release was described on September 17, 2026, by Harvard SEAS press contact Anne J. Manning.

RLE-Bench contains 48 tasks that examine whether general-purpose coding agents can perform the kinds of engineering work required to build and operate robotic systems. Tasks range from developing control and perception algorithms to designing robot hardware and interfaces.

The tasks run in simulated environments where agents can generate, execute, inspect, and refine their solutions. The problems are grounded in physical constraints, so a design that appears functional may become unstable once a payload is added or fail because the agent has not correctly accounted for mass, torque, or geometry.

In one example task, an agent must design a universal mobile base for multiple robotic arms that needs to reach objects at different heights on a shelf. A proposed design may allow the arm to reach every target while still failing certain stability tests where the complete robot would tip over.

A Standardized Exam For Robot Learning Engineers

RLE-Bench organizes its tasks into four broad areas: interactive control, policy development, perception and estimation, and mechanical design. Together, the areas test not only an agent's ability to generate code, but also its ability to reason about sensing, dynamics, stability, hardware constraints, and the interaction between software and the physical world.

The benchmark description compares the tool to a standardized test for an AI robot learning engineer. That role combines machine learning, robotics, control, software, and hardware, according to the Harvard SEAS report.

The four areas were selected to cover complementary parts of the engineering workflow. Interactive control addresses operating robots in closed-loop settings. Policy development addresses learning control policies. Perception and estimation addresses sensing and state estimation. Mechanical design addresses hardware and physical structure.

Harvard computer scientists released RLE-Bench as an evaluation tool that tests how well AI coding agents can perform the task of engineering a physical robot. The open-source benchmark allows researchers and developers to evaluate new and existing AI systems on a common set of robotic tasks.

The researchers view the current release as a foundation for a broader community benchmark. Robot learning engineers work on many problems that are not yet represented in RLE-Bench, including new forms of hardware design, simulation, system integration, debugging, safety, and deployment.

Na Li said RLE-Bench 1.0 is intended as a starting point and invited contributions for expansion. The stated goal for RLE-Bench 2.0 is to capture a much broader picture of what an AI system needs to do to function as a robot learning engineer. The team said it hopes researchers and robotics practitioners will contribute tasks based on the engineering problems they encounter in their own work.

Why Coding Agents Face A Harder Test In Robotics

With the rapid rise of artificial intelligence in daily life, software coding has become increasingly automated, with powerful AI systems known as coding agents able to write and revise computer programs almost autonomously. The Harvard SEAS report poses the follow-up question of what happens when an AI agent must contend not just with digital command lines, but with the physical world of robotics.

Na Li said benchmarks give the field a common measuring stick, allowing comparison of systems under the same conditions and identification of where important capability gaps remain. Li said robotics already has many valuable benchmarks, but many focus primarily on learning and evaluating control policies. Li said RLE-Bench takes a broader view and asks whether an AI system can perform the range of engineering tasks required to make a robotic system actually work.

That broader view includes sensing, dynamics, stability, hardware constraints, and the interaction between software and the physical world. The benchmark tests code generation together with reasoning about those physical factors.

Haitong Ma said physical reasoning becomes critical because an agent can produce something that looks reasonable computationally, but once the physics of the complete system is considered, important failure modes can appear. The mobile-base illustration released with the benchmark shows a coding agent engineering a mobile manipulator base that tips over.

A separate LinkedIn summary of the project by Haitong Ma on September 14, 2026, said the team started exploring the question a few months earlier with collaborators Chenxiao Gao, Rushi Qiang and advisors Bo Dai and Na Li. That summary described interest not only in a new control paradigm beyond VLAs and world models, but in the broader question of whether coding agents can improve the entire robotic system.

The same summary listed the four complementary areas as interactive control and embodied reasoning, policy learning and deployment, perception and estimation, and mechanical design and co-design. It said results were uneven across models, with GPT-6 Astra particularly strong at visual grounding, perception, and interactive control compared with other models. It said mechanical design exposed a different limitation, with agents producing solutions that looked reasonable while missing basic physical effects such as stability, load, contact, and dynamics.

Inside The 48 Tasks And Four Workflows

RLE-Bench contains 48 tasks that test whether general-purpose coding agents can perform engineering work required to build and operate robotic systems. The tasks span control and perception algorithms as well as robot hardware and interface design.

The tasks run in simulated environments where agents can generate, execute, inspect, and refine their solutions. Agents work under task-specific limits in the broader project description, and their work is evaluated under hidden physical conditions in independent testing described by collaborators.

A September 14, 2026, collaborator summary by Lening Li described RLE-Bench as a qualifying exam for coding agents as robot learning engineers with 48 tasks, 9 families, and 4 workflows spanning the arc of the job. That summary listed interactive control as operating the robot and building a harness for another agent to reuse zero-shot. It listed policy development as humanoid motion tracking and VLA training recipes tested on transfer. It listed perception and estimation as pose under occlusion and bin clearing with visual and force feedback. It listed mechanical design as mobile bases and gravity-compensated GELLO arms scored on unseen physical instances.

The same collaborator summary said visual grounding separated the systems, with the gap on perception much wider than the gap on policy learning, where leading models clustered close together. It said one agent's mobile base earned full credit for shelf access and payload margin, then scored zero on static stability and lateral and turning checks. It said more scaffolding in the form of perception models, transforms, and motion primitives lifted most systems, but the strongest low-level performer performed worse with the extra harness.

Another collaborator note from Haitong Ma said the harder question is whether an agent can handle the full engineering loop of observe, reason, act, and reflect, and make the right decisions when the physical world pushes back. The note said the interesting question is no longer only whether AI can control a robot, because that capability is increasingly present.

When Reaching The Target Is Not Enough

The universal mobile base example illustrates why physical reasoning is central to RLE-Bench. In that task, an agent must design a universal mobile base for multiple robotic arms that needs to reach objects at different heights on a shelf.

A proposed design may allow the arm to reach every target while still failing certain stability tests where the complete robot would tip over. The Harvard SEAS report presents that outcome as an illustration of failure modes that appear only after the physics of the complete system is considered.

More generally, a design that appears functional may become unstable once a payload is added. A design may also fail because the agent has not correctly accounted for mass, torque, or geometry.

Haitong Ma said an agent can produce something that looks reasonable computationally, but important failure modes can appear once the physics of the complete system is considered. That statement was made in connection with the mobile-manipulator tipping illustration.

The collaborator summary from September 14, 2026, said mechanical design exposes limitations around stability, load, contact, and dynamics. It said agents can produce solutions that look reasonable while still missing those basic physical effects.

The September 14 collaborator analysis by Lening Li gave a concrete scoring illustration in which a mobile base earned full credit for shelf access and payload margin and then scored zero on static stability and lateral and turning checks. That analysis said reaching the target is only part of the problem, and that a design must also anticipate how it behaves under load and motion.

A Parallel Advance On Hard Safety Constraints

A separate MIT development reported on September 14, 2026, addresses a related requirement for AI systems that act in the physical world: satisfying nonnegotiable safety, physical, or task-specific requirements known as hard constraints, according to the MIT News report titled New method enables AI for safety-critical situations.

MIT researchers developed a method called HardFlow that helps generative AI models meet strict requirements without sacrificing the quality of their outputs. The technique gives the model more freedom during the generation process and enforces hard constraints on the final output, rather than at every intermediate step.

HardFlow was developed by lead author Zeyang Li, a graduate student in mechanical engineering and the Laboratory for Information and Decision Systems (LIDS), Kaveh Alim, a graduate student in the Institute for Data, Systems, and Society (IDSS) and LIDS, and senior author Navid Azizan, the Alfred H. and Jean M. Hayes Career Development Associate Professor in the Department of Mechanical Engineering and IDSS and a principal investigator of LIDS. The research appears in the IEEE Transactions on Pattern Analysis and Machine Intelligence.

The MIT News report said the adaptable, plug-and-play technique works at deployment time, so it can be applied to pretrained generative models without retraining them. It said the method can make such models more useful in applications where safety rules, physical laws, or other strict requirements cannot be violated.

In experiments spanning robotics, control of physical processes, and computer vision, the new method consistently satisfied the required constraints while identifying better solutions than existing techniques. The report describes applications spanning robotics, control of physical systems, and computer vision.

Pretrained generative AI models such as diffusion models like Stable Diffusion and flow-matching models like FLUX learn to create new data by transforming random noise. The MIT News report says those highly capable models excel at providing answers that come close to satisfying most queries, but in safety-critical applications like robot path planning on a crowded factory floor, an answer that is nearly correct may not be good enough.

For instance, a nearly correct path from one machine to another might still result in the robot colliding with a human co-worker. The report presents that collision example to explain why plausible answers are insufficient when hard constraints govern safety.

Users often employ a technique called projection-based sampling, which repeatedly forces the model's partial solutions, called intermediate samples, to satisfy strict requirements during the generation process. The MIT News report says constraining the entire generation process can prevent the model from reaching a better final solution. It says those methods also typically focus only on satisfying the hard constraints, missing the opportunity to improve other qualities of the solution, like reducing the length of the robot's trajectory.

Zeyang Li said what ultimately matters for constraint satisfaction is the model's final output, since the internal process is discarded. Li said not requiring every intermediate step to satisfy the constraints gives the model more freedom to find high-quality solutions that are still feasible in the end.

HardFlow reformulates hard-constrained sampling as a trajectory-optimization problem, using tools from optimal control. The framework steers the model's sampling trajectory toward a goal, making subtle corrections along the way while enforcing hard constraints on the final output.

Navid Azizan said control theory provides a framework for formalizing the optimal way of making those corrections. Azizan said the promise of generative AI is its ability to explore a rich space of possibilities, but the real world places boundaries on which possibilities are acceptable. Azizan said the approach preserves generative power while enforcing the nonnegotiable requirements of high-stakes or safety-critical applications.

To make the problem tractable, the researchers leveraged the structure of flow-matching models to decompose the problem into a sequence of smaller, single-step subproblems. The model may have hundreds of interconnected layers that process data, and solving the trajectory-optimization problem around that enormous neural network required decomposition.

What RLE-Bench 1.0 Leaves Open

RLE-Bench is open source and publicly available, allowing researchers to evaluate new AI systems on a common set of tasks. The Harvard SEAS report presents that availability as a way for researchers and developers to evaluate new and existing AI systems on the same robotic tasks.

The researchers view the current release as a foundation for a broader community benchmark. The current 48 tasks do not yet represent many problems that robot learning engineers encounter, including new forms of hardware design, simulation, system integration, debugging, safety, and deployment.

Li said RLE-Bench 1.0 is only a starting point and said the team hopes researchers and robotics practitioners will contribute tasks based on engineering problems they encounter in their own work. The goal for RLE-Bench 2.0 is to capture a much broader picture of what an AI system needs to do to function as a robot learning engineer.

The September 14, 2026, project summary said the team built RLE-Bench after starting work a few months earlier with collaborators Chenxiao Gao and Rushi Qiang and advisors Bo Dai and Na Li. The summary said the team was interested in whether coding agents can improve the entire robotic system, beyond vision-language-action models and world models.

The press contact for the Harvard SEAS announcement is Anne J. Manning at [email protected]. The school lists addresses at 150 Western Ave, Allston, MA 02134 and 29 Oxford Street, Cambridge, MA 02138.

For readers tracking AI for robotics, the two September 2026 developments point to complementary test and method needs. RLE-Bench provides a common measuring stick for engineering tasks across interactive control, policy development, perception and estimation, and mechanical design. HardFlow provides a deployment-time method for steering pretrained generative models so the final output satisfies hard constraints while improving solution quality.