An end-to-end experiment connecting Large Language Models to robotic simulators
through the Model Context Protocol — what worked, what broke, and what it unlocks.
Pavan Kumar · March 2026 · Ubuntu 24.04 · HP ZBook Power G11
Executive Summary
In a single session, we constructed a working pipeline where natural language typed into an LLM
is translated into real-time robot commands across multiple physics simulators. The pipeline
chains Claude Code (via Anthropic's Claude) through the Model Context Protocol (MCP) to a
ROS2 bridge, which dispatches commands to both a Gazebo mobile robot and a MuJoCo robotic arm
— simultaneously. NVIDIA Isaac Sim was installed but encountered compatibility issues on
Ubuntu 24.04.
2
Simulators Connected
13+
Live ROS2 Topics
1
MCP Pipeline
The Working Pipeline
Claude Code→MCP Server→rosbridge (9090)→ROS2 Jazzy→Gazebo / MuJoCo
· · ·
The Build Journey
What follows is a chronological account of every step taken — the installations that
succeeded on the first try, the ones that required debugging, and the walls we hit.
Each step is marked with its outcome.
Phase 1 — ROS2 Foundation
The first challenge: Ubuntu 24.04 (Noble) ships with Python 3.12 and is not compatible
with ROS2 Humble (which targets Ubuntu 22.04). The pivot to ROS2 Jazzy
was the correct call and resolved the distribution mismatch.
curl not found
Ubuntu 24.04 minimal doesn't include curl. Quick fix.
sudo apt install -y curl
ROS2 Humble incompatible with Ubuntu 24.04
GPG key errors and missing packages. Humble targets Jammy (22.04), not Noble (24.04). Pivoted to ROS2 Jazzy.
ROS2 Jazzy installed successfully
Full desktop install completed. ros2 --help confirmed working.
sudo apt install -y ros-jazzy-desktop
TurtleBot3 packages installed but not found
Initial apt install with the gazebo-ros-pkgs glob failed, rolling back turtlebot3 packages silently. Had to install them separately.
The ROS MCP Server by robotmcp is the bridge that translates LLM tool calls into
ROS2 WebSocket commands. Getting it running required navigating a project that had
recently restructured from a flat script to a proper Python package.
ros-mcp-server cloned and installed
The project uses pyproject.toml (no requirements.txt). Installed with uv pip install -e .
cli.claude.com doesn't exist. Correct URL is claude.ai/install.sh.
curl -fsSL https://claude.ai/install.sh | bash
Claude Code installed and authenticated
Native binary installer, no Node.js required.
MCP server connected to Claude Code
Claude can now list all ROS topics, publish velocity commands, subscribe to sensor data.
claude mcp add ros-mcp-server uv -- --directory /home/pavan-kumar/ros-mcp-server run server.py
First LLM → Robot command executed
Asked Claude "What ROS topics are available?" — it returned all 13 topics with types. The full pipeline was live.
"What ROS topics are available?" — a natural language question that traverses
Claude Code → MCP → WebSocket → ROS2 and returns live robot data.
· · ·
Phase 3 — MuJoCo Robotic Arm
To demonstrate multi-simulator control, we added a MuJoCo-based 3-DOF robotic arm
that publishes to ROS2 topics. This meant Claude could control two entirely different
robots through the same MCP connection.
MuJoCo 3.5.0 installed
Clean pip install, no issues.
pip install mujoco --break-system-packages
mujoco_ros2_control failed to build
C++ API incompatibility — the ResourceManager constructor changed in Jazzy's ros2_control. The community package hasn't been updated.
Custom Python bridge created
Wrote a Python node that runs MuJoCo simulation headlessly and publishes /mujoco/joint_states, /mujoco/end_effector_pos, subscribes to /mujoco/joint_commands.
python3 ~/mujoco_ros_bridge.py
MuJoCo viewer crashed with segfault
Thread-safety issue when viewer and ROS2 spin access MjData simultaneously. mujoco.viewer.launch() is not thread-safe.
Fixed with launch_passive()
Switched to mujoco.viewer.launch_passive() with viewer.sync() in the ROS spin loop. Arm now visible and controllable.
Dual-simulator control achieved
Claude controlling TurtleBot3 in Gazebo AND the MuJoCo arm simultaneously through one MCP connection.
· · ·
Phase 4 — NVIDIA Isaac Sim (Attempted)
Isaac Sim represents the high end of robotics simulation — photorealistic rendering,
GPU-accelerated physics, native ROS2 bridge. Getting it running on Ubuntu 24.04 with
8GB VRAM proved challenging.
Isaac Sim requires Python 3.11
Ubuntu 24.04 ships Python 3.12. Had to install Python 3.11 and create a separate venv.
Application launched via isaacsim isaacsim.exp.full.kit but kept closing. SimulationApp Python API returned NoneType. RTX 2000 Ada (8GB VRAM) at minimum spec threshold.
ROS2 bridge not loading
No isaacsim-ros2-bridge pip package available. The --enable isaacsim.ros2.bridge flag didn't produce ROS2 topics. Ubuntu 24.04 is not fully supported by Isaac Sim.
· · ·
Final Status Matrix
Component
Status
Version
Notes
ROS2 Jazzy
Working
Jazzy Jalisco
Full desktop install on Ubuntu 24.04 Noble
Gazebo + TurtleBot3
Working
Harmonic
Mobile robot with LiDAR, IMU, odometry — all controllable via Claude
rosbridge
Working
Port 9090
WebSocket bridge between MCP server and ROS2
ROS MCP Server
Working
3.0.1
robotmcp/ros-mcp-server — FastMCP-based, stdio transport
Claude Code
Working
Native installer
LLM client with MCP support on Linux
MuJoCo
Working
3.5.0
3-DOF arm via custom Python-ROS2 bridge with passive viewer
mujoco_ros2_control
Failed
—
C++ API mismatch with Jazzy's ros2_control ResourceManager
NVIDIA Isaac Sim
Partial
5.1.0
Installed, GUI opens but unstable. ROS2 bridge not functional. Ubuntu 24.04 not fully supported, 8GB VRAM at minimum threshold.
· · ·
Working Architecture
The final architecture requires four simultaneous terminal sessions. The LLM never
directly touches ROS2 — it operates through a clean abstraction layer of MCP tools
that translate intent into action.
Ubuntu 24.04 (Noble) is too new for many robotics packages — ROS2 Humble, Isaac Sim, and mujoco_ros2_control all target 22.04
RTX 2000 Ada with 8GB VRAM is at minimum spec for Isaac Sim — complex scenes require 16GB+
No official Claude Desktop for Linux — used Claude Code CLI as the MCP client instead
MuJoCo viewer thread-safety requires launch_passive() pattern, not launch()
Software Ecosystem Constraints
Isaac Sim 5.1 pip package requires Python 3.11 exactly — not 3.10, not 3.12
The mujoco_ros2_control C++ package hasn't been updated for Jazzy's ros2_control API changes
Isaac Sim ROS2 bridge has no standalone pip package — tightly coupled to the Omniverse runtime
rosbridge is the single point of failure — all MCP-to-ROS communication flows through one WebSocket
· · ·
What This Makes Possible
Immediate Capabilities (Working Today)
Natural language control of a mobile robot — "move forward 1 meter, turn left 90 degrees"
Natural language control of a robotic arm — "set shoulder to 1.5 radians"
Real-time sensor reading via conversation — "what does the LiDAR see?"
Multi-robot orchestration — control Gazebo and MuJoCo robots simultaneously
Robot debugging via natural language — browse topics, inspect message types, call services
Near-Term Extensions
Voice-controlled robotics — pipe speech-to-text into Claude Code
Vision-language robot reasoning — subscribe to camera topics, send frames to VLMs
Multi-step autonomous task planning — LLM decomposes "explore the room" into sensor reads and motor commands
Anomaly detection — LLM monitors sensor streams and alerts on unusual readings
Add Blender MCP for 3D environment design + Home Assistant MCP for smart device integration
Two different robots, two different physics engines, one natural language interface.
The protocol layer — MCP — is what makes this composable.
· · ·
Key Takeaways
1. The MCP abstraction is powerful. By placing an MCP server between the LLM and ROS2,
we decouple the intelligence layer from the robot layer. Any MCP-compatible LLM (Claude, GPT, Gemini)
can control any ROS robot without modification.
2. Ubuntu 22.04 is still the safe bet for robotics. Every compatibility issue we hit
— ROS2 Humble, Isaac Sim, mujoco_ros2_control — traces back to being on 24.04. The ecosystem hasn't
fully migrated yet.
3. Python bridges beat C++ packages for rapid prototyping. When the compiled
mujoco_ros2_control package failed, a 90-line Python script achieved the same result in minutes.
For LLM-driven robotics, the speed of iteration matters more than the speed of execution.
4. The ecosystem is nascent but accelerating. The ros-mcp-server project went from
first release in September 2025 to 825+ GitHub stars by March 2026. Research papers on "Agentic
Embodied AI" via MCP are appearing monthly. This is early, but the direction is clear.