MMSW v59: From Multimodal Architecture to Robotics
MMSW — Multicasting Multimodal Story Worlds — reaches a new architectural milestone with version v59, incorporating robotic systems into a distributed infrastructure that connects entities, agents, memory, media, and shared states.
The innovation is not simply about connecting a robotic arm to artificial intelligence, but about establishing an architecture capable of translating intentions into bounded, observable, and verifiable actions.
This advance expands the role of World Models and Digital Twins in the relationship between perception, decision-making, and physical action.
From Story Worlds to physical action
In previous versions of MMSW, the architecture enabled the coordination of agents, communication, memory, media, and tools across distributed multimodal environments. The integration of robotics introduces new responsibilities related to executing actions in the physical world.
With version v59, robotics becomes an explicit participant in the system graph. Dedicated protocols, adapters, access control, execution queues, task logs, and verification mechanisms make it possible to distinguish a communicated intention, an issued command, and an observed outcome.
The system also begins exploring multiple transport and integration methods, retaining WebSocket as its central mechanism while establishing interfaces for HTTP, optional MQTT, serial bridges, and ROS 2.
This evolution does not replace the previous multimodal capabilities; it extends them toward embodied interaction and measurable physical action.
The MMSW v59 architectural milestone
The following comparison presents eight dimensions of MMSW’s evolution, from the previously documented multimodal runtime to the robotics-enabled architecture of version v59.
| Dimension | Previous MMSW | MMSW v59 — Robotics |
|---|---|---|
| Primary role of the system | Distributed multimodal Story Worlds runtime coordinating humans, agents, memory, media, tools, and shared states. | Distributed, embodied multimodal runtime in which robotics becomes an explicit participant in the system graph. |
| Embodiment | Embodied interfaces and robotics were part of the system’s conceptual and deployment horizon. | Explicit robotic protocols, adapters, components, ownership, control leases, queues, task logs, observation, verification, and recovery. |
| Action boundaries | Plugin and tool actions remained separate from conversational speech and were confirmed through orchestration logs. | Natural-language intentions are translated into bounded robotic commands. Execution, evidence, terminal state, and narrative remain explicitly separated. |
| Network transport | WebSocket-centered multicast runtime with HTTP services and plugin interfaces. | WebSocket remains the default, with HTTP compatibility, optional hybrid MQTT, serial bridges, and ROS 2 integration interfaces, allowing heterogeneous devices to coexist. |
| Physical and simulated worlds | External devices and embodied media could be connected to the Story Worlds runtime. | Digital Twins become execution and validation layers for the Franka Panda and Waveshare-class robotic arms, keeping physical outcomes distinct from simulated outcomes. |
| Perception and evidence | Multimodal perception and external-state checks grounded media and tool responses. | Cameras, scene state, telemetry, execution feedback, and terminal logs establish an evidence trail between embodied action and the shared world. |
| Relationship with AI entities | Persistent identity and multimodal expression supported the transition from bots to agents and entities. | A robotic subsystem can function as a connected extension of an AI entity’s embodied presence while maintaining independent control, observability, and verifiability. |
| Repository scale* | 114 files; 34,733 raw lines of code; approximately 27,150 lines of source code; 44 unique routes; 158 asynchronous functions. | 339 measured source files; 150,583 raw lines; 127 unique HTTP/WebSocket method-and-path pairs; 458 asynchronous Python functions. The robotics subsystem accounts for 156 files and 24,115 raw lines within the measured scope. |
Table 1. MMSW’s architectural evolution: from the previously documented multimodal runtime to the v59 robotics milestone.
* These figures come from specific repository surveys with different measurement scopes. Code metrics alone are not indicators of performance.
Celestia Borealis and the robotic extension of presence
One of the questions explored at this stage is how a synthetic entity can extend its presence into physical space without its identity being confused with the device through which it acts.
Celestia Borealis is a non-human entity integrated into MMSW. Aster, a Franka Panda robotic arm, serves as an extension of her capacity to act in physical and simulated environments.
When interpreting an instruction such as “pick up the red cube from the side”, Celestia must consider the spatial configuration, obstacles, and available opportunities for action — the affordances — before determining an execution strategy.
The robotic arm performs movements through dedicated interfaces, while the architecture keeps interpretation, planning, execution, and verification as distinct processes.
Digital Twins and execution evidence
Digital Twins make it possible to explore movement strategies in simulated environments, observe changes in configuration, and test different operating conditions.
However, a successful action in simulation does not automatically constitute proof of physical execution. Distinguishing these two states is an important part of the architecture.
Version v59 introduces mechanisms for recording observations, telemetry, execution feedback, and terminal outcomes, allowing issued commands to be related to observable evidence.
This process broadens the possibilities for investigating systems in which synthetic entities express their presence through different devices, while preserving traceability and control boundaries.
Presentation at VISGRAF/IMPA — October 14
These advances will be discussed at the upcoming VISGRAF/IMPA Computer Graphics Seminar on October 14, 2026.
Matteo Moriconi, VFXRio co-founder and VISGRAF collaborating researcher, will present the work alongside Luiz Velho, a senior researcher and professor at IMPA, lead researcher at VISGRAF, and one of the pioneers of computer graphics in Brazil.
The seminar will address World Models, transmultimodality, Digital Twins, affordances, human–synthetic interaction, and robotic extensions of non-human entities, with examples involving MMSW, Celestia Borealis, and Aster.
October 14, 2026
1:30–3:00 p.m. — Brasília time
Auditorium 3 — IMPA, Rio de Janeiro
A new stage of research
The robotics integration in version v59 marks a milestone in MMSW’s evolution, extending the concept of shared multimodal worlds to systems in which physical action can also participate in a distributed architecture.
The challenge is not merely to enable a synthetic entity to control a robot, but to understand how perception, world representation, intention, action, and evidence can remain connected in a coherent and verifiable way.
Related research: