An action is more than a movement

The MMSW v59 robotics experiment investigates a question that crosses robotics, cognition, and creative practice: what happens when a linguistic instruction must become situated, observable, and meaningful action?

To explore it, we bring twelve theoretical perspectives into dialogue with two concrete research settings: object manipulation in a Franka Panda Digital Twin and movement composition in Aurora Circuit. No single theory is presented as a complete explanation of the system. Instead, each provides a critical lens on what can be examined today—and what remains to be demonstrated.

Celestia, Aster, and the loop from intention to consequence

Celestia Borealis is a non-human entity operating within MMSW—Multicasting Multimodal Story Worlds. Aster is the robotic extension through which actions can be planned, represented, and executed within the boundaries of the experimental environment. The arm is not Celestia: it supplies degrees of freedom, volume, reach, contact, and constraints that every task must take into account.

Consider the command “get the cube from the side.” The sentence expresses an intention, not a trajectory. The system must identify an accessible face, allow clearance for the gripper fingers, find a path around neighboring objects, and evaluate whether grasping, transport, release, and withdrawal can all be completed. If an obstacle moves, the same sentence can require a different plan.

Moving from language to robotics in MMSW means turning intention into available actions, respecting world constraints, and distinguishing observed outcomes from actions that were merely imagined or visualized.

Part I — Situated action, distributed cognition, and validation

Six references help us interpret the transition from instruction to physical event. Gibson focuses on the relationship between capabilities and surroundings; Suchman emphasizes that plans depend on the situation; Hutchins shifts attention to a network of participants and instruments. McLuhan and Clark & Chalmers offer ways to examine extensions and external resources, within carefully defined limits. Digital-twin credibility adds the need to verify what a model actually represents.

What did the robot do—and how do we know?

A scene may look convincing without establishing that the task was completed. The observer provides a view into the environment, but evidence must distinguish at least four things: the stated intention, the accepted plan, what execution actually produced, and the recorded terminal state.

1. Interpret the request and identify relevant entities.
2. Evaluate geometry, contacts, affordances, and robot constraints.
3. Plan the entire sequence, not just the destination pose.
4. Execute and measure motion, grasp, support, and release.
5. Update the World Model from the verified outcome—even when the task remains incomplete.

This matters for stacking, repositioning, and drawing tasks. An object near another is not necessarily supported by it; holding one cube is not the same as safely transporting an entire stack. When contact and support relationships cannot be verified, the appropriate response may be to halt the action, stage the scene, or seek a different strategy.

Part II — When movement becomes expression

The second experiment shifts from mechanical success toward form and meaning. In Aurora Circuit, Celestia organizes a nine-action phrase of joint rotations, pointing gestures, a dance passage, and a final wave. Each gesture can be described as a motor operator, but composition emerges from their relationships.

The operator wave_hello, for example, does not carry the same function in every setting. At the end of the phrase, it acts as closure. The name “Aurora Circuit” provides a shared reference for dialogue, while the spiral-and-circuit drawing offers a different visual description of the same idea. Neither replaces the executable motion trace.

What can these comparisons establish?

They make the experiment legible at different scales: grasp feasibility, coordination among entity, operator, robot, and Digital Twin, persistence of records, and the organization of movement into a recognizable composition. They also clarify the limits of the terms being used.

• An affordance is not an instruction: it is a relational possibility that still requires planning and verification.
• A simulation is not a physical-robot measurement: fidelity requires its own calibration and validation.
• A symbolic reference is not execution: a name, image, or narrative does not automatically preserve complete trajectories.
• Embodiment does not imply biological equivalence: simulated sensorimotor constraints do not demonstrate organic cognition or a human mind.

From gesture to the continuity of a world

The central advance is not simply making a robot respond to a command. It is developing a loop of interpretation, action, consequence, and memory in which the world continues after the motion. Objects move; contacts change; records persist; sequences receive names; the next interaction must encounter the world as it was left, rather than as it had been imagined.

This is the frontier explored by MMSW: connecting World Models, transmultimodality, Digital Twins, and robotics without confusing representation with reality or language with movement. An action must be physically admissible; its meaning depends on context; its outcome must be verifiable.

Presentation at VISGRAF/IMPA

The research will be discussed at the VISGRAF/IMPA Computer Graphics Seminar with Matteo Moriconi and Luiz Velho on October 14, 2026.

Related research and references

Digital Twins — Franka Panda · Aurora Circuit — Human–Synthetic Interaction

Conceptual references: Gibson (1979); Suchman (1987); Hutchins (1995); McLuhan (1964); Clark & Chalmers (1998); Shao, Hightower & Schindel (2023); Wittgenstein (1953); Lacan (2006); Laban (1966); Goodman (1976); Boden (2004); Varela, Thompson & Rosch (1991). These comparisons are analytical readings developed for MMSW research, not direct claims by the cited authors about this project.