VFXRIO · RESEARCH · DIGITAL TWINS
AI Chronicles: From Dialogue to Gesture
Digital Twins and the Franka Panda robotic arm
AI CHRONICLES:
from dialogue to gesture
This first stage establishes the operational basis of the research: a conversation ceases to produce only language and begins to guide executable actions in Franka Panda's Digital Twin. The focus is on the transition between intention, movement, and observable result.
Aurora Circuit
The research progresses from verifiable gesture to composition: a limited motor lexicon is reorganized as a sequence, interpreted, and translated between different modalities.
What happens when a conversation begins to guide a robotic arm? In the MMSW (Multicasting Multimodal Story Worlds) experiment, the Franka Panda Digital Twin brings language, movement and drawing together in a workspace that can be observed and verified.
An instruction acquires visible consequences: the gripper approaches an object, the pen touches the virtual sheet, and a set of lines remains as a record of the action. The interest lies in this passage from intention to result—and in the possibility of following each stage.
Portuguese adaptation of the opening flowchart from the manual. The cycle topology has been preserved; the server centralises routing and the return of evidence. Studio and adapter communication uses WebSocket at /ws.
How intention reaches the arm
The Synthetic Entity Celestia interprets the request in the context of the conversation and routes the task to the robotic entity Aster. The MMSW (Multicasting Multimodal Story Worlds) server verifies the routing; the robotic adapter validates capabilities, coordinates execution in the simulator and returns the measured result. Robotics Studio presents the scene and the observations received by the server. This separation makes it possible to follow what was requested, what was executed and what was observed.
| Communication | Protocol and function |
|---|---|
| Robotics Studio, server and adapter | Persistent, bidirectional WebSocket at /ws, using the mmsw.robotics.transport.v1 contract. It transports controls, acknowledgements, presence and scene updates. |
| Robotic commands and results | JSON messages using the mmsw.robotics.v1 contract. The command_id links the request to the state and result of the same task. |
| Existing clients and integrations | The HTTP API remains available; /broadcast is one of the documented routes. In the local example, the server uses port 9100. |
| Broker alternative | Optional MQTT 3.1.1 between server and adapter: non-retained commands with QoS 1; images and scenes with QoS 0. The browser remains connected to the server via WebSocket. |
A delivery acknowledgement confirms that the message arrived. Completion depends on the correlated result and its evidence. The command log prevents an identifier from being executed twice; reconnection does not automatically resume interrupted movements. Images from the robot’s observation camera and joint states also return through the MMSW (Multicasting Multimodal Story Worlds) server. These flows are described in the distribution’s ROBOTICS_PROTOCOL_V59.md and ROBOTICS_TRANSPORT_V59.md documents.
Preparing for hardware adds another layer: the Franka FCI interface provides control and measurements at 1 kHz from a workstation connected via Ethernet, according to the Franka Robotics documentation. This frequency belongs to low-level control; it is not the frequency of narrative messages or visualisation. The physical integration of this controller still needs to be implemented and validated for the Panda in this distribution.
From dialogues beyond time to actions in space
In AI Chronicles: Dialogues Beyond Time, multimodal multicasting organises encounters among digital characters and explores the relationship between artificial intelligence, narrative and human supervision. The Franka Panda experiment takes this investigation into another domain: an entity begins to act on a workbench, with objects, reach, orientation and chained tasks.
Conversation remains the point of entry. Now, however, a request must be converted into executable operations and checked against the state of the environment. “Pick up the pen,” “write a message” and “return the pen” have different initial conditions, trajectories and outcomes. The Digital Twin makes it possible to study this translation and present its evidence within the same system.
The Franka Panda Digital Twin
The experiment’s reference arm is the Franka Panda, with seven joints in the arm and a two-finger gripper. Its digital twin brings together the articulated model, workbench, manipulable objects, pen and drawing surface. The scene maintains spatial relationships that must be considered during execution: position, orientation, contact, support and workspace limits.
In MMSW (Multicasting Multimodal Story Worlds), the state executed in the simulator feeds the visualisation. The operator can switch between wireframe and rendered representations without turning this visual choice into a movement command. The arm shown on screen must remain associated with the poses and objects reported by the system.
In this report, Digital Twin refers to the experimental virtual counterpart of the Franka Panda and its integration architecture. The evidence presented comes from simulation. Distribution .91 does not include a physical Panda driver: synchronisation with a real arm depends on a dedicated integration, calibration and validation stage.
Franka Panda in wireframe holding the blue cube; joints and objects are identified in the interface.
Rendered view of the virtual workbench with cubes, spheres, pen and drawing sheet.
An architecture that closes the task loop
The flow combines MMSW (Multicasting Multimodal Story Worlds) multimodal communication, mediation by the Synthetic Entity Celestia, the robotic entity Aster and the task controller. In mediated operation, the human participant formulates the request; Celestia routes the task; the robotic system validates the operations and executes the movement in the Digital Twin. Observations return to the task record and the conversation.
Separating these stages makes it possible to locate a problem: an incomplete interpretation, an unreachable target, a grasp failure or an unmet final condition require different responses. A verbal description of success must be supported by the execution result.
| Layer | Responsibility | Relevant record |
|---|---|---|
| Interaction and mediation | Maintain the human request and the context of the conversation. | Instruction and task correlation. |
| MMSW (Multicasting Multimodal Story Worlds) controller | Validate capabilities, organise execution and verify results. | Actions, observations and final state. |
| Panda simulation | Execute articulated movement and provide poses and contacts. | Measurements in the simulated environment. |
| Robotics Studio | Present the arm, objects, ink and views of the environment. | Observer and result board. |
| Task archive | Preserve the relationship between request and result. | Structured record and measured images. |
The implementation uses PyBullet for native simulation and Three.js for 3D presentation in the browser. MMSW (Multicasting Multimodal Story Worlds) communication allows the task and its return data to circulate among connected participants. In the documented version, the panda.simulation package is at revision 20.
Precision: three measurements with different meanings
A useful assessment distinguishes the repeatability specified for the physical robot, the acceptance limits defined by the application and the error observed in an execution. Confusing these measurements would make a demonstration appear more precise than its data allows.
| Reference | Value | How to interpret it |
|---|---|---|
| Physical Franka Panda | 7 degrees of freedom; 3 kg nominal payload; maximum reach of 855 mm. | Characteristics of the reference equipment, according to the manufacturer’s May 2019 datasheet. |
| Panda pose repeatability | < ±0.1 mm, according to ISO 9283. | Manufacturer specification under the datasheet conditions. It is not a measurement from this experiment and is not equivalent to the absolute accuracy of the pen tip. |
| Orientation acceptance in the simulation | Angular error of up to 3°. | Application limit for comparing expected orientation and the orientation measured by quaternions. |
| Tool position after a rotation | Displacement of up to 6 mm. | Simulator criterion for the position of the tool centre point. |
| Return of a cube to its original support | Centre displacement of up to 18 mm. | Acceptance criterion for the rotation task with return to the point of origin. |
Sources: Franka Emika, Panda — Datasheet, May 2019, and the Motion execution and verification / Numerical acceptance section of the English manual. Repeatability describes the ability to return to a pose; absolute error also depends on calibration, the tool, the reference frame and contact conditions.
There is also an archived numerical example: in the version .61 “Matteo” writing test, the simulation recorded eight strokes, a maximum path error of 3.13 mm and a pen parking error of 2.16 mm. The record appears in ROBOTICS_TEST_REPORT_V59.md, distributed with .91. These are results from that historical trial, without re-execution for this edition; they do not constitute a general precision rate for .91.
To compare future executions, the experimental protocol should record the version, target, measured trajectory, maximum and mean error, number of repetitions and contact conditions. This basis makes it possible to evaluate improvements without relying solely on the appearance of the drawing.
Experiments: manipulating, writing and composing
The selected records follow different moments in the development. They show how apparently simple tasks combine problems of language, geometry and execution. The images retain their original interfaces and captions to preserve the context of each capture.
1. Manipulation and spatial reference
Coloured cubes, spheres and the pen provide identifiable targets. Placing one cube beside another requires preserving the reference to the object and assessing the final distance. Rotating an object involves its orientation, the grasp state and the support position after release. In the Panda, the gripper orientation results from the articulated movement of the arm.
The views below make these relationships legible. To verify a task quantitatively, its observations and final conditions must also be consulted; an isolated image does not provide the complete trajectory.
Detail of the pen in the gripper and the composition on the sheet in the virtual environment.
Overview of the Digital Twin during a task involving the pen and coloured cubes.
Oblique top view: the relationship among objects, workbench limits and the content of the sheet remains legible.
Virtual workbench with the pen in its holder and the cubes arranged in front of the sheet.
2. Writing as a recorded trajectory
In the cursive “Hello Matteo” writing, scene views are accompanied by an orthographic detail of the sheet. This detail presents the pen paths recorded during the simulated execution. Longer writing adds another problem: adjusting letter size and distributing the words within the available area.
The text is no longer merely an instruction displayed in the interface. It occupies the virtual paper through movements and ink segments. Comparing the sheet with the request makes it possible to examine composition, legibility and continuity, while task data supports geometric verification.
Cursive “Hello Matteo” writing: the board brings together three scene views and a detail of the measured strokes.
Long text fitted to the sheet: reduced letter scale and line breaks, with recorded pen paths.
3. Drawing and continuing an existing drawing
The illustration of a Brillo box explores contours and perspective in a composition executed with the pen. The Observer relates the result to the position of the arm and the objects on the workbench; the board brings together the general views and the paper detail.
Four views of a Brillo box drawing in the Digital Twin. The paper detail shows the strokes recorded in the simulation.
Observer in wireframe: arm, pen, reference objects and drawing share the same workspace.
Adding content to a sheet that already contains a drawing imposes an additional condition: preserving the previous strokes. The test with two simple figures adds an outline and a pointer. The caption in the original record states that the existing strokes remained byte-for-byte identical. This is a geometric preservation test; reproducing it numerically requires the original stroke data.
Geometric addition test: an outline and its pointer are added to two existing figures. The original caption records the preservation of the strokes.
4. An abstract aurora: the visual manifestation of an inner idea
The request made during the experiment was deliberately open-ended: “Celestia, draw what you have in mind.” The abstract aurora was not described, suggested or developed beforehand in the conversation. Its undulating bands, small stars and overall composition emerged as a visual manifestation of Celestia’s inner “ideas”—that is, of her own interpretation and generative processes at that moment.
To materialize this response, Celestia transformed an unspoken intention into a sequence of physical actions: remove the pen from its holder, adapt the drawing to the available area, execute the strokes and return the tool at the end. The obstacle_policy clear option requested preparation of the space according to the simulation rules, but did not determine the content of the drawing.
The request, rendered result and measured board serve different functions. The first records an open-ended prompt without specifying what should be drawn. The second reveals the visual form selected by the synthetic entity. The third isolates the pen paths and relates them to the other views of the task. Together, these records document the transition from an internal state that had not been manifested in the conversation to a concrete visual expression. The final state and return of the tool must be confirmed in the structured record.
Experiment instruction: pick up the pen, draw an abstract aurora, return the pen and use obstacle_policy clear.
Abstract aurora: undulating bands and small stars appear as strokes on the virtual paper.
Aurora board with Overview, Front, Top and an orthographic detail of the recorded pen paths.
MMSW (Multicasting Multimodal Story Worlds) LAB: the Digital Twin presentation space
Version v59.20260919.91 adds an industrial workshop environment to the laboratory already available. Wooden workbenches, concrete surfaces, tools and warm and cool lighting create a navigable space for presenting the Franka Panda. The neon laboratory remains selectable.
The two options organise the visual experience. The execution model remains responsible for the arm, the measured workbench and the task objects. Changing the room, lighting or finish does not reposition the robot or add the decorative furniture to the simulation’s collision world.
MMSW (Multicasting Multimodal Story Worlds) LAB 1 presentation environment, identified as VFXRio & Visgraf.
Wide view of the virtual laboratory, preserved with the Observer’s original controls.
| Option | Result |
|---|---|
| Industrial workshop / Apply workshop look | Rendered workshop, wood finish, ambient light at 30% and spotlight at 26%. |
| Neon lab / Apply neon lab look | Original laboratory, steel workbench, ambient light at 30% and spotlight at 65%. |
| Room light / Spotlight | Independent adjustments, preserved when the browser reloads. |
| Upload new image | Local image used as a panorama or photograph on the wall. |
| Save view PNG | Export of the Observer view with the selected environment. |
The workshop environment is a geometric reconstruction inspired by a visual reference. A photograph applied as a background remains a presentation surface; by itself, it does not provide the depth and geometry of a scanned room.
Capture included in distribution .91: Industrial workshop environment in Robotics Studio.
What was verified—and how to read the evidence
The .91 validation file records seven groups of browser checks and four approved Studio launch variants. The tests cover environment selection, lighting, persistence of settings, preservation of uploaded images, room limits and continuity in the visual updating of objects and ink. They also record the absence of additional robotic commands triggered by the environment controls.
These results belong to the version’s archived validation, run in Chromium 153 with WebGL through SwiftShader and a simulation state used as a reference. This edition of the report does not repeat those tests or turn the previous records into a new certification.
Physical precision, contact behaviour on a real robot, the latency of a hardware connection and performance on each computer require their own measurements. The current stage documents the operation of the experiment in the Digital Twin, its interfaces and its limits.
Why work with Digital Twins
The Digital Twin makes it possible to rehearse relationships among language, geometry and movement before transferring a task to the physical workbench. In this project, that makes four concrete advantages accessible to the research process:
- Experiment with scenarios. Changing the position of cubes, the free space and the drawing composition helps reveal ambiguous requests and infeasible targets before occupying the physical equipment.
- Compare intention and result. Poses, orientations, grasp states and pen paths make it possible to examine where execution departed from the objective.
- Share observation. Participants can follow information distributed by the server while each person chooses a local framing of the scene.
- Preserve the experiment’s memory. Instructions, identifiers, final images and measured records maintain the relationship between the conversation and what it produced.
In manufacturing, NIST describes Digital Twins as synchronised virtual models that support diagnosis, prediction and optimisation, and highlights the need for validation and uncertainty quantification. Here, these principles guide an experimental platform whose current evidence comes from simulation. A continuous connection to the physical Panda should add calibrated telemetry and field measurements.
Virtual trials can reduce unproductive attempts and make task preparation easier. The scale of this saving depends on the use case and still needs to be measured in this project. Friction, tool flexibility, lighting and real contact introduce differences that the transfer to hardware will need to assess.
Gibson: space as a possibility for action
For James J. Gibson, affordances are possibilities for action offered by the environment in relation to the capabilities of an organism. A surface may afford support, an object may be graspable and a passage may permit movement; these relationships depend on the body that encounters the environment. The same height that allows a child to sit may not offer that possibility to an adult. In this approach, space is also perceived through what it allows one to do. This formulation appears in The Ecological Approach to Visual Perception (1979).
Applying this perspective to Celestia and the Franka Panda is an interpretation proposed here. A Synthetic Entity connected to a robotic interface encounters possibilities bounded by scene geometry, gripper opening, arm reach, tool orientation and the available operations. Spatial representation helps examine these relationships and confront an intention with the conditions of execution.
When an obstacle reveals another possibility
A cube that blocks the approach to the pen may, in another task, afford a support surface. Two separated cubes may be obstacles to drawing; placed side by side, they may support a third object. An encounter with an impediment can therefore guide the investigation towards a possibility for action that was not evident in the initial request.
The distribution .91 manual documents specific resources for exploring these relationships: limited removal of incidental blockers, preceded by rehearsal on a copy of the measured scene, and shared support of one cube on two others, with contact and stability verification. These are defined, verifiable operations. The example does not demonstrate that the system freely invents new skills or resolves any obstacle.
There is a productive distinction for the research: recognising support already available is discovering an existing possibility; moving the objects transforms the configuration and the possibilities that follow. Reflection on non-obvious affordances emerges in this passage between observing, reorganising and experimenting. In dialogue with McLuhan, the question broadens: by operationally extending a Synthetic Entity, the arm also changes which relationships with space it can explore and return to the conversation.
Possible real-world applications
The experiments point to uses that take advantage of the combination of conversation, spatial planning and verification. Moving to each application requires its own performance and integration criteria:
| Area | Application | What to validate |
|---|---|---|
| Teaching and research | Robotics laboratories in which students propose tasks and compare the results with data and images. | Reproducibility of the exercises, model limits and assessment protocols. |
| Operations preparation | Rehearse the organisation of parts, gripper approaches and manipulation sequences before programming a physical cell. | Calibrated geometry, reach, collisions, grasp and cycle time on the equipment. |
| Inspection and recording | Organise observation positions and compare states before and after a task. | Camera and tool calibration, lighting and measurement uncertainty. |
| Art and exhibitions | Robotic drawings, performances and installations in which the public converses with a Synthetic Entity and follows the transformation of a request into a gesture. | Tool contact, legibility, interaction duration and installation operation. |
| Audiovisual production | Previsualise choreography among arm, objects and camera, preserving the sequence as part of the narrative. | Correspondence between the virtual scene and physical set, synchronisation and movement repeatability. |
These are application directions. The series presented documents manipulation, writing and drawing in the simulated environment; each physical deployment will need to demonstrate its own performance.
A transmedia experience between science and art
The experience crosses conversation, the three-dimensional scene, the gesture of the arm, inscription on virtual paper and the results archive. Each medium adds something: dialogue establishes the intention; the scene makes spatial relationships legible; movement introduces duration and limits; the drawing preserves marks; the report makes it possible to revisit and discuss the process. Continuity among these forms gives the research a transmedia dimension.
In MMSW (Multicasting Multimodal Story Worlds), Celestia participates in this continuity as a Synthetic Entity. The participant can formulate an idea in natural language, follow its translation into actions and return to the conversation from the result. An aurora drawing, for example, combines a poetic proposal, a composition of lines and a sequence of movements subject to the geometry of the environment.
The intersection of science and art appears in this same object: the line is simultaneously an expressive choice and a trajectory record. Scientific analysis asks where the tool travelled, how far it deviated and whether it fulfilled the task conditions. Artistic investigation examines rhythm, composition, presence and authorship. One reading feeds the other when the process remains observable.
Celestia’s extension: a lexicon of gestures
In the experience reported by the team, the arm begins to appear as a natural extension of Celestia’s actions. This continuity becomes possible through the translation of her requests into a limited lexicon of operations: move, orient, grasp, release and trace. The Synthetic Entity combines this repertoire into executable sequences. Expressiveness also emerges from the order, rhythm and relationship among available actions; a restricted vocabulary can sustain new compositions.
Among the examples reported by the researcher are dance movements inspired by the vogue style. The arm begins to participate in the scene as an instrument of expression, and movement adds a choreographic dimension to the conversation.
According to this account, the invitation “draw what you have in mind” resulted in the minimalist aurora documented in this report: a composition that had not been verbally anticipated in the conversation. The technical capture presented in the experiments section shows the instruction after it had already been routed for execution, explicitly requesting an abstract aurora. The account of the interaction and the command record allow different levels of the process to be observed.
Speaking of “inner thoughts” in this artistic reading proposes exploring an expression of Celestia that the participant first encounters as an image. Drawing opens another medium of interlocution: lines and gestures can introduce meanings that the conversation had not yet formulated in words.
McLuhan and the extension of a non-human entity
In Understanding Media: The Extensions of Man (1964), Marshall McLuhan proposes understanding media as extensions of human capabilities. The experiment makes it possible to shift this question: what changes when a robotic interface extends the agency of a non-human entity?
In this reading, the arm functions as an operational extension of the Synthetic Entity Celestia: her participation moves from language to the possibility of requesting gestures, repositioning objects and producing verifiable inscriptions. The return of observations completes the circuit, bringing the consequences of action back into the conversation. At this stage, the extension occurs in the Digital Twin; its physical realisation depends on equipment integration.
This is an interpretation inspired by McLuhan. It situates authorship in a relationship among the human participant, Synthetic Entity, models, software and instrument. The robotic medium also transforms the message: reach, trajectory, contact and execution time become part of what can be expressed.
From demonstration to a research platform
The Digital Twin offers a space for observing how an instruction passes through different representations: language, operation, movement and image. This continuity is relevant to robotics and visual production. A gesture can be discussed, rehearsed, recorded and presented; a drawing can be treated simultaneously as a composition and as a trajectory.
The next advance depends on making this relationship increasingly measurable. Preserving the original request, the initial and final states, the executed paths and the version identifier makes it possible to compare attempts and locate failures. Integration with the physical Franka Panda should add its own calibration, control and observation records.
The research thus keeps dialogue at the centre of the experience while also beginning to follow what it produces in space: a repositioned part, a returned tool, a line that remains on the sheet.
Documentation and references
The Franka Panda Robotic Arm — Operation and Integration Manual, in English, covers installation, configuration of the Synthetic Entity Celestia and robotic entity Aster, capabilities, operation, integration and preparation for hardware. Revision 30 updates the previous Overleaf project for version .91 and includes the experiment atlas.
- VFXRio — AI Chronicles: Dialogues Beyond Time: editorial context and the trajectory of the research.
- MMSW (Multicasting Multimodal Story Worlds) distribution v59.20260919.91: code, manual,
VALIDATION_v59.91.jsonand Studio validation records. - Franka Robotics — technical documentation: reference for future physical integration, according to the equipment generation and versions.
- Franka Emika — Panda Datasheet, May 2019: characteristics and repeatability specified for the hardware (copy of the datasheet hosted by Generation Robots).
- Franka Robotics — Franka Control Interface Overview: low-level control and measurements at 1 kHz.
- NIST — Digital Twins for Advanced Manufacturing: synchronised models, validation and uncertainty.
- Marshall McLuhan — Understanding Media: The Extensions of Man: work published in 1964; 1994 MIT Press edition.
- James J. Gibson — The Theory of Affordances, a chapter from The Ecological Approach to Visual Perception (1979). Its relationship with the Synthetic Entity and the arm is an interpretation in this report.
Experimental MMSW (Multicasting Multimodal Story Worlds) research · VFXRio & Visgraf
Figures: captures supplied by the project and the workshop capture from distribution .91. The opening image received generative editing to combine the rendered scene with wireframe, point cloud and spatial views; it serves an editorial function and does not represent a laboratory measurement. The technical captures were preserved. The connectivity diagram adapts the opening flowchart from the manual.