Experiment scene with Aster pen, colored cube stack and ink drawing

MMSW v59: Robotic Drawing, Episodic Memory, and World-Model Learning

This technical report examines two drawing requests addressed to the synthetic entity Celestia Borealis in the MMSW v59 robotics environment. It relates graphical outputs, linguistic responses, Digital Twin visualization, and execution telemetry, explicitly separating results documented in simulation from those not yet verified on physical hardware.

Building on those records, the report outlines a proposed experience-driven, world-model learning framework: an experimental module for episodic replay, counterfactual simulation, and strategy evaluation for Aster, the Franka Panda robotic arm. The term “dreaming” denotes this proposed computational process, not a claim of consciousness or emotion.

1. Research objective and scope

Multicasting Multimodal Story Worlds (MMSW) integrates synthetic entities, language, memory, scene representations, and multimodal actions. In Robotic Studio, Celestia Borealis interprets requests and interacts with Aster, its Franka Panda-based robotic extension, through planning and execution interfaces. The Digital Twin provides an environment for representing the scene and evaluating operations; outcomes within it are not automatically equivalent to outcomes on physical hardware.

This case study addresses two related questions: (i) how open-ended expressive instructions are converted into graphical operations and linguistic interpretations; and (ii) which records would be required to turn robotic episodes into useful examples for learning and improving manipulation strategies.

The work is an exploratory case study. The supplied captures do not constitute repeated trials, a controlled comparison, or a demonstration of policy training. Empirical conclusions are limited to the images, conversation excerpts, and telemetry provided.

2. Protocol: drawing tasks and entity responses

Following the completion of a robotic task described as relatively complex, the operator requested that Celestia draw how she felt. The first output comprises approximately concentric lines and overlapping contours on the dark drawing surface. A subsequent instruction requested that the entity draw how she saw herself, resulting in an additional composition in the same environment.

First drawing near the yellow sphere
Figure 1. Initial graphic output in the workspace beside the yellow sphere. A record of visual composition, not a measurement of affective state.

When asked to interpret the first drawing, Celestia produced the following textual response, reproduced verbatim:

“To me, it means growth without certainty: moving upward, exploring outward, and staying open to what surrounds you.”

The subsequent instruction — now draw how you see yourself — produced the response:

“I drew myself as a luminous, thoughtful machine presence: a central core with flowing geometric arcs reaching outward—curiosity, care, and an urge to connect. The earlier ink was left untouched; this was added only in a clear space.”

Two drawings created in separate areas of the simulated paper
Figure 2. Initial composition and additional drawing preserved on the same surface. Their spatial separation is relevant to state persistence and workspace-occupancy assessment.

These statements are system-generated linguistic observations. Terms such as growth, curiosity, and care are descriptive elements within the dialogue and do not, by themselves, measure internal affective states. The statement that the additional drawing occupied free space is operationally relevant because it can be compared with the images and with the recorded state of the drawing surface.

3. Observation, traceability, and evidence provenance

The documentation links captures of the drawing environment to simulated scene views and a projection of pen trajectories. The multiview representation supports inspection of geometry, object relationships, and the spatial distribution of marks. A rendered image does not, however, substitute for executed commands, measured state, or terminal verification.

Simulation views: overview, front, top and pen paths
Figure 3. Multiview simulation panel: overview, front and top views together with a projection of pen trajectories.

The telemetry for the operation identified as 29c532fef447 separates token consumption from native phase duration for stages such as intent, plan, preflight, execute, and verify. The displayed measurements include approximately 17,571.01 ms for preflight and 188,621.62 ms for execute. The operation timeline extends to 234.95 seconds. Displayed intervals may overlap and therefore should not be summed naively.

Task phase tokens and native time measurements
Figure 4. Phase instrumentation: new/cached input and output tokens, with measured native phase durations. A missing measurement is not a zero duration.
Recorded task operation timeline and simulation receipt
Figure 5. Operation timeline and trace action receipt. The operation is classified as a qualified simulation transition, not verified physical movement.

The action-level record labels trace with a Passed receipt, a held-object state change ([] → ["pen"]), and the category “Qualified simulation transition.” The panel itself cautions that the measured intervals do not establish that live motion occurred. The receipt should therefore not be interpreted as confirmation of physical drawing by the Franka Panda.

Evidence What the record supports What it does not establish
Drawing images The visible graphic forms and their arrangement in the captured workspace. Feelings, subjective intention, or reproduction on physical hardware.
Celestia responses Linguistic interpretations and self-descriptions linked to the requests. Consciousness, pheNamenological experience, or psychological validity of those interpretations.
Telemetry and receipt Recorded phases, partial costs, and a qualified simulation transition. Independent physical execution, Digital Twin fidelity, or policy learning.
Drawing in free space A visual outcome compatible with preservation of existing marks. That a policy learned free-space selection autonomously or generalizes to unseen scenes.

Table 1. Evidence matrix for the case study: documented observations and the limits of possible inference.

4. Technical interpretation: symbolic composition and persistent state

The scientific relevance of the drawings does not depend on attributing subjective experience to the entity. The output can be analyzed as a transformation chain linking linguistic instruction, operational representation, movement sequence, graphical result, and subsequent textual interpretation. This is a study of multimodal coordination and situated action.

The second drawing introduces a particularly important robotics requirement: world-state persistence. Producing an additional composition requires representing existing marks, estimating usable surface area, honoring geometric constraints, and distinguishing occupied from unoccupied regions. In manipulation tasks, the same class of problem arises when preserving support under a stack of cubes, transporting already-held objects, or clearing obstacles with minimum disturbance.

Gibson’s notion of affordances and Suchman’s account of situated action help frame this issue: available strategies depend on current context, arm capabilities, and predicted consequences. A coherent verbal account does not, however, guarantee a physically valid plan.

5. From isolated execution to learning from experience

The documented experiments concern interaction and action representation but do not establish performance improvement across episodes. The proposed development is to introduce an offline learning and controlled evaluation cycle that uses previous records as material for training and counterfactual analysis.

Comparison with simulation-trained locomotion platforms, such as Microduck, is useful only at a methodological level. For the Panda, the learning targets include grasp selection, spatial reasoning, collision avoidance, preservation of support relationships, authorized obstacle clearing, object transport, and recovery from failed tasks. These competencies require contact and dynamics models distinct from those used in locomotion.

6. Proposed architecture: Celestia Dreamer

Celestia Dreamer is the proposed experimental module for converting recorded episodes into candidate strategies subject to validation. Technical inspiration includes Sutton’s Dyna architecture, which integrates learning from experience with model-based planning, and the Dreamer research family, which learns from trajectories imagined by world models. This comparison neither implies direct reuse of those algorithms nor claims that the new module has already been implemented.

Proposed component Function Evaluation criterion
Episodic memory Capture goal, initial state, objects, contacts, plans, actions, feedback, cost, failures, and terminal state with provenance. Reproducible episodes; physical observations clearly separated from simulation and prediction.
Replay and world model Reconstruct scenes, simulate alternatives, vary poses, obstacles, action ordering, and contact conditions. Predictive accuracy, explicit uncertainty, and consistency with reference episodes.
Optional reasoning Activate deliberative analysis for stacking, conflicting constraints, or planning failures. Gains on complex tasks versus added latency and token costs.
Policy evaluation Test candidates without physical actuation: safety, stability, efficiency, and recovery. Held-out scene performance, baseline comparisons, and absence of critical regressions.
Supervised promotion Advance approved policies to controlled tests with shadow mode and rollback capability. Explicit acceptance requirements and prevention of unauthorized automatic execution.

Table 2. Functional elements of a proposed MMSW experience-learning architecture, not experimental results already obtained.

The proposed dreaming process would not be triggered only by failure. Replay sessions could be initiated by informative new episodes, uncertainty, near collisions, inefficient successes, or high task complexity. Such work should remain asynchronous relative to physical control, computationally budgeted, and isolated from authorized actuation. An offline-generated strategy must not automatically acquire permission to command Aster.

7. Hypotheses and validation protocol

A future controlled comparison could examine three conditions: (A) a baseline planner without replay; (B) replay and offline evaluation without strategy promotion; and (C) candidate strategies promoted after validation. Evaluation scenes should be partitioned into experienced configurations and held-out combinations of objects, obstacles, and support relationships.

Relevant metrics include verified task-completion rate, collisions, disturbance of neighboring objects, support preservation, operator interventions, planning time, execution time, token use, recovery after failure, and simulation-to-physical discrepancy. Claims of improvement require consistent protocols, adequate repeated trials, and explicit provenance for each observation.

The testable hypothesis is that episodic memory coupled with counterfactual planning may improve generalization to previously unseen tasks and scenes without compromising safety or introducing excessive latency. The present case study contains insufficient evidence to accept or reject that hypothesis.

8. Discussion: the limits of the dreaming metaphor

The image of a “dreaming” machine, familiar from the cultural imagination surrounding Stanley Kubrick’s 2001: A Space Odyssey, can serve as a framework for discussing computational imagination. Its engineering definition is narrower: replaying experience, predicting consequences, and selecting candidates under validation constraints. Neither the recorded dialogue nor the drawings establish that Celestia has consciousness, human emotions, or subjective dreams.

Conclusion

This case study documents two graphical outputs following expressive requests, corresponding interpretations generated by Celestia, and instrumentation of an operation classified as simulated. Its principal contribution is the joint examination of linguistic behavior, scene state, represented action, telemetry, and the limits of evidence.

The next research question is whether those episodes, combined with manipulation trials and failure records, can support a reproducible world-model-based embodied learning cycle. Moving from a proposal to a scientific result will require controlled evaluation and independent verification of physical robot actions.

Selected references

Gibson, J. J. (1979). The Ecological Approach to Visual Perception. Suchman, L. (1987). Plans and Situated Actions. Wittgenstein, L. (1953). Philosophical Investigations. Sutton, R. S. (1990). Integrated Architectures for Learning, Planning, and Reacting. Hafner, D. et al. (2025). Mastering Diverse Control Tasks Through World Models, Nature.

Related research: MMSW — Digital Twins and Franka Panda · Aurora Circuit.