What is an affordance?

In ecological psychology, James J. Gibson developed the concept of affordances to describe the possibilities for action that an environment offers to an organism. An affordance is not simply a visual property or a function assigned to an object in advance. It depends on the relationship between the properties of the world and the capabilities of the entity acting within it.

A surface may afford support, an opening may afford passage, and a cube may afford different kinds of grasping. Yet what is accessible to a human hand may not be accessible to a robotic gripper. And what can be grasped in one configuration may become inaccessible when neighboring objects block the path.

For this reason, seeing a blue cube and identifying a feasible side grasp are two different problems.

What changes in MMSW v59?

MMSW brings together entities, agents, memory, media, and shared states in distributed multimodal worlds. The robotics integration in version v59 extends this architecture to include bounded commands, execution, observation, and outcome verification in physical and simulated environments.

In this research, Celestia Borealis is a non-human entity, while Aster is a robotic extension of her ability to act. The arm is not the entity: it provides an embodiment with specific mechanical, kinematic, and operational capabilities through which an intention may be executed.

Affordances form a conceptual layer between world representation and motion planning. They help the system ask not only where is the object?, but also what does the current state of the scene allow us to do with it?

One scene, multiple possibilities

The experimental environment shown in the image contains the robotic arm, colored cubes, spheres, a work surface, and a bounded dark area. The affordance visualization makes regions and directions associated with possible interactions available for inspection.

Visual markers should not be confused with a guarantee of successful execution. A direction that appears unobstructed must still be checked against gripper volume, arm kinematics, obstacles, contacts, and the continuity of the entire motion.

Same objective. Different affordances. Different strategies.

Consider the instruction: “Pick up the blue cube from the side.” The goal sounds simple, but execution raises concrete questions: which face is unobstructed? Is there room for the gripper fingers to approach? Can the arm reach a stable orientation? Does the green cube obstruct access? Can the cube be transported and the gripper withdrawn without striking neighboring objects?

A small change in the green cube’s position may alter the availability of a side grasp. The strategy might require a different orientation, an approach from another face, or, when permitted, rearranging an object first. The instruction is unchanged; the opportunities for action change with the state of the world.

The aim is not to map a verbal command onto a fixed motion, but to evaluate which actions are supported by the current environment and the available embodiment.

From object recognition to physically feasible planning

To treat affordances as operational information, the research combines several layers of representation and validation:

A World Model that changes after each action

Affordances are not permanent labels. When a cube is moved, when a stack is formed, or when the robot begins holding an object, the scene relationships change, along with the movements that may be available.

An important research direction is to construct a dynamic scene graph containing geometry, support relationships, contacts, and manipulation states, coupled with an affordance system and motion planning. Such a model should account for the complete sequence, including gripper withdrawal and the settling of objects after release.

This gives rise to a cycle of reasoning and verification:

1. Perceive the current state and locate physical entities.
2. Represent geometry, supports, obstacles, and contacts.
3. Evaluate affordances and constraints for the requested task.
4. Plan a complete sequence and check its feasibility.
5. Execute with control boundaries and telemetry.
6. Observe and update the World Model using the verified outcome.

In the MMSW architecture, a successful action in the Digital Twin does not automatically establish that the physical robot performed that action. Visualization, simulation, execution, and evidence must remain distinguishable.

From language to the possibility of action

The conceptual advance is not simply about enabling a synthetic entity to control a robotic arm. It is about investigating how an intention expressed through language can be connected to concrete physical possibilities that are evaluated and revised within a continuously changing world.

For Celestia Borealis, this extends synthetic presence from dialogue toward embodied interaction. For MMSW, it reinforces the role of World Models as systems that do not merely describe scenes, but help organize decisions about what can be done within them.

Understanding the instruction is not enough: the system must discover which possibilities in the environment could turn that instruction into action—and verify what actually happened.

Presentation at VISGRAF/IMPA — October 14

This line of research is among the topics of the VISGRAF/IMPA Computer Graphics Seminar, presented by Matteo Moriconi and Luiz Velho, focusing on World Models, transmultimodality, Digital Twins, and robotic extensions of non-human entities.

Related research and reference

Aurora Circuit — Human–Synthetic Interaction

Digital Twins — Franka Panda

Conceptual reference: Gibson, J. J. (1979). The Ecological Approach to Visual Perception. Houghton Mifflin.