Cognitive Physical AI: From Experience to Transferable Intelligence

Every interaction gives a robot more than an outcome. It creates an opportunity to understand what happened, learn from it, and apply that experience somewhere new.

A robot that retrieves an object, responds to a person, or recovers from a failed grasp encounters information that could matter long after the task is complete. Yet much of that information remains tied to the particular episode in which it was collected.

Foundation models, vision-language-action policies, and world models have expanded what robots can perceive and do. The next question is how to connect these capabilities to experience: how can a robot use what it has learned in one task or environment to make better decisions in another?

At AlpsLab, we are exploring Cognitive Physical AI: a research direction in which cognition connects physical interaction, memory, reasoning, and action. Our long-term vision is to develop Cognizers that transform embodied experience into reusable knowledge, adapt that knowledge to new situations, and improve the strategies that guide future behavior.

01Why Physical AI needs cognition

A capable model gives a robot a powerful starting point, but physical environments are open-ended. An object may move after a grasp is planned. A person may change a request. A drawer may still be partly closed even though an opening action reports success. In each case, the robot must connect what it observes to what it intended and decide what to do next.

Cognition gives this process a broader purpose. It helps a system organize experience into representations, task decompositions, and strategies that can guide later decisions. In our broader Cognizer vision, the system should also be able to evaluate and revise the structures that generate those strategies. Remembering an episode or adapting a plan is a step toward that goal; neither alone establishes a complete Cognizer.

We organize this research vision around two complementary loops of adaptive physical intelligence.

Physical intelligence should adapt to the situation at hand and turn verified experience into knowledge that can guide future tasks.

The fast loop operates during execution. It combines current observations, relevant memory, task reasoning, and actions to respond as a situation changes. The slow loop operates across episodes. It examines accumulated experience, proposes updates to knowledge or capabilities, and evaluates those updates before they are used more broadly. Fast and slow describe their relative roles, not fixed clock rates.

02From physical experience to transferable intelligence

Experience becomes useful when a system can relate an outcome to the conditions that produced it. An episodic record can connect the task, the observed scene, the robot's available skills, its plan and actions, and the feedback that followed. These relationships provide a basis for retrieving a relevant precedent and assessing whether it applies to the present situation.

From a single episode to a reusable strategy

Consider a robot that tries to retrieve an object from a drawer. A recorded failure might show that the drawer had not opened far enough for the object to be reachable. A more useful lesson is a conditional strategy: before retrieval, check whether access and reachability are sufficient; if they are not, address that condition before attempting the grasp.

The same idea could help with a different task in a different setting, such as collecting supplies from a kitchen cabinet. The robot would still need to identify the cabinet, observe its state, check its own reach and available skills, and produce actions for the new geometry. What transfers is the reasoning pattern and its conditions, not an assumption that the original motion can be replayed unchanged.

This illustrates the role we envision for cognition in transfer: retrieve relevant experience, identify the relationship between the old and new situations, adapt a strategy to the current constraints, and verify the result through execution. Across tasks, the goal or sequence may change; across environments, objects, layouts, and people may change. Across embodiments, sensing, reach, and action capabilities may also differ. Each kind of transfer needs to be evaluated on its own terms.

Evidence from experience-guided planning

Our research on memory-guided planning investigates how experience can revise reusable planning procedures: structures that organize a class of tasks and specify checks or recovery steps. Such revisions can change future plans even when the underlying foundation model remains fixed.

In our planning evaluations, experience-revised procedures increased the planning pass rate from 61.7% to 83.3% for mobile manipulation and from 66.7% to 90.0% for tabletop manipulation, compared with fixed procedures. These correspond to gains of 21.6 and 23.3 percentage points. Planning pass rate measures whether plans meet the evaluation criteria; it is not a measure of successful physical execution.

The process that proposes revisions can itself be improved. In a separate preliminary evaluation, updating the revision strategy increased accepted beneficial updates from 12 of 30 to 23 of 30 attempts under predefined criteria. This measures update acceptance, not robot task success.

Together, these studies motivate the use of experience to improve future planning procedures. The drawer-to-cabinet example above is an illustration of the intended transfer mechanism, not a result established by those evaluations. Demonstrating reliable transfer to held-out tasks and environments remains an important next step.

03Grounding cognition in physical action

Reusable knowledge is only valuable if it remains connected to the world the robot is acting in. During execution, new observations can invalidate a previous assumption, alter a person's goal, or make a planned action unsafe or infeasible. The system must therefore combine prior experience with current feedback.

Adapting plans as human context changes

In a collaborative workflow, a person may change the order of requested objects while a robot is moving between workstations. Some completed actions and parts of the remaining plan may still be useful, even though other dependencies need to change.

Our research on interactive human-humanoid planning maintains an evolving representation of human activity alongside the robot's executable plan. It uses these representations to identify affected dependencies and revise remaining actions while preserving completed work and still-valid plan structure.

In multi-object assistance experiments, this approach achieved 85.7% end-to-end task success, compared with 57.1% for a general agentic baseline. In a separate request-order evaluation, selective replanning reduced mean plan-update latency from 3.62 to 1.20 seconds, approximately 67%, while post-update token use fell by approximately 43% relative to full replanning.

This work illustrates adaptation at the level of goals and task dependencies. Its value depends on maintaining a representation that remains consistent with actual execution and the actions that are still feasible.

Adapting actions as observations arrive

Physical responsiveness also depends on the relationship between observation and action. When a policy generates a sequence of actions, later actions may become outdated if an object moves or contact conditions change before the sequence is completed.

Our research on streaming action generation explores executing near-term actions while refining future actions with updated observations. In a standardized manipulation evaluation, the approach reached 97.7% task success and increased measured inference-level action throughput from 80.7 to 369.1 actions per second, approximately 4.6 times the conventional short-horizon setting. This throughput describes inference, not physical control frequency or robot motion speed.

In separate real-robot evaluations under matched inference-compute budgets, task success improved by 7.5 to 32.5 percentage points across six static and dynamic settings. These results show that responsiveness depends on the setting; they do not imply a uniform gain.

Adaptive task planning and observation-conditioned action generation address different parts of physical interaction. In the proposed architecture, cognition would help connect decisions about what to do with feedback about what actually happened; reliable execution would, in turn, provide evidence for future learning.

04Toward Cognizer-driven physical intelligence

Our long-term goal is a system that can act in the present and improve across repeated use. In the fast loop, it would perceive the current situation, retrieve relevant experience, reason about the task, and coordinate available skills. In the slow loop, it would examine what succeeded or failed and determine whether the evidence supports a lasting change.

Different failures call for different interventions. A missing precondition may suggest revising a planning procedure. An unreliable grasp may require more demonstrations or policy optimization. Delayed observations may call for a change in inference scheduling. A Cognizer should be able to reason about these alternatives rather than treating every failure as the same kind of problem.

Proposed updates also need verification beyond the episode that produced them. Held-out tasks and environments can test transfer; regression checks can reveal whether an update harms previously successful behavior; physical constraints can test whether it remains suitable for deployment. Accepted changes can then become versioned capabilities, with a way to revert them if later evidence exposes a problem.

Our current studies provide evidence for individual components: experience-revised planning, adaptive human-robot coordination, and observation-conditioned action generation. They were evaluated in different settings and do not yet demonstrate one integrated, autonomously improving Cognizer. Reliable failure attribution, sustained cross-task and cross-environment transfer, and transfer across embodiments remain open research questions.

The next step is to test whether experience acquired in one set of tasks and environments can improve performance in genuinely unseen ones, while preserving established capabilities. Progress should be measured not only by whether a robot completes a new task, but also by what it transfers, when that transfer helps, and when it fails.

The future of Physical AI is not only about robots that act. It is about systems that turn physical experience into cognition—and cognition into capabilities that can travel across tasks and environments.