Robot memory system flags and corrects its own uncertain object labels

Researchers at MIT have built a memory system that lets a mobile robot recognise when its stored description of an object may be wrong, and repair it. The system, called UQ-DAAAM, improved the accuracy of object descriptions by 5 per cent over the state of the art and gave better answers to questions about objects, their locations and when they were last seen. The work will be presented at the Conference on Neural Information Processing Systems.

The problem of persistent errors

UQ-DAAAM extends an earlier system, DAAAM, in which a vision-language model (VLM) writes detailed captions for objects seen by a robot's onboard cameras. The captions are stored in a 3D, map-based representation of the environment, which the robot can later query to answer questions.

The weakness is that objects are sometimes seen only in part, from a distance, or in blurred or dim images. In such cases the VLM can produce a wrong caption, for example describing a grey chimney as a squirrel on a roof. Without a way to detect the mistake, the error stays in memory and can affect any later task that relies on it. Harry Zhang, a co-lead author and MIT graduate student, noted that such a memory would persist as the robot moves on.

Measuring what the robot does not know

The new method adds uncertainty quantification to the framework. It combines two signals into a single score. The first is the VLM's own confidence in the words it generated for an object. The second is consistency: how well that description agrees with captions of the same object taken from other vantage points, which the robot gathers naturally as it travels.

If the score passes a set threshold, the entry is treated as unreliable and is refined using further images. A simple rule, or heuristic, picks which extra viewpoints are most likely to reduce uncertainty; in the chimney example, it might choose a frame a few moments later that clearly shows smoke. A large language model then merges the selected captions into one description, giving more weight to newer and less ambiguous captions.

Keeping it fast enough for a moving robot

Running a VLM repeatedly is costly, so the team designed the heuristic to avoid doing so. The researchers report a mathematical proof that it can identify the best frames without rerunning the VLM, which sharply cuts computation. Senior author Luca Carlone, an associate professor in MIT's Department of Aeronautics and Astronautics, described the central challenge as getting this to run in a fraction of a second on a robot in motion. Co-lead author Nicolas Gorlo is also an MIT graduate student.

According to the researchers, the refinement step resolves ambiguities for about 92 per cent of the objects in the robot's environment while needing only two additional captions per object. The source does not give details of the test environments or baseline methods, so the size and generality of the gains are best judged from the full paper.

Why it matters and what follows

For researchers working on embodied AI, the study addresses a practical gap: memory-based robots need to know when their stored knowledge is unreliable, not only to store it. Carlone argued that, unlike a chatbot, a physical robot cannot rely on a general disclaimer that it may be wrong, and that awareness of its own ignorance will be central to safe deployment. Potential applications include household and industrial robots handling multistep tasks, such as clearing a table or helping assemble machinery.

The team plans to apply UQ-DAAAM to other robot types, including mobile manipulators, and to use its uncertainty methods in other domains. They are also studying how the system could point a robot towards areas of high uncertainty to revisit, and exploring uncertainty quantification in systems that predict how an environment will change. The research was supported in part by the U.S. Army Research Laboratory and the U.S. Office of Naval Research.

Source: Clearing up a robot’s fuzzy memory (MIT News: research, 5 November 2026).

Image: AI-generated illustration.