• Scene captioning with a Vision-Language Model. As the robot moves through its environment, a VLM generates captions for the scenes it encounters. These descriptive summaries capture the essence of what the robot sees, providing rich, contextual information about the surroundings rather than a raw image or point cloud.
• Memory building with ROS 2. The captions, along with their associated positional and rotational data, are embedded and saved in a vector database using a ROS 2 node. This database becomes the robot’s long-term memory, allowing it to recall and understand its environment even after extended periods of operation.
• Intelligent retrieval with RAG. When a task is assigned, such as finding the elevator, the robot uses an Agentic Retrieval-Augmented Generation application to embed the question and perform a similarity search through its memory database. The relevant data is then processed by a large language model deployed locally on Connect Tech Edge ai systems, which generates the positional data needed to guide the robot.
• Seamless navigation with Nav2 and SLAM. The resulting positional and rotational data is fed into the Nav2 stack, enabling the robot to navigate to the goal position while avoiding obstacles using Simultaneous Localization and Mapping algorithms (Open Navigation, n.d.).

Together, these four stages give the robot access to contextual information about its environment, combining spatial awareness, long-term memory, and language-based reasoning in a single pipeline. Each stage is decoupled as its own ROS 2 node, so the VLM captioning model, vector database, RAG retrieval logic, and Nav2 stack can each be updated or swapped independently as a deployment’s requirements evolve.