What is Neural Cube?
This project explores building a fully autonomous robot orchestrated using multi-agent LLMs, enabling humans to interact with it more naturally, see how it reasons, and explore its understanding evolve in real time.
Overview
Problem Statement
Current robotic systems are largely reactive, with limited capacity to reason about themselves or anticipate outcomes in their environment. This project explores how a robot can develop a richer understanding of itself and the world it inhabits through multi-agent LLMs, world modeling, and simulation.
Proposed Approach
The robot will maintain a digital twin as a unified, continuously updated representation of its own internal state and surrounding environment. This serves as the foundation for self-understanding.
For more intensive, physics-aware reasoning tasks, generative frameworks and simulation frameworks can be leveraged on top of this foundation. These enable the robot to generate and simulate hypothetical scenarios derived from its own sensory experience.
The robot will then use iterative experience to refine its understanding and close the loop between perception, thinking, and action. All of these combined inform a multi-agent AI system to make informed decisions.
Goals
- Demonstrate interpretable robot reasoning so that the robot's decision-making process — including simulated what-if scenarios — is visible and understandable to non-experts.
- Have a modular configuration so that different faculties or research groups can adapt the platform to their own questions and experiments.
- Collect anonymous interaction data for future analysis.
- Serve as a living research platform that students, educators, and researchers can observe, query, and contribute to as the project continues to improve.
Minimum Viable Product
The MVP is the project in its simplest complete form: a human interacting with a robot using natural language and visual expression to perform a multi-step navigation task.
What it demonstrates:
- The existing wheeled prototype running ROS 2, equipped with a sensor suite sufficient for basic environment perception (depth camera + IMU at minimum).
- A live web dashboard displaying the robot's digital twin visualized as a continuously updated 3D representation of the robot's pose, sensor state, and immediate environment.
- A natural language interface through which a user can ask the robot what it's thinking ("what do you see?", "why did you stop?") or give it a simple task ("move to the table").
Success criterion:
- A non-expert user can converse with the robot.
- The robot can successfully maneuver in an indoor space to complete a task that requires multi-step reasoning and navigation.
Team
| Name | Role |
|---|---|
| Anbara Lutfullaeva | Design Advisor |
| Josh Thompson | Electrical Systems Engineer |
| Isak Jorgenson | Mechanical Systems Engineer |
| Steve Wufeng | Developer |
| Chanel Cheng | Developer |