Engineering Interactive Learning in Artificial Systems
We look to develop machines that learn through autonomous exploration of and interaction with their environments -- as humans learn. To do this, we use deep reinforcement learning and employ and develop techniques in curiosity, active learning, and self-supervised learning. In doing so, we hope to create artificial systems that can learn more autonomously, flexibly, and robustly, with less demand on data.
Most successful machine learning algorithms of today rely on either carefully curated, human-labeled datasets, or extremely large amounts of interactive experience with simple environments. This reliance has critical drawbacks: the expansive and careful human effort in curating the data is expensive, and models trained in this way struggle to generalize beyond the scope of the data. In short, current AI is data-hungry, and in particular, for large-scale, carefully-crafted human input. This creates challenges, not only of expense and scale, but also of ethics: the desire to collect more and better data conflicts with people’s desire for privacy.
People, on the other hand, learn by virtue of their agency: they interact with various environments, exploring and building complex mental models of their world so as to flexibly adapt to a wide variety of tasks. This loop of action and perception is inseparable from the human learning process, as they decide what to look at, how to manipulate objects, and what to say to others.
Thus far, we have developed curious, self-supervised deep reinforcement learning agents that explore and learn to model 3D physical environments and other agents. In doing so, they gain predictive models and useful visual representations. In the long term, we would like to grow these learning algorithms into a variety of robust technologies:
Curious, exploring robots that autonomously adapt to new surroundings and perform a variety of tasks.
Social AI that interacts with you and understand your beliefs & understandings.
AI that accesses, harnesses, and welds human knowledge.
World models are self-supervised predictive models of how the world evolves. Humans learn world models by curiously exploring their environment, in the process acquiring compact abstractions of high bandwidth sensory inputs, the ability to plan across long temporal horizons, and an understanding of the behavioral patterns of other agents. In this work, we study how to design such a curiosity-driven Active World Model Learning (AWML) system. To do so, we construct a curious agent building world models while visually exploring a 3D physical environment rich with distillations of representative real-world agents. We propose an AWML system driven by -Progress: a scalable and effective learning progress-based curiosity signal and show that -Progress naturally gives rise to an exploration policy that directs attention to complex but learnable dynamics in a balanced manner, as a result overcoming the white noise problem. As a result, our -Progress-driven controller achieves significantly higher AWML performance than baseline controllers equipped with state-of-the-art exploration strategies such as Random Network Distillation and Model Disagreement.
@article{kim_active_2020,author={Kim, Kuno and Sano, Megumi and Freitas, Julian De and Haber, Nick and Yamins, Daniel},citation={International conference on machine learning, 5306-5315, 2020},conference={International conference on machine learning},pages={5306-5315},year={2020},publisher={PMLR},title={Active world model learning with progress curiosity},_google_scholar_id={QIV2ME_5wuYC},}
2018
Learning to play with intrinsically-motivated, self-aware agents
Nick Haber, Damian Mrowca, Stephanie Wang, and 2 more authors
Advances in neural information processing systems, 2018
Infants are experts at playing, with an amazing ability to generate novel structured behaviors in unstructured environments that lack clear extrinsic reward signals. We seek to mathematically formalize these abilities using a neural network that implements curiosity-driven intrinsic motivation. Using a simple but ecologically naturalistic simulated environment in which an agent can move and interact with objects it sees, we propose a" world-model" network that learns to predict the dynamic consequences of the agent’s actions. Simultaneously, we train a separate explicit" self-model" that allows the agent to track the error map of its world-model. It then uses the self-model to adversarially challenge the developing world-model. We demonstrate that this policy causes the agent to explore novel and informative interactions with its environment, leading to the generation of a spectrum of complex behaviors, including ego-motion prediction, object attention, and object gathering. Moreover, the world-model that the agent learns supports improved performance on object dynamics prediction, detection, localization and recognition tasks. Taken together, our results are initial steps toward creating flexible autonomous agents that self-supervise in realistic physical environments.
@article{haber_learning_2018,author={Haber, Nick and Mrowca, Damian and Wang, Stephanie and Fei-Fei, Li F and Yamins, Daniel L},citation={Advances in neural information processing systems 31, 2018},journal={Advances in neural information processing systems},year={2018},title={Learning to play with intrinsically-motivated, self-aware agents},volume={31},_google_scholar_id={W7OEmFMy1HYC},}
Flexible neural representation for physics prediction
Damian Mrowca, Chengxu Zhuang, Elias Wang, and 4 more authors
Advances in neural information processing systems, 2018
Humans have a remarkable capacity to understand the physical dynamics of objects in their environment, flexibly capturing complex structures and interactions at multiple levels of detail.
@article{mrowca_flexible_2018,author={Mrowca, Damian and Zhuang, Chengxu and Wang, Elias and Haber, Nick and Fei-Fei, Li F and Tenenbaum, Josh and Yamins, Daniel L},citation={Advances in neural information processing systems 31, 2018},journal={Advances in neural information processing systems},year={2018},title={Flexible neural representation for physics prediction},volume={31},_google_scholar_id={ULOm3_A8WrAC},}
