publications
2025
-
Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-ThoughViolet Xiang, Charlie Snell, Kanishk Gandhi, and 11 more authorsarXiv preprint arXiv:2501.04682, 2025We propose a novel framework, Meta Chain-of-Thought (Meta-CoT), which extends traditional Chain-of-Thought (CoT) by explicitly modeling the underlying reasoning required to arrive at a particular CoT. We present empirical evidence from state-of-the-art models exhibiting behaviors consistent with in-context search, and explore methods for producing Meta-CoT via process supervision, synthetic data generation, and search algorithms. Finally, we outline a concrete pipeline for training a model to produce Meta-CoTs, incorporating instruction tuning with linearized search traces and reinforcement learning post-training. Finally, we discuss open research questions, including scaling laws, verifier roles, and the potential for discovering novel reasoning algorithms. This work provides a theoretical and practical roadmap to enable Meta-CoT in LLMs, paving the way for more powerful and human-like reasoning in artificial intelligence.
@article{xiang_towards_2025, author = {Xiang, Violet and Snell, Charlie and Gandhi, Kanishk and Albalak, Alon and Singh, Anikait and Blagden, Chase and Phung, Duy and Rafailov, Rafael and Lile, Nathan and Mahan, Dakota and Castricato, Louis and Franken, Jan-Philipp and Haber, Nick and Finn, Chelsea}, citation = {arXiv preprint arXiv:2501.04682, 2025}, journal = {arXiv preprint arXiv:2501.04682}, year = {2025}, title = {Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Though}, _google_scholar_id = {bFI3QPDXJZMC}, } - Prediction of Item Difficulty for Reading Comprehension Items by Creation of Annotated Item RepositoryRadhika Kapoor, Sang T Truong, Nick Haber, and 2 more authorsarXiv preprint arXiv:2502.20663, 2025
Prediction of item difficulty based on its text content is of substantial interest. In this paper, we focus on the related problem of recovering IRT-based difficulty when the data originally reported item p-value (percent correct responses). We model this item difficulty using a repository of reading passages and student data from US standardized tests from New York and Texas for grades 3-8 spanning the years 2017-23. This repository is annotated with meta-data on (1) linguistic features of the reading items, (2) test features of the passage, and (3) context features. A penalized regression prediction model with all these features can predict item difficulty with RMSE 0.52 compared to baseline RMSE of 0.92, and with a correlation of 0.77 between true and predicted difficulty. We supplement these features with embeddings from LLMs (ModernBERT, BERT, and LlAMA), which marginally improve item difficulty prediction. When models use only item linguistic features or LLM embeddings, prediction performance is similar, which suggests that only one of these feature categories may be required. This item difficulty prediction model can be used to filter and categorize reading items and will be made publicly available for use by other stakeholders.
@article{kapoor_prediction_2025, author = {Kapoor, Radhika and Truong, Sang T and Haber, Nick and Ruiz-Primo, Maria Araceli and Domingue, Benjamin W}, citation = {arXiv preprint arXiv:2502.20663, 2025}, journal = {arXiv preprint arXiv:2502.20663}, year = {2025}, title = {Prediction of Item Difficulty for Reading Comprehension Items by Creation of Annotated Item Repository}, _google_scholar_id = {CHSYGLWDkRkC}, } - Person-Specific Analyses of Smartphone Use and Mental Health: Intensive Longitudinal StudyMerve Cerit, Angela Y Lee, Jeffrey Hancock, and 8 more authorsJMIR Formative Research, 2025
Background: Contrary to popular concerns about the harmful effects of media use on mental health, research on this relationship is ambiguous, stalling advances in theory, interventions, and policy. Scientific explorations of the relationship between media and mental health have mostly been found null or have small associations, with the results often blamed on the use of cross-sectional study designs or imprecise measures of media use and mental health.Objective: This exploratory empirical demonstration aims to answer whether mental health effects are associated with media use experiences by (1) redirecting research investments to granular and intensive longitudinal recordings of digital experiences to build models of media use and mental health for single individuals over the course of 1 year,(2) using new metrics of fragmented media use to propose explanations of mental health effects that will advance person-specific theorizing in media psychology, and (3) identifying combinations of media behaviors and mental health symptoms that may be more useful for studying media effects than single measures of dosage and affect or assessments of clinical symptoms related to specific disorders.Methods: The activity on individuals smartphone screens was recorded every 5 seconds when devices were in use over 1 year, resulting in a dataset of 6,744,013 screenshots and 123 fortnightly surveys from 5 adult participants. Each participant contributed between 0.8 and 2.7 million screens. Six media use metrics were derived from smartphone metadata. Fortnightly surveys captured symptoms of depression, attention-deficit/hyperactivity disorder
@article{cerit_person-specific_2025, author = {Cerit, Merve and Lee, Angela Y and Hancock, Jeffrey and Miner, Adam and Cho, Mu-Jung and Muise, Daniel and Torres, Anna-Angelina Garrn and Haber, Nick and Ram, Nilam and Robinson, Thomas N and Reeves, Byron}, citation = {JMIR Formative Research 9 (1), e59875, 2025}, journal = {JMIR Formative Research}, number = {1}, pages = {e59875}, year = {2025}, publisher = {JMIR Publications Inc., Toronto, Canada}, title = {Person-Specific Analyses of Smartphone Use and Mental Health: Intensive Longitudinal Study}, volume = {9}, _google_scholar_id = {xtRiw3GOFMkC}, } - Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language ModelsAlon Albalak, Duy Phung, Nathan Lile, and 8 more authorsarXiv preprint arXiv:2502.17387, 2025
Increasing interest in reasoning models has led math to become a prominent testing ground for algorithmic and methodological improvements. However, existing open math datasets either contain a small collection of high-quality, human-written problems or a large corpus of machine-generated problems of uncertain quality, forcing researchers to choose between quality and quantity. In this work, we present Big-Math, a dataset of over 250,000 high-quality math questions with verifiable answers, purposefully made for reinforcement learning (RL). To create Big-Math, we rigorously filter, clean, and curate openly available datasets, extracting questions that satisfy our three desiderata: (1) problems with uniquely verifiable solutions, (2) problems that are open-ended, (3) and problems with a closed-form solution. To ensure the quality of Big-Math, we manually verify each step in our filtering process. Based on the findings from our filtering process, we introduce 47,000 new questions with verified answers, Big-Math-Reformulated: closed-ended questions (i.e. multiple choice questions) that have been reformulated as open-ended questions through a systematic reformulation algorithm. Compared to the most commonly used existing open-source datasets for math reasoning, GSM8k and MATH, Big-Math is an order of magnitude larger, while our rigorous filtering ensures that we maintain the questions most suitable for RL. We also provide a rigorous analysis of the dataset, finding that Big-Math contains a high degree of diversity across problem domains, and incorporates a wide range of problem difficulties, enabling a wide range of downstream uses for
@article{albalak_big-math:_2025, author = {Albalak, Alon and Phung, Duy and Lile, Nathan and Rafailov, Rafael and Gandhi, Kanishk and Castricato, Louis and Singh, Anikait and Blagden, Chase and Xiang, Violet and Mahan, Dakota and Haber, Nick}, citation = {arXiv preprint arXiv:2502.17387, 2025}, journal = {arXiv preprint arXiv:2502.17387}, year = {2025}, title = {Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Models}, _google_scholar_id = {NhqRSupF_l8C}, } - Symmetrical Visual Contrastive Optimization: Aligning Vision-Language Models with Minimal Contrastive ImagesShengguang Wu, Fan-Yun Sun, Kaiyue Wen, and 1 more authorarXiv preprint arXiv:2502.13928, 2025
Recent studies have shown that Large Vision-Language Models (VLMs) tend to neglect image content and over-rely on language-model priors, resulting in errors in visually grounded tasks and hallucinations. We hypothesize that this issue arises because existing VLMs are not explicitly trained to generate texts that are accurately grounded in fine-grained image details. To enhance visual feedback during VLM training, we propose S-VCO (Symmetrical Visual Contrastive Optimization), a novel finetuning objective that steers the model toward capturing important visual details and aligning them with corresponding text tokens. To further facilitate this detailed alignment, we introduce MVC, a paired image-text dataset built by automatically filtering and augmenting visual counterfactual data to challenge the model with hard contrastive cases involving Minimal Visual Contrasts. Experiments show that our method consistently improves VLM performance across diverse benchmarks covering various abilities and domains, achieving up to a 22% reduction in hallucinations, and significant gains in vision-centric and general tasks. Notably, these improvements become increasingly pronounced in benchmarks with higher visual dependency. In short, S-VCO offers a significant enhancement of VLM’s visually-dependent task performance while retaining or even improving the model’s general abilities. We opensource our code at https://s-vco.github.io/
@article{wu_symmetrical_2025, author = {Wu, Shengguang and Sun, Fan-Yun and Wen, Kaiyue and Haber, Nick}, citation = {arXiv preprint arXiv:2502.13928, 2025}, journal = {arXiv preprint arXiv:2502.13928}, year = {2025}, title = {Symmetrical Visual Contrastive Optimization: Aligning Vision-Language Models with Minimal Contrastive Images}, _google_scholar_id = {abG-DnoFyZgC}, }
2024
- Quiet-star: Language models can teach themselves to think before speakingEric Zelikman, Georges Raif Harik, Yijia Shao, and 3 more authors2024
When writing and talking, people sometimes pause to think. Although reasoning-focused works have often framed reasoning as a method of answering questions or completing agentic tasks, reasoning is implicit in almost all written text. For example, this applies to the steps not stated between the lines of a proof or to the theory of mind underlying a conversation. In the Self-Taught Reasoner (STaR, Zelikman et al. 2022), useful thinking is learned by inferring rationales from few-shot examples in question-answering and learning from those that lead to a correct answer. This is a highly constrained setting – ideally, a language model could instead learn to infer unstated rationales in arbitrary text. We present Quiet-STaR, a generalization of STaR in which LMs learn to generate rationales at each token to explain future text, improving their predictions. We address key challenges, including 1) the computational cost of generating continuations, 2) the fact that the LM does not initially know how to generate or use internal thoughts, and 3) the need to predict beyond individual next tokens. To resolve these, we propose a tokenwise parallel sampling algorithm, using learnable tokens indicating a thought’s start and end, and an extended teacher-forcing technique. Encouragingly, generated rationales disproportionately help model difficult-to-predict tokens and improve the LM’s ability to directly answer difficult questions. In particular, after continued pretraining of an LM on a corpus of internet text with Quiet-STaR, we find zero-shot improvements on GSM8K (5.9%10.9%) and CommonsenseQA (36.3%47.2%) and observe a perplexity improvement of
@article{zelikman_quiet-star:_2024, author = {Zelikman, Eric and Harik, Georges Raif and Shao, Yijia and Jayasiri, Varuna and Haber, Nick and Goodman, Noah}, citation = {First Conference on Language Modeling, 2024}, conference = {First Conference on Language Modeling}, year = {2024}, title = {Quiet-star: Language models can teach themselves to think before speaking}, _google_scholar_id = {g5m5HwL7SMYC}, } - Holodeck: Language guided generation of 3d embodied ai environmentsYue Yang, Fan-Yun Sun, Luca Weihs, and 11 more authors2024
3D simulated environments play a critical role in Embodied AI but their creation requires expertise and extensive manual effort restricting their diversity and scope. To mitigate this limitation we present Holodeck a system that generates 3D environments to match a user-supplied prompt fully automatedly. Holodeck can generate diverse scenes eg arcades spas and museums adjust the designs for styles and can capture the semantics of complex queries such as" apartment for a researcher with a cat" and" office of a professor who is a fan of Star Wars". Holodeck leverages a large language model (ie GPT-4) for common sense knowledge about what the scene might look like and uses a large collection of 3D assets from Objaverse to populate the scene with diverse objects. To address the challenge of positioning objects correctly we prompt GPT-4 to generate spatial relational constraints between objects and then optimize the layout to satisfy those constraints. Our large-scale human evaluation shows that annotators prefer Holodeck over manually designed procedural baselines in residential scenes and that Holodeck can produce high-quality outputs for diverse scene types. We also demonstrate an exciting application of Holodeck in Embodied AI training agents to navigate in novel scenes like music rooms and daycares without human-constructed data which is a significant step forward in developing general-purpose embodied agents.
@article{yang_holodeck:_2024, author = {Yang, Yue and Sun, Fan-Yun and Weihs, Luca and VanderBilt, Eli and Herrasti, Alvaro and Han, Winson and Wu, Jiajun and Haber, Nick and Krishna, Ranjay and Liu, Lingjie and Callison-Burch, Chris and Yatskar, Mark and Kembhavi, Aniruddha and Clark, Christopher}, citation = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern , 2024}, conference = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages = {16227-16237}, year = {2024}, title = {Holodeck: Language guided generation of 3d embodied ai environments}, _google_scholar_id = {2P1L_qKh6hAC}, } - Examining the potential and pitfalls of ChatGPT in science and engineering problem-solvingKaren D Wang, Eric Burkholder, Carl Wieman, and 2 more authorsFrontiers in Education, 2024
The study explores the capabilities of OpenAI’s ChatGPT in solving different types of physics problems. ChatGPT (with GPT-4) was queried to solve a total of 40 problems from a college-level engineering physics course. These problems ranged from well-specified problems, where all data required for solving the problem was provided, to under-specified, real-world problems where not all necessary data were given. Our findings show that ChatGPT could successfully solve 62.5% of the well-specified problems, but its accuracy drops to 8.3% for under-specified problems. Analysis of the model’s incorrect solutions revealed three distinct failure modes: (1) failure to construct accurate models of the physical world, (2) failure to make reasonable assumptions about missing data, and (3) calculation errors. The study offers implications for how to leverage LLM-augmented instructional materials to enhance STEM education. The insights also contribute to the broader discourse on AI’s strengths and limitations, serving both educators aiming to leverage the technology and researchers investigating human-AI collaboration frameworks for problem-solving and decision-making.
@article{wang_examining_2024, author = {Wang, Karen D and Burkholder, Eric and Wieman, Carl and Salehi, Shima and Haber, Nick}, citation = {Frontiers in Education 8, 1330486, 2024}, journal = {Frontiers in Education}, pages = {1330486}, year = {2024}, publisher = {Frontiers Media SA}, title = {Examining the potential and pitfalls of ChatGPT in science and engineering problem-solving}, volume = {8}, _google_scholar_id = {J_g5lzvAfSwC}, } - Learning formal mathematics from intrinsic motivationGabriel Poesia, David Broman, Nick Haber, and 1 more authorAdvances in Neural Information Processing Systems, 2024
How did humanity coax mathematics from the aether? We explore the Platonic view that mathematics can be discovered from its axioms—a game of conjecture and proof. We describe an agent that jointly learns to pose challenging problems for itself (conjecturing) and solve them (theorem proving). Given a mathematical domain axiomatized in dependent type theory, we first combine methods for constrained decoding and type-directed synthesis to sample valid conjectures from a language model. Our method guarantees well-formed conjectures by construction, even as we start with a randomly initialized model. We use the same model to represent a policy and value function for guiding proof search. Our agent targets generating hard but provable conjectures—a moving target, since its own theorem proving ability also improves as it trains. We propose novel methods for hindsight relabeling on proof search trees to significantly improve the agent’s sample efficiency in both tasks. Experiments on 3 axiomatic domains (propositional logic, arithmetic and group theory) demonstrate that our agent can bootstrap from only the axioms, self-improving in generating true and challenging conjectures and in finding proofs.
@article{poesia_learning_2024, author = {Poesia, Gabriel and Broman, David and Haber, Nick and Goodman, Noah}, citation = {Advances in Neural Information Processing Systems 37, 43032-43057, 2024}, journal = {Advances in Neural Information Processing Systems}, pages = {43032-43057}, year = {2024}, title = {Learning formal mathematics from intrinsic motivation}, volume = {37}, _google_scholar_id = {u_35RYKgDlwC}, } - Binding the person-specific approach to modern AI in the human screenome project: moving past generalizability to transferabilityNilam Ram, Nick Haber, Thomas N Robinson, and 1 more authorMultivariate Behavioral Research, 2024
Advances in ability to comprehensively record individuals digital lives and in AI modeling of those data facilitate new possibilities for describing, predicting, and generating a wide variety of behavioral processes. In this paper, we consider these advances from a person-specific perspective, including whether the pervasive concerns about generalizability of results might be productively reframed with respect to transferability of models, and how self-supervision and new deep neural network architectures that facilitate transfer learning can be applied in a person-specific way to the super-intensive longitudinal data arriving in the Human Screenome Project. In developing the possibilities, we suggest Molenaar add a statement to the person-specific Manifesto In short, the concerns about generalizability commonly leveled at the person-specific paradigm are unfounded and can be fully and completely replaced with
@article{ram_binding_2024, author = {Ram, Nilam and Haber, Nick and Robinson, Thomas N and Reeves, Byron}, citation = {Multivariate Behavioral Research 59 (6), 1211-1219, 2024}, journal = {Multivariate Behavioral Research}, number = {6}, pages = {1211-1219}, year = {2024}, publisher = {Routledge}, title = {Binding the person-specific approach to modern AI in the human screenome project: moving past generalizability to transferability}, volume = {59}, _google_scholar_id = {YFjsv_pBGBYC}, } - Hypothetical minds: Scaffolding theory of mind for multi-agent tasks with large language modelsLogan Cross, Violet Xiang, Agam Bhatia, and 2 more authorsarXiv preprint arXiv:2407.07086, 2024
Multi-agent reinforcement learning (MARL) methods struggle with the non-stationarity of multi-agent systems and fail to adaptively learn online when tested with novel agents. Here, we leverage large language models (LLMs) to create an autonomous agent that can handle these challenges. Our agent, Hypothetical Minds, consists of a cognitively-inspired architecture, featuring modular components for perception, memory, and hierarchical planning over two levels of abstraction. We introduce the Theory of Mind module that scaffolds the high-level planning process by generating hypotheses about other agents’ strategies in natural language. It then evaluates and iteratively refines these hypotheses by reinforcing hypotheses that make correct predictions about the other agents’ behavior. Hypothetical Minds significantly improves performance over previous LLM-agent and RL baselines on a range of competitive, mixed motive, and collaborative domains in the Melting Pot benchmark, including both dyadic and population-based environments. Additionally, comparisons against LLM-agent baselines and ablations reveal the importance of hypothesis evaluation and refinement for succeeding on complex scenarios.
@article{cross_hypothetical_2024, author = {Cross, Logan and Xiang, Violet and Bhatia, Agam and Yamins, Daniel LK and Haber, Nick}, citation = {arXiv preprint arXiv:2407.07086, 2024}, journal = {arXiv preprint arXiv:2407.07086}, year = {2024}, title = {Hypothetical minds: Scaffolding theory of mind for multi-agent tasks with large language models}, _google_scholar_id = {fPk4N6BV_jEC}, } - Building a domain-specific guardrail model in productionMohammad Niknazar, Paul V Haley, Latha Ramanan, and 12 more authorsarXiv preprint arXiv:2408.01452, 2024
Generative AI holds the promise of enabling a range of sought-after capabilities and revolutionizing workflows in various consumer and enterprise verticals. However, putting a model in production involves much more than just generating an output. It involves ensuring the model is reliable, safe, performant and also adheres to the policy of operation in a particular domain. Guardrails as a necessity for models has evolved around the need to enforce appropriate behavior of models, especially when they are in production. In this paper, we use education as a use case, given its stringent requirements of the appropriateness of content in the domain, to demonstrate how a guardrail model can be trained and deployed in production. Specifically, we describe our experience in building a production-grade guardrail model for a K-12 educational platform. We begin by formulating the requirements for deployment to this sensitive domain. We then describe the training and benchmarking of our domain-specific guardrail model, which outperforms competing open- and closed- instruction-tuned models of similar and larger size, on proprietary education-related benchmarks and public benchmarks related to general aspects of safety. Finally, we detail the choices we made on architecture and the optimizations for deploying this service in production; these range across the stack from the hardware infrastructure to the serving layer to language model inference optimizations. We hope this paper will be instructive to other practitioners looking to create production-grade domain-specific services based on generative AI and large language models.
@article{niknazar_building_2024, author = {Niknazar, Mohammad and Haley, Paul V and Ramanan, Latha and Truong, Sang T and Shrinivasan, Yedendra and Bhowmick, Ayan Kumar and Dey, Prasenjit and Jagmohan, Ashish and Maheshwari, Hema and Ponoth, Shom and Smith, Robert and Vempaty, Aditya and Haber, Nick and Koyejo, Sanmi and Sundararajan, Sharad}, citation = {arXiv preprint arXiv:2408.01452, 2024}, journal = {arXiv preprint arXiv:2408.01452}, year = {2024}, title = {Building a domain-specific guardrail model in production}, _google_scholar_id = {EUQCXRtRnyEC}, } - Scaffold or Crutch? Examining College Students’ Use and Views of Generative AI Tools for STEM EducationKaren D Wang, Zhangyang Wu, L’Nard Tufts II, and 3 more authorsarXiv preprint arXiv:2412.02653, 2024
Developing problem-solving competency is central to Science, Technology, Engineering, and Mathematics (STEM) education, yet translating this priority into effective approaches to problem-solving instruction and assessment remain a significant challenge. The recent proliferation of generative artificial intelligence (genAI) tools like ChatGPT in higher education introduces new considerations about how these tools can help or hinder students’ development of STEM problem-solving competency. Our research examines these considerations by studying how and why college students use genAI tools in their STEM coursework, focusing on their problem-solving support. We surveyed 40 STEM college students from diverse U.S. institutions and 28 STEM faculty to understand instructor perspectives on effective genAI tool use and guidance in STEM courses. Our findings reveal high adoption rates and diverse applications of genAI tools among STEM students. The most common use cases include finding explanations, exploring related topics, summarizing readings, and helping with problem-set questions. The primary motivation for using genAI tools was to save time. Moreover, over half of student participants reported simply inputting problems for AI to generate solutions, potentially bypassing their own problem-solving processes. These findings indicate that despite high adoption rates, students’ current approaches to utilizing genAI tools often fall short in enhancing their own STEM problem-solving competencies. The study also explored students’ and STEM instructors’ perceptions of the benefits and risks associated with using genAI tools in STEM
@article{wang_scaffold_2024, author = {Wang, Karen D and Wu, Zhangyang and II, L'Nard Tufts and Wieman, Carl and Salehi, Shima and Haber, Nick}, citation = {arXiv preprint arXiv:2412.02653, 2024}, journal = {arXiv preprint arXiv:2412.02653}, year = {2024}, title = {Scaffold or Crutch? Examining College Students' Use and Views of Generative AI Tools for STEM Education}, _google_scholar_id = {pyW8ca7W8N0C}, } - Discovering Players Problem-Solving Behavioral Characteristics in a Puzzle Game through Sequence MiningKaren D Wang, Haoyu Liu, David DeLiema, and 2 more authors2024
Digital games offer promising platforms for assessing student higher-order competencies such as problem-solving. However, processing and analyzing the large volume of interaction log data generated in these platforms to uncover meaningful behavioral patterns remain a complex research challenge. In this study, we employ sequence mining and clustering techniques to examine students log data in an interactive puzzle game that requires player to change rules to win the game. Our goal is to identify behavioral characteristics associated with the problem-solving practices adopted by individual students. The findings indicate that the most effective problem solvers made fewer rule changes and took longer time to make those changes across both an introductory and a more advanced level of the game. Conversely, rapid rule change actions were linked to ineffective problem-solving. This research underscores
@article{wang_discovering_2024, author = {Wang, Karen D and Liu, Haoyu and DeLiema, David and Haber, Nick and Salehi, Shima}, citation = {Proceedings of the 14th Learning Analytics and Knowledge Conference, 498-506, 2024}, pages = {498-506}, year = {2024}, title = {Discovering Players Problem-Solving Behavioral Characteristics in a Puzzle Game through Sequence Mining}, _google_scholar_id = {M05iB0D1s5AC}, } - LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language ModelsFan-Yun Sun, Weiyu Liu, Siyi Gu, and 6 more authorsarXiv preprint arXiv:2412.02193, 2024
Open-universe 3D layout generation arranges unlabeled 3D assets conditioned on language instruction. Large language models (LLMs) struggle with generating physically plausible 3D scenes and adherence to input instructions, particularly in cluttered scenes. We introduce LayoutVLM, a framework and scene layout representation that exploits the semantic knowledge of Vision-Language Models (VLMs) and supports differentiable optimization to ensure physical plausibility. LayoutVLM employs VLMs to generate two mutually reinforcing representations from visually marked images, and a self-consistent decoding process to improve VLMs spatial planning. Our experiments show that LayoutVLM addresses the limitations of existing LLM and constraint-based approaches, producing physically plausible 3D layouts better aligned with the semantic intent of input language instructions. We also demonstrate that fine-tuning VLMs with the proposed scene layout representation extracted from existing scene datasets can improve performance.
@article{sun_layoutvlm:_2024, author = {Sun, Fan-Yun and Liu, Weiyu and Gu, Siyi and Lim, Dylan and Bhat, Goutam and Tombari, Federico and Li, Manling and Haber, Nick and Wu, Jiajun}, citation = {arXiv preprint arXiv:2412.02193, 2024}, journal = {arXiv preprint arXiv:2412.02193}, year = {2024}, title = {LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models}, _google_scholar_id = {a0OBvERweLwC}, } - Partial-View Object View Synthesis via Filtering InversionFan-Yun Sun, Jonathan Tremblay, Valts Blukis, and 11 more authors2024
We propose Filtering Inversion (FINV), a learning framework and optimization process that predicts a renderable 3D object representation from one or few partial views. FINV addresses the challenge of synthesizing novel views of objects from partial observations, spanning cases where the object is not entirely in view, is partially occluded, or is only observed from similar views. To achieve this, FINV learns shape priors by training a 3D generative model. At inference, given one or more views of a novel real-world object, FINV first finds a set of latent codes for the object by inverting the generative model from multiple initial seeds. Maintaining the set of latent codes, FINV filters and resamples them after receiving each new observation, akin to particle filtering. The generator is then finetuned for each latent code on the available views in order to adapt to novel objects. We show that FINV successfully synthesizes novel
@article{sun_partial-view_2024, author = {Sun, Fan-Yun and Tremblay, Jonathan and Blukis, Valts and Lin, Kevin and Xu, Danfei and Ivanovic, Boris and Karkus, Peter and Birchfield, Stan and Fox, Dieter and Zhang, Ruohan and Li, Yunzhu and Wu, Jiajun and Pavone, Marco and Haber, Nick}, citation = {2024 International Conference on 3D Vision (3DV), 453-463, 2024}, conference = {2024 International Conference on 3D Vision (3DV)}, pages = {453-463}, year = {2024}, publisher = {IEEE}, title = {Partial-View Object View Synthesis via Filtering Inversion}, _google_scholar_id = {rO6llkc54NcC}, } - FactorSim: Generative Simulation via Factorized RepresentationFan-Yun Sun, Harini SI, Angela Yi, and 6 more authorsAdvances in Neural Information Processing Systems, 2024
Generating simulations to train intelligent agents in game-playing and robotics from natural language input, user input, or task documentation remains an open-ended challenge. Existing approaches focus on parts of this challenge, such as generating reward functions or task hyperparameters. Unlike previous work, we introduce FACTORSIM that generates full simulations in code from language input that can be used to train agents. Exploiting the structural modularity specific to coded simulations, we propose to use a factored partially observable Markov decision process representation that allows us to reduce context dependence during each step of the generation. For evaluation, we introduce a generative simulation benchmark that assesses the generated simulation codes accuracy and effectiveness in facilitating zero-shot transfers in reinforcement learning settings. We show that FACTORSIM outperforms existing methods in generating simulations regarding prompt alignment (ie, accuracy), zero-shot transfer abilities, and human evaluation. We also demonstrate its effectiveness in generating robotic tasks.
@article{sun_factorsim:_2024, author = {Sun, Fan-Yun and SI, Harini and Yi, Angela and Zhou, Yihan and Zook, Alex and Tremblay, Jonathan and Cross, Logan and Wu, Jiajun and Haber, Nick}, citation = {Advances in Neural Information Processing Systems 37, 87438-87472, 2024}, journal = {Advances in Neural Information Processing Systems}, pages = {87438-87472}, year = {2024}, title = {FactorSim: Generative Simulation via Factorized Representation}, volume = {37}, _google_scholar_id = {4OULZ7Gr8RgC}, } - Policy-shaped prediction: avoiding distractions in model-based reinforcement learningMiles Hutson, Isaac Kauvar, and Nick HaberarXiv preprint arXiv:2412.05766, 2024
Model-based reinforcement learning (MBRL) is a promising route to sample-efficient policy optimization. However, a known vulnerability of reconstruction-based MBRL consists of scenarios in which detailed aspects of the world are highly predictable, but irrelevant to learning a good policy. Such scenarios can lead the model to exhaust its capacity on meaningless content, at the cost of neglecting important environment dynamics. While existing approaches attempt to solve this problem, we highlight its continuing impact on leading MBRL methods – including DreamerV3 and DreamerPro – with a novel environment where background distractions are intricate, predictable, and useless for planning future actions. To address this challenge we develop a method for focusing the capacity of the world model through synergy of a pretrained segmentation model, a task-aware reconstruction loss, and adversarial learning. Our method outperforms a variety of other approaches designed to reduce the impact of distractors, and is an advance towards robust model-based reinforcement learning.
@article{hutson_policy-shaped_2024, author = {Hutson, Miles and Kauvar, Isaac and Haber, Nick}, citation = {arXiv preprint arXiv:2412.05766, 2024}, journal = {arXiv preprint arXiv:2412.05766}, year = {2024}, title = {Policy-shaped prediction: avoiding distractions in model-based reinforcement learning}, _google_scholar_id = {cFHS6HbyZ2cC}, } - ChatCollab: Exploring Collaboration Between Humans and AI Agents in Software TeamsBenjamin Klieger, Charis Charitsis, Miroslav Suzara, and 3 more authorsarXiv preprint arXiv:2412.01992, 2024
We explore the potential for productive team-based collaboration between humans and Artificial Intelligence (AI) by presenting and conducting initial tests with a general framework that enables multiple human and AI agents to work together as peers. ChatCollab’s novel architecture allows agents - human or AI - to join collaborations in any role, autonomously engage in tasks and communication within Slack, and remain agnostic to whether their collaborators are human or AI. Using software engineering as a case study, we find that our AI agents successfully identify their roles and responsibilities, coordinate with other agents, and await requested inputs or deliverables before proceeding. In relation to three prior multi-agent AI systems for software development, we find ChatCollab AI agents produce comparable or better software in an interactive game development task. We also propose an automated method for analyzing collaboration dynamics that effectively identifies behavioral characteristics of agents with distinct roles, allowing us to quantitatively compare collaboration dynamics in a range of experimental conditions. For example, in comparing ChatCollab AI agents, we find that an AI CEO agent generally provides suggestions 2-4 times more often than an AI product manager or AI developer, suggesting agents within ChatCollab can meaningfully adopt differentiated collaborative roles. Our code and data can be found at: https://github.com/ChatCollab.
@article{klieger_chatcollab:_2024, author = {Klieger, Benjamin and Charitsis, Charis and Suzara, Miroslav and Wang, Sierra and Haber, Nick and Mitchell, John C}, citation = {arXiv preprint arXiv:2412.01992, 2024}, journal = {arXiv preprint arXiv:2412.01992}, year = {2024}, title = {ChatCollab: Exploring Collaboration Between Humans and AI Agents in Software Teams}, _google_scholar_id = {D03iK_w7-QYC}, } - Tutorly: Turning Programming Videos Into Apprenticeship Learning Environments with LLMsWengxi Li, Roy Pea, Nick Haber, and 1 more authorarXiv preprint arXiv:2405.12946, 2024
Online programming videos, including tutorials and streamcasts, are widely popular and contain a wealth of expert knowledge. However, effectively utilizing these resources to achieve targeted learning goals can be challenging. Unlike direct tutoring, video content lacks tailored guidance based on individual learning paces, personalized feedback, and interactive engagement necessary for support and monitoring. Our work transforms programming videos into one-on-one tutoring experiences using the cognitive apprenticeship framework. Tutorly, developed as a JupyterLab Plugin, allows learners to (1) set personalized learning goals, (2) engage in learning-by-doing through a conversational LLM-based mentor agent, (3) receive guidance and feedback based on a student model that steers the mentor moves. In a within-subject study with 16 participants learning exploratory data analysis from a streamcast, Tutorly significantly improved their performance from 61.9% to 76.6% based on a post-test questionnaire. Tutorly demonstrates the potential for enhancing programming video learning experiences with LLM and learner modeling.
@article{li_tutorly:_2024, author = {Li, Wengxi and Pea, Roy and Haber, Nick and Subramonyam, Hari}, citation = {arXiv preprint arXiv:2405.12946, 2024}, journal = {arXiv preprint arXiv:2405.12946}, year = {2024}, title = {Tutorly: Turning Programming Videos Into Apprenticeship Learning Environments with LLMs}, _google_scholar_id = {SeFeTyx0c_EC}, } - Math IDE: A Platform for Creating with MathSierra Wang, John Mitchell, Nick Haber, and 1 more author2024
To inspire student engagement in middle school math, we explore the possibility of using generative AI to enhance the creativity of math learning. We present the Math IDE, a math education environment in which students learn about math concepts by building artifacts. We aimed to create a platform in which students can engage with mathematical concepts, create an artifact that embodies the math that they are learning about, and practice their high-level specification skills. In the current iteration of the Math IDE, students can create custom web pages by describing and demonstrating understanding of the math that is involved in the web page. In this short overview, we describe our process and discuss several open questions regarding the design and application of this novel method of math education.
@article{wang_math_2024, author = {Wang, Sierra and Mitchell, John and Haber, Nick and Piech, Chris}, citation = {Proceedings of the 55th ACM Technical Symposium on Computer Science , 2024}, pages = {1844-1845}, year = {2024}, title = {Math IDE: A Platform for Creating with Math}, _google_scholar_id = {pqnbT2bcN3wC}, } - Animate Agent World Modeling BenchmarkLogan Matthew Cross, Violet Xiang, Nick Haber, and 1 more authorProceedings of the Annual Meeting of the Cognitive Science Society, 2024
To advance the capacity of intuitive psychology in machines, we introduce the Animate Agent World Modeling Benchmark. This benchmark features agents engaged in a diverse repertoire of behaviors, such as goal-directed interactions with objects and multi-agent interactions, all governed by realistic physics. Humans tend to predict the future based on expected events rather than simulating step-by-step. Thus, our benchmark includes a cognitively-inspired evaluation pipeline designed to assess whether the simulated trajectories of world models capture the correct sequences of events. To perform well, models need to leverage predictive cues from the observations to accurately simulate the goals of animate agents over long horizons. We demonstrate that current state-of-the-art models perform poorly in our evaluations. A hierarchical oracle model sets an upper bound for performance, suggesting that to excel, a model should scaffold their predictions with abstractions like goals that guide the simulation process towards relevant future events
@article{cross_animate_2024, author = {Cross, Logan Matthew and Xiang, Violet and Haber, Nick and Yamins, Daniel}, citation = {Proceedings of the Annual Meeting of the Cognitive Science Society 46, 2024}, journal = {Proceedings of the Annual Meeting of the Cognitive Science Society}, year = {2024}, title = {Animate Agent World Modeling Benchmark}, volume = {46}, _google_scholar_id = {zA6iFVUQeVQC}, } - Simulating Infants Attachment: Behavioral Patterns of Caregiver Proximity Seeking and Environment Exploration Using Reinforcement Learning Models.Xi Jia Zhou, Chris Doyle, Michael C Frank, and 1 more authorProceedings of the Annual Meeting of the Cognitive Science Society, 2024
Attachment is crucial for infants’ cognitive development and social relationships. Traditional attachment research has been qualitative, lacking a model to explain how infants’ attachment styles develop from experience and how these are influenced by personal traits and environmental factors. We propose such a model, predicting how infants balance interaction with caregivers against exploring their surroundings. Our study is based in a grid-world environment containing an infant and caregiver agent. We vary the infants temperamental factors (e.g., ability to regulate emotions and preferences for social vs. environmental reward), and caregiver behavior (whether positive or negative interactions are more likely). We find that different equilibria result that qualitatively correspond to different attachment styles. Our findings suggest that the characteristic exploratory behavior of each attachment style in real infants may arise from interactions of infant temperament and caregiver behaviors.
@article{zhou_simulating_2024, author = {Zhou, Xi Jia and Doyle, Chris and Frank, Michael C and Haber, Nick}, citation = {Proceedings of the Annual Meeting of the Cognitive Science Society 46, 2024}, journal = {Proceedings of the Annual Meeting of the Cognitive Science Society}, year = {2024}, title = {Simulating Infants Attachment: Behavioral Patterns of Caregiver Proximity Seeking and Environment Exploration Using Reinforcement Learning Models.}, volume = {46}, _google_scholar_id = {3s1wT3WcHBgC}, } - Nonmyopic Bayesian Optimization in Dynamic Cost SettingsSang T Truong, Duc Quang Nguyen, Willie Neiswanger, and 4 more authors2024
Bayesian optimization (BO) is a popular framework for optimizing black-box functions, leveraging probabilistic models such as Gaussian processes. However, conventional BO assumes static query costs, which limits its applicability to real-world problems with dynamic cost structures, such as geological surveys or biological sequence design, where query costs vary based on previous actions. To address this, we propose a cost-constrained nonmyopic BO algorithm that incorporates dynamic cost models. Our method employs a neural network policy for variational optimization over multi-step lookahead horizons to plan ahead in dynamic cost environments. Empirically, we benchmark our method on synthetic functions exhibiting a variety of dynamic cost structures. Furthermore, we apply our method to a real-world application in protein sequence design using a large language model-based policy, demonstrating its scalability and effectiveness in handling multi-step planning in a large and complex query space. Our nonmyopic BO algorithm consistently outperforms its myopic counterparts in both synthetic and real-world settings, achieving significant improvements in both efficiency and solution quality.
@article{truong_nonmyopic_2024, author = {Truong, Sang T and Nguyen, Duc Quang and Neiswanger, Willie and Griffiths, Ryan-Rhys and Ermon, Stefano and Haber, Nick and Koyejo, Sanmi}, year = {2024}, title = {Nonmyopic Bayesian Optimization in Dynamic Cost Settings}, _google_scholar_id = {b0M2c_1WBrUC}, } - Person-Specific Analyses of Smartphone Use and Mental Health: An Intensive Longitudinal Study Over One YearMerve Cerit, Angela Y Lee, Jeffrey Hancock, and 8 more authorsJMIR formative research, 2024
Contrary to popular concerns about the harmful effects of media use on mental health, research on this relationship is ambiguous, stalling advances in theory, interventions, and policy. Scientific explorations of the relationship between media and mental health have mostly found null or small associations, with the results often blamed on the use of cross-sectional study designs or imprecise measures of media use and mental health.This exploratory empirical demonstration aimed to answer whether mental health effects are associated with media use experiences by (1) redirecting research investments to granular and intensive longitudinal recordings of digital experiences to build models of media use and mental health for single individuals over the course of one entire year, (2) using new metrics of fragmented media use to propose explanations of mental health effects that will advance
@article{cerit_person-specific_2024, author = {Cerit, Merve and Lee, Angela Y and Hancock, Jeffrey and Miner, Adam and Cho, Mu-Jung and Muise, Daniel and Torres, Anna-Angelina Garrn and Haber, Nick and Ram, Nilam and Robinson, Thomas and Reeves, Byron}, citation = {JMIR formative research, 0}, year = {2024}, journal = {JMIR formative research}, title = {Person-Specific Analyses of Smartphone Use and Mental Health: An Intensive Longitudinal Study Over One Year}, _google_scholar_id = {_xSYboBqXhAC}, } - Policy-shaped prediction: improving world modeling through interpretabilityMiles Richard Hutson, Isaac Kauvar, and Nick Haber2024
Model-based reinforcement learning (MBRL) offers sample-efficient policy optimization but is susceptible to distractions. We address this by developing Policy-Shaped Prediction (PSP), a method that empowers agents to interpret their own policies and shape their world models accordingly. By combining gradient-based interpretability, pretrained segmentation models, and adversarial learning, PSP outperforms existing distractor-reduction approaches. This work represents an interpretability-driven advance towards robust MBRL.
@article{hutson_policy-shaped_2025, author = {Hutson, Miles Richard and Kauvar, Isaac and Haber, Nick}, citation = {Interpretable AI: Past, Present and Future, 0}, year = {2024}, conference = {Interpretable AI: Past, Present and Future}, title = {Policy-shaped prediction: improving world modeling through interpretability}, _google_scholar_id = {f2IySw72cVMC}, } - Does Infantile Attachment Require Intrinsic Reward?Xi Jia Zhou, Logan Cross, Wanqiao Xu, and 1 more author2024
Infant attachment behaviors, essential for cognitive and social development, are debated in terms of their originswhether they arise from intrinsic drives or external reinforcement. This study uses a reinforcement learning (RL) framework to examine if infant attachment behaviors can be replicated in a simulated 1D grid world environment. Two groups of simulated infants were modeled: one with intrinsic rewards for staying close to a caregiver, and another relying solely on extrinsic rewards from both the caregiver and the environment. Preliminary results show that external reinforcement can generate infant-like attachment behaviors, though adding intrinsic attachment rewards made some overall infant behaviors more realistic while distorting subgroup patterns. These findings highlight the complex interaction between intrinsic and extrinsic factors in shaping attachment behaviors, suggesting more work is required
@article{zhou_does_2024, author = {Zhou, Xi Jia and Cross, Logan and Xu, Wanqiao and Haber, Nick}, citation = {Intrinsically-Motivated and Open-Ended Learning Workshop@ NeurIPS2024, 0}, conference = {Intrinsically-Motivated and Open-Ended Learning Workshop@ NeurIPS2024}, title = {Does Infantile Attachment Require Intrinsic Reward?}, year = {2024}, _google_scholar_id = {yD5IFk8b50cC}, }
2023
- Hypothesis search: Inductive reasoning with language modelsRuocheng Wang, Eric Zelikman, Gabriel Poesia, and 3 more authorsarXiv preprint arXiv:2309.05660, 2023
Inductive reasoning is a core problem-solving capacity: humans can identify underlying principles from a few examples, which robustly generalize to novel scenarios. Recent work evaluates large language models (LLMs) on inductive reasoning tasks by directly prompting them yielding "in context learning." This works well for straightforward inductive tasks but performs poorly on complex tasks such as the Abstraction and Reasoning Corpus (ARC). In this work, we propose to improve the inductive reasoning ability of LLMs by generating explicit hypotheses at multiple levels of abstraction: we prompt the LLM to propose multiple abstract hypotheses about the problem, in natural language, then implement the natural language hypotheses as concrete Python programs. These programs can be verified by running on observed examples and generalized to novel inputs. To reduce the hypothesis search space, we explore steps to filter the set of hypotheses to implement: we either ask the LLM to summarize them into a smaller set of hypotheses or ask human annotators to select a subset. We verify our pipeline’s effectiveness on the ARC visual inductive reasoning benchmark, its variant 1D-ARC, string transformation dataset SyGuS, and list transformation dataset List Functions. On a random 100-problem subset of ARC, our automated pipeline using LLM summaries achieves 30% accuracy, outperforming the direct prompting baseline (accuracy of 17%). With the minimal human input of selecting from LLM-generated candidates, performance is boosted to 33%. Our ablations show that both abstract hypothesis generation and concrete program
@article{wang_hypothesis_2023, author = {Wang, Ruocheng and Zelikman, Eric and Poesia, Gabriel and Pu, Yewen and Haber, Nick and Goodman, Noah D}, citation = {arXiv preprint arXiv:2309.05660, 2023}, journal = {arXiv preprint arXiv:2309.05660}, year = {2023}, title = {Hypothesis search: Inductive reasoning with language models}, _google_scholar_id = {BqipwSGYUEgC}, } - Parsel: Algorithmic Reasoning with Language Models by Composing DecompositionsEric Zelikman, Qian Huang, Gabriel Poesia, and 2 more authorsAdvances in Neural Information Processing Systems, 2023
Despite recent success in large language model (LLM) reasoning, LLMs struggle with hierarchical multi-step reasoning tasks like generating complex programs. For these tasks, humans often start with a high-level algorithmic design and implement each part gradually. We introduce Parsel, a framework enabling automatic implementation and validation of complex algorithms with code LLMs. With Parsel, we automatically decompose algorithmic tasks into hierarchical natural language function descriptions and then search over combinations of possible function implementations using tests. We show that Parsel can be used across domains requiring hierarchical reasoning, including program synthesis and robotic planning. We find that, using Parsel, LLMs solve more competition-level problems in the APPS dataset, resulting in pass rates over 75\% higher than prior results from directly sampling AlphaCode and Codex, while often using a smaller sample budget. Moreover, with automatically generated tests, we find that Parsel can improve the state-of-the-art pass@ 1 performance on HumanEval from 67\% to 85\%. We also find that LLM-generated robotic plans using Parsel are more than twice as likely to be considered accurate than directly generated plans. Lastly, we explore how Parsel addresses LLM limitations and discuss how Parsel may be useful for human programmers. We release our code at https://github. com/ezelikman/parsel.
@article{zelikman_parsel:_2023, author = {Zelikman, Eric and Huang, Qian and Poesia, Gabriel and Goodman, Noah and Haber, Nick}, citation = {Advances in Neural Information Processing Systems 36, 31466-31523, 2023}, journal = {Advances in Neural Information Processing Systems}, pages = {31466-31523}, year = {2023}, title = {Parsel: Algorithmic Reasoning with Language Models by Composing Decompositions}, volume = {36}, _google_scholar_id = {RYcK_YlVTxYC}, } - Curious replay for model-based adaptationIsaac Kauvar, Chris Doyle, Linqi Zhou, and 1 more authorarXiv preprint arXiv:2306.15934, 2023
Agents must be able to adapt quickly as an environment changes. We find that existing model-based reinforcement learning agents are unable to do this well, in part because of how they use past experiences to train their world model. Here, we present Curious Replay – a form of prioritized experience replay tailored to model-based agents through use of a curiosity-based priority signal. Agents using Curious Replay exhibit improved performance in an exploration paradigm inspired by animal behavior and on the Crafter benchmark. DreamerV3 with Curious Replay surpasses state-of-the-art performance on Crafter, achieving a mean score of 19.4 that substantially improves on the previous high score of 14.5 by DreamerV3 with uniform replay, while also maintaining similar performance on the Deepmind Control Suite. Code for Curious Replay is available at https://github.com/AutonomousAgentsLab/curiousreplay
@article{kauvar_curious_2023, author = {Kauvar, Isaac and Doyle, Chris and Zhou, Linqi and Haber, Nick}, citation = {arXiv preprint arXiv:2306.15934, 2023}, journal = {arXiv preprint arXiv:2306.15934}, year = {2023}, title = {Curious replay for model-based adaptation}, _google_scholar_id = {blknAaTinKkC}, } - Just one byte (per gradient): A note on low-bandwidth decentralized language model finetuning using shared randomnessEric Zelikman, Qian Huang, Percy Liang, and 2 more authorsarXiv preprint arXiv:2306.10015, 2023
Language model training in distributed settings is limited by the communication cost of gradient exchanges. In this short note, we extend recent work from Malladi et al. (2023), using shared randomness to perform distributed fine-tuning with low bandwidth. The method is a natural decentralized extension of memory-efficient Simultaneous Perturbation Stochastic Approximation (SPSA). Each iteration, each machine seeds a Random Number Generator (RNG) to perform local reproducible perturbations on model weights and calculate and exchange scalar projected gradients, which are then used to update each model. By using a (machine, sample) identifier as the random seed, each model can regenerate one another’s perturbations. As machines only exchange single-byte projected gradients, this is highly communication efficient. There are also potential privacy benefits, as projected gradients may be calculated on different training data, and models never access the other’s data. Our approach not only drastically reduces communication bandwidth requirements but also accommodates dynamic addition or removal of machines during the training process and retains the memory-efficient and inference-only advantages of recent work. We perform proof-of-concept experiments to demonstrate the potential usefulness of this method, building off of rich literature on distributed optimization and memory-efficient training.
@article{zelikman_just_2023, author = {Zelikman, Eric and Huang, Qian and Liang, Percy and Haber, Nick and Goodman, Noah D}, citation = {arXiv preprint arXiv:2306.10015, 2023}, journal = {arXiv preprint arXiv:2306.10015}, year = {2023}, title = {Just one byte (per gradient): A note on low-bandwidth decentralized language model finetuning using shared randomness}, _google_scholar_id = {RGFaLdJalmkC}, } - Generating and evaluating tests for k-12 students with language model simulations: A case study on sentence reading efficiencyEric Zelikman, Wanjing Anya Ma, Jasmine E Tran, and 3 more authorsarXiv preprint arXiv:2310.06837, 2023
Developing an educational test can be expensive and time-consuming, as each item must be written by experts and then evaluated by collecting hundreds of student responses. Moreover, many tests require multiple distinct sets of questions administered throughout the school year to closely monitor students’ progress, known as parallel tests. In this study, we focus on tests of silent sentence reading efficiency, used to assess students’ reading ability over time. To generate high-quality parallel tests, we propose to fine-tune large language models (LLMs) to simulate how previous students would have responded to unseen items. With these simulated responses, we can estimate each item’s difficulty and ambiguity. We first use GPT-4 to generate new test items following a list of expert-developed rules and then apply a fine-tuned LLM to filter the items based on criteria from psychological measurements. We also propose an optimal-transport-inspired technique for generating parallel tests and show the generated tests closely correspond to the original test’s difficulty and reliability based on crowdworker responses. Our evaluation of a generated test with 234 students from grades 2 to 8 produces test scores highly correlated (r=0.93) to those of a standard test form written by human experts and evaluated across thousands of K-12 students.
@article{zelikman_generating_2023, author = {Zelikman, Eric and Ma, Wanjing Anya and Tran, Jasmine E and Yang, Diyi and Yeatman, Jason D and Haber, Nick}, citation = {arXiv preprint arXiv:2310.06837, 2023}, journal = {arXiv preprint arXiv:2310.06837}, year = {2023}, title = {Generating and evaluating tests for k-12 students with language model simulations: A case study on sentence reading efficiency}, _google_scholar_id = {NaGl4SEjCO4C}, } - ContextRef: evaluating referenceless metrics for image description generationElisa Kreiss, Eric Zelikman, Christopher Potts, and 1 more authorarXiv preprint arXiv:2309.11710, 2023
Referenceless metrics (e.g., CLIPScore) use pretrained vision–language models to assess image descriptions directly without costly ground-truth reference texts. Such methods can facilitate rapid progress, but only if they truly align with human preference judgments. In this paper, we introduce ContextRef, a benchmark for assessing referenceless metrics for such alignment. ContextRef has two components: human ratings along a variety of established quality dimensions, and ten diverse robustness checks designed to uncover fundamental weaknesses. A crucial aspect of ContextRef is that images and descriptions are presented in context, reflecting prior work showing that context is important for description quality. Using ContextRef, we assess a variety of pretrained models, scoring functions, and techniques for incorporating context. None of the methods is successful with ContextRef, but we show that careful fine-tuning yields substantial improvements. ContextRef remains a challenging benchmark though, in large part due to the challenge of context dependence.
@article{kreiss_contextref:_2023, author = {Kreiss, Elisa and Zelikman, Eric and Potts, Christopher and Haber, Nick}, citation = {arXiv preprint arXiv:2309.11710, 2023}, journal = {arXiv preprint arXiv:2309.11710}, year = {2023}, title = {ContextRef: evaluating referenceless metrics for image description generation}, _google_scholar_id = {ns9cj8rnVeAC}, } - Measuring and Modeling Physical Intrinsic MotivationJulio Martinez, Felix Binder, Haoliang Wang, and 3 more authorsarXiv preprint arXiv:2305.13452, 2023
Humans are interactive agents driven to seek out situations with interesting physical dynamics. Here we formalize the functional form of physical intrinsic motivation. We first collect ratings of how interesting humans find a variety of physics scenarios. We then model human interestingness responses by implementing various hypotheses of intrinsic motivation including models that rely on simple scene features to models that depend on forward physics prediction. We find that the single best predictor of human responses is adversarial reward, a model derived from physical prediction loss. We also find that simple scene feature models do not generalize their prediction of human responses across all scenarios. Finally, linearly combining the adversarial model with the number of collisions in a scene leads to the greatest improvement in predictivity of human responses, suggesting humans are driven towards scenarios that result in high information gain and physical activity.
@article{martinez_measuring_2023, author = {Martinez, Julio and Binder, Felix and Wang, Haoliang and Haber, Nick and Fan, Judith and Yamins, Daniel LK}, citation = {arXiv preprint arXiv:2305.13452, 2023}, journal = {arXiv preprint arXiv:2305.13452}, year = {2023}, title = {Measuring and Modeling Physical Intrinsic Motivation}, _google_scholar_id = {HoB7MX3m0LUC}, } - Developmental curiosity and social interaction in virtual agentsChris Doyle, Sarah Shader, Michelle Lau, and 3 more authorsarXiv preprint arXiv:2305.13396, 2023
Infants explore their complex physical and social environment in an organized way. To gain insight into what intrinsic motivations may help structure this exploration, we create a virtual infant agent and place it in a developmentally-inspired 3D environment with no external rewards. The environment has a virtual caregiver agent with the capability to interact contingently with the infant agent in ways that resemble play. We test intrinsic reward functions that are similar to motivations that have been proposed to drive exploration in humans: surprise, uncertainty, novelty, and learning progress. These generic reward functions lead the infant agent to explore its environment and discover the contingencies that are embedded into the caregiver agent. The reward functions that are proxies for novelty and uncertainty are the most successful in generating diverse experiences and activating the environment contingencies. We also find that learning a world model in the presence of an attentive caregiver helps the infant agent learn how to predict scenarios with challenging social and physical dynamics. Taken together, our findings provide insight into how curiosity-like intrinsic rewards and contingent social interaction lead to dynamic social behavior and the creation of a robust predictive world model.
@article{doyle_developmental_2023, author = {Doyle, Chris and Shader, Sarah and Lau, Michelle and Sano, Megumi and Yamins, Daniel LK and Haber, Nick}, citation = {arXiv preprint arXiv:2305.13396, 2023}, journal = {arXiv preprint arXiv:2305.13396}, year = {2023}, title = {Developmental curiosity and social interaction in virtual agents}, _google_scholar_id = {JV2RwH3_ST0C}, } - Understanding generative artificial intelligence and its relationship to copyrightChristopher Callison-Burch, A Zhu, L Dugan, and 7 more authorsProceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL 2023), University of Pennsylvania, School of Engineering and Applied Sciences, 2023
BackgroundGenerative AI had a breakthrough moment last November with the release of OpenAIs ChatGPT, bringing my field of research into the public eye and generating significant excitement. I had early access to OpenAI’s private Beta in June 2021 and was a visiting researcher at Google in 2019 and 2020, where I used their LaMDA language model before its public announcement. Despite my two decades of experience in the field, I was similarly astonished by the capabilities of these large language models (LLMs), and I believe that this is a truly transformative technology. It is crucial that we carefully consider and draft legislation that simultaneously encourages innovation and guards against potential negative impacts on society.
@article{callison-burch_understanding_2023, author = {Callison-Burch, Christopher and Zhu, A and Dugan, L and Hwang, A and Callison-Burch, C and Lyu, Q and Havaldar, S and Stein, A and Zhang, L and Rao, D}, citation = {Proceedings of the 13th International Joint Conference on Natural Language , 2023}, journal = {Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL 2023), University of Pennsylvania, School of Engineering and Applied Sciences}, pages = {370-387}, year = {2023}, title = {Understanding generative artificial intelligence and its relationship to copyright}, _google_scholar_id = {dfsIfKJdRG4C}, } - Tingets 2.0: Computer Vision-Powered Interactive Social and Emotional Learning ToolMerve Cerit, Daniela Vainer, and Nick Haber2023
Young children between the ages of 3 and 6 often struggle with identifying, expressing, and managing their emotions, especially when it comes to negative emotions. Research indicates that children with higher emotional skills tend to experience greater long-term mental health and well-being benefits. Building on our previous work, we introduce Tingets 2.0, an interactive storytelling app powered by computer vision to facilitate social and emotional learning (SEL) through tangible and digital play. We propose the TinkeRERR framework: Reflect, Express, Regulate, and Repeat, and utilize computer vision technology to combine tangible and digital play seamlessly. Tingets 2.0 aims to help children understand their own and others emotions, express emotions constructively, manage negative emotions and cultivate positive emotions. Early results indicate that our novel approach has effectively increased the depth
@article{cerit_tingets_2023, author = {Cerit, Merve and Vainer, Daniela and Haber, Nick}, citation = {Proceedings of the 2023 Symposium on Learning, Design and Technology, 53-59, 2023}, pages = {53-59}, year = {2023}, title = {Tingets 2.0: Computer Vision-Powered Interactive Social and Emotional Learning Tool}, _google_scholar_id = {hMod-77fHWUC}, } - Intrinsically motivated social play in virtual infantsChris Doyle, Sarah Shader, Michelle Lau, and 3 more authors2023
Infants explore their complex physical and social environment in an organized way. To gain insight into what intrinsic motivations may help structure this exploration, we create a virtual infant agent and place it in a developmentally-inspired 3D environment with no external rewards. The environment has a virtual caregiver agent with the capability to interact contingently with the infant agent in ways that resemble play. We test intrinsic reward functions that are similar to motivations that have been proposed to drive exploration in humans: surprise, uncertainty, novelty, and learning progress. The reward functions that are proxies for novelty and uncertainty are the most successful in generating diverse experiences and activating the environment contingencies. We also find that learning a world model in the presence of an attentive caregiver helps the infant agent learn how to predict scenarios with challenging social and physical dynamics. Our findings provide insight into how curiosity-like intrinsic rewards and contingent social interaction lead to social behavior and the creation of a robust predictive world model.
@article{doyle_intrinsically_2023, author = {Doyle, Chris and Shader, Sarah and Lau, Michelle and Sano, Megumi and Yamins, Daniel and Haber, Nick}, citation = {Intrinsically-Motivated and Open-Ended Learning Workshop@ NeurIPS2023, 2023}, conference = {Intrinsically-Motivated and Open-Ended Learning Workshop@ NeurIPS2023}, year = {2023}, title = {Intrinsically motivated social play in virtual infants}, _google_scholar_id = {35N4QoGY0k4C}, } - Modeling Social Learning Through Demonstration in Multi-Armed BanditsJulio Martinez, Michael C Frank, and Nick HaberProceedings of the Annual Meeting of the Cognitive Science Society, 2023
Humans are efficient social learners who leverage social information to rapidly adapt to new environments, but the computations by which we combine social information with prior knowledge are poorly understood. We study social learning within the context of multi-armed bandits using a novel asteroid mining video game where participants learn through active play and passive observation of expert and novice players. We simulate human exploration and social learning using naive versions of Thompson and Upper Confidence Bound (UCB) solvers and hybrid models that use Thompson and UCB solvers for direct learning together with a multi-layer perceptron to estimate what should be learned from other players. Two variants of the hybrid models provide good, parameter-free fits to human performance across a range of learning conditions. Our work shows a route for integrating social learning into reinforcement learning models and suggests that human social learning conforms to the predictions of such models.
@article{martinez_modeling_2023, author = {Martinez, Julio and Frank, Michael C and Haber, Nick}, citation = {Proceedings of the Annual Meeting of the Cognitive Science Society 46, 2023}, journal = {Proceedings of the Annual Meeting of the Cognitive Science Society}, year = {2023}, title = {Modeling Social Learning Through Demonstration in Multi-Armed Bandits}, volume = {46}, _google_scholar_id = {ZHo1McVdvXMC}, } - From Centralized to Self-Supervised: Pursuing Realistic Multi-Agent Reinforcement LearningViolet Xiang, Logan Cross, Jan-Philipp Frnken, and 1 more authorarXiv preprint arXiv:2312.08662, 2023
In real-world environments, autonomous agents rely on their egocentric observations. They must learn adaptive strategies to interact with others who possess mixed motivations, discernible only through visible cues. Several Multi-Agent Reinforcement Learning (MARL) methods adopt centralized approaches that involve either centralized training or reward-sharing, often violating the realistic ways in which living organisms, like animals or humans, process information and interact. MARL strategies deploying decentralized training with intrinsic motivation offer a self-supervised approach, enable agents to develop flexible social strategies through the interaction of autonomous agents. However, by contrasting the self-supervised and centralized methods, we reveal that populations trained with reward-sharing methods surpass those using self-supervised methods in a mixed-motive environment. We link this superiority to specialized role emergence and an agent’s expertise in its role. Interestingly, this gap shrinks in pure-motive settings, emphasizing the need for evaluations in more complex, realistic environments (mixed-motive). Our preliminary results suggest a gap in population performance that can be closed by improving self-supervised methods and thereby pushing MARL closer to real-world readiness.
@article{xiang_from_2023, author = {Xiang, Violet and Cross, Logan and Frnken, Jan-Philipp and Haber, Nick}, citation = {arXiv preprint arXiv:2312.08662, 2023}, journal = {arXiv preprint arXiv:2312.08662}, year = {2023}, title = {From Centralized to Self-Supervised: Pursuing Realistic Multi-Agent Reinforcement Learning}, _google_scholar_id = {lSLTfruPkqcC}, } - Partial-View Object View Synthesis via Filtered InversionFan-Yun Sun, Jonathan Tremblay, Valts Blukis, and 11 more authorsarXiv preprint arXiv:2304.00673, 2023
We propose Filtering Inversion (FINV), a learning framework and optimization process that predicts a renderable 3D object representation from one or few partial views. FINV addresses the challenge of synthesizing novel views of objects from partial observations, spanning cases where the object is not entirely in view, is partially occluded, or is only observed from similar views. To achieve this, FINV learns shape priors by training a 3D generative model. At inference, given one or more views of a novel real-world object, FINV first finds a set of latent codes for the object by inverting the generative model from multiple initial seeds. Maintaining the set of latent codes, FINV filters and resamples them after receiving each new observation, akin to particle filtering. The generator is then finetuned for each latent code on the available views in order to adapt to novel objects. We show that FINV successfully synthesizes novel views of real-world objects (e.g., chairs, tables, and cars), even if the generative prior is trained only on synthetic objects. The ability to address the sim-to-real problem allows FINV to be used for object categories without real-world datasets. FINV achieves state-of-the-art performance on multiple real-world datasets, recovers object shape and texture from partial and sparse views, is robust to occlusion, and is able to incrementally improve its representation with more observations.
@article{sun_partial-view_2023, author = {Sun, Fan-Yun and Tremblay, Jonathan and Blukis, Valts and Lin, Kevin and Xu, Danfei and Ivanovic, Boris and Karkus, Peter and Birchfield, Stan and Fox, Dieter and Zhang, Ruohan and Li, Yunzhu and Wu, Jiajun and Pavone, Marco and Haber, Nick}, citation = {arXiv preprint arXiv:2304.00673, 2023}, journal = {arXiv preprint arXiv:2304.00673}, year = {2023}, title = {Partial-View Object View Synthesis via Filtered Inversion}, _google_scholar_id = {M3NEmzRMIkIC}, } - Reinforcement learning models of tradeoffs between infant attachment and curiosityXi Jia Zhou, Frieda Rong, Michael Frank, and 1 more authorProceedings of the Annual Meeting of the Cognitive Science Society, 2023
Infants must balance interacting with a caregiver and navigating the environment, driven by their curiosity and attachment. This decision involves trading off proximity seeking with the caregiver and novelty seeking in the surroundings. Our hypothesis is that different attachment styles may arise as adaptive reward-seeking strategies based on different frequencies and scales of rewards from both caregiver and environment, including negative rewards due to distressing encounters. We implement a computational model using a reinforcement learning framework to model infants attachment and exploratory behaviors as a bandit problem. We study how infants exploratory behaviors emerge as they learn from their caregiver and how these learned behaviors transfer to a new caregiver. Our work expands upon existing computational models of attachment and models how social factors can influence curiosity-seeking as infants learn to explore versus exploit.
@article{zhou_reinforcement_2023, author = {Zhou, Xi Jia and Rong, Frieda and Frank, Michael and Haber, Nick}, citation = {Proceedings of the Annual Meeting of the Cognitive Science Society, 2023}, journal = {Proceedings of the Annual Meeting of the Cognitive Science Society}, year = {2023}, title = {Reinforcement learning models of tradeoffs between infant attachment and curiosity}, _google_scholar_id = {vV6vV6tmYwMC}, } - Neurobehavior of exploring AI agentsIsaac Kauvar, Chris Doyle, and Nick Haber2023
We study intrinsically motivated exploration by artificially intelligent (AI) agents in animal-inspired settings. We construct virtual environments that are 3D, vision-based, physics-simulated, and based on two established animal assays: labyrinth exploration, and novel object interaction. We assess Plan2Explore (P2E), a leading model-based, intrinsically motivated deep reinforcement learning agent, in these environments. We characterize and compare the behavior of the AI agents to animal behavior, using measures devised for animal neuroethology. P2E exhibits some similarities to animal behavior, but is dramatically less efficient than mice at labyrinth exploration. We further characterize the neural dynamics associated with world modeling in the novel-object assay. We identify latent neural population activity axes linearly associated with representing object proximity. These results identify areas of improvement for existing AI agents, and make strides toward understanding the learned neural dynamics that guide their behavior.
@article{kauvar_neurobehavior_2023, author = {Kauvar, Isaac and Doyle, Chris and Haber, Nick}, citation = {Intrinsically-Motivated and Open-Ended Learning Workshop@ NeurIPS2023, 0}, conference = {Intrinsically-Motivated and Open-Ended Learning Workshop@ NeurIPS2023}, title = {Neurobehavior of exploring AI agents}, year = {2023}, _google_scholar_id = {70eg2SAEIzsC}, }
2022
- Improved digital therapy for developmental pediatrics using domain-specific artificial intelligence: machine learning studyPeter Washington, Haik Kalantarian, John Kent, and 16 more authorsJMIR pediatrics and parenting, 2022
Background: Automated emotion classification could aid those who struggle to recognize emotions, including children with developmental behavioral conditions such as autism. However, most computer vision emotion recognition models are trained on adult emotion and therefore underperform when applied to child faces.Objective: We designed a strategy to gamify the collection and labeling of child emotionenriched images to boost the performance of automatic child emotion recognition models to a level closer to what will be needed for digital health care approaches.Methods: We leveraged our prototype therapeutic smartphone game, GuessWhat, which was designed in large part for children with developmental and behavioral conditions, to gamify the secure collection of video data of children expressing a variety of emotions prompted by the game. Independently, we created a secure web interface to gamify the human labeling effort, called HollywoodSquares, tailored for use by any qualified labeler. We gathered and labeled 2155 videos, 39,968 emotion frames, and 106,001 labels on all images. With this drastically expanded pediatric emotioncentric database (> 30 times larger than existing public pediatric emotion data sets), we trained a convolutional neural network (CNN) computer vision classifier of happy, sad, surprised, fearful, angry, disgust, and neutral expressions evoked by children.Results: The classifier achieved a 66.9% balanced accuracy and 67.4% F1-score on the entirety of the Child Affective Facial Expression (CAFE) as well as a 79.1% balanced accuracy and 78% F1-score on CAFE Subset A, a subset containing at
@article{washington_improved_2022, author = {Washington, Peter and Kalantarian, Haik and Kent, John and Husic, Arman and Kline, Aaron and Leblanc, Emilie and Hou, Cathy and Mutlu, Onur Cezmi and Dunlap, Kaitlyn and Penev, Yordan and Varma, Maya and Stockham, Nate Tyler and Chrisman, Brianna and Paskov, Kelley and Sun, Min Woo and Jung, Jae-Yoon and Voss, Catalin and Haber, Nick and Wall, Dennis Paul}, citation = {JMIR pediatrics and parenting 5 (2), e26760, 2022}, journal = {JMIR pediatrics and parenting}, number = {2}, pages = {e26760}, year = {2022}, publisher = {JMIR Publications Inc., Toronto, Canada}, title = {Improved digital therapy for developmental pediatrics using domain-specific artificial intelligence: machine learning study}, volume = {5}, _google_scholar_id = {iH-uZ7U-co4C}, } - Interaction modeling with multiplex attentionFan-Yun Sun, Isaac Kauvar, Ruohan Zhang, and 4 more authorsAdvances in Neural Information Processing Systems, 2022
Modeling multi-agent systems requires understanding how agents interact. Such systems are often difficult to model because they can involve a variety of types of interactions that layer together to drive rich social behavioral dynamics. Here we introduce a method for accurately modeling multi-agent systems. We present Interaction Modeling with Multiplex Attention (IMMA), a forward prediction model that uses a multiplex latent graph to represent multiple independent types of interactions and attention to account for relations of different strengths. We also introduce Progressive Layer Training, a training strategy for this architecture. We show that our approach outperforms state-of-the-art models in trajectory forecasting and relation inference, spanning three multi-agent scenarios: social navigation, cooperative task achievement, and team sports. We further demonstrate that our approach can improve zero-shot generalization and allows us to probe how different interactions impact agent behavior.
@article{sun_interaction_2022, author = {Sun, Fan-Yun and Kauvar, Isaac and Zhang, Ruohan and Li, Jiachen and Kochenderfer, Mykel J and Wu, Jiajun and Haber, Nick}, citation = {Advances in Neural Information Processing Systems 35, 20038-20050, 2022}, journal = {Advances in Neural Information Processing Systems}, pages = {20038-20050}, year = {2022}, title = {Interaction modeling with multiplex attention}, volume = {35}, _google_scholar_id = {k_IJM867U9cC}, } - Communication skills training using remote augmented reality medical simulation: a feasibility and acceptability qualitative studyOlivia Hess, Jimmy Qian, Janine Bruce, and 4 more authorsMedical science educator, 2022
Augmented reality (AR) has promise as a clinical teaching tool, particularly for remote learning. The Chariot Augmented Reality Medical (CHARM) simulator integrates real-time communication into a portable medical simulator with a holographic patient and monitor. The primary aim was to analyze feedback from medical and physician assistant students regarding acceptability and feasibility of the simulator.Using the CHARM simulator, we created an advanced cardiovascular life support (ACLS) simulation scenario. After IRB approval, preclinical medical and physician assistant students volunteered to participate from August to September 2020. We delivered augmented reality headsets (Magic Leap One) to students before the study. Prior to the simulation, via video conference, we introduced students to effective communication skills during a cardiac arrest. Participants then, individually and
@article{hess_communication_2022, author = {Hess, Olivia and Qian, Jimmy and Bruce, Janine and Wang, Ellen and Rodriguez, Samuel and Haber, Nick and Caruso, Thomas J}, citation = {Medical science educator 32 (5), 1005-1014, 2022}, journal = {Medical science educator}, number = {5}, pages = {1005-1014}, year = {2022}, publisher = {Springer US}, title = {Communication skills training using remote augmented reality medical simulation: a feasibility and acceptability qualitative study}, volume = {32}, _google_scholar_id = {isC4tDSrTZIC}, } - Challenges and opportunities for machine learning classification of behavior and mental state from imagesPeter Washington, Cezmi Onur Mutlu, Aaron Kline, and 7 more authors2022
Computer Vision (CV) classifiers which distinguish and detect nonverbal social human behavior and mental state can aid digital diagnostics and therapeutics for psychiatry and the behavioral sciences. While CV classifiers for traditional and structured classification tasks can be developed with standard machine learning pipelines for supervised learning consisting of data labeling, preprocessing, and training a convolutional neural network, there are several pain points which arise when attempting this process for behavioral phenotyping. Here, we discuss the challenges and corresponding opportunities in this space, including handling heterogeneous data, avoiding biased models, labeling massive and repetitive data sets, working with ambiguous or compound class labels, managing privacy concerns, creating appropriate representations, and personalizing models. We discuss current state-of-the-art research endeavors in CV such as data curation, data augmentation, crowdsourced labeling, active learning, reinforcement learning, generative models, representation learning, federated learning, and meta-learning. We highlight at least some of the machine learning advancements needed for imaging classifiers to detect human social cues successfully and reliably.
@article{washington_challenges_2022, author = {Washington, Peter and Mutlu, Cezmi Onur and Kline, Aaron and Paskov, Kelley and Stockham, Nate Tyler and Chrisman, Brianna and Deveau, Nick and Surhabi, Mourya and Haber, Nick and Wall, Dennis P}, citation = {arXiv preprint arXiv:2201.11197, 2022}, year = {2022}, title = {Challenges and opportunities for machine learning classification of behavior and mental state from images}, _google_scholar_id = {j3f4tGmQtD8C}, } - Curiosity and Interactive Learning in Artificial Systems2022
If we were to distill the learning that we see in a childs playroom into a computer program, how would we? We might start by describing essential propertiesthe engineering specifications of childhood learning. Early childhood learning is incredibly interactive (Fantz 1964; Gopnik et al. 1999; Begus et al. 2014; Goupil et al. 2016; Twomey and Westermann 2018). Children play, grabbing and manipulating objects, learning about the properties and affordances of their worlds. Their learning is both autonomous and social. They engage in incredibly complex self-play, yet they also learn from demonstration and imitation (Tomasello et al. 1993; Tomasello 2016). Further, their behavior is curiosity-driven, satisfying not only instrumental needs, but also intrinsic motivations to understand and control (Kidd et al. 2012; Dweck 2017). In engaging in these activities, they build powerful, general representations about their worlds, including those that give them a sense of intuitive physics (Spelke 1985) and intuitive psychology (Colle et al. 2007; Woodward 2009).
@article{haber_curiosity_2022, author = {Haber, Nick}, citation = {AI in learning: Designing the future, 37-54, 2022}, pages = {37-54}, year = {2022}, publisher = {Springer International Publishing}, title = {Curiosity and Interactive Learning in Artificial Systems}, _google_scholar_id = {maZDTaKrznsC}, } - Measuring social curiosity-driven attentional differences in children with autism using an augmented reality-based phone appSamaher F Radwan, Aaron Kline, Alejandro Galindo, and 3 more authorsProceedings of the Annual Meeting of the Cognitive Science Society, 2022
In learning about and building models of their social and physical worlds, children exhibit a wide range of complex, spontaneous, intrinsically motivated, curiosity-driven behaviors. To understand how curiosity-driven behavior in social environments differs in diverse learners, we experimentally measure attention allocation in children on the autism spectrum and their typically-developing peers. We designed an augmented reality smartphone activity where children freely explore novel stimuli. Some agents behave socially (i.e., animate), while others have more regular (i.e., periodic, static) or irregular (i.e., random) inanimate behavioral patterns. This project is part of a broader program in which we attempt to model curiosity with artificial intelligence. Our augmented reality systems will lead to large and diverse data acquisition, allowing us to model a comprehensive range of learning behaviors and enable more inclusive, personalized pedagogical and diagnostic tools.
@article{radwan_measuring_2022, author = {Radwan, Samaher F and Kline, Aaron and Galindo, Alejandro and Frank, Michael C and Wall, Dennis and Haber, Nick}, citation = {Proceedings of the Annual Meeting of the Cognitive Science Society 44 (44), 2022}, journal = {Proceedings of the Annual Meeting of the Cognitive Science Society}, number = {44}, year = {2022}, title = {Measuring social curiosity-driven attentional differences in children with autism using an augmented reality-based phone app}, volume = {44}, _google_scholar_id = {TFP_iSt0sucC}, }
2021
- Crowdsourced privacy-preserved feature tagging of short home videos for machine learning ASD detectionPeter Washington, Qandeel Tariq, Emilie Leblanc, and 14 more authorsScientific reports, 2021
Standard medical diagnosis of mental health conditions requires licensed experts who are increasingly outnumbered by those at risk, limiting reach. We test the hypothesis that a trustworthy crowd of non-experts can efficiently annotate behavioral features needed for accurate machine learning detection of the common childhood developmental disorder Autism Spectrum Disorder (ASD) for children under 8 years old. We implement a novel process for identifying and certifying a trustworthy distributed workforce for video feature extraction, selecting a workforce of 102 workers from a pool of 1,107. Two previously validated ASD logistic regression classifiers, evaluated against parent-reported diagnoses, were used to assess the accuracy of the trusted crowds ratings of unstructured home videos. A representative balanced sample (N = 50 videos) of videos were evaluated with and without face box and pitch shift
@article{washington_crowdsourced_2021, author = {Washington, Peter and Tariq, Qandeel and Leblanc, Emilie and Chrisman, Brianna and Dunlap, Kaitlyn and Kline, Aaron and Kalantarian, Haik and Penev, Yordan and Paskov, Kelley and Voss, Catalin and Stockham, Nathaniel and Varma, Maya and Husic, Arman and Kent, Jack and Haber, Nick and Winograd, Terry and Wall, Dennis P}, citation = {Scientific reports 11 (1), 7620, 2021}, journal = {Scientific reports}, number = {1}, pages = {7620}, year = {2021}, publisher = {Nature Publishing Group UK}, title = {Crowdsourced privacy-preserved feature tagging of short home videos for machine learning ASD detection}, volume = {11}, _google_scholar_id = {_Qo2XoVZTnwC}, } - Selection of trustworthy crowd workers for telemedical diagnosis of pediatric autism spectrum disorderPeter Washington, Emilie Leblanc, Kaitlyn Dunlap, and 11 more authorsPacific Symposium on Biocomputing. Pacific Symposium on Biocomputing, 2021
Crowd-powered telemedicine has the potential to revolutionize healthcare, especially during times that require remote access to care. However, sharing private health data with strangers from around the world is not compatible with data privacy standards, requiring a stringent filtration process to recruit reliable and trustworthy workers who can go through the proper training and security steps. The key challenge, then, is to identify capable, trustworthy, and reliable workers through high-fidelity evaluation tasks without exposing any sensitive patient data during the evaluation process. We contribute a set of experimentally validated metrics for assessing the trustworthiness and reliability of crowd workers tasked with providing behavioral feature tags to unstructured videos of children with autism and matched neurotypical controls. The workers are blinded to diagnosis and blinded to the goal of using the features to
@article{washington_selection_2021, author = {Washington, Peter and Leblanc, Emilie and Dunlap, Kaitlyn and Penev, Yordan and Varma, Maya and Jung, Jae-Yoon and Chrisman, Brianna and Sun, Min Woo and Stockham, Nathaniel and Paskov, Kelley Marie and Kalantarian, Haik and Voss, Catalin and Haber, Nick and Wall, Dennis P}, citation = {Pacific Symposium on Biocomputing. Pacific Symposium on Biocomputing 26, 14, 2021}, journal = {Pacific Symposium on Biocomputing. Pacific Symposium on Biocomputing}, pages = {14}, year = {2021}, title = {Selection of trustworthy crowd workers for telemedical diagnosis of pediatric autism spectrum disorder}, volume = {26}, _google_scholar_id = {-f6ydRqryjwC}, } - Training affective computer vision models by crowdsourcing soft-target labelsPeter Washington, Haik Kalantarian, Jack Kent, and 14 more authorsCognitive computation, 2021
Emotion detection classifiers traditionally predict discrete emotions. However, emotion expressions are often subjective, thus requiring a method to handle compound and ambiguous labels. We explore the feasibility of using crowdsourcing to acquire reliable soft-target labels and evaluate an emotion detection classifier trained with these labels. We hypothesize that training with labels that are representative of the diversity of human interpretation of an image will result in predictions that are similarly representative on a disjoint test set. We also hypothesize that crowdsourcing can generate distributions which mirror those generated in a lab setting. We center our study on the Child Affective Facial Expression (CAFE) dataset, a gold standard collection of images depicting pediatric facial expressions along with 100 human labels per image. To test the feasibility of crowdsourcing to generate these labels, we
@article{washington_training_2021, author = {Washington, Peter and Kalantarian, Haik and Kent, Jack and Husic, Arman and Kline, Aaron and Leblanc, Emilie and Hou, Cathy and Mutlu, Cezmi and Dunlap, Kaitlyn and Penev, Yordan and Stockham, Nate and Chrisman, Brianna and Paskov, Kelley and Jung, Jae-Yoon and Voss, Catalin and Haber, Nick and Wall, Dennis P}, citation = {Cognitive computation 13, 1363-1373, 2021}, journal = {Cognitive computation}, pages = {1363-1373}, year = {2021}, publisher = {Springer US}, title = {Training affective computer vision models by crowdsourcing soft-target labels}, volume = {13}, _google_scholar_id = {RHpTSmoSYBkC}, } - Integrated eye tracking on Magic Leap One during augmented reality medical simulation: a technical reportThomas J Caruso, Olivia Hess, Kenny Roy, and 4 more authorsBMJ simulation & technology enhanced learning, 2021
Augmented reality (AR) has been studied as a clinical teaching tool, however eye-tracking capabilities integrated within an AR medical simulator have limited research. The recently developed Chariot Augmented Reality Medical (CHARM) simulator integrates real-time communication into a portable medical simulator. The purpose of this project was to refine the gaze-tracking capabilities of the CHARM simulator on the Magic Leap One (ML1). Adults aged 18 years and older were recruited using convenience sampling. Participants were provided with an ML1 headset that projected a hologram of a patient, bed and monitor. They were instructed via audio recording to gaze at variables in this scenario. The participant gaze targets from the ML1 output were compared with the specified gaze points from the audio recording. A priori investigators planned to iterative modifications of the eye-tracking software until a
@article{caruso_integrated_2021, author = {Caruso, Thomas J and Hess, Olivia and Roy, Kenny and Wang, Ellen and Rodriguez, Samuel and Palivathukal, Coby and Haber, Nick}, citation = {BMJ simulation \& technology enhanced learning 7 (5), 431, 2021}, journal = {BMJ simulation \& technology enhanced learning}, number = {5}, pages = {431}, year = {2021}, title = {Integrated eye tracking on Magic Leap One during augmented reality medical simulation: a technical report}, volume = {7}, _google_scholar_id = {r0BpntZqJG4C}, } - Using crowdsourcing to train facial emotion machine learning models with ambiguous labelsPeter Washington, Onur Cezmi Mutlu, Emilie Leblanc, and 8 more authorsarXiv preprint arXiv:2101.03477, 2021
@article{washington_using_2021, author = {Washington, Peter and Mutlu, Onur Cezmi and Leblanc, Emilie and Kline, Aaron and Hou, Cathy and Chrisman, Brianna and Stockham, Nate and Paskov, Kelley and Voss, Catalin and Haber, Nick and Wall, Dennis P}, citation = {arXiv preprint arXiv:2101.03477, 2021}, journal = {arXiv preprint arXiv:2101.03477}, year = {2021}, title = {Using crowdsourcing to train facial emotion machine learning models with ambiguous labels}, _google_scholar_id = {R3hNpaxXUhUC}, }
2020
- Threedworld: A platform for interactive multi-modal physical simulationChuang Gan, Jeremy Schwartz, Seth Alter, and 21 more authorsarXiv preprint arXiv:2007.04954, 2020
We introduce ThreeDWorld (TDW), a platform for interactive multi-modal physical simulation. TDW enables simulation of high-fidelity sensory data and physical interactions between mobile agents and objects in rich 3D environments. Unique properties include: real-time near-photo-realistic image rendering; a library of objects and environments, and routines for their customization; generative procedures for efficiently building classes of new environments; high-fidelity audio rendering; realistic physical interactions for a variety of material types, including cloths, liquid, and deformable objects; customizable agents that embody AI agents; and support for human interactions with VR devices. TDW’s API enables multiple agents to interact within a simulation and returns a range of sensor and physics data representing the state of the world. We present initial experiments enabled by TDW in emerging research directions in computer vision, machine learning, and cognitive science, including multi-modal physical scene understanding, physical dynamics predictions, multi-agent interactions, models that learn like a child, and attention studies in humans and neural networks.
@article{gan_threedworld:_2020, author = {Gan, Chuang and Schwartz, Jeremy and Alter, Seth and Mrowca, Damian and Schrimpf, Martin and Traer, James and Freitas, Julian De and Kubilius, Jonas and Bhandwaldar, Abhishek and Haber, Nick and Sano, Megumi and Kim, Kuno and Wang, Elias and Lingelbach, Michael and Curtis, Aidan and Feigelis, Kevin and Bear, Daniel M and Gutfreund, Dan and Cox, David and Torralba, Antonio and DiCarlo, James J and Tenenbaum, Joshua B and McDermott, Josh H and Yamins, Daniel LK}, citation = {arXiv preprint arXiv:2007.04954, 2020}, journal = {arXiv preprint arXiv:2007.04954}, year = {2020}, title = {Threedworld: A platform for interactive multi-modal physical simulation}, _google_scholar_id = {IWHjjKOFINEC}, } - Data-driven diagnostics and the potential of mobile artificial intelligence for digital therapeutic phenotyping in computational psychiatryPeter Washington, Natalie Park, Parishkrita Srivastava, and 12 more authors2020
Data science and digital technologies have the potential to transform diagnostic classification. Digital technologies enable the collection of big data, and advances in machine learning and artificial intelligence enable scalable, rapid, and automated classification of medical conditions. In this review, we summarize and categorize various data-driven methods for diagnostic classification. In particular, we focus on autism as an example of a challenging disorder due to its highly heterogeneous nature. We begin by describing the frontier of data science methods for the neuropsychiatry of autism. We discuss early signs of autism as defined by existing pen-and-paperbased diagnostic instruments and describe data-driven feature selection techniques for determining the behaviors that are most salient for distinguishing children with autism from neurologically typical children. We then describe data-driven detection
@article{washington_data-driven_2020, author = {Washington, Peter and Park, Natalie and Srivastava, Parishkrita and Voss, Catalin and Kline, Aaron and Varma, Maya and Tariq, Qandeel and Kalantarian, Haik and Schwartz, Jessey and Patnaik, Ritik and Chrisman, Brianna and Stockham, Nathaniel and Paskov, Kelley and Haber, Nick and Wall, Dennis P}, citation = {Biological Psychiatry: Cognitive Neuroscience and Neuroimaging 5 (8), 759-769, 2020}, number = {8}, pages = {759-769}, year = {2020}, publisher = {Elsevier}, title = {Data-driven diagnostics and the potential of mobile artificial intelligence for digital therapeutic phenotyping in computational psychiatry}, volume = {5}, _google_scholar_id = {Se3iqnhoufwC}, } - Active world model learning with progress curiosityKuno Kim, Megumi Sano, Julian De Freitas, and 2 more authors2020
World models are self-supervised predictive models of how the world evolves. Humans learn world models by curiously exploring their environment, in the process acquiring compact abstractions of high bandwidth sensory inputs, the ability to plan across long temporal horizons, and an understanding of the behavioral patterns of other agents. In this work, we study how to design such a curiosity-driven Active World Model Learning (AWML) system. To do so, we construct a curious agent building world models while visually exploring a 3D physical environment rich with distillations of representative real-world agents. We propose an AWML system driven by -Progress: a scalable and effective learning progress-based curiosity signal and show that -Progress naturally gives rise to an exploration policy that directs attention to complex but learnable dynamics in a balanced manner, as a result overcoming the white noise problem. As a result, our -Progress-driven controller achieves significantly higher AWML performance than baseline controllers equipped with state-of-the-art exploration strategies such as Random Network Distillation and Model Disagreement.
@article{kim_active_2020, author = {Kim, Kuno and Sano, Megumi and Freitas, Julian De and Haber, Nick and Yamins, Daniel}, citation = {International conference on machine learning, 5306-5315, 2020}, conference = {International conference on machine learning}, pages = {5306-5315}, year = {2020}, publisher = {PMLR}, title = {Active world model learning with progress curiosity}, _google_scholar_id = {QIV2ME_5wuYC}, } - Toward continuous social phenotyping: analyzing gaze patterns in an emotion recognition task for children with autism through wearable smart glassesAnish Nag, Nick Haber, Catalin Voss, and 9 more authorsJournal of medical Internet research, 2020
Several studies have shown that facial attention differs in children with autism. Measuring eye gaze and emotion recognition in children with autism is challenging, as standard clinical assessments must be delivered in clinical settings by a trained clinician. Wearable technologies may be able to bring eye gaze and emotion recognition into natural social interactions and settings.This study aimed to test: (1) the feasibility of tracking gaze using wearable smart glasses during a facial expression recognition task and (2) the ability of these gaze-tracking data, together with facial expression recognition responses, to distinguish children with autism from neurotypical controls (NCs).We compared the eye gaze and emotion recognition patterns of 16 children with autism spectrum disorder (ASD) and 17 children without ASD via wearable smart glasses fitted with a custom eye tracker. Children identified static facial expressions of images presented on a computer screen along with nonsocial distractors while wearing Google Glass and the eye tracker. Faces were presented in three trials, during one of which children received feedback in the form of the correct classification. We employed hybrid human-labeling and computer visionenabled methods for pupil tracking and worldgaze translation calibration. We analyzed the impact of gaze and emotion recognition features in a prediction task aiming to distinguish children with ASD from NC participants.Gaze and emotion recognition patterns enabled the training of a classifier that distinguished
@article{nag_toward_2020, author = {Nag, Anish and Haber, Nick and Voss, Catalin and Tamura, Serena and Daniels, Jena and Ma, Jeffrey and Chiang, Bryan and Ramachandran, Shasta and Schwartz, Jessey and Winograd, Terry and Feinstein, Carl and Wall, Dennis P}, citation = {Journal of medical Internet research 22 (4), e13810, 2020}, journal = {Journal of medical Internet research}, number = {4}, pages = {e13810}, year = {2020}, publisher = {JMIR Publications}, title = {Toward continuous social phenotyping: analyzing gaze patterns in an emotion recognition task for children with autism through wearable smart glasses}, volume = {22}, _google_scholar_id = {zYLM7Y9cAGgC}, } - Precision telemedicine through crowdsourced machine learning: testing variability of crowd workers for video-based autism feature recognitionPeter Washington, Emilie Leblanc, Kaitlyn Dunlap, and 10 more authorsJournal of personalized medicine, 2020
Mobilized telemedicine is becoming a key, and even necessary, facet of both precision health and precision medicine. In this study, we evaluate the capability and potential of a crowd of virtual workersdefined as vetted members of popular crowdsourcing platformsto aid in the task of diagnosing autism. We evaluate workers when crowdsourcing the task of providing categorical ordinal behavioral ratings to unstructured public YouTube videos of children with autism and neurotypical controls. To evaluate emerging patterns that are consistent across independent crowds, we target workers from distinct geographic loci on two crowdsourcing platforms: an international group of workers on Amazon Mechanical Turk (MTurk) (N = 15) and Microworkers from Bangladesh (N = 56), Kenya (N = 23), and the Philippines (N = 25). We feed worker responses as input to a validated diagnostic machine learning classifier trained on clinician-filled electronic health records. We find that regardless of crowd platform or targeted country, workers vary in the average confidence of the correct diagnosis predicted by the classifier. The best worker responses produce a mean probability of the correct class above 80% and over one standard deviation above 50%, accuracy and variability on par with experts according to prior studies. There is a weak correlation between mean time spent on task and mean performance (r = 0.358, p = 0.005). These results demonstrate that while the crowd can produce accurate diagnoses, there are intrinsic differences in crowdworker ability to rate behavioral features. We propose a novel strategy for recruitment of crowdsourced workers to
@article{washington_precision_2020, author = {Washington, Peter and Leblanc, Emilie and Dunlap, Kaitlyn and Penev, Yordan and Kline, Aaron and Paskov, Kelley and Sun, Min Woo and Chrisman, Brianna and Stockham, Nathaniel and Varma, Maya and Voss, Catalin and Haber, Nick and Wall, Dennis P}, citation = {Journal of personalized medicine 10 (3), 86, 2020}, journal = {Journal of personalized medicine}, number = {3}, pages = {86}, year = {2020}, publisher = {MDPI}, title = {Precision telemedicine through crowdsourced machine learning: testing variability of crowd workers for video-based autism feature recognition}, volume = {10}, _google_scholar_id = {hC7cP41nSMkC}, } - Feature selection and dimension reduction of social autism dataPeter Washington, Kelley Marie Paskov, Haik Kalantarian, and 11 more authorsPacific Symposium on Biocomputing. Pacific Symposium on Biocomputing, 2020
Autism Spectrum Disorder (ASD) is a complex neuropsychiatric condition with a highly heterogeneous phenotype. Following the work of Duda et al., which uses a reduced feature set from the Social Responsiveness Scale, Second Edition (SRS) to distinguish ASD from ADHD, we performed item-level question selection on answers to the SRS to determine whether ASD can be distinguished from non-ASD using a similarly small subset of questions. To explore feature redundancies between the SRS questions, we performed filter, wrapper, and embedded feature selection analyses. To explore the linearity of the SRS-related ASD phenotype, we then compressed the 65-question SRS into low-dimension representations using PCA, t-SNE, and a denoising autoencoder. We measured the performance of a multi-layer perceptron (MLP) classifier with the top-ranking questions as input. Classification using only the top
@article{washington_feature_2020, author = {Washington, Peter and Paskov, Kelley Marie and Kalantarian, Haik and Stockham, Nathaniel and Voss, Catalin and Kline, Aaron and Patnaik, Ritik and Chrisman, Brianna and Varma, Maya and Tariq, Qandeel and Dunlap, Kaitlyn and Schwartz, Jessey and Haber, Nick and Wall, Dennis P}, citation = {Pacific Symposium on Biocomputing. Pacific Symposium on Biocomputing 25, 707, 2020}, journal = {Pacific Symposium on Biocomputing. Pacific Symposium on Biocomputing}, pages = {707}, year = {2020}, title = {Feature selection and dimension reduction of social autism data}, volume = {25}, _google_scholar_id = {9ZlFYXVOiuMC}, } - Making emotions transparent: Google Glass helps autistic kids understand facial expressions through augmented-reaiity therapyNick Haber, Catalin Voss, and Dennis WallIEEE Spectrum, 2020
Now imagine that Jimmy is wearing a special kind of Google Glass, the augmented-reality headset that Google introduced in 2013. When he looks up at his mom, the head-up display lights up with a green box, which alerts Jimmy that he’s "found a face." As he focuses on her face, an emoji pops up, which tells Jimmy, "You found an angry face." He thinks about why his mom might be annoyed. Maybe he should stop what he’s doing with the silverware and ask her.
@article{haber_making_2020, author = {Haber, Nick and Voss, Catalin and Wall, Dennis}, citation = {IEEE Spectrum 57 (4), 46-52, 2020}, journal = {IEEE Spectrum}, number = {4}, pages = {46-52}, year = {2020}, publisher = {IEEE}, title = {Making emotions transparent: Google Glass helps autistic kids understand facial expressions through augmented-reaiity therapy}, volume = {57}, _google_scholar_id = {M3ejUd6NZC8C}, } - Training an emotion detection classifier using frames from a mobile therapeutic game for children with developmental disordersPeter Washington, Haik Kalantarian, Jack Kent, and 16 more authorsarXiv e-prints, 2020
Automated emotion classification could aid those who struggle to recognize emotion, including children with developmental behavioral conditions such as autism. However, most computer vision emotion models are trained on adult affect and therefore underperform on child faces. In this study, we designed a strategy to gamify the collection and the labeling of child affect data in an effort to boost the performance of automatic child emotion detection to a level closer to what will be needed for translational digital healthcare. We leveraged our therapeutic smartphone game, GuessWhat, which was designed in large part for children with developmental and behavioral conditions, to gamify the secure collection of video data of children expressing a variety of emotions prompted by the game. Through a secure web interface gamifying the human labeling effort, we gathered and labeled 2,155 videos, 39,968 emotion
@article{washington_training_2020, author = {Washington, Peter and Kalantarian, Haik and Kent, Jack and Husic, Arman and Kline, Aaron and Leblanc, Emilie and Hou, Cathy and Mutlu, Cezmi and Dunlap, Kaitlyn and Penev, Yordan and Varma, Maya and Stockham, Nate and Chrisman, Brianna and Paskov, Kelley and Sun, Min Woo and Jung, Jae-Yoon and Voss, Catalin and Haber, Nick and Wall, Dennis P}, citation = {arXiv e-prints, arXiv: 2012.08678, 2020}, journal = {arXiv e-prints}, pages = {arXiv: 2012.08678}, year = {2020}, title = {Training an emotion detection classifier using frames from a mobile therapeutic game for children with developmental disorders}, _google_scholar_id = {hFOr9nPyWt4C}, } - Designing a holistic at-home learning aid for autismCatalin Voss, Nick Haber, Peter Washington, and 8 more authorsarXiv preprint arXiv:2002.04263, 2020
In recent years, much focus has been put on employing technology to make novel behavioural aids for those with autism. Most of these are digital adaptations of tools used in standard behavioural therapy to enforce normative skills. These digital counterparts are often used outside of both the larger therapeutic context and the real world, in which the learned skills might apply. To address this, we are designing a system of automatic expression recognition on wearable devices that integrates directly into the families daily social interactions, to give children and their caregivers the tools and information they need to design their own holistic therapy. In order to develop a tool that will be truly useful to families, we proactively include children with autism and their families as co-designers in the development process. By providing an app and interface with interchangeable social feedback options, we aim to produce a framework for therapy that folds into their daily lives, tailored to their specific needs.
@article{voss_designing_2020, author = {Voss, Catalin and Haber, Nick and Washington, Peter and Kline, Aaron and McCarthy, Beth and Daniels, Jena and Fazel, Azar and De, Titas and Feinstein, Carl and Winograd, Terry and Wall, Dennis}, citation = {arXiv preprint arXiv:2002.04263, 2020}, journal = {arXiv preprint arXiv:2002.04263}, year = {2020}, title = {Designing a holistic at-home learning aid for autism}, _google_scholar_id = {KlAtU1dfN6UC}, } - A wearable social interaction aid for children with autismNick Haber, Catalin Voss, Jena Daniels, and 7 more authorsarXiv preprint arXiv:2004.14281, 2020
With most recent estimates giving an incidence rate of 1 in 68 children in the United States, the autism spectrum disorder (ASD) is a growing public health crisis. Many of these children struggle to make eye contact, recognize facial expressions, and engage in social interactions. Today the standard for treatment of the core autism-related deficits focuses on a form of behavior training known as Applied Behavioral Analysis. To address perceived deficits in expression recognition, ABA approaches routinely involve the use of prompts such as flash cards for repetitive emotion recognition training via memorization. These techniques must be administered by trained practitioners and often at clinical centers that are far outnumbered by and out of reach from the many children and families in need of attention. Waitlists for access are up to 18 months long, and this wait may lead to children regressing down a path of isolation that worsens their long-term prognosis. There is an urgent need to innovate new methods of care delivery that can appropriately empower caregivers of children at risk or with a diagnosis of autism, and that capitalize on mobile tools and wearable devices for use outside of clinical settings.
@article{haber_a_2020, author = {Haber, Nick and Voss, Catalin and Daniels, Jena and Washington, Peter and Fazel, Azar and Kline, Aaron and De, Titas and Winograd, Terry and Feinstein, Carl and Wall, Dennis P}, citation = {arXiv preprint arXiv:2004.14281, 2020}, journal = {arXiv preprint arXiv:2004.14281}, year = {2020}, title = {A wearable social interaction aid for children with autism}, _google_scholar_id = {YsMSGLbcyi4C}, } - Crowdsourced feature tagging for scalable and privacy-preserved autism diagnosisPeter Washington, Qandeel Tariq, Emilie Leblanc, and 14 more authorsmedRxiv, 2020
Standard medical diagnosis of mental health conditions often requires licensed experts who are increasingly outnumbered by those at risk, limiting reach. We test the hypothesis that a trustworthy crowd of non-experts can efficiently label features needed for accurate machine learning detection of the common childhood developmental disorder autism. We implement a novel process for creating a trustworthy distributed workforce for video feature extraction, selecting a workforce of 102 workers from a pool of 1,107. Two previously validated binary autism logistic regression classifiers were used to evaluate the quality of the curated crowds ratings on unstructured home videos. A clinically representative balanced sample (N=50 videos) of videos were evaluated with and without face box and pitch shift privacy alterations, with AUROC and AUPRC scores >0.98. With both privacy-preserving modifications, sensitivity is preserved (96.0%) while maintaining specificity (80.0%) and accuracy (88.0%) at levels that exceed classification methods without alterations. We find that machine learning classification from features extracted by a curated nonexpert crowd achieves clinical performance for pediatric autism videos and maintains acceptable performance when privacy-preserving mechanisms are applied. These results suggest that privacy-based crowdsourcing of short videos can be leveraged for rapid and mobile assessment of behavioral health.
@article{washington_crowdsourced_2020, author = {Washington, Peter and Tariq, Qandeel and Leblanc, Emilie and Chrisman, Brianna and Dunlap, Kaitlyn and Kline, Aaron and Kalantarian, Haik and Penev, Yordan and Paskov, Kelley and Voss, Catalin and Stockham, Nathaniel and Varma, Maya and Husic, Arman and Kent, Jack and Haber, Nick and Winograd, Terry and Wall, Dennis P}, citation = {medRxiv, 2020.12. 15.20248283, 2020}, journal = {medRxiv}, pages = {2020.12. 15.20248283}, year = {2020}, publisher = {Cold Spring Harbor Laboratory Press}, title = {Crowdsourced feature tagging for scalable and privacy-preserved autism diagnosis}, _google_scholar_id = {mB3voiENLucC}, } - Towards curiosity-driven learning of physical dynamicsMichael Lingelbach, Damian Mrowca, Nick Haber, and 2 more authorsICLR Bridging AI and Cognitive Science (BAICS) Workshop, 2020
Throughout our lives, we as humans acquire an intuitive understanding of our physical environments, a capacity that supports our imagination and planning abilities. Driven by our own curiosity, we learn about object motion and properties via self-curated targeted experiments, that teach us what we do not know. Recently, neural network models have been proposed that learn forward object dynamics from observations like humans. Unlike humans, these models do not actively interact with surrounding objects but learn from human-curated datasets as passive observers. In this work-in-progress, we propose a closed-loop system that teaches itself about forward object dynamics without any human intervention. Our model consists of two parts. A forward dynamics model that models the transition between states and a policy model that tries to predict the dynamics models error conditioned on object interactions as its intrinsic reward. We show that our method is able to train forward dynamics models that generalize to unseen physical scenarios and approaches the upper bound of models trained on human-curated data. The model generates complex behaviors with a preference to novel objects.
@article{lingelbach_towards_2020, author = {Lingelbach, Michael and Mrowca, Damian and Haber, Nick and Fei-Fei, Li and Yamins, DL}, citation = {ICLR Bridging AI and Cognitive Science (BAICS) Workshop, 2020}, journal = {ICLR Bridging AI and Cognitive Science (BAICS) Workshop}, year = {2020}, title = {Towards curiosity-driven learning of physical dynamics}, _google_scholar_id = {WF5omc3nYNoC}, } - superpower glass: An augmented reality intervention for improving social deficits in children with autism spectrum disorderAaron Kline, Michael Ning, Arman Husic, and 7 more authorsINSAR 2020 Virtual Meeting, 2020
@article{kline_superpower_2020, author = {Kline, Aaron and Ning, Michael and Husic, Arman and Washington, Peter and Voss, Catalin and Dunlap, Kaitlyn L and Penev, Yordan and Leblanc, Emilie and Haber, Nick and Wall, Dennis}, citation = {INSAR 2020 Virtual Meeting, 2020}, journal = {INSAR 2020 Virtual Meeting}, year = {2020}, publisher = {INSAR}, title = {superpower glass: An augmented reality intervention for improving social deficits in children with autism spectrum disorder}, _google_scholar_id = {HDshCWvjkbEC}, } - Learning in social environments with curious neural agentsMegumi Sano, Julian De Freitas, Nick Haber, and 1 more authorProceedings of the Annual Meeting of the Cognitive Science Society, 2020
From an early age, humans are capable of learning about theirsocial environment, making predictions of how other agentswill operate and decisions about how they themselves will in-teract. In this work, we address the problem of formalizing thelearning principles underlying these abilities. We construct a cu-rious neural agent that can efficiently learn predictive models ofsocial environments that are rich with external agents inspiredby real-world animate behaviors such as peekaboo, chasing,and mimicry. Our curious neural agent consists of a controllerdriven by -Progress, a scalable and effective curiosity signal,and a disentangled world model that allocates separate networksfor interdependent components of the world. We show that ourdisentangled curiosity-driven agent achieves higher learning ef-ficiency and prediction performance than strong baselines. Cru-cially, we find that a preference for animate attention emergesnaturally in our model, and is a key driver of performance. Fi-nally we discuss future directions including applications of ourframework to modeling human behavior and designing earlyindicators for developmental variability.
@article{sano_learning_2020, author = {Sano, Megumi and Freitas, Julian De and Haber, Nick and Yamins, Daniel LK}, citation = {Proceedings of the Annual Meeting of the Cognitive Science Society 42, 2020}, journal = {Proceedings of the Annual Meeting of the Cognitive Science Society}, year = {2020}, title = {Learning in social environments with curious neural agents}, volume = {42}, _google_scholar_id = {ZeXyd9-uunAC}, } - Improved Digital Therapy for Developmental Pediatrics Using Domain-Specific Artificial IntelligencePeter Washington, Haik Kalantarian, Jack Kent, and 16 more authors2020
Automated emotion classification could aid those who struggle to recognize emotion, including children with developmental behavioral conditions such as autism. However, most computer vision emotion recognition models are trained on adult affect and therefore underperform when used on child faces. In this study, we designed a strategy to gamify the collection and the labeling of child affect data in an effort to boost the performance of automatic child emotion detection to a level closer to what will be needed for digital healthcare approaches. We leveraged our prototype therapeutic smartphone game, GuessWhat, which was designed in large part for children with developmental and behavioral conditions, to gamify the secure collection of video data of children expressing a variety of emotions prompted by the game. Independently, we created a secure web interface to gamify the human labeling effort HollywoodSquares, tailored for use by any qualified labeler. We gathered and labeled 2,155 videos, 39,968 emotion frames, and 106,001 labels on all images. With this drastically expanded pediatric emotion centric database (> 30x larger than existing public pediatric affect datasets), we trained a pediatric emotion classification convolutional neural network (CNN) classifier of happy, sad, surprised, fearful, angry, disgust, and neutral expressions in children. The classifier achieved 66.9% balanced accuracy and 67.4% F1-score on the entirety of CAFE as well as 79.1% balanced accuracy and 78.0% F1-score on CAFE Subset A, a subset containing at least 60% human agreement on emotions labels. This performance is at least 10% higher than all
@article{washington_improved_2020, author = {Washington, Peter and Kalantarian, Haik and Kent, Jack and Husic, Arman and Kline, Aaron and Leblanc, Emilie and Hou, Cathy and Mutlu, Cezmi and Dunlap, Kaitlyn and Penev, Yordan and Varma, Maya and Stockham, Nate and Chrisman, Brianna and Paskov, Kelley and Sun, Min Woo and Jung, Jae-Yoon and Voss, Catalin and Haber, Nick and Wall, Dennis Paul}, citation = {}, year = {2020}, title = {Improved Digital Therapy for Developmental Pediatrics Using Domain-Specific Artificial Intelligence}, _google_scholar_id = {TQgYirikUcIC}, } - TOWARDS MODELING THE DEVELOPMENTAL VARIABIL-ITY OF HUMAN ATTENTIONKuno Kim, Megumi Sano, Julian De Freitas, and 2 more authors2020
Children exhibit extraordinary exploratory behaviors hypothesized to contribute to the building of models of their world. Harnessing this capacity in artificial systems promises not only more flexible technology but also cognitive models of the developmental processes we seek to mimic. Yet not all children learn the same way, and for instance children with autism exhibit characteristically different exploratory strategies early in life. What if we could, by developing artificial systems that learn through exploration, model not only typically development, but all its variations? In this work, we present a preliminary analysis of curiosity-driven agents in social environments that establishes links between early behavior and later acuity, with implications for the future of both diagnostics and personalized learning.
@article{kim_towards_2020, author = {Kim, Kuno and Sano, Megumi and Freitas, Julian De and Yamins, Daniel LK and Haber, Nick}, citation = {}, year = {2020}, title = {TOWARDS MODELING THE DEVELOPMENTAL VARIABIL-ITY OF HUMAN ATTENTION}, _google_scholar_id = {YOwf2qJgpHMC}, }
2019
- Effect of wearable digital intervention for improving socialization in children with autism spectrum disorder: a randomized clinical trialCatalin Voss, Jessey Schwartz, Jena Daniels, and 10 more authorsJAMA pediatrics, 2019
Autism behavioral therapy is effective but expensive and difficult to access. While mobile technologybased therapy can alleviate wait-lists and scale for increasing demand, few clinical trials exist to support its use for autism spectrum disorder (ASD) care.To evaluate the efficacy of Superpower Glass, an artificial intelligencedriven wearable behavioral intervention for improving social outcomes of children with ASD.A randomized clinical trial in which participants received the Superpower Glass intervention plus standard of care applied behavioral analysis therapy and control participants received only applied behavioral analysis therapy. Assessments were completed at the Stanford University Medical School, and enrolled participants used the Superpower Glass intervention in their homes. Children aged 6 to 12 years with a formal ASD diagnosis who were
@article{voss_effect_2019, author = {Voss, Catalin and Schwartz, Jessey and Daniels, Jena and Kline, Aaron and Haber, Nick and Washington, Peter and Tariq, Qandeel and Robinson, Thomas N and Desai, Manisha and Phillips, Jennifer M and Feinstein, Carl and Winograd, Terry and Wall, Dennis P}, citation = {JAMA pediatrics 173 (5), 446-454, 2019}, journal = {JAMA pediatrics}, number = {5}, pages = {446-454}, year = {2019}, publisher = {American Medical Association}, title = {Effect of wearable digital intervention for improving socialization in children with autism spectrum disorder: a randomized clinical trial}, volume = {173}, _google_scholar_id = {qjMakFHDy7sC}, } - Guess what? Towards understanding autism from structured video using facial affectHaik Kalantarian, Peter Washington, Jessey Schwartz, and 3 more authorsJournal of healthcare informatics research, 2019
Autism Spectrum Disorder (ASD) is a condition affecting an estimated 1 in 59 children in the United States. Due to delays in diagnosis and imbalances in coverage, it is necessary to develop new methods of care delivery that can appropriately empower children and caregivers by capitalizing on mobile tools and wearable devices for use outside of clinical settings. In this paper, we present a mobile charades-style game, Guess What?, used for the acquisition of structured video from children with ASD for behavioral disease research. We then apply face tracking and emotion recognition algorithms to videos acquired through Guess What? game play. By analyzing facial affect in response to various prompts, we demonstrate that engagement and facial affect can be quantified and measured using real-time image processing algorithms: an important first-step for future therapies, at-home screenings, and
@article{kalantarian_guess_2019, author = {Kalantarian, Haik and Washington, Peter and Schwartz, Jessey and Daniels, Jena and Haber, Nick and Wall, Dennis P}, citation = {Journal of healthcare informatics research 3, 43-66, 2019}, journal = {Journal of healthcare informatics research}, pages = {43-66}, year = {2019}, publisher = {Springer International Publishing}, title = {Guess what? Towards understanding autism from structured video using facial affect}, volume = {3}, _google_scholar_id = {5nxA0vEk-isC}, } - Superpower glassAaron Kline, Catalin Voss, Peter Washington, and 6 more authorsGetMobile: Mobile Computing and Communications, 2019
Autism Spectrum Disorder (ASD) is quickly becoming a global health crisis estimated to affect one in 40 children in the United States [8]. ASD consists of social deficiencies, such as impaired communication, eye contact, facial expression recognition, and social interaction. The current standard of care, applied behavioral analysis (ABA), relies on teaching these skills primarily in clinical environments with tools such as static flashcards. Such tools are largely removed from real world emotional contexts. While ABA can lead to improvements [1, 2], the therapy often generalizes poorly to situations that extend beyond the routines practiced in clinical contexts [3]. Furthermore, access to such treatment is constrained by the availability of therapists, who struggle to keep up with the increasing demand for care.
@article{kline_superpower_2019, author = {Kline, Aaron and Voss, Catalin and Washington, Peter and Haber, Nick and Schwartz, Hessey and Tariq, Qandeel and Winograd, Terry and Feinstein, Carl and Wall, Dennis P}, citation = {GetMobile: Mobile Computing and Communications 23 (2), 35-38, 2019}, journal = {GetMobile: Mobile Computing and Communications}, number = {2}, pages = {35-38}, year = {2019}, publisher = {ACM}, title = {Superpower glass}, volume = {23}, _google_scholar_id = {dhFuZR0502QC}, } - Validity of online screening for autism: crowdsourcing study comparing paid and unpaid diagnostic tasksPeter Washington, Haik Kalantarian, Qandeel Tariq, and 11 more authorsJournal of medical Internet research, 2019
Obtaining a diagnosis of neuropsychiatric disorders such as autism requires long waiting times that can exceed a year and can be prohibitively expensive. Crowdsourcing approaches may provide a scalable alternative that can accelerate general access to care and permit underserved populations to obtain an accurate diagnosis.We aimed to perform a series of studies to explore whether paid crowd workers on Amazon Mechanical Turk (AMT) and citizen crowd workers on a public website shared on social media can provide accurate online detection of autism, conducted via crowdsourced ratings of short home video clips.Three online studies were performed: (1) a paid crowdsourcing task on AMT (N=54) where crowd workers were asked to classify 10 short video clips of children as Autism or Not autism, (2) a more complex paid crowdsourcing task (N=27) with only those raters who correctly rated 8 of the 10 videos during the first study, and (3) a public unpaid study (N=115) identical to the first study.For Study 1, the mean score of the participants who completed all questions was 7.50/10 (SD 1.46). When only analyzing the workers who scored 8/10 (n=27/54), there was a weak negative correlation between the time spent rating the videos and the sensitivity (=0.44, P=.02). For Study 2, the mean score of the participants rating new videos was 6.76/10 (SD 0.59). The average deviation between the crowdsourced answers and gold standard ratings provided by two expert clinical research coordinators was 0.56, with an SD
@article{washington_validity_2019, author = {Washington, Peter and Kalantarian, Haik and Tariq, Qandeel and Schwartz, Jessey and Dunlap, Kaitlyn and Chrisman, Brianna and Varma, Maya and Ning, Michael and Kline, Aaron and Stockham, Nathaniel and Paskov, Kelley and Voss, Catalin and Haber, Nick and Wall, Dennis Paul}, citation = {Journal of medical Internet research 21 (5), e13668, 2019}, journal = {Journal of medical Internet research}, number = {5}, pages = {e13668}, year = {2019}, publisher = {JMIR Publications}, title = {Validity of online screening for autism: crowdsourcing study comparing paid and unpaid diagnostic tasks}, volume = {21}, _google_scholar_id = {_kc_bZDykSQC}, } - The potential for machine learningbased wearables to improve socialization in teenagers and adults with autism spectrum disorderreplyCatalin Voss, Nick Haber, and Dennis P WallJAMA pediatrics, 2019
In Reply We agree with the commentary by Ahuja on our article. 1 Adolescents and adults need innovative digital solutions that can scale to meet what is an enormous and enormously diverse demand. Complex social interactions, such as dating, interviewing, managing conflict, and even just basic daily living skills, challenge this teenager-to-adult population in myriad ways. Naturally, this makes the focus on tools development significantly more challenging. What are the biggest unmet needs where the combination of artificial intelligence and augmented reality wearables (or similar tools) can make a positive effect? We can generate a list, and probably come up with a fairly exhaustive one, but how to prioritize that list is daunting.Our focus has been on the pediatric population namely because addressing developmental issues early can pay dividends downstream; children may progress to a point where they no
@article{voss_the_2019, author = {Voss, Catalin and Haber, Nick and Wall, Dennis P}, citation = {JAMA pediatrics 173 (11), 1106-1106, 2019}, journal = {JAMA pediatrics}, number = {11}, pages = {1106-1106}, year = {2019}, publisher = {American Medical Association}, title = {The potential for machine learningbased wearables to improve socialization in teenagers and adults with autism spectrum disorderreply}, volume = {173}, _google_scholar_id = {LkGwnXOMwfcC}, } - Positive Childhood Experiences and Adult Mental and Relational Health in a Statewide Sample Associations Across Adverse Childhood Experiences Levels In ReplyCatalin Voss, Nick Haber, and Dennis P WallJAMA PEDIATRICS, 2019
@article{voss_positive_2019, author = {Voss, Catalin and Haber, Nick and Wall, Dennis P}, citation = {JAMA PEDIATRICS 173 (11), 1106-1106, 2019}, journal = {JAMA PEDIATRICS}, number = {11}, pages = {1106-1106}, year = {2019}, publisher = {AMER MEDICAL ASSOC}, title = {Positive Childhood Experiences and Adult Mental and Relational Health in a Statewide Sample Associations Across Adverse Childhood Experiences Levels In Reply}, volume = {173}, _google_scholar_id = {MXK_kJrjxJIC}, } - Intrinsic curiosity may give rise to animate attentionJulian De Freitas, Kun Ho Kim, Nick Haber, and 3 more authorsJournal of Vision, 2019
Humans naturally pay attention to other animate agents in their environment, a prosocial behavior that has been documented as early as a few weeks. What internal mechanisms give rise to this behavior? A standard hypothesis is that the human brain has a built-in module that specifically detects animacy from visual input. We hypothesize that animate attention naturally arises from a more general process of curiosity driven learning. We ran experiments to measure important features of animate attention. Observers (N= 12, A gerange= 43 years) wore a mobile eye tracker while watching a display consisting of four self-propelled, spherical robots travelling along a mat. Using kinematics alone, the robots were made to appear animate, random, periodic, or static. Average fixations proportions were animate 55%, random 24%, periodic 14%, and static 7%, with 10 of 12 observers fixating most on animate. We also
@article{freitas_intrinsic_2019, author = {Freitas, Julian De and Kim, Kun Ho and Haber, Nick and Conwell, Colin and Alvarez, George A and Yamins, Daniel LK}, citation = {Journal of Vision 19 (10), 17d-17d, 2019}, journal = {Journal of Vision}, number = {10}, pages = {17d-17d}, year = {2019}, publisher = {The Association for Research in Vision and Ophthalmology}, title = {Intrinsic curiosity may give rise to animate attention}, volume = {19}, _google_scholar_id = {UebtZRa9Y70C}, } - Addendum to the Acknowledgements: Validity of Online Screening for Autism: Crowdsourcing Study Comparing Paid and Unpaid Diagnostic TasksPeter Washington, Haik Kalantarian, Qandeel Tariq, and 11 more authorsJournal of Medical Internet Research, 2019
The correction will appear in the online version of the paper on the JMIR website on June 27, 2019, together with the publication of this correction notice. Because this was made after submission to PubMed, PubMed Central, and other full-text repositories, the corrected article also has been resubmitted to those repositories.
@article{washington_addendum_2019, author = {Washington, Peter and Kalantarian, Haik and Tariq, Qandeel and Schwartz, Jessey and Dunlap, Kaitlyn and Chrisman, Brianna and Varma, Maya and Ning, Michael and Kline, Aaron and Stockham, Nathaniel and Paskov, Kelley and Voss, Catalin and Haber, Nick and Wall, Dennis Paul}, citation = {Journal of Medical Internet Research 21 (6), e14950, 2019}, journal = {Journal of Medical Internet Research}, number = {6}, pages = {e14950}, year = {2019}, publisher = {JMIR Publications}, title = {Addendum to the Acknowledgements: Validity of Online Screening for Autism: Crowdsourcing Study Comparing Paid and Unpaid Diagnostic Tasks}, volume = {21}, _google_scholar_id = {IjCSPb-OGe4C}, } - Autism Insight Score (AIS): A Crowdsourced Study of Paid and Unpaid Citizen HealthcarePeter Washington, Haik Kalantarian, Qandeel Tariq, and 11 more authors2019
Background: Obtaining a diagnosis of neuropsychiatric disorders such as autism requires long waiting times often exceeding a year. Furthermore, obtaining a professional diagnosis is often prohibitively expensive for much of the global population. Crowdsourcing provides a cheap and possibly free alternative that can allow underserved populations to obtain an accurate diagnosis.Objective: To determine whether paid crowd workers on Amazon Mechanical Turk (AMT) as well as citizen crowd workers on a public website shared on social media can provide accurate online screening for autism from watching a short video clip.Methods: Three online studies:(1) a paid crowdsourcing task on AMT (N= 55) where crowd workers were asked to rate ten short video clips of children as having autism or no autism,(2) a follow-up, more complex paid crowdsourcing task on AMT (N= 27) with only those raters who correctly rated 8 or more of the 10 videos during the first study, and (3) a public unpaid citizen healthcare study (N= 71) with the same task as in the first study. 50% of the videos in all studies contained children with autism, 50% of videos contained females, and the age range of children in the videos ranged from 2 to 5 years.Results: Study 1: The mean score of the participants who completed all questions was 7.50 (SD= 1.46) out of 10. When only analyzing the workers who scored reasonably well (scored 8 out of 10 or higher; n= 27 out of 55), there was a weak negative correlation between the time spent rating the videos and the sensitivity (r=-0.44, p= 0.02). Study 2: The mean score of the participants rating new videos was 6.76 (SD= 0.59) out
@article{washington_autism_2019, author = {Washington, Peter and Kalantarian, Haik and Tariq, Qandeel and Schwartz, Jessey and Dunlap, Kaitlyn and Chrisman, Brianna and Varma, Maya and Ning, Michael and Kline, Aaron and Stockham, Nathaniel and Paskov, Kelley and Voss, Catalin and Haber, Nick and Wall, Dennis Paul}, citation = {}, title = {Autism Insight Score (AIS): A Crowdsourced Study of Paid and Unpaid Citizen Healthcare}, _google_scholar_id = {4JMBOYKVnBMC}, year = {2019} }
2018
- Flexible neural representation for physics predictionDamian Mrowca, Chengxu Zhuang, Elias Wang, and 4 more authorsAdvances in neural information processing systems, 2018
Humans have a remarkable capacity to understand the physical dynamics of objects in their environment, flexibly capturing complex structures and interactions at multiple levels of detail.
@article{mrowca_flexible_2018, author = {Mrowca, Damian and Zhuang, Chengxu and Wang, Elias and Haber, Nick and Fei-Fei, Li F and Tenenbaum, Josh and Yamins, Daniel L}, citation = {Advances in neural information processing systems 31, 2018}, journal = {Advances in neural information processing systems}, year = {2018}, title = {Flexible neural representation for physics prediction}, volume = {31}, _google_scholar_id = {ULOm3_A8WrAC}, } - Learning to play with intrinsically-motivated, self-aware agentsNick Haber, Damian Mrowca, Stephanie Wang, and 2 more authorsAdvances in neural information processing systems, 2018
Infants are experts at playing, with an amazing ability to generate novel structured behaviors in unstructured environments that lack clear extrinsic reward signals. We seek to mathematically formalize these abilities using a neural network that implements curiosity-driven intrinsic motivation. Using a simple but ecologically naturalistic simulated environment in which an agent can move and interact with objects it sees, we propose a" world-model" network that learns to predict the dynamic consequences of the agent’s actions. Simultaneously, we train a separate explicit" self-model" that allows the agent to track the error map of its world-model. It then uses the self-model to adversarially challenge the developing world-model. We demonstrate that this policy causes the agent to explore novel and informative interactions with its environment, leading to the generation of a spectrum of complex behaviors, including ego-motion prediction, object attention, and object gathering. Moreover, the world-model that the agent learns supports improved performance on object dynamics prediction, detection, localization and recognition tasks. Taken together, our results are initial steps toward creating flexible autonomous agents that self-supervise in realistic physical environments.
@article{haber_learning_2018, author = {Haber, Nick and Mrowca, Damian and Wang, Stephanie and Fei-Fei, Li F and Yamins, Daniel L}, citation = {Advances in neural information processing systems 31, 2018}, journal = {Advances in neural information processing systems}, year = {2018}, title = {Learning to play with intrinsically-motivated, self-aware agents}, volume = {31}, _google_scholar_id = {W7OEmFMy1HYC}, } - Exploratory study examining the at-home feasibility of a wearable tool for social-affective learning in children with autismJena Daniels, Jessey N Schwartz, Catalin Voss, and 7 more authorsNPJ digital medicine, 2018
Although standard behavioral interventions for autism spectrum disorder (ASD) are effective therapies for social deficits, they face criticism for being time-intensive and overdependent on specialists. Earlier starting age of therapy is a strong predictor of later success, but waitlists for therapies can be 18 months long. To address these complications, we developed Superpower Glass, a machine-learning-assisted software system that runs on Google Glass and an Android smartphone, designed for use during social interactions. This pilot exploratory study examines our prototype tools potential for social-affective learning for children with autism. We sent our tool home with 14 families and assessed changes from intake to conclusion through the Social Responsiveness Scale (SRS-2), a facial affect recognition task (EGG), and qualitative parent reports. A repeated-measures one-way ANOVA demonstrated a decrease in
@article{daniels_exploratory_2018, author = {Daniels, Jena and Schwartz, Jessey N and Voss, Catalin and Haber, Nick and Fazel, Azar and Kline, Aaron and Washington, Peter and Feinstein, Carl and Winograd, Terry and Wall, Dennis P}, citation = {NPJ digital medicine 1 (1), 32, 2018}, journal = {NPJ digital medicine}, number = {1}, pages = {32}, year = {2018}, publisher = {Nature Publishing Group UK}, title = {Exploratory study examining the at-home feasibility of a wearable tool for social-affective learning in children with autism}, volume = {1}, _google_scholar_id = {hqOjcs7Dif8C}, } - Feasibility testing of a wearable behavioral aid for social learning in children with autismJena Daniels, Nick Haber, Catalin Voss, and 9 more authorsApplied clinical informatics, 2018
Background Recent advances in computer vision and wearable technology have created an opportunity to introduce mobile therapy systems for autism spectrum disorders (ASD) that can respond to the increasing demand for therapeutic interventions; however, feasibility questions must be answered first. Objective We studied the feasibility of a prototype therapeutic tool for children with ASD using Google Glass, examining whether children with ASD would wear such a device, if providing the emotion classification will improve emotion recognition, and how emotion recognition differs between ASD participants and neurotypical controls (NC). Methods We ran a controlled laboratory experiment with 43 children: 23 with ASD and 20 NC. Children identified static facial images on a computer screen with one of 7 emotions in 3 successive batches: the first with no information about emotion provided to the child, the
@article{daniels_feasibility_2018, author = {Daniels, Jena and Haber, Nick and Voss, Catalin and Schwartz, Jessey and Tamura, Serena and Fazel, Azar and Kline, Aaron and Washington, Peter and Phillips, Jennifer and Winograd, Terry and Feinstein, Carl and Wall, Dennis P}, citation = {Applied clinical informatics 9 (01), 129-140, 2018}, journal = {Applied clinical informatics}, number = {01}, pages = {129-140}, year = {2018}, publisher = {Schattauer GmbH}, title = {Feasibility testing of a wearable behavioral aid for social learning in children with autism}, volume = {9}, _google_scholar_id = {u5HHmVD_uO8C}, } - A gamified mobile system for crowdsourcing video for autism researchHaik Kalantarian, Peter Washington, Jessey Schwartz, and 3 more authors2018
Autism Spectrum Disorder (ASD) is a condition affecting 70 million children worldwide. Due to delays in diagnosis and imbalances in coverage, it is necessary to develop new methods of care delivery for use outside of clinical settings., We present a mobile charades-style game, Guess What?, used for acquisition of structured video from children with ASD for behavioral disease research. By analyzing facial affect in response to various prompts, we demonstrate that engagement and facial affect can be quantified and measured using real-time image processing algorithms: an important step for future therapies, diagnostics, and outcome measures based on home video. Our study of thirteen subjects reveals that game sessions displaying "faces" to the player produced the most emotive facial expressions in the player by a considerable margin. Videos from the younger neurotypical group contained 73.9% more frames
@article{kalantarian_a_2018, author = {Kalantarian, Haik and Washington, Peter and Schwartz, Jessey and Daniels, Jena and Haber, Nick and Wall, Dennis}, citation = {2018 IEEE international conference on healthcare informatics (ICHI), 350-352, 2018}, conference = {2018 IEEE international conference on healthcare informatics (ICHI)}, pages = {350-352}, year = {2018}, publisher = {IEEE}, title = {A gamified mobile system for crowdsourcing video for autism research}, _google_scholar_id = {d1gkVwhDpl0C}, } - Emergence of structured behaviors from curiosity-based intrinsic motivationNick Haber, Damian Mrowca, Li Fei-Fei, and 1 more authorarXiv preprint arXiv:1802.07461, 2018
Infants are experts at playing, with an amazing ability to generate novel structured behaviors in unstructured environments that lack clear extrinsic reward signals. We seek to replicate some of these abilities with a neural network that implements curiosity-driven intrinsic motivation. Using a simple but ecologically naturalistic simulated environment in which the agent can move and interact with objects it sees, the agent learns a world model predicting the dynamic consequences of its actions. Simultaneously, the agent learns to take actions that adversarially challenge the developing world model, pushing the agent to explore novel and informative interactions with its environment. We demonstrate that this policy leads to the self-supervised emergence of a spectrum of complex behaviors, including ego motion prediction, object attention, and object gathering. Moreover, the world model that the agent learns supports improved performance on object dynamics prediction and localization tasks. Our results are a proof-of-principle that computational models of intrinsic motivation might account for key features of developmental visuomotor learning in infants.
@article{haber_emergence_2018, author = {Haber, Nick and Mrowca, Damian and Fei-Fei, Li and Yamins, Daniel LK}, citation = {arXiv preprint arXiv:1802.07461, 2018}, journal = {arXiv preprint arXiv:1802.07461}, year = {2018}, title = {Emergence of structured behaviors from curiosity-based intrinsic motivation}, _google_scholar_id = {aqlVkmm33-oC}, } - Modeling the scientist in the cribNick Haber, Damian Mrowca, Li Fei-Fei, and 1 more authorJournal of Vision, 2018
Deep convolutional neural networks, when trained on difficult supervised tasks such as object classification on large datasets, have shown the remarkable property of learning general representations useful for many other tasks and predictive of responses in the ventral visual stream. These successes have led to the pursuit of tasks that are able to generate a visual backbone in developmentally-realistic ways. Yet large gaps remain to be filled.-Lack of supervision Training must happen without large amounts of manually-labeled data.-Interaction and agency The developmental behavioral literature intertwines such developments with interaction with the world, explaining rich behaviors through this training. We built a simulated environment based on a game engine that provides 3D visual stimuli and allows for agent interactions with different types of objects. In this, we have an artificial agent gather experience, and
@article{haber_modeling_2018, author = {Haber, Nick and Mrowca, Damian and Fei-Fei, Li and Yamins, Daniel}, citation = {Journal of Vision 18 (10), 10-10, 2018}, journal = {Journal of Vision}, number = {10}, pages = {10-10}, year = {2018}, publisher = {The Association for Research in Vision and Ophthalmology}, title = {Modeling the scientist in the crib}, volume = {18}, _google_scholar_id = {0EnyYjriUFMC}, } - To View More...Yelda Semizer, Melchi Michel, Karla Evans, and 11 more authorsJournal of Vision, 2018
Volume 18 Issue 10 | JOV | ARVO Journals jov Issues Topics For Authors About Editorial Board Journals Home MANAGE ALERTS Access Provided By: Google Indexer Forgot password? Create an Account To View More... Purchase this article with an account. or Subscribe Now AdvancedSearch Issues Topics For Authors About Editorial Board Current Issue Past Issues Special Issues/Collections Select a back issue 2024 2023 2022 2021 2020 2019 2018 2017 2016 2015 2014 2013 2012 2011 2010 2009 2008 2007 2006 2005 2004 2003 2002 2001 Select a back issue June 2024 Volume 24 Issue 6 Articles 1 - 18 May 2024 Volume 24 Issue 5 Articles 1 - 17 April 2024 Volume 24 Issue 4 Articles 1 - 24 March 2024 Volume 24 Issue 3 Articles 1 - 9 February 2024 Volume 24 Issue 2 Articles 1 - 14 January 2024 Volume 24 Issue 1 Articles 1 - 11 December 2023 Volume 23 Issue 14 Articles 1 - 7 December 2023
@article{semizer_to_2018, author = {Semizer, Yelda and Michel, Melchi and Evans, Karla and Wolfe, Jeremy and Penacchio, Olivier and Harris, Julie and Jonnalagadda, Aditya and Deza, Arturo and Eckstein, Miguel and Cox, Patrick and Kravitz, Dwight and Mitroff, Stephen and Hayes, Taylor and Henderson, John}, citation = {Journal of Vision 18, 1, 2018}, journal = {Journal of Vision}, pages = {1}, year = {2018}, title = {To View More...}, volume = {18}, _google_scholar_id = {8k81kl-MbHgC}, }
2017
- SuperpowerGlass: a wearable aid for the at-home therapy of children with autismPeter Washington, Catalin Voss, Aaron Kline, and 7 more authorsProceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies, 2017
We have developed a system for automatic facial expression recognition running on Google Glass, delivering real-time social cues to children with Autism Spectrum Disorder (ASD). The system includes multiple mechanisms to engage children and their parents, who administer this technology within the home. We completed an at-home design trial with 14 families that used the learning aid over a 3-month period. We found that children with ASD generally respond well to wearing the system at home and opt for the most expressive feedback choice. We further evaluated app usage, facial engagement, and model accuracy. We found that the device can act as a powerful training aid when used periodically in the home, that interactive video content from wearable therapy sessions should be augmented with sufficient context about the content to produce long-term engagement, and that the design of wearable systems
@article{washington_superpowerglass:_2017, author = {Washington, Peter and Voss, Catalin and Kline, Aaron and Haber, Nick and Daniels, Jena and Fazel, Azar and De, Titas and Feinstein, Carl and Winograd, Terry and Wall, Dennis}, citation = {Proceedings of the ACM on interactive, mobile, wearable and ubiquitous , 2017}, journal = {Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies}, number = {3}, pages = {1-22}, year = {2017}, publisher = {ACM}, title = {SuperpowerGlass: a wearable aid for the at-home therapy of children with autism}, volume = {1}, _google_scholar_id = {Y0pCki6q_DkC}, } - Crowdsourced validation of a machine-learning classification system for autism and ADHDM Duda , N Haber, J Daniels, and 1 more authorTranslational psychiatry, 2017
Autism spectrum disorder (ASD) and attention deficit hyperactivity disorder (ADHD) together affect> 10% of the children in the United States, but considerable behavioral overlaps between the two disorders can often complicate differential diagnosis. Currently, there is no screening test designed to differentiate between the two disorders, and with waiting times from initial suspicion to diagnosis upwards of a year, methods to quickly and accurately assess risk for these and other developmental disorders are desperately needed. In a previous study, we found that four machine-learning algorithms were able to accurately (area under the curve (AUC)> 0.96) distinguish ASD from ADHD using only a small subset of items from the Social Responsiveness Scale (SRS). Here, we expand upon our prior work by including a novel crowdsourced data set of responses to our predefined top 15 SRS-derived questions from
@article{duda_crowdsourced_2017, author = {Duda, M and Haber, N and Daniels, J and Wall, DP}, citation = {Translational psychiatry 7 (5), e1133-e1133, 2017}, journal = {Translational psychiatry}, number = {5}, pages = {e1133-e1133}, year = {2017}, publisher = {Nature Publishing Group}, title = {Crowdsourced validation of a machine-learning classification system for autism and ADHD}, volume = {7}, _google_scholar_id = {Zph67rFs4hoC}, } - Sparsifying machine learning models identify stable subsets of predictive features for behavioral detection of autismSebastien Levy, Marlena Duda, Nick Haber, and 1 more authorMolecular autism, 2017
Autism spectrum disorder (ASD) diagnosis can be delayed due in part to the time required for administration of standard exams, such as the Autism Diagnostic Observation Schedule (ADOS). Shorter and potentially mobilized approaches would help to alleviate bottlenecks in the healthcare system. Previous work using machine learning suggested that a subset of the behaviors measured by ADOS can achieve clinically acceptable levels of accuracy. Here we expand on this initial work to build sparse models that have higher potential to generalize to the clinical population.We assembled a collection of score sheets for two ADOS modules, one for children with phrased speech (Module 2; 1319 ASD cases, 70 controls) and the other for children with verbal fluency (Module 3; 2870 ASD cases, 273 controls). We used sparsity/parsimony
@article{levy_sparsifying_2017, author = {Levy, Sebastien and Duda, Marlena and Haber, Nick and Wall, Dennis P}, citation = {Molecular autism 8, 1-17, 2017}, journal = {Molecular autism}, pages = {1-17}, year = {2017}, publisher = {BioMed Central}, title = {Sparsifying machine learning models identify stable subsets of predictive features for behavioral detection of autism}, volume = {8}, _google_scholar_id = {Wp0gIr-vW9MC}, } - 5.13 Design and efficacy of a wearable device for social affective learning in children with autismASD AGGJournal of the American Academy of Child & Adolescent Psychiatry, 2017
Objectives: We investigated the feasibility of using physiological biomarkers to predict the onset of aggression in minimally verbal (MV) youth with autism spectrum disorder (ASD).Methods: Nine MV youth with confirmed ASD wore a wristband-mounted E4 biosensor during repeated unstructured observation periods while they were hospitalized in a specialized child psychiatry unit. Physiological and three axis acceleration data were collected concurrent with coding of aggressive behavior. Physiological arousal was measured by: 1) heart rate and heart rate variability, both derived from blood volume pulse (BVP) and interbeat interval (IBI) via photoplethysmography at 64 Hz; and 2) electrodermal activity (EDA), which reflects autonomic innervation of sweat glands. Advanced signal processing and machine learning algorithms were then applied to predict aggression onset. The area under the curve (AUC) accuracy (based on true/false positive rates) was calculated to predict the onset of aggression in the next one minute from present time (t). The predictions were made through a ridge-regularized logistic regression using: 1) previous t 3 minutes of motor movement acceleration (ACC) signals; 2) time elapsed since last aggression event; 3) previous t 3 minutes of BVP, EDA, and IBI signals; and 4) all of the above signals combined.Results: All youth tolerated the sensor after desensitization, usable data were obtained in all cases, and there was an average of 9.67 (range 0-44) aggressive episodes per four-hour observation period. Time-synced coding of aggression and concurrent E4 signal data predicted the onset of aggression with AUC
@article{agg_5.13_2017, author = {AGG, ASD}, citation = {Journal of the American Academy of Child \& Adolescent Psychiatry 56 (10S), 2017}, journal = {Journal of the American Academy of Child \& Adolescent Psychiatry}, number = {10S}, year = {2017}, title = {5.13 Design and efficacy of a wearable device for social affective learning in children with autism}, volume = {56}, _google_scholar_id = {2osOgNQ5qMEC}, }
2016
- Use of machine learning for behavioral distinction of autism and ADHDM Duda, R Ma , N Haber, and 1 more authorTranslational psychiatry, 2016
Although autism spectrum disorder (ASD) and attention deficit hyperactivity disorder (ADHD) continue to rise in prevalence, together affecting> 10% of todays pediatric population, the methods of diagnosis remain subjective, cumbersome and time intensive. With gaps upward of a year between initial suspicion and diagnosis, valuable time where treatments and behavioral interventions could be applied is lost as these disorders remain undetected. Methods to quickly and accurately assess risk for these, and other, developmental disorders are necessary to streamline the process of diagnosis and provide families access to much-needed therapies sooner. Using forward feature selection, as well as undersampling and 10-fold cross-validation, we trained and tested six machine learning models on complete 65-item Social Responsiveness Scale score sheets from 2925 individuals with either ASD (n= 2775) or ADHD
@article{duda_use_2016, author = {Duda, M and Ma, R and Haber, N and Wall, DP}, citation = {Translational psychiatry 6 (2), e732-e732, 2016}, journal = {Translational psychiatry}, number = {2}, pages = {e732-e732}, year = {2016}, publisher = {Nature Publishing Group}, title = {Use of machine learning for behavioral distinction of autism and ADHD}, volume = {6}, _google_scholar_id = {Tyk-4Ss8FVUC}, } - A wearable social interaction aid for children with autismPeter Washington, Catalin Voss, Nick Haber, and 5 more authors2016
Over 1 million children under the age of 17 in the US have been identified with Autism Spectrum Disorder (ASD). These children struggle to recognize facial expressions, make eye contact, and engage in social interactions. Gaining these skills requires intensive behavioral interventions that are often expensive, difficult to access, and inconsistently administered.nWe have developed a system to automate facial expression recognition that runs on wearable glasses and delivers real time social cues, with the goal of creating a behavioral aid for children with ASD that maximizes behavioral feedback while minimizing the distractions to the child. This paper describes the design of our system and interface decisions resulting from initial observations gathered during multiple preliminary trials.
@article{washington_a_2016, author = {Washington, Peter and Voss, Catalin and Haber, Nick and Tanaka, Serena and Daniels, Jena and Feinstein, Carl and Winograd, Terry and Wall, Dennis}, citation = {Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors , 2016}, pages = {2348-2354}, year = {2016}, title = {A wearable social interaction aid for children with autism}, _google_scholar_id = {4TOpqqG69KYC}, } - Superpower glass: delivering unobtrusive real-time social cues in wearable systemsCatalin Voss, Peter Washington, Nick Haber, and 8 more authors2016
We have developed a system for automatic facial expression recognition, which runs on Google Glass and delivers real-time social cues to the wearer. We evaluate the system as a behavioral aid for children with Autism Spectrum Disorder (ASD), who can greatly benefit from real-time non-invasive emotional cues and are more sensitive to sensory input than neurotypically developing children. In addition, we present a mobile application that enables users of the wearable aid to review their videos along with auto-curated emotional information on the video playback bar. This integrates our learning aid into the context of behavioral therapy. Expanding on our previous work describing in-lab trials, this paper presents our system and application-level design decisions in depth as well as the interface learnings gathered during the use of the system by multiple children with ASD in an at-home iterative trial.
@article{voss_superpower_2016, author = {Voss, Catalin and Washington, Peter and Haber, Nick and Kline, Aaron and Daniels, Jena and Fazel, Azar and De, Titas and McCarthy, Beth and Feinstein, Carl and Winograd, Terry and Wall, Dennis}, citation = {Proceedings of the 2016 ACM International Joint Conference on Pervasive and , 2016}, pages = {1218-1226}, year = {2016}, title = {Superpower glass: delivering unobtrusive real-time social cues in wearable systems}, _google_scholar_id = {eQOLeE2rZwMC}, } - The Feynman propagator on perturbations of Minkowski spaceJesse Gell-Redman, Nick Haber, and Andrs VasyCommunications in Mathematical Physics, 2016
In this paper we analyze the Feynman wave equation on Lorentzian scattering spaces. We prove that the Feynman propagator exists as a map between certain Banach spaces defined by decay and microlocal Sobolev regularity properties. We go on to show that certain nonlinear wave equations arising in QFT are well-posed for small data in the Feynman setting.
@article{gell-redman_the_2016, author = {Gell-Redman, Jesse and Haber, Nick and Vasy, Andrs}, citation = {Communications in Mathematical Physics 342, 333-384, 2016}, journal = {Communications in Mathematical Physics}, pages = {333-384}, year = {2016}, publisher = {Springer Berlin Heidelberg}, title = {The Feynman propagator on perturbations of Minkowski space}, volume = {342}, _google_scholar_id = {ufrVoPGSRksC}, } - A practical approach to real-time neutral feature subtraction for facial expression recognitionNick Haber, Catalin Voss, Azar Fazel, and 2 more authors2016
Methods for automated facial expression recognition - identifying faces as happy, sad, angry, etc. - typically rely on the classification of features extracted from images. These features, designed to encode shape and texture information, depend on both (1) the expression an individual is making, and (2) the individual’s physical characteristics and lighting conditions of the image. To reduce the effect of (2), a common strategy is to establish a "baseline" for an individual and subtract out this individual’s baseline neutral feature. This extra neutral feature information often is not available - in particular for in-the-wild, real-time classification of a previously unseen subject. Thus, in order to implement "neutral subtraction," one must estimate the individual’s neutral feature. Existing methods to do this are susceptible to class imbalance at test time (e.g., averaging over all facial features), require a more complex model specific to
@article{haber_a_2016, author = {Haber, Nick and Voss, Catalin and Fazel, Azar and Winograd, Terry and Wall, Dennis P}, citation = {2016 IEEE Winter Conference on Applications of Computer Vision (WACV), 1-9, 2016}, conference = {2016 IEEE Winter Conference on Applications of Computer Vision (WACV)}, pages = {1-9}, year = {2016}, publisher = {IEEE}, title = {A practical approach to real-time neutral feature subtraction for facial expression recognition}, _google_scholar_id = {qxL8FJ1GzNcC}, } - Titas De, Beth McCarthy, Carl Feinstein, Terry Winograd, and Dennis Wall. 2016. Superpower glass: delivering unobtrusive real-time social cues in wearable systemsCatalin Voss, Peter Washington, Nick Haber, and 3 more authorsProceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct (Heidelberg, Germany)(UbiComp16). Association for Computing Machinery, New York, NY, USA, 2016
@article{voss_titas_2016, author = {Voss, Catalin and Washington, Peter and Haber, Nick and Kline, Aaron and Daniels, Jena and Fazel, Azar}, citation = {Proceedings of the 2016 ACM International Joint Conference on Pervasive and , 0}, journal = {Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct (Heidelberg, Germany)(UbiComp16). Association for Computing Machinery, New York, NY, USA}, pages = {1218-1226}, year = {2016}, title = {Titas De, Beth McCarthy, Carl Feinstein, Terry Winograd, and Dennis Wall. 2016. Superpower glass: delivering unobtrusive real-time social cues in wearable systems}, _google_scholar_id = {mVmsd5A6BfQC}, } - Superpower glassC Voss, P Washington , and N Haber2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, 2016
@article{voss_superpower_2017, author = {Voss, C and Washington, P and Haber, N}, citation = {2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, 2016}, journal = {2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing}, year = {2016}, title = {Superpower glass}, _google_scholar_id = {kNdYIx-mwKoC}, } - Invited Talks at the AAAI Symposium on Well-Being Computing.Rafael A Calvo, Nick Haber, Catalin Voss, and 4 more authors2016
This paper contains abstracts of the invited talks presented at the AAAI 2016 Symposium on Well-Being Computing.
@article{calvo_invited_2016, author = {Calvo, Rafael A and Haber, Nick and Voss, Catalin and Nova, Michael and Salins, Dennis and Snyder, Michael and Wall, Dennis P}, citation = {AAAI Spring Symposia, 2016}, conference = {AAAI Spring Symposia}, year = {2016}, title = {Invited Talks at the AAAI Symposium on Well-Being Computing.}, _google_scholar_id = {UeHWp8X0CEIC}, }
2015
- Propagation of singularities around a Lagrangian submanifold of radial pointsNick Haber, and Andrs VasyBull. Soc. Math. France, 2015
In this work we study the wavefront set of a solution u to Pu= f, where P is a pseudodifferential operator on a manifold with real-valued homogeneous principal symbol p, when the Hamilton vector field corresponding to p is radial on a Lagrangian submanifold contained in the characteristic set of P. The standard propagation of singularities theorem of Duistermaat-Hrmander gives no information at . By adapting the standard positive-commutator estimate proof of this theorem, we are able to conclude additional regularity at a point q in this radial set, assuming some regularity around this point. That is, the a priori assumption is either a weaker regularity assumption at q, or a regularity assumption near but not at q. Earlier results of Melrose and Vasy give a more global version of such analysis. Given some regularity assumptions around the Lagrangian submanifold, they obtain some regularity at the Lagrangian
@article{haber_propagation_2015, author = {Haber, Nick and Vasy, Andrs}, citation = {Bull. Soc. Math. France 143 (4), 679-726, 2015}, journal = {Bull. Soc. Math. France}, number = {4}, pages = {679-726}, year = {2015}, title = {Propagation of singularities around a Lagrangian submanifold of radial points}, volume = {143}, _google_scholar_id = {NMxIlDl6LWMC}, }
2014
- A normal form around a Lagrangian submanifold of radial pointsInternational Mathematics Research Notices, 2014
In this work, we produce microlocal normal forms for pseudodifferential operators which have a Lagrangian submanifold of radial points. This answers natural questions about such operators and their associated classical dynamics. In a sequel, we will give a microlocal parametrix construction, as well as a construction of a microlocal Poisson operator, for such pseudodifferential operators.
@article{haber_a_2014, author = {Haber, Nick}, citation = {International Mathematics Research Notices 2014 (17), 4804-4821, 2014}, journal = {International Mathematics Research Notices}, number = {17}, pages = {4804-4821}, year = {2014}, publisher = {OUP}, title = {A normal form around a Lagrangian submanifold of radial points}, volume = {2014}, _google_scholar_id = {_FxGoFyzp5QC}, }
2013
- Microlocal analysis of Lagrangian submanifolds of radial points2013
Microlocal analysis relies on correspondences between quantum physics and classical physics to give information about certain PDEs–for instance, linear variable-coefficient PDEs on manifolds. PDEs are interpreted as quantum systems. The corresponding classical systems tell us, for example, function spaces on which problems are solvable or almost solvable, existence and uniqueness results, and the structure of solution operators. Landmark papers of Hrmander and Duistermaat and Hrmander establish key results for the standard calculus of microlocal analysis, which gives a broad framework for dealing with variable-coefficient PDEs on manifolds. Their work is well-suited for dealing with PDEs which, in a generalized sense, are hyperbolic, with corresponding classical dynamics looking like wave propagation of geometric optics. In this thesis, we aim to extend many of their results to situations in which the
@article{haber_microlocal_2013, author = {Haber, Nick}, citation = {Stanford University, 2013}, year = {2013}, title = {Microlocal analysis of Lagrangian submanifolds of radial points}, _google_scholar_id = {u-x6o8ySG0sC}, }
2012
- Propagation of singularities around a Lagrangian submanifold of radial pointsNick Haber, and Andrs Vasy2012
Para>This talk discusses the wavefront set of a solution u to Pu = f, where P is a pseudodifferential operator on a manifold with real-valued homogeneous principal symbol p, when the Hamilton vector field corresponding to p is radial on a Lagrangian submanifold contained in the characteristic set of P. According to a theorem of Duistermaat-Hrmander [2], singularities propagate along bicharacteristics of this Hamilton vector field. This theorem gives us no information about the wavefront set when the Hamilton vector field is radial.
@article{haber_propagation_2012, author = {Haber, Nick and Vasy, Andrs}, citation = {Microlocal Methods in Mathematical Physics and Global Analysis, 113-116, 2012}, pages = {113-116}, year = {2012}, publisher = {Springer Basel}, title = {Propagation of singularities around a Lagrangian submanifold of radial points}, _google_scholar_id = {4DMP91E08xMC}, } - Mobile Crowdsourcing of Emotive Video for Autism ResearchHaik Kalantarian, and Nick Haber2012
Autism Spectrum Disorder (ASD) is a condition affecting 70 million children worldwide. Due to delays in diagnosis and imbalances in coverage, it is necessary to develop new methods of care delivery for use outside of clinical settings. In this paper, we present a mobile charades-style game, Guess What?, used for crowdsourcing the acquisition of structured video from children with ASD for behavioral disease research. By analyzing facial affect in response to various prompts, we demonstrate that engagement and facial affect can be quantified and measured using real-time image processing algorithms: an important first-step for future therapies, diagnostics, and outcome measures based on home video. Our study of thirteen subjects reveals that game sessions displaying faces to the player produced the most emotive facial expressions in the player by a considerable margin. Videos from the younger neurotypical group contained 73.9% more frames with emotion in this category, compared to the older group with ASD. Moreover, results from all categories indicated 20.7% less facial engagement among older children with ASD compared to younger children with no diagnosis. We will attempt to replicate these findings in a larger cohort in future work.
@article{kalantarian_mobile_2022, author = {Kalantarian, Haik and Haber, Nick}, citation = {}, year = {2012}, title = {Mobile Crowdsourcing of Emotive Video for Autism Research}, _google_scholar_id = {9yKSN-GCB0IC}, }