Khimya Khetarpal

Khimya Khetarpal

I am a Research Scientist at Google Deepmind and an Affiliate Faculty Member of (Mila). I earned my Ph.D. in Computer Science from Reasoning and Learning Lab at McGill University and Mila, where I was advised by Doina Precup. I am broadly interested in artificial intelligence and reinforcement learning.

Research Summary: I am interested in the capability of AI agents to understand and develop broadly intelligent behavior. My research focuses on how agents can efficiently represent the world's knowledge, plan with it, and adapt to changes over time through learning and interaction. See my research page for more details.

Email  /  CV  /  GitHub  /  Google Scholar  /  Twitter  /  YouTube

profile photo
  • DeepMind December 2022 - Now
    Research Scientist
  • Microsoft Research Fall 2021, Winter 2022
    Research Intern [Host: Katja Hofmann, Harm van Seijen]
  • DeepMind Summer 2021
    Research Scientist Intern [Host: Satinder Singh, Tom Zahavy]
  • DeepMind Fall 2019
    Research Scientist Intern [Host: Doina Precup]
  • McGill University 2017 - Now
    Ph.D. in Computer Science
  • Intel 2016 - 2017
    Perceptual Computing Engineer
  • Univerity of Florida 2014 - 2016
    Masters in Computer Engineering
  • IIT Kanpur 2013 - 2014
    Research Associate
  • Robert Bosch 2011-2012
    Embedded Software Development
  • VIT University 2007 - 2011
    B.Tech in Electronics & Communication Engineering



Highlights and News



Media
sym A concept in psychology is helping AI to better navigate our world

MIT Technology Review , Hao, K. (July, 2020)

sym UF Engineers Display Intelligent Machines At Robot Demo Day

WUFT News , Whitson, C. (December, 2014)

The Gainesville Sun , Finger, D. (December, 2014)

Research

My research aims to (1) understand intelligent behavior that bridges both action and perception grounded in theoretical foundations of reinforcement learning, and (2) build AI agents to efficiently represent the world knowledge, plan with it, and adapt to changes over time through learning and interaction. I approach this with the following research directions:

Selective Attention for Fast Adaptation and Robustness

Humans adapt robustly in complex and dynamic environments by using selective attention, which is then aggregated in a representation. How can do we enable more efficient learning of such representations to guide behavior?

Papers
Attend Before You Act: Leveraging human visual attention for Continual Learning [ICML Workshop 2018]
Options of Interest: Temporal abstraction with interest functions [AAAI 2020]
Self-Supervised Attention-Aware Reinforcement Learning [AAAI 2021]

 

Learning Abstractions and Affordances

Interactive behavior requires dynamically tracking and updating action possibilities, i.e. ``affordances'' (defined as set of state and actions that achieve desired consequences). How can AI agents learn to represent and reason about their environment through this lens?

Papers
What can I do here? A theory of Affordances in Reinforcement Learning [ICML 2020]
Temporally Abstract Partial Models [NeurIPS 2021]
The Paradox of Choice: On the Role of Attention in HRL [NeurIPS Workshop, 2022]
Toward Human-AI Alignment in Large-Scale Multi-Player Games [Preprint, 2024]

 

Discovery and Continual Reinforcement Learning

For practical applications, to what extent can RL agents transfer what was learned from previous experience to new situations on-the-fly and adapt to non-stationarity, with the ability to discover more complex capabilities on top of those already developed?

Papers
Continual Reinforcement Learning: A Review and Perspectives [JAIR 2022]
POMRL: No-Regret Learning-to-Plan with Increasing Horizons [TMLR 2023]
Discovering Object-Centric Generalized Value Functions From Pixels [ICML 2023]

Papers

Representative papers are highlighted.

sym

Toward Human-AI Alignment in Large-Scale Multi-Player Games
Sugandha Sharma, Guy Davidson, Khimya Khetarpal, Anssi Kanervisto, Udit Arora, Katja Hofmann, Ida Momennejad,
Under Review, 2024.

sym

Disentangling the Causes of Plasticity Loss in Neural Networks
Clare Lyle, Zeyu Zheng, Khimya Khetarpal, Hado van Hasselt, Razvan Pascanu, James Martens, Will Dabney
Under Review, 2024.

sym

Forecaster: Towards Temporally Abstract Tree-Search Planning from Pixels
Thomas Jiralerspong, Flemming Kondrup, Doina Precup, Khimya Khetarpal,
GenPlan Workshop, Neural Information Processing Systems (NeurIPS), 2023.

sym

Discovering Object-Centric Generalized Value Functions From Pixels
Somjit Nath, Gopeshh Subbaraj, Khimya Khetarpal, Samira Ebrahimi Kahou
International Conference on Machine Learning (ICML), 2023.

sym

POMRL: No-Regret Learning-to-Plan with Increasing Horizons
Khimya Khetarpal*, Claire Vernade*, Brendan O'Donoghue, Satinder Singh, Tom Zahavy
Transactions on Machine Learning Research (TMLR), 2023. (Expert Reviewer Certification.)
GenPlan Workshop, Neural Information Processing Systems (NeurIPS), 2023.

sym

The Paradox of Choice: Using Attention in Hierarchical Reinforcement Learning
Andrei Nica*, Khimya Khetarpal*, Doina Precup
All Things Attention Workshop, Neural Information Processing Systems (NeurIPS), 2022.

paper | code |
sym

Temporally Abstract Partial Models
Khimya Khetarpal, Zafarali Ahmed, Gheorghe Comanici, Doina Precup
Neural Information Processing Systems (NeurIPS), 2021

sym

Towards Continual Reinforcement Learning: A Review and Perspectives
Khimya Khetarpal*, Matthew Riemer*, Irina Rish, Doina Precup
Journal of Artificial Intelligence Research (JAIR), 2022

sym

Learning Robust State Abstractions for Hidden-Parameter Block MDPs
Amy Zhang, Shagun Sodhani, Khimya Khetarpal, Joelle Pineau
International Conference on Learning Representations (ICLR), 2021

sym

Sequoia: A Software Framework to Unify Continual Learning Research
Fabrice Normandin, Florian Golemo, Oleksiy Ostapenko, Pau Rodriguez, Matthew D Riemer, Julio Hurtado, Khimya Khetarpal, Dominic Zhao, Ryan Lindeborg, Timothée Lesort, Laurent Charlin, Irina Rish, Massimo Caccia
Workshop on Theory and Foundation of Continual Learning (ICML Workshop), 2021

sym

Self-Supervised Attention-Aware Reinforcement Learning
Haiping Wu, Khimya Khetarpal, Doina Precup
Association for the Advancement of Artificial Intelligence (AAAI), 2021

sym

Variance Penalized On-Policy and Off-Policy Actor-Critic
Arushi Jain, Gandharv Patil, Ayush Jain, Khimya Khetarpal, Doina Precup
Association for the Advancement of Artificial Intelligence (AAAI), 2021

sym

What can I do here? A Theory of Affordances in Reinforcement Learning (Featured in MIT Technology Review)
Khimya Khetarpal, Zafarali Ahmed, Gheorghe Comanici, David Abel, Doina Precup
International Conference on Machine Learning (ICML), 2020

 
sym

Options of Interest: Temporal Abstraction with Interest Functions
Khimya Khetarpal, Martin Klissarov, Maxime Chevalier-Boisvert, Pierre-Luc Bacon, Doina Precup
Association for the Advancement of Artificial Intelligence (AAAI), 2020

sym

Value Preserving State-Action Abstractions
David Abel, Nathan Umbanhowar, Khimya Khetarpal, Dilip Arumugam, Doina Precup, and Michael L. Littman
International Conference on Artificial Intelligence and Statistics (AISTATS), 2020

sym

Learning Generalized Temporal Abstractions across Both Action and Perception (Scholarship Award)
Khimya Khetarpal
Association for the Advancement of Artificial Intelligence (AAAI), 2019
Doctorial Consortium Track

sym

Learning Options with Interest Functions (3 Minute Thesis Finalist)
Khimya Khetarpal, Doina Precup
Association for the Advancement of Artificial Intelligence (AAAI), 2019
Student Abstract Track

sym

Variational State Encoding as Intrinsic Motivation in Reinforcement Learning
Martin Klissarov*, Riashat Islam*, Khimya Khetarpal, Doina Precup
The Multi-disciplinary Conference on Reinforcement Learning and Decision Making (RLDM), 2019

sym

Attend before you act: Leveraging human visual attention for continual learning (Best Paper Award- 3rd Place)
Khimya Khetarpal, Doina Precup
Lifelong Learning: A Reinforcement Learning Approach Workshop (ICML), 2018

sym

Safe option-critic: Learning safety in the option-critic architecture
Arushi Jain*, Khimya Khetarpal*, Doina Precup
Adaptive Learning Agents Workshop, (ICML), 2018.
Invited for submission to special issue of The Knowledge Engineering Review (Cambridge University Press journal)

sym

Re-evaluate: Reproducibility in evaluating reinforcement learning algorithms
Khimya Khetarpal*, Zafarali Ahmed*, Andre Cianflone, Riashat Islam, Joelle Pineau
Reproducibility in Machine Learning Workshop, (ICML), 2018.

sym

Environments for Lifelong Reinforcement Learning
Khimya Khetarpal*, Shagun Sodhani*, Sarath Chandar, Doina Precup
Continual Learning Workshop, Workshop, (NeurIPS), 2018.

sym

Creating Segments and Effects on Comics by Clustering Gaze Data
Ishwarya Thirunarayanan, Khimya Khetarpal, Sanjeev Koppal, Olivier Le Meur, John Shea and Eakta Jain
ACM Transactions on Multimedia Computing, Communications, and Applications, (TOMM), 2017.

sym

A Preliminary Benchmark Of Four Saliency Algorithms On Comic Art
Khimya Khetarpal, Eakta Jain
International Workshop on Multimedia Artworks Analysis (MMArt),
(IEEE ICME), 2016.

sym

Mobile robot navigation using evolving neural controller in unstructured environments
AwhanPatnaik, Khimya Khetarpal, Laxmidhar Behera
International Conference on – Advances in Control and Optimization of Dynamical Systems,
(IFAC), 2014.




Talks
sym POMRL: No-Regret Learning-to-Plan with Increasing Horizons

Upper Bound, Amii, Edmonton, Invited Talk, May, 2023
Meta Paris, Virtual, Invited Talk, Aug, 2023
WEIRD Lab, UW, Seattle, Invited Talk, Apr, 2023
sym Bridging State and Action: Towards Continual Reinforcement Learning (PhD Defence)

RL Lab, McGill University, Mila, Montreal, October, 2022
slides | pictures
sym Bridging State and Action: Towards Continual Reinforcement Learning

RLAI Lab, University of Alberta, Edmonton, Invited Talk, 2022
Microsoft Research, NYC, Invited Talk, 2022 (virtual)
Microsoft Research, Montreal, Invited Talk, 2022 (virtual)
Brown Robotics Lab, Brown University, Invited Talk, 2022, (virtual)
Deepmind , Invited Talk, 2022, (virtual)
Google Research , Invited Talk, 2022 (virtual)
sym Temporally Abstract Partial Models
Neural Information Processing Systems (NeurIPS), 2021
Microsoft Research RL Reading Group, Invited Talk, 2021
sym Towards Continual Reinforcement Learning: A Review and Perspectives
RIKEN Center for Advanced Intelligence Project- Approximate Bayesian Inference Team (Japan), Invited Talk, 2021
sym What can I do here? A Theory of Affordances in Reinforcement Learning (Featured in MIT Technology Review)
International Conference on Machine Learning (ICML), Virtual Vienna, 2020 Northeast Reinforcement Learning and Decision Making Symposium (NERDS), 2020
sym Options of Interest: Temporal Abstraction with Interest Functions
Association for the Advancement of Artificial Intelligence (AAAI), New York, 2020.
sym Learning Generalized Temporal Abstractions across Both Action and Perception
Association for the Advancement of Artificial Intelligence (AAAI), Hawaii, 2019
Doctorial Consortium Track, (Mentor: Michael Littman)
sym Learning Options with Interest Functions
Association for the Advancement of Artificial Intelligence (AAAI), Hawaii, 2019
Student Abstract Track, (3 Minute Thesis Finalist)
sym Attend before you act: Leveraging human visual attention for continual learning (Best Paper Award- 3rd Place)
Lifelong Learning: A Reinforcement Learning Approach Workshop (ICML), Stockholm, 2018
Advising/Mentorship
Teaching
sym Teaching Assistant, COMP-767 Reinforcement Learning, Winter 2020

Teaching Assistant, COMP-208 Computers in Engineering, Winter 2018

Guest Lecture, Hierarchical RL, Management Studies, 2019 [slides]
sym Reinforcement Learning, IVADO Deep Learning Summer School, 2019 [slides]
sym Lecturer, Reinforcement Learning, 2020 [slides]

Lecturer, Deep Reinforcement Learning, 2019 [slides]

Teaching Assistant, 2018 [resources]
Organizational Roles
sym All Things Attention: Bridging Different Perspectives on Attention, NeurIPS 2022

A Roadmap to Never-Ending Reinforcement Learning, Workshop, ICLR 2021

Rethinking ML Papers, Workshop, ICLR 2021

Continual Reinforcement Learning, Un-Workshop WiML, ICML 2020

Lifelong Learning: A Reinforcement Learning Approach (LLARLA), RLDM 2019

Multi-Task and Lifelong Reinforcement Learning Workshop, ICML 2019

Area Chair, Women in Machine Learning (WiML), NeurIPS 2018
Reviewing
sym Reviewer, JMLR, Journal of Machine Learning Research

Reviewer, TMLR, Transactions on Machine Learning Research

Reviewer, EWRL, European Workshop on Reinforcement Learning('22)

Reviewer, AISTATS ('21), ICLR ('20, '21), NeurIPS ('20, '21, '22)

Reviewer, NeurIPS, Deep RL Workshop ('20), Reproducibility Challenge ('19)

Program Committee, Continual Learning Workshop, NeurIPS 2018

Reviewer, AI for Social Good Workshop, NeurIPS 2018

© Khimya Khetarpal
Base Design & CSS courtsey Jon Barron.