SpatioTemporal Reconstruction of Interacting People for pErceiving Systems

People constantly interact with objects to perform tasks.

To help people accomplish these, computers need to perceive Human-Object Interactions (HOI), and for this, they need to reconstruct HOI from whole-body color images of people interacting with objects or scenes.

This is challenging, due to the occlusions between bodies and objects, motion blur, depth ambiguities, and the low image resolution of hands and graspable object parts.

There has been significant prior work on estimating 3D humans without considering objects, and estimating 3D objects without considering humans.

Little prior work estimates these jointly, but, for tractability, focuses either on interacting hands, ignoring the body, or on interacting bodies, ignoring hands.

Only recent work addresses dexterous interaction of whole bodies, but instruments bodies with intrusive markers or sensors, and uses non-standard cameras to capture video of interactions.

Moreover, reconstruction lacks hand detail that is crucial for grasping, and videos are captured in constrained settings, consequently, methods trained on these struggle generalizing. My goal is to infer HOI from natural whole-body images/videos.

To this end, I present an ambitious 5-year research agenda with novelties in four directions: (1) developing strong generative 3D shape models for objects and humans for a novel HOI representation; (2) developing methods that estimate 3D HOI from a color image with rich contact and proximal awareness; (3) instilling spatiotemporal reasoning into the heart of these for estimating 4D HOI from color video; and (4) extending these methods to also infer their own confidence that will be correlated with the reconstruction quality.

The outcome will be novel and robust methods for HOI reconstruction from natural images/videos.

This will fill an important gap, enabling future intelligent systems to amplify people’s skills and help them accomplish tasks, e.g. for assistive robots or virtual 3D assistants or trainers.