Integrating Human and Machine Learning for Enabling Co-Adaptive Body-Machine Interfaces

This project for the Mind, Machine, and Motor Nexus (M3X) program will advance understanding of how people learn new neuromotor skills, and subsequently apply that understanding to the creation of innovative wearable device controllers called body-machine interfaces (BoMIs). Individuals with neuromuscular impairment -- perhaps due to stroke or spinal cord injury -- may have difficulty carrying out the activities of everyday life.

This project explores novel interfaces through which an individual can use their body's residual mobility to issue commands to assistive devices such as computer cursors, wheelchairs, or robotic arms. The project has three main research goals. The first is to improve existing methods for translating small body movements into controller commands for assistive devices.

The second is to model the process by which the human user learns over time to use the body-machine interface. The third is to apply the obtained model of the learning process to enable the body-machine interface to adjust to the evolving characteristics of the human user. An interface that does not adapt to changes in its user may significantly degrade in performance over time.

On the other hand, an interface whose properties instantly change with every small shift in user behavior will be difficult to control. The ultimate outcome of this project will be human-machine interfaces based on body movement that consider the user and the interface as two components of an integrated system in which each component continually learns from and adapts to the other.

The results of the project will lead to assistive devices that more affordable, and provide more versatile control and ease of use. The underlying principles of co-adaptation to be identified through this work are also relevant to rehabilitation from disease or injury, as well as to increasing the capabilities of human-operated robotic systems.

Recent work has shown that linear methods such as principal component analysis (PCA) may be effectively used in a body-machine interface (BoMI) to map elements from a higher dimensional feature space of body movements onto a lower dimensional space of device commands. In this project, the features that provide input to the BoMI are generated by multiple inertial measurement units (IMUs) worn by the user; the IMUs report their current orientation in an inertial reference space.

The output from the BoMI are commands used to control a sequence of representative devices, specifically a computer cursor, a simulated wheelchair, an actual wheelchair, and a simulated manipulator arm. The three technical goals of the project are as follows: 1) Compare the performance of a linear map based on PCA to a nonlinear map based on an autoencoder network (AEN) for providing input features to the BoMI that translates residual mobility space features into device commands.

The AEN is capable of representing a richer variety of features than PCA, but it remains to be shown, for example, whether human users can make effective use of that variety. 2) Obtain a computable representation of the process by which humans learn neuromotor skills. This representation will be based on the premise that humans simultaneously learn both a forward and inverse map of the relationship between neuromotor signals and the resulting physical outcomes.

Once learned, the forward map predicts the outcomes that will result from a certain set of signals, while the inverse map is used to generate the signals that correspond to a given desired physical outcome. As a person learns mastery of a neuromotor skill, the forward and inverse maps become more accurate predictors of actual behavior, and the degree of learning can be monitored through estimates of these maps. 3) Incorporate a co-adaptation algorithm into the BoMI for maintaining performance as the user's mastery of the BoMI changes.

In most current approaches to human-machine interfaces, the interface is fixed following an initial calibration stage, and the user must learn to control that interface configuration. In this project, the learning representation of objective (2) will be used to monitor and periodically update the BoMI map parameters. The implementation of this objective is aided by parallels between the human learning model and the AEN training method, which automatically generates a decoder network that captures the inverse map between desired device commands and the corresponding residual mobility features needed to produce them.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.