^{1}

^{2}

^{3}

^{2}

^{2}

The authors have declared that no competing interests exist.

Conceived and designed the experiments: JM JPC LP. Performed the experiments: JM. Analyzed the data: JM. Contributed reagents/materials/analysis tools: JM DMP JPC LP. Wrote the paper: JM DMP JPC LP.

Neuroprosthetic brain-computer interfaces are systems that decode neural activity into useful control signals for effectors, such as a cursor on a computer screen. It has long been recognized that both the user and decoding system can adapt to increase the accuracy of the end effector. Co-adaptation is the process whereby a user learns to control the system in conjunction with the decoder adapting to learn the user's neural patterns. We provide a mathematical framework for co-adaptation and relate co-adaptation to the joint optimization of the user's control scheme ("encoding model") and the decoding algorithm's parameters. When the assumptions of that framework are respected, co-adaptation cannot yield better performance than that obtainable by an optimal initial choice of fixed decoder, coupled with optimal user learning. For a specific case, we provide numerical methods to obtain such an optimized decoder. We demonstrate our approach in a model brain-computer interface system using an online prosthesis simulator, a simple human-in-the-loop pyschophysics setup which provides a non-invasive simulation of the BCI setting. These experiments support two claims: that users can learn encoders matched to fixed, optimal decoders and that, once learned, our approach yields expected performance advantages.

Brain-computer interfaces are systems which allow a user to control a device in their environment via their neural activity. The system consists of hardware used to acquire signals from the brain of the user, algorithms to decode the signals, and some effector in the world that the user will be able to control, such as a cursor on a computer screen. When the user can see the effector under control, the system is closed-loop, such that the user can learn based on discrepancies between intended and actual kinematic outcomes. During training sessions where the user has specified objectives, the decoding algorithm can be updated as well based on discrepancies between what the user is supposed to be doing and what was decoded. When both the user and the decoding algorithm are simultaneously co-adapting, performance can improve. We propose a mathematical framework which contextualizes co-adaptation as a joint optimization of the user’s control scheme and the decoding algorithm, and we relate co-adaptation to optimal, fixed (non-adaptive) choices of decoder. We use simulation and human psychophysics experiments intended to model the BCI setting to demonstrate the utility of this approach.

Brain-computer interfaces (BCI) allow for a user to control effectors, such as a cursor on a computer screen, by the modulation of neural activity. For this approach to be practical for clinical settings, performance optimization through user learning and decoder tuning is a critical issue (see [

When the BCI system is closed-loop such that both the user can learn and the decoding algorithm can adapt, we have a setting which permits co-adaptation [

While previous work has examined how decoder parameters should adapt in closed-loop settings in order to improve over static decoders [

Conceptually motivated by co-adaptation and BCI-user learning, we propose a framework for jointly optimizing the encoding model and decoder of a BCI under a minimum mean square error (MMSE) objective. The central premise of this work is that closed-loop co-adaptation is a special case of this joint encoder-decoder optimization problem, so an optimal decoder can be computed in advance, effectively circumventing the co-adaptation process. The core of our approach amounts to “pre-computing” the limit of an idealized co-adaptation process in which the user (hypothetically) optimally learns in order to obtain an optimized encoder-decoder pair. Instead of focusing on the temporal dynamics of learning in closed-loop [

In a very general sense, learning by both the user and the decoder is equivalent to an optimization of performance with respect to these components. This perspective is essential, as it allows us to critically investigate co-adaptation as a mathematical optimization procedure, rather than an ill-defined “synergistic” training process. Co-adaptation then amounts to a specific approach to optimizing this objective over time—a coordinate descent approach where the user updates the encoder, and then the BCI system updates the decoder, iteratively. Seen from this perspective, it becomes clear that this objective function over encoder-decoder pairs could instead be descended using some other optimization strategy, and then a fixed, pre-optimized decoder could be presented to the user and learned. Such a setting would obviate co-adaptation. This approach could break down in a few key places: (1) The optimization could be intractable. (2) We may not be able to characterize the user objective. (3) We may not be able to characterize the constraints on user learning.

It is worth emphasizing that these issues are present for co-adaptation as much as any generic optimization approach. Co-adaptation may not do a good job of obtaining optima of the objective function, and without knowing the user’s objective or constraints on user learning, co-adaptive decoder updates may be very difficult to tune or suboptimal.

In the remainder of this work, we specify a MSE objective, and we work through the joint encoder-decoder optimization framework for the case when encoding is linear and the decoder is a steady state Kalman Filter (SSKF). We assume that the SNR of the neurons is constrained and that it is difficult for the user to learn encoding models in which the correlation structure of the neural activity must change too much. We also assume that the user is aware of the MSE objective and is motivated to optimize this objective. Our model form is broadly consistent with other Kalman Filter decoding schemes employed in contemporary BCI. However, in particular settings, the objective function and the constraints on learning of the encoding model could be specialized. We return to the opportunity for specialization in the discussion after presenting our results.

Finally, we validate the pre-computed decoder in computer simulations as well as in an online prosthesis simulator (OPS) [

Allowing for changes to both the decoder and encoder, we here show how to obtain encoder-decoder pairs which theoretically yield better performance than would be obtained either by learning an arbitrary, fixed decoder or adapting the decoder when the user is not learning. Although conceptually applicable generally, we apply our framework to the conventional Kalman filter (KF), which serves as a reasonable choice of decoder for BCI system. KF decoding approaches have been made to have adaptive parameters in various ways [

To build a Bayesian decoder, we will need both an “encoding model” (or encoder), mapping user intention to neural activity, as well as a “prior model”, which relates the variables capturing user intention across time (

Top figure depicts the general setting where neural activity is taken together with prior information to decode an estimate of the underlying intention. When in closed loop, sensory (visual) feedback is provided which can allow the user to modify intention and also permit changes to the encoding model. Bottom figure depicts the steady state Kalman filter (SSKF), a simple exemplar of the general setting in which _{t}, _{t}, or _{t} and

For the KF, the kinematic variables are related by a first order autoregressive process (AR(1)), which serves as a prior model over cursor trajectories.
_{t} is a vector of size

We then specify an observation model, which in our case will be the linear encoding of the kinematic variable in the neural activity with additive Gaussian noise.
_{t} is a vector of size

The Kalman filter has transients due to initial conditions, but we are interested in the performance of the user and decoder on long timescales, so we will prefer the simplification that the system is in steady state. It has been shown that very little performance is lost in BCI settings using steady state approaches [_{SS} is the fixed point solution to the Riccati equation:

The problem we want to solve then is the joint optimization of the encoder and decoder. In the general Kalman filter decoder setting, we solve for parameters

For the problems we consider, we are provided with a source distribution for the kinematic intention variables _{t}, given by the AR(1) model parameters _{t}, predominantly given by the observation noise (

In trying to determine the optimal encoder-decoder pair, we consider the objective function to be MMSE, so the objective is (see equations (15)–(18) of _{x} is the marginal covariance for _{t} given by the (zero-mean) process distribution in _{x} (and thereby the marginal distribution _{t} ∼ 𝒩(0, Σ_{x})) by iterating the matrix recursion to convergence or by vectorizing and solving for the covariance in closed form [

The choice of encoding _{t}, which also carries an additive noise component with covariance

To capture the physiological requirement that neurons cannot be active with arbitrary magnitude and/or arbitrarily low noise, we must penalize the signal power while encouraging it to be as distinct as possible from the noise power. Additionally, when considering the optimal encoder-decoder pair, we want to constrain the optimal encoding model to a set which is feasible for the user to learn. Several studies suggest that BCI users can only adjust neural activity such that it does not violate certain features of the neural activity structure [

For our class of models, this joint activity constraint ensures that both signal and noise correlations in neural activity are appropriately respected. If future experiments reveal additional constraints that restrict the set of encoding models learnable by the user, it would be straightforward to model them and incorporate them into an updated penalty. In the simpler case where neural signal constraints are unknown and we only wish to focus on noise constraints, we can replace Σ_{y} with _{SNR}(^{−1} _{x} ^{T}).

We add the 𝒢_{joint}(

We optimize the full objective (

Conceptually, we compare the optimal coding solutions of the form described in the previous section with those that follow from using a static, linear decoder instead of a Kalman filter:

Such a decoder corresponds to the setting where each timestep of the process _{1…T} is sampled independently and identically distributed (i.i.d.) from a Gaussian, rather than from an AR(1) process. Work on “robust coding” has solved the optimal encoder-decoder problem for i.i.d. input without temporal correlations—versions of this problem are explored in the context of neuroscience in [

For the AR(1) case, we have empirically studied the objective function using both penalties with up to k = 200 neural channels to control n = 3 dimensions. As long as parameters are initialized to the correct order of magnitude (for numerical stability), optimized encoders and decoders obtained from different, random initializations converge to the same value of the objective function. The parameters (_{n × n} and Σ_{x} ∝ _{n × n}. In general, if there are other variables such as velocities, then _{n × n} and other blocks which relate the variables to position according to physical relationships (see [_{joint}(

Having provided the framework and described the encoder-decoder optimization methods, we validate our approach in two settings. We first examine the coding schemes arising from simulations and verify that our approach yields interpretable solutions. Following the simulation results, we show the results of psychophysics experiments implementing our approach on an online prosthesis simulator (OPS), a system which tracks overt human body movements and generates synthetic, closed-loop data.

In this section, we wish to build intuition for the optimized encoder-decoder pairs we obtain. Consider a generic case in which there are _{t} to the _{t}. The decoder is a 1 × _{t} to an estimate

We can simulate this setting by constructing signal and noise covariance matrices, and we can numerically compute the optimal decoder parameters. The optimization procedure finds encoders which have good signal subject to constraints on the magnitude of the encoded variables (i.e. for fixed SNR level). The encoder corresponds to the

In (a) & (c) we generate a signal covariance (Σ_{sig}) and noise covariance (_{y}). These covariances are sufficient statistics for the optimization procedure and allow us to determine the optimal encoder-decoder pair for these channels. In (b) & (d) we visualize the corresponding optimal encoder and decoder. For example 1 (a) & (b), the optimized encoder-decoder pair transmits information along dimensions 2 & 5 which have the highest SNR (i.e. signal magnitude is same for all dimensions and noise is lowest for dimensions 2 & 5). For example 2 (c) & (d), the signal is a rank 1 matrix and the encoder parameters preferentially encode higher signal dimensions. The obtained decoder parameters again reflect preferential decoding of dimensions with high signal and low noise.

Indeed, in simple example simulations here, we see that signal is coded along dimensions with less noise. In example 1 (top row of

The key point is that the encoder and decoder are matched. Recall that in the prosthetic setting, the user’s initial encoding will be arbitrary, but users who change their encoding (i.e. adapt their control scheme) to match the optimized encoder, will potentially be able to obtain optimal performance when using the corresponding decoder. The feature of our framework, that neurons not relevant for the decoder will have decreased encoding weight, has experimental corroboration in BCI settings from work showing that neurons not used by a decoder have decreased modulation [

In order to thoroughly examine our approach when we know how neural activity is generated, we tested our framework with human subjects in a general setting where all features of the system are observed and controlled. To this end, we used an online prosthesis simulator (OPS) to validate our approach in experimental demonstrations [

The top-most plot depicts target position observed by user, in response to which the user makes overt movements reflecting their intention. The second plot depicts these overt user movements captured by the motion-capture sensor and pose tracking software. The third plot depicts simulated neural activity which was synthesized by reweighting, mixing and adding noise to the raw movement data. The bottom-most plot depicts decoded cursor positions (1D) during the task. Target remains fixed until acquired by the cursor, noted by black markings around target transition times. Trajectories are smooth and under/overshoots are common, visual feedback being crucial to performance.

If the appropriate signals are available, it is natural for a neuroprosthesis to use biomimetic control—for example, using the neurons which correspond to imagined left-right movement of the hands to control left-right movement of a cursor. With the OPS, we set out to explore a relevant regime where the user is inclined to use a biomimetic decoder (e.g. moving their hands naturally to control a cursor), but higher SNR is available in other, non-biomimetic dimensions which might be less likely to be explored. By deciding what body movements generate neural activity and how much noise is present for various neural channels, we can construct cases which require the user to learn in order to achieve optimal performance. Naïve motor-imitation control will not suffice because the native movement dimensions are noisier. Closed-loop learning is required in this setting so the user can learn the non-biomimetic scheme.

The purposes of these OPS demonstrations are (1) to examine how much performance might be expected to improve if optimal encoder-decoder pairs are used instead of sub-optimal, initial mappings and (2) to examine the extent to which non-biomimetic schemes are learnable. To this end, we compared the use of a pre-computed, fixed decoder for the cursor task with biomimetic schemes to illustrate the sense in which our approach finds better solutions. Given that we only receive signal from a small number of control dimensions (i.e. overt movement channels) and we wish to make some of these control dimensions noisy, we restricted attention to 1D cursor versions of the task. We compute the optimal decoder according to our framework and use it for the cursor task, expecting that the user should be able to learn it. As described when presenting our framework, this decoder corresponds to a theoretically obtainable end-result of a co-adaptation process having occurred, for which user learning was permitted.

After a few minutes of free exploration during which the user could become comfortable controlling the cursor (see

We first consider single-subject comparisons using each of the two classes of decoders. We observe that our method performs visibly better than the motor-imitation decoder (see

The top plot depicts single trial data with a motor-imitation initialized decoder. The cursor moves more smoothly (i.e. oversmoothed) and it is difficult to reach extremally-located targets. This smoothness is reflected in a slow falloff for the autocorrelation function (ACF) of the cursor position. The bottom plot depicts a matched trial with a pre-computed decoder. The user’s better control derives from a decoder with comparable local smoothness, but more responsiveness. Empirically the ACF has correspondingly sharper falloff.

The details of the single-trial data shown in

For each SNR case, motor-imitation decoder curves are depicted in blue and the pre-computed decoder overlays it in red. Better controllability would be expected to be reflected in faster ACF falloff. In almost every case, falloff was faster for the cursor control using the pre-computed decoder (SNR-case 2 did not follow this trend as reliably—see main text for discussion).

The quantity compared is the number of targets per time and and significance is tested with one-sided unpaired t-tests (

All three SNR cases were designed with the expectation that user “motor-imitation” behavior would consist of moving both hands together left or right to control a horizontal cursor (i.e. moving both hands rightwards to get the cursor to move rightwards, etc.). However, subjects 2 and 3 predominantly used one hand during SNR-case 2 motor-imitation trials, and this happened to correspond to a good non-biomimetic scheme—the distance between hands served as a signal which strongly drove neural activity. Consequently, the motor-imitation decoder had decent signal as these subjects’ motor-imitation preferences coincidentally aligned with high SNR dimensions. This meant that the pre-computed decoder did not have enough room to do meaningfully better. This is reflected both in the insignificant differences in performance (

In addition to the performance differences between decoders, we examined how well the subjects learned to control the cursor via the decoders, essentially to verify the learnability of the pre-computed decoder. To demonstrate this, we provide plots comparing the parameters of the optimal encoding models against the estimated actual encoding model of the user (

Each plot compares, for each decoder, an estimate of the user’s learned encoder parameters against the optimal encoder parameters for a given subject, aggregated across signal cases. The optimal parameters for the two decoders are unrelated. A reference line is depicted on each plot and correlation coefficients are provided for each decoder (All correlations are significantly different from zero,

The key contributions of this paper are to establish co-adaptation methods as joint optimization problems over encoder-decoder pairs and to propose our approach for pre-computing an optimized decoder. For Kalman filter decoders and under our modeling assumptions, we showed that co-adaptation should not be able to do better than our approach in theory. To test the approach empirically, we also presented 1D OPS validation, in which our approach provided significantly better performance than the motor-imitation decoding approach. Instead of devising rules for adapting the decoder, we have presented an approach which characterizes what the user and decoder ought to do to complement one another, and we demonstrated that the user can match the optimal strategy in the 1D setting.

Our framework points to some qualitative insights into co-adaptation. In presenting our framework, we pointed out a non-obvious property of KF-decoder updates—updates to the decoder that are designed to be optimal for the current encoding model can allow for improvement if the user learns further (i.e. adapts the encoding model). This is effectively why co-adaptation can serve as a coordinate descent algorithm. Implicitly, this property, coupled with the recognition that the user encoding can change, has permitted various approaches that attempt to track these changing encoders, using Kalman Filtering (e.g. CLDA [

We can contextualize our pre-computation approach relative to standard co-adaptive methods, by viewing pre-computed decoders as an attempt to push towards a certain logical extreme, essentially with the decoder “adapting” entirely before any user learning takes place. Leveraging available information about the statistics of the recorded signals, the decoder can change pro-actively to prompt the user, who may not be learning very efficiently, to move in a direction in which better performance is expected. This is effectively what our algorithm was designed to do. A core intuition in assessing the relationship between these approaches is the observation that if one dimension of neural activity has less noise than another, optimal decoding will rely more heavily on the less noisy dimension. If noisy and non-noisy dimensions initially have non-zero weight, perhaps co-adaptation can be used to tune the relative strengths. However, this tuning can also be computed directly given estimates of signal and noise statistics. We believe our framework helps clarify the performance opportunities available when using fixed or adaptive decoders.

Our approach is broadly consistent with the literature on joint source-channel coding, but we are not aware of neural modeling being performed using our formulation. In the control literature, others have derived solutions to the related problem of optimizing time-varying observation models for the time-varying Kalman filter, but their result does not extend to the steady state case and concludes with a numerical analysis showing that their solution can be very suboptimal for our setting [

There also exists other research which has looked into control schemes for “body-machine interfaces”, for example using a sensor-wired glove with many degrees of freedom (DOF) to control a cursor-on-a-screen [

There are a few points to be aware of when interpreting the results of this paper. We optimize encoder-decoder pairs for the target tracing task, and we rely on similarity between tracing and the pinball tasks for the derived optimal solutions to hold in the pinball case. Additionally, we still have a free parameter in our optimization,

A limitation of the present study is that learning in an OPS setting may be different from learning in real BCI or across classes of BCI [

That said, both the decoding approach and the OPS can be modified to incorporate other features of the neural activity such as neural dynamics. While not conventionally modeled in BCI, recent research has revealed dynamical components to neural activity in motor cortices [

We lastly emphasize that our methods require estimation of the neural noise covariance, so a calibration session used to examine neural activity is of crucial importance. For real neural applications, the neural signal and noise components could be estimated using various neural factor analysis methods [

We have made an initial attempt to build learnable decoders using a new joint optimization framework. We have focused on the linear-Gaussian setting which is implicit when KF decoders are employed. There will certainly be model mismatch, and insofar as this model proves too simple, the way forward is to incorporate additional features of the encoding model and constraints on user learning. While our approach is very clear about the extent to which it relies upon the accuracy of these assumptions, it is worth emphasizing that co-adaptation approaches are equally (if implicitly) dependent on these assumptions. For example, the structure and timescale of decoder updates in co-adaptive approaches directly impact how well users can learn [

Additionally, while KF decoders are still widely used in BCI, future improvements may require nonlinear decoding methods, so we may wish to extend our framework to certain classes of nonlinear encoder-decoder pairs. For example we could consider Poisson-observation encoding with point-process decoders or otherwise parameterize the neural noise covariance to scale with the magnitude of the signal (i.e. signal-dependent noise as in [

In our OPS experiments the modeling assumptions hold by construction, so how significantly results will deviate from those of this paper in physiological experiments is open. Also, we admittedly focused on low dimensional control in order to gain conceptual ground, but this is simpler than high dimensional control. It remains to be explored how much more difficult user learning is in high dimensional settings, and what modifications will be required to assist users in learning optimal encodings. Additionally, more detailed characterization of the learning process may become relevant.

If applying our framework to human BCI, there are a number of practical opportunities. For example, it may be interesting to develop interfaces which indicate to the user how much control is available for some number of DOF and permit the user to select how to map control to the BCI effector. Users may have specific preferences to invert certain dimensions of control or request more precision on a certain effector DOF even at the expense of precision for other DOF. Such user-specific modifications are entirely amenable within our approach.

All simulations, optimizations, and experiments were conducted using custom code in Matlab (

Human protocols were approved by the Columbia University Morningside Institutional Review Board—all subjects read an IRB-approved informed consent document and provided verbal consent (Protocol Number: IRB-AAAM6407).

The Online prosthesis simulator (OPS) is an experimental platform that tracks overt movements of the user, uses the movements to simulate noisy neural data, and then decodes the synthetic data by a brain-computer interface (BCI) decoding algorithm [

The cartoon on the left depicts the OPS: (a) Actual overt movements of the user are detected. (b) Synthetic neural activity is generated in some fashion derived from the overt movements. (c) The decoder sees only the simulated activity and provides a decoded estimate of the cursor position. (d) The cursor is placed on the screen along with the target permitting visual feedback. The flow diagrams on the right compare how data is generated in a real BCI versus the OPS and the relationship between the variables. While there is no direct link relating intended kinematics and overt movement (or volition), these variables are conditionally related. If the user desires to produce a certain decoded kinematics and doing so requires some modulations of the neural activity which are controlled by the overt movement, then the user should modulate overt movement in a way that reflects their intended kinematics.

The user’s kinematic intention (_{t}) should correspond to an attempt to control the cursor towards a target presented on the screen. The overt arm movements of the user are used to simulate a population of synthetic, noisy neural units (_{t}). The encoding model (_{t} to _{t}. During the task, the neural activity is combined with the decoded estimate of the previous timestep (

In some high level sense, the user could explore the space of mappings from their behavior to visible feedback. While the user could not change how their overt movements map to neural activity, the user could change the relationship between “kinematic intention” and simulated neural activity—this is the relationship we call the encoding model. That is, changes in how the user made overt movements when intending to obtain certain decoded kinematics will cause changes in the encoding model (estimated parameters

We emphasize that these overt movements are not supposed to correspond to overt movements in a BCI setting; rather, these overt movements are simply supposed to provide arbitrary volitional signals which the BCI system will exploit to perform a task. BCI permits the use of any neural signals which are correlated with user volition—for example, even imagined speech has been used to control a 1D cursor [

To analyze optimal encoder-decoder pairs, we have been considering the objective function of a “tracing” task (a.k.a. pursuit tracking), in which the user has a time-varying kinematic intention and the decoder attempts to optimally estimate the kinematics. This objective is standard, physically meaningful, and the methods developed from it yield smooth priors. We validated our methods using the OPS on a “pinball” task (a.k.a. point-to-point reaching) where the user controls a smoothly moving cursor towards targets, as this is more realistic for applications.

In this work, we present the results of a 1D pinball task. We present data from 4 subjects (none of whom are authors of the study or otherwise directly affiliated with this project). Subjects were explained the rules of the “game” they were to play but were not described details of simulated neural activity (some had performed prototype versions of the experiment previous to participation in the experiments reported here). Before beginning the task, we collected calibration data for each user—the user was instructed to move around freely in an unstructured way. During this time, the user saw real-time video (RGB image frames) of themselves with skeletal tracking superimposed, in order to see how well the Kinect tracking worked. This unstructured phase was to get an estimate of the empirical neural activity covariance (i.e. estimated as the sample covariance of the neural responses).

The subjects participated in one session for each of two decoders (“motor-imitation” and “pre-computed”, described more below) in each of three “signal cases”. Signal cases consisted of specifications of how strongly the simulated neural channels were driven by different tracked body movements of the user (described more below). Between each of these six conditions, subjects were given a break until they felt ready for the next. For each signal case, the two decoder sessions were presented with the motor-imitation trial first, followed by a trial where the cursor was controlled by the pre-computed decoder. These trials were independent and the decoders were unrelated so the order mattered only so the control strategy the user learned when using the pre-computed decoder would not affect the motor-imitation control scheme. For block-order validation see

Each session consisted of two key phases, “free exploration” and “testing.” During free exploration, the user controlled a cursor on a screen via closed-loop OPS control, and there was no target. During this time, the user was prompted to explore and try to learn as well as possible how to control the cursor—this phase gave the subject an opportunity to learn, and nothing about the decoder was changed. Exploration rapidly provided information about how the user’s intentions relate to the movement of the cursor. The user can adjust their control strategy, in turn, adjusting the tuning of the simulated neurons—this process constitutes learning in this task. After a few minutes of exploration, a target appeared and the user was prompted verbally to move to the testing phase when ready by holding the cursor at the target for a few seconds (subject 1 moved directly to the testing phase without having to hold a target first, and this subject reported having had adequate time in free exploration). During testing, a single target at a time was presented, surrounded by a visible halo. The user attempted to control their cursor to the target and attempted to “acquire” the target by holding the cursor in the halo-region (holding is indicated by a color change of the region inside the halo). After a short period of time, if the cursor is held within the halo (≈1s), the target is acquired, disappears, and a new target is randomly placed at a new location. Targets that were not acquired after ≈15s were treated as misses, and a new target replaced them. The subject was instructed to acquire as many targets as possible in the ≈3min duration of the testing phase. Before beginning the task, the entire trial structure was verbally described to each subject, and verbal reminders of trial-stage (at transitions) were provided by the experimenter. A video depicting the experimental procedures is included as

The two decoders tested were selected either by “motor-imitation” initialization or by pre-computing the optimal encoder-decoder pair and presenting the optimal decoder. Trials with “motor-imitation” decoders began with an extra phase (at the beginning of the free exploration phase), during which a target moved smoothly on the screen. The user was instructed to “trace” the target with their hand movements, thereby providing the “native” control mapping of making rightwards and leftwards movements of their hands to control a cursor along the horizontal axis (without visual feedback, i.e. open-loop). This OPS phase provided supervised data which permitted estimation (by recursive least squares—an online algorithm for linear regression) of a motor-imitation encoding model, a control scheme which is as close as possible to user default control. The motor-imitation decoder refers to the optimal decoder corresponding to this encoding model, which can be computed directly from the estimated encoding model. Alternatively, for the pre-computed decoder trials, we used covariance information determined from the calibration data to estimate an optimal encoder-decoder pair according to our joint optimization approach presented in this paper, and we presented the pre-computed optimal decoder to the user. This second approach does not require labeled data; rather it merely optimizes the encoder-decoder pair such that good task performance results after the user learns how to use the scheme. Selection of the decoder is unsupervised, and this approach relies on the user to adapt to match the fixed decoder.

We note that

Using the OPS, we are able to arbitrarily specify mappings of user movements to the simulated neural data as well as setting noise levels across channels—this corresponds to the blue box of the OPS flow diagram in

We compared all subjects on all three cases of signal-noise structure using both motor-imitation initialized decoders and pre-computed decoders. The above cases were selected with the intent of highlighting differences between naïve, motor-imitation schemes and schemes which permit better performance by coding in dimensions that are not natively leveraged. The intention was to see if optimization of the decoder yielded performance differences and how learnable the schemes were.

In the results, we made use of estimated user encoding models, comparing them with optimal encoding models. To estimate the user encoding models during the pinball task, we simply performed linear regression from target positions to neural activity during the pinball phase. This assumes that the user is attempting to control the cursor such that it moves directly towards the target.

(PDF)

Side-by-side video of tracked user pose along with cursor control task. For high SNR, user initializes the decoder by motor-imitation, freely explores to become comfortable with the scheme, and then proceeds to the testing phase. The data used in the video was collected for demonstration purposes only.

(MOV)

Two additional subjects were run for alternating blocks of testing with the motor-imitation and pre-computed decoder on signal case 1 (three tests blocks using each decoder per subject). The initialization for the motor-imitation decoder was run once at the beginning. Between decoder switches, subjects were given short exploration phases to refamiliarize themselves before testing. One subject began with the motor-imitation decoder, and the other began with the pre-computed decoder. Performance across subjects and blocks was significantly better using the pre-computed decoder (using a one-sided t-test on targets/time across repeats and subjects). Across subjects and repeats, no statistically significant linear trend was found for performance with the motor-imitation decoder. A weak trend was found for the pre-computed decoder across repeats (i.e. for these subjects there was some evidence that subjects continued to improve when using the pre-computed decoder

(TIFF)

Earlier versions of related experiments were assisted by Antony Qian. Thanks to Hagai Lalazar for comments on a draft of the manuscript. Also, thanks to Conor Dempsey, Lars Buesing, Eftychios Pnevmatikakis, Grace Lindsay as well as other members of the Center for Theoretical Neuroscience at Columbia for helpful discussions.