Motor Control and Neural Networks: What the Brain's Movement Circuits Teach Us About Building Embodied AI

What the brain's movement circuits teach us about building embodied AI

Large language models are the most visible neural networks today, so it is easy to forget that most of what a brain does is not language. It moves a body through a noisy, delayed, unforgiving physical world. That problem has a different shape from next-token prediction, and the brain solves it with a different set of specialized circuits.

This post walks through the motor system region by region, maps each region to a concrete neural network family, shows how the pieces connect, and lays out how you would train them. Equations are written in LaTeX; the diagrams are Mermaid.

TL;DR

• Motor control is continuous, noisy, real-time and closed-loop, with sensory feedback arriving 50 to 150+ ms late. Language is discrete, offline and open-loop. The architectures differ because the problems differ.
• A useful frame (Doya, 1999) assigns the three big learning systems to three learning types: cortex to self-supervised learning, cerebellum to supervised learning from error, basal ganglia to reinforcement learning. An LLM is essentially the cortex-like piece only.
• Region to network, in one line each: M1 is a recurrent policy, premotor/SMA is a hierarchical planner, the cerebellum is a forward model with a fast error-driven readout, the basal ganglia are an actor-critic gate, the thalamus is a router, and the spinal cord is a fast low-level controller.
• The modules run at different clock rates and talk through compact latent vectors, not raw signals.
• Train bottom-up: world model first (no reward needed), then imitation, then RL against the world model, then the planner, then keep learning from deployment data.

1. Why motor control is not language modeling

The last row deserves a comment. Reaching for a cup can be done with infinitely many joint trajectories (Bernstein's "degrees of freedom problem"). The brain does not pick the single optimal one. It picks one that is good enough and corrects only the deviations that matter, which is exactly what optimal feedback control predicts (section 3).

2. The big picture: three learners and a loop

Doya (1999, 2000) proposed that the cerebellum, basal ganglia and cerebral cortex are specialized for three learning paradigms:

This is a functional hypothesis, not a wiring diagram, and later work has complicated it (the cerebellum also contributes to cognition, dopamine does more than carry a TD error). But it is a very productive design heuristic: pick the learning rule that matches the signal available at that level.

Here is the motor loop that connects them:

Reading the loop: the planner proposes, the basal ganglia decide whether and which, the thalamus routes the decision back, M1 turns it into commands, the spinal cord and body execute, and the cerebellum compares what was predicted with what happened and feeds the difference back.

3. The problem being solved

3.1 Dynamics, noise and delay

Model the body plus environment as a noisy dynamical system with delayed, noisy observation:

$$ x_{t+1} = f(x_t, u_t) + \epsilon_t, \qquad y_t = g(x_{t-\tau}) + \eta_t $$

where $u_t$ is the motor command, $\tau$ is the sensory delay, and the command noise $\epsilon_t$ grows with $|u_t|$ (signal-dependent noise). If you act only on $y_t$, you are always reacting to the world as it was $\tau$ milliseconds ago, and a naive feedback loop with that much delay oscillates or goes unstable.

3.2 The brain's fix: predict, then correct

The standard fix in control theory is a state estimator. In its simplest form (a Kalman-style filter):

$$ \hat{x}_{t+1|t} = f(\hat{x}_t, u_t) \qquad \text{(predict using the efference copy)} $$

$$ \hat{x}{t+1} = \hat{x}{t+1|t} + K,\big(y_{t+1} - g(\hat{x}_{t+1|t})\big) \qquad \text{(correct with delayed sensing)} $$

The prediction step is a forward model: given my current state estimate and the command I just sent (the efference copy), what will I sense next? The correction step uses the prediction error. The cerebellum is the leading candidate for implementing the forward model; the idea that it acts as a Smith predictor that compensates for feedback delay goes back to Miall et al. (1993).

3.3 Optimal feedback control and the minimal intervention principle

Todorov and Jordan (2002) showed that many properties of human movement follow from optimal feedback control: minimize a cost that trades task error against effort, with state estimation and control working together (the LQG setting: a linear-quadratic regulator plus a Kalman filter). The key behavioral prediction is the minimal intervention principle: the controller corrects only deviations that interfere with the task and ignores task-irrelevant variability. This explains why repeated reaches vary in irrelevant dimensions but stay tight in the dimensions that matter.

Takeaway for ML: a good motor policy is not "reproduce this trajectory." It is a feedback law, a function from estimated state and goal to command, evaluated every tick.

4. Module by module

4.1 Primary motor cortex (M1): a recurrent policy that behaves like a dynamical system

Biology. M1 sends commands down the corticospinal tract. Classic work tried to decode movement direction from single neurons. Churchland, Shenoy and colleagues (2012) found something different at the population level: during reaching, neural population activity shows consistent rotational dynamics, and the preparatory activity before movement sets the initial condition from which the dynamics unfold. In this view, M1 is less a table of "muscle commands" and more a dynamical system:

$$ \dot{r} = F(r) + B,u(t), \qquad \text{muscle activity} = C,r $$

with $r$ the population state, $u(t)$ inputs from premotor areas and sensory feedback, and a low-dimensional readout $C$ to muscles. Gallego et al. (2017, 2020) added that this activity lives on a low-dimensional neural manifold that is stable across days and similar across animals doing the same task.

ML analogue. A recurrent network that maps (state estimate, goal latent) to joint targets or muscle-like activations. Sussillo et al. (2015) trained an RNN to produce muscle activity for reaches and found that with regularization that favors smooth, input-insensitive internal dynamics, the trained network developed rotational population dynamics resembling monkey M1.

Architecture options.

• GRU or LSTM: the safe default, constant cost per step.
• Linear-recurrent and state-space models (LRU, S4, Mamba-style): attractive for real-time control because inference is a cheap recurrent update with fixed latency.
• Small transformer over a short history window: works well when you need to look back over several observations; keep the window short for latency.

Training.

1. Behavior cloning on demonstrations: regress actions from observation and goal. Add noise injection to the states during data collection (DART) so the policy sees recoveries, not just perfect trajectories.
2. Add regularizers that mirror the biological findings: penalize action jerk, penalize sensitivity of the hidden state to input noise, and optionally pass the hidden state through a low-dimensional bottleneck to encourage a manifold.
3. Fine-tune with RL once the policy is competent (section 6).

Common failure. Plain MSE behavior cloning averages multiple valid modes. If a demonstrator goes left or right around an obstacle, the average goes straight into it. That problem is handled one level up, in premotor/SMA.

4.2 Premotor cortex and SMA: planning, sequencing, chunking

Biology. Dorsal premotor cortex shows preparatory activity before movement; in the dynamical-systems view it sets M1's initial state. The supplementary motor area and pre-SMA are involved in sequencing and in organizing learned sequences of movements.

ML analogue. A model that turns a goal and a scene into a short structured plan, as sub-goals or chunks of future actions. Several families map well:

• Action chunking (ACT, Zhao et al. 2023). Predict the next $k$ actions at once instead of one. Executed with temporal ensembling (average overlapping predictions), this reduces compounding error and smooths motion. It is the clearest ML counterpart of "plan a chunk, then execute it."
• Diffusion Policy (Chi et al. 2023). Model the distribution over action sequences with a diffusion model. This directly fixes the mode-averaging problem above, because it can commit to one mode.
• Decision Transformer (Chen et al. 2021). Treat control as sequence modeling conditioned on a desired return or goal.
• Hierarchical RL. Options (Sutton, Precup, Singh 1999) define temporally extended skills with an initiation set, an intra-option policy and a termination condition. Goal-conditioned policies with hindsight relabeling (HER, Andrychowicz et al. 2017) let a planner issue goals that a lower policy learns to reach.
• Language-model planners such as SayCan (2022) sit here too: the LLM proposes skills and a learned affordance model filters them for what is feasible.

What the planner outputs. Not torques. It outputs a goal embedding or a skill identifier plus parameters, updated slowly (roughly 1 to 10 Hz).

4.3 Cerebellum: a forward model and a fast error-driven learner

The cerebellum holds roughly 80% of the brain's neurons, mostly tiny granule cells, in a strikingly regular repeating circuit. That regularity is a hint: it is a massively parallel, uniform computation.

The Marr-Albus-Ito idea. Marr (1969) and Albus (1971) proposed that the circuit does pattern separation followed by supervised readout:

1. Mossy fibres carry a modest number of signals (sensory context, efference copy).
2. Granule cells expand these into a vastly larger, sparse code. Expansion plus sparsity makes similar inputs more separable, so a linear readout can solve problems that were not linearly separable in the input space.
3. Purkinje cells compute a weighted sum of the parallel fibres. Climbing fibres from the inferior olive deliver a sparse error signal, and Ito's work showed that coincident climbing-fibre activity depresses the parallel-fibre synapses (long-term depression).

That is the delta rule on a random-feature layer:

$$ \Delta w_i = -\eta,\big(\hat{y} - y\big),\phi_i(x) $$

In ML terms: a fixed (or slowly learned) wide sparse random-feature layer with a trainable linear readout, trained online by gradient descent or recursive least squares on prediction error. Because the readout is linear in fixed features, learning is convex and can be extremely fast and stable, which is exactly what you want for on-the-fly adaptation.

Adaptive filter view. Fujita (1982), and later Dean and Porrill, modelled the cerebellum as an adaptive filter: a bank of basis filters (a tapped delay line) feeding an adaptive linear combination. This connects directly to adaptive control and to learning temporal structure, not just static mappings.

Forward and inverse models.

• Forward model: $(x_t, u_t) \mapsto \hat{x}_{t+1}$. Used for state estimation, delay compensation, and "mental rehearsal."
• Inverse model: $(x_t, x^{\ast}_{t+1}) \mapsto u_t$. Used to compute the command that achieves a desired next state.
• Feedback-error learning (Kawato 1987). A crude fixed feedback controller produces a correction command whenever the movement deviates. Use that correction as the training error for the inverse model. As the inverse model improves, the feedback correction shrinks toward zero. This is an elegant way to learn a controller with no explicit teacher.
• MOSAIC (Wolpert and Kawato 1998). Maintain several paired forward/inverse models. A "responsibility" signal, computed from how well each forward model predicts, selects or blends them. It is a mixture of experts for contexts such as lifting a heavy versus a light object.

Modern ML analogues.

• World models with a recurrent latent state, such as the RSSM in Dreamer (Hafner et al. 2020, 2023).
• Rapid Motor Adaptation (Kumar et al. 2021): an adaptation module infers hidden environment parameters (payload, friction, terrain) from a short history of states and actions, and conditions the policy on them. Functionally this is cerebellum-like contextual adaptation.
• Online ridge regression or RLS on top of frozen features, for fast per-session calibration.

Training the cerebellum module. Self-supervised, no reward needed:

$$ \mathcal{L}{\text{world}} = \sum{h=1}^{H} \big| \hat{x}{t+h} - x{t+h} \big|^2 $$

with multi-step rollouts ($H > 1$), because one-step accuracy hides compounding error.

4.4 Basal ganglia: action selection and a reinforcement learner

Biology. The striatum receives cortical input and projects through two main routes to the output nuclei (GPi/SNr), which tonically inhibit the thalamus.

• Direct pathway ("go") inhibits the output nuclei, releasing the brake on the thalamus and letting the selected action through.
• Indirect pathway ("no-go") strengthens the brake on competing actions.
• Hyperdirect pathway (cortex to subthalamic nucleus) provides a fast global stop signal.

The net effect is selection among competing actions: release the brake on one channel, keep it on the rest (Mink 1996; Redgrave et al. 1999; Gurney et al. 2001).

Dopamine as a TD error. Schultz, Dayan and Montague (1997) showed that midbrain dopamine neurons behave like the temporal-difference error of reinforcement learning:

$$ \delta_t = r_t + \gamma,V(s_{t+1}) - V(s_t) $$

Positive $\delta$ strengthens the corticostriatal synapses that produced the action; negative $\delta$ weakens them. A popular reading maps the dorsal striatum to the actor and the ventral striatum to the critic. More recent work (Dabney et al. 2020) suggests dopamine neurons carry something closer to a distribution over returns, which aligns with distributional RL.

ML analogue. An actor-critic pair:

• Critic: a value network $V_\phi(s)$ trained by TD learning.
• Actor: a policy or a categorical gate over skills trained with policy gradient (PPO) or maximum-entropy methods (SAC).
• The gate decides which skill is active and when to switch. In options terms, it learns the termination function $\beta(s)$. Switching too eagerly causes thrashing; too lazily causes sluggishness.

Two further ideas are worth borrowing:

1. Model-free versus model-based arbitration (Daw et al. 2005). Early in learning, behavior is goal-directed and leans on a model; with practice it becomes habitual, cheap and fast. The ML version is distilling slow planning (search in the world model) into a fast policy.
2. The hyperdirect stop as a safety layer. A fast, simple override that can veto any selected action maps naturally to control barrier functions, action masking, or a shield wrapped around the learned policy.

4.5 Thalamus: a gate and router, not a mere relay

The thalamus sits in the middle of the loop. Older textbooks call it a relay; more recent work shows that higher-order thalamic nuclei participate in cortico-cortical communication, and that thalamic circuits (including the reticular nucleus) gate which signals get through. For motor control it connects the basal ganglia and cerebellum outputs back to cortex.

ML analogue. Anything that conditionally routes information:

• Cross-attention from a goal or context vector.
• FiLM modulation: $h' = \gamma(z) \odot h + \beta(z)$, where $z$ is the context.
• Mixture-of-experts routers and gated linear units.
• Hypernetworks that generate weights from a context.

In a practical stack, "the thalamus" is simply the interface layer where the planner latent, the cerebellar correction and the gate decision are combined and fed to the policy. You rarely train it separately; it is learned end-to-end with whatever it conditions.

4.6 Brainstem and spinal cord: fast, cheap, reliable

Biology. The spinal cord is far from a passive cable. It contains reflex arcs (the stretch reflex acts within tens of milliseconds) and central pattern generators (CPGs): oscillatory circuits that produce rhythmic output for walking and swimming with no input from the cortex. The classic half-center model dates to Brown (1911), and Bizzi and others proposed that the spinal cord stores a small library of motor primitives (synergies, force fields) that higher centers combine.

ML analogue.

• Low-level controllers: PD or impedance control tracking joint targets. Empirically, letting an RL policy output joint-position targets for a PD loop works far better than outputting raw torques (Peng and van de Panne 2017), because the PD loop does the high-rate stabilization.
• Pattern generators: coupled oscillators (Matsuoka, Hopf), neural ODEs, or CPG layers that the policy modulates with a few parameters (Ijspeert 2008).
• Learned locomotion: legged robots trained with RL in massively parallel simulation (Hwangbo et al. 2019; Lee et al. 2020), using privileged information in a teacher and distilling to a student that only sees onboard sensors.

Engineering advice. If a classical controller (MPC, whole-body control, simple PD) does the job, keep it classical. Spend learning where classical methods struggle: perception-rich decisions, contact-rich manipulation, adaptation.

5. How the modules interlink

5.1 Different clock rates, compact interfaces

Design rules that fall out of this:

• Latest-value buffers. Each layer reads the most recent output of the layer above and holds it until a new one arrives. If the planner is late, the policy keeps executing the last goal.
• Timestamps everywhere. Sensory delay compensation only works if you know the delay.
• A reflex layer that can always override. The low-level controller should be able to stop or stabilize regardless of what the learned layers say.
• Watchdogs. If a slow module stops responding, fall back to a safe behavior.

5.2 One control cycle

5.3 Interface contracts

5.4 Where gradients flow

Training everything end-to-end through all of these boundaries is fragile. A pragmatic arrangement:

Freeze modules that are working, and unfreeze them one at a time for joint fine-tuning.

6. How to train it

Stage 1: world model (cerebellum)

• Data: any logged (state, action, next state) sequences. Random exploration data is fine; it needs no reward.
• Loss: multi-step prediction error, plus a KL or reconstruction term if using a latent RSSM.
• Pitfalls: one-step accuracy is misleading. Evaluate error as a function of horizon. Use an ensemble to get uncertainty; use short imagined rollouts and branch from real states (the MBPO idea) to limit compounding error.

Stage 2: motor policy (M1)

• Data: demonstrations from teleoperation, scripted controllers or a privileged teacher.
• Loss: regression on actions (Huber or MSE) for unimodal tasks; diffusion or mixture-density heads when demonstrations are multimodal.
• Tricks: noise injection during collection, DAgger-style relabeling when you can query an expert, jerk penalty, history stacking.

Stage 3: critic and gate (basal ganglia)

• Data: imagined rollouts in the world model plus a smaller amount of real experience.
• Loss: TD error for the critic; PPO or SAC objective for the actor; entropy bonus to keep exploring.
• Gate specifics: add a switching cost so the gate does not thrash between skills, and learn termination explicitly.
• Offline option: if you only have a fixed dataset, use offline RL (IQL, CQL) to avoid exploiting world-model errors.

Stage 4: planner (premotor/SMA, or an LLM)

• Data: task descriptions and successful skill sequences; hindsight relabeling to turn failures into valid goal-reaching examples.
• Loss: imitation on chunks, plus hierarchical RL where the reward is task success and the action space is skills or goals.
• Note: train this last. It needs competent skills to plan over, otherwise it learns to ask for things the lower layers cannot do.

Stage 5: joint fine-tuning and continued learning

• Unfreeze one module at a time with a small learning rate.
• Keep a replay buffer of deployment data and feed it back into the world model, which is the loop in the diagram.
• To limit forgetting, rely on modularity (the world model adapts online, the policy changes slowly) and replay of old data.

Sim-to-real

• Domain randomization (Tobin et al. 2017): randomize mass, friction, latency, sensor noise so that the real world looks like one more sample.
• Teacher-student distillation: train a teacher with privileged simulator state, then distill to a student that sees only deployable sensors (the recipe in Lee et al. 2020).
• Online adaptation: an RMA-style context encoder (the cerebellum analogue) closes the remaining gap.

What to measure at each level

7. A reference skeleton in code

This is a schematic PyTorch skeleton to show the shape of the stack, not a tested implementation.

import torch
import torch.nn as nn
import torch.nn.functional as F


class CerebellumLike(nn.Module):
    """Fixed random expansion -> k-winner-take-all -> trainable linear readout."""
    def __init__(self, d_in, d_out, d_expand=8192, k=128):
        super().__init__()
        self.register_buffer("W", torch.randn(d_in, d_expand) / d_in ** 0.5)
        self.k = k
        self.readout = nn.Linear(d_expand, d_out)

    def features(self, x):
        h = x @ self.W                                   # mossy -> granule expansion
        thr = h.topk(self.k, dim=-1).values[..., -1:]    # sparsify: keep top-k
        return F.relu(h - thr)

    def forward(self, x):
        return self.readout(self.features(x))            # Purkinje-like readout


class ForwardModel(nn.Module):
    """Predicts next-state delta from (state, command): the efference-copy path."""
    def __init__(self, d_state, d_act):
        super().__init__()
        self.core = CerebellumLike(d_state + d_act, d_state)

    def forward(self, s, a):
        return s + self.core(torch.cat([s, a], dim=-1))


class MotorPolicy(nn.Module):
    """M1-like recurrent policy: (state estimate, goal latent, prediction error) -> joint targets."""
    def __init__(self, d_state, d_goal, d_act, d_h=256):
        super().__init__()
        self.rnn = nn.GRUCell(d_state + d_goal + d_state, d_h)
        self.head = nn.Linear(d_h, d_act)

    def forward(self, s_hat, goal, pred_err, h):
        h = self.rnn(torch.cat([s_hat, goal, pred_err], dim=-1), h)
        return torch.tanh(self.head(h)), h               # normalized joint-target offsets


class BasalGangliaGate(nn.Module):
    """Actor-critic over skills. The actor selects, the critic predicts value."""
    def __init__(self, d_state, d_goal, n_skills, d_h=256):
        super().__init__()
        self.actor = nn.Sequential(nn.Linear(d_state + d_goal, d_h), nn.ReLU(), nn.Linear(d_h, n_skills))
        self.critic = nn.Sequential(nn.Linear(d_state + d_goal, d_h), nn.ReLU(), nn.Linear(d_h, 1))

    def forward(self, s, goal):
        x = torch.cat([s, goal], dim=-1)
        return self.actor(x), self.critic(x).squeeze(-1)  # skill logits, value


def td_error(r, v, v_next, done, gamma=0.99):
    """The 'dopamine' signal."""
    return r + gamma * v_next * (1.0 - done) - v


def control_loop(env, planner, estimator, world, policy, pd, steps,
                 plan_every=20, low_per_tick=10):
    h = torch.zeros(1, 256)
    pred_next = None
    for t in range(steps):
        obs = env.observe()                                   # delayed, noisy
        if t % plan_every == 0:
            goal = planner(obs)                               # slow: LLM / VLM / HRL
        err = torch.zeros_like(obs) if pred_next is None else obs - pred_next
        s_hat = estimator(obs, err)                           # predict + correct
        a, h = policy(s_hat, goal, err, h)                    # fast: motor policy
        pred_next = world(s_hat, a).detach()                  # efference copy -> prediction
        q_target = env.joint_target(a)
        for _ in range(low_per_tick):                         # fastest: reflex layer
            env.apply(pd(q_target, env.q, env.qd))
        env.step()

Training the forward model online is one line, because the readout is linear on fixed features: loss = F.mse_loss(world(s, a), s_next), followed by an optimizer step (or an RLS update for even faster adaptation).

8. Where LLMs and vision-language-action models fit

A language or vision-language model is a good candidate for the planner slot: it holds world knowledge, parses instructions and decomposes tasks. Everything below it should stay non-linguistic.

Published robotics systems already follow this structure: RT-2 and OpenVLA tokenize actions inside a vision-language backbone; $\pi_0$ attaches a flow-matching action expert to a VLM; and dual-system designs such as Figure's Helix and NVIDIA's GR00T N1 explicitly split a slow reasoning system from a fast motor system. (Check each system's own documentation for current details, since this area changes quickly.)

The mapping also shows what is commonly thin or missing: an explicit world model with prediction-error feedback (cerebellum) and an explicit value-driven gate (basal ganglia). Adding those is one of the more promising directions.

9. Where the analogy breaks

It is worth being honest about the limits.

• Functional, not literal. The mapping says what each region seems to compute, not that a GRU is a cortex. Modern ML works fine without copying biology; treat this as a source of design hypotheses.
• The tidy triad is a simplification. The cerebellum also contributes to cognition and timing. Dopamine also relates to motivation, vigor and salience, not only prediction error. The cortex does supervised and reinforcement-like learning too.
• Learning rules. The brain appears to use local, largely backprop-free learning (whether it approximates backprop is debated; see Lillicrap et al. 2020). The ML stack above uses gradient descent.
• Priors from evolution and development. Infants learn to reach far faster than an RL agent, partly because the architecture, body and reflexes arrive pre-tuned. Sample-efficiency gaps are largely a gap in priors.
• The body computes too. Muscle elasticity and limb mechanics simplify control (morphological computation). A software-only model of the brain leaves out part of the solution.
• Energy. The brain does all this on roughly 20 W. That motivates sparse, event-driven and neuromorphic hardware, and is not something the architecture above addresses.

10. A concrete roadmap and experiments

Start small and let experiments test each brain-inspired claim.

Build order

1. A simulated 2-link arm (or a MuJoCo reacher) with an adjustable sensory delay and signal-dependent noise.
2. A feedback-only baseline policy. Watch it degrade as delay increases.
3. Add the forward model and state estimator. Show recovery under delay.
4. Add the recurrent policy trained by imitation, then RL fine-tuning.
5. Add skills, a gate with termination learning, and a simple planner.
6. Randomize dynamics, add an RMA-style context encoder, and test generalization.

Ablations that test the story

Open problems worth chasing

• Continual learning without forgetting, using modular structure and replay.
• Learning world models that stay accurate over long horizons.
• Gates that learn good termination and temporal abstraction from sparse reward.
• Tight latency guarantees for learned components on real hardware.
• Safe exploration: a learned, verifiable version of the hyperdirect "stop."

References

Concepts and works named above (look up each by author and year):

• Albus (1971), a theory of cerebellar function; Marr (1969), a theory of cerebellar cortex; Ito, work on cerebellar long-term depression.
• Andrychowicz et al. (2017), Hindsight Experience Replay.
• Bernstein (1967), the coordination and regulation of movements; Bizzi and colleagues on motor primitives.
• Chen et al. (2021), Decision Transformer.
• Chi et al. (2023), Diffusion Policy.
• Churchland et al. (2012), neural population dynamics during reaching.
• Dabney et al. (2020), a distributional code for value in dopamine-based reinforcement learning.
• Daw, Niv, Dayan (2005), uncertainty-based competition between prefrontal and dorsolateral striatal systems.
• Doya (1999, 2000), what are the computations of the cerebellum, the basal ganglia and the cerebral cortex.
• Fujita (1982), adaptive filter model of the cerebellum; Dean and Porrill, later developments.
• Gallego et al. (2017, 2020), neural manifolds for the control of movement.
• Gurney, Prescott, Redgrave (2001), a computational model of action selection in the basal ganglia.
• Hafner et al. (2020, 2023), Dreamer and DreamerV3.
• Hwangbo et al. (2019), learning agile and dynamic motor skills for legged robots; Lee et al. (2020), learning quadrupedal locomotion over challenging terrain.
• Ijspeert (2008), central pattern generators for locomotion control.
• Kawato (1987), feedback-error learning; Wolpert and Kawato (1998), MOSAIC multiple paired forward and inverse models.
• Kumar et al. (2021), RMA: Rapid Motor Adaptation.
• Lillicrap et al. (2020), backpropagation and the brain.
• Miall et al. (1993), the cerebellum as a Smith predictor.
• Mink (1996), the basal ganglia and focused selection of movement.
• Peng and van de Panne (2017), learning locomotion skills using deep RL: does the choice of action space matter.
• Redgrave, Prescott, Gurney (1999), the basal ganglia: a vertebrate solution to the selection problem.
• Schultz, Dayan, Montague (1997), a neural substrate of prediction and reward.
• Sussillo et al. (2015), a neural network that finds a naturalistic solution for the production of muscle activity.
• Sutton, Precup, Singh (1999), between MDPs and semi-MDPs (options).
• Tobin et al. (2017), domain randomization for sim-to-real transfer.
• Todorov and Jordan (2002), optimal feedback control as a theory of motor coordination.
• Zhao et al. (2023), learning fine-grained bimanual manipulation with low-cost hardware (ACT).
• Robotics systems: RT-2, OpenVLA, $\pi_0$, Helix, GR00T N1 (see each project's own documentation).