Exploration and Exploitation

In the previous post I discussed the "neuroeconomics of movement", which is an approach to the study of motor control that assumes the nervous system chooses motor patterns by computing their estimated costs and benefits. One aspect of this computation that I didn't mention, but which was mentioned by my friend Eric Kruger in a comment, is the trade-off between exploration and exploitation. I have been meaning to talk about this trade-off for a long time, and Eric's comment prompted me to write about it, as it has a lot of relevance for some topics I discuss frequently, such as the role of play in motor learning.

The tradeoff

The concept of the exploration-exploitation tradeoff originated in computer science, and was later applied in many other fields seeking to understand complex decision-making by intelligent systems, such as organisms and businesses.

The basic dilemma posed by the trade-off is this: in trying to solve a problem, should you rely on a well-known method that delivers decent results with high confidence (exploitation), or try an untested method that might work much better, or lead to the discovery of useful information (exploration)? By exploiting a known resource you are assured a certain level of performance, but you are also denied the possibility of improvement that might come from exploring different options. On the other hand, exploration brings risk and uncertainty, but also the possibility of learning something new and useful that may be exploited in the future.

In the context of motor control, exploitation refers to reliance on familiar, habitual movement patterns. These are the strategies the nervous system has previously determined to be efficient for a given task. Exploitation is about perfecting and utilizing these known patterns to ensure consistent, reliable outcomes. It's following a well-trodden path or staying in a groove.

Exploration involves getting off the path and into uncharted territory. It's about varying movements, trying new strategies, and experimenting with different techniques. While this might introduce temporary inefficiency or awkwardness, it is required for discovering better movement patterns that might eventually be exploited as new habits.

The landscape analogy

We can add some nuance to these distinctions by imagining that the search for better solutions to movement problems is like exploring a vast landscape of different options. Solutions are represented by peaks in the landscape, and non-solutions are represented by valleys. The highest peak is the optimal solution.

Exploitation means finding a peak and then simply moving upward to the top. This strategy will eventually bring you to the top of the local area, but there may be far higher peaks on more distant parts of the landscape. To find them, you need to explore by descending from your local peak, and maybe even walking through a few valleys before finding the base of the highest mountain.

Exploration/exploitation and skill level

The preference for exploration or exploitation should depend on your level of skill and education. In general, early learners should engage in lots of exploration, so they don't lock themselves into suboptimal habits. But as they get better at what they are doing and become reasonably assured that their problem-solving methods are nearly optimized, they should spend less time exploring and more time exploiting what they know.

We can use this metaphor to understand how a golfer makes swing changes. A beginner will need to engage in significant exploration to find a good technique, because the first swing attempts will probably be pretty awkward. For example, a novice may find it easier to make solid contact by taking a short backswing and