How variability shapes learning and generalization

The Role of Variability: From Playing Tennis to Language Learning

An example of visual data augmentation techniques used in machine learning that captures the main principle of variability effects: exposure to variation along nondiscriminatory dimensions (i.e., by rotating, changing color, or partially masking the target image) improves neural networks’ ability to generalize ( in this case – to identify a fox), but at the expense of slowing down initial learning. Humans show a similar effect: more variable input is more difficult to learn, but ultimately increases our ability to generalize learned knowledge to new contexts. This is because variability helps highlight which features of the category are actually relevant and which are not. Credit: Limor Raviv

Variability is key to learning new skills. Consider learning how to serve in tennis. Should you always practice serving from the exact same spot on the court and aiming for the exact same spot? Although practicing in changing conditions will be slower at first, it will likely make you a better tennis player in the end. Because variability leads to better generalization of what has been learned.

Chihuahuas and Great Danes

This principle is found in many areas, including language perception, grammar, and learning words and categories. For example, infants will have trouble learning the “dog” category if they are only exposed to Chihuahuas instead of many different types of dogs (Chihuahuas, Poodles, and Great Danes).

“There are over 10 different names for this basic principle,” says Limor Raviv from the MPI, lead researcher of the study published in Trends in the cognitive sciences. “Learning from less variable inputs is often rapid but may not be transferrable to new stimuli. But these important insights have not been brought together in a single theoretical framework, which obscures the bigger picture.”

To identify key patterns and understand the underlying principles of variability effects, Raviv and her colleagues reviewed over 150 studies on variability and generalization across fields including computer science, linguistics, categorization, motor learning, visual cognition, and formal education.

Mr Miyagi

The researchers discovered that there are at least four different types of variability across studies, such as: . “These four types of variability have never been directly compared, which means we don’t currently know which one is most effective for learning,” says Raviv.

The impact of the variability depends on whether it is relevant to the task or not (the color of the tennis court is arguably irrelevant to serving practice). But according to the “Mr. Miyagi Principle” (inspired by the classic 1984 film The Karate Kid), practicing seemingly unrelated skills (like waxing cars) can actually encourage learning of other skills (like martial arts).

The Role of Variability: From Playing Tennis to Language Learning

An example of the effect of more or less variation in learning to identify the letter “A”. Initial training items are shown in the center circle of each panel, and the color gradient symbolizes generalization performance: Greater accuracy and/or certainty in our generalization is represented by shades of yellow, while lower accuracy and/or certainty in our generalization is represented by shades of blue. Less variability during initial training (Panel A) may lead learners to form more conservative hypotheses about what the letter “A” may look like, leading to tighter generalization to less common occurrences of the letter “A”. More variable examples during initial training (Panel B) lead to broader hypotheses/categorizations and allow learners to more accurately and/or more confidently classify different instances of the letter ‘A’ that they encounter later. Credit: Limor Raviv

Competing theories

But why does variability affect learning and generalization? One theory is that more variable input can highlight which aspects of a task are relevant and which aren’t (color is useful for distinguishing between lemons and limes, but not for distinguishing between cars and trucks).

Another theory holds that greater variability leads to broader generalizations. This is because variability better represents the real world, including atypical examples (like Chihuahuas).

A third reason has to do with how memory works: when the training is variable, learners are forced to actively reconstruct their memories.

face recognition

“Understanding the effects of variability is important to literally every aspect of our daily lives. Aside from how we learn language, motor skills and categories, it even has an impact on our social life,” explains Raviv. “For example, facial recognition is affected by whether people grew up in a small community (less than 1,000 people) or in a larger community (more than 30,000 people). Exposure to fewer faces during childhood is associated with decreased facial memory.”

“We hope that this work will pique people’s curiosity and generate more work on the subject,” concludes Raviv. “Our work raises many unanswered questions. For example: is the relationship between variability and learning broadly similar across species, or are there species-specific adaptations? Can we find similar effects of variability beyond the brain, for example in the immune system.” ?”

The size of the community matters when people create a new language

More information:
Limor Raviv et al, How variability shapes learning and generalization, Trends in the cognitive sciences (2022). DOI: 10.1016/j.tics.2022.03.007

Provided by the Max Planck Society

Citation: How variability shape learning and generalization (2022, May 13), retrieved May 13, 2022 from

This document is protected by copyright. Except for fair trade for the purpose of private study or research, no part may be reproduced without written permission. The content is for informational purposes only.

Leave a Comment