Mixture Of Experts

What is a mixture of experts?

An AI architecture where only a subset of parameters, such as 3-7%, are activated at any given time.