Let’s start from evaluation. When have a computer that does something, we want to see if it does it well. So, this (“seeing if it does well”) is called evaluation.
Evaluation is at the core of pretty much every computational technique, such as algorithm, model, system, device, etc. We want these things to work and improve people’s lives.
And that needs to be shown through evaluation.
So, getting back to expert evaluation.
Expert is someone who has specific expertise to evaluate.
For example, you develop a translation system that translates from Language A to Language B; yet, you yourself don’t speak either language. You are not equipped to evaluate this system. Instead, you need someone who knows both Language A and Language B.
Some tasks are “universal” and can be evaluated by anyone. For example, “tell us if this image has a picture of a person.” This is a generic human intelligence task (HIT) that does not require expert evaluators.
Identifying when you need expert evaluators and when not is a vital research skill.
Also, the idea is connected to sampling: for example, when you do a survey, you are interested in what a population (=group of people) think. Can your population include anyone or does it include specific people? If “specific people”, this idea is similar to experts—you have to be mindful that you reach such people.