Three ways of justifying/arguing for research choices (e.g., hyperparameter values, thresholds, sample sizes):
- Principled approach — “principled” means the same as well-established, as in: robust, reliable, good. For example, in calculating a sample size, there is a formula. Therefore, when you do a study a calculate the sample size using this formulate, you’re applying a principled approach in
- Data-driven approach — this is a type of principled approach, in which you determine the correct option from the data. A classic example is clustering in which you test several cluster numbers and plot them (or calculate some metric) in order to determine what number is “optimal”.
- Heuristic approach — this is based on a rule of thumb or general convention. For example, “minimum cell size == 5” or “minimum sample size == 30” are rules of thumb; you can see a lot of defaults used in studies based on other studies. However, a heuristic might not be principled – for example, “n = 30” is based on convenience in use that predates modern statistical software.