Model Comparison and Selection

Understand how to compare models, choose between them, and balance fit, complexity, and decision relevance.
Tip

What you’ll build today: a practical framework for comparing models and selecting the one that best supports decisions—not just the one that fits best.

Learning Objectives

By the end of this lesson, you will be able to:

  • Explain why multiple models can fit the same data
  • Understand criteria for model comparison (AIC, BIC, likelihood)
  • Balance model fit vs complexity
  • Choose models based on decision relevance

Key Ideas

In pharmacometrics, it is common to have multiple models that fit the data reasonably well.

This raises an important question:

Which model should we use?

Model selection is not just about fit—it is about:

  • predictive performance
  • interpretability
  • decision relevance

Why This Lesson Matters

Different models can:

  • produce similar fits
  • but lead to different predictions

This means:

Model choice can directly impact clinical and regulatory decisions.

Worked Example: Two Competing Models

Imagine:

  • Model A → simpler and easier to interpret
  • Model B → more flexible but more complex

Both explain the observed data.

But they may behave differently when predicting new scenarios.

Likelihood-Based Comparison

Models can be compared using likelihood:

\[ L(\theta) = P(\text{data} \mid \theta) \]

Higher likelihood → better fit

But:

  • more complex models often have higher likelihood

Objective Function Values (OFV)

In pharmacometrics, likelihood is often reported as an objective function value (OFV):

\[ OFV = -2\log L \]

where:

  • \(L\) is the likelihood
  • lower OFV indicates better agreement between the model and the data

Conceptually:

Higher Likelihood
↓
Lower OFV
↓
Better Fit

This is the quantity commonly reported by software such as NONMEM and nlmixr2.

Comparing Nested Models

When one model is a modification of another, models can be compared using the change in objective function value:

\[ \Delta OFV = OFV_{old} - OFV_{new} \]

Interpretation:

  • Positive ΔOFV indicates improved fit
  • Larger ΔOFV indicates stronger evidence that the new model better explains the data

For example:

Base Model
OFV = 1250

New Model
OFV = 1240

ΔOFV = 10

The new model provides a better fit because the OFV decreased.

Information Criteria

To balance fit and complexity:

AIC (Akaike Information Criterion)

\[ AIC = -2 \log L + 2k \]

BIC (Bayesian Information Criterion)

\[ BIC = -2 \log L + k \log n \]

Where:

  • \(k\) = number of parameters
  • \(n\) = number of observations

Interpretation

  • Lower AIC/BIC → stronger statistical support among compared models
  • Penalizes overly complex models

Model Selection Is Sequential

Model comparison is rarely based on a single number.

Typical workflow:

  1. Check convergence and stability
  2. Inspect diagnostics
  3. Compare fit statistics (OFV, AIC, BIC)
  4. Evaluate predictive performance
  5. Ask whether the model supports the decision

Each step eliminates unsuitable models.

Insight

The best model is not the one that fits best—it is the one that balances fit and simplicity.

Note

Overfitting improves fit but harms prediction.

Beyond Statistics: Decision Relevance

A model should be selected based on:

  • ability to answer the scientific question
  • predictive performance
  • robustness

Example:

Model A: - slightly higher AIC
- stable predictions
- interpretable parameters

Model B: - lower AIC
- unstable extrapolation

👉 Model A may still be preferred.

Strategies

  • Compare models using multiple criteria
  • Consider simplicity and interpretability
  • Focus on prediction and decision impact
  • Validate models using diagnostics

Common Mistakes

  • Choosing the model with lowest AIC blindly
  • Ignoring biological plausibility
  • Overfitting noise
  • Ignoring predictive performance

Practice Problems

  1. Why can multiple models fit the same data?
  2. What is the purpose of AIC?
  3. Why is the lowest AIC not always the best model?
  1. Because different structures can explain the same observations
  2. To balance model fit and complexity
  3. Because it may still overfit or lack interpretability

Summary

Model selection:

  • compares competing explanations
  • balances fit and complexity
  • focuses on decision relevance

The goal is not the best-fitting model, but the most useful model.

  • Fit is necessary, not sufficient
  • Use AIC/BIC as guides, not rules
  • Prefer simpler models when possible
  • Always consider the decision context