Why Model Evaluation Matters

Understand why pharmacometric models must be evaluated, what can go wrong, and how model assumptions impact decisions.
Tip

What you’ll build today: a clear understanding of why model evaluation is essential, what “good” and “bad” models look like, and how assumptions impact decisions.

Learning Objectives

By the end of this lesson, you will be able to:

  • Explain why model evaluation is necessary
  • Understand the role of assumptions in modeling
  • Recognize what can go wrong even when models “fit”
  • Connect model evaluation to decision-making

Key Ideas

A model is not useful just because it can be estimated.

It must be evaluated.

All models rely on assumptions:

  • structural assumptions
  • statistical assumptions
  • data assumptions

If those assumptions are wrong:

  • the model may still fit
  • but the conclusions may be incorrect

Insight: A good fit does not guarantee a good model.

Warning

It is possible for a model to describe observed data well and still fail when used for prediction or decision-making.

Why This Lesson Matters

Up to this point, you’ve learned:

  • how models are built
  • how variability is represented
  • how parameters are estimated

Now comes the critical question:

Can we trust the model?

This is where model evaluation comes in.

A Mental Model for Evaluation

Model evaluation asks progressively harder questions.

flowchart LR

FIT["Does it fit?"]

-->

INTERP["Does it make sense?"]

-->

PRED["Does it predict?"]

-->

DEC["Can we trust decisions?"]

A model should move successfully through all stages.

Good fit alone is not enough.

Worked Example: Good Fit, Wrong Model

Imagine:

  • Model A assumes linear elimination
  • Model B assumes nonlinear elimination

Inside the observed region:

  • both models appear acceptable

But beyond the observed data:

  • Model A predicts continued decline
  • Model B predicts changing behavior

👉 Similar fit does not guarantee similar decisions.

Expanding the Example

Evaluation asks a different question from estimation.

Estimation asks:

“Can the model explain observed data?”

Evaluation asks:

“Will the model remain reliable when used?”

A model can be:

  • descriptively adequate (fits observations)
  • predictively unreliable (fails outside observed conditions)

This often occurs because:

  • assumptions are incorrect
  • structure is misspecified
  • variability is poorly represented

Insight

Model evaluation is about understanding not just fit, but reliability.

Note

Always ask:

“Will this model still work outside the observed data?”

The following lessons introduce diagnostic tools used to answer that question.

Types of Assumptions

Structural Assumptions

  • compartment structure
  • functional forms

Statistical Assumptions

  • distribution of variability
  • residual error

Data Assumptions

  • measurement accuracy
  • sampling design

Each can fail independently.

Why This Matters for Decisions

Models are used for:

  • dose selection
  • trial design
  • regulatory justification

If a model is wrong:

  • doses may be incorrect
  • risks may be underestimated
  • decisions may fail

Strategies

  • Evaluate models from multiple perspectives
  • Focus on prediction, not just fit
  • Question assumptions explicitly
  • Compare alternative models

Common Mistakes

  • Trusting models solely based on fit
  • Ignoring assumptions
  • Over-relying on numerical output
  • Confusing fit with validity

Practice Problems

  1. Why is model evaluation necessary?
  2. Can a model fit data well and still be wrong?
  3. What types of assumptions do models rely on?
  1. Because fitting alone does not guarantee correctness or predictive ability
  2. Yes, if assumptions are incorrect
  3. Structural, statistical, and data assumptions

Summary

Model evaluation:

  • tests whether a model is reliable
  • identifies assumption failures
  • connects modeling to decision-making

A model is only useful if it remains reliable for the decision it supports.

  • Fit ≠ truth
  • Always question assumptions
  • Evaluate predictions, not just fit
  • Use multiple diagnostics
  • Models support decisions—not replace thinking