Validity Threat Audit AI Prompt

Audit my study design for threats to internal and external validity.

Study description: {{study_description}}

Apply the four validity frameworks systematically.

1. Internal validity (did the treatment really cause the observed outcome?):
   Check for each threat:
   - History: did any external event occur during the study that could explain the outcome?
   - Maturation: could participants have changed naturally over the study period independent of the treatment?
   - Testing: could repeated measurement itself change participants' responses?
   - Instrumentation: did the measurement tools or procedures change during the study?
   - Regression to the mean: were extreme scorers selected? Their scores would likely move toward the mean naturally.
   - Selection bias: were treatment and control groups systematically different at baseline?
   - Attrition / mortality: did participants drop out differentially across conditions?
   - Contamination: did control participants receive elements of the treatment inadvertently?

2. Construct validity (are you measuring and manipulating what you think you are?):
   - Construct underrepresentation: does your operationalization miss important aspects of the construct?
   - Construct-irrelevant variance: does your measure capture things other than the construct of interest?
   - Manipulation check: how do you know the treatment actually changed what it was intended to change?

3. Statistical conclusion validity (are your statistical inferences correct?):
   - Low statistical power: are you likely to detect a real effect if it exists?
   - Multiple comparisons: are you testing many outcomes without adjustment?
   - Assumption violations: do the data meet the assumptions of your planned analyses?
   - Fishing and flexibility in data analysis: are analysis decisions made post-hoc after seeing results?

4. External validity (do results generalize?):
   - Population validity: how similar is your sample to the population of interest?
   - Ecological validity: how similar are your study conditions to real-world conditions?
   - Temporal validity: are results likely to hold at other time points?
   - Treatment variation: does your treatment represent how it would actually be delivered in practice?

5. For each identified threat:
   - Severity: how likely is this threat to bias results and in what direction?
   - Mitigation: what design features address this threat?
   - Residual risk: what threat remains after mitigation?
   - Disclosure: how will this be acknowledged in the limitations section?

Return: validity audit table (threat, severity, mitigation, residual risk), overall validity assessment, and limitations section draft.