Is This Difference Real? AI Prompt

I see a difference in my data between two groups or time periods. Help me figure out whether this difference is meaningful or just random variation.

Observation: {{observation}} (e.g. 'Group A has a 12% conversion rate and Group B has a 14% conversion rate')
Sample sizes: {{sample_sizes}}

1. Explain the core problem in plain English:
   - Small samples are noisy: if you flip a coin 10 times and get 6 heads, that does not mean the coin is biased
   - The same applies to business data: a small difference on a small sample is often just luck
   - We need to know if the difference is large enough relative to the sample size to be trustworthy

2. The quick gut-check:
   - How large is the difference in percentage terms and in absolute terms?
   - How large is each group? Fewer than 30 in either group → the difference is probably unreliable. Fewer than 100 → be cautious.
   - Has this pattern held up over multiple time periods, or is this one observation?

3. The proper test (explained simply):
   - For comparing two rates or percentages between groups: use a proportion test
   - For comparing two averages: use a t-test
   - Explain what 'p-value' means without jargon: 'A p-value of 0.05 means that if there were truly no difference between the groups, we would only see a gap this large or larger by chance 5% of the time — so we can be reasonably confident the difference is real'
   - Run the appropriate test and tell me the result

4. Practical vs statistical significance:
   - Statistical significance just means the difference is real, not random
   - It does not mean the difference is large enough to matter for the business
   - A difference can be statistically significant but too small to act on (e.g. 3.1% vs 3.2% conversion rate on 100,000 users)
   - Is this difference both statistically significant AND large enough to care about?

5. Plain English verdict:
   - Give me a single sentence conclusion: is this difference real, probably real, probably not real, or impossible to tell with this data?