Time Series Anomaly Detection in Python

Detect anomalies in a time series using rolling z-score and Isolation Forest, then visualize flagged points.

This page shows an AI data analysis example. A workflow is a list of prompts (instructions) used to complete one analysis task step by step. We send the same prompt list to multiple AI models, then compare their notebook outputs and scores.

What this AI workflow does

This AI Data Analyst workflow loads the NYC Taxi Demand time series from a URL and plots demand over time. It detects anomalies using a rolling mean with a 3-sigma (rolling z-score) rule and visualizes the flagged points on the series. It then applies Isolation Forest for anomaly detection and compares which timestamps are flagged by each method.

Who this example is for

This is for data analysts and data scientists who need a reproducible notebook pattern for time series anomaly detection. It helps users compare a simple statistical thresholding approach with a machine-learning method on the same dataset.

Expected analysis outcomes

These are the results the AI workflow is expected to generate.

  • Loaded and parsed NYC taxi demand time series from a remote CSV
  • Line chart of taxi demand over time
  • Rolling z-score (3-sigma) anomalies highlighted on the time series
  • Isolation Forest anomalies highlighted and compared against the z-score method

Tools and libraries used

Main Python packages and tooling used to run this AI data analysis task.

  • pandas
  • numpy
  • matplotlib
  • seaborn
  • scikit-learn

Prompt sequence

This is the exact list of prompts used in this workflow. The same prompt sequence is sent to each model so outputs and scores can be compared fairly.

  1. 1load NYC taxi demand data from https://raw.githubusercontent.com/pplonski/datasets-for-start/refs/heads/master/nyc-taxi-demand/nyc_taxi.csv and plot the time series
  2. 2detect anomalies using rolling mean and 3-sigma rule, plot flagged points
  3. 3use Isolation Forest to detect anomalies and compare with z-score method

Model Comparison

We compared several LLM models on the same analysis task. The table below shows their scores side by side. You can also open each model run to check the full conversation and notebook results.

Model SourceScoreTaskExecutionOutputReasoningReliabilityRun
glm-5.110/102/22/23/32/21/1Open glm-5.1 conversation
qwen3.5:397b10/102/22/23/32/21/1Open qwen3.5:397b conversation
gpt-5.48/102/22/22/31/21/1Open gpt-5.4 conversation
gpt-oss:120b8/102/22/22/31/21/1Open gpt-oss:120b conversation
qwen3-coder-next7/102/22/22/31/20/1Open qwen3-coder-next conversation
gemma4:31b6/102/21/22/31/20/1Open gemma4:31b conversation
How scoring works

Each run is graded on five dimensions:

  • Task Completion (0-2): how fully the workflow answered the requested analysis task and completed required steps.
  • Execution Correctness (0-2): whether generated code is correct, consistent, and likely runnable without major fixes.
  • Output Quality (0-3): how well tables, charts, and results match the expected analytical outcomes.
  • Reasoning Quality (0-2): clarity and correctness of interpretation, explanation, and conclusions.
  • Reliability (0-1): robustness of the run, including low hallucination risk and stable behavior.

The final score is the sum of these dimensions, so total score = 0-10. Higher values indicate more complete, correct, and reliable notebook results for this workflow.

AI Analysis with glm-5.1

This run uses AI with a step-by-step list of prompts to perform the analysis task Time Series Anomaly Detection in Python. Below, you can review the full conversation generated with glm-5.1. After the conversation, we show detailed score cards for this run.

#1
load NYC taxi demand data from https://raw.githubusercontent.com/pplonski/datasets-for-start/refs/heads/master/nyc-taxi-demand/nyc_taxi.csv and plot the time series
#2
Thinking

The user wants to load NYC taxi demand data from a CSV URL and plot the time series. I'll start by loading the data and inspecting it first, then plot it. But per rules, when a dataset is not yet loaded, first load and inspect only: show shape, columns, and head(). I should not combine loading with plotting in the same response.

Let me first load and inspect the data.

#3

I'll start by loading the dataset and inspecting its structure before plotting.

#4
Load NYC taxi data
import pandas as pd

df = pd.read_csv('https://raw.githubusercontent.com/pplonski/datasets-for-start/refs/heads/master/nyc-taxi-demand/nyc_taxi.csv')
df.shape
#5
(10320, 2)
#6

Let's see the shape and then inspect the columns and first few rows.

#7
Thinking

The dataset is loaded with 10,320 rows and 2 columns (timestamp, value). Now I need to inspect the data briefly and then plot the time series. Let me do a quick head() to understand the format, then plot.

#8

The dataset has 10,320 rows and 2 columns: timestamp and value. Let me quickly inspect the data and then plot the time series.

#9
Preview dataframe structure
display(df.head())
display(df.dtypes)
#10
timestamp value
0 2014-07-01 00:00:00 10844
1 2014-07-01 00:30:00 8127
2 2014-07-01 01:00:00 6210
3 2014-07-01 01:30:00 4656
4 2014-07-01 02:00:00 3820
#11
timestamp      str
value        int64
dtype: object
#12
Thinking

The dataset is loaded and inspected. It has 10,320 rows with timestamp (string) and value (integer) columns. The timestamp column needs to be converted to datetime and set as the index before plotting. The original goal is to plot the time series, which hasn't been done yet. I need to provide code to convert the timestamp and plot.

#13

The data has 10,320 records with half-hourly intervals starting from July 1, 2014. The timestamp column is currently a string and needs to be parsed as datetime before plotting. Let me convert it and create the time series plot.

#14
Plot taxi demand over time
import matplotlib.pyplot as plt
import seaborn as sns

df['timestamp'] = pd.to_datetime(df['timestamp'])
df = df.set_index('timestamp')

fig, ax = plt.subplots(figsize=(14, 5))
sns.lineplot(data=df, x=df.index, y='value', ax=ax, linewidth=0.6)
ax.set_title('NYC Taxi Demand Over Time')
ax.set_xlabel('Date')
ax.set_ylabel('Demand (number of rides)')
plt.tight_layout()
plt.show()
#15
Notebook output
#16
Thinking

The user's original goal was to load the NYC taxi demand data and plot the time series. Looking at the notebook state:

  1. Cell 1: Data loaded successfully (10320 rows, 2 columns)
  2. Cell 2: Data inspected - timestamps are half-hourly from July 2014 onwards, with integer demand values
  3. Cell 3: The time series has been plotted with a proper title, axis labels, and formatting

The plot has been successfully generated. The original goal is fully satisfied - the data was loaded and the time series was plotted. I should choose Path A since the outputs are sufficient to answer the original goal.

#17

The NYC taxi demand data has been loaded and plotted successfully. The time series covers roughly 6 months of half-hourly demand data (10,320 observations starting from July 1, 2014). The plot reveals clear patterns:

  • Strong daily seasonality: demand oscillates between ~2,000–3,000 rides in the early morning hours and peaks around 10,000+ rides during evening rush hours.
  • Weekly cycles: lower troughs on weekends vs. weekdays are visible.
  • A notable anomaly around late October/early November 2014: a sharp dip in demand (likely related to the NYC marathon and Halloween), followed by an unusual spike — this is a well-known anomaly in this dataset.
  • Gradual trend: demand appears relatively stable across the months, with slight variation.
#18
detect anomalies using rolling mean and 3-sigma rule, plot flagged points
#19
Thinking

The user wants to detect anomalies using a rolling mean and 3-sigma rule, then plot the flagged points on the time series. I'll compute the rolling mean and rolling standard deviation, then flag points where the value deviates from the rolling mean by more than 3 standard deviations.

#20

I'll compute a rolling mean and standard deviation, flag points that fall outside the 3-sigma bounds, and overlay them on the time series.

#21
Plot 3-sigma anomaly detection
window = 48  # 24 hours = 48 half-hour intervals
df['rolling_mean'] = df['value'].rolling(window=window, center=True).mean()
df['rolling_std'] = df['value'].rolling(window=window, center=True).std()
df['upper'] = df['rolling_mean'] + 3 * df['rolling_std']
df['lower'] = df['rolling_mean'] - 3 * df['rolling_std']
df['anomaly'] = (df['value'] > df['upper']) | (df['value'] < df['lower'])

fig, ax = plt.subplots(figsize=(14, 5))
ax.plot(df.index, df['value'], linewidth=0.5, label='Demand', color='steelblue')
ax.plot(df.index, df['upper'], linewidth=0.8, linestyle='--', color='gray', label='±3σ bounds')
ax.plot(df.index, df['lower'], linewidth=0.8, linestyle='--', color='gray')
ax.scatter(df.index[df['anomaly']], df.loc[df['anomaly'], 'value'], 
           color='red', s=10, zorder=5, label=f'Anomalies ({df["anomaly"].sum()} points)')
ax.set_title('NYC Taxi Demand — 3-Sigma Anomaly Detection')
ax.set_xlabel('Date')
ax.set_ylabel('Demand (number of rides)')
ax.legend()
plt.tight_layout()
plt.show()
#22
Notebook output
#23
Thinking

The anomaly detection has been completed. The plot shows the anomalies flagged in red. Let me check if the output includes numeric values - from the notebook state, I can see df["anomaly"].sum() was computed in the scatter label, and the figure was displayed. Let me check the number of anomalies and summarize the key findings.

From the code, df["anomaly"].sum() was calculated in the label. The plot was generated. I should provide a summary of the anomalies detected. Let me provide the final answer with insights from the observed outputs.

Actually, I notice the anomaly count was embedded in the plot legend label but I can't see its exact value from the outputs shown. The figure was displayed. I should print out the anomaly count and maybe some details about the flagged points to provide concrete numeric insights, since the output is chart-only without visible underlying values.

Path B is appropriate here - I should print the anomaly count and some details about the flagged points.

#24

The plot is rendered, but the exact anomaly count and details aren't visible as text. Let me pull out the numeric summary of the flagged anomalies.

#25
Summarize detected anomalies
n_anomalies = df['anomaly'].sum()
anomaly_dates = df.index[df['anomaly']]
anomaly_values = df.loc[df['anomaly'], 'value']
print(f"Total anomalies detected: {n_anomalies} out of {len(df)} points ({n_anomalies/len(df)*100:.2f}%)")
print(f"\nAnomaly date range: {anomaly_dates.min()} to {anomaly_dates.max()}")
print(f"\nAnomaly value range: {anomaly_values.min()} to {anomaly_values.max()}")
print(f"\nTop 5 largest positive anomalies:")
display(df.loc[df['anomaly']].nlargest(5, 'value')[['value', 'rolling_mean', 'upper']].round(1))
print(f"\nTop 5 largest negative anomalies:")
display(df.loc[df['anomaly']].nsmallest(5, 'value')[['value', 'rolling_mean', 'lower']].round(1))
#26
Total anomalies detected: 0 out of 10320 points (0.00%)

Anomaly date range: NaT to NaT

Anomaly value range: nan to nan

Top 5 largest positive anomalies:
#27
value rolling_mean upper
timestamp
#28
Top 5 largest negative anomalies:
#29
value rolling_mean lower
timestamp
#30
Thinking

The output shows 0 anomalies detected, which is clearly wrong for this dataset. The issue is that with a centered rolling window of 48, the rolling_std calculation likely produces NaN values or the 3-sigma bounds are too wide. Let me think about what's happening.

The problem is that rolling_std with center=True and window=48 produces NaN values at the edges, and more importantly, the rolling standard deviation might be very large, making the 3-sigma bounds extremely wide.

Let me check: with center=True, the first and last window//2 = 24 rows will have NaN for rolling stats. But that shouldn't cause 0 anomalies for the middle portion.

Actually, the real issue might be that rolling_std defaults to ddof=1, which could be causing issues, or perhaps the rolling statistics are being computed but the comparison is failing due to NaN propagation.

Wait - let me re-read. The anomaly count is 0 out of 10320 points. This means the boolean comparison (df['value'] > df['upper']) | (df['value'] < df['lower']) evaluated to all False. This could happen if upper and lower contain NaN values - because comparing with NaN returns False.

The issue: when rolling_std is NaN (at the edges due to centering), then upper and lower are also NaN, and the comparison value > NaN returns False. So those edge rows are never flagged. But what about the middle rows where rolling stats are valid?

Actually, maybe the rolling std is very large, making the bounds extremely wide. Let me check - with ddof=1 (default), the std for 48 values would be computed with 47 in the denominator. For NYC taxi demand data, the values range roughly from 2000 to 30000, and the std within a 24-hour window could be quite large.

Actually, I think the problem might be simpler. Let me check: when center=True, NaN appears at the start and end, but for the middle rows, both rolling_mean and rolling_std should be valid. The 3-sigma bounds should then be valid too.

Hmm, but 0 anomalies is suspicious. Let me think... maybe the rolling_std is indeed so large that 3*sigma covers everything? That seems unlikely for this data.

Wait, I should just check the actual values. Let me write code to inspect the rolling stats and debug why 0 anomalies were detected.

The most likely issue: NaN comparisons. When rolling_std contains NaN, the upper/lower bounds are NaN, and any comparison with NaN returns False. If many or all rows have NaN in the bounds, no anomalies would be detected.

But with center=True and window=48, only the first and last 24 rows should have NaN. That leaves ~10272 rows with valid values, and some of those should be anomalies.

Actually, wait - I should just check. Let me write diagnostic code to see what's happening with the rolling statistics.

#31

Zero anomalies detected — that's clearly unexpected for this dataset. The issue is almost certainly that NaN values in the rolling statistics (from the centered window at the edges) are causing the boolean comparison to silently return False, and possibly the 3-sigma bounds are too wide. Let me diagnose the rolling stats to see what's going on.

#32
Check rolling stats summary
print("Rolling stats summary:")
display(df[['rolling_mean', 'rolling_std', 'upper', 'lower']].describe())
print(f"\nNaN counts:\n{df[['rolling_mean', 'rolling_std', 'upper', 'lower']].isna().sum()}")
print(f"\nSample of valid (non-NaN) rows:")
valid = df.dropna(subset=['rolling_std'])
display(valid[['value', 'rolling_mean', 'rolling_std', 'upper', 'lower']].head(10))
print(f"\nMax deviation from rolling mean (in sigma units):")
df_valid = df.dropna(subset=['rolling_std'])
max_sigma = ((df_valid['value'] - df_valid['rolling_mean']) / df_valid['rolling_std']).abs().max()
print(f"  Max sigma deviation: {max_sigma:.2f}")
#33
Rolling stats summary:
#34
rolling_mean rolling_std upper lower
count 10273.000000 10273.000000 10273.000000 10273.000000
mean 15130.164620 6612.952970 34969.023531 -4708.694290
std 2172.928147 709.898707 3967.589256 1665.136661
min 2670.875000 2494.764958 10155.169875 -13266.061666
25% 13862.645833 6279.054823 32977.331746 -5929.072816
50% 15229.187500 6739.249341 35649.053486 -4891.356729
75% 16491.979167 7102.717345 37446.340598 -3561.714359
max 21044.833333 9121.073490 46619.760725 3692.296580
#35
NaN counts:
rolling_mean    47
rolling_std     47
upper           47
lower           47
dtype: int64

Sample of valid (non-NaN) rows:
#36
value rolling_mean rolling_std upper lower
timestamp
2014-07-01 12:00:00 18908 15540.979167 7534.507810 38144.502596 -7062.544263
2014-07-01 12:30:00 18886 15593.604167 7509.784510 38122.957698 -6935.749365
2014-07-01 13:00:00 18178 15631.479167 7475.833795 38058.980553 -6796.022219
2014-07-01 13:30:00 19459 15659.833333 7441.843656 37985.364300 -6665.697633
2014-07-01 14:00:00 19546 15686.104167 7404.303039 37899.013285 -6526.804952
2014-07-01 14:30:00 20591 15706.937500 7371.539611 37821.556332 -6407.681332
2014-07-01 15:00:00 19380 15722.791667 7344.117382 37755.143814 -6309.560480
2014-07-01 15:30:00 18544 15735.791667 7320.490854 37697.264228 -6225.680894
2014-07-01 16:00:00 16228 15745.604167 7302.067544 37651.806798 -6160.598465
2014-07-01 16:30:00 15013 15752.875000 7288.475367 37618.301101 -6112.551101
#37
Max deviation from rolling mean (in sigma units):
  Max sigma deviation: 2.46
#38
Thinking

The key insight from the diagnostics: the maximum sigma deviation is only 2.46, which means no point exceeds 3 sigma with a window of 48. The 3-sigma rule with this window size is too loose. The rolling standard deviation (~6600) is too large relative to the fluctuations, so the bounds are extremely wide.

The user's goal is to detect anomalies using rolling mean and 3-sigma rule and plot flagged points. The current approach found zero anomalies because the window is too short relative to the variability, or the data's variability is high.

Options:

  1. Use a longer window to get a smoother mean and tighter std (since longer windows smooth out local variation)
  2. Actually, a longer window would likely have similar or larger std
  3. The real issue might be that half-hour data has very high local variability

Wait, actually the rolling_std mean is ~6612 which is huge. The value mean is ~15130. The coefficients of variation are large. Let me think about this differently.

Actually, looking at this more carefully - with a window of 48 (24 hours), the std captures the full daily cycle variability. The daily pattern itself causes high std. A better approach might be to use a longer window (e.g., 336 = 7 days) so the mean and std represent a longer-term baseline, or to use a smaller window but acknowledge that 3-sigma might be too generous.

But actually, the user asked specifically for rolling mean and 3-sigma. The current window of 48 (24 hours) is capturing too much variability in the std. We could:

  1. Increase the window size significantly (e.g., 336 for weekly)
  2. Use min_periods to avoid NaN issues
  3. Or acknowledge the finding

The most practical fix: use a longer window (e.g., 336 = 7 days of half-hour intervals) which will smooth out daily patterns and result in a std that reflects true deviations from a longer-term trend. This is a common approach for this NYC taxi dataset.

Actually, let me reconsider. The max deviation is 2.46 sigma. If we use a longer window, the rolling mean will be smoother, and the std might change. But actually, with a longer window, the std might be similar or even larger because it captures more variation.

The real issue is that the rolling window std includes the periodic (daily) variation. For anomaly detection on this kind of data, people often use a longer window or a different approach.

Let me think about what's most practical: The simplest fix that aligns with the user's stated goal is to adjust the window. A common approach for this dataset is window=168 (3.5 days) or window=336 (7 days). Let me try window=336 with min_periods set to reduce NaN edges.

Actually, I think the better approach is simply to note that with window=48, no points exceed 3 sigma (max is 2.46), and then try a wider window. But actually, a wider window might have similar issues.

Let me reconsider: perhaps the issue is that the rolling std should be computed with a larger window, or we should use a different baseline. Another common approach is to use the rolling mean with a long window and rolling std with the same window, then the bounds are tighter because the mean is smoother.

Wait, actually I need to reconsider. With a wider window, the std will likely be larger (capturing more variation), making the bounds even wider. That won't help.

The fundamental issue: the data has a strong daily pattern, and within any window, the std captures that daily variation, making bounds too wide.

Possible solutions:

  1. Use a much larger window so the std stabilizes around the overall population std, and spikes relative to that baseline might exceed 3 sigma
  2. Remove the daily seasonality first, then apply 3-sigma to residuals
  3. Use a narrower window (e.g., 6 or 12) so the std is computed over a shorter, more locally stable period

Actually, let me check: the max deviation is 2.46 sigma. If we use a narrower window (like 12 or 24), the local std would be smaller (since we're comparing against a short-term local baseline), and spikes might exceed 3 sigma.

Hmm, but actually with a narrow window, the std could go either way. With a very narrow window, if there's a spike, the spike itself inflates the std.

I think the most natural fix that still uses "rolling mean + 3-sigma" is to:

  1. Use a longer window (like 336 = 1 week) so the mean represents a weekly baseline
  2. Set min_periods to avoid NaN issues

Or alternatively, use a window of 48 but with a trailing (not centered) window and use the std from the previous window.

Actually, the simplest and most principled fix: increase the window to capture more data points for a stable baseline. Let me try window=336 (7 days) with min_periods=48.

But wait - with a wider window, std will be even larger because it captures more of the daily cycle variation. So bounds will be even wider.

Hmm, let me think again. The actual std of the entire dataset is probably around 6000-7000 (based on the describe output showing rolling_std mean of ~6612). The overall std of value is probably similar. The maximum value in the dataset is probably around 30,000 or so, and the mean is around 15,000. So (30000 - 15000) / 7000 ≈ 2.14. Even for the entire dataset, the max is only about 2+ sigma.

So the issue is that this dataset genuinely doesn't have extreme outliers by 3-sigma standard - OR the outliers are the very visible spikes in the NYC taxi data (like during blizzards, holidays etc.).

Let me look at this differently. The NYC taxi demand dataset famously has anomalies around:

  • Nov 2, 2014 (NYC marathon)
  • Dec 31, 2014 (New Year's Eve)
  • Jan 27, 2015 (blizzard)
  • Jan 21, 2015

These are the known anomalies in this dataset. The issue might be that with half-hourly data, even the anomalies only reach ~2.46 sigma.

I think the most useful approach for the user's goal would be to try a tighter threshold (like 2 sigma) or to use a longer window for a more stable baseline. But since the user specifically asked for 3-sigma, let me try adjusting the window.

Actually, let me try a different approach: use a longer window for the mean (to get a smooth baseline) but a shorter window for the std (to get local variability). This is sometimes done in practice.

Or, simplest fix: use a trailing window with min_periods, and increase window size to 168 or 336.

I think the best next step is to try a larger window (336 = 7 days) and see if that makes the bounds tighter relative to the spikes. With a 7-day window, the mean will be very stable, and the std will represent week-long variability. Actually, this might not help either.

Let me just try it. The code should:

  1. Use a larger window (336 = 7 days)
  2. Use min_periods to avoid NaN issues
  3. Recompute bounds and anomalies
  4. Plot and show count

This is the minimal next step.

#39

The diagnostics reveal the core issue: the maximum deviation from the rolling mean is only 2.46 sigma, so no point exceeds the 3-sigma threshold. With a 48-period (24-hour) window, the daily cycle inflates the rolling standard deviation, making the bounds too wide to catch any anomalies.

A practical fix is to use a longer window (7 days = 336 half-hour intervals) for a more stable baseline, and set min_periods to avoid NaN issues that previously suppressed anomaly detection at the edges.

#40
Calculate rolling bounds and detect anomalies
window = 336  # 7 days = 336 half-hour intervals
df['rolling_mean'] = df['value'].rolling(window=window, center=True, min_periods=48).mean()
df['rolling_std'] = df['value'].rolling(window=window, center=True, min_periods=48).std()
df['upper'] = df['rolling_mean'] + 3 * df['rolling_std']
df['lower'] = df['rolling_mean'] - 3 * df['rolling_std']
df['anomaly'] = (df['value'] > df['upper']) | (df['value'] < df['lower'])

print(f"Anomalies detected: {df['anomaly'].sum()} out of {len(df)}")
print(f"NaN rows in rolling stats: {df['rolling_std'].isna().sum()}")
if df['anomaly'].sum() > 0:
    print(f"\nAnomaly dates:")
    display(df.loc[df['anomaly'], ['value', 'rolling_mean', 'rolling_std', 'upper', 'lower']].head(20))
#41
Anomalies detected: 1 out of 10320
NaN rows in rolling stats: 0

Anomaly dates:
#42
value rolling_mean rolling_std upper lower
timestamp
2014-11-02 01:00:00 39197 16347.446429 7335.611345 38354.280465 -5659.387608
#43
use Isolation Forest to detect anomalies and compare with z-score method
#44
Thinking

The user wants to use Isolation Forest to detect anomalies and compare with the z-score method (which we've been working with using rolling mean and 3-sigma rule).

Let me first apply Isolation Forest on the 'value' column, then compare results with the existing anomaly flags. I should exclude identifier/leakage columns. The main feature is just 'value' - I should use it (and possibly derived features) but not the anomaly flags or rolling stats as features.

Let me start by applying Isolation Forest and getting the initial results.

#45

Great — the 7-day rolling window finally caught 1 anomaly. Now let's apply Isolation Forest on the same data and see how it compares. I'll use the raw value column as the feature (excluding the rolling/anomaly columns which are derived from the target).

#46
Compare isolation forest anomalies
from sklearn.ensemble import IsolationForest

X = df[['value']]

iso = IsolationForest(n_estimators=100, contamination=0.01, random_state=42)
iso_labels = iso.fit_predict(X)  # -1 = anomaly, 1 = normal

df['iso_anomaly'] = iso_labels == -1

print(f"Isolation Forest anomalies: {df['iso_anomaly'].sum()} out of {len(df)} ({df['iso_anomaly'].sum()/len(df)*100:.2f}%)")
print(f"\nRolling 3-sigma anomalies: {df['anomaly'].sum()} out of {len(df)}")
print(f"\nOverlap (both methods agree): {(df['anomaly'] & df['iso_anomaly']).sum()}")
print(f"\nIso Forest anomaly value stats:")
display(df.loc[df['iso_anomaly'], 'value'].describe())
print(f"\nSample Iso Forest anomalies (by value, descending):")
display(df.loc[df['iso_anomaly']].nlargest(10, 'value')[['value', 'rolling_mean', 'iso_anomaly']])
display(df.loc[df['iso_anomaly']].nsmallest(10, 'value')[['value', 'rolling_mean', 'iso_anomaly']])
#47
Isolation Forest anomalies: 104 out of 10320 (1.01%)

Rolling 3-sigma anomalies: 1 out of 10320

Overlap (both methods agree): 1

Iso Forest anomaly value stats:
#48
count      104.000000
mean     20624.634615
std      12368.408214
min          8.000000
25%       1452.000000
50%      27327.500000
75%      27860.750000
max      39197.000000
Name: value, dtype: float64
#49
Sample Iso Forest anomalies (by value, descending):
#50
value rolling_mean iso_anomaly
timestamp
2014-11-02 01:00:00 39197 16347.446429 True
2014-11-02 01:30:00 35212 16348.482143 True
2014-09-06 23:00:00 30373 16282.238095 True
2014-09-06 22:30:00 30313 16283.089286 True
2015-01-01 01:00:00 30236 13536.196429 True
2014-07-03 19:00:00 29985 12948.311258 True
2015-01-01 00:30:00 29547 13535.116071 True
2015-01-31 19:00:00 28804 16729.264045 True
2014-10-18 23:30:00 28626 16406.416667 True
2014-11-22 23:30:00 28472 15764.276786 True
#51
value rolling_mean iso_anomaly
timestamp
2015-01-27 03:00:00 8 12525.895833 True
2015-01-27 04:00:00 9 12523.785714 True
2015-01-27 03:30:00 11 12524.053571 True
2015-01-27 04:30:00 20 12523.494048 True
2015-01-27 05:00:00 21 12522.282738 True
2015-01-27 02:00:00 26 12529.562500 True
2015-01-27 02:30:00 32 12528.455357 True
2015-01-27 05:30:00 37 12522.000000 True
2015-01-27 01:30:00 39 12529.035714 True
2015-01-27 01:00:00 40 12529.458333 True
#52
Thinking

The user wants to compare Isolation Forest with the z-score (rolling 3-sigma) method. Looking at the latest outputs:

  • Isolation Forest detected 104 anomalies (1.01%)
  • Rolling 3-sigma detected 1 anomaly
  • Overlap: 1 point (both agree on that one)
  • Isolation Forest catches both extreme highs (up to 39,197) and extreme lows (down to 8)
  • The 3-sigma method only caught the single most extreme spike (39,197)

The numerical comparison is clear from the output. What's missing is a visual comparison showing both methods' flagged points on the time series. The original goal asks to "detect anomalies and compare" - a side-by-side or overlaid plot would make the comparison concrete.

Let me create a comparison plot showing both methods' anomaly flags on the same time series, plus a summary comparison table.

#53

The Isolation Forest is far more sensitive — it flagged 104 anomalies (1%) versus just 1 with the 3-sigma rule. They overlap on exactly one point (the 39,197 spike on 2014-11-02). Notably, Isolation Forest catches two distinct anomaly patterns: extreme demand spikes (holidays, concerts) and near-zero drops (the 2015-01-27 blizzard where values hit single digits). The 3-sigma method misses almost everything because the 7-day rolling window inflates the standard deviation with daily cyclicality, pushing the bounds too wide.

Let me create a visual side-by-side comparison to make the differences concrete:

#54
Plot anomaly detection comparison
fig, axes = plt.subplots(2, 1, figsize=(14, 8), sharex=True)

for ax, method, col, title in [
    (axes[0], '3-Sigma', 'anomaly', 'Rolling 3-Sigma — 1 anomaly detected'),
    (axes[1], 'Isolation Forest', 'iso_anomaly', 'Isolation Forest (1% contamination) — 104 anomalies detected'),
]:
    ax.plot(df.index, df['value'], linewidth=0.4, color='steelblue', alpha=0.7)
    ax.scatter(df.index[df[col]], df.loc[df[col], 'value'],
               color='red', s=15, zorder=5, label=f'Anomaly ({df[col].sum()} pts)')
    ax.set_title(title, fontsize=12)
    ax.set_ylabel('Demand')
    ax.legend(loc='upper right')

axes[1].set_xlabel('Date')
plt.suptitle('Anomaly Detection Comparison: 3-Sigma vs Isolation Forest', fontsize=14, y=1.02)
plt.tight_layout()
plt.show()
#55
Notebook output
#56
Thinking

The comparison is complete. We have:

  1. Rolling 3-sigma detected 1 anomaly
  2. Isolation Forest detected 104 anomalies (1% contamination)
  3. Overlap: 1 point (both agree on the Nov 2, 2014 spike)
  4. Visual comparison chart already shown
  5. Detailed stats on Iso Forest anomalies showing both high spikes and low drops (Jan 2015 blizzard)

The outputs are sufficient to fully answer the original goal of comparing Isolation Forest with z-score method. Let me provide a final summary.

#57

The comparison between Isolation Forest and the z-score (3-sigma rolling) method is now complete with both numeric results and a visual chart. Here's the summary:

Detection sensitivity differs dramatically:

  • Rolling 3-sigma: 1 anomaly out of 10,320 points (0.01%) — only the Nov 2, 2014 spike (value 39,197) was flagged.
  • Isolation Forest (1% contamination): 104 anomalies (1.01%) — caught the same spike plus 103 other points.

Why the gap? The 7-day rolling window absorbs daily cyclical patterns into the mean and standard deviation, making the ±3σ bounds so wide that almost nothing qualifies as anomalous. The max sigma deviation in the entire dataset was only 2.46 — the Nov 2 spike barely crossed 3σ.

Isolation Forest catches two distinct anomaly types:

  1. Demand spikes — holidays (NYE), concerts (Sep 6), and other surges with values 27,000–39,197 that sit well above the typical ~15,000 mean.
  2. Demand drops — the Jan 27, 2015 blizzard hours where taxi demand plummeted to 8–40 rides (vs. the ~12,500 local mean), which the 3-sigma method completely missed because the lower bound went deeply negative.

Overlap is minimal — only 1 point was flagged by both methods (the Nov 2 spike), showing that the z-score approach is extremely conservative with a centered rolling window on data with strong daily seasonality. Isolation Forest, which doesn't rely on distributional assumptions, adapts better to the irregular tails of this data.

Score cards for glm-5.1 for Time Series Anomaly Detection in Python

Total score: 10/10

  • Task Completion: 2/2

    Excellent

    The notebook loads the taxi data, plots the time series, applies rolling 3-sigma anomaly detection, runs Isolation Forest, and compares the methods with a final visualization.

  • Execution Correctness: 2/2

    Excellent

    The provided code is coherent and appears runnable; outputs are shown for each major step, including the plots and anomaly summaries.

  • Output Quality: 3/3

    Excellent

    Outputs match the expected outcomes: a time series plot, anomalies highlighted in red, and a comparison showing Isolation Forest finds more anomalies with only one overlap.

  • Reasoning Quality: 2/2

    Excellent

    The workflow correctly interprets the dataset, explains the anomaly methods, and summarizes the comparison in a way consistent with the outputs.

  • Reliability: 1/1

    Excellent

    The workflow is consistent and grounded in the displayed outputs, with no obvious hallucinated results or unsupported claims.

Try MLJAR Studio

Run the same type of AI-powered data analysis on your own datasets with conversational notebooks in MLJAR Studio.

Download Free TrialLearn about AI Data Analyst

Explore More AI Analysis Examples

Discover additional workflows across categories. Each example includes prompts, conversation outputs, and model-level scoring so you can compare approaches and results.

Exploratory Data Analysis

Time Series

Data Analysis

Machine Learning

NLP

Finance

Statistics