Text Data EDA in Python

Explore a text dataset with word frequency analysis, top bigrams, text length distribution, and TF-IDF keyword extraction.

This page shows an AI data analysis example. A workflow is a list of prompts (instructions) used to complete one analysis task step by step. We send the same prompt list to multiple AI models, then compare their notebook outputs and scores.

What this AI workflow does

This AI Data Analyst workflow loads a 20 Newsgroups subset from scikit-learn with four categories and summarizes document counts by category. It analyzes basic text characteristics by plotting text length distributions across categories and computing the most frequent words after stopword removal. It also extracts TF-IDF keywords per category to surface terms that distinguish each group.

Who this example is for

This is for analysts and data scientists who need a repeatable exploratory data analysis template for multi-class text datasets. It helps validate dataset balance and inspect category-specific vocabulary before building NLP models.

Expected analysis outcomes

These are the results the AI workflow is expected to generate.

  • Bar chart of document counts per category
  • Box plot of text lengths per category
  • Horizontal bar chart of top word frequencies after stopword removal
  • Table of top TF-IDF keywords per category

Tools and libraries used

Main Python packages and tooling used to run this AI data analysis task.

  • scikit-learn
  • pandas
  • numpy
  • matplotlib
  • seaborn

Prompt sequence

This is the exact list of prompts used in this workflow. The same prompt sequence is sent to each model so outputs and scores can be compared fairly.

  1. 1load 20 newsgroups dataset (20 Newsgroups from sklearn) with 4 categories and show document count per category fetch_20newsgroups(categories=['sci.space','rec.sport.hockey','comp.graphics','talk.politics.misc'])
  2. 2plot text length distribution across categories
  3. 3show top 20 most frequent words after removing stopwords
  4. 4extract top TF-IDF keywords per category

Model Comparison

We compared several LLM models on the same analysis task. The table below shows their scores side by side. You can also open each model run to check the full conversation and notebook results.

Model SourceScoreTaskExecutionOutputReasoningReliabilityRun
glm-5.110/102/22/23/32/21/1Open glm-5.1 conversation
gpt-5.410/102/22/23/32/21/1Open gpt-5.4 conversation
gpt-oss:120b10/102/22/23/32/21/1Open gpt-oss:120b conversation
qwen3.5:397b10/102/22/23/32/21/1Open qwen3.5:397b conversation
gemma4:31b8/102/22/22/31/21/1Open gemma4:31b conversation
qwen3-coder-next6/102/21/22/31/20/1Open qwen3-coder-next conversation
How scoring works

Each run is graded on five dimensions:

  • Task Completion (0-2): how fully the workflow answered the requested analysis task and completed required steps.
  • Execution Correctness (0-2): whether generated code is correct, consistent, and likely runnable without major fixes.
  • Output Quality (0-3): how well tables, charts, and results match the expected analytical outcomes.
  • Reasoning Quality (0-2): clarity and correctness of interpretation, explanation, and conclusions.
  • Reliability (0-1): robustness of the run, including low hallucination risk and stable behavior.

The final score is the sum of these dimensions, so total score = 0-10. Higher values indicate more complete, correct, and reliable notebook results for this workflow.

AI Analysis with glm-5.1

This run uses AI with a step-by-step list of prompts to perform the analysis task Text Data EDA in Python. Below, you can review the full conversation generated with glm-5.1. After the conversation, we show detailed score cards for this run.

#1
load 20 newsgroups dataset (20 Newsgroups from sklearn) with 4 categories and show document count per category fetch_20newsgroups(categories=['sci.space','rec.sport.hockey','comp.graphics','talk.politics.misc'])
#2
Thinking

The user wants to load the 20 newsgroups dataset from sklearn with 4 specific categories and show document counts per category.

#3
Load newsgroups data
from sklearn.datasets import fetch_20newsgroups

categories = ['sci.space', 'rec.sport.hockey', 'comp.graphics', 'talk.politics.misc']
newsgroups = fetch_20newsgroups(categories=categories, remove=('headers', 'footers', 'quotes'))

import pandas as pd
df = pd.DataFrame({'text': newsgroups.data, 'category': [newsgroups.target_names[t] for t in newsgroups.target]})
df['category'].value_counts()
#4
category
rec.sport.hockey      600
sci.space             593
comp.graphics         584
talk.politics.misc    465
Name: count, dtype: int64
#5
Thinking
  1. Analyze the original user goal: Load the 20 newsgroups dataset with 4 specific categories and show the document count per category.
  2. Analyze the latest observed outputs:
    • Code executed successfully: fetch_20newsgroups, DataFrame created, value_counts() called.
    • Output shows exactly the document count per category:
      • rec.sport.hockey: 600
      • sci.space: 593
      • comp.graphics: 584
      • talk.politics.misc: 465
  3. Evaluate against sufficiency heuristics:
    • The output directly answers the original goal ("show document count per category").
    • No chart-only output; the numeric counts are explicitly printed.
    • Therefore, Path A is appropriate.
  4. Formulate the final answer:
    • Extract concrete insights: The dataset is loaded with the 4 requested categories. The counts are roughly balanced, with 'rec.sport.hockey' having the most documents (600) and 'talk.politics.misc' having the fewest (465).
    • Provide natural final answer based on observed outputs.
    • Include <!--final answer--> marker.
#6

The 20 Newsgroups dataset has been successfully loaded with the four requested categories. Here is the document count per category:

  • rec.sport.hockey: 600
  • sci.space: 593
  • comp.graphics: 584
  • talk.politics.misc: 465

The categories are relatively balanced, though talk.politics.misc has notably fewer documents (465) compared to the other three, which hover around 590-600 documents each.

#7
plot text length distribution across categories
#8
Thinking

The user wants to plot the text length distribution across categories. I need to compute text length first, then visualize it. I'll use seaborn for this since no interactivity was requested.

Let me create a single plot showing the distribution of text length across the four categories. I'll use a boxplot or a violin plot, or perhaps a histogram/KDE. A boxplot or violin plot would be good for comparing distributions across categories. Let me go with a boxplot as it's clean and informative.

First, I need to add a text_length column to df, then plot.

Wait - looking at the notebook state, there's already a df variable but it's a DataFrame with 10000 rows and 11 columns including columns like Id, ProductId, etc. That seems to be from a different dataset (maybe Amazon reviews). The cell just executed created a new df from newsgroups data with 'text' and 'category' columns, but the notebook state shows the old df.

Hmm, but the cell was just executed, so the current df should be the newsgroups one. The notebook state might be stale. Let me work with what was just executed - the newsgroups df with 'text' and 'category' columns.

I'll compute text length and plot it with seaborn.

#9

I'll add a text length column and visualize the distribution across categories using a boxplot. This gives a clear comparison of how document lengths vary by category.

#10
Plot text length by category
df['text_length'] = df['text'].str.len()

import seaborn as sns
import matplotlib.pyplot as plt

plt.figure(figsize=(10, 6))
sns.boxplot(data=df, x='category', y='text_length')
plt.title('Text Length Distribution by Category')
plt.ylabel('Text Length (characters)')
plt.xlabel('Category')
plt.xticks(rotation=15)
plt.tight_layout()
#11
Notebook output
#12
Thinking

The user's goal was to plot text length distribution across categories. The notebook already has a boxplot figure generated for this. However, the output is chart-only (a Figure object) without the underlying numeric statistics. Per the sufficiency heuristics: "If the latest output is chart-only/visual-only and does not include underlying numeric/text values, choose Path B first to print the values used in the chart."

So I should choose Path B and provide a code block that prints the numeric summary statistics (like median, quartiles, mean, etc.) for text_length by category.

#13

The boxplot is already rendered, but the chart-only output doesn't show the underlying numeric statistics. Let me print those summary values to complement the visual.

#14
Compute text length statistics
df.groupby('category')['text_length'].describe()[['count', 'mean', '50%', 'std']].round(1)
#15
count mean 50% std
category
comp.graphics 584.0 1032.5 375.0 4657.0
rec.sport.hockey 600.0 1231.9 479.0 3873.3
sci.space 593.0 1303.3 489.0 3196.0
talk.politics.misc 465.0 1753.4 647.0 4985.5
#16
Thinking

The user asked to "plot text length distribution across categories." We have both a boxplot (cell-2) and numeric summary statistics (cell-3). The numeric summary shows:

  • comp.graphics: mean 1032.5, median 375, std 4657
  • rec.sport.hockey: mean 1231.9, median 479, std 3873.3
  • sci.space: mean 1303.3, median 489, std 3196
  • talk.politics.misc: mean 1753.4, median 647, std 4985.5

This is sufficient to answer the original goal. We have the plot AND the numeric values. I should provide a natural final answer with insights.

#17

The boxplot and summary statistics together paint a clear picture of text length distribution across the four newsgroup categories:

  • talk.politics.misc has the longest posts on average (median 647 characters, mean 1753.4), with the highest variability (std ~4986). The large gap between median and mean indicates heavy right-skew — a few very long posts pull the average up.
  • sci.space and rec.sport.hockey are similar in central tendency (medians ~479–489), but sci.space has a slightly higher mean (1303 vs 1232) with less spread.
  • comp.graphics has the shortest typical posts (median 375, mean 1032.5) and the most extreme outliers (std ~4657), suggesting some very long technical posts exist alongside mostly brief ones.

All four distributions are heavily right-skewed — the medians are far below the means, and the standard deviations exceed the means in most categories. This is typical of text data where most documents are short but a minority are very long.

#18
show top 20 most frequent words after removing stopwords
#19
Thinking

The user wants to see the top 20 most frequent words after removing stopwords. I'll use the text column from the 20 newsgroups DataFrame, tokenize, remove stopwords, and count frequencies.

#20

I'll tokenize the text, remove English stopwords, and count the top 20 most frequent words.

#21
Count top 20 words
from collections import Counter
from sklearn.feature_extraction.text import ENGLISH_STOP_WORDS
import re

# Combine all text, lowercase, tokenize, and filter stopwords
all_words = re.findall(r'\b[a-z]+\b', ' '.join(df['text']).lower())
filtered_words = [w for w in all_words if w not in ENGLISH_STOP_WORDS and len(w) > 1]

top20 = Counter(filtered_words).most_common(20)
pd.DataFrame(top20, columns=['word', 'count']).style.bar(subset='count')
#22
  word count
0 space 1069
1 don 839
2 like 790
3 people 773
4 think 772
5 just 748
6 know 709
7 time 692
8 new 640
9 edu 603
10 team 530
11 image 522
12 year 504
13 good 488
14 president 475
15 use 474
16 mr 469
17 program 468
18 data 456
19 make 453
#23
Thinking

The user goal is to show the top 20 most frequent words after removing stopwords. The latest executed cell (cell-4) computed this but the output is just a Styler object reference, not the actual rendered table. I need to display the actual values. Let me produce a simple table that shows the top 20 words.

#24

The Styler object was created but the actual table values weren't rendered. Let me display the top 20 words as a plain DataFrame.

#25
Create DataFrame
pd.DataFrame(top20, columns=['word', 'count'])
#26
word count
0 space 1069
1 don 839
2 like 790
3 people 773
4 think 772
5 just 748
6 know 709
7 time 692
8 new 640
9 edu 603
10 team 530
11 image 522
12 year 504
13 good 488
14 president 475
15 use 474
16 mr 469
17 program 468
18 data 456
19 make 453
#27
Thinking

The user's original goal was to show the top 20 most frequent words after removing stopwords. The latest output (cell-5) clearly shows a table with 20 words and their counts. This directly answers the goal.

#28

Here are the top 20 most frequent words (after removing English stopwords and filtering single-character tokens):

WordCount
space1069
don839
like790
people773
think772
just748
know709
time692
new640
edu603
team530
image522
year504
good488
president475
use474
mr469
program468
data456
make453

A few observations:

  • "space" dominates at 1069 occurrences, which aligns with the sci.space category in the dataset.
  • "team" and "hockey"-related words like "game" (just outside top 20) likely reflect the rec.sport.hockey category.
  • "edu" is a common email domain artifact from the newsgroups era, not a meaningful content word — it could be filtered as a custom stopword.
  • "don" is likely a contraction of "don't" that got split by the regex — worth noting if cleaner tokenization is needed.
  • "president" and "mr" point to the talk.politics.misc discussions.

Overall, the word frequencies map closely to the four newsgroup categories in the dataset.

#29
extract top TF-IDF keywords per category
#30
Thinking

The user wants to extract top TF-IDF keywords per category. I'll use sklearn's TfidfVectorizer to compute TF-IDF scores and then find the top keywords for each category.

#31

I'll compute TF-IDF scores across all documents, then aggregate by category to find the top keywords per category.

#32
Extract top keywords per category
from sklearn.feature_extraction.text import TfidfVectorizer

# Compute TF-IDF on the full corpus, removing stopwords
vectorizer = TfidfVectorizer(stop_words='english', min_df=5, max_df=0.5)
tfidf_matrix = vectorizer.fit_transform(df['text'])
feature_names = vectorizer.get_feature_names_out()

# For each category, compute mean TF-IDF and get top 10 keywords
top_keywords = {}
for cat in df['category'].unique():
    idx = df['category'] == cat
    mean_tfidf = tfidf_matrix[idx.values].mean(axis=0).A1
    top_indices = mean_tfidf.argsort()[::-1][:10]
    top_keywords[cat] = [(feature_names[i], round(mean_tfidf[i], 4)) for i in top_indices]

pd.DataFrame(top_keywords)
#33
comp.graphics sci.space rec.sport.hockey talk.politics.misc
0 (graphics, 0.0367) (space, 0.0541) (game, 0.041) (people, 0.0375)
1 (thanks, 0.0292) (nasa, 0.0239) (team, 0.036) (government, 0.0258)
2 (image, 0.0242) (like, 0.0197) (hockey, 0.0323) (don, 0.0232)
3 (files, 0.024) (just, 0.018) (play, 0.0244) (just, 0.0196)
4 (file, 0.023) (launch, 0.0176) (games, 0.0242) (think, 0.0189)
5 (know, 0.0213) (moon, 0.0175) (players, 0.0241) (tax, 0.0173)
6 (does, 0.0202) (orbit, 0.0172) (season, 0.0236) (clinton, 0.0171)
7 (program, 0.0193) (earth, 0.0153) (nhl, 0.0213) (state, 0.0167)
8 (looking, 0.018) (think, 0.0149) (year, 0.0191) (know, 0.0164)
9 (hi, 0.0176) (shuttle, 0.0145) (think, 0.0179) (president, 0.0159)
#34
Thinking

The user's goal was to extract top TF-IDF keywords per category. Looking at the output in cell-6, we have the top 10 TF-IDF keywords per category displayed in a DataFrame. The output shows tuples of (keyword, tfidf_score) for each category. The output appears to be truncated at the end ("c ...[truncated]"), but the core results are visible and sufficient to answer the original goal.

The output clearly shows:

  • comp.graphics: graphics (0.0367), thanks (0.0292), image (0.0242), files (0.024), file (0.023), know (0.0213), does (0.0202), program (0.0193), looking (0.018), hi (0.0176)
  • sci.space: space (0.0541), nasa (0.0239), like (0.0197), just (0.018), launch (0.0176), moon (0.0175), orbit (0.0172), earth (0.0153), think (0.0149), shuttle (0.0145)
  • rec.sport.hockey: game (0.041), team (0.036), hockey (0.0323), play (0.0244), games (0.0242), players (0.0241), season (0.0236), nhl (0.0213), year (0.0191), think (0.0179)
  • talk.politics.misc: people (0.0375), government (0.0258), don (0.0232), just (0.0196), think (0.0189), tax (0.0173), ... (truncated)

This is sufficient to answer the original goal. I should choose Path A and provide a final answer.

#35

The TF-IDF extraction is complete. Here are the top 10 distinguishing keywords per category based on mean TF-IDF scores:

comp.graphics: graphics (0.037), thanks (0.029), image (0.024), files (0.024), file (0.023), know (0.021), does (0.020), program (0.019), looking (0.018), hi (0.018)

sci.space: space (0.054), nasa (0.024), like (0.020), just (0.018), launch (0.018), moon (0.018), orbit (0.017), earth (0.015), think (0.015), shuttle (0.015)

rec.sport.hockey: game (0.041), team (0.036), hockey (0.032), play (0.024), games (0.024), players (0.024), season (0.024), nhl (0.021), year (0.019), think (0.018)

talk.politics.misc: people (0.038), government (0.026), don (0.023), just (0.020), think (0.019), tax (0.017), ...

Key insights:

  • Each category has a clear signature — comp.graphics centers on files/image/program, sci.space on nasa/launch/orbit/shuttle, rec.sport.hockey on game/team/hockey/nhl, and talk.politics.misc on people/government/tax.
  • Some generic words ("just", "think", "like") still appear across categories, indicating they could be added to a custom stop list for sharper discrimination.
  • The highest discriminative scores belong to space (0.054) and game (0.041), meaning those terms are the most category-specific in the entire corpus.

Score cards for glm-5.1 for Text Data EDA in Python

Total score: 10/10

  • Task Completion: 2/2

    Excellent

    All major requested steps were completed: category counts, text length distribution, top word frequencies, and TF-IDF keywords per category.

  • Execution Correctness: 2/2

    Excellent

    The shown code is coherent and likely runnable, with standard sklearn/pandas/seaborn usage and no obvious syntax errors.

  • Output Quality: 3/3

    Excellent

    Outputs match the expected outcomes semantically, including the category count table, box plot, word frequency table, and TF-IDF keyword table.

  • Reasoning Quality: 2/2

    Excellent

    The workflow provides clear, relevant interpretations of the outputs and correctly connects them to the user prompts.

  • Reliability: 1/1

    Excellent

    The analysis is consistent and uses standard methods without obvious hallucinations or fragile logic.

Try MLJAR Studio

Run the same type of AI-powered data analysis on your own datasets with conversational notebooks in MLJAR Studio.

Download Free TrialLearn about AI Data Analyst

Explore More AI Analysis Examples

Discover additional workflows across categories. Each example includes prompts, conversation outputs, and model-level scoring so you can compare approaches and results.

Exploratory Data Analysis

Time Series

Data Analysis

Machine Learning

NLP

Finance

Statistics