Exact duplicates + fuzzy matches
Start with identical rows. Then find customer or company records that differ by spelling, punctuation, spacing, or abbreviations. MLJAR Studio writes real Python, runs it locally, and lets you review every match before changing the workbook.
Choose the right check
Run the exact check first. Use fuzzy matching only when records can describe the same customer without containing identical text.
Exact duplicates
Use pandas to count complete copies, check repeated customer IDs, and remove only the duplicates that contain no conflicting information.
Near-duplicates
Use RapidFuzz to shortlist similar names, then compare supporting fields and approve each match with an explicit survivor rule.
Demo outcomes
The exact and fuzzy examples use separate demo workbooks. Each result is checked against its own source before anything is saved.
30
exact duplicate rows removed
Demo: customers.xlsx
10
near-duplicate pairs reviewed
Demo: companies-with-duplicates.xlsx
$683,350
annual spend overstatement found
Demo: companies-with-duplicates.xlsx
Four checkpoints
AI handles repetitive code generation. You inspect the candidates, decide what the records mean, and approve the final workbook.
Count identical rows and repeated identifiers before applying a more subjective matching rule.
Use RapidFuzz to score company names that differ by spelling, punctuation, spacing, or abbreviations.
Compare contacts, email addresses, phone numbers, cities, dates, and values before accepting a pair.
Choose which record survives, verify the totals, and write a new workbook without changing the source file.
Visible Python
Prompts, generated code, review tables, and outputs remain together. Change the input file and rerun the checks when the next export arrives.
# Exact copies — safe to count before any fuzzy matching
exact_duplicate_rows = df.duplicated().sum()
print(f"Exact duplicate rows: {exact_duplicate_rows}")# Similar text — a candidate score, not a deletion decision
from rapidfuzz.fuzz import token_sort_ratio
score = round(token_sort_ratio(
"Mueller Technik GmbH",
"Muller Technik GmbH",
), 1)
print(score) # 97.4# Save only after the candidate pairs and survivor rule are reviewed
deduplicated_df.to_excel(
"companies-deduplicated.xlsx",
index=False,
)Exact equality is deterministic. A fuzzy score creates a candidate; a person still decides whether the records should be merged.
Human judgment
A parent company and subsidiary can share most of a name. Two branches may need separate records. Similar names only tell you where to look next.
The MLJAR Studio desktop app reads Excel files and runs Python locally. A cloud AI model may receive prompts, schema, and preview context. Connect a local model through Ollama, vLLM, LM Studio, Jan, or llama.cpp when that context must remain on your own hardware too.
Compare local and cloud LLMs
Reusable output
Write the reviewed rows to a new Excel file while preserving the original. The notebook records the prompts, generated Python, thresholds, matching evidence, and survivor rule used to create it.
Need to clean other fields first? Follow the category-standardization tutorial. When colleagues need to run the reviewed workflow without editing notebook cells, turn it into a Python internal tool.
Related guides
Follow the detailed method that matches your duplicate problem, or continue into a broader cleaning workflow.
Count complete copies, check repeated customer IDs, remove the safe duplicates, and save 500 reviewed rows.
Read nextUse RapidFuzz, inspect supporting fields, test the threshold, and merge ten reviewed company pairs.
Read nextCombine duplicate checks with missing-value handling, mixed-date parsing, and category standardization.
Read nextFAQ
Exact duplicates contain the same compared values. Fuzzy duplicates differ in some way — for example, a shortened company name, different punctuation, a typo, or another phone-number format — but may still represent the same real record.
No. Excel Remove Duplicates and pandas drop_duplicates compare values for equality. They do not infer that “GreenLeaf Foods” and “Green Leaf Foods” may refer to the same company. A fuzzy-matching step is needed to shortlist those records.
There is no universal threshold. In the demo, 70 finds all ten reviewed pairs, while 75 misses three real matches. Start with a candidate threshold, inspect false positives and missed matches, and adjust it for your names and business rules.
That is risky. Two subsidiaries, branches, or unrelated companies can have similar names. Treat a high score as a review candidate and compare other fields such as email domain, phone, address, city, and signup date before merging records.
Make the rule explicit. You might keep the earliest record, the latest verified record, the row with the most complete fields, or create a new row that combines useful information from both. MLJAR Studio writes the Python, but you approve the business rule.
No. The desktop app reads the workbook from your disk and runs Python locally. A cloud AI provider may receive prompts, column names, dataframe shape, and preview context. Connect Studio to a local LLM when that context must remain on your hardware too.
Yes. The conversation becomes a normal Jupyter notebook containing prompts, Python, and outputs. Change the input path and rerun the same exact checks, fuzzy scoring, review tables, totals, and export.
Use MLJAR Studio on your own Excel workbook and keep a person in control of every fuzzy match.