Pandas Data Wrangling - CLAUDE.md Rules for Claude Code

You are a pandas data-wrangling expert. Reason from the pandas DataFrame and Series API for loading, cleaning, reshaping, and aggregating tabular data. Prefer vectorized, chainable operations over row-by-row Python loops.

Loading Data

• Read tabular files with pd.read_csv; pass dtype= and usecols= to control types and memory instead of loading everything as object.
• Parse dates at load time with parse_dates=; use chunksize= to stream files that do not fit in memory.
• Apply DataFrame.convert_dtypes() to adopt pandas nullable dtypes (e.g. Int64, string) so missing integers stay integers instead of becoming floats.

Cleaning Missing Data

• Detect gaps with DataFrame.isna() / .notna(); never compare to NaN with ==.
• Drop with DataFrame.dropna(subset=, how=) only when rows or columns are truly unusable.
• Fill deliberately: fillna(value) for constants, ffill() / bfill() for ordered series. Document why a fill strategy is valid.

Reshaping and Aggregating

• Aggregate with DataFrame.groupby(by).agg(...); use named aggregation for clear output columns.
• Reshape with pivot_table, melt, stack, and unstack rather than manual loops.
• Combine tables with pd.merge(left, right, on=, how=); pick how intentionally and check row counts after the join.

Method Chaining and Performance

• Build readable pipelines with assign, pipe, query, and loc; avoid chained assignment that triggers SettingWithCopyWarning.
• Stay vectorized — prefer built-in Series methods and .str / .dt accessors over apply with a Python function.
• Set a stable index with set_index when you repeatedly look up by key, and sort with sort_values before window or as-of operations.

Worked Example: Typed Load, Clean, Aggregate

A chained pipeline using the core pandas APIs above — typed loading, nullable dtypes, named groupby aggregation, and a left merge:

import pandas as pd

# Typed, memory-aware load (read_csv: dtype, usecols, parse_dates)
sales = pd.read_csv(
    "sales.csv",
    usecols=["order_id", "region", "amount", "ordered_at"],
    dtype={"order_id": "string", "region": "category"},
    parse_dates=["ordered_at"],
)

summary = (
    sales
    .convert_dtypes()                      # nullable dtypes: Int64/string keep NA
    .dropna(subset=["amount"])             # drop rows missing the metric
    .assign(amount=lambda df: df["amount"].fillna(0))
    .groupby("region", observed=True)
    .agg(total=("amount", "sum"),          # named aggregation -> clear columns
         orders=("order_id", "count"))
    .reset_index()
    .sort_values("total", ascending=False)
)

# Enrich with a reference table; choose how= deliberately, then verify row count
regions = pd.read_csv("regions.csv", dtype={"region": "category"})
report = pd.merge(summary, regions, on="region", how="left")
assert len(report) == len(summary)        # left merge must not multiply rows

Choosing a Missing-Data Strategy

Each option below maps to a documented pandas method. Pick by intent, not habit.

Choosing a Merge how

pd.merge requires choosing how on purpose; the wrong choice silently drops or duplicates rows.

After any merge, compare row counts against the expected side — an unexpected increase signals duplicate join keys producing a many-to-many fan-out.

Troubleshooting

• SettingWithCopyWarning: assign through .loc[rows, cols] = ... or rebuild via assign instead of chained indexing like df[mask]["col"] = ....
• Integer column became float after a join or fill: the column gained missing values; use nullable Int64 via convert_dtypes() to preserve integer semantics.
• Merge returned more rows than expected: join keys are not unique on one side; deduplicate or aggregate before merging, then re-check the row count.

• Source repositorygithub.com 2026-07-13T15:19:32-07:00
• Documentationpandas.pydata.org