Learn how to handle missing values in Python pandas using structured detection, safe deletion, and statistical imputation, and be careful to avoid data leakage in your machine‑learning pipelines.
Quick answer
First find missing entries with `df.isna().sum()`.
If you want to remove them, use `df.dropna()`.
If you prefer to fill them, use `df.fillna()` or `df.interpolate()`.
For machine‑learning pipelines, apply scikit‑learn’s `SimpleImputer`.
Fit the imputer only on the training data so you avoid data leakage, rather than using pandas operations on the entire dataset.
Understanding Missing Data Sentinels in Pandas
Before learning how to handle missing values in Python pandas, you need to know how pandas shows missing data. In the past, pandas used only NumPy’s floating‑point NaN (Not a Number) to mark missing values. That works fine for numbers that are already floats, but it creates side effects. For instance, if an integer column has a missing entry, pandas automatically changes its type to float64. This type change can increase memory use and cause problems with strict database schemas. Python’s built‑in None is usually used for object‑type columns, like strings. To avoid the issues caused by float‑only NaNs, pandas added nullable data types.
These newer dtypes (such as Int64, Float64, and boolean) rely on a special marker, pd.NA. The marker lets a column keep its original integer or boolean type even when some rows are missing. When you build data‑ingestion pipelines, setting these nullable dtypes explicitly stops unwanted type conversions.
For a closely related example, see Train Test Split in Machine Learning Python Example: Avoiding Data Leakage.
Pandas Missing Value Sentinels Comparison
Pandas uses different representations for missing data depending on the column type. Understanding these differences helps avoid unexpected type casting.
| Factor | Engineering view | Why it matters |
|---|---|---|
| np.nan | Floating-point representation used for float columns. Automatically casts integer columns to float64. | Standard NumPy null |
| None | Python object-type null. Typically used in object or string columns. | Native Python |
| pd.NA | Pandas native null sentinel. Supports nullable integer, float, and boolean dtypes without type casting. | Introduced in pandas 1.0 |
Detecting and Quantifying Missing Data
Good data cleaning starts with a quick check. First, locate the missing values and see what share of the whole data they make up. In pandas you can use either isna() or isnull() to spot nulls; both give a DataFrame of True/False where True means the value is missing. If you want the exact number of missing entries in each column, add sum() to the mask you get from isna() or isnull(). To see how big the problem is relative to the column size, take the mean of that boolean mask. The result will be a number between 0 and 1 for each column, showing the percentage of missing rows.
This check helps you spot columns that have too many gaps to be useful.
Dropping Missing Values Safely
If only a few values are missing or the gaps are in columns that aren’t crucial, you can simply drop the affected rows or columns. The dropna() function lets you control how this works. By default, dropna() deletes any row that contains at least one missing value, which can remove a lot of data when you have many columns. To use dropping more safely, set the subset argument to the columns you care about. This way a row is removed only when a critical field is missing. The thresh argument lets you require a minimum number of non‑null entries before a row or column is kept.
You can also change the axis parameter from 0 to 1 to drop whole columns that don’t meet your quality threshold.
For a closely related example, see Model Predictive Current Control and Direct Torque Control for SRM With Cost Function‐Based Switching States to Optimize System Performance.
Choosing a Missing Value Strategy
Use this decision aid to choose the correct approach based on your dataset characteristics and downstream goals.
| Factor | Engineering view | Why it matters |
|---|---|---|
| High Missingness (>50% in a column) | Drop the entire column using df.dropna(axis=1). | Saves processing time; avoids introducing excessive noise |
| Low Missingness (<5% randomly spread) | Drop rows using df.dropna(subset=['col_name']). | Simple and quick; has minimal impact on statistical power |
| Skewed Numerical Data | Impute with the column median. | Prevents outliers from skewing the imputed values |
| Sequential or Time-Series Data | Use forward fill (ffill) or interpolation. | Preserves temporal trends and patterns |
| Categorical Data | Impute with the mode or create a new 'Missing' category. | Preserves categorical structure without adding bias |
Imputation Strategies for Numeric Features
When you can’t drop rows because the sample size is small or the missing data follows a pattern, you need to fill the gaps; this is called imputation. For numeric columns, pandas `fillna()` lets you replace missing values with a statistical estimate. The median is often chosen over the mean because it is less affected by extreme outliers. For data that has a natural order, such as time‑series measurements from IoT sensors, using a simple average can break the temporal pattern. In those cases, forward filling (`ffill`) or backward filling (`bfill`) is usually better. These methods copy the nearest known value forward or backward.
If you want smoother transitions, the `interpolate()` method can generate linear, polynomial, or time‑weighted estimates based on the surrounding data.
For a closely related example, see Spiking Neural Network-Based Neuromorphic Signal Processing for Real-Time Audio Event Detection in Low-Power Embedded Smart Sensors.
Handling Missing Categorical Values
Categorical columns can’t be filled with numeric means or medians. The easiest statistical trick is mode imputation: replace each missing value with the category that appears most often in that column. This works, but it can make the dominant class look even more common and it can hide the true variability of the data. A different option is to treat “missing” as its own category. You can change null entries to a label such as `'Unknown'` or `'Missing'`. This keeps the fact that the data was not available.
It is especially helpful when the missingness is systematic rather than random, because the lack of a value can itself be a useful signal for machine‑learning models.
Preventing Data Leakage in Machine Learning Pipelines
When learning how to handle missing values in Python pandas, many people mistakenly compute imputation statistics on the entire dataset before splitting it into training and test sets. Doing that lets information from the test set sneak into the training step, which makes the model look better than it really is. To avoid this leakage, split the data first. Compute the imputation numbers, like the median or the mode, using only the training part. Then use those same numbers to fill missing values in both the training and test sets.
You could do this by hand in pandas, but scikit-learn’s SimpleImputer inside a pipeline does the job more cleanly and works well for production.
Verifiable Implementation Example
Below is a complete, verifiable Python example demonstrating how to split a dataset, calculate imputation values on the training set, and apply them to the test set to prevent data leakage.
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.impute import SimpleImputer
# Construct a structured dataset with missing values
data = {
'sensor_a': [1.2, 1.5, np.nan, 2.1, 1.9, np.nan, 3.0, 3.2, np.nan, 4.0],
'sensor_b': [10.5, np.nan, 12.1, 11.8, np.nan, 14.2, 15.0, np.nan, 16.1, 17.5],
'label': [0, 0, 0, 1, 1, 1, 0, 1, 0, 1]
}
df = pd.DataFrame(data)
# Isolate features and targets
X = df[['sensor_a', 'sensor_b']]
y = df['label']
# Split into training and testing sets first
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.3, stratify=y, random_state=42
)
# Initialize the imputer to use the median value
imputer = SimpleImputer(strategy='median')
# Fit on training data ONLY and transform it
X_train_clean = imputer.fit_transform(X_train)
# Transform test data using the training medians
X_test_clean = imputer.transform(X_test)
# Convert back to pandas DataFrames for downstream work
X_train_imputed = pd.DataFrame(X_train_clean, columns=X_train.columns, index=X_train.index)
X_test_imputed = pd.DataFrame(X_test_clean, columns=X_test.columns, index=X_test.index)
Useful research starting points
Key takeaways
- Traditional pandas NaN values are floats and will force integer columns to upcast to float64.
- Use nullable data types like 'Int64' to store integers with missing values without changing the column type.
- Always check the proportion of missing values using df.isna().mean() before deciding on a cleanup strategy.
- When dropping rows with dropna(), use the subset parameter to avoid losing data from unrelated columns.
- To prevent data leakage, calculate imputation values using only the training dataset, never the entire dataset.
Questions engineers often ask
What is the difference between isna() and isnull() in pandas?
There’s no functional difference between isna() and isnull(); they are simply two names for the same underlying function. The library offers both so that users from different programming backgrounds feel comfortable: isnull() is familiar to SQL users, while isna() matches the naming used in R.
Why does pandas convert integers to floats when there are missing values?
Standard NumPy arrays can’t store missing values in integer columns. Because pandas originally built its core on NumPy, it used the floating‑point NaN to mark missing numbers. Since NaN is a float, pandas has to convert the whole integer column to float64 to hold it.
How can I drop columns only if they exceed a specific percentage of missing values?
You can calculate the missingness ratio for each column and use boolean indexing to filter them out. For example, to drop columns with more than 30% missing values, use: limit = len(df) * 0.7; df = df.dropna(thresh=limit, axis=1). This keeps only columns with at least 70% non-null values.
Is it better to use fillna() or scikit-learn's SimpleImputer?
For simple exploratory analysis, fillna() is faster and more convenient. However, for machine learning workflows, scikit-learn's SimpleImputer is preferred. SimpleImputer integrates into pipeline objects, making it easier to fit imputation parameters on the training set and transform the test set without risking data leakage.
Continue learning: how to handle missing values in python pandas
For more practical guides on data engineering and software architecture, explore our development resources.