Pandas Data Cleaning Cheat Sheet

Posted : admin On 1/29/2022

First thing we need to do is read our data into pandas and take a look for ourselves. Import pandas as pd. Df = pd.readcsv('/user/home/test.csv') df.head Here we import pandas using the alias 'pd', then we read in our data. Df.head - shows us the first 5 rows and headers - it gives us an idea what to expect. Df.tail - shows us the last 5 rows. This cheat sheet will help you quickly find and recall things you've already learned about pandas; it isn't designed to teach you pandas from scratch! It's also a good idea to check to the official pandas documentation from time to time, even if you can find what you need in the cheat sheet.

For working with data in python, Pandas is an essential tool you must use. This is a fast, powerful, flexible and easy to use open source data analysis and manipulation tool, built on top of the Python programming language.

But even when you’ve learned pandas in python, it’s easy to forget the specific syntax for doing something. That’s why today I am giving you a cheat sheet to help you easily reference the most common pandas tasks.

It’s also a good idea to check to the official pandas documentation from time to time, even if you can find what you need in the cheat sheet. Reading documentation is a skill every data professional needs, and the documentation goes into a lot more detail than we can fit in a single sheet anyway!

Importing Data:

Use these commands to import data from a variety of different sources and formats.

Exporting Data:

Use these commands to export a DataFrame to CSV, .xlsx, SQL, or JSON.

Viewing/Inspecting Data:

Use these commands to take a look at specific sections of your pandas DataFrame or Series.

Selection:

Use these commands to select a specific subset of your data.

Data Cleaning:

Use these commands to perform a variety of data cleaning tasks.

Filter, Sort, and Groupby:

Use these commands to filter, sort, and group your data.

Join/Combine:

Use these commands to combine multiple dataframes into a single one.

Statistics:

These commands perform various statistical tests. (They can be applied to a series as well)

I hope this cheat sheet will be useful to you no matter you are new to python who is learning python for data science or a data professional. Happy Programming.

You can alsodownload the printable PDF file from here.

.

Data can be messy: it often comes from various sources, doesn’t have structure or contains errors and missing fields. Working with data requires to clean, refine and filter the dataset before making use of it.

Pandas is one of the most popular tools to perform such data transformations. It is an open source library for Python offering a simple way to aggregate, filter and analyze data. The library is often used together with Jupyter notebooks to empower data exploration in various research and data visualization projects.

Pandas introduces the concept of a DataFrame – a table-like data structure similar to a spreadsheet. You can import data in a data frame, join frames together, filter rows and columns and export the results in various file formats. Here is a pandas cheat sheet of the most common data operations:

Getting Started

Import Pandas & Numpy

Get the first 5 rows in a dataframe:

Pandas Data Cleaning Tutorial

Get the last 5 rows in a dataframe:

Import Data

Create DataFrame from dictionary:

Import data from a CSV file:

Import data from an Excel Spreadsheet:

Import data from an Excel Spreadsheet without the header:

Export Data

Export as an Excel Spreadsheet:

Export to a CSV file:

Convert Data Types

Convert column data to string:

Pandas functions cheat sheet

Convert column data to integer (nan values are set to -1):

Convert column data to numeric type:

Get / Set Values

Get the value of a column on a row with index idx:

Set column value on a given row:

Count

Number of rows in a DataFrame:

Count rows where column is equal to a value:

Count unique values in a column:

Count rows based on a value:

Pandas data science cheat sheet

Filter Data

Filter rows based on a value:

Filter rows based on multiple values:

Filter rows that contain a string:

Filter rows containing some of the strings:

Filter rows where value is in a list:

Filter rows where value is _not_ in a list:

Filter all rows that have valid values (not null):

Sort Data

Sort rows by value:

Sort Columns By Name:

Rename columns

Rename particular columns:

Rename all columns:

Make all columns lowercase:

Drop data

Drop column named col

Drop all rows with null index:

Drop rows that have missing values in some columns:

Drop duplicate rows:

Create columns

Create a new column based on row data:

Create a new column based on another column:

Create multiple new columns based on row data:

Match id to label:

Data Joins

Join data frames by columns:

Concatenate two data frames (one after the other):

Utilities

Increase the number of table rows & columns shown:

Learn More

Pandas Data Cleaning Cheat Sheet 2019

We are covering data analysis and visualization in our upcoming course “Data & the City”. The course will discuss how to collect, store and visualize urban data in a useful way. Subscribe bellow and we’ll notify you when the course becomes available.