6.0 DataFrame Basics

Choose Library:

Pandas

Polars

Data Frames are the core data structure for data analytics in both Python. They derive from the R programming language’s data.frame concept. Data frames are two-dimensional, tabular data structures that can hold heterogeneous data types across columns, making them ideal for handling structured data. The first implementation of data frames in Python came with the NumPy library that was released as Numeric back in 1996, but then modernized in 2006 with its transformation into NumPy. NumPy provided the foundational array structures and operations that paved the way for more advanced data manipulation libraries, but it only works with homogeneous data types within its arrays.

Note

Arrays are a grid of values, all the same type, and are indexed by a tuple of non-negative integers. The number of dimensions is the rank of the array; the shape of an array is a tuple of integers giving the size of the array along each dimension.

Arrays are homogeneous, meaning all elements must be of the same data type (e.g., all integers or all floats). In contrast, data frames can contain columns with different data types (e.g., integers, floats, strings) within the same structure.

Pandas filled an important gap beginning in 2008. Despite NumPy being amazing for numerical computations, it is not flexible to handle tabular data that includes various data types. In other words, it is not designed to work with the sort of categorical data that data analysts frequently encounter outside the scientific community. Pandas solves that very problem. It built on top of NumPy and provided powerful data manipulation capabilities, including support for heterogeneous data types, missing data handling, and intuitive data indexing. Since then, Pandas has become the de facto standard for data analysis in Python. Over the years between 2008 and now, it has seen widespread adoption in both academia and industry, becoming an essential tool for data scientists, analysts, and researchers. However, as datasets have grown larger and more complex, Pandas has been transformed and updated to meet these new challenges, and it has done so effectively. Nevertheless, it has also produced some inconsistent syntax and performance issues with very large datasets.

Polars is a newer library that was introduced in 2020 to address some of the limitations of Pandas, particularly in terms of performance and scalability. Polars is designed to be faster and more memory-efficient than Pandas, making it well-suited for handling large datasets. It achieves this through a combination of techniques, including lazy evaluation, multi-threading, and efficient memory management. Polars also offers a more consistent and expressive API compared to Pandas, which can help reduce the learning curve for new users. While Polars is still relatively new compared to Pandas, it has gained significant traction in the data science community and is increasingly being adopted for data analysis tasks.

This textbook will introduce all the core concept moving forward in both Pandas and Polars, allowing you to choose the library that best fits your needs and preferences.

Installing and Importing Pandas

Note

For more on Pandas, see the Pandas documentation.

Pandas is a library designed for reading and manipulating data in Python. Pandas provides data structures and functions that make it easy to work with structured data, such as tables or time series. It is built on the top of NumPy, a library that is the cornerstone of mathematical and scientific computing for Python. However, whereas NumPy is built to work with homogenous (where all data in the dataset are of the same type) datasets, Pandas extends its capabilities to hetergenous datasets and time-series data. As the Pandas documentation states,

pandas is a Python package providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real-world data analysis in Python. Additionally, it has the broader goal of becoming the most powerful and flexible open source data analysis/manipulation tool available in any language.

Pandas is a powerful tool for data analysis and manipulation, and it is widely used in data science, machine learning, and statistical analysis. It provides a wide range of functions for reading and writing data, filtering and transforming data, and performing statistical operations. Pandas is also highly optimized for performance, making it suitable for working with large datasets. In fact, Pandas will be the key focus throughout the rest of this book.

Tip

Find the Pandas Cheat Sheet here.

Downloading and Importing Modules

To use Pandas, you first need to install it. Pip, the official package manager for Python, makes it easy to install and manage Python libraries. You can think of a package manager like Pip as similar to the App Store on Apple devices or the Play Store on Android. It provides a centralized location to download and manage software. In fact, package managers like Pip existed long before app stores. Pip is included with Python when you install it, so you already have it on your system. To use Pip, you need to use a terminal, a text-based interface that allows you to interact with your operating system by typing commands. Through the terminal, you can execute tasks such as navigating the file system, running programs, and managing processes.

If you’re using Windows, I recommend Git Bash, which is installed alongside Git. Git Bash is a terminal emulator that provides a Bash shell, enabling you to run Unix-like commands on Windows. For more details, see the Git Setup section. Most programming workflows rely on Unix-like environments, which use Bash commands. If you’re on macOS, you can use the built-in Apple Terminal because macOS is a Unix-like system. The following commands will work seamlessly in the Apple Terminal.

Here is how you can install Pandas using Pip:

pip install Pandas

Luckily, you don’t have to use the terminal for these commands. We downloaded Anaconda (see the Anaconda setup page{.target=“_blank” .rel=“noreferrer noopenner”}, if you do not have Anaconda downloaded) earlier to make this process easier. Use the following steps in the Anaconda Navigator to download Pandas.

On the Environments page, select myenvironment (or whatever name you gave it) from the environment list to activate it.
Open the package filter dropdown and select “Not installed” to view a list of packages that are available to install from the channels you are connected to.
In the Search packages box, type “pandas” to filter the list of packages.
Select the checkbox beside pandas in the package list.
Click the Apply button to install the package.

A dialog will appear that lists all packages to be installed. Click Apply.

Now you can type “Pandas” into the Search-packages box and the newly-installed Pandas package will now be displayed in the list of installed programs in the current environment.

For more on installing packages, see the Anaconda documentation{.target=“_blank” .rel=“noreferrer noopener”}.

Importing Pandas

To access Pandas and its tools, we have to import it into our Python code. We can do this by using the import statement. The convention is to import Pandas as pd, which allows us to use the shorthand pd instead of typing out Pandas every time we want to use it.

import pandas as pd

This import will import the entire Pandas library. For our purposes, it is perfectly acceptable to do so. But know that Pandas is a very large library, and its entirety will be included within the codebase. If you are sending your program to run on a server, it is not advisable to import an entire library this way. Instead, only import the parts of the library (each function, class, etc.) that you actually use in the project. We can import only specific aspects this way:

from pandas import Series

If we want to import multiple specific functions or classes, we can separate each one with a comma.

from pandas import Series, DataFrame

The Pandas Series

A Series is a one-dimensional labeled array capable of holding any data type (integers, strings, floating point numbers, Python objects, etc.). It is similar to a column in a spreadsheet or a SQL table. Think of a series as similar to a list, but with the major exception that each index has an explicit index that can be either numeric or a label (a list can only use implicit intiger-based indexing starting at 0). A Series can be created from a list, dictionary, or NumPy array. Each element in a Series has an associated label, which is used to access the elements. The labels are called the index of the Series.

Creating a Series

Array-like objects include Pandas Series objects, lists, tuples,

The Pandas DataFrame

A DataFrame is a two-dimensional, size-mutable, potentially heterogeneous tabular data structure with labeled axes (rows and columns). It is similar to a spreadsheet or SQL table, or a dictionary of Series objects. DataFrames are generally the most commonly used Pandas object.

Creating a DataFrames

A DataFrame, in its simplest form, is a matrix, a collection of Pandas Series objects.

[[1, 2, 3],
 [4, 5, 6],
 [7, 8, 9]]

Together they form columns and rows. In this way, we can create a DataFrame as rows or as columns. To create the DataFrame from a collection of columns, we use a dictionary of array-like object (the same types of data that can form a Pandas Series as well as Pandas Series). When we construct a DataFrame this way, we automatically create the names for each column as well, which derive from the keys of the dictionary.

import pandas as pd

data = {
    'A': [1, 2, 3],
    'B': [4, 5, 6],
    'C': [7, 8, 9]
}

df = pd.DataFrame(data=data)
df

As you can see, each list in the dictionary becomes a column in the DataFrame. The keys of the dictionary become the names of the columns. The index of the DataFrame is automatically generated, starting at 0 and incrementing by 1 for each row.

When we create a DataFrame by row, we use a list of array-like objects. But we have to create the column names separately.

data1 = [
    [1, 2, 3],
    [4, 5, 6],
    [7, 8, 9]
]

column_names = ['A', 'B', 'C']

df1 = pd.DataFrame(data=data1, columns=column_names)

Now let’s look at how Pandas DataFrames stores the data we include. We will use the french-ambs.csv dataset, which contains information on French ambassadors from the 16th century.

Installing and Importing Polars

Note

For more on Polars, see the Polars documentation.

Polars is a library (or collection of pre-written code) that is not included with Python. It provides extensive structures and functions that make it easy to work with very large tabular data. Unlike Pandas, Polars is build in Rust, a systems programming language that is designed for speed and safety. As a result, Polars can handle larger datasets and perform operations faster than Pandas in many cases.

Polars is the new kid on the block in data analytics with Python. It was first released in 2020, whereas Pandas has been around since 2008. However, Polars has quickly gained popularity due to its speed and efficiency, especially when working with large datasets. Polars is designed to be easy to use, with a syntax that is similar to Pandas, making it easy for users familiar with Pandas to transition to Polars. In fact, many think it provides a more expressive and user-friendly API than Pandas—if also more verbose.

Tip

Find the Polars Cheat Sheet here.

Downloading and Importing Modules

To use Polars, you first need to install it. Pip, the official package manager for Python, makes it easy to install and manage Python libraries. You can think of a package manager like Pip as similar to the App Store on Apple devices or the Play Store on Android. It provides a centralized location to download and manage software. In fact, package managers like Pip existed long before app stores. Pip is included with Python when you install it, so you already have it on your system. To use Pip, you need to use a terminal, a text-based interface that allows you to interact with your operating system by typing commands. Through the terminal, you can execute tasks such as navigating the file system, running programs, and managing processes.

Here is how you can install Polars using Pip:

pip install polars

Luckily, you don’t have to use the terminal for these commands. We downloaded Anaconda (see the Anaconda setup page, if you do not have Anaconda downloaded) earlier to make this process easier. Use the following steps in the Anaconda Navigator to download Polars.

On the Environments page, select myenvironment (or whatever name you gave it) from the environment list to activate it.
Open the package filter dropdown and select “Not installed” to view a list of packages that are available to install from the channels you are connected to.
In the Search packages box, type “polars” to filter the list of packages.
Select the checkbox beside polars in the package list.
Click the Apply button to install the package.

A dialog will appear that lists all packages to be installed. Click Apply.

Now you can type “polars” into the Search-packages box and the newly-installed Polars package will now be displayed in the list of installed programs in the current environment.

For more on installing packages, see the Anaconda documentation.

If you want to use Anaconda from the command line (aka terminal), you can also install Polars using the following command:

conda install -c conda-forge polars

Importing Polars

To access Polars and its tools, we have to import it into our Python environment. We can do this by using the import statement. The convention is to import Polars as pl, which allows us to use the shorthand pl instead of typing out polars every time we want to use it.

import polars as pl

This import will import the entire Polars library. For our purposes, it is perfectly acceptable to do so. But know that Polars is a very large library, and its entirety will be included within the codebase. If you are sending your program to run on a server, it is not advisable to import an entire library this way. Instead, only import the parts of the library (each function, class, etc.) that you actually use in the project. We can import only specific aspects this way:

from polars import DataFrame

If we want to import multiple specific functions or classes, we can separate each one with a comma.

from polars import DataFrame, Series

The Polars Series

The Polars series is the building block of the Polars DataFrame. A Series is a one-dimensional array similar to a Python list or a NumPy array, but with additional functionality and features. Each Series has a name and a data type, and it can hold various types of data, including integers, floats, strings, and more. Series are designed to be efficient and fast, allowing for quick data manipulation and analysis. Unlike a Pandas Series, a Polars Series is immutable, meaning that once it is created, its values cannot be changed. Instead, any operation that modifies a Series will return a new Series with the updated values. In addition, the Polars Series does not support labeled indexing like Pandas; instead, it relies on positional indexing.

Let’s create a simple Series in Polars and see some of the operations we can perform on it.

We will create a Series from a list of integers.
1. The most important argument in the construction of a Series is the data: it can accept a Sequence, a Series, a pyarrow Array, and numpy ndarray.
We will generate the mean of the Series.
We will filter the Series to include only values greater than a certain threshold.
We will demonstrate how to perform arithmetic operations on the Series.

import polars as pl

data = [10, 20, 30, 40, 50]
series = pl.Series(data)
series

shape: (5,)


i64
10
20
30
40
50

mean_value = series.mean()
mean_value

30.0

We can filter the Series using boolean indexing with the .filter() expression, which accepts the

filtered_series = series.filter(series > 25)
filtered_series

shape: (3,)


i64
30
40
50

The Polars DataFrame

See more on the Polars documentation

A Polars DataFrame is a two-dimensional collection that represents data as a table with columns and row. Unlike Pandas, rows are not named and given a non-negative integer as their index. Creating a datframe is fairly simple. The data parameter, the first parameter ordinally, can be

import polars as pl
from IPython.display import display

data = {"a": [1,2,3], "b": [4,5,6], "c": [7,8,9]}
data2 = [[1,2,3], [4,5,6], [7,8,9]]
df = pl.DataFrame(data)
df2 = pl.DataFrame(
  data2, 
  schema=["a", "b", "c"], 
)
display("df:")
display(df)
display("df2:")
display(df2)

'df:'

shape: (3, 3)

a	b	c
i64	i64	i64
1	4	7
2	5	8
3	6	9

'df2:'

shape: (3, 3)

a	b	c
i64	i64	i64
1	4	7
2	5	8
3	6	9

When creating a data frame with a dictionary, the keys of the dictionary will act as the column names. When creating a data frame from other collections (such as a list as df2 does above or a Polars Series or an Arrow object), one must set the schema. What is neat about Polars is that types are declared and cannot change. This implements type safety which help prevent bugs. The schema not only declares the name of the columns but can also be used to declare the data types for each column.