Advanced DataFrame Operations
Pandas has extensive functionality for working with dates and times. The datetime module in Python’s standard library provides the basic building blocks, while Pandas builds on this with its own Timestamp and Timedelta types.
To work with dates and times in a DataFrame, we often need to convert string representations of dates into Pandas datetime objects. This can be done using the pd.to_datetime() function.
import pandas as pd
# Sample data
data = {"date": ["2023-01-01", "2023-02-01", "2023-03-01"], "value": [10, 20, 30]}
df = pd.DataFrame(data)
# Convert the 'date' column to datetime
df["date"] = pd.to_datetime(df["date"])
df| date | value | |
|---|---|---|
| 0 | 2023-01-01 | 10 |
| 1 | 2023-02-01 | 20 |
| 2 | 2023-03-01 | 30 |
Once we have a datetime column, we can easily extract components such as the year, month, day, etc.
df["year"] = df["date"].dt.year
df["month"] = df["date"].dt.month
df["day"] = df["date"].dt.day
df| date | value | year | month | day | |
|---|---|---|---|---|---|
| 0 | 2023-01-01 | 10 | 2023 | 1 | 1 |
| 1 | 2023-02-01 | 20 | 2023 | 2 | 1 |
| 2 | 2023-03-01 | 30 | 2023 | 3 | 1 |
Pandas allows for easy date arithmetic using Timedelta objects. We can add or subtract time from datetime columns.
# Add 10 days to each date
df["date_plus_10_days"] = df["date"] + pd.Timedelta(days=10)
df| date | value | year | month | day | date_plus_10_days | |
|---|---|---|---|---|---|---|
| 0 | 2023-01-01 | 10 | 2023 | 1 | 1 | 2023-01-11 |
| 1 | 2023-02-01 | 20 | 2023 | 2 | 1 | 2023-02-11 |
| 2 | 2023-03-01 | 30 | 2023 | 3 | 1 | 2023-03-11 |
We can filter DataFrames based on date conditions.
# Filter rows where the date is after February 1, 2023
filtered_df = df[df["date"] > "2023-02-01"]
filtered_df| date | value | year | month | day | date_plus_10_days | |
|---|---|---|---|---|---|---|
| 2 | 2023-03-01 | 30 | 2023 | 3 | 1 | 2023-03-11 |
We can be even more specific here as well. For instance if we want to find only the rows that are in February, we can do that too.
# Filter rows where the month is February
february_df = df[df["date"].dt.month == 2]
february_df| date | value | year | month | day | date_plus_10_days | |
|---|---|---|---|---|---|---|
| 1 | 2023-02-01 | 20 | 2023 | 2 | 1 | 2023-02-11 |
Or if we want to find only rows in either January or February of the year:
# Filter rows where the month is in the last quarter (October, November, December)
last_quarter_df = df[df["date"].dt.month <= 2]
last_quarter_df| date | value | year | month | day | date_plus_10_days | |
|---|---|---|---|---|---|---|
| 0 | 2023-01-01 | 10 | 2023 | 1 | 1 | 2023-01-11 |
| 1 | 2023-02-01 | 20 | 2023 | 2 | 1 | 2023-02-11 |