8.1 EDA Basics

Univariate Statistics

Basic Descriptions of Data

Exploratory Data Analysis (EDA) is a crucial step in understanding the characteristics of your dataset. It involves summarizing the main features of the data, often using visual methods. In this section, we will explore some basic descriptions of data using univariate statistics.

The first way to get going on this is to use the df.describe() method in pandas, which provides a quick overview of the numerical columns in your DataFrame.

For our examples, we will begin working with data that comes loaded already in Seaborn. Seaborn is a powerful visualization library in Python that comes with several built-in datasets. You can also load your own datasets using pandas. In order to work with this data, you will need to uv add seaborn if you haven’t already. This should have been done in Appendix C.

from seaborn import load_dataset
import pandas as pd

# Load the Titanic dataset
df = load_dataset("titanic")

# Display basic statistics
df.describe()
survived pclass age sibsp parch fare
count 891.000000 891.000000 714.000000 891.000000 891.000000 891.000000
mean 0.383838 2.308642 29.699118 0.523008 0.381594 32.204208
std 0.486592 0.836071 14.526497 1.102743 0.806057 49.693429
min 0.000000 1.000000 0.420000 0.000000 0.000000 0.000000
25% 0.000000 2.000000 20.125000 0.000000 0.000000 7.910400
50% 0.000000 3.000000 28.000000 0.000000 0.000000 14.454200
75% 1.000000 3.000000 38.000000 1.000000 0.000000 31.000000
max 1.000000 3.000000 80.000000 8.000000 6.000000 512.329200

Pandas has a number of built-in methods to get basic descriptions of data. These include count(), nunique(), and unique() among others. We can also use value_counts() to see the distribution of categorical variables. These provide raw numbers of observations, unique values, and frequency counts. But sometimes there are nullish values in the data, and we need to be careful about how we handle them. Pandas also provides methods like isnull() and notnull() to help identify missing data.

To begin, we likely want to know the available columns in our DataFrame. We can use the columns attribute to get a list of all column names. Then we probably want to know the data types of each column. The two primary data categories are numerical and categorical. Numerical data can be further divided into continuous and discrete types, while categorical data can be nominal or ordinal. Understanding the data types is crucial for selecting appropriate statistical methods and visualizations.

  1. Numerical data
    • continuous: numerical values that can take any value within a range (e.g., height, weight)
    • discrete: numerical values that can only take specific, separate values (e.g., months in a year, shoe sizes)
  2. Categorical data
    • nominal: categories without an inherent order
    • ordinal: categories with a meaningful order

Luckily, we can use the pd.api.types module to help us identify the data type of each column by looping over all columns in the DataFrame.

for col in df.columns:
    if pd.api.types.is_numeric_dtype(df[col]):
        print(f"{col} is numerical")
    else:
        print(f"{col} is categorical")
survived is numerical
pclass is numerical
sex is categorical
age is numerical
sibsp is numerical
parch is numerical
fare is numerical
embarked is categorical
class is categorical
who is categorical
adult_male is numerical
deck is categorical
embark_town is categorical
alive is categorical
alone is numerical

We can now expand on this logic to create a summary of the DataFrame, including the count of non-null values, number of unique values, and data type for each column.

col_num = df.shape[1]
row_num = df.shape[0]
print(f"Dataframe has {row_num} rows and {col_num} columns.")
for col in df.columns:
    if pd.api.types.is_numeric_dtype(df[col]):
        print(f"{col}: numerical")
        print(f"\tUnique: {df[col].nunique()}")
        print(f"\tData Type: {df[col].dtype}")
        print(f"\tNulls: {df[col].isnull().sum()}")
        if df[col].isnull().sum() > 0:
            print(
                f"\tNote: {(df[col].isnull().sum() / row_num)*100:.2f}% of values are null."
            )
    else:
        print(f"{col}: categorical")
        print(f"\tUnique: {df[col].nunique()}")
        print(f"\tData Type: {df[col].dtype}")
        print(f"\tNulls: {df[col].isnull().sum()}")
        if df[col].isnull().sum() > 0:
            print(
                f"\tNote: {(df[col].isnull().sum() / row_num)*100:.2f}% of values are null."
            )
Dataframe has 891 rows and 15 columns.
survived: numerical
    Unique: 2
    Data Type: int64
    Nulls: 0
pclass: numerical
    Unique: 3
    Data Type: int64
    Nulls: 0
sex: categorical
    Unique: 2
    Data Type: str
    Nulls: 0
age: numerical
    Unique: 88
    Data Type: float64
    Nulls: 177
    Note: 19.87% of values are null.
sibsp: numerical
    Unique: 7
    Data Type: int64
    Nulls: 0
parch: numerical
    Unique: 7
    Data Type: int64
    Nulls: 0
fare: numerical
    Unique: 248
    Data Type: float64
    Nulls: 0
embarked: categorical
    Unique: 3
    Data Type: str
    Nulls: 2
    Note: 0.22% of values are null.
class: categorical
    Unique: 3
    Data Type: category
    Nulls: 0
who: categorical
    Unique: 3
    Data Type: str
    Nulls: 0
adult_male: numerical
    Unique: 2
    Data Type: bool
    Nulls: 0
deck: categorical
    Unique: 7
    Data Type: category
    Nulls: 688
    Note: 77.22% of values are null.
embark_town: categorical
    Unique: 3
    Data Type: str
    Nulls: 2
    Note: 0.22% of values are null.
alive: categorical
    Unique: 2
    Data Type: str
    Nulls: 0
alone: numerical
    Unique: 2
    Data Type: bool
    Nulls: 0

As you can see, the .nunique() method provides the number of unique values in each column, while the isnull().sum() method counts the number of missing values. This information is essential for understanding the completeness and variability of your data. These methods provide important context for further analysis and visualization. For example, only 2 unique values in the sex column indicates that it is a binary categorical variable, while 7 unique values in the embarked column suggests a nominal categorical variable with multiple categories. While the age column has 88 unique values, indicating it is a continuous numerical variable similar to fare with its 248 unique values. The pclass column has only 3 unique values, indicating it is a discrete numerical variable representing passenger class.

Moreover, we can see that there are not many null values in most of the columns, except for age and deck. The age column has 177 null values, which is about 19.87% of the total rows, while the deck column has a significant number of null values (688), which is about 77.04% of the total rows. This indicates that we may need to handle these missing values appropriately before conducting further analysis. We will discuss strategies for handling missing data in a later section.

Boundaries and Middle

To further understand the distribution of numerical data, we can look at measures of central tendency and spread. These include the minimum, maximum, mean, median, and mode, as well as quartiles and quantiles which divide the data into intervals. Pandas has built-in methods to calculate these statistics, such as min(), max(), mean(), median(), and mode().

mean = df["age"].mean()
med = df["age"].median()
mod = df["age"].mode()[0]
quantile = df["age"].quantile([0.25, 0.5, 0.75])
min_val = df["age"].min()
max_val = df["age"].max()

print(f"Mean: {mean}")
print(f"Median: {med}")
print(f"Mode: {mod}")
print(f"Quantiles:\n\t{quantile}")
print(f"Min: {min_val}")
print(f"Max: {max_val}")    
Mean: 29.69911764705882
Median: 28.0
Mode: 24.0
Quantiles:
    0.25    20.125
0.50    28.000
0.75    38.000
Name: age, dtype: float64
Min: 0.42
Max: 80.0

These statistics can be separated into two categories: measures of central tendency (mean, median, mode) and measures of spread (min, max, quartiles, quantiles).

  • central tendency: mean: the average value, median: the middle value when the data is sorted, mode: the most frequently occurring value
  • measures of spread: min: the minimum value, max: the maximum value, quartiles and quantiles: values that divide the data into intervals

Happily, pandas provides built-in methods to calculate these statistics, such as min(), max(), mean(), median(), and mode().

But we can also access them from the statistics library in python. If you have not already installed it, you can do so with Anaconda Navigator or by running uv add statistics in your terminal.

Once we have it we can do all of the above calculations with the statistics library, but we will need to drop the null values first. Also, statistics is typically aliased as stats so we will do that here.

import statistics as stats

age_no_null = df["age"].dropna()  # drop null valuse to calculate propertly
mean = stats.mean(age_no_null)
med = stats.median(age_no_null)
mod = stats.mode(age_no_null)
min_val = min(age_no_null)  # min is built-in function
max_val = max(age_no_null)  # max is built-in function
quartiles = stats.quantiles(age_no_null, n=4)  # quartiles
terciles = stats.quantiles(age_no_null, n=3)  # terciles
print(f"Mean: {mean}")
print(f"Median: {med}")
print(f"Mode: {mod}")
print(f"Quartiles: {quartiles}")
print(f"Terciles: {terciles}")
print(f"Min: {min_val}")
print(f"Max: {max_val}")
Mean: 29.699117647058824
Median: 28.0
Mode: 24.0
Quartiles: [20.0, 28.0, 38.0]
Terciles: [23.0, 34.0]
Min: 0.42
Max: 80.0

Notice that min() and max() are built-in Python functions, so we don’t need to import them from the statistics library. For our quantiles, we specify n=4 to get quartiles (dividing the data into four equal parts) and n=3 to get terciles (dividing the data into three equal parts). This means that the first quartile (Q1) is the value below which 25% of the data falls, the second quartile (Q2) is the median (50%), and the third quartile (Q3) is the value below which 75% of the data falls. Similarly, for terciles, the first tercile (T1) is the value below which 33.33% of the data falls, and the second tercile (T2) is the value below which 66.67% of the data falls.