10.2 The Pearson R in Python

Bivariate Numerical-to-Numerical Statistics

To explore how easily it is to calculate the Pearson correlation coefficient in Python, we will use the scipy.stats library, which provides a straightforward function to compute r. We will use this dataset from the palmerpenguins package, which contains measurements for different penguin species.

import pandas as pd
import scipy.stats as stats

url = "https://raw.githubusercontent.com/dataprofessor/data/refs/heads/master/penguins_cleaned.csv"
penguins = pd.read_csv(url)
penguins.head()

	species	island	bill_length_mm	bill_depth_mm	flipper_length_mm	body_mass_g	sex
0	Adelie	Torgersen	39.1	18.7	181	3750	male
1	Adelie	Torgersen	39.5	17.4	186	3800	female
2	Adelie	Torgersen	40.3	18.0	195	3250	female
3	Adelie	Torgersen	36.7	19.3	193	3450	female
4	Adelie	Torgersen	39.3	20.6	190	3650	male

The fastest way to calculate the Pearson correlation coefficient between two numerical columns is to use the pandas corr method. We will need to also need to pass in the numeric_only=True argument to avoid issues with non-numeric columns.

penguins.corr(numeric_only=True)

	bill_length_mm	bill_depth_mm	flipper_length_mm	body_mass_g
bill_length_mm	1.000000	-0.228626	0.653096	0.589451
bill_depth_mm	-0.228626	1.000000	-0.577792	-0.472016
flipper_length_mm	0.653096	-0.577792	1.000000	0.872979
body_mass_g	0.589451	-0.472016	0.872979	1.000000

df.corr() is a great place to start. It provides a matrix of correlation coefficients for all pairs of numerical columns in the DataFrame. Each cell represents the Pearson correlation coefficient between the two variables corresponding to that row and column. It’s a great place to start because we can instantly see which pairs of variables have strong positive or negative correlations. But it doesn’t give us all the information. Specifically, it doesn’t provide p-values to assess the statistical significance of these correlations.

Once we find a pair of variables that we are interested in, we can use the scipy.stats.pearsonr function to get both the Pearson correlation coefficient and the p-value. For instance, let’s calculate the correlation between flipper_length_mm and body_mass_g because it has a positive correlation in the matrix above. All we need to do is pass in the two columns as arguments to the function. The order of the arguments does not matter.

st = stats.pearsonr(penguins["flipper_length_mm"], penguins["body_mass_g"])
st

PearsonRResult(statistic=np.float64(0.8729788985653614), pvalue=np.float64(3.1328362509712445e-105))

The output is a tuple where the first element is the Pearson correlation coefficient (r), and the second element is the p-value. The r value indicates the strength and direction of the linear relationship between the two variables, while the p-value helps us determine whether this correlation is statistically significant.

We typically deconstruct the tuple into two separate variables for easier interpretation. Make sure to set the order correctly: the first variable is r, and the second is p.

r, p = stats.pearsonr(
    penguins["flipper_length_mm"], 
    penguins["body_mass_g"]
    )
print(f"Pearson r: {r}")
print(f"P-value: {p}")

Pearson r: 0.8729788985653614
P-value: 3.1328362509712445e-105

From the output, we can see that the Pearson correlation coefficient (r) is approximately 0.872, indicating a strong positive linear relationship between flipper length and body mass. The p-value is extremely small (close to zero), suggesting that this correlation is statistically significant. Unfortunately, it is so small that python is using scientific notation to represent it. 3.1224…e_105 means 3.1224 x 10^-105, which is a decimal point followed by 104 zeros and then 31224. In other words, it is effectively zero for our purposes. We can use :.4f to format the p-value to four decimal places, which will display it as 0.0000.

print(f"P-value: {p:.4f}")

P-value: 0.0000

From this output, we can confidently conclude that there is a statistically significant positive correlation between flipper length and body mass in penguins. This is because the r value is positive and above 0.7, and the p-value is effectively zero, indicating strong evidence against the null hypothesis of no correlation. In other words, if we ran this analysis on many different samples of penguins, we would expect to see a similar strong positive correlation between these two variables most of the time.