How to compute covariance and correlation coefficients (in Python, using pandas and NumPy)

See all solutions.

Task

Covariance is a measure of how much two variables “change together.” It is positive when the variables tend to increase or decrease together, and negative when they upward motion of one variable is correlated with downward motion of the other. Correlation normalizes covariance to the interval $[-1,1]$.

Solution

We will construct some random data here, but when applying this, you would use your own data, of course.

1
2
3
4
5
import pandas as pd
import numpy as np
df = pd.DataFrame(np.random.rand(10,5))
df.columns = [ 'col1','col2','col3','col4','col5' ]
df.head()

If you have two pandas Series, you can compute the covariance of just those two variables. Note that every column in a DataFrame is a pandas series.

1
np.cov( df['col1'], df['col2'] )

1
2
array([[ 0.04524431, -0.02545402],
       [-0.02545402,  0.12901528]])

You can also compare all of a DataFrame’s columns among one another, each as a separate variable.

1
df.cov()

The Pearson correlation coefficient can be computed with np.corrcoef in place of np.cov.

1
np.corrcoef( df['col1'], df['col2'] )

1
2
array([[ 1.        , -0.33316075],
       [-0.33316075,  1.        ]])

And pandas DataFrames have a built in method to do this for all numeric columns.

1
df.corr()

Content last modified on 24 July 2023.

See a problem? Tell us or edit the source.

Contributed by Nathan Carter (ncarter@bentley.edu)