Standard Deviation of Each Group in Pandas Groupby

You can use Pandas groupby to group the underlying data on one or more columns and estimate useful statistics like count, mean, median, std, min, max etc. In this tutorial, we will look at how to get the standard deviation of a column (or columns) for each group in pandas groupby with the help of some examples.

Pandas Groupby Standard Deviation

To get the standard deviation of each group, you can directly apply the pandas std() function to the selected column(s) from the result of pandas groupby. The following is a step-by-step guide of what you need to do.

1. Group the dataframe on the column(s) you want.
2. Select the field(s) for which you want to estimate the standard deviation.
3. Apply the pandas std() function directly or pass ‘std’ to the agg() function.

The following is the syntax –

# groupby columns on Col1 and estimate the std dev of column Col2 for each group
df.groupby([Col1])[Col2].std()
# alternatively, you can pass 'std' to the agg() function
df.groupby([Col1])[Col2].agg('std')

Examples

Let’s look at how to get the standard deviation for each group with the help of some examples. First, we will create a sample dataframe that we will be using throughout this tutorial.

import pandas as pd

# create a dataframe of car models by two companies
df = pd.DataFrame({
    'Company': ['A', 'A', 'A', 'B', 'B', 'B', 'B'],
    'Model': ['A1', 'A2', 'A3', 'B1', 'B2', 'B3', 'B4'],
    'Year': [2019, 2020, 2021, 2018, 2019, 2020, 2021],
    'Transmission': ['Manual', 'Automatic', 'Automatic', 'Manual', 'Automatic', 'Automatic', 'Manual'],
    'EngineSize': [1.4, 2.0, 1.4, 1.5, 2.0, 1.5, 1.5],
    'MPG': [55.4, 67.3, 58.9, 52.3, 64.2, 68.9, 83.1]
})
# display the dataframe
df

Output:

We now have a dataframe containing the specifications of the car models by two different companies. The “EngineSize” column is the size of the engine in litres and the “MPG” column is the mileage of the car in miles-per-gallon.

1. Groupby std of a Single Column

Let’s get the standard deviation of the mileage “MPG” column for each “Company” in the dataframe df. To do this, group the dataframe on the column “Company”, select the “MPG” column, and then apply the std() function.

# std dev in MPG for each Company
df.groupby('Company')['MPG'].std()

Output:

Company
A     6.115826
B    12.736921
Name: MPG, dtype: float64

You can see that std dev in “MPG” for company “A” is less than that of company “B”. This tells us that the mileage of cars from company “A” do not vary as much as the mileage for cars from company “B”.

Alternatively, you can also use the pandas agg() function on the resulting groups.

# std dev in MPG for each Company
df.groupby('Company')['MPG'].agg('std')

Output:

Company
A     6.115826
B    12.736921
Name: MPG, dtype: float64

We get the same result as above.

Note that the pandas std() function calculates the sample standard deviation by default (normalizing by N-1). To get the population standard deviation, pass ddof = 0 to the std() function. To see an example, check out our tutorial on calculating standard deviation in Python. Also, here’s a link to the official documentation.

If you want to see common descriptive stats for each group, like mean, median, standard deviation, etc., you can apply the pandas describe() function on the result of groupby.

# useful statistics on MPG for each Company
df.groupby('Company')['MPG'].describe()

Output:

You can see that we get the standard deviation as well as other common descriptive stats like mean, median, min, max, etc. for each group.

2. Groupby std of Multiple Columns

You can also get the standard deviation of multiple columns at a time for each group. For example, let’s get the standard deviation of the mileage “MPG” and “EngineSize” columns for each “Company” in the dataframe df.

# std dev in MPG and EngineSize for each Company
df.groupby('Company')[['MPG', 'EngineSize']].std()

Output:

Here we selected the columns that we wanted to compute the std dev on from the resulting groupby object and then applied the std() function. We already know that the standard deviation for “MPG” is smaller for company “A”. Here we additionally find that the standard deviation for “EngineSize” is larger for company “A” compared to “B”.

Let’s now do the same thing using the pandas agg() function.

# std dev in MPG and EngineSize for each Company
df.groupby('Company')[['MPG', 'EngineSize']].agg('std')

Output:

We get the same result as above.

With this, we come to the end of this tutorial. The code examples and results presented in this tutorial have been implemented in a Jupyter Notebook with a python (version 3.8.3) kernel having pandas version 1.0.5