Calculate Standard Deviation in Python

Standard deviation is an important metric that is used to measure the spread in the data. It has useful applications in describing the data, statistical testing, etc. There are a number of ways in which you can calculate the standard deviation of a list of values in Python which is covered in this tutorial with examples.

In this tutorial, we will look at –

• What is standard deviation?
• Manually calculate standard deviation
• How to calculate standard deviation of a list in Python?
• Standard deviation of a numpy array
• Standard deviation of a pandas series

What is standard deviation?

Standard deviation is a measure of spread in the data. This means that if the standard deviation is higher, the data is more spread out and if it’s lower, the data is more centered. It is calculated by taking the square root of the variance. The following is the formula of standard deviation.

Note that the above is the formula for the population standard deviation. For sample standard deviation, we use the sample mean in place of the population mean and (sample size – 1) in place of the population size.

Both variance and standard deviation are measures of spread but the standard deviation is more commonly used. This is because the standard deviation is in the same units as the data.

Manually calculate standard deviation

Before we proceed to the computing standard deviation in Python, let’s calculate it manually to get an idea of what’s happening. For example, let’s calculate the standard deviation of the list of values [7, 2, 4, 3, 9, 12, 10, 1].

To calculate the standard deviation, let’s first calculate the mean of the list of values.

The mean comes out to be six (μ = 6).

Now, to calculate the standard deviation, using the above formula, we sum the squares of the difference between the value and the mean and then divide this sum by n to get the variance.

The variance comes out to be 14.5

The standard deviation can then be calculated by taking the square root of the variance.

How to calculate standard deviation in Python?

There are a number of ways to compute standard deviation in Python. You can write your own function to calculate the standard deviation or use off-the-shelf methods from numpy or pandas.

Let’s write a vanilla implementation of calculating std dev from scratch in Python without using any external libraries.

def get_std_dev(ls):
    n = len(ls)
    mean = sum(ls) / n
    var = sum((x - mean)**2 for x in ls) / n
    std_dev = var ** 0.5
    return std_dev

# create a list of data points
ls = [7, 2, 4, 3, 9, 12, 10, 2]
get_std_dev(ls)

Output:

3.8078865529319543

Here, we created a function to return the standard deviation of a list of values. Notice that we used the Python built-in sum() function to compute the sum for mean and variance. This function computes the sum of the sequence passed.

The above method is not the only way to get the standard deviation of a list of values. You can store the values as a numpy array or a pandas series and then use the simple one-line implementations for calculating standard deviations from these libraries.

Standard deviation of a numpy array

You can store the list of values as a numpy array and then use the numpy ndarray std() function to directly calculate the standard deviation. Here’s an example –

import numpy as np

# list of data points
ls = [7, 2, 4, 3, 9, 12, 10, 2]
# create numpy array of list values
ar = np.array(ls)
# get the standard deviation
print(ar.std())

Output:

3.8078865529319543

You can see that we get the same result as above.

Standard deviation of a pandas series

You can also store the list of values as pandas series and then compute its standard deviation using the pandas series std() function.

This method is very similar to the numpy array method. In fact, under the hood, a number of pandas methods are wrappers on numpy methods.

Let’s compute the standard deviation of the same list of values using pandas this time.

import pandas as pd

# list of data points
ls = [7, 2, 4, 3, 9, 12, 10, 2]
# create pandas series of list values
col = pd.Series(ls)
# get the standard deviation
print(col.std())

Output:

4.0708019567928595

You can see that the result is higher compared to the previous two examples. This is because pandas calculates the sample standard deviation by default (normalizing by N – 1). To get the population standard deviation, pass ddof = 0 to the std() function.

# get the standard deviation
print(col.std(ddof=0))

Output:

3.8078865529319543

Now we get the same standard deviation as the above two examples.

Note that pandas is generally used for working with two-dimensional data and offers a range of methods to manipulate, aggregate, and analyze data. For example, you can calculate the standard deviation of each column in a pandas dataframe.

With this, we come to the end of this tutorial. The code examples and results presented in this tutorial have been implemented in a Jupyter Notebook with a python (version 3.8.3) kernel having numpy version 1.18.5 and pandas version 1.0.5