Pyspark – Get Distinct Values in a Column

In this tutorial, we will look at how to get the distinct values in a Pyspark column with the help of some examples.

How to get distinct values in a Pyspark column?

You can use the Pyspark distinct() function to get the distinct values in a Pyspark column. The following is the syntax –

# distinct values in a column in pyspark dataframe
df.select("col").distinct().show()

Here, we use the select() function to first select the column (or columns) we want to get the distinct values for and then apply the distinct() function.

Examples

Let’s look at some examples of getting the distinct values in a Pyspark column. First, we’ll create a Pyspark dataframe that we’ll be using throughout this tutorial.

# import the pyspark module
import pyspark
  
# import the  sparksession class  from pyspark.sql
from pyspark.sql import SparkSession
  
# create an app from SparkSession class
spark = SparkSession.builder.appName('datascience_parichay').getOrCreate()

# data of competition participants
data = [["Tim", "Germany", "A"],
      ["Viraj", "India", "A"],
      ["Emma", "USA", "B"],
      ["Jack", "USA", "B"],
      ["Max", "Germany", "A"]]

# create a Pyspark dataframe using the above data
df = spark.createDataFrame(data, ["Name", "Country", "Team"])

# display 
df.show()

Output:

+-----+-------+----+
| Name|Country|Team|
+-----+-------+----+
|  Tim|Germany|   A|
|Viraj|  India|   A|
| Emma|    USA|   B|
| Jack|    USA|   B|
|  Max|Germany|   A|
+-----+-------+----+

We now have a dataframe containing the information on the name, country, and the respective team of some students in a case-study competition.

Distinct values in a single column in Pyspark

Let’s get the distinct values in the “Country” column. For this, use the Pyspark select() function to select the column and then apply the distinct() function and finally apply the show() function to display the results.

# distinct values in Country column
df.select("Country").distinct().show()

Output:

+-------+
|Country|
+-------+
|Germany|
|  India|
|    USA|
+-------+

You can see that we only get the unique values from the “Country” column – “Germany”, “India”, and “USA”.

Distinct values in multiple columns in Pyspark

You can also get distinct values in the multiple columns at once in Pyspark. For example, let’s get the unique values in the columns “Country” and “Team” from the above dataframe.

The syntax is similar to the example above with additional columns in the select statement for which you want to get the distinct values.

# distinct values in Country and Team columns
df.select("Country", "Team").distinct().show()

Output:

+-------+----+
|Country|Team|
+-------+----+
|Germany|   A|
|  India|   A|
|    USA|   B|
+-------+----+

You can see that we get the distinct values for each of the two columns above.

You might also be interested in –

• Display DataFrame in Pyspark with show()
• Filter PySpark DataFrame with where()