get distinct values in a pyspark column

Pyspark – Get Distinct Values in a Column

In this tutorial, we will look at how to get the distinct values in a Pyspark column with the help of some examples.

How to get distinct values in a Pyspark column?

get distinct values in a pyspark column

You can use the Pyspark distinct() function to get the distinct values in a Pyspark column. The following is the syntax –

# distinct values in a column in pyspark dataframe"col").distinct().show()

Here, we use the select() function to first select the column (or columns) we want to get the distinct values for and then apply the distinct() function.


Let’s look at some examples of getting the distinct values in a Pyspark column. First, we’ll create a Pyspark dataframe that we’ll be using throughout this tutorial.

# import the pyspark module
import pyspark
# import the  sparksession class  from pyspark.sql
from pyspark.sql import SparkSession
# create an app from SparkSession class
spark = SparkSession.builder.appName('datascience_parichay').getOrCreate()

# data of competition participants
data = [["Tim", "Germany", "A"],
      ["Viraj", "India", "A"],
      ["Emma", "USA", "B"],
      ["Jack", "USA", "B"],
      ["Max", "Germany", "A"]]

# create a Pyspark dataframe using the above data
df = spark.createDataFrame(data, ["Name", "Country", "Team"])

# display


| Name|Country|Team|
|  Tim|Germany|   A|
|Viraj|  India|   A|
| Emma|    USA|   B|
| Jack|    USA|   B|
|  Max|Germany|   A|

We now have a dataframe containing the information on the name, country, and the respective team of some students in a case-study competition.

Distinct values in a single column in Pyspark

Let’s get the distinct values in the “Country” column. For this, use the Pyspark select() function to select the column and then apply the distinct() function and finally apply the show() function to display the results.

# distinct values in Country column"Country").distinct().show()


📚 Data Science Programs By Skill Level


Intermediate ⭐⭐⭐

Advanced ⭐⭐⭐⭐⭐

🔎 Find Data Science Programs 👨‍💻 111,889 already enrolled

Disclaimer: Data Science Parichay is reader supported. When you purchase a course through a link on this site, we may earn a small commission at no additional cost to you. Earned commissions help support this website and its team of writers.

|  India|
|    USA|

You can see that we only get the unique values from the “Country” column – “Germany”, “India”, and “USA”.

Distinct values in multiple columns in Pyspark

You can also get distinct values in the multiple columns at once in Pyspark. For example, let’s get the unique values in the columns “Country” and “Team” from the above dataframe.

The syntax is similar to the example above with additional columns in the select statement for which you want to get the distinct values.

# distinct values in Country and Team columns"Country", "Team").distinct().show()


|Germany|   A|
|  India|   A|
|    USA|   B|

You can see that we get the distinct values for each of the two columns above.

You might also be interested in –

Subscribe to our newsletter for more informative guides and tutorials.
We do not spam and you can opt out any time.


  • Piyush Raj

    Piyush is a data professional passionate about using data to understand things better and make informed decisions. He has experience working as a Data Scientist in the consulting domain and holds an engineering degree from IIT Roorkee. His hobbies include watching cricket, reading, and working on side projects.

  • Gottumukkala Sravan Kumar
Scroll to Top