In this tutorial, we will look at how to get the distinct values in a Pyspark column with the help of some examples.
How to get distinct values in a Pyspark column?
You can use the Pyspark distinct() function to get the distinct values in a Pyspark column. The following is the syntax –
# distinct values in a column in pyspark dataframe
df.select("col").distinct().show()
Here, we use the select() function to first select the column (or columns) we want to get the distinct values for and then apply the distinct() function.
Examples
Let’s look at some examples of getting the distinct values in a Pyspark column. First, we’ll create a Pyspark dataframe that we’ll be using throughout this tutorial.
# import the pyspark module
import pyspark
# import the sparksession class from pyspark.sql
from pyspark.sql import SparkSession
# create an app from SparkSession class
spark = SparkSession.builder.appName('datascience_parichay').getOrCreate()
# data of competition participants
data = [["Tim", "Germany", "A"],
["Viraj", "India", "A"],
["Emma", "USA", "B"],
["Jack", "USA", "B"],
["Max", "Germany", "A"]]
# create a Pyspark dataframe using the above data
df = spark.createDataFrame(data, ["Name", "Country", "Team"])
# display
df.show()
Output:
Highlighted programs for you
Flatiron School
Flatiron School
University of Maryland Global Campus
University of Maryland Global Campus
Creighton University
Creighton University
+-----+-------+----+ | Name|Country|Team| +-----+-------+----+ | Tim|Germany| A| |Viraj| India| A| | Emma| USA| B| | Jack| USA| B| | Max|Germany| A| +-----+-------+----+
We now have a dataframe containing the information on the name, country, and the respective team of some students in a case-study competition.
Distinct values in a single column in Pyspark
Let’s get the distinct values in the “Country” column. For this, use the Pyspark select() function to select the column and then apply the distinct() function and finally apply the show() function to display the results.
# distinct values in Country column
df.select("Country").distinct().show()
Output:
+-------+ |Country| +-------+ |Germany| | India| | USA| +-------+
You can see that we only get the unique values from the “Country” column – “Germany”, “India”, and “USA”.
Distinct values in multiple columns in Pyspark
You can also get distinct values in the multiple columns at once in Pyspark. For example, let’s get the unique values in the columns “Country” and “Team” from the above dataframe.
The syntax is similar to the example above with additional columns in the select statement for which you want to get the distinct values.
# distinct values in Country and Team columns
df.select("Country", "Team").distinct().show()
Output:
+-------+----+ |Country|Team| +-------+----+ |Germany| A| | India| A| | USA| B| +-------+----+
You can see that we get the distinct values for each of the two columns above.
You might also be interested in –
Subscribe to our newsletter for more informative guides and tutorials.
We do not spam and you can opt out any time.
