In this article, We will see how to order data in a Pyspark dataframe based on one or more columns with the help of examples.
How to order data in a Pyspark dataframe?
You can use the Pyspark dataframe orderBy
function to order (that is, sort) the data based on one or more columns. The following is the syntax –
DataFrame.orderBy(*cols, **kwargs)
The orderBy
function takes the following parameters –
- cols – The column or list of column names to sort by.
- ascending – Boolean or list of boolean. Use a list for multiple sort orders. By default, it sorts in ascending order.
Examples
Let’s look at some examples of using the orderBy
function to sort a Pyspark dataframe. First, let’s create a dataframe that we will be using throughout this tutorial.
#import the pyspark module import pyspark # import the sparksession class from pyspark.sql from pyspark.sql import SparkSession # create an app from SparkSession class spark = SparkSession.builder.appName('datascience_parichay').getOrCreate() # books data as list of lists df = [[1, "PHP", "Sravan", 250], [2, "SQL", "Chandra", 300], [3, "Python", "Harsha", 250], [4, "R", "Rohith", 1200], [5, "Hadoop", "Manasa", 700], ] # creating dataframe from books data dataframe = spark.createDataFrame(df, ['Book_Id', 'Book_Name', 'Author', 'Price']) # display the dataframe dataframe.show()
Output:
+-------+---------+-------+-----+ |Book_Id|Book_Name| Author|Price| +-------+---------+-------+-----+ | 1| PHP| Sravan| 250| | 2| SQL|Chandra| 300| | 3| Python| Harsha| 250| | 4| R| Rohith| 1200| | 5| Hadoop| Manasa| 700| +-------+---------+-------+-----+
We now have a dataframe containing information about books and their prices in a particular store.
Sort dataframe in ascending order
Let’s sort the above dataframe on the column “Price” in ascending order. We pass the column name, “Price” as an argument to the orderBy()
function.
# order by Price in ascending order dataframe.orderBy("Price").show()
Output:
Introductory ⭐
- Harvard University Data Science: Learn R Basics for Data Science
- Standford University Data Science: Introduction to Machine Learning
- UC Davis Data Science: Learn SQL Basics for Data Science
- IBM Data Science: Professional Certificate in Data Science
- IBM Data Analysis: Professional Certificate in Data Analytics
- Google Data Analysis: Professional Certificate in Data Analytics
- IBM Data Science: Professional Certificate in Python Data Science
- IBM Data Engineering Fundamentals: Python Basics for Data Science
Intermediate ⭐⭐⭐
- Harvard University Learning Python for Data Science: Introduction to Data Science with Python
- Harvard University Computer Science Courses: Using Python for Research
- IBM Python Data Science: Visualizing Data with Python
- DeepLearning.AI Data Science and Machine Learning: Deep Learning Specialization
Advanced ⭐⭐⭐⭐⭐
- UC San Diego Data Science: Python for Data Science
- UC San Diego Data Science: Probability and Statistics in Data Science using Python
- Google Data Analysis: Professional Certificate in Advanced Data Analytics
- MIT Statistics and Data Science: Machine Learning with Python - from Linear Models to Deep Learning
- MIT Statistics and Data Science: MicroMasters® Program in Statistics and Data Science
🔎 Find Data Science Programs 👨💻 111,889 already enrolled
Disclaimer: Data Science Parichay is reader supported. When you purchase a course through a link on this site, we may earn a small commission at no additional cost to you. Earned commissions help support this website and its team of writers.
+-------+---------+-------+-----+ |Book_Id|Book_Name| Author|Price| +-------+---------+-------+-----+ | 3| Python| Harsha| 250| | 1| PHP| Sravan| 250| | 2| SQL|Chandra| 300| | 5| Hadoop| Manasa| 700| | 4| R| Rohith| 1200| +-------+---------+-------+-----+
You can see that the output contains dataframe rows sorted by the “Price” column in ascending order. Note that the orderBy()
function sorts a column in ascending order by default and hence we didn’t need to explicitly specify it.
Sort dataframe in descending order
Let’s now sort the same dataframe on the “Price” column in descending order. This time, we pass False
as the value for the ascending
parameter.
# order by Price in descending order dataframe.orderBy("Price", ascending=False).show()
Output:
+-------+---------+-------+-----+ |Book_Id|Book_Name| Author|Price| +-------+---------+-------+-----+ | 4| R| Rohith| 1200| | 5| Hadoop| Manasa| 700| | 2| SQL|Chandra| 300| | 1| PHP| Sravan| 250| | 3| Python| Harsha| 250| +-------+---------+-------+-----+
You can see that the output contains dataframe rows sorted by the “Price” column in descending order.
Order dataframe by more than one column
You can also use the orderBy()
function to sort a Pyspark dataframe by more than one column. For this, pass the columns to sort by as a list. You can also pass sort order as a list to the ascending
parameter for custom sort order for each column.
Let’s sort the above dataframe by “Price” and “Book_Id” both in descending order.
# order by Price and Book_Id in descending order dataframe.orderBy(["Price", "Book_Id"], ascending=[False, False]).show()
Output:
+-------+---------+-------+-----+ |Book_Id|Book_Name| Author|Price| +-------+---------+-------+-----+ | 4| R| Rohith| 1200| | 5| Hadoop| Manasa| 700| | 2| SQL|Chandra| 300| | 3| Python| Harsha| 250| | 1| PHP| Sravan| 250| +-------+---------+-------+-----+
You can see that the output is sorted first on the “Price” column in descending order and then on the “Book_Id” column in descending order.
You might also be interested in –
- Display DataFrame in Pyspark with show()
- Pandas – Sort a DataFrame
- Randomly Shuffle Pandas DataFrame Rows
Subscribe to our newsletter for more informative guides and tutorials.
We do not spam and you can opt out any time.