In this tutorial, we will look at how to drop one or more columns from a Pyspark dataframe with the help of some examples.
How to drop Pyspark dataframe columns?
You can use the Pyspark drop()
function to drop one or more columns from a Pyspark dataframe. Pass the column (or columns) you want to drop as arguments to the function. The following is the syntax –
# drop column from dataframe df.drop("column1", "column2", ...)
It returns a Pyspark dataframe resulting from removing the passed column(s).
Examples
Let’s look at some examples of removing columns from a Pypsark dataframe. First, we will create a Pyspark dataframe that we’ll be using throughout this tutorial.
# import the pyspark module import pyspark # import the sparksession class from pyspark.sql from pyspark.sql import SparkSession # create an app from SparkSession class spark = SparkSession.builder.appName('datascience_parichay').getOrCreate() # data of competition participants data = [["Tim", 23, "Germany", "A"], ["Viraj", 22, "India", "A"], ["Emma", 24, "USA", "B"], ["Jack", 28, "USA", "B"], ["Max", 25, "Germany", "A"]] # create a Pyspark dataframe using the above data df = spark.createDataFrame(data, ["Name", "Age", "Country", "Team"]) # display df.show()
Output:
+-----+---+-------+----+ | Name|Age|Country|Team| +-----+---+-------+----+ | Tim| 23|Germany| A| |Viraj| 22| India| A| | Emma| 24| USA| B| | Jack| 28| USA| B| | Max| 25|Germany| A| +-----+---+-------+----+
We now have a dataframe containing the name, age, country, and team information of some students participating in a case-study competition.
Drop a column from Pyspark dataframe
Let’s drop the “Age” column from the above dataframe. For this, apply the drop()
function to the dataframe and pass “Age” as the argument.
# drop "Age" column df.drop("Age").show()
Output:
Introductory ⭐
- Harvard University Data Science: Learn R Basics for Data Science
- Standford University Data Science: Introduction to Machine Learning
- UC Davis Data Science: Learn SQL Basics for Data Science
- IBM Data Science: Professional Certificate in Data Science
- IBM Data Analysis: Professional Certificate in Data Analytics
- Google Data Analysis: Professional Certificate in Data Analytics
- IBM Data Science: Professional Certificate in Python Data Science
- IBM Data Engineering Fundamentals: Python Basics for Data Science
Intermediate ⭐⭐⭐
- Harvard University Learning Python for Data Science: Introduction to Data Science with Python
- Harvard University Computer Science Courses: Using Python for Research
- IBM Python Data Science: Visualizing Data with Python
- DeepLearning.AI Data Science and Machine Learning: Deep Learning Specialization
Advanced ⭐⭐⭐⭐⭐
- UC San Diego Data Science: Python for Data Science
- UC San Diego Data Science: Probability and Statistics in Data Science using Python
- Google Data Analysis: Professional Certificate in Advanced Data Analytics
- MIT Statistics and Data Science: Machine Learning with Python - from Linear Models to Deep Learning
- MIT Statistics and Data Science: MicroMasters® Program in Statistics and Data Science
🔎 Find Data Science Programs 👨💻 111,889 already enrolled
Disclaimer: Data Science Parichay is reader supported. When you purchase a course through a link on this site, we may earn a small commission at no additional cost to you. Earned commissions help support this website and its team of writers.
+-----+-------+----+ | Name|Country|Team| +-----+-------+----+ | Tim|Germany| A| |Viraj| India| A| | Emma| USA| B| | Jack| USA| B| | Max|Germany| A| +-----+-------+----+
The resulting dataframe has the “Age” column removed. Note that the drop()
function doesn’t modify the original dataframe in place. To modify the original dataframe, assign the resulting dataframe from the drop()
function to the original dataframe variable.
# drop "Age" column df = df.drop("Age") # display the dataframe df.show()
Output:
+-----+-------+----+ | Name|Country|Team| +-----+-------+----+ | Tim|Germany| A| |Viraj| India| A| | Emma| USA| B| | Jack| USA| B| | Max|Germany| A| +-----+-------+----+
You can see that the dataframe now doesn’t have the “Age” column.
Drop multiple columns from Pyspark dataframe
You can also use the drop()
function to remove more than one column from a Pyspark dataframe. Pass the columns you want to drop as arguments to the drop()
function.
For example, let’s drop the columns “Country” and “Team” from the above dataframe.
# drop "Country" and "Team" columns df.drop("Country", "Team").show()
Output:
+-----+ | Name| +-----+ | Tim| |Viraj| | Emma| | Jack| | Max| +-----+
You can see that the resulting dataframe has both columns removed.
You might also be interested in –
- Order PySpark DataFrame using orderBy()
- Filter PySpark DataFrame with where()
- Aggregate Functions in PySpark
Subscribe to our newsletter for more informative guides and tutorials.
We do not spam and you can opt out any time.