difference between sort by and order by

Published: None

Source: https://www.linkedin.com/pulse/difference-between-sort-order-arabinda-mohapatra-scgnc?trackingId=fJ3OTm4OQfCQnxnrWneABA%3D%3D

---

difference between sort by and order by

Arabinda Mohapatra

Running Kafka streams after dark, diving into genetic code by daylight, and wrestling with Databricks and Tableflow in every spare moment—sleep is optional

April 28, 2024

difference between sort by and order by

Order By and Sort By both are not same in sql. Order by will do sorting an entire data. sort by will do partition wise sorting

orderBy and sort both are same in pyspark. sortWintinPartitions as same as sort By in sql.

df=spark.read.format("csv")\

.option("header","True")\

.load("/FileStore/tables/Sample DataSource/emp.csv")

df.createOrReplaceTempView("df")

df.rdd.getNumPartitions()

df=df.repartition(4,"JOB")

df=df.withColumn("PARTITION_ID",spark_partition_id())

df.createOrReplaceTempView("df")

display(df)

df.rdd.getNumPartitions()
df=df.repartition(4,"JOB")

#Ascending
#df.orderBy(col("SAL")).show()
#descending order


df.orderBy(col("SAL").desc()).show()

 #df.sort("SAL").show()


df.sort(col("SAL").desc()).show()

df.sortWithinPartitions(desc(col("SAL"))).show()