Posts

Showing posts with the label 𝗨𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱𝗶𝗻𝗴 𝗦𝗽𝗮𝗿𝗸 𝗝𝗼𝗯𝘀

𝗨𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱𝗶𝗻𝗴 𝗦𝗽𝗮𝗿𝗸 𝗝𝗼𝗯𝘀, 𝗦𝘁𝗮𝗴𝗲𝘀, 𝗮𝗻𝗱 𝗧𝗮𝘀𝗸𝘀

𝗨𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱𝗶𝗻𝗴 𝗦𝗽𝗮𝗿𝗸 𝗝𝗼𝗯𝘀, 𝗦𝘁𝗮𝗴𝗲𝘀, 𝗮𝗻𝗱 𝗧𝗮𝘀𝗸𝘀 🚀𝟭. 𝗦𝗽𝗮𝗿𝗸 𝗝𝗼𝗯  🔶: A Spark job is a complete computation task that you submit to a Spark cluster, which includes all the actions and transformations you want to perform on your data.  🔶: It consists of multiple stages, each containing a sequence of tasks.  🔶: A job is triggered by an action (e.g., collect(), save(), count()) in Spark. Actions prompt the execution of all the preceding transformations.  🔶: Actions like collect(), save(), count(), and take().  🔶: Jobs are the high-level units of work in Spark, representing the full data processing task. They encompass everything from data reading to transformation and writing. 🚀𝟮. 𝗦𝘁𝗮𝗴𝗲  🔶A stage is a set of tasks that can be executed together without needing to shuffle data across the network.  🔶Stages are created during a job’s execution based on the transformations and actions in the data lineage. ...