This is my first blog and am going to start with current trending topic :
Spark.
Apache Spark is a booming technology nowadays. Hence it is very important
to know each and every aspect of Apache Spark as well as Spark Interview Questions.
Here I am going to share the top 35 Spark Questions with answers
which are asked now a days in IT companies Interview.
All these questions are based on my personal experience by attending multiple IT companies Interview in the Year of 2020.
I assume, reader would have good understanding of all Spark concept
prior going through this content. Answer to each question
is very precise and can be referred by all Spark experienced folks who
are looking for job change and hopefully it can help them to crack most of
the Spark Interviews. Please note, spark fresher / beginner should
understand spark concepts in details first before going through it.
So let's start:
Question-1: What is the difference between Mapreduce and Spark? OR How does Spark achieve 100x times faster performance than Mapreduce ?
Answer:
Main difference between Hadoop MapReduce and Spark
processing are mentioned below:
1. MapReduce has to read from
and write to a disk, as a result, it slows down the processing speed while
Spark can do it in-memory. As a result, the speed of processing differs
significantly ans hence Spark may be up to 100 times faster than
MapReduce.
2. The volume of data processed also differs: Hadoop
MapReduce is able to work with far larger data sets than Spark.
3.
Map reduce process data in batch mode whereas with Spark Streaming, near
real time data can also be processed
4. MapReduce is complex. As a
result, we need to handle low-level APIs to process the data, which
requires lots of hand coding whereas Spark process the data using
high-level operators. It also provides rich APIs in Java, Scala, Python,
and R.
5.Hadoop MapReduce is developed in Java whereas Spark is
developed in Scala.
6. There is no interactive shell in Hadoop
MapReduce. Spark provides REPL(Read Evaluate Print Loop).
Question-2: What is the difference between RDD and Dataframe? OR What is there in DF but not in RDD?
Answer:
There are lot many differences between the two. Few are mentioned
below:
1. RDDs are Resilient distributed Datasets where data is stored in form
of objects. It has no schema. Whereas in Dataframes, data is organized
into named columns similar to table structure.
3. RDD is low level of API, whereas Dataframe is high level API.
4. No inbuilt optimization engine is available in RDD. But Optimization
takes place in dataframes using catalyst optimizer.
5. RDDs are typed-safe hence can get analysis errors at compile time. On
the other hand, a DataFrames are untyped and can lead to runtime
errors.
6. RDD is slower in performing simple grouping and aggregation operations
as compared to DataFrame.
Question-3: What is the difference between Dataframe and Datasets?
Answer:
1. Dataframe and Datasets both are high level APIs.
2. Data is organized into named columns both in Dataframe and Datasets.
Basically, they are similar to tables in a relational
database.
3. Dataset is an extension of DataFrame API that provides the
functionality of – type-safe, object-oriented programming interface of
the RDD API and performance benefits of the Catalyst query optimizer and
off heap storage mechanism of a DataFrame API.
4. Datasets are DataFrames with the types of the columns defined. When
converting a Dataset to DataFrame only the type info is lost otherwise
the object is the same and vica versa i.e. you can convert a DataFrame
to Dataset by defining the types of the columns in a case class.
5. Dataset brings the best of both worlds with a mix of relational
(DataFrame) and functional (RDD) transformations.
6. If you are trying to access the column which does not exist in the
table in such case Dataframe APIs does not support compile-time error.
It detects attribute error only at runtime. whereas Datasets provides
compile-time type safety.
Question-4: Define RDD in Spark.
Answer:
Spark uses RDD (Resilient Distributed Dataset) which is a
fault-tolerant collection of elements that can be operated in
parallel. RDDs are distributed collection of objects. Distributed means, each RDD
is divided into multiple partitions. Each of these partitions can
reside in memory or stored on disk of different machines in a
cluster.
They are immutable (Read Only) in nature. You can’t change original RDD, but you can always transform it into different RDD with all changes you want
RDDs can be created by 2 ways:
1. Parallelizing existing collection(sc.parallelize())
2. Loading external dataset from HDFS(sc.textFile())
Question-5: What are RDD Operations in Spark?
Answer:
Spark RDD supports two types of Operations:
Transformations - are lazy operations on a RDD that create one
or many new RDDs. It takes RDD as input and produces one or more RDD
as output. Each time it creates new RDD when we apply any
transformation.
After the transformation, the resultant RDD is always different from
its parent RDD.
It can be smaller (e.g. filter, count, distinct,
sample), bigger (e.g. flatMap(), union(), Cartesian()) or the same
size (e.g. map).
Some frequently used RDD Transformations are - map(), flatmap(),
filter(), union(), distinct(), reduceByKey(),
groupByKey(),coalesce(),repartition(), join() etc.
Actions - Actions are the functions that perform some kind of computation over the transformed RDD and sends the computed result from executors to driver.
Actions triggers execution using lineage graph to
load the data into original RDD, carry out all intermediate
transformations and return final results to Driver program or write it
out to file system.
Some frequently used RDD Actions are - reduce(func), collect(),
count(), first(), take(), takeSample(), saveAsTextFile(),
foreach(func) etc.
Question-6: What are Narrow and Wide transformations?
Answer:
Transformations can further be divided into 2 types:
Narrow transformation: A pipeline of operations that can be
executed as one stage and does not require the data to be shuffled
across the partitions —
For example, Map, flatmap, filter, sample, union, etc..
Wide transformation: Here each operation requires the data
to be shuffled, henceforth for each wide transformation a new stage
will be created —
For example, Intersection, distinct, reduceByKey, groupByKey, join,
repartition, coalesce, etc..
Note: When compared to Narrow transformations, wider
transformations are expensive operations due to shuffling.
Question-7: What are the Shared Variables in Spark?
Answer:
There are two different types of Shared Variables in spark
-Broadcast Variable and Accumulator:
Broadcast Variable - Broadcast variables are read-only
variables that are distributed across worker nodes in-memory instead
of shipping a copy of data with tasks. The data broadcasted this way
is cached in a serialized form and deserialized before running each
task. They are used to cache a value in memory on all nodes.
Generally small datasets can be broadcasted.
Accumulator - Accumulators are shared variables which are
used to update the variables in parallel during execution and share
the results from workers to the driver program running on master
node. They are similar to counters in map-reduce. Are those
variables that are used to perform associative and commutative
operations such as counters or sums.
Question-8: What is the difference between Persist and Cache?
Answer:
Cache and Persist are optimization techniques for both iterative
and interactive Spark applications to improve the performance of the
jobs or applications.
Iterative means Reuse intermediate results. Whereas interactive
means allowing a two-way flow of information.
Spark provides an optimization mechanism to store the intermediate
results of an RDD, DataFrame, and Dataset, so they can be reused in
subsequent stages.
Both caching and persisting are used to save the intermediate
resultsets but, the difference among them is that cache() will cache
the RDD into memory(MEMORY_ONLY), whereas persist(level) can cache
in memory, on disk, or off-heap memory according to the caching
strategy specified by level. Those storage levels are like
MEMORY_ONLY, MEMORY_AND_DISK, MEMORY_ONLY_SER, MEMORY_AND_DISK_SER,
DISK_ONLY
Note:
- persist() without an argument is equivalent with cache().
- unpersist() can be used to Freeing up space from the Storage memory.
Question-9: What is Spark checkpointing and how it is different from Persist to disk?
Answer:
“When should I cache or checkpoint?”
- to determining if the
results of a set of transformations can be reused for a very long
time or not.
If the answer is yes, use
checkpointing. If the answer is no, use caching.
Caching is extremely effective and more useful than checkpointing,
as it’s typically much faster when used properly, when you have a
lot of available memory, as RDDs and DataFrames can be massive in
size, like giga or terabyte size. Essentially, caching will maintain
the result of applying a set of transformations to an RDD or
DataFrame so that those transformations will not have to be
recomputed again when an additional transformation is applied or
fault occurs within a RDD or DataFrame.
Checkpointing will persist the transformed RDD or DataFrame
forever. Now you’re thinking “this is great, I can throw away old
history for free and not have Spark need to redo transformations
from time to time like caching.” Well not for free exactly. The main
problem with checkpointing is that Spark must be able to persist any
checkpoint RDD or DataFrame to HDFS which is slower and less
flexible than caching. You also need to setup checkpointing to a
location on HDFS, where a RDD or DataFrame’s transformations can be
persisted, whereas caching is part of Spark’s implicit default
setup.
Question-10: What is SparkSession and how it is different from SparkContext?
Answer:
Spark Context -
SparkContext is the entry point of Apache Spark
functionality.
It allows your Spark Application to access Spark Cluster with the
help of Resource Manager
To create SparkContext, first SparkConf should be made, which had
all the cluster configs and parameters to create a Spark Context
object.
Only one SparkContext may be active per JVM, its a expensive
operation
Spark Session -
Spark session is a unified entry point of a spark application from
Spark 2.0
Spark session is a combination of spark context, hive context and
SQL context
Spark session provides with spark.implicits._ which is one of the
most useful imports in all of the spark packages which comes in
handy with a lot of implicit methods for converting Scala objects to
Datasets and some other handy utils.
You can think of Spark Session as a data structure where the driver
maintains all the information including the executor location and
their status.
Question-11: What parameters to specify in spark-submit command -
Answer:
To submit a Spark job, below command has to be
triggered:
Parameters can be passed as per your project
configuration
spark-submit --num-executers a --executor-cores b --driver-memory c
-- executor-memory d --class com.spark.examples.TestSparkJob
--master yarn --deploy-mode cluster
/path/to/examples.jar
Values of a,b,c,d can be substituted as below:
a= No of executors required. Eg: 15
b= No of cores per executor. 5 cores(default)
c= Driver memory. Eg: 1G
d= Executor memery. Eg: 2G
Question-12: What are the different types of Cluster manager available in Spark?
Answer:
Apache Spark supports four different cluster managers:
Apache YARN - YARN is the cluster manager for Hadoop. As of
date, YARN is the most widely used cluster manager for Apache
Spark.
Apache Mesos - Apache Mesos is another general-purpose
cluster manager. If you are not using Hadoop, you might be using
Mesos for your Spark cluster.
Kubernetes - it's a general purpose container orchestration
platform from Google.
Standalone - Finally, the standalone. The Standalone
is a simple and basic cluster manager that comes with Apache Spark
and makes it easy to set up a Spark cluster very quickly. This is
basically used during development.
No matter which cluster manager do we use, primarily, all of them
delivers the same purpose.
Question-13: What are deploy modes in Spark?
Answer:
When you start an application, you have a choice to specify the
execution mode, and there are three options - Local, Client &
Cluster :
1. Client Mode -
In Client mode, driver starts on the local machine and then as soon
as the driver create a Spark Session, a request goes to YARN
resource manager to create a YARN application. The YARN resource
manager starts an Application Master. For the client mode, the AM
acts as an Executor Launcher. So, the YARN application master will
reach out to YARN resource manager and request for further
containers. The resource manager will allocate new containers, and
the Application Master starts an executor in each container. After
the initial setup, these executors directly communicate with the
driver.
 |
| Client Deploy Mode |
2. Cluster Mode -
In Cluster mode, the spark-submit utility will send a YARN
application request to the YARN resource manager. The YARN resource
manager starts an application master. And then, the driver starts in
the AM container. That's where the client mode and cluster mode
differs.
In the client mode, the YARN AM acts as an executor launcher, and
the driver resides on your local machine, but in the cluster mode,
the YARN AM starts the driver, and you don't have any dependency on
your local computer. Once started, the driver will reach out to
resource manager with a request for more Containers. The resource
manager will allocate new Containers, and the driver starts an
executor in each Container.
 |
| Cluster Deploy Mode |
3. Local Mode -
Local mode is a for debugging purpose. The local mode doesn't use
the cluster at all and everything runs in a single JVM on your local
machine.
Question-14: What is the difference between executor & executor core?
Answer:
An Executor is a JVM process within a YARN container and an
executor core is a thread within the JVM. Each task is processed by
a thread. Each task work on a subset of the data. An Executor is
dedicated to a specific Spark application and terminated when the
application completes. A Spark program normally consists of many
Executors, often working in parallel.
Executor Core defines number of concurrent tasks each
executor can run. If we give executors cores very high, then spark
will create very high number of tasks for each executor. These tasks
will compete each other for resources and It will reduce data I/O
throughput. The more cores we have, the more work we can do. In
spark, this controls the number of parallel tasks an executor can
run.
Question-15: What is Shuffling concept in Spark?
Answer:
Shuffling is a process of redistributing data across multiple
partitions. Its a process of data transfer between stages. By
default, shuffling doesn’t change the number of partitions, but
their content. It is a costly and complex operation.
When an “Action” is called, Spark sends out tasks to the worker
nodes. If the action is a reduction, data shuffling takes
place.
When the DAGScheduler generates the physical execution plan from
our logical plan it pipelines together all RDDs into one stage that
are connected by a narrow dependency. Consequently, it cuts the
logical plan between all RDDs with wide dependencies
In general, avoiding shuffle will make your program run faster. All
shuffle data must be written to disk and then transferred over the
network.
Repartition, Join, cogroup, distinct, and any of the *By or *ByKey
transformations can result in shuffles.
map, filter and union generate a only stage (no shuffling).
Question-16: How to decide number of stages created in Spark job?
Answer:
DAG is a collection of stages in spark, which contains different
task.
A Stage is a sequence of Tasks that don't require a Shuffle
in-between Or you can say a Stage is collection of transformations
which does not include shuffling.
When an wide transformation executed then data will shuffled
between partitions. Due to this a new stage is created in DAG.
In spark, stages are of two types:
-ShuffleMapStage
-ResultStage
Basically, Number of stages to be created completely depends on
shuffling, i.e. whenever you perform any transformation where Spark
needs to shuffle the data by communicating to the other partitions,
it creates other stages for such transformations. And the
transformation does not require the shuffling of your data; it
creates a single stage for it.
And the number of tasks to be created depends on your number of
partitions. Lets say we have only two partitions, so each stage is
divided into two tasks. And a single task runs on a single
partition.
The number of tasks for a job is = ( no of your stages * no of your
partitions )
Question-17: What happens when Spark job is submitted OR How Spark Internally Executes a Program?
Answer:
When we do a transformation on any RDD, it gives us a new RDD. But
it does not start the execution of those transformations. The
execution is performed only when an action is performed on the new
RDD and gives us a final result.
So once you perform any action on an RDD, Spark context gives your
program to the driver.
The driver creates the DAG (directed acyclic graph) or execution
plan (job) for your program. Once the DAG is created, the driver
divides this DAG into a number of stages. These stages are then
divided into smaller tasks and all the tasks are given to the
executors for execution on the worker node.
Question-18: Can output be directly stored without sending it back to driver?
Answer:
Yes. Code needs to be written in such a way that all explicit
result collection at the driver should be avoided. We can very well
delegate this task to one of the executors.
E.g., if we want to save the results to a particular file, either
we can collect it at the driver or assign an executor to do that for
you.
Question-19: What is
coalesce and repartition used for ?
Answer:
These are the functions used to increase/decrease the number of
partitions generated.
Coalesce is used to decrease the number of partitions
without invoking Shuffling.It should be used if the number of output
partitions is less than the input. It can trigger RDD shuffling
depending on the shuffle flag which is disabled by default (i.e.
false).
While Repartition function helps to increase or decrease the
number of partitions by doing Shuffling of data.
Question-20: What is physical and logical plan optimization in Spark?
Answer:
Logical Plan:
In this phase, an RDD is created using a set of transformations, It
keeps track of those transformations in the driver program by
building a computing chain (a series of RDD)as a Graph of
transformations to produce one RDD called a Lineage Graph.
Logical Plan is divided into three parts:
a. Unresolved Logical Plan OR Parsed Logical Plan
- When code
is valid and the syntax is correct.
b. Resolved Logical Plan OR Analyzed Logical Plan OR Logical Plan
- This plan is then passed on
to a “Catalyst Optimizer” which will apply its own rule and will try
to optimize the plan.
c. Optimized Logical Plan
- Once our Optimized Logical Plan
is created then further, Physical Plan is generated.
 |
| Catalyst Optimizer |
Physical Plan:
It simply specifies how our Logical Plan is going to be executed on
the cluster. It generates different kinds of execution strategies
and then keeps comparing them in the “Cost Model”. Spark decides
which partitions should be joined first (basically it decides the
order of joining the partitions), the type of join, etc for better
optimization.
Physical Plan is specific to Spark operation and for this it will
do check up of multiple physical plans and decide the best optimal
physical plan. And finally the Best Physical Plan runs in our
cluster.
Question-21: How catalyst optimizer work internally?
Answer:
The Catalyst optimizer handles: analysis, logical optimization,
physical planning, and code generation to compile parts of queries
to Java byte code. Catalyst now supports both rule-based and
cost-based optimization.
- Analyzing logical plan --
Rule based
- Logical plan optimization
-- Rule based
- Physical planning -- Cost
based
- Code generation to
generate Java bytecode-- Rule based
*Refer Question(20) for more detailed answer on this.
Question-22: What are new features have Tungsten brought?
Answer:
Tungsten Engine has been introduced in Spark 2.x. The goal of
Project Tungsten is to improve Spark execution by optimizing Spark
jobs for CPU and memory efficiency. It does so by offering the
following optimization features:
a. Memory Management and Binary Processing: to eliminate the
overhead of JVM object model and garbage collection
- Tungsten is a Spark SQL
component that provides increased performance by rewriting Spark
operations in bytecode, at runtime.
b. Cache-aware Computation:
c. Code generation: For performance reasons,
Spark tries to group together multiple operators inside a
Whole-Stage CodeGen.
Question-23: What is Predicate Pushdown in Spark?
Answer:
Predicate pushdown is a technique to process only the required
data. Predicates can be applied to SparkSQL by defining filters in
where conditions. By using explain command to query we can check the
query processing stages. If the query plan contains PushedFilter
than the query is optimized to select only required data as every
predicate returns either True or False. If there is no PushedFilter
found in query plan than better is to cast the where condition.
Predicate Pushdowns limits the number of files and partitions that
SparkSQL reads while querying, thus reducing disk I/O. Querying on
data in buckets with predicate push downs produce results faster
with less shuffle.
*Predicate Pushdown does not work with all the datatypes in
Parquet, rather when used with Int, pushes down the values.
Question-24: How to handle skewed data? What are the steps?
Answer:
Data skew means that data distribution is uneven or asymmetric. Data
skew can severely downgrade performance of queries, especially those
with joins.
Data skew is not an issue with Spark, rather it is a data
problem.
- Common Symptoms of data skew are -
- Stuck stages &
tasks
- Low utilization of
CPU
- Out of memory errors
There are several ways to solve data skew problem.
- Data Broadcast - If we are doing a join operation on a
skewed dataset one of the tricks is to increase the
spark.sql.autoBroadcastJoinThreshold value so that smaller tables get
broadcasted. This should be done to ensure sufficient driver and
executor memory.
-
Data processes - If there are too many null values in a join or group-by key they
would skew the operation. Try to preprocess the null values with some
random ids and handle them in the application
- Salting - The
idea is to invent a new key, which guarantees an even distribution of
data. In a SQL join operation, the join key is changed to redistribute
data in an even manner so that processing for a partition does not
take more time. This technique is called salting.
Salting is a technique where we will
add random values to join key of one of the tables. In the other
table, we need to replicate the rows to match the random keys.The idea
is if the join condition is satisfied by key1 == key1, it should also
get satisfied by key1_<salt> = key1_<salt>. The value of
salt will help the dataset to be more evenly distributed.
Question-25: What are Spark optimization techniques? OR What are the best practices and optimization tips used in Spark?
Answer:
Optimization in Spark can be done in many ways. Few and important
ones are mentioned below:
- Partitioning of data - If
you join a very large table with a smaller table, pre partition the
large table before joining with the smaller table to minimize the
amount of data getting shuffled (i.e. only the data in the smaller
table will be shuffled).
- Broadcast small dataset -
Utilize Broadcast Joining for joining Smaller Datasets to Larger Ones.
Broadcast when the data is less than 1 GB or 1 million rows. Load
small dataset into memory for joining with large dataset to avoid
shuffling.
- Using Accumulators
- Caching of data - Caching
is the best technique for Apache Spark Optimization when we need some
data again and again. But it is always not acceptable to cache
data.
- Use DataFrame’s instead of
RDDs
- Predicate Pushdown
Optimization
- Avoid or minimize shuffle -
Shuffling is an expensive operation since it involves disk I/O, data
serialization, and network I/O. The primary goal when choosing an
arrangement of operators is to reduce the number of shuffles and the
amount of data shuffled.
- Skewed data: Skewing
happens when you don’t distribute the data evenly across the cluster
nodes. One or two nodes will be processing most of the data. For
example, when you join two tables by keys and say 60% of the keys are
say either null or a particular value (e.g. most traded stock id),
then 60% of all the data will be processed by a single node. This will
adversely impact performance in a distributed computing
environment.
- Define Right number of
Executors and Executor cores
Question-26: What Points to be taken care while writing Spark Application?
Answer:
- Prefer ReduceByKey than
groupByKey
- Choosing Right File
formats
- Handling multiple output
files by using repartition/coalesce
- Load small dataset into
memory for joining with large dataset to avoid shuffling using
Broadcast variables.
- Spark applications should
be continuously monitored using DAG(a data structure used in Spark that
describes various stages of tasks in Graph format)
- Use the power of Tungsten -
Tungsten is a Spark SQL component that provides increased performance
by rewriting Spark operations in byte code, at runtime
Question-27: How DAG optimization takes place in Spark?
Answer:
DAGScheduler is the scheduling layer of Apache Spark that implements
stage-oriented scheduling. It transforms a logical execution plan
(i.e. RDD lineage of dependencies built using RDD transformations) to
a physical execution plan (using stages).
The DAG scheduler splits the graph into multiple stages, the stages
are created based on the transformations.
The narrow transformations will be grouped together into a single
stage. Wide transformation define the boundary of 2 stages. Stages are
separated by 2 shuffle operations.
The DAG scheduler will then submit the stages into the task
scheduler. The number of tasks submitted depends on the number of
partitions present in the textFile.
Once the Best Physical Plan is selected, it’s the time to generate
the executable code (DAG of RDDs) for the query that is to be executed
in a cluster in a distributed fashion. This process is called Codegen
and that’s the job of Spark’s Tungsten Execution Engine.
- Spark creates DAG.
- Once action is triggered on
the RDD, DAG is submitted to the DAGScheduler.
- DAGScheduler looks at RDD
lineage and comes up with the best execution plan by dividing it into
stages of the task
- Stages set is given to
TaskScheduler and TaskScheduler will launch tasks through Cluster
manager.
- TaskScheduler with help of
cluster manager will check the data/resources availability in
different nodes to execute the tasks. It will distribute the tasks to
different executors.
Question-28: Why we prefer treereduce and treeaggregate over reduceByKey and aggregateByKey?
Answer:
In a regular reduce or aggregate functions in Spark (and the original
MapReduce) all partitions have to send their reduced value to the
driver machine, and that machine spends linear time on the number of
partitions. It becomes a bottleneck when there are many partitions and
the data from each partition is big.
So Spark has introduced a new aggregation communication pattern based
on multi-level aggregation trees.
In this setup, data are combined partially on a small set of
executors before they are sent to the driver, which dramatically
reduces the load the driver has to deal with.
So, in treeReduce and in treeAggregate, the partitions talk to each
other in a logarithmic number of rounds.
Also would like to mention difference between reduceByKey and
treeReduce -
reduceByKey is only available
on key-value pair RDDs, while treeReduce is a generalization of reduce
operation on any RDD.
reduceByKey performs
reduction for each key, resulting in an RDD; it is not an action but a
transformation that returns a Shuffled RDD.
On the other hand, treeReduce
perform the reduction in parallel using reduceByKey .
Question-29: In what cases Accumulators are reliable? OR If executors are crashed and we are using accumulators, then can we rely on accumulators output. If yes, why and if no, why?
Answer:
Accumulators are variables that are used for aggregating information
across the executors. For example, this information can be used to
find out how many records are corrupted or how many times a particular
library API was called.
Spark automatically deals with failed or slow machines by
re-executing failed or slow tasks.
Example :- if the node running a partition of a map() operation
crashes, Spark will rerun it on another node; and even if the node
does not crash but is simply much slower than other nodes, Spark can
preemptively launch a “speculative” copy of the task on another node,
and take its result if that finishes.
Even if no nodes fail,Spark may have to rerun a task to rebuild a
cached value that falls out of memory.The net result is therefore that
the same function may run multiple times on the same data depending on
what happens on the cluster.
So Accumulators are truely reliable when they are present in an
Action operation.
Accumulator updates are sent back to the driver when a task is
successfully completed. So your accumulator results are guaranteed to
be correct when you are certain that each task will have been executed
exactly once and each task did as you expected.
Question-30: What are Stragglers? How to handle stragglers and how to detect them.
Answer:
“Stragglers” are tasks within a stage that take much longer to
execute than other tasks.
Stragglers in general are those tasks that take more time to complete
than their peers. This could happen due to many reasons such as load
imbalance, I/O blocks, garbage collections, etc.
Once a Spark job is submitted, it gets broken down into multiple
stages and each stage spawns many tasks which will be executed on each
node. One or more stage might be dependent on the previous stages. So,
when a task of the earlier stage takes a longer time to complete, then
the dependent stages execution in the pipeline also gets delayed. Such
tasks that takes longer time to complete are termed as
straggler.
As the size of the cluster and the size of the stages/tasks grows,
the impact of stragglers increases dramatically impacting the job
completion time.
Thus, addressing the problem of straggler becomes prudent in order to
speed up the job completion time and also improve the cluster
efficiency
The Straggler Mitigation strategy in Spark is based on Speculation. A
task is identified as a Straggler (or Speculatable) and is
duplicated.
The Straggler identification strategy starts when
spark.speculation.quantile fraction of tasks in a stage have
completed.
By default, spark.speculation.quantile is set as 0.75. Thereafter, if
a currently running task exceeds spark.speculation.multiplier(1.5)
times the median task time from the set of successful task completions
of this particular stage, then this task is identified as a
straggler.
The default value of spark.speculation.multiplier is 1.5.
Question-31: Difference between groupByKey and reduceByKey
Answer:
Both groupByKey and reduceByKey produce the same answer but concept
to produce results are different.
reduceByKey works much better on a large dataset because Spark can be
combine output with a common key on each partition before shuffling
the data.
While on the other side in groupByKey, all the key-value pairs are
shuffled around. This is a lot of unnecessary data to being transferred
over the network.
Question-32: How will you handle OOM errors in Spark?
Answer:
Out of memory issues can be observed for the driver nodes or for
executor nodes:
- Out-of-memory errors in the driver. -
Caused by actions
Common causes which result in driver
OOM are:
- rdd.collect()
- sparkContext.broadcast
- Low driver memory configured as per the application
requirements
- Misconfiguration of spark.sql.autoBroadcastJoinThreshold.
- Out-of-memory errors on the executor
nodes. - Caused by shuffles associated with wide transformations
Common causes which result in executor OOM are:
- HIGH CONCURRENCY
- INEFFICIENT QUERIES
- INCORRECT CONFIGURATION
**OutofMemoryError due to operations: Increase parallelism so each
task input is smaller. Since Spark reuses executor JVM for many tasks,
increase parallelism to more than the number of cores.
Question-33: What is the command to see RDD lineage ?
Answer:
"ToDebugString" Method - Returns “A description of the RDD and
its recursive dependencies for debugging.
Question-34: What is Explode function and how it can be used?
Answer:
explode() takes in an array (or a map) as an input and outputs the
elements of the array (map) as separate rows.
In Spark, we can use “explode” method to convert single column values
into multiple rows.
+---+----+---------+
| id|col1|
col2|
+---+----+---------+
| 1| abc|[p, q,
r]|
| 2| def|[x, y,
z]|
+---+----+---------+
df.withColumn("col2",explode($"col2")).show()
+---+----+----+
| id|col1|col2|
+---+----+----+
| 1| abc|
p|
| 1| abc|
q|
| 1| abc|
r|
| 2| def|
x|
| 2| def|
y|
| 2| def|
z|
+---+----+----+
Question-35: Does Parquet file contains the header of data?
Answer: Yes
--------------------------------------------------------------------------------------------------------------------------------------------------------
