Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Exam Questions & Answers

To extract unique values from a column and sort them alphabetically:

distinct() is required to remove duplicate values.

orderBy() is needed to sort the results alphabetically (ascending by default).

Correct code:

df.select('Name').distinct().orderBy(df['Name'])

This is directly aligned with standard DataFrame API usage in PySpark, as documented in the official Databricks Spark APIs. Option A is incorrect because it may not remove duplicates. Option C omits sorting. Option D sorts in descending order, which doesn't meet the requirement for alphabetical (ascending) order.

Pandas API on Spark provides a distributed implementation of the Pandas DataFrame API on top of Apache Spark.

Advantages:

Executes transformations in parallel across all nodes and cores in the cluster.

Maintains Pandas-like syntax, making it easy for Python users to transition.

Enables scaling of existing Pandas code to handle large datasets without memory limits.

Therefore, it combines Pandas usability with Spark's distributed power, offering both speed and scalability.

Why the other options are incorrect:

B: While it uses Python, that's not its main advantage.

C: It runs distributed across the cluster, not on a single node.

D: Pandas API on Spark uses lazy evaluation, not eager computation.

PySpark Pandas API Overview --- advantages of distributed execution.

Databricks Exam Guide (June 2025): Section ''Using Pandas API on Apache Spark'' --- explains the benefits of Pandas API integration for scalable transformations.

===========

When writing a DataFrame using .partitionBy() in Spark, the data is physically organized into directory structures corresponding to unique combinations of the partition columns.

Example:

/path/to/output/color=Red/taste=Sweet/part-0001.parquet

/path/to/output/color=Green/taste=Sour/part-0002.parquet

This structure improves query performance by pruning partitions when filtering on these columns.

Why the other options are incorrect:

A: Appending requires .mode('append'), which isn't used here.

B: Null values in partition columns are handled; they don't raise errors.

D: Partitioning prevents storing all data in a single file.

PySpark DataFrameWriter API --- partitionBy() and .parquet() methods.

Databricks Exam Guide (June 2025): Section ''Using Spark SQL'' --- partitioning and writing optimized output files.

===========

According to Spark's watermarking rules:

''Records that are older than the watermark (event time < current watermark) are considered too late and are dropped.''

So, if a record's event_time is earlier than (max event_time seen so far - 10 minutes), it is discarded.

When running in local mode (e.g., local[4]), the number inside the brackets defines how many threads Spark will use.

Using local[*] ensures Spark uses all available CPU cores for parallelism.

Example:

spark-submit --master local[*]

Dynamic allocation and executor memory apply to cluster-based deployments, not local mode.