Verified Databricks-Certified-Professional-Data-Engineer dumps Q&As – 2024 Latest Databricks-Certified-Professional-Data-Engineer Download [Q34-Q53]

Rate this post

Verified Databricks-Certified-Professional-Data-Engineer dumps Q&As – 2024 Latest Databricks-Certified-Professional-Data-Engineer Download

Dumps Questions [2024] Pass for Databricks-Certified-Professional-Data-Engineer Exam

NO.34 Which of the following locations hosts the driver and worker nodes of a Databricks-managed clus-ter?

Data plane

Control plane

Databricks Filesystem

JDBC data source

Databricks web application

NO.35 Which statement regarding stream-static joins and static Delta tables is correct?

Each microbatch of a stream-static join will use the most recent version of the static Delta table as of each microbatch.

Each microbatch of a stream-static join will use the most recent version of the static Delta table as of the job’s initialization.

The checkpoint directory will be used to track state information for the unique keys present in the join.

Stream-static joins cannot use static Delta tables because of consistency issues.

The checkpoint directory will be used to track updates to the static Delta table.

NO.36 Which of the following developer operations in CI/CD flow can be implemented in Databricks Re-pos?

Merge when code is committed

Pull request and review process

Trigger Databricks Repos API to pull the latest version of code into production folder

Resolve merge conflicts

Delete a branch

NO.37 You have accidentally deleted records from a table called transactions, what is the easiest way to restore the records deleted or the previous state of the table? Prior to deleting the version of the table is 3 and after delete the version of the table is 4.

RESTORE TABLE transactions FROM VERSION as of 4

RESTORE TABLE transactions TO VERSION as of 3
C .
1.INSERT INTO OVERWRITE transactions
2.SELECT * FROM transactions VERSION AS OF 3
3.MINUS
4.SELECT * FROM transactions

1.INSERT INTO OVERWRITE transactions
2.SELECT * FROM transactions VERSION AS OF 4
3.INTERSECT
4.SELECT * FROM transactions

COPY OVERWRITE transactions from VERSION as of 3

NO.38 Which of the following SQL commands are used to append rows to an existing delta table?

APPEND INTO DELTA table_name

APPEND INTO table_name

COPY DELTA INTO table_name

INSERT INTO table_name

UPDATE table_name

NO.39 you are currently working on creating a spark stream process to read and write in for a one-time micro batch, and also rewrite the existing target table, fill in the blanks to complete the below command sucesfully.
1.spark.table(“source_table”)
2..writeStream
3..option(“____”, “dbfs:/location/silver”)
4..outputMode(“____”)
5..trigger(Once=____)
6..table(“target_table”)

checkpointlocation, complete, True

targetlocation, overwrite, True

checkpointlocation, True, overwrite

checkpointlocation, True, complete

checkpointlocation, overwrite, True

NO.40 What is the purpose of the silver layer in a Multi hop architecture?

Replaces a traditional data lake

Efficient storage and querying of full, unprocessed history of data

Eliminates duplicate data, quarantines bad data

Refined views with aggregated data

Optimized query performance for business-critical data

NO.41 A Delta table of weather records is partitioned by date and has the below schema:
date DATE, device_id INT, temp FLOAT, latitude FLOAT, longitude FLOAT
To find all the records from within the Arctic Circle, you execute a query with the below filter:
latitude > 66.3
Which statement describes how the Delta engine identifies which files to load?

All records are cached to an operational database and then the filter is applied

The Parquet file footers are scanned for min and max statistics for the latitude column

All records are cached to attached storage and then the filter is applied

The Delta log is scanned for min and max statistics for the latitude column

The Hive metastore is scanned for min and max statistics for the latitude column

NO.42 If E1 and E2 are two events, how do you represent the conditional probability given that E2 occurs given that
E1 has occurred?

P(E1)/P(E2)

P(E1+E2)/P(E1)

P(E2)/P(E1)

P(E2)/(P(E1+E2)

NO.43 Which statement describes Delta Lake Auto Compaction?

An asynchronous job runs after the write completes to detect if files could be further compacted; if yes, an optimize job is executed toward a default of 1 GB.

Before a Jobs cluster terminates, optimize is executed on all tables modified during the most recent job.

Optimized writes use logical partitions instead of directory partitions; because partition boundaries are only represented in metadata, fewer small files are written.

Data is queued in a messaging bus instead of committing data directly to memory; all data is committed from the messaging bus in one batch once the job is complete.

An asynchronous job runs after the write completes to detect if files could be further compacted; if yes, an optimize job is executed toward a default of 128 MB.

NO.44 At the end of the inventory process a file gets uploaded to the cloud object storage, you are asked to build a process to ingest data which of the following method can be used to ingest the data incrementally, the schema of the file is expected to change overtime ingestion process should be able to handle these changes automatically. Below is the auto loader command to load the data, fill in the blanks for successful execution of the below code.
1.spark.readStream
2..format(“cloudfiles”)
3..option(“cloudfiles.format”,”csv)
4..option(“_______”, ‘dbfs:/location/checkpoint/’)
5..load(data_source)
6..writeStream
7..option(“_______”,’ dbfs:/location/checkpoint/’)
8..option(“mergeSchema”, “true”)
9..table(table_name))

checkpointlocation, schemalocation

checkpointlocation, cloudfiles.schemalocation

schemalocation, checkpointlocation

cloudfiles.schemalocation, checkpointlocation

cloudfiles.schemalocation, cloudfiles.checkpointlocation

NO.45 A junior data engineer needs to create a Spark SQL table my_table for which Spark manages both the data and
the metadata. The metadata and data should also be stored in the Databricks Filesystem (DBFS).
Which of the following commands should a senior data engineer share with the junior data engineer to
complete this task?

1. CREATE MANAGED TABLE my_table (id STRING, value STRING) USING
2. org.apache.spark.sql.parquet OPTIONS (PATH “storage-path”);

1. CREATE TABLE my_table (id STRING, value STRING) USING DBFS;

1. CREATE TABLE my_table (id STRING, value STRING) USING
2. org.apache.spark.sql.parquet OPTIONS (PATH “storage-path”)

1. CREATE TABLE my_table (id STRING, value STRING);

1. CREATE MANAGED TABLE my_table (id STRING, value STRING);

NO.46 What steps need to be taken to set up a DELTA LIVE PIPELINE as a job using the workspace UI?

DELTA LIVE TABLES do not support job cluster

Select Workflows UI and Delta live tables tab, under task type select Delta live tables pipeline and select the notebook

Select Workflows UI and Delta live tables tab, under task type select Delta live tables pipeline and select the pipeline JSON file

Use Pipeline creation UI, select a new pipeline and job cluster

NO.47 One of the queries in the Databricks SQL Dashboard takes a long time to refresh, which of the be-low steps can be taken to identify the root cause of this issue?

Restart the SQL endpoint

Select the SQL endpoint cluster, spark UI, SQL tab to see the execution plan and time spent in each step

Run optimize and Z ordering

Change the Spot Instance Policy from “Cost optimized” to “Reliability Optimized.”

Use Query History, to view queries and select query, and check query profile to time spent in each step

NO.48 You are tasked to set up a set notebook as a job for six departments and each department can run the task parallelly, the notebook takes an input parameter dept number to process the data by department, how do you go about to setup this up in job?

Use a single notebook as task in the job and use dbutils.notebook.run to run each note-book with parameter in a different cell

A task in the job cannot take an input parameter, create six notebooks with hardcoded dept number and setup six tasks with linear dependency in the job

A task accepts key-value pair parameters, creates six tasks pass department number as parameter foreach task with no dependency in the job as they can all run in parallel.
(Correct)

A parameter can only be passed at the job level, create six jobs pass department number to each job with linear job dependency

A parameter can only be passed at the job level, create six jobs pass department number to each job with no job dependency

NO.49 A data engineering team has created a series of tables using Parquet data stored in an external sys-tem. The
team is noticing that after appending new rows to the data in the external system, their queries within
Databricks are not returning the new rows. They identify the caching of the previous data as the cause of this
issue.
Which of the following approaches will ensure that the data returned by queries is always up-to-date?

The tables should be updated before the next query is run

The tables should be converted to the Delta format

The tables should be refreshed in the writing cluster before the next query is run

The tables should be altered to include metadata to not cache

The tables should be stored in a cloud-based external system

NO.50 You have configured AUTO LOADER to process incoming IOT data from cloud object storage every 15 mins, recently a change was made to the notebook code to update the processing logic but the team later realized that the notebook was failing for the last 24 hours, what steps team needs to take to reprocess the data that was not loaded after the notebook was corrected?

Move the files that were not processed to another location and manually copy the files into the ingestion path to reprocess them

Enable back_fill = TRUE to reprocess the data

Delete the checkpoint folder and run the autoloader again

Autoloader automatically re-processes data that was not loaded

Manually re-load the data

NO.51 Data engineering team has provided 10 queries and asked Data Analyst team to build a dashboard and refresh the data every day at 8 AM, identify the best approach to set up data refresh for this dashaboard?

Each query requires a separate task and setup 10 tasks under a single job to run at 8 AM to refresh the dashboard

The entire dashboard with 10 queries can be refreshed at once, single schedule needs to be set up to refresh at 8 AM.

Setup JOB with linear dependency to all load all 10 queries into a table so the dashboard can be refreshed at once.

A dashboard can only refresh one query at a time, 10 schedules to set up the refresh.

Use Incremental refresh to run at 8 AM every day.

NO.52 Which of the following is not a privilege in the Unity catalog?

SELECT

MODIFY

DELETE

CREATE TABLE

EXECUTE

NO.53 Which of the following scenarios is the best fit for the AUTO LOADER solution?

Efficiently process new data incrementally from cloud object storage

Incrementally process new streaming data from Apache Kafa into delta lake

Incrementally process new data from relational databases like MySQL

Efficiently copy data from data lake location to another data lake location

Efficiently move data incrementally from one delta table to another delta table

Explanation
The answer is, Efficiently process new data incrementally from cloud object storage.
Please note: AUTO LOADER only works on data/files located in cloud object storage like S3 or Azure Blob Storage it does not have the ability to read other data sources, although AU-TO LOADER is built on top of structured streaming it only supports files in the cloud object stor-age. If you want to use Apache Kafka then you can just use structured streaming.
Diagram Description automatically generated

Auto Loader and Cloud Storage Integration
Auto Loader supports a couple of ways to ingest data incrementally
1.Directory listing – List Directory and maintain the state in RocksDB, supports incremental file listing
2.File notification – Uses a trigger+queue to store the file notification which can be later used to retrieve the file, unlike Directory listing File notification can scale up to millions of files per day.
[OPTIONAL]
Auto Loader vs COPY INTO?
Auto Loader
Auto Loader incrementally and efficiently processes new data files as they arrive in cloud storage without any additional setup. Auto Loader provides a new Structured Streaming source called cloudFiles. Given an input directory path on the cloud file storage, the cloudFiles source automatically processes new files as they arrive, with the option of also processing existing files in that directory.
When to use Auto Loader instead of the COPY INTO?
*You want to load data from a file location that contains files in the order of millions or higher. Auto Loader can discover files more efficiently than the COPY INTO SQL command and can split file processing into multiple batches.
*You do not plan to load subsets of previously uploaded files. With Auto Loader, it can be more difficult to reprocess subsets of files. However, you can use the COPY INTO SQL command to reload subsets of files while an Auto Loader stream is simultaneously running.
Refer to more documentation here,
https://docs.microsoft.com/en-us/azure/databricks/ingestion/auto-loader