Implementing Data Engineering Solutions Using Azure Databricks v1.0

Your company has sales, finance, and HR departments.
You have an Azure Databricks workspace that is enabled for Unity Catalog.
You need to implement Unity Catalog to meet the following requirements:
Access to Unity Catalog for each department must be managed independently from that of the other departments.
The data assets of each department must be isolated logically from those of the other departments.
The solution must support centralized governance.
What should you do for each department?

DRAG DROP -
You have an Azure Databricks workspace named Workspace1.
You have a user named User1 that is a non-admin user for Workspace1.
You need to ensure that User1 can perform the following tasks:
Provision clusters of any size.
Run Databricks jobs.
The solution must follow the principle of least privilege.
What should you do?
To answer, drag the appropriate options to the correct requirements. Each option may be used once, more than once, or not at all. You may need to drag the split bar between panes or scroll to view content.
NOTE: Each correct selection is worth one point.

You have an Azure Databricks workspace that is enabled for Unity Catalog.
You have a CSV file stored in an Azure Data Lake Storage Gen2 container.
You plan to ingest the data into an existing table by running the following SQL statement.

COPY INTO Customer -
FROM ‘abfss://[email protected]/data/customer’

FILEFORMAT = CSV -
You need to ensure that the statement can access the data in the container.
What should you configure?

HOTSPOT -
You have an Azure Databricks workspace that is enabled for Unity Catalog and contains a catalog named catalog1.
You have a group named group1. Group1 already has the USE CATALOG privilege on catalog1.
You create a schema named schema1 in catalog1.
You need to ensure that group1 can create tables in schema1. Group1 must not be able to grant permissions on the schema or its objects. The solution must follow the principle of least privilege.
How should you complete the SQL statement? To answer, select the appropriate options in the answer area.
NOTE: Each correct selection is worth one point.

HOTSPOT -
You have an Azure Databricks workspace that is attached to a Unity Catalog metastore named metastore1.metastore1 contains:

A catalog named Sales -
A schema named Customers in the Sales catalog
A table named Customer_details in the Customers schema
You need to ensure that a user named User1 can update the data in Customer_details. The solution must meet the following requirements:
Ensure that User1 cannot create new tables.
Follow the principle of least privilege.
Which permission should you grant to User1 for each object? To answer, select the appropriate options in the answer area.
NOTE: Each correct selection is worth one point.

HOTSPOT
-

You have an Azure Databricks workspace that is enabled for Unity Catalog and contains:

• A catalog named Corpdb
• A schema named Finance in the Corpdb catalog
• A table named Sales in the Finance schema

You have a group named Analysts.

You assign permissions to Analysts as shown in the following table.



For each of the following statements, select Yes if the statement is true. Otherwise, select No.

NOTE: Each correct selection is worth one point.

You have an Azure Databricks workspace that is enabled for Unity Catalog.

You need to share curated data with an external organization. The solution must meet the following requirements:

• The organization will use its own compute platform to query the data.
• Access to the data must be centrally governed by using Unity Catalog.
• Administrative effort must be minimized.

What should you do?

DRAG DROP
-

You have an Azure Databricks workspace that is enabled for Unity Catalog and contains:

• A catalog named Finance
• A schema named Purchases in the Finance catalog
• A table named Transactions in the Purchases schema

You need to ensure that a user named finance_user can query the Transactions table. The solution must follow the principle of least privilege.

Which permission should you grant to finance_user for each object? To answer, drag the appropriate permissions to the correct objects. Each permission may be used once, more than once, or not at all. You may need to drag the split bar between panes or scroll to view content.

NOTE: Each correct selection is worth one point.

You have an Azure Databricks account that contains workspaces enabled for Unity Catalog.

You need to implement audit logging to meet the following requirements:

• Capture audit logs for all the workspaces in the account.
• Retain the audit logs for 90 days.
• Minimize storage and ingestion costs.

The logs will be reviewed only during security investigations and will NOT be queried regularly.

To where should you send the audit logs?

You have an Azure Databricks workspace named Workspace1 that contains a cluster named Cluster1.

You need to enable a user named User1 to install a Maven package on all the nodes of Cluster1. The solution must follow the principle of least privilege.

What should you do?

This is a case study. Case studies are not timed separately from other exam sections. You can use as much exam time as you would like to complete each case study. However, there might be additional case studies or other exam sections. Manage your time to ensure that you can complete all the exam sections in the time provided. Pay attention to the Exam Progress at the top of the screen so you have sufficient time to complete any exam sections that follow this case study.

To answer the case study questions, you will need to reference information that is provided in the case. Case studies and associated questions might contain exhibits or other resources that provide more information about the scenario described in the case. Information provided in an individual question does not apply to the other questions in the case study.

A Review Screen will appear at the end of this case study. From the Review Screen, you can review and change your answers before you move to the next exam section. After you leave this case study, you will NOT be able to return to it.


To start the case study -
To display the first question in this case study, select the “Next” button. To the left of the question, a menu provides links to information such as business requirements, the existing environment, and problem statements. Please read through all this information before answering any questions. When you are ready to answer a question, select the “Question” button to return to the question.


Overview -


Company Information -

Contoso, Inc. is a renewable energy provider that operates solar and wind farms across North America.


Existing Environment -


Azure Environment -

Contoso has a single Azure Databricks workspace named Workspace1 in the West US Azure region. Workspace1 is enabled for Unity Catalog.

Workspace1 contains all-purpose clusters for both development and production workloads.

The company's Azure environment contains:

• In the West US, Central US, and East US Azure regions, Azure event hubs that stream telemetry data and an Azure Data Lake Storage Gen2 account in each region for each hub
• A single Azure SQL database in the West US region that hosts enterprise resource planning (ERP) data
• An Azure Database for PostgreSQL server in the West US region that stores operational maintenance data


Data Environment -

Contoso ingests the following operational and business data:

• Telemetry data: More than 40,000 IoT sensors across 28 sites emit JSON telemetry events every few seconds. Each site sends the events to the nearest event hub, which writes the data into the corresponding Data Lake Storage Gen2 account. These files frequently experience schema drift.
• Maintenance logs: Maintenance systems generate historical repair logs, daily incremental updates, technician notes, and unstructured attachments that are stored in the Data Lake Storage Gen2 accounts.
• Operational maintenance data: Structured operational maintenance data is stored on the Azure Database for PostgreSQL server.
• External weather data: Hourly weather forecasts are retrieved from a REST API and written to the Data Lake Storage Gen2 accounts.
• ERP data: Daily CSV extracts of 50 to 100 GB contain equipment metadata, work orders, and purchase order information.


Problem Statements -

The company’s existing analytics environment has several issues:


Ingestion -
• Telemetry pipelines fall behind during peak loads.
• Telemetry ingestion fails when schema drift occurs.
• Streaming pipelines reprocess events after a pipeline restarts.


Compute -
• Production and development workloads run on the same all-purpose clusters.
• Production and development workloads do NOT support autoscaling or workload isolation.


Governance -
• The ERP data is duplicated across systems and development teams.
• Naming conventions are inconsistent across development teams, regions, and products.
• Ownership of the IoT sensors changes over time, and analysts must track the full history of the ownership.
• Occasionally, equipment manufacturers must correct data-entry mistakes in equipment names. Historical values are NOT required.


Pipeline operations -
• Pipelines lack resiliency, alerting, and centralized scheduling.


Requirements -


Planned Changes -

Contoso plans to implement the following changes:

• Implement scalable data pipeline orchestration.
• Create a managed analytics catalog in Unity Catalog.
• Implement a consistent approach to creating curated datasets.
• Establish a centralized governance model across ingestion, cleansed, and curated layers.
• Grant data engineers access to the ERP tables by using minimal development effort.
• Adopt a compute strategy that isolates production workloads and supports autoscaling.
• Adopt a slowly changing dimension (SCD) approach to address current data modeling issues.


Technical Requirements -

Contoso identifies the following environment and compute requirements:
• Ensure that production ingestion workloads run on compute clusters that can scale automatically during telemetry spikes.
• Provide fast and consistent performance for business intelligence (BI) workloads.
• Prevent development activity from affecting production pipelines.
• Production ingestion workloads must run as scheduled, non-interactive pipelines rather than on shared interactive development clusters.

Contoso identifies the following data ingestion and processing requirements:
• Auto-scale ingestion pipelines to handle bursty workloads.
• Handle schema drift for the maintenance and telemetry data.
• Ingest file-based telemetry data by using minimal operational effort.
• Store all the ingested data in a format that supports incremental processing.
• Support the continuous ingestion of telemetry data from the event hubs by using exactly-once semantics.
• Support the ingestion of the structured maintenance data from the Azure Database for PostgreSQL server.
• Build a new telemetry pipeline that ingests raw events from the event hubs, cleanses the data, and publishes curated tables to Unity Catalog.
• Ensure that the Apache Spark Structured Streaming pipelines reading from the event hubs write the data into a managed Delta table named telemetry.raw_events. The pipelines must support schema drift and resume processing after failures without reprocessing the data.

Contoso identifies the following data modeling and optimization requirements:
• Build curated tables that standardize business logic.
• Overwrite equipment metadata attributes, such as name, manufacturer, model, and commissioning date, when the attributes change. Historical values are NOT required.

Contoso identifies the following pipeline deployment and operation requirements:
• Orchestrate multi-step ingestion and transformation workflows.
• Define a clear execution order and dependencies.
• Automatically retry failed steps and notify operators.
• Schedule ingestion and transformation workloads consistently.


Governance Requirements -

Contoso identifies the following governance requirements:

• Centralize the metadata catalog.
• Provide isolated development areas that follow standard naming conventions.
• Establish a consistent structure for organizing raw, cleansed, and curated data.
• Provide a read-only mechanism to reference the ERP data through a foreign catalog.


Business Requirements -

Contoso identifies the following business requirements:

• Improve ingestion reliability and reduce operational effort.
• Standardize data definitions across development teams.


You need to curate Unity Catalog objects that reference the ERP data. The solution must meet the governance requirements.

What should you do?

HOTSPOT -
You have an Azure Databricks account that contains a single workspace named Workspace1. Workspace1 is enabled for Unity Catalog.
You discover that data access events for Unity Catalog tables fail to appear in the logs.
You need to ensure that all the data access events are captured centrally for auditing purposes. The log data must be available for analysis as quickly as possible.
What should you do?
To answer, select the appropriate options in the answer area.
NOTE: Each correct selection is worth one point.

DRAG DROP -
This is a case study. Case studies are not timed separately from other exam sections. You can use as much exam time as you would like to complete each case study. However, there might be additional case studies or other exam sections. Manage your time to ensure that you can complete all the exam sections in the time provided. Pay attention to the Exam Progress at the top of the screen so you have sufficient time to complete any exam sections that follow this case study.
To answer the case study questions, you will need to reference information that is provided in the case. Case studies and associated questions might contain exhibits or other resources that provide more information about the scenario described in the case. Information provided in an individual question does not apply to the other questions in the case study.
A Review Screen will appear at the end of this case study. From the Review Screen, you can review and change your answers before you move to the next exam section. After you leave this case study, you will NOT be able to return to it.

To start the case study -
To display the first question in this case study, select the “Next” button. To the left of the question, a menu provides links to information such as business requirements, the existing environment, and problem statements. Please read through all this information before answering any questions. When you are ready to answer a question, select the “Question” button to return to the question.

Overview -

Company Information -
Contoso, Inc. is a renewable energy provider that operates solar and wind farms across North America.

Existing Environment -

Azure Environment -
Contoso has a single Azure Databricks workspace named Workspace1 in the West US Azure region. Workspace1 is enabled for Unity Catalog.
Workspace1 contains all-purpose clusters for both development and production workloads.
The company's Azure environment contains:
In the West US, Central US, and East US Azure regions, Azure event hubs that stream telemetry data and an Azure Data Lake Storage Gen2 account in each region for each hub
A single Azure SQL database in the West US region that hosts enterprise resource planning (ERP) data
An Azure Database for PostgreSQL server in the West US region that stores operational maintenance data

Data Environment -
Contoso ingests the following operational and business data:
Telemetry data: More than 40,000 IoT sensors across 28 sites emit JSON telemetry events every few seconds. Each site sends the events to the nearest event hub, which writes the data into the corresponding Data Lake Storage Gen2 account. These files frequently experience schema drift.
Maintenance logs: Maintenance systems generate historical repair logs, daily incremental updates, technician notes, and unstructured attachments that are stored in the Data Lake Storage Gen2 accounts.
Operational maintenance data: Structured operational maintenance data is stored on the Azure Database for PostgreSQL server.
External weather data: Hourly weather forecasts are retrieved from a REST API and written to the Data Lake Storage Gen2 accounts.
ERP data: Daily CSV extracts of 50 to 100 GB contain equipment metadata, work orders, and purchase order information.

Problem Statements -
The company’s existing analytics environment has several issues:

Ingestion -
Telemetry pipelines fall behind during peak loads.
Telemetry ingestion fails when schema drift occurs.
Streaming pipelines reprocess events after a pipeline restarts.

Compute -
Production and development workloads run on the same all-purpose clusters.
Production and development workloads do NOT support autoscaling or workload isolation.

Governance -
The ERP data is duplicated across systems and development teams.
Naming conventions are inconsistent across development teams, regions, and products.
Ownership of the IoT sensors changes over time, and analysts must track the full history of the ownership.
Occasionally, equipment manufacturers must correct data-entry mistakes in equipment names. Historical values are NOT required.

Pipeline operations -
Pipelines lack resiliency, alerting, and centralized scheduling.

Requirements -

Planned Changes -
Contoso plans to implement the following changes:
Implement scalable data pipeline orchestration.
Create a managed analytics catalog in Unity Catalog.
Implement a consistent approach to creating curated datasets.
Establish a centralized governance model across ingestion, cleansed, and curated layers.
Grant data engineers access to the ERP tables by using minimal development effort.
Adopt a compute strategy that isolates production workloads and supports autoscaling.
Adopt a slowly changing dimension (SCD) approach to address current data modeling issues.

Technical Requirements -
Contoso identifies the following environment and compute requirements:
Ensure that production ingestion workloads run on compute clusters that can scale automatically during telemetry spikes.
Provide fast and consistent performance for business intelligence (BI) workloads.
Prevent development activity from affecting production pipelines.
Production ingestion workloads must run as scheduled, non-interactive pipelines rather than on shared interactive development clusters.
Contoso identifies the following data ingestion and processing requirements:
Auto-scale ingestion pipelines to handle bursty workloads.
Handle schema drift for the maintenance and telemetry data.
Ingest file-based telemetry data by using minimal operational effort.
Store all the ingested data in a format that supports incremental processing.
Support the continuous ingestion of telemetry data from the event hubs by using exactly-once semantics.
Support the ingestion of the structured maintenance data from the Azure Database for PostgreSQL server.
Build a new telemetry pipeline that ingests raw events from the event hubs, cleanses the data, and publishes curated tables to Unity Catalog.
Ensure that the Apache Spark Structured Streaming pipelines reading from the event hubs write the data into a managed Delta table named telemetry.raw_events. The pipelines must support schema drift and resume processing after failures without reprocessing the data.
Contoso identifies the following data modeling and optimization requirements:
Build curated tables that standardize business logic.
Overwrite equipment metadata attributes, such as name, manufacturer, model, and commissioning date, when the attributes change. Historical values are NOT required.
Contoso identifies the following pipeline deployment and operation requirements:
Orchestrate multi-step ingestion and transformation workflows.
Define a clear execution order and dependencies.
Automatically retry failed steps and notify operators.
Schedule ingestion and transformation workloads consistently.

Governance Requirements -
Contoso identifies the following governance requirements:
Centralize the metadata catalog.
Provide isolated development areas that follow standard naming conventions.
Establish a consistent structure for organizing raw, cleansed, and curated data.
Provide a read-only mechanism to reference the ERP data through a foreign catalog.

Business Requirements -
Contoso identifies the following business requirements:
Improve ingestion reliability and reduce operational effort.
Standardize data definitions across development teams.
Which SCD type should you use to support the planned data modeling changes? To answer, drag the appropriate types to the correct issues. Each type may be used once, more than once, or not at all. You may need to drag the split bar between panes or scroll to view content.
NOTE: Each correct selection is worth one point.

DRAG DROP -
This is a case study. Case studies are not timed separately from other exam sections. You can use as much exam time as you would like to complete each case study. However, there might be additional case studies or other exam sections. Manage your time to ensure that you can complete all the exam sections in the time provided. Pay attention to the Exam Progress at the top of the screen so you have sufficient time to complete any exam sections that follow this case study.
To answer the case study questions, you will need to reference information that is provided in the case. Case studies and associated questions might contain exhibits or other resources that provide more information about the scenario described in the case. Information provided in an individual question does not apply to the other questions in the case study.
A Review Screen will appear at the end of this case study. From the Review Screen, you can review and change your answers before you move to the next exam section. After you leave this case study, you will NOT be able to return to it.

To start the case study -
To display the first question in this case study, select the “Next” button. To the left of the question, a menu provides links to information such as business requirements, the existing environment, and problem statements. Please read through all this information before answering any questions. When you are ready to answer a question, select the “Question” button to return to the question.

Overview -

Company Information -
Contoso, Inc. is a renewable energy provider that operates solar and wind farms across North America.

Existing Environment -

Azure Environment -
Contoso has a single Azure Databricks workspace named Workspace1 in the West US Azure region. Workspace1 is enabled for Unity Catalog.
Workspace1 contains all-purpose clusters for both development and production workloads.
The company's Azure environment contains:
In the West US, Central US, and East US Azure regions, Azure event hubs that stream telemetry data and an Azure Data Lake Storage Gen2 account in each region for each hub
A single Azure SQL database in the West US region that hosts enterprise resource planning (ERP) data
An Azure Database for PostgreSQL server in the West US region that stores operational maintenance data

Data Environment -
Contoso ingests the following operational and business data:
Telemetry data: More than 40,000 IoT sensors across 28 sites emit JSON telemetry events every few seconds. Each site sends the events to the nearest event hub, which writes the data into the corresponding Data Lake Storage Gen2 account. These files frequently experience schema drift.
Maintenance logs: Maintenance systems generate historical repair logs, daily incremental updates, technician notes, and unstructured attachments that are stored in the Data Lake Storage Gen2 accounts.
Operational maintenance data: Structured operational maintenance data is stored on the Azure Database for PostgreSQL server.
External weather data: Hourly weather forecasts are retrieved from a REST API and written to the Data Lake Storage Gen2 accounts.
ERP data: Daily CSV extracts of 50 to 100 GB contain equipment metadata, work orders, and purchase order information.

Problem Statements -
The company’s existing analytics environment has several issues:

Ingestion -
Telemetry pipelines fall behind during peak loads.
Telemetry ingestion fails when schema drift occurs.
Streaming pipelines reprocess events after a pipeline restarts.

Compute -
Production and development workloads run on the same all-purpose clusters.
Production and development workloads do NOT support autoscaling or workload isolation.

Governance -
The ERP data is duplicated across systems and development teams.
Naming conventions are inconsistent across development teams, regions, and products.
Ownership of the IoT sensors changes over time, and analysts must track the full history of the ownership.
Occasionally, equipment manufacturers must correct data-entry mistakes in equipment names. Historical values are NOT required.

Pipeline operations -
Pipelines lack resiliency, alerting, and centralized scheduling.

Requirements -

Planned Changes -
Contoso plans to implement the following changes:
Implement scalable data pipeline orchestration.
Create a managed analytics catalog in Unity Catalog.
Implement a consistent approach to creating curated datasets.
Establish a centralized governance model across ingestion, cleansed, and curated layers.
Grant data engineers access to the ERP tables by using minimal development effort.
Adopt a compute strategy that isolates production workloads and supports autoscaling.
Adopt a slowly changing dimension (SCD) approach to address current data modeling issues.

Technical Requirements -
Contoso identifies the following environment and compute requirements:
Ensure that production ingestion workloads run on compute clusters that can scale automatically during telemetry spikes.
Provide fast and consistent performance for business intelligence (BI) workloads.
Prevent development activity from affecting production pipelines.
Production ingestion workloads must run as scheduled, non-interactive pipelines rather than on shared interactive development clusters.
Contoso identifies the following data ingestion and processing requirements:
Auto-scale ingestion pipelines to handle bursty workloads.
Handle schema drift for the maintenance and telemetry data.
Ingest file-based telemetry data by using minimal operational effort.
Store all the ingested data in a format that supports incremental processing.
Support the continuous ingestion of telemetry data from the event hubs by using exactly-once semantics.
Support the ingestion of the structured maintenance data from the Azure Database for PostgreSQL server.
Build a new telemetry pipeline that ingests raw events from the event hubs, cleanses the data, and publishes curated tables to Unity Catalog.
Ensure that the Apache Spark Structured Streaming pipelines reading from the event hubs write the data into a managed Delta table named telemetry.raw_events. The pipelines must support schema drift and resume processing after failures without reprocessing the data.
Contoso identifies the following data modeling and optimization requirements:
Build curated tables that standardize business logic.
Overwrite equipment metadata attributes, such as name, manufacturer, model, and commissioning date, when the attributes change. Historical values are NOT required.
Contoso identifies the following pipeline deployment and operation requirements:
Orchestrate multi-step ingestion and transformation workflows.
Define a clear execution order and dependencies.
Automatically retry failed steps and notify operators.
Schedule ingestion and transformation workloads consistently.

Governance Requirements -
Contoso identifies the following governance requirements:
Centralize the metadata catalog.
Provide isolated development areas that follow standard naming conventions.
Establish a consistent structure for organizing raw, cleansed, and curated data.
Provide a read-only mechanism to reference the ERP data through a foreign catalog.

Business Requirements -
Contoso identifies the following business requirements:
Improve ingestion reliability and reduce operational effort.
Standardize data definitions across development teams.
Which ingestion option should you recommend for each data source? To answer, drag the appropriate options to the correct data sources. Each option may be used once, more than once, or not at all. You may need to drag the split bar between panes or scroll to view content.
NOTE: Each correct selection is worth one point.