Data Governance & Quality: Databricks Data Engineer Associate Practice Questions

A data engineering team is migrating a legacy Hive metastore data warehouse to Unity Catalog. They have two types of data assets: - **Dataset A**: Internal Silver-layer Delta tables produced entirely by Databricks pipelines. No external tool reads these tables directly from cloud storage. - **Dataset B**: Raw CSV and Parquet landing files produced by an upstream on-premises ETL system that writes directly to an S3 bucket. Databricks only reads this data. A data engineer proposes registering Dataset B as a managed table in Unity Catalog by running CREATE TABLE ... AS SELECT ... to import the data. A colleague argues this would be the wrong table type for Dataset B and instead recommends an external table. Which statement BEST explains the colleague's reasoning, and what is the KEY behavioral difference between a managed and external table in Unity Catalog when a DROP TABLE command is executed?

1. AThe colleague is correct. Unity Catalog fully manages a managed table's lifecycle: DROP TABLE deletes its metadata and, after a recovery period (7 days by default), its data files. For Dataset B, whose files an upstream system owns in S3, an external table fits: DROP TABLE only removes the metadata and leaves the files untouched.
2. BThe colleague is correct. An external table in Unity Catalog enables full read and write access from both Databricks and any external cloud storage tool simultaneously, allowing the on-premises ETL system to append new CSV files while Databricks reads them concurrently without any consistency issues. By contrast, a managed table exclusively locks the storage path, preventing the upstream on-premises system from writing new files to the same S3 bucket location while the table is registered in Unity Catalog.
3. CThe colleague is incorrect. Managed tables and external tables are functionally identical after registration in Unity Catalog—both types support the same file formats (Delta, CSV, Parquet, Avro), and DROP TABLE behaves identically for both, removing only the metadata registration while always preserving the underlying physical data files to prevent accidental data loss in a shared environment.
4. DThe colleague is incorrect. Since Dataset B is read-only from Databricks, registering it as a managed table is the right choice because managed tables in Unity Catalog offer superior query performance through predictive optimization and automatic clustering, regardless of whether the underlying data was created by Databricks. The DROP TABLE behavior is irrelevant for read-only datasets because Unity Catalog enforces a data protection policy that prevents DROP TABLE from deleting files when the table has been read by two or more users in the past 30 days.

A data engineering team is evaluating whether to use **Lakehouse Federation** or **managed ingestion (Auto Loader / COPY INTO)** for connecting to several external data sources. They describe the following four scenarios: - **Scenario A**: A data scientist wants to run exploratory SQL queries against a production PostgreSQL database to understand data shape before designing an ETL pipeline. Speed of setup is more important than query performance. - **Scenario B**: A critical nightly pipeline must join 500 million records from an internal Delta table with 200 million records from a Snowflake data warehouse. The result feeds a real-time executive dashboard that requires sub-3-second query times. - **Scenario C**: A compliance team needs to query a MySQL database that stores audit records from a legacy HR system. They want the data governed by Unity Catalog (lineage, fine-grained access control) without moving data into Databricks. - **Scenario D**: A streaming pipeline must ingest IoT sensor data arriving at 100,000 events per second from an external Kafka topic and land it into a Unity Catalog Delta table for downstream real-time analytics. For which scenarios is **Lakehouse Federation** the MOST appropriate solution?

1. ALakehouse Federation is most appropriate for Scenarios A and B. Scenario A benefits from Lakehouse Federation's fast setup for ad-hoc exploration without ingestion. Scenario B benefits from Lakehouse Federation's 'advanced query planning' that pushes computation down to the Snowflake engine, meaning the 500M × 200M join is executed entirely within Snowflake's compute layer, eliminating data transfer costs and achieving sub-3-second dashboard performance at zero Databricks compute cost.
2. BLakehouse Federation is most appropriate for Scenarios B and D. Scenario B is well-suited because Lakehouse Federation supports high-throughput joins between Delta tables and external warehouse tables, and its caching layer automatically materializes the Snowflake results for dashboard serving. Scenario D is appropriate because Lakehouse Federation's foreign catalog feature supports real-time streaming connections to Kafka topics as a federated external source, with Unity Catalog governing the stream's schema and access policies.
3. CLakehouse Federation is most appropriate for Scenarios A and C. Scenario A (ad-hoc exploration of PostgreSQL before building an ETL pipeline) is an ideal use case because Lakehouse Federation enables query federation without data migration, providing fast setup for proof-of-concept work. Scenario C (querying a MySQL audit database with Unity Catalog governance—lineage, fine-grained access control—without moving data) is also a primary use case for Lakehouse Federation, which extends UC governance capabilities (lineage, row/column-level security) to external data sources in place.
4. DLakehouse Federation is most appropriate for Scenarios C and D. Scenario C requires Unity Catalog governance over external MySQL data, which Lakehouse Federation provides via foreign catalogs. Scenario D requires high-throughput real-time data ingestion, and Lakehouse Federation's streaming foreign catalog feature handles Kafka sources natively within Unity Catalog, while automatically applying Delta Lake ACID guarantees to incoming IoT events without requiring a separate Auto Loader configuration.

A data platform team at a retail company needs to share sales analytics data with three different recipients: - **Recipient 1**: An internal data science team at the same company using a different Databricks workspace in a different cloud region. - **Recipient 2**: A strategic retail analytics partner that uses Apache Spark (open source, not Databricks) to process shared data in their own environment. - **Recipient 3**: A business intelligence team at a subsidiary company using Tableau connected to their own Databricks SQL warehouse. After reviewing Delta Sharing documentation, a data engineer lists the following statements about Delta Sharing capabilities and limitations. Which statement is ACCURATE?

1. ADelta Sharing grants recipients full read/write access to the shared data, allowing Recipient 1's data science team to write model output results back to the provider's shared tables directly. This bidirectional access is the primary advantage of Databricks-to-Databricks Delta Sharing over open sharing, because open sharing only supports one-directional, read-only access for non-Databricks recipients like Recipient 2.
2. BFor Recipient 1 (different cloud region), Databricks-to-Databricks Delta Sharing can be used, but the team should be aware that lineage graphs in Unity Catalog do not cross region or platform boundaries—lineage tracking will not show cross-region data flow. For Recipient 2 (open-source Spark), open Delta Sharing (non-Databricks) is used; the recipient consumes shared data as a read-only external dataset without needing a Databricks account, using the open Delta Sharing protocol. For all recipients, Delta Sharing only grants read-only access—it is not possible for any recipient to write back to provider tables through Delta Sharing.
3. CDelta Sharing requires recipients to always be on the Databricks platform regardless of whether they are internal or external, because the Delta Sharing protocol relies on Unity Catalog's proprietary credential vending mechanism that is only compatible with Databricks Runtime. Recipient 2 using open-source Spark cannot consume Delta Sharing data unless they install the official Databricks Spark connector and authenticate with a Databricks personal access token issued by the provider organization.
4. DDelta Sharing is exclusively designed for sharing data across different organizations and cloud providers, and cannot be used for internal cross-workspace or cross-region sharing within the same Databricks account. For Recipient 1 (same company, different workspace and region), the correct approach is to register the data as an external table accessible from both workspaces via a shared external location, while Delta Sharing should only be used for Recipient 2 and Recipient 3, who are external to the company's primary Databricks metastore.