Ensuring Data Security and Compliance: Databricks Data Engineer Professional Practice Questions

Correct answer: A

WHY A is correct: Generalisation replaces specific values with broader categories or ranges, reducing the granularity of the data so that individuals cannot be singled out while the aggregate/demographic signal is preserved. Replacing exact ages with 10-year buckets (e.g., 34 → '30-39') is the canonical application of generalisation and is widely used in k-anonymity implementations for healthcare and research data. WHY NOT D: Hashing converts exact ages to opaque fixed-length strings. While this prevents reading the age, the hashed values are useless for demographic analysis — you cannot derive any statistical distribution or trend from a set of SHA-256 digests. Demographic utility is lost entirely. WHY NOT B: Suppression removes the column entirely. This satisfies the privacy requirement but completely eliminates demographic analysis utility, which contradicts the requirement. WHY NOT C: Tokenization replaces the age with a random token while retaining a lookup table. The tokens themselves carry no demographic meaning and cannot be used for statistical analysis. Additionally, the retention of a reverse-mapping table means the data has not been truly anonymized — it remains pseudonymized.

Correct answer: B

WHY B is correct: Per Databricks documentation, materialized views automatically handle deletions from their source tables. A materialized view always returns a correct result — it uses incremental computation unless full recomputation is required for correctness (e.g., after a deletion that changes aggregations). You do NOT need to manually delete records from materialized views. The team only needs to refresh the materialized view and run maintenance to ensure deletions are completely processed and the underlying data files are cleaned up. WHY NOT A: Materialized views are derived objects; you cannot run DML (DELETE) statements directly on a materialized view in Databricks. Materalized views are refreshed from their source definitions, not modified directly. WHY NOT C: Dropping and recreating materialized views is unnecessary and expensive. Materialized views are designed to handle source deletions via refresh — recreation would also lose any additional configuration or metadata associated with the view. WHY NOT D: skipChangeCommits is a streaming table option, not a materialized view option. Materialized views do not use this concept — they handle corrections automatically through full or incremental recomputation. WHY NOT E: Enabling change data feed on bronze tables helps with streaming CDC pipelines but it is not a requirement for materialized view refresh after deletions. Materialized views compute correctness independently of whether CDF is enabled on source tables.

Correct answer: C

WHY C is correct: Unity Catalog row filters and column masks are enforced with a 'fail-secure' design. On Databricks Runtime versions below 12.2 LTS, any query against a table that has row filters or column masks attached will fail and return NO data. This prevents accidental exposure of restricted data on older runtimes that cannot correctly enforce the filter logic. WHY NOT A: Returning all rows on an unsupported runtime would be a security vulnerability. Databricks explicitly chose the fail-secure approach — no data is returned rather than all data. WHY NOT B: Row filters are NOT enforced at the metastore level independently of the compute runtime. The enforcement occurs in the query engine, which is runtime-dependent. Older runtimes cannot enforce filters, hence the fail-secure fallback. WHY NOT D: The fail-secure design does not degrade silently. The query does not proceed to fetch rows and then apply the filter; instead it fails entirely on unsupported runtimes. WHY NOT E: There is no 'warning row' behavior. The query simply fails and returns an error with no data to prevent any data leakage.

Correct answer: C

WHY C is correct: When deletion vectors are enabled on a Delta table, DELETE operations are 'soft deletes' — they write a deletion vector (a metadata bitmap file) marking which rows are logically deleted, but the actual Parquet data files containing the original rows are NOT rewritten or removed. This means the raw user data physically remains in the current Parquet files. VACUUM removes older data files that are no longer referenced by the transaction log, but it does not rewrite current files to remove soft-deleted data. The correct GDPR purge workflow is: (1) DELETE the records, (2) run REORG TABLE catalog1.schema1.users APPLY (PURGE) to physically rewrite the current Parquet files and eliminate the soft-deleted rows from them, and then (3) run VACUUM to remove the older (pre-REORG) files that are now outside the retention window. WHY NOT A: Deletion vectors are specifically designed to avoid rewriting Parquet files for performance — DELETE does NOT immediately rewrite any files when deletion vectors are enabled. The auditor's concern is valid. WHY NOT B: VACUUM removes older files that are no longer referenced by the transaction log beyond the retention threshold. It does not rewrite current, actively referenced Parquet files. The user's data physically residing in the current files (soft-deleted via deletion vector) remains intact after VACUUM alone. WHY NOT D: The claim that data 'will remain forever' is inaccurate. After REORG TABLE ... APPLY (PURGE) is run, the data is physically removed from the current files. Subsequently, VACUUM removes the now-old pre-REORG files. The combination completes the physical deletion. WHY NOT E: VACUUM does not encrypt data; it deletes files. Databricks encryption-at-rest is a storage-layer protection that is unrelated to whether raw data bytes are readable by someone with direct cloud storage access.

Correct answer: D

WHY D is correct: Databricks enforces a safety check that prevents VACUUM from running with a retention threshold below 7 days by default. To override this, the engineer must set the Spark configuration property spark.databricks.delta.retentionDurationCheck.enabled to false on the cluster or session. The documented risk is significant: if any running jobs (e.g., long-running streaming or batch jobs) are writing data files that have not yet been committed to the Delta transaction log, VACUUM with a short threshold may permanently delete those uncommitted files before the job finishes — causing the job to fail or producing corrupted/incomplete tables. WHY NOT A: Setting delta.deletedFileRetentionDuration as a table property controls retention for tables on Databricks Runtime 18.0+ (or Unity Catalog managed tables on DBR 12.2+). However, the safety check that prevents VACUUM from running below 7 days is a separate, independent mechanism. Even after setting the property, VACUUM will still fail unless retentionDurationCheck.enabled is also set to false. WHY NOT B: VACUUM LITE mode is a Public Preview feature (DBR 16.1+) that uses the transaction log to identify files to remove, avoiding full directory listings. It does NOT bypass the safety check for short retention — the same minimum threshold constraints apply regardless of FULL or LITE mode. WHY NOT C: The retentionDurationCheck.enabled configuration is a standard, user-accessible Spark property. There is no workspace-level restriction that requires Databricks support intervention. WHY NOT E: VACUUM FULL is the default mode of VACUUM and lists all files to identify stale ones. It offers no special bypass of the sub-7-day retention safety check compared to standard VACUUM. Both FULL and LITE modes require disabling the safety check to use thresholds below 7 days.

Correct answer: E

WHY E is correct: Unity Catalog enforces a hierarchical privilege model. To SELECT from a table, a user must also have USE SCHEMA on the parent schema and USE CATALOG on the parent catalog. All three privileges are required — SELECT alone is insufficient even though it is granted directly on the table. WHY NOT A: Granting SELECT on the table alone is not sufficient. Without USE SCHEMA and USE CATALOG, the query engine cannot navigate to the table and the request will be denied. WHY NOT C: ALL PRIVILEGES grants a broad set of capabilities (SELECT, MODIFY, APPLY TAG) but granting more than the minimum violates the principle of least privilege. Additionally, USE SCHEMA and USE CATALOG still need to be granted separately as they are not table-level privileges. WHY NOT D: MODIFY implies write access (INSERT, UPDATE, DELETE) and is far beyond what read-only analysts need. Also, USE SCHEMA on the schema is still missing, so the query would fail. WHY NOT B: USE SCHEMA on catalog1.schema1 is also required. Without it, the catalog navigation stops at the catalog level and the schema and its objects are inaccessible.

Correct answer: A

WHY A is correct: According to the Unity Catalog privilege reference, ALL PRIVILEGES for a table includes SELECT, MODIFY, and APPLY TAG — but it explicitly does NOT include MANAGE. MANAGE is a separate, Public Preview privilege that must be granted independently. Similarly, ALL PRIVILEGES does not include EXTERNAL USE LOCATION or EXTERNAL USE SCHEMA. WHY NOT D: SELECT is included in ALL PRIVILEGES on a table. It grants the ability to read data from the table. WHY NOT B: MODIFY is included in ALL PRIVILEGES on a table. It grants the ability to add, update, and delete data (INSERT, UPDATE, DELETE, COPY INTO). WHY NOT C: APPLY TAG is included in ALL PRIVILEGES on a table. It grants the ability to apply and remove tags (key-value metadata) on the table.

Correct answer: D

WHY D is correct: Deterministic (consistent) hashing maps each unique email to the same hash value every time. This means analysts can JOIN two hashed datasets on the hash column to correlate activity across tables — without ever accessing the raw email. Because SHA-256 (or similar) is a one-way function, analysts cannot reverse the hash to recover the original value, satisfying the anonymization requirement. WHY NOT A: Suppression removes the data entirely, which means there is no identifier left to join on. This would make correlation across datasets impossible, which contradicts the requirement. WHY NOT B: Generalisation replaces specific values with broader categories (e.g., exact age → age range, full email → domain). While it reduces identifiability, joining on a domain like 'gmail.com' is not meaningful for identifying individual users across datasets. WHY NOT C: Tokenization is a pseudonymization technique — it replaces PII with a token while retaining a reverse-mapping. Because re-identification is possible via the mapping table, it does not fully anonymize the data. Additionally, if the goal is solely to enable joins without re-identification risk, hashing is simpler and does not require a mapping store. WHY NOT E: Partial masking (showing part of the email) still exposes PII and would not prevent a determined analyst from identifying users. It is also reversible for common email patterns and does not constitute anonymization.