Development and Ingestion — DE Associate Practice Questions

1 Development and Ingestion

A data engineer is building an Auto Loader pipeline to ingest product catalog CSV files from an S3 bucket. The CSV files include a header row, may have trailing whitespace in values, and contain a product_launch_date column that the engineer wants stored as a DATE type rather than a string for downstream compatibility. The engineer wants to enable schema inference so the schema does not need to be fully hardcoded, but wants to enforce the DATE type for product_launch_date via a schema hint. The cloudFiles.schemaLocation should be set to /checkpoints/catalog/schema, and the checkpointLocation should be /checkpoints/catalog. Which Auto Loader configuration fragment correctly satisfies all requirements?

A``python .format("cloudFiles") .option("cloudFiles.format", "csv") .option("header", "true") .option("ignoreTrailingWhiteSpace", "true") .option("cloudFiles.schemaLocation", "/checkpoints/catalog/schema") .option("cloudFiles.schemaHints", "product_launch_date DATE")`This is the correct and complete configuration. ThecloudFiles.schemaHints option enforces the DATE type for product_launch_date while allowing all other columns to be inferred, the header option ensures the CSV header row is used, and ignoreTrailingWhiteSpace` handles the whitespace cleanup.
B``python .format("cloudFiles") .option("cloudFiles.format", "csv") .option("header", "true") .option("ignoreTrailingWhiteSpace", "false") .option("cloudFiles.schemaLocation", "/checkpoints/catalog/schema") .option("inferSchema", "true") .option("schema", "product_launch_date DATE")`UsinginferSchema=true alongside a partial schema` declaration for one column provides the hybrid inference + type-enforcement behavior needed for this use case.
C``python .format("json") .option("cloudFiles.format", "csv") .option("header", "true") .option("ignoreTrailingWhiteSpace", "true") .option("cloudFiles.schemaLocation", "/checkpoints/catalog/schema") .option("cloudFiles.schemaHints", "product_launch_date DATE")`The outer.format("json") declaration is required because Auto Loader internally converts CSV to JSON before applying schema inference; cloudFiles.format` overrides this to ensure CSV parsing.
D``python .format("cloudFiles") .option("cloudFiles.format", "csv") .option("header", "true") .option("ignoreTrailingWhiteSpace", "true") .option("cloudFiles.schemaLocation", "/checkpoints/catalog/schema") .option("cloudFiles.schemaHints", "product_launch_date DATE") .option("cloudFiles.inferColumnTypes", "true")`This is the most complete configuration:cloudFiles.inferColumnTypes=true ensures non-string types (integers, booleans, etc.) are inferred rather than defaulting to strings for all CSV columns, cloudFiles.schemaHints enforces DATE for product_launch_date, header=true processes the CSV header, and ignoreTrailingWhiteSpace=true` handles whitespace cleanup.

Show answer & explanation

Correct answer: D

WHY D: Option D is the most complete and accurate. For CSV files, Auto Loader defaults to inferring all columns as strings. To get actual typed inference (integers, booleans, dates, etc.), cloudFiles.inferColumnTypes=true must be set. cloudFiles.schemaHints then allows specifying the exact type for product_launch_date as DATE — schema hints work alongside inference to enforce known column types. header=true is needed for CSV header processing, and ignoreTrailingWhiteSpace=true trims trailing whitespace. WHY NOT A: Partially correct but incomplete. Without cloudFiles.inferColumnTypes=true, all CSV columns (other than the schema-hinted product_launch_date) are inferred as strings rather than their native types (e.g., numeric columns stay as strings). WHY NOT B: inferSchema is a standard Spark CSV option, not the Auto Loader equivalent. For Auto Loader, use cloudFiles.inferColumnTypes. The schema option sets a full explicit schema, not a partial type hint. ignoreTrailingWhiteSpace=false also disables the required whitespace trimming. WHY NOT C: Using .format('json') would make Databricks attempt to read the files as JSON — the cloudFiles.format option does not override the outer .format() call. Auto Loader requires .format('cloudFiles') as the stream format.

2 Development and Ingestion

A data engineer manages an Auto Loader stream ingesting JSON files. After two weeks of operation, incoming JSON files from the data provider start including two new fields: device_firmware_version (string) and battery_level (integer). The engineer is reviewing how Auto Loader's cloudFiles.schemaEvolutionMode setting will handle these new columns under each available mode. The current stream was started with an explicit schema provided and cloudFiles.schemaEvolutionMode was not explicitly configured. A colleague asks what the default schema evolution behavior is, and how the four available modes differ. Which statement correctly summarizes all four evolution modes and identifies the default mode when a schema is explicitly provided?

AWhen a schema is provided, the default cloudFiles.schemaEvolutionMode is addNewColumns, which automatically adds new columns to the schema without interrupting the stream; the rescue mode sends schema change events to a DLT expectations log instead of failing; the failOnNewColumns mode throws an unrecoverable exception and requires the pipeline to be fully deleted and recreated with the new schema; and the none mode ignores all new columns silently without rescuing their data.
BAll four modes behave identically when a schema is provided — they all stop the stream and wait for manual schema intervention since user-provided schemas are treated as immutable contracts; cloudFiles.schemaEvolutionMode only has an effect when schema inference (without a user-provided schema) is active, because Auto Loader cannot modify a schema that the user has explicitly declared.
CWhen an explicit schema is provided, the default cloudFiles.schemaEvolutionMode is none (new columns are dropped silently, stream does not fail); the addNewColumns mode is not allowed when an explicit schema is provided (it works only with schema hints); the rescue mode captures unmatched columns in _rescued_data without failing the stream; and the failOnNewColumns mode stops the stream when new columns are encountered and requires either an updated schema or removal of the offending files before restart.
DWhen an explicit schema is provided, the default cloudFiles.schemaEvolutionMode is failOnNewColumns, which stops the stream immediately upon detecting new columns to prevent silent data loss; the addNewColumns mode automatically evolves the schema and resumes the stream without any interruption; the rescue mode collects all unmatched column data into a JSON blob in a _rescued_data column; and none ignores structural changes entirely and processes each file as if the schema never changed.

Show answer & explanation

Correct answer: C

WHY C: Per official Databricks documentation, when a user provides an explicit schema, the default cloudFiles.schemaEvolutionMode is none — new columns are ignored, data is not rescued unless rescuedDataColumn is set, and the stream does not fail. The four modes are: addNewColumns (default when NO schema is provided — stream fails, new columns added to schema, existing types unchanged; NOT allowed when an explicit schema is provided unless schema hints are used), rescue (schema never evolves, new columns go to _rescued_data, stream never fails on schema changes), failOnNewColumns (stream stops, requires schema update or file removal before restart), and none (schema unchanged, new columns dropped, no failure). WHY NOT A: The default when a schema IS provided is none, not addNewColumns. addNewColumns is the default only when no schema is provided. The rescue mode does not send events to DLT expectations logs — it captures data in _rescued_data column. WHY NOT B: The four modes have distinct behaviors even when a schema is provided. cloudFiles.schemaEvolutionMode is fully functional regardless of whether schema inference or explicit schema is used. WHY NOT D: The default when an explicit schema is provided is none, not failOnNewColumns. The addNewColumns behavior causes a stream stop/restart cycle, not a seamless automatic resume without interruption.

3 Development and Ingestion

A data engineer at a logistics company needs to choose the right ingestion approach for three different data scenarios: (1) A continuous feed of IoT sensor JSON files arriving at a rate of 10,000 files per hour into an S3 bucket, requiring near-real-time Bronze table updates with exactly-once semantics and fault tolerance after failures. (2) A one-time historical backfill of 2 billion legacy Parquet files from ADLS Gen2 that were generated over 5 years and need to be loaded into a Delta table. (3) A daily batch of 50 CSV report files dropped into a GCS bucket each morning, which should be loaded incrementally so only new files added since the last run are processed. In which scenario(s) is Auto Loader the most appropriate and recommended ingestion tool?

AAuto Loader is the correct tool exclusively for Scenario 1 (continuous near-real-time streaming) because it uses Spark Structured Streaming under the hood; for Scenario 2, COPY INTO (legacy batch) is the only tool capable of handling billions of files; and for Scenario 3, a standard spark.read.csv() batch job with a custom date-filter predicate on file modification timestamps is the recommended approach since Auto Loader adds unnecessary complexity for small daily batches.
BAuto Loader is not suitable for any of the three scenarios in production; Scenario 1 should use Kafka Connect with a Delta Lake sink connector for exactly-once semantics; Scenario 2 requires the Databricks COPY INTO command for bulk file ingestion; and Scenario 3 requires a manual Python script that lists GCS bucket contents, filters for files added in the last 24 hours using the GCS metadata API, and then runs a batch spark.read.csv() load with a MERGE INTO command to avoid duplicates.
CAuto Loader is the recommended tool for all three scenarios: it natively supports Structured Streaming with exactly-once guarantees and RocksDB-backed checkpointing for Scenario 1; it can efficiently process billions of files for large-scale backfills in Scenario 2 using the cloudFiles.includeExistingFiles=true option; and for Scenario 3, incremental processing of only new files is Auto Loader's default behavior, making it ideal for daily batch ingestion without requiring a manual file-tracking mechanism.
DAuto Loader is appropriate for Scenarios 1 and 3 only; for Scenario 2 (2 billion historical Parquet files), Auto Loader is explicitly unsupported for backfills because it cannot process files that existed before the stream was first started — the cloudFiles.includeExistingFiles option only applies to files added in the 7 days prior to stream start and silently ignores older files, making a full 5-year historical backfill impossible without COPY INTO.

Show answer & explanation

Correct answer: C

WHY C: Auto Loader is designed and recommended for all three scenarios. For Scenario 1, it provides Structured Streaming with exactly-once guarantees via RocksDB checkpointing — capable of handling millions of files per hour. For Scenario 2, the official Databricks docs explicitly state: 'You can use Auto Loader to process billions of files to migrate or backfill a table' — cloudFiles.includeExistingFiles=true (which is the default) includes all existing files. For Scenario 3, Auto Loader's default behavior processes only new files since the last run by tracking file metadata in its checkpoint — making it perfect for incremental daily batch ingestion. WHY NOT A: Auto Loader supports all three scenarios. The official docs specifically call out billion-file backfills as a supported use case. Small daily batches also benefit from Auto Loader's built-in exactly-once tracking vs. manual timestamp filtering. WHY NOT B: Auto Loader is production-grade for all three scenarios. Kafka Connect is for streaming from Kafka topics, not S3 files. Manual GCS listing scripts are error-prone and unnecessary when Auto Loader provides built-in file tracking. WHY NOT D: The cloudFiles.includeExistingFiles=true option (the default) includes ALL existing files regardless of age — there is no 7-day limit. Auto Loader can process files from any time period for historical backfills.

4 Development and Ingestion

A data engineer working in VS Code has installed Databricks Connect (v14.x) and configured it to connect to a Databricks workspace using a personal access token. They have written the following PySpark code in a local Python file transform.py: ``python from databricks.connect import DatabricksSession spark = DatabricksSession.builder.remote().getOrCreate() df = spark.read.table("main.sales.transactions") result = df.filter(df.amount > 1000).groupBy("region").agg({"amount": "sum"}) result.show()`` The engineer runs the file locally. A colleague asks: at which exact line does Databricks Connect send an execution request to the remote cluster, and what happens on the preceding lines? Which answer correctly identifies the execution trigger point and the nature of the preceding operations?

AAll five lines execute locally without any cluster communication until the program exits, at which point Databricks Connect batches and submits all accumulated DataFrame operations to the remote cluster as a single optimized query plan; the show() call at the end merely signals the end of the local plan accumulation phase before the batch submission occurs.
BThe cluster is contacted on line 2 when DatabricksSession.builder.remote().getOrCreate() is called to establish a gRPC connection and open a remote Spark session; subsequent lines build unresolved logical plans locally; show() on line 5 triggers the actual data scan and computation on the cluster and materializes and returns results to the local client.
CThe cluster is not contacted at all during local execution of transform.py; instead, Databricks Connect queues all five lines in a local execution manifest that is automatically submitted as a scheduled job to the Databricks Jobs API the next time the workspace syncs, and results are stored in a Delta table on the cluster rather than being materialized in the local terminal output.
DLines 1 through 4 execute entirely on the cluster because importing DatabricksSession triggers an automatic remote code upload through the Spark Connect protocol; result.show() on line 5 is the only line that runs locally, as it calls the Databricks REST API to retrieve and display the pre-computed result that was already stored in the cluster's driver memory by the preceding remote execution steps.

Show answer & explanation

Correct answer: B

WHY B: Line 2 (getOrCreate()) establishes the remote Spark session by opening the gRPC connection to the Databricks cluster. Lines 3–4 build lazy, unresolved logical plans locally — read.table() declares a data source, and filter().groupBy().agg() chain transformations without sending any data. Line 5 result.show() is the action that triggers remote execution: the accumulated logical plan is sent to the cluster, executed, and results are materialized and returned to the local client. This matches Spark's lazy evaluation model under Databricks Connect. WHY NOT A: Databricks Connect does not batch-submit all operations on program exit. Actions like show() immediately trigger remote execution when called — there is no deferred batch submission phase. WHY NOT C: Databricks Connect is an interactive live session library, not a job queuing system. Results are returned to the local terminal immediately when show() is called, not queued or stored in a Delta table. WHY NOT D: Importing DatabricksSession and building DataFrames do not upload or execute code on the cluster. Transformations are lazy plan builders — only the show() action on line 5 sends the plan to the cluster for computation.

5 Development and Ingestion

A senior data engineer is reviewing a junior engineer's Auto Loader pipeline code before it goes to production. The pipeline ingests JSON files from an ADLS Gen2 path into a Bronze Delta table. The junior engineer's code is shown below: ``python (spark.readStream .format("cloudFiles") .option("cloudFiles.format", "json") .option("cloudFiles.schemaLocation", "/checkpoints/bronze/events/schema") .option("cloudFiles.schemaEvolutionMode", "rescue") .option("cloudFiles.useNotifications", "true") .option("cloudFiles.maxFilesPerTrigger", "500") .load("abfss://raw@storageacct.dfs.core.windows.net/events/") .writeStream .format("delta") .option("checkpointLocation", "/checkpoints/bronze/events") .trigger(availableNow=True) .table("main.bronze.events") )`` The senior engineer identifies two specific behaviors the junior engineer should understand about this configuration before deploying to production. Which statement correctly identifies both behaviors?

AFirst, cloudFiles.useNotifications=true requires that an Azure Event Grid subscription and Azure Queue Storage resource have been manually created and configured by a cloud administrator before the stream starts, or Databricks will not automatically provision them and will raise a CloudFilesNotificationSetupException. Second, cloudFiles.maxFilesPerTrigger=500 is incompatible with trigger(availableNow=True) — the availableNow trigger always processes ALL pending files in a single batch regardless of maxFilesPerTrigger, making the limit setting meaningless in this configuration.
BFirst, the cloudFiles.schemaEvolutionMode=rescue setting means that when new JSON fields appear, they will NOT cause the stream to fail or update the schema — instead, unmapped fields are captured in a _rescued_data JSON column, allowing the stream to continue processing without interruption; the downstream events Delta table will receive a _rescued_data column containing any fields not in the current schema. Second, trigger(availableNow=True) runs the stream as an incremental batch — processing all files that have arrived since the last checkpoint in one or more microbatches and then stopping — making it suitable for Lakeflow Jobs-scheduled runs rather than a continuously running stream.
CFirst, cloudFiles.schemaEvolutionMode=rescue is invalid when cloudFiles.useNotifications=true because file notification mode requires strict schema validation (failOnNewColumns) to ensure that cloud event payloads always match the registered table schema in Unity Catalog; mixing these two options will cause a SchemaEvolutionConflictException at stream startup. Second, setting cloudFiles.maxFilesPerTrigger to any value greater than 100 causes the Auto Loader file-tracking RocksDB store to grow unboundedly, eventually exceeding the driver node's disk allocation and crashing the stream after several days of production operation.
DFirst, using cloudFiles.schemaEvolutionMode=rescue means Auto Loader will attempt to automatically migrate the target Delta table schema (adding new columns) on every microbatch using a MERGE INTO schema operation, which makes the stream incompatible with Delta table OPTIMIZE and VACUUM operations running concurrently. Second, trigger(availableNow=True) is not a valid Spark Structured Streaming trigger for Auto Loader streams — only trigger(processingTime='N seconds') and trigger(once=True) (deprecated) are valid; using availableNow will silently fall back to trigger(processingTime='0 seconds') continuous mode.

Show answer & explanation

Correct answer: B

WHY B: Both observations are accurate. (1) cloudFiles.schemaEvolutionMode=rescue means Auto Loader never fails or modifies the schema when new JSON fields arrive — instead, unexpected fields are collected into _rescued_data as a JSON blob. This is ideal for pipelines where schema stability is critical. (2) trigger(availableNow=True) is a valid Spark Structured Streaming trigger (introduced to replace the deprecated Trigger.Once()) that processes all pending data in one or more incremental batches and then terminates — making it well-suited for scheduled batch jobs in Lakeflow Jobs, as opposed to a continuously running stream that never stops. WHY NOT A: Databricks can automatically provision Azure Event Grid and Queue Storage if the workspace has the necessary Azure service principal permissions configured. Manual pre-provisioning is not always required. Also, maxFilesPerTrigger does interact with availableNow — it controls the number of files per microbatch even in availableNow mode (multiple microbatches may run). WHY NOT C: There is no conflict between rescue evolution mode and useNotifications=true — these options operate independently. There is also no documented RocksDB growth limitation related to maxFilesPerTrigger values greater than 100. WHY NOT D: rescue mode does not perform Delta schema migrations — it explicitly does NOT evolve the schema (that's the behavior of addNewColumns). Also, trigger(availableNow=True) is a fully valid and documented trigger type for Auto Loader streams — it does not fall back to continuous mode.

6 Development and Ingestion

A data platform architect is evaluating Databricks Connect as the local development solution for a team of five Python data engineers who build ETL pipelines using PySpark. The team uses Python virtual environments managed by pyenv, connects to a Databricks workspace on Azure, and needs to install custom Python libraries both locally (for non-Spark logic) and on the cluster (for UDFs). One engineer raises concerns about dependency management: where should application dependencies be installed, and where should UDF dependencies be installed when using Databricks Connect? Which statement correctly describes the dependency management model for Databricks Connect?

AAll dependencies — both application-level and UDF-level — must be installed exclusively on the Databricks cluster using %pip install commands in an initialization notebook that runs before the remote Spark session starts, because Databricks Connect transmits all Python code (including non-Spark application logic) to the remote cluster for execution; no Python packages need to be installed in the local virtual environment for the pipeline to function.
BApplication-level dependencies that are used in local Python code should be installed in the local Python virtual environment (for example, as part of the project's requirements.txt), while UDF dependencies — libraries required by functions dispatched to cluster workers via UDFs, foreach, or foreachBatch — must be installed on the Databricks cluster environment because UDF code is serialized and executed remotely on the cluster.
CDatabricks Connect eliminates the need to manage cluster-level dependencies entirely because all UDFs are executed locally on the developer's machine using the local virtual environment's installed packages, meaning that any library installed via pip install in the local environment will automatically be available within UDFs that run on cluster workers through the Spark Connect gRPC serialization protocol.
DDatabricks Connect requires all Python dependencies to be defined in a shared Conda environment YAML file that is synchronized with the cluster via the Databricks Workspace Files API before every development session; without this synchronization step, any import statement in the pipeline code — whether in local Python logic or in Spark UDFs — will raise a ModuleNotFoundError on both the local machine and the cluster.

Show answer & explanation

Correct answer: B

WHY B: Per official Databricks Connect documentation, the dependency split follows the execution split: application dependencies used in local Python code run locally and must be installed in the local project environment (e.g., the Python virtual environment). UDF dependencies — libraries required by udf, foreach, foreachBatch, or transformWithState code that executes on cluster workers — must be installed on the Databricks cluster, since UDF code is serialized locally and deserialized/executed on remote workers. WHY NOT A: Databricks Connect does not run all Python code on the cluster. Non-Spark Python logic runs locally, so application-level dependencies must be in the local environment, not just on the cluster. WHY NOT C: UDFs run on cluster workers, not locally. Local library installations are not automatically propagated to cluster workers through the gRPC protocol. WHY NOT D: There is no requirement for a Conda YAML synchronization step via the Workspace Files API. Databricks Connect manages execution splitting without requiring synchronized environment YAML files.

7 Development and Ingestion

A data engineering team develops complex PySpark transformation pipelines that are version-controlled in a private GitHub repository. Developers currently clone notebooks to their local machines, edit them offline, and then manually copy the code back into the Databricks workspace, causing frequent merge conflicts and making interactive debugging with their IDE nearly impossible. The team lead wants to adopt Databricks Connect so developers can run and debug PySpark code directly from PyCharm or VS Code against live Databricks compute. A new team member asks how Databricks Connect routes code execution — specifically, which parts of the code run locally on the developer's machine versus which parts run on the Databricks cluster. Which statement most accurately describes the execution model of Databricks Connect (DBR 13.3 LTS and above)?

AWith Databricks Connect, general Python control flow and non-Spark code runs locally on the developer's machine, while all Spark DataFrame transformations are converted to logical plans and executed on the remote Databricks compute; UDFs serialized from the local environment are transmitted to and run on the cluster; and results of actions like collect(), show(), or toPandas() are materialized on the client side after remote execution.
BDatabricks Connect sends the entire Python script file to the Databricks cluster as a remote execution request via SSH, where it is executed in full on the cluster's driver node — meaning all code, including non-Spark Python logic, runs remotely — and the final output is streamed back to the developer's terminal as standard output, with no differentiation between local and remote execution boundaries.
CDatabricks Connect compiles the entire PySpark notebook into a serialized JVM bytecode bundle that is submitted to the Databricks cluster through the Jobs API, where it is queued and executed in batch mode as a standard job run; developers benefit from IDE integration only at the code-writing stage but cannot interactively inspect runtime variable states or step through DataFrame transformations using IDE breakpoints.
DDatabricks Connect requires all Python code — including imports, variable assignments, and control flow logic — to be written as pure Spark SQL expressions before it can be submitted to the remote cluster, because the underlying Spark Connect protocol only supports SQL-based logical plan serialization and does not natively handle Python object serialization or general-purpose Python function calls outside of registered UDFs.

Show answer & explanation

Correct answer: A

WHY A: Databricks Connect uses the open-source Spark Connect protocol (gRPC + Arrow) to split execution: general Python and Scala code runs locally on the client, while Spark DataFrame API calls are converted to unresolved logical plans and executed on remote Databricks compute. UDFs defined locally are serialized and transmitted to the cluster for execution. Actions like collect(), show(), and toPandas() trigger remote execution and materialize results on the client. This model enables true IDE debugging of Python logic locally while leveraging scalable cluster compute for Spark transformations. WHY NOT B: Databricks Connect does not send the entire script over SSH and execute it fully on the driver. The execution is split — Python code runs locally and only Spark plans are sent remotely through gRPC. WHY NOT C: Databricks Connect is not a job submission system. It provides an interactive, live remote Spark session that supports runtime debugging, not async batch job execution through the Jobs API. WHY NOT D: Databricks Connect supports the full Python/PySpark DataFrame API, not just SQL expressions. The Spark Connect protocol serializes DataFrame logical plans, not SQL strings, and does support general Python code execution locally alongside Spark operations.

8 Development and Ingestion

A data engineering team is designing an Auto Loader pipeline to ingest daily transaction files that arrive in an Azure Data Lake Storage Gen2 (ADLS Gen2) container. A new team member asks which cloud storage sources are officially supported by Auto Loader and whether it can also read from Databricks File System (DBFS) paths. The team lead also wants to confirm which file formats are supported. Which statement correctly identifies all supported Auto Loader source storage systems and a complete list of supported file formats?

AAuto Loader supports only cloud-native object storage services (Amazon S3, Azure ADLS Gen2, and Google Cloud Storage) as source storage systems; DBFS paths are explicitly excluded because DBFS is a virtual filesystem overlay that cannot generate the file-event notifications required by Auto Loader's internal processing engine. Supported file formats include JSON, CSV, Parquet, and Avro only — XML, ORC, text, and binary files must be read using standard Spark batch readers and cannot be ingested with Auto Loader.
BAuto Loader can ingest from Amazon S3 (s3://), Azure Data Lake Storage Gen2 (abfss://), Google Cloud Storage (gs://), Azure Blob Storage (wasbs://, though deprecated in favor of ABFS), and Databricks File System (dbfs:/); supported file formats include JSON, CSV, XML, Parquet, Avro, ORC, TEXT, and BINARYFILE — covering all eight formats.
CAuto Loader supports only Amazon S3 and Azure ADLS Gen2 as source storage systems on their respective cloud platforms (AWS and Azure); it does not support Google Cloud Storage or DBFS as sources. Supported file formats extend to JSON, CSV, Parquet, Avro, ORC, TEXT, and BINARYFILE — seven formats total — while XML support requires a separately licensed Databricks XML connector library that is not bundled with the standard Auto Loader implementation.
DAuto Loader acts as a metadata-only ingestion controller that does not directly interface with cloud storage APIs; instead, it reads file listings generated by the Lakeflow Connect ingestion service and delegates actual file reads to Spark's standard batch format readers. As a result, it natively supports only the storage formats that are registered in the workspace's Unity Catalog external locations, and any cloud storage path not registered as an external location will raise an AuthorizationException even if the cluster's instance profile has direct S3 or ADLS access.

Show answer & explanation

Correct answer: B

WHY B: Per official Databricks documentation, Auto Loader supports the following source storage systems: Amazon S3 (s3://), Azure Data Lake Storage Gen2 (abfss://), Google Cloud Storage (gs://), Azure Blob Storage (wasbs://, deprecated in favor of ABFS), and Databricks File System (dbfs:/). Supported file formats include JSON, CSV, XML, PARQUET, AVRO, ORC, TEXT, and BINARYFILE — all eight. This makes Auto Loader extremely versatile for multi-cloud and multi-format ingestion scenarios. WHY NOT A: DBFS (dbfs:/) is a supported Auto Loader source. Furthermore, the format list is incomplete — XML, ORC, TEXT, and BINARYFILE are all officially supported. WHY NOT C: GCS is a fully supported Auto Loader source. XML is natively supported by Auto Loader without any separate licensed connector library since Databricks Runtime 14.3 LTS. WHY NOT D: Auto Loader directly interfaces with cloud storage APIs (using cloud-native listing and file notification services) and does not delegate to Lakeflow Connect. Unity Catalog external location registration is not required on all cloud deployments for Auto Loader to function.

Take the full DE Associate practice test →

Keep going

Development and Ingestion: Databricks Data Engineer Associate Practice Questions

Other DE Associate topics