Databricks glossary
Key Databricks certification terms and acronyms.
Definitions are AI-assisted and reviewed for general accuracy — verify critical details against Databricks's official documentation.
ACIDAtomicity, Consistency, Isolation, Durability
A set of transaction properties that guarantee database operations are processed reliably, ensuring data integrity even during failures.
All-Purpose Cluster
A persistent Databricks cluster that remains running and can be shared among multiple users for interactive development and ad hoc analysis.
AUC-ROCArea Under the Receiver Operating Characteristic Curve
A scalar metric summarizing a binary classifier's ability to discriminate between classes across all decision thresholds; ranges from 0 to 1.
Auto Loader
A Databricks structured streaming source that incrementally and efficiently ingests new files from cloud object storage as they arrive.
AutoMLAutomated Machine Learning
Databricks AutoML automatically trains and tunes multiple models on a dataset, generating Python notebooks with the best-found configurations.
Batch Inference
Running a trained model against a large dataset all at once using Spark or pandas, writing predictions back to storage rather than serving live requests.
CDFChange Data Feed
A Delta Lake feature that records row-level changes (insert, update, delete) to a table so downstream consumers can process incremental changes.
Checkpoint
In Structured Streaming, a directory storing the offset and state progress of a streaming query so it can resume after failure without reprocessing data.
Cluster
A set of computation resources in Databricks consisting of a driver node and optional worker nodes that execute Spark jobs and notebook code.
Cluster Policy
An admin-defined set of rules that limits or presets cluster configuration options to enforce cost controls and security standards for workspace users.
Cross-Validation
A model evaluation technique that partitions data into k folds, training on k-1 folds and validating on the remaining fold repeatedly.
Data Drift
A statistical change in the distribution of input features over time that can degrade model performance in production.
Databricks Workflows
A fully managed orchestration service within Databricks for scheduling and running multi-task jobs composed of notebooks, Python scripts, or DLT pipelines.
DBFSDatabricks File System
A distributed file system abstraction mounted in a Databricks workspace that maps to underlying cloud object storage.
Delta Lake
An open-source storage layer that adds ACID transactions, scalable metadata handling, and unified batch/streaming to data lakes on cloud object storage.
Delta Lake
An open-source storage layer that adds ACID transactions, schema enforcement, and time travel to Apache Spark and Databricks data pipelines.
DLTDelta Live Tables
A Databricks framework for building reliable, maintainable, and testable data pipelines using declarative Python or SQL definitions.
Estimator
A Spark MLlib component that implements a fit() method, consuming a DataFrame to produce a trained Transformer (Model).
Expectation
A Delta Live Tables data quality constraint defined with @dlt.expect that specifies a boolean condition rows must satisfy, with configurable failure behavior.
Experiment
An MLflow organizational unit that groups related training runs, identified by a name or ID, within a workspace or notebook.
External Table
A Databricks/Spark table that references data stored at a user-specified location; dropping the table removes only the metadata, not the data files.
Feature Store
A Databricks service for creating, storing, and sharing curated ML features, ensuring consistent feature computation between training and inference.
Hyperparameter Tuning
The process of systematically searching for optimal model configuration values that are set before training rather than learned from data.
Job Cluster
An ephemeral Databricks cluster created automatically when a job starts and terminated when the job completes, optimizing cost for scheduled workloads.
Managed Table
A Databricks/Spark table whose data files are stored in the default metastore location and are deleted when the table is dropped.
Medallion Architecture
A data design pattern that organizes data into Bronze, Silver, and Gold layers representing raw, cleansed, and aggregated quality levels respectively.
MERGE
A Delta Lake SQL command that performs upsert operations by matching source rows to target rows and applying INSERT, UPDATE, or DELETE actions.
MLflow
An open-source platform for managing the ML lifecycle, including experiment tracking, model packaging, and model registry, natively integrated into Databricks.
MLflow Model Registry
A centralized MLflow component for versioning, annotating, and managing model lifecycle stages such as Staging, Production, and Archived.
MLflow Projects
A convention for packaging ML code and its dependencies into a reproducible, reusable format defined by an MLproject YAML file.
MLflow Tracking
The MLflow component that logs parameters, metrics, artifacts, and source code for each training run to enable experiment comparison.
Model Serving
Databricks Model Serving deploys registered MLflow models as scalable REST API endpoints for real-time inference.
Photon
A Databricks native vectorized query engine written in C++ that accelerates Apache Spark SQL workloads on the Databricks Runtime.
Pipeline
A Spark MLlib abstraction that chains multiple Transformers and Estimators into a single reproducible workflow for feature engineering and model training.
pyfuncPython Function Flavor
A generic MLflow model flavor that wraps any Python model with a standard predict() interface, enabling framework-agnostic deployment.
Run
A single MLflow execution instance within an experiment that records parameters, metrics, tags, and artifacts for one training attempt.
SHAPSHapley Additive exPlanations
A game-theory-based framework for explaining individual predictions by computing each feature's marginal contribution to the model output.
Spark MLlibApache Spark Machine Learning Library
Spark's built-in distributed ML library providing algorithms and utilities for classification, regression, clustering, and feature engineering at scale.
Spark UI
A web interface embedded in Databricks that displays execution plans, DAGs, stages, and task metrics for running and completed Spark jobs.
StringIndexer
A Spark MLlib Estimator that encodes a categorical string column to a numeric index column ordered by label frequency.
Structured Streaming
A scalable, fault-tolerant stream processing engine built on Apache Spark that treats a live data stream as an unbounded table.
Time Travel
A Delta Lake feature that allows querying historical versions of a table using a version number or timestamp.
Transaction Log
The Delta Lake _delta_log directory that stores an ordered JSON record of every transaction committed to a table, enabling ACID guarantees and time travel.
Transformer
A Spark MLlib component that implements a transform() method to convert one DataFrame into another, typically for feature engineering steps.
Unity Catalog
Databricks' unified governance solution for data and AI assets, providing centralized access control, auditing, and lineage across workspaces.
Unity Catalog
A unified governance solution for Databricks that provides centralized access control, auditing, lineage, and data discovery across workspaces.
VACUUM
A Delta Lake command that removes files no longer referenced by the table transaction log that are older than a specified retention threshold.
VectorAssembler
A Spark MLlib Transformer that combines multiple numeric and vector columns into a single feature vector column required by MLlib algorithms.
Widgets
Parameterized input controls in Databricks notebooks (text, dropdown, combobox, multiselect) that allow dynamic values to be passed to notebook code.
Z-Ordering
A Delta Lake data-skipping optimization technique that co-locates related information in the same set of files to reduce query scan times.