Unity Catalog notes

databricks Unity Catalog notes

Logical hierarchy

Managed storage

Metastore, catalog and schema are 3 levels that can be configured with their own manages storage locations (e.g. point to an S3 bucket)

This is for physically separating data.

External storage

External volumes are recommended for the below scenarios:

1. Register landing areas for raw data produced by external systems to support ETL pipelines 

2. Register staging locations for ingestion, for example, using Auto Loader, COPY INTO

3. Provide file storage locations for data scientists, data analysts, when managed volumes are not an option.

4. Give Databricks users access to arbitrary files in other systems, for example image, audio, video

5. Store operational data, such as logging or checkpointing files, when managed volumes are not an option.