Command-line interface CLI
The databricks command-line interface https://docs.databricks.com/dev-tools/cli/index.html provides a way to manage everything related to a databricks instance, including secrets, cluster, dbfs, etc. The CLI is built on top of the REST API https://docs.databricks.com/dev-tools/api/2.0/index.html so an alternative to CLI is to call the REST API directly in program.
Here is an example of using CLI to add scope and key to Databricks secrets.
Pre-requisite
Install the CLI library for connecting to databricks, and also generate a personal access token in databricks to authenticate the connection first.
Install Databricks CLI. Use windows admin account so databricks command is added to path.
pip install databricks-cli
Configure databricks host and access token
databricks configure --token
It will prompt to enter the host and access token.
The token is invisible on the command line, just copy paste and press enter
When finished, a .databrickscfg file (as below) is created under users/username directory. The Config file has the host and token that just put in.
[DEFAULT]
host = https://adb-123345667.19.azuredatabricks.net
token = 1233445556563546345643564
jobs-api-version = 2.0
TO test if it can connect successfully,
databricks clusters list
It returns the clusters. Looks good
Add Secrets
To manage secrets in databricks, it has to use the CLI or through rest api.
It doesn't provide any other user interface in databricks due to security reasons.
Firstly create a scope https://docs.databricks.com/security/secrets/secret-scopes.html
databricks secrets create-scope --scope <scope-name>
Add a key by given a key name https://docs.databricks.com/security/secrets/secrets.html
databricks secrets put --scope <scope-name> --key <key-name>
An editor will subsequently pops up for entering the key. As noted in the editor, enter the password above the dash line. When finished, save and close the editor.
# ----------------------------------------------------------------------
# Do not edit the above line. Everything that follows it will be ignored.
# Please input your secret value above the line. Text will be stored in
# UTF-8 (MB4) form and any trailing new line will be stripped.
# Exit without saving will abort writing secret.
Note a scope can have multiple keys. For example, create a scope for ODBC connection to Something.
Add two keys username and password to the scope. So in this case the scope has two keys.
To list the keys:
databricks secrets list --scope <scope-name>
When programming in databricks notebooks, dbutils can access the key, but you wont be able to print and see it.
dbutils.secrets.get(scope=scope_name,key=key_name)
Running the above command in databricks notebook will show only '[REDACTED]'
dbutils.secrets.get(scope="my-scope", key="my-key")
# Out[14]: '[REDACTED]'
However, the getBytes() will get the secret as per the documentation:
Gets the bytes representation of a secret value for the specified scope and key.
To display help for this command, run dbutils.secrets.help("getBytes").
This example gets the secret value (a1!b2@c3#) for the scope named my-scope and the key named my-key.
my_secret = dbutils.secrets.getBytes(scope="my-scope", key="my-key")
my_secret.decode("utf-8")
# Out[1]: 'a1!b2@c3#'
This doesn't seem to be True. It will show '[REDACTED]' as well even the user is admin.
A work around to show the secret. It seems reading char by char in a loop avoids redaction.
x = dbutils.secrets.get(scope="scope name", key="key name")
for y in x:
print(y)
It will show every char of the secret like
a
1
!
b
2
@
c
3
#