Databricks

In this topic we will be exploring how to integrate KNIME with Databricks. In the example below I will connect to my Workspace.  I am using Databricks Free Edition for this, so if you want you can set up your own and follow along.

The first thing I have to do is configure the Databricks Workspace Connector. I need two parameters, the Databricks workspace URL and a Personal access token. On the screenshot below the workflow you can see how to get the URL, it is in the red box (please substitute this with your own Databricks URL if you want to try this).

To generate a Personal access token, from the Databricks home, click on your Account icon on the top right of the page, and then select 'Settings' and then 'Developer'. There you will see 'Access tokens' , click on 'Manage' and 'Generate new token'. Copy the generated token and paste it into the Databricks Workspace Connector node. You should now be able to successfully execute the node!

Next, open the configuration dialogue of the Databricks SQL Warehouse Connector and select 'Serverless Starter Warehouse' as the warehouse you want to use.

Next, from the DB Table Selector you can select the table you want to access (in this example, I selected 'call_center'  from 'tpcds_sf1'

In the workflow below I have added two more connectors. The first one - in the red box - will allow me to access the files in the Databricks 'directory' specified in the path. The second one - in the blue box - allows me access - through Databricks - to the Postgres Airlines database stored on the Aiven cloud we talked about on this page