Building on the excellent PowerShell Databricks module created by Gerhard Brueckl here, I’ve added another layer of code to recursively export all items in a given Databricks workspace using PowerShell.
I accept this does need to be hardened as a PowerShell cmdlet on its own and added to a module. However, I wanted to share the first version sooner rather than later to support a couple of ongoing projects where Databricks Notebook CI/CD is becoming an important issue.
The code snippet requires four parameters as follows:
- Token – An access token for your Databricks service which can be generated from the User Setting menu.
- The URL – Your service. In the case of Azure this can just be the region.
- OutputPath – Where you want the exported items to land in your local environment.
- WorkspacePath – The level within your Databricks workspace where you want to recursively export items from.
For the first use you’ll probably need to install/import the DatabricksPS module.
Download the full script from my GitHub repo here:
Beware that currently the export of libraries (eg. jar files) is not support by the workspace API.
Full API details here: https://docs.azuredatabricks.net/api/index.html