Databricks Community

Cary · 05-21-2025

Introduction Every data warehousing project once defined and deemed necessary contains three primary tasks: Ingest data from source.Populate and transform data for reportingLeverage Slowly Changing Dimensions Type II methodology (maintain history) ...

Cary · 05-16-2024

It looks like a couple of things. You may not have setup the abfss path as an external location. You may not have added the location to the IAM controls for the container to the Databricks Connector. I am assuming that you are a workspace and cata...

Cary · 05-16-2024

Two other things we would generally recommend are 1.) Direct Connect and 2.) Using AWS DMS, or any other CDC tool set. There are lots. We have Query Federation now in the Unity Catalog, so you could set up the connection that way, but JDBC/ODBC con...

Cary · 05-15-2024

I would use widgets in the notebook which will process in Jobs. SQL in Notebooks can use parameters, as would the SQL in the jobs with parameterized queries now supported.

Cary · 09-02-2021

One clarification to the point above. The data is stored in your (the customer's) account and S3. All of the ephemeral workers are spun up in your own account and VPC. Very little information is actually stored in the Databricks account, other tha...

Cary · 09-02-2021

Without knowing all that you are trying to do, the answer is yes, with the Instance Profile API. https://docs.databricks.com/dev-tools/api/latest/instance-profiles.html. You might also check out the SCIM APIs to associate the Instance Profile to a g...

Databricks Community

User Stats

User Activity

How to perform CDC Data Load and SCD Type II with Databricks Lakeflow Connect and Pipelines

Re: Error Message when Creating Catalog

Re: Set up connection to on prem sql server

Re: Using parameters in a SQL Notebook and COPY INTO statement

Re: Where the data is stored in processing

Re: Is there a way to add an instance profile to a Databricks group using REST API?