Databricks Community

JesseS · 01-12-2023

Here is the situation I am working with. I am trying to extract source data using Databricks JDBC connector using SQL Server databases as my data source. I want to write those into a directory in my data lake as JSON files, then have AutoLoader ing...

JesseS

I'm eagerly awating the next part. I do have some questions that I hope will get addressed, or maybe someone can share some insights on.1. Are the framework tables stored as Delta tables in the lakehouse? I envision a seperate config schema that wo...

JesseS

I know it's a bit after the fact, but in case you didn't solve it, I came across this article in the Databricks documentation. You can set pipelines.reset.allowed to false on a table to prevent a full refresh of a table. Ref: https://docs.databrick...

JesseS

I would absolutely love a meet-up in the central Wisconsin area (around Wausau), or even in the Fox Valley region (Green Bay, Appleton, etc..). I've been using Databricks for several years and am currently in the processing of migrating our ETL syst...

Databricks Community

User Stats

User Activity

How to extract source data from on-premise databases into a data lake and load with AutoLoader?

Re: Metadata-Driven ETL Framework in Databricks (Part-1)

Re: DLT full refresh

Re: Would You Join Us for a Local Meetup?