Databricks

Aashita · 08-08-2023

In this blog, I would like to introduce to you the Databricks lakehouse platform and explain concepts like batch processing, streaming, apache spark at a high level and how it all ties together with structured streaming.Table of ContentsWhat is Datab...

Aashita · 01-12-2023

To add to @werners point, I would use ADF to load SQL server data into ADLS Gen 2 as json. Then Load these Raw Json files from your ADLS base location into a Delta table using Autoloader.Delta Live Tables can be used in this scenario.You can also reg...

Aashita · 10-30-2022

Currently there is no way to integrate these two. This is a work in progress.

Aashita · 05-13-2022

@Christopher Flach , can you paste the trail when you run this in the notebook?%sh #!/bin/bash #Start with update sudo apt-get update #######rgdal #This installs gdal on the linux machine but not the R library (done in R script) sudo apt-get i...

Aashita · 05-13-2022

@Eric P , To keep the cluster active at all times, Disable "Automatic termination" create a notebook with simple print "%sql select 1" commands and schedule it to run at regular intervals(avoid scheduling forever) to keep the cluster active all the ...

Aashita · 05-13-2022

@Krishna Kommineni ,MERGE command allows you to perform “upserts”, which are a mix of an UPDATE and an INSERTTo understand upserts, imagine that you have an existing table (a.k.a. a target table), and a source table that contains a mix of new record...

Databricks

User Stats

User Activity

Beginners Guide to Databricks: Batch Processing and Streaming

Re: How to extract source data from on-premise databases into a data lake and load with AutoLoader?

Re: Is there a way to integrate Databricks secrets with AWS secrets manager?

Re: How do you install R packages?

Re: Keep long-running notebook alive when closing browser

Re: How to merge delta data..