Databricks Community

alesventus · 08-09-2024

Hi, I have several Power BI reports based on Delta Lake tables that are refreshed every 4 hours. ETL process in Databricks is much cheaper that refresh of these Power BI reports. My questions are: if approach described below is correct and if there i...

alesventus · 07-29-2024

My task is to sync 300 tables from on prem sql server to delta lake. I will load CDC from Raw. First step is to move CDC data to bronze with autoloader. Then using delta stream get changes from bronze, make simple datatype changes and merge this data...

alesventus · 07-04-2024

I have databricks job with around 70 notebooks. When job starts, only one notebook gets executed and the rest of the notebooks that are at the beginning of the line are in state PENDING (not blocked). Looks like notebooks cannot run in parallel for t...

Change users status in account console · 11-15-2023

One of our users has status = Inactive in Admin setting. I tried to set it to active using rest api and the documentation but seems like api is not changing status.I tested api and I can list users so api setting are ok.api request: import requests i...

alesventus · 11-08-2023

Hi,My scenario is this: autoloader loads data from ADLS once a day.data are merged into the bronze zone.bronze table has CDF turned on.read CDF and merge into the silver.all works fine, but when there is not new file for autoloader to process, CDF ta...

alesventus · 08-06-2024

Hi @szymon_dybczak I followed your advice, and it worked pretty well. I have generic notebook which accepts parameters as table name and primary keys.In databricks with CLI command databricks jobs create --json {...} I can create job with hundreds o...

alesventus · 07-04-2024

Autoscaling is turned off because I set up manually 5 workers. DLT is not used, these notebooks just merge new data from the source to the bronze zone

alesventus · 07-04-2024

This issue is solely related to the VNET. Azure engineer must set up connection within VNET correctly.

alesventus · 10-17-2023

Thanks @Debayan for your help.Question is why is cluster trying to connect to default hive metastore when we use unity catalog only and we never set up hive metastore? Do we need to set up also Hive metastore?See attached docs.

alesventus · 10-15-2023

Thanks for check of stdout and stderr.Here is log4j.pdf. Its just first 50 pages because this log was huge.

Databricks Community

User Stats

User Activity

Effectively refresh Power BI report based on Delta Lake

How to handle load of 300 tables to delta lake

Tasks in job are in pending state

Change users status in account console

How to check if autoloader processed new file

Re: How to handle load of 300 tables to delta lake

Re: Tasks in job are in pending state

Re: Unity Catalog metastore is down error

Re: Specify bottleneck for databricks cluster

Re: Specify bottleneck for databricks cluster