Databricks Community

For your consideration: To address the challenge of passing message handles from executors back to the driver within the DataSourceStreamReader, consider the following approaches: Challenges in Spark Architecture 1. Executor Memory Isolation: Execut...

Here are some things considerations:   The errors experienced in your production streaming jobs—ERR_NGROK_3200 and Spark driver failed to start within 900 seconds—stem from distinct causes related to connectivity, underlying system constraints, and d...

@Sidhant07 also using Kylin4 and Sqoop I am getting: "Unsupported transaction isolation level: 2" when running sqoop list-tables. 

I just accomplished by modifying the DatabricksClient by adding the feature_store_uri parameter to it and passing it to each call of get_host_creds. Lets see if databricks releases this simple change soon...

I disabled the compute schema and then enabled it again

Hello @ViniciusFC54! Did you try following the steps mentioned here: Managing access to the Personal Compute policy? It shows how admins can manage access to the Personal Compute policy and control who can use it.

Thanks @Renu_ . That makes sense, I understand that in the DBX workspaces, you can choose between two 16.4 spark versions when creating a compute.My confusion was with using the docker image in local environment. I pull it from the registry but then ...

Hi @Walter_C , Thanks for your suggestion. Exporting the images (=> duplicating the storage) is a no-go in our case. We are talking about very large automotive datasets.Is there another option or should we consider moving this data out of the catalog...

add me https://www.linkedin.com/in/aviralb/ we will do live debugging 

An approach that works well when using a Databricks SQL Warehouse is to use the replace_where strategy - I've just tested this. It also works with partitioned tables:{{ config( materialized='incremental', incremental_strategy='replace_where', ...

If we talk about traditional data warehouse way of building the architecture, we can consider Silver layer as Data mart with star schema kind of relations for dimensions and fact. Can we build entire DWH enterprise scale using databrikcs? I see in pr...

Here is something to consider: The issue you're experiencing likely stems from differences in behavior when accessing Oracle database objects via Spark JDBC versus other database clients like DBeaver. Specifically, Spark's JDBC interface may perform ...