Databricks Community

Hi @ankitmit, You may want to see this article: https://www.databricks.com/blog/2023/03/17/production-ready-and-resilient-disaster-recovery-dlt-pipelines.html

The Git Server Proxy is needed for various tasks within the job, and multiple jobs may rely on it. Manually terminating it at the end of job1 could lead to job2 failing.

Hi @Alberto_Umana and @jay250terry , thank you for your reply. I know that spark execute task in parallel and the sum of the each task execution time does not correspond with the overall total job duration.What I don't get from the text execution sum...

Hi @SivaPK , It may happen that the CSV is fine, but Excel often does not automatically recognize CSV files as UTF-8 (it might assume ANSI or Windows-1252). This is why characters can appear incorrect. Instead, open the CSV in a text editor like Note...

I tried lot of combinations , but it seems it doesn't support lot of html tags. No style no color or formating.if anyone aware please provide solution. Also alert sending only 100 rows to email.

Hello @HoussemBL, Here are a few points to consider: Initialization of Generated Columns: Generated columns, such as is_current, rely on the values of other columns (__END_AT in this case) to be correctly populated. During the first run, if the sequ...

This can happen if the table was not properly dropped or if there is a naming conflict. Before creating or refreshing the table, check if it already exists in the catalog.  SHOW TABLES IN <database_name>;

Hi @subhas_hati, Enable AQE to dynamically optimize the join strategy at runtime based on the actual data distribution. This can help in choosing the best join strategy automatically If you are using Delta tables, you can leverage the MERGE statement...

Hello @vineet_chaure, By default, Spark converts LongType to float64 when transferring data to pandas. You can use Arrow-optimized pandas UDFs introduced in Apache Spark 3.5. Please try with below code: import pandas as pdimport pyarrow as pafrom pys...

@TejeshS  You can alternatively use, mask function instead of hardcoding with value 'Masked'.CREATE OR REPLACE VIEW masked_employees AS SELECT Name, Department, CASE WHEN current_user() IN ('ab***@gmail.com', 'xy***@gmail.com') THEN Salary ELSE mask(...

Did you try this in your databricks.yml?experimental: python_wheel_wrapper: true

I've found the JDBC query to be faster than the federated query because in our testing, the federated query does not pass down the full query to the source database. Instead, it's running "select * from table", pulling all of the data into Databricks...

Thanks for the details, are you using any type of container (docker) image? Or any special setting on the cluster?Nothing special, I just click on create compute button, then select 16.1 from the dropdown and click create compute button again.If you ...