Databricks Community

Hi @subhas_hati, Enable AQE to dynamically optimize the join strategy at runtime based on the actual data distribution. This can help in choosing the best join strategy automatically If you are using Delta tables, you can leverage the MERGE statement...

Hello @vineet_chaure, By default, Spark converts LongType to float64 when transferring data to pandas. You can use Arrow-optimized pandas UDFs introduced in Apache Spark 3.5. Please try with below code: import pandas as pdimport pyarrow as pafrom pys...

@TejeshS  You can alternatively use, mask function instead of hardcoding with value 'Masked'.CREATE OR REPLACE VIEW masked_employees AS SELECT Name, Department, CASE WHEN current_user() IN ('ab***@gmail.com', 'xy***@gmail.com') THEN Salary ELSE mask(...

Did you try this in your databricks.yml?experimental: python_wheel_wrapper: true

I've found the JDBC query to be faster than the federated query because in our testing, the federated query does not pass down the full query to the source database. Instead, it's running "select * from table", pulling all of the data into Databricks...

Thanks for the details, are you using any type of container (docker) image? Or any special setting on the cluster?Nothing special, I just click on create compute button, then select 16.1 from the dropdown and click create compute button again.If you ...

Hi @dbuserng, It is possible but it requires custom code setup based on your use-case. You can use job REST API: https://docs.databricks.com/api/workspace/jobs. Create a Monitoring Job: Set up a job that will monitor the completion status of the thre...

While running the UCX tool in workspace we are not able to access the tables which are created in hive metastore on default schema. When we run the migrate-table workflow we get the error that the tables are not accessible. We get the following error...

@Alberto_Umana  In addition to my last comment:For adjusting the spark.sql.shuffle.partitions and spark.executor.memory, I tried this but I was still seeing the same error  spark = (    SparkSession.builder    .appName("GA4 Bronze Table Ingestion")  ...

Our initial approach was to have catalogs for sources and uses, but that was confusing and the grade of data wasn't obvious.  And, as we began to join data, it was unclear where to put the final datasets.  We switched to three catalogs--one for bronz...