- 17 Views
- 1 replies
- 0 kudos
Default ACL for Jobs and Clusters
Hi,I want to set default ACL that applies to all created jobs and clusters, according to a cluster policy for example, but currently I need to apply my ACL at every created job/cluster separately.is there a way to do that?BR,
- 17 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @murtadha_s Can you please clarify what you are after? The second part of your question sounded more like a statement: "but currently I need to apply my ACL at every created job/cluster separately," and that confused me a bit. To make sure we poi...
- 0 kudos
- 14 Views
- 1 replies
- 1 kudos
Resolved! Databricks optimization for query perfomance and pipeline run
I am currently working on optimizing several Spark pipelines and wanted to gather community insights on advanced performance tuning. Typically, my workflow for traditional SQL optimization involves a deep dive into the execution plan to identify bott...
- 14 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @sai_sakhamuri You're clearly past the basics. Let me give you a practitioner-level breakdown of each layer you mentioned, plus a few things that often get overlooked.Spark Catalyst Optimizer — Working With the Rules EngineCatalyst operates in fou...
- 1 kudos
- 59 Views
- 3 replies
- 1 kudos
Resolved! Parametrize the DLT pipeline for dynamic loading of many tables
I need to load many tables into Bronze layer connecting to sql server DB. How can i pass the tables names dynamically in DLT. Means one code pass many tables and load into bronze layer
- 59 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi Ashwin Thanks for the quick response. Yes I want to pass all the tables through config parameter/param file and load that into bronze layerI will try this approach. Thanks
- 1 kudos
- 102 Views
- 8 replies
- 0 kudos
Autoloader inserts null rows in delta table while reading json file
Hi,I am exploring Schema inference and Schema evolution using Autoloader.I am reading a single line json file and writing in a delta table which does not exist already (creating it on the fly), using pyspark (below is the code).Code :spark.readStream...
- 102 Views
- 8 replies
- 0 kudos
- 0 kudos
Hi @mits1 can you try adding this option as well:{"multiLine": "true"}
- 0 kudos
- 12 Views
- 0 replies
- 0 kudos
Databricks Workspace - Unknow IP access
Azure monitor log showing unknow ip authentication requests to Databricks workspace . -- When searched ip below url, result showing its from AZURE CLOUD : <Region> (the region is same as workspace)https://azureipranges.azurewebsites.net/SearchFor -- ...
- 12 Views
- 0 replies
- 0 kudos
- 129 Views
- 2 replies
- 0 kudos
DLT Pipeline Error -key not found: all_info_dlt_cx_utils_cod resulting in a NoSuchElementException.
Databricks ETL pipeline, specifically an error with the @DP.expectorfail decorator causing the pipeline update to fail. The error message indicated a 'key not found: all_info_dlt_cx_utils_cod ' resulting in a NoSuchElementException.Note: if we commen...
- 129 Views
- 2 replies
- 0 kudos
- 0 kudos
@MoJaMa Thanks for the reply, The issue was in the code, corrected code worked
- 0 kudos
- 18 Views
- 0 replies
- 0 kudos
How to import python modules in a notebook?
I have a job with a notebook task that utilizes python modules in another folder than the notebook itself. When I try to import the module in the notebook, it raises module not found error. I solved the problem using sys.pathBut I am curious if there...
- 18 Views
- 0 replies
- 0 kudos
- 19 Views
- 0 replies
- 0 kudos
Databricks SQL Alerts and Jobs Integration: Legacy vs V2
Hi,I’m facing a challenge integrating Databricks SQL alerts with a Databricks Job.After reviewing the documentation, I arrived at the following understanding and would appreciate confirmation from the community or Databricks team:Legacy SQL alerts ca...
- 19 Views
- 0 replies
- 0 kudos
- 59 Views
- 2 replies
- 0 kudos
Databricks to Salesforce Core (Not cloud)
Hi,Is there any native connector available to connect salesforce core (not cloud) in Databricks? If no native connector, what are all recommended approaches to connect to Salesforce coreThanks,Subashini
- 59 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @sdurai, Yes. Databricks has a native Salesforce connector for core Salesforce (Sales Cloud / Service Cloud / Platform objects) via Lakeflow Connect - Salesforce ingestion connector. It lets you create fully managed, incremental pipelines from Sal...
- 0 kudos
- 1010 Views
- 19 replies
- 3 kudos
Resolved! Lakeflow SDP failed with DELTA_STREAMING_INCOMPATIBLE_SCHEMA_CHANGE_USE_LOG
Hi,A column was deleted on the source table, when I ran LSDP it failed with error DELTA_STREAMING_INCOMPATIBLE_SCHEMA_CHANGE_USE_LOG : Streaming read is not supported on tables with read-incompatible schema changes( e.g: rename or drop or datatype ch...
- 1010 Views
- 19 replies
- 3 kudos
- 3 kudos
This looks like a very practical template, especially for teams trying to structure their Data & AI strategy without overcomplicating things. The step-by-step format and examples should be really helpful for workshops and collaborative sessions. Curi...
- 3 kudos
- 435 Views
- 7 replies
- 0 kudos
update on iceberg table creating duplicate records
We are using databricks to connect to a glue catalog which contains iceberg tables. We are using DBR 17.2 and adding the jars org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.10.0org.apache.iceberg:iceberg-aws-bundle:1.10.0the spark config is then...
- 435 Views
- 7 replies
- 0 kudos
- 0 kudos
Hi @stemill , The way of connecting to Iceberg tables managed by Glue catalog that you described is not officially supported. Because spark_catalog is not a generic catalog slot – it’s a special, tightly‑wired session catalog with a lot of assumptio...
- 0 kudos
- 40 Views
- 0 replies
- 0 kudos
Databricks workflows for APIs with different frequencies (cluster keeps restarting)
Title: Databricks workflows for APIs with different frequencies (cluster keeps restarting)Hey everyone,I’m stuck with a Databricks workflow design and could use some advice.Currently, we are calling 70+ APIs Right now the workflow looks something l...
- 40 Views
- 0 replies
- 0 kudos
- 1619 Views
- 3 replies
- 2 kudos
Resolved! Using cloudFiles.inferColumnTypes with inferSchema and without defining schema checkpoint
Two Issues:1. What is the behavior of cloudFiles.inferColumnTypes with and without cloudFiles.inferSchema? Why would you use both?2. When can cloudFiles.inferColumnTypes be used without a schema checkpoint? How does that affect the behavior of cloud...
- 1619 Views
- 3 replies
- 2 kudos
- 2 kudos
Behavior of cloudFiles.inferColumnTypes with and without cloudFiles.inferSchema:When cloudFiles.inferColumnTypes is enabled, Auto Loader attempts to identify the appropriate data types for columns instead of defaulting everything to strings, which i...
- 2 kudos
- 49 Views
- 0 replies
- 0 kudos
Best Practices for Implementing Automated, Scalable, and Auditable Purge Mechanism on Azure Databric
Hi All, I'm looking to implement an automated, scalable, and auditable purge mechanism on Azure Databricks to manage data retention, deletion and archival policies across our Unity Catalog-governed Delta tables.I've come across various approaches, s...
- 49 Views
- 0 replies
- 0 kudos
- 133 Views
- 4 replies
- 5 kudos
Python Data Source API — worth using?
Hi all,I’ve been looking into the Python Data Source API and wanted to get some feedback from others who may be experimenting with it.One of the more common challenges I run into is working with applications that expose APIs but don’t have out-of-the...
- 133 Views
- 4 replies
- 5 kudos
- 5 kudos
Adding on to @edonaire, which are accurate. @beaglerot , your contacts project is the right use case for the pattern you have. Small data, infrequent changes, direct read into bronze. That works. The real question you're asking is what happens when t...
- 5 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
