- 31695 Views
- 9 replies
- 5 kudos
Resolved! Can I change from default showing first 1,000 to return all records when query?
I have to query a data for showing in my dashboard.But it truncated the results and showing only first 1,000 rows.In the dashboard view, there is no option to re-execute with maximum result limits.I don't want to switch back to standard view and clic...
- 31695 Views
- 9 replies
- 5 kudos
- 2891 Views
- 2 replies
- 0 kudos
Problem connecting vscode with databricks
I have a problem connecting VSCode with databricks. I am doing all the steps to add databricks in VSCode but after the last step I get this error in VSCode:“Error connecting to the workspace: "Can't set configuration 'authProfile' without selecting a...
- 2891 Views
- 2 replies
- 0 kudos
- 0 kudos
I faced the same problem today. The case was that the databricks.yml file inside my bundle did not include a targets mapping. I created one like so then added the relevant profile in the .databrickscfg fileand it got detected by the extension Hope i...
- 0 kudos
- 1097 Views
- 2 replies
- 1 kudos
ModuleNotFoundError or Permission denied are occurring when running Job with Git and All purpose
"ModuleNotFoundError" or "Permission denied" are occurring when running Jobs with Git provider and All purpose clusters.Which one occurs depends on the workspace, but it is reproducible.Is that no longer recommended?I can run it by serverless cluster...
- 1097 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Renu_ ,Thank you so much for your advice. I will try to both patterns.Regards,
- 1 kudos
- 1416 Views
- 2 replies
- 2 kudos
Resolved! Permission denied on shallow cloned table write on single cluster
If I want to modify a shallow cloned table with partitionOverwriteMode dynamic on a "dedicated/single user" cluster DBR 16.4 i get following error message: Py4JJavaError: An error occurred while calling o483.saveAsTable.: org.apache.spark.SparkExcept...
- 1416 Views
- 2 replies
- 2 kudos
- 3813 Views
- 8 replies
- 1 kudos
Cannot apply liquid clustering via DLT pipeline
I want to use liquid clustering on a materialised view created via a DLT pipeline, however, there doesn't appear to be a valid way to do this.Via table properties:@Dlt.table( name="<table name>, comment="<table description", table_propert...
- 3813 Views
- 8 replies
- 1 kudos
- 1 kudos
Hi everyone, in our project we are trying to implement liquid clustering. We are testing liquid clustering with a test table called status_update, where we need to update the status for different market IDs. We are trying to update the status_update ...
- 1 kudos
- 1140 Views
- 2 replies
- 2 kudos
Resolved! Actively Seeking Data Engineering Opportunities – Impact-Driven & Committed Profile
Hello everyone,My name is Tchalim M'Bandakpa, a passionate Data Engineer based in West Africa (Lomé, Togo), with a strong interest in distributed systems, large-scale data processing performance, and modern architectures such as the Lakehouse paradig...
- 1140 Views
- 2 replies
- 2 kudos
- 2 kudos
Hello @Tchalim! It’s great to have such a passionate and skilled Data Engineer join the Community . Your background and technical strengths are highly valuable here.I encourage you to engage, ask questions, and share your insights. If you're looking ...
- 2 kudos
- 1628 Views
- 3 replies
- 0 kudos
Resolved! Databricks Salesforce Connector - ActivityMetric error
Hi all,I'm experiencing an issue starting 13th May 2025 where a previously successful pipeline using the salesforce connector is now failing, complaining that "Object ActivityMetric is not supported by the Salesforce connector", despite no change to ...
- 1628 Views
- 3 replies
- 0 kudos
- 0 kudos
Hello @ble! This was a known issue that the engineering team had been investigating. Could you kindly confirm whether you’re still experiencing the issue or if it has been resolved?
- 0 kudos
- 1927 Views
- 1 replies
- 1 kudos
Autoloader move file to archive immediately after processing
Hi,We are using autoloader with spark streaming (Databricks: file detection mode) and Want to move files to archive folder from source immediately after processing file. But I cannot reduce retention window beyond 7 days. Code:.option("cloudFiles.cle...
- 1927 Views
- 1 replies
- 1 kudos
- 1 kudos
cloudFiles.cleanSource.retentionDurationType: Interval StringAmount of time to wait before processed files become candidates for archival with cleanSource. Must be greater than 7 days for DELETE. No minimum restriction for MOVE.Available in Databrick...
- 1 kudos
- 3994 Views
- 5 replies
- 0 kudos
Error Unable to Register Model to Unity Catalog
I am following the examples outlined here: https://learn.microsoft.com/en-us/azure/databricks/machine-learning/manage-model-lifecycle/ to register a model to unity catalog.I keep getting this error : BlockingIOError: [Errno 11] Resource temporarily ...
- 3994 Views
- 5 replies
- 0 kudos
- 0 kudos
Thank you very much for your response and explanation.Unfortunately, this takes me back to the original error. I have tried changing the set_tracking_uri to my workspace url and I get the same error.Any ideas what this could be?
- 0 kudos
- 1018 Views
- 1 replies
- 0 kudos
Downloading the query result through rest API
Hi all i have a specific requirements to download the query result. i have created a table on data bricks using SQL warehouse. I have to fetch the query from a custom UI using data API token. Now I am able to fetch the query, but the problem is what ...
- 1018 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @Datagyan ,If your query result is larger than 25MB, Databricks automatically uses disposition=EXTERNAL_LINKS, which returns the result in multiple chunked files(external links).Currently, there's no option to get a single file directly from the ...
- 0 kudos
- 1334 Views
- 1 replies
- 0 kudos
Service Principal cannot access its own workspace folder
We are using Asset bundles with databricks runtime 14.3LTS. During DAB deployment, the wheel is built and stored in the folder of the service principal running the deployment via GH workflow. The full path is/Workspace/Users/SERVICE-PRINCIPAL-ID/.bun...
- 1334 Views
- 1 replies
- 0 kudos
- 0 kudos
You're encountering a common issue when using service principals and job clusters with workspace-scoped paths. This typically happens due topermission mismatches or cluster identity issues. Here’s a breakdown of the root cause and a recommended solut...
- 0 kudos
- 3861 Views
- 1 replies
- 2 kudos
Resolved! Access Mode: Dedicated (assigned to a group) VS Standard
Dedicated Access mode on Azure Databricks clusters provides the option to give access to a GROUP.Trying to understand the use casewhen compared to Standard (formerly: Shared)?When compared to Dedicated (access given to single user)?Ignoring - Languag...
- 3861 Views
- 1 replies
- 2 kudos
- 2 kudos
Dedicated Access mode on Azure Databricks clusters is an upgraded feature that extends the capabilities of single-user access mode. This mode allows a compute resource to be assigned either to a single user or to a group. It offers secure sharing amo...
- 2 kudos
- 1718 Views
- 1 replies
- 0 kudos
Spark custom data sources - SQS streaming reader [DLT]
Hey,I’m working on pulling data from AWS SQS into Databricks using Spark custom data sources and DLT (see https://docs.databricks.com/aws/en/pyspark/datasources). I started with a batch reader/writer based on this example: https://medium.com/@zcking/...
- 1718 Views
- 1 replies
- 0 kudos
- 0 kudos
For your consideration: To address the challenge of passing message handles from executors back to the driver within the DataSourceStreamReader, consider the following approaches: Challenges in Spark Architecture 1. Executor Memory Isolation: Execut...
- 0 kudos
- 3239 Views
- 2 replies
- 1 kudos
Databricks Job Cluster became unreachable
We have production streaming jobs running on Job Clusters. We face cluster related errors now and then, one such example is below error. Run failed with error message Cluster became unreachable during run Cause: Got invalid response: 404 /ERR_NGROK_3...
- 3239 Views
- 2 replies
- 1 kudos
- 1 kudos
Here are some things considerations: The errors experienced in your production streaming jobs—ERR_NGROK_3200 and Spark driver failed to start within 900 seconds—stem from distinct causes related to connectivity, underlying system constraints, and d...
- 1 kudos
- 5069 Views
- 3 replies
- 2 kudos
Connect databricks delta table to Apache Kyln using JDBC
I am using Apache Kylin for Data Analytics and Databricks for data modelling and filtering. I have my final data in gold tables and I would like to integrate this data with Apache Kylin using JDBC where the gold table will be the Data Source. I would...
- 5069 Views
- 3 replies
- 2 kudos
- 2 kudos
@Sidhant07 also using Kylin4 and Sqoop I am getting: "Unsupported transaction isolation level: 2" when running sqoop list-tables.
- 2 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 554 | |
| 349 | |
| 287 |
