- 1986 Views
- 1 replies
- 0 kudos
Databricks OutOfMemory error on code that previously worked without issue
I have a notebook in Azure Databricks that does some transformations on a bronze tier table and inserts the transformed data into a silver tier table. This notebook is used to do an initial load of the data from our existing system into our new datal...
- 1986 Views
- 1 replies
- 0 kudos
- 0 kudos
Please review your Spark UI from the old job execution versus the new job execution. You might need to check if the data volume has increase and that could be the reason of the OOM
- 0 kudos
- 1978 Views
- 1 replies
- 0 kudos
org.apache.spark.SparkException: Job aborted due to stage failure when writing to Cosmos
I am writing data to cosmos DB using Python & Spark on DatabricksI am getting below error :org.apache.spark.SparkException: Job aborted due to stage failure: Authorized committer (attemptNumber=0, stage=192, partition=105) failed; but task commit suc...
- 1978 Views
- 1 replies
- 0 kudos
- 0 kudos
The configs are for cluster:Worker Type & Driver type : Standard_D16ads_v5RUs for Cosmos : 1.5L
- 0 kudos
- 4285 Views
- 2 replies
- 0 kudos
Unable to access unity catalog volume via /Volumes in notebook
I have set up a volume in unity catalog in the format catalog/schema/volume, and granted all permissions to all users on the catalog, schema and volume.From the notebook I can see the /Volumes directory in the root of the file system but am unable to...
- 4285 Views
- 2 replies
- 0 kudos
- 0 kudos
Thanks for your comments. The problem turned out to be the compute resource not having unity catalog enabled.
- 0 kudos
- 1669 Views
- 1 replies
- 0 kudos
SparkR or sparklyr not showing history
Hi,for some reason Azure Databricks doesn't show History if the data is saved with SparkR (2 in the figure below) or Sparklyr (3), but it does show it with Data Ingestion (0) or with PySpark (1). Is this a known bug or am I doing something wrong? Is ...
- 1669 Views
- 1 replies
- 0 kudos
- 12990 Views
- 2 replies
- 0 kudos
connect timed out error - Connecting to SQL Server from Databricks
I am getting a connect timed out error when attempting to access a sql server. I can successfully ping the server from Databricks. I have used the jdbc connection and the sqlserver included driver and both result in the same error. I have also attemp...
- 12990 Views
- 2 replies
- 0 kudos
- 0 kudos
Can you run the following command in a notebook using the same cluster you are using to connect:%sh nc -vz <hostname> <port> This test will confirm us if we are able to communicate with the SQL server by using the port you are defining to connect. If...
- 0 kudos
- 1039 Views
- 0 replies
- 0 kudos
Some columns are being dropped when moving to pandas data set.
Some columns are being dropped when moving to pandas data set. I see part of the dataset, but it does not show when displaying..
- 1039 Views
- 0 replies
- 0 kudos
- 3932 Views
- 2 replies
- 1 kudos
Machine Type for VACUUM operation
Dear allI have a workflow with 2 tasks : one that does OPTIMIZE, followed by one that does VACUUM. I used a cluster with F32s driver and F64s - 8 workers (auto-scaling enabled). All 8 workers are launched by Databricks as soon as OPTIMIZE starts. As ...
- 3932 Views
- 2 replies
- 1 kudos
- 2569 Views
- 2 replies
- 0 kudos
How to migrate Git repos with DLT configurations
Hi!I want to migrate all my databricks related code from one github repo to another. I knew this wouldn't be straight forward. When I copy my code for one DLT, I get the errororg.apache.spark.sql.catalyst.ExtendedAnalysisException: Table 'vessel_batt...
- 2569 Views
- 2 replies
- 0 kudos
- 0 kudos
Does cloning take considerably less time then recreating the tables?Can I resume append operations to a cloned table?
- 0 kudos
- 1508 Views
- 0 replies
- 0 kudos
Job cluster configuration for 24/7
Hi Team,We intend to activate the job cluster around the clock. We consider the following parameters regarding cost: - Data volumes - Client SLA for job completion- Starting with a small cluster configuration Please advise on any other options we s...
- 1508 Views
- 0 replies
- 0 kudos
- 4444 Views
- 1 replies
- 1 kudos
Masking rules with Delta Sharing
Hi,We tried Delta sharing to PBI which worked fine, But facing issues while trying to apply row, column level filtering or data masking. It fails with error that its not supported.Can anyone please confirm, if delta sharing with masking rules works w...
- 4444 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Anshul_DBX good day! The issue you are encountering is due to a limitation in Delta Sharing. As per the provided information, Delta Sharing does not support row-level security or column masks. This means that you cannot apply row and column level...
- 1 kudos
- 5147 Views
- 1 replies
- 0 kudos
Databricks cli workflow
Is there a way that I can set up and configure a Databricks workflow job and tasks from Databricks cli or api tools by using python? Any help would be appreciated. #databricksworkflow #databricks
- 5147 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello and yes, you can set up and configure a Databricks workflow job and tasks using Databricks CLI or API tools with Python. Here are some resources and steps to guide you: Create and run Databricks Jobs: This document: ( https://docs.databrick...
- 0 kudos
- 4684 Views
- 2 replies
- 1 kudos
Address Validation, Correction and Enrichment with Databricks Spark Engine
Hi all!In our project, we're thinking about "Validation, Correction and Enrichment of Postal Addresses" with Databricks. For sure we'd need some kind of batch processing, because we have millions of addresses in our system.I'm aware of Address Valida...
- 4684 Views
- 2 replies
- 1 kudos
- 1 kudos
Happy to help. Feel free to reach out https://www.linkedin.com/in/saleh-sultan-143ab036?utm_source=share&utm_campaign=share_via&utm_content=profile&utm_medium=android_app
- 1 kudos
- 2734 Views
- 1 replies
- 0 kudos
udf in databricks
Hi Team,Is there a particular reason why we should avoid using UDF and instead convert to DataFrame code?Are there any restrictions or limitations (in terms of performance or governance) when using UDFs in Databricks? Regards,Janga
- 2734 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello some of the things you need to take in consideration is that:UDFs might introduce significant processing bottlenecks into code execution. Databricks uses a number of different optimizers automatically for code written with included Apache Spark...
- 0 kudos
- 45387 Views
- 2 replies
- 8 kudos
What's the difference between a Global view and a Temp view?
The difference between Global and Temp is how the lifetime of the view is tied to the application:http://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.DataFrame.createOrReplaceTempView.html?highlight=createorreplacetempview#pyspar...
- 45387 Views
- 2 replies
- 8 kudos
- 8 kudos
Correct A Temp View is scoped to the SparkSession and dropped when that session closes. Each notebook runs in its own SparkSession. The Global Temp View is scoped to the cluster and dropped when the cluster re-starts or you drop it. ---------------...
- 8 kudos
- 18208 Views
- 5 replies
- 4 kudos
Resolved! Pass date value as parameter in Databricks SQL notebook
I want to pass yesterday date (In the example 20230115*.csv) in the csv file. Don't know how to create parameter and use it here.CREATE OR REPLACE TEMPORARY VIEW abc_delivery_logUSING CSVOPTIONS ( header="true", delimiter=",", inferSchema="true", pat...
- 18208 Views
- 5 replies
- 4 kudos
- 4 kudos
@Retired_mod @sp1 @Chaitanya_Raju @daniel_sahal Hi Everyone,I need the same scenario on SQL code, because my DBR cluster not allowed me to run python codeError: Unsupported cell during execution. SQL warehouses only support executing SQL cells.I appr...
- 4 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
