- 6447 Views
- 3 replies
- 0 kudos
Resolved! Command output disappearing (Not sure what's the root cause)
I have a workflow which will run every month and it will create a new notebook containing the outputs from the main notebook. However, after some time, the outputs from the created notebook will disappear. Is there anyway I can retain the outputs?
- 6447 Views
- 3 replies
- 0 kudos
- 0 kudos
@Shaun Ang​ :There are a few possible reasons why the outputs from the created notebook might be disappearing:Notebook permissions: It's possible that the user or service account running the workflow does not have permission to write to the destinati...
- 0 kudos
- 11498 Views
- 2 replies
- 1 kudos
How to apply a UDF to a property in an array of structs
I have a column that contains an array of structs as follows:"column" : [ { "struct_field1": "struct_value", "struct_field2": "struct_value" }, { "struct_field1": "struct_value", "struct_field2": "struct_value" } ]I want to apply a udf to each f...
- 11498 Views
- 2 replies
- 1 kudos
- 9622 Views
- 7 replies
- 0 kudos
Databricks Workflow cost on running in interactive cluster
Databricks Workflow cost on running in interactive cluster
- 9622 Views
- 7 replies
- 0 kudos
- 3962 Views
- 3 replies
- 0 kudos
Azure Databricks DBFS Root, Storage Account Networking
For an Azure Databricks with vnet injection, we would like to change the networking on the default managed Azure Databricks storage account (dbstorage) from Enabled from all networks to Enabled from selected virtual networks and IP addresses.Can this...
- 3962 Views
- 3 replies
- 0 kudos
- 0 kudos
@Sander Sintjorissen​ usually root storage bucket has below directories present in article https://learn.microsoft.com/en-us/azure/databricks/dbfs/root-locationsto store logs related to auditing you can create another storage and add that. hope this ...
- 0 kudos
- 1421 Views
- 1 replies
- 0 kudos
Request for unlock workspace
please unlock my workspace that am easily to login our workspace am waiting a few days ago
- 1421 Views
- 1 replies
- 0 kudos
- 0 kudos
Adding @Vidula Khanna​ and @Kaniz Fatma​ for visibility to help you with your request
- 0 kudos
- 3094 Views
- 1 replies
- 2 kudos
Azure Blob Storage sas-keys expired for Apache Spark Tutorial
"Apache Spark programming with databricks" tutorial uses Blob storage parquet files on Azure. To access those files a sas key is used in the configuration files. Those keys were generated 5 years ago, however they expired in the begining of this mont...
- 3094 Views
- 1 replies
- 2 kudos
- 2 kudos
Adding @Vidula Khanna​ and @Kaniz Fatma​ for visibility to help with your request
- 2 kudos
- 4368 Views
- 3 replies
- 0 kudos
Notebook connectivity issue with aws s3 bucket using mounting
When connecting to aws s3 bucket using dbfs, application throws error like org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 7864387.0 failed 4 times, most recent failure: Lost task 0.3 in stage 7864387.0 (TID 1709732...
- 4368 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Amrendra Kumar​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us s...
- 0 kudos
- 24125 Views
- 8 replies
- 19 kudos
Resolved! Delta Live Tables failed to launch pipeline cluster
I'm trying to run through the Delta Live Tables quickstart example on Azure Databricks. When trying to start the pipeline I get the following error:Failed to launch pipeline cluster 0408-131049-n3g9vr4r: The operation could not be performed on your a...
- 24125 Views
- 8 replies
- 19 kudos
- 19 kudos
This communication really helped me. I am now successfully able to execute DLT pipeline. Thanks to all contributor.
- 19 kudos
- 2673 Views
- 1 replies
- 2 kudos
Your administrator has forbidden Scala UDFs from being run on this cluster. How to enable access to Scala UDF on Azure Databricks cluster ?
Hi All,When i try to run a scala UDF in Azuredatabricks 10.1 (includes Apache Spark 3.2.0, Scala 2.12) cluster i was able to run the udf. However when i tried to run the same notebook in 10.4 LTS (includes Apache Spark 3.2.1, Scala 2.12) cluster i ha...
- 2673 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi, Are you trying this with High concurrency clusters? Also, please tag @Debayan Mukherjee​ with your next response so that I will get notified.
- 2 kudos
- 19969 Views
- 9 replies
- 1 kudos
Pushdown of datetime filter to date partition.
Hi Everybody,I have 20 years of data, 600m rows.I have partitioned them on year and month to generated a files size which seems reasonable.(128Mb)All data is queried using timestamp, as all queries needs to filter on the exact hours.So my requirement...
- 19969 Views
- 9 replies
- 1 kudos
- 1 kudos
Hi Guys, thanks for your advices. I found a solution. We upgrade the Databricks Runtime to 12.2 and now the pushdown of the partitionfilter works. The documentation said that 10.4 would be adequate, but obviously it wasn't enough.
- 1 kudos
- 7272 Views
- 1 replies
- 2 kudos
New text editor "Monaco" does not find SQL-tables or aliases.
I am having troubles with the new text editor "Monaco", it does not find my SQL-tables as soon as they are nested in SQL-commands and marks them with red underscoring which is really annoying. Even though the code works perfectly...It also have troub...
- 7272 Views
- 1 replies
- 2 kudos
- 2092 Views
- 1 replies
- 0 kudos
Garbage Collection on AutoLoader
Once a week, I get very long run times with AutoLoader. The spark job says it is done, but garbage collection keeps rising on the driver. I assume this is because of the backfill interval that I am using with FileNotification Type. I have this set to...
- 2092 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @nolanlavender008​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us...
- 0 kudos
- 18746 Views
- 8 replies
- 6 kudos
Databricks notebook ipywidgets not working as expected ( button click issue)
I am working on Azure databricks(IDE). I wanted to create a button which takes a text value as input and on the click of a button a function needed to be run which prints the value entered.For that I created this code:from IPython.display import disp...
- 18746 Views
- 8 replies
- 6 kudos
- 6 kudos
Hi @Shubham Ringne​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us s...
- 6 kudos
- 5613 Views
- 2 replies
- 0 kudos
How disable disk cache in SQL Warehouse (Azure Databricks)?
Hi! I want to disable disk cache for SQL Warehouse in Azure Databricks, but it seems that is not possible. Is it correct?You can't use this configuration for SQL Warehouse (https://learn.microsoft.com/en-US/azure/databricks/optimizations/disk-cache#-...
- 5613 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @jlgr jlgr​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we ...
- 0 kudos
- 10936 Views
- 4 replies
- 0 kudos
I am struggling to optimize my Spark Application Code. Is there someone who can assist me in optimizing it? I am using Spark over Hadoop Yarn.
I will elaborate my problem. I am using a 6-node Spark cluster over Hadoop Yarn out of which one node acts as the master and the other 5 are acting as worker nodes. I am running my Spark application over the cluster. After completion, when I check th...
- 10936 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @T. V. K. Hanuman​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 566 | |
| 349 | |
| 287 |
