- 2283 Views
- 3 replies
- 0 kudos
Azure Databricks DBFS Root, Storage Account Networking
For an Azure Databricks with vnet injection, we would like to change the networking on the default managed Azure Databricks storage account (dbstorage) from Enabled from all networks to Enabled from selected virtual networks and IP addresses.Can this...
- 2283 Views
- 3 replies
- 0 kudos
- 0 kudos
@Sander Sintjorissen​ usually root storage bucket has below directories present in article https://learn.microsoft.com/en-us/azure/databricks/dbfs/root-locationsto store logs related to auditing you can create another storage and add that. hope this ...
- 0 kudos
- 765 Views
- 1 replies
- 0 kudos
Request for unlock workspace
please unlock my workspace that am easily to login our workspace am waiting a few days ago
- 765 Views
- 1 replies
- 0 kudos
- 0 kudos
Adding @Vidula Khanna​ and @Kaniz Fatma​ for visibility to help you with your request
- 0 kudos
- 1425 Views
- 1 replies
- 2 kudos
Azure Blob Storage sas-keys expired for Apache Spark Tutorial
"Apache Spark programming with databricks" tutorial uses Blob storage parquet files on Azure. To access those files a sas key is used in the configuration files. Those keys were generated 5 years ago, however they expired in the begining of this mont...
- 1425 Views
- 1 replies
- 2 kudos
- 2 kudos
Adding @Vidula Khanna​ and @Kaniz Fatma​ for visibility to help with your request
- 2 kudos
- 2641 Views
- 3 replies
- 0 kudos
Notebook connectivity issue with aws s3 bucket using mounting
When connecting to aws s3 bucket using dbfs, application throws error like org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 7864387.0 failed 4 times, most recent failure: Lost task 0.3 in stage 7864387.0 (TID 1709732...
- 2641 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Amrendra Kumar​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us s...
- 0 kudos
- 16502 Views
- 8 replies
- 19 kudos
Resolved! Delta Live Tables failed to launch pipeline cluster
I'm trying to run through the Delta Live Tables quickstart example on Azure Databricks. When trying to start the pipeline I get the following error:Failed to launch pipeline cluster 0408-131049-n3g9vr4r: The operation could not be performed on your a...
- 16502 Views
- 8 replies
- 19 kudos
- 19 kudos
This communication really helped me. I am now successfully able to execute DLT pipeline. Thanks to all contributor.
- 19 kudos
- 1754 Views
- 1 replies
- 2 kudos
Your administrator has forbidden Scala UDFs from being run on this cluster. How to enable access to Scala UDF on Azure Databricks cluster ?
Hi All,When i try to run a scala UDF in Azuredatabricks 10.1 (includes Apache Spark 3.2.0, Scala 2.12) cluster i was able to run the udf. However when i tried to run the same notebook in 10.4 LTS (includes Apache Spark 3.2.1, Scala 2.12) cluster i ha...
- 1754 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi, Are you trying this with High concurrency clusters? Also, please tag @Debayan Mukherjee​ with your next response so that I will get notified.
- 2 kudos
- 4192 Views
- 3 replies
- 0 kudos
disable downloading files for specific group of users ?
I see we can disable/enable download button for entire workspace using download button for notebook results.is there a way to disable/enable this just for specific group of users ?
- 4192 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi Vidula/Suteja,Sorry no, I could not find what you mentioned. Can you please provide some screenshots ?I only see Admin settings when I click on User icon in the top right corner of the Databricks workspace Under Admin Settings, I see below for Acc...
- 0 kudos
- 14338 Views
- 9 replies
- 1 kudos
Pushdown of datetime filter to date partition.
Hi Everybody,I have 20 years of data, 600m rows.I have partitioned them on year and month to generated a files size which seems reasonable.(128Mb)All data is queried using timestamp, as all queries needs to filter on the exact hours.So my requirement...
- 14338 Views
- 9 replies
- 1 kudos
- 1 kudos
Hi Guys, thanks for your advices. I found a solution. We upgrade the Databricks Runtime to 12.2 and now the pushdown of the partitionfilter works. The documentation said that 10.4 would be adequate, but obviously it wasn't enough.
- 1 kudos
- 6268 Views
- 1 replies
- 2 kudos
New text editor "Monaco" does not find SQL-tables or aliases.
I am having troubles with the new text editor "Monaco", it does not find my SQL-tables as soon as they are nested in SQL-commands and marks them with red underscoring which is really annoying. Even though the code works perfectly...It also have troub...
- 6268 Views
- 1 replies
- 2 kudos
- 2 kudos
I am having troubles with the new text editor "Monaco", it does not find my SQL-tables as soon as they are nested in SQL-commands and marks them with red underscoring which is really annoying. Even though the code works perfectly...It also have troub...
- 2 kudos
- 1214 Views
- 1 replies
- 0 kudos
Garbage Collection on AutoLoader
Once a week, I get very long run times with AutoLoader. The spark job says it is done, but garbage collection keeps rising on the driver. I assume this is because of the backfill interval that I am using with FileNotification Type. I have this set to...
- 1214 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @nolanlavender008​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us...
- 0 kudos
- 12980 Views
- 8 replies
- 6 kudos
Databricks notebook ipywidgets not working as expected ( button click issue)
I am working on Azure databricks(IDE). I wanted to create a button which takes a text value as input and on the click of a button a function needed to be run which prints the value entered.For that I created this code:from IPython.display import disp...
- 12980 Views
- 8 replies
- 6 kudos
- 6 kudos
Hi @Shubham Ringne​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us s...
- 6 kudos
- 3985 Views
- 2 replies
- 0 kudos
How disable disk cache in SQL Warehouse (Azure Databricks)?
Hi! I want to disable disk cache for SQL Warehouse in Azure Databricks, but it seems that is not possible. Is it correct?You can't use this configuration for SQL Warehouse (https://learn.microsoft.com/en-US/azure/databricks/optimizations/disk-cache#-...
- 3985 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @jlgr jlgr​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we ...
- 0 kudos
- 7331 Views
- 4 replies
- 0 kudos
I am struggling to optimize my Spark Application Code. Is there someone who can assist me in optimizing it? I am using Spark over Hadoop Yarn.
I will elaborate my problem. I am using a 6-node Spark cluster over Hadoop Yarn out of which one node acts as the master and the other 5 are acting as worker nodes. I am running my Spark application over the cluster. After completion, when I check th...
- 7331 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @T. V. K. Hanuman​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us...
- 0 kudos
- 1409 Views
- 2 replies
- 3 kudos
SQL endpoint increased response times
We have observed that an SQL endpoint has increased response times after a long time being idle. This endpoint is always running and does not terminate. Are there any checks/overheads due to being idle that could impact performance?
- 1409 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @EDDatabricks EDDatabricks​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, pleas...
- 3 kudos
- 1905 Views
- 2 replies
- 1 kudos
Pyspark 3.3.0 exceptAll working on 11.3 LTS but not locally
Hello,Currently I'm in process of upgrading the DBR version in my jobs to version 11.3 LTS. After upgrading pyspark version to 3.3.0 on my local machine I found that exceptAll function is broken (it looks like others have similar problem). It throws ...
- 1905 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Jacek Dembowiak​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us ...
- 1 kudos