- 4086 Views
- 2 replies
- 1 kudos
Using Init scripts using DBX
I specify init scripts in my deployment.conf, as below: basic-static-cluster: &basic-static-cluster new_cluster: spark_version: "13.0.x-scala2.12" num_workers: 1 node_type_id: "n2-highmem-2" init_scripts: - worksp...
- 4086 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Naga Vaibhav Elluru We haven't heard from you since the last response from @Debayan Mukherjee , and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be he...
- 1 kudos
- 8164 Views
- 4 replies
- 1 kudos
Resolved! Disable iFraming prevention not working
For my own workspace disable iFraming prevention stopped working on 9th May 2023. Till Friday it was working fine and I have already disabled the iFraming prevention.Is there something not right with the databricks platform?
- 8164 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @Ankit Goyal Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so w...
- 1 kudos
- 6964 Views
- 1 replies
- 0 kudos
Resolved! Job cluster failed to start with custom docker image
Using Databricks Container Services, we have created two custom docker image: one based on nvidia/cuda:11.8.0-runtime-ubuntu22.04 and another based on databricksruntime/standard:12.2-LTS. In either case, we got this error with no specific diagnostics...
- 6964 Views
- 1 replies
- 0 kudos
- 0 kudos
I managed to get databricksruntime/standard:12.2-LTS to run in Databricks. However, for the CUDA image (nvidia/cuda:11.8.0-runtime-ubuntu22.04), I have only managed to get it to run with Databricks runtime 10.4 LTS. Does anyone know if Databricks run...
- 0 kudos
- 4517 Views
- 1 replies
- 0 kudos
Batch Doesn't Exist Failure
I have a job that's been working perfectly fine since I deployed it earlier this month. Last night, however, one of the tasks within the job started failing with the following error:java.lang.IllegalStateException: batch 4 doesn't exist at org.apac...
- 4517 Views
- 1 replies
- 0 kudos
- 0 kudos
I tried FSCK REPAIR just on the chance that it would work and it had no effect.
- 0 kudos
- 1382 Views
- 0 replies
- 0 kudos
Databricks MountPoints Need to be refreshed when created by an ADF Job cluster?
In my particular use case, the creation of the mount is initiated through a notebook activity in Azure Data Factory (ADF). This activity utilizes a job cluster for the current execution. However, it has come to my attention that the mounts generated ...
- 1382 Views
- 0 replies
- 0 kudos
- 2830 Views
- 2 replies
- 3 kudos
Resolved! Grant Databricks Access
In the above printscreen of Grant Databricks Access, we see we need to give the rights to a certain Bucket at the highest level. Why is this so? Are we able to limit the rights to only certain directories in a bucket, when we need Databricks to have ...
- 2830 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @THIAM HUAT TAN Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...
- 3 kudos
- 8509 Views
- 2 replies
- 1 kudos
Resolved! DLT setup taking longer than actually building the tables
Hi all!We are using DLT for our ETL jobs, and we're noticing the setup steps (Initializing, Resetting tables. Setting up tables, Rendering graph) are taking much longer than actually ETL'ing the data into our tables. We have about 110 tables combined...
- 8509 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @daan duppen Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...
- 1 kudos
- 2090 Views
- 0 replies
- 0 kudos
How to configure Spark to adjust the number of output partitions after a join or groupBy?
I know you can set "spark.sql.shuffle.partitions" and "spark.sql.adaptive.advisoryPartitionSizeInBytes". The former will not work with adaptive query execution, and the latter only works for the first shuffle for some reason, after which it just uses...
- 2090 Views
- 0 replies
- 0 kudos
- 5627 Views
- 0 replies
- 0 kudos
Identity Column Issues
Hello.I'm experiencing what I'm believe are pretty severe (current) shortcomings regarding Identity columns in Databricks.I'm defining a SQL table using spark SQL - the table creates as exepcted; I've tried using both column definitions for this iden...
- 5627 Views
- 0 replies
- 0 kudos
- 4165 Views
- 3 replies
- 2 kudos
Resolved! How to filter a dashboard by the current user email?
Hello,I would like to know if it is possible to filter a dashboard by the current user email?For example, I have a table result of a group of people with the following columns: user_id, user_email, date, productivity. So with this table I create som...
- 4165 Views
- 3 replies
- 2 kudos
- 2 kudos
Hey guys, After some research on the documentation, I found out that if a filter the query using the current_user() function, I will get the result that I was looking for.If anyone need look at this:https://docs.databricks.com/sql/language-manual/fun...
- 2 kudos
- 1282 Views
- 0 replies
- 0 kudos
How to receive data from azure event hub in parquet ?
import asyncioimport osfrom azure.eventhub.aio import EventHubConsumerClientCONNECTION_STR = "Connection_string"EVENTHUB_NAME = "event_hub"async def on_event(partition_context, event): # Put your code here. # If the operation is i/o intensive, ...
- 1282 Views
- 0 replies
- 0 kudos
- 7230 Views
- 0 replies
- 0 kudos
Ingesting data from oracle to databricks through IICS
While ingesting the data from oracle to databricks through IICS, target table were created however data is not getting inserted. Below is the error. Could someone please help meException occurred when initializing data session. Root cause: java.lang....
- 7230 Views
- 0 replies
- 0 kudos
- 2448 Views
- 3 replies
- 0 kudos
Improve iteration time on implementing jobs
Hey there, I am using dbx to create Databricks tasks and deploy the job. I find it not ideal since the iteration circles are sometimes a bit long when I have to wait for a job with several tasks to complete and see where it failed. I am already tryin...
- 2448 Views
- 3 replies
- 0 kudos
- 0 kudos
Hello, thanks for the answer. Unfortunately, this did not help me, since it is general best practice. @Debayan Mukherjee
- 0 kudos
- 7889 Views
- 0 replies
- 0 kudos
What is the performance impact of changing dataSkippingNumIndexedCols from 32 to 16?
I already improved a lot the performances of our ETL (x20 !) but I still want to know where I can improve performances. I seems that tables stats and column indexing slow down a bit writings so I want to decrease dataSkippingNumIndexedCols to match t...
- 7889 Views
- 0 replies
- 0 kudos
- 29507 Views
- 4 replies
- 1 kudos
Resolved! How to add pre-commit hook to the Git Client on Databricks Cluster?
I'd like to add a Git pre-commit hook to the Databricks Cluster.This pre-commit hook should be executed when pushing to GitHub.Why would I need a pre-commit hook on a Databricks Cluster?My goal is to run blackbricks and format all notebooks automatic...
- 29507 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @Dejan Hrubenja Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...
- 1 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 484 | |
| 349 | |
| 287 |
