- 1113 Views
- 1 replies
- 0 kudos
How to use the same job cluster in diferents job runs inside the one workflow
I created a Workflow with notebooks and some job runs, but I would to use only one job cluster to run every job runs, without creating a new job cluster for each job run. Because I didn't want to increase the execution time with each new job cluster ...
- 1113 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi,If I understand correctly, you are hoping to reduce overall job execution time by reducing the Cloud Service Provider instance provisioning time. Is that correct?If so, you may want to consider: Using a Pool of instances: https://docs.databricks.c...
- 0 kudos
- 4701 Views
- 1 replies
- 0 kudos
Streaming foreachBatch _jdf jvm attribute not supported
I'm trying to perform a merge inside a streaming foreachbatch using the command: microBatchDF._jdf.sparkSession().sql(self.merge_query)Streaming runs fine if I use a Personal cluster while if I use a Shared cluster streaming fails with the following ...
- 4701 Views
- 1 replies
- 0 kudos
- 0 kudos
Can you share what runtime your cluster is using? This error doesn't surprise me, Unity Catalog Shared clusters have many security limitations, but the list is reducing over time. https://docs.databricks.com/en/compute/access-mode-limitations.html#s...
- 0 kudos
- 1549 Views
- 2 replies
- 0 kudos
Databricks dashboard publish
How to publish the dashboard created from a notebook? I don't see publish option within the File anymoreWhen referred to the old video they seem to have an option to publish the dashboard
- 1549 Views
- 2 replies
- 0 kudos
- 0 kudos
Can you share the link to the video you are referring?As per docs, no publish option is currently available, you can do a Present Dashboard to see it.https://docs.databricks.com/en/notebooks/dashboards.html
- 0 kudos
- 1227 Views
- 2 replies
- 0 kudos
Certifications questions
What are the timings and days between which we can get certified? Can we purchase a certification at the location or are we supposed to purchase it beforehand?
- 1227 Views
- 2 replies
- 0 kudos
- 0 kudos
Online exams can be purchased and taken anytime. Is this question related to DAIS?
- 0 kudos
- 1201 Views
- 0 replies
- 0 kudos
How to make a DLT pipeline trigger another pipeline?
For example, I have 2 DLT pipelines, one is used for computing user gender distribution, another is used for compute user location distribution. In the first pipeline, I follow the medallion architecture creating the bronze, silver, gold table one by...
- 1201 Views
- 0 replies
- 0 kudos
- 6994 Views
- 7 replies
- 1 kudos
Databrick IP address
I'm trying to call api in databricks notebook .But while calling api inside databricks notebook it is giving error saying "403" forbidden. I think it seems issue with IP address can any one help me to know which IP of databricks need to be whitelis...
- 6994 Views
- 7 replies
- 1 kudos
- 1 kudos
@jacovangelder can you please let me know where can I get compute IP address
- 1 kudos
- 1760 Views
- 0 replies
- 0 kudos
UC_COMMAND_NOT_SUPPORTED.WITHOUT_RECOMMENDATION in shared access mode
I'm using a shared access cluster and am getting this error while trying to upload to Qdrant. This is the error. Anyway I can make it worked on shared access mode? It works on the personal cluster.[UC_COMMAND_NOT_SUPPORTED.WITHOUT_RECOMMENDATION] The...
- 1760 Views
- 0 replies
- 0 kudos
- 6020 Views
- 6 replies
- 3 kudos
Resolved! Why is My MIN MAX Query Still Slow on a 29TB Delta Table After Liquid Clustering and Optimization?
Hello,I have a large Delta table with a size of 29TB. I implemented Liquid Clustering on this table, but running a simple MIN MAX query on the set cluster column is still extremely slow. I have already optimized the table. Am I missing something in m...
- 6020 Views
- 6 replies
- 3 kudos
- 3 kudos
Hithis operation should take seconds because it use the precomputed statistics for the table. Then few elements to verify:if the data type is datetime or integer should work, if it is string data type then it needs to read all data.verify the column ...
- 3 kudos
- 6522 Views
- 1 replies
- 0 kudos
Change Default Parallelism ?
HII attempted to parallelize my Spark read process by setting the default parallelism using spark.conf.set("spark.default.parallelism", "X"). However, despite setting this configuration, when I checked sc.defaultParallelism in my notebook, it display...
- 6522 Views
- 1 replies
- 0 kudos
- 0 kudos
sc.defaultParallelism is based on the number of worker cores in the cluster. It can't be overridden. The reason you are seeing 200 tasks is because of spark.sql.shuffle.partitions (whose default value is 200). This determines the number of shuffle pa...
- 0 kudos
- 8091 Views
- 8 replies
- 0 kudos
Resolved! Pandas_UDF not working on shared access mode but works on personal cluster
The "dense_vector" column does not output on show(). Instead I get the error below. Any idea why it doesn't work on the shared access mode? Any alternatives? from fastembed import TextEmbedding, SparseTextEmbedding from pyspark.sql.pandas.functions ...
- 8091 Views
- 8 replies
- 0 kudos
- 0 kudos
For some reason a moderator is removing my pip freeze? no idea why. Maybe too long/spammy for a comment.Anyway, I am using DBR 14.3 LTS with Shared Access Mode. I haven't installed any other version apart from fastembed==0.3.1. Included a screenshot ...
- 0 kudos
- 613 Views
- 0 replies
- 0 kudos
How to capture cuurent_user in custom audit logs in job
Hi,I want to build custom audit log in my job.But when I do so, I'm able to capture everything except the current_user() who is running the job, How can I capture it in my custom audit log in my job itself.
- 613 Views
- 0 replies
- 0 kudos
- 1341 Views
- 1 replies
- 0 kudos
Cost comparison of DLT to custom pipeline
Hello, our company currently has a number of custom pipelines written in python for ETL, and I want to do an evaluation of DLT to see if that will make things more efficient. A problem is that there is a restriction on using DLT "because it is too e...
- 1341 Views
- 1 replies
- 0 kudos
- 0 kudos
DLT is expensive in my opinion. I tried to run a simple notebook that just reads a parquet file into a dataframe and write it out to a cloud storage and I got an error that i hit my CPU instance limit for my azure subscription. I just gave up after t...
- 0 kudos
- 6271 Views
- 8 replies
- 1 kudos
DLT pipeline error key not found: user
When I try to create a DLT pipeline from a foreign catalog (BigQuery), I get this error: java.util.NoSuchElementException: key not found: user.I used the same script to copy Salesforce data and that worked completely fine.
- 6271 Views
- 8 replies
- 1 kudos
- 1 kudos
Hi @lucasrocha ,Any luck with this error? I guess it's something with connection to BigQuery, but I didn't find anything.Best regards,Ksenija
- 1 kudos
- 4524 Views
- 3 replies
- 2 kudos
How do I access to DLT advanced configuration from python notebook?
Hi Team, Im trying to get DLT Advanced Configuration value from the python dlt notebook. For example, I set "something": "some path" in Advanced configuration in DLT and I want to get the value from my dlt notebook. I tried "dbutils.widgets.get("some...
- 4524 Views
- 3 replies
- 2 kudos
- 2 kudos
here you can find the documentation on how to use parameters in dlt (sql and python): https://docs.databricks.com/en/delta-live-tables/settings.html#parameterize-dataset-declarations-in-python-or-sql
- 2 kudos
- 3261 Views
- 5 replies
- 0 kudos
validate bundle does not work on windows 10 PRO x64
I use the databricks clidatabricks_cli_0.221.1_windows_amd64-signedand when I rundatabricks bundle validatein my project I get"Error: no shell found" In git-bash it works but I have other problems there.
- 3261 Views
- 5 replies
- 0 kudos
- 0 kudos
That's strange. Just checking, are you running it from the right folder? (location of databricks.yml file)?
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 791 | |
| 553 | |
| 349 | |
| 287 |
