- 4639 Views
- 3 replies
- 3 kudos
Resolved! Generating an Account console PAT token
I can't seem to find any documentation on generating an account console PAT token, Can anyone link me to it or guide me?
- 4639 Views
- 3 replies
- 3 kudos
- 3 kudos
You can't create a Personal Access Token on account level to use REST APIs. If you want to use SCIM on account level, on the account console settings you'll find the user provisioning tab. In there you can generate de SCIM token. If you want to acces...
- 3 kudos
- 49296 Views
- 30 replies
- 15 kudos
Resolved! Repos configuration for Azure Service Principal
Hello community!I would like to update a repo from within my Azure DevOps release pipeline. In the pipeline I generate a token using a AAD Service Principal as recommended, and I setup the databricks api using that token.When I pass the databricks re...
- 49296 Views
- 30 replies
- 15 kudos
- 15 kudos
traditional PAT may have long lifespn, but the new SP feature uses an AAD token which should have a much shorter lifespqn, maybe around one hour, this could be a limiting factor.However, I haven't tested this yet, so these are merely hypotheses.Neve...
- 15 kudos
- 4271 Views
- 2 replies
- 1 kudos
Integration Dolly with Databricks
Hi Databricks Team,Could you please share any links /docs/Sample notebooks to integrate Dolly with Databricks, our aim is to generate SQL queries based on the free text and execute it via databricks cluster/SQL warehouse.
- 4271 Views
- 2 replies
- 1 kudos
- 1 kudos
https://www.dbdemos.ai/demo.html?demoName=llm-dolly-chatbot is a good demonstration of Dolly (or really any LLM) for question answering. LLMs like this are not for SQL generation, but other LLMs are, like starcoderbase
- 1 kudos
- 3305 Views
- 2 replies
- 1 kudos
Resolved! How can I prioritize message in autoloader
Hi,I am using autoloader, it picks data from AWS S3 and stores in delta table. In case there are large number of messages, I like to process messages by priority. Is it possible to prioritize messages in autoloader.Regards,Sanjay
- 3305 Views
- 2 replies
- 1 kudos
- 1 kudos
Thank you Sandeep. Other option is I can keep messages in 2 different folders in S3. Can autoloader read message from multiple folders
- 1 kudos
- 7049 Views
- 8 replies
- 0 kudos
Resolved! Disable scheduling of notebooks
Hi,We are wondering if it is possible to disable the possibility to disable scheduling of a notebook. A client wants to allow many analysts access to databricks, but a concern is the possibility of setting schedules (the fastest is every minute!). Is...
- 7049 Views
- 8 replies
- 0 kudos
- 0 kudos
Hi @Paulo Rijnberg Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedba...
- 0 kudos
- 31278 Views
- 16 replies
- 9 kudos
Resolved! Schedule job to run sequentially after another job
Is there a way to schedule a job to run after some other job is complete?E.g. Schedule Job A, then upon it's completion run Job B.
- 31278 Views
- 16 replies
- 9 kudos
- 9 kudos
Here is the User Guide for Jobs-as-Tasks - https://docs.google.com/document/d/1OJsc-g7IwAJjYooCp7T01Rxyt_xFkMPjmAAGdDGPkY4/edit#heading=h.oudvb5fyfd0n
- 9 kudos
- 6655 Views
- 5 replies
- 1 kudos
Can Spark History server be created in Databricks?
We have a Spark pipeline producing more than 3k Spark jobs. After the pipeline finishes and the cluster shuts down, only a subset (<1k) of these can be recovered from the Spark UI.We would like to have access to the full Spark UI after the pipeline t...
- 6655 Views
- 5 replies
- 1 kudos
- 1 kudos
@Vlad Crisan , you can use the Databricks clusters to replay the events. Please follow this kb: https://kb.databricks.com/clusters/replay-cluster-spark-eventsNote: Please spin up a cluster with version 10.4 LTS.
- 1 kudos
- 1938 Views
- 0 replies
- 3 kudos
In any Spark application, Spark driver plays a critical role and performs the following functions: 1. Initiating a Spark Session 2. Communicating with...
In any Spark application, Spark driver plays a critical role and performs the following functions:1. Initiating a Spark Session2. Communicating with the cluster manager to request resources (CPU, memory, etc) from the cluster manager for Spark's exec...
- 1938 Views
- 0 replies
- 3 kudos
- 7375 Views
- 8 replies
- 0 kudos
R packages not getting installed on cluster when creating cluster from dockerfile
I'm trying to use dockerfile to create a cluster which has Robyn (https://facebookexperimental.github.io/Robyn/) and other R libraries installed. But it is failing to install the R libraries to the cluster. When I run the container in interactive mod...
- 7375 Views
- 8 replies
- 0 kudos
- 0 kudos
What there has been no answer here! @Debayan Mukherjee @Vartika Nain So I am running into this same problem as the idea of having to wait 45 minutes for libraries to install is absolutely wild as well as I have done everything outside of working w...
- 0 kudos
- 10596 Views
- 1 replies
- 3 kudos
Delta Live Table Schema Error
I'm using Delta Live Tables to load a set of csv files in a directory. I am pre-defining the schema to avoid issues with schema inference. This works with autoloader on a regular delta table, but is failing for Delta Live Tables. Below is an example ...
- 10596 Views
- 1 replies
- 3 kudos
- 3 kudos
i was facing similar issue in loading json files through autoloader for delta live tables.Was able to fix with this option .option("cloudFiles.inferColumnTypes", "True")From the docs "For formats that don’t encode data types (JSON and CSV), Auto Load...
- 3 kudos
- 2853 Views
- 4 replies
- 0 kudos
Databricks Certification Exam Got Suspended. Need help in resolving the issue
Hi Team,I have taken online exam for Databricks Certified Associate Developer for Apache Spark 3.0 - Python on 21-May-2023 6:30 , In between the exam my session got suspended. by proctor eventhough I was in my seat and looking at camera . Again I cou...
- 2853 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Vidula Khanna , I got the relevant details from the team , was able to complete the certification as well . Thanks for help .
- 0 kudos
- 2707 Views
- 3 replies
- 0 kudos
session suspended , for the databricks exam , how to restart it.
session suspended , please enable launch option
- 2707 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @sindhu goyal Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback...
- 0 kudos
- 4822 Views
- 2 replies
- 0 kudos
Resolved! Input File Path from Autoloader in Delta Live Tables
Hello everyone!I was wondering if there is any way to get the subdirectories in which the file resides while loading while loading using Autoloader with DLT. For example:def customer(): return ( spark.readStream.format('cloudfiles') .option('clou...
- 4822 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Parsa Bahraminejad We haven't heard from you since the last response from @Vigneshraja Palaniraj , and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be...
- 0 kudos
- 3701 Views
- 2 replies
- 2 kudos
merge vs MERGE INTO
from 10.4 LTS version we have low shuffle merge, so merge is more faster. But what about MERGE INTO function that we run in sql notebook of databricks. Is there any performance difference when we use databrciks pyspark ".merge" function vs databricks...
- 3701 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Roshan RC Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers you...
- 2 kudos
- 3438 Views
- 2 replies
- 1 kudos
Search for Databricks Jobs By Name
The Databricks CLI has the ability to list jobs by exact name using “Databricks jobs list —name my_job”. Is there a way to search for jobs using this same method, where I could put a partial name of a job and get all the jobs that match? Ex: “databri...
- 3438 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Eric Keniuk Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers y...
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 554 | |
| 349 | |
| 287 |
