cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

KarenBT
by New Contributor III
  • 7871 Views
  • 15 replies
  • 4 kudos

Welcome 2023 Virtual hackathon participants, we're happy to have you! ✋  Please use this space to ask questions, we'll have some folks from Da...

Welcome 2023 Virtual hackathon participants, we're happy to have you! Please use this space to ask questions, we'll have some folks from Databricks and the community join to help out. We're really excited to see what you work on and if you have any ...

  • 7871 Views
  • 15 replies
  • 4 kudos
Latest Reply
sanggusti
New Contributor II
  • 4 kudos

Hi, I also have another question. Do we get any Databricks platform access for the period of hackathon? My company didn't use one and the trial is only 14 days. I'm pretty aware of the capability and since the hackathon are held by Databricks I think...

  • 4 kudos
14 More Replies
Michelle_-_Devp
by New Contributor III
  • 1394 Views
  • 1 replies
  • 1 kudos

Resolved! How is brainstorming going?

Wondering if anyone is willing to share their project ideas here. It would be great to know how things are going and if anyone has a good open-source dataset they are willing to share.

  • 1394 Views
  • 1 replies
  • 1 kudos
Latest Reply
bayang
New Contributor III
  • 1 kudos

Good, read their docs to get a lot of info to sharpen this hackathon

  • 1 kudos
IndihomeTV
by New Contributor
  • 1410 Views
  • 1 replies
  • 0 kudos

Databricks to redash

We have an issued security in redash, if we used databrick as a connector to redash, Can you support us?https://www.databricks.com/blog/2020/06/24/welcoming-redash-to-databricks.html

  • 1410 Views
  • 1 replies
  • 0 kudos
Latest Reply
arpit
Databricks Employee
  • 0 kudos

Hi @Probis Useetv​ Thank you for reaching out to us.Would you please elaborate your use case about the "issued security in redash" ?

  • 0 kudos
Ismail1
by New Contributor III
  • 3772 Views
  • 3 replies
  • 3 kudos

Resolved! Generating an Account console PAT token

I can't seem to find any documentation on generating an account console PAT token, Can anyone link me to it or guide me?

  • 3772 Views
  • 3 replies
  • 3 kudos
Latest Reply
fkseki
New Contributor III
  • 3 kudos

You can't create a Personal Access Token on account level to use REST APIs. If you want to use SCIM on account level, on the account console settings you'll find the user provisioning tab. In there you can generate de SCIM token. If you want to acces...

  • 3 kudos
2 More Replies
pantelis_mare
by Contributor III
  • 42604 Views
  • 30 replies
  • 15 kudos

Resolved! Repos configuration for Azure Service Principal

Hello community!I would like to update a repo from within my Azure DevOps release pipeline. In the pipeline I generate a token using a AAD Service Principal as recommended, and I setup the databricks api using that token.When I pass the databricks re...

  • 42604 Views
  • 30 replies
  • 15 kudos
Latest Reply
xiangzhu
Contributor III
  • 15 kudos

traditional PAT may have long lifespn, but the new SP feature uses an AAD token which should have a much shorter lifespqn, maybe around one hour, this could be a limiting factor.However, I haven't tested this yet, so these are merely hypotheses.​Neve...

  • 15 kudos
29 More Replies
Phani1
by Valued Contributor II
  • 3554 Views
  • 2 replies
  • 1 kudos

Integration Dolly with Databricks

Hi Databricks Team,Could you please share any links /docs/Sample notebooks to integrate Dolly with Databricks, our aim is to generate SQL queries based on the free text and execute it via databricks cluster/SQL warehouse.

  • 3554 Views
  • 2 replies
  • 1 kudos
Latest Reply
sean_owen
Databricks Employee
  • 1 kudos

https://www.dbdemos.ai/demo.html?demoName=llm-dolly-chatbot is a good demonstration of Dolly (or really any LLM) for question answering. LLMs like this are not for SQL generation, but other LLMs are, like starcoderbase

  • 1 kudos
1 More Replies
sanjay
by Valued Contributor II
  • 2992 Views
  • 2 replies
  • 1 kudos

Resolved! How can I prioritize message in autoloader

Hi,I am using autoloader, it picks data from AWS S3 and stores in delta table. In case there are large number of messages, I like to process messages by priority. Is it possible to prioritize messages in autoloader.Regards,Sanjay

  • 2992 Views
  • 2 replies
  • 1 kudos
Latest Reply
sanjay
Valued Contributor II
  • 1 kudos

Thank you Sandeep. Other option is I can keep messages in 2 different folders in S3. Can autoloader read message from multiple folders

  • 1 kudos
1 More Replies
pauloquantile
by New Contributor III
  • 6170 Views
  • 8 replies
  • 0 kudos

Resolved! Disable scheduling of notebooks

Hi,We are wondering if it is possible to disable the possibility to disable scheduling of a notebook. A client wants to allow many analysts access to databricks, but a concern is the possibility of setting schedules (the fastest is every minute!). Is...

  • 6170 Views
  • 8 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Paulo Rijnberg​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedba...

  • 0 kudos
7 More Replies
deep_thought
by Contributor
  • 27820 Views
  • 16 replies
  • 9 kudos

Resolved! Schedule job to run sequentially after another job

Is there a way to schedule a job to run after some other job is complete?E.g. Schedule Job A, then upon it's completion run Job B.

  • 27820 Views
  • 16 replies
  • 9 kudos
Latest Reply
claytonseverson
Databricks Employee
  • 9 kudos

Here is the User Guide for Jobs-as-Tasks - https://docs.google.com/document/d/1OJsc-g7IwAJjYooCp7T01Rxyt_xFkMPjmAAGdDGPkY4/edit#heading=h.oudvb5fyfd0n

  • 9 kudos
15 More Replies
vladcrisan
by New Contributor II
  • 5663 Views
  • 5 replies
  • 1 kudos

Can Spark History server be created in Databricks?

We have a Spark pipeline producing more than 3k Spark jobs. After the pipeline finishes and the cluster shuts down, only a subset (<1k) of these can be recovered from the Spark UI.We would like to have access to the full Spark UI after the pipeline t...

  • 5663 Views
  • 5 replies
  • 1 kudos
Latest Reply
Sandeep
Contributor III
  • 1 kudos

@Vlad Crisan​ , you can use the Databricks clusters to replay the events. Please follow this kb: https://kb.databricks.com/clusters/replay-cluster-spark-eventsNote: Please spin up a cluster with version 10.4 LTS.

  • 1 kudos
4 More Replies
yunna_wei
by Databricks Employee
  • 1483 Views
  • 0 replies
  • 3 kudos

In any Spark application, Spark driver plays a critical role and performs the following functions: 1. Initiating a Spark Session 2. Communicating with...

In any Spark application, Spark driver plays a critical role and performs the following functions:1. Initiating a Spark Session2. Communicating with the cluster manager to request resources (CPU, memory, etc) from the cluster manager for Spark's exec...

  • 1483 Views
  • 0 replies
  • 3 kudos
nav
by New Contributor II
  • 6339 Views
  • 8 replies
  • 0 kudos

R packages not getting installed on cluster when creating cluster from dockerfile

I'm trying to use dockerfile to create a cluster which has Robyn (https://facebookexperimental.github.io/Robyn/) and other R libraries installed. But it is failing to install the R libraries to the cluster. When I run the container in interactive mod...

  • 6339 Views
  • 8 replies
  • 0 kudos
Latest Reply
workingtogetdbw
New Contributor II
  • 0 kudos

What there has been no answer here!  @Debayan Mukherjee​ @Vartika Nain​ So I am running into this same problem as the idea of having to wait 45 minutes for libraries to install is absolutely wild as well as I have done everything outside of working w...

  • 0 kudos
7 More Replies
Dave_Nithio
by Contributor II
  • 9652 Views
  • 1 replies
  • 3 kudos

Delta Live Table Schema Error

I'm using Delta Live Tables to load a set of csv files in a directory. I am pre-defining the schema to avoid issues with schema inference. This works with autoloader on a regular delta table, but is failing for Delta Live Tables. Below is an example ...

  • 9652 Views
  • 1 replies
  • 3 kudos
Latest Reply
shagun
New Contributor III
  • 3 kudos

i was facing similar issue in loading json files through autoloader for delta live tables.Was able to fix with this option .option("cloudFiles.inferColumnTypes", "True")From the docs "For formats that don’t encode data types (JSON and CSV), Auto Load...

  • 3 kudos
Kannan1206
by New Contributor II
  • 2440 Views
  • 4 replies
  • 0 kudos

Databricks Certification Exam Got Suspended. Need help in resolving the issue

Hi Team,I have taken online exam for Databricks Certified Associate Developer for Apache Spark 3.0 - Python on 21-May-2023 6:30 , In between the exam my session got suspended. by proctor eventhough I was in my seat and looking at camera . Again I cou...

  • 2440 Views
  • 4 replies
  • 0 kudos
Latest Reply
Kannan1206
New Contributor II
  • 0 kudos

Hi @Vidula Khanna​ , I got the relevant details from the team , was able to complete the certification as well . Thanks for help .

  • 0 kudos
3 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels