cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Hubert-Dudek
by Esteemed Contributor III
  • 1503 Views
  • 1 replies
  • 7 kudos

Databricks recently added SQL alerts feature that enables users to create notifications based on various conditions and trigger them within their job ...

Databricks recently added SQL alerts feature that enables users to create notifications based on various conditions and trigger them within their job workflows.SQL alerts inform users about potential issues and easily ensure critical data availabilit...

ezgif-3-324d970c83
  • 1503 Views
  • 1 replies
  • 7 kudos
Latest Reply
Anonymous
Not applicable
  • 7 kudos

Thanks for sharing this @Hubert Dudek​ .

  • 7 kudos
sintsan
by New Contributor II
  • 1277 Views
  • 1 replies
  • 1 kudos

Resolved! spark.sparkContext.setCheckpointDir - External Azure Storage

Is it possible to direct spark.sparkContext.setCheckpointDir to an external Azure Storage Container location (instead of DBFS) & if so how, there's very little documentation on that.

  • 1277 Views
  • 1 replies
  • 1 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 1 kudos

yes,the directory must be an HDFS path if running on a cluster.All you need to do is provide the correct path.

  • 1 kudos
Praveen
by New Contributor II
  • 8992 Views
  • 8 replies
  • 1 kudos

Resolved! Pass Typesafe config file to the Spark Submit Job

Hello everyone ! I am trying to pass a Typesafe config file to the spark submit task and print the details in the config file. Code: import org.slf4j.{Logger, LoggerFactory} import com.typesafe.config.{Config, ConfigFactory} import org.apache.spa...

  • 8992 Views
  • 8 replies
  • 1 kudos
Latest Reply
source2sea
Contributor
  • 1 kudos

I've experenced similar issues; please help to answer how to get this working;I've tried using below to be either /dbfs/mnt/blah path or dbfs:/mnt/blah pathin either spark_submit_task or spark_jar_task (via cluster spark_conf for java optinos); no su...

  • 1 kudos
7 More Replies
Anonymous
by Not applicable
  • 924 Views
  • 0 replies
  • 1 kudos

Hi Everyone, As we continue to see the rise of AI, particularly with technologies such as ChatGPT, it's important to consider how this will impact...

Hi Everyone,As we continue to see the rise of AI, particularly with technologies such as ChatGPT, it's important to consider how this will impact the future of our workplaces and long-term career goals. With that in mind, I would love to hear your th...

  • 924 Views
  • 0 replies
  • 1 kudos
Vinay123
by New Contributor III
  • 5098 Views
  • 2 replies
  • 1 kudos

Unity catlog replication or Disaster recovery implementation

I am working on Disaster recovery implementation on databricks on aws.I am not able to find how to implement with unity catalog. I am planning to create two workspaces in two different regions, one would be primary workspace which will be active and ...

  • 5098 Views
  • 2 replies
  • 1 kudos
Latest Reply
karthik_p
Esteemed Contributor
  • 1 kudos

@Suram Vinay​ From my end i have not implemented this, but just cheked this blog previously. terraform script will help for DR setup. https://www.databricks.com/blog/2022/07/18/disaster-recovery-automation-and-tooling-for-a-databricks-workspace.htmlc...

  • 1 kudos
1 More Replies
pranathisg97
by New Contributor III
  • 2967 Views
  • 2 replies
  • 1 kudos

readStream query throws exception if there's no data in delta location.

Hi,I have a scenario where writeStream query writes the stream data to bronze location and I have to read from bronze, do some processing and finally write it to silver. I use S3 location for delta tablesBut for the very first execution , readStream ...

  • 2967 Views
  • 2 replies
  • 1 kudos
Latest Reply
Vartika
Databricks Employee
  • 1 kudos

Hi @Pranathi Girish​,Hope all is well!Checking in. If @Suteja Kanuri​'s answer helped, would you let us know and mark the answer as best? If not, would you be happy to give us more information?We'd love to hear from you.Thanks!

  • 1 kudos
1 More Replies
Hubert-Dudek
by Esteemed Contributor III
  • 2549 Views
  • 2 replies
  • 9 kudos

databricks Photon is a next-generation engine on the Databricks Lakehouse Platform that provides speedy query performance at a low cost.- Its function...

databricks Photon is a next-generation engine on the Databricks Lakehouse Platform that provides speedy query performance at a low cost.- Its function coverage is growing, and UDF under Photon is coming, which can bring significant improvements in us...

ezgif-5-724cb0ccf8
  • 2549 Views
  • 2 replies
  • 9 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 9 kudos

 

  • 9 kudos
1 More Replies
gdev
by New Contributor
  • 6267 Views
  • 6 replies
  • 3 kudos

Resolved! Migrate notebooks and workflows and others .

I want to move notebooks , workflows , data from one users to another user in Azure Databricks. We move have access to that databricks. Is it possible? If, yes. How to move it.

  • 6267 Views
  • 6 replies
  • 3 kudos
Latest Reply
deedstoke
New Contributor II
  • 3 kudos

Hope all is well!

  • 3 kudos
5 More Replies
Paru
by New Contributor II
  • 1418 Views
  • 1 replies
  • 1 kudos
  • 1418 Views
  • 1 replies
  • 1 kudos
Latest Reply
Vartika
Databricks Employee
  • 1 kudos

Hi @Parvez Mushaf Maniyar​,Hope everything is going well.Does @Kaniz Fatma​'s response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 1 kudos
owen1
by New Contributor
  • 1146 Views
  • 2 replies
  • 2 kudos

workflow cluster was create error

I set the workflow to run at 12:00 every day in the workflow, but the workflow failed with the error message below, and I don't know why.Run result unavailable: run failed with error message Unexpected failure while waiting for the cluster (0506-0233...

  • 1146 Views
  • 2 replies
  • 2 kudos
Latest Reply
Murthy1
Contributor II
  • 2 kudos

Hello @Sangwoo Lee​ ,As mentioned by vignesh, it seems like an infra related issue. > Does the user (which executes the job) has access to start a cluster?> Incase if it is not an access issue and Incase if you are starting a lot of workflow jobs tog...

  • 2 kudos
1 More Replies
DrK
by New Contributor III
  • 1547 Views
  • 1 replies
  • 1 kudos

Current_Database() function unexpected results when queried with PowerBI

Hi,I`m creating some views to be queried by PowerBI. In our delta tables we have a column called database name which contains the source systems database name. What I`m doing is using this to filter data WHERE databaseName = current_database(). No...

  • 1547 Views
  • 1 replies
  • 1 kudos
Latest Reply
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 1 kudos

This widget could not be displayed.
Hi,I`m creating some views to be queried by PowerBI. In our delta tables we have a column called database name which contains the source systems database name. What I`m doing is using this to filter data WHERE databaseName = current_database(). No...

This widget could not be displayed.
  • 1 kudos
This widget could not be displayed.
Muskan
by New Contributor III
  • 5544 Views
  • 9 replies
  • 1 kudos

Unable to launch notebook

I have created a 12.2 LTS cluster and trying to launch notebook attached to this cluster. But unable to launch, it is not giving any error instead it is still showing the same home page.

  • 5544 Views
  • 9 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Muskan Bansal​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers...

  • 1 kudos
8 More Replies
kunaldeb
by New Contributor III
  • 3946 Views
  • 3 replies
  • 1 kudos

databricks cluster creation error

Hi Databricks Community,My Databricks workspace created on Azure pay-as-go subscription.I am facing two sided challenges.First>>   I am not able to create Delta-Live-table pipeline or any other all-purpose multi-node cluster as it is throwing below e...

  • 3946 Views
  • 3 replies
  • 1 kudos
Latest Reply
kunaldeb
New Contributor III
  • 1 kudos

HI All, Thanks for your reply. Just to update you, I am now able to to create DLT pipeline as well as all purpose multi-node cluster with minimum resources. This is due to Quota limit and I was able to increase it. But observation is, if I try to us...

  • 1 kudos
2 More Replies
Kit
by New Contributor III
  • 2856 Views
  • 6 replies
  • 1 kudos

Resolved! How can I get the list of downstream dashboards of a query

I want to deprecate a query from our workspace. However, I don't know if there is any downstream dashboard needs it.In case there is downstream dependency, then I can't just delete the query.I tried the search by query name, but it only returns the ...

  • 2856 Views
  • 6 replies
  • 1 kudos
Latest Reply
Kit
New Contributor III
  • 1 kudos

Databricks support suggests me to check the query history, to find out any history of the query, then check which dashboard is using it.Not an ideal solution but it works for me.

  • 1 kudos
5 More Replies
Nick_Hughes
by New Contributor III
  • 5034 Views
  • 2 replies
  • 1 kudos

Authorised views in databricks?

In GCP you can give a user access to a view, and then the view itself access to the underlying object, meaning you don't have to give end users access to the tables themselves.Is there a similar way of managing these permissions in databricks? The vi...

  • 5034 Views
  • 2 replies
  • 1 kudos
Latest Reply
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 1 kudos

This widget could not be displayed.
In GCP you can give a user access to a view, and then the view itself access to the underlying object, meaning you don't have to give end users access to the tables themselves.Is there a similar way of managing these permissions in databricks? The vi...

This widget could not be displayed.
  • 1 kudos
This widget could not be displayed.
1 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels