cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Madman
by • New Contributor II
  • 13719 Views
  • 5 replies
  • 6 kudos

Snowflake connection to Databricks error

When I am trying to read snowflake table from my databricks notebook, it is giving the error as:df1.read.format("snowflake") \.options(**options) \.option("query", "select * from abc") \.save()Getting below errorjava.sql.SQLException: No suitable dri...

Data Engineering
Reply
  • 13719 Views
  • 5 replies
  • 6 kudos
View Replies
Latest Reply
pdiegop
New Contributor II
  • 6 kudos

@anurag2192 did you managed to solve it?

  • 6 kudos
4 More Replies
Axatar
by • New Contributor III
  • 11566 Views
  • 3 replies
  • 3 kudos

Resolved! query takes too long to write into delta table.

hello, am running into in issue while trying to write the data into a delta table, the query is a join between 3 tables and it takes 5 minutes to fetch the data but 3hours to write the data into the table, the select has 700 records. here are the app...

Data Engineering
Reply
  • 11566 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
Axatar
New Contributor III
  • 3 kudos

it turned out that the issue was not in the writing side, even when i was getting the results in under 5min, the issue was in the cross join in my query i resolved the issue by doing the same cross joins via dataframes got the results computed and wr...

  • 3 kudos
2 More Replies
Ajay-Pandey
by • Esteemed Contributor III
  • 4715 Views
  • 4 replies
  • 3 kudos

Resolved! Issue while re-starting Streaming DLT pipeline - PUBSUB

I am not able to restart our streaming DLT pipeline as it's giving "__tmp_path_dir" number format error, I am using PUB-SUB as a source for streaming.Any solutions will help - #pusub #databricks 

fff.jpg
Data Engineering
dlt
pubsub
streaming
Reply
  • 4715 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
Ajay-Pandey
Esteemed Contributor III
  • 3 kudos

 @Retired_mod @Amine 

  • 3 kudos
3 More Replies
elgeo
by • Valued Contributor II
  • 35236 Views
  • 9 replies
  • 5 kudos

Resolved! SQL Declare Variable equivalent in databricks

Hello. What would be the equivalent of the below in databricks?DECLARE @LastChangeDate as dateSET @LastChangeDate = GetDate()I already tried the below and worked. However I need to know how set a sql variable dynamicallySET da.dbname = test;SELECT "$...

Data Engineering
Reply
  • 35236 Views
  • 9 replies
  • 5 kudos
View Replies
Latest Reply
srinitechworld
New Contributor II
  • 5 kudos

hi try to to control the variables  

  • 5 kudos
8 More Replies
p4pratikjain
by • Contributor
  • 1754 Views
  • 1 replies
  • 0 kudos

Resolved! How to send Task Specific Parameters using run-now REST API.

I am using Following job trigger API -https://docs.databricks.com/api/workspace/jobs/runnowHere is what I want to achieve -I have complex workflow with multiple tasks and each task has python parameter named "--conf-file".While using runnow REST API ...

Data Engineering
Reply
  • 1754 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
victormarcel
New Contributor III
  • 0 kudos

@p4pratikjain wrote:I am using Following job trigger API - Popeyes Guest Experience Surveyhttps://docs.databricks.com/api/workspace/jobs/runnowHere is what I want to achieve - I have complex workflow with multiple tasks and each task has python param...

  • 0 kudos
Sivaji
by • New Contributor
  • 660 Views
  • 0 replies
  • 0 kudos

Data bricks Exam got suspended

Hello Team, I encountered Pathetic experience while attempting my 1st DataBricks certification. Abruptly, Proctor asked me to show my desk, after showing he/she asked multiple times.. wasted my time and then suspended my exam. I want to file a compla...

Data Engineering
Reply
  • 660 Views
  • 0 replies
  • 0 kudos
Juha
by • New Contributor II
  • 2216 Views
  • 3 replies
  • 2 kudos

I'm trying to create a table in databricks sql using widget values in table naming. The idea is that the users could select / enter table naming values as they create their tables. This can be done in notebooks but I can't get the syntax working in DBSQL.

Here's an example: CREATE OR REPLACE TABLE {{workspace}}.{{TableNameFirstPart}}_{{TableNameEndPart}} AS SELECT ...

Data Engineering
Reply
  • 2216 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
lawrence009
Contributor
  • 2 kudos

Have you figured out what the problem was? Could the issue be permission related?

  • 2 kudos
2 More Replies
Fast_Lanes
by • New Contributor II
  • 1350 Views
  • 2 replies
  • 3 kudos

Why am I getting shown estimated costs and charges (DBU in $) during community trial edition?

I recently signed up for the 14 day community trial and I noticed it was incurring or showing estimate costs on my Usage page shortly after I created my first workspace linked through AWS. Is this only for monitoring purposes or am I actually going t...

Data Engineering
Reply
  • 1350 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Ajay-Pandey
Esteemed Contributor III
  • 3 kudos

Hi @Fast_Lanes No, you will not be charged for the trial period on databricks usage.It's just showing the usual charges that will be charged when you will not be using trial version.

  • 3 kudos
1 More Replies
ADataBricksSP
by • New Contributor
  • 1106 Views
  • 1 replies
  • 1 kudos

Resolved! Community Edition SignUp option not visible

I would like to do some practical scenarios testing thro communication edition. For that, i check signup for community edition , but this option is not visible/shows in login page itself whereas shows only sign-in .. Can anyone help?

Data Engineering
Reply
  • 1106 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Ajay-Pandey
Esteemed Contributor III
  • 1 kudos

Hi @ADataBricksSP Please find below URL for signup as new user please all the required details click continue then you will find the option to signup as community edition -Databricks Community Signup Link 

  • 1 kudos
danatsafe
by • New Contributor
  • 5608 Views
  • 3 replies
  • 0 kudos

Amazon returns a 403 error code when trying to access an S3 Bucket

Hey! So far I have followed along with the Configure S3 access with instance profiles article to grant my cluster access to an S3 bucket. I have also made sure to disable IAM role passthrough on the cluster. Upon querying the bucket through a noteboo...

Data Engineering
Reply
  • 5608 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
winojoe
New Contributor III
  • 0 kudos

I had the same issue and I found a solutionFor me, the permission problems only exist when the Cluster's (compute's) Access mode is "Shared No Isolation".  When the Access Mode is either "Shared" or "Single User" then the IAM configuration seems to a...

  • 0 kudos
2 More Replies
cmditch
by • New Contributor II
  • 1843 Views
  • 1 replies
  • 0 kudos

Intermittent secret resolution error service fault in GCP

Experiencing the error below in GCP when starting a cluster (both manually and in jobs). It's causing our ETL and other production jobs to fail multiple times a week. Its intermittent, but requires manual intervention to retry scheduled jobs. run fai...

Data Engineering
Reply
  • 1843 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
cmditch
New Contributor II
  • 0 kudos

Thanks @Retired_mod . 1 and 2 are confirmed fine. I would imagine 3 to not result in intermittent failures if it were a config issue, but perhaps it's another network related issue that would be susceptible to intermittent failure.The link you provid...

  • 0 kudos
Starki
by • New Contributor III
  • 3157 Views
  • 3 replies
  • 2 kudos

StreamingQueryListener onQueryTerminated in Databricks Job

I am defining a StreamingQueryListener that collects metrics on my Spark Structured Streaming tasks and sends them to a Prometheus Pushgateway.When the job is terminated, I want to use the onQueryTerminated to cleanup the metrics for each job from th...

Data Engineering
onQueryTerminated
StreamingQueryListener
Reply
  • 3157 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
shan_chandra
Databricks Employee
  • 2 kudos

@Starki  - Per documentation, StreamingQueryListener.onQueryTerminated is called when the query is stopped, e.g., StreamingQuery.stop.and each of these Python observable APIs work asynchronously. https://www.databricks.com/blog/2022/05/27/how-to-moni...

  • 2 kudos
2 More Replies
User16765131552
by • Contributor III
  • 2008 Views
  • 2 replies
  • 0 kudos

Resolved! Disable welcome emails

Is it possible to disable the welcome emails that go to users when added to the workspace?

Data Engineering
Reply
  • 2008 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
User16765131552
Contributor III
  • 0 kudos

I have found that it is possible to suppress welcome emails if users are added via the API using flags.

  • 0 kudos
1 More Replies
Loki
by • New Contributor III
  • 3527 Views
  • 4 replies
  • 1 kudos

Resolved! Accessing ADLS Gen 2 Raw Files with UC ?

We are using service principal to access data from raw files such as json, CSV .I saw a video suggesting that it could be done via unity catalog as well.Could someone comment on this please ?

Data Engineering
Reply
  • 3527 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
donkyhotes
New Contributor II
  • 1 kudos

@Loki wrote:We are using service principal to access data from raw files such as json, CSV .Car GamesI saw a video suggesting that it could be done via unity catalog as well.Could someone comment on this please ?That's great! Service principals are a...

  • 1 kudos
3 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels