cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
brickster_2018
by • Esteemed Contributor
  • 2371 Views
  • 2 replies
  • 0 kudos

Resolved! The driver is temporarily unavailable

My job fails with Driver is temporarily unavailable. Apparently, it's permanently unavailable, because the job is not pausing but failing.

Data Engineering
Reply
  • 2371 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Chalki
New Contributor III
  • 0 kudos

I am facing the same issues .  I am writing in batches using a simple for loop. I don't have any collect statements inside the loop. I am rewriting the partitions with partition overwrite dynamic mode in a huge wide delta table - several tb. The incr...

  • 0 kudos
1 More Replies
Kratik
by • New Contributor III
  • 6080 Views
  • 10 replies
  • 1 kudos

How to run sql query/notebook present on S3 on databricks cluster?

I have code files (.hql) in s3 which was running on Hive. But now, I need to run them on the Databricks cluster. I can rename the files to .sql and add comment "-- Databricks notebook source" on top of the files to treat them as single cell notebook....

Data Engineering
Reply
  • 6080 Views
  • 10 replies
  • 1 kudos
View Replies
Latest Reply
Debayan
Esteemed Contributor III
  • 1 kudos

Hi, Sorry missed seeing the post, did the issue got resolved? 

  • 1 kudos
9 More Replies
PriyanshuKumar
by • New Contributor
  • 1395 Views
  • 1 replies
  • 1 kudos

try_to_timestamp not working as expected

try_to_timestamp should return NULL for following expression but it is throwing error: try_to_timestamp( '2019-02-28 23:59:59.000', 'yyyy-MM-dd HH:mm:ss' ) I understand above expression is wrong as the date format does no...

Data Engineering
Reply
  • 1395 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Kumaran
Valued Contributor III
  • 1 kudos

Hi @PriyanshuKumar,Thank you for your question in the Databricks community.The try_to_timestamp() function is designed to attempt to convert a string to timestamp format based on a specified pattern and return null if it fails to do so. However, in t...

  • 1 kudos
Mihai_Cog
by • Contributor
  • 6157 Views
  • 6 replies
  • 10 kudos

Resolved! Change Data Feed Databricks

Hello,I am doing some testing with this feature Change Data Feed using Databricks and Pyspark, of course the Delta format and I don't understand something:I created a tableSaved some data insideEnabled Change Data Feed featureApply a merge with a dat...

Data Engineering
Reply
  • 6157 Views
  • 6 replies
  • 10 kudos
View Replies
Latest Reply
Tharun-Kumar
Honored Contributor II
  • 10 kudos

@Mihai_Cog You have to split your merge statement into 2 parts. (Update and Insert/Delete).MERGE INTO test t USING src s ON s.Id = t.Id and s.date_field = t.date_field and s.fields <> t.fields WHEN MATCHED THEN UPDATE SET * MERGE INTO test t USING sr...

  • 10 kudos
5 More Replies
Smitha1
by • Valued Contributor II
  • 5669 Views
  • 10 replies
  • 9 kudos

Resolved! Request for reattempt voucher. Databricks Certified Associate Developer for Apache Spark 3.0 exam

Hi,I gave Databricks Certified Associate Developer for Apache Spark 3.0 exam today but missed by one percent. I got 68.33% and pass is 70%.I am planning to reattempt the exam, could you kindly give me another opportunity and provide reattempt voucher...

Data Engineering
Reply
  • 5669 Views
  • 10 replies
  • 9 kudos
View Replies
Latest Reply
shriya
New Contributor II
  • 9 kudos

Hi,I gave Databricks Certified Associate Developer for Apache Spark 3.0 Python exam yesterday but missed by three percent. I got 66.66% and pass is 70%.I am planning to reattempt the exam, could you kindly give me another opportunity and provide reat...

  • 9 kudos
9 More Replies
-werners-
by • Esteemed Contributor III
  • 5263 Views
  • 6 replies
  • 12 kudos

Resolved! SSRS (on-prem) on Databricks SQL

Has anybody succeeded in querying Databricks SQL with an on-prem SSRS (so an on-prem Report Server and Report Builder)?I manage to create a connection that works (according to the connection test anyway), but the moment I try to create a dataset on t...

Data Engineering
Reply
  • 5263 Views
  • 6 replies
  • 12 kudos
View Replies
Latest Reply
Haider93
New Contributor III
  • 12 kudos

Hi @-werners- , I am able to build connection between Microsoft visual studio and data bricks using Simba Spark ODBC driver. I can query delta tables sitting in Databricks from Microsoft Visual studio (SSRS). However, when I am deploying the report t...

  • 12 kudos
5 More Replies
Ronis
by • New Contributor
  • 4471 Views
  • 4 replies
  • 1 kudos

SSRS Connect to Databricks

Hi ,I need to connect databricks query from microsoft SSRS.is it possible ? How do you make the connection?

Data Engineering
Reply
  • 4471 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
Tharun-Kumar
Honored Contributor II
  • 1 kudos

@Haider93 https://www.databricks.com/company/partners/technology This is the list of supported databricks technology partners. SSRS is not yet part of it. We do have official support for Power BI. You can use this document to mimic it for SSRS - http...

  • 1 kudos
3 More Replies
aiwithqasim
by • Contributor
  • 628 Views
  • 0 replies
  • 0 kudos

Why no Practice Test for Databricks Professional Exam ?

Hi Databricks Team/Community,I'm planning for Databricks Data Engineering Professional Certification but what I have seen is that Databricks don't give any practice test for this professional/advanced certification. I don't know why, If anyone is rea...

Data Engineering
Reply
  • 628 Views
  • 0 replies
  • 0 kudos
andresfp14
by • New Contributor II
  • 26816 Views
  • 2 replies
  • 0 kudos

Can you delete a widget, or force a value to it?

Hello everyone! Basically, I have a dropdown widget, and at some point I want to change the options available, but still maintain the same selected option. The problems that I find are: - If I want to delete the widget and create a new one, it seems...

0693f000007OoHmAAK
Data Engineering
Reply
  • 26816 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Gim
Contributor
  • 0 kudos

I know this is 5+ years late but I would like to add an answer (or work around) to this in case anyone else was looking for the same solution.What I did was to essentially 'reset' the widget whenever I had to assign (or force) a value to it.#Cell #1 ...

  • 0 kudos
1 More Replies
Swostiman
by • New Contributor II
  • 4817 Views
  • 5 replies
  • 4 kudos

Consuming data from databricks[Hive metastore] sql endpoint using pyspark

I was trying to read some delta data from databricks[Hive metastore] sql endpoint using pyspark, but while doing so I encountered that all the values of the table after fetching are same as the column name.Even when I try to just show the data it giv...

Data Engineering
Reply
  • 4817 Views
  • 5 replies
  • 4 kudos
View Replies
Latest Reply
sucan
New Contributor II
  • 4 kudos

Encountered the same issue and downgrading to 2.6.22 helped me resolve this issue.

  • 4 kudos
4 More Replies
pg1
by • New Contributor II
  • 1045 Views
  • 2 replies
  • 1 kudos

How to expand display of workspace name in top right window?

How to expand display of workspace name in top right window? My workspace name is little bit longer and it cuts out, could not find a way to expand name window.

Data Engineering
Reply
  • 1045 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Tharun-Kumar
Honored Contributor II
  • 1 kudos

@pg1 Is your workspace Unity Catalog enabled?We have identified this behaviour in Unity Catalog enabled workspaces and already working to show up the entire workspace name when we hover over it.

  • 1 kudos
1 More Replies
cbabay
by • New Contributor II
  • 1662 Views
  • 3 replies
  • 2 kudos

Resolved! Adding Duration Warning During Workflow Deployment

Hi Team,Thank you very much for making it so simple to add notifications on failure and duration warnings to workflows.Each time we deploy to dev, through Jenkins, our email is unsubscribed and we stop getting alerts. Is there any documentation on ho...

Data Engineering
Reply
  • 1662 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
alexott
Valued Contributor II
  • 2 kudos

It heavily depends on how do you perform deployment - what APIs are used, etc. Most probably you're using Job Reset API instead of Job Update API

  • 2 kudos
2 More Replies
Yasser
by • New Contributor
  • 658 Views
  • 0 replies
  • 0 kudos

[sql warehouse] Invalid configuration value detected for fs.azure.account.key with 'force' = 'true'

Hello,I am getting the following error when trying to copy data to databricks from an ADLS with SQL and using a SAS tokenFailure to initialize configuration for storage account <storage account>: Invalid configuration value detected for fs.azure.acco...

Data Engineering
Reply
  • 658 Views
  • 0 replies
  • 0 kudos
Chris_Shehu
by • Valued Contributor III
  • 14660 Views
  • 5 replies
  • 5 kudos

Resolved! What is the best way to handle big data sets?

I'm trying to find the best strategy for handling big data sets. In this case I have something that is 450 million records. I'm pulling the data from SQL Server very quickly but when I try to push the data to the Delta Table OR a Azure Container the...

Data Engineering
Reply
  • 14660 Views
  • 5 replies
  • 5 kudos
View Replies
Latest Reply
Wilynan
New Contributor II
  • 5 kudos

I think you should consult experts in Big Data for advice on this issue

  • 5 kudos
4 More Replies
parthsalvi
by • Contributor
  • 2138 Views
  • 1 replies
  • 2 kudos

Amazon SES : boto3 credentials not found. DBR 11.2 Shared mode

We're trying to send email using Amazon SES using boto3.client in python. We've added SES Full access in clusters IAM Role.We were able to send email in "No isolation shared" mode in DBR 11.2 using ses = boto3.client('ses', region_name='us-****-2') s...

image
Data Engineering
Reply
  • 2138 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
JameDavi_51481
New Contributor III
  • 2 kudos

This appears to be an intentional design choice to prevent users from using the credentials of the host machine to carry out arbitrary AWS API calls. I really wish there was a workaround or setting to disable this behavior because we put a lot of wor...

  • 2 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels