cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
KalingaSena
by • New Contributor II
  • 4168 Views
  • 3 replies
  • 0 kudos

Not able to execute below SQL query in databricks notebook because of Pare error

Hi Team,I am unable to run the below command and it is giving me a parse error. Can any one point out the issue with the code:   

KalingaSena_1-1689140837096.png
Data Engineering
Reply
  • 4168 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
BkP
Contributor
  • 0 kudos

Hi,From the error , it looks like there is no space between the brackets and the "in" keyword after the where clause. Can you please try again see if you facing the same error.  

  • 0 kudos
2 More Replies
apiury
by • New Contributor III
  • 2452 Views
  • 2 replies
  • 1 kudos

Consume gold data layer from web application

Hello!We are developing a web application in .NET, we need to consume data in gold layer, (as if we had a relational database), how can we do it? export data to sql server from gold layer?

Data Engineering
Reply
  • 2452 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @apiury  Thank you for posting your question in our community! We are happy to assist you. To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your ...

  • 1 kudos
1 More Replies
NithinTiruveedh
by • New Contributor II
  • 24091 Views
  • 12 replies
  • 0 kudos

How can I split a Spark Dataframe into n equal Dataframes (by rows)? I tried to add a Row ID column to acheive this but was unsuccessful.

I have a dataframe that has 5M rows. I need to split it up into 5 dataframes of ~1M rows each. This would be easy if I could create a column that contains Row ID. Is that possible?

Data Engineering
Reply
  • 24091 Views
  • 12 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @NithinTiruveedh  Thank you for posting your question in our community! We are happy to assist you. To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answ...

  • 0 kudos
11 More Replies
Anonymous
by • Not applicable
  • 2248 Views
  • 2 replies
  • 3 kudos

Databricks streaming dataframe into Snowflake

Any suggestions on how to stream data from databricks into snowflake?. Is snowpipe is the only option?. Snowpipe is not faster since it runs copy into in a small batch intervals and not in few seconds. If no option other than snowpipe, how to call it...

Data Engineering
Reply
  • 2248 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Anonymous  Hope everything is going great. Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we...

  • 3 kudos
1 More Replies
Gil
by • New Contributor III
  • 6899 Views
  • 10 replies
  • 7 kudos

DLT optimize and vacuum

We were finally able to get DLT pipelines to run the optimize and vacuum automatically.  We verified this via the the table history.   However I am able to still query versions older than 7 days.   Has anyone been experiencing this and how were you a...

Data Engineering
Reply
  • 6899 Views
  • 10 replies
  • 7 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 7 kudos

Hi @Gil  Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.  We'd love to hear from you. Thanks!

  • 7 kudos
9 More Replies
Anonymous
by • Not applicable
  • 2521 Views
  • 2 replies
  • 5 kudos

 Dear @Werner Stinckens​  and @Tyler Retzlaff​  We would like to express our gratitude for your participation and dedication in the Databricks Commun...

 Dear @Werner Stinckens​  and @Tyler Retzlaff​ We would like to express our gratitude for your participation and dedication in the Databricks Community last week. Your interactions with customers have been valuable and we truly appreciate the time...

Screenshot 2023-06-13 at 8.42.49 PM
Data Engineering
Reply
  • 2521 Views
  • 2 replies
  • 5 kudos
View Replies
Latest Reply
dplante
Contributor II
  • 5 kudos

Congratulations guys!

  • 5 kudos
1 More Replies
Constantine
by • Contributor III
  • 7590 Views
  • 2 replies
  • 4 kudos

Resolved! How does merge schema work

Let's say I create a table like CREATE TABLE IF NOT EXISTS new_db.data_table ( key STRING, value STRING, last_updated_time TIMESTAMP ) USING DELTA LOCATION 's3://......';Now when I insert into this table I insert data which has say 20 columns a...

Data Engineering
Reply
  • 7590 Views
  • 2 replies
  • 4 kudos
View Replies
Latest Reply
timdriscoll22
New Contributor II
  • 4 kudos

I tried running "REFRESH TABLE tablename;" but I still do not see the added columns in the data explorer columns, while I do see the added columns in the sample data 

  • 4 kudos
1 More Replies
pjain
by • New Contributor II
  • 3594 Views
  • 4 replies
  • 0 kudos

_sqldf value in case of query failure in %sql cell

I am trying to write a code for Error Handling in Databricks notebook in case of a SQL magic cell failure. I have a %sql cell followed by some python code in next cells. I want to abort the notebook if the query in %sql cell fails. To do so I am look...

Data Engineering
Reply
  • 3594 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @pjain  We haven't heard from you since the last response from @daniel_sahal ​, and I was checking back to see if her suggestions helped you. Or else, If you have any solution, please share it with the community, as it can be helpful to others.  A...

  • 0 kudos
3 More Replies
GC-James
by • Contributor II
  • 16418 Views
  • 15 replies
  • 5 kudos

Resolved! Lost memory when using dbutils

Why does copying a 9GB file from a container to the /dbfs lose me 50GB of memory? (Which doesn't come back until I restarted the cluster)

image
Data Engineering
Reply
  • 16418 Views
  • 15 replies
  • 5 kudos
View Replies
Latest Reply
AdrianP
New Contributor II
  • 5 kudos

Hi James,Did you get to the bottom of this? We are experiencing the same issue, and all the suggested solutions don't seem to work.Thanks,Adrian

  • 5 kudos
14 More Replies
Vadim1
by • New Contributor III
  • 3714 Views
  • 3 replies
  • 3 kudos

Resolved! Error on Azure-Databricks write RDD to storage account with wsabs://

Hi, I'm trying to write data from RDD to the storage account:Adding storage account key:spark.conf.set("fs.azure.account.key.y.blob.core.windows.net", "myStorageAccountKey")Read and write to the same storage:val path = "wasbs://x@y.blob.core.windows....

Data Engineering
Reply
  • 3714 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
TheoDeSo
New Contributor III
  • 3 kudos

Hello @Vadim1 and @User16764241763. I'm wondering if you find a way to avoid adding the hardcoded key in the advanced options spark config section in the cluster configuration. Is there a similar command to spark.conf.set("spark.hadoop.fs.azure.accou...

  • 3 kudos
2 More Replies
jdobken
by • New Contributor III
  • 8946 Views
  • 8 replies
  • 11 kudos

As the Databricks account manager; I cannot login: "Your user already belongs to a Databricks account"

On GCP I subscribed to Databricks in one project within the organization.Then I canceled this subscription and subscribed to Databricks in another project.When I try to login onto the newly subscribed databricks with google SSO:> There was an error s...

Screenshot 2023-06-07 at 11.32.30
Data Engineering
Reply
  • 8946 Views
  • 8 replies
  • 11 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 11 kudos

I can see the issue might be related to organizations or billing accounts. The new Databricks project I tried creating was on a different organization/billing-account than the test Databricks subscription I created a month back.I went back to the ori...

  • 11 kudos
7 More Replies
Distributed_Com
by • New Contributor III
  • 12927 Views
  • 4 replies
  • 6 kudos

Resolved! Location not empty but not a Delta table

I need help or insight regarding the following errors. My instructors (Brooke Wenig with Conor Murphy) ran this code successfully on our course video, but I cannot replicate what she did. Here is the code and below it is the outcome from my Cours...

Data Engineering
Reply
  • 12927 Views
  • 4 replies
  • 6 kudos
View Replies
Latest Reply
gilo12
New Contributor III
  • 6 kudos

DELETE the original Parquet table as a separate statementHow can this be done? simple query "DROP TABLE .... " still failing with "cannot be found" 

  • 6 kudos
3 More Replies
Siravich
by • New Contributor
  • 611 Views
  • 0 replies
  • 0 kudos

Permission on Unity catalog

I am facing an issue when assign permission on view created on unity catalog. The problem is I had create a user defined function (UDFs) in order to encrypt sensitive column, I create a view which call the functions and source table within the catalo...

Data Engineering
Reply
  • 611 Views
  • 0 replies
  • 0 kudos
glebex
by • New Contributor II
  • 10247 Views
  • 7 replies
  • 7 kudos

Resolved! Accessing workspace files within cluster init script

Greetings all!I am currently facing an issue while accessing workspace files from the init script.As it was explained in the documentation, it is possible to place init script inside workspace files (link). This works perfectly fine and init script i...

Data Engineering
Reply
  • 10247 Views
  • 7 replies
  • 7 kudos
View Replies
Latest Reply
jacob_hill_prof
New Contributor II
  • 7 kudos

@Gleb Smolnik​ You might also want to try cloning a github repo in your init script and then storing dependencies like requirements.txt files and other init scripts there. By doing this you can pull a whole slew of init scripts to be utilized in your...

  • 7 kudos
6 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels