cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
Kv1
by • New Contributor III
  • 3183 Views
  • 5 replies
  • 6 kudos

Databricks Professional Exam - . No Transcript provided when you fail

When I passed associate exam , I was provided with Transcript which showcase strength and weakness in learning area like the one below which help us to understand gap in skill set and which area to concentrate to improve furtherYesterday I failed i...

image
Data Engineering
Reply
  • 3183 Views
  • 5 replies
  • 6 kudos
View Replies
Latest Reply
jose_gonzalez
Moderator
  • 6 kudos

Adding @Vidula Khanna​ and @Kaniz Fatma​ for visibility

  • 6 kudos
4 More Replies
SDG_Peter
by • New Contributor III
  • 4666 Views
  • 7 replies
  • 7 kudos

Spark image failed to download or does not exist

Good morning, and thank you for the supportIn our scheduled job one cluster failed to start with the following error:```Run result unavailable: job failed with error messageUnexpected failure while waiting for the cluster to be ready.Cause Unexpected...

Data Engineering
Reply
  • 4666 Views
  • 7 replies
  • 7 kudos
View Replies
Latest Reply
LandanG
Honored Contributor
  • 7 kudos

@Rita Fernandes​ @Kajetan Gęgotek​ @Yoshi Coppens​ @Viktor Fulop​ @Andrius Vitkauskas​ @Pietro Maria Nobili​ It looks like an issue with Azure account limits, the Databricks eng team is looking into it. Apart from retries, I'd suggest running jobs no...

  • 7 kudos
6 More Replies
pawank9
by • New Contributor II
  • 2342 Views
  • 2 replies
  • 0 kudos

SQL dashboard report: Select table column dynamically

I am using SQL Dashboard report, there are around 400 columns in a table, I want to display some of the columns based on data, not all 400 columns should be displayed.eg: a table have column named c1, c2,c3,c4 and rows r1, r2, r3, r4. The report shou...

Data Engineering
Reply
  • 2342 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 0 kudos

You can build own logic using query parameters, and/or just select required columns based on case.

  • 0 kudos
1 More Replies
KVNARK
by • Honored Contributor II
  • 2209 Views
  • 4 replies
  • 3 kudos

Error while reading a file from ADLS 2.0 in databricks platformbelow is the error. Anyone faced the similar issue and solution to fix this.

Error while reading a file from ADLS 2.0 in databricks platformbelow is the error. Anyone faced the similar issue and solution to fix this.

image
Data Engineering
Reply
  • 2209 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
Vivian_Wilfred
Honored Contributor
  • 3 kudos

Hey @KVNARK .​ did you run nslookup to the storage endpoint and confirm if this isn’t a DNS issue? If not, run this from the same notebook -%sh nslookup <storage account>.dfs.core.windows.net

  • 3 kudos
3 More Replies
Yoshe1101
by • New Contributor III
  • 2892 Views
  • 2 replies
  • 0 kudos

Resolved! HTTP Response code: 302

When trying to access a Databricks Warehouse from an external workspace I get the following error.Both, the Warehouse and the workspace are hosted on different AWS subscriptions.But curiously, with the same script I can access the Databricks Warehous...

Data Engineering
Reply
  • 2892 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 0 kudos

It is network related issue. Please check the private link, VPC peering, DNS names resolving to ip (from AWS and Azure), and routing of that IP inside AWS. You can also open a support ticket with AWS.

  • 0 kudos
1 More Replies
abhi_1825
by • New Contributor III
  • 2946 Views
  • 6 replies
  • 1 kudos

Resolved! Databricks certified Data Engineer Associate V3 Exam - Voucher code not working.

So I have a voucher code which I received after completing the Lakehouse Fundamentals exam. However, I am not able to use it for Databricks certified Data Engineer Associate V3 Exam in public section. Its working for V2 version along with other exams...

Data Engineering
Reply
  • 2946 Views
  • 6 replies
  • 1 kudos
View Replies
Latest Reply
abhi_1825
New Contributor III
  • 1 kudos

Thanks guys. My problem of voucher code is resolved after raising a ticket with Training team.

  • 1 kudos
5 More Replies
Toy
by • New Contributor II
  • 2362 Views
  • 3 replies
  • 0 kudos

Pipeline Error [Py4JJavaError] com.databricks.WorkflowException: com.databricks.NotebookExecutionException: FAILED

I have a pipeline that used used to run successfully and now all of a sudden is returning this error that I cannot resolve: [Py4JJavaError] 

image
Data Engineering
Reply
  • 2362 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Toy
New Contributor II
  • 0 kudos

Hi Guys, You're right the problem is with the child notebook.All my notebooks are failing at this point. I can't seem to be wining with solving this error

  • 0 kudos
2 More Replies
Phani1
by • Valued Contributor II
  • 1129 Views
  • 1 replies
  • 0 kudos

Databricks Issue with the returning results to PowerBI

While returning results to PowerBI, Databricks Completed the session (in 9 mins) but PowerBI waiting for the results (more than 7 hrs. for 20 GB of data).Could you please help us on this.

Data Engineering
Reply
  • 1129 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
daniel_sahal
Esteemed Contributor
  • 0 kudos

@Janga Reddy​ It's really hard to find a solution here without further investigating PBI Gateway performance/Data models etc.If Databricks completed the session in 9 mins then I assume that the issue could be with the performance of PBI datasets.

  • 0 kudos
MachuPichu
by • New Contributor III
  • 5705 Views
  • 9 replies
  • 4 kudos

Copying Delta table from QA to Prod

I am looking for a way to copy large managed Delta table like 4TB from one environment (QA) to other(Prod) . QA and Prod are in different subscription and in different region. I understand Databricks provides a way to clone table. But I am not sure i...

Data Engineering
Reply
  • 5705 Views
  • 9 replies
  • 4 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 4 kudos

I would use a data factory to copy 4TB files as it has gigantic throughput. After completing a copy of everything, I would register as a table in the new metastore.

  • 4 kudos
8 More Replies
521746
by • New Contributor
  • 431 Views
  • 0 replies
  • 0 kudos

Deleting account level group through API throws error

Hi team, we are getting an error in deleting an account level group with the scim API{ "schemas": [ "urn:ietf:params:scim:api:messages:2.0:Error" ], "detail": "INTERNAL_ERROR: Unexpected error: getTenantIdForAccountId is not implemented in HybridWor...

Data Engineering
Reply
  • 431 Views
  • 0 replies
  • 0 kudos
SivaRamaKrishna
by • New Contributor
  • 6554 Views
  • 2 replies
  • 0 kudos

Resolved! I got this error help me how to sort out?

The file actually in DEV environment... My job was sync data in QA ENVIRONMENT.The query execute in QA ENVIRONMENT I got this error ...how to solve this issue​

Data Engineering
Reply
  • 6554 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
jose_gonzalez
Moderator
  • 0 kudos

Hi @SivaRamaKrishna Mulagalapati​,Could you copy and paste the full error stack trace? and also share the code you are trying to run.

  • 0 kudos
1 More Replies
hussi
by • New Contributor
  • 1189 Views
  • 3 replies
  • 0 kudos

I also have the same issue, can someone please help ?I just passed the Lakehouse Fundamentals Accreditation and I haven&#39;t received any badge or ce...

I also have the same issue, can someone please help ?I just passed the Lakehouse Fundamentals Accreditation and I haven't received any badge or certificate for it. I understand that I need to go to credentials.databricks.com but it is not there. How ...

Data Engineering
Reply
  • 1189 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
jose_gonzalez
Moderator
  • 0 kudos

Hi @Husain Sadriwala​ ,Just a friendly follow-up. Did you receive your badge yet? please let us know if you still need help.

  • 0 kudos
2 More Replies
Craig_
by • New Contributor III
  • 3146 Views
  • 4 replies
  • 0 kudos

Resolved! Caveats when importing functions from REPO stored .py files

The ability to import .py files into notebooks looked like a clean and easy way to reuse code and to ensure all notebooks are using the same version of code. However, two items remain unclear after scouring documentation and forums.Are these the rig...

Data Engineering
Reply
  • 3146 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 0 kudos

"does not have access to the spark session by default" yes, that is correct you need to pass the reference to the spark variable inside Class or Function, something like when you call from notebook function_from_file(spark=spark)displayHTML() is desi...

  • 0 kudos
3 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels