cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
pjp94
by • Contributor
  • 2099 Views
  • 1 replies
  • 1 kudos

DB Connect failing

For some reason, my dbconnect failed and I haven't been able to resolve the issue. I am connecting to an enterprise server. I was getting the following errors which (I believe) are now resolved.I defined the PYSPARK_DRIVER_PYTHON and PYSPARK_PYTHON v...

image.png image.png
Data Engineering
Reply
  • 2099 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Debayan
Databricks Employee
  • 1 kudos

Hi, Could you please confirm if you are providing the right shard ID/URL? Also, please let us know which guide has been followed to configure the connectivity. Also, please provide the screenshot of the error after entering the shard URL.

  • 1 kudos
cybersam
by • New Contributor II
  • 1185 Views
  • 2 replies
  • 0 kudos

How do I find the documentation for a Databricks "platform release"?

My DB portal says the platform version is v3.86, and provides a link to all the releases. But none of those releases state the "platform version". And I can't find "v3.86" by searching in the Databricks docs.So, how does one find the documentation fo...

Data Engineering
Reply
  • 1185 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 0 kudos

Hey @Samuel Yang​ ,Here you will find all detailshttps://docs.databricks.com/release-notes/index.htmlThanksAviral Bhardwaj

  • 0 kudos
1 More Replies
cmilligan
by • Contributor II
  • 2715 Views
  • 3 replies
  • 4 kudos

Link a visio diagram in a markdown cell

Is there a way to have databricks pull a diagram directly from visio? I've tried to use the embed links from visio but the image won't render. I'm trying to get around loading the image to DBFS as there may be updates to the image that I want it to g...

Data Engineering
Reply
  • 2715 Views
  • 3 replies
  • 4 kudos
View Replies
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 4 kudos

thanks for this

  • 4 kudos
2 More Replies
prashantp
by • New Contributor III
  • 2334 Views
  • 1 replies
  • 1 kudos

Resolved! CloudFormation failure while creating workspace

The cloud formation script is fails to create workspace due to following error. The solution to this error is not available in the standard troubleshooting guide. Following is the error message from cloud watch logs. It seems like a bug in the api, b...

Data Engineering
Reply
  • 2334 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
prashantp
New Contributor III
  • 1 kudos

I found a solution to this by creating a new role following these articles -https://docs.databricks.com/administration-guide/account-api/iam-role.htmlhttps://docs.databricks.com/administration-guide/account-api/aws-storage.html

  • 1 kudos
Sharath
by • New Contributor II
  • 2144 Views
  • 7 replies
  • 0 kudos

Hi Databricks Team,​ I passed the associate data engineer exam day before but still haven't received on accredible or on db academy. My registere...

Hi Databricks Team,​ I passed the associate data engineer exam day before but still haven't received on accredible or on db academy. My registered email id for exam is sharath.koushik@gmail.com. Could you please help ?​

Data Engineering
Reply
  • 2144 Views
  • 7 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Sharath K​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 0 kudos
6 More Replies
shamly
by • New Contributor III
  • 3533 Views
  • 3 replies
  • 2 kudos

How to remove extra ENTER line in csv UTF-16 while reading

Dear Friends,I have a csv and it looks like this‡‡Id‡‡,‡‡Version‡‡,‡‡Questionnaire‡‡,‡‡Date‡‡‡‡123456‡‡,‡‡Version2‡‡,‡‡All questions have been answered accurately and the guidance in the questionnaire was understood and followed‡‡,‡‡2010-12-16 00:01:...

Data Engineering
Reply
  • 3533 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 2 kudos

This is working fine, from pyspark.sql.functions import regexp_replace   path="dbfs:/FileStore/df/test.csv" dff = spark.read.option("header", "true").option("inferSchema", "true").option('multiline', 'true').option('encoding', 'UTF-8').option("delimi...

  • 2 kudos
2 More Replies
databicky
by • Contributor II
  • 4551 Views
  • 6 replies
  • 8 kudos

Resolved! How can we move the excel file from adls to share point

I have one excel file in the adls, i want to move that file into sharepoint, but i tried this method in data factory, but in sink sharepoint is not available, is there any possible way to do this?​

Data Engineering
Reply
  • 4551 Views
  • 6 replies
  • 8 kudos
View Replies
Latest Reply
daniel_sahal
Esteemed Contributor
  • 8 kudos

If you want to upload raw Excel file (not Sharepoint List) then IMO it would be easier to simply mount the Sharepoint Site as a network drive (https://support.microsoft.com/en-us/office/map-a-network-drive-to-a-sharepoint-library-751148de-f579-42f9-b...

  • 8 kudos
5 More Replies
299305
by • New Contributor II
  • 1982 Views
  • 4 replies
  • 3 kudos

where can I review the exam

Data Engineering
Reply
  • 1982 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @A DB​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 3 kudos
3 More Replies
Anatoly
by • New Contributor III
  • 5991 Views
  • 5 replies
  • 4 kudos

"Detected schema change" error while reading from delta table in streaming after applying "ALTER COLUMN DROP NOT NULL" to more than one columns.

Hi!I have a delta table and a process that reading a stream from this table.I need to drop the NOT NULL constraint from some of the columns of this table.The first drop command does not affect the reading stream.But the second command results in erro...

Data Engineering
Reply
  • 5991 Views
  • 5 replies
  • 4 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Anatoly Tikhonov​ Hope everything is going great.Does @Kaniz Fatma​  response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 4 kudos
4 More Replies
gj0904
by • New Contributor III
  • 3490 Views
  • 7 replies
  • 4 kudos

Certificate not received -

Hi ThereI successfully passed the exam on 27th Oct 2022 - but I haven't received the certificate yet.

Data Engineering
Reply
  • 3490 Views
  • 7 replies
  • 4 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Gaurav Jhamb​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...

  • 4 kudos
6 More Replies
Anuroop
by • New Contributor II
  • 3212 Views
  • 6 replies
  • 0 kudos

I've successfully passed Databricks Data Engineer Associate Certified exam but still have not received the certificate. Could you help on it pleas...

I've successfully passed Databricks Data Engineer Associate Certified exam but still have not received the certificate. Could you help on it please.

Data Engineering
Reply
  • 3212 Views
  • 6 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Venkata Sai Anuroop Samudrala​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to he...

  • 0 kudos
5 More Replies
Justin_Stuparit
by • New Contributor II
  • 1618 Views
  • 2 replies
  • 1 kudos

Configure DLT Pipeline to use existing running cluster

How can I configure a DLT pipeline to use an existing running cluster? I don't see where in the settings to set the pipeline to use an existing cluster. Instead it wants to always standup a new cluster.

Data Engineering
Reply
  • 1618 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Justin Stuparitz​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.T...

  • 1 kudos
1 More Replies
alejandrofm
by • Valued Contributor
  • 2737 Views
  • 3 replies
  • 0 kudos

Cluster not following the zone: auto configuration

Hi, I'm trying some new instance types, so to be sure I will get one I set the Availability zone to 'auto'.But this is happening, the image of the error and JSON are attached.So, the cluster is trying three times to upscale and the three times is usi...

image image image.png
Data Engineering
Reply
  • 2737 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Alejandro Martinez​ Hope everything is going great.Does @Sivaprasad C S​  response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 0 kudos
2 More Replies
Anonymous
by • Not applicable
  • 1464 Views
  • 2 replies
  • 2 kudos

Time Parameter Returns Incorrect Time

For some reason, the parameter keeps showing me my local time in UTC. When converting this parameter from EST, the offset is wrong. In the screenshot, you can see that my local session TZ is set to ETC/UTC (for whatever reason). It looks like the par...

Data Engineering
Reply
  • 1464 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Zachary Higgins​ Hope everything is going great.Does @Prasad Wagh​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 2 kudos
1 More Replies
Smitha1
by • Valued Contributor II
  • 3769 Views
  • 6 replies
  • 10 kudos

databricks learning Full screen disabled in selfpaced course

databricks learning self paced course Databricks Certified Associate Developer for Apache Spark 3 has below issues 1) Full screen button is disabled, difficult to see small font and stress on eyes.PFA2) courses do not have captions. Sometimes it is d...

font is very small as full screen is disabled.
Data Engineering
Reply
  • 3769 Views
  • 6 replies
  • 10 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 10 kudos

Hi @Smitha Nelapati​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us ...

  • 10 kudos
5 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels