cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Elon
by • New Contributor III
  • 1458 Views
  • 2 replies
  • 6 kudos

When are you gonna fix this site?

Unacceptable user experience for this forum: What markdown support?My colleagues agree the email verification looks sketchy.The website is SUPER slow. (10-56 seconds per page)Uses color red for email verified?!No feedback on repeated logins.The fonts...

ugly slow great 2023-03-15_10-09 databricks-smarting
Data Engineering
Reply
  • 1458 Views
  • 2 replies
  • 6 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 6 kudos

Hi @Elon Musk​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback wi...

  • 6 kudos
1 More Replies
sanq
by • New Contributor II
  • 2468 Views
  • 3 replies
  • 7 kudos

what formatter is used to format SQL cell in databricks

Databricks launched formatter Black which formats python cells, I can also see SQL cell getting formatted, but not sure which formatter is being used for SQL cell formatting. No clarity given on docs.

Data Engineering
Reply
  • 2468 Views
  • 3 replies
  • 7 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 7 kudos

Hi @SANJAY qq​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 7 kudos
2 More Replies
Unilever
by • New Contributor II
  • 834 Views
  • 2 replies
  • 1 kudos

I would like to get rid of the error

the SPN we use for the mount points has access to the dataset in question, but for some reason I get this errorPlease find the attached screenshot for the error details.

Screenshot (4)
Data Engineering
Reply
  • 834 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @sai chandu palkapati​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from y...

  • 1 kudos
1 More Replies
bd
by • New Contributor III
  • 2253 Views
  • 3 replies
  • 0 kudos

Resolved! Job aborted due to stage failure: ModuleNotFoundError

I'm getting this Failure Reason on a fairly simple streaming job. I'm running the job in a notebook. The notebook relies on a python module that I'm syncing to DBFS with `dbx`. Within the notebook generally, the module is available, i.e. `import mymo...

Data Engineering
Reply
  • 2253 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Benjamin Dean​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...

  • 0 kudos
2 More Replies
Dayaa
by • New Contributor II
  • 1536 Views
  • 3 replies
  • 3 kudos

Resolved! Load data into Azure SQL Database from Azure Databricks ( restricted table not a whole workspace tables)

Hi ,I want to share limited tables in my databricks workspace and users will connect to my databricks through Azure Data factory and will load data into Azure SQL. Is this possible using Delta Sharing? Or any other method or tool?

Data Engineering
Reply
  • 1536 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Dayananthan Marimuthu​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your...

  • 3 kudos
2 More Replies
Arunsundar
by • New Contributor III
  • 2125 Views
  • 5 replies
  • 4 kudos

The possibility of finding the workload dynamically and spin up the cluster based on the workload

Hi Team,Good morning. I would like to understand if there is a possibility to determine the workload automatically through code (data load from a file to a table, determine the file size, kind of a benchmark that we can check), based on which we can ...

Data Engineering
Reply
  • 2125 Views
  • 5 replies
  • 4 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 4 kudos

Hi @Arunsundar Muthumanickam​ ​​, We haven't heard from you since the last response from @Vigneshraja Palaniraj​ and @Debayan Mukherjee​​, and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it...

  • 4 kudos
4 More Replies
ossinova
by • Contributor II
  • 1197 Views
  • 2 replies
  • 2 kudos

PIVOT on month and quarter

I want to simplify this query:SELECT year(EntryDate) Year, AccountNumber, sum(CreditBase - DebitBase) FILTER(WHERE month(EntryDate) = 1) AS jan_total, sum(CreditBase - DebitBase) FILTER(WHERE month(EntryDate) = 2) AS feb_total, sum(CreditBase - Debi...

Data Engineering
Reply
  • 1197 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 2 kudos

Hi @Oscar Dyremyhr​, We haven't heard from you since the last response from @Lakshay Goel​ ​, and I was checking back to see if his suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to ...

  • 2 kudos
1 More Replies
Harun
by • Honored Contributor
  • 3054 Views
  • 6 replies
  • 6 kudos

how to load structured stream data into delta table whose location is in ADLS Gen2

Hi All,I am working on a streaming data processing. As a intial step i have read the data from azure eventhub using readstream. now i want to writestream this into a delta table. My requirement is, The data should present in external location (adls g...

Data Engineering
Reply
  • 3054 Views
  • 6 replies
  • 6 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 6 kudos

Hi @Harun Raseed Basheer​​​, We haven't heard from you since the last response from @Werner Stinckens​​, and I was checking back to see if his suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be ...

  • 6 kudos
5 More Replies
sanjay
by • Valued Contributor II
  • 5732 Views
  • 4 replies
  • 4 kudos

Resolved! PySpark UDF is taking long to process

Hi,I have UDF which runs for each spark dataframe row, does some complex processing and return string output. But it takes very long if data is 15000 rows. I have configured cluster with autoscaling, but its not spinning more servers.Please suggest h...

Data Engineering
Reply
  • 5732 Views
  • 4 replies
  • 4 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 4 kudos

Hi @Sanjay Jain​ ​​, We haven't heard from you since the last response from @Lakshay Goel​, @rishabh and @Vigneshraja Palaniraj​​, and I was checking back to see if their suggestions helped you.Or else, If you have any solution, please share it with ...

  • 4 kudos
3 More Replies
Dataengineer_mm
by • New Contributor
  • 1775 Views
  • 2 replies
  • 1 kudos

Surrogate key using identity column.

I want to create a surrogate in the delta table And i used the identity column id-Generated as DefaultCan i insert rows into the delta table using only spark.sql like Insert query ? or i can also use write delta format options? If i use the df.write ...

Data Engineering
Reply
  • 1775 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @Menaka Murugesan​(Customer)​, We haven’t heard from you since the last response from @Nandini N​  (Customer)​, and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as ...

  • 1 kudos
1 More Replies
bluesky111
by • New Contributor II
  • 1295 Views
  • 2 replies
  • 3 kudos

Resolved! I Input the wrong schedule time for the exams can it be reschedule ?

Helo today ,i think i was scheduled to do an exams at 2.15 PM but unfortunately i made a mistake put the time to 2.15 AM, could it be rescheduled? i already submit a ticket to https://help.databricks.com/s/contact-us?ReqType=training but no reply yet...

Data Engineering
Reply
  • 1295 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 3 kudos

Hi @heron halim​ (Customer)​​, We haven't heard from you since the last response from @Akshay Padmanabhan​​, and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please share it with the community, as it ca...

  • 3 kudos
1 More Replies
Philearner
by • New Contributor II
  • 1709 Views
  • 3 replies
  • 3 kudos

Unable to find input by typing input in the Multiselect Widget

In the AWS databricks widgets.multiselect, I'm unable to find input by typing input in the mulitselect bar. It was working before. Although I can find the inputs by scrolling down the list, it's annoying if the list is long.​​Here's my script:measlis...

databrick widget problem databrick widget problem 2
Data Engineering
Reply
  • 1709 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Philip Teu​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 3 kudos
2 More Replies
Sas
by • New Contributor II
  • 2901 Views
  • 3 replies
  • 4 kudos

Resolved! Confusion in string comparison

Hello expertI am new to spark. I am using same price of code but getting different resultsWhen i am using below piece of code, i am getting errorpy4j.Py4JException: Method or([class java.lang.String]) does not existdf.filter(F.col("state").isNull()  ...

Data Engineering
Reply
  • 2901 Views
  • 3 replies
  • 4 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Saswata Dutta​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedbac...

  • 4 kudos
2 More Replies
sagiatul
by • New Contributor II
  • 3538 Views
  • 2 replies
  • 3 kudos

Databricks driver logs

I am running jobs on databricks clusters. When the cluster is running I am able to find the executor logs by going to Spark Cluster UI Master dropdown, selecting a worker and going through the stderr logs. However, once the job is finished and cluste...

image
Data Engineering
Reply
  • 3538 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Atul Arora​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback w...

  • 3 kudos
1 More Replies
saikrishna3390
by • New Contributor II
  • 5163 Views
  • 2 replies
  • 2 kudos

How do I configure managed identity to databricks cluster and access azure storage using spark config

Partner want to use adf managed identity to connect to my databricks cluster and connect to my azure storage and copy the data from my azure storage to their azure storage storage

Data Engineering
Reply
  • 5163 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @SAI PUSALA​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback w...

  • 2 kudos
1 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels
Top Kudoed Authors
View all