cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
uchombo
by • New Contributor II
  • 5145 Views
  • 3 replies
  • 1 kudos

Resolved! Alerts Notification Fail

Hi,I set up an alert and apparently it worked because status changes according to the Trigger condition. However I don't received any notification on my email, or Teams channel. Any idea what is happening?Thanks

Data Engineering
Reply
  • 5145 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @uriel chombo​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...

  • 1 kudos
2 More Replies
MichelSmits
by • New Contributor III
  • 7743 Views
  • 4 replies
  • 3 kudos

Connection to on-prem database occasionally goes to the wrong server

We run a dbx job that uses a jdbc connection to three (SQL Server) on-prem databases. Very occasionally a connection will go to the wrong server.In the job log we see the error "com.microsoft.sqlserver.jdbc.SQLServerException: Login failed for user '...

Data Engineering
Reply
  • 7743 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Michel Smits​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...

  • 3 kudos
3 More Replies
Anonymous
by • Not applicable
  • 10333 Views
  • 3 replies
  • 0 kudos

Resolved! What is the best practice for managing different environments (Staging vs Production) on Databricks?

Should we create separate workspaces for Dev/Test/Prod ? Or should we have 1 workspace and create separate folders for Dev/Test/Prod?

Data Engineering
Reply
  • 10333 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Srikanth_Gupta_
Databricks Employee
  • 0 kudos

as per my previous experience, its always good to have different workspaces for different envs, its easy to maintain and helps better with CICD pipeline as well, because lot of organizations provide deployment access to Developers in Dev env but not ...

  • 0 kudos
2 More Replies
seefoods
by • New Contributor III
  • 1590 Views
  • 1 replies
  • 0 kudos

run jar file into databricks

I have created a job which run a jar files, but i have this error NoClassDefFoundError: com/google/cloud/hadoop/gcsio/GoogleCloudStorageFileSystemOptions$TimestampUpdatePredicateCaused by: ClassNotFoundException: com.google.cloud.hadoop.gcsio.GoogleC...

Data Engineering
Reply
  • 1590 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
aiNdata
New Contributor II
  • 0 kudos

Hey Aubert, seems you are missing dependent class in jar. Either package the dependent classes in jar or add them into class path.

  • 0 kudos
Teja07
by • New Contributor II
  • 7659 Views
  • 2 replies
  • 0 kudos

File copy from local to dbfs

How to copy a file from local disk to databricks dbfs path. I tried as below but it is throwing me error:code i tried: dbutils.fs.cp("file://c:/user/file.txt",dbfs/data/) and dbutils.fs.cp("file:///c:/user/file.txt",dbfs/data/) error: File not found ...

Data Engineering
Reply
  • 7659 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
venkatcrc
New Contributor III
  • 0 kudos

I assume you cannot copy files from Local machine to dbfs using dbutils. you can upload files to dbfs using below gui option . Data --> Browse DFS --> Upload  

  • 0 kudos
1 More Replies
az38
by • New Contributor II
  • 7226 Views
  • 2 replies
  • 3 kudos

load files filtered by last_modified in PySpark

Hi, community!How do you think what is the best way to load from Azure ADLS (actually, filesystem doesn't matter) into df onli files modified after some point in time?Is there any function like input_file_name() but for last_modified to use it in a w...

Data Engineering
Reply
  • 7226 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
venkatcrc
New Contributor III
  • 3 kudos

_metadata will provide file modification timestamp. I tried on dbfs but not sure for ADLS.https://docs.databricks.com/ingestion/file-metadata-column.html

  • 3 kudos
1 More Replies
matt_at_ark
by • New Contributor
  • 1661 Views
  • 0 replies
  • 0 kudos

Bokeh plotting with ipywidgets or in Databricks Dashboard

Databricks' documentation for using bokeh shows using displayHTML, which seems to be incompatible with using ipywidgets, or putting them in Databricks dashboards.I found there is undocumented functionality with plotting interactive altair charts in D...

Data Engineering
Reply
  • 1661 Views
  • 0 replies
  • 0 kudos
William_Scardua
by • Valued Contributor
  • 5788 Views
  • 3 replies
  • 1 kudos

Resolved! How to integrate pipeline with Dynatrace ?

Hi guys,Do you know how I integrate pipeline some data to Dynatrace ?Have any idea ?Thank you

Data Engineering
Reply
  • 5788 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @William Scardua​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answe...

  • 1 kudos
2 More Replies
sudhanshu1
by • New Contributor III
  • 1440 Views
  • 0 replies
  • 0 kudos

SCD Type-2

Hi All, I have 22 postgress tables and i need to implement SCD type 2 and create azure Databricks pipeline . However my project team doesn't want to use delta tables concept . Have anyone implemented this ? below is how i planned to do try: df_src = ...

Data Engineering
Reply
  • 1440 Views
  • 0 replies
  • 0 kudos
Aviral-Bhardwaj
by • Esteemed Contributor III
  • 9560 Views
  • 2 replies
  • 2 kudos

Resolved! can anyone help with Spill Question

Spill occurs as a result of executing various wide transformations. However, diagnosing a spill requires one to proactively look for key indicators.Where in the Spark UI are two of the primary indicators that a partition is spilling to disk?a-   Exec...

Data Engineering
Reply
  • 9560 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
pvignesh92
Honored Contributor
  • 2 kudos

@Aviral Bhardwaj​  I feel it is Option e. Stage and executor log files. Consolidated details at the Stage LevelDetails at the task and Executor Level Please let me know if you feel any other option is better.

  • 2 kudos
1 More Replies
pvignesh92
by • Honored Contributor
  • 1717 Views
  • 1 replies
  • 3 kudos

lnkd.in

Databricks Auto Loader is an interesting feature that can be used to load data incrementally.✳ It can process new data files as they arrive in the cloud object stores✳ It can be used to ingest JSON, CSV, PARQUET, AVRO, ORC, TEXT and even Binary file ...

Data Engineering
Reply
  • 1717 Views
  • 1 replies
  • 3 kudos
View Replies
Latest Reply
Ajay-Pandey
Esteemed Contributor III
  • 3 kudos

Thanks for sharing

  • 3 kudos
Kevin_Kang
by • New Contributor III
  • 2493 Views
  • 3 replies
  • 0 kudos

Resolved! Account Registration Help for Partner-Academy

Hi there,I used my company email to register an account for customer-academy.databricks.com a while ago. And I need to create an account in the partner-academy.databricks.com using the same company email.However, when I tried for partner-academy, it ...

Data Engineering
Reply
  • 2493 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Geunhee Kang​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...

  • 0 kudos
2 More Replies
DenisMcd
by • New Contributor
  • 2627 Views
  • 2 replies
  • 0 kudos

"Endpoint not found for /2.0/sql/statements/"

Hey everyone! I´m trying to access table row using databricks api. Using Insomnia or postman, to test and the error are the same: { "error_code": "ENDPOINT_NOT_FOUND", "message": "Endpoint not found for /2.0/sql/statements/"}Below is my request:(for ...

Data Engineering
Reply
  • 2627 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Denis Macedo​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...

  • 0 kudos
1 More Replies
Papa_PSL
by • New Contributor
  • 1725 Views
  • 2 replies
  • 1 kudos

I want to import a notebook

Want to import a note book, facing this following error:Importation fails: could not deserialize: Unkown language htm

Data Engineering
Reply
  • 1725 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @BITAIGOT Guy-Serge​ We haven't heard from you since the last response from @Debayan Mukherjee​ â€‹, and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be hel...

  • 1 kudos
1 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels