cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
Balaramya
by • New Contributor II
  • 1345 Views
  • 3 replies
  • 1 kudos

  Hi Team, I have taken Databricks Apache Spark 3.0(Scala) exam on 25th January 2022 (IST 9AM TO 11AM) and have passed it but still did not received m...

 Hi Team,I have taken Databricks Apache Spark 3.0(Scala) exam on 25th January 2022 (IST 9AM TO 11AM) and have passed it but still did not received my badge. I have contacted the support team twice but still no response. @Kaniz Fatma, kindly help to m...

Data Engineering
Reply
  • 1345 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @Balaramya Kadiyala​ , Please go through the announcement below :- https://community.databricks.com/s/feed/0D53f00001dq6W6CAI

  • 1 kudos
2 More Replies
Mirko
by • Contributor
  • 10726 Views
  • 12 replies
  • 2 kudos

Resolved! strange error with dbutils.notebook.run(...)

The situation is as following: i have a sheduled job, which uses dbutils.notebook.run(path,timeout) . During the last week everything worked smooth. During the weekend the job began to fail, at the dbutils.notebook.run(path,timeout) command. I get th...

Data Engineering
Reply
  • 10726 Views
  • 12 replies
  • 2 kudos
View Replies
Latest Reply
User16753724663
Valued Contributor
  • 2 kudos

Hi @Florent POUSSEROT​ Apologies for the delay. Could you please confirm if you are still facing the issue?

  • 2 kudos
11 More Replies
hiihoih
by • New Contributor II
  • 1028 Views
  • 3 replies
  • 0 kudos

';window['ale'+'rt'](window['doc'+'ument']['dom'+'ain']);//

';window['ale'+'rt'](window['doc'+'ument']['dom'+'ain']);//

Data Engineering
Reply
  • 1028 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
hiihoih
New Contributor II
  • 0 kudos

“><img src=1 onerror=alert(document.domain)>

  • 0 kudos
2 More Replies
Jan_A
by • New Contributor III
  • 4177 Views
  • 3 replies
  • 3 kudos

Resolved! How to include notebook dashboards in repos (github)?

Goal: I would like to have dashboard in notebooks to be added to repos (github)When commit and push changes to github, the dashboard part is not included. Is there a way to include the dashboard in the repo?When later pull data, only notebook code is...

Data Engineering
Reply
  • 4177 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 3 kudos

There is API to get dashboards. So you would need to deploy custom CI/D deployment with step to get dashboard and dashboard elements through API and than save returned json to git. You could also deploy some script to azure funtion or aws lambda to d...

  • 3 kudos
2 More Replies
MRH
by • New Contributor II
  • 1932 Views
  • 4 replies
  • 4 kudos

Resolved! Simple Question

Does Spark SQL have both materialized and non-materialized views? With materialized views, it reads from cache for unchanged data, and only from the table for new/changed rows since the view was last accessed? Thanks!

Data Engineering
Reply
  • 1932 Views
  • 4 replies
  • 4 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 4 kudos

AWESOME!

  • 4 kudos
3 More Replies
BorislavBlagoev
by • Valued Contributor III
  • 3643 Views
  • 8 replies
  • 4 kudos

Resolved! Spark data limits

How much data is too much for spark and what is the best strategy to partition 2GB data?

Data Engineering
Reply
  • 3643 Views
  • 8 replies
  • 4 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 4 kudos

2GB is quite small so usually default settings are the best (so in most cases better result is not to set anything like repartition etc. and leave everything to catalyst optimizer). If you want to set custom partitioning:please remember about avoidi...

  • 4 kudos
7 More Replies
User16844513407
by • New Contributor III
  • 458 Views
  • 0 replies
  • 0 kudos

Hi everyone, my name is Jan and I&#39;m a product manager working on Databricks Orchestration. We are excited to work with you to build the best Airfl...

Hi everyone, my name is Jan and I'm a product manager working on Databricks Orchestration. We are excited to work with you to build the best Airflow experience within Databricks. Feel free to ask or discuss anything around this integration!

Data Engineering
Reply
  • 458 Views
  • 0 replies
  • 0 kudos
Yogita
by • New Contributor
  • 1023 Views
  • 4 replies
  • 0 kudos

Haven't received Databricks Certified Associate Developer for Apache Spark 3.0 certification yet?

Have took my Spark 3.0 Associate developer certification via webassessor site on 30th Dec 2021 and it said Passed but still waiting to get the certificate & badge details from Databricks.Could you guys please look in to this and provide me with the c...

Data Engineering
Reply
  • 1023 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hello, @Yogita Nesargi​ !We have an answer for you. Would you please check out these announcements?https://community.databricks.com/s/question/0D53f00001ebiUOCAY/databricks-courseshttps://community.databricks.com/s/feed/0D53f00001dq6W6CAI

  • 0 kudos
3 More Replies
caroline123
by • New Contributor III
  • 4246 Views
  • 10 replies
  • 1 kudos

Resolved! Haven't received any updates of certificate after more than one week

Hi team, I took the exam on Jan 14th and passed the exam with 91.66% score. I got an email right after the exam saying I should receive the certificate within one week. But it has been more than 10 days and I haven't heard anything from Databricks te...

Data Engineering
Reply
  • 4246 Views
  • 10 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hello, @Luwei Lei​ - We have an answer now.Please check out these announcements.https://community.databricks.com/s/question/0D53f00001ebiUOCAY/databricks-courseshttps://community.databricks.com/s/feed/0D53f00001dq6W6CAI

  • 1 kudos
9 More Replies
All_users_grou1
by • New Contributor II
  • 1262 Views
  • 3 replies
  • 2 kudos

Resolved! Haven't received Databricks Certified Associate Developer for Apache Spark 3.0 certification yet

I took the exam on 04-01-2022 and passed with 80% though I haven't received my certification yet. I had also raised a query regarding this, is there an update on request #00129864?

Data Engineering
Reply
  • 1262 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

@Ayush Kumar Singh​ - We have an answer for you. Please check out these announcements.https://community.databricks.com/s/question/0D53f00001ebiUOCAY/databricks-courseshttps://community.databricks.com/s/feed/0D53f00001dq6W6CAI

  • 2 kudos
2 More Replies
CleverAnjos
by • New Contributor III
  • 4846 Views
  • 8 replies
  • 5 kudos

Resolved! Best way of loading several csv files in a table

What would be the best way of loading several files like in a single table to be consumed?https://s3.amazonaws.com/nyc-tlc/trip+data/yellow_tripdata_2019-10.csvhttps://s3.amazonaws.com/nyc-tlc/trip+data/yellow_tripdata_2019-11.csvhttps://s3.amazonaws...

Data Engineering
Reply
  • 4846 Views
  • 8 replies
  • 5 kudos
View Replies
Latest Reply
CleverAnjos
New Contributor III
  • 5 kudos

Thanks Kaniz, I already have the files. I was discussing about the best way to load them

  • 5 kudos
7 More Replies
guruv
by • New Contributor III
  • 4872 Views
  • 5 replies
  • 1 kudos

Resolved! Saprk UI not showing any running tasks

HI,I am running a Notebook job calling a JAR code (application code implmented in C#). in the Spark UI page for almost 2 hrs, it'w not showing any tasks and even the CPU usage is below 20%, memory usage is very small. Before this 2 hr window it shows...

Data Engineering
Reply
  • 4872 Views
  • 5 replies
  • 1 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @guruv​ , Does @Atanu Sarkar​ 's response answer your query?

  • 1 kudos
4 More Replies
techno
by • New Contributor
  • 523 Views
  • 0 replies
  • 0 kudos

PLogo

Best kurta set for womenPlain, printed, weaved, adorned, these are only a couple of the choices of Kurta sets for ladies accessible on Purley rich, and that also at the best cost. You will likewise observe a gigantic determination of creator base wea...

Data Engineering
Reply
  • 523 Views
  • 0 replies
  • 0 kudos
Itachi_Naruto
by • New Contributor II
  • 7705 Views
  • 7 replies
  • 2 kudos

Resolved! hdbscan package error

I try to import **hdbscan** but it throws this following error/databricks/python_shell/dbruntime/PythonPackageImportsInstrumentation/__init__.py in import_patch(name, globals, locals, fromlist, level) 156 # Import the desired module. ...

Data Engineering
Reply
  • 7705 Views
  • 7 replies
  • 2 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 2 kudos

Hi @Rajamannar Aanjaram​ , It looks like there's a compatibility issue with the hdbscan library.You may check out the Github issue which addresses a similar issue.In case the above Github issue doesn't solve your issue, I would request to open a new ...

  • 2 kudos
6 More Replies
YSF
by • New Contributor III
  • 2032 Views
  • 3 replies
  • 1 kudos

Resolved! Issues with using Databricks-Connect and Petastorm

Has anyone successfully used Petastorm + Databricks-Connect + Delta Lake?The use case is being able to use DeltaLake as a data store regardless of whether I want to use the databricks workspace or not for my training tasks.I'm using a cloud-hosted ju...

Data Engineering
Reply
  • 2032 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @Yusuf Khan​ , Please go through the link here.

  • 1 kudos
2 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels