cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Barb
by New Contributor III
  • 3607 Views
  • 6 replies
  • 0 kudos

How do I get my notebooks displayed in the order that they were last modified

I use databricks for analytic work, and I have a lot of notebooks looking at different things. Right now they are listed in alphabetic order by the name of the notebook. How do I make the order display the most recently created/modified notebooks at...

  • 3607 Views
  • 6 replies
  • 0 kudos
Latest Reply
Vidula
Honored Contributor
  • 0 kudos

Hi @Barb Krienke​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...

  • 0 kudos
5 More Replies
159312
by New Contributor III
  • 2213 Views
  • 4 replies
  • 2 kudos

Access workflow settings from within a notebook.

I have a notebook used for a dlt pipeline. The pipeline should perform an extra task if the pipeline is run as a full refresh. Right now, I have to set an extra configuration parameter when I run a full refresh. Is there a way to programmatically...

  • 2213 Views
  • 4 replies
  • 2 kudos
Latest Reply
Vidula
Honored Contributor
  • 2 kudos

Hi @Ben Bogart​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 2 kudos
3 More Replies
Newyn
by New Contributor III
  • 3839 Views
  • 5 replies
  • 6 kudos

Resolved! Where does Python "import" statement in a Notebook search (on Azure) for libraries?

I know that import can find installed cluster libraries, workspace libraries etc. as well as how to create wheel libraries externally and upload them.However, in a project I came across the feature that import was able to find the folder in, if I rem...

  • 3839 Views
  • 5 replies
  • 6 kudos
Latest Reply
Vidula
Honored Contributor
  • 6 kudos

Hi @Jonas Mellin​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...

  • 6 kudos
4 More Replies
Vashista
by New Contributor II
  • 3280 Views
  • 4 replies
  • 0 kudos

Blank page after logging into community cloud

After logging in on community.cloud.databricks.com. A blank white page appears as if still loading. Does not open even when I refresh. Not an internet connectivity issue. How do I access the page ? I'm running on safari

  • 3280 Views
  • 4 replies
  • 0 kudos
Latest Reply
Vidula
Honored Contributor
  • 0 kudos

Hi @Vashista Thakuri​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.T...

  • 0 kudos
3 More Replies
949991
by New Contributor
  • 1212 Views
  • 1 replies
  • 0 kudos

"assertion failed: Invalid batch"

My autoloader based streaming use microbatch process. I can not pass the following error. "java.lang.AssertionError: assertion failed: Invalid batch: payrollOutputs#217439260,meta#217439261,_metadata#217439267 != payrollOutputs#217484957,meta#2174849...

  • 1212 Views
  • 1 replies
  • 0 kudos
Latest Reply
Vidula
Honored Contributor
  • 0 kudos

Hi @Julian.He​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 0 kudos
Sneha1594
by New Contributor III
  • 3712 Views
  • 6 replies
  • 4 kudos

Databricks job/task shows success but original notebook is not updated

I have uploaded parquet files to hive meta store tables then performed some transformations on data and generated some visualizations. All this is done in a notebook. I have scheduled the notebook for every morning so that I get a refreshed view of d...

  • 3712 Views
  • 6 replies
  • 4 kudos
Latest Reply
Vidula
Honored Contributor
  • 4 kudos

Hi @Sneha Mulrajani​ Does @Prabakar Ammeappin​  response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 4 kudos
5 More Replies
Christine
by Contributor II
  • 7612 Views
  • 7 replies
  • 4 kudos

Resolved! autoML' is not found when using databricks.automl with runtime 112.ML (and runtime 10.4 LTS ML).

I have tried to set up a autoML experiment with runtime 11.2ML and data from a delta table. However I receive the error "ModuleNotFoundError: No module named 'databricks.automl'" and "AutoML not available: Use Databricks Runtime 8.3 ML or above." tho...

image
  • 7612 Views
  • 7 replies
  • 4 kudos
Latest Reply
Christine
Contributor II
  • 4 kudos

I deleted the cluster and created a new with runtime 9.1 LTS ML which solved the problem.

  • 4 kudos
6 More Replies
Siva_K
by New Contributor II
  • 1749 Views
  • 2 replies
  • 2 kudos

While exporting Dashboard as HTML displaying image as blank (during Job Run).

During Job run, am trying to convert the dashboard to HTML form. Images in notebook are displayed as blank. am able to see the images displayed in Result only or Dashboard mode. Only during export to HTML, there is an issue with images. Could you p...

  • 1749 Views
  • 2 replies
  • 2 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 2 kudos

Hi @Siva K​,Just a friendly follow-up. Did you have time to check Atanu's docs? did it help to resolve your question? please let us know if you need any help.

  • 2 kudos
1 More Replies
pjp94
by Contributor
  • 2618 Views
  • 1 replies
  • 0 kudos

ERROR - Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.

I get the below error when trying to run multi-threading - fails towards the end of the run. My guess is it's related to memory/worker config. I've seen some solutions involving modifying the number of workers or CPU on the cluster - however that's n...

  • 2618 Views
  • 1 replies
  • 0 kudos
Latest Reply
pjp94
Contributor
  • 0 kudos

Since I don't have permissions to change cluster configurations, the only solution that ended up working was setting a max thread count to about half of the actual max so I don't overload the containers. However, open to any other optimization ideas!

  • 0 kudos
avnerrhh
by New Contributor III
  • 4390 Views
  • 6 replies
  • 4 kudos

Resolved! How do I import class/functions so it work in Databricks and in my IDE

I already saw this postI want my code to work on both platforms (Databricks and PyCharm), is there any way to do it?

  • 4390 Views
  • 6 replies
  • 4 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 4 kudos

yes.one way is to develop everything locally on your pc, so you also need to have spark installed.This is of course not ideal as you will not have some interesting stuff that databricks provides.But it can be done. What you have to do is create a whl...

  • 4 kudos
5 More Replies
martikev
by New Contributor II
  • 1426 Views
  • 2 replies
  • 1 kudos

How to avoid losing dbutils.widgets parameters when connecting databricks repos folder with azure dev ops?

Hi guys We want to move from databricks workspace to databricks repos. Now when we create a new folder under repos and connect to our azure dev ops repository all our dbutils.widget parameters are lost. How to avoid this? We want to fully depend on a...

  • 1426 Views
  • 2 replies
  • 1 kudos
Latest Reply
Vidula
Honored Contributor
  • 1 kudos

Hi @Kevin Peter M. Marti​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from y...

  • 1 kudos
1 More Replies
Shubhamgoyal
by New Contributor
  • 1765 Views
  • 2 replies
  • 1 kudos

Access Databricks SQL databases using rest API

Hi All,We want to read/write data to Databricks SQL using powerapps. I have been looking for documentation around accessing databases in databricks SQL via rest api.Appreciate your help on this.

  • 1765 Views
  • 2 replies
  • 1 kudos
Latest Reply
byrdman
New Contributor III
  • 1 kudos

With the databricks api you can start a workflow job. build the job to ingest your data into tables.

  • 1 kudos
1 More Replies
turagittech
by New Contributor
  • 6862 Views
  • 2 replies
  • 1 kudos

PYODBC very slow - 30 minutes to write 6000 rows

Along withh several other issues I'm encountering, I am finding pandas dataframe to_sql being very slowI am writing to an Azure SQL database and performance is woeful. This is a test database and it has S3 100DTU and one user, me as it's configuratio...

  • 6862 Views
  • 2 replies
  • 1 kudos
Latest Reply
Vidula
Honored Contributor
  • 1 kudos

Hi @Peter McLarty​ Does @Debayan Mukherjee​  response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 1 kudos
1 More Replies
dibo
by New Contributor II
  • 631 Views
  • 0 replies
  • 0 kudos

I can't login to https://community.cloud.databricks.com/login.html

Now, I can't login to https://community.cloud.databricks.com/login.html with the correct username and password, later I click the button to reset my password and I receive the email for modifying password, I have modified password, But I still can't ...

  • 631 Views
  • 0 replies
  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels