cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Odoo_ERP
by New Contributor II
  • 2340 Views
  • 0 replies
  • 1 kudos

Odoo ERP customization Odoo is one of the most popular ERP software. It is widely use by companies. Odoo customization mainly includes changing the sy...

Odoo ERP customizationOdoo is one of the most popular ERP software. It is widely use by companies. Odoo customization mainly includes changing the system by including new features and functionalities in accordance with the business needs of the clien...

  • 2340 Views
  • 0 replies
  • 1 kudos
Pat
by Honored Contributor III
  • 993 Views
  • 0 replies
  • 14 kudos

Automating DDLs and Privilege management

How do you automate table creation and management of privilege / grants on securable objects (tables, views, etc.).I had idea to use terraform to manage above, but terraform doesn't work with managed tables - it allows to create table but then there ...

  • 993 Views
  • 0 replies
  • 14 kudos
elgeo
by Valued Contributor II
  • 1332 Views
  • 1 replies
  • 2 kudos

Databricks - DBeaver error

Hello experts. While trying to setup the connection between DBeaver and Databricks, we receive the following error:[Databricks][DatabricksJDBCDriver](700120) Host adb-xxxxxxxxxx.azuredatabricks.net cannot be resolved through DnsResolver com.databrick...

  • 1332 Views
  • 1 replies
  • 2 kudos
Latest Reply
elgeo
Valued Contributor II
  • 2 kudos

We identified the problem. It was due to the proxy that the hostname couldn't be resolved.

  • 2 kudos
Anonymous
by Not applicable
  • 9100 Views
  • 4 replies
  • 17 kudos

Resolved! SQL query takes too long to run

Hi all,I have a set up SQL query run with 5 hours but the SQL endpoint take too long to start up with each run. Currently I don't know how to fix this Could you please help me how to improve this?

  • 9100 Views
  • 4 replies
  • 17 kudos
Latest Reply
Unforgiven
Valued Contributor III
  • 17 kudos

or refer on this : Faster SQL Queries on Delta Lake with Dynamic File Pruninghttps://www.databricks.com/blog/2020/04/30/faster-sql-queries-on-delta-lake-with-dynamic-file-pruning.html

  • 17 kudos
3 More Replies
lawrence009
by Contributor
  • 2509 Views
  • 4 replies
  • 8 kudos

Photon does not fully support the query because of dynamic pruning

Does it still make sense to run this job on a cluster with Photon enable when I am receiving the following?This is the code I ran:CREATE OR REPLACE TABLE ${tbl_name}_dups SELECT src.*, ROW_NUMBER() OVER ( PARTITION BY src.id ...

  • 2509 Views
  • 4 replies
  • 8 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 8 kudos

Hi @Lawrence Chen​, Please tell us your DBR version.

  • 8 kudos
3 More Replies
farbodr
by New Contributor II
  • 3730 Views
  • 4 replies
  • 1 kudos

Shapley Progressbar

The shapley progress bar or tqdm progress bar in general doesn't show in notebooks. Do I need to set something special to get this or any other similar widgets to work?

  • 3730 Views
  • 4 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Fred Rahmanian​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 1 kudos
3 More Replies
638555
by New Contributor III
  • 4131 Views
  • 4 replies
  • 8 kudos

Resolved! Auto termination for clusters, jobs, and delta live tables does not terminate clusters on GCP.

Hello,I am new to Databricks, and I have been trying to understand how auto termination works, but I am unsure if it is a problem based on my configuration or something else. The problem is the same in all cases, the cluster on GCP created by Databri...

  • 4131 Views
  • 4 replies
  • 8 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 8 kudos

Hi @Tilemachos Charalampous​ , We haven’t heard from you since the last response from @Landan George​, and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please share it with the community, as it can be h...

  • 8 kudos
3 More Replies
Mado
by Valued Contributor II
  • 898 Views
  • 0 replies
  • 0 kudos

How to change branch without using UI?

Hi, I want to change the branch in Databricks Repo. When I click on the current branch name, there is a message that: As you can see in the above image, I cannot see the list of branches to select. In fact, I selected a branch by mistake and I am not...

Change_Branch
  • 898 Views
  • 0 replies
  • 0 kudos
xiangzhu
by Contributor III
  • 3529 Views
  • 3 replies
  • 2 kudos

Could jobs do everything delta live tables do ?

Hello,I've read the posts:Jobs - Delta Live tables difference (databricks.com)andDifference between Delta Live Tables and Multitask Jobs (databricks.com)My understanding is that delta live tables are more like a DSL that simplfies the workflow defini...

  • 3529 Views
  • 3 replies
  • 2 kudos
Latest Reply
xiangzhu
Contributor III
  • 2 kudos

@Landan George​ "Jobs won't be able to do what DLT does", I read some blogs, and watched some videos too, but I still cannot figure out the difference between jobs vs DLT. Does it mean without Databricks DLT, Databricks jobs cannot handle delta table...

  • 2 kudos
2 More Replies
apayne
by New Contributor III
  • 664 Views
  • 0 replies
  • 0 kudos

Python Databricks SQL Connector vs Databricks Connect?

Connecting several Databricks tables to a Plotly Dash application. Can't seem to find much documentation on the differences between SQL Connector and Connect. Currently using the Connect approach to read tables into pyspark dataframes, is one better ...

  • 664 Views
  • 0 replies
  • 0 kudos
Deiry
by New Contributor III
  • 854 Views
  • 0 replies
  • 0 kudos

Why is the whole list not displayed in dbutil.widgets.multiselect?

I have been studying the Apache Spark in Databricks Academy and I don't understand why the whole list is nos displayed? Creation of widgets:dbutils.widgets.text("name", "Brickster", "Name") dbutils.widgets.multiselect("colors","orange", ["orange", "r...

image
  • 854 Views
  • 0 replies
  • 0 kudos
Kash
by Contributor III
  • 2441 Views
  • 5 replies
  • 0 kudos

Cannot install TA-LIB via cluster libraries

Hi there,I can't seem to find a way to install Ta-lib on a databricks server. I can manually install it in the notebook using the code below but if we detach the notebook then I have to install it again. Please let me know if you've found a fix for t...

  • 2441 Views
  • 5 replies
  • 0 kudos
Latest Reply
Kash
Contributor III
  • 0 kudos

Hi Vivian,Thanks for the note. I tried that and it shows the package was installed but when I try to import talib in the notebook it says no module names 'talib'Any thoughts on what might be happening?Thanks,K

  • 0 kudos
4 More Replies
Kavin
by New Contributor II
  • 1541 Views
  • 2 replies
  • 2 kudos

Issue converting the datasets into JSON

Im a newbie to Databricks, I need to convert the data sets into JSON. i tried bth FOR JSON AUTO AND FOR JSON PATH, However im getting an issue - [PARSE_SYNTAX_ERROR] Syntax error at or near 'json'line My Query works fine without FOR JSON AUTO AND FOR...

  • 1541 Views
  • 2 replies
  • 2 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 2 kudos

Hi @Kavin Natarajan​, We haven’t heard from you since the last response from @Debayan Mukherjee​​, and I was checking back to see if their suggestions helped you. Or else, If you have any solution, please share it with the community, as it can be hel...

  • 2 kudos
1 More Replies
Bujji
by New Contributor II
  • 4349 Views
  • 2 replies
  • 3 kudos

How to resolve our of memory error?

Hi, I am working as azure support engineerI found this error while I am checking the pipeline failure, and showing below error"org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 72403.0 failed 4 times, most recent fail...

  • 4349 Views
  • 2 replies
  • 3 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 3 kudos

Hi @mahesh bmk​, We haven’t heard from you since the last response from @Pat Sienkiewicz​​, and I was checking back to see if their suggestions helped you. Or else, If you have any solution, please share it with the community, as it can be helpful to...

  • 3 kudos
1 More Replies
natadhorcross
by New Contributor III
  • 1090 Views
  • 1 replies
  • 4 kudos

Hi, we encountered a problem of timeout every (60 / 80 minutes ) on a long execution of copy json into parquet file in datalake Gen 2.

First, our process was triggered by the datafactory. First the connexion was set with token access, then with managed service identity.We prove the untimely time out was not due to the datafactory by running directly the notebook. Secondly, we tried ...

  • 1090 Views
  • 1 replies
  • 4 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 4 kudos

Hi @Natacha D'hour​, Does this thread help resolve your question?

  • 4 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels