cancel
Turn on suggestions
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
xiangzhu
by • Contributor III
  • 5177 Views
  • 3 replies
  • 2 kudos

Could jobs do everything delta live tables do ?

Hello,I've read the posts:Jobs - Delta Live tables difference (databricks.com)andDifference between Delta Live Tables and Multitask Jobs (databricks.com)My understanding is that delta live tables are more like a DSL that simplfies the workflow defini...

Data Engineering
Reply
  • 5177 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
xiangzhu
Contributor III
  • 2 kudos

@Landan George​ "Jobs won't be able to do what DLT does", I read some blogs, and watched some videos too, but I still cannot figure out the difference between jobs vs DLT. Does it mean without Databricks DLT, Databricks jobs cannot handle delta table...

  • 2 kudos
2 More Replies
apayne
by • New Contributor III
  • 967 Views
  • 0 replies
  • 0 kudos

Python Databricks SQL Connector vs Databricks Connect?

Connecting several Databricks tables to a Plotly Dash application. Can't seem to find much documentation on the differences between SQL Connector and Connect. Currently using the Connect approach to read tables into pyspark dataframes, is one better ...

Data Engineering
Reply
  • 967 Views
  • 0 replies
  • 0 kudos
Deiry
by • New Contributor III
  • 1183 Views
  • 0 replies
  • 0 kudos

Why is the whole list not displayed in dbutil.widgets.multiselect?

I have been studying the Apache Spark in Databricks Academy and I don't understand why the whole list is nos displayed? Creation of widgets:dbutils.widgets.text("name", "Brickster", "Name") dbutils.widgets.multiselect("colors","orange", ["orange", "r...

image
Data Engineering
Reply
  • 1183 Views
  • 0 replies
  • 0 kudos
Kash
by • Contributor III
  • 3302 Views
  • 5 replies
  • 0 kudos

Cannot install TA-LIB via cluster libraries

Hi there,I can't seem to find a way to install Ta-lib on a databricks server. I can manually install it in the notebook using the code below but if we detach the notebook then I have to install it again. Please let me know if you've found a fix for t...

Data Engineering
Reply
  • 3302 Views
  • 5 replies
  • 0 kudos
View Replies
Latest Reply
Kash
Contributor III
  • 0 kudos

Hi Vivian,Thanks for the note. I tried that and it shows the package was installed but when I try to import talib in the notebook it says no module names 'talib'Any thoughts on what might be happening?Thanks,K

  • 0 kudos
4 More Replies
huy_le
by • New Contributor II
  • 869 Views
  • 0 replies
  • 1 kudos

How to Update openapi specs ?

I found a few errors/differences between https://docs.databricks.com/_static/api-refs/permissions-2.0-aws.yaml and the actual api. Where can we update this specs for everyone because engineers are using this specs to generate their clients

Data Engineering
Reply
  • 869 Views
  • 0 replies
  • 1 kudos
mickniz
by • Contributor
  • 2964 Views
  • 3 replies
  • 3 kudos

Dbx installation for local development on Vscode

Hi Folks,Since databricks is now asking to use DBX instead of databricks-connect ,We are trying to set up our local environment following the guide.dbx by Databricks Labs - Azure Databricks | Microsoft LearnHave create conf/deployment.yml and dbx/pro...

Data Engineering
Reply
  • 2964 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
mickniz
Contributor
  • 3 kudos

fixed this issue. but I am getting another issue while syncing local repo with Workspace in Databricks UI.When I run commanddbx sync repo -d workspace name --source.Command runs fine . I can see that dbfs but not under workspace in Databricks page.An...

  • 3 kudos
2 More Replies
rishabh4312
by • Contributor II
  • 7782 Views
  • 15 replies
  • 47 kudos

Voucher code error

Hi,I received a voucher in Nov 2020 for 'Databricks Certified Associate Developer for Apache Spark 3.0 exam' with an expiry date on 10th Nov 2022. However I receive an error stating the promotion code has been used. I have never used the code. Please...

Data Engineering
Reply
  • 7782 Views
  • 15 replies
  • 47 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 47 kudos

Hi @Rishabh Jain​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...

  • 47 kudos
14 More Replies
Anonymous
by • Not applicable
  • 1276 Views
  • 1 replies
  • 11 kudos

Explain about "Activate Gift Certificate" section in Databricks Community Rewards.

Hi every,Could some one explain about the Active Gift Certificate in  in Databricks Community Rewards. And how to use it?Thanks vere much 

Image
Data Engineering
Reply
  • 1276 Views
  • 1 replies
  • 11 kudos
View Replies
Latest Reply
Unforgiven
Valued Contributor III
  • 11 kudos

@Jensen Ackles​ give ur info i had try to this function does not working. I used cert id lakehouse input into box active gift certificate not help for claim 100points to rewards point in my account

  • 11 kudos
B_Seibert
by • New Contributor III
  • 2490 Views
  • 2 replies
  • 8 kudos

Restore Delta table after adding columns.

At version 3 of our Delta Lake table we added a column. We later restored from version 11 back to version 10, which is now the most current version. But now when we run the table build from Azure Data Factory (ADF) on the full history of the data, we...

Data Engineering
Reply
  • 2490 Views
  • 2 replies
  • 8 kudos
View Replies
Latest Reply
B_Seibert
New Contributor III
  • 8 kudos

ignoreDeletes works. But I recommend to other developers that you have a think aboout all of the schema change scenarios and solve this problem above as part of a complete solution to every schema change scenario, instead of dealing with it as a one...

  • 8 kudos
1 More Replies
bharani
by • New Contributor II
  • 792 Views
  • 0 replies
  • 0 kudos

coverage in databricks

Hi Prabakar, I would like to do test coverage in databricks notebooks so is there any way that I can do and also can I do testing with pytest in databricks? Are there any other tests like nutter library and doc test supports coverage? 

Data Engineering
Reply
  • 792 Views
  • 0 replies
  • 0 kudos
Nilave
by • New Contributor III
  • 6373 Views
  • 4 replies
  • 8 kudos

Resolved! Azure Databricks Python Test Coverage report

Hi,I'm trying to enable coverage report for Azure Databricks python code. Found a solution using pytest with databricks connect but faced below challenges Pytest framework using Databricks Connect:- This works for simple notebooks. But doesnt work fo...

Data Engineering
Reply
  • 6373 Views
  • 4 replies
  • 8 kudos
View Replies
Latest Reply
Prabakar
Databricks Employee
  • 8 kudos

@Nilave Chakraborty​ , There is no support for code coverage functionality yet. I don't think it will be available in the near future. However, it will be considered later. I do not have any timeline for now.

  • 8 kudos
3 More Replies
PhoenBounce
by • New Contributor III
  • 4092 Views
  • 3 replies
  • 11 kudos

Resolved! Unable to see the Delta Live Tables tab in Workflows on Azure Databricks

I have looked everywhere but I am unable to find the option but doing the training indicates I need to do this to create a pipeline.I must be missing something obvious as I am new to thisThanksSteve

Data Engineering
Reply
  • 4092 Views
  • 3 replies
  • 11 kudos
View Replies
Latest Reply
Unforgiven
Valued Contributor III
  • 11 kudos

Delta Live Tables (DLT) is the first ETL framework that uses a simple declarative approach to building reliable data pipelines and automatically managing your infrastructure at scale so data analysts and engineers can spend less time on tooling and f...

  • 11 kudos
2 More Replies
AB_MN
by • New Contributor III
  • 3554 Views
  • 5 replies
  • 1 kudos

Error when using DLT with SCD type 1

I'm trying to load data using DLT and SCD 1 and am running into the error message "Detected a data update in the source table at version x. This is currently not supported. If you'd like to ignore updates, set the option 'ignoreChanges' to 'true'.I h...

image
Data Engineering
Reply
  • 3554 Views
  • 5 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Andy Bolk​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 1 kudos
4 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels
Top