- 5246 Views
- 2 replies
- 0 kudos
When would you not want to use autoloader?
I am genuinely curious why would you ever not use Autoloader? I see it in one-off downloads of course. When you pull data from another platform, say Salesforce, is it better to append to a table without Autoloader? There must be cases I am missing. T...
- 5246 Views
- 2 replies
- 0 kudos
- 0 kudos
Autoloader is pretty handy, but not open source. That is one reason f.e.Another reason is f.e. if you cannot guarantee lexicographically generated files, or you do not want to use streaming, or you do not land your raw data into a data lake (read fr...
- 0 kudos
- 3944 Views
- 3 replies
- 1 kudos
Resolved! DATABRICKS TO AZ SQL??
Hi All,, quick question:Is this correct data flow pattern: Databricks -> Az SQL -> Tableau??Or does it have to go through ADLS: Databricks -> ADLS -> Az SQL - > Tableau? Also, is it better to leverage databricks lakehouse sql warehouse capability as ...
- 3944 Views
- 3 replies
- 1 kudos
- 1 kudos
I would not call it 'better' per se. A lakehouse is a more modern approach to a classic datawarehouse, using flexible distributed cloud compute, cheap storage and open file formats.If you have an existing environment, which works well, that is heavi...
- 1 kudos
- 7815 Views
- 1 replies
- 0 kudos
What are the Delta Live Tables limitations with relation to Unity Catalog?
Hi community!I was in a Databricks webinar and one of the participants said "Delta Live Tables seems to have some limitations when using with Unity Catalog. Is the idea to get parity with Hive?" and someone answered "DLT + Unity Catalog combination h...
- 7815 Views
- 1 replies
- 0 kudos
- 28381 Views
- 1 replies
- 1 kudos
Hashing Functions in PySpark
Hashes are commonly used in SCD2 merges to determine whether data has changed by comparing the hashes of the new rows in the source with the hashes of the existing rows in the target table. PySpark offers multiple different hashing functions like:MD5...
- 28381 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Retired_mod ,thank you for your comprehensive answer. What is your opinion on the trade-off between using a hash like xxHASH64 which returns a LongType column and thus would offer good performance when there is a need to join on the hash column v...
- 1 kudos
- 1756 Views
- 0 replies
- 0 kudos
Delta Live Tables saving as corrupt files
Hello,I am using Delta Live Tables to store data and then trying to save them to ADLS. I've specified the storage location of the Delta Live Tables in my Delta Live Tables pipeline. However, when I check the files that are saved in ADLS, they are cor...
- 1756 Views
- 0 replies
- 0 kudos
- 8172 Views
- 3 replies
- 1 kudos
PermissionError / Operation not Permitted with Files-in-Repos
I've been running a notebook using files-in-repo. Previously this has worked fine. I'm unsure what's changed (I was testing integration with DCS on older runtimes, but don't think I made any persistent changes)--but now it's throwing an error (always...
- 8172 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @jfarmer , did you solved this issue? I'm having exactly the same challenge.Thanks!
- 1 kudos
- 1705 Views
- 0 replies
- 0 kudos
Failed to run the job on databricks version LTS 9.x and 10.x(AWS)
Hi Team,When we tried to change the databricks version from 7.3 to 9.x or 10.x we are getting below error. Caused by: java.lang.RuntimeException: MetaException(message:Unable to verify existence of default database: com.amazonaws.services.glue.model....
- 1705 Views
- 0 replies
- 0 kudos
- 2887 Views
- 2 replies
- 2 kudos
How can we create streaming tables as external delta tables ?
We would like to introduce DLT, Streaming tables to our medallion architecture but we are unable to create the streaming tables with concerned schemas. STREAMING Tables doesn't have an option to be stored with custom schemas. The requirement we have ...
- 2887 Views
- 2 replies
- 2 kudos
- 2 kudos
If unity catalog is used, by default tables under that would be managed
- 2 kudos
- 5137 Views
- 2 replies
- 1 kudos
[INTERNAL_ERROR] Cannot generate code for expression: claimsconifer.default.decrypt_colA(
A column contains encrypted data at rest. I am trying to create a sql function which will decrypt the data if the user is a part of a particular group. Below is the function: %sql CREATE OR REPLACE FUNCTION test.default.decrypt_if_valid_user(col_a ST...
- 5137 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Retired_mod After removing "TABLE" keyword from create or replace statement this function got registered as builtin function. Just to verify that I displayed all the functions and I can see that function--> decrypt_if_valid_user:Now I am trying t...
- 1 kudos
- 3864 Views
- 3 replies
- 1 kudos
Resolved! Are data health check expectations available only on Delta Live tables?
I love the idea of "expectations" being available for Delta Live tables: https://docs.databricks.com/delta-live-tables/expectations.htmlI'd like to know if they are also available for regular delta tables?Thank you in advance!
- 3864 Views
- 3 replies
- 1 kudos
- 1 kudos
Hello @Oliver_Angelil, so have you found a way to implement something resembling expectations for delta tables outside of a DLT pipeline ?
- 1 kudos
- 5304 Views
- 1 replies
- 1 kudos
How to display shap waterfall plot
Hi,I have managed to display force plot for a single observation using the advice from this thread:Solved: How to display SHAP plots? - Databricks - 28315But is there anyway to display the newer "waterfall"-plot shap.plots.waterfall — SHAP latest doc...
- 5304 Views
- 1 replies
- 1 kudos
- 1 kudos
Thank you for the swift response. I made a minimal example and it does work as you said. However when I try with my own model it does not work, the only output is<Figure size 576x468 with 3 Axes>I tried to save the figure as a file and then I do get ...
- 1 kudos
- 1238 Views
- 0 replies
- 0 kudos
How to write test code in databricks
from databricks.connect import DatabricksSession from data.dbx_conn_info import DbxConnInfo class SparkSessionManager: _instance = None _spark = None def __new__(cls): if cls._instance is None: cls._instance = s...
- 1238 Views
- 0 replies
- 0 kudos
- 11858 Views
- 3 replies
- 4 kudos
- 11858 Views
- 3 replies
- 4 kudos
- 4 kudos
Use Lakehouse Monitoring: https://docs.databricks.com/en/lakehouse-monitoring/index.html Specifically: https://docs.databricks.com/en/lakehouse-monitoring/monitor-output.html#drift-metrics-table
- 4 kudos
- 7024 Views
- 2 replies
- 2 kudos
InputWidgetNotDefined exception when running a notebook as a job
I have a notebook that reads a value of a text input in a Scala command:var startTimeStr = dbutils.widgets.get("Run Date")What doesn't make any sense that this notebook fails with InputWidgetNotDefined error when being scheduled as a job, but works j...
- 7024 Views
- 2 replies
- 2 kudos
- 13559 Views
- 4 replies
- 1 kudos
Connection to Azure SQL Server: ODBC Driver 18 for SQL Server
Task: Setup connection to Azure SQL Server.A couple things have changed...*We've started using Unity Catalog, so need Unity Catalog -enabled clusters*Legacy init scripts have been deprecated, and this is how we had our pyodbc setup, etc. defined.Code...
- 13559 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @js54123875 did you manage to find a solution for this? I'm facing a similar problem.ThanksDiego
- 1 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 486 | |
| 349 | |
| 287 |
