- 4961 Views
- 8 replies
- 4 kudos
Hi Community,I have successfully run a job through the API but would need to be able to pass parameters (configuration) to the DLT workflow via the APII have tried passing JSON in this format:{
"full_refresh": "true",
"configuration": [
...
- 4961 Views
- 8 replies
- 4 kudos
by
pokus
• New Contributor III
- 2740 Views
- 3 replies
- 2 kudos
I need to use DeltaLog class in the code to get the AddFiles dataset. I have to keep the implemented code in a repo and run it in databricks cluster. Some docs say to use org.apache.spark.sql.delta.DeltaLog class, but it seems databricks gets rid of ...
- 2740 Views
- 3 replies
- 2 kudos
Latest Reply
Thanks for providing a solution @pokus .What I dont understand is why Databricks cannot provide the DeltaLog at runtime. How can this be the official solution? We need a better solution for this instead of depending on reflections.
2 More Replies
- 699 Views
- 3 replies
- 7 kudos
Rename and drop columns with Delta Lake column mapping. Hi all,Now databricks started supporting column rename and drop.Column mapping requires the following Delta protocols:Reader version 2 or above.Writer version 5 or above.Blog URL##Available in D...
- 699 Views
- 3 replies
- 7 kudos
Latest Reply
Above mentioned feature is not working in the DLT pipeline. if the scrip has more than 4 columns
2 More Replies
- 4185 Views
- 2 replies
- 0 kudos
Delta creates more small files during merge and updates operations.
- 4185 Views
- 2 replies
- 0 kudos
Latest Reply
Delta solves the large number of small file problems using the below operations available for a Delta table. Optimize writes helps to optimizes the write operation by adding an additional shuffle step and reducing the number of output files. By defau...
1 More Replies
- 4554 Views
- 10 replies
- 4 kudos
Suppose I have a Delta Live Tables framework with 2 tables: Table 1 ingests from a json source, Table 2 reads from Table 1 and runs some transformation.In other words, the data flow is json source -> Table 1 -> Table 2. Now if I find some bugs in the...
- 4554 Views
- 10 replies
- 4 kudos
Latest Reply
Answering my own question: nowadays (February 2024) this can all be done via the UI.When viewing your DLT pipeline there is a "Select tables for refresh" button in the header. If you click this, you can select individual tables, and then in the botto...
9 More Replies
- 2145 Views
- 3 replies
- 3 kudos
woahhh #Excel plug in for #DeltaSharing.Now I can import delta tables directly into my spreadsheet using Delta Sharing.It puts the power of #DeltaLake into the hands of millions of business users.What does this mean?Imagine a data provider delivering...
- 2145 Views
- 3 replies
- 3 kudos
Latest Reply
If you have any uncertainties, feel free to inquire here or connect with me on my LinkedIn profile for further assistance.https://whatsgbpro.org/
2 More Replies
- 5669 Views
- 2 replies
- 3 kudos
What is the difference between Databricks Auto-Loader and Delta Live Tables? Both seem to manage ETL for you but I'm confused on where to use one vs. the other.
- 5669 Views
- 2 replies
- 3 kudos
Latest Reply
You say "...__would__ be a piece..." and "...DLT __would__ pick up...".Is DLT built upon AL?
1 More Replies
- 2349 Views
- 6 replies
- 3 kudos
Hello, we have Databricks Python workbooks accessing Delta tables. These workbooks are scheduled/invoked by Azure Data Factory. How can I enable Photon on the linked services that are used to call Databricks?If I specify new job cluster, there does n...
- 2349 Views
- 6 replies
- 3 kudos
Latest Reply
When you create a cluster on Databricks, you can enable Photon by selecting the "Photon" option in the cluster configuration settings. This is typically done when creating a new cluster, and you would find the option in the advanced cluster configura...
5 More Replies
- 8067 Views
- 3 replies
- 3 kudos
I have setup my Databricks notebook to use Service Principal to access ADLS using below configuration.service_credential = dbutils.secrets.get(scope="<scope>",key="<service-credential-key>")
spark.conf.set("fs.azure.account.auth.type.<storage-accou...
- 8067 Views
- 3 replies
- 3 kudos
Latest Reply
found the solution, need one additional configuration. spark._jsc.hadoopConfiguration().set("fs.azure.account.key.<account name>.dfs.core.windows.net",'<account key>')
2 More Replies
by
sanjay
• Valued Contributor II
- 5660 Views
- 8 replies
- 0 kudos
Hi,I have pipeline running. I have updated one file in delta table which is already processed. Now I am getting errorcom.databricks.sql.transaction.tahoe.DeltaUnsupportedOperationException: Detected a data update. This is currently not supported. If ...
- 5660 Views
- 8 replies
- 0 kudos
Latest Reply
Hi Sanjay, You can try adding .option("overwriteSchema", "true")
7 More Replies
by
shagun
• New Contributor III
- 2406 Views
- 3 replies
- 0 kudos
The first time i run my delta live table pipeline after setup, I get this error on starting it :-------------------------------------org.apache.spark.sql.catalyst.parser.ParseException: Possibly unquoted identifier my-schema-name detected. Please con...
- 2406 Views
- 3 replies
- 0 kudos
Latest Reply
This still errors on internal databricks spark/python code likedeltaTable.history()@shagun wrote:The first time i run my delta live table pipeline after setup, I get this error on starting it :-------------------------------------org.apache.spark.sql...
2 More Replies
- 1577 Views
- 2 replies
- 1 kudos
Does anyone have any recent examples of using Tableau and Delta Sharing? The video below mentions using web connector but this connector has been depreciated in Tableau 2023.1. https://www.youtube.com/watch?v=Yg-5LXH9K1I&t=913shttps://help.tableau.co...
- 1577 Views
- 2 replies
- 1 kudos
Latest Reply
Hi,I am still trying to figure out how to use delta sharing with tableau. I've looking for information for a month without any success. As mentionned before, web data connector is deprecatedAny help would be appreciated.thanks, Johnattan
1 More Replies
- 4219 Views
- 9 replies
- 12 kudos
I recently created a table on a cluster in Azure running Databricks Runtime 11.1. The table is partitioned by a "date" column. I enabled column mapping, like this:ALTER TABLE {schema}.{table_name} SET TBLPROPERTIES('delta.columnMapping.mode' = 'nam...
- 4219 Views
- 9 replies
- 12 kudos
Latest Reply
Hi @Gary_Irick, @gongasxavi , @Pete_Cotton , @aleks1601 ,
Certainly, let’s address your questions regarding Delta table partition directories and column mapping.
Directory Names with Column Mapping:
When you enable column mapping in a Delta tabl...
8 More Replies
by
DJey
• New Contributor III
- 4445 Views
- 5 replies
- 2 kudos
- 4445 Views
- 5 replies
- 2 kudos
Latest Reply
@Vidula Khanna​ Enabling the below property resolved my issue:spark.conf.set("spark.databricks.delta.schema.autoMerge.enabled",True) Thanks v much!
4 More Replies
- 1385 Views
- 4 replies
- 2 kudos
I love the idea of "expectations" being available for Delta Live tables: https://docs.databricks.com/delta-live-tables/expectations.htmlI'd like to know if they are also available for regular delta tables?Thank you in advance!
- 1385 Views
- 4 replies
- 2 kudos
Latest Reply
Hello @Oliver_Angelil, so have you found a way to implement something resembling expectations for delta tables outside of a DLT pipeline ?
3 More Replies