- 6961 Views
- 3 replies
- 1 kudos
Error while trying to implement Change Data Capture
Hi All,I'm new to databricks and learning towards taking up Associate Engineer Certification.While going through the section "Build Data Pipelines with Delta Live Tables". I'm trying to implement Change Data Capture, but it is erroring out when execu...
- 6961 Views
- 3 replies
- 1 kudos
- 1 kudos
Having had a quick look, I think your error is because you are trying to add SCD to a STREAMING LIVE table. I believe APPLY CHANGES INTO cannot be used on a streaming table.You can use a streaming table as a source though.Simply changing this line:C...
- 1 kudos
- 5599 Views
- 2 replies
- 1 kudos
Issue with inserting multiple rows in Delta table with identity column
Running DBR 11.3 / Azure DatabricksTable definition below:%sql CREATE OR REPLACE TABLE demo2 ( id BIGINT GENERATED BY DEFAULT AS IDENTITY, product_type STRING, sales BIGINT ) USING DELTA LOCATION '/folderlocation/' TBLPROPERTIES ( 'delta.column...
- 5599 Views
- 2 replies
- 1 kudos
- 1 kudos
Just updating it is possible this issue has now been addressed.As before working on Azure Databricks 11.3 DBRInserting into managed table:Also appears to be addressed for autoloader insertion into unmanaged table
- 1 kudos
- 2261 Views
- 1 replies
- 7 kudos
Starting from databricks 12.2 LTS, the explode function can be used in the FROM statement to manipulate data in new and powerful ways. This function t...
Starting from databricks 12.2 LTS, the explode function can be used in the FROM statement to manipulate data in new and powerful ways. This function takes an array column as input and returns a new row for each element in the array, offering new poss...
- 2261 Views
- 1 replies
- 7 kudos
- 7 kudos
It's very useful for SQL developers.
- 7 kudos
- 6029 Views
- 2 replies
- 0 kudos
Overwriting the existing table in Databricks; Mechanism and History?
Hi,Assume that I have a delta table stored on an Azure storage account. When new records arrive, I repeat the transformation and overwrite the existing table. (DF.write .format("delta") .mode("overwrite") .option("...
- 6029 Views
- 2 replies
- 0 kudos
- 0 kudos
the overwrite will add new files, keep the old ones and in a log keeps track of what is current data and what is old data.If the overwrite fails, you will get an error message in the spark program, and the data to be overwritten will still be the cur...
- 0 kudos
- 11134 Views
- 2 replies
- 0 kudos
Resolved! Convert date to integer
Hello. Is there a way in Databricks sql to convert a date to integer? In Db2, there is days function DAYS - IBM Documentation .For example '2023-03-01' is converted to 738580 value.Thank you in advance
- 11134 Views
- 2 replies
- 0 kudos
- 0 kudos
TRy this:CREATE OR REPLACE FUNCTION days(dt DATE) RETURN unix_date(dt) - unix_date(DATE'0001-01-01') + 1;SELECT current_date, days(current_date); 2023-03-09 738588I verified on Db2 for LUW and it matches up.
- 0 kudos
- 12511 Views
- 5 replies
- 3 kudos
Resolved! Unable to import tkinter in notebook
Hello,I am unable to import tkinter (or Tkinter) into a python notebook.I also tried %pip install tkinter at the top of the notebook.Has anyone else been successful at this, or if it's impossible, why? Thank you
- 12511 Views
- 5 replies
- 3 kudos
- 3 kudos
Hi @Will Griffin​ Can you confirm if this worked for you? I get a message `ERROR: No matching distribution found for python3-tk`.
- 3 kudos
- 4641 Views
- 4 replies
- 0 kudos
I added a secret scope to the databricks environment, and I get this error when trying to run either of the following:
Commands Attempted:dbutils.secrets.listScopes()dbutils.secrets.get(scope = "{InsertScope}", key = "{InsertKey}") Error: "shaded.v245.com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (number, ...
- 4641 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Haylee Gaddy​,Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.
- 0 kudos
- 5619 Views
- 6 replies
- 0 kudos
GeneratedAlwaysAs' along with dataframe.write
Is it possible to use a calculated column (as like in the delta table using generatedAlwaysAs) definition while writing the data frame as a delta file like df.write.format("delta").Any options are there with the dataframe.write method to achieve this...
- 5619 Views
- 6 replies
- 0 kudos
- 0 kudos
Hi @Thushar R​ ,This option is not a part of Dataframe write API as GeneratedAlwaysAs feature is only applicable to Delta format and df.write is a common API to handle writes for all formats. If you to achieve this programmatically, you can still use...
- 0 kudos
- 7976 Views
- 4 replies
- 7 kudos
Resolved! Delta Live Table Schema Comment
I predefined my schema for a Delta Live Table Autoload. This included comments for some attributes. When performing a standard readStream, my comments appear, but when in Delta Live Tables I get no comments. Is there anything I need to do get comment...
- 7976 Views
- 4 replies
- 7 kudos
- 7 kudos
You need to add your schema to dlt declaration:@dlt.table( name="test_bronze", comment = "test account data incrementally ingested from S3 Raw landing zone", table_properties={ "quality": "bronze" }, schema=schema)
- 7 kudos
- 35764 Views
- 4 replies
- 1 kudos
Resolved! How to get rid of "Command result size exceeds limit"
I am working on Databricks Notebook and trying to display a map using Floium and I keep getting this error > Command result size exceeds limit: Exceeded 20971520 bytes (current = 20973510)How can I get increase the memory limit?I already reduced the...
- 35764 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi, I have the same problem with keplergl, and the save to disk option, whilst helpful isn't super practical... So how does one plot large datasets in kepler?Any thought welcome
- 1 kudos
- 3252 Views
- 2 replies
- 3 kudos
Resolved! Data lineage graph now working
I created the below tables but when I click the lineage graph not able to see the upstream or downstream table .... the + sign goes away after a few sec but not able to click it ... anyone else having this issue?CREATE TABLE IF NOT EXISTS lineage_d...
- 3252 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Raj Sharma​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 3 kudos
- 2852 Views
- 2 replies
- 1 kudos
Resolved! How to load multiple xlsx that are storage in different folders with the same name in a blob storage in a delta table ?
Hi i have a blob storage with multile unzip folders with the same suffix folder_report_name_01_2023_01_02 -> file_name_2023_01_02.xlsxBut i want to load all of this data using pandas or pyspark and insert in my delta table.I'm trying to using widget...
- 2852 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Fernando Vázquez​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.T...
- 1 kudos
- 3370 Views
- 2 replies
- 2 kudos
Why Azure Databricks needs to store data in temp storage in Azure before writing to the synapse.
I was following the tutorial about data transformation with azure databricks, and it says before loading data into azure synapse analytics, the data transformed by azure databricks would be saved on temp storage in azure blob storage first before loa...
- 3370 Views
- 2 replies
- 2 kudos
- 2 kudos
@Ajay Pandey​ Saving the transformed data to temporary storage in Azure Blob Storage before loading into Azure Synapse Analytics provides a number of benefits to ensure that the data is accurate, optimized, and performs well in the target environmen...
- 2 kudos
- 2088 Views
- 1 replies
- 0 kudos
What is the upper bound limit for dataSkippingNumIndexedCols, to keeps stats in delta log file?
Is there an upper bound of number that i can assign to delta.dataSkippingNumIndexedCols for computing statistics. Is there some tradeoff benchmark available for increasing this number beyond 32.
- 2088 Views
- 1 replies
- 0 kudos
- 0 kudos
@Chhavi Bansal​ :The delta.dataSkippingNumIndexedCols configuration property controls the maximum number of columns that Delta Lake will build statistics on during data skipping. By default, this value is set to 32. There is no hard upper bound on th...
- 0 kudos
- 8223 Views
- 1 replies
- 1 kudos
how can i export my dashboard en format html using databriks api
hi everyone, i would like to export my dashbord in html format and embed it in my body of my email in order to send it to my teamso there is my code python for the databriks api and i got this error and when i put my htm in the body of my message i...
- 8223 Views
- 1 replies
- 1 kudos
- 1 kudos
@mathild noun​ :import databricks.workspace as workspace_api import requests # set up your Databricks workspace credentials domain = "<your Databricks workspace domain>" token = "<your Databricks API token>" # set up the workspace client workspac...
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 576 | |
| 349 | |
| 287 |
