- 5158 Views
- 2 replies
- 2 kudos
Resolved! What is the best way to find deltalake version on OSS and Databricks at runtime?
To identify certain deltalake features available on a certain installation, it is important to have a robust way to identify deltalake version. For OSS, I found that the below Scala snippet will do the job.import io.delta println(io.delta.VERSION)Not...
- 5158 Views
- 2 replies
- 2 kudos
- 2 kudos
@Yousry Mohamed​ - could you please check the DBR runtime release notes for the Delta lake API compatibility matrix section ( DBR version vs Delta lake compatible version) for the mapping.Reference: https://docs.databricks.com/release-notes/runtime/r...
- 2 kudos
- 9133 Views
- 6 replies
- 12 kudos
ignoreDeletes' option with Delta Live Table streaming source
We have a delta streaming source in our delta live table pipelines that may have data deleted from time to time. The error message is pretty self explanatory:...from streaming source at version 191. This is currently not supported. If you'd like to i...
- 9133 Views
- 6 replies
- 12 kudos
- 12 kudos
I'd am looking at this as well and would like to understand my options here.
- 12 kudos
- 2664 Views
- 3 replies
- 2 kudos
Resolved! Limitation as of now in delta live table
I am thinking of using delta live table, before that I want to be aware of the limitations it has as of now when it s announced on datasummit 2021
- 2664 Views
- 3 replies
- 2 kudos
- 2 kudos
There doesn't appear to be a way to enforce a retention policy on source tables when defining a structured stream. Setting the options for "ignoreChanges" and "ignoreDeletes" doesn't seem to have any effect at all. CDC does not fill this role either,...
- 2 kudos
- 4002 Views
- 2 replies
- 0 kudos
GRANT PRIVILEGES or REVOKE not working in databricks: Operation not allowed
Hi everyone!I am having a problem! I can Grant or Revoke privileges from users using the UI on databricks, but when I try to do that using SQL commands such as GRANT SELECT ON SCHEMA [... ] TO [USER]; I get an error stating Operation not allowed.I am...
- 4002 Views
- 2 replies
- 0 kudos
- 1570 Views
- 2 replies
- 0 kudos
Issue with visualizing dataframe from a job
Greetings,I have the following data set:```sqlSELECT * FROM ( VALUES ('2023-02',113.81::decimal(27,2),'A','X'), ('2023-02',112.66::decimal(27,2),'A','Y'), ('2023-02',1223.8::decimal(27,2),'B','X'), ('2023-02',1234.56::decimal(27,2),'B',...
- 1570 Views
- 2 replies
- 0 kudos
- 0 kudos
Attaching some more screenshots to add more details.This seems to be a bug in the bar chart visualization widget when displaying from job run.
- 0 kudos
- 913 Views
- 1 replies
- 3 kudos
Updated UI for Create, run, and manage Databricks JobsÂ
Updated UI for Create, run, and manage Databricks Jobs
- 913 Views
- 1 replies
- 3 kudos
- 3 kudos
Hi Ajay, Could you please clarify the ask here? Thanks.
- 3 kudos
- 1294 Views
- 1 replies
- 4 kudos
Resolved! REVOKE access from users
There is a use-case where we want to REVOKE access from users so that they can't run VACUUM Command on Delta Table.Can anyone please help here.
- 1294 Views
- 1 replies
- 4 kudos
- 4 kudos
Hello @KVNARK .​ We cannot specifically restrict Vacuum operation alone.You need to remove "MODIFY" access on the table and restrict only to the "Read" (SELECT) operationPlease note if you restrict to only "read" it will also affect all the write, up...
- 4 kudos
- 1394 Views
- 1 replies
- 1 kudos
Using your own docker container to launch databricks cluster.
When using your own docker container while creating a databricks cluster, what is the mapping between the number of containers launched and the nodes launched?Is it 1:1 mapping? or is it similar to other orchestration framework like Kubernetes?Or is ...
- 1394 Views
- 1 replies
- 1 kudos
- 1 kudos
+1
- 1 kudos
- 2572 Views
- 4 replies
- 5 kudos
Data Engineer Associate and Professional tittle holders count
How many people hold tittles of certified Databricks Data Engineers Asociate/Professional right now?Is there any place I can check the global certificate count?
- 2572 Views
- 4 replies
- 5 kudos
- 5 kudos
check here: https://credentials.databricks.com/collection/da21363e-5c7d-410a-b144-dd07d3e22942?_ga=2.163643839.1823848454.1674389186-2106443313.1667211405&_gac=1.49521364.1672812437.CjwKCAiAwc-dBhA7EiwAxPRylBN9S-JeQ8779ec3GXJYBQPfnu_qkv5l_MKO1u4jw2w-...
- 5 kudos
- 2876 Views
- 3 replies
- 1 kudos
Resolved! Azure CosmosDB change feed ingestion via DLT
Is there a way to ingest Azure CosmosDB data via Delta Live Tables? If I use regular workflows it works well, but with DLT I'm not able to set CosmosDB Connector on a cluster.
- 2876 Views
- 3 replies
- 1 kudos
- 1 kudos
Thanks a lot! Just wanted to doublecheck whether this natively exists.
- 1 kudos
- 2587 Views
- 2 replies
- 0 kudos
depth of view exceeds the maximum view resolution depth (100).
I got this error after updating a view. How can I increase the value of spark.sql.view.maNestedViewDepth to work around this? Thanks!
- 2587 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi, Could you please confirm if you are showing the view? (https://docs.databricks.com/sql/language-manual/sql-ref-syntax-aux-show-views.html) also, it will be helpful if you post the screenshot of the error.
- 0 kudos
- 1684 Views
- 1 replies
- 3 kudos
Delta Sharing Costs
When Delta Sharing is enabled and a link is shared, I understand that the data transfer happens directly and not through the sharing server. I'm curious how costs are calculated. Is the entity making the share available charged for data egress and ...
- 1684 Views
- 1 replies
- 3 kudos
- 3 kudos
Any news
- 3 kudos
- 4122 Views
- 5 replies
- 2 kudos
Cannot install com.microsoft.azure.kusto:kusto-spark
Hello,I'm trying to install/update the library com.microsoft.azure.kusto:kusto-spark_3.0_2.12:3.1.xTried to install with Maven central repository and using Terraform.It was working previously and now the installation always ends with error:│ Error: c...
- 4122 Views
- 5 replies
- 2 kudos
- 2 kudos
I have the same problem with a slightly different version of the connector (change on the minor version). I have a job that runs every hour and specifically, this started to happen on the 23rd of January onwards. The error indeed does say the same:Ru...
- 2 kudos
- 4543 Views
- 4 replies
- 2 kudos
Pausing a scheduled Azure Databricks job after failure
Hi All,I have a job/workflow scheduled in Databricks to run after every hour.How can I configure my Job to pause whenever a job run fails? (Pause the job/workflow on first failure)I would want to prevent triggering multiple runs due to the scheduled/...
- 4543 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @Hubert Dudek​ , Thank you for your suggestion.I understand that we can use Jobs API to change the pasue_status of job on errors, but sometimes we observed that the workflow/job fails due to cluster issues (while the job clusters are getting creat...
- 2 kudos
- 2135 Views
- 1 replies
- 1 kudos
Understanding file retention with Vacuum
I have seen few instances where users reported that they run OPTIMIZE for the past week worth of data and they follow by VACUUM with RETAIN of 168 HOURS (for example), the old files aren't being deleted, "VACUUM is not removing old files from the tab...
- 2135 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @Venkatesh Kottapalli​ VACUUM removes all files from the table directory that are not managed by Delta, as well as data files that are no longer in the latest state of the transaction log for the table and are older than a retention threshold. ...
- 1 kudos