- 11103 Views
- 7 replies
- 24 kudos
Resolved! Couldn't create new catalog?
I used DBR version 11.0
- 11103 Views
- 7 replies
- 24 kudos
- 24 kudos
In the 2.9 Comprehensive lab of getting started with data engineering on data bricks, I try to run:%run ../Includes/Classroom-Setup-11and get the following errorThe execution of this command did not finish successfully Resetting the learning envir...
- 24 kudos
- 1467 Views
- 0 replies
- 0 kudos
How to force delta live tables legacy execution mode?
We've been running delta live tables for some time with unity catalog and it's as slow as a sloth on a Hawaiian vacation.Anyway, DLT had three consecutive failures (due to the data source being unreliable) and then the logs printed: "MaxRetryThreshol...
- 1467 Views
- 0 replies
- 0 kudos
- 27241 Views
- 4 replies
- 7 kudos
Resolved! Error: The associated location ... is not empty but it's not a Delta table
I try to create a table but I get this error: AnalysisException: Cannot create table ('`spark_catalog`.`default`.`citation_all_tenants`'). The associated location ('dbfs:/user/hive/warehouse/citation_all_tenants') is not empty but it's not a Delta t...
- 27241 Views
- 4 replies
- 7 kudos
- 7 kudos
Hi Team, I am facing the same issue. When we try to load data to table in production batch getting error as table not in delta format. there is no recent change in table. and we are not trying any create or replace table. this is existing table in pr...
- 7 kudos
- 5343 Views
- 1 replies
- 1 kudos
How to get different dynamic value for each task in workflow
I created a workflow with two tasks. It runs the first notebook and then it wait for that to finish to start the second notebook. I want to use this dynamic value as one of the parameters {{job.start_time.iso_datetime}} for both tasks. This should gi...
- 5343 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @mh_db , The dynamic value {{job.start_time.iso_datetime}} you are using in your workflow is designed to capture the start time of the job run, not the individual tasks within the job. This is why you are seeing the same date and time for both ...
- 1 kudos
- 1634 Views
- 1 replies
- 1 kudos
Identifying invalid views
Is there a way to identify all invalid views in a schema or catalog without querying the view to see if it succeeds?
- 1634 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @WWoman, I don't think there's a feature for that. If you think this would be a cool feature you could submit an idea in Databricks' Ideas Portal.
- 1 kudos
- 3338 Views
- 3 replies
- 0 kudos
Resolved! Disk cache for csv file in Databricks
Dear team,I'm investigate to improve performance when reading large csv file as input and find this https://learn.microsoft.com/en-us/azure/databricks/optimizations/disk-cache.I just wonder Do disk-cache also apply for csv file?Thanks!
- 3338 Views
- 3 replies
- 0 kudos
- 0 kudos
Thanks @-werners-,That's right, I tried and get some significantly performance.
- 0 kudos
- 6879 Views
- 9 replies
- 0 kudos
Not able to read the file content completely using head
Hi,We want to read the file content of the file and encode the content into base64. For that we have used below code file_path = "/path/to/your/file.csv"file_content = dbutils.fs.head(file_path, 512000000)encode_content = base64.b64encode(file_conten...
- 6879 Views
- 9 replies
- 0 kudos
- 0 kudos
I am curious what the use case if for wanting to load large files into github, which is a code repo.Depending on the file format different parsing is necessary. you could foresee logic for that in your program.
- 0 kudos
- 2893 Views
- 2 replies
- 0 kudos
AWS Email sending challenge from Databricks with UNITY CATALOG and Multinode cluster
Hi,I have implemented the UNITY CATALOG with multinode cluster in databricks. The workspace instance profile with EC2 access is also created in IAM. but still having a challenge in sending emails from databricks using SES service.The same is working ...
- 2893 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @DataEngineer ,Are you able to resolve the issue. We are having the same issue when we try to use MultiNode cluster for UnityCatalog. Email functionality was working fine with Single node cluster.We are getting "ConnectionRefusedError: [Errno 111]...
- 0 kudos
- 1984 Views
- 0 replies
- 0 kudos
Instance profile failure while installing Databricks Overwatch
Despite following the steps mentioned in the provided link to create an instance profile, we encountered a problem in step 6 where we couldn't successfully add the instance profile to Databricks(Step 6: Add the instance profile to Databricks).https:/...
- 1984 Views
- 0 replies
- 0 kudos
- 1591 Views
- 0 replies
- 0 kudos
Different result between manual and automated task run
I have a notebook where I bring info about a previous task run metadata from the API ".... /jobs/runs/get". The response should be a dictionary that contains information such as task key, run if, run page URL etc. When I run the notebook as part of ...
- 1591 Views
- 0 replies
- 0 kudos
- 4606 Views
- 4 replies
- 2 kudos
Bug: Shallow Clone `create or replace` causing [TABLE_OR_VIEW_NOT_FOUND]
I am having an issue where when I do a shallow clone using :create or replace table `catalog_a_test`.`schema_a`.`table_a` shallow clone `catalog_a`.`schema_a`.`table_a` I get:[TABLE_OR_VIEW_NOT_FOUND] The table or view catalog_a_test.schema_a.table_a...
- 4606 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi StevenThis is really a strange issue. First let's exclude some possible causes for this. We need to check the following:- The permission to table A and Catalog B. take a look at the following link to check what permission is needed: https://docs.d...
- 2 kudos
- 2501 Views
- 1 replies
- 1 kudos
Resolved! Can I delete specific partition from a Delta Live Table?
if I have created a Delta Live Table with partition on a column (lets say a date column) from a Stream Source, can I delete the partition for specific date values later to save on cost & to keep the table lean? if I can, then -1- how to do it?2- do I...
- 2501 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @gauravchaturved , You can remove the partition by filtering it in your source code and triggering a full refresh in your pipeline. There is no need to run vacuum, as DLT has maintenance clusters that perform OPTIMIZE and VACUUM operations on y...
- 1 kudos
- 5145 Views
- 3 replies
- 2 kudos
Unable to connect with Databricks Serverless SQL using Dbeaver
I am trying to connect to databricks serverless SQL pool using DBeaver as mentioned in the documentation below:https://learn.microsoft.com/en-us/azure/databricks/dev-tools/dbeaverI am trying to use the Browser based authentication i.e (OAuth user-to-...
- 5145 Views
- 3 replies
- 2 kudos
- 5168 Views
- 3 replies
- 2 kudos
Resolved! Databricks Expectations
In the example in https://docs.databricks.com/en/delta-live-tables/expectations.html#fail-on-invalid-records, it wrote that one is able to query the DLT event log for such expectations violation. In Databricks, I can use expectation to fail or drop r...
- 5168 Views
- 3 replies
- 2 kudos
- 2 kudos
That's right, the "reason" would be "x1 is negative" in your example and "valid_max_length" in the example JSON payload that I shared.If you are looking for a descriptive reason, you would name the expectation accordingly such as: @Dlt.expect_or_fail...
- 2 kudos
- 7877 Views
- 8 replies
- 0 kudos
Databricks Autoloader File Notification Not Working As Expected
Hello Everyone,In my project I am using databricks autoloader to incrementally and efficiently processes new data files as they arrive in cloud storage.I am using file notification mode with event grid and queue service setup in azure storage account...
- 7877 Views
- 8 replies
- 0 kudos
- 0 kudos
Hi @Sambit_S , I misread inputRows as inputFiles which aren't the same thing. Considering the limitation on Azure queue, if you are already at the limit then you may need to consider to switching to an event source such as Kafka or Event Hub to get b...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
