- 2382 Views
- 3 replies
- 0 kudos
Resolved! Databricks Workflow design
I have 7 - 8 different dlt pipelines which have to be run at the same time according to their batch type i.e. hourly and daily. Right now they are triggered effectively according to their batch type. I want to move to a next stage where I want to clu...
- 2382 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @VZLA , I got the idea. There will be a small change in the way, we will use it. Since we don't schedule the workflow in databricks we trigger it using the API. So I will pass a job parameter along with the trigger according to the timestamp wheth...
- 0 kudos
- 1051 Views
- 3 replies
- 0 kudos
History load from Source and
Hi As part of our requirement we wanted to load a huge historical data from the Source System to Databricks in Bronze and then process it to Gold, We wanted to use batch with read and Write so that the historical load is done and then for the delta o...
- 1051 Views
- 3 replies
- 0 kudos
- 0 kudos
I imported 16 TB of data using ADF. In this scenario I'd create a process that will extract from a source data using ADF and then execute the rest of logic to populate tables in the gold. For the new data I'd create a separate process using Autoloade...
- 0 kudos
- 2726 Views
- 4 replies
- 4 kudos
How to consume Fabric Datawarehouse inside a Databricks notebook
Hello,I'm having a hard time figuring out (and finding the right documentation) to be able to connect my databricks notebook to consume tables from a fabric datawarehouse. I've checked this, but seems to work only with onelake and this, but I'm not ...
- 2726 Views
- 4 replies
- 4 kudos
- 4 kudos
Hello, I would like to get a bit more options regarding reading Views. Using the abfss is fine for reading tables, but I don't know how to load Views, which are visible in the SQL Endpoint. Is there any alternative for connecting to Fabric and be abl...
- 4 kudos
- 1639 Views
- 3 replies
- 4 kudos
Redshift to Databricks Migration
Hi,I want a detailed plan steps to migrate my data from redshift to databricks.where to start, what to assess and what to migrate.It could really help me if you provide the detailed explaination on migration.Thanks in Advance.
- 1639 Views
- 3 replies
- 4 kudos
- 4 kudos
I migrated Oracle to Databricks and have an experience with Redshift. The cost and effort will depend on your technical stuck:- What do you use for ETL?- What do you use for data ingestion?- Reporting tools?In general, simplest steps are: data and mo...
- 4 kudos
- 4224 Views
- 1 replies
- 0 kudos
Deployed DABs job via Gitlab CICD. It is creating duplicate jobs.
We had error in DABs deploy and then subsequent retries resulted in a locked stateAnd as suggested in the logs, we use --force-lock option and the deploy succeededHowever, it created duplicate jobs for all assets in the bundle instead of updating the...
- 4224 Views
- 1 replies
- 0 kudos
- 0 kudos
@ahen When you used the --force-lock option during the Databricks Asset Bundle (DAB) deployment, it likely bypassed certain checks that would normally prevent duplicate resource creation. This option is used to force a deployment even when a lock is ...
- 0 kudos
- 2657 Views
- 6 replies
- 0 kudos
Resolved! Need urgent help and guidance on information/details with reference links on below topics:
Dear experts,I need urgent help and guidance on information/details with reference links on below topics:Steps on Package Installation with Serverless in Databricks.What are Delta Lake Connector with serverless ? How to run Delta Lake queries outside...
- 2657 Views
- 6 replies
- 0 kudos
- 0 kudos
Were you able to review the documentation provided here: https://docs.databricks.com/en/compute/serverless/dependencies.html#install-notebook-dependencies?
- 0 kudos
- 888 Views
- 2 replies
- 0 kudos
Databricks connect, set spark config
Hi, Iam using databricks connect to compute with databricks cluster. I need to set some spark configurations, namely spark.files.ignoreCorruptFiles. As I have experienced, setting spark configuration in databricks connect for the current session, has...
- 888 Views
- 2 replies
- 0 kudos
- 0 kudos
Have you tried setting it up in your code as: from pyspark.sql import SparkSession # Create a Spark session spark = SparkSession.builder \ .appName("YourAppName") \ .config("spark.files.ignoreCorruptFiles", "true") \ .getOrCreate() # Yo...
- 0 kudos
- 2377 Views
- 4 replies
- 4 kudos
Error when accessing 'num_inserted_rows' in Spark SQL (DBR 15.4 LTS)
Hello Databricks Community,I've encountered an issue while trying to capture the number of rows inserted after executing an SQL insert statement in Databricks (DBR 15.4 LTS). My code is attempting to access the number of inserted rows as follows: row...
- 2377 Views
- 4 replies
- 4 kudos
- 4 kudos
Hi,we are experiencing the same issue. We also turned on liquid clustering on table and we had additional checks on the inserted data information, which was really helpful.@GavinReeves3 did you manage to solve the issue?@MuthuLakshmi any idea? Thank ...
- 4 kudos
- 1976 Views
- 4 replies
- 3 kudos
Resolved! Unable to Create Alert Using API
Hi All, I'm trying to create an alert using the Databricks REST API, but I keep encountering the following error:Error creating alert: 400 {"message": "Alert name cannot be empty or whitespace"}:{"alert": {"seconds_to_retrigger": 0,"display_name": "A...
- 1976 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @zg ,You are sending the payload related to the new endpoint (/api/2.0/sql/alerts) to the old endpoint (/api/2.0/preview/sql/alerts).That are the docs of the old endpoint:https://docs.databricks.com/api/workspace/alertslegacy/createAs you can see ...
- 3 kudos
- 3528 Views
- 4 replies
- 0 kudos
Unity Catalog is missing column in Catalog Explorer
I have just altered one of my tables and added a column.ALTER TABLE tpch.customer ADD COLUMN C_CUSTDETAILS struct<key:string,another_key:string,boolean_key:boolean,extra_key:string,int_key:long,nested_object:struct<more:long,arrayOne:array<string>>>A...
- 3528 Views
- 4 replies
- 0 kudos
- 0 kudos
Can you share the steps for reproducing this issue? If it is reproducible, I can investigate.
- 0 kudos
- 1836 Views
- 3 replies
- 0 kudos
How to increase timeout in Databricks Workflows DBT task
Hi,I have a Databricks Workflows DBT task that targets a PRO SQL warehouse. However, the task fails with a "to many retries" error (see below) if the PRO SQL warehouse is not up and running when the task starts. How can I increase the timeout or allo...
- 1836 Views
- 3 replies
- 0 kudos
- 0 kudos
One option seems to be to reference a custom "profiles.yml" in the job configuration and specify a custom DBT Databricks connector timeout there (https://docs.getdbt.com/docs/core/connect-data-platform/databricks-setup#additional-parameters).However,...
- 0 kudos
- 1590 Views
- 1 replies
- 0 kudos
Joining tables across DLT pipelines
How can I join a silver table (s1) from a DLT pipeline (D1) to another silver table (S2) from a different DLT pipeline (D2)?#DLT #DeltaLiveTables
- 1590 Views
- 1 replies
- 0 kudos
- 0 kudos
@Mkk1 Did you get to get this completed? We are in the similar situation, how did you get to acheive this?
- 0 kudos
- 2735 Views
- 3 replies
- 0 kudos
How to compare two databricks notebooks which are in different folders? note we dont have GIT setup
we would to like compare two notebooks which are in different folders , we are yet set up a GIT repo for these folders.?what are the other options we have to compare two notebooks?thanksNAnda
- 2735 Views
- 3 replies
- 0 kudos
- 4757 Views
- 3 replies
- 7 kudos
Resolved! DLT Apply Changes Tables corrupt
Got a weird DLT error.Test harness using the new(ish) 'Apply Changes from Snapshot' Functionality and DLT Serverless (Current Channel). Azure Aus East Region.Has been working for several months without issue - but within the last week these DLT table...
- 4757 Views
- 3 replies
- 7 kudos
- 7 kudos
We have an open ticket on this issue. The issue is caused by the maintenance pipeline renaming the backing table. We expect the fix to be rolled out soon for this issue.
- 7 kudos
- 1296 Views
- 1 replies
- 0 kudos
Urgent !! Need information/details and reference link on below two topics:
Dear experts,I need urgent help and guidance on information/details with reference links on below topics:Steps on Package Installation with Serverless in Databricks.What are Delta Lake Connector with serverless ? How to run Delta Lake queries outside...
- 1296 Views
- 1 replies
- 0 kudos
- 0 kudos
Seems like a duplicate: https://community.databricks.com/t5/data-engineering/urgent-need-information-details-and-reference-link-on-below-two/td-p/107260
- 0 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 493 | |
| 349 | |
| 287 |
