- 834 Views
- 7 replies
- 1 kudos
Resolved! copy into - statement executed - no records inserted
I am able to run the copy command but records are not inserting and there is no error raised on execution.
- 834 Views
- 7 replies
- 1 kudos
- 1 kudos
Records not inserted with the COPY INTO statement usually indicate either no matching source data or unmet conditions in the query.
- 1 kudos
- 4446 Views
- 3 replies
- 1 kudos
Resolved! Spark UI simulator is not available online
About 2 weeks ago I started course on "Optimizing Apache Spark on Databricks" from official Databricks academy. It is heavily based on Spark UI simulator experiments that were available here: https://www.databricks.training/spark-ui-simulator and for...
- 4446 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @LandanG The link is still not working. Any idea when it will be live again?http://www.databricks.training/spark-ui-simulator/GitHub - databricks-academy/spark-ui-simulator: Apache Spark UI simulator with 30+ educational experiments, pre-recorded ...
- 1 kudos
- 6932 Views
- 3 replies
- 3 kudos
Resolved! Delta live tables - foreign keys
I'm creating ingestions using delta live tables, the dlt support the use of schema, with constraints like foreign keys. The problem is: how can i create foreign keys between the same pipeline, that has no read/write relation, but has foreign key rela...
- 6932 Views
- 3 replies
- 3 kudos
- 3 kudos
Was there any update from Databricks to @User12350 comment above? We have the exact issue - you cannot add foreign key constraint on materialized view (even if its not enforced).
- 3 kudos
- 434 Views
- 2 replies
- 0 kudos
Resolved! Which of my Lakeflow pipelines are using the same gateway?
Does anyone know of a way to see what Lakeflow pipelines are using the same gateway? We have a gateway connected to a SQL Server that serves multiple individual pipelines but I cannot find a way to see what those are. I've tried system tables. Any in...
- 434 Views
- 2 replies
- 0 kudos
- 0 kudos
Thanks @szymon_dybczak - this worked perfectly. This is what I ended up throwing together to see what I was looking for:from databricks.sdk import WorkspaceClient url = f"{DATABRICKS_HOST}/api/2.0/pipelines" headers = wc.config.authenticate() wc = ...
- 0 kudos
- 12803 Views
- 8 replies
- 7 kudos
Reading Iceberg table present in S3 from databricks console using spark given none error .
Hi Team , I am facing issue while reading iceberg table from S3 and getting none error when read the data . below steps I followed .Added Iceberg Spark connector library to your Databricks cluster. 2. Cluster Configuration to Enable Iceberg ...
- 12803 Views
- 8 replies
- 7 kudos
- 7 kudos
https://community.databricks.com/t5/technical-blog/secure-migration-of-aws-glue-iceberg-tables-to-databricks-unity/ba-p/148414
- 7 kudos
- 520 Views
- 4 replies
- 1 kudos
Resolved! Table listener
Hello Community,I would like to ask whether it’s possible to define a job that checks for updates in a table at a specified frequency.Here is my use case:Data is uploaded to a table located in Catalog A, Schema B, Table C (a.b.c).I need to transfer t...
- 520 Views
- 4 replies
- 1 kudos
- 1 kudos
Also I have a question about failure handling in case of triggered jobs. Let's say new data has come to the source table and job failed for any reason. If I rerun it or the next batch of data comes to source table, will the data from the failed job b...
- 1 kudos
- 675 Views
- 1 replies
- 3 kudos
Resolved! Asset Bundles: Extending complex variables
Hi all,In my Asset Bundle I have some setting for a cluster like in the example at Substitutions and variables in Databricks Asset Bundles (section Define a complex variable).Now I want to add some additional attribute when using this variables, like...
- 675 Views
- 1 replies
- 3 kudos
- 3 kudos
Hi @Daniel_dlh ,you can try to use YAML anchor, have a look at this example:before: databricks.yml variables: my_cluster: description: "Base cluster configuration" default: spark_version: "15.4.x-scala2.12" node_type_id: "Standa...
- 3 kudos
- 1577 Views
- 1 replies
- 2 kudos
Resolved! Cost-Effective Databricks Pipeline for API Ingestion - Best Practices?
Hi Community,As a senior data engineer migrating ETL workloads to Databricks (with Unity Catalog and Delta Lake), I'm building a cost-effective pipeline to ingest data from a REST API. Goals: minimize DBU costs, handle incremental loads, ensure scala...
- 1577 Views
- 1 replies
- 2 kudos
- 2 kudos
HI @TheDataMaverick ,The most efficient approach for your REST API ingestion pipeline on Databricks is to use an external service like Azure Functions (or AWS Lambda) to handle API calls, then land raw JSON/Parquet in ADLS/S3 for Auto Loader ingestio...
- 2 kudos
- 393 Views
- 1 replies
- 1 kudos
Resolved! How to manage shared parameters between different jobs?
I have few jobs that run serverless. Currently there is only one task for each job. These tasks have multiple parameters, some of which are shared between the jobs, some are unique to each jobs.Is it possible to define all the parameters for all the ...
- 393 Views
- 1 replies
- 1 kudos
- 1 kudos
It's weird, I have tried your definition and it works well when I repleace ${value1} and ${value2}. Not sure how you are using those, you should prefix with var. or use some static. values, like I did.
- 1 kudos
- 612 Views
- 2 replies
- 0 kudos
Servicenow integration with AWS databricks
Hi Databricks Community,We’re running Databricks on AWS and would like to improve operational incident management for production workloads.Is there any official Databricks documentation or recommended approach to integrate with ServiceNow for automat...
- 612 Views
- 2 replies
- 0 kudos
- 0 kudos
@anshu_roy Hello, Thanks for the reply,Any sample code for rest API (that will fetch the job long along with name and time ) would be helpful.
- 0 kudos
- 13419 Views
- 2 replies
- 0 kudos
- 13419 Views
- 2 replies
- 0 kudos
- 0 kudos
These are action methods that return data -first() : Returns the very first row of the dataframe as a single row.head() : This does the same as first(), returns the first rowhead(n): Returns an array or list of the first n rowstake(n): Similar to hea...
- 0 kudos
- 364 Views
- 1 replies
- 1 kudos
Resolved! new to databricks
Hi this is phani can some one guide me with best resources to learn databricks as a fresherThank you,Phaneendra
- 364 Views
- 1 replies
- 1 kudos
- 1 kudos
Welcome to Databricks community. You should head to https://www.databricks.com/learn and practice them here on the Free edition https://www.databricks.com/learn/free-edition Start with the free, short fundamentals courses to build a strong base: tak...
- 1 kudos
- 1828 Views
- 3 replies
- 1 kudos
ERROR: Could not find a version that satisfies the requirement
I'm using Databricks workflows along with asset bundle to run my pipeline and recently we moved our pipeline from development to acceptance. We are also using JFrog as our package artifactory.Even though I do see the release candidate version under t...
- 1828 Views
- 3 replies
- 1 kudos
- 1 kudos
I do have init script install on the cluster level as well but still getting the same error, any other solutions for this error?And I have not made any changes to my asset bundle it was running well other day all of sudden I got this error.
- 1 kudos
- 574 Views
- 2 replies
- 0 kudos
Resolved! unable to create new catalog in databricks free edition
unable to create new catalog in databricks free edition
- 574 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @AbhiBange ,I had the same issue using UI approach. Open notebook and use SQL to create catalog. It will work CREATE CATALOG your_catalog_name
- 0 kudos
- 7180 Views
- 3 replies
- 4 kudos
Resolved! How do I track databricks cluster users ?
Hi, is there a way to find out/monitor which users has used my cluster, how long and how many times in an azure databricks workspace ?
- 7180 Views
- 3 replies
- 4 kudos
- 4 kudos
Do you have any query we can get the details from system table who all used the cluster in between this window
- 4 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 567 | |
| 349 | |
| 287 |
