- 5833 Views
- 4 replies
- 0 kudos
How do i retrieve timestamp data from history in databricks sql not using DELTA table,its data is coming from SAP
I am not using delta tables my data is from SAP ..how do i retrieve timestamp(history) dynamically from SAP table using databricks SQL
- 5833 Views
- 4 replies
- 0 kudos
- 0 kudos
Hello,If you’re trying to bring timestamp data or any other SAP Table from SAP (SAP HANA) into Databricks, our SAP HANA to Databricks Connector can help streamline this process. The connector enables you to extract data directly from SAP HANA tables ...
- 0 kudos
- 1910 Views
- 2 replies
- 0 kudos
Identify a job name in case notebook is triggered from another notebook/
I am running an another notebook from one of the notebook as below. The main notebook is scheuled with a job/workflow.dbutils.notebook.run("./ABC/XYZ/another_notebook", 1000) Usually to get the workflow/job name i use any of the below 2 options. 1. ...
- 1910 Views
- 2 replies
- 0 kudos
- 0 kudos
Thanks Alberto. I am aware of that solution but trying to get an option to get job name in the scenario i gave. So i assume there is no way to get the job name in that scenario.
- 0 kudos
- 2645 Views
- 3 replies
- 0 kudos
Configure cluster single node with service principal in Azure
Hi! We can configure a cluster single node with single user as service principal using this command:databricks clusters create --json ' { "cluster_name": "my-cluster", "spark_version": "13.3.x-scala2.12", "node_type_id": "Standard_DS3_v2"...
- 2645 Views
- 3 replies
- 0 kudos
- 0 kudos
Can you share an screenshot of the cluster on the UI side please, if you are creating a cluster assigned to a service principal this means that it should have already the Private preview enabled. If the private preview is not enabled it will throw th...
- 0 kudos
- 16539 Views
- 5 replies
- 3 kudos
Resolved! Feature request delta tables : drop duplicate rows
A deltaTable.dropDuplicates(columns) would be a very nice feature, simplifying the complex procedures that are suggested online. Or am I missing any existing procedures that can be done withouth merge operations or similar?
- 16539 Views
- 5 replies
- 3 kudos
- 3 kudos
I created a feature request in the delta table project: [Feature Request] data deduplication on existing delta table · Issue #1767 · delta-io/delta (github.com)
- 3 kudos
- 2297 Views
- 1 replies
- 0 kudos
Read Data from Updating Streaming Table
We are working on a project with the following setup:The Bronze table is append-only.The Silver table uses "APPLY CHANGES INTO" from the Bronze table and implements SCD Type 1, meaning records can be updated at any time.We are trying to use the Silv...
- 2297 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @smit_tw, you can consider the following approach: Materialized View: Convert the Gold table to a materialized view instead of a streaming table. This approach will allow you to handle updates and deletes in the Silver table without causing errors...
- 0 kudos
- 2599 Views
- 7 replies
- 2 kudos
RDD Parallelism without delta_log
I've set up my script to be able to use a multinode cluster, but am running into an issue when iterating on a list of .json files to sink to sql table via a JDBC driver. The primary response is that a delta_log file (that I can't see in my blob conta...
- 2599 Views
- 7 replies
- 2 kudos
- 2 kudos
Could it be that our cluster doesn't have the delta lake libraries loaded?maven JAR coordinates: Maven: io.delta:delta-core_2.12:2.4.0
- 2 kudos
- 11863 Views
- 8 replies
- 3 kudos
DatabricksSession broken for 15.1
This code fails with exception:[NOT_COLUMN_OR_STR] Argument `col` should be a Column or str, got Column.File <command-4420517954891674>, line 7 4 spark = DatabricksSession.builder.getOrCreate() 6 df = spark.read.table("samples.nyctaxi.trips") ---->...
- 11863 Views
- 8 replies
- 3 kudos
- 3 kudos
I also get the same for runtime 13.3 LTS. The same code with 15.2 LTS seems to work.df.withColumn("new_col", concat("col1", lit("-"), "col2"))
- 3 kudos
- 6234 Views
- 7 replies
- 0 kudos
Resolved! Not able to insert into identity column through spark
I have a delta table with identity column and not able to insert data using spark. I am using 15.4LTS. any idea what needed to be done Table Name : accountColumn Def: account_dimension_id BIGINT GENERATED BY DEFAULT AS IDENTITY,df = spark.read.format...
- 6234 Views
- 7 replies
- 0 kudos
- 0 kudos
Thanks. Still giving data mismatch error -table : accountaccount_dimension_id BIGINT GENERATED BY DEFAULT AS IDENTITY,df = spark.read.format("csv").option("header", "true").load("abfss://databricks-storage@sa14127e1dv0101.dfs.core.windows.net/catalog...
- 0 kudos
- 973 Views
- 0 replies
- 0 kudos
Digital Natives on Databricks our Experience.
They say we should craft solutions tailored to the unique journeys of those we aim to support, paving the way for their success. Digital Native enterprises, with their distinct needs and ambitions in Data and AI, often seek clarity on where to start ...
- 973 Views
- 0 replies
- 0 kudos
- 1692 Views
- 0 replies
- 1 kudos
Learn Data Engineering on Databricks Step By Step.
They say we should build bridges along the paths we’ve already traveled, making it easier for others to follow. Learning Data Engineering has often been a confusing journey for many, especially when it comes to figuring out where to start.I faced thi...
- 1692 Views
- 0 replies
- 1 kudos
- 6075 Views
- 4 replies
- 1 kudos
Failed to Create a Workspace Service Principal
I am attempting to add a service principal to my workspace. Each time I attempt this though I get the error message "Failed to create service principal. Please try again." There are no additional resources regarding addressing this error. Has anyone ...
- 6075 Views
- 4 replies
- 1 kudos
- 1 kudos
Got it, can you try to create the SP at workspace level and share with me the timestamp and if possible the workspace id and cloud region to take a look on my side?
- 1 kudos
- 2101 Views
- 2 replies
- 0 kudos
Avoiding full refresh in CDF data on DLT updates
Hi,We have a DLT pipeline with the following source code:import dlt from pyspark.sql.functions import col, from_json from pyspark.sql.types import LongType, StringType, StructField, StructType KAFKA_TOPIC = "..." BOOTSTRAP_SERVERS = "..." # Define ...
- 2101 Views
- 2 replies
- 0 kudos
- 0 kudos
Thank you, @Alberto_Umana,One of the main reasons we chose to use DLT was its ease of setup and monitoring, as well as the ability to avoid creating custom Spark Streaming jobs. From what I understand, your suggestion is to use both DLT and Spark Str...
- 0 kudos
- 8905 Views
- 13 replies
- 1 kudos
Import failed with error: Could not deserialize (No such file or directory)
I tried to import a notebook (.dbc) into my workspace today and this error popped up. I have never encountered this before. Is this a temporary bug or something has changed for Community Version? Error Message:Import failed with error: Could not dese...
- 8905 Views
- 13 replies
- 1 kudos
- 1 kudos
Importing .DBC file is not working in Community edition. Please guide if we can resolve this issue. Thanks.
- 1 kudos
- 5047 Views
- 3 replies
- 0 kudos
Handling single table in multiple dlt pipeines
So this is my workflow which i am trying to buildThere are 2 databricks job Lets say A and B.So Job A has 3 tasks and the3rd task Check if target table is present or not if present it updates its schema if any changes or normally refreshes the pipeli...
- 5047 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @ashraf1395 Yes, your understanding is partially correct. Let me clarify:Only one pipeline can own and manage a target table, including operations like schema evolution, maintenance, and refreshes, etcWhen other pipelines are mentioned as "produci...
- 0 kudos
- 999 Views
- 1 replies
- 0 kudos
Updating a streaming table in dlt
Can we update a streaming table in dlt my source and target will be same. That is the update should be made on the same table. If yes then can you guide me how.I tried append_flow but it just appends dataCDC I am not sure whether we can have both tar...
- 999 Views
- 1 replies
- 0 kudos
- 0 kudos
You can define the CDC flow to update the streaming table. This involves reading from the same table and applying changes. @Dlt.table( name="my_streaming_table", comment="This table is updated using CDC", table_properties={"quality": "sil...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
