- 3458 Views
- 3 replies
- 0 kudos
Overwriting a delta table using DLT
Hello,We are trying to ingest bunch of csv files that we receive on daily basis using DLT, we chose streaming table for this purpose since streaming table is append only records keep adding up on a daily basis which will cause multiple rows in downst...
- 3458 Views
- 3 replies
- 0 kudos
- 0 kudos
In your scenario, if the data loaded on day 2 also includes all the data from day 1, you can still apply a "remove duplicates" logic. For instance, you could compute a hashdiff by hashing all the columns and use this to exclude rows you've already se...
- 0 kudos
- 3745 Views
- 1 replies
- 1 kudos
Resolved! Read and process large CSV files that updates regularly
I've got a lot of large CSV files (> 1 GB) that updates regularly (stored in Data Lake Gen 2). The task is to concatenate these files into a single dataframe that is written to parquet format. However, since these files updates very often I get a rea...
- 3745 Views
- 1 replies
- 1 kudos
- 1 kudos
@Kjetil Since they are getting updated often then IMO making a copy would make sense.What you could try is to create Microsoft.Storage.BlobCreated event to replicate the .CSV into the secondary bucket.However, best practice would be to have some kind...
- 1 kudos
- 1332 Views
- 0 replies
- 0 kudos
Understanding this Ipython related error in cluster logs
Hi Databricks Community !I'm having this error from a cluster's logs : [IPKernelApp] ERROR | Exception in control handler:Traceback (most recent call last):File "/databricks/python/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 334, in p...
- 1332 Views
- 0 replies
- 0 kudos
- 8531 Views
- 4 replies
- 3 kudos
Cluster Failed to Start - Cluster scoped init scrip failed: Script exit status is non-zero
i have been using cluster scoped init script for around 1 year already and everything is working fine. But suddenly, Databricks cluster has failed to restart since last week Thursday (13th June 2024). It returns this error:” Failed to add 2 container...
- 8531 Views
- 4 replies
- 3 kudos
- 3 kudos
Just maybe - there is no outbound connection on DEV from Cluster VNET to URL you are trying to get ? You can spin ALl purpose cluster and try testing connection with %sh magic command
- 3 kudos
- 6305 Views
- 1 replies
- 0 kudos
Resolved! Getting com.databricks.client.jdbc.Driver is not found error while connecting to databricks
Hi Community,I need help regarding the class not found issue. I'm trying to connect databricks in python via jaydebeapi, provided proper class name `com.databricks.client.jdbc.Driver` and jarpath `databricks-jdbc-2.6.34.jar`, but i'm getting com.data...
- 6305 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @HASSAN_UPPAL123 The class name is correct, for the jar please try downloading the latest from here This issue may also be a classpath issue were the jar is not exported correctly in your client setup, I see similar issues/ suggested solutions ...
- 0 kudos
- 9132 Views
- 2 replies
- 0 kudos
export notebook
Hi,I want to export notebook in python programming.is there a way to leverage databricks cli in python.Or any other way to export the notebook to my local PC.
- 9132 Views
- 2 replies
- 0 kudos
- 0 kudos
Is there a way to export a notebook though terraform? If so please provide examples?With terraform-provider-databricks.exe we are able to export all the notebooks from workspace but not a single notebook. Any suggestions ?
- 0 kudos
- 6286 Views
- 5 replies
- 1 kudos
Resolved! Retrieve job-level parameters in spark_python_task (not notebooks)
Hello,I would like to use job parameters in spark_python_task (not notebook_task), does anyone know how to retrieve these parameters inside pure Python ?I tried:1/ dbutils.widgets.get("debug"), got error: com.databricks.dbutils_v1.InputWidgetNotD...
- 6286 Views
- 5 replies
- 1 kudos
- 1 kudos
@daniel_sahal just tested {{job.parameters.[name]}}, it works, thanks again !
- 1 kudos
- 4000 Views
- 2 replies
- 0 kudos
Resolved! Cannot pass arrays to spark.sql() using named parameter markers
Hello all,I am attempting to use named parameter markers as shown in this article: https://docs.databricks.com/en/sql/language-manual/sql-ref-parameter-marker.html#named-parameter-markersI can pass strings and numbers in perfectly fine, but the issue...
- 4000 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @runninsavvy , The following code sample can be used in such case val argArray = Array(1, 2, 3) val argMap = Map("param" -> argArray.mkString(",")) spark.sql("SELECT 1 IN (SELECT explode(split(:param, ',')))",argMap).show()
- 0 kudos
- 2654 Views
- 1 replies
- 0 kudos
DLT fails with Queries with streaming sources must be executed with writeStream.start();
Hi guys!I am having an issue with passing the "streaming flow" between layers of the DLT.first layer "ETD_Bz" is passing through, but then "ETD_Flattened_Bz" is failing with "pyspark.errors.exceptions.captured.AnalysisException: Queries with streamin...
- 2654 Views
- 1 replies
- 0 kudos
- 0 kudos
UPDATE: tried adding writeStream.start() like error suggested + as per other posts and ended up with following error/code: @dlt.table( name="ETD_Bz", temporary=False)def Bronze(): return (spark.readStream .format("delta") ...
- 0 kudos
- 4758 Views
- 1 replies
- 0 kudos
Automatic conversion of timestamp to the default timezone
I am encountering the issue when ingesting data from adls xml or json files to process them via Pyspark (Autoloader or just reading df). The timestamp is automatically converted to the default timezone.And I have dynamically timezone values. Did any...
- 4758 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Retired_mod , The point is that in the aim of our project, we need the timestamp attribute to be as they are from the source system. So basically our aim would be to have the attribute without the timezone conversion. I did the below tests so fa...
- 0 kudos
- 1844 Views
- 0 replies
- 0 kudos
DLT live tables error while reading file from datalake gen2
I am getting following error while running cell in python. Same file is run fine when i upload json file into databrics and then give this path to df.read syntex while reading it. When i use DLT for same file which is in datalake it gives me follo...
- 1844 Views
- 0 replies
- 0 kudos
- 12332 Views
- 2 replies
- 2 kudos
Delete Delta Live Table Completely
I've been struggling with figuring out how to delete a managed delta live table. If I run a drop command in Databricks SQL I get:[STREAMING_TABLE_OPERATION_NOT_ALLOWED.DROP_DELTA_LIVE_TABLE] The operation DROP is not allowed: The operation does not a...
- 12332 Views
- 2 replies
- 2 kudos
- 879 Views
- 0 replies
- 0 kudos
Databricks Cross platform data access
Hi Team,We have a requirement . data storage on the S3 platform, while our databricks is hosted on Azure.Our objective is to access the data from the S3 location.Could you kindly provide us with the most suitable approach for this scenario? ex- exte...
- 879 Views
- 0 replies
- 0 kudos
- 6202 Views
- 1 replies
- 0 kudos
Databricks asset bundles error "failed to instantiate provider"
Hi all,I'm trying to deploy with databricks asset bundles. When running bundle deploy, the process fails with following error message:failed execution pid=25092 exit_code=1 error="terraform apply: exit status 1\n\nError: failed to read schema for dat...
- 6202 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi,sorry for my late answer and thank you for your help. I was on parental leave in the meantime.I solved the problem by installing the latest Databricks extension. With the new extension and updated Databricks-CLI I got the same error.However: The l...
- 0 kudos
- 9696 Views
- 4 replies
- 0 kudos
How to deal with delete records from the source Files in DLT .
Can apply_changes feature deal with deleted records in incoming source Files?By delete I mean record is being removed (Not a soft delete with Flag).If not, how to automate with deleting records from Bronze Streaming table based on source Files.
- 9696 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi Manoj,Did you get the solution or design change for this problem. We have 200K files on to S3 bucket and when there is change in upstream app we get new feed, feed name is fixed. On DLT we should have only new records from replaced file but in dlt...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
