- 8914 Views
- 4 replies
- 3 kudos
Cluster Failed to Start - Cluster scoped init scrip failed: Script exit status is non-zero
i have been using cluster scoped init script for around 1 year already and everything is working fine. But suddenly, Databricks cluster has failed to restart since last week Thursday (13th June 2024). It returns this error:” Failed to add 2 container...
- 8914 Views
- 4 replies
- 3 kudos
- 3 kudos
Just maybe - there is no outbound connection on DEV from Cluster VNET to URL you are trying to get ? You can spin ALl purpose cluster and try testing connection with %sh magic command
- 3 kudos
- 6557 Views
- 1 replies
- 0 kudos
Resolved! Getting com.databricks.client.jdbc.Driver is not found error while connecting to databricks
Hi Community,I need help regarding the class not found issue. I'm trying to connect databricks in python via jaydebeapi, provided proper class name `com.databricks.client.jdbc.Driver` and jarpath `databricks-jdbc-2.6.34.jar`, but i'm getting com.data...
- 6557 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @HASSAN_UPPAL123 The class name is correct, for the jar please try downloading the latest from here This issue may also be a classpath issue were the jar is not exported correctly in your client setup, I see similar issues/ suggested solutions ...
- 0 kudos
- 9354 Views
- 2 replies
- 0 kudos
export notebook
Hi,I want to export notebook in python programming.is there a way to leverage databricks cli in python.Or any other way to export the notebook to my local PC.
- 9354 Views
- 2 replies
- 0 kudos
- 0 kudos
Is there a way to export a notebook though terraform? If so please provide examples?With terraform-provider-databricks.exe we are able to export all the notebooks from workspace but not a single notebook. Any suggestions ?
- 0 kudos
- 6672 Views
- 5 replies
- 1 kudos
Resolved! Retrieve job-level parameters in spark_python_task (not notebooks)
Hello,I would like to use job parameters in spark_python_task (not notebook_task), does anyone know how to retrieve these parameters inside pure Python ?I tried:1/ dbutils.widgets.get("debug"), got error: com.databricks.dbutils_v1.InputWidgetNotD...
- 6672 Views
- 5 replies
- 1 kudos
- 1 kudos
@daniel_sahal just tested {{job.parameters.[name]}}, it works, thanks again !
- 1 kudos
- 4220 Views
- 2 replies
- 0 kudos
Resolved! Cannot pass arrays to spark.sql() using named parameter markers
Hello all,I am attempting to use named parameter markers as shown in this article: https://docs.databricks.com/en/sql/language-manual/sql-ref-parameter-marker.html#named-parameter-markersI can pass strings and numbers in perfectly fine, but the issue...
- 4220 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @runninsavvy , The following code sample can be used in such case val argArray = Array(1, 2, 3) val argMap = Map("param" -> argArray.mkString(",")) spark.sql("SELECT 1 IN (SELECT explode(split(:param, ',')))",argMap).show()
- 0 kudos
- 2794 Views
- 1 replies
- 0 kudos
DLT fails with Queries with streaming sources must be executed with writeStream.start();
Hi guys!I am having an issue with passing the "streaming flow" between layers of the DLT.first layer "ETD_Bz" is passing through, but then "ETD_Flattened_Bz" is failing with "pyspark.errors.exceptions.captured.AnalysisException: Queries with streamin...
- 2794 Views
- 1 replies
- 0 kudos
- 0 kudos
UPDATE: tried adding writeStream.start() like error suggested + as per other posts and ended up with following error/code: @dlt.table( name="ETD_Bz", temporary=False)def Bronze(): return (spark.readStream .format("delta") ...
- 0 kudos
- 4902 Views
- 1 replies
- 0 kudos
Automatic conversion of timestamp to the default timezone
I am encountering the issue when ingesting data from adls xml or json files to process them via Pyspark (Autoloader or just reading df). The timestamp is automatically converted to the default timezone.And I have dynamically timezone values. Did any...
- 4902 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Retired_mod , The point is that in the aim of our project, we need the timestamp attribute to be as they are from the source system. So basically our aim would be to have the attribute without the timezone conversion. I did the below tests so fa...
- 0 kudos
- 1944 Views
- 0 replies
- 0 kudos
DLT live tables error while reading file from datalake gen2
I am getting following error while running cell in python. Same file is run fine when i upload json file into databrics and then give this path to df.read syntex while reading it. When i use DLT for same file which is in datalake it gives me follo...
- 1944 Views
- 0 replies
- 0 kudos
- 12547 Views
- 2 replies
- 2 kudos
Delete Delta Live Table Completely
I've been struggling with figuring out how to delete a managed delta live table. If I run a drop command in Databricks SQL I get:[STREAMING_TABLE_OPERATION_NOT_ALLOWED.DROP_DELTA_LIVE_TABLE] The operation DROP is not allowed: The operation does not a...
- 12547 Views
- 2 replies
- 2 kudos
- 947 Views
- 0 replies
- 0 kudos
Databricks Cross platform data access
Hi Team,We have a requirement . data storage on the S3 platform, while our databricks is hosted on Azure.Our objective is to access the data from the S3 location.Could you kindly provide us with the most suitable approach for this scenario? ex- exte...
- 947 Views
- 0 replies
- 0 kudos
- 6370 Views
- 1 replies
- 0 kudos
Databricks asset bundles error "failed to instantiate provider"
Hi all,I'm trying to deploy with databricks asset bundles. When running bundle deploy, the process fails with following error message:failed execution pid=25092 exit_code=1 error="terraform apply: exit status 1\n\nError: failed to read schema for dat...
- 6370 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi,sorry for my late answer and thank you for your help. I was on parental leave in the meantime.I solved the problem by installing the latest Databricks extension. With the new extension and updated Databricks-CLI I got the same error.However: The l...
- 0 kudos
- 10020 Views
- 4 replies
- 0 kudos
How to deal with delete records from the source Files in DLT .
Can apply_changes feature deal with deleted records in incoming source Files?By delete I mean record is being removed (Not a soft delete with Flag).If not, how to automate with deleting records from Bronze Streaming table based on source Files.
- 10020 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi Manoj,Did you get the solution or design change for this problem. We have 200K files on to S3 bucket and when there is change in upstream app we get new feed, feed name is fixed. On DLT we should have only new records from replaced file but in dlt...
- 0 kudos
- 2945 Views
- 1 replies
- 0 kudos
Credential passthrough and Hive metastore table access controls are deprecated
Hello,Based on the recent platform release, the credential passthrough will be deprecated for runtime 15.0 and later. Our current setup involves using Databricks alongside AWS Glue and Athena, i.e. registering delta tables in AWS Glue and running oth...
- 2945 Views
- 1 replies
- 0 kudos
- 3222 Views
- 2 replies
- 0 kudos
Connecting to ADLS azure storage and reading csv file from adls
Hi Friends , How are you all? I am Anwitha. I am a newbie in azure data bricks community. I am stuck with this error while practising with azure storage account. Could anyone throw a light on this error and provide a solution?I tried connecting to or...
- 3222 Views
- 2 replies
- 0 kudos
- 0 kudos
Ermm, you might want to edit your post and remove that storage account access key! In combination with the storage account name (that is also in your post), everyone will be able to access it if it isn't behind a firewall! Btw, you might want to stud...
- 0 kudos
- 2255 Views
- 3 replies
- 0 kudos
Error while reading steaming source from DLT Pipeline
Hi All, I am getting this error, when I am reading the streaming sourceFull load - It loadsIncremental load - am facing this errorThis is the piece of code that am using def gim_suppliers_ln(): logger.info("Starting __cn_gim_suppliers") overall...
- 2255 Views
- 3 replies
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
