- 2997 Views
- 2 replies
- 1 kudos
running notebook job from remote github repository fails, but do not fail on python script type
Hi allI am trying to run a notebook from a remote repository, but the job fails. I setup the job as follows:my project structure is as such:but the output i get is like such: The thing is if i set the job type to "Python Script" i dont encounter this...
- 2997 Views
- 2 replies
- 1 kudos
- 1 kudos
@NielsMH if you want to run your jobs based o job name, please use new preview service that databricks released which are DAB format. there you can run your job based on your job name.remote repo in the sense, are you using github actions or api, loo...
- 1 kudos
- 1666 Views
- 1 replies
- 0 kudos
Query Load/ Warehouse Load missing
For observability, warehouse load and query load percentage are two major requirements. How do I fetch those details?
- 1666 Views
- 1 replies
- 0 kudos
- 1767 Views
- 1 replies
- 1 kudos
Query Hash missing
From the following Databricks API /api/2.0/preview/sql/queries query_hash is missing from the actual response.But the sample response mentioned in the API documentation has it.{ "count": 0, "page": 0, "page_size": 0, "results": [ { ...
- 1767 Views
- 1 replies
- 1 kudos
- 1955 Views
- 1 replies
- 1 kudos
Databricks cluster automated
Is there any way to automatically start a Databricks cluster when an event occurs, such as the cluster terminating for some reason, and have the Databricks cluster restart automatically thereafter ? It should avoid manual start.
- 1955 Views
- 1 replies
- 1 kudos
- 3486 Views
- 1 replies
- 0 kudos
problem with workspace after metastore deleted
I am completely new to Databricks AWS and start working on it a week ago. Pls excuse me if I ask or did something silly.I created a workspace and a single node cluster for testing. A metastore was created from Databricks quickstart and it was automa...
- 3486 Views
- 1 replies
- 0 kudos
- 0 kudos
I restarted the compute node and this problem went away.ErrorClass=METASTORE_DOES_NOT_EXIST] Metastore 'b11fb1a0-a462-4dfb-b91b-e0795fde10b0' does not exist.New question: I am testing Databricks with non-AWS S3 object storage. I can access the non-A...
- 0 kudos
- 3997 Views
- 3 replies
- 1 kudos
drop duplicates within watermark
Recently we are using structured streaming to ingest data. We want to use watermark to drop duplicated event. But We encountered some wired behavior and unexpected exception. Anyone can help me to explain what is the expected behavior and how should ...
- 3997 Views
- 3 replies
- 1 kudos
- 5884 Views
- 2 replies
- 1 kudos
Resolved! Read zstd file from Databricks
I just started to read `zstd` compressed file in Databricks on Azure, Runtime 14.1 on Spark 3.5.0I've set PySpark commands as followspath = f"wasbs://{container}@{storageaccount}.blob.core.windows.net/test-zstd" schema = "some schema" df = spark.read...
- 5884 Views
- 2 replies
- 1 kudos
- 1 kudos
The available compression types are format dependent.For json, zstd is not (yet) available, whereas for parquet it is.
- 1 kudos
- 3323 Views
- 0 replies
- 0 kudos
Can Error Message be un Redacted
I there a way to un-redact the logging of error message ?Alternatively would be nice to have access to the source code of involved classes like : com.databricks.backend.common.util.CommandLineHelper or com.databricks.util.UntrustedUtils I'm getting t...
- 3323 Views
- 0 replies
- 0 kudos
- 5609 Views
- 1 replies
- 1 kudos
How to schedule/refresh databricks alerts using REST API?
Hi, I am deploying Databricks SQL alerts using REST API. But I can't seem to figure out how to schedule their refresh task.I went through the documentation it says "Alerts can be scheduled using the sql_task type of the Jobs API, e.g. Jobs/Create"How...
- 5609 Views
- 1 replies
- 1 kudos
- 1 kudos
What they mention in the API docs is that you can create a job with sql_task of type Alert. To make it easier you can try creating the job first in the UI first and downloading the JSON config. Here is an example with the main parameters that should ...
- 1 kudos
- 1082 Views
- 0 replies
- 0 kudos
Small files and discrepancy in S3 vs catalog
Hello all,I'm in the process of optimizing my tables and I'm running into a confusing situation. I have a table named "trace_messages_fg_streaming_event". If I navigate to the Databricks catalog, it shows stats:Size: 6.7GB, Files: 464But when I look ...
- 1082 Views
- 0 replies
- 0 kudos
- 11889 Views
- 1 replies
- 1 kudos
Shared access vs Single user access mode
I am running a notebook to get secret value from GCP Secret Manager. This is working well with Single user Access Mode, however it fail when i use a cluster with Shared Access mode. I have specified the same GCP service account on both of these clust...
- 11889 Views
- 1 replies
- 1 kudos
- 1 kudos
Thanks for your response.I am using a cloud service account (same account that was used to create the workspace) on the cluster properties in case of both the single user cluster and on the shared user cluster. This service account has all the necess...
- 1 kudos
- 17043 Views
- 6 replies
- 0 kudos
Specify bottleneck for databricks cluster
Hi, Im trying to find out what is bottleneck on cluster when running loading process.Scenario: Loading CDC changes from sql server to Raw zone and merge changes into Bronze zone and then merge Bronze to Silver. All is orchestrated in data factory as ...
- 17043 Views
- 6 replies
- 0 kudos
- 0 kudos
stdout and stderr looks okay, do you have the log4j to share? You can make a doc out of it and share the doc here.
- 0 kudos
- 29399 Views
- 3 replies
- 3 kudos
Facing Issues with Databricks JDBC Connectivity after Idle time
Hello team, I am using commons(commons-dbcp2) Datasource which supports default connection pooling in Spring Java application (rest services to fetch databricks data via JDBC template).Initially all works fine and can get the data from databricks via...
- 29399 Views
- 3 replies
- 3 kudos
- 3 kudos
I am seeing the same issue with hikari. When a pooled connection is created then the databricks cluster is terminated (or restarted), the HikariDataSource retains a stale session handle.Why does connection.isValid() returns true then executing any qu...
- 3 kudos
- 4711 Views
- 2 replies
- 0 kudos
Resolved! Unable to edit Catalog Owner
I created a Catalog and ownership was assigned to meI created databricks account-group on UC, added my user to this account-group, Assigned ownership of the catalog to this account-group.I deleted the account-groupNow, the catalog ownership is showin...
- 4711 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi, In addition to the previous message, you can refer to https://docs.databricks.com/en/data-governance/unity-catalog/manage-privileges/index.html#assign-a-metastore-admin to get more information on metastore and etc.
- 0 kudos
- 2956 Views
- 2 replies
- 0 kudos
Merge version data files of Delta table
Hi,I am having one CDC enabled Delta table. In 256th version, table is having 50 data files. I want all to merge and create a single file. How can I merge all 50 data file and when I query for 256th version, I should get 1 data file? Is there any com...
- 2956 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi, ae you talking about merging CSV files? https://community.databricks.com/t5/machine-learning/merge-12-csv-files-in-databricks/td-p/3551#:~:text=Use%20Union()%20method%20to,from%20the%20specified%20set%2Fs.
- 0 kudos
