- 3304 Views
- 1 replies
- 0 kudos
Are init scripts breaking clusters?
My Jobs were running just fine, but for some reason all of a sudden they all started failing. When I looked into it I saw it was an error due to an init script error (we do use an init script). run failed with error message Cluster 1117-045226-l...
- 3304 Views
- 1 replies
- 0 kudos
- 0 kudos
Thank you for reaching out to Databricks Support. Could you please specify the location of the initialization script you are referring to? Additionally, it would be helpful to know whether this is a global init script or one specific to a cluster. ...
- 0 kudos
- 8421 Views
- 1 replies
- 0 kudos
Databricks PAT Logs
As an admin, how can i check which external applications are being connected with databricks by people through Personal Access Token. I have used the token API to get the token list but i couldn't find any other REST API reference for obtaining the i...
- 8421 Views
- 1 replies
- 0 kudos
- 2252 Views
- 1 replies
- 0 kudos
Disable CDF on DLT tables
Hi, I noticed Change Data Feed (CDF) is enabled by default for the bronze and gold tables running in DLT. How to check the size of the delta log? Can it be turned off?
- 2252 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, I dont believe CDF is enabled by default, please see:"Change data feed is not enabled by default..." in this doc: https://docs.databricks.com/en/delta/delta-change-data-feed.html. If it was mistakenly enabled at a table-level it could be disabled...
- 0 kudos
- 2517 Views
- 0 replies
- 0 kudos
Is Autoloader suitable to load full dumps?
Hi,I recently completed the fundamentals & advanced data engineer exam, yet I've got a question about Autoloader. Please don't go too hard on me, since I lack practical experience at this point in time Docs say this is incremental ingestion, so it's ...
- 2517 Views
- 0 replies
- 0 kudos
- 2605 Views
- 0 replies
- 0 kudos
I want to practice apache spark.
I want to practice apache spark.
- 2605 Views
- 0 replies
- 0 kudos
- 6841 Views
- 4 replies
- 1 kudos
Spark streaming: Checkpoint not recognising new data
Hello everyone!We are currently facing an issue with a stream that is not updating new data since the 20 of July.We've validated and bronze table has data that silver doesn't have.Also seeing the logs the silver stream is running but writing 0 files....
- 6841 Views
- 4 replies
- 1 kudos
- 1 kudos
Also the trigger is configured to run once, but when we start the job it never ends, it keeps in an endless loop.
- 1 kudos
- 17795 Views
- 1 replies
- 0 kudos
Resolved! Error: cannot create mws storage configurations: default auth: cannot configure default credentials.
I’ve run into an error that I can't figure out how to debug. We're trying to use terraform through a service account. I don’t know if it’s a permissions issue on Databricks, in our account, or in AWS, but it seems that something is being blocked some...
- 17795 Views
- 1 replies
- 0 kudos
- 0 kudos
Ok. I found the issue here. We had a *second* place where we were setting up the databricks provider, which I had not updated with the proper client credentials.
- 0 kudos
- 19484 Views
- 4 replies
- 2 kudos
Resolved! If exists in databrickds sql
what is the equivalent of "IF EXISTS" in databricks? I would like to first check something later after that use the insert into statement.
- 19484 Views
- 4 replies
- 2 kudos
- 2 kudos
Is there a way to check if a table exists, without trying to drop it? something like :select table_name from system_catalogs where database_name = 'mydb' and schema_name = 'myschema' and object_name = 'mytab';
- 2 kudos
- 2876 Views
- 0 replies
- 0 kudos
synapse pyspark delta lake merge scd type2 without primary key
ProblemI have a set of rows coming from previous process which has no primary key, and the composite keys are bound to change which are not a good case for composite key, only way the rows are unique is the whole row( including all keys and all value...
- 2876 Views
- 0 replies
- 0 kudos
- 22352 Views
- 0 replies
- 0 kudos
curl: (26) Failed to open/read local data from file/application in DBFS
Hi all,I am trying to upload a parquet file from S3 to dbfs with airflow bash operator curl command using Databricks python Rest API's as shown below databricks_load_task = BashOperator( task_id="upload_to_databricks", bash_command ...
- 22352 Views
- 0 replies
- 0 kudos
- 2596 Views
- 1 replies
- 0 kudos
Run Query from another notebook in streaming job
Hi!We want to run query located in another notebook every streaming microbatch.We were trying dbutils.run.notebook but we always get errorContext not valid. If you are calling this outside the main thread, you must set the Notebook context via dbutil...
- 2596 Views
- 1 replies
- 0 kudos
- 0 kudos
Query Parameters means that you have to pass all parameters as a part of URL after question mark not in the body"/api/1.2/commands/status?clusterId=$cid&contextId=$ec_id&commandId=$command_id"
- 0 kudos
- 1723 Views
- 0 replies
- 0 kudos
Not able to generate Excel file in blob through databricks
Hi team,.I am using cluster 9.1 8n databricks not able to generate Excel file in blob and below are conf Cluster:9.1.8Spark version -3.1.1Scala version 3.1.1Library:Com.crealyticsSpark.excel_2.12Version-3.1.1_0.18.2Dependency:Org.apachr.poi-poi-5.2...
- 1723 Views
- 0 replies
- 0 kudos
- 10249 Views
- 2 replies
- 1 kudos
Tableau Delta Sharing
Does anyone have any recent examples of using Tableau and Delta Sharing? The video below mentions using web connector but this connector has been depreciated in Tableau 2023.1. https://www.youtube.com/watch?v=Yg-5LXH9K1I&t=913shttps://help.tableau.co...
- 10249 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi,I am still trying to figure out how to use delta sharing with tableau. I've looking for information for a month without any success. As mentionned before, web data connector is deprecatedAny help would be appreciated.thanks, Johnattan
- 1 kudos
- 9465 Views
- 1 replies
- 1 kudos
notebooks naming convention
I have read info about objects names but are there any best practices regarding notebooks naming convention?
- 9465 Views
- 1 replies
- 1 kudos
- 1 kudos
It is recommended to name notebooks descriptively so that it is easy to understand their purpose and content. A good practice is to follow a consistent naming convention to help keep notebooks organized. These are some of general practices Use naming...
- 1 kudos
- 2381 Views
- 1 replies
- 0 kudos
Orchestration
Is it possible to use redwood orchestration over Databricks ?
- 2381 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @AG2 , We don't currently support Redwood Orchestration over Databricks. Best, Miguel
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
