Hi,I recently completed the fundamentals & advanced data engineer exam, yet I've got a question about Autoloader. Please don't go too hard on me, since I lack practical experience at this point in time Docs say this is incremental ingestion, so it's ...
I want to practice apache spark.
Hello everyone!We are currently facing an issue with a stream that is not updating new data since the 20 of July.We've validated and bronze table has data that silver doesn't have.Also seeing the logs the silver stream is running but writing 0 files....
Also the trigger is configured to run once, but when we start the job it never ends, it keeps in an endless loop.
I’ve run into an error that I can't figure out how to debug. We're trying to use terraform through a service account. I don’t know if it’s a permissions issue on Databricks, in our account, or in AWS, but it seems that something is being blocked some...
Ok. I found the issue here. We had a *second* place where we were setting up the databricks provider, which I had not updated with the proper client credentials.
what is the equivalent of "IF EXISTS" in databricks? I would like to first check something later after that use the insert into statement.
Is there a way to check if a table exists, without trying to drop it? something like :select table_name from system_catalogs where database_name = 'mydb' and schema_name = 'myschema' and object_name = 'mytab';
ProblemI have a set of rows coming from previous process which has no primary key, and the composite keys are bound to change which are not a good case for composite key, only way the rows are unique is the whole row( including all keys and all value...
Hi all,I am trying to upload a parquet file from S3 to dbfs with airflow bash operator curl command using Databricks python Rest API's as shown below databricks_load_task = BashOperator( task_id="upload_to_databricks", bash_command ...
Hi!We want to run query located in another notebook every streaming microbatch.We were trying dbutils.run.notebook but we always get errorContext not valid. If you are calling this outside the main thread, you must set the Notebook context via dbutil...
Query Parameters means that you have to pass all parameters as a part of URL after question mark not in the body"/api/1.2/commands/status?clusterId=$cid&contextId=$ec_id&commandId=$command_id"
Hi team,.I am using cluster 9.1 8n databricks not able to generate Excel file in blob and below are conf Cluster:9.1.8Spark version -3.1.1Scala version 3.1.1Library:Com.crealyticsSpark.excel_2.12Version-3.1.1_0.18.2Dependency:Org.apachr.poi-poi-5.2...
Does anyone have any recent examples of using Tableau and Delta Sharing? The video below mentions using web connector but this connector has been depreciated in Tableau 2023.1. https://www.youtube.com/watch?v=Yg-5LXH9K1I&t=913shttps://help.tableau.co...
Hi,I am still trying to figure out how to use delta sharing with tableau. I've looking for information for a month without any success. As mentionned before, web data connector is deprecatedAny help would be appreciated.thanks, Johnattan
I have read info about objects names but are there any best practices regarding notebooks naming convention?
It is recommended to name notebooks descriptively so that it is easy to understand their purpose and content. A good practice is to follow a consistent naming convention to help keep notebooks organized. These are some of general practices Use naming...
Is it possible to use redwood orchestration over Databricks ?
Hi @AG2 , We don't currently support Redwood Orchestration over Databricks. Best, Miguel
With the append_flow method in Delta Live Tables, you can effortlessly combine data from multiple streams into a single Delta table.
Thank you for sharing this information @Hubert-Dudek
Databricks Runtime 14.2 now has row-level concurrency generally available and enabled by default for Delta tables with deletion vectors. This feature dramatically reduces conflicts between concurrent write operations.
Thank you for sharing this @Hubert-Dudek !!!
We currently have several workflows that are basically copies with the only difference being that they run with different service principals and so have different permissions and configuration based on who is running. The way this is managed today is...
| User | Count |
|---|---|
| 1613 | |
| 770 | |
| 349 | |
| 286 | |
| 253 |
