- 4380 Views
- 2 replies
- 0 kudos
Databricks Managing Materialized Views in Delta Live Tables: Selective Refresh Behavior
Hi Community,I have 200 complex SQL Queries and I can't create a Streaming tables using these queries. So, I have created as Materialized Views in Delta Live Tables using these SQL queries and the DLT pipeline should be run continuously.My question i...
- 4380 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello team, thanks for reaching out us, it will be a pleasure to help you on this ask. That's a great catch to run through a materialized, view and regarding the question: If any record of underlying table is inserted, updated or deleted, the only re...
- 0 kudos
- 6343 Views
- 3 replies
- 3 kudos
Resolved! No userid, username, job when making modifications on tables
Hi, everyone!I'm in this situationI have some jobs that makes changes on a particular table. I use only one user to make this modifications, but then there's a process i can't identify that also makes changes on my table.The question is, there's a re...
- 6343 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi, @shan_chandra, @LuisRSanchez,Just found that there are some .jar that are executed and are writting on this table but this .jar is called through batchSo, we think this is the cause Thanks!Ingrid
- 3 kudos
- 1229 Views
- 1 replies
- 0 kudos
Databricks job cluster logs
Hello, how can I enable Databricks to generate a separate spark log for each job run?What parameters should I use in spark configuration?
- 1229 Views
- 1 replies
- 0 kudos
- 0 kudos
@Jackson1111 - If you are talking about workflow jobs, you can try running using a job cluster to generate spark logs for a each of the workflow jobs. But, If this is of Spark Jobs within the Spark UI, you wanted to separate out the logs. This is a ...
- 0 kudos
- 3798 Views
- 6 replies
- 1 kudos
Resolved! Azure Devops CI/CD - AWS Databricks
Hello, there is documentation for integrating Azure Devops CI/CD pipeline with AWS Databricks.Thanks.
- 3798 Views
- 6 replies
- 1 kudos
- 1 kudos
You'll need to install the AWS Toolkit in Azure DevOps, that way you can make a service connection inside your Azure DevOps project that authenticates using an AWS Access Key ID/Secret Access Key (the AWS equivalent to Azure Service Principals). Hope...
- 1 kudos
- 1566 Views
- 1 replies
- 0 kudos
Overwriting same table
I have a table A that is used in a spark.sql and joins with multiple other tables to get data. this data will be overwritten to the same table A.When i tried this, i get an error consistently as below: ERROR: An error occurred while calling o382.save...
- 1566 Views
- 1 replies
- 0 kudos
- 0 kudos
Found this to be a transient error. Once i restarted the cluster, the overwrite was successful.
- 0 kudos
- 1375 Views
- 1 replies
- 0 kudos
Why my DLT is not working with UC?
My IAM profile is not working when accessing the SQS for file notification based ingestion?
- 1375 Views
- 1 replies
- 0 kudos
- 0 kudos
I'm not sure if I fully understand the question, but what location are you monitoring? Is it a DBFS path or mount? If so, consider using a UC Volume.
- 0 kudos
- 654 Views
- 0 replies
- 1 kudos
Python project || write write_micro_batch in structure streaming
Hi Team,We are in progress of developing a framework in python using datatbricks-connect in VSO,However when we are trying to run micro batches in the foreachbatch function it is giving us error message saying that "few objects are not serializable: ...
- 654 Views
- 0 replies
- 1 kudos
- 1551 Views
- 2 replies
- 1 kudos
Bug with enabling UniForm Data Format?
In the documentation for enabling iceberg compatibility on delta tables, it states that the minReaderVersion for IcebergCompatV1 and IcebergCompatV2 is 2 (https://docs.databricks.com/en/delta/uniform.html#requirements).However, when you run the REORG...
- 1551 Views
- 2 replies
- 1 kudos
- 1 kudos
@stevenayers-bge I've just checked source code of delta and you're right - documentation states that tat minReaderVersion should be >=2, but source code is upgrading it to 3https://github.com/delta-io/delta/blob/78970abd96dfc0278e21c04cda442bb05ccde4...
- 1 kudos
- 1384 Views
- 1 replies
- 0 kudos
How to disable DBFS storage
Hello,I administer a self-service oriented Databricks workspace and I notice that more and more users are storing their data in DBFS due to lack of knowledge.They are not specifying a location when creating their schema or they are not specifying a s...
- 1384 Views
- 1 replies
- 0 kudos
- 0 kudos
Replacing "/mnt/adl2" by "dbfs:/mnt/adl2" fixed the issue.
- 0 kudos
- 896 Views
- 1 replies
- 0 kudos
- 896 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @diego_andres, The billing model is the same. The Databricks Pricing Calculator might help with predicting the costs.
- 0 kudos
- 594 Views
- 0 replies
- 0 kudos
Attended Data +AI summit at SFO
It was great experience attending this conference.Got great insights about new features.Got to know about new advances in data industry.Attending the conference was an enriching and transformative experience. I gained invaluable insights into the lat...
- 594 Views
- 0 replies
- 0 kudos
- 425 Views
- 0 replies
- 0 kudos
Using Variant to consolidate small tables with different schemas
With the announcement of Variant format, I'm wondering if this is a good way to consolidate small tables with varying columns and formats into a single normalized table? Any ideas?
- 425 Views
- 0 replies
- 0 kudos
- 725 Views
- 0 replies
- 0 kudos
Unity catalog
Great learning on serverless compute, Unity catalog, etc
- 725 Views
- 0 replies
- 0 kudos
- 6415 Views
- 4 replies
- 2 kudos
Resolved! Handle updates from bronze to silver table stream
Hi Databricks Community, I am trying to stream from a bronze to a silver table, however, I have the problem that there may be updates in the bronze table. Delta table streaming reads and write does not support skipChangeCommits=false, i.e. handle mo...
- 6415 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi, You can use dlt apply changes to deal with changing source.Delta Live Tables Python language reference | Databricks on AWSThank you
- 2 kudos
- 843 Views
- 0 replies
- 0 kudos
java.net.SocketTimeoutException - Apptio Datalink Failure
We use a software called Apptio Datalink where it allows you to call databases, specifically, we are using it to retrieve data from our Prod Databricks Database. Their connector is returning the following message: { "time" : "1717506136286", "errorC...
- 843 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 1616 | |
| 787 | |
| 384 | |
| 349 | |
| 287 |
