- 3006 Views
- 1 replies
- 0 kudos
Spark Scala Vs Pyspark
With the release of spark connect and used defined table functions for pyspark, I wonder, what are the remaining advantages (if any) of using scala Spark?
- 3006 Views
- 1 replies
- 0 kudos
- 0 kudos
The main remaining advantages of Scala are performance as there will always be some interoperation overhead when using PySpark. While I don't have any stats on me, I would assume the differences in performance are negligible at this point until very ...
- 0 kudos
- 6138 Views
- 2 replies
- 0 kudos
Databricks Managing Materialized Views in Delta Live Tables: Selective Refresh Behavior
Hi Community,I have 200 complex SQL Queries and I can't create a Streaming tables using these queries. So, I have created as Materialized Views in Delta Live Tables using these SQL queries and the DLT pipeline should be run continuously.My question i...
- 6138 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello team, thanks for reaching out us, it will be a pleasure to help you on this ask. That's a great catch to run through a materialized, view and regarding the question: If any record of underlying table is inserted, updated or deleted, the only re...
- 0 kudos
- 7042 Views
- 3 replies
- 3 kudos
Resolved! No userid, username, job when making modifications on tables
Hi, everyone!I'm in this situationI have some jobs that makes changes on a particular table. I use only one user to make this modifications, but then there's a process i can't identify that also makes changes on my table.The question is, there's a re...
- 7042 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi, @shan_chandra, @LuisRSanchez,Just found that there are some .jar that are executed and are writting on this table but this .jar is called through batchSo, we think this is the cause Thanks!Ingrid
- 3 kudos
- 1569 Views
- 1 replies
- 0 kudos
Databricks job cluster logs
Hello, how can I enable Databricks to generate a separate spark log for each job run?What parameters should I use in spark configuration?
- 1569 Views
- 1 replies
- 0 kudos
- 0 kudos
@Jackson1111 - If you are talking about workflow jobs, you can try running using a job cluster to generate spark logs for a each of the workflow jobs. But, If this is of Spark Jobs within the Spark UI, you wanted to separate out the logs. This is a ...
- 0 kudos
- 5023 Views
- 6 replies
- 1 kudos
Resolved! Azure Devops CI/CD - AWS Databricks
Hello, there is documentation for integrating Azure Devops CI/CD pipeline with AWS Databricks.Thanks.
- 5023 Views
- 6 replies
- 1 kudos
- 1 kudos
You'll need to install the AWS Toolkit in Azure DevOps, that way you can make a service connection inside your Azure DevOps project that authenticates using an AWS Access Key ID/Secret Access Key (the AWS equivalent to Azure Service Principals). Hope...
- 1 kudos
- 2388 Views
- 1 replies
- 0 kudos
Overwriting same table
I have a table A that is used in a spark.sql and joins with multiple other tables to get data. this data will be overwritten to the same table A.When i tried this, i get an error consistently as below: ERROR: An error occurred while calling o382.save...
- 2388 Views
- 1 replies
- 0 kudos
- 0 kudos
Found this to be a transient error. Once i restarted the cluster, the overwrite was successful.
- 0 kudos
- 1653 Views
- 1 replies
- 0 kudos
Why my DLT is not working with UC?
My IAM profile is not working when accessing the SQS for file notification based ingestion?
- 1653 Views
- 1 replies
- 0 kudos
- 0 kudos
I'm not sure if I fully understand the question, but what location are you monitoring? Is it a DBFS path or mount? If so, consider using a UC Volume.
- 0 kudos
- 1652 Views
- 0 replies
- 1 kudos
Python project || write write_micro_batch in structure streaming
Hi Team,We are in progress of developing a framework in python using datatbricks-connect in VSO,However when we are trying to run micro batches in the foreachbatch function it is giving us error message saying that "few objects are not serializable: ...
- 1652 Views
- 0 replies
- 1 kudos
- 2156 Views
- 2 replies
- 1 kudos
Bug with enabling UniForm Data Format?
In the documentation for enabling iceberg compatibility on delta tables, it states that the minReaderVersion for IcebergCompatV1 and IcebergCompatV2 is 2 (https://docs.databricks.com/en/delta/uniform.html#requirements).However, when you run the REORG...
- 2156 Views
- 2 replies
- 1 kudos
- 1 kudos
@stevenayers-bge I've just checked source code of delta and you're right - documentation states that tat minReaderVersion should be >=2, but source code is upgrading it to 3https://github.com/delta-io/delta/blob/78970abd96dfc0278e21c04cda442bb05ccde4...
- 1 kudos
- 1817 Views
- 1 replies
- 0 kudos
How to disable DBFS storage
Hello,I administer a self-service oriented Databricks workspace and I notice that more and more users are storing their data in DBFS due to lack of knowledge.They are not specifying a location when creating their schema or they are not specifying a s...
- 1817 Views
- 1 replies
- 0 kudos
- 0 kudos
Replacing "/mnt/adl2" by "dbfs:/mnt/adl2" fixed the issue.
- 0 kudos
- 1302 Views
- 1 replies
- 0 kudos
- 1302 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @diego_andres, The billing model is the same. The Databricks Pricing Calculator might help with predicting the costs.
- 0 kudos
- 929 Views
- 0 replies
- 0 kudos
Attended Data +AI summit at SFO
It was great experience attending this conference.Got great insights about new features.Got to know about new advances in data industry.Attending the conference was an enriching and transformative experience. I gained invaluable insights into the lat...
- 929 Views
- 0 replies
- 0 kudos
- 599 Views
- 0 replies
- 0 kudos
Using Variant to consolidate small tables with different schemas
With the announcement of Variant format, I'm wondering if this is a good way to consolidate small tables with varying columns and formats into a single normalized table? Any ideas?
- 599 Views
- 0 replies
- 0 kudos
- 865 Views
- 0 replies
- 0 kudos
Unity catalog
Great learning on serverless compute, Unity catalog, etc
- 865 Views
- 0 replies
- 0 kudos
- 8678 Views
- 4 replies
- 2 kudos
Resolved! Handle updates from bronze to silver table stream
Hi Databricks Community, I am trying to stream from a bronze to a silver table, however, I have the problem that there may be updates in the bronze table. Delta table streaming reads and write does not support skipChangeCommits=false, i.e. handle mo...
- 8678 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi, You can use dlt apply changes to deal with changing source.Delta Live Tables Python language reference | Databricks on AWSThank you
- 2 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 566 | |
| 349 | |
| 287 |
