- 2905 Views
- 1 replies
- 4 kudos
How to reduce storage space consumed by delta with many updates
I have 1 delta table that I continuously append events into, and a 2nd delta table that I continuously merge into (streamed from the 1st table) that has unique ID's where properties are updated from the events (An ID represents a unique thing that ge...
- 2905 Views
- 1 replies
- 4 kudos
- 4 kudos
Did you already solved this problem?
- 4 kudos
- 5339 Views
- 2 replies
- 0 kudos
DLT Pipeline Out Of Memory Errors
I have a DLT pipeline that has been running for weeks. Now, trying to rerun the pipeline with the same code and same data fails. I've even tried updating the compute on the cluster to about 3x of what was previously working and it still fails with ou...
- 5339 Views
- 2 replies
- 0 kudos
- 0 kudos
I'd focus on understanding the codebase first. It'll help you decide what logic or data asset to keep or not keep when you try to optimize it. If you share the architecture of the application, the problem it solves, and some sample code here, it'll h...
- 0 kudos
- 11796 Views
- 1 replies
- 0 kudos
How to stop SparkSession within notebook without errr
I want to run an ETL job and when the job ends I would like to stop SparkSession to free my cluster's resources, by doing this I could avoid restarting the cluster, but when calling spark.stop() the job returns with status failed even though it has f...
- 11796 Views
- 1 replies
- 0 kudos
- 0 kudos
Please refer to this Job fails, but Apache Spark tasks finish - Databricks
- 0 kudos
- 1668 Views
- 0 replies
- 0 kudos
Add data manually to DLT
Hi Team,Is there a way that we can add data manually to the tables that are generated by DLT?We have done a PoC using DLT for Sep 15 to current data. Now, that they are happy, they wanted the previous data from Synapse and put into Databricks.I can e...
- 1668 Views
- 0 replies
- 0 kudos
- 12065 Views
- 2 replies
- 1 kudos
Referring to Azure Keyvault secrets in spark config
Hi allIn spark config for a cluster, it works well to refer to a Azure Keyvault secret in the "value" part of the name/value combo on a config row/setting.For example, this works fine (I've removed the string that is our specific storage account name...
- 12065 Views
- 2 replies
- 1 kudos
- 1 kudos
Hello,Is there any update on this issue please? Databricks no longer recommend mounting external location, so the other way to access Azure storage is to use spark config as mentioned in this document - https://learn.microsoft.com/en-us/azure/databri...
- 1 kudos
- 2311 Views
- 0 replies
- 0 kudos
Why I am getting a parse syntax error while running SQL?
Can anyone please help me what is wrong here in the syntax: dbutils.fs.ls;Error in SQL statement: ParseException: [PARSE_SYNTAX_ERROR] Syntax error at or near 'dbutils'.(line 1, pos 0)
- 2311 Views
- 0 replies
- 0 kudos
- 1595 Views
- 0 replies
- 0 kudos
Does DLT use one single SparkSession?
Hi! Does DLT use one single SparkSession for all notebooks in a Delta Live Tables Pipeline?
- 1595 Views
- 0 replies
- 0 kudos
- 5363 Views
- 0 replies
- 0 kudos
Permanently delete dropped table (Unity Catalog)
The recommendation before dropping a table is to do a DELETE then VACUUM RETENTION 0 (recommended in DEV).If you DROP the table without doing a DELETE|VACUUM, your table will be soft deleted with your entire data (permanently deletedin 30 days) and y...
- 5363 Views
- 0 replies
- 0 kudos
- 2992 Views
- 0 replies
- 0 kudos
Statistics for rearranged columns
The table property dataSkippingNumIndexedCols that gets statistics for a table starts from left to right. I am wondering what will happen to the statistics for both new and old records if we add a column in between using FIRST|AFTER identifier.
- 2992 Views
- 0 replies
- 0 kudos
- 878 Views
- 0 replies
- 0 kudos
Bamboolib error
What is the status of bamboolib? I understand that it is public preview but I'm unable to find any support references. I am getting error below. I've tried installing in a notebook, on a cluster, creating a pandas dataframe and running bam, etc. ...
- 878 Views
- 0 replies
- 0 kudos
- 9222 Views
- 0 replies
- 0 kudos
Spark is not reading Kinesis Data as fast as specified
Hi Databricks community team,I have code as below"""df = spark.readStream \.format("kinesis") \.option("endpointUrl", endpoint_url) \.option("streamName", stream_name) \.option("initialPosition", "latest") \.option("consumerMode", "efo") \.option("ma...
- 9222 Views
- 0 replies
- 0 kudos
- 1760 Views
- 0 replies
- 0 kudos
SparkSession conf : strange bug when injecting property "hourl" in SQL query
Assuming you have a catalog "my_catalog" and a schema "my_schema", the following code is not working : full_table_location = "`my_catalog`.`my_schema`.`my_table_hourl`" spark.conf.set("fullTableName", full_table_location) spark.sql("""SELECT * FRO...
- 1760 Views
- 0 replies
- 0 kudos
- 6120 Views
- 4 replies
- 0 kudos
Create cluster error: Backend service unavailable
hello,i'm new to Databricks (community edition account) and encountered a problem just now.When creating a new cluster (default 10.4 LTS) it fails with the following error: Backend service unavailable.I've tried a different runtime > same issue.I've ...
- 6120 Views
- 4 replies
- 0 kudos
- 0 kudos
Hey mbvb_py,I'm sorry to hear you're facing this "Backend service unavailable" issue with Databricks. I've encountered similar problems in the past, and it can be frustrating. Don't worry; you're not alone in this!From my experience, this error can o...
- 0 kudos
- 3352 Views
- 2 replies
- 0 kudos
How does Job Cluster knows how many resources to assign to an Application ?
Hi All Enthusiasts !As per my understanding when a user submits an application in spark cluster it specifies how much memory, executors etc. it would need . But in Data bricks notebooks we never specify that anywhere. If we have submitted the noteboo...
- 3352 Views
- 2 replies
- 0 kudos
- 0 kudos
@DBEnthusiast great question! Today, with Job Clusters, you have to specify this. As @btafur note, you do this by setting CPU, memory etc. We are in early preview of Serverless Job Clusters where you no longer specify this configuration, instead Data...
- 0 kudos
- 10483 Views
- 3 replies
- 1 kudos
Databricks Job scheduling - continuous mode
While scheduling the Databricks job using continuous mode - what will happen if the job is configured to run with Job cluster.At the end of each run will the cluster be terminated and re-created again for the next run? The official documentation is n...
- 10483 Views
- 3 replies
- 1 kudos
- 1 kudos
Hello @youssefmrini So how is the DBU calculated? As the cluster is reused, the DBU should be calculated per hour on all the jobs run in an hour correct? Or will it be calculated based on each run?I would like to know the cost calculation when runnin...
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
