- 11373 Views
- 12 replies
- 3 kudos
Delay in files update on filesystem
Hi, I noticed that there is quite a significant delay (2 - 10s) between making a change to some file in Repos via Databricks file edit window and propagation of such change to the filesystem. Our engineers and scientists use YAML config files. If the...
- 11373 Views
- 12 replies
- 3 kudos
- 3 kudos
Similar issue where opening up the git interface can take 30+ seconds to show changes.
- 3 kudos
- 4347 Views
- 3 replies
- 4 kudos
Resolved! TABLE_OR_VIEW_ALREADY_EXISTS when using saveAsTable with append mode
I ran into an unexpected error while writing a DataFrame to a managed table df.write \ .mode("append") \ .option("overwriteSchema", "true") \ .option("mergeSchema", "true") \ .saveAsTable(bsr_sink_table_fqn)Instead of appending, Spark attemp...
- 4347 Views
- 3 replies
- 4 kudos
- 4 kudos
Thanks @szymon_dybczak and @WiliamRosa for the reply. I will probably keep mergeSchema then. If the options are incompatible or might cause issues I wish there was a warning of sorts.
- 4 kudos
- 4584 Views
- 5 replies
- 3 kudos
Resolved! Incrementalization issue in Materialized views
I am trying to implement the incremental updates to the Materialized views. The source is the Could you tell how to resolve the following issue? { "planning_information": { "technique_information": [ { "maintenance_type": "MAINTENANCE...
- 4584 Views
- 5 replies
- 3 kudos
- 1545 Views
- 4 replies
- 5 kudos
Resolved! Lakeflow Connect SchemaParseException: Illegal character
Hi,i'm trying to setup Lakeflow Connect for SQL Server. The created gateway is failing with "org.apache.avro.SchemaParseException: Illegal character in: LN.FWH-ID"Unfortunately, don't have control over the source database to change the column names.I...
- 1545 Views
- 4 replies
- 5 kudos
- 5 kudos
Thanks for your response. I ended up solving this with federation. It's not ideal but given the data volumes and the minimal impact on source database, it works for us for now.
- 5 kudos
- 2324 Views
- 2 replies
- 0 kudos
Accessing Spark Runtime Metrics Using PySpark – Seeking Best Practices
Hi everyone,I’m currently working on a solution to access Spark runtime metrics for better monitoring and analysis of our workloads.From my research, I understand that this can be implemented using SparkListener, which is a JVM interface available in...
- 2324 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi saicharandeepb,How are you doing today? as per my understanding, since SparkListener is native to Scala/Java, getting detailed runtime metrics in PySpark can be tricky, but there are some workarounds. If you need deep metrics (like stage-level and...
- 0 kudos
- 5422 Views
- 2 replies
- 3 kudos
Resolved! HELP!
Hey Databricks Community, This may be a silly question but, are we able to share Databricks related job posts on here?
- 5422 Views
- 2 replies
- 3 kudos
- 3 kudos
Hello @TomFielding! If your question is about posting job openings in the Community, then the answer is No. The Community is intended for discussions and knowledge sharing around Databricks products.
- 3 kudos
- 4108 Views
- 11 replies
- 3 kudos
Resolved! Programmatically setting TAGs on VIEWs
Hi all,In order to achieve data stability in our Workspace, our IT team has given us access to the data through VIEWs on top of an ingestion schema.Now I want to provide metadata to VIEWs in form of TAGs (IT does not want to cover this used case). Th...
- 4108 Views
- 11 replies
- 3 kudos
- 3 kudos
Hi @szymon_dybczak and @WiliamRosa,I have used backticks from the very start (you'll see it if you re-check my original post).It is definitely a cluster issue as I am able to assign tags using serverless cluster. The reason I was using a dedicated on...
- 3 kudos
- 2215 Views
- 9 replies
- 6 kudos
Resolved! transformWithStateInPandas throws "Spark connect directory is not ready" error
Hello,we employ arbitrary stateful aggregations in our data processing streams on Azure Databricks, and would like to migrate from applyInPandasWithState to transformWithStateInPandas. We employ the Python API throughout our solution, and some of our...
- 2215 Views
- 9 replies
- 6 kudos
- 6 kudos
Update: This is working fine with earlier DBR versions, but the issue seems to occur specifically with DBR 17.1.I’ve flagged this behaviour with the internal team for further investigation.
- 6 kudos
- 6921 Views
- 7 replies
- 2 kudos
Databricks Key Vault Secret - Is it available in Databricks on AWS?
@Hubert-Dudek, I see your post regarding Key Value Secret handling via UI for Databricks on Azure.Is this feature available for Databricks on AWS as well?#Secrets #Scopes #DatabricksOnAWSThanksRamana
- 6921 Views
- 7 replies
- 2 kudos
- 2 kudos
Hi, do you have any update on this topic? We are looking for the same thing. We are using Databricks (UC) on AWS and have a developer group that wants to implement 90-day rotation for an AWS IAM secret using Secrets Manager. I do not see anything i...
- 2 kudos
- 2237 Views
- 3 replies
- 0 kudos
serverless environment v3 JavaPackage object is not callable
run into this issue when use serverless environment v3JavaPackage object is not callable V2 works fine, any idea
- 2237 Views
- 3 replies
- 0 kudos
- 1403 Views
- 2 replies
- 2 kudos
Resolved! Lakebridge Transpiler Fails with UnicodeDecodeError while Analyzer Works Successfully
Hello Team,I am facing an issue with Lakebridge transpiler.The Analyzer step runs successfully and produces the expected analysis files. However, when I run the Transpiler, it fails with the following error: ERROR [src/databricks/labs/Lakebridge.tr...
- 1403 Views
- 2 replies
- 2 kudos
- 2 kudos
Root CauseThe trailing “unexpected end of JSON input” suggests the decoder aborted midway, producing invalid JSON.This mismatch between file content (likely UTF-8 or containing special characters) and default Windows decoding causes the issue.Suggest...
- 2 kudos
- 1297 Views
- 2 replies
- 1 kudos
Achieving batch-level overwrite for streaming SCD1 in DLT
Hi all,I am working with Databricks Delta Live Tables (DLT) and have the following scenario:Setup:Source data is delivered as weekly snapshots (not CDC).I have a bronze layer (streaming table) and a silver layer (also streaming).I am implementing SCD...
- 1297 Views
- 2 replies
- 1 kudos
- 1 kudos
One can achieve this with dlt.apply_changes — but you need to configure it carefully to emulate key-based batch overwrite.Step 1 — Define Bronze as Streaming Sourceimport dltfrom pyspark.sql.functions import col@Dlt.table(comment="Bronze snapshot dat...
- 1 kudos
- 2314 Views
- 5 replies
- 3 kudos
Resolved! Streaming Solution
Maintain Zonemaps with Streaming Writes Challenge: Streaming breaks zonemaps due to constant micro-batches.Solution: Incremental Updatesdef write_streaming_with_zonemap(stream_df, table_path):def update_zonemap(batch_df, batch_id):# Write databatch_d...
- 2314 Views
- 5 replies
- 3 kudos
- 3 kudos
@ck7007 brainstormed some solution approaches ., do you have some test data to test these hands on Approach Throughput Query Speed Complexity NotesPartition-level zonemapsHighMediumLowScales with micro-batches; prune at pa...
- 3 kudos
- 2090 Views
- 3 replies
- 3 kudos
Resolved! Table Fields Have a Different Value and Data Type in SQL Editor vs a SQL Notebook Cell
When I query a numeric field in the SQL Editor it returns a value of 0.02875 and the data type is decimal but when I run the same query in a SQL notebook cell it returns 0.0287500 and decimal(7,7). I'm assuming this is expected behavior but is there ...
- 2090 Views
- 3 replies
- 3 kudos
- 3 kudos
Hello @help_needed_445 Good day!its very indeed interesting case study!I found below from LLM models. Yes, this difference in decimal display between the Databricks SQL Editor (which uses the Photon engine in Databricks SQL) and notebooks (which use ...
- 3 kudos
- 1106 Views
- 1 replies
- 1 kudos
Material View to External Delta Table using sink api
Hi Team,While executing the below code i am able to create the sink and my data is getting written into delta tables from materialized view. import dlt@Dlt.table(name = "employee_bronze3")def create_table():df = spark.read.table("dev.default.employee...
- 1106 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi nkrom456,How are you doing today? as per my understanding, when you use dlt.read_stream() inside the same DLT pipeline, Databricks allows it to stream from that materialized view because everything is being managed within one pipeline — it underst...
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 579 | |
| 349 | |
| 287 |
