- 345 Views
- 4 replies
- 0 kudos
[DAB] registered_model aliases not being applied to Unity Catalog despite successful deploy
HiI'm experiencing an issue with Databricks Asset Bundles where model aliases defined in the bundle configuration are not being applied to Unity Catalog, even though the deployment succeeds and the Terraform state shows the aliases are set.Environmen...
- 345 Views
- 4 replies
- 0 kudos
- 0 kudos
Can you try by explicitly adding: databricks model-versions get-by-alias <catalog>.<schema>.<model> staging
- 0 kudos
- 349 Views
- 1 replies
- 2 kudos
Resolved! Inconsistent behaviour when using read_files to read UTF-8 BOM encoded csv
I have a simple piece of code to read a csv file from an AWS s3 bucket: SELECT * FROM read_files( myfile, format => 'csv', header => true, inferSchema => true, mode => 'FAILFAST') It's a large file with over 100 column...
- 349 Views
- 1 replies
- 2 kudos
- 2 kudos
Short version: this is (unfortunately) a Databricks quirk, not you going mad. The SQL read_files path and the PySpark spark.read.csv path do not use the exact same schema inference code, and CSVs with a UTF-8 BOM hit a corner case where read_files fa...
- 2 kudos
- 780 Views
- 6 replies
- 1 kudos
Pushing data from databricks (cloud) to Oracle (on-prem) instance?
Pushing data from databricks (cloud) to Oracle (on-prem) instance?===================================================Thanks for reviewing my threads. I find some threads on this subject dated in 2022 by @ Ajay-PandeyDatabricks to Oracle We find many...
- 780 Views
- 6 replies
- 1 kudos
- 1 kudos
Option 1: Spark JDBC write from Databricks to Oracle (recommended for “push”/ingestion) Use the built‑in Spark JDBC writer with Oracle’s JDBC driver. It’s the most direct path for writing into on‑prem Oracle and gives you control over batching, paral...
- 1 kudos
- 784 Views
- 9 replies
- 1 kudos
Resolved! Databricks external table lagging behind source files
I have a databricks external table which is pointed at an S3 bucket which contains an ever-growing number of parquet files (currently around 2000 of them). Each row in the file is timestamped to indicate when it was written. A new parquet file is add...
- 784 Views
- 9 replies
- 1 kudos
- 1 kudos
Thanks for your answers.I got a solution in the end, but it was more weirdness. A colleague fired the same query at the same database on his machine, and got the latest data! So I rebooted my PC and opened a new Databricks session, and I got the late...
- 1 kudos
- 4776 Views
- 3 replies
- 5 kudos
Resolved! SQL Warehouse - Table does not support overwrite by expression:
I'm copying data from a foreign catalog using a replace where logic in the target table, this work fine for two other tables. But for a specific one, I keep getting this error:Table does not support overwrite by expression: DeltaTableV2(org.apache.sp...
- 4776 Views
- 3 replies
- 5 kudos
- 5 kudos
Thank you @adriennn I encountered same issue and your post helped me resolved this. I agree that the error message given by databricks is not so helpful and I wasted my time in investigating table properties a lot before I found your post
- 5 kudos
- 383 Views
- 4 replies
- 2 kudos
Resolved! Naming question about SQL server database schemas
I have an MS SQL server database that has several schemas we need to ingest data from. Call them "SCHEMA1" tables and "SCHEMA2" tables. Let's call the server S and the database D. In unity catalog I have a catalog called "staging" where the staging (...
- 383 Views
- 4 replies
- 2 kudos
- 2 kudos
Thanks for the responses! @K_Anudeep suggestion makes sense in the context of our current lakehouse architecture so I think I will migrate to that.
- 2 kudos
- 437 Views
- 4 replies
- 2 kudos
Spark SQL INITCAP not capitalizing letters after periods in abbreviations
Using SELECT INITCAP("text (e.g., text, text, etc.)") abbreviations with periods like e.g. are not being fully capitalized.Current behavior:Input: "text (e.g., text, text, etc.)"Output: "Text (e.g., Text, Text, Etc.)"Expected behavior:Output: "Text ...
- 437 Views
- 4 replies
- 2 kudos
- 2 kudos
Yes similar to what @Coffee77 has told, you can alternatively create an SQL function and use it directly with the custom logic using the regexp: CREATE OR REPLACE FUNCTION PROPER_WITH_ABBREVIATIONS(input STRING)RETURNS STRINGRETURN regexp_replace(INI...
- 2 kudos
- 368 Views
- 1 replies
- 0 kudos
Resolved! Best Practices for implementing DLT, Autoloader in Workflows
I am in the process of designing a Medallion architecture where the data sources include REST API calls, JSON files, SQL Server, and Azure Event Hubs.For the Silver and Gold layers, I plan to leverage Delta Live Tables (DLT). However, I am seeking gu...
- 368 Views
- 1 replies
- 0 kudos
- 0 kudos
The optimal approach for implementing the Bronze layer in a Medallion architecture with Delta Live Tables (DLT) involves balancing batch and streaming ingestion patterns, especially when combining DLT and Autoloader. The trigger(availableNow=True) op...
- 0 kudos
- 332 Views
- 1 replies
- 2 kudos
Databricks always loads built-in BigQuery connector (0.22.2), can’t override with 0.43.x
I am using Databricks Runtime 15.4 (Spark 3.5 / Scala 2.12) on AWS.My goal is to use the latest Google BigQuery connector because I need the direct write method (BigQuery Storage Write API):option("writeMethod", "direct")This allows writing directly ...
- 332 Views
- 1 replies
- 2 kudos
- 2 kudos
There is no supported way on Databricks Runtime 15.4 to override or replace the built-in BigQuery connector to use your own version (such as 0.43.x) in order to access the direct write method. Databricks clusters come preloaded with their own managed...
- 2 kudos
- 176 Views
- 1 replies
- 0 kudos
Question on how to properly write a dataset of custom objects to MonogDB
Hi, I am implementing a Spark Job in Kotlin (unfortunately a must-have) which reads from and writes to MongoDB. The reason for this is to reuse existing code in a MapFunction. The result of applying that map is a DataSet of type Consumer, a custom ob...
- 176 Views
- 1 replies
- 0 kudos
- 0 kudos
You are correct—when you pass a BsonDocument to Spark's MongoDB connector using .write().format("mongodb"), Spark treats unknown types as generic serialized blobs, leading to documents stored as a single binary field (as you observed) rather than as ...
- 0 kudos
- 259 Views
- 2 replies
- 0 kudos
Trigger a Databricks job using sql query
Trying to run a SQL call function to trigger a Databricks job and am stuck with an error stating:"The schema <schema_name>.system cannot be found. Verify the spelling and correctness of the schema and catalog. If you did not qualify the name with a c...
- 259 Views
- 2 replies
- 0 kudos
- 0 kudos
When you say "Databricks Job", do you mean a custom SQL code you created in an stored procedure or function? Saying this because if you're really trying to run a real Databricks Job (lakeflow), it would be much better to establish another strategy di...
- 0 kudos
- 431 Views
- 3 replies
- 1 kudos
Resolved! Delta Sharing Issue between AWS and Azure
HiWe have attempted to setup a delta share between from Azure to AWS.We can see the delta share table and meta data in AWS, however when we attempt to query the table we hit a problem.If we use serverless SQL or Notebook and whitelist the IP address ...
- 431 Views
- 3 replies
- 1 kudos
- 1 kudos
Thank you @ManojkMohan.I can see the correct IP address when if IPIFY from a compute notebook. So from what you are saying above, that implies the issue is with the Azure Storage firewall right?
- 1 kudos
- 518 Views
- 9 replies
- 1 kudos
How do i Create a workspace object with SP ownership
Hi Team,I have a scenario that i have a jar file(24MB) to be put on workspace directory. But the ownership should be associated to the SP with any Individual ID ownership. Tried the Databricks CLI export option but it has limitation of 10 MB max.Plea...
- 518 Views
- 9 replies
- 1 kudos
- 1 kudos
Inspecting underlying HTTP traffic while using Databricks UI to import files in Workspace, it turns out (as expected) Databricks API is used, with requests similar to:So, @Naveenkumar1811 use Databricks API with SP identity in a similar way as expect...
- 1 kudos
- 252 Views
- 2 replies
- 0 kudos
"Something went wrong, please try again later." On Sync tables for PostgreSQL
I'm using the Sync feature to load up a Snowflake view from a catalog to postgreSQL (to expose data to API's for faster response times).I'm been playing around scripting the creation of the sync. And when I create + delete and recreate the same sync/...
- 252 Views
- 2 replies
- 0 kudos
- 0 kudos
Thank you for your response @stbjelcevic ,So tried to refreh the catalog and the schema when the table was deleted in Postgre + unity catalog (the sync one) and removed the pipeline:from databricks.sdk import WorkspaceClient from databricks.sdk.servi...
- 0 kudos
- 418 Views
- 4 replies
- 6 kudos
Resolved! VACUUM vs VACUUM LITE
Hey Team,I have a few questions regarding VACUUM and VACUUM LITE1. How do they work internally, do both of them scan the entire table storage directory?2. How should we use these in our prod jobs..I mean should we always run VACUUM LITE or VACUUM or ...
- 418 Views
- 4 replies
- 6 kudos
| User | Count |
|---|---|
| 1637 | |
| 791 | |
| 518 | |
| 349 | |
| 287 |
