- 1903 Views
- 3 replies
- 1 kudos
Best practice to handle SQL table archives?
Many of our source data are setup in a way that the main table only keep small amount of data, and historical data are move to another archive table with very similar schema.My goal is have one table in Databricks, maybe with a flag to indicate if th...
- 1903 Views
- 3 replies
- 1 kudos
- 1 kudos
I would need to dive deeper in your scenario but it sounds to me a strategy could be:1) Create a view in your SQL Server database with "current data" UNION "historical data". You can set an additional boolean field with True in first query and False ...
- 1 kudos
- 1061 Views
- 4 replies
- 0 kudos
[DAB] registered_model aliases not being applied to Unity Catalog despite successful deploy
HiI'm experiencing an issue with Databricks Asset Bundles where model aliases defined in the bundle configuration are not being applied to Unity Catalog, even though the deployment succeeds and the Terraform state shows the aliases are set.Environmen...
- 1061 Views
- 4 replies
- 0 kudos
- 0 kudos
Can you try by explicitly adding: databricks model-versions get-by-alias <catalog>.<schema>.<model> staging
- 0 kudos
- 1792 Views
- 1 replies
- 2 kudos
Resolved! Inconsistent behaviour when using read_files to read UTF-8 BOM encoded csv
I have a simple piece of code to read a csv file from an AWS s3 bucket: SELECT * FROM read_files( myfile, format => 'csv', header => true, inferSchema => true, mode => 'FAILFAST') It's a large file with over 100 column...
- 1792 Views
- 1 replies
- 2 kudos
- 2 kudos
Short version: this is (unfortunately) a Databricks quirk, not you going mad. The SQL read_files path and the PySpark spark.read.csv path do not use the exact same schema inference code, and CSVs with a UTF-8 BOM hit a corner case where read_files fa...
- 2 kudos
- 1963 Views
- 9 replies
- 1 kudos
Resolved! Databricks external table lagging behind source files
I have a databricks external table which is pointed at an S3 bucket which contains an ever-growing number of parquet files (currently around 2000 of them). Each row in the file is timestamped to indicate when it was written. A new parquet file is add...
- 1963 Views
- 9 replies
- 1 kudos
- 1 kudos
Thanks for your answers.I got a solution in the end, but it was more weirdness. A colleague fired the same query at the same database on his machine, and got the latest data! So I rebooted my PC and opened a new Databricks session, and I got the late...
- 1 kudos
- 5926 Views
- 3 replies
- 5 kudos
Resolved! SQL Warehouse - Table does not support overwrite by expression:
I'm copying data from a foreign catalog using a replace where logic in the target table, this work fine for two other tables. But for a specific one, I keep getting this error:Table does not support overwrite by expression: DeltaTableV2(org.apache.sp...
- 5926 Views
- 3 replies
- 5 kudos
- 5 kudos
Thank you @adriennn I encountered same issue and your post helped me resolved this. I agree that the error message given by databricks is not so helpful and I wasted my time in investigating table properties a lot before I found your post
- 5 kudos
- 941 Views
- 4 replies
- 2 kudos
Resolved! Naming question about SQL server database schemas
I have an MS SQL server database that has several schemas we need to ingest data from. Call them "SCHEMA1" tables and "SCHEMA2" tables. Let's call the server S and the database D. In unity catalog I have a catalog called "staging" where the staging (...
- 941 Views
- 4 replies
- 2 kudos
- 2 kudos
Thanks for the responses! @K_Anudeep suggestion makes sense in the context of our current lakehouse architecture so I think I will migrate to that.
- 2 kudos
- 1040 Views
- 4 replies
- 2 kudos
Spark SQL INITCAP not capitalizing letters after periods in abbreviations
Using SELECT INITCAP("text (e.g., text, text, etc.)") abbreviations with periods like e.g. are not being fully capitalized.Current behavior:Input: "text (e.g., text, text, etc.)"Output: "Text (e.g., Text, Text, Etc.)"Expected behavior:Output: "Text ...
- 1040 Views
- 4 replies
- 2 kudos
- 2 kudos
Yes similar to what @Coffee77 has told, you can alternatively create an SQL function and use it directly with the custom logic using the regexp: CREATE OR REPLACE FUNCTION PROPER_WITH_ABBREVIATIONS(input STRING)RETURNS STRINGRETURN regexp_replace(INI...
- 2 kudos
- 1665 Views
- 1 replies
- 0 kudos
Resolved! Best Practices for implementing DLT, Autoloader in Workflows
I am in the process of designing a Medallion architecture where the data sources include REST API calls, JSON files, SQL Server, and Azure Event Hubs.For the Silver and Gold layers, I plan to leverage Delta Live Tables (DLT). However, I am seeking gu...
- 1665 Views
- 1 replies
- 0 kudos
- 0 kudos
The optimal approach for implementing the Bronze layer in a Medallion architecture with Delta Live Tables (DLT) involves balancing batch and streaming ingestion patterns, especially when combining DLT and Autoloader. The trigger(availableNow=True) op...
- 0 kudos
- 947 Views
- 1 replies
- 2 kudos
Databricks always loads built-in BigQuery connector (0.22.2), can’t override with 0.43.x
I am using Databricks Runtime 15.4 (Spark 3.5 / Scala 2.12) on AWS.My goal is to use the latest Google BigQuery connector because I need the direct write method (BigQuery Storage Write API):option("writeMethod", "direct")This allows writing directly ...
- 947 Views
- 1 replies
- 2 kudos
- 2 kudos
There is no supported way on Databricks Runtime 15.4 to override or replace the built-in BigQuery connector to use your own version (such as 0.43.x) in order to access the direct write method. Databricks clusters come preloaded with their own managed...
- 2 kudos
- 475 Views
- 1 replies
- 0 kudos
Question on how to properly write a dataset of custom objects to MonogDB
Hi, I am implementing a Spark Job in Kotlin (unfortunately a must-have) which reads from and writes to MongoDB. The reason for this is to reuse existing code in a MapFunction. The result of applying that map is a DataSet of type Consumer, a custom ob...
- 475 Views
- 1 replies
- 0 kudos
- 0 kudos
You are correct—when you pass a BsonDocument to Spark's MongoDB connector using .write().format("mongodb"), Spark treats unknown types as generic serialized blobs, leading to documents stored as a single binary field (as you observed) rather than as ...
- 0 kudos
- 890 Views
- 2 replies
- 0 kudos
Trigger a Databricks job using sql query
Trying to run a SQL call function to trigger a Databricks job and am stuck with an error stating:"The schema <schema_name>.system cannot be found. Verify the spelling and correctness of the schema and catalog. If you did not qualify the name with a c...
- 890 Views
- 2 replies
- 0 kudos
- 0 kudos
When you say "Databricks Job", do you mean a custom SQL code you created in an stored procedure or function? Saying this because if you're really trying to run a real Databricks Job (lakeflow), it would be much better to establish another strategy di...
- 0 kudos
- 1339 Views
- 3 replies
- 1 kudos
Resolved! Delta Sharing Issue between AWS and Azure
HiWe have attempted to setup a delta share between from Azure to AWS.We can see the delta share table and meta data in AWS, however when we attempt to query the table we hit a problem.If we use serverless SQL or Notebook and whitelist the IP address ...
- 1339 Views
- 3 replies
- 1 kudos
- 1 kudos
Thank you @ManojkMohan.I can see the correct IP address when if IPIFY from a compute notebook. So from what you are saying above, that implies the issue is with the Azure Storage firewall right?
- 1 kudos
- 1641 Views
- 9 replies
- 1 kudos
How do i Create a workspace object with SP ownership
Hi Team,I have a scenario that i have a jar file(24MB) to be put on workspace directory. But the ownership should be associated to the SP with any Individual ID ownership. Tried the Databricks CLI export option but it has limitation of 10 MB max.Plea...
- 1641 Views
- 9 replies
- 1 kudos
- 1 kudos
Inspecting underlying HTTP traffic while using Databricks UI to import files in Workspace, it turns out (as expected) Databricks API is used, with requests similar to:So, @Naveenkumar1811 use Databricks API with SP identity in a similar way as expect...
- 1 kudos
- 616 Views
- 2 replies
- 0 kudos
"Something went wrong, please try again later." On Sync tables for PostgreSQL
I'm using the Sync feature to load up a Snowflake view from a catalog to postgreSQL (to expose data to API's for faster response times).I'm been playing around scripting the creation of the sync. And when I create + delete and recreate the same sync/...
- 616 Views
- 2 replies
- 0 kudos
- 0 kudos
Thank you for your response @stbjelcevic ,So tried to refreh the catalog and the schema when the table was deleted in Postgre + unity catalog (the sync one) and removed the pipeline:from databricks.sdk import WorkspaceClient from databricks.sdk.servi...
- 0 kudos
- 1148 Views
- 4 replies
- 6 kudos
Resolved! VACUUM vs VACUUM LITE
Hey Team,I have a few questions regarding VACUUM and VACUUM LITE1. How do they work internally, do both of them scan the entire table storage directory?2. How should we use these in our prod jobs..I mean should we always run VACUUM LITE or VACUUM or ...
- 1148 Views
- 4 replies
- 6 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 573 | |
| 349 | |
| 287 |
