- 627 Views
- 2 replies
- 1 kudos
Resolved! ML Specific computes in data bricks free edition
Given free edition data bricks has serverless compute only is there any work around to chose ML Specific computes like belowis paying for it the only option ?
- 627 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @ManojkMohan , as part of Databricks Free Edition you have access to serverless compute resources only.Databricks Runtime for Machine Learning and Apache Spark MLlib are not supported.Resources:Databricks Free Edition limitations | Databricks Docu...
- 1 kudos
- 580 Views
- 1 replies
- 1 kudos
Resolved! Testing Databricks Auto Loader File Notification (File Event) in Public Preview - Spark Termination
I tried to test the Databricks Auto Loader file notification (file event) feature, which is currently in public preview, using a notebook for work purposes. However, when I ran display(df), Spark terminated and threw the error shown in the attached i...
- 580 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @dawn-dot-py! Auto Loader’s managed file events are indeed in Public Preview, but they’re available to allowlisted workspaces. The error you encountered means your workspace hasn’t been enrolled in the preview, which is expected unless you’ve b...
- 1 kudos
- 1711 Views
- 1 replies
- 1 kudos
DLT Incrimental Load And Metadata Capture
Hello,I'm building a Delta Live Tables (DLT) pipeline to load data from a cloud source into an on-premise warehouse. My source tables have Change Data Feed (CDF) enabled, and my pipeline code is complex, involving joins of multiple Slowly Changing Di...
- 1711 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Ovasheli ,The thing is with Declarative Pipelines (former DLT) you can't always force incremental load. For example, if you're using materialized views in your pipeline there is an optimizer called Enzyme that can selectively incrementally load m...
- 1 kudos
- 5585 Views
- 5 replies
- 1 kudos
Resolved! Connecting to a On Prem SQL Server Instance using JDBC
Hi,We are trying to connect to an On-prem SQL Server instances using JDBC (we really want to use a Federated connection, testing JDBC first). We have successfully done this for one of the SQL Servers we have, but cannot get it to work for the other....
- 5585 Views
- 5 replies
- 1 kudos
- 1 kudos
The main difference was that the failing one was a named instance (hostname\instancename) instead of just hostname. After trying different connection string variations and confirming traffic hit the server through the firewall, we found the root caus...
- 1 kudos
- 1009 Views
- 1 replies
- 1 kudos
Resolved! Create A data Pipeline between 2 databricks Instance one using unity catalog the Other Databrick not
I have 2 Databricks Instance. Databricks A and Databricks B.The Application Hightouch is consuming Data from Databricks B through Unity Catalog.I have to create a Data pipeline to push data from Databricks A to B Without using Delta sharing.Diagram: ...
- 1009 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Pratikmsbsvm You could use Lakehouse Federation. Attach 1 instance to another instance as a connection. https://docs.databricks.com/aws/en/query-federation/databricks
- 1 kudos
- 28529 Views
- 16 replies
- 5 kudos
Resolved! Databricks SQL - Unable to Escape Dollar Sign ($) in Column Name
It seems that due to how Databricks processes SQL cells, it's impossible to escape the $ when it comes to a column name.I would expect the following to work:%sql SELECT 'hi' `$id`The backticks ought to escape everything. And indeed that's exactly wha...
- 28529 Views
- 16 replies
- 5 kudos
- 5 kudos
I needed a $ character in the context of Java format strings in the SQL printf function.With some trial and error, I found workarounds:A trailing backslash escapes the dollar character successfully, like soSELECT printf('%1$\s, %1$\s', 'hi') AS testC...
- 5 kudos
- 568 Views
- 1 replies
- 1 kudos
Data Transfer Between 2 Databricks Instance without using Delta share.
Hello,I am not allow to use Delta share.What could be best approach to send Data from Databricks A to Databricks B. AS Shown in diagram.what mechanism we can used to transfer the data. for example:- Do I need to open port or any other mechanism like ...
- 568 Views
- 1 replies
- 1 kudos
- 1 kudos
If the two workspaces are in the same region (same control plane), then there is no need to transfer data. Remember, Databricks does not store your data or put it into a proprietary format. You give Databricks the permissions to access the data tha...
- 1 kudos
- 1186 Views
- 2 replies
- 1 kudos
Resolved! Prakash Hinduja Switzerland (Swiss) How do I build an ETL pipeline in Databricks?
Hi I’m Prakash Hinduja, a visionary financial strategist, was born in Amritsar (India) and now resides in Geneva, Switzerland (Swiss). I’m looking to build an ETL pipeline in Databricks and would love some guidance. What are the key steps I should fo...
- 1186 Views
- 2 replies
- 1 kudos
- 1 kudos
@prakashhinduja1 this is a great resource if you wanna get stuck straight in to an example: https://community.databricks.com/t5/get-started-guides/getting-started-with-databricks-build-a-simple-lakehouse/ta-p/67404All the best,BS
- 1 kudos
- 692 Views
- 1 replies
- 0 kudos
Resolved! Kamal Hinduja Switzerland (Swiss) What is the best way to manage Delta Lake tables in Databricks?
Hi,I'm Kamal Hinduja. I was born in Chennai, India, and I now reside in Geneva, Switzerland(Swiss) . Can anyone explain in detail what is the best way to manage Delta Lake tables in Databricks?Thanks, RegardsKamal Hinduja Geneva, Switzerland(Swiss)
- 692 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @kamalhinduja, There's a great article here: https://docs.databricks.com/aws/en/delta/best-practices If you look down the left-hand side navigation pane on the link above, you'll find a bunch of useful articles surrounding the Delta Lake and Delta...
- 0 kudos
- 637 Views
- 2 replies
- 1 kudos
Resolved! Migrating to new SQL parameters
How do I migrate this to the new SQL paramaters? %sqlCREATE OR REPLACE TABLE ${environment_name}.${schema_name}.cmip6_max_rainfall_${run_version} ASSELECT *FROM read_files('/Volumes/${environment_name}/${schema_name}/pluvial_flood/scratch/gfes_parqu...
- 637 Views
- 2 replies
- 1 kudos
- 1 kudos
Thanks for the help on how to change. I must say it seemed better how it was before!
- 1 kudos
- 471 Views
- 1 replies
- 0 kudos
Resolved! Databricks Free Edition | RETRIES_EXCEEDED issue
Hi Team,I am not able to fire below commands i am getting error %sqlCREATE SCHEMA IF NOT EXISTS workspace.gold; The maximum number of retries has been exceeded.Tried all the available option but not working ThanksNikhil Samaiya
- 471 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @samaiyanik! Could you please try the suggestions shared in the post below and let us know if that helps resolve the issue?Similar Post: error: [RETRIES_EXCEEDED] The maximum number of retries has been exceeded
- 0 kudos
- 1446 Views
- 3 replies
- 1 kudos
Databricks recommended Approach to load data vault 2.0
Hi,Please share the recommended approach to load Data Vault 2.0 .Overview1. Current Landscape - Lakehouse (Bronze/Silver/Gold)2. Data Vault 2.0 to be created in Silver layer.3. Bronze data will be made available in delta table using ETL Questions1. ...
- 1446 Views
- 3 replies
- 1 kudos
- 1 kudos
Kindly provide your valuable input and suggestion for the above questions
- 1 kudos
- 4597 Views
- 5 replies
- 0 kudos
Spark SQL vs serverless SQL
Are there any benchmarks showing performance and cost differences between running SQL workloads on Spark SQL vs Databricks SQL (specially serverless SQL)?Our customer is hesitant about getting locked into Databricks SQL as opposed to being able to ru...
- 4597 Views
- 5 replies
- 0 kudos
- 0 kudos
Spark SQL serves as the SQL interface for Spark applications, whereas Databricks SQL is a more advanced, warehouse-optimized product built around SQL Warehouses, which utilize multiple Spark clusters. This architectural difference can lead to noticea...
- 0 kudos
- 1125 Views
- 3 replies
- 4 kudos
DLT table deletion
If we delete the DLT pipeline, the tables would get deleted.But in a DLT pipeline which creates 5 tables , if I comment out logic of 1 table, that table is not deleted from the catalog, even though full refresh of the pipeline is done.Does anyone kno...
- 1125 Views
- 3 replies
- 4 kudos
- 4 kudos
Don't confuse DLT and LDP (Lakeflow Declarative Pipelines) as though behind the scenes they work very similarly, the UI and the developer experience has changed immensely and very important new features have been added. I used DLT extensively and in ...
- 4 kudos
- 535 Views
- 1 replies
- 0 kudos
Troubleshooting AutoLoader
Hi there, I am running into a bit of an issue displaying some AutoLoader readStream data. Can I get some assistance to understand how to properly troubleshoot this? I've looked at logs before, but frankly it's not clear where to look exactly:First, "...
- 535 Views
- 1 replies
- 0 kudos
- 0 kudos
I'm also working with AutoLoader these days to create an ingestion pattern and troubleshooting it can be tricky.I wonder if you could pick a single file (whose full path / location / uri you exactly know) and read it without autoloader. Just with spa...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
