- 6538 Views
- 2 replies
- 0 kudos
Resolved! Pathway advice on how to Data Engineer Associate
Hi everyone,I am new to this community and I am a BI/Data Engineer by trade in Microsoft Azure/On prem context. I want some advice on how to be a certified Data Engineer Associate in Databiricks. The training, lesson or courses to be eligible for tak...
- 6538 Views
- 2 replies
- 0 kudos
- 0 kudos
Great post, very helpful insights on the Data Engineer Associate pathway I’m also preparing for this exam and found similar resources really useful for understanding the core concepts and practice approach. Thanks for sharing! If anyone else has add...
- 0 kudos
- 812 Views
- 3 replies
- 0 kudos
Resolved! How to update alias for catalogs
Greetings,Is there a way to create an alias for a Databricks catalog? Current catalog name: trainingDesired alias: development_training The goal is that users connecting to either name should see the same schemas, tables, and data
- 812 Views
- 3 replies
- 0 kudos
- 0 kudos
I have a use case where I need to rename a catalog without impacting existing pipelines and notebooks, as the current catalog name is referenced across multiple applications. Instead of coordinating with multiple teams to update it everywhere, I was ...
- 0 kudos
- 2428 Views
- 7 replies
- 11 kudos
Resolved! Designing Reliable Data Versioning Strategies in Databricks
Hi everyone,I’m working on a use case where I need to retain 30 days of historical data in a Delta table and use it to build trend reports.I’m looking for the best approach to reliably maintain this historical data while also making it suitable for r...
- 2428 Views
- 7 replies
- 11 kudos
- 11 kudos
Hey @Raj_DB , The TLDR is time travel is great for short-term ops and debugging, but brittle as your primary reporting history, and its cost profile is harder to control and reason about than a purpose-built history table. Docs 1,2 explicitly say De...
- 11 kudos
- 1517 Views
- 4 replies
- 5 kudos
Valid init script for installing ODBC Driver 18 for SQL Server to a job cluster
I need to execute stored procedures in my notebook, to do that I created an init script that installs ODBC driver to my job cluster. But the script stops working after some time and I can't figure out why, so the cluster can't start. Can someone send...
- 1517 Views
- 4 replies
- 5 kudos
- 5 kudos
You don't need an ODBC driver inside a Databricks job cluster to run stored procedures, and init scripts are fragile enough that they easily break cluster startup. Options: Option 1 (Recommended): Use built-in JDBC / connectors instead of ODBC For S...
- 5 kudos
- 1951 Views
- 3 replies
- 2 kudos
Structured Streaming Real-Time Mode Doesn’t Support Delta — What’s the Plan?
Real-time mode doesn’t currently support Delta tables in Structured Streaming. Is there a planned timeline for launching this support?Also, are there any plans for declarative pipelines to support real-time mode or provide an equivalent capability?
- 1951 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi again !Tere usually is not a single detailed public roadmap with dates for Databricks features. I think the best official places to track what is coming are release notes, preview or beta announcements the blog and the product feedback and ideas c...
- 2 kudos
- 588 Views
- 2 replies
- 2 kudos
Why do you want to migrate from azure synapse analytics or Azure data factory to databricks
It's a simple answer bro. According to our analysis Azure pipelines and not books match process approximately 40% faster than the snaps analytics. If we really want to optimise your pipelines and perform cost optimisations in your team please migrate...
- 588 Views
- 2 replies
- 2 kudos
- 2 kudos
Hello !The only thing I can add here is migrating from ASA or ADF to dbks can make sense when your workloads need more scalable spark based processing, batch and streaming support, advanced transformation logic, lakehouse architecture or ML and AI ca...
- 2 kudos
- 3536 Views
- 5 replies
- 1 kudos
Transition from partitioned table to Liquid clustered table
Hi all,I have a table called classes, which is already partitioned on three different columns. I want to create a Liquid Clustered Table, but as far as I understand from the documentation—and from Dany Lee and his team—it was not possible as of 2024 ...
- 3536 Views
- 5 replies
- 1 kudos
- 1 kudos
Is there a plan to implement a way to migrate to liquid clustering for an existing table that has traditional partitioning and that is quite large (over 4 TB)? Re-creating such tables from scratch is not always ideal.
- 1 kudos
- 836 Views
- 1 replies
- 0 kudos
Resolved! Databricks not able to create cluster with Amazon free trial version
Error : Cannot launch the cluster because the user specified an invalid argument.Instance ID: failed-2d901c0f-d88d-499a-aInternal error message: The VM launch request to AWS failed, please check your configuration. [details] InvalidParameterCombinati...
- 836 Views
- 1 replies
- 0 kudos
- 0 kudos
The error is coming from AWS, not Databricks: your AWS account is restricted to Free Tier–eligible instance types, but the node type you picked in Databricks maps to an EC2 instance that is not Free Tier–eligible, so AWS rejects the launch request wi...
- 0 kudos
- 275 Views
- 1 replies
- 0 kudos
FileNotFoundError: [Errno 2] No such file or directory: '../00_configuration/prd/main_configuration.
Maybe someone has encountered this problem before?I’m running parallel loading for 10 objects using pool.map. Nine of them complete successfully, but one fails when trying to read a configuration file. The problem occurs occasionally and doesn’t foll...
- 275 Views
- 1 replies
- 0 kudos
- 0 kudos
@AdrianLobacz You can read the configuration once and pass the object into your function instead of reading the same file multiple times. It eliminates the IO overhead and avoids hitting the FUSE layer. When the code triggers parallel processes, they...
- 0 kudos
- 741 Views
- 1 replies
- 0 kudos
Resolved! databricks autoloader source files
Hello, How can handle this error when we use autoloader with spark.readStream (com.databricks.sql.cloudfiles.errors.CloudFilesException) [CF_EMPTY_DIR_FOR_SCHEMA_INFERENCE] Cannot infer schema when the input path `/Volumes/default/landing/source/bund...
- 741 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @seefoods, The error message seems to indicate there are no files in the source path? You can either define the schema yourself and pass it to schema(...) so Auto Loader doesn’t need to infer anything.. and as soon as files arrive, the stream will...
- 0 kudos
- 2210 Views
- 8 replies
- 0 kudos
- 2210 Views
- 8 replies
- 0 kudos
- 0 kudos
Hi @Ashwin_DSA Thanks for the response ‌‌I was thinking if the results are precomputed using cube and persisted using mv that would be retrieve results faster than metric view. Could u please let me know if my understanding is correct
- 0 kudos
- 1112 Views
- 1 replies
- 1 kudos
Resolved! Auto Loader with ignoreMissingFiles and useManagedFileEvents fails on Classic Compute
Hi everyone,I am seeing an unexpected behavior with Auto Loader when using Managed File Events on Classic Compute.The error message itself seems inconsistent with the behavior I am seeing:[FAILED_READ_FILE.DBR_FILE_NOT_EXIST] Error while reading file...
- 1112 Views
- 1 replies
- 1 kudos
- 1 kudos
Just sharing a solution in case anyone runs into the same issue.The error was caused by the cluster configuration including spark.master: "local[*]". After removing this setting, the error stopped occurring and the Auto Loader finished correctly.This...
- 1 kudos
- 685 Views
- 1 replies
- 0 kudos
Resolved! Network Configuration
I have a Databricks workspace on AWS (serverless compute). I created a network policy with "Allow access to all destinations" enabled and attached it to my workspace. When I run a Python notebook and try to make an HTTP request or curl to any externa...
- 685 Views
- 1 replies
- 0 kudos
- 0 kudos
Most likely the egress policy change hasn’t actually taken effect on the serverless compute that’s running your notebook. Check these things in order: Verify the network policy itself (Account Console → Security → Networking → Context-based ingress ...
- 0 kudos
- 725 Views
- 2 replies
- 1 kudos
Resolved! LDP Materialized View Incremental Refreshes - Changeset Size Thresholds
Is there any documentation available around the changeset size thresholds for materialized view incremental refreshes? Are these configurable at all? Are they constant or do the thresholds change depending on the number of rows/size of the material...
- 725 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi, On top of Pradeep's reply, which I'd recommend trying, I'd also suggest you raise a support ticket for this. They will potentially be able to tweak the settings in the backend (not guaranteed), but it may help. Thanks,Emma
- 1 kudos
- 965 Views
- 1 replies
- 0 kudos
Disable Public Network Access on Databricks Managed Storage Account - Deny Assignment
Issue Description:I am attempting to disable public network access on the Azure Databricks managed storage account. However, I am encountering the following error:Failed to save resource settings — access is denied due to a deny assignment created by...
- 965 Views
- 1 replies
- 0 kudos
- 0 kudos
@MyProfile This would be helpful, check once - https://learn.microsoft.com/en-us/answers/questions/1707749/managed-storage-accounts-compliance
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 580 | |
| 349 | |
| 287 |
