- 580 Views
- 1 replies
- 2 kudos
Resolved! AutoLoader Pros/Cons When Extracting Data (Cross-Post)
Cross-posting from: https://community.databricks.com/t5/data-engineering/autoloader-pros-cons-when-extracting-data/td-p/127400Hi there, I am interested in using AutoLoader, but I'd like to get a bit of clarity if it makes sense in my case. Based on e...
- 580 Views
- 1 replies
- 2 kudos
- 2 kudos
You’ve already identified data duplication as a potential con of landing the data first, but there are several benefits to this approach that might not be immediately obvious:Schema Inference and Evolution: AutoLoader can automatically infer the sche...
- 2 kudos
- 594 Views
- 3 replies
- 2 kudos
Resolved! Python module import with Dedicated access mode
I currently have a repo connected in databricks and I was able to correctly import a python module from src folder located in the same root.Since I am using a Machine Learning runtime, I am force to choose a Dedicated (formerly: Single user) access m...
- 594 Views
- 3 replies
- 2 kudos
- 2 kudos
Thanks @szymon_dybczak ! I confirm that's a permission issue and assigning "CAN MANAGE" solves it.I still find it not really intuitive, since the goal is to use a shared cluster (with ML runtime) for development purposes. I mean, it would make sense ...
- 2 kudos
- 382 Views
- 0 replies
- 0 kudos
Automating technical documentation in ETL pipelines using LLMs
Generate pipeline documentation using LLMs and rich metadata extract As enterprise data environments expand, the complexity of maintaining accurate and current documentation across ETL pipelines has intensified. While modern platforms such as Databri...
- 382 Views
- 0 replies
- 0 kudos
- 263 Views
- 1 replies
- 0 kudos
Unity Catalog tool function with custom parameters not being used
I have created a UC tool that takes in a few custom STRING parameters. I gave this tool to an ai agent using the mosaic ai agent framework with hardcoded parameter values for testing. The issue is my ai agent hallucinates and injects its own ai gener...
- 263 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Boban12335,Can we get UC function definition to understand your problem better?Best Regards,Nivethan V
- 0 kudos
- 481 Views
- 3 replies
- 3 kudos
Resolved! AutoLoader - Write To Console (Notebook Cell) Long Running Issue
Hi there,I am likely misunderstanding how to use AutoLoader properly while developing/testing. I am trying to write a simple AutoLoader notebook cell to *read* the contents of a path with json files, and *write* them to console (i.e. notebook cell) i...
- 481 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @ChristianRRL ,It looks like spark.readStream with Auto Loader creates a continuous streaming job by default, which means it keeps running while waiting for new files.To avoid this, you can control the behaviour using trigger(availableNow=True), w...
- 3 kudos
- 3003 Views
- 2 replies
- 3 kudos
Resolved! Documentation for spatial SQL public preview - Where is it?
Hi everybody,since DBR 17.1 spatial sql functions (st_point(), st_distancesphere, ... ) are in public preview.The functionality is presented in this talk Geospatial Insights With Databricks SQL: Techniques and Applications or discussed here in the fo...
- 3003 Views
- 2 replies
- 3 kudos
- 3 kudos
Is this what you were after?https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-st-geospatial-functions
- 3 kudos
- 345 Views
- 1 replies
- 1 kudos
Resolved! Run_type has some null
Just wondering — we know that the run_type column in the job run timeline usually has only three values: JOB_RUN, SUBMIT_RUN, and WORKFLOW_RUN. So why do we also see a null value there? Any reason?
- 345 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Danish1105 ,One possible explanation is that you see null values because of the following reason they stated in documentation:"Not populated for rows emitted before late August 2024."In case of my workspace, this seems valid. I have only nulls wh...
- 1 kudos
- 1464 Views
- 3 replies
- 2 kudos
PERMISSION_DENIED: Cannot access Spark Connect. when trying to run serverless databricks connect
I am not able to run a file as "run as workflow" nor "run with databricks connect" when I choose serverless run on my paid account. However I can perform this action in my free edition account . See error : pyspark.errors.exceptions.connect.SparkCon...
- 1464 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @ivan7256 ,This might be because serverless compute isn't enabled for workflows in your paid workspace.
- 2 kudos
- 832 Views
- 3 replies
- 5 kudos
Databricks Free Edition Needs Transparency About Data Access
When I first discovered the Databricks Free Edition, I thought it was a generous offering for data enthusiasts, researchers, and developers who just needed a personal sandbox. No cost. Easy setup. Promises of productivity. But what caught me off guar...
- 832 Views
- 3 replies
- 5 kudos
- 5 kudos
Thanks again for all the perspectives shared so far. I want to re-emphasize that the Databricks Free Edition offers real value. For data enthusiasts, learners, and builders, it’s a genuinely powerful environment to get hands-on without jumping throug...
- 5 kudos
- 4982 Views
- 3 replies
- 8 kudos
Resolved! Rss feeds for databricks releases
Hi,are there any rss feeds for the databricks platform, sql & runtime releases? We have a big tech stack so it is sometimes hard to keep up with the ever changing technologies. We are using rss feeds to keep up with all of that.Cant find anything for...
- 4982 Views
- 3 replies
- 8 kudos
- 8 kudos
Databricks recently published an RSS feed for all their updates. As far as I can find, it is only for AWS at the moment.https://docs.databricks.com/aws/en/feed.xml
- 8 kudos
- 83851 Views
- 7 replies
- 7 kudos
Resolved! How to create temporary table in databricks
Hi Team,I have a requirement where I need to create temporary table not temporary view.Can you tell me how to create temporary table in data bricks ?
- 83851 Views
- 7 replies
- 7 kudos
- 7 kudos
I see, thanks for sharing, can you mark the solution which worked for you @abueno as Accepted.
- 7 kudos
- 512 Views
- 1 replies
- 0 kudos
CLI: Export-dir provides LatestClone
Hi everyone,I want to download the current databricks codebase out of a workspace and tried viadatabricks databricks workspace export-dir /Sandbox/foo .Surprisingly, some of the subfolders are twice in the export target: One with the expected name (`...
- 512 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @holunder ,This could be because the backend stores both the original and cloned versions of folders, even if only one appears in the web UI. The Databricks CLI exports everything from the backend, not just what's visible in the UI.
- 0 kudos
- 3816 Views
- 5 replies
- 1 kudos
"PutWithBucketOwnerFullControl" privilege missing for storage configuration
Hi. I've been unable to create workspaces manually for a while now. The error I get is "MALFORMED_REQUEST: Failed storage configuration validation checks: List,Put,PutWithBucketOwnerFullControl,Delete". The storage configuration is on a bucket that ...
- 3816 Views
- 5 replies
- 1 kudos
- 11316 Views
- 9 replies
- 3 kudos
Stored Procedure creation in Unity Catalog
Does Databricks support the direct creation of stored procedures using the CREATE PROCEDURE syntax?
- 11316 Views
- 9 replies
- 3 kudos
- 9062 Views
- 12 replies
- 35 kudos
Are you part of a Databricks user group in your city/region?
Joining a regional user group is a great way to connect with data and AI professionals near you. These groups bring together practitioners to learn, network, and share real-world experiences — all within your local context. To join a group: Select y...
- 9062 Views
- 12 replies
- 35 kudos
- 35 kudos
@Rishabh_Tiwari appreciate this initiative! I think user groups are the best way to bring together the community as well as to learn, share and grow. I would like to start a local user group in my city since there is none already. Could you please gu...
- 35 kudos
