Hi, I need to join three streams/streamed data sets in a DLT pipeline. I am reading from a Kinesis data stream a sequence of events per group key. The logically first of the events per group contains a marker which determines whether that group is re...
Hi, we are ingesting data using Databricks Lake flow SQL connector from two different SQL Server databases hosted on separate servers. As part of the setup:We created two separate Ingestion Gateways.We created two separate ingestion pipelines.Both pi...
Hi @Louis_Frolio , I’ve successfully ingested one SQL database using the Lakeflow SQL connector. As part of the setup, I created an ingestion pipeline along with a gateway, and it is working as expected - when I run or re-run the pipeline, it picks u...
Hello I have jobs that perform a number of activities.Sometimes I want to disable one or more of these activities.I can do that easily in an app like ADF.Reading around I cannot find an easy way to do this although, what I've read suggest that it was...
Thanks @Raman_Unifeye Do you know if there are any plans to add this feature?
I'm experiencing "Network is unreachable" errors when trying to access external APIs from serverless compute jobs, despite having a network policy configured to allow access to all destinations. I have the "Serverless egress control" in the account s...
The ongoing "Network is unreachable" errors in Databricks serverless compute suggest that network egress is being blocked, even though you have "Full access" set for Serverless Egress Control and a permissive network policy. The partial DNS resolutio...
Hello all,I am trying to enable file events on my Azure Workspace for the File Arrival Trigger trigger mode for Databricks Workflows. I'm following this documentation exactly (I think) but I'm not seeing the option to enable them. As you can see here...
I also have an external location that is a ADLS v2 however from databricks I only want to grant read access to the managed identity. I want to use file arrival trigger but want to have option of using more than 50 such triggers. Is it possible to onl...
As of now DBFS option is disabled in the Databricks community edition. What are the other ways to use file in the Databricks notebooks for learning? When I go to catalog it show default option only with the AWS S3. Is it the only option to access the...
Databricks Architecture Center — Your Blueprint for Building Modern Data & AI PlatformsThe Databricks Architecture Center is a centralized knowledge hub that provides:End-to-end reference architecturesIndustry-specific patternsArchitecture decision ...
Its very useful. I suppose you missed to provide the link. Here is the link for easy accesshttps://www.databricks.com/resources/architectures
When should I use a single-user cluster, and when should I use a multi-user/shared cluster? What’s the difference and how do I pick the right one?
Agree with previous responses BUT consider using a dedicated cluster if using it under an "automation security principal" identity for a subset of your jobs, if that makes sense. To summarize, you need to run a job, under a non-human identity and you...
I need to do some POC with spark 3.5.7 and 4.x and need some local setup with some sample Kafka source. The POC would read data from Kafka via streaming job and write to delta table and I would like to do this on spark-4.x ..Do you know of any quick ...
Hello @RevanthV , I did some digging and here are some helpful tips. Got it — here are fast, reproducible ways to stand up a local Kafka source and run a Spark Structured Streaming job that writes to a Delta table, plus the common fixes for the conne...
HiIs there a way to have a Single or multiple filter in a Dashboard that shows titles/names but on selection passes in the identifier as query parameter filters. The cost management dashboard shipped by Databricks seems to do this for the workspace s...
I wanted to share my solution to this, hope this helps. Create a SQL function that accepts an ARRAY<STRING>, extracts a numeric ID enclosed in parentheses from the end of each string, and returns an ARRAY<BIGINT>.Example UsageInput: ["Item (1)", "Ite...
Hello,I've recently been testing out Databricks experiments for a project of mine. I wanted to nest runs, and then see these runs grouped by their parent in the experiments UI. For the longest time, I couldn't figure out how to do this. I was seeing ...
@africke , If you’re happy with the results, please go ahead and accept this as the solution so others know it worked.
Hello Community,In our AWS Databricks environment, we’ve encountered some behavior we don’t understand while performing the following operation.When we clone a query file that already has content, a new file is created with the same name and “(clone)...
@kyeongmin_baek - There is no aut-save or CMD+S for Query as it get saved only once it is attached to a cluster AND 'SAVE' icon is used. However, it still stays in cache as unsaved in that 'query' window but cloning or other file operations may lose ...
I am trying to create a simple databricks custom app but I am getting Error: Could not import 'app'. error.app.yaml fileenv: - name: FLASK_APP value: '/Workspace/Users/sam@xxx.com/databricks_apps/hello-world_2025_11_13-16_19/Gaap_commentry/app'comm...
Thanks, I have modified the yaml file but still getting Error: Could not import 'app' errorenv: - name: FLASK_APP value: '/Workspace/Users/xxx@zzz.com/databricks_apps/hello-world_2025_11_13-16_19/Gaap_commentry'command: [ "flask", "--app", "...
Hello,I have been looking at the new Dashboard created by @CodyA ( great job! ) and I was wondering if the cost displaying only provides visibility to only the databricks mark-up on each job ( i.e. $ DBUs ) or to the overall cost including cloud prov...
As the dashboard uses System Tables (system.billing.usage) to show spend across Jobs, SQL, and Notebooks, I dont beleive it includes cloud provider VM costs.
Hi All,I need your help, as I am running jobs it is getting successful, when I click on job and there we can find lineage > View run events option when click on it. I see below steps.Job Started: The job is triggered.Waiting for Cluster: The job wait...
Hi Team/Member,As I am running jobs it is getting successful, when I click on job and there we can find lineage > View run events option when click on it. We find below steps and also added screenshot of it. I want screenshot stages logs, where i wil...
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 492 | |
| 349 | |
| 287 |
