- 4693 Views
- 1 replies
- 0 kudos
Dataframe getting updated after creating temporary view
I'm observing different behavior between Databricks Runtime versions when working with DataFrames and temporary views, and would appreciate any clarification.In both environments, I performed the following steps in a notebook (each connected to its o...
- 4693 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Einsatz, this is expected in DBR version 14.3 and above since we don't have a Spark context. This is happening due to the cache invalidation. To resolve the issue, please go with the dynamic name for the view every time.
- 0 kudos
- 2258 Views
- 1 replies
- 0 kudos
Analyzing Serverless SQL Warehouse Cost Projection Using System Tables
Hello everyone,I'm working on analyzing cost projections for Serverless SQL Warehouses using system tables, and I’d like to share a visualization approach I’m using to highlight some key differences between classic and serverless SQL warehouses. (Loo...
- 2258 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @mstfkmlbsbdk, great analysis. However, as you mentioned, the classic warehouse is not terminating even though there are no active queries on it. The reason for that is while creating a classic warehouse, by default it takes 45 minutes to termina...
- 0 kudos
- 13646 Views
- 6 replies
- 8 kudos
Resolved! How to sort widgets in a specific order?
I'd like to have a couple of widgets, one for the start and another for end date. I want them to appear in that order but when I run the code below, end date shows up before the start date. How can order in the way I I desired?dbutils.widgets.text("s...
- 13646 Views
- 6 replies
- 8 kudos
- 8 kudos
Doing it manually is not optimal. It should be possible to do this automatically - by creation date or extra function to sort widgets.
- 8 kudos
- 2598 Views
- 1 replies
- 0 kudos
How to Implement Incremental Loading in Azure Databricks for ETL
Hi everyone,I'm currently working on an ETL process using Azure Databricks (Standard Tier) where I load data from Azure SQL Database into Databricks. I run a notebook daily to extract, transform, and load the data for Power BI reports.Right now, the ...
- 2598 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @chexa_Wee, you can leverage DLT feature to do so. Please check: https://docs.databricks.com/aws/en/dlt/transform https://docs.databricks.com/aws/en/dlt/stateful-processing Here is the step-by-step tutorial: https://docs.databricks.com/aws/en/dlt...
- 0 kudos
- 1533 Views
- 1 replies
- 1 kudos
Resolved! Migrating DLT tables from TEST to PROD catalogs
Why can't we just copy all the DLT tables and materialized views from one UC catalog to another to get the historical data in place and then run the DLT pipelines on those UC tables?We are migrating many very large tables from our TEST catalog to our...
- 1533 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @BF7, we do not support moving the streaming table yet. If we clone the DLT streaming table, it will get converted into a normal delta table instead of a streaming table. In that condition, we need to go with the "full refresh all" option and inde...
- 1 kudos
- 7274 Views
- 4 replies
- 0 kudos
List all users groups and the actual users in them in sql
We have a bunch of cloud AD groups in Databricks and I can see what users are in each group by using the user interface Manage Account -> Users and groups -> GroupsI would like to be able to produce this full list in SQL. I have found the below code ...
- 7274 Views
- 4 replies
- 0 kudos
- 0 kudos
Ok. Well, my last suggestion is to have a look at the SCIM Users API, and SCIM Groups API. You should be able to make the API calls right in the notebook. Cheers, Lou.
- 0 kudos
- 4390 Views
- 2 replies
- 2 kudos
Resolved! DBFS with Google Cloud Storage(GCS)
Does DBFS support GCS?
- 4390 Views
- 2 replies
- 2 kudos
- 2 kudos
Yes you need just to create service account for databricks and than assign storage admin role to bucket. After that you can mount GCS standard way:bucket_name = "<bucket-name>" mount_name = "<mount-name>" dbutils.fs.mount("gs://%s" % bucket_name, "/m...
- 2 kudos
- 2326 Views
- 1 replies
- 1 kudos
Resolved! File Arrival Trigger - Reduce Listing
Hi there, the file arrival trigger seems handy, but I have questions about the performance and cost implications of using it. Per file arrival trigger documentation:"File arrival triggers do not incur additional costs other than cloud provider costs ...
- 2326 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @ChristianRRL Your Assumptions - Partially CorrectYou're correct about several key points:1. File listing overhead: Yes, the trigger does need to list files in the monitored location to detect new arrivals2. Cloud provider costs: Listing operation...
- 1 kudos
- 2840 Views
- 1 replies
- 0 kudos
Resolved! Is it possible to selectively overwrite data in a partitioned delta table with a sql warehouse
My team has a workflow that currently runs with databricks sql using a standalone cluster. We are trying to switch this job to using a sql warehouse but I keep getting errors. The current job runs in a for-each loop to break up the work into smaller ...
- 2840 Views
- 1 replies
- 0 kudos
- 0 kudos
Teammate helped me realize that `replace where` doesn't work with insert overwrite, but does work with insert into.
- 0 kudos
- 4472 Views
- 2 replies
- 2 kudos
Resolved! Notebook cell won't finish running or cancelling. Interrupt button greyed out.
A cell in a notebook that is using the %run magic command to run another notebook was running for what I considered too long and so I clicked the interrupt button and now the button is greyed out and giving a spinning circle.The interrupt button at t...
- 4472 Views
- 2 replies
- 2 kudos
- 2 kudos
Hello @help_needed_445! Are you still experiencing this issue? You can try restarting your cluster to force-stop any ongoing processes. If that doesn’t resolve the problem, detaching and reattaching the notebook to the cluster might help.
- 2 kudos
- 18547 Views
- 6 replies
- 0 kudos
Databrikcs SDK - create new job using JSON
Hello,I am trying to create a Job via Databricks SDK. As input, I use the JSON generated via Workflows UI (Worklflows->Jobs->View YAML/JSON->JSON API->Create) generating pavel_job.json. When trying to run SDK function jobs.create asdbk = WorkspaceCli...
- 18547 Views
- 6 replies
- 0 kudos
- 0 kudos
This is probably the easiest way to create job from JSON:import json from databricks.sdk import WorkspaceClient from databricks.sdk.service.jobs import CreateJob client = WorkspaceClient( host=WORKSPACE_DICT[WORKSPACE_NAME]["host_name"], token=...
- 0 kudos
- 1360 Views
- 1 replies
- 0 kudos
Not able to login to Partner Academy
I am not able to login to databricks partner academy. Also I raised a ticket (ticket id: 00670650) for the support team. Can someone please help me in resolving the issue?
- 1360 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @abhishekv5055! The support team takes around 24-48 hours to review and respond to submitted tickets. They will follow up with you directly through the ticket you’ve raised. We appreciate your patience in the meantime.
- 0 kudos
- 3162 Views
- 3 replies
- 0 kudos
Databricks Asset Bundles Firewall Issue
HelloI'm trying to use Databricks Asset Bundles within a network that has limited access to the internet. When I try to deploy I get the error message "error downloading Terraform: Get "https://releases.hashicorp.com/terraform/1.5.5/index.json". Is...
- 3162 Views
- 3 replies
- 0 kudos
- 0 kudos
HelloAfter testing the following urls need to be whitelisted on the firewall... - [Terraform Registry](https://registry.terraform.io) - [Terrafrom Checkpoint API](https://checkpoint-api.hashicorp.com) - [Terraform Releases](https://releases.hashi...
- 0 kudos
- 1020 Views
- 1 replies
- 0 kudos
S3 Private Connection from Databricks Serverless Workspace
I'm looking for reference to privately connect to an S3 bucket from a Serverless Workspace deployed on the same region.
- 1020 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @mgcasas! To enable private access from a Serverless Workspace to an S3 bucket in the same region, you can use an AWS Gateway Endpoint. Specifically, create an S3 Gateway Endpoint in your AWS VPC to allow direct and secure connectivity without ...
- 0 kudos
- 2012 Views
- 2 replies
- 0 kudos
IP address configuration for databricks workspace
Hi All Im running the below code in notebook but always returns 400 error even though everything seems correct, Can you please help me how to see the current IP address list or enable the IP access list for the databricks workspaceyour help will be a...
- 2012 Views
- 2 replies
- 0 kudos
- 0 kudos
HI @Shua42 - I tried installing the CLI and tried providing the necessary commands from the below documentation, but workspace commands are not working as workspace -conf is not working , can you help here Link - Configure IP access lists for worksp...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
