- 967 Views
- 2 replies
- 4 kudos
Resolved! Vacuum on DLT
We are currently using DLT tables in our target tables. The tables are getting loaded in continuous job pipelines.The liquid cluster is enabled in the tables. Will Vacuum work on these tables when it is getting loaded in continuous mode? How to run t...
- 967 Views
- 2 replies
- 4 kudos
- 4 kudos
VACUUM works fine on DLT tables running in continuous mode. DLT does automatic maintenance (OPTIMIZE + VACUUM) roughly every 24 hours if the pipeline has a maintenance cluster configured. Q: The liquid cluster is enabled in the tables. Will Vacuum wo...
- 4 kudos
- 931 Views
- 4 replies
- 1 kudos
Catalog tag filter error
When trying to filter in the catalog on "Tag", it throws an error that it failed to load values:The other filters do load:I have tried it with different computes and I have a view that has a tag (as shown in the screenshot).I have the following privi...
- 931 Views
- 4 replies
- 1 kudos
- 1 kudos
Hello @DylanStout! Did the suggestions shared above help resolve your concern? If so, please consider marking the response as the accepted solution.If you found a different approach that worked, sharing it would be helpful for others in the community...
- 1 kudos
- 486 Views
- 1 replies
- 1 kudos
Regression: Dashboard slicer paste now commits invalid filter values instead of searching
Hi Team,We appear to be experiencing a recent regression in the AI/BI dashboard filter slicer behaviour.Steps to reproduceOpen a dashboard containing a single-select or multi-select filter slicer.Click into the slicer’s text input.Paste text from the...
- 486 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi Tim, I can't find any mention of this internally. But I suspect it will be related to this change Multi-select filter paste: Viewers can now copy a column of values from a spreadsheet and paste them into a multi-select filter. My recommendation w...
- 1 kudos
- 760 Views
- 3 replies
- 1 kudos
Data Ingestions errors
I was going to ingestion Data from website to databricks but it is showing Public DBFS is not enableb message. is there any other way to automate data ingestion to databricks?
- 760 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @sher_1222 yes you can upload to cloud storage and then connect using unity catalog: Connect to cloud object storage using Unity Catalog - Azure Databricks | Microsoft Learnand then use What is Auto Loader? | Databricks on AWS to automatically ing...
- 1 kudos
- 1214 Views
- 3 replies
- 0 kudos
Serverless notebook DNS failure (gai error / name resolution)
I’m using a Databricks workspace on AWS (region: us-west-2). My Serverless notebook (CPU) cannot access any external URL — every outbound request fails at DNS resolution.Minimal test in a notebook:import urllib.requesturllib.request.urlopen("https://...
- 1214 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi, Here are some troubleshooting steps: 1. Network Connectivity Configuration (NCC) Confirm that the correct NCC (such as ncc_public_internet) is attached specifically to Serverless compute, not just to SQL Warehouses or other resources.After making...
- 0 kudos
- 45780 Views
- 8 replies
- 7 kudos
Python mocking dbutils in unittests
I am trying to write some unittests using pytest, but I am coming accross the problem of how to mock my dbutils method when dbutils isn't being defined in my notebook.Is there a way to do this so that I can unit test individual functions that are uti...
- 45780 Views
- 8 replies
- 7 kudos
- 7 kudos
If this helps anyone here is how we do this:We rely on databricks_test for injecting dbutils into the notebooks that we're testing (which is a 3rd party package mind you and hasn't been updated in a while but still works). And in our notebooks we put...
- 7 kudos
- 1290 Views
- 4 replies
- 2 kudos
Resolved! st_point is disabled or unsupported.
On my DLT pipeline,I Installed the Databricks-mosaic libraryPhoton is activatedI'm using a workspace premium tierSELECT id, city_name, st_point(latitude, longitude) AS city_point FROM city_data ;st_point is disabled or unsupported. Co...
- 1290 Views
- 4 replies
- 2 kudos
- 2 kudos
DLT pipelines are still on runtime 16.4 which doesn't have support for st_point yet. See details here https://learn.microsoft.com/en-us/azure/databricks/release-notes/dlt/ You should be able to use st_point in normal SQL editor as long as the cluster...
- 2 kudos
- 1775 Views
- 1 replies
- 1 kudos
Resolved! Declarative Pipelines - Dynamic Overwrite
Regarding the limitations of declarative pipelines—specifically the inability to use replaceWhere—I discovered through testing that materialized views actually support dynamic overwrites. This handles several scenarios where replaceWhere would typica...
- 1775 Views
- 1 replies
- 1 kudos
- 1 kudos
This is a really interesting find, and honestly not something most people expect from materialized views.Under the hood, MVs in Databricks declarative pipelines are still Delta tables. So when you set partitionOverwriteMode=dynamic and partition by a...
- 1 kudos
- 1358 Views
- 2 replies
- 0 kudos
power BI Vnet data gateway to Databricks using import mode
we are using Power Bi Vnet data gateway and data source connection as databricks and using import mode.databricks is behind Vnet.refreshing model working fine for 400 records but larger volume throwing errors.i tried with different way kind of increm...
- 1358 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @szymon_dybczak thanks but that is what we set when we do make power Bi desktop model ,i used this query only and made semantic model in power BI desktop and then we did Publish this into power bi service and do the refresh in web UI there it is f...
- 0 kudos
- 1958 Views
- 7 replies
- 7 kudos
Resolved! List budget policies applying filter_by
I'm trying to list budget policies using the parameter "filter_by" to filter policies that start with "aaaa" but I'm getting an error "400 Bad Request"{'error_code': 'MALFORMED_REQUEST', 'message': "Could not parse request object: Expected 'START_OB...
- 1958 Views
- 7 replies
- 7 kudos
- 7 kudos
Thanks for the reply, @szymon_dybczak and @lingareddy_Alva.I tried both approaches but none was successful.url = f'{account_url}/api/2.1/accounts/{account_id}/budget-policies'filter_by_json = json.dumps({"policy_name": "aaaa"})params = {"filter_by": ...
- 7 kudos
- 2558 Views
- 4 replies
- 0 kudos
Using Lakehouse Federation for SQL Server with Serverless Compute
Hi,My team was able to create a Foreign Catalog that connects to a SQL Server instance hosted on an Azure VM, however when trying to query the catalog, we cannot access it using serverless compute (or a serverless sql warehouse). We have tried lookin...
- 2558 Views
- 4 replies
- 0 kudos
- 0 kudos
I'm trying to get something similar to work: Lakehouse Federation for Oracle with SQL warehouse serverless. We are using Azure Databricks and our Oracle DB runs on-prem. I've been able to use classic compute to query the database, but now I'd like to...
- 0 kudos
- 2935 Views
- 3 replies
- 5 kudos
Resolved! Why Does Azure Databricks Consume So Much Memory When Running in the Background?
I had two Azure Databricks pages open in my browser without performing any computations. When I returned from lunch, I noticed that they were occupying about 80% of the memory in the task manager. What happened? This issue never occurred in the past,...
- 2935 Views
- 3 replies
- 5 kudos
- 5 kudos
@szymon_dybczak This issue still exists and is getting worse. Even a 32GB memory & ultra 7 processor laptop cannot seem to handle this issue if there are many cells in the notebook. Do you know when it'll be fixed?
- 5 kudos
- 5287 Views
- 6 replies
- 4 kudos
Resolved! Read Array of Arrays of Objects JSON file using Spark
Hi Databricks Community! This is my first post in this forum, so I hope you can forgive me if it's not according to the forum best practices After lots of searching, I decided to share the peculiar issue I'm running into in this community.I try to lo...
- 5287 Views
- 6 replies
- 4 kudos
- 4 kudos
I guess I was a bit over enthusiastic by accepting the answer.When I run the following on the single object array of arrays (as shown in the original post) I get a single row with column "value" and value null. from pyspark.sql import functions as F,...
- 4 kudos
- 1987 Views
- 1 replies
- 1 kudos
Resolved! Azure databricks streamlit app unity catalog access
Hi allI am developing a Databricks app. I will use Databricks asset bundles for deployment.How can I connect Databricks streamlit app into Databricks unity catalog?Where should I define the credentials? (Databricks host for dev, qa and prod environme...
- 1987 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi, As a starter you may want to try deploying the streamlit starter app from the app UI, this will show you the pattern to connect and pull data into your streamlit app. The following then gives some best practise guidelines on your questions: 1. U...
- 1 kudos
- 1021 Views
- 3 replies
- 3 kudos
Resolved! Comments not updating on a SDP streaming table
We have a pipeline in a job which dynamically creates a set of streaming tables based on a list of kafka topics like this: # inside a loop @DP.table(name=table_name, comment=markdown_info) def topic_flow(topic_name=topic_name): ...
- 1021 Views
- 3 replies
- 3 kudos
- 3 kudos
Ah, my code is correct. There was just a mistake further up when producing the comments that lead me down the wrong path. Comments (and metadata) are correctly updated as expected!
- 3 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 576 | |
| 349 | |
| 287 |
