- 688 Views
- 2 replies
- 3 kudos
Resolved! Databricks shared folder area permissions issue
We have some notebook code that I would like to share with our team only in the "shared folder" area of Databricks. I know by default this area is meant as a area to share stuff with the entire organization but from what I have read you should be abl...
- 688 Views
- 2 replies
- 3 kudos
- 3 kudos
Hello @eballinger In Databricks, the users group (sometimes shown in the UI as All workspace users) has default permissions that cannot be revoked at the top-level Shared folder. DocsSo looks like:It’s not possible to create a folder under /Shared th...
- 3 kudos
- 463 Views
- 1 replies
- 0 kudos
Databricks Lakeflow Connector for PostgreSQL on GCP Cloud
Lakeflow connection for PosgresHi all,I hope this message finds you well.I am currently trying to create a Lakeflow connection in Databricks for a PostgreSQL database hosted on Google Cloud Platform (GCP). However, when testing the connection, I am e...
- 463 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @David_M To better support you we’d need to clarify a few points: PostgreSQL locationIs this PostgreSQL deployed inside a private VPC in GCP or is it exposed through a public IP accessible from the internet? This is key to understand what type ...
- 0 kudos
- 374 Views
- 1 replies
- 1 kudos
Solution Design Recommendation on Databricks
Hi Team,We need to design a pipeline in Databricks to:1. Call a metadata API (returns XML per keyword), parse, and consolidate into a combined JSON.2. Use this metadata to generate dynamic links for a second API, download ZIPs, unzip, and extract spe...
- 374 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @tyhatwar785 1. Should metadata and file download be separate jobs/notebooks or combined?Keep them in separate notebooks but orchestrate them under a single Databricks Job.for better error handling, and retries .2. Cluster recommendationsstart wit...
- 1 kudos
- 410 Views
- 2 replies
- 0 kudos
How to connect to a local instance of SQL Server
How can I connect my Databricks AWS account to a local instance of SQL Server?
- 410 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @MGAutomation @szymon_dybczak You may also need to open the firewall of your on-premises SQL Server to the CIDR range of your Databricks VPC. This ensures that the EC2 instances used by Databricks have valid IPs that can reach your database.If...
- 0 kudos
- 1282 Views
- 3 replies
- 0 kudos
Access Databricks Volume through CLI
Hi,I am able to connect to DBFS and transfer files there or download from there. But when I change the path to Volumes, it doesn't work. Even though I created the volume I still get this error message:Error: no such directory: /Volumes/bgem_dev/text_...
- 1282 Views
- 3 replies
- 0 kudos
- 551 Views
- 1 replies
- 2 kudos
Resolved! What is "External tables backed by Delta Lake"?
Goal: event-driven without implementing job triggereed on file arrivalI see hope to incrementally update materialized views which have external tables as their sources.This is quite a game changer if it works for various data formats.(since MV starte...
- 551 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @tana_sakakimiya ,Yes, only external tables that are in delta format are supported. Databricks supports other table formats, but to be able to use this particular feature, your table needs to be in Delta format. But if you have parquet files it's ...
- 2 kudos
- 41460 Views
- 8 replies
- 2 kudos
SSL_connect: certificate verify failed with Power BI
Hi, I'm getting this error with Power BI:Any ideas?Thanks in advance,Andres
- 41460 Views
- 8 replies
- 2 kudos
- 2 kudos
In the proxy setup, PowerBI is not aware of process to fetch intermediate certificate like a browser. hence it fails. Recent PowerBI comes with additional option such as"Automatic Proxy Discovery (Optional): Enabled"Implementation (optional) : 2.0(be...
- 2 kudos
- 27107 Views
- 16 replies
- 9 kudos
Resolved! Delta Live Tables: dynamic schema
Does anyone know if there's a way to specify an alternate Unity schema in a DLT workflow using the @Dlt.table syntax? In my case, I’m looping through folders in Azure datalake storage to ingest data. I’d like those folders to get created in different...
- 27107 Views
- 16 replies
- 9 kudos
- 9 kudos
if you add these settings in the pipeline JSON, the issue should get fixed:"pipelines.setMigrationHints" = "true""pipelines.enableDPMForExistingPipeline" = "true"I tried it on my side, and now it no longer throws the materialization error.
- 9 kudos
- 18046 Views
- 21 replies
- 16 kudos
Resolved! Was DBFS disabled from community edition?
HelloI'm trying to use the Upload data to DBFS... from Databricks community edition but it's disabled. I'm also trying to activate it using the path Settings>Advanced>Other but the option is not longer in the list. Is this a temporary or permanent mo...
- 18046 Views
- 21 replies
- 16 kudos
- 16 kudos
Noticed there a several questions/'workarounds' on this topic. I may have overlooked but again, has this feature been removed or is it currently integrated naturally? Appreciate the assistance, in advance.
- 16 kudos
- 524 Views
- 2 replies
- 1 kudos
Resolved! dashboard cost databricks - AWS instance
Hello guys, I hope your day going well ! I have some question about your dashboard cost publish on github databricks quick lab: this dashboard describe both the cost of databricks and AWS instance EC2 or just databricks cost ? Thanx
- 524 Views
- 2 replies
- 1 kudos
- 1 kudos
@seefoods, also worth noting that you might not be able to see the system tables without a metastore admin granting the appropriate access. That was something I learned in Azure : https://docs.databricks.com/aws/en/admin/ All the best,BS
- 1 kudos
- 956 Views
- 1 replies
- 2 kudos
Move/Merge Catalogs
HiWe have a couple of unity catalogs. In each are schema's, some used, some old, same named temp_data etc. etc Within those schema are tables with the same erratic approach. The result of a couple of years of bad housekeeping. (who not experienced th...
- 956 Views
- 1 replies
- 2 kudos
- 2 kudos
For moving tables and entire schemas consider looking at the "SYNC" command. The SYNC Command - Your Primary Migration Tool Databricks provides a SYNC command specifically designed for this exact use case - migrating tables from one location to ano...
- 2 kudos
- 615 Views
- 3 replies
- 2 kudos
Permission issue for pandas to read local files
I can use pandas to read local files in a notebook, such as those located in tmp.However, when I run two consecutive notebooks within the same job and read files with pandas in both, I encounter a permission error in the second notebook stating that ...
- 615 Views
- 3 replies
- 2 kudos
- 2 kudos
@liu is there anything preventing you from moving this file into a Volume on the Unity Catalog & then reading it from there?Based on your original message, the "permission" message is making me feel like it's having an issue with opening the file; pe...
- 2 kudos
- 563 Views
- 1 replies
- 0 kudos
unable to create table on external location
Currently trying to connect a table on external location and it fails with error -[UNAUTHORIZED_ACCESS] Unauthorized access: PERMISSION_DENIED: request not authorized SQLSTATE: 42501which seems like a pretty straight forward error but unable to find ...
- 563 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @PratikRudra Thank you for sharing the error: I think probably there is a component that is missing. Writing table metadata (for example, to the _delta_log directory) requires the CREATE EXTERNAL TABLE capability on the external location; this ...
- 0 kudos
- 1097 Views
- 7 replies
- 7 kudos
Resolved! MATERIALZIED VIEW TRIGGER ON UPDATE with external table as upstream table
goal: implement event driven architecture without trigger on file arrivalI would like to know whether materialzied view can update itself when its source table which is external table updated.given that the source external table referencing data in d...
- 1097 Views
- 7 replies
- 7 kudos
- 7 kudos
it seems that my idea is a bad idea because it seems that materialzied view doesn't support incremental udpate for external locationIncremental refresh for materialized views - Azure Databricks | Microsoft Learn
- 7 kudos
- 456 Views
- 1 replies
- 0 kudos
Changing paths to tables
HiMy organization has many notebooks that reference tables in schemas with the three part pathcatalog.schema.tablenameWith a lack of foresight we hardcoded all of these paths in the code and now the inevitable is happening and there is a need to rest...
- 456 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @IONA ,Definitely, I would say that’s even a common practice. Create a feature branch and make the necessary changes there. But once a day, merge into that feature branch all the changes that have appeared on your main branch. That way, you will a...
- 0 kudos
| User | Count |
|---|---|
| 1629 | |
| 790 | |
| 511 | |
| 349 | |
| 287 |
