cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

ErikApption
by New Contributor II
  • 3076 Views
  • 1 replies
  • 0 kudos

Notebook runs not found due to retention limits with dbutils.notebook.run

We saw this odd error in an AWS deployment, we have one notebook calling another one through dbutils.notebook.run(...) and this suddenly stopped working and failed with "Notebook runs not found due to retention limits", the "learn more" points to Dat...

  • 3076 Views
  • 1 replies
  • 0 kudos
Latest Reply
Isi
Honored Contributor II
  • 0 kudos

Hey @ErikApption ,Maybe I am wrong but I will give you my opinion.Each time you execute dbutils.notebook.run(), it launches a new and independent execution within the same cluster. So, if you run the cell today and then run it again tomorrow, there s...

  • 0 kudos
tom_1
by New Contributor III
  • 2580 Views
  • 0 replies
  • 0 kudos

Customer Managed VPC: Databricks IP Address Ranges

Hello,how often does Databricks change its public ip addresses (the ones that must be whitelisted in a customer managed vpc) and where can I find them?I found this list, but it seems to be incomplete.We moved from a managed vpc to a customer-managed ...

  • 2580 Views
  • 0 replies
  • 0 kudos
zsucic1
by New Contributor III
  • 9827 Views
  • 3 replies
  • 1 kudos

Current Azure Managed Identity capabilities 2024?

Hello everyone, I have a few questions about MI capabilites: Is it possible to define a managed identity for Azure Databricks Service resource and use it for e.g.: Writing to Azure SQL Server database Authenticating to Azure Devops in order to downlo...

  • 9827 Views
  • 3 replies
  • 1 kudos
Latest Reply
zsucic1
New Contributor III
  • 1 kudos

Kaniz, thank you very much, you are the best! I will get to work implementing your advice

  • 1 kudos
2 More Replies
sam_tw
by New Contributor II
  • 2695 Views
  • 1 replies
  • 1 kudos

Resolved! Community Edition - Photon enabled possible?

Is it possible to use a photon enabled cluster in the community edition? I want to use DBR 13.3 LTS, but choosing that there is no option to enable photon.  I want to use test the spatial functionality in Databricks library Mosiac, and appears photon...

  • 2695 Views
  • 1 replies
  • 1 kudos
Latest Reply
Stefan-Koch
Valued Contributor II
  • 1 kudos

Hi @sam_tw Photon is not available in Community Edition

  • 1 kudos
SarahA
by New Contributor II
  • 2036 Views
  • 1 replies
  • 1 kudos

Resolved! Databricks app - permissions needed

Hi. I am trying to create a new databricks app and I get the following error"Failed to create app [appname]. User does not have permission to grant resource sql-warehouse."Can someone tell me what level of access I require in order to generate a data...

  • 2036 Views
  • 1 replies
  • 1 kudos
Latest Reply
lingareddy_Alva
Honored Contributor III
  • 1 kudos

Hi @SarahA ,Requied Permissions to Create a Data AppYou'll need the following permissions in Databricks:Permission Required Role / Grant Purpose CAN MANAGE on the SQL WarehouseSQL Warehouse Admin or OwnerTo manage warehouse settings and assign it to ...

  • 1 kudos
DavidSzedlak
by New Contributor
  • 1486 Views
  • 2 replies
  • 0 kudos

Mismatch of Columns in databricks vs Athena.

We are trying to expose one of our external tables to databricks via unity catalog, but we are having an issue with column mismatch, ie few of our columns are not visible in databricks. Is this a known issue? If so, can anyone best advise me on where...

  • 1486 Views
  • 2 replies
  • 0 kudos
Latest Reply
lingareddy_Alva
Honored Contributor III
  • 0 kudos

Hi @DavidSzedlak ,1.  Unity Catalog caches metadata for performance. If new columns were added to the source table after the initial creation, they may not be reflected.Run the following command to refresh metadata: ALTER TABLE <catalog>.<schema>.<ta...

  • 0 kudos
1 More Replies
kch
by New Contributor II
  • 984 Views
  • 3 replies
  • 0 kudos

Timeout on docker pull in Databricks Container Services

Hello,There is a timeout that limits the size of images used in Docker Container Service. When using images containing large ML libraries, the size often exceeds the limit that could be pulled. Is there any plan to add parametrization of this timeout...

  • 984 Views
  • 3 replies
  • 0 kudos
Latest Reply
kch
New Contributor II
  • 0 kudos

Are there any new or planned changes in the policy?

  • 0 kudos
2 More Replies
Aaron_Wong
by New Contributor II
  • 2716 Views
  • 8 replies
  • 0 kudos

How do I simply disable someone's user account

I'm trying to do something seemingly very simple - disable someone's user account. I don't even want to delete the user, just disable it for the time being. How do I go about doing that? 

  • 2716 Views
  • 8 replies
  • 0 kudos
Latest Reply
maaz1
New Contributor II
  • 0 kudos

 In Databricks, go to the Admin Console.Navigate to the Service Principals section.Create a Service Role (make sure it has necessary permissions (Admin Access).Generate OAuth TokenFollow the instructions in the Databricks documentation to generate ap...

  • 0 kudos
7 More Replies
harripy
by New Contributor III
  • 2355 Views
  • 2 replies
  • 0 kudos

Proper way to collect Statement ID from JDBC Connection

Hi, We are executing DML calls on Databricks SQL Warehouse programmatically, with Java and Python.There can be thousands of executions running on daily level, so in case of an error occurs, it would be very beneficial to spot the Statement ID of the ...

  • 2355 Views
  • 2 replies
  • 0 kudos
Latest Reply
oprime
New Contributor II
  • 0 kudos

Found a way to extract it for below dbx-java library.-> databricks-jdbc library version : `2.6.32` private static String extractQueryIdFromDbxStatement(Hive42PreparedStatement statement) { byte[] guid = ((HiveJDBCNativeQueryExecutor) state...

  • 0 kudos
1 More Replies
bhanu_dp
by New Contributor III
  • 2764 Views
  • 0 replies
  • 0 kudos

How to know Legacy Metastore connection to SQL DB (used to store metadata)

I am logged into a workspace and trying to check the schemas in legacy hive_metastore using a serverless compute, I can see the schemas listed.However, when I am creating all-purpose cluster and trying to check the schemas in legacy hive_metastore. I...

Administration & Architecture
configuration
Hive
metastore
  • 2764 Views
  • 0 replies
  • 0 kudos
AnkurMittal008
by New Contributor III
  • 1166 Views
  • 1 replies
  • 0 kudos

Secret Creation for Service Principal using API

There is API available to create a secret for Service Principal./api/2.0/accounts/{account_id}/servicePrincipals/{service_principal_id}/credentials/secretsCan anyone please help what has to be passed as authentication for this API ? This is looking a...

  • 1166 Views
  • 1 replies
  • 0 kudos
Latest Reply
Stefan-Koch
Valued Contributor II
  • 0 kudos

hi @AnkurMittal008 In general, I do not recommend using tokens anymore. Instead, if you want to log in via databricks cli, you can use this command:databricks auth login --host https://accounts.azuredatabricks.net/ --account-id <YOUR_ACCOUNT_ID>This ...

  • 0 kudos
jeet414
by New Contributor
  • 672 Views
  • 1 replies
  • 0 kudos

Reading delta table from ADLS gen2 using ABFS driver

Scenario - I have an ADLS g2 account and trying to read a delta table using ABFS driver. I am using Databricks Serverless compute. There are no firewalls in place as I am working with sample data. There's network line of sight between Databricks serv...

  • 672 Views
  • 1 replies
  • 0 kudos
Latest Reply
Nivethan_Venkat
Contributor III
  • 0 kudos

Hi @jeet414,You can always read Delta table from ADLS storage in Databricks mount points: https://learn.microsoft.com/en-us/azure/databricks/dbfs/mountsThis is just experimental, please give it a try.Best Regards,Nivethan V

  • 0 kudos
PaulJones
by New Contributor
  • 408 Views
  • 1 replies
  • 0 kudos

Querying view gives spurious error

When trying to query a view, my_view, we sometimes see a spurious error. This seems to occur after the table underlying the view has been updated. The error persists for a while and then it seems to fix itself. Error running query: [42501] [Simba][Ha...

  • 408 Views
  • 1 replies
  • 0 kudos
Latest Reply
mnorland
Valued Contributor
  • 0 kudos

Are there DDL updates of the view going on?

  • 0 kudos