cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Lakehouse Architecture


Forum Posts

Start a conversation
scott1
by • New Contributor II
  • 3127 Views
  • 1 replies
  • 0 kudos

Automatic UC enablement with existing Unity Catalog

Hello,The environment I'm working in has an existing account and unity catalog in our primary region. These have been primarily PoC, and have existed for a year or so.Now we're building out our IaC and getting ready to move towards production. In new...

Administration & Architecture
Reply
  • 3127 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
Hello,The environment I'm working in has an existing account and unity catalog in our primary region. These have been primarily PoC, and have existed for a year or so.Now we're building out our IaC and getting ready to move towards production. In new...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
leelee3000
by • Databricks Employee
  • 2685 Views
  • 1 replies
  • 0 kudos

Connect to Salesforce

Curious if there's a Databricks connector for Salesforce on AWS? 

Administration & Architecture
Reply
  • 2685 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Wojciech_BUK
Valued Contributor III
  • 0 kudos

There is no "databricks" connector like the on you have in Unity Fedarating e.g. for Snowflake.You can use partner ecosystem e.g. Fivetran https://www.fivetran.com/connectors/salesforceto integrate Salesforce data to your Lakehouse. You also have spa...

  • 0 kudos
riturralde-p
by • New Contributor II
  • 3413 Views
  • 1 replies
  • 1 kudos

How to allocate costs per SQL query?

By using System Tables (systen.billing.usage) I'm able to identity DBU usage per query, but I'm not able to identify who ran each query because is not part of the table. I'm also aware of query history where all the queries and who ran them is listed...

Administration & Architecture
Reply
  • 3413 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
riturralde-p
New Contributor II
  • 1 kudos

thanks @Retired_mod for the reply, however query_id is not part of the system.billing.usage table, so no way to join them by IDs. What my Databricks account team suggested me is to join them by timestamps since both tables contain a column like that....

  • 1 kudos
lgepp11
by • New Contributor III
  • 4455 Views
  • 2 replies
  • 0 kudos

NPIP tunnel setup failure during launch

In AWS with this current error when spinning up the SQL warehouse or personal compute.Backend private link is enabled.Error: NPIP tunnel setup failure during launch. Please try again later and contact Databricks if the problem persists. Instance boot...

Administration & Architecture
Reply
  • 4455 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
User16539034020
Databricks Employee
  • 0 kudos

Hello,  Thanks for contacting Databricks Support.  Based on the error message: NPIP_TUNNEL_SETUP_FAILURE. It indicates that bootstrap failed due to network connectivity issues between the data plane and control plane.  Seems like you have already dow...

  • 0 kudos
1 More Replies
alexlod
by • New Contributor III
  • 4246 Views
  • 2 replies
  • 1 kudos

How to monitor a python wheel job with Prometheus?

Hi Community,We have a Databricks job with a single Python wheel task that runs our streaming pyspark job. The job runs on a single-node compute cluster and consumes from Kafka.Our monitoring stack is Prometheus + Grafana.I want the job's metrics to ...

Administration & Architecture
Reply
  • 4246 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
amelia1
New Contributor II
  • 1 kudos

Hi I'm trying to use the metrics registry object inside an UDF function, but I can't because it's not serializable due to Lock. Our goal is to be able to count the number of messages parsed, and the number of messages we can't parsed (due to exceptio...

  • 1 kudos
1 More Replies
leelee3000
by • Databricks Employee
  • 1031 Views
  • 0 replies
  • 0 kudos

Handling Kafka Topics with Avro Schema

Our input data resides in a Kafka topic, and we utilize the Kafka schema registry with Avro schemas. While I can retrieve the schema from the registry, I am facing challenges creating a Spark DataFrame that correctly serializes data for streaming rea...

Administration & Architecture
Reply
  • 1031 Views
  • 0 replies
  • 0 kudos
leelee3000
by • Databricks Employee
  • 674 Views
  • 0 replies
  • 0 kudos

Scalable API/binary lookups

We sometimes process large dataframes that contain a column of IP addresses and we need to associate an Autonomous System Number (ASN) per IP address. The ASN information is provided by MaxMind in the form of a binary data file only accessible via a ...

Administration & Architecture
Reply
  • 674 Views
  • 0 replies
  • 0 kudos
leelee3000
by • Databricks Employee
  • 1137 Views
  • 0 replies
  • 0 kudos

auto-deleted workspace on GCP

Our production Databricks workspaces were auto-deleted when our subscription from our GCP account was canceled due to system error.  We have a backup of the GCS buckets that Databricks was running on (not workspace exports).  Is it possible to recove...

Administration & Architecture
Reply
  • 1137 Views
  • 0 replies
  • 0 kudos
552532
by • New Contributor II
  • 1503 Views
  • 1 replies
  • 0 kudos

Databricks Private link connectivity for External SaaS Application

We need your guidance on completing the set-up around private link set-up with a Customer who is in the same region in AWS where our application is hosted in AWS. Our Customer has already enabled Private Link in their account and they are using custo...

Administration & Architecture
Reply
  • 1503 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
552532
New Contributor II
  • 0 kudos

Hi Databricks Support,We followed the instructions above, but we are getting an error when registering the endpoint in customer databricks environment using "Registering Endpoint".Following is the error message we see:"INVALID_PARAMETER_VALUE" Endpoi...

  • 0 kudos
Databricks24
by • New Contributor
  • 1622 Views
  • 0 replies
  • 0 kudos

UserAgentEntry added to JDBC URL but not visible in Audit logs

Hi,As part of Databricks Best Practices, I have added 'UserAgentEntry' to JDBC URL that is being created when we are executing SQL statements through the JDBC driver.Sample url - jdbc:databricks://<host>:443;httpPath=<httpPath>; AuthMech=3;UID=token;...

Administration & Architecture
user-agent
Reply
  • 1622 Views
  • 0 replies
  • 0 kudos
alexometis
by • New Contributor III
  • 7107 Views
  • 2 replies
  • 3 kudos

System Tables Preview - retention period?

The new System Tables for billing, pricing & compute look really useful and easier to consume than getting it via the APIs.However I can't see in the documentation:Does data only start being gathered when you turn them on or is there immediately a hi...

Administration & Architecture
Reply
  • 7107 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Avvar2022
Contributor
  • 3 kudos

@Retired_mod  -We are customer of databricks. Have databricks premium workspace with unity catalog enabled. and we have also legacy workspaces (non-unity enabled).I can see history is available for all workspaces (unity and non-unity) in same meta st...

  • 3 kudos
1 More Replies
smart5mk
by • New Contributor III
  • 1945 Views
  • 0 replies
  • 0 kudos

Destination Path of Cloned Notebooks

Hi, for my project I need to get destination paths of cloned notebooks. But when I run the query to get them: ''SELECT DISTINCT request_params.destinationPathFROM system.access.auditWHERE service_name = "notebook"andaction_name = 'cloneNotebook'LIMIT...

Administration & Architecture
Reply
  • 1945 Views
  • 0 replies
  • 0 kudos
JDL
by • New Contributor III
  • 12377 Views
  • 4 replies
  • 2 kudos

Get number of rows in delta lake table from metadata without count(*)

Hello folks,Is there a way with sql query to get count from delta table metadata without doing count(*) on each of table? Wondering, if this information is stored in any of INFORMATION_SCHEMA tables.I have a use-case to get counts from 1000's of delt...

Administration & Architecture
Reply
  • 12377 Views
  • 4 replies
  • 2 kudos
View Replies
Latest Reply
SSundaram
Contributor
  • 2 kudos

 Here is a related one.https://community.databricks.com/t5/data-engineering/how-to-get-the-total-number-of-records-in-a-delta-table-from-the/td-p/20441

  • 2 kudos
3 More Replies
karthik_p
by • Esteemed Contributor
  • 1551 Views
  • 3 replies
  • 0 kudos

Disaster Recovery Issue

We are trying to create Disaster Recovery for UC enabled Workspaces in Azure. our UC metastore are in different regions.1. we are trying to use Deep Clone2. In source we are adding region2 metastore as external location3. able to do deep cloneproblem...

Administration & Architecture
Reply
  • 1551 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

Right I get it.Actually cloning it as external seems logical to me, for the moment, as unity cannot manage the other metastore.For the moment I would go with cloning the data and then creating an external table of that.Not ideal, but at least you hav...

  • 0 kudos
2 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels