cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

alexlod
by New Contributor III
  • 3126 Views
  • 4 replies
  • 1 kudos

How to monitor a python wheel job with Prometheus?

Hi Community,We have a Databricks job with a single Python wheel task that runs our streaming pyspark job. The job runs on a single-node compute cluster and consumes from Kafka.Our monitoring stack is Prometheus + Grafana.I want the job's metrics to ...

  • 3126 Views
  • 4 replies
  • 1 kudos
Latest Reply
amelia1
New Contributor II
  • 1 kudos

Hi I'm trying to use the metrics registry object inside an UDF function, but I can't because it's not serializable due to Lock. Our goal is to be able to count the number of messages parsed, and the number of messages we can't parsed (due to exceptio...

  • 1 kudos
3 More Replies
leelee3000
by New Contributor III
  • 734 Views
  • 0 replies
  • 0 kudos

Handling Kafka Topics with Avro Schema

Our input data resides in a Kafka topic, and we utilize the Kafka schema registry with Avro schemas. While I can retrieve the schema from the registry, I am facing challenges creating a Spark DataFrame that correctly serializes data for streaming rea...

  • 734 Views
  • 0 replies
  • 0 kudos
leelee3000
by New Contributor III
  • 521 Views
  • 1 replies
  • 0 kudos

auto-deleted workspace on GCP

Our production Databricks workspaces were auto-deleted when our subscription from our GCP account was canceled due to system error.  We have a backup of the GCS buckets that Databricks was running on (not workspace exports).  Is it possible to recove...

  • 521 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @leelee3000, I apologize for the inconvenience you’ve faced with the auto-deletion of your Databricks workspaces.   Let’s explore the possibilities for recovery:   Notebooks Recovery: Deleted notebooks are moved to the Trash folder within Databric...

  • 0 kudos
leelee3000
by New Contributor III
  • 367 Views
  • 1 replies
  • 0 kudos

Scalable API/binary lookups

We sometimes process large dataframes that contain a column of IP addresses and we need to associate an Autonomous System Number (ASN) per IP address. The ASN information is provided by MaxMind in the form of a binary data file only accessible via a ...

  • 367 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @leelee3000, You have a couple of options to associate Autonomous System Numbers (ASNs) with IP addresses in Databricks.   Let’s explore them:   Using MaxMind’s GeoLite2 ASN Database: MaxMind provides a GeoLite2 ASN Database that contains informat...

  • 0 kudos
552532
by New Contributor II
  • 848 Views
  • 2 replies
  • 0 kudos

Databricks Private link connectivity for External SaaS Application

We need your guidance on completing the set-up around private link set-up with a Customer who is in the same region in AWS where our application is hosted in AWS. Our Customer has already enabled Private Link in their account and they are using custo...

  • 848 Views
  • 2 replies
  • 0 kudos
Latest Reply
552532
New Contributor II
  • 0 kudos

Hi Databricks Support,We followed the instructions above, but we are getting an error when registering the endpoint in customer databricks environment using "Registering Endpoint".Following is the error message we see:"INVALID_PARAMETER_VALUE" Endpoi...

  • 0 kudos
1 More Replies
harvey-c
by New Contributor III
  • 4520 Views
  • 3 replies
  • 0 kudos

Resolved! Any way to move the unity catalog to a new external storage location?

Dear Databricks CommunityThe question is about changing an existing unity catalog to a new storage location. For example: With an existing unity catalog (i.e. catalog1)  includeing schemas and volumes. The catalog is based on an external location (i....

  • 4520 Views
  • 3 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?This will also help other community members who may have similar questions in the fut...

  • 0 kudos
2 More Replies
smart5mk
by New Contributor III
  • 1470 Views
  • 1 replies
  • 0 kudos

Resolved! Destination Path of Cloned Notebooks

Hi, for my project I need to get destination paths of cloned notebooks. But when I run the query to get them: ''SELECT DISTINCT request_params.destinationPathFROM system.access.auditWHERE service_name = "notebook"andaction_name = 'cloneNotebook'LIMIT...

  • 1470 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @smart5mk, The numbers you’re seeing in the destination paths of cloned notebooks are likely unique identifiers rather than human-readable paths. These identifiers are often used internally by the system to manage resources efficiently.   To decod...

  • 0 kudos
JDL
by New Contributor III
  • 5380 Views
  • 7 replies
  • 2 kudos

Get number of rows in delta lake table from metadata without count(*)

Hello folks,Is there a way with sql query to get count from delta table metadata without doing count(*) on each of table? Wondering, if this information is stored in any of INFORMATION_SCHEMA tables.I have a use-case to get counts from 1000's of delt...

  • 5380 Views
  • 7 replies
  • 2 kudos
Latest Reply
SSundaram
Contributor
  • 2 kudos

 Here is a related one.https://community.databricks.com/t5/data-engineering/how-to-get-the-total-number-of-records-in-a-delta-table-from-the/td-p/20441

  • 2 kudos
6 More Replies
alexometis
by New Contributor III
  • 3309 Views
  • 5 replies
  • 5 kudos

Resolved! System Tables Preview - retention period?

The new System Tables for billing, pricing & compute look really useful and easier to consume than getting it via the APIs.However I can't see in the documentation:Does data only start being gathered when you turn them on or is there immediately a hi...

  • 3309 Views
  • 5 replies
  • 5 kudos
Latest Reply
Avvar2022
New Contributor III
  • 5 kudos

@Kaniz  -We are customer of databricks. Have databricks premium workspace with unity catalog enabled. and we have also legacy workspaces (non-unity enabled).I can see history is available for all workspaces (unity and non-unity) in same meta store re...

  • 5 kudos
4 More Replies
karthik_p
by Esteemed Contributor
  • 977 Views
  • 3 replies
  • 0 kudos

Disaster Recovery Issue

We are trying to create Disaster Recovery for UC enabled Workspaces in Azure. our UC metastore are in different regions.1. we are trying to use Deep Clone2. In source we are adding region2 metastore as external location3. able to do deep cloneproblem...

  • 977 Views
  • 3 replies
  • 0 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

Right I get it.Actually cloning it as external seems logical to me, for the moment, as unity cannot manage the other metastore.For the moment I would go with cloning the data and then creating an external table of that.Not ideal, but at least you hav...

  • 0 kudos
2 More Replies
Jen_Harvey
by New Contributor
  • 522 Views
  • 1 replies
  • 1 kudos

Source data for Raw layer - API's vs Microservice

What does everyone think about ingesting source data to the raw layer via a microservice rather than direct from the source API?

  • 522 Views
  • 1 replies
  • 1 kudos
Latest Reply
Kaniz
Community Manager
  • 1 kudos

Hi @Jen_Harvey, the discussion around microservices and APIs in software architecture often sparks intriguing debates. Let’s delve into the nuances of each: Microservices: Definition: Microservices are small, self-contained programs that break down a...

  • 1 kudos
Kaviana
by New Contributor III
  • 868 Views
  • 2 replies
  • 0 kudos

error when trying to create a cluster in databricks

We are trying to create the cluster within the databricks workspace but it is generating the error attach

  • 868 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @Kaviana, Hey, I noticed you're facing an issue setting up a cluster in your Databricks workspace.   Let's troubleshoot this.   Access Denied Error: The "Access Denied" error suggests you lack the permissions needed to create a cluster. Users with...

  • 0 kudos
1 More Replies
JDL
by New Contributor III
  • 1818 Views
  • 2 replies
  • 2 kudos

Resolved! Tie Parquet files in Azure ADLS to Databricks table

Hello All,I have databricks delta table with files residing in Azure Data Lake. I understand, when I load create table and load data from databricks, it creates respective folder and files for table in ADLS. I am wondering if there is reverse way to ...

  • 1818 Views
  • 2 replies
  • 2 kudos
Latest Reply
JDL
New Contributor III
  • 2 kudos

Thanks @Kaniz this is helpful

  • 2 kudos
1 More Replies
RonMoody
by New Contributor II
  • 2773 Views
  • 2 replies
  • 0 kudos

service principal table accesses not showing up in system.audit

When we run jobs using service principals system.audit doesn't show any table accesses (getTable). Volume (getVolume) shows up for service principals. Same query when run as a user shows up in system.audit. I know system.audit is in public preview. W...

  • 2773 Views
  • 2 replies
  • 0 kudos
Latest Reply
RonMoody
New Contributor II
  • 0 kudos

hi @Kaniz thanks so much for your reply ! I was referring to https://docs.databricks.com/en/administration-guide/system-tables/audit-logs.html which is part of databricks core offering and isn't related to ServiceNow's offering. I am assuming they ar...

  • 0 kudos
1 More Replies
cltj
by New Contributor III
  • 955 Views
  • 2 replies
  • 1 kudos

Three level name space naming standard

Hi all, I have not been successful in getting a good grip of the naming conventions for the three level name space. Initially i learned about bronze, silver and gold, but i am confused where to put this. The obvious choice may be to use the {catalog}...

  • 955 Views
  • 2 replies
  • 1 kudos
Latest Reply
Kaniz
Community Manager
  • 1 kudos

Hi @cltj ," I am also looking for others comments on my attached picture. " Could not find any.  

  • 1 kudos
1 More Replies
Labels