We sometimes process large dataframes that contain a column of IP addresses and we need to associate an Autonomous System Number (ASN) per IP address. The ASN information is provided by MaxMind in the form of a binary data file only accessible via a ...
Our production Databricks workspaces were auto-deleted when our subscription from our GCP account was canceled due to system error. We have a backup of the GCS buckets that Databricks was running on (not workspace exports). Is it possible to recove...
We need your guidance on completing the set-up around private link set-up with a Customer who is in the same region in AWS where our application is hosted in AWS. Our Customer has already enabled Private Link in their account and they are using custo...
Hi Databricks Support,We followed the instructions above, but we are getting an error when registering the endpoint in customer databricks environment using "Registering Endpoint".Following is the error message we see:"INVALID_PARAMETER_VALUE" Endpoi...
Hi community We are using Unity Catalog, SCIM and Identity Federation, so we have users, groups and service principals at Account level. In what scenarios do users, groups and service principals need explicitly added to a Workspace?
1. If you enable Unity Catalog in a workspace, users in that workspace may be able to access the same data that users in other workspaces in your account can access. Data guardians can control who has access to what data across all workspaces from on...
The new System Tables for billing, pricing & compute look really useful and easier to consume than getting it via the APIs.However I can't see in the documentation:Does data only start being gathered when you turn them on or is there immediately a hi...
@Retired_mod -We are customer of databricks. Have databricks premium workspace with unity catalog enabled. and we have also legacy workspaces (non-unity enabled).I can see history is available for all workspaces (unity and non-unity) in same meta st...
Hi, for my project I need to get destination paths of cloned notebooks. But when I run the query to get them: ''SELECT DISTINCT request_params.destinationPathFROM system.access.auditWHERE service_name = "notebook"andaction_name = 'cloneNotebook'LIMIT...
Hello folks,Is there a way with sql query to get count from delta table metadata without doing count(*) on each of table? Wondering, if this information is stored in any of INFORMATION_SCHEMA tables.I have a use-case to get counts from 1000's of delt...
Here is a related one.https://community.databricks.com/t5/data-engineering/how-to-get-the-total-number-of-records-in-a-delta-table-from-the/td-p/20441
We are trying to create Disaster Recovery for UC enabled Workspaces in Azure. our UC metastore are in different regions.1. we are trying to use Deep Clone2. In source we are adding region2 metastore as external location3. able to do deep cloneproblem...
Right I get it.Actually cloning it as external seems logical to me, for the moment, as unity cannot manage the other metastore.For the moment I would go with cloning the data and then creating an external table of that.Not ideal, but at least you hav...
What does everyone think about ingesting source data to the raw layer via a microservice rather than direct from the source API?
We are trying to create the cluster within the databricks workspace but it is generating the error attach
Hello All,I have databricks delta table with files residing in Azure Data Lake. I understand, when I load create table and load data from databricks, it creates respective folder and files for table in ADLS. I am wondering if there is reverse way to ...
Thanks @Retired_mod this is helpful
When we run jobs using service principals system.audit doesn't show any table accesses (getTable). Volume (getVolume) shows up for service principals. Same query when run as a user shows up in system.audit. I know system.audit is in public preview. W...
hi @Retired_mod thanks so much for your reply ! I was referring to https://docs.databricks.com/en/administration-guide/system-tables/audit-logs.html which is part of databricks core offering and isn't related to ServiceNow's offering. I am assuming t...
Hi all, I have not been successful in getting a good grip of the naming conventions for the three level name space. Initially i learned about bronze, silver and gold, but i am confused where to put this. The obvious choice may be to use the {catalog}...
I am currently using a personal computer cluster [13.3 LTS (includes Apache Spark 3.4.1, Scala 2.12)] on GCP attached to a notebook. After running a few command lines without an issue, I end up getting this error Internal error. Attach your notebook...
I am running into an issue where "Standard_NC8as_T4_v3" and "Standard_NC4as_T4_v3" instances are behaving differently for a 30gb custom docker image, and I am a bit stumped.when using NC4 instances, I get a timeout, with the exact message shown below...
