- 22992 Views
- 4 replies
- 2 kudos
Get number of rows in delta lake table from metadata without count(*)
Hello folks,Is there a way with sql query to get count from delta table metadata without doing count(*) on each of table? Wondering, if this information is stored in any of INFORMATION_SCHEMA tables.I have a use-case to get counts from 1000's of delt...
- 22992 Views
- 4 replies
- 2 kudos
- 2 kudos
Here is a related one.https://community.databricks.com/t5/data-engineering/how-to-get-the-total-number-of-records-in-a-delta-table-from-the/td-p/20441
- 2 kudos
- 2832 Views
- 3 replies
- 0 kudos
Disaster Recovery Issue
We are trying to create Disaster Recovery for UC enabled Workspaces in Azure. our UC metastore are in different regions.1. we are trying to use Deep Clone2. In source we are adding region2 metastore as external location3. able to do deep cloneproblem...
- 2832 Views
- 3 replies
- 0 kudos
- 0 kudos
Right I get it.Actually cloning it as external seems logical to me, for the moment, as unity cannot manage the other metastore.For the moment I would go with cloning the data and then creating an external table of that.Not ideal, but at least you hav...
- 0 kudos
- 2418 Views
- 0 replies
- 0 kudos
Source data for Raw layer - API's vs Microservice
What does everyone think about ingesting source data to the raw layer via a microservice rather than direct from the source API?
- 2418 Views
- 0 replies
- 0 kudos
- 3055 Views
- 1 replies
- 0 kudos
error when trying to create a cluster in databricks
We are trying to create the cluster within the databricks workspace but it is generating the error attach
- 3055 Views
- 1 replies
- 0 kudos
- 3921 Views
- 1 replies
- 1 kudos
Tie Parquet files in Azure ADLS to Databricks table
Hello All,I have databricks delta table with files residing in Azure Data Lake. I understand, when I load create table and load data from databricks, it creates respective folder and files for table in ADLS. I am wondering if there is reverse way to ...
- 3921 Views
- 1 replies
- 1 kudos
- 1 kudos
Thanks @Retired_mod this is helpful
- 1 kudos
- 7914 Views
- 1 replies
- 0 kudos
service principal table accesses not showing up in system.audit
When we run jobs using service principals system.audit doesn't show any table accesses (getTable). Volume (getVolume) shows up for service principals. Same query when run as a user shows up in system.audit. I know system.audit is in public preview. W...
- 7914 Views
- 1 replies
- 0 kudos
- 0 kudos
hi @Retired_mod thanks so much for your reply ! I was referring to https://docs.databricks.com/en/administration-guide/system-tables/audit-logs.html which is part of databricks core offering and isn't related to ServiceNow's offering. I am assuming t...
- 0 kudos
- 4075 Views
- 1 replies
- 1 kudos
Three level name space naming standard
Hi all, I have not been successful in getting a good grip of the naming conventions for the three level name space. Initially i learned about bronze, silver and gold, but i am confused where to put this. The obvious choice may be to use the {catalog}...
- 4075 Views
- 1 replies
- 1 kudos
- 6144 Views
- 4 replies
- 2 kudos
Internal error: Attach your notebook to a different compute or restart the current compute.
I am currently using a personal computer cluster [13.3 LTS (includes Apache Spark 3.4.1, Scala 2.12)] on GCP attached to a notebook. After running a few command lines without an issue, I end up getting this error Internal error. Attach your notebook...
- 6144 Views
- 4 replies
- 2 kudos
- 2744 Views
- 0 replies
- 0 kudos
Standard_NC8as_T4_v3" and "Standard_NC4as_T4_v3" instances
I am running into an issue where "Standard_NC8as_T4_v3" and "Standard_NC4as_T4_v3" instances are behaving differently for a 30gb custom docker image, and I am a bit stumped.when using NC4 instances, I get a timeout, with the exact message shown below...
- 2744 Views
- 0 replies
- 0 kudos
- 5636 Views
- 1 replies
- 0 kudos
Resolved! Error: cannot create metastore data access
I'm in the progress of enabling Databricks Unity Catalog and encountered a problem with the databricks_metastore_data_access Terraform resource:resource "databricks_metastore_data_access" "this" { provider = databricks.account-level metastore_id...
- 5636 Views
- 1 replies
- 0 kudos
- 0 kudos
Found a solution. Please see my answer on Stack Overflow:https://stackoverflow.com/questions/77440091/databricks-unity-catalog-error-cannot-create-metastore-data-access/77506306#77506306
- 0 kudos
- 5194 Views
- 4 replies
- 2 kudos
Failed to start cluster: Large docker image
I have a large Docker image in our AWS ECR repo. The image is 27.4 GB locally and 11539.79 MB compressed in ECR.The error from the Event Log is:Failed to add 2 containers to the compute. Will attempt retry: true. Reason: Docker image pull failureJSON...
- 5194 Views
- 4 replies
- 2 kudos
- 2 kudos
I have a similar problem. a 10gb image pulls fine but a 31gb image doesnt. both workers and drivers have 64gb memory. i get the timeout error with "Cannot launch the cluster because pulling the docker image failed. Please double check connectivity fr...
- 2 kudos
- 5762 Views
- 0 replies
- 0 kudos
Is it possible to change the Azure storage account of Unity Catalog?
We have unity catalog metastore set up in storage account prod_1. Can we move this to prod_2 storage account and delete prod_1?Also, is it possible to rename the catalogs once they are created?
- 5762 Views
- 0 replies
- 0 kudos
- 6180 Views
- 3 replies
- 2 kudos
Connect to databricks from external non-spark cluster
Hi,I have an app/service on a non-spark kubernetes cluster. Is there a way to access/query a databricks service from my app/service? I see documentations on connectors, particularly on scala which is the code of my app/service. Can I use these connec...
- 6180 Views
- 3 replies
- 2 kudos
- 3465 Views
- 0 replies
- 0 kudos
Databricks deployment and automation tools comparison.
Hello All, As a newcomer to databricks, I am seeking guidance on automation within databricks environments. What are the best best practices for deployment, and how do Terraform, the REST API, and the databricks SDK compare in terms of advantages and...
- 3465 Views
- 0 replies
- 0 kudos
- 2330 Views
- 1 replies
- 0 kudos
Notebook Id level of uniqueness
Hi there,We know that notebook ids are unique. https://docs.databricks.com/en/workspace/workspace-details.html but I want to know in what level they're unique. For example, if Notebook Ids are unique within a workspace, or are they universally unique...
- 2330 Views
- 1 replies
- 0 kudos
