- 2401 Views
- 1 replies
- 0 kudos
Resolved! Unable to create a Managed table in Unity Catalog defaul location
We have setup the metastore with Manged Identity and when trying to create a managed table in the default location I am hitting below error. The storage is ADLS Gen2. AbfsRestOperationException: Operation failed: "This request is not authorized to pe...
- 2401 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Kris2 , I completely understand your hesitation and appreciate your approach to seeking guidance! This error generally means that the cluster has connectivity to Unity Catalog configured storage location and it is not authorized to access storag...
- 0 kudos
- 743 Views
- 0 replies
- 1 kudos
Error Signing Up for Databricks Community Edition
@Retired_mod I've been trying to sign up for Databricks Community Edition using different email addresses over the past 24 hours, but I keep getting the error message: "An error has occurred. Please try again later." Can anyone help?Tags: #Databricks...
- 743 Views
- 0 replies
- 1 kudos
- 1754 Views
- 0 replies
- 0 kudos
Unity Catalog: Databricks *Specific* Features
Good day,Deceptively simple question, are there any "Databricks only" specific features that Unity Catalog offers? I understand that generally speaking enabling UC offers some of the following:Data Discovery and LineageAuditing and MonitoringAccess C...
- 1754 Views
- 0 replies
- 0 kudos
- 3481 Views
- 2 replies
- 0 kudos
Problem login in
Hello allI´m new in this platform, I sign up, validated my email, create my password everything is fine and when I try to log in and start a message came upI create a new password but same happen again! But it works a few times, I think like 3 times....
- 3481 Views
- 2 replies
- 0 kudos
- 0 kudos
Can you confirm the username was created all lower case? Login is case sensitive so you need to make sure the username is set same exact as you add it in the console or workspace
- 0 kudos
- 6427 Views
- 3 replies
- 3 kudos
Resolved! DLT Job Clusters: Continuous vs Triggered Cluster Start Times
Hi there,I'm curious if anyone is able to definitively help me answer how DLT Job Clusters operate/run.For example, the following is my baseline understanding of DLT Job Clusters. If I run a Triggered DLT Pipeline (e.g. daily) the job cluster takes m...
- 6427 Views
- 3 replies
- 3 kudos
- 3 kudos
Ideally one would expect clusters used for DLT pipeline to terminate after the pipeline execution has finished. However, while running in `development` environment, you'll notice it doesn't terminate on its own, whereas in `production` it terminates ...
- 3 kudos
- 2608 Views
- 1 replies
- 0 kudos
Databricks Java - Create Jar in Java 11
I am trying to a run simple print java program which is not working and getting compilation version issues though i changed the environment variable points to java 11. Can you please help me ? Can we create java with spark session and execute as a ja...
- 2608 Views
- 1 replies
- 0 kudos
- 0 kudos
@Databricks_Java You can run command like this: spark-submit --class com.test.Main example.jarand make sure to check the java version and match with the DBR compatibility
- 0 kudos
- 1466 Views
- 1 replies
- 0 kudos
Disable personal compute with the Databricks API or UI
For a production environment, I want to disable the personal compute policy, because I do not want that all users can create personal compute clusters in production. Unfortunately, I am not able to access the account console, so I want to revoke perm...
- 1466 Views
- 1 replies
- 0 kudos
- 0 kudos
@mathijs-fish You need to be admin to disable a policy.
- 0 kudos
- 1686 Views
- 3 replies
- 0 kudos
Autoloader file latency
Hi Team,I would like to understand if there is a metadata table for the autoloader in Databricks that captures information about file arrival and processing.The reason we are experiencing data issues is because our table A receives hundreds of files ...
- 1686 Views
- 3 replies
- 0 kudos
- 0 kudos
Check with cloud_files_state() API You can find examples here https://docs.databricks.com/en/ingestion/auto-loader/production.html#querying-files-discovered-by-auto-loader
- 0 kudos
- 4144 Views
- 2 replies
- 0 kudos
Resolved! spark context in databricks
Hi @all,In Azure Databricks,I am using structured streaming for each batch functionality, in one of the functions I am creating tempview with pyspark dataframe (*Not GlobalTempView) and trying to access the same temp view by using spark.sql functiona...
- 4144 Views
- 2 replies
- 0 kudos
- 0 kudos
Do you face this issue without spark streaming as well? Also, could you share a minimal repo code preferably without streaming?
- 0 kudos
- 1649 Views
- 1 replies
- 0 kudos
Generate Excel for a SQL query
Greetings,I am using a Java Spring boot application that is supposed to respond with an excel based on request. My current approach involves reading data using jdbc drivers, storing them in appropriate data structures, writing them to an excel which ...
- 1649 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks for putting this together @Retired_mod ,I see that this approach will help to generate an excel after receiving the data from data bricks in the form of resultSet which has to be parsed.I believe this approach is the appropriate way to generat...
- 0 kudos
- 1540 Views
- 1 replies
- 0 kudos
Data masking best practices
Hi Team,Could you please suggest any best practices/blogs on implementing data masking, row level ,column level ,access control, role-based access control (RBAC), and attribute-based access control (ABAC)? Regards.Phanindra
- 1540 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, Can you check if this document answers your question: https://www.databricks.com/blog/2020/11/20/enforcing-column-level-encryption-and-avoiding-data-duplication-with-pii.html
- 0 kudos
- 978 Views
- 0 replies
- 1 kudos
feasibility of using user groups
Hi Team,Can you provide me the details of feasibility of using user groups for granting access to both Personally Identifiable Information (PII) and Non-PII in SQL PoolsRegards,Phanindra
- 978 Views
- 0 replies
- 1 kudos
- 4929 Views
- 2 replies
- 1 kudos
DLT Primary Key Deduplication: Expectations vs. Constraints vs. Other?
I'm trying to figure out what's the best way to "de-duplicate" data via DLT. Currently, my only leads are:Manage data quality with Delta Live Tables | Databricks on AWSVia "Drop invalid records"Constraints on Databricks | Databricks on AWSVia "pre-de...
- 4929 Views
- 2 replies
- 1 kudos
- 1 kudos
Hey @ChristianRRL ,Based on my understanding you want to de-duplicate your data during your DLT pipeline processing unfortunately I was not able to find a solution to this when I ran into this problem due to the native feature limitations.Limitations...
- 1 kudos
- 18931 Views
- 2 replies
- 1 kudos
ADF vs Databricks
Hi Team ,I would appreciate your suggestion on which scenario to choose between ADF (Azure Data Factory) and Databricks for orchestration, as well as any significant differences between them.Regards,Phanindra
- 18931 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi, I work with both, so it depends on the usecase.ADF is easy to set up and good for data integration, e.g. "copy data" job to transfer files from storage 1 to storage 2ADF data flows (data transformations) can be used to some level, but when the tr...
- 1 kudos
- 2584 Views
- 4 replies
- 0 kudos
DLT Performance question with Unity Catalog
Dear Community MembersThis question is about debugging performance issue of DLT pipeline with unity catalog.I had a DLT pipeline in Azure Databricks running on local store i.g. hive_metastore. And the processes took about 2 hour with the auto scalain...
- 2584 Views
- 4 replies
- 0 kudos
- 0 kudos
Hey Harvey, I getting around the same performance problems as you:From around 25 minutes in a normal workspace to an 1 hour and 20mins in UC workspace. Which is roughly 3x slower.Did you manage to solve this? I've also noticed dbutil.fs.ls() is much ...
- 0 kudos
