- 1587 Views
- 1 replies
- 0 kudos
Generate Excel for a SQL query
Greetings,I am using a Java Spring boot application that is supposed to respond with an excel based on request. My current approach involves reading data using jdbc drivers, storing them in appropriate data structures, writing them to an excel which ...
- 1587 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks for putting this together @Retired_mod ,I see that this approach will help to generate an excel after receiving the data from data bricks in the form of resultSet which has to be parsed.I believe this approach is the appropriate way to generat...
- 0 kudos
- 1451 Views
- 1 replies
- 0 kudos
Data masking best practices
Hi Team,Could you please suggest any best practices/blogs on implementing data masking, row level ,column level ,access control, role-based access control (RBAC), and attribute-based access control (ABAC)? Regards.Phanindra
- 1451 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, Can you check if this document answers your question: https://www.databricks.com/blog/2020/11/20/enforcing-column-level-encryption-and-avoiding-data-duplication-with-pii.html
- 0 kudos
- 768 Views
- 0 replies
- 1 kudos
feasibility of using user groups
Hi Team,Can you provide me the details of feasibility of using user groups for granting access to both Personally Identifiable Information (PII) and Non-PII in SQL PoolsRegards,Phanindra
- 768 Views
- 0 replies
- 1 kudos
- 4339 Views
- 2 replies
- 1 kudos
DLT Primary Key Deduplication: Expectations vs. Constraints vs. Other?
I'm trying to figure out what's the best way to "de-duplicate" data via DLT. Currently, my only leads are:Manage data quality with Delta Live Tables | Databricks on AWSVia "Drop invalid records"Constraints on Databricks | Databricks on AWSVia "pre-de...
- 4339 Views
- 2 replies
- 1 kudos
- 1 kudos
Hey @ChristianRRL ,Based on my understanding you want to de-duplicate your data during your DLT pipeline processing unfortunately I was not able to find a solution to this when I ran into this problem due to the native feature limitations.Limitations...
- 1 kudos
- 17759 Views
- 2 replies
- 1 kudos
ADF vs Databricks
Hi Team ,I would appreciate your suggestion on which scenario to choose between ADF (Azure Data Factory) and Databricks for orchestration, as well as any significant differences between them.Regards,Phanindra
- 17759 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi, I work with both, so it depends on the usecase.ADF is easy to set up and good for data integration, e.g. "copy data" job to transfer files from storage 1 to storage 2ADF data flows (data transformations) can be used to some level, but when the tr...
- 1 kudos
- 2437 Views
- 4 replies
- 0 kudos
DLT Performance question with Unity Catalog
Dear Community MembersThis question is about debugging performance issue of DLT pipeline with unity catalog.I had a DLT pipeline in Azure Databricks running on local store i.g. hive_metastore. And the processes took about 2 hour with the auto scalain...
- 2437 Views
- 4 replies
- 0 kudos
- 0 kudos
Hey Harvey, I getting around the same performance problems as you:From around 25 minutes in a normal workspace to an 1 hour and 20mins in UC workspace. Which is roughly 3x slower.Did you manage to solve this? I've also noticed dbutil.fs.ls() is much ...
- 0 kudos
- 3791 Views
- 0 replies
- 0 kudos
Auto-Update API Data
Not sure if this has come up before, but I'm wondering if Databricks has any kind of functionality to "watch" an API call for changes?E.g. Currently I have a frequently running job that pulls data via an API call and overwrites the old data. This see...
- 3791 Views
- 0 replies
- 0 kudos
- 7269 Views
- 7 replies
- 4 kudos
System table with state UNAVAILABLE
Hello,When I check the system table's status, it seems that they are in UNAVAILABLE state. I would like to know if anyone have faced this issue ?Because of that, I can't enable the system table. {"schemas":[{"schema":"storage","state":"UNAVAILABLE"},...
- 7269 Views
- 7 replies
- 4 kudos
- 3630 Views
- 1 replies
- 0 kudos
Unable to connect Azure kafka server with public IP from databricks notebook
Hi Team,I am unable to connect (SSH connection) from Azure Databricks Notebook to Azure Kafka server.Kafka Server and Databricks both are under same resource group and region. Also in Inbound rule the port is added in Kafka server.Please help me to r...
- 3630 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, This looks like issue with networking config. Could you please check on the routing configs, firewall routes etc to make sure destination IP to 9092 is added in the Azure console?
- 0 kudos
- 1572 Views
- 1 replies
- 0 kudos
user email invitation to workspace not received
HelloSince December 2023, I cannot anymore invite users to connect to my workspace as I used to. For no reason, the users I add through my admin dashboard do not receive the invitation email and thus the link to connect to the workspace. I tried my...
- 1572 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, Could you also please try to add users through account console if Identity Federation is enabled? Refer: https://docs.databricks.com/en/administration-guide/users-groups/users.html#assign-a-user-to-a-workspace-using-the-account-console
- 0 kudos
- 12614 Views
- 11 replies
- 6 kudos
databricks-connect 13.1.0 limitations
Hi,Quite excited to see the new release of databricks-connect, I started writing unit tests running pyspark on a databricks cluster using databricks-connect.After some successful basic unit tests, I tested just more chained transformations on a dataf...
- 12614 Views
- 11 replies
- 6 kudos
- 6 kudos
I doubled the `spark.connect.grpc.maxInboundMessageSize` parameter to 256mb but that didn't appear to resolve anything.
- 6 kudos
- 965 Views
- 1 replies
- 0 kudos
Running sql command on Single User cluster vs Shared.
Hi, when i am running the below simple code over my Unity Catalog on a Shared cluster, it works very well.But on a Single User - i am getting : Failed to acquire a SAS token for list on /__unitystorage/schemas/1bb5b053-ac96-471b-8077-8288c56c8a20/tab...
- 965 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, Could you please refer to the limitations here: https://docs.databricks.com/en/compute/access-mode-limitations.html . Please let us know if this helps.
- 0 kudos
- 2436 Views
- 1 replies
- 0 kudos
how to access data in one databricks in another databricks
I want to acces data in another databricks in my databricks, how to do that
- 2436 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello, many thanks for your question, to be able to provide you with a more precise response we required some additional information:1. When you refer databricks in my databricks are you refering to access data that is in one workspace to another wor...
- 0 kudos
- 6100 Views
- 3 replies
- 2 kudos
Resolved! Move multiple notebooks at the same time (programmatically)
If I want to move multiple (hundreds of) notebooks at the same time from one folder to another, what is the best way to do that? Other than going to each individual notebook and clicking "Move".Is there a way to programmatically move notebooks? Like ...
- 6100 Views
- 3 replies
- 2 kudos
- 2 kudos
You should be redirected to the KB page, but this is the information contained: Problem How to migrate Shared folders and the notebooks Cause Shared notebooks are not migrated into new workspace by default Solution Please find the script to migrate t...
- 2 kudos
- 2772 Views
- 1 replies
- 2 kudos
Databricks API using the personal access token
We can access the Azure databricks API using the personal access token which is created by us manually.The objective is that client don’t want to store the personal access token which may not be secure .Do we have option to generate the token during ...
- 2772 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Phani1 ,Yes, now you can use databricks Create a user token API for create access token via automated API call.Please refer below doc - Create a user token | Token API | REST API reference | Azure Databricks
- 2 kudos
