cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

baatchus
by New Contributor III
  • 3789 Views
  • 6 replies
  • 7 kudos

Resolved! Architecture choice, streaming data

I have sensor data coming into Azure Event Hub and need some help in deciding how to best ingest it into the Data Lake and Delta Lake:Option 1:azure event hub > databricks structured streaming > delta lake (bronze)Option 2:azure event hub > event hu...

  • 3789 Views
  • 6 replies
  • 7 kudos
Latest Reply
Kaniz
Community Manager
  • 7 kudos

Hi @baatch us​ , Just a friendly follow-up. Do you still need help, or does the above response help you to find the solution? Please let us know.

  • 7 kudos
5 More Replies
Dunken
by New Contributor III
  • 2064 Views
  • 5 replies
  • 4 kudos

Resolved! Databricks and DDD

Our architecture is according to Domain Driven Design. The data is therefore distributed among different domains.We would like to run workloads on top of our data but we would like to avoid to have a dedicated (duplicated) data lake just for Databric...

  • 2064 Views
  • 5 replies
  • 4 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 4 kudos

So basically you do not want to persist data outside of your source systems.I think the so called 'Kappa architecture' could be a fit, where everything is treated like a stream.Hubert already mentioned Kafka, which is an excellent source to build thi...

  • 4 kudos
4 More Replies
DataDNA
by New Contributor III
  • 6547 Views
  • 9 replies
  • 0 kudos

Resolved! AWS Databricks Pay-as-you-Go

Hi Team,We are currently using AWS cloud service for our AI/ML Project. We wanted to use Databricks based GPU service for model training purposes. We do not find a Pay-as-you-go option in marketplace for this purposes.Can someone help me on this !Tha...

  • 6547 Views
  • 9 replies
  • 0 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

https://databricks.com/product/aws-pricingI guess you try to roll out something else?

  • 0 kudos
8 More Replies
William_Scardua
by Valued Contributor
  • 1452 Views
  • 4 replies
  • 5 kudos

Resolved! GDPR/LGPD Compliance

How do you work to compliance the GDPR/LGPD, in special to people data ?have any suggestion ?

  • 1452 Views
  • 4 replies
  • 5 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 5 kudos

there is identifying PII data and handling/storing PII data.Identifying can be done with Purview, Macie, other tools. Those are not free ofc, so if your env is pretty big it can be interesting. Otherwise, you could also do manual checks.For storin...

  • 5 kudos
3 More Replies
User16752244127
by Contributor
  • 1069 Views
  • 2 replies
  • 1 kudos
  • 1069 Views
  • 2 replies
  • 1 kudos
Latest Reply
Brian_Dirking
New Contributor II
  • 1 kudos

A great place to learn more about Databricks integrations with AWS services is https://www.databricks.com/aws There is information on this page regarding integrations with Glue, SageMaker, Redshift and others. Many of these pages also point to our bl...

  • 1 kudos
1 More Replies
Anonymous
by Not applicable
  • 5557 Views
  • 3 replies
  • 0 kudos
  • 5557 Views
  • 3 replies
  • 0 kudos
Latest Reply
BigRoux
New Contributor III
  • 0 kudos

Multiple writers (inserts/appends) present no problems with Delta. You can have two users appending data to a Delta table at the same time without issue. Updates, deletes, merges, and compaction can run into issues when multiple user are trying to d...

  • 0 kudos
2 More Replies
User16826987838
by Contributor
  • 861 Views
  • 1 replies
  • 0 kudos
  • 861 Views
  • 1 replies
  • 0 kudos
Latest Reply
Taha
New Contributor III
  • 0 kudos

Not as of this comment, but our product team is aware of this request and are working on getting this implemented soon!Keep in mind that for feature requests and direct updates/communication from our product team, you can post here (or just vote up o...

  • 0 kudos
User16790091296
by Contributor II
  • 651 Views
  • 1 replies
  • 0 kudos
  • 651 Views
  • 1 replies
  • 0 kudos
Latest Reply
Taha
New Contributor III
  • 0 kudos

The admin console exists within the workspace and let's you control access and privileges for that specific workspace. An existing admin can get to it from the drop down in the very top right and selecting Admin Console.The first screen you'll land o...

  • 0 kudos
MoJaMa
by Valued Contributor II
  • 587 Views
  • 1 replies
  • 0 kudos
  • 587 Views
  • 1 replies
  • 0 kudos
Latest Reply
MoJaMa
Valued Contributor II
  • 0 kudos

Anything that can reach the control plane and use the SCIM API should work. For Azure AD Premium, there is specifically an enterprise App that does this for the customer. 

  • 0 kudos
User16869510359
by Esteemed Contributor
  • 1281 Views
  • 1 replies
  • 1 kudos
  • 1281 Views
  • 1 replies
  • 1 kudos
Latest Reply
User16869510359
Esteemed Contributor
  • 1 kudos

Ganglia metrics are available only if the job runs for more than 15 minutes. For jobs that are completed within 15 minutes, the metrics won't be available

  • 1 kudos
User16765131552
by Contributor III
  • 638 Views
  • 1 replies
  • 0 kudos

Resolved! Databricks SQL dashboard refresh

In Databricks SQL, can you prohibit a dashboard from being refreshed?

  • 638 Views
  • 1 replies
  • 0 kudos
Latest Reply
User16765131552
Contributor III
  • 0 kudos

It looks like this can be done by not granting CAN_RUN to a user/grouphttps://docs.databricks.com/sql/user/security/access-control/dashboard-acl.html#dashboard-permissions

  • 0 kudos
Anonymous
by Not applicable
  • 494 Views
  • 1 replies
  • 0 kudos
  • 494 Views
  • 1 replies
  • 0 kudos
Latest Reply
User16826994223
Honored Contributor III
  • 0 kudos

One way to increase the CIDR range id the ip list is available or create a complete different work space on same vpc with different subnets

  • 0 kudos
Labels