Databricks Community

RicksDB · 06-16-2024

Hello,Is there any plan to support the equivalent of fabric semantic-link on Databricks? Essentially, having the ability to query PowerBi Dataset using SQL on interactive/jobs cluster and SQL warehouse (i.e a Unity Catalog federated source being Powe...

RicksDB · 09-03-2022

Hello,I started testing Unity catalog and was surprised by the following mention in the documentation about shared clusters not supporting third-party1 libraries. Nevertheless, I tried doing imports and using pip install and both were working correct...

RicksDB · 08-29-2022

What are the best practices when using DLT with a unity catalog aware workspace since it is not supported at the moment?Do you create a copy of every table you need in the hive metastore in order to reference them? Or is it recommended to wait until ...

RicksDB · 08-26-2022

Hello,Before Unity catalog, we were able to rename spark files to a single file (ex: accounts.csv instead of part001) using mounts with pandas or dbutils.fs. How can we achieve the same today without mounts? Is it possible to use external locations w...

RicksDB · 07-01-2022

Hi, what is the maximum number of jobs we can execute in an hour for a given workspace?This page mentions 5000https://docs.microsoft.com/en-us/azure/databricks/data-engineering/jobs/jobsThe number of jobs a workspace can create in an hour is limited ...

RicksDB · 02-02-2022

Thanks for the quick answer Hubert.We currently have workspace object access control enabled on our workspace. Although, we can use it to add ACL on workspaces ACL folders that store notebooks, we haven't seen anyway to secure DBFS root itself.Are th...

RicksDB · 01-31-2022

Indeed, there isn't much missing to meet our security requirements. In our case, the Hive is also our weak spot. Fortunately, interactive (credential passthrough) are contained in a different workspace so there is no impact on our "jobs" if somebody ...

RicksDB · 01-28-2022

We had the same issue and never could find any combination that satisfied our needs.We had to separate the use cases in 2 different workspaces as mention here:https://docs.microsoft.com/en-us/azure/cloud-adoption-framework/scenarios/data-management/b...

RicksDB · 01-04-2022

@Bilal Aslam In my case, it usually depends on the customers and their SLA. Most of them usually do not have a "true" high SLA requirement thus prefer the jobs to be throttled when the actual cost is within a certain range of the budget instead of ...

RicksDB · 01-03-2022

Hi @Bilal Aslam Serverless clusters would definitely help regarding the speed required for the small jobs of most of my clients.That being said, most of these clients requires calculating the "worst case" for most technologies when presenting a busi...

Databricks Community

User Stats

User Activity

Semantic-link on Databricks

Unity Catalog shared cluster & third-party1 libraries

DLT and Unity catalog best practices

Renaming spark files using External Locations

Maximum job execution per hour

Re: Restricting file upload to DBFS

Re: Overall security/access rights concept needed (combine Table Access Control and Credential Passthrough), how to allow users the benefits of both worlds

Re: Overall security/access rights concept needed (combine Table Access Control and Credential Passthrough), how to allow users the benefits of both worlds

Re: SingleNode all-purpose cluster for small ETLs

Re: SingleNode all-purpose cluster for small ETLs