- 22112 Views
- 6 replies
- 1 kudos
Installed Library / Module not found through Databricks connect LST 12.2
Hi all,We recently upgraded our databricks compute cluster from runtime version 10.4 LST, to 12.2 LST.After the upgrade one of our python scripts suddenly fails with a module not found error; indicating that our customly created module "xml_parser" i...
- 22112 Views
- 6 replies
- 1 kudos
- 1 kudos
Hi @maartenvr , hi @Debayan ,Are there any updates on this? Have you found a solution, or can the problem at least be narrowed down to specific DBR versions? I am on a cluster with 11.3 LTS and deploy my custom packaged code (named simply 'src') as P...
- 1 kudos
- 5019 Views
- 3 replies
- 0 kudos
Databricks grant update calatog catlog_name --json @privileges.json not updating privileges
Hi Team, I am trying to update the catalog permission privileges using databricks cli command Grant by appending json file but which is not updating the prIviliges, please help on grant update command usage.Command using : databricks grants update c...
- 5019 Views
- 3 replies
- 0 kudos
- 0 kudos
If someone needs this in the future, like I did.The issue is with your JSON structure. The Databricks CLI uses "changes" with "add" instead of "privilege_assignments" with "privileges".{ "changes": [ { "principal": "mailid", "add": ...
- 0 kudos
- 1405 Views
- 1 replies
- 0 kudos
Logging: Unable to read a /volume based file
Hi We've just started using databricks and so am a little naive into the file system, especially regarding unity catalog.The issue is that we're creating a loggeer and wanting to write the files based on a queue handler/listener pattern. The patternn...
- 1405 Views
- 1 replies
- 0 kudos
- 0 kudos
When using the CLI you need to add the scheme:dbfs:/Volumes/​...The rest should be fine to refer with "/Volumes/...", for more info Manage files in volumes | Databricks Documentation.Hope this solves the issue!
- 0 kudos
- 1779 Views
- 1 replies
- 0 kudos
Resolved! Workspace Consolidation Strategy in Databricks
Hi Team,The customer is facing a challenge related to increasing Databricks workspace maintenance costs. Apparently, every project is creating its own workspace for specific functionalities, and this has become a standard practice. As a result, the n...
- 1779 Views
- 1 replies
- 0 kudos
- 0 kudos
This is something that you should discuss with your Databricks rep imo. Even with standard tools, migrating consolidating 200 workspaces is something that needs very careful planning and testing.
- 0 kudos
- 1537 Views
- 0 replies
- 0 kudos
Introduction Dario Schiraldi Deutsche Bank Executive
Dario Schiraldi Deutsche Bank Executive, known for his strong leadership in the financial and banking sector. Dario Schiraldi brings 20 years of leadership experience to major worldwide organizations where his expertise extends into both market acqui...
- 1537 Views
- 0 replies
- 0 kudos
- 2775 Views
- 0 replies
- 1 kudos
SAS TO DATABRICKS MIGRATION
SAS to PY is an AI/ML-based Accelerator designed for "SAS to Python or PySpark" code migration. This Accelerator is engineered to convert SAS legacy proprietary codes to the more flexible, open-source Python or PySpark environment with 95% automatica...
- 2775 Views
- 0 replies
- 1 kudos
- 1137 Views
- 1 replies
- 0 kudos
Dario Schiraldi : How do I integrate Databricks with AWS?
Hi everyone,I am Dario Schiraldi, CEO of Travel Works, and I am reaching out to the community for some insights. We are in the process of integrating Databricks with AWS for a new project, and I have love to hear from anyone who has experience with t...
- 1137 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello Dario Good to meet you. You can connect with your account manager of databricks. Also Azure provides first partner assistance to databricks. you can check Azure services as well. Thank you.
- 0 kudos
- 6930 Views
- 4 replies
- 0 kudos
Resolved! vscode python project for development
Hi,I'm trying to set up a local development environment using python / vscode / poetry. Also, linting is enabled (Microsoft pylance extension) and the python.analysis.typeCheckingMode is set to strict.We are using python files for our code (.py) whit...
- 6930 Views
- 4 replies
- 0 kudos
- 0 kudos
How did you solve the type error checks on `pyspark.sql ` ? mypy doesn't create the missing stubs for that one?
- 0 kudos
- 4408 Views
- 1 replies
- 1 kudos
Resolved! How to trigger Power BI refresh from Databricks pipeline without keeping cluster alive?
I have a Databricks pipeline that pulls data from AWS, which takes ~90 minutes. After this, I need to refresh a series of Power BI dataflows (~45 mins) and then datasets (~45 mins).I want to trigger the Power BI refresh automatically from Databricks ...
- 4408 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @chandataeng ,The current Power BI task that is available in databricks workflow will wait for refresh process to return correct status (whether it succeeded or failed).But you can start refresh process by using asynchronous REST API call. The ref...
- 1 kudos
- 7417 Views
- 2 replies
- 0 kudos
Resolved! information_schema not populating with columns
We started migrating databases from hive_metastore into unity catalog back in October 2024 and ive noticed that periodically the Catalog UI will not show columns or a data preview for some tables, but not all of them that were migrated. After some di...
- 7417 Views
- 2 replies
- 0 kudos
- 0 kudos
this is definitely a bug related to older instances of azure databricks that were upgraded to use unity platform. after going back and forth with MS support for 2+ months, we made the decision to just spin up a new instance of azure databricks and co...
- 0 kudos
- 10094 Views
- 8 replies
- 0 kudos
Delta Sharing - Alternative to config.share
I was recently given a credential file to access shared data via delta sharing. I am following the documentation from https://docs.databricks.com/en/data-sharing/read-data-open.html. The documentation wants the contents of the credential file in a fo...
- 10094 Views
- 8 replies
- 0 kudos
- 0 kudos
Hi, the most feasible way would be to convert the contents of your key file into base64 and only mention the spark config as below: credentials <base 64 encoded code>
- 0 kudos
- 5738 Views
- 3 replies
- 2 kudos
Resolved! is Spark UI available on the Databricks Free Edition?
Hi allI have a noob question, I am currently using the Databricks free edition, which runs on serverless compute.To access the Spark UI normally one would click on the attached compute, however, with serverless, I can not find the menu to access Spar...
- 5738 Views
- 3 replies
- 2 kudos
- 2 kudos
So, there is no way we can run spark in free edition as we need general purpose clusters?
- 2 kudos
- 4165 Views
- 4 replies
- 0 kudos
Databricks Features
Hi All, I am new to the Databricks, am using community version. So far, I have noticed some limitations , features like DBFS (File System) are restricted and Cluster Configuration is Locked. So I am thinking to use trial version, it will give 14 da...
- 4165 Views
- 4 replies
- 0 kudos
- 0 kudos
Translator Hello However, the DBFS file browser is often disabled by default in the user interface. It can typically be re-enabled through the admin settings. In the free edition, you would face some limitations with cluster size. However, if you...
- 0 kudos
- 9043 Views
- 7 replies
- 0 kudos
Capture data from a Specific SharePoint Site (List) in M365 into Azure DataBricks
Hello. We are using Azure Databricks and would like to ingest data from a specific M365 SharePoint Online Site/List. I was originally trying to use this recommendation, https://learn.microsoft.com/en-us/answers/questions/2116616/service-principal-a...
- 9043 Views
- 7 replies
- 0 kudos
- 0 kudos
We achieved the same using the SharePoint API. You can follow the steps outlined in this documentation: https://learn.microsoft.com/en-us/graph/auth-v2-service?tabs=http.Additionally, you can grant the Sites.Selected permission to the Azure AD applic...
- 0 kudos
- 1782 Views
- 2 replies
- 1 kudos
Dario Schiraldi Deutsche Bank Executive : Excited to Join
I’m Dario Schiraldi Deutsche Bank Executive. During my time there, I led global institutional sales and investment businesses, honing my expertise in strategy, leadership, and financial markets. As someone who’s passionate about the transformative p...
- 1782 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @ds01 ,Welcome, Dario! It’s great to have someone with your deep experience in finance and leadership join the Databricks community. Looking forward to your insights and contributions!
- 1 kudos
