Dario Schiraldi Deutsche Bank Executive, known for his strong leadership in the financial and banking sector. Dario Schiraldi brings 20 years of leadership experience to major worldwide organizations where his expertise extends into both market acqui...
SAS to PY is an AI/ML-based Accelerator designed for "SAS to Python or PySpark" code migration. This Accelerator is engineered to convert SAS legacy proprietary codes to the more flexible, open-source Python or PySpark environment with 95% automatica...
Hi everyone,I am Dario Schiraldi, CEO of Travel Works, and I am reaching out to the community for some insights. We are in the process of integrating Databricks with AWS for a new project, and I have love to hear from anyone who has experience with t...
Hello Dario Good to meet you. You can connect with your account manager of databricks. Also Azure provides first partner assistance to databricks. you can check Azure services as well. Thank you.
Hi,I'm trying to set up a local development environment using python / vscode / poetry. Also, linting is enabled (Microsoft pylance extension) and the python.analysis.typeCheckingMode is set to strict.We are using python files for our code (.py) whit...
How did you solve the type error checks on `pyspark.sql ` ? mypy doesn't create the missing stubs for that one?
I have a Databricks pipeline that pulls data from AWS, which takes ~90 minutes. After this, I need to refresh a series of Power BI dataflows (~45 mins) and then datasets (~45 mins).I want to trigger the Power BI refresh automatically from Databricks ...
Hi @chandataeng ,The current Power BI task that is available in databricks workflow will wait for refresh process to return correct status (whether it succeeded or failed).But you can start refresh process by using asynchronous REST API call. The ref...
We started migrating databases from hive_metastore into unity catalog back in October 2024 and ive noticed that periodically the Catalog UI will not show columns or a data preview for some tables, but not all of them that were migrated. After some di...
this is definitely a bug related to older instances of azure databricks that were upgraded to use unity platform. after going back and forth with MS support for 2+ months, we made the decision to just spin up a new instance of azure databricks and co...
I was recently given a credential file to access shared data via delta sharing. I am following the documentation from https://docs.databricks.com/en/data-sharing/read-data-open.html. The documentation wants the contents of the credential file in a fo...
Hi, the most feasible way would be to convert the contents of your key file into base64 and only mention the spark config as below: credentials <base 64 encoded code>
Hi allI have a noob question, I am currently using the Databricks free edition, which runs on serverless compute.To access the Spark UI normally one would click on the attached compute, however, with serverless, I can not find the menu to access Spar...
So, there is no way we can run spark in free edition as we need general purpose clusters?
Hi All, I am new to the Databricks, am using community version. So far, I have noticed some limitations , features like DBFS (File System) are restricted and Cluster Configuration is Locked. So I am thinking to use trial version, it will give 14 da...
Translator Hello However, the DBFS file browser is often disabled by default in the user interface. It can typically be re-enabled through the admin settings. In the free edition, you would face some limitations with cluster size. However, if you...
Hello. We are using Azure Databricks and would like to ingest data from a specific M365 SharePoint Online Site/List. I was originally trying to use this recommendation, https://learn.microsoft.com/en-us/answers/questions/2116616/service-principal-a...
We achieved the same using the SharePoint API. You can follow the steps outlined in this documentation: https://learn.microsoft.com/en-us/graph/auth-v2-service?tabs=http.Additionally, you can grant the Sites.Selected permission to the Azure AD applic...
I’m Dario Schiraldi Deutsche Bank Executive. During my time there, I led global institutional sales and investment businesses, honing my expertise in strategy, leadership, and financial markets. As someone who’s passionate about the transformative p...
Hi @ds01 ,Welcome, Dario! It’s great to have someone with your deep experience in finance and leadership join the Databricks community. Looking forward to your insights and contributions!
I found some hidden features in Databricks Notebooks, but sometimes when copying, Titles are lost. Maybe someone knows the reason?
When you copy cells in Databricks, titles (headers) can be lost because:TOC (Table of Contents) comes from Markdown titles like # Level 1.If you copy only the content and not the Markdown format, it won’t show in TOC.Hidden cells under the title migh...
I have all the prerequisites met as per the documentation but I don't have the option to create a serverless warehouse, which looks like is mandatory for delta share.https://learn.microsoft.com/en-us/azure/databricks/admin/sql/serverless- The workspa...
Magnus, if this satisfies your query please feel free to "Accept as Solution" so that others will benefit should they be in the same situation. Cheers, Lou.
Hi,I am trying to install mosaic on my cluster, but get the error once I use 'enable_mosaic': ImportError: cannot import name '_to_java_column' from 'pyspark.sql.functions' (/databricks/spark/python/pyspark/sql/functions/__init__.py) File <command-14...
Hello all, do we know when databricks-mosaic will get an update support the latest spark/scala runtime? Thanks
Hello all,everytime I try to log into Databricks Free Edition, I have to sign in with my email and password and then verification code via my email account.It gets really tiresome after a while.Is there a way for Databricks to remember my account log...
