- 12636 Views
- 3 replies
- 1 kudos
Databricks connect and spark session best practice
Hi all!I am using databricks-connect to develop and test pyspark code pure python (not notebook) files in my local IDE, running on a Databricks cluster. These files are part of a deployment setup with dbx so that they are run as tasks in a workflow.E...
- 12636 Views
- 3 replies
- 1 kudos
- 1 kudos
I personally do something similar by checking an environment variable, the example was for a notebook but should work for a python file as well: import os if not os.environ.get("DATABRICKS_RUNTIME_VERSION"): from databricks.connect import Databr...
- 1 kudos
- 476 Views
- 0 replies
- 0 kudos
Overview of Dario Schiraldi Deutsche Bank Executive
Hello Community, Dario Schiraldi, an executive at Deutsche Bank, is widely recognized for his strategic vision and in-depth expertise in global financial systems. He expert at optimizing financial services and guiding the bank’s international strateg...
- 476 Views
- 0 replies
- 0 kudos
- 2095 Views
- 1 replies
- 0 kudos
Databricks workspace in failed state
Hi Community,I had my Databricks workspace up and running and it was managed through terraform, and encryption was enabled through cmk, there were some updation in the code, so I put terraform plan, one of the key changes(replace) it showed me was"az...
- 2095 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @sumitkuma49 ,Did you try what the error is suggesting? You could try re-creating the workspace. Once the workspace is created, set up the necessary Key Vault access policies or role assignments. After confirming the permissions are in place, upda...
- 0 kudos
- 977 Views
- 1 replies
- 1 kudos
Resolved! Accessing views using unitycatalog module
Hi, I'm trying to access to views in my catalog in databricks using unitycatalog open source modulewhen I try to do so I get an error message that indicates this is not possible:cannot be accessed from outside of Databricks Compute Environment due to...
- 977 Views
- 1 replies
- 1 kudos
- 1 kudos
Here are helpful tips/tricks: Based on the latest Databricks documentation and internal guides, it is currently not possible to grant external access (via open source Unity Catalog APIs or credential vending) to Unity Catalog views (i.e., objects w...
- 1 kudos
- 7534 Views
- 4 replies
- 0 kudos
Unable to create Iceberg tables pointing to data in S3 and run queries against the tables.
I need to to set up Iceberg tables in Databricks environment, but the data resides in an S3 bucket. Then read these tables by running SQL queries.Databricks environment has access to S3. This is done bysetting up the access by mapping the Instance Pr...
- 7534 Views
- 4 replies
- 0 kudos
- 0 kudos
@JohnsonBDSouza , @PujithaKarnati , @Venkat5 ,There are 3 concepts to use Iceberg format in databricks based on recent updates in DAIS 2025.1) Managed Iceberg tables 2) Foreign Iceberg tables 3) Enabling Iceberg reads on delta tables. Please refer be...
- 0 kudos
- 3281 Views
- 1 replies
- 0 kudos
How to search on empty string on text filter with Lakeview Dashboards
Hi,I have created a lakeview dashboard with a couple of filters and a table. Now I would like to search if a certain filter (column) has an empty string but if I search for ' ' then it goes 'no data'. I am wondering how can I search for an empty stri...
- 3281 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi! Could you get a resolution for this problem ?
- 0 kudos
- 19870 Views
- 6 replies
- 1 kudos
Installed Library / Module not found through Databricks connect LST 12.2
Hi all,We recently upgraded our databricks compute cluster from runtime version 10.4 LST, to 12.2 LST.After the upgrade one of our python scripts suddenly fails with a module not found error; indicating that our customly created module "xml_parser" i...
- 19870 Views
- 6 replies
- 1 kudos
- 1 kudos
Hi @maartenvr , hi @Debayan ,Are there any updates on this? Have you found a solution, or can the problem at least be narrowed down to specific DBR versions? I am on a cluster with 11.3 LTS and deploy my custom packaged code (named simply 'src') as P...
- 1 kudos
- 3308 Views
- 3 replies
- 0 kudos
Databricks grant update calatog catlog_name --json @privileges.json not updating privileges
Hi Team, I am trying to update the catalog permission privileges using databricks cli command Grant by appending json file but which is not updating the prIviliges, please help on grant update command usage.Command using : databricks grants update c...
- 3308 Views
- 3 replies
- 0 kudos
- 0 kudos
If someone needs this in the future, like I did.The issue is with your JSON structure. The Databricks CLI uses "changes" with "add" instead of "privilege_assignments" with "privileges".{ "changes": [ { "principal": "mailid", "add": ...
- 0 kudos
- 843 Views
- 1 replies
- 0 kudos
Logging: Unable to read a /volume based file
Hi We've just started using databricks and so am a little naive into the file system, especially regarding unity catalog.The issue is that we're creating a loggeer and wanting to write the files based on a queue handler/listener pattern. The patternn...
- 843 Views
- 1 replies
- 0 kudos
- 0 kudos
When using the CLI you need to add the scheme:dbfs:/Volumes/​...The rest should be fine to refer with "/Volumes/...", for more info Manage files in volumes | Databricks Documentation.Hope this solves the issue!
- 0 kudos
- 1231 Views
- 1 replies
- 0 kudos
Resolved! Workspace Consolidation Strategy in Databricks
Hi Team,The customer is facing a challenge related to increasing Databricks workspace maintenance costs. Apparently, every project is creating its own workspace for specific functionalities, and this has become a standard practice. As a result, the n...
- 1231 Views
- 1 replies
- 0 kudos
- 0 kudos
This is something that you should discuss with your Databricks rep imo. Even with standard tools, migrating consolidating 200 workspaces is something that needs very careful planning and testing.
- 0 kudos
- 1108 Views
- 0 replies
- 0 kudos
Introduction Dario Schiraldi Deutsche Bank Executive
Dario Schiraldi Deutsche Bank Executive, known for his strong leadership in the financial and banking sector. Dario Schiraldi brings 20 years of leadership experience to major worldwide organizations where his expertise extends into both market acqui...
- 1108 Views
- 0 replies
- 0 kudos
- 1226 Views
- 0 replies
- 0 kudos
SAS TO DATABRICKS MIGRATION
SAS to PY is an AI/ML-based Accelerator designed for "SAS to Python or PySpark" code migration. This Accelerator is engineered to convert SAS legacy proprietary codes to the more flexible, open-source Python or PySpark environment with 95% automatica...
- 1226 Views
- 0 replies
- 0 kudos
- 701 Views
- 1 replies
- 0 kudos
Dario Schiraldi : How do I integrate Databricks with AWS?
Hi everyone,I am Dario Schiraldi, CEO of Travel Works, and I am reaching out to the community for some insights. We are in the process of integrating Databricks with AWS for a new project, and I have love to hear from anyone who has experience with t...
- 701 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello Dario Good to meet you. You can connect with your account manager of databricks. Also Azure provides first partner assistance to databricks. you can check Azure services as well. Thank you.
- 0 kudos
- 5241 Views
- 4 replies
- 0 kudos
Resolved! vscode python project for development
Hi,I'm trying to set up a local development environment using python / vscode / poetry. Also, linting is enabled (Microsoft pylance extension) and the python.analysis.typeCheckingMode is set to strict.We are using python files for our code (.py) whit...
- 5241 Views
- 4 replies
- 0 kudos
- 0 kudos
How did you solve the type error checks on `pyspark.sql ` ? mypy doesn't create the missing stubs for that one?
- 0 kudos
- 1923 Views
- 1 replies
- 1 kudos
Resolved! How to trigger Power BI refresh from Databricks pipeline without keeping cluster alive?
I have a Databricks pipeline that pulls data from AWS, which takes ~90 minutes. After this, I need to refresh a series of Power BI dataflows (~45 mins) and then datasets (~45 mins).I want to trigger the Power BI refresh automatically from Databricks ...
- 1923 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @chandataeng ,The current Power BI task that is available in databricks workflow will wait for refresh process to return correct status (whether it succeeded or failed).But you can start refresh process by using asynchronous REST API call. The ref...
- 1 kudos
