- 235 Views
- 1 replies
- 0 kudos
Context:job running using a job clustered configured in Standard access mode ( Shared Access mode )scala 2.12.15 / spark 3.5.0 jar programDatabricks runtime 15.4 LTSIn this context, it is not possible to get the sparkSession.sparkContext, as confirme...
- 235 Views
- 1 replies
- 0 kudos
Latest Reply
In Unity Catalog standard access mode (formerly shared access mode) with Databricks Runtime 15.4 LTS, direct access to `sparkSession.sparkContext` is restricted as part of the security limitations. However, there are still ways to access the Hadoop c...
by
pg289
• New Contributor II
- 884 Views
- 1 replies
- 0 kudos
I manage a large data lake of Iceberg tables stored on premise in S3 storage from MinIO. I need a Spark cluster to run ETL jobs. I decided to try Databricks as there were no other good options. However, I'm unable to properly access my tables or even...
- 884 Views
- 1 replies
- 0 kudos
Latest Reply
Not sure, but Databricks may default to AWS-style paths if the configurations are incomplete. Try setting the MinIO endpoint by configuring spark.hadoop.fs.s3a.endpoint to your MinIO server's URL. If MinIO uses HTTP, disable SSL by setting spark.hado...
- 1047 Views
- 4 replies
- 0 kudos
Hello. We are using Azure Databricks and would like to ingest data from a specific M365 SharePoint Online Site/List. I was originally trying to use this recommendation, https://learn.microsoft.com/en-us/answers/questions/2116616/service-principal-a...
- 1047 Views
- 4 replies
- 0 kudos
Latest Reply
Maybe, but at this point, we just need a method to get data from the specific SharePoint Online site reliability to pull into Databricks. So open to the easiest, most efficient method. Thanks.
3 More Replies
- 550 Views
- 0 replies
- 0 kudos
Hi Everyone,I am currently facing an issue with in our Test Environment where Data bricks is not able to mount with the storage account and we are using the same mount in other environments those are Dev,Preprod and Prod and it works fine there witho...
- 550 Views
- 0 replies
- 0 kudos
by
Malthe
• New Contributor II
- 986 Views
- 2 replies
- 0 kudos
In the documentation, most examples use the CREATE OR REFRESH STREAMING TABLE command.Meanwhile, from a role segregation perspective, create and refresh operations should happen in a separate context. That is, we want to create these objects (which e...
- 986 Views
- 2 replies
- 0 kudos
Latest Reply
Renu_
New Contributor III
Hi @Malthe, refreshing is automatically handled during pipeline runs in here. To implement effective role segregation, you should define separate DLT pipelines for deployment and execution, each with its own set of roles and permissions. This approac...
1 More Replies
by
Krthk
• New Contributor
- 314 Views
- 1 replies
- 1 kudos
Hi, I have a py notebook that I want to execute in an automated manner. One way I found this was to attach this to a job/task and hit it using the api from my local. However this seems to be adding significant overhead, my code even if it’s just one ...
- 314 Views
- 1 replies
- 1 kudos
Latest Reply
Hey @Krthk If you want to orchestrate a notebook, the easiest way is to go to File > Schedule directly from the notebook. My recommendation is to use cron syntax to define when it should run, and attach it to a predefined cluster or configure a new j...
by
phguk
• New Contributor III
- 30616 Views
- 5 replies
- 3 kudos
I am trying to configure access to Azure Storage Account (ADLS2) using OAUTH. The doc here gives an example of how to specify a secret in a cluster's spark configuration{{secrets/<secret-scope>/<service-credential-key>}}I can see how this works for ...
- 30616 Views
- 5 replies
- 3 kudos
Latest Reply
New doc link : https://learn.microsoft.com/en-us/azure/databricks/security/secrets/
4 More Replies
- 714 Views
- 2 replies
- 1 kudos
- 714 Views
- 2 replies
- 1 kudos
Latest Reply
Hi @Nexusss7
Out of curiosity, I tried to retrieve the resolved task parameter values. Finding a way to retrieve executed sub-tasks by the for_each task using APIs was challenging. So, I devised a solution using API and system tables.
I simplified t...
1 More Replies
- 507 Views
- 1 replies
- 0 kudos
Hi, Please help me configure/choose the cluster configuration. I need to process and merge 6 million records into Azure SQL DB. At the end of the week, 9 billion records need to be processed and merged into Azure SQL DB, and a few transformations nee...
- 507 Views
- 1 replies
- 0 kudos
Latest Reply
@Pu_123 Option 1 Daily Load (6M Records) - Cost-OptimizedCluster Mode: Single NodeVM Type: Standard_DS4_v2 or Standard_E4ds_v5Workers: 1Driver Node: Same as workerDatabricks Runtime: 13.x LTS (Photon Optional)Terminate after: 10-15 mins of inactivit...
- 1344 Views
- 2 replies
- 0 kudos
Databricks Apps is a new way to build and deploy internal data and AI applications is now available in Public Preview.Databricks Apps let developers build native apps using frameworks like Dash, Shiny and Streamlit, enabling data applications for non...
- 1344 Views
- 2 replies
- 0 kudos
Latest Reply
Which regions is it currently available in during public preview? @Sourav-Kundu How long is the public preview and what are next steps?
1 More Replies
- 277 Views
- 1 replies
- 0 kudos
Hi all,Looking for an honest review for anyone has had experience with the Databricks labs. Would it be more beneficial to learn without the labs and setup own infrastructure?Any advice would be greatly appreciated, newbie over here Thanks,Stringer
- 277 Views
- 1 replies
- 0 kudos
Latest Reply
Hello @Stringer!
From my experience, Databricks Labs makes learning easier by handling the setup and eliminating cloud costs. This is perfect if you’re just starting out or want to focus purely on Databricks. But since it abstracts things like networ...
- 810 Views
- 3 replies
- 0 kudos
Hi,This is the context of my issue:I have an AWS RDS Postgres database instance setup. I have also set up a Postgres CONNECTION in Databricks and can view the Postgres tables under a newly created FOREIGN CATALOG in Databricks Unity Catalog.Using the...
- 810 Views
- 3 replies
- 0 kudos
Latest Reply
Hi @pankj0510,
DML for tables is blocked from Databricks SQL, you can only read from DBSQL. I think you can set up a JDBC URL to the Postgres database and use Spark/Pandas DataFrame write methods to insert data
2 More Replies
- 420 Views
- 1 replies
- 0 kudos
Hi All,How can I align the text inside the Dashboard markdown to the middle?Is there an option to do this?Thanks,Gal
- 420 Views
- 1 replies
- 0 kudos
Latest Reply
Hello @Gal_Sb!
Databricks markdown does not support text alignment, and HTML/CSS do not work for this purpose in Databricks dashboards. You can try formatting options like headers or spacing adjustments. I'll also check with the team to explore possi...
- 518 Views
- 0 replies
- 0 kudos
Hi there, I would like to get some feedback on what are the ideal/suggested ways to get UMF data from our Azure cloud into Databricks. For context, UMF can mean either:User Managed FileUser Maintained FileBasically, a UMF could be something like a si...
- 518 Views
- 0 replies
- 0 kudos
by
T0M
• New Contributor III
- 765 Views
- 3 replies
- 1 kudos
Hi all!I've started learning DLT-Pipelines but I am struggling with the development of a pipeline.As far as I understand it, once I click on “Validate” a cluster will spin-up and stay (by default for 2hours), if the pipeline is in “Development” mode....
- 765 Views
- 3 replies
- 1 kudos
Latest Reply
Well, turns out if I do not make any changes to the cluster settings when creating a new pipeline (i.e. keep default) it works as expected (every new "validate" skips the "waiting for resources"-step).Initially, I reduced the number of workers to a m...
2 More Replies
