cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

pg289
by New Contributor II
  • 3801 Views
  • 1 replies
  • 0 kudos

How to connect to an on-premise implementation of S3 storage (such as Minio) in Databricks Notebooks

I manage a large data lake of Iceberg tables stored on premise in S3 storage from MinIO. I need a Spark cluster to run ETL jobs. I decided to try Databricks as there were no other good options. However, I'm unable to properly access my tables or even...

  • 3801 Views
  • 1 replies
  • 0 kudos
Latest Reply
SP_6721
Contributor III
  • 0 kudos

Not sure, but Databricks may default to AWS-style paths if the configurations are incomplete. Try setting the MinIO endpoint by configuring spark.hadoop.fs.s3a.endpoint to your MinIO server's URL. If MinIO uses HTTP, disable SSL by setting spark.hado...

  • 0 kudos
Malthe
by Contributor
  • 3346 Views
  • 2 replies
  • 0 kudos

Create DLT pipeline in CI/CD with role segregation

In the documentation, most examples use the CREATE OR REFRESH STREAMING TABLE command.Meanwhile, from a role segregation perspective, create and refresh operations should happen in a separate context. That is, we want to create these objects (which e...

  • 3346 Views
  • 2 replies
  • 0 kudos
Latest Reply
Renu_
Valued Contributor II
  • 0 kudos

Hi @Malthe, refreshing is automatically handled during pipeline runs in here. To implement effective role segregation, you should define separate DLT pipelines for deployment and execution, each with its own set of roles and permissions. This approac...

  • 0 kudos
1 More Replies
Krthk
by New Contributor
  • 976 Views
  • 1 replies
  • 1 kudos

Resolved! Jobs overhead why ?

Hi, I have a py notebook that I want to execute in an automated manner. One way I found this was to attach this to a job/task and hit it using the api from my local. However this seems to be adding significant overhead, my code even if it’s just one ...

Get Started Discussions
API
automation
jobs
Jobs api spark
spark
  • 976 Views
  • 1 replies
  • 1 kudos
Latest Reply
Isi
Honored Contributor II
  • 1 kudos

Hey @Krthk If you want to orchestrate a notebook, the easiest way is to go to File > Schedule directly from the notebook. My recommendation is to use cron syntax to define when it should run, and attach it to a predefined cluster or configure a new j...

  • 1 kudos
phguk
by New Contributor III
  • 38876 Views
  • 5 replies
  • 3 kudos

Using Azure Key Vault secret to access Azure Storage

I am trying to configure access to Azure Storage Account (ADLS2) using OAUTH.  The doc here gives an example of how to specify a secret in a cluster's spark configuration{{secrets/<secret-scope>/<service-credential-key>}}I can see how this works for ...

  • 38876 Views
  • 5 replies
  • 3 kudos
Latest Reply
bot_axel
New Contributor II
  • 3 kudos

New doc link : https://learn.microsoft.com/en-us/azure/databricks/security/secrets/

  • 3 kudos
4 More Replies
Nexusss7
by New Contributor II
  • 1400 Views
  • 2 replies
  • 1 kudos

Resolved! Query: Extracting Resolved 'Input' Parameter from a Databricks Workflow Run

Hi Everyone,I have a query regarding extracting the resolved value of the 'Input' parameter (highlighted in yellow in the attached images) from a Databricks workflow run.The images show:The foreach task receives its input from the Metadata_Fetcher ta...

Nexusss7_0-1741764899129.png Nexusss7_1-1741764908789.png Nexusss7_3-1741764955126.png
  • 1400 Views
  • 2 replies
  • 1 kudos
Latest Reply
koji_kawamura
Databricks Employee
  • 1 kudos

Hi @Nexusss7  Out of curiosity, I tried to retrieve the resolved task parameter values. Finding a way to retrieve executed sub-tasks by the for_each task using APIs was challenging. So, I devised a solution using API and system tables. I simplified t...

  • 1 kudos
1 More Replies
Pu_123
by New Contributor
  • 1445 Views
  • 1 replies
  • 0 kudos

Cluster configuration

Hi, Please help me configure/choose the cluster configuration. I need to process and merge 6 million records into Azure SQL DB. At the end of the week, 9 billion records need to be processed and merged into Azure SQL DB, and a few transformations nee...

  • 1445 Views
  • 1 replies
  • 0 kudos
Latest Reply
Ajay-Pandey
Esteemed Contributor III
  • 0 kudos

@Pu_123  Option 1 Daily Load (6M Records) - Cost-OptimizedCluster Mode: Single NodeVM Type: Standard_DS4_v2 or Standard_E4ds_v5Workers: 1Driver Node: Same as workerDatabricks Runtime: 13.x LTS (Photon Optional)Terminate after: 10-15 mins of inactivit...

  • 0 kudos
Stringer
by New Contributor
  • 652 Views
  • 1 replies
  • 0 kudos

Databricks labs $200 or not

Hi all,Looking for an honest review for anyone has had experience with the Databricks labs. Would it be more beneficial to learn without the labs and setup own infrastructure?Any advice would be greatly appreciated, newbie over here Thanks,Stringer 

  • 652 Views
  • 1 replies
  • 0 kudos
Latest Reply
Advika
Databricks Employee
  • 0 kudos

Hello @Stringer! From my experience, Databricks Labs makes learning easier by handling the setup and eliminating cloud costs. This is perfect if you’re just starting out or want to focus purely on Databricks. But since it abstracts things like networ...

  • 0 kudos
pankj0510
by New Contributor II
  • 1764 Views
  • 3 replies
  • 0 kudos

Resolved! Error when executing an INSERT statement on an External Postgres table from Databricks SQL Editor

Hi,This is the context of my issue:I have an AWS RDS Postgres database instance setup. I have also set up a Postgres CONNECTION in Databricks and can view the Postgres tables under a newly created FOREIGN CATALOG in Databricks Unity Catalog.Using the...

Get Started Discussions
Connection with Postgres DB
External Table
Unity Catalog
  • 1764 Views
  • 3 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hi @pankj0510, DML for tables is blocked from Databricks SQL, you can only read from DBSQL. I think you can set up a JDBC URL to the Postgres database and use Spark/Pandas DataFrame write methods to insert data

  • 0 kudos
2 More Replies
Gal_Sb
by New Contributor
  • 1259 Views
  • 1 replies
  • 0 kudos

Text alignment in databricks dashboard markdown

Hi All,How can I align the text inside the Dashboard markdown to the middle?Is there an option to do this?Thanks,Gal

  • 1259 Views
  • 1 replies
  • 0 kudos
Latest Reply
Advika
Databricks Employee
  • 0 kudos

Hello @Gal_Sb! Databricks markdown does not support text alignment, and HTML/CSS do not work for this purpose in Databricks dashboards. You can try formatting options like headers or spacing adjustments. I'll also check with the team to explore possi...

  • 0 kudos
ChristianRRL
by Valued Contributor III
  • 2814 Views
  • 0 replies
  • 0 kudos

Databricks UMF Best Practice

Hi there, I would like to get some feedback on what are the ideal/suggested ways to get UMF data from our Azure cloud into Databricks. For context, UMF can mean either:User Managed FileUser Maintained FileBasically, a UMF could be something like a si...

Get Started Discussions
Data ingestion
UMF
User Maintained File
User Managed File
  • 2814 Views
  • 0 replies
  • 0 kudos
T0M
by New Contributor III
  • 1328 Views
  • 3 replies
  • 1 kudos

Resolved! DLT Pipeline Validate will always spawn new cluster

Hi all!I've started learning DLT-Pipelines but I am struggling with the development of a pipeline.As far as I understand it, once I click on “Validate” a cluster will spin-up and stay (by default for 2hours), if the pipeline is in “Development” mode....

  • 1328 Views
  • 3 replies
  • 1 kudos
Latest Reply
T0M
New Contributor III
  • 1 kudos

Well, turns out if I do not make any changes to the cluster settings when creating a new pipeline (i.e. keep default) it works as expected (every new "validate" skips the "waiting for resources"-step).Initially, I reduced the number of workers to a m...

  • 1 kudos
2 More Replies
surajitDE
by New Contributor III
  • 1187 Views
  • 4 replies
  • 0 kudos

DLT refresh time for combination of streaming and non streaming tables?

@dlt.tabledef joined_table():    dim_df = spark.read.table("dim_table")  # Reloads every batch    fact_df = spark.readStream.table("fact_stream")    return fact_df.join(dim_df, "id", "left")

  • 1187 Views
  • 4 replies
  • 0 kudos
Latest Reply
brycejune
New Contributor III
  • 0 kudos

Hi,Current approach reloads dim_df in every batch, which can be inefficient. To optimize, consider broadcasting dim_df if it's small or using a mapGroupsWithState function for stateful joins. Also, ensure that fact_df has sufficient watermarking to h...

  • 0 kudos
3 More Replies
dollyb
by Contributor II
  • 9409 Views
  • 2 replies
  • 0 kudos

How to detect if running in a workflow job?

Hi there,what's the best way to differentiate in what environment my Spark session is running? Locally I develop with databricks-connect's DatabricksSession, but that doesn't work when running a workflow job which requires SparkSession.getOrCreate()....

  • 9409 Views
  • 2 replies
  • 0 kudos
Latest Reply
Rob-Altmiller
Databricks Employee
  • 0 kudos

import json def get_job_context(): """Retrieve job-related context from the current Databricks notebook.""" # Retrieve the notebook context ctx = dbutils.notebook.entry_point.getDbutils().notebook().getContext() # Convert the context...

  • 0 kudos
1 More Replies
SB93
by New Contributor II
  • 854 Views
  • 1 replies
  • 0 kudos

Help Needed: Executor Lost Error in Multi-Node Distributed Training with PyTorch

Hi everyone,I'm currently working on distributed training of a PyTorch model, following the example provided here. The training runs perfectly on a single node with a single GPU. However, when I attempt multi-node training using the following configu...

  • 854 Views
  • 1 replies
  • 0 kudos
Latest Reply
cgrant
Databricks Employee
  • 0 kudos

We do not recommend using spot instances with distributed ML training workloads that use barrier mode, like TorchDistributor as these workloads are extremely sensitive to executor loss. Please disable spot/pre-emption and try again.

  • 0 kudos
manoj_2355ca
by New Contributor III
  • 4569 Views
  • 2 replies
  • 0 kudos

cannot create external location: invalid Databricks Workspace configuration

HI AllI am trying to create databricks storage credentials , external location and catalog with terraform.cloud : AzureMy storage credentials code is working correctly . But the external location code is throwing below error when executing the Terraf...

Get Started Discussions
azuredatabricks
  • 4569 Views
  • 2 replies
  • 0 kudos
Latest Reply
badari_narayan
New Contributor II
  • 0 kudos

Hi @manoj_2355ca , I am also facing the same error, did you get the solution for it?

  • 0 kudos
1 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels