- 471 Views
- 1 replies
- 1 kudos
Resolved! Jobs overhead why ?
Hi, I have a py notebook that I want to execute in an automated manner. One way I found this was to attach this to a job/task and hit it using the api from my local. However this seems to be adding significant overhead, my code even if it’s just one ...
- 471 Views
- 1 replies
- 1 kudos
- 1 kudos
Hey @Krthk If you want to orchestrate a notebook, the easiest way is to go to File > Schedule directly from the notebook. My recommendation is to use cron syntax to define when it should run, and attach it to a predefined cluster or configure a new j...
- 1 kudos
- 32253 Views
- 5 replies
- 3 kudos
Using Azure Key Vault secret to access Azure Storage
I am trying to configure access to Azure Storage Account (ADLS2) using OAUTH. The doc here gives an example of how to specify a secret in a cluster's spark configuration{{secrets/<secret-scope>/<service-credential-key>}}I can see how this works for ...
- 32253 Views
- 5 replies
- 3 kudos
- 3 kudos
New doc link : https://learn.microsoft.com/en-us/azure/databricks/security/secrets/
- 3 kudos
- 966 Views
- 2 replies
- 1 kudos
Resolved! Query: Extracting Resolved 'Input' Parameter from a Databricks Workflow Run
Hi Everyone,I have a query regarding extracting the resolved value of the 'Input' parameter (highlighted in yellow in the attached images) from a Databricks workflow run.The images show:The foreach task receives its input from the Metadata_Fetcher ta...
- 966 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Nexusss7 Out of curiosity, I tried to retrieve the resolved task parameter values. Finding a way to retrieve executed sub-tasks by the for_each task using APIs was challenging. So, I devised a solution using API and system tables. I simplified t...
- 1 kudos
- 778 Views
- 1 replies
- 0 kudos
Cluster configuration
Hi, Please help me configure/choose the cluster configuration. I need to process and merge 6 million records into Azure SQL DB. At the end of the week, 9 billion records need to be processed and merged into Azure SQL DB, and a few transformations nee...
- 778 Views
- 1 replies
- 0 kudos
- 0 kudos
@Pu_123 Option 1 Daily Load (6M Records) - Cost-OptimizedCluster Mode: Single NodeVM Type: Standard_DS4_v2 or Standard_E4ds_v5Workers: 1Driver Node: Same as workerDatabricks Runtime: 13.x LTS (Photon Optional)Terminate after: 10-15 mins of inactivit...
- 0 kudos
- 375 Views
- 1 replies
- 0 kudos
Databricks labs $200 or not
Hi all,Looking for an honest review for anyone has had experience with the Databricks labs. Would it be more beneficial to learn without the labs and setup own infrastructure?Any advice would be greatly appreciated, newbie over here Thanks,Stringer
- 375 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Stringer! From my experience, Databricks Labs makes learning easier by handling the setup and eliminating cloud costs. This is perfect if you’re just starting out or want to focus purely on Databricks. But since it abstracts things like networ...
- 0 kudos
- 1051 Views
- 3 replies
- 0 kudos
Resolved! Error when executing an INSERT statement on an External Postgres table from Databricks SQL Editor
Hi,This is the context of my issue:I have an AWS RDS Postgres database instance setup. I have also set up a Postgres CONNECTION in Databricks and can view the Postgres tables under a newly created FOREIGN CATALOG in Databricks Unity Catalog.Using the...
- 1051 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @pankj0510, DML for tables is blocked from Databricks SQL, you can only read from DBSQL. I think you can set up a JDBC URL to the Postgres database and use Spark/Pandas DataFrame write methods to insert data
- 0 kudos
- 666 Views
- 1 replies
- 0 kudos
Text alignment in databricks dashboard markdown
Hi All,How can I align the text inside the Dashboard markdown to the middle?Is there an option to do this?Thanks,Gal
- 666 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Gal_Sb! Databricks markdown does not support text alignment, and HTML/CSS do not work for this purpose in Databricks dashboards. You can try formatting options like headers or spacing adjustments. I'll also check with the team to explore possi...
- 0 kudos
- 865 Views
- 0 replies
- 0 kudos
Databricks UMF Best Practice
Hi there, I would like to get some feedback on what are the ideal/suggested ways to get UMF data from our Azure cloud into Databricks. For context, UMF can mean either:User Managed FileUser Maintained FileBasically, a UMF could be something like a si...
- 865 Views
- 0 replies
- 0 kudos
- 873 Views
- 3 replies
- 1 kudos
DLT Pipeline Validate will always spawn new cluster
Hi all!I've started learning DLT-Pipelines but I am struggling with the development of a pipeline.As far as I understand it, once I click on “Validate” a cluster will spin-up and stay (by default for 2hours), if the pipeline is in “Development” mode....
- 873 Views
- 3 replies
- 1 kudos
- 1 kudos
Well, turns out if I do not make any changes to the cluster settings when creating a new pipeline (i.e. keep default) it works as expected (every new "validate" skips the "waiting for resources"-step).Initially, I reduced the number of workers to a m...
- 1 kudos
- 552 Views
- 4 replies
- 0 kudos
DLT refresh time for combination of streaming and non streaming tables?
@dlt.tabledef joined_table(): dim_df = spark.read.table("dim_table") # Reloads every batch fact_df = spark.readStream.table("fact_stream") return fact_df.join(dim_df, "id", "left")
- 552 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi,Current approach reloads dim_df in every batch, which can be inefficient. To optimize, consider broadcasting dim_df if it's small or using a mapGroupsWithState function for stateful joins. Also, ensure that fact_df has sufficient watermarking to h...
- 0 kudos
- 8496 Views
- 2 replies
- 0 kudos
How to detect if running in a workflow job?
Hi there,what's the best way to differentiate in what environment my Spark session is running? Locally I develop with databricks-connect's DatabricksSession, but that doesn't work when running a workflow job which requires SparkSession.getOrCreate()....
- 8496 Views
- 2 replies
- 0 kudos
- 0 kudos
import json def get_job_context(): """Retrieve job-related context from the current Databricks notebook.""" # Retrieve the notebook context ctx = dbutils.notebook.entry_point.getDbutils().notebook().getContext() # Convert the context...
- 0 kudos
- 434 Views
- 1 replies
- 0 kudos
Help Needed: Executor Lost Error in Multi-Node Distributed Training with PyTorch
Hi everyone,I'm currently working on distributed training of a PyTorch model, following the example provided here. The training runs perfectly on a single node with a single GPU. However, when I attempt multi-node training using the following configu...
- 434 Views
- 1 replies
- 0 kudos
- 0 kudos
We do not recommend using spot instances with distributed ML training workloads that use barrier mode, like TorchDistributor as these workloads are extremely sensitive to executor loss. Please disable spot/pre-emption and try again.
- 0 kudos
- 3895 Views
- 2 replies
- 0 kudos
cannot create external location: invalid Databricks Workspace configuration
HI AllI am trying to create databricks storage credentials , external location and catalog with terraform.cloud : AzureMy storage credentials code is working correctly . But the external location code is throwing below error when executing the Terraf...
- 3895 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @manoj_2355ca , I am also facing the same error, did you get the solution for it?
- 0 kudos
- 4209 Views
- 5 replies
- 0 kudos
typing extensions import match error
I am trying to install the stanza library and try to create a udf function to create NER tags for my chunk_text in the dataframe.Cluster Config: DBR 14.3 LTS SPARK 3.5.0 SCALA 2.12below code:def extract_entities(text import stanza nlp = stanza....
- 4209 Views
- 5 replies
- 0 kudos
- 0 kudos
@SaadhikaB Hi, when I run dbutils.library.restartPython(), I get the following error
- 0 kudos
- 980 Views
- 0 replies
- 0 kudos
PYTEST: Module not found error
Hi,Apologies, as I am trying to use Pytest first time. I know this question has been raised but I went through previous answers but the issue still exists.I am following DAtabricks and other articles using pytest. My structure is simple as -tests--co...
- 980 Views
- 0 replies
- 0 kudos
