- 3128 Views
- 2 replies
- 1 kudos
Resolved! Databricks AI + Data Summit discount coupon
Hi Community,I hope you're doing well.I wanted to ask you the following: I want to go to Databricks AI + Data Summit this year, but it's super expensive for me. And hotels in San Francisco, as you know, are super expensive.So, I wanted to know how I ...
- 3128 Views
- 2 replies
- 1 kudos
- 2099 Views
- 1 replies
- 0 kudos
Can we get the branch name from Notebook
Hi Folks,Is there a way to display the current git branch name from Databricks notebook Thanks
- 2099 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes, you can display the current git branch name from a Databricks notebook in several ways: Using the Databricks UI The simplest method is to use the Databricks UI, which already shows the current branch name:- In a notebook, look for the button nex...
- 0 kudos
- 1213 Views
- 1 replies
- 0 kudos
DLT Pipeline
Hello, I have written below simple code to write data to Catalogue table using simple DLT pipeline .As part of Below program am reading a file from blob container and trying to write to a Catalogue table . New catalogue table got created but table d...
- 1213 Views
- 1 replies
- 0 kudos
- 0 kudos
The issue with your DLT pipeline is that you've defined the table and schema correctly, but you haven't actually implemented the data loading logic in your `ingest_from_storage()` function. While you've created the function, you're not calling it any...
- 0 kudos
- 1615 Views
- 1 replies
- 0 kudos
How to get the hadoopConfiguration in a unity catalog standard access mode app ?
Context:job running using a job clustered configured in Standard access mode ( Shared Access mode )scala 2.12.15 / spark 3.5.0 jar programDatabricks runtime 15.4 LTSIn this context, it is not possible to get the sparkSession.sparkContext, as confirme...
- 1615 Views
- 1 replies
- 0 kudos
- 0 kudos
In Unity Catalog standard access mode (formerly shared access mode) with Databricks Runtime 15.4 LTS, direct access to `sparkSession.sparkContext` is restricted as part of the security limitations. However, there are still ways to access the Hadoop c...
- 0 kudos
- 8248 Views
- 1 replies
- 0 kudos
How to connect to an on-premise implementation of S3 storage (such as Minio) in Databricks Notebooks
I manage a large data lake of Iceberg tables stored on premise in S3 storage from MinIO. I need a Spark cluster to run ETL jobs. I decided to try Databricks as there were no other good options. However, I'm unable to properly access my tables or even...
- 8248 Views
- 1 replies
- 0 kudos
- 0 kudos
Not sure, but Databricks may default to AWS-style paths if the configurations are incomplete. Try setting the MinIO endpoint by configuring spark.hadoop.fs.s3a.endpoint to your MinIO server's URL. If MinIO uses HTTP, disable SSL by setting spark.hado...
- 0 kudos
- 4666 Views
- 2 replies
- 0 kudos
Create DLT pipeline in CI/CD with role segregation
In the documentation, most examples use the CREATE OR REFRESH STREAMING TABLE command.Meanwhile, from a role segregation perspective, create and refresh operations should happen in a separate context. That is, we want to create these objects (which e...
- 4666 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Malthe, refreshing is automatically handled during pipeline runs in here. To implement effective role segregation, you should define separate DLT pipelines for deployment and execution, each with its own set of roles and permissions. This approac...
- 0 kudos
- 2010 Views
- 1 replies
- 1 kudos
Resolved! Jobs overhead why ?
Hi, I have a py notebook that I want to execute in an automated manner. One way I found this was to attach this to a job/task and hit it using the api from my local. However this seems to be adding significant overhead, my code even if it’s just one ...
- 2010 Views
- 1 replies
- 1 kudos
- 1 kudos
Hey @Krthk If you want to orchestrate a notebook, the easiest way is to go to File > Schedule directly from the notebook. My recommendation is to use cron syntax to define when it should run, and attach it to a predefined cluster or configure a new j...
- 1 kudos
- 55872 Views
- 5 replies
- 3 kudos
Using Azure Key Vault secret to access Azure Storage
I am trying to configure access to Azure Storage Account (ADLS2) using OAUTH. The doc here gives an example of how to specify a secret in a cluster's spark configuration{{secrets/<secret-scope>/<service-credential-key>}}I can see how this works for ...
- 55872 Views
- 5 replies
- 3 kudos
- 3 kudos
New doc link : https://learn.microsoft.com/en-us/azure/databricks/security/secrets/
- 3 kudos
- 2691 Views
- 2 replies
- 1 kudos
Resolved! Query: Extracting Resolved 'Input' Parameter from a Databricks Workflow Run
Hi Everyone,I have a query regarding extracting the resolved value of the 'Input' parameter (highlighted in yellow in the attached images) from a Databricks workflow run.The images show:The foreach task receives its input from the Metadata_Fetcher ta...
- 2691 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Nexusss7 Out of curiosity, I tried to retrieve the resolved task parameter values. Finding a way to retrieve executed sub-tasks by the for_each task using APIs was challenging. So, I devised a solution using API and system tables. I simplified t...
- 1 kudos
- 2303 Views
- 1 replies
- 0 kudos
Cluster configuration
Hi, Please help me configure/choose the cluster configuration. I need to process and merge 6 million records into Azure SQL DB. At the end of the week, 9 billion records need to be processed and merged into Azure SQL DB, and a few transformations nee...
- 2303 Views
- 1 replies
- 0 kudos
- 0 kudos
@Pu_123 Option 1 Daily Load (6M Records) - Cost-OptimizedCluster Mode: Single NodeVM Type: Standard_DS4_v2 or Standard_E4ds_v5Workers: 1Driver Node: Same as workerDatabricks Runtime: 13.x LTS (Photon Optional)Terminate after: 10-15 mins of inactivit...
- 0 kudos
- 1925 Views
- 1 replies
- 0 kudos
Databricks labs $200 or not
Hi all,Looking for an honest review for anyone has had experience with the Databricks labs. Would it be more beneficial to learn without the labs and setup own infrastructure?Any advice would be greatly appreciated, newbie over here Thanks,Stringer
- 1925 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Stringer! From my experience, Databricks Labs makes learning easier by handling the setup and eliminating cloud costs. This is perfect if you’re just starting out or want to focus purely on Databricks. But since it abstracts things like networ...
- 0 kudos
- 3427 Views
- 3 replies
- 0 kudos
Resolved! Error when executing an INSERT statement on an External Postgres table from Databricks SQL Editor
Hi,This is the context of my issue:I have an AWS RDS Postgres database instance setup. I have also set up a Postgres CONNECTION in Databricks and can view the Postgres tables under a newly created FOREIGN CATALOG in Databricks Unity Catalog.Using the...
- 3427 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @pankj0510, DML for tables is blocked from Databricks SQL, you can only read from DBSQL. I think you can set up a JDBC URL to the Postgres database and use Spark/Pandas DataFrame write methods to insert data
- 0 kudos
- 2300 Views
- 1 replies
- 0 kudos
Text alignment in databricks dashboard markdown
Hi All,How can I align the text inside the Dashboard markdown to the middle?Is there an option to do this?Thanks,Gal
- 2300 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Gal_Sb! Databricks markdown does not support text alignment, and HTML/CSS do not work for this purpose in Databricks dashboards. You can try formatting options like headers or spacing adjustments. I'll also check with the team to explore possi...
- 0 kudos
- 2345 Views
- 3 replies
- 1 kudos
Resolved! DLT Pipeline Validate will always spawn new cluster
Hi all!I've started learning DLT-Pipelines but I am struggling with the development of a pipeline.As far as I understand it, once I click on “Validate” a cluster will spin-up and stay (by default for 2hours), if the pipeline is in “Development” mode....
- 2345 Views
- 3 replies
- 1 kudos
- 1 kudos
Well, turns out if I do not make any changes to the cluster settings when creating a new pipeline (i.e. keep default) it works as expected (every new "validate" skips the "waiting for resources"-step).Initially, I reduced the number of workers to a m...
- 1 kudos
- 2158 Views
- 4 replies
- 0 kudos
DLT refresh time for combination of streaming and non streaming tables?
@dlt.tabledef joined_table(): dim_df = spark.read.table("dim_table") # Reloads every batch fact_df = spark.readStream.table("fact_stream") return fact_df.join(dim_df, "id", "left")
- 2158 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi,Current approach reloads dim_df in every batch, which can be inefficient. To optimize, consider broadcasting dim_df if it's small or using a mapGroupsWithState function for stateful joins. Also, ensure that fact_df has sufficient watermarking to h...
- 0 kudos
