- 775 Views
- 2 replies
- 1 kudos
Delta comparison architecture using flatMapGroupsWithState in Structured Streaming
I am designing structured streaming job in Azure data bricks(using Scala) which will consume messages from two event hubs, lets call them source and target.I would like your feedback on below flow, whether it is will survive the production load and ...
- 775 Views
- 2 replies
- 1 kudos
- 1 kudos
It is hard to understand what the source is and what the target is. Some charts could be useful. Also, information on how long the state is kept. My solution usually is:- Use declarative lakeflow pipelines if possible (dlt) - if not, consider handlin...
- 1 kudos
- 2431 Views
- 6 replies
- 4 kudos
Resolved! Databricks partner Tech Summit FY26 access
I'm trying to access the recordings of Partner Tech Summit FY26 which happened a month back. It says lobby is closed.Is there any other way i can access the recordings. I'm yet to watch the day 2 sessions.
- 2431 Views
- 6 replies
- 4 kudos
- 4 kudos
Hi @saurabh18cs , check link shared by @Advika . Make sure you are logged in using partner account.Link - https://partner-academy.databricks.com/learn/catalog/view/168SS:
- 4 kudos
- 1606 Views
- 4 replies
- 5 kudos
Resolved! serialized_dashboard
I have a dashboard.json file, for example: {select * from ${{var.table_name}}}. I have job.yml and section serialized_dashboard there? bcs my job runs parallel with dashboard. Can I use variables in databrics.yml if I define the table_variable variab...
- 1606 Views
- 4 replies
- 5 kudos
- 5 kudos
I currently use the parameter inside IDENTIFIER(:schema || 'my_table') and the 'bundle scripts' feature to perform substitutions, but I hope for better support soon.
- 5 kudos
- 2786 Views
- 4 replies
- 5 kudos
Resolved! Need help understanding Databricks
Hi,I come from a traditional ETL background and am having trouble understanding some of the cloud hyper scalar features and use cases.I understand Databricks is hosted on a cloud providers. I see the cloud providers have their own tools for ETL, ML/A...
- 2786 Views
- 4 replies
- 5 kudos
- 5 kudos
Thanks a lot Gema. For the detailed and meticulous answers.I guess I have to unlearn and relearn everything starting today.
- 5 kudos
- 2071 Views
- 5 replies
- 3 kudos
Resolved! Stateless streaming with aggregations on a DLT/Lakeflow pipeline
In a DLT pipeline I have a bronze table that ingest files using Autoloader, and a derived silver table that, for this example, just stores the number of rows for each file ingested into bronze. The basic code example: import dlt from pyspark.sql impo...
- 2071 Views
- 5 replies
- 3 kudos
- 3 kudos
For scenarios in Databricks where lower latency is needed for Silver tables but continuous streaming pipelines are not feasible, using jobs or notebooks with foreachBatch running in Structured Streaming mode is a common and recommended approach. This...
- 3 kudos
- 900 Views
- 4 replies
- 0 kudos
Data analyst learning plan lab files
Hi all,I am very new to databricks and to this community. I recently signed up for the data analyst learning plan and the data engineering one.The learning platform page seems like confusing maze to navigate! In the course material for the data analy...
- 900 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi,I managed to find the lab. It wasn't straight-forward at all. It was part of another link and no in the learning path I had signed upThe lab series I am trying to work on is thishttps://partner-academy.databricks.com/learn/courses/3701/aibi-for-da...
- 0 kudos
- 2929 Views
- 2 replies
- 1 kudos
More than expected number of Jobs created in Databricks
Hi Databricks Gurus !I am trying to run a very simple snippet :data_emp=[["1","sarvan","1"],["2","John","2"],["3","Jose","1"]]emp_columns=["EmpId","Name","Dept"]df=spark.createDataFrame(data=data_emp, schema=emp_columns)df.show() --------Based on a g...
- 2929 Views
- 2 replies
- 1 kudos
- 1806 Views
- 6 replies
- 6 kudos
Resolved! Cluster cannot find init script stored in Volume
I have created an init script stored in a Volume which I want to execute on a cluster with runtime 16.4 LTS. The cluster has policy = Unrestricted and Access mode = Standard. I have additionally added the init script to the allowlist. This should be ...
- 1806 Views
- 6 replies
- 6 kudos
- 6 kudos
Hi @jimoskar ,Since you're using standard access mode you need to add init script to allowlist. Did you add your init script to allowlist? If not, do the following:In your Databricks workspace, click Catalog.Click the gear icon .Click the metastore ...
- 6 kudos
- 884 Views
- 2 replies
- 3 kudos
Resolved! Delta sharing with Celonis
Is there is any way/plans of Databricks use Delta sharing to provide data access to Celonis?
- 884 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @cbhoga ,Delta Sharing is an open protocol for secure data sharing. Databricks already supports it natively, so you can publish data using Delta Sharing. However, whether Celonis can directly consume that shared data depends on whether Celonis sup...
- 3 kudos
- 1842 Views
- 3 replies
- 4 kudos
Performance Comparison: spark.read vs. Autoloader
Hi there, I would appreciate some help to compare the runtime performance of two approaches to performing ELT in Databricks: spark.read vs. Autoloader. We already have a process in place to extract highly nested json data into a landing path, and fro...
- 1842 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @ChristianRRL ,For that kind of ingestion scenario autoloader is a winner . It will scale much better than batch approach - especially if we are talking about large number of files.If you configure autoloader with file notification mode it can sca...
- 4 kudos
- 1140 Views
- 1 replies
- 2 kudos
Resolved! AutoLoader Ingestion Best Practice
Hi there, I would appreciate some input on AutoLoader best practice. I've read that some people recommend that the latest data should be loaded in its rawest form into a raw delta table (i.e. highly nested json-like schema) and from that data the app...
- 1140 Views
- 1 replies
- 2 kudos
- 2 kudos
I think the key thing with holding the raw data in a table, and not transforming that table, is that you have more flexibility at your disposal. There's a great resource available via Databricks Docs for best practices in the Lakehouse. I'd highly re...
- 2 kudos
- 2194 Views
- 2 replies
- 4 kudos
Resolved! What is `read_files`?
Bit of a silly question, but wondering if someone can help me better understand what is `read_files`?read_files table-valued function | Databricks on AWSThere's at least 3 ways to pull raw json data into a spark dataframe:df = spark.read...df = spark...
- 2194 Views
- 2 replies
- 4 kudos
- 4 kudos
Also, @ChristianRRL , with a slight adjustment to the syntax, it does indeed behave like Autoloaderhttps://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/patterns?language=SQL I'd also advise looking at the different options th...
- 4 kudos
- 6649 Views
- 8 replies
- 0 kudos
Need help migrating company customer and partner academy accounts to work properly
Hi, originally I accidentally made a customer academy account with my company that is a databricks partner. Then I made an account using my personal email and listed my company email as the partner email for the partner academy account. that account ...
- 6649 Views
- 8 replies
- 0 kudos
- 0 kudos
Need help to merge my customer portal id with partner mail id my case number is 00754330
- 0 kudos
- 1695 Views
- 4 replies
- 2 kudos
Trying to reduce latency on DLT pipelines with Autoloader and derived tables
What I'm trying to achieve: ingest files into bronze tables with Autoloader, then produce Kafka messages for each file ingested using a DLT sink.The issue: latency between file ingested and message produced get exponentially higher the more tables ar...
- 1695 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi, I think it is a delay of the autoloader as it doesn't know about the ingested files. It is nothing in common with the state, as it is just an autoloader and it keeps a list of processed files. Autloader scans the directory every minute, usually a...
- 2 kudos
- 1070 Views
- 2 replies
- 2 kudos
Resolved! how to import sample notebook to azure databricks workspace
In the second onboarding video, the Quickstart Notebook is shown. I found that notebook here:https://www.databricks.com/notebooks/gcp-qs-notebook.htmlI wanted to import it to my workspace in Azure Databricks account, to play with it. However, selecti...
- 1070 Views
- 2 replies
- 2 kudos
