- 534 Views
- 2 replies
- 2 kudos
Resolved! Looking for resources to learn Databricks
Hi Community Members,I'm working as a Power BI Developer and interested to upskill into Databricks platform as a Data Analyst and Data Engineering.Request to share the resources(documentation/video tutorials) in a sequential order.Thank You!Best Rega...
- 534 Views
- 2 replies
- 2 kudos
- 2 kudos
For Bite-size overviews check the demo center - https://www.databricks.com/resources/demos/library This youtube channel is great for more detailed oriented discussion around specific features .https://www.youtube.com/@nextgenlakehouseFor more structu...
- 2 kudos
- 4565 Views
- 3 replies
- 0 kudos
Unlocking the Power of Databricks: A Comprehensive Guide for Beginners
In the rapidly evolving world of big data, Databricks has emerged as a leading platform for data engineering, data science, and machine learning. Whether you're a data professional or someone looking to expand your knowledge, understanding Databricks...
- 4565 Views
- 3 replies
- 0 kudos
- 0 kudos
This is a clear and helpful guide to Databricks. You explained its key features and learning steps in a beginner-friendly way, making it easy for readers to get started and build practical skills in data analytics and machine learning. There is sone ...
- 0 kudos
- 8609 Views
- 6 replies
- 2 kudos
Resolved! Understanding Autoscaling in Databricks: Under What Conditions Does Spark Add a New Worker Node?
I’m currently working with Databricks autoscaling configurations and trying to better understand how Spark decides when to spin up additional worker nodes. My cluster has a minimum of one worker and can scale up to five. I know that tasks are assigne...
- 8609 Views
- 6 replies
- 2 kudos
- 2 kudos
Is the above information true for job clusters as well? Looks like the enhanced auto scalar is only available for pipelines
- 2 kudos
- 280 Views
- 2 replies
- 3 kudos
Resolved! Am i publishing article in a correct way or not?
Hello Community,I’d like to check with the contributors whether the article I recently published follows the correct approach. Did I choose the right options and the appropriate place to publish it in the Databricks Community?https://community.databr...
- 280 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Kirankumarbs ,Yes, you did everything in correct manner. You put your article in correct place which is "Community Articles".Anyway, thanks for sharing with us
- 3 kudos
- 231 Views
- 0 replies
- 1 kudos
Better Diff for Jupyter Notebooks in Bitbucket
Comparing versions of Jupyter Notebooks (new preferred format on Databricks) in Bitbucket is much more difficult than the previous format. TPlease use the link below vote on adding better Jupyter Notebooks comparison to Bitbucket.Enable rich renderin...
- 231 Views
- 0 replies
- 1 kudos
- 4981 Views
- 6 replies
- 1 kudos
Cannot import editable installed module in notebook
Hi,I have the following directory structure:- mypkg/ - setup.py - mypkg/ - __init__.py - module.py - scripts/ - main # notebook From the `main` notebok I have a cell that runs:%pip install -e /path/to/mypkgThis command appears to succ...
- 4981 Views
- 6 replies
- 1 kudos
- 1 kudos
Sorry to triple post but I have another update: it seems to work for standalone clusters, but it refuses to build the wheel (I get a write permission error) on the job clusters.
- 1 kudos
- 3380 Views
- 2 replies
- 0 kudos
Best practices for tableau to connect to Databricks
Having problem in connecting to Databrikcs with service principal from tableau . Wanted to how how tableau extracts refreshing connecting to databricks , is it via individual Oauth or service principal
- 3380 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @cheerwthraj, To connect Tableau to Databricks and refresh extracts, you can use either OAuth or service principal authentication. For best practices, please refer to the below link, https://docs.databricks.com/en/partners/bi/tableau.html#best-pr...
- 0 kudos
- 542 Views
- 2 replies
- 1 kudos
Resolved! cluster and workflow issue
com.databricks:spark-xml_2.12:0.18.0 com.crealytics:spark-excel_2.12:3.4.3_0.20.4 in prerequisites_maven.yml and i created cluster and ran from this updated cluster notebook running but jobs failing UnknownException: (java.util.ServiceConfiguratio...
- 542 Views
- 2 replies
- 1 kudos
- 1 kudos
You can now natively read Excel files https://docs.databricks.com/aws/en/query/formats/excel
- 1 kudos
- 433 Views
- 1 replies
- 1 kudos
Resolved! cannot see "User Provisioning " in settings in Databricks Account management console
Hi Team , I came across below issues , need help to resolve the issue's .Issue 1 :- cannot see "User Provisioning " in settings in Databricks Account management console.Issue 2: - Account Admin -Toggle - Failed to provision user. Please ensure the...
- 433 Views
- 1 replies
- 1 kudos
- 1 kudos
Hey dpavanbo! 1. In the account console, go to Security > User provisioning. If you see “Automatic identity management,” that’s expected on Azure; it replaces traditional SCIM UI and handles JIT on first sign‑in. 2. Automatic identity management: ...
- 1 kudos
- 182 Views
- 1 replies
- 0 kudos
I placed an swag but did not reiceve it just want to lnow the status
I want to know the status of my swag
- 182 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Ritika-08! Could you please share a few more details about the swag you’re referring to, such as which program or event it was associated with?
- 0 kudos
- 2330 Views
- 1 replies
- 1 kudos
Left Outer Join returns an Inner Join in Delta Live Tables
In our Delta Live Table pipeline I am simply joining two streaming tables to a new streaming table.We use the following code: @Dlt.create_table() def fact_event_faults(): events = dlt.read_stream('event_list').withWatermark('TimeStamp', '4 hours'...
- 2330 Views
- 1 replies
- 1 kudos
- 1 kudos
did you ever get this resolved? struggling with a similar problem
- 1 kudos
- 618 Views
- 3 replies
- 3 kudos
Resolved! CDC / Event Driven Data Ingestion
Hello Guys,I am planning to implement Event Driven Data Ingestion from Bronze -> Silver -> Gold layer in my project. Currently we are having batch processing approach for our data ingestion pipelines. We have decided to move away from batch process t...
- 618 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi Mey,Please also consider databrick file arrival trigger for your event driven data ingestion journey.https://docs.databricks.com/aws/en/jobs/file-arrival-triggersRegards, Kartik
- 3 kudos
- 721 Views
- 2 replies
- 4 kudos
A Smarter Approach to Data Quality Monitoring
For a long time, data quality has been one of the most painful parts of data engineering.Most of us have written rules and thresholds that looked correct but didn’t reflect how data was actually used. We ended up with too many alerts that didn’t matt...
- 721 Views
- 2 replies
- 4 kudos
- 4 kudos
agentic data quality monitoring is a focused approach for what really matters...
- 4 kudos
- 4762 Views
- 2 replies
- 2 kudos
Stream failure JsonParseException
Hi all! I am having the following issue with a couple of pyspark streams. I have some notebooks running each of them an independent file structured streaming using delta bronze table (gzip parquet files) dumped from kinesis to S3 in a previous job....
- 4762 Views
- 2 replies
- 2 kudos
- 2 kudos
Thanks for the detail answer I've been searching for. If you play at online casinos, you should check out the best online casinos that payout that offer the best gaming experiences.
- 2 kudos
- 359 Views
- 1 replies
- 1 kudos
Resolved! Databricks Spatial SQL - examples and tutorials?
Hello,I recently learned about Databricks being now able to handle geospatial data. With my background in geospatial, that got me curious for sure I was wondering if anybody has used examples, maybe a tutorial, to learn about geospatial in databricks...
- 359 Views
- 1 replies
- 1 kudos
- 1 kudos
HI @AnneEst I havent played around with spatial data but if I had to i would start with these two blogs . https://www.databricks.com/blog/introducing-spatial-sql-databricks-80-functions-high-performance-geospatial-analyticshttps://www.databricks.com/...
- 1 kudos
