- 851 Views
- 4 replies
- 0 kudos
Row count is higher than expected after breaking cases into 15-minute segments
I’m building a fact table in Databricks where the grain is one row per case per 15-minute segment. For example, if a case lasts 90 minutes, I expect about 6 rows.I generate the 15-minute segments and MERGE into a Delta table (incremental load). Howev...
- 851 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @CuriousEngineer, This is a common scenario, and there are good solutions. Nice job including the code and the minimal reproducible example -- that makes it much easier to help. There are two places in your query that are very likely introducing d...
- 0 kudos
- 3323 Views
- 8 replies
- 2 kudos
Resolved! Continuous Job - How to set max_retries
Hello Community,I have a couple of continuous workflows (jobs) running in production, and they’ve been working well so far. However, we’re seeing some transient failures that are causing the entire job to restart — which I’d prefer to avoid.While we ...
- 3323 Views
- 8 replies
- 2 kudos
- 2 kudos
Hi @Kirankumarbs, Thanks for posting this. I can see from the thread that the existing replies confirmed the limitation but did not offer much in the way of alternatives. Let me provide a fuller picture. THE SHORT ANSWER You cannot set a custom max_r...
- 2 kudos
- 1473 Views
- 3 replies
- 4 kudos
Resolved! Where is Lakeflow Designer in the databricks UI
Hello everyone,Yesterday I saw a demo of Lakeflow Designer, and it looked very interesting. I tried to find this option in the UI, but I was not able to locate it.Could anyone please guide me on where I can access or try out this feature?I would real...
- 1473 Views
- 3 replies
- 4 kudos
- 4 kudos
If you are talking about the No-code version that is described here, then that is still in Private Preview as far as I am aware. Talk to your Databricks Account if you want to opt into the preview. If you do get access to it, you should be able to fi...
- 4 kudos
- 1375 Views
- 2 replies
- 1 kudos
Resolved! Share dashboard with external customers
I’m new to Databricks and still exploring the platform. One of our requirements is to share a Databricks dashboard with multiple external customers (for example, Customer 1 and Customer 2).The key requirement is data isolation:When a user from Custom...
- 1375 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @RJ1 - Welcome to Databricks Platform! Great question. This is a very common requirement. At a high level, you’ll want to separate the concerns: How users authenticate and are identified (who is the user / which customer are they from?)How data i...
- 1 kudos
- 783 Views
- 3 replies
- 2 kudos
Resolved! Data governance solution
I am here looking for Data governance solution for organization. I also searched this on many different website and found many solutions. We are bit confused to which one to choose. One of my friend suggest me this platform, and i am hoping i will ge...
- 783 Views
- 3 replies
- 2 kudos
- 2 kudos
You can use Unity Catalog for Databricks Lakehouse. You can use Collibra/Open Metadata along with Unity Catalog for complete governance
- 2 kudos
- 688 Views
- 2 replies
- 2 kudos
Resolved! Looking for resources to learn Databricks
Hi Community Members,I'm working as a Power BI Developer and interested to upskill into Databricks platform as a Data Analyst and Data Engineering.Request to share the resources(documentation/video tutorials) in a sequential order.Thank You!Best Rega...
- 688 Views
- 2 replies
- 2 kudos
- 2 kudos
For Bite-size overviews check the demo center - https://www.databricks.com/resources/demos/library This youtube channel is great for more detailed oriented discussion around specific features .https://www.youtube.com/@nextgenlakehouseFor more structu...
- 2 kudos
- 9582 Views
- 6 replies
- 2 kudos
Resolved! Understanding Autoscaling in Databricks: Under What Conditions Does Spark Add a New Worker Node?
I’m currently working with Databricks autoscaling configurations and trying to better understand how Spark decides when to spin up additional worker nodes. My cluster has a minimum of one worker and can scale up to five. I know that tasks are assigne...
- 9582 Views
- 6 replies
- 2 kudos
- 2 kudos
Is the above information true for job clusters as well? Looks like the enhanced auto scalar is only available for pipelines
- 2 kudos
- 401 Views
- 2 replies
- 3 kudos
Resolved! Am i publishing article in a correct way or not?
Hello Community,I’d like to check with the contributors whether the article I recently published follows the correct approach. Did I choose the right options and the appropriate place to publish it in the Databricks Community?https://community.databr...
- 401 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Kirankumarbs ,Yes, you did everything in correct manner. You put your article in correct place which is "Community Articles".Anyway, thanks for sharing with us
- 3 kudos
- 300 Views
- 0 replies
- 1 kudos
Better Diff for Jupyter Notebooks in Bitbucket
Comparing versions of Jupyter Notebooks (new preferred format on Databricks) in Bitbucket is much more difficult than the previous format. TPlease use the link below vote on adding better Jupyter Notebooks comparison to Bitbucket.Enable rich renderin...
- 300 Views
- 0 replies
- 1 kudos
- 5452 Views
- 6 replies
- 1 kudos
Cannot import editable installed module in notebook
Hi,I have the following directory structure:- mypkg/ - setup.py - mypkg/ - __init__.py - module.py - scripts/ - main # notebook From the `main` notebok I have a cell that runs:%pip install -e /path/to/mypkgThis command appears to succ...
- 5452 Views
- 6 replies
- 1 kudos
- 1 kudos
Sorry to triple post but I have another update: it seems to work for standalone clusters, but it refuses to build the wheel (I get a write permission error) on the job clusters.
- 1 kudos
- 3794 Views
- 2 replies
- 0 kudos
Best practices for tableau to connect to Databricks
Having problem in connecting to Databrikcs with service principal from tableau . Wanted to how how tableau extracts refreshing connecting to databricks , is it via individual Oauth or service principal
- 3794 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @cheerwthraj, To connect Tableau to Databricks and refresh extracts, you can use either OAuth or service principal authentication. For best practices, please refer to the below link, https://docs.databricks.com/en/partners/bi/tableau.html#best-pr...
- 0 kudos
- 801 Views
- 2 replies
- 1 kudos
Resolved! cluster and workflow issue
com.databricks:spark-xml_2.12:0.18.0 com.crealytics:spark-excel_2.12:3.4.3_0.20.4 in prerequisites_maven.yml and i created cluster and ran from this updated cluster notebook running but jobs failing UnknownException: (java.util.ServiceConfiguratio...
- 801 Views
- 2 replies
- 1 kudos
- 1 kudos
You can now natively read Excel files https://docs.databricks.com/aws/en/query/formats/excel
- 1 kudos
- 586 Views
- 1 replies
- 1 kudos
Resolved! cannot see "User Provisioning " in settings in Databricks Account management console
Hi Team , I came across below issues , need help to resolve the issue's .Issue 1 :- cannot see "User Provisioning " in settings in Databricks Account management console.Issue 2: - Account Admin -Toggle - Failed to provision user. Please ensure the...
- 586 Views
- 1 replies
- 1 kudos
- 1 kudos
Hey dpavanbo! 1. In the account console, go to Security > User provisioning. If you see “Automatic identity management,” that’s expected on Azure; it replaces traditional SCIM UI and handles JIT on first sign‑in. 2. Automatic identity management: ...
- 1 kudos
- 276 Views
- 1 replies
- 0 kudos
I placed an swag but did not reiceve it just want to lnow the status
I want to know the status of my swag
- 276 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Ritika-08! Could you please share a few more details about the swag you’re referring to, such as which program or event it was associated with?
- 0 kudos
- 2494 Views
- 1 replies
- 1 kudos
Left Outer Join returns an Inner Join in Delta Live Tables
In our Delta Live Table pipeline I am simply joining two streaming tables to a new streaming table.We use the following code: @Dlt.create_table() def fact_event_faults(): events = dlt.read_stream('event_list').withWatermark('TimeStamp', '4 hours'...
- 2494 Views
- 1 replies
- 1 kudos
- 1 kudos
did you ever get this resolved? struggling with a similar problem
- 1 kudos
