cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
188386
by • New Contributor II
  • 1166 Views
  • 2 replies
  • 0 kudos

Databricks Learning - Get Started with Databricks for Data Engineering -> Next button not active

Hi,Databricks Learning - Get Started with Databricks for Data Engineering (ID: E-03ZW80) got stuck at lesson where file "get-started-with-data-engineering-on-databricks-2.1.zip" is downloaded. The "Next" button is not active - see attached picture.Li...

Data Engineering
Reply
  • 1166 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Ajay-Pandey
Esteemed Contributor III
  • 0 kudos

Hi @188386 As I can see you have skipped multiple lesson(Introduction to Data on Databricks) first complete it in sequence then next button will be enabled for you.

  • 0 kudos
1 More Replies
bgerhardi
by • New Contributor III
  • 11145 Views
  • 12 replies
  • 13 kudos

Surrogate Keys with Delta Live

We are considering moving to Delta Live tables from a traditional sql-based data warehouse. Worrying me is this FAQ on identity columns Delta Live Tables frequently asked questions | Databricks on AWS this seems to suggest that we basically can't cre...

Data Engineering
Reply
  • 11145 Views
  • 12 replies
  • 13 kudos
View Replies
Latest Reply
Pelle123
New Contributor II
  • 13 kudos

I am wondering if there is an answer to this question now.

  • 13 kudos
11 More Replies
IN
by • New Contributor II
  • 508 Views
  • 1 replies
  • 1 kudos

Connect to remote SQL Server (add databricks cluster IP to the whitelist)

Hi, I would need to connect from a notebook on workspaces to a remote SQL server instance. This server is protected by a firewall, thus I would need to add an IP address to the whitelist. Ideally, if it would be possible to setup/allocate static IP a...

Data Engineering
Reply
  • 508 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
szymon_dybczak
Esteemed Contributor III
  • 1 kudos

Hi @IN ,In Azure, you can deploy Workspace in VNET Injection mode and attach NAT Gateway to your VNET.NAT GW require Public IP.  This IP will be your static egress IP for all Cluster in for this Workspace.I've never worked with GCP,  but I think you ...

  • 1 kudos
Henrik_
by • New Contributor III
  • 1559 Views
  • 1 replies
  • 1 kudos

Resolved! Optimizing recursive joins on group and UNION-operations.

The code snippet below takes each group (based on id) and perform recursive joins to build parent-child relations  (id1 and id2) within a group. The code produce the correct output, an array in column 'path'.However, in my real world use-case, this c...

Data Engineering
Reply
  • 1559 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 1 kudos

The recursive join is definitely a performance killer.  It will blow up the query plan.So I would advice against using it.Alternatives?  Well, a fixed amount of joins for example, if that is an option of course.Using a graph algorithm is also an opti...

  • 1 kudos
n_joy
by • Contributor
  • 2520 Views
  • 6 replies
  • 2 kudos

Resolved! Change data feed for tables with allowColumnDefaults property "enabled"

I have a Delta table already created, with both enabled the #enableChangeDataFeed option and #allowColumnDefaults properties. However when writing to CDC table with streaming queries it fails with the following error [CREATE TABLE command because it ...

Data Engineering
Reply
  • 2520 Views
  • 6 replies
  • 2 kudos
View Replies
Latest Reply
n_joy
Contributor
  • 2 kudos

@filipniziolYes, that is what I do  Thanks for feedback ! 

  • 2 kudos
5 More Replies
enda
by • New Contributor II
  • 594 Views
  • 3 replies
  • 1 kudos

Course missing required content.

https://customer-academy.databricks.com/learn/course/1266/play/7847/navigate-the-workspace-user-interface;lp=10 Its missing the file you import to your workspace theres no file to download?

Data Engineering
Reply
  • 594 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
enda
New Contributor II
  • 1 kudos

Cheers thanks i see now!

  • 1 kudos
2 More Replies
ashraf1395
by • Honored Contributor
  • 1293 Views
  • 0 replies
  • 0 kudos

Databricks Finops Assessment

We have to deliver a Databricks Finops Assessment project. I am trying to write a proposal for it. I haven't done one before. I have created a general process of how the assessment will look like and then restructured it using gpt.Plz give your feedb...

Data Engineering
Reply
  • 1293 Views
  • 0 replies
  • 0 kudos
Rishabh-Pandey
by • Esteemed Contributor
  • 782 Views
  • 1 replies
  • 3 kudos

Creating a shareable dashboard

AI/BI Dashboards offer a robust solution for securely sharing visualizations, and insights throughout your organization. You can easily share these dashboards with users within your Databricks workspace, across other workspaces in your organization, ...

Data Engineering
Reply
  • 782 Views
  • 1 replies
  • 3 kudos
View Replies
Latest Reply
Anushree_Tatode
Honored Contributor II
  • 3 kudos

Hi Rishabh,Nice post, AI/BI Dashboards make it easy to share data securely within and across workspaces, even with view-only users. This way, everyone gets the right info while keeping things controlled. Excited to learn more about the key features!A...

  • 3 kudos
AndyG
by • New Contributor II
  • 3149 Views
  • 6 replies
  • 1 kudos

Remove partition column from delta table

I have delta tables with multiple partition columns. I want to remove most of the partition columns and retain just one. I can see there are ALTER TABLE...PARTITION options but these are not supported for delta lake tables. So is there a way to do th...

Data Engineering
Reply
  • 3149 Views
  • 6 replies
  • 1 kudos
View Replies
Latest Reply
szymon_dybczak
Esteemed Contributor III
  • 1 kudos

Hi @AndyG ,Maybe try the way official delta guid is suggesting:Adding and Deleting Partitions in Delta Lake tables | Delta LakeYou can delete all rows from a given partition to remove the partition from the Delta table.Here’s how to delete all the ro...

  • 1 kudos
5 More Replies
guangyi
by • Contributor III
  • 772 Views
  • 2 replies
  • 2 kudos

Resolved! How to use dlt.expect to validate table level constraints?

I know how to validate the column level constraint, like checking whether the specified column value is larger than target value.Can I validate some table level constraints? For example, validate whether the total records count of a table is larger t...

Data Engineering
Reply
  • 772 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
szymon_dybczak
Esteemed Contributor III
  • 2 kudos

Hi @guangyi ,Unfortunately, there is no out of the box solution for this requirement in dlt. But as a workaround you can add an additional view/table to your pipeline that defines an expectation in similar way to below: CREATE OR REFRESH MATERIALIZED...

  • 2 kudos
1 More Replies
modest2va
by • New Contributor II
  • 1527 Views
  • 6 replies
  • 2 kudos

AnalysisException: is not a Delta table. but that table is Delta table

When running a Databricks notebook,an error occurs stating that SOME_TABLE is not a Delta table.However, after executing the describe detail command and checking the format,the table is shown as Delta.Without taking any specific actions, re-running t...

Data Engineering
Reply
  • 1527 Views
  • 6 replies
  • 2 kudos
View Replies
Latest Reply
Witold
Honored Contributor
  • 2 kudos

Another thing you could check is how the underlying data looks like. Maybe the actual writer of the data, messed it up.

  • 2 kudos
5 More Replies
Fnazar
by • New Contributor II
  • 1177 Views
  • 3 replies
  • 0 kudos

Resolved! Migrate delta live tables from a non-uc to uc enabled databricks workspace

Hello Team,I have a usecase where I need to migrate Delta Live Tables from a non-Unity Catalog Databricks workspace to a Unity Catalog enabled Databricks workspace. What will be the efficient approach to achieve this scenario, please suggest?

Data Engineering
Reply
  • 1177 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
szymon_dybczak
Esteemed Contributor III
  • 0 kudos

Hi @Fnazar ,You can try to use UCX by Databricks Labs. It automates whole process. databrickslabs/ucx: Automated migrations to Unity Catalog (github.com)

  • 0 kudos
2 More Replies
varshini_reddy
by • New Contributor III
  • 2920 Views
  • 2 replies
  • 1 kudos

Databricks Jobs For Each Functionality

Hi Team, I have two questions, please clarigy.1. can we use multiple tasks inside a for each for databricks jobs2. is nested for each functionality available?

Data Engineering
Reply
  • 2920 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
filipniziol
Esteemed Contributor
  • 1 kudos

Hi @varshini_reddy,While running multiple tasks within a for each loop is not available, there is a simple workaround.Workaround: create a main notebook to run multiple child notebooks:Suppose you need to run notebook1, notebook2, and notebook3 withi...

  • 1 kudos
1 More Replies
Ravikumashi
by • Contributor
  • 4472 Views
  • 4 replies
  • 0 kudos

issue with azure databricks workspace after we disable public network access

Hi All,We had azure databricks workspaces created thru terraform with public network access enabled to true and everything was working great. recently we have disabled the public network access and started to face issues.terraform is uanble to add us...

Data Engineering
Reply
  • 4472 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
Mohamednazeer
New Contributor III
  • 0 kudos

Do you got any solution for this issue? I am also facing the same issue.

  • 0 kudos
3 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels