cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
jeremy98
by • New Contributor III
  • 88 Views
  • 8 replies
  • 0 kudos

How to deploy unique workflows that running on production

Hello, community!I have a question about deploying workflows in a production environment. Specifically, how can we deploy a group of workflows to production so that they are created only once and cannot be duplicated by others?Currently, if someone d...

Data Engineering
Reply
  • 88 Views
  • 8 replies
  • 0 kudos
View Replies
Latest Reply
jeremy98
New Contributor III
  • 0 kudos

 I had this night another issue:run failed with error message Unable to access the notebook "/Workspace/Users/<user email>/.bundle/rnc_data_pipelines/prod/files/notebook/prod/db_sync_initial_wip". Either it does not exist, or the identity used to run...

  • 0 kudos
7 More Replies
Fikrat
by • New Contributor
  • 28 Views
  • 1 replies
  • 0 kudos

Can SQL task pass its outputs to ForEach task?

Hi there,If I understood correctly, Roland said output SQL task can be used as input to ForEach task in Workflows. I tried that and used the expression sqlTaskName.output.rows, but Databricks rejected that expression. Anyone know how to do that? 

Data Engineering
Reply
  • 28 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Can you confirm if this are the steps being followed: Create the SQL Task: Ensure your SQL task is correctly set up and produces the desired output. For example: SELECT customer_name, market FROM example_customers; Reference the SQL Task Output i...

  • 0 kudos
zuzsad
by • Visitor
  • 41 Views
  • 2 replies
  • 0 kudos

Azure Asset Bundle deploy removes the continous: true configuration

I have this pipeline configuration that I'm deploying using Azure Asset Bundles:ingest-pipeline.test.yml```resources:  pipelines:    ingest-pipeline-test:      name: ingest-pipeline-test-2      clusters:        - label: default          node_type_id:...

Data Engineering
Reply
  • 41 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Which CLI version are you using? Is it the latest version?

  • 0 kudos
1 More Replies
David_Billa
by • New Contributor II
  • 21 Views
  • 1 replies
  • 0 kudos

Unable to convert to date from datetime string with AM and PM

Any help to understand why it's showing 'null' instead of the date value? It's showing null only for 12:00:00 AM and for any other values it's showing date correctlyTO_DATE("12/30/2022 12:00:00 AM", "MM/dd/yyyy HH:mm:ss a") AS tsDate 

Data Engineering
Reply
  • 21 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hi @David_Billa, Can you try with: TO_TIMESTAMP("12/30/2022 12:00:00 AM", "MM/dd/yyyy hh:mm:ss a") AS tsDate The issue you are encountering with the TO_DATE function returning null for the value "12:00:00 AM" is likely due to the format string not ma...

  • 0 kudos
Svish
by • New Contributor
  • 78 Views
  • 3 replies
  • 0 kudos

Resolved! DLT: Schema mismatch error

HiI am encountering the following error when writing a DLT pipeline. Here is my workflow:Read a bronze delta tableCheck Data Quality RulesWrite clean records to a silver table with defined schema. I use TRY_CAST for columns where there is mismatch be...

Data Engineering
Reply
  • 78 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
filipniziol
Contributor III
  • 0 kudos

Hi @Svish ,You have one line that differs:JOB_CERTREP_CONTRACT_INT: string (nullable = true) vs. JOB_CERTREP_CONTRACT_NUMBER: string (nullable = true) 

  • 0 kudos
2 More Replies
stevewb
by • Visitor
  • 54 Views
  • 2 replies
  • 1 kudos

Resolved! databricks bundle deploy fails when job includes dbt task and git_source

I am trying to deploy a dbt task as part of a databricks job using databricks asset bundles.However, there seems to be a clash that occurs when specifying a job that includes a dbt task that causes a bizarre failure.I am using v0.237.0 of the CLI.Min...

Data Engineering
Reply
  • 54 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
madams
Contributor
  • 1 kudos

Thanks for providing that whole example, it was really easy to fiddle with.  I think I've found your solution.  Update the original two tasks on the job (if you want to keep them) like this: tasks: - task_key: notebook_task job...

  • 1 kudos
1 More Replies
Taja
by • New Contributor II
  • 26 Views
  • 0 replies
  • 0 kudos

Delta Live Tables: large use

Does anyone use Delta Live Table on large scale in production pipelines ? Are they satisfied with the product ?Recently, I´ve started a PoC to evaluate the DLT and notice some concerns:- Excessive use of compute resources when you check the cluster m...

Data Engineering
Reply
  • 26 Views
  • 0 replies
  • 0 kudos
filipniziol
by • Contributor III
  • 61 Views
  • 5 replies
  • 2 kudos

Magic Commands (%sql) Not Working with Databricks Extension for VS Code

Hi Community,I’ve encountered an issue with the Databricks Extension for VS Code that seems to contradict the documentation. According to the Databricks documentation, the extension supports magic commands like %sql when used with Databricks Connect:...

filipniziol_0-1734692630751.png
Data Engineering
Reply
  • 61 Views
  • 5 replies
  • 2 kudos
View Replies
Latest Reply
Walter_C
Databricks Employee
  • 2 kudos

Got it, I will check with my internal team to validate if there is any issue around.

  • 2 kudos
4 More Replies
HoussemBL
by • New Contributor II
  • 23 Views
  • 1 replies
  • 0 kudos

Impact of deleting workspace on associated catalogs

Hello Community,I have a specific scenario regarding Unity Catalog and workspace deletion that I'd like to clarify:Current Setup:Two DataBricks workspaces: W1 and W2Single Unity Catalog instanceCatalog1: Created in W1, shared and accessible in W2Cata...

Data Engineering
Reply
  • 23 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hi @HoussemBL  When you delete a Databricks workspace, it does not directly impact the Unity Catalog or the data within it. Unity Catalog is a separate entity that manages data access and governance across multiple workspaces. Here’s what happens in ...

  • 0 kudos
thisisthemurph
by • New Contributor II
  • 32 Views
  • 1 replies
  • 0 kudos

Databricks dashboards across multiple Databricks instances

We have multiple Databricks instances, one per environment (Dev-UK, Live-UK Live-EU, Live-US, etc), and we would like to create dashboards to present stats on our data in each of these environments. Each of these environments also has a differently n...

Data Engineering
Reply
  • 32 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Hello, as you have mentioned you could create an script in Python that uses the api call https://docs.databricks.com/api/workspace/lakeview/create to generate the dashboard for each environment, the process to create the visualizations will be comple...

  • 0 kudos
dollyb
by • Contributor
  • 99 Views
  • 5 replies
  • 0 kudos

Accessing Workspace / Repo file works in notebook, but not from job

In a notebook attached to.a normal personal cluster I can successfully do this:%fs ls file:/Workspace/Repos/$userName/$repoName/$folderNameWhen I run an init-script on a UC volume that the does the same thing, I'm getting this error:ls: cannot access...

Data Engineering
Reply
  • 99 Views
  • 5 replies
  • 0 kudos
View Replies
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hi @dollyb, Can you try with just "ls /Workspace/Repos/my_user_name@company.com/my_repo_name/my_folder_name" I'm not sure dbutils will be useful in an init script, I will try to test it out

  • 0 kudos
4 More Replies
minhhung0507
by • New Contributor II
  • 66 Views
  • 2 replies
  • 0 kudos

Handling Dropped Records in Delta Live Tables with Watermark - Need Optimization Strategy

Hi Databricks Community,I'm encountering an issue with watermarks in Delta Live Tables that's causing data loss in my streaming pipeline. Let me explain my specific problem:Current SituationI've implemented watermarks for stateful processing in my De...

Data Engineering
Reply
  • 66 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
minhhung0507
New Contributor II
  • 0 kudos

 Dear @Walter_C, thank you for your detailed response regarding watermark handling in Delta Live Tables (DLT). I appreciate the guidance provided, but I would like further clarification on a couple of points related to our use case.1. Auto-Saving Dro...

  • 0 kudos
1 More Replies
lauraxyz
by • New Contributor III
  • 57 Views
  • 3 replies
  • 0 kudos

refresh online table: How to get update_id and check status of a specific update

Hi!I have a workflow job to trigger a refresh of an online table. How can I get the update_id with this specific refresh?Also, is it possible to get the status from this specific update_id?Thanks!

Data Engineering
Reply
  • 57 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
lauraxyz
New Contributor III
  • 0 kudos

Another qq: Since online table has 3 sync mode: Snapshot, Triggered, and Continuous.   when refreshing the online table with w.pipelines.start_update(pipeline_id='{pipeline_id}', full_refresh=True) which sync mode is used by default? 

  • 0 kudos
2 More Replies
jeremy98
by • New Contributor III
  • 434 Views
  • 10 replies
  • 2 kudos

Resolved! ImportError: cannot import name 'AnalyzeArgument' from 'pyspark.sql.udtf'

Hello community,I installed databricks extension on my vscode ide. How to fix this error? I created the environment to run locally my notebooks and selected the available remote cluster to execute my notebook, what else?I Have this error: ImportError...

Data Engineering
Reply
  • 434 Views
  • 10 replies
  • 2 kudos
View Replies
Latest Reply
jeremy98
New Contributor III
  • 2 kudos

@unj1m yes, as Alberto said you don't need to install pyspark, it is included in your cluster configuration.

  • 2 kudos
9 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels