cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
jdata
by • Visitor
  • 858 Views
  • 4 replies
  • 1 kudos

Dashboard Usage

Hi there,My team is developing some SQL Dashboards. I would like to know how many people view that dashboard/or at least click to it and then queries triggered.I found out that there is one endpoint provided by Databricks: List Queries | Query Histor...

Data Engineering
Reply
  • 858 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
TakuyaOmi
Contributor III
  • 1 kudos

If your requirement is to retrieve the number of dashboard viewers, you can achieve this using the SQL query described in the previously shared audit logs documentation.If you prefer to retrieve this information via an API, you can use the Statement ...

  • 1 kudos
3 More Replies
Tarun_sahu
by • Visitor
  • 51 Views
  • 2 replies
  • 2 kudos

Databricks exam got suspended without any reason.

Hi Team,My Databricks certified Data engineer Associate exam got suspended on 22nd December and it is in progress state.I was continuously in front of camera and suddenly the alert appeared , and paused the session with the message that contact with ...

Data Engineering
Reply
  • 51 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
seraer
Visitor
  • 2 kudos

The best

  • 2 kudos
1 More Replies
Akash_Wadhankar
by • New Contributor II
  • 27 Views
  • 0 replies
  • 1 kudos

Data Engineering Journey on Databricks

For any new Data Engineering aspirant, it has always been a difficult where to start the learning journey. I faced this challenge a decade ago. In order to help new aspirants I created a series of medium article for new learners. I hope it brings mor...

Data Engineering
Reply
  • 27 Views
  • 0 replies
  • 1 kudos
Fikrat
by • New Contributor II
  • 84 Views
  • 2 replies
  • 0 kudos

Can SQL task pass its outputs to ForEach task?

Hi there,If I understood correctly, Roland said output SQL task can be used as input to ForEach task in Workflows. I tried that and used the expression sqlTaskName.output.rows, but Databricks rejected that expression. Anyone know how to do that? 

Data Engineering
Reply
  • 84 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Can you confirm if this are the steps being followed: Create the SQL Task: Ensure your SQL task is correctly set up and produces the desired output. For example: SELECT customer_name, market FROM example_customers; Reference the SQL Task Output i...

  • 0 kudos
1 More Replies
vijaypodili
by • New Contributor II
  • 58 Views
  • 3 replies
  • 0 kudos

databricks job taking longer time to load 2.3 gb data from bolb to ssms table

df_CorpBond= spark.read.format("parquet").option("header", "true").load(f"/mnt/{container_name}/raw_data/dsl.corporate.parquet") df_CorpBond.repartition(20).write\ .format("jdbc")\ .option("url", url_connector)\ .option("dbtable", "MarkIt...

Data Engineering
datarbricks
performance
Reply
  • 58 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
vijaypodili
New Contributor II
  • 0 kudos

i'm trying to say my cluster have enough storage space 94gb so it can easy handle 2.3 gb data but my job taking longer time time job 2  and 3 completed with in the 3 min but job 4 taking longer to to complete its tasks

  • 0 kudos
2 More Replies
jeremy98
by • New Contributor III
  • 139 Views
  • 8 replies
  • 0 kudos

How to deploy unique workflows that running on production

Hello, community!I have a question about deploying workflows in a production environment. Specifically, how can we deploy a group of workflows to production so that they are created only once and cannot be duplicated by others?Currently, if someone d...

Data Engineering
Reply
  • 139 Views
  • 8 replies
  • 0 kudos
View Replies
Latest Reply
jeremy98
New Contributor III
  • 0 kudos

 I had this night another issue:run failed with error message Unable to access the notebook "/Workspace/Users/<user email>/.bundle/rnc_data_pipelines/prod/files/notebook/prod/db_sync_initial_wip". Either it does not exist, or the identity used to run...

  • 0 kudos
7 More Replies
zuzsad
by • New Contributor
  • 72 Views
  • 2 replies
  • 0 kudos

Azure Asset Bundle deploy removes the continous: true configuration

I have this pipeline configuration that I'm deploying using Azure Asset Bundles:ingest-pipeline.test.yml```resources:  pipelines:    ingest-pipeline-test:      name: ingest-pipeline-test-2      clusters:        - label: default          node_type_id:...

Data Engineering
Reply
  • 72 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Which CLI version are you using? Is it the latest version?

  • 0 kudos
1 More Replies
robbe
by • New Contributor III
  • 1630 Views
  • 3 replies
  • 1 kudos

Resolved! Get job ID from Asset Bundles

When using Asset Bundles to deploy jobs, how does one get the job ID of the resources that are created?I would like to deploy some jobs through asset bundles, get the job IDs, and then trigger these jobs programmatically outside the CI/CD pipeline us...

Data Engineering
Reply
  • 1630 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
nvashisth
New Contributor II
  • 1 kudos

Refer this answer and this can be a solution to above scenario -> https://community.databricks.com/t5/data-engineering/getting-job-id-dynamically-to-create-another-job-to-refer-as-job/m-p/102860/highlight/true#M41252

  • 1 kudos
2 More Replies
David_Billa
by • New Contributor II
  • 38 Views
  • 1 replies
  • 0 kudos

Unable to convert to date from datetime string with AM and PM

Any help to understand why it's showing 'null' instead of the date value? It's showing null only for 12:00:00 AM and for any other values it's showing date correctlyTO_DATE("12/30/2022 12:00:00 AM", "MM/dd/yyyy HH:mm:ss a") AS tsDate 

Data Engineering
Reply
  • 38 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hi @David_Billa, Can you try with: TO_TIMESTAMP("12/30/2022 12:00:00 AM", "MM/dd/yyyy hh:mm:ss a") AS tsDate The issue you are encountering with the TO_DATE function returning null for the value "12:00:00 AM" is likely due to the format string not ma...

  • 0 kudos
najmead
by • Contributor
  • 19512 Views
  • 7 replies
  • 13 kudos

How to convert string to datetime with correct timezone?

I have a field stored as a string in the format "12/30/2022 10:30:00 AM"If I use the function TO_DATE, I only get the date part... I want the full date and time.If I use the function TO_TIMESTAMP, I get the date and time, but it's assumed to be UTC, ...

Data Engineering
Reply
  • 19512 Views
  • 7 replies
  • 13 kudos
View Replies
Latest Reply
Rajeev_Basu
Contributor III
  • 13 kudos

use from_utc_timestamp(to_timestam("<string>", <format>),<timezone>)

  • 13 kudos
6 More Replies
Svish
by • New Contributor II
  • 103 Views
  • 3 replies
  • 0 kudos

Resolved! DLT: Schema mismatch error

HiI am encountering the following error when writing a DLT pipeline. Here is my workflow:Read a bronze delta tableCheck Data Quality RulesWrite clean records to a silver table with defined schema. I use TRY_CAST for columns where there is mismatch be...

Data Engineering
Reply
  • 103 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
filipniziol
Contributor III
  • 0 kudos

Hi @Svish ,You have one line that differs:JOB_CERTREP_CONTRACT_INT: string (nullable = true) vs. JOB_CERTREP_CONTRACT_NUMBER: string (nullable = true) 

  • 0 kudos
2 More Replies
stevewb
by • New Contributor
  • 86 Views
  • 2 replies
  • 1 kudos

Resolved! databricks bundle deploy fails when job includes dbt task and git_source

I am trying to deploy a dbt task as part of a databricks job using databricks asset bundles.However, there seems to be a clash that occurs when specifying a job that includes a dbt task that causes a bizarre failure.I am using v0.237.0 of the CLI.Min...

Data Engineering
Reply
  • 86 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
madams
Contributor
  • 1 kudos

Thanks for providing that whole example, it was really easy to fiddle with.  I think I've found your solution.  Update the original two tasks on the job (if you want to keep them) like this: tasks: - task_key: notebook_task job...

  • 1 kudos
1 More Replies
Taja
by • New Contributor II
  • 44 Views
  • 0 replies
  • 0 kudos

Delta Live Tables: large use

Does anyone use Delta Live Table on large scale in production pipelines ? Are they satisfied with the product ?Recently, I´ve started a PoC to evaluate the DLT and notice some concerns:- Excessive use of compute resources when you check the cluster m...

Data Engineering
Reply
  • 44 Views
  • 0 replies
  • 0 kudos
filipniziol
by • Contributor III
  • 108 Views
  • 5 replies
  • 2 kudos

Magic Commands (%sql) Not Working with Databricks Extension for VS Code

Hi Community,I’ve encountered an issue with the Databricks Extension for VS Code that seems to contradict the documentation. According to the Databricks documentation, the extension supports magic commands like %sql when used with Databricks Connect:...

filipniziol_0-1734692630751.png
Data Engineering
Reply
  • 108 Views
  • 5 replies
  • 2 kudos
View Replies
Latest Reply
Walter_C
Databricks Employee
  • 2 kudos

Got it, I will check with my internal team to validate if there is any issue around.

  • 2 kudos
4 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels