cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Kai
by • New Contributor II
  • 2989 Views
  • 1 replies
  • 0 kudos

Resolved! Differences Between "TEMPORARY STREAMING TABLE" and "TEMPORARY STREAMING LIVE VIEW" in DLT

Hello Databricks community,I'm seeking clarification on the distinctions between the following two syntaxes:CREATE OR REFRESH TEMPORARY STREAMING TABLECREATE TEMPORARY STREAMING LIVE VIEWAs of my understanding, both of these methods do not store data...

Data Engineering
Reply
  • 2989 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
gabsylvain
Databricks Employee
  • 0 kudos

Hi @Kai, The two syntaxes you're asking about, CREATE OR REFRESH TEMPORARY STREAMING TABLE and CREATE TEMPORARY STREAMING LIVE VIEW, are used in Delta Live Tables and have distinct purposes. CREATE OR REFRESH TEMPORARY STREAMING TABLE: This syntax i...

  • 0 kudos
RyHubb
by • New Contributor III
  • 4597 Views
  • 5 replies
  • 0 kudos

Resolved! Databricks asset bundles job and pipeline

Hello, I'm looking to create a job which is linked to a delta live table.  Given the job code like this: my_job_name: name: thejobname schedule: quartz_cron_expression: 56 30 12 * * ? timezone_id: UTC pause_stat...

Data Engineering
Reply
  • 4597 Views
  • 5 replies
  • 0 kudos
View Replies
Latest Reply
Yeshwanth
Databricks Employee
  • 0 kudos

@RyHubb  You can specify the variable of the ID and it will be materialized at deploy time. No need to do this yourself. An example is at https://github.com/databricks/bundle-examples/blob/24678f538415ab936e341a04fce207dce91093a8/default_python/...

  • 0 kudos
4 More Replies
leaw
by • New Contributor III
  • 6531 Views
  • 7 replies
  • 0 kudos

Resolved! How to load xml files with spark-xml ?

Hello,I cannot load xml files.First, I tried to install Maven library com.databricks:spark-xml_2.12:0.14.0 as it is told in documentation, but I could not find it. I only have HyukjinKwon:spark-xml:0.1.1-s_2.10, with this one I have this error: DRIVE...

Data Engineering
Reply
  • 6531 Views
  • 7 replies
  • 0 kudos
View Replies
Latest Reply
Frustrated_DE
New Contributor III
  • 0 kudos

Mismatch on Scala version, my bad! Sorted

  • 0 kudos
6 More Replies
rsamant07
by • New Contributor III
  • 4053 Views
  • 2 replies
  • 1 kudos

DBT JOBS FAILING

HI ,we have dbt workflow jobs and its been failing randomly from last few days  with below error.  is there any known issue for this , any help on the root cause will be helpful.Encountered an error: Runtime Error Database Error __init__() got an une...

Data Engineering
dbt
Reply
  • 4053 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
rsamant07
New Contributor III
  • 1 kudos

setting dbt-databricks==1.7.3 solved this issue but now we randomly get the below error . it gets fixed after restrating the cluster sometimes. but is there any permanent solution for this ? from dbt.events import types_pb2 File "/databricks/python3/...

  • 1 kudos
1 More Replies
Andyt
by • New Contributor
  • 1401 Views
  • 1 replies
  • 0 kudos

Restore sql editor

any Options restore sql editors query after workspace was accidentally deleted and restored 

Data Engineering
Reply
  • 1401 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
arpit
Databricks Employee
  • 0 kudos

@Andyt If the workspace is accidentally deleted, there is not way to retrieve content from SQL editor.

  • 0 kudos
WhistlePodu
by • New Contributor
  • 2385 Views
  • 1 replies
  • 0 kudos

How to get Workflow status and error description programmatically ?

Hi,I want to take some basic info by running workflow and populate a table with those data. I want to add logic programmatically in a notebook and will run it by attaching it in a task of workflow.Information required to be populated in table:Job idJ...

Data Engineering
Reply
  • 2385 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
arpit
Databricks Employee
  • 0 kudos

@WhistlePodu You can review the jobs API for getting the other fields like jobs status etc

  • 0 kudos
Sangram
by • New Contributor III
  • 963 Views
  • 1 replies
  • 1 kudos

data engineer course materials are throwing error

Your course material for data engineering associate program is throwing error.Please correct the below error: -This is from section 2.2 "ETL with Spark".

Sangram_0-1705846100252.png
Data Engineering
data engineering
pyspark
spark
Reply
  • 963 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
arpit
Databricks Employee
  • 1 kudos

@Sangram Can you please confirm if that DBFS exists in the specified location?  

  • 1 kudos
pshuk
by • New Contributor III
  • 903 Views
  • 1 replies
  • 0 kudos

Access Databricks Volume through CLI

Hi,I am able to connect to DBFS and transfer files there or download from there. But when I change the path to Volumes, it doesn't work. Even though I created the volume I still get this error message:Error: no such directory: /Volumes/bgem_dev/text_...

Data Engineering
Reply
  • 903 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
arpit
Databricks Employee
  • 0 kudos

@pshuk If you are still facing the issue, can you please share the commands you are using ?May be you need to use the latest CLI version to use Volumes which is a new feature?

  • 0 kudos
Fnazar
by • New Contributor II
  • 997 Views
  • 1 replies
  • 0 kudos

Streaming delta table - Performance with incremental refresh

Hi Team,We are hitting performance issues with Streaming live delta table specifically when evaluating large tables of more than 10million rows. What are the workarounds to handle these streaming live tables in an attempt to load these large tables. ...

Data Engineering
Reply
  • 997 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Priyanka_Biswas
Databricks Employee
  • 0 kudos

Hi @Fnazar  When dealing with streaming data, you might end up with many small files, which can be inefficient. Use Delta Lake's OPTIMIZE command to compact files into larger ones and ZORDER to colocate related information in the same set of files. T...

  • 0 kudos
TCorr15
by • New Contributor
  • 6976 Views
  • 1 replies
  • 0 kudos

Databricks Connect V2 - OPENSSL_internal: CERTIFICATE_VERIFY_FAILED

I am getting an error when using Databricks V2 in when running anything relating to databricks-sql-connector/databricks.sql.connect(). Would anyone know how to resolve this issue?Sample Error Message Additional DetailsPython Version 3.11.4Sample Code...

TCorr15_0-1706177740099.png
Data Engineering
Reply
  • 6976 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
arpit
Databricks Employee
  • 0 kudos

Can you directly use Databricks connect and validate if it works from CLI?Also, confirm the databrics-connect version please 

  • 0 kudos
nitinsingh1
by • New Contributor II
  • 3818 Views
  • 5 replies
  • 2 kudos

Databricks Runtime compatibility error with latest version while reading from (ADLS) Dynamic 365 .

We are trying to establish ingestion from dynamic 365 >> ADLS >> Databricks, While reading information we need to use databricks runtime 6.4 to read the raw data from ADLS into Databricks. Latest databricks runtime couldn’t be used, Need your help to...

Data Engineering
Reply
  • 3818 Views
  • 5 replies
  • 2 kudos
View Replies
Latest Reply
BobBubble2000
New Contributor II
  • 2 kudos

Hi @nitinsingh1 Thank you for bringing up this topic, I'm also currently looking into how to ingest exported Dynamics 365 FO data (csv files with CDM) from ADLS into Databricks. Could you share how you achieved this? I'd be very curious to see your a...

  • 2 kudos
4 More Replies
Manjusha
by • New Contributor II
  • 1651 Views
  • 3 replies
  • 0 kudos

Failed to create notebook on community edition

Hi,I am unable to create new notebook on databricks community edition.  getting error 'failed to create notebook' when I click on  create-> notebookIs anyone else facing the same issue? if so, any tips on how to resolve it?

Data Engineering
Reply
  • 1651 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 0 kudos

Thank you for the update. Please select the best response as a solution, so other community members will be able to get unblock if they have this issue

  • 0 kudos
2 More Replies
Shivanshu_
by • Contributor
  • 3637 Views
  • 4 replies
  • 3 kudos

parallelizing function call in databricks

I have a use case where I have to process stream data and have to create categorical table's(500 table count). I'm using concurrent threadpools to parallelize the whole process, but while seeing the spark UI, my code dosen't utilizes all the workers(...

Data Engineering
parallelism
threading
threadpool executor
Reply
  • 3637 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 3 kudos

You can use DLT, read from many-to-one table.

  • 3 kudos
3 More Replies
Fnazar
by • New Contributor II
  • 1572 Views
  • 3 replies
  • 0 kudos

Streaming live table

I am trying to create a streaming live table using the below syntax : CREATE OR REFRESH STREAMING LIVE TABLE revenue_stream AS (SELECT * FROM stream (finance_silver.finance_db.revenue)) And as I am trying to execute this notebook via DLT pipeline i a...

Fnazar_0-1706699205272.png
Data Engineering
Reply
  • 1572 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 0 kudos

You can use materialized views in serverless only

  • 0 kudos
2 More Replies
FurqanAmin
by • New Contributor II
  • 1077 Views
  • 1 replies
  • 0 kudos

Spark Logs inaccessible - from the UI and dbfs (GCS)

We have a lot of jobs with spark-submit tasks, previously we were able to see the logs for the jobs. Now we are not able to see the logs in the DBX UI.We created a test job for this 'test_job_2' in our workspace to test it out. When the job finishes ...

Data Engineering
Reply
  • 1077 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Yeshwanth
Databricks Employee
  • 0 kudos

@FurqanAmin Could you please attach a screenshot of this?

  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels