cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
Boyan
by • New Contributor II
  • 1383 Views
  • 1 replies
  • 0 kudos

Running unit tests and hyperopt causes a broadcast variable exception

Hello,We are using hyperopt to train a model with relatively large train dataset.We've experience some performance issues and following the suggestions in this notebook, we broadcasted the dataset.To verify that broadcasting the dataset resolved the ...

Data Engineering
Reply
  • 1383 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
Hello,We are using hyperopt to train a model with relatively large train dataset.We've experience some performance issues and following the suggestions in this notebook, we broadcasted the dataset.To verify that broadcasting the dataset resolved the ...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
tinai_long
by • New Contributor III
  • 8042 Views
  • 9 replies
  • 4 kudos

Resolved! How to refresh a single table in Delta Live Tables?

Suppose I have a Delta Live Tables framework with 2 tables: Table 1 ingests from a json source, Table 2 reads from Table 1 and runs some transformation.In other words, the data flow is json source -> Table 1 -> Table 2. Now if I find some bugs in the...

Data Engineering
Reply
  • 8042 Views
  • 9 replies
  • 4 kudos
View Replies
Latest Reply
cpayne_vax
New Contributor III
  • 4 kudos

Answering my own question: nowadays (February 2024) this can all be done via the UI.When viewing your DLT pipeline there is a "Select tables for refresh" button in the header. If you click this, you can select individual tables, and then in the botto...

  • 4 kudos
8 More Replies
brickster_2018
by • Databricks Employee
  • 11095 Views
  • 3 replies
  • 6 kudos

Resolved! How to add I custom logging in Databricks

I want to add custom logs that redirect in the Spark driver logs. Can I use the existing logger classes to have my application logs or progress message in the Spark driver logs.

Data Engineering
Reply
  • 11095 Views
  • 3 replies
  • 6 kudos
View Replies
Latest Reply
Kaizen
Valued Contributor
  • 6 kudos

1) Is it possible to save all the custom logging to its own file? Currently it is being logging with all other cluster logs (see image) 2) Also Databricks it seems like a lot of blank files are also being created for this. Is this a bug? this include...

  • 6 kudos
2 More Replies
sha
by • New Contributor
  • 1138 Views
  • 1 replies
  • 0 kudos

Importing data from S3 to Azure DataBricks Cluster with Unity Catalog in Shared Mode

Environment details:DataBricks on Azure, 13.3 LTS, Unity Catalog, Shared Cluster mode.Currently in the environment I'm in, we run imports from S3 with code like:spark.read.option('inferSchema', 'true').json(s3_path).  When running on a cluster in Sha...

Data Engineering
Reply
  • 1138 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
BR_DatabricksAI
Contributor
  • 0 kudos

Hello Sha, We usually get such error while working with shared cluster mode assuming this your dev environment just to avoid errors, please use different clusters. However as a alternative solution in case if would like to keep the shared cluster the...

  • 0 kudos
Dhruv-22
by • New Contributor III
  • 3572 Views
  • 4 replies
  • 0 kudos

CREATE TABLE does not overwrite location whereas CREATE OR REPLACE TABLE does

I am working on Azure Databricks, with Databricks Runtime version being - 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12). I am facing the following issue.Suppose I have a view named v1 and a database f1_processed created from the following comman...

Data Engineering
Reply
  • 3572 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
Ayushi_Suthar
Databricks Employee
  • 0 kudos

Hi @Dhruv-22 ,  Based on the information you shared above, the "CREATE OR REPLACE" and "CREATE" commands in Databricks do have different behaviours, particularly when it comes to handling tables with specific target locations. The "CREATE OR REPLACE"...

  • 0 kudos
3 More Replies
DApt
by • New Contributor II
  • 8362 Views
  • 1 replies
  • 2 kudos

REDACTED_POSSIBLE_SECRET_ACCESS_KEY as part of column value result form aes_encrypt

Hi, i've encountered an error using base64/aes_encrypt, as result the string saved contains 'REDACTED_POSSIBLE_SECRET_ACCESS_KEY' at the end destroying the original data, rendering it useless undecryptable, is there a way to avoid this replacement in...

Captura de pantalla 2023-12-11 152523.png DApt_0-1702326511602.png DApt_3-1702327037748.png DApt_1-1702326665014.png
Data Engineering
Reply
  • 8362 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
DataEnthusiast1
New Contributor II
  • 2 kudos

I had the same issue, and my usage was similar to OP:base64(aes_encrypt(<clear_text>, unbase64(secret(<scope>, <key>))))Databricks support suggested to not call secret within the insert/update operation that writes to the table. After updating the py...

  • 2 kudos
Dhruv-22
by • New Contributor III
  • 3209 Views
  • 3 replies
  • 1 kudos

Resolved! REPLACE TABLE AS SELECT is not working with parquet whereas it works fine for delta

I am working on Azure Databricks, with Databricks Runtime version being - 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12). I am facing the following issue.Suppose I have a view named v1 and a database f1_processed created from the following comman...

Data Engineering
Reply
  • 3209 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Ayushi_Suthar
Databricks Employee
  • 1 kudos

Hi @Dhruv-22  We understand that you are facing the following error when using REPLACE TABLE AS SELECT  on the Parquet Table but at this moment the REPLACE TABLE AS SELECT operation you're trying to perform is not supported for Parquet tables. Accord...

  • 1 kudos
2 More Replies
Kroy
by • Contributor
  • 1400 Views
  • 2 replies
  • 0 kudos

Near Real time Solutioning on data from Core System which gets updated

We are trying to build solution , where customer data stored in one of RDBM database SQL server and we are moving this data to delta lake in medallion architecture and want to this to be near real time by using DLT pipeline.Problem is that source tab...

Data Engineering
Reply
  • 1400 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Kroy
Contributor
  • 0 kudos

came across this matrix while reading about DLT what is read from complete and write to Incremental means ?

  • 0 kudos
1 More Replies
billfoster
by • New Contributor II
  • 15158 Views
  • 8 replies
  • 4 kudos

how can I learn DataBricks

I am currently enrolled in data engineering boot camp. We go over various technologies azure , pyspark , airflow , Hadoop ,nosql,SQL, python. But not over something like databricks. I am in contact with lots of recent graduates who landed a job. Almo...

Data Engineering
Reply
  • 15158 Views
  • 8 replies
  • 4 kudos
View Replies
Latest Reply
Ali23
New Contributor II
  • 4 kudos

 I'd be glad to help you on your journey to learning Databricks! Whether you're a beginner or aiming to advance your skills, here's a comprehensive guide:Foundations:Solid understanding of core concepts: Begin with foundational knowledge in big data,...

  • 4 kudos
7 More Replies
John_Rotenstein
by • New Contributor II
  • 7056 Views
  • 1 replies
  • 0 kudos

Resolved! "Run Job" without waiting for target job to finish?

We have configured a task in Job-A to run Job-B.However, the task in Job-A continues to 'run' until Job-B has completed.I can see this would be useful if we wanted to wait for Job-B and then perform another task, but we would actually like Job-A to e...

Data Engineering
Reply
  • 7056 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
feiyun0112
Honored Contributor
  • 0 kudos

you can create and run job use sdkdatabricks-sdk-py/examples/jobs/run_now_jobs_api_full_integration.py at main · databricks/databricks-sdk-py (github.com)

  • 0 kudos
DavidMooreZA
by • New Contributor II
  • 1758 Views
  • 2 replies
  • 0 kudos

Structure Streaming - Table(s) to File(s) - Is it possible?

Hi,I'm trying to do something that's probably considered a no-no. The documentation makes me believe it should be possible. But, I'm getting lots of weird errors when trying to make it work.If anyone has managed to get something similar to work, plea...

Data Engineering
Reply
  • 1758 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
DavidMooreZA
New Contributor II
  • 0 kudos

I created a new schema and volume specifically for this testing and the names are all distinct from any other objects in the catalog.I did a quick double check just in case I somehow missed a duplicate and there were none.

  • 0 kudos
1 More Replies
pgagliardi
by • New Contributor II
  • 1770 Views
  • 1 replies
  • 2 kudos

Latest pushed code is not taken into account by Notebook

Hello, I cloned a repo my_repo in the Dataricks space Repos.Inside my_repo, I created a notebook new_experiment where I can import functions from my_repo, which is really handy. When I want to modify a function in my_repo, I open my local IDE, do the...

Data Engineering
Reply
  • 1770 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
Jnguyen
Databricks Employee
  • 2 kudos

Use  %reload_ext autoreload instead, it will do your expected behavior.You just need to run it once, like %load_ext autoreload %autoreload 2

  • 2 kudos
jcoggs
by • New Contributor II
  • 3235 Views
  • 2 replies
  • 1 kudos

Handling Exceptions from dbutils.fs in Python

I have a notebook that calls dbutils.fs.ls() for some derived file path in azure. Occasionally, this path may not exist, and in general I can't always guarantee that the path exists. When the path doesn't exist it throws an "ExecutionError" which app...

Data Engineering
dbutils
Error
Exceptions
Reply
  • 3235 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Palash01
Valued Contributor
  • 1 kudos

Hey @jcoggs The problem looks legit though never occurred to me as I try to keep my mounts manually fed to the pipeline using a parameters or a variable by doing this you will have more control over your pipelines see if you could do the same in your...

  • 1 kudos
1 More Replies
Maaax
by • New Contributor
  • 919 Views
  • 0 replies
  • 0 kudos

Table shared via Delta sharingschema not shown in Unity-Catalog

Hello dear community,I use the following command to register the provider including shares to unity-catalog:databricks unity-catalog create-provider --name company --recipient-profile-json-file ~/Develop/profile.jsonOnce it is registered, I could see...

Data Engineering
Reply
  • 919 Views
  • 0 replies
  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels