cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
tobi
by • New Contributor II
  • 414 Views
  • 3 replies
  • 0 kudos

Deleted workspace

Hello guys, I have a question. We have databricks on gcp, we forgot to pay for subscription and they removed our workspace. I had code notebooks on that workspace. If there is any way to reproduce this code? Or maybe it’s automatically saved this cod...

Data Engineering
Reply
  • 414 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
szymon_dybczak
Contributor III
  • 0 kudos

Hi @tobi ,Unfortunately, if the data was stored directly within the Workspace and not backed up externally, there is no much you can do.Once a Databricks subscription is cancelled, all workspaces associated with that account are deleted and this dele...

  • 0 kudos
2 More Replies
mannepk85
by • New Contributor III
  • 377 Views
  • 0 replies
  • 0 kudos

Databricks academy courses are defaulting to hive metastore

So far, I started 2 Databricks Academy Courses. In both the course, the default is hive-metastore where the schema is created. In my org, hive metastore is blocked and we have been asked to use Unity Catalog. Is there a way the course material in dat...

Data Engineering
Reply
  • 377 Views
  • 0 replies
  • 0 kudos
CaptainJack
by • New Contributor III
  • 565 Views
  • 4 replies
  • 1 kudos

Get taskValue from job as task, and then pass it to next task.

I have workflow like this.1 task: job as a task. Inside this job there is task which is seting parameter x as taskValue using dbutils.jobs.taskValues.set. 2. task dependent on previous job as a task. I would like to access this parameter x. I tried t...

Data Engineering
Reply
  • 565 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
NandiniN
Honored Contributor
  • 1 kudos

I see, I have requested for someone else to guide you on this. cc: @Retired_mod 

  • 1 kudos
3 More Replies
turtleXturtle
by • New Contributor II
  • 623 Views
  • 1 replies
  • 0 kudos

Delta share existing parquet files in R2

Hi - I have existing parquet files in Cloudflare R2 storage (created outside of Databricks).  I would like to share them via Delta Share, but I keep running into an error.  Is it possible to share existing parquet files without duplicating them?I did...

Data Engineering
Reply
  • 623 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
turtleXturtle
New Contributor II
  • 0 kudos

Thanks @Retired_mod.  It's currently possible to share a delta table stored in an S3 external location without duplication or doing the `DEEP CLONE` first.  Is it on the roadmap to support this for R2 as well?

  • 0 kudos
MYB24
by • New Contributor III
  • 6586 Views
  • 6 replies
  • 0 kudos

Resolved! Error: cannot create mws credentials: invalid Databricks Account configuration

Good Evening, I am configuring databricks_mws_credentials through Terraform on AWS.  I am getting the following error:Error: cannot create mws credentials: invalid Databricks Account configuration││ with module.databricks.databricks_mws_credentials.t...

Data Engineering
AWS
credentials
Databricks
Terraform
Reply
  • 6586 Views
  • 6 replies
  • 0 kudos
View Replies
Latest Reply
Alexandre467
New Contributor II
  • 0 kudos

Hello, I'm facing a similaire Issue. I try to update my TF with properly authentification and I have this error ?! â•· │ Error: cannot create mws credentials: failed visitor: context canceled │ │ with databricks_mws_credentials.this, │ on main.tf ...

  • 0 kudos
5 More Replies
riccostamendes
by • New Contributor II
  • 33335 Views
  • 3 replies
  • 0 kudos

Just a doubt, can we develop a kedro project in databricks?

I am asking this because up to now I have just seen some examples of deploying a pre-existent kedro project in databricks in order to run some pipelines...

Data Engineering
Reply
  • 33335 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
noklam
New Contributor II
  • 0 kudos

Hi! Kedro Dev here. You can surely develop Kedro on Databricks, in fact we have a lot of Kedro project running on Databricks. In the past there has been some friction, mainly because Kedro are project based while Databricks focus a lot on notebook. T...

  • 0 kudos
2 More Replies
georgef
by • New Contributor III
  • 1549 Views
  • 2 replies
  • 0 kudos

Resolved! Cannot import relative python paths

Hello,Some variations of this question have been asked before but there doesn't seem to be an answer for the following simple use case:I have the following file structure on a Databricks Asset Bundles project: src --dir1 ----file1.py --dir2 ----file2...

Data Engineering
Reply
  • 1549 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
m997al
Contributor III
  • 0 kudos

Hi.  This was a long-standing issue for me too.  This solution may not be what is desired, but it works perfectly for my needs.In my python code, I have this structure:if __name__ == '__main__': # directory structure where "mycode" is this code ...

  • 0 kudos
1 More Replies
Pierre1
by • New Contributor
  • 600 Views
  • 1 replies
  • 2 kudos

DLT with Unity Catalog: Multipart table name

Hello,I can't seem to find up to date info on how to handle catalog.schema.table in DLT live table and Unity catalog.My statement is the following and is failing with the error: Multipart table name is not supported. Any workaround possible?Thanks a ...

Data Engineering
Reply
  • 600 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
szymon_dybczak
Contributor III
  • 2 kudos

Hi @Pierre1 ,Actually, you don't provide this infromation in the code. You specify this information when you create dlt pipeline.If you do not select a catalog and target schema for a pipeline, tables are not published to Unity Catalog and can only b...

  • 2 kudos
darkolexis
by • New Contributor
  • 592 Views
  • 1 replies
  • 1 kudos

Service Principal types in Azure Databricks

In Azure Databricks, we can create two types of Service Principals, namely:1. Databricks Managed SP2. Microsoft Entra ID Managed SP What is the difference between two, other than one being specific to single workspace, and another being usable from m...

Data Engineering
Reply
  • 592 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
szymon_dybczak
Contributor III
  • 1 kudos

Hi @darkolexis ,So the main difference is that in case of Azure Databricks managed service principals you  can authenticate to Azure Databricks using Databricks OAuth authentication and personal access tokens.In case of Microsoft Entra ID managed ser...

  • 1 kudos
Youngwb
by • New Contributor
  • 496 Views
  • 0 replies
  • 0 kudos

Databricks ODBC driver tooks long time to list columns

I'm testing the performance of databricks, when I use ODBC driver to submit query, I found it slower than notebook,and it looks like odbc driver will send "list columns" request to databricks, 1. Is there a way to prevent the ODBC driver from sending...

Youngwb_0-1722594481062.png
Data Engineering
Databricks
deltalake
ODBC driver
Reply
  • 496 Views
  • 0 replies
  • 0 kudos
mbdata
by • New Contributor II
  • 32230 Views
  • 6 replies
  • 5 kudos

Resolved! Toggle line comment

I work with Azure Databricks. The shortcut Ctrl + / to toggle line comment doesn't work on AZERTY keyboard on Firefox... Do you know this issue ? Is there an other shortcut I can try ? Thanks !

Data Engineering
Reply
  • 32230 Views
  • 6 replies
  • 5 kudos
View Replies
Latest Reply
Flo
New Contributor III
  • 5 kudos

'cmd + shift + 7' works for me!I'm using an AZERTY keyboard on Chrome for MacOS.

  • 5 kudos
5 More Replies
vishal48
by • New Contributor II
  • 285 Views
  • 0 replies
  • 1 kudos

Integrating row and column level security in parent child tables with masking only selected rows

Currently I am working with a project where we need to mask PIIs in few columns for VIP customers only.Let me explain briefly with example:Table A: [personid, status, address, UID, VIPFLAG]   --> Mask "UID" and "address" only where VIPFLAG is 1Table ...

Data Engineering
Reply
  • 285 Views
  • 0 replies
  • 1 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels