cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

vijay_boopathy
by • New Contributor
  • 12862 Views
  • 1 replies
  • 1 kudos

Hive vs Delta

I'm curious about your experiences with Hive and Delta Lake. What are the advantages of using Delta over Hive, and in what scenarios would you recommend choosing Delta for data processing tasks? I'd appreciate any insights or recommendations based on...

  • 12862 Views
  • 1 replies
  • 1 kudos
Latest Reply
Walter_C
Databricks Employee
  • 1 kudos

Delta Lake offers several advantages over Hive. One of the key benefits is its design for petabyte-scale data lakes with streaming and fast access at the forefront. This makes it more suitable for near-real-time streams, unlike Hive. Delta Lake also ...

  • 1 kudos
William_Scardua
by • Valued Contributor
  • 4059 Views
  • 2 replies
  • 0 kudos

Drop array in a struct field

Hi guys,look my table definitionwell, I need to remove 'med array' inside that 'equip' field.have any idea ?Thank you

Screenshot 2024-04-02 at 19.03.38.png
  • 4059 Views
  • 2 replies
  • 0 kudos
Latest Reply
Sampath_Kumar
New Contributor II
  • 0 kudos

Hi William,There is array_remove method that can help to remove the elements from an array. Here med array is an element in equip array.If it not helpful, please share some sample data so that we can solve it together.Reference: array_removeThanks

  • 0 kudos
1 More Replies
sharma_kamal
by • New Contributor III
  • 4767 Views
  • 2 replies
  • 1 kudos

Resolved! Getting errors while reading data from URL

I'm encountering some issues while trying to read a public dataset from a URL using Databricks. Here's the code snippet(along with errors) I'm working with: I'm confused about Delta format error here.When I read data from a URL, how would it have a D...

sharma_kamal_1-1710132330915.png
  • 4767 Views
  • 2 replies
  • 1 kudos
Latest Reply
MuthuLakshmi
Databricks Employee
  • 1 kudos

@sharma_kamal  Please disable the formatCheck in notebook and check if you could read the data The configuration command %sql SET spark.databricks.delta.formatCheck.enabled=false will disable the format check for Delta tables in Databricks. Databrick...

  • 1 kudos
1 More Replies
Yuki
by • Contributor
  • 3400 Views
  • 2 replies
  • 1 kudos

Can I use Git provider with using Service Principal in job

Hi everyone,I'm trying to use Git provider in Databricks job.First, I was using my personal user account to `Run as`.But when I change `Run as` to Service Principal, it was failed because of permission error.And I can't find a way to solve it.Could I...

Yuki_0-1699340000007.png
  • 3400 Views
  • 2 replies
  • 1 kudos
Latest Reply
martindlarsson
New Contributor III
  • 1 kudos

The documentation is lacking in this area which should be easy to set up. Instead we are forced to search among community topics such as these.

  • 1 kudos
1 More Replies
MAR1
by • New Contributor
  • 5137 Views
  • 0 replies
  • 0 kudos

[ Databricks - Delta sharing ] Issue with Delta Sharing in Databricks: Unable to Query Shared Views

Hi guys, I've encountered an issue while attempting to query shared views via Delta Sharing in Databricks. We are using Delta Sharing Databricks-to-Databricks protocol to share data from a databricks environment deployed on azure to another databrick...

  • 5137 Views
  • 0 replies
  • 0 kudos
Gabrosciurti
by • New Contributor
  • 2350 Views
  • 0 replies
  • 0 kudos

Can't run Delta Live Tables pipeline while using Mosaic

I'm just learning to use mosaic and delta live table in Databricks. I was following this example https://github.com/databrickslabs/mosaic/tree/main/notebooks/examples/python/OpenStreetMaps and after configuring everything I'm not able to run the pipe...

Data Engineering
Delta Live Tables
mosaic
  • 2350 Views
  • 0 replies
  • 0 kudos
Leszek
by • Contributor
  • 1814 Views
  • 0 replies
  • 0 kudos

Delta Sharing - RESOURCE_LIMIT_EXCEEDED (AddFiles, RemoveFiles)

Hi,I would like to start using Delta Sharing but I would need to check if my tables hit limitations which are in the following article: RESOURCE_LIMIT_EXCEEDED error when querying a Delta Sharing table - Databricks.Delta Sharing has limits on the met...

  • 1814 Views
  • 0 replies
  • 0 kudos
r-goswami
by • New Contributor II
  • 3001 Views
  • 3 replies
  • 0 kudos

Unable to create/save job of type "python script"

Hi All,We are facing an issue while creating a simple job of type "python script". A python file in workspace is selected as a source. No arguments/job parameters are provided. This is a strange behavior and just started occurring since today morning...

  • 3001 Views
  • 3 replies
  • 0 kudos
Latest Reply
r-goswami
New Contributor II
  • 0 kudos

Hi Ayushi,How can I call call RESET API? this issue is occurring when creating a new job from databricks web UI. It looks like REST API is for resetting job settings of an existing job.Can this be an issue with the databricks workspace we are using?A...

  • 0 kudos
2 More Replies
johnp
by • New Contributor III
  • 2884 Views
  • 0 replies
  • 0 kudos

Access Azure App service failed with 403 response

We have an Azure app service written in Django.  From databricks notebook we sent curl command to test the connection between databricks and Azure AppService.  We got the following repsonse:Error 403 - ForbiddenThe web app you have attempted to reach...

  • 2884 Views
  • 0 replies
  • 0 kudos
AxelBrsn
by • Databricks Partner
  • 9553 Views
  • 3 replies
  • 2 kudos

Resolved! Use DLT from another pipeline

Hello, I have a question.Context :I have a Unity Catalog organized with three schemas (bronze, silver and gold). Logically, I would like to create tables in each schemas.I tried to organize my pipelines on the layers, which mean that I would like to ...

  • 9553 Views
  • 3 replies
  • 2 kudos
Latest Reply
AxelBrsn
Databricks Partner
  • 2 kudos

Hello, thanks for the answers @YuliyanBogdanov, @standup1.So the solution is to use catalog.schema.table, and not LIVE.table, that's the key, you were right standup!But, you won't have the visibility of the tables on Bronze Pipeline, if you are on Si...

  • 2 kudos
2 More Replies
cosminsanda
by • New Contributor III
  • 8427 Views
  • 8 replies
  • 0 kudos

Adding a new column triggers reprocessing of Auto Loader source table

I have a source table A in Unity Catalog. This table is constantly written to and is a streaming table.I also have another table B in Unity Catalog. This is a managed table with liquid clustering.Using Auto Loader I move new data from A to B using a ...

Data Engineering
auto-loader
  • 8427 Views
  • 8 replies
  • 0 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

change data feed might be a solution for you perhaps.https://docs.databricks.com/en/delta/delta-change-data-feed.html

  • 0 kudos
7 More Replies
databrick_usert
by • New Contributor
  • 3627 Views
  • 1 replies
  • 0 kudos

Workspace client creation error

Hi,We are trying to use Python SDK and create a workspace client using the following code:%pip install databricks-sdk --upgrade dbutils.library.restartPython()from databricks.sdk import WorkspaceClientw = WorkspaceClient()Here is the notebook: https:...

  • 3627 Views
  • 1 replies
  • 0 kudos
Latest Reply
Ayushi_Suthar
Databricks Employee
  • 0 kudos

Hi @databrick_usert , Hope you are doing well!  Can you check the version of the SDK running in this notebook? If it's not an upgraded version then could you please try to upgrade the SDK version and then restart the python after the pip install?  %p...

  • 0 kudos
Cloudtek
by • New Contributor
  • 2043 Views
  • 0 replies
  • 0 kudos

delete non empty folders from workspace

Looking for a solution to delete 1000's of non-empty folders that I created wrongly while migration. (looking for solution that delete thoudands of folders created with below name)Folders have a path like "shared/db/..../..../..../yippy.dbc"

  • 2043 Views
  • 0 replies
  • 0 kudos
Kingston
by • New Contributor II
  • 4911 Views
  • 3 replies
  • 0 kudos

Unable to overwrite table to Azure sql db

Hi I have a requirement to read table from azure sql db and update the table in azure databricks with transformations and overwrite updated table to the azure sql db but due to lazy evaluation of pyspark im unable to overwrite the table in azure sql ...

  • 4911 Views
  • 3 replies
  • 0 kudos
Latest Reply
YuliyanBogdanov
New Contributor III
  • 0 kudos

 Hi @Kingston Make sure that you have the proper permissions on the SQL server for the user you do the authentication through JDBC with, i.e. database reader / database writer. Then your approach can go in two directions, push the data from Databrick...

  • 0 kudos
2 More Replies
Labels