- 10494 Views
- 1 replies
- 1 kudos
Hive vs Delta
I'm curious about your experiences with Hive and Delta Lake. What are the advantages of using Delta over Hive, and in what scenarios would you recommend choosing Delta for data processing tasks? I'd appreciate any insights or recommendations based on...
- 10494 Views
- 1 replies
- 1 kudos
- 1 kudos
Delta Lake offers several advantages over Hive. One of the key benefits is its design for petabyte-scale data lakes with streaming and fast access at the forefront. This makes it more suitable for near-real-time streams, unlike Hive. Delta Lake also ...
- 1 kudos
- 2652 Views
- 2 replies
- 0 kudos
Drop array in a struct field
Hi guys,look my table definitionwell, I need to remove 'med array' inside that 'equip' field.have any idea ?Thank you
- 2652 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi William,There is array_remove method that can help to remove the elements from an array. Here med array is an element in equip array.If it not helpful, please share some sample data so that we can solve it together.Reference: array_removeThanks
- 0 kudos
- 2581 Views
- 0 replies
- 1 kudos
Register more redirect URIs for OAuth U2M
I am following this guide on allowing OAuth U2M for Azure Databricks.When I get to Step 2, I make a request to account.azuredatabricks.net and specify a redirect URI to receive a code.The redirect URI in the example is localhost:8020. If I change thi...
- 2581 Views
- 0 replies
- 1 kudos
- 2915 Views
- 2 replies
- 1 kudos
Resolved! Getting errors while reading data from URL
I'm encountering some issues while trying to read a public dataset from a URL using Databricks. Here's the code snippet(along with errors) I'm working with: I'm confused about Delta format error here.When I read data from a URL, how would it have a D...
- 2915 Views
- 2 replies
- 1 kudos
- 1 kudos
@sharma_kamal Please disable the formatCheck in notebook and check if you could read the data The configuration command %sql SET spark.databricks.delta.formatCheck.enabled=false will disable the format check for Delta tables in Databricks. Databrick...
- 1 kudos
- 2505 Views
- 2 replies
- 1 kudos
Can I use Git provider with using Service Principal in job
Hi everyone,I'm trying to use Git provider in Databricks job.First, I was using my personal user account to `Run as`.But when I change `Run as` to Service Principal, it was failed because of permission error.And I can't find a way to solve it.Could I...
- 2505 Views
- 2 replies
- 1 kudos
- 1 kudos
The documentation is lacking in this area which should be easy to set up. Instead we are forced to search among community topics such as these.
- 1 kudos
- 3614 Views
- 0 replies
- 0 kudos
[ Databricks - Delta sharing ] Issue with Delta Sharing in Databricks: Unable to Query Shared Views
Hi guys, I've encountered an issue while attempting to query shared views via Delta Sharing in Databricks. We are using Delta Sharing Databricks-to-Databricks protocol to share data from a databricks environment deployed on azure to another databrick...
- 3614 Views
- 0 replies
- 0 kudos
- 1795 Views
- 0 replies
- 0 kudos
Can't run Delta Live Tables pipeline while using Mosaic
I'm just learning to use mosaic and delta live table in Databricks. I was following this example https://github.com/databrickslabs/mosaic/tree/main/notebooks/examples/python/OpenStreetMaps and after configuring everything I'm not able to run the pipe...
- 1795 Views
- 0 replies
- 0 kudos
- 1157 Views
- 0 replies
- 0 kudos
Delta Sharing - RESOURCE_LIMIT_EXCEEDED (AddFiles, RemoveFiles)
Hi,I would like to start using Delta Sharing but I would need to check if my tables hit limitations which are in the following article: RESOURCE_LIMIT_EXCEEDED error when querying a Delta Sharing table - Databricks.Delta Sharing has limits on the met...
- 1157 Views
- 0 replies
- 0 kudos
- 1806 Views
- 3 replies
- 0 kudos
Unable to create/save job of type "python script"
Hi All,We are facing an issue while creating a simple job of type "python script". A python file in workspace is selected as a source. No arguments/job parameters are provided. This is a strange behavior and just started occurring since today morning...
- 1806 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi Ayushi,How can I call call RESET API? this issue is occurring when creating a new job from databricks web UI. It looks like REST API is for resetting job settings of an existing job.Can this be an issue with the databricks workspace we are using?A...
- 0 kudos
- 2033 Views
- 0 replies
- 0 kudos
Access Azure App service failed with 403 response
We have an Azure app service written in Django. From databricks notebook we sent curl command to test the connection between databricks and Azure AppService. We got the following repsonse:Error 403 - ForbiddenThe web app you have attempted to reach...
- 2033 Views
- 0 replies
- 0 kudos
- 5914 Views
- 3 replies
- 2 kudos
Resolved! Use DLT from another pipeline
Hello, I have a question.Context :I have a Unity Catalog organized with three schemas (bronze, silver and gold). Logically, I would like to create tables in each schemas.I tried to organize my pipelines on the layers, which mean that I would like to ...
- 5914 Views
- 3 replies
- 2 kudos
- 2 kudos
Hello, thanks for the answers @YuliyanBogdanov, @standup1.So the solution is to use catalog.schema.table, and not LIVE.table, that's the key, you were right standup!But, you won't have the visibility of the tables on Bronze Pipeline, if you are on Si...
- 2 kudos
- 5021 Views
- 8 replies
- 0 kudos
Adding a new column triggers reprocessing of Auto Loader source table
I have a source table A in Unity Catalog. This table is constantly written to and is a streaming table.I also have another table B in Unity Catalog. This is a managed table with liquid clustering.Using Auto Loader I move new data from A to B using a ...
- 5021 Views
- 8 replies
- 0 kudos
- 0 kudos
change data feed might be a solution for you perhaps.https://docs.databricks.com/en/delta/delta-change-data-feed.html
- 0 kudos
- 935 Views
- 0 replies
- 0 kudos
Migration of Synapse Data bricks activity runs from 'All purpose cluster' to 'New job cluster'
Could someone please help me with the below question:1. How much time does the VM takes to deallocate the CPUs?2. Can we check how much quota is required to run a script?
- 935 Views
- 0 replies
- 0 kudos
- 2070 Views
- 1 replies
- 0 kudos
Workspace client creation error
Hi,We are trying to use Python SDK and create a workspace client using the following code:%pip install databricks-sdk --upgrade dbutils.library.restartPython()from databricks.sdk import WorkspaceClientw = WorkspaceClient()Here is the notebook: https:...
- 2070 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @databrick_usert , Hope you are doing well! Can you check the version of the SDK running in this notebook? If it's not an upgraded version then could you please try to upgrade the SDK version and then restart the python after the pip install? %p...
- 0 kudos
- 1514 Views
- 0 replies
- 0 kudos
delete non empty folders from workspace
Looking for a solution to delete 1000's of non-empty folders that I created wrongly while migration. (looking for solution that delete thoudands of folders created with below name)Folders have a path like "shared/db/..../..../..../yippy.dbc"
- 1514 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 1613 | |
| 771 | |
| 349 | |
| 286 | |
| 253 |
