- 2230 Views
- 2 replies
- 1 kudos
[STREAMING_TABLE_OPERATION_NOT_ALLOWED.REQUIRES_SHARED_COMPUTE]
Currently trying to refresh a Delta Live Table using a Full Refresh but an error keeps coming up saying that we have to use a shared cluster or a SQL warehouse. I've tried both a shared cluster and a SQL warehouse and the same error keeps coming up. ...
- 2230 Views
- 2 replies
- 1 kudos
- 1 kudos
You are not using "No Isolation Shared" mode, right? Also, can you share the chunk of code that is causing the failure? Thanks, Louis.
- 1 kudos
- 555 Views
- 6 replies
- 3 kudos
Spark UI Simulator Not Accessible
Hello,The Spark UI Simulator is not accessible since the last few days. I was able to refer to it last week, at https://www.databricks.training/spark-ui-simulator/index.html. I already have access to partner academy (if that is any relevant). <Error...
- 555 Views
- 6 replies
- 3 kudos
- 3 kudos
Just a short update: the request I raised was closed saying there is no active support contract with the org (from the email I used) to look into this. Perhaps someone else could try raising a request using the link above.
- 3 kudos
- 817 Views
- 3 replies
- 0 kudos
Multiple Notebooks Migration from one workspace to another without using Git.
Hi all!I need to migrate multiple notebooks from one workspace to another. Is there any way to do it without using Git?Since Manual Import and Export is difficult to do for multiple notebooks and folders, need an alternate solution.Please reply as so...
- 817 Views
- 3 replies
- 0 kudos
- 0 kudos
Hello @HaripriyaP , @rabia_farooq check for the databricks cli documentation here - https://docs.databricks.com/aws/en/dev-tools/cli/commandsYou can work with databricks workflow <command_such_as_export> and move the notebooks/code files around.Cheer...
- 0 kudos
- 101 Views
- 3 replies
- 1 kudos
Data Migration using Bladebridge
Hi,We are planning to migrate from Teradata to Databricks using Bladebridge. Going through various portals, I am not able to conclude the component that facilitates Data movement between Teradata and databricks.Please clarify end to end tool and acti...
- 101 Views
- 3 replies
- 1 kudos
- 1 kudos
I'm not aware if blade bridge has data movement tool handy with them.I don't see anything advertised by them though.Let me know if you find anything on this.
- 1 kudos
- 246 Views
- 11 replies
- 2 kudos
API for Restarting Individual Failed Tasks within a Job?
Hi everyone,I'm exploring ways to streamline my workflow in Databricks and could really use some expert advice. In my current setup, I have a job (named job_silver) with multiple tasks (e.g., task 1, task 2, task 3). When one of these tasks fails—say...
- 246 Views
- 11 replies
- 2 kudos
- 2 kudos
Hey @minhhung0507 - quick question - what is the cluster type you're using to run your workflow?I'm using a shared, interactive cluster, so I'm passing the parameter {'existing_cluster_id' : task['existing_cluster_id']}in the payload. This parameter ...
- 2 kudos
- 39 Views
- 0 replies
- 0 kudos
How can I use Terraform to assign an external location to multiple workspaces?
How can I use Terraform to assign an external location to multiple workspaces?When I create an external location with Terraform, I do not see any option to directly link workspaces. it also only links to the workspace of the databricks profile that I...
- 39 Views
- 0 replies
- 0 kudos
- 140 Views
- 1 replies
- 0 kudos
Error while running update statement using delta lake linked service through ADF
Hi All, I am getting the below error while running an update query in a lookup activity using the delta lake linked service:ErrorCode=AzureDatabricksCommandError,Hit an error when running the command in Azure Databricks. Error details: <span class='a...
- 140 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @yashojha1995 EOL while scanning string literal hints that there might be a syntax error in the update query.could you share your update query here, and any other info such as how are you creating a Linked service to your delta lake? Does it mean ...
- 0 kudos
- 2146 Views
- 5 replies
- 3 kudos
Resolved! How to decide on creating views vs Tables in Gold layer?
We have the following use case:We receive raw form of data from an application and that is ingested in the Iron Layer. The raw data is in the JSON FormatThe Bronze layer will the first level of transformation. The flattening of the JSON file happens ...
- 2146 Views
- 5 replies
- 3 kudos
- 3 kudos
The whole medallion architecture is unnecesarily complex. Bronze, Silver, Gold. Some places I have seen people do - RAW , Trusted RAW , Silver, Trusted Silver, Gold
- 3 kudos
- 73 Views
- 0 replies
- 0 kudos
Query limiting to only 10000 rows
Hi I am query my Azure Databricks table using VB script/Simba Spark ODBC driver.While querying into the DB(lets say 'Select * from table_1') it is not returning any data. However while querying a limit (lets say 'Select TOP 10000 ID from table_1'), i...
- 73 Views
- 0 replies
- 0 kudos
- 645 Views
- 4 replies
- 0 kudos
Resolved! [Databricks Asset Bundles] Triggering Delta Live Tables
I would like to know how to schedule a DLT pipeline using DAB's.I'm trying to trigger a Delta Live Table pipeline using Databricks Asset Bundles. Below is my YAML code:resources: pipelines: data_quality_pipelines: name: data_quality_pipeline...
- 645 Views
- 4 replies
- 0 kudos
- 0 kudos
As of now, Databricks Asset Bundles do not support direct scheduling of DLT pipelines using cron expressions within the bundle configuration. Instead, you can achieve scheduling by creating a Databricks job that triggers the DLT pipeline and then sch...
- 0 kudos
- 224 Views
- 4 replies
- 1 kudos
DLT Pipeline & Automatic Liquid Clustering Syntax
Hi everyone,I noticed Databricks recently released the automatic liquid clustering feature, which looks very promising. I'm currently implementing a DLT pipeline and would like to leverage this new functionality.However, I'm having trouble figuring o...
- 224 Views
- 4 replies
- 1 kudos
- 1 kudos
Hey @HoussemBL You're correct about DLT not support Auto LC. You can assign any columns in the cluster_by but if you set it to auto, it will throw an error complaining about auto not being present in the list of columns.Maybe, altering thee table to ...
- 1 kudos
- 319 Views
- 5 replies
- 3 kudos
Databricks streamlit app for data ingestion in a table
I am using this code in a notebook to save data row on table. And it is working perfectly. And now I am using the same function to save data from a chatbot in streamlit chatbot application of databricks and I am getting error for ERROR ##############...
- 319 Views
- 5 replies
- 3 kudos
- 3 kudos
Hi @manish_tanwar how can we work with streamlit apps in databricks , i have a use case where i want to ingest data from different csv files and ingest it into delta tables
- 3 kudos
- 185 Views
- 3 replies
- 0 kudos
Serverless Compute
Hi Team,We are using Azure Databricks Serverless Compute to execute workflows and notebooks. My question is :Does serverless compute support Maven library installations?I appreciate any insights or suggestions you might have. Thanks in advance for yo...
- 185 Views
- 3 replies
- 0 kudos
- 0 kudos
So, it appears that the there is conflicting documentation on this topic. I checked with our internal documention and what I found was that you CANNOT install JDBC or ODBC drivers on Serverless. See limitations here: https://docs.databricks.com/aws...
- 0 kudos
- 3386 Views
- 6 replies
- 5 kudos
Resolved! How to get git commit ID of the repository the script runs on?
I have a script in a repository on DataBricks. The script should log the current git commit ID of the repository. How can that be implemented? I tried various command, for example: result = subprocess.run(['git', 'rev-parse', 'HEAD'], stdout=subproce...
- 3386 Views
- 6 replies
- 5 kudos
- 5 kudos
Here is a version of @vr 's solution that can be run from any folder within the rep. It uses regex to extract the root from the path in the form of \Repos\<username>\<some-repo:import os import re from databricks.sdk import WorkspaceClient w = Worksp...
- 5 kudos
- 117 Views
- 3 replies
- 0 kudos
Default Code generated by Bladebridge converter
Hello all ,1. What is the default code generated by Bladebridge converter.for eg : When we migrate Teradat, Oracle to Databricks using Bladebridge whats the default code base.2.If the generated code is PYSPARK, do I have any control over the generate...
- 117 Views
- 3 replies
- 0 kudos
- 0 kudos
Hello @Vasu_Kumar_T - We've used Bladebridge to convert from Redshift to Databricks. Bladebridge can definetly convert to Spark SQL, not sure about Scala Spark though.
- 0 kudos
| User | Count |
|---|---|
| 1611 | |
| 768 | |
| 348 | |
| 286 | |
| 252 |
