Hello,the costs regarding the databricks service from cost management in azure portal (45,869...) are not allligned with costs calculated from usage system table (75,34). The costs from the portal are filtered based on the desired period (usage_date ...
I am not being able to run jobs/runnow endpoint. I am getting an error asError fetching files: 403 - {"error_code":"PERMISSION_DENIED","message":"User xxxx-dxxxx-xxx-xxxx does not have Manage Run or Owner or Admin permissions on job 437174060919465",...
I have a very basic view with 3 inner joins that will only do a full refresh. Is there a limit to the number of joins you can have and still get an incremental refresh?"incrementalization_issues": [{"issue_type": "INCREMENTAL_PLAN_REJECTED_BY_COST_MO...
@GregTyndall Yes, the current limit is 2 by default. But we can increase up to 5 with the below flag added to the pipeline settings. pipelines.enzyme.numberOfJoinsThreshold 5
I am using delta live table and pub sub to ingest message from 30 different topics in parallel. I noticed that initialization time can be very long around 15 minutes. Does someone knows how to reduced initialization time in dlt ? Thanks You
Currently trying to refresh a Delta Live Table using a Full Refresh but an error keeps coming up saying that we have to use a shared cluster or a SQL warehouse. I've tried both a shared cluster and a SQL warehouse and the same error keeps coming up. ...
You are not using "No Isolation Shared" mode, right? Also, can you share the chunk of code that is causing the failure? Thanks, Louis.
How do I deploy or run a single job if I have 2 or more jobs defined in my asset bundle?$databricks bundle deploy job1 #does not work I do not see a flag to identify what job to run.
Hello,The Spark UI Simulator is not accessible since the last few days. I was able to refer to it last week, at https://www.databricks.training/spark-ui-simulator/index.html. I already have access to partner academy (if that is any relevant). <Error...
Just a short update: the request I raised was closed saying there is no active support contract with the org (from the email I used) to look into this. Perhaps someone else could try raising a request using the link above.
Hi all!I need to migrate multiple notebooks from one workspace to another. Is there any way to do it without using Git?Since Manual Import and Export is difficult to do for multiple notebooks and folders, need an alternate solution.Please reply as so...
Hello @HaripriyaP , @rabia_farooq check for the databricks cli documentation here - https://docs.databricks.com/aws/en/dev-tools/cli/commandsYou can work with databricks workflow <command_such_as_export> and move the notebooks/code files around.Cheer...
Hi,We are planning to migrate from Teradata to Databricks using Bladebridge. Going through various portals, I am not able to conclude the component that facilitates Data movement between Teradata and databricks.Please clarify end to end tool and acti...
I'm not aware if blade bridge has data movement tool handy with them.I don't see anything advertised by them though.Let me know if you find anything on this.
Hi everyone,I'm exploring ways to streamline my workflow in Databricks and could really use some expert advice. In my current setup, I have a job (named job_silver) with multiple tasks (e.g., task 1, task 2, task 3). When one of these tasks fails—say...
Hey @minhhung0507 - quick question - what is the cluster type you're using to run your workflow?I'm using a shared, interactive cluster, so I'm passing the parameter {'existing_cluster_id' : task['existing_cluster_id']}in the payload. This parameter ...
Hi Community.Since I wasn't able to find a way to set a job tag dynamically at runtime based on a parameter that is passed to the job, I was wondering if it is possible or if there is an equivalent way to do it.Thank you. Regards.
Based on the provided context, it appears that there isn't a direct way within Databricks to dynamically set job tags at runtime based on a parameter passed to the job. However, there are alternative approaches you can consider to work around this li...
How can I use Terraform to assign an external location to multiple workspaces?When I create an external location with Terraform, I do not see any option to directly link workspaces. it also only links to the workspace of the databricks profile that I...
Hi All, I am getting the below error while running an update query in a lookup activity using the delta lake linked service:ErrorCode=AzureDatabricksCommandError,Hit an error when running the command in Azure Databricks. Error details: <span class='a...
Hi @yashojha1995 EOL while scanning string literal hints that there might be a syntax error in the update query.could you share your update query here, and any other info such as how are you creating a Linked service to your delta lake? Does it mean ...
Hi I am query my Azure Databricks table using VB script/Simba Spark ODBC driver.While querying into the DB(lets say 'Select * from table_1') it is not returning any data. However while querying a limit (lets say 'Select TOP 10000 ID from table_1'), i...
Hello!The issue where a full table query in Azure Databricks via VB Script and Simba Spark ODBC returns no data, but TOP works, likely stems from the ODBC driver's inability to handle the large result set at once. Investigate and increase the driver'...
We have the following use case:We receive raw form of data from an application and that is ingested in the Iron Layer. The raw data is in the JSON FormatThe Bronze layer will the first level of transformation. The flattening of the JSON file happens ...
The whole medallion architecture is unnecesarily complex. Bronze, Silver, Gold. Some places I have seen people do - RAW , Trusted RAW , Silver, Trusted Silver, Gold
| User | Count |
|---|---|
| 1611 | |
| 768 | |
| 347 | |
| 286 | |
| 252 |
