- 2822 Views
- 2 replies
- 0 kudos
Can't enable CLI 2.1 on CI
Hi! this is my CI configuration, I added the databricks jobs configure --version=2.1 command but it stills showing this error, any idea of what can I be doing wrong?Error:Resetting Databricks Job with job_id 1036...WARN: Your CLI is configured to use...
- 2822 Views
- 2 replies
- 0 kudos
- 594 Views
- 3 replies
- 2 kudos
Resolved! Asset Bundles Job If Condition
HiI am trying to use Asset Bundles to deploy a job with email notifications active only on production environment.I tried to use such if statement but it does not seem to do anything.resources: jobs: Master_Load: name: Master Load e...
- 594 Views
- 3 replies
- 2 kudos
- 2 kudos
Okay i made it work with a variable, it was pretty easy. The variable accepted multiline text easily.Thanks!databricks.ymltargets: dev: mode: development ... variables: notifications: default: - user1@smth.com ...
- 2 kudos
- 417 Views
- 3 replies
- 0 kudos
What does DLT INITIALIZING phase do?
In Delta Live Tables, the INITIALIZING phase takes sometimes a minute, sometimes 5 minutes. I'd like to learn what is it doing in the background, and can it be optimized in any way.
- 417 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @antr, In Delta Live Tables (DLT), a feature of Databricks, the "Initializing" phase refers to the first step in the lifecycle of a DLT pipeline run. During this phase, the platform sets up the necessary resources, configurations, and dependencies...
- 0 kudos
- 245 Views
- 1 replies
- 0 kudos
Databricks as a "pure" data streaming software like Confluent
DearsI was wondering if anyone has leveraged Databricks as a "pure" data streaming software in place of Confluent, Flink, Kafka etc.I see the reference architectures placing Databricks on the data processing side mostly once data is made available by...
- 245 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @noorbasha534 ,It depends on what you're asking for. Kafka is primarily a messaging system, optimized for handling high-throughput, distributed message logs. Databricks can read from Kafka as a data source but doesn't replace Kafka's role in messa...
- 0 kudos
- 9479 Views
- 7 replies
- 0 kudos
Connect to Databricks from PowerApps
Hi All,Currently I trying to connect databricks Unity Catalog from Powerapps Dataflow by using spark connector specifying http url and using databricks personal access token as specified in below screenshot: I am able to connect but the issue is when...
- 9479 Views
- 7 replies
- 0 kudos
- 0 kudos
Do you have any firewall restriction? Maybe if the public IP are not allowed for your databricks workspace it is not allowing thew connection, besides that I don't have much clue on how to help. Would suggest open a support ticket.
- 0 kudos
- 311 Views
- 1 replies
- 0 kudos
how to get node and executor id and log
Hi Team,We have df has 70 M row count and we are calling an API for 6000 rows per set using df.repartition(rep_count).foreachpartition(func_name) so in func_name we are calling API post req for that partition but when we are trying to print/log somet...
- 311 Views
- 1 replies
- 0 kudos
- 0 kudos
any update on this???
- 0 kudos
- 201 Views
- 1 replies
- 1 kudos
Streaming Live Table - What is actually computed?
Can anyone please share in a DLT or structured streaming task, what group of rows are computed?Specific scenarios:1. when a streaming table A joining a delta table B. Is each of the minibatches in A joining the whole delta table? Does Spark compute t...
- 201 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @tliuzillow ,1. Stream-static Join: Each minibatch from the streaming table (A) is joined with the entire Delta table (B). 2. Stream-stream Join: Each minibatch from the streaming table(A) is joined with minibatch from the streaming table(B). Howe...
- 1 kudos
- 599 Views
- 4 replies
- 2 kudos
Resolved! Network bottleneck
Within a script, I noticed that the network connection between driver and the mounted network drives is often a huge bottleneck. It seems that the network through speed is unreasonable low for being an Azure Single node: Standard_DS12_v2 · DBR: 14.3....
- 599 Views
- 4 replies
- 2 kudos
- 2 kudos
You are right. I am creating 200 small files with the size of roughly 6 MB (in the quality system) and a few 100000s files in production. The partition is motivated by the original business need and further processing. Let me test with a the differen...
- 2 kudos
- 412 Views
- 1 replies
- 1 kudos
Resolved! Is there a way to use name prefixes for asset bundle deployments to prevent collisions
Hi, we are using asset bundles to deploy artefacts in Databricks. We use a service account and github workflows for that. Currently, we deploy in dev mode whenever a PR is created against the main branch of our gh repo. So far so good. However, we ar...
- 412 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Mathias_Peters ,Yes, it is possible to do so.1. Configure variable that will be added to the path.variables: feature_name: description: This will be added to the path2. Configure workspace root_path:workspace: root_path: /Shared/.bundle/...
- 1 kudos
- 304 Views
- 1 replies
- 0 kudos
post /api/2.0/sql/statements/sql statement id/cancel forbidden error in databricks
when i tried executing this requestpost /api/2.0/sql/statements/${SQL_STATEMENT_ID}/cancelI am getting forbidden error.how do we get access to execute this
- 304 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @slakshmanan ,The 403 Forbidden error usually means that the API request is not authorized, meaning you likely do not have the proper permissions or your authentication credentials are not being recognized. Here are 2 main reasons:1. Missing or In...
- 0 kudos
- 9251 Views
- 16 replies
- 27 kudos
Resolved! Do not received Databricks Certification: Fully Sponsored after order on Reward Store
Hi team.Would you please help check on my case?From 30-Nov I have placed an order "Databricks Certification: Fully Sponsored" on https://communitydatabricks.mybrightsites.com/ and after waiting 10 bussiness days. I still not receive that voucher.Is t...
- 9251 Views
- 16 replies
- 27 kudos
- 27 kudos
If you have ordered a fully sponsored Databricks Certification through a rewards store (e.g., an online rewards platform for employees or learners) and have not received it, there are several possible reasons and steps to resolve the issue:Possible R...
- 27 kudos
- 11627 Views
- 8 replies
- 2 kudos
Resolved! dbutils.fs.mv - 1 folder and 1 file with the same name and only move the folder
Hello!I am contacting you because of the following problem I am having:In an ADLS folder I have two items, a folder and an automatically generated Block blob file with the same name as the folder.I want to use the dbutils.fs.mv command to move the fo...
- 11627 Views
- 8 replies
- 2 kudos
- 2 kudos
What are the possible reasons for the generation of those extra files with same name with zero bytes of data?
- 2 kudos
- 680 Views
- 4 replies
- 0 kudos
how to cancel or kill a long running sql query from databricks python notebook.I have a long running
how to cancel or kill a long running sql query from databricks python notebook.I have a long running sql query in sql warehouse
- 680 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @slakshmanan ,To stop or interrupt a running notebook, select the interrupt button in the notebook toolbar. You can also select Run > Interrupt execution, or use the keyboard shortcut I I.https://learn.microsoft.com/en-us/azure/databricks/notebook...
- 0 kudos
- 4708 Views
- 9 replies
- 0 kudos
Schema migration of production delta tables
GoalWe would like to be in control of schema migrations of delta tables in all dev and production environments, and it must be automatically deployed.I anticipated this to be a common problem with a well-known standard solution. But unfortunately, I ...
- 4708 Views
- 9 replies
- 0 kudos
- 352 Views
- 3 replies
- 7 kudos
Resolved! Gold table for Analyst
Hi team,we are running data pipeline from bronze to gold, and another team need to refer the gold table. however the team doesn't have technical skills to query, and they would like to use the data on spreadsheets. Do we have any good work around in ...
- 352 Views
- 3 replies
- 7 kudos
- 7 kudos
It was easy to install. I'm not sure if this information is useful, but I'd like to share it for those who might be in the same situation. https://bricksheet.amukin.com/export-data-from-databricks-to-google-sheet
- 7 kudos