cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
mickniz
by • Contributor
  • 13327 Views
  • 7 replies
  • 0 kudos

Connect to Databricks from PowerApps

Hi All,Currently I trying to connect databricks Unity Catalog from Powerapps Dataflow by using spark connector specifying http url and using databricks personal access token as specified in below screenshot: I am able to connect but the issue is when...

mickniz_0-1714487746554.png mickniz_1-1714487891958.png
Data Engineering
Reply
  • 13327 Views
  • 7 replies
  • 0 kudos
View Replies
Latest Reply
william_lenotis
New Contributor II
  • 0 kudos

Do you have any firewall restriction? Maybe if the public IP are not allowed for your databricks workspace it is not allowing thew connection, besides that I don't have much clue on how to help. Would suggest open a support ticket.

  • 0 kudos
6 More Replies
mayur_05
by • New Contributor II
  • 488 Views
  • 1 replies
  • 0 kudos

how to get node and executor id and log

Hi Team,We have df has 70 M row count and we are calling an API for 6000 rows per set using df.repartition(rep_count).foreachpartition(func_name) so in func_name we are calling API post req for that partition but when we are trying to print/log somet...

Data Engineering
Reply
  • 488 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
mayur_05
New Contributor II
  • 0 kudos

any update on this???

  • 0 kudos
tliuzillow
by • New Contributor
  • 307 Views
  • 1 replies
  • 1 kudos

Streaming Live Table - What is actually computed?

Can anyone please share in a DLT or structured streaming task, what group of rows are computed?Specific scenarios:1. when a streaming table A joining a delta table B. Is each of the minibatches in A joining the whole delta table? Does Spark compute t...

Data Engineering
Reply
  • 307 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
filipniziol
Contributor III
  • 1 kudos

Hi @tliuzillow ,1. Stream-static Join: Each minibatch from the streaming table (A) is joined with the entire Delta table (B). 2. Stream-stream Join: Each minibatch from the streaming table(A) is joined with minibatch from the streaming table(B). Howe...

  • 1 kudos
jenshumrich
by • Contributor
  • 831 Views
  • 4 replies
  • 2 kudos

Resolved! Network bottleneck

Within a script, I noticed that the network connection between driver and the mounted network drives is often a huge bottleneck. It seems that the network through speed is unreasonable low for being an Azure Single node: Standard_DS12_v2 · DBR: 14.3....

Data Engineering
Reply
  • 831 Views
  • 4 replies
  • 2 kudos
View Replies
Latest Reply
jenshumrich
Contributor
  • 2 kudos

You are right. I am creating 200 small files with the size of roughly 6 MB (in the quality system) and a few 100000s files in production. The partition is motivated by the original business need and further processing. Let me test with a the differen...

  • 2 kudos
3 More Replies
Mathias_Peters
by • Contributor
  • 750 Views
  • 1 replies
  • 1 kudos

Resolved! Is there a way to use name prefixes for asset bundle deployments to prevent collisions

Hi, we are using asset bundles to deploy artefacts in Databricks. We use a service account and github workflows for that. Currently, we deploy in dev mode whenever a PR is created against the main branch of our gh repo. So far so good. However, we ar...

Data Engineering
Reply
  • 750 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
filipniziol
Contributor III
  • 1 kudos

Hi @Mathias_Peters ,Yes, it is possible to do so.1. Configure variable that will be added to the path.variables: feature_name: description: This will be added to the path2. Configure workspace root_path:workspace: root_path: /Shared/.bundle/...

  • 1 kudos
slakshmanan
by • New Contributor III
  • 418 Views
  • 1 replies
  • 0 kudos

post /api/2.0/sql/statements/sql statement id/cancel forbidden error in databricks

when i tried executing this requestpost /api/2.0/sql/statements/${SQL_STATEMENT_ID}/cancelI am getting forbidden error.how do we get access to execute this

Data Engineering
Reply
  • 418 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
filipniziol
Contributor III
  • 0 kudos

Hi @slakshmanan ,The 403 Forbidden error usually means that the API request is not authorized, meaning you likely do not have the proper permissions or your authentication credentials are not being recognized. Here are 2 main reasons:1. Missing or In...

  • 0 kudos
Nhan_Nguyen
by • Valued Contributor
  • 12232 Views
  • 16 replies
  • 27 kudos

Resolved! Do not received Databricks Certification: Fully Sponsored after order on Reward Store

Hi team.Would you please help check on my case?From 30-Nov I have placed an order "Databricks Certification: Fully Sponsored" on https://communitydatabricks.mybrightsites.com/ and after waiting 10 bussiness days. I still not receive that voucher.Is t...

Data Engineering
Reply
  • 12232 Views
  • 16 replies
  • 27 kudos
View Replies
Latest Reply
domenichancock
New Contributor II
  • 27 kudos

If you have ordered a fully sponsored Databricks Certification through a rewards store (e.g., an online rewards platform for employees or learners) and have not received it, there are several possible reasons and steps to resolve the issue:Possible R...

  • 27 kudos
15 More Replies
SaraCorralLou
by • New Contributor III
  • 12921 Views
  • 8 replies
  • 2 kudos

Resolved! dbutils.fs.mv - 1 folder and 1 file with the same name and only move the folder

Hello!I am contacting you because of the following problem I am having:In an ADLS folder I have two items, a folder and an automatically generated Block blob file with the same name as the folder.I want to use the dbutils.fs.mv command to move the fo...

Data Engineering
Reply
  • 12921 Views
  • 8 replies
  • 2 kudos
View Replies
Latest Reply
deep_coder16
New Contributor II
  • 2 kudos

What are the possible reasons for the generation of those extra files with same name with zero bytes of data? 

  • 2 kudos
7 More Replies
slakshmanan
by • New Contributor III
  • 1298 Views
  • 4 replies
  • 0 kudos

how to cancel or kill a long running sql query from databricks python notebook.I have a long running

how to cancel or kill a long running sql query from databricks python notebook.I have a long running sql query in sql warehouse

Data Engineering
Reply
  • 1298 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
szymon_dybczak
Esteemed Contributor III
  • 0 kudos

Hi @slakshmanan ,To stop or interrupt a running notebook, select the interrupt button in the notebook toolbar. You can also select Run > Interrupt execution, or use the keyboard shortcut I I.https://learn.microsoft.com/en-us/azure/databricks/notebook...

  • 0 kudos
3 More Replies
BjarkeM
by • New Contributor II
  • 8115 Views
  • 9 replies
  • 0 kudos

Schema migration of production delta tables

GoalWe would like to be in control of schema migrations of delta tables in all dev and production environments, and it must be automatically deployed.I anticipated this to be a common problem with a well-known standard solution. But unfortunately, I ...

Data Engineering
Reply
  • 8115 Views
  • 9 replies
  • 0 kudos
View Replies
Latest Reply
worlordv
New Contributor II
  • 0 kudos

GitHub - liquibase/liquibase-databricks

  • 0 kudos
8 More Replies
KennethKnewman
by • New Contributor III
  • 490 Views
  • 3 replies
  • 7 kudos

Resolved! Gold table for Analyst

Hi team,we are running data pipeline from bronze to gold, and another team need to refer the gold table. however the team doesn't have technical skills to query, and they would like to use the data on spreadsheets. Do we have any good work around in ...

Data Engineering
Reply
  • 490 Views
  • 3 replies
  • 7 kudos
View Replies
Latest Reply
KennethKnewman
New Contributor III
  • 7 kudos

It was easy to install. I'm not sure if this information is useful, but I'd like to share it for those who might be in the same situation. https://bricksheet.amukin.com/export-data-from-databricks-to-google-sheet

  • 7 kudos
2 More Replies
mr_poola49
by • New Contributor III
  • 554 Views
  • 3 replies
  • 0 kudos

ADLS gen2 config issue

I am new to Azure Databricks. I am trying to access ADLS gen2 from Azure Databricks. I've set all the required configurations in the notebook but when I try to query the table using SPAR.SQL(), it is throwing exception "Failure to initialize configur...

Data Engineering
Reply
  • 554 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
mr_poola49
New Contributor III
  • 0 kudos

Issue is resolved! dropped the table from hive_metastore which is pointing to  ppeadlsg2  storage container and re-created it using prodadlsg2 storage.

  • 0 kudos
2 More Replies
-werners-
by • Esteemed Contributor III
  • 1229 Views
  • 3 replies
  • 2 kudos

Resolved! asset bundles and compute policies

Did anyone succeed in using already existing compute policies (created using the UI) in asset bundles for creating a job?I defined the policy_id in the resources/job yml for the job_cluster, but when deploying I get errors saying spark version is not...

Data Engineering
Reply
  • 1229 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 2 kudos

So I figured it out.You can actually refer existing cluster policies, but I made the mistake thinking all cluster config was added automatically by doing that.In fact you still have to add some cluster config in the resources yaml:- spark_version- sp...

  • 2 kudos
2 More Replies
Erfan
by • New Contributor III
  • 897 Views
  • 3 replies
  • 3 kudos

Resolved! Liquid Clustering With more than 4 columns

Hi there,I’m trying to join a small table (a few million records) with a much larger table (around 1 TB in size, containing a few billion records).The small table isn’t quite small enough to use Broadcast. Additionally, our join clause involves more ...

Data Engineering
Reply
  • 897 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
filipniziol
Contributor III
  • 3 kudos

Hi @Erfan ,What you can do is to create an additional column that concatenates the values of multiple columns and then apply Liquid Clustering on that new column.

  • 3 kudos
2 More Replies
Constantine
by • Contributor III
  • 6110 Views
  • 5 replies
  • 1 kudos

Resolved! How to use Databricks Query History API (REST API)

I have setup authentication using this page https://docs.databricks.com/sql/api/authentication.html and run curl -n -X GET https://<databricks-instance>.cloud.databricks.com/api/2.0/sql/history/queriesTo get history of all sql endpoint queries, but I...

Data Engineering
Reply
  • 6110 Views
  • 5 replies
  • 1 kudos
View Replies
Latest Reply
yegorski
New Contributor III
  • 1 kudos

Here's how to query with databricks-sdk-py (working code). I had a frustrating time doing it with vanilla python + requests/urllib and couldn't figure it out. import datetime import os from databricks.sdk import WorkspaceClient from databricks.sdk.se...

  • 1 kudos
4 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels