cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
philHarasz
by • New Contributor
  • 28 Views
  • 1 replies
  • 0 kudos

Writing a small pyspark dataframe to a table is taking a very long time

My experience with Databricks pyspark up to this point has always been to execute a SQL query against existing Databricks tables, then write the resulting pyspark dataframe into a new table. For the first time, I am now getting data via an API which ...

Data Engineering
Reply
  • 28 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
MariuszK
Contributor II
  • 0 kudos

Can you share the code? You need to remember that spark uses lazy evaluation so it can give you impression that code works fast and saving works slowly because a code is executed when you it hit an action.

  • 0 kudos
akshay716
by • Visitor
  • 37 Views
  • 1 replies
  • 0 kudos

How to create Service Principal and access APIs like clusters list without adding to admin group

I have created a Databricks Managed Service Principal and trying to access the APIs like clusters list, job lists pipelines but without adding it to admin group I am getting empty list in response. There are other ways to get clusters by adding polic...

Data Engineering
Reply
  • 37 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hi @akshay716, You can assign specific permissions directly to the service principal without granting it broader admin access

  • 0 kudos
tebodelpino1234
by • Visitor
  • 26 Views
  • 0 replies
  • 0 kudos

can view allow_expectations_col in unit catalog

I am developing a dlt that manages expectations and it works correctly.but I need to see the columns__DROP_EXPECTATIONS_COL__MEETS_DROP_EXPECTATIONS__ALLOW_EXPECTATIONS_COLin the unified catalog, I can see them in the delta table that the dlt generat...

tebodelpino1234_0-1739993377613.png tebodelpino1234_1-1739993707990.png tebodelpino1234_2-1739993775604.png
Data Engineering
Reply
  • 26 Views
  • 0 replies
  • 0 kudos
Ramonrcn
by • New Contributor III
  • 26 Views
  • 0 replies
  • 0 kudos

Cant read/write tables with shared cluster

Hi!I have a pipeline that i cant execute sucessfully in a shared cluster. Basically i read a query from multiple sources on my databricks instance, including streaming tables (thats the reason i have to use a shared cluster).But when comes to the par...

Data Engineering
Reply
  • 26 Views
  • 0 replies
  • 0 kudos
Sergio_Linares
by • Visitor
  • 23 Views
  • 1 replies
  • 0 kudos

When Sign in databricks partner-academy i can not see the courses

Dear partner academy team, I am writing to report an issue I am experiencing when trying to access the partner academy courses. Despite using my credentials, I am unable to view any of the courses. Could you please look into this and assist me in res...

Data Engineering
Reply
  • 23 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Advika
Databricks Employee
  • 0 kudos

Hello @Sergio_Linares! Please file a ticket with the Databricks support team to get assistance with this issue.

  • 0 kudos
DataEnginerrOO1
by • Visitor
  • 82 Views
  • 4 replies
  • 0 kudos

Access for delta lake with serverless

I have an issue when trying to use the command display(dbutils.fs.ls("abfss://test@test.dfs.core.windows.net")). When I execute the command on my personal cluster, it works, and I can see the files. Before that, I set the following configurations:spa...

Data Engineering
Reply
  • 82 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
Rjdudley
Valued Contributor II
  • 0 kudos

Can your serverless compute access any storage in that storage account?  Something else to check is if your NCC is configured correctly: Configure private connectivity from serverless compute - Azure Databricks | Microsoft Learn.  However, if your se...

  • 0 kudos
3 More Replies
KSB
by • Visitor
  • 21 Views
  • 0 replies
  • 0 kudos

databricks

Hi Team,Having excel file in sharepoint folder, and has to insert excel data into SQL table from databricks notebook . can i have clear steps on it. Dont have access to Azure Active Directory.  can anyone gives solution without using AZURE Active Dir...

Data Engineering
Reply
  • 21 Views
  • 0 replies
  • 0 kudos
BabakBastan
by • New Contributor
  • 18 Views
  • 0 replies
  • 0 kudos

Missing Delta-live-Table in hive-metastore catalog

Hi experts,I defined my delta table in an external location as following:%sqlCREATE OR REFRESH STREAMING TABLE pumpdata (Body string,EnqueuedTimeUtc string,SystemProperties string,_rescued_data string,Properties string)USING DELTALOCATION 'abfss://md...

Bild1.png Bild2.png Bild3.png Bild4.png
Data Engineering
Delta Live Tables
Reply
  • 18 Views
  • 0 replies
  • 0 kudos
mkEngineer
by • New Contributor III
  • 29 Views
  • 2 replies
  • 0 kudos

How to Version & Deploy Databricks Workflows with Azure DevOps (CI/CD)?

Hi everyone,I’m trying to set up versioning and CI/CD for my Databricks workflows using Azure DevOps and Git. While I’ve successfully versioned notebooks in a Git repo, I’m struggling with handling workflows (which define orchestration, dependencies,...

Data Engineering
Reply
  • 29 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
mkEngineer
New Contributor III
  • 0 kudos

As of now, my current approach is to manually copy/paste YAMLs across workspaces and version them using Git/Azure DevOps by saving them as DBFS files. The CD process is then handled using Databricks DBFS File Deployment by Data Thirst Ltd.While this ...

  • 0 kudos
1 More Replies
BillBishop
by • New Contributor II
  • 68 Views
  • 2 replies
  • 0 kudos

DAB for_each_task python wheel fail

using python_wheel_wrapper experimental true allows me to use python_wheel_task on an older cluster.However, if I embed the python_wheel_task in a for_each_task it fails at runtime with: "Library installation failed for library due to user error.  Er...

Data Engineering
Reply
  • 68 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hi @BillBishop, I will check on this internally as outcome does not seem to be correct. If possible, upgrade your cluster to DBR 14.1 or later. This would resolve the issue without relying on the experimental feature

  • 0 kudos
1 More Replies
Yuppp
by • Visitor
  • 22 Views
  • 0 replies
  • 0 kudos

Need help with setting up ForEach task in Databricks

Hi everyone,I have a workflow involving two notebooks: Notebook A and Notebook B. At the end of Notebook A, we generate a variable number of files, let's call it N. I want to run Notebook B for each of these N files.I know Databricks has a Foreach ta...

For Each.jpg Task.jpg
Data Engineering
ForEach
Workflows
Reply
  • 22 Views
  • 0 replies
  • 0 kudos
rushi29
by • New Contributor III
  • 1668 Views
  • 5 replies
  • 0 kudos

sparkContext in Runtime 15.3

Hello All, Our Azure databricks cluster is running under "Legacy Shared Compute" policy with 15.3 runtime. One of the python notebooks is used to connect to an Azure SQL database to read/insert data. The following snippet of code is responsible for r...

Data Engineering
Reply
  • 1668 Views
  • 5 replies
  • 0 kudos
View Replies
Latest Reply
jayct
New Contributor II
  • 0 kudos

@rushi29 @GangsterI ended up implementing pyodbc with the mssql driver using init scripts.Spark context is no longer usable in shared compute so that was the only approach we could take. 

  • 0 kudos
4 More Replies
ewe
by • Visitor
  • 28 Views
  • 0 replies
  • 0 kudos

Databricks apps (streamlit) not able to install python libs

So, I have a databricks streamlit app that is not able to install any python lib defined on the requirements.txt.Issue is not specific to one lib, tried other ones but no python lib can be installed. Anyone with similar issue to help ? [2025-02-19 10...

Data Engineering
Reply
  • 28 Views
  • 0 replies
  • 0 kudos
p_romm
by • Visitor
  • 30 Views
  • 1 replies
  • 0 kudos

Structured Streaming writeStream - Query is no longer active causes task to fail

Hi, I execute readStream/writeStream in workflow task. Write stream uses .trigger(availableNow=True) option. After writeStream I'm waiting query to finish with query.awaitTermination(). However from time to time, pipeline ends with "Query <id> is no ...

Data Engineering
Reply
  • 30 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hello @p_romm  - Are you using serverless compute?

  • 0 kudos
ila-de
by • New Contributor II
  • 225 Views
  • 7 replies
  • 1 kudos

Resolved! databricks workspace import_dir not working without any failure message

Morning everyone!I`m trying to copy from the repo into the databricks workspace all the notebooks. I`m using the command: databricks workspace import_dir . /Shared/Notebooks, it will just print all the info regarding the Workspace API.If I launch dat...

Data Engineering
Reply
  • 225 Views
  • 7 replies
  • 1 kudos
View Replies
Latest Reply
ila-de
New Contributor II
  • 1 kudos

Hi all,I`ve disinstalled and installed again databricks-cli and now worked.Is not a real solution but still it worked after one week...

  • 1 kudos
6 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels