cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

277745
by New Contributor
  • 332 Views
  • 1 replies
  • 0 kudos

Pandas_Udod max batch size not working in notebook

Hello I am trying to set max batch size for pandas-udf in Databricks notebook, but in my tests it doesn’t have any effect on size. spark.conf.set("spark.sql.execution.arrow.enabled", "true")spark.conf.set('spark.sql.execution.arrow.maxRecordsPerBatch...

  • 332 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @277745, It seems you’re working with Pandas UDF in a Databricks Notebook and trying to set the maximum batch size. Let’s address your query: Setting Max Batch Size for Pandas UDF: You’ve already taken the right steps by configuring the follo...

  • 0 kudos
Surajv
by New Contributor III
  • 365 Views
  • 3 replies
  • 0 kudos

Limit the scope of workspace level access token to access only specific REST APIs of Databricks

Hi Community, Is there a way to limit the scope of workspace level token to hit only certain REST APIs of Databricks.In short, Once we generate a workspace level token following this doc. Link: https://docs.databricks.com/en/dev-tools/auth/oauth-m2m....

  • 365 Views
  • 3 replies
  • 0 kudos
Latest Reply
Surajv
New Contributor III
  • 0 kudos

 <Replied to previous message as response to @Kaniz's answer> 

  • 0 kudos
2 More Replies
Sardenberg
by New Contributor II
  • 453 Views
  • 2 replies
  • 0 kudos

How list all USERs FROM a especific GROUP USING SQL?

I want to list, using sql editor, all users name from a specific group.Reading documentation, I only learned how to show the groups or the users, using simples filters, like:SHOW GROUPS LIKE '*XPTO*';SHOW GROUPS WITH USER `test@gmail.com`SHOW USERS L...

  • 453 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @Sardenberg,  To retrieve a list of users from a specific group using SQL, you can follow these steps:   Assumptions: Let’s assume you have three tables: USERS, GROUPS, and GROUP_USERS.The USERS table contains user information.The GROUPS table con...

  • 0 kudos
1 More Replies
Mal
by New Contributor
  • 533 Views
  • 1 replies
  • 0 kudos

FileNotFoundError: [Errno 2] No such file or directory: 'ffmpeg'

import whisperimport ffmpegmodel = whisper.load_model("base")transcription = model.transcribe("dbfs:/FileStore/Call_Center_Conversation__03.mp3")print(transcription["text"])FileNotFoundError: [Errno 2] No such file or directory: 'ffmpeg'I have import...

  • 533 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @Mal, It appears you're facing a FileNotFoundError with ffmpeg while using OpenAI Whisper. To troubleshoot, first try installing ffmpeg as a Python module with `pip install ffmpeg` in your Python environment. Next, ensure ffmpeg is accessible by r...

  • 0 kudos
db_allrails
by New Contributor II
  • 2989 Views
  • 2 replies
  • 1 kudos

Resolved! Configuring NCC does not show option to add private endpoints

Hi!I am following this guide: https://learn.microsoft.com/en-us/azure/databricks/security/network/serverless-network-security/serverless-private-linkHowever in Step 3: Create private endpoint rules, number 6 there is no option for me to Add a private...

  • 2989 Views
  • 2 replies
  • 1 kudos
Latest Reply
db_allrails
New Contributor II
  • 1 kudos

@saikumar246  you were correct. It was really super easy to set up and works flawlessly! Good job dev-team!

  • 1 kudos
1 More Replies
TestuserAva
by New Contributor II
  • 1494 Views
  • 7 replies
  • 2 kudos

Getting HTML sign I page as api response from databricks api with statuscode 200

Response:<!doctype html><html><head>    <meta charset="utf-8" />    <meta http-equiv="Content-Language" content="en" />    <title>Databricks - Sign In</title>    <meta name="viewport" content="width=960" />    <link rel="icon" type="image/png" href="...

TestuserAva_0-1701165195616.png
  • 1494 Views
  • 7 replies
  • 2 kudos
Latest Reply
SJR
New Contributor III
  • 2 kudos

Hello @Abhishek10745 It was just like you said! We have a completely private instance of Databricks and the DevOps Pipeline that I was using didin't have access to the private vnet. Switching pools solved the problem. Thanks for all the help!

  • 2 kudos
6 More Replies
B_J_Innov
by New Contributor III
  • 543 Views
  • 2 replies
  • 0 kudos

Make API Call to run job

Hi everyone,I want to trigger a run for a job using API Call.Here's my code"""import shleximport subprocessdef call_curl(curl):args = shlex.split(curl)process = subprocess.Popen(args, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)stdout...

  • 543 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @B_J_Innov, To resolve the "Unauthorized" error when triggering a Databricks job via the API, ensure you authenticate using a personal access token (PAT) and include it as the `Bearer` token in your request headers. Verify that your API endpoint i...

  • 0 kudos
1 More Replies
BillGuyTheScien
by New Contributor II
  • 411 Views
  • 1 replies
  • 0 kudos

Resolved! combining accounts

I have an AWS based databricks account with a few workspaces and an Azure Databricks workspace.  How do I combine them into one account?I am particularly interested in setting up a single billing drop with all my Databricks costs.  

  • 411 Views
  • 1 replies
  • 0 kudos
Latest Reply
AlliaKhosla
New Contributor III
  • 0 kudos

Hi @BillGuyTheScien  Greetings! Currently, we do not have such a feature to combine multiple cloud usage into a single account. We do have a feature request for the same and it is considered for future. Currently, there is no ETA on that. You can bro...

  • 0 kudos
anushajalesh28
by New Contributor II
  • 1449 Views
  • 2 replies
  • 1 kudos

Catalog issue

When i was trying to create catalog i got an error saying to mention azure storage account and storage container in the following query -CREATE CATALOG IF NOT EXISTS Databricks_Anu_Jal_27022024MANAGED LOCATION 'abfss://<databricks-workspace-stack-anu...

Get Started Discussions
Azure Databricks
  • 1449 Views
  • 2 replies
  • 1 kudos
Latest Reply
Kaniz
Community Manager
  • 1 kudos

Hi @anushajalesh28, To create a catalog in Azure Databricks, you need to specify the Azure storage account and storage container in the MANAGED LOCATION clause.    Let’s break down the query:   CREATE CATALOG IF NOT EXISTS Databricks_Anu_Jal_270220...

  • 1 kudos
1 More Replies
Ariaa
by New Contributor II
  • 464 Views
  • 2 replies
  • 0 kudos

How to overwrite git_source configuration in Asset Bundles

I'm using DAB to deploy a "jobs" resource into Databeicks and into  two environments: "dev" and "prod". I pull the notebooks from a remote git repository using "git_resource", and defined the default job to use a tag to find which version to pull. Ho...

  • 464 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @Ariaa, It sounds like you’re working with Databricks Asset Bundles (DAB) to deploy jobs into different environments. Let’s address your specific scenario. In your configuration, you’ve defined a default job that uses a tag to determine which ...

  • 0 kudos
1 More Replies
nan
by New Contributor II
  • 1151 Views
  • 1 replies
  • 0 kudos

TIMEZONE

Can I get some help from Databricks to help me understand how those timestamps being interpreted? Some are really confusing me. I have timestamp coming into AWS Databricks as String type. And the string timestamp is represented in UTC. I ran below qu...

  • 1151 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @nan,  Let’s dive into the intricacies of timestamp interpretation in Databricks on AWS. Timestamp Type in Databricks: A timestamp in Databricks represents an absolute point in time, comprising values for year, month, day, hour, minute, and se...

  • 0 kudos
Surajv
by New Contributor III
  • 565 Views
  • 1 replies
  • 0 kudos

Run spark code in notebook by setting spark conf instead of databricks connect configure in runtime

Hi community, I wanted to understand if there is a way to pass config values to spark session in runtime than using databricks-connect configure to run spark code. One way I found out is given here: https://stackoverflow.com/questions/63088121/config...

  • 565 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @Surajv, To pass configuration values to a Spark session at runtime in PySpark without relying on the Databricks Connect configuration, you can access and set Spark configuration parameters programmatically. First, retrieve the current Spark conte...

  • 0 kudos
Flachboard84
by New Contributor II
  • 2358 Views
  • 4 replies
  • 1 kudos

sparkR.session

Why might this be erroring out? My understanding is that SparkR is built into Databricks.Code:library(SparkR, include.only=c('read.parquet', 'collect'))sparkR.session() Error:Error in sparkR.session(): could not find function "sparkR.session"

  • 2358 Views
  • 4 replies
  • 1 kudos
Latest Reply
Flachboard84
New Contributor II
  • 1 kudos

It happens with any code; even something as simple as...x <- 2 + 2

  • 1 kudos
3 More Replies
Sujitha
by Community Manager
  • 511 Views
  • 0 replies
  • 0 kudos

🌟 Welcome Newcomers! 🌟

Hello and welcome to our wonderful Community!Whether you are here by chance or intention, we're thrilled to have you join us. Before you dive into the plethora of discussions and activities happening here, we'd love to get to know you better! ...

  • 511 Views
  • 0 replies
  • 0 kudos
Data_Engineer3
by Contributor II
  • 682 Views
  • 4 replies
  • 0 kudos

Delete databricks community post

Hi All,If in case if I make any mistake in my previous post in the databricks community, how can delete the post which I was posted and got replied on the same. Is this possible to delete the old post which was already replied by someone else.Thanks,

  • 682 Views
  • 4 replies
  • 0 kudos
Latest Reply
Data_Engineer3
Contributor II
  • 0 kudos

I need to delete my old post which contain the details which should not be shared.https://community.databricks.com/t5/data-engineering/need-to-define-the-struct-and-array-of-struct-field-colum-in-the/m-p/58131#M31022 

  • 0 kudos
3 More Replies
Labels
Top Kudoed Authors