- 7011 Views
- 1 replies
- 0 kudos
How to detect if running in a workflow job?
Hi there,what's the best way to differentiate in what environment my Spark session is running? Locally I develop with databricks-connect's DatabricksSession, but that doesn't work when running a workflow job which requires SparkSession.getOrCreate()....
- 7011 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks, dbutils.notebook.getContext does indeed contain information about the job run.
- 0 kudos
- 2530 Views
- 2 replies
- 0 kudos
Limit the scope of workspace level access token to access only specific REST APIs of Databricks
Hi Community, Is there a way to limit the scope of workspace level token to hit only certain REST APIs of Databricks.In short, Once we generate a workspace level token following this doc. Link: https://docs.databricks.com/en/dev-tools/auth/oauth-m2m....
- 2530 Views
- 2 replies
- 0 kudos
- 9525 Views
- 2 replies
- 1 kudos
Resolved! Configuring NCC does not show option to add private endpoints
Hi!I am following this guide: https://learn.microsoft.com/en-us/azure/databricks/security/network/serverless-network-security/serverless-private-linkHowever in Step 3: Create private endpoint rules, number 6 there is no option for me to Add a private...
- 9525 Views
- 2 replies
- 1 kudos
- 1 kudos
@saikumar246 you were correct. It was really super easy to set up and works flawlessly! Good job dev-team!
- 1 kudos
- 3784 Views
- 0 replies
- 0 kudos
FileNotFoundError: [Errno 2] No such file or directory: 'ffmpeg'
import whisperimport ffmpegmodel = whisper.load_model("base")transcription = model.transcribe("dbfs:/FileStore/Call_Center_Conversation__03.mp3")print(transcription["text"])FileNotFoundError: [Errno 2] No such file or directory: 'ffmpeg'I have import...
- 3784 Views
- 0 replies
- 0 kudos
- 3298 Views
- 1 replies
- 0 kudos
Make API Call to run job
Hi everyone,I want to trigger a run for a job using API Call.Here's my code"""import shleximport subprocessdef call_curl(curl):args = shlex.split(curl)process = subprocess.Popen(args, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)stdout...
- 3298 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi everyone,I want to trigger a run for a job using API Call.Here's my code"""import shleximport subprocessdef call_curl(curl):args = shlex.split(curl)process = subprocess.Popen(args, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)stdout...
- 0 kudos
- 2743 Views
- 1 replies
- 0 kudos
Resolved! combining accounts
I have an AWS based databricks account with a few workspaces and an Azure Databricks workspace. How do I combine them into one account?I am particularly interested in setting up a single billing drop with all my Databricks costs.
- 2743 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @BillGuyTheScien Greetings! Currently, we do not have such a feature to combine multiple cloud usage into a single account. We do have a feature request for the same and it is considered for future. Currently, there is no ETA on that. You can bro...
- 0 kudos
- 3872 Views
- 1 replies
- 0 kudos
Catalog issue
When i was trying to create catalog i got an error saying to mention azure storage account and storage container in the following query -CREATE CATALOG IF NOT EXISTS Databricks_Anu_Jal_27022024MANAGED LOCATION 'abfss://<databricks-workspace-stack-anu...
- 3872 Views
- 1 replies
- 0 kudos
- 0 kudos
When i was trying to create catalog i got an error saying to mention azure storage account and storage container in the following query -CREATE CATALOG IF NOT EXISTS Databricks_Anu_Jal_27022024MANAGED LOCATION 'abfss://<databricks-workspace-stack-anu...
- 0 kudos
- 4843 Views
- 0 replies
- 0 kudos
Run spark code in notebook by setting spark conf instead of databricks connect configure in runtime
Hi community, I wanted to understand if there is a way to pass config values to spark session in runtime than using databricks-connect configure to run spark code. One way I found out is given here: https://stackoverflow.com/questions/63088121/config...
- 4843 Views
- 0 replies
- 0 kudos
- 3930 Views
- 4 replies
- 1 kudos
sparkR.session
Why might this be erroring out? My understanding is that SparkR is built into Databricks.Code:library(SparkR, include.only=c('read.parquet', 'collect'))sparkR.session() Error:Error in sparkR.session(): could not find function "sparkR.session"
- 3930 Views
- 4 replies
- 1 kudos
- 2040 Views
- 0 replies
- 0 kudos
🌟 Welcome Newcomers! 🌟
Hello and welcome to our wonderful Community!Whether you are here by chance or intention, we're thrilled to have you join us. Before you dive into the plethora of discussions and activities happening here, we'd love to get to know you better! ...
- 2040 Views
- 0 replies
- 0 kudos
- 4685 Views
- 0 replies
- 0 kudos
TIMEZONE
Can I get some help from Databricks to help me understand how those timestamps being interpreted? Some are really confusing me. I have timestamp coming into AWS Databricks as String type. And the string timestamp is represented in UTC. I ran below qu...
- 4685 Views
- 0 replies
- 0 kudos
- 7390 Views
- 1 replies
- 0 kudos
Syntax of UPDATE Command in DataBricks
Hi All,I am testing the sql generated by our ETL software to see if it can run on data bricks SQL which I believe is Delta Tables underneath. This is the statement we are testing. As far as I can tell from the manual the from clause is not supported ...
- 7390 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi All,I am testing the sql generated by our ETL software to see if it can run on data bricks SQL which I believe is Delta Tables underneath. This is the statement we are testing. As far as I can tell from the manual the from clause is not supported ...
- 0 kudos
- 2363 Views
- 0 replies
- 0 kudos
Pandas_Udod max batch size not working in notebook
Hello I am trying to set max batch size for pandas-udf in Databricks notebook, but in my tests it doesn’t have any effect on size. spark.conf.set("spark.sql.execution.arrow.enabled", "true")spark.conf.set('spark.sql.execution.arrow.maxRecordsPerBatch...
- 2363 Views
- 0 replies
- 0 kudos
- 6663 Views
- 13 replies
- 1 kudos
Databricks Connect Scala -
Hi,I'm using Databricks Connect to run Scala code from IntelliJ on a Databricks single node cluster.Even with the simplest code, I'm experiencing this error:org.apache.spark.SparkException: grpc_shaded.io.grpc.StatusRuntimeException: INTERNAL: org.ap...
- 6663 Views
- 13 replies
- 1 kudos
- 1299 Views
- 1 replies
- 0 kudos
bitmap_count() function's output is different in databricks compared to snowflake
I have found that the results of the bitmap_count() function output differs significantly between databricks and snowflake.eg: snowflake returns a value of '1' for this code. "select bitmap_count(X'0001056c000000000000') " while Databricks returns a...
- 1299 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @vigneshp , Good Day! In Databricks, bitmap_count function returns the number of bits set in a BINARY string representing a bitmap. This function is typically used to count distinct values in combination with the bitmap_bucket_number() and the bi...
- 0 kudos
