- 5229 Views
- 2 replies
- 1 kudos
Resolved! Okta and Unified login
Hey Folks anyone put Databricks behind Okta and enabled Unified Login with workspaces that have a Unity Catalog metastore applied and some that don't?There are some workspaces we can't move over yet and it isn't clear in documentation if Unity Catalo...
- 5229 Views
- 2 replies
- 1 kudos
- 1 kudos
Yes, users should be able to use a single Okta application for all workspaces, regardless of whether the Unity Catalog metastore has been applied or not. The Unity Catalog is a feature that allows you to manage and secure access to your data across a...
- 1 kudos
- 6417 Views
- 5 replies
- 7 kudos
Incremental ingestion of Snowflake data with Delta Live Table (CDC)
Hello,I have some data which are lying into Snowflake, so I want to apply CDC on them using delta live table but I am having some issues.Here is what I am trying to do: @dlt.view() def table1(): return spark.read.format("snowflake").options(**opt...
- 6417 Views
- 5 replies
- 7 kudos
- 7 kudos
The CDC for delta live works fine for delta tables, as you have noticed. However it is not a full blown CDC implementation/software.If you want to capture changes in Snowflake, you will have to implement some CDC method on Snowflake itself, and read...
- 7 kudos
- 3678 Views
- 0 replies
- 0 kudos
Database: Delta Lake or PostgreSQL
Hey all,I am searching for a non-political answer to my database questions. Please know that I am a data newbie and litteraly do not know anything about this topic, but I want to learn, so please be gentle. Some context: I am working for an OEM that...
- 3678 Views
- 0 replies
- 0 kudos
- 1996 Views
- 0 replies
- 0 kudos
New draft for every post I visit
When I visit my profile page, under the drafts section I see an entry for every post I visit in the discussions. Is this normal?
- 1996 Views
- 0 replies
- 0 kudos
- 3348 Views
- 2 replies
- 0 kudos
Databricks jdbc driver connectiion issue with apache solr
Hi,databricks jdbc version - 2.6.34I am facing the below issue with connecting databricks sql from apache solr Caused by: java.sql.SQLFeatureNotSupportedException: [Databricks][JDBC](10220) Driver does not support this optional feature.at com.databri...
- 3348 Views
- 2 replies
- 0 kudos
- 0 kudos
Databricks team recommended to set IgnoreTransactions=1 and autocommit=false in the connection string but that didn't resolve the issue .Ultimately I had to use solr update API for uploading documents
- 0 kudos
- 5239 Views
- 3 replies
- 4 kudos
Resolved! Error not a delta table for Unity Catalog table
Is anyone able to advise why I am getting the error not a delta table? The table was created in Unity Catalog. I've also tried DeltaTable.forName and also using 13.3 LTS and 14.3 LTS clusters. Any advice would be much appreciated
- 5239 Views
- 3 replies
- 4 kudos
- 4 kudos
@StogponI believe if you are using DeltaTable.forPath then you have to pass the path where the table is. You can get this path from the Catalog. It is available in the details tab of the table.Example:delta_table_path = "dbfs:/user/hive/warehouse/xyz...
- 4 kudos
- 2434 Views
- 4 replies
- 3 kudos
BAD_REQUEST: ExperimentIds cannot be empty when checking ACLs in bulk
I was going through this tutorial https://mlflow.org/docs/latest/getting-started/tracking-server-overview/index.html#method-2-start-your-own-mlflow-server, I ran the whole script and when I try to open the experiment on the databricks website I get t...
- 2434 Views
- 4 replies
- 3 kudos
- 994 Views
- 0 replies
- 0 kudos
Using com.databricks:databricks-jdbc:2.6.36 inside oracle stored proc
Hi dear Databricks community,We tried to use databricks-jdbc inside oracle store procedure to load something from hive. However Oracle marked databricks-jdbc invalid because some classes (for example com.databricks.client.jdbc42.internal.io.netty.ut...
- 994 Views
- 0 replies
- 0 kudos
- 6922 Views
- 1 replies
- 0 kudos
How to detect if running in a workflow job?
Hi there,what's the best way to differentiate in what environment my Spark session is running? Locally I develop with databricks-connect's DatabricksSession, but that doesn't work when running a workflow job which requires SparkSession.getOrCreate()....
- 6922 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks, dbutils.notebook.getContext does indeed contain information about the job run.
- 0 kudos
- 2491 Views
- 2 replies
- 0 kudos
Limit the scope of workspace level access token to access only specific REST APIs of Databricks
Hi Community, Is there a way to limit the scope of workspace level token to hit only certain REST APIs of Databricks.In short, Once we generate a workspace level token following this doc. Link: https://docs.databricks.com/en/dev-tools/auth/oauth-m2m....
- 2491 Views
- 2 replies
- 0 kudos
- 9463 Views
- 2 replies
- 1 kudos
Resolved! Configuring NCC does not show option to add private endpoints
Hi!I am following this guide: https://learn.microsoft.com/en-us/azure/databricks/security/network/serverless-network-security/serverless-private-linkHowever in Step 3: Create private endpoint rules, number 6 there is no option for me to Add a private...
- 9463 Views
- 2 replies
- 1 kudos
- 1 kudos
@saikumar246 you were correct. It was really super easy to set up and works flawlessly! Good job dev-team!
- 1 kudos
- 3680 Views
- 0 replies
- 0 kudos
FileNotFoundError: [Errno 2] No such file or directory: 'ffmpeg'
import whisperimport ffmpegmodel = whisper.load_model("base")transcription = model.transcribe("dbfs:/FileStore/Call_Center_Conversation__03.mp3")print(transcription["text"])FileNotFoundError: [Errno 2] No such file or directory: 'ffmpeg'I have import...
- 3680 Views
- 0 replies
- 0 kudos
- 3238 Views
- 1 replies
- 0 kudos
Make API Call to run job
Hi everyone,I want to trigger a run for a job using API Call.Here's my code"""import shleximport subprocessdef call_curl(curl):args = shlex.split(curl)process = subprocess.Popen(args, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)stdout...
- 3238 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi everyone,I want to trigger a run for a job using API Call.Here's my code"""import shleximport subprocessdef call_curl(curl):args = shlex.split(curl)process = subprocess.Popen(args, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)stdout...
- 0 kudos
- 2710 Views
- 1 replies
- 0 kudos
Resolved! combining accounts
I have an AWS based databricks account with a few workspaces and an Azure Databricks workspace. How do I combine them into one account?I am particularly interested in setting up a single billing drop with all my Databricks costs.
- 2710 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @BillGuyTheScien Greetings! Currently, we do not have such a feature to combine multiple cloud usage into a single account. We do have a feature request for the same and it is considered for future. Currently, there is no ETA on that. You can bro...
- 0 kudos
- 3858 Views
- 1 replies
- 0 kudos
Catalog issue
When i was trying to create catalog i got an error saying to mention azure storage account and storage container in the following query -CREATE CATALOG IF NOT EXISTS Databricks_Anu_Jal_27022024MANAGED LOCATION 'abfss://<databricks-workspace-stack-anu...
- 3858 Views
- 1 replies
- 0 kudos
- 0 kudos
When i was trying to create catalog i got an error saying to mention azure storage account and storage container in the following query -CREATE CATALOG IF NOT EXISTS Databricks_Anu_Jal_27022024MANAGED LOCATION 'abfss://<databricks-workspace-stack-anu...
- 0 kudos
