I have a notebook in Databricks in which I am streaming a Pub/sub topic. The code for this looks like following-%pip install --upgrade google-cloud-pubsub[pandas] from pyspark.sql import SparkSession authOptions={"clientId" : "123","clientEmail"...
Hi I didn't see any conversations regarding using python RPA package on Data bricks clusters. Is anyone doing this or have gotten it to successfully work on the clusters? I ran into the following errors:1) Initially I was getting the error below rega...
If you want to capture browser screenshot, you can use playwright%sh pip install playwright playwright install sudo apt-get update playwright install-deps from playwright.async_api import async_playwright async with async_playwright() as p: ...
I am new to using volumes on databricks. Is there a way to create volume using CLI commands.On the similar note, is there a way to create DBFS directories and subdirectories using single command.for example: I want to copy file here dbfs:/FileStore/T...
Creates a new volume. The user could create either an external volume or a managed volume. An external volume will be created in the specified external location, while a managed volume will be located in the default location which is specified bythe...
https://docs.databricks.com/api/workspace/tablesIt seems I could only list/delete tables, is there a way to update a table's metadata like comment or detail fields by REST API?
Hi @al2co33 , We don't currently provide any APIs for updating table comments, however you can utilize the SQL Statement Execution API to do it. You can use the following tutorial to ALTER TABLE/COLUMN COMMENT. https://learn.microsoft.com/en-us/azure...
Hello, I'm writing to bring to your attention an issue that we have encountered while working with Data bricks and seek your assistance in resolving it.Context of the Error : When a sql query(1700 lines) is ran, corresponding data bricks job is faili...
are processing Parquet files or what is the format of your tables? can you split your sql query instead of having a huge query with 1700 lines
Hi all,I am recently started using databricks. I want to my git repo under workspace/Users/user_name path which I can't able to do it. But i can able to clone only under repo directory by default.Can anyone pls advice me regarding this Thank you
Hi team, I wanted to know if there is a way to connect a piece of my pyspark code running in ECS to Databricks cluster and leverage the databricks compute using Databricks connect?I see Databricks connect is for connecting local ide code to databrick...
Noted @Retired_mod @RonDeFreitas. I am currently using Databricks runtime v12.2 (which is < v13.0). I followed this doc (Databricks Connect for Databricks Runtime 12.2 LTS and below) and connected my local terminal to Databricks cluster and was able ...
Databricks is packaging a old version of big-query jar(Databricks also repackaged and created a fat jar), and our application needs a latest jar. Now the latest jar depends on spark-bigquery-connector.properties file for a property scala.binary.vers...
I try to get my head around the quality monitoring functionality in Unity Catalog. I configured one of the tables in our unity catalog. My assumption is that the profile and drift metrics tables are automatically created. But when I get an internal e...
Hi, were you able to resolve this, am having a similar issue - thanks
I am referencing this doc to initialize a databricks.sdk.WorkspaceClient object instance via Azure Native Authentication. I am initializing this WorkspaceClient within a databricks notebook, but I am trying to use the client to access the Jobs api of...
What makes Unity Catalog a game-changer? The blog intricately dissects five main value levers: mitigating data and architectural risks, ensuring compliance, accelerating innovation, reducing platform complexity and costs while improving operational e...
I’m relying on workspace assignment via REST API to have the account user created in the workspace. This is like the workspace assignment screen at account level or adding existing user screen at workspace level. The reference URL is below.Workspace ...
It turns out, the problem is the documentation. It says that the permission parameter (that's supplied in) is an array of strings. It really just expects a string, either UNKNOWN, USER, or ADMIN. It would be great if the team could fix the documentat...
Hello Allcan someone please suggest me how can I change the config IsBlindAppend true from false.I need to do this not for a data table but a custom log table .Also is there any concern If I toggle the value as standard practises. pls suggest
IsBlindAppend is not a config but an operation metrics that is used in Delta Lake History. The value of this changes based on the type of operation performed on Delta table. https://docs.databricks.com/en/delta/history.html
Hello Databricks Community, I'm trying to connect to Databricks via the Python library (databricks-sql-connector-3.0.1). This below code was working few months ago. Now, it is failing to connect.hostname, http_path and access_token are valid values a...
Hi,When i running this command over my private cluster (Single User) it works well :dbutils.fs.cp(ituff_file, protocol_local_file_path) When i try to run it over a shared cluster, i am getting : java.lang.SecurityException: Cannot use com.databricks....
Hi, Could you please refer to the limitations here: https://docs.databricks.com/en/compute/access-mode-limitations.html#shared-access-mode-limitations-on-unity-catalog (Volumes do not support dbutils.fs commands distributed to executors.) , Also, fo...
