- 1769 Views
- 2 replies
- 2 kudos
Resolved! Regarding cloning my gitrepo under workspace/Users/user_name
Hi all,I am recently started using databricks. I want to my git repo under workspace/Users/user_name path which I can't able to do it. But i can able to clone only under repo directory by default.Can anyone pls advice me regarding this Thank you
- 1769 Views
- 2 replies
- 2 kudos
- 2545 Views
- 4 replies
- 0 kudos
Connect my spark code running in AWS ECS to databricks cluster
Hi team, I wanted to know if there is a way to connect a piece of my pyspark code running in ECS to Databricks cluster and leverage the databricks compute using Databricks connect?I see Databricks connect is for connecting local ide code to databrick...
- 2545 Views
- 4 replies
- 0 kudos
- 0 kudos
Noted @Retired_mod @RonDeFreitas. I am currently using Databricks runtime v12.2 (which is < v13.0). I followed this doc (Databricks Connect for Databricks Runtime 12.2 LTS and below) and connected my local terminal to Databricks cluster and was able ...
- 0 kudos
- 641 Views
- 0 replies
- 0 kudos
Databricks & Bigquery
Databricks is packaging a old version of big-query jar(Databricks also repackaged and created a fat jar), and our application needs a latest jar. Now the latest jar depends on spark-bigquery-connector.properties file for a property scala.binary.vers...
- 641 Views
- 0 replies
- 0 kudos
- 1013 Views
- 1 replies
- 0 kudos
Unity catalog internal error - quality monitoring
I try to get my head around the quality monitoring functionality in Unity Catalog. I configured one of the tables in our unity catalog. My assumption is that the profile and drift metrics tables are automatically created. But when I get an internal e...
- 1013 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, were you able to resolve this, am having a similar issue - thanks
- 0 kudos
- 3181 Views
- 0 replies
- 0 kudos
Error authenticating databricks.sdk.WorkspaceClient with external workspace via Azure Native Auth
I am referencing this doc to initialize a databricks.sdk.WorkspaceClient object instance via Azure Native Authentication. I am initializing this WorkspaceClient within a databricks notebook, but I am trying to use the client to access the Jobs api of...
- 3181 Views
- 0 replies
- 0 kudos
- 6686 Views
- 0 replies
- 3 kudos
Unity Catalog Governance Value Levers
What makes Unity Catalog a game-changer? The blog intricately dissects five main value levers: mitigating data and architectural risks, ensuring compliance, accelerating innovation, reducing platform complexity and costs while improving operational e...
- 6686 Views
- 0 replies
- 3 kudos
- 1149 Views
- 1 replies
- 0 kudos
Workspace Assignment Issue via REST API
I’m relying on workspace assignment via REST API to have the account user created in the workspace. This is like the workspace assignment screen at account level or adding existing user screen at workspace level. The reference URL is below.Workspace ...
- 1149 Views
- 1 replies
- 0 kudos
- 0 kudos
It turns out, the problem is the documentation. It says that the permission parameter (that's supplied in) is an array of strings. It really just expects a string, either UNKNOWN, USER, or ADMIN. It would be great if the team could fix the documentat...
- 0 kudos
- 2840 Views
- 1 replies
- 0 kudos
Unable to create Iceberg tables pointing to data in S3 and run queries against the tables.
I need to to set up Iceberg tables in Databricks environment, but the data resides in an S3 bucket. Then read these tables by running SQL queries.Databricks environment has access to S3. This is done bysetting up the access by mapping the Instance Pr...
- 2840 Views
- 1 replies
- 0 kudos
- 0 kudos
@JohnsonBDSouza - could you please let me know if you had a chance to review the Uniform feature that allows to create iceberg tables from the delta format. Based on what i could understand from the above, you can create a delta table and use the b...
- 0 kudos
- 1559 Views
- 1 replies
- 0 kudos
Write stream to Kafka topic with DLT
Hi,Is it possible to write stream to Kafka topic with Delta Live Table?I would like to do something like this:@dlt.view(name="kafka_pub",comment="Publish to kafka")def kafka_pub():return (dlt.readStream("source_table").selectExpr("to_json (struct (*)...
- 1559 Views
- 1 replies
- 0 kudos
- 0 kudos
@gb - Currently, DLT does not support writing to non-delta sink. we can divide the pipeline currently into two by writing to a materialized view or delta sink and build a non-DLT job to write out to Kafka sink. Databricks DLT engineering team is wo...
- 0 kudos
- 959 Views
- 2 replies
- 0 kudos
IsBlindAppend config change
Hello Allcan someone please suggest me how can I change the config IsBlindAppend true from false.I need to do this not for a data table but a custom log table .Also is there any concern If I toggle the value as standard practises. pls suggest
- 959 Views
- 2 replies
- 0 kudos
- 0 kudos
IsBlindAppend is not a config but an operation metrics that is used in Delta Lake History. The value of this changes based on the type of operation performed on Delta table. https://docs.databricks.com/en/delta/history.html
- 0 kudos
- 2166 Views
- 1 replies
- 0 kudos
Connection Error Python databricks-sql-connector
Hello Databricks Community, I'm trying to connect to Databricks via the Python library (databricks-sql-connector-3.0.1). This below code was working few months ago. Now, it is failing to connect.hostname, http_path and access_token are valid values a...
- 2166 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello Databricks Community, I'm trying to connect to Databricks via the Python library (databricks-sql-connector-3.0.1). This below code was working few months ago. Now, it is failing to connect.hostname, http_path and access_token are valid values a...
- 0 kudos
- 2351 Views
- 1 replies
- 0 kudos
local filesystem access is forbidden
Hi,When i running this command over my private cluster (Single User) it works well :dbutils.fs.cp(ituff_file, protocol_local_file_path) When i try to run it over a shared cluster, i am getting : java.lang.SecurityException: Cannot use com.databricks....
- 2351 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, Could you please refer to the limitations here: https://docs.databricks.com/en/compute/access-mode-limitations.html#shared-access-mode-limitations-on-unity-catalog (Volumes do not support dbutils.fs commands distributed to executors.) , Also, fo...
- 0 kudos
- 1516 Views
- 0 replies
- 0 kudos
Py4JSecurityException for file access in azure data storage - seeking help
I am trying to access a file in azure data storage using databricks in python. when i access I am getting py4SecurityException (py4j.security.Py4JSecurityException: Method public com.databricks.backend.daemon.dbutils.DBUtilsCore$Result com.databricks...
- 1516 Views
- 0 replies
- 0 kudos
- 3415 Views
- 1 replies
- 0 kudos
How to update a value in a column in a delta table with Map of Struct datatype?
I have a delta table in Databricks named "prod.silver.control_table". It has a few columns including "table_name" with string data type and "transform_options" with the below structure: |-- transform_options: map (nullable = true) | |-- key: str...
- 3415 Views
- 1 replies
- 0 kudos
- 0 kudos
I have a delta table in Databricks named "prod.silver.control_table". It has a few columns including "table_name" with string data type and "transform_options" with the below structure: |-- transform_options: map (nullable = true) | |-- key: str...
- 0 kudos
- 1456 Views
- 0 replies
- 0 kudos
Unable to login to Databricks Community edition
I signed up to Databricks Community Edition with gmail and verified my account and used it to create a notebook as well but having issue in re-logging in even though email and password provided are correct. I did try "Forget Password" step but I am...
- 1456 Views
- 0 replies
- 0 kudos
