- 5194 Views
- 2 replies
- 0 kudos
Resolved! Using Python RPA Library on Databricks
Hi I didn't see any conversations regarding using python RPA package on Data bricks clusters. Is anyone doing this or have gotten it to successfully work on the clusters? I ran into the following errors:1) Initially I was getting the error below rega...
- 5194 Views
- 2 replies
- 0 kudos
- 0 kudos
If you want to capture browser screenshot, you can use playwright%sh pip install playwright playwright install sudo apt-get update playwright install-deps from playwright.async_api import async_playwright async with async_playwright() as p: ...
- 0 kudos
- 2430 Views
- 0 replies
- 0 kudos
Unity Catalog: Databricks *Specific* Features
Good day,Deceptively simple question, are there any "Databricks only" specific features that Unity Catalog offers? I understand that generally speaking enabling UC offers some of the following:Data Discovery and LineageAuditing and MonitoringAccess C...
- 2430 Views
- 0 replies
- 0 kudos
- 3941 Views
- 2 replies
- 0 kudos
Problem login in
Hello allI´m new in this platform, I sign up, validated my email, create my password everything is fine and when I try to log in and start a message came upI create a new password but same happen again! But it works a few times, I think like 3 times....
- 3941 Views
- 2 replies
- 0 kudos
- 0 kudos
Can you confirm the username was created all lower case? Login is case sensitive so you need to make sure the username is set same exact as you add it in the console or workspace
- 0 kudos
- 2861 Views
- 1 replies
- 0 kudos
how to create volume using databricks cli commands
I am new to using volumes on databricks. Is there a way to create volume using CLI commands.On the similar note, is there a way to create DBFS directories and subdirectories using single command.for example: I want to copy file here dbfs:/FileStore/T...
- 2861 Views
- 1 replies
- 0 kudos
- 0 kudos
Creates a new volume. The user could create either an external volume or a managed volume. An external volume will be created in the specified external location, while a managed volume will be located in the default location which is specified bythe...
- 0 kudos
- 9160 Views
- 3 replies
- 3 kudos
Resolved! DLT Job Clusters: Continuous vs Triggered Cluster Start Times
Hi there,I'm curious if anyone is able to definitively help me answer how DLT Job Clusters operate/run.For example, the following is my baseline understanding of DLT Job Clusters. If I run a Triggered DLT Pipeline (e.g. daily) the job cluster takes m...
- 9160 Views
- 3 replies
- 3 kudos
- 3 kudos
Ideally one would expect clusters used for DLT pipeline to terminate after the pipeline execution has finished. However, while running in `development` environment, you'll notice it doesn't terminate on its own, whereas in `production` it terminates ...
- 3 kudos
- 2536 Views
- 1 replies
- 0 kudos
Can I update a table comment using REST API?
https://docs.databricks.com/api/workspace/tablesIt seems I could only list/delete tables, is there a way to update a table's metadata like comment or detail fields by REST API?
- 2536 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @al2co33 , We don't currently provide any APIs for updating table comments, however you can utilize the SQL Statement Execution API to do it. You can use the following tutorial to ALTER TABLE/COLUMN COMMENT. https://learn.microsoft.com/en-us/azure...
- 0 kudos
- 3791 Views
- 1 replies
- 0 kudos
Databricks Java - Create Jar in Java 11
I am trying to a run simple print java program which is not working and getting compilation version issues though i changed the environment variable points to java 11. Can you please help me ? Can we create java with spark session and execute as a ja...
- 3791 Views
- 1 replies
- 0 kudos
- 0 kudos
@Databricks_Java You can run command like this: spark-submit --class com.test.Main example.jarand make sure to check the java version and match with the DBR compatibility
- 0 kudos
- 2197 Views
- 1 replies
- 0 kudos
Disable personal compute with the Databricks API or UI
For a production environment, I want to disable the personal compute policy, because I do not want that all users can create personal compute clusters in production. Unfortunately, I am not able to access the account console, so I want to revoke perm...
- 2197 Views
- 1 replies
- 0 kudos
- 0 kudos
@mathijs-fish You need to be admin to disable a policy.
- 0 kudos
- 4630 Views
- 1 replies
- 0 kudos
No space left on device and IllegalStateException: Have already allocated a maximum of 8192 pages
Hello, I'm writing to bring to your attention an issue that we have encountered while working with Data bricks and seek your assistance in resolving it.Context of the Error : When a sql query(1700 lines) is ran, corresponding data bricks job is faili...
- 4630 Views
- 1 replies
- 0 kudos
- 0 kudos
are processing Parquet files or what is the format of your tables? can you split your sql query instead of having a huge query with 1700 lines
- 0 kudos
- 2382 Views
- 3 replies
- 0 kudos
Autoloader file latency
Hi Team,I would like to understand if there is a metadata table for the autoloader in Databricks that captures information about file arrival and processing.The reason we are experiencing data issues is because our table A receives hundreds of files ...
- 2382 Views
- 3 replies
- 0 kudos
- 0 kudos
Check with cloud_files_state() API You can find examples here https://docs.databricks.com/en/ingestion/auto-loader/production.html#querying-files-discovered-by-auto-loader
- 0 kudos
- 2616 Views
- 2 replies
- 2 kudos
Resolved! Regarding cloning my gitrepo under workspace/Users/user_name
Hi all,I am recently started using databricks. I want to my git repo under workspace/Users/user_name path which I can't able to do it. But i can able to clone only under repo directory by default.Can anyone pls advice me regarding this Thank you
- 2616 Views
- 2 replies
- 2 kudos
- 4511 Views
- 4 replies
- 0 kudos
Connect my spark code running in AWS ECS to databricks cluster
Hi team, I wanted to know if there is a way to connect a piece of my pyspark code running in ECS to Databricks cluster and leverage the databricks compute using Databricks connect?I see Databricks connect is for connecting local ide code to databrick...
- 4511 Views
- 4 replies
- 0 kudos
- 0 kudos
Noted @Retired_mod @RonDeFreitas. I am currently using Databricks runtime v12.2 (which is < v13.0). I followed this doc (Databricks Connect for Databricks Runtime 12.2 LTS and below) and connected my local terminal to Databricks cluster and was able ...
- 0 kudos
- 5543 Views
- 2 replies
- 0 kudos
Resolved! spark context in databricks
Hi @all,In Azure Databricks,I am using structured streaming for each batch functionality, in one of the functions I am creating tempview with pyspark dataframe (*Not GlobalTempView) and trying to access the same temp view by using spark.sql functiona...
- 5543 Views
- 2 replies
- 0 kudos
- 0 kudos
Do you face this issue without spark streaming as well? Also, could you share a minimal repo code preferably without streaming?
- 0 kudos
- 974 Views
- 0 replies
- 0 kudos
Databricks & Bigquery
Databricks is packaging a old version of big-query jar(Databricks also repackaged and created a fat jar), and our application needs a latest jar. Now the latest jar depends on spark-bigquery-connector.properties file for a property scala.binary.vers...
- 974 Views
- 0 replies
- 0 kudos
- 2129 Views
- 1 replies
- 0 kudos
Unity catalog internal error - quality monitoring
I try to get my head around the quality monitoring functionality in Unity Catalog. I configured one of the tables in our unity catalog. My assumption is that the profile and drift metrics tables are automatically created. But when I get an internal e...
- 2129 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, were you able to resolve this, am having a similar issue - thanks
- 0 kudos
