- 6723 Views
- 1 replies
- 0 kudos
Resolved! Issue with View Ownership Reassignment in Unity Catalog
Hello,It appears that the ownership rules for views and functions in Unity Catalog do not align with the guidelines provided in the “Manage Unity Catalog object ownership” documentation on Microsoft Learn.When attempting to reassign the ownership of ...
- 6723 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Filippo To prevent privilege escalations, only a metastore admin can transfer ownership of a view, function, or model to any user, service principal, or group in the account. Current owners and users with the MANAGE privilege are restricted to tr...
- 0 kudos
- 5530 Views
- 1 replies
- 0 kudos
Resolved! dbx with serverless clusters
With dbx, is it impossible to create tasks that run on serverless clusters? Is it necessary to use Databricks bundles for it?https://dbx.readthedocs.io/en/latest/reference/deployment/https://learn.microsoft.com/en-us/azure/databricks/jobs/run-serverl...
- 5530 Views
- 1 replies
- 0 kudos
- 0 kudos
It is possible to create tasks that run on serverless clusters using dbx. Also, please note that Databricks recommends that you use Databricks Asset Bundles instead of dbx by Databricks Labs. See What are Databricks Asset Bundles and Migrate from dbx...
- 0 kudos
- 79688 Views
- 6 replies
- 0 kudos
Resolved! dbutils.notebook.exit() executing from except in try/except block even if there is no error.
I am using Python notebooks as part of a concurrently running workflow with Databricks Runtime 6.1. Within the notebooks I am using try/except blocks to return an error message to the main concurrent notebook if a section of code fails. However I h...
- 79688 Views
- 6 replies
- 0 kudos
- 0 kudos
because the dbutils.notebook.exit() is an 'Exception' it will always trigger the except Exception as e: part of the code. When can use this to our advantage to solve the problem by adding an 'if else' to the except block. query = "SELECT 'a' as Colum...
- 0 kudos
- 2452 Views
- 1 replies
- 0 kudos
How do i stop pyspark from outputting text
I am using a tqdm progress bar to monitor the amount of data records I have collected via API. I am temporarily writing them to a file in the DBFS, then uploading to a Spark DataFrame. Each time I write to a file, I get a message like 'Wrote 8873925 ...
- 2452 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @halox6000, You could temporarily redirect console output to a null device for these write operations. Try this out: @contextlib.contextmanager def silence_dbutils(): with contextlib.redirect_stdout(io.StringIO()): yield # Usage in...
- 0 kudos
- 2727 Views
- 2 replies
- 0 kudos
remove empty folders with pyspark
Hi,I am trying to search a mnt point for any empty folders and remove them. Does anyone know of a way to do this? I have tried dbutils.fs.walk but this does not seem to work.Thanks
- 2727 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @Nathant93, You could use dbutils.fs.ls and iterate on all the directories found to accomplish this task. Something like this: def find_empty_dirs(path): directories = dbutils.fs.ls(path) for directory in directories: if directo...
- 0 kudos
- 6146 Views
- 5 replies
- 1 kudos
Executing an HTTP call from worker to localhost:80 (Unity Catalog)
hi,trying to implement this industry accelerator in UC (DBR 15.4, Shared Access Mode), I'm encountering an error when running an HTTP query (python requests) from within a udf (see this github issue). Are there any restrictions when it comes to run a...
- 6146 Views
- 5 replies
- 1 kudos
- 4495 Views
- 11 replies
- 1 kudos
Databricks SQL Warehouse Querys went to orphan state
We're experiencing an issue with our Databricks dbt workflow and workflow job is using the SQL warehouse L size cluster that's been working smoothly for the past couple of weeks. However, today we've noticed that at a specific time, all queries are g...
- 4495 Views
- 11 replies
- 1 kudos
- 9066 Views
- 2 replies
- 1 kudos
Cant connect power BI desktop to Azure databricks
Hello,I am trying to connect Power BI desktop to azure databricks (source: delta table) by downloading a connection file from Databricks. I see an error message like below when I open the connection file with power BI. Repeated attempts have given th...
- 9066 Views
- 2 replies
- 1 kudos
- 2861 Views
- 1 replies
- 0 kudos
VSCode Databricks-Connect can't find config file, says it doesn't exist, but it does
Hi all,I am getting an error that I previously didn't have within VSCode when authenticating to Databricks-Connect in PowerShell.When I run "databricks auth login" and choose a profile, I get the following:> Error: cannot load Databricks config file:...
- 2861 Views
- 1 replies
- 0 kudos
- 0 kudos
PrerequisitesDatabricks CLI installedMatch your databricks-connect version with your cluster runtime (e.g. runtime 14.3 LTS, needs databricks-connect 14.3.x)Match your local python installation with the Databricks Python version.Databricks extension ...
- 0 kudos
- 2974 Views
- 1 replies
- 0 kudos
ExecutorLostFailure
ExecutorLostFailure (executor 22 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
- 2974 Views
- 1 replies
- 0 kudos
- 0 kudos
he ExecutorLostFailure error in Spark indicates that an executor was lost during the execution of a task. Review the driver logs for any WARN or ERROR messages that might provide more context about why the executor was lost. Also, Ensure that the exe...
- 0 kudos
- 2917 Views
- 4 replies
- 2 kudos
Resolved! Differentiate null values in Variant Data type
Hello, Based on the documentation here, in both scenarios below try_variant_get returns a null: If the object cannot be foundif the object cannot be cast How does one differentiate between the two scenarios?
- 2917 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @vgautam, In the try_variant_get function, NULL is returned in two scenarios: Object Not Found: If the specified path does not exist in the JSON object.Invalid Cast: If the object at the specified path cannot be cast to the target type. To differe...
- 2 kudos
- 3853 Views
- 8 replies
- 2 kudos
Getting OOM error while processing xml data
I have a table in which one of the column contains xml raw data , approx. size of each row is 3MB, The volume of data is very huge, I have chunked it into 1 hour processing, On observing Memory Utilization metrics everything seems fine, but receiving...
- 3853 Views
- 8 replies
- 2 kudos
- 2 kudos
filteredDataframe=spark.table(f'{sourceConfig["srcDatabaseName"]}.{sourceConfig["srcTableName"]}').filter(f.col("load_dt")==current_start_time.date()).filter(f.col("load_ts")>=current_start_time).filter(f.col("load_ts")<current_end_time).filter("col...
- 2 kudos
- 5800 Views
- 11 replies
- 0 kudos
How to deploy unique workflows that running on production
Hello, community!I have a question about deploying workflows in a production environment. Specifically, how can we deploy a group of workflows to production so that they are created only once and cannot be duplicated by others?Currently, if someone d...
- 5800 Views
- 11 replies
- 0 kudos
- 0 kudos
I got some information from my internal team:The main thing to help here is deploying as a service principal and setting mode: production on the target. This is best done by setting up automation, such as Github Actions or Azure DevOps pipeline. You ...
- 0 kudos
- 2743 Views
- 2 replies
- 0 kudos
Unable to communicate with AWS DocumentDB
Runtime version: 15.4 LTS (includes Apache Spark 3.5.0, Scala 2.12)Spark config:'''spark.hadoop.datanucleus.fixedDatastore false spark.driver.extraJavaOptions -Djavax.net.ssl.trustStore=$JAVA_HOME/lib/security/cacerts spark.hadoop.javax.jdo.option.Co...
- 2743 Views
- 2 replies
- 0 kudos
- 0 kudos
The error seems to be related to writing data to a MongoDB data source, as indicated by the com.mongodb.spark.sql.connector.exceptions.DataException. It appears that the error is occurring during the execution of a Spark job that involves writing dat...
- 0 kudos
- 4966 Views
- 1 replies
- 0 kudos
Unable to read data from unity catalog, cannot store results of commands to DBFS
When attempting to read an excel (or any file) from a Unity Catalog volume (external or managed) the directory/file is not found. Additionally, when trying to use display() or any function like it to show results of a code chunk I am presented with"F...
- 4966 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @randythedataguy The error message "Failed to upload command result to DBFS. Error message: dbstoragekbbxxxxxxxxx.dfs.core.windows.net: Name or service not known" suggests a potential DNS resolution issue. Ensure that the DNS settings are correctl...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
