- 2030 Views
- 5 replies
- 0 kudos
Resolved! Query separate data loads from python spark.readStream
I am using python spark.readStream in a Delta Live Tables pipeline to read json data files from a S3 folder path. Each load is a daily snapshot of a very similar set of products showing changes in price and inventory. How do i distinguish and query e...
- 2030 Views
- 5 replies
- 0 kudos
- 0 kudos
The problem was fixed by this importfrom pyspark.sql import functions as F then using F.lit() instead of F.col.withColumn('ingestion_date', F.lit(folder_date)) Sorry code formatting is not working at the moment.
- 0 kudos
- 5235 Views
- 6 replies
- 5 kudos
Resolved! Issue with DeltaFileNotFoundException After Vacuum and Missing Data Changes in Delta Log
Dear Databricks experts,I encountered the following error in Databricks:`com.databricks.sql.transaction.tahoe.DeltaFileNotFoundException: [DELTA_EMPTY_DIRECTORY] No file found in the directory: gs://cimb-prod-lakehouse/bronze-layer/losdb/pl_message/_...
- 5235 Views
- 6 replies
- 5 kudos
- 5 kudos
Hi @minhhung0507,The VACUUM command on a Delta table does not delete the _delta_log folder, as this folder contains all the metadata related to the Delta table. The _delta_log folder acts as a pointer where all changes are tracked. In the event that ...
- 5 kudos
- 9220 Views
- 7 replies
- 3 kudos
Errors calling databricks rest api /api/2.1/jobs/run-now with job_parameters
Hello! I have been using the databricks rest api for running workflows using this endpoint: /api/2.1/jobs/run-now. But now i wanted to also include job_parameters in my api call, i have put job parameters inside my workflow: param1, param2, and in my...
- 9220 Views
- 7 replies
- 3 kudos
- 3 kudos
I encountered a null job_id in my post, when a notebook parameter was set in the job GUI. But it runs just fine (I get a valid job_id with active run) if I delete the notebook parameter in the job GUI.Is this a documented behavior, or a bug? If it's ...
- 3 kudos
- 869 Views
- 1 replies
- 0 kudos
JBDC RBMS Table Overwrite Transaction Incomplete
Spark version: spark-3.4.1-bin-hadoop3JBDC Driver: mysql-connector-j-8.4.0.jarAssumptions:have all the proper read/write permissionsdataset isn't large: ~2 million recordsreading flat files, writing to a databaseDoes not read from the database at al...
- 869 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @diegohMoodys, Can you try in debug mode? spark.sparkContext.setLogLevel("DEBUG")
- 0 kudos
- 7059 Views
- 16 replies
- 2 kudos
agents.deploy NOT_FOUND: The directory being accessed is not found. error
I keep getting the following error although the model definitely does exist and version names and model name is correct RestException: NOT_FOUND: The directory being accessed is not found. when calling # Deploy the model to the review app and a model...
- 7059 Views
- 16 replies
- 2 kudos
- 5382 Views
- 1 replies
- 0 kudos
Resolved! Databricks Asset Bundle - Job Cluster - JDBC HTTP Path
I am currently working on deploying dbt jobs using a Databricks Asset Bundle. In my existing job configuration, I am using an all-purpose cluster and the JDBC HTTP Path was manually copied from the web UI. Now that I am trying to switch to using a jo...
- 5382 Views
- 1 replies
- 0 kudos
- 0 kudos
To reference the HTTP Path using substitutions in Databricks Asset Bundles and job clusters, you can use the variables section in your databricks.yml configuration file In your databricks.yml file, you can define a variable for the HTTP Path. For e...
- 0 kudos
- 6173 Views
- 1 replies
- 0 kudos
Resolved! Issue with View Ownership Reassignment in Unity Catalog
Hello,It appears that the ownership rules for views and functions in Unity Catalog do not align with the guidelines provided in the “Manage Unity Catalog object ownership” documentation on Microsoft Learn.When attempting to reassign the ownership of ...
- 6173 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Filippo To prevent privilege escalations, only a metastore admin can transfer ownership of a view, function, or model to any user, service principal, or group in the account. Current owners and users with the MANAGE privilege are restricted to tr...
- 0 kudos
- 5166 Views
- 1 replies
- 0 kudos
Resolved! dbx with serverless clusters
With dbx, is it impossible to create tasks that run on serverless clusters? Is it necessary to use Databricks bundles for it?https://dbx.readthedocs.io/en/latest/reference/deployment/https://learn.microsoft.com/en-us/azure/databricks/jobs/run-serverl...
- 5166 Views
- 1 replies
- 0 kudos
- 0 kudos
It is possible to create tasks that run on serverless clusters using dbx. Also, please note that Databricks recommends that you use Databricks Asset Bundles instead of dbx by Databricks Labs. See What are Databricks Asset Bundles and Migrate from dbx...
- 0 kudos
- 76389 Views
- 6 replies
- 0 kudos
Resolved! dbutils.notebook.exit() executing from except in try/except block even if there is no error.
I am using Python notebooks as part of a concurrently running workflow with Databricks Runtime 6.1. Within the notebooks I am using try/except blocks to return an error message to the main concurrent notebook if a section of code fails. However I h...
- 76389 Views
- 6 replies
- 0 kudos
- 0 kudos
because the dbutils.notebook.exit() is an 'Exception' it will always trigger the except Exception as e: part of the code. When can use this to our advantage to solve the problem by adding an 'if else' to the except block. query = "SELECT 'a' as Colum...
- 0 kudos
- 2219 Views
- 1 replies
- 0 kudos
How do i stop pyspark from outputting text
I am using a tqdm progress bar to monitor the amount of data records I have collected via API. I am temporarily writing them to a file in the DBFS, then uploading to a Spark DataFrame. Each time I write to a file, I get a message like 'Wrote 8873925 ...
- 2219 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @halox6000, You could temporarily redirect console output to a null device for these write operations. Try this out: @contextlib.contextmanager def silence_dbutils(): with contextlib.redirect_stdout(io.StringIO()): yield # Usage in...
- 0 kudos
- 2417 Views
- 2 replies
- 0 kudos
remove empty folders with pyspark
Hi,I am trying to search a mnt point for any empty folders and remove them. Does anyone know of a way to do this? I have tried dbutils.fs.walk but this does not seem to work.Thanks
- 2417 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @Nathant93, You could use dbutils.fs.ls and iterate on all the directories found to accomplish this task. Something like this: def find_empty_dirs(path): directories = dbutils.fs.ls(path) for directory in directories: if directo...
- 0 kudos
- 5618 Views
- 5 replies
- 1 kudos
Executing an HTTP call from worker to localhost:80 (Unity Catalog)
hi,trying to implement this industry accelerator in UC (DBR 15.4, Shared Access Mode), I'm encountering an error when running an HTTP query (python requests) from within a udf (see this github issue). Are there any restrictions when it comes to run a...
- 5618 Views
- 5 replies
- 1 kudos
- 3555 Views
- 11 replies
- 1 kudos
Databricks SQL Warehouse Querys went to orphan state
We're experiencing an issue with our Databricks dbt workflow and workflow job is using the SQL warehouse L size cluster that's been working smoothly for the past couple of weeks. However, today we've noticed that at a specific time, all queries are g...
- 3555 Views
- 11 replies
- 1 kudos
- 8765 Views
- 2 replies
- 1 kudos
Cant connect power BI desktop to Azure databricks
Hello,I am trying to connect Power BI desktop to azure databricks (source: delta table) by downloading a connection file from Databricks. I see an error message like below when I open the connection file with power BI. Repeated attempts have given th...
- 8765 Views
- 2 replies
- 1 kudos
- 2446 Views
- 1 replies
- 0 kudos
VSCode Databricks-Connect can't find config file, says it doesn't exist, but it does
Hi all,I am getting an error that I previously didn't have within VSCode when authenticating to Databricks-Connect in PowerShell.When I run "databricks auth login" and choose a profile, I get the following:> Error: cannot load Databricks config file:...
- 2446 Views
- 1 replies
- 0 kudos
- 0 kudos
PrerequisitesDatabricks CLI installedMatch your databricks-connect version with your cluster runtime (e.g. runtime 14.3 LTS, needs databricks-connect 14.3.x)Match your local python installation with the Databricks Python version.Databricks extension ...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 570 | |
| 349 | |
| 287 |
