- 9227 Views
- 1 replies
- 0 kudos
Resolved! "Run Job" without waiting for target job to finish?
We have configured a task in Job-A to run Job-B.However, the task in Job-A continues to 'run' until Job-B has completed.I can see this would be useful if we wanted to wait for Job-B and then perform another task, but we would actually like Job-A to e...
- 9227 Views
- 1 replies
- 0 kudos
- 0 kudos
you can create and run job use sdkdatabricks-sdk-py/examples/jobs/run_now_jobs_api_full_integration.py at main · databricks/databricks-sdk-py (github.com)
- 0 kudos
- 3191 Views
- 1 replies
- 2 kudos
Latest pushed code is not taken into account by Notebook
Hello, I cloned a repo my_repo in the Dataricks space Repos.Inside my_repo, I created a notebook new_experiment where I can import functions from my_repo, which is really handy. When I want to modify a function in my_repo, I open my local IDE, do the...
- 3191 Views
- 1 replies
- 2 kudos
- 2 kudos
Use %reload_ext autoreload instead, it will do your expected behavior.You just need to run it once, like %load_ext autoreload %autoreload 2
- 2 kudos
- 11481 Views
- 2 replies
- 1 kudos
Handling Exceptions from dbutils.fs in Python
I have a notebook that calls dbutils.fs.ls() for some derived file path in azure. Occasionally, this path may not exist, and in general I can't always guarantee that the path exists. When the path doesn't exist it throws an "ExecutionError" which app...
- 11481 Views
- 2 replies
- 1 kudos
- 1 kudos
Hey @jcoggs The problem looks legit though never occurred to me as I try to keep my mounts manually fed to the pipeline using a parameters or a variable by doing this you will have more control over your pipelines see if you could do the same in your...
- 1 kudos
- 2472 Views
- 1 replies
- 0 kudos
Databricks import api with lakeview dashboard "error_code":"INVALID_PARAMETER_VALUE"
Hi Community!I was trying to use the import api to replicate a lakehouse dashboard in a workspace but I keep bumping into INVALID_PARAMETER_VALUE error. After spending some time with getting the "content" property to (probably) correct base64 string ...
- 2472 Views
- 1 replies
- 0 kudos
- 0 kudos
There may be more information in the log. Look for "Cannot parse the zip archive."
- 0 kudos
- 32442 Views
- 12 replies
- 13 kudos
Option "delta.columnMapping.mode","name" introduces unexpected result
Hi, I am trying to write and create a delta table by enable "delta.columnMapping.mode","name", and the partition is date. But I found that when I enable this option, the partition folder name is not date any more while it is some random two letters.A...
- 32442 Views
- 12 replies
- 13 kudos
- 13 kudos
Hello, I'm a bit late to the party, but I'll put that for posterity:There's a way to rename your weird two letter named folders and still have your table working, but it violates the good practices guidelines suggested by Data Bricks, and I don't thi...
- 13 kudos
- 3065 Views
- 2 replies
- 1 kudos
Unable to write to Volume from DLT pipeline
Hi,I've a DLT pipeline running in Unity Catalog, and one of the task is to write content into a file within volume.I was able to write to file within volume using just PySpark, however when I do the same in DLT, I get an error:OSError: [Errno 30] Rea...
- 3065 Views
- 2 replies
- 1 kudos
- 1 kudos
It seems like a permission error. Can you check if the managed identity has correct permissions to write to the volume?DLT should be supported to write into volumes.
- 1 kudos
- 9511 Views
- 1 replies
- 0 kudos
What is the best practice for data model in silver layer in lakehouse
As per databricks https://www.databricks.com/glossary/medallion-architecture silver layers typically represent the "enterprise view" with improved quality than bronze (cleansed, deduplicated , augmented ) and mostly has 3NF like normalised data . The...
- 9511 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @param_sen Given your concerns about expensive joins and prioritizing analytics with flat raw data, here are some suggestions:Analyze the most common queries and reports you anticipate. Do they heavily rely on joins across dimensions? If not, the...
- 0 kudos
- 6551 Views
- 1 replies
- 1 kudos
Resolved! Asset Bundles Scala
Do asset bundles support Scala notebooks? I am trying to run a simple job that uses Scala notebook and getting an error: "run failed with error message Your administrator has only allowed sql and python commands on this cluster. This execution contai...
- 6551 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @VitaliiK,Thanks for bringing up your concerns; I am always happy to help The error means that the cluster is configured to only allow SQL and Python languages to be used in notebooks and the notebook you are trying to run contains a Scala langua...
- 1 kudos
- 4461 Views
- 3 replies
- 1 kudos
Resolved! autoloader stops working if I do not drop table each time
I first create a catalog and schema and ingest some data into it as follows:catalogName = 'neo'schemaName ='indicators'locationPath = 's3a://databricks-workspace-olaptrader-stack-1-bucket/unity-catalog/99999999xxx'sqlContext.sql(f"CREATE CATALOG IF N...
- 4461 Views
- 3 replies
- 1 kudos
- 1 kudos
My issue was due to the fact that I have over 300 columns and due to datatype mismatches, the rows were actually written to the table, but values were all null. That's why I didnt get any errors. I am doing manual datatype mapping now and I am able t...
- 1 kudos
- 4748 Views
- 2 replies
- 0 kudos
Resolved! Auto optimize config
Does auto-optimize work for existing tables only or will it work for both existing and new tables when we enable at the cluster config level?
- 4748 Views
- 2 replies
- 0 kudos
- 0 kudos
If you're referring to Delta Tables, Auto-Optimize will work for both.For new tables:CREATE TABLE student (id INT, name STRING, age INT) TBLPROPERTIES (delta.autoOptimize.optimizeWrite = true, delta.autoOptimize.autoCompact = true)For existing tables...
- 0 kudos
- 2811 Views
- 0 replies
- 0 kudos
Databricks-jdbc and vulnerabilities CVE-2021-36090 CVE-2023-6378 CVE-2023-6481
The latest version of Databricks-jdbc available through Maven (2.6.36) now has these three vulnerabilities:https://www.cve.org/CVERecord?id=CVE-2021-36090https://www.cve.org/CVERecord?id=CVE-2023-6378https://www.cve.org/CVERecord?id=CVE-2023-6481All ...
- 2811 Views
- 0 replies
- 0 kudos
- 2452 Views
- 3 replies
- 0 kudos
Internal Error when querying a doubleType column of a delta table using ">" "<" operators
Hi there,we are currently facing a pretty confusing issue:We have a delta table (~2TB) which has been working just fine over the last few years and months. For a few days or weeks now, querying the table on one of its columns, let´s call it double_co...
- 2452 Views
- 3 replies
- 0 kudos
- 0 kudos
it might be a bug which is already logged, or a new one. You can check the Spark Jira pages.
- 0 kudos
- 4118 Views
- 1 replies
- 0 kudos
Resolved! IllegalArgumentException: Mount failed due to invalid mount source
We are trying to create mount for containers from two different storage accounts. We are using Azure Storage Account and Azure Data bricks.We could able to create mount for containers from one storage account, but when we try to create the mount for ...
- 4118 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi community,The issue was becoz of cross vent access. The storage account and the databricks workspace both are in different vnet. Since that we had to create private end point to access the cross vnet resources. Once we crated the private endpoint ...
- 0 kudos
- 2209 Views
- 0 replies
- 0 kudos
Course: Apache Spark Programming with Databricks ID: E-P0W7ZV // Issue Classroom-Setup
Hello,I am trying to run the Classroom-Setup from the course files notebook (ASP 1.1 - Databricks Platform)(Course:Apache Sparkâ„¢ Programming with DatabricksID: E-P0W7ZV)Instructions: "Setup:Run classroom setup to mount Databricks training datasets an...
- 2209 Views
- 0 replies
- 0 kudos
- 18037 Views
- 6 replies
- 3 kudos
upload files to dbfs:/volume using databricks cli
In our azure pipeline we are using databricks-cli command to upload jar files at dbfs:/FileStore location and that works perfectly fine. But when we try to use the same command to upload files at dbfs:/Volume/dev/default/files, it does not work and g...
- 18037 Views
- 6 replies
- 3 kudos
- 3 kudos
@Hardy I think you are using the word volume in the path but it should be Volumes(plural), not Volume(singular). Do one thing, copy the volume path directly from the Workspace and try.
- 3 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
