- 1773 Views
- 4 replies
- 3 kudos
Issue with os and sys Operations in Repo Path on Databricks
Hi,Starting from today, I have encountered an issue when performing operations using the os and sys modules within the Repo path in my Databricks environment. Specifically, any operation that involves these modules results in a timeout error. However...
- 1773 Views
- 4 replies
- 3 kudos
- 3 kudos
https://status.azuredatabricks.net/pages/incident/5d49ec10226b9e13cb6a422e/667c08fa17fef71767abda04"Degraded performance" is a pretty mild way of saying almost nothing productve can be done ATM...
- 3 kudos
- 744 Views
- 0 replies
- 0 kudos
DLT, combine LIVE table with data masking and row filter
I need to apply data masking and row filters to my table. At the same time I would like to use DLT Live tables. However, as far as I can see, DLT Live tables are not compatble with Live tables. What are my options? Move the tables from out of the mat...
- 744 Views
- 0 replies
- 0 kudos
- 1089 Views
- 1 replies
- 0 kudos
System Tables / Audit Logs action_name createWarehouse/createEndpoint
I am creating a cost dashboard across multiple accounts. I am working get sql warehouse names and warehouse ids so I can combine with system.access.billing on warehouse_id. But the only action_names that include both the warehouse_id and warehouse_n...
- 1089 Views
- 1 replies
- 0 kudos
- 0 kudos
I just wanted to circle back to this. It appears that the ID is returned in the response column of the create action_name.
- 0 kudos
- 1378 Views
- 1 replies
- 0 kudos
SPARK_GEN_SUBQ_0 WHERE 1=0, Error message from Server: Configuration schema is not available
Hi Community,I'm trying to read the data from sample schema from table nation from data-bricks catalog via spark but i'm getting this error.com.databricks.client.support.exceptions.GeneralException: [Databricks][JDBCDriver](500051) ERROR processing q...
- 1378 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Community,I'm still facing the issue can someone please provide me any solution how to fix above error.
- 0 kudos
- 956 Views
- 1 replies
- 0 kudos
Unity Catalog Shared compute Issues
Am I the only one experiencing challenges in migrating to Databricks Unity Catalog? I observed that in Unity Catalog-enabled compute, the "Shared" access mode is still tagged as a Preview feature. This means it is not yet safe for use in production w...
- 956 Views
- 1 replies
- 0 kudos
- 0 kudos
Have you tried creating a volume on top of the external location, and using the volume in spark.read.parquet?i.e. spark.read.parquet('/Volumes/<volume_name>/<folder_name>/<file_name.parquet>') Edit: also, not sure why the Databricks community mana...
- 0 kudos
- 691 Views
- 1 replies
- 1 kudos
Resolved! Is Datbricks-Salesforce already available to use?
Reference: Salesforce and Databricks Announce Strategic Partnership to Bring Lakehouse Data Sharing and Shared ...I was going through this article and wanted to know if this is already released. My assumption is that there’s no need to use third-part...
- 691 Views
- 1 replies
- 1 kudos
- 1 kudos
Looks like it has been released - Salesforce BYOM
- 1 kudos
- 599 Views
- 1 replies
- 0 kudos
How to use job.run_id as the running parameter of jar job to trigger job through REST API
"[,\"\{\{job.run_id\}\}\"]" {"error_code": "INVALID_PARAMETER_VALUE","message": "Legacy parameters cannot contain references."}
- 599 Views
- 1 replies
- 0 kudos
- 0 kudos
How to get the Job ID and Run ID in job runing?
- 0 kudos
- 2257 Views
- 1 replies
- 0 kudos
Get the triggering task's name
Hi,I have tasks that depend on each other. I would like to get variables from task1 that triggers task2.This is how I solved for my problem:Following suggestion in https://community.databricks.com/t5/data-engineering/how-to-pass-parameters-to-a-quot-...
- 2257 Views
- 1 replies
- 0 kudos
- 1896 Views
- 3 replies
- 2 kudos
Resolved! Autoloader to concatenate CSV files that updates regularly into a single parquet dataframe.
I have multiple large CSV files. One or more of these files changes now and then (a few times a day). The changes in the CSV files are both of type update and append (so both new rows) and updates of old. I want to combine all CSV files into a datafr...
- 1896 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Kjetil, Please let us know if you still have issue or if @-werners- response could be mark as a best solution. Thank you
- 2 kudos
- 1040 Views
- 1 replies
- 0 kudos
Variant datatype
I'm checking on variant datatype and noted that whenever a JSON string is stored as a variant datatype in order to filter and value it needs to be casted: i.eSELECT sum(jsondatavar:Value::double )FROM tableWHERE jsondatavar:customer ::int= 1000Here j...
- 1040 Views
- 1 replies
- 0 kudos
- 0 kudos
Could you please try using SQL functions: SELECT SUM(CAST(get_json_object(jsondatavar, '$.Value') AS DOUBLE)) AS total_value FROM table WHERE CAST(get_json_object(jsondatavar, '$.customer') AS INT) = 1000
- 0 kudos
- 6950 Views
- 11 replies
- 3 kudos
Delay in files update on filesystem
Hi, I noticed that there is quite a significant delay (2 - 10s) between making a change to some file in Repos via Databricks file edit window and propagation of such change to the filesystem. Our engineers and scientists use YAML config files. If the...
- 6950 Views
- 11 replies
- 3 kudos
- 3 kudos
Is there a solution to this?BTW, the "ls" command trick didn't work for me
- 3 kudos
- 1864 Views
- 3 replies
- 0 kudos
Overwriting a delta table using DLT
Hello,We are trying to ingest bunch of csv files that we receive on daily basis using DLT, we chose streaming table for this purpose since streaming table is append only records keep adding up on a daily basis which will cause multiple rows in downst...
- 1864 Views
- 3 replies
- 0 kudos
- 0 kudos
In your scenario, if the data loaded on day 2 also includes all the data from day 1, you can still apply a "remove duplicates" logic. For instance, you could compute a hashdiff by hashing all the columns and use this to exclude rows you've already se...
- 0 kudos
- 1908 Views
- 1 replies
- 1 kudos
Resolved! Read and process large CSV files that updates regularly
I've got a lot of large CSV files (> 1 GB) that updates regularly (stored in Data Lake Gen 2). The task is to concatenate these files into a single dataframe that is written to parquet format. However, since these files updates very often I get a rea...
- 1908 Views
- 1 replies
- 1 kudos
- 1 kudos
@Kjetil Since they are getting updated often then IMO making a copy would make sense.What you could try is to create Microsoft.Storage.BlobCreated event to replicate the .CSV into the secondary bucket.However, best practice would be to have some kind...
- 1 kudos
- 813 Views
- 0 replies
- 0 kudos
Understanding this Ipython related error in cluster logs
Hi Databricks Community !I'm having this error from a cluster's logs : [IPKernelApp] ERROR | Exception in control handler:Traceback (most recent call last):File "/databricks/python/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 334, in p...
- 813 Views
- 0 replies
- 0 kudos
- 4284 Views
- 4 replies
- 2 kudos
Cluster Failed to Start - Cluster scoped init scrip failed: Script exit status is non-zero
i have been using cluster scoped init script for around 1 year already and everything is working fine. But suddenly, Databricks cluster has failed to restart since last week Thursday (13th June 2024). It returns this error:” Failed to add 2 container...
- 4284 Views
- 4 replies
- 2 kudos
- 2 kudos
Just maybe - there is no outbound connection on DEV from Cluster VNET to URL you are trying to get ? You can spin ALl purpose cluster and try testing connection with %sh magic command
- 2 kudos
| User | Count |
|---|---|
| 1611 | |
| 768 | |
| 348 | |
| 286 | |
| 252 |
