I need to apply data masking and row filters to my table. At the same time I would like to use DLT Live tables. However, as far as I can see, DLT Live tables are not compatble with Live tables. What are my options? Move the tables from out of the mat...
I am creating a cost dashboard across multiple accounts. I am working get sql warehouse names and warehouse ids so I can combine with system.access.billing on warehouse_id. But the only action_names that include both the warehouse_id and warehouse_n...
I just wanted to circle back to this. It appears that the ID is returned in the response column of the create action_name.
Hi Community,I'm trying to read the data from sample schema from table nation from data-bricks catalog via spark but i'm getting this error.com.databricks.client.support.exceptions.GeneralException: [Databricks][JDBCDriver](500051) ERROR processing q...
Hi Community,I'm still facing the issue can someone please provide me any solution how to fix above error.
Am I the only one experiencing challenges in migrating to Databricks Unity Catalog? I observed that in Unity Catalog-enabled compute, the "Shared" access mode is still tagged as a Preview feature. This means it is not yet safe for use in production w...
Have you tried creating a volume on top of the external location, and using the volume in spark.read.parquet?i.e. spark.read.parquet('/Volumes/<volume_name>/<folder_name>/<file_name.parquet>') Edit: also, not sure why the Databricks community mana...
Reference: Salesforce and Databricks Announce Strategic Partnership to Bring Lakehouse Data Sharing and Shared ...I was going through this article and wanted to know if this is already released. My assumption is that there’s no need to use third-part...
Looks like it has been released - Salesforce BYOM
"[,\"\{\{job.run_id\}\}\"]" {"error_code": "INVALID_PARAMETER_VALUE","message": "Legacy parameters cannot contain references."}
How to get the Job ID and Run ID in job runing?
Hi,I have tasks that depend on each other. I would like to get variables from task1 that triggers task2.This is how I solved for my problem:Following suggestion in https://community.databricks.com/t5/data-engineering/how-to-pass-parameters-to-a-quot-...
I have multiple large CSV files. One or more of these files changes now and then (a few times a day). The changes in the CSV files are both of type update and append (so both new rows) and updates of old. I want to combine all CSV files into a datafr...
Hi @Kjetil, Please let us know if you still have issue or if @-werners- response could be mark as a best solution. Thank you
I'm checking on variant datatype and noted that whenever a JSON string is stored as a variant datatype in order to filter and value it needs to be casted: i.eSELECT sum(jsondatavar:Value::double )FROM tableWHERE jsondatavar:customer ::int= 1000Here j...
Could you please try using SQL functions: SELECT SUM(CAST(get_json_object(jsondatavar, '$.Value') AS DOUBLE)) AS total_value FROM table WHERE CAST(get_json_object(jsondatavar, '$.customer') AS INT) = 1000
Hello,We are trying to ingest bunch of csv files that we receive on daily basis using DLT, we chose streaming table for this purpose since streaming table is append only records keep adding up on a daily basis which will cause multiple rows in downst...
In your scenario, if the data loaded on day 2 also includes all the data from day 1, you can still apply a "remove duplicates" logic. For instance, you could compute a hashdiff by hashing all the columns and use this to exclude rows you've already se...
I've got a lot of large CSV files (> 1 GB) that updates regularly (stored in Data Lake Gen 2). The task is to concatenate these files into a single dataframe that is written to parquet format. However, since these files updates very often I get a rea...
@Kjetil Since they are getting updated often then IMO making a copy would make sense.What you could try is to create Microsoft.Storage.BlobCreated event to replicate the .CSV into the secondary bucket.However, best practice would be to have some kind...
Hi Databricks Community !I'm having this error from a cluster's logs : [IPKernelApp] ERROR | Exception in control handler:Traceback (most recent call last):File "/databricks/python/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 334, in p...
i have been using cluster scoped init script for around 1 year already and everything is working fine. But suddenly, Databricks cluster has failed to restart since last week Thursday (13th June 2024). It returns this error:” Failed to add 2 container...
Just maybe - there is no outbound connection on DEV from Cluster VNET to URL you are trying to get ? You can spin ALl purpose cluster and try testing connection with %sh magic command
Hi Community,I need help regarding the class not found issue. I'm trying to connect databricks in python via jaydebeapi, provided proper class name `com.databricks.client.jdbc.Driver` and jarpath `databricks-jdbc-2.6.34.jar`, but i'm getting com.data...
Hello @HASSAN_UPPAL123 The class name is correct, for the jar please try downloading the latest from here This issue may also be a classpath issue were the jar is not exported correctly in your client setup, I see similar issues/ suggested solutions ...
Hi,I want to export notebook in python programming.is there a way to leverage databricks cli in python.Or any other way to export the notebook to my local PC.
Is there a way to export a notebook though terraform? If so please provide examples?With terraform-provider-databricks.exe we are able to export all the notebooks from workspace but not a single notebook. Any suggestions ?
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 573 | |
| 349 | |
| 287 |
