- 17134 Views
- 2 replies
- 0 kudos
Intermittently unavailable: Maven library com.crealytics:spark-excel_2.12:3.5.0_0.20.3
The issue is that the package com.crealytics:spark-excel_2.12:3.5.0_0.20.3 is intermittently unavailable i.e. most of the times excel import works and few times it fails with exception (org.apache.spark.SparkClassNotFoundException).I have installed m...
- 17134 Views
- 2 replies
- 0 kudos
- 0 kudos
"Looks like the issue is source is not able to reach" - Can you please let me know what you mean by this.Libraries installed on the databricks cluster are as below, I have a cluster with14.2 version on which I have installed maven library(com.crealyt...
- 0 kudos
- 3063 Views
- 2 replies
- 0 kudos
"Last Access" information for external delta tables (no UC)
Hi,Is there a way to make audit on all tables in hive_metastore (no UC), all are external, to check when each has been used for the last time (queried / updated / etc). ?
- 3063 Views
- 2 replies
- 0 kudos
- 0 kudos
Apache Ranger or Apache Sentry can be used for auditing Hive activities. If you have set up auditing in one of these tools, you can review the audit logs to see when tables were accessed. Audit logs are typically stored in a separate location, and yo...
- 0 kudos
- 10142 Views
- 5 replies
- 2 kudos
Resolved! Rest API Error 404
I am trying to export a notebook or directory using /api/2.0/workspace/export.When I run /api/2.0/workspace/list with a particular url and path, I get the results that I expect, a list of objects (notebooks and folders) at that location.But when I ru...
- 10142 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi, Could you please remove the parameters , (format and direct_download) and confirm?
- 2 kudos
- 1605 Views
- 1 replies
- 0 kudos
Shared Mode Cluster Permission Issue: Editing Folders Across Users
Hi everyone,Currently, I save logs to a specific folder at the root level in Databricks. However, I need to use a Shared Mode cluster, and it seems I no longer have permission to save to the folder or even open its terminal to access the underlying i...
- 1605 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, If workspace access control is enabled, by default objects in this folder are private to that user. You can refer to https://docs.databricks.com/en/workspace/workspace-objects.html and let us know if this helps.
- 0 kudos
- 3104 Views
- 3 replies
- 1 kudos
databricks cli error : command >> databricks fs ls # getting error Error: accepts 1 arg(s), received
Hi team, I installed databricks cli on my mac using homebrew, below is the linkhttps://docs.databricks.com/en/dev-tools/cli/install.html#homebrew-installstep1:ran >> databricks configure , configured successfully.however, when I ran I am getting belo...
- 3104 Views
- 3 replies
- 1 kudos
- 1 kudos
Thanks for the reply, when I install databricks cli in my windows, it was actually returning some directories even with databricks fs ls.I installed in windows with pip. You think pip install different t from brew install in mac
- 1 kudos
- 2208 Views
- 1 replies
- 0 kudos
cloud fetch Qlik Sense
Hi Team,Cloud Fetch will improve data transfer efficiency from DataBricks to Power BI and is it compatible with Qlik Sense as well ?
- 2208 Views
- 1 replies
- 0 kudos
- 0 kudos
Cloud Fetch is a feature introduced in Databricks Runtime 8.3 and Simba ODBC 2.6.17 driver that significantly improves data transfer efficiency from Databricks to BI tools like Power BI. It achieves this by fetching data in parallel via cloud storage...
- 0 kudos
- 3041 Views
- 1 replies
- 0 kudos
Same path producing different counts on Databricks and EMR
We're in the middle of migrating to Databricks and found that the same path on s3 is producing different counts between EMR (Spark 2.4.4) and Databricks (Spark 3.4.1) it is a simple spark.read.parquet().count(), tried multiple solutions like making t...
- 3041 Views
- 1 replies
- 0 kudos
- 0 kudos
The discrepancy in counts between EMR (Spark 2.4.4) and Databricks (Spark 3.4.1) could be due to several reasons:1. Different versions of Spark: The two environments are running different versions of Spark which might have different optimizations or ...
- 0 kudos
- 8285 Views
- 1 replies
- 0 kudos
applyInPandas hangs on runtime 13.3 LTS ML and above
Hello, recently I've tried to upgrade my runtime env to the 13.3 LTS ML and found that it breaks my workload during applyInPandas.My job started to hang during the applyInPandas execution. Thread dump shows that it hangs on direct memory allocation: ...
- 8285 Views
- 1 replies
- 0 kudos
- 0 kudos
Same post: https://community.databricks.com/t5/data-engineering/applyinpandas-function-hangs-in-runtime-13-3-lts-ml-and-above/td-p/56795
- 0 kudos
- 11075 Views
- 3 replies
- 0 kudos
How to implement merge multiple rows in single row with array and do not result in OOM?
Hi, Everyone.Currently I try to implement spark structured streaming with Pyspark. And I would like to merge multiple rows in single row with array and sink to downstream message queue for another service to use. Related example can follow as:* Befor...
- 11075 Views
- 3 replies
- 0 kudos
- 0 kudos
Is there any solution to this, @MarsSu were you able to solve this, kindly shed some light on this if you resolve this.
- 0 kudos
- 3700 Views
- 0 replies
- 0 kudos
ipywidgets
I am having an issue with getting a display upon interaction with the ipywidgets dropdown menu. Once I've selected an option from the dropdown, nothing happens. I am inclined to believe it has to do with how I've structure my on_select function and n...
- 3700 Views
- 0 replies
- 0 kudos
- 8122 Views
- 4 replies
- 1 kudos
Resolved! Z-Ordering a Unity Catalog Materialized View
Hey everyone, We're making the move to Unity Catalog from Hive_Metastore and we're running into some issues performing Z-order optimizations on some of our tables. These tables are, in either place, materialized views created with a "create or refres...
- 8122 Views
- 4 replies
- 1 kudos
- 1 kudos
For anyone who's reading this later: You can still Z-order your materialized views, but you can't run it as a SQL command. Instead, you can set it as one of the TBLPROPERTIES when you define the table. Here's an example:create or refresh live table {...
- 1 kudos
- 4970 Views
- 4 replies
- 1 kudos
Apparent bug with dbutils.fs.cp on S3 using DBR 13.3LTS
If I use dbutils.fs.cp on a cluster running DBR 13.3LTS to try to copy an object on S3 from one prefix to another, I don't get the expected results.For example, if I try the following command:dbutils.fs.cp('s3://some-bucket/some/prefix/some_file.gz',...
- 4970 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @DManowitz-BAH , The correct syntax to use the dbutils.fs.cp command is to provide the file name in the destination path. Please check the document here: https://docs.databricks.com/en/dev-tools/databricks-utils.html#cp-command-dbutilsfscp
- 1 kudos
- 2502 Views
- 1 replies
- 0 kudos
Need to set parameter order and Note(Tooltip) on Databricks notebook task parameter
Hi Team,I have created a workflow job on Databricks which has 5 parameters. I created parameters using deploument.json like:"tasks": [{"task_key": "Test1","notebook_task": {"notebook_path": "Notebooks/F/UF/FileUpload","base_parameters": {"File_name":...
- 2502 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks for your reply @Retired_mod , will you help me understand which order Databricks uses to arrange parameters on UI so I can use that as a prefix to order parameters? Like ASCII, binary or something else?
- 0 kudos
- 3391 Views
- 1 replies
- 0 kudos
Undrop a table from built-in catalogs Azure Databricks
How can I undrop a table from a built-in catalog in Azure Databricks
- 3391 Views
- 1 replies
- 0 kudos
- 0 kudos
If you are using Unity Catalog, you can simply run the UnDrop command. Ref Doc:- https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-undrop-table.html
- 0 kudos
- 4203 Views
- 1 replies
- 2 kudos
Unity Catalog and Data Accessibility
Hi,I got a few question about the internals of #Unity Catalog in #Databricks1. Understand that we can customize the UC metastore at different levels (catalog/schema). Wondering where is the information about UC permission model stored for every data ...
- 4203 Views
- 1 replies
- 2 kudos
- 2 kudos
thank you @Retired_mod ,your response really helps. A quick follow up - when Unity Catalog uses its permissions to access objects across workspaces, what kind of connection method does it use to access the data object i.e. in this case, when User Y q...
- 2 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
