cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

hbs59
by New Contributor III
  • 6524 Views
  • 5 replies
  • 2 kudos

Resolved! Rest API Error 404

I am trying to export a notebook or directory using /api/2.0/workspace/export.When I run /api/2.0/workspace/list with a particular url and path, I get the results that I expect, a list of objects (notebooks and folders) at that location.But when I ru...

  • 6524 Views
  • 5 replies
  • 2 kudos
Latest Reply
Debayan
Databricks Employee
  • 2 kudos

Hi, Could you please remove the parameters , (format and direct_download) and confirm? 

  • 2 kudos
4 More Replies
drii_cavalcanti
by New Contributor III
  • 1069 Views
  • 1 replies
  • 0 kudos

Shared Mode Cluster Permission Issue: Editing Folders Across Users

Hi everyone,Currently, I save logs to a specific folder at the root level in Databricks. However, I need to use a Shared Mode cluster, and it seems I no longer have permission to save to the folder or even open its terminal to access the underlying i...

  • 1069 Views
  • 1 replies
  • 0 kudos
Latest Reply
Debayan
Databricks Employee
  • 0 kudos

Hi, If workspace access control is enabled, by default objects in this folder are private to that user. You can refer to https://docs.databricks.com/en/workspace/workspace-objects.html and let us know if this helps. 

  • 0 kudos
therealDE
by New Contributor II
  • 2024 Views
  • 3 replies
  • 1 kudos

databricks cli error : command >> databricks fs ls # getting error Error: accepts 1 arg(s), received

Hi team, I installed databricks cli on my mac using homebrew, below is the linkhttps://docs.databricks.com/en/dev-tools/cli/install.html#homebrew-installstep1:ran >> databricks configure , configured successfully.however, when I ran I am getting belo...

  • 2024 Views
  • 3 replies
  • 1 kudos
Latest Reply
therealDE
New Contributor II
  • 1 kudos

Thanks for the reply, when I install databricks cli in my windows, it was actually returning some directories even with databricks fs ls.I installed in windows with pip. You think pip install different t from brew install in mac

  • 1 kudos
2 More Replies
Phani1
by Valued Contributor II
  • 1671 Views
  • 1 replies
  • 0 kudos

cloud fetch Qlik Sense

 Hi Team,Cloud Fetch will improve data transfer efficiency from DataBricks to Power BI and is it compatible with Qlik Sense as well ?

  • 1671 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Cloud Fetch is a feature introduced in Databricks Runtime 8.3 and Simba ODBC 2.6.17 driver that significantly improves data transfer efficiency from Databricks to BI tools like Power BI. It achieves this by fetching data in parallel via cloud storage...

  • 0 kudos
Vishwanath_Rao
by New Contributor II
  • 2211 Views
  • 1 replies
  • 0 kudos

Same path producing different counts on Databricks and EMR

We're in the middle of migrating to Databricks and found that the same path on s3 is producing different counts between EMR (Spark 2.4.4) and Databricks (Spark 3.4.1) it is a simple spark.read.parquet().count(), tried multiple solutions like making t...

  • 2211 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

The discrepancy in counts between EMR (Spark 2.4.4) and Databricks (Spark 3.4.1) could be due to several reasons:1. Different versions of Spark: The two environments are running different versions of Spark which might have different optimizations or ...

  • 0 kudos
DmitriyLamzin
by New Contributor II
  • 6488 Views
  • 1 replies
  • 0 kudos

applyInPandas hangs on runtime 13.3 LTS ML and above

Hello, recently I've tried to upgrade my runtime env to the 13.3 LTS ML and found that it breaks my workload during applyInPandas.My job started to hang during the applyInPandas execution. Thread dump shows that it hangs on direct memory allocation: ...

Data Engineering
pandas PythonRunner
  • 6488 Views
  • 1 replies
  • 0 kudos
Latest Reply
Debayan
Databricks Employee
  • 0 kudos

Same post: https://community.databricks.com/t5/data-engineering/applyinpandas-function-hangs-in-runtime-13-3-lts-ml-and-above/td-p/56795

  • 0 kudos
MarsSu
by New Contributor II
  • 9137 Views
  • 3 replies
  • 0 kudos

How to implement merge multiple rows in single row with array and do not result in OOM?

Hi, Everyone.Currently I try to implement spark structured streaming with Pyspark. And I would like to merge multiple rows in single row with array and sink to downstream message queue for another service to use. Related example can follow as:* Befor...

  • 9137 Views
  • 3 replies
  • 0 kudos
Latest Reply
917074
New Contributor II
  • 0 kudos

Is there any solution to this, @MarsSu  were you able to solve this, kindly shed some light on this if you resolve this.

  • 0 kudos
2 More Replies
uncle_rufus
by New Contributor
  • 3182 Views
  • 0 replies
  • 0 kudos

ipywidgets

I am having an issue with getting a display upon interaction with the ipywidgets dropdown menu. Once I've selected an option from the dropdown, nothing happens. I am inclined to believe it has to do with how I've structure my on_select function and n...

  • 3182 Views
  • 0 replies
  • 0 kudos
Jake2
by New Contributor III
  • 6013 Views
  • 4 replies
  • 1 kudos

Resolved! Z-Ordering a Unity Catalog Materialized View

Hey everyone, We're making the move to Unity Catalog from Hive_Metastore and we're running into some issues performing Z-order optimizations on some of our tables. These tables are, in either place, materialized views created with a "create or refres...

  • 6013 Views
  • 4 replies
  • 1 kudos
Latest Reply
Jake2
New Contributor III
  • 1 kudos

For anyone who's reading this later: You can still Z-order your materialized views, but you can't run it as a SQL command. Instead, you can set it as one of the TBLPROPERTIES when you define the table. Here's an example:create or refresh live table {...

  • 1 kudos
3 More Replies
DManowitz-BAH
by New Contributor II
  • 3625 Views
  • 4 replies
  • 1 kudos

Apparent bug with dbutils.fs.cp on S3 using DBR 13.3LTS

If I use dbutils.fs.cp on a cluster running DBR 13.3LTS to try to copy an object on S3 from one prefix to another, I don't get the expected results.For example, if I try the following command:dbutils.fs.cp('s3://some-bucket/some/prefix/some_file.gz',...

  • 3625 Views
  • 4 replies
  • 1 kudos
Latest Reply
Lakshay
Databricks Employee
  • 1 kudos

Hi @DManowitz-BAH , The correct syntax to use the dbutils.fs.cp command is to provide the file name in the destination path. Please check the document here: https://docs.databricks.com/en/dev-tools/databricks-utils.html#cp-command-dbutilsfscp

  • 1 kudos
3 More Replies
JagadishMori
by New Contributor II
  • 1691 Views
  • 1 replies
  • 0 kudos

Need to set parameter order and Note(Tooltip) on Databricks notebook task parameter

Hi Team,I have created a workflow job on Databricks which has 5 parameters. I created parameters using deploument.json like:"tasks": [{"task_key": "Test1","notebook_task": {"notebook_path": "Notebooks/F/UF/FileUpload","base_parameters": {"File_name":...

  • 1691 Views
  • 1 replies
  • 0 kudos
Latest Reply
JagadishMori
New Contributor II
  • 0 kudos

Thanks for your reply @Retired_mod , will you help me understand which order Databricks uses to arrange parameters on UI so I can use that as a prefix to order parameters? Like ASCII, binary or something else?

  • 0 kudos
Rags98
by New Contributor II
  • 2493 Views
  • 1 replies
  • 0 kudos

Undrop a table from built-in catalogs Azure Databricks

How can I undrop a table from a built-in catalog in Azure Databricks

  • 2493 Views
  • 1 replies
  • 0 kudos
Latest Reply
Lakshay
Databricks Employee
  • 0 kudos

If you are using Unity Catalog, you can simply run the UnDrop command. Ref Doc:- https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-undrop-table.html

  • 0 kudos
SenthilJ
by New Contributor III
  • 3477 Views
  • 1 replies
  • 2 kudos

Unity Catalog and Data Accessibility

Hi,I got a few question about the internals of #Unity Catalog in #Databricks1. Understand that we can customize the UC metastore at different levels (catalog/schema). Wondering where is the information about UC permission model stored for every data ...

Data Engineering
Databricks
Unity Catalog
  • 3477 Views
  • 1 replies
  • 2 kudos
Latest Reply
SenthilJ
New Contributor III
  • 2 kudos

thank you @Retired_mod ,your response really helps. A quick follow up - when Unity Catalog uses its permissions to access objects across workspaces, what kind of connection method does it use to access the data object i.e. in this case, when User Y q...

  • 2 kudos
Simon_T
by New Contributor III
  • 2617 Views
  • 1 replies
  • 0 kudos

CURL API - Error while parsing token: io.jsonwebtoken.ExpiredJwtException: JWT expired

I am running this code:curl -X --request GET -H "Authorization: Bearer <databricks token>" "https://adb-1817728758721967.7.azuredatabricks.net/api/2.0/clusters/list"And I am getting this error:2024-01-17T13:21:41.4245092Z </head>2024-01-17T13:21:41.4...

  • 2617 Views
  • 1 replies
  • 0 kudos
Latest Reply
Debayan
Databricks Employee
  • 0 kudos

Hi, Could you please renew the token and confirm? 

  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels