- 9778 Views
- 6 replies
- 0 kudos
Resolved! Not able to create external table in a schema under a Catalog.
Problem StatementCluster 1 (Shared Cluster) is not able to read the file location at "dbfs:/mnt/landingzone/landingzonecontainer/Inbound/" and hence we are not able to create an external table in a schema inside Enterprise Catalog.Cluster 2 (No Isola...
- 9778 Views
- 6 replies
- 0 kudos
- 0 kudos
External table creation failing with error :- UnityCatalogServiceException:[RequestId=**** ErrorClass=INVALID_PARAMETER_VALUE] Unsupported path operation PATH_CREATE_TABLE on volume.Able to access and create files on external location.
- 0 kudos
- 2493 Views
- 1 replies
- 0 kudos
run md5 using CLI
Hi,I want to run a md5 checksum on the uploaded file to databricks. I can generate md5 on the local file but how do I generate one on uploaded file on databricks using CLI (Command line interface). Any help would be appreciated.I tried running databr...
- 2493 Views
- 1 replies
- 0 kudos
- 10152 Views
- 3 replies
- 1 kudos
Connect Databricks hosted on Azure, with RDS on AWS.
We have Databricks set up and running on Azure. Now we want to connect it with RDS (AWS) to transfer data from RDS to Azure DataLake using the Databricks.I could find the documentation on how to do it within the same cloud (Either AWS or Azure) but n...
- 10152 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Danial Malik​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so ...
- 1 kudos
- 1981 Views
- 0 replies
- 0 kudos
Capture Automatically Added tags
Can we capture automatically added tags (ex: RunName) from azure data bricks job cluster to parameters or custom tags in azure data factory
- 1981 Views
- 0 replies
- 0 kudos
- 4846 Views
- 4 replies
- 0 kudos
Shallow clone and issue with MODIFY permission to source table
Hi,I'm running shallow clone for external delta tables. The shallow clone is failing for source tables where I don't have MODIFY permission. I'm getting below exception. I don't understand why MODIFY permission to source table is required. Is there a...
- 4846 Views
- 4 replies
- 0 kudos
- 0 kudos
Also check this documentation on access mode :Shallow clone for Unity Catalog tables | Databricks on AWS Working with Unity Catalog shallow clones in Single User access mode, you must have permissions on the resources for the cloned table source as w...
- 0 kudos
- 3228 Views
- 1 replies
- 0 kudos
Fixed interval micro-batches and AvailableNow Trigger
What is the fundamental difference between Fixed interval micro-batches and AvailableNow Trigger, given that both can consume in micro-batch based on the desire size of the micro batch ? Is the fundamental difference the fact that AvailableNow shut d...
- 3228 Views
- 1 replies
- 0 kudos
- 13375 Views
- 9 replies
- 0 kudos
Pulsar Streaming (Read) - Benchmarking Information
We are doing a first time implementation of data streaming reading from a partitioned pulsar topics to a delta table managed by UC. We are unable to scale the job beyond about ~ 40k msgs/sec. Beyond 40k msgs/sec , the job fails. I'd imagine Databric...
- 13375 Views
- 9 replies
- 0 kudos
- 1116 Views
- 0 replies
- 0 kudos
Autoloader Failure Creating EventSubscription
Posting this here too in case anyone else has run into this issue... Trying to set up Autoloader File Notifications but keep getting an "Internal Server Error" message.Failure on Write EventSubscription - Internal error - Microsoft Q&A
- 1116 Views
- 0 replies
- 0 kudos
- 7600 Views
- 3 replies
- 0 kudos
Resolved! Iterating over a pyspark.pandas.groupby.DataFrameGroupBy
I have a pyspark.pandas.frame.DataFrame object (that I called from `pandas_api` on a pyspark.sql.dataframe.DataFrame object). I have a complicated transformation that I would like to apply to this data, and in particular I would like to apply it in ...
- 7600 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi,The error indicates that the Unity Catalog does not support Spark higher-order functions, such as those used in pandas_udf. This limitation likely comes from architectural or compatibility constraints. To resolve the issue, consider alternative ap...
- 0 kudos
- 1159 Views
- 0 replies
- 0 kudos
On Unity Catalog - what is the best way to adding members to groups
Hi All, On Unity Catalog - what is the best way to adding members to groups using API or CLI? API should be the best option, but thought to check with you all.
- 1159 Views
- 0 replies
- 0 kudos
- 1465 Views
- 1 replies
- 0 kudos
Unity Catalog External Tables
what if I delete the external tables files manually on storage account for external table without dropping the table itself?
- 1465 Views
- 1 replies
- 0 kudos
- 0 kudos
This change won't be registered as metadata thus the table will still appear on unity catalog, but you'll get an error when trying to access it, as the table metadata will point to deleted files.
- 0 kudos
- 1461 Views
- 0 replies
- 0 kudos
Application Deployment in Marketplace
Hi,I want to deploy my flask application in Databricks Marketplace.How can I do it?Can you please share the details
- 1461 Views
- 0 replies
- 0 kudos
- 8348 Views
- 5 replies
- 4 kudos
Is there a way to add Features descriptions for each of the features in a Feature Store table?
If not, then I believe that it will be beneficial because the feature tables contain engineered features that its a good idea to document their calc logic for the benefit of other data scientists. Also, even non-engineered features are many times no...
- 8348 Views
- 5 replies
- 4 kudos
- 4 kudos
I also would like to see support added for feature description get/set methods.
- 4 kudos
- 27591 Views
- 8 replies
- 3 kudos
UnauthorizedAccessException: PERMISSION_DENIED: User does not have READ FILES on External Location
Our jobs have been running fine so far w/o any issues on a specific workspace. These jobs read data from files on Azure ADLS storage containers and dont use the hive metastore data at all.Now we attached the unity metastore to this workspace, created...
- 27591 Views
- 8 replies
- 3 kudos
- 3 kudos
@Wojciech_BUK I granted both in the GUI:) you can either search for display name there (and then it uses the Managed Identity Object ID), or you can search directly for the value of the "Managed Identity Application ID" and then it works correctly! ...
- 3 kudos
- 1422 Views
- 0 replies
- 1 kudos
Importing JSON files when format is subject to evolution
Hi there,I'm reaching out for some assistance with importing JSON files into Databricks. Still a beginner even if I've gained experience working with various data import batches (CSV/JSON) for application monitoring: I'm currently facing a challenge...
- 1422 Views
- 0 replies
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 580 | |
| 349 | |
| 287 |
