- 4452 Views
- 2 replies
- 1 kudos
Can't import local files under repo
I have a yaml file inside one of the sub dir in Databricks, I have appended the repo path to sys. Still I can't access this file. https://docs.databricks.com/_static/notebooks/files-in-repos.html
- 4452 Views
- 2 replies
- 1 kudos
- 1 kudos
Hello @chandan_a_v ,were you able to solve this issue?I am also experiencing the same thing where i cannot move file with extension .yml from repo folder to shared workspace folder.As per documentation, this is the limitation or functionality of data...
- 1 kudos
- 1894 Views
- 1 replies
- 0 kudos
Delta live table is inserting data multiple times
So I have created a delta live table Which uses spark.sql() to execute a query And uses df.write.mode(append).insert intoTo insert data into the respective table And at the end i return a dumy table Since this was the requirement So now I have also ...
- 1894 Views
- 1 replies
- 0 kudos
- 0 kudos
whats your source? your sink is a delta table correct? how do you verify that there are no inserts happening?
- 0 kudos
- 10049 Views
- 6 replies
- 0 kudos
Resolved! Not able to create external table in a schema under a Catalog.
Problem StatementCluster 1 (Shared Cluster) is not able to read the file location at "dbfs:/mnt/landingzone/landingzonecontainer/Inbound/" and hence we are not able to create an external table in a schema inside Enterprise Catalog.Cluster 2 (No Isola...
- 10049 Views
- 6 replies
- 0 kudos
- 0 kudos
External table creation failing with error :- UnityCatalogServiceException:[RequestId=**** ErrorClass=INVALID_PARAMETER_VALUE] Unsupported path operation PATH_CREATE_TABLE on volume.Able to access and create files on external location.
- 0 kudos
- 2580 Views
- 1 replies
- 0 kudos
run md5 using CLI
Hi,I want to run a md5 checksum on the uploaded file to databricks. I can generate md5 on the local file but how do I generate one on uploaded file on databricks using CLI (Command line interface). Any help would be appreciated.I tried running databr...
- 2580 Views
- 1 replies
- 0 kudos
- 10401 Views
- 3 replies
- 1 kudos
Connect Databricks hosted on Azure, with RDS on AWS.
We have Databricks set up and running on Azure. Now we want to connect it with RDS (AWS) to transfer data from RDS to Azure DataLake using the Databricks.I could find the documentation on how to do it within the same cloud (Either AWS or Azure) but n...
- 10401 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Danial Malik​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so ...
- 1 kudos
- 2022 Views
- 0 replies
- 0 kudos
Capture Automatically Added tags
Can we capture automatically added tags (ex: RunName) from azure data bricks job cluster to parameters or custom tags in azure data factory
- 2022 Views
- 0 replies
- 0 kudos
- 5043 Views
- 4 replies
- 0 kudos
Shallow clone and issue with MODIFY permission to source table
Hi,I'm running shallow clone for external delta tables. The shallow clone is failing for source tables where I don't have MODIFY permission. I'm getting below exception. I don't understand why MODIFY permission to source table is required. Is there a...
- 5043 Views
- 4 replies
- 0 kudos
- 0 kudos
Also check this documentation on access mode :Shallow clone for Unity Catalog tables | Databricks on AWS Working with Unity Catalog shallow clones in Single User access mode, you must have permissions on the resources for the cloned table source as w...
- 0 kudos
- 3320 Views
- 1 replies
- 0 kudos
Fixed interval micro-batches and AvailableNow Trigger
What is the fundamental difference between Fixed interval micro-batches and AvailableNow Trigger, given that both can consume in micro-batch based on the desire size of the micro batch ? Is the fundamental difference the fact that AvailableNow shut d...
- 3320 Views
- 1 replies
- 0 kudos
- 14686 Views
- 9 replies
- 0 kudos
Pulsar Streaming (Read) - Benchmarking Information
We are doing a first time implementation of data streaming reading from a partitioned pulsar topics to a delta table managed by UC. We are unable to scale the job beyond about ~ 40k msgs/sec. Beyond 40k msgs/sec , the job fails. I'd imagine Databric...
- 14686 Views
- 9 replies
- 0 kudos
- 1160 Views
- 0 replies
- 0 kudos
Autoloader Failure Creating EventSubscription
Posting this here too in case anyone else has run into this issue... Trying to set up Autoloader File Notifications but keep getting an "Internal Server Error" message.Failure on Write EventSubscription - Internal error - Microsoft Q&A
- 1160 Views
- 0 replies
- 0 kudos
- 7793 Views
- 3 replies
- 0 kudos
Resolved! Iterating over a pyspark.pandas.groupby.DataFrameGroupBy
I have a pyspark.pandas.frame.DataFrame object (that I called from `pandas_api` on a pyspark.sql.dataframe.DataFrame object). I have a complicated transformation that I would like to apply to this data, and in particular I would like to apply it in ...
- 7793 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi,The error indicates that the Unity Catalog does not support Spark higher-order functions, such as those used in pandas_udf. This limitation likely comes from architectural or compatibility constraints. To resolve the issue, consider alternative ap...
- 0 kudos
- 1185 Views
- 0 replies
- 0 kudos
On Unity Catalog - what is the best way to adding members to groups
Hi All, On Unity Catalog - what is the best way to adding members to groups using API or CLI? API should be the best option, but thought to check with you all.
- 1185 Views
- 0 replies
- 0 kudos
- 1524 Views
- 1 replies
- 0 kudos
Unity Catalog External Tables
what if I delete the external tables files manually on storage account for external table without dropping the table itself?
- 1524 Views
- 1 replies
- 0 kudos
- 0 kudos
This change won't be registered as metadata thus the table will still appear on unity catalog, but you'll get an error when trying to access it, as the table metadata will point to deleted files.
- 0 kudos
- 1517 Views
- 0 replies
- 0 kudos
Application Deployment in Marketplace
Hi,I want to deploy my flask application in Databricks Marketplace.How can I do it?Can you please share the details
- 1517 Views
- 0 replies
- 0 kudos
- 8610 Views
- 5 replies
- 4 kudos
Is there a way to add Features descriptions for each of the features in a Feature Store table?
If not, then I believe that it will be beneficial because the feature tables contain engineered features that its a good idea to document their calc logic for the benefit of other data scientists. Also, even non-engineered features are many times no...
- 8610 Views
- 5 replies
- 4 kudos
- 4 kudos
I also would like to see support added for feature description get/set methods.
- 4 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
