- 2895 Views
- 2 replies
- 1 kudos
Resolved! Databricks Asset Bundles: issue with python_file path in spark_python_task
Hi,I am trying to run a python file which is stored in src folder. However, I am getting the following error: Error: cannot update job: Invalid python file reference: src/get_git_credentials.py. Please visit the Databricks user guide for supported py...
- 2895 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @szymon_dybczak,Thanks for you reply. The issue is related to the fact if am also providing git_source parameters in this job (even though the source parameter of my task is set to WORKSPACE). When I comment the git_source part, specifying the pat...
- 1 kudos
- 679 Views
- 1 replies
- 0 kudos
Resolved! Unable to login to community edition
Hi All,I am not able to login into "databricks community edition". When I hit sign up its redirecting me into "free edition" as there’s currently no way to create or enable clusters in the Databricks Free Edition. Please suggest someway where I can u...
- 679 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @SahabazKhan! If you’re trying to create a new Community Edition account, that’s not possible, it will redirect you to create a Free Edition account instead. In the Free Edition, you’ll have access to serverless compute and other features. Want...
- 0 kudos
- 2596 Views
- 8 replies
- 3 kudos
Resolved! Error in Viewing the Table
Facing below error while accessing a table with mutiple row filters but i am not able to delete the row filter using SQL query. Please help Failed to request /ajax-api/2.1/unity-catalog/tables/product_return_prediction_dev.bronze.customers_data?incl...
- 2596 Views
- 8 replies
- 3 kudos
- 3 kudos
Hi @pranaav93 @szymon_dybczak below query worked for me "DROP POLICY city_filter_policy ON product_return_prediction_dev.bronze.customers_data"
- 3 kudos
- 3177 Views
- 5 replies
- 0 kudos
[INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have permission MODIFY on any file
Just wondering if anyone could help me understand why we are hitting this error: `[INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have permission MODIFY on any file`A job is trying to create a table with an external location (alread...
- 3177 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @rachelh As I understand , you need to look for azure access connector setup for your unity catalog because Serverless clusters run under a Azure Databricks-managed identity, not the service principal.Access Connector (Azure Managed Identity): Use...
- 0 kudos
- 2419 Views
- 2 replies
- 2 kudos
Capturing Streaming Metrics in Near Real-Time Using Cluster Logs
Over the past few weeks, I’ve been exploring ways to capture streaming metrics from our data load jobs. The goal is to monitor job performance and behavior in real time, without disrupting our existing data load pipelines.Initial Exploration: Streami...
- 2419 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @saicharandeepb Good job on doing such detailed research on monitoring structured streaming. If you need lower latency than rolling log permits, then have you tried this:Cluster-wide listener injection: Use spark.extraListeners to register a cust...
- 2 kudos
- 1498 Views
- 1 replies
- 1 kudos
Resolved! AttributeError: module 'numpy' has no attribute 'typing'
We started experiencing failures in several Databricks jobs without any changes on our side. The error occurs during Python job execution and seems related to package dependencies.The job error:Run failed with error message Cannot read the python fil...
- 1498 Views
- 1 replies
- 1 kudos
- 1 kudos
The problem was numexpr lib version, 2.14.0, I needed to pin 2.13.1
- 1 kudos
- 1518 Views
- 2 replies
- 1 kudos
Resolved! Data profiling monitoring with foreign catalog
Hi team,I’m currently working with Azure Databricks and have created a foreign catalog for my source database in Azure SQL. I can successfully run SELECT statements from Databricks to the Azure SQL database.However, I would like to set up data profil...
- 1518 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @sta_gas ,Since data quality monitoring is in beta I'm quite sure they don't support foreign tables as of now (but they forgot to mentioned it in docs).But more important question if they ever will be supported. For me data quality monitoring appl...
- 1 kudos
- 1081 Views
- 2 replies
- 1 kudos
Wheel permissions issue
I get a : org.apache.spark.SparkSecurityException: [INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have permission MODIFY,SELECT on any file. SQLSTATE: 42501 at com.databricks.sql.acl.Unauthorized.throwInsufficientPermissionsError(P...
- 1081 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @adrianhernandez , The permissions error indicates you need to have the privileges for "any file". To resolve this, Can you try by adding the corresponding permissions and see if it works: %sql GRANT SELECT ON ANY FILE TO `username` %sql GRANT MO...
- 1 kudos
- 4851 Views
- 4 replies
- 2 kudos
Resolved! Executing Bash Scripts or Binaries Directly in Databricks Jobs on Single Node Cluster
Hi,Is it possible to directly execute a Bash script or a binary executable from the operating system of a Databricks job compute node using a single node cluster?I’m using databricks asset bundels for job initialization and execution. When the job s...
- 4851 Views
- 4 replies
- 2 kudos
- 2 kudos
Hello @jorperort , I did some research internally and have some tips/suggestions for you to consider: Based on the research and available documentation, it is not possible to directly execute a Bash script or binary executable from the operating sy...
- 2 kudos
- 4853 Views
- 5 replies
- 0 kudos
Enabling enableChangeDataFeed on Streaming Table created in DLT
Hello, Can I enable Change Data Feed on Streaming Tables? How should I do this? I couldn't find this in the existing documentation https://learn.microsoft.com/en-us/azure/databricks/delta/delta-change-data-feed .
- 4853 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @Vsleg i think you cannot enable cdf like this for streaming tables. it is not natively supported for DLT streaming tables , please have a look here = Propagating Deletes: Managing Data Removal using D... - Databricks Community - 90978
- 0 kudos
- 1559 Views
- 3 replies
- 1 kudos
Unable to configure clustering on DLT tables
Hi TeamI have a DLT pipeline with `cluster_by` property configured for all my tables. The code looks something like below:@Dlt.table( name="flows", cluster_by=["from"] ) def flows(): <LOGIC>It was all working fine and in couple of days, the queries w...
- 1559 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Chris_N , You have mentioned - "I couldn't find any cluster properties configured." If they existed and were changed, you can use the delta history command to check if someone changed on the clustering information. It is possible there were ch...
- 1 kudos
- 2745 Views
- 1 replies
- 2 kudos
Best approach for writing/updating delta tables from python?
Hi,We are migrating a local dash app to the Databricks infrastructure (using databricks apps and our delta lake).The local app does the following (among others):- takes Excel files form the end-user- read in-memory and transforms to pandas dataframe-...
- 2745 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @georgemichael40 My suggestion wud be to try using MERGE INTO for delta tables which works with connector then using delete/insert statements. This will also keep your code in SQL as you wanted. your tables are not large so this shud be sufficient...
- 2 kudos
- 2685 Views
- 2 replies
- 0 kudos
Salesforce Bulk API 2.0 not getting all rows from large table
Has anyone run into an incomplete data extraction issue with the Salesforce Bulk API 2.0 where very large source object tables with more than 260k rows (s/b approx 13M) - result in only extracting approx 250k on attempt?
- 2685 Views
- 2 replies
- 0 kudos
- 0 kudos
@AlanDanque I am working on a similar use case and will share screen shots shortlyBut to reach the root cause can you share the below detailsChecks at SalesforceDescriptionHeader used?Was Sforce-Enable-PKChunking: chunkSize=250000 explicitly included...
- 0 kudos
- 3050 Views
- 4 replies
- 4 kudos
Resolved! Node type not available in Central India (Student Subscription)
Hi Community,I have deployed an Azure Databricks workspace in the Central India region using a student subscription. While trying to create a compute resource, I encountered an error stating that the selected node type is not available in Central Ind...
- 3050 Views
- 4 replies
- 4 kudos
- 4 kudos
@Abrarali8708 As discussed can you trymanaging the Azure Policy definition:Locate the policy definition ID /providers/Microsoft.Authorization/policyDefinitions/b86dabb9-b578-4d7b-b842-3b45e95769a1.Modify the parameter listOfAllowedLocations to inclu...
- 4 kudos
- 2978 Views
- 3 replies
- 3 kudos
Resolved! To change ownership of a materialized view
working in a Unity Catalog-enabled Databricks workspace, and we have several materialized views (MVs) that were created through a Delta Live Tables (DLT) / Lakeflow pipeline.Currently, the original owner of the pipeline has moved out of the project,...
- 2978 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Data_NXT ,You can change the owner of a materialized view if you are a both a metastore admin and a workspace admin.Use the following steps to change a materialized views owner:Open the materialized view in Catalog Explorer, then on the Overview ...
- 3 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
