cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

stevewb
by New Contributor III
  • 12 Views
  • 0 replies
  • 0 kudos

Errors in runtime 17 today

Anyone else getting a bunch of errors on runtime 17 today? A load of our pipelines that were running smoothly suddenly stopped working with driver crashes. I was able to get us running again by downgrading to runtime 16, but curious if anyone else hi...

  • 12 Views
  • 0 replies
  • 0 kudos
donlxz
by New Contributor
  • 16 Views
  • 1 replies
  • 0 kudos

Error occurs on create materialized view with spark.sql

When creating materialized view with spark.sql function it returns following error message.[MATERIALIZED_VIEW_OPERATION_NOT_ALLOWED.MV_NOT_ENABLED] The materialized view operation CREATE is not allowed: Materialized view features are not enabled for ...

  • 16 Views
  • 1 replies
  • 0 kudos
Latest Reply
szymon_dybczak
Esteemed Contributor III
  • 0 kudos

Hi @donlxz ,When you look at documentation, they only available way to create materialized view is by executing CREATE MATERIALIZED VIEW statement. You can submit that statement via SQL editor, Azure Databricks UI, Databricsk CLI or Databricks SQL AP...

  • 0 kudos
SahabazKhan
by Visitor
  • 14 Views
  • 1 replies
  • 0 kudos

Unable to login to community edition

Hi All,I am not able to login into "databricks community edition". When I hit sign up its redirecting me into "free edition" as there’s currently no way to create or enable clusters in the Databricks Free Edition. Please suggest someway where I can u...

  • 14 Views
  • 1 replies
  • 0 kudos
Latest Reply
Advika
Databricks Employee
  • 0 kudos

Hello @SahabazKhan! If you’re trying to create a new Community Edition account, that’s not possible, it will redirect you to create a Free Edition account instead. In the Free Edition, you’ll have access to serverless compute and other features. Want...

  • 0 kudos
vamsi_simbus
by New Contributor III
  • 52 Views
  • 8 replies
  • 1 kudos

Error in Viewing the Table

Facing below error while accessing a table with mutiple row filters but i am not able to delete the row filter using SQL query. Please help  Failed to request /ajax-api/2.1/unity-catalog/tables/product_return_prediction_dev.bronze.customers_data?incl...

  • 52 Views
  • 8 replies
  • 1 kudos
Latest Reply
vamsi_simbus
New Contributor III
  • 1 kudos

Hi @pranaav93  @szymon_dybczak  below query worked for me "DROP POLICY city_filter_policy ON product_return_prediction_dev.bronze.customers_data"

  • 1 kudos
7 More Replies
rachelh
by New Contributor
  • 101 Views
  • 5 replies
  • 0 kudos

[INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have permission MODIFY on any file

Just wondering if anyone could help me understand why we are hitting this error: `[INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have permission MODIFY on any file`A job is trying to create a table with an external location (alread...

  • 101 Views
  • 5 replies
  • 0 kudos
Latest Reply
saurabh18cs
Honored Contributor II
  • 0 kudos

Hi @rachelh As I understand , you need to look for azure access connector setup for your unity catalog because Serverless clusters run under a Azure Databricks-managed identity, not the service principal.Access Connector (Azure Managed Identity): Use...

  • 0 kudos
4 More Replies
pranaav93
by New Contributor II
  • 16 Views
  • 0 replies
  • 0 kudos

Databricks Compute Metrics Alerts

Hi All,Im looking for some implementation ideas where i can use information from the system.compute.node_timeline table to catch memory spikes and if above a given threshold restart the cluster through an API call. Have any of you implemented a simil...

  • 16 Views
  • 0 replies
  • 0 kudos
saicharandeepb
by New Contributor III
  • 188 Views
  • 2 replies
  • 1 kudos

Capturing Streaming Metrics in Near Real-Time Using Cluster Logs

Over the past few weeks, I’ve been exploring ways to capture streaming metrics from our data load jobs. The goal is to monitor job performance and behavior in real time, without disrupting our existing data load pipelines.Initial Exploration: Streami...

saicharandeepb_0-1760081131866.png
  • 188 Views
  • 2 replies
  • 1 kudos
Latest Reply
Krishna_S
Databricks Employee
  • 1 kudos

Hi @saicharandeepb  Good job on doing such detailed research on monitoring structured streaming. If you need lower latency than rolling log permits, then have you tried this:Cluster-wide listener injection: Use spark.extraListeners to register a cust...

  • 1 kudos
1 More Replies
j_unspeakable
by New Contributor III
  • 1159 Views
  • 3 replies
  • 3 kudos

Resolved! Permission Denied when Creating External Tables Using Workspace Default Credential

I’m building out schemas, volumes, and external Delta tables in Unity Catalog via Terraform. The schemas and volumes are created successfully, but all external tables are failing.The error message from Terraform doesn't highlight what the issue is bu...

image.png image.png Screenshot 2025-06-15 152848.png
  • 1159 Views
  • 3 replies
  • 3 kudos
Latest Reply
artopihlaja
  • 3 kudos

Feature or bug, I discovered the same. I couldn't create tables with the default credential. To test, I assigned the default credential and a custom credential the same access rights to the storage container that is the target of the external locatio...

  • 3 kudos
2 More Replies
MarcoRezende
by New Contributor III
  • 70 Views
  • 1 replies
  • 1 kudos

AttributeError: module 'numpy' has no attribute 'typing'

We started experiencing failures in several Databricks jobs without any changes on our side. The error occurs during Python job execution and seems related to package dependencies.The job error:Run failed with error message Cannot read the python fil...

  • 70 Views
  • 1 replies
  • 1 kudos
Latest Reply
MarcoRezende
New Contributor III
  • 1 kudos

The problem was numexpr lib version, 2.14.0, I needed to pin 2.13.1

  • 1 kudos
sta_gas
by New Contributor
  • 106 Views
  • 2 replies
  • 1 kudos

Resolved! Data profiling monitoring with foreign catalog

Hi team,I’m currently working with Azure Databricks and have created a foreign catalog for my source database in Azure SQL. I can successfully run SELECT statements from Databricks to the Azure SQL database.However, I would like to set up data profil...

sta_gas_0-1760357690503.png
  • 106 Views
  • 2 replies
  • 1 kudos
Latest Reply
szymon_dybczak
Esteemed Contributor III
  • 1 kudos

Hi @sta_gas ,Since data quality monitoring is in beta I'm quite sure they don't support foreign tables as of now (but they forgot to mentioned it in docs).But more important question if they ever will be supported. For me data quality monitoring appl...

  • 1 kudos
1 More Replies
adrianhernandez
by New Contributor III
  • 159 Views
  • 2 replies
  • 1 kudos

Wheel permissions issue

I get a : org.apache.spark.SparkSecurityException: [INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have permission MODIFY,SELECT on any file. SQLSTATE: 42501 at com.databricks.sql.acl.Unauthorized.throwInsufficientPermissionsError(P...

  • 159 Views
  • 2 replies
  • 1 kudos
Latest Reply
NandiniN
Databricks Employee
  • 1 kudos

Hi @adrianhernandez ,  The permissions error indicates you need to have the privileges for "any file". To resolve this, Can you try by adding the corresponding permissions and see if it works: %sql GRANT SELECT ON ANY FILE TO `username` %sql GRANT MO...

  • 1 kudos
1 More Replies
Hritik_Moon
by New Contributor II
  • 142 Views
  • 5 replies
  • 8 kudos

Stop Cache in free edition

Hello,I am using databricks free edition, is there a way to turn off IO caching.I am trying to learn optimization and cant see any difference in query run time with caching enabled.

  • 142 Views
  • 5 replies
  • 8 kudos
Latest Reply
szymon_dybczak
Esteemed Contributor III
  • 8 kudos

Hi @Hritik_Moon ,I guess you cannot. To disable disk cache you need to have ability to run following command:spark.conf.set("spark.databricks.io.cache.enabled", "[true | false]")But serverless compute does not support setting most Spark properties fo...

  • 8 kudos
4 More Replies
jorperort
by Contributor
  • 2050 Views
  • 4 replies
  • 2 kudos

Resolved! Executing Bash Scripts or Binaries Directly in Databricks Jobs on Single Node Cluster

Hi,Is it possible to directly execute a Bash script or a binary executable from the operating system of a Databricks job compute node using a single node cluster?I’m using databricks asset bundels  for job initialization and execution. When the job s...

  • 2050 Views
  • 4 replies
  • 2 kudos
Latest Reply
Louis_Frolio
Databricks Employee
  • 2 kudos

Hello @jorperort , I did some research internally and have some tips/suggestions for you to consider:   Based on the research and available documentation, it is not possible to directly execute a Bash script or binary executable from the operating sy...

  • 2 kudos
3 More Replies
Vsleg
by Contributor
  • 3166 Views
  • 5 replies
  • 0 kudos

Enabling enableChangeDataFeed on Streaming Table created in DLT

Hello, Can I enable Change Data Feed on Streaming Tables? How should I do this? I couldn't find this in the existing documentation https://learn.microsoft.com/en-us/azure/databricks/delta/delta-change-data-feed .

  • 3166 Views
  • 5 replies
  • 0 kudos
Latest Reply
saurabh18cs
Honored Contributor II
  • 0 kudos

Hi @Vsleg i think you cannot enable cdf like this for streaming tables. it is not natively supported for DLT streaming tables , please have a look here = Propagating Deletes: Managing Data Removal using D... - Databricks Community - 90978

  • 0 kudos
4 More Replies
Chris_N
by New Contributor
  • 165 Views
  • 3 replies
  • 1 kudos

Unable to configure clustering on DLT tables

Hi TeamI have a DLT pipeline with `cluster_by` property configured for all my tables. The code looks something like below:@Dlt.table( name="flows", cluster_by=["from"] ) def flows(): <LOGIC>It was all working fine and in couple of days, the queries w...

  • 165 Views
  • 3 replies
  • 1 kudos
Latest Reply
NandiniN
Databricks Employee
  • 1 kudos

Hi @Chris_N ,   You have mentioned - "I couldn't find any cluster properties configured." If they existed and were changed, you can use the delta history command to check if someone changed on the clustering information.  It is possible there were ch...

  • 1 kudos
2 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels