- 13 Views
- 1 replies
- 0 kudos
Set default tblproperties for pipeline
I like to set tblproperties ("delta.feature.timestampNtz" = "supported") for all tables in a pipeline. instead of set this option for every table definition. The property must be set direct on creation. I have tried it in the pipeline settings - conf...
- 13 Views
- 1 replies
- 0 kudos
- 0 kudos
Databricks does not allow you to set a global default for all TBLPROPERTIES. However, you can use the spark.databricks.delta.properties.defaults configuration key to set defaults for new Delta tables created in a specific session or pipeline.If you w...
- 0 kudos
- 39 Views
- 1 replies
- 0 kudos
Slow PySpark operations after long DAG that contains many joins and transformations
We are using PySpark and notice that when we are doing many transformations/aggregations/joins of the data then at some point the execution time of simple task (count, display, union of 2 tables, ...) become very slow even if we have a small data (ex...
- 39 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Jonathan_ Good day!To answer first, I think as you mentioned pandas are faster than spark because, spark works better with big data. Pandas runs in-memory on a single machine without this distributed overhead, making it faster for small data. ...
- 0 kudos
- 83 Views
- 3 replies
- 1 kudos
Not able to refresh powerbi dashboar form databricks jobs
I am trying to refresh Power BI Dashboard using Databricks jobs and constantly getting this error, but I am providing optional parameters which includes catalog and database. Also, things to note that I am able to do refresh on Power BI UI using both...
- 83 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @sumitkumar_284 ,Can you provide us more details? Are you using Unity Catalog? Which authentication mechanism you have? In which version of Power BI Desktop you've developed your semantic model/dashboard? Do you meet all below requirements?Publish...
- 1 kudos
- 54 Views
- 2 replies
- 3 kudos
Resolved! Error occurs on create materialized view with spark.sql
When creating materialized view with spark.sql function it returns following error message.[MATERIALIZED_VIEW_OPERATION_NOT_ALLOWED.MV_NOT_ENABLED] The materialized view operation CREATE is not allowed: Materialized view features are not enabled for ...
- 54 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi, @szymon_dybczak Thank you for your response.You're right, it was mentioned in the documentation—I missed it when checking.I understand now that it's not possible to do this with spark.sql. Thanks for clarifying!
- 3 kudos
- 1951 Views
- 9 replies
- 3 kudos
Resolved! How to create parameters that works in Power BI Report Builder (SSRS)
Hello!I'm trying to create an item in Power Bi Report Server (SSRS) connected to Databricks. I can connect normally, but I'm having trouble using a parameter that Databricks recognizes.First, I'll illustrate what I do when I connect to SQL Server and...
- 1951 Views
- 9 replies
- 3 kudos
- 3 kudos
@fellipeao This is the only way I found that works well with databricks since positional arguments (?) was a fail for me. This is the latest version of paginated report builder.https://learn.microsoft.com/en-us/power-bi/paginated-reports/report-build...
- 3 kudos
- 104 Views
- 10 replies
- 3 kudos
Workspace got disabled
Hi everyone,I was creating a database linked service in ADF, but I got an error: "unauthorized network access to workspace." After that, I went to ADB networking and changed the public setting to "enable"; previously, it was disabled.I think it is b...
- 104 Views
- 10 replies
- 3 kudos
- 3 kudos
Also adding to @szymon_dybczak reply. You can refer to this github page.https://github.com/databricks/terraform-databricks-sra/tree/main/azureIt has all the biceps and terraform templates. if you are deploying a secured workspace that have security c...
- 3 kudos
- 464 Views
- 4 replies
- 0 kudos
Databricks Delta MERGE fails with row filter — “Cannot find column index for attribute 'account_id'”
Problem:I’m getting the below error when performing a MERGE (or any other DML command) on a Delta table with a row filter in Databricks.Error: Cannot find column index for attribute 'account_id#48219' in: Map(transaction_id#47260 -> 5, file_path#4725...
- 464 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @vim17, I see "partitionValues_parsed#47264" in the Error trace. Is the table partitioned, by any chance? Which DBR version are you using?
- 0 kudos
- 106 Views
- 2 replies
- 1 kudos
Databricks Asset Bundles: issue with python_file path in spark_python_task
Hi,I am trying to run a python file which is stored in src folder. However, I am getting the following error: Error: cannot update job: Invalid python file reference: src/get_git_credentials.py. Please visit the Databricks user guide for supported py...
- 106 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @szymon_dybczak,Thanks for you reply. The issue is related to the fact if am also providing git_source parameters in this job (even though the source parameter of my task is set to WORKSPACE). When I comment the git_source part, specifying the pat...
- 1 kudos
- 34 Views
- 0 replies
- 0 kudos
Errors in runtime 17 today
Anyone else getting a bunch of errors on runtime 17 today? A load of our pipelines that were running smoothly suddenly stopped working with driver crashes. I was able to get us running again by downgrading to runtime 16, but curious if anyone else hi...
- 34 Views
- 0 replies
- 0 kudos
- 29 Views
- 1 replies
- 0 kudos
Unable to login to community edition
Hi All,I am not able to login into "databricks community edition". When I hit sign up its redirecting me into "free edition" as there’s currently no way to create or enable clusters in the Databricks Free Edition. Please suggest someway where I can u...
- 29 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @SahabazKhan! If you’re trying to create a new Community Edition account, that’s not possible, it will redirect you to create a Free Edition account instead. In the Free Edition, you’ll have access to serverless compute and other features. Want...
- 0 kudos
- 64 Views
- 8 replies
- 1 kudos
Error in Viewing the Table
Facing below error while accessing a table with mutiple row filters but i am not able to delete the row filter using SQL query. Please help Failed to request /ajax-api/2.1/unity-catalog/tables/product_return_prediction_dev.bronze.customers_data?incl...
- 64 Views
- 8 replies
- 1 kudos
- 1 kudos
Hi @pranaav93 @szymon_dybczak below query worked for me "DROP POLICY city_filter_policy ON product_return_prediction_dev.bronze.customers_data"
- 1 kudos
- 113 Views
- 5 replies
- 0 kudos
[INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have permission MODIFY on any file
Just wondering if anyone could help me understand why we are hitting this error: `[INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have permission MODIFY on any file`A job is trying to create a table with an external location (alread...
- 113 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @rachelh As I understand , you need to look for azure access connector setup for your unity catalog because Serverless clusters run under a Azure Databricks-managed identity, not the service principal.Access Connector (Azure Managed Identity): Use...
- 0 kudos
- 30 Views
- 0 replies
- 0 kudos
Databricks Compute Metrics Alerts
Hi All,Im looking for some implementation ideas where i can use information from the system.compute.node_timeline table to catch memory spikes and if above a given threshold restart the cluster through an API call. Have any of you implemented a simil...
- 30 Views
- 0 replies
- 0 kudos
- 199 Views
- 2 replies
- 2 kudos
Capturing Streaming Metrics in Near Real-Time Using Cluster Logs
Over the past few weeks, I’ve been exploring ways to capture streaming metrics from our data load jobs. The goal is to monitor job performance and behavior in real time, without disrupting our existing data load pipelines.Initial Exploration: Streami...
- 199 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @saicharandeepb Good job on doing such detailed research on monitoring structured streaming. If you need lower latency than rolling log permits, then have you tried this:Cluster-wide listener injection: Use spark.extraListeners to register a cust...
- 2 kudos
- 1178 Views
- 3 replies
- 3 kudos
Resolved! Permission Denied when Creating External Tables Using Workspace Default Credential
I’m building out schemas, volumes, and external Delta tables in Unity Catalog via Terraform. The schemas and volumes are created successfully, but all external tables are failing.The error message from Terraform doesn't highlight what the issue is bu...
- 1178 Views
- 3 replies
- 3 kudos
- 3 kudos
Feature or bug, I discovered the same. I couldn't create tables with the default credential. To test, I assigned the default credential and a custom credential the same access rights to the storage container that is the target of the external locatio...
- 3 kudos
| User | Count |
|---|---|
| 1616 | |
| 790 | |
| 455 | |
| 349 | |
| 287 |
