- 929 Views
- 1 replies
- 0 kudos
Decoupling power bi reports from schemas
HiI'm sure many of you have power bi reports that use the native Databricks connector to pull data from a schema to fuel a wonderful dashboard. If the source schemes was moved to a different catalog or a table renamed then the pbi connection would be...
- 929 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @IONA ,You can parametrize your data source using power query parameters. Just define parameter for catalog, schema etc. I have such setup in one of the project. We parametrized our PBI semantic models and when we are deploying semantic model to a...
- 0 kudos
- 3051 Views
- 7 replies
- 4 kudos
- 3051 Views
- 7 replies
- 4 kudos
- 4 kudos
Did anyone know when we will be going to receive the partner tech summit badge?
- 4 kudos
- 1363 Views
- 3 replies
- 6 kudos
Resolved! Issue with Auto Liquid clustering
I have written data to a table using clusterByAuto set to trueBut the clustering keys are not selected automatically when i do a desc detail on the table.Screenshot belowWhy are clustering columns not being selected automatically?Repro steps:Create ...
- 1363 Views
- 3 replies
- 6 kudos
- 6 kudos
Hi @RevanthV ,As @K_Anudeep correctly suggested it could be the case that your table is to small to benefit from liquid clustering.Another possibility it that you're using runtime lower than 15.4 LTS.
- 6 kudos
- 804 Views
- 1 replies
- 2 kudos
- 804 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello @Soumenkumar ,I believe OGG has a Databricks target that stages to cloud storage (ADLS on S3) and then runs MERGE into Delta tables. This is designed for UC and documented in the official Oracle Docs.In Unity Catalog, create a storage credentia...
- 2 kudos
- 607 Views
- 1 replies
- 0 kudos
Cross region delta table creation in Azure
Hi Community.I have a usecase as below.1. I have a Azure Databricks [ADB] in NCUS which has Unity Catalog [UC] enabled using the NCUS region metastore.2. The ADLS Gen2 storage is in EUSI am trying to registers a existing delta table in UC and also cr...
- 607 Views
- 1 replies
- 0 kudos
- 0 kudos
Seems like an networking issue. In a notebook attached to the custom cluster, try dbutils.fs.ls("abfss://<container>@<account>.dfs.core.windows.net/<path>") to confirm reachability before running CREATE EXTERNAL TABLE. (If this hangs or errors, it’s ...
- 0 kudos
- 2606 Views
- 5 replies
- 4 kudos
Resolved! How to add webhook notification in DLT pipeline through yml
Hi team,I am trying to create a slack webhook notification in a DLT pipeline leveraging jobs.yml. targets: dev: # The default target uses 'mode: development' to create a development copy. # - Deployed resources get prefixed with '[dev my_user_...
- 2606 Views
- 5 replies
- 4 kudos
- 4 kudos
@szymon_dybczak , I gone through the Databricks docs.. I have added the webhook notifications in DLT pipeline. it works fine for me . # The job that triggers api_data_pipeline.resources: jobs: api_data_job: name: api_data_job schedule: ...
- 4 kudos
- 1247 Views
- 4 replies
- 0 kudos
Azure not getting listed in create external location
Hi,I'm trying to create a pipeline using Azure, however, Azure is not getting listed in the drop down of Catalog Explorer -> Create External Location. I'm using community version for practice. Please advice.
- 1247 Views
- 4 replies
- 0 kudos
- 0 kudos
@AanchalSoni Yes Databricks free edition have this limitation you cannot create customized external location It has only support for s3 now and that location is managed by databricks. Look at the limitations below.
- 0 kudos
- 2523 Views
- 2 replies
- 0 kudos
Materialized Views Incremental Load
My question is: can materialized views be updated incrementally. For example:In my case, we store the data in Iron layer and it gets flattened in bronze and silver layer as separate tables. The required transformations happen from silver to gold laye...
- 2523 Views
- 2 replies
- 0 kudos
- 0 kudos
So, I am using DLT declarative framework to work with MV in the gold layer. I am sharing code for the sample. So, can someone tell me that how we can do only the incremental refresh, as this code is doing a full refresh daily. I don't to want do agai...
- 0 kudos
- 3879 Views
- 4 replies
- 1 kudos
Incremental refresh of materialized view in serverless DLT
Hello, Every time that I run a delta live table materialized view in serverless , I get a log of "COMPLETE RECOMPUTE" . How can I achieve incremental refresh in serverless in DLT pipelines?
- 3879 Views
- 4 replies
- 1 kudos
- 1 kudos
So, I am using DLT declarative framework to work with MV in the gold layer. I am sharing code for the sample. So, can someone tell me that how we can do only the incremental refresh, as this code is doing a full refresh daily. I don't to want do agai...
- 1 kudos
- 2365 Views
- 4 replies
- 6 kudos
Resolved! Databricks clusters unresponsive
Hello everyone,we are experiencing issues on one of our Databricks workspaces:Notebooks and SQL queries are executing, but results are not returned to the UI.On the screenshots you can see examples where cells in notebooks and queries in a SQL wareho...
- 2365 Views
- 4 replies
- 6 kudos
- 6 kudos
Thanks a lot for the quick replies!It seems the issue was related to a faulty iPython version (or something similar inside the workspace). The problem resolved itself later the same day in the evening, and everything is working normally again now.
- 6 kudos
- 5706 Views
- 11 replies
- 2 kudos
Resolved! Issue with Installing Remorph Reconcile Tool and Compatibility Clarification
I am currently working on a table migration project from a source Hive Metastore workspace to a target Unity Catalog workspace. After migrating the tables, I intend to write table validation scripts using the Remorph Reconcile tool. However, I am enc...
- 5706 Views
- 11 replies
- 2 kudos
- 2 kudos
I would just like to mention that it might not be due to remorph or your python version that you encounter this error. I got a similar error message when trying to apply changes to the metastore grants through terraform.It worked when I authenticate...
- 2 kudos
- 1069 Views
- 1 replies
- 2 kudos
Resolved! New Scroll bar appears in cells with more than 300 lines
Hi,In a Databricks notebook, I noticed a new behaviour which I'd like to deactivate.Essentially, when a cell reaches 300 lines or more, a new scroll bar within the cell appears. I'd rather have the cell displayed in full and keep using the page scrol...
- 1069 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @ToNiOZ45 ,Unfortunately, there's no option to disable it in workspace settings. You can create ticket in Databricks Ideas and suggest them to add that option if you find this scroll bar frustrating:Feedback | Databricks
- 2 kudos
- 2243 Views
- 5 replies
- 4 kudos
Serverless Compute - pySpark - Any alternative for rdd.getNumPartitions()
Hello Community,We have been trying to migrate our jobs from Classic Compute to Serverless Compute. As part of this process, we face several challenges, and this is one of them.When we read CSV or JSON files with multiLine=true, the load becomes sing...
- 2243 Views
- 5 replies
- 4 kudos
- 4 kudos
Hi @Ramana ,Yep, RDD API is not supported on ServelessAs a workaround you can obtain number of partitions in following way - using spark_partiton_id and then counting distinct occurance of each idfrom pyspark.sql.functions import spark_partition_id,...
- 4 kudos
- 1783 Views
- 1 replies
- 0 kudos
Deploy asset bundle without recreating Lakeflow SQL gateway and DLT Pipeline
Hello all,I have a pre-existing Databricks Asset Bundle that deploys a workflow.I am starting to investigate Lakeflow Connect using a SQL Server connector. In my existing Databricks Asset Bundle, I added a pipeline YAML file that defines a gateway an...
- 1783 Views
- 1 replies
- 0 kudos
- 0 kudos
@LJacobsen If i understand you correctly, you have manually created objects (in your case a DLT pipeline) that you now want to manage in your asset bundle. If that's the case "bundle deployment bind" is your friend:https://learn.microsoft.com/en-us/a...
- 0 kudos
- 1086 Views
- 2 replies
- 0 kudos
Serverless Compute - Python - Custom Emails via SMTP (smtplib.SMTP(host_name)) - Any alternative?
Hello Community,We have been trying to migrate our jobs from Classic Compute to Serverless Compute. As part of this process, we face several challenges, and this is one of them.We have several scenarios where we need to send an inline email via Pytho...
- 1086 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Ramana ,What error do you get in serverless? Could you provide error message?
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 567 | |
| 349 | |
| 287 |
