- 324 Views
- 4 replies
- 5 kudos
Resolved! Python Wheel in Serverless Job in DAB
Hey,I am trying to run a job with serverless compute, that runs python scripts.I need the paramiko package to get my scripts to work. I managed to get it working by doing:environments:- environment_key: default# Full documentation of this spec can be...
- 324 Views
- 4 replies
- 5 kudos
- 5 kudos
Hi @georgemichael40 ,Put your whl file in the volume and then you can reference it in following way in your DAB file:dependencies: - " /Volumes/workspace/default/my_volume/hellopkg-0.0.1-py3-none-any.whl"https://docs.databricks.com/aws/en/compute/s...
- 5 kudos
- 8812 Views
- 5 replies
- 3 kudos
Can we use multiple git repos for a job running multiple tasks?
I have a job running multiple tasks :Task 1 runs a machine learning pipeline from git repo 1Task 2 runs an ETL pipeline from git repo 1Task 2 is actually a generic pipeline and should not be checked in repo 1, and will be made available in another re...
- 8812 Views
- 5 replies
- 3 kudos
- 3 kudos
Had this same problem. Fix was to have two workflows with no triggers, each pointing to the respective git repo. Then setup a 3rd workflow with appropriate triggers/schedule which calls the first 2 workflows. A workflow can run other workflows.
- 3 kudos
- 1945 Views
- 1 replies
- 2 kudos
Resolved! Geometry Type not converted into proper binary format when reading through Federated Catalog
Hi,When reading a geometry column from a sql server into Databricks through foreign/federated catalog the tranformation of geometry type to binary type is not in proper format or I am not able to find a way I can decode that binary.for example, for p...
- 1945 Views
- 1 replies
- 2 kudos
- 2 kudos
Give this a shotCreate a view in SQL Server that converts geometry to Well-Known Text before federating:-- Create view in SQL ServerCREATE VIEW dbo.vw_spatial_converted ASSELECTid,location_name,location.STAsText() AS geom_wkt,location.STSrid() AS sri...
- 2 kudos
- 295 Views
- 4 replies
- 4 kudos
Resolved! Lost access to Databricks account console on Free Edition
Hi everyone,I'm having trouble accessing the Databricks account console and need some guidance.Background:I successfully set up Databricks Free Edition with Terraform using my personal accountI was able to access accounts.cloud.databricks.com to obta...
- 295 Views
- 4 replies
- 4 kudos
- 4 kudos
I just double checked, I was able to manage my personal workspace through terraform without account console. Thanks again.
- 4 kudos
- 163 Views
- 1 replies
- 0 kudos
Resolved! Errors in runtime 17 today
Anyone else getting a bunch of errors on runtime 17 today? A load of our pipelines that were running smoothly suddenly stopped working with driver crashes. I was able to get us running again by downgrading to runtime 16, but curious if anyone else hi...
- 163 Views
- 1 replies
- 0 kudos
- 0 kudos
@stevewb Driver crash is very generic. We may need to dig deeper here to understand the root cause. Can you raise a support ticket with us?
- 0 kudos
- 402 Views
- 2 replies
- 0 kudos
Question on assigning email_notification_group to DLT Job Notifications?
Hi Folks,I wanted to check if there’s a way to assign an email notification group to a Delta Live Tables (DLT) job for notifications.I know that it’s possible to configure Teams workflows and email notification groups for Databricks jobs, but in the ...
- 402 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @surajitDE ,At the moment, DLT doesn’t support linking existing email notification groups or Teams workflows directly. You can only add individual email addresses in the DLT UI.If you have a group email alias, you can use it as a single address so...
- 0 kudos
- 290 Views
- 1 replies
- 0 kudos
Orphaned __dlt_materialization schemas left behind after dropping materialized views
Hi everyone,I’m seeing several internal schemas under the __databricks_internal catalog that were auto-created when I built a few materialized views in Databricks SQL. However, after dropping the materialized views, the schemas were not automatically...
- 290 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes, this is expected behavior in Databricks. The __databricks_internal catalog contains system-owned schemas that support features like materialized views and Delta Live Tables (DLT). When you create materialized views, Databricks generates internal...
- 0 kudos
- 193 Views
- 1 replies
- 1 kudos
Resolved! Databricks Compute Metrics Alerts
Hi All,Im looking for some implementation ideas where i can use information from the system.compute.node_timeline table to catch memory spikes and if above a given threshold restart the cluster through an API call. Have any of you implemented a simil...
- 193 Views
- 1 replies
- 1 kudos
- 1 kudos
Hey @pranaav93 A very common use case for using system table system.compute.node_timeline to build alerting and remediation. Check this KB https://kb.databricks.com/en_US/clusters/getting-node-specific-instead-of-cluster-wide-memory-usage-data-from-...
- 1 kudos
- 2261 Views
- 1 replies
- 0 kudos
Databricks-connect OpenSSL Handshake failed on WSL2
When trying to setup databricks-connect on WSL2 using 13.3 cluster, I receive the following error regarding OpenSSL CERTIFICATE_ERIFY_FAILED.The authentication is done via SPARK_REMOTE env. variable. E0415 11:24:26.646129568 142172 ssl_transport_sec...
- 2261 Views
- 1 replies
- 0 kudos
- 0 kudos
@vpacik Was it solved? I have the same issue
- 0 kudos
- 410 Views
- 7 replies
- 3 kudos
Resolved! create delta table in free edition
table_name = f"project.bronze.{file_name}"spark.sql( f""" CREATE TABLE IF NOT EXISTS {table_name} USING DELTA """) what am I getting wrong?
- 410 Views
- 7 replies
- 3 kudos
- 3 kudos
yes, multiline solved it. .Is there any better approach to this scenario?
- 3 kudos
- 191 Views
- 1 replies
- 2 kudos
Resolved! Set default tblproperties for pipeline
I like to set tblproperties ("delta.feature.timestampNtz" = "supported") for all tables in a pipeline. instead of set this option for every table definition. The property must be set direct on creation. I have tried it in the pipeline settings - conf...
- 191 Views
- 1 replies
- 2 kudos
- 2 kudos
Databricks does not allow you to set a global default for all TBLPROPERTIES. However, you can use the spark.databricks.delta.properties.defaults configuration key to set defaults for new Delta tables created in a specific session or pipeline.If you w...
- 2 kudos
- 296 Views
- 2 replies
- 4 kudos
Resolved! Error occurs on create materialized view with spark.sql
When creating materialized view with spark.sql function it returns following error message.[MATERIALIZED_VIEW_OPERATION_NOT_ALLOWED.MV_NOT_ENABLED] The materialized view operation CREATE is not allowed: Materialized view features are not enabled for ...
- 296 Views
- 2 replies
- 4 kudos
- 4 kudos
Hi, @szymon_dybczak Thank you for your response.You're right, it was mentioned in the documentation—I missed it when checking.I understand now that it's not possible to do this with spark.sql. Thanks for clarifying!
- 4 kudos
- 2277 Views
- 9 replies
- 3 kudos
Resolved! How to create parameters that works in Power BI Report Builder (SSRS)
Hello!I'm trying to create an item in Power Bi Report Server (SSRS) connected to Databricks. I can connect normally, but I'm having trouble using a parameter that Databricks recognizes.First, I'll illustrate what I do when I connect to SQL Server and...
- 2277 Views
- 9 replies
- 3 kudos
- 3 kudos
@fellipeao This is the only way I found that works well with databricks since positional arguments (?) was a fail for me. This is the latest version of paginated report builder.https://learn.microsoft.com/en-us/power-bi/paginated-reports/report-build...
- 3 kudos
- 470 Views
- 10 replies
- 3 kudos
Workspace got disabled
Hi everyone,I was creating a database linked service in ADF, but I got an error: "unauthorized network access to workspace." After that, I went to ADB networking and changed the public setting to "enable"; previously, it was disabled.I think it is b...
- 470 Views
- 10 replies
- 3 kudos
- 3 kudos
Also adding to @szymon_dybczak reply. You can refer to this github page.https://github.com/databricks/terraform-databricks-sra/tree/main/azureIt has all the biceps and terraform templates. if you are deploying a secured workspace that have security c...
- 3 kudos
- 568 Views
- 4 replies
- 0 kudos
Databricks Delta MERGE fails with row filter — “Cannot find column index for attribute 'account_id'”
Problem:I’m getting the below error when performing a MERGE (or any other DML command) on a Delta table with a row filter in Databricks.Error: Cannot find column index for attribute 'account_id#48219' in: Map(transaction_id#47260 -> 5, file_path#4725...
- 568 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @vim17, I see "partitionValues_parsed#47264" in the Error trace. Is the table partitioned, by any chance? Which DBR version are you using?
- 0 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 483 | |
| 349 | |
| 287 |
