- 3600 Views
- 1 replies
- 0 kudos
mqtt to Delta Live Table
Greetings,I see that Delta Live Tables has various real-time connectors such as Kafka, Kinesis, Google's Pub Sub, and so on. I also see that Apache had maintained an mqtt connector to Spark through the 2.x series called Bahir, but dropped it in versi...
- 3600 Views
- 1 replies
- 0 kudos
- 0 kudos
How this solution end up?
- 0 kudos
- 27337 Views
- 7 replies
- 1 kudos
ModuleNotFoundError: No module named 'databricks.sdk' in module installed via Pip
Hello. I'm currently having an issue that I simply cannot understand nor find an adequate work-around for. Recently, my team within our organization has undergone the effort of migrating our Python code from Databricks notebooks into regular Python m...
- 27337 Views
- 7 replies
- 1 kudos
- 1 kudos
Lol, OK so in my case it was because I had a file called databricks.py with clashed with the installed databricks. Renaming my file to databricks_utils.py solved it.
- 1 kudos
- 4263 Views
- 1 replies
- 2 kudos
Resolved! AI Agents in ETL
Hi, I recently found a blog online about Databricks using AI Agents to automate ETL, but I can't find where these capabilities are located in Databricks. Does anyone know?Here is the blog,https://www.heliverse.com/blog/databricks-ai-agents-streamlini...
- 4263 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi NiilDatabrick has introduced AI Agent sub-categories as part of its Generative AI capabilities. We can now automate tasks such as Extract, Transformation, and Load (ETL).Ex, Information Extraction Agent - we can able to transform large a volume ...
- 2 kudos
- 5870 Views
- 7 replies
- 0 kudos
Internal GRPC errors when using databricks connect
Hey there, in our local development flow we heavily rely on databricks asset bundles and databricks connect. Recently, locally run workflows (i.e. just pyspark python files) have begun to frequently fail with the following grpc error:pyspark.errors.e...
- 5870 Views
- 7 replies
- 0 kudos
- 0 kudos
@marcelhfm it might be a Spark Connect issueI would say it is the same for the rest of you, guys Nothing much to do until the situation is fixed by Databricks
- 0 kudos
- 3223 Views
- 3 replies
- 1 kudos
Observability and monitoring accross multiple workspaces(both job clusters and serverless compute)
Hi all, Today what are the best option available today for observability and monitoring databricks jobs accross all workspaces. We have 100 of workspaces and it hard to do monitoring to check failed and successeded jobs.We tried using: 1. Team webhoo...
- 3223 Views
- 3 replies
- 1 kudos
- 1 kudos
Hey Brahmareddy, Thanks so much for responding. Sorry I forgot to mention we are in Azure. Lets go through one by one.1. Audit logs(Azure Monitor) : AFAIK this requires init scripts and jar build that will not support in serverless or its not the ca...
- 1 kudos
- 2470 Views
- 2 replies
- 1 kudos
Resolved! Not readable format of event_logs traceback in dlt pipeline
This is my dlt pipeline event_log - why is it not in readable foramt how can i correct it.This is my pipeline code : import logging logger = logging.getLogger(__name__) logger.info("Error") raise "Error is error"
- 2470 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @ashraf1395 ,I'm working with Delta Live Tables (DLT) and the event_log table. I would like to know if it is possible to access the event handler that DLT uses to write custom logs and send them to this table when events are published.If this is n...
- 1 kudos
- 1902 Views
- 2 replies
- 1 kudos
How to write event_log destination into DLT Settings JSON via Asset Bundles
Hi all,I would like to publish the event_log of my DLT Pipeline to a specific schema in Unity Catalog.Following this article (https://docs.databricks.com/gcp/en/dlt/observability#query-the-event-log) this can be done by writing this into the DLTs set...
- 1902 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @susanne , indeed , i tried to create it using dabs as well. This feature is not available using dabs I guess, maybe they will add it once event_logs moves to ga from public preview.databricks API will be a good alternative but if you try it using...
- 1 kudos
- 1737 Views
- 1 replies
- 1 kudos
Resolved! Creating event_log for dlt pipeline using dabs
Hi there, i have dlt pipeline and I recently came to know about the event_log feature, i want to deploy my dlt pipeline along with the event_log using databricks asset bundles but i am not able to find any resources for it.If anyone has tried it , yo...
- 1737 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi there @21f3001806 ,I guess you are talking about the this : https://docs.databricks.com/api/workspace/pipelines/create#event_logIts still in public preview I tried creating it through UI or by chaing the pipeline settings it worked. But when I imp...
- 1 kudos
- 1668 Views
- 3 replies
- 0 kudos
Data migration from S3 to Databricks
I currently have an S3 bucket with around ~80 tables, each of which has hive-style partition columns S3RootFolder/Table1Name/Year=2024/Month=12/Day=1/xxx.parquetS3RootFolder/Table1Name/Year=2024/Month=12/Day=2/xxx.parquetS3RootFolder/Table2Name/Year...
- 1668 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi DBStudent,You’re absolutely right—coalescing during write won’t help with the initial bottleneck, since the issue is really with the read side, where Spark has to list and open 110k small files. If you can’t pre-compact them before reading, then o...
- 0 kudos
- 5126 Views
- 10 replies
- 6 kudos
ISOLATION_STARTUP_FAILURE for Serverless
Hi, I'm getting an error in my job which is using Serverless cluster: `[ISOLATION_STARTUP_FAILURE] Failed to start isolated execution environment. Please contact Databricks support. SQLSTATE: XXKSS`It was working fine and then, this error appeared on...
- 5126 Views
- 10 replies
- 6 kudos
- 6 kudos
@Alberto_Umana Looks like the issue resurfaced again. We started seeing "[ISOLATION_STARTUP_FAILURE] Failed to start isolated execution environment. Please contact Databricks support. SQLSTATE: XXKSS`" error with serverless compute from last night.Co...
- 6 kudos
- 8485 Views
- 5 replies
- 5 kudos
ISOLATION_STARTUP_FAILURE
Hi I'm getting error in my data pipeline[ISOLATION_STARTUP_FAILURE] Failed to start isolated execution environment. Please contact Databricks support. SQLSTATE: XXKSSit was working fine and suddenly it is keep failing. Please advice.
- 8485 Views
- 5 replies
- 5 kudos
- 5 kudos
I am running on serverless and only yesterday I started seeing this error. Any thoughts?[ISOLATION_STARTUP_FAILURE] Failed to start isolated execution environment. Please contact Databricks support. SQLSTATE: XXKSS
- 5 kudos
- 1378 Views
- 1 replies
- 0 kudos
Unable to create Cluster via REST API
I have tried all 3 of the example payloads listed here to create a Compute Cluster: https://docs.databricks.com/api/azure/workspace/clusters/createAll return the same error: Invalid JSON given in the body of the request - expected a mapOther Compute ...
- 1378 Views
- 1 replies
- 0 kudos
- 0 kudos
@jeffn - I was able to successfully create a cluster using the same Azure documentation, both via the REST API and the Databricks SDK. Hopefully it helps.from databricks.sdk import WorkspaceClient from databricks.sdk.service.compute import CreateClu...
- 0 kudos
- 1299 Views
- 1 replies
- 0 kudos
Resolved! Regexp_replaces pattern only once
select regexp_replace('asdfhsdf&&1&&asdfasdf&&2&&asdf','&&[0-100]&&','') output here it replaces first pattern but not the 2nd patterno/poutputasdfhsdfasdfasdf&&2&&asdf even if we use position its not working something like both wont work select ...
- 1299 Views
- 1 replies
- 0 kudos
- 0 kudos
changing to [0-9] works this can be closed.
- 0 kudos
- 13774 Views
- 3 replies
- 2 kudos
Resolved! CREATE TEMP TABLE
The Databricks assistant tells me (sometimes) that `CREATE TEMP TABLE` is a valid SQL operation. And other sources (e.g., https://www.freecodecamp.org/news/sql-temp-table-how-to-create-a-temporary-sql-table/) say the same.But in actual practice, thi...
- 13774 Views
- 3 replies
- 2 kudos
- 2 kudos
In addition to accepting KaranamS's answer, I will note a longer and useful discussion, with caveats, at https://community.databricks.com/t5/data-engineering/how-do-temp-views-actually-work/m-p/20137/highlight/true#M13584.
- 2 kudos
- 3286 Views
- 5 replies
- 0 kudos
Remote SQL Server Instance Connection using JDBC
Hi All,I am connecting Remote SQL Server Instance using JDBC Driver, I have enabled TCP/IP and Setup the Firewall Rule. When i am querying the instance i am getting this error:(com.microsoft.sqlserver.jdbc.SQLServerException) The TCP/IP connection to...
- 3286 Views
- 5 replies
- 0 kudos
- 0 kudos
If you want to access a local SQL server, you'll need a Private Link to access the server. If it's on your own local machine, that's likely not possible. Creating a VPN to your machine is a unique problem, and you would be better off using a VM or a ...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
