- 331 Views
- 1 replies
- 1 kudos
Use of Genie agents for ETL migration
I am currently working on a project to migrate existing ETL workflows from SSIS to Databricks. I would like to understand how Databricks Genie, Genie Code, or other Databricks AI-assisted capabilities can support this migration.Can these tools analyz...
- 331 Views
- 1 replies
- 1 kudos
- 1 kudos
Genie code can meaningfully accelerate an SSIS migration, but they should be viewed as a productivity layer around the migration process rather than a push-button solution. In practice, the most reliable automation today is around assessment, invento...
- 1 kudos
- 522 Views
- 2 replies
- 1 kudos
Resolved! Community Custom Connector - Defining Non-Serverless Job Compute Runtime?
Hi there,I'm exploring Community Custom Connectors and I'm wondering about the Compute. By default, it's set to Serverless:But due to IT network limitations at our company, we cannot use serverless. In the Compute settings, I can untoggle the Serverl...
- 522 Views
- 2 replies
- 1 kudos
- 1 kudos
You generally cannot select DBR version in Lakeflow / Spark Declarative Pipelines because the pipeline runtime is Databricks-managed and versionless. Databricks auto-upgrades the runtime for pipelines.The settings has a JSON view, and some advanced o...
- 1 kudos
- 6714 Views
- 3 replies
- 0 kudos
Resolved! Getting connection reset issue while connecting to a SQL server
Hello All,I am unable to connect to a SQL server instance that is installed in a on-premise network from databricks. I am able to successfully ping the server from the notebook using this command [nc -vz <hostname> <port>] which means I am able to e...
- 6714 Views
- 3 replies
- 0 kudos
- 0 kudos
The error you are encountering, "com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset," even after a successful nc (netcat) connection, is a common but nuanced problem when connecting Databricks to an on-premise SQL Server. Although your...
- 0 kudos
- 542 Views
- 6 replies
- 1 kudos
My Apps get stopped automatically
Hi team,I have 2 applications in my workspace which they will get stopped sometimes without me knowing the reason. I’m on premium account and I have to run them when I can’t have access to my app. I need help.    
- 542 Views
- 6 replies
- 1 kudos
- 1 kudos
Could you check the app Logs and tell me what you see when one of them stops?A stack trace or "OOM / killed" → it's crashing (code/memory fix)Clean shutdown with no error → it's the compute lifecycle (keep-alive job fix)Nothing / app never fully star...
- 1 kudos
- 890 Views
- 4 replies
- 2 kudos
Resolved! sdp-meta (dlt-meta) vs lakeflow_framework: when should we use which?
Hi All,I'm evaluating metadata-driven frameworks for Lakeflow Spark Declarative Pipelines, and I've found two that appear to serve the same purpose, both from Databricks organizations:- databrickslabs/dlt-meta, whose feature/sdp-meta branch renames t...
- 890 Views
- 4 replies
- 2 kudos
- 2 kudos
Since you are already on Lakeflow Connect for MySQL, your CDC ingestion into Bronze is the managed, supported path, so you do not need a metadata framework for it. Resources by layer: Ingestion to Bronze (what you have): stay on Lakeflow Connect. It...
- 2 kudos
- 543 Views
- 2 replies
- 2 kudos
Resolved! What is the compute for Lakeflow Connect SharePoint Connector
Can someone help me understand what compute the SharePoint Connector is using by default? I'm trying to use Job Compute, but it "seems" like it may be defaulting to serverless.
- 543 Views
- 2 replies
- 2 kudos
- 2 kudos
The managed SharePoint connector uses serverless compute; it can't be switched to classic Job Compute. The Job you see is the orchestration layer: Lakeflow Connect creates it for the schedule and runs the ingestion pipeline as a task, while the pipel...
- 2 kudos
- 594 Views
- 2 replies
- 0 kudos
Resolved! Issue: Lakeflow Connect Microsoft Teams Community Connector - No module named 'databricks.labs'
Hi there, I'm trying to run a pretty barebones test here with community connectors. I'm trying out the "vanilla" Microsoft Teams connector by:(A) going to the Data Ingestion tab in our workspace,(B) selecting the Microsoft Teams community connector,(...
- 594 Views
- 2 replies
- 0 kudos
- 0 kudos
Both issues trace back to running on non-serverless compute. Community connectors were designed to run on serverless Spark Declarative Pipelines (SDP), and serverless is on by default in the CLI config for a reason. The No module named 'databricks.la...
- 0 kudos
- 1747 Views
- 4 replies
- 2 kudos
Pull JAR from private Maven repository (Azure Artifactory)
Hi,I currently struggle on the following task:We want to push our code to a private repository (Azure Artifactory) and then pull it from databricks when the job runs. It currently works only with wheels inside a PyPi repo in the artifactory. I found ...
- 1747 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @iyashk-DB ,Any update on this? Is there any official suggestion for integrating JFrog and install JAR files using libraries section in asset bundles?
- 2 kudos
- 703 Views
- 4 replies
- 3 kudos
Resolved! How do you keep a plain Delta copy of a streaming table? CLONE is not supported on streaming tables
Hi All, I need an up-to-date plain Delta copy of a streaming table created by Lakeflow pipeline. My first thought was CLONE, and that path is closed. The clone docs say streaming tables and materialized views cannot be the source or target of a deep ...
- 703 Views
- 4 replies
- 3 kudos
- 3 kudos
Thanks @adnan_alvee , yes, we did test Compatibility Mode. The output is a copy of files at a storage location, not a Unity Catalog table.Context: We are setting up an external engine (duckdb) to read UC Delta tables through credential vending, which...
- 3 kudos
- 1357 Views
- 5 replies
- 4 kudos
Resolved! Lakeflow Community Connector Issue - ModuleNotFoundError: No module named 'databricks.labs'
Hi there, I have what should be a somewhat straightforward implementation of a Custom Community Connector. I've tried to Dry run a few times but am encountering PYTHON.MODULE_NOT_FOUND_ERRORBelow is some context about my setup:Source Relative path: l...
- 1357 Views
- 5 replies
- 4 kudos
- 4 kudos
the sys.path approach won’t fix this for Lakeflow Connect pipelines. During a dry run, Databricks imports databricks.labs.community_connector before your ingest notebook runs, using only the pipeline’s Environment dependencies. Fix: Pipeline Settings...
- 4 kudos
- 2519 Views
- 5 replies
- 4 kudos
Resolved! Does VACUUM on Delta Lake also clean Iceberg metadata when using Iceberg Uniform feature?
I'm working with Delta tables using the Iceberg Uniform feature to enable Iceberg-compatible reads. I’m trying to understand how metadata cleanup works in this setup.Specifically, does the VACUUM operation—which removes old Delta Lake metadata based ...
- 2519 Views
- 5 replies
- 4 kudos
- 4 kudos
Sound discussion overall in this thread. Two hard earned lessons that came out of running this in production that I can think of sharing:(1) Rehearse the cleanup on a scratch table before relying on it: point your reader at a test table, expire snaps...
- 4 kudos
- 279 Views
- 1 replies
- 0 kudos
Community Connectors confusion with `connector_spec.yaml` usage (or lack of usage)
Hi there, I'm a bit confused about how connections for Community Connectors setup should be. I assumed that the `connector_spec.yaml` file should outline the specifics of what is required for a Community Connector, but in practice it doesn't seem lik...
- 279 Views
- 1 replies
- 0 kudos
- 0 kudos
The connector_spec.yaml isn't what controls the UI for custom connectors. When you use the "Add Custom Community Connector" path in the workspace UI, Databricks shows a generic connection form with all available auth types. That form is the same for ...
- 0 kudos
- 265 Views
- 1 replies
- 1 kudos
Materialized View backing pipeline retains old Unity Catalog after catalog rename
Materialized View backing pipeline retains old Unity Catalog after catalog rename (dev_cat → dap_catalog_dev) resulting in NO_SUCH_CATALOG_EXCEPTIONThis issue blocks the creation and refresh of Materialized Views after a Unity Catalog rename.Workspac...
- 265 Views
- 1 replies
- 1 kudos
- 1 kudos
What you're hitting is expected behavior, not something you did wrong. Every materialized view is backed by a managed serverless pipeline, and that pipeline pins its catalog, target schema, and internal storage location at the moment the M...
- 1 kudos
- 16013 Views
- 8 replies
- 9 kudos
Reading an Iceberg table with AWS Glue Data Catalog as metastore
I have created an Iceberg table using AWS Glue, however whenever I try to read it using a Databricks cluster, I get `java.lang.InstantiationException`. I have tried every combination of Spark configs for my Databricks compute cluster that I can think...
- 16013 Views
- 8 replies
- 9 kudos
- 9 kudos
Hello, I get the same issue "Failed to instantiate org.apache.hadoop.mapred.FileInputFormat" while trying to read Iceberg tables through Glue catalog federation. Provider appears as hive in Databricks while it is registered as an Iceberg table in Glu...
- 9 kudos
- 195 Views
- 1 replies
- 0 kudos
Ingesting data from a SharePoint Excel file with two worksheets in one pipeline
Hi,I would like to ask about ingesting data from an Excel file stored in SharePoint.Currently, I have one Excel file containing two worksheets:one worksheet contains dimension data,the other worksheet contains target values.I would like to ingest bot...
- 195 Views
- 1 replies
- 0 kudos
- 0 kudos
Based on the behavior you described, this looks like a Managed SharePoint ingestion bug or preview limitation, not an obvious error in your configuration.Databricks’ Excel reader supports selecting one worksheet with dataAddress, and your successful ...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
