- 4267 Views
- 2 replies
- 1 kudos
Autloader Data Reprocess
Hi ,If possible can any please help me with some autloader options I have 2 open queries ,(i) Let assume I am running some autoloader stream and if my job fails, so instead of resetting the whole checkpoint, I want to run stream from specified timest...
- 4267 Views
- 2 replies
- 1 kudos
- 1 kudos
In Databricks Autoloader, controlling the starting point for streaming data after a job failure requires careful management of checkpoints and configuration options. By default, Autoloader uses checkpoints to remember where the stream last left off, ...
- 1 kudos
- 741 Views
- 1 replies
- 2 kudos
Resolved! Global Parameter at the Pipeline level in Lakeflow Job
Hi ,any work around or Databricks can enable global parameters feature at the pipeline level in the lakeflow job.Currently I am working on migrating adf pipeline schedule set up to lakeflow job.
- 741 Views
- 1 replies
- 2 kudos
- 2 kudos
Databricks Lakeflow Declarative Pipelines do not currently support truly global parameters at the pipeline level in the same way that Azure Data Factory (ADF) allows, but there are workarounds that enable parameterization to streamline migration from...
- 2 kudos
- 439 Views
- 1 replies
- 0 kudos
Resolved! transformWithStateInPandas. Invalid pickle opcode when updating ValueState with large (float) array
I am getting an error when the entity I need to store in a ValueState is a large array (over 15k-20k items). No error (and works correctly) if I trim the array to under 10k samples. The same error is raised when using it as a value for MapState or as...
- 439 Views
- 1 replies
- 0 kudos
- 0 kudos
The error you’re facing, specifically PySparkRuntimeError: Error updating value state: invalid pickle opcod, usually points to a serialization (pickling) problem when storing large arrays in Flink/Spark state such as ValueState, ListState, or MapStat...
- 0 kudos
- 354 Views
- 1 replies
- 0 kudos
Time window for "All tables are updated" option in job Table Update Trigger
I've been using the Table Update Trigger for some SQL alert workflows. I have a job that uses 3 tables with an "All tables updated" trigger:Table 1 was updated at 07:20 UTCTable 2 was updated at 16:48 UTCTable 3 was updated at 16:50 UTC-> Job is trig...
- 354 Views
- 1 replies
- 0 kudos
- 0 kudos
There is no fixed or documented “window” time for the interval between updates to all monitored tables before a job with an "All tables updated" trigger runs in Databricks. The job is triggered as soon as every table in the set has seen at least one ...
- 0 kudos
- 5268 Views
- 9 replies
- 2 kudos
Databricks connection to on-prem cloudera
Hello, we are trying to evaluate Databricks solution to extract the data from existing cloudera schema hosted on physical server. We are using the Databricks serverless compute provided by databricks express setup and we assume we will not need t...
- 5268 Views
- 9 replies
- 2 kudos
- 2 kudos
I work for a databricks partner called Cirata. Our Data migrator offering allows both data and metadata replication from cloudera to be delivered to the databricks environment , whether this is just delivering it to the ADLS2 object storage or to ...
- 2 kudos
- 1183 Views
- 2 replies
- 2 kudos
Resolved! Environment in serverless
I'm playing little bit with on the Databricks free environment and I'm super confused by the documentation vs actual behavior. Maybe you could help me to understand better.For the workspace I can define base environment which I can use in serverless ...
- 1183 Views
- 2 replies
- 2 kudos
- 2 kudos
Hello @pepco , Is it possible to use environments with notebook tasks? Yes—but only in a very specific way. Notebook tasks can use base environments, but you don’t attach them in the job’s YAML. You pick the base env in the notebook’s Environment sid...
- 2 kudos
- 651 Views
- 1 replies
- 0 kudos
On Prem MS sql to Azure Databricks
Hi allI need to ingest data from on prem MS sql tables using Databricks to Azure Cloud. For the ingest, previously I used notebooks, jdbc connectors, read sql tables and write in unity catalog tables. Now, I want to experiment Databricks connectors f...
- 651 Views
- 1 replies
- 0 kudos
- 0 kudos
This is feature is good to go... I can't think of any disadvantages. Here is a guide. https://landang.ca/2025/01/31/simple-data-ingestion-from-sql-server-to-databricks-using-lakeflow-connect/
- 0 kudos
- 282 Views
- 1 replies
- 0 kudos
How have you set up a governance structure (data access control, workspace management, cluster polic
If your company uses Databricks with many people, how do you manage security, organize teams, and control costs — and what tools do you use to make it all work smoothly?
- 282 Views
- 1 replies
- 0 kudos
- 0 kudos
Please take a look here to get some initial ideas. https://medium.com/databricks-unity-catalog-sme/a-practical-guide-to-catalog-layout-data-sharing-and-distribution-with-databricks-unity-catalog-763e4c7b7351
- 0 kudos
- 27545 Views
- 14 replies
- 10 kudos
i am getting the below error while making a GET request to job in databrick after successfully running it
"error_code": "INVALID_PARAMETER_VALUE", "message": "Retrieving the output of runs with multiple tasks is not supported. Please retrieve the output of each individual task run instead."}
- 27545 Views
- 14 replies
- 10 kudos
- 10 kudos
Hi @Debayan I'd suggest to also mention this explicitly in the documentation of the workspace client for get_run_outputOne has to pay extra attention to the examplerun_id=run.tasks[0].run_id otherwise it can be easily missed.
- 10 kudos
- 11817 Views
- 5 replies
- 1 kudos
Getting non-traceable NullPointerExceptions
We're running a job that's issuing NullPointerException without traces of our job's code.Does anybody know what would be the best course of action when it comes to debugging these issues?The job is a Scala job running on DBR 11.3 LTS.In case it's rel...
- 11817 Views
- 5 replies
- 1 kudos
- 1 kudos
You could try enabling full stack traces and checking the Spark executor logs for hidden errors. Null Pointer Exceptions in Scala on DBR often come from lazy evaluations or missing schema fields during I/O. Reviewing your Data Frame transformations a...
- 1 kudos
- 5409 Views
- 4 replies
- 2 kudos
Convert EBCDIC (Binary) file format to ASCII
Hi Team,How can we convert EBCDIC (Binary) file format to ASCII in databricks? Do we have any libraries in Databricks?
- 5409 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi Phani1 Were you able to do that successfully ? Can you share the details and steps please. Thanks.
- 2 kudos
- 397 Views
- 1 replies
- 1 kudos
Simple integration to push data from third-party into a client's Databricks instance
Hi there, we have an industry data platform with multiple customers using it. We provide each customer with their own data every night via .csv. Some of our customers use Databricks, and import their data from us into it.We would like to offer a more...
- 397 Views
- 1 replies
- 1 kudos
- 1 kudos
You could use external volumes with a Cloudflare R2 bucket as an intermediary - you write the nightly data files to R2 (using S3-compatible API), and your customers create external volumes in their Databricks workspace pointing to their designated R2...
- 1 kudos
- 619 Views
- 2 replies
- 1 kudos
Resolved! org.apache.spark.SparkRuntimeException: [UDF_USER_CODE_ERROR.GENERIC]
Hi EveryoneI am writing a small function, with spark read from a csv and spark write into a table. I could execute this function within the notebook. But, when I register the same function as a unity catalog function and calling it from Playground, i...
- 619 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @GiriSreerangam, You cannot use a Unity Catalog user-defined function (UDF) in Databricks to perform Spark read from a CSV and write to a table. Unity Catalog Python UDFs execute in a secure, isolated environment without access to the file system ...
- 1 kudos
- 645 Views
- 1 replies
- 2 kudos
dbt Cloud + Databricks SQL Warehouse with microbatching (48h lookback) — intermittent failures
Hey everyone,I’m currently running hourly dbt Cloud job (27 models with 8 threads) on a Databricks SQL Warehouse using the dbt microbatch approach, with a 48-hour lookback window.But I’m running into some recurring issues:Jobs failing intermittentlyO...
- 645 Views
- 1 replies
- 2 kudos
- 2 kudos
Here are few options you can try and see if it resolves your issue.1. SQL Warehouse TuningUse Serverless SQL Warehouse with Photon for faster spin-up and query execution. [docs.getdbt.com]Size Appropriately: Start with Medium or Large, and enable au...
- 2 kudos
- 515 Views
- 2 replies
- 0 kudos
Databricks CLI binding storage credential to a workspace
In the documentation from Databricks it says to run the below for binding a storage credential to a workspace (after already completing step 1 to update the `isolation-mode` to be `ISOLATED`): databricks workspace-bindings update-bindings storage-cre...
- 515 Views
- 2 replies
- 0 kudos
- 0 kudos
This appears to be a documentation inconsistency. The CLI implementation seems to: 1. Require binding_type to be explicitly specified (contradicting the docs) 2. Require it to be placed within each workspace object, not as a top-level parameter ...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 555 | |
| 349 | |
| 287 |
