- 2140 Views
- 2 replies
- 3 kudos
Resolved! handling both Pyspark and Python exceptions
In a Python notebook, I am using error handling according to the official documentation. try:[some data transformation steps]except PySparkException as ex:[logging steps to log the error condition and error message in a table]However, this catches o...
- 2140 Views
- 2 replies
- 3 kudos
- 3 kudos
To handle both PySpark exceptions and general Python exceptions without double-logging or overwriting error details, the recommended approach is to use multiple except clauses that distinguish the exception type clearly. In Python, exception handlers...
- 3 kudos
- 2699 Views
- 5 replies
- 1 kudos
Resolved! BUG in Job Task of Type DBT
Hi, just wanted to let the Databricks Team know, that there is a bug in the task ui.Currently it is not possible to save a task of "Type: dbt" if the "SQL Warehouse" is set to "None (Manual)".Some weeks ago this was possible, also the "Profiles Direc...
- 2699 Views
- 5 replies
- 1 kudos
- 1 kudos
I am facing this bug still. I don't want to select any SQL warehouse, what do I do? and from where can I pass the profiles directory
- 1 kudos
- 950 Views
- 3 replies
- 1 kudos
Resolved! I recently made new account on databricks under Free edition
It by default made SQL warehouse compute, but I want all-purpose compute, as I want test and learn capabilities of PySpark and Databricks.I can't connect with the serverless compute in the notebook; it gives a mean error as follows: "An error occurr...
- 950 Views
- 3 replies
- 1 kudos
- 1 kudos
In the free Databricks edition, to use PySpark you need to create and start a cluster, since the SQL Warehouse is only for SQL queries; go to Compute > Create Cluster, set up a free cluster, click Start, and then attach your notebook to it this will ...
- 1 kudos
- 2825 Views
- 2 replies
- 1 kudos
Unable to access metrics from Driver node on localhost:4040
Greetings,I am trying to setup monitoring in Grafana for all my databricks clustersI have added 2 things as part of thisUnder Compute > Configuration > Advanced > Spark > Spark Config, I have addedspark.ui.prometheus.enabled trueUnder init_scripts, I...
- 2825 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @vishal_balaji ,You're following guides that were prepared for OSS Apache Spark. For sure localhost won't work in this case because in Databricks all compute is cloud-based. Please follow below guide how to configure it properly on databricks:Data...
- 1 kudos
- 2440 Views
- 4 replies
- 3 kudos
Autoloader - File Notification Mode
Hello All,We have started to consume source messages/files via autoloader directory listing mode at the moment and want to convert this to file notification mode instead so consumption can be faster with no more entire directories/folder scanning. I ...
- 2440 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @K_Anudeep @szymon_dybczak how do i understand a situation when 100 jobs are running in parallel with minimal latency needed. does autoloader directly connect to the cloud queue service ? or databricks stores and manages detected files somewhere? ...
- 3 kudos
- 1010 Views
- 2 replies
- 0 kudos
Foreign Catalog Wrong Mapping - Azure SQL Database Binary Column
Hi all.I've used foreign catalog attached to azure sql databases and never had problems except in 2 situations:1) Foreign Catalogs don't support sql schemas/objects like [xxxx.yyyy].tablename. The workaround is creating views on sql database2) This i...
- 1010 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @RobsonNLPT One thing that might help to narrow this down: could you check whether the problem occurs for the entire column, or if some of the batches you receive actually contain the full (non-truncated) value?If some batches are complete but ...
- 0 kudos
- 3017 Views
- 5 replies
- 3 kudos
Schedule databricks job based on custom calendar
I want to schedule the databricks jobs based on the custom calender, like skip the job run on random days or holidays.#databricks @DataBricks @DATA
- 3017 Views
- 5 replies
- 3 kudos
- 3 kudos
Hello @DataDev! Did the suggestions shared above help address your question? If so, please consider marking one or more responses as the accepted solution. If you found another approach that worked for you, sharing it with the community would be real...
- 3 kudos
- 905 Views
- 3 replies
- 3 kudos
How to load all the previous day's data only into the newly added column of the existing delta table
How to load all the previous day's data only into the newly added column of the existing delta table? Is there any option available to do that without writing any logic?
- 905 Views
- 3 replies
- 3 kudos
- 3 kudos
Hello @shan-databricks! Did the suggestions shared above help resolve your concern? If so, please consider marking one of the responses as the accepted solution. If you found a different approach that worked for you, it would be great if you could sh...
- 3 kudos
- 6131 Views
- 8 replies
- 3 kudos
Resolved! How to create a managed iceberg table via REST catalog
We're iceberg's java lib to write managed iceberg tables in databricks. We actually can create these tables using databricks as iceberg REST catalog. But this only works when we provide a partitioning spec. This is then picked up as cluster_columns f...
- 6131 Views
- 8 replies
- 3 kudos
- 3 kudos
Why are you using the iceberg-core Java library instead of an existing open source Iceberg client (like Apache Spark)? Any of these can create a table with partitions when using Unity Catalog.
- 3 kudos
- 4411 Views
- 1 replies
- 2 kudos
Resolved! uidance Required for Informatica to Databricks Workflow Migration Using AI
Hi Team,I am currently exploring approaches to convert Informatica PowerCenter workflows into Databricks-compatible code using AI capabilities. As part of this effort, I would like to highlight that Informatica generates individual XML files for each...
- 4411 Views
- 1 replies
- 2 kudos
- 2 kudos
Greetings @chirag_nagar , as you can imagine or know, migrations are extremely complex and time consuming. There are a few approaches to migrations but I want to focus on one - Bladebridge. This is a free tool provided by Databricks that is AI powe...
- 2 kudos
- 1352 Views
- 2 replies
- 0 kudos
pyspark.testing.assertSchemaEqual() ignoreColumnOrder parameter exists in 3.5.0 only on Databricks
Hi, I am using the pyspark.testing.assertSchemaEqual() function in my code using the ignoreColumnOrder parameter that is available since pyspark 4.0.0. https://spark.apache.org/docs/4.0.0/api/python/reference/api/pyspark.testing.assertSchemaEqual.htm...
- 1352 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Rainer When you use Databricks Connect, your local code is executed against the Databricks cluster, which uses the Databricks Runtime’s PySpark, not your local PySpark installation. meaning your master driver node is also running on remote comput...
- 0 kudos
- 1554 Views
- 1 replies
- 0 kudos
git actions to deploy dabs
HI All,I am trying to create a dabs deployment via git actions. However I am getting below error constantly. Requesting your suggestion on what I am doing wrong here? Thanks.Note we are using OIDC authentication from git. Our company has disabled tok...
- 1554 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @karthikmani Databricks CLI is not able to find or use the right authentication method/environment variables for OIDC. can you try with this : In your databricks.yaml, you have auth_type: github-oidc and in your workflow you use DATABRICKS_AUTH_TY...
- 0 kudos
- 2766 Views
- 3 replies
- 2 kudos
Resolved! org.apache.hadoop.hive.ql.metadata.HiveException: MetaException
Hi Data Enthusiasts,I have been facing few errors in SQL warehouse for quiet a long time and its happening pretty randomly.We checked query runs and captured the errors below.I believe this is something to do with hive. And I am facing this when ther...
- 2766 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @rohith_23 , These errors all relate to problems communicating with the Hive Metastore Service (HMS), which is the central component to store metadata (schemas, table locations, column types, etc.) about your tables. The core of the issue in all t...
- 2 kudos
- 1095 Views
- 1 replies
- 1 kudos
Resolved! Power BI refresh history info is different from ADF monitor info
In Azure Data Factory, I have a pipeline defined with Bronze - Silver - Gold layers + the final step 4 entails the refresh of a Power BI semantic model.This final step is executed via a Databricks notebook and contains the following tasks:# getting p...
- 1095 Views
- 1 replies
- 1 kudos
- 1 kudos
considering the logging of the executed notebook:Failed to refresh dataset: {"error":{"code":"ItemNotFound","message":"Dataset \"xxxxxxxxxxxxxxxx\" is not found!Please verify datasetId is correct and user have sufficient permissions."}}The dataset wa...
- 1 kudos
- 2720 Views
- 1 replies
- 1 kudos
Resolved! Service principle used in Bitbucket CICD pipelines not working
The Databricks Asset Bundle is used for the Bitbucket CICD pipelines.The service principle is used in both of the local Databricks configuration and Bitbucket CICD environment.The service principle is only working in the local environment for deploym...
- 2720 Views
- 1 replies
- 1 kudos
- 1 kudos
Hey PaulChen Having your Databricks service principal (SP) work locally but fail in Bitbucket CI/CD usually means environment variables aren’t set up right, or the pipeline is falling back to an unexpected config. Quick checklist: Set all SP creden...
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 576 | |
| 349 | |
| 287 |
