- 1405 Views
- 5 replies
- 3 kudos
Schedule databricks job based on custom calendar
I want to schedule the databricks jobs based on the custom calender, like skip the job run on random days or holidays.#databricks @DataBricks @DATA
- 1405 Views
- 5 replies
- 3 kudos
- 3 kudos
Hello @DataDev! Did the suggestions shared above help address your question? If so, please consider marking one or more responses as the accepted solution. If you found another approach that worked for you, sharing it with the community would be real...
- 3 kudos
- 587 Views
- 3 replies
- 3 kudos
How to load all the previous day's data only into the newly added column of the existing delta table
How to load all the previous day's data only into the newly added column of the existing delta table? Is there any option available to do that without writing any logic?
- 587 Views
- 3 replies
- 3 kudos
- 3 kudos
Hello @shan-databricks! Did the suggestions shared above help resolve your concern? If so, please consider marking one of the responses as the accepted solution. If you found a different approach that worked for you, it would be great if you could sh...
- 3 kudos
- 3959 Views
- 8 replies
- 3 kudos
Resolved! How to create a managed iceberg table via REST catalog
We're iceberg's java lib to write managed iceberg tables in databricks. We actually can create these tables using databricks as iceberg REST catalog. But this only works when we provide a partitioning spec. This is then picked up as cluster_columns f...
- 3959 Views
- 8 replies
- 3 kudos
- 3 kudos
Why are you using the iceberg-core Java library instead of an existing open source Iceberg client (like Apache Spark)? Any of these can create a table with partitions when using Unity Catalog.
- 3 kudos
- 3252 Views
- 1 replies
- 2 kudos
Resolved! uidance Required for Informatica to Databricks Workflow Migration Using AI
Hi Team,I am currently exploring approaches to convert Informatica PowerCenter workflows into Databricks-compatible code using AI capabilities. As part of this effort, I would like to highlight that Informatica generates individual XML files for each...
- 3252 Views
- 1 replies
- 2 kudos
- 2 kudos
Greetings @chirag_nagar , as you can imagine or know, migrations are extremely complex and time consuming. There are a few approaches to migrations but I want to focus on one - Bladebridge. This is a free tool provided by Databricks that is AI powe...
- 2 kudos
- 2275 Views
- 1 replies
- 1 kudos
Resolved! Azure Databricks Serverless private connection to S3 bucket
I'm looking for technical references to connect an Azure Databricks serverless workspace to an S3 bucket over a private site-to-site VPN connection. Found the following to connect AWS (consumer) to Azure (provider), but I'm looking for the other way....
- 2275 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @mgcasas-aws Thank you for your question! We’re currently working on a solution for private cross-cloud Delta Sharing (Azure → AWS). In the meantime, here’s a possible approach: Update your Azure Storage Account network settings from private e...
- 1 kudos
- 925 Views
- 2 replies
- 0 kudos
pyspark.testing.assertSchemaEqual() ignoreColumnOrder parameter exists in 3.5.0 only on Databricks
Hi, I am using the pyspark.testing.assertSchemaEqual() function in my code using the ignoreColumnOrder parameter that is available since pyspark 4.0.0. https://spark.apache.org/docs/4.0.0/api/python/reference/api/pyspark.testing.assertSchemaEqual.htm...
- 925 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Rainer When you use Databricks Connect, your local code is executed against the Databricks cluster, which uses the Databricks Runtime’s PySpark, not your local PySpark installation. meaning your master driver node is also running on remote comput...
- 0 kudos
- 952 Views
- 1 replies
- 0 kudos
git actions to deploy dabs
HI All,I am trying to create a dabs deployment via git actions. However I am getting below error constantly. Requesting your suggestion on what I am doing wrong here? Thanks.Note we are using OIDC authentication from git. Our company has disabled tok...
- 952 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @karthikmani Databricks CLI is not able to find or use the right authentication method/environment variables for OIDC. can you try with this : In your databricks.yaml, you have auth_type: github-oidc and in your workflow you use DATABRICKS_AUTH_TY...
- 0 kudos
- 1545 Views
- 3 replies
- 2 kudos
Resolved! org.apache.hadoop.hive.ql.metadata.HiveException: MetaException
Hi Data Enthusiasts,I have been facing few errors in SQL warehouse for quiet a long time and its happening pretty randomly.We checked query runs and captured the errors below.I believe this is something to do with hive. And I am facing this when ther...
- 1545 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @rohith_23 , These errors all relate to problems communicating with the Hive Metastore Service (HMS), which is the central component to store metadata (schemas, table locations, column types, etc.) about your tables. The core of the issue in all t...
- 2 kudos
- 762 Views
- 1 replies
- 1 kudos
Resolved! Power BI refresh history info is different from ADF monitor info
In Azure Data Factory, I have a pipeline defined with Bronze - Silver - Gold layers + the final step 4 entails the refresh of a Power BI semantic model.This final step is executed via a Databricks notebook and contains the following tasks:# getting p...
- 762 Views
- 1 replies
- 1 kudos
- 1 kudos
considering the logging of the executed notebook:Failed to refresh dataset: {"error":{"code":"ItemNotFound","message":"Dataset \"xxxxxxxxxxxxxxxx\" is not found!Please verify datasetId is correct and user have sufficient permissions."}}The dataset wa...
- 1 kudos
- 2249 Views
- 1 replies
- 1 kudos
Resolved! Service principle used in Bitbucket CICD pipelines not working
The Databricks Asset Bundle is used for the Bitbucket CICD pipelines.The service principle is used in both of the local Databricks configuration and Bitbucket CICD environment.The service principle is only working in the local environment for deploym...
- 2249 Views
- 1 replies
- 1 kudos
- 1 kudos
Hey PaulChen Having your Databricks service principal (SP) work locally but fail in Bitbucket CI/CD usually means environment variables aren’t set up right, or the pipeline is falling back to an unexpected config. Quick checklist: Set all SP creden...
- 1 kudos
- 2554 Views
- 2 replies
- 1 kudos
Monitoring structured streaming and Log4J properties
Hi guys, I would like to monitor streaming job on metrics like delay, processing time and more. I found this documentation but I get message on starting and terminating phase and not while I process a record. The job is a pretty easy streaming which ...
- 2554 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @cz0 The StreamingQueryListener in Spark is designed to give you metrics at the micro-batch level (not per individual record), which is typical for Spark Structured StreamingonQueryStarted: Called when the streaming job starts.onQueryProgress: Ca...
- 1 kudos
- 1720 Views
- 1 replies
- 1 kudos
Resolved! When embedding redash, is it possible to make it visible without an account?
Same as the title.I created a redash in Databricks and want to embed it to show it in another portal. However, other users have accounts in the portal but not in Databricks.In this case, is it possible to show Redash to all these users?
- 1720 Views
- 1 replies
- 1 kudos
- 1 kudos
It is possible to embed a Redash dashboard created in Databricks into another portal so that users without Databricks accounts can view it, but this requires specific setup and permission management. How Embedding Works To show the dashboard to users...
- 1 kudos
- 1281 Views
- 3 replies
- 1 kudos
Resolved! Summarized Data from Source system into Bronze
Hello,We are just starting with Databricks. Quick question. We have a table in our legacy source system that summarizes values that are used on legacy reports and used for payment in our legacy system. The business wants a dashboard on our new plat...
- 1281 Views
- 3 replies
- 1 kudos
- 1 kudos
Hello @TinaDouglass! Did the suggestions shared above help address your concern? If so, please consider marking one as the accepted solution.
- 1 kudos
- 2183 Views
- 16 replies
- 4 kudos
Resolved! Control Databricks Platform version
Hello there! I have noticed that my Databricks UI has been changed from time to time, I was wondering how can I control the Databricks Platform version so I don't keep having new changes and new ways/names in my UI. I have found a release page https:...
- 2183 Views
- 16 replies
- 4 kudos
- 4 kudos
Hi, Can we get an official feedback when the issue with adding new users/groups in Databricks apps UI (not working) will be fixed? I tried with python SDK as well and it does not work. The issue was posted previously and applies to my case as well....
- 4 kudos
- 2719 Views
- 2 replies
- 2 kudos
Temporary view
Hi,I wrote a query to create temp view in my catalog, query execution was successful and returned the result as 'OK' in SQL editor window. However, when I executed the command 'Show Tables' and' Select * {temp_view}', it's not identifying the view. W...
- 2719 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @y_sanjay ,I guess that is somehow related to how sessions are managed within SQL Editor. For instance when I ran following queries in SQL Editor all at once it worked and I've got 3 result sets:1) First result set with OK status - which means tha...
- 2 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
