- 1010 Views
- 1 replies
- 0 kudos
An error occurred when loading Jobs and Workflows App.
Hi,I was trying to open the Workflows but there is an error "An error occurred when loading Jobs and Workflows App." we need help to know why it happened and how we can resolve it please.
- 1010 Views
- 1 replies
- 0 kudos
- 0 kudos
Same...and the weirdest is that all of the services looks healthy in https://status.databricks.com/Region: eu-central-1Provider: AWSCould anyone provide some info here?
- 0 kudos
- 1196 Views
- 0 replies
- 0 kudos
Migrate a notebook that reside in workspace using Databricks Asset Bundle
Hello Community Folks -Did anyone implemented migration of notebooks that is in workspace to production databricks workspace using Databricks Asset Bundle? If so can you please help me with any documentation which I can refer? Thanks!!RegardsNiruban ...
- 1196 Views
- 0 replies
- 0 kudos
- 1214 Views
- 1 replies
- 0 kudos
Cached Views in MERGE INTO operation
Hi everyone!I want to use in-memory cached views in a merge into operation, but I am not entirely sure if the exactly saved in-memory view is used in this operation or not.So, suppose I have a table named table_1 and a cached view named cached_view_1...
- 1214 Views
- 1 replies
- 0 kudos
- 0 kudos
@deng_dev - Are you using external metastore by any chance. From the physical plan, we could see the catalog`.`db`.`table_1` is not cached. If it is glue catalog, then caching can be enabled based on the below configs in the article below https://do...
- 0 kudos
- 16365 Views
- 15 replies
- 8 kudos
Resolved! What are some best practices for CICD?
A number of people have questions on using Databricks in a productionalized environment. What are the best practices to enable CICD automation?
- 16365 Views
- 15 replies
- 8 kudos
- 8 kudos
Any leads/posts for Databricks CI/CD integration with Bitbucket pipeline. I am facing the below error while I creation my CICD pipeline pipelines:branches:master:- step:name: Deploy Databricks Changesimage: docker:19.03.12services:- dockerscript:# U...
- 8 kudos
- 1329 Views
- 0 replies
- 0 kudos
Error during deserializing protobuf data
I am receiving protobuf data in a json attribute and along with it I receive a descriptor file.I am using from_protobuf to deserialize the data as below,It works most of the time but giving error when there are some recursive fields within the protob...
- 1329 Views
- 0 replies
- 0 kudos
- 14787 Views
- 7 replies
- 0 kudos
Spark CSV file read option to read blank/empty value from file as empty value only instead Null
Hi,I am trying to read one file which having some blank value in column and we know spark convert blank value to null value during reading, how to read blank/empty value as empty value ?? tried DBR 13.2,14.3I have tried all possible way but its not w...
- 14787 Views
- 7 replies
- 0 kudos
- 0 kudos
OK, after some tests:The trick is in surrounding text in your csv with quotes. Like that spark can actually make a difference between a missing value and an empty value. Missing values are null and can only be converted to something else implicitel...
- 0 kudos
- 1660 Views
- 2 replies
- 0 kudos
Pushdown in Postgres
Hi team,In Databricks I need to query a postgres source likeselect * from postgres_tbl where id in (select id from df)the df is got from a hive table. If I use JDBC driver, and doquery = '(select * from postgres_tbl) as t' src_df = spark.read.format(...
- 1660 Views
- 2 replies
- 0 kudos
- 0 kudos
Thanks for response. I cannot do that as we incrementally loading from source very frequently. We cannot read full data each time.
- 0 kudos
- 4056 Views
- 2 replies
- 1 kudos
Resolved! Best way to parse Google Analytics data in Databricks notebook
I managed to extract the Google Analytics data via lakehouse federation and the Big Query connection but the events table values are in a weird JSON format{"v":[{"v":{"f":[{"v":"ga_session_number"},{"v":{"f":[{"v":null},{"v":"2"},{"v":null},{"v":null...
- 4056 Views
- 2 replies
- 1 kudos
- 1 kudos
@AnaMocanu I was using this function, with a little modifications on my end:https://gist.github.com/shreyasms17/96f74e45d862f8f1dce0532442cc95b2Maybe this will be helpful for you
- 1 kudos
- 1686 Views
- 1 replies
- 0 kudos
Can I see queries sent to All Purpose Compute from Power BI?
I am brand new to Databricks and am working on connecting a power bi semantic model to our databricks instance. I have successfully connected it to an All Purpose Compute but was wondering if there was a way I could see the queries that power bi is ...
- 1686 Views
- 1 replies
- 0 kudos
- 0 kudos
For All purpose compute, best bet would be to use the system tables,specifically the system.access.audit table. https://docs.databricks.com/en/administration-guide/system-tables/index.html
- 0 kudos
- 13009 Views
- 1 replies
- 0 kudos
How to Implement Custom Logging in Databricks without Using _jvm Attribute with Spark Connect?
Hello Databricks Community,I am currently working in a Databricks environment and trying to set up custom logging using Log4j in a Python notebook. However, I've run into a problem due to the use of Spark Connect, which does not support the _jvm attr...
- 13009 Views
- 1 replies
- 0 kudos
- 0 kudos
import logging logging.getLogger().setLevel(logging.WARN) log = logging.getLogger("DATABRICKS-LOGGER") log.warning("Hello")
- 0 kudos
- 4367 Views
- 4 replies
- 1 kudos
Resolved! Pyspark operations slowness in CLuster 14.3LTS as compared to 13.3 LTS
In my notebook, i am performing few join operations which are taking more than 30s in cluster 14.3 LTS where same operation is taking less than 4s in 13.3 LTS cluster. Can someone help me how can i optimize pyspark operations like joins and withColum...
- 4367 Views
- 4 replies
- 1 kudos
- 2805 Views
- 3 replies
- 1 kudos
Customize job run name when running jobs from adf
Hi guys, i am running my Databricks jobs on a cluster job from azure datafactory using a databricks Python activity When I monitor my jobs in workflow-> job runs . I see that the run name is a concatenation of adf pipeline name , Databricks python ac...
- 2805 Views
- 3 replies
- 1 kudos
- 1 kudos
I don't think that level of customisation is provided. However, I can suggest some workarounds:REST API: Create a job on the fly with desired name within ADF and trigger it using REST API in Web activity. This way you can track job completion status ...
- 1 kudos
- 5923 Views
- 2 replies
- 3 kudos
Resolved! Could not initialize class error
User is running a job triggered from ADF in Databricks. In this job they need to use custom libraries that are in jars. Most of the times jobs are running fine, however sometimes it fails with:java.lang.NoClassDefFoundError: Could not initializeAny s...
- 5923 Views
- 2 replies
- 3 kudos
- 3 kudos
Can you please check if there are more than one jar containing this class . If multiple jars of the same type are available on the cluster, then there is no guarantee of JVM picking the proper classes for processing, which results in the intermittent...
- 3 kudos
- 4049 Views
- 3 replies
- 2 kudos
Resolved! [Databricks Assets Bundles] Workflow trigger on file arrival
Hi everyone!I'm setting up a workflow using Databricks Assets Bundles (DABs). And I want to configure my workflow to be trigger on file arrival. However all the examples I've found in the documentation use schedule triggers. Does anyone know if it is...
- 4049 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Jorge3 Yes, you can use continues mode also.Please find syntax below - resources: jobs: dbx_job: name: continuous_job_name continuous: pause_status: UNPAUSED queue: enabled: true
- 2 kudos
- 3097 Views
- 1 replies
- 1 kudos
Addressing Pipeline Error Handling in Databricks bundle run with CI/CD when SUCCESS WITH FAILURES
I'm using Databricks asset bundles and I have pipelines that contain "if all done rules". When running on CI/CD, if a task fails, the pipeline returns a message like "the job xxxx SUCCESS_WITH_FAILURES" and it passes, potentially deploying a broken p...
- 3097 Views
- 1 replies
- 1 kudos
- 1 kudos
Awesome answer, I will try the first approach. I think it is a less intrusive solution than changing the rules of my pipeline in development scenarios. This way, I can maintain a general pipeline for deployment across all environments. We plan to imp...
- 1 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 485 | |
| 349 | |
| 287 |
