Hello I am trying to set max batch size for pandas-udf in Databricks notebook, but in my tests it doesn’t have any effect on size. spark.conf.set("spark.sql.execution.arrow.enabled", "true")spark.conf.set('spark.sql.execution.arrow.maxRecordsPerBatch...
Hi,I'm using Databricks Connect to run Scala code from IntelliJ on a Databricks single node cluster.Even with the simplest code, I'm experiencing this error:org.apache.spark.SparkException: grpc_shaded.io.grpc.StatusRuntimeException: INTERNAL: org.ap...
Could it be nice to enable the use of one own photo instead of a ready avatar?
I think you can tag the Admins or the Community managers where they can help you with some relevant answers , tagging a few I know.@Kaizen@Sujitha @s_park @
I and 3 of my friends can't open https://community.databricks.com/ from Poland. The page returns 403. Different LTE and internet providers
I'm having the same issue while connecting from my ISP from Poland. @Retired_mod can you please help us with this?
Hello,I am seeing this message on all databricks environments.Action Required: Init Scripts on DBFS will stop working on March 31, 2024You must migrate init scripts of DBFS or opt-in to a september 1, 2024 extension by March 31Not sure what this is f...
How are yall managing your init script versions? I have mine in a repo i connected into databricks workspace and im using it within there. With multiple users they can push it to the repo and we can merge it in.
Hello, I'm setting up a python environment to work with databricks in vscode using Windows.I followed the instructions in Install Databricks Connect for Python, and found some issues/conflicts/errors:- There is a useless tip in the Set up the client ...
Additionally, there are no comments on how to avoid uploading virtual environment to the catalog when using databricks extension for vscode
Hey there, we're using DBR 13.3 (no Docker) as general purpose cluster and init the cluster using the following init script:```#!/usr/bin/env bashexport DEBIAN_FRONTEND=noninteractiveset -euxo pipefailif [[ $DB_IS_DRIVER = "TRUE" ]]; thenecho "I am d...
I was trying the new lakehouse monitoring feature for one of my unity tables, and when I create a monitoring dashboard for my table, the 2 metric tables({output_schema}.{table_name}_profile_metrics and {output_schema}.{table_name}_drift_metrics) are ...
Hi Team,This is the second time I am tryng to post a discussion in the community. The first time it was marked SPAM. It has been a terrible experience so far with my Databricks Data Engineer Associate exam getting suspended by the proctor for reasons...
@RohitB91 the support team is awaiting your response for a date / time you are available to resume your exam. Your registration confirmation email should contain this detail: Preparation for taking your Online proctored Exam Understand the computer r...
Hi Team,Good morning, I am Sai Rekhawar, currently residing in Hamburg, Germany. I am reaching out regarding an issue with the search engine functionality in Databricks.I have created several catalogs and tables within the catalog, and I am the owner...
Hi @SaiRekhawar , try using the filters in under the table bar in ctrl+P as below
HelloI have a query in one of my notebooks that took around 3.5 hours on D12_V2 cluster and workers between 5 to 25 .is there a way to write the query in diffrenet way in order to improve performance and cost : select /*+ BROADCAST(b) */ MD5(CONCAT(N...
Hello Folks, We are recently working on Databricks integration in to our products and one of the Best practices suggested is to send user-agent information for any REST API or JDBC connect we make from the product to Databricks. We have made all the ...
Hi @bdraitcak , Here are the steps to be followed to get the logs: Log in to your Azure portal.Go to / Search for "Log analytics workspace."Create a new Log Analytics workspace by specifying your Resource group and Instance details. [subscription + r...
I have found that the results of the bitmap_count() function output differs significantly between databricks and snowflake.eg: snowflake returns a value of '1' for this code. "select bitmap_count(X'0001056c000000000000') " while Databricks returns a...
Hi @vigneshp , Good Day! In Databricks, bitmap_count function returns the number of bits set in a BINARY string representing a bitmap. This function is typically used to count distinct values in combination with the bitmap_bucket_number() and the bi...
I am looking to convert pkl file to parquet format. I am reading this data from Azure Data Lake and convert and create parquet file outputs to store them in my hive metastore.
Hi - we had a few issues with some of our init scripts recently. Investigating I found that mesa packages were throwing issues when trying to install. Posting this to help the community and raise awareness to Databricks to fix itI believe the image f...
