Hi Data bricks team , It is informed that all the lifetime points on community will be retired by June 2023. Is it the same for brightsites reward store also .Will the brightsites get restored or lost along with community life time points ? @Vidula...
@Priyadarshini G So, we are updating our community platform to provide a better experience for our users and members. As part of the updates, we plan on introducing a new recognition protocol that will better celebrate members for their contribution...
Hi y'all.I'm trying to export metrics and logs to AWS cloudwatch, but while following their tutorial to do so, I ended up facing this error when trying to initialize my cluster with an init script they provided.This is the part where the script fail...
I am facing the same issue . error details|+ tree /home/ubuntu/databricks/spark/dbconf | |+ cat /home/ubuntu/databricks/spark/dbconf/log4j/driver/log4j2.xml | |+ sed -i '/^log4j.appender.publicFile.layout/ s/^/#/g' /home/ubuntu/databricks/spark/dbcon...
When ever I am trying to run and load multiple files in single dataframe for processing (overall file size is more than 15 gb in single dataframe at the end of the loop, my code is crashing everytime with the below error...ConnectException error: Thi...
@Satish Agarwal It seems your system memory is not sufficient to load the 15GB file. I believe you are using Python Pandas data frame for loading 15GB file and not using Spark. Is there any particular reason that you cannot use Spark for this.
I'm utilizing SQL to perform aggregation operations within a gold layer of a DLT pipeline. However, I'm encountering an error when running the pipeline while attempting to return a data frame using spark.sql.Could anyone please assist me with the SQL...
Very often, we need to know how many files my table path contains and the overall size of the path for various optimizations. In the past, I had to write my own logic to accomplish this.Delta Lake is making life easier. See how simple it is to obtain...
I have created two notebooks in my workspace one(A) is having function and other one(B) is having the main code.I am trying to use the %run magic function to use the functions available in notebook A from notebook B. Both notebooks are on the same pa...
@SK ASIF ALI It's not supported in community if you are using premium version then this should workCan you please send the snippet of the error message? #DAIS2023
Hi guys, I am trying to delete an external table in Databricks that is connected to a Synapse table (via the com.databricks.spark.sqldw connector) but got the error shown below:Apart from that, when I try to execute an DELETE and INSERT actions (like...
Is there a way to manually update the cuda required file in the db runtime?There are some rather annoying bugs still in TF 2.11 that have been fixed in TF 2.12.Sadly the latest DB runtime 13.1 (beta) only supports the older TF 2.11 even tho 2.12 was ...
Hi @Pedro Medeiros Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answer...
While working on video analytics task I need to save the image bytes to the delta table earlier extracted into the spark dataframe. While I want to over write a same delta table over the period of complete task and also the size of input data differs...
I want to use an Init Script on ADLS Gen2 location for my Azure Databricks 11.3 and 12.2 clusters. The init_script.sh is placed in a directory that has spaces in it:https://storageaccount1.blob.core.windows.net/container1/directory%20with%20spaces/su...
Hi Team, Is it possible I can use "query_start_time_range" filter from the api call to get the query data only from now to 5 mins ago?Im using telegraf to call query history api but it looks like Im reaching the max return and I can't find how to use...
Have you checked this https://docs.databricks.com/api-explorer/workspace/queryhistory/list you can list the queries based on time range as well. So you can try passing the fields in the filter_by parameter. Then pass the value as (current time - 5 m...
I would like to understand who provides the server when using Delta sharing? If a customer exposes their delta table through Delta sharing, is it the customer who needs to setup a cluster or server to process the incoming requests?
The producer does need a cluster to set up Delta Sharing. However, once the handoff happens no cluster is needed, the data will be delivered via storage services.
