Hi All,We have a couple of jars stored in a workspace folder.We are using init scripts to copy the jars in the workspace to the /databricks/jars path.The init scripts do not seem to be able to find the files. The scripts are failing saying the files ...
Hi @Bharath Kumar Ramachandran You're welcome! I'm glad you found the link useful. I empathize with your hope that Databricks would consider adding this option. It's possible that Databricks will take user feedback into account when planning future ...
I have the following code:from pyspark.sql.functions import *
!pip install dbl-tempo
from tempo import TSDF
from pyspark.sql.functions import *
# interpolate target_cols column linearly for tsdf dataframe
def interpolate_tsdf(tsdf_data, target_c...
Consider we have two tables A & B.qry = """INSERT INTO Table ASelect * from Table B where Id is null """spark.sql(qry)I need to get the number of records inserted after running this in databricks.
Almost same advice than Hubert, I use the history of the delta table :df_history.select(F.col('operationMetrics')).collect()[0].operationMetrics['numOutputRows']You can find also other 'operationMetrics' values, like 'numTargetRowsDeleted'.
When running some SQL queries using spark.sql(...), we sometimes get a variant of the following error:AnalysisException: Undefined function: current_timestamp. This function is neither a built-in/temporary function, nor a persistent function that is ...
I want to overwrite a Postgresql table transactionStats which is used by the customer facing dashboards.This table needs to be updated every 30 mins. I am writing a AWS Glue Spark job via JDBC connection to perform this operation.Spark dataframe writ...
Hi @Siddharth Kanojiya We haven't heard from you since the last response from @werners (Customer) . Kindly share the information with us, and in return, we will provide you with the necessary solution.Thanks and Regards
I am running this Delta Live Tables PoC from databricks-industry-solutions/industry-solutions-blueprintshttps://github.com/databricks-industry-solutions/pos-dltI have Standard_DS4_v2 with 28GB and 8 cores x 2 workers - so a total of 16 cores. This is...
Hi @Prasenjit Biswas We haven't heard from you since the last response from @Jose Gonzalez . Kindly share the information with us, and in return, we will provide you with the necessary solution.Thanks and Regards
I need to login and I can't to try Databricks..so you have a OAUTH issue...I cant try Databricks at all because the country icon doesn't work and it sends a URI issue from your front end to the back end..."Request_URI=&Geo_country_code=&Geo_country_i...
Hi @Jessica Woods Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers...
I have been struggling to install Detectron 2. I think it is an issue with Cuda. Any adivise?install!pip install -q "detectron2@git+https://github.com/facebookresearch/detectron2.git@e2ce8dc#egg=detectron2"Error:qm8vsdal/detectron2_85faeed5ce7945dbad...
Hi Team,We have few prod tables which are created in s3 bucket, that have grown now very large, these tables are getting real time data continuously from round the clock databricks workflows; we would like run the optimization commands(optimize, zord...
Hi @Sriram Kumar We haven't heard from you since the last response from @Suteja Kanuri . Kindly share the information with us, and in return, we will provide you with the necessary solution.Thanks and Regards
I'm using Azure Databricks and I'd like to create a project virtual environment, persisted on a shared compute cluster. As the cluster is shared for many projects, it is necessary to have virtual environments if I want to execute code runs from withi...
Hi @Joshua L We haven't heard from you since the last response from @Debayan Mukherjee , and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to ot...
I am trying to start the same Jobs multiple times using the python sdk's "run_now" command.If the number of requests exceeds the Maximum concurrent runs, the status of the run will be Skipped and the run will not be executed.Is there any way to queue...
Hi, We do have a private preview feature which will be enabled shortly for queueing. Please tag me (@Debayan Mukherjee ) with your next update so that I will get notified.
I am getting an error about the `drop function of pyspark` at a cluster using 12.2 LTS. When I check the error I see spark solved that bug, see SPARK-42444. Also when I check maintenance updates page, I saw this solved issue included the Databricks R...
Hi @Sevval Mehder Elevate our community by acknowledging exceptional contributions. Your participation in marking the best answers is a testament to our collective pursuit of knowledge.
Hi @Ramdas Panicher Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answe...
Hi @Ayush Raj Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers you...
I have integrated gitlab with my azure databricks repo, I am able to push and pull commits from the databricks UI, I want to checkout to a specific commit version via databricks UI.Note: I am aware that via the gitlab i have checkout to specific vers...
After getting more context on databricks repo in details,Currently databricks doesn't support checkout of repo to specific commit.databricks provides only limited functionality mentioned belowAdd a repo and connect remotely laterClone a repo connecte...
