cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
sanjay
Valued Contributor II
since ‎10-15-2021
‎10-30-2024

User Stats

  • 64 Posts
  • 2 Solutions
  • 7 Kudos given
  • 19 Kudos received

User Activity

Remove duplicate records using pyspark

08-28-2024
Hi,I am trying to remove duplicate records from pyspark dataframe and keep the latest one. But somehow df.dropDuplicates["id"] keeps the first one instead of latest. One of the option is to use pandas drop_duplicates, Is there any solution in pyspark...

maxFilesPerTrigger not working while loading data from Unity Catalogue table

08-28-2024
Hi,I am using streaming on unity catalogue tables and trying to limit the number of records read in each batch. Here is my code but its not respecting maxFilesPerTrigger, instead reads all available data. (spark.readStream.option("skipChangeCommits",...

how to reduce scale to zero time in MLFlow Serving

07-28-2024
Hi,I am deploying MLflow models using Databrick serverless serving but seems servers scale down to 0 only after 30 minute of inactivity. Is there any way to reduce this time?Also, Is it possible to deploy multiple models under single endpoint. I want...

Error while running Job sparkSession is null while trying to executeCollectResult

06-28-2024
Hi,I have started getting following error while running jobs in databrick. It started failing since last few days. Recently I have migrated to unity catalogue, no other change was made recently. I am running on DBR 13.3 LTS. com.google.common.util.co...

Deploy mlflow model to Sagemaker

02-19-2024
Hi,I am trying to deploy mlflow model in Sagemaker. My mlflow model is registered in Databrick.Followed below url to deploy and it need ECR for deployment. For ECR, either I can create custom image and push to ECR or its mentioned in below url to get...
View more