- 2114 Views
- 1 replies
- 0 kudos
Even the Unfinished Experiment in Mlflow is getting saved as finished
when I start the experiment with mlflow.start_run(),even if my script is interrupted or failed before executing mlflow.end_run() ,the run gets tagged as finished instead of unfinished , Can any one help why it is happening here
- 2114 Views
- 1 replies
- 0 kudos
- 0 kudos
In note book the mlflow tagas ias the command travels and once failed or exit there itself it logs and finishes the experiment even if the noteboolsfails. However, if you want to continue logging metrics or artifacts to that run, you just need to use...
- 0 kudos
- 2154 Views
- 1 replies
- 0 kudos
Resolved! Why is my streaming job not resuming even though I specified checkpoint directory
I have provided the checkpointLocation as below, however I see the config is ignored for my streaming queryoption("checkpointLocation", "path/to/checkpoint/dir")
- 2154 Views
- 1 replies
- 0 kudos
- 0 kudos
This is a common question from many users. If the streaming checkpoint directory is specified correctly then this behavior is expected. Below is an example of specifying the checkpoint correctlydf.writeStream .format("parquet") .option("checkpo...
- 0 kudos
- 2726 Views
- 1 replies
- 0 kudos
Resolved! Is there any way to control the autoOptimize interval?
I can see my streaming jobs running optimize jobs more frequently, Is there any property so I can control autoOptimize duration
- 2726 Views
- 1 replies
- 0 kudos
- 0 kudos
The autoOptimize is not performed on a time basis. It's an event-based trigger. Once the delta table/partition has 50 (default value of spark.databricks.delta.autoCompact.minNumFiles) files, auto-compaction is triggered. To reduce the frequency, inc...
- 0 kudos
- 4496 Views
- 1 replies
- 0 kudos
How to change the time zone in notebook ,
How to change the time zone in notebook ,
- 4496 Views
- 1 replies
- 0 kudos
- 0 kudos
import java.util.TimeZone spark.conf.set("spark.sql.session.timeZone", "Asia/Calcutta")TimeZone.setDefault(TimeZone.getTimeZone("Asia/Calcutta"))Scalaimport java.timeval s: String = time.LocalDateTime.now().toStringprintln(s)sql %sqlselect current_t...
- 0 kudos
- 3304 Views
- 1 replies
- 0 kudos
Resolved! How Can I update the DBR versions of all my jobs in one go?
I keep it a point to use the latest DBR versions for my workloads and mostly we leverage those new features. But I have 300 jobs on the Databricks workspace and updating the DBR versions for each job manually is difficult to do. Any quick hack
- 3304 Views
- 1 replies
- 0 kudos
- 0 kudos
Below code snippet can be helpful if you are using Databricks CLIfor jobid in `databricks jobs list | awk '{print $1}'`; do databricks jobs get --job-id $jobid | jq .settings > /tmp/jobs/$jobid.json; done sed -i 's/"spark_version": ".*"/"spark_ver...
- 0 kudos
- 10003 Views
- 1 replies
- 0 kudos
How to add a new datetime column to a spark dataFrame from existing timestamp column
I have a data frame in Spark that has a column timestamp. I want to add a new column to this data frame that has the DateTime in the below format created from this existing timestamp column.“YYYY-MM-DD HH:MM:SS”
- 10003 Views
- 1 replies
- 0 kudos
- 0 kudos
val df = Seq(("2021-11-05 02:46:47.154410"),("2019-10-05 2:46:47.154410")).toDF("old_column")display(df)import org.apache.spark.sql.functions._val df2 = df.withColumn("new_column", from_unixtime(unix_timestamp(col("old_column"), "yyyy-MM-dd HH:mm:ss....
- 0 kudos
- 2565 Views
- 1 replies
- 0 kudos
Resolved! What is the trade-off of using an unsupported DBR version on my cluster?
I do not want to upgrade my cluster every one month. I am looking for stability over new features.
- 2565 Views
- 1 replies
- 0 kudos
- 0 kudos
The strong recommendation is not to use an unsupported version of DBR on your cluster. For production workloads where you don't welcome newer versions, then check the Databricks LTS DBR versions. if using an unsupported version then you don't receiv...
- 0 kudos
- 3041 Views
- 1 replies
- 0 kudos
Resolved! Getting file permission issues even though I have the right IAM role attached
I am reading data from S3 from a Databricks cluster and the read operation seldom fails with 403 permission errors. Restarting the cluster fixes my issue.
- 3041 Views
- 1 replies
- 0 kudos
- 0 kudos
The main reason for this behavior is : AWS keys are used in addition to the IAM role. Using global init scripts to set the AWS keys can cause this behavior.The IAM role has the required permission to access the S3 data, but AWS keys are set in the Sp...
- 0 kudos
- 3050 Views
- 1 replies
- 0 kudos
Resolved! Why do I see data loss with Structured streaming jobs?
I have a Spark structured streaming job reading data from Kafka and loading it to the Delta table. I have some transformations and aggregations on the streaming data before writing to Delta table
- 3050 Views
- 1 replies
- 0 kudos
- 0 kudos
The typical reason for data loss on a Structured streaming application is having an incorrect value set for watermarking. The watermarking is done to ensure the application does not develop the state for a long period, However, it should be ensured ...
- 0 kudos
- 2303 Views
- 1 replies
- 0 kudos
Resolved! Does Table ACL support column-level security like Ranger?
I have used Ranger in Apache Hadoop and it works fine for my use case. Now that I am migrating my workloads from Apache Hadoop to Databricks
- 2303 Views
- 1 replies
- 0 kudos
- 0 kudos
Currently, Table ACL does not support column-level security. There are several tools like Privcera which has better integration with Databricks.
- 0 kudos
- 10635 Views
- 1 replies
- 1 kudos
When to use cache vs checkpoint?
I've seen .cache() and .checkpoint() used similarly in some workflows I've come across. What's the difference, and when should I use one over the other?
- 10635 Views
- 1 replies
- 1 kudos
- 1 kudos
Caching is extremely useful than checkpointing when you have lot of available memory to store your RDD or Dataframes if they are massive.Caching will maintain the result of your transformations so that those transformations will not have to be recomp...
- 1 kudos
- 1774 Views
- 1 replies
- 0 kudos
- 1774 Views
- 1 replies
- 0 kudos
- 0 kudos
1. We want a venue in which we can rapidly iterate and make new releases. The overhead of making a release as a separate project is minuscule (in the order of minutes). A release on Spark takes a lot longer (in the order of days)2. Koalas takes a dif...
- 0 kudos
- 2062 Views
- 1 replies
- 0 kudos
- 2062 Views
- 1 replies
- 0 kudos
- 0 kudos
Different projects have different focuses. Spark is already deployed in virtually every organization, and often is the primary interface to the massive amount of data stored in data lakes. Koalas was inspired by Dask, and aims to make the transition ...
- 0 kudos
- 6660 Views
- 1 replies
- 0 kudos
Do login sessions into Databricks have an idle timeout?
Do login sessions into Databricks have an idle timeout?
- 6660 Views
- 1 replies
- 0 kudos
- 0 kudos
Short Answer:YesDetailed Answer:User sessions automatically timeout after six hours of idle time. This timeout is not configurable. User sessions are terminated if the user is removed from the workspace. To trigger session end for users who were remo...
- 0 kudos
- 1988 Views
- 1 replies
- 0 kudos
- 1988 Views
- 1 replies
- 0 kudos
- 0 kudos
for any other non-private previews, they can check out admin console --> advanced tab. there are tons of toggles there to enable/disable features. if it’s not there, there usually isn’t an easy (or direct) way of disabling
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
