cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results forĀ 
Search instead forĀ 
Did you mean:Ā 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results forĀ 
Search instead forĀ 
Did you mean:Ā 

Lost Databricks' dependency in a job.

KosmaS
New Contributor III

ā€Ž07-19-2024 10:56 AM - edited ā€Ž07-19-2024 10:58 AM

Hey,

I had a stable notebook within the whole job. It contains one action defined as dumping data to s3. Currently, it started generating some issues. Maybe someone can suggest either how to investigate it further or what to try to do with such kinds of issues.

AQE is turned on. I heard a suggestion it is good to turn it off in the production.
Maybe that is causing this?

Failure:

 

 

 

---------------------------------------------------------------------------
Py4JJavaError                             Traceback (most recent call last)
File <command-230092523435878>, line 5
      3 if "lin" in obj_name:
      4   coal_val = 3
----> 5 df_master.coalesce(coal_val).write.mode('overwrite').parquet(output_path)

File /databricks/spark/python/pyspark/instrumentation_utils.py:48, in _wrap_function.<locals>.wrapper(*args, **kwargs)
     46 start = time.perf_counter()
     47 try:
---> 48     res = func(*args, **kwargs)
     49     logger.log_success(
     50         module_name, class_name, function_name, time.perf_counter() - start, signature
     51     )
     52     return res

File /databricks/spark/python/pyspark/sql/readwriter.py:1721, in DataFrameWriter.parquet(self, path, mode, partitionBy, compression)
   1719     self.partitionBy(partitionBy)
   1720 self._set_opts(compression=compression)
-> 1721 self._jwrite.parquet(path)

File /databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py:1355, in JavaMember.__call__(self, *args)
   1349 command = proto.CALL_COMMAND_NAME +\
   1350     self.command_header +\
   1351     args_command +\
   1352     proto.END_COMMAND_PART
   1354 answer = self.gateway_client.send_command(command)
-> 1355 return_value = get_return_value(
   1356     answer, self.gateway_client, self.target_id, self.name)
   1358 for temp_arg in temp_args:
   1359     if hasattr(temp_arg, "_detach"):

File /databricks/spark/python/pyspark/errors/exceptions/captured.py:188, in capture_sql_exception.<locals>.deco(*a, **kw)
    186 def deco(*a: Any, **kw: Any) -> Any:
    187     try:
--> 188         return f(*a, **kw)
    189     except Py4JJavaError as e:
    190         converted = convert_exception(e.java_exception)

File /databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py:326, in get_return_value(answer, gateway_client, target_id, name)
    324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
    325 if answer[1] == REFERENCE_TYPE:
--> 326     raise Py4JJavaError(
    327         "An error occurred while calling {0}{1}{2}.\n".
    328         format(target_id, ".", name), value)
    329 else:
    330     raise Py4JError(
    331         "An error occurred while calling {0}{1}{2}. Trace:\n{3}\n".
    332         format(target_id, ".", name, value))

Py4JJavaError: An error occurred while calling o514.parquet.
: org.apache.spark.SparkException: Job aborted due to stage failure: ResultStage 276 ($anonfun$withThreadLocalCaptured$5 at LexicalThreadLocal.scala:63) has failed the maximum allowable number of times: 4. Most recent failure reason:
org.apache.spark.shuffle.FetchFailedException
	at org.apache.spark.errors.SparkCoreErrors$.fetchFailedError(SparkCoreErrors.scala:440)
	at org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:1285)
	at org.apache.spark.storage.ShuffleBlockFetcherIterator.$anonfun$next$1(ShuffleBlockFetcherIterator.scala:1011)
	at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:831)
	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:86)
	at org.apache.spark.util.CompletionIterator.next(CompletionIterator.scala:29)
	at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:31)
	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage6.hashAgg_doAggregateWithKeys_0$(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage6.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:43)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:490)
	at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.writeWithIterator(FileFormatDataWriter.scala:91)
	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeTask$2(FileFormatWriter.scala:534)
	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1743)
	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:541)
	at org.apache.spark.sql.execution.datasources.WriteFilesExec.$anonfun$doExecuteWrite$1(WriteFiles.scala:116)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:934)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:934)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:60)
	at org.apache.spark.rdd.RDD.$anonfun$computeOrReadCheckpoint$1(RDD.scala:410)
	at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:407)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:374)
	at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$3(ResultTask.scala:82)
	at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
	at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$1(ResultTask.scala:82)
	at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
	at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:196)
	at org.apache.spark.scheduler.Task.doRunTask(Task.scala:181)
	at org.apache.spark.scheduler.Task.$anonfun$run$5(Task.scala:146)
	at com.databricks.unity.UCSEphemeralState$Handle.runWith(UCSEphemeralState.scala:45)
	at com.databricks.unity.HandleImpl.runWith(UCSHandle.scala:103)
	at com.databricks.unity.HandleImpl.$anonfun$runWithAndClose$1(UCSHandle.scala:108)
	at scala.util.Using$.resource(Using.scala:269)
	at com.databricks.unity.HandleImpl.runWithAndClose(UCSHandle.scala:107)
	at org.apache.spark.scheduler.Task.$anonfun$run$1(Task.scala:146)
	at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
	at org.apache.spark.scheduler.Task.run(Task.scala:99)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$8(Executor.scala:900)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1709)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:903)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:798)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
Caused by: java.io.IOException: Failed to connect to /10.13.64.101:4048
	at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:297)
	at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:217)
	at org.apache.spark.network.shuffle.ExternalBlockStoreClient.lambda$fetchBlocks$0(ExternalBlockStoreClient.java:128)
	at org.apache.spark.network.shuffle.RetryingBlockTransferor.transferAllOutstanding(RetryingBlockTransferor.java:173)
	at org.apache.spark.network.shuffle.RetryingBlockTransferor.lambda$initiateRetry$0(RetryingBlockTransferor.java:206)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	... 1 more
Caused by: io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: No route to host: /10.13.64.101:4048
Caused by: java.net.NoRouteToHostException: No route to host
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:750)

	at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:3681)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:3603)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:3590)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:3590)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskCompletion(DAGScheduler.scala:2856)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3920)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3838)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3826)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:51)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$runJob$1(DAGScheduler.scala:1259)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:94)
	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:1247)
	at org.apache.spark.SparkContext.runJobInternal(SparkContext.scala:2966)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2949)
	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeWrite$4(FileFormatWriter.scala:410)
	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.writeAndCommit(FileFormatWriter.scala:374)
	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeWrite(FileFormatWriter.scala:407)
	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$1(FileFormatWriter.scala:292)
	at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:94)
	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:117)
	at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:194)
	at org.apache.spark.sql.execution.command.DataWritingCommandExec.$anonfun$sideEffectResult$3(commands.scala:132)
	at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:94)
	at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:130)
	at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:129)
	at org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:144)
	at org.apache.spark.sql.execution.collect.Collector$.collect(Collector.scala:102)
	at org.apache.spark.sql.execution.collect.Collector$.collect(Collector.scala:124)
	at org.apache.spark.sql.execution.qrc.InternalRowFormat$.collect(cachedSparkResults.scala:126)
	at org.apache.spark.sql.execution.qrc.InternalRowFormat$.collect(cachedSparkResults.scala:114)
	at org.apache.spark.sql.execution.qrc.InternalRowFormat$.collect(cachedSparkResults.scala:94)
	at org.apache.spark.sql.execution.qrc.ResultCacheManager.$anonfun$computeResult$1(ResultCacheManager.scala:557)
	at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:94)
	at org.apache.spark.sql.execution.qrc.ResultCacheManager.collectResult$1(ResultCacheManager.scala:545)
	at org.apache.spark.sql.execution.qrc.ResultCacheManager.$anonfun$computeResult$2(ResultCacheManager.scala:563)
	at org.apache.spark.sql.execution.adaptive.ResultQueryStageExec.$anonfun$doMaterialize$1(QueryStageExec.scala:530)
	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:1148)
	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withThreadLocalCaptured$5(SQLExecution.scala:605)
	at com.databricks.util.LexicalThreadLocal$Handle.runWith(LexicalThreadLocal.scala:63)
	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withThreadLocalCaptured$4(SQLExecution.scala:605)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withThreadLocalCaptured$3(SQLExecution.scala:604)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withThreadLocalCaptured$2(SQLExecution.scala:603)
	at org.apache.spark.sql.execution.SQLExecution$.withOptimisticTransaction(SQLExecution.scala:625)
	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withThreadLocalCaptured$1(SQLExecution.scala:602)
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
	at org.apache.spark.util.threads.SparkThreadLocalCapturingRunnable.$anonfun$run$1(SparkThreadLocalForwardingThreadPoolExecutor.scala:118)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at com.databricks.spark.util.IdentityClaim$.withClaim(IdentityClaim.scala:48)
	at org.apache.spark.util.threads.SparkThreadLocalCapturingHelper.$anonfun$runWithCaptured$4(SparkThreadLocalForwardingThreadPoolExecutor.scala:81)
	at com.databricks.unity.UCSEphemeralState$Handle.runWith(UCSEphemeralState.scala:45)
	at org.apache.spark.util.threads.SparkThreadLocalCapturingHelper.runWithCaptured(SparkThreadLocalForwardingThreadPoolExecutor.scala:80)
	at org.apache.spark.util.threads.SparkThreadLocalCapturingHelper.runWithCaptured$(SparkThreadLocalForwardingThreadPoolExecutor.scala:66)
	at org.apache.spark.util.threads.SparkThreadLocalCapturingRunnable.runWithCaptured(SparkThreadLocalForwardingThreadPoolExecutor.scala:115)
	at org.apache.spark.util.threads.SparkThreadLocalCapturingRunnable.run(SparkThreadLocalForwardingThreadPoolExecutor.scala:118)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

 

 

 

 

When I review Log4j at the end I see that some Databricks' dependency was not installed correctly and caused an error:

 

 

 

 78.0 in stage 1125.0 (TID 28009) (10.13.70.215, executor 126, partition 78, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 79.0 in stage 1125.0 (TID 28010) (10.13.67.111, executor 106, partition 79, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 80.0 in stage 1125.0 (TID 28011) (10.13.69.209, executor 117, partition 80, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 81.0 in stage 1125.0 (TID 28012) (10.13.68.167, executor 133, partition 81, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 82.0 in stage 1125.0 (TID 28013) (10.13.69.166, executor 121, partition 82, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 83.0 in stage 1125.0 (TID 28014) (10.13.67.67, executor 108, partition 83, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO MemoryStore: Block broadcast_1505 stored as values in memory (estimated size 221.4 KiB, free 25.2 GiB)
24/07/19 15:25:25 INFO MemoryStore: Block broadcast_1505_piece0 stored as bytes in memory (estimated size 73.7 KiB, free 25.2 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1505_piece0 in memory on 10.13.66.62:42183 (size: 73.7 KiB, free: 25.6 GiB)
24/07/19 15:25:25 INFO SparkContext: Created broadcast 1505 from broadcast at TaskSetManager.scala:711
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1505_piece0 in memory on 10.13.67.242:34949 (size: 73.7 KiB, free: 11.0 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1505_piece0 in memory on 10.13.64.47:39169 (size: 73.7 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1505_piece0 in memory on 10.13.70.254:42683 (size: 73.7 KiB, free: 11.0 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1505_piece0 in memory on 10.13.67.111:40323 (size: 73.7 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1505_piece0 in memory on 10.13.64.125:44919 (size: 73.7 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1505_piece0 in memory on 10.13.69.102:44669 (size: 73.7 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1505_piece0 in memory on 10.13.69.209:36605 (size: 73.7 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1505_piece0 in memory on 10.13.70.66:39213 (size: 73.7 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1505_piece0 in memory on 10.13.71.160:39905 (size: 73.7 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1505_piece0 in memory on 10.13.67.67:37729 (size: 73.7 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1505_piece0 in memory on 10.13.68.167:44723 (size: 73.7 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1505_piece0 in memory on 10.13.71.124:39087 (size: 73.7 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1505_piece0 in memory on 10.13.69.60:37447 (size: 73.7 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1505_piece0 in memory on 10.13.71.86:34131 (size: 73.7 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1505_piece0 in memory on 10.13.66.126:40899 (size: 73.7 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1505_piece0 in memory on 10.13.70.215:44777 (size: 73.7 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1505_piece0 in memory on 10.13.64.49:34851 (size: 73.7 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1505_piece0 in memory on 10.13.66.167:40621 (size: 73.7 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1505_piece0 in memory on 10.13.69.166:37577 (size: 73.7 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1505_piece0 in memory on 10.13.70.189:46789 (size: 73.7 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1505_piece0 in memory on 10.13.71.34:35213 (size: 73.7 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 252 to 10.13.67.242:37200
24/07/19 15:25:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 252 to 10.13.70.254:41952
24/07/19 15:25:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 252 to 10.13.64.47:53370
24/07/19 15:25:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 252 to 10.13.69.209:55396
24/07/19 15:25:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 252 to 10.13.71.160:40752
24/07/19 15:25:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 252 to 10.13.66.167:38178
24/07/19 15:25:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 252 to 10.13.70.66:42354
24/07/19 15:25:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 252 to 10.13.71.34:53106
24/07/19 15:25:25 INFO ClusterLoadAvgHelper: Current cluster load: 1, Old Ema: 1.0, New Ema: 1.0 
24/07/19 15:25:25 INFO TaskSetManager: Finished task 74.0 in stage 1125.0 (TID 28005) in 94 ms on 10.13.70.254 (executor 84) (1/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 32.0 in stage 1125.0 (TID 27963) in 98 ms on 10.13.70.254 (executor 84) (2/84)
24/07/19 15:25:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 252 to 10.13.68.167:39234
24/07/19 15:25:25 INFO TaskSetManager: Finished task 1.0 in stage 1125.0 (TID 27932) in 99 ms on 10.13.67.242 (executor 100) (3/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 22.0 in stage 1125.0 (TID 27953) in 99 ms on 10.13.67.242 (executor 100) (4/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 64.0 in stage 1125.0 (TID 27995) in 98 ms on 10.13.67.242 (executor 100) (5/84)
24/07/19 15:25:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 252 to 10.13.67.111:47552
24/07/19 15:25:25 INFO TaskSetManager: Finished task 43.0 in stage 1125.0 (TID 27974) in 100 ms on 10.13.67.242 (executor 100) (6/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 53.0 in stage 1125.0 (TID 27984) in 100 ms on 10.13.70.254 (executor 84) (7/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 11.0 in stage 1125.0 (TID 27942) in 100 ms on 10.13.70.254 (executor 84) (8/84)
24/07/19 15:25:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 252 to 10.13.66.126:55012
24/07/19 15:25:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 252 to 10.13.64.125:36268
24/07/19 15:25:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 252 to 10.13.70.215:41546
24/07/19 15:25:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 252 to 10.13.69.166:44434
24/07/19 15:25:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 252 to 10.13.69.102:47188
24/07/19 15:25:25 INFO TaskSetManager: Finished task 44.0 in stage 1125.0 (TID 27975) in 118 ms on 10.13.64.47 (executor 101) (9/84)
24/07/19 15:25:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 252 to 10.13.69.60:53406
24/07/19 15:25:25 INFO TaskSetManager: Finished task 2.0 in stage 1125.0 (TID 27933) in 119 ms on 10.13.64.47 (executor 101) (10/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 23.0 in stage 1125.0 (TID 27954) in 126 ms on 10.13.64.47 (executor 101) (11/84)
24/07/19 15:25:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 252 to 10.13.71.124:33330
24/07/19 15:25:25 INFO TaskSetManager: Finished task 65.0 in stage 1125.0 (TID 27996) in 126 ms on 10.13.64.47 (executor 101) (12/84)
24/07/19 15:25:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 252 to 10.13.70.189:46350
24/07/19 15:25:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 252 to 10.13.67.67:42254
24/07/19 15:25:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 252 to 10.13.64.49:48748
24/07/19 15:25:25 INFO TaskSetManager: Finished task 52.0 in stage 1125.0 (TID 27983) in 159 ms on 10.13.66.167 (executor 111) (13/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 10.0 in stage 1125.0 (TID 27941) in 163 ms on 10.13.66.167 (executor 111) (14/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 73.0 in stage 1125.0 (TID 28004) in 162 ms on 10.13.66.167 (executor 111) (15/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 31.0 in stage 1125.0 (TID 27962) in 165 ms on 10.13.66.167 (executor 111) (16/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 54.0 in stage 1125.0 (TID 27985) in 169 ms on 10.13.71.34 (executor 132) (17/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 38.0 in stage 1125.0 (TID 27969) in 171 ms on 10.13.69.209 (executor 117) (18/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 33.0 in stage 1125.0 (TID 27964) in 171 ms on 10.13.71.34 (executor 132) (19/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 59.0 in stage 1125.0 (TID 27990) in 172 ms on 10.13.69.209 (executor 117) (20/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 12.0 in stage 1125.0 (TID 27943) in 172 ms on 10.13.71.34 (executor 132) (21/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 75.0 in stage 1125.0 (TID 28006) in 171 ms on 10.13.71.34 (executor 132) (22/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 70.0 in stage 1125.0 (TID 28001) in 172 ms on 10.13.71.160 (executor 127) (23/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 80.0 in stage 1125.0 (TID 28011) in 173 ms on 10.13.69.209 (executor 117) (24/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 17.0 in stage 1125.0 (TID 27948) in 175 ms on 10.13.69.209 (executor 117) (25/84)
24/07/19 15:25:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 252 to 10.13.71.86:43090
24/07/19 15:25:25 INFO TaskSetManager: Finished task 49.0 in stage 1125.0 (TID 27980) in 185 ms on 10.13.71.160 (executor 127) (26/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 68.0 in stage 1125.0 (TID 27999) in 188 ms on 10.13.70.66 (executor 119) (27/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 26.0 in stage 1125.0 (TID 27957) in 189 ms on 10.13.70.66 (executor 119) (28/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 47.0 in stage 1125.0 (TID 27978) in 190 ms on 10.13.70.66 (executor 119) (29/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 5.0 in stage 1125.0 (TID 27936) in 190 ms on 10.13.70.66 (executor 119) (30/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 7.0 in stage 1125.0 (TID 27938) in 192 ms on 10.13.71.160 (executor 127) (31/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 28.0 in stage 1125.0 (TID 27959) in 192 ms on 10.13.71.160 (executor 127) (32/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 39.0 in stage 1125.0 (TID 27970) in 230 ms on 10.13.68.167 (executor 133) (33/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 18.0 in stage 1125.0 (TID 27949) in 234 ms on 10.13.68.167 (executor 133) (34/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 81.0 in stage 1125.0 (TID 28012) in 239 ms on 10.13.68.167 (executor 133) (35/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 60.0 in stage 1125.0 (TID 27991) in 243 ms on 10.13.68.167 (executor 133) (36/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 36.0 in stage 1125.0 (TID 27967) in 244 ms on 10.13.70.215 (executor 126) (37/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 15.0 in stage 1125.0 (TID 27946) in 244 ms on 10.13.70.215 (executor 126) (38/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 57.0 in stage 1125.0 (TID 27988) in 244 ms on 10.13.70.215 (executor 126) (39/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 50.0 in stage 1125.0 (TID 27981) in 245 ms on 10.13.64.125 (executor 120) (40/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 8.0 in stage 1125.0 (TID 27939) in 245 ms on 10.13.64.125 (executor 120) (41/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 40.0 in stage 1125.0 (TID 27971) in 247 ms on 10.13.69.166 (executor 121) (42/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 46.0 in stage 1125.0 (TID 27977) in 248 ms on 10.13.66.126 (executor 125) (43/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 82.0 in stage 1125.0 (TID 28013) in 247 ms on 10.13.69.166 (executor 121) (44/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 29.0 in stage 1125.0 (TID 27960) in 248 ms on 10.13.64.125 (executor 120) (45/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 19.0 in stage 1125.0 (TID 27950) in 248 ms on 10.13.69.166 (executor 121) (46/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 61.0 in stage 1125.0 (TID 27992) in 251 ms on 10.13.69.166 (executor 121) (47/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 0.0 in stage 1125.0 (TID 27931) in 254 ms on 10.13.69.102 (executor 109) (48/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 71.0 in stage 1125.0 (TID 28002) in 253 ms on 10.13.64.125 (executor 120) (49/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 21.0 in stage 1125.0 (TID 27952) in 255 ms on 10.13.69.102 (executor 109) (50/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 42.0 in stage 1125.0 (TID 27973) in 255 ms on 10.13.69.102 (executor 109) (51/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 63.0 in stage 1125.0 (TID 27994) in 256 ms on 10.13.69.102 (executor 109) (52/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 79.0 in stage 1125.0 (TID 28010) in 256 ms on 10.13.67.111 (executor 106) (53/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 37.0 in stage 1125.0 (TID 27968) in 258 ms on 10.13.67.111 (executor 106) (54/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 78.0 in stage 1125.0 (TID 28009) in 257 ms on 10.13.70.215 (executor 126) (55/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 16.0 in stage 1125.0 (TID 27947) in 258 ms on 10.13.67.111 (executor 106) (56/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 25.0 in stage 1125.0 (TID 27956) in 259 ms on 10.13.66.126 (executor 125) (57/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 4.0 in stage 1125.0 (TID 27935) in 261 ms on 10.13.66.126 (executor 125) (58/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 67.0 in stage 1125.0 (TID 27998) in 261 ms on 10.13.66.126 (executor 125) (59/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 77.0 in stage 1125.0 (TID 28008) in 261 ms on 10.13.69.60 (executor 107) (60/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 58.0 in stage 1125.0 (TID 27989) in 262 ms on 10.13.67.111 (executor 106) (61/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 35.0 in stage 1125.0 (TID 27966) in 266 ms on 10.13.69.60 (executor 107) (62/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 56.0 in stage 1125.0 (TID 27987) in 277 ms on 10.13.69.60 (executor 107) (63/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 14.0 in stage 1125.0 (TID 27945) in 283 ms on 10.13.69.60 (executor 107) (64/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 9.0 in stage 1125.0 (TID 27940) in 292 ms on 10.13.70.189 (executor 128) (65/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 51.0 in stage 1125.0 (TID 27982) in 293 ms on 10.13.70.189 (executor 128) (66/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 30.0 in stage 1125.0 (TID 27961) in 293 ms on 10.13.70.189 (executor 128) (67/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 27.0 in stage 1125.0 (TID 27958) in 295 ms on 10.13.71.124 (executor 103) (68/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 72.0 in stage 1125.0 (TID 28003) in 296 ms on 10.13.70.189 (executor 128) (69/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 48.0 in stage 1125.0 (TID 27979) in 298 ms on 10.13.71.124 (executor 103) (70/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 69.0 in stage 1125.0 (TID 28000) in 298 ms on 10.13.71.124 (executor 103) (71/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 6.0 in stage 1125.0 (TID 27937) in 300 ms on 10.13.71.124 (executor 103) (72/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 62.0 in stage 1125.0 (TID 27993) in 307 ms on 10.13.67.67 (executor 108) (73/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 20.0 in stage 1125.0 (TID 27951) in 307 ms on 10.13.67.67 (executor 108) (74/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 41.0 in stage 1125.0 (TID 27972) in 308 ms on 10.13.67.67 (executor 108) (75/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 45.0 in stage 1125.0 (TID 27976) in 326 ms on 10.13.64.49 (executor 116) (76/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 24.0 in stage 1125.0 (TID 27955) in 336 ms on 10.13.64.49 (executor 116) (77/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 66.0 in stage 1125.0 (TID 27997) in 336 ms on 10.13.64.49 (executor 116) (78/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 3.0 in stage 1125.0 (TID 27934) in 337 ms on 10.13.64.49 (executor 116) (79/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 83.0 in stage 1125.0 (TID 28014) in 365 ms on 10.13.67.67 (executor 108) (80/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 13.0 in stage 1125.0 (TID 27944) in 388 ms on 10.13.71.86 (executor 130) (81/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 55.0 in stage 1125.0 (TID 27986) in 389 ms on 10.13.71.86 (executor 130) (82/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 76.0 in stage 1125.0 (TID 28007) in 388 ms on 10.13.71.86 (executor 130) (83/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 34.0 in stage 1125.0 (TID 27965) in 391 ms on 10.13.71.86 (executor 130) (84/84)
24/07/19 15:25:25 INFO TaskSchedulerImpl: Removed TaskSet 1125.0, whose tasks have all completed, from pool 4085144645627145219
24/07/19 15:25:25 INFO DAGScheduler: ResultStage 1125 (wrapper at /root/.ipykernel/18206/command-230092523437668-2624768249:13) finished in 0.394 s
24/07/19 15:25:25 INFO DAGScheduler: Job 499 is finished. Cancelling potential speculative or zombie tasks for this job
24/07/19 15:25:25 INFO TaskSchedulerImpl: Killing all running tasks in stage 1125: Stage finished
24/07/19 15:25:25 INFO DAGScheduler: Job 499 finished: wrapper at /root/.ipykernel/18206/command-230092523437668-2624768249:13, took 0.396146 s
24/07/19 15:25:25 INFO ClusterLoadMonitor: Removed query with execution ID:227. Current active queries:0
24/07/19 15:25:25 INFO SparkEBJCleanupListener: Cleaning up ebj join relation block with ID spark_join_relation_252_370064497)
24/07/19 15:25:25 WARN BlockManager: Asked to remove block spark_join_relation_252_370064497, which does not exist
24/07/19 15:25:25 INFO QueryProfileListener: Query profile sent to logger, seq number: 227, app id: app-20240719133525-0000
24/07/19 15:25:25 INFO ClusterLoadMonitor: Added query with execution ID:228. Current active queries:1
24/07/19 15:25:25 INFO SparkContext: Starting job: wrapper at /root/.ipykernel/18206/command-230092523437668-2624768249:14
24/07/19 15:25:25 INFO DAGScheduler: Got job 500 (wrapper at /root/.ipykernel/18206/command-230092523437668-2624768249:14) with 84 output partitions
24/07/19 15:25:25 INFO DAGScheduler: Final stage: ResultStage 1127 (wrapper at /root/.ipykernel/18206/command-230092523437668-2624768249:14)
24/07/19 15:25:25 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 1126)
24/07/19 15:25:25 INFO DAGScheduler: Missing parents: List()
24/07/19 15:25:25 INFO DAGScheduler: Submitting ResultStage 1127 (MapPartitionsRDD[5100] at wrapper at /root/.ipykernel/18206/command-230092523437668-2624768249:14), which has no missing parents
24/07/19 15:25:25 INFO DAGScheduler: Submitting 84 missing tasks from ResultStage 1127 (MapPartitionsRDD[5100] at wrapper at /root/.ipykernel/18206/command-230092523437668-2624768249:14) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))
24/07/19 15:25:25 INFO TaskSchedulerImpl: Adding task set 1127.0 with 84 tasks resource profile 0
24/07/19 15:25:25 INFO TaskSetManager: TaskSet 1127.0 using PreferredLocationsV1
24/07/19 15:25:25 INFO FairSchedulableBuilder: Added task set TaskSet_1127.0 tasks to pool 4085144645627145219
24/07/19 15:25:25 INFO TaskSetManager: Starting task 0.0 in stage 1127.0 (TID 28015) (10.13.71.86, executor 130, partition 0, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 1.0 in stage 1127.0 (TID 28016) (10.13.71.124, executor 103, partition 1, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 2.0 in stage 1127.0 (TID 28017) (10.13.70.215, executor 126, partition 2, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 3.0 in stage 1127.0 (TID 28018) (10.13.70.189, executor 128, partition 3, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 4.0 in stage 1127.0 (TID 28019) (10.13.69.60, executor 107, partition 4, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 5.0 in stage 1127.0 (TID 28020) (10.13.71.34, executor 132, partition 5, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 6.0 in stage 1127.0 (TID 28021) (10.13.64.47, executor 101, partition 6, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 7.0 in stage 1127.0 (TID 28022) (10.13.69.102, executor 109, partition 7, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 8.0 in stage 1127.0 (TID 28023) (10.13.64.125, executor 120, partition 8, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 9.0 in stage 1127.0 (TID 28024) (10.13.68.167, executor 133, partition 9, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 10.0 in stage 1127.0 (TID 28025) (10.13.67.242, executor 100, partition 10, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 11.0 in stage 1127.0 (TID 28026) (10.13.69.166, executor 121, partition 11, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 12.0 in stage 1127.0 (TID 28027) (10.13.70.66, executor 119, partition 12, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 13.0 in stage 1127.0 (TID 28028) (10.13.71.160, executor 127, partition 13, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 14.0 in stage 1127.0 (TID 28029) (10.13.66.167, executor 111, partition 14, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 15.0 in stage 1127.0 (TID 28030) (10.13.67.67, executor 108, partition 15, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 16.0 in stage 1127.0 (TID 28031) (10.13.70.254, executor 84, partition 16, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 17.0 in stage 1127.0 (TID 28032) (10.13.67.111, executor 106, partition 17, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 18.0 in stage 1127.0 (TID 28033) (10.13.69.209, executor 117, partition 18, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 19.0 in stage 1127.0 (TID 28034) (10.13.66.126, executor 125, partition 19, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 20.0 in stage 1127.0 (TID 28035) (10.13.64.49, executor 116, partition 20, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 21.0 in stage 1127.0 (TID 28036) (10.13.71.86, executor 130, partition 21, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 22.0 in stage 1127.0 (TID 28037) (10.13.71.124, executor 103, partition 22, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 23.0 in stage 1127.0 (TID 28038) (10.13.70.215, executor 126, partition 23, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 24.0 in stage 1127.0 (TID 28039) (10.13.70.189, executor 128, partition 24, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 25.0 in stage 1127.0 (TID 28040) (10.13.69.60, executor 107, partition 25, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 26.0 in stage 1127.0 (TID 28041) (10.13.71.34, executor 132, partition 26, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 27.0 in stage 1127.0 (TID 28042) (10.13.64.47, executor 101, partition 27, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 28.0 in stage 1127.0 (TID 28043) (10.13.69.102, executor 109, partition 28, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 29.0 in stage 1127.0 (TID 28044) (10.13.64.125, executor 120, partition 29, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 30.0 in stage 1127.0 (TID 28045) (10.13.68.167, executor 133, partition 30, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 31.0 in stage 1127.0 (TID 28046) (10.13.67.242, executor 100, partition 31, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 32.0 in stage 1127.0 (TID 28047) (10.13.69.166, executor 121, partition 32, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 33.0 in stage 1127.0 (TID 28048) (10.13.70.66, executor 119, partition 33, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 34.0 in stage 1127.0 (TID 28049) (10.13.71.160, executor 127, partition 34, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 35.0 in stage 1127.0 (TID 28050) (10.13.66.167, executor 111, partition 35, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 36.0 in stage 1127.0 (TID 28051) (10.13.67.67, executor 108, partition 36, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 37.0 in stage 1127.0 (TID 28052) (10.13.70.254, executor 84, partition 37, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 38.0 in stage 1127.0 (TID 28053) (10.13.67.111, executor 106, partition 38, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 39.0 in stage 1127.0 (TID 28054) (10.13.69.209, executor 117, partition 39, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 40.0 in stage 1127.0 (TID 28055) (10.13.66.126, executor 125, partition 40, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 41.0 in stage 1127.0 (TID 28056) (10.13.64.49, executor 116, partition 41, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 42.0 in stage 1127.0 (TID 28057) (10.13.71.86, executor 130, partition 42, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 43.0 in stage 1127.0 (TID 28058) (10.13.71.124, executor 103, partition 43, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 44.0 in stage 1127.0 (TID 28059) (10.13.70.215, executor 126, partition 44, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 45.0 in stage 1127.0 (TID 28060) (10.13.70.189, executor 128, partition 45, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 46.0 in stage 1127.0 (TID 28061) (10.13.69.60, executor 107, partition 46, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 47.0 in stage 1127.0 (TID 28062) (10.13.71.34, executor 132, partition 47, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 48.0 in stage 1127.0 (TID 28063) (10.13.64.47, executor 101, partition 48, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 49.0 in stage 1127.0 (TID 28064) (10.13.69.102, executor 109, partition 49, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 50.0 in stage 1127.0 (TID 28065) (10.13.64.125, executor 120, partition 50, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 51.0 in stage 1127.0 (TID 28066) (10.13.68.167, executor 133, partition 51, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 52.0 in stage 1127.0 (TID 28067) (10.13.67.242, executor 100, partition 52, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 53.0 in stage 1127.0 (TID 28068) (10.13.69.166, executor 121, partition 53, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 54.0 in stage 1127.0 (TID 28069) (10.13.70.66, executor 119, partition 54, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 55.0 in stage 1127.0 (TID 28070) (10.13.71.160, executor 127, partition 55, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 56.0 in stage 1127.0 (TID 28071) (10.13.66.167, executor 111, partition 56, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 57.0 in stage 1127.0 (TID 28072) (10.13.67.67, executor 108, partition 57, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 58.0 in stage 1127.0 (TID 28073) (10.13.70.254, executor 84, partition 58, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 59.0 in stage 1127.0 (TID 28074) (10.13.67.111, executor 106, partition 59, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 60.0 in stage 1127.0 (TID 28075) (10.13.69.209, executor 117, partition 60, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 61.0 in stage 1127.0 (TID 28076) (10.13.66.126, executor 125, partition 61, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 62.0 in stage 1127.0 (TID 28077) (10.13.64.49, executor 116, partition 62, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 63.0 in stage 1127.0 (TID 28078) (10.13.71.86, executor 130, partition 63, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 64.0 in stage 1127.0 (TID 28079) (10.13.71.124, executor 103, partition 64, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 65.0 in stage 1127.0 (TID 28080) (10.13.70.215, executor 126, partition 65, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 66.0 in stage 1127.0 (TID 28081) (10.13.70.189, executor 128, partition 66, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 67.0 in stage 1127.0 (TID 28082) (10.13.69.60, executor 107, partition 67, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 68.0 in stage 1127.0 (TID 28083) (10.13.71.34, executor 132, partition 68, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 69.0 in stage 1127.0 (TID 28084) (10.13.64.47, executor 101, partition 69, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 70.0 in stage 1127.0 (TID 28085) (10.13.69.102, executor 109, partition 70, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 71.0 in stage 1127.0 (TID 28086) (10.13.64.125, executor 120, partition 71, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 72.0 in stage 1127.0 (TID 28087) (10.13.68.167, executor 133, partition 72, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 73.0 in stage 1127.0 (TID 28088) (10.13.67.242, executor 100, partition 73, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 74.0 in stage 1127.0 (TID 28089) (10.13.69.166, executor 121, partition 74, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 75.0 in stage 1127.0 (TID 28090) (10.13.70.66, executor 119, partition 75, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 76.0 in stage 1127.0 (TID 28091) (10.13.71.160, executor 127, partition 76, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 77.0 in stage 1127.0 (TID 28092) (10.13.66.167, executor 111, partition 77, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 78.0 in stage 1127.0 (TID 28093) (10.13.67.67, executor 108, partition 78, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 79.0 in stage 1127.0 (TID 28094) (10.13.70.254, executor 84, partition 79, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 80.0 in stage 1127.0 (TID 28095) (10.13.67.111, executor 106, partition 80, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 81.0 in stage 1127.0 (TID 28096) (10.13.69.209, executor 117, partition 81, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 82.0 in stage 1127.0 (TID 28097) (10.13.66.126, executor 125, partition 82, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO TaskSetManager: Starting task 83.0 in stage 1127.0 (TID 28098) (10.13.64.49, executor 116, partition 83, PROCESS_LOCAL, 
24/07/19 15:25:25 INFO MemoryStore: Block broadcast_1506 stored as values in memory (estimated size 223.6 KiB, free 25.2 GiB)
24/07/19 15:25:25 INFO MemoryStore: Block broadcast_1506_piece0 stored as bytes in memory (estimated size 74.2 KiB, free 25.2 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1506_piece0 in memory on 10.13.66.62:42183 (size: 74.2 KiB, free: 25.6 GiB)
24/07/19 15:25:25 INFO SparkContext: Created broadcast 1506 from broadcast at TaskSetManager.scala:711
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1506_piece0 in memory on 10.13.67.242:34949 (size: 74.2 KiB, free: 11.0 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1506_piece0 in memory on 10.13.64.47:39169 (size: 74.2 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1506_piece0 in memory on 10.13.69.60:37447 (size: 74.2 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1506_piece0 in memory on 10.13.67.67:37729 (size: 74.2 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1506_piece0 in memory on 10.13.71.34:35213 (size: 74.2 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1506_piece0 in memory on 10.13.70.254:42683 (size: 74.2 KiB, free: 11.0 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1506_piece0 in memory on 10.13.70.66:39213 (size: 74.2 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1506_piece0 in memory on 10.13.71.160:39905 (size: 74.2 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1506_piece0 in memory on 10.13.67.111:40323 (size: 74.2 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1506_piece0 in memory on 10.13.64.125:44919 (size: 74.2 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1506_piece0 in memory on 10.13.66.167:40621 (size: 74.2 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1506_piece0 in memory on 10.13.66.126:40899 (size: 74.2 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1506_piece0 in memory on 10.13.69.209:36605 (size: 74.2 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1506_piece0 in memory on 10.13.71.86:34131 (size: 74.2 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1506_piece0 in memory on 10.13.69.102:44669 (size: 74.2 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1506_piece0 in memory on 10.13.64.49:34851 (size: 74.2 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1506_piece0 in memory on 10.13.71.124:39087 (size: 74.2 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1506_piece0 in memory on 10.13.68.167:44723 (size: 74.2 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1506_piece0 in memory on 10.13.70.189:46789 (size: 74.2 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1506_piece0 in memory on 10.13.70.215:44777 (size: 74.2 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO BlockManagerInfo: Added broadcast_1506_piece0 in memory on 10.13.69.166:37577 (size: 74.2 KiB, free: 11.1 GiB)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 10.0 in stage 1127.0 (TID 28025) in 67 ms on 10.13.67.242 (executor 100) (1/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 52.0 in stage 1127.0 (TID 28067) in 67 ms on 10.13.67.242 (executor 100) (2/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 58.0 in stage 1127.0 (TID 28073) in 70 ms on 10.13.70.254 (executor 84) (3/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 31.0 in stage 1127.0 (TID 28046) in 72 ms on 10.13.67.242 (executor 100) (4/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 79.0 in stage 1127.0 (TID 28094) in 76 ms on 10.13.70.254 (executor 84) (5/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 29.0 in stage 1127.0 (TID 28044) in 77 ms on 10.13.64.125 (executor 120) (6/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 37.0 in stage 1127.0 (TID 28052) in 77 ms on 10.13.70.254 (executor 84) (7/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 16.0 in stage 1127.0 (TID 28031) in 78 ms on 10.13.70.254 (executor 84) (8/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 43.0 in stage 1127.0 (TID 28058) in 78 ms on 10.13.71.124 (executor 103) (9/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 48.0 in stage 1127.0 (TID 28063) in 79 ms on 10.13.64.47 (executor 101) (10/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 6.0 in stage 1127.0 (TID 28021) in 80 ms on 10.13.64.47 (executor 101) (11/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 64.0 in stage 1127.0 (TID 28079) in 80 ms on 10.13.71.124 (executor 103) (12/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 71.0 in stage 1127.0 (TID 28086) in 80 ms on 10.13.64.125 (executor 120) (13/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 73.0 in stage 1127.0 (TID 28088) in 82 ms on 10.13.67.242 (executor 100) (14/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 76.0 in stage 1127.0 (TID 28091) in 83 ms on 10.13.71.160 (executor 127) (15/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 27.0 in stage 1127.0 (TID 28042) in 83 ms on 10.13.64.47 (executor 101) (16/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 26.0 in stage 1127.0 (TID 28041) in 83 ms on 10.13.71.34 (executor 132) (17/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 77.0 in stage 1127.0 (TID 28092) in 84 ms on 10.13.66.167 (executor 111) (18/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 30.0 in stage 1127.0 (TID 28045) in 86 ms on 10.13.68.167 (executor 133) (19/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 72.0 in stage 1127.0 (TID 28087) in 86 ms on 10.13.68.167 (executor 133) (20/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 69.0 in stage 1127.0 (TID 28084) in 87 ms on 10.13.64.47 (executor 101) (21/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 53.0 in stage 1127.0 (TID 28068) in 88 ms on 10.13.69.166 (executor 121) (22/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 32.0 in stage 1127.0 (TID 28047) in 88 ms on 10.13.69.166 (executor 121) (23/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 1.0 in stage 1127.0 (TID 28016) in 89 ms on 10.13.71.124 (executor 103) (24/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 21.0 in stage 1127.0 (TID 28036) in 89 ms on 10.13.71.86 (executor 130) (25/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 56.0 in stage 1127.0 (TID 28071) in 89 ms on 10.13.66.167 (executor 111) (26/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 66.0 in stage 1127.0 (TID 28081) in 90 ms on 10.13.70.189 (executor 128) (27/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 3.0 in stage 1127.0 (TID 28018) in 91 ms on 10.13.70.189 (executor 128) (28/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 38.0 in stage 1127.0 (TID 28053) in 90 ms on 10.13.67.111 (executor 106) (29/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 70.0 in stage 1127.0 (TID 28085) in 90 ms on 10.13.69.102 (executor 109) (30/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 59.0 in stage 1127.0 (TID 28074) in 90 ms on 10.13.67.111 (executor 106) (31/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 49.0 in stage 1127.0 (TID 28064) in 90 ms on 10.13.69.102 (executor 109) (32/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 17.0 in stage 1127.0 (TID 28032) in 91 ms on 10.13.67.111 (executor 106) (33/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 9.0 in stage 1127.0 (TID 28024) in 92 ms on 10.13.68.167 (executor 133) (34/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 18.0 in stage 1127.0 (TID 28033) in 91 ms on 10.13.69.209 (executor 117) (35/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 47.0 in stage 1127.0 (TID 28062) in 91 ms on 10.13.71.34 (executor 132) (36/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 20.0 in stage 1127.0 (TID 28035) in 92 ms on 10.13.64.49 (executor 116) (37/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 8.0 in stage 1127.0 (TID 28023) in 93 ms on 10.13.64.125 (executor 120) (38/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 35.0 in stage 1127.0 (TID 28050) in 92 ms on 10.13.66.167 (executor 111) (39/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 7.0 in stage 1127.0 (TID 28022) in 95 ms on 10.13.69.102 (executor 109) (40/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 50.0 in stage 1127.0 (TID 28065) in 94 ms on 10.13.64.125 (executor 120) (41/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 80.0 in stage 1127.0 (TID 28095) in 95 ms on 10.13.67.111 (executor 106) (42/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 25.0 in stage 1127.0 (TID 28040) in 95 ms on 10.13.69.60 (executor 107) (43/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 67.0 in stage 1127.0 (TID 28082) in 95 ms on 10.13.69.60 (executor 107) (44/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 22.0 in stage 1127.0 (TID 28037) in 95 ms on 10.13.71.124 (executor 103) (45/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 46.0 in stage 1127.0 (TID 28061) in 96 ms on 10.13.69.60 (executor 107) (46/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 82.0 in stage 1127.0 (TID 28097) in 96 ms on 10.13.66.126 (executor 125) (47/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 14.0 in stage 1127.0 (TID 28029) in 96 ms on 10.13.66.167 (executor 111) (48/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 42.0 in stage 1127.0 (TID 28057) in 97 ms on 10.13.71.86 (executor 130) (49/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 51.0 in stage 1127.0 (TID 28066) in 97 ms on 10.13.68.167 (executor 133) (50/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 4.0 in stage 1127.0 (TID 28019) in 98 ms on 10.13.69.60 (executor 107) (51/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 62.0 in stage 1127.0 (TID 28077) in 97 ms on 10.13.64.49 (executor 116) (52/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 13.0 in stage 1127.0 (TID 28028) in 98 ms on 10.13.71.160 (executor 127) (53/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 55.0 in stage 1127.0 (TID 28070) in 98 ms on 10.13.71.160 (executor 127) (54/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 33.0 in stage 1127.0 (TID 28048) in 98 ms on 10.13.70.66 (executor 119) (55/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 81.0 in stage 1127.0 (TID 28096) in 99 ms on 10.13.69.209 (executor 117) (56/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 34.0 in stage 1127.0 (TID 28049) in 99 ms on 10.13.71.160 (executor 127) (57/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 12.0 in stage 1127.0 (TID 28027) in 100 ms on 10.13.70.66 (executor 119) (58/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 28.0 in stage 1127.0 (TID 28043) in 100 ms on 10.13.69.102 (executor 109) (59/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 11.0 in stage 1127.0 (TID 28026) in 101 ms on 10.13.69.166 (executor 121) (60/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 75.0 in stage 1127.0 (TID 28090) in 101 ms on 10.13.70.66 (executor 119) (61/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 54.0 in stage 1127.0 (TID 28069) in 101 ms on 10.13.70.66 (executor 119) (62/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 39.0 in stage 1127.0 (TID 28054) in 102 ms on 10.13.69.209 (executor 117) (63/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 60.0 in stage 1127.0 (TID 28075) in 102 ms on 10.13.69.209 (executor 117) (64/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 74.0 in stage 1127.0 (TID 28089) in 102 ms on 10.13.69.166 (executor 121) (65/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 24.0 in stage 1127.0 (TID 28039) in 102 ms on 10.13.70.189 (executor 128) (66/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 78.0 in stage 1127.0 (TID 28093) in 103 ms on 10.13.67.67 (executor 108) (67/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 57.0 in stage 1127.0 (TID 28072) in 103 ms on 10.13.67.67 (executor 108) (68/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 5.0 in stage 1127.0 (TID 28020) in 106 ms on 10.13.71.34 (executor 132) (69/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 63.0 in stage 1127.0 (TID 28078) in 105 ms on 10.13.71.86 (executor 130) (70/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 0.0 in stage 1127.0 (TID 28015) in 107 ms on 10.13.71.86 (executor 130) (71/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 36.0 in stage 1127.0 (TID 28051) in 106 ms on 10.13.67.67 (executor 108) (72/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 61.0 in stage 1127.0 (TID 28076) in 107 ms on 10.13.66.126 (executor 125) (73/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 15.0 in stage 1127.0 (TID 28030) in 110 ms on 10.13.67.67 (executor 108) (74/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 68.0 in stage 1127.0 (TID 28083) in 111 ms on 10.13.71.34 (executor 132) (75/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 40.0 in stage 1127.0 (TID 28055) in 115 ms on 10.13.66.126 (executor 125) (76/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 45.0 in stage 1127.0 (TID 28060) in 120 ms on 10.13.70.189 (executor 128) (77/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 41.0 in stage 1127.0 (TID 28056) in 127 ms on 10.13.64.49 (executor 116) (78/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 19.0 in stage 1127.0 (TID 28034) in 128 ms on 10.13.66.126 (executor 125) (79/84)
24/07/19 15:25:25 INFO TaskSetManager: Finished task 83.0 in stage 1127.0 (TID 28098) in 153 ms on 10.13.64.49 (executor 116) (80/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 65.0 in stage 1127.0 (TID 28080) in 259 ms on 10.13.70.215 (executor 126) (81/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 23.0 in stage 1127.0 (TID 28038) in 265 ms on 10.13.70.215 (executor 126) (82/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 2.0 in stage 1127.0 (TID 28017) in 266 ms on 10.13.70.215 (executor 126) (83/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 44.0 in stage 1127.0 (TID 28059) in 265 ms on 10.13.70.215 (executor 126) (84/84)
24/07/19 15:25:26 INFO TaskSchedulerImpl: Removed TaskSet 1127.0, whose tasks have all completed, from pool 4085144645627145219
24/07/19 15:25:26 INFO DAGScheduler: ResultStage 1127 (wrapper at /root/.ipykernel/18206/command-230092523437668-2624768249:14) finished in 0.268 s
24/07/19 15:25:26 INFO DAGScheduler: Job 500 is finished. Cancelling potential speculative or zombie tasks for this job
24/07/19 15:25:26 INFO TaskSchedulerImpl: Killing all running tasks in stage 1127: Stage finished
24/07/19 15:25:26 INFO DAGScheduler: Job 500 finished: wrapper at /root/.ipykernel/18206/command-230092523437668-2624768249:14, took 0.270714 s
24/07/19 15:25:26 INFO ClusterLoadMonitor: Removed query with execution ID:228. Current active queries:0
24/07/19 15:25:26 INFO SparkEBJCleanupListener: Cleaning up ebj join relation block with ID spark_join_relation_252_370064497)
24/07/19 15:25:26 WARN BlockManager: Asked to remove block spark_join_relation_252_370064497, which does not exist
24/07/19 15:25:26 INFO QueryProfileListener: Query profile sent to logger, seq number: 228, app id: app-20240719133525-0000
24/07/19 15:25:26 INFO ProgressReporter$: Removed result fetcher for 4085144645627145219_4958586356645017151_job-23196080691842-run-226089366195953-action-6604081895629205
24/07/19 15:25:26 INFO ProgressReporter$: Added result fetcher for 4085144645627145219_6406455263148317922_job-23196080691842-run-226089366195953-action-6604081895629205
24/07/19 15:25:26 INFO ClusterLoadMonitor: Added query with execution ID:229. Current active queries:1
24/07/19 15:25:26 WARN HintErrorLogger: Hint (strategy=broadcast) is not supported in the query: build left for left outer join.
24/07/19 15:25:26 INFO FileSourceStrategy: Pushed Filters: IsNotNull(duration),IsNotNull(views),EqualTo(duration,180_days),LessThan(views,250)
24/07/19 15:25:26 INFO FileSourceStrategy: Post-Scan Filters: isnotnull(duration#146463),isnotnull(views#146459L),(duration#146463 = 180_days),(views#146459L < 250)
24/07/19 15:25:26 INFO MemoryStore: Block broadcast_1507 stored as values in memory (estimated size 558.3 KiB, free 25.2 GiB)
24/07/19 15:25:26 INFO MemoryStore: Block broadcast_1507_piece0 stored as bytes in memory (estimated size 15.9 KiB, free 25.2 GiB)
24/07/19 15:25:26 INFO BlockManagerInfo: Added broadcast_1507_piece0 in memory on 10.13.66.62:42183 (size: 15.9 KiB, free: 25.6 GiB)
24/07/19 15:25:26 INFO SparkContext: Created broadcast 1507 from wrapper at /root/.ipykernel/18206/command-230092523437669-2836746030:13
24/07/19 15:25:26 INFO DAGScheduler: Registering RDD 5102 (wrapper at /root/.ipykernel/18206/command-230092523437669-2836746030:13) as input to shuffle 253
24/07/19 15:25:26 INFO DAGScheduler: Got map stage job 501 (wrapper at /root/.ipykernel/18206/command-230092523437669-2836746030:13) with 84 output partitions
24/07/19 15:25:26 INFO DAGScheduler: Final stage: ShuffleMapStage 1128 (wrapper at /root/.ipykernel/18206/command-230092523437669-2836746030:13)
24/07/19 15:25:26 INFO DAGScheduler: Parents of final stage: List()
24/07/19 15:25:26 INFO FileSourceScanExec: Planning scan with bin packing, max split size: 134217728 bytes, max partition size: 4194304, open cost is considered as scanning 4194304 bytes.
24/07/19 15:25:26 INFO DAGScheduler: Missing parents: List()
24/07/19 15:25:26 INFO DAGScheduler: Submitting ShuffleMapStage 1128 (MapPartitionsRDD[5102] at wrapper at /root/.ipykernel/18206/command-230092523437669-2836746030:13), which has no missing parents
24/07/19 15:25:26 INFO DAGScheduler: Submitting 84 missing tasks from ShuffleMapStage 1128 (MapPartitionsRDD[5102] at wrapper at /root/.ipykernel/18206/command-230092523437669-2836746030:13) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))
24/07/19 15:25:26 INFO TaskSchedulerImpl: Adding task set 1128.0 with 84 tasks resource profile 0
24/07/19 15:25:26 INFO TaskSetManager: TaskSet 1128.0 using PreferredLocationsV1
24/07/19 15:25:26 INFO FairSchedulableBuilder: Added task set TaskSet_1128.0 tasks to pool 4085144645627145219
24/07/19 15:25:26 INFO TaskSetManager: Starting task 0.0 in stage 1128.0 (TID 28099) (10.13.64.49, executor 116, partition 0, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 1.0 in stage 1128.0 (TID 28100) (10.13.71.34, executor 132, partition 1, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 2.0 in stage 1128.0 (TID 28101) (10.13.67.242, executor 100, partition 2, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 3.0 in stage 1128.0 (TID 28102) (10.13.67.111, executor 106, partition 3, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 4.0 in stage 1128.0 (TID 28103) (10.13.64.47, executor 101, partition 4, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 5.0 in stage 1128.0 (TID 28104) (10.13.69.209, executor 117, partition 5, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 6.0 in stage 1128.0 (TID 28105) (10.13.71.86, executor 130, partition 6, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 7.0 in stage 1128.0 (TID 28106) (10.13.66.167, executor 111, partition 7, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 8.0 in stage 1128.0 (TID 28107) (10.13.66.126, executor 125, partition 8, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 9.0 in stage 1128.0 (TID 28108) (10.13.70.215, executor 126, partition 9, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 10.0 in stage 1128.0 (TID 28109) (10.13.64.125, executor 120, partition 10, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 11.0 in stage 1128.0 (TID 28110) (10.13.68.167, executor 133, partition 11, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 12.0 in stage 1128.0 (TID 28111) (10.13.71.160, executor 127, partition 12, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 13.0 in stage 1128.0 (TID 28112) (10.13.70.66, executor 119, partition 13, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 14.0 in stage 1128.0 (TID 28113) (10.13.67.67, executor 108, partition 14, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 15.0 in stage 1128.0 (TID 28114) (10.13.69.102, executor 109, partition 15, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 16.0 in stage 1128.0 (TID 28115) (10.13.70.254, executor 84, partition 16, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 17.0 in stage 1128.0 (TID 28116) (10.13.70.189, executor 128, partition 17, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 18.0 in stage 1128.0 (TID 28117) (10.13.69.60, executor 107, partition 18, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 19.0 in stage 1128.0 (TID 28118) (10.13.69.166, executor 121, partition 19, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 20.0 in stage 1128.0 (TID 28119) (10.13.71.124, executor 103, partition 20, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 21.0 in stage 1128.0 (TID 28120) (10.13.64.49, executor 116, partition 21, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 22.0 in stage 1128.0 (TID 28121) (10.13.71.34, executor 132, partition 22, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 23.0 in stage 1128.0 (TID 28122) (10.13.67.242, executor 100, partition 23, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 24.0 in stage 1128.0 (TID 28123) (10.13.67.111, executor 106, partition 24, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 25.0 in stage 1128.0 (TID 28124) (10.13.64.47, executor 101, partition 25, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 26.0 in stage 1128.0 (TID 28125) (10.13.69.209, executor 117, partition 26, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 27.0 in stage 1128.0 (TID 28126) (10.13.71.86, executor 130, partition 27, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 28.0 in stage 1128.0 (TID 28127) (10.13.66.167, executor 111, partition 28, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 29.0 in stage 1128.0 (TID 28128) (10.13.66.126, executor 125, partition 29, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 30.0 in stage 1128.0 (TID 28129) (10.13.70.215, executor 126, partition 30, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 31.0 in stage 1128.0 (TID 28130) (10.13.64.125, executor 120, partition 31, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 32.0 in stage 1128.0 (TID 28131) (10.13.68.167, executor 133, partition 32, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 33.0 in stage 1128.0 (TID 28132) (10.13.71.160, executor 127, partition 33, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 34.0 in stage 1128.0 (TID 28133) (10.13.70.66, executor 119, partition 34, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 35.0 in stage 1128.0 (TID 28134) (10.13.67.67, executor 108, partition 35, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 36.0 in stage 1128.0 (TID 28135) (10.13.69.102, executor 109, partition 36, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 37.0 in stage 1128.0 (TID 28136) (10.13.70.254, executor 84, partition 37, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 38.0 in stage 1128.0 (TID 28137) (10.13.70.189, executor 128, partition 38, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 39.0 in stage 1128.0 (TID 28138) (10.13.69.60, executor 107, partition 39, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 40.0 in stage 1128.0 (TID 28139) (10.13.69.166, executor 121, partition 40, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 41.0 in stage 1128.0 (TID 28140) (10.13.71.124, executor 103, partition 41, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 42.0 in stage 1128.0 (TID 28141) (10.13.64.49, executor 116, partition 42, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 43.0 in stage 1128.0 (TID 28142) (10.13.71.34, executor 132, partition 43, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 44.0 in stage 1128.0 (TID 28143) (10.13.67.242, executor 100, partition 44, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 45.0 in stage 1128.0 (TID 28144) (10.13.67.111, executor 106, partition 45, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 46.0 in stage 1128.0 (TID 28145) (10.13.64.47, executor 101, partition 46, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 47.0 in stage 1128.0 (TID 28146) (10.13.69.209, executor 117, partition 47, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 48.0 in stage 1128.0 (TID 28147) (10.13.71.86, executor 130, partition 48, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 49.0 in stage 1128.0 (TID 28148) (10.13.66.167, executor 111, partition 49, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 50.0 in stage 1128.0 (TID 28149) (10.13.66.126, executor 125, partition 50, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 51.0 in stage 1128.0 (TID 28150) (10.13.70.215, executor 126, partition 51, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 52.0 in stage 1128.0 (TID 28151) (10.13.64.125, executor 120, partition 52, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 53.0 in stage 1128.0 (TID 28152) (10.13.68.167, executor 133, partition 53, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 54.0 in stage 1128.0 (TID 28153) (10.13.71.160, executor 127, partition 54, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 55.0 in stage 1128.0 (TID 28154) (10.13.70.66, executor 119, partition 55, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 56.0 in stage 1128.0 (TID 28155) (10.13.67.67, executor 108, partition 56, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 57.0 in stage 1128.0 (TID 28156) (10.13.69.102, executor 109, partition 57, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 58.0 in stage 1128.0 (TID 28157) (10.13.70.254, executor 84, partition 58, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 59.0 in stage 1128.0 (TID 28158) (10.13.70.189, executor 128, partition 59, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 60.0 in stage 1128.0 (TID 28159) (10.13.69.60, executor 107, partition 60, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 61.0 in stage 1128.0 (TID 28160) (10.13.69.166, executor 121, partition 61, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 62.0 in stage 1128.0 (TID 28161) (10.13.71.124, executor 103, partition 62, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 63.0 in stage 1128.0 (TID 28162) (10.13.64.49, executor 116, partition 63, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 64.0 in stage 1128.0 (TID 28163) (10.13.71.34, executor 132, partition 64, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 65.0 in stage 1128.0 (TID 28164) (10.13.67.242, executor 100, partition 65, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 66.0 in stage 1128.0 (TID 28165) (10.13.67.111, executor 106, partition 66, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 67.0 in stage 1128.0 (TID 28166) (10.13.64.47, executor 101, partition 67, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 68.0 in stage 1128.0 (TID 28167) (10.13.69.209, executor 117, partition 68, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 69.0 in stage 1128.0 (TID 28168) (10.13.71.86, executor 130, partition 69, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 70.0 in stage 1128.0 (TID 28169) (10.13.66.167, executor 111, partition 70, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 71.0 in stage 1128.0 (TID 28170) (10.13.66.126, executor 125, partition 71, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 72.0 in stage 1128.0 (TID 28171) (10.13.70.215, executor 126, partition 72, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 73.0 in stage 1128.0 (TID 28172) (10.13.64.125, executor 120, partition 73, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 74.0 in stage 1128.0 (TID 28173) (10.13.68.167, executor 133, partition 74, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 75.0 in stage 1128.0 (TID 28174) (10.13.71.160, executor 127, partition 75, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 76.0 in stage 1128.0 (TID 28175) (10.13.70.66, executor 119, partition 76, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 77.0 in stage 1128.0 (TID 28176) (10.13.67.67, executor 108, partition 77, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 78.0 in stage 1128.0 (TID 28177) (10.13.69.102, executor 109, partition 78, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 79.0 in stage 1128.0 (TID 28178) (10.13.70.254, executor 84, partition 79, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 80.0 in stage 1128.0 (TID 28179) (10.13.70.189, executor 128, partition 80, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 81.0 in stage 1128.0 (TID 28180) (10.13.69.60, executor 107, partition 81, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 82.0 in stage 1128.0 (TID 28181) (10.13.69.166, executor 121, partition 82, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Starting task 83.0 in stage 1128.0 (TID 28182) (10.13.71.124, executor 103, partition 83, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO MemoryStore: Block broadcast_1508 stored as values in memory (estimated size 161.2 KiB, free 25.2 GiB)
24/07/19 15:25:26 INFO DAGScheduler: Registering RDD 5106 (wrapper at /root/.ipykernel/18206/command-230092523437669-2836746030:13) as input to shuffle 254
24/07/19 15:25:26 INFO DAGScheduler: Got map stage job 502 (wrapper at /root/.ipykernel/18206/command-230092523437669-2836746030:13) with 1 output partitions
24/07/19 15:25:26 INFO DAGScheduler: Final stage: ShuffleMapStage 1129 (wrapper at /root/.ipykernel/18206/command-230092523437669-2836746030:13)
24/07/19 15:25:26 INFO DAGScheduler: Parents of final stage: List()
24/07/19 15:25:26 INFO DAGScheduler: Missing parents: List()
24/07/19 15:25:26 INFO MemoryStore: Block broadcast_1508_piece0 stored as bytes in memory (estimated size 43.0 KiB, free 25.2 GiB)
24/07/19 15:25:26 INFO BlockManagerInfo: Added broadcast_1508_piece0 in memory on 10.13.66.62:42183 (size: 43.0 KiB, free: 25.6 GiB)
24/07/19 15:25:26 INFO DAGScheduler: Submitting ShuffleMapStage 1129 (MapPartitionsRDD[5106] at wrapper at /root/.ipykernel/18206/command-230092523437669-2836746030:13), which has no missing parents
24/07/19 15:25:26 INFO SparkContext: Created broadcast 1508 from broadcast at TaskSetManager.scala:711
24/07/19 15:25:26 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 1129 (MapPartitionsRDD[5106] at wrapper at /root/.ipykernel/18206/command-230092523437669-2836746030:13) (first 15 tasks are for partitions Vector(0))
24/07/19 15:25:26 INFO TaskSchedulerImpl: Adding task set 1129.0 with 1 tasks resource profile 0
24/07/19 15:25:26 INFO TaskSetManager: TaskSet 1129.0 using PreferredLocationsV1
24/07/19 15:25:26 INFO FairSchedulableBuilder: Added task set TaskSet_1129.0 tasks to pool 4085144645627145219
24/07/19 15:25:26 INFO BlockManagerInfo: Added broadcast_1508_piece0 in memory on 10.13.67.242:34949 (size: 43.0 KiB, free: 11.0 GiB)
24/07/19 15:25:26 INFO BlockManagerInfo: Added broadcast_1508_piece0 in memory on 10.13.70.254:42683 (size: 43.0 KiB, free: 11.0 GiB)
24/07/19 15:25:26 INFO BlockManagerInfo: Added broadcast_1508_piece0 in memory on 10.13.64.47:39169 (size: 43.0 KiB, free: 11.1 GiB)
24/07/19 15:25:26 INFO BlockManagerInfo: Added broadcast_1508_piece0 in memory on 10.13.71.124:39087 (size: 43.0 KiB, free: 11.1 GiB)
24/07/19 15:25:26 INFO BlockManagerInfo: Added broadcast_1508_piece0 in memory on 10.13.67.67:37729 (size: 43.0 KiB, free: 11.1 GiB)
24/07/19 15:25:26 INFO BlockManagerInfo: Added broadcast_1508_piece0 in memory on 10.13.68.167:44723 (size: 43.0 KiB, free: 11.1 GiB)
24/07/19 15:25:26 INFO BlockManagerInfo: Added broadcast_1508_piece0 in memory on 10.13.71.86:34131 (size: 43.0 KiB, free: 11.1 GiB)
24/07/19 15:25:26 INFO BlockManagerInfo: Added broadcast_1508_piece0 in memory on 10.13.66.167:40621 (size: 43.0 KiB, free: 11.1 GiB)
24/07/19 15:25:26 INFO BlockManagerInfo: Added broadcast_1508_piece0 in memory on 10.13.67.111:40323 (size: 43.0 KiB, free: 11.1 GiB)
24/07/19 15:25:26 INFO BlockManagerInfo: Added broadcast_1508_piece0 in memory on 10.13.70.215:44777 (size: 43.0 KiB, free: 11.1 GiB)
24/07/19 15:25:26 INFO BlockManagerInfo: Added broadcast_1508_piece0 in memory on 10.13.71.34:35213 (size: 43.0 KiB, free: 11.1 GiB)
24/07/19 15:25:26 INFO BlockManagerInfo: Added broadcast_1508_piece0 in memory on 10.13.70.189:46789 (size: 43.0 KiB, free: 11.1 GiB)
24/07/19 15:25:26 INFO BlockManagerInfo: Added broadcast_1508_piece0 in memory on 10.13.66.126:40899 (size: 43.0 KiB, free: 11.1 GiB)
24/07/19 15:25:26 INFO BlockManagerInfo: Added broadcast_1508_piece0 in memory on 10.13.64.49:34851 (size: 43.0 KiB, free: 11.1 GiB)
24/07/19 15:25:26 INFO BlockManagerInfo: Added broadcast_1508_piece0 in memory on 10.13.64.125:44919 (size: 43.0 KiB, free: 11.1 GiB)
24/07/19 15:25:26 INFO BlockManagerInfo: Added broadcast_1508_piece0 in memory on 10.13.71.160:39905 (size: 43.0 KiB, free: 11.1 GiB)
24/07/19 15:25:26 INFO BlockManagerInfo: Added broadcast_1508_piece0 in memory on 10.13.69.60:37447 (size: 43.0 KiB, free: 11.1 GiB)
24/07/19 15:25:26 INFO BlockManagerInfo: Added broadcast_1508_piece0 in memory on 10.13.70.66:39213 (size: 43.0 KiB, free: 11.1 GiB)
24/07/19 15:25:26 INFO BlockManagerInfo: Added broadcast_1508_piece0 in memory on 10.13.69.166:37577 (size: 43.0 KiB, free: 11.1 GiB)
24/07/19 15:25:26 INFO BlockManagerInfo: Added broadcast_1508_piece0 in memory on 10.13.69.102:44669 (size: 43.0 KiB, free: 11.1 GiB)
24/07/19 15:25:26 INFO BlockManagerInfo: Added broadcast_1508_piece0 in memory on 10.13.69.209:36605 (size: 43.0 KiB, free: 11.1 GiB)
24/07/19 15:25:26 INFO TaskSetManager: Starting task 0.0 in stage 1129.0 (TID 28183) (10.13.67.242, executor 100, partition 0, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO TaskSetManager: Finished task 2.0 in stage 1128.0 (TID 28101) in 66 ms on 10.13.67.242 (executor 100) (1/84)
24/07/19 15:25:26 INFO MemoryStore: Block broadcast_1509 stored as values in memory (estimated size 178.1 KiB, free 25.2 GiB)
24/07/19 15:25:26 INFO MemoryStore: Block broadcast_1509_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 25.2 GiB)
24/07/19 15:25:26 INFO BlockManagerInfo: Added broadcast_1509_piece0 in memory on 10.13.66.62:42183 (size: 50.2 KiB, free: 25.6 GiB)
24/07/19 15:25:26 INFO SparkContext: Created broadcast 1509 from broadcast at TaskSetManager.scala:711
24/07/19 15:25:26 INFO TaskSetManager: Finished task 23.0 in stage 1128.0 (TID 28122) in 69 ms on 10.13.67.242 (executor 100) (2/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 37.0 in stage 1128.0 (TID 28136) in 70 ms on 10.13.70.254 (executor 84) (3/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 71.0 in stage 1128.0 (TID 28170) in 70 ms on 10.13.66.126 (executor 125) (4/84)
24/07/19 15:25:26 INFO BlockManagerInfo: Added broadcast_1509_piece0 in memory on 10.13.67.242:34949 (size: 50.2 KiB, free: 11.0 GiB)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 65.0 in stage 1128.0 (TID 28164) in 72 ms on 10.13.67.242 (executor 100) (5/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 46.0 in stage 1128.0 (TID 28145) in 72 ms on 10.13.64.47 (executor 101) (6/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 79.0 in stage 1128.0 (TID 28178) in 73 ms on 10.13.70.254 (executor 84) (7/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 30.0 in stage 1128.0 (TID 28129) in 73 ms on 10.13.70.215 (executor 126) (8/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 33.0 in stage 1128.0 (TID 28132) in 74 ms on 10.13.71.160 (executor 127) (9/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 49.0 in stage 1128.0 (TID 28148) in 75 ms on 10.13.66.167 (executor 111) (10/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 64.0 in stage 1128.0 (TID 28163) in 75 ms on 10.13.71.34 (executor 132) (11/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 52.0 in stage 1128.0 (TID 28151) in 76 ms on 10.13.64.125 (executor 120) (12/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 68.0 in stage 1128.0 (TID 28167) in 77 ms on 10.13.69.209 (executor 117) (13/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 59.0 in stage 1128.0 (TID 28158) in 77 ms on 10.13.70.189 (executor 128) (14/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 45.0 in stage 1128.0 (TID 28144) in 79 ms on 10.13.67.111 (executor 106) (15/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 44.0 in stage 1128.0 (TID 28143) in 79 ms on 10.13.67.242 (executor 100) (16/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 73.0 in stage 1128.0 (TID 28172) in 80 ms on 10.13.64.125 (executor 120) (17/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 76.0 in stage 1128.0 (TID 28175) in 80 ms on 10.13.70.66 (executor 119) (18/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 56.0 in stage 1128.0 (TID 28155) in 80 ms on 10.13.67.67 (executor 108) (19/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 4.0 in stage 1128.0 (TID 28103) in 81 ms on 10.13.64.47 (executor 101) (20/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 43.0 in stage 1128.0 (TID 28142) in 81 ms on 10.13.71.34 (executor 132) (21/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 75.0 in stage 1128.0 (TID 28174) in 81 ms on 10.13.71.160 (executor 127) (22/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 81.0 in stage 1128.0 (TID 28180) in 81 ms on 10.13.69.60 (executor 107) (23/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 15.0 in stage 1128.0 (TID 28114) in 82 ms on 10.13.69.102 (executor 109) (24/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 34.0 in stage 1128.0 (TID 28133) in 83 ms on 10.13.70.66 (executor 119) (25/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 25.0 in stage 1128.0 (TID 28124) in 84 ms on 10.13.64.47 (executor 101) (26/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 17.0 in stage 1128.0 (TID 28116) in 84 ms on 10.13.70.189 (executor 128) (27/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 41.0 in stage 1128.0 (TID 28140) in 84 ms on 10.13.71.124 (executor 103) (28/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 29.0 in stage 1128.0 (TID 28128) in 84 ms on 10.13.66.126 (executor 125) (29/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 48.0 in stage 1128.0 (TID 28147) in 84 ms on 10.13.71.86 (executor 130) (30/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 58.0 in stage 1128.0 (TID 28157) in 84 ms on 10.13.70.254 (executor 84) (31/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 13.0 in stage 1128.0 (TID 28112) in 84 ms on 10.13.70.66 (executor 119) (32/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 36.0 in stage 1128.0 (TID 28135) in 84 ms on 10.13.69.102 (executor 109) (33/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 16.0 in stage 1128.0 (TID 28115) in 84 ms on 10.13.70.254 (executor 84) (34/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 55.0 in stage 1128.0 (TID 28154) in 85 ms on 10.13.70.66 (executor 119) (35/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 60.0 in stage 1128.0 (TID 28159) in 86 ms on 10.13.69.60 (executor 107) (36/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 18.0 in stage 1128.0 (TID 28117) in 86 ms on 10.13.69.60 (executor 107) (37/84)
24/07/19 15:25:26 INFO BlockManagerInfo: Added broadcast_1507_piece0 in memory on 10.13.67.242:34949 (size: 15.9 KiB, free: 11.0 GiB)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 3.0 in stage 1128.0 (TID 28102) in 88 ms on 10.13.67.111 (executor 106) (38/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 11.0 in stage 1128.0 (TID 28110) in 87 ms on 10.13.68.167 (executor 133) (39/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 8.0 in stage 1128.0 (TID 28107) in 88 ms on 10.13.66.126 (executor 125) (40/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 67.0 in stage 1128.0 (TID 28166) in 87 ms on 10.13.64.47 (executor 101) (41/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 27.0 in stage 1128.0 (TID 28126) in 88 ms on 10.13.71.86 (executor 130) (42/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 40.0 in stage 1128.0 (TID 28139) in 88 ms on 10.13.69.166 (executor 121) (43/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 74.0 in stage 1128.0 (TID 28173) in 88 ms on 10.13.68.167 (executor 133) (44/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 50.0 in stage 1128.0 (TID 28149) in 89 ms on 10.13.66.126 (executor 125) (45/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 82.0 in stage 1128.0 (TID 28181) in 89 ms on 10.13.69.166 (executor 121) (46/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 42.0 in stage 1128.0 (TID 28141) in 90 ms on 10.13.64.49 (executor 116) (47/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 53.0 in stage 1128.0 (TID 28152) in 91 ms on 10.13.68.167 (executor 133) (48/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 47.0 in stage 1128.0 (TID 28146) in 92 ms on 10.13.69.209 (executor 117) (49/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 51.0 in stage 1128.0 (TID 28150) in 92 ms on 10.13.70.215 (executor 126) (50/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 22.0 in stage 1128.0 (TID 28121) in 92 ms on 10.13.71.34 (executor 132) (51/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 0.0 in stage 1128.0 (TID 28099) in 94 ms on 10.13.64.49 (executor 116) (52/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 63.0 in stage 1128.0 (TID 28162) in 93 ms on 10.13.64.49 (executor 116) (53/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 26.0 in stage 1128.0 (TID 28125) in 93 ms on 10.13.69.209 (executor 117) (54/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 78.0 in stage 1128.0 (TID 28177) in 93 ms on 10.13.69.102 (executor 109) (55/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 7.0 in stage 1128.0 (TID 28106) in 94 ms on 10.13.66.167 (executor 111) (56/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 70.0 in stage 1128.0 (TID 28169) in 94 ms on 10.13.66.167 (executor 111) (57/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 72.0 in stage 1128.0 (TID 28171) in 94 ms on 10.13.70.215 (executor 126) (58/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 66.0 in stage 1128.0 (TID 28165) in 95 ms on 10.13.67.111 (executor 106) (59/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 1.0 in stage 1128.0 (TID 28100) in 96 ms on 10.13.71.34 (executor 132) (60/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 31.0 in stage 1128.0 (TID 28130) in 96 ms on 10.13.64.125 (executor 120) (61/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 10.0 in stage 1128.0 (TID 28109) in 96 ms on 10.13.64.125 (executor 120) (62/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 38.0 in stage 1128.0 (TID 28137) in 97 ms on 10.13.70.189 (executor 128) (63/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 5.0 in stage 1128.0 (TID 28104) in 98 ms on 10.13.69.209 (executor 117) (64/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 77.0 in stage 1128.0 (TID 28176) in 97 ms on 10.13.67.67 (executor 108) (65/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 80.0 in stage 1128.0 (TID 28179) in 97 ms on 10.13.70.189 (executor 128) (66/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 14.0 in stage 1128.0 (TID 28113) in 98 ms on 10.13.67.67 (executor 108) (67/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 62.0 in stage 1128.0 (TID 28161) in 98 ms on 10.13.71.124 (executor 103) (68/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 57.0 in stage 1128.0 (TID 28156) in 98 ms on 10.13.69.102 (executor 109) (69/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 39.0 in stage 1128.0 (TID 28138) in 98 ms on 10.13.69.60 (executor 107) (70/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 20.0 in stage 1128.0 (TID 28119) in 98 ms on 10.13.71.124 (executor 103) (71/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 6.0 in stage 1128.0 (TID 28105) in 100 ms on 10.13.71.86 (executor 130) (72/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 54.0 in stage 1128.0 (TID 28153) in 99 ms on 10.13.71.160 (executor 127) (73/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 32.0 in stage 1128.0 (TID 28131) in 101 ms on 10.13.68.167 (executor 133) (74/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 9.0 in stage 1128.0 (TID 28108) in 102 ms on 10.13.70.215 (executor 126) (75/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 12.0 in stage 1128.0 (TID 28111) in 102 ms on 10.13.71.160 (executor 127) (76/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 28.0 in stage 1128.0 (TID 28127) in 104 ms on 10.13.66.167 (executor 111) (77/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 19.0 in stage 1128.0 (TID 28118) in 105 ms on 10.13.69.166 (executor 121) (78/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 21.0 in stage 1128.0 (TID 28120) in 106 ms on 10.13.64.49 (executor 116) (79/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 24.0 in stage 1128.0 (TID 28123) in 109 ms on 10.13.67.111 (executor 106) (80/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 35.0 in stage 1128.0 (TID 28134) in 111 ms on 10.13.67.67 (executor 108) (81/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 69.0 in stage 1128.0 (TID 28168) in 113 ms on 10.13.71.86 (executor 130) (82/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 61.0 in stage 1128.0 (TID 28160) in 116 ms on 10.13.69.166 (executor 121) (83/84)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 0.0 in stage 1129.0 (TID 28183) in 90 ms on 10.13.67.242 (executor 100) (1/1)
24/07/19 15:25:26 INFO TaskSchedulerImpl: Removed TaskSet 1129.0, whose tasks have all completed, from pool 4085144645627145219
24/07/19 15:25:26 INFO DAGScheduler: ShuffleMapStage 1129 (wrapper at /root/.ipykernel/18206/command-230092523437669-2836746030:13) finished in 0.153 s
24/07/19 15:25:26 INFO DAGScheduler: looking for newly runnable stages
24/07/19 15:25:26 INFO DAGScheduler: running: Set(ShuffleMapStage 1128)
24/07/19 15:25:26 INFO DAGScheduler: waiting: Set()
24/07/19 15:25:26 INFO DAGScheduler: failed: Set()
24/07/19 15:25:26 WARN HintErrorLogger: Hint (strategy=broadcast) is not supported in the query: build left for left outer join.
24/07/19 15:25:26 INFO TaskSetManager: Finished task 83.0 in stage 1128.0 (TID 28182) in 177 ms on 10.13.71.124 (executor 103) (84/84)
24/07/19 15:25:26 INFO TaskSchedulerImpl: Removed TaskSet 1128.0, whose tasks have all completed, from pool 4085144645627145219
24/07/19 15:25:26 INFO DAGScheduler: ShuffleMapStage 1128 (wrapper at /root/.ipykernel/18206/command-230092523437669-2836746030:13) finished in 0.180 s
24/07/19 15:25:26 INFO DAGScheduler: looking for newly runnable stages
24/07/19 15:25:26 INFO DAGScheduler: running: Set()
24/07/19 15:25:26 INFO DAGScheduler: waiting: Set()
24/07/19 15:25:26 INFO DAGScheduler: failed: Set()
24/07/19 15:25:26 WARN HintErrorLogger: Hint (strategy=broadcast) is not supported in the query: build left for left outer join.
24/07/19 15:25:26 INFO ShufflePartitionsUtil: For shuffle(254), advisory target size: 67108864, actual target size 1048576, minimum partition size: 1048576
24/07/19 15:25:26 INFO CodeGenerator: Code generated in 10.936932 ms
24/07/19 15:25:26 INFO SparkContext: Starting job: wrapper at /root/.ipykernel/18206/command-230092523437669-2836746030:13
24/07/19 15:25:26 INFO DAGScheduler: Got job 503 (wrapper at /root/.ipykernel/18206/command-230092523437669-2836746030:13) with 1 output partitions
24/07/19 15:25:26 INFO DAGScheduler: Final stage: ResultStage 1132 (wrapper at /root/.ipykernel/18206/command-230092523437669-2836746030:13)
24/07/19 15:25:26 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 1130, ShuffleMapStage 1131)
24/07/19 15:25:26 INFO DAGScheduler: Missing parents: List()
24/07/19 15:25:26 INFO DAGScheduler: Submitting ResultStage 1132 (MapPartitionsRDD[5113] at wrapper at /root/.ipykernel/18206/command-230092523437669-2836746030:13), which has no missing parents
24/07/19 15:25:26 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 1132 (MapPartitionsRDD[5113] at wrapper at /root/.ipykernel/18206/command-230092523437669-2836746030:13) (first 15 tasks are for partitions Vector(0))
24/07/19 15:25:26 INFO TaskSchedulerImpl: Adding task set 1132.0 with 1 tasks resource profile 0
24/07/19 15:25:26 INFO TaskSetManager: TaskSet 1132.0 using PreferredLocationsV1
24/07/19 15:25:26 INFO FairSchedulableBuilder: Added task set TaskSet_1132.0 tasks to pool 4085144645627145219
24/07/19 15:25:26 INFO TaskSetManager: Starting task 0.0 in stage 1132.0 (TID 28184) (10.13.67.242, executor 100, partition 0, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO MemoryStore: Block broadcast_1510 stored as values in memory (estimated size 230.1 KiB, free 25.2 GiB)
24/07/19 15:25:26 INFO MemoryStore: Block broadcast_1510_piece0 stored as bytes in memory (estimated size 75.1 KiB, free 25.2 GiB)
24/07/19 15:25:26 INFO BlockManagerInfo: Added broadcast_1510_piece0 in memory on 10.13.66.62:42183 (size: 75.1 KiB, free: 25.6 GiB)
24/07/19 15:25:26 INFO SparkContext: Created broadcast 1510 from broadcast at TaskSetManager.scala:711
24/07/19 15:25:26 INFO BlockManagerInfo: Added broadcast_1510_piece0 in memory on 10.13.67.242:34949 (size: 75.1 KiB, free: 11.0 GiB)
24/07/19 15:25:26 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 254 to 10.13.67.242:37200
24/07/19 15:25:26 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 253 to 10.13.67.242:37200
24/07/19 15:25:26 INFO TaskSetManager: Finished task 0.0 in stage 1132.0 (TID 28184) in 39 ms on 10.13.67.242 (executor 100) (1/1)
24/07/19 15:25:26 INFO TaskSchedulerImpl: Removed TaskSet 1132.0, whose tasks have all completed, from pool 4085144645627145219
24/07/19 15:25:26 INFO DAGScheduler: ResultStage 1132 (wrapper at /root/.ipykernel/18206/command-230092523437669-2836746030:13) finished in 0.042 s
24/07/19 15:25:26 INFO DAGScheduler: Job 503 is finished. Cancelling potential speculative or zombie tasks for this job
24/07/19 15:25:26 INFO TaskSchedulerImpl: Killing all running tasks in stage 1132: Stage finished
24/07/19 15:25:26 INFO DAGScheduler: Job 503 finished: wrapper at /root/.ipykernel/18206/command-230092523437669-2836746030:13, took 0.043626 s
24/07/19 15:25:26 INFO ClusterLoadMonitor: Removed query with execution ID:229. Current active queries:0
24/07/19 15:25:26 INFO SparkEBJCleanupListener: Cleaning up ebj join relation block with ID spark_join_relation_253_1455983219)
24/07/19 15:25:26 WARN BlockManager: Asked to remove block spark_join_relation_253_1455983219, which does not exist
24/07/19 15:25:26 INFO QueryProfileListener: Query profile sent to logger, seq number: 229, app id: app-20240719133525-0000
24/07/19 15:25:26 INFO ClusterLoadMonitor: Added query with execution ID:230. Current active queries:1
24/07/19 15:25:26 INFO SparkContext: Starting job: wrapper at /root/.ipykernel/18206/command-230092523437669-2836746030:14
24/07/19 15:25:26 INFO DAGScheduler: Got job 504 (wrapper at /root/.ipykernel/18206/command-230092523437669-2836746030:14) with 1 output partitions
24/07/19 15:25:26 INFO DAGScheduler: Final stage: ResultStage 1135 (wrapper at /root/.ipykernel/18206/command-230092523437669-2836746030:14)
24/07/19 15:25:26 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 1134, ShuffleMapStage 1133)
24/07/19 15:25:26 INFO DAGScheduler: Missing parents: List()
24/07/19 15:25:26 INFO DAGScheduler: Submitting ResultStage 1135 (MapPartitionsRDD[5118] at wrapper at /root/.ipykernel/18206/command-230092523437669-2836746030:14), which has no missing parents
24/07/19 15:25:26 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 1135 (MapPartitionsRDD[5118] at wrapper at /root/.ipykernel/18206/command-230092523437669-2836746030:14) (first 15 tasks are for partitions Vector(0))
24/07/19 15:25:26 INFO TaskSchedulerImpl: Adding task set 1135.0 with 1 tasks resource profile 0
24/07/19 15:25:26 INFO TaskSetManager: TaskSet 1135.0 using PreferredLocationsV1
24/07/19 15:25:26 INFO FairSchedulableBuilder: Added task set TaskSet_1135.0 tasks to pool 4085144645627145219
24/07/19 15:25:26 INFO TaskSetManager: Starting task 0.0 in stage 1135.0 (TID 28185) (10.13.67.242, executor 100, partition 0, PROCESS_LOCAL, 
24/07/19 15:25:26 INFO MemoryStore: Block broadcast_1511 stored as values in memory (estimated size 233.3 KiB, free 25.2 GiB)
24/07/19 15:25:26 INFO MemoryStore: Block broadcast_1511_piece0 stored as bytes in memory (estimated size 75.5 KiB, free 25.2 GiB)
24/07/19 15:25:26 INFO BlockManagerInfo: Added broadcast_1511_piece0 in memory on 10.13.66.62:42183 (size: 75.5 KiB, free: 25.6 GiB)
24/07/19 15:25:26 INFO SparkContext: Created broadcast 1511 from broadcast at TaskSetManager.scala:711
24/07/19 15:25:26 INFO BlockManagerInfo: Added broadcast_1511_piece0 in memory on 10.13.67.242:34949 (size: 75.5 KiB, free: 11.0 GiB)
24/07/19 15:25:26 INFO TaskSetManager: Finished task 0.0 in stage 1135.0 (TID 28185) in 42 ms on 10.13.67.242 (executor 100) (1/1)
24/07/19 15:25:26 INFO TaskSchedulerImpl: Removed TaskSet 1135.0, whose tasks have all completed, from pool 4085144645627145219
24/07/19 15:25:26 INFO DAGScheduler: ResultStage 1135 (wrapper at /root/.ipykernel/18206/command-230092523437669-2836746030:14) finished in 0.044 s
24/07/19 15:25:26 INFO DAGScheduler: Job 504 is finished. Cancelling potential speculative or zombie tasks for this job
24/07/19 15:25:26 INFO TaskSchedulerImpl: Killing all running tasks in stage 1135: Stage finished
24/07/19 15:25:26 INFO DAGScheduler: Job 504 finished: wrapper at /root/.ipykernel/18206/command-230092523437669-2836746030:14, took 0.045943 s
24/07/19 15:25:26 INFO ClusterLoadMonitor: Removed query with execution ID:230. Current active queries:0
24/07/19 15:25:26 INFO SparkEBJCleanupListener: Cleaning up ebj join relation block with ID spark_join_relation_253_1455983219)
24/07/19 15:25:26 WARN BlockManager: Asked to remove block spark_join_relation_253_1455983219, which does not exist
24/07/19 15:25:26 INFO QueryProfileListener: Query profile sent to logger, seq number: 230, app id: app-20240719133525-0000
24/07/19 15:25:28 INFO ClusterLoadAvgHelper: Current cluster load: 0, Old Ema: 1.0, New Ema: 0.85 
24/07/19 15:25:31 INFO ClusterLoadAvgHelper: Current cluster load: 0, Old Ema: 0.85, New Ema: 0.0 
24/07/19 15:25:32 INFO ProgressReporter$: Removed result fetcher for 4085144645627145219_6406455263148317922_job-23196080691842-run-226089366195953-action-6604081895629205
24/07/19 15:25:32 INFO ProgressReporter$: Added result fetcher for 4085144645627145219_5037104740190529778_1a9d9c12-6ac3-44c2-8aa6-593a3e35f5aa
24/07/19 15:25:32 INFO ProgressReporter$: Removed result fetcher for 4085144645627145219_5037104740190529778_1a9d9c12-6ac3-44c2-8aa6-593a3e35f5aa
24/07/19 15:25:34 INFO DriverCorral$: Cleaning the wrapper ReplId-38b15-96942-b1ec0-3 (currently in status Idle(ReplId-38b15-96942-b1ec0-3))
24/07/19 15:25:34 INFO DriverCorral$: sending shutdown signal for REPL ReplId-38b15-96942-b1ec0-3
24/07/19 15:25:34 INFO PythonDriverLocalBase$RedirectThread: Python RedirectThread exit
24/07/19 15:25:34 INFO PythonDriverLocalBase$RedirectThread: Python RedirectThread exit
24/07/19 15:25:34 INFO ReplCrashUtils$: python shell exit code: 143; replId: ReplId-38b15-96942-b1ec0-3, pid: 18206
24/07/19 15:25:34 INFO DriverCorral$: sending the interrupt signal for REPL ReplId-38b15-96942-b1ec0-3
24/07/19 15:25:34 INFO DriverCorral$: waiting for localThread to stop for REPL ReplId-38b15-96942-b1ec0-3
24/07/19 15:25:34 INFO DriverCorral$: ReplId-38b15-96942-b1ec0-3 successfully discarded
24/07/19 15:25:35 WARN DynamicSparkConfContextImpl: Ignored update because id 1721395931188 < 1721395931188; source: ConfigFile
24/07/19 15:25:35 INFO DriverCorral: Received SAFEr configs with version 1721395931188
24/07/19 15:25:35 INFO DriverCorral: DBFS health check ok
24/07/19 15:25:35 INFO HiveMetaStore: 1: get_database: default
24/07/19 15:25:35 INFO audit: ugi=root	ip=unknown-ip-addr	cmd=get_database: default	
24/07/19 15:25:35 INFO DriverCorral: Metastore health check ok
24/07/19 15:25:36 WARN SQLConf: The SQL config 'spark.sql.hive.convertCTAS' has been deprecated in Spark v3.1 and may be removed in the future. Set 'spark.sql.legacy.createHiveTableByDefault' to false instead.
24/07/19 15:25:36 WARN SQLConf: The SQL config 'spark.sql.hive.convertCTAS' has been deprecated in Spark v3.1 and may be removed in the future. Set 'spark.sql.legacy.createHiveTableByDefault' to false instead.
24/07/19 15:25:36 INFO DriverCorral: Starting python repl ReplId-19faa-a9550-74320
24/07/19 15:25:36 INFO PyPIPkgInfo$$anon$1: create pypi file /local_disk0/tmp/clusterWideLibDir/__clusterWide_lib_00000_7181440e2693324539d1591450f3526d.pypi for project CurrencyConverter
24/07/19 15:25:36 INFO JupyterDriverLocal: Starting gateway server for repl ReplId-19faa-a9550-74320
24/07/19 15:25:36 INFO PythonPy4JUtil: Using pinned thread mode in Py4J
24/07/19 15:25:37 INFO VirtualenvCloneHelper: Creating notebook-scoped virtualenv for 656133fc-8dca-4de1-b5f3-f95b26619ca3 with python user root
24/07/19 15:25:37 INFO Utils: resolved command to be run: List(virtualenv, /local_disk0/.ephemeral_nfs/envs/pythonEnv-656133fc-8dca-4de1-b5f3-f95b26619ca3, -p, /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/python, --no-download, --no-setuptools, --no-wheel)
24/07/19 15:25:37 INFO PythonEnvCloneHelper$: Created python virtualenv: /local_disk0/.ephemeral_nfs/envs/pythonEnv-656133fc-8dca-4de1-b5f3-f95b26619ca3
24/07/19 15:25:37 INFO Utils: resolved command to be run: List(/local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/python, -c, import sys; dirs=[p for p in sys.path if 'package' in p]; print(' '.join(dirs)))
24/07/19 15:25:37 INFO Utils: resolved command to be run: List(/local_disk0/.ephemeral_nfs/envs/pythonEnv-656133fc-8dca-4de1-b5f3-f95b26619ca3/bin/python, -c, from sysconfig import get_path; print(get_path('purelib')))
24/07/19 15:25:37 INFO PythonEnvCloneHelper$: Created sites.pth at /local_disk0/.ephemeral_nfs/envs/pythonEnv-656133fc-8dca-4de1-b5f3-f95b26619ca3/lib/python3.10/site-packages/sites.pth
24/07/19 15:25:37 INFO NotebookScopedPythonEnvManager: Time spent to start virtualenv /local_disk0/.ephemeral_nfs/envs/pythonEnv-656133fc-8dca-4de1-b5f3-f95b26619ca3 is 229(ms)
24/07/19 15:25:37 INFO NotebookScopedPythonEnvManager: Registered /local_disk0/.ephemeral_nfs/envs/pythonEnv-656133fc-8dca-4de1-b5f3-f95b26619ca3/lib/python3.10/site-packages with the WatchService sun.nio.fs.LinuxWatchService$LinuxWatchKey@18bce245
24/07/19 15:25:37 INFO IpykernelUtils$: Python process builder: [bash, /local_disk0/.ephemeral_nfs/envs/pythonEnv-656133fc-8dca-4de1-b5f3-f95b26619ca3/python_start_notebook_scoped.sh, /databricks/spark/python/pyspark/wrapped_python.py, root, /local_disk0/.ephemeral_nfs/envs/pythonEnv-656133fc-8dca-4de1-b5f3-f95b26619ca3/bin/python, /databricks/python_shell/scripts/db_ipykernel_launcher.py, -f, /databricks/kernel-connections/eac711e5ccf914b5365875e8843d2ba72d0a52a7f42c9f5f5d830157061aac36.json]
24/07/19 15:25:37 INFO IpykernelUtils$: Cgroup isolation disabled, not placing python process with ReplId=ReplId-19faa-a9550-74320 in repl cgroup
24/07/19 15:25:39 INFO PythonDriverWrapper: setupRepl:ReplId-19faa-a9550-74320: finished to load
24/07/19 15:25:39 INFO ProgressReporter$: Added result fetcher for 116999759587656480_8465232889942420702_e088c378-5f98-42e4-820c-4e8a28635a4f
24/07/19 15:25:39 INFO ProgressReporter$: Removed result fetcher for 116999759587656480_8465232889942420702_e088c378-5f98-42e4-820c-4e8a28635a4f
24/07/19 15:25:39 INFO ProgressReporter$: Added result fetcher for 116999759587656480_7649902306085856559_job-23196080691842-run-303359891138582-action-2203242799067577
24/07/19 15:25:39 INFO ProgressReporter$: Removed result fetcher for 116999759587656480_7649902306085856559_job-23196080691842-run-303359891138582-action-2203242799067577
24/07/19 15:25:39 INFO ProgressReporter$: Added result fetcher for 116999759587656480_6577393541675970279_job-23196080691842-run-303359891138582-action-2203242799067577
24/07/19 15:25:39 INFO ReliableJobsSessionClient: Created ReliableJobsSessionClient with timeout of 60 seconds
24/07/19 15:25:40 INFO ProgressReporter$: Removed result fetcher for 116999759587656480_6577393541675970279_job-23196080691842-run-303359891138582-action-2203242799067577
24/07/19 15:25:40 INFO ProgressReporter$: Added result fetcher for 116999759587656480_7003668038968523249_job-23196080691842-run-303359891138582-action-2203242799067577
24/07/19 15:25:40 INFO ProgressReporter$: Removed result fetcher for 116999759587656480_7003668038968523249_job-23196080691842-run-303359891138582-action-2203242799067577
24/07/19 15:25:40 INFO ProgressReporter$: Added result fetcher for 116999759587656480_6176117216235452428_job-23196080691842-run-303359891138582-action-2203242799067577
24/07/19 15:25:40 INFO ProgressReporter$: Removed result fetcher for 116999759587656480_6176117216235452428_job-23196080691842-run-303359891138582-action-2203242799067577
24/07/19 15:25:40 INFO ProgressReporter$: Added result fetcher for 116999759587656480_8261889283528090835_job-23196080691842-run-303359891138582-action-2203242799067577
24/07/19 15:25:40 INFO ProgressReporter$: Removed result fetcher for 116999759587656480_8261889283528090835_job-23196080691842-run-303359891138582-action-2203242799067577
24/07/19 15:25:40 INFO ProgressReporter$: Added result fetcher for 116999759587656480_8013011612161194847_ad67fe9a-daa8-4704-8688-b063bebd4d1c
24/07/19 15:25:40 WARN JupyterDriverLocal: User code returned error with traceback: ---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
File ~/.ipykernel/18356/command--1-1238724229:14
     11         os.remove(filepath)
     12         print(file_content)
---> 14     ____databricks_read_dependency_file()
     16 finally:
     17     del ____databricks_read_dependency_file

File ~/.ipykernel/18356/command--1-1238724229:8, in ____databricks_read_dependency_file()
      5 def ____databricks_read_dependency_file():
      6     filepath = """/tmp/run-303359891138582-pip-list-output"""
----> 8     with open(filepath, 'r') as file:
      9         file_content = file.read()
     11     os.remove(filepath)

File /databricks/python/lib/python3.10/site-packages/IPython/core/interactiveshell.py:282, in _modified_open(file, *args, **kwargs)
    275 if file in {0, 1, 2}:
    276     raise ValueError(
    277         f"IPython won't let you open fd={file} by default "
    278         "as it is likely to crash IPython. If you know what you are doing, "
    279         "you can use builtins' open."
    280     )
--> 282 return io_open(file, *args, **kwargs)

FileNotFoundError: [Errno 2] No such file or directory: '/tmp/run-303359891138582-pip-list-output'
24/07/19 15:25:40 INFO ProgressReporter$: Removed result fetcher for 116999759587656480_8013011612161194847_ad67fe9a-daa8-4704-8688-b063bebd4d1c
24/07/19 15:25:43 INFO DriverCorral$: Cleaning the wrapper ReplId-19faa-a9550-74320 (currently in status Idle(ReplId-19faa-a9550-74320))
24/07/19 15:25:43 INFO DriverCorral$: sending shutdown signal for REPL ReplId-19faa-a9550-74320
24/07/19 15:25:43 INFO PythonDriverLocalBase$RedirectThread: Python RedirectThread exit
24/07/19 15:25:43 INFO PythonDriverLocalBase$RedirectThread: Python RedirectThread exit
24/07/19 15:25:43 INFO ReplCrashUtils$: python shell exit code: 143; replId: ReplId-19faa-a9550-74320, pid: 18356
24/07/19 15:25:43 INFO DriverCorral$: sending the interrupt signal for REPL ReplId-19faa-a9550-74320
24/07/19 15:25:43 INFO DriverCorral$: waiting for localThread to stop for REPL ReplId-19faa-a9550-74320
24/07/19 15:25:43 INFO DriverCorral$: ReplId-19faa-a9550-74320 successfully discarded

 

 

 Failed task:

Screenshot 2024-07-19 at 19.55.48.png

ā€ƒ

0 Kudos
0 REPLIES 0

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonā€™t want to miss the chance to attend and share knowledge.

If there isnā€™t a group near you, start one and help create a community that brings people together.

Request a New Group
Announcements