2020-11-27 02:09:34 [INFO] ------------Environment Information------------- platform: Linux-3.10.0_3-0-0-34-x86_64-with-centos-7.5.1804-Core Python: 3.7.9 (default, Aug 31 2020, 12:42:55) [GCC 7.3.0] Paddle compiled with cuda: True NVCC: Cuda compilation tools, release 10.2, V10.2.89 cudnn: 7.6 GPUs used: 4 CUDA_VISIBLE_DEVICES: 0,1,2,3 GPU: ['GPU 0: Tesla V100-SXM2-16GB', 'GPU 1: Tesla V100-SXM2-16GB', 'GPU 2: Tesla V100-SXM2-16GB', 'GPU 3: Tesla V100-SXM2-16GB', 'GPU 4: Tesla V100-SXM2-16GB', 'GPU 5: Tesla V100-SXM2-16GB', 'GPU 6: Tesla V100-SXM2-16GB', 'GPU 7: Tesla V100-SXM2-16GB'] GCC: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39) PaddlePaddle: 2.0.0-rc0 OpenCV: 4.1.0 ------------------------------------------------ 2020-11-27 02:09:34 [INFO] ---------------Config Information--------------- batch_size: 4 iters: 40000 learning_rate: decay: end_lr: 0.0 power: 0.9 type: poly value: 0.01 loss: coef: - 1 types: - ignore_index: 255 type: CrossEntropyLoss model: align_corners: false aspp_out_channels: 256 aspp_ratios: - 1 - 12 - 24 - 36 backbone: multi_grid: - 1 - 2 - 4 output_stride: 8 pretrained: https://bj.bcebos.com/paddleseg/dygraph/resnet101_vd_ssld.tar.gz type: ResNet101_vd backbone_indices: - 3 pretrained: null type: DeepLabV3 optimizer: momentum: 0.9 type: sgd weight_decay: 4.0e-05 train_dataset: dataset_root: data/VOCdevkit/ mode: trainaug transforms: - max_scale_factor: 2.0 min_scale_factor: 0.5 scale_step_size: 0.25 type: ResizeStepScaling - crop_size: - 512 - 512 type: RandomPaddingCrop - type: RandomHorizontalFlip - brightness_range: 0.4 contrast_range: 0.4 saturation_range: 0.4 type: RandomDistort - type: Normalize type: PascalVOC val_dataset: dataset_root: data/VOCdevkit/ mode: val transforms: - target_size: - 512 - 512 type: Padding - type: Normalize type: PascalVOC ------------------------------------------------ 2020-11-27 02:09:40 [INFO] Loading pretrained model from https://bj.bcebos.com/paddleseg/dygraph/resnet101_vd_ssld.tar.gz 2020-11-27 02:09:42 [INFO] There are 530/530 variables loaded into ResNet_vd. 2020-11-27 02:12:00 [INFO] [TRAIN] epoch=1, iter=100/40000, loss=1.4073, lr=0.009978, batch_cost=1.3138, reader_cost=0.0109 | ETA 14:33:40 2020-11-27 02:14:07 [INFO] [TRAIN] epoch=1, iter=200/40000, loss=1.1208, lr=0.009955, batch_cost=1.2713, reader_cost=0.0006 | ETA 14:03:18 2020-11-27 02:16:13 [INFO] [TRAIN] epoch=1, iter=300/40000, loss=0.9720, lr=0.009933, batch_cost=1.2639, reader_cost=0.0005 | ETA 13:56:18 2020-11-27 02:18:20 [INFO] [TRAIN] epoch=1, iter=400/40000, loss=0.8605, lr=0.009910, batch_cost=1.2661, reader_cost=0.0005 | ETA 13:55:39 2020-11-27 02:20:27 [INFO] [TRAIN] epoch=1, iter=500/40000, loss=0.7556, lr=0.009888, batch_cost=1.2699, reader_cost=0.0009 | ETA 13:56:01 2020-11-27 02:22:33 [INFO] [TRAIN] epoch=1, iter=600/40000, loss=0.8232, lr=0.009865, batch_cost=1.2661, reader_cost=0.0006 | ETA 13:51:24 2020-11-27 02:24:41 [INFO] [TRAIN] epoch=2, iter=700/40000, loss=0.7527, lr=0.009843, batch_cost=1.2775, reader_cost=0.0068 | ETA 13:56:44 2020-11-27 02:26:47 [INFO] [TRAIN] epoch=2, iter=800/40000, loss=0.7220, lr=0.009820, batch_cost=1.2631, reader_cost=0.0004 | ETA 13:45:15 2020-11-27 02:28:54 [INFO] [TRAIN] epoch=2, iter=900/40000, loss=0.7060, lr=0.009797, batch_cost=1.2660, reader_cost=0.0005 | ETA 13:45:01 2020-11-27 02:31:01 [INFO] [TRAIN] epoch=2, iter=1000/40000, loss=0.6891, lr=0.009775, batch_cost=1.2650, reader_cost=0.0006 | ETA 13:42:14 2020-11-27 02:33:07 [INFO] [TRAIN] epoch=2, iter=1100/40000, loss=0.6114, lr=0.009752, batch_cost=1.2649, reader_cost=0.0008 | ETA 13:40:03 2020-11-27 02:35:14 [INFO] [TRAIN] epoch=2, iter=1200/40000, loss=0.5655, lr=0.009730, batch_cost=1.2646, reader_cost=0.0007 | ETA 13:37:46 2020-11-27 02:37:20 [INFO] [TRAIN] epoch=2, iter=1300/40000, loss=0.5990, lr=0.009707, batch_cost=1.2692, reader_cost=0.0004 | ETA 13:38:37 2020-11-27 02:39:28 [INFO] [TRAIN] epoch=3, iter=1400/40000, loss=0.5105, lr=0.009685, batch_cost=1.2709, reader_cost=0.0069 | ETA 13:37:34 2020-11-27 02:41:34 [INFO] [TRAIN] epoch=3, iter=1500/40000, loss=0.5535, lr=0.009662, batch_cost=1.2616, reader_cost=0.0002 | ETA 13:29:30 2020-11-27 02:43:41 [INFO] [TRAIN] epoch=3, iter=1600/40000, loss=0.5083, lr=0.009639, batch_cost=1.2695, reader_cost=0.0003 | ETA 13:32:29 2020-11-27 02:45:47 [INFO] [TRAIN] epoch=3, iter=1700/40000, loss=0.4586, lr=0.009617, batch_cost=1.2616, reader_cost=0.0005 | ETA 13:25:19 2020-11-27 02:47:52 [INFO] [TRAIN] epoch=3, iter=1800/40000, loss=0.5246, lr=0.009594, batch_cost=1.2546, reader_cost=0.0006 | ETA 13:18:46 2020-11-27 02:49:59 [INFO] [TRAIN] epoch=3, iter=1900/40000, loss=0.4961, lr=0.009572, batch_cost=1.2625, reader_cost=0.0005 | ETA 13:21:41 2020-11-27 02:52:06 [INFO] [TRAIN] epoch=4, iter=2000/40000, loss=0.4813, lr=0.009549, batch_cost=1.2741, reader_cost=0.0063 | ETA 13:26:54 2020-11-27 02:54:12 [INFO] [TRAIN] epoch=4, iter=2100/40000, loss=0.4855, lr=0.009526, batch_cost=1.2658, reader_cost=0.0002 | ETA 13:19:34 2020-11-27 02:56:18 [INFO] [TRAIN] epoch=4, iter=2200/40000, loss=0.4989, lr=0.009504, batch_cost=1.2569, reader_cost=0.0003 | ETA 13:11:52 2020-11-27 02:58:25 [INFO] [TRAIN] epoch=4, iter=2300/40000, loss=0.5188, lr=0.009481, batch_cost=1.2664, reader_cost=0.0009 | ETA 13:15:43 2020-11-27 03:00:31 [INFO] [TRAIN] epoch=4, iter=2400/40000, loss=0.4758, lr=0.009459, batch_cost=1.2612, reader_cost=0.0007 | ETA 13:10:20 2020-11-27 03:02:38 [INFO] [TRAIN] epoch=4, iter=2500/40000, loss=0.5133, lr=0.009436, batch_cost=1.2703, reader_cost=0.0004 | ETA 13:13:57 2020-11-27 03:04:44 [INFO] [TRAIN] epoch=4, iter=2600/40000, loss=0.4267, lr=0.009413, batch_cost=1.2637, reader_cost=0.0003 | ETA 13:07:43 2020-11-27 03:06:51 [INFO] [TRAIN] epoch=5, iter=2700/40000, loss=0.4233, lr=0.009391, batch_cost=1.2667, reader_cost=0.0071 | ETA 13:07:28 2020-11-27 03:08:57 [INFO] [TRAIN] epoch=5, iter=2800/40000, loss=0.4101, lr=0.009368, batch_cost=1.2637, reader_cost=0.0003 | ETA 13:03:28 2020-11-27 03:11:05 [INFO] [TRAIN] epoch=5, iter=2900/40000, loss=0.4727, lr=0.009345, batch_cost=1.2725, reader_cost=0.0007 | ETA 13:06:49 2020-11-27 03:13:10 [INFO] [TRAIN] epoch=5, iter=3000/40000, loss=0.4934, lr=0.009323, batch_cost=1.2544, reader_cost=0.0005 | ETA 12:53:33 2020-11-27 03:15:16 [INFO] [TRAIN] epoch=5, iter=3100/40000, loss=0.4871, lr=0.009300, batch_cost=1.2599, reader_cost=0.0005 | ETA 12:54:49 2020-11-27 03:17:22 [INFO] [TRAIN] epoch=5, iter=3200/40000, loss=0.4462, lr=0.009277, batch_cost=1.2612, reader_cost=0.0006 | ETA 12:53:31 2020-11-27 03:19:29 [INFO] [TRAIN] epoch=5, iter=3300/40000, loss=0.3775, lr=0.009255, batch_cost=1.2670, reader_cost=0.0007 | ETA 12:54:58 2020-11-27 03:21:36 [INFO] [TRAIN] epoch=6, iter=3400/40000, loss=0.3492, lr=0.009232, batch_cost=1.2722, reader_cost=0.0063 | ETA 12:56:01 2020-11-27 03:23:43 [INFO] [TRAIN] epoch=6, iter=3500/40000, loss=0.4074, lr=0.009209, batch_cost=1.2646, reader_cost=0.0008 | ETA 12:49:19 2020-11-27 03:25:50 [INFO] [TRAIN] epoch=6, iter=3600/40000, loss=0.3864, lr=0.009186, batch_cost=1.2708, reader_cost=0.0007 | ETA 12:50:56 2020-11-27 03:27:56 [INFO] [TRAIN] epoch=6, iter=3700/40000, loss=0.4435, lr=0.009164, batch_cost=1.2676, reader_cost=0.0012 | ETA 12:46:52 2020-11-27 03:30:03 [INFO] [TRAIN] epoch=6, iter=3800/40000, loss=0.4384, lr=0.009141, batch_cost=1.2628, reader_cost=0.0010 | ETA 12:41:52 2020-11-27 03:32:09 [INFO] [TRAIN] epoch=6, iter=3900/40000, loss=0.3738, lr=0.009118, batch_cost=1.2637, reader_cost=0.0006 | ETA 12:40:19 2020-11-27 03:34:17 [INFO] [TRAIN] epoch=7, iter=4000/40000, loss=0.4113, lr=0.009096, batch_cost=1.2749, reader_cost=0.0067 | ETA 12:44:57 2020-11-27 03:34:17 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-27 03:34:50 [INFO] [EVAL] #Images=1449 mIoU=0.6204 Acc=0.9092 Kappa=0.7989 2020-11-27 03:34:50 [INFO] [EVAL] Class IoU: [0.9114 0.5849 0.3686 0.8082 0.5602 0.566 0.787 0.7714 0.7014 0.2731 0.7646 0.4932 0.6832 0.7038 0.7559 0.7765 0.4049 0.6343 0.2143 0.665 0.6004] 2020-11-27 03:34:50 [INFO] [EVAL] Class Acc: [0.9487 0.6121 0.3931 0.9334 0.8245 0.8431 0.9228 0.8504 0.9682 0.3988 0.8359 0.6359 0.7704 0.8888 0.7938 0.9128 0.8278 0.7387 0.8232 0.7068 0.7347] 2020-11-27 03:34:58 [INFO] [EVAL] The model with the best validation mIoU (0.6204) was saved at iter 4000. 2020-11-27 03:37:05 [INFO] [TRAIN] epoch=7, iter=4100/40000, loss=0.3495, lr=0.009073, batch_cost=1.2650, reader_cost=0.0007 | ETA 12:36:51 2020-11-27 03:39:10 [INFO] [TRAIN] epoch=7, iter=4200/40000, loss=0.3813, lr=0.009050, batch_cost=1.2531, reader_cost=0.0004 | ETA 12:27:41 2020-11-27 03:41:16 [INFO] [TRAIN] epoch=7, iter=4300/40000, loss=0.3832, lr=0.009027, batch_cost=1.2571, reader_cost=0.0005 | ETA 12:27:56 2020-11-27 03:43:22 [INFO] [TRAIN] epoch=7, iter=4400/40000, loss=0.3508, lr=0.009005, batch_cost=1.2633, reader_cost=0.0005 | ETA 12:29:33 2020-11-27 03:45:29 [INFO] [TRAIN] epoch=7, iter=4500/40000, loss=0.3636, lr=0.008982, batch_cost=1.2641, reader_cost=0.0006 | ETA 12:27:55 2020-11-27 03:47:35 [INFO] [TRAIN] epoch=7, iter=4600/40000, loss=0.3780, lr=0.008959, batch_cost=1.2672, reader_cost=0.0004 | ETA 12:27:39 2020-11-27 03:49:43 [INFO] [TRAIN] epoch=8, iter=4700/40000, loss=0.3942, lr=0.008936, batch_cost=1.2782, reader_cost=0.0065 | ETA 12:32:00 2020-11-27 03:51:50 [INFO] [TRAIN] epoch=8, iter=4800/40000, loss=0.4010, lr=0.008913, batch_cost=1.2659, reader_cost=0.0005 | ETA 12:22:38 2020-11-27 03:53:56 [INFO] [TRAIN] epoch=8, iter=4900/40000, loss=0.3778, lr=0.008891, batch_cost=1.2671, reader_cost=0.0002 | ETA 12:21:16 2020-11-27 03:56:03 [INFO] [TRAIN] epoch=8, iter=5000/40000, loss=0.4108, lr=0.008868, batch_cost=1.2702, reader_cost=0.0003 | ETA 12:20:58 2020-11-27 03:58:10 [INFO] [TRAIN] epoch=8, iter=5100/40000, loss=0.3805, lr=0.008845, batch_cost=1.2653, reader_cost=0.0003 | ETA 12:16:00 2020-11-27 04:00:16 [INFO] [TRAIN] epoch=8, iter=5200/40000, loss=0.3458, lr=0.008822, batch_cost=1.2596, reader_cost=0.0004 | ETA 12:10:33 2020-11-27 04:02:23 [INFO] [TRAIN] epoch=9, iter=5300/40000, loss=0.3534, lr=0.008799, batch_cost=1.2721, reader_cost=0.0062 | ETA 12:15:40 2020-11-27 04:04:29 [INFO] [TRAIN] epoch=9, iter=5400/40000, loss=0.3822, lr=0.008777, batch_cost=1.2597, reader_cost=0.0004 | ETA 12:06:26 2020-11-27 04:06:35 [INFO] [TRAIN] epoch=9, iter=5500/40000, loss=0.3280, lr=0.008754, batch_cost=1.2600, reader_cost=0.0004 | ETA 12:04:28 2020-11-27 04:08:41 [INFO] [TRAIN] epoch=9, iter=5600/40000, loss=0.3792, lr=0.008731, batch_cost=1.2625, reader_cost=0.0004 | ETA 12:03:49 2020-11-27 04:10:47 [INFO] [TRAIN] epoch=9, iter=5700/40000, loss=0.3431, lr=0.008708, batch_cost=1.2606, reader_cost=0.0005 | ETA 12:00:39 2020-11-27 04:12:54 [INFO] [TRAIN] epoch=9, iter=5800/40000, loss=0.3300, lr=0.008685, batch_cost=1.2611, reader_cost=0.0002 | ETA 11:58:50 2020-11-27 04:15:00 [INFO] [TRAIN] epoch=9, iter=5900/40000, loss=0.2972, lr=0.008662, batch_cost=1.2604, reader_cost=0.0003 | ETA 11:56:19 2020-11-27 04:17:07 [INFO] [TRAIN] epoch=10, iter=6000/40000, loss=0.3388, lr=0.008639, batch_cost=1.2699, reader_cost=0.0061 | ETA 11:59:34 2020-11-27 04:19:14 [INFO] [TRAIN] epoch=10, iter=6100/40000, loss=0.2621, lr=0.008617, batch_cost=1.2729, reader_cost=0.0007 | ETA 11:59:12 2020-11-27 04:21:20 [INFO] [TRAIN] epoch=10, iter=6200/40000, loss=0.3394, lr=0.008594, batch_cost=1.2634, reader_cost=0.0004 | ETA 11:51:42 2020-11-27 04:23:27 [INFO] [TRAIN] epoch=10, iter=6300/40000, loss=0.3223, lr=0.008571, batch_cost=1.2658, reader_cost=0.0009 | ETA 11:50:58 2020-11-27 04:25:33 [INFO] [TRAIN] epoch=10, iter=6400/40000, loss=0.2911, lr=0.008548, batch_cost=1.2590, reader_cost=0.0003 | ETA 11:45:02 2020-11-27 04:27:40 [INFO] [TRAIN] epoch=10, iter=6500/40000, loss=0.3247, lr=0.008525, batch_cost=1.2713, reader_cost=0.0004 | ETA 11:49:47 2020-11-27 04:29:46 [INFO] [TRAIN] epoch=10, iter=6600/40000, loss=0.3384, lr=0.008502, batch_cost=1.2612, reader_cost=0.0007 | ETA 11:42:04 2020-11-27 04:31:53 [INFO] [TRAIN] epoch=11, iter=6700/40000, loss=0.3155, lr=0.008479, batch_cost=1.2657, reader_cost=0.0061 | ETA 11:42:29 2020-11-27 04:33:59 [INFO] [TRAIN] epoch=11, iter=6800/40000, loss=0.3428, lr=0.008456, batch_cost=1.2651, reader_cost=0.0004 | ETA 11:40:01 2020-11-27 04:36:05 [INFO] [TRAIN] epoch=11, iter=6900/40000, loss=0.2517, lr=0.008433, batch_cost=1.2611, reader_cost=0.0003 | ETA 11:35:42 2020-11-27 04:38:12 [INFO] [TRAIN] epoch=11, iter=7000/40000, loss=0.2778, lr=0.008410, batch_cost=1.2656, reader_cost=0.0007 | ETA 11:36:05 2020-11-27 04:40:18 [INFO] [TRAIN] epoch=11, iter=7100/40000, loss=0.2998, lr=0.008388, batch_cost=1.2601, reader_cost=0.0003 | ETA 11:30:58 2020-11-27 04:42:24 [INFO] [TRAIN] epoch=11, iter=7200/40000, loss=0.2911, lr=0.008365, batch_cost=1.2623, reader_cost=0.0004 | ETA 11:30:02 2020-11-27 04:44:31 [INFO] [TRAIN] epoch=12, iter=7300/40000, loss=0.2924, lr=0.008342, batch_cost=1.2666, reader_cost=0.0064 | ETA 11:30:19 2020-11-27 04:46:37 [INFO] [TRAIN] epoch=12, iter=7400/40000, loss=0.3328, lr=0.008319, batch_cost=1.2681, reader_cost=0.0002 | ETA 11:28:58 2020-11-27 04:48:45 [INFO] [TRAIN] epoch=12, iter=7500/40000, loss=0.2844, lr=0.008296, batch_cost=1.2710, reader_cost=0.0004 | ETA 11:28:26 2020-11-27 04:50:51 [INFO] [TRAIN] epoch=12, iter=7600/40000, loss=0.2716, lr=0.008273, batch_cost=1.2651, reader_cost=0.0003 | ETA 11:23:10 2020-11-27 04:52:58 [INFO] [TRAIN] epoch=12, iter=7700/40000, loss=0.2998, lr=0.008250, batch_cost=1.2726, reader_cost=0.0004 | ETA 11:25:04 2020-11-27 04:55:05 [INFO] [TRAIN] epoch=12, iter=7800/40000, loss=0.3348, lr=0.008227, batch_cost=1.2631, reader_cost=0.0004 | ETA 11:17:53 2020-11-27 04:57:10 [INFO] [TRAIN] epoch=12, iter=7900/40000, loss=0.2761, lr=0.008204, batch_cost=1.2549, reader_cost=0.0002 | ETA 11:11:21 2020-11-27 04:59:17 [INFO] [TRAIN] epoch=13, iter=8000/40000, loss=0.2831, lr=0.008181, batch_cost=1.2645, reader_cost=0.0062 | ETA 11:14:23 2020-11-27 04:59:17 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-27 04:59:50 [INFO] [EVAL] #Images=1449 mIoU=0.6543 Acc=0.9135 Kappa=0.8160 2020-11-27 04:59:50 [INFO] [EVAL] Class IoU: [0.9135 0.7788 0.3746 0.6676 0.5882 0.5575 0.8931 0.7322 0.8351 0.253 0.5568 0.5038 0.6644 0.7623 0.686 0.8015 0.4636 0.81 0.4631 0.8099 0.625 ] 2020-11-27 04:59:50 [INFO] [EVAL] Class Acc: [0.9673 0.8602 0.4006 0.6959 0.7607 0.8568 0.9371 0.7842 0.9058 0.3018 0.9335 0.7281 0.6939 0.8321 0.918 0.8795 0.52 0.8864 0.7736 0.8843 0.6748] 2020-11-27 04:59:59 [INFO] [EVAL] The model with the best validation mIoU (0.6543) was saved at iter 8000. 2020-11-27 05:02:06 [INFO] [TRAIN] epoch=13, iter=8100/40000, loss=0.2678, lr=0.008158, batch_cost=1.2663, reader_cost=0.0007 | ETA 11:13:13 2020-11-27 05:04:12 [INFO] [TRAIN] epoch=13, iter=8200/40000, loss=0.2866, lr=0.008135, batch_cost=1.2583, reader_cost=0.0002 | ETA 11:06:53 2020-11-27 05:06:18 [INFO] [TRAIN] epoch=13, iter=8300/40000, loss=0.2414, lr=0.008112, batch_cost=1.2663, reader_cost=0.0004 | ETA 11:09:03 2020-11-27 05:08:25 [INFO] [TRAIN] epoch=13, iter=8400/40000, loss=0.2581, lr=0.008089, batch_cost=1.2701, reader_cost=0.0004 | ETA 11:08:54 2020-11-27 05:10:32 [INFO] [TRAIN] epoch=13, iter=8500/40000, loss=0.2412, lr=0.008066, batch_cost=1.2628, reader_cost=0.0003 | ETA 11:02:58 2020-11-27 05:12:39 [INFO] [TRAIN] epoch=14, iter=8600/40000, loss=0.2722, lr=0.008043, batch_cost=1.2719, reader_cost=0.0066 | ETA 11:05:36 2020-11-27 05:14:46 [INFO] [TRAIN] epoch=14, iter=8700/40000, loss=0.2159, lr=0.008020, batch_cost=1.2677, reader_cost=0.0006 | ETA 11:01:20 2020-11-27 05:16:52 [INFO] [TRAIN] epoch=14, iter=8800/40000, loss=0.2585, lr=0.007996, batch_cost=1.2685, reader_cost=0.0004 | ETA 10:59:38 2020-11-27 05:18:59 [INFO] [TRAIN] epoch=14, iter=8900/40000, loss=0.2449, lr=0.007973, batch_cost=1.2704, reader_cost=0.0006 | ETA 10:58:29 2020-11-27 05:21:05 [INFO] [TRAIN] epoch=14, iter=9000/40000, loss=0.2788, lr=0.007950, batch_cost=1.2585, reader_cost=0.0003 | ETA 10:50:14 2020-11-27 05:23:11 [INFO] [TRAIN] epoch=14, iter=9100/40000, loss=0.2284, lr=0.007927, batch_cost=1.2615, reader_cost=0.0004 | ETA 10:49:40 2020-11-27 05:25:18 [INFO] [TRAIN] epoch=14, iter=9200/40000, loss=0.2544, lr=0.007904, batch_cost=1.2674, reader_cost=0.0005 | ETA 10:50:36 2020-11-27 05:27:25 [INFO] [TRAIN] epoch=15, iter=9300/40000, loss=0.2813, lr=0.007881, batch_cost=1.2725, reader_cost=0.0062 | ETA 10:51:07 2020-11-27 05:29:33 [INFO] [TRAIN] epoch=15, iter=9400/40000, loss=0.2466, lr=0.007858, batch_cost=1.2704, reader_cost=0.0006 | ETA 10:47:53 2020-11-27 05:31:39 [INFO] [TRAIN] epoch=15, iter=9500/40000, loss=0.2483, lr=0.007835, batch_cost=1.2667, reader_cost=0.0008 | ETA 10:43:55 2020-11-27 05:33:46 [INFO] [TRAIN] epoch=15, iter=9600/40000, loss=0.2795, lr=0.007812, batch_cost=1.2675, reader_cost=0.0007 | ETA 10:42:11 2020-11-27 05:35:52 [INFO] [TRAIN] epoch=15, iter=9700/40000, loss=0.2557, lr=0.007789, batch_cost=1.2639, reader_cost=0.0003 | ETA 10:38:16 2020-11-27 05:37:59 [INFO] [TRAIN] epoch=15, iter=9800/40000, loss=0.2885, lr=0.007765, batch_cost=1.2618, reader_cost=0.0006 | ETA 10:35:06 2020-11-27 05:40:05 [INFO] [TRAIN] epoch=15, iter=9900/40000, loss=0.3006, lr=0.007742, batch_cost=1.2605, reader_cost=0.0003 | ETA 10:32:22 2020-11-27 05:42:11 [INFO] [TRAIN] epoch=16, iter=10000/40000, loss=0.2710, lr=0.007719, batch_cost=1.2691, reader_cost=0.0062 | ETA 10:34:32 2020-11-27 05:44:18 [INFO] [TRAIN] epoch=16, iter=10100/40000, loss=0.2411, lr=0.007696, batch_cost=1.2655, reader_cost=0.0005 | ETA 10:30:38 2020-11-27 05:46:25 [INFO] [TRAIN] epoch=16, iter=10200/40000, loss=0.2722, lr=0.007673, batch_cost=1.2674, reader_cost=0.0004 | ETA 10:29:28 2020-11-27 05:48:30 [INFO] [TRAIN] epoch=16, iter=10300/40000, loss=0.2254, lr=0.007650, batch_cost=1.2552, reader_cost=0.0005 | ETA 10:21:18 2020-11-27 05:50:37 [INFO] [TRAIN] epoch=16, iter=10400/40000, loss=0.2341, lr=0.007626, batch_cost=1.2641, reader_cost=0.0008 | ETA 10:23:37 2020-11-27 05:52:43 [INFO] [TRAIN] epoch=16, iter=10500/40000, loss=0.2136, lr=0.007603, batch_cost=1.2662, reader_cost=0.0003 | ETA 10:22:33 2020-11-27 05:54:50 [INFO] [TRAIN] epoch=17, iter=10600/40000, loss=0.2371, lr=0.007580, batch_cost=1.2714, reader_cost=0.0066 | ETA 10:22:58 2020-11-27 05:56:57 [INFO] [TRAIN] epoch=17, iter=10700/40000, loss=0.2291, lr=0.007557, batch_cost=1.2658, reader_cost=0.0008 | ETA 10:18:07 2020-11-27 05:59:04 [INFO] [TRAIN] epoch=17, iter=10800/40000, loss=0.2395, lr=0.007534, batch_cost=1.2660, reader_cost=0.0007 | ETA 10:16:07 2020-11-27 06:01:10 [INFO] [TRAIN] epoch=17, iter=10900/40000, loss=0.2276, lr=0.007510, batch_cost=1.2668, reader_cost=0.0003 | ETA 10:14:25 2020-11-27 06:03:17 [INFO] [TRAIN] epoch=17, iter=11000/40000, loss=0.2300, lr=0.007487, batch_cost=1.2677, reader_cost=0.0007 | ETA 10:12:42 2020-11-27 06:05:24 [INFO] [TRAIN] epoch=17, iter=11100/40000, loss=0.2482, lr=0.007464, batch_cost=1.2661, reader_cost=0.0007 | ETA 10:09:49 2020-11-27 06:07:30 [INFO] [TRAIN] epoch=17, iter=11200/40000, loss=0.2034, lr=0.007441, batch_cost=1.2668, reader_cost=0.0009 | ETA 10:08:02 2020-11-27 06:09:38 [INFO] [TRAIN] epoch=18, iter=11300/40000, loss=0.2430, lr=0.007417, batch_cost=1.2756, reader_cost=0.0067 | ETA 10:10:09 2020-11-27 06:11:45 [INFO] [TRAIN] epoch=18, iter=11400/40000, loss=0.2387, lr=0.007394, batch_cost=1.2679, reader_cost=0.0007 | ETA 10:04:21 2020-11-27 06:13:50 [INFO] [TRAIN] epoch=18, iter=11500/40000, loss=0.2185, lr=0.007371, batch_cost=1.2555, reader_cost=0.0007 | ETA 09:56:22 2020-11-27 06:15:57 [INFO] [TRAIN] epoch=18, iter=11600/40000, loss=0.2181, lr=0.007348, batch_cost=1.2639, reader_cost=0.0007 | ETA 09:58:14 2020-11-27 06:18:04 [INFO] [TRAIN] epoch=18, iter=11700/40000, loss=0.2211, lr=0.007324, batch_cost=1.2691, reader_cost=0.0006 | ETA 09:58:35 2020-11-27 06:20:11 [INFO] [TRAIN] epoch=18, iter=11800/40000, loss=0.2251, lr=0.007301, batch_cost=1.2715, reader_cost=0.0004 | ETA 09:57:35 2020-11-27 06:22:18 [INFO] [TRAIN] epoch=19, iter=11900/40000, loss=0.2315, lr=0.007278, batch_cost=1.2755, reader_cost=0.0070 | ETA 09:57:22 2020-11-27 06:24:25 [INFO] [TRAIN] epoch=19, iter=12000/40000, loss=0.2273, lr=0.007254, batch_cost=1.2668, reader_cost=0.0006 | ETA 09:51:09 2020-11-27 06:24:25 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-27 06:24:59 [INFO] [EVAL] #Images=1449 mIoU=0.7441 Acc=0.9406 Kappa=0.8694 2020-11-27 06:24:59 [INFO] [EVAL] Class IoU: [0.9359 0.8916 0.4288 0.8515 0.631 0.7362 0.9009 0.8629 0.865 0.3811 0.789 0.5257 0.7891 0.8385 0.8546 0.8336 0.5441 0.8805 0.5592 0.8581 0.6694] 2020-11-27 06:24:59 [INFO] [EVAL] Class Acc: [0.9649 0.9304 0.451 0.9253 0.701 0.8766 0.9433 0.9466 0.9108 0.5702 0.895 0.8343 0.8857 0.8815 0.9217 0.9282 0.8698 0.9307 0.6942 0.9325 0.8665] 2020-11-27 06:25:08 [INFO] [EVAL] The model with the best validation mIoU (0.7441) was saved at iter 12000. 2020-11-27 06:27:15 [INFO] [TRAIN] epoch=19, iter=12100/40000, loss=0.2350, lr=0.007231, batch_cost=1.2695, reader_cost=0.0005 | ETA 09:50:19 2020-11-27 06:29:22 [INFO] [TRAIN] epoch=19, iter=12200/40000, loss=0.2157, lr=0.007208, batch_cost=1.2673, reader_cost=0.0006 | ETA 09:47:09 2020-11-27 06:31:28 [INFO] [TRAIN] epoch=19, iter=12300/40000, loss=0.2208, lr=0.007184, batch_cost=1.2649, reader_cost=0.0006 | ETA 09:43:57 2020-11-27 06:33:35 [INFO] [TRAIN] epoch=19, iter=12400/40000, loss=0.2072, lr=0.007161, batch_cost=1.2701, reader_cost=0.0004 | ETA 09:44:14 2020-11-27 06:35:42 [INFO] [TRAIN] epoch=19, iter=12500/40000, loss=0.1978, lr=0.007138, batch_cost=1.2668, reader_cost=0.0003 | ETA 09:40:36 2020-11-27 06:37:49 [INFO] [TRAIN] epoch=20, iter=12600/40000, loss=0.2223, lr=0.007114, batch_cost=1.2695, reader_cost=0.0068 | ETA 09:39:44 2020-11-27 06:39:55 [INFO] [TRAIN] epoch=20, iter=12700/40000, loss=0.1946, lr=0.007091, batch_cost=1.2611, reader_cost=0.0002 | ETA 09:33:46 2020-11-27 06:42:03 [INFO] [TRAIN] epoch=20, iter=12800/40000, loss=0.2166, lr=0.007068, batch_cost=1.2753, reader_cost=0.0002 | ETA 09:38:08 2020-11-27 06:44:09 [INFO] [TRAIN] epoch=20, iter=12900/40000, loss=0.2020, lr=0.007044, batch_cost=1.2681, reader_cost=0.0003 | ETA 09:32:45 2020-11-27 06:46:16 [INFO] [TRAIN] epoch=20, iter=13000/40000, loss=0.1701, lr=0.007021, batch_cost=1.2648, reader_cost=0.0002 | ETA 09:29:10 2020-11-27 06:48:23 [INFO] [TRAIN] epoch=20, iter=13100/40000, loss=0.1767, lr=0.006997, batch_cost=1.2675, reader_cost=0.0004 | ETA 09:28:16 2020-11-27 06:50:29 [INFO] [TRAIN] epoch=20, iter=13200/40000, loss=0.1670, lr=0.006974, batch_cost=1.2668, reader_cost=0.0002 | ETA 09:25:49 2020-11-27 06:52:38 [INFO] [TRAIN] epoch=21, iter=13300/40000, loss=0.1710, lr=0.006951, batch_cost=1.2831, reader_cost=0.0072 | ETA 09:30:58 2020-11-27 06:54:45 [INFO] [TRAIN] epoch=21, iter=13400/40000, loss=0.2089, lr=0.006927, batch_cost=1.2694, reader_cost=0.0004 | ETA 09:22:45 2020-11-27 06:56:51 [INFO] [TRAIN] epoch=21, iter=13500/40000, loss=0.1965, lr=0.006904, batch_cost=1.2661, reader_cost=0.0006 | ETA 09:19:10 2020-11-27 06:58:58 [INFO] [TRAIN] epoch=21, iter=13600/40000, loss=0.1759, lr=0.006880, batch_cost=1.2682, reader_cost=0.0004 | ETA 09:18:01 2020-11-27 07:01:05 [INFO] [TRAIN] epoch=21, iter=13700/40000, loss=0.1670, lr=0.006857, batch_cost=1.2650, reader_cost=0.0004 | ETA 09:14:28 2020-11-27 07:03:11 [INFO] [TRAIN] epoch=21, iter=13800/40000, loss=0.2417, lr=0.006833, batch_cost=1.2601, reader_cost=0.0005 | ETA 09:10:13 2020-11-27 07:05:17 [INFO] [TRAIN] epoch=22, iter=13900/40000, loss=0.2366, lr=0.006810, batch_cost=1.2595, reader_cost=0.0065 | ETA 09:07:52 2020-11-27 07:07:23 [INFO] [TRAIN] epoch=22, iter=14000/40000, loss=0.2116, lr=0.006786, batch_cost=1.2598, reader_cost=0.0005 | ETA 09:05:55 2020-11-27 07:09:29 [INFO] [TRAIN] epoch=22, iter=14100/40000, loss=0.2000, lr=0.006763, batch_cost=1.2674, reader_cost=0.0003 | ETA 09:07:05 2020-11-27 07:11:37 [INFO] [TRAIN] epoch=22, iter=14200/40000, loss=0.2151, lr=0.006739, batch_cost=1.2788, reader_cost=0.0005 | ETA 09:09:51 2020-11-27 07:13:44 [INFO] [TRAIN] epoch=22, iter=14300/40000, loss=0.1718, lr=0.006716, batch_cost=1.2688, reader_cost=0.0002 | ETA 09:03:27 2020-11-27 07:15:51 [INFO] [TRAIN] epoch=22, iter=14400/40000, loss=0.1810, lr=0.006692, batch_cost=1.2683, reader_cost=0.0004 | ETA 09:01:08 2020-11-27 07:17:58 [INFO] [TRAIN] epoch=22, iter=14500/40000, loss=0.2097, lr=0.006669, batch_cost=1.2695, reader_cost=0.0006 | ETA 08:59:32 2020-11-27 07:20:05 [INFO] [TRAIN] epoch=23, iter=14600/40000, loss=0.1562, lr=0.006645, batch_cost=1.2759, reader_cost=0.0068 | ETA 09:00:07 2020-11-27 07:22:13 [INFO] [TRAIN] epoch=23, iter=14700/40000, loss=0.1852, lr=0.006622, batch_cost=1.2710, reader_cost=0.0006 | ETA 08:55:57 2020-11-27 07:24:19 [INFO] [TRAIN] epoch=23, iter=14800/40000, loss=0.2050, lr=0.006598, batch_cost=1.2671, reader_cost=0.0006 | ETA 08:52:11 2020-11-27 07:26:26 [INFO] [TRAIN] epoch=23, iter=14900/40000, loss=0.1832, lr=0.006575, batch_cost=1.2632, reader_cost=0.0005 | ETA 08:48:25 2020-11-27 07:28:33 [INFO] [TRAIN] epoch=23, iter=15000/40000, loss=0.1766, lr=0.006551, batch_cost=1.2732, reader_cost=0.0005 | ETA 08:50:30 2020-11-27 07:30:39 [INFO] [TRAIN] epoch=23, iter=15100/40000, loss=0.1800, lr=0.006527, batch_cost=1.2568, reader_cost=0.0006 | ETA 08:41:33 2020-11-27 07:32:45 [INFO] [TRAIN] epoch=23, iter=15200/40000, loss=0.2097, lr=0.006504, batch_cost=1.2609, reader_cost=0.0007 | ETA 08:41:09 2020-11-27 07:34:52 [INFO] [TRAIN] epoch=24, iter=15300/40000, loss=0.1846, lr=0.006480, batch_cost=1.2743, reader_cost=0.0063 | ETA 08:44:35 2020-11-27 07:36:59 [INFO] [TRAIN] epoch=24, iter=15400/40000, loss=0.1779, lr=0.006457, batch_cost=1.2648, reader_cost=0.0002 | ETA 08:38:33 2020-11-27 07:39:05 [INFO] [TRAIN] epoch=24, iter=15500/40000, loss=0.1495, lr=0.006433, batch_cost=1.2670, reader_cost=0.0004 | ETA 08:37:22 2020-11-27 07:41:12 [INFO] [TRAIN] epoch=24, iter=15600/40000, loss=0.1803, lr=0.006409, batch_cost=1.2661, reader_cost=0.0003 | ETA 08:34:53 2020-11-27 07:43:19 [INFO] [TRAIN] epoch=24, iter=15700/40000, loss=0.1643, lr=0.006386, batch_cost=1.2667, reader_cost=0.0003 | ETA 08:32:59 2020-11-27 07:45:25 [INFO] [TRAIN] epoch=24, iter=15800/40000, loss=0.1548, lr=0.006362, batch_cost=1.2683, reader_cost=0.0007 | ETA 08:31:32 2020-11-27 07:47:33 [INFO] [TRAIN] epoch=25, iter=15900/40000, loss=0.1641, lr=0.006338, batch_cost=1.2723, reader_cost=0.0067 | ETA 08:31:02 2020-11-27 07:49:39 [INFO] [TRAIN] epoch=25, iter=16000/40000, loss=0.1891, lr=0.006315, batch_cost=1.2635, reader_cost=0.0003 | ETA 08:25:24 2020-11-27 07:49:39 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-27 07:50:12 [INFO] [EVAL] #Images=1449 mIoU=0.7503 Acc=0.9425 Kappa=0.8743 2020-11-27 07:50:12 [INFO] [EVAL] Class IoU: [0.9389 0.8636 0.4457 0.8617 0.6185 0.8107 0.905 0.8009 0.8816 0.4016 0.8559 0.6119 0.8267 0.8259 0.8309 0.8272 0.5578 0.8964 0.4325 0.8369 0.7259] 2020-11-27 07:50:12 [INFO] [EVAL] Class Acc: [0.9693 0.8927 0.4694 0.9087 0.6968 0.9156 0.9674 0.8503 0.91 0.5831 0.9276 0.8506 0.9266 0.8664 0.899 0.8763 0.7849 0.9604 0.8155 0.8916 0.904 ] 2020-11-27 07:50:21 [INFO] [EVAL] The model with the best validation mIoU (0.7503) was saved at iter 16000. 2020-11-27 07:52:28 [INFO] [TRAIN] epoch=25, iter=16100/40000, loss=0.1538, lr=0.006291, batch_cost=1.2671, reader_cost=0.0005 | ETA 08:24:43 2020-11-27 07:54:35 [INFO] [TRAIN] epoch=25, iter=16200/40000, loss=0.1595, lr=0.006267, batch_cost=1.2688, reader_cost=0.0003 | ETA 08:23:17 2020-11-27 07:56:39 [INFO] [TRAIN] epoch=25, iter=16300/40000, loss=0.1678, lr=0.006244, batch_cost=1.2489, reader_cost=0.0003 | ETA 08:13:17 2020-11-27 07:58:45 [INFO] [TRAIN] epoch=25, iter=16400/40000, loss=0.1830, lr=0.006220, batch_cost=1.2588, reader_cost=0.0003 | ETA 08:15:08 2020-11-27 08:00:52 [INFO] [TRAIN] epoch=25, iter=16500/40000, loss=0.1676, lr=0.006196, batch_cost=1.2666, reader_cost=0.0003 | ETA 08:16:04 2020-11-27 08:02:59 [INFO] [TRAIN] epoch=26, iter=16600/40000, loss=0.1561, lr=0.006172, batch_cost=1.2715, reader_cost=0.0063 | ETA 08:15:53 2020-11-27 08:05:06 [INFO] [TRAIN] epoch=26, iter=16700/40000, loss=0.1435, lr=0.006149, batch_cost=1.2731, reader_cost=0.0005 | ETA 08:14:23 2020-11-27 08:07:13 [INFO] [TRAIN] epoch=26, iter=16800/40000, loss=0.1425, lr=0.006125, batch_cost=1.2654, reader_cost=0.0004 | ETA 08:09:16 2020-11-27 08:09:20 [INFO] [TRAIN] epoch=26, iter=16900/40000, loss=0.1857, lr=0.006101, batch_cost=1.2658, reader_cost=0.0005 | ETA 08:07:19 2020-11-27 08:11:26 [INFO] [TRAIN] epoch=26, iter=17000/40000, loss=0.1619, lr=0.006077, batch_cost=1.2683, reader_cost=0.0004 | ETA 08:06:11 2020-11-27 08:13:33 [INFO] [TRAIN] epoch=26, iter=17100/40000, loss=0.1484, lr=0.006054, batch_cost=1.2647, reader_cost=0.0005 | ETA 08:02:40 2020-11-27 08:15:40 [INFO] [TRAIN] epoch=27, iter=17200/40000, loss=0.1458, lr=0.006030, batch_cost=1.2672, reader_cost=0.0069 | ETA 08:01:31 2020-11-27 08:17:46 [INFO] [TRAIN] epoch=27, iter=17300/40000, loss=0.1845, lr=0.006006, batch_cost=1.2611, reader_cost=0.0002 | ETA 07:57:07 2020-11-27 08:19:52 [INFO] [TRAIN] epoch=27, iter=17400/40000, loss=0.1498, lr=0.005982, batch_cost=1.2652, reader_cost=0.0006 | ETA 07:56:32 2020-11-27 08:21:59 [INFO] [TRAIN] epoch=27, iter=17500/40000, loss=0.1533, lr=0.005958, batch_cost=1.2653, reader_cost=0.0004 | ETA 07:54:29 2020-11-27 08:24:04 [INFO] [TRAIN] epoch=27, iter=17600/40000, loss=0.1611, lr=0.005935, batch_cost=1.2555, reader_cost=0.0004 | ETA 07:48:43 2020-11-27 08:26:11 [INFO] [TRAIN] epoch=27, iter=17700/40000, loss=0.1475, lr=0.005911, batch_cost=1.2650, reader_cost=0.0004 | ETA 07:50:10 2020-11-27 08:28:17 [INFO] [TRAIN] epoch=27, iter=17800/40000, loss=0.1381, lr=0.005887, batch_cost=1.2658, reader_cost=0.0006 | ETA 07:48:20 2020-11-27 08:30:25 [INFO] [TRAIN] epoch=28, iter=17900/40000, loss=0.1388, lr=0.005863, batch_cost=1.2743, reader_cost=0.0076 | ETA 07:49:21 2020-11-27 08:32:32 [INFO] [TRAIN] epoch=28, iter=18000/40000, loss=0.1483, lr=0.005839, batch_cost=1.2699, reader_cost=0.0007 | ETA 07:45:37 2020-11-27 08:34:38 [INFO] [TRAIN] epoch=28, iter=18100/40000, loss=0.1456, lr=0.005815, batch_cost=1.2623, reader_cost=0.0003 | ETA 07:40:44 2020-11-27 08:36:45 [INFO] [TRAIN] epoch=28, iter=18200/40000, loss=0.1785, lr=0.005791, batch_cost=1.2656, reader_cost=0.0004 | ETA 07:39:49 2020-11-27 08:38:51 [INFO] [TRAIN] epoch=28, iter=18300/40000, loss=0.1784, lr=0.005767, batch_cost=1.2641, reader_cost=0.0003 | ETA 07:37:11 2020-11-27 08:40:58 [INFO] [TRAIN] epoch=28, iter=18400/40000, loss=0.1417, lr=0.005743, batch_cost=1.2674, reader_cost=0.0007 | ETA 07:36:15 2020-11-27 08:43:05 [INFO] [TRAIN] epoch=28, iter=18500/40000, loss=0.1723, lr=0.005720, batch_cost=1.2728, reader_cost=0.0006 | ETA 07:36:05 2020-11-27 08:45:12 [INFO] [TRAIN] epoch=29, iter=18600/40000, loss=0.1745, lr=0.005696, batch_cost=1.2703, reader_cost=0.0064 | ETA 07:33:04 2020-11-27 08:47:17 [INFO] [TRAIN] epoch=29, iter=18700/40000, loss=0.1598, lr=0.005672, batch_cost=1.2512, reader_cost=0.0002 | ETA 07:24:10 2020-11-27 08:49:23 [INFO] [TRAIN] epoch=29, iter=18800/40000, loss=0.1478, lr=0.005648, batch_cost=1.2551, reader_cost=0.0002 | ETA 07:23:27 2020-11-27 08:51:29 [INFO] [TRAIN] epoch=29, iter=18900/40000, loss=0.1696, lr=0.005624, batch_cost=1.2658, reader_cost=0.0002 | ETA 07:25:08 2020-11-27 08:53:35 [INFO] [TRAIN] epoch=29, iter=19000/40000, loss=0.1499, lr=0.005600, batch_cost=1.2575, reader_cost=0.0004 | ETA 07:20:08 2020-11-27 08:55:41 [INFO] [TRAIN] epoch=29, iter=19100/40000, loss=0.1295, lr=0.005576, batch_cost=1.2591, reader_cost=0.0003 | ETA 07:18:36 2020-11-27 08:57:48 [INFO] [TRAIN] epoch=30, iter=19200/40000, loss=0.1863, lr=0.005552, batch_cost=1.2735, reader_cost=0.0071 | ETA 07:21:29 2020-11-27 08:59:54 [INFO] [TRAIN] epoch=30, iter=19300/40000, loss=0.1217, lr=0.005528, batch_cost=1.2593, reader_cost=0.0004 | ETA 07:14:27 2020-11-27 09:02:01 [INFO] [TRAIN] epoch=30, iter=19400/40000, loss=0.1318, lr=0.005504, batch_cost=1.2645, reader_cost=0.0004 | ETA 07:14:08 2020-11-27 09:04:08 [INFO] [TRAIN] epoch=30, iter=19500/40000, loss=0.1285, lr=0.005480, batch_cost=1.2735, reader_cost=0.0006 | ETA 07:15:06 2020-11-27 09:06:14 [INFO] [TRAIN] epoch=30, iter=19600/40000, loss=0.1329, lr=0.005455, batch_cost=1.2631, reader_cost=0.0003 | ETA 07:09:27 2020-11-27 09:08:21 [INFO] [TRAIN] epoch=30, iter=19700/40000, loss=0.1571, lr=0.005431, batch_cost=1.2661, reader_cost=0.0004 | ETA 07:08:21 2020-11-27 09:10:28 [INFO] [TRAIN] epoch=30, iter=19800/40000, loss=0.1756, lr=0.005407, batch_cost=1.2653, reader_cost=0.0004 | ETA 07:05:58 2020-11-27 09:12:35 [INFO] [TRAIN] epoch=31, iter=19900/40000, loss=0.1590, lr=0.005383, batch_cost=1.2744, reader_cost=0.0067 | ETA 07:06:54 2020-11-27 09:14:40 [INFO] [TRAIN] epoch=31, iter=20000/40000, loss=0.1381, lr=0.005359, batch_cost=1.2521, reader_cost=0.0002 | ETA 06:57:22 2020-11-27 09:14:40 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-27 09:15:09 [INFO] [EVAL] #Images=1449 mIoU=0.7715 Acc=0.9488 Kappa=0.8869 2020-11-27 09:15:09 [INFO] [EVAL] Class IoU: [0.9445 0.8818 0.4104 0.8553 0.608 0.7934 0.9262 0.8776 0.8851 0.4217 0.8985 0.5922 0.8374 0.8904 0.8854 0.8594 0.6215 0.8803 0.5127 0.8963 0.7238] 2020-11-27 09:15:09 [INFO] [EVAL] Class Acc: [0.9675 0.9199 0.4235 0.901 0.72 0.8821 0.9754 0.9616 0.9276 0.5682 0.954 0.9063 0.9485 0.9585 0.9433 0.9255 0.7824 0.9365 0.8162 0.9417 0.9358] 2020-11-27 09:15:18 [INFO] [EVAL] The model with the best validation mIoU (0.7715) was saved at iter 20000. 2020-11-27 09:17:23 [INFO] [TRAIN] epoch=31, iter=20100/40000, loss=0.1314, lr=0.005335, batch_cost=1.2499, reader_cost=0.0003 | ETA 06:54:32 2020-11-27 09:19:28 [INFO] [TRAIN] epoch=31, iter=20200/40000, loss=0.1374, lr=0.005311, batch_cost=1.2500, reader_cost=0.0003 | ETA 06:52:29 2020-11-27 09:21:33 [INFO] [TRAIN] epoch=31, iter=20300/40000, loss=0.1305, lr=0.005287, batch_cost=1.2513, reader_cost=0.0002 | ETA 06:50:49 2020-11-27 09:23:39 [INFO] [TRAIN] epoch=31, iter=20400/40000, loss=0.1641, lr=0.005263, batch_cost=1.2558, reader_cost=0.0002 | ETA 06:50:13 2020-11-27 09:25:44 [INFO] [TRAIN] epoch=32, iter=20500/40000, loss=0.1347, lr=0.005238, batch_cost=1.2559, reader_cost=0.0064 | ETA 06:48:10 2020-11-27 09:27:50 [INFO] [TRAIN] epoch=32, iter=20600/40000, loss=0.1517, lr=0.005214, batch_cost=1.2593, reader_cost=0.0001 | ETA 06:47:11 2020-11-27 09:29:56 [INFO] [TRAIN] epoch=32, iter=20700/40000, loss=0.1593, lr=0.005190, batch_cost=1.2570, reader_cost=0.0002 | ETA 06:44:20 2020-11-27 09:32:02 [INFO] [TRAIN] epoch=32, iter=20800/40000, loss=0.1523, lr=0.005166, batch_cost=1.2647, reader_cost=0.0004 | ETA 06:44:42 2020-11-27 09:34:08 [INFO] [TRAIN] epoch=32, iter=20900/40000, loss=0.1364, lr=0.005142, batch_cost=1.2609, reader_cost=0.0002 | ETA 06:41:23 2020-11-27 09:36:14 [INFO] [TRAIN] epoch=32, iter=21000/40000, loss=0.1201, lr=0.005117, batch_cost=1.2598, reader_cost=0.0003 | ETA 06:38:56 2020-11-27 09:38:20 [INFO] [TRAIN] epoch=32, iter=21100/40000, loss=0.1486, lr=0.005093, batch_cost=1.2583, reader_cost=0.0002 | ETA 06:36:21 2020-11-27 09:40:27 [INFO] [TRAIN] epoch=33, iter=21200/40000, loss=0.1366, lr=0.005069, batch_cost=1.2662, reader_cost=0.0066 | ETA 06:36:44 2020-11-27 09:42:32 [INFO] [TRAIN] epoch=33, iter=21300/40000, loss=0.1640, lr=0.005045, batch_cost=1.2502, reader_cost=0.0002 | ETA 06:29:38 2020-11-27 09:44:38 [INFO] [TRAIN] epoch=33, iter=21400/40000, loss=0.1320, lr=0.005020, batch_cost=1.2616, reader_cost=0.0002 | ETA 06:31:06 2020-11-27 09:46:45 [INFO] [TRAIN] epoch=33, iter=21500/40000, loss=0.1443, lr=0.004996, batch_cost=1.2672, reader_cost=0.0002 | ETA 06:30:42 2020-11-27 09:48:51 [INFO] [TRAIN] epoch=33, iter=21600/40000, loss=0.1237, lr=0.004972, batch_cost=1.2617, reader_cost=0.0002 | ETA 06:26:55 2020-11-27 09:50:57 [INFO] [TRAIN] epoch=33, iter=21700/40000, loss=0.1287, lr=0.004947, batch_cost=1.2623, reader_cost=0.0002 | ETA 06:24:59 2020-11-27 09:53:04 [INFO] [TRAIN] epoch=33, iter=21800/40000, loss=0.1372, lr=0.004923, batch_cost=1.2676, reader_cost=0.0002 | ETA 06:24:30 2020-11-27 09:55:10 [INFO] [TRAIN] epoch=34, iter=21900/40000, loss=0.1325, lr=0.004899, batch_cost=1.2607, reader_cost=0.0065 | ETA 06:20:19 2020-11-27 09:57:15 [INFO] [TRAIN] epoch=34, iter=22000/40000, loss=0.1236, lr=0.004874, batch_cost=1.2559, reader_cost=0.0002 | ETA 06:16:45 2020-11-27 09:59:21 [INFO] [TRAIN] epoch=34, iter=22100/40000, loss=0.1310, lr=0.004850, batch_cost=1.2507, reader_cost=0.0004 | ETA 06:13:08 2020-11-27 10:01:26 [INFO] [TRAIN] epoch=34, iter=22200/40000, loss=0.1435, lr=0.004826, batch_cost=1.2587, reader_cost=0.0007 | ETA 06:13:24 2020-11-27 10:03:32 [INFO] [TRAIN] epoch=34, iter=22300/40000, loss=0.1223, lr=0.004801, batch_cost=1.2541, reader_cost=0.0004 | ETA 06:09:56 2020-11-27 10:05:38 [INFO] [TRAIN] epoch=34, iter=22400/40000, loss=0.1317, lr=0.004777, batch_cost=1.2621, reader_cost=0.0005 | ETA 06:10:12 2020-11-27 10:07:44 [INFO] [TRAIN] epoch=35, iter=22500/40000, loss=0.1360, lr=0.004752, batch_cost=1.2640, reader_cost=0.0075 | ETA 06:08:39 2020-11-27 10:09:50 [INFO] [TRAIN] epoch=35, iter=22600/40000, loss=0.1281, lr=0.004728, batch_cost=1.2539, reader_cost=0.0002 | ETA 06:03:38 2020-11-27 10:11:55 [INFO] [TRAIN] epoch=35, iter=22700/40000, loss=0.1040, lr=0.004703, batch_cost=1.2526, reader_cost=0.0003 | ETA 06:01:10 2020-11-27 10:14:00 [INFO] [TRAIN] epoch=35, iter=22800/40000, loss=0.1114, lr=0.004679, batch_cost=1.2515, reader_cost=0.0002 | ETA 05:58:45 2020-11-27 10:16:06 [INFO] [TRAIN] epoch=35, iter=22900/40000, loss=0.1190, lr=0.004654, batch_cost=1.2541, reader_cost=0.0004 | ETA 05:57:24 2020-11-27 10:18:11 [INFO] [TRAIN] epoch=35, iter=23000/40000, loss=0.1322, lr=0.004630, batch_cost=1.2563, reader_cost=0.0002 | ETA 05:55:56 2020-11-27 10:20:17 [INFO] [TRAIN] epoch=35, iter=23100/40000, loss=0.1189, lr=0.004605, batch_cost=1.2582, reader_cost=0.0002 | ETA 05:54:22 2020-11-27 10:22:23 [INFO] [TRAIN] epoch=36, iter=23200/40000, loss=0.1055, lr=0.004581, batch_cost=1.2595, reader_cost=0.0056 | ETA 05:52:38 2020-11-27 10:24:29 [INFO] [TRAIN] epoch=36, iter=23300/40000, loss=0.1196, lr=0.004556, batch_cost=1.2550, reader_cost=0.0002 | ETA 05:49:18 2020-11-27 10:26:34 [INFO] [TRAIN] epoch=36, iter=23400/40000, loss=0.1073, lr=0.004532, batch_cost=1.2549, reader_cost=0.0004 | ETA 05:47:12 2020-11-27 10:28:39 [INFO] [TRAIN] epoch=36, iter=23500/40000, loss=0.1120, lr=0.004507, batch_cost=1.2520, reader_cost=0.0003 | ETA 05:44:17 2020-11-27 10:30:45 [INFO] [TRAIN] epoch=36, iter=23600/40000, loss=0.1101, lr=0.004483, batch_cost=1.2539, reader_cost=0.0002 | ETA 05:42:43 2020-11-27 10:32:51 [INFO] [TRAIN] epoch=36, iter=23700/40000, loss=0.1128, lr=0.004458, batch_cost=1.2628, reader_cost=0.0006 | ETA 05:43:03 2020-11-27 10:34:57 [INFO] [TRAIN] epoch=37, iter=23800/40000, loss=0.1213, lr=0.004433, batch_cost=1.2658, reader_cost=0.0065 | ETA 05:41:45 2020-11-27 10:37:03 [INFO] [TRAIN] epoch=37, iter=23900/40000, loss=0.1310, lr=0.004409, batch_cost=1.2571, reader_cost=0.0002 | ETA 05:37:18 2020-11-27 10:39:08 [INFO] [TRAIN] epoch=37, iter=24000/40000, loss=0.1218, lr=0.004384, batch_cost=1.2480, reader_cost=0.0003 | ETA 05:32:48 2020-11-27 10:39:08 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-27 10:39:38 [INFO] [EVAL] #Images=1449 mIoU=0.7781 Acc=0.9493 Kappa=0.8890 2020-11-27 10:39:38 [INFO] [EVAL] Class IoU: [0.9448 0.8953 0.4381 0.8995 0.7069 0.8005 0.8866 0.8711 0.9206 0.4195 0.8889 0.5639 0.8688 0.8842 0.8616 0.8705 0.6192 0.8592 0.5146 0.8601 0.7658] 2020-11-27 10:39:38 [INFO] [EVAL] Class Acc: [0.972 0.9231 0.4549 0.9535 0.8507 0.8817 0.9773 0.93 0.9464 0.6264 0.9171 0.9466 0.9119 0.9441 0.9476 0.9397 0.8056 0.9579 0.6108 0.886 0.921 ] 2020-11-27 10:39:46 [INFO] [EVAL] The model with the best validation mIoU (0.7781) was saved at iter 24000. 2020-11-27 10:41:52 [INFO] [TRAIN] epoch=37, iter=24100/40000, loss=0.1014, lr=0.004359, batch_cost=1.2586, reader_cost=0.0005 | ETA 05:33:31 2020-11-27 10:43:57 [INFO] [TRAIN] epoch=37, iter=24200/40000, loss=0.1040, lr=0.004335, batch_cost=1.2527, reader_cost=0.0003 | ETA 05:29:52 2020-11-27 10:46:03 [INFO] [TRAIN] epoch=37, iter=24300/40000, loss=0.1112, lr=0.004310, batch_cost=1.2552, reader_cost=0.0003 | ETA 05:28:27 2020-11-27 10:48:08 [INFO] [TRAIN] epoch=37, iter=24400/40000, loss=0.0998, lr=0.004285, batch_cost=1.2500, reader_cost=0.0003 | ETA 05:25:00 2020-11-27 10:50:13 [INFO] [TRAIN] epoch=38, iter=24500/40000, loss=0.1167, lr=0.004261, batch_cost=1.2564, reader_cost=0.0065 | ETA 05:24:33 2020-11-27 10:52:18 [INFO] [TRAIN] epoch=38, iter=24600/40000, loss=0.1157, lr=0.004236, batch_cost=1.2496, reader_cost=0.0005 | ETA 05:20:43 2020-11-27 10:54:23 [INFO] [TRAIN] epoch=38, iter=24700/40000, loss=0.1044, lr=0.004211, batch_cost=1.2527, reader_cost=0.0005 | ETA 05:19:26 2020-11-27 10:56:28 [INFO] [TRAIN] epoch=38, iter=24800/40000, loss=0.1063, lr=0.004186, batch_cost=1.2483, reader_cost=0.0002 | ETA 05:16:13 2020-11-27 10:58:34 [INFO] [TRAIN] epoch=38, iter=24900/40000, loss=0.1005, lr=0.004162, batch_cost=1.2538, reader_cost=0.0004 | ETA 05:15:33 2020-11-27 11:00:39 [INFO] [TRAIN] epoch=38, iter=25000/40000, loss=0.0981, lr=0.004137, batch_cost=1.2551, reader_cost=0.0005 | ETA 05:13:45 2020-11-27 11:02:45 [INFO] [TRAIN] epoch=38, iter=25100/40000, loss=0.1104, lr=0.004112, batch_cost=1.2534, reader_cost=0.0004 | ETA 05:11:15 2020-11-27 11:04:51 [INFO] [TRAIN] epoch=39, iter=25200/40000, loss=0.1110, lr=0.004087, batch_cost=1.2645, reader_cost=0.0064 | ETA 05:11:53 2020-11-27 11:06:57 [INFO] [TRAIN] epoch=39, iter=25300/40000, loss=0.1064, lr=0.004062, batch_cost=1.2625, reader_cost=0.0005 | ETA 05:09:19 2020-11-27 11:09:04 [INFO] [TRAIN] epoch=39, iter=25400/40000, loss=0.1093, lr=0.004037, batch_cost=1.2659, reader_cost=0.0006 | ETA 05:08:02 2020-11-27 11:11:10 [INFO] [TRAIN] epoch=39, iter=25500/40000, loss=0.1082, lr=0.004012, batch_cost=1.2653, reader_cost=0.0003 | ETA 05:05:47 2020-11-27 11:13:18 [INFO] [TRAIN] epoch=39, iter=25600/40000, loss=0.0904, lr=0.003987, batch_cost=1.2746, reader_cost=0.0010 | ETA 05:05:53 2020-11-27 11:15:24 [INFO] [TRAIN] epoch=39, iter=25700/40000, loss=0.1008, lr=0.003963, batch_cost=1.2653, reader_cost=0.0004 | ETA 05:01:34 2020-11-27 11:17:31 [INFO] [TRAIN] epoch=40, iter=25800/40000, loss=0.1091, lr=0.003938, batch_cost=1.2671, reader_cost=0.0063 | ETA 04:59:52 2020-11-27 11:19:37 [INFO] [TRAIN] epoch=40, iter=25900/40000, loss=0.1090, lr=0.003913, batch_cost=1.2554, reader_cost=0.0003 | ETA 04:55:01 2020-11-27 11:21:43 [INFO] [TRAIN] epoch=40, iter=26000/40000, loss=0.1072, lr=0.003888, batch_cost=1.2617, reader_cost=0.0005 | ETA 04:54:23 2020-11-27 11:23:49 [INFO] [TRAIN] epoch=40, iter=26100/40000, loss=0.1188, lr=0.003863, batch_cost=1.2657, reader_cost=0.0007 | ETA 04:53:13 2020-11-27 11:25:55 [INFO] [TRAIN] epoch=40, iter=26200/40000, loss=0.1010, lr=0.003838, batch_cost=1.2578, reader_cost=0.0004 | ETA 04:49:17 2020-11-27 11:28:01 [INFO] [TRAIN] epoch=40, iter=26300/40000, loss=0.0895, lr=0.003813, batch_cost=1.2562, reader_cost=0.0004 | ETA 04:46:49 2020-11-27 11:30:07 [INFO] [TRAIN] epoch=40, iter=26400/40000, loss=0.0999, lr=0.003788, batch_cost=1.2616, reader_cost=0.0004 | ETA 04:45:57 2020-11-27 11:32:14 [INFO] [TRAIN] epoch=41, iter=26500/40000, loss=0.0989, lr=0.003762, batch_cost=1.2677, reader_cost=0.0069 | ETA 04:45:14 2020-11-27 11:34:18 [INFO] [TRAIN] epoch=41, iter=26600/40000, loss=0.1042, lr=0.003737, batch_cost=1.2455, reader_cost=0.0002 | ETA 04:38:10 2020-11-27 11:36:23 [INFO] [TRAIN] epoch=41, iter=26700/40000, loss=0.1025, lr=0.003712, batch_cost=1.2514, reader_cost=0.0002 | ETA 04:37:23 2020-11-27 11:38:29 [INFO] [TRAIN] epoch=41, iter=26800/40000, loss=0.1079, lr=0.003687, batch_cost=1.2539, reader_cost=0.0003 | ETA 04:35:51 2020-11-27 11:40:34 [INFO] [TRAIN] epoch=41, iter=26900/40000, loss=0.1008, lr=0.003662, batch_cost=1.2478, reader_cost=0.0002 | ETA 04:32:26 2020-11-27 11:42:38 [INFO] [TRAIN] epoch=41, iter=27000/40000, loss=0.1015, lr=0.003637, batch_cost=1.2487, reader_cost=0.0002 | ETA 04:30:32 2020-11-27 11:44:44 [INFO] [TRAIN] epoch=41, iter=27100/40000, loss=0.1191, lr=0.003612, batch_cost=1.2510, reader_cost=0.0002 | ETA 04:28:57 2020-11-27 11:46:50 [INFO] [TRAIN] epoch=42, iter=27200/40000, loss=0.1086, lr=0.003586, batch_cost=1.2625, reader_cost=0.0067 | ETA 04:29:19 2020-11-27 11:48:55 [INFO] [TRAIN] epoch=42, iter=27300/40000, loss=0.0989, lr=0.003561, batch_cost=1.2476, reader_cost=0.0001 | ETA 04:24:04 2020-11-27 11:51:00 [INFO] [TRAIN] epoch=42, iter=27400/40000, loss=0.1043, lr=0.003536, batch_cost=1.2540, reader_cost=0.0003 | ETA 04:23:19 2020-11-27 11:53:05 [INFO] [TRAIN] epoch=42, iter=27500/40000, loss=0.1091, lr=0.003511, batch_cost=1.2496, reader_cost=0.0002 | ETA 04:20:19 2020-11-27 11:55:10 [INFO] [TRAIN] epoch=42, iter=27600/40000, loss=0.0924, lr=0.003485, batch_cost=1.2541, reader_cost=0.0002 | ETA 04:19:10 2020-11-27 11:57:15 [INFO] [TRAIN] epoch=42, iter=27700/40000, loss=0.0925, lr=0.003460, batch_cost=1.2497, reader_cost=0.0002 | ETA 04:16:11 2020-11-27 11:59:21 [INFO] [TRAIN] epoch=43, iter=27800/40000, loss=0.0988, lr=0.003435, batch_cost=1.2571, reader_cost=0.0063 | ETA 04:15:36 2020-11-27 12:01:26 [INFO] [TRAIN] epoch=43, iter=27900/40000, loss=0.0973, lr=0.003409, batch_cost=1.2506, reader_cost=0.0004 | ETA 04:12:12 2020-11-27 12:03:32 [INFO] [TRAIN] epoch=43, iter=28000/40000, loss=0.0989, lr=0.003384, batch_cost=1.2548, reader_cost=0.0002 | ETA 04:10:57 2020-11-27 12:03:32 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-27 12:04:01 [INFO] [EVAL] #Images=1449 mIoU=0.7954 Acc=0.9552 Kappa=0.9001 2020-11-27 12:04:01 [INFO] [EVAL] Class IoU: [0.9485 0.8953 0.4382 0.9073 0.7144 0.8076 0.9524 0.8901 0.9379 0.4312 0.9195 0.5806 0.8898 0.8954 0.8703 0.8811 0.6546 0.9113 0.5487 0.918 0.7122] 2020-11-27 12:04:01 [INFO] [EVAL] Class Acc: [0.966 0.9377 0.4535 0.9489 0.8491 0.9024 0.9785 0.9495 0.9692 0.6962 0.9806 0.9341 0.9363 0.9508 0.9395 0.9584 0.8589 0.9555 0.8215 0.9672 0.8907] 2020-11-27 12:04:10 [INFO] [EVAL] The model with the best validation mIoU (0.7954) was saved at iter 28000. 2020-11-27 12:06:15 [INFO] [TRAIN] epoch=43, iter=28100/40000, loss=0.1016, lr=0.003359, batch_cost=1.2562, reader_cost=0.0003 | ETA 04:09:09 2020-11-27 12:08:21 [INFO] [TRAIN] epoch=43, iter=28200/40000, loss=0.0953, lr=0.003333, batch_cost=1.2576, reader_cost=0.0003 | ETA 04:07:19 2020-11-27 12:10:26 [INFO] [TRAIN] epoch=43, iter=28300/40000, loss=0.0947, lr=0.003308, batch_cost=1.2547, reader_cost=0.0002 | ETA 04:04:40 2020-11-27 12:12:32 [INFO] [TRAIN] epoch=43, iter=28400/40000, loss=0.0873, lr=0.003282, batch_cost=1.2532, reader_cost=0.0002 | ETA 04:02:17 2020-11-27 12:14:38 [INFO] [TRAIN] epoch=44, iter=28500/40000, loss=0.0905, lr=0.003257, batch_cost=1.2584, reader_cost=0.0064 | ETA 04:01:11 2020-11-27 12:16:43 [INFO] [TRAIN] epoch=44, iter=28600/40000, loss=0.1042, lr=0.003231, batch_cost=1.2510, reader_cost=0.0002 | ETA 03:57:41 2020-11-27 12:18:48 [INFO] [TRAIN] epoch=44, iter=28700/40000, loss=0.0987, lr=0.003206, batch_cost=1.2557, reader_cost=0.0002 | ETA 03:56:29 2020-11-27 12:20:53 [INFO] [TRAIN] epoch=44, iter=28800/40000, loss=0.1077, lr=0.003180, batch_cost=1.2474, reader_cost=0.0002 | ETA 03:52:50 2020-11-27 12:22:58 [INFO] [TRAIN] epoch=44, iter=28900/40000, loss=0.0969, lr=0.003155, batch_cost=1.2503, reader_cost=0.0002 | ETA 03:51:17 2020-11-27 12:25:03 [INFO] [TRAIN] epoch=44, iter=29000/40000, loss=0.0924, lr=0.003129, batch_cost=1.2535, reader_cost=0.0003 | ETA 03:49:48 2020-11-27 12:27:09 [INFO] [TRAIN] epoch=45, iter=29100/40000, loss=0.0883, lr=0.003104, batch_cost=1.2532, reader_cost=0.0065 | ETA 03:47:39 2020-11-27 12:29:13 [INFO] [TRAIN] epoch=45, iter=29200/40000, loss=0.1087, lr=0.003078, batch_cost=1.2449, reader_cost=0.0003 | ETA 03:44:05 2020-11-27 12:31:19 [INFO] [TRAIN] epoch=45, iter=29300/40000, loss=0.0969, lr=0.003052, batch_cost=1.2539, reader_cost=0.0002 | ETA 03:43:36 2020-11-27 12:33:23 [INFO] [TRAIN] epoch=45, iter=29400/40000, loss=0.1021, lr=0.003027, batch_cost=1.2488, reader_cost=0.0002 | ETA 03:40:37 2020-11-27 12:35:29 [INFO] [TRAIN] epoch=45, iter=29500/40000, loss=0.0929, lr=0.003001, batch_cost=1.2547, reader_cost=0.0002 | ETA 03:39:34 2020-11-27 12:37:34 [INFO] [TRAIN] epoch=45, iter=29600/40000, loss=0.0948, lr=0.002975, batch_cost=1.2484, reader_cost=0.0002 | ETA 03:36:23 2020-11-27 12:39:39 [INFO] [TRAIN] epoch=45, iter=29700/40000, loss=0.0860, lr=0.002949, batch_cost=1.2504, reader_cost=0.0002 | ETA 03:34:38 2020-11-27 12:41:45 [INFO] [TRAIN] epoch=46, iter=29800/40000, loss=0.0904, lr=0.002924, batch_cost=1.2578, reader_cost=0.0062 | ETA 03:33:50 2020-11-27 12:43:50 [INFO] [TRAIN] epoch=46, iter=29900/40000, loss=0.1032, lr=0.002898, batch_cost=1.2547, reader_cost=0.0002 | ETA 03:31:12 2020-11-27 12:45:57 [INFO] [TRAIN] epoch=46, iter=30000/40000, loss=0.0918, lr=0.002872, batch_cost=1.2672, reader_cost=0.0004 | ETA 03:31:12 2020-11-27 12:48:03 [INFO] [TRAIN] epoch=46, iter=30100/40000, loss=0.0915, lr=0.002846, batch_cost=1.2571, reader_cost=0.0003 | ETA 03:27:25 2020-11-27 12:50:09 [INFO] [TRAIN] epoch=46, iter=30200/40000, loss=0.0816, lr=0.002820, batch_cost=1.2607, reader_cost=0.0004 | ETA 03:25:54 2020-11-27 12:52:15 [INFO] [TRAIN] epoch=46, iter=30300/40000, loss=0.0874, lr=0.002794, batch_cost=1.2687, reader_cost=0.0008 | ETA 03:25:06 2020-11-27 12:54:22 [INFO] [TRAIN] epoch=46, iter=30400/40000, loss=0.0914, lr=0.002768, batch_cost=1.2640, reader_cost=0.0006 | ETA 03:22:14 2020-11-27 12:56:28 [INFO] [TRAIN] epoch=47, iter=30500/40000, loss=0.0907, lr=0.002742, batch_cost=1.2590, reader_cost=0.0067 | ETA 03:19:20 2020-11-27 12:58:34 [INFO] [TRAIN] epoch=47, iter=30600/40000, loss=0.0853, lr=0.002716, batch_cost=1.2621, reader_cost=0.0003 | ETA 03:17:43 2020-11-27 13:00:40 [INFO] [TRAIN] epoch=47, iter=30700/40000, loss=0.0849, lr=0.002690, batch_cost=1.2624, reader_cost=0.0005 | ETA 03:15:40 2020-11-27 13:02:46 [INFO] [TRAIN] epoch=47, iter=30800/40000, loss=0.0906, lr=0.002664, batch_cost=1.2611, reader_cost=0.0004 | ETA 03:13:22 2020-11-27 13:04:53 [INFO] [TRAIN] epoch=47, iter=30900/40000, loss=0.0863, lr=0.002638, batch_cost=1.2674, reader_cost=0.0008 | ETA 03:12:13 2020-11-27 13:06:59 [INFO] [TRAIN] epoch=47, iter=31000/40000, loss=0.0926, lr=0.002612, batch_cost=1.2589, reader_cost=0.0006 | ETA 03:08:49 2020-11-27 13:09:06 [INFO] [TRAIN] epoch=48, iter=31100/40000, loss=0.0875, lr=0.002586, batch_cost=1.2694, reader_cost=0.0066 | ETA 03:08:17 2020-11-27 13:11:11 [INFO] [TRAIN] epoch=48, iter=31200/40000, loss=0.0884, lr=0.002560, batch_cost=1.2541, reader_cost=0.0003 | ETA 03:03:56 2020-11-27 13:13:17 [INFO] [TRAIN] epoch=48, iter=31300/40000, loss=0.0841, lr=0.002534, batch_cost=1.2525, reader_cost=0.0006 | ETA 03:01:36 2020-11-27 13:15:22 [INFO] [TRAIN] epoch=48, iter=31400/40000, loss=0.0856, lr=0.002507, batch_cost=1.2570, reader_cost=0.0007 | ETA 03:00:09 2020-11-27 13:17:29 [INFO] [TRAIN] epoch=48, iter=31500/40000, loss=0.0846, lr=0.002481, batch_cost=1.2642, reader_cost=0.0005 | ETA 02:59:05 2020-11-27 13:19:34 [INFO] [TRAIN] epoch=48, iter=31600/40000, loss=0.0866, lr=0.002455, batch_cost=1.2566, reader_cost=0.0004 | ETA 02:55:55 2020-11-27 13:21:41 [INFO] [TRAIN] epoch=48, iter=31700/40000, loss=0.0840, lr=0.002429, batch_cost=1.2645, reader_cost=0.0009 | ETA 02:54:55 2020-11-27 13:23:48 [INFO] [TRAIN] epoch=49, iter=31800/40000, loss=0.0865, lr=0.002402, batch_cost=1.2675, reader_cost=0.0066 | ETA 02:53:13 2020-11-27 13:25:54 [INFO] [TRAIN] epoch=49, iter=31900/40000, loss=0.0822, lr=0.002376, batch_cost=1.2649, reader_cost=0.0006 | ETA 02:50:46 2020-11-27 13:28:00 [INFO] [TRAIN] epoch=49, iter=32000/40000, loss=0.0867, lr=0.002350, batch_cost=1.2598, reader_cost=0.0003 | ETA 02:47:58 2020-11-27 13:28:00 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-27 13:28:30 [INFO] [EVAL] #Images=1449 mIoU=0.7999 Acc=0.9562 Kappa=0.9029 2020-11-27 13:28:30 [INFO] [EVAL] Class IoU: [0.9504 0.8959 0.444 0.905 0.7248 0.8331 0.9436 0.895 0.9357 0.4241 0.9347 0.5977 0.8814 0.8936 0.8779 0.8834 0.6427 0.9113 0.5861 0.916 0.7217] 2020-11-27 13:28:30 [INFO] [EVAL] Class Acc: [0.9692 0.9349 0.4612 0.9523 0.8522 0.8916 0.9786 0.95 0.9573 0.6308 0.9786 0.9177 0.9378 0.9598 0.9485 0.9581 0.8202 0.9744 0.8286 0.9588 0.866 ] 2020-11-27 13:28:39 [INFO] [EVAL] The model with the best validation mIoU (0.7999) was saved at iter 32000. 2020-11-27 13:30:44 [INFO] [TRAIN] epoch=49, iter=32100/40000, loss=0.0900, lr=0.002323, batch_cost=1.2554, reader_cost=0.0004 | ETA 02:45:17 2020-11-27 13:32:50 [INFO] [TRAIN] epoch=49, iter=32200/40000, loss=0.0852, lr=0.002297, batch_cost=1.2576, reader_cost=0.0005 | ETA 02:43:29 2020-11-27 13:34:56 [INFO] [TRAIN] epoch=49, iter=32300/40000, loss=0.0816, lr=0.002270, batch_cost=1.2593, reader_cost=0.0004 | ETA 02:41:36 2020-11-27 13:37:02 [INFO] [TRAIN] epoch=50, iter=32400/40000, loss=0.0879, lr=0.002244, batch_cost=1.2612, reader_cost=0.0065 | ETA 02:39:45 2020-11-27 13:39:07 [INFO] [TRAIN] epoch=50, iter=32500/40000, loss=0.0826, lr=0.002217, batch_cost=1.2495, reader_cost=0.0002 | ETA 02:36:11 2020-11-27 13:41:13 [INFO] [TRAIN] epoch=50, iter=32600/40000, loss=0.0880, lr=0.002190, batch_cost=1.2621, reader_cost=0.0005 | ETA 02:35:39 2020-11-27 13:43:19 [INFO] [TRAIN] epoch=50, iter=32700/40000, loss=0.0891, lr=0.002164, batch_cost=1.2568, reader_cost=0.0003 | ETA 02:32:54 2020-11-27 13:45:24 [INFO] [TRAIN] epoch=50, iter=32800/40000, loss=0.0848, lr=0.002137, batch_cost=1.2562, reader_cost=0.0002 | ETA 02:30:44 2020-11-27 13:47:30 [INFO] [TRAIN] epoch=50, iter=32900/40000, loss=0.0828, lr=0.002110, batch_cost=1.2530, reader_cost=0.0004 | ETA 02:28:16 2020-11-27 13:49:35 [INFO] [TRAIN] epoch=50, iter=33000/40000, loss=0.0814, lr=0.002083, batch_cost=1.2532, reader_cost=0.0003 | ETA 02:26:12 2020-11-27 13:51:41 [INFO] [TRAIN] epoch=51, iter=33100/40000, loss=0.0839, lr=0.002057, batch_cost=1.2604, reader_cost=0.0062 | ETA 02:24:56 2020-11-27 13:53:46 [INFO] [TRAIN] epoch=51, iter=33200/40000, loss=0.0863, lr=0.002030, batch_cost=1.2512, reader_cost=0.0001 | ETA 02:21:48 2020-11-27 13:55:52 [INFO] [TRAIN] epoch=51, iter=33300/40000, loss=0.0763, lr=0.002003, batch_cost=1.2556, reader_cost=0.0004 | ETA 02:20:12 2020-11-27 13:57:57 [INFO] [TRAIN] epoch=51, iter=33400/40000, loss=0.0801, lr=0.001976, batch_cost=1.2482, reader_cost=0.0002 | ETA 02:17:18 2020-11-27 14:00:02 [INFO] [TRAIN] epoch=51, iter=33500/40000, loss=0.0878, lr=0.001949, batch_cost=1.2517, reader_cost=0.0002 | ETA 02:15:36 2020-11-27 14:02:07 [INFO] [TRAIN] epoch=51, iter=33600/40000, loss=0.0822, lr=0.001922, batch_cost=1.2552, reader_cost=0.0002 | ETA 02:13:52 2020-11-27 14:04:13 [INFO] [TRAIN] epoch=51, iter=33700/40000, loss=0.0847, lr=0.001895, batch_cost=1.2573, reader_cost=0.0002 | ETA 02:12:01 2020-11-27 14:06:19 [INFO] [TRAIN] epoch=52, iter=33800/40000, loss=0.0745, lr=0.001868, batch_cost=1.2561, reader_cost=0.0065 | ETA 02:09:47 2020-11-27 14:08:24 [INFO] [TRAIN] epoch=52, iter=33900/40000, loss=0.0900, lr=0.001841, batch_cost=1.2543, reader_cost=0.0002 | ETA 02:07:31 2020-11-27 14:10:30 [INFO] [TRAIN] epoch=52, iter=34000/40000, loss=0.0767, lr=0.001814, batch_cost=1.2587, reader_cost=0.0002 | ETA 02:05:51 2020-11-27 14:12:36 [INFO] [TRAIN] epoch=52, iter=34100/40000, loss=0.0748, lr=0.001786, batch_cost=1.2590, reader_cost=0.0003 | ETA 02:03:48 2020-11-27 14:14:41 [INFO] [TRAIN] epoch=52, iter=34200/40000, loss=0.0824, lr=0.001759, batch_cost=1.2554, reader_cost=0.0002 | ETA 02:01:21 2020-11-27 14:16:46 [INFO] [TRAIN] epoch=52, iter=34300/40000, loss=0.0745, lr=0.001732, batch_cost=1.2511, reader_cost=0.0002 | ETA 01:58:51 2020-11-27 14:18:53 [INFO] [TRAIN] epoch=53, iter=34400/40000, loss=0.0889, lr=0.001704, batch_cost=1.2634, reader_cost=0.0066 | ETA 01:57:54 2020-11-27 14:20:59 [INFO] [TRAIN] epoch=53, iter=34500/40000, loss=0.0743, lr=0.001677, batch_cost=1.2615, reader_cost=0.0002 | ETA 01:55:38 2020-11-27 14:23:06 [INFO] [TRAIN] epoch=53, iter=34600/40000, loss=0.0777, lr=0.001650, batch_cost=1.2656, reader_cost=0.0002 | ETA 01:53:54 2020-11-27 14:25:12 [INFO] [TRAIN] epoch=53, iter=34700/40000, loss=0.0872, lr=0.001622, batch_cost=1.2643, reader_cost=0.0003 | ETA 01:51:40 2020-11-27 14:27:19 [INFO] [TRAIN] epoch=53, iter=34800/40000, loss=0.0772, lr=0.001594, batch_cost=1.2672, reader_cost=0.0004 | ETA 01:49:49 2020-11-27 14:29:25 [INFO] [TRAIN] epoch=53, iter=34900/40000, loss=0.0810, lr=0.001567, batch_cost=1.2644, reader_cost=0.0002 | ETA 01:47:28 2020-11-27 14:31:32 [INFO] [TRAIN] epoch=53, iter=35000/40000, loss=0.0820, lr=0.001539, batch_cost=1.2715, reader_cost=0.0004 | ETA 01:45:57 2020-11-27 14:33:39 [INFO] [TRAIN] epoch=54, iter=35100/40000, loss=0.0770, lr=0.001511, batch_cost=1.2688, reader_cost=0.0064 | ETA 01:43:37 2020-11-27 14:35:45 [INFO] [TRAIN] epoch=54, iter=35200/40000, loss=0.0770, lr=0.001484, batch_cost=1.2574, reader_cost=0.0003 | ETA 01:40:35 2020-11-27 14:37:51 [INFO] [TRAIN] epoch=54, iter=35300/40000, loss=0.0798, lr=0.001456, batch_cost=1.2594, reader_cost=0.0006 | ETA 01:38:39 2020-11-27 14:39:58 [INFO] [TRAIN] epoch=54, iter=35400/40000, loss=0.0921, lr=0.001428, batch_cost=1.2672, reader_cost=0.0008 | ETA 01:37:09 2020-11-27 14:42:04 [INFO] [TRAIN] epoch=54, iter=35500/40000, loss=0.0742, lr=0.001400, batch_cost=1.2615, reader_cost=0.0006 | ETA 01:34:36 2020-11-27 14:44:10 [INFO] [TRAIN] epoch=54, iter=35600/40000, loss=0.0847, lr=0.001372, batch_cost=1.2589, reader_cost=0.0008 | ETA 01:32:19 2020-11-27 14:46:16 [INFO] [TRAIN] epoch=55, iter=35700/40000, loss=0.0824, lr=0.001344, batch_cost=1.2637, reader_cost=0.0068 | ETA 01:30:33 2020-11-27 14:48:21 [INFO] [TRAIN] epoch=55, iter=35800/40000, loss=0.0803, lr=0.001316, batch_cost=1.2545, reader_cost=0.0002 | ETA 01:27:48 2020-11-27 14:50:27 [INFO] [TRAIN] epoch=55, iter=35900/40000, loss=0.0767, lr=0.001287, batch_cost=1.2550, reader_cost=0.0002 | ETA 01:25:45 2020-11-27 14:52:32 [INFO] [TRAIN] epoch=55, iter=36000/40000, loss=0.0754, lr=0.001259, batch_cost=1.2487, reader_cost=0.0002 | ETA 01:23:14 2020-11-27 14:52:32 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-27 14:53:01 [INFO] [EVAL] #Images=1449 mIoU=0.8021 Acc=0.9566 Kappa=0.9034 2020-11-27 14:53:01 [INFO] [EVAL] Class IoU: [0.9495 0.8972 0.441 0.9001 0.7122 0.823 0.9561 0.8997 0.938 0.4405 0.9378 0.5673 0.8846 0.8984 0.8818 0.8857 0.6618 0.9132 0.5884 0.9178 0.7499] 2020-11-27 14:53:01 [INFO] [EVAL] Class Acc: [0.9669 0.9359 0.4555 0.9528 0.8389 0.9017 0.9802 0.9562 0.9605 0.7407 0.9817 0.9527 0.9361 0.9581 0.9539 0.9572 0.8215 0.9748 0.841 0.9607 0.9168] 2020-11-27 14:53:10 [INFO] [EVAL] The model with the best validation mIoU (0.8021) was saved at iter 36000. 2020-11-27 14:55:15 [INFO] [TRAIN] epoch=55, iter=36100/40000, loss=0.0788, lr=0.001231, batch_cost=1.2509, reader_cost=0.0002 | ETA 01:21:18 2020-11-27 14:57:21 [INFO] [TRAIN] epoch=55, iter=36200/40000, loss=0.0799, lr=0.001202, batch_cost=1.2546, reader_cost=0.0002 | ETA 01:19:27 2020-11-27 14:59:26 [INFO] [TRAIN] epoch=55, iter=36300/40000, loss=0.0745, lr=0.001174, batch_cost=1.2555, reader_cost=0.0003 | ETA 01:17:25 2020-11-27 15:01:32 [INFO] [TRAIN] epoch=56, iter=36400/40000, loss=0.0688, lr=0.001145, batch_cost=1.2562, reader_cost=0.0068 | ETA 01:15:22 2020-11-27 15:03:37 [INFO] [TRAIN] epoch=56, iter=36500/40000, loss=0.0798, lr=0.001117, batch_cost=1.2524, reader_cost=0.0002 | ETA 01:13:03 2020-11-27 15:05:42 [INFO] [TRAIN] epoch=56, iter=36600/40000, loss=0.0920, lr=0.001088, batch_cost=1.2527, reader_cost=0.0002 | ETA 01:10:59 2020-11-27 15:07:48 [INFO] [TRAIN] epoch=56, iter=36700/40000, loss=0.0784, lr=0.001059, batch_cost=1.2526, reader_cost=0.0003 | ETA 01:08:53 2020-11-27 15:09:53 [INFO] [TRAIN] epoch=56, iter=36800/40000, loss=0.0676, lr=0.001030, batch_cost=1.2571, reader_cost=0.0001 | ETA 01:07:02 2020-11-27 15:11:58 [INFO] [TRAIN] epoch=56, iter=36900/40000, loss=0.0789, lr=0.001001, batch_cost=1.2495, reader_cost=0.0002 | ETA 01:04:33 2020-11-27 15:14:03 [INFO] [TRAIN] epoch=56, iter=37000/40000, loss=0.0804, lr=0.000972, batch_cost=1.2519, reader_cost=0.0002 | ETA 01:02:35 2020-11-27 15:16:09 [INFO] [TRAIN] epoch=57, iter=37100/40000, loss=0.0742, lr=0.000943, batch_cost=1.2585, reader_cost=0.0065 | ETA 01:00:49 2020-11-27 15:18:15 [INFO] [TRAIN] epoch=57, iter=37200/40000, loss=0.0727, lr=0.000914, batch_cost=1.2543, reader_cost=0.0002 | ETA 00:58:31 2020-11-27 15:20:20 [INFO] [TRAIN] epoch=57, iter=37300/40000, loss=0.0771, lr=0.000884, batch_cost=1.2536, reader_cost=0.0004 | ETA 00:56:24 2020-11-27 15:22:25 [INFO] [TRAIN] epoch=57, iter=37400/40000, loss=0.0781, lr=0.000855, batch_cost=1.2524, reader_cost=0.0002 | ETA 00:54:16 2020-11-27 15:24:31 [INFO] [TRAIN] epoch=57, iter=37500/40000, loss=0.0815, lr=0.000825, batch_cost=1.2570, reader_cost=0.0008 | ETA 00:52:22 2020-11-27 15:26:36 [INFO] [TRAIN] epoch=57, iter=37600/40000, loss=0.0748, lr=0.000795, batch_cost=1.2534, reader_cost=0.0003 | ETA 00:50:08 2020-11-27 15:28:43 [INFO] [TRAIN] epoch=58, iter=37700/40000, loss=0.0737, lr=0.000765, batch_cost=1.2619, reader_cost=0.0063 | ETA 00:48:22 2020-11-27 15:30:48 [INFO] [TRAIN] epoch=58, iter=37800/40000, loss=0.0739, lr=0.000735, batch_cost=1.2514, reader_cost=0.0002 | ETA 00:45:53 2020-11-27 15:32:52 [INFO] [TRAIN] epoch=58, iter=37900/40000, loss=0.0758, lr=0.000705, batch_cost=1.2464, reader_cost=0.0002 | ETA 00:43:37 2020-11-27 15:34:57 [INFO] [TRAIN] epoch=58, iter=38000/40000, loss=0.0811, lr=0.000675, batch_cost=1.2513, reader_cost=0.0002 | ETA 00:41:42 2020-11-27 15:37:02 [INFO] [TRAIN] epoch=58, iter=38100/40000, loss=0.0705, lr=0.000645, batch_cost=1.2465, reader_cost=0.0002 | ETA 00:39:28 2020-11-27 15:39:07 [INFO] [TRAIN] epoch=58, iter=38200/40000, loss=0.0753, lr=0.000614, batch_cost=1.2466, reader_cost=0.0002 | ETA 00:37:23 2020-11-27 15:41:11 [INFO] [TRAIN] epoch=58, iter=38300/40000, loss=0.0758, lr=0.000583, batch_cost=1.2471, reader_cost=0.0002 | ETA 00:35:20 2020-11-27 15:43:17 [INFO] [TRAIN] epoch=59, iter=38400/40000, loss=0.0776, lr=0.000552, batch_cost=1.2558, reader_cost=0.0065 | ETA 00:33:29 2020-11-27 15:45:22 [INFO] [TRAIN] epoch=59, iter=38500/40000, loss=0.0795, lr=0.000521, batch_cost=1.2520, reader_cost=0.0004 | ETA 00:31:18 2020-11-27 15:47:28 [INFO] [TRAIN] epoch=59, iter=38600/40000, loss=0.0770, lr=0.000490, batch_cost=1.2527, reader_cost=0.0002 | ETA 00:29:13 2020-11-27 15:49:33 [INFO] [TRAIN] epoch=59, iter=38700/40000, loss=0.0707, lr=0.000458, batch_cost=1.2531, reader_cost=0.0004 | ETA 00:27:08 2020-11-27 15:51:38 [INFO] [TRAIN] epoch=59, iter=38800/40000, loss=0.0781, lr=0.000426, batch_cost=1.2544, reader_cost=0.0004 | ETA 00:25:05 2020-11-27 15:53:43 [INFO] [TRAIN] epoch=59, iter=38900/40000, loss=0.0765, lr=0.000394, batch_cost=1.2515, reader_cost=0.0002 | ETA 00:22:56 2020-11-27 15:55:49 [INFO] [TRAIN] epoch=60, iter=39000/40000, loss=0.0817, lr=0.000362, batch_cost=1.2566, reader_cost=0.0065 | ETA 00:20:56 2020-11-27 15:57:55 [INFO] [TRAIN] epoch=60, iter=39100/40000, loss=0.0770, lr=0.000329, batch_cost=1.2562, reader_cost=0.0002 | ETA 00:18:50 2020-11-27 16:00:01 [INFO] [TRAIN] epoch=60, iter=39200/40000, loss=0.0725, lr=0.000296, batch_cost=1.2595, reader_cost=0.0004 | ETA 00:16:47 2020-11-27 16:02:06 [INFO] [TRAIN] epoch=60, iter=39300/40000, loss=0.0820, lr=0.000263, batch_cost=1.2562, reader_cost=0.0005 | ETA 00:14:39 2020-11-27 16:04:12 [INFO] [TRAIN] epoch=60, iter=39400/40000, loss=0.0801, lr=0.000229, batch_cost=1.2582, reader_cost=0.0006 | ETA 00:12:34 2020-11-27 16:06:18 [INFO] [TRAIN] epoch=60, iter=39500/40000, loss=0.0756, lr=0.000194, batch_cost=1.2626, reader_cost=0.0007 | ETA 00:10:31 2020-11-27 16:08:25 [INFO] [TRAIN] epoch=60, iter=39600/40000, loss=0.0791, lr=0.000159, batch_cost=1.2645, reader_cost=0.0008 | ETA 00:08:25 2020-11-27 16:10:31 [INFO] [TRAIN] epoch=61, iter=39700/40000, loss=0.0741, lr=0.000123, batch_cost=1.2634, reader_cost=0.0074 | ETA 00:06:19 2020-11-27 16:12:36 [INFO] [TRAIN] epoch=61, iter=39800/40000, loss=0.0777, lr=0.000085, batch_cost=1.2534, reader_cost=0.0002 | ETA 00:04:10 2020-11-27 16:14:41 [INFO] [TRAIN] epoch=61, iter=39900/40000, loss=0.0820, lr=0.000046, batch_cost=1.2479, reader_cost=0.0002 | ETA 00:02:04 2020-11-27 16:16:46 [INFO] [TRAIN] epoch=61, iter=40000/40000, loss=0.0784, lr=0.000001, batch_cost=1.2515, reader_cost=0.0002 | ETA 00:00:00 2020-11-27 16:16:46 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-27 16:17:17 [INFO] [EVAL] #Images=1449 mIoU=0.8062 Acc=0.9573 Kappa=0.9051 2020-11-27 16:17:17 [INFO] [EVAL] Class IoU: [0.9501 0.8982 0.4436 0.9068 0.7018 0.835 0.9542 0.9053 0.9379 0.4792 0.9415 0.5742 0.8851 0.9036 0.8899 0.886 0.6516 0.922 0.5869 0.9178 0.76 ] 2020-11-27 16:17:17 [INFO] [EVAL] Class Acc: [0.9682 0.9314 0.4581 0.9515 0.834 0.8932 0.9807 0.9617 0.9624 0.7396 0.9798 0.9466 0.9322 0.9578 0.9531 0.958 0.8227 0.9683 0.8378 0.9578 0.9223] 2020-11-27 16:17:25 [INFO] [EVAL] The model with the best validation mIoU (0.8062) was saved at iter 40000.