2020-11-29 20:13:35 [INFO] ------------Environment Information------------- platform: Linux-3.10.0_3-0-0-34-x86_64-with-centos-7.5.1804-Core Python: 3.7.9 (default, Aug 31 2020, 12:42:55) [GCC 7.3.0] Paddle compiled with cuda: True NVCC: Cuda compilation tools, release 10.2, V10.2.89 cudnn: 7.6 GPUs used: 4 CUDA_VISIBLE_DEVICES: 0,1,2,3 GPU: ['GPU 0: Tesla V100-SXM2-16GB', 'GPU 1: Tesla V100-SXM2-16GB', 'GPU 2: Tesla V100-SXM2-16GB', 'GPU 3: Tesla V100-SXM2-16GB', 'GPU 4: Tesla V100-SXM2-16GB', 'GPU 5: Tesla V100-SXM2-16GB', 'GPU 6: Tesla V100-SXM2-16GB', 'GPU 7: Tesla V100-SXM2-16GB'] GCC: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39) PaddlePaddle: 2.0.0-rc0 OpenCV: 4.1.0 ------------------------------------------------ 2020-11-29 20:13:35 [INFO] ---------------Config Information--------------- batch_size: 4 iters: 40000 learning_rate: decay: end_lr: 0.0 power: 0.9 type: poly value: 0.01 loss: coef: - 1 types: - ignore_index: 255 type: CrossEntropyLoss model: align_corners: false aspp_out_channels: 256 aspp_ratios: - 1 - 12 - 24 - 36 backbone: multi_grid: - 1 - 2 - 4 output_stride: 8 pretrained: https://bj.bcebos.com/paddleseg/dygraph/resnet50_vd_ssld_v2.tar.gz type: ResNet50_vd backbone_indices: - 0 - 3 num_classes: 19 pretrained: null type: DeepLabV3P optimizer: momentum: 0.9 type: sgd weight_decay: 4.0e-05 train_dataset: dataset_root: data/VOCdevkit/ mode: trainaug transforms: - max_scale_factor: 2.0 min_scale_factor: 0.5 scale_step_size: 0.25 type: ResizeStepScaling - crop_size: - 512 - 512 type: RandomPaddingCrop - type: RandomHorizontalFlip - brightness_range: 0.4 contrast_range: 0.4 saturation_range: 0.4 type: RandomDistort - type: Normalize type: PascalVOC val_dataset: dataset_root: data/VOCdevkit/ mode: val transforms: - target_size: - 512 - 512 type: Padding - type: Normalize type: PascalVOC ------------------------------------------------ 2020-11-29 20:13:41 [INFO] Loading pretrained model from https://bj.bcebos.com/paddleseg/dygraph/resnet50_vd_ssld_v2.tar.gz 2020-11-29 20:13:42 [INFO] There are 275/275 variables loaded into ResNet_vd. 2020-11-29 20:15:00 [INFO] [TRAIN] epoch=1, iter=100/40000, loss=1.1484, lr=0.009978, batch_cost=0.7137, reader_cost=0.0097 | ETA 07:54:37 2020-11-29 20:16:06 [INFO] [TRAIN] epoch=1, iter=200/40000, loss=0.7083, lr=0.009955, batch_cost=0.6677, reader_cost=0.0004 | ETA 07:22:53 2020-11-29 20:17:13 [INFO] [TRAIN] epoch=1, iter=300/40000, loss=0.7142, lr=0.009933, batch_cost=0.6669, reader_cost=0.0002 | ETA 07:21:17 2020-11-29 20:18:20 [INFO] [TRAIN] epoch=1, iter=400/40000, loss=0.6012, lr=0.009910, batch_cost=0.6655, reader_cost=0.0002 | ETA 07:19:13 2020-11-29 20:19:26 [INFO] [TRAIN] epoch=1, iter=500/40000, loss=0.5536, lr=0.009888, batch_cost=0.6622, reader_cost=0.0003 | ETA 07:15:56 2020-11-29 20:20:32 [INFO] [TRAIN] epoch=1, iter=600/40000, loss=0.5443, lr=0.009865, batch_cost=0.6663, reader_cost=0.0002 | ETA 07:17:30 2020-11-29 20:21:40 [INFO] [TRAIN] epoch=2, iter=700/40000, loss=0.4770, lr=0.009843, batch_cost=0.6733, reader_cost=0.0060 | ETA 07:21:02 2020-11-29 20:22:46 [INFO] [TRAIN] epoch=2, iter=800/40000, loss=0.4813, lr=0.009820, batch_cost=0.6618, reader_cost=0.0002 | ETA 07:12:23 2020-11-29 20:23:53 [INFO] [TRAIN] epoch=2, iter=900/40000, loss=0.4734, lr=0.009797, batch_cost=0.6668, reader_cost=0.0003 | ETA 07:14:31 2020-11-29 20:24:59 [INFO] [TRAIN] epoch=2, iter=1000/40000, loss=0.4709, lr=0.009775, batch_cost=0.6671, reader_cost=0.0002 | ETA 07:13:37 2020-11-29 20:26:06 [INFO] [TRAIN] epoch=2, iter=1100/40000, loss=0.4634, lr=0.009752, batch_cost=0.6633, reader_cost=0.0004 | ETA 07:10:02 2020-11-29 20:27:12 [INFO] [TRAIN] epoch=2, iter=1200/40000, loss=0.4989, lr=0.009730, batch_cost=0.6603, reader_cost=0.0003 | ETA 07:07:00 2020-11-29 20:28:19 [INFO] [TRAIN] epoch=2, iter=1300/40000, loss=0.4119, lr=0.009707, batch_cost=0.6690, reader_cost=0.0008 | ETA 07:11:29 2020-11-29 20:29:26 [INFO] [TRAIN] epoch=3, iter=1400/40000, loss=0.4276, lr=0.009685, batch_cost=0.6703, reader_cost=0.0065 | ETA 07:11:14 2020-11-29 20:30:32 [INFO] [TRAIN] epoch=3, iter=1500/40000, loss=0.4296, lr=0.009662, batch_cost=0.6642, reader_cost=0.0006 | ETA 07:06:13 2020-11-29 20:31:38 [INFO] [TRAIN] epoch=3, iter=1600/40000, loss=0.3911, lr=0.009639, batch_cost=0.6601, reader_cost=0.0004 | ETA 07:02:28 2020-11-29 20:32:44 [INFO] [TRAIN] epoch=3, iter=1700/40000, loss=0.3598, lr=0.009617, batch_cost=0.6623, reader_cost=0.0003 | ETA 07:02:45 2020-11-29 20:33:50 [INFO] [TRAIN] epoch=3, iter=1800/40000, loss=0.4692, lr=0.009594, batch_cost=0.6601, reader_cost=0.0005 | ETA 07:00:17 2020-11-29 20:34:56 [INFO] [TRAIN] epoch=3, iter=1900/40000, loss=0.4408, lr=0.009572, batch_cost=0.6612, reader_cost=0.0002 | ETA 06:59:51 2020-11-29 20:36:04 [INFO] [TRAIN] epoch=4, iter=2000/40000, loss=0.3773, lr=0.009549, batch_cost=0.6707, reader_cost=0.0064 | ETA 07:04:47 2020-11-29 20:37:10 [INFO] [TRAIN] epoch=4, iter=2100/40000, loss=0.3754, lr=0.009526, batch_cost=0.6645, reader_cost=0.0003 | ETA 06:59:44 2020-11-29 20:38:16 [INFO] [TRAIN] epoch=4, iter=2200/40000, loss=0.3479, lr=0.009504, batch_cost=0.6620, reader_cost=0.0003 | ETA 06:57:05 2020-11-29 20:39:22 [INFO] [TRAIN] epoch=4, iter=2300/40000, loss=0.3772, lr=0.009481, batch_cost=0.6598, reader_cost=0.0002 | ETA 06:54:34 2020-11-29 20:40:29 [INFO] [TRAIN] epoch=4, iter=2400/40000, loss=0.3440, lr=0.009459, batch_cost=0.6674, reader_cost=0.0002 | ETA 06:58:15 2020-11-29 20:41:35 [INFO] [TRAIN] epoch=4, iter=2500/40000, loss=0.3392, lr=0.009436, batch_cost=0.6633, reader_cost=0.0003 | ETA 06:54:34 2020-11-29 20:42:42 [INFO] [TRAIN] epoch=4, iter=2600/40000, loss=0.3383, lr=0.009413, batch_cost=0.6664, reader_cost=0.0003 | ETA 06:55:24 2020-11-29 20:43:49 [INFO] [TRAIN] epoch=5, iter=2700/40000, loss=0.3199, lr=0.009391, batch_cost=0.6728, reader_cost=0.0063 | ETA 06:58:15 2020-11-29 20:44:56 [INFO] [TRAIN] epoch=5, iter=2800/40000, loss=0.2774, lr=0.009368, batch_cost=0.6648, reader_cost=0.0003 | ETA 06:52:09 2020-11-29 20:46:02 [INFO] [TRAIN] epoch=5, iter=2900/40000, loss=0.3506, lr=0.009345, batch_cost=0.6627, reader_cost=0.0003 | ETA 06:49:45 2020-11-29 20:47:09 [INFO] [TRAIN] epoch=5, iter=3000/40000, loss=0.3152, lr=0.009323, batch_cost=0.6709, reader_cost=0.0009 | ETA 06:53:44 2020-11-29 20:48:16 [INFO] [TRAIN] epoch=5, iter=3100/40000, loss=0.3876, lr=0.009300, batch_cost=0.6686, reader_cost=0.0005 | ETA 06:51:11 2020-11-29 20:49:23 [INFO] [TRAIN] epoch=5, iter=3200/40000, loss=0.3616, lr=0.009277, batch_cost=0.6671, reader_cost=0.0006 | ETA 06:49:07 2020-11-29 20:50:30 [INFO] [TRAIN] epoch=5, iter=3300/40000, loss=0.3002, lr=0.009255, batch_cost=0.6700, reader_cost=0.0006 | ETA 06:49:50 2020-11-29 20:51:37 [INFO] [TRAIN] epoch=6, iter=3400/40000, loss=0.2782, lr=0.009232, batch_cost=0.6696, reader_cost=0.0067 | ETA 06:48:27 2020-11-29 20:52:43 [INFO] [TRAIN] epoch=6, iter=3500/40000, loss=0.3062, lr=0.009209, batch_cost=0.6664, reader_cost=0.0005 | ETA 06:45:25 2020-11-29 20:53:50 [INFO] [TRAIN] epoch=6, iter=3600/40000, loss=0.2994, lr=0.009186, batch_cost=0.6646, reader_cost=0.0003 | ETA 06:43:12 2020-11-29 20:54:56 [INFO] [TRAIN] epoch=6, iter=3700/40000, loss=0.3732, lr=0.009164, batch_cost=0.6651, reader_cost=0.0002 | ETA 06:42:22 2020-11-29 20:56:03 [INFO] [TRAIN] epoch=6, iter=3800/40000, loss=0.3721, lr=0.009141, batch_cost=0.6682, reader_cost=0.0006 | ETA 06:43:09 2020-11-29 20:57:10 [INFO] [TRAIN] epoch=6, iter=3900/40000, loss=0.3154, lr=0.009118, batch_cost=0.6694, reader_cost=0.0004 | ETA 06:42:44 2020-11-29 20:58:17 [INFO] [TRAIN] epoch=7, iter=4000/40000, loss=0.3088, lr=0.009096, batch_cost=0.6717, reader_cost=0.0059 | ETA 06:43:02 2020-11-29 20:58:17 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-29 20:58:37 [INFO] [EVAL] #Images=1449 mIoU=0.7034 Acc=0.9302 Kappa=0.8464 2020-11-29 20:58:37 [INFO] [EVAL] Class IoU: [0.9285 0.8667 0.4091 0.8591 0.5838 0.734 0.9067 0.829 0.8689 0.2954 0.687 0.5284 0.7547 0.7936 0.8371 0.8173 0.5168 0.6984 0.3723 0.8196 0.6641] 2020-11-29 20:58:37 [INFO] [EVAL] Class Acc: [0.9608 0.9414 0.434 0.9467 0.6502 0.8276 0.9523 0.9714 0.9199 0.3908 0.9685 0.6089 0.851 0.8594 0.8739 0.9358 0.8139 0.7993 0.8726 0.8666 0.8834] 2020-11-29 20:58:42 [INFO] [EVAL] The model with the best validation mIoU (0.7034) was saved at iter 4000. 2020-11-29 20:59:48 [INFO] [TRAIN] epoch=7, iter=4100/40000, loss=0.2337, lr=0.009073, batch_cost=0.6683, reader_cost=0.0003 | ETA 06:39:53 2020-11-29 21:00:55 [INFO] [TRAIN] epoch=7, iter=4200/40000, loss=0.3255, lr=0.009050, batch_cost=0.6681, reader_cost=0.0005 | ETA 06:38:37 2020-11-29 21:02:02 [INFO] [TRAIN] epoch=7, iter=4300/40000, loss=0.2929, lr=0.009027, batch_cost=0.6652, reader_cost=0.0006 | ETA 06:35:48 2020-11-29 21:03:09 [INFO] [TRAIN] epoch=7, iter=4400/40000, loss=0.3106, lr=0.009005, batch_cost=0.6716, reader_cost=0.0004 | ETA 06:38:27 2020-11-29 21:04:16 [INFO] [TRAIN] epoch=7, iter=4500/40000, loss=0.3261, lr=0.008982, batch_cost=0.6682, reader_cost=0.0005 | ETA 06:35:19 2020-11-29 21:05:22 [INFO] [TRAIN] epoch=7, iter=4600/40000, loss=0.3204, lr=0.008959, batch_cost=0.6661, reader_cost=0.0008 | ETA 06:32:59 2020-11-29 21:06:30 [INFO] [TRAIN] epoch=8, iter=4700/40000, loss=0.2984, lr=0.008936, batch_cost=0.6762, reader_cost=0.0070 | ETA 06:37:48 2020-11-29 21:07:37 [INFO] [TRAIN] epoch=8, iter=4800/40000, loss=0.2635, lr=0.008913, batch_cost=0.6714, reader_cost=0.0006 | ETA 06:33:51 2020-11-29 21:08:44 [INFO] [TRAIN] epoch=8, iter=4900/40000, loss=0.2589, lr=0.008891, batch_cost=0.6656, reader_cost=0.0005 | ETA 06:29:23 2020-11-29 21:09:50 [INFO] [TRAIN] epoch=8, iter=5000/40000, loss=0.2652, lr=0.008868, batch_cost=0.6684, reader_cost=0.0010 | ETA 06:29:52 2020-11-29 21:10:57 [INFO] [TRAIN] epoch=8, iter=5100/40000, loss=0.2792, lr=0.008845, batch_cost=0.6660, reader_cost=0.0003 | ETA 06:27:24 2020-11-29 21:12:04 [INFO] [TRAIN] epoch=8, iter=5200/40000, loss=0.2531, lr=0.008822, batch_cost=0.6705, reader_cost=0.0008 | ETA 06:28:51 2020-11-29 21:13:11 [INFO] [TRAIN] epoch=9, iter=5300/40000, loss=0.2611, lr=0.008799, batch_cost=0.6720, reader_cost=0.0065 | ETA 06:28:39 2020-11-29 21:14:18 [INFO] [TRAIN] epoch=9, iter=5400/40000, loss=0.2671, lr=0.008777, batch_cost=0.6647, reader_cost=0.0006 | ETA 06:23:20 2020-11-29 21:15:24 [INFO] [TRAIN] epoch=9, iter=5500/40000, loss=0.2432, lr=0.008754, batch_cost=0.6639, reader_cost=0.0004 | ETA 06:21:43 2020-11-29 21:16:30 [INFO] [TRAIN] epoch=9, iter=5600/40000, loss=0.2688, lr=0.008731, batch_cost=0.6617, reader_cost=0.0005 | ETA 06:19:23 2020-11-29 21:17:37 [INFO] [TRAIN] epoch=9, iter=5700/40000, loss=0.3214, lr=0.008708, batch_cost=0.6621, reader_cost=0.0003 | ETA 06:18:29 2020-11-29 21:18:43 [INFO] [TRAIN] epoch=9, iter=5800/40000, loss=0.2810, lr=0.008685, batch_cost=0.6617, reader_cost=0.0002 | ETA 06:17:10 2020-11-29 21:19:49 [INFO] [TRAIN] epoch=9, iter=5900/40000, loss=0.2767, lr=0.008662, batch_cost=0.6669, reader_cost=0.0004 | ETA 06:19:02 2020-11-29 21:20:56 [INFO] [TRAIN] epoch=10, iter=6000/40000, loss=0.2398, lr=0.008639, batch_cost=0.6675, reader_cost=0.0056 | ETA 06:18:15 2020-11-29 21:22:03 [INFO] [TRAIN] epoch=10, iter=6100/40000, loss=0.2060, lr=0.008617, batch_cost=0.6629, reader_cost=0.0004 | ETA 06:14:33 2020-11-29 21:23:09 [INFO] [TRAIN] epoch=10, iter=6200/40000, loss=0.2588, lr=0.008594, batch_cost=0.6651, reader_cost=0.0003 | ETA 06:14:41 2020-11-29 21:24:16 [INFO] [TRAIN] epoch=10, iter=6300/40000, loss=0.2421, lr=0.008571, batch_cost=0.6649, reader_cost=0.0002 | ETA 06:13:28 2020-11-29 21:25:22 [INFO] [TRAIN] epoch=10, iter=6400/40000, loss=0.2348, lr=0.008548, batch_cost=0.6623, reader_cost=0.0002 | ETA 06:10:53 2020-11-29 21:26:28 [INFO] [TRAIN] epoch=10, iter=6500/40000, loss=0.2515, lr=0.008525, batch_cost=0.6605, reader_cost=0.0004 | ETA 06:08:45 2020-11-29 21:27:35 [INFO] [TRAIN] epoch=10, iter=6600/40000, loss=0.2497, lr=0.008502, batch_cost=0.6673, reader_cost=0.0003 | ETA 06:11:28 2020-11-29 21:28:42 [INFO] [TRAIN] epoch=11, iter=6700/40000, loss=0.2167, lr=0.008479, batch_cost=0.6732, reader_cost=0.0060 | ETA 06:13:39 2020-11-29 21:29:48 [INFO] [TRAIN] epoch=11, iter=6800/40000, loss=0.2443, lr=0.008456, batch_cost=0.6608, reader_cost=0.0003 | ETA 06:05:37 2020-11-29 21:30:54 [INFO] [TRAIN] epoch=11, iter=6900/40000, loss=0.2310, lr=0.008433, batch_cost=0.6583, reader_cost=0.0003 | ETA 06:03:08 2020-11-29 21:32:00 [INFO] [TRAIN] epoch=11, iter=7000/40000, loss=0.2543, lr=0.008410, batch_cost=0.6597, reader_cost=0.0003 | ETA 06:02:48 2020-11-29 21:33:06 [INFO] [TRAIN] epoch=11, iter=7100/40000, loss=0.2439, lr=0.008388, batch_cost=0.6649, reader_cost=0.0005 | ETA 06:04:36 2020-11-29 21:34:12 [INFO] [TRAIN] epoch=11, iter=7200/40000, loss=0.2671, lr=0.008365, batch_cost=0.6619, reader_cost=0.0003 | ETA 06:01:49 2020-11-29 21:35:19 [INFO] [TRAIN] epoch=12, iter=7300/40000, loss=0.2529, lr=0.008342, batch_cost=0.6649, reader_cost=0.0067 | ETA 06:02:21 2020-11-29 21:36:26 [INFO] [TRAIN] epoch=12, iter=7400/40000, loss=0.2894, lr=0.008319, batch_cost=0.6696, reader_cost=0.0007 | ETA 06:03:48 2020-11-29 21:37:32 [INFO] [TRAIN] epoch=12, iter=7500/40000, loss=0.2582, lr=0.008296, batch_cost=0.6637, reader_cost=0.0005 | ETA 05:59:29 2020-11-29 21:38:39 [INFO] [TRAIN] epoch=12, iter=7600/40000, loss=0.2171, lr=0.008273, batch_cost=0.6633, reader_cost=0.0003 | ETA 05:58:10 2020-11-29 21:39:45 [INFO] [TRAIN] epoch=12, iter=7700/40000, loss=0.2430, lr=0.008250, batch_cost=0.6611, reader_cost=0.0004 | ETA 05:55:52 2020-11-29 21:40:51 [INFO] [TRAIN] epoch=12, iter=7800/40000, loss=0.2827, lr=0.008227, batch_cost=0.6631, reader_cost=0.0003 | ETA 05:55:52 2020-11-29 21:41:57 [INFO] [TRAIN] epoch=12, iter=7900/40000, loss=0.2273, lr=0.008204, batch_cost=0.6630, reader_cost=0.0002 | ETA 05:54:42 2020-11-29 21:43:04 [INFO] [TRAIN] epoch=13, iter=8000/40000, loss=0.2157, lr=0.008181, batch_cost=0.6699, reader_cost=0.0064 | ETA 05:57:16 2020-11-29 21:43:04 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-29 21:43:24 [INFO] [EVAL] #Images=1449 mIoU=0.7290 Acc=0.9372 Kappa=0.8601 2020-11-29 21:43:24 [INFO] [EVAL] Class IoU: [0.9321 0.8968 0.398 0.8417 0.6715 0.7207 0.833 0.7911 0.9041 0.3705 0.8295 0.5933 0.8654 0.8336 0.8155 0.82 0.5459 0.7776 0.4175 0.7833 0.6669] 2020-11-29 21:43:24 [INFO] [EVAL] Class Acc: [0.9576 0.9659 0.4121 0.9025 0.7405 0.893 0.9692 0.8689 0.9516 0.5581 0.8745 0.9007 0.9417 0.8856 0.913 0.9507 0.7505 0.9282 0.8222 0.8161 0.7962] 2020-11-29 21:43:28 [INFO] [EVAL] The model with the best validation mIoU (0.7290) was saved at iter 8000. 2020-11-29 21:44:34 [INFO] [TRAIN] epoch=13, iter=8100/40000, loss=0.2094, lr=0.008158, batch_cost=0.6662, reader_cost=0.0007 | ETA 05:54:11 2020-11-29 21:45:41 [INFO] [TRAIN] epoch=13, iter=8200/40000, loss=0.2339, lr=0.008135, batch_cost=0.6689, reader_cost=0.0004 | ETA 05:54:29 2020-11-29 21:46:48 [INFO] [TRAIN] epoch=13, iter=8300/40000, loss=0.1794, lr=0.008112, batch_cost=0.6665, reader_cost=0.0002 | ETA 05:52:08 2020-11-29 21:47:55 [INFO] [TRAIN] epoch=13, iter=8400/40000, loss=0.2261, lr=0.008089, batch_cost=0.6729, reader_cost=0.0007 | ETA 05:54:24 2020-11-29 21:49:01 [INFO] [TRAIN] epoch=13, iter=8500/40000, loss=0.2188, lr=0.008066, batch_cost=0.6617, reader_cost=0.0004 | ETA 05:47:23 2020-11-29 21:50:08 [INFO] [TRAIN] epoch=14, iter=8600/40000, loss=0.1925, lr=0.008043, batch_cost=0.6701, reader_cost=0.0062 | ETA 05:50:40 2020-11-29 21:51:15 [INFO] [TRAIN] epoch=14, iter=8700/40000, loss=0.1653, lr=0.008020, batch_cost=0.6683, reader_cost=0.0007 | ETA 05:48:38 2020-11-29 21:52:21 [INFO] [TRAIN] epoch=14, iter=8800/40000, loss=0.1828, lr=0.007996, batch_cost=0.6628, reader_cost=0.0003 | ETA 05:44:38 2020-11-29 21:53:28 [INFO] [TRAIN] epoch=14, iter=8900/40000, loss=0.1927, lr=0.007973, batch_cost=0.6679, reader_cost=0.0003 | ETA 05:46:12 2020-11-29 21:54:34 [INFO] [TRAIN] epoch=14, iter=9000/40000, loss=0.2439, lr=0.007950, batch_cost=0.6602, reader_cost=0.0002 | ETA 05:41:06 2020-11-29 21:55:41 [INFO] [TRAIN] epoch=14, iter=9100/40000, loss=0.2536, lr=0.007927, batch_cost=0.6672, reader_cost=0.0007 | ETA 05:43:36 2020-11-29 21:56:47 [INFO] [TRAIN] epoch=14, iter=9200/40000, loss=0.2798, lr=0.007904, batch_cost=0.6600, reader_cost=0.0004 | ETA 05:38:47 2020-11-29 21:57:54 [INFO] [TRAIN] epoch=15, iter=9300/40000, loss=0.2187, lr=0.007881, batch_cost=0.6715, reader_cost=0.0062 | ETA 05:43:35 2020-11-29 21:59:01 [INFO] [TRAIN] epoch=15, iter=9400/40000, loss=0.2024, lr=0.007858, batch_cost=0.6728, reader_cost=0.0009 | ETA 05:43:06 2020-11-29 22:00:08 [INFO] [TRAIN] epoch=15, iter=9500/40000, loss=0.2253, lr=0.007835, batch_cost=0.6647, reader_cost=0.0004 | ETA 05:37:52 2020-11-29 22:01:14 [INFO] [TRAIN] epoch=15, iter=9600/40000, loss=0.2160, lr=0.007812, batch_cost=0.6636, reader_cost=0.0006 | ETA 05:36:12 2020-11-29 22:02:20 [INFO] [TRAIN] epoch=15, iter=9700/40000, loss=0.1981, lr=0.007789, batch_cost=0.6597, reader_cost=0.0007 | ETA 05:33:08 2020-11-29 22:03:27 [INFO] [TRAIN] epoch=15, iter=9800/40000, loss=0.2161, lr=0.007765, batch_cost=0.6649, reader_cost=0.0009 | ETA 05:34:40 2020-11-29 22:04:32 [INFO] [TRAIN] epoch=15, iter=9900/40000, loss=0.1760, lr=0.007742, batch_cost=0.6580, reader_cost=0.0006 | ETA 05:30:07 2020-11-29 22:05:40 [INFO] [TRAIN] epoch=16, iter=10000/40000, loss=0.1965, lr=0.007719, batch_cost=0.6787, reader_cost=0.0067 | ETA 05:39:21 2020-11-29 22:06:47 [INFO] [TRAIN] epoch=16, iter=10100/40000, loss=0.1912, lr=0.007696, batch_cost=0.6642, reader_cost=0.0005 | ETA 05:30:59 2020-11-29 22:07:53 [INFO] [TRAIN] epoch=16, iter=10200/40000, loss=0.1983, lr=0.007673, batch_cost=0.6658, reader_cost=0.0006 | ETA 05:30:41 2020-11-29 22:08:59 [INFO] [TRAIN] epoch=16, iter=10300/40000, loss=0.1784, lr=0.007650, batch_cost=0.6599, reader_cost=0.0006 | ETA 05:26:39 2020-11-29 22:10:06 [INFO] [TRAIN] epoch=16, iter=10400/40000, loss=0.1755, lr=0.007626, batch_cost=0.6642, reader_cost=0.0008 | ETA 05:27:40 2020-11-29 22:11:12 [INFO] [TRAIN] epoch=16, iter=10500/40000, loss=0.2117, lr=0.007603, batch_cost=0.6638, reader_cost=0.0004 | ETA 05:26:22 2020-11-29 22:12:19 [INFO] [TRAIN] epoch=17, iter=10600/40000, loss=0.2172, lr=0.007580, batch_cost=0.6679, reader_cost=0.0057 | ETA 05:27:17 2020-11-29 22:13:25 [INFO] [TRAIN] epoch=17, iter=10700/40000, loss=0.1592, lr=0.007557, batch_cost=0.6605, reader_cost=0.0002 | ETA 05:22:33 2020-11-29 22:14:31 [INFO] [TRAIN] epoch=17, iter=10800/40000, loss=0.1754, lr=0.007534, batch_cost=0.6624, reader_cost=0.0002 | ETA 05:22:21 2020-11-29 22:15:38 [INFO] [TRAIN] epoch=17, iter=10900/40000, loss=0.1705, lr=0.007510, batch_cost=0.6667, reader_cost=0.0005 | ETA 05:23:20 2020-11-29 22:16:44 [INFO] [TRAIN] epoch=17, iter=11000/40000, loss=0.1567, lr=0.007487, batch_cost=0.6648, reader_cost=0.0010 | ETA 05:21:19 2020-11-29 22:17:50 [INFO] [TRAIN] epoch=17, iter=11100/40000, loss=0.2118, lr=0.007464, batch_cost=0.6612, reader_cost=0.0007 | ETA 05:18:28 2020-11-29 22:18:57 [INFO] [TRAIN] epoch=17, iter=11200/40000, loss=0.1566, lr=0.007441, batch_cost=0.6625, reader_cost=0.0006 | ETA 05:18:00 2020-11-29 22:20:04 [INFO] [TRAIN] epoch=18, iter=11300/40000, loss=0.1598, lr=0.007417, batch_cost=0.6732, reader_cost=0.0055 | ETA 05:22:00 2020-11-29 22:21:10 [INFO] [TRAIN] epoch=18, iter=11400/40000, loss=0.1858, lr=0.007394, batch_cost=0.6649, reader_cost=0.0002 | ETA 05:16:56 2020-11-29 22:22:17 [INFO] [TRAIN] epoch=18, iter=11500/40000, loss=0.1662, lr=0.007371, batch_cost=0.6634, reader_cost=0.0002 | ETA 05:15:05 2020-11-29 22:23:23 [INFO] [TRAIN] epoch=18, iter=11600/40000, loss=0.1870, lr=0.007348, batch_cost=0.6636, reader_cost=0.0003 | ETA 05:14:04 2020-11-29 22:24:29 [INFO] [TRAIN] epoch=18, iter=11700/40000, loss=0.1803, lr=0.007324, batch_cost=0.6614, reader_cost=0.0002 | ETA 05:11:56 2020-11-29 22:25:36 [INFO] [TRAIN] epoch=18, iter=11800/40000, loss=0.2104, lr=0.007301, batch_cost=0.6667, reader_cost=0.0003 | ETA 05:13:19 2020-11-29 22:26:43 [INFO] [TRAIN] epoch=19, iter=11900/40000, loss=0.1649, lr=0.007278, batch_cost=0.6711, reader_cost=0.0066 | ETA 05:14:17 2020-11-29 22:27:49 [INFO] [TRAIN] epoch=19, iter=12000/40000, loss=0.1965, lr=0.007254, batch_cost=0.6591, reader_cost=0.0002 | ETA 05:07:35 2020-11-29 22:27:49 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-29 22:28:06 [INFO] [EVAL] #Images=1449 mIoU=0.7587 Acc=0.9439 Kappa=0.8773 2020-11-29 22:28:06 [INFO] [EVAL] Class IoU: [0.9391 0.8813 0.4223 0.8743 0.6124 0.7669 0.9249 0.8545 0.8831 0.3601 0.908 0.6046 0.8415 0.8522 0.8556 0.8497 0.562 0.8386 0.5117 0.8704 0.7193] 2020-11-29 22:28:06 [INFO] [EVAL] Class Acc: [0.9693 0.9535 0.4412 0.9602 0.6689 0.8697 0.9491 0.932 0.9741 0.4884 0.9491 0.8957 0.8734 0.8986 0.9456 0.9322 0.7233 0.9162 0.679 0.9158 0.9057] 2020-11-29 22:28:10 [INFO] [EVAL] The model with the best validation mIoU (0.7587) was saved at iter 12000. 2020-11-29 22:29:16 [INFO] [TRAIN] epoch=19, iter=12100/40000, loss=0.2063, lr=0.007231, batch_cost=0.6584, reader_cost=0.0002 | ETA 05:06:08 2020-11-29 22:30:22 [INFO] [TRAIN] epoch=19, iter=12200/40000, loss=0.2081, lr=0.007208, batch_cost=0.6547, reader_cost=0.0002 | ETA 05:03:19 2020-11-29 22:31:28 [INFO] [TRAIN] epoch=19, iter=12300/40000, loss=0.2082, lr=0.007184, batch_cost=0.6583, reader_cost=0.0002 | ETA 05:03:55 2020-11-29 22:32:33 [INFO] [TRAIN] epoch=19, iter=12400/40000, loss=0.1862, lr=0.007161, batch_cost=0.6566, reader_cost=0.0002 | ETA 05:02:01 2020-11-29 22:33:39 [INFO] [TRAIN] epoch=19, iter=12500/40000, loss=0.1789, lr=0.007138, batch_cost=0.6588, reader_cost=0.0002 | ETA 05:01:56 2020-11-29 22:34:46 [INFO] [TRAIN] epoch=20, iter=12600/40000, loss=0.1860, lr=0.007114, batch_cost=0.6713, reader_cost=0.0058 | ETA 05:06:32 2020-11-29 22:35:53 [INFO] [TRAIN] epoch=20, iter=12700/40000, loss=0.1910, lr=0.007091, batch_cost=0.6625, reader_cost=0.0006 | ETA 05:01:26 2020-11-29 22:36:59 [INFO] [TRAIN] epoch=20, iter=12800/40000, loss=0.1919, lr=0.007068, batch_cost=0.6692, reader_cost=0.0009 | ETA 05:03:23 2020-11-29 22:38:06 [INFO] [TRAIN] epoch=20, iter=12900/40000, loss=0.1707, lr=0.007044, batch_cost=0.6618, reader_cost=0.0005 | ETA 04:58:55 2020-11-29 22:39:12 [INFO] [TRAIN] epoch=20, iter=13000/40000, loss=0.1811, lr=0.007021, batch_cost=0.6630, reader_cost=0.0003 | ETA 04:58:19 2020-11-29 22:40:18 [INFO] [TRAIN] epoch=20, iter=13100/40000, loss=0.1735, lr=0.006997, batch_cost=0.6568, reader_cost=0.0002 | ETA 04:54:27 2020-11-29 22:41:24 [INFO] [TRAIN] epoch=20, iter=13200/40000, loss=0.1697, lr=0.006974, batch_cost=0.6663, reader_cost=0.0002 | ETA 04:57:36 2020-11-29 22:42:31 [INFO] [TRAIN] epoch=21, iter=13300/40000, loss=0.1613, lr=0.006951, batch_cost=0.6662, reader_cost=0.0056 | ETA 04:56:28 2020-11-29 22:43:38 [INFO] [TRAIN] epoch=21, iter=13400/40000, loss=0.1770, lr=0.006927, batch_cost=0.6668, reader_cost=0.0006 | ETA 04:55:37 2020-11-29 22:44:45 [INFO] [TRAIN] epoch=21, iter=13500/40000, loss=0.1676, lr=0.006904, batch_cost=0.6712, reader_cost=0.0006 | ETA 04:56:27 2020-11-29 22:45:51 [INFO] [TRAIN] epoch=21, iter=13600/40000, loss=0.1393, lr=0.006880, batch_cost=0.6656, reader_cost=0.0005 | ETA 04:52:51 2020-11-29 22:46:58 [INFO] [TRAIN] epoch=21, iter=13700/40000, loss=0.1682, lr=0.006857, batch_cost=0.6686, reader_cost=0.0003 | ETA 04:53:04 2020-11-29 22:48:04 [INFO] [TRAIN] epoch=21, iter=13800/40000, loss=0.2227, lr=0.006833, batch_cost=0.6638, reader_cost=0.0004 | ETA 04:49:52 2020-11-29 22:49:11 [INFO] [TRAIN] epoch=22, iter=13900/40000, loss=0.1719, lr=0.006810, batch_cost=0.6680, reader_cost=0.0058 | ETA 04:50:35 2020-11-29 22:50:18 [INFO] [TRAIN] epoch=22, iter=14000/40000, loss=0.1538, lr=0.006786, batch_cost=0.6629, reader_cost=0.0003 | ETA 04:47:15 2020-11-29 22:51:24 [INFO] [TRAIN] epoch=22, iter=14100/40000, loss=0.1537, lr=0.006763, batch_cost=0.6632, reader_cost=0.0006 | ETA 04:46:16 2020-11-29 22:52:30 [INFO] [TRAIN] epoch=22, iter=14200/40000, loss=0.1587, lr=0.006739, batch_cost=0.6655, reader_cost=0.0002 | ETA 04:46:10 2020-11-29 22:53:37 [INFO] [TRAIN] epoch=22, iter=14300/40000, loss=0.1577, lr=0.006716, batch_cost=0.6660, reader_cost=0.0002 | ETA 04:45:17 2020-11-29 22:54:43 [INFO] [TRAIN] epoch=22, iter=14400/40000, loss=0.1313, lr=0.006692, batch_cost=0.6602, reader_cost=0.0002 | ETA 04:41:42 2020-11-29 22:55:50 [INFO] [TRAIN] epoch=22, iter=14500/40000, loss=0.1806, lr=0.006669, batch_cost=0.6676, reader_cost=0.0003 | ETA 04:43:43 2020-11-29 22:56:57 [INFO] [TRAIN] epoch=23, iter=14600/40000, loss=0.1299, lr=0.006645, batch_cost=0.6750, reader_cost=0.0069 | ETA 04:45:43 2020-11-29 22:58:04 [INFO] [TRAIN] epoch=23, iter=14700/40000, loss=0.1412, lr=0.006622, batch_cost=0.6674, reader_cost=0.0003 | ETA 04:41:24 2020-11-29 22:59:11 [INFO] [TRAIN] epoch=23, iter=14800/40000, loss=0.1600, lr=0.006598, batch_cost=0.6703, reader_cost=0.0005 | ETA 04:41:32 2020-11-29 23:00:18 [INFO] [TRAIN] epoch=23, iter=14900/40000, loss=0.1906, lr=0.006575, batch_cost=0.6703, reader_cost=0.0008 | ETA 04:40:24 2020-11-29 23:01:25 [INFO] [TRAIN] epoch=23, iter=15000/40000, loss=0.1719, lr=0.006551, batch_cost=0.6692, reader_cost=0.0008 | ETA 04:38:49 2020-11-29 23:02:32 [INFO] [TRAIN] epoch=23, iter=15100/40000, loss=0.2148, lr=0.006527, batch_cost=0.6709, reader_cost=0.0003 | ETA 04:38:25 2020-11-29 23:03:39 [INFO] [TRAIN] epoch=23, iter=15200/40000, loss=0.2018, lr=0.006504, batch_cost=0.6649, reader_cost=0.0003 | ETA 04:34:49 2020-11-29 23:04:46 [INFO] [TRAIN] epoch=24, iter=15300/40000, loss=0.1473, lr=0.006480, batch_cost=0.6715, reader_cost=0.0058 | ETA 04:36:27 2020-11-29 23:05:52 [INFO] [TRAIN] epoch=24, iter=15400/40000, loss=0.1505, lr=0.006457, batch_cost=0.6662, reader_cost=0.0003 | ETA 04:33:09 2020-11-29 23:06:59 [INFO] [TRAIN] epoch=24, iter=15500/40000, loss=0.1510, lr=0.006433, batch_cost=0.6631, reader_cost=0.0002 | ETA 04:30:44 2020-11-29 23:08:06 [INFO] [TRAIN] epoch=24, iter=15600/40000, loss=0.1708, lr=0.006409, batch_cost=0.6679, reader_cost=0.0002 | ETA 04:31:37 2020-11-29 23:09:12 [INFO] [TRAIN] epoch=24, iter=15700/40000, loss=0.1630, lr=0.006386, batch_cost=0.6619, reader_cost=0.0004 | ETA 04:28:03 2020-11-29 23:10:19 [INFO] [TRAIN] epoch=24, iter=15800/40000, loss=0.1800, lr=0.006362, batch_cost=0.6682, reader_cost=0.0002 | ETA 04:29:29 2020-11-29 23:11:26 [INFO] [TRAIN] epoch=25, iter=15900/40000, loss=0.1462, lr=0.006338, batch_cost=0.6708, reader_cost=0.0057 | ETA 04:29:26 2020-11-29 23:12:32 [INFO] [TRAIN] epoch=25, iter=16000/40000, loss=0.1401, lr=0.006315, batch_cost=0.6659, reader_cost=0.0003 | ETA 04:26:22 2020-11-29 23:12:32 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-29 23:12:53 [INFO] [EVAL] #Images=1449 mIoU=0.7581 Acc=0.9451 Kappa=0.8778 2020-11-29 23:12:53 [INFO] [EVAL] Class IoU: [0.9394 0.8988 0.4472 0.8796 0.6961 0.8341 0.894 0.8639 0.8683 0.36 0.848 0.5695 0.8257 0.8591 0.8872 0.8551 0.6087 0.8241 0.4289 0.8586 0.6746] 2020-11-29 23:12:53 [INFO] [EVAL] Class Acc: [0.9618 0.9208 0.474 0.9309 0.7455 0.9196 0.912 0.9218 0.9681 0.7012 0.9063 0.9269 0.8648 0.9578 0.9361 0.9167 0.8214 0.9569 0.7719 0.9627 0.9197] 2020-11-29 23:12:56 [INFO] [EVAL] The model with the best validation mIoU (0.7587) was saved at iter 12000. 2020-11-29 23:14:02 [INFO] [TRAIN] epoch=25, iter=16100/40000, loss=0.1552, lr=0.006291, batch_cost=0.6652, reader_cost=0.0004 | ETA 04:24:57 2020-11-29 23:15:08 [INFO] [TRAIN] epoch=25, iter=16200/40000, loss=0.1479, lr=0.006267, batch_cost=0.6609, reader_cost=0.0002 | ETA 04:22:09 2020-11-29 23:16:15 [INFO] [TRAIN] epoch=25, iter=16300/40000, loss=0.1377, lr=0.006244, batch_cost=0.6652, reader_cost=0.0002 | ETA 04:22:46 2020-11-29 23:17:21 [INFO] [TRAIN] epoch=25, iter=16400/40000, loss=0.1686, lr=0.006220, batch_cost=0.6653, reader_cost=0.0002 | ETA 04:21:39 2020-11-29 23:18:28 [INFO] [TRAIN] epoch=25, iter=16500/40000, loss=0.1663, lr=0.006196, batch_cost=0.6619, reader_cost=0.0002 | ETA 04:19:14 2020-11-29 23:19:35 [INFO] [TRAIN] epoch=26, iter=16600/40000, loss=0.1816, lr=0.006172, batch_cost=0.6701, reader_cost=0.0058 | ETA 04:21:19 2020-11-29 23:20:40 [INFO] [TRAIN] epoch=26, iter=16700/40000, loss=0.1235, lr=0.006149, batch_cost=0.6564, reader_cost=0.0002 | ETA 04:14:53 2020-11-29 23:21:47 [INFO] [TRAIN] epoch=26, iter=16800/40000, loss=0.1206, lr=0.006125, batch_cost=0.6647, reader_cost=0.0003 | ETA 04:17:00 2020-11-29 23:22:52 [INFO] [TRAIN] epoch=26, iter=16900/40000, loss=0.1412, lr=0.006101, batch_cost=0.6528, reader_cost=0.0002 | ETA 04:11:20 2020-11-29 23:23:58 [INFO] [TRAIN] epoch=26, iter=17000/40000, loss=0.1460, lr=0.006077, batch_cost=0.6653, reader_cost=0.0005 | ETA 04:15:00 2020-11-29 23:25:06 [INFO] [TRAIN] epoch=26, iter=17100/40000, loss=0.1810, lr=0.006054, batch_cost=0.6701, reader_cost=0.0004 | ETA 04:15:44 2020-11-29 23:26:13 [INFO] [TRAIN] epoch=27, iter=17200/40000, loss=0.1563, lr=0.006030, batch_cost=0.6737, reader_cost=0.0060 | ETA 04:15:59 2020-11-29 23:27:19 [INFO] [TRAIN] epoch=27, iter=17300/40000, loss=0.1599, lr=0.006006, batch_cost=0.6635, reader_cost=0.0001 | ETA 04:11:01 2020-11-29 23:28:26 [INFO] [TRAIN] epoch=27, iter=17400/40000, loss=0.1327, lr=0.005982, batch_cost=0.6646, reader_cost=0.0001 | ETA 04:10:20 2020-11-29 23:29:31 [INFO] [TRAIN] epoch=27, iter=17500/40000, loss=0.1434, lr=0.005958, batch_cost=0.6541, reader_cost=0.0002 | ETA 04:05:17 2020-11-29 23:30:37 [INFO] [TRAIN] epoch=27, iter=17600/40000, loss=0.1539, lr=0.005935, batch_cost=0.6587, reader_cost=0.0002 | ETA 04:05:55 2020-11-29 23:31:43 [INFO] [TRAIN] epoch=27, iter=17700/40000, loss=0.2092, lr=0.005911, batch_cost=0.6588, reader_cost=0.0002 | ETA 04:04:51 2020-11-29 23:32:49 [INFO] [TRAIN] epoch=27, iter=17800/40000, loss=0.1362, lr=0.005887, batch_cost=0.6646, reader_cost=0.0002 | ETA 04:05:53 2020-11-29 23:33:56 [INFO] [TRAIN] epoch=28, iter=17900/40000, loss=0.1316, lr=0.005863, batch_cost=0.6680, reader_cost=0.0058 | ETA 04:06:01 2020-11-29 23:35:02 [INFO] [TRAIN] epoch=28, iter=18000/40000, loss=0.1305, lr=0.005839, batch_cost=0.6590, reader_cost=0.0002 | ETA 04:01:37 2020-11-29 23:36:08 [INFO] [TRAIN] epoch=28, iter=18100/40000, loss=0.1269, lr=0.005815, batch_cost=0.6646, reader_cost=0.0004 | ETA 04:02:34 2020-11-29 23:37:15 [INFO] [TRAIN] epoch=28, iter=18200/40000, loss=0.1421, lr=0.005791, batch_cost=0.6663, reader_cost=0.0004 | ETA 04:02:06 2020-11-29 23:38:22 [INFO] [TRAIN] epoch=28, iter=18300/40000, loss=0.1366, lr=0.005767, batch_cost=0.6674, reader_cost=0.0008 | ETA 04:01:23 2020-11-29 23:39:28 [INFO] [TRAIN] epoch=28, iter=18400/40000, loss=0.1370, lr=0.005743, batch_cost=0.6597, reader_cost=0.0003 | ETA 03:57:30 2020-11-29 23:40:34 [INFO] [TRAIN] epoch=28, iter=18500/40000, loss=0.1371, lr=0.005720, batch_cost=0.6636, reader_cost=0.0002 | ETA 03:57:48 2020-11-29 23:41:42 [INFO] [TRAIN] epoch=29, iter=18600/40000, loss=0.1212, lr=0.005696, batch_cost=0.6731, reader_cost=0.0065 | ETA 04:00:04 2020-11-29 23:42:48 [INFO] [TRAIN] epoch=29, iter=18700/40000, loss=0.1276, lr=0.005672, batch_cost=0.6624, reader_cost=0.0002 | ETA 03:55:08 2020-11-29 23:43:54 [INFO] [TRAIN] epoch=29, iter=18800/40000, loss=0.1287, lr=0.005648, batch_cost=0.6610, reader_cost=0.0002 | ETA 03:53:32 2020-11-29 23:45:01 [INFO] [TRAIN] epoch=29, iter=18900/40000, loss=0.1400, lr=0.005624, batch_cost=0.6698, reader_cost=0.0008 | ETA 03:55:31 2020-11-29 23:46:08 [INFO] [TRAIN] epoch=29, iter=19000/40000, loss=0.1284, lr=0.005600, batch_cost=0.6668, reader_cost=0.0005 | ETA 03:53:23 2020-11-29 23:47:14 [INFO] [TRAIN] epoch=29, iter=19100/40000, loss=0.1137, lr=0.005576, batch_cost=0.6642, reader_cost=0.0005 | ETA 03:51:21 2020-11-29 23:48:21 [INFO] [TRAIN] epoch=30, iter=19200/40000, loss=0.1350, lr=0.005552, batch_cost=0.6688, reader_cost=0.0057 | ETA 03:51:51 2020-11-29 23:49:27 [INFO] [TRAIN] epoch=30, iter=19300/40000, loss=0.1078, lr=0.005528, batch_cost=0.6626, reader_cost=0.0003 | ETA 03:48:34 2020-11-29 23:50:34 [INFO] [TRAIN] epoch=30, iter=19400/40000, loss=0.1389, lr=0.005504, batch_cost=0.6653, reader_cost=0.0004 | ETA 03:48:24 2020-11-29 23:51:40 [INFO] [TRAIN] epoch=30, iter=19500/40000, loss=0.1185, lr=0.005480, batch_cost=0.6605, reader_cost=0.0004 | ETA 03:45:39 2020-11-29 23:52:46 [INFO] [TRAIN] epoch=30, iter=19600/40000, loss=0.1284, lr=0.005455, batch_cost=0.6636, reader_cost=0.0002 | ETA 03:45:36 2020-11-29 23:53:52 [INFO] [TRAIN] epoch=30, iter=19700/40000, loss=0.1401, lr=0.005431, batch_cost=0.6624, reader_cost=0.0004 | ETA 03:44:06 2020-11-29 23:54:58 [INFO] [TRAIN] epoch=30, iter=19800/40000, loss=0.1269, lr=0.005407, batch_cost=0.6579, reader_cost=0.0005 | ETA 03:41:29 2020-11-29 23:56:05 [INFO] [TRAIN] epoch=31, iter=19900/40000, loss=0.1108, lr=0.005383, batch_cost=0.6654, reader_cost=0.0059 | ETA 03:42:54 2020-11-29 23:57:10 [INFO] [TRAIN] epoch=31, iter=20000/40000, loss=0.1182, lr=0.005359, batch_cost=0.6549, reader_cost=0.0002 | ETA 03:38:18 2020-11-29 23:57:10 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-29 23:57:31 [INFO] [EVAL] #Images=1449 mIoU=0.7804 Acc=0.9505 Kappa=0.8900 2020-11-29 23:57:31 [INFO] [EVAL] Class IoU: [0.9441 0.8932 0.4371 0.8767 0.7011 0.8226 0.9227 0.8597 0.9066 0.4408 0.8971 0.5968 0.8483 0.9025 0.85 0.8635 0.6405 0.8391 0.5535 0.876 0.7159] 2020-11-29 23:57:31 [INFO] [EVAL] Class Acc: [0.9649 0.9296 0.4515 0.9374 0.7712 0.9015 0.98 0.9168 0.9749 0.7233 0.9243 0.9372 0.8764 0.9491 0.9491 0.9525 0.7991 0.954 0.8038 0.9464 0.94 ] 2020-11-29 23:57:35 [INFO] [EVAL] The model with the best validation mIoU (0.7804) was saved at iter 20000. 2020-11-29 23:58:42 [INFO] [TRAIN] epoch=31, iter=20100/40000, loss=0.1101, lr=0.005335, batch_cost=0.6654, reader_cost=0.0003 | ETA 03:40:41 2020-11-29 23:59:48 [INFO] [TRAIN] epoch=31, iter=20200/40000, loss=0.1326, lr=0.005311, batch_cost=0.6610, reader_cost=0.0003 | ETA 03:38:07 2020-11-30 00:00:55 [INFO] [TRAIN] epoch=31, iter=20300/40000, loss=0.1578, lr=0.005287, batch_cost=0.6691, reader_cost=0.0007 | ETA 03:39:40 2020-11-30 00:02:01 [INFO] [TRAIN] epoch=31, iter=20400/40000, loss=0.1240, lr=0.005263, batch_cost=0.6621, reader_cost=0.0005 | ETA 03:36:17 2020-11-30 00:03:08 [INFO] [TRAIN] epoch=32, iter=20500/40000, loss=0.1325, lr=0.005238, batch_cost=0.6682, reader_cost=0.0066 | ETA 03:37:09 2020-11-30 00:04:14 [INFO] [TRAIN] epoch=32, iter=20600/40000, loss=0.1156, lr=0.005214, batch_cost=0.6636, reader_cost=0.0003 | ETA 03:34:33 2020-11-30 00:05:20 [INFO] [TRAIN] epoch=32, iter=20700/40000, loss=0.1235, lr=0.005190, batch_cost=0.6599, reader_cost=0.0002 | ETA 03:32:16 2020-11-30 00:06:27 [INFO] [TRAIN] epoch=32, iter=20800/40000, loss=0.1106, lr=0.005166, batch_cost=0.6634, reader_cost=0.0002 | ETA 03:32:18 2020-11-30 00:07:33 [INFO] [TRAIN] epoch=32, iter=20900/40000, loss=0.1159, lr=0.005142, batch_cost=0.6668, reader_cost=0.0002 | ETA 03:32:15 2020-11-30 00:08:39 [INFO] [TRAIN] epoch=32, iter=21000/40000, loss=0.1267, lr=0.005117, batch_cost=0.6582, reader_cost=0.0004 | ETA 03:28:25 2020-11-30 00:09:45 [INFO] [TRAIN] epoch=32, iter=21100/40000, loss=0.1400, lr=0.005093, batch_cost=0.6610, reader_cost=0.0002 | ETA 03:28:11 2020-11-30 00:10:52 [INFO] [TRAIN] epoch=33, iter=21200/40000, loss=0.1260, lr=0.005069, batch_cost=0.6698, reader_cost=0.0065 | ETA 03:29:52 2020-11-30 00:11:59 [INFO] [TRAIN] epoch=33, iter=21300/40000, loss=0.1504, lr=0.005045, batch_cost=0.6650, reader_cost=0.0003 | ETA 03:27:15 2020-11-30 00:13:05 [INFO] [TRAIN] epoch=33, iter=21400/40000, loss=0.1300, lr=0.005020, batch_cost=0.6653, reader_cost=0.0003 | ETA 03:26:15 2020-11-30 00:14:11 [INFO] [TRAIN] epoch=33, iter=21500/40000, loss=0.1269, lr=0.004996, batch_cost=0.6612, reader_cost=0.0002 | ETA 03:23:52 2020-11-30 00:15:18 [INFO] [TRAIN] epoch=33, iter=21600/40000, loss=0.1201, lr=0.004972, batch_cost=0.6677, reader_cost=0.0008 | ETA 03:24:46 2020-11-30 00:16:25 [INFO] [TRAIN] epoch=33, iter=21700/40000, loss=0.1168, lr=0.004947, batch_cost=0.6656, reader_cost=0.0007 | ETA 03:22:59 2020-11-30 00:17:31 [INFO] [TRAIN] epoch=33, iter=21800/40000, loss=0.1251, lr=0.004923, batch_cost=0.6655, reader_cost=0.0005 | ETA 03:21:51 2020-11-30 00:18:38 [INFO] [TRAIN] epoch=34, iter=21900/40000, loss=0.1082, lr=0.004899, batch_cost=0.6688, reader_cost=0.0059 | ETA 03:21:46 2020-11-30 00:19:44 [INFO] [TRAIN] epoch=34, iter=22000/40000, loss=0.1137, lr=0.004874, batch_cost=0.6609, reader_cost=0.0002 | ETA 03:18:15 2020-11-30 00:20:51 [INFO] [TRAIN] epoch=34, iter=22100/40000, loss=0.1241, lr=0.004850, batch_cost=0.6656, reader_cost=0.0002 | ETA 03:18:33 2020-11-30 00:21:58 [INFO] [TRAIN] epoch=34, iter=22200/40000, loss=0.1077, lr=0.004826, batch_cost=0.6688, reader_cost=0.0006 | ETA 03:18:25 2020-11-30 00:23:05 [INFO] [TRAIN] epoch=34, iter=22300/40000, loss=0.1044, lr=0.004801, batch_cost=0.6696, reader_cost=0.0004 | ETA 03:17:31 2020-11-30 00:24:11 [INFO] [TRAIN] epoch=34, iter=22400/40000, loss=0.1250, lr=0.004777, batch_cost=0.6676, reader_cost=0.0007 | ETA 03:15:49 2020-11-30 00:25:19 [INFO] [TRAIN] epoch=35, iter=22500/40000, loss=0.1125, lr=0.004752, batch_cost=0.6711, reader_cost=0.0062 | ETA 03:15:45 2020-11-30 00:26:25 [INFO] [TRAIN] epoch=35, iter=22600/40000, loss=0.1124, lr=0.004728, batch_cost=0.6660, reader_cost=0.0003 | ETA 03:13:08 2020-11-30 00:27:32 [INFO] [TRAIN] epoch=35, iter=22700/40000, loss=0.1023, lr=0.004703, batch_cost=0.6642, reader_cost=0.0005 | ETA 03:11:30 2020-11-30 00:28:38 [INFO] [TRAIN] epoch=35, iter=22800/40000, loss=0.1028, lr=0.004679, batch_cost=0.6643, reader_cost=0.0004 | ETA 03:10:25 2020-11-30 00:29:45 [INFO] [TRAIN] epoch=35, iter=22900/40000, loss=0.1088, lr=0.004654, batch_cost=0.6653, reader_cost=0.0004 | ETA 03:09:37 2020-11-30 00:30:51 [INFO] [TRAIN] epoch=35, iter=23000/40000, loss=0.1059, lr=0.004630, batch_cost=0.6604, reader_cost=0.0002 | ETA 03:07:06 2020-11-30 00:31:57 [INFO] [TRAIN] epoch=35, iter=23100/40000, loss=0.1019, lr=0.004605, batch_cost=0.6598, reader_cost=0.0002 | ETA 03:05:50 2020-11-30 00:33:03 [INFO] [TRAIN] epoch=36, iter=23200/40000, loss=0.1105, lr=0.004581, batch_cost=0.6675, reader_cost=0.0055 | ETA 03:06:54 2020-11-30 00:34:10 [INFO] [TRAIN] epoch=36, iter=23300/40000, loss=0.1175, lr=0.004556, batch_cost=0.6631, reader_cost=0.0002 | ETA 03:04:33 2020-11-30 00:35:17 [INFO] [TRAIN] epoch=36, iter=23400/40000, loss=0.1085, lr=0.004532, batch_cost=0.6712, reader_cost=0.0003 | ETA 03:05:42 2020-11-30 00:36:24 [INFO] [TRAIN] epoch=36, iter=23500/40000, loss=0.1098, lr=0.004507, batch_cost=0.6700, reader_cost=0.0005 | ETA 03:04:15 2020-11-30 00:37:30 [INFO] [TRAIN] epoch=36, iter=23600/40000, loss=0.1272, lr=0.004483, batch_cost=0.6630, reader_cost=0.0002 | ETA 03:01:12 2020-11-30 00:38:37 [INFO] [TRAIN] epoch=36, iter=23700/40000, loss=0.0956, lr=0.004458, batch_cost=0.6658, reader_cost=0.0002 | ETA 03:00:52 2020-11-30 00:39:44 [INFO] [TRAIN] epoch=37, iter=23800/40000, loss=0.1075, lr=0.004433, batch_cost=0.6734, reader_cost=0.0061 | ETA 03:01:48 2020-11-30 00:40:50 [INFO] [TRAIN] epoch=37, iter=23900/40000, loss=0.1028, lr=0.004409, batch_cost=0.6621, reader_cost=0.0003 | ETA 02:57:40 2020-11-30 00:41:56 [INFO] [TRAIN] epoch=37, iter=24000/40000, loss=0.1135, lr=0.004384, batch_cost=0.6609, reader_cost=0.0003 | ETA 02:56:14 2020-11-30 00:41:56 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-30 00:42:16 [INFO] [EVAL] #Images=1449 mIoU=0.7841 Acc=0.9515 Kappa=0.8927 2020-11-30 00:42:16 [INFO] [EVAL] Class IoU: [0.9447 0.8979 0.4197 0.887 0.7153 0.8139 0.9524 0.8928 0.8995 0.4753 0.8885 0.5022 0.8757 0.8727 0.891 0.8626 0.6271 0.8328 0.606 0.8938 0.7144] 2020-11-30 00:42:16 [INFO] [EVAL] Class Acc: [0.9673 0.9321 0.4286 0.9379 0.8216 0.8771 0.9728 0.9419 0.9294 0.7027 0.9653 0.9495 0.9531 0.906 0.9392 0.9474 0.8552 0.897 0.8063 0.937 0.8706] 2020-11-30 00:42:20 [INFO] [EVAL] The model with the best validation mIoU (0.7841) was saved at iter 24000. 2020-11-30 00:43:27 [INFO] [TRAIN] epoch=37, iter=24100/40000, loss=0.1009, lr=0.004359, batch_cost=0.6660, reader_cost=0.0005 | ETA 02:56:29 2020-11-30 00:44:33 [INFO] [TRAIN] epoch=37, iter=24200/40000, loss=0.1017, lr=0.004335, batch_cost=0.6647, reader_cost=0.0004 | ETA 02:55:02 2020-11-30 00:45:39 [INFO] [TRAIN] epoch=37, iter=24300/40000, loss=0.1076, lr=0.004310, batch_cost=0.6608, reader_cost=0.0002 | ETA 02:52:54 2020-11-30 00:46:45 [INFO] [TRAIN] epoch=37, iter=24400/40000, loss=0.1098, lr=0.004285, batch_cost=0.6623, reader_cost=0.0003 | ETA 02:52:12 2020-11-30 00:47:52 [INFO] [TRAIN] epoch=38, iter=24500/40000, loss=0.1135, lr=0.004261, batch_cost=0.6656, reader_cost=0.0061 | ETA 02:51:56 2020-11-30 00:48:58 [INFO] [TRAIN] epoch=38, iter=24600/40000, loss=0.1033, lr=0.004236, batch_cost=0.6603, reader_cost=0.0004 | ETA 02:49:28 2020-11-30 00:50:04 [INFO] [TRAIN] epoch=38, iter=24700/40000, loss=0.1007, lr=0.004211, batch_cost=0.6600, reader_cost=0.0002 | ETA 02:48:17 2020-11-30 00:51:10 [INFO] [TRAIN] epoch=38, iter=24800/40000, loss=0.0971, lr=0.004186, batch_cost=0.6594, reader_cost=0.0002 | ETA 02:47:02 2020-11-30 00:52:16 [INFO] [TRAIN] epoch=38, iter=24900/40000, loss=0.1025, lr=0.004162, batch_cost=0.6614, reader_cost=0.0003 | ETA 02:46:26 2020-11-30 00:53:22 [INFO] [TRAIN] epoch=38, iter=25000/40000, loss=0.0934, lr=0.004137, batch_cost=0.6613, reader_cost=0.0004 | ETA 02:45:19 2020-11-30 00:54:28 [INFO] [TRAIN] epoch=38, iter=25100/40000, loss=0.1057, lr=0.004112, batch_cost=0.6595, reader_cost=0.0006 | ETA 02:43:47 2020-11-30 00:55:35 [INFO] [TRAIN] epoch=39, iter=25200/40000, loss=0.0952, lr=0.004087, batch_cost=0.6637, reader_cost=0.0053 | ETA 02:43:43 2020-11-30 00:56:41 [INFO] [TRAIN] epoch=39, iter=25300/40000, loss=0.1013, lr=0.004062, batch_cost=0.6624, reader_cost=0.0002 | ETA 02:42:17 2020-11-30 00:57:47 [INFO] [TRAIN] epoch=39, iter=25400/40000, loss=0.0985, lr=0.004037, batch_cost=0.6621, reader_cost=0.0002 | ETA 02:41:06 2020-11-30 00:58:53 [INFO] [TRAIN] epoch=39, iter=25500/40000, loss=0.1058, lr=0.004012, batch_cost=0.6634, reader_cost=0.0002 | ETA 02:40:19 2020-11-30 01:00:00 [INFO] [TRAIN] epoch=39, iter=25600/40000, loss=0.0873, lr=0.003987, batch_cost=0.6678, reader_cost=0.0002 | ETA 02:40:16 2020-11-30 01:01:07 [INFO] [TRAIN] epoch=39, iter=25700/40000, loss=0.0970, lr=0.003963, batch_cost=0.6668, reader_cost=0.0002 | ETA 02:38:55 2020-11-30 01:02:14 [INFO] [TRAIN] epoch=40, iter=25800/40000, loss=0.1104, lr=0.003938, batch_cost=0.6722, reader_cost=0.0063 | ETA 02:39:05 2020-11-30 01:03:20 [INFO] [TRAIN] epoch=40, iter=25900/40000, loss=0.1038, lr=0.003913, batch_cost=0.6639, reader_cost=0.0002 | ETA 02:36:01 2020-11-30 01:04:27 [INFO] [TRAIN] epoch=40, iter=26000/40000, loss=0.1062, lr=0.003888, batch_cost=0.6606, reader_cost=0.0003 | ETA 02:34:08 2020-11-30 01:05:33 [INFO] [TRAIN] epoch=40, iter=26100/40000, loss=0.1216, lr=0.003863, batch_cost=0.6607, reader_cost=0.0003 | ETA 02:33:03 2020-11-30 01:06:39 [INFO] [TRAIN] epoch=40, iter=26200/40000, loss=0.0991, lr=0.003838, batch_cost=0.6599, reader_cost=0.0002 | ETA 02:31:46 2020-11-30 01:07:44 [INFO] [TRAIN] epoch=40, iter=26300/40000, loss=0.0968, lr=0.003813, batch_cost=0.6590, reader_cost=0.0002 | ETA 02:30:28 2020-11-30 01:08:51 [INFO] [TRAIN] epoch=40, iter=26400/40000, loss=0.1030, lr=0.003788, batch_cost=0.6627, reader_cost=0.0002 | ETA 02:30:13 2020-11-30 01:09:57 [INFO] [TRAIN] epoch=41, iter=26500/40000, loss=0.1002, lr=0.003762, batch_cost=0.6656, reader_cost=0.0054 | ETA 02:29:44 2020-11-30 01:11:04 [INFO] [TRAIN] epoch=41, iter=26600/40000, loss=0.0947, lr=0.003737, batch_cost=0.6635, reader_cost=0.0002 | ETA 02:28:11 2020-11-30 01:12:09 [INFO] [TRAIN] epoch=41, iter=26700/40000, loss=0.1064, lr=0.003712, batch_cost=0.6549, reader_cost=0.0002 | ETA 02:25:10 2020-11-30 01:13:16 [INFO] [TRAIN] epoch=41, iter=26800/40000, loss=0.0963, lr=0.003687, batch_cost=0.6668, reader_cost=0.0002 | ETA 02:26:42 2020-11-30 01:14:22 [INFO] [TRAIN] epoch=41, iter=26900/40000, loss=0.1067, lr=0.003662, batch_cost=0.6581, reader_cost=0.0002 | ETA 02:23:40 2020-11-30 01:15:28 [INFO] [TRAIN] epoch=41, iter=27000/40000, loss=0.1004, lr=0.003637, batch_cost=0.6616, reader_cost=0.0002 | ETA 02:23:21 2020-11-30 01:16:34 [INFO] [TRAIN] epoch=41, iter=27100/40000, loss=0.0925, lr=0.003612, batch_cost=0.6583, reader_cost=0.0002 | ETA 02:21:31 2020-11-30 01:17:41 [INFO] [TRAIN] epoch=42, iter=27200/40000, loss=0.0895, lr=0.003586, batch_cost=0.6701, reader_cost=0.0056 | ETA 02:22:57 2020-11-30 01:18:47 [INFO] [TRAIN] epoch=42, iter=27300/40000, loss=0.1049, lr=0.003561, batch_cost=0.6608, reader_cost=0.0003 | ETA 02:19:52 2020-11-30 01:19:53 [INFO] [TRAIN] epoch=42, iter=27400/40000, loss=0.1065, lr=0.003536, batch_cost=0.6618, reader_cost=0.0004 | ETA 02:18:58 2020-11-30 01:20:59 [INFO] [TRAIN] epoch=42, iter=27500/40000, loss=0.1112, lr=0.003511, batch_cost=0.6659, reader_cost=0.0002 | ETA 02:18:43 2020-11-30 01:22:06 [INFO] [TRAIN] epoch=42, iter=27600/40000, loss=0.1007, lr=0.003485, batch_cost=0.6616, reader_cost=0.0002 | ETA 02:16:43 2020-11-30 01:23:12 [INFO] [TRAIN] epoch=42, iter=27700/40000, loss=0.0943, lr=0.003460, batch_cost=0.6641, reader_cost=0.0006 | ETA 02:16:08 2020-11-30 01:24:19 [INFO] [TRAIN] epoch=43, iter=27800/40000, loss=0.1084, lr=0.003435, batch_cost=0.6676, reader_cost=0.0049 | ETA 02:15:44 2020-11-30 01:25:25 [INFO] [TRAIN] epoch=43, iter=27900/40000, loss=0.0947, lr=0.003409, batch_cost=0.6614, reader_cost=0.0003 | ETA 02:13:22 2020-11-30 01:26:31 [INFO] [TRAIN] epoch=43, iter=28000/40000, loss=0.0942, lr=0.003384, batch_cost=0.6640, reader_cost=0.0003 | ETA 02:12:48 2020-11-30 01:26:31 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-30 01:26:51 [INFO] [EVAL] #Images=1449 mIoU=0.7897 Acc=0.9534 Kappa=0.8963 2020-11-30 01:26:51 [INFO] [EVAL] Class IoU: [0.9466 0.918 0.4489 0.8837 0.7216 0.844 0.9453 0.8955 0.9432 0.421 0.8918 0.4392 0.902 0.8786 0.8759 0.8748 0.6509 0.9059 0.5593 0.8926 0.7456] 2020-11-30 01:26:51 [INFO] [EVAL] Class Acc: [0.9663 0.9625 0.4662 0.9328 0.7938 0.9135 0.9646 0.9473 0.9644 0.6801 0.9772 0.9476 0.9361 0.912 0.9356 0.9468 0.8441 0.9697 0.7975 0.9485 0.9175] 2020-11-30 01:26:56 [INFO] [EVAL] The model with the best validation mIoU (0.7897) was saved at iter 28000. 2020-11-30 01:28:02 [INFO] [TRAIN] epoch=43, iter=28100/40000, loss=0.0937, lr=0.003359, batch_cost=0.6659, reader_cost=0.0005 | ETA 02:12:03 2020-11-30 01:29:09 [INFO] [TRAIN] epoch=43, iter=28200/40000, loss=0.0957, lr=0.003333, batch_cost=0.6633, reader_cost=0.0004 | ETA 02:10:26 2020-11-30 01:30:15 [INFO] [TRAIN] epoch=43, iter=28300/40000, loss=0.0953, lr=0.003308, batch_cost=0.6617, reader_cost=0.0002 | ETA 02:09:01 2020-11-30 01:31:22 [INFO] [TRAIN] epoch=43, iter=28400/40000, loss=0.0972, lr=0.003282, batch_cost=0.6688, reader_cost=0.0003 | ETA 02:09:17 2020-11-30 01:32:29 [INFO] [TRAIN] epoch=44, iter=28500/40000, loss=0.0879, lr=0.003257, batch_cost=0.6724, reader_cost=0.0058 | ETA 02:08:52 2020-11-30 01:33:36 [INFO] [TRAIN] epoch=44, iter=28600/40000, loss=0.0927, lr=0.003231, batch_cost=0.6675, reader_cost=0.0002 | ETA 02:06:49 2020-11-30 01:34:42 [INFO] [TRAIN] epoch=44, iter=28700/40000, loss=0.0866, lr=0.003206, batch_cost=0.6649, reader_cost=0.0002 | ETA 02:05:13 2020-11-30 01:35:48 [INFO] [TRAIN] epoch=44, iter=28800/40000, loss=0.0898, lr=0.003180, batch_cost=0.6613, reader_cost=0.0002 | ETA 02:03:26 2020-11-30 01:36:55 [INFO] [TRAIN] epoch=44, iter=28900/40000, loss=0.0911, lr=0.003155, batch_cost=0.6676, reader_cost=0.0006 | ETA 02:03:30 2020-11-30 01:38:02 [INFO] [TRAIN] epoch=44, iter=29000/40000, loss=0.0957, lr=0.003129, batch_cost=0.6663, reader_cost=0.0004 | ETA 02:02:09 2020-11-30 01:39:08 [INFO] [TRAIN] epoch=45, iter=29100/40000, loss=0.0983, lr=0.003104, batch_cost=0.6658, reader_cost=0.0052 | ETA 02:00:56 2020-11-30 01:40:14 [INFO] [TRAIN] epoch=45, iter=29200/40000, loss=0.0890, lr=0.003078, batch_cost=0.6602, reader_cost=0.0002 | ETA 01:58:49 2020-11-30 01:41:20 [INFO] [TRAIN] epoch=45, iter=29300/40000, loss=0.0915, lr=0.003052, batch_cost=0.6555, reader_cost=0.0002 | ETA 01:56:54 2020-11-30 01:42:26 [INFO] [TRAIN] epoch=45, iter=29400/40000, loss=0.0948, lr=0.003027, batch_cost=0.6559, reader_cost=0.0002 | ETA 01:55:52 2020-11-30 01:43:32 [INFO] [TRAIN] epoch=45, iter=29500/40000, loss=0.0938, lr=0.003001, batch_cost=0.6618, reader_cost=0.0002 | ETA 01:55:48 2020-11-30 01:44:37 [INFO] [TRAIN] epoch=45, iter=29600/40000, loss=0.0905, lr=0.002975, batch_cost=0.6554, reader_cost=0.0001 | ETA 01:53:36 2020-11-30 01:45:43 [INFO] [TRAIN] epoch=45, iter=29700/40000, loss=0.0897, lr=0.002949, batch_cost=0.6604, reader_cost=0.0002 | ETA 01:53:22 2020-11-30 01:46:50 [INFO] [TRAIN] epoch=46, iter=29800/40000, loss=0.0957, lr=0.002924, batch_cost=0.6683, reader_cost=0.0063 | ETA 01:53:37 2020-11-30 01:47:56 [INFO] [TRAIN] epoch=46, iter=29900/40000, loss=0.0930, lr=0.002898, batch_cost=0.6626, reader_cost=0.0003 | ETA 01:51:32 2020-11-30 01:49:03 [INFO] [TRAIN] epoch=46, iter=30000/40000, loss=0.0907, lr=0.002872, batch_cost=0.6626, reader_cost=0.0005 | ETA 01:50:26 2020-11-30 01:50:09 [INFO] [TRAIN] epoch=46, iter=30100/40000, loss=0.0895, lr=0.002846, batch_cost=0.6626, reader_cost=0.0002 | ETA 01:49:19 2020-11-30 01:51:15 [INFO] [TRAIN] epoch=46, iter=30200/40000, loss=0.0893, lr=0.002820, batch_cost=0.6603, reader_cost=0.0005 | ETA 01:47:50 2020-11-30 01:52:21 [INFO] [TRAIN] epoch=46, iter=30300/40000, loss=0.0924, lr=0.002794, batch_cost=0.6638, reader_cost=0.0005 | ETA 01:47:18 2020-11-30 01:53:27 [INFO] [TRAIN] epoch=46, iter=30400/40000, loss=0.0913, lr=0.002768, batch_cost=0.6537, reader_cost=0.0003 | ETA 01:44:35 2020-11-30 01:54:33 [INFO] [TRAIN] epoch=47, iter=30500/40000, loss=0.0869, lr=0.002742, batch_cost=0.6637, reader_cost=0.0058 | ETA 01:45:05 2020-11-30 01:55:38 [INFO] [TRAIN] epoch=47, iter=30600/40000, loss=0.0825, lr=0.002716, batch_cost=0.6499, reader_cost=0.0001 | ETA 01:41:48 2020-11-30 01:56:44 [INFO] [TRAIN] epoch=47, iter=30700/40000, loss=0.0863, lr=0.002690, batch_cost=0.6559, reader_cost=0.0002 | ETA 01:41:39 2020-11-30 01:57:49 [INFO] [TRAIN] epoch=47, iter=30800/40000, loss=0.0948, lr=0.002664, batch_cost=0.6561, reader_cost=0.0002 | ETA 01:40:36 2020-11-30 01:58:55 [INFO] [TRAIN] epoch=47, iter=30900/40000, loss=0.0841, lr=0.002638, batch_cost=0.6538, reader_cost=0.0002 | ETA 01:39:09 2020-11-30 02:00:00 [INFO] [TRAIN] epoch=47, iter=31000/40000, loss=0.0911, lr=0.002612, batch_cost=0.6527, reader_cost=0.0002 | ETA 01:37:54 2020-11-30 02:01:06 [INFO] [TRAIN] epoch=48, iter=31100/40000, loss=0.0889, lr=0.002586, batch_cost=0.6621, reader_cost=0.0058 | ETA 01:38:12 2020-11-30 02:02:11 [INFO] [TRAIN] epoch=48, iter=31200/40000, loss=0.0791, lr=0.002560, batch_cost=0.6521, reader_cost=0.0001 | ETA 01:35:38 2020-11-30 02:03:17 [INFO] [TRAIN] epoch=48, iter=31300/40000, loss=0.0830, lr=0.002534, batch_cost=0.6553, reader_cost=0.0001 | ETA 01:35:01 2020-11-30 02:04:22 [INFO] [TRAIN] epoch=48, iter=31400/40000, loss=0.0882, lr=0.002507, batch_cost=0.6539, reader_cost=0.0002 | ETA 01:33:43 2020-11-30 02:05:28 [INFO] [TRAIN] epoch=48, iter=31500/40000, loss=0.0808, lr=0.002481, batch_cost=0.6525, reader_cost=0.0001 | ETA 01:32:25 2020-11-30 02:06:33 [INFO] [TRAIN] epoch=48, iter=31600/40000, loss=0.0907, lr=0.002455, batch_cost=0.6540, reader_cost=0.0002 | ETA 01:31:33 2020-11-30 02:07:38 [INFO] [TRAIN] epoch=48, iter=31700/40000, loss=0.0872, lr=0.002429, batch_cost=0.6513, reader_cost=0.0001 | ETA 01:30:05 2020-11-30 02:08:44 [INFO] [TRAIN] epoch=49, iter=31800/40000, loss=0.0886, lr=0.002402, batch_cost=0.6590, reader_cost=0.0055 | ETA 01:30:04 2020-11-30 02:09:49 [INFO] [TRAIN] epoch=49, iter=31900/40000, loss=0.0848, lr=0.002376, batch_cost=0.6510, reader_cost=0.0001 | ETA 01:27:53 2020-11-30 02:10:54 [INFO] [TRAIN] epoch=49, iter=32000/40000, loss=0.0899, lr=0.002350, batch_cost=0.6541, reader_cost=0.0001 | ETA 01:27:13 2020-11-30 02:10:55 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-30 02:11:12 [INFO] [EVAL] #Images=1449 mIoU=0.8066 Acc=0.9568 Kappa=0.9044 2020-11-30 02:11:12 [INFO] [EVAL] Class IoU: [0.9499 0.9137 0.4466 0.8879 0.7414 0.8515 0.9516 0.8871 0.9482 0.4733 0.9147 0.5457 0.906 0.901 0.8699 0.8759 0.6877 0.8917 0.6331 0.9003 0.7618] 2020-11-30 02:11:12 [INFO] [EVAL] Class Acc: [0.9693 0.9573 0.4639 0.9379 0.8508 0.9012 0.9785 0.944 0.9699 0.6944 0.9602 0.9482 0.9433 0.9401 0.9457 0.952 0.7961 0.9708 0.8313 0.9578 0.9271] 2020-11-30 02:11:16 [INFO] [EVAL] The model with the best validation mIoU (0.8066) was saved at iter 32000. 2020-11-30 02:12:22 [INFO] [TRAIN] epoch=49, iter=32100/40000, loss=0.0908, lr=0.002323, batch_cost=0.6597, reader_cost=0.0002 | ETA 01:26:51 2020-11-30 02:13:28 [INFO] [TRAIN] epoch=49, iter=32200/40000, loss=0.0877, lr=0.002297, batch_cost=0.6578, reader_cost=0.0002 | ETA 01:25:30 2020-11-30 02:14:33 [INFO] [TRAIN] epoch=49, iter=32300/40000, loss=0.0921, lr=0.002270, batch_cost=0.6546, reader_cost=0.0002 | ETA 01:24:00 2020-11-30 02:15:39 [INFO] [TRAIN] epoch=50, iter=32400/40000, loss=0.0988, lr=0.002244, batch_cost=0.6582, reader_cost=0.0060 | ETA 01:23:22 2020-11-30 02:16:44 [INFO] [TRAIN] epoch=50, iter=32500/40000, loss=0.0794, lr=0.002217, batch_cost=0.6528, reader_cost=0.0001 | ETA 01:21:36 2020-11-30 02:17:50 [INFO] [TRAIN] epoch=50, iter=32600/40000, loss=0.0906, lr=0.002190, batch_cost=0.6582, reader_cost=0.0002 | ETA 01:21:10 2020-11-30 02:18:55 [INFO] [TRAIN] epoch=50, iter=32700/40000, loss=0.0856, lr=0.002164, batch_cost=0.6527, reader_cost=0.0002 | ETA 01:19:24 2020-11-30 02:20:01 [INFO] [TRAIN] epoch=50, iter=32800/40000, loss=0.0845, lr=0.002137, batch_cost=0.6561, reader_cost=0.0002 | ETA 01:18:43 2020-11-30 02:21:07 [INFO] [TRAIN] epoch=50, iter=32900/40000, loss=0.0827, lr=0.002110, batch_cost=0.6586, reader_cost=0.0002 | ETA 01:17:56 2020-11-30 02:22:12 [INFO] [TRAIN] epoch=50, iter=33000/40000, loss=0.0821, lr=0.002083, batch_cost=0.6554, reader_cost=0.0002 | ETA 01:16:28 2020-11-30 02:23:19 [INFO] [TRAIN] epoch=51, iter=33100/40000, loss=0.0926, lr=0.002057, batch_cost=0.6618, reader_cost=0.0057 | ETA 01:16:06 2020-11-30 02:24:24 [INFO] [TRAIN] epoch=51, iter=33200/40000, loss=0.0802, lr=0.002030, batch_cost=0.6543, reader_cost=0.0002 | ETA 01:14:09 2020-11-30 02:25:29 [INFO] [TRAIN] epoch=51, iter=33300/40000, loss=0.0860, lr=0.002003, batch_cost=0.6514, reader_cost=0.0002 | ETA 01:12:44 2020-11-30 02:26:35 [INFO] [TRAIN] epoch=51, iter=33400/40000, loss=0.0821, lr=0.001976, batch_cost=0.6549, reader_cost=0.0002 | ETA 01:12:02 2020-11-30 02:27:40 [INFO] [TRAIN] epoch=51, iter=33500/40000, loss=0.0843, lr=0.001949, batch_cost=0.6552, reader_cost=0.0002 | ETA 01:10:58 2020-11-30 02:28:45 [INFO] [TRAIN] epoch=51, iter=33600/40000, loss=0.0898, lr=0.001922, batch_cost=0.6510, reader_cost=0.0002 | ETA 01:09:26 2020-11-30 02:29:51 [INFO] [TRAIN] epoch=51, iter=33700/40000, loss=0.0853, lr=0.001895, batch_cost=0.6542, reader_cost=0.0002 | ETA 01:08:41 2020-11-30 02:30:57 [INFO] [TRAIN] epoch=52, iter=33800/40000, loss=0.0830, lr=0.001868, batch_cost=0.6585, reader_cost=0.0053 | ETA 01:08:02 2020-11-30 02:32:02 [INFO] [TRAIN] epoch=52, iter=33900/40000, loss=0.0878, lr=0.001841, batch_cost=0.6556, reader_cost=0.0002 | ETA 01:06:39 2020-11-30 02:33:07 [INFO] [TRAIN] epoch=52, iter=34000/40000, loss=0.0773, lr=0.001814, batch_cost=0.6529, reader_cost=0.0002 | ETA 01:05:17 2020-11-30 02:34:13 [INFO] [TRAIN] epoch=52, iter=34100/40000, loss=0.0899, lr=0.001786, batch_cost=0.6522, reader_cost=0.0002 | ETA 01:04:07 2020-11-30 02:35:18 [INFO] [TRAIN] epoch=52, iter=34200/40000, loss=0.0863, lr=0.001759, batch_cost=0.6491, reader_cost=0.0002 | ETA 01:02:44 2020-11-30 02:36:23 [INFO] [TRAIN] epoch=52, iter=34300/40000, loss=0.0846, lr=0.001732, batch_cost=0.6519, reader_cost=0.0002 | ETA 01:01:55 2020-11-30 02:37:29 [INFO] [TRAIN] epoch=53, iter=34400/40000, loss=0.0853, lr=0.001704, batch_cost=0.6605, reader_cost=0.0058 | ETA 01:01:39 2020-11-30 02:38:34 [INFO] [TRAIN] epoch=53, iter=34500/40000, loss=0.0761, lr=0.001677, batch_cost=0.6508, reader_cost=0.0001 | ETA 00:59:39 2020-11-30 02:39:40 [INFO] [TRAIN] epoch=53, iter=34600/40000, loss=0.0844, lr=0.001650, batch_cost=0.6572, reader_cost=0.0003 | ETA 00:59:08 2020-11-30 02:40:45 [INFO] [TRAIN] epoch=53, iter=34700/40000, loss=0.0824, lr=0.001622, batch_cost=0.6517, reader_cost=0.0002 | ETA 00:57:34 2020-11-30 02:41:50 [INFO] [TRAIN] epoch=53, iter=34800/40000, loss=0.0878, lr=0.001594, batch_cost=0.6539, reader_cost=0.0002 | ETA 00:56:40 2020-11-30 02:42:55 [INFO] [TRAIN] epoch=53, iter=34900/40000, loss=0.0902, lr=0.001567, batch_cost=0.6532, reader_cost=0.0002 | ETA 00:55:31 2020-11-30 02:44:01 [INFO] [TRAIN] epoch=53, iter=35000/40000, loss=0.0910, lr=0.001539, batch_cost=0.6570, reader_cost=0.0002 | ETA 00:54:44 2020-11-30 02:45:07 [INFO] [TRAIN] epoch=54, iter=35100/40000, loss=0.0785, lr=0.001511, batch_cost=0.6604, reader_cost=0.0062 | ETA 00:53:55 2020-11-30 02:46:12 [INFO] [TRAIN] epoch=54, iter=35200/40000, loss=0.0857, lr=0.001484, batch_cost=0.6494, reader_cost=0.0003 | ETA 00:51:57 2020-11-30 02:47:17 [INFO] [TRAIN] epoch=54, iter=35300/40000, loss=0.0853, lr=0.001456, batch_cost=0.6521, reader_cost=0.0002 | ETA 00:51:04 2020-11-30 02:48:23 [INFO] [TRAIN] epoch=54, iter=35400/40000, loss=0.0871, lr=0.001428, batch_cost=0.6570, reader_cost=0.0002 | ETA 00:50:21 2020-11-30 02:49:29 [INFO] [TRAIN] epoch=54, iter=35500/40000, loss=0.0879, lr=0.001400, batch_cost=0.6544, reader_cost=0.0001 | ETA 00:49:04 2020-11-30 02:50:34 [INFO] [TRAIN] epoch=54, iter=35600/40000, loss=0.0867, lr=0.001372, batch_cost=0.6525, reader_cost=0.0002 | ETA 00:47:50 2020-11-30 02:51:40 [INFO] [TRAIN] epoch=55, iter=35700/40000, loss=0.0766, lr=0.001344, batch_cost=0.6598, reader_cost=0.0063 | ETA 00:47:17 2020-11-30 02:52:45 [INFO] [TRAIN] epoch=55, iter=35800/40000, loss=0.0834, lr=0.001316, batch_cost=0.6536, reader_cost=0.0002 | ETA 00:45:45 2020-11-30 02:53:51 [INFO] [TRAIN] epoch=55, iter=35900/40000, loss=0.0820, lr=0.001287, batch_cost=0.6595, reader_cost=0.0001 | ETA 00:45:03 2020-11-30 02:54:57 [INFO] [TRAIN] epoch=55, iter=36000/40000, loss=0.0815, lr=0.001259, batch_cost=0.6590, reader_cost=0.0002 | ETA 00:43:56 2020-11-30 02:54:57 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-30 02:55:15 [INFO] [EVAL] #Images=1449 mIoU=0.7973 Acc=0.9553 Kappa=0.9006 2020-11-30 02:55:15 [INFO] [EVAL] Class IoU: [0.9483 0.9151 0.4489 0.8951 0.7183 0.8095 0.9519 0.8885 0.9433 0.4653 0.9022 0.5401 0.9044 0.88 0.8706 0.88 0.643 0.9086 0.586 0.8729 0.7718] 2020-11-30 02:55:15 [INFO] [EVAL] Class Acc: [0.9669 0.9623 0.4655 0.9386 0.8254 0.9084 0.9804 0.9446 0.9648 0.7098 0.9724 0.9432 0.9412 0.9157 0.9391 0.9483 0.8531 0.9711 0.8467 0.9435 0.9373] 2020-11-30 02:55:17 [INFO] [EVAL] The model with the best validation mIoU (0.8066) was saved at iter 32000. 2020-11-30 02:56:24 [INFO] [TRAIN] epoch=55, iter=36100/40000, loss=0.0818, lr=0.001231, batch_cost=0.6625, reader_cost=0.0002 | ETA 00:43:03 2020-11-30 02:57:30 [INFO] [TRAIN] epoch=55, iter=36200/40000, loss=0.0921, lr=0.001202, batch_cost=0.6606, reader_cost=0.0003 | ETA 00:41:50 2020-11-30 02:58:35 [INFO] [TRAIN] epoch=55, iter=36300/40000, loss=0.0906, lr=0.001174, batch_cost=0.6549, reader_cost=0.0006 | ETA 00:40:23 2020-11-30 02:59:41 [INFO] [TRAIN] epoch=56, iter=36400/40000, loss=0.0703, lr=0.001145, batch_cost=0.6598, reader_cost=0.0057 | ETA 00:39:35 2020-11-30 03:00:47 [INFO] [TRAIN] epoch=56, iter=36500/40000, loss=0.0747, lr=0.001117, batch_cost=0.6587, reader_cost=0.0004 | ETA 00:38:25 2020-11-30 03:01:53 [INFO] [TRAIN] epoch=56, iter=36600/40000, loss=0.0890, lr=0.001088, batch_cost=0.6556, reader_cost=0.0003 | ETA 00:37:09 2020-11-30 03:02:58 [INFO] [TRAIN] epoch=56, iter=36700/40000, loss=0.0820, lr=0.001059, batch_cost=0.6560, reader_cost=0.0002 | ETA 00:36:04 2020-11-30 03:04:03 [INFO] [TRAIN] epoch=56, iter=36800/40000, loss=0.0745, lr=0.001030, batch_cost=0.6515, reader_cost=0.0001 | ETA 00:34:44 2020-11-30 03:05:09 [INFO] [TRAIN] epoch=56, iter=36900/40000, loss=0.0833, lr=0.001001, batch_cost=0.6531, reader_cost=0.0002 | ETA 00:33:44 2020-11-30 03:06:14 [INFO] [TRAIN] epoch=56, iter=37000/40000, loss=0.0799, lr=0.000972, batch_cost=0.6542, reader_cost=0.0002 | ETA 00:32:42 2020-11-30 03:07:20 [INFO] [TRAIN] epoch=57, iter=37100/40000, loss=0.0794, lr=0.000943, batch_cost=0.6619, reader_cost=0.0059 | ETA 00:31:59 2020-11-30 03:08:26 [INFO] [TRAIN] epoch=57, iter=37200/40000, loss=0.0736, lr=0.000914, batch_cost=0.6526, reader_cost=0.0002 | ETA 00:30:27 2020-11-30 03:09:31 [INFO] [TRAIN] epoch=57, iter=37300/40000, loss=0.0788, lr=0.000884, batch_cost=0.6541, reader_cost=0.0002 | ETA 00:29:26 2020-11-30 03:10:36 [INFO] [TRAIN] epoch=57, iter=37400/40000, loss=0.0846, lr=0.000855, batch_cost=0.6536, reader_cost=0.0002 | ETA 00:28:19 2020-11-30 03:11:42 [INFO] [TRAIN] epoch=57, iter=37500/40000, loss=0.0881, lr=0.000825, batch_cost=0.6570, reader_cost=0.0002 | ETA 00:27:22 2020-11-30 03:12:48 [INFO] [TRAIN] epoch=57, iter=37600/40000, loss=0.0790, lr=0.000795, batch_cost=0.6591, reader_cost=0.0002 | ETA 00:26:21 2020-11-30 03:13:54 [INFO] [TRAIN] epoch=58, iter=37700/40000, loss=0.0788, lr=0.000765, batch_cost=0.6616, reader_cost=0.0062 | ETA 00:25:21 2020-11-30 03:14:59 [INFO] [TRAIN] epoch=58, iter=37800/40000, loss=0.0850, lr=0.000735, batch_cost=0.6529, reader_cost=0.0001 | ETA 00:23:56 2020-11-30 03:16:05 [INFO] [TRAIN] epoch=58, iter=37900/40000, loss=0.0741, lr=0.000705, batch_cost=0.6539, reader_cost=0.0002 | ETA 00:22:53 2020-11-30 03:17:10 [INFO] [TRAIN] epoch=58, iter=38000/40000, loss=0.0856, lr=0.000675, batch_cost=0.6544, reader_cost=0.0001 | ETA 00:21:48 2020-11-30 03:18:16 [INFO] [TRAIN] epoch=58, iter=38100/40000, loss=0.0847, lr=0.000645, batch_cost=0.6521, reader_cost=0.0002 | ETA 00:20:39 2020-11-30 03:19:21 [INFO] [TRAIN] epoch=58, iter=38200/40000, loss=0.0804, lr=0.000614, batch_cost=0.6558, reader_cost=0.0002 | ETA 00:19:40 2020-11-30 03:20:27 [INFO] [TRAIN] epoch=58, iter=38300/40000, loss=0.0807, lr=0.000583, batch_cost=0.6542, reader_cost=0.0002 | ETA 00:18:32 2020-11-30 03:21:33 [INFO] [TRAIN] epoch=59, iter=38400/40000, loss=0.0869, lr=0.000552, batch_cost=0.6602, reader_cost=0.0058 | ETA 00:17:36 2020-11-30 03:22:38 [INFO] [TRAIN] epoch=59, iter=38500/40000, loss=0.0818, lr=0.000521, batch_cost=0.6537, reader_cost=0.0002 | ETA 00:16:20 2020-11-30 03:23:44 [INFO] [TRAIN] epoch=59, iter=38600/40000, loss=0.0803, lr=0.000490, batch_cost=0.6567, reader_cost=0.0002 | ETA 00:15:19 2020-11-30 03:24:49 [INFO] [TRAIN] epoch=59, iter=38700/40000, loss=0.0792, lr=0.000458, batch_cost=0.6577, reader_cost=0.0002 | ETA 00:14:15 2020-11-30 03:25:55 [INFO] [TRAIN] epoch=59, iter=38800/40000, loss=0.0786, lr=0.000426, batch_cost=0.6543, reader_cost=0.0002 | ETA 00:13:05 2020-11-30 03:27:00 [INFO] [TRAIN] epoch=59, iter=38900/40000, loss=0.0781, lr=0.000394, batch_cost=0.6537, reader_cost=0.0002 | ETA 00:11:59 2020-11-30 03:28:06 [INFO] [TRAIN] epoch=60, iter=39000/40000, loss=0.0821, lr=0.000362, batch_cost=0.6593, reader_cost=0.0060 | ETA 00:10:59 2020-11-30 03:29:12 [INFO] [TRAIN] epoch=60, iter=39100/40000, loss=0.0775, lr=0.000329, batch_cost=0.6572, reader_cost=0.0002 | ETA 00:09:51 2020-11-30 03:30:17 [INFO] [TRAIN] epoch=60, iter=39200/40000, loss=0.0832, lr=0.000296, batch_cost=0.6492, reader_cost=0.0001 | ETA 00:08:39 2020-11-30 03:31:22 [INFO] [TRAIN] epoch=60, iter=39300/40000, loss=0.0832, lr=0.000263, batch_cost=0.6561, reader_cost=0.0002 | ETA 00:07:39 2020-11-30 03:32:27 [INFO] [TRAIN] epoch=60, iter=39400/40000, loss=0.0838, lr=0.000229, batch_cost=0.6492, reader_cost=0.0001 | ETA 00:06:29 2020-11-30 03:33:32 [INFO] [TRAIN] epoch=60, iter=39500/40000, loss=0.0801, lr=0.000194, batch_cost=0.6514, reader_cost=0.0002 | ETA 00:05:25 2020-11-30 03:34:38 [INFO] [TRAIN] epoch=60, iter=39600/40000, loss=0.0778, lr=0.000159, batch_cost=0.6519, reader_cost=0.0002 | ETA 00:04:20 2020-11-30 03:35:44 [INFO] [TRAIN] epoch=61, iter=39700/40000, loss=0.0797, lr=0.000123, batch_cost=0.6617, reader_cost=0.0055 | ETA 00:03:18 2020-11-30 03:36:49 [INFO] [TRAIN] epoch=61, iter=39800/40000, loss=0.0803, lr=0.000085, batch_cost=0.6497, reader_cost=0.0002 | ETA 00:02:09 2020-11-30 03:37:54 [INFO] [TRAIN] epoch=61, iter=39900/40000, loss=0.0807, lr=0.000046, batch_cost=0.6478, reader_cost=0.0002 | ETA 00:01:04 2020-11-30 03:38:58 [INFO] [TRAIN] epoch=61, iter=40000/40000, loss=0.0816, lr=0.000001, batch_cost=0.6498, reader_cost=0.0001 | ETA 00:00:00 2020-11-30 03:38:59 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-30 03:39:16 [INFO] [EVAL] #Images=1449 mIoU=0.8053 Acc=0.9568 Kappa=0.9042 2020-11-30 03:39:16 [INFO] [EVAL] Class IoU: [0.9494 0.9132 0.4492 0.8944 0.7212 0.8242 0.9558 0.8954 0.9491 0.4716 0.936 0.5986 0.9088 0.9143 0.875 0.8797 0.6447 0.8968 0.5713 0.8901 0.7716] 2020-11-30 03:39:16 [INFO] [EVAL] Class Acc: [0.9686 0.9579 0.4645 0.9389 0.8307 0.8993 0.9789 0.9462 0.9716 0.7134 0.9674 0.9331 0.9418 0.9537 0.9442 0.9469 0.8315 0.9631 0.8495 0.9475 0.9345] 2020-11-30 03:39:18 [INFO] [EVAL] The model with the best validation mIoU (0.8066) was saved at iter 32000.