2020-12-12 10:39:47 [INFO] ------------Environment Information------------- platform: Linux-3.10.0_3-0-0-34-x86_64-with-centos-7.5.1804-Core Python: 3.7.9 (default, Aug 31 2020, 12:42:55) [GCC 7.3.0] Paddle compiled with cuda: True NVCC: Cuda compilation tools, release 10.2, V10.2.89 cudnn: 7.6 GPUs used: 4 CUDA_VISIBLE_DEVICES: 4,5,6,7 GPU: ['GPU 0: Tesla V100-SXM2-16GB', 'GPU 1: Tesla V100-SXM2-16GB', 'GPU 2: Tesla V100-SXM2-16GB', 'GPU 3: Tesla V100-SXM2-16GB', 'GPU 4: Tesla V100-SXM2-16GB', 'GPU 5: Tesla V100-SXM2-16GB', 'GPU 6: Tesla V100-SXM2-16GB', 'GPU 7: Tesla V100-SXM2-16GB'] GCC: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39) PaddlePaddle: 2.0.0-rc0 OpenCV: 4.1.0 ------------------------------------------------ 2020-12-12 10:39:48 [INFO] ---------------Config Information--------------- batch_size: 4 iters: 160000 learning_rate: decay: end_lr: 0.0 power: 0.9 type: poly value: 0.02 loss: coef: - 1 types: - ignore_index: 255 loss_th: 0.3 min_K: 4096 type: BootstrappedCrossEntropyLoss model: pretrained: null type: HarDNet optimizer: momentum: 0.9 type: sgd weight_decay: 0.0005 train_dataset: dataset_root: data/cityscapes mode: train transforms: - max_scale_factor: 2.0 min_scale_factor: 0.5 scale_step_size: 0.25 type: ResizeStepScaling - crop_size: - 1024 - 1024 type: RandomPaddingCrop - type: RandomHorizontalFlip - brightness_range: 0.4 contrast_range: 0.4 saturation_range: 0.4 type: RandomDistort - type: Normalize type: Cityscapes val_dataset: dataset_root: data/cityscapes mode: val transforms: - type: Normalize type: Cityscapes ------------------------------------------------ 2020-12-12 10:41:59 [INFO] [TRAIN] epoch=2, iter=200/160000, loss=1.6328, lr=0.019978, batch_cost=0.6044, reader_cost=0.0261 | ETA 26:49:46 2020-12-12 10:43:53 [INFO] [TRAIN] epoch=3, iter=400/160000, loss=1.3663, lr=0.019955, batch_cost=0.5695, reader_cost=0.0139 | ETA 25:14:44 2020-12-12 10:45:46 [INFO] [TRAIN] epoch=4, iter=600/160000, loss=1.3091, lr=0.019933, batch_cost=0.5658, reader_cost=0.0130 | ETA 25:03:01 2020-12-12 10:47:38 [INFO] [TRAIN] epoch=5, iter=800/160000, loss=1.2606, lr=0.019910, batch_cost=0.5609, reader_cost=0.0127 | ETA 24:48:19 2020-12-12 10:49:30 [INFO] [TRAIN] epoch=6, iter=1000/160000, loss=1.2168, lr=0.019888, batch_cost=0.5579, reader_cost=0.0135 | ETA 24:38:30 2020-12-12 10:51:22 [INFO] [TRAIN] epoch=7, iter=1200/160000, loss=1.2440, lr=0.019865, batch_cost=0.5609, reader_cost=0.0134 | ETA 24:44:26 2020-12-12 10:53:13 [INFO] [TRAIN] epoch=8, iter=1400/160000, loss=1.1923, lr=0.019843, batch_cost=0.5565, reader_cost=0.0131 | ETA 24:31:00 2020-12-12 10:55:04 [INFO] [TRAIN] epoch=9, iter=1600/160000, loss=1.1953, lr=0.019820, batch_cost=0.5546, reader_cost=0.0139 | ETA 24:24:13 2020-12-12 10:56:55 [INFO] [TRAIN] epoch=10, iter=1800/160000, loss=1.1672, lr=0.019797, batch_cost=0.5540, reader_cost=0.0132 | ETA 24:20:47 2020-12-12 10:58:45 [INFO] [TRAIN] epoch=11, iter=2000/160000, loss=1.1686, lr=0.019775, batch_cost=0.5517, reader_cost=0.0119 | ETA 24:12:52 2020-12-12 11:00:35 [INFO] [TRAIN] epoch=12, iter=2200/160000, loss=1.1672, lr=0.019752, batch_cost=0.5500, reader_cost=0.0131 | ETA 24:06:30 2020-12-12 11:02:25 [INFO] [TRAIN] epoch=13, iter=2400/160000, loss=1.1322, lr=0.019730, batch_cost=0.5494, reader_cost=0.0121 | ETA 24:03:03 2020-12-12 11:04:15 [INFO] [TRAIN] epoch=14, iter=2600/160000, loss=1.1591, lr=0.019707, batch_cost=0.5479, reader_cost=0.0119 | ETA 23:57:16 2020-12-12 11:06:07 [INFO] [TRAIN] epoch=16, iter=2800/160000, loss=1.1290, lr=0.019685, batch_cost=0.5604, reader_cost=0.0225 | ETA 24:28:18 2020-12-12 11:07:57 [INFO] [TRAIN] epoch=17, iter=3000/160000, loss=1.1106, lr=0.019662, batch_cost=0.5513, reader_cost=0.0133 | ETA 24:02:26 2020-12-12 11:09:47 [INFO] [TRAIN] epoch=18, iter=3200/160000, loss=1.1020, lr=0.019640, batch_cost=0.5491, reader_cost=0.0130 | ETA 23:54:51 2020-12-12 11:11:37 [INFO] [TRAIN] epoch=19, iter=3400/160000, loss=1.1138, lr=0.019617, batch_cost=0.5473, reader_cost=0.0126 | ETA 23:48:22 2020-12-12 11:13:27 [INFO] [TRAIN] epoch=20, iter=3600/160000, loss=1.0859, lr=0.019595, batch_cost=0.5500, reader_cost=0.0134 | ETA 23:53:47 2020-12-12 11:15:16 [INFO] [TRAIN] epoch=21, iter=3800/160000, loss=1.0895, lr=0.019572, batch_cost=0.5470, reader_cost=0.0136 | ETA 23:44:07 2020-12-12 11:17:05 [INFO] [TRAIN] epoch=22, iter=4000/160000, loss=1.0737, lr=0.019550, batch_cost=0.5470, reader_cost=0.0127 | ETA 23:42:04 2020-12-12 11:17:05 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-12 11:17:42 [INFO] [EVAL] #Images=500 mIoU=0.4406 Acc=0.8989 Kappa=0.8682 2020-12-12 11:17:42 [INFO] [EVAL] Class IoU: [0.9482 0.6441 0.8176 0.1214 0.259 0.385 0.2497 0.4057 0.8429 0.3848 0.8833 0.4633 0.1089 0.8064 0.0117 0.2672 0.1507 0.1307 0.4914] 2020-12-12 11:17:42 [INFO] [EVAL] Class Acc: [0.9589 0.859 0.9361 0.5627 0.5942 0.6522 0.4128 0.6578 0.8671 0.6237 0.9155 0.597 0.4652 0.8332 0.1824 0.6051 0.2389 0.3131 0.5655] 2020-12-12 11:17:43 [INFO] [EVAL] The model with the best validation mIoU (0.4406) was saved at iter 4000. 2020-12-12 11:19:32 [INFO] [TRAIN] epoch=23, iter=4200/160000, loss=1.0837, lr=0.019527, batch_cost=0.5484, reader_cost=0.0152 | ETA 23:43:54 2020-12-12 11:21:22 [INFO] [TRAIN] epoch=24, iter=4400/160000, loss=1.1001, lr=0.019504, batch_cost=0.5480, reader_cost=0.0131 | ETA 23:41:13 2020-12-12 11:23:12 [INFO] [TRAIN] epoch=25, iter=4600/160000, loss=1.0624, lr=0.019482, batch_cost=0.5487, reader_cost=0.0140 | ETA 23:41:06 2020-12-12 11:25:01 [INFO] [TRAIN] epoch=26, iter=4800/160000, loss=1.0559, lr=0.019459, batch_cost=0.5475, reader_cost=0.0132 | ETA 23:36:17 2020-12-12 11:26:51 [INFO] [TRAIN] epoch=27, iter=5000/160000, loss=1.0721, lr=0.019437, batch_cost=0.5490, reader_cost=0.0140 | ETA 23:38:19 2020-12-12 11:28:41 [INFO] [TRAIN] epoch=28, iter=5200/160000, loss=1.0647, lr=0.019414, batch_cost=0.5487, reader_cost=0.0141 | ETA 23:35:39 2020-12-12 11:30:32 [INFO] [TRAIN] epoch=30, iter=5400/160000, loss=1.0396, lr=0.019392, batch_cost=0.5565, reader_cost=0.0241 | ETA 23:53:58 2020-12-12 11:32:22 [INFO] [TRAIN] epoch=31, iter=5600/160000, loss=1.0601, lr=0.019369, batch_cost=0.5483, reader_cost=0.0147 | ETA 23:31:03 2020-12-12 11:34:11 [INFO] [TRAIN] epoch=32, iter=5800/160000, loss=1.0509, lr=0.019346, batch_cost=0.5469, reader_cost=0.0152 | ETA 23:25:34 2020-12-12 11:36:00 [INFO] [TRAIN] epoch=33, iter=6000/160000, loss=1.0623, lr=0.019324, batch_cost=0.5451, reader_cost=0.0143 | ETA 23:19:04 2020-12-12 11:37:50 [INFO] [TRAIN] epoch=34, iter=6200/160000, loss=1.0288, lr=0.019301, batch_cost=0.5462, reader_cost=0.0142 | ETA 23:20:12 2020-12-12 11:39:39 [INFO] [TRAIN] epoch=35, iter=6400/160000, loss=1.0538, lr=0.019279, batch_cost=0.5465, reader_cost=0.0121 | ETA 23:19:01 2020-12-12 11:41:28 [INFO] [TRAIN] epoch=36, iter=6600/160000, loss=1.0287, lr=0.019256, batch_cost=0.5452, reader_cost=0.0136 | ETA 23:13:57 2020-12-12 11:43:17 [INFO] [TRAIN] epoch=37, iter=6800/160000, loss=1.0447, lr=0.019233, batch_cost=0.5451, reader_cost=0.0141 | ETA 23:11:48 2020-12-12 11:45:06 [INFO] [TRAIN] epoch=38, iter=7000/160000, loss=1.0462, lr=0.019211, batch_cost=0.5463, reader_cost=0.0136 | ETA 23:12:59 2020-12-12 11:46:55 [INFO] [TRAIN] epoch=39, iter=7200/160000, loss=1.0361, lr=0.019188, batch_cost=0.5461, reader_cost=0.0122 | ETA 23:10:45 2020-12-12 11:48:44 [INFO] [TRAIN] epoch=40, iter=7400/160000, loss=1.0048, lr=0.019166, batch_cost=0.5439, reader_cost=0.0127 | ETA 23:03:20 2020-12-12 11:50:33 [INFO] [TRAIN] epoch=41, iter=7600/160000, loss=1.0375, lr=0.019143, batch_cost=0.5462, reader_cost=0.0123 | ETA 23:07:24 2020-12-12 11:52:22 [INFO] [TRAIN] epoch=42, iter=7800/160000, loss=1.0274, lr=0.019120, batch_cost=0.5426, reader_cost=0.0128 | ETA 22:56:23 2020-12-12 11:54:13 [INFO] [TRAIN] epoch=44, iter=8000/160000, loss=1.0111, lr=0.019098, batch_cost=0.5542, reader_cost=0.0214 | ETA 23:23:57 2020-12-12 11:54:13 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-12 11:54:42 [INFO] [EVAL] #Images=500 mIoU=0.4644 Acc=0.9018 Kappa=0.8728 2020-12-12 11:54:42 [INFO] [EVAL] Class IoU: [0.9575 0.7192 0.8478 0.227 0.2539 0.3921 0.1583 0.5083 0.8067 0.3006 0.8829 0.619 0.0834 0.8509 0.109 0.2963 0.2142 0.0096 0.5861] 2020-12-12 11:54:42 [INFO] [EVAL] Class Acc: [0.9758 0.8819 0.894 0.5687 0.2854 0.745 0.889 0.7205 0.9637 0.3227 0.8981 0.7604 0.8765 0.8847 0.6156 0.7089 0.284 0.4339 0.7167] 2020-12-12 11:54:43 [INFO] [EVAL] The model with the best validation mIoU (0.4644) was saved at iter 8000. 2020-12-12 11:56:32 [INFO] [TRAIN] epoch=45, iter=8200/160000, loss=1.0285, lr=0.019075, batch_cost=0.5467, reader_cost=0.0141 | ETA 23:03:10 2020-12-12 11:58:21 [INFO] [TRAIN] epoch=46, iter=8400/160000, loss=1.0303, lr=0.019053, batch_cost=0.5419, reader_cost=0.0124 | ETA 22:49:14 2020-12-12 12:00:09 [INFO] [TRAIN] epoch=47, iter=8600/160000, loss=0.9945, lr=0.019030, batch_cost=0.5427, reader_cost=0.0140 | ETA 22:49:27 2020-12-12 12:01:58 [INFO] [TRAIN] epoch=48, iter=8800/160000, loss=0.9990, lr=0.019007, batch_cost=0.5447, reader_cost=0.0140 | ETA 22:52:43 2020-12-12 12:03:47 [INFO] [TRAIN] epoch=49, iter=9000/160000, loss=1.0154, lr=0.018985, batch_cost=0.5441, reader_cost=0.0145 | ETA 22:49:12 2020-12-12 12:05:36 [INFO] [TRAIN] epoch=50, iter=9200/160000, loss=0.9889, lr=0.018962, batch_cost=0.5427, reader_cost=0.0124 | ETA 22:44:03 2020-12-12 12:07:25 [INFO] [TRAIN] epoch=51, iter=9400/160000, loss=1.0234, lr=0.018939, batch_cost=0.5452, reader_cost=0.0152 | ETA 22:48:31 2020-12-12 12:09:13 [INFO] [TRAIN] epoch=52, iter=9600/160000, loss=1.0044, lr=0.018917, batch_cost=0.5423, reader_cost=0.0132 | ETA 22:39:20 2020-12-12 12:11:02 [INFO] [TRAIN] epoch=53, iter=9800/160000, loss=1.0115, lr=0.018894, batch_cost=0.5425, reader_cost=0.0125 | ETA 22:38:02 2020-12-12 12:12:50 [INFO] [TRAIN] epoch=54, iter=10000/160000, loss=0.9798, lr=0.018872, batch_cost=0.5430, reader_cost=0.0129 | ETA 22:37:33 2020-12-12 12:14:39 [INFO] [TRAIN] epoch=55, iter=10200/160000, loss=1.0023, lr=0.018849, batch_cost=0.5440, reader_cost=0.0141 | ETA 22:38:06 2020-12-12 12:16:28 [INFO] [TRAIN] epoch=56, iter=10400/160000, loss=0.9932, lr=0.018826, batch_cost=0.5439, reader_cost=0.0137 | ETA 22:36:10 2020-12-12 12:18:17 [INFO] [TRAIN] epoch=57, iter=10600/160000, loss=0.9901, lr=0.018804, batch_cost=0.5443, reader_cost=0.0136 | ETA 22:35:13 2020-12-12 12:20:07 [INFO] [TRAIN] epoch=59, iter=10800/160000, loss=1.0062, lr=0.018781, batch_cost=0.5523, reader_cost=0.0217 | ETA 22:53:30 2020-12-12 12:21:56 [INFO] [TRAIN] epoch=60, iter=11000/160000, loss=1.0144, lr=0.018758, batch_cost=0.5462, reader_cost=0.0139 | ETA 22:36:23 2020-12-12 12:23:45 [INFO] [TRAIN] epoch=61, iter=11200/160000, loss=0.9739, lr=0.018736, batch_cost=0.5449, reader_cost=0.0139 | ETA 22:31:27 2020-12-12 12:25:34 [INFO] [TRAIN] epoch=62, iter=11400/160000, loss=0.9837, lr=0.018713, batch_cost=0.5427, reader_cost=0.0128 | ETA 22:24:03 2020-12-12 12:27:23 [INFO] [TRAIN] epoch=63, iter=11600/160000, loss=1.0142, lr=0.018690, batch_cost=0.5437, reader_cost=0.0135 | ETA 22:24:44 2020-12-12 12:29:11 [INFO] [TRAIN] epoch=64, iter=11800/160000, loss=1.0044, lr=0.018668, batch_cost=0.5433, reader_cost=0.0128 | ETA 22:22:04 2020-12-12 12:31:00 [INFO] [TRAIN] epoch=65, iter=12000/160000, loss=0.9783, lr=0.018645, batch_cost=0.5428, reader_cost=0.0129 | ETA 22:18:49 2020-12-12 12:31:00 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-12 12:31:31 [INFO] [EVAL] #Images=500 mIoU=0.5113 Acc=0.8927 Kappa=0.8606 2020-12-12 12:31:31 [INFO] [EVAL] Class IoU: [0.919 0.6436 0.8208 0.1424 0.4008 0.4876 0.3598 0.6001 0.7885 0.3892 0.9119 0.6233 0.3108 0.8182 0.2125 0.4217 0.1566 0.109 0.5987] 2020-12-12 12:31:31 [INFO] [EVAL] Class Acc: [0.9497 0.7502 0.8789 0.5062 0.548 0.6873 0.7005 0.8329 0.9431 0.5715 0.9321 0.712 0.6215 0.8892 0.4009 0.5702 0.1803 0.682 0.7467] 2020-12-12 12:31:32 [INFO] [EVAL] The model with the best validation mIoU (0.5113) was saved at iter 12000. 2020-12-12 12:33:20 [INFO] [TRAIN] epoch=66, iter=12200/160000, loss=0.9706, lr=0.018622, batch_cost=0.5412, reader_cost=0.0131 | ETA 22:13:09 2020-12-12 12:35:08 [INFO] [TRAIN] epoch=67, iter=12400/160000, loss=0.9951, lr=0.018600, batch_cost=0.5429, reader_cost=0.0133 | ETA 22:15:26 2020-12-12 12:36:57 [INFO] [TRAIN] epoch=68, iter=12600/160000, loss=0.9963, lr=0.018577, batch_cost=0.5444, reader_cost=0.0148 | ETA 22:17:26 2020-12-12 12:38:46 [INFO] [TRAIN] epoch=69, iter=12800/160000, loss=0.9975, lr=0.018554, batch_cost=0.5442, reader_cost=0.0131 | ETA 22:15:06 2020-12-12 12:40:35 [INFO] [TRAIN] epoch=70, iter=13000/160000, loss=0.9614, lr=0.018531, batch_cost=0.5441, reader_cost=0.0129 | ETA 22:13:07 2020-12-12 12:42:23 [INFO] [TRAIN] epoch=71, iter=13200/160000, loss=0.9966, lr=0.018509, batch_cost=0.5414, reader_cost=0.0135 | ETA 22:04:36 2020-12-12 12:44:14 [INFO] [TRAIN] epoch=73, iter=13400/160000, loss=0.9993, lr=0.018486, batch_cost=0.5556, reader_cost=0.0231 | ETA 22:37:36 2020-12-12 12:46:03 [INFO] [TRAIN] epoch=74, iter=13600/160000, loss=0.9836, lr=0.018463, batch_cost=0.5425, reader_cost=0.0132 | ETA 22:03:43 2020-12-12 12:47:51 [INFO] [TRAIN] epoch=75, iter=13800/160000, loss=0.9710, lr=0.018441, batch_cost=0.5422, reader_cost=0.0129 | ETA 22:01:03 2020-12-12 12:49:40 [INFO] [TRAIN] epoch=76, iter=14000/160000, loss=0.9827, lr=0.018418, batch_cost=0.5425, reader_cost=0.0127 | ETA 22:00:11 2020-12-12 12:51:28 [INFO] [TRAIN] epoch=77, iter=14200/160000, loss=0.9601, lr=0.018395, batch_cost=0.5426, reader_cost=0.0141 | ETA 21:58:31 2020-12-12 12:53:16 [INFO] [TRAIN] epoch=78, iter=14400/160000, loss=0.9576, lr=0.018373, batch_cost=0.5402, reader_cost=0.0122 | ETA 21:50:50 2020-12-12 12:55:05 [INFO] [TRAIN] epoch=79, iter=14600/160000, loss=0.9673, lr=0.018350, batch_cost=0.5449, reader_cost=0.0123 | ETA 22:00:35 2020-12-12 12:56:54 [INFO] [TRAIN] epoch=80, iter=14800/160000, loss=0.9800, lr=0.018327, batch_cost=0.5420, reader_cost=0.0139 | ETA 21:51:42 2020-12-12 12:58:42 [INFO] [TRAIN] epoch=81, iter=15000/160000, loss=0.9834, lr=0.018304, batch_cost=0.5403, reader_cost=0.0135 | ETA 21:45:44 2020-12-12 13:00:31 [INFO] [TRAIN] epoch=82, iter=15200/160000, loss=0.9864, lr=0.018282, batch_cost=0.5428, reader_cost=0.0130 | ETA 21:49:59 2020-12-12 13:02:19 [INFO] [TRAIN] epoch=83, iter=15400/160000, loss=0.9661, lr=0.018259, batch_cost=0.5428, reader_cost=0.0126 | ETA 21:48:09 2020-12-12 13:04:08 [INFO] [TRAIN] epoch=84, iter=15600/160000, loss=0.9591, lr=0.018236, batch_cost=0.5431, reader_cost=0.0130 | ETA 21:46:58 2020-12-12 13:05:56 [INFO] [TRAIN] epoch=85, iter=15800/160000, loss=0.9787, lr=0.018214, batch_cost=0.5419, reader_cost=0.0122 | ETA 21:42:27 2020-12-12 13:07:47 [INFO] [TRAIN] epoch=87, iter=16000/160000, loss=0.9740, lr=0.018191, batch_cost=0.5524, reader_cost=0.0220 | ETA 22:05:39 2020-12-12 13:07:47 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-12 13:08:17 [INFO] [EVAL] #Images=500 mIoU=0.5398 Acc=0.9337 Kappa=0.9134 2020-12-12 13:08:17 [INFO] [EVAL] Class IoU: [0.9681 0.7681 0.8792 0.2395 0.4338 0.4908 0.3568 0.6286 0.9049 0.5284 0.9062 0.699 0.4416 0.8677 0.0662 0.1521 0.1199 0.1604 0.6454] 2020-12-12 13:08:17 [INFO] [EVAL] Class Acc: [0.979 0.8977 0.9036 0.7088 0.61 0.7913 0.878 0.909 0.9503 0.8381 0.9762 0.801 0.7684 0.8915 0.5031 0.8293 0.4629 0.758 0.7347] 2020-12-12 13:08:17 [INFO] [EVAL] The model with the best validation mIoU (0.5398) was saved at iter 16000. 2020-12-12 13:10:06 [INFO] [TRAIN] epoch=88, iter=16200/160000, loss=0.9685, lr=0.018168, batch_cost=0.5438, reader_cost=0.0135 | ETA 21:43:24 2020-12-12 13:11:54 [INFO] [TRAIN] epoch=89, iter=16400/160000, loss=0.9611, lr=0.018145, batch_cost=0.5410, reader_cost=0.0130 | ETA 21:34:40 2020-12-12 13:13:42 [INFO] [TRAIN] epoch=90, iter=16600/160000, loss=0.9829, lr=0.018123, batch_cost=0.5417, reader_cost=0.0121 | ETA 21:34:37 2020-12-12 13:15:31 [INFO] [TRAIN] epoch=91, iter=16800/160000, loss=0.9795, lr=0.018100, batch_cost=0.5439, reader_cost=0.0131 | ETA 21:38:05 2020-12-12 13:17:20 [INFO] [TRAIN] epoch=92, iter=17000/160000, loss=0.9614, lr=0.018077, batch_cost=0.5431, reader_cost=0.0122 | ETA 21:34:28 2020-12-12 13:19:08 [INFO] [TRAIN] epoch=93, iter=17200/160000, loss=0.9830, lr=0.018054, batch_cost=0.5397, reader_cost=0.0130 | ETA 21:24:33 2020-12-12 13:20:56 [INFO] [TRAIN] epoch=94, iter=17400/160000, loss=0.9715, lr=0.018032, batch_cost=0.5424, reader_cost=0.0137 | ETA 21:29:02 2020-12-12 13:22:44 [INFO] [TRAIN] epoch=95, iter=17600/160000, loss=0.9560, lr=0.018009, batch_cost=0.5403, reader_cost=0.0129 | ETA 21:22:13 2020-12-12 13:24:33 [INFO] [TRAIN] epoch=96, iter=17800/160000, loss=0.9708, lr=0.017986, batch_cost=0.5409, reader_cost=0.0137 | ETA 21:21:50 2020-12-12 13:26:21 [INFO] [TRAIN] epoch=97, iter=18000/160000, loss=0.9625, lr=0.017963, batch_cost=0.5402, reader_cost=0.0116 | ETA 21:18:25 2020-12-12 13:28:09 [INFO] [TRAIN] epoch=98, iter=18200/160000, loss=0.9524, lr=0.017940, batch_cost=0.5414, reader_cost=0.0126 | ETA 21:19:25 2020-12-12 13:29:57 [INFO] [TRAIN] epoch=99, iter=18400/160000, loss=0.9335, lr=0.017918, batch_cost=0.5407, reader_cost=0.0126 | ETA 21:15:59 2020-12-12 13:31:45 [INFO] [TRAIN] epoch=100, iter=18600/160000, loss=0.9706, lr=0.017895, batch_cost=0.5398, reader_cost=0.0114 | ETA 21:12:10 2020-12-12 13:33:36 [INFO] [TRAIN] epoch=102, iter=18800/160000, loss=0.9556, lr=0.017872, batch_cost=0.5542, reader_cost=0.0219 | ETA 21:44:06 2020-12-12 13:35:24 [INFO] [TRAIN] epoch=103, iter=19000/160000, loss=0.9506, lr=0.017849, batch_cost=0.5427, reader_cost=0.0135 | ETA 21:15:20 2020-12-12 13:37:13 [INFO] [TRAIN] epoch=104, iter=19200/160000, loss=0.9658, lr=0.017827, batch_cost=0.5425, reader_cost=0.0129 | ETA 21:13:01 2020-12-12 13:39:02 [INFO] [TRAIN] epoch=105, iter=19400/160000, loss=0.9462, lr=0.017804, batch_cost=0.5439, reader_cost=0.0129 | ETA 21:14:33 2020-12-12 13:40:50 [INFO] [TRAIN] epoch=106, iter=19600/160000, loss=0.9479, lr=0.017781, batch_cost=0.5393, reader_cost=0.0113 | ETA 21:02:03 2020-12-12 13:42:38 [INFO] [TRAIN] epoch=107, iter=19800/160000, loss=0.9666, lr=0.017758, batch_cost=0.5405, reader_cost=0.0124 | ETA 21:02:54 2020-12-12 13:44:26 [INFO] [TRAIN] epoch=108, iter=20000/160000, loss=0.9567, lr=0.017735, batch_cost=0.5415, reader_cost=0.0126 | ETA 21:03:28 2020-12-12 13:44:26 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-12 13:44:56 [INFO] [EVAL] #Images=500 mIoU=0.5942 Acc=0.9376 Kappa=0.9187 2020-12-12 13:44:56 [INFO] [EVAL] Class IoU: [0.9703 0.7758 0.8787 0.2327 0.3864 0.5489 0.3726 0.6662 0.9013 0.5554 0.9304 0.7182 0.4442 0.9146 0.3411 0.4224 0.2604 0.3009 0.6694] 2020-12-12 13:44:56 [INFO] [EVAL] Class Acc: [0.9822 0.89 0.9004 0.7499 0.711 0.8203 0.9002 0.8468 0.9591 0.7567 0.9513 0.8637 0.7619 0.9476 0.7553 0.7155 0.2871 0.53 0.7717] 2020-12-12 13:44:56 [INFO] [EVAL] The model with the best validation mIoU (0.5942) was saved at iter 20000. 2020-12-12 13:46:45 [INFO] [TRAIN] epoch=109, iter=20200/160000, loss=0.9570, lr=0.017713, batch_cost=0.5408, reader_cost=0.0112 | ETA 21:00:04 2020-12-12 13:48:33 [INFO] [TRAIN] epoch=110, iter=20400/160000, loss=0.9648, lr=0.017690, batch_cost=0.5410, reader_cost=0.0122 | ETA 20:58:43 2020-12-12 13:50:21 [INFO] [TRAIN] epoch=111, iter=20600/160000, loss=0.9615, lr=0.017667, batch_cost=0.5400, reader_cost=0.0116 | ETA 20:54:41 2020-12-12 13:52:09 [INFO] [TRAIN] epoch=112, iter=20800/160000, loss=0.9477, lr=0.017644, batch_cost=0.5422, reader_cost=0.0125 | ETA 20:57:51 2020-12-12 13:53:57 [INFO] [TRAIN] epoch=113, iter=21000/160000, loss=0.9709, lr=0.017621, batch_cost=0.5387, reader_cost=0.0126 | ETA 20:48:04 2020-12-12 13:55:45 [INFO] [TRAIN] epoch=114, iter=21200/160000, loss=0.9580, lr=0.017598, batch_cost=0.5412, reader_cost=0.0117 | ETA 20:51:59 2020-12-12 13:57:36 [INFO] [TRAIN] epoch=116, iter=21400/160000, loss=0.9486, lr=0.017576, batch_cost=0.5513, reader_cost=0.0218 | ETA 21:13:35 2020-12-12 13:59:24 [INFO] [TRAIN] epoch=117, iter=21600/160000, loss=0.9531, lr=0.017553, batch_cost=0.5441, reader_cost=0.0125 | ETA 20:55:08 2020-12-12 14:01:13 [INFO] [TRAIN] epoch=118, iter=21800/160000, loss=0.9613, lr=0.017530, batch_cost=0.5416, reader_cost=0.0130 | ETA 20:47:35 2020-12-12 14:03:01 [INFO] [TRAIN] epoch=119, iter=22000/160000, loss=0.9329, lr=0.017507, batch_cost=0.5402, reader_cost=0.0130 | ETA 20:42:25 2020-12-12 14:04:49 [INFO] [TRAIN] epoch=120, iter=22200/160000, loss=0.9258, lr=0.017484, batch_cost=0.5403, reader_cost=0.0135 | ETA 20:40:52 2020-12-12 14:06:37 [INFO] [TRAIN] epoch=121, iter=22400/160000, loss=0.9496, lr=0.017461, batch_cost=0.5392, reader_cost=0.0120 | ETA 20:36:37 2020-12-12 14:08:25 [INFO] [TRAIN] epoch=122, iter=22600/160000, loss=0.9529, lr=0.017439, batch_cost=0.5420, reader_cost=0.0134 | ETA 20:41:11 2020-12-12 14:10:13 [INFO] [TRAIN] epoch=123, iter=22800/160000, loss=0.9387, lr=0.017416, batch_cost=0.5415, reader_cost=0.0135 | ETA 20:38:08 2020-12-12 14:12:02 [INFO] [TRAIN] epoch=124, iter=23000/160000, loss=0.9457, lr=0.017393, batch_cost=0.5406, reader_cost=0.0129 | ETA 20:34:21 2020-12-12 14:13:50 [INFO] [TRAIN] epoch=125, iter=23200/160000, loss=0.9546, lr=0.017370, batch_cost=0.5400, reader_cost=0.0117 | ETA 20:31:11 2020-12-12 14:15:38 [INFO] [TRAIN] epoch=126, iter=23400/160000, loss=0.9478, lr=0.017347, batch_cost=0.5405, reader_cost=0.0141 | ETA 20:30:34 2020-12-12 14:17:26 [INFO] [TRAIN] epoch=127, iter=23600/160000, loss=0.9601, lr=0.017324, batch_cost=0.5407, reader_cost=0.0136 | ETA 20:29:08 2020-12-12 14:19:14 [INFO] [TRAIN] epoch=128, iter=23800/160000, loss=0.9458, lr=0.017302, batch_cost=0.5430, reader_cost=0.0144 | ETA 20:32:31 2020-12-12 14:21:05 [INFO] [TRAIN] epoch=130, iter=24000/160000, loss=0.9674, lr=0.017279, batch_cost=0.5519, reader_cost=0.0233 | ETA 20:50:55 2020-12-12 14:21:05 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-12 14:21:34 [INFO] [EVAL] #Images=500 mIoU=0.5438 Acc=0.9137 Kappa=0.8882 2020-12-12 14:21:34 [INFO] [EVAL] Class IoU: [0.9465 0.7061 0.8602 0.2286 0.3856 0.4728 0.465 0.6056 0.8546 0.4349 0.9303 0.6987 0.4785 0.7826 0.2078 0.3766 0.1314 0.1317 0.6354] 2020-12-12 14:21:34 [INFO] [EVAL] Class Acc: [0.9829 0.8428 0.8964 0.6919 0.4524 0.7983 0.8737 0.9191 0.9501 0.4979 0.9598 0.8774 0.6222 0.7996 0.7612 0.6247 0.6662 0.6969 0.7933] 2020-12-12 14:21:35 [INFO] [EVAL] The model with the best validation mIoU (0.5942) was saved at iter 20000. 2020-12-12 14:23:23 [INFO] [TRAIN] epoch=131, iter=24200/160000, loss=0.9529, lr=0.017256, batch_cost=0.5413, reader_cost=0.0135 | ETA 20:25:15 2020-12-12 14:25:11 [INFO] [TRAIN] epoch=132, iter=24400/160000, loss=0.9481, lr=0.017233, batch_cost=0.5412, reader_cost=0.0117 | ETA 20:23:01 2020-12-12 14:27:00 [INFO] [TRAIN] epoch=133, iter=24600/160000, loss=0.9329, lr=0.017210, batch_cost=0.5412, reader_cost=0.0129 | ETA 20:21:16 2020-12-12 14:28:48 [INFO] [TRAIN] epoch=134, iter=24800/160000, loss=0.9418, lr=0.017187, batch_cost=0.5405, reader_cost=0.0124 | ETA 20:17:58 2020-12-12 14:30:36 [INFO] [TRAIN] epoch=135, iter=25000/160000, loss=0.9241, lr=0.017164, batch_cost=0.5427, reader_cost=0.0136 | ETA 20:20:58 2020-12-12 14:32:24 [INFO] [TRAIN] epoch=136, iter=25200/160000, loss=0.9415, lr=0.017141, batch_cost=0.5403, reader_cost=0.0129 | ETA 20:13:58 2020-12-12 14:34:13 [INFO] [TRAIN] epoch=137, iter=25400/160000, loss=0.9475, lr=0.017118, batch_cost=0.5414, reader_cost=0.0134 | ETA 20:14:36 2020-12-12 14:36:01 [INFO] [TRAIN] epoch=138, iter=25600/160000, loss=0.9373, lr=0.017096, batch_cost=0.5404, reader_cost=0.0129 | ETA 20:10:34 2020-12-12 14:37:49 [INFO] [TRAIN] epoch=139, iter=25800/160000, loss=0.9558, lr=0.017073, batch_cost=0.5406, reader_cost=0.0120 | ETA 20:09:13 2020-12-12 14:39:37 [INFO] [TRAIN] epoch=140, iter=26000/160000, loss=0.9562, lr=0.017050, batch_cost=0.5401, reader_cost=0.0136 | ETA 20:06:17 2020-12-12 14:41:25 [INFO] [TRAIN] epoch=141, iter=26200/160000, loss=0.9413, lr=0.017027, batch_cost=0.5392, reader_cost=0.0120 | ETA 20:02:23 2020-12-12 14:43:13 [INFO] [TRAIN] epoch=142, iter=26400/160000, loss=0.9402, lr=0.017004, batch_cost=0.5396, reader_cost=0.0125 | ETA 20:01:26 2020-12-12 14:45:03 [INFO] [TRAIN] epoch=144, iter=26600/160000, loss=0.9478, lr=0.016981, batch_cost=0.5499, reader_cost=0.0217 | ETA 20:22:37 2020-12-12 14:46:50 [INFO] [TRAIN] epoch=145, iter=26800/160000, loss=0.9350, lr=0.016958, batch_cost=0.5367, reader_cost=0.0132 | ETA 19:51:26 2020-12-12 14:48:37 [INFO] [TRAIN] epoch=146, iter=27000/160000, loss=0.9297, lr=0.016935, batch_cost=0.5373, reader_cost=0.0121 | ETA 19:50:55 2020-12-12 14:50:25 [INFO] [TRAIN] epoch=147, iter=27200/160000, loss=0.9351, lr=0.016912, batch_cost=0.5395, reader_cost=0.0120 | ETA 19:54:08 2020-12-12 14:52:14 [INFO] [TRAIN] epoch=148, iter=27400/160000, loss=0.9343, lr=0.016889, batch_cost=0.5406, reader_cost=0.0137 | ETA 19:54:49 2020-12-12 14:54:01 [INFO] [TRAIN] epoch=149, iter=27600/160000, loss=0.9135, lr=0.016866, batch_cost=0.5382, reader_cost=0.0134 | ETA 19:47:35 2020-12-12 14:55:49 [INFO] [TRAIN] epoch=150, iter=27800/160000, loss=0.9291, lr=0.016844, batch_cost=0.5407, reader_cost=0.0133 | ETA 19:51:25 2020-12-12 14:57:37 [INFO] [TRAIN] epoch=151, iter=28000/160000, loss=0.9508, lr=0.016821, batch_cost=0.5372, reader_cost=0.0122 | ETA 19:41:51 2020-12-12 14:57:37 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-12 14:58:07 [INFO] [EVAL] #Images=500 mIoU=0.6539 Acc=0.9436 Kappa=0.9265 2020-12-12 14:58:07 [INFO] [EVAL] Class IoU: [0.9719 0.7851 0.8927 0.3186 0.3977 0.5757 0.582 0.7019 0.9075 0.522 0.9247 0.75 0.4929 0.9183 0.4092 0.6787 0.4963 0.3986 0.7 ] 2020-12-12 14:58:07 [INFO] [EVAL] Class Acc: [0.9811 0.9054 0.9194 0.7332 0.6472 0.8207 0.8117 0.9024 0.9544 0.8678 0.9379 0.8701 0.7652 0.9389 0.7435 0.8337 0.7118 0.717 0.8358] 2020-12-12 14:58:08 [INFO] [EVAL] The model with the best validation mIoU (0.6539) was saved at iter 28000. 2020-12-12 14:59:55 [INFO] [TRAIN] epoch=152, iter=28200/160000, loss=0.9343, lr=0.016798, batch_cost=0.5383, reader_cost=0.0135 | ETA 19:42:26 2020-12-12 15:01:43 [INFO] [TRAIN] epoch=153, iter=28400/160000, loss=0.9498, lr=0.016775, batch_cost=0.5368, reader_cost=0.0113 | ETA 19:37:28 2020-12-12 15:03:31 [INFO] [TRAIN] epoch=154, iter=28600/160000, loss=0.9261, lr=0.016752, batch_cost=0.5403, reader_cost=0.0138 | ETA 19:43:12 2020-12-12 15:05:19 [INFO] [TRAIN] epoch=155, iter=28800/160000, loss=0.9354, lr=0.016729, batch_cost=0.5398, reader_cost=0.0124 | ETA 19:40:17 2020-12-12 15:07:06 [INFO] [TRAIN] epoch=156, iter=29000/160000, loss=0.9221, lr=0.016706, batch_cost=0.5373, reader_cost=0.0135 | ETA 19:33:02 2020-12-12 15:08:53 [INFO] [TRAIN] epoch=157, iter=29200/160000, loss=0.9393, lr=0.016683, batch_cost=0.5369, reader_cost=0.0128 | ETA 19:30:26 2020-12-12 15:10:44 [INFO] [TRAIN] epoch=159, iter=29400/160000, loss=0.9402, lr=0.016660, batch_cost=0.5503, reader_cost=0.0225 | ETA 19:57:46 2020-12-12 15:12:31 [INFO] [TRAIN] epoch=160, iter=29600/160000, loss=0.9364, lr=0.016637, batch_cost=0.5392, reader_cost=0.0112 | ETA 19:31:55 2020-12-12 15:14:19 [INFO] [TRAIN] epoch=161, iter=29800/160000, loss=0.9293, lr=0.016614, batch_cost=0.5387, reader_cost=0.0135 | ETA 19:28:59 2020-12-12 15:16:06 [INFO] [TRAIN] epoch=162, iter=30000/160000, loss=0.9387, lr=0.016591, batch_cost=0.5367, reader_cost=0.0120 | ETA 19:22:49 2020-12-12 15:17:54 [INFO] [TRAIN] epoch=163, iter=30200/160000, loss=0.9259, lr=0.016568, batch_cost=0.5385, reader_cost=0.0116 | ETA 19:24:51 2020-12-12 15:19:42 [INFO] [TRAIN] epoch=164, iter=30400/160000, loss=0.9442, lr=0.016545, batch_cost=0.5392, reader_cost=0.0109 | ETA 19:24:37 2020-12-12 15:21:30 [INFO] [TRAIN] epoch=165, iter=30600/160000, loss=0.9343, lr=0.016522, batch_cost=0.5381, reader_cost=0.0129 | ETA 19:20:32 2020-12-12 15:23:18 [INFO] [TRAIN] epoch=166, iter=30800/160000, loss=0.9250, lr=0.016499, batch_cost=0.5406, reader_cost=0.0131 | ETA 19:24:06 2020-12-12 15:25:05 [INFO] [TRAIN] epoch=167, iter=31000/160000, loss=0.9301, lr=0.016476, batch_cost=0.5370, reader_cost=0.0115 | ETA 19:14:32 2020-12-12 15:26:53 [INFO] [TRAIN] epoch=168, iter=31200/160000, loss=0.9142, lr=0.016453, batch_cost=0.5372, reader_cost=0.0131 | ETA 19:13:11 2020-12-12 15:28:41 [INFO] [TRAIN] epoch=169, iter=31400/160000, loss=0.9360, lr=0.016430, batch_cost=0.5402, reader_cost=0.0137 | ETA 19:17:44 2020-12-12 15:30:28 [INFO] [TRAIN] epoch=170, iter=31600/160000, loss=0.9322, lr=0.016407, batch_cost=0.5379, reader_cost=0.0119 | ETA 19:11:11 2020-12-12 15:32:16 [INFO] [TRAIN] epoch=171, iter=31800/160000, loss=0.9254, lr=0.016384, batch_cost=0.5376, reader_cost=0.0119 | ETA 19:08:42 2020-12-12 15:34:05 [INFO] [TRAIN] epoch=173, iter=32000/160000, loss=0.9196, lr=0.016361, batch_cost=0.5482, reader_cost=0.0205 | ETA 19:29:35 2020-12-12 15:34:05 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-12 15:34:36 [INFO] [EVAL] #Images=500 mIoU=0.6139 Acc=0.9415 Kappa=0.9240 2020-12-12 15:34:36 [INFO] [EVAL] Class IoU: [0.9713 0.7892 0.9031 0.2036 0.4479 0.591 0.5989 0.7135 0.9048 0.5284 0.9336 0.6586 0.2237 0.9197 0.4554 0.5996 0.3813 0.2679 0.573 ] 2020-12-12 15:34:36 [INFO] [EVAL] Class Acc: [0.9854 0.8758 0.9354 0.8072 0.668 0.7899 0.8201 0.8903 0.9541 0.6407 0.9664 0.6926 0.7541 0.9513 0.5897 0.8423 0.4332 0.3959 0.9197] 2020-12-12 15:34:37 [INFO] [EVAL] The model with the best validation mIoU (0.6539) was saved at iter 28000. 2020-12-12 15:36:24 [INFO] [TRAIN] epoch=174, iter=32200/160000, loss=0.9293, lr=0.016338, batch_cost=0.5381, reader_cost=0.0132 | ETA 19:06:05 2020-12-12 15:38:12 [INFO] [TRAIN] epoch=175, iter=32400/160000, loss=0.9253, lr=0.016315, batch_cost=0.5372, reader_cost=0.0125 | ETA 19:02:21 2020-12-12 15:39:59 [INFO] [TRAIN] epoch=176, iter=32600/160000, loss=0.9285, lr=0.016292, batch_cost=0.5370, reader_cost=0.0135 | ETA 19:00:18 2020-12-12 15:41:47 [INFO] [TRAIN] epoch=177, iter=32800/160000, loss=0.9418, lr=0.016269, batch_cost=0.5387, reader_cost=0.0122 | ETA 19:01:57 2020-12-12 15:43:35 [INFO] [TRAIN] epoch=178, iter=33000/160000, loss=0.9305, lr=0.016246, batch_cost=0.5396, reader_cost=0.0131 | ETA 19:02:14 2020-12-12 15:45:22 [INFO] [TRAIN] epoch=179, iter=33200/160000, loss=0.9353, lr=0.016223, batch_cost=0.5384, reader_cost=0.0116 | ETA 18:57:43 2020-12-12 15:47:10 [INFO] [TRAIN] epoch=180, iter=33400/160000, loss=0.9363, lr=0.016200, batch_cost=0.5391, reader_cost=0.0119 | ETA 18:57:34 2020-12-12 15:48:58 [INFO] [TRAIN] epoch=181, iter=33600/160000, loss=0.9129, lr=0.016177, batch_cost=0.5380, reader_cost=0.0126 | ETA 18:53:20 2020-12-12 15:50:46 [INFO] [TRAIN] epoch=182, iter=33800/160000, loss=0.9224, lr=0.016154, batch_cost=0.5387, reader_cost=0.0125 | ETA 18:53:04 2020-12-12 15:52:33 [INFO] [TRAIN] epoch=183, iter=34000/160000, loss=0.9205, lr=0.016131, batch_cost=0.5374, reader_cost=0.0114 | ETA 18:48:37 2020-12-12 15:54:21 [INFO] [TRAIN] epoch=184, iter=34200/160000, loss=0.9284, lr=0.016108, batch_cost=0.5375, reader_cost=0.0129 | ETA 18:46:59 2020-12-12 15:56:08 [INFO] [TRAIN] epoch=185, iter=34400/160000, loss=0.9465, lr=0.016085, batch_cost=0.5370, reader_cost=0.0140 | ETA 18:44:07 2020-12-12 15:57:58 [INFO] [TRAIN] epoch=187, iter=34600/160000, loss=0.9234, lr=0.016062, batch_cost=0.5495, reader_cost=0.0227 | ETA 19:08:25 2020-12-12 15:59:45 [INFO] [TRAIN] epoch=188, iter=34800/160000, loss=0.9344, lr=0.016039, batch_cost=0.5364, reader_cost=0.0132 | ETA 18:39:17 2020-12-12 16:01:32 [INFO] [TRAIN] epoch=189, iter=35000/160000, loss=0.9017, lr=0.016016, batch_cost=0.5369, reader_cost=0.0127 | ETA 18:38:35 2020-12-12 16:03:20 [INFO] [TRAIN] epoch=190, iter=35200/160000, loss=0.9431, lr=0.015993, batch_cost=0.5389, reader_cost=0.0119 | ETA 18:40:54 2020-12-12 16:05:08 [INFO] [TRAIN] epoch=191, iter=35400/160000, loss=0.9156, lr=0.015970, batch_cost=0.5372, reader_cost=0.0133 | ETA 18:35:39 2020-12-12 16:06:55 [INFO] [TRAIN] epoch=192, iter=35600/160000, loss=0.9505, lr=0.015946, batch_cost=0.5371, reader_cost=0.0122 | ETA 18:33:39 2020-12-12 16:08:43 [INFO] [TRAIN] epoch=193, iter=35800/160000, loss=0.9484, lr=0.015923, batch_cost=0.5400, reader_cost=0.0136 | ETA 18:37:43 2020-12-12 16:10:31 [INFO] [TRAIN] epoch=194, iter=36000/160000, loss=0.9172, lr=0.015900, batch_cost=0.5395, reader_cost=0.0135 | ETA 18:34:53 2020-12-12 16:10:31 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-12 16:11:02 [INFO] [EVAL] #Images=500 mIoU=0.6372 Acc=0.9442 Kappa=0.9276 2020-12-12 16:11:02 [INFO] [EVAL] Class IoU: [0.9739 0.8002 0.8982 0.3891 0.4843 0.5544 0.6023 0.7145 0.9115 0.5462 0.9408 0.7507 0.443 0.9188 0.4886 0.5176 0.3487 0.1371 0.6861] 2020-12-12 16:11:02 [INFO] [EVAL] Class Acc: [0.9848 0.8972 0.952 0.72 0.6021 0.6313 0.8611 0.7974 0.9557 0.7545 0.9606 0.8325 0.7518 0.9426 0.6211 0.9226 0.5248 0.5943 0.8582] 2020-12-12 16:11:03 [INFO] [EVAL] The model with the best validation mIoU (0.6539) was saved at iter 28000. 2020-12-12 16:12:50 [INFO] [TRAIN] epoch=195, iter=36200/160000, loss=0.9248, lr=0.015877, batch_cost=0.5373, reader_cost=0.0117 | ETA 18:28:43 2020-12-12 16:14:38 [INFO] [TRAIN] epoch=196, iter=36400/160000, loss=0.9477, lr=0.015854, batch_cost=0.5378, reader_cost=0.0124 | ETA 18:27:46 2020-12-12 16:16:26 [INFO] [TRAIN] epoch=197, iter=36600/160000, loss=0.9193, lr=0.015831, batch_cost=0.5393, reader_cost=0.0131 | ETA 18:29:10 2020-12-12 16:18:13 [INFO] [TRAIN] epoch=198, iter=36800/160000, loss=0.9158, lr=0.015808, batch_cost=0.5383, reader_cost=0.0132 | ETA 18:25:19 2020-12-12 16:20:01 [INFO] [TRAIN] epoch=199, iter=37000/160000, loss=0.9131, lr=0.015785, batch_cost=0.5370, reader_cost=0.0131 | ETA 18:20:51 2020-12-12 16:21:48 [INFO] [TRAIN] epoch=200, iter=37200/160000, loss=0.9179, lr=0.015762, batch_cost=0.5361, reader_cost=0.0125 | ETA 18:17:13 2020-12-12 16:23:37 [INFO] [TRAIN] epoch=202, iter=37400/160000, loss=0.9366, lr=0.015739, batch_cost=0.5485, reader_cost=0.0214 | ETA 18:40:46 2020-12-12 16:25:25 [INFO] [TRAIN] epoch=203, iter=37600/160000, loss=0.9275, lr=0.015716, batch_cost=0.5367, reader_cost=0.0116 | ETA 18:14:55 2020-12-12 16:27:13 [INFO] [TRAIN] epoch=204, iter=37800/160000, loss=0.9273, lr=0.015692, batch_cost=0.5383, reader_cost=0.0119 | ETA 18:16:18 2020-12-12 16:29:00 [INFO] [TRAIN] epoch=205, iter=38000/160000, loss=0.9292, lr=0.015669, batch_cost=0.5396, reader_cost=0.0142 | ETA 18:17:16 2020-12-12 16:30:49 [INFO] [TRAIN] epoch=206, iter=38200/160000, loss=0.9331, lr=0.015646, batch_cost=0.5408, reader_cost=0.0136 | ETA 18:17:46 2020-12-12 16:32:36 [INFO] [TRAIN] epoch=207, iter=38400/160000, loss=0.9181, lr=0.015623, batch_cost=0.5369, reader_cost=0.0125 | ETA 18:08:01 2020-12-12 16:34:23 [INFO] [TRAIN] epoch=208, iter=38600/160000, loss=0.9169, lr=0.015600, batch_cost=0.5361, reader_cost=0.0126 | ETA 18:04:46 2020-12-12 16:36:11 [INFO] [TRAIN] epoch=209, iter=38800/160000, loss=0.9100, lr=0.015577, batch_cost=0.5395, reader_cost=0.0130 | ETA 18:09:51 2020-12-12 16:37:58 [INFO] [TRAIN] epoch=210, iter=39000/160000, loss=0.9347, lr=0.015554, batch_cost=0.5359, reader_cost=0.0119 | ETA 18:00:45 2020-12-12 16:39:46 [INFO] [TRAIN] epoch=211, iter=39200/160000, loss=0.9238, lr=0.015531, batch_cost=0.5364, reader_cost=0.0124 | ETA 17:59:53 2020-12-12 16:41:33 [INFO] [TRAIN] epoch=212, iter=39400/160000, loss=0.9408, lr=0.015507, batch_cost=0.5386, reader_cost=0.0119 | ETA 18:02:35 2020-12-12 16:43:21 [INFO] [TRAIN] epoch=213, iter=39600/160000, loss=0.9078, lr=0.015484, batch_cost=0.5387, reader_cost=0.0119 | ETA 18:00:56 2020-12-12 16:45:08 [INFO] [TRAIN] epoch=214, iter=39800/160000, loss=0.9407, lr=0.015461, batch_cost=0.5371, reader_cost=0.0111 | ETA 17:55:54 2020-12-12 16:46:58 [INFO] [TRAIN] epoch=216, iter=40000/160000, loss=0.9375, lr=0.015438, batch_cost=0.5482, reader_cost=0.0219 | ETA 18:16:28 2020-12-12 16:46:58 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-12 16:47:29 [INFO] [EVAL] #Images=500 mIoU=0.5548 Acc=0.9202 Kappa=0.8965 2020-12-12 16:47:29 [INFO] [EVAL] Class IoU: [0.9615 0.7424 0.8775 0.1862 0.4715 0.5255 0.5315 0.7127 0.8728 0.4506 0.9336 0.5911 0.4108 0.8175 0.2672 0.5363 0.3371 0.0645 0.2513] 2020-12-12 16:47:29 [INFO] [EVAL] Class Acc: [0.982 0.8676 0.9562 0.6682 0.7473 0.7725 0.6718 0.8649 0.901 0.8042 0.9572 0.6447 0.5032 0.8791 0.3825 0.7701 0.6128 0.0652 0.8226] 2020-12-12 16:47:30 [INFO] [EVAL] The model with the best validation mIoU (0.6539) was saved at iter 28000. 2020-12-12 16:49:17 [INFO] [TRAIN] epoch=217, iter=40200/160000, loss=0.9325, lr=0.015415, batch_cost=0.5374, reader_cost=0.0137 | ETA 17:52:55 2020-12-12 16:51:05 [INFO] [TRAIN] epoch=218, iter=40400/160000, loss=0.9190, lr=0.015392, batch_cost=0.5376, reader_cost=0.0125 | ETA 17:51:37 2020-12-12 16:52:53 [INFO] [TRAIN] epoch=219, iter=40600/160000, loss=0.9156, lr=0.015368, batch_cost=0.5400, reader_cost=0.0124 | ETA 17:54:39 2020-12-12 16:54:40 [INFO] [TRAIN] epoch=220, iter=40800/160000, loss=0.9379, lr=0.015345, batch_cost=0.5354, reader_cost=0.0137 | ETA 17:43:34 2020-12-12 16:56:27 [INFO] [TRAIN] epoch=221, iter=41000/160000, loss=0.9178, lr=0.015322, batch_cost=0.5355, reader_cost=0.0126 | ETA 17:42:06 2020-12-12 16:58:14 [INFO] [TRAIN] epoch=222, iter=41200/160000, loss=0.9343, lr=0.015299, batch_cost=0.5376, reader_cost=0.0125 | ETA 17:44:25 2020-12-12 17:00:02 [INFO] [TRAIN] epoch=223, iter=41400/160000, loss=0.9266, lr=0.015276, batch_cost=0.5377, reader_cost=0.0125 | ETA 17:42:51 2020-12-12 17:01:49 [INFO] [TRAIN] epoch=224, iter=41600/160000, loss=0.9155, lr=0.015253, batch_cost=0.5370, reader_cost=0.0132 | ETA 17:39:38 2020-12-12 17:03:37 [INFO] [TRAIN] epoch=225, iter=41800/160000, loss=0.9190, lr=0.015229, batch_cost=0.5374, reader_cost=0.0137 | ETA 17:38:45 2020-12-12 17:05:24 [INFO] [TRAIN] epoch=226, iter=42000/160000, loss=0.9111, lr=0.015206, batch_cost=0.5360, reader_cost=0.0124 | ETA 17:34:04 2020-12-12 17:07:11 [INFO] [TRAIN] epoch=227, iter=42200/160000, loss=0.9232, lr=0.015183, batch_cost=0.5368, reader_cost=0.0142 | ETA 17:33:51 2020-12-12 17:08:59 [INFO] [TRAIN] epoch=228, iter=42400/160000, loss=0.9174, lr=0.015160, batch_cost=0.5359, reader_cost=0.0125 | ETA 17:30:25 2020-12-12 17:10:48 [INFO] [TRAIN] epoch=230, iter=42600/160000, loss=0.9021, lr=0.015137, batch_cost=0.5474, reader_cost=0.0220 | ETA 17:51:00 2020-12-12 17:12:36 [INFO] [TRAIN] epoch=231, iter=42800/160000, loss=0.9340, lr=0.015113, batch_cost=0.5383, reader_cost=0.0116 | ETA 17:31:24 2020-12-12 17:14:23 [INFO] [TRAIN] epoch=232, iter=43000/160000, loss=0.9329, lr=0.015090, batch_cost=0.5392, reader_cost=0.0127 | ETA 17:31:24 2020-12-12 17:16:11 [INFO] [TRAIN] epoch=233, iter=43200/160000, loss=0.9131, lr=0.015067, batch_cost=0.5380, reader_cost=0.0136 | ETA 17:27:23 2020-12-12 17:17:59 [INFO] [TRAIN] epoch=234, iter=43400/160000, loss=0.9017, lr=0.015044, batch_cost=0.5396, reader_cost=0.0127 | ETA 17:28:37 2020-12-12 17:19:47 [INFO] [TRAIN] epoch=235, iter=43600/160000, loss=0.9225, lr=0.015020, batch_cost=0.5401, reader_cost=0.0139 | ETA 17:27:47 2020-12-12 17:21:35 [INFO] [TRAIN] epoch=236, iter=43800/160000, loss=0.9196, lr=0.014997, batch_cost=0.5376, reader_cost=0.0127 | ETA 17:21:06 2020-12-12 17:23:22 [INFO] [TRAIN] epoch=237, iter=44000/160000, loss=0.9110, lr=0.014974, batch_cost=0.5377, reader_cost=0.0123 | ETA 17:19:29 2020-12-12 17:23:22 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-12 17:23:51 [INFO] [EVAL] #Images=500 mIoU=0.6004 Acc=0.9245 Kappa=0.9015 2020-12-12 17:23:51 [INFO] [EVAL] Class IoU: [0.9445 0.6402 0.8825 0.2384 0.4371 0.5585 0.5518 0.6976 0.8834 0.359 0.9261 0.5709 0.4607 0.8915 0.4155 0.5667 0.4582 0.277 0.648 ] 2020-12-12 17:23:51 [INFO] [EVAL] Class Acc: [0.9547 0.8959 0.9321 0.6133 0.761 0.7781 0.6405 0.7982 0.9416 0.8778 0.947 0.5961 0.6192 0.9229 0.5257 0.8449 0.755 0.4278 0.8145] 2020-12-12 17:23:52 [INFO] [EVAL] The model with the best validation mIoU (0.6539) was saved at iter 28000. 2020-12-12 17:25:39 [INFO] [TRAIN] epoch=238, iter=44200/160000, loss=0.9310, lr=0.014951, batch_cost=0.5373, reader_cost=0.0119 | ETA 17:16:56 2020-12-12 17:27:27 [INFO] [TRAIN] epoch=239, iter=44400/160000, loss=0.9214, lr=0.014928, batch_cost=0.5398, reader_cost=0.0127 | ETA 17:19:59 2020-12-12 17:29:15 [INFO] [TRAIN] epoch=240, iter=44600/160000, loss=0.9091, lr=0.014904, batch_cost=0.5380, reader_cost=0.0134 | ETA 17:14:50 2020-12-12 17:31:02 [INFO] [TRAIN] epoch=241, iter=44800/160000, loss=0.9078, lr=0.014881, batch_cost=0.5390, reader_cost=0.0114 | ETA 17:14:47 2020-12-12 17:32:50 [INFO] [TRAIN] epoch=242, iter=45000/160000, loss=0.9141, lr=0.014858, batch_cost=0.5392, reader_cost=0.0127 | ETA 17:13:28 2020-12-12 17:34:39 [INFO] [TRAIN] epoch=244, iter=45200/160000, loss=0.9290, lr=0.014835, batch_cost=0.5454, reader_cost=0.0225 | ETA 17:23:36 2020-12-12 17:36:27 [INFO] [TRAIN] epoch=245, iter=45400/160000, loss=0.9032, lr=0.014811, batch_cost=0.5369, reader_cost=0.0122 | ETA 17:05:30 2020-12-12 17:38:14 [INFO] [TRAIN] epoch=246, iter=45600/160000, loss=0.9094, lr=0.014788, batch_cost=0.5373, reader_cost=0.0116 | ETA 17:04:25 2020-12-12 17:40:02 [INFO] [TRAIN] epoch=247, iter=45800/160000, loss=0.9393, lr=0.014765, batch_cost=0.5385, reader_cost=0.0124 | ETA 17:04:53 2020-12-12 17:41:49 [INFO] [TRAIN] epoch=248, iter=46000/160000, loss=0.9211, lr=0.014741, batch_cost=0.5356, reader_cost=0.0122 | ETA 16:57:33 2020-12-12 17:43:36 [INFO] [TRAIN] epoch=249, iter=46200/160000, loss=0.9162, lr=0.014718, batch_cost=0.5371, reader_cost=0.0119 | ETA 16:58:47 2020-12-12 17:45:24 [INFO] [TRAIN] epoch=250, iter=46400/160000, loss=0.9164, lr=0.014695, batch_cost=0.5369, reader_cost=0.0131 | ETA 16:56:32 2020-12-12 17:47:11 [INFO] [TRAIN] epoch=251, iter=46600/160000, loss=0.9133, lr=0.014672, batch_cost=0.5370, reader_cost=0.0124 | ETA 16:55:00 2020-12-12 17:48:59 [INFO] [TRAIN] epoch=252, iter=46800/160000, loss=0.8989, lr=0.014648, batch_cost=0.5375, reader_cost=0.0124 | ETA 16:54:03 2020-12-12 17:50:46 [INFO] [TRAIN] epoch=253, iter=47000/160000, loss=0.9051, lr=0.014625, batch_cost=0.5345, reader_cost=0.0122 | ETA 16:46:43 2020-12-12 17:52:33 [INFO] [TRAIN] epoch=254, iter=47200/160000, loss=0.9157, lr=0.014602, batch_cost=0.5366, reader_cost=0.0119 | ETA 16:48:44 2020-12-12 17:54:21 [INFO] [TRAIN] epoch=255, iter=47400/160000, loss=0.9128, lr=0.014578, batch_cost=0.5380, reader_cost=0.0133 | ETA 16:49:35 2020-12-12 17:56:08 [INFO] [TRAIN] epoch=256, iter=47600/160000, loss=0.9300, lr=0.014555, batch_cost=0.5394, reader_cost=0.0126 | ETA 16:50:25 2020-12-12 17:57:56 [INFO] [TRAIN] epoch=257, iter=47800/160000, loss=0.9028, lr=0.014532, batch_cost=0.5355, reader_cost=0.0121 | ETA 16:41:24 2020-12-12 17:59:45 [INFO] [TRAIN] epoch=259, iter=48000/160000, loss=0.9045, lr=0.014508, batch_cost=0.5498, reader_cost=0.0228 | ETA 17:06:12 2020-12-12 17:59:45 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-12 18:00:16 [INFO] [EVAL] #Images=500 mIoU=0.6221 Acc=0.9403 Kappa=0.9226 2020-12-12 18:00:16 [INFO] [EVAL] Class IoU: [0.9698 0.765 0.892 0.3102 0.4278 0.5973 0.6055 0.7093 0.9088 0.5669 0.9413 0.7614 0.5021 0.9207 0.4535 0.5269 0.2352 0.0241 0.7017] 2020-12-12 18:00:16 [INFO] [EVAL] Class Acc: [0.9777 0.9097 0.958 0.7997 0.4809 0.7077 0.8679 0.8071 0.9551 0.7359 0.9605 0.8255 0.7417 0.9439 0.7919 0.5628 0.2664 0.9383 0.8963] 2020-12-12 18:00:17 [INFO] [EVAL] The model with the best validation mIoU (0.6539) was saved at iter 28000. 2020-12-12 18:02:05 [INFO] [TRAIN] epoch=260, iter=48200/160000, loss=0.9111, lr=0.014485, batch_cost=0.5375, reader_cost=0.0132 | ETA 16:41:37 2020-12-12 18:03:52 [INFO] [TRAIN] epoch=261, iter=48400/160000, loss=0.9298, lr=0.014462, batch_cost=0.5358, reader_cost=0.0124 | ETA 16:36:30 2020-12-12 18:05:39 [INFO] [TRAIN] epoch=262, iter=48600/160000, loss=0.9136, lr=0.014439, batch_cost=0.5383, reader_cost=0.0132 | ETA 16:39:31 2020-12-12 18:07:27 [INFO] [TRAIN] epoch=263, iter=48800/160000, loss=0.9282, lr=0.014415, batch_cost=0.5370, reader_cost=0.0127 | ETA 16:35:17 2020-12-12 18:09:15 [INFO] [TRAIN] epoch=264, iter=49000/160000, loss=0.9183, lr=0.014392, batch_cost=0.5396, reader_cost=0.0138 | ETA 16:38:13 2020-12-12 18:11:02 [INFO] [TRAIN] epoch=265, iter=49200/160000, loss=0.9069, lr=0.014368, batch_cost=0.5381, reader_cost=0.0134 | ETA 16:33:43 2020-12-12 18:12:50 [INFO] [TRAIN] epoch=266, iter=49400/160000, loss=0.9064, lr=0.014345, batch_cost=0.5384, reader_cost=0.0143 | ETA 16:32:22 2020-12-12 18:14:38 [INFO] [TRAIN] epoch=267, iter=49600/160000, loss=0.9047, lr=0.014322, batch_cost=0.5388, reader_cost=0.0147 | ETA 16:31:22 2020-12-12 18:16:25 [INFO] [TRAIN] epoch=268, iter=49800/160000, loss=0.9142, lr=0.014298, batch_cost=0.5376, reader_cost=0.0138 | ETA 16:27:20 2020-12-12 18:18:13 [INFO] [TRAIN] epoch=269, iter=50000/160000, loss=0.8925, lr=0.014275, batch_cost=0.5380, reader_cost=0.0112 | ETA 16:26:20 2020-12-12 18:20:00 [INFO] [TRAIN] epoch=270, iter=50200/160000, loss=0.9004, lr=0.014252, batch_cost=0.5357, reader_cost=0.0131 | ETA 16:20:15 2020-12-12 18:21:48 [INFO] [TRAIN] epoch=271, iter=50400/160000, loss=0.9205, lr=0.014228, batch_cost=0.5389, reader_cost=0.0137 | ETA 16:24:27 2020-12-12 18:23:37 [INFO] [TRAIN] epoch=273, iter=50600/160000, loss=0.9141, lr=0.014205, batch_cost=0.5456, reader_cost=0.0233 | ETA 16:34:46 2020-12-12 18:25:25 [INFO] [TRAIN] epoch=274, iter=50800/160000, loss=0.9211, lr=0.014182, batch_cost=0.5376, reader_cost=0.0114 | ETA 16:18:29 2020-12-12 18:27:13 [INFO] [TRAIN] epoch=275, iter=51000/160000, loss=0.8996, lr=0.014158, batch_cost=0.5404, reader_cost=0.0148 | ETA 16:21:47 2020-12-12 18:29:00 [INFO] [TRAIN] epoch=276, iter=51200/160000, loss=0.9029, lr=0.014135, batch_cost=0.5348, reader_cost=0.0129 | ETA 16:09:50 2020-12-12 18:30:47 [INFO] [TRAIN] epoch=277, iter=51400/160000, loss=0.9021, lr=0.014111, batch_cost=0.5377, reader_cost=0.0124 | ETA 16:13:12 2020-12-12 18:32:34 [INFO] [TRAIN] epoch=278, iter=51600/160000, loss=0.9219, lr=0.014088, batch_cost=0.5353, reader_cost=0.0109 | ETA 16:07:10 2020-12-12 18:34:21 [INFO] [TRAIN] epoch=279, iter=51800/160000, loss=0.8964, lr=0.014065, batch_cost=0.5365, reader_cost=0.0127 | ETA 16:07:30 2020-12-12 18:36:09 [INFO] [TRAIN] epoch=280, iter=52000/160000, loss=0.8971, lr=0.014041, batch_cost=0.5363, reader_cost=0.0134 | ETA 16:05:16 2020-12-12 18:36:09 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-12 18:36:39 [INFO] [EVAL] #Images=500 mIoU=0.6148 Acc=0.9243 Kappa=0.9021 2020-12-12 18:36:39 [INFO] [EVAL] Class IoU: [0.9548 0.7288 0.8646 0.2014 0.4206 0.5345 0.4684 0.6988 0.86 0.4238 0.9287 0.7328 0.5078 0.8842 0.4573 0.642 0.3648 0.3263 0.6811] 2020-12-12 18:36:39 [INFO] [EVAL] Class Acc: [0.9913 0.7832 0.8987 0.4373 0.7983 0.8237 0.5984 0.8878 0.952 0.494 0.9507 0.8931 0.7004 0.9081 0.6787 0.7954 0.4825 0.4387 0.8328] 2020-12-12 18:36:39 [INFO] [EVAL] The model with the best validation mIoU (0.6539) was saved at iter 28000. 2020-12-12 18:38:27 [INFO] [TRAIN] epoch=281, iter=52200/160000, loss=0.9062, lr=0.014018, batch_cost=0.5368, reader_cost=0.0129 | ETA 16:04:26 2020-12-12 18:40:14 [INFO] [TRAIN] epoch=282, iter=52400/160000, loss=0.8907, lr=0.013994, batch_cost=0.5382, reader_cost=0.0119 | ETA 16:05:08 2020-12-12 18:42:02 [INFO] [TRAIN] epoch=283, iter=52600/160000, loss=0.8995, lr=0.013971, batch_cost=0.5374, reader_cost=0.0117 | ETA 16:01:54 2020-12-12 18:43:49 [INFO] [TRAIN] epoch=284, iter=52800/160000, loss=0.9041, lr=0.013948, batch_cost=0.5347, reader_cost=0.0116 | ETA 15:55:15 2020-12-12 18:45:36 [INFO] [TRAIN] epoch=285, iter=53000/160000, loss=0.8994, lr=0.013924, batch_cost=0.5378, reader_cost=0.0121 | ETA 15:59:01 2020-12-12 18:47:26 [INFO] [TRAIN] epoch=287, iter=53200/160000, loss=0.8993, lr=0.013901, batch_cost=0.5456, reader_cost=0.0221 | ETA 16:11:06 2020-12-12 18:49:13 [INFO] [TRAIN] epoch=288, iter=53400/160000, loss=0.9038, lr=0.013877, batch_cost=0.5384, reader_cost=0.0136 | ETA 15:56:30 2020-12-12 18:51:00 [INFO] [TRAIN] epoch=289, iter=53600/160000, loss=0.9125, lr=0.013854, batch_cost=0.5351, reader_cost=0.0118 | ETA 15:48:52 2020-12-12 18:52:47 [INFO] [TRAIN] epoch=290, iter=53800/160000, loss=0.9002, lr=0.013830, batch_cost=0.5360, reader_cost=0.0147 | ETA 15:48:47 2020-12-12 18:54:35 [INFO] [TRAIN] epoch=291, iter=54000/160000, loss=0.9011, lr=0.013807, batch_cost=0.5356, reader_cost=0.0135 | ETA 15:46:17 2020-12-12 18:56:22 [INFO] [TRAIN] epoch=292, iter=54200/160000, loss=0.9228, lr=0.013784, batch_cost=0.5370, reader_cost=0.0135 | ETA 15:46:59 2020-12-12 18:58:10 [INFO] [TRAIN] epoch=293, iter=54400/160000, loss=0.9038, lr=0.013760, batch_cost=0.5381, reader_cost=0.0134 | ETA 15:47:06 2020-12-12 18:59:57 [INFO] [TRAIN] epoch=294, iter=54600/160000, loss=0.9038, lr=0.013737, batch_cost=0.5365, reader_cost=0.0112 | ETA 15:42:31 2020-12-12 19:01:44 [INFO] [TRAIN] epoch=295, iter=54800/160000, loss=0.8997, lr=0.013713, batch_cost=0.5364, reader_cost=0.0146 | ETA 15:40:30 2020-12-12 19:03:32 [INFO] [TRAIN] epoch=296, iter=55000/160000, loss=0.8912, lr=0.013690, batch_cost=0.5395, reader_cost=0.0137 | ETA 15:44:06 2020-12-12 19:05:20 [INFO] [TRAIN] epoch=297, iter=55200/160000, loss=0.9186, lr=0.013666, batch_cost=0.5388, reader_cost=0.0131 | ETA 15:41:06 2020-12-12 19:07:07 [INFO] [TRAIN] epoch=298, iter=55400/160000, loss=0.9060, lr=0.013643, batch_cost=0.5378, reader_cost=0.0125 | ETA 15:37:29 2020-12-12 19:08:55 [INFO] [TRAIN] epoch=299, iter=55600/160000, loss=0.8951, lr=0.013619, batch_cost=0.5369, reader_cost=0.0120 | ETA 15:34:08 2020-12-12 19:10:42 [INFO] [TRAIN] epoch=300, iter=55800/160000, loss=0.9114, lr=0.013596, batch_cost=0.5348, reader_cost=0.0127 | ETA 15:28:44 2020-12-12 19:12:31 [INFO] [TRAIN] epoch=302, iter=56000/160000, loss=0.9020, lr=0.013572, batch_cost=0.5461, reader_cost=0.0227 | ETA 15:46:30 2020-12-12 19:12:31 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-12 19:13:01 [INFO] [EVAL] #Images=500 mIoU=0.6119 Acc=0.9290 Kappa=0.9080 2020-12-12 19:13:01 [INFO] [EVAL] Class IoU: [0.9527 0.7397 0.8867 0.2016 0.3699 0.5332 0.5688 0.7439 0.8828 0.4474 0.9332 0.7255 0.4997 0.8883 0.4968 0.602 0.1509 0.3201 0.682 ] 2020-12-12 19:13:01 [INFO] [EVAL] Class Acc: [0.9869 0.8944 0.9335 0.3679 0.4563 0.8013 0.7127 0.9203 0.917 0.603 0.9765 0.7861 0.7815 0.9173 0.7165 0.6591 0.7715 0.5142 0.8215] 2020-12-12 19:13:01 [INFO] [EVAL] The model with the best validation mIoU (0.6539) was saved at iter 28000. 2020-12-12 19:14:49 [INFO] [TRAIN] epoch=303, iter=56200/160000, loss=0.8958, lr=0.013549, batch_cost=0.5390, reader_cost=0.0134 | ETA 15:32:26 2020-12-12 19:16:36 [INFO] [TRAIN] epoch=304, iter=56400/160000, loss=0.9216, lr=0.013525, batch_cost=0.5362, reader_cost=0.0133 | ETA 15:25:53 2020-12-12 19:18:25 [INFO] [TRAIN] epoch=305, iter=56600/160000, loss=0.9075, lr=0.013502, batch_cost=0.5430, reader_cost=0.0130 | ETA 15:35:49 2020-12-12 19:20:13 [INFO] [TRAIN] epoch=306, iter=56800/160000, loss=0.9086, lr=0.013478, batch_cost=0.5396, reader_cost=0.0127 | ETA 15:28:11 2020-12-12 19:22:00 [INFO] [TRAIN] epoch=307, iter=57000/160000, loss=0.9004, lr=0.013455, batch_cost=0.5363, reader_cost=0.0126 | ETA 15:20:42 2020-12-12 19:23:48 [INFO] [TRAIN] epoch=308, iter=57200/160000, loss=0.9023, lr=0.013431, batch_cost=0.5397, reader_cost=0.0146 | ETA 15:24:45 2020-12-12 19:25:35 [INFO] [TRAIN] epoch=309, iter=57400/160000, loss=0.8973, lr=0.013408, batch_cost=0.5374, reader_cost=0.0136 | ETA 15:18:55 2020-12-12 19:27:23 [INFO] [TRAIN] epoch=310, iter=57600/160000, loss=0.8863, lr=0.013384, batch_cost=0.5374, reader_cost=0.0131 | ETA 15:17:11 2020-12-12 19:29:11 [INFO] [TRAIN] epoch=311, iter=57800/160000, loss=0.8877, lr=0.013361, batch_cost=0.5385, reader_cost=0.0137 | ETA 15:17:14 2020-12-12 19:30:58 [INFO] [TRAIN] epoch=312, iter=58000/160000, loss=0.9132, lr=0.013337, batch_cost=0.5373, reader_cost=0.0130 | ETA 15:13:22 2020-12-12 19:32:45 [INFO] [TRAIN] epoch=313, iter=58200/160000, loss=0.9006, lr=0.013314, batch_cost=0.5357, reader_cost=0.0133 | ETA 15:08:56 2020-12-12 19:34:33 [INFO] [TRAIN] epoch=314, iter=58400/160000, loss=0.9086, lr=0.013290, batch_cost=0.5375, reader_cost=0.0129 | ETA 15:10:06 2020-12-12 19:36:22 [INFO] [TRAIN] epoch=316, iter=58600/160000, loss=0.9032, lr=0.013267, batch_cost=0.5473, reader_cost=0.0230 | ETA 15:24:53 2020-12-12 19:38:09 [INFO] [TRAIN] epoch=317, iter=58800/160000, loss=0.9066, lr=0.013243, batch_cost=0.5361, reader_cost=0.0129 | ETA 15:04:15 2020-12-12 19:39:57 [INFO] [TRAIN] epoch=318, iter=59000/160000, loss=0.8849, lr=0.013220, batch_cost=0.5395, reader_cost=0.0138 | ETA 15:08:09 2020-12-12 19:41:45 [INFO] [TRAIN] epoch=319, iter=59200/160000, loss=0.8970, lr=0.013196, batch_cost=0.5400, reader_cost=0.0128 | ETA 15:07:13 2020-12-12 19:43:33 [INFO] [TRAIN] epoch=320, iter=59400/160000, loss=0.9036, lr=0.013172, batch_cost=0.5387, reader_cost=0.0123 | ETA 15:03:14 2020-12-12 19:45:20 [INFO] [TRAIN] epoch=321, iter=59600/160000, loss=0.8975, lr=0.013149, batch_cost=0.5371, reader_cost=0.0130 | ETA 14:58:48 2020-12-12 19:47:08 [INFO] [TRAIN] epoch=322, iter=59800/160000, loss=0.8904, lr=0.013125, batch_cost=0.5375, reader_cost=0.0129 | ETA 14:57:34 2020-12-12 19:48:56 [INFO] [TRAIN] epoch=323, iter=60000/160000, loss=0.8901, lr=0.013102, batch_cost=0.5382, reader_cost=0.0132 | ETA 14:56:55 2020-12-12 19:48:56 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-12 19:49:26 [INFO] [EVAL] #Images=500 mIoU=0.5363 Acc=0.9086 Kappa=0.8811 2020-12-12 19:49:26 [INFO] [EVAL] Class IoU: [0.9514 0.734 0.8397 0.1168 0.3994 0.4479 0.5152 0.7037 0.8122 0.5427 0.9329 0.6756 0.3652 0.7416 0.1162 0.3419 0.1208 0.1806 0.6515] 2020-12-12 19:49:26 [INFO] [EVAL] Class Acc: [0.9765 0.9202 0.8681 0.6939 0.8137 0.8871 0.7376 0.8529 0.9516 0.6668 0.972 0.7352 0.4496 0.7532 0.5624 0.5019 0.2796 0.3006 0.7721] 2020-12-12 19:49:27 [INFO] [EVAL] The model with the best validation mIoU (0.6539) was saved at iter 28000. 2020-12-12 19:51:14 [INFO] [TRAIN] epoch=324, iter=60200/160000, loss=0.8932, lr=0.013078, batch_cost=0.5353, reader_cost=0.0118 | ETA 14:50:24 2020-12-12 19:53:02 [INFO] [TRAIN] epoch=325, iter=60400/160000, loss=0.9094, lr=0.013054, batch_cost=0.5397, reader_cost=0.0135 | ETA 14:55:50 2020-12-12 19:54:49 [INFO] [TRAIN] epoch=326, iter=60600/160000, loss=0.8927, lr=0.013031, batch_cost=0.5368, reader_cost=0.0127 | ETA 14:49:13 2020-12-12 19:56:37 [INFO] [TRAIN] epoch=327, iter=60800/160000, loss=0.8943, lr=0.013007, batch_cost=0.5376, reader_cost=0.0135 | ETA 14:48:54 2020-12-12 19:58:24 [INFO] [TRAIN] epoch=328, iter=61000/160000, loss=0.8875, lr=0.012984, batch_cost=0.5380, reader_cost=0.0136 | ETA 14:47:43 2020-12-12 20:00:13 [INFO] [TRAIN] epoch=330, iter=61200/160000, loss=0.9054, lr=0.012960, batch_cost=0.5442, reader_cost=0.0229 | ETA 14:56:06 2020-12-12 20:02:00 [INFO] [TRAIN] epoch=331, iter=61400/160000, loss=0.9225, lr=0.012936, batch_cost=0.5360, reader_cost=0.0124 | ETA 14:40:52 2020-12-12 20:03:48 [INFO] [TRAIN] epoch=332, iter=61600/160000, loss=0.8935, lr=0.012913, batch_cost=0.5373, reader_cost=0.0118 | ETA 14:41:11 2020-12-12 20:05:35 [INFO] [TRAIN] epoch=333, iter=61800/160000, loss=0.8879, lr=0.012889, batch_cost=0.5353, reader_cost=0.0120 | ETA 14:36:07 2020-12-12 20:07:23 [INFO] [TRAIN] epoch=334, iter=62000/160000, loss=0.9003, lr=0.012866, batch_cost=0.5382, reader_cost=0.0122 | ETA 14:39:07 2020-12-12 20:09:10 [INFO] [TRAIN] epoch=335, iter=62200/160000, loss=0.9026, lr=0.012842, batch_cost=0.5364, reader_cost=0.0132 | ETA 14:34:20 2020-12-12 20:10:57 [INFO] [TRAIN] epoch=336, iter=62400/160000, loss=0.8997, lr=0.012818, batch_cost=0.5348, reader_cost=0.0129 | ETA 14:29:57 2020-12-12 20:12:44 [INFO] [TRAIN] epoch=337, iter=62600/160000, loss=0.8995, lr=0.012795, batch_cost=0.5352, reader_cost=0.0129 | ETA 14:28:45 2020-12-12 20:14:31 [INFO] [TRAIN] epoch=338, iter=62800/160000, loss=0.8940, lr=0.012771, batch_cost=0.5348, reader_cost=0.0120 | ETA 14:26:24 2020-12-12 20:16:18 [INFO] [TRAIN] epoch=339, iter=63000/160000, loss=0.8847, lr=0.012747, batch_cost=0.5375, reader_cost=0.0134 | ETA 14:28:57 2020-12-12 20:18:06 [INFO] [TRAIN] epoch=340, iter=63200/160000, loss=0.8971, lr=0.012724, batch_cost=0.5377, reader_cost=0.0143 | ETA 14:27:27 2020-12-12 20:19:54 [INFO] [TRAIN] epoch=341, iter=63400/160000, loss=0.8971, lr=0.012700, batch_cost=0.5393, reader_cost=0.0135 | ETA 14:28:16 2020-12-12 20:21:42 [INFO] [TRAIN] epoch=342, iter=63600/160000, loss=0.9090, lr=0.012676, batch_cost=0.5398, reader_cost=0.0129 | ETA 14:27:13 2020-12-12 20:23:31 [INFO] [TRAIN] epoch=344, iter=63800/160000, loss=0.8926, lr=0.012653, batch_cost=0.5469, reader_cost=0.0228 | ETA 14:36:52 2020-12-12 20:25:18 [INFO] [TRAIN] epoch=345, iter=64000/160000, loss=0.8936, lr=0.012629, batch_cost=0.5369, reader_cost=0.0135 | ETA 14:18:58 2020-12-12 20:25:18 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-12 20:25:48 [INFO] [EVAL] #Images=500 mIoU=0.5957 Acc=0.9282 Kappa=0.9069 2020-12-12 20:25:48 [INFO] [EVAL] Class IoU: [0.9584 0.7376 0.8754 0.2948 0.4469 0.538 0.4685 0.7273 0.8742 0.5119 0.929 0.7423 0.5231 0.8896 0.0305 0.4814 0.5348 0.0958 0.6592] 2020-12-12 20:25:48 [INFO] [EVAL] Class Acc: [0.9801 0.8532 0.9152 0.421 0.5674 0.7237 0.9021 0.8821 0.9604 0.7066 0.9435 0.8129 0.5963 0.9437 0.9155 0.5546 0.7105 0.8679 0.7345] 2020-12-12 20:25:48 [INFO] [EVAL] The model with the best validation mIoU (0.6539) was saved at iter 28000. 2020-12-12 20:27:36 [INFO] [TRAIN] epoch=346, iter=64200/160000, loss=0.8790, lr=0.012605, batch_cost=0.5367, reader_cost=0.0128 | ETA 14:16:57 2020-12-12 20:29:23 [INFO] [TRAIN] epoch=347, iter=64400/160000, loss=0.8897, lr=0.012582, batch_cost=0.5361, reader_cost=0.0126 | ETA 14:14:14 2020-12-12 20:31:10 [INFO] [TRAIN] epoch=348, iter=64600/160000, loss=0.9060, lr=0.012558, batch_cost=0.5365, reader_cost=0.0149 | ETA 14:13:02 2020-12-12 20:32:57 [INFO] [TRAIN] epoch=349, iter=64800/160000, loss=0.8836, lr=0.012534, batch_cost=0.5365, reader_cost=0.0139 | ETA 14:11:15 2020-12-12 20:34:45 [INFO] [TRAIN] epoch=350, iter=65000/160000, loss=0.9144, lr=0.012511, batch_cost=0.5379, reader_cost=0.0132 | ETA 14:11:43 2020-12-12 20:36:32 [INFO] [TRAIN] epoch=351, iter=65200/160000, loss=0.8917, lr=0.012487, batch_cost=0.5370, reader_cost=0.0130 | ETA 14:08:32 2020-12-12 20:38:20 [INFO] [TRAIN] epoch=352, iter=65400/160000, loss=0.9003, lr=0.012463, batch_cost=0.5379, reader_cost=0.0137 | ETA 14:08:09 2020-12-12 20:40:07 [INFO] [TRAIN] epoch=353, iter=65600/160000, loss=0.9046, lr=0.012439, batch_cost=0.5366, reader_cost=0.0131 | ETA 14:04:14 2020-12-12 20:41:55 [INFO] [TRAIN] epoch=354, iter=65800/160000, loss=0.8856, lr=0.012416, batch_cost=0.5377, reader_cost=0.0142 | ETA 14:04:15 2020-12-12 20:43:42 [INFO] [TRAIN] epoch=355, iter=66000/160000, loss=0.8958, lr=0.012392, batch_cost=0.5375, reader_cost=0.0119 | ETA 14:02:09 2020-12-12 20:45:30 [INFO] [TRAIN] epoch=356, iter=66200/160000, loss=0.8918, lr=0.012368, batch_cost=0.5389, reader_cost=0.0141 | ETA 14:02:26 2020-12-12 20:47:17 [INFO] [TRAIN] epoch=357, iter=66400/160000, loss=0.9020, lr=0.012345, batch_cost=0.5343, reader_cost=0.0132 | ETA 13:53:27 2020-12-12 20:49:07 [INFO] [TRAIN] epoch=359, iter=66600/160000, loss=0.8980, lr=0.012321, batch_cost=0.5479, reader_cost=0.0232 | ETA 14:12:56 2020-12-12 20:50:54 [INFO] [TRAIN] epoch=360, iter=66800/160000, loss=0.8895, lr=0.012297, batch_cost=0.5382, reader_cost=0.0128 | ETA 13:55:57 2020-12-12 20:52:42 [INFO] [TRAIN] epoch=361, iter=67000/160000, loss=0.9016, lr=0.012273, batch_cost=0.5388, reader_cost=0.0134 | ETA 13:55:10 2020-12-12 20:54:29 [INFO] [TRAIN] epoch=362, iter=67200/160000, loss=0.8830, lr=0.012250, batch_cost=0.5354, reader_cost=0.0132 | ETA 13:48:01 2020-12-12 20:56:17 [INFO] [TRAIN] epoch=363, iter=67400/160000, loss=0.9025, lr=0.012226, batch_cost=0.5387, reader_cost=0.0138 | ETA 13:51:21 2020-12-12 20:58:04 [INFO] [TRAIN] epoch=364, iter=67600/160000, loss=0.8808, lr=0.012202, batch_cost=0.5384, reader_cost=0.0137 | ETA 13:49:12 2020-12-12 20:59:52 [INFO] [TRAIN] epoch=365, iter=67800/160000, loss=0.8929, lr=0.012178, batch_cost=0.5367, reader_cost=0.0123 | ETA 13:44:41 2020-12-12 21:01:39 [INFO] [TRAIN] epoch=366, iter=68000/160000, loss=0.9030, lr=0.012154, batch_cost=0.5335, reader_cost=0.0136 | ETA 13:38:06 2020-12-12 21:01:39 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-12 21:02:08 [INFO] [EVAL] #Images=500 mIoU=0.6601 Acc=0.9440 Kappa=0.9272 2020-12-12 21:02:08 [INFO] [EVAL] Class IoU: [0.9723 0.7864 0.8999 0.3803 0.527 0.5846 0.6464 0.7468 0.8992 0.5979 0.9448 0.6337 0.4799 0.9279 0.4745 0.7473 0.3473 0.2523 0.6925] 2020-12-12 21:02:08 [INFO] [EVAL] Class Acc: [0.9816 0.9129 0.9308 0.7217 0.6262 0.8589 0.8022 0.9007 0.9649 0.7785 0.971 0.6559 0.7004 0.9571 0.8338 0.8596 0.3905 0.7213 0.8882] 2020-12-12 21:02:09 [INFO] [EVAL] The model with the best validation mIoU (0.6601) was saved at iter 68000. 2020-12-12 21:03:56 [INFO] [TRAIN] epoch=367, iter=68200/160000, loss=0.8794, lr=0.012131, batch_cost=0.5356, reader_cost=0.0133 | ETA 13:39:28 2020-12-12 21:05:43 [INFO] [TRAIN] epoch=368, iter=68400/160000, loss=0.8832, lr=0.012107, batch_cost=0.5380, reader_cost=0.0121 | ETA 13:41:19 2020-12-12 21:07:31 [INFO] [TRAIN] epoch=369, iter=68600/160000, loss=0.8935, lr=0.012083, batch_cost=0.5372, reader_cost=0.0116 | ETA 13:38:20 2020-12-12 21:09:19 [INFO] [TRAIN] epoch=370, iter=68800/160000, loss=0.8806, lr=0.012059, batch_cost=0.5399, reader_cost=0.0129 | ETA 13:40:40 2020-12-12 21:11:06 [INFO] [TRAIN] epoch=371, iter=69000/160000, loss=0.8787, lr=0.012035, batch_cost=0.5352, reader_cost=0.0134 | ETA 13:31:44 2020-12-12 21:12:55 [INFO] [TRAIN] epoch=373, iter=69200/160000, loss=0.8915, lr=0.012012, batch_cost=0.5481, reader_cost=0.0236 | ETA 13:49:23 2020-12-12 21:14:43 [INFO] [TRAIN] epoch=374, iter=69400/160000, loss=0.9021, lr=0.011988, batch_cost=0.5374, reader_cost=0.0133 | ETA 13:31:32 2020-12-12 21:16:30 [INFO] [TRAIN] epoch=375, iter=69600/160000, loss=0.8788, lr=0.011964, batch_cost=0.5372, reader_cost=0.0127 | ETA 13:29:24 2020-12-12 21:18:18 [INFO] [TRAIN] epoch=376, iter=69800/160000, loss=0.8827, lr=0.011940, batch_cost=0.5387, reader_cost=0.0136 | ETA 13:29:48 2020-12-12 21:20:05 [INFO] [TRAIN] epoch=377, iter=70000/160000, loss=0.8858, lr=0.011916, batch_cost=0.5333, reader_cost=0.0120 | ETA 13:19:56 2020-12-12 21:21:52 [INFO] [TRAIN] epoch=378, iter=70200/160000, loss=0.9058, lr=0.011893, batch_cost=0.5372, reader_cost=0.0134 | ETA 13:23:57 2020-12-12 21:23:39 [INFO] [TRAIN] epoch=379, iter=70400/160000, loss=0.8823, lr=0.011869, batch_cost=0.5357, reader_cost=0.0127 | ETA 13:19:56 2020-12-12 21:25:27 [INFO] [TRAIN] epoch=380, iter=70600/160000, loss=0.8903, lr=0.011845, batch_cost=0.5374, reader_cost=0.0141 | ETA 13:20:39 2020-12-12 21:27:14 [INFO] [TRAIN] epoch=381, iter=70800/160000, loss=0.8943, lr=0.011821, batch_cost=0.5364, reader_cost=0.0124 | ETA 13:17:29 2020-12-12 21:29:02 [INFO] [TRAIN] epoch=382, iter=71000/160000, loss=0.8879, lr=0.011797, batch_cost=0.5375, reader_cost=0.0131 | ETA 13:17:19 2020-12-12 21:30:49 [INFO] [TRAIN] epoch=383, iter=71200/160000, loss=0.8780, lr=0.011773, batch_cost=0.5375, reader_cost=0.0122 | ETA 13:15:27 2020-12-12 21:32:36 [INFO] [TRAIN] epoch=384, iter=71400/160000, loss=0.8856, lr=0.011749, batch_cost=0.5351, reader_cost=0.0132 | ETA 13:10:08 2020-12-12 21:34:24 [INFO] [TRAIN] epoch=385, iter=71600/160000, loss=0.8915, lr=0.011726, batch_cost=0.5376, reader_cost=0.0133 | ETA 13:12:04 2020-12-12 21:36:13 [INFO] [TRAIN] epoch=387, iter=71800/160000, loss=0.9099, lr=0.011702, batch_cost=0.5483, reader_cost=0.0232 | ETA 13:26:00 2020-12-12 21:38:01 [INFO] [TRAIN] epoch=388, iter=72000/160000, loss=0.8869, lr=0.011678, batch_cost=0.5373, reader_cost=0.0134 | ETA 13:07:59 2020-12-12 21:38:01 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-12 21:38:32 [INFO] [EVAL] #Images=500 mIoU=0.6444 Acc=0.9381 Kappa=0.9198 2020-12-12 21:38:32 [INFO] [EVAL] Class IoU: [0.9642 0.7619 0.8889 0.3382 0.4976 0.5948 0.631 0.7376 0.9045 0.4813 0.9358 0.7193 0.581 0.8936 0.3564 0.6493 0.1391 0.4538 0.7159] 2020-12-12 21:38:32 [INFO] [EVAL] Class Acc: [0.9863 0.8272 0.9499 0.4702 0.6516 0.7849 0.8369 0.9086 0.9469 0.7811 0.9534 0.8037 0.6985 0.9167 0.5711 0.7442 0.8305 0.6859 0.8803] 2020-12-12 21:38:32 [INFO] [EVAL] The model with the best validation mIoU (0.6601) was saved at iter 68000. 2020-12-12 21:40:20 [INFO] [TRAIN] epoch=389, iter=72200/160000, loss=0.8986, lr=0.011654, batch_cost=0.5367, reader_cost=0.0123 | ETA 13:05:18 2020-12-12 21:42:07 [INFO] [TRAIN] epoch=390, iter=72400/160000, loss=0.8938, lr=0.011630, batch_cost=0.5370, reader_cost=0.0140 | ETA 13:04:04 2020-12-12 21:43:55 [INFO] [TRAIN] epoch=391, iter=72600/160000, loss=0.8750, lr=0.011606, batch_cost=0.5377, reader_cost=0.0130 | ETA 13:03:15 2020-12-12 21:45:42 [INFO] [TRAIN] epoch=392, iter=72800/160000, loss=0.8947, lr=0.011582, batch_cost=0.5380, reader_cost=0.0137 | ETA 13:01:52 2020-12-12 21:47:29 [INFO] [TRAIN] epoch=393, iter=73000/160000, loss=0.8894, lr=0.011558, batch_cost=0.5340, reader_cost=0.0125 | ETA 12:54:15 2020-12-12 21:49:17 [INFO] [TRAIN] epoch=394, iter=73200/160000, loss=0.8853, lr=0.011534, batch_cost=0.5394, reader_cost=0.0127 | ETA 13:00:22 2020-12-12 21:51:04 [INFO] [TRAIN] epoch=395, iter=73400/160000, loss=0.8838, lr=0.011510, batch_cost=0.5366, reader_cost=0.0137 | ETA 12:54:26 2020-12-12 21:52:52 [INFO] [TRAIN] epoch=396, iter=73600/160000, loss=0.8817, lr=0.011487, batch_cost=0.5371, reader_cost=0.0128 | ETA 12:53:25 2020-12-12 21:54:39 [INFO] [TRAIN] epoch=397, iter=73800/160000, loss=0.8936, lr=0.011463, batch_cost=0.5363, reader_cost=0.0129 | ETA 12:50:29 2020-12-12 21:56:26 [INFO] [TRAIN] epoch=398, iter=74000/160000, loss=0.8862, lr=0.011439, batch_cost=0.5367, reader_cost=0.0130 | ETA 12:49:14 2020-12-12 21:58:13 [INFO] [TRAIN] epoch=399, iter=74200/160000, loss=0.8927, lr=0.011415, batch_cost=0.5360, reader_cost=0.0138 | ETA 12:46:28 2020-12-12 22:00:00 [INFO] [TRAIN] epoch=400, iter=74400/160000, loss=0.8983, lr=0.011391, batch_cost=0.5351, reader_cost=0.0114 | ETA 12:43:26 2020-12-12 22:01:49 [INFO] [TRAIN] epoch=402, iter=74600/160000, loss=0.8969, lr=0.011367, batch_cost=0.5453, reader_cost=0.0220 | ETA 12:56:10 2020-12-12 22:03:37 [INFO] [TRAIN] epoch=403, iter=74800/160000, loss=0.8992, lr=0.011343, batch_cost=0.5396, reader_cost=0.0135 | ETA 12:46:15 2020-12-12 22:05:25 [INFO] [TRAIN] epoch=404, iter=75000/160000, loss=0.8816, lr=0.011319, batch_cost=0.5369, reader_cost=0.0128 | ETA 12:40:37 2020-12-12 22:07:12 [INFO] [TRAIN] epoch=405, iter=75200/160000, loss=0.8833, lr=0.011295, batch_cost=0.5380, reader_cost=0.0137 | ETA 12:40:21 2020-12-12 22:09:00 [INFO] [TRAIN] epoch=406, iter=75400/160000, loss=0.8967, lr=0.011271, batch_cost=0.5374, reader_cost=0.0119 | ETA 12:37:43 2020-12-12 22:10:47 [INFO] [TRAIN] epoch=407, iter=75600/160000, loss=0.8930, lr=0.011247, batch_cost=0.5373, reader_cost=0.0134 | ETA 12:35:51 2020-12-12 22:12:35 [INFO] [TRAIN] epoch=408, iter=75800/160000, loss=0.9048, lr=0.011223, batch_cost=0.5397, reader_cost=0.0132 | ETA 12:37:18 2020-12-12 22:14:23 [INFO] [TRAIN] epoch=409, iter=76000/160000, loss=0.8926, lr=0.011199, batch_cost=0.5381, reader_cost=0.0136 | ETA 12:33:18 2020-12-12 22:14:23 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-12 22:14:53 [INFO] [EVAL] #Images=500 mIoU=0.6554 Acc=0.9402 Kappa=0.9225 2020-12-12 22:14:53 [INFO] [EVAL] Class IoU: [0.9651 0.7729 0.8879 0.3163 0.5283 0.6124 0.6338 0.7628 0.8933 0.5327 0.9368 0.7387 0.4703 0.9248 0.6162 0.6706 0.1741 0.3261 0.6899] 2020-12-12 22:14:53 [INFO] [EVAL] Class Acc: [0.9868 0.8597 0.9241 0.611 0.7527 0.7471 0.7449 0.8973 0.9543 0.8094 0.9534 0.7818 0.6348 0.9646 0.6845 0.701 0.3084 0.783 0.8937] 2020-12-12 22:14:53 [INFO] [EVAL] The model with the best validation mIoU (0.6601) was saved at iter 68000. 2020-12-12 22:16:40 [INFO] [TRAIN] epoch=410, iter=76200/160000, loss=0.8847, lr=0.011175, batch_cost=0.5360, reader_cost=0.0126 | ETA 12:28:36 2020-12-12 22:18:28 [INFO] [TRAIN] epoch=411, iter=76400/160000, loss=0.8790, lr=0.011151, batch_cost=0.5367, reader_cost=0.0130 | ETA 12:27:50 2020-12-12 22:20:16 [INFO] [TRAIN] epoch=412, iter=76600/160000, loss=0.8892, lr=0.011127, batch_cost=0.5390, reader_cost=0.0129 | ETA 12:29:16 2020-12-12 22:22:03 [INFO] [TRAIN] epoch=413, iter=76800/160000, loss=0.8778, lr=0.011103, batch_cost=0.5357, reader_cost=0.0134 | ETA 12:22:47 2020-12-12 22:23:50 [INFO] [TRAIN] epoch=414, iter=77000/160000, loss=0.8749, lr=0.011079, batch_cost=0.5359, reader_cost=0.0130 | ETA 12:21:20 2020-12-12 22:25:40 [INFO] [TRAIN] epoch=416, iter=77200/160000, loss=0.8785, lr=0.011055, batch_cost=0.5498, reader_cost=0.0229 | ETA 12:38:46 2020-12-12 22:27:27 [INFO] [TRAIN] epoch=417, iter=77400/160000, loss=0.8913, lr=0.011031, batch_cost=0.5357, reader_cost=0.0143 | ETA 12:17:28 2020-12-12 22:29:14 [INFO] [TRAIN] epoch=418, iter=77600/160000, loss=0.8796, lr=0.011007, batch_cost=0.5356, reader_cost=0.0128 | ETA 12:15:36 2020-12-12 22:31:02 [INFO] [TRAIN] epoch=419, iter=77800/160000, loss=0.8810, lr=0.010983, batch_cost=0.5387, reader_cost=0.0138 | ETA 12:18:00 2020-12-12 22:32:49 [INFO] [TRAIN] epoch=420, iter=78000/160000, loss=0.8912, lr=0.010959, batch_cost=0.5368, reader_cost=0.0137 | ETA 12:13:39 2020-12-12 22:34:36 [INFO] [TRAIN] epoch=421, iter=78200/160000, loss=0.8923, lr=0.010935, batch_cost=0.5362, reader_cost=0.0138 | ETA 12:10:58 2020-12-12 22:36:24 [INFO] [TRAIN] epoch=422, iter=78400/160000, loss=0.8884, lr=0.010911, batch_cost=0.5367, reader_cost=0.0142 | ETA 12:09:51 2020-12-12 22:38:11 [INFO] [TRAIN] epoch=423, iter=78600/160000, loss=0.8768, lr=0.010887, batch_cost=0.5367, reader_cost=0.0142 | ETA 12:08:06 2020-12-12 22:39:58 [INFO] [TRAIN] epoch=424, iter=78800/160000, loss=0.8764, lr=0.010862, batch_cost=0.5354, reader_cost=0.0123 | ETA 12:04:38 2020-12-12 22:41:45 [INFO] [TRAIN] epoch=425, iter=79000/160000, loss=0.8910, lr=0.010838, batch_cost=0.5359, reader_cost=0.0128 | ETA 12:03:30 2020-12-12 22:43:33 [INFO] [TRAIN] epoch=426, iter=79200/160000, loss=0.8952, lr=0.010814, batch_cost=0.5376, reader_cost=0.0135 | ETA 12:03:57 2020-12-12 22:45:20 [INFO] [TRAIN] epoch=427, iter=79400/160000, loss=0.8601, lr=0.010790, batch_cost=0.5357, reader_cost=0.0139 | ETA 11:59:38 2020-12-12 22:47:07 [INFO] [TRAIN] epoch=428, iter=79600/160000, loss=0.8853, lr=0.010766, batch_cost=0.5360, reader_cost=0.0128 | ETA 11:58:17 2020-12-12 22:48:56 [INFO] [TRAIN] epoch=430, iter=79800/160000, loss=0.8938, lr=0.010742, batch_cost=0.5457, reader_cost=0.0228 | ETA 12:09:23 2020-12-12 22:50:44 [INFO] [TRAIN] epoch=431, iter=80000/160000, loss=0.8704, lr=0.010718, batch_cost=0.5372, reader_cost=0.0127 | ETA 11:56:13 2020-12-12 22:50:44 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-12 22:51:15 [INFO] [EVAL] #Images=500 mIoU=0.6837 Acc=0.9504 Kappa=0.9355 2020-12-12 22:51:15 [INFO] [EVAL] Class IoU: [0.9753 0.811 0.9114 0.4802 0.5494 0.6123 0.6606 0.7498 0.914 0.5583 0.9237 0.7847 0.4798 0.9194 0.6005 0.6721 0.6043 0.1548 0.6279] 2020-12-12 22:51:15 [INFO] [EVAL] Class Acc: [0.984 0.9204 0.9434 0.7149 0.7493 0.847 0.8295 0.8529 0.9506 0.875 0.9324 0.8828 0.7851 0.94 0.8862 0.8348 0.7076 0.2449 0.9145] 2020-12-12 22:51:15 [INFO] [EVAL] The model with the best validation mIoU (0.6837) was saved at iter 80000. 2020-12-12 22:53:03 [INFO] [TRAIN] epoch=432, iter=80200/160000, loss=0.8744, lr=0.010694, batch_cost=0.5369, reader_cost=0.0142 | ETA 11:54:06 2020-12-12 22:54:50 [INFO] [TRAIN] epoch=433, iter=80400/160000, loss=0.8704, lr=0.010670, batch_cost=0.5375, reader_cost=0.0148 | ETA 11:53:04 2020-12-12 22:56:38 [INFO] [TRAIN] epoch=434, iter=80600/160000, loss=0.8797, lr=0.010645, batch_cost=0.5372, reader_cost=0.0133 | ETA 11:50:54 2020-12-12 22:58:25 [INFO] [TRAIN] epoch=435, iter=80800/160000, loss=0.8798, lr=0.010621, batch_cost=0.5372, reader_cost=0.0132 | ETA 11:49:04 2020-12-12 23:00:13 [INFO] [TRAIN] epoch=436, iter=81000/160000, loss=0.8881, lr=0.010597, batch_cost=0.5404, reader_cost=0.0132 | ETA 11:51:33 2020-12-12 23:02:01 [INFO] [TRAIN] epoch=437, iter=81200/160000, loss=0.8767, lr=0.010573, batch_cost=0.5367, reader_cost=0.0143 | ETA 11:44:55 2020-12-12 23:03:48 [INFO] [TRAIN] epoch=438, iter=81400/160000, loss=0.8824, lr=0.010549, batch_cost=0.5362, reader_cost=0.0132 | ETA 11:42:27 2020-12-12 23:05:35 [INFO] [TRAIN] epoch=439, iter=81600/160000, loss=0.8763, lr=0.010525, batch_cost=0.5377, reader_cost=0.0134 | ETA 11:42:33 2020-12-12 23:07:23 [INFO] [TRAIN] epoch=440, iter=81800/160000, loss=0.8890, lr=0.010501, batch_cost=0.5397, reader_cost=0.0135 | ETA 11:43:23 2020-12-12 23:09:11 [INFO] [TRAIN] epoch=441, iter=82000/160000, loss=0.8744, lr=0.010476, batch_cost=0.5387, reader_cost=0.0128 | ETA 11:40:16 2020-12-12 23:10:58 [INFO] [TRAIN] epoch=442, iter=82200/160000, loss=0.8758, lr=0.010452, batch_cost=0.5346, reader_cost=0.0118 | ETA 11:33:10 2020-12-12 23:12:47 [INFO] [TRAIN] epoch=444, iter=82400/160000, loss=0.8750, lr=0.010428, batch_cost=0.5453, reader_cost=0.0213 | ETA 11:45:18 2020-12-12 23:14:34 [INFO] [TRAIN] epoch=445, iter=82600/160000, loss=0.8863, lr=0.010404, batch_cost=0.5359, reader_cost=0.0128 | ETA 11:31:19 2020-12-12 23:16:21 [INFO] [TRAIN] epoch=446, iter=82800/160000, loss=0.8827, lr=0.010380, batch_cost=0.5355, reader_cost=0.0130 | ETA 11:29:00 2020-12-12 23:18:09 [INFO] [TRAIN] epoch=447, iter=83000/160000, loss=0.8791, lr=0.010355, batch_cost=0.5369, reader_cost=0.0135 | ETA 11:29:02 2020-12-12 23:19:55 [INFO] [TRAIN] epoch=448, iter=83200/160000, loss=0.8678, lr=0.010331, batch_cost=0.5332, reader_cost=0.0129 | ETA 11:22:27 2020-12-12 23:21:43 [INFO] [TRAIN] epoch=449, iter=83400/160000, loss=0.8694, lr=0.010307, batch_cost=0.5366, reader_cost=0.0126 | ETA 11:25:00 2020-12-12 23:23:30 [INFO] [TRAIN] epoch=450, iter=83600/160000, loss=0.8812, lr=0.010283, batch_cost=0.5362, reader_cost=0.0136 | ETA 11:22:43 2020-12-12 23:25:17 [INFO] [TRAIN] epoch=451, iter=83800/160000, loss=0.8837, lr=0.010259, batch_cost=0.5374, reader_cost=0.0120 | ETA 11:22:26 2020-12-12 23:27:04 [INFO] [TRAIN] epoch=452, iter=84000/160000, loss=0.8799, lr=0.010234, batch_cost=0.5353, reader_cost=0.0138 | ETA 11:18:02 2020-12-12 23:27:04 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-12 23:27:35 [INFO] [EVAL] #Images=500 mIoU=0.6609 Acc=0.9479 Kappa=0.9322 2020-12-12 23:27:35 [INFO] [EVAL] Class IoU: [0.9754 0.8058 0.9042 0.4304 0.4708 0.5695 0.6572 0.7541 0.9106 0.5582 0.9444 0.768 0.5707 0.9156 0.2921 0.6257 0.3081 0.3792 0.7165] 2020-12-12 23:27:35 [INFO] [EVAL] Class Acc: [0.9887 0.8777 0.9319 0.6423 0.7879 0.8612 0.8656 0.8938 0.9504 0.8538 0.9668 0.8737 0.8133 0.9381 0.952 0.7191 0.7518 0.5276 0.8468] 2020-12-12 23:27:35 [INFO] [EVAL] The model with the best validation mIoU (0.6837) was saved at iter 80000. 2020-12-12 23:29:22 [INFO] [TRAIN] epoch=453, iter=84200/160000, loss=0.8904, lr=0.010210, batch_cost=0.5349, reader_cost=0.0138 | ETA 11:15:43 2020-12-12 23:31:10 [INFO] [TRAIN] epoch=454, iter=84400/160000, loss=0.8738, lr=0.010186, batch_cost=0.5384, reader_cost=0.0128 | ETA 11:18:21 2020-12-12 23:32:57 [INFO] [TRAIN] epoch=455, iter=84600/160000, loss=0.8637, lr=0.010162, batch_cost=0.5355, reader_cost=0.0132 | ETA 11:12:57 2020-12-12 23:34:44 [INFO] [TRAIN] epoch=456, iter=84800/160000, loss=0.8776, lr=0.010137, batch_cost=0.5360, reader_cost=0.0125 | ETA 11:11:49 2020-12-12 23:36:32 [INFO] [TRAIN] epoch=457, iter=85000/160000, loss=0.8778, lr=0.010113, batch_cost=0.5388, reader_cost=0.0142 | ETA 11:13:31 2020-12-12 23:38:21 [INFO] [TRAIN] epoch=459, iter=85200/160000, loss=0.8805, lr=0.010089, batch_cost=0.5477, reader_cost=0.0235 | ETA 11:22:46 2020-12-12 23:40:08 [INFO] [TRAIN] epoch=460, iter=85400/160000, loss=0.8833, lr=0.010064, batch_cost=0.5348, reader_cost=0.0130 | ETA 11:04:57 2020-12-12 23:41:56 [INFO] [TRAIN] epoch=461, iter=85600/160000, loss=0.8663, lr=0.010040, batch_cost=0.5378, reader_cost=0.0130 | ETA 11:06:55 2020-12-12 23:43:44 [INFO] [TRAIN] epoch=462, iter=85800/160000, loss=0.8835, lr=0.010016, batch_cost=0.5378, reader_cost=0.0131 | ETA 11:05:08 2020-12-12 23:45:30 [INFO] [TRAIN] epoch=463, iter=86000/160000, loss=0.8800, lr=0.009992, batch_cost=0.5342, reader_cost=0.0128 | ETA 10:58:49 2020-12-12 23:47:18 [INFO] [TRAIN] epoch=464, iter=86200/160000, loss=0.8661, lr=0.009967, batch_cost=0.5375, reader_cost=0.0132 | ETA 11:01:09 2020-12-12 23:49:05 [INFO] [TRAIN] epoch=465, iter=86400/160000, loss=0.8776, lr=0.009943, batch_cost=0.5350, reader_cost=0.0131 | ETA 10:56:15 2020-12-12 23:50:52 [INFO] [TRAIN] epoch=466, iter=86600/160000, loss=0.8874, lr=0.009919, batch_cost=0.5343, reader_cost=0.0125 | ETA 10:53:36 2020-12-12 23:52:39 [INFO] [TRAIN] epoch=467, iter=86800/160000, loss=0.8725, lr=0.009894, batch_cost=0.5374, reader_cost=0.0145 | ETA 10:55:40 2020-12-12 23:54:26 [INFO] [TRAIN] epoch=468, iter=87000/160000, loss=0.8797, lr=0.009870, batch_cost=0.5349, reader_cost=0.0126 | ETA 10:50:46 2020-12-12 23:56:13 [INFO] [TRAIN] epoch=469, iter=87200/160000, loss=0.8839, lr=0.009846, batch_cost=0.5355, reader_cost=0.0125 | ETA 10:49:44 2020-12-12 23:58:00 [INFO] [TRAIN] epoch=470, iter=87400/160000, loss=0.8748, lr=0.009821, batch_cost=0.5337, reader_cost=0.0118 | ETA 10:45:44 2020-12-12 23:59:47 [INFO] [TRAIN] epoch=471, iter=87600/160000, loss=0.8653, lr=0.009797, batch_cost=0.5349, reader_cost=0.0136 | ETA 10:45:29 2020-12-13 00:01:36 [INFO] [TRAIN] epoch=473, iter=87800/160000, loss=0.8767, lr=0.009773, batch_cost=0.5449, reader_cost=0.0208 | ETA 10:55:38 2020-12-13 00:03:23 [INFO] [TRAIN] epoch=474, iter=88000/160000, loss=0.8663, lr=0.009748, batch_cost=0.5349, reader_cost=0.0129 | ETA 10:41:50 2020-12-13 00:03:23 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-13 00:03:53 [INFO] [EVAL] #Images=500 mIoU=0.6655 Acc=0.9436 Kappa=0.9267 2020-12-13 00:03:53 [INFO] [EVAL] Class IoU: [0.9661 0.7656 0.9025 0.3347 0.5183 0.5995 0.6308 0.7583 0.9071 0.5567 0.9444 0.7762 0.5202 0.9186 0.4776 0.632 0.5128 0.267 0.657 ] 2020-12-13 00:03:53 [INFO] [EVAL] Class Acc: [0.9756 0.9127 0.9477 0.7156 0.6812 0.7536 0.7794 0.8502 0.9503 0.7661 0.964 0.851 0.8088 0.9417 0.5571 0.9182 0.7525 0.299 0.8428] 2020-12-13 00:03:54 [INFO] [EVAL] The model with the best validation mIoU (0.6837) was saved at iter 80000. 2020-12-13 00:05:41 [INFO] [TRAIN] epoch=475, iter=88200/160000, loss=0.8776, lr=0.009724, batch_cost=0.5347, reader_cost=0.0123 | ETA 10:39:52 2020-12-13 00:07:28 [INFO] [TRAIN] epoch=476, iter=88400/160000, loss=0.8624, lr=0.009699, batch_cost=0.5363, reader_cost=0.0126 | ETA 10:40:02 2020-12-13 00:09:15 [INFO] [TRAIN] epoch=477, iter=88600/160000, loss=0.8766, lr=0.009675, batch_cost=0.5347, reader_cost=0.0126 | ETA 10:36:17 2020-12-13 00:11:02 [INFO] [TRAIN] epoch=478, iter=88800/160000, loss=0.8763, lr=0.009651, batch_cost=0.5359, reader_cost=0.0141 | ETA 10:35:59 2020-12-13 00:12:49 [INFO] [TRAIN] epoch=479, iter=89000/160000, loss=0.8641, lr=0.009626, batch_cost=0.5347, reader_cost=0.0131 | ETA 10:32:42 2020-12-13 00:14:36 [INFO] [TRAIN] epoch=480, iter=89200/160000, loss=0.8895, lr=0.009602, batch_cost=0.5346, reader_cost=0.0126 | ETA 10:30:51 2020-12-13 00:16:23 [INFO] [TRAIN] epoch=481, iter=89400/160000, loss=0.8738, lr=0.009577, batch_cost=0.5350, reader_cost=0.0134 | ETA 10:29:31 2020-12-13 00:18:10 [INFO] [TRAIN] epoch=482, iter=89600/160000, loss=0.8644, lr=0.009553, batch_cost=0.5359, reader_cost=0.0137 | ETA 10:28:47 2020-12-13 00:19:58 [INFO] [TRAIN] epoch=483, iter=89800/160000, loss=0.8572, lr=0.009529, batch_cost=0.5379, reader_cost=0.0119 | ETA 10:29:17 2020-12-13 00:21:45 [INFO] [TRAIN] epoch=484, iter=90000/160000, loss=0.8813, lr=0.009504, batch_cost=0.5358, reader_cost=0.0138 | ETA 10:25:06 2020-12-13 00:23:33 [INFO] [TRAIN] epoch=485, iter=90200/160000, loss=0.8723, lr=0.009480, batch_cost=0.5380, reader_cost=0.0142 | ETA 10:25:50 2020-12-13 00:25:22 [INFO] [TRAIN] epoch=487, iter=90400/160000, loss=0.8681, lr=0.009455, batch_cost=0.5449, reader_cost=0.0212 | ETA 10:32:07 2020-12-13 00:27:09 [INFO] [TRAIN] epoch=488, iter=90600/160000, loss=0.8673, lr=0.009431, batch_cost=0.5358, reader_cost=0.0137 | ETA 10:19:43 2020-12-13 00:28:56 [INFO] [TRAIN] epoch=489, iter=90800/160000, loss=0.8806, lr=0.009406, batch_cost=0.5357, reader_cost=0.0135 | ETA 10:17:48 2020-12-13 00:30:44 [INFO] [TRAIN] epoch=490, iter=91000/160000, loss=0.8760, lr=0.009382, batch_cost=0.5376, reader_cost=0.0126 | ETA 10:18:13 2020-12-13 00:32:31 [INFO] [TRAIN] epoch=491, iter=91200/160000, loss=0.8741, lr=0.009357, batch_cost=0.5353, reader_cost=0.0139 | ETA 10:13:50 2020-12-13 00:34:18 [INFO] [TRAIN] epoch=492, iter=91400/160000, loss=0.8672, lr=0.009333, batch_cost=0.5368, reader_cost=0.0140 | ETA 10:13:45 2020-12-13 00:36:05 [INFO] [TRAIN] epoch=493, iter=91600/160000, loss=0.8595, lr=0.009308, batch_cost=0.5350, reader_cost=0.0137 | ETA 10:09:55 2020-12-13 00:37:52 [INFO] [TRAIN] epoch=494, iter=91800/160000, loss=0.8800, lr=0.009284, batch_cost=0.5363, reader_cost=0.0125 | ETA 10:09:38 2020-12-13 00:39:39 [INFO] [TRAIN] epoch=495, iter=92000/160000, loss=0.8682, lr=0.009259, batch_cost=0.5337, reader_cost=0.0117 | ETA 10:04:51 2020-12-13 00:39:39 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-13 00:40:09 [INFO] [EVAL] #Images=500 mIoU=0.6783 Acc=0.9490 Kappa=0.9336 2020-12-13 00:40:09 [INFO] [EVAL] Class IoU: [0.9693 0.78 0.9139 0.5055 0.489 0.6397 0.6921 0.7835 0.9117 0.5653 0.9467 0.8018 0.5872 0.933 0.3902 0.529 0.3344 0.357 0.7578] 2020-12-13 00:40:09 [INFO] [EVAL] Class Acc: [0.9764 0.9305 0.9539 0.6584 0.7781 0.8293 0.8205 0.8938 0.9506 0.6741 0.9615 0.8669 0.8394 0.9558 0.9306 0.9233 0.3535 0.8351 0.8477] 2020-12-13 00:40:09 [INFO] [EVAL] The model with the best validation mIoU (0.6837) was saved at iter 80000. 2020-12-13 00:41:56 [INFO] [TRAIN] epoch=496, iter=92200/160000, loss=0.8581, lr=0.009235, batch_cost=0.5349, reader_cost=0.0124 | ETA 10:04:24 2020-12-13 00:43:43 [INFO] [TRAIN] epoch=497, iter=92400/160000, loss=0.8620, lr=0.009210, batch_cost=0.5363, reader_cost=0.0137 | ETA 10:04:14 2020-12-13 00:45:31 [INFO] [TRAIN] epoch=498, iter=92600/160000, loss=0.8641, lr=0.009186, batch_cost=0.5357, reader_cost=0.0128 | ETA 10:01:44 2020-12-13 00:47:18 [INFO] [TRAIN] epoch=499, iter=92800/160000, loss=0.8595, lr=0.009161, batch_cost=0.5350, reader_cost=0.0124 | ETA 09:59:11 2020-12-13 00:49:04 [INFO] [TRAIN] epoch=500, iter=93000/160000, loss=0.8591, lr=0.009137, batch_cost=0.5347, reader_cost=0.0121 | ETA 09:57:03 2020-12-13 00:50:54 [INFO] [TRAIN] epoch=502, iter=93200/160000, loss=0.8718, lr=0.009112, batch_cost=0.5463, reader_cost=0.0220 | ETA 10:08:15 2020-12-13 00:52:41 [INFO] [TRAIN] epoch=503, iter=93400/160000, loss=0.8769, lr=0.009088, batch_cost=0.5362, reader_cost=0.0138 | ETA 09:55:09 2020-12-13 00:54:29 [INFO] [TRAIN] epoch=504, iter=93600/160000, loss=0.8708, lr=0.009063, batch_cost=0.5387, reader_cost=0.0138 | ETA 09:56:07 2020-12-13 00:56:16 [INFO] [TRAIN] epoch=505, iter=93800/160000, loss=0.8637, lr=0.009039, batch_cost=0.5356, reader_cost=0.0127 | ETA 09:50:53 2020-12-13 00:58:03 [INFO] [TRAIN] epoch=506, iter=94000/160000, loss=0.8595, lr=0.009014, batch_cost=0.5353, reader_cost=0.0126 | ETA 09:48:52 2020-12-13 00:59:50 [INFO] [TRAIN] epoch=507, iter=94200/160000, loss=0.8675, lr=0.008989, batch_cost=0.5369, reader_cost=0.0138 | ETA 09:48:46 2020-12-13 01:01:38 [INFO] [TRAIN] epoch=508, iter=94400/160000, loss=0.8877, lr=0.008965, batch_cost=0.5366, reader_cost=0.0128 | ETA 09:46:40 2020-12-13 01:03:25 [INFO] [TRAIN] epoch=509, iter=94600/160000, loss=0.8696, lr=0.008940, batch_cost=0.5389, reader_cost=0.0122 | ETA 09:47:22 2020-12-13 01:05:13 [INFO] [TRAIN] epoch=510, iter=94800/160000, loss=0.8638, lr=0.008916, batch_cost=0.5361, reader_cost=0.0116 | ETA 09:42:34 2020-12-13 01:07:00 [INFO] [TRAIN] epoch=511, iter=95000/160000, loss=0.8814, lr=0.008891, batch_cost=0.5391, reader_cost=0.0140 | ETA 09:44:00 2020-12-13 01:08:48 [INFO] [TRAIN] epoch=512, iter=95200/160000, loss=0.8685, lr=0.008866, batch_cost=0.5357, reader_cost=0.0133 | ETA 09:38:31 2020-12-13 01:10:35 [INFO] [TRAIN] epoch=513, iter=95400/160000, loss=0.8555, lr=0.008842, batch_cost=0.5355, reader_cost=0.0132 | ETA 09:36:34 2020-12-13 01:12:22 [INFO] [TRAIN] epoch=514, iter=95600/160000, loss=0.8730, lr=0.008817, batch_cost=0.5347, reader_cost=0.0123 | ETA 09:33:55 2020-12-13 01:14:10 [INFO] [TRAIN] epoch=516, iter=95800/160000, loss=0.8732, lr=0.008792, batch_cost=0.5442, reader_cost=0.0233 | ETA 09:42:18 2020-12-13 01:15:57 [INFO] [TRAIN] epoch=517, iter=96000/160000, loss=0.8733, lr=0.008768, batch_cost=0.5339, reader_cost=0.0124 | ETA 09:29:27 2020-12-13 01:15:57 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-13 01:16:28 [INFO] [EVAL] #Images=500 mIoU=0.7111 Acc=0.9524 Kappa=0.9383 2020-12-13 01:16:28 [INFO] [EVAL] Class IoU: [0.977 0.8257 0.9096 0.4596 0.5586 0.6205 0.6662 0.7637 0.9148 0.5993 0.9374 0.8082 0.6041 0.9328 0.6416 0.6986 0.413 0.4386 0.7421] 2020-12-13 01:16:28 [INFO] [EVAL] Class Acc: [0.9896 0.8889 0.9563 0.7847 0.8222 0.7265 0.7633 0.862 0.9506 0.7631 0.9498 0.8735 0.7799 0.9642 0.7976 0.9688 0.4361 0.7515 0.8636] 2020-12-13 01:16:29 [INFO] [EVAL] The model with the best validation mIoU (0.7111) was saved at iter 96000. 2020-12-13 01:18:16 [INFO] [TRAIN] epoch=518, iter=96200/160000, loss=0.8794, lr=0.008743, batch_cost=0.5350, reader_cost=0.0119 | ETA 09:28:55 2020-12-13 01:20:03 [INFO] [TRAIN] epoch=519, iter=96400/160000, loss=0.8722, lr=0.008718, batch_cost=0.5362, reader_cost=0.0134 | ETA 09:28:19 2020-12-13 01:21:50 [INFO] [TRAIN] epoch=520, iter=96600/160000, loss=0.8621, lr=0.008694, batch_cost=0.5361, reader_cost=0.0121 | ETA 09:26:26 2020-12-13 01:23:37 [INFO] [TRAIN] epoch=521, iter=96800/160000, loss=0.8741, lr=0.008669, batch_cost=0.5367, reader_cost=0.0145 | ETA 09:25:22 2020-12-13 01:25:25 [INFO] [TRAIN] epoch=522, iter=97000/160000, loss=0.8744, lr=0.008644, batch_cost=0.5360, reader_cost=0.0143 | ETA 09:22:49 2020-12-13 01:27:12 [INFO] [TRAIN] epoch=523, iter=97200/160000, loss=0.8612, lr=0.008620, batch_cost=0.5358, reader_cost=0.0132 | ETA 09:20:47 2020-12-13 01:28:58 [INFO] [TRAIN] epoch=524, iter=97400/160000, loss=0.8541, lr=0.008595, batch_cost=0.5335, reader_cost=0.0127 | ETA 09:16:40 2020-12-13 01:30:46 [INFO] [TRAIN] epoch=525, iter=97600/160000, loss=0.8571, lr=0.008570, batch_cost=0.5369, reader_cost=0.0143 | ETA 09:18:22 2020-12-13 01:32:33 [INFO] [TRAIN] epoch=526, iter=97800/160000, loss=0.8642, lr=0.008546, batch_cost=0.5349, reader_cost=0.0131 | ETA 09:14:30 2020-12-13 01:34:20 [INFO] [TRAIN] epoch=527, iter=98000/160000, loss=0.8554, lr=0.008521, batch_cost=0.5338, reader_cost=0.0120 | ETA 09:11:32 2020-12-13 01:36:07 [INFO] [TRAIN] epoch=528, iter=98200/160000, loss=0.8641, lr=0.008496, batch_cost=0.5350, reader_cost=0.0132 | ETA 09:11:05 2020-12-13 01:37:55 [INFO] [TRAIN] epoch=530, iter=98400/160000, loss=0.8703, lr=0.008471, batch_cost=0.5433, reader_cost=0.0240 | ETA 09:17:50 2020-12-13 01:39:42 [INFO] [TRAIN] epoch=531, iter=98600/160000, loss=0.8552, lr=0.008447, batch_cost=0.5336, reader_cost=0.0130 | ETA 09:06:01 2020-12-13 01:41:29 [INFO] [TRAIN] epoch=532, iter=98800/160000, loss=0.8739, lr=0.008422, batch_cost=0.5358, reader_cost=0.0124 | ETA 09:06:31 2020-12-13 01:43:16 [INFO] [TRAIN] epoch=533, iter=99000/160000, loss=0.8730, lr=0.008397, batch_cost=0.5363, reader_cost=0.0131 | ETA 09:05:14 2020-12-13 01:45:04 [INFO] [TRAIN] epoch=534, iter=99200/160000, loss=0.8799, lr=0.008372, batch_cost=0.5365, reader_cost=0.0149 | ETA 09:03:41 2020-12-13 01:46:51 [INFO] [TRAIN] epoch=535, iter=99400/160000, loss=0.8706, lr=0.008347, batch_cost=0.5372, reader_cost=0.0139 | ETA 09:02:31 2020-12-13 01:48:38 [INFO] [TRAIN] epoch=536, iter=99600/160000, loss=0.8717, lr=0.008323, batch_cost=0.5355, reader_cost=0.0118 | ETA 08:59:02 2020-12-13 01:50:26 [INFO] [TRAIN] epoch=537, iter=99800/160000, loss=0.8607, lr=0.008298, batch_cost=0.5371, reader_cost=0.0124 | ETA 08:58:51 2020-12-13 01:52:13 [INFO] [TRAIN] epoch=538, iter=100000/160000, loss=0.8628, lr=0.008273, batch_cost=0.5375, reader_cost=0.0124 | ETA 08:57:27 2020-12-13 01:52:13 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-13 01:52:43 [INFO] [EVAL] #Images=500 mIoU=0.6753 Acc=0.9450 Kappa=0.9285 2020-12-13 01:52:43 [INFO] [EVAL] Class IoU: [0.9728 0.7847 0.902 0.3104 0.4614 0.5243 0.6448 0.7581 0.9078 0.5731 0.9361 0.7877 0.5939 0.9259 0.5138 0.6792 0.3657 0.4514 0.7368] 2020-12-13 01:52:43 [INFO] [EVAL] Class Acc: [0.984 0.9002 0.9455 0.7154 0.6504 0.605 0.87 0.9278 0.9518 0.8053 0.9536 0.8821 0.8274 0.9628 0.6631 0.7391 0.7183 0.7312 0.7995] 2020-12-13 01:52:43 [INFO] [EVAL] The model with the best validation mIoU (0.7111) was saved at iter 96000. 2020-12-13 01:54:31 [INFO] [TRAIN] epoch=539, iter=100200/160000, loss=0.8549, lr=0.008248, batch_cost=0.5383, reader_cost=0.0124 | ETA 08:56:27 2020-12-13 01:56:18 [INFO] [TRAIN] epoch=540, iter=100400/160000, loss=0.8557, lr=0.008223, batch_cost=0.5371, reader_cost=0.0132 | ETA 08:53:30 2020-12-13 01:58:06 [INFO] [TRAIN] epoch=541, iter=100600/160000, loss=0.8716, lr=0.008199, batch_cost=0.5367, reader_cost=0.0125 | ETA 08:51:20 2020-12-13 01:59:53 [INFO] [TRAIN] epoch=542, iter=100800/160000, loss=0.8603, lr=0.008174, batch_cost=0.5373, reader_cost=0.0133 | ETA 08:50:06 2020-12-13 02:01:43 [INFO] [TRAIN] epoch=544, iter=101000/160000, loss=0.8624, lr=0.008149, batch_cost=0.5473, reader_cost=0.0238 | ETA 08:58:12 2020-12-13 02:03:30 [INFO] [TRAIN] epoch=545, iter=101200/160000, loss=0.8591, lr=0.008124, batch_cost=0.5352, reader_cost=0.0132 | ETA 08:44:32 2020-12-13 02:05:17 [INFO] [TRAIN] epoch=546, iter=101400/160000, loss=0.8666, lr=0.008099, batch_cost=0.5374, reader_cost=0.0136 | ETA 08:44:50 2020-12-13 02:07:05 [INFO] [TRAIN] epoch=547, iter=101600/160000, loss=0.8606, lr=0.008074, batch_cost=0.5375, reader_cost=0.0115 | ETA 08:43:08 2020-12-13 02:08:52 [INFO] [TRAIN] epoch=548, iter=101800/160000, loss=0.8491, lr=0.008049, batch_cost=0.5354, reader_cost=0.0138 | ETA 08:39:19 2020-12-13 02:10:39 [INFO] [TRAIN] epoch=549, iter=102000/160000, loss=0.8670, lr=0.008024, batch_cost=0.5374, reader_cost=0.0132 | ETA 08:39:26 2020-12-13 02:12:27 [INFO] [TRAIN] epoch=550, iter=102200/160000, loss=0.8666, lr=0.008000, batch_cost=0.5376, reader_cost=0.0147 | ETA 08:37:54 2020-12-13 02:14:14 [INFO] [TRAIN] epoch=551, iter=102400/160000, loss=0.8573, lr=0.007975, batch_cost=0.5357, reader_cost=0.0144 | ETA 08:34:17 2020-12-13 02:16:01 [INFO] [TRAIN] epoch=552, iter=102600/160000, loss=0.8725, lr=0.007950, batch_cost=0.5377, reader_cost=0.0155 | ETA 08:34:25 2020-12-13 02:17:48 [INFO] [TRAIN] epoch=553, iter=102800/160000, loss=0.8603, lr=0.007925, batch_cost=0.5352, reader_cost=0.0138 | ETA 08:30:13 2020-12-13 02:19:35 [INFO] [TRAIN] epoch=554, iter=103000/160000, loss=0.8706, lr=0.007900, batch_cost=0.5349, reader_cost=0.0117 | ETA 08:28:11 2020-12-13 02:21:22 [INFO] [TRAIN] epoch=555, iter=103200/160000, loss=0.8513, lr=0.007875, batch_cost=0.5347, reader_cost=0.0131 | ETA 08:26:11 2020-12-13 02:23:09 [INFO] [TRAIN] epoch=556, iter=103400/160000, loss=0.8522, lr=0.007850, batch_cost=0.5356, reader_cost=0.0134 | ETA 08:25:13 2020-12-13 02:24:56 [INFO] [TRAIN] epoch=557, iter=103600/160000, loss=0.8521, lr=0.007825, batch_cost=0.5343, reader_cost=0.0140 | ETA 08:22:13 2020-12-13 02:26:45 [INFO] [TRAIN] epoch=559, iter=103800/160000, loss=0.8522, lr=0.007800, batch_cost=0.5448, reader_cost=0.0232 | ETA 08:30:19 2020-12-13 02:28:32 [INFO] [TRAIN] epoch=560, iter=104000/160000, loss=0.8631, lr=0.007775, batch_cost=0.5350, reader_cost=0.0128 | ETA 08:19:21 2020-12-13 02:28:32 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-13 02:29:03 [INFO] [EVAL] #Images=500 mIoU=0.7054 Acc=0.9514 Kappa=0.9368 2020-12-13 02:29:03 [INFO] [EVAL] Class IoU: [0.9714 0.7903 0.9144 0.4125 0.5444 0.6426 0.6773 0.7788 0.9196 0.5424 0.9433 0.7932 0.5611 0.9357 0.5526 0.7225 0.622 0.3405 0.7384] 2020-12-13 02:29:03 [INFO] [EVAL] Class Acc: [0.9838 0.8784 0.9483 0.6803 0.7339 0.8423 0.7955 0.9285 0.9532 0.9078 0.9605 0.9006 0.6938 0.9667 0.7449 0.757 0.8491 0.8726 0.8063] 2020-12-13 02:29:03 [INFO] [EVAL] The model with the best validation mIoU (0.7111) was saved at iter 96000. 2020-12-13 02:30:50 [INFO] [TRAIN] epoch=561, iter=104200/160000, loss=0.8483, lr=0.007750, batch_cost=0.5327, reader_cost=0.0126 | ETA 08:15:25 2020-12-13 02:32:36 [INFO] [TRAIN] epoch=562, iter=104400/160000, loss=0.8412, lr=0.007725, batch_cost=0.5326, reader_cost=0.0112 | ETA 08:13:30 2020-12-13 02:34:23 [INFO] [TRAIN] epoch=563, iter=104600/160000, loss=0.8627, lr=0.007700, batch_cost=0.5328, reader_cost=0.0125 | ETA 08:11:58 2020-12-13 02:36:10 [INFO] [TRAIN] epoch=564, iter=104800/160000, loss=0.8796, lr=0.007675, batch_cost=0.5357, reader_cost=0.0137 | ETA 08:12:48 2020-12-13 02:37:57 [INFO] [TRAIN] epoch=565, iter=105000/160000, loss=0.8509, lr=0.007650, batch_cost=0.5363, reader_cost=0.0134 | ETA 08:11:38 2020-12-13 02:39:44 [INFO] [TRAIN] epoch=566, iter=105200/160000, loss=0.8571, lr=0.007625, batch_cost=0.5362, reader_cost=0.0145 | ETA 08:09:45 2020-12-13 02:41:31 [INFO] [TRAIN] epoch=567, iter=105400/160000, loss=0.8503, lr=0.007600, batch_cost=0.5347, reader_cost=0.0135 | ETA 08:06:35 2020-12-13 02:43:18 [INFO] [TRAIN] epoch=568, iter=105600/160000, loss=0.8597, lr=0.007575, batch_cost=0.5337, reader_cost=0.0125 | ETA 08:03:51 2020-12-13 02:45:05 [INFO] [TRAIN] epoch=569, iter=105800/160000, loss=0.8532, lr=0.007550, batch_cost=0.5366, reader_cost=0.0122 | ETA 08:04:41 2020-12-13 02:46:52 [INFO] [TRAIN] epoch=570, iter=106000/160000, loss=0.8476, lr=0.007525, batch_cost=0.5344, reader_cost=0.0130 | ETA 08:00:55 2020-12-13 02:48:39 [INFO] [TRAIN] epoch=571, iter=106200/160000, loss=0.8563, lr=0.007500, batch_cost=0.5344, reader_cost=0.0131 | ETA 07:59:13 2020-12-13 02:50:28 [INFO] [TRAIN] epoch=573, iter=106400/160000, loss=0.8516, lr=0.007474, batch_cost=0.5459, reader_cost=0.0211 | ETA 08:07:37 2020-12-13 02:52:15 [INFO] [TRAIN] epoch=574, iter=106600/160000, loss=0.8555, lr=0.007449, batch_cost=0.5344, reader_cost=0.0123 | ETA 07:55:34 2020-12-13 02:54:02 [INFO] [TRAIN] epoch=575, iter=106800/160000, loss=0.8443, lr=0.007424, batch_cost=0.5346, reader_cost=0.0131 | ETA 07:53:58 2020-12-13 02:55:49 [INFO] [TRAIN] epoch=576, iter=107000/160000, loss=0.8661, lr=0.007399, batch_cost=0.5346, reader_cost=0.0127 | ETA 07:52:11 2020-12-13 02:57:36 [INFO] [TRAIN] epoch=577, iter=107200/160000, loss=0.8636, lr=0.007374, batch_cost=0.5342, reader_cost=0.0116 | ETA 07:50:07 2020-12-13 02:59:23 [INFO] [TRAIN] epoch=578, iter=107400/160000, loss=0.8571, lr=0.007349, batch_cost=0.5378, reader_cost=0.0119 | ETA 07:51:28 2020-12-13 03:01:10 [INFO] [TRAIN] epoch=579, iter=107600/160000, loss=0.8614, lr=0.007324, batch_cost=0.5349, reader_cost=0.0133 | ETA 07:47:10 2020-12-13 03:02:58 [INFO] [TRAIN] epoch=580, iter=107800/160000, loss=0.8617, lr=0.007298, batch_cost=0.5372, reader_cost=0.0150 | ETA 07:47:22 2020-12-13 03:04:45 [INFO] [TRAIN] epoch=581, iter=108000/160000, loss=0.8648, lr=0.007273, batch_cost=0.5343, reader_cost=0.0121 | ETA 07:43:01 2020-12-13 03:04:45 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-13 03:05:15 [INFO] [EVAL] #Images=500 mIoU=0.6991 Acc=0.9509 Kappa=0.9361 2020-12-13 03:05:15 [INFO] [EVAL] Class IoU: [0.9726 0.8196 0.9094 0.2606 0.5469 0.6356 0.6533 0.7725 0.9167 0.5415 0.9277 0.7794 0.5243 0.9293 0.5302 0.7476 0.6515 0.4262 0.7388] 2020-12-13 03:05:15 [INFO] [EVAL] Class Acc: [0.982 0.9026 0.9418 0.8062 0.7296 0.8647 0.8633 0.8776 0.951 0.8632 0.9427 0.8727 0.8125 0.9545 0.7324 0.9265 0.8051 0.7926 0.8503] 2020-12-13 03:05:15 [INFO] [EVAL] The model with the best validation mIoU (0.7111) was saved at iter 96000. 2020-12-13 03:07:02 [INFO] [TRAIN] epoch=582, iter=108200/160000, loss=0.8660, lr=0.007248, batch_cost=0.5341, reader_cost=0.0140 | ETA 07:41:06 2020-12-13 03:08:49 [INFO] [TRAIN] epoch=583, iter=108400/160000, loss=0.8453, lr=0.007223, batch_cost=0.5353, reader_cost=0.0134 | ETA 07:40:22 2020-12-13 03:10:36 [INFO] [TRAIN] epoch=584, iter=108600/160000, loss=0.8409, lr=0.007198, batch_cost=0.5363, reader_cost=0.0143 | ETA 07:39:24 2020-12-13 03:12:23 [INFO] [TRAIN] epoch=585, iter=108800/160000, loss=0.8535, lr=0.007173, batch_cost=0.5350, reader_cost=0.0141 | ETA 07:36:32 2020-12-13 03:14:12 [INFO] [TRAIN] epoch=587, iter=109000/160000, loss=0.8605, lr=0.007147, batch_cost=0.5445, reader_cost=0.0225 | ETA 07:42:47 2020-12-13 03:15:59 [INFO] [TRAIN] epoch=588, iter=109200/160000, loss=0.8640, lr=0.007122, batch_cost=0.5350, reader_cost=0.0119 | ETA 07:32:56 2020-12-13 03:17:46 [INFO] [TRAIN] epoch=589, iter=109400/160000, loss=0.8638, lr=0.007097, batch_cost=0.5350, reader_cost=0.0138 | ETA 07:31:12 2020-12-13 03:19:33 [INFO] [TRAIN] epoch=590, iter=109600/160000, loss=0.8477, lr=0.007072, batch_cost=0.5308, reader_cost=0.0135 | ETA 07:25:53 2020-12-13 03:21:19 [INFO] [TRAIN] epoch=591, iter=109800/160000, loss=0.8511, lr=0.007046, batch_cost=0.5342, reader_cost=0.0145 | ETA 07:26:56 2020-12-13 03:23:06 [INFO] [TRAIN] epoch=592, iter=110000/160000, loss=0.8418, lr=0.007021, batch_cost=0.5329, reader_cost=0.0134 | ETA 07:24:04 2020-12-13 03:24:53 [INFO] [TRAIN] epoch=593, iter=110200/160000, loss=0.8574, lr=0.006996, batch_cost=0.5355, reader_cost=0.0138 | ETA 07:24:26 2020-12-13 03:26:40 [INFO] [TRAIN] epoch=594, iter=110400/160000, loss=0.8465, lr=0.006970, batch_cost=0.5346, reader_cost=0.0136 | ETA 07:21:53 2020-12-13 03:28:26 [INFO] [TRAIN] epoch=595, iter=110600/160000, loss=0.8445, lr=0.006945, batch_cost=0.5325, reader_cost=0.0129 | ETA 07:18:26 2020-12-13 03:30:14 [INFO] [TRAIN] epoch=596, iter=110800/160000, loss=0.8458, lr=0.006920, batch_cost=0.5353, reader_cost=0.0141 | ETA 07:18:57 2020-12-13 03:32:01 [INFO] [TRAIN] epoch=597, iter=111000/160000, loss=0.8503, lr=0.006895, batch_cost=0.5359, reader_cost=0.0139 | ETA 07:17:38 2020-12-13 03:33:48 [INFO] [TRAIN] epoch=598, iter=111200/160000, loss=0.8451, lr=0.006869, batch_cost=0.5354, reader_cost=0.0136 | ETA 07:15:25 2020-12-13 03:35:35 [INFO] [TRAIN] epoch=599, iter=111400/160000, loss=0.8483, lr=0.006844, batch_cost=0.5349, reader_cost=0.0136 | ETA 07:13:18 2020-12-13 03:37:22 [INFO] [TRAIN] epoch=600, iter=111600/160000, loss=0.8294, lr=0.006819, batch_cost=0.5354, reader_cost=0.0137 | ETA 07:11:51 2020-12-13 03:39:11 [INFO] [TRAIN] epoch=602, iter=111800/160000, loss=0.8533, lr=0.006793, batch_cost=0.5467, reader_cost=0.0228 | ETA 07:19:09 2020-12-13 03:40:58 [INFO] [TRAIN] epoch=603, iter=112000/160000, loss=0.8576, lr=0.006768, batch_cost=0.5356, reader_cost=0.0145 | ETA 07:08:29 2020-12-13 03:40:58 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-13 03:41:29 [INFO] [EVAL] #Images=500 mIoU=0.6527 Acc=0.9271 Kappa=0.9056 2020-12-13 03:41:29 [INFO] [EVAL] Class IoU: [0.9455 0.6535 0.887 0.2298 0.4165 0.5956 0.6666 0.7694 0.9004 0.5668 0.9455 0.5539 0.5146 0.8777 0.5868 0.6825 0.5481 0.381 0.6803] 2020-12-13 03:41:29 [INFO] [EVAL] Class Acc: [0.9795 0.9253 0.9182 0.3762 0.4951 0.7598 0.8136 0.8961 0.9628 0.7862 0.9617 0.5865 0.5861 0.9021 0.8376 0.8074 0.7465 0.47 0.8457] 2020-12-13 03:41:29 [INFO] [EVAL] The model with the best validation mIoU (0.7111) was saved at iter 96000. 2020-12-13 03:43:16 [INFO] [TRAIN] epoch=604, iter=112200/160000, loss=0.8391, lr=0.006742, batch_cost=0.5347, reader_cost=0.0136 | ETA 07:05:58 2020-12-13 03:45:03 [INFO] [TRAIN] epoch=605, iter=112400/160000, loss=0.8569, lr=0.006717, batch_cost=0.5352, reader_cost=0.0144 | ETA 07:04:35 2020-12-13 03:46:51 [INFO] [TRAIN] epoch=606, iter=112600/160000, loss=0.8561, lr=0.006692, batch_cost=0.5368, reader_cost=0.0142 | ETA 07:04:04 2020-12-13 03:48:37 [INFO] [TRAIN] epoch=607, iter=112800/160000, loss=0.8439, lr=0.006666, batch_cost=0.5338, reader_cost=0.0115 | ETA 06:59:56 2020-12-13 03:50:24 [INFO] [TRAIN] epoch=608, iter=113000/160000, loss=0.8462, lr=0.006641, batch_cost=0.5332, reader_cost=0.0126 | ETA 06:57:39 2020-12-13 03:52:11 [INFO] [TRAIN] epoch=609, iter=113200/160000, loss=0.8634, lr=0.006615, batch_cost=0.5329, reader_cost=0.0127 | ETA 06:55:41 2020-12-13 03:53:58 [INFO] [TRAIN] epoch=610, iter=113400/160000, loss=0.8621, lr=0.006590, batch_cost=0.5355, reader_cost=0.0148 | ETA 06:55:54 2020-12-13 03:55:45 [INFO] [TRAIN] epoch=611, iter=113600/160000, loss=0.8359, lr=0.006564, batch_cost=0.5387, reader_cost=0.0134 | ETA 06:56:33 2020-12-13 03:57:32 [INFO] [TRAIN] epoch=612, iter=113800/160000, loss=0.8512, lr=0.006539, batch_cost=0.5345, reader_cost=0.0139 | ETA 06:51:34 2020-12-13 03:59:19 [INFO] [TRAIN] epoch=613, iter=114000/160000, loss=0.8312, lr=0.006513, batch_cost=0.5341, reader_cost=0.0134 | ETA 06:49:29 2020-12-13 04:01:06 [INFO] [TRAIN] epoch=614, iter=114200/160000, loss=0.8512, lr=0.006488, batch_cost=0.5329, reader_cost=0.0123 | ETA 06:46:48 2020-12-13 04:02:54 [INFO] [TRAIN] epoch=616, iter=114400/160000, loss=0.8472, lr=0.006462, batch_cost=0.5437, reader_cost=0.0222 | ETA 06:53:10 2020-12-13 04:04:41 [INFO] [TRAIN] epoch=617, iter=114600/160000, loss=0.8455, lr=0.006437, batch_cost=0.5342, reader_cost=0.0124 | ETA 06:44:14 2020-12-13 04:06:28 [INFO] [TRAIN] epoch=618, iter=114800/160000, loss=0.8386, lr=0.006411, batch_cost=0.5321, reader_cost=0.0116 | ETA 06:40:52 2020-12-13 04:08:14 [INFO] [TRAIN] epoch=619, iter=115000/160000, loss=0.8339, lr=0.006386, batch_cost=0.5325, reader_cost=0.0130 | ETA 06:39:24 2020-12-13 04:10:01 [INFO] [TRAIN] epoch=620, iter=115200/160000, loss=0.8352, lr=0.006360, batch_cost=0.5348, reader_cost=0.0122 | ETA 06:39:16 2020-12-13 04:11:48 [INFO] [TRAIN] epoch=621, iter=115400/160000, loss=0.8457, lr=0.006335, batch_cost=0.5346, reader_cost=0.0143 | ETA 06:37:21 2020-12-13 04:13:35 [INFO] [TRAIN] epoch=622, iter=115600/160000, loss=0.8405, lr=0.006309, batch_cost=0.5341, reader_cost=0.0134 | ETA 06:35:14 2020-12-13 04:15:22 [INFO] [TRAIN] epoch=623, iter=115800/160000, loss=0.8592, lr=0.006284, batch_cost=0.5331, reader_cost=0.0140 | ETA 06:32:44 2020-12-13 04:17:08 [INFO] [TRAIN] epoch=624, iter=116000/160000, loss=0.8354, lr=0.006258, batch_cost=0.5330, reader_cost=0.0126 | ETA 06:30:50 2020-12-13 04:17:08 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-13 04:17:39 [INFO] [EVAL] #Images=500 mIoU=0.6835 Acc=0.9477 Kappa=0.9321 2020-12-13 04:17:39 [INFO] [EVAL] Class IoU: [0.9743 0.8124 0.9017 0.3467 0.4838 0.6218 0.6643 0.7549 0.9049 0.5968 0.944 0.7305 0.585 0.9246 0.3976 0.609 0.5647 0.4265 0.743 ] 2020-12-13 04:17:39 [INFO] [EVAL] Class Acc: [0.9897 0.8812 0.9351 0.7618 0.7862 0.8133 0.8157 0.8459 0.9472 0.7945 0.9666 0.7939 0.8065 0.9509 0.6293 0.8703 0.9487 0.5808 0.8598] 2020-12-13 04:17:39 [INFO] [EVAL] The model with the best validation mIoU (0.7111) was saved at iter 96000. 2020-12-13 04:19:26 [INFO] [TRAIN] epoch=625, iter=116200/160000, loss=0.8510, lr=0.006232, batch_cost=0.5330, reader_cost=0.0136 | ETA 06:29:07 2020-12-13 04:21:13 [INFO] [TRAIN] epoch=626, iter=116400/160000, loss=0.8432, lr=0.006207, batch_cost=0.5348, reader_cost=0.0131 | ETA 06:28:35 2020-12-13 04:23:00 [INFO] [TRAIN] epoch=627, iter=116600/160000, loss=0.8534, lr=0.006181, batch_cost=0.5349, reader_cost=0.0130 | ETA 06:26:55 2020-12-13 04:24:46 [INFO] [TRAIN] epoch=628, iter=116800/160000, loss=0.8390, lr=0.006156, batch_cost=0.5326, reader_cost=0.0132 | ETA 06:23:27 2020-12-13 04:26:35 [INFO] [TRAIN] epoch=630, iter=117000/160000, loss=0.8457, lr=0.006130, batch_cost=0.5450, reader_cost=0.0218 | ETA 06:30:35 2020-12-13 04:28:23 [INFO] [TRAIN] epoch=631, iter=117200/160000, loss=0.8455, lr=0.006104, batch_cost=0.5359, reader_cost=0.0127 | ETA 06:22:18 2020-12-13 04:30:09 [INFO] [TRAIN] epoch=632, iter=117400/160000, loss=0.8418, lr=0.006079, batch_cost=0.5340, reader_cost=0.0123 | ETA 06:19:10 2020-12-13 04:31:56 [INFO] [TRAIN] epoch=633, iter=117600/160000, loss=0.8357, lr=0.006053, batch_cost=0.5327, reader_cost=0.0138 | ETA 06:16:26 2020-12-13 04:33:43 [INFO] [TRAIN] epoch=634, iter=117800/160000, loss=0.8475, lr=0.006027, batch_cost=0.5343, reader_cost=0.0130 | ETA 06:15:47 2020-12-13 04:35:30 [INFO] [TRAIN] epoch=635, iter=118000/160000, loss=0.8526, lr=0.006001, batch_cost=0.5355, reader_cost=0.0146 | ETA 06:14:52 2020-12-13 04:37:17 [INFO] [TRAIN] epoch=636, iter=118200/160000, loss=0.8434, lr=0.005976, batch_cost=0.5350, reader_cost=0.0154 | ETA 06:12:44 2020-12-13 04:39:04 [INFO] [TRAIN] epoch=637, iter=118400/160000, loss=0.8502, lr=0.005950, batch_cost=0.5338, reader_cost=0.0138 | ETA 06:10:04 2020-12-13 04:40:51 [INFO] [TRAIN] epoch=638, iter=118600/160000, loss=0.8309, lr=0.005924, batch_cost=0.5355, reader_cost=0.0140 | ETA 06:09:31 2020-12-13 04:42:37 [INFO] [TRAIN] epoch=639, iter=118800/160000, loss=0.8479, lr=0.005898, batch_cost=0.5332, reader_cost=0.0126 | ETA 06:06:07 2020-12-13 04:44:24 [INFO] [TRAIN] epoch=640, iter=119000/160000, loss=0.8473, lr=0.005873, batch_cost=0.5352, reader_cost=0.0134 | ETA 06:05:43 2020-12-13 04:46:12 [INFO] [TRAIN] epoch=641, iter=119200/160000, loss=0.8400, lr=0.005847, batch_cost=0.5363, reader_cost=0.0125 | ETA 06:04:40 2020-12-13 04:47:58 [INFO] [TRAIN] epoch=642, iter=119400/160000, loss=0.8364, lr=0.005821, batch_cost=0.5327, reader_cost=0.0120 | ETA 06:00:27 2020-12-13 04:49:48 [INFO] [TRAIN] epoch=644, iter=119600/160000, loss=0.8241, lr=0.005795, batch_cost=0.5464, reader_cost=0.0221 | ETA 06:07:52 2020-12-13 04:51:35 [INFO] [TRAIN] epoch=645, iter=119800/160000, loss=0.8491, lr=0.005769, batch_cost=0.5349, reader_cost=0.0137 | ETA 05:58:21 2020-12-13 04:53:21 [INFO] [TRAIN] epoch=646, iter=120000/160000, loss=0.8501, lr=0.005744, batch_cost=0.5343, reader_cost=0.0139 | ETA 05:56:11 2020-12-13 04:53:21 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-13 04:53:51 [INFO] [EVAL] #Images=500 mIoU=0.7223 Acc=0.9563 Kappa=0.9432 2020-12-13 04:53:51 [INFO] [EVAL] Class IoU: [0.9798 0.8354 0.9228 0.4093 0.5898 0.657 0.7125 0.7702 0.9204 0.6371 0.9478 0.7338 0.4888 0.9421 0.6623 0.8144 0.6762 0.2993 0.7258] 2020-12-13 04:53:51 [INFO] [EVAL] Class Acc: [0.9892 0.9168 0.9535 0.8278 0.7689 0.8288 0.8273 0.9204 0.9517 0.8203 0.969 0.902 0.5893 0.9621 0.8161 0.94 0.8433 0.3167 0.8259] 2020-12-13 04:53:51 [INFO] [EVAL] The model with the best validation mIoU (0.7223) was saved at iter 120000. 2020-12-13 04:55:39 [INFO] [TRAIN] epoch=647, iter=120200/160000, loss=0.8453, lr=0.005718, batch_cost=0.5355, reader_cost=0.0147 | ETA 05:55:14 2020-12-13 04:57:26 [INFO] [TRAIN] epoch=648, iter=120400/160000, loss=0.8379, lr=0.005692, batch_cost=0.5354, reader_cost=0.0138 | ETA 05:53:21 2020-12-13 04:59:12 [INFO] [TRAIN] epoch=649, iter=120600/160000, loss=0.8392, lr=0.005666, batch_cost=0.5333, reader_cost=0.0116 | ETA 05:50:11 2020-12-13 05:00:59 [INFO] [TRAIN] epoch=650, iter=120800/160000, loss=0.8412, lr=0.005640, batch_cost=0.5338, reader_cost=0.0139 | ETA 05:48:44 2020-12-13 05:02:46 [INFO] [TRAIN] epoch=651, iter=121000/160000, loss=0.8420, lr=0.005614, batch_cost=0.5344, reader_cost=0.0139 | ETA 05:47:20 2020-12-13 05:04:33 [INFO] [TRAIN] epoch=652, iter=121200/160000, loss=0.8418, lr=0.005588, batch_cost=0.5349, reader_cost=0.0134 | ETA 05:45:53 2020-12-13 05:06:20 [INFO] [TRAIN] epoch=653, iter=121400/160000, loss=0.8440, lr=0.005562, batch_cost=0.5337, reader_cost=0.0135 | ETA 05:43:20 2020-12-13 05:08:07 [INFO] [TRAIN] epoch=654, iter=121600/160000, loss=0.8410, lr=0.005536, batch_cost=0.5350, reader_cost=0.0138 | ETA 05:42:23 2020-12-13 05:09:53 [INFO] [TRAIN] epoch=655, iter=121800/160000, loss=0.8169, lr=0.005510, batch_cost=0.5308, reader_cost=0.0113 | ETA 05:37:58 2020-12-13 05:11:39 [INFO] [TRAIN] epoch=656, iter=122000/160000, loss=0.8332, lr=0.005485, batch_cost=0.5333, reader_cost=0.0136 | ETA 05:37:43 2020-12-13 05:13:26 [INFO] [TRAIN] epoch=657, iter=122200/160000, loss=0.8301, lr=0.005459, batch_cost=0.5319, reader_cost=0.0143 | ETA 05:35:05 2020-12-13 05:15:14 [INFO] [TRAIN] epoch=659, iter=122400/160000, loss=0.8398, lr=0.005433, batch_cost=0.5427, reader_cost=0.0234 | ETA 05:40:04 2020-12-13 05:17:02 [INFO] [TRAIN] epoch=660, iter=122600/160000, loss=0.8144, lr=0.005407, batch_cost=0.5355, reader_cost=0.0122 | ETA 05:33:48 2020-12-13 05:18:48 [INFO] [TRAIN] epoch=661, iter=122800/160000, loss=0.8299, lr=0.005380, batch_cost=0.5345, reader_cost=0.0138 | ETA 05:31:21 2020-12-13 05:20:35 [INFO] [TRAIN] epoch=662, iter=123000/160000, loss=0.8362, lr=0.005354, batch_cost=0.5345, reader_cost=0.0141 | ETA 05:29:35 2020-12-13 05:22:23 [INFO] [TRAIN] epoch=663, iter=123200/160000, loss=0.8477, lr=0.005328, batch_cost=0.5381, reader_cost=0.0136 | ETA 05:30:01 2020-12-13 05:24:10 [INFO] [TRAIN] epoch=664, iter=123400/160000, loss=0.8274, lr=0.005302, batch_cost=0.5374, reader_cost=0.0145 | ETA 05:27:49 2020-12-13 05:25:57 [INFO] [TRAIN] epoch=665, iter=123600/160000, loss=0.8465, lr=0.005276, batch_cost=0.5343, reader_cost=0.0127 | ETA 05:24:08 2020-12-13 05:27:44 [INFO] [TRAIN] epoch=666, iter=123800/160000, loss=0.8266, lr=0.005250, batch_cost=0.5323, reader_cost=0.0123 | ETA 05:21:10 2020-12-13 05:29:31 [INFO] [TRAIN] epoch=667, iter=124000/160000, loss=0.8392, lr=0.005224, batch_cost=0.5344, reader_cost=0.0147 | ETA 05:20:38 2020-12-13 05:29:31 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-13 05:30:00 [INFO] [EVAL] #Images=500 mIoU=0.6155 Acc=0.9220 Kappa=0.8987 2020-12-13 05:30:00 [INFO] [EVAL] Class IoU: [0.9635 0.7828 0.8403 0.2663 0.4472 0.5468 0.5688 0.7241 0.8223 0.5251 0.9397 0.605 0.5028 0.8672 0.3311 0.5781 0.4395 0.2694 0.6752] 2020-12-13 05:30:00 [INFO] [EVAL] Class Acc: [0.9874 0.8738 0.8677 0.6236 0.8166 0.8536 0.7817 0.9136 0.9407 0.7393 0.9601 0.667 0.6383 0.8995 0.5019 0.8086 0.862 0.4355 0.8862] 2020-12-13 05:30:01 [INFO] [EVAL] The model with the best validation mIoU (0.7223) was saved at iter 120000. 2020-12-13 05:31:47 [INFO] [TRAIN] epoch=668, iter=124200/160000, loss=0.8310, lr=0.005198, batch_cost=0.5338, reader_cost=0.0134 | ETA 05:18:31 2020-12-13 05:33:34 [INFO] [TRAIN] epoch=669, iter=124400/160000, loss=0.8326, lr=0.005172, batch_cost=0.5329, reader_cost=0.0128 | ETA 05:16:12 2020-12-13 05:35:20 [INFO] [TRAIN] epoch=670, iter=124600/160000, loss=0.8465, lr=0.005146, batch_cost=0.5328, reader_cost=0.0122 | ETA 05:14:20 2020-12-13 05:37:07 [INFO] [TRAIN] epoch=671, iter=124800/160000, loss=0.8271, lr=0.005119, batch_cost=0.5339, reader_cost=0.0134 | ETA 05:13:12 2020-12-13 05:38:56 [INFO] [TRAIN] epoch=673, iter=125000/160000, loss=0.8268, lr=0.005093, batch_cost=0.5430, reader_cost=0.0208 | ETA 05:16:46 2020-12-13 05:40:42 [INFO] [TRAIN] epoch=674, iter=125200/160000, loss=0.8383, lr=0.005067, batch_cost=0.5326, reader_cost=0.0108 | ETA 05:08:53 2020-12-13 05:42:29 [INFO] [TRAIN] epoch=675, iter=125400/160000, loss=0.8269, lr=0.005041, batch_cost=0.5343, reader_cost=0.0136 | ETA 05:08:06 2020-12-13 05:44:16 [INFO] [TRAIN] epoch=676, iter=125600/160000, loss=0.8268, lr=0.005015, batch_cost=0.5341, reader_cost=0.0141 | ETA 05:06:11 2020-12-13 05:46:03 [INFO] [TRAIN] epoch=677, iter=125800/160000, loss=0.8362, lr=0.004988, batch_cost=0.5345, reader_cost=0.0130 | ETA 05:04:40 2020-12-13 05:47:49 [INFO] [TRAIN] epoch=678, iter=126000/160000, loss=0.8295, lr=0.004962, batch_cost=0.5309, reader_cost=0.0129 | ETA 05:00:50 2020-12-13 05:49:35 [INFO] [TRAIN] epoch=679, iter=126200/160000, loss=0.8261, lr=0.004936, batch_cost=0.5316, reader_cost=0.0135 | ETA 04:59:26 2020-12-13 05:51:22 [INFO] [TRAIN] epoch=680, iter=126400/160000, loss=0.8290, lr=0.004910, batch_cost=0.5338, reader_cost=0.0136 | ETA 04:58:56 2020-12-13 05:53:09 [INFO] [TRAIN] epoch=681, iter=126600/160000, loss=0.8258, lr=0.004883, batch_cost=0.5330, reader_cost=0.0133 | ETA 04:56:42 2020-12-13 05:54:56 [INFO] [TRAIN] epoch=682, iter=126800/160000, loss=0.8414, lr=0.004857, batch_cost=0.5337, reader_cost=0.0128 | ETA 04:55:17 2020-12-13 05:56:42 [INFO] [TRAIN] epoch=683, iter=127000/160000, loss=0.8345, lr=0.004831, batch_cost=0.5321, reader_cost=0.0138 | ETA 04:52:39 2020-12-13 05:58:28 [INFO] [TRAIN] epoch=684, iter=127200/160000, loss=0.8229, lr=0.004804, batch_cost=0.5321, reader_cost=0.0135 | ETA 04:50:51 2020-12-13 06:00:15 [INFO] [TRAIN] epoch=685, iter=127400/160000, loss=0.8325, lr=0.004778, batch_cost=0.5319, reader_cost=0.0137 | ETA 04:49:00 2020-12-13 06:02:03 [INFO] [TRAIN] epoch=687, iter=127600/160000, loss=0.8414, lr=0.004751, batch_cost=0.5419, reader_cost=0.0238 | ETA 04:52:37 2020-12-13 06:03:50 [INFO] [TRAIN] epoch=688, iter=127800/160000, loss=0.8230, lr=0.004725, batch_cost=0.5321, reader_cost=0.0123 | ETA 04:45:34 2020-12-13 06:05:36 [INFO] [TRAIN] epoch=689, iter=128000/160000, loss=0.8277, lr=0.004699, batch_cost=0.5334, reader_cost=0.0124 | ETA 04:44:28 2020-12-13 06:05:36 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-13 06:06:07 [INFO] [EVAL] #Images=500 mIoU=0.7599 Acc=0.9576 Kappa=0.9448 2020-12-13 06:06:07 [INFO] [EVAL] Class IoU: [0.978 0.8313 0.9204 0.4335 0.5898 0.6521 0.7093 0.8013 0.9201 0.6181 0.9479 0.8142 0.6102 0.9472 0.7302 0.8497 0.7457 0.5745 0.7653] 2020-12-13 06:06:07 [INFO] [EVAL] Class Acc: [0.9857 0.9349 0.9448 0.7455 0.7635 0.8496 0.8637 0.9225 0.9598 0.7682 0.9638 0.8951 0.8091 0.9743 0.8584 0.9361 0.8891 0.6917 0.8667] 2020-12-13 06:06:07 [INFO] [EVAL] The model with the best validation mIoU (0.7599) was saved at iter 128000. 2020-12-13 06:07:54 [INFO] [TRAIN] epoch=690, iter=128200/160000, loss=0.8268, lr=0.004672, batch_cost=0.5332, reader_cost=0.0147 | ETA 04:42:34 2020-12-13 06:09:41 [INFO] [TRAIN] epoch=691, iter=128400/160000, loss=0.8319, lr=0.004646, batch_cost=0.5323, reader_cost=0.0124 | ETA 04:40:20 2020-12-13 06:11:27 [INFO] [TRAIN] epoch=692, iter=128600/160000, loss=0.8364, lr=0.004619, batch_cost=0.5310, reader_cost=0.0113 | ETA 04:37:54 2020-12-13 06:13:13 [INFO] [TRAIN] epoch=693, iter=128800/160000, loss=0.8271, lr=0.004593, batch_cost=0.5335, reader_cost=0.0142 | ETA 04:37:25 2020-12-13 06:15:00 [INFO] [TRAIN] epoch=694, iter=129000/160000, loss=0.8195, lr=0.004566, batch_cost=0.5313, reader_cost=0.0117 | ETA 04:34:31 2020-12-13 06:16:46 [INFO] [TRAIN] epoch=695, iter=129200/160000, loss=0.8290, lr=0.004540, batch_cost=0.5336, reader_cost=0.0146 | ETA 04:33:56 2020-12-13 06:18:33 [INFO] [TRAIN] epoch=696, iter=129400/160000, loss=0.8270, lr=0.004513, batch_cost=0.5325, reader_cost=0.0131 | ETA 04:31:35 2020-12-13 06:20:20 [INFO] [TRAIN] epoch=697, iter=129600/160000, loss=0.8180, lr=0.004487, batch_cost=0.5325, reader_cost=0.0125 | ETA 04:29:49 2020-12-13 06:22:06 [INFO] [TRAIN] epoch=698, iter=129800/160000, loss=0.8138, lr=0.004460, batch_cost=0.5301, reader_cost=0.0122 | ETA 04:26:50 2020-12-13 06:23:52 [INFO] [TRAIN] epoch=699, iter=130000/160000, loss=0.8296, lr=0.004433, batch_cost=0.5328, reader_cost=0.0137 | ETA 04:26:23 2020-12-13 06:25:38 [INFO] [TRAIN] epoch=700, iter=130200/160000, loss=0.8404, lr=0.004407, batch_cost=0.5317, reader_cost=0.0141 | ETA 04:24:03 2020-12-13 06:27:27 [INFO] [TRAIN] epoch=702, iter=130400/160000, loss=0.8317, lr=0.004380, batch_cost=0.5423, reader_cost=0.0224 | ETA 04:27:30 2020-12-13 06:29:13 [INFO] [TRAIN] epoch=703, iter=130600/160000, loss=0.8245, lr=0.004354, batch_cost=0.5299, reader_cost=0.0121 | ETA 04:19:38 2020-12-13 06:30:59 [INFO] [TRAIN] epoch=704, iter=130800/160000, loss=0.8217, lr=0.004327, batch_cost=0.5303, reader_cost=0.0127 | ETA 04:18:04 2020-12-13 06:32:45 [INFO] [TRAIN] epoch=705, iter=131000/160000, loss=0.8212, lr=0.004300, batch_cost=0.5309, reader_cost=0.0128 | ETA 04:16:37 2020-12-13 06:34:31 [INFO] [TRAIN] epoch=706, iter=131200/160000, loss=0.8155, lr=0.004274, batch_cost=0.5313, reader_cost=0.0129 | ETA 04:15:02 2020-12-13 06:36:18 [INFO] [TRAIN] epoch=707, iter=131400/160000, loss=0.8248, lr=0.004247, batch_cost=0.5326, reader_cost=0.0150 | ETA 04:13:51 2020-12-13 06:38:04 [INFO] [TRAIN] epoch=708, iter=131600/160000, loss=0.8165, lr=0.004220, batch_cost=0.5316, reader_cost=0.0132 | ETA 04:11:37 2020-12-13 06:39:51 [INFO] [TRAIN] epoch=709, iter=131800/160000, loss=0.8190, lr=0.004193, batch_cost=0.5329, reader_cost=0.0122 | ETA 04:10:28 2020-12-13 06:41:37 [INFO] [TRAIN] epoch=710, iter=132000/160000, loss=0.8293, lr=0.004167, batch_cost=0.5326, reader_cost=0.0128 | ETA 04:08:33 2020-12-13 06:41:37 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-13 06:42:07 [INFO] [EVAL] #Images=500 mIoU=0.7354 Acc=0.9579 Kappa=0.9453 2020-12-13 06:42:07 [INFO] [EVAL] Class IoU: [0.9792 0.8445 0.9202 0.4486 0.5306 0.6643 0.7138 0.7957 0.9232 0.617 0.9438 0.8168 0.6328 0.9488 0.6586 0.7578 0.3852 0.622 0.7701] 2020-12-13 06:42:07 [INFO] [EVAL] Class Acc: [0.9858 0.9293 0.9473 0.68 0.8354 0.823 0.8767 0.9117 0.9616 0.8963 0.9571 0.8753 0.7986 0.9721 0.9097 0.8141 0.9043 0.7794 0.8541] 2020-12-13 06:42:07 [INFO] [EVAL] The model with the best validation mIoU (0.7599) was saved at iter 128000. 2020-12-13 06:43:54 [INFO] [TRAIN] epoch=711, iter=132200/160000, loss=0.8302, lr=0.004140, batch_cost=0.5332, reader_cost=0.0143 | ETA 04:07:02 2020-12-13 06:45:41 [INFO] [TRAIN] epoch=712, iter=132400/160000, loss=0.8240, lr=0.004113, batch_cost=0.5336, reader_cost=0.0125 | ETA 04:05:27 2020-12-13 06:47:27 [INFO] [TRAIN] epoch=713, iter=132600/160000, loss=0.8212, lr=0.004086, batch_cost=0.5306, reader_cost=0.0140 | ETA 04:02:19 2020-12-13 06:49:13 [INFO] [TRAIN] epoch=714, iter=132800/160000, loss=0.8156, lr=0.004059, batch_cost=0.5307, reader_cost=0.0131 | ETA 04:00:34 2020-12-13 06:51:01 [INFO] [TRAIN] epoch=716, iter=133000/160000, loss=0.8189, lr=0.004032, batch_cost=0.5416, reader_cost=0.0232 | ETA 04:03:43 2020-12-13 06:52:48 [INFO] [TRAIN] epoch=717, iter=133200/160000, loss=0.8106, lr=0.004006, batch_cost=0.5341, reader_cost=0.0139 | ETA 03:58:34 2020-12-13 06:54:35 [INFO] [TRAIN] epoch=718, iter=133400/160000, loss=0.8188, lr=0.003979, batch_cost=0.5324, reader_cost=0.0139 | ETA 03:56:00 2020-12-13 06:56:21 [INFO] [TRAIN] epoch=719, iter=133600/160000, loss=0.8098, lr=0.003952, batch_cost=0.5309, reader_cost=0.0124 | ETA 03:53:36 2020-12-13 06:58:07 [INFO] [TRAIN] epoch=720, iter=133800/160000, loss=0.8115, lr=0.003925, batch_cost=0.5311, reader_cost=0.0117 | ETA 03:51:55 2020-12-13 06:59:53 [INFO] [TRAIN] epoch=721, iter=134000/160000, loss=0.8214, lr=0.003898, batch_cost=0.5309, reader_cost=0.0125 | ETA 03:50:04 2020-12-13 07:01:39 [INFO] [TRAIN] epoch=722, iter=134200/160000, loss=0.8315, lr=0.003871, batch_cost=0.5307, reader_cost=0.0130 | ETA 03:48:12 2020-12-13 07:03:26 [INFO] [TRAIN] epoch=723, iter=134400/160000, loss=0.8090, lr=0.003844, batch_cost=0.5326, reader_cost=0.0135 | ETA 03:47:14 2020-12-13 07:05:12 [INFO] [TRAIN] epoch=724, iter=134600/160000, loss=0.8192, lr=0.003817, batch_cost=0.5316, reader_cost=0.0140 | ETA 03:45:02 2020-12-13 07:06:59 [INFO] [TRAIN] epoch=725, iter=134800/160000, loss=0.8192, lr=0.003790, batch_cost=0.5319, reader_cost=0.0143 | ETA 03:43:24 2020-12-13 07:08:45 [INFO] [TRAIN] epoch=726, iter=135000/160000, loss=0.8133, lr=0.003763, batch_cost=0.5330, reader_cost=0.0139 | ETA 03:42:04 2020-12-13 07:10:32 [INFO] [TRAIN] epoch=727, iter=135200/160000, loss=0.8098, lr=0.003735, batch_cost=0.5339, reader_cost=0.0132 | ETA 03:40:40 2020-12-13 07:12:19 [INFO] [TRAIN] epoch=728, iter=135400/160000, loss=0.8172, lr=0.003708, batch_cost=0.5326, reader_cost=0.0137 | ETA 03:38:22 2020-12-13 07:14:07 [INFO] [TRAIN] epoch=730, iter=135600/160000, loss=0.8181, lr=0.003681, batch_cost=0.5431, reader_cost=0.0228 | ETA 03:40:50 2020-12-13 07:15:54 [INFO] [TRAIN] epoch=731, iter=135800/160000, loss=0.8180, lr=0.003654, batch_cost=0.5341, reader_cost=0.0124 | ETA 03:35:25 2020-12-13 07:17:40 [INFO] [TRAIN] epoch=732, iter=136000/160000, loss=0.8026, lr=0.003627, batch_cost=0.5318, reader_cost=0.0130 | ETA 03:32:42 2020-12-13 07:17:40 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-13 07:18:10 [INFO] [EVAL] #Images=500 mIoU=0.7707 Acc=0.9589 Kappa=0.9466 2020-12-13 07:18:10 [INFO] [EVAL] Class IoU: [0.9773 0.8267 0.9245 0.5278 0.5543 0.6619 0.7087 0.7981 0.9251 0.6384 0.9502 0.8272 0.6371 0.9472 0.7057 0.8761 0.7551 0.6295 0.7733] 2020-12-13 07:18:10 [INFO] [EVAL] Class Acc: [0.9883 0.9064 0.9525 0.7381 0.758 0.8568 0.8792 0.9349 0.9563 0.7988 0.9719 0.9115 0.8237 0.9694 0.9315 0.9421 0.8809 0.8135 0.8808] 2020-12-13 07:18:11 [INFO] [EVAL] The model with the best validation mIoU (0.7707) was saved at iter 136000. 2020-12-13 07:19:58 [INFO] [TRAIN] epoch=733, iter=136200/160000, loss=0.8183, lr=0.003600, batch_cost=0.5338, reader_cost=0.0131 | ETA 03:31:45 2020-12-13 07:21:44 [INFO] [TRAIN] epoch=734, iter=136400/160000, loss=0.8129, lr=0.003572, batch_cost=0.5298, reader_cost=0.0121 | ETA 03:28:22 2020-12-13 07:23:30 [INFO] [TRAIN] epoch=735, iter=136600/160000, loss=0.8169, lr=0.003545, batch_cost=0.5303, reader_cost=0.0142 | ETA 03:26:50 2020-12-13 07:25:16 [INFO] [TRAIN] epoch=736, iter=136800/160000, loss=0.8152, lr=0.003518, batch_cost=0.5309, reader_cost=0.0128 | ETA 03:25:16 2020-12-13 07:27:02 [INFO] [TRAIN] epoch=737, iter=137000/160000, loss=0.8133, lr=0.003491, batch_cost=0.5310, reader_cost=0.0130 | ETA 03:23:33 2020-12-13 07:28:48 [INFO] [TRAIN] epoch=738, iter=137200/160000, loss=0.8222, lr=0.003463, batch_cost=0.5294, reader_cost=0.0106 | ETA 03:21:10 2020-12-13 07:30:34 [INFO] [TRAIN] epoch=739, iter=137400/160000, loss=0.8092, lr=0.003436, batch_cost=0.5314, reader_cost=0.0125 | ETA 03:20:08 2020-12-13 07:32:21 [INFO] [TRAIN] epoch=740, iter=137600/160000, loss=0.8087, lr=0.003408, batch_cost=0.5314, reader_cost=0.0140 | ETA 03:18:24 2020-12-13 07:34:07 [INFO] [TRAIN] epoch=741, iter=137800/160000, loss=0.8052, lr=0.003381, batch_cost=0.5316, reader_cost=0.0146 | ETA 03:16:41 2020-12-13 07:35:53 [INFO] [TRAIN] epoch=742, iter=138000/160000, loss=0.8276, lr=0.003354, batch_cost=0.5320, reader_cost=0.0143 | ETA 03:15:03 2020-12-13 07:37:42 [INFO] [TRAIN] epoch=744, iter=138200/160000, loss=0.8092, lr=0.003326, batch_cost=0.5414, reader_cost=0.0207 | ETA 03:16:42 2020-12-13 07:39:28 [INFO] [TRAIN] epoch=745, iter=138400/160000, loss=0.7997, lr=0.003299, batch_cost=0.5312, reader_cost=0.0136 | ETA 03:11:14 2020-12-13 07:41:15 [INFO] [TRAIN] epoch=746, iter=138600/160000, loss=0.8116, lr=0.003271, batch_cost=0.5327, reader_cost=0.0130 | ETA 03:09:58 2020-12-13 07:43:01 [INFO] [TRAIN] epoch=747, iter=138800/160000, loss=0.8054, lr=0.003244, batch_cost=0.5302, reader_cost=0.0126 | ETA 03:07:19 2020-12-13 07:44:47 [INFO] [TRAIN] epoch=748, iter=139000/160000, loss=0.8146, lr=0.003216, batch_cost=0.5322, reader_cost=0.0127 | ETA 03:06:17 2020-12-13 07:46:34 [INFO] [TRAIN] epoch=749, iter=139200/160000, loss=0.8150, lr=0.003189, batch_cost=0.5324, reader_cost=0.0134 | ETA 03:04:32 2020-12-13 07:48:20 [INFO] [TRAIN] epoch=750, iter=139400/160000, loss=0.8120, lr=0.003161, batch_cost=0.5308, reader_cost=0.0124 | ETA 03:02:15 2020-12-13 07:50:06 [INFO] [TRAIN] epoch=751, iter=139600/160000, loss=0.8137, lr=0.003133, batch_cost=0.5317, reader_cost=0.0130 | ETA 03:00:46 2020-12-13 07:51:53 [INFO] [TRAIN] epoch=752, iter=139800/160000, loss=0.8109, lr=0.003106, batch_cost=0.5326, reader_cost=0.0135 | ETA 02:59:18 2020-12-13 07:53:39 [INFO] [TRAIN] epoch=753, iter=140000/160000, loss=0.8122, lr=0.003078, batch_cost=0.5329, reader_cost=0.0136 | ETA 02:57:37 2020-12-13 07:53:39 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-13 07:54:10 [INFO] [EVAL] #Images=500 mIoU=0.7020 Acc=0.9508 Kappa=0.9361 2020-12-13 07:54:10 [INFO] [EVAL] Class IoU: [0.9745 0.809 0.9131 0.343 0.5712 0.6187 0.7018 0.8016 0.9107 0.5756 0.9496 0.7614 0.5896 0.9248 0.485 0.6944 0.4466 0.5127 0.7552] 2020-12-13 07:54:10 [INFO] [EVAL] Class Acc: [0.9879 0.8918 0.9473 0.7655 0.7867 0.7939 0.861 0.9225 0.9525 0.8031 0.9694 0.8241 0.6941 0.9428 0.8006 0.7519 0.5736 0.6849 0.8658] 2020-12-13 07:54:11 [INFO] [EVAL] The model with the best validation mIoU (0.7707) was saved at iter 136000. 2020-12-13 07:55:57 [INFO] [TRAIN] epoch=754, iter=140200/160000, loss=0.8199, lr=0.003050, batch_cost=0.5329, reader_cost=0.0145 | ETA 02:55:50 2020-12-13 07:57:43 [INFO] [TRAIN] epoch=755, iter=140400/160000, loss=0.7951, lr=0.003023, batch_cost=0.5320, reader_cost=0.0125 | ETA 02:53:46 2020-12-13 07:59:30 [INFO] [TRAIN] epoch=756, iter=140600/160000, loss=0.8014, lr=0.002995, batch_cost=0.5303, reader_cost=0.0117 | ETA 02:51:27 2020-12-13 08:01:16 [INFO] [TRAIN] epoch=757, iter=140800/160000, loss=0.8092, lr=0.002967, batch_cost=0.5315, reader_cost=0.0138 | ETA 02:50:04 2020-12-13 08:03:04 [INFO] [TRAIN] epoch=759, iter=141000/160000, loss=0.8057, lr=0.002939, batch_cost=0.5409, reader_cost=0.0230 | ETA 02:51:17 2020-12-13 08:04:50 [INFO] [TRAIN] epoch=760, iter=141200/160000, loss=0.8079, lr=0.002911, batch_cost=0.5303, reader_cost=0.0128 | ETA 02:46:09 2020-12-13 08:06:36 [INFO] [TRAIN] epoch=761, iter=141400/160000, loss=0.8041, lr=0.002883, batch_cost=0.5302, reader_cost=0.0119 | ETA 02:44:21 2020-12-13 08:08:23 [INFO] [TRAIN] epoch=762, iter=141600/160000, loss=0.8064, lr=0.002855, batch_cost=0.5334, reader_cost=0.0147 | ETA 02:43:35 2020-12-13 08:10:09 [INFO] [TRAIN] epoch=763, iter=141800/160000, loss=0.8071, lr=0.002828, batch_cost=0.5319, reader_cost=0.0118 | ETA 02:41:20 2020-12-13 08:11:56 [INFO] [TRAIN] epoch=764, iter=142000/160000, loss=0.8032, lr=0.002800, batch_cost=0.5341, reader_cost=0.0135 | ETA 02:40:13 2020-12-13 08:13:42 [INFO] [TRAIN] epoch=765, iter=142200/160000, loss=0.8066, lr=0.002772, batch_cost=0.5323, reader_cost=0.0114 | ETA 02:37:54 2020-12-13 08:15:29 [INFO] [TRAIN] epoch=766, iter=142400/160000, loss=0.8028, lr=0.002744, batch_cost=0.5313, reader_cost=0.0131 | ETA 02:35:50 2020-12-13 08:17:15 [INFO] [TRAIN] epoch=767, iter=142600/160000, loss=0.7982, lr=0.002715, batch_cost=0.5305, reader_cost=0.0125 | ETA 02:33:49 2020-12-13 08:19:01 [INFO] [TRAIN] epoch=768, iter=142800/160000, loss=0.7978, lr=0.002687, batch_cost=0.5297, reader_cost=0.0120 | ETA 02:31:51 2020-12-13 08:20:47 [INFO] [TRAIN] epoch=769, iter=143000/160000, loss=0.8039, lr=0.002659, batch_cost=0.5333, reader_cost=0.0130 | ETA 02:31:06 2020-12-13 08:22:34 [INFO] [TRAIN] epoch=770, iter=143200/160000, loss=0.7988, lr=0.002631, batch_cost=0.5308, reader_cost=0.0132 | ETA 02:28:37 2020-12-13 08:24:19 [INFO] [TRAIN] epoch=771, iter=143400/160000, loss=0.7974, lr=0.002603, batch_cost=0.5275, reader_cost=0.0122 | ETA 02:25:57 2020-12-13 08:26:07 [INFO] [TRAIN] epoch=773, iter=143600/160000, loss=0.7965, lr=0.002575, batch_cost=0.5404, reader_cost=0.0230 | ETA 02:27:42 2020-12-13 08:27:53 [INFO] [TRAIN] epoch=774, iter=143800/160000, loss=0.8033, lr=0.002546, batch_cost=0.5309, reader_cost=0.0129 | ETA 02:23:21 2020-12-13 08:29:39 [INFO] [TRAIN] epoch=775, iter=144000/160000, loss=0.7869, lr=0.002518, batch_cost=0.5297, reader_cost=0.0129 | ETA 02:21:14 2020-12-13 08:29:39 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-13 08:30:11 [INFO] [EVAL] #Images=500 mIoU=0.7329 Acc=0.9548 Kappa=0.9414 2020-12-13 08:30:11 [INFO] [EVAL] Class IoU: [0.978 0.8264 0.9148 0.4627 0.573 0.6439 0.7 0.7872 0.9142 0.6346 0.9492 0.7831 0.5939 0.9396 0.6052 0.7506 0.5741 0.548 0.7472] 2020-12-13 08:30:11 [INFO] [EVAL] Class Acc: [0.9888 0.9068 0.9465 0.8108 0.797 0.8388 0.8037 0.877 0.9568 0.8005 0.9712 0.844 0.6673 0.9632 0.7179 0.8631 0.8578 0.6785 0.86 ] 2020-12-13 08:30:11 [INFO] [EVAL] The model with the best validation mIoU (0.7707) was saved at iter 136000. 2020-12-13 08:31:57 [INFO] [TRAIN] epoch=776, iter=144200/160000, loss=0.7959, lr=0.002490, batch_cost=0.5312, reader_cost=0.0134 | ETA 02:19:52 2020-12-13 08:33:43 [INFO] [TRAIN] epoch=777, iter=144400/160000, loss=0.7874, lr=0.002461, batch_cost=0.5308, reader_cost=0.0147 | ETA 02:18:00 2020-12-13 08:35:30 [INFO] [TRAIN] epoch=778, iter=144600/160000, loss=0.8107, lr=0.002433, batch_cost=0.5306, reader_cost=0.0138 | ETA 02:16:11 2020-12-13 08:37:16 [INFO] [TRAIN] epoch=779, iter=144800/160000, loss=0.8037, lr=0.002404, batch_cost=0.5315, reader_cost=0.0139 | ETA 02:14:38 2020-12-13 08:39:02 [INFO] [TRAIN] epoch=780, iter=145000/160000, loss=0.8046, lr=0.002376, batch_cost=0.5315, reader_cost=0.0128 | ETA 02:12:52 2020-12-13 08:40:48 [INFO] [TRAIN] epoch=781, iter=145200/160000, loss=0.7986, lr=0.002347, batch_cost=0.5289, reader_cost=0.0136 | ETA 02:10:28 2020-12-13 08:42:35 [INFO] [TRAIN] epoch=782, iter=145400/160000, loss=0.8117, lr=0.002319, batch_cost=0.5330, reader_cost=0.0130 | ETA 02:09:41 2020-12-13 08:44:21 [INFO] [TRAIN] epoch=783, iter=145600/160000, loss=0.7956, lr=0.002290, batch_cost=0.5321, reader_cost=0.0131 | ETA 02:07:41 2020-12-13 08:46:07 [INFO] [TRAIN] epoch=784, iter=145800/160000, loss=0.7988, lr=0.002262, batch_cost=0.5320, reader_cost=0.0132 | ETA 02:05:54 2020-12-13 08:47:54 [INFO] [TRAIN] epoch=785, iter=146000/160000, loss=0.7938, lr=0.002233, batch_cost=0.5332, reader_cost=0.0127 | ETA 02:04:24 2020-12-13 08:49:42 [INFO] [TRAIN] epoch=787, iter=146200/160000, loss=0.7973, lr=0.002204, batch_cost=0.5400, reader_cost=0.0219 | ETA 02:04:11 2020-12-13 08:51:28 [INFO] [TRAIN] epoch=788, iter=146400/160000, loss=0.7950, lr=0.002175, batch_cost=0.5320, reader_cost=0.0127 | ETA 02:00:35 2020-12-13 08:53:15 [INFO] [TRAIN] epoch=789, iter=146600/160000, loss=0.7891, lr=0.002147, batch_cost=0.5322, reader_cost=0.0132 | ETA 01:58:51 2020-12-13 08:55:01 [INFO] [TRAIN] epoch=790, iter=146800/160000, loss=0.7897, lr=0.002118, batch_cost=0.5299, reader_cost=0.0131 | ETA 01:56:34 2020-12-13 08:56:47 [INFO] [TRAIN] epoch=791, iter=147000/160000, loss=0.7925, lr=0.002089, batch_cost=0.5304, reader_cost=0.0131 | ETA 01:54:55 2020-12-13 08:58:34 [INFO] [TRAIN] epoch=792, iter=147200/160000, loss=0.7977, lr=0.002060, batch_cost=0.5334, reader_cost=0.0134 | ETA 01:53:46 2020-12-13 09:00:20 [INFO] [TRAIN] epoch=793, iter=147400/160000, loss=0.7987, lr=0.002031, batch_cost=0.5331, reader_cost=0.0130 | ETA 01:51:57 2020-12-13 09:02:06 [INFO] [TRAIN] epoch=794, iter=147600/160000, loss=0.7948, lr=0.002002, batch_cost=0.5310, reader_cost=0.0119 | ETA 01:49:44 2020-12-13 09:03:53 [INFO] [TRAIN] epoch=795, iter=147800/160000, loss=0.7894, lr=0.001973, batch_cost=0.5308, reader_cost=0.0123 | ETA 01:47:55 2020-12-13 09:05:39 [INFO] [TRAIN] epoch=796, iter=148000/160000, loss=0.7881, lr=0.001944, batch_cost=0.5329, reader_cost=0.0141 | ETA 01:46:34 2020-12-13 09:05:39 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-13 09:06:08 [INFO] [EVAL] #Images=500 mIoU=0.7567 Acc=0.9620 Kappa=0.9507 2020-12-13 09:06:08 [INFO] [EVAL] Class IoU: [0.9831 0.8599 0.9278 0.539 0.6201 0.6693 0.7355 0.8037 0.9271 0.6417 0.9497 0.8343 0.6449 0.9498 0.6788 0.766 0.4764 0.5921 0.7786] 2020-12-13 09:06:08 [INFO] [EVAL] Class Acc: [0.992 0.92 0.9591 0.8779 0.8245 0.8563 0.8491 0.9192 0.9529 0.8267 0.9662 0.9014 0.7626 0.9716 0.8336 0.8027 0.9038 0.7444 0.8676] 2020-12-13 09:06:09 [INFO] [EVAL] The model with the best validation mIoU (0.7707) was saved at iter 136000. 2020-12-13 09:07:55 [INFO] [TRAIN] epoch=797, iter=148200/160000, loss=0.7865, lr=0.001914, batch_cost=0.5314, reader_cost=0.0135 | ETA 01:44:30 2020-12-13 09:09:41 [INFO] [TRAIN] epoch=798, iter=148400/160000, loss=0.7933, lr=0.001885, batch_cost=0.5314, reader_cost=0.0129 | ETA 01:42:44 2020-12-13 09:11:28 [INFO] [TRAIN] epoch=799, iter=148600/160000, loss=0.7862, lr=0.001856, batch_cost=0.5305, reader_cost=0.0128 | ETA 01:40:47 2020-12-13 09:13:14 [INFO] [TRAIN] epoch=800, iter=148800/160000, loss=0.7848, lr=0.001827, batch_cost=0.5314, reader_cost=0.0136 | ETA 01:39:11 2020-12-13 09:15:02 [INFO] [TRAIN] epoch=802, iter=149000/160000, loss=0.7854, lr=0.001797, batch_cost=0.5432, reader_cost=0.0224 | ETA 01:39:35 2020-12-13 09:16:49 [INFO] [TRAIN] epoch=803, iter=149200/160000, loss=0.7888, lr=0.001768, batch_cost=0.5332, reader_cost=0.0138 | ETA 01:35:58 2020-12-13 09:18:35 [INFO] [TRAIN] epoch=804, iter=149400/160000, loss=0.7886, lr=0.001738, batch_cost=0.5315, reader_cost=0.0139 | ETA 01:33:53 2020-12-13 09:20:22 [INFO] [TRAIN] epoch=805, iter=149600/160000, loss=0.7965, lr=0.001709, batch_cost=0.5323, reader_cost=0.0142 | ETA 01:32:16 2020-12-13 09:22:08 [INFO] [TRAIN] epoch=806, iter=149800/160000, loss=0.7907, lr=0.001679, batch_cost=0.5314, reader_cost=0.0144 | ETA 01:30:19 2020-12-13 09:23:54 [INFO] [TRAIN] epoch=807, iter=150000/160000, loss=0.7855, lr=0.001650, batch_cost=0.5309, reader_cost=0.0136 | ETA 01:28:29 2020-12-13 09:25:41 [INFO] [TRAIN] epoch=808, iter=150200/160000, loss=0.7866, lr=0.001620, batch_cost=0.5311, reader_cost=0.0138 | ETA 01:26:45 2020-12-13 09:27:27 [INFO] [TRAIN] epoch=809, iter=150400/160000, loss=0.7905, lr=0.001590, batch_cost=0.5311, reader_cost=0.0144 | ETA 01:24:58 2020-12-13 09:29:13 [INFO] [TRAIN] epoch=810, iter=150600/160000, loss=0.7911, lr=0.001560, batch_cost=0.5300, reader_cost=0.0143 | ETA 01:23:02 2020-12-13 09:30:59 [INFO] [TRAIN] epoch=811, iter=150800/160000, loss=0.7864, lr=0.001530, batch_cost=0.5307, reader_cost=0.0134 | ETA 01:21:22 2020-12-13 09:32:45 [INFO] [TRAIN] epoch=812, iter=151000/160000, loss=0.7757, lr=0.001500, batch_cost=0.5320, reader_cost=0.0145 | ETA 01:19:47 2020-12-13 09:34:31 [INFO] [TRAIN] epoch=813, iter=151200/160000, loss=0.7851, lr=0.001470, batch_cost=0.5306, reader_cost=0.0141 | ETA 01:17:49 2020-12-13 09:36:18 [INFO] [TRAIN] epoch=814, iter=151400/160000, loss=0.7870, lr=0.001440, batch_cost=0.5308, reader_cost=0.0127 | ETA 01:16:05 2020-12-13 09:38:06 [INFO] [TRAIN] epoch=816, iter=151600/160000, loss=0.7758, lr=0.001410, batch_cost=0.5417, reader_cost=0.0216 | ETA 01:15:50 2020-12-13 09:39:52 [INFO] [TRAIN] epoch=817, iter=151800/160000, loss=0.7873, lr=0.001380, batch_cost=0.5320, reader_cost=0.0132 | ETA 01:12:42 2020-12-13 09:41:38 [INFO] [TRAIN] epoch=818, iter=152000/160000, loss=0.7815, lr=0.001349, batch_cost=0.5299, reader_cost=0.0132 | ETA 01:10:38 2020-12-13 09:41:38 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-13 09:42:09 [INFO] [EVAL] #Images=500 mIoU=0.7700 Acc=0.9578 Kappa=0.9453 2020-12-13 09:42:09 [INFO] [EVAL] Class IoU: [0.9762 0.8237 0.9224 0.5237 0.6041 0.6627 0.7261 0.8106 0.9223 0.6289 0.9499 0.8255 0.6488 0.9445 0.7344 0.8093 0.745 0.599 0.772 ] 2020-12-13 09:42:09 [INFO] [EVAL] Class Acc: [0.9911 0.8889 0.9539 0.7172 0.7899 0.8081 0.8451 0.9149 0.9605 0.8114 0.9654 0.8934 0.786 0.9669 0.9163 0.9329 0.8815 0.809 0.8497] 2020-12-13 09:42:10 [INFO] [EVAL] The model with the best validation mIoU (0.7707) was saved at iter 136000. 2020-12-13 09:43:56 [INFO] [TRAIN] epoch=819, iter=152200/160000, loss=0.7880, lr=0.001319, batch_cost=0.5292, reader_cost=0.0129 | ETA 01:08:48 2020-12-13 09:45:42 [INFO] [TRAIN] epoch=820, iter=152400/160000, loss=0.7884, lr=0.001289, batch_cost=0.5330, reader_cost=0.0142 | ETA 01:07:30 2020-12-13 09:47:28 [INFO] [TRAIN] epoch=821, iter=152600/160000, loss=0.7748, lr=0.001258, batch_cost=0.5290, reader_cost=0.0129 | ETA 01:05:14 2020-12-13 09:49:14 [INFO] [TRAIN] epoch=822, iter=152800/160000, loss=0.7853, lr=0.001227, batch_cost=0.5309, reader_cost=0.0133 | ETA 01:03:42 2020-12-13 09:51:00 [INFO] [TRAIN] epoch=823, iter=153000/160000, loss=0.7820, lr=0.001197, batch_cost=0.5316, reader_cost=0.0140 | ETA 01:02:00 2020-12-13 09:52:47 [INFO] [TRAIN] epoch=824, iter=153200/160000, loss=0.7744, lr=0.001166, batch_cost=0.5323, reader_cost=0.0142 | ETA 01:00:19 2020-12-13 09:54:33 [INFO] [TRAIN] epoch=825, iter=153400/160000, loss=0.7737, lr=0.001135, batch_cost=0.5317, reader_cost=0.0146 | ETA 00:58:29 2020-12-13 09:56:19 [INFO] [TRAIN] epoch=826, iter=153600/160000, loss=0.7709, lr=0.001104, batch_cost=0.5304, reader_cost=0.0132 | ETA 00:56:34 2020-12-13 09:58:06 [INFO] [TRAIN] epoch=827, iter=153800/160000, loss=0.7738, lr=0.001073, batch_cost=0.5310, reader_cost=0.0147 | ETA 00:54:51 2020-12-13 09:59:52 [INFO] [TRAIN] epoch=828, iter=154000/160000, loss=0.7704, lr=0.001042, batch_cost=0.5307, reader_cost=0.0131 | ETA 00:53:04 2020-12-13 10:01:41 [INFO] [TRAIN] epoch=830, iter=154200/160000, loss=0.7870, lr=0.001010, batch_cost=0.5441, reader_cost=0.0218 | ETA 00:52:35 2020-12-13 10:03:26 [INFO] [TRAIN] epoch=831, iter=154400/160000, loss=0.7623, lr=0.000979, batch_cost=0.5285, reader_cost=0.0122 | ETA 00:49:19 2020-12-13 10:05:13 [INFO] [TRAIN] epoch=832, iter=154600/160000, loss=0.7852, lr=0.000947, batch_cost=0.5331, reader_cost=0.0129 | ETA 00:47:58 2020-12-13 10:06:59 [INFO] [TRAIN] epoch=833, iter=154800/160000, loss=0.7775, lr=0.000916, batch_cost=0.5328, reader_cost=0.0129 | ETA 00:46:10 2020-12-13 10:08:46 [INFO] [TRAIN] epoch=834, iter=155000/160000, loss=0.7680, lr=0.000884, batch_cost=0.5315, reader_cost=0.0128 | ETA 00:44:17 2020-12-13 10:10:32 [INFO] [TRAIN] epoch=835, iter=155200/160000, loss=0.7835, lr=0.000852, batch_cost=0.5309, reader_cost=0.0137 | ETA 00:42:28 2020-12-13 10:12:18 [INFO] [TRAIN] epoch=836, iter=155400/160000, loss=0.7735, lr=0.000820, batch_cost=0.5313, reader_cost=0.0135 | ETA 00:40:43 2020-12-13 10:14:04 [INFO] [TRAIN] epoch=837, iter=155600/160000, loss=0.7654, lr=0.000788, batch_cost=0.5309, reader_cost=0.0128 | ETA 00:38:56 2020-12-13 10:15:51 [INFO] [TRAIN] epoch=838, iter=155800/160000, loss=0.7725, lr=0.000756, batch_cost=0.5318, reader_cost=0.0167 | ETA 00:37:13 2020-12-13 10:17:37 [INFO] [TRAIN] epoch=839, iter=156000/160000, loss=0.7816, lr=0.000723, batch_cost=0.5307, reader_cost=0.0142 | ETA 00:35:22 2020-12-13 10:17:37 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-13 10:18:07 [INFO] [EVAL] #Images=500 mIoU=0.7903 Acc=0.9630 Kappa=0.9520 2020-12-13 10:18:07 [INFO] [EVAL] Class IoU: [0.9809 0.8501 0.9306 0.6098 0.6226 0.6874 0.7375 0.8183 0.929 0.6533 0.9509 0.8382 0.6564 0.9535 0.7365 0.8615 0.7694 0.6467 0.7828] 2020-12-13 10:18:07 [INFO] [EVAL] Class Acc: [0.9911 0.9102 0.96 0.8061 0.8309 0.8387 0.8599 0.9235 0.9604 0.8207 0.9689 0.9066 0.7759 0.9749 0.8921 0.9347 0.8923 0.8139 0.8728] 2020-12-13 10:18:08 [INFO] [EVAL] The model with the best validation mIoU (0.7903) was saved at iter 156000. 2020-12-13 10:19:54 [INFO] [TRAIN] epoch=840, iter=156200/160000, loss=0.7659, lr=0.000691, batch_cost=0.5303, reader_cost=0.0139 | ETA 00:33:35 2020-12-13 10:21:40 [INFO] [TRAIN] epoch=841, iter=156400/160000, loss=0.7718, lr=0.000658, batch_cost=0.5285, reader_cost=0.0145 | ETA 00:31:42 2020-12-13 10:23:25 [INFO] [TRAIN] epoch=842, iter=156600/160000, loss=0.7697, lr=0.000625, batch_cost=0.5282, reader_cost=0.0130 | ETA 00:29:55 2020-12-13 10:25:12 [INFO] [TRAIN] epoch=844, iter=156800/160000, loss=0.7768, lr=0.000592, batch_cost=0.5351, reader_cost=0.0211 | ETA 00:28:32 2020-12-13 10:26:57 [INFO] [TRAIN] epoch=845, iter=157000/160000, loss=0.7667, lr=0.000558, batch_cost=0.5261, reader_cost=0.0118 | ETA 00:26:18 2020-12-13 10:28:43 [INFO] [TRAIN] epoch=846, iter=157200/160000, loss=0.7696, lr=0.000525, batch_cost=0.5285, reader_cost=0.0121 | ETA 00:24:39 2020-12-13 10:30:28 [INFO] [TRAIN] epoch=847, iter=157400/160000, loss=0.7725, lr=0.000491, batch_cost=0.5261, reader_cost=0.0103 | ETA 00:22:47 2020-12-13 10:32:13 [INFO] [TRAIN] epoch=848, iter=157600/160000, loss=0.7710, lr=0.000457, batch_cost=0.5254, reader_cost=0.0131 | ETA 00:21:00 2020-12-13 10:33:59 [INFO] [TRAIN] epoch=849, iter=157800/160000, loss=0.7726, lr=0.000422, batch_cost=0.5273, reader_cost=0.0114 | ETA 00:19:20 2020-12-13 10:35:44 [INFO] [TRAIN] epoch=850, iter=158000/160000, loss=0.7672, lr=0.000388, batch_cost=0.5271, reader_cost=0.0135 | ETA 00:17:34 2020-12-13 10:37:30 [INFO] [TRAIN] epoch=851, iter=158200/160000, loss=0.7648, lr=0.000353, batch_cost=0.5260, reader_cost=0.0132 | ETA 00:15:46 2020-12-13 10:39:15 [INFO] [TRAIN] epoch=852, iter=158400/160000, loss=0.7593, lr=0.000317, batch_cost=0.5254, reader_cost=0.0144 | ETA 00:14:00 2020-12-13 10:41:00 [INFO] [TRAIN] epoch=853, iter=158600/160000, loss=0.7637, lr=0.000281, batch_cost=0.5275, reader_cost=0.0136 | ETA 00:12:18 2020-12-13 10:42:46 [INFO] [TRAIN] epoch=854, iter=158800/160000, loss=0.7574, lr=0.000245, batch_cost=0.5277, reader_cost=0.0137 | ETA 00:10:33 2020-12-13 10:44:31 [INFO] [TRAIN] epoch=855, iter=159000/160000, loss=0.7626, lr=0.000208, batch_cost=0.5270, reader_cost=0.0130 | ETA 00:08:47 2020-12-13 10:46:16 [INFO] [TRAIN] epoch=856, iter=159200/160000, loss=0.7646, lr=0.000170, batch_cost=0.5267, reader_cost=0.0120 | ETA 00:07:01 2020-12-13 10:48:01 [INFO] [TRAIN] epoch=857, iter=159400/160000, loss=0.7700, lr=0.000131, batch_cost=0.5241, reader_cost=0.0109 | ETA 00:05:14 2020-12-13 10:49:49 [INFO] [TRAIN] epoch=859, iter=159600/160000, loss=0.7602, lr=0.000091, batch_cost=0.5367, reader_cost=0.0216 | ETA 00:03:34 2020-12-13 10:51:34 [INFO] [TRAIN] epoch=860, iter=159800/160000, loss=0.7746, lr=0.000049, batch_cost=0.5268, reader_cost=0.0121 | ETA 00:01:45 2020-12-13 10:53:20 [INFO] [TRAIN] epoch=861, iter=160000/160000, loss=0.7695, lr=0.000000, batch_cost=0.5303, reader_cost=0.0133 | ETA 00:00:00 2020-12-13 10:53:20 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-13 10:53:49 [INFO] [EVAL] #Images=500 mIoU=0.7886 Acc=0.9636 Kappa=0.9528 2020-12-13 10:53:49 [INFO] [EVAL] Class IoU: [0.9822 0.8577 0.9311 0.5941 0.627 0.693 0.7449 0.8185 0.9293 0.6457 0.9521 0.8408 0.6626 0.9535 0.7378 0.8508 0.7392 0.6373 0.7854] 2020-12-13 10:53:49 [INFO] [EVAL] Class Acc: [0.9922 0.9147 0.9608 0.83 0.8242 0.843 0.862 0.9169 0.9592 0.818 0.9695 0.904 0.7883 0.973 0.9159 0.902 0.9214 0.7857 0.8727] 2020-12-13 10:53:50 [INFO] [EVAL] The model with the best validation mIoU (0.7903) was saved at iter 156000.