2020-12-02 15:32:51 [INFO] ------------Environment Information------------- platform: Linux-3.10.0-1062.18.1.el7.x86_64-x86_64-with-centos-7.7.1908-Core Python: 3.7.9 (default, Aug 31 2020, 12:42:55) [GCC 7.3.0] Paddle compiled with cuda: True NVCC: Cuda compilation tools, release 10.2, V10.2.89 cudnn: 7.6 GPUs used: 4 CUDA_VISIBLE_DEVICES: 4,5,6,7 GPU: ['GPU 0: Tesla V100-SXM2-16GB', 'GPU 1: Tesla V100-SXM2-16GB', 'GPU 2: Tesla V100-SXM2-16GB', 'GPU 3: Tesla V100-SXM2-16GB', 'GPU 4: Tesla V100-SXM2-16GB', 'GPU 5: Tesla V100-SXM2-16GB', 'GPU 6: Tesla V100-SXM2-16GB', 'GPU 7: Tesla V100-SXM2-16GB'] GCC: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39) PaddlePaddle: 2.0.0-rc0 OpenCV: 4.1.1 ------------------------------------------------ 2020-12-02 15:32:51 [INFO] ---------------Config Information--------------- batch_size: 4 iters: 40000 learning_rate: decay: end_lr: 0.0 power: 0.9 type: poly value: 0.01 loss: coef: - 1 - 0.4 types: - ignore_index: 255 type: CrossEntropyLoss model: align_corners: false backbone: output_stride: 8 pretrained: https://bj.bcebos.com/paddleseg/dygraph/resnet101_vd_ssld.tar.gz type: ResNet101_vd enable_auxiliary_loss: true pretrained: null type: PSPNet optimizer: momentum: 0.9 type: sgd weight_decay: 4.0e-05 train_dataset: dataset_root: data/VOCdevkit/ mode: trainaug transforms: - crop_size: - 512 - 512 type: RandomPaddingCrop - type: RandomHorizontalFlip - brightness_range: 0.4 contrast_range: 0.4 saturation_range: 0.4 type: RandomDistort - type: Normalize type: PascalVOC val_dataset: dataset_root: data/VOCdevkit/ mode: val transforms: - target_size: - 512 - 512 type: Padding - type: Normalize type: PascalVOC ------------------------------------------------ 2020-12-02 15:32:58 [INFO] Loading pretrained model from https://bj.bcebos.com/paddleseg/dygraph/resnet101_vd_ssld.tar.gz 2020-12-02 15:33:01 [INFO] There are 530/530 variables loaded into ResNet_vd. 2020-12-02 15:34:36 [INFO] [TRAIN] epoch=1, iter=100/40000, loss=1.3707, lr=0.009978, batch_cost=0.8868, reader_cost=0.0151 | ETA 09:49:43 2020-12-02 15:35:59 [INFO] [TRAIN] epoch=1, iter=200/40000, loss=0.8521, lr=0.009955, batch_cost=0.8283, reader_cost=0.0002 | ETA 09:09:27 2020-12-02 15:37:22 [INFO] [TRAIN] epoch=1, iter=300/40000, loss=0.7921, lr=0.009933, batch_cost=0.8341, reader_cost=0.0002 | ETA 09:11:55 2020-12-02 15:38:50 [INFO] [TRAIN] epoch=1, iter=400/40000, loss=0.6706, lr=0.009910, batch_cost=0.8763, reader_cost=0.0002 | ETA 09:38:22 2020-12-02 15:40:16 [INFO] [TRAIN] epoch=1, iter=500/40000, loss=0.5951, lr=0.009888, batch_cost=0.8590, reader_cost=0.0002 | ETA 09:25:30 2020-12-02 15:41:39 [INFO] [TRAIN] epoch=1, iter=600/40000, loss=0.6563, lr=0.009865, batch_cost=0.8353, reader_cost=0.0002 | ETA 09:08:30 2020-12-02 15:43:05 [INFO] [TRAIN] epoch=2, iter=700/40000, loss=0.5494, lr=0.009843, batch_cost=0.8525, reader_cost=0.0103 | ETA 09:18:25 2020-12-02 15:44:29 [INFO] [TRAIN] epoch=2, iter=800/40000, loss=0.5284, lr=0.009820, batch_cost=0.8450, reader_cost=0.0002 | ETA 09:12:04 2020-12-02 15:45:51 [INFO] [TRAIN] epoch=2, iter=900/40000, loss=0.4933, lr=0.009797, batch_cost=0.8199, reader_cost=0.0002 | ETA 08:54:19 2020-12-02 15:47:14 [INFO] [TRAIN] epoch=2, iter=1000/40000, loss=0.5124, lr=0.009775, batch_cost=0.8247, reader_cost=0.0004 | ETA 08:56:03 2020-12-02 15:48:39 [INFO] [TRAIN] epoch=2, iter=1100/40000, loss=0.4966, lr=0.009752, batch_cost=0.8532, reader_cost=0.0002 | ETA 09:13:07 2020-12-02 15:50:10 [INFO] [TRAIN] epoch=2, iter=1200/40000, loss=0.5225, lr=0.009730, batch_cost=0.9125, reader_cost=0.0003 | ETA 09:50:06 2020-12-02 15:51:31 [INFO] [TRAIN] epoch=2, iter=1300/40000, loss=0.4971, lr=0.009707, batch_cost=0.8097, reader_cost=0.0002 | ETA 08:42:13 2020-12-02 15:52:55 [INFO] [TRAIN] epoch=3, iter=1400/40000, loss=0.4555, lr=0.009685, batch_cost=0.8334, reader_cost=0.0088 | ETA 08:56:08 2020-12-02 15:54:14 [INFO] [TRAIN] epoch=3, iter=1500/40000, loss=0.4195, lr=0.009662, batch_cost=0.7938, reader_cost=0.0002 | ETA 08:29:21 2020-12-02 15:55:36 [INFO] [TRAIN] epoch=3, iter=1600/40000, loss=0.4049, lr=0.009639, batch_cost=0.8200, reader_cost=0.0002 | ETA 08:44:48 2020-12-02 15:56:55 [INFO] [TRAIN] epoch=3, iter=1700/40000, loss=0.3817, lr=0.009617, batch_cost=0.7901, reader_cost=0.0002 | ETA 08:24:20 2020-12-02 15:58:19 [INFO] [TRAIN] epoch=3, iter=1800/40000, loss=0.4553, lr=0.009594, batch_cost=0.8428, reader_cost=0.0002 | ETA 08:56:35 2020-12-02 15:59:41 [INFO] [TRAIN] epoch=3, iter=1900/40000, loss=0.3851, lr=0.009572, batch_cost=0.8137, reader_cost=0.0005 | ETA 08:36:40 2020-12-02 16:01:04 [INFO] [TRAIN] epoch=4, iter=2000/40000, loss=0.4407, lr=0.009549, batch_cost=0.8327, reader_cost=0.0092 | ETA 08:47:21 2020-12-02 16:01:04 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-12-02 16:01:43 [INFO] [EVAL] #Images=1449 mIoU=0.6496 Acc=0.9211 Kappa=0.8280 2020-12-02 16:01:43 [INFO] [EVAL] Class IoU: [0.9247 0.5889 0.3491 0.7942 0.5138 0.6507 0.8604 0.671 0.8623 0.3692 0.5665 0.423 0.7419 0.7725 0.6559 0.8071 0.6159 0.5365 0.4711 0.8012 0.6668] 2020-12-02 16:01:43 [INFO] [EVAL] Class Acc: [0.9629 0.5943 0.3698 0.8475 0.794 0.849 0.9384 0.7084 0.9217 0.6372 0.6584 0.8276 0.8066 0.8983 0.9147 0.8759 0.8485 0.6588 0.7068 0.9576 0.8974] 2020-12-02 16:02:02 [INFO] [EVAL] The model with the best validation mIoU (0.6496) was saved at iter 2000. 2020-12-02 16:03:19 [INFO] [TRAIN] epoch=4, iter=2100/40000, loss=0.3131, lr=0.009526, batch_cost=0.7776, reader_cost=0.0003 | ETA 08:11:11 2020-12-02 16:04:39 [INFO] [TRAIN] epoch=4, iter=2200/40000, loss=0.3717, lr=0.009504, batch_cost=0.7940, reader_cost=0.0002 | ETA 08:20:12 2020-12-02 16:05:56 [INFO] [TRAIN] epoch=4, iter=2300/40000, loss=0.3387, lr=0.009481, batch_cost=0.7705, reader_cost=0.0002 | ETA 08:04:06 2020-12-02 16:07:18 [INFO] [TRAIN] epoch=4, iter=2400/40000, loss=0.3504, lr=0.009459, batch_cost=0.8233, reader_cost=0.0005 | ETA 08:35:57 2020-12-02 16:08:42 [INFO] [TRAIN] epoch=4, iter=2500/40000, loss=0.3535, lr=0.009436, batch_cost=0.8358, reader_cost=0.0002 | ETA 08:42:23 2020-12-02 16:10:02 [INFO] [TRAIN] epoch=4, iter=2600/40000, loss=0.3420, lr=0.009413, batch_cost=0.7978, reader_cost=0.0002 | ETA 08:17:17 2020-12-02 16:11:29 [INFO] [TRAIN] epoch=5, iter=2700/40000, loss=0.3284, lr=0.009391, batch_cost=0.8712, reader_cost=0.0114 | ETA 09:01:37 2020-12-02 16:12:51 [INFO] [TRAIN] epoch=5, iter=2800/40000, loss=0.2461, lr=0.009368, batch_cost=0.8278, reader_cost=0.0002 | ETA 08:33:14 2020-12-02 16:14:12 [INFO] [TRAIN] epoch=5, iter=2900/40000, loss=0.2681, lr=0.009345, batch_cost=0.8032, reader_cost=0.0003 | ETA 08:16:37 2020-12-02 16:15:32 [INFO] [TRAIN] epoch=5, iter=3000/40000, loss=0.2797, lr=0.009323, batch_cost=0.8027, reader_cost=0.0002 | ETA 08:14:59 2020-12-02 16:16:56 [INFO] [TRAIN] epoch=5, iter=3100/40000, loss=0.3240, lr=0.009300, batch_cost=0.8381, reader_cost=0.0002 | ETA 08:35:26 2020-12-02 16:18:14 [INFO] [TRAIN] epoch=5, iter=3200/40000, loss=0.3178, lr=0.009277, batch_cost=0.7790, reader_cost=0.0002 | ETA 07:57:48 2020-12-02 16:19:37 [INFO] [TRAIN] epoch=5, iter=3300/40000, loss=0.2826, lr=0.009255, batch_cost=0.8291, reader_cost=0.0002 | ETA 08:27:08 2020-12-02 16:20:57 [INFO] [TRAIN] epoch=6, iter=3400/40000, loss=0.2238, lr=0.009232, batch_cost=0.8005, reader_cost=0.0094 | ETA 08:08:19 2020-12-02 16:22:21 [INFO] [TRAIN] epoch=6, iter=3500/40000, loss=0.2561, lr=0.009209, batch_cost=0.8452, reader_cost=0.0002 | ETA 08:34:10 2020-12-02 16:23:42 [INFO] [TRAIN] epoch=6, iter=3600/40000, loss=0.2675, lr=0.009186, batch_cost=0.8048, reader_cost=0.0002 | ETA 08:08:12 2020-12-02 16:25:04 [INFO] [TRAIN] epoch=6, iter=3700/40000, loss=0.2600, lr=0.009164, batch_cost=0.8180, reader_cost=0.0002 | ETA 08:14:52 2020-12-02 16:26:22 [INFO] [TRAIN] epoch=6, iter=3800/40000, loss=0.2519, lr=0.009141, batch_cost=0.7892, reader_cost=0.0002 | ETA 07:56:08 2020-12-02 16:27:40 [INFO] [TRAIN] epoch=6, iter=3900/40000, loss=0.2420, lr=0.009118, batch_cost=0.7760, reader_cost=0.0002 | ETA 07:46:54 2020-12-02 16:29:00 [INFO] [TRAIN] epoch=7, iter=4000/40000, loss=0.2779, lr=0.009096, batch_cost=0.8036, reader_cost=0.0092 | ETA 08:02:08 2020-12-02 16:29:00 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-12-02 16:29:41 [INFO] [EVAL] #Images=1449 mIoU=0.7497 Acc=0.9437 Kappa=0.8750 2020-12-02 16:29:41 [INFO] [EVAL] Class IoU: [0.9392 0.8675 0.4066 0.8902 0.6745 0.7204 0.9421 0.8572 0.8716 0.3826 0.826 0.6208 0.7844 0.84 0.8793 0.8652 0.576 0.808 0.4227 0.8345 0.7357] 2020-12-02 16:29:41 [INFO] [EVAL] Class Acc: [0.9632 0.9343 0.4361 0.9758 0.8311 0.9421 0.9713 0.9467 0.9606 0.556 0.8696 0.8146 0.8316 0.9425 0.924 0.946 0.8547 0.8716 0.818 0.8766 0.9062] 2020-12-02 16:30:00 [INFO] [EVAL] The model with the best validation mIoU (0.7497) was saved at iter 4000. 2020-12-02 16:31:21 [INFO] [TRAIN] epoch=7, iter=4100/40000, loss=0.2041, lr=0.009073, batch_cost=0.8192, reader_cost=0.0009 | ETA 08:10:07 2020-12-02 16:32:43 [INFO] [TRAIN] epoch=7, iter=4200/40000, loss=0.2287, lr=0.009050, batch_cost=0.8175, reader_cost=0.0002 | ETA 08:07:44 2020-12-02 16:34:02 [INFO] [TRAIN] epoch=7, iter=4300/40000, loss=0.2079, lr=0.009027, batch_cost=0.7856, reader_cost=0.0002 | ETA 07:47:26 2020-12-02 16:35:20 [INFO] [TRAIN] epoch=7, iter=4400/40000, loss=0.2252, lr=0.009005, batch_cost=0.7791, reader_cost=0.0002 | ETA 07:42:16 2020-12-02 16:36:40 [INFO] [TRAIN] epoch=7, iter=4500/40000, loss=0.2272, lr=0.008982, batch_cost=0.8065, reader_cost=0.0002 | ETA 07:57:10 2020-12-02 16:37:58 [INFO] [TRAIN] epoch=7, iter=4600/40000, loss=0.2574, lr=0.008959, batch_cost=0.7814, reader_cost=0.0002 | ETA 07:41:02 2020-12-02 16:39:16 [INFO] [TRAIN] epoch=8, iter=4700/40000, loss=0.2197, lr=0.008936, batch_cost=0.7774, reader_cost=0.0091 | ETA 07:37:23 2020-12-02 16:40:33 [INFO] [TRAIN] epoch=8, iter=4800/40000, loss=0.2022, lr=0.008913, batch_cost=0.7685, reader_cost=0.0002 | ETA 07:30:50 2020-12-02 16:41:55 [INFO] [TRAIN] epoch=8, iter=4900/40000, loss=0.1908, lr=0.008891, batch_cost=0.8161, reader_cost=0.0002 | ETA 07:57:24 2020-12-02 16:43:17 [INFO] [TRAIN] epoch=8, iter=5000/40000, loss=0.1854, lr=0.008868, batch_cost=0.8186, reader_cost=0.0002 | ETA 07:57:32 2020-12-02 16:44:36 [INFO] [TRAIN] epoch=8, iter=5100/40000, loss=0.1927, lr=0.008845, batch_cost=0.7943, reader_cost=0.0002 | ETA 07:41:59 2020-12-02 16:45:55 [INFO] [TRAIN] epoch=8, iter=5200/40000, loss=0.2023, lr=0.008822, batch_cost=0.7916, reader_cost=0.0002 | ETA 07:39:06 2020-12-02 16:47:17 [INFO] [TRAIN] epoch=9, iter=5300/40000, loss=0.2245, lr=0.008799, batch_cost=0.8218, reader_cost=0.0088 | ETA 07:55:16 2020-12-02 16:48:39 [INFO] [TRAIN] epoch=9, iter=5400/40000, loss=0.1628, lr=0.008777, batch_cost=0.8156, reader_cost=0.0002 | ETA 07:50:20 2020-12-02 16:50:01 [INFO] [TRAIN] epoch=9, iter=5500/40000, loss=0.1720, lr=0.008754, batch_cost=0.8201, reader_cost=0.0002 | ETA 07:51:33 2020-12-02 16:51:21 [INFO] [TRAIN] epoch=9, iter=5600/40000, loss=0.1867, lr=0.008731, batch_cost=0.8020, reader_cost=0.0003 | ETA 07:39:47 2020-12-02 16:52:42 [INFO] [TRAIN] epoch=9, iter=5700/40000, loss=0.1816, lr=0.008708, batch_cost=0.8046, reader_cost=0.0002 | ETA 07:39:58 2020-12-02 16:54:04 [INFO] [TRAIN] epoch=9, iter=5800/40000, loss=0.1895, lr=0.008685, batch_cost=0.8235, reader_cost=0.0002 | ETA 07:49:24 2020-12-02 16:55:27 [INFO] [TRAIN] epoch=9, iter=5900/40000, loss=0.1820, lr=0.008662, batch_cost=0.8255, reader_cost=0.0002 | ETA 07:49:09 2020-12-02 16:56:49 [INFO] [TRAIN] epoch=10, iter=6000/40000, loss=0.1815, lr=0.008639, batch_cost=0.8299, reader_cost=0.0089 | ETA 07:50:17 2020-12-02 16:56:50 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-12-02 16:57:27 [INFO] [EVAL] #Images=1449 mIoU=0.7757 Acc=0.9495 Kappa=0.8870 2020-12-02 16:57:27 [INFO] [EVAL] Class IoU: [0.9438 0.8881 0.4111 0.9124 0.6989 0.8346 0.8959 0.8134 0.9063 0.436 0.8652 0.6185 0.8556 0.8782 0.8173 0.8633 0.6643 0.8502 0.5384 0.8494 0.7499] 2020-12-02 16:57:27 [INFO] [EVAL] Class Acc: [0.9625 0.9152 0.4416 0.9746 0.8366 0.9026 0.979 0.8884 0.9497 0.7151 0.9266 0.8659 0.9352 0.968 0.8861 0.9578 0.8816 0.9504 0.7945 0.948 0.9421] 2020-12-02 16:57:47 [INFO] [EVAL] The model with the best validation mIoU (0.7757) was saved at iter 6000. 2020-12-02 16:59:13 [INFO] [TRAIN] epoch=10, iter=6100/40000, loss=0.1414, lr=0.008617, batch_cost=0.8593, reader_cost=0.0003 | ETA 08:05:30 2020-12-02 17:00:31 [INFO] [TRAIN] epoch=10, iter=6200/40000, loss=0.1870, lr=0.008594, batch_cost=0.7775, reader_cost=0.0002 | ETA 07:18:01 2020-12-02 17:01:53 [INFO] [TRAIN] epoch=10, iter=6300/40000, loss=0.1787, lr=0.008571, batch_cost=0.8217, reader_cost=0.0003 | ETA 07:41:32 2020-12-02 17:03:10 [INFO] [TRAIN] epoch=10, iter=6400/40000, loss=0.1624, lr=0.008548, batch_cost=0.7725, reader_cost=0.0002 | ETA 07:12:35 2020-12-02 17:04:29 [INFO] [TRAIN] epoch=10, iter=6500/40000, loss=0.1585, lr=0.008525, batch_cost=0.7922, reader_cost=0.0002 | ETA 07:22:19 2020-12-02 17:05:52 [INFO] [TRAIN] epoch=10, iter=6600/40000, loss=0.1659, lr=0.008502, batch_cost=0.8228, reader_cost=0.0002 | ETA 07:38:01 2020-12-02 17:07:17 [INFO] [TRAIN] epoch=11, iter=6700/40000, loss=0.1359, lr=0.008479, batch_cost=0.8548, reader_cost=0.0106 | ETA 07:54:24 2020-12-02 17:08:41 [INFO] [TRAIN] epoch=11, iter=6800/40000, loss=0.1428, lr=0.008456, batch_cost=0.8357, reader_cost=0.0002 | ETA 07:42:24 2020-12-02 17:10:05 [INFO] [TRAIN] epoch=11, iter=6900/40000, loss=0.1345, lr=0.008433, batch_cost=0.8470, reader_cost=0.0002 | ETA 07:47:16 2020-12-02 17:11:28 [INFO] [TRAIN] epoch=11, iter=7000/40000, loss=0.1430, lr=0.008410, batch_cost=0.8300, reader_cost=0.0002 | ETA 07:36:28 2020-12-02 17:12:52 [INFO] [TRAIN] epoch=11, iter=7100/40000, loss=0.1454, lr=0.008388, batch_cost=0.8368, reader_cost=0.0002 | ETA 07:38:50 2020-12-02 17:14:12 [INFO] [TRAIN] epoch=11, iter=7200/40000, loss=0.1514, lr=0.008365, batch_cost=0.7982, reader_cost=0.0003 | ETA 07:16:20 2020-12-02 17:15:33 [INFO] [TRAIN] epoch=12, iter=7300/40000, loss=0.1458, lr=0.008342, batch_cost=0.8130, reader_cost=0.0110 | ETA 07:23:05 2020-12-02 17:16:55 [INFO] [TRAIN] epoch=12, iter=7400/40000, loss=0.1620, lr=0.008319, batch_cost=0.8167, reader_cost=0.0002 | ETA 07:23:43 2020-12-02 17:18:13 [INFO] [TRAIN] epoch=12, iter=7500/40000, loss=0.1796, lr=0.008296, batch_cost=0.7826, reader_cost=0.0002 | ETA 07:03:54 2020-12-02 17:19:34 [INFO] [TRAIN] epoch=12, iter=7600/40000, loss=0.1429, lr=0.008273, batch_cost=0.8070, reader_cost=0.0002 | ETA 07:15:45 2020-12-02 17:21:01 [INFO] [TRAIN] epoch=12, iter=7700/40000, loss=0.1593, lr=0.008250, batch_cost=0.8692, reader_cost=0.0005 | ETA 07:47:54 2020-12-02 17:22:21 [INFO] [TRAIN] epoch=12, iter=7800/40000, loss=0.1657, lr=0.008227, batch_cost=0.8063, reader_cost=0.0004 | ETA 07:12:43 2020-12-02 17:23:40 [INFO] [TRAIN] epoch=12, iter=7900/40000, loss=0.1731, lr=0.008204, batch_cost=0.7834, reader_cost=0.0002 | ETA 06:59:05 2020-12-02 17:24:59 [INFO] [TRAIN] epoch=13, iter=8000/40000, loss=0.1476, lr=0.008181, batch_cost=0.7932, reader_cost=0.0076 | ETA 07:03:02 2020-12-02 17:24:59 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-12-02 17:25:34 [INFO] [EVAL] #Images=1449 mIoU=0.7843 Acc=0.9506 Kappa=0.8913 2020-12-02 17:25:34 [INFO] [EVAL] Class IoU: [0.9439 0.9014 0.4084 0.9014 0.7364 0.8251 0.9359 0.8721 0.8961 0.4609 0.8887 0.5357 0.8806 0.8957 0.8471 0.8639 0.6205 0.8565 0.5388 0.8952 0.7662] 2020-12-02 17:25:34 [INFO] [EVAL] Class Acc: [0.9687 0.9289 0.426 0.9374 0.8284 0.9133 0.9738 0.9319 0.9276 0.6549 0.9525 0.8893 0.926 0.9336 0.9123 0.9293 0.8424 0.9261 0.8187 0.9273 0.8917] 2020-12-02 17:25:51 [INFO] [EVAL] The model with the best validation mIoU (0.7843) was saved at iter 8000. 2020-12-02 17:27:13 [INFO] [TRAIN] epoch=13, iter=8100/40000, loss=0.1182, lr=0.008158, batch_cost=0.8199, reader_cost=0.0003 | ETA 07:15:53 2020-12-02 17:28:36 [INFO] [TRAIN] epoch=13, iter=8200/40000, loss=0.1427, lr=0.008135, batch_cost=0.8341, reader_cost=0.0002 | ETA 07:22:03 2020-12-02 17:30:00 [INFO] [TRAIN] epoch=13, iter=8300/40000, loss=0.1272, lr=0.008112, batch_cost=0.8427, reader_cost=0.0002 | ETA 07:25:14 2020-12-02 17:31:20 [INFO] [TRAIN] epoch=13, iter=8400/40000, loss=0.1521, lr=0.008089, batch_cost=0.8009, reader_cost=0.0002 | ETA 07:01:49 2020-12-02 17:32:43 [INFO] [TRAIN] epoch=13, iter=8500/40000, loss=0.1288, lr=0.008066, batch_cost=0.8235, reader_cost=0.0003 | ETA 07:12:20 2020-12-02 17:34:06 [INFO] [TRAIN] epoch=14, iter=8600/40000, loss=0.1655, lr=0.008043, batch_cost=0.8379, reader_cost=0.0088 | ETA 07:18:30 2020-12-02 17:35:34 [INFO] [TRAIN] epoch=14, iter=8700/40000, loss=0.1370, lr=0.008020, batch_cost=0.8800, reader_cost=0.0002 | ETA 07:39:02 2020-12-02 17:36:54 [INFO] [TRAIN] epoch=14, iter=8800/40000, loss=0.1276, lr=0.007996, batch_cost=0.7956, reader_cost=0.0002 | ETA 06:53:43 2020-12-02 17:38:15 [INFO] [TRAIN] epoch=14, iter=8900/40000, loss=0.1139, lr=0.007973, batch_cost=0.8077, reader_cost=0.0003 | ETA 06:58:39 2020-12-02 17:39:34 [INFO] [TRAIN] epoch=14, iter=9000/40000, loss=0.1387, lr=0.007950, batch_cost=0.7920, reader_cost=0.0002 | ETA 06:49:10 2020-12-02 17:40:58 [INFO] [TRAIN] epoch=14, iter=9100/40000, loss=0.1208, lr=0.007927, batch_cost=0.8382, reader_cost=0.0003 | ETA 07:11:39 2020-12-02 17:42:24 [INFO] [TRAIN] epoch=14, iter=9200/40000, loss=0.1259, lr=0.007904, batch_cost=0.8579, reader_cost=0.0004 | ETA 07:20:24 2020-12-02 17:43:45 [INFO] [TRAIN] epoch=15, iter=9300/40000, loss=0.1154, lr=0.007881, batch_cost=0.8105, reader_cost=0.0096 | ETA 06:54:42 2020-12-02 17:45:08 [INFO] [TRAIN] epoch=15, iter=9400/40000, loss=0.1006, lr=0.007858, batch_cost=0.8309, reader_cost=0.0002 | ETA 07:03:46 2020-12-02 17:46:29 [INFO] [TRAIN] epoch=15, iter=9500/40000, loss=0.1051, lr=0.007835, batch_cost=0.8129, reader_cost=0.0003 | ETA 06:53:12 2020-12-02 17:47:49 [INFO] [TRAIN] epoch=15, iter=9600/40000, loss=0.1045, lr=0.007812, batch_cost=0.8014, reader_cost=0.0002 | ETA 06:46:01 2020-12-02 17:49:12 [INFO] [TRAIN] epoch=15, iter=9700/40000, loss=0.1003, lr=0.007789, batch_cost=0.8308, reader_cost=0.0005 | ETA 06:59:33 2020-12-02 17:50:36 [INFO] [TRAIN] epoch=15, iter=9800/40000, loss=0.1313, lr=0.007765, batch_cost=0.8376, reader_cost=0.0002 | ETA 07:01:34 2020-12-02 17:51:55 [INFO] [TRAIN] epoch=15, iter=9900/40000, loss=0.1107, lr=0.007742, batch_cost=0.7922, reader_cost=0.0002 | ETA 06:37:25 2020-12-02 17:53:16 [INFO] [TRAIN] epoch=16, iter=10000/40000, loss=0.1035, lr=0.007719, batch_cost=0.8080, reader_cost=0.0086 | ETA 06:43:58 2020-12-02 17:53:16 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-12-02 17:53:55 [INFO] [EVAL] #Images=1449 mIoU=0.7819 Acc=0.9503 Kappa=0.8896 2020-12-02 17:53:55 [INFO] [EVAL] Class IoU: [0.9444 0.8862 0.4401 0.9257 0.6708 0.8373 0.9444 0.8937 0.9333 0.4116 0.8888 0.5575 0.8841 0.9012 0.8782 0.8697 0.7104 0.8568 0.3774 0.8836 0.7248] 2020-12-02 17:53:55 [INFO] [EVAL] Class Acc: [0.9653 0.9392 0.4588 0.971 0.7984 0.9208 0.982 0.9693 0.969 0.5382 0.9395 0.8825 0.9214 0.9658 0.9144 0.9388 0.8808 0.9632 0.8247 0.9401 0.8692] 2020-12-02 17:54:07 [INFO] [EVAL] The model with the best validation mIoU (0.7843) was saved at iter 8000. 2020-12-02 17:55:28 [INFO] [TRAIN] epoch=16, iter=10100/40000, loss=0.0927, lr=0.007696, batch_cost=0.8042, reader_cost=0.0002 | ETA 06:40:46 2020-12-02 17:56:53 [INFO] [TRAIN] epoch=16, iter=10200/40000, loss=0.1005, lr=0.007673, batch_cost=0.8492, reader_cost=0.0002 | ETA 07:01:46 2020-12-02 17:58:11 [INFO] [TRAIN] epoch=16, iter=10300/40000, loss=0.0864, lr=0.007650, batch_cost=0.7839, reader_cost=0.0002 | ETA 06:28:01 2020-12-02 17:59:29 [INFO] [TRAIN] epoch=16, iter=10400/40000, loss=0.0962, lr=0.007626, batch_cost=0.7813, reader_cost=0.0002 | ETA 06:25:27 2020-12-02 18:00:49 [INFO] [TRAIN] epoch=16, iter=10500/40000, loss=0.0950, lr=0.007603, batch_cost=0.8024, reader_cost=0.0002 | ETA 06:34:30 2020-12-02 18:02:11 [INFO] [TRAIN] epoch=17, iter=10600/40000, loss=0.1066, lr=0.007580, batch_cost=0.8119, reader_cost=0.0127 | ETA 06:37:50 2020-12-02 18:03:33 [INFO] [TRAIN] epoch=17, iter=10700/40000, loss=0.0863, lr=0.007557, batch_cost=0.8231, reader_cost=0.0002 | ETA 06:41:56 2020-12-02 18:04:51 [INFO] [TRAIN] epoch=17, iter=10800/40000, loss=0.0930, lr=0.007534, batch_cost=0.7787, reader_cost=0.0002 | ETA 06:18:58 2020-12-02 18:06:15 [INFO] [TRAIN] epoch=17, iter=10900/40000, loss=0.0940, lr=0.007510, batch_cost=0.8422, reader_cost=0.0016 | ETA 06:48:28 2020-12-02 18:07:40 [INFO] [TRAIN] epoch=17, iter=11000/40000, loss=0.0842, lr=0.007487, batch_cost=0.8505, reader_cost=0.0002 | ETA 06:51:04 2020-12-02 18:09:06 [INFO] [TRAIN] epoch=17, iter=11100/40000, loss=0.0839, lr=0.007464, batch_cost=0.8598, reader_cost=0.0002 | ETA 06:54:08 2020-12-02 18:10:25 [INFO] [TRAIN] epoch=17, iter=11200/40000, loss=0.0859, lr=0.007441, batch_cost=0.7886, reader_cost=0.0002 | ETA 06:18:31 2020-12-02 18:11:44 [INFO] [TRAIN] epoch=18, iter=11300/40000, loss=0.0876, lr=0.007417, batch_cost=0.7862, reader_cost=0.0090 | ETA 06:16:04 2020-12-02 18:13:04 [INFO] [TRAIN] epoch=18, iter=11400/40000, loss=0.0906, lr=0.007394, batch_cost=0.8006, reader_cost=0.0004 | ETA 06:21:36 2020-12-02 18:14:23 [INFO] [TRAIN] epoch=18, iter=11500/40000, loss=0.0822, lr=0.007371, batch_cost=0.7949, reader_cost=0.0002 | ETA 06:17:34 2020-12-02 18:15:46 [INFO] [TRAIN] epoch=18, iter=11600/40000, loss=0.0863, lr=0.007348, batch_cost=0.8288, reader_cost=0.0002 | ETA 06:32:18 2020-12-02 18:17:07 [INFO] [TRAIN] epoch=18, iter=11700/40000, loss=0.0789, lr=0.007324, batch_cost=0.8150, reader_cost=0.0002 | ETA 06:24:25 2020-12-02 18:18:31 [INFO] [TRAIN] epoch=18, iter=11800/40000, loss=0.0885, lr=0.007301, batch_cost=0.8313, reader_cost=0.0002 | ETA 06:30:43 2020-12-02 18:19:49 [INFO] [TRAIN] epoch=19, iter=11900/40000, loss=0.0840, lr=0.007278, batch_cost=0.7834, reader_cost=0.0085 | ETA 06:06:53 2020-12-02 18:21:06 [INFO] [TRAIN] epoch=19, iter=12000/40000, loss=0.0798, lr=0.007254, batch_cost=0.7740, reader_cost=0.0002 | ETA 06:01:11 2020-12-02 18:21:06 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-12-02 18:21:50 [INFO] [EVAL] #Images=1449 mIoU=0.8022 Acc=0.9551 Kappa=0.9005 2020-12-02 18:21:50 [INFO] [EVAL] Class IoU: [0.9475 0.9078 0.4388 0.9066 0.7124 0.8454 0.9633 0.8899 0.9308 0.4182 0.9241 0.582 0.899 0.9045 0.8685 0.877 0.6847 0.9009 0.5771 0.8994 0.7673] 2020-12-02 18:21:50 [INFO] [EVAL] Class Acc: [0.9683 0.9519 0.4556 0.9491 0.7791 0.9292 0.9831 0.9624 0.9637 0.7449 0.964 0.833 0.9471 0.9546 0.9235 0.9503 0.8718 0.9621 0.7632 0.9615 0.9326] 2020-12-02 18:22:08 [INFO] [EVAL] The model with the best validation mIoU (0.8022) was saved at iter 12000. 2020-12-02 18:23:30 [INFO] [TRAIN] epoch=19, iter=12100/40000, loss=0.0796, lr=0.007231, batch_cost=0.8188, reader_cost=0.0003 | ETA 06:20:45 2020-12-02 18:24:51 [INFO] [TRAIN] epoch=19, iter=12200/40000, loss=0.0783, lr=0.007208, batch_cost=0.8084, reader_cost=0.0002 | ETA 06:14:34 2020-12-02 18:26:16 [INFO] [TRAIN] epoch=19, iter=12300/40000, loss=0.0786, lr=0.007184, batch_cost=0.8563, reader_cost=0.0002 | ETA 06:35:19 2020-12-02 18:27:35 [INFO] [TRAIN] epoch=19, iter=12400/40000, loss=0.0788, lr=0.007161, batch_cost=0.7884, reader_cost=0.0002 | ETA 06:02:39 2020-12-02 18:28:58 [INFO] [TRAIN] epoch=19, iter=12500/40000, loss=0.0750, lr=0.007138, batch_cost=0.8231, reader_cost=0.0003 | ETA 06:17:14 2020-12-02 18:30:18 [INFO] [TRAIN] epoch=20, iter=12600/40000, loss=0.0757, lr=0.007114, batch_cost=0.8022, reader_cost=0.0099 | ETA 06:06:20 2020-12-02 18:31:37 [INFO] [TRAIN] epoch=20, iter=12700/40000, loss=0.0739, lr=0.007091, batch_cost=0.7932, reader_cost=0.0002 | ETA 06:00:53 2020-12-02 18:32:56 [INFO] [TRAIN] epoch=20, iter=12800/40000, loss=0.0790, lr=0.007068, batch_cost=0.7864, reader_cost=0.0002 | ETA 05:56:29 2020-12-02 18:34:12 [INFO] [TRAIN] epoch=20, iter=12900/40000, loss=0.0768, lr=0.007044, batch_cost=0.7648, reader_cost=0.0002 | ETA 05:45:24 2020-12-02 18:35:35 [INFO] [TRAIN] epoch=20, iter=13000/40000, loss=0.0713, lr=0.007021, batch_cost=0.8232, reader_cost=0.0002 | ETA 06:10:26 2020-12-02 18:36:55 [INFO] [TRAIN] epoch=20, iter=13100/40000, loss=0.0693, lr=0.006997, batch_cost=0.8050, reader_cost=0.0010 | ETA 06:00:53 2020-12-02 18:38:14 [INFO] [TRAIN] epoch=20, iter=13200/40000, loss=0.0649, lr=0.006974, batch_cost=0.7894, reader_cost=0.0003 | ETA 05:52:34 2020-12-02 18:39:33 [INFO] [TRAIN] epoch=21, iter=13300/40000, loss=0.0691, lr=0.006951, batch_cost=0.7951, reader_cost=0.0078 | ETA 05:53:50 2020-12-02 18:40:52 [INFO] [TRAIN] epoch=21, iter=13400/40000, loss=0.0730, lr=0.006927, batch_cost=0.7806, reader_cost=0.0002 | ETA 05:46:03 2020-12-02 18:42:11 [INFO] [TRAIN] epoch=21, iter=13500/40000, loss=0.0721, lr=0.006904, batch_cost=0.7956, reader_cost=0.0002 | ETA 05:51:22 2020-12-02 18:43:32 [INFO] [TRAIN] epoch=21, iter=13600/40000, loss=0.0630, lr=0.006880, batch_cost=0.8092, reader_cost=0.0002 | ETA 05:56:03 2020-12-02 18:44:55 [INFO] [TRAIN] epoch=21, iter=13700/40000, loss=0.0719, lr=0.006857, batch_cost=0.8282, reader_cost=0.0002 | ETA 06:03:02 2020-12-02 18:46:15 [INFO] [TRAIN] epoch=21, iter=13800/40000, loss=0.0744, lr=0.006833, batch_cost=0.8062, reader_cost=0.0002 | ETA 05:52:02 2020-12-02 18:47:44 [INFO] [TRAIN] epoch=22, iter=13900/40000, loss=0.0683, lr=0.006810, batch_cost=0.8855, reader_cost=0.0090 | ETA 06:25:10 2020-12-02 18:49:05 [INFO] [TRAIN] epoch=22, iter=14000/40000, loss=0.0698, lr=0.006786, batch_cost=0.8062, reader_cost=0.0002 | ETA 05:49:22 2020-12-02 18:49:05 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-12-02 18:49:48 [INFO] [EVAL] #Images=1449 mIoU=0.7931 Acc=0.9532 Kappa=0.8961 2020-12-02 18:49:48 [INFO] [EVAL] Class IoU: [0.9459 0.8919 0.4396 0.9259 0.6857 0.8593 0.9561 0.8779 0.9081 0.4607 0.8892 0.505 0.8777 0.9002 0.8665 0.8828 0.6941 0.8568 0.567 0.8916 0.7733] 2020-12-02 18:49:48 [INFO] [EVAL] Class Acc: [0.9667 0.9273 0.4621 0.9659 0.7625 0.9319 0.9855 0.9414 0.9349 0.6826 0.9318 0.8386 0.9447 0.9555 0.9314 0.9546 0.8592 0.9482 0.8294 0.954 0.9364] 2020-12-02 18:49:58 [INFO] [EVAL] The model with the best validation mIoU (0.8022) was saved at iter 12000. 2020-12-02 18:51:26 [INFO] [TRAIN] epoch=22, iter=14100/40000, loss=0.0714, lr=0.006763, batch_cost=0.8745, reader_cost=0.0002 | ETA 06:17:29 2020-12-02 18:52:59 [INFO] [TRAIN] epoch=22, iter=14200/40000, loss=0.0721, lr=0.006739, batch_cost=0.9282, reader_cost=0.0002 | ETA 06:39:08 2020-12-02 18:54:21 [INFO] [TRAIN] epoch=22, iter=14300/40000, loss=0.0653, lr=0.006716, batch_cost=0.8229, reader_cost=0.0003 | ETA 05:52:27 2020-12-02 18:55:42 [INFO] [TRAIN] epoch=22, iter=14400/40000, loss=0.0661, lr=0.006692, batch_cost=0.8139, reader_cost=0.0002 | ETA 05:47:16 2020-12-02 18:57:11 [INFO] [TRAIN] epoch=22, iter=14500/40000, loss=0.0740, lr=0.006669, batch_cost=0.8924, reader_cost=0.0002 | ETA 06:19:16 2020-12-02 18:58:39 [INFO] [TRAIN] epoch=23, iter=14600/40000, loss=0.0645, lr=0.006645, batch_cost=0.8714, reader_cost=0.0117 | ETA 06:08:52 2020-12-02 19:00:02 [INFO] [TRAIN] epoch=23, iter=14700/40000, loss=0.0658, lr=0.006622, batch_cost=0.8348, reader_cost=0.0005 | ETA 05:52:00 2020-12-02 19:01:22 [INFO] [TRAIN] epoch=23, iter=14800/40000, loss=0.0666, lr=0.006598, batch_cost=0.8003, reader_cost=0.0013 | ETA 05:36:08 2020-12-02 19:02:44 [INFO] [TRAIN] epoch=23, iter=14900/40000, loss=0.0642, lr=0.006575, batch_cost=0.8216, reader_cost=0.0002 | ETA 05:43:42 2020-12-02 19:04:08 [INFO] [TRAIN] epoch=23, iter=15000/40000, loss=0.0644, lr=0.006551, batch_cost=0.8328, reader_cost=0.0002 | ETA 05:46:59 2020-12-02 19:05:28 [INFO] [TRAIN] epoch=23, iter=15100/40000, loss=0.0652, lr=0.006527, batch_cost=0.8011, reader_cost=0.0002 | ETA 05:32:28 2020-12-02 19:06:44 [INFO] [TRAIN] epoch=23, iter=15200/40000, loss=0.0652, lr=0.006504, batch_cost=0.7622, reader_cost=0.0002 | ETA 05:15:01 2020-12-02 19:08:06 [INFO] [TRAIN] epoch=24, iter=15300/40000, loss=0.0634, lr=0.006480, batch_cost=0.8187, reader_cost=0.0085 | ETA 05:37:01 2020-12-02 19:09:27 [INFO] [TRAIN] epoch=24, iter=15400/40000, loss=0.0643, lr=0.006457, batch_cost=0.8169, reader_cost=0.0004 | ETA 05:34:56 2020-12-02 19:10:50 [INFO] [TRAIN] epoch=24, iter=15500/40000, loss=0.0634, lr=0.006433, batch_cost=0.8269, reader_cost=0.0003 | ETA 05:37:39 2020-12-02 19:12:09 [INFO] [TRAIN] epoch=24, iter=15600/40000, loss=0.0645, lr=0.006409, batch_cost=0.7848, reader_cost=0.0002 | ETA 05:19:08 2020-12-02 19:13:35 [INFO] [TRAIN] epoch=24, iter=15700/40000, loss=0.0615, lr=0.006386, batch_cost=0.8605, reader_cost=0.0010 | ETA 05:48:29 2020-12-02 19:14:56 [INFO] [TRAIN] epoch=24, iter=15800/40000, loss=0.0624, lr=0.006362, batch_cost=0.8171, reader_cost=0.0002 | ETA 05:29:34 2020-12-02 19:16:23 [INFO] [TRAIN] epoch=25, iter=15900/40000, loss=0.0601, lr=0.006338, batch_cost=0.8657, reader_cost=0.0133 | ETA 05:47:43 2020-12-02 19:17:45 [INFO] [TRAIN] epoch=25, iter=16000/40000, loss=0.0572, lr=0.006315, batch_cost=0.8163, reader_cost=0.0003 | ETA 05:26:30 2020-12-02 19:17:45 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-12-02 19:18:26 [INFO] [EVAL] #Images=1449 mIoU=0.7936 Acc=0.9537 Kappa=0.8972 2020-12-02 19:18:26 [INFO] [EVAL] Class IoU: [0.9469 0.8927 0.4434 0.9224 0.6682 0.844 0.956 0.8955 0.9082 0.4369 0.8916 0.5491 0.8693 0.8956 0.8661 0.8851 0.7074 0.858 0.5586 0.9026 0.7674] 2020-12-02 19:18:26 [INFO] [EVAL] Class Acc: [0.967 0.9239 0.4621 0.9715 0.7589 0.9334 0.9852 0.9656 0.9321 0.6723 0.9338 0.8506 0.9444 0.9621 0.9233 0.9525 0.8673 0.9463 0.8345 0.9659 0.9199] 2020-12-02 19:18:37 [INFO] [EVAL] The model with the best validation mIoU (0.8022) was saved at iter 12000. 2020-12-02 19:19:56 [INFO] [TRAIN] epoch=25, iter=16100/40000, loss=0.0597, lr=0.006291, batch_cost=0.7949, reader_cost=0.0002 | ETA 05:16:38 2020-12-02 19:21:16 [INFO] [TRAIN] epoch=25, iter=16200/40000, loss=0.0607, lr=0.006267, batch_cost=0.8001, reader_cost=0.0002 | ETA 05:17:21 2020-12-02 19:22:39 [INFO] [TRAIN] epoch=25, iter=16300/40000, loss=0.0683, lr=0.006244, batch_cost=0.8258, reader_cost=0.0002 | ETA 05:26:12 2020-12-02 19:24:07 [INFO] [TRAIN] epoch=25, iter=16400/40000, loss=0.0626, lr=0.006220, batch_cost=0.8855, reader_cost=0.0002 | ETA 05:48:17 2020-12-02 19:25:32 [INFO] [TRAIN] epoch=25, iter=16500/40000, loss=0.0674, lr=0.006196, batch_cost=0.8496, reader_cost=0.0002 | ETA 05:32:44 2020-12-02 19:27:09 [INFO] [TRAIN] epoch=26, iter=16600/40000, loss=0.0596, lr=0.006172, batch_cost=0.9672, reader_cost=0.0105 | ETA 06:17:11 2020-12-02 19:28:30 [INFO] [TRAIN] epoch=26, iter=16700/40000, loss=0.0572, lr=0.006149, batch_cost=0.8132, reader_cost=0.0020 | ETA 05:15:48 2020-12-02 19:29:54 [INFO] [TRAIN] epoch=26, iter=16800/40000, loss=0.0553, lr=0.006125, batch_cost=0.8356, reader_cost=0.0002 | ETA 05:23:05 2020-12-02 19:31:22 [INFO] [TRAIN] epoch=26, iter=16900/40000, loss=0.0616, lr=0.006101, batch_cost=0.8832, reader_cost=0.0002 | ETA 05:40:01 2020-12-02 19:32:51 [INFO] [TRAIN] epoch=26, iter=17000/40000, loss=0.0610, lr=0.006077, batch_cost=0.8871, reader_cost=0.0003 | ETA 05:40:03 2020-12-02 19:34:14 [INFO] [TRAIN] epoch=26, iter=17100/40000, loss=0.0632, lr=0.006054, batch_cost=0.8315, reader_cost=0.0002 | ETA 05:17:20 2020-12-02 19:35:38 [INFO] [TRAIN] epoch=27, iter=17200/40000, loss=0.0597, lr=0.006030, batch_cost=0.8380, reader_cost=0.0088 | ETA 05:18:26 2020-12-02 19:36:58 [INFO] [TRAIN] epoch=27, iter=17300/40000, loss=0.0547, lr=0.006006, batch_cost=0.8067, reader_cost=0.0002 | ETA 05:05:12 2020-12-02 19:38:21 [INFO] [TRAIN] epoch=27, iter=17400/40000, loss=0.0562, lr=0.005982, batch_cost=0.8260, reader_cost=0.0002 | ETA 05:11:07 2020-12-02 19:39:43 [INFO] [TRAIN] epoch=27, iter=17500/40000, loss=0.0568, lr=0.005958, batch_cost=0.8163, reader_cost=0.0002 | ETA 05:06:06 2020-12-02 19:41:01 [INFO] [TRAIN] epoch=27, iter=17600/40000, loss=0.0583, lr=0.005935, batch_cost=0.7784, reader_cost=0.0003 | ETA 04:50:36 2020-12-02 19:42:26 [INFO] [TRAIN] epoch=27, iter=17700/40000, loss=0.0566, lr=0.005911, batch_cost=0.8525, reader_cost=0.0003 | ETA 05:16:50 2020-12-02 19:43:55 [INFO] [TRAIN] epoch=27, iter=17800/40000, loss=0.0550, lr=0.005887, batch_cost=0.8959, reader_cost=0.0002 | ETA 05:31:29 2020-12-02 19:45:17 [INFO] [TRAIN] epoch=28, iter=17900/40000, loss=0.0574, lr=0.005863, batch_cost=0.8181, reader_cost=0.0095 | ETA 05:01:18 2020-12-02 19:46:40 [INFO] [TRAIN] epoch=28, iter=18000/40000, loss=0.0545, lr=0.005839, batch_cost=0.8247, reader_cost=0.0002 | ETA 05:02:23 2020-12-02 19:46:40 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-12-02 19:47:21 [INFO] [EVAL] #Images=1449 mIoU=0.7970 Acc=0.9542 Kappa=0.8984 2020-12-02 19:47:21 [INFO] [EVAL] Class IoU: [0.9469 0.9004 0.4403 0.9219 0.6701 0.8428 0.9501 0.9013 0.9174 0.4453 0.9036 0.5889 0.8813 0.8818 0.8622 0.8798 0.6959 0.884 0.5585 0.8905 0.7731] 2020-12-02 19:47:21 [INFO] [EVAL] Class Acc: [0.9667 0.939 0.4618 0.9654 0.7674 0.9277 0.9825 0.9669 0.9475 0.7165 0.9701 0.8499 0.9425 0.9274 0.9242 0.9519 0.8809 0.9454 0.829 0.9587 0.9165] 2020-12-02 19:47:31 [INFO] [EVAL] The model with the best validation mIoU (0.8022) was saved at iter 12000. 2020-12-02 19:48:55 [INFO] [TRAIN] epoch=28, iter=18100/40000, loss=0.0537, lr=0.005815, batch_cost=0.8390, reader_cost=0.0003 | ETA 05:06:14 2020-12-02 19:50:18 [INFO] [TRAIN] epoch=28, iter=18200/40000, loss=0.0576, lr=0.005791, batch_cost=0.8315, reader_cost=0.0002 | ETA 05:02:05 2020-12-02 19:51:46 [INFO] [TRAIN] epoch=28, iter=18300/40000, loss=0.0604, lr=0.005767, batch_cost=0.8733, reader_cost=0.0002 | ETA 05:15:49 2020-12-02 19:53:09 [INFO] [TRAIN] epoch=28, iter=18400/40000, loss=0.0569, lr=0.005743, batch_cost=0.8317, reader_cost=0.0002 | ETA 04:59:24 2020-12-02 19:54:35 [INFO] [TRAIN] epoch=28, iter=18500/40000, loss=0.0571, lr=0.005720, batch_cost=0.8624, reader_cost=0.0003 | ETA 05:09:01 2020-12-02 19:56:14 [INFO] [TRAIN] epoch=29, iter=18600/40000, loss=0.0550, lr=0.005696, batch_cost=0.9882, reader_cost=0.0109 | ETA 05:52:26 2020-12-02 19:57:34 [INFO] [TRAIN] epoch=29, iter=18700/40000, loss=0.0525, lr=0.005672, batch_cost=0.8007, reader_cost=0.0002 | ETA 04:44:13 2020-12-02 19:58:54 [INFO] [TRAIN] epoch=29, iter=18800/40000, loss=0.0524, lr=0.005648, batch_cost=0.8002, reader_cost=0.0002 | ETA 04:42:44 2020-12-02 20:00:17 [INFO] [TRAIN] epoch=29, iter=18900/40000, loss=0.0563, lr=0.005624, batch_cost=0.8334, reader_cost=0.0002 | ETA 04:53:05 2020-12-02 20:01:50 [INFO] [TRAIN] epoch=29, iter=19000/40000, loss=0.0562, lr=0.005600, batch_cost=0.9303, reader_cost=0.0002 | ETA 05:25:36 2020-12-02 20:03:15 [INFO] [TRAIN] epoch=29, iter=19100/40000, loss=0.0515, lr=0.005576, batch_cost=0.8497, reader_cost=0.0003 | ETA 04:55:58 2020-12-02 20:04:36 [INFO] [TRAIN] epoch=30, iter=19200/40000, loss=0.0605, lr=0.005552, batch_cost=0.8012, reader_cost=0.0074 | ETA 04:37:45 2020-12-02 20:05:54 [INFO] [TRAIN] epoch=30, iter=19300/40000, loss=0.0522, lr=0.005528, batch_cost=0.7882, reader_cost=0.0002 | ETA 04:31:56 2020-12-02 20:07:15 [INFO] [TRAIN] epoch=30, iter=19400/40000, loss=0.0558, lr=0.005504, batch_cost=0.8030, reader_cost=0.0002 | ETA 04:35:41 2020-12-02 20:08:37 [INFO] [TRAIN] epoch=30, iter=19500/40000, loss=0.0542, lr=0.005480, batch_cost=0.8236, reader_cost=0.0002 | ETA 04:41:24 2020-12-02 20:09:59 [INFO] [TRAIN] epoch=30, iter=19600/40000, loss=0.0546, lr=0.005455, batch_cost=0.8166, reader_cost=0.0009 | ETA 04:37:39 2020-12-02 20:11:21 [INFO] [TRAIN] epoch=30, iter=19700/40000, loss=0.0576, lr=0.005431, batch_cost=0.8205, reader_cost=0.0002 | ETA 04:37:35 2020-12-02 20:12:48 [INFO] [TRAIN] epoch=30, iter=19800/40000, loss=0.0560, lr=0.005407, batch_cost=0.8735, reader_cost=0.0002 | ETA 04:54:04 2020-12-02 20:14:14 [INFO] [TRAIN] epoch=31, iter=19900/40000, loss=0.0532, lr=0.005383, batch_cost=0.8544, reader_cost=0.0110 | ETA 04:46:13 2020-12-02 20:15:35 [INFO] [TRAIN] epoch=31, iter=20000/40000, loss=0.0523, lr=0.005359, batch_cost=0.8136, reader_cost=0.0002 | ETA 04:31:11 2020-12-02 20:15:35 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-12-02 20:16:13 [INFO] [EVAL] #Images=1449 mIoU=0.7979 Acc=0.9541 Kappa=0.8981 2020-12-02 20:16:13 [INFO] [EVAL] Class IoU: [0.9469 0.8892 0.4403 0.9217 0.6832 0.8319 0.9572 0.8935 0.9081 0.4551 0.9141 0.5559 0.8682 0.9043 0.8651 0.8822 0.7115 0.898 0.5495 0.9024 0.7776] 2020-12-02 20:16:13 [INFO] [EVAL] Class Acc: [0.967 0.9248 0.4576 0.9654 0.7978 0.9093 0.9809 0.9608 0.9383 0.6649 0.9741 0.8586 0.9436 0.9612 0.9184 0.9534 0.8554 0.9553 0.852 0.9539 0.9211] 2020-12-02 20:16:23 [INFO] [EVAL] The model with the best validation mIoU (0.8022) was saved at iter 12000. 2020-12-02 20:17:41 [INFO] [TRAIN] epoch=31, iter=20100/40000, loss=0.0519, lr=0.005335, batch_cost=0.7813, reader_cost=0.0002 | ETA 04:19:08 2020-12-02 20:19:12 [INFO] [TRAIN] epoch=31, iter=20200/40000, loss=0.0560, lr=0.005311, batch_cost=0.9066, reader_cost=0.0003 | ETA 04:59:10 2020-12-02 20:20:37 [INFO] [TRAIN] epoch=31, iter=20300/40000, loss=0.0546, lr=0.005287, batch_cost=0.8556, reader_cost=0.0003 | ETA 04:40:54 2020-12-02 20:21:58 [INFO] [TRAIN] epoch=31, iter=20400/40000, loss=0.0540, lr=0.005263, batch_cost=0.8053, reader_cost=0.0002 | ETA 04:23:02 2020-12-02 20:23:23 [INFO] [TRAIN] epoch=32, iter=20500/40000, loss=0.0528, lr=0.005238, batch_cost=0.8491, reader_cost=0.0087 | ETA 04:35:57 2020-12-02 20:24:46 [INFO] [TRAIN] epoch=32, iter=20600/40000, loss=0.0493, lr=0.005214, batch_cost=0.8347, reader_cost=0.0003 | ETA 04:29:52 2020-12-02 20:26:06 [INFO] [TRAIN] epoch=32, iter=20700/40000, loss=0.0517, lr=0.005190, batch_cost=0.7924, reader_cost=0.0003 | ETA 04:14:53 2020-12-02 20:27:23 [INFO] [TRAIN] epoch=32, iter=20800/40000, loss=0.0504, lr=0.005166, batch_cost=0.7784, reader_cost=0.0002 | ETA 04:09:04 2020-12-02 20:28:45 [INFO] [TRAIN] epoch=32, iter=20900/40000, loss=0.0494, lr=0.005142, batch_cost=0.8153, reader_cost=0.0002 | ETA 04:19:31 2020-12-02 20:30:11 [INFO] [TRAIN] epoch=32, iter=21000/40000, loss=0.0522, lr=0.005117, batch_cost=0.8550, reader_cost=0.0003 | ETA 04:30:44 2020-12-02 20:31:33 [INFO] [TRAIN] epoch=32, iter=21100/40000, loss=0.0511, lr=0.005093, batch_cost=0.8246, reader_cost=0.0002 | ETA 04:19:44 2020-12-02 20:32:57 [INFO] [TRAIN] epoch=33, iter=21200/40000, loss=0.0515, lr=0.005069, batch_cost=0.8385, reader_cost=0.0101 | ETA 04:22:44 2020-12-02 20:34:20 [INFO] [TRAIN] epoch=33, iter=21300/40000, loss=0.0517, lr=0.005045, batch_cost=0.8305, reader_cost=0.0002 | ETA 04:18:50 2020-12-02 20:35:42 [INFO] [TRAIN] epoch=33, iter=21400/40000, loss=0.0515, lr=0.005020, batch_cost=0.8168, reader_cost=0.0002 | ETA 04:13:12 2020-12-02 20:37:05 [INFO] [TRAIN] epoch=33, iter=21500/40000, loss=0.0500, lr=0.004996, batch_cost=0.8351, reader_cost=0.0002 | ETA 04:17:29 2020-12-02 20:38:29 [INFO] [TRAIN] epoch=33, iter=21600/40000, loss=0.0515, lr=0.004972, batch_cost=0.8357, reader_cost=0.0002 | ETA 04:16:16 2020-12-02 20:39:53 [INFO] [TRAIN] epoch=33, iter=21700/40000, loss=0.0542, lr=0.004947, batch_cost=0.8449, reader_cost=0.0002 | ETA 04:17:41 2020-12-02 20:41:15 [INFO] [TRAIN] epoch=33, iter=21800/40000, loss=0.0539, lr=0.004923, batch_cost=0.8162, reader_cost=0.0003 | ETA 04:07:34 2020-12-02 20:42:36 [INFO] [TRAIN] epoch=34, iter=21900/40000, loss=0.0478, lr=0.004899, batch_cost=0.8086, reader_cost=0.0096 | ETA 04:03:56 2020-12-02 20:43:57 [INFO] [TRAIN] epoch=34, iter=22000/40000, loss=0.0521, lr=0.004874, batch_cost=0.8157, reader_cost=0.0002 | ETA 04:04:42 2020-12-02 20:43:57 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-12-02 20:44:37 [INFO] [EVAL] #Images=1449 mIoU=0.7945 Acc=0.9539 Kappa=0.8970 2020-12-02 20:44:37 [INFO] [EVAL] Class IoU: [0.9464 0.899 0.4424 0.9208 0.6587 0.8249 0.9549 0.8908 0.9098 0.4683 0.9193 0.5638 0.8752 0.8967 0.8779 0.8835 0.6726 0.8792 0.5481 0.8833 0.7682] 2020-12-02 20:44:37 [INFO] [EVAL] Class Acc: [0.9643 0.9471 0.4637 0.9738 0.7666 0.932 0.9879 0.9686 0.941 0.692 0.9738 0.8583 0.9495 0.9587 0.9307 0.9544 0.8789 0.9513 0.8621 0.969 0.9301] 2020-12-02 20:44:50 [INFO] [EVAL] The model with the best validation mIoU (0.8022) was saved at iter 12000. 2020-12-02 20:46:19 [INFO] [TRAIN] epoch=34, iter=22100/40000, loss=0.0489, lr=0.004850, batch_cost=0.8863, reader_cost=0.0002 | ETA 04:24:24 2020-12-02 20:47:36 [INFO] [TRAIN] epoch=34, iter=22200/40000, loss=0.0499, lr=0.004826, batch_cost=0.7695, reader_cost=0.0002 | ETA 03:48:17 2020-12-02 20:49:01 [INFO] [TRAIN] epoch=34, iter=22300/40000, loss=0.0518, lr=0.004801, batch_cost=0.8477, reader_cost=0.0002 | ETA 04:10:04 2020-12-02 20:50:22 [INFO] [TRAIN] epoch=34, iter=22400/40000, loss=0.0518, lr=0.004777, batch_cost=0.8122, reader_cost=0.0002 | ETA 03:58:14 2020-12-02 20:51:45 [INFO] [TRAIN] epoch=35, iter=22500/40000, loss=0.0499, lr=0.004752, batch_cost=0.8282, reader_cost=0.0104 | ETA 04:01:33 2020-12-02 20:53:01 [INFO] [TRAIN] epoch=35, iter=22600/40000, loss=0.0509, lr=0.004728, batch_cost=0.7593, reader_cost=0.0002 | ETA 03:40:12 2020-12-02 20:54:26 [INFO] [TRAIN] epoch=35, iter=22700/40000, loss=0.0459, lr=0.004703, batch_cost=0.8510, reader_cost=0.0015 | ETA 04:05:21 2020-12-02 20:55:47 [INFO] [TRAIN] epoch=35, iter=22800/40000, loss=0.0475, lr=0.004679, batch_cost=0.8116, reader_cost=0.0003 | ETA 03:52:38 2020-12-02 20:57:20 [INFO] [TRAIN] epoch=35, iter=22900/40000, loss=0.0478, lr=0.004654, batch_cost=0.9359, reader_cost=0.0010 | ETA 04:26:44 2020-12-02 20:58:48 [INFO] [TRAIN] epoch=35, iter=23000/40000, loss=0.0514, lr=0.004630, batch_cost=0.8763, reader_cost=0.0002 | ETA 04:08:16 2020-12-02 21:00:12 [INFO] [TRAIN] epoch=35, iter=23100/40000, loss=0.0649, lr=0.004605, batch_cost=0.8361, reader_cost=0.0002 | ETA 03:55:29 2020-12-02 21:01:35 [INFO] [TRAIN] epoch=36, iter=23200/40000, loss=0.0532, lr=0.004581, batch_cost=0.8347, reader_cost=0.0094 | ETA 03:53:42 2020-12-02 21:03:04 [INFO] [TRAIN] epoch=36, iter=23300/40000, loss=0.0537, lr=0.004556, batch_cost=0.8892, reader_cost=0.0016 | ETA 04:07:29 2020-12-02 21:04:29 [INFO] [TRAIN] epoch=36, iter=23400/40000, loss=0.0534, lr=0.004532, batch_cost=0.8447, reader_cost=0.0002 | ETA 03:53:41 2020-12-02 21:05:51 [INFO] [TRAIN] epoch=36, iter=23500/40000, loss=0.0499, lr=0.004507, batch_cost=0.8246, reader_cost=0.0004 | ETA 03:46:45 2020-12-02 21:07:12 [INFO] [TRAIN] epoch=36, iter=23600/40000, loss=0.0488, lr=0.004483, batch_cost=0.8124, reader_cost=0.0003 | ETA 03:42:02 2020-12-02 21:08:33 [INFO] [TRAIN] epoch=36, iter=23700/40000, loss=0.0488, lr=0.004458, batch_cost=0.8038, reader_cost=0.0002 | ETA 03:38:22 2020-12-02 21:09:52 [INFO] [TRAIN] epoch=37, iter=23800/40000, loss=0.0529, lr=0.004433, batch_cost=0.7930, reader_cost=0.0086 | ETA 03:34:05 2020-12-02 21:11:10 [INFO] [TRAIN] epoch=37, iter=23900/40000, loss=0.0490, lr=0.004409, batch_cost=0.7780, reader_cost=0.0002 | ETA 03:28:45 2020-12-02 21:12:29 [INFO] [TRAIN] epoch=37, iter=24000/40000, loss=0.0482, lr=0.004384, batch_cost=0.7890, reader_cost=0.0002 | ETA 03:30:23 2020-12-02 21:12:29 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-12-02 21:13:09 [INFO] [EVAL] #Images=1449 mIoU=0.7943 Acc=0.9541 Kappa=0.8978 2020-12-02 21:13:09 [INFO] [EVAL] Class IoU: [0.9469 0.8936 0.4415 0.9231 0.6651 0.8334 0.9541 0.8897 0.9173 0.4526 0.8846 0.5771 0.8823 0.8737 0.8604 0.8844 0.6972 0.8711 0.5572 0.8815 0.7945] 2020-12-02 21:13:09 [INFO] [EVAL] Class Acc: [0.9655 0.9392 0.4652 0.9702 0.7831 0.9143 0.9877 0.9643 0.9517 0.7447 0.9541 0.8592 0.9471 0.9338 0.9124 0.9578 0.8707 0.9593 0.8392 0.9405 0.921 ] 2020-12-02 21:13:20 [INFO] [EVAL] The model with the best validation mIoU (0.8022) was saved at iter 12000. 2020-12-02 21:14:42 [INFO] [TRAIN] epoch=37, iter=24100/40000, loss=0.0476, lr=0.004359, batch_cost=0.8272, reader_cost=0.0003 | ETA 03:39:13 2020-12-02 21:16:01 [INFO] [TRAIN] epoch=37, iter=24200/40000, loss=0.0475, lr=0.004335, batch_cost=0.7867, reader_cost=0.0002 | ETA 03:27:10 2020-12-02 21:17:25 [INFO] [TRAIN] epoch=37, iter=24300/40000, loss=0.0500, lr=0.004310, batch_cost=0.8374, reader_cost=0.0008 | ETA 03:39:07 2020-12-02 21:18:48 [INFO] [TRAIN] epoch=37, iter=24400/40000, loss=0.0476, lr=0.004285, batch_cost=0.8369, reader_cost=0.0002 | ETA 03:37:35 2020-12-02 21:20:17 [INFO] [TRAIN] epoch=38, iter=24500/40000, loss=0.0486, lr=0.004261, batch_cost=0.8851, reader_cost=0.0115 | ETA 03:48:38 2020-12-02 21:21:38 [INFO] [TRAIN] epoch=38, iter=24600/40000, loss=0.0458, lr=0.004236, batch_cost=0.8095, reader_cost=0.0002 | ETA 03:27:46 2020-12-02 21:23:01 [INFO] [TRAIN] epoch=38, iter=24700/40000, loss=0.0476, lr=0.004211, batch_cost=0.8280, reader_cost=0.0002 | ETA 03:31:08 2020-12-02 21:24:20 [INFO] [TRAIN] epoch=38, iter=24800/40000, loss=0.0472, lr=0.004186, batch_cost=0.7978, reader_cost=0.0002 | ETA 03:22:07 2020-12-02 21:25:39 [INFO] [TRAIN] epoch=38, iter=24900/40000, loss=0.0471, lr=0.004162, batch_cost=0.7825, reader_cost=0.0002 | ETA 03:16:55 2020-12-02 21:26:57 [INFO] [TRAIN] epoch=38, iter=25000/40000, loss=0.0460, lr=0.004137, batch_cost=0.7872, reader_cost=0.0002 | ETA 03:16:47 2020-12-02 21:28:22 [INFO] [TRAIN] epoch=38, iter=25100/40000, loss=0.0495, lr=0.004112, batch_cost=0.8486, reader_cost=0.0002 | ETA 03:30:44 2020-12-02 21:29:48 [INFO] [TRAIN] epoch=39, iter=25200/40000, loss=0.0443, lr=0.004087, batch_cost=0.8610, reader_cost=0.0088 | ETA 03:32:22 2020-12-02 21:31:08 [INFO] [TRAIN] epoch=39, iter=25300/40000, loss=0.0464, lr=0.004062, batch_cost=0.7931, reader_cost=0.0002 | ETA 03:14:19 2020-12-02 21:32:26 [INFO] [TRAIN] epoch=39, iter=25400/40000, loss=0.0469, lr=0.004037, batch_cost=0.7857, reader_cost=0.0002 | ETA 03:11:11 2020-12-02 21:33:53 [INFO] [TRAIN] epoch=39, iter=25500/40000, loss=0.0470, lr=0.004012, batch_cost=0.8620, reader_cost=0.0002 | ETA 03:28:19 2020-12-02 21:35:16 [INFO] [TRAIN] epoch=39, iter=25600/40000, loss=0.0443, lr=0.003987, batch_cost=0.8318, reader_cost=0.0002 | ETA 03:19:37 2020-12-02 21:36:40 [INFO] [TRAIN] epoch=39, iter=25700/40000, loss=0.0468, lr=0.003963, batch_cost=0.8479, reader_cost=0.0002 | ETA 03:22:05 2020-12-02 21:38:04 [INFO] [TRAIN] epoch=40, iter=25800/40000, loss=0.0480, lr=0.003938, batch_cost=0.8305, reader_cost=0.0086 | ETA 03:16:33 2020-12-02 21:39:21 [INFO] [TRAIN] epoch=40, iter=25900/40000, loss=0.0453, lr=0.003913, batch_cost=0.7786, reader_cost=0.0002 | ETA 03:02:57 2020-12-02 21:40:46 [INFO] [TRAIN] epoch=40, iter=26000/40000, loss=0.0459, lr=0.003888, batch_cost=0.8441, reader_cost=0.0002 | ETA 03:16:57 2020-12-02 21:40:46 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-12-02 21:41:22 [INFO] [EVAL] #Images=1449 mIoU=0.7944 Acc=0.9537 Kappa=0.8968 2020-12-02 21:41:22 [INFO] [EVAL] Class IoU: [0.9462 0.8988 0.44 0.9215 0.6631 0.836 0.9532 0.8922 0.9124 0.465 0.9087 0.556 0.8751 0.8767 0.8612 0.8825 0.6806 0.8838 0.557 0.8804 0.7928] 2020-12-02 21:41:22 [INFO] [EVAL] Class Acc: [0.9652 0.9412 0.4591 0.9658 0.7616 0.9242 0.9867 0.9598 0.9463 0.7245 0.9759 0.883 0.9462 0.936 0.9173 0.9564 0.8708 0.9467 0.8288 0.9474 0.936 ] 2020-12-02 21:41:33 [INFO] [EVAL] The model with the best validation mIoU (0.8022) was saved at iter 12000. 2020-12-02 21:42:54 [INFO] [TRAIN] epoch=40, iter=26100/40000, loss=0.0473, lr=0.003863, batch_cost=0.8050, reader_cost=0.0002 | ETA 03:06:29 2020-12-02 21:44:23 [INFO] [TRAIN] epoch=40, iter=26200/40000, loss=0.0446, lr=0.003838, batch_cost=0.8889, reader_cost=0.0005 | ETA 03:24:26 2020-12-02 21:45:43 [INFO] [TRAIN] epoch=40, iter=26300/40000, loss=0.0439, lr=0.003813, batch_cost=0.8046, reader_cost=0.0002 | ETA 03:03:43 2020-12-02 21:47:02 [INFO] [TRAIN] epoch=40, iter=26400/40000, loss=0.0452, lr=0.003788, batch_cost=0.7881, reader_cost=0.0002 | ETA 02:58:37 2020-12-02 21:48:23 [INFO] [TRAIN] epoch=41, iter=26500/40000, loss=0.0436, lr=0.003762, batch_cost=0.8055, reader_cost=0.0103 | ETA 03:01:14 2020-12-02 21:49:49 [INFO] [TRAIN] epoch=41, iter=26600/40000, loss=0.0451, lr=0.003737, batch_cost=0.8629, reader_cost=0.0002 | ETA 03:12:43 2020-12-02 21:51:09 [INFO] [TRAIN] epoch=41, iter=26700/40000, loss=0.0484, lr=0.003712, batch_cost=0.8002, reader_cost=0.0002 | ETA 02:57:22 2020-12-02 21:52:34 [INFO] [TRAIN] epoch=41, iter=26800/40000, loss=0.0467, lr=0.003687, batch_cost=0.8471, reader_cost=0.0002 | ETA 03:06:21 2020-12-02 21:53:55 [INFO] [TRAIN] epoch=41, iter=26900/40000, loss=0.0459, lr=0.003662, batch_cost=0.8134, reader_cost=0.0002 | ETA 02:57:35 2020-12-02 21:55:15 [INFO] [TRAIN] epoch=41, iter=27000/40000, loss=0.0447, lr=0.003637, batch_cost=0.8037, reader_cost=0.0004 | ETA 02:54:08 2020-12-02 21:56:38 [INFO] [TRAIN] epoch=41, iter=27100/40000, loss=0.0453, lr=0.003612, batch_cost=0.8253, reader_cost=0.0002 | ETA 02:57:26 2020-12-02 21:58:07 [INFO] [TRAIN] epoch=42, iter=27200/40000, loss=0.0447, lr=0.003586, batch_cost=0.8850, reader_cost=0.0095 | ETA 03:08:48 2020-12-02 21:59:32 [INFO] [TRAIN] epoch=42, iter=27300/40000, loss=0.0449, lr=0.003561, batch_cost=0.8575, reader_cost=0.0002 | ETA 03:01:29 2020-12-02 22:00:55 [INFO] [TRAIN] epoch=42, iter=27400/40000, loss=0.0462, lr=0.003536, batch_cost=0.8272, reader_cost=0.0008 | ETA 02:53:42 2020-12-02 22:02:16 [INFO] [TRAIN] epoch=42, iter=27500/40000, loss=0.0466, lr=0.003511, batch_cost=0.8063, reader_cost=0.0002 | ETA 02:47:59 2020-12-02 22:03:47 [INFO] [TRAIN] epoch=42, iter=27600/40000, loss=0.0427, lr=0.003485, batch_cost=0.9171, reader_cost=0.0002 | ETA 03:09:32 2020-12-02 22:05:11 [INFO] [TRAIN] epoch=42, iter=27700/40000, loss=0.0445, lr=0.003460, batch_cost=0.8339, reader_cost=0.0003 | ETA 02:50:56 2020-12-02 22:06:38 [INFO] [TRAIN] epoch=43, iter=27800/40000, loss=0.0452, lr=0.003435, batch_cost=0.8733, reader_cost=0.0096 | ETA 02:57:34 2020-12-02 22:07:59 [INFO] [TRAIN] epoch=43, iter=27900/40000, loss=0.0441, lr=0.003409, batch_cost=0.8097, reader_cost=0.0002 | ETA 02:43:17 2020-12-02 22:09:19 [INFO] [TRAIN] epoch=43, iter=28000/40000, loss=0.0437, lr=0.003384, batch_cost=0.7968, reader_cost=0.0002 | ETA 02:39:21 2020-12-02 22:09:19 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-12-02 22:09:59 [INFO] [EVAL] #Images=1449 mIoU=0.7943 Acc=0.9536 Kappa=0.8963 2020-12-02 22:09:59 [INFO] [EVAL] Class IoU: [0.9457 0.8868 0.4444 0.9251 0.6751 0.8379 0.9517 0.8945 0.9113 0.4625 0.9144 0.5522 0.8732 0.8904 0.859 0.8822 0.6873 0.8862 0.5449 0.8707 0.7848] 2020-12-02 22:09:59 [INFO] [EVAL] Class Acc: [0.9636 0.9302 0.4683 0.9699 0.7944 0.9302 0.9864 0.9656 0.9451 0.7406 0.974 0.9031 0.9438 0.9525 0.9114 0.9554 0.8674 0.9474 0.8449 0.9495 0.9405] 2020-12-02 22:10:10 [INFO] [EVAL] The model with the best validation mIoU (0.8022) was saved at iter 12000. 2020-12-02 22:11:31 [INFO] [TRAIN] epoch=43, iter=28100/40000, loss=0.0451, lr=0.003359, batch_cost=0.8147, reader_cost=0.0002 | ETA 02:41:34 2020-12-02 22:12:53 [INFO] [TRAIN] epoch=43, iter=28200/40000, loss=0.0442, lr=0.003333, batch_cost=0.8158, reader_cost=0.0003 | ETA 02:40:26 2020-12-02 22:14:11 [INFO] [TRAIN] epoch=43, iter=28300/40000, loss=0.0479, lr=0.003308, batch_cost=0.7822, reader_cost=0.0002 | ETA 02:32:31 2020-12-02 22:15:37 [INFO] [TRAIN] epoch=43, iter=28400/40000, loss=0.0448, lr=0.003282, batch_cost=0.8566, reader_cost=0.0002 | ETA 02:45:36 2020-12-02 22:17:03 [INFO] [TRAIN] epoch=44, iter=28500/40000, loss=0.0415, lr=0.003257, batch_cost=0.8627, reader_cost=0.0085 | ETA 02:45:21 2020-12-02 22:18:25 [INFO] [TRAIN] epoch=44, iter=28600/40000, loss=0.0460, lr=0.003231, batch_cost=0.8190, reader_cost=0.0003 | ETA 02:35:36 2020-12-02 22:19:50 [INFO] [TRAIN] epoch=44, iter=28700/40000, loss=0.0434, lr=0.003206, batch_cost=0.8449, reader_cost=0.0005 | ETA 02:39:07 2020-12-02 22:21:11 [INFO] [TRAIN] epoch=44, iter=28800/40000, loss=0.0434, lr=0.003180, batch_cost=0.8141, reader_cost=0.0002 | ETA 02:31:58 2020-12-02 22:22:36 [INFO] [TRAIN] epoch=44, iter=28900/40000, loss=0.0442, lr=0.003155, batch_cost=0.8493, reader_cost=0.0002 | ETA 02:37:07 2020-12-02 22:24:00 [INFO] [TRAIN] epoch=44, iter=29000/40000, loss=0.0458, lr=0.003129, batch_cost=0.8390, reader_cost=0.0002 | ETA 02:33:49 2020-12-02 22:25:22 [INFO] [TRAIN] epoch=45, iter=29100/40000, loss=0.0435, lr=0.003104, batch_cost=0.8191, reader_cost=0.0104 | ETA 02:28:48 2020-12-02 22:26:43 [INFO] [TRAIN] epoch=45, iter=29200/40000, loss=0.0429, lr=0.003078, batch_cost=0.8081, reader_cost=0.0002 | ETA 02:25:27 2020-12-02 22:28:02 [INFO] [TRAIN] epoch=45, iter=29300/40000, loss=0.0436, lr=0.003052, batch_cost=0.7902, reader_cost=0.0002 | ETA 02:20:55 2020-12-02 22:29:25 [INFO] [TRAIN] epoch=45, iter=29400/40000, loss=0.0438, lr=0.003027, batch_cost=0.8333, reader_cost=0.0002 | ETA 02:27:12 2020-12-02 22:30:50 [INFO] [TRAIN] epoch=45, iter=29500/40000, loss=0.0419, lr=0.003001, batch_cost=0.8504, reader_cost=0.0002 | ETA 02:28:49 2020-12-02 22:32:14 [INFO] [TRAIN] epoch=45, iter=29600/40000, loss=0.0435, lr=0.002975, batch_cost=0.8413, reader_cost=0.0004 | ETA 02:25:49 2020-12-02 22:33:34 [INFO] [TRAIN] epoch=45, iter=29700/40000, loss=0.0433, lr=0.002949, batch_cost=0.7947, reader_cost=0.0002 | ETA 02:16:25 2020-12-02 22:34:55 [INFO] [TRAIN] epoch=46, iter=29800/40000, loss=0.0411, lr=0.002924, batch_cost=0.8138, reader_cost=0.0103 | ETA 02:18:21 2020-12-02 22:36:16 [INFO] [TRAIN] epoch=46, iter=29900/40000, loss=0.0438, lr=0.002898, batch_cost=0.8101, reader_cost=0.0002 | ETA 02:16:21 2020-12-02 22:37:37 [INFO] [TRAIN] epoch=46, iter=30000/40000, loss=0.0429, lr=0.002872, batch_cost=0.8090, reader_cost=0.0002 | ETA 02:14:49 2020-12-02 22:37:37 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-12-02 22:38:10 [INFO] [EVAL] #Images=1449 mIoU=0.7949 Acc=0.9537 Kappa=0.8969 2020-12-02 22:38:10 [INFO] [EVAL] Class IoU: [0.9463 0.893 0.4449 0.9248 0.6675 0.8352 0.954 0.8893 0.9055 0.4817 0.9145 0.5456 0.8647 0.8863 0.8648 0.8832 0.6813 0.8824 0.5578 0.8778 0.7926] 2020-12-02 22:38:10 [INFO] [EVAL] Class Acc: [0.9651 0.9368 0.4688 0.9673 0.7764 0.9245 0.9872 0.9634 0.9383 0.7158 0.9722 0.8847 0.9408 0.9524 0.9175 0.9535 0.8754 0.9485 0.8412 0.9507 0.9307] 2020-12-02 22:38:20 [INFO] [EVAL] The model with the best validation mIoU (0.8022) was saved at iter 12000. 2020-12-02 22:39:37 [INFO] [TRAIN] epoch=46, iter=30100/40000, loss=0.0444, lr=0.002846, batch_cost=0.7638, reader_cost=0.0002 | ETA 02:06:01 2020-12-02 22:40:54 [INFO] [TRAIN] epoch=46, iter=30200/40000, loss=0.0399, lr=0.002820, batch_cost=0.7782, reader_cost=0.0002 | ETA 02:07:06 2020-12-02 22:42:14 [INFO] [TRAIN] epoch=46, iter=30300/40000, loss=0.0446, lr=0.002794, batch_cost=0.7937, reader_cost=0.0002 | ETA 02:08:19 2020-12-02 22:43:36 [INFO] [TRAIN] epoch=46, iter=30400/40000, loss=0.0425, lr=0.002768, batch_cost=0.8193, reader_cost=0.0002 | ETA 02:11:05 2020-12-02 22:44:58 [INFO] [TRAIN] epoch=47, iter=30500/40000, loss=0.0428, lr=0.002742, batch_cost=0.8199, reader_cost=0.0103 | ETA 02:09:49 2020-12-02 22:46:21 [INFO] [TRAIN] epoch=47, iter=30600/40000, loss=0.0405, lr=0.002716, batch_cost=0.8345, reader_cost=0.0002 | ETA 02:10:44 2020-12-02 22:47:42 [INFO] [TRAIN] epoch=47, iter=30700/40000, loss=0.0427, lr=0.002690, batch_cost=0.8118, reader_cost=0.0002 | ETA 02:05:49 2020-12-02 22:49:02 [INFO] [TRAIN] epoch=47, iter=30800/40000, loss=0.0432, lr=0.002664, batch_cost=0.7978, reader_cost=0.0002 | ETA 02:02:20 2020-12-02 22:50:21 [INFO] [TRAIN] epoch=47, iter=30900/40000, loss=0.0436, lr=0.002638, batch_cost=0.7882, reader_cost=0.0002 | ETA 01:59:32 2020-12-02 22:51:40 [INFO] [TRAIN] epoch=47, iter=31000/40000, loss=0.0438, lr=0.002612, batch_cost=0.7954, reader_cost=0.0002 | ETA 01:59:19 2020-12-02 22:53:02 [INFO] [TRAIN] epoch=48, iter=31100/40000, loss=0.0426, lr=0.002586, batch_cost=0.8160, reader_cost=0.0126 | ETA 02:01:02 2020-12-02 22:54:29 [INFO] [TRAIN] epoch=48, iter=31200/40000, loss=0.0422, lr=0.002560, batch_cost=0.8727, reader_cost=0.0002 | ETA 02:08:00 2020-12-02 22:55:49 [INFO] [TRAIN] epoch=48, iter=31300/40000, loss=0.0402, lr=0.002534, batch_cost=0.7949, reader_cost=0.0002 | ETA 01:55:15 2020-12-02 22:57:13 [INFO] [TRAIN] epoch=48, iter=31400/40000, loss=0.0396, lr=0.002507, batch_cost=0.8397, reader_cost=0.0002 | ETA 02:00:21 2020-12-02 22:58:31 [INFO] [TRAIN] epoch=48, iter=31500/40000, loss=0.0411, lr=0.002481, batch_cost=0.7771, reader_cost=0.0002 | ETA 01:50:05 2020-12-02 22:59:52 [INFO] [TRAIN] epoch=48, iter=31600/40000, loss=0.0413, lr=0.002455, batch_cost=0.8171, reader_cost=0.0002 | ETA 01:54:23 2020-12-02 23:01:13 [INFO] [TRAIN] epoch=48, iter=31700/40000, loss=0.0428, lr=0.002429, batch_cost=0.8051, reader_cost=0.0002 | ETA 01:51:22 2020-12-02 23:02:33 [INFO] [TRAIN] epoch=49, iter=31800/40000, loss=0.0413, lr=0.002402, batch_cost=0.8069, reader_cost=0.0079 | ETA 01:50:16 2020-12-02 23:03:55 [INFO] [TRAIN] epoch=49, iter=31900/40000, loss=0.0417, lr=0.002376, batch_cost=0.8157, reader_cost=0.0002 | ETA 01:50:06 2020-12-02 23:05:17 [INFO] [TRAIN] epoch=49, iter=32000/40000, loss=0.0410, lr=0.002350, batch_cost=0.8218, reader_cost=0.0002 | ETA 01:49:34 2020-12-02 23:05:17 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-12-02 23:05:51 [INFO] [EVAL] #Images=1449 mIoU=0.7957 Acc=0.9538 Kappa=0.8972 2020-12-02 23:05:51 [INFO] [EVAL] Class IoU: [0.9464 0.8878 0.4389 0.9222 0.6545 0.8333 0.9562 0.8917 0.91 0.4706 0.9231 0.5554 0.8724 0.8936 0.8615 0.8819 0.6945 0.8905 0.5517 0.8792 0.794 ] 2020-12-02 23:05:51 [INFO] [EVAL] Class Acc: [0.9659 0.9292 0.4632 0.9618 0.7687 0.9188 0.9874 0.9618 0.9408 0.6989 0.9781 0.8699 0.9441 0.9539 0.9104 0.9539 0.8718 0.9488 0.8358 0.9514 0.9235] 2020-12-02 23:06:02 [INFO] [EVAL] The model with the best validation mIoU (0.8022) was saved at iter 12000. 2020-12-02 23:07:22 [INFO] [TRAIN] epoch=49, iter=32100/40000, loss=0.0432, lr=0.002323, batch_cost=0.8091, reader_cost=0.0006 | ETA 01:46:31 2020-12-02 23:08:47 [INFO] [TRAIN] epoch=49, iter=32200/40000, loss=0.0415, lr=0.002297, batch_cost=0.8447, reader_cost=0.0002 | ETA 01:49:48 2020-12-02 23:10:13 [INFO] [TRAIN] epoch=49, iter=32300/40000, loss=0.0409, lr=0.002270, batch_cost=0.8657, reader_cost=0.0002 | ETA 01:51:05 2020-12-02 23:11:39 [INFO] [TRAIN] epoch=50, iter=32400/40000, loss=0.0406, lr=0.002244, batch_cost=0.8552, reader_cost=0.0091 | ETA 01:48:19 2020-12-02 23:13:00 [INFO] [TRAIN] epoch=50, iter=32500/40000, loss=0.0405, lr=0.002217, batch_cost=0.8088, reader_cost=0.0002 | ETA 01:41:06 2020-12-02 23:14:21 [INFO] [TRAIN] epoch=50, iter=32600/40000, loss=0.0417, lr=0.002190, batch_cost=0.8114, reader_cost=0.0002 | ETA 01:40:04 2020-12-02 23:15:46 [INFO] [TRAIN] epoch=50, iter=32700/40000, loss=0.0413, lr=0.002164, batch_cost=0.8495, reader_cost=0.0002 | ETA 01:43:21 2020-12-02 23:17:10 [INFO] [TRAIN] epoch=50, iter=32800/40000, loss=0.0408, lr=0.002137, batch_cost=0.8370, reader_cost=0.0003 | ETA 01:40:26 2020-12-02 23:18:30 [INFO] [TRAIN] epoch=50, iter=32900/40000, loss=0.0417, lr=0.002110, batch_cost=0.8022, reader_cost=0.0002 | ETA 01:34:55 2020-12-02 23:19:55 [INFO] [TRAIN] epoch=50, iter=33000/40000, loss=0.0416, lr=0.002083, batch_cost=0.8464, reader_cost=0.0002 | ETA 01:38:44 2020-12-02 23:21:13 [INFO] [TRAIN] epoch=51, iter=33100/40000, loss=0.0440, lr=0.002057, batch_cost=0.7848, reader_cost=0.0079 | ETA 01:30:14 2020-12-02 23:22:29 [INFO] [TRAIN] epoch=51, iter=33200/40000, loss=0.0401, lr=0.002030, batch_cost=0.7587, reader_cost=0.0002 | ETA 01:25:59 2020-12-02 23:23:49 [INFO] [TRAIN] epoch=51, iter=33300/40000, loss=0.0389, lr=0.002003, batch_cost=0.8026, reader_cost=0.0002 | ETA 01:29:37 2020-12-02 23:25:06 [INFO] [TRAIN] epoch=51, iter=33400/40000, loss=0.0381, lr=0.001976, batch_cost=0.7712, reader_cost=0.0002 | ETA 01:24:50 2020-12-02 23:26:31 [INFO] [TRAIN] epoch=51, iter=33500/40000, loss=0.0428, lr=0.001949, batch_cost=0.8480, reader_cost=0.0004 | ETA 01:31:51 2020-12-02 23:27:57 [INFO] [TRAIN] epoch=51, iter=33600/40000, loss=0.0403, lr=0.001922, batch_cost=0.8614, reader_cost=0.0008 | ETA 01:31:52 2020-12-02 23:29:19 [INFO] [TRAIN] epoch=51, iter=33700/40000, loss=0.0416, lr=0.001895, batch_cost=0.8129, reader_cost=0.0002 | ETA 01:25:21 2020-12-02 23:30:44 [INFO] [TRAIN] epoch=52, iter=33800/40000, loss=0.0388, lr=0.001868, batch_cost=0.8524, reader_cost=0.0083 | ETA 01:28:04 2020-12-02 23:32:06 [INFO] [TRAIN] epoch=52, iter=33900/40000, loss=0.0418, lr=0.001841, batch_cost=0.8207, reader_cost=0.0003 | ETA 01:23:26 2020-12-02 23:33:25 [INFO] [TRAIN] epoch=52, iter=34000/40000, loss=0.0390, lr=0.001814, batch_cost=0.7874, reader_cost=0.0002 | ETA 01:18:44 2020-12-02 23:33:25 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-12-02 23:34:11 [INFO] [EVAL] #Images=1449 mIoU=0.7947 Acc=0.9538 Kappa=0.8973 2020-12-02 23:34:11 [INFO] [EVAL] Class IoU: [0.9466 0.8893 0.443 0.9239 0.6637 0.8388 0.948 0.8928 0.9088 0.4762 0.9155 0.551 0.8721 0.8779 0.8646 0.8826 0.6903 0.8786 0.5612 0.8753 0.7891] 2020-12-02 23:34:11 [INFO] [EVAL] Class Acc: [0.9662 0.9263 0.4649 0.9664 0.7781 0.9109 0.988 0.9615 0.9415 0.7267 0.9805 0.8723 0.9418 0.9369 0.9126 0.9527 0.8678 0.9421 0.8384 0.9454 0.9114] 2020-12-02 23:34:22 [INFO] [EVAL] The model with the best validation mIoU (0.8022) was saved at iter 12000. 2020-12-02 23:35:51 [INFO] [TRAIN] epoch=52, iter=34100/40000, loss=0.0394, lr=0.001786, batch_cost=0.8914, reader_cost=0.0003 | ETA 01:27:39 2020-12-02 23:37:12 [INFO] [TRAIN] epoch=52, iter=34200/40000, loss=0.0407, lr=0.001759, batch_cost=0.8137, reader_cost=0.0002 | ETA 01:18:39 2020-12-02 23:38:33 [INFO] [TRAIN] epoch=52, iter=34300/40000, loss=0.0397, lr=0.001732, batch_cost=0.8095, reader_cost=0.0002 | ETA 01:16:53 2020-12-02 23:39:52 [INFO] [TRAIN] epoch=53, iter=34400/40000, loss=0.0415, lr=0.001704, batch_cost=0.7863, reader_cost=0.0093 | ETA 01:13:23 2020-12-02 23:41:18 [INFO] [TRAIN] epoch=53, iter=34500/40000, loss=0.0389, lr=0.001677, batch_cost=0.8617, reader_cost=0.0002 | ETA 01:18:59 2020-12-02 23:42:48 [INFO] [TRAIN] epoch=53, iter=34600/40000, loss=0.0388, lr=0.001650, batch_cost=0.8988, reader_cost=0.0004 | ETA 01:20:53 2020-12-02 23:44:14 [INFO] [TRAIN] epoch=53, iter=34700/40000, loss=0.0415, lr=0.001622, batch_cost=0.8629, reader_cost=0.0002 | ETA 01:16:13 2020-12-02 23:45:39 [INFO] [TRAIN] epoch=53, iter=34800/40000, loss=0.0395, lr=0.001594, batch_cost=0.8491, reader_cost=0.0002 | ETA 01:13:35 2020-12-02 23:46:58 [INFO] [TRAIN] epoch=53, iter=34900/40000, loss=0.0404, lr=0.001567, batch_cost=0.7913, reader_cost=0.0002 | ETA 01:07:15 2020-12-02 23:48:19 [INFO] [TRAIN] epoch=53, iter=35000/40000, loss=0.0434, lr=0.001539, batch_cost=0.8067, reader_cost=0.0002 | ETA 01:07:13 2020-12-02 23:49:53 [INFO] [TRAIN] epoch=54, iter=35100/40000, loss=0.0394, lr=0.001511, batch_cost=0.9427, reader_cost=0.0088 | ETA 01:16:59 2020-12-02 23:51:16 [INFO] [TRAIN] epoch=54, iter=35200/40000, loss=0.0393, lr=0.001484, batch_cost=0.8299, reader_cost=0.0006 | ETA 01:06:23 2020-12-02 23:52:41 [INFO] [TRAIN] epoch=54, iter=35300/40000, loss=0.0408, lr=0.001456, batch_cost=0.8460, reader_cost=0.0002 | ETA 01:06:16 2020-12-02 23:54:05 [INFO] [TRAIN] epoch=54, iter=35400/40000, loss=0.0393, lr=0.001428, batch_cost=0.8401, reader_cost=0.0002 | ETA 01:04:24 2020-12-02 23:55:30 [INFO] [TRAIN] epoch=54, iter=35500/40000, loss=0.0388, lr=0.001400, batch_cost=0.8474, reader_cost=0.0002 | ETA 01:03:33 2020-12-02 23:56:54 [INFO] [TRAIN] epoch=54, iter=35600/40000, loss=0.0442, lr=0.001372, batch_cost=0.8460, reader_cost=0.0002 | ETA 01:02:02 2020-12-02 23:58:27 [INFO] [TRAIN] epoch=55, iter=35700/40000, loss=0.0389, lr=0.001344, batch_cost=0.9300, reader_cost=0.0086 | ETA 01:06:38 2020-12-02 23:59:53 [INFO] [TRAIN] epoch=55, iter=35800/40000, loss=0.0415, lr=0.001316, batch_cost=0.8525, reader_cost=0.0004 | ETA 00:59:40 2020-12-03 00:01:16 [INFO] [TRAIN] epoch=55, iter=35900/40000, loss=0.0381, lr=0.001287, batch_cost=0.8302, reader_cost=0.0002 | ETA 00:56:43 2020-12-03 00:02:42 [INFO] [TRAIN] epoch=55, iter=36000/40000, loss=0.0408, lr=0.001259, batch_cost=0.8670, reader_cost=0.0004 | ETA 00:57:48 2020-12-03 00:02:42 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-12-03 00:03:19 [INFO] [EVAL] #Images=1449 mIoU=0.7925 Acc=0.9533 Kappa=0.8958 2020-12-03 00:03:19 [INFO] [EVAL] Class IoU: [0.9459 0.8882 0.442 0.9222 0.6574 0.8356 0.949 0.8911 0.9048 0.4748 0.9182 0.5469 0.8662 0.8855 0.8597 0.8815 0.6716 0.8866 0.553 0.8732 0.789 ] 2020-12-03 00:03:19 [INFO] [EVAL] Class Acc: [0.9647 0.9356 0.4643 0.9668 0.7689 0.9196 0.9877 0.9605 0.9374 0.722 0.9777 0.8833 0.9461 0.947 0.9143 0.9556 0.8714 0.9439 0.8405 0.9445 0.9249] 2020-12-03 00:03:30 [INFO] [EVAL] The model with the best validation mIoU (0.8022) was saved at iter 12000. 2020-12-03 00:04:56 [INFO] [TRAIN] epoch=55, iter=36100/40000, loss=0.0394, lr=0.001231, batch_cost=0.8585, reader_cost=0.0007 | ETA 00:55:48 2020-12-03 00:06:17 [INFO] [TRAIN] epoch=55, iter=36200/40000, loss=0.0417, lr=0.001202, batch_cost=0.8095, reader_cost=0.0002 | ETA 00:51:16 2020-12-03 00:07:36 [INFO] [TRAIN] epoch=55, iter=36300/40000, loss=0.0412, lr=0.001174, batch_cost=0.7948, reader_cost=0.0002 | ETA 00:49:00 2020-12-03 00:09:00 [INFO] [TRAIN] epoch=56, iter=36400/40000, loss=0.0357, lr=0.001145, batch_cost=0.8350, reader_cost=0.0104 | ETA 00:50:05 2020-12-03 00:10:20 [INFO] [TRAIN] epoch=56, iter=36500/40000, loss=0.0395, lr=0.001117, batch_cost=0.8000, reader_cost=0.0002 | ETA 00:46:40 2020-12-03 00:11:39 [INFO] [TRAIN] epoch=56, iter=36600/40000, loss=0.0410, lr=0.001088, batch_cost=0.7909, reader_cost=0.0002 | ETA 00:44:48 2020-12-03 00:12:58 [INFO] [TRAIN] epoch=56, iter=36700/40000, loss=0.0393, lr=0.001059, batch_cost=0.7891, reader_cost=0.0002 | ETA 00:43:24 2020-12-03 00:14:17 [INFO] [TRAIN] epoch=56, iter=36800/40000, loss=0.0377, lr=0.001030, batch_cost=0.7927, reader_cost=0.0002 | ETA 00:42:16 2020-12-03 00:15:33 [INFO] [TRAIN] epoch=56, iter=36900/40000, loss=0.0412, lr=0.001001, batch_cost=0.7619, reader_cost=0.0002 | ETA 00:39:22 2020-12-03 00:16:51 [INFO] [TRAIN] epoch=56, iter=37000/40000, loss=0.0397, lr=0.000972, batch_cost=0.7813, reader_cost=0.0002 | ETA 00:39:03 2020-12-03 00:18:12 [INFO] [TRAIN] epoch=57, iter=37100/40000, loss=0.0397, lr=0.000943, batch_cost=0.8097, reader_cost=0.0109 | ETA 00:39:08 2020-12-03 00:19:30 [INFO] [TRAIN] epoch=57, iter=37200/40000, loss=0.0371, lr=0.000914, batch_cost=0.7755, reader_cost=0.0002 | ETA 00:36:11 2020-12-03 00:20:50 [INFO] [TRAIN] epoch=57, iter=37300/40000, loss=0.0382, lr=0.000884, batch_cost=0.7994, reader_cost=0.0002 | ETA 00:35:58 2020-12-03 00:22:11 [INFO] [TRAIN] epoch=57, iter=37400/40000, loss=0.0399, lr=0.000855, batch_cost=0.8086, reader_cost=0.0002 | ETA 00:35:02 2020-12-03 00:23:27 [INFO] [TRAIN] epoch=57, iter=37500/40000, loss=0.0406, lr=0.000825, batch_cost=0.7681, reader_cost=0.0002 | ETA 00:32:00 2020-12-03 00:24:54 [INFO] [TRAIN] epoch=57, iter=37600/40000, loss=0.0402, lr=0.000795, batch_cost=0.8631, reader_cost=0.0002 | ETA 00:34:31 2020-12-03 00:26:17 [INFO] [TRAIN] epoch=58, iter=37700/40000, loss=0.0388, lr=0.000765, batch_cost=0.8308, reader_cost=0.0116 | ETA 00:31:50 2020-12-03 00:27:39 [INFO] [TRAIN] epoch=58, iter=37800/40000, loss=0.0388, lr=0.000735, batch_cost=0.8197, reader_cost=0.0003 | ETA 00:30:03 2020-12-03 00:29:09 [INFO] [TRAIN] epoch=58, iter=37900/40000, loss=0.0375, lr=0.000705, batch_cost=0.9068, reader_cost=0.0006 | ETA 00:31:44 2020-12-03 00:30:32 [INFO] [TRAIN] epoch=58, iter=38000/40000, loss=0.0393, lr=0.000675, batch_cost=0.8222, reader_cost=0.0002 | ETA 00:27:24 2020-12-03 00:30:32 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-12-03 00:31:06 [INFO] [EVAL] #Images=1449 mIoU=0.7945 Acc=0.9537 Kappa=0.8968 2020-12-03 00:31:06 [INFO] [EVAL] Class IoU: [0.946 0.8913 0.4404 0.9214 0.6527 0.8329 0.9487 0.8899 0.9152 0.4728 0.9232 0.5536 0.8786 0.89 0.8613 0.8813 0.6797 0.8883 0.5543 0.8768 0.7855] 2020-12-03 00:31:06 [INFO] [EVAL] Class Acc: [0.9653 0.9335 0.4622 0.9646 0.7669 0.9095 0.9872 0.9598 0.9495 0.7205 0.9757 0.8805 0.9435 0.9566 0.912 0.9539 0.8726 0.9458 0.8405 0.9443 0.9205] 2020-12-03 00:31:16 [INFO] [EVAL] The model with the best validation mIoU (0.8022) was saved at iter 12000. 2020-12-03 00:32:37 [INFO] [TRAIN] epoch=58, iter=38100/40000, loss=0.0387, lr=0.000645, batch_cost=0.8089, reader_cost=0.0002 | ETA 00:25:36 2020-12-03 00:34:07 [INFO] [TRAIN] epoch=58, iter=38200/40000, loss=0.0406, lr=0.000614, batch_cost=0.8983, reader_cost=0.0002 | ETA 00:26:56 2020-12-03 00:35:34 [INFO] [TRAIN] epoch=58, iter=38300/40000, loss=0.0401, lr=0.000583, batch_cost=0.8645, reader_cost=0.0002 | ETA 00:24:29 2020-12-03 00:36:55 [INFO] [TRAIN] epoch=59, iter=38400/40000, loss=0.0408, lr=0.000552, batch_cost=0.8126, reader_cost=0.0127 | ETA 00:21:40 2020-12-03 00:38:28 [INFO] [TRAIN] epoch=59, iter=38500/40000, loss=0.0382, lr=0.000521, batch_cost=0.9318, reader_cost=0.0002 | ETA 00:23:17 2020-12-03 00:39:52 [INFO] [TRAIN] epoch=59, iter=38600/40000, loss=0.0398, lr=0.000490, batch_cost=0.8358, reader_cost=0.0002 | ETA 00:19:30 2020-12-03 00:41:11 [INFO] [TRAIN] epoch=59, iter=38700/40000, loss=0.0362, lr=0.000458, batch_cost=0.7938, reader_cost=0.0002 | ETA 00:17:11 2020-12-03 00:42:31 [INFO] [TRAIN] epoch=59, iter=38800/40000, loss=0.0415, lr=0.000426, batch_cost=0.8017, reader_cost=0.0005 | ETA 00:16:02 2020-12-03 00:43:55 [INFO] [TRAIN] epoch=59, iter=38900/40000, loss=0.0406, lr=0.000394, batch_cost=0.8411, reader_cost=0.0002 | ETA 00:15:25 2020-12-03 00:45:20 [INFO] [TRAIN] epoch=60, iter=39000/40000, loss=0.0392, lr=0.000362, batch_cost=0.8434, reader_cost=0.0118 | ETA 00:14:03 2020-12-03 00:46:39 [INFO] [TRAIN] epoch=60, iter=39100/40000, loss=0.0387, lr=0.000329, batch_cost=0.7889, reader_cost=0.0002 | ETA 00:11:50 2020-12-03 00:47:57 [INFO] [TRAIN] epoch=60, iter=39200/40000, loss=0.0388, lr=0.000296, batch_cost=0.7862, reader_cost=0.0003 | ETA 00:10:28 2020-12-03 00:49:18 [INFO] [TRAIN] epoch=60, iter=39300/40000, loss=0.0393, lr=0.000263, batch_cost=0.8109, reader_cost=0.0002 | ETA 00:09:27 2020-12-03 00:50:41 [INFO] [TRAIN] epoch=60, iter=39400/40000, loss=0.0395, lr=0.000229, batch_cost=0.8280, reader_cost=0.0013 | ETA 00:08:16 2020-12-03 00:52:05 [INFO] [TRAIN] epoch=60, iter=39500/40000, loss=0.0393, lr=0.000194, batch_cost=0.8364, reader_cost=0.0003 | ETA 00:06:58 2020-12-03 00:53:32 [INFO] [TRAIN] epoch=60, iter=39600/40000, loss=0.0406, lr=0.000159, batch_cost=0.8702, reader_cost=0.0011 | ETA 00:05:48 2020-12-03 00:54:58 [INFO] [TRAIN] epoch=61, iter=39700/40000, loss=0.0395, lr=0.000123, batch_cost=0.8606, reader_cost=0.0088 | ETA 00:04:18 2020-12-03 00:56:23 [INFO] [TRAIN] epoch=61, iter=39800/40000, loss=0.0375, lr=0.000085, batch_cost=0.8527, reader_cost=0.0002 | ETA 00:02:50 2020-12-03 00:57:44 [INFO] [TRAIN] epoch=61, iter=39900/40000, loss=0.0407, lr=0.000046, batch_cost=0.8101, reader_cost=0.0002 | ETA 00:01:21 2020-12-03 00:59:03 [INFO] [TRAIN] epoch=61, iter=40000/40000, loss=0.0397, lr=0.000001, batch_cost=0.7873, reader_cost=0.0002 | ETA 00:00:00 2020-12-03 00:59:03 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-12-03 00:59:36 [INFO] [EVAL] #Images=1449 mIoU=0.7953 Acc=0.9538 Kappa=0.8972 2020-12-03 00:59:36 [INFO] [EVAL] Class IoU: [0.9462 0.8897 0.4408 0.9218 0.6509 0.8362 0.9526 0.8906 0.9156 0.4772 0.9231 0.5553 0.8789 0.8927 0.8598 0.8816 0.6827 0.8838 0.5576 0.8762 0.7874] 2020-12-03 00:59:36 [INFO] [EVAL] Class Acc: [0.9658 0.9325 0.4634 0.9664 0.7626 0.9131 0.9861 0.96 0.9504 0.7109 0.9757 0.8664 0.9458 0.9552 0.9144 0.9528 0.8753 0.9349 0.8403 0.9474 0.9193] 2020-12-03 00:59:47 [INFO] [EVAL] The model with the best validation mIoU (0.8022) was saved at iter 12000.