2020-10-30 13:20:09 [INFO] ------------Environment Information------------- platform: Linux-3.10.0_3-0-0-34-x86_64-with-centos-7.5.1804-Core Python: 3.7.9 (default, Aug 31 2020, 12:42:55) [GCC 7.3.0] Paddle compiled with cuda: True NVCC: Cuda compilation tools, release 10.2, V10.2.89 cudnn: 7.6 GPUs used: 4 CUDA_VISIBLE_DEVICES: 0,1,2,3 GPU: ['GPU 0: Tesla V100-SXM2-16GB', 'GPU 1: Tesla V100-SXM2-16GB', 'GPU 2: Tesla V100-SXM2-16GB', 'GPU 3: Tesla V100-SXM2-16GB', 'GPU 4: Tesla V100-SXM2-16GB', 'GPU 5: Tesla V100-SXM2-16GB', 'GPU 6: Tesla V100-SXM2-16GB', 'GPU 7: Tesla V100-SXM2-16GB'] GCC: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36) PaddlePaddle: 2.0.0-rc0 OpenCV: 4.1.0 ------------------------------------------------ 2020-10-30 13:20:09 [INFO] ---------------Config Information--------------- batch_size: 2 iters: 80000 learning_rate: decay: end_lr: 1.0e-05 power: 0.9 type: poly value: 0.01 loss: coef: - 1 - 0.4 types: - ignore_index: 255 type: CrossEntropyLoss model: backbone: output_stride: 8 pretrained: https://bj.bcebos.com/paddleseg/dygraph/resnet101_vd_ssld.tar.gz type: ResNet101_vd backbone_indices: - 2 - 3 enable_auxiliary_loss: true inter_channels: 512 key_value_channels: 256 pretrained: null psp_size: - 1 - 3 - 6 - 8 type: ANN optimizer: momentum: 0.9 type: sgd weight_decay: 4.0e-05 train_dataset: dataset_root: data/cityscapes mode: train transforms: - max_scale_factor: 2.0 min_scale_factor: 0.5 scale_step_size: 0.25 type: ResizeStepScaling - crop_size: - 1024 - 512 type: RandomPaddingCrop - type: RandomHorizontalFlip - brightness_range: 0.4 contrast_range: 0.4 saturation_range: 0.4 type: RandomDistort - type: Normalize type: Cityscapes val_dataset: dataset_root: data/cityscapes mode: val transforms: - type: Normalize type: Cityscapes ------------------------------------------------ 2020-10-30 13:20:13 [INFO] Loading pretrained model from https://bj.bcebos.com/paddleseg/dygraph/resnet101_vd_ssld.tar.gz 2020-10-30 13:20:15 [INFO] There are 530/530 variables loaded into ResNet_vd. 2020-10-30 13:21:39 [INFO] [TRAIN] epoch=1, iter=100/80000, loss=1.4150, lr=0.009989, batch_cost=0.7839, reader_cost=0.0124 | ETA 17:23:49 2020-10-30 13:22:53 [INFO] [TRAIN] epoch=1, iter=200/80000, loss=0.9170, lr=0.009978, batch_cost=0.7404, reader_cost=0.0004 | ETA 16:24:44 2020-10-30 13:24:08 [INFO] [TRAIN] epoch=1, iter=300/80000, loss=0.8712, lr=0.009966, batch_cost=0.7448, reader_cost=0.0002 | ETA 16:29:20 2020-10-30 13:25:23 [INFO] [TRAIN] epoch=2, iter=400/80000, loss=0.6295, lr=0.009955, batch_cost=0.7555, reader_cost=0.0082 | ETA 16:42:17 2020-10-30 13:26:38 [INFO] [TRAIN] epoch=2, iter=500/80000, loss=0.6577, lr=0.009944, batch_cost=0.7479, reader_cost=0.0008 | ETA 16:30:59 2020-10-30 13:27:53 [INFO] [TRAIN] epoch=2, iter=600/80000, loss=0.4679, lr=0.009933, batch_cost=0.7488, reader_cost=0.0002 | ETA 16:30:56 2020-10-30 13:29:10 [INFO] [TRAIN] epoch=2, iter=700/80000, loss=0.6252, lr=0.009921, batch_cost=0.7655, reader_cost=0.0009 | ETA 16:51:42 2020-10-30 13:30:27 [INFO] [TRAIN] epoch=3, iter=800/80000, loss=0.5810, lr=0.009910, batch_cost=0.7765, reader_cost=0.0097 | ETA 17:05:00 2020-10-30 13:31:44 [INFO] [TRAIN] epoch=3, iter=900/80000, loss=0.6250, lr=0.009899, batch_cost=0.7676, reader_cost=0.0004 | ETA 16:51:59 2020-10-30 13:33:01 [INFO] [TRAIN] epoch=3, iter=1000/80000, loss=0.4285, lr=0.009888, batch_cost=0.7663, reader_cost=0.0003 | ETA 16:48:57 2020-10-30 13:34:18 [INFO] [TRAIN] epoch=3, iter=1100/80000, loss=0.6557, lr=0.009876, batch_cost=0.7703, reader_cost=0.0004 | ETA 16:52:53 2020-10-30 13:35:35 [INFO] [TRAIN] epoch=4, iter=1200/80000, loss=0.4001, lr=0.009865, batch_cost=0.7775, reader_cost=0.0082 | ETA 17:01:09 2020-10-30 13:36:52 [INFO] [TRAIN] epoch=4, iter=1300/80000, loss=0.4000, lr=0.009854, batch_cost=0.7679, reader_cost=0.0009 | ETA 16:47:14 2020-10-30 13:38:09 [INFO] [TRAIN] epoch=4, iter=1400/80000, loss=0.4499, lr=0.009843, batch_cost=0.7714, reader_cost=0.0012 | ETA 16:50:33 2020-10-30 13:39:27 [INFO] [TRAIN] epoch=5, iter=1500/80000, loss=0.3938, lr=0.009831, batch_cost=0.7793, reader_cost=0.0085 | ETA 16:59:36 2020-10-30 13:40:44 [INFO] [TRAIN] epoch=5, iter=1600/80000, loss=0.5110, lr=0.009820, batch_cost=0.7716, reader_cost=0.0003 | ETA 16:48:10 2020-10-30 13:42:02 [INFO] [TRAIN] epoch=5, iter=1700/80000, loss=0.4271, lr=0.009809, batch_cost=0.7718, reader_cost=0.0006 | ETA 16:47:14 2020-10-30 13:43:19 [INFO] [TRAIN] epoch=5, iter=1800/80000, loss=0.4774, lr=0.009798, batch_cost=0.7735, reader_cost=0.0005 | ETA 16:48:06 2020-10-30 13:44:37 [INFO] [TRAIN] epoch=6, iter=1900/80000, loss=0.4245, lr=0.009786, batch_cost=0.7783, reader_cost=0.0084 | ETA 16:53:03 2020-10-30 13:45:54 [INFO] [TRAIN] epoch=6, iter=2000/80000, loss=0.3271, lr=0.009775, batch_cost=0.7731, reader_cost=0.0006 | ETA 16:45:03 2020-10-30 13:47:11 [INFO] [TRAIN] epoch=6, iter=2100/80000, loss=0.3052, lr=0.009764, batch_cost=0.7707, reader_cost=0.0005 | ETA 16:40:39 2020-10-30 13:48:29 [INFO] [TRAIN] epoch=6, iter=2200/80000, loss=0.4066, lr=0.009753, batch_cost=0.7749, reader_cost=0.0003 | ETA 16:44:49 2020-10-30 13:49:47 [INFO] [TRAIN] epoch=7, iter=2300/80000, loss=0.4273, lr=0.009741, batch_cost=0.7849, reader_cost=0.0082 | ETA 16:56:29 2020-10-30 13:51:04 [INFO] [TRAIN] epoch=7, iter=2400/80000, loss=0.3739, lr=0.009730, batch_cost=0.7717, reader_cost=0.0006 | ETA 16:38:03 2020-10-30 13:52:22 [INFO] [TRAIN] epoch=7, iter=2500/80000, loss=0.3735, lr=0.009719, batch_cost=0.7728, reader_cost=0.0011 | ETA 16:38:10 2020-10-30 13:53:39 [INFO] [TRAIN] epoch=7, iter=2600/80000, loss=0.3435, lr=0.009707, batch_cost=0.7701, reader_cost=0.0006 | ETA 16:33:25 2020-10-30 13:54:56 [INFO] [TRAIN] epoch=8, iter=2700/80000, loss=0.4718, lr=0.009696, batch_cost=0.7727, reader_cost=0.0079 | ETA 16:35:31 2020-10-30 13:56:12 [INFO] [TRAIN] epoch=8, iter=2800/80000, loss=0.3804, lr=0.009685, batch_cost=0.7649, reader_cost=0.0010 | ETA 16:24:10 2020-10-30 13:57:29 [INFO] [TRAIN] epoch=8, iter=2900/80000, loss=0.3885, lr=0.009674, batch_cost=0.7677, reader_cost=0.0007 | ETA 16:26:29 2020-10-30 13:58:47 [INFO] [TRAIN] epoch=9, iter=3000/80000, loss=0.3157, lr=0.009662, batch_cost=0.7800, reader_cost=0.0084 | ETA 16:41:00 2020-10-30 14:00:04 [INFO] [TRAIN] epoch=9, iter=3100/80000, loss=0.4231, lr=0.009651, batch_cost=0.7713, reader_cost=0.0002 | ETA 16:28:34 2020-10-30 14:01:21 [INFO] [TRAIN] epoch=9, iter=3200/80000, loss=0.3248, lr=0.009640, batch_cost=0.7690, reader_cost=0.0003 | ETA 16:24:15 2020-10-30 14:02:38 [INFO] [TRAIN] epoch=9, iter=3300/80000, loss=0.3414, lr=0.009628, batch_cost=0.7669, reader_cost=0.0004 | ETA 16:20:24 2020-10-30 14:03:55 [INFO] [TRAIN] epoch=10, iter=3400/80000, loss=0.3085, lr=0.009617, batch_cost=0.7751, reader_cost=0.0095 | ETA 16:29:35 2020-10-30 14:05:12 [INFO] [TRAIN] epoch=10, iter=3500/80000, loss=0.3478, lr=0.009606, batch_cost=0.7689, reader_cost=0.0007 | ETA 16:20:23 2020-10-30 14:06:29 [INFO] [TRAIN] epoch=10, iter=3600/80000, loss=0.2869, lr=0.009595, batch_cost=0.7679, reader_cost=0.0003 | ETA 16:17:48 2020-10-30 14:07:47 [INFO] [TRAIN] epoch=10, iter=3700/80000, loss=0.3712, lr=0.009583, batch_cost=0.7742, reader_cost=0.0010 | ETA 16:24:34 2020-10-30 14:09:04 [INFO] [TRAIN] epoch=11, iter=3800/80000, loss=0.3528, lr=0.009572, batch_cost=0.7786, reader_cost=0.0092 | ETA 16:28:50 2020-10-30 14:10:21 [INFO] [TRAIN] epoch=11, iter=3900/80000, loss=0.3296, lr=0.009561, batch_cost=0.7677, reader_cost=0.0006 | ETA 16:13:42 2020-10-30 14:11:38 [INFO] [TRAIN] epoch=11, iter=4000/80000, loss=0.3153, lr=0.009549, batch_cost=0.7682, reader_cost=0.0006 | ETA 16:13:05 2020-10-30 14:12:56 [INFO] [TRAIN] epoch=12, iter=4100/80000, loss=0.2604, lr=0.009538, batch_cost=0.7781, reader_cost=0.0086 | ETA 16:24:21 2020-10-30 14:14:13 [INFO] [TRAIN] epoch=12, iter=4200/80000, loss=0.3605, lr=0.009527, batch_cost=0.7722, reader_cost=0.0007 | ETA 16:15:29 2020-10-30 14:15:30 [INFO] [TRAIN] epoch=12, iter=4300/80000, loss=0.2771, lr=0.009516, batch_cost=0.7701, reader_cost=0.0007 | ETA 16:11:34 2020-10-30 14:16:47 [INFO] [TRAIN] epoch=12, iter=4400/80000, loss=0.2932, lr=0.009504, batch_cost=0.7685, reader_cost=0.0007 | ETA 16:08:19 2020-10-30 14:18:05 [INFO] [TRAIN] epoch=13, iter=4500/80000, loss=0.4035, lr=0.009493, batch_cost=0.7788, reader_cost=0.0087 | ETA 16:19:57 2020-10-30 14:19:22 [INFO] [TRAIN] epoch=13, iter=4600/80000, loss=0.3129, lr=0.009482, batch_cost=0.7687, reader_cost=0.0011 | ETA 16:05:59 2020-10-30 14:20:39 [INFO] [TRAIN] epoch=13, iter=4700/80000, loss=0.2854, lr=0.009470, batch_cost=0.7774, reader_cost=0.0010 | ETA 16:15:38 2020-10-30 14:21:58 [INFO] [TRAIN] epoch=13, iter=4800/80000, loss=0.3574, lr=0.009459, batch_cost=0.7811, reader_cost=0.0008 | ETA 16:18:55 2020-10-30 14:23:15 [INFO] [TRAIN] epoch=14, iter=4900/80000, loss=0.3158, lr=0.009448, batch_cost=0.7760, reader_cost=0.0096 | ETA 16:11:15 2020-10-30 14:24:32 [INFO] [TRAIN] epoch=14, iter=5000/80000, loss=0.3369, lr=0.009436, batch_cost=0.7685, reader_cost=0.0007 | ETA 16:00:38 2020-10-30 14:25:48 [INFO] [TRAIN] epoch=14, iter=5100/80000, loss=0.3064, lr=0.009425, batch_cost=0.7640, reader_cost=0.0007 | ETA 15:53:40 2020-10-30 14:27:04 [INFO] [TRAIN] epoch=14, iter=5200/80000, loss=0.3051, lr=0.009414, batch_cost=0.7554, reader_cost=0.0005 | ETA 15:41:44 2020-10-30 14:28:20 [INFO] [TRAIN] epoch=15, iter=5300/80000, loss=0.3165, lr=0.009402, batch_cost=0.7613, reader_cost=0.0082 | ETA 15:47:52 2020-10-30 14:29:36 [INFO] [TRAIN] epoch=15, iter=5400/80000, loss=0.2760, lr=0.009391, batch_cost=0.7559, reader_cost=0.0008 | ETA 15:39:50 2020-10-30 14:30:51 [INFO] [TRAIN] epoch=15, iter=5500/80000, loss=0.3332, lr=0.009380, batch_cost=0.7580, reader_cost=0.0005 | ETA 15:41:12 2020-10-30 14:32:08 [INFO] [TRAIN] epoch=16, iter=5600/80000, loss=0.3033, lr=0.009368, batch_cost=0.7681, reader_cost=0.0080 | ETA 15:52:26 2020-10-30 14:33:25 [INFO] [TRAIN] epoch=16, iter=5700/80000, loss=0.2698, lr=0.009357, batch_cost=0.7705, reader_cost=0.0009 | ETA 15:54:11 2020-10-30 14:34:41 [INFO] [TRAIN] epoch=16, iter=5800/80000, loss=0.2739, lr=0.009346, batch_cost=0.7566, reader_cost=0.0003 | ETA 15:35:40 2020-10-30 14:35:55 [INFO] [TRAIN] epoch=16, iter=5900/80000, loss=0.2822, lr=0.009335, batch_cost=0.7438, reader_cost=0.0002 | ETA 15:18:37 2020-10-30 14:37:12 [INFO] [TRAIN] epoch=17, iter=6000/80000, loss=0.2354, lr=0.009323, batch_cost=0.7659, reader_cost=0.0074 | ETA 15:44:37 2020-10-30 14:38:28 [INFO] [TRAIN] epoch=17, iter=6100/80000, loss=0.3038, lr=0.009312, batch_cost=0.7576, reader_cost=0.0006 | ETA 15:33:09 2020-10-30 14:39:44 [INFO] [TRAIN] epoch=17, iter=6200/80000, loss=0.2502, lr=0.009301, batch_cost=0.7656, reader_cost=0.0005 | ETA 15:41:42 2020-10-30 14:41:01 [INFO] [TRAIN] epoch=17, iter=6300/80000, loss=0.2647, lr=0.009289, batch_cost=0.7677, reader_cost=0.0008 | ETA 15:42:57 2020-10-30 14:42:18 [INFO] [TRAIN] epoch=18, iter=6400/80000, loss=0.3425, lr=0.009278, batch_cost=0.7725, reader_cost=0.0083 | ETA 15:47:34 2020-10-30 14:43:35 [INFO] [TRAIN] epoch=18, iter=6500/80000, loss=0.2721, lr=0.009267, batch_cost=0.7633, reader_cost=0.0009 | ETA 15:35:01 2020-10-30 14:44:51 [INFO] [TRAIN] epoch=18, iter=6600/80000, loss=0.2608, lr=0.009255, batch_cost=0.7632, reader_cost=0.0005 | ETA 15:33:37 2020-10-30 14:46:08 [INFO] [TRAIN] epoch=19, iter=6700/80000, loss=0.2427, lr=0.009244, batch_cost=0.7724, reader_cost=0.0089 | ETA 15:43:37 2020-10-30 14:47:24 [INFO] [TRAIN] epoch=19, iter=6800/80000, loss=0.2555, lr=0.009233, batch_cost=0.7578, reader_cost=0.0005 | ETA 15:24:32 2020-10-30 14:48:40 [INFO] [TRAIN] epoch=19, iter=6900/80000, loss=0.2810, lr=0.009221, batch_cost=0.7647, reader_cost=0.0007 | ETA 15:31:37 2020-10-30 14:49:57 [INFO] [TRAIN] epoch=19, iter=7000/80000, loss=0.3165, lr=0.009210, batch_cost=0.7667, reader_cost=0.0002 | ETA 15:32:50 2020-10-30 14:51:15 [INFO] [TRAIN] epoch=20, iter=7100/80000, loss=0.2879, lr=0.009199, batch_cost=0.7777, reader_cost=0.0092 | ETA 15:44:54 2020-10-30 14:52:31 [INFO] [TRAIN] epoch=20, iter=7200/80000, loss=0.3553, lr=0.009187, batch_cost=0.7652, reader_cost=0.0004 | ETA 15:28:27 2020-10-30 14:53:48 [INFO] [TRAIN] epoch=20, iter=7300/80000, loss=0.3007, lr=0.009176, batch_cost=0.7644, reader_cost=0.0003 | ETA 15:26:12 2020-10-30 14:55:04 [INFO] [TRAIN] epoch=20, iter=7400/80000, loss=0.3019, lr=0.009164, batch_cost=0.7606, reader_cost=0.0006 | ETA 15:20:18 2020-10-30 14:56:22 [INFO] [TRAIN] epoch=21, iter=7500/80000, loss=0.2470, lr=0.009153, batch_cost=0.7783, reader_cost=0.0080 | ETA 15:40:26 2020-10-30 14:57:38 [INFO] [TRAIN] epoch=21, iter=7600/80000, loss=0.2758, lr=0.009142, batch_cost=0.7632, reader_cost=0.0004 | ETA 15:20:58 2020-10-30 14:58:55 [INFO] [TRAIN] epoch=21, iter=7700/80000, loss=0.2238, lr=0.009130, batch_cost=0.7690, reader_cost=0.0003 | ETA 15:26:41 2020-10-30 15:00:11 [INFO] [TRAIN] epoch=21, iter=7800/80000, loss=0.2657, lr=0.009119, batch_cost=0.7650, reader_cost=0.0003 | ETA 15:20:34 2020-10-30 15:01:29 [INFO] [TRAIN] epoch=22, iter=7900/80000, loss=0.2895, lr=0.009108, batch_cost=0.7749, reader_cost=0.0086 | ETA 15:31:11 2020-10-30 15:02:46 [INFO] [TRAIN] epoch=22, iter=8000/80000, loss=0.2500, lr=0.009096, batch_cost=0.7672, reader_cost=0.0002 | ETA 15:20:37 2020-10-30 15:02:53 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-10-30 15:09:09 [INFO] [EVAL] #Images=500 mIoU=0.7148 Acc=0.9511 Kappa=0.9363 2020-10-30 15:09:09 [INFO] [EVAL] Category IoU: [0.9732 0.8066 0.9065 0.4942 0.4904 0.588 0.6684 0.7529 0.9175 0.6102 0.9328 0.7907 0.5342 0.931 0.5165 0.7476 0.5882 0.5907 0.7415] 2020-10-30 15:09:09 [INFO] [EVAL] Category Acc: [0.9838 0.9106 0.9388 0.7438 0.8699 0.7934 0.8212 0.8655 0.9465 0.8313 0.9769 0.8642 0.737 0.957 0.9447 0.8801 0.956 0.8621 0.8482] 2020-10-30 15:09:12 [INFO] [EVAL] The model with the best validation mIoU (0.7148) was saved at iter 8000. 2020-10-30 15:10:28 [INFO] [TRAIN] epoch=22, iter=8100/80000, loss=0.2182, lr=0.009085, batch_cost=0.7563, reader_cost=0.0007 | ETA 15:06:18 2020-10-30 15:11:44 [INFO] [TRAIN] epoch=23, iter=8200/80000, loss=0.2760, lr=0.009074, batch_cost=0.7662, reader_cost=0.0106 | ETA 15:16:51 2020-10-30 15:13:00 [INFO] [TRAIN] epoch=23, iter=8300/80000, loss=0.2811, lr=0.009062, batch_cost=0.7575, reader_cost=0.0003 | ETA 15:05:15 2020-10-30 15:14:17 [INFO] [TRAIN] epoch=23, iter=8400/80000, loss=0.2314, lr=0.009051, batch_cost=0.7672, reader_cost=0.0008 | ETA 15:15:30 2020-10-30 15:15:33 [INFO] [TRAIN] epoch=23, iter=8500/80000, loss=0.2596, lr=0.009040, batch_cost=0.7638, reader_cost=0.0005 | ETA 15:10:13 2020-10-30 15:16:50 [INFO] [TRAIN] epoch=24, iter=8600/80000, loss=0.3009, lr=0.009028, batch_cost=0.7662, reader_cost=0.0090 | ETA 15:11:48 2020-10-30 15:18:06 [INFO] [TRAIN] epoch=24, iter=8700/80000, loss=0.3164, lr=0.009017, batch_cost=0.7599, reader_cost=0.0009 | ETA 15:03:01 2020-10-30 15:19:22 [INFO] [TRAIN] epoch=24, iter=8800/80000, loss=0.2526, lr=0.009005, batch_cost=0.7647, reader_cost=0.0006 | ETA 15:07:29 2020-10-30 15:20:38 [INFO] [TRAIN] epoch=24, iter=8900/80000, loss=0.2488, lr=0.008994, batch_cost=0.7610, reader_cost=0.0005 | ETA 15:01:46 2020-10-30 15:21:55 [INFO] [TRAIN] epoch=25, iter=9000/80000, loss=0.2616, lr=0.008983, batch_cost=0.7686, reader_cost=0.0090 | ETA 15:09:30 2020-10-30 15:23:11 [INFO] [TRAIN] epoch=25, iter=9100/80000, loss=0.2317, lr=0.008971, batch_cost=0.7610, reader_cost=0.0004 | ETA 14:59:17 2020-10-30 15:24:27 [INFO] [TRAIN] epoch=25, iter=9200/80000, loss=0.2511, lr=0.008960, batch_cost=0.7582, reader_cost=0.0002 | ETA 14:54:37 2020-10-30 15:25:44 [INFO] [TRAIN] epoch=25, iter=9300/80000, loss=0.3001, lr=0.008949, batch_cost=0.7638, reader_cost=0.0003 | ETA 15:00:00 2020-10-30 15:27:01 [INFO] [TRAIN] epoch=26, iter=9400/80000, loss=0.2416, lr=0.008937, batch_cost=0.7722, reader_cost=0.0094 | ETA 15:08:37 2020-10-30 15:28:18 [INFO] [TRAIN] epoch=26, iter=9500/80000, loss=0.2350, lr=0.008926, batch_cost=0.7702, reader_cost=0.0003 | ETA 15:05:00 2020-10-30 15:29:34 [INFO] [TRAIN] epoch=26, iter=9600/80000, loss=0.2477, lr=0.008914, batch_cost=0.7595, reader_cost=0.0002 | ETA 14:51:07 2020-10-30 15:30:51 [INFO] [TRAIN] epoch=27, iter=9700/80000, loss=0.2447, lr=0.008903, batch_cost=0.7768, reader_cost=0.0085 | ETA 15:10:09 2020-10-30 15:32:08 [INFO] [TRAIN] epoch=27, iter=9800/80000, loss=0.3088, lr=0.008892, batch_cost=0.7674, reader_cost=0.0006 | ETA 14:57:49 2020-10-30 15:33:24 [INFO] [TRAIN] epoch=27, iter=9900/80000, loss=0.2507, lr=0.008880, batch_cost=0.7620, reader_cost=0.0008 | ETA 14:50:19 2020-10-30 15:34:40 [INFO] [TRAIN] epoch=27, iter=10000/80000, loss=0.2857, lr=0.008869, batch_cost=0.7588, reader_cost=0.0007 | ETA 14:45:14 2020-10-30 15:35:56 [INFO] [TRAIN] epoch=28, iter=10100/80000, loss=0.2586, lr=0.008857, batch_cost=0.7579, reader_cost=0.0087 | ETA 14:42:53 2020-10-30 15:37:10 [INFO] [TRAIN] epoch=28, iter=10200/80000, loss=0.2388, lr=0.008846, batch_cost=0.7442, reader_cost=0.0002 | ETA 14:25:45 2020-10-30 15:38:25 [INFO] [TRAIN] epoch=28, iter=10300/80000, loss=0.2580, lr=0.008835, batch_cost=0.7461, reader_cost=0.0002 | ETA 14:26:39 2020-10-30 15:39:41 [INFO] [TRAIN] epoch=28, iter=10400/80000, loss=0.2651, lr=0.008823, batch_cost=0.7555, reader_cost=0.0002 | ETA 14:36:22 2020-10-30 15:40:57 [INFO] [TRAIN] epoch=29, iter=10500/80000, loss=0.2305, lr=0.008812, batch_cost=0.7602, reader_cost=0.0090 | ETA 14:40:31 2020-10-30 15:42:12 [INFO] [TRAIN] epoch=29, iter=10600/80000, loss=0.2512, lr=0.008801, batch_cost=0.7522, reader_cost=0.0002 | ETA 14:30:02 2020-10-30 15:43:28 [INFO] [TRAIN] epoch=29, iter=10700/80000, loss=0.2891, lr=0.008789, batch_cost=0.7584, reader_cost=0.0004 | ETA 14:35:57 2020-10-30 15:44:45 [INFO] [TRAIN] epoch=30, iter=10800/80000, loss=0.2440, lr=0.008778, batch_cost=0.7713, reader_cost=0.0078 | ETA 14:49:32 2020-10-30 15:46:00 [INFO] [TRAIN] epoch=30, iter=10900/80000, loss=0.2166, lr=0.008766, batch_cost=0.7554, reader_cost=0.0006 | ETA 14:29:56 2020-10-30 15:47:16 [INFO] [TRAIN] epoch=30, iter=11000/80000, loss=0.2259, lr=0.008755, batch_cost=0.7563, reader_cost=0.0004 | ETA 14:29:48 2020-10-30 15:48:32 [INFO] [TRAIN] epoch=30, iter=11100/80000, loss=0.2262, lr=0.008743, batch_cost=0.7608, reader_cost=0.0005 | ETA 14:33:36 2020-10-30 15:49:49 [INFO] [TRAIN] epoch=31, iter=11200/80000, loss=0.2466, lr=0.008732, batch_cost=0.7711, reader_cost=0.0096 | ETA 14:44:11 2020-10-30 15:51:05 [INFO] [TRAIN] epoch=31, iter=11300/80000, loss=0.2508, lr=0.008721, batch_cost=0.7599, reader_cost=0.0005 | ETA 14:30:03 2020-10-30 15:52:21 [INFO] [TRAIN] epoch=31, iter=11400/80000, loss=0.2342, lr=0.008709, batch_cost=0.7539, reader_cost=0.0004 | ETA 14:21:57 2020-10-30 15:53:36 [INFO] [TRAIN] epoch=31, iter=11500/80000, loss=0.2258, lr=0.008698, batch_cost=0.7556, reader_cost=0.0003 | ETA 14:22:39 2020-10-30 15:54:53 [INFO] [TRAIN] epoch=32, iter=11600/80000, loss=0.2309, lr=0.008686, batch_cost=0.7638, reader_cost=0.0083 | ETA 14:30:44 2020-10-30 15:56:09 [INFO] [TRAIN] epoch=32, iter=11700/80000, loss=0.2182, lr=0.008675, batch_cost=0.7602, reader_cost=0.0003 | ETA 14:25:22 2020-10-30 15:57:25 [INFO] [TRAIN] epoch=32, iter=11800/80000, loss=0.2114, lr=0.008664, batch_cost=0.7651, reader_cost=0.0006 | ETA 14:29:37 2020-10-30 15:58:41 [INFO] [TRAIN] epoch=32, iter=11900/80000, loss=0.2521, lr=0.008652, batch_cost=0.7641, reader_cost=0.0006 | ETA 14:27:12 2020-10-30 15:59:58 [INFO] [TRAIN] epoch=33, iter=12000/80000, loss=0.2515, lr=0.008641, batch_cost=0.7668, reader_cost=0.0092 | ETA 14:29:02 2020-10-30 16:01:14 [INFO] [TRAIN] epoch=33, iter=12100/80000, loss=0.2287, lr=0.008629, batch_cost=0.7605, reader_cost=0.0004 | ETA 14:20:40 2020-10-30 16:02:30 [INFO] [TRAIN] epoch=33, iter=12200/80000, loss=0.3136, lr=0.008618, batch_cost=0.7570, reader_cost=0.0005 | ETA 14:15:22 2020-10-30 16:03:46 [INFO] [TRAIN] epoch=34, iter=12300/80000, loss=0.2348, lr=0.008606, batch_cost=0.7649, reader_cost=0.0090 | ETA 14:23:05 2020-10-30 16:05:02 [INFO] [TRAIN] epoch=34, iter=12400/80000, loss=0.2328, lr=0.008595, batch_cost=0.7600, reader_cost=0.0003 | ETA 14:16:13 2020-10-30 16:06:19 [INFO] [TRAIN] epoch=34, iter=12500/80000, loss=0.1939, lr=0.008584, batch_cost=0.7637, reader_cost=0.0003 | ETA 14:19:11 2020-10-30 16:07:35 [INFO] [TRAIN] epoch=34, iter=12600/80000, loss=0.2301, lr=0.008572, batch_cost=0.7667, reader_cost=0.0006 | ETA 14:21:12 2020-10-30 16:08:53 [INFO] [TRAIN] epoch=35, iter=12700/80000, loss=0.2228, lr=0.008561, batch_cost=0.7750, reader_cost=0.0098 | ETA 14:29:17 2020-10-30 16:10:09 [INFO] [TRAIN] epoch=35, iter=12800/80000, loss=0.2235, lr=0.008549, batch_cost=0.7640, reader_cost=0.0005 | ETA 14:15:40 2020-10-30 16:11:25 [INFO] [TRAIN] epoch=35, iter=12900/80000, loss=0.2180, lr=0.008538, batch_cost=0.7587, reader_cost=0.0008 | ETA 14:08:32 2020-10-30 16:12:41 [INFO] [TRAIN] epoch=35, iter=13000/80000, loss=0.2231, lr=0.008526, batch_cost=0.7608, reader_cost=0.0003 | ETA 14:09:34 2020-10-30 16:13:58 [INFO] [TRAIN] epoch=36, iter=13100/80000, loss=0.2240, lr=0.008515, batch_cost=0.7634, reader_cost=0.0109 | ETA 14:11:12 2020-10-30 16:15:13 [INFO] [TRAIN] epoch=36, iter=13200/80000, loss=0.2215, lr=0.008504, batch_cost=0.7565, reader_cost=0.0003 | ETA 14:02:14 2020-10-30 16:16:29 [INFO] [TRAIN] epoch=36, iter=13300/80000, loss=0.2370, lr=0.008492, batch_cost=0.7545, reader_cost=0.0004 | ETA 13:58:47 2020-10-30 16:17:45 [INFO] [TRAIN] epoch=37, iter=13400/80000, loss=0.2631, lr=0.008481, batch_cost=0.7647, reader_cost=0.0086 | ETA 14:08:49 2020-10-30 16:19:01 [INFO] [TRAIN] epoch=37, iter=13500/80000, loss=0.2601, lr=0.008469, batch_cost=0.7556, reader_cost=0.0008 | ETA 13:57:24 2020-10-30 16:20:17 [INFO] [TRAIN] epoch=37, iter=13600/80000, loss=0.2290, lr=0.008458, batch_cost=0.7590, reader_cost=0.0004 | ETA 13:59:56 2020-10-30 16:21:33 [INFO] [TRAIN] epoch=37, iter=13700/80000, loss=0.2985, lr=0.008446, batch_cost=0.7615, reader_cost=0.0002 | ETA 14:01:30 2020-10-30 16:22:49 [INFO] [TRAIN] epoch=38, iter=13800/80000, loss=0.2428, lr=0.008435, batch_cost=0.7630, reader_cost=0.0086 | ETA 14:01:49 2020-10-30 16:24:05 [INFO] [TRAIN] epoch=38, iter=13900/80000, loss=0.2370, lr=0.008423, batch_cost=0.7630, reader_cost=0.0004 | ETA 14:00:34 2020-10-30 16:25:22 [INFO] [TRAIN] epoch=38, iter=14000/80000, loss=0.2071, lr=0.008412, batch_cost=0.7609, reader_cost=0.0004 | ETA 13:56:56 2020-10-30 16:26:37 [INFO] [TRAIN] epoch=38, iter=14100/80000, loss=0.2060, lr=0.008400, batch_cost=0.7589, reader_cost=0.0006 | ETA 13:53:30 2020-10-30 16:27:54 [INFO] [TRAIN] epoch=39, iter=14200/80000, loss=0.2358, lr=0.008389, batch_cost=0.7654, reader_cost=0.0086 | ETA 13:59:24 2020-10-30 16:29:10 [INFO] [TRAIN] epoch=39, iter=14300/80000, loss=0.2278, lr=0.008378, batch_cost=0.7583, reader_cost=0.0003 | ETA 13:50:19 2020-10-30 16:30:26 [INFO] [TRAIN] epoch=39, iter=14400/80000, loss=0.2202, lr=0.008366, batch_cost=0.7597, reader_cost=0.0003 | ETA 13:50:37 2020-10-30 16:31:42 [INFO] [TRAIN] epoch=39, iter=14500/80000, loss=0.2746, lr=0.008355, batch_cost=0.7617, reader_cost=0.0006 | ETA 13:51:34 2020-10-30 16:32:58 [INFO] [TRAIN] epoch=40, iter=14600/80000, loss=0.2234, lr=0.008343, batch_cost=0.7633, reader_cost=0.0101 | ETA 13:52:02 2020-10-30 16:34:14 [INFO] [TRAIN] epoch=40, iter=14700/80000, loss=0.2167, lr=0.008332, batch_cost=0.7558, reader_cost=0.0005 | ETA 13:42:36 2020-10-30 16:35:29 [INFO] [TRAIN] epoch=40, iter=14800/80000, loss=0.2408, lr=0.008320, batch_cost=0.7555, reader_cost=0.0005 | ETA 13:40:56 2020-10-30 16:36:46 [INFO] [TRAIN] epoch=41, iter=14900/80000, loss=0.2149, lr=0.008309, batch_cost=0.7614, reader_cost=0.0092 | ETA 13:46:08 2020-10-30 16:38:01 [INFO] [TRAIN] epoch=41, iter=15000/80000, loss=0.2372, lr=0.008297, batch_cost=0.7543, reader_cost=0.0006 | ETA 13:37:10 2020-10-30 16:39:15 [INFO] [TRAIN] epoch=41, iter=15100/80000, loss=0.2071, lr=0.008286, batch_cost=0.7431, reader_cost=0.0002 | ETA 13:23:44 2020-10-30 16:40:30 [INFO] [TRAIN] epoch=41, iter=15200/80000, loss=0.2139, lr=0.008274, batch_cost=0.7446, reader_cost=0.0002 | ETA 13:24:10 2020-10-30 16:41:45 [INFO] [TRAIN] epoch=42, iter=15300/80000, loss=0.2209, lr=0.008263, batch_cost=0.7552, reader_cost=0.0104 | ETA 13:34:23 2020-10-30 16:43:00 [INFO] [TRAIN] epoch=42, iter=15400/80000, loss=0.2283, lr=0.008251, batch_cost=0.7515, reader_cost=0.0003 | ETA 13:29:03 2020-10-30 16:44:16 [INFO] [TRAIN] epoch=42, iter=15500/80000, loss=0.1944, lr=0.008240, batch_cost=0.7552, reader_cost=0.0002 | ETA 13:31:51 2020-10-30 16:45:32 [INFO] [TRAIN] epoch=42, iter=15600/80000, loss=0.2916, lr=0.008228, batch_cost=0.7601, reader_cost=0.0002 | ETA 13:35:51 2020-10-30 16:46:48 [INFO] [TRAIN] epoch=43, iter=15700/80000, loss=0.2643, lr=0.008217, batch_cost=0.7609, reader_cost=0.0095 | ETA 13:35:28 2020-10-30 16:48:03 [INFO] [TRAIN] epoch=43, iter=15800/80000, loss=0.2959, lr=0.008205, batch_cost=0.7534, reader_cost=0.0003 | ETA 13:26:05 2020-10-30 16:49:19 [INFO] [TRAIN] epoch=43, iter=15900/80000, loss=0.2036, lr=0.008194, batch_cost=0.7603, reader_cost=0.0003 | ETA 13:32:13 2020-10-30 16:50:37 [INFO] [TRAIN] epoch=44, iter=16000/80000, loss=0.2086, lr=0.008182, batch_cost=0.7751, reader_cost=0.0098 | ETA 13:46:47 2020-10-30 16:50:43 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-10-30 16:56:54 [INFO] [EVAL] #Images=500 mIoU=0.7320 Acc=0.9529 Kappa=0.9389 2020-10-30 16:56:54 [INFO] [EVAL] Category IoU: [0.9756 0.8274 0.9093 0.4485 0.5083 0.6149 0.6906 0.7588 0.9121 0.6346 0.9386 0.8058 0.595 0.9372 0.6807 0.7684 0.5531 0.5879 0.7617] 2020-10-30 16:56:54 [INFO] [EVAL] Category Acc: [0.9899 0.9066 0.9539 0.7298 0.8184 0.8145 0.8043 0.8469 0.9347 0.815 0.9652 0.8648 0.7682 0.9574 0.9287 0.934 0.6319 0.7683 0.8408] 2020-10-30 16:56:58 [INFO] [EVAL] The model with the best validation mIoU (0.7320) was saved at iter 16000. 2020-10-30 16:58:13 [INFO] [TRAIN] epoch=44, iter=16100/80000, loss=0.2184, lr=0.008171, batch_cost=0.7475, reader_cost=0.0004 | ETA 13:16:08 2020-10-30 16:59:28 [INFO] [TRAIN] epoch=44, iter=16200/80000, loss=0.2204, lr=0.008159, batch_cost=0.7593, reader_cost=0.0003 | ETA 13:27:24 2020-10-30 17:00:44 [INFO] [TRAIN] epoch=44, iter=16300/80000, loss=0.2240, lr=0.008148, batch_cost=0.7568, reader_cost=0.0005 | ETA 13:23:28 2020-10-30 17:02:01 [INFO] [TRAIN] epoch=45, iter=16400/80000, loss=0.2029, lr=0.008136, batch_cost=0.7682, reader_cost=0.0088 | ETA 13:34:19 2020-10-30 17:03:17 [INFO] [TRAIN] epoch=45, iter=16500/80000, loss=0.2377, lr=0.008125, batch_cost=0.7596, reader_cost=0.0004 | ETA 13:23:52 2020-10-30 17:04:33 [INFO] [TRAIN] epoch=45, iter=16600/80000, loss=0.1955, lr=0.008113, batch_cost=0.7591, reader_cost=0.0004 | ETA 13:22:07 2020-10-30 17:05:49 [INFO] [TRAIN] epoch=45, iter=16700/80000, loss=0.1997, lr=0.008102, batch_cost=0.7643, reader_cost=0.0003 | ETA 13:26:21 2020-10-30 17:07:06 [INFO] [TRAIN] epoch=46, iter=16800/80000, loss=0.2060, lr=0.008090, batch_cost=0.7691, reader_cost=0.0084 | ETA 13:30:08 2020-10-30 17:08:22 [INFO] [TRAIN] epoch=46, iter=16900/80000, loss=0.1969, lr=0.008079, batch_cost=0.7622, reader_cost=0.0008 | ETA 13:21:37 2020-10-30 17:09:38 [INFO] [TRAIN] epoch=46, iter=17000/80000, loss=0.2124, lr=0.008067, batch_cost=0.7595, reader_cost=0.0003 | ETA 13:17:30 2020-10-30 17:10:55 [INFO] [TRAIN] epoch=46, iter=17100/80000, loss=0.2062, lr=0.008056, batch_cost=0.7655, reader_cost=0.0004 | ETA 13:22:32 2020-10-30 17:12:12 [INFO] [TRAIN] epoch=47, iter=17200/80000, loss=0.2074, lr=0.008044, batch_cost=0.7699, reader_cost=0.0092 | ETA 13:25:52 2020-10-30 17:13:28 [INFO] [TRAIN] epoch=47, iter=17300/80000, loss=0.2114, lr=0.008033, batch_cost=0.7636, reader_cost=0.0003 | ETA 13:17:59 2020-10-30 17:14:45 [INFO] [TRAIN] epoch=47, iter=17400/80000, loss=0.2074, lr=0.008021, batch_cost=0.7621, reader_cost=0.0003 | ETA 13:15:05 2020-10-30 17:16:01 [INFO] [TRAIN] epoch=48, iter=17500/80000, loss=0.2051, lr=0.008010, batch_cost=0.7693, reader_cost=0.0091 | ETA 13:21:18 2020-10-30 17:17:18 [INFO] [TRAIN] epoch=48, iter=17600/80000, loss=0.2001, lr=0.007998, batch_cost=0.7614, reader_cost=0.0004 | ETA 13:11:48 2020-10-30 17:18:33 [INFO] [TRAIN] epoch=48, iter=17700/80000, loss=0.2167, lr=0.007987, batch_cost=0.7588, reader_cost=0.0003 | ETA 13:07:55 2020-10-30 17:19:49 [INFO] [TRAIN] epoch=48, iter=17800/80000, loss=0.2157, lr=0.007975, batch_cost=0.7586, reader_cost=0.0001 | ETA 13:06:27 2020-10-30 17:21:06 [INFO] [TRAIN] epoch=49, iter=17900/80000, loss=0.1695, lr=0.007964, batch_cost=0.7705, reader_cost=0.0074 | ETA 13:17:29 2020-10-30 17:22:22 [INFO] [TRAIN] epoch=49, iter=18000/80000, loss=0.2134, lr=0.007952, batch_cost=0.7606, reader_cost=0.0003 | ETA 13:05:59 2020-10-30 17:23:38 [INFO] [TRAIN] epoch=49, iter=18100/80000, loss=0.2132, lr=0.007941, batch_cost=0.7592, reader_cost=0.0002 | ETA 13:03:11 2020-10-30 17:24:54 [INFO] [TRAIN] epoch=49, iter=18200/80000, loss=0.2552, lr=0.007929, batch_cost=0.7604, reader_cost=0.0003 | ETA 13:03:10 2020-10-30 17:26:11 [INFO] [TRAIN] epoch=50, iter=18300/80000, loss=0.2281, lr=0.007918, batch_cost=0.7671, reader_cost=0.0085 | ETA 13:08:49 2020-10-30 17:27:27 [INFO] [TRAIN] epoch=50, iter=18400/80000, loss=0.1949, lr=0.007906, batch_cost=0.7607, reader_cost=0.0003 | ETA 13:01:02 2020-10-30 17:28:43 [INFO] [TRAIN] epoch=50, iter=18500/80000, loss=0.2038, lr=0.007895, batch_cost=0.7591, reader_cost=0.0005 | ETA 12:58:04 2020-10-30 17:29:59 [INFO] [TRAIN] epoch=50, iter=18600/80000, loss=0.1849, lr=0.007883, batch_cost=0.7558, reader_cost=0.0007 | ETA 12:53:23 2020-10-30 17:31:16 [INFO] [TRAIN] epoch=51, iter=18700/80000, loss=0.1800, lr=0.007871, batch_cost=0.7706, reader_cost=0.0085 | ETA 13:07:19 2020-10-30 17:32:32 [INFO] [TRAIN] epoch=51, iter=18800/80000, loss=0.1985, lr=0.007860, batch_cost=0.7670, reader_cost=0.0010 | ETA 13:02:19 2020-10-30 17:33:49 [INFO] [TRAIN] epoch=51, iter=18900/80000, loss=0.1866, lr=0.007848, batch_cost=0.7656, reader_cost=0.0011 | ETA 12:59:36 2020-10-30 17:35:07 [INFO] [TRAIN] epoch=52, iter=19000/80000, loss=0.1915, lr=0.007837, batch_cost=0.7753, reader_cost=0.0076 | ETA 13:08:13 2020-10-30 17:36:23 [INFO] [TRAIN] epoch=52, iter=19100/80000, loss=0.2000, lr=0.007825, batch_cost=0.7678, reader_cost=0.0009 | ETA 12:59:17 2020-10-30 17:37:40 [INFO] [TRAIN] epoch=52, iter=19200/80000, loss=0.1876, lr=0.007814, batch_cost=0.7633, reader_cost=0.0005 | ETA 12:53:30 2020-10-30 17:38:55 [INFO] [TRAIN] epoch=52, iter=19300/80000, loss=0.2411, lr=0.007802, batch_cost=0.7584, reader_cost=0.0004 | ETA 12:47:15 2020-10-30 17:40:12 [INFO] [TRAIN] epoch=53, iter=19400/80000, loss=0.2265, lr=0.007791, batch_cost=0.7688, reader_cost=0.0081 | ETA 12:56:29 2020-10-30 17:41:28 [INFO] [TRAIN] epoch=53, iter=19500/80000, loss=0.2169, lr=0.007779, batch_cost=0.7519, reader_cost=0.0005 | ETA 12:38:09 2020-10-30 17:42:42 [INFO] [TRAIN] epoch=53, iter=19600/80000, loss=0.1878, lr=0.007768, batch_cost=0.7487, reader_cost=0.0007 | ETA 12:33:43 2020-10-30 17:43:57 [INFO] [TRAIN] epoch=53, iter=19700/80000, loss=0.2459, lr=0.007756, batch_cost=0.7492, reader_cost=0.0004 | ETA 12:32:57 2020-10-30 17:45:14 [INFO] [TRAIN] epoch=54, iter=19800/80000, loss=0.2051, lr=0.007744, batch_cost=0.7621, reader_cost=0.0082 | ETA 12:44:39 2020-10-30 17:46:30 [INFO] [TRAIN] epoch=54, iter=19900/80000, loss=0.1876, lr=0.007733, batch_cost=0.7598, reader_cost=0.0006 | ETA 12:41:03 2020-10-30 17:47:45 [INFO] [TRAIN] epoch=54, iter=20000/80000, loss=0.1992, lr=0.007721, batch_cost=0.7544, reader_cost=0.0004 | ETA 12:34:21 2020-10-30 17:49:01 [INFO] [TRAIN] epoch=55, iter=20100/80000, loss=0.2128, lr=0.007710, batch_cost=0.7643, reader_cost=0.0091 | ETA 12:43:02 2020-10-30 17:50:17 [INFO] [TRAIN] epoch=55, iter=20200/80000, loss=0.2325, lr=0.007698, batch_cost=0.7573, reader_cost=0.0009 | ETA 12:34:45 2020-10-30 17:51:34 [INFO] [TRAIN] epoch=55, iter=20300/80000, loss=0.2173, lr=0.007687, batch_cost=0.7658, reader_cost=0.0011 | ETA 12:41:59 2020-10-30 17:52:50 [INFO] [TRAIN] epoch=55, iter=20400/80000, loss=0.1693, lr=0.007675, batch_cost=0.7648, reader_cost=0.0009 | ETA 12:39:43 2020-10-30 17:54:08 [INFO] [TRAIN] epoch=56, iter=20500/80000, loss=0.2204, lr=0.007663, batch_cost=0.7736, reader_cost=0.0091 | ETA 12:47:11 2020-10-30 17:55:24 [INFO] [TRAIN] epoch=56, iter=20600/80000, loss=0.2785, lr=0.007652, batch_cost=0.7661, reader_cost=0.0003 | ETA 12:38:26 2020-10-30 17:56:40 [INFO] [TRAIN] epoch=56, iter=20700/80000, loss=0.1991, lr=0.007640, batch_cost=0.7604, reader_cost=0.0004 | ETA 12:31:32 2020-10-30 17:57:56 [INFO] [TRAIN] epoch=56, iter=20800/80000, loss=0.2133, lr=0.007629, batch_cost=0.7608, reader_cost=0.0009 | ETA 12:30:36 2020-10-30 17:59:13 [INFO] [TRAIN] epoch=57, iter=20900/80000, loss=0.1904, lr=0.007617, batch_cost=0.7675, reader_cost=0.0090 | ETA 12:35:56 2020-10-30 18:00:29 [INFO] [TRAIN] epoch=57, iter=21000/80000, loss=0.1790, lr=0.007606, batch_cost=0.7597, reader_cost=0.0005 | ETA 12:27:04 2020-10-30 18:01:45 [INFO] [TRAIN] epoch=57, iter=21100/80000, loss=0.1761, lr=0.007594, batch_cost=0.7574, reader_cost=0.0005 | ETA 12:23:33 2020-10-30 18:03:00 [INFO] [TRAIN] epoch=57, iter=21200/80000, loss=0.1941, lr=0.007582, batch_cost=0.7566, reader_cost=0.0003 | ETA 12:21:27 2020-10-30 18:04:17 [INFO] [TRAIN] epoch=58, iter=21300/80000, loss=0.2043, lr=0.007571, batch_cost=0.7634, reader_cost=0.0089 | ETA 12:26:51 2020-10-30 18:05:32 [INFO] [TRAIN] epoch=58, iter=21400/80000, loss=0.2039, lr=0.007559, batch_cost=0.7569, reader_cost=0.0002 | ETA 12:19:11 2020-10-30 18:06:48 [INFO] [TRAIN] epoch=58, iter=21500/80000, loss=0.1973, lr=0.007548, batch_cost=0.7598, reader_cost=0.0003 | ETA 12:20:46 2020-10-30 18:08:06 [INFO] [TRAIN] epoch=59, iter=21600/80000, loss=0.1760, lr=0.007536, batch_cost=0.7768, reader_cost=0.0083 | ETA 12:36:02 2020-10-30 18:09:22 [INFO] [TRAIN] epoch=59, iter=21700/80000, loss=0.2088, lr=0.007524, batch_cost=0.7592, reader_cost=0.0003 | ETA 12:17:43 2020-10-30 18:10:38 [INFO] [TRAIN] epoch=59, iter=21800/80000, loss=0.1666, lr=0.007513, batch_cost=0.7591, reader_cost=0.0003 | ETA 12:16:19 2020-10-30 18:11:54 [INFO] [TRAIN] epoch=59, iter=21900/80000, loss=0.2039, lr=0.007501, batch_cost=0.7609, reader_cost=0.0004 | ETA 12:16:50 2020-10-30 18:13:11 [INFO] [TRAIN] epoch=60, iter=22000/80000, loss=0.2411, lr=0.007490, batch_cost=0.7741, reader_cost=0.0081 | ETA 12:28:18 2020-10-30 18:14:28 [INFO] [TRAIN] epoch=60, iter=22100/80000, loss=0.2088, lr=0.007478, batch_cost=0.7638, reader_cost=0.0003 | ETA 12:17:05 2020-10-30 18:15:44 [INFO] [TRAIN] epoch=60, iter=22200/80000, loss=0.1995, lr=0.007466, batch_cost=0.7577, reader_cost=0.0003 | ETA 12:09:53 2020-10-30 18:17:01 [INFO] [TRAIN] epoch=60, iter=22300/80000, loss=0.1821, lr=0.007455, batch_cost=0.7694, reader_cost=0.0002 | ETA 12:19:52 2020-10-30 18:18:17 [INFO] [TRAIN] epoch=61, iter=22400/80000, loss=0.1852, lr=0.007443, batch_cost=0.7680, reader_cost=0.0086 | ETA 12:17:15 2020-10-30 18:19:33 [INFO] [TRAIN] epoch=61, iter=22500/80000, loss=0.2111, lr=0.007432, batch_cost=0.7564, reader_cost=0.0005 | ETA 12:04:55 2020-10-30 18:20:49 [INFO] [TRAIN] epoch=61, iter=22600/80000, loss=0.2106, lr=0.007420, batch_cost=0.7585, reader_cost=0.0005 | ETA 12:05:36 2020-10-30 18:22:06 [INFO] [TRAIN] epoch=62, iter=22700/80000, loss=0.2126, lr=0.007408, batch_cost=0.7696, reader_cost=0.0104 | ETA 12:15:00 2020-10-30 18:23:21 [INFO] [TRAIN] epoch=62, iter=22800/80000, loss=0.2207, lr=0.007397, batch_cost=0.7568, reader_cost=0.0003 | ETA 12:01:28 2020-10-30 18:24:38 [INFO] [TRAIN] epoch=62, iter=22900/80000, loss=0.1717, lr=0.007385, batch_cost=0.7631, reader_cost=0.0005 | ETA 12:06:13 2020-10-30 18:25:53 [INFO] [TRAIN] epoch=62, iter=23000/80000, loss=0.2463, lr=0.007373, batch_cost=0.7563, reader_cost=0.0003 | ETA 11:58:27 2020-10-30 18:27:11 [INFO] [TRAIN] epoch=63, iter=23100/80000, loss=0.1790, lr=0.007362, batch_cost=0.7748, reader_cost=0.0086 | ETA 12:14:47 2020-10-30 18:28:27 [INFO] [TRAIN] epoch=63, iter=23200/80000, loss=0.2479, lr=0.007350, batch_cost=0.7653, reader_cost=0.0006 | ETA 12:04:27 2020-10-30 18:29:44 [INFO] [TRAIN] epoch=63, iter=23300/80000, loss=0.2268, lr=0.007339, batch_cost=0.7627, reader_cost=0.0011 | ETA 12:00:46 2020-10-30 18:31:00 [INFO] [TRAIN] epoch=63, iter=23400/80000, loss=0.2549, lr=0.007327, batch_cost=0.7650, reader_cost=0.0008 | ETA 12:01:41 2020-10-30 18:32:17 [INFO] [TRAIN] epoch=64, iter=23500/80000, loss=0.1839, lr=0.007315, batch_cost=0.7693, reader_cost=0.0088 | ETA 12:04:25 2020-10-30 18:33:33 [INFO] [TRAIN] epoch=64, iter=23600/80000, loss=0.1918, lr=0.007304, batch_cost=0.7613, reader_cost=0.0003 | ETA 11:55:37 2020-10-30 18:34:49 [INFO] [TRAIN] epoch=64, iter=23700/80000, loss=0.1842, lr=0.007292, batch_cost=0.7604, reader_cost=0.0004 | ETA 11:53:31 2020-10-30 18:36:05 [INFO] [TRAIN] epoch=64, iter=23800/80000, loss=0.1962, lr=0.007280, batch_cost=0.7617, reader_cost=0.0008 | ETA 11:53:26 2020-10-30 18:37:23 [INFO] [TRAIN] epoch=65, iter=23900/80000, loss=0.1919, lr=0.007269, batch_cost=0.7715, reader_cost=0.0092 | ETA 12:01:20 2020-10-30 18:38:39 [INFO] [TRAIN] epoch=65, iter=24000/80000, loss=0.2146, lr=0.007257, batch_cost=0.7623, reader_cost=0.0004 | ETA 11:51:26 2020-10-30 18:38:45 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-10-30 18:45:02 [INFO] [EVAL] #Images=500 mIoU=0.7484 Acc=0.9522 Kappa=0.9381 2020-10-30 18:45:02 [INFO] [EVAL] Category IoU: [0.9677 0.8014 0.9136 0.4709 0.5735 0.6287 0.702 0.779 0.9143 0.6109 0.9431 0.8095 0.6019 0.9382 0.6509 0.8305 0.7001 0.618 0.7648] 2020-10-30 18:45:02 [INFO] [EVAL] Category Acc: [0.9884 0.8597 0.9485 0.7723 0.8181 0.7933 0.8256 0.9136 0.9465 0.9148 0.9599 0.8845 0.7167 0.9604 0.9269 0.9371 0.806 0.7986 0.8838] 2020-10-30 18:45:06 [INFO] [EVAL] The model with the best validation mIoU (0.7484) was saved at iter 24000. 2020-10-30 18:46:20 [INFO] [TRAIN] epoch=65, iter=24100/80000, loss=0.2118, lr=0.007245, batch_cost=0.7438, reader_cost=0.0003 | ETA 11:32:57 2020-10-30 18:47:36 [INFO] [TRAIN] epoch=66, iter=24200/80000, loss=0.1899, lr=0.007234, batch_cost=0.7542, reader_cost=0.0094 | ETA 11:41:22 2020-10-30 18:48:51 [INFO] [TRAIN] epoch=66, iter=24300/80000, loss=0.1952, lr=0.007222, batch_cost=0.7512, reader_cost=0.0005 | ETA 11:37:19 2020-10-30 18:50:07 [INFO] [TRAIN] epoch=66, iter=24400/80000, loss=0.1726, lr=0.007210, batch_cost=0.7600, reader_cost=0.0004 | ETA 11:44:17 2020-10-30 18:51:23 [INFO] [TRAIN] epoch=66, iter=24500/80000, loss=0.1903, lr=0.007199, batch_cost=0.7605, reader_cost=0.0005 | ETA 11:43:26 2020-10-30 18:52:40 [INFO] [TRAIN] epoch=67, iter=24600/80000, loss=0.1726, lr=0.007187, batch_cost=0.7718, reader_cost=0.0099 | ETA 11:52:35 2020-10-30 18:53:56 [INFO] [TRAIN] epoch=67, iter=24700/80000, loss=0.2012, lr=0.007175, batch_cost=0.7595, reader_cost=0.0007 | ETA 11:39:58 2020-10-30 18:55:13 [INFO] [TRAIN] epoch=67, iter=24800/80000, loss=0.1745, lr=0.007164, batch_cost=0.7651, reader_cost=0.0004 | ETA 11:43:52 2020-10-30 18:56:29 [INFO] [TRAIN] epoch=67, iter=24900/80000, loss=0.1986, lr=0.007152, batch_cost=0.7616, reader_cost=0.0014 | ETA 11:39:22 2020-10-30 18:57:46 [INFO] [TRAIN] epoch=68, iter=25000/80000, loss=0.1857, lr=0.007140, batch_cost=0.7687, reader_cost=0.0082 | ETA 11:44:38 2020-10-30 18:59:02 [INFO] [TRAIN] epoch=68, iter=25100/80000, loss=0.1783, lr=0.007129, batch_cost=0.7631, reader_cost=0.0002 | ETA 11:38:15 2020-10-30 19:00:18 [INFO] [TRAIN] epoch=68, iter=25200/80000, loss=0.2308, lr=0.007117, batch_cost=0.7604, reader_cost=0.0003 | ETA 11:34:32 2020-10-30 19:01:35 [INFO] [TRAIN] epoch=69, iter=25300/80000, loss=0.1955, lr=0.007105, batch_cost=0.7689, reader_cost=0.0092 | ETA 11:41:01 2020-10-30 19:02:51 [INFO] [TRAIN] epoch=69, iter=25400/80000, loss=0.1696, lr=0.007094, batch_cost=0.7584, reader_cost=0.0005 | ETA 11:30:07 2020-10-30 19:04:07 [INFO] [TRAIN] epoch=69, iter=25500/80000, loss=0.1988, lr=0.007082, batch_cost=0.7641, reader_cost=0.0004 | ETA 11:34:03 2020-10-30 19:05:23 [INFO] [TRAIN] epoch=69, iter=25600/80000, loss=0.2045, lr=0.007070, batch_cost=0.7593, reader_cost=0.0005 | ETA 11:28:25 2020-10-30 19:06:40 [INFO] [TRAIN] epoch=70, iter=25700/80000, loss=0.1936, lr=0.007059, batch_cost=0.7702, reader_cost=0.0086 | ETA 11:37:04 2020-10-30 19:07:56 [INFO] [TRAIN] epoch=70, iter=25800/80000, loss=0.1931, lr=0.007047, batch_cost=0.7620, reader_cost=0.0003 | ETA 11:28:22 2020-10-30 19:09:13 [INFO] [TRAIN] epoch=70, iter=25900/80000, loss=0.1789, lr=0.007035, batch_cost=0.7620, reader_cost=0.0003 | ETA 11:27:02 2020-10-30 19:10:29 [INFO] [TRAIN] epoch=70, iter=26000/80000, loss=0.2058, lr=0.007024, batch_cost=0.7655, reader_cost=0.0009 | ETA 11:28:54 2020-10-30 19:11:46 [INFO] [TRAIN] epoch=71, iter=26100/80000, loss=0.2003, lr=0.007012, batch_cost=0.7707, reader_cost=0.0078 | ETA 11:32:19 2020-10-30 19:13:02 [INFO] [TRAIN] epoch=71, iter=26200/80000, loss=0.2081, lr=0.007000, batch_cost=0.7588, reader_cost=0.0003 | ETA 11:20:21 2020-10-30 19:14:18 [INFO] [TRAIN] epoch=71, iter=26300/80000, loss=0.1749, lr=0.006989, batch_cost=0.7573, reader_cost=0.0009 | ETA 11:17:45 2020-10-30 19:15:34 [INFO] [TRAIN] epoch=71, iter=26400/80000, loss=0.1999, lr=0.006977, batch_cost=0.7584, reader_cost=0.0005 | ETA 11:17:28 2020-10-30 19:16:51 [INFO] [TRAIN] epoch=72, iter=26500/80000, loss=0.1899, lr=0.006965, batch_cost=0.7703, reader_cost=0.0093 | ETA 11:26:49 2020-10-30 19:18:07 [INFO] [TRAIN] epoch=72, iter=26600/80000, loss=0.2178, lr=0.006954, batch_cost=0.7603, reader_cost=0.0003 | ETA 11:16:37 2020-10-30 19:19:23 [INFO] [TRAIN] epoch=72, iter=26700/80000, loss=0.2039, lr=0.006942, batch_cost=0.7597, reader_cost=0.0010 | ETA 11:14:50 2020-10-30 19:20:39 [INFO] [TRAIN] epoch=73, iter=26800/80000, loss=0.1676, lr=0.006930, batch_cost=0.7673, reader_cost=0.0091 | ETA 11:20:21 2020-10-30 19:21:55 [INFO] [TRAIN] epoch=73, iter=26900/80000, loss=0.1979, lr=0.006918, batch_cost=0.7604, reader_cost=0.0008 | ETA 11:12:55 2020-10-30 19:23:12 [INFO] [TRAIN] epoch=73, iter=27000/80000, loss=0.1741, lr=0.006907, batch_cost=0.7633, reader_cost=0.0009 | ETA 11:14:16 2020-10-30 19:24:27 [INFO] [TRAIN] epoch=73, iter=27100/80000, loss=0.1743, lr=0.006895, batch_cost=0.7560, reader_cost=0.0004 | ETA 11:06:32 2020-10-30 19:25:44 [INFO] [TRAIN] epoch=74, iter=27200/80000, loss=0.1912, lr=0.006883, batch_cost=0.7688, reader_cost=0.0089 | ETA 11:16:34 2020-10-30 19:27:00 [INFO] [TRAIN] epoch=74, iter=27300/80000, loss=0.1981, lr=0.006872, batch_cost=0.7590, reader_cost=0.0006 | ETA 11:06:36 2020-10-30 19:28:17 [INFO] [TRAIN] epoch=74, iter=27400/80000, loss=0.1661, lr=0.006860, batch_cost=0.7646, reader_cost=0.0004 | ETA 11:10:17 2020-10-30 19:29:33 [INFO] [TRAIN] epoch=74, iter=27500/80000, loss=0.1933, lr=0.006848, batch_cost=0.7595, reader_cost=0.0004 | ETA 11:04:32 2020-10-30 19:30:49 [INFO] [TRAIN] epoch=75, iter=27600/80000, loss=0.1813, lr=0.006836, batch_cost=0.7634, reader_cost=0.0082 | ETA 11:06:40 2020-10-30 19:32:05 [INFO] [TRAIN] epoch=75, iter=27700/80000, loss=0.1879, lr=0.006825, batch_cost=0.7563, reader_cost=0.0003 | ETA 10:59:13 2020-10-30 19:33:21 [INFO] [TRAIN] epoch=75, iter=27800/80000, loss=0.1858, lr=0.006813, batch_cost=0.7640, reader_cost=0.0009 | ETA 11:04:43 2020-10-30 19:34:37 [INFO] [TRAIN] epoch=75, iter=27900/80000, loss=0.1833, lr=0.006801, batch_cost=0.7632, reader_cost=0.0003 | ETA 11:02:41 2020-10-30 19:35:54 [INFO] [TRAIN] epoch=76, iter=28000/80000, loss=0.2336, lr=0.006789, batch_cost=0.7714, reader_cost=0.0089 | ETA 11:08:30 2020-10-30 19:37:11 [INFO] [TRAIN] epoch=76, iter=28100/80000, loss=0.1727, lr=0.006778, batch_cost=0.7613, reader_cost=0.0004 | ETA 10:58:33 2020-10-30 19:38:27 [INFO] [TRAIN] epoch=76, iter=28200/80000, loss=0.1878, lr=0.006766, batch_cost=0.7606, reader_cost=0.0003 | ETA 10:56:37 2020-10-30 19:39:44 [INFO] [TRAIN] epoch=77, iter=28300/80000, loss=0.1783, lr=0.006754, batch_cost=0.7724, reader_cost=0.0088 | ETA 11:05:34 2020-10-30 19:41:00 [INFO] [TRAIN] epoch=77, iter=28400/80000, loss=0.1917, lr=0.006743, batch_cost=0.7573, reader_cost=0.0003 | ETA 10:51:18 2020-10-30 19:42:15 [INFO] [TRAIN] epoch=77, iter=28500/80000, loss=0.1623, lr=0.006731, batch_cost=0.7580, reader_cost=0.0003 | ETA 10:50:37 2020-10-30 19:43:32 [INFO] [TRAIN] epoch=77, iter=28600/80000, loss=0.2142, lr=0.006719, batch_cost=0.7642, reader_cost=0.0007 | ETA 10:54:37 2020-10-30 19:44:49 [INFO] [TRAIN] epoch=78, iter=28700/80000, loss=0.1706, lr=0.006707, batch_cost=0.7737, reader_cost=0.0102 | ETA 11:01:29 2020-10-30 19:46:05 [INFO] [TRAIN] epoch=78, iter=28800/80000, loss=0.1851, lr=0.006696, batch_cost=0.7618, reader_cost=0.0007 | ETA 10:50:02 2020-10-30 19:47:22 [INFO] [TRAIN] epoch=78, iter=28900/80000, loss=0.1724, lr=0.006684, batch_cost=0.7641, reader_cost=0.0010 | ETA 10:50:44 2020-10-30 19:48:38 [INFO] [TRAIN] epoch=78, iter=29000/80000, loss=0.1829, lr=0.006672, batch_cost=0.7630, reader_cost=0.0011 | ETA 10:48:32 2020-10-30 19:49:55 [INFO] [TRAIN] epoch=79, iter=29100/80000, loss=0.1725, lr=0.006660, batch_cost=0.7708, reader_cost=0.0089 | ETA 10:53:53 2020-10-30 19:51:12 [INFO] [TRAIN] epoch=79, iter=29200/80000, loss=0.1983, lr=0.006648, batch_cost=0.7653, reader_cost=0.0005 | ETA 10:47:58 2020-10-30 19:52:28 [INFO] [TRAIN] epoch=79, iter=29300/80000, loss=0.1714, lr=0.006637, batch_cost=0.7667, reader_cost=0.0005 | ETA 10:47:53 2020-10-30 19:53:46 [INFO] [TRAIN] epoch=80, iter=29400/80000, loss=0.1992, lr=0.006625, batch_cost=0.7786, reader_cost=0.0080 | ETA 10:56:36 2020-10-30 19:55:02 [INFO] [TRAIN] epoch=80, iter=29500/80000, loss=0.1827, lr=0.006613, batch_cost=0.7608, reader_cost=0.0002 | ETA 10:40:21 2020-10-30 19:56:18 [INFO] [TRAIN] epoch=80, iter=29600/80000, loss=0.1597, lr=0.006601, batch_cost=0.7555, reader_cost=0.0002 | ETA 10:34:35 2020-10-30 19:57:33 [INFO] [TRAIN] epoch=80, iter=29700/80000, loss=0.1707, lr=0.006590, batch_cost=0.7511, reader_cost=0.0002 | ETA 10:29:40 2020-10-30 19:58:49 [INFO] [TRAIN] epoch=81, iter=29800/80000, loss=0.1675, lr=0.006578, batch_cost=0.7594, reader_cost=0.0092 | ETA 10:35:21 2020-10-30 20:00:04 [INFO] [TRAIN] epoch=81, iter=29900/80000, loss=0.1613, lr=0.006566, batch_cost=0.7501, reader_cost=0.0005 | ETA 10:26:19 2020-10-30 20:01:19 [INFO] [TRAIN] epoch=81, iter=30000/80000, loss=0.1571, lr=0.006554, batch_cost=0.7556, reader_cost=0.0003 | ETA 10:29:42 2020-10-30 20:02:36 [INFO] [TRAIN] epoch=81, iter=30100/80000, loss=0.2038, lr=0.006543, batch_cost=0.7647, reader_cost=0.0004 | ETA 10:35:56 2020-10-30 20:03:53 [INFO] [TRAIN] epoch=82, iter=30200/80000, loss=0.1817, lr=0.006531, batch_cost=0.7743, reader_cost=0.0091 | ETA 10:42:40 2020-10-30 20:05:09 [INFO] [TRAIN] epoch=82, iter=30300/80000, loss=0.1692, lr=0.006519, batch_cost=0.7571, reader_cost=0.0003 | ETA 10:27:09 2020-10-30 20:06:25 [INFO] [TRAIN] epoch=82, iter=30400/80000, loss=0.1958, lr=0.006507, batch_cost=0.7613, reader_cost=0.0002 | ETA 10:29:22 2020-10-30 20:07:41 [INFO] [TRAIN] epoch=82, iter=30500/80000, loss=0.2032, lr=0.006495, batch_cost=0.7596, reader_cost=0.0010 | ETA 10:26:39 2020-10-30 20:08:57 [INFO] [TRAIN] epoch=83, iter=30600/80000, loss=0.1665, lr=0.006484, batch_cost=0.7630, reader_cost=0.0092 | ETA 10:28:13 2020-10-30 20:10:14 [INFO] [TRAIN] epoch=83, iter=30700/80000, loss=0.1874, lr=0.006472, batch_cost=0.7628, reader_cost=0.0004 | ETA 10:26:44 2020-10-30 20:11:30 [INFO] [TRAIN] epoch=83, iter=30800/80000, loss=0.1971, lr=0.006460, batch_cost=0.7594, reader_cost=0.0004 | ETA 10:22:44 2020-10-30 20:12:47 [INFO] [TRAIN] epoch=84, iter=30900/80000, loss=0.1759, lr=0.006448, batch_cost=0.7751, reader_cost=0.0086 | ETA 10:34:16 2020-10-30 20:14:03 [INFO] [TRAIN] epoch=84, iter=31000/80000, loss=0.2028, lr=0.006436, batch_cost=0.7594, reader_cost=0.0004 | ETA 10:20:12 2020-10-30 20:15:19 [INFO] [TRAIN] epoch=84, iter=31100/80000, loss=0.1600, lr=0.006425, batch_cost=0.7607, reader_cost=0.0003 | ETA 10:20:00 2020-10-30 20:16:35 [INFO] [TRAIN] epoch=84, iter=31200/80000, loss=0.1723, lr=0.006413, batch_cost=0.7617, reader_cost=0.0007 | ETA 10:19:29 2020-10-30 20:17:52 [INFO] [TRAIN] epoch=85, iter=31300/80000, loss=0.1885, lr=0.006401, batch_cost=0.7699, reader_cost=0.0090 | ETA 10:24:51 2020-10-30 20:19:09 [INFO] [TRAIN] epoch=85, iter=31400/80000, loss=0.1907, lr=0.006389, batch_cost=0.7636, reader_cost=0.0002 | ETA 10:18:33 2020-10-30 20:20:25 [INFO] [TRAIN] epoch=85, iter=31500/80000, loss=0.1600, lr=0.006377, batch_cost=0.7596, reader_cost=0.0002 | ETA 10:14:02 2020-10-30 20:21:41 [INFO] [TRAIN] epoch=85, iter=31600/80000, loss=0.1996, lr=0.006366, batch_cost=0.7624, reader_cost=0.0002 | ETA 10:15:00 2020-10-30 20:22:58 [INFO] [TRAIN] epoch=86, iter=31700/80000, loss=0.1923, lr=0.006354, batch_cost=0.7711, reader_cost=0.0079 | ETA 10:20:44 2020-10-30 20:24:14 [INFO] [TRAIN] epoch=86, iter=31800/80000, loss=0.1938, lr=0.006342, batch_cost=0.7559, reader_cost=0.0002 | ETA 10:07:14 2020-10-30 20:25:30 [INFO] [TRAIN] epoch=86, iter=31900/80000, loss=0.1663, lr=0.006330, batch_cost=0.7595, reader_cost=0.0003 | ETA 10:08:54 2020-10-30 20:26:46 [INFO] [TRAIN] epoch=87, iter=32000/80000, loss=0.1780, lr=0.006318, batch_cost=0.7676, reader_cost=0.0083 | ETA 10:14:03 2020-10-30 20:26:53 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-10-30 20:33:07 [INFO] [EVAL] #Images=500 mIoU=0.7754 Acc=0.9591 Kappa=0.9469 2020-10-30 20:33:07 [INFO] [EVAL] Category IoU: [0.9762 0.8272 0.9231 0.5168 0.6324 0.6549 0.7258 0.7973 0.9247 0.6806 0.9461 0.8287 0.6211 0.9499 0.7831 0.8254 0.7091 0.6367 0.7733] 2020-10-30 20:33:07 [INFO] [EVAL] Category Acc: [0.9865 0.9219 0.9583 0.8785 0.7873 0.7837 0.8421 0.9037 0.951 0.8587 0.9688 0.8914 0.7824 0.9717 0.9599 0.9413 0.8537 0.8435 0.8622] 2020-10-30 20:33:10 [INFO] [EVAL] The model with the best validation mIoU (0.7754) was saved at iter 32000. 2020-10-30 20:34:25 [INFO] [TRAIN] epoch=87, iter=32100/80000, loss=0.2017, lr=0.006306, batch_cost=0.7503, reader_cost=0.0007 | ETA 09:58:57 2020-10-30 20:35:41 [INFO] [TRAIN] epoch=87, iter=32200/80000, loss=0.1690, lr=0.006295, batch_cost=0.7572, reader_cost=0.0002 | ETA 10:03:12 2020-10-30 20:36:57 [INFO] [TRAIN] epoch=87, iter=32300/80000, loss=0.1949, lr=0.006283, batch_cost=0.7575, reader_cost=0.0003 | ETA 10:02:11 2020-10-30 20:38:13 [INFO] [TRAIN] epoch=88, iter=32400/80000, loss=0.2025, lr=0.006271, batch_cost=0.7631, reader_cost=0.0083 | ETA 10:05:25 2020-10-30 20:39:29 [INFO] [TRAIN] epoch=88, iter=32500/80000, loss=0.1807, lr=0.006259, batch_cost=0.7599, reader_cost=0.0003 | ETA 10:01:33 2020-10-30 20:40:45 [INFO] [TRAIN] epoch=88, iter=32600/80000, loss=0.1662, lr=0.006247, batch_cost=0.7588, reader_cost=0.0006 | ETA 09:59:27 2020-10-30 20:42:01 [INFO] [TRAIN] epoch=88, iter=32700/80000, loss=0.1664, lr=0.006235, batch_cost=0.7583, reader_cost=0.0004 | ETA 09:57:48 2020-10-30 20:43:18 [INFO] [TRAIN] epoch=89, iter=32800/80000, loss=0.1661, lr=0.006224, batch_cost=0.7677, reader_cost=0.0085 | ETA 10:03:54 2020-10-30 20:44:33 [INFO] [TRAIN] epoch=89, iter=32900/80000, loss=0.1790, lr=0.006212, batch_cost=0.7591, reader_cost=0.0002 | ETA 09:55:55 2020-10-30 20:45:49 [INFO] [TRAIN] epoch=89, iter=33000/80000, loss=0.1727, lr=0.006200, batch_cost=0.7599, reader_cost=0.0004 | ETA 09:55:16 2020-10-30 20:47:06 [INFO] [TRAIN] epoch=89, iter=33100/80000, loss=0.1783, lr=0.006188, batch_cost=0.7619, reader_cost=0.0002 | ETA 09:55:30 2020-10-30 20:48:22 [INFO] [TRAIN] epoch=90, iter=33200/80000, loss=0.1652, lr=0.006176, batch_cost=0.7684, reader_cost=0.0086 | ETA 09:59:22 2020-10-30 20:49:38 [INFO] [TRAIN] epoch=90, iter=33300/80000, loss=0.1761, lr=0.006164, batch_cost=0.7581, reader_cost=0.0002 | ETA 09:50:05 2020-10-30 20:50:55 [INFO] [TRAIN] epoch=90, iter=33400/80000, loss=0.1755, lr=0.006152, batch_cost=0.7646, reader_cost=0.0003 | ETA 09:53:52 2020-10-30 20:52:12 [INFO] [TRAIN] epoch=91, iter=33500/80000, loss=0.1807, lr=0.006141, batch_cost=0.7736, reader_cost=0.0091 | ETA 09:59:33 2020-10-30 20:53:28 [INFO] [TRAIN] epoch=91, iter=33600/80000, loss=0.2497, lr=0.006129, batch_cost=0.7579, reader_cost=0.0003 | ETA 09:46:05 2020-10-30 20:54:44 [INFO] [TRAIN] epoch=91, iter=33700/80000, loss=0.1616, lr=0.006117, batch_cost=0.7583, reader_cost=0.0002 | ETA 09:45:09 2020-10-30 20:56:00 [INFO] [TRAIN] epoch=91, iter=33800/80000, loss=0.1910, lr=0.006105, batch_cost=0.7604, reader_cost=0.0005 | ETA 09:45:29 2020-10-30 20:57:17 [INFO] [TRAIN] epoch=92, iter=33900/80000, loss=0.1540, lr=0.006093, batch_cost=0.7672, reader_cost=0.0105 | ETA 09:49:26 2020-10-30 20:58:33 [INFO] [TRAIN] epoch=92, iter=34000/80000, loss=0.1688, lr=0.006081, batch_cost=0.7630, reader_cost=0.0008 | ETA 09:44:55 2020-10-30 20:59:49 [INFO] [TRAIN] epoch=92, iter=34100/80000, loss=0.1608, lr=0.006069, batch_cost=0.7612, reader_cost=0.0008 | ETA 09:42:21 2020-10-30 21:01:06 [INFO] [TRAIN] epoch=92, iter=34200/80000, loss=0.1648, lr=0.006057, batch_cost=0.7693, reader_cost=0.0005 | ETA 09:47:15 2020-10-30 21:02:23 [INFO] [TRAIN] epoch=93, iter=34300/80000, loss=0.1734, lr=0.006046, batch_cost=0.7685, reader_cost=0.0087 | ETA 09:45:22 2020-10-30 21:03:39 [INFO] [TRAIN] epoch=93, iter=34400/80000, loss=0.1598, lr=0.006034, batch_cost=0.7601, reader_cost=0.0002 | ETA 09:37:38 2020-10-30 21:04:55 [INFO] [TRAIN] epoch=93, iter=34500/80000, loss=0.1638, lr=0.006022, batch_cost=0.7583, reader_cost=0.0006 | ETA 09:35:02 2020-10-30 21:06:11 [INFO] [TRAIN] epoch=94, iter=34600/80000, loss=0.1778, lr=0.006010, batch_cost=0.7676, reader_cost=0.0082 | ETA 09:40:50 2020-10-30 21:07:27 [INFO] [TRAIN] epoch=94, iter=34700/80000, loss=0.1734, lr=0.005998, batch_cost=0.7588, reader_cost=0.0010 | ETA 09:32:52 2020-10-30 21:08:43 [INFO] [TRAIN] epoch=94, iter=34800/80000, loss=0.1861, lr=0.005986, batch_cost=0.7590, reader_cost=0.0007 | ETA 09:31:47 2020-10-30 21:09:59 [INFO] [TRAIN] epoch=94, iter=34900/80000, loss=0.1653, lr=0.005974, batch_cost=0.7550, reader_cost=0.0006 | ETA 09:27:32 2020-10-30 21:11:14 [INFO] [TRAIN] epoch=95, iter=35000/80000, loss=0.1498, lr=0.005962, batch_cost=0.7555, reader_cost=0.0082 | ETA 09:26:39 2020-10-30 21:12:28 [INFO] [TRAIN] epoch=95, iter=35100/80000, loss=0.1701, lr=0.005950, batch_cost=0.7413, reader_cost=0.0002 | ETA 09:14:44 2020-10-30 21:13:44 [INFO] [TRAIN] epoch=95, iter=35200/80000, loss=0.1553, lr=0.005938, batch_cost=0.7534, reader_cost=0.0003 | ETA 09:22:30 2020-10-30 21:15:00 [INFO] [TRAIN] epoch=95, iter=35300/80000, loss=0.1801, lr=0.005927, batch_cost=0.7587, reader_cost=0.0007 | ETA 09:25:14 2020-10-30 21:16:17 [INFO] [TRAIN] epoch=96, iter=35400/80000, loss=0.1790, lr=0.005915, batch_cost=0.7700, reader_cost=0.0095 | ETA 09:32:20 2020-10-30 21:17:33 [INFO] [TRAIN] epoch=96, iter=35500/80000, loss=0.3061, lr=0.005903, batch_cost=0.7608, reader_cost=0.0012 | ETA 09:24:17 2020-10-30 21:18:49 [INFO] [TRAIN] epoch=96, iter=35600/80000, loss=0.2062, lr=0.005891, batch_cost=0.7637, reader_cost=0.0005 | ETA 09:25:10 2020-10-30 21:20:06 [INFO] [TRAIN] epoch=96, iter=35700/80000, loss=0.1847, lr=0.005879, batch_cost=0.7659, reader_cost=0.0007 | ETA 09:25:30 2020-10-30 21:21:23 [INFO] [TRAIN] epoch=97, iter=35800/80000, loss=0.1960, lr=0.005867, batch_cost=0.7723, reader_cost=0.0082 | ETA 09:28:54 2020-10-30 21:22:39 [INFO] [TRAIN] epoch=97, iter=35900/80000, loss=0.1817, lr=0.005855, batch_cost=0.7631, reader_cost=0.0002 | ETA 09:20:50 2020-10-30 21:23:55 [INFO] [TRAIN] epoch=97, iter=36000/80000, loss=0.1655, lr=0.005843, batch_cost=0.7570, reader_cost=0.0003 | ETA 09:15:09 2020-10-30 21:25:12 [INFO] [TRAIN] epoch=98, iter=36100/80000, loss=0.1960, lr=0.005831, batch_cost=0.7672, reader_cost=0.0093 | ETA 09:21:21 2020-10-30 21:26:27 [INFO] [TRAIN] epoch=98, iter=36200/80000, loss=0.1634, lr=0.005819, batch_cost=0.7542, reader_cost=0.0003 | ETA 09:10:34 2020-10-30 21:27:43 [INFO] [TRAIN] epoch=98, iter=36300/80000, loss=0.1536, lr=0.005807, batch_cost=0.7616, reader_cost=0.0013 | ETA 09:14:41 2020-10-30 21:28:59 [INFO] [TRAIN] epoch=98, iter=36400/80000, loss=0.2190, lr=0.005795, batch_cost=0.7562, reader_cost=0.0008 | ETA 09:09:29 2020-10-30 21:30:15 [INFO] [TRAIN] epoch=99, iter=36500/80000, loss=0.1490, lr=0.005783, batch_cost=0.7675, reader_cost=0.0083 | ETA 09:16:25 2020-10-30 21:31:32 [INFO] [TRAIN] epoch=99, iter=36600/80000, loss=0.1762, lr=0.005771, batch_cost=0.7634, reader_cost=0.0010 | ETA 09:12:13 2020-10-30 21:32:48 [INFO] [TRAIN] epoch=99, iter=36700/80000, loss=0.1416, lr=0.005760, batch_cost=0.7599, reader_cost=0.0011 | ETA 09:08:22 2020-10-30 21:34:05 [INFO] [TRAIN] epoch=99, iter=36800/80000, loss=0.1585, lr=0.005748, batch_cost=0.7673, reader_cost=0.0006 | ETA 09:12:28 2020-10-30 21:35:21 [INFO] [TRAIN] epoch=100, iter=36900/80000, loss=0.1517, lr=0.005736, batch_cost=0.7632, reader_cost=0.0079 | ETA 09:08:15 2020-10-30 21:36:37 [INFO] [TRAIN] epoch=100, iter=37000/80000, loss=0.1954, lr=0.005724, batch_cost=0.7582, reader_cost=0.0003 | ETA 09:03:20 2020-10-30 21:37:53 [INFO] [TRAIN] epoch=100, iter=37100/80000, loss=0.1517, lr=0.005712, batch_cost=0.7598, reader_cost=0.0003 | ETA 09:03:13 2020-10-30 21:39:08 [INFO] [TRAIN] epoch=100, iter=37200/80000, loss=0.1867, lr=0.005700, batch_cost=0.7548, reader_cost=0.0003 | ETA 08:58:26 2020-10-30 21:40:25 [INFO] [TRAIN] epoch=101, iter=37300/80000, loss=0.2065, lr=0.005688, batch_cost=0.7655, reader_cost=0.0105 | ETA 09:04:45 2020-10-30 21:41:41 [INFO] [TRAIN] epoch=101, iter=37400/80000, loss=0.1700, lr=0.005676, batch_cost=0.7609, reader_cost=0.0007 | ETA 09:00:12 2020-10-30 21:42:57 [INFO] [TRAIN] epoch=101, iter=37500/80000, loss=0.1805, lr=0.005664, batch_cost=0.7615, reader_cost=0.0007 | ETA 08:59:24 2020-10-30 21:44:14 [INFO] [TRAIN] epoch=102, iter=37600/80000, loss=0.1977, lr=0.005652, batch_cost=0.7675, reader_cost=0.0091 | ETA 09:02:21 2020-10-30 21:45:30 [INFO] [TRAIN] epoch=102, iter=37700/80000, loss=0.1749, lr=0.005640, batch_cost=0.7633, reader_cost=0.0004 | ETA 08:58:09 2020-10-30 21:46:46 [INFO] [TRAIN] epoch=102, iter=37800/80000, loss=0.1567, lr=0.005628, batch_cost=0.7591, reader_cost=0.0003 | ETA 08:53:55 2020-10-30 21:48:02 [INFO] [TRAIN] epoch=102, iter=37900/80000, loss=0.1650, lr=0.005616, batch_cost=0.7607, reader_cost=0.0003 | ETA 08:53:44 2020-10-30 21:49:19 [INFO] [TRAIN] epoch=103, iter=38000/80000, loss=0.1441, lr=0.005604, batch_cost=0.7691, reader_cost=0.0095 | ETA 08:58:23 2020-10-30 21:50:35 [INFO] [TRAIN] epoch=103, iter=38100/80000, loss=0.1641, lr=0.005592, batch_cost=0.7614, reader_cost=0.0004 | ETA 08:51:41 2020-10-30 21:51:51 [INFO] [TRAIN] epoch=103, iter=38200/80000, loss=0.1568, lr=0.005580, batch_cost=0.7610, reader_cost=0.0009 | ETA 08:50:09 2020-10-30 21:53:07 [INFO] [TRAIN] epoch=103, iter=38300/80000, loss=0.1938, lr=0.005568, batch_cost=0.7621, reader_cost=0.0008 | ETA 08:49:38 2020-10-30 21:54:24 [INFO] [TRAIN] epoch=104, iter=38400/80000, loss=0.1612, lr=0.005556, batch_cost=0.7638, reader_cost=0.0091 | ETA 08:49:32 2020-10-30 21:55:40 [INFO] [TRAIN] epoch=104, iter=38500/80000, loss=0.1633, lr=0.005544, batch_cost=0.7594, reader_cost=0.0003 | ETA 08:45:16 2020-10-30 21:56:56 [INFO] [TRAIN] epoch=104, iter=38600/80000, loss=0.1762, lr=0.005532, batch_cost=0.7585, reader_cost=0.0005 | ETA 08:43:21 2020-10-30 21:58:13 [INFO] [TRAIN] epoch=105, iter=38700/80000, loss=0.1627, lr=0.005520, batch_cost=0.7698, reader_cost=0.0085 | ETA 08:49:51 2020-10-30 21:59:28 [INFO] [TRAIN] epoch=105, iter=38800/80000, loss=0.1604, lr=0.005508, batch_cost=0.7578, reader_cost=0.0009 | ETA 08:40:21 2020-10-30 22:00:45 [INFO] [TRAIN] epoch=105, iter=38900/80000, loss=0.1521, lr=0.005496, batch_cost=0.7622, reader_cost=0.0005 | ETA 08:42:05 2020-10-30 22:02:01 [INFO] [TRAIN] epoch=105, iter=39000/80000, loss=0.1519, lr=0.005484, batch_cost=0.7645, reader_cost=0.0002 | ETA 08:42:24 2020-10-30 22:03:18 [INFO] [TRAIN] epoch=106, iter=39100/80000, loss=0.1469, lr=0.005472, batch_cost=0.7708, reader_cost=0.0082 | ETA 08:45:26 2020-10-30 22:04:34 [INFO] [TRAIN] epoch=106, iter=39200/80000, loss=0.1547, lr=0.005460, batch_cost=0.7606, reader_cost=0.0002 | ETA 08:37:13 2020-10-30 22:05:50 [INFO] [TRAIN] epoch=106, iter=39300/80000, loss=0.1542, lr=0.005448, batch_cost=0.7556, reader_cost=0.0002 | ETA 08:32:31 2020-10-30 22:07:05 [INFO] [TRAIN] epoch=106, iter=39400/80000, loss=0.1463, lr=0.005436, batch_cost=0.7581, reader_cost=0.0003 | ETA 08:33:00 2020-10-30 22:08:22 [INFO] [TRAIN] epoch=107, iter=39500/80000, loss=0.1553, lr=0.005424, batch_cost=0.7647, reader_cost=0.0090 | ETA 08:36:12 2020-10-30 22:09:38 [INFO] [TRAIN] epoch=107, iter=39600/80000, loss=0.1838, lr=0.005412, batch_cost=0.7635, reader_cost=0.0004 | ETA 08:34:04 2020-10-30 22:10:54 [INFO] [TRAIN] epoch=107, iter=39700/80000, loss=0.1521, lr=0.005400, batch_cost=0.7603, reader_cost=0.0002 | ETA 08:30:38 2020-10-30 22:12:10 [INFO] [TRAIN] epoch=107, iter=39800/80000, loss=0.1581, lr=0.005388, batch_cost=0.7589, reader_cost=0.0003 | ETA 08:28:27 2020-10-30 22:13:26 [INFO] [TRAIN] epoch=108, iter=39900/80000, loss=0.1473, lr=0.005376, batch_cost=0.7535, reader_cost=0.0083 | ETA 08:23:35 2020-10-30 22:14:41 [INFO] [TRAIN] epoch=108, iter=40000/80000, loss=0.1521, lr=0.005364, batch_cost=0.7528, reader_cost=0.0005 | ETA 08:21:51 2020-10-30 22:14:48 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-10-30 22:21:01 [INFO] [EVAL] #Images=500 mIoU=0.7768 Acc=0.9607 Kappa=0.9490 2020-10-30 22:21:01 [INFO] [EVAL] Category IoU: [0.9806 0.8572 0.9222 0.4178 0.6236 0.6575 0.7228 0.8005 0.9254 0.6384 0.945 0.8273 0.6215 0.9536 0.8197 0.8521 0.7442 0.6642 0.7848] 2020-10-30 22:21:01 [INFO] [EVAL] Category Acc: [0.9903 0.9291 0.9529 0.8605 0.8012 0.8231 0.8597 0.9024 0.9496 0.9167 0.9613 0.8826 0.8106 0.9732 0.9516 0.9515 0.8316 0.8857 0.878 ] 2020-10-30 22:21:04 [INFO] [EVAL] The model with the best validation mIoU (0.7768) was saved at iter 40000. 2020-10-30 22:22:19 [INFO] [TRAIN] epoch=108, iter=40100/80000, loss=0.1643, lr=0.005352, batch_cost=0.7504, reader_cost=0.0003 | ETA 08:19:00 2020-10-30 22:23:36 [INFO] [TRAIN] epoch=109, iter=40200/80000, loss=0.1454, lr=0.005340, batch_cost=0.7671, reader_cost=0.0084 | ETA 08:28:50 2020-10-30 22:24:52 [INFO] [TRAIN] epoch=109, iter=40300/80000, loss=0.1594, lr=0.005327, batch_cost=0.7560, reader_cost=0.0003 | ETA 08:20:13 2020-10-30 22:26:08 [INFO] [TRAIN] epoch=109, iter=40400/80000, loss=0.1917, lr=0.005315, batch_cost=0.7602, reader_cost=0.0003 | ETA 08:21:43 2020-10-30 22:27:24 [INFO] [TRAIN] epoch=109, iter=40500/80000, loss=0.1897, lr=0.005303, batch_cost=0.7582, reader_cost=0.0003 | ETA 08:19:07 2020-10-30 22:28:40 [INFO] [TRAIN] epoch=110, iter=40600/80000, loss=0.1410, lr=0.005291, batch_cost=0.7681, reader_cost=0.0089 | ETA 08:24:23 2020-10-30 22:29:56 [INFO] [TRAIN] epoch=110, iter=40700/80000, loss=0.1812, lr=0.005279, batch_cost=0.7611, reader_cost=0.0008 | ETA 08:18:31 2020-10-30 22:31:13 [INFO] [TRAIN] epoch=110, iter=40800/80000, loss=0.1453, lr=0.005267, batch_cost=0.7647, reader_cost=0.0008 | ETA 08:19:35 2020-10-30 22:32:29 [INFO] [TRAIN] epoch=110, iter=40900/80000, loss=0.1502, lr=0.005255, batch_cost=0.7657, reader_cost=0.0005 | ETA 08:18:57 2020-10-30 22:33:46 [INFO] [TRAIN] epoch=111, iter=41000/80000, loss=0.1713, lr=0.005243, batch_cost=0.7679, reader_cost=0.0093 | ETA 08:19:07 2020-10-30 22:35:02 [INFO] [TRAIN] epoch=111, iter=41100/80000, loss=0.1568, lr=0.005231, batch_cost=0.7589, reader_cost=0.0003 | ETA 08:12:00 2020-10-30 22:36:18 [INFO] [TRAIN] epoch=111, iter=41200/80000, loss=0.1498, lr=0.005219, batch_cost=0.7582, reader_cost=0.0004 | ETA 08:10:20 2020-10-30 22:37:34 [INFO] [TRAIN] epoch=112, iter=41300/80000, loss=0.1640, lr=0.005207, batch_cost=0.7641, reader_cost=0.0091 | ETA 08:12:49 2020-10-30 22:38:51 [INFO] [TRAIN] epoch=112, iter=41400/80000, loss=0.1908, lr=0.005195, batch_cost=0.7618, reader_cost=0.0009 | ETA 08:10:05 2020-10-30 22:40:07 [INFO] [TRAIN] epoch=112, iter=41500/80000, loss=0.1667, lr=0.005183, batch_cost=0.7637, reader_cost=0.0003 | ETA 08:10:02 2020-10-30 22:41:23 [INFO] [TRAIN] epoch=112, iter=41600/80000, loss=0.1789, lr=0.005171, batch_cost=0.7581, reader_cost=0.0003 | ETA 08:05:10 2020-10-30 22:42:40 [INFO] [TRAIN] epoch=113, iter=41700/80000, loss=0.1453, lr=0.005158, batch_cost=0.7701, reader_cost=0.0084 | ETA 08:11:34 2020-10-30 22:43:55 [INFO] [TRAIN] epoch=113, iter=41800/80000, loss=0.1493, lr=0.005146, batch_cost=0.7568, reader_cost=0.0005 | ETA 08:01:48 2020-10-30 22:45:12 [INFO] [TRAIN] epoch=113, iter=41900/80000, loss=0.1688, lr=0.005134, batch_cost=0.7607, reader_cost=0.0004 | ETA 08:03:04 2020-10-30 22:46:28 [INFO] [TRAIN] epoch=113, iter=42000/80000, loss=0.1488, lr=0.005122, batch_cost=0.7653, reader_cost=0.0007 | ETA 08:04:42 2020-10-30 22:47:45 [INFO] [TRAIN] epoch=114, iter=42100/80000, loss=0.1559, lr=0.005110, batch_cost=0.7691, reader_cost=0.0102 | ETA 08:05:48 2020-10-30 22:49:01 [INFO] [TRAIN] epoch=114, iter=42200/80000, loss=0.1705, lr=0.005098, batch_cost=0.7617, reader_cost=0.0006 | ETA 07:59:52 2020-10-30 22:50:17 [INFO] [TRAIN] epoch=114, iter=42300/80000, loss=0.1769, lr=0.005086, batch_cost=0.7563, reader_cost=0.0008 | ETA 07:55:11 2020-10-30 22:51:33 [INFO] [TRAIN] epoch=114, iter=42400/80000, loss=0.1943, lr=0.005074, batch_cost=0.7645, reader_cost=0.0008 | ETA 07:59:05 2020-10-30 22:52:50 [INFO] [TRAIN] epoch=115, iter=42500/80000, loss=0.1494, lr=0.005062, batch_cost=0.7676, reader_cost=0.0082 | ETA 07:59:45 2020-10-30 22:54:06 [INFO] [TRAIN] epoch=115, iter=42600/80000, loss=0.1599, lr=0.005049, batch_cost=0.7568, reader_cost=0.0004 | ETA 07:51:45 2020-10-30 22:55:22 [INFO] [TRAIN] epoch=115, iter=42700/80000, loss=0.1527, lr=0.005037, batch_cost=0.7586, reader_cost=0.0004 | ETA 07:51:34 2020-10-30 22:56:38 [INFO] [TRAIN] epoch=116, iter=42800/80000, loss=0.1727, lr=0.005025, batch_cost=0.7672, reader_cost=0.0085 | ETA 07:55:39 2020-10-30 22:57:55 [INFO] [TRAIN] epoch=116, iter=42900/80000, loss=0.1427, lr=0.005013, batch_cost=0.7631, reader_cost=0.0005 | ETA 07:51:50 2020-10-30 22:59:11 [INFO] [TRAIN] epoch=116, iter=43000/80000, loss=0.1593, lr=0.005001, batch_cost=0.7624, reader_cost=0.0006 | ETA 07:50:10 2020-10-30 23:00:27 [INFO] [TRAIN] epoch=116, iter=43100/80000, loss=0.1381, lr=0.004989, batch_cost=0.7642, reader_cost=0.0008 | ETA 07:49:57 2020-10-30 23:01:45 [INFO] [TRAIN] epoch=117, iter=43200/80000, loss=0.1520, lr=0.004977, batch_cost=0.7765, reader_cost=0.0092 | ETA 07:56:14 2020-10-30 23:03:01 [INFO] [TRAIN] epoch=117, iter=43300/80000, loss=0.1732, lr=0.004964, batch_cost=0.7647, reader_cost=0.0006 | ETA 07:47:43 2020-10-30 23:04:17 [INFO] [TRAIN] epoch=117, iter=43400/80000, loss=0.1659, lr=0.004952, batch_cost=0.7606, reader_cost=0.0003 | ETA 07:43:58 2020-10-30 23:05:34 [INFO] [TRAIN] epoch=117, iter=43500/80000, loss=0.1615, lr=0.004940, batch_cost=0.7614, reader_cost=0.0008 | ETA 07:43:10 2020-10-30 23:06:51 [INFO] [TRAIN] epoch=118, iter=43600/80000, loss=0.1750, lr=0.004928, batch_cost=0.7698, reader_cost=0.0086 | ETA 07:46:59 2020-10-30 23:08:07 [INFO] [TRAIN] epoch=118, iter=43700/80000, loss=0.1465, lr=0.004916, batch_cost=0.7614, reader_cost=0.0004 | ETA 07:40:39 2020-10-30 23:09:23 [INFO] [TRAIN] epoch=118, iter=43800/80000, loss=0.1418, lr=0.004904, batch_cost=0.7664, reader_cost=0.0003 | ETA 07:42:22 2020-10-30 23:10:41 [INFO] [TRAIN] epoch=119, iter=43900/80000, loss=0.1430, lr=0.004891, batch_cost=0.7726, reader_cost=0.0090 | ETA 07:44:52 2020-10-30 23:11:57 [INFO] [TRAIN] epoch=119, iter=44000/80000, loss=0.1495, lr=0.004879, batch_cost=0.7653, reader_cost=0.0008 | ETA 07:39:09 2020-10-30 23:13:13 [INFO] [TRAIN] epoch=119, iter=44100/80000, loss=0.1486, lr=0.004867, batch_cost=0.7578, reader_cost=0.0003 | ETA 07:33:26 2020-10-30 23:14:29 [INFO] [TRAIN] epoch=119, iter=44200/80000, loss=0.1512, lr=0.004855, batch_cost=0.7597, reader_cost=0.0004 | ETA 07:33:18 2020-10-30 23:15:45 [INFO] [TRAIN] epoch=120, iter=44300/80000, loss=0.1462, lr=0.004843, batch_cost=0.7604, reader_cost=0.0084 | ETA 07:32:24 2020-10-30 23:16:59 [INFO] [TRAIN] epoch=120, iter=44400/80000, loss=0.1462, lr=0.004831, batch_cost=0.7434, reader_cost=0.0003 | ETA 07:21:05 2020-10-30 23:18:14 [INFO] [TRAIN] epoch=120, iter=44500/80000, loss=0.1416, lr=0.004818, batch_cost=0.7495, reader_cost=0.0003 | ETA 07:23:28 2020-10-30 23:19:29 [INFO] [TRAIN] epoch=120, iter=44600/80000, loss=0.1487, lr=0.004806, batch_cost=0.7497, reader_cost=0.0003 | ETA 07:22:19 2020-10-30 23:20:46 [INFO] [TRAIN] epoch=121, iter=44700/80000, loss=0.1495, lr=0.004794, batch_cost=0.7697, reader_cost=0.0086 | ETA 07:32:49 2020-10-30 23:22:02 [INFO] [TRAIN] epoch=121, iter=44800/80000, loss=0.1676, lr=0.004782, batch_cost=0.7569, reader_cost=0.0004 | ETA 07:24:02 2020-10-30 23:23:18 [INFO] [TRAIN] epoch=121, iter=44900/80000, loss=0.1616, lr=0.004770, batch_cost=0.7656, reader_cost=0.0009 | ETA 07:27:54 2020-10-30 23:24:34 [INFO] [TRAIN] epoch=121, iter=45000/80000, loss=0.1645, lr=0.004757, batch_cost=0.7602, reader_cost=0.0011 | ETA 07:23:26 2020-10-30 23:25:51 [INFO] [TRAIN] epoch=122, iter=45100/80000, loss=0.1529, lr=0.004745, batch_cost=0.7686, reader_cost=0.0084 | ETA 07:27:04 2020-10-30 23:27:07 [INFO] [TRAIN] epoch=122, iter=45200/80000, loss=0.1617, lr=0.004733, batch_cost=0.7585, reader_cost=0.0002 | ETA 07:19:55 2020-10-30 23:28:23 [INFO] [TRAIN] epoch=122, iter=45300/80000, loss=0.1562, lr=0.004721, batch_cost=0.7615, reader_cost=0.0009 | ETA 07:20:25 2020-10-30 23:29:40 [INFO] [TRAIN] epoch=123, iter=45400/80000, loss=0.1489, lr=0.004709, batch_cost=0.7673, reader_cost=0.0092 | ETA 07:22:27 2020-10-30 23:30:56 [INFO] [TRAIN] epoch=123, iter=45500/80000, loss=0.1608, lr=0.004696, batch_cost=0.7614, reader_cost=0.0010 | ETA 07:17:49 2020-10-30 23:32:13 [INFO] [TRAIN] epoch=123, iter=45600/80000, loss=0.1471, lr=0.004684, batch_cost=0.7666, reader_cost=0.0007 | ETA 07:19:31 2020-10-30 23:33:29 [INFO] [TRAIN] epoch=123, iter=45700/80000, loss=0.1491, lr=0.004672, batch_cost=0.7608, reader_cost=0.0007 | ETA 07:14:56 2020-10-30 23:34:46 [INFO] [TRAIN] epoch=124, iter=45800/80000, loss=0.1558, lr=0.004660, batch_cost=0.7672, reader_cost=0.0092 | ETA 07:17:17 2020-10-30 23:36:01 [INFO] [TRAIN] epoch=124, iter=45900/80000, loss=0.1498, lr=0.004647, batch_cost=0.7577, reader_cost=0.0004 | ETA 07:10:37 2020-10-30 23:37:18 [INFO] [TRAIN] epoch=124, iter=46000/80000, loss=0.1395, lr=0.004635, batch_cost=0.7647, reader_cost=0.0003 | ETA 07:13:20 2020-10-30 23:38:35 [INFO] [TRAIN] epoch=124, iter=46100/80000, loss=0.1438, lr=0.004623, batch_cost=0.7697, reader_cost=0.0005 | ETA 07:14:51 2020-10-30 23:39:51 [INFO] [TRAIN] epoch=125, iter=46200/80000, loss=0.1598, lr=0.004611, batch_cost=0.7649, reader_cost=0.0078 | ETA 07:10:53 2020-10-30 23:41:07 [INFO] [TRAIN] epoch=125, iter=46300/80000, loss=0.1757, lr=0.004598, batch_cost=0.7541, reader_cost=0.0002 | ETA 07:03:31 2020-10-30 23:42:23 [INFO] [TRAIN] epoch=125, iter=46400/80000, loss=0.1432, lr=0.004586, batch_cost=0.7585, reader_cost=0.0002 | ETA 07:04:46 2020-10-30 23:43:39 [INFO] [TRAIN] epoch=125, iter=46500/80000, loss=0.1442, lr=0.004574, batch_cost=0.7610, reader_cost=0.0002 | ETA 07:04:52 2020-10-30 23:44:56 [INFO] [TRAIN] epoch=126, iter=46600/80000, loss=0.1504, lr=0.004562, batch_cost=0.7751, reader_cost=0.0105 | ETA 07:11:27 2020-10-30 23:46:13 [INFO] [TRAIN] epoch=126, iter=46700/80000, loss=0.1700, lr=0.004549, batch_cost=0.7682, reader_cost=0.0010 | ETA 07:06:20 2020-10-30 23:47:29 [INFO] [TRAIN] epoch=126, iter=46800/80000, loss=0.1465, lr=0.004537, batch_cost=0.7588, reader_cost=0.0004 | ETA 06:59:50 2020-10-30 23:48:46 [INFO] [TRAIN] epoch=127, iter=46900/80000, loss=0.1491, lr=0.004525, batch_cost=0.7749, reader_cost=0.0080 | ETA 07:07:29 2020-10-30 23:50:03 [INFO] [TRAIN] epoch=127, iter=47000/80000, loss=0.2138, lr=0.004513, batch_cost=0.7650, reader_cost=0.0008 | ETA 07:00:45 2020-10-30 23:51:20 [INFO] [TRAIN] epoch=127, iter=47100/80000, loss=0.1436, lr=0.004500, batch_cost=0.7669, reader_cost=0.0006 | ETA 07:00:30 2020-10-30 23:52:36 [INFO] [TRAIN] epoch=127, iter=47200/80000, loss=0.1829, lr=0.004488, batch_cost=0.7629, reader_cost=0.0009 | ETA 06:57:02 2020-10-30 23:53:53 [INFO] [TRAIN] epoch=128, iter=47300/80000, loss=0.1408, lr=0.004476, batch_cost=0.7708, reader_cost=0.0094 | ETA 07:00:05 2020-10-30 23:55:09 [INFO] [TRAIN] epoch=128, iter=47400/80000, loss=0.1440, lr=0.004463, batch_cost=0.7579, reader_cost=0.0006 | ETA 06:51:47 2020-10-30 23:56:25 [INFO] [TRAIN] epoch=128, iter=47500/80000, loss=0.1417, lr=0.004451, batch_cost=0.7615, reader_cost=0.0008 | ETA 06:52:28 2020-10-30 23:57:41 [INFO] [TRAIN] epoch=128, iter=47600/80000, loss=0.1693, lr=0.004439, batch_cost=0.7649, reader_cost=0.0004 | ETA 06:53:04 2020-10-30 23:58:59 [INFO] [TRAIN] epoch=129, iter=47700/80000, loss=0.1403, lr=0.004427, batch_cost=0.7713, reader_cost=0.0086 | ETA 06:55:14 2020-10-31 00:00:14 [INFO] [TRAIN] epoch=129, iter=47800/80000, loss=0.1432, lr=0.004414, batch_cost=0.7566, reader_cost=0.0008 | ETA 06:46:03 2020-10-31 00:01:30 [INFO] [TRAIN] epoch=129, iter=47900/80000, loss=0.1516, lr=0.004402, batch_cost=0.7602, reader_cost=0.0006 | ETA 06:46:43 2020-10-31 00:02:47 [INFO] [TRAIN] epoch=130, iter=48000/80000, loss=0.1324, lr=0.004390, batch_cost=0.7645, reader_cost=0.0095 | ETA 06:47:43 2020-10-31 00:02:53 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-10-31 00:09:11 [INFO] [EVAL] #Images=500 mIoU=0.7786 Acc=0.9613 Kappa=0.9498 2020-10-31 00:09:11 [INFO] [EVAL] Category IoU: [0.9817 0.8561 0.9255 0.4366 0.6244 0.6599 0.7244 0.8026 0.9255 0.6152 0.9491 0.8323 0.6363 0.9537 0.7684 0.8919 0.8252 0.6029 0.7819] 2020-10-31 00:09:11 [INFO] [EVAL] Category Acc: [0.9916 0.9219 0.9577 0.8305 0.7884 0.8113 0.8132 0.9009 0.9498 0.9134 0.968 0.8895 0.812 0.9714 0.9751 0.9511 0.9215 0.8861 0.8424] 2020-10-31 00:09:15 [INFO] [EVAL] The model with the best validation mIoU (0.7786) was saved at iter 48000. 2020-10-31 00:10:30 [INFO] [TRAIN] epoch=130, iter=48100/80000, loss=0.1387, lr=0.004377, batch_cost=0.7499, reader_cost=0.0002 | ETA 06:38:42 2020-10-31 00:11:46 [INFO] [TRAIN] epoch=130, iter=48200/80000, loss=0.1539, lr=0.004365, batch_cost=0.7565, reader_cost=0.0002 | ETA 06:40:56 2020-10-31 00:13:01 [INFO] [TRAIN] epoch=130, iter=48300/80000, loss=0.1455, lr=0.004353, batch_cost=0.7560, reader_cost=0.0005 | ETA 06:39:23 2020-10-31 00:14:18 [INFO] [TRAIN] epoch=131, iter=48400/80000, loss=0.1363, lr=0.004340, batch_cost=0.7688, reader_cost=0.0101 | ETA 06:44:54 2020-10-31 00:15:34 [INFO] [TRAIN] epoch=131, iter=48500/80000, loss=0.1478, lr=0.004328, batch_cost=0.7586, reader_cost=0.0004 | ETA 06:38:16 2020-10-31 00:16:50 [INFO] [TRAIN] epoch=131, iter=48600/80000, loss=0.1467, lr=0.004316, batch_cost=0.7564, reader_cost=0.0003 | ETA 06:35:50 2020-10-31 00:18:06 [INFO] [TRAIN] epoch=131, iter=48700/80000, loss=0.1528, lr=0.004303, batch_cost=0.7633, reader_cost=0.0003 | ETA 06:38:12 2020-10-31 00:19:22 [INFO] [TRAIN] epoch=132, iter=48800/80000, loss=0.1439, lr=0.004291, batch_cost=0.7568, reader_cost=0.0102 | ETA 06:33:33 2020-10-31 00:20:36 [INFO] [TRAIN] epoch=132, iter=48900/80000, loss=0.1465, lr=0.004279, batch_cost=0.7480, reader_cost=0.0002 | ETA 06:27:44 2020-10-31 00:21:51 [INFO] [TRAIN] epoch=132, iter=49000/80000, loss=0.1336, lr=0.004266, batch_cost=0.7473, reader_cost=0.0002 | ETA 06:26:05 2020-10-31 00:23:06 [INFO] [TRAIN] epoch=132, iter=49100/80000, loss=0.1496, lr=0.004254, batch_cost=0.7490, reader_cost=0.0003 | ETA 06:25:45 2020-10-31 00:24:23 [INFO] [TRAIN] epoch=133, iter=49200/80000, loss=0.1472, lr=0.004241, batch_cost=0.7718, reader_cost=0.0097 | ETA 06:36:09 2020-10-31 00:25:40 [INFO] [TRAIN] epoch=133, iter=49300/80000, loss=0.1389, lr=0.004229, batch_cost=0.7681, reader_cost=0.0005 | ETA 06:33:01 2020-10-31 00:26:56 [INFO] [TRAIN] epoch=133, iter=49400/80000, loss=0.1383, lr=0.004217, batch_cost=0.7613, reader_cost=0.0004 | ETA 06:28:16 2020-10-31 00:28:13 [INFO] [TRAIN] epoch=134, iter=49500/80000, loss=0.1471, lr=0.004204, batch_cost=0.7708, reader_cost=0.0098 | ETA 06:31:47 2020-10-31 00:29:30 [INFO] [TRAIN] epoch=134, iter=49600/80000, loss=0.1442, lr=0.004192, batch_cost=0.7675, reader_cost=0.0009 | ETA 06:28:52 2020-10-31 00:30:46 [INFO] [TRAIN] epoch=134, iter=49700/80000, loss=0.1370, lr=0.004180, batch_cost=0.7567, reader_cost=0.0006 | ETA 06:22:06 2020-10-31 00:32:02 [INFO] [TRAIN] epoch=134, iter=49800/80000, loss=0.1481, lr=0.004167, batch_cost=0.7592, reader_cost=0.0006 | ETA 06:22:06 2020-10-31 00:33:19 [INFO] [TRAIN] epoch=135, iter=49900/80000, loss=0.1441, lr=0.004155, batch_cost=0.7737, reader_cost=0.0077 | ETA 06:28:09 2020-10-31 00:34:35 [INFO] [TRAIN] epoch=135, iter=50000/80000, loss=0.1273, lr=0.004142, batch_cost=0.7597, reader_cost=0.0005 | ETA 06:19:51 2020-10-31 00:35:52 [INFO] [TRAIN] epoch=135, iter=50100/80000, loss=0.1480, lr=0.004130, batch_cost=0.7654, reader_cost=0.0009 | ETA 06:21:24 2020-10-31 00:37:07 [INFO] [TRAIN] epoch=135, iter=50200/80000, loss=0.1696, lr=0.004118, batch_cost=0.7575, reader_cost=0.0003 | ETA 06:16:12 2020-10-31 00:38:24 [INFO] [TRAIN] epoch=136, iter=50300/80000, loss=0.1449, lr=0.004105, batch_cost=0.7699, reader_cost=0.0081 | ETA 06:21:07 2020-10-31 00:39:41 [INFO] [TRAIN] epoch=136, iter=50400/80000, loss=0.1383, lr=0.004093, batch_cost=0.7635, reader_cost=0.0004 | ETA 06:16:39 2020-10-31 00:40:57 [INFO] [TRAIN] epoch=136, iter=50500/80000, loss=0.1330, lr=0.004080, batch_cost=0.7633, reader_cost=0.0004 | ETA 06:15:17 2020-10-31 00:42:14 [INFO] [TRAIN] epoch=137, iter=50600/80000, loss=0.1257, lr=0.004068, batch_cost=0.7693, reader_cost=0.0082 | ETA 06:16:58 2020-10-31 00:43:30 [INFO] [TRAIN] epoch=137, iter=50700/80000, loss=0.1388, lr=0.004056, batch_cost=0.7581, reader_cost=0.0003 | ETA 06:10:12 2020-10-31 00:44:45 [INFO] [TRAIN] epoch=137, iter=50800/80000, loss=0.1531, lr=0.004043, batch_cost=0.7559, reader_cost=0.0002 | ETA 06:07:52 2020-10-31 00:46:02 [INFO] [TRAIN] epoch=137, iter=50900/80000, loss=0.1409, lr=0.004031, batch_cost=0.7618, reader_cost=0.0006 | ETA 06:09:29 2020-10-31 00:47:18 [INFO] [TRAIN] epoch=138, iter=51000/80000, loss=0.1244, lr=0.004018, batch_cost=0.7668, reader_cost=0.0087 | ETA 06:10:36 2020-10-31 00:48:35 [INFO] [TRAIN] epoch=138, iter=51100/80000, loss=0.1440, lr=0.004006, batch_cost=0.7637, reader_cost=0.0004 | ETA 06:07:51 2020-10-31 00:49:51 [INFO] [TRAIN] epoch=138, iter=51200/80000, loss=0.1327, lr=0.003993, batch_cost=0.7602, reader_cost=0.0002 | ETA 06:04:53 2020-10-31 00:51:07 [INFO] [TRAIN] epoch=138, iter=51300/80000, loss=0.1399, lr=0.003981, batch_cost=0.7659, reader_cost=0.0004 | ETA 06:06:20 2020-10-31 00:52:25 [INFO] [TRAIN] epoch=139, iter=51400/80000, loss=0.1382, lr=0.003968, batch_cost=0.7749, reader_cost=0.0097 | ETA 06:09:21 2020-10-31 00:53:40 [INFO] [TRAIN] epoch=139, iter=51500/80000, loss=0.1554, lr=0.003956, batch_cost=0.7581, reader_cost=0.0003 | ETA 06:00:05 2020-10-31 00:54:57 [INFO] [TRAIN] epoch=139, iter=51600/80000, loss=0.1445, lr=0.003944, batch_cost=0.7643, reader_cost=0.0006 | ETA 06:01:44 2020-10-31 00:56:13 [INFO] [TRAIN] epoch=139, iter=51700/80000, loss=0.1431, lr=0.003931, batch_cost=0.7619, reader_cost=0.0005 | ETA 05:59:22 2020-10-31 00:57:30 [INFO] [TRAIN] epoch=140, iter=51800/80000, loss=0.1342, lr=0.003919, batch_cost=0.7663, reader_cost=0.0086 | ETA 06:00:08 2020-10-31 00:58:46 [INFO] [TRAIN] epoch=140, iter=51900/80000, loss=0.1653, lr=0.003906, batch_cost=0.7622, reader_cost=0.0005 | ETA 05:56:57 2020-10-31 01:00:02 [INFO] [TRAIN] epoch=140, iter=52000/80000, loss=0.1399, lr=0.003894, batch_cost=0.7627, reader_cost=0.0007 | ETA 05:55:56 2020-10-31 01:01:19 [INFO] [TRAIN] epoch=141, iter=52100/80000, loss=0.1282, lr=0.003881, batch_cost=0.7704, reader_cost=0.0103 | ETA 05:58:14 2020-10-31 01:02:35 [INFO] [TRAIN] epoch=141, iter=52200/80000, loss=0.1508, lr=0.003869, batch_cost=0.7599, reader_cost=0.0008 | ETA 05:52:06 2020-10-31 01:03:51 [INFO] [TRAIN] epoch=141, iter=52300/80000, loss=0.1287, lr=0.003856, batch_cost=0.7616, reader_cost=0.0004 | ETA 05:51:35 2020-10-31 01:05:08 [INFO] [TRAIN] epoch=141, iter=52400/80000, loss=0.1502, lr=0.003844, batch_cost=0.7668, reader_cost=0.0007 | ETA 05:52:43 2020-10-31 01:06:25 [INFO] [TRAIN] epoch=142, iter=52500/80000, loss=0.1373, lr=0.003831, batch_cost=0.7666, reader_cost=0.0091 | ETA 05:51:20 2020-10-31 01:07:41 [INFO] [TRAIN] epoch=142, iter=52600/80000, loss=0.1343, lr=0.003819, batch_cost=0.7608, reader_cost=0.0004 | ETA 05:47:26 2020-10-31 01:08:57 [INFO] [TRAIN] epoch=142, iter=52700/80000, loss=0.1343, lr=0.003806, batch_cost=0.7586, reader_cost=0.0005 | ETA 05:45:09 2020-10-31 01:10:13 [INFO] [TRAIN] epoch=142, iter=52800/80000, loss=0.1430, lr=0.003794, batch_cost=0.7595, reader_cost=0.0005 | ETA 05:44:18 2020-10-31 01:11:29 [INFO] [TRAIN] epoch=143, iter=52900/80000, loss=0.1330, lr=0.003781, batch_cost=0.7676, reader_cost=0.0105 | ETA 05:46:42 2020-10-31 01:12:46 [INFO] [TRAIN] epoch=143, iter=53000/80000, loss=0.1417, lr=0.003769, batch_cost=0.7656, reader_cost=0.0004 | ETA 05:44:32 2020-10-31 01:14:02 [INFO] [TRAIN] epoch=143, iter=53100/80000, loss=0.1283, lr=0.003756, batch_cost=0.7620, reader_cost=0.0008 | ETA 05:41:37 2020-10-31 01:15:19 [INFO] [TRAIN] epoch=144, iter=53200/80000, loss=0.1430, lr=0.003744, batch_cost=0.7724, reader_cost=0.0099 | ETA 05:44:59 2020-10-31 01:16:35 [INFO] [TRAIN] epoch=144, iter=53300/80000, loss=0.1299, lr=0.003731, batch_cost=0.7546, reader_cost=0.0004 | ETA 05:35:46 2020-10-31 01:17:51 [INFO] [TRAIN] epoch=144, iter=53400/80000, loss=0.1390, lr=0.003718, batch_cost=0.7611, reader_cost=0.0007 | ETA 05:37:26 2020-10-31 01:19:07 [INFO] [TRAIN] epoch=144, iter=53500/80000, loss=0.1471, lr=0.003706, batch_cost=0.7625, reader_cost=0.0006 | ETA 05:36:45 2020-10-31 01:20:24 [INFO] [TRAIN] epoch=145, iter=53600/80000, loss=0.1378, lr=0.003693, batch_cost=0.7676, reader_cost=0.0094 | ETA 05:37:45 2020-10-31 01:21:39 [INFO] [TRAIN] epoch=145, iter=53700/80000, loss=0.1359, lr=0.003681, batch_cost=0.7497, reader_cost=0.0002 | ETA 05:28:37 2020-10-31 01:22:54 [INFO] [TRAIN] epoch=145, iter=53800/80000, loss=0.1481, lr=0.003668, batch_cost=0.7456, reader_cost=0.0002 | ETA 05:25:34 2020-10-31 01:24:09 [INFO] [TRAIN] epoch=145, iter=53900/80000, loss=0.1342, lr=0.003656, batch_cost=0.7521, reader_cost=0.0005 | ETA 05:27:08 2020-10-31 01:25:25 [INFO] [TRAIN] epoch=146, iter=54000/80000, loss=0.1306, lr=0.003643, batch_cost=0.7576, reader_cost=0.0086 | ETA 05:28:18 2020-10-31 01:26:39 [INFO] [TRAIN] epoch=146, iter=54100/80000, loss=0.1401, lr=0.003631, batch_cost=0.7475, reader_cost=0.0004 | ETA 05:22:40 2020-10-31 01:27:55 [INFO] [TRAIN] epoch=146, iter=54200/80000, loss=0.1321, lr=0.003618, batch_cost=0.7595, reader_cost=0.0002 | ETA 05:26:35 2020-10-31 01:29:11 [INFO] [TRAIN] epoch=146, iter=54300/80000, loss=0.1529, lr=0.003605, batch_cost=0.7612, reader_cost=0.0002 | ETA 05:26:02 2020-10-31 01:30:28 [INFO] [TRAIN] epoch=147, iter=54400/80000, loss=0.1421, lr=0.003593, batch_cost=0.7676, reader_cost=0.0089 | ETA 05:27:30 2020-10-31 01:31:45 [INFO] [TRAIN] epoch=147, iter=54500/80000, loss=0.1363, lr=0.003580, batch_cost=0.7662, reader_cost=0.0009 | ETA 05:25:36 2020-10-31 01:33:01 [INFO] [TRAIN] epoch=147, iter=54600/80000, loss=0.1381, lr=0.003568, batch_cost=0.7600, reader_cost=0.0004 | ETA 05:21:42 2020-10-31 01:34:18 [INFO] [TRAIN] epoch=148, iter=54700/80000, loss=0.1343, lr=0.003555, batch_cost=0.7680, reader_cost=0.0085 | ETA 05:23:50 2020-10-31 01:35:34 [INFO] [TRAIN] epoch=148, iter=54800/80000, loss=0.1322, lr=0.003542, batch_cost=0.7687, reader_cost=0.0012 | ETA 05:22:50 2020-10-31 01:36:50 [INFO] [TRAIN] epoch=148, iter=54900/80000, loss=0.1355, lr=0.003530, batch_cost=0.7589, reader_cost=0.0005 | ETA 05:17:29 2020-10-31 01:38:06 [INFO] [TRAIN] epoch=148, iter=55000/80000, loss=0.1623, lr=0.003517, batch_cost=0.7621, reader_cost=0.0009 | ETA 05:17:31 2020-10-31 01:39:24 [INFO] [TRAIN] epoch=149, iter=55100/80000, loss=0.1328, lr=0.003504, batch_cost=0.7740, reader_cost=0.0100 | ETA 05:21:12 2020-10-31 01:40:40 [INFO] [TRAIN] epoch=149, iter=55200/80000, loss=0.1504, lr=0.003492, batch_cost=0.7647, reader_cost=0.0006 | ETA 05:16:03 2020-10-31 01:41:56 [INFO] [TRAIN] epoch=149, iter=55300/80000, loss=0.1402, lr=0.003479, batch_cost=0.7562, reader_cost=0.0007 | ETA 05:11:18 2020-10-31 01:43:13 [INFO] [TRAIN] epoch=149, iter=55400/80000, loss=0.1524, lr=0.003467, batch_cost=0.7654, reader_cost=0.0003 | ETA 05:13:49 2020-10-31 01:44:29 [INFO] [TRAIN] epoch=150, iter=55500/80000, loss=0.1443, lr=0.003454, batch_cost=0.7677, reader_cost=0.0093 | ETA 05:13:29 2020-10-31 01:45:46 [INFO] [TRAIN] epoch=150, iter=55600/80000, loss=0.1391, lr=0.003441, batch_cost=0.7622, reader_cost=0.0002 | ETA 05:09:57 2020-10-31 01:47:02 [INFO] [TRAIN] epoch=150, iter=55700/80000, loss=0.1412, lr=0.003429, batch_cost=0.7648, reader_cost=0.0003 | ETA 05:09:44 2020-10-31 01:48:18 [INFO] [TRAIN] epoch=150, iter=55800/80000, loss=0.1498, lr=0.003416, batch_cost=0.7574, reader_cost=0.0002 | ETA 05:05:28 2020-10-31 01:49:34 [INFO] [TRAIN] epoch=151, iter=55900/80000, loss=0.1359, lr=0.003403, batch_cost=0.7660, reader_cost=0.0093 | ETA 05:07:41 2020-10-31 01:50:50 [INFO] [TRAIN] epoch=151, iter=56000/80000, loss=0.1360, lr=0.003391, batch_cost=0.7560, reader_cost=0.0006 | ETA 05:02:25 2020-10-31 01:50:57 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-10-31 01:57:11 [INFO] [EVAL] #Images=500 mIoU=0.7924 Acc=0.9618 Kappa=0.9504 2020-10-31 01:57:11 [INFO] [EVAL] Category IoU: [0.9805 0.8529 0.9256 0.457 0.6339 0.6669 0.7383 0.8093 0.9264 0.6328 0.9483 0.8362 0.6626 0.955 0.8572 0.8833 0.8317 0.6677 0.7907] 2020-10-31 01:57:11 [INFO] [EVAL] Category Acc: [0.9905 0.9252 0.9572 0.9026 0.7912 0.8172 0.8396 0.9013 0.9514 0.8867 0.9656 0.8875 0.8148 0.9729 0.9605 0.9523 0.9342 0.8377 0.8789] 2020-10-31 01:57:14 [INFO] [EVAL] The model with the best validation mIoU (0.7924) was saved at iter 56000. 2020-10-31 01:58:30 [INFO] [TRAIN] epoch=151, iter=56100/80000, loss=0.1457, lr=0.003378, batch_cost=0.7539, reader_cost=0.0003 | ETA 05:00:17 2020-10-31 01:59:46 [INFO] [TRAIN] epoch=152, iter=56200/80000, loss=0.1417, lr=0.003365, batch_cost=0.7613, reader_cost=0.0083 | ETA 05:01:58 2020-10-31 02:01:02 [INFO] [TRAIN] epoch=152, iter=56300/80000, loss=0.1457, lr=0.003353, batch_cost=0.7593, reader_cost=0.0007 | ETA 04:59:56 2020-10-31 02:02:18 [INFO] [TRAIN] epoch=152, iter=56400/80000, loss=0.1357, lr=0.003340, batch_cost=0.7610, reader_cost=0.0005 | ETA 04:59:18 2020-10-31 02:03:34 [INFO] [TRAIN] epoch=152, iter=56500/80000, loss=0.1380, lr=0.003327, batch_cost=0.7608, reader_cost=0.0002 | ETA 04:57:59 2020-10-31 02:04:51 [INFO] [TRAIN] epoch=153, iter=56600/80000, loss=0.1343, lr=0.003314, batch_cost=0.7692, reader_cost=0.0091 | ETA 04:59:59 2020-10-31 02:06:07 [INFO] [TRAIN] epoch=153, iter=56700/80000, loss=0.1543, lr=0.003302, batch_cost=0.7600, reader_cost=0.0006 | ETA 04:55:08 2020-10-31 02:07:22 [INFO] [TRAIN] epoch=153, iter=56800/80000, loss=0.1373, lr=0.003289, batch_cost=0.7525, reader_cost=0.0005 | ETA 04:50:59 2020-10-31 02:08:38 [INFO] [TRAIN] epoch=153, iter=56900/80000, loss=0.1395, lr=0.003276, batch_cost=0.7570, reader_cost=0.0002 | ETA 04:51:26 2020-10-31 02:09:55 [INFO] [TRAIN] epoch=154, iter=57000/80000, loss=0.1332, lr=0.003264, batch_cost=0.7672, reader_cost=0.0089 | ETA 04:54:06 2020-10-31 02:11:10 [INFO] [TRAIN] epoch=154, iter=57100/80000, loss=0.1363, lr=0.003251, batch_cost=0.7565, reader_cost=0.0007 | ETA 04:48:43 2020-10-31 02:12:26 [INFO] [TRAIN] epoch=154, iter=57200/80000, loss=0.1221, lr=0.003238, batch_cost=0.7590, reader_cost=0.0004 | ETA 04:48:25 2020-10-31 02:13:43 [INFO] [TRAIN] epoch=155, iter=57300/80000, loss=0.1338, lr=0.003225, batch_cost=0.7692, reader_cost=0.0090 | ETA 04:51:00 2020-10-31 02:14:59 [INFO] [TRAIN] epoch=155, iter=57400/80000, loss=0.1341, lr=0.003213, batch_cost=0.7554, reader_cost=0.0002 | ETA 04:44:32 2020-10-31 02:16:15 [INFO] [TRAIN] epoch=155, iter=57500/80000, loss=0.1278, lr=0.003200, batch_cost=0.7600, reader_cost=0.0007 | ETA 04:44:59 2020-10-31 02:17:31 [INFO] [TRAIN] epoch=155, iter=57600/80000, loss=0.1733, lr=0.003187, batch_cost=0.7612, reader_cost=0.0004 | ETA 04:44:10 2020-10-31 02:18:48 [INFO] [TRAIN] epoch=156, iter=57700/80000, loss=0.1380, lr=0.003174, batch_cost=0.7685, reader_cost=0.0087 | ETA 04:45:37 2020-10-31 02:20:03 [INFO] [TRAIN] epoch=156, iter=57800/80000, loss=0.1302, lr=0.003162, batch_cost=0.7569, reader_cost=0.0002 | ETA 04:40:03 2020-10-31 02:21:20 [INFO] [TRAIN] epoch=156, iter=57900/80000, loss=0.1306, lr=0.003149, batch_cost=0.7613, reader_cost=0.0003 | ETA 04:40:24 2020-10-31 02:22:36 [INFO] [TRAIN] epoch=156, iter=58000/80000, loss=0.1192, lr=0.003136, batch_cost=0.7639, reader_cost=0.0006 | ETA 04:40:06 2020-10-31 02:23:52 [INFO] [TRAIN] epoch=157, iter=58100/80000, loss=0.1318, lr=0.003123, batch_cost=0.7655, reader_cost=0.0080 | ETA 04:39:24 2020-10-31 02:25:09 [INFO] [TRAIN] epoch=157, iter=58200/80000, loss=0.1321, lr=0.003110, batch_cost=0.7630, reader_cost=0.0006 | ETA 04:37:13 2020-10-31 02:26:26 [INFO] [TRAIN] epoch=157, iter=58300/80000, loss=0.1361, lr=0.003098, batch_cost=0.7673, reader_cost=0.0007 | ETA 04:37:30 2020-10-31 02:27:42 [INFO] [TRAIN] epoch=157, iter=58400/80000, loss=0.1228, lr=0.003085, batch_cost=0.7604, reader_cost=0.0006 | ETA 04:33:44 2020-10-31 02:28:58 [INFO] [TRAIN] epoch=158, iter=58500/80000, loss=0.1192, lr=0.003072, batch_cost=0.7669, reader_cost=0.0085 | ETA 04:34:47 2020-10-31 02:30:14 [INFO] [TRAIN] epoch=158, iter=58600/80000, loss=0.1293, lr=0.003059, batch_cost=0.7577, reader_cost=0.0006 | ETA 04:30:15 2020-10-31 02:31:30 [INFO] [TRAIN] epoch=158, iter=58700/80000, loss=0.1388, lr=0.003046, batch_cost=0.7636, reader_cost=0.0004 | ETA 04:31:04 2020-10-31 02:32:48 [INFO] [TRAIN] epoch=159, iter=58800/80000, loss=0.1385, lr=0.003033, batch_cost=0.7725, reader_cost=0.0092 | ETA 04:32:57 2020-10-31 02:34:04 [INFO] [TRAIN] epoch=159, iter=58900/80000, loss=0.1412, lr=0.003021, batch_cost=0.7589, reader_cost=0.0008 | ETA 04:26:53 2020-10-31 02:35:20 [INFO] [TRAIN] epoch=159, iter=59000/80000, loss=0.1260, lr=0.003008, batch_cost=0.7633, reader_cost=0.0005 | ETA 04:27:08 2020-10-31 02:36:36 [INFO] [TRAIN] epoch=159, iter=59100/80000, loss=0.1437, lr=0.002995, batch_cost=0.7597, reader_cost=0.0002 | ETA 04:24:36 2020-10-31 02:37:53 [INFO] [TRAIN] epoch=160, iter=59200/80000, loss=0.1313, lr=0.002982, batch_cost=0.7674, reader_cost=0.0087 | ETA 04:26:02 2020-10-31 02:39:08 [INFO] [TRAIN] epoch=160, iter=59300/80000, loss=0.1292, lr=0.002969, batch_cost=0.7558, reader_cost=0.0004 | ETA 04:20:44 2020-10-31 02:40:24 [INFO] [TRAIN] epoch=160, iter=59400/80000, loss=0.1259, lr=0.002956, batch_cost=0.7610, reader_cost=0.0003 | ETA 04:21:16 2020-10-31 02:41:40 [INFO] [TRAIN] epoch=160, iter=59500/80000, loss=0.1436, lr=0.002943, batch_cost=0.7601, reader_cost=0.0006 | ETA 04:19:41 2020-10-31 02:42:57 [INFO] [TRAIN] epoch=161, iter=59600/80000, loss=0.1313, lr=0.002931, batch_cost=0.7643, reader_cost=0.0095 | ETA 04:19:52 2020-10-31 02:44:13 [INFO] [TRAIN] epoch=161, iter=59700/80000, loss=0.1323, lr=0.002918, batch_cost=0.7618, reader_cost=0.0003 | ETA 04:17:44 2020-10-31 02:45:29 [INFO] [TRAIN] epoch=161, iter=59800/80000, loss=0.1321, lr=0.002905, batch_cost=0.7569, reader_cost=0.0002 | ETA 04:14:49 2020-10-31 02:46:46 [INFO] [TRAIN] epoch=162, iter=59900/80000, loss=0.1219, lr=0.002892, batch_cost=0.7726, reader_cost=0.0079 | ETA 04:18:49 2020-10-31 02:48:02 [INFO] [TRAIN] epoch=162, iter=60000/80000, loss=0.1202, lr=0.002879, batch_cost=0.7594, reader_cost=0.0002 | ETA 04:13:07 2020-10-31 02:49:18 [INFO] [TRAIN] epoch=162, iter=60100/80000, loss=0.1224, lr=0.002866, batch_cost=0.7577, reader_cost=0.0007 | ETA 04:11:17 2020-10-31 02:50:33 [INFO] [TRAIN] epoch=162, iter=60200/80000, loss=0.1279, lr=0.002853, batch_cost=0.7573, reader_cost=0.0003 | ETA 04:09:53 2020-10-31 02:51:50 [INFO] [TRAIN] epoch=163, iter=60300/80000, loss=0.1195, lr=0.002840, batch_cost=0.7659, reader_cost=0.0095 | ETA 04:11:28 2020-10-31 02:53:06 [INFO] [TRAIN] epoch=163, iter=60400/80000, loss=0.1302, lr=0.002827, batch_cost=0.7570, reader_cost=0.0003 | ETA 04:07:16 2020-10-31 02:54:22 [INFO] [TRAIN] epoch=163, iter=60500/80000, loss=0.1323, lr=0.002814, batch_cost=0.7641, reader_cost=0.0006 | ETA 04:08:20 2020-10-31 02:55:39 [INFO] [TRAIN] epoch=163, iter=60600/80000, loss=0.1337, lr=0.002801, batch_cost=0.7694, reader_cost=0.0004 | ETA 04:08:47 2020-10-31 02:56:55 [INFO] [TRAIN] epoch=164, iter=60700/80000, loss=0.1115, lr=0.002788, batch_cost=0.7659, reader_cost=0.0093 | ETA 04:06:21 2020-10-31 02:58:11 [INFO] [TRAIN] epoch=164, iter=60800/80000, loss=0.1509, lr=0.002776, batch_cost=0.7532, reader_cost=0.0003 | ETA 04:01:01 2020-10-31 02:59:26 [INFO] [TRAIN] epoch=164, iter=60900/80000, loss=0.1217, lr=0.002763, batch_cost=0.7509, reader_cost=0.0003 | ETA 03:59:01 2020-10-31 03:00:41 [INFO] [TRAIN] epoch=164, iter=61000/80000, loss=0.1315, lr=0.002750, batch_cost=0.7508, reader_cost=0.0006 | ETA 03:57:45 2020-10-31 03:01:57 [INFO] [TRAIN] epoch=165, iter=61100/80000, loss=0.1311, lr=0.002737, batch_cost=0.7600, reader_cost=0.0085 | ETA 03:59:23 2020-10-31 03:03:12 [INFO] [TRAIN] epoch=165, iter=61200/80000, loss=0.1317, lr=0.002724, batch_cost=0.7463, reader_cost=0.0002 | ETA 03:53:49 2020-10-31 03:04:28 [INFO] [TRAIN] epoch=165, iter=61300/80000, loss=0.1438, lr=0.002711, batch_cost=0.7599, reader_cost=0.0006 | ETA 03:56:49 2020-10-31 03:05:45 [INFO] [TRAIN] epoch=166, iter=61400/80000, loss=0.1171, lr=0.002698, batch_cost=0.7787, reader_cost=0.0092 | ETA 04:01:23 2020-10-31 03:07:01 [INFO] [TRAIN] epoch=166, iter=61500/80000, loss=0.1280, lr=0.002685, batch_cost=0.7595, reader_cost=0.0005 | ETA 03:54:11 2020-10-31 03:08:17 [INFO] [TRAIN] epoch=166, iter=61600/80000, loss=0.1346, lr=0.002672, batch_cost=0.7573, reader_cost=0.0008 | ETA 03:52:13 2020-10-31 03:09:33 [INFO] [TRAIN] epoch=166, iter=61700/80000, loss=0.1338, lr=0.002659, batch_cost=0.7610, reader_cost=0.0009 | ETA 03:52:06 2020-10-31 03:10:50 [INFO] [TRAIN] epoch=167, iter=61800/80000, loss=0.1176, lr=0.002646, batch_cost=0.7669, reader_cost=0.0099 | ETA 03:52:38 2020-10-31 03:12:06 [INFO] [TRAIN] epoch=167, iter=61900/80000, loss=0.1340, lr=0.002633, batch_cost=0.7641, reader_cost=0.0008 | ETA 03:50:30 2020-10-31 03:13:22 [INFO] [TRAIN] epoch=167, iter=62000/80000, loss=0.1346, lr=0.002619, batch_cost=0.7593, reader_cost=0.0005 | ETA 03:47:47 2020-10-31 03:14:38 [INFO] [TRAIN] epoch=167, iter=62100/80000, loss=0.1266, lr=0.002606, batch_cost=0.7621, reader_cost=0.0009 | ETA 03:47:22 2020-10-31 03:15:56 [INFO] [TRAIN] epoch=168, iter=62200/80000, loss=0.1292, lr=0.002593, batch_cost=0.7703, reader_cost=0.0107 | ETA 03:48:30 2020-10-31 03:17:12 [INFO] [TRAIN] epoch=168, iter=62300/80000, loss=0.1273, lr=0.002580, batch_cost=0.7619, reader_cost=0.0003 | ETA 03:44:45 2020-10-31 03:18:28 [INFO] [TRAIN] epoch=168, iter=62400/80000, loss=0.1409, lr=0.002567, batch_cost=0.7609, reader_cost=0.0008 | ETA 03:43:11 2020-10-31 03:19:45 [INFO] [TRAIN] epoch=169, iter=62500/80000, loss=0.1265, lr=0.002554, batch_cost=0.7709, reader_cost=0.0085 | ETA 03:44:50 2020-10-31 03:21:00 [INFO] [TRAIN] epoch=169, iter=62600/80000, loss=0.1352, lr=0.002541, batch_cost=0.7546, reader_cost=0.0002 | ETA 03:38:50 2020-10-31 03:22:16 [INFO] [TRAIN] epoch=169, iter=62700/80000, loss=0.1335, lr=0.002528, batch_cost=0.7569, reader_cost=0.0003 | ETA 03:38:14 2020-10-31 03:23:32 [INFO] [TRAIN] epoch=169, iter=62800/80000, loss=0.1385, lr=0.002515, batch_cost=0.7612, reader_cost=0.0002 | ETA 03:38:13 2020-10-31 03:24:49 [INFO] [TRAIN] epoch=170, iter=62900/80000, loss=0.1194, lr=0.002502, batch_cost=0.7659, reader_cost=0.0092 | ETA 03:38:17 2020-10-31 03:26:04 [INFO] [TRAIN] epoch=170, iter=63000/80000, loss=0.1291, lr=0.002489, batch_cost=0.7568, reader_cost=0.0007 | ETA 03:34:25 2020-10-31 03:27:21 [INFO] [TRAIN] epoch=170, iter=63100/80000, loss=0.1174, lr=0.002476, batch_cost=0.7626, reader_cost=0.0004 | ETA 03:34:47 2020-10-31 03:28:37 [INFO] [TRAIN] epoch=170, iter=63200/80000, loss=0.1256, lr=0.002462, batch_cost=0.7631, reader_cost=0.0005 | ETA 03:33:40 2020-10-31 03:29:53 [INFO] [TRAIN] epoch=171, iter=63300/80000, loss=0.1187, lr=0.002449, batch_cost=0.7637, reader_cost=0.0087 | ETA 03:32:33 2020-10-31 03:31:08 [INFO] [TRAIN] epoch=171, iter=63400/80000, loss=0.1388, lr=0.002436, batch_cost=0.7510, reader_cost=0.0005 | ETA 03:27:46 2020-10-31 03:32:25 [INFO] [TRAIN] epoch=171, iter=63500/80000, loss=0.1180, lr=0.002423, batch_cost=0.7602, reader_cost=0.0006 | ETA 03:29:03 2020-10-31 03:33:40 [INFO] [TRAIN] epoch=171, iter=63600/80000, loss=0.1335, lr=0.002410, batch_cost=0.7584, reader_cost=0.0005 | ETA 03:27:17 2020-10-31 03:34:56 [INFO] [TRAIN] epoch=172, iter=63700/80000, loss=0.1351, lr=0.002397, batch_cost=0.7604, reader_cost=0.0095 | ETA 03:26:34 2020-10-31 03:36:12 [INFO] [TRAIN] epoch=172, iter=63800/80000, loss=0.1207, lr=0.002383, batch_cost=0.7555, reader_cost=0.0002 | ETA 03:23:58 2020-10-31 03:37:28 [INFO] [TRAIN] epoch=172, iter=63900/80000, loss=0.1400, lr=0.002370, batch_cost=0.7571, reader_cost=0.0002 | ETA 03:23:09 2020-10-31 03:38:44 [INFO] [TRAIN] epoch=173, iter=64000/80000, loss=0.1216, lr=0.002357, batch_cost=0.7601, reader_cost=0.0092 | ETA 03:22:40 2020-10-31 03:38:51 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-10-31 03:45:05 [INFO] [EVAL] #Images=500 mIoU=0.8040 Acc=0.9652 Kappa=0.9548 2020-10-31 03:45:05 [INFO] [EVAL] Category IoU: [0.9848 0.8702 0.9339 0.5595 0.6337 0.6783 0.7409 0.8141 0.9282 0.6487 0.9505 0.8414 0.6664 0.9574 0.8638 0.888 0.8439 0.6784 0.7934] 2020-10-31 03:45:05 [INFO] [EVAL] Category Acc: [0.9917 0.9352 0.9634 0.9115 0.8058 0.8179 0.8536 0.9092 0.9532 0.871 0.9699 0.9127 0.7759 0.9762 0.9623 0.9627 0.9538 0.8473 0.8635] 2020-10-31 03:45:10 [INFO] [EVAL] The model with the best validation mIoU (0.8040) was saved at iter 64000. 2020-10-31 03:46:24 [INFO] [TRAIN] epoch=173, iter=64100/80000, loss=0.1222, lr=0.002344, batch_cost=0.7475, reader_cost=0.0002 | ETA 03:18:04 2020-10-31 03:47:40 [INFO] [TRAIN] epoch=173, iter=64200/80000, loss=0.1293, lr=0.002331, batch_cost=0.7517, reader_cost=0.0006 | ETA 03:17:57 2020-10-31 03:48:55 [INFO] [TRAIN] epoch=173, iter=64300/80000, loss=0.1414, lr=0.002317, batch_cost=0.7551, reader_cost=0.0006 | ETA 03:17:35 2020-10-31 03:50:12 [INFO] [TRAIN] epoch=174, iter=64400/80000, loss=0.1100, lr=0.002304, batch_cost=0.7641, reader_cost=0.0093 | ETA 03:18:40 2020-10-31 03:51:28 [INFO] [TRAIN] epoch=174, iter=64500/80000, loss=0.1304, lr=0.002291, batch_cost=0.7690, reader_cost=0.0006 | ETA 03:18:39 2020-10-31 03:52:45 [INFO] [TRAIN] epoch=174, iter=64600/80000, loss=0.1217, lr=0.002278, batch_cost=0.7632, reader_cost=0.0002 | ETA 03:15:53 2020-10-31 03:54:01 [INFO] [TRAIN] epoch=174, iter=64700/80000, loss=0.1311, lr=0.002264, batch_cost=0.7594, reader_cost=0.0002 | ETA 03:13:39 2020-10-31 03:55:17 [INFO] [TRAIN] epoch=175, iter=64800/80000, loss=0.1355, lr=0.002251, batch_cost=0.7665, reader_cost=0.0091 | ETA 03:14:10 2020-10-31 03:56:33 [INFO] [TRAIN] epoch=175, iter=64900/80000, loss=0.1333, lr=0.002238, batch_cost=0.7572, reader_cost=0.0006 | ETA 03:10:33 2020-10-31 03:57:49 [INFO] [TRAIN] epoch=175, iter=65000/80000, loss=0.1138, lr=0.002225, batch_cost=0.7559, reader_cost=0.0004 | ETA 03:08:58 2020-10-31 03:59:04 [INFO] [TRAIN] epoch=175, iter=65100/80000, loss=0.1309, lr=0.002211, batch_cost=0.7549, reader_cost=0.0003 | ETA 03:07:27 2020-10-31 04:00:21 [INFO] [TRAIN] epoch=176, iter=65200/80000, loss=0.1154, lr=0.002198, batch_cost=0.7651, reader_cost=0.0085 | ETA 03:08:43 2020-10-31 04:01:36 [INFO] [TRAIN] epoch=176, iter=65300/80000, loss=0.1142, lr=0.002185, batch_cost=0.7568, reader_cost=0.0006 | ETA 03:05:24 2020-10-31 04:02:52 [INFO] [TRAIN] epoch=176, iter=65400/80000, loss=0.1274, lr=0.002171, batch_cost=0.7606, reader_cost=0.0002 | ETA 03:05:05 2020-10-31 04:04:09 [INFO] [TRAIN] epoch=177, iter=65500/80000, loss=0.1183, lr=0.002158, batch_cost=0.7692, reader_cost=0.0078 | ETA 03:05:53 2020-10-31 04:05:25 [INFO] [TRAIN] epoch=177, iter=65600/80000, loss=0.1271, lr=0.002145, batch_cost=0.7580, reader_cost=0.0002 | ETA 03:01:54 2020-10-31 04:06:41 [INFO] [TRAIN] epoch=177, iter=65700/80000, loss=0.1247, lr=0.002131, batch_cost=0.7576, reader_cost=0.0004 | ETA 03:00:33 2020-10-31 04:07:57 [INFO] [TRAIN] epoch=177, iter=65800/80000, loss=0.1242, lr=0.002118, batch_cost=0.7598, reader_cost=0.0004 | ETA 02:59:49 2020-10-31 04:09:14 [INFO] [TRAIN] epoch=178, iter=65900/80000, loss=0.1274, lr=0.002105, batch_cost=0.7672, reader_cost=0.0095 | ETA 03:00:18 2020-10-31 04:10:30 [INFO] [TRAIN] epoch=178, iter=66000/80000, loss=0.1309, lr=0.002091, batch_cost=0.7598, reader_cost=0.0012 | ETA 02:57:17 2020-10-31 04:11:46 [INFO] [TRAIN] epoch=178, iter=66100/80000, loss=0.1253, lr=0.002078, batch_cost=0.7644, reader_cost=0.0005 | ETA 02:57:04 2020-10-31 04:13:02 [INFO] [TRAIN] epoch=178, iter=66200/80000, loss=0.1369, lr=0.002064, batch_cost=0.7626, reader_cost=0.0010 | ETA 02:55:24 2020-10-31 04:14:19 [INFO] [TRAIN] epoch=179, iter=66300/80000, loss=0.1222, lr=0.002051, batch_cost=0.7633, reader_cost=0.0086 | ETA 02:54:17 2020-10-31 04:15:34 [INFO] [TRAIN] epoch=179, iter=66400/80000, loss=0.1362, lr=0.002038, batch_cost=0.7582, reader_cost=0.0004 | ETA 02:51:51 2020-10-31 04:16:50 [INFO] [TRAIN] epoch=179, iter=66500/80000, loss=0.1225, lr=0.002024, batch_cost=0.7578, reader_cost=0.0002 | ETA 02:50:30 2020-10-31 04:18:07 [INFO] [TRAIN] epoch=180, iter=66600/80000, loss=0.1255, lr=0.002011, batch_cost=0.7667, reader_cost=0.0089 | ETA 02:51:13 2020-10-31 04:19:22 [INFO] [TRAIN] epoch=180, iter=66700/80000, loss=0.1236, lr=0.001997, batch_cost=0.7518, reader_cost=0.0007 | ETA 02:46:39 2020-10-31 04:20:38 [INFO] [TRAIN] epoch=180, iter=66800/80000, loss=0.1301, lr=0.001984, batch_cost=0.7599, reader_cost=0.0006 | ETA 02:47:10 2020-10-31 04:21:54 [INFO] [TRAIN] epoch=180, iter=66900/80000, loss=0.1362, lr=0.001970, batch_cost=0.7594, reader_cost=0.0010 | ETA 02:45:48 2020-10-31 04:23:11 [INFO] [TRAIN] epoch=181, iter=67000/80000, loss=0.1078, lr=0.001957, batch_cost=0.7678, reader_cost=0.0084 | ETA 02:46:21 2020-10-31 04:24:27 [INFO] [TRAIN] epoch=181, iter=67100/80000, loss=0.1229, lr=0.001944, batch_cost=0.7594, reader_cost=0.0006 | ETA 02:43:16 2020-10-31 04:25:43 [INFO] [TRAIN] epoch=181, iter=67200/80000, loss=0.1159, lr=0.001930, batch_cost=0.7578, reader_cost=0.0005 | ETA 02:41:40 2020-10-31 04:26:58 [INFO] [TRAIN] epoch=181, iter=67300/80000, loss=0.1186, lr=0.001917, batch_cost=0.7583, reader_cost=0.0006 | ETA 02:40:30 2020-10-31 04:28:15 [INFO] [TRAIN] epoch=182, iter=67400/80000, loss=0.1155, lr=0.001903, batch_cost=0.7696, reader_cost=0.0084 | ETA 02:41:36 2020-10-31 04:29:31 [INFO] [TRAIN] epoch=182, iter=67500/80000, loss=0.1270, lr=0.001889, batch_cost=0.7586, reader_cost=0.0004 | ETA 02:38:02 2020-10-31 04:30:47 [INFO] [TRAIN] epoch=182, iter=67600/80000, loss=0.1146, lr=0.001876, batch_cost=0.7604, reader_cost=0.0003 | ETA 02:37:08 2020-10-31 04:32:03 [INFO] [TRAIN] epoch=182, iter=67700/80000, loss=0.1210, lr=0.001862, batch_cost=0.7572, reader_cost=0.0006 | ETA 02:35:13 2020-10-31 04:33:20 [INFO] [TRAIN] epoch=183, iter=67800/80000, loss=0.1147, lr=0.001849, batch_cost=0.7686, reader_cost=0.0088 | ETA 02:36:17 2020-10-31 04:34:37 [INFO] [TRAIN] epoch=183, iter=67900/80000, loss=0.1317, lr=0.001835, batch_cost=0.7667, reader_cost=0.0010 | ETA 02:34:36 2020-10-31 04:35:51 [INFO] [TRAIN] epoch=183, iter=68000/80000, loss=0.1257, lr=0.001822, batch_cost=0.7489, reader_cost=0.0004 | ETA 02:29:47 2020-10-31 04:37:08 [INFO] [TRAIN] epoch=184, iter=68100/80000, loss=0.1027, lr=0.001808, batch_cost=0.7632, reader_cost=0.0088 | ETA 02:31:22 2020-10-31 04:38:23 [INFO] [TRAIN] epoch=184, iter=68200/80000, loss=0.1213, lr=0.001794, batch_cost=0.7513, reader_cost=0.0005 | ETA 02:27:44 2020-10-31 04:39:38 [INFO] [TRAIN] epoch=184, iter=68300/80000, loss=0.1173, lr=0.001781, batch_cost=0.7559, reader_cost=0.0006 | ETA 02:27:23 2020-10-31 04:40:54 [INFO] [TRAIN] epoch=184, iter=68400/80000, loss=0.1299, lr=0.001767, batch_cost=0.7555, reader_cost=0.0005 | ETA 02:26:03 2020-10-31 04:42:11 [INFO] [TRAIN] epoch=185, iter=68500/80000, loss=0.1218, lr=0.001754, batch_cost=0.7673, reader_cost=0.0094 | ETA 02:27:03 2020-10-31 04:43:27 [INFO] [TRAIN] epoch=185, iter=68600/80000, loss=0.1264, lr=0.001740, batch_cost=0.7590, reader_cost=0.0010 | ETA 02:24:12 2020-10-31 04:44:43 [INFO] [TRAIN] epoch=185, iter=68700/80000, loss=0.1228, lr=0.001726, batch_cost=0.7645, reader_cost=0.0009 | ETA 02:23:59 2020-10-31 04:45:59 [INFO] [TRAIN] epoch=185, iter=68800/80000, loss=0.1445, lr=0.001713, batch_cost=0.7622, reader_cost=0.0008 | ETA 02:22:16 2020-10-31 04:47:16 [INFO] [TRAIN] epoch=186, iter=68900/80000, loss=0.1444, lr=0.001699, batch_cost=0.7705, reader_cost=0.0093 | ETA 02:22:32 2020-10-31 04:48:33 [INFO] [TRAIN] epoch=186, iter=69000/80000, loss=0.1374, lr=0.001685, batch_cost=0.7646, reader_cost=0.0007 | ETA 02:20:10 2020-10-31 04:49:50 [INFO] [TRAIN] epoch=186, iter=69100/80000, loss=0.1280, lr=0.001672, batch_cost=0.7695, reader_cost=0.0015 | ETA 02:19:47 2020-10-31 04:51:08 [INFO] [TRAIN] epoch=187, iter=69200/80000, loss=0.1234, lr=0.001658, batch_cost=0.7810, reader_cost=0.0088 | ETA 02:20:34 2020-10-31 04:52:25 [INFO] [TRAIN] epoch=187, iter=69300/80000, loss=0.1209, lr=0.001644, batch_cost=0.7668, reader_cost=0.0005 | ETA 02:16:44 2020-10-31 04:53:41 [INFO] [TRAIN] epoch=187, iter=69400/80000, loss=0.1335, lr=0.001630, batch_cost=0.7596, reader_cost=0.0002 | ETA 02:14:11 2020-10-31 04:54:56 [INFO] [TRAIN] epoch=187, iter=69500/80000, loss=0.1268, lr=0.001617, batch_cost=0.7573, reader_cost=0.0008 | ETA 02:12:31 2020-10-31 04:56:14 [INFO] [TRAIN] epoch=188, iter=69600/80000, loss=0.1445, lr=0.001603, batch_cost=0.7733, reader_cost=0.0086 | ETA 02:14:01 2020-10-31 04:57:29 [INFO] [TRAIN] epoch=188, iter=69700/80000, loss=0.1256, lr=0.001589, batch_cost=0.7590, reader_cost=0.0014 | ETA 02:10:17 2020-10-31 04:58:45 [INFO] [TRAIN] epoch=188, iter=69800/80000, loss=0.1091, lr=0.001575, batch_cost=0.7582, reader_cost=0.0008 | ETA 02:08:54 2020-10-31 05:00:02 [INFO] [TRAIN] epoch=188, iter=69900/80000, loss=0.1232, lr=0.001561, batch_cost=0.7642, reader_cost=0.0013 | ETA 02:08:38 2020-10-31 05:01:19 [INFO] [TRAIN] epoch=189, iter=70000/80000, loss=0.1165, lr=0.001548, batch_cost=0.7686, reader_cost=0.0083 | ETA 02:08:05 2020-10-31 05:02:34 [INFO] [TRAIN] epoch=189, iter=70100/80000, loss=0.1178, lr=0.001534, batch_cost=0.7549, reader_cost=0.0002 | ETA 02:04:33 2020-10-31 05:03:49 [INFO] [TRAIN] epoch=189, iter=70200/80000, loss=0.1224, lr=0.001520, batch_cost=0.7522, reader_cost=0.0003 | ETA 02:02:51 2020-10-31 05:05:05 [INFO] [TRAIN] epoch=189, iter=70300/80000, loss=0.1235, lr=0.001506, batch_cost=0.7603, reader_cost=0.0005 | ETA 02:02:55 2020-10-31 05:06:22 [INFO] [TRAIN] epoch=190, iter=70400/80000, loss=0.1190, lr=0.001492, batch_cost=0.7696, reader_cost=0.0093 | ETA 02:03:08 2020-10-31 05:07:38 [INFO] [TRAIN] epoch=190, iter=70500/80000, loss=0.1111, lr=0.001478, batch_cost=0.7598, reader_cost=0.0008 | ETA 02:00:18 2020-10-31 05:08:54 [INFO] [TRAIN] epoch=190, iter=70600/80000, loss=0.1278, lr=0.001464, batch_cost=0.7609, reader_cost=0.0005 | ETA 01:59:12 2020-10-31 05:10:11 [INFO] [TRAIN] epoch=191, iter=70700/80000, loss=0.1199, lr=0.001450, batch_cost=0.7616, reader_cost=0.0092 | ETA 01:58:02 2020-10-31 05:11:26 [INFO] [TRAIN] epoch=191, iter=70800/80000, loss=0.1291, lr=0.001436, batch_cost=0.7576, reader_cost=0.0006 | ETA 01:56:10 2020-10-31 05:12:42 [INFO] [TRAIN] epoch=191, iter=70900/80000, loss=0.1213, lr=0.001422, batch_cost=0.7539, reader_cost=0.0004 | ETA 01:54:20 2020-10-31 05:13:58 [INFO] [TRAIN] epoch=191, iter=71000/80000, loss=0.1461, lr=0.001408, batch_cost=0.7593, reader_cost=0.0004 | ETA 01:53:53 2020-10-31 05:15:14 [INFO] [TRAIN] epoch=192, iter=71100/80000, loss=0.1130, lr=0.001394, batch_cost=0.7620, reader_cost=0.0088 | ETA 01:53:01 2020-10-31 05:16:29 [INFO] [TRAIN] epoch=192, iter=71200/80000, loss=0.1319, lr=0.001380, batch_cost=0.7498, reader_cost=0.0002 | ETA 01:49:58 2020-10-31 05:17:45 [INFO] [TRAIN] epoch=192, iter=71300/80000, loss=0.1217, lr=0.001366, batch_cost=0.7617, reader_cost=0.0002 | ETA 01:50:27 2020-10-31 05:19:01 [INFO] [TRAIN] epoch=192, iter=71400/80000, loss=0.1273, lr=0.001352, batch_cost=0.7620, reader_cost=0.0004 | ETA 01:49:13 2020-10-31 05:20:18 [INFO] [TRAIN] epoch=193, iter=71500/80000, loss=0.1232, lr=0.001338, batch_cost=0.7638, reader_cost=0.0079 | ETA 01:48:12 2020-10-31 05:21:33 [INFO] [TRAIN] epoch=193, iter=71600/80000, loss=0.1214, lr=0.001324, batch_cost=0.7561, reader_cost=0.0002 | ETA 01:45:51 2020-10-31 05:22:49 [INFO] [TRAIN] epoch=193, iter=71700/80000, loss=0.1154, lr=0.001310, batch_cost=0.7588, reader_cost=0.0002 | ETA 01:44:57 2020-10-31 05:24:06 [INFO] [TRAIN] epoch=194, iter=71800/80000, loss=0.1251, lr=0.001296, batch_cost=0.7667, reader_cost=0.0076 | ETA 01:44:46 2020-10-31 05:25:22 [INFO] [TRAIN] epoch=194, iter=71900/80000, loss=0.1228, lr=0.001282, batch_cost=0.7580, reader_cost=0.0010 | ETA 01:42:19 2020-10-31 05:26:38 [INFO] [TRAIN] epoch=194, iter=72000/80000, loss=0.1182, lr=0.001268, batch_cost=0.7620, reader_cost=0.0010 | ETA 01:41:35 2020-10-31 05:26:45 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-10-31 05:33:07 [INFO] [EVAL] #Images=500 mIoU=0.8015 Acc=0.9643 Kappa=0.9536 2020-10-31 05:33:07 [INFO] [EVAL] Category IoU: [0.9831 0.8637 0.9318 0.5796 0.6307 0.6762 0.7433 0.812 0.9288 0.6328 0.9504 0.8378 0.6495 0.9568 0.8523 0.8928 0.8382 0.6748 0.7936] 2020-10-31 05:33:07 [INFO] [EVAL] Category Acc: [0.9924 0.9235 0.9605 0.8682 0.8676 0.8109 0.8414 0.9045 0.9565 0.8565 0.9674 0.892 0.8072 0.9735 0.9645 0.9657 0.9325 0.8643 0.8688] 2020-10-31 05:33:07 [INFO] [EVAL] The model with the best validation mIoU (0.8040) was saved at iter 64000. 2020-10-31 05:34:22 [INFO] [TRAIN] epoch=194, iter=72100/80000, loss=0.1264, lr=0.001254, batch_cost=0.7572, reader_cost=0.0012 | ETA 01:39:41 2020-10-31 05:35:39 [INFO] [TRAIN] epoch=195, iter=72200/80000, loss=0.1333, lr=0.001239, batch_cost=0.7666, reader_cost=0.0087 | ETA 01:39:39 2020-10-31 05:36:56 [INFO] [TRAIN] epoch=195, iter=72300/80000, loss=0.1315, lr=0.001225, batch_cost=0.7651, reader_cost=0.0006 | ETA 01:38:11 2020-10-31 05:38:12 [INFO] [TRAIN] epoch=195, iter=72400/80000, loss=0.1158, lr=0.001211, batch_cost=0.7629, reader_cost=0.0007 | ETA 01:36:37 2020-10-31 05:39:28 [INFO] [TRAIN] epoch=195, iter=72500/80000, loss=0.1190, lr=0.001197, batch_cost=0.7585, reader_cost=0.0002 | ETA 01:34:49 2020-10-31 05:40:44 [INFO] [TRAIN] epoch=196, iter=72600/80000, loss=0.1325, lr=0.001183, batch_cost=0.7636, reader_cost=0.0095 | ETA 01:34:10 2020-10-31 05:41:59 [INFO] [TRAIN] epoch=196, iter=72700/80000, loss=0.1270, lr=0.001168, batch_cost=0.7529, reader_cost=0.0006 | ETA 01:31:36 2020-10-31 05:43:15 [INFO] [TRAIN] epoch=196, iter=72800/80000, loss=0.1146, lr=0.001154, batch_cost=0.7566, reader_cost=0.0002 | ETA 01:30:47 2020-10-31 05:44:31 [INFO] [TRAIN] epoch=196, iter=72900/80000, loss=0.1251, lr=0.001140, batch_cost=0.7566, reader_cost=0.0002 | ETA 01:29:31 2020-10-31 05:45:47 [INFO] [TRAIN] epoch=197, iter=73000/80000, loss=0.1226, lr=0.001125, batch_cost=0.7651, reader_cost=0.0092 | ETA 01:29:15 2020-10-31 05:47:03 [INFO] [TRAIN] epoch=197, iter=73100/80000, loss=0.1229, lr=0.001111, batch_cost=0.7624, reader_cost=0.0006 | ETA 01:27:40 2020-10-31 05:48:21 [INFO] [TRAIN] epoch=197, iter=73200/80000, loss=0.1217, lr=0.001097, batch_cost=0.7718, reader_cost=0.0013 | ETA 01:27:28 2020-10-31 05:49:38 [INFO] [TRAIN] epoch=198, iter=73300/80000, loss=0.1044, lr=0.001082, batch_cost=0.7720, reader_cost=0.0088 | ETA 01:26:12 2020-10-31 05:50:54 [INFO] [TRAIN] epoch=198, iter=73400/80000, loss=0.1185, lr=0.001068, batch_cost=0.7626, reader_cost=0.0010 | ETA 01:23:53 2020-10-31 05:52:11 [INFO] [TRAIN] epoch=198, iter=73500/80000, loss=0.1206, lr=0.001053, batch_cost=0.7699, reader_cost=0.0012 | ETA 01:23:24 2020-10-31 05:53:28 [INFO] [TRAIN] epoch=198, iter=73600/80000, loss=0.1136, lr=0.001039, batch_cost=0.7669, reader_cost=0.0010 | ETA 01:21:48 2020-10-31 05:54:45 [INFO] [TRAIN] epoch=199, iter=73700/80000, loss=0.1144, lr=0.001025, batch_cost=0.7716, reader_cost=0.0088 | ETA 01:21:01 2020-10-31 05:56:01 [INFO] [TRAIN] epoch=199, iter=73800/80000, loss=0.1238, lr=0.001010, batch_cost=0.7604, reader_cost=0.0006 | ETA 01:18:34 2020-10-31 05:57:17 [INFO] [TRAIN] epoch=199, iter=73900/80000, loss=0.1193, lr=0.000995, batch_cost=0.7592, reader_cost=0.0012 | ETA 01:17:10 2020-10-31 05:58:34 [INFO] [TRAIN] epoch=199, iter=74000/80000, loss=0.1215, lr=0.000981, batch_cost=0.7691, reader_cost=0.0007 | ETA 01:16:54 2020-10-31 05:59:50 [INFO] [TRAIN] epoch=200, iter=74100/80000, loss=0.1108, lr=0.000966, batch_cost=0.7655, reader_cost=0.0109 | ETA 01:15:16 2020-10-31 06:01:06 [INFO] [TRAIN] epoch=200, iter=74200/80000, loss=0.1198, lr=0.000952, batch_cost=0.7587, reader_cost=0.0010 | ETA 01:13:20 2020-10-31 06:02:22 [INFO] [TRAIN] epoch=200, iter=74300/80000, loss=0.1127, lr=0.000937, batch_cost=0.7609, reader_cost=0.0014 | ETA 01:12:16 2020-10-31 06:03:38 [INFO] [TRAIN] epoch=200, iter=74400/80000, loss=0.1194, lr=0.000922, batch_cost=0.7592, reader_cost=0.0013 | ETA 01:10:51 2020-10-31 06:04:56 [INFO] [TRAIN] epoch=201, iter=74500/80000, loss=0.1185, lr=0.000908, batch_cost=0.7791, reader_cost=0.0103 | ETA 01:11:25 2020-10-31 06:06:13 [INFO] [TRAIN] epoch=201, iter=74600/80000, loss=0.1258, lr=0.000893, batch_cost=0.7663, reader_cost=0.0008 | ETA 01:08:57 2020-10-31 06:07:29 [INFO] [TRAIN] epoch=201, iter=74700/80000, loss=0.1252, lr=0.000878, batch_cost=0.7643, reader_cost=0.0005 | ETA 01:07:30 2020-10-31 06:08:47 [INFO] [TRAIN] epoch=202, iter=74800/80000, loss=0.1187, lr=0.000864, batch_cost=0.7744, reader_cost=0.0082 | ETA 01:07:06 2020-10-31 06:10:02 [INFO] [TRAIN] epoch=202, iter=74900/80000, loss=0.1163, lr=0.000849, batch_cost=0.7566, reader_cost=0.0003 | ETA 01:04:18 2020-10-31 06:11:19 [INFO] [TRAIN] epoch=202, iter=75000/80000, loss=0.1064, lr=0.000834, batch_cost=0.7652, reader_cost=0.0010 | ETA 01:03:46 2020-10-31 06:12:34 [INFO] [TRAIN] epoch=202, iter=75100/80000, loss=0.1229, lr=0.000819, batch_cost=0.7500, reader_cost=0.0005 | ETA 01:01:15 2020-10-31 06:13:50 [INFO] [TRAIN] epoch=203, iter=75200/80000, loss=0.1062, lr=0.000804, batch_cost=0.7560, reader_cost=0.0103 | ETA 01:00:28 2020-10-31 06:15:05 [INFO] [TRAIN] epoch=203, iter=75300/80000, loss=0.1281, lr=0.000789, batch_cost=0.7533, reader_cost=0.0006 | ETA 00:59:00 2020-10-31 06:16:20 [INFO] [TRAIN] epoch=203, iter=75400/80000, loss=0.1201, lr=0.000774, batch_cost=0.7540, reader_cost=0.0003 | ETA 00:57:48 2020-10-31 06:17:36 [INFO] [TRAIN] epoch=203, iter=75500/80000, loss=0.1160, lr=0.000759, batch_cost=0.7580, reader_cost=0.0005 | ETA 00:56:51 2020-10-31 06:18:53 [INFO] [TRAIN] epoch=204, iter=75600/80000, loss=0.1093, lr=0.000744, batch_cost=0.7690, reader_cost=0.0095 | ETA 00:56:23 2020-10-31 06:20:08 [INFO] [TRAIN] epoch=204, iter=75700/80000, loss=0.1319, lr=0.000729, batch_cost=0.7521, reader_cost=0.0002 | ETA 00:53:54 2020-10-31 06:21:25 [INFO] [TRAIN] epoch=204, iter=75800/80000, loss=0.1251, lr=0.000714, batch_cost=0.7647, reader_cost=0.0003 | ETA 00:53:31 2020-10-31 06:22:42 [INFO] [TRAIN] epoch=205, iter=75900/80000, loss=0.1121, lr=0.000699, batch_cost=0.7736, reader_cost=0.0100 | ETA 00:52:51 2020-10-31 06:23:58 [INFO] [TRAIN] epoch=205, iter=76000/80000, loss=0.1117, lr=0.000684, batch_cost=0.7634, reader_cost=0.0008 | ETA 00:50:53 2020-10-31 06:25:15 [INFO] [TRAIN] epoch=205, iter=76100/80000, loss=0.1141, lr=0.000669, batch_cost=0.7621, reader_cost=0.0004 | ETA 00:49:32 2020-10-31 06:26:30 [INFO] [TRAIN] epoch=205, iter=76200/80000, loss=0.1253, lr=0.000654, batch_cost=0.7589, reader_cost=0.0008 | ETA 00:48:03 2020-10-31 06:27:47 [INFO] [TRAIN] epoch=206, iter=76300/80000, loss=0.1088, lr=0.000638, batch_cost=0.7694, reader_cost=0.0101 | ETA 00:47:26 2020-10-31 06:29:04 [INFO] [TRAIN] epoch=206, iter=76400/80000, loss=0.1158, lr=0.000623, batch_cost=0.7651, reader_cost=0.0006 | ETA 00:45:54 2020-10-31 06:30:20 [INFO] [TRAIN] epoch=206, iter=76500/80000, loss=0.1202, lr=0.000608, batch_cost=0.7615, reader_cost=0.0007 | ETA 00:44:25 2020-10-31 06:31:37 [INFO] [TRAIN] epoch=206, iter=76600/80000, loss=0.1049, lr=0.000592, batch_cost=0.7681, reader_cost=0.0010 | ETA 00:43:31 2020-10-31 06:32:54 [INFO] [TRAIN] epoch=207, iter=76700/80000, loss=0.1082, lr=0.000577, batch_cost=0.7696, reader_cost=0.0088 | ETA 00:42:19 2020-10-31 06:34:10 [INFO] [TRAIN] epoch=207, iter=76800/80000, loss=0.1162, lr=0.000561, batch_cost=0.7627, reader_cost=0.0008 | ETA 00:40:40 2020-10-31 06:35:27 [INFO] [TRAIN] epoch=207, iter=76900/80000, loss=0.1265, lr=0.000546, batch_cost=0.7727, reader_cost=0.0005 | ETA 00:39:55 2020-10-31 06:36:44 [INFO] [TRAIN] epoch=207, iter=77000/80000, loss=0.1201, lr=0.000530, batch_cost=0.7653, reader_cost=0.0007 | ETA 00:38:16 2020-10-31 06:38:02 [INFO] [TRAIN] epoch=208, iter=77100/80000, loss=0.1159, lr=0.000515, batch_cost=0.7776, reader_cost=0.0094 | ETA 00:37:34 2020-10-31 06:39:19 [INFO] [TRAIN] epoch=208, iter=77200/80000, loss=0.1195, lr=0.000499, batch_cost=0.7706, reader_cost=0.0012 | ETA 00:35:57 2020-10-31 06:40:36 [INFO] [TRAIN] epoch=208, iter=77300/80000, loss=0.1216, lr=0.000483, batch_cost=0.7732, reader_cost=0.0012 | ETA 00:34:47 2020-10-31 06:41:53 [INFO] [TRAIN] epoch=209, iter=77400/80000, loss=0.1054, lr=0.000468, batch_cost=0.7670, reader_cost=0.0101 | ETA 00:33:14 2020-10-31 06:43:08 [INFO] [TRAIN] epoch=209, iter=77500/80000, loss=0.1169, lr=0.000452, batch_cost=0.7563, reader_cost=0.0006 | ETA 00:31:30 2020-10-31 06:44:25 [INFO] [TRAIN] epoch=209, iter=77600/80000, loss=0.1202, lr=0.000436, batch_cost=0.7619, reader_cost=0.0009 | ETA 00:30:28 2020-10-31 06:45:40 [INFO] [TRAIN] epoch=209, iter=77700/80000, loss=0.1081, lr=0.000420, batch_cost=0.7573, reader_cost=0.0007 | ETA 00:29:01 2020-10-31 06:46:57 [INFO] [TRAIN] epoch=210, iter=77800/80000, loss=0.1054, lr=0.000404, batch_cost=0.7705, reader_cost=0.0079 | ETA 00:28:15 2020-10-31 06:48:13 [INFO] [TRAIN] epoch=210, iter=77900/80000, loss=0.1181, lr=0.000388, batch_cost=0.7568, reader_cost=0.0003 | ETA 00:26:29 2020-10-31 06:49:29 [INFO] [TRAIN] epoch=210, iter=78000/80000, loss=0.1091, lr=0.000371, batch_cost=0.7575, reader_cost=0.0004 | ETA 00:25:14 2020-10-31 06:50:45 [INFO] [TRAIN] epoch=210, iter=78100/80000, loss=0.1073, lr=0.000355, batch_cost=0.7583, reader_cost=0.0002 | ETA 00:24:00 2020-10-31 06:52:01 [INFO] [TRAIN] epoch=211, iter=78200/80000, loss=0.1187, lr=0.000339, batch_cost=0.7667, reader_cost=0.0081 | ETA 00:23:00 2020-10-31 06:53:17 [INFO] [TRAIN] epoch=211, iter=78300/80000, loss=0.1298, lr=0.000322, batch_cost=0.7622, reader_cost=0.0006 | ETA 00:21:35 2020-10-31 06:54:34 [INFO] [TRAIN] epoch=211, iter=78400/80000, loss=0.1128, lr=0.000306, batch_cost=0.7644, reader_cost=0.0002 | ETA 00:20:23 2020-10-31 06:55:51 [INFO] [TRAIN] epoch=212, iter=78500/80000, loss=0.1280, lr=0.000289, batch_cost=0.7730, reader_cost=0.0088 | ETA 00:19:19 2020-10-31 06:57:07 [INFO] [TRAIN] epoch=212, iter=78600/80000, loss=0.1131, lr=0.000272, batch_cost=0.7586, reader_cost=0.0002 | ETA 00:17:42 2020-10-31 06:58:23 [INFO] [TRAIN] epoch=212, iter=78700/80000, loss=0.1178, lr=0.000255, batch_cost=0.7601, reader_cost=0.0009 | ETA 00:16:28 2020-10-31 06:59:39 [INFO] [TRAIN] epoch=212, iter=78800/80000, loss=0.1217, lr=0.000238, batch_cost=0.7613, reader_cost=0.0012 | ETA 00:15:13 2020-10-31 07:00:56 [INFO] [TRAIN] epoch=213, iter=78900/80000, loss=0.1174, lr=0.000221, batch_cost=0.7723, reader_cost=0.0098 | ETA 00:14:09 2020-10-31 07:02:12 [INFO] [TRAIN] epoch=213, iter=79000/80000, loss=0.1211, lr=0.000204, batch_cost=0.7555, reader_cost=0.0007 | ETA 00:12:35 2020-10-31 07:03:27 [INFO] [TRAIN] epoch=213, iter=79100/80000, loss=0.1150, lr=0.000186, batch_cost=0.7545, reader_cost=0.0002 | ETA 00:11:19 2020-10-31 07:04:43 [INFO] [TRAIN] epoch=213, iter=79200/80000, loss=0.1165, lr=0.000169, batch_cost=0.7576, reader_cost=0.0008 | ETA 00:10:06 2020-10-31 07:05:59 [INFO] [TRAIN] epoch=214, iter=79300/80000, loss=0.1037, lr=0.000151, batch_cost=0.7598, reader_cost=0.0082 | ETA 00:08:51 2020-10-31 07:07:16 [INFO] [TRAIN] epoch=214, iter=79400/80000, loss=0.1216, lr=0.000132, batch_cost=0.7670, reader_cost=0.0010 | ETA 00:07:40 2020-10-31 07:08:32 [INFO] [TRAIN] epoch=214, iter=79500/80000, loss=0.1143, lr=0.000114, batch_cost=0.7657, reader_cost=0.0009 | ETA 00:06:22 2020-10-31 07:09:49 [INFO] [TRAIN] epoch=214, iter=79600/80000, loss=0.1142, lr=0.000095, batch_cost=0.7665, reader_cost=0.0004 | ETA 00:05:06 2020-10-31 07:11:06 [INFO] [TRAIN] epoch=215, iter=79700/80000, loss=0.1165, lr=0.000076, batch_cost=0.7716, reader_cost=0.0093 | ETA 00:03:51 2020-10-31 07:12:23 [INFO] [TRAIN] epoch=215, iter=79800/80000, loss=0.1134, lr=0.000056, batch_cost=0.7654, reader_cost=0.0011 | ETA 00:02:33 2020-10-31 07:13:39 [INFO] [TRAIN] epoch=215, iter=79900/80000, loss=0.1224, lr=0.000035, batch_cost=0.7654, reader_cost=0.0008 | ETA 00:01:16 2020-10-31 07:14:57 [INFO] [TRAIN] epoch=216, iter=80000/80000, loss=0.1173, lr=0.000010, batch_cost=0.7747, reader_cost=0.0100 | ETA 00:00:00 2020-10-31 07:15:05 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-10-31 07:21:18 [INFO] [EVAL] #Images=500 mIoU=0.8061 Acc=0.9653 Kappa=0.9550 2020-10-31 07:21:18 [INFO] [EVAL] Category IoU: [0.9848 0.8698 0.9329 0.5729 0.6335 0.6804 0.7444 0.8141 0.9295 0.6379 0.951 0.8433 0.6661 0.9582 0.8588 0.9052 0.8477 0.688 0.7982] 2020-10-31 07:21:18 [INFO] [EVAL] Category Acc: [0.992 0.9337 0.962 0.8954 0.8378 0.8204 0.8399 0.9053 0.9544 0.887 0.9689 0.9005 0.8029 0.9755 0.9592 0.9651 0.9423 0.8554 0.8763] 2020-10-31 07:21:22 [INFO] [EVAL] The model with the best validation mIoU (0.8061) was saved at iter 80000.