2020-11-03 08:10:45 [INFO] ------------Environment Information------------- platform: Linux-3.10.0_3-0-0-34-x86_64-with-centos-7.5.1804-Core Python: 3.7.9 (default, Aug 31 2020, 12:42:55) [GCC 7.3.0] Paddle compiled with cuda: True NVCC: Cuda compilation tools, release 10.2, V10.2.89 cudnn: 7.6 GPUs used: 4 CUDA_VISIBLE_DEVICES: 0,1,2,3 GPU: ['GPU 0: Tesla V100-SXM2-16GB', 'GPU 1: Tesla V100-SXM2-16GB', 'GPU 2: Tesla V100-SXM2-16GB', 'GPU 3: Tesla V100-SXM2-16GB', 'GPU 4: Tesla V100-SXM2-16GB', 'GPU 5: Tesla V100-SXM2-16GB', 'GPU 6: Tesla V100-SXM2-16GB', 'GPU 7: Tesla V100-SXM2-16GB'] GCC: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36) PaddlePaddle: 2.0.0-rc0 OpenCV: 4.1.0 ------------------------------------------------ 2020-11-03 08:10:45 [INFO] ---------------Config Information--------------- batch_size: 2 iters: 80000 learning_rate: decay: end_lr: 1.0e-05 power: 0.9 type: poly value: 0.01 loss: coef: - 1 - 0.4 types: - ignore_index: 255 type: CrossEntropyLoss model: align_corners: false backbone: output_stride: 8 pretrained: https://bj.bcebos.com/paddleseg/dygraph/resnet50_vd_ssld_v2.tar.gz type: ResNet50_vd enable_auxiliary_loss: true pretrained: null type: PSPNet optimizer: momentum: 0.9 type: sgd weight_decay: 4.0e-05 train_dataset: dataset_root: data/cityscapes mode: train transforms: - max_scale_factor: 2.0 min_scale_factor: 0.5 scale_step_size: 0.25 type: ResizeStepScaling - crop_size: - 1024 - 512 type: RandomPaddingCrop - type: RandomHorizontalFlip - brightness_range: 0.4 contrast_range: 0.4 saturation_range: 0.4 type: RandomDistort - type: Normalize type: Cityscapes val_dataset: dataset_root: data/cityscapes mode: val transforms: - type: Normalize type: Cityscapes ------------------------------------------------ 2020-11-03 08:10:50 [INFO] Loading pretrained model from https://bj.bcebos.com/paddleseg/dygraph/resnet50_vd_ssld_v2.tar.gz 2020-11-03 08:10:51 [INFO] There are 275/275 variables loaded into ResNet_vd. 2020-11-03 08:11:55 [INFO] [TRAIN] epoch=1, iter=100/80000, loss=1.7607, lr=0.009989, batch_cost=0.5754, reader_cost=0.0135 | ETA 12:46:16 2020-11-03 08:12:48 [INFO] [TRAIN] epoch=1, iter=200/80000, loss=1.4146, lr=0.009978, batch_cost=0.5324, reader_cost=0.0004 | ETA 11:48:04 2020-11-03 08:13:42 [INFO] [TRAIN] epoch=1, iter=300/80000, loss=1.2818, lr=0.009966, batch_cost=0.5345, reader_cost=0.0004 | ETA 11:49:56 2020-11-03 08:14:36 [INFO] [TRAIN] epoch=2, iter=400/80000, loss=0.8663, lr=0.009955, batch_cost=0.5456, reader_cost=0.0095 | ETA 12:03:53 2020-11-03 08:15:30 [INFO] [TRAIN] epoch=2, iter=500/80000, loss=0.7211, lr=0.009944, batch_cost=0.5365, reader_cost=0.0005 | ETA 11:50:50 2020-11-03 08:16:24 [INFO] [TRAIN] epoch=2, iter=600/80000, loss=0.6962, lr=0.009933, batch_cost=0.5379, reader_cost=0.0008 | ETA 11:51:47 2020-11-03 08:17:17 [INFO] [TRAIN] epoch=2, iter=700/80000, loss=0.6531, lr=0.009921, batch_cost=0.5372, reader_cost=0.0009 | ETA 11:49:57 2020-11-03 08:18:12 [INFO] [TRAIN] epoch=3, iter=800/80000, loss=0.5860, lr=0.009910, batch_cost=0.5454, reader_cost=0.0091 | ETA 11:59:58 2020-11-03 08:19:06 [INFO] [TRAIN] epoch=3, iter=900/80000, loss=0.5924, lr=0.009899, batch_cost=0.5374, reader_cost=0.0007 | ETA 11:48:29 2020-11-03 08:19:59 [INFO] [TRAIN] epoch=3, iter=1000/80000, loss=0.5373, lr=0.009888, batch_cost=0.5374, reader_cost=0.0010 | ETA 11:47:38 2020-11-03 08:20:53 [INFO] [TRAIN] epoch=3, iter=1100/80000, loss=0.5517, lr=0.009876, batch_cost=0.5332, reader_cost=0.0005 | ETA 11:41:09 2020-11-03 08:21:47 [INFO] [TRAIN] epoch=4, iter=1200/80000, loss=0.5089, lr=0.009865, batch_cost=0.5415, reader_cost=0.0092 | ETA 11:51:12 2020-11-03 08:22:40 [INFO] [TRAIN] epoch=4, iter=1300/80000, loss=0.5211, lr=0.009854, batch_cost=0.5341, reader_cost=0.0003 | ETA 11:40:36 2020-11-03 08:23:34 [INFO] [TRAIN] epoch=4, iter=1400/80000, loss=0.4709, lr=0.009843, batch_cost=0.5356, reader_cost=0.0004 | ETA 11:41:35 2020-11-03 08:24:28 [INFO] [TRAIN] epoch=5, iter=1500/80000, loss=0.4364, lr=0.009831, batch_cost=0.5443, reader_cost=0.0074 | ETA 11:52:11 2020-11-03 08:25:22 [INFO] [TRAIN] epoch=5, iter=1600/80000, loss=0.4185, lr=0.009820, batch_cost=0.5343, reader_cost=0.0003 | ETA 11:38:11 2020-11-03 08:26:15 [INFO] [TRAIN] epoch=5, iter=1700/80000, loss=0.4348, lr=0.009809, batch_cost=0.5324, reader_cost=0.0004 | ETA 11:34:48 2020-11-03 08:27:08 [INFO] [TRAIN] epoch=5, iter=1800/80000, loss=0.4492, lr=0.009798, batch_cost=0.5316, reader_cost=0.0004 | ETA 11:32:52 2020-11-03 08:28:03 [INFO] [TRAIN] epoch=6, iter=1900/80000, loss=0.4405, lr=0.009786, batch_cost=0.5476, reader_cost=0.0069 | ETA 11:52:49 2020-11-03 08:28:57 [INFO] [TRAIN] epoch=6, iter=2000/80000, loss=0.4326, lr=0.009775, batch_cost=0.5396, reader_cost=0.0007 | ETA 11:41:29 2020-11-03 08:29:51 [INFO] [TRAIN] epoch=6, iter=2100/80000, loss=0.4424, lr=0.009764, batch_cost=0.5375, reader_cost=0.0006 | ETA 11:37:52 2020-11-03 08:30:44 [INFO] [TRAIN] epoch=6, iter=2200/80000, loss=0.4435, lr=0.009753, batch_cost=0.5381, reader_cost=0.0009 | ETA 11:37:43 2020-11-03 08:31:39 [INFO] [TRAIN] epoch=7, iter=2300/80000, loss=0.4128, lr=0.009741, batch_cost=0.5439, reader_cost=0.0089 | ETA 11:44:19 2020-11-03 08:32:32 [INFO] [TRAIN] epoch=7, iter=2400/80000, loss=0.4114, lr=0.009730, batch_cost=0.5362, reader_cost=0.0004 | ETA 11:33:26 2020-11-03 08:33:26 [INFO] [TRAIN] epoch=7, iter=2500/80000, loss=0.4468, lr=0.009719, batch_cost=0.5389, reader_cost=0.0005 | ETA 11:36:01 2020-11-03 08:34:20 [INFO] [TRAIN] epoch=7, iter=2600/80000, loss=0.3939, lr=0.009707, batch_cost=0.5389, reader_cost=0.0012 | ETA 11:35:12 2020-11-03 08:35:15 [INFO] [TRAIN] epoch=8, iter=2700/80000, loss=0.4643, lr=0.009696, batch_cost=0.5461, reader_cost=0.0096 | ETA 11:43:32 2020-11-03 08:36:08 [INFO] [TRAIN] epoch=8, iter=2800/80000, loss=0.3722, lr=0.009685, batch_cost=0.5350, reader_cost=0.0007 | ETA 11:28:19 2020-11-03 08:37:02 [INFO] [TRAIN] epoch=8, iter=2900/80000, loss=0.3809, lr=0.009674, batch_cost=0.5371, reader_cost=0.0010 | ETA 11:30:11 2020-11-03 08:37:57 [INFO] [TRAIN] epoch=9, iter=3000/80000, loss=0.3211, lr=0.009662, batch_cost=0.5459, reader_cost=0.0083 | ETA 11:40:30 2020-11-03 08:38:50 [INFO] [TRAIN] epoch=9, iter=3100/80000, loss=0.3777, lr=0.009651, batch_cost=0.5395, reader_cost=0.0007 | ETA 11:31:27 2020-11-03 08:39:44 [INFO] [TRAIN] epoch=9, iter=3200/80000, loss=0.4143, lr=0.009640, batch_cost=0.5384, reader_cost=0.0003 | ETA 11:29:07 2020-11-03 08:40:38 [INFO] [TRAIN] epoch=9, iter=3300/80000, loss=0.3689, lr=0.009628, batch_cost=0.5377, reader_cost=0.0007 | ETA 11:27:18 2020-11-03 08:41:33 [INFO] [TRAIN] epoch=10, iter=3400/80000, loss=0.3092, lr=0.009617, batch_cost=0.5442, reader_cost=0.0089 | ETA 11:34:44 2020-11-03 08:42:26 [INFO] [TRAIN] epoch=10, iter=3500/80000, loss=0.3102, lr=0.009606, batch_cost=0.5350, reader_cost=0.0006 | ETA 11:22:07 2020-11-03 08:43:20 [INFO] [TRAIN] epoch=10, iter=3600/80000, loss=0.4306, lr=0.009595, batch_cost=0.5358, reader_cost=0.0006 | ETA 11:22:15 2020-11-03 08:44:13 [INFO] [TRAIN] epoch=10, iter=3700/80000, loss=0.4684, lr=0.009583, batch_cost=0.5387, reader_cost=0.0004 | ETA 11:25:00 2020-11-03 08:45:08 [INFO] [TRAIN] epoch=11, iter=3800/80000, loss=0.3483, lr=0.009572, batch_cost=0.5441, reader_cost=0.0092 | ETA 11:31:00 2020-11-03 08:46:02 [INFO] [TRAIN] epoch=11, iter=3900/80000, loss=0.3937, lr=0.009561, batch_cost=0.5367, reader_cost=0.0005 | ETA 11:20:44 2020-11-03 08:46:55 [INFO] [TRAIN] epoch=11, iter=4000/80000, loss=0.3413, lr=0.009549, batch_cost=0.5361, reader_cost=0.0006 | ETA 11:19:04 2020-11-03 08:47:50 [INFO] [TRAIN] epoch=12, iter=4100/80000, loss=0.3326, lr=0.009538, batch_cost=0.5462, reader_cost=0.0076 | ETA 11:30:56 2020-11-03 08:48:43 [INFO] [TRAIN] epoch=12, iter=4200/80000, loss=0.3779, lr=0.009527, batch_cost=0.5366, reader_cost=0.0004 | ETA 11:17:57 2020-11-03 08:49:37 [INFO] [TRAIN] epoch=12, iter=4300/80000, loss=0.3784, lr=0.009516, batch_cost=0.5347, reader_cost=0.0005 | ETA 11:14:38 2020-11-03 08:50:30 [INFO] [TRAIN] epoch=12, iter=4400/80000, loss=0.3481, lr=0.009504, batch_cost=0.5348, reader_cost=0.0004 | ETA 11:13:50 2020-11-03 08:51:25 [INFO] [TRAIN] epoch=13, iter=4500/80000, loss=0.3316, lr=0.009493, batch_cost=0.5436, reader_cost=0.0080 | ETA 11:23:58 2020-11-03 08:52:18 [INFO] [TRAIN] epoch=13, iter=4600/80000, loss=0.3931, lr=0.009482, batch_cost=0.5373, reader_cost=0.0003 | ETA 11:15:08 2020-11-03 08:53:12 [INFO] [TRAIN] epoch=13, iter=4700/80000, loss=0.3655, lr=0.009470, batch_cost=0.5352, reader_cost=0.0004 | ETA 11:11:39 2020-11-03 08:54:06 [INFO] [TRAIN] epoch=13, iter=4800/80000, loss=0.4359, lr=0.009459, batch_cost=0.5404, reader_cost=0.0004 | ETA 11:17:15 2020-11-03 08:55:01 [INFO] [TRAIN] epoch=14, iter=4900/80000, loss=0.4036, lr=0.009448, batch_cost=0.5445, reader_cost=0.0077 | ETA 11:21:35 2020-11-03 08:55:54 [INFO] [TRAIN] epoch=14, iter=5000/80000, loss=0.3832, lr=0.009436, batch_cost=0.5365, reader_cost=0.0009 | ETA 11:10:39 2020-11-03 08:56:48 [INFO] [TRAIN] epoch=14, iter=5100/80000, loss=0.3627, lr=0.009425, batch_cost=0.5366, reader_cost=0.0006 | ETA 11:09:54 2020-11-03 08:57:42 [INFO] [TRAIN] epoch=14, iter=5200/80000, loss=0.3503, lr=0.009414, batch_cost=0.5379, reader_cost=0.0006 | ETA 11:10:33 2020-11-03 08:58:36 [INFO] [TRAIN] epoch=15, iter=5300/80000, loss=0.3169, lr=0.009402, batch_cost=0.5476, reader_cost=0.0077 | ETA 11:21:43 2020-11-03 08:59:30 [INFO] [TRAIN] epoch=15, iter=5400/80000, loss=0.3115, lr=0.009391, batch_cost=0.5383, reader_cost=0.0007 | ETA 11:09:19 2020-11-03 09:00:24 [INFO] [TRAIN] epoch=15, iter=5500/80000, loss=0.3330, lr=0.009380, batch_cost=0.5379, reader_cost=0.0009 | ETA 11:07:50 2020-11-03 09:01:18 [INFO] [TRAIN] epoch=16, iter=5600/80000, loss=0.3344, lr=0.009368, batch_cost=0.5444, reader_cost=0.0072 | ETA 11:15:03 2020-11-03 09:02:12 [INFO] [TRAIN] epoch=16, iter=5700/80000, loss=0.3571, lr=0.009357, batch_cost=0.5392, reader_cost=0.0005 | ETA 11:07:39 2020-11-03 09:03:06 [INFO] [TRAIN] epoch=16, iter=5800/80000, loss=0.3027, lr=0.009346, batch_cost=0.5367, reader_cost=0.0005 | ETA 11:03:42 2020-11-03 09:04:00 [INFO] [TRAIN] epoch=16, iter=5900/80000, loss=0.3129, lr=0.009335, batch_cost=0.5370, reader_cost=0.0005 | ETA 11:03:13 2020-11-03 09:04:54 [INFO] [TRAIN] epoch=17, iter=6000/80000, loss=0.3413, lr=0.009323, batch_cost=0.5439, reader_cost=0.0087 | ETA 11:10:50 2020-11-03 09:05:48 [INFO] [TRAIN] epoch=17, iter=6100/80000, loss=0.3099, lr=0.009312, batch_cost=0.5384, reader_cost=0.0008 | ETA 11:03:04 2020-11-03 09:06:42 [INFO] [TRAIN] epoch=17, iter=6200/80000, loss=0.2736, lr=0.009301, batch_cost=0.5378, reader_cost=0.0003 | ETA 11:01:30 2020-11-03 09:07:36 [INFO] [TRAIN] epoch=17, iter=6300/80000, loss=0.3381, lr=0.009289, batch_cost=0.5391, reader_cost=0.0007 | ETA 11:02:14 2020-11-03 09:08:30 [INFO] [TRAIN] epoch=18, iter=6400/80000, loss=0.3440, lr=0.009278, batch_cost=0.5438, reader_cost=0.0086 | ETA 11:07:02 2020-11-03 09:09:24 [INFO] [TRAIN] epoch=18, iter=6500/80000, loss=0.2919, lr=0.009267, batch_cost=0.5377, reader_cost=0.0007 | ETA 10:58:37 2020-11-03 09:10:18 [INFO] [TRAIN] epoch=18, iter=6600/80000, loss=0.2967, lr=0.009255, batch_cost=0.5383, reader_cost=0.0008 | ETA 10:58:30 2020-11-03 09:11:12 [INFO] [TRAIN] epoch=19, iter=6700/80000, loss=0.2938, lr=0.009244, batch_cost=0.5460, reader_cost=0.0089 | ETA 11:06:59 2020-11-03 09:12:06 [INFO] [TRAIN] epoch=19, iter=6800/80000, loss=0.2843, lr=0.009233, batch_cost=0.5356, reader_cost=0.0005 | ETA 10:53:29 2020-11-03 09:13:00 [INFO] [TRAIN] epoch=19, iter=6900/80000, loss=0.3188, lr=0.009221, batch_cost=0.5376, reader_cost=0.0004 | ETA 10:54:59 2020-11-03 09:13:53 [INFO] [TRAIN] epoch=19, iter=7000/80000, loss=0.3046, lr=0.009210, batch_cost=0.5376, reader_cost=0.0006 | ETA 10:54:07 2020-11-03 09:14:48 [INFO] [TRAIN] epoch=20, iter=7100/80000, loss=0.2444, lr=0.009199, batch_cost=0.5430, reader_cost=0.0085 | ETA 10:59:42 2020-11-03 09:15:41 [INFO] [TRAIN] epoch=20, iter=7200/80000, loss=0.2854, lr=0.009187, batch_cost=0.5383, reader_cost=0.0004 | ETA 10:53:04 2020-11-03 09:16:35 [INFO] [TRAIN] epoch=20, iter=7300/80000, loss=0.3466, lr=0.009176, batch_cost=0.5399, reader_cost=0.0007 | ETA 10:54:14 2020-11-03 09:17:29 [INFO] [TRAIN] epoch=20, iter=7400/80000, loss=0.3076, lr=0.009164, batch_cost=0.5399, reader_cost=0.0009 | ETA 10:53:14 2020-11-03 09:18:24 [INFO] [TRAIN] epoch=21, iter=7500/80000, loss=0.3111, lr=0.009153, batch_cost=0.5460, reader_cost=0.0086 | ETA 10:59:45 2020-11-03 09:19:18 [INFO] [TRAIN] epoch=21, iter=7600/80000, loss=0.3011, lr=0.009142, batch_cost=0.5398, reader_cost=0.0010 | ETA 10:51:24 2020-11-03 09:20:12 [INFO] [TRAIN] epoch=21, iter=7700/80000, loss=0.2957, lr=0.009130, batch_cost=0.5382, reader_cost=0.0010 | ETA 10:48:29 2020-11-03 09:21:06 [INFO] [TRAIN] epoch=21, iter=7800/80000, loss=0.3345, lr=0.009119, batch_cost=0.5398, reader_cost=0.0011 | ETA 10:49:33 2020-11-03 09:22:00 [INFO] [TRAIN] epoch=22, iter=7900/80000, loss=0.2808, lr=0.009108, batch_cost=0.5415, reader_cost=0.0084 | ETA 10:50:45 2020-11-03 09:22:53 [INFO] [TRAIN] epoch=22, iter=8000/80000, loss=0.2511, lr=0.009096, batch_cost=0.5337, reader_cost=0.0004 | ETA 10:40:25 2020-11-03 09:23:00 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-03 09:28:24 [INFO] [EVAL] #Images=500 mIoU=0.7058 Acc=0.9461 Kappa=0.9298 2020-11-03 09:28:24 [INFO] [EVAL] Category IoU: [0.9671 0.7683 0.9011 0.3648 0.5475 0.5744 0.6368 0.7321 0.9089 0.5293 0.9213 0.7723 0.4737 0.93 0.6642 0.7813 0.6423 0.5584 0.7372] 2020-11-03 09:28:24 [INFO] [EVAL] Category Acc: [0.9789 0.8901 0.9359 0.8379 0.7249 0.775 0.8066 0.8567 0.9391 0.8827 0.9626 0.8693 0.7834 0.9646 0.925 0.8667 0.8914 0.7899 0.8735] 2020-11-03 09:28:27 [INFO] [EVAL] The model with the best validation mIoU (0.7058) was saved at iter 8000. 2020-11-03 09:29:21 [INFO] [TRAIN] epoch=22, iter=8100/80000, loss=0.2901, lr=0.009085, batch_cost=0.5334, reader_cost=0.0007 | ETA 10:39:11 2020-11-03 09:30:15 [INFO] [TRAIN] epoch=23, iter=8200/80000, loss=0.3004, lr=0.009074, batch_cost=0.5424, reader_cost=0.0094 | ETA 10:49:05 2020-11-03 09:31:08 [INFO] [TRAIN] epoch=23, iter=8300/80000, loss=0.3309, lr=0.009062, batch_cost=0.5346, reader_cost=0.0008 | ETA 10:38:53 2020-11-03 09:32:02 [INFO] [TRAIN] epoch=23, iter=8400/80000, loss=0.3396, lr=0.009051, batch_cost=0.5358, reader_cost=0.0005 | ETA 10:39:22 2020-11-03 09:32:55 [INFO] [TRAIN] epoch=23, iter=8500/80000, loss=0.3308, lr=0.009040, batch_cost=0.5345, reader_cost=0.0004 | ETA 10:36:58 2020-11-03 09:33:49 [INFO] [TRAIN] epoch=24, iter=8600/80000, loss=0.2865, lr=0.009028, batch_cost=0.5412, reader_cost=0.0068 | ETA 10:44:03 2020-11-03 09:34:43 [INFO] [TRAIN] epoch=24, iter=8700/80000, loss=0.2924, lr=0.009017, batch_cost=0.5338, reader_cost=0.0003 | ETA 10:34:20 2020-11-03 09:35:37 [INFO] [TRAIN] epoch=24, iter=8800/80000, loss=0.2751, lr=0.009005, batch_cost=0.5374, reader_cost=0.0005 | ETA 10:37:46 2020-11-03 09:36:30 [INFO] [TRAIN] epoch=24, iter=8900/80000, loss=0.2673, lr=0.008994, batch_cost=0.5340, reader_cost=0.0008 | ETA 10:32:50 2020-11-03 09:37:25 [INFO] [TRAIN] epoch=25, iter=9000/80000, loss=0.2679, lr=0.008983, batch_cost=0.5460, reader_cost=0.0080 | ETA 10:46:06 2020-11-03 09:38:18 [INFO] [TRAIN] epoch=25, iter=9100/80000, loss=0.2932, lr=0.008971, batch_cost=0.5347, reader_cost=0.0003 | ETA 10:31:48 2020-11-03 09:39:12 [INFO] [TRAIN] epoch=25, iter=9200/80000, loss=0.2952, lr=0.008960, batch_cost=0.5356, reader_cost=0.0005 | ETA 10:31:59 2020-11-03 09:40:05 [INFO] [TRAIN] epoch=25, iter=9300/80000, loss=0.2789, lr=0.008949, batch_cost=0.5362, reader_cost=0.0008 | ETA 10:31:50 2020-11-03 09:41:00 [INFO] [TRAIN] epoch=26, iter=9400/80000, loss=0.2526, lr=0.008937, batch_cost=0.5445, reader_cost=0.0072 | ETA 10:40:44 2020-11-03 09:41:53 [INFO] [TRAIN] epoch=26, iter=9500/80000, loss=0.2826, lr=0.008926, batch_cost=0.5379, reader_cost=0.0006 | ETA 10:31:59 2020-11-03 09:42:47 [INFO] [TRAIN] epoch=26, iter=9600/80000, loss=0.2967, lr=0.008914, batch_cost=0.5397, reader_cost=0.0009 | ETA 10:33:18 2020-11-03 09:43:42 [INFO] [TRAIN] epoch=27, iter=9700/80000, loss=0.2589, lr=0.008903, batch_cost=0.5423, reader_cost=0.0071 | ETA 10:35:20 2020-11-03 09:44:35 [INFO] [TRAIN] epoch=27, iter=9800/80000, loss=0.2716, lr=0.008892, batch_cost=0.5349, reader_cost=0.0004 | ETA 10:25:50 2020-11-03 09:45:29 [INFO] [TRAIN] epoch=27, iter=9900/80000, loss=0.2806, lr=0.008880, batch_cost=0.5362, reader_cost=0.0005 | ETA 10:26:24 2020-11-03 09:46:22 [INFO] [TRAIN] epoch=27, iter=10000/80000, loss=0.2842, lr=0.008869, batch_cost=0.5350, reader_cost=0.0004 | ETA 10:24:11 2020-11-03 09:47:17 [INFO] [TRAIN] epoch=28, iter=10100/80000, loss=0.2395, lr=0.008857, batch_cost=0.5424, reader_cost=0.0083 | ETA 10:31:52 2020-11-03 09:48:10 [INFO] [TRAIN] epoch=28, iter=10200/80000, loss=0.2640, lr=0.008846, batch_cost=0.5326, reader_cost=0.0004 | ETA 10:19:34 2020-11-03 09:49:03 [INFO] [TRAIN] epoch=28, iter=10300/80000, loss=0.2685, lr=0.008835, batch_cost=0.5364, reader_cost=0.0006 | ETA 10:23:08 2020-11-03 09:49:57 [INFO] [TRAIN] epoch=28, iter=10400/80000, loss=0.2661, lr=0.008823, batch_cost=0.5335, reader_cost=0.0005 | ETA 10:18:52 2020-11-03 09:50:51 [INFO] [TRAIN] epoch=29, iter=10500/80000, loss=0.2793, lr=0.008812, batch_cost=0.5447, reader_cost=0.0078 | ETA 10:30:54 2020-11-03 09:51:45 [INFO] [TRAIN] epoch=29, iter=10600/80000, loss=0.2621, lr=0.008801, batch_cost=0.5345, reader_cost=0.0004 | ETA 10:18:15 2020-11-03 09:52:38 [INFO] [TRAIN] epoch=29, iter=10700/80000, loss=0.2626, lr=0.008789, batch_cost=0.5364, reader_cost=0.0005 | ETA 10:19:31 2020-11-03 09:53:33 [INFO] [TRAIN] epoch=30, iter=10800/80000, loss=0.2450, lr=0.008778, batch_cost=0.5442, reader_cost=0.0087 | ETA 10:27:38 2020-11-03 09:54:27 [INFO] [TRAIN] epoch=30, iter=10900/80000, loss=0.2428, lr=0.008766, batch_cost=0.5381, reader_cost=0.0005 | ETA 10:19:42 2020-11-03 09:55:20 [INFO] [TRAIN] epoch=30, iter=11000/80000, loss=0.2711, lr=0.008755, batch_cost=0.5366, reader_cost=0.0008 | ETA 10:17:06 2020-11-03 09:56:14 [INFO] [TRAIN] epoch=30, iter=11100/80000, loss=0.2605, lr=0.008743, batch_cost=0.5406, reader_cost=0.0012 | ETA 10:20:49 2020-11-03 09:57:09 [INFO] [TRAIN] epoch=31, iter=11200/80000, loss=0.2404, lr=0.008732, batch_cost=0.5440, reader_cost=0.0089 | ETA 10:23:50 2020-11-03 09:58:03 [INFO] [TRAIN] epoch=31, iter=11300/80000, loss=0.2410, lr=0.008721, batch_cost=0.5396, reader_cost=0.0004 | ETA 10:17:47 2020-11-03 09:58:57 [INFO] [TRAIN] epoch=31, iter=11400/80000, loss=0.2681, lr=0.008709, batch_cost=0.5384, reader_cost=0.0009 | ETA 10:15:35 2020-11-03 09:59:50 [INFO] [TRAIN] epoch=31, iter=11500/80000, loss=0.2638, lr=0.008698, batch_cost=0.5377, reader_cost=0.0007 | ETA 10:13:52 2020-11-03 10:00:45 [INFO] [TRAIN] epoch=32, iter=11600/80000, loss=0.2242, lr=0.008686, batch_cost=0.5453, reader_cost=0.0095 | ETA 10:21:36 2020-11-03 10:01:38 [INFO] [TRAIN] epoch=32, iter=11700/80000, loss=0.2807, lr=0.008675, batch_cost=0.5357, reader_cost=0.0008 | ETA 10:09:48 2020-11-03 10:02:32 [INFO] [TRAIN] epoch=32, iter=11800/80000, loss=0.2713, lr=0.008664, batch_cost=0.5351, reader_cost=0.0009 | ETA 10:08:14 2020-11-03 10:03:25 [INFO] [TRAIN] epoch=32, iter=11900/80000, loss=0.2368, lr=0.008652, batch_cost=0.5327, reader_cost=0.0007 | ETA 10:04:38 2020-11-03 10:04:19 [INFO] [TRAIN] epoch=33, iter=12000/80000, loss=0.2319, lr=0.008641, batch_cost=0.5418, reader_cost=0.0071 | ETA 10:14:01 2020-11-03 10:05:13 [INFO] [TRAIN] epoch=33, iter=12100/80000, loss=0.2661, lr=0.008629, batch_cost=0.5356, reader_cost=0.0008 | ETA 10:06:05 2020-11-03 10:06:07 [INFO] [TRAIN] epoch=33, iter=12200/80000, loss=0.2564, lr=0.008618, batch_cost=0.5377, reader_cost=0.0012 | ETA 10:07:37 2020-11-03 10:07:01 [INFO] [TRAIN] epoch=34, iter=12300/80000, loss=0.2477, lr=0.008606, batch_cost=0.5447, reader_cost=0.0088 | ETA 10:14:37 2020-11-03 10:07:55 [INFO] [TRAIN] epoch=34, iter=12400/80000, loss=0.2443, lr=0.008595, batch_cost=0.5391, reader_cost=0.0008 | ETA 10:07:22 2020-11-03 10:08:49 [INFO] [TRAIN] epoch=34, iter=12500/80000, loss=0.2695, lr=0.008584, batch_cost=0.5402, reader_cost=0.0012 | ETA 10:07:44 2020-11-03 10:09:43 [INFO] [TRAIN] epoch=34, iter=12600/80000, loss=0.2646, lr=0.008572, batch_cost=0.5350, reader_cost=0.0008 | ETA 10:00:58 2020-11-03 10:10:37 [INFO] [TRAIN] epoch=35, iter=12700/80000, loss=0.2416, lr=0.008561, batch_cost=0.5481, reader_cost=0.0082 | ETA 10:14:45 2020-11-03 10:11:31 [INFO] [TRAIN] epoch=35, iter=12800/80000, loss=0.2752, lr=0.008549, batch_cost=0.5353, reader_cost=0.0005 | ETA 09:59:32 2020-11-03 10:12:24 [INFO] [TRAIN] epoch=35, iter=12900/80000, loss=0.2455, lr=0.008538, batch_cost=0.5293, reader_cost=0.0003 | ETA 09:51:55 2020-11-03 10:13:17 [INFO] [TRAIN] epoch=35, iter=13000/80000, loss=0.2703, lr=0.008526, batch_cost=0.5294, reader_cost=0.0004 | ETA 09:51:11 2020-11-03 10:14:11 [INFO] [TRAIN] epoch=36, iter=13100/80000, loss=0.2672, lr=0.008515, batch_cost=0.5412, reader_cost=0.0069 | ETA 10:03:25 2020-11-03 10:15:04 [INFO] [TRAIN] epoch=36, iter=13200/80000, loss=0.2643, lr=0.008504, batch_cost=0.5327, reader_cost=0.0003 | ETA 09:53:01 2020-11-03 10:15:58 [INFO] [TRAIN] epoch=36, iter=13300/80000, loss=0.2871, lr=0.008492, batch_cost=0.5341, reader_cost=0.0007 | ETA 09:53:44 2020-11-03 10:16:52 [INFO] [TRAIN] epoch=37, iter=13400/80000, loss=0.2491, lr=0.008481, batch_cost=0.5404, reader_cost=0.0086 | ETA 09:59:47 2020-11-03 10:17:45 [INFO] [TRAIN] epoch=37, iter=13500/80000, loss=0.2170, lr=0.008469, batch_cost=0.5320, reader_cost=0.0007 | ETA 09:49:36 2020-11-03 10:18:38 [INFO] [TRAIN] epoch=37, iter=13600/80000, loss=0.2809, lr=0.008458, batch_cost=0.5343, reader_cost=0.0004 | ETA 09:51:19 2020-11-03 10:19:32 [INFO] [TRAIN] epoch=37, iter=13700/80000, loss=0.2702, lr=0.008446, batch_cost=0.5381, reader_cost=0.0008 | ETA 09:54:38 2020-11-03 10:20:27 [INFO] [TRAIN] epoch=38, iter=13800/80000, loss=0.2320, lr=0.008435, batch_cost=0.5472, reader_cost=0.0078 | ETA 10:03:44 2020-11-03 10:21:21 [INFO] [TRAIN] epoch=38, iter=13900/80000, loss=0.2554, lr=0.008423, batch_cost=0.5369, reader_cost=0.0008 | ETA 09:51:27 2020-11-03 10:22:14 [INFO] [TRAIN] epoch=38, iter=14000/80000, loss=0.2268, lr=0.008412, batch_cost=0.5379, reader_cost=0.0011 | ETA 09:51:44 2020-11-03 10:23:08 [INFO] [TRAIN] epoch=38, iter=14100/80000, loss=0.2829, lr=0.008400, batch_cost=0.5378, reader_cost=0.0010 | ETA 09:50:44 2020-11-03 10:24:03 [INFO] [TRAIN] epoch=39, iter=14200/80000, loss=0.2356, lr=0.008389, batch_cost=0.5462, reader_cost=0.0095 | ETA 09:58:57 2020-11-03 10:24:56 [INFO] [TRAIN] epoch=39, iter=14300/80000, loss=0.2313, lr=0.008378, batch_cost=0.5344, reader_cost=0.0006 | ETA 09:45:08 2020-11-03 10:25:50 [INFO] [TRAIN] epoch=39, iter=14400/80000, loss=0.2362, lr=0.008366, batch_cost=0.5357, reader_cost=0.0005 | ETA 09:45:39 2020-11-03 10:26:44 [INFO] [TRAIN] epoch=39, iter=14500/80000, loss=0.2244, lr=0.008355, batch_cost=0.5398, reader_cost=0.0010 | ETA 09:49:14 2020-11-03 10:27:38 [INFO] [TRAIN] epoch=40, iter=14600/80000, loss=0.2676, lr=0.008343, batch_cost=0.5453, reader_cost=0.0072 | ETA 09:54:22 2020-11-03 10:28:32 [INFO] [TRAIN] epoch=40, iter=14700/80000, loss=0.2311, lr=0.008332, batch_cost=0.5381, reader_cost=0.0004 | ETA 09:45:39 2020-11-03 10:29:25 [INFO] [TRAIN] epoch=40, iter=14800/80000, loss=0.2338, lr=0.008320, batch_cost=0.5324, reader_cost=0.0005 | ETA 09:38:35 2020-11-03 10:30:20 [INFO] [TRAIN] epoch=41, iter=14900/80000, loss=0.1992, lr=0.008309, batch_cost=0.5461, reader_cost=0.0084 | ETA 09:52:30 2020-11-03 10:31:13 [INFO] [TRAIN] epoch=41, iter=15000/80000, loss=0.2377, lr=0.008297, batch_cost=0.5345, reader_cost=0.0005 | ETA 09:39:02 2020-11-03 10:32:07 [INFO] [TRAIN] epoch=41, iter=15100/80000, loss=0.3024, lr=0.008286, batch_cost=0.5378, reader_cost=0.0005 | ETA 09:41:40 2020-11-03 10:33:01 [INFO] [TRAIN] epoch=41, iter=15200/80000, loss=0.2548, lr=0.008274, batch_cost=0.5352, reader_cost=0.0006 | ETA 09:38:00 2020-11-03 10:33:55 [INFO] [TRAIN] epoch=42, iter=15300/80000, loss=0.2032, lr=0.008263, batch_cost=0.5445, reader_cost=0.0073 | ETA 09:47:09 2020-11-03 10:34:48 [INFO] [TRAIN] epoch=42, iter=15400/80000, loss=0.2934, lr=0.008251, batch_cost=0.5330, reader_cost=0.0005 | ETA 09:33:50 2020-11-03 10:35:42 [INFO] [TRAIN] epoch=42, iter=15500/80000, loss=0.2480, lr=0.008240, batch_cost=0.5378, reader_cost=0.0004 | ETA 09:38:07 2020-11-03 10:36:36 [INFO] [TRAIN] epoch=42, iter=15600/80000, loss=0.2722, lr=0.008228, batch_cost=0.5388, reader_cost=0.0004 | ETA 09:38:17 2020-11-03 10:37:31 [INFO] [TRAIN] epoch=43, iter=15700/80000, loss=0.2099, lr=0.008217, batch_cost=0.5471, reader_cost=0.0080 | ETA 09:46:18 2020-11-03 10:38:25 [INFO] [TRAIN] epoch=43, iter=15800/80000, loss=0.2438, lr=0.008205, batch_cost=0.5377, reader_cost=0.0009 | ETA 09:35:18 2020-11-03 10:39:18 [INFO] [TRAIN] epoch=43, iter=15900/80000, loss=0.2580, lr=0.008194, batch_cost=0.5358, reader_cost=0.0009 | ETA 09:32:25 2020-11-03 10:40:12 [INFO] [TRAIN] epoch=44, iter=16000/80000, loss=0.2569, lr=0.008182, batch_cost=0.5431, reader_cost=0.0080 | ETA 09:39:18 2020-11-03 10:40:20 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-03 10:45:39 [INFO] [EVAL] #Images=500 mIoU=0.7316 Acc=0.9484 Kappa=0.9328 2020-11-03 10:45:39 [INFO] [EVAL] Category IoU: [0.9657 0.7755 0.8967 0.4005 0.4821 0.5924 0.67 0.7536 0.9167 0.5983 0.9336 0.7981 0.5622 0.9413 0.7576 0.8244 0.7321 0.5552 0.745 ] 2020-11-03 10:45:39 [INFO] [EVAL] Category Acc: [0.9781 0.92 0.9309 0.8289 0.8407 0.8149 0.8003 0.8992 0.9408 0.8963 0.9608 0.857 0.7167 0.9662 0.9266 0.9509 0.9025 0.7299 0.8712] 2020-11-03 10:45:43 [INFO] [EVAL] The model with the best validation mIoU (0.7316) was saved at iter 16000. 2020-11-03 10:46:36 [INFO] [TRAIN] epoch=44, iter=16100/80000, loss=0.2226, lr=0.008171, batch_cost=0.5333, reader_cost=0.0005 | ETA 09:27:54 2020-11-03 10:47:29 [INFO] [TRAIN] epoch=44, iter=16200/80000, loss=0.2684, lr=0.008159, batch_cost=0.5313, reader_cost=0.0003 | ETA 09:24:59 2020-11-03 10:48:23 [INFO] [TRAIN] epoch=44, iter=16300/80000, loss=0.2392, lr=0.008148, batch_cost=0.5344, reader_cost=0.0006 | ETA 09:27:19 2020-11-03 10:49:17 [INFO] [TRAIN] epoch=45, iter=16400/80000, loss=0.2168, lr=0.008136, batch_cost=0.5451, reader_cost=0.0084 | ETA 09:37:49 2020-11-03 10:50:11 [INFO] [TRAIN] epoch=45, iter=16500/80000, loss=0.2252, lr=0.008125, batch_cost=0.5346, reader_cost=0.0005 | ETA 09:25:46 2020-11-03 10:51:05 [INFO] [TRAIN] epoch=45, iter=16600/80000, loss=0.2153, lr=0.008113, batch_cost=0.5377, reader_cost=0.0008 | ETA 09:28:11 2020-11-03 10:51:58 [INFO] [TRAIN] epoch=45, iter=16700/80000, loss=0.2195, lr=0.008102, batch_cost=0.5377, reader_cost=0.0005 | ETA 09:27:19 2020-11-03 10:52:53 [INFO] [TRAIN] epoch=46, iter=16800/80000, loss=0.2497, lr=0.008090, batch_cost=0.5468, reader_cost=0.0081 | ETA 09:35:56 2020-11-03 10:53:47 [INFO] [TRAIN] epoch=46, iter=16900/80000, loss=0.2154, lr=0.008079, batch_cost=0.5370, reader_cost=0.0006 | ETA 09:24:46 2020-11-03 10:54:41 [INFO] [TRAIN] epoch=46, iter=17000/80000, loss=0.2542, lr=0.008067, batch_cost=0.5398, reader_cost=0.0012 | ETA 09:26:49 2020-11-03 10:55:35 [INFO] [TRAIN] epoch=46, iter=17100/80000, loss=0.2995, lr=0.008056, batch_cost=0.5376, reader_cost=0.0011 | ETA 09:23:32 2020-11-03 10:56:29 [INFO] [TRAIN] epoch=47, iter=17200/80000, loss=0.2363, lr=0.008044, batch_cost=0.5421, reader_cost=0.0091 | ETA 09:27:25 2020-11-03 10:57:22 [INFO] [TRAIN] epoch=47, iter=17300/80000, loss=0.2559, lr=0.008033, batch_cost=0.5339, reader_cost=0.0004 | ETA 09:17:56 2020-11-03 10:58:16 [INFO] [TRAIN] epoch=47, iter=17400/80000, loss=0.2419, lr=0.008021, batch_cost=0.5357, reader_cost=0.0004 | ETA 09:18:51 2020-11-03 10:59:10 [INFO] [TRAIN] epoch=48, iter=17500/80000, loss=0.2262, lr=0.008010, batch_cost=0.5445, reader_cost=0.0091 | ETA 09:27:12 2020-11-03 11:00:04 [INFO] [TRAIN] epoch=48, iter=17600/80000, loss=0.2665, lr=0.007998, batch_cost=0.5387, reader_cost=0.0011 | ETA 09:20:16 2020-11-03 11:00:58 [INFO] [TRAIN] epoch=48, iter=17700/80000, loss=0.2319, lr=0.007987, batch_cost=0.5410, reader_cost=0.0006 | ETA 09:21:43 2020-11-03 11:01:52 [INFO] [TRAIN] epoch=48, iter=17800/80000, loss=0.2467, lr=0.007975, batch_cost=0.5363, reader_cost=0.0011 | ETA 09:15:57 2020-11-03 11:02:46 [INFO] [TRAIN] epoch=49, iter=17900/80000, loss=0.2525, lr=0.007964, batch_cost=0.5462, reader_cost=0.0084 | ETA 09:25:17 2020-11-03 11:03:40 [INFO] [TRAIN] epoch=49, iter=18000/80000, loss=0.1951, lr=0.007952, batch_cost=0.5378, reader_cost=0.0007 | ETA 09:15:46 2020-11-03 11:04:34 [INFO] [TRAIN] epoch=49, iter=18100/80000, loss=0.2726, lr=0.007941, batch_cost=0.5394, reader_cost=0.0009 | ETA 09:16:30 2020-11-03 11:05:28 [INFO] [TRAIN] epoch=49, iter=18200/80000, loss=0.2720, lr=0.007929, batch_cost=0.5384, reader_cost=0.0010 | ETA 09:14:33 2020-11-03 11:06:22 [INFO] [TRAIN] epoch=50, iter=18300/80000, loss=0.2713, lr=0.007918, batch_cost=0.5427, reader_cost=0.0076 | ETA 09:18:06 2020-11-03 11:07:16 [INFO] [TRAIN] epoch=50, iter=18400/80000, loss=0.2442, lr=0.007906, batch_cost=0.5356, reader_cost=0.0010 | ETA 09:09:51 2020-11-03 11:08:10 [INFO] [TRAIN] epoch=50, iter=18500/80000, loss=0.3014, lr=0.007895, batch_cost=0.5387, reader_cost=0.0009 | ETA 09:12:11 2020-11-03 11:09:03 [INFO] [TRAIN] epoch=50, iter=18600/80000, loss=0.2292, lr=0.007883, batch_cost=0.5358, reader_cost=0.0009 | ETA 09:08:19 2020-11-03 11:09:58 [INFO] [TRAIN] epoch=51, iter=18700/80000, loss=0.2236, lr=0.007871, batch_cost=0.5464, reader_cost=0.0084 | ETA 09:18:12 2020-11-03 11:10:52 [INFO] [TRAIN] epoch=51, iter=18800/80000, loss=0.2350, lr=0.007860, batch_cost=0.5372, reader_cost=0.0006 | ETA 09:07:54 2020-11-03 11:11:46 [INFO] [TRAIN] epoch=51, iter=18900/80000, loss=0.2045, lr=0.007848, batch_cost=0.5392, reader_cost=0.0013 | ETA 09:09:03 2020-11-03 11:12:40 [INFO] [TRAIN] epoch=52, iter=19000/80000, loss=0.1942, lr=0.007837, batch_cost=0.5474, reader_cost=0.0086 | ETA 09:16:29 2020-11-03 11:13:34 [INFO] [TRAIN] epoch=52, iter=19100/80000, loss=0.2289, lr=0.007825, batch_cost=0.5382, reader_cost=0.0007 | ETA 09:06:13 2020-11-03 11:14:28 [INFO] [TRAIN] epoch=52, iter=19200/80000, loss=0.2238, lr=0.007814, batch_cost=0.5363, reader_cost=0.0005 | ETA 09:03:25 2020-11-03 11:15:21 [INFO] [TRAIN] epoch=52, iter=19300/80000, loss=0.2444, lr=0.007802, batch_cost=0.5369, reader_cost=0.0002 | ETA 09:03:10 2020-11-03 11:16:16 [INFO] [TRAIN] epoch=53, iter=19400/80000, loss=0.2356, lr=0.007791, batch_cost=0.5445, reader_cost=0.0071 | ETA 09:09:59 2020-11-03 11:17:10 [INFO] [TRAIN] epoch=53, iter=19500/80000, loss=0.2065, lr=0.007779, batch_cost=0.5407, reader_cost=0.0009 | ETA 09:05:10 2020-11-03 11:18:04 [INFO] [TRAIN] epoch=53, iter=19600/80000, loss=0.2313, lr=0.007768, batch_cost=0.5374, reader_cost=0.0007 | ETA 09:00:57 2020-11-03 11:18:58 [INFO] [TRAIN] epoch=53, iter=19700/80000, loss=0.2385, lr=0.007756, batch_cost=0.5384, reader_cost=0.0010 | ETA 09:01:06 2020-11-03 11:19:52 [INFO] [TRAIN] epoch=54, iter=19800/80000, loss=0.2289, lr=0.007744, batch_cost=0.5455, reader_cost=0.0081 | ETA 09:07:17 2020-11-03 11:20:46 [INFO] [TRAIN] epoch=54, iter=19900/80000, loss=0.2980, lr=0.007733, batch_cost=0.5393, reader_cost=0.0010 | ETA 09:00:14 2020-11-03 11:21:40 [INFO] [TRAIN] epoch=54, iter=20000/80000, loss=0.2394, lr=0.007721, batch_cost=0.5368, reader_cost=0.0008 | ETA 08:56:48 2020-11-03 11:22:34 [INFO] [TRAIN] epoch=55, iter=20100/80000, loss=0.2765, lr=0.007710, batch_cost=0.5446, reader_cost=0.0075 | ETA 09:03:42 2020-11-03 11:23:28 [INFO] [TRAIN] epoch=55, iter=20200/80000, loss=0.2281, lr=0.007698, batch_cost=0.5370, reader_cost=0.0008 | ETA 08:55:14 2020-11-03 11:24:21 [INFO] [TRAIN] epoch=55, iter=20300/80000, loss=0.2530, lr=0.007687, batch_cost=0.5351, reader_cost=0.0006 | ETA 08:52:27 2020-11-03 11:25:15 [INFO] [TRAIN] epoch=55, iter=20400/80000, loss=0.2101, lr=0.007675, batch_cost=0.5378, reader_cost=0.0012 | ETA 08:54:10 2020-11-03 11:26:10 [INFO] [TRAIN] epoch=56, iter=20500/80000, loss=0.2257, lr=0.007663, batch_cost=0.5462, reader_cost=0.0080 | ETA 09:01:39 2020-11-03 11:27:03 [INFO] [TRAIN] epoch=56, iter=20600/80000, loss=0.2384, lr=0.007652, batch_cost=0.5359, reader_cost=0.0005 | ETA 08:50:32 2020-11-03 11:27:57 [INFO] [TRAIN] epoch=56, iter=20700/80000, loss=0.2079, lr=0.007640, batch_cost=0.5364, reader_cost=0.0007 | ETA 08:50:07 2020-11-03 11:28:51 [INFO] [TRAIN] epoch=56, iter=20800/80000, loss=0.2450, lr=0.007629, batch_cost=0.5377, reader_cost=0.0007 | ETA 08:50:31 2020-11-03 11:29:46 [INFO] [TRAIN] epoch=57, iter=20900/80000, loss=0.2400, lr=0.007617, batch_cost=0.5478, reader_cost=0.0081 | ETA 08:59:35 2020-11-03 11:30:39 [INFO] [TRAIN] epoch=57, iter=21000/80000, loss=0.2336, lr=0.007606, batch_cost=0.5360, reader_cost=0.0007 | ETA 08:47:04 2020-11-03 11:31:33 [INFO] [TRAIN] epoch=57, iter=21100/80000, loss=0.2389, lr=0.007594, batch_cost=0.5363, reader_cost=0.0004 | ETA 08:46:26 2020-11-03 11:32:27 [INFO] [TRAIN] epoch=57, iter=21200/80000, loss=0.2031, lr=0.007582, batch_cost=0.5371, reader_cost=0.0011 | ETA 08:46:19 2020-11-03 11:33:21 [INFO] [TRAIN] epoch=58, iter=21300/80000, loss=0.2058, lr=0.007571, batch_cost=0.5471, reader_cost=0.0075 | ETA 08:55:12 2020-11-03 11:34:15 [INFO] [TRAIN] epoch=58, iter=21400/80000, loss=0.2019, lr=0.007559, batch_cost=0.5371, reader_cost=0.0007 | ETA 08:44:31 2020-11-03 11:35:09 [INFO] [TRAIN] epoch=58, iter=21500/80000, loss=0.2519, lr=0.007548, batch_cost=0.5387, reader_cost=0.0009 | ETA 08:45:16 2020-11-03 11:36:03 [INFO] [TRAIN] epoch=59, iter=21600/80000, loss=0.2035, lr=0.007536, batch_cost=0.5428, reader_cost=0.0082 | ETA 08:48:18 2020-11-03 11:36:56 [INFO] [TRAIN] epoch=59, iter=21700/80000, loss=0.2406, lr=0.007524, batch_cost=0.5339, reader_cost=0.0006 | ETA 08:38:47 2020-11-03 11:37:50 [INFO] [TRAIN] epoch=59, iter=21800/80000, loss=0.2014, lr=0.007513, batch_cost=0.5341, reader_cost=0.0007 | ETA 08:38:04 2020-11-03 11:38:44 [INFO] [TRAIN] epoch=59, iter=21900/80000, loss=0.2337, lr=0.007501, batch_cost=0.5362, reader_cost=0.0008 | ETA 08:39:11 2020-11-03 11:39:38 [INFO] [TRAIN] epoch=60, iter=22000/80000, loss=0.2385, lr=0.007490, batch_cost=0.5449, reader_cost=0.0075 | ETA 08:46:44 2020-11-03 11:40:32 [INFO] [TRAIN] epoch=60, iter=22100/80000, loss=0.2307, lr=0.007478, batch_cost=0.5356, reader_cost=0.0012 | ETA 08:36:52 2020-11-03 11:41:25 [INFO] [TRAIN] epoch=60, iter=22200/80000, loss=0.2259, lr=0.007466, batch_cost=0.5378, reader_cost=0.0006 | ETA 08:38:06 2020-11-03 11:42:19 [INFO] [TRAIN] epoch=60, iter=22300/80000, loss=0.2297, lr=0.007455, batch_cost=0.5373, reader_cost=0.0004 | ETA 08:36:41 2020-11-03 11:43:14 [INFO] [TRAIN] epoch=61, iter=22400/80000, loss=0.2229, lr=0.007443, batch_cost=0.5453, reader_cost=0.0080 | ETA 08:43:29 2020-11-03 11:44:07 [INFO] [TRAIN] epoch=61, iter=22500/80000, loss=0.2297, lr=0.007432, batch_cost=0.5366, reader_cost=0.0008 | ETA 08:34:12 2020-11-03 11:45:00 [INFO] [TRAIN] epoch=61, iter=22600/80000, loss=0.2248, lr=0.007420, batch_cost=0.5314, reader_cost=0.0004 | ETA 08:28:20 2020-11-03 11:45:55 [INFO] [TRAIN] epoch=62, iter=22700/80000, loss=0.2481, lr=0.007408, batch_cost=0.5441, reader_cost=0.0081 | ETA 08:39:34 2020-11-03 11:46:48 [INFO] [TRAIN] epoch=62, iter=22800/80000, loss=0.2368, lr=0.007397, batch_cost=0.5362, reader_cost=0.0009 | ETA 08:31:08 2020-11-03 11:47:42 [INFO] [TRAIN] epoch=62, iter=22900/80000, loss=0.2865, lr=0.007385, batch_cost=0.5355, reader_cost=0.0006 | ETA 08:29:39 2020-11-03 11:48:36 [INFO] [TRAIN] epoch=62, iter=23000/80000, loss=0.2323, lr=0.007373, batch_cost=0.5376, reader_cost=0.0004 | ETA 08:30:43 2020-11-03 11:49:30 [INFO] [TRAIN] epoch=63, iter=23100/80000, loss=0.1955, lr=0.007362, batch_cost=0.5462, reader_cost=0.0071 | ETA 08:37:59 2020-11-03 11:50:24 [INFO] [TRAIN] epoch=63, iter=23200/80000, loss=0.2121, lr=0.007350, batch_cost=0.5377, reader_cost=0.0008 | ETA 08:29:00 2020-11-03 11:51:18 [INFO] [TRAIN] epoch=63, iter=23300/80000, loss=0.2207, lr=0.007339, batch_cost=0.5382, reader_cost=0.0008 | ETA 08:28:35 2020-11-03 11:52:12 [INFO] [TRAIN] epoch=63, iter=23400/80000, loss=0.2207, lr=0.007327, batch_cost=0.5369, reader_cost=0.0012 | ETA 08:26:27 2020-11-03 11:53:06 [INFO] [TRAIN] epoch=64, iter=23500/80000, loss=0.2159, lr=0.007315, batch_cost=0.5479, reader_cost=0.0086 | ETA 08:35:55 2020-11-03 11:54:00 [INFO] [TRAIN] epoch=64, iter=23600/80000, loss=0.2460, lr=0.007304, batch_cost=0.5355, reader_cost=0.0004 | ETA 08:23:19 2020-11-03 11:54:53 [INFO] [TRAIN] epoch=64, iter=23700/80000, loss=0.2221, lr=0.007292, batch_cost=0.5336, reader_cost=0.0003 | ETA 08:20:41 2020-11-03 11:55:47 [INFO] [TRAIN] epoch=64, iter=23800/80000, loss=0.2124, lr=0.007280, batch_cost=0.5373, reader_cost=0.0007 | ETA 08:23:14 2020-11-03 11:56:41 [INFO] [TRAIN] epoch=65, iter=23900/80000, loss=0.2545, lr=0.007269, batch_cost=0.5441, reader_cost=0.0090 | ETA 08:28:42 2020-11-03 11:57:35 [INFO] [TRAIN] epoch=65, iter=24000/80000, loss=0.2133, lr=0.007257, batch_cost=0.5343, reader_cost=0.0005 | ETA 08:18:40 2020-11-03 11:57:41 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-03 12:02:59 [INFO] [EVAL] #Images=500 mIoU=0.7306 Acc=0.9499 Kappa=0.9348 2020-11-03 12:02:59 [INFO] [EVAL] Category IoU: [0.9675 0.781 0.9078 0.3173 0.5618 0.6076 0.6793 0.7621 0.9143 0.6149 0.9323 0.8001 0.5646 0.9377 0.6987 0.7923 0.7498 0.5309 0.7615] 2020-11-03 12:02:59 [INFO] [EVAL] Category Acc: [0.9803 0.8985 0.9348 0.8373 0.7946 0.8171 0.8469 0.8876 0.9515 0.7858 0.9645 0.8645 0.7769 0.9617 0.9499 0.9464 0.9404 0.8884 0.874 ] 2020-11-03 12:02:59 [INFO] [EVAL] The model with the best validation mIoU (0.7316) was saved at iter 16000. 2020-11-03 12:03:52 [INFO] [TRAIN] epoch=65, iter=24100/80000, loss=0.2204, lr=0.007245, batch_cost=0.5272, reader_cost=0.0005 | ETA 08:11:10 2020-11-03 12:04:46 [INFO] [TRAIN] epoch=66, iter=24200/80000, loss=0.2175, lr=0.007234, batch_cost=0.5367, reader_cost=0.0078 | ETA 08:19:07 2020-11-03 12:05:39 [INFO] [TRAIN] epoch=66, iter=24300/80000, loss=0.2073, lr=0.007222, batch_cost=0.5328, reader_cost=0.0006 | ETA 08:14:39 2020-11-03 12:06:32 [INFO] [TRAIN] epoch=66, iter=24400/80000, loss=0.2422, lr=0.007210, batch_cost=0.5331, reader_cost=0.0005 | ETA 08:14:01 2020-11-03 12:07:26 [INFO] [TRAIN] epoch=66, iter=24500/80000, loss=0.2196, lr=0.007199, batch_cost=0.5311, reader_cost=0.0003 | ETA 08:11:16 2020-11-03 12:08:19 [INFO] [TRAIN] epoch=67, iter=24600/80000, loss=0.2070, lr=0.007187, batch_cost=0.5387, reader_cost=0.0087 | ETA 08:17:26 2020-11-03 12:09:13 [INFO] [TRAIN] epoch=67, iter=24700/80000, loss=0.1884, lr=0.007175, batch_cost=0.5313, reader_cost=0.0004 | ETA 08:09:40 2020-11-03 12:10:06 [INFO] [TRAIN] epoch=67, iter=24800/80000, loss=0.2518, lr=0.007164, batch_cost=0.5320, reader_cost=0.0009 | ETA 08:09:25 2020-11-03 12:10:59 [INFO] [TRAIN] epoch=67, iter=24900/80000, loss=0.2584, lr=0.007152, batch_cost=0.5338, reader_cost=0.0006 | ETA 08:10:11 2020-11-03 12:11:53 [INFO] [TRAIN] epoch=68, iter=25000/80000, loss=0.2469, lr=0.007140, batch_cost=0.5405, reader_cost=0.0083 | ETA 08:15:28 2020-11-03 12:12:46 [INFO] [TRAIN] epoch=68, iter=25100/80000, loss=0.2053, lr=0.007129, batch_cost=0.5311, reader_cost=0.0004 | ETA 08:05:57 2020-11-03 12:13:40 [INFO] [TRAIN] epoch=68, iter=25200/80000, loss=0.2208, lr=0.007117, batch_cost=0.5333, reader_cost=0.0006 | ETA 08:07:06 2020-11-03 12:14:34 [INFO] [TRAIN] epoch=69, iter=25300/80000, loss=0.2364, lr=0.007105, batch_cost=0.5405, reader_cost=0.0082 | ETA 08:12:43 2020-11-03 12:15:27 [INFO] [TRAIN] epoch=69, iter=25400/80000, loss=0.2196, lr=0.007094, batch_cost=0.5307, reader_cost=0.0011 | ETA 08:02:58 2020-11-03 12:16:20 [INFO] [TRAIN] epoch=69, iter=25500/80000, loss=0.2510, lr=0.007082, batch_cost=0.5296, reader_cost=0.0006 | ETA 08:01:03 2020-11-03 12:17:13 [INFO] [TRAIN] epoch=69, iter=25600/80000, loss=0.2054, lr=0.007070, batch_cost=0.5283, reader_cost=0.0004 | ETA 07:59:00 2020-11-03 12:18:07 [INFO] [TRAIN] epoch=70, iter=25700/80000, loss=0.2222, lr=0.007059, batch_cost=0.5394, reader_cost=0.0081 | ETA 08:08:09 2020-11-03 12:19:00 [INFO] [TRAIN] epoch=70, iter=25800/80000, loss=0.2144, lr=0.007047, batch_cost=0.5320, reader_cost=0.0005 | ETA 08:00:32 2020-11-03 12:19:53 [INFO] [TRAIN] epoch=70, iter=25900/80000, loss=0.2049, lr=0.007035, batch_cost=0.5324, reader_cost=0.0009 | ETA 08:00:00 2020-11-03 12:20:46 [INFO] [TRAIN] epoch=70, iter=26000/80000, loss=0.2039, lr=0.007024, batch_cost=0.5292, reader_cost=0.0005 | ETA 07:56:19 2020-11-03 12:21:40 [INFO] [TRAIN] epoch=71, iter=26100/80000, loss=0.2246, lr=0.007012, batch_cost=0.5390, reader_cost=0.0079 | ETA 08:04:13 2020-11-03 12:22:33 [INFO] [TRAIN] epoch=71, iter=26200/80000, loss=0.1904, lr=0.007000, batch_cost=0.5319, reader_cost=0.0003 | ETA 07:56:53 2020-11-03 12:23:26 [INFO] [TRAIN] epoch=71, iter=26300/80000, loss=0.2208, lr=0.006989, batch_cost=0.5306, reader_cost=0.0008 | ETA 07:54:53 2020-11-03 12:24:19 [INFO] [TRAIN] epoch=71, iter=26400/80000, loss=0.2225, lr=0.006977, batch_cost=0.5309, reader_cost=0.0003 | ETA 07:54:14 2020-11-03 12:25:13 [INFO] [TRAIN] epoch=72, iter=26500/80000, loss=0.2043, lr=0.006965, batch_cost=0.5363, reader_cost=0.0078 | ETA 07:58:10 2020-11-03 12:26:06 [INFO] [TRAIN] epoch=72, iter=26600/80000, loss=0.2111, lr=0.006954, batch_cost=0.5282, reader_cost=0.0004 | ETA 07:50:04 2020-11-03 12:26:59 [INFO] [TRAIN] epoch=72, iter=26700/80000, loss=0.1934, lr=0.006942, batch_cost=0.5312, reader_cost=0.0007 | ETA 07:51:51 2020-11-03 12:27:53 [INFO] [TRAIN] epoch=73, iter=26800/80000, loss=0.2054, lr=0.006930, batch_cost=0.5394, reader_cost=0.0064 | ETA 07:58:17 2020-11-03 12:28:46 [INFO] [TRAIN] epoch=73, iter=26900/80000, loss=0.2259, lr=0.006918, batch_cost=0.5296, reader_cost=0.0005 | ETA 07:48:44 2020-11-03 12:29:39 [INFO] [TRAIN] epoch=73, iter=27000/80000, loss=0.2410, lr=0.006907, batch_cost=0.5322, reader_cost=0.0008 | ETA 07:50:08 2020-11-03 12:30:32 [INFO] [TRAIN] epoch=73, iter=27100/80000, loss=0.2073, lr=0.006895, batch_cost=0.5294, reader_cost=0.0004 | ETA 07:46:43 2020-11-03 12:31:26 [INFO] [TRAIN] epoch=74, iter=27200/80000, loss=0.2251, lr=0.006883, batch_cost=0.5391, reader_cost=0.0082 | ETA 07:54:23 2020-11-03 12:32:19 [INFO] [TRAIN] epoch=74, iter=27300/80000, loss=0.2031, lr=0.006872, batch_cost=0.5330, reader_cost=0.0007 | ETA 07:48:10 2020-11-03 12:33:12 [INFO] [TRAIN] epoch=74, iter=27400/80000, loss=0.1975, lr=0.006860, batch_cost=0.5317, reader_cost=0.0005 | ETA 07:46:08 2020-11-03 12:34:05 [INFO] [TRAIN] epoch=74, iter=27500/80000, loss=0.2632, lr=0.006848, batch_cost=0.5328, reader_cost=0.0004 | ETA 07:46:10 2020-11-03 12:35:00 [INFO] [TRAIN] epoch=75, iter=27600/80000, loss=0.1936, lr=0.006836, batch_cost=0.5435, reader_cost=0.0086 | ETA 07:54:41 2020-11-03 12:35:53 [INFO] [TRAIN] epoch=75, iter=27700/80000, loss=0.2214, lr=0.006825, batch_cost=0.5325, reader_cost=0.0006 | ETA 07:44:12 2020-11-03 12:36:46 [INFO] [TRAIN] epoch=75, iter=27800/80000, loss=0.2254, lr=0.006813, batch_cost=0.5323, reader_cost=0.0007 | ETA 07:43:07 2020-11-03 12:37:39 [INFO] [TRAIN] epoch=75, iter=27900/80000, loss=0.2013, lr=0.006801, batch_cost=0.5318, reader_cost=0.0007 | ETA 07:41:46 2020-11-03 12:38:34 [INFO] [TRAIN] epoch=76, iter=28000/80000, loss=0.2411, lr=0.006789, batch_cost=0.5405, reader_cost=0.0089 | ETA 07:48:26 2020-11-03 12:39:27 [INFO] [TRAIN] epoch=76, iter=28100/80000, loss=0.2154, lr=0.006778, batch_cost=0.5313, reader_cost=0.0006 | ETA 07:39:34 2020-11-03 12:40:20 [INFO] [TRAIN] epoch=76, iter=28200/80000, loss=0.2067, lr=0.006766, batch_cost=0.5319, reader_cost=0.0008 | ETA 07:39:10 2020-11-03 12:41:14 [INFO] [TRAIN] epoch=77, iter=28300/80000, loss=0.1911, lr=0.006754, batch_cost=0.5406, reader_cost=0.0084 | ETA 07:45:51 2020-11-03 12:42:07 [INFO] [TRAIN] epoch=77, iter=28400/80000, loss=0.1962, lr=0.006743, batch_cost=0.5317, reader_cost=0.0008 | ETA 07:37:14 2020-11-03 12:43:00 [INFO] [TRAIN] epoch=77, iter=28500/80000, loss=0.2176, lr=0.006731, batch_cost=0.5304, reader_cost=0.0003 | ETA 07:35:18 2020-11-03 12:43:53 [INFO] [TRAIN] epoch=77, iter=28600/80000, loss=0.2338, lr=0.006719, batch_cost=0.5298, reader_cost=0.0003 | ETA 07:33:53 2020-11-03 12:44:47 [INFO] [TRAIN] epoch=78, iter=28700/80000, loss=0.1904, lr=0.006707, batch_cost=0.5395, reader_cost=0.0083 | ETA 07:41:15 2020-11-03 12:45:40 [INFO] [TRAIN] epoch=78, iter=28800/80000, loss=0.1944, lr=0.006696, batch_cost=0.5280, reader_cost=0.0004 | ETA 07:30:31 2020-11-03 12:46:33 [INFO] [TRAIN] epoch=78, iter=28900/80000, loss=0.2022, lr=0.006684, batch_cost=0.5301, reader_cost=0.0005 | ETA 07:31:28 2020-11-03 12:47:26 [INFO] [TRAIN] epoch=78, iter=29000/80000, loss=0.2206, lr=0.006672, batch_cost=0.5311, reader_cost=0.0006 | ETA 07:31:26 2020-11-03 12:48:20 [INFO] [TRAIN] epoch=79, iter=29100/80000, loss=0.1893, lr=0.006660, batch_cost=0.5427, reader_cost=0.0085 | ETA 07:40:22 2020-11-03 12:49:13 [INFO] [TRAIN] epoch=79, iter=29200/80000, loss=0.1890, lr=0.006648, batch_cost=0.5310, reader_cost=0.0006 | ETA 07:29:32 2020-11-03 12:50:07 [INFO] [TRAIN] epoch=79, iter=29300/80000, loss=0.2004, lr=0.006637, batch_cost=0.5333, reader_cost=0.0007 | ETA 07:30:40 2020-11-03 12:51:01 [INFO] [TRAIN] epoch=80, iter=29400/80000, loss=0.1971, lr=0.006625, batch_cost=0.5404, reader_cost=0.0091 | ETA 07:35:43 2020-11-03 12:51:54 [INFO] [TRAIN] epoch=80, iter=29500/80000, loss=0.1978, lr=0.006613, batch_cost=0.5322, reader_cost=0.0007 | ETA 07:27:57 2020-11-03 12:52:47 [INFO] [TRAIN] epoch=80, iter=29600/80000, loss=0.2060, lr=0.006601, batch_cost=0.5313, reader_cost=0.0004 | ETA 07:26:16 2020-11-03 12:53:40 [INFO] [TRAIN] epoch=80, iter=29700/80000, loss=0.2083, lr=0.006590, batch_cost=0.5320, reader_cost=0.0003 | ETA 07:26:01 2020-11-03 12:54:34 [INFO] [TRAIN] epoch=81, iter=29800/80000, loss=0.1881, lr=0.006578, batch_cost=0.5370, reader_cost=0.0078 | ETA 07:29:16 2020-11-03 12:55:27 [INFO] [TRAIN] epoch=81, iter=29900/80000, loss=0.2705, lr=0.006566, batch_cost=0.5308, reader_cost=0.0004 | ETA 07:23:12 2020-11-03 12:56:20 [INFO] [TRAIN] epoch=81, iter=30000/80000, loss=0.2328, lr=0.006554, batch_cost=0.5314, reader_cost=0.0003 | ETA 07:22:51 2020-11-03 12:57:13 [INFO] [TRAIN] epoch=81, iter=30100/80000, loss=0.2343, lr=0.006543, batch_cost=0.5291, reader_cost=0.0003 | ETA 07:20:04 2020-11-03 12:58:07 [INFO] [TRAIN] epoch=82, iter=30200/80000, loss=0.2400, lr=0.006531, batch_cost=0.5384, reader_cost=0.0094 | ETA 07:26:54 2020-11-03 12:59:00 [INFO] [TRAIN] epoch=82, iter=30300/80000, loss=0.1972, lr=0.006519, batch_cost=0.5310, reader_cost=0.0005 | ETA 07:19:49 2020-11-03 12:59:53 [INFO] [TRAIN] epoch=82, iter=30400/80000, loss=0.2144, lr=0.006507, batch_cost=0.5308, reader_cost=0.0006 | ETA 07:18:46 2020-11-03 13:00:46 [INFO] [TRAIN] epoch=82, iter=30500/80000, loss=0.1801, lr=0.006495, batch_cost=0.5280, reader_cost=0.0004 | ETA 07:15:37 2020-11-03 13:01:40 [INFO] [TRAIN] epoch=83, iter=30600/80000, loss=0.2108, lr=0.006484, batch_cost=0.5367, reader_cost=0.0079 | ETA 07:21:50 2020-11-03 13:02:32 [INFO] [TRAIN] epoch=83, iter=30700/80000, loss=0.2427, lr=0.006472, batch_cost=0.5276, reader_cost=0.0004 | ETA 07:13:30 2020-11-03 13:03:25 [INFO] [TRAIN] epoch=83, iter=30800/80000, loss=0.2064, lr=0.006460, batch_cost=0.5304, reader_cost=0.0004 | ETA 07:14:53 2020-11-03 13:04:19 [INFO] [TRAIN] epoch=84, iter=30900/80000, loss=0.1949, lr=0.006448, batch_cost=0.5384, reader_cost=0.0075 | ETA 07:20:37 2020-11-03 13:05:12 [INFO] [TRAIN] epoch=84, iter=31000/80000, loss=0.1804, lr=0.006436, batch_cost=0.5306, reader_cost=0.0005 | ETA 07:13:19 2020-11-03 13:06:05 [INFO] [TRAIN] epoch=84, iter=31100/80000, loss=0.1873, lr=0.006425, batch_cost=0.5292, reader_cost=0.0002 | ETA 07:11:18 2020-11-03 13:06:58 [INFO] [TRAIN] epoch=84, iter=31200/80000, loss=0.1945, lr=0.006413, batch_cost=0.5297, reader_cost=0.0006 | ETA 07:10:48 2020-11-03 13:07:52 [INFO] [TRAIN] epoch=85, iter=31300/80000, loss=0.1928, lr=0.006401, batch_cost=0.5395, reader_cost=0.0091 | ETA 07:17:54 2020-11-03 13:08:45 [INFO] [TRAIN] epoch=85, iter=31400/80000, loss=0.2020, lr=0.006389, batch_cost=0.5332, reader_cost=0.0005 | ETA 07:11:54 2020-11-03 13:09:39 [INFO] [TRAIN] epoch=85, iter=31500/80000, loss=0.2226, lr=0.006377, batch_cost=0.5302, reader_cost=0.0004 | ETA 07:08:33 2020-11-03 13:10:31 [INFO] [TRAIN] epoch=85, iter=31600/80000, loss=0.2201, lr=0.006366, batch_cost=0.5283, reader_cost=0.0004 | ETA 07:06:11 2020-11-03 13:11:25 [INFO] [TRAIN] epoch=86, iter=31700/80000, loss=0.2063, lr=0.006354, batch_cost=0.5398, reader_cost=0.0086 | ETA 07:14:34 2020-11-03 13:12:19 [INFO] [TRAIN] epoch=86, iter=31800/80000, loss=0.1911, lr=0.006342, batch_cost=0.5333, reader_cost=0.0006 | ETA 07:08:23 2020-11-03 13:13:12 [INFO] [TRAIN] epoch=86, iter=31900/80000, loss=0.1845, lr=0.006330, batch_cost=0.5307, reader_cost=0.0009 | ETA 07:05:26 2020-11-03 13:14:06 [INFO] [TRAIN] epoch=87, iter=32000/80000, loss=0.2239, lr=0.006318, batch_cost=0.5399, reader_cost=0.0087 | ETA 07:11:55 2020-11-03 13:14:13 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-03 13:19:21 [INFO] [EVAL] #Images=500 mIoU=0.7337 Acc=0.9548 Kappa=0.9413 2020-11-03 13:19:21 [INFO] [EVAL] Category IoU: [0.9727 0.8137 0.9125 0.4372 0.5742 0.6312 0.7071 0.7837 0.9239 0.6397 0.9405 0.8183 0.6223 0.9456 0.7573 0.7477 0.2678 0.6677 0.7768] 2020-11-03 13:19:21 [INFO] [EVAL] Category Acc: [0.9863 0.9105 0.9407 0.8581 0.8229 0.7911 0.8215 0.9009 0.9547 0.8863 0.9604 0.8966 0.7659 0.9738 0.8381 0.7764 0.9661 0.8299 0.866 ] 2020-11-03 13:19:25 [INFO] [EVAL] The model with the best validation mIoU (0.7337) was saved at iter 32000. 2020-11-03 13:20:18 [INFO] [TRAIN] epoch=87, iter=32100/80000, loss=0.1907, lr=0.006306, batch_cost=0.5324, reader_cost=0.0005 | ETA 07:05:01 2020-11-03 13:21:11 [INFO] [TRAIN] epoch=87, iter=32200/80000, loss=0.2236, lr=0.006295, batch_cost=0.5309, reader_cost=0.0006 | ETA 07:02:58 2020-11-03 13:22:04 [INFO] [TRAIN] epoch=87, iter=32300/80000, loss=0.2074, lr=0.006283, batch_cost=0.5308, reader_cost=0.0004 | ETA 07:02:00 2020-11-03 13:22:58 [INFO] [TRAIN] epoch=88, iter=32400/80000, loss=0.2006, lr=0.006271, batch_cost=0.5364, reader_cost=0.0077 | ETA 07:05:30 2020-11-03 13:23:51 [INFO] [TRAIN] epoch=88, iter=32500/80000, loss=0.1808, lr=0.006259, batch_cost=0.5281, reader_cost=0.0004 | ETA 06:58:06 2020-11-03 13:24:44 [INFO] [TRAIN] epoch=88, iter=32600/80000, loss=0.1771, lr=0.006247, batch_cost=0.5313, reader_cost=0.0004 | ETA 06:59:45 2020-11-03 13:25:37 [INFO] [TRAIN] epoch=88, iter=32700/80000, loss=0.2142, lr=0.006235, batch_cost=0.5292, reader_cost=0.0004 | ETA 06:57:10 2020-11-03 13:26:31 [INFO] [TRAIN] epoch=89, iter=32800/80000, loss=0.1758, lr=0.006224, batch_cost=0.5380, reader_cost=0.0081 | ETA 07:03:12 2020-11-03 13:27:24 [INFO] [TRAIN] epoch=89, iter=32900/80000, loss=0.2079, lr=0.006212, batch_cost=0.5296, reader_cost=0.0008 | ETA 06:55:41 2020-11-03 13:28:17 [INFO] [TRAIN] epoch=89, iter=33000/80000, loss=0.2101, lr=0.006200, batch_cost=0.5316, reader_cost=0.0006 | ETA 06:56:24 2020-11-03 13:29:10 [INFO] [TRAIN] epoch=89, iter=33100/80000, loss=0.2048, lr=0.006188, batch_cost=0.5313, reader_cost=0.0009 | ETA 06:55:19 2020-11-03 13:30:04 [INFO] [TRAIN] epoch=90, iter=33200/80000, loss=0.2096, lr=0.006176, batch_cost=0.5414, reader_cost=0.0089 | ETA 07:02:16 2020-11-03 13:30:57 [INFO] [TRAIN] epoch=90, iter=33300/80000, loss=0.1998, lr=0.006164, batch_cost=0.5321, reader_cost=0.0005 | ETA 06:54:09 2020-11-03 13:31:50 [INFO] [TRAIN] epoch=90, iter=33400/80000, loss=0.1984, lr=0.006152, batch_cost=0.5311, reader_cost=0.0004 | ETA 06:52:28 2020-11-03 13:32:44 [INFO] [TRAIN] epoch=91, iter=33500/80000, loss=0.1894, lr=0.006141, batch_cost=0.5397, reader_cost=0.0100 | ETA 06:58:14 2020-11-03 13:33:38 [INFO] [TRAIN] epoch=91, iter=33600/80000, loss=0.1951, lr=0.006129, batch_cost=0.5339, reader_cost=0.0005 | ETA 06:52:53 2020-11-03 13:34:31 [INFO] [TRAIN] epoch=91, iter=33700/80000, loss=0.1992, lr=0.006117, batch_cost=0.5282, reader_cost=0.0004 | ETA 06:47:36 2020-11-03 13:35:24 [INFO] [TRAIN] epoch=91, iter=33800/80000, loss=0.2091, lr=0.006105, batch_cost=0.5312, reader_cost=0.0005 | ETA 06:49:01 2020-11-03 13:36:18 [INFO] [TRAIN] epoch=92, iter=33900/80000, loss=0.1673, lr=0.006093, batch_cost=0.5395, reader_cost=0.0079 | ETA 06:54:30 2020-11-03 13:37:11 [INFO] [TRAIN] epoch=92, iter=34000/80000, loss=0.1874, lr=0.006081, batch_cost=0.5324, reader_cost=0.0003 | ETA 06:48:10 2020-11-03 13:38:04 [INFO] [TRAIN] epoch=92, iter=34100/80000, loss=0.1853, lr=0.006069, batch_cost=0.5320, reader_cost=0.0004 | ETA 06:46:59 2020-11-03 13:38:57 [INFO] [TRAIN] epoch=92, iter=34200/80000, loss=0.1858, lr=0.006057, batch_cost=0.5309, reader_cost=0.0004 | ETA 06:45:14 2020-11-03 13:39:51 [INFO] [TRAIN] epoch=93, iter=34300/80000, loss=0.1964, lr=0.006046, batch_cost=0.5393, reader_cost=0.0093 | ETA 06:50:44 2020-11-03 13:40:44 [INFO] [TRAIN] epoch=93, iter=34400/80000, loss=0.1655, lr=0.006034, batch_cost=0.5307, reader_cost=0.0004 | ETA 06:43:21 2020-11-03 13:41:37 [INFO] [TRAIN] epoch=93, iter=34500/80000, loss=0.1954, lr=0.006022, batch_cost=0.5316, reader_cost=0.0004 | ETA 06:43:09 2020-11-03 13:42:32 [INFO] [TRAIN] epoch=94, iter=34600/80000, loss=0.2012, lr=0.006010, batch_cost=0.5416, reader_cost=0.0075 | ETA 06:49:48 2020-11-03 13:43:25 [INFO] [TRAIN] epoch=94, iter=34700/80000, loss=0.2031, lr=0.005998, batch_cost=0.5300, reader_cost=0.0004 | ETA 06:40:06 2020-11-03 13:44:18 [INFO] [TRAIN] epoch=94, iter=34800/80000, loss=0.1898, lr=0.005986, batch_cost=0.5315, reader_cost=0.0004 | ETA 06:40:25 2020-11-03 13:45:11 [INFO] [TRAIN] epoch=94, iter=34900/80000, loss=0.1883, lr=0.005974, batch_cost=0.5304, reader_cost=0.0004 | ETA 06:38:40 2020-11-03 13:46:05 [INFO] [TRAIN] epoch=95, iter=35000/80000, loss=0.1730, lr=0.005962, batch_cost=0.5393, reader_cost=0.0084 | ETA 06:44:29 2020-11-03 13:46:58 [INFO] [TRAIN] epoch=95, iter=35100/80000, loss=0.1796, lr=0.005950, batch_cost=0.5320, reader_cost=0.0004 | ETA 06:38:05 2020-11-03 13:47:51 [INFO] [TRAIN] epoch=95, iter=35200/80000, loss=0.1753, lr=0.005938, batch_cost=0.5287, reader_cost=0.0003 | ETA 06:34:44 2020-11-03 13:48:44 [INFO] [TRAIN] epoch=95, iter=35300/80000, loss=0.1906, lr=0.005927, batch_cost=0.5307, reader_cost=0.0004 | ETA 06:35:23 2020-11-03 13:49:38 [INFO] [TRAIN] epoch=96, iter=35400/80000, loss=0.1675, lr=0.005915, batch_cost=0.5387, reader_cost=0.0075 | ETA 06:40:26 2020-11-03 13:50:31 [INFO] [TRAIN] epoch=96, iter=35500/80000, loss=0.1754, lr=0.005903, batch_cost=0.5318, reader_cost=0.0004 | ETA 06:34:23 2020-11-03 13:51:24 [INFO] [TRAIN] epoch=96, iter=35600/80000, loss=0.2055, lr=0.005891, batch_cost=0.5305, reader_cost=0.0006 | ETA 06:32:32 2020-11-03 13:52:17 [INFO] [TRAIN] epoch=96, iter=35700/80000, loss=0.2026, lr=0.005879, batch_cost=0.5312, reader_cost=0.0003 | ETA 06:32:10 2020-11-03 13:53:11 [INFO] [TRAIN] epoch=97, iter=35800/80000, loss=0.2037, lr=0.005867, batch_cost=0.5419, reader_cost=0.0094 | ETA 06:39:12 2020-11-03 13:54:04 [INFO] [TRAIN] epoch=97, iter=35900/80000, loss=0.1899, lr=0.005855, batch_cost=0.5296, reader_cost=0.0004 | ETA 06:29:16 2020-11-03 13:54:58 [INFO] [TRAIN] epoch=97, iter=36000/80000, loss=0.2081, lr=0.005843, batch_cost=0.5337, reader_cost=0.0005 | ETA 06:31:23 2020-11-03 13:55:51 [INFO] [TRAIN] epoch=98, iter=36100/80000, loss=0.1929, lr=0.005831, batch_cost=0.5383, reader_cost=0.0087 | ETA 06:33:50 2020-11-03 13:56:44 [INFO] [TRAIN] epoch=98, iter=36200/80000, loss=0.1961, lr=0.005819, batch_cost=0.5290, reader_cost=0.0004 | ETA 06:26:10 2020-11-03 13:57:37 [INFO] [TRAIN] epoch=98, iter=36300/80000, loss=0.2079, lr=0.005807, batch_cost=0.5289, reader_cost=0.0006 | ETA 06:25:12 2020-11-03 13:58:30 [INFO] [TRAIN] epoch=98, iter=36400/80000, loss=0.1966, lr=0.005795, batch_cost=0.5306, reader_cost=0.0004 | ETA 06:25:32 2020-11-03 13:59:24 [INFO] [TRAIN] epoch=99, iter=36500/80000, loss=0.1774, lr=0.005783, batch_cost=0.5396, reader_cost=0.0075 | ETA 06:31:11 2020-11-03 14:00:17 [INFO] [TRAIN] epoch=99, iter=36600/80000, loss=0.1923, lr=0.005771, batch_cost=0.5295, reader_cost=0.0004 | ETA 06:23:00 2020-11-03 14:01:10 [INFO] [TRAIN] epoch=99, iter=36700/80000, loss=0.1802, lr=0.005760, batch_cost=0.5286, reader_cost=0.0004 | ETA 06:21:30 2020-11-03 14:02:03 [INFO] [TRAIN] epoch=99, iter=36800/80000, loss=0.1809, lr=0.005748, batch_cost=0.5329, reader_cost=0.0005 | ETA 06:23:41 2020-11-03 14:02:57 [INFO] [TRAIN] epoch=100, iter=36900/80000, loss=0.1873, lr=0.005736, batch_cost=0.5401, reader_cost=0.0096 | ETA 06:27:57 2020-11-03 14:03:51 [INFO] [TRAIN] epoch=100, iter=37000/80000, loss=0.1696, lr=0.005724, batch_cost=0.5321, reader_cost=0.0004 | ETA 06:21:18 2020-11-03 14:04:44 [INFO] [TRAIN] epoch=100, iter=37100/80000, loss=0.1772, lr=0.005712, batch_cost=0.5320, reader_cost=0.0005 | ETA 06:20:24 2020-11-03 14:05:37 [INFO] [TRAIN] epoch=100, iter=37200/80000, loss=0.1884, lr=0.005700, batch_cost=0.5293, reader_cost=0.0005 | ETA 06:17:33 2020-11-03 14:06:30 [INFO] [TRAIN] epoch=101, iter=37300/80000, loss=0.1938, lr=0.005688, batch_cost=0.5378, reader_cost=0.0084 | ETA 06:22:43 2020-11-03 14:07:23 [INFO] [TRAIN] epoch=101, iter=37400/80000, loss=0.2174, lr=0.005676, batch_cost=0.5294, reader_cost=0.0004 | ETA 06:15:53 2020-11-03 14:08:17 [INFO] [TRAIN] epoch=101, iter=37500/80000, loss=0.2215, lr=0.005664, batch_cost=0.5324, reader_cost=0.0005 | ETA 06:17:09 2020-11-03 14:09:11 [INFO] [TRAIN] epoch=102, iter=37600/80000, loss=0.1944, lr=0.005652, batch_cost=0.5416, reader_cost=0.0083 | ETA 06:22:45 2020-11-03 14:10:04 [INFO] [TRAIN] epoch=102, iter=37700/80000, loss=0.1900, lr=0.005640, batch_cost=0.5303, reader_cost=0.0005 | ETA 06:13:53 2020-11-03 14:10:57 [INFO] [TRAIN] epoch=102, iter=37800/80000, loss=0.1869, lr=0.005628, batch_cost=0.5331, reader_cost=0.0006 | ETA 06:14:56 2020-11-03 14:11:50 [INFO] [TRAIN] epoch=102, iter=37900/80000, loss=0.1796, lr=0.005616, batch_cost=0.5317, reader_cost=0.0006 | ETA 06:13:03 2020-11-03 14:12:45 [INFO] [TRAIN] epoch=103, iter=38000/80000, loss=0.1817, lr=0.005604, batch_cost=0.5433, reader_cost=0.0099 | ETA 06:20:19 2020-11-03 14:13:38 [INFO] [TRAIN] epoch=103, iter=38100/80000, loss=0.2226, lr=0.005592, batch_cost=0.5309, reader_cost=0.0007 | ETA 06:10:43 2020-11-03 14:14:31 [INFO] [TRAIN] epoch=103, iter=38200/80000, loss=0.1820, lr=0.005580, batch_cost=0.5316, reader_cost=0.0007 | ETA 06:10:21 2020-11-03 14:15:24 [INFO] [TRAIN] epoch=103, iter=38300/80000, loss=0.1825, lr=0.005568, batch_cost=0.5331, reader_cost=0.0007 | ETA 06:10:31 2020-11-03 14:16:18 [INFO] [TRAIN] epoch=104, iter=38400/80000, loss=0.1782, lr=0.005556, batch_cost=0.5427, reader_cost=0.0080 | ETA 06:16:16 2020-11-03 14:17:11 [INFO] [TRAIN] epoch=104, iter=38500/80000, loss=0.1671, lr=0.005544, batch_cost=0.5301, reader_cost=0.0005 | ETA 06:06:40 2020-11-03 14:18:05 [INFO] [TRAIN] epoch=104, iter=38600/80000, loss=0.1968, lr=0.005532, batch_cost=0.5310, reader_cost=0.0002 | ETA 06:06:24 2020-11-03 14:18:59 [INFO] [TRAIN] epoch=105, iter=38700/80000, loss=0.1868, lr=0.005520, batch_cost=0.5481, reader_cost=0.0077 | ETA 06:17:14 2020-11-03 14:19:54 [INFO] [TRAIN] epoch=105, iter=38800/80000, loss=0.1746, lr=0.005508, batch_cost=0.5429, reader_cost=0.0004 | ETA 06:12:48 2020-11-03 14:20:48 [INFO] [TRAIN] epoch=105, iter=38900/80000, loss=0.1991, lr=0.005496, batch_cost=0.5438, reader_cost=0.0012 | ETA 06:12:29 2020-11-03 14:21:42 [INFO] [TRAIN] epoch=105, iter=39000/80000, loss=0.2053, lr=0.005484, batch_cost=0.5417, reader_cost=0.0015 | ETA 06:10:11 2020-11-03 14:22:37 [INFO] [TRAIN] epoch=106, iter=39100/80000, loss=0.1813, lr=0.005472, batch_cost=0.5515, reader_cost=0.0101 | ETA 06:15:56 2020-11-03 14:23:32 [INFO] [TRAIN] epoch=106, iter=39200/80000, loss=0.1610, lr=0.005460, batch_cost=0.5459, reader_cost=0.0009 | ETA 06:11:14 2020-11-03 14:24:26 [INFO] [TRAIN] epoch=106, iter=39300/80000, loss=0.1814, lr=0.005448, batch_cost=0.5414, reader_cost=0.0012 | ETA 06:07:16 2020-11-03 14:25:21 [INFO] [TRAIN] epoch=106, iter=39400/80000, loss=0.1783, lr=0.005436, batch_cost=0.5466, reader_cost=0.0010 | ETA 06:09:52 2020-11-03 14:26:16 [INFO] [TRAIN] epoch=107, iter=39500/80000, loss=0.1843, lr=0.005424, batch_cost=0.5534, reader_cost=0.0076 | ETA 06:13:30 2020-11-03 14:27:11 [INFO] [TRAIN] epoch=107, iter=39600/80000, loss=0.1755, lr=0.005412, batch_cost=0.5453, reader_cost=0.0021 | ETA 06:07:09 2020-11-03 14:28:05 [INFO] [TRAIN] epoch=107, iter=39700/80000, loss=0.1771, lr=0.005400, batch_cost=0.5459, reader_cost=0.0006 | ETA 06:06:38 2020-11-03 14:29:00 [INFO] [TRAIN] epoch=107, iter=39800/80000, loss=0.1916, lr=0.005388, batch_cost=0.5433, reader_cost=0.0009 | ETA 06:03:58 2020-11-03 14:29:55 [INFO] [TRAIN] epoch=108, iter=39900/80000, loss=0.1840, lr=0.005376, batch_cost=0.5537, reader_cost=0.0100 | ETA 06:10:02 2020-11-03 14:30:49 [INFO] [TRAIN] epoch=108, iter=40000/80000, loss=0.2044, lr=0.005364, batch_cost=0.5443, reader_cost=0.0011 | ETA 06:02:52 2020-11-03 14:30:58 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-03 14:36:17 [INFO] [EVAL] #Images=500 mIoU=0.7588 Acc=0.9566 Kappa=0.9436 2020-11-03 14:36:17 [INFO] [EVAL] Category IoU: [0.9773 0.8246 0.9157 0.3925 0.5713 0.6364 0.7131 0.7814 0.9208 0.6196 0.9423 0.8116 0.5523 0.9478 0.6872 0.8549 0.8202 0.6717 0.7771] 2020-11-03 14:36:17 [INFO] [EVAL] Category Acc: [0.9846 0.9339 0.9419 0.8496 0.8793 0.7954 0.842 0.9028 0.9513 0.8957 0.9634 0.8654 0.8166 0.9698 0.9447 0.9314 0.9639 0.8386 0.8843] 2020-11-03 14:36:21 [INFO] [EVAL] The model with the best validation mIoU (0.7588) was saved at iter 40000. 2020-11-03 14:37:16 [INFO] [TRAIN] epoch=108, iter=40100/80000, loss=0.1925, lr=0.005352, batch_cost=0.5434, reader_cost=0.0017 | ETA 06:01:20 2020-11-03 14:38:11 [INFO] [TRAIN] epoch=109, iter=40200/80000, loss=0.1841, lr=0.005340, batch_cost=0.5505, reader_cost=0.0104 | ETA 06:05:11 2020-11-03 14:39:05 [INFO] [TRAIN] epoch=109, iter=40300/80000, loss=0.1746, lr=0.005327, batch_cost=0.5455, reader_cost=0.0018 | ETA 06:00:56 2020-11-03 14:40:00 [INFO] [TRAIN] epoch=109, iter=40400/80000, loss=0.1690, lr=0.005315, batch_cost=0.5446, reader_cost=0.0019 | ETA 05:59:26 2020-11-03 14:40:54 [INFO] [TRAIN] epoch=109, iter=40500/80000, loss=0.2091, lr=0.005303, batch_cost=0.5412, reader_cost=0.0020 | ETA 05:56:18 2020-11-03 14:41:49 [INFO] [TRAIN] epoch=110, iter=40600/80000, loss=0.1582, lr=0.005291, batch_cost=0.5548, reader_cost=0.0089 | ETA 06:04:17 2020-11-03 14:42:44 [INFO] [TRAIN] epoch=110, iter=40700/80000, loss=0.1956, lr=0.005279, batch_cost=0.5470, reader_cost=0.0020 | ETA 05:58:18 2020-11-03 14:43:39 [INFO] [TRAIN] epoch=110, iter=40800/80000, loss=0.1834, lr=0.005267, batch_cost=0.5444, reader_cost=0.0016 | ETA 05:55:38 2020-11-03 14:44:33 [INFO] [TRAIN] epoch=110, iter=40900/80000, loss=0.1966, lr=0.005255, batch_cost=0.5458, reader_cost=0.0025 | ETA 05:55:40 2020-11-03 14:45:28 [INFO] [TRAIN] epoch=111, iter=41000/80000, loss=0.1857, lr=0.005243, batch_cost=0.5524, reader_cost=0.0096 | ETA 05:59:02 2020-11-03 14:46:23 [INFO] [TRAIN] epoch=111, iter=41100/80000, loss=0.1727, lr=0.005231, batch_cost=0.5460, reader_cost=0.0013 | ETA 05:54:00 2020-11-03 14:47:17 [INFO] [TRAIN] epoch=111, iter=41200/80000, loss=0.1914, lr=0.005219, batch_cost=0.5445, reader_cost=0.0009 | ETA 05:52:06 2020-11-03 14:48:13 [INFO] [TRAIN] epoch=112, iter=41300/80000, loss=0.2137, lr=0.005207, batch_cost=0.5571, reader_cost=0.0109 | ETA 05:59:18 2020-11-03 14:49:07 [INFO] [TRAIN] epoch=112, iter=41400/80000, loss=0.2204, lr=0.005195, batch_cost=0.5435, reader_cost=0.0019 | ETA 05:49:40 2020-11-03 14:50:02 [INFO] [TRAIN] epoch=112, iter=41500/80000, loss=0.1820, lr=0.005183, batch_cost=0.5434, reader_cost=0.0014 | ETA 05:48:39 2020-11-03 14:50:57 [INFO] [TRAIN] epoch=112, iter=41600/80000, loss=0.1661, lr=0.005171, batch_cost=0.5475, reader_cost=0.0016 | ETA 05:50:24 2020-11-03 14:51:52 [INFO] [TRAIN] epoch=113, iter=41700/80000, loss=0.1889, lr=0.005158, batch_cost=0.5573, reader_cost=0.0092 | ETA 05:55:44 2020-11-03 14:52:47 [INFO] [TRAIN] epoch=113, iter=41800/80000, loss=0.1654, lr=0.005146, batch_cost=0.5457, reader_cost=0.0017 | ETA 05:47:25 2020-11-03 14:53:41 [INFO] [TRAIN] epoch=113, iter=41900/80000, loss=0.1780, lr=0.005134, batch_cost=0.5441, reader_cost=0.0017 | ETA 05:45:29 2020-11-03 14:54:36 [INFO] [TRAIN] epoch=113, iter=42000/80000, loss=0.1873, lr=0.005122, batch_cost=0.5423, reader_cost=0.0020 | ETA 05:43:26 2020-11-03 14:55:31 [INFO] [TRAIN] epoch=114, iter=42100/80000, loss=0.1929, lr=0.005110, batch_cost=0.5584, reader_cost=0.0096 | ETA 05:52:44 2020-11-03 14:56:26 [INFO] [TRAIN] epoch=114, iter=42200/80000, loss=0.2029, lr=0.005098, batch_cost=0.5474, reader_cost=0.0015 | ETA 05:44:52 2020-11-03 14:57:21 [INFO] [TRAIN] epoch=114, iter=42300/80000, loss=0.1915, lr=0.005086, batch_cost=0.5445, reader_cost=0.0020 | ETA 05:42:09 2020-11-03 14:58:15 [INFO] [TRAIN] epoch=114, iter=42400/80000, loss=0.2003, lr=0.005074, batch_cost=0.5438, reader_cost=0.0018 | ETA 05:40:46 2020-11-03 14:59:10 [INFO] [TRAIN] epoch=115, iter=42500/80000, loss=0.1776, lr=0.005062, batch_cost=0.5555, reader_cost=0.0095 | ETA 05:47:09 2020-11-03 15:00:05 [INFO] [TRAIN] epoch=115, iter=42600/80000, loss=0.1742, lr=0.005049, batch_cost=0.5459, reader_cost=0.0015 | ETA 05:40:15 2020-11-03 15:00:59 [INFO] [TRAIN] epoch=115, iter=42700/80000, loss=0.2081, lr=0.005037, batch_cost=0.5358, reader_cost=0.0007 | ETA 05:33:05 2020-11-03 15:01:53 [INFO] [TRAIN] epoch=116, iter=42800/80000, loss=0.1838, lr=0.005025, batch_cost=0.5413, reader_cost=0.0080 | ETA 05:35:35 2020-11-03 15:02:46 [INFO] [TRAIN] epoch=116, iter=42900/80000, loss=0.1811, lr=0.005013, batch_cost=0.5328, reader_cost=0.0005 | ETA 05:29:25 2020-11-03 15:03:40 [INFO] [TRAIN] epoch=116, iter=43000/80000, loss=0.1980, lr=0.005001, batch_cost=0.5378, reader_cost=0.0009 | ETA 05:31:38 2020-11-03 15:04:34 [INFO] [TRAIN] epoch=116, iter=43100/80000, loss=0.1731, lr=0.004989, batch_cost=0.5416, reader_cost=0.0014 | ETA 05:33:06 2020-11-03 15:05:29 [INFO] [TRAIN] epoch=117, iter=43200/80000, loss=0.1665, lr=0.004977, batch_cost=0.5469, reader_cost=0.0095 | ETA 05:35:26 2020-11-03 15:06:23 [INFO] [TRAIN] epoch=117, iter=43300/80000, loss=0.1878, lr=0.004964, batch_cost=0.5435, reader_cost=0.0014 | ETA 05:32:25 2020-11-03 15:07:17 [INFO] [TRAIN] epoch=117, iter=43400/80000, loss=0.2096, lr=0.004952, batch_cost=0.5408, reader_cost=0.0012 | ETA 05:29:53 2020-11-03 15:08:11 [INFO] [TRAIN] epoch=117, iter=43500/80000, loss=0.1939, lr=0.004940, batch_cost=0.5362, reader_cost=0.0006 | ETA 05:26:11 2020-11-03 15:09:06 [INFO] [TRAIN] epoch=118, iter=43600/80000, loss=0.1835, lr=0.004928, batch_cost=0.5534, reader_cost=0.0078 | ETA 05:35:41 2020-11-03 15:10:01 [INFO] [TRAIN] epoch=118, iter=43700/80000, loss=0.1818, lr=0.004916, batch_cost=0.5445, reader_cost=0.0020 | ETA 05:29:24 2020-11-03 15:10:55 [INFO] [TRAIN] epoch=118, iter=43800/80000, loss=0.1980, lr=0.004904, batch_cost=0.5423, reader_cost=0.0014 | ETA 05:27:09 2020-11-03 15:11:50 [INFO] [TRAIN] epoch=119, iter=43900/80000, loss=0.1743, lr=0.004891, batch_cost=0.5512, reader_cost=0.0093 | ETA 05:31:39 2020-11-03 15:12:44 [INFO] [TRAIN] epoch=119, iter=44000/80000, loss=0.1699, lr=0.004879, batch_cost=0.5412, reader_cost=0.0010 | ETA 05:24:42 2020-11-03 15:13:38 [INFO] [TRAIN] epoch=119, iter=44100/80000, loss=0.1952, lr=0.004867, batch_cost=0.5389, reader_cost=0.0006 | ETA 05:22:25 2020-11-03 15:14:32 [INFO] [TRAIN] epoch=119, iter=44200/80000, loss=0.1926, lr=0.004855, batch_cost=0.5377, reader_cost=0.0012 | ETA 05:20:48 2020-11-03 15:15:27 [INFO] [TRAIN] epoch=120, iter=44300/80000, loss=0.1834, lr=0.004843, batch_cost=0.5508, reader_cost=0.0078 | ETA 05:27:43 2020-11-03 15:16:28 [INFO] [TRAIN] epoch=120, iter=44400/80000, loss=0.1960, lr=0.004831, batch_cost=0.6171, reader_cost=0.0029 | ETA 06:06:09 2020-11-03 15:18:05 [INFO] [TRAIN] epoch=120, iter=44500/80000, loss=0.1793, lr=0.004818, batch_cost=0.9682, reader_cost=0.0044 | ETA 09:32:52 2020-11-03 15:19:34 [INFO] [TRAIN] epoch=120, iter=44600/80000, loss=0.2004, lr=0.004806, batch_cost=0.8898, reader_cost=0.0060 | ETA 08:44:57 2020-11-03 15:20:41 [INFO] [TRAIN] epoch=121, iter=44700/80000, loss=0.1754, lr=0.004794, batch_cost=0.6675, reader_cost=0.0109 | ETA 06:32:42 2020-11-03 15:21:51 [INFO] [TRAIN] epoch=121, iter=44800/80000, loss=0.1812, lr=0.004782, batch_cost=0.7001, reader_cost=0.0037 | ETA 06:50:44 2020-11-03 15:23:01 [INFO] [TRAIN] epoch=121, iter=44900/80000, loss=0.1957, lr=0.004770, batch_cost=0.7025, reader_cost=0.0029 | ETA 06:50:58 2020-11-03 15:24:08 [INFO] [TRAIN] epoch=121, iter=45000/80000, loss=0.1738, lr=0.004757, batch_cost=0.6639, reader_cost=0.0021 | ETA 06:27:17 2020-11-03 15:25:10 [INFO] [TRAIN] epoch=122, iter=45100/80000, loss=0.1807, lr=0.004745, batch_cost=0.6257, reader_cost=0.0090 | ETA 06:03:56 2020-11-03 15:26:07 [INFO] [TRAIN] epoch=122, iter=45200/80000, loss=0.1659, lr=0.004733, batch_cost=0.5720, reader_cost=0.0007 | ETA 05:31:45 2020-11-03 15:27:02 [INFO] [TRAIN] epoch=122, iter=45300/80000, loss=0.1859, lr=0.004721, batch_cost=0.5409, reader_cost=0.0011 | ETA 05:12:50 2020-11-03 15:27:57 [INFO] [TRAIN] epoch=123, iter=45400/80000, loss=0.1759, lr=0.004709, batch_cost=0.5521, reader_cost=0.0090 | ETA 05:18:22 2020-11-03 15:28:51 [INFO] [TRAIN] epoch=123, iter=45500/80000, loss=0.1664, lr=0.004696, batch_cost=0.5446, reader_cost=0.0012 | ETA 05:13:09 2020-11-03 15:29:45 [INFO] [TRAIN] epoch=123, iter=45600/80000, loss=0.1709, lr=0.004684, batch_cost=0.5408, reader_cost=0.0008 | ETA 05:10:02 2020-11-03 15:30:40 [INFO] [TRAIN] epoch=123, iter=45700/80000, loss=0.1919, lr=0.004672, batch_cost=0.5459, reader_cost=0.0015 | ETA 05:12:04 2020-11-03 15:31:35 [INFO] [TRAIN] epoch=124, iter=45800/80000, loss=0.1857, lr=0.004660, batch_cost=0.5534, reader_cost=0.0090 | ETA 05:15:27 2020-11-03 15:32:30 [INFO] [TRAIN] epoch=124, iter=45900/80000, loss=0.1901, lr=0.004647, batch_cost=0.5439, reader_cost=0.0014 | ETA 05:09:07 2020-11-03 15:33:24 [INFO] [TRAIN] epoch=124, iter=46000/80000, loss=0.1558, lr=0.004635, batch_cost=0.5486, reader_cost=0.0014 | ETA 05:10:50 2020-11-03 15:34:19 [INFO] [TRAIN] epoch=124, iter=46100/80000, loss=0.1807, lr=0.004623, batch_cost=0.5453, reader_cost=0.0015 | ETA 05:08:06 2020-11-03 15:35:15 [INFO] [TRAIN] epoch=125, iter=46200/80000, loss=0.1757, lr=0.004611, batch_cost=0.5555, reader_cost=0.0124 | ETA 05:12:54 2020-11-03 15:36:10 [INFO] [TRAIN] epoch=125, iter=46300/80000, loss=0.1893, lr=0.004598, batch_cost=0.5499, reader_cost=0.0024 | ETA 05:08:50 2020-11-03 15:37:04 [INFO] [TRAIN] epoch=125, iter=46400/80000, loss=0.1818, lr=0.004586, batch_cost=0.5465, reader_cost=0.0026 | ETA 05:06:03 2020-11-03 15:37:59 [INFO] [TRAIN] epoch=125, iter=46500/80000, loss=0.1778, lr=0.004574, batch_cost=0.5485, reader_cost=0.0017 | ETA 05:06:16 2020-11-03 15:38:55 [INFO] [TRAIN] epoch=126, iter=46600/80000, loss=0.1697, lr=0.004562, batch_cost=0.5601, reader_cost=0.0104 | ETA 05:11:47 2020-11-03 15:39:50 [INFO] [TRAIN] epoch=126, iter=46700/80000, loss=0.1895, lr=0.004549, batch_cost=0.5481, reader_cost=0.0021 | ETA 05:04:12 2020-11-03 15:40:45 [INFO] [TRAIN] epoch=126, iter=46800/80000, loss=0.1723, lr=0.004537, batch_cost=0.5490, reader_cost=0.0017 | ETA 05:03:48 2020-11-03 15:41:41 [INFO] [TRAIN] epoch=127, iter=46900/80000, loss=0.1685, lr=0.004525, batch_cost=0.5597, reader_cost=0.0115 | ETA 05:08:45 2020-11-03 15:42:36 [INFO] [TRAIN] epoch=127, iter=47000/80000, loss=0.1817, lr=0.004513, batch_cost=0.5507, reader_cost=0.0021 | ETA 05:02:53 2020-11-03 15:43:30 [INFO] [TRAIN] epoch=127, iter=47100/80000, loss=0.1649, lr=0.004500, batch_cost=0.5459, reader_cost=0.0021 | ETA 04:59:20 2020-11-03 15:44:25 [INFO] [TRAIN] epoch=127, iter=47200/80000, loss=0.1838, lr=0.004488, batch_cost=0.5471, reader_cost=0.0021 | ETA 04:59:06 2020-11-03 15:45:21 [INFO] [TRAIN] epoch=128, iter=47300/80000, loss=0.1639, lr=0.004476, batch_cost=0.5616, reader_cost=0.0106 | ETA 05:06:05 2020-11-03 15:46:16 [INFO] [TRAIN] epoch=128, iter=47400/80000, loss=0.1742, lr=0.004463, batch_cost=0.5486, reader_cost=0.0015 | ETA 04:58:05 2020-11-03 15:47:11 [INFO] [TRAIN] epoch=128, iter=47500/80000, loss=0.1857, lr=0.004451, batch_cost=0.5501, reader_cost=0.0017 | ETA 04:57:59 2020-11-03 15:48:06 [INFO] [TRAIN] epoch=128, iter=47600/80000, loss=0.1852, lr=0.004439, batch_cost=0.5499, reader_cost=0.0024 | ETA 04:56:55 2020-11-03 15:49:02 [INFO] [TRAIN] epoch=129, iter=47700/80000, loss=0.1778, lr=0.004427, batch_cost=0.5616, reader_cost=0.0108 | ETA 05:02:19 2020-11-03 15:49:57 [INFO] [TRAIN] epoch=129, iter=47800/80000, loss=0.1618, lr=0.004414, batch_cost=0.5451, reader_cost=0.0015 | ETA 04:52:31 2020-11-03 15:50:51 [INFO] [TRAIN] epoch=129, iter=47900/80000, loss=0.1727, lr=0.004402, batch_cost=0.5419, reader_cost=0.0011 | ETA 04:49:53 2020-11-03 15:51:46 [INFO] [TRAIN] epoch=130, iter=48000/80000, loss=0.1500, lr=0.004390, batch_cost=0.5477, reader_cost=0.0091 | ETA 04:52:07 2020-11-03 15:51:53 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-03 15:57:22 [INFO] [EVAL] #Images=500 mIoU=0.7758 Acc=0.9586 Kappa=0.9462 2020-11-03 15:57:22 [INFO] [EVAL] Category IoU: [0.9768 0.8236 0.922 0.5078 0.6048 0.6463 0.7181 0.7903 0.924 0.6234 0.9405 0.8201 0.5859 0.9499 0.7448 0.8818 0.8308 0.668 0.7818] 2020-11-03 15:57:22 [INFO] [EVAL] Category Acc: [0.9863 0.9191 0.9527 0.8748 0.8646 0.7898 0.8452 0.9156 0.9513 0.8913 0.9603 0.8762 0.8297 0.972 0.9288 0.9278 0.9504 0.8422 0.8559] 2020-11-03 15:57:26 [INFO] [EVAL] The model with the best validation mIoU (0.7758) was saved at iter 48000. 2020-11-03 15:58:20 [INFO] [TRAIN] epoch=130, iter=48100/80000, loss=0.1511, lr=0.004377, batch_cost=0.5414, reader_cost=0.0010 | ETA 04:47:51 2020-11-03 15:59:14 [INFO] [TRAIN] epoch=130, iter=48200/80000, loss=0.1689, lr=0.004365, batch_cost=0.5405, reader_cost=0.0008 | ETA 04:46:27 2020-11-03 16:00:09 [INFO] [TRAIN] epoch=130, iter=48300/80000, loss=0.1957, lr=0.004353, batch_cost=0.5421, reader_cost=0.0008 | ETA 04:46:24 2020-11-03 16:01:04 [INFO] [TRAIN] epoch=131, iter=48400/80000, loss=0.1584, lr=0.004340, batch_cost=0.5543, reader_cost=0.0092 | ETA 04:51:56 2020-11-03 16:01:58 [INFO] [TRAIN] epoch=131, iter=48500/80000, loss=0.1687, lr=0.004328, batch_cost=0.5422, reader_cost=0.0009 | ETA 04:44:40 2020-11-03 16:02:53 [INFO] [TRAIN] epoch=131, iter=48600/80000, loss=0.1717, lr=0.004316, batch_cost=0.5435, reader_cost=0.0014 | ETA 04:44:24 2020-11-03 16:03:47 [INFO] [TRAIN] epoch=131, iter=48700/80000, loss=0.1642, lr=0.004303, batch_cost=0.5445, reader_cost=0.0008 | ETA 04:44:01 2020-11-03 16:04:42 [INFO] [TRAIN] epoch=132, iter=48800/80000, loss=0.1622, lr=0.004291, batch_cost=0.5532, reader_cost=0.0091 | ETA 04:47:40 2020-11-03 16:05:37 [INFO] [TRAIN] epoch=132, iter=48900/80000, loss=0.1822, lr=0.004279, batch_cost=0.5420, reader_cost=0.0015 | ETA 04:40:55 2020-11-03 16:06:31 [INFO] [TRAIN] epoch=132, iter=49000/80000, loss=0.1665, lr=0.004266, batch_cost=0.5423, reader_cost=0.0007 | ETA 04:40:12 2020-11-03 16:07:25 [INFO] [TRAIN] epoch=132, iter=49100/80000, loss=0.1898, lr=0.004254, batch_cost=0.5395, reader_cost=0.0012 | ETA 04:37:50 2020-11-03 16:08:20 [INFO] [TRAIN] epoch=133, iter=49200/80000, loss=0.1822, lr=0.004241, batch_cost=0.5521, reader_cost=0.0096 | ETA 04:43:24 2020-11-03 16:09:15 [INFO] [TRAIN] epoch=133, iter=49300/80000, loss=0.1616, lr=0.004229, batch_cost=0.5451, reader_cost=0.0007 | ETA 04:38:53 2020-11-03 16:10:09 [INFO] [TRAIN] epoch=133, iter=49400/80000, loss=0.1563, lr=0.004217, batch_cost=0.5410, reader_cost=0.0011 | ETA 04:35:54 2020-11-03 16:11:04 [INFO] [TRAIN] epoch=134, iter=49500/80000, loss=0.1644, lr=0.004204, batch_cost=0.5553, reader_cost=0.0092 | ETA 04:42:17 2020-11-03 16:11:58 [INFO] [TRAIN] epoch=134, iter=49600/80000, loss=0.1647, lr=0.004192, batch_cost=0.5428, reader_cost=0.0014 | ETA 04:35:01 2020-11-03 16:12:53 [INFO] [TRAIN] epoch=134, iter=49700/80000, loss=0.1714, lr=0.004180, batch_cost=0.5413, reader_cost=0.0016 | ETA 04:33:22 2020-11-03 16:13:47 [INFO] [TRAIN] epoch=134, iter=49800/80000, loss=0.1867, lr=0.004167, batch_cost=0.5424, reader_cost=0.0021 | ETA 04:33:00 2020-11-03 16:14:42 [INFO] [TRAIN] epoch=135, iter=49900/80000, loss=0.1721, lr=0.004155, batch_cost=0.5535, reader_cost=0.0080 | ETA 04:37:39 2020-11-03 16:15:37 [INFO] [TRAIN] epoch=135, iter=50000/80000, loss=0.1638, lr=0.004142, batch_cost=0.5452, reader_cost=0.0013 | ETA 04:32:37 2020-11-03 16:16:31 [INFO] [TRAIN] epoch=135, iter=50100/80000, loss=0.1810, lr=0.004130, batch_cost=0.5445, reader_cost=0.0013 | ETA 04:31:20 2020-11-03 16:17:25 [INFO] [TRAIN] epoch=135, iter=50200/80000, loss=0.1887, lr=0.004118, batch_cost=0.5419, reader_cost=0.0013 | ETA 04:29:08 2020-11-03 16:18:21 [INFO] [TRAIN] epoch=136, iter=50300/80000, loss=0.1980, lr=0.004105, batch_cost=0.5518, reader_cost=0.0093 | ETA 04:33:09 2020-11-03 16:19:15 [INFO] [TRAIN] epoch=136, iter=50400/80000, loss=0.1653, lr=0.004093, batch_cost=0.5436, reader_cost=0.0013 | ETA 04:28:11 2020-11-03 16:20:09 [INFO] [TRAIN] epoch=136, iter=50500/80000, loss=0.1678, lr=0.004080, batch_cost=0.5453, reader_cost=0.0013 | ETA 04:28:05 2020-11-03 16:21:05 [INFO] [TRAIN] epoch=137, iter=50600/80000, loss=0.1759, lr=0.004068, batch_cost=0.5532, reader_cost=0.0090 | ETA 04:31:03 2020-11-03 16:21:59 [INFO] [TRAIN] epoch=137, iter=50700/80000, loss=0.1667, lr=0.004056, batch_cost=0.5437, reader_cost=0.0014 | ETA 04:25:30 2020-11-03 16:22:53 [INFO] [TRAIN] epoch=137, iter=50800/80000, loss=0.1668, lr=0.004043, batch_cost=0.5428, reader_cost=0.0017 | ETA 04:24:09 2020-11-03 16:23:48 [INFO] [TRAIN] epoch=137, iter=50900/80000, loss=0.1639, lr=0.004031, batch_cost=0.5427, reader_cost=0.0014 | ETA 04:23:13 2020-11-03 16:24:43 [INFO] [TRAIN] epoch=138, iter=51000/80000, loss=0.1443, lr=0.004018, batch_cost=0.5552, reader_cost=0.0092 | ETA 04:28:20 2020-11-03 16:25:38 [INFO] [TRAIN] epoch=138, iter=51100/80000, loss=0.1743, lr=0.004006, batch_cost=0.5455, reader_cost=0.0013 | ETA 04:22:46 2020-11-03 16:26:32 [INFO] [TRAIN] epoch=138, iter=51200/80000, loss=0.1759, lr=0.003993, batch_cost=0.5431, reader_cost=0.0010 | ETA 04:20:42 2020-11-03 16:27:26 [INFO] [TRAIN] epoch=138, iter=51300/80000, loss=0.1769, lr=0.003981, batch_cost=0.5423, reader_cost=0.0017 | ETA 04:19:23 2020-11-03 16:28:22 [INFO] [TRAIN] epoch=139, iter=51400/80000, loss=0.1585, lr=0.003968, batch_cost=0.5537, reader_cost=0.0088 | ETA 04:23:54 2020-11-03 16:29:16 [INFO] [TRAIN] epoch=139, iter=51500/80000, loss=0.1617, lr=0.003956, batch_cost=0.5435, reader_cost=0.0008 | ETA 04:18:09 2020-11-03 16:30:10 [INFO] [TRAIN] epoch=139, iter=51600/80000, loss=0.1609, lr=0.003944, batch_cost=0.5418, reader_cost=0.0012 | ETA 04:16:26 2020-11-03 16:31:04 [INFO] [TRAIN] epoch=139, iter=51700/80000, loss=0.1765, lr=0.003931, batch_cost=0.5383, reader_cost=0.0010 | ETA 04:13:52 2020-11-03 16:32:00 [INFO] [TRAIN] epoch=140, iter=51800/80000, loss=0.1529, lr=0.003919, batch_cost=0.5549, reader_cost=0.0076 | ETA 04:20:48 2020-11-03 16:32:54 [INFO] [TRAIN] epoch=140, iter=51900/80000, loss=0.1717, lr=0.003906, batch_cost=0.5440, reader_cost=0.0007 | ETA 04:14:46 2020-11-03 16:33:48 [INFO] [TRAIN] epoch=140, iter=52000/80000, loss=0.1808, lr=0.003894, batch_cost=0.5423, reader_cost=0.0009 | ETA 04:13:04 2020-11-03 16:34:43 [INFO] [TRAIN] epoch=141, iter=52100/80000, loss=0.1744, lr=0.003881, batch_cost=0.5511, reader_cost=0.0105 | ETA 04:16:16 2020-11-03 16:35:36 [INFO] [TRAIN] epoch=141, iter=52200/80000, loss=0.1665, lr=0.003869, batch_cost=0.5295, reader_cost=0.0004 | ETA 04:05:18 2020-11-03 16:36:29 [INFO] [TRAIN] epoch=141, iter=52300/80000, loss=0.1635, lr=0.003856, batch_cost=0.5280, reader_cost=0.0004 | ETA 04:03:45 2020-11-03 16:37:22 [INFO] [TRAIN] epoch=141, iter=52400/80000, loss=0.2010, lr=0.003844, batch_cost=0.5319, reader_cost=0.0004 | ETA 04:04:40 2020-11-03 16:38:18 [INFO] [TRAIN] epoch=142, iter=52500/80000, loss=0.1633, lr=0.003831, batch_cost=0.5555, reader_cost=0.0089 | ETA 04:14:35 2020-11-03 16:39:12 [INFO] [TRAIN] epoch=142, iter=52600/80000, loss=0.1511, lr=0.003819, batch_cost=0.5412, reader_cost=0.0013 | ETA 04:07:09 2020-11-03 16:40:06 [INFO] [TRAIN] epoch=142, iter=52700/80000, loss=0.1755, lr=0.003806, batch_cost=0.5432, reader_cost=0.0012 | ETA 04:07:08 2020-11-03 16:41:01 [INFO] [TRAIN] epoch=142, iter=52800/80000, loss=0.1727, lr=0.003794, batch_cost=0.5445, reader_cost=0.0009 | ETA 04:06:49 2020-11-03 16:41:56 [INFO] [TRAIN] epoch=143, iter=52900/80000, loss=0.1695, lr=0.003781, batch_cost=0.5531, reader_cost=0.0080 | ETA 04:09:48 2020-11-03 16:42:50 [INFO] [TRAIN] epoch=143, iter=53000/80000, loss=0.1501, lr=0.003769, batch_cost=0.5447, reader_cost=0.0013 | ETA 04:05:07 2020-11-03 16:43:45 [INFO] [TRAIN] epoch=143, iter=53100/80000, loss=0.1550, lr=0.003756, batch_cost=0.5448, reader_cost=0.0011 | ETA 04:04:15 2020-11-03 16:44:40 [INFO] [TRAIN] epoch=144, iter=53200/80000, loss=0.1724, lr=0.003744, batch_cost=0.5535, reader_cost=0.0088 | ETA 04:07:13 2020-11-03 16:45:34 [INFO] [TRAIN] epoch=144, iter=53300/80000, loss=0.1513, lr=0.003731, batch_cost=0.5421, reader_cost=0.0009 | ETA 04:01:13 2020-11-03 16:46:28 [INFO] [TRAIN] epoch=144, iter=53400/80000, loss=0.1765, lr=0.003718, batch_cost=0.5398, reader_cost=0.0009 | ETA 03:59:17 2020-11-03 16:47:23 [INFO] [TRAIN] epoch=144, iter=53500/80000, loss=0.1671, lr=0.003706, batch_cost=0.5444, reader_cost=0.0012 | ETA 04:00:26 2020-11-03 16:48:18 [INFO] [TRAIN] epoch=145, iter=53600/80000, loss=0.1643, lr=0.003693, batch_cost=0.5527, reader_cost=0.0098 | ETA 04:03:10 2020-11-03 16:49:12 [INFO] [TRAIN] epoch=145, iter=53700/80000, loss=0.1733, lr=0.003681, batch_cost=0.5428, reader_cost=0.0012 | ETA 03:57:56 2020-11-03 16:50:07 [INFO] [TRAIN] epoch=145, iter=53800/80000, loss=0.1632, lr=0.003668, batch_cost=0.5417, reader_cost=0.0009 | ETA 03:56:32 2020-11-03 16:51:01 [INFO] [TRAIN] epoch=145, iter=53900/80000, loss=0.1738, lr=0.003656, batch_cost=0.5410, reader_cost=0.0014 | ETA 03:55:19 2020-11-03 16:51:56 [INFO] [TRAIN] epoch=146, iter=54000/80000, loss=0.1805, lr=0.003643, batch_cost=0.5518, reader_cost=0.0091 | ETA 03:59:07 2020-11-03 16:52:50 [INFO] [TRAIN] epoch=146, iter=54100/80000, loss=0.1587, lr=0.003631, batch_cost=0.5429, reader_cost=0.0007 | ETA 03:54:22 2020-11-03 16:53:44 [INFO] [TRAIN] epoch=146, iter=54200/80000, loss=0.1803, lr=0.003618, batch_cost=0.5426, reader_cost=0.0006 | ETA 03:53:19 2020-11-03 16:54:39 [INFO] [TRAIN] epoch=146, iter=54300/80000, loss=0.1758, lr=0.003605, batch_cost=0.5429, reader_cost=0.0008 | ETA 03:52:33 2020-11-03 16:55:34 [INFO] [TRAIN] epoch=147, iter=54400/80000, loss=0.1644, lr=0.003593, batch_cost=0.5531, reader_cost=0.0088 | ETA 03:55:59 2020-11-03 16:56:28 [INFO] [TRAIN] epoch=147, iter=54500/80000, loss=0.1463, lr=0.003580, batch_cost=0.5410, reader_cost=0.0011 | ETA 03:49:56 2020-11-03 16:57:22 [INFO] [TRAIN] epoch=147, iter=54600/80000, loss=0.1636, lr=0.003568, batch_cost=0.5394, reader_cost=0.0008 | ETA 03:48:19 2020-11-03 16:58:17 [INFO] [TRAIN] epoch=148, iter=54700/80000, loss=0.1690, lr=0.003555, batch_cost=0.5527, reader_cost=0.0083 | ETA 03:53:04 2020-11-03 16:59:11 [INFO] [TRAIN] epoch=148, iter=54800/80000, loss=0.1665, lr=0.003542, batch_cost=0.5405, reader_cost=0.0011 | ETA 03:46:59 2020-11-03 17:00:06 [INFO] [TRAIN] epoch=148, iter=54900/80000, loss=0.1834, lr=0.003530, batch_cost=0.5407, reader_cost=0.0011 | ETA 03:46:11 2020-11-03 17:01:00 [INFO] [TRAIN] epoch=148, iter=55000/80000, loss=0.1780, lr=0.003517, batch_cost=0.5423, reader_cost=0.0010 | ETA 03:45:56 2020-11-03 17:01:55 [INFO] [TRAIN] epoch=149, iter=55100/80000, loss=0.1452, lr=0.003504, batch_cost=0.5516, reader_cost=0.0090 | ETA 03:48:55 2020-11-03 17:02:49 [INFO] [TRAIN] epoch=149, iter=55200/80000, loss=0.1726, lr=0.003492, batch_cost=0.5446, reader_cost=0.0019 | ETA 03:45:06 2020-11-03 17:03:43 [INFO] [TRAIN] epoch=149, iter=55300/80000, loss=0.1623, lr=0.003479, batch_cost=0.5383, reader_cost=0.0010 | ETA 03:41:34 2020-11-03 17:04:37 [INFO] [TRAIN] epoch=149, iter=55400/80000, loss=0.1717, lr=0.003467, batch_cost=0.5424, reader_cost=0.0005 | ETA 03:42:22 2020-11-03 17:05:33 [INFO] [TRAIN] epoch=150, iter=55500/80000, loss=0.1770, lr=0.003454, batch_cost=0.5531, reader_cost=0.0093 | ETA 03:45:51 2020-11-03 17:06:27 [INFO] [TRAIN] epoch=150, iter=55600/80000, loss=0.1648, lr=0.003441, batch_cost=0.5428, reader_cost=0.0011 | ETA 03:40:44 2020-11-03 17:07:21 [INFO] [TRAIN] epoch=150, iter=55700/80000, loss=0.1679, lr=0.003429, batch_cost=0.5377, reader_cost=0.0016 | ETA 03:37:46 2020-11-03 17:08:15 [INFO] [TRAIN] epoch=150, iter=55800/80000, loss=0.1753, lr=0.003416, batch_cost=0.5430, reader_cost=0.0010 | ETA 03:39:01 2020-11-03 17:09:10 [INFO] [TRAIN] epoch=151, iter=55900/80000, loss=0.1685, lr=0.003403, batch_cost=0.5534, reader_cost=0.0084 | ETA 03:42:17 2020-11-03 17:10:05 [INFO] [TRAIN] epoch=151, iter=56000/80000, loss=0.1676, lr=0.003391, batch_cost=0.5459, reader_cost=0.0014 | ETA 03:38:22 2020-11-03 17:10:12 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-03 17:15:29 [INFO] [EVAL] #Images=500 mIoU=0.7803 Acc=0.9598 Kappa=0.9477 2020-11-03 17:15:29 [INFO] [EVAL] Category IoU: [0.9796 0.8381 0.9219 0.5013 0.5999 0.6425 0.7202 0.7887 0.9229 0.6479 0.9482 0.825 0.6183 0.9499 0.7928 0.8911 0.8379 0.6179 0.7823] 2020-11-03 17:15:29 [INFO] [EVAL] Category Acc: [0.9877 0.9357 0.9548 0.9029 0.8659 0.7792 0.8482 0.9174 0.946 0.8707 0.9697 0.8844 0.781 0.9706 0.9246 0.9468 0.9564 0.8628 0.8708] 2020-11-03 17:15:33 [INFO] [EVAL] The model with the best validation mIoU (0.7803) was saved at iter 56000. 2020-11-03 17:16:27 [INFO] [TRAIN] epoch=151, iter=56100/80000, loss=0.1560, lr=0.003378, batch_cost=0.5371, reader_cost=0.0017 | ETA 03:33:56 2020-11-03 17:17:21 [INFO] [TRAIN] epoch=152, iter=56200/80000, loss=0.1504, lr=0.003365, batch_cost=0.5469, reader_cost=0.0094 | ETA 03:36:56 2020-11-03 17:18:16 [INFO] [TRAIN] epoch=152, iter=56300/80000, loss=0.1667, lr=0.003353, batch_cost=0.5430, reader_cost=0.0015 | ETA 03:34:28 2020-11-03 17:19:10 [INFO] [TRAIN] epoch=152, iter=56400/80000, loss=0.1630, lr=0.003340, batch_cost=0.5466, reader_cost=0.0016 | ETA 03:34:59 2020-11-03 17:20:04 [INFO] [TRAIN] epoch=152, iter=56500/80000, loss=0.1574, lr=0.003327, batch_cost=0.5385, reader_cost=0.0012 | ETA 03:30:55 2020-11-03 17:20:58 [INFO] [TRAIN] epoch=153, iter=56600/80000, loss=0.1493, lr=0.003314, batch_cost=0.5403, reader_cost=0.0076 | ETA 03:30:43 2020-11-03 17:21:51 [INFO] [TRAIN] epoch=153, iter=56700/80000, loss=0.1608, lr=0.003302, batch_cost=0.5309, reader_cost=0.0004 | ETA 03:26:09 2020-11-03 17:22:45 [INFO] [TRAIN] epoch=153, iter=56800/80000, loss=0.1650, lr=0.003289, batch_cost=0.5337, reader_cost=0.0004 | ETA 03:26:22 2020-11-03 17:23:39 [INFO] [TRAIN] epoch=153, iter=56900/80000, loss=0.1646, lr=0.003276, batch_cost=0.5425, reader_cost=0.0018 | ETA 03:28:51 2020-11-03 17:24:34 [INFO] [TRAIN] epoch=154, iter=57000/80000, loss=0.1466, lr=0.003264, batch_cost=0.5525, reader_cost=0.0088 | ETA 03:31:47 2020-11-03 17:25:28 [INFO] [TRAIN] epoch=154, iter=57100/80000, loss=0.1466, lr=0.003251, batch_cost=0.5407, reader_cost=0.0007 | ETA 03:26:21 2020-11-03 17:26:23 [INFO] [TRAIN] epoch=154, iter=57200/80000, loss=0.1786, lr=0.003238, batch_cost=0.5424, reader_cost=0.0008 | ETA 03:26:05 2020-11-03 17:27:18 [INFO] [TRAIN] epoch=155, iter=57300/80000, loss=0.1602, lr=0.003225, batch_cost=0.5554, reader_cost=0.0093 | ETA 03:30:08 2020-11-03 17:28:13 [INFO] [TRAIN] epoch=155, iter=57400/80000, loss=0.1558, lr=0.003213, batch_cost=0.5470, reader_cost=0.0016 | ETA 03:26:02 2020-11-03 17:29:07 [INFO] [TRAIN] epoch=155, iter=57500/80000, loss=0.1669, lr=0.003200, batch_cost=0.5443, reader_cost=0.0017 | ETA 03:24:05 2020-11-03 17:30:02 [INFO] [TRAIN] epoch=155, iter=57600/80000, loss=0.1619, lr=0.003187, batch_cost=0.5429, reader_cost=0.0013 | ETA 03:22:40 2020-11-03 17:30:57 [INFO] [TRAIN] epoch=156, iter=57700/80000, loss=0.1496, lr=0.003174, batch_cost=0.5556, reader_cost=0.0098 | ETA 03:26:29 2020-11-03 17:31:52 [INFO] [TRAIN] epoch=156, iter=57800/80000, loss=0.1516, lr=0.003162, batch_cost=0.5444, reader_cost=0.0009 | ETA 03:21:26 2020-11-03 17:32:46 [INFO] [TRAIN] epoch=156, iter=57900/80000, loss=0.1688, lr=0.003149, batch_cost=0.5445, reader_cost=0.0012 | ETA 03:20:32 2020-11-03 17:33:41 [INFO] [TRAIN] epoch=156, iter=58000/80000, loss=0.1686, lr=0.003136, batch_cost=0.5456, reader_cost=0.0016 | ETA 03:20:02 2020-11-03 17:34:36 [INFO] [TRAIN] epoch=157, iter=58100/80000, loss=0.1661, lr=0.003123, batch_cost=0.5558, reader_cost=0.0103 | ETA 03:22:52 2020-11-03 17:35:31 [INFO] [TRAIN] epoch=157, iter=58200/80000, loss=0.1488, lr=0.003110, batch_cost=0.5474, reader_cost=0.0008 | ETA 03:18:53 2020-11-03 17:36:49 [INFO] [TRAIN] epoch=157, iter=58300/80000, loss=0.1593, lr=0.003098, batch_cost=0.7841, reader_cost=0.0060 | ETA 04:43:34 2020-11-03 17:38:23 [INFO] [TRAIN] epoch=157, iter=58400/80000, loss=0.1502, lr=0.003085, batch_cost=0.9365, reader_cost=0.0070 | ETA 05:37:07 2020-11-03 17:39:39 [INFO] [TRAIN] epoch=158, iter=58500/80000, loss=0.1519, lr=0.003072, batch_cost=0.7605, reader_cost=0.0192 | ETA 04:32:30 2020-11-03 17:40:46 [INFO] [TRAIN] epoch=158, iter=58600/80000, loss=0.1513, lr=0.003059, batch_cost=0.6735, reader_cost=0.0033 | ETA 04:00:12 2020-11-03 17:41:53 [INFO] [TRAIN] epoch=158, iter=58700/80000, loss=0.1715, lr=0.003046, batch_cost=0.6659, reader_cost=0.0041 | ETA 03:56:24 2020-11-03 17:43:01 [INFO] [TRAIN] epoch=159, iter=58800/80000, loss=0.1582, lr=0.003033, batch_cost=0.6810, reader_cost=0.0129 | ETA 04:00:36 2020-11-03 17:44:05 [INFO] [TRAIN] epoch=159, iter=58900/80000, loss=0.1605, lr=0.003021, batch_cost=0.6412, reader_cost=0.0029 | ETA 03:45:29 2020-11-03 17:45:03 [INFO] [TRAIN] epoch=159, iter=59000/80000, loss=0.1611, lr=0.003008, batch_cost=0.5755, reader_cost=0.0021 | ETA 03:21:25 2020-11-03 17:46:02 [INFO] [TRAIN] epoch=159, iter=59100/80000, loss=0.1570, lr=0.002995, batch_cost=0.5893, reader_cost=0.0021 | ETA 03:25:15 2020-11-03 17:46:57 [INFO] [TRAIN] epoch=160, iter=59200/80000, loss=0.1411, lr=0.002982, batch_cost=0.5527, reader_cost=0.0096 | ETA 03:11:36 2020-11-03 17:47:51 [INFO] [TRAIN] epoch=160, iter=59300/80000, loss=0.1501, lr=0.002969, batch_cost=0.5377, reader_cost=0.0015 | ETA 03:05:30 2020-11-03 17:48:45 [INFO] [TRAIN] epoch=160, iter=59400/80000, loss=0.1511, lr=0.002956, batch_cost=0.5387, reader_cost=0.0017 | ETA 03:04:57 2020-11-03 17:49:39 [INFO] [TRAIN] epoch=160, iter=59500/80000, loss=0.1696, lr=0.002943, batch_cost=0.5409, reader_cost=0.0012 | ETA 03:04:47 2020-11-03 17:50:34 [INFO] [TRAIN] epoch=161, iter=59600/80000, loss=0.1602, lr=0.002931, batch_cost=0.5536, reader_cost=0.0106 | ETA 03:08:12 2020-11-03 17:51:28 [INFO] [TRAIN] epoch=161, iter=59700/80000, loss=0.1528, lr=0.002918, batch_cost=0.5388, reader_cost=0.0018 | ETA 03:02:18 2020-11-03 17:52:22 [INFO] [TRAIN] epoch=161, iter=59800/80000, loss=0.1724, lr=0.002905, batch_cost=0.5373, reader_cost=0.0021 | ETA 03:00:53 2020-11-03 17:53:17 [INFO] [TRAIN] epoch=162, iter=59900/80000, loss=0.1626, lr=0.002892, batch_cost=0.5519, reader_cost=0.0108 | ETA 03:04:53 2020-11-03 17:54:11 [INFO] [TRAIN] epoch=162, iter=60000/80000, loss=0.1571, lr=0.002879, batch_cost=0.5409, reader_cost=0.0014 | ETA 03:00:18 2020-11-03 17:55:05 [INFO] [TRAIN] epoch=162, iter=60100/80000, loss=0.1735, lr=0.002866, batch_cost=0.5411, reader_cost=0.0011 | ETA 02:59:28 2020-11-03 17:55:59 [INFO] [TRAIN] epoch=162, iter=60200/80000, loss=0.1457, lr=0.002853, batch_cost=0.5396, reader_cost=0.0021 | ETA 02:58:04 2020-11-03 17:56:54 [INFO] [TRAIN] epoch=163, iter=60300/80000, loss=0.1533, lr=0.002840, batch_cost=0.5491, reader_cost=0.0107 | ETA 03:00:16 2020-11-03 17:57:47 [INFO] [TRAIN] epoch=163, iter=60400/80000, loss=0.1534, lr=0.002827, batch_cost=0.5360, reader_cost=0.0006 | ETA 02:55:05 2020-11-03 17:58:41 [INFO] [TRAIN] epoch=163, iter=60500/80000, loss=0.1664, lr=0.002814, batch_cost=0.5397, reader_cost=0.0020 | ETA 02:55:23 2020-11-03 17:59:35 [INFO] [TRAIN] epoch=163, iter=60600/80000, loss=0.1611, lr=0.002801, batch_cost=0.5381, reader_cost=0.0012 | ETA 02:53:59 2020-11-03 18:00:31 [INFO] [TRAIN] epoch=164, iter=60700/80000, loss=0.1532, lr=0.002788, batch_cost=0.5544, reader_cost=0.0093 | ETA 02:58:19 2020-11-03 18:01:25 [INFO] [TRAIN] epoch=164, iter=60800/80000, loss=0.1643, lr=0.002776, batch_cost=0.5394, reader_cost=0.0013 | ETA 02:52:35 2020-11-03 18:02:18 [INFO] [TRAIN] epoch=164, iter=60900/80000, loss=0.1638, lr=0.002763, batch_cost=0.5368, reader_cost=0.0012 | ETA 02:50:52 2020-11-03 18:03:12 [INFO] [TRAIN] epoch=164, iter=61000/80000, loss=0.1571, lr=0.002750, batch_cost=0.5382, reader_cost=0.0010 | ETA 02:50:26 2020-11-03 18:04:07 [INFO] [TRAIN] epoch=165, iter=61100/80000, loss=0.1478, lr=0.002737, batch_cost=0.5522, reader_cost=0.0099 | ETA 02:53:56 2020-11-03 18:05:01 [INFO] [TRAIN] epoch=165, iter=61200/80000, loss=0.1670, lr=0.002724, batch_cost=0.5383, reader_cost=0.0010 | ETA 02:48:39 2020-11-03 18:05:55 [INFO] [TRAIN] epoch=165, iter=61300/80000, loss=0.1558, lr=0.002711, batch_cost=0.5383, reader_cost=0.0014 | ETA 02:47:45 2020-11-03 18:06:50 [INFO] [TRAIN] epoch=166, iter=61400/80000, loss=0.1709, lr=0.002698, batch_cost=0.5511, reader_cost=0.0109 | ETA 02:50:51 2020-11-03 18:07:45 [INFO] [TRAIN] epoch=166, iter=61500/80000, loss=0.1521, lr=0.002685, batch_cost=0.5436, reader_cost=0.0009 | ETA 02:47:36 2020-11-03 18:08:38 [INFO] [TRAIN] epoch=166, iter=61600/80000, loss=0.1598, lr=0.002672, batch_cost=0.5388, reader_cost=0.0016 | ETA 02:45:13 2020-11-03 18:09:33 [INFO] [TRAIN] epoch=166, iter=61700/80000, loss=0.1888, lr=0.002659, batch_cost=0.5427, reader_cost=0.0025 | ETA 02:45:30 2020-11-03 18:10:28 [INFO] [TRAIN] epoch=167, iter=61800/80000, loss=0.1608, lr=0.002646, batch_cost=0.5558, reader_cost=0.0117 | ETA 02:48:36 2020-11-03 18:11:23 [INFO] [TRAIN] epoch=167, iter=61900/80000, loss=0.1542, lr=0.002633, batch_cost=0.5445, reader_cost=0.0027 | ETA 02:44:16 2020-11-03 18:12:17 [INFO] [TRAIN] epoch=167, iter=62000/80000, loss=0.1622, lr=0.002619, batch_cost=0.5388, reader_cost=0.0018 | ETA 02:41:38 2020-11-03 18:13:10 [INFO] [TRAIN] epoch=167, iter=62100/80000, loss=0.1580, lr=0.002606, batch_cost=0.5338, reader_cost=0.0006 | ETA 02:39:15 2020-11-03 18:14:04 [INFO] [TRAIN] epoch=168, iter=62200/80000, loss=0.1495, lr=0.002593, batch_cost=0.5420, reader_cost=0.0079 | ETA 02:40:47 2020-11-03 18:14:58 [INFO] [TRAIN] epoch=168, iter=62300/80000, loss=0.1721, lr=0.002580, batch_cost=0.5349, reader_cost=0.0011 | ETA 02:37:48 2020-11-03 18:15:52 [INFO] [TRAIN] epoch=168, iter=62400/80000, loss=0.1819, lr=0.002567, batch_cost=0.5415, reader_cost=0.0015 | ETA 02:38:50 2020-11-03 18:16:47 [INFO] [TRAIN] epoch=169, iter=62500/80000, loss=0.1809, lr=0.002554, batch_cost=0.5486, reader_cost=0.0114 | ETA 02:40:00 2020-11-03 18:17:41 [INFO] [TRAIN] epoch=169, iter=62600/80000, loss=0.1621, lr=0.002541, batch_cost=0.5413, reader_cost=0.0011 | ETA 02:36:59 2020-11-03 18:18:35 [INFO] [TRAIN] epoch=169, iter=62700/80000, loss=0.1623, lr=0.002528, batch_cost=0.5442, reader_cost=0.0017 | ETA 02:36:55 2020-11-03 18:19:29 [INFO] [TRAIN] epoch=169, iter=62800/80000, loss=0.1429, lr=0.002515, batch_cost=0.5397, reader_cost=0.0012 | ETA 02:34:43 2020-11-03 18:20:24 [INFO] [TRAIN] epoch=170, iter=62900/80000, loss=0.1468, lr=0.002502, batch_cost=0.5483, reader_cost=0.0089 | ETA 02:36:15 2020-11-03 18:21:18 [INFO] [TRAIN] epoch=170, iter=63000/80000, loss=0.1614, lr=0.002489, batch_cost=0.5423, reader_cost=0.0009 | ETA 02:33:39 2020-11-03 18:22:13 [INFO] [TRAIN] epoch=170, iter=63100/80000, loss=0.1477, lr=0.002476, batch_cost=0.5433, reader_cost=0.0013 | ETA 02:33:01 2020-11-03 18:23:07 [INFO] [TRAIN] epoch=170, iter=63200/80000, loss=0.1567, lr=0.002462, batch_cost=0.5414, reader_cost=0.0018 | ETA 02:31:36 2020-11-03 18:24:02 [INFO] [TRAIN] epoch=171, iter=63300/80000, loss=0.1579, lr=0.002449, batch_cost=0.5523, reader_cost=0.0101 | ETA 02:33:42 2020-11-03 18:24:56 [INFO] [TRAIN] epoch=171, iter=63400/80000, loss=0.1517, lr=0.002436, batch_cost=0.5423, reader_cost=0.0019 | ETA 02:30:02 2020-11-03 18:25:50 [INFO] [TRAIN] epoch=171, iter=63500/80000, loss=0.1518, lr=0.002423, batch_cost=0.5385, reader_cost=0.0014 | ETA 02:28:05 2020-11-03 18:26:44 [INFO] [TRAIN] epoch=171, iter=63600/80000, loss=0.1947, lr=0.002410, batch_cost=0.5410, reader_cost=0.0017 | ETA 02:27:52 2020-11-03 18:27:39 [INFO] [TRAIN] epoch=172, iter=63700/80000, loss=0.1643, lr=0.002397, batch_cost=0.5492, reader_cost=0.0100 | ETA 02:29:12 2020-11-03 18:28:33 [INFO] [TRAIN] epoch=172, iter=63800/80000, loss=0.1390, lr=0.002383, batch_cost=0.5428, reader_cost=0.0017 | ETA 02:26:33 2020-11-03 18:29:27 [INFO] [TRAIN] epoch=172, iter=63900/80000, loss=0.1814, lr=0.002370, batch_cost=0.5394, reader_cost=0.0013 | ETA 02:24:45 2020-11-03 18:30:22 [INFO] [TRAIN] epoch=173, iter=64000/80000, loss=0.1481, lr=0.002357, batch_cost=0.5485, reader_cost=0.0095 | ETA 02:26:16 2020-11-03 18:30:30 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-03 18:35:58 [INFO] [EVAL] #Images=500 mIoU=0.7781 Acc=0.9604 Kappa=0.9485 2020-11-03 18:35:58 [INFO] [EVAL] Category IoU: [0.9791 0.8394 0.9241 0.5456 0.6286 0.6507 0.7254 0.7976 0.9251 0.649 0.9434 0.8278 0.6341 0.952 0.8024 0.8534 0.6487 0.6761 0.7816] 2020-11-03 18:35:58 [INFO] [EVAL] Category Acc: [0.9887 0.9277 0.9545 0.8713 0.8641 0.783 0.8475 0.9082 0.9553 0.8472 0.9593 0.898 0.7607 0.9718 0.9235 0.9016 0.9843 0.8299 0.8622] 2020-11-03 18:35:58 [INFO] [EVAL] The model with the best validation mIoU (0.7803) was saved at iter 56000. 2020-11-03 18:36:53 [INFO] [TRAIN] epoch=173, iter=64100/80000, loss=0.1507, lr=0.002344, batch_cost=0.5443, reader_cost=0.0029 | ETA 02:24:14 2020-11-03 18:37:47 [INFO] [TRAIN] epoch=173, iter=64200/80000, loss=0.1977, lr=0.002331, batch_cost=0.5441, reader_cost=0.0023 | ETA 02:23:16 2020-11-03 18:38:41 [INFO] [TRAIN] epoch=173, iter=64300/80000, loss=0.1513, lr=0.002317, batch_cost=0.5398, reader_cost=0.0022 | ETA 02:21:15 2020-11-03 18:39:36 [INFO] [TRAIN] epoch=174, iter=64400/80000, loss=0.1423, lr=0.002304, batch_cost=0.5494, reader_cost=0.0110 | ETA 02:22:50 2020-11-03 18:40:30 [INFO] [TRAIN] epoch=174, iter=64500/80000, loss=0.1548, lr=0.002291, batch_cost=0.5392, reader_cost=0.0015 | ETA 02:19:18 2020-11-03 18:41:24 [INFO] [TRAIN] epoch=174, iter=64600/80000, loss=0.1580, lr=0.002278, batch_cost=0.5399, reader_cost=0.0011 | ETA 02:18:34 2020-11-03 18:42:18 [INFO] [TRAIN] epoch=174, iter=64700/80000, loss=0.1592, lr=0.002264, batch_cost=0.5392, reader_cost=0.0007 | ETA 02:17:30 2020-11-03 18:43:14 [INFO] [TRAIN] epoch=175, iter=64800/80000, loss=0.1637, lr=0.002251, batch_cost=0.5552, reader_cost=0.0115 | ETA 02:20:38 2020-11-03 18:44:08 [INFO] [TRAIN] epoch=175, iter=64900/80000, loss=0.1587, lr=0.002238, batch_cost=0.5417, reader_cost=0.0013 | ETA 02:16:19 2020-11-03 18:45:02 [INFO] [TRAIN] epoch=175, iter=65000/80000, loss=0.1559, lr=0.002225, batch_cost=0.5382, reader_cost=0.0007 | ETA 02:14:33 2020-11-03 18:45:55 [INFO] [TRAIN] epoch=175, iter=65100/80000, loss=0.1560, lr=0.002211, batch_cost=0.5367, reader_cost=0.0008 | ETA 02:13:16 2020-11-03 18:46:50 [INFO] [TRAIN] epoch=176, iter=65200/80000, loss=0.1493, lr=0.002198, batch_cost=0.5490, reader_cost=0.0113 | ETA 02:15:24 2020-11-03 18:47:44 [INFO] [TRAIN] epoch=176, iter=65300/80000, loss=0.1530, lr=0.002185, batch_cost=0.5353, reader_cost=0.0014 | ETA 02:11:09 2020-11-03 18:48:38 [INFO] [TRAIN] epoch=176, iter=65400/80000, loss=0.1689, lr=0.002171, batch_cost=0.5391, reader_cost=0.0018 | ETA 02:11:10 2020-11-03 18:49:33 [INFO] [TRAIN] epoch=177, iter=65500/80000, loss=0.1602, lr=0.002158, batch_cost=0.5527, reader_cost=0.0113 | ETA 02:13:33 2020-11-03 18:50:27 [INFO] [TRAIN] epoch=177, iter=65600/80000, loss=0.1489, lr=0.002145, batch_cost=0.5393, reader_cost=0.0014 | ETA 02:09:25 2020-11-03 18:51:21 [INFO] [TRAIN] epoch=177, iter=65700/80000, loss=0.1601, lr=0.002131, batch_cost=0.5438, reader_cost=0.0013 | ETA 02:09:36 2020-11-03 18:52:15 [INFO] [TRAIN] epoch=177, iter=65800/80000, loss=0.1613, lr=0.002118, batch_cost=0.5410, reader_cost=0.0017 | ETA 02:08:02 2020-11-03 18:53:10 [INFO] [TRAIN] epoch=178, iter=65900/80000, loss=0.1490, lr=0.002105, batch_cost=0.5510, reader_cost=0.0105 | ETA 02:09:28 2020-11-03 18:54:05 [INFO] [TRAIN] epoch=178, iter=66000/80000, loss=0.1746, lr=0.002091, batch_cost=0.5435, reader_cost=0.0011 | ETA 02:06:48 2020-11-03 18:54:59 [INFO] [TRAIN] epoch=178, iter=66100/80000, loss=0.1586, lr=0.002078, batch_cost=0.5400, reader_cost=0.0018 | ETA 02:05:05 2020-11-03 18:55:53 [INFO] [TRAIN] epoch=178, iter=66200/80000, loss=0.1704, lr=0.002064, batch_cost=0.5417, reader_cost=0.0023 | ETA 02:04:35 2020-11-03 18:56:48 [INFO] [TRAIN] epoch=179, iter=66300/80000, loss=0.1629, lr=0.002051, batch_cost=0.5512, reader_cost=0.0113 | ETA 02:05:50 2020-11-03 18:57:42 [INFO] [TRAIN] epoch=179, iter=66400/80000, loss=0.1635, lr=0.002038, batch_cost=0.5402, reader_cost=0.0018 | ETA 02:02:26 2020-11-03 18:58:36 [INFO] [TRAIN] epoch=179, iter=66500/80000, loss=0.1508, lr=0.002024, batch_cost=0.5408, reader_cost=0.0012 | ETA 02:01:41 2020-11-03 18:59:31 [INFO] [TRAIN] epoch=180, iter=66600/80000, loss=0.1635, lr=0.002011, batch_cost=0.5490, reader_cost=0.0101 | ETA 02:02:36 2020-11-03 19:00:25 [INFO] [TRAIN] epoch=180, iter=66700/80000, loss=0.1592, lr=0.001997, batch_cost=0.5367, reader_cost=0.0009 | ETA 01:58:58 2020-11-03 19:01:19 [INFO] [TRAIN] epoch=180, iter=66800/80000, loss=0.1602, lr=0.001984, batch_cost=0.5389, reader_cost=0.0006 | ETA 01:58:33 2020-11-03 19:02:13 [INFO] [TRAIN] epoch=180, iter=66900/80000, loss=0.1764, lr=0.001970, batch_cost=0.5430, reader_cost=0.0008 | ETA 01:58:32 2020-11-03 19:03:07 [INFO] [TRAIN] epoch=181, iter=67000/80000, loss=0.1470, lr=0.001957, batch_cost=0.5434, reader_cost=0.0103 | ETA 01:57:44 2020-11-03 19:04:00 [INFO] [TRAIN] epoch=181, iter=67100/80000, loss=0.1348, lr=0.001944, batch_cost=0.5297, reader_cost=0.0005 | ETA 01:53:52 2020-11-03 19:04:54 [INFO] [TRAIN] epoch=181, iter=67200/80000, loss=0.1510, lr=0.001930, batch_cost=0.5324, reader_cost=0.0003 | ETA 01:53:35 2020-11-03 19:05:47 [INFO] [TRAIN] epoch=181, iter=67300/80000, loss=0.1440, lr=0.001917, batch_cost=0.5392, reader_cost=0.0011 | ETA 01:54:07 2020-11-03 19:06:42 [INFO] [TRAIN] epoch=182, iter=67400/80000, loss=0.1654, lr=0.001903, batch_cost=0.5497, reader_cost=0.0098 | ETA 01:55:26 2020-11-03 19:07:36 [INFO] [TRAIN] epoch=182, iter=67500/80000, loss=0.1574, lr=0.001889, batch_cost=0.5402, reader_cost=0.0014 | ETA 01:52:32 2020-11-03 19:08:30 [INFO] [TRAIN] epoch=182, iter=67600/80000, loss=0.1560, lr=0.001876, batch_cost=0.5376, reader_cost=0.0015 | ETA 01:51:06 2020-11-03 19:09:24 [INFO] [TRAIN] epoch=182, iter=67700/80000, loss=0.1487, lr=0.001862, batch_cost=0.5400, reader_cost=0.0011 | ETA 01:50:41 2020-11-03 19:10:19 [INFO] [TRAIN] epoch=183, iter=67800/80000, loss=0.1316, lr=0.001849, batch_cost=0.5509, reader_cost=0.0094 | ETA 01:52:01 2020-11-03 19:11:13 [INFO] [TRAIN] epoch=183, iter=67900/80000, loss=0.1469, lr=0.001835, batch_cost=0.5417, reader_cost=0.0009 | ETA 01:49:14 2020-11-03 19:12:07 [INFO] [TRAIN] epoch=183, iter=68000/80000, loss=0.1744, lr=0.001822, batch_cost=0.5385, reader_cost=0.0014 | ETA 01:47:42 2020-11-03 19:13:02 [INFO] [TRAIN] epoch=184, iter=68100/80000, loss=0.1354, lr=0.001808, batch_cost=0.5495, reader_cost=0.0110 | ETA 01:48:59 2020-11-03 19:13:56 [INFO] [TRAIN] epoch=184, iter=68200/80000, loss=0.1497, lr=0.001794, batch_cost=0.5395, reader_cost=0.0013 | ETA 01:46:06 2020-11-03 19:14:50 [INFO] [TRAIN] epoch=184, iter=68300/80000, loss=0.1439, lr=0.001781, batch_cost=0.5382, reader_cost=0.0007 | ETA 01:44:56 2020-11-03 19:15:44 [INFO] [TRAIN] epoch=184, iter=68400/80000, loss=0.1687, lr=0.001767, batch_cost=0.5390, reader_cost=0.0012 | ETA 01:44:12 2020-11-03 19:16:39 [INFO] [TRAIN] epoch=185, iter=68500/80000, loss=0.1384, lr=0.001754, batch_cost=0.5517, reader_cost=0.0100 | ETA 01:45:44 2020-11-03 19:17:33 [INFO] [TRAIN] epoch=185, iter=68600/80000, loss=0.1521, lr=0.001740, batch_cost=0.5435, reader_cost=0.0007 | ETA 01:43:15 2020-11-03 19:18:27 [INFO] [TRAIN] epoch=185, iter=68700/80000, loss=0.1552, lr=0.001726, batch_cost=0.5394, reader_cost=0.0016 | ETA 01:41:35 2020-11-03 19:19:21 [INFO] [TRAIN] epoch=185, iter=68800/80000, loss=0.1708, lr=0.001713, batch_cost=0.5379, reader_cost=0.0013 | ETA 01:40:24 2020-11-03 19:20:16 [INFO] [TRAIN] epoch=186, iter=68900/80000, loss=0.1451, lr=0.001699, batch_cost=0.5499, reader_cost=0.0095 | ETA 01:41:43 2020-11-03 19:21:10 [INFO] [TRAIN] epoch=186, iter=69000/80000, loss=0.1588, lr=0.001685, batch_cost=0.5386, reader_cost=0.0009 | ETA 01:38:44 2020-11-03 19:22:04 [INFO] [TRAIN] epoch=186, iter=69100/80000, loss=0.1596, lr=0.001672, batch_cost=0.5382, reader_cost=0.0012 | ETA 01:37:45 2020-11-03 19:22:59 [INFO] [TRAIN] epoch=187, iter=69200/80000, loss=0.1530, lr=0.001658, batch_cost=0.5516, reader_cost=0.0122 | ETA 01:39:17 2020-11-03 19:23:53 [INFO] [TRAIN] epoch=187, iter=69300/80000, loss=0.1390, lr=0.001644, batch_cost=0.5408, reader_cost=0.0014 | ETA 01:36:26 2020-11-03 19:24:47 [INFO] [TRAIN] epoch=187, iter=69400/80000, loss=0.1740, lr=0.001630, batch_cost=0.5384, reader_cost=0.0014 | ETA 01:35:07 2020-11-03 19:25:41 [INFO] [TRAIN] epoch=187, iter=69500/80000, loss=0.1559, lr=0.001617, batch_cost=0.5396, reader_cost=0.0008 | ETA 01:34:25 2020-11-03 19:26:36 [INFO] [TRAIN] epoch=188, iter=69600/80000, loss=0.1406, lr=0.001603, batch_cost=0.5535, reader_cost=0.0092 | ETA 01:35:56 2020-11-03 19:27:31 [INFO] [TRAIN] epoch=188, iter=69700/80000, loss=0.1491, lr=0.001589, batch_cost=0.5428, reader_cost=0.0016 | ETA 01:33:10 2020-11-03 19:28:25 [INFO] [TRAIN] epoch=188, iter=69800/80000, loss=0.1585, lr=0.001575, batch_cost=0.5434, reader_cost=0.0020 | ETA 01:32:22 2020-11-03 19:29:19 [INFO] [TRAIN] epoch=188, iter=69900/80000, loss=0.1477, lr=0.001561, batch_cost=0.5411, reader_cost=0.0015 | ETA 01:31:05 2020-11-03 19:30:14 [INFO] [TRAIN] epoch=189, iter=70000/80000, loss=0.1467, lr=0.001548, batch_cost=0.5479, reader_cost=0.0098 | ETA 01:31:18 2020-11-03 19:31:08 [INFO] [TRAIN] epoch=189, iter=70100/80000, loss=0.1334, lr=0.001534, batch_cost=0.5383, reader_cost=0.0012 | ETA 01:28:49 2020-11-03 19:32:01 [INFO] [TRAIN] epoch=189, iter=70200/80000, loss=0.1499, lr=0.001520, batch_cost=0.5375, reader_cost=0.0009 | ETA 01:27:47 2020-11-03 19:32:55 [INFO] [TRAIN] epoch=189, iter=70300/80000, loss=0.1680, lr=0.001506, batch_cost=0.5394, reader_cost=0.0011 | ETA 01:27:12 2020-11-03 19:33:51 [INFO] [TRAIN] epoch=190, iter=70400/80000, loss=0.1485, lr=0.001492, batch_cost=0.5558, reader_cost=0.0105 | ETA 01:28:55 2020-11-03 19:34:45 [INFO] [TRAIN] epoch=190, iter=70500/80000, loss=0.1504, lr=0.001478, batch_cost=0.5412, reader_cost=0.0007 | ETA 01:25:40 2020-11-03 19:35:39 [INFO] [TRAIN] epoch=190, iter=70600/80000, loss=0.1736, lr=0.001464, batch_cost=0.5398, reader_cost=0.0026 | ETA 01:24:34 2020-11-03 19:36:34 [INFO] [TRAIN] epoch=191, iter=70700/80000, loss=0.1412, lr=0.001450, batch_cost=0.5513, reader_cost=0.0103 | ETA 01:25:27 2020-11-03 19:37:28 [INFO] [TRAIN] epoch=191, iter=70800/80000, loss=0.1488, lr=0.001436, batch_cost=0.5428, reader_cost=0.0008 | ETA 01:23:13 2020-11-03 19:38:22 [INFO] [TRAIN] epoch=191, iter=70900/80000, loss=0.1667, lr=0.001422, batch_cost=0.5393, reader_cost=0.0006 | ETA 01:21:47 2020-11-03 19:39:16 [INFO] [TRAIN] epoch=191, iter=71000/80000, loss=0.1652, lr=0.001408, batch_cost=0.5367, reader_cost=0.0007 | ETA 01:20:30 2020-11-03 19:40:11 [INFO] [TRAIN] epoch=192, iter=71100/80000, loss=0.1394, lr=0.001394, batch_cost=0.5541, reader_cost=0.0097 | ETA 01:22:11 2020-11-03 19:41:05 [INFO] [TRAIN] epoch=192, iter=71200/80000, loss=0.1401, lr=0.001380, batch_cost=0.5389, reader_cost=0.0014 | ETA 01:19:02 2020-11-03 19:41:59 [INFO] [TRAIN] epoch=192, iter=71300/80000, loss=0.1585, lr=0.001366, batch_cost=0.5380, reader_cost=0.0018 | ETA 01:18:00 2020-11-03 19:42:53 [INFO] [TRAIN] epoch=192, iter=71400/80000, loss=0.1648, lr=0.001352, batch_cost=0.5413, reader_cost=0.0018 | ETA 01:17:35 2020-11-03 19:43:48 [INFO] [TRAIN] epoch=193, iter=71500/80000, loss=0.1644, lr=0.001338, batch_cost=0.5505, reader_cost=0.0103 | ETA 01:17:59 2020-11-03 19:44:42 [INFO] [TRAIN] epoch=193, iter=71600/80000, loss=0.1612, lr=0.001324, batch_cost=0.5378, reader_cost=0.0016 | ETA 01:15:17 2020-11-03 19:45:36 [INFO] [TRAIN] epoch=193, iter=71700/80000, loss=0.1557, lr=0.001310, batch_cost=0.5415, reader_cost=0.0010 | ETA 01:14:54 2020-11-03 19:46:31 [INFO] [TRAIN] epoch=194, iter=71800/80000, loss=0.1555, lr=0.001296, batch_cost=0.5481, reader_cost=0.0096 | ETA 01:14:54 2020-11-03 19:47:25 [INFO] [TRAIN] epoch=194, iter=71900/80000, loss=0.1544, lr=0.001282, batch_cost=0.5392, reader_cost=0.0008 | ETA 01:12:47 2020-11-03 19:48:19 [INFO] [TRAIN] epoch=194, iter=72000/80000, loss=0.1430, lr=0.001268, batch_cost=0.5397, reader_cost=0.0019 | ETA 01:11:57 2020-11-03 19:48:28 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-03 19:53:55 [INFO] [EVAL] #Images=500 mIoU=0.7858 Acc=0.9612 Kappa=0.9496 2020-11-03 19:53:55 [INFO] [EVAL] Category IoU: [0.9816 0.8524 0.9241 0.4431 0.6496 0.6544 0.7244 0.8032 0.9241 0.6399 0.9468 0.8283 0.6285 0.9516 0.7996 0.8975 0.7988 0.6932 0.7899] 2020-11-03 19:53:55 [INFO] [EVAL] Category Acc: [0.9895 0.9329 0.9564 0.8662 0.8535 0.7973 0.8549 0.906 0.9499 0.8547 0.9675 0.8859 0.8105 0.9704 0.9495 0.9504 0.9717 0.8477 0.8673] 2020-11-03 19:53:59 [INFO] [EVAL] The model with the best validation mIoU (0.7858) was saved at iter 72000. 2020-11-03 19:54:52 [INFO] [TRAIN] epoch=194, iter=72100/80000, loss=0.1374, lr=0.001254, batch_cost=0.5349, reader_cost=0.0007 | ETA 01:10:25 2020-11-03 19:55:48 [INFO] [TRAIN] epoch=195, iter=72200/80000, loss=0.1499, lr=0.001239, batch_cost=0.5546, reader_cost=0.0115 | ETA 01:12:05 2020-11-03 19:56:42 [INFO] [TRAIN] epoch=195, iter=72300/80000, loss=0.1579, lr=0.001225, batch_cost=0.5445, reader_cost=0.0020 | ETA 01:09:52 2020-11-03 19:57:36 [INFO] [TRAIN] epoch=195, iter=72400/80000, loss=0.1504, lr=0.001211, batch_cost=0.5412, reader_cost=0.0021 | ETA 01:08:33 2020-11-03 19:58:30 [INFO] [TRAIN] epoch=195, iter=72500/80000, loss=0.1468, lr=0.001197, batch_cost=0.5416, reader_cost=0.0021 | ETA 01:07:42 2020-11-03 19:59:26 [INFO] [TRAIN] epoch=196, iter=72600/80000, loss=0.1503, lr=0.001183, batch_cost=0.5546, reader_cost=0.0119 | ETA 01:08:23 2020-11-03 20:00:20 [INFO] [TRAIN] epoch=196, iter=72700/80000, loss=0.1345, lr=0.001168, batch_cost=0.5422, reader_cost=0.0017 | ETA 01:05:58 2020-11-03 20:01:14 [INFO] [TRAIN] epoch=196, iter=72800/80000, loss=0.1448, lr=0.001154, batch_cost=0.5430, reader_cost=0.0019 | ETA 01:05:09 2020-11-03 20:02:09 [INFO] [TRAIN] epoch=196, iter=72900/80000, loss=0.1557, lr=0.001140, batch_cost=0.5432, reader_cost=0.0022 | ETA 01:04:16 2020-11-03 20:03:04 [INFO] [TRAIN] epoch=197, iter=73000/80000, loss=0.1514, lr=0.001125, batch_cost=0.5540, reader_cost=0.0110 | ETA 01:04:37 2020-11-03 20:03:59 [INFO] [TRAIN] epoch=197, iter=73100/80000, loss=0.1667, lr=0.001111, batch_cost=0.5445, reader_cost=0.0019 | ETA 01:02:37 2020-11-03 20:04:53 [INFO] [TRAIN] epoch=197, iter=73200/80000, loss=0.1584, lr=0.001097, batch_cost=0.5432, reader_cost=0.0014 | ETA 01:01:33 2020-11-03 20:05:48 [INFO] [TRAIN] epoch=198, iter=73300/80000, loss=0.1394, lr=0.001082, batch_cost=0.5529, reader_cost=0.0104 | ETA 01:01:44 2020-11-03 20:06:42 [INFO] [TRAIN] epoch=198, iter=73400/80000, loss=0.1383, lr=0.001068, batch_cost=0.5420, reader_cost=0.0018 | ETA 00:59:37 2020-11-03 20:07:37 [INFO] [TRAIN] epoch=198, iter=73500/80000, loss=0.1546, lr=0.001053, batch_cost=0.5440, reader_cost=0.0023 | ETA 00:58:56 2020-11-03 20:08:31 [INFO] [TRAIN] epoch=198, iter=73600/80000, loss=0.1657, lr=0.001039, batch_cost=0.5379, reader_cost=0.0012 | ETA 00:57:22 2020-11-03 20:09:26 [INFO] [TRAIN] epoch=199, iter=73700/80000, loss=0.1335, lr=0.001025, batch_cost=0.5508, reader_cost=0.0110 | ETA 00:57:50 2020-11-03 20:10:20 [INFO] [TRAIN] epoch=199, iter=73800/80000, loss=0.1459, lr=0.001010, batch_cost=0.5417, reader_cost=0.0018 | ETA 00:55:58 2020-11-03 20:11:14 [INFO] [TRAIN] epoch=199, iter=73900/80000, loss=0.1410, lr=0.000995, batch_cost=0.5382, reader_cost=0.0010 | ETA 00:54:43 2020-11-03 20:12:08 [INFO] [TRAIN] epoch=199, iter=74000/80000, loss=0.1529, lr=0.000981, batch_cost=0.5399, reader_cost=0.0017 | ETA 00:53:59 2020-11-03 20:13:03 [INFO] [TRAIN] epoch=200, iter=74100/80000, loss=0.1430, lr=0.000966, batch_cost=0.5502, reader_cost=0.0104 | ETA 00:54:05 2020-11-03 20:13:57 [INFO] [TRAIN] epoch=200, iter=74200/80000, loss=0.1394, lr=0.000952, batch_cost=0.5421, reader_cost=0.0017 | ETA 00:52:24 2020-11-03 20:14:51 [INFO] [TRAIN] epoch=200, iter=74300/80000, loss=0.1449, lr=0.000937, batch_cost=0.5374, reader_cost=0.0005 | ETA 00:51:02 2020-11-03 20:15:44 [INFO] [TRAIN] epoch=200, iter=74400/80000, loss=0.1681, lr=0.000922, batch_cost=0.5345, reader_cost=0.0010 | ETA 00:49:53 2020-11-03 20:16:39 [INFO] [TRAIN] epoch=201, iter=74500/80000, loss=0.1518, lr=0.000908, batch_cost=0.5531, reader_cost=0.0103 | ETA 00:50:41 2020-11-03 20:17:33 [INFO] [TRAIN] epoch=201, iter=74600/80000, loss=0.1531, lr=0.000893, batch_cost=0.5397, reader_cost=0.0011 | ETA 00:48:34 2020-11-03 20:18:27 [INFO] [TRAIN] epoch=201, iter=74700/80000, loss=0.1610, lr=0.000878, batch_cost=0.5405, reader_cost=0.0009 | ETA 00:47:44 2020-11-03 20:19:23 [INFO] [TRAIN] epoch=202, iter=74800/80000, loss=0.1404, lr=0.000864, batch_cost=0.5531, reader_cost=0.0096 | ETA 00:47:56 2020-11-03 20:20:17 [INFO] [TRAIN] epoch=202, iter=74900/80000, loss=0.1538, lr=0.000849, batch_cost=0.5419, reader_cost=0.0011 | ETA 00:46:03 2020-11-03 20:21:11 [INFO] [TRAIN] epoch=202, iter=75000/80000, loss=0.1567, lr=0.000834, batch_cost=0.5406, reader_cost=0.0011 | ETA 00:45:03 2020-11-03 20:22:05 [INFO] [TRAIN] epoch=202, iter=75100/80000, loss=0.1510, lr=0.000819, batch_cost=0.5401, reader_cost=0.0011 | ETA 00:44:06 2020-11-03 20:23:00 [INFO] [TRAIN] epoch=203, iter=75200/80000, loss=0.1476, lr=0.000804, batch_cost=0.5497, reader_cost=0.0107 | ETA 00:43:58 2020-11-03 20:23:54 [INFO] [TRAIN] epoch=203, iter=75300/80000, loss=0.1309, lr=0.000789, batch_cost=0.5429, reader_cost=0.0015 | ETA 00:42:31 2020-11-03 20:24:48 [INFO] [TRAIN] epoch=203, iter=75400/80000, loss=0.1515, lr=0.000774, batch_cost=0.5392, reader_cost=0.0008 | ETA 00:41:20 2020-11-03 20:25:42 [INFO] [TRAIN] epoch=203, iter=75500/80000, loss=0.1741, lr=0.000759, batch_cost=0.5394, reader_cost=0.0007 | ETA 00:40:27 2020-11-03 20:26:37 [INFO] [TRAIN] epoch=204, iter=75600/80000, loss=0.1497, lr=0.000744, batch_cost=0.5517, reader_cost=0.0095 | ETA 00:40:27 2020-11-03 20:27:31 [INFO] [TRAIN] epoch=204, iter=75700/80000, loss=0.1463, lr=0.000729, batch_cost=0.5383, reader_cost=0.0011 | ETA 00:38:34 2020-11-03 20:28:25 [INFO] [TRAIN] epoch=204, iter=75800/80000, loss=0.1536, lr=0.000714, batch_cost=0.5349, reader_cost=0.0006 | ETA 00:37:26 2020-11-03 20:29:20 [INFO] [TRAIN] epoch=205, iter=75900/80000, loss=0.1501, lr=0.000699, batch_cost=0.5537, reader_cost=0.0103 | ETA 00:37:50 2020-11-03 20:30:14 [INFO] [TRAIN] epoch=205, iter=76000/80000, loss=0.1373, lr=0.000684, batch_cost=0.5390, reader_cost=0.0007 | ETA 00:35:55 2020-11-03 20:31:08 [INFO] [TRAIN] epoch=205, iter=76100/80000, loss=0.1592, lr=0.000669, batch_cost=0.5374, reader_cost=0.0009 | ETA 00:34:55 2020-11-03 20:32:01 [INFO] [TRAIN] epoch=205, iter=76200/80000, loss=0.1493, lr=0.000654, batch_cost=0.5376, reader_cost=0.0008 | ETA 00:34:02 2020-11-03 20:32:56 [INFO] [TRAIN] epoch=206, iter=76300/80000, loss=0.1514, lr=0.000638, batch_cost=0.5509, reader_cost=0.0098 | ETA 00:33:58 2020-11-03 20:33:50 [INFO] [TRAIN] epoch=206, iter=76400/80000, loss=0.1358, lr=0.000623, batch_cost=0.5390, reader_cost=0.0015 | ETA 00:32:20 2020-11-03 20:34:44 [INFO] [TRAIN] epoch=206, iter=76500/80000, loss=0.1482, lr=0.000608, batch_cost=0.5390, reader_cost=0.0009 | ETA 00:31:26 2020-11-03 20:35:38 [INFO] [TRAIN] epoch=206, iter=76600/80000, loss=0.1652, lr=0.000592, batch_cost=0.5386, reader_cost=0.0007 | ETA 00:30:31 2020-11-03 20:36:33 [INFO] [TRAIN] epoch=207, iter=76700/80000, loss=0.1376, lr=0.000577, batch_cost=0.5456, reader_cost=0.0100 | ETA 00:30:00 2020-11-03 20:37:27 [INFO] [TRAIN] epoch=207, iter=76800/80000, loss=0.1465, lr=0.000561, batch_cost=0.5415, reader_cost=0.0022 | ETA 00:28:52 2020-11-03 20:38:21 [INFO] [TRAIN] epoch=207, iter=76900/80000, loss=0.1480, lr=0.000546, batch_cost=0.5414, reader_cost=0.0017 | ETA 00:27:58 2020-11-03 20:39:15 [INFO] [TRAIN] epoch=207, iter=77000/80000, loss=0.1574, lr=0.000530, batch_cost=0.5417, reader_cost=0.0013 | ETA 00:27:05 2020-11-03 20:40:10 [INFO] [TRAIN] epoch=208, iter=77100/80000, loss=0.1423, lr=0.000515, batch_cost=0.5526, reader_cost=0.0108 | ETA 00:26:42 2020-11-03 20:41:05 [INFO] [TRAIN] epoch=208, iter=77200/80000, loss=0.1436, lr=0.000499, batch_cost=0.5419, reader_cost=0.0019 | ETA 00:25:17 2020-11-03 20:41:59 [INFO] [TRAIN] epoch=208, iter=77300/80000, loss=0.1569, lr=0.000483, batch_cost=0.5400, reader_cost=0.0017 | ETA 00:24:17 2020-11-03 20:42:53 [INFO] [TRAIN] epoch=209, iter=77400/80000, loss=0.1361, lr=0.000468, batch_cost=0.5420, reader_cost=0.0085 | ETA 00:23:29 2020-11-03 20:43:46 [INFO] [TRAIN] epoch=209, iter=77500/80000, loss=0.1615, lr=0.000452, batch_cost=0.5330, reader_cost=0.0004 | ETA 00:22:12 2020-11-03 20:44:39 [INFO] [TRAIN] epoch=209, iter=77600/80000, loss=0.1757, lr=0.000436, batch_cost=0.5311, reader_cost=0.0005 | ETA 00:21:14 2020-11-03 20:45:33 [INFO] [TRAIN] epoch=209, iter=77700/80000, loss=0.1503, lr=0.000420, batch_cost=0.5409, reader_cost=0.0016 | ETA 00:20:43 2020-11-03 20:46:29 [INFO] [TRAIN] epoch=210, iter=77800/80000, loss=0.1399, lr=0.000404, batch_cost=0.5527, reader_cost=0.0101 | ETA 00:20:15 2020-11-03 20:47:22 [INFO] [TRAIN] epoch=210, iter=77900/80000, loss=0.1431, lr=0.000388, batch_cost=0.5396, reader_cost=0.0008 | ETA 00:18:53 2020-11-03 20:48:16 [INFO] [TRAIN] epoch=210, iter=78000/80000, loss=0.1711, lr=0.000371, batch_cost=0.5393, reader_cost=0.0008 | ETA 00:17:58 2020-11-03 20:49:11 [INFO] [TRAIN] epoch=210, iter=78100/80000, loss=0.1550, lr=0.000355, batch_cost=0.5428, reader_cost=0.0018 | ETA 00:17:11 2020-11-03 20:50:06 [INFO] [TRAIN] epoch=211, iter=78200/80000, loss=0.1551, lr=0.000339, batch_cost=0.5505, reader_cost=0.0111 | ETA 00:16:30 2020-11-03 20:50:59 [INFO] [TRAIN] epoch=211, iter=78300/80000, loss=0.1474, lr=0.000322, batch_cost=0.5368, reader_cost=0.0013 | ETA 00:15:12 2020-11-03 20:51:53 [INFO] [TRAIN] epoch=211, iter=78400/80000, loss=0.1451, lr=0.000306, batch_cost=0.5393, reader_cost=0.0015 | ETA 00:14:22 2020-11-03 20:52:48 [INFO] [TRAIN] epoch=212, iter=78500/80000, loss=0.1514, lr=0.000289, batch_cost=0.5477, reader_cost=0.0104 | ETA 00:13:41 2020-11-03 20:53:42 [INFO] [TRAIN] epoch=212, iter=78600/80000, loss=0.1401, lr=0.000272, batch_cost=0.5407, reader_cost=0.0016 | ETA 00:12:37 2020-11-03 20:54:36 [INFO] [TRAIN] epoch=212, iter=78700/80000, loss=0.1585, lr=0.000255, batch_cost=0.5407, reader_cost=0.0014 | ETA 00:11:42 2020-11-03 20:55:30 [INFO] [TRAIN] epoch=212, iter=78800/80000, loss=0.1607, lr=0.000238, batch_cost=0.5418, reader_cost=0.0015 | ETA 00:10:50 2020-11-03 20:56:26 [INFO] [TRAIN] epoch=213, iter=78900/80000, loss=0.1486, lr=0.000221, batch_cost=0.5512, reader_cost=0.0094 | ETA 00:10:06 2020-11-03 20:57:20 [INFO] [TRAIN] epoch=213, iter=79000/80000, loss=0.1550, lr=0.000204, batch_cost=0.5409, reader_cost=0.0017 | ETA 00:09:00 2020-11-03 20:58:14 [INFO] [TRAIN] epoch=213, iter=79100/80000, loss=0.1507, lr=0.000186, batch_cost=0.5394, reader_cost=0.0010 | ETA 00:08:05 2020-11-03 20:59:08 [INFO] [TRAIN] epoch=213, iter=79200/80000, loss=0.1628, lr=0.000169, batch_cost=0.5411, reader_cost=0.0017 | ETA 00:07:12 2020-11-03 21:00:03 [INFO] [TRAIN] epoch=214, iter=79300/80000, loss=0.1438, lr=0.000151, batch_cost=0.5530, reader_cost=0.0109 | ETA 00:06:27 2020-11-03 21:00:57 [INFO] [TRAIN] epoch=214, iter=79400/80000, loss=0.1464, lr=0.000132, batch_cost=0.5399, reader_cost=0.0008 | ETA 00:05:23 2020-11-03 21:01:51 [INFO] [TRAIN] epoch=214, iter=79500/80000, loss=0.1512, lr=0.000114, batch_cost=0.5420, reader_cost=0.0026 | ETA 00:04:30 2020-11-03 21:02:45 [INFO] [TRAIN] epoch=214, iter=79600/80000, loss=0.1528, lr=0.000095, batch_cost=0.5413, reader_cost=0.0019 | ETA 00:03:36 2020-11-03 21:03:40 [INFO] [TRAIN] epoch=215, iter=79700/80000, loss=0.1490, lr=0.000076, batch_cost=0.5501, reader_cost=0.0102 | ETA 00:02:45 2020-11-03 21:04:34 [INFO] [TRAIN] epoch=215, iter=79800/80000, loss=0.1321, lr=0.000056, batch_cost=0.5368, reader_cost=0.0008 | ETA 00:01:47 2020-11-03 21:05:28 [INFO] [TRAIN] epoch=215, iter=79900/80000, loss=0.1540, lr=0.000035, batch_cost=0.5399, reader_cost=0.0018 | ETA 00:00:53 2020-11-03 21:06:23 [INFO] [TRAIN] epoch=216, iter=80000/80000, loss=0.1492, lr=0.000010, batch_cost=0.5522, reader_cost=0.0114 | ETA 00:00:00 2020-11-03 21:06:31 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-03 21:11:49 [INFO] [EVAL] #Images=500 mIoU=0.7883 Acc=0.9607 Kappa=0.9490 2020-11-03 21:11:49 [INFO] [EVAL] Category IoU: [0.9793 0.8405 0.9245 0.5347 0.6343 0.6555 0.7268 0.8032 0.9252 0.6268 0.9467 0.826 0.6235 0.9514 0.7892 0.8945 0.8273 0.6794 0.788 ] 2020-11-03 21:11:49 [INFO] [EVAL] Category Acc: [0.9897 0.9217 0.9533 0.8801 0.8717 0.7953 0.8422 0.907 0.9532 0.8822 0.9666 0.8814 0.8062 0.9701 0.9528 0.9502 0.9644 0.8471 0.8736] 2020-11-03 21:11:53 [INFO] [EVAL] The model with the best validation mIoU (0.7883) was saved at iter 80000.