2020-10-31 01:26:04 [INFO] ------------Environment Information------------- platform: Linux-3.10.0_3-0-0-34-x86_64-with-centos-7.5.1804-Core Python: 3.7.9 (default, Aug 31 2020, 12:42:55) [GCC 7.3.0] Paddle compiled with cuda: True NVCC: Cuda compilation tools, release 10.2, V10.2.89 cudnn: 7.6 GPUs used: 4 CUDA_VISIBLE_DEVICES: 4,5,6,7 GPU: ['GPU 0: Tesla V100-SXM2-16GB', 'GPU 1: Tesla V100-SXM2-16GB', 'GPU 2: Tesla V100-SXM2-16GB', 'GPU 3: Tesla V100-SXM2-16GB', 'GPU 4: Tesla V100-SXM2-16GB', 'GPU 5: Tesla V100-SXM2-16GB', 'GPU 6: Tesla V100-SXM2-16GB', 'GPU 7: Tesla V100-SXM2-16GB'] GCC: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36) PaddlePaddle: 2.0.0-rc0 OpenCV: 4.1.0 ------------------------------------------------ 2020-10-31 01:26:04 [INFO] ---------------Config Information--------------- batch_size: 2 iters: 80000 learning_rate: decay: end_lr: 1.0e-05 power: 0.9 type: poly value: 0.01 loss: coef: - 1 - 0.4 types: - ignore_index: 255 type: CrossEntropyLoss model: backbone: output_stride: 8 pretrained: https://bj.bcebos.com/paddleseg/dygraph/resnet101_vd_ssld.tar.gz type: ResNet101_vd enable_auxiliary_loss: true gc_channels: 512 pretrained: null ratio: 0.25 type: GCNet optimizer: momentum: 0.9 type: sgd weight_decay: 4.0e-05 train_dataset: dataset_root: data/cityscapes mode: train transforms: - max_scale_factor: 2.0 min_scale_factor: 0.5 scale_step_size: 0.25 type: ResizeStepScaling - crop_size: - 1024 - 512 type: RandomPaddingCrop - type: RandomHorizontalFlip - brightness_range: 0.4 contrast_range: 0.4 saturation_range: 0.4 type: RandomDistort - type: Normalize type: Cityscapes val_dataset: dataset_root: data/cityscapes mode: val transforms: - type: Normalize type: Cityscapes ------------------------------------------------ 2020-10-31 01:26:09 [INFO] Loading pretrained model from https://bj.bcebos.com/paddleseg/dygraph/resnet101_vd_ssld.tar.gz 2020-10-31 01:26:10 [INFO] There are 530/530 variables loaded into ResNet_vd. 2020-10-31 01:27:29 [INFO] [TRAIN] epoch=1, iter=100/80000, loss=1.5680, lr=0.009989, batch_cost=0.7212, reader_cost=0.0138 | ETA 16:00:20 2020-10-31 01:28:37 [INFO] [TRAIN] epoch=1, iter=200/80000, loss=1.0895, lr=0.009978, batch_cost=0.6815, reader_cost=0.0008 | ETA 15:06:23 2020-10-31 01:29:45 [INFO] [TRAIN] epoch=1, iter=300/80000, loss=0.7422, lr=0.009966, batch_cost=0.6836, reader_cost=0.0007 | ETA 15:07:59 2020-10-31 01:30:54 [INFO] [TRAIN] epoch=2, iter=400/80000, loss=0.9358, lr=0.009955, batch_cost=0.6891, reader_cost=0.0101 | ETA 15:14:11 2020-10-31 01:32:03 [INFO] [TRAIN] epoch=2, iter=500/80000, loss=0.6425, lr=0.009944, batch_cost=0.6866, reader_cost=0.0006 | ETA 15:09:47 2020-10-31 01:33:11 [INFO] [TRAIN] epoch=2, iter=600/80000, loss=0.5909, lr=0.009933, batch_cost=0.6851, reader_cost=0.0003 | ETA 15:06:38 2020-10-31 01:34:19 [INFO] [TRAIN] epoch=2, iter=700/80000, loss=0.5032, lr=0.009921, batch_cost=0.6804, reader_cost=0.0002 | ETA 14:59:17 2020-10-31 01:35:28 [INFO] [TRAIN] epoch=3, iter=800/80000, loss=0.4903, lr=0.009910, batch_cost=0.6846, reader_cost=0.0095 | ETA 15:03:37 2020-10-31 01:36:36 [INFO] [TRAIN] epoch=3, iter=900/80000, loss=0.5859, lr=0.009899, batch_cost=0.6812, reader_cost=0.0002 | ETA 14:58:00 2020-10-31 01:37:44 [INFO] [TRAIN] epoch=3, iter=1000/80000, loss=0.5310, lr=0.009888, batch_cost=0.6814, reader_cost=0.0004 | ETA 14:57:10 2020-10-31 01:38:52 [INFO] [TRAIN] epoch=3, iter=1100/80000, loss=0.4767, lr=0.009876, batch_cost=0.6793, reader_cost=0.0002 | ETA 14:53:19 2020-10-31 01:40:01 [INFO] [TRAIN] epoch=4, iter=1200/80000, loss=0.9016, lr=0.009865, batch_cost=0.6890, reader_cost=0.0091 | ETA 15:04:53 2020-10-31 01:41:09 [INFO] [TRAIN] epoch=4, iter=1300/80000, loss=0.6138, lr=0.009854, batch_cost=0.6823, reader_cost=0.0004 | ETA 14:54:56 2020-10-31 01:42:17 [INFO] [TRAIN] epoch=4, iter=1400/80000, loss=0.5556, lr=0.009843, batch_cost=0.6767, reader_cost=0.0004 | ETA 14:46:30 2020-10-31 01:43:26 [INFO] [TRAIN] epoch=5, iter=1500/80000, loss=0.4871, lr=0.009831, batch_cost=0.6884, reader_cost=0.0096 | ETA 15:00:39 2020-10-31 01:44:34 [INFO] [TRAIN] epoch=5, iter=1600/80000, loss=0.4526, lr=0.009820, batch_cost=0.6820, reader_cost=0.0004 | ETA 14:51:05 2020-10-31 01:45:42 [INFO] [TRAIN] epoch=5, iter=1700/80000, loss=0.4847, lr=0.009809, batch_cost=0.6783, reader_cost=0.0003 | ETA 14:45:07 2020-10-31 01:46:50 [INFO] [TRAIN] epoch=5, iter=1800/80000, loss=0.4969, lr=0.009798, batch_cost=0.6797, reader_cost=0.0003 | ETA 14:45:49 2020-10-31 01:47:59 [INFO] [TRAIN] epoch=6, iter=1900/80000, loss=0.6032, lr=0.009786, batch_cost=0.6939, reader_cost=0.0097 | ETA 15:03:17 2020-10-31 01:49:07 [INFO] [TRAIN] epoch=6, iter=2000/80000, loss=0.4173, lr=0.009775, batch_cost=0.6745, reader_cost=0.0002 | ETA 14:36:49 2020-10-31 01:50:15 [INFO] [TRAIN] epoch=6, iter=2100/80000, loss=0.4137, lr=0.009764, batch_cost=0.6794, reader_cost=0.0002 | ETA 14:42:03 2020-10-31 01:51:22 [INFO] [TRAIN] epoch=6, iter=2200/80000, loss=0.3169, lr=0.009753, batch_cost=0.6785, reader_cost=0.0004 | ETA 14:39:50 2020-10-31 01:52:31 [INFO] [TRAIN] epoch=7, iter=2300/80000, loss=0.4240, lr=0.009741, batch_cost=0.6892, reader_cost=0.0091 | ETA 14:52:32 2020-10-31 01:53:39 [INFO] [TRAIN] epoch=7, iter=2400/80000, loss=0.4210, lr=0.009730, batch_cost=0.6777, reader_cost=0.0002 | ETA 14:36:28 2020-10-31 01:54:47 [INFO] [TRAIN] epoch=7, iter=2500/80000, loss=0.4270, lr=0.009719, batch_cost=0.6754, reader_cost=0.0001 | ETA 14:32:22 2020-10-31 01:55:54 [INFO] [TRAIN] epoch=7, iter=2600/80000, loss=0.4016, lr=0.009707, batch_cost=0.6734, reader_cost=0.0001 | ETA 14:28:40 2020-10-31 01:57:03 [INFO] [TRAIN] epoch=8, iter=2700/80000, loss=0.3278, lr=0.009696, batch_cost=0.6890, reader_cost=0.0100 | ETA 14:47:37 2020-10-31 01:58:11 [INFO] [TRAIN] epoch=8, iter=2800/80000, loss=0.4067, lr=0.009685, batch_cost=0.6804, reader_cost=0.0001 | ETA 14:35:25 2020-10-31 01:59:19 [INFO] [TRAIN] epoch=8, iter=2900/80000, loss=0.3208, lr=0.009674, batch_cost=0.6836, reader_cost=0.0002 | ETA 14:38:28 2020-10-31 02:00:29 [INFO] [TRAIN] epoch=9, iter=3000/80000, loss=0.3178, lr=0.009662, batch_cost=0.6945, reader_cost=0.0100 | ETA 14:51:18 2020-10-31 02:01:38 [INFO] [TRAIN] epoch=9, iter=3100/80000, loss=0.3927, lr=0.009651, batch_cost=0.6911, reader_cost=0.0005 | ETA 14:45:45 2020-10-31 02:02:47 [INFO] [TRAIN] epoch=9, iter=3200/80000, loss=0.4036, lr=0.009640, batch_cost=0.6890, reader_cost=0.0002 | ETA 14:41:51 2020-10-31 02:03:56 [INFO] [TRAIN] epoch=9, iter=3300/80000, loss=0.3412, lr=0.009628, batch_cost=0.6889, reader_cost=0.0003 | ETA 14:40:39 2020-10-31 02:05:05 [INFO] [TRAIN] epoch=10, iter=3400/80000, loss=0.3490, lr=0.009617, batch_cost=0.6913, reader_cost=0.0093 | ETA 14:42:29 2020-10-31 02:06:13 [INFO] [TRAIN] epoch=10, iter=3500/80000, loss=0.3279, lr=0.009606, batch_cost=0.6852, reader_cost=0.0008 | ETA 14:33:41 2020-10-31 02:07:22 [INFO] [TRAIN] epoch=10, iter=3600/80000, loss=0.4012, lr=0.009595, batch_cost=0.6833, reader_cost=0.0006 | ETA 14:30:03 2020-10-31 02:08:30 [INFO] [TRAIN] epoch=10, iter=3700/80000, loss=0.3628, lr=0.009583, batch_cost=0.6873, reader_cost=0.0006 | ETA 14:33:59 2020-10-31 02:09:40 [INFO] [TRAIN] epoch=11, iter=3800/80000, loss=0.3702, lr=0.009572, batch_cost=0.6919, reader_cost=0.0104 | ETA 14:38:45 2020-10-31 02:10:48 [INFO] [TRAIN] epoch=11, iter=3900/80000, loss=0.3354, lr=0.009561, batch_cost=0.6826, reader_cost=0.0002 | ETA 14:25:49 2020-10-31 02:11:56 [INFO] [TRAIN] epoch=11, iter=4000/80000, loss=0.3327, lr=0.009549, batch_cost=0.6818, reader_cost=0.0002 | ETA 14:23:33 2020-10-31 02:13:05 [INFO] [TRAIN] epoch=12, iter=4100/80000, loss=0.3680, lr=0.009538, batch_cost=0.6915, reader_cost=0.0101 | ETA 14:34:44 2020-10-31 02:14:14 [INFO] [TRAIN] epoch=12, iter=4200/80000, loss=0.3716, lr=0.009527, batch_cost=0.6870, reader_cost=0.0002 | ETA 14:27:56 2020-10-31 02:15:23 [INFO] [TRAIN] epoch=12, iter=4300/80000, loss=0.4069, lr=0.009516, batch_cost=0.6893, reader_cost=0.0004 | ETA 14:29:41 2020-10-31 02:16:31 [INFO] [TRAIN] epoch=12, iter=4400/80000, loss=0.3107, lr=0.009504, batch_cost=0.6827, reader_cost=0.0002 | ETA 14:20:15 2020-10-31 02:17:41 [INFO] [TRAIN] epoch=13, iter=4500/80000, loss=0.3170, lr=0.009493, batch_cost=0.6965, reader_cost=0.0088 | ETA 14:36:22 2020-10-31 02:18:49 [INFO] [TRAIN] epoch=13, iter=4600/80000, loss=0.2921, lr=0.009482, batch_cost=0.6877, reader_cost=0.0007 | ETA 14:24:09 2020-10-31 02:19:58 [INFO] [TRAIN] epoch=13, iter=4700/80000, loss=0.3090, lr=0.009470, batch_cost=0.6879, reader_cost=0.0007 | ETA 14:23:17 2020-10-31 02:21:07 [INFO] [TRAIN] epoch=13, iter=4800/80000, loss=0.2986, lr=0.009459, batch_cost=0.6828, reader_cost=0.0005 | ETA 14:15:45 2020-10-31 02:22:15 [INFO] [TRAIN] epoch=14, iter=4900/80000, loss=0.3400, lr=0.009448, batch_cost=0.6882, reader_cost=0.0092 | ETA 14:21:26 2020-10-31 02:23:23 [INFO] [TRAIN] epoch=14, iter=5000/80000, loss=0.4004, lr=0.009436, batch_cost=0.6737, reader_cost=0.0001 | ETA 14:02:08 2020-10-31 02:24:31 [INFO] [TRAIN] epoch=14, iter=5100/80000, loss=0.3140, lr=0.009425, batch_cost=0.6872, reader_cost=0.0006 | ETA 14:17:49 2020-10-31 02:25:40 [INFO] [TRAIN] epoch=14, iter=5200/80000, loss=0.3170, lr=0.009414, batch_cost=0.6846, reader_cost=0.0003 | ETA 14:13:25 2020-10-31 02:26:49 [INFO] [TRAIN] epoch=15, iter=5300/80000, loss=0.3242, lr=0.009402, batch_cost=0.6925, reader_cost=0.0092 | ETA 14:22:13 2020-10-31 02:27:58 [INFO] [TRAIN] epoch=15, iter=5400/80000, loss=0.3363, lr=0.009391, batch_cost=0.6880, reader_cost=0.0005 | ETA 14:15:22 2020-10-31 02:29:06 [INFO] [TRAIN] epoch=15, iter=5500/80000, loss=0.3252, lr=0.009380, batch_cost=0.6842, reader_cost=0.0003 | ETA 14:09:36 2020-10-31 02:30:16 [INFO] [TRAIN] epoch=16, iter=5600/80000, loss=0.2854, lr=0.009368, batch_cost=0.6923, reader_cost=0.0098 | ETA 14:18:27 2020-10-31 02:31:23 [INFO] [TRAIN] epoch=16, iter=5700/80000, loss=0.2823, lr=0.009357, batch_cost=0.6784, reader_cost=0.0006 | ETA 14:00:06 2020-10-31 02:32:32 [INFO] [TRAIN] epoch=16, iter=5800/80000, loss=0.2991, lr=0.009346, batch_cost=0.6831, reader_cost=0.0004 | ETA 14:04:48 2020-10-31 02:33:40 [INFO] [TRAIN] epoch=16, iter=5900/80000, loss=0.2727, lr=0.009335, batch_cost=0.6846, reader_cost=0.0008 | ETA 14:05:31 2020-10-31 02:34:49 [INFO] [TRAIN] epoch=17, iter=6000/80000, loss=0.3253, lr=0.009323, batch_cost=0.6887, reader_cost=0.0087 | ETA 14:09:26 2020-10-31 02:35:57 [INFO] [TRAIN] epoch=17, iter=6100/80000, loss=0.3288, lr=0.009312, batch_cost=0.6738, reader_cost=0.0002 | ETA 13:49:52 2020-10-31 02:37:05 [INFO] [TRAIN] epoch=17, iter=6200/80000, loss=0.2755, lr=0.009301, batch_cost=0.6842, reader_cost=0.0002 | ETA 14:01:32 2020-10-31 02:38:14 [INFO] [TRAIN] epoch=17, iter=6300/80000, loss=0.2963, lr=0.009289, batch_cost=0.6880, reader_cost=0.0001 | ETA 14:05:02 2020-10-31 02:39:23 [INFO] [TRAIN] epoch=18, iter=6400/80000, loss=0.3185, lr=0.009278, batch_cost=0.6920, reader_cost=0.0088 | ETA 14:08:51 2020-10-31 02:40:31 [INFO] [TRAIN] epoch=18, iter=6500/80000, loss=0.2874, lr=0.009267, batch_cost=0.6852, reader_cost=0.0002 | ETA 13:59:21 2020-10-31 02:41:40 [INFO] [TRAIN] epoch=18, iter=6600/80000, loss=0.3089, lr=0.009255, batch_cost=0.6828, reader_cost=0.0002 | ETA 13:55:18 2020-10-31 02:42:49 [INFO] [TRAIN] epoch=19, iter=6700/80000, loss=0.2961, lr=0.009244, batch_cost=0.6883, reader_cost=0.0102 | ETA 14:00:52 2020-10-31 02:43:56 [INFO] [TRAIN] epoch=19, iter=6800/80000, loss=0.3073, lr=0.009233, batch_cost=0.6781, reader_cost=0.0003 | ETA 13:47:19 2020-10-31 02:45:05 [INFO] [TRAIN] epoch=19, iter=6900/80000, loss=0.3046, lr=0.009221, batch_cost=0.6874, reader_cost=0.0007 | ETA 13:57:26 2020-10-31 02:46:13 [INFO] [TRAIN] epoch=19, iter=7000/80000, loss=0.2813, lr=0.009210, batch_cost=0.6776, reader_cost=0.0004 | ETA 13:44:22 2020-10-31 02:47:22 [INFO] [TRAIN] epoch=20, iter=7100/80000, loss=0.3049, lr=0.009199, batch_cost=0.6879, reader_cost=0.0093 | ETA 13:55:47 2020-10-31 02:48:31 [INFO] [TRAIN] epoch=20, iter=7200/80000, loss=0.3935, lr=0.009187, batch_cost=0.6887, reader_cost=0.0003 | ETA 13:55:38 2020-10-31 02:49:39 [INFO] [TRAIN] epoch=20, iter=7300/80000, loss=0.2891, lr=0.009176, batch_cost=0.6800, reader_cost=0.0003 | ETA 13:43:58 2020-10-31 02:50:47 [INFO] [TRAIN] epoch=20, iter=7400/80000, loss=0.2924, lr=0.009164, batch_cost=0.6815, reader_cost=0.0002 | ETA 13:44:34 2020-10-31 02:51:56 [INFO] [TRAIN] epoch=21, iter=7500/80000, loss=0.3301, lr=0.009153, batch_cost=0.6909, reader_cost=0.0106 | ETA 13:54:50 2020-10-31 02:53:03 [INFO] [TRAIN] epoch=21, iter=7600/80000, loss=0.3786, lr=0.009142, batch_cost=0.6757, reader_cost=0.0001 | ETA 13:35:19 2020-10-31 02:54:12 [INFO] [TRAIN] epoch=21, iter=7700/80000, loss=0.2593, lr=0.009130, batch_cost=0.6853, reader_cost=0.0006 | ETA 13:45:48 2020-10-31 02:55:20 [INFO] [TRAIN] epoch=21, iter=7800/80000, loss=0.2478, lr=0.009119, batch_cost=0.6847, reader_cost=0.0003 | ETA 13:43:54 2020-10-31 02:56:29 [INFO] [TRAIN] epoch=22, iter=7900/80000, loss=0.2903, lr=0.009108, batch_cost=0.6911, reader_cost=0.0101 | ETA 13:50:25 2020-10-31 02:57:38 [INFO] [TRAIN] epoch=22, iter=8000/80000, loss=0.2506, lr=0.009096, batch_cost=0.6836, reader_cost=0.0006 | ETA 13:40:17 2020-10-31 02:57:45 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-10-31 03:03:29 [INFO] [EVAL] #Images=500 mIoU=0.7033 Acc=0.9442 Kappa=0.9273 2020-10-31 03:03:29 [INFO] [EVAL] Category IoU: [0.961 0.7194 0.9046 0.402 0.5119 0.5959 0.6629 0.7455 0.9153 0.5432 0.9369 0.7753 0.5347 0.9308 0.6706 0.669 0.5963 0.5597 0.7285] 2020-10-31 03:03:29 [INFO] [EVAL] Category Acc: [0.9696 0.9119 0.9454 0.5713 0.7292 0.791 0.7747 0.8869 0.9475 0.815 0.9667 0.8842 0.6301 0.9608 0.8566 0.9316 0.7585 0.7587 0.8142] 2020-10-31 03:03:33 [INFO] [EVAL] The model with the best validation mIoU (0.7033) was saved at iter 8000. 2020-10-31 03:04:40 [INFO] [TRAIN] epoch=22, iter=8100/80000, loss=0.2339, lr=0.009085, batch_cost=0.6742, reader_cost=0.0004 | ETA 13:27:57 2020-10-31 03:05:49 [INFO] [TRAIN] epoch=23, iter=8200/80000, loss=0.4832, lr=0.009074, batch_cost=0.6834, reader_cost=0.0089 | ETA 13:37:46 2020-10-31 03:06:57 [INFO] [TRAIN] epoch=23, iter=8300/80000, loss=0.3061, lr=0.009062, batch_cost=0.6810, reader_cost=0.0006 | ETA 13:33:51 2020-10-31 03:08:05 [INFO] [TRAIN] epoch=23, iter=8400/80000, loss=0.2885, lr=0.009051, batch_cost=0.6776, reader_cost=0.0003 | ETA 13:28:33 2020-10-31 03:09:13 [INFO] [TRAIN] epoch=23, iter=8500/80000, loss=0.3451, lr=0.009040, batch_cost=0.6829, reader_cost=0.0005 | ETA 13:33:44 2020-10-31 03:10:22 [INFO] [TRAIN] epoch=24, iter=8600/80000, loss=0.3260, lr=0.009028, batch_cost=0.6920, reader_cost=0.0085 | ETA 13:43:29 2020-10-31 03:11:30 [INFO] [TRAIN] epoch=24, iter=8700/80000, loss=0.3335, lr=0.009017, batch_cost=0.6769, reader_cost=0.0001 | ETA 13:24:21 2020-10-31 03:12:38 [INFO] [TRAIN] epoch=24, iter=8800/80000, loss=0.2773, lr=0.009005, batch_cost=0.6795, reader_cost=0.0007 | ETA 13:26:20 2020-10-31 03:13:46 [INFO] [TRAIN] epoch=24, iter=8900/80000, loss=0.2450, lr=0.008994, batch_cost=0.6780, reader_cost=0.0002 | ETA 13:23:26 2020-10-31 03:14:54 [INFO] [TRAIN] epoch=25, iter=9000/80000, loss=0.2561, lr=0.008983, batch_cost=0.6880, reader_cost=0.0101 | ETA 13:34:09 2020-10-31 03:16:03 [INFO] [TRAIN] epoch=25, iter=9100/80000, loss=0.2790, lr=0.008971, batch_cost=0.6832, reader_cost=0.0005 | ETA 13:27:16 2020-10-31 03:17:11 [INFO] [TRAIN] epoch=25, iter=9200/80000, loss=0.2575, lr=0.008960, batch_cost=0.6820, reader_cost=0.0002 | ETA 13:24:49 2020-10-31 03:18:19 [INFO] [TRAIN] epoch=25, iter=9300/80000, loss=0.2534, lr=0.008949, batch_cost=0.6826, reader_cost=0.0002 | ETA 13:24:18 2020-10-31 03:19:28 [INFO] [TRAIN] epoch=26, iter=9400/80000, loss=0.2839, lr=0.008937, batch_cost=0.6914, reader_cost=0.0089 | ETA 13:33:32 2020-10-31 03:20:36 [INFO] [TRAIN] epoch=26, iter=9500/80000, loss=0.2894, lr=0.008926, batch_cost=0.6794, reader_cost=0.0004 | ETA 13:18:16 2020-10-31 03:21:44 [INFO] [TRAIN] epoch=26, iter=9600/80000, loss=0.2404, lr=0.008914, batch_cost=0.6807, reader_cost=0.0005 | ETA 13:18:37 2020-10-31 03:22:53 [INFO] [TRAIN] epoch=27, iter=9700/80000, loss=0.2423, lr=0.008903, batch_cost=0.6901, reader_cost=0.0093 | ETA 13:28:36 2020-10-31 03:24:02 [INFO] [TRAIN] epoch=27, iter=9800/80000, loss=0.2519, lr=0.008892, batch_cost=0.6818, reader_cost=0.0002 | ETA 13:17:42 2020-10-31 03:25:10 [INFO] [TRAIN] epoch=27, iter=9900/80000, loss=0.2450, lr=0.008880, batch_cost=0.6802, reader_cost=0.0006 | ETA 13:14:38 2020-10-31 03:26:18 [INFO] [TRAIN] epoch=27, iter=10000/80000, loss=0.2678, lr=0.008869, batch_cost=0.6816, reader_cost=0.0007 | ETA 13:15:12 2020-10-31 03:27:26 [INFO] [TRAIN] epoch=28, iter=10100/80000, loss=0.3345, lr=0.008857, batch_cost=0.6872, reader_cost=0.0085 | ETA 13:20:35 2020-10-31 03:28:34 [INFO] [TRAIN] epoch=28, iter=10200/80000, loss=0.2500, lr=0.008846, batch_cost=0.6799, reader_cost=0.0002 | ETA 13:10:54 2020-10-31 03:29:43 [INFO] [TRAIN] epoch=28, iter=10300/80000, loss=0.3080, lr=0.008835, batch_cost=0.6819, reader_cost=0.0007 | ETA 13:12:07 2020-10-31 03:30:50 [INFO] [TRAIN] epoch=28, iter=10400/80000, loss=0.2561, lr=0.008823, batch_cost=0.6772, reader_cost=0.0002 | ETA 13:05:33 2020-10-31 03:32:00 [INFO] [TRAIN] epoch=29, iter=10500/80000, loss=0.2652, lr=0.008812, batch_cost=0.6931, reader_cost=0.0100 | ETA 13:22:46 2020-10-31 03:33:08 [INFO] [TRAIN] epoch=29, iter=10600/80000, loss=0.2425, lr=0.008801, batch_cost=0.6841, reader_cost=0.0003 | ETA 13:11:14 2020-10-31 03:34:16 [INFO] [TRAIN] epoch=29, iter=10700/80000, loss=0.3285, lr=0.008789, batch_cost=0.6834, reader_cost=0.0003 | ETA 13:09:22 2020-10-31 03:35:26 [INFO] [TRAIN] epoch=30, iter=10800/80000, loss=0.3481, lr=0.008778, batch_cost=0.6962, reader_cost=0.0100 | ETA 13:22:59 2020-10-31 03:36:34 [INFO] [TRAIN] epoch=30, iter=10900/80000, loss=0.2552, lr=0.008766, batch_cost=0.6827, reader_cost=0.0002 | ETA 13:06:12 2020-10-31 03:37:42 [INFO] [TRAIN] epoch=30, iter=11000/80000, loss=0.2944, lr=0.008755, batch_cost=0.6818, reader_cost=0.0002 | ETA 13:04:04 2020-10-31 03:38:51 [INFO] [TRAIN] epoch=30, iter=11100/80000, loss=0.2571, lr=0.008743, batch_cost=0.6823, reader_cost=0.0002 | ETA 13:03:33 2020-10-31 03:39:59 [INFO] [TRAIN] epoch=31, iter=11200/80000, loss=0.3079, lr=0.008732, batch_cost=0.6858, reader_cost=0.0105 | ETA 13:06:21 2020-10-31 03:41:06 [INFO] [TRAIN] epoch=31, iter=11300/80000, loss=0.2715, lr=0.008721, batch_cost=0.6685, reader_cost=0.0002 | ETA 12:45:26 2020-10-31 03:42:13 [INFO] [TRAIN] epoch=31, iter=11400/80000, loss=0.3002, lr=0.008709, batch_cost=0.6683, reader_cost=0.0002 | ETA 12:44:05 2020-10-31 03:43:20 [INFO] [TRAIN] epoch=31, iter=11500/80000, loss=0.2463, lr=0.008698, batch_cost=0.6701, reader_cost=0.0002 | ETA 12:45:01 2020-10-31 03:44:28 [INFO] [TRAIN] epoch=32, iter=11600/80000, loss=0.2394, lr=0.008686, batch_cost=0.6787, reader_cost=0.0087 | ETA 12:53:40 2020-10-31 03:45:35 [INFO] [TRAIN] epoch=32, iter=11700/80000, loss=0.2458, lr=0.008675, batch_cost=0.6725, reader_cost=0.0002 | ETA 12:45:29 2020-10-31 03:46:43 [INFO] [TRAIN] epoch=32, iter=11800/80000, loss=0.2456, lr=0.008664, batch_cost=0.6814, reader_cost=0.0003 | ETA 12:54:30 2020-10-31 03:47:52 [INFO] [TRAIN] epoch=32, iter=11900/80000, loss=0.2771, lr=0.008652, batch_cost=0.6850, reader_cost=0.0002 | ETA 12:57:27 2020-10-31 03:49:01 [INFO] [TRAIN] epoch=33, iter=12000/80000, loss=0.2813, lr=0.008641, batch_cost=0.6919, reader_cost=0.0100 | ETA 13:04:11 2020-10-31 03:50:09 [INFO] [TRAIN] epoch=33, iter=12100/80000, loss=0.2818, lr=0.008629, batch_cost=0.6830, reader_cost=0.0002 | ETA 12:52:55 2020-10-31 03:51:17 [INFO] [TRAIN] epoch=33, iter=12200/80000, loss=0.2262, lr=0.008618, batch_cost=0.6787, reader_cost=0.0007 | ETA 12:46:53 2020-10-31 03:52:26 [INFO] [TRAIN] epoch=34, iter=12300/80000, loss=0.2542, lr=0.008606, batch_cost=0.6842, reader_cost=0.0082 | ETA 12:51:58 2020-10-31 03:53:33 [INFO] [TRAIN] epoch=34, iter=12400/80000, loss=0.2296, lr=0.008595, batch_cost=0.6793, reader_cost=0.0002 | ETA 12:45:18 2020-10-31 03:54:42 [INFO] [TRAIN] epoch=34, iter=12500/80000, loss=0.2124, lr=0.008584, batch_cost=0.6838, reader_cost=0.0002 | ETA 12:49:15 2020-10-31 03:55:50 [INFO] [TRAIN] epoch=34, iter=12600/80000, loss=0.2350, lr=0.008572, batch_cost=0.6782, reader_cost=0.0002 | ETA 12:41:50 2020-10-31 03:56:59 [INFO] [TRAIN] epoch=35, iter=12700/80000, loss=0.2398, lr=0.008561, batch_cost=0.6899, reader_cost=0.0087 | ETA 12:53:48 2020-10-31 03:58:06 [INFO] [TRAIN] epoch=35, iter=12800/80000, loss=0.2388, lr=0.008549, batch_cost=0.6771, reader_cost=0.0002 | ETA 12:38:23 2020-10-31 03:59:14 [INFO] [TRAIN] epoch=35, iter=12900/80000, loss=0.2111, lr=0.008538, batch_cost=0.6745, reader_cost=0.0002 | ETA 12:34:20 2020-10-31 04:00:22 [INFO] [TRAIN] epoch=35, iter=13000/80000, loss=0.2352, lr=0.008526, batch_cost=0.6795, reader_cost=0.0005 | ETA 12:38:47 2020-10-31 04:01:30 [INFO] [TRAIN] epoch=36, iter=13100/80000, loss=0.2452, lr=0.008515, batch_cost=0.6871, reader_cost=0.0085 | ETA 12:46:08 2020-10-31 04:02:39 [INFO] [TRAIN] epoch=36, iter=13200/80000, loss=0.2875, lr=0.008504, batch_cost=0.6834, reader_cost=0.0005 | ETA 12:40:53 2020-10-31 04:03:47 [INFO] [TRAIN] epoch=36, iter=13300/80000, loss=0.2688, lr=0.008492, batch_cost=0.6797, reader_cost=0.0006 | ETA 12:35:38 2020-10-31 04:04:55 [INFO] [TRAIN] epoch=37, iter=13400/80000, loss=0.2522, lr=0.008481, batch_cost=0.6832, reader_cost=0.0084 | ETA 12:38:24 2020-10-31 04:06:02 [INFO] [TRAIN] epoch=37, iter=13500/80000, loss=0.2417, lr=0.008469, batch_cost=0.6729, reader_cost=0.0001 | ETA 12:25:48 2020-10-31 04:07:11 [INFO] [TRAIN] epoch=37, iter=13600/80000, loss=0.2952, lr=0.008458, batch_cost=0.6813, reader_cost=0.0002 | ETA 12:33:58 2020-10-31 04:08:19 [INFO] [TRAIN] epoch=37, iter=13700/80000, loss=0.2761, lr=0.008446, batch_cost=0.6883, reader_cost=0.0005 | ETA 12:40:36 2020-10-31 04:09:29 [INFO] [TRAIN] epoch=38, iter=13800/80000, loss=0.2720, lr=0.008435, batch_cost=0.6929, reader_cost=0.0091 | ETA 12:44:32 2020-10-31 04:10:37 [INFO] [TRAIN] epoch=38, iter=13900/80000, loss=0.2470, lr=0.008423, batch_cost=0.6799, reader_cost=0.0005 | ETA 12:29:02 2020-10-31 04:11:45 [INFO] [TRAIN] epoch=38, iter=14000/80000, loss=0.2582, lr=0.008412, batch_cost=0.6813, reader_cost=0.0003 | ETA 12:29:28 2020-10-31 04:12:53 [INFO] [TRAIN] epoch=38, iter=14100/80000, loss=0.2356, lr=0.008400, batch_cost=0.6834, reader_cost=0.0002 | ETA 12:30:33 2020-10-31 04:14:03 [INFO] [TRAIN] epoch=39, iter=14200/80000, loss=0.2522, lr=0.008389, batch_cost=0.6936, reader_cost=0.0091 | ETA 12:40:36 2020-10-31 04:15:10 [INFO] [TRAIN] epoch=39, iter=14300/80000, loss=0.2479, lr=0.008378, batch_cost=0.6758, reader_cost=0.0002 | ETA 12:19:59 2020-10-31 04:16:18 [INFO] [TRAIN] epoch=39, iter=14400/80000, loss=0.2503, lr=0.008366, batch_cost=0.6777, reader_cost=0.0004 | ETA 12:20:57 2020-10-31 04:17:25 [INFO] [TRAIN] epoch=39, iter=14500/80000, loss=0.2571, lr=0.008355, batch_cost=0.6760, reader_cost=0.0002 | ETA 12:17:55 2020-10-31 04:18:35 [INFO] [TRAIN] epoch=40, iter=14600/80000, loss=0.2555, lr=0.008343, batch_cost=0.6929, reader_cost=0.0095 | ETA 12:35:16 2020-10-31 04:19:43 [INFO] [TRAIN] epoch=40, iter=14700/80000, loss=0.2423, lr=0.008332, batch_cost=0.6787, reader_cost=0.0002 | ETA 12:18:36 2020-10-31 04:20:51 [INFO] [TRAIN] epoch=40, iter=14800/80000, loss=0.2718, lr=0.008320, batch_cost=0.6810, reader_cost=0.0002 | ETA 12:20:02 2020-10-31 04:21:59 [INFO] [TRAIN] epoch=41, iter=14900/80000, loss=0.2121, lr=0.008309, batch_cost=0.6864, reader_cost=0.0084 | ETA 12:24:42 2020-10-31 04:23:08 [INFO] [TRAIN] epoch=41, iter=15000/80000, loss=0.2760, lr=0.008297, batch_cost=0.6831, reader_cost=0.0003 | ETA 12:19:59 2020-10-31 04:24:15 [INFO] [TRAIN] epoch=41, iter=15100/80000, loss=0.2223, lr=0.008286, batch_cost=0.6749, reader_cost=0.0002 | ETA 12:09:58 2020-10-31 04:25:23 [INFO] [TRAIN] epoch=41, iter=15200/80000, loss=0.2451, lr=0.008274, batch_cost=0.6806, reader_cost=0.0003 | ETA 12:15:01 2020-10-31 04:26:32 [INFO] [TRAIN] epoch=42, iter=15300/80000, loss=0.2065, lr=0.008263, batch_cost=0.6875, reader_cost=0.0094 | ETA 12:21:21 2020-10-31 04:27:40 [INFO] [TRAIN] epoch=42, iter=15400/80000, loss=0.2326, lr=0.008251, batch_cost=0.6788, reader_cost=0.0004 | ETA 12:10:53 2020-10-31 04:28:48 [INFO] [TRAIN] epoch=42, iter=15500/80000, loss=0.2384, lr=0.008240, batch_cost=0.6840, reader_cost=0.0007 | ETA 12:15:18 2020-10-31 04:29:56 [INFO] [TRAIN] epoch=42, iter=15600/80000, loss=0.2217, lr=0.008228, batch_cost=0.6794, reader_cost=0.0005 | ETA 12:09:14 2020-10-31 04:31:05 [INFO] [TRAIN] epoch=43, iter=15700/80000, loss=0.2126, lr=0.008217, batch_cost=0.6884, reader_cost=0.0101 | ETA 12:17:43 2020-10-31 04:32:14 [INFO] [TRAIN] epoch=43, iter=15800/80000, loss=0.2311, lr=0.008205, batch_cost=0.6911, reader_cost=0.0005 | ETA 12:19:27 2020-10-31 04:33:22 [INFO] [TRAIN] epoch=43, iter=15900/80000, loss=0.2137, lr=0.008194, batch_cost=0.6807, reader_cost=0.0002 | ETA 12:07:12 2020-10-31 04:34:31 [INFO] [TRAIN] epoch=44, iter=16000/80000, loss=0.2400, lr=0.008182, batch_cost=0.6907, reader_cost=0.0101 | ETA 12:16:43 2020-10-31 04:34:38 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-10-31 04:40:17 [INFO] [EVAL] #Images=500 mIoU=0.7447 Acc=0.9514 Kappa=0.9369 2020-10-31 04:40:17 [INFO] [EVAL] Category IoU: [0.9682 0.8132 0.9024 0.3668 0.5505 0.6121 0.6837 0.7748 0.9194 0.6076 0.9399 0.8135 0.6042 0.9454 0.8043 0.8503 0.6611 0.5711 0.7613] 2020-10-31 04:40:17 [INFO] [EVAL] Category Acc: [0.9881 0.8955 0.9372 0.743 0.714 0.813 0.7902 0.899 0.9481 0.8238 0.962 0.8858 0.7418 0.973 0.8957 0.9485 0.7998 0.7624 0.8532] 2020-10-31 04:40:21 [INFO] [EVAL] The model with the best validation mIoU (0.7447) was saved at iter 16000. 2020-10-31 04:41:28 [INFO] [TRAIN] epoch=44, iter=16100/80000, loss=0.2196, lr=0.008171, batch_cost=0.6693, reader_cost=0.0004 | ETA 11:52:50 2020-10-31 04:42:35 [INFO] [TRAIN] epoch=44, iter=16200/80000, loss=0.2540, lr=0.008159, batch_cost=0.6711, reader_cost=0.0005 | ETA 11:53:39 2020-10-31 04:43:43 [INFO] [TRAIN] epoch=44, iter=16300/80000, loss=0.2360, lr=0.008148, batch_cost=0.6765, reader_cost=0.0007 | ETA 11:58:14 2020-10-31 04:44:52 [INFO] [TRAIN] epoch=45, iter=16400/80000, loss=0.2010, lr=0.008136, batch_cost=0.6917, reader_cost=0.0088 | ETA 12:13:10 2020-10-31 04:46:00 [INFO] [TRAIN] epoch=45, iter=16500/80000, loss=0.2488, lr=0.008125, batch_cost=0.6812, reader_cost=0.0002 | ETA 12:00:53 2020-10-31 04:47:08 [INFO] [TRAIN] epoch=45, iter=16600/80000, loss=0.2163, lr=0.008113, batch_cost=0.6827, reader_cost=0.0002 | ETA 12:01:20 2020-10-31 04:48:17 [INFO] [TRAIN] epoch=45, iter=16700/80000, loss=0.2439, lr=0.008102, batch_cost=0.6864, reader_cost=0.0004 | ETA 12:04:07 2020-10-31 04:49:26 [INFO] [TRAIN] epoch=46, iter=16800/80000, loss=0.2174, lr=0.008090, batch_cost=0.6937, reader_cost=0.0088 | ETA 12:10:39 2020-10-31 04:50:34 [INFO] [TRAIN] epoch=46, iter=16900/80000, loss=0.2428, lr=0.008079, batch_cost=0.6779, reader_cost=0.0002 | ETA 11:52:53 2020-10-31 04:51:41 [INFO] [TRAIN] epoch=46, iter=17000/80000, loss=0.2647, lr=0.008067, batch_cost=0.6752, reader_cost=0.0001 | ETA 11:49:00 2020-10-31 04:52:49 [INFO] [TRAIN] epoch=46, iter=17100/80000, loss=0.2166, lr=0.008056, batch_cost=0.6748, reader_cost=0.0001 | ETA 11:47:23 2020-10-31 04:53:58 [INFO] [TRAIN] epoch=47, iter=17200/80000, loss=0.2382, lr=0.008044, batch_cost=0.6898, reader_cost=0.0083 | ETA 12:02:02 2020-10-31 04:55:06 [INFO] [TRAIN] epoch=47, iter=17300/80000, loss=0.2062, lr=0.008033, batch_cost=0.6796, reader_cost=0.0004 | ETA 11:50:08 2020-10-31 04:56:14 [INFO] [TRAIN] epoch=47, iter=17400/80000, loss=0.2191, lr=0.008021, batch_cost=0.6791, reader_cost=0.0003 | ETA 11:48:32 2020-10-31 04:57:23 [INFO] [TRAIN] epoch=48, iter=17500/80000, loss=0.2160, lr=0.008010, batch_cost=0.6918, reader_cost=0.0092 | ETA 12:00:37 2020-10-31 04:58:31 [INFO] [TRAIN] epoch=48, iter=17600/80000, loss=0.2401, lr=0.007998, batch_cost=0.6834, reader_cost=0.0004 | ETA 11:50:42 2020-10-31 04:59:40 [INFO] [TRAIN] epoch=48, iter=17700/80000, loss=0.2478, lr=0.007987, batch_cost=0.6822, reader_cost=0.0004 | ETA 11:48:20 2020-10-31 05:00:47 [INFO] [TRAIN] epoch=48, iter=17800/80000, loss=0.2092, lr=0.007975, batch_cost=0.6793, reader_cost=0.0002 | ETA 11:44:14 2020-10-31 05:01:57 [INFO] [TRAIN] epoch=49, iter=17900/80000, loss=0.2066, lr=0.007964, batch_cost=0.6910, reader_cost=0.0091 | ETA 11:55:12 2020-10-31 05:03:05 [INFO] [TRAIN] epoch=49, iter=18000/80000, loss=0.2077, lr=0.007952, batch_cost=0.6840, reader_cost=0.0006 | ETA 11:46:48 2020-10-31 05:04:12 [INFO] [TRAIN] epoch=49, iter=18100/80000, loss=0.2250, lr=0.007941, batch_cost=0.6734, reader_cost=0.0003 | ETA 11:34:43 2020-10-31 05:05:20 [INFO] [TRAIN] epoch=49, iter=18200/80000, loss=0.1882, lr=0.007929, batch_cost=0.6804, reader_cost=0.0003 | ETA 11:40:48 2020-10-31 05:06:30 [INFO] [TRAIN] epoch=50, iter=18300/80000, loss=0.2303, lr=0.007918, batch_cost=0.6951, reader_cost=0.0082 | ETA 11:54:49 2020-10-31 05:07:38 [INFO] [TRAIN] epoch=50, iter=18400/80000, loss=0.2250, lr=0.007906, batch_cost=0.6848, reader_cost=0.0004 | ETA 11:43:04 2020-10-31 05:08:47 [INFO] [TRAIN] epoch=50, iter=18500/80000, loss=0.2391, lr=0.007895, batch_cost=0.6861, reader_cost=0.0005 | ETA 11:43:16 2020-10-31 05:09:56 [INFO] [TRAIN] epoch=50, iter=18600/80000, loss=0.2427, lr=0.007883, batch_cost=0.6879, reader_cost=0.0007 | ETA 11:43:59 2020-10-31 05:11:05 [INFO] [TRAIN] epoch=51, iter=18700/80000, loss=0.2070, lr=0.007871, batch_cost=0.6908, reader_cost=0.0097 | ETA 11:45:47 2020-10-31 05:12:13 [INFO] [TRAIN] epoch=51, iter=18800/80000, loss=0.2162, lr=0.007860, batch_cost=0.6827, reader_cost=0.0007 | ETA 11:36:23 2020-10-31 05:13:22 [INFO] [TRAIN] epoch=51, iter=18900/80000, loss=0.1891, lr=0.007848, batch_cost=0.6898, reader_cost=0.0006 | ETA 11:42:25 2020-10-31 05:14:32 [INFO] [TRAIN] epoch=52, iter=19000/80000, loss=0.2777, lr=0.007837, batch_cost=0.6946, reader_cost=0.0097 | ETA 11:46:13 2020-10-31 05:15:40 [INFO] [TRAIN] epoch=52, iter=19100/80000, loss=0.2998, lr=0.007825, batch_cost=0.6841, reader_cost=0.0002 | ETA 11:34:20 2020-10-31 05:16:48 [INFO] [TRAIN] epoch=52, iter=19200/80000, loss=0.2371, lr=0.007814, batch_cost=0.6824, reader_cost=0.0005 | ETA 11:31:32 2020-10-31 05:17:56 [INFO] [TRAIN] epoch=52, iter=19300/80000, loss=0.2113, lr=0.007802, batch_cost=0.6812, reader_cost=0.0004 | ETA 11:29:07 2020-10-31 05:19:05 [INFO] [TRAIN] epoch=53, iter=19400/80000, loss=0.2410, lr=0.007791, batch_cost=0.6912, reader_cost=0.0099 | ETA 11:38:04 2020-10-31 05:20:14 [INFO] [TRAIN] epoch=53, iter=19500/80000, loss=0.2246, lr=0.007779, batch_cost=0.6826, reader_cost=0.0002 | ETA 11:28:16 2020-10-31 05:21:22 [INFO] [TRAIN] epoch=53, iter=19600/80000, loss=0.2265, lr=0.007768, batch_cost=0.6818, reader_cost=0.0002 | ETA 11:26:19 2020-10-31 05:22:30 [INFO] [TRAIN] epoch=53, iter=19700/80000, loss=0.2402, lr=0.007756, batch_cost=0.6783, reader_cost=0.0002 | ETA 11:21:40 2020-10-31 05:23:38 [INFO] [TRAIN] epoch=54, iter=19800/80000, loss=0.2158, lr=0.007744, batch_cost=0.6876, reader_cost=0.0093 | ETA 11:29:53 2020-10-31 05:24:47 [INFO] [TRAIN] epoch=54, iter=19900/80000, loss=0.2061, lr=0.007733, batch_cost=0.6828, reader_cost=0.0003 | ETA 11:23:58 2020-10-31 05:25:55 [INFO] [TRAIN] epoch=54, iter=20000/80000, loss=0.1880, lr=0.007721, batch_cost=0.6774, reader_cost=0.0002 | ETA 11:17:22 2020-10-31 05:27:03 [INFO] [TRAIN] epoch=55, iter=20100/80000, loss=0.2651, lr=0.007710, batch_cost=0.6877, reader_cost=0.0092 | ETA 11:26:34 2020-10-31 05:28:10 [INFO] [TRAIN] epoch=55, iter=20200/80000, loss=0.2230, lr=0.007698, batch_cost=0.6651, reader_cost=0.0001 | ETA 11:02:55 2020-10-31 05:29:17 [INFO] [TRAIN] epoch=55, iter=20300/80000, loss=0.2542, lr=0.007687, batch_cost=0.6686, reader_cost=0.0002 | ETA 11:05:15 2020-10-31 05:30:24 [INFO] [TRAIN] epoch=55, iter=20400/80000, loss=0.2041, lr=0.007675, batch_cost=0.6712, reader_cost=0.0002 | ETA 11:06:41 2020-10-31 05:31:32 [INFO] [TRAIN] epoch=56, iter=20500/80000, loss=0.2095, lr=0.007663, batch_cost=0.6808, reader_cost=0.0097 | ETA 11:15:06 2020-10-31 05:32:39 [INFO] [TRAIN] epoch=56, iter=20600/80000, loss=0.2131, lr=0.007652, batch_cost=0.6724, reader_cost=0.0004 | ETA 11:05:43 2020-10-31 05:33:47 [INFO] [TRAIN] epoch=56, iter=20700/80000, loss=0.2370, lr=0.007640, batch_cost=0.6781, reader_cost=0.0003 | ETA 11:10:10 2020-10-31 05:34:55 [INFO] [TRAIN] epoch=56, iter=20800/80000, loss=0.2007, lr=0.007629, batch_cost=0.6769, reader_cost=0.0003 | ETA 11:07:50 2020-10-31 05:36:03 [INFO] [TRAIN] epoch=57, iter=20900/80000, loss=0.2260, lr=0.007617, batch_cost=0.6889, reader_cost=0.0093 | ETA 11:18:31 2020-10-31 05:37:12 [INFO] [TRAIN] epoch=57, iter=21000/80000, loss=0.2258, lr=0.007606, batch_cost=0.6823, reader_cost=0.0002 | ETA 11:10:54 2020-10-31 05:38:19 [INFO] [TRAIN] epoch=57, iter=21100/80000, loss=0.2145, lr=0.007594, batch_cost=0.6765, reader_cost=0.0002 | ETA 11:04:06 2020-10-31 05:39:27 [INFO] [TRAIN] epoch=57, iter=21200/80000, loss=0.2355, lr=0.007582, batch_cost=0.6785, reader_cost=0.0002 | ETA 11:04:55 2020-10-31 05:40:36 [INFO] [TRAIN] epoch=58, iter=21300/80000, loss=0.2072, lr=0.007571, batch_cost=0.6847, reader_cost=0.0081 | ETA 11:09:50 2020-10-31 05:41:44 [INFO] [TRAIN] epoch=58, iter=21400/80000, loss=0.2208, lr=0.007559, batch_cost=0.6796, reader_cost=0.0002 | ETA 11:03:42 2020-10-31 05:42:52 [INFO] [TRAIN] epoch=58, iter=21500/80000, loss=0.1848, lr=0.007548, batch_cost=0.6787, reader_cost=0.0002 | ETA 11:01:41 2020-10-31 05:44:01 [INFO] [TRAIN] epoch=59, iter=21600/80000, loss=0.2689, lr=0.007536, batch_cost=0.6917, reader_cost=0.0091 | ETA 11:13:13 2020-10-31 05:45:09 [INFO] [TRAIN] epoch=59, iter=21700/80000, loss=0.2205, lr=0.007524, batch_cost=0.6818, reader_cost=0.0002 | ETA 11:02:31 2020-10-31 05:46:17 [INFO] [TRAIN] epoch=59, iter=21800/80000, loss=0.2252, lr=0.007513, batch_cost=0.6850, reader_cost=0.0004 | ETA 11:04:27 2020-10-31 05:47:25 [INFO] [TRAIN] epoch=59, iter=21900/80000, loss=0.2107, lr=0.007501, batch_cost=0.6792, reader_cost=0.0002 | ETA 10:57:39 2020-10-31 05:48:34 [INFO] [TRAIN] epoch=60, iter=22000/80000, loss=0.4036, lr=0.007490, batch_cost=0.6888, reader_cost=0.0094 | ETA 11:05:49 2020-10-31 05:49:42 [INFO] [TRAIN] epoch=60, iter=22100/80000, loss=0.2333, lr=0.007478, batch_cost=0.6791, reader_cost=0.0004 | ETA 10:55:20 2020-10-31 05:50:50 [INFO] [TRAIN] epoch=60, iter=22200/80000, loss=0.2183, lr=0.007466, batch_cost=0.6802, reader_cost=0.0004 | ETA 10:55:14 2020-10-31 05:51:58 [INFO] [TRAIN] epoch=60, iter=22300/80000, loss=0.2478, lr=0.007455, batch_cost=0.6803, reader_cost=0.0005 | ETA 10:54:15 2020-10-31 05:53:07 [INFO] [TRAIN] epoch=61, iter=22400/80000, loss=0.2312, lr=0.007443, batch_cost=0.6871, reader_cost=0.0081 | ETA 10:59:37 2020-10-31 05:54:15 [INFO] [TRAIN] epoch=61, iter=22500/80000, loss=0.2029, lr=0.007432, batch_cost=0.6810, reader_cost=0.0006 | ETA 10:52:35 2020-10-31 05:55:24 [INFO] [TRAIN] epoch=61, iter=22600/80000, loss=0.2168, lr=0.007420, batch_cost=0.6861, reader_cost=0.0003 | ETA 10:56:21 2020-10-31 05:56:32 [INFO] [TRAIN] epoch=62, iter=22700/80000, loss=0.2495, lr=0.007408, batch_cost=0.6868, reader_cost=0.0086 | ETA 10:55:50 2020-10-31 05:57:40 [INFO] [TRAIN] epoch=62, iter=22800/80000, loss=0.1979, lr=0.007397, batch_cost=0.6794, reader_cost=0.0003 | ETA 10:47:39 2020-10-31 05:58:48 [INFO] [TRAIN] epoch=62, iter=22900/80000, loss=0.1871, lr=0.007385, batch_cost=0.6819, reader_cost=0.0004 | ETA 10:48:56 2020-10-31 05:59:56 [INFO] [TRAIN] epoch=62, iter=23000/80000, loss=0.2062, lr=0.007373, batch_cost=0.6812, reader_cost=0.0003 | ETA 10:47:09 2020-10-31 06:01:05 [INFO] [TRAIN] epoch=63, iter=23100/80000, loss=0.2090, lr=0.007362, batch_cost=0.6878, reader_cost=0.0088 | ETA 10:52:17 2020-10-31 06:02:13 [INFO] [TRAIN] epoch=63, iter=23200/80000, loss=0.2148, lr=0.007350, batch_cost=0.6795, reader_cost=0.0003 | ETA 10:43:15 2020-10-31 06:03:21 [INFO] [TRAIN] epoch=63, iter=23300/80000, loss=0.2545, lr=0.007339, batch_cost=0.6826, reader_cost=0.0002 | ETA 10:45:03 2020-10-31 06:04:30 [INFO] [TRAIN] epoch=63, iter=23400/80000, loss=0.2389, lr=0.007327, batch_cost=0.6820, reader_cost=0.0002 | ETA 10:43:21 2020-10-31 06:05:39 [INFO] [TRAIN] epoch=64, iter=23500/80000, loss=0.2001, lr=0.007315, batch_cost=0.6887, reader_cost=0.0092 | ETA 10:48:32 2020-10-31 06:06:47 [INFO] [TRAIN] epoch=64, iter=23600/80000, loss=0.1736, lr=0.007304, batch_cost=0.6811, reader_cost=0.0002 | ETA 10:40:16 2020-10-31 06:07:55 [INFO] [TRAIN] epoch=64, iter=23700/80000, loss=0.1972, lr=0.007292, batch_cost=0.6802, reader_cost=0.0005 | ETA 10:38:13 2020-10-31 06:09:03 [INFO] [TRAIN] epoch=64, iter=23800/80000, loss=0.2281, lr=0.007280, batch_cost=0.6807, reader_cost=0.0003 | ETA 10:37:37 2020-10-31 06:10:12 [INFO] [TRAIN] epoch=65, iter=23900/80000, loss=0.2547, lr=0.007269, batch_cost=0.6874, reader_cost=0.0094 | ETA 10:42:45 2020-10-31 06:11:19 [INFO] [TRAIN] epoch=65, iter=24000/80000, loss=0.2183, lr=0.007257, batch_cost=0.6792, reader_cost=0.0001 | ETA 10:33:56 2020-10-31 06:11:27 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-10-31 06:17:10 [INFO] [EVAL] #Images=500 mIoU=0.7256 Acc=0.9481 Kappa=0.9328 2020-10-31 06:17:10 [INFO] [EVAL] Category IoU: [0.9728 0.7932 0.8943 0.4597 0.5732 0.6294 0.7003 0.7541 0.8978 0.564 0.9406 0.8108 0.622 0.9396 0.7829 0.7181 0.4378 0.5362 0.7598] 2020-10-31 06:17:10 [INFO] [EVAL] Category Acc: [0.9915 0.8429 0.9574 0.8877 0.7352 0.762 0.8178 0.8429 0.9231 0.8818 0.9675 0.897 0.7775 0.9678 0.9034 0.7651 0.8226 0.683 0.8422] 2020-10-31 06:17:10 [INFO] [EVAL] The model with the best validation mIoU (0.7447) was saved at iter 16000. 2020-10-31 06:18:17 [INFO] [TRAIN] epoch=65, iter=24100/80000, loss=0.2426, lr=0.007245, batch_cost=0.6701, reader_cost=0.0002 | ETA 10:24:17 2020-10-31 06:19:26 [INFO] [TRAIN] epoch=66, iter=24200/80000, loss=0.2149, lr=0.007234, batch_cost=0.6860, reader_cost=0.0096 | ETA 10:37:59 2020-10-31 06:20:33 [INFO] [TRAIN] epoch=66, iter=24300/80000, loss=0.1962, lr=0.007222, batch_cost=0.6786, reader_cost=0.0004 | ETA 10:29:57 2020-10-31 06:21:41 [INFO] [TRAIN] epoch=66, iter=24400/80000, loss=0.2125, lr=0.007210, batch_cost=0.6780, reader_cost=0.0002 | ETA 10:28:16 2020-10-31 06:22:49 [INFO] [TRAIN] epoch=66, iter=24500/80000, loss=0.2009, lr=0.007199, batch_cost=0.6766, reader_cost=0.0002 | ETA 10:25:53 2020-10-31 06:23:58 [INFO] [TRAIN] epoch=67, iter=24600/80000, loss=0.2102, lr=0.007187, batch_cost=0.6874, reader_cost=0.0093 | ETA 10:34:39 2020-10-31 06:25:06 [INFO] [TRAIN] epoch=67, iter=24700/80000, loss=0.1818, lr=0.007175, batch_cost=0.6804, reader_cost=0.0005 | ETA 10:27:03 2020-10-31 06:26:14 [INFO] [TRAIN] epoch=67, iter=24800/80000, loss=0.1836, lr=0.007164, batch_cost=0.6801, reader_cost=0.0004 | ETA 10:25:42 2020-10-31 06:27:21 [INFO] [TRAIN] epoch=67, iter=24900/80000, loss=0.1804, lr=0.007152, batch_cost=0.6755, reader_cost=0.0002 | ETA 10:20:20 2020-10-31 06:28:30 [INFO] [TRAIN] epoch=68, iter=25000/80000, loss=0.2138, lr=0.007140, batch_cost=0.6907, reader_cost=0.0101 | ETA 10:33:05 2020-10-31 06:29:39 [INFO] [TRAIN] epoch=68, iter=25100/80000, loss=0.1936, lr=0.007129, batch_cost=0.6821, reader_cost=0.0005 | ETA 10:24:06 2020-10-31 06:30:47 [INFO] [TRAIN] epoch=68, iter=25200/80000, loss=0.1959, lr=0.007117, batch_cost=0.6845, reader_cost=0.0005 | ETA 10:25:09 2020-10-31 06:31:56 [INFO] [TRAIN] epoch=69, iter=25300/80000, loss=0.2817, lr=0.007105, batch_cost=0.6947, reader_cost=0.0094 | ETA 10:33:18 2020-10-31 06:33:05 [INFO] [TRAIN] epoch=69, iter=25400/80000, loss=0.2111, lr=0.007094, batch_cost=0.6840, reader_cost=0.0008 | ETA 10:22:28 2020-10-31 06:34:14 [INFO] [TRAIN] epoch=69, iter=25500/80000, loss=0.2068, lr=0.007082, batch_cost=0.6895, reader_cost=0.0010 | ETA 10:26:15 2020-10-31 06:35:22 [INFO] [TRAIN] epoch=69, iter=25600/80000, loss=0.1992, lr=0.007070, batch_cost=0.6812, reader_cost=0.0003 | ETA 10:17:38 2020-10-31 06:36:31 [INFO] [TRAIN] epoch=70, iter=25700/80000, loss=0.2372, lr=0.007059, batch_cost=0.6886, reader_cost=0.0097 | ETA 10:23:08 2020-10-31 06:37:39 [INFO] [TRAIN] epoch=70, iter=25800/80000, loss=0.1869, lr=0.007047, batch_cost=0.6854, reader_cost=0.0006 | ETA 10:19:08 2020-10-31 06:38:47 [INFO] [TRAIN] epoch=70, iter=25900/80000, loss=0.2097, lr=0.007035, batch_cost=0.6771, reader_cost=0.0003 | ETA 10:10:33 2020-10-31 06:39:55 [INFO] [TRAIN] epoch=70, iter=26000/80000, loss=0.2089, lr=0.007024, batch_cost=0.6810, reader_cost=0.0003 | ETA 10:12:52 2020-10-31 06:41:04 [INFO] [TRAIN] epoch=71, iter=26100/80000, loss=0.1839, lr=0.007012, batch_cost=0.6856, reader_cost=0.0093 | ETA 10:15:54 2020-10-31 06:42:12 [INFO] [TRAIN] epoch=71, iter=26200/80000, loss=0.2093, lr=0.007000, batch_cost=0.6822, reader_cost=0.0007 | ETA 10:11:43 2020-10-31 06:43:20 [INFO] [TRAIN] epoch=71, iter=26300/80000, loss=0.2145, lr=0.006989, batch_cost=0.6824, reader_cost=0.0006 | ETA 10:10:47 2020-10-31 06:44:29 [INFO] [TRAIN] epoch=71, iter=26400/80000, loss=0.2328, lr=0.006977, batch_cost=0.6903, reader_cost=0.0010 | ETA 10:16:40 2020-10-31 06:45:39 [INFO] [TRAIN] epoch=72, iter=26500/80000, loss=0.1845, lr=0.006965, batch_cost=0.6954, reader_cost=0.0111 | ETA 10:20:04 2020-10-31 06:46:47 [INFO] [TRAIN] epoch=72, iter=26600/80000, loss=0.1902, lr=0.006954, batch_cost=0.6803, reader_cost=0.0006 | ETA 10:05:28 2020-10-31 06:47:55 [INFO] [TRAIN] epoch=72, iter=26700/80000, loss=0.1746, lr=0.006942, batch_cost=0.6839, reader_cost=0.0009 | ETA 10:07:34 2020-10-31 06:49:04 [INFO] [TRAIN] epoch=73, iter=26800/80000, loss=0.1932, lr=0.006930, batch_cost=0.6933, reader_cost=0.0097 | ETA 10:14:45 2020-10-31 06:50:12 [INFO] [TRAIN] epoch=73, iter=26900/80000, loss=0.1971, lr=0.006918, batch_cost=0.6799, reader_cost=0.0002 | ETA 10:01:42 2020-10-31 06:51:21 [INFO] [TRAIN] epoch=73, iter=27000/80000, loss=0.2062, lr=0.006907, batch_cost=0.6835, reader_cost=0.0003 | ETA 10:03:43 2020-10-31 06:52:29 [INFO] [TRAIN] epoch=73, iter=27100/80000, loss=0.1758, lr=0.006895, batch_cost=0.6784, reader_cost=0.0003 | ETA 09:58:06 2020-10-31 06:53:37 [INFO] [TRAIN] epoch=74, iter=27200/80000, loss=0.1632, lr=0.006883, batch_cost=0.6839, reader_cost=0.0104 | ETA 10:01:52 2020-10-31 06:54:45 [INFO] [TRAIN] epoch=74, iter=27300/80000, loss=0.1965, lr=0.006872, batch_cost=0.6752, reader_cost=0.0002 | ETA 09:53:01 2020-10-31 06:55:52 [INFO] [TRAIN] epoch=74, iter=27400/80000, loss=0.1892, lr=0.006860, batch_cost=0.6702, reader_cost=0.0001 | ETA 09:47:30 2020-10-31 06:57:00 [INFO] [TRAIN] epoch=74, iter=27500/80000, loss=0.1840, lr=0.006848, batch_cost=0.6824, reader_cost=0.0003 | ETA 09:57:08 2020-10-31 06:58:09 [INFO] [TRAIN] epoch=75, iter=27600/80000, loss=0.2004, lr=0.006836, batch_cost=0.6893, reader_cost=0.0093 | ETA 10:02:01 2020-10-31 06:59:17 [INFO] [TRAIN] epoch=75, iter=27700/80000, loss=0.1778, lr=0.006825, batch_cost=0.6858, reader_cost=0.0006 | ETA 09:57:49 2020-10-31 07:00:26 [INFO] [TRAIN] epoch=75, iter=27800/80000, loss=0.1771, lr=0.006813, batch_cost=0.6829, reader_cost=0.0005 | ETA 09:54:07 2020-10-31 07:01:34 [INFO] [TRAIN] epoch=75, iter=27900/80000, loss=0.2000, lr=0.006801, batch_cost=0.6836, reader_cost=0.0005 | ETA 09:53:37 2020-10-31 07:02:43 [INFO] [TRAIN] epoch=76, iter=28000/80000, loss=0.1767, lr=0.006789, batch_cost=0.6935, reader_cost=0.0108 | ETA 10:01:00 2020-10-31 07:03:51 [INFO] [TRAIN] epoch=76, iter=28100/80000, loss=0.2268, lr=0.006778, batch_cost=0.6782, reader_cost=0.0001 | ETA 09:46:40 2020-10-31 07:04:59 [INFO] [TRAIN] epoch=76, iter=28200/80000, loss=0.2042, lr=0.006766, batch_cost=0.6794, reader_cost=0.0003 | ETA 09:46:30 2020-10-31 07:06:08 [INFO] [TRAIN] epoch=77, iter=28300/80000, loss=0.1847, lr=0.006754, batch_cost=0.6880, reader_cost=0.0086 | ETA 09:52:51 2020-10-31 07:07:16 [INFO] [TRAIN] epoch=77, iter=28400/80000, loss=0.1900, lr=0.006743, batch_cost=0.6763, reader_cost=0.0001 | ETA 09:41:39 2020-10-31 07:08:23 [INFO] [TRAIN] epoch=77, iter=28500/80000, loss=0.1675, lr=0.006731, batch_cost=0.6708, reader_cost=0.0001 | ETA 09:35:48 2020-10-31 07:09:30 [INFO] [TRAIN] epoch=77, iter=28600/80000, loss=0.1769, lr=0.006719, batch_cost=0.6778, reader_cost=0.0002 | ETA 09:40:40 2020-10-31 07:10:40 [INFO] [TRAIN] epoch=78, iter=28700/80000, loss=0.1897, lr=0.006707, batch_cost=0.6928, reader_cost=0.0095 | ETA 09:52:21 2020-10-31 07:11:48 [INFO] [TRAIN] epoch=78, iter=28800/80000, loss=0.2019, lr=0.006696, batch_cost=0.6867, reader_cost=0.0008 | ETA 09:45:58 2020-10-31 07:12:56 [INFO] [TRAIN] epoch=78, iter=28900/80000, loss=0.1799, lr=0.006684, batch_cost=0.6797, reader_cost=0.0003 | ETA 09:38:54 2020-10-31 07:14:04 [INFO] [TRAIN] epoch=78, iter=29000/80000, loss=0.1764, lr=0.006672, batch_cost=0.6775, reader_cost=0.0002 | ETA 09:35:52 2020-10-31 07:15:13 [INFO] [TRAIN] epoch=79, iter=29100/80000, loss=0.1859, lr=0.006660, batch_cost=0.6894, reader_cost=0.0086 | ETA 09:44:48 2020-10-31 07:16:20 [INFO] [TRAIN] epoch=79, iter=29200/80000, loss=0.1728, lr=0.006648, batch_cost=0.6686, reader_cost=0.0002 | ETA 09:26:05 2020-10-31 07:17:27 [INFO] [TRAIN] epoch=79, iter=29300/80000, loss=0.1762, lr=0.006637, batch_cost=0.6661, reader_cost=0.0001 | ETA 09:22:52 2020-10-31 07:18:34 [INFO] [TRAIN] epoch=80, iter=29400/80000, loss=0.1943, lr=0.006625, batch_cost=0.6768, reader_cost=0.0098 | ETA 09:30:43 2020-10-31 07:19:41 [INFO] [TRAIN] epoch=80, iter=29500/80000, loss=0.1743, lr=0.006613, batch_cost=0.6682, reader_cost=0.0002 | ETA 09:22:25 2020-10-31 07:20:48 [INFO] [TRAIN] epoch=80, iter=29600/80000, loss=0.2132, lr=0.006601, batch_cost=0.6675, reader_cost=0.0002 | ETA 09:20:40 2020-10-31 07:21:55 [INFO] [TRAIN] epoch=80, iter=29700/80000, loss=0.1872, lr=0.006590, batch_cost=0.6698, reader_cost=0.0002 | ETA 09:21:30 2020-10-31 07:23:02 [INFO] [TRAIN] epoch=81, iter=29800/80000, loss=0.2168, lr=0.006578, batch_cost=0.6753, reader_cost=0.0088 | ETA 09:25:00 2020-10-31 07:24:09 [INFO] [TRAIN] epoch=81, iter=29900/80000, loss=0.1958, lr=0.006566, batch_cost=0.6666, reader_cost=0.0002 | ETA 09:16:36 2020-10-31 07:25:15 [INFO] [TRAIN] epoch=81, iter=30000/80000, loss=0.1772, lr=0.006554, batch_cost=0.6632, reader_cost=0.0002 | ETA 09:12:38 2020-10-31 07:26:22 [INFO] [TRAIN] epoch=81, iter=30100/80000, loss=0.1698, lr=0.006543, batch_cost=0.6685, reader_cost=0.0005 | ETA 09:16:00 2020-10-31 07:27:30 [INFO] [TRAIN] epoch=82, iter=30200/80000, loss=0.1994, lr=0.006531, batch_cost=0.6806, reader_cost=0.0086 | ETA 09:24:56 2020-10-31 07:28:37 [INFO] [TRAIN] epoch=82, iter=30300/80000, loss=0.2485, lr=0.006519, batch_cost=0.6710, reader_cost=0.0002 | ETA 09:15:50 2020-10-31 07:29:44 [INFO] [TRAIN] epoch=82, iter=30400/80000, loss=0.1812, lr=0.006507, batch_cost=0.6643, reader_cost=0.0003 | ETA 09:09:11 2020-10-31 07:30:50 [INFO] [TRAIN] epoch=82, iter=30500/80000, loss=0.1787, lr=0.006495, batch_cost=0.6652, reader_cost=0.0002 | ETA 09:08:45 2020-10-31 07:31:58 [INFO] [TRAIN] epoch=83, iter=30600/80000, loss=0.1918, lr=0.006484, batch_cost=0.6780, reader_cost=0.0095 | ETA 09:18:13 2020-10-31 07:33:05 [INFO] [TRAIN] epoch=83, iter=30700/80000, loss=0.2003, lr=0.006472, batch_cost=0.6662, reader_cost=0.0001 | ETA 09:07:23 2020-10-31 07:34:12 [INFO] [TRAIN] epoch=83, iter=30800/80000, loss=0.1867, lr=0.006460, batch_cost=0.6697, reader_cost=0.0001 | ETA 09:09:09 2020-10-31 07:35:20 [INFO] [TRAIN] epoch=84, iter=30900/80000, loss=0.1995, lr=0.006448, batch_cost=0.6797, reader_cost=0.0092 | ETA 09:16:14 2020-10-31 07:36:26 [INFO] [TRAIN] epoch=84, iter=31000/80000, loss=0.1881, lr=0.006436, batch_cost=0.6674, reader_cost=0.0001 | ETA 09:05:02 2020-10-31 07:37:33 [INFO] [TRAIN] epoch=84, iter=31100/80000, loss=0.1815, lr=0.006425, batch_cost=0.6701, reader_cost=0.0001 | ETA 09:06:07 2020-10-31 07:38:40 [INFO] [TRAIN] epoch=84, iter=31200/80000, loss=0.1791, lr=0.006413, batch_cost=0.6688, reader_cost=0.0001 | ETA 09:03:55 2020-10-31 07:39:48 [INFO] [TRAIN] epoch=85, iter=31300/80000, loss=0.2408, lr=0.006401, batch_cost=0.6745, reader_cost=0.0099 | ETA 09:07:30 2020-10-31 07:40:54 [INFO] [TRAIN] epoch=85, iter=31400/80000, loss=0.2138, lr=0.006389, batch_cost=0.6676, reader_cost=0.0002 | ETA 09:00:46 2020-10-31 07:42:01 [INFO] [TRAIN] epoch=85, iter=31500/80000, loss=0.2055, lr=0.006377, batch_cost=0.6652, reader_cost=0.0002 | ETA 08:57:40 2020-10-31 07:43:08 [INFO] [TRAIN] epoch=85, iter=31600/80000, loss=0.1864, lr=0.006366, batch_cost=0.6745, reader_cost=0.0004 | ETA 09:04:06 2020-10-31 07:44:16 [INFO] [TRAIN] epoch=86, iter=31700/80000, loss=0.2007, lr=0.006354, batch_cost=0.6774, reader_cost=0.0092 | ETA 09:05:17 2020-10-31 07:45:23 [INFO] [TRAIN] epoch=86, iter=31800/80000, loss=0.1807, lr=0.006342, batch_cost=0.6663, reader_cost=0.0002 | ETA 08:55:16 2020-10-31 07:46:30 [INFO] [TRAIN] epoch=86, iter=31900/80000, loss=0.1526, lr=0.006330, batch_cost=0.6689, reader_cost=0.0002 | ETA 08:56:13 2020-10-31 07:47:37 [INFO] [TRAIN] epoch=87, iter=32000/80000, loss=0.2108, lr=0.006318, batch_cost=0.6755, reader_cost=0.0086 | ETA 09:00:26 2020-10-31 07:47:44 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-10-31 07:53:18 [INFO] [EVAL] #Images=500 mIoU=0.7543 Acc=0.9593 Kappa=0.9472 2020-10-31 07:53:18 [INFO] [EVAL] Category IoU: [0.9818 0.8531 0.9215 0.4778 0.596 0.6422 0.6919 0.7882 0.923 0.6187 0.9492 0.8254 0.6249 0.9513 0.8463 0.7497 0.4976 0.6126 0.7799] 2020-10-31 07:53:18 [INFO] [EVAL] Category Acc: [0.9928 0.9104 0.9557 0.7239 0.7432 0.8082 0.8903 0.9261 0.9506 0.8689 0.9731 0.9075 0.7732 0.9732 0.9221 0.9009 0.5825 0.7825 0.8744] 2020-10-31 07:53:22 [INFO] [EVAL] The model with the best validation mIoU (0.7543) was saved at iter 32000. 2020-10-31 07:54:28 [INFO] [TRAIN] epoch=87, iter=32100/80000, loss=0.1518, lr=0.006306, batch_cost=0.6635, reader_cost=0.0002 | ETA 08:49:41 2020-10-31 07:55:34 [INFO] [TRAIN] epoch=87, iter=32200/80000, loss=0.1968, lr=0.006295, batch_cost=0.6649, reader_cost=0.0001 | ETA 08:49:42 2020-10-31 07:56:41 [INFO] [TRAIN] epoch=87, iter=32300/80000, loss=0.1878, lr=0.006283, batch_cost=0.6669, reader_cost=0.0002 | ETA 08:50:09 2020-10-31 07:57:49 [INFO] [TRAIN] epoch=88, iter=32400/80000, loss=0.1872, lr=0.006271, batch_cost=0.6768, reader_cost=0.0094 | ETA 08:56:54 2020-10-31 07:58:56 [INFO] [TRAIN] epoch=88, iter=32500/80000, loss=0.1669, lr=0.006259, batch_cost=0.6710, reader_cost=0.0003 | ETA 08:51:12 2020-10-31 08:00:03 [INFO] [TRAIN] epoch=88, iter=32600/80000, loss=0.1918, lr=0.006247, batch_cost=0.6728, reader_cost=0.0001 | ETA 08:51:32 2020-10-31 08:01:11 [INFO] [TRAIN] epoch=88, iter=32700/80000, loss=0.1637, lr=0.006235, batch_cost=0.6740, reader_cost=0.0002 | ETA 08:51:20 2020-10-31 08:02:19 [INFO] [TRAIN] epoch=89, iter=32800/80000, loss=0.1580, lr=0.006224, batch_cost=0.6807, reader_cost=0.0088 | ETA 08:55:28 2020-10-31 08:03:26 [INFO] [TRAIN] epoch=89, iter=32900/80000, loss=0.2486, lr=0.006212, batch_cost=0.6720, reader_cost=0.0001 | ETA 08:47:31 2020-10-31 08:04:33 [INFO] [TRAIN] epoch=89, iter=33000/80000, loss=0.1751, lr=0.006200, batch_cost=0.6706, reader_cost=0.0001 | ETA 08:45:20 2020-10-31 08:05:41 [INFO] [TRAIN] epoch=89, iter=33100/80000, loss=0.1915, lr=0.006188, batch_cost=0.6768, reader_cost=0.0002 | ETA 08:49:03 2020-10-31 08:06:48 [INFO] [TRAIN] epoch=90, iter=33200/80000, loss=0.1682, lr=0.006176, batch_cost=0.6771, reader_cost=0.0102 | ETA 08:48:07 2020-10-31 08:07:55 [INFO] [TRAIN] epoch=90, iter=33300/80000, loss=0.1668, lr=0.006164, batch_cost=0.6678, reader_cost=0.0001 | ETA 08:39:44 2020-10-31 08:09:02 [INFO] [TRAIN] epoch=90, iter=33400/80000, loss=0.1702, lr=0.006152, batch_cost=0.6693, reader_cost=0.0001 | ETA 08:39:50 2020-10-31 08:10:10 [INFO] [TRAIN] epoch=91, iter=33500/80000, loss=0.1833, lr=0.006141, batch_cost=0.6776, reader_cost=0.0107 | ETA 08:45:06 2020-10-31 08:11:18 [INFO] [TRAIN] epoch=91, iter=33600/80000, loss=0.1930, lr=0.006129, batch_cost=0.6785, reader_cost=0.0002 | ETA 08:44:42 2020-10-31 08:12:27 [INFO] [TRAIN] epoch=91, iter=33700/80000, loss=0.1739, lr=0.006117, batch_cost=0.6903, reader_cost=0.0001 | ETA 08:52:40 2020-10-31 08:13:35 [INFO] [TRAIN] epoch=91, iter=33800/80000, loss=0.1907, lr=0.006105, batch_cost=0.6871, reader_cost=0.0002 | ETA 08:49:05 2020-10-31 08:14:44 [INFO] [TRAIN] epoch=92, iter=33900/80000, loss=0.1642, lr=0.006093, batch_cost=0.6881, reader_cost=0.0085 | ETA 08:48:40 2020-10-31 08:15:52 [INFO] [TRAIN] epoch=92, iter=34000/80000, loss=0.1835, lr=0.006081, batch_cost=0.6820, reader_cost=0.0002 | ETA 08:42:53 2020-10-31 08:17:01 [INFO] [TRAIN] epoch=92, iter=34100/80000, loss=0.1824, lr=0.006069, batch_cost=0.6824, reader_cost=0.0005 | ETA 08:42:01 2020-10-31 08:18:10 [INFO] [TRAIN] epoch=92, iter=34200/80000, loss=0.1769, lr=0.006057, batch_cost=0.6891, reader_cost=0.0004 | ETA 08:45:58 2020-10-31 08:19:18 [INFO] [TRAIN] epoch=93, iter=34300/80000, loss=0.1935, lr=0.006046, batch_cost=0.6880, reader_cost=0.0093 | ETA 08:44:00 2020-10-31 08:20:26 [INFO] [TRAIN] epoch=93, iter=34400/80000, loss=0.1641, lr=0.006034, batch_cost=0.6748, reader_cost=0.0003 | ETA 08:32:52 2020-10-31 08:21:34 [INFO] [TRAIN] epoch=93, iter=34500/80000, loss=0.1729, lr=0.006022, batch_cost=0.6766, reader_cost=0.0002 | ETA 08:33:06 2020-10-31 08:22:42 [INFO] [TRAIN] epoch=94, iter=34600/80000, loss=0.1675, lr=0.006010, batch_cost=0.6843, reader_cost=0.0095 | ETA 08:37:46 2020-10-31 08:23:49 [INFO] [TRAIN] epoch=94, iter=34700/80000, loss=0.1799, lr=0.005998, batch_cost=0.6663, reader_cost=0.0001 | ETA 08:23:05 2020-10-31 08:24:56 [INFO] [TRAIN] epoch=94, iter=34800/80000, loss=0.2078, lr=0.005986, batch_cost=0.6700, reader_cost=0.0003 | ETA 08:24:42 2020-10-31 08:26:02 [INFO] [TRAIN] epoch=94, iter=34900/80000, loss=0.1973, lr=0.005974, batch_cost=0.6687, reader_cost=0.0002 | ETA 08:22:37 2020-10-31 08:27:10 [INFO] [TRAIN] epoch=95, iter=35000/80000, loss=0.1681, lr=0.005962, batch_cost=0.6766, reader_cost=0.0086 | ETA 08:27:25 2020-10-31 08:28:17 [INFO] [TRAIN] epoch=95, iter=35100/80000, loss=0.1839, lr=0.005950, batch_cost=0.6654, reader_cost=0.0002 | ETA 08:17:55 2020-10-31 08:29:23 [INFO] [TRAIN] epoch=95, iter=35200/80000, loss=0.1652, lr=0.005938, batch_cost=0.6670, reader_cost=0.0002 | ETA 08:17:59 2020-10-31 08:30:30 [INFO] [TRAIN] epoch=95, iter=35300/80000, loss=0.1581, lr=0.005927, batch_cost=0.6672, reader_cost=0.0005 | ETA 08:17:05 2020-10-31 08:31:38 [INFO] [TRAIN] epoch=96, iter=35400/80000, loss=0.1936, lr=0.005915, batch_cost=0.6777, reader_cost=0.0095 | ETA 08:23:47 2020-10-31 08:32:45 [INFO] [TRAIN] epoch=96, iter=35500/80000, loss=0.1738, lr=0.005903, batch_cost=0.6758, reader_cost=0.0002 | ETA 08:21:12 2020-10-31 08:33:53 [INFO] [TRAIN] epoch=96, iter=35600/80000, loss=0.1782, lr=0.005891, batch_cost=0.6802, reader_cost=0.0002 | ETA 08:23:19 2020-10-31 08:35:01 [INFO] [TRAIN] epoch=96, iter=35700/80000, loss=0.1729, lr=0.005879, batch_cost=0.6801, reader_cost=0.0002 | ETA 08:22:10 2020-10-31 08:36:10 [INFO] [TRAIN] epoch=97, iter=35800/80000, loss=0.2076, lr=0.005867, batch_cost=0.6846, reader_cost=0.0093 | ETA 08:24:17 2020-10-31 08:37:17 [INFO] [TRAIN] epoch=97, iter=35900/80000, loss=0.1852, lr=0.005855, batch_cost=0.6703, reader_cost=0.0002 | ETA 08:12:42 2020-10-31 08:38:24 [INFO] [TRAIN] epoch=97, iter=36000/80000, loss=0.2005, lr=0.005843, batch_cost=0.6665, reader_cost=0.0002 | ETA 08:08:47 2020-10-31 08:39:31 [INFO] [TRAIN] epoch=98, iter=36100/80000, loss=0.1941, lr=0.005831, batch_cost=0.6757, reader_cost=0.0092 | ETA 08:14:21 2020-10-31 08:40:38 [INFO] [TRAIN] epoch=98, iter=36200/80000, loss=0.1809, lr=0.005819, batch_cost=0.6661, reader_cost=0.0001 | ETA 08:06:13 2020-10-31 08:41:44 [INFO] [TRAIN] epoch=98, iter=36300/80000, loss=0.2018, lr=0.005807, batch_cost=0.6640, reader_cost=0.0001 | ETA 08:03:36 2020-10-31 08:42:51 [INFO] [TRAIN] epoch=98, iter=36400/80000, loss=0.1854, lr=0.005795, batch_cost=0.6648, reader_cost=0.0001 | ETA 08:03:07 2020-10-31 08:43:58 [INFO] [TRAIN] epoch=99, iter=36500/80000, loss=0.1798, lr=0.005783, batch_cost=0.6766, reader_cost=0.0099 | ETA 08:10:33 2020-10-31 08:45:06 [INFO] [TRAIN] epoch=99, iter=36600/80000, loss=0.2150, lr=0.005771, batch_cost=0.6777, reader_cost=0.0001 | ETA 08:10:13 2020-10-31 08:46:14 [INFO] [TRAIN] epoch=99, iter=36700/80000, loss=0.1834, lr=0.005760, batch_cost=0.6812, reader_cost=0.0002 | ETA 08:11:38 2020-10-31 08:47:22 [INFO] [TRAIN] epoch=99, iter=36800/80000, loss=0.1808, lr=0.005748, batch_cost=0.6796, reader_cost=0.0001 | ETA 08:09:16 2020-10-31 08:48:30 [INFO] [TRAIN] epoch=100, iter=36900/80000, loss=0.1924, lr=0.005736, batch_cost=0.6806, reader_cost=0.0092 | ETA 08:08:53 2020-10-31 08:49:37 [INFO] [TRAIN] epoch=100, iter=37000/80000, loss=0.1893, lr=0.005724, batch_cost=0.6671, reader_cost=0.0002 | ETA 07:58:05 2020-10-31 08:50:44 [INFO] [TRAIN] epoch=100, iter=37100/80000, loss=0.1623, lr=0.005712, batch_cost=0.6721, reader_cost=0.0001 | ETA 08:00:35 2020-10-31 08:51:51 [INFO] [TRAIN] epoch=100, iter=37200/80000, loss=0.1985, lr=0.005700, batch_cost=0.6692, reader_cost=0.0002 | ETA 07:57:21 2020-10-31 08:52:59 [INFO] [TRAIN] epoch=101, iter=37300/80000, loss=0.1854, lr=0.005688, batch_cost=0.6821, reader_cost=0.0099 | ETA 08:05:26 2020-10-31 08:54:07 [INFO] [TRAIN] epoch=101, iter=37400/80000, loss=0.1733, lr=0.005676, batch_cost=0.6753, reader_cost=0.0001 | ETA 07:59:26 2020-10-31 08:55:14 [INFO] [TRAIN] epoch=101, iter=37500/80000, loss=0.1607, lr=0.005664, batch_cost=0.6714, reader_cost=0.0002 | ETA 07:55:33 2020-10-31 08:56:22 [INFO] [TRAIN] epoch=102, iter=37600/80000, loss=0.1856, lr=0.005652, batch_cost=0.6786, reader_cost=0.0086 | ETA 07:59:31 2020-10-31 08:57:29 [INFO] [TRAIN] epoch=102, iter=37700/80000, loss=0.1712, lr=0.005640, batch_cost=0.6750, reader_cost=0.0002 | ETA 07:55:54 2020-10-31 08:58:36 [INFO] [TRAIN] epoch=102, iter=37800/80000, loss=0.1632, lr=0.005628, batch_cost=0.6690, reader_cost=0.0001 | ETA 07:50:31 2020-10-31 08:59:43 [INFO] [TRAIN] epoch=102, iter=37900/80000, loss=0.1453, lr=0.005616, batch_cost=0.6718, reader_cost=0.0001 | ETA 07:51:21 2020-10-31 09:00:51 [INFO] [TRAIN] epoch=103, iter=38000/80000, loss=0.1583, lr=0.005604, batch_cost=0.6791, reader_cost=0.0090 | ETA 07:55:23 2020-10-31 09:01:59 [INFO] [TRAIN] epoch=103, iter=38100/80000, loss=0.1909, lr=0.005592, batch_cost=0.6721, reader_cost=0.0002 | ETA 07:49:22 2020-10-31 09:03:06 [INFO] [TRAIN] epoch=103, iter=38200/80000, loss=0.1835, lr=0.005580, batch_cost=0.6699, reader_cost=0.0002 | ETA 07:46:41 2020-10-31 09:04:12 [INFO] [TRAIN] epoch=103, iter=38300/80000, loss=0.1778, lr=0.005568, batch_cost=0.6686, reader_cost=0.0002 | ETA 07:44:39 2020-10-31 09:05:20 [INFO] [TRAIN] epoch=104, iter=38400/80000, loss=0.1673, lr=0.005556, batch_cost=0.6807, reader_cost=0.0103 | ETA 07:51:57 2020-10-31 09:06:28 [INFO] [TRAIN] epoch=104, iter=38500/80000, loss=0.1787, lr=0.005544, batch_cost=0.6771, reader_cost=0.0002 | ETA 07:48:21 2020-10-31 09:07:36 [INFO] [TRAIN] epoch=104, iter=38600/80000, loss=0.1540, lr=0.005532, batch_cost=0.6734, reader_cost=0.0002 | ETA 07:44:38 2020-10-31 09:08:44 [INFO] [TRAIN] epoch=105, iter=38700/80000, loss=0.1522, lr=0.005520, batch_cost=0.6872, reader_cost=0.0097 | ETA 07:53:02 2020-10-31 09:09:52 [INFO] [TRAIN] epoch=105, iter=38800/80000, loss=0.1673, lr=0.005508, batch_cost=0.6753, reader_cost=0.0004 | ETA 07:43:43 2020-10-31 09:10:59 [INFO] [TRAIN] epoch=105, iter=38900/80000, loss=0.1598, lr=0.005496, batch_cost=0.6769, reader_cost=0.0002 | ETA 07:43:38 2020-10-31 09:12:08 [INFO] [TRAIN] epoch=105, iter=39000/80000, loss=0.1512, lr=0.005484, batch_cost=0.6831, reader_cost=0.0003 | ETA 07:46:47 2020-10-31 09:13:17 [INFO] [TRAIN] epoch=106, iter=39100/80000, loss=0.1655, lr=0.005472, batch_cost=0.6908, reader_cost=0.0095 | ETA 07:50:55 2020-10-31 09:14:26 [INFO] [TRAIN] epoch=106, iter=39200/80000, loss=0.1625, lr=0.005460, batch_cost=0.6872, reader_cost=0.0003 | ETA 07:47:17 2020-10-31 09:15:34 [INFO] [TRAIN] epoch=106, iter=39300/80000, loss=0.1830, lr=0.005448, batch_cost=0.6811, reader_cost=0.0003 | ETA 07:42:01 2020-10-31 09:16:42 [INFO] [TRAIN] epoch=106, iter=39400/80000, loss=0.1644, lr=0.005436, batch_cost=0.6859, reader_cost=0.0007 | ETA 07:44:06 2020-10-31 09:17:51 [INFO] [TRAIN] epoch=107, iter=39500/80000, loss=0.1799, lr=0.005424, batch_cost=0.6915, reader_cost=0.0090 | ETA 07:46:44 2020-10-31 09:19:00 [INFO] [TRAIN] epoch=107, iter=39600/80000, loss=0.1657, lr=0.005412, batch_cost=0.6842, reader_cost=0.0002 | ETA 07:40:39 2020-10-31 09:20:08 [INFO] [TRAIN] epoch=107, iter=39700/80000, loss=0.1497, lr=0.005400, batch_cost=0.6813, reader_cost=0.0002 | ETA 07:37:37 2020-10-31 09:21:17 [INFO] [TRAIN] epoch=107, iter=39800/80000, loss=0.1743, lr=0.005388, batch_cost=0.6863, reader_cost=0.0006 | ETA 07:39:50 2020-10-31 09:22:26 [INFO] [TRAIN] epoch=108, iter=39900/80000, loss=0.1734, lr=0.005376, batch_cost=0.6911, reader_cost=0.0103 | ETA 07:41:54 2020-10-31 09:23:35 [INFO] [TRAIN] epoch=108, iter=40000/80000, loss=0.1778, lr=0.005364, batch_cost=0.6874, reader_cost=0.0008 | ETA 07:38:17 2020-10-31 09:23:42 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-10-31 09:29:19 [INFO] [EVAL] #Images=500 mIoU=0.7838 Acc=0.9622 Kappa=0.9509 2020-10-31 09:29:19 [INFO] [EVAL] Category IoU: [0.9824 0.8599 0.9268 0.5277 0.6296 0.6499 0.7279 0.7911 0.9265 0.6626 0.9474 0.825 0.623 0.9534 0.8271 0.8639 0.7804 0.6066 0.7808] 2020-10-31 09:29:19 [INFO] [EVAL] Category Acc: [0.9918 0.9278 0.9548 0.8378 0.783 0.842 0.8401 0.9274 0.9545 0.8452 0.9677 0.8965 0.7446 0.9785 0.8871 0.9152 0.8447 0.8193 0.8805] 2020-10-31 09:29:23 [INFO] [EVAL] The model with the best validation mIoU (0.7838) was saved at iter 40000. 2020-10-31 09:30:31 [INFO] [TRAIN] epoch=108, iter=40100/80000, loss=0.1803, lr=0.005352, batch_cost=0.6781, reader_cost=0.0003 | ETA 07:30:54 2020-10-31 09:31:39 [INFO] [TRAIN] epoch=109, iter=40200/80000, loss=0.1747, lr=0.005340, batch_cost=0.6873, reader_cost=0.0086 | ETA 07:35:52 2020-10-31 09:32:48 [INFO] [TRAIN] epoch=109, iter=40300/80000, loss=0.1636, lr=0.005327, batch_cost=0.6868, reader_cost=0.0007 | ETA 07:34:24 2020-10-31 09:33:56 [INFO] [TRAIN] epoch=109, iter=40400/80000, loss=0.1612, lr=0.005315, batch_cost=0.6821, reader_cost=0.0003 | ETA 07:30:12 2020-10-31 09:35:04 [INFO] [TRAIN] epoch=109, iter=40500/80000, loss=0.1730, lr=0.005303, batch_cost=0.6807, reader_cost=0.0005 | ETA 07:28:06 2020-10-31 09:36:13 [INFO] [TRAIN] epoch=110, iter=40600/80000, loss=0.1540, lr=0.005291, batch_cost=0.6851, reader_cost=0.0092 | ETA 07:29:53 2020-10-31 09:37:21 [INFO] [TRAIN] epoch=110, iter=40700/80000, loss=0.1516, lr=0.005279, batch_cost=0.6778, reader_cost=0.0003 | ETA 07:23:57 2020-10-31 09:38:29 [INFO] [TRAIN] epoch=110, iter=40800/80000, loss=0.1706, lr=0.005267, batch_cost=0.6851, reader_cost=0.0003 | ETA 07:27:36 2020-10-31 09:39:38 [INFO] [TRAIN] epoch=110, iter=40900/80000, loss=0.1657, lr=0.005255, batch_cost=0.6873, reader_cost=0.0005 | ETA 07:27:51 2020-10-31 09:40:47 [INFO] [TRAIN] epoch=111, iter=41000/80000, loss=0.1656, lr=0.005243, batch_cost=0.6952, reader_cost=0.0091 | ETA 07:31:51 2020-10-31 09:41:56 [INFO] [TRAIN] epoch=111, iter=41100/80000, loss=0.1794, lr=0.005231, batch_cost=0.6868, reader_cost=0.0004 | ETA 07:25:15 2020-10-31 09:43:05 [INFO] [TRAIN] epoch=111, iter=41200/80000, loss=0.1715, lr=0.005219, batch_cost=0.6851, reader_cost=0.0004 | ETA 07:23:00 2020-10-31 09:44:14 [INFO] [TRAIN] epoch=112, iter=41300/80000, loss=0.1672, lr=0.005207, batch_cost=0.6947, reader_cost=0.0101 | ETA 07:28:04 2020-10-31 09:45:23 [INFO] [TRAIN] epoch=112, iter=41400/80000, loss=0.1613, lr=0.005195, batch_cost=0.6889, reader_cost=0.0004 | ETA 07:23:13 2020-10-31 09:46:32 [INFO] [TRAIN] epoch=112, iter=41500/80000, loss=0.1826, lr=0.005183, batch_cost=0.6916, reader_cost=0.0006 | ETA 07:23:45 2020-10-31 09:47:41 [INFO] [TRAIN] epoch=112, iter=41600/80000, loss=0.1548, lr=0.005171, batch_cost=0.6840, reader_cost=0.0004 | ETA 07:17:45 2020-10-31 09:48:50 [INFO] [TRAIN] epoch=113, iter=41700/80000, loss=0.1601, lr=0.005158, batch_cost=0.6975, reader_cost=0.0099 | ETA 07:25:15 2020-10-31 09:49:59 [INFO] [TRAIN] epoch=113, iter=41800/80000, loss=0.1569, lr=0.005146, batch_cost=0.6887, reader_cost=0.0003 | ETA 07:18:27 2020-10-31 09:51:08 [INFO] [TRAIN] epoch=113, iter=41900/80000, loss=0.1581, lr=0.005134, batch_cost=0.6855, reader_cost=0.0002 | ETA 07:15:17 2020-10-31 09:52:17 [INFO] [TRAIN] epoch=113, iter=42000/80000, loss=0.1600, lr=0.005122, batch_cost=0.6888, reader_cost=0.0002 | ETA 07:16:15 2020-10-31 09:53:26 [INFO] [TRAIN] epoch=114, iter=42100/80000, loss=0.1701, lr=0.005110, batch_cost=0.6929, reader_cost=0.0092 | ETA 07:17:40 2020-10-31 09:54:34 [INFO] [TRAIN] epoch=114, iter=42200/80000, loss=0.1549, lr=0.005098, batch_cost=0.6806, reader_cost=0.0004 | ETA 07:08:46 2020-10-31 09:55:43 [INFO] [TRAIN] epoch=114, iter=42300/80000, loss=0.1556, lr=0.005086, batch_cost=0.6855, reader_cost=0.0005 | ETA 07:10:44 2020-10-31 09:56:51 [INFO] [TRAIN] epoch=114, iter=42400/80000, loss=0.1567, lr=0.005074, batch_cost=0.6874, reader_cost=0.0002 | ETA 07:10:45 2020-10-31 09:58:01 [INFO] [TRAIN] epoch=115, iter=42500/80000, loss=0.1441, lr=0.005062, batch_cost=0.6923, reader_cost=0.0090 | ETA 07:12:40 2020-10-31 09:59:09 [INFO] [TRAIN] epoch=115, iter=42600/80000, loss=0.2014, lr=0.005049, batch_cost=0.6855, reader_cost=0.0005 | ETA 07:07:16 2020-10-31 10:00:18 [INFO] [TRAIN] epoch=115, iter=42700/80000, loss=0.1629, lr=0.005037, batch_cost=0.6907, reader_cost=0.0006 | ETA 07:09:23 2020-10-31 10:01:28 [INFO] [TRAIN] epoch=116, iter=42800/80000, loss=0.1754, lr=0.005025, batch_cost=0.6952, reader_cost=0.0099 | ETA 07:11:00 2020-10-31 10:02:36 [INFO] [TRAIN] epoch=116, iter=42900/80000, loss=0.1481, lr=0.005013, batch_cost=0.6828, reader_cost=0.0007 | ETA 07:02:13 2020-10-31 10:03:45 [INFO] [TRAIN] epoch=116, iter=43000/80000, loss=0.1603, lr=0.005001, batch_cost=0.6856, reader_cost=0.0010 | ETA 07:02:46 2020-10-31 10:04:53 [INFO] [TRAIN] epoch=116, iter=43100/80000, loss=0.1434, lr=0.004989, batch_cost=0.6842, reader_cost=0.0002 | ETA 07:00:47 2020-10-31 10:06:02 [INFO] [TRAIN] epoch=117, iter=43200/80000, loss=0.1511, lr=0.004977, batch_cost=0.6947, reader_cost=0.0109 | ETA 07:06:03 2020-10-31 10:07:11 [INFO] [TRAIN] epoch=117, iter=43300/80000, loss=0.1695, lr=0.004964, batch_cost=0.6851, reader_cost=0.0004 | ETA 06:59:04 2020-10-31 10:08:19 [INFO] [TRAIN] epoch=117, iter=43400/80000, loss=0.1702, lr=0.004952, batch_cost=0.6786, reader_cost=0.0005 | ETA 06:53:55 2020-10-31 10:09:27 [INFO] [TRAIN] epoch=117, iter=43500/80000, loss=0.1776, lr=0.004940, batch_cost=0.6846, reader_cost=0.0004 | ETA 06:56:27 2020-10-31 10:10:37 [INFO] [TRAIN] epoch=118, iter=43600/80000, loss=0.1645, lr=0.004928, batch_cost=0.6938, reader_cost=0.0091 | ETA 07:00:55 2020-10-31 10:11:45 [INFO] [TRAIN] epoch=118, iter=43700/80000, loss=0.1570, lr=0.004916, batch_cost=0.6842, reader_cost=0.0003 | ETA 06:53:54 2020-10-31 10:12:53 [INFO] [TRAIN] epoch=118, iter=43800/80000, loss=0.1478, lr=0.004904, batch_cost=0.6821, reader_cost=0.0003 | ETA 06:51:30 2020-10-31 10:14:03 [INFO] [TRAIN] epoch=119, iter=43900/80000, loss=0.1919, lr=0.004891, batch_cost=0.6942, reader_cost=0.0094 | ETA 06:57:40 2020-10-31 10:15:11 [INFO] [TRAIN] epoch=119, iter=44000/80000, loss=0.1641, lr=0.004879, batch_cost=0.6842, reader_cost=0.0002 | ETA 06:50:29 2020-10-31 10:16:20 [INFO] [TRAIN] epoch=119, iter=44100/80000, loss=0.1771, lr=0.004867, batch_cost=0.6887, reader_cost=0.0006 | ETA 06:52:02 2020-10-31 10:17:29 [INFO] [TRAIN] epoch=119, iter=44200/80000, loss=0.1881, lr=0.004855, batch_cost=0.6859, reader_cost=0.0005 | ETA 06:49:14 2020-10-31 10:18:38 [INFO] [TRAIN] epoch=120, iter=44300/80000, loss=0.1678, lr=0.004843, batch_cost=0.6981, reader_cost=0.0102 | ETA 06:55:23 2020-10-31 10:19:47 [INFO] [TRAIN] epoch=120, iter=44400/80000, loss=0.1731, lr=0.004831, batch_cost=0.6905, reader_cost=0.0007 | ETA 06:49:40 2020-10-31 10:20:56 [INFO] [TRAIN] epoch=120, iter=44500/80000, loss=0.1545, lr=0.004818, batch_cost=0.6868, reader_cost=0.0007 | ETA 06:46:20 2020-10-31 10:22:04 [INFO] [TRAIN] epoch=120, iter=44600/80000, loss=0.1581, lr=0.004806, batch_cost=0.6836, reader_cost=0.0005 | ETA 06:43:20 2020-10-31 10:23:14 [INFO] [TRAIN] epoch=121, iter=44700/80000, loss=0.1655, lr=0.004794, batch_cost=0.6982, reader_cost=0.0109 | ETA 06:50:46 2020-10-31 10:24:23 [INFO] [TRAIN] epoch=121, iter=44800/80000, loss=0.1681, lr=0.004782, batch_cost=0.6844, reader_cost=0.0004 | ETA 06:41:29 2020-10-31 10:25:31 [INFO] [TRAIN] epoch=121, iter=44900/80000, loss=0.1576, lr=0.004770, batch_cost=0.6851, reader_cost=0.0003 | ETA 06:40:46 2020-10-31 10:26:40 [INFO] [TRAIN] epoch=121, iter=45000/80000, loss=0.1560, lr=0.004757, batch_cost=0.6853, reader_cost=0.0002 | ETA 06:39:46 2020-10-31 10:27:49 [INFO] [TRAIN] epoch=122, iter=45100/80000, loss=0.1700, lr=0.004745, batch_cost=0.6941, reader_cost=0.0096 | ETA 06:43:43 2020-10-31 10:28:57 [INFO] [TRAIN] epoch=122, iter=45200/80000, loss=0.1650, lr=0.004733, batch_cost=0.6829, reader_cost=0.0003 | ETA 06:36:03 2020-10-31 10:30:06 [INFO] [TRAIN] epoch=122, iter=45300/80000, loss=0.1842, lr=0.004721, batch_cost=0.6885, reader_cost=0.0005 | ETA 06:38:12 2020-10-31 10:31:16 [INFO] [TRAIN] epoch=123, iter=45400/80000, loss=0.1510, lr=0.004709, batch_cost=0.6960, reader_cost=0.0088 | ETA 06:41:23 2020-10-31 10:32:25 [INFO] [TRAIN] epoch=123, iter=45500/80000, loss=0.1482, lr=0.004696, batch_cost=0.6873, reader_cost=0.0005 | ETA 06:35:10 2020-10-31 10:33:34 [INFO] [TRAIN] epoch=123, iter=45600/80000, loss=0.1695, lr=0.004684, batch_cost=0.6901, reader_cost=0.0005 | ETA 06:35:38 2020-10-31 10:34:42 [INFO] [TRAIN] epoch=123, iter=45700/80000, loss=0.1721, lr=0.004672, batch_cost=0.6863, reader_cost=0.0004 | ETA 06:32:21 2020-10-31 10:35:51 [INFO] [TRAIN] epoch=124, iter=45800/80000, loss=0.1862, lr=0.004660, batch_cost=0.6909, reader_cost=0.0109 | ETA 06:33:47 2020-10-31 10:37:00 [INFO] [TRAIN] epoch=124, iter=45900/80000, loss=0.1543, lr=0.004647, batch_cost=0.6829, reader_cost=0.0002 | ETA 06:28:06 2020-10-31 10:38:08 [INFO] [TRAIN] epoch=124, iter=46000/80000, loss=0.1563, lr=0.004635, batch_cost=0.6877, reader_cost=0.0002 | ETA 06:29:41 2020-10-31 10:39:18 [INFO] [TRAIN] epoch=124, iter=46100/80000, loss=0.1428, lr=0.004623, batch_cost=0.6920, reader_cost=0.0002 | ETA 06:30:59 2020-10-31 10:40:28 [INFO] [TRAIN] epoch=125, iter=46200/80000, loss=0.1619, lr=0.004611, batch_cost=0.7020, reader_cost=0.0090 | ETA 06:35:27 2020-10-31 10:41:36 [INFO] [TRAIN] epoch=125, iter=46300/80000, loss=0.1661, lr=0.004598, batch_cost=0.6797, reader_cost=0.0002 | ETA 06:21:44 2020-10-31 10:42:44 [INFO] [TRAIN] epoch=125, iter=46400/80000, loss=0.1516, lr=0.004586, batch_cost=0.6841, reader_cost=0.0003 | ETA 06:23:06 2020-10-31 10:43:52 [INFO] [TRAIN] epoch=125, iter=46500/80000, loss=0.1651, lr=0.004574, batch_cost=0.6808, reader_cost=0.0003 | ETA 06:20:07 2020-10-31 10:45:02 [INFO] [TRAIN] epoch=126, iter=46600/80000, loss=0.1563, lr=0.004562, batch_cost=0.6974, reader_cost=0.0090 | ETA 06:28:14 2020-10-31 10:46:11 [INFO] [TRAIN] epoch=126, iter=46700/80000, loss=0.1649, lr=0.004549, batch_cost=0.6843, reader_cost=0.0002 | ETA 06:19:47 2020-10-31 10:47:19 [INFO] [TRAIN] epoch=126, iter=46800/80000, loss=0.1415, lr=0.004537, batch_cost=0.6875, reader_cost=0.0002 | ETA 06:20:24 2020-10-31 10:48:28 [INFO] [TRAIN] epoch=127, iter=46900/80000, loss=0.1500, lr=0.004525, batch_cost=0.6918, reader_cost=0.0093 | ETA 06:21:39 2020-10-31 10:49:36 [INFO] [TRAIN] epoch=127, iter=47000/80000, loss=0.1796, lr=0.004513, batch_cost=0.6782, reader_cost=0.0004 | ETA 06:13:02 2020-10-31 10:50:45 [INFO] [TRAIN] epoch=127, iter=47100/80000, loss=0.1620, lr=0.004500, batch_cost=0.6870, reader_cost=0.0004 | ETA 06:16:43 2020-10-31 10:51:53 [INFO] [TRAIN] epoch=127, iter=47200/80000, loss=0.1469, lr=0.004488, batch_cost=0.6810, reader_cost=0.0008 | ETA 06:12:15 2020-10-31 10:53:02 [INFO] [TRAIN] epoch=128, iter=47300/80000, loss=0.1498, lr=0.004476, batch_cost=0.6880, reader_cost=0.0088 | ETA 06:14:56 2020-10-31 10:54:10 [INFO] [TRAIN] epoch=128, iter=47400/80000, loss=0.1599, lr=0.004463, batch_cost=0.6829, reader_cost=0.0002 | ETA 06:11:03 2020-10-31 10:55:19 [INFO] [TRAIN] epoch=128, iter=47500/80000, loss=0.1552, lr=0.004451, batch_cost=0.6854, reader_cost=0.0004 | ETA 06:11:13 2020-10-31 10:56:27 [INFO] [TRAIN] epoch=128, iter=47600/80000, loss=0.1468, lr=0.004439, batch_cost=0.6783, reader_cost=0.0002 | ETA 06:06:16 2020-10-31 10:57:35 [INFO] [TRAIN] epoch=129, iter=47700/80000, loss=0.1554, lr=0.004427, batch_cost=0.6890, reader_cost=0.0100 | ETA 06:10:55 2020-10-31 10:58:44 [INFO] [TRAIN] epoch=129, iter=47800/80000, loss=0.1469, lr=0.004414, batch_cost=0.6893, reader_cost=0.0003 | ETA 06:09:55 2020-10-31 10:59:52 [INFO] [TRAIN] epoch=129, iter=47900/80000, loss=0.1409, lr=0.004402, batch_cost=0.6794, reader_cost=0.0004 | ETA 06:03:29 2020-10-31 11:01:01 [INFO] [TRAIN] epoch=130, iter=48000/80000, loss=0.1648, lr=0.004390, batch_cost=0.6897, reader_cost=0.0086 | ETA 06:07:51 2020-10-31 11:01:09 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-10-31 11:06:47 [INFO] [EVAL] #Images=500 mIoU=0.7843 Acc=0.9609 Kappa=0.9494 2020-10-31 11:06:47 [INFO] [EVAL] Category IoU: [0.98 0.8522 0.9267 0.5747 0.6132 0.6619 0.7261 0.8046 0.9268 0.6656 0.9482 0.835 0.6408 0.9448 0.7383 0.8644 0.7901 0.6398 0.7687] 2020-10-31 11:06:47 [INFO] [EVAL] Category Acc: [0.992 0.9194 0.9637 0.7298 0.7973 0.7954 0.8154 0.8924 0.9574 0.8103 0.9649 0.9037 0.749 0.9765 0.7932 0.9089 0.9557 0.8047 0.8294] 2020-10-31 11:06:51 [INFO] [EVAL] The model with the best validation mIoU (0.7843) was saved at iter 48000. 2020-10-31 11:07:58 [INFO] [TRAIN] epoch=130, iter=48100/80000, loss=0.1532, lr=0.004377, batch_cost=0.6714, reader_cost=0.0003 | ETA 05:56:58 2020-10-31 11:09:05 [INFO] [TRAIN] epoch=130, iter=48200/80000, loss=0.1543, lr=0.004365, batch_cost=0.6743, reader_cost=0.0002 | ETA 05:57:22 2020-10-31 11:10:14 [INFO] [TRAIN] epoch=130, iter=48300/80000, loss=0.1412, lr=0.004353, batch_cost=0.6840, reader_cost=0.0004 | ETA 06:01:24 2020-10-31 11:11:22 [INFO] [TRAIN] epoch=131, iter=48400/80000, loss=0.1440, lr=0.004340, batch_cost=0.6863, reader_cost=0.0086 | ETA 06:01:26 2020-10-31 11:12:30 [INFO] [TRAIN] epoch=131, iter=48500/80000, loss=0.1620, lr=0.004328, batch_cost=0.6793, reader_cost=0.0003 | ETA 05:56:39 2020-10-31 11:13:38 [INFO] [TRAIN] epoch=131, iter=48600/80000, loss=0.1538, lr=0.004316, batch_cost=0.6773, reader_cost=0.0001 | ETA 05:54:26 2020-10-31 11:14:46 [INFO] [TRAIN] epoch=131, iter=48700/80000, loss=0.1390, lr=0.004303, batch_cost=0.6830, reader_cost=0.0002 | ETA 05:56:18 2020-10-31 11:15:55 [INFO] [TRAIN] epoch=132, iter=48800/80000, loss=0.1546, lr=0.004291, batch_cost=0.6877, reader_cost=0.0088 | ETA 05:57:37 2020-10-31 11:17:03 [INFO] [TRAIN] epoch=132, iter=48900/80000, loss=0.1517, lr=0.004279, batch_cost=0.6762, reader_cost=0.0001 | ETA 05:50:31 2020-10-31 11:18:11 [INFO] [TRAIN] epoch=132, iter=49000/80000, loss=0.1519, lr=0.004266, batch_cost=0.6796, reader_cost=0.0003 | ETA 05:51:06 2020-10-31 11:19:19 [INFO] [TRAIN] epoch=132, iter=49100/80000, loss=0.1513, lr=0.004254, batch_cost=0.6864, reader_cost=0.0005 | ETA 05:53:28 2020-10-31 11:20:29 [INFO] [TRAIN] epoch=133, iter=49200/80000, loss=0.1577, lr=0.004241, batch_cost=0.6969, reader_cost=0.0082 | ETA 05:57:43 2020-10-31 11:21:37 [INFO] [TRAIN] epoch=133, iter=49300/80000, loss=0.1576, lr=0.004229, batch_cost=0.6851, reader_cost=0.0003 | ETA 05:50:31 2020-10-31 11:22:46 [INFO] [TRAIN] epoch=133, iter=49400/80000, loss=0.1428, lr=0.004217, batch_cost=0.6876, reader_cost=0.0006 | ETA 05:50:41 2020-10-31 11:23:56 [INFO] [TRAIN] epoch=134, iter=49500/80000, loss=0.1814, lr=0.004204, batch_cost=0.7002, reader_cost=0.0108 | ETA 05:55:56 2020-10-31 11:25:04 [INFO] [TRAIN] epoch=134, iter=49600/80000, loss=0.1535, lr=0.004192, batch_cost=0.6821, reader_cost=0.0003 | ETA 05:45:35 2020-10-31 11:26:13 [INFO] [TRAIN] epoch=134, iter=49700/80000, loss=0.1641, lr=0.004180, batch_cost=0.6828, reader_cost=0.0005 | ETA 05:44:49 2020-10-31 11:27:21 [INFO] [TRAIN] epoch=134, iter=49800/80000, loss=0.1498, lr=0.004167, batch_cost=0.6834, reader_cost=0.0002 | ETA 05:43:59 2020-10-31 11:28:30 [INFO] [TRAIN] epoch=135, iter=49900/80000, loss=0.1388, lr=0.004155, batch_cost=0.6897, reader_cost=0.0086 | ETA 05:45:58 2020-10-31 11:29:38 [INFO] [TRAIN] epoch=135, iter=50000/80000, loss=0.1442, lr=0.004142, batch_cost=0.6809, reader_cost=0.0003 | ETA 05:40:27 2020-10-31 11:30:47 [INFO] [TRAIN] epoch=135, iter=50100/80000, loss=0.1619, lr=0.004130, batch_cost=0.6852, reader_cost=0.0002 | ETA 05:41:28 2020-10-31 11:31:55 [INFO] [TRAIN] epoch=135, iter=50200/80000, loss=0.1645, lr=0.004118, batch_cost=0.6789, reader_cost=0.0006 | ETA 05:37:11 2020-10-31 11:33:04 [INFO] [TRAIN] epoch=136, iter=50300/80000, loss=0.1478, lr=0.004105, batch_cost=0.6937, reader_cost=0.0089 | ETA 05:43:24 2020-10-31 11:34:12 [INFO] [TRAIN] epoch=136, iter=50400/80000, loss=0.1539, lr=0.004093, batch_cost=0.6854, reader_cost=0.0003 | ETA 05:38:08 2020-10-31 11:35:21 [INFO] [TRAIN] epoch=136, iter=50500/80000, loss=0.1423, lr=0.004080, batch_cost=0.6819, reader_cost=0.0001 | ETA 05:35:14 2020-10-31 11:36:30 [INFO] [TRAIN] epoch=137, iter=50600/80000, loss=0.1511, lr=0.004068, batch_cost=0.6920, reader_cost=0.0077 | ETA 05:39:06 2020-10-31 11:37:38 [INFO] [TRAIN] epoch=137, iter=50700/80000, loss=0.1509, lr=0.004056, batch_cost=0.6814, reader_cost=0.0002 | ETA 05:32:44 2020-10-31 11:38:45 [INFO] [TRAIN] epoch=137, iter=50800/80000, loss=0.1483, lr=0.004043, batch_cost=0.6709, reader_cost=0.0001 | ETA 05:26:29 2020-10-31 11:39:52 [INFO] [TRAIN] epoch=137, iter=50900/80000, loss=0.1446, lr=0.004031, batch_cost=0.6649, reader_cost=0.0001 | ETA 05:22:28 2020-10-31 11:40:59 [INFO] [TRAIN] epoch=138, iter=51000/80000, loss=0.1523, lr=0.004018, batch_cost=0.6775, reader_cost=0.0087 | ETA 05:27:28 2020-10-31 11:42:06 [INFO] [TRAIN] epoch=138, iter=51100/80000, loss=0.1370, lr=0.004006, batch_cost=0.6675, reader_cost=0.0001 | ETA 05:21:31 2020-10-31 11:43:14 [INFO] [TRAIN] epoch=138, iter=51200/80000, loss=0.1583, lr=0.003993, batch_cost=0.6779, reader_cost=0.0006 | ETA 05:25:23 2020-10-31 11:44:22 [INFO] [TRAIN] epoch=138, iter=51300/80000, loss=0.1493, lr=0.003981, batch_cost=0.6792, reader_cost=0.0003 | ETA 05:24:51 2020-10-31 11:45:31 [INFO] [TRAIN] epoch=139, iter=51400/80000, loss=0.1321, lr=0.003968, batch_cost=0.6875, reader_cost=0.0093 | ETA 05:27:43 2020-10-31 11:46:38 [INFO] [TRAIN] epoch=139, iter=51500/80000, loss=0.1552, lr=0.003956, batch_cost=0.6778, reader_cost=0.0005 | ETA 05:21:56 2020-10-31 11:47:47 [INFO] [TRAIN] epoch=139, iter=51600/80000, loss=0.1454, lr=0.003944, batch_cost=0.6833, reader_cost=0.0004 | ETA 05:23:24 2020-10-31 11:48:55 [INFO] [TRAIN] epoch=139, iter=51700/80000, loss=0.1571, lr=0.003931, batch_cost=0.6816, reader_cost=0.0003 | ETA 05:21:30 2020-10-31 11:50:04 [INFO] [TRAIN] epoch=140, iter=51800/80000, loss=0.1491, lr=0.003919, batch_cost=0.6912, reader_cost=0.0091 | ETA 05:24:53 2020-10-31 11:51:12 [INFO] [TRAIN] epoch=140, iter=51900/80000, loss=0.1551, lr=0.003906, batch_cost=0.6815, reader_cost=0.0006 | ETA 05:19:10 2020-10-31 11:52:20 [INFO] [TRAIN] epoch=140, iter=52000/80000, loss=0.1560, lr=0.003894, batch_cost=0.6760, reader_cost=0.0007 | ETA 05:15:28 2020-10-31 11:53:29 [INFO] [TRAIN] epoch=141, iter=52100/80000, loss=0.1552, lr=0.003881, batch_cost=0.6899, reader_cost=0.0089 | ETA 05:20:48 2020-10-31 11:54:37 [INFO] [TRAIN] epoch=141, iter=52200/80000, loss=0.1663, lr=0.003869, batch_cost=0.6825, reader_cost=0.0004 | ETA 05:16:14 2020-10-31 11:55:45 [INFO] [TRAIN] epoch=141, iter=52300/80000, loss=0.1618, lr=0.003856, batch_cost=0.6841, reader_cost=0.0005 | ETA 05:15:49 2020-10-31 11:56:53 [INFO] [TRAIN] epoch=141, iter=52400/80000, loss=0.2225, lr=0.003844, batch_cost=0.6754, reader_cost=0.0003 | ETA 05:10:41 2020-10-31 11:58:02 [INFO] [TRAIN] epoch=142, iter=52500/80000, loss=0.1378, lr=0.003831, batch_cost=0.6889, reader_cost=0.0089 | ETA 05:15:44 2020-10-31 11:59:09 [INFO] [TRAIN] epoch=142, iter=52600/80000, loss=0.1577, lr=0.003819, batch_cost=0.6763, reader_cost=0.0003 | ETA 05:08:51 2020-10-31 12:00:18 [INFO] [TRAIN] epoch=142, iter=52700/80000, loss=0.1486, lr=0.003806, batch_cost=0.6832, reader_cost=0.0006 | ETA 05:10:50 2020-10-31 12:01:26 [INFO] [TRAIN] epoch=142, iter=52800/80000, loss=0.1443, lr=0.003794, batch_cost=0.6803, reader_cost=0.0004 | ETA 05:08:25 2020-10-31 12:02:35 [INFO] [TRAIN] epoch=143, iter=52900/80000, loss=0.1395, lr=0.003781, batch_cost=0.6933, reader_cost=0.0096 | ETA 05:13:08 2020-10-31 12:03:44 [INFO] [TRAIN] epoch=143, iter=53000/80000, loss=0.1454, lr=0.003769, batch_cost=0.6869, reader_cost=0.0007 | ETA 05:09:05 2020-10-31 12:04:53 [INFO] [TRAIN] epoch=143, iter=53100/80000, loss=0.1402, lr=0.003756, batch_cost=0.6878, reader_cost=0.0007 | ETA 05:08:22 2020-10-31 12:06:02 [INFO] [TRAIN] epoch=144, iter=53200/80000, loss=0.1569, lr=0.003744, batch_cost=0.6953, reader_cost=0.0090 | ETA 05:10:34 2020-10-31 12:07:10 [INFO] [TRAIN] epoch=144, iter=53300/80000, loss=0.1478, lr=0.003731, batch_cost=0.6815, reader_cost=0.0004 | ETA 05:03:14 2020-10-31 12:08:18 [INFO] [TRAIN] epoch=144, iter=53400/80000, loss=0.1638, lr=0.003718, batch_cost=0.6778, reader_cost=0.0005 | ETA 05:00:30 2020-10-31 12:09:26 [INFO] [TRAIN] epoch=144, iter=53500/80000, loss=0.1619, lr=0.003706, batch_cost=0.6802, reader_cost=0.0004 | ETA 05:00:26 2020-10-31 12:10:35 [INFO] [TRAIN] epoch=145, iter=53600/80000, loss=0.1575, lr=0.003693, batch_cost=0.6909, reader_cost=0.0086 | ETA 05:04:00 2020-10-31 12:11:43 [INFO] [TRAIN] epoch=145, iter=53700/80000, loss=0.1606, lr=0.003681, batch_cost=0.6805, reader_cost=0.0003 | ETA 04:58:17 2020-10-31 12:12:51 [INFO] [TRAIN] epoch=145, iter=53800/80000, loss=0.1535, lr=0.003668, batch_cost=0.6750, reader_cost=0.0002 | ETA 04:54:44 2020-10-31 12:13:58 [INFO] [TRAIN] epoch=145, iter=53900/80000, loss=0.1442, lr=0.003656, batch_cost=0.6769, reader_cost=0.0002 | ETA 04:54:27 2020-10-31 12:15:07 [INFO] [TRAIN] epoch=146, iter=54000/80000, loss=0.1480, lr=0.003643, batch_cost=0.6890, reader_cost=0.0092 | ETA 04:58:34 2020-10-31 12:16:16 [INFO] [TRAIN] epoch=146, iter=54100/80000, loss=0.1314, lr=0.003631, batch_cost=0.6841, reader_cost=0.0010 | ETA 04:55:17 2020-10-31 12:17:25 [INFO] [TRAIN] epoch=146, iter=54200/80000, loss=0.1399, lr=0.003618, batch_cost=0.6893, reader_cost=0.0006 | ETA 04:56:24 2020-10-31 12:18:33 [INFO] [TRAIN] epoch=146, iter=54300/80000, loss=0.1422, lr=0.003605, batch_cost=0.6816, reader_cost=0.0006 | ETA 04:51:56 2020-10-31 12:19:42 [INFO] [TRAIN] epoch=147, iter=54400/80000, loss=0.1333, lr=0.003593, batch_cost=0.6909, reader_cost=0.0104 | ETA 04:54:48 2020-10-31 12:20:50 [INFO] [TRAIN] epoch=147, iter=54500/80000, loss=0.1377, lr=0.003580, batch_cost=0.6835, reader_cost=0.0004 | ETA 04:50:28 2020-10-31 12:21:59 [INFO] [TRAIN] epoch=147, iter=54600/80000, loss=0.1481, lr=0.003568, batch_cost=0.6839, reader_cost=0.0007 | ETA 04:49:30 2020-10-31 12:23:08 [INFO] [TRAIN] epoch=148, iter=54700/80000, loss=0.2349, lr=0.003555, batch_cost=0.6921, reader_cost=0.0094 | ETA 04:51:50 2020-10-31 12:24:16 [INFO] [TRAIN] epoch=148, iter=54800/80000, loss=0.1665, lr=0.003542, batch_cost=0.6838, reader_cost=0.0005 | ETA 04:47:10 2020-10-31 12:25:24 [INFO] [TRAIN] epoch=148, iter=54900/80000, loss=0.1552, lr=0.003530, batch_cost=0.6808, reader_cost=0.0005 | ETA 04:44:48 2020-10-31 12:26:32 [INFO] [TRAIN] epoch=148, iter=55000/80000, loss=0.1350, lr=0.003517, batch_cost=0.6800, reader_cost=0.0002 | ETA 04:43:20 2020-10-31 12:27:42 [INFO] [TRAIN] epoch=149, iter=55100/80000, loss=0.1364, lr=0.003504, batch_cost=0.6940, reader_cost=0.0109 | ETA 04:47:59 2020-10-31 12:28:50 [INFO] [TRAIN] epoch=149, iter=55200/80000, loss=0.1602, lr=0.003492, batch_cost=0.6867, reader_cost=0.0002 | ETA 04:43:50 2020-10-31 12:29:58 [INFO] [TRAIN] epoch=149, iter=55300/80000, loss=0.1760, lr=0.003479, batch_cost=0.6788, reader_cost=0.0004 | ETA 04:39:25 2020-10-31 12:31:07 [INFO] [TRAIN] epoch=149, iter=55400/80000, loss=0.1435, lr=0.003467, batch_cost=0.6824, reader_cost=0.0002 | ETA 04:39:46 2020-10-31 12:32:16 [INFO] [TRAIN] epoch=150, iter=55500/80000, loss=0.1465, lr=0.003454, batch_cost=0.6926, reader_cost=0.0094 | ETA 04:42:48 2020-10-31 12:33:24 [INFO] [TRAIN] epoch=150, iter=55600/80000, loss=0.1553, lr=0.003441, batch_cost=0.6853, reader_cost=0.0005 | ETA 04:38:41 2020-10-31 12:34:33 [INFO] [TRAIN] epoch=150, iter=55700/80000, loss=0.1406, lr=0.003429, batch_cost=0.6855, reader_cost=0.0005 | ETA 04:37:37 2020-10-31 12:35:41 [INFO] [TRAIN] epoch=150, iter=55800/80000, loss=0.1712, lr=0.003416, batch_cost=0.6843, reader_cost=0.0004 | ETA 04:36:01 2020-10-31 12:36:50 [INFO] [TRAIN] epoch=151, iter=55900/80000, loss=0.1488, lr=0.003403, batch_cost=0.6883, reader_cost=0.0099 | ETA 04:36:29 2020-10-31 12:37:59 [INFO] [TRAIN] epoch=151, iter=56000/80000, loss=0.1492, lr=0.003391, batch_cost=0.6837, reader_cost=0.0004 | ETA 04:33:28 2020-10-31 12:38:06 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-10-31 12:43:45 [INFO] [EVAL] #Images=500 mIoU=0.7969 Acc=0.9638 Kappa=0.9531 2020-10-31 12:43:45 [INFO] [EVAL] Category IoU: [0.9827 0.865 0.932 0.6034 0.639 0.6613 0.7245 0.8027 0.9269 0.6704 0.9487 0.8324 0.628 0.9545 0.8293 0.8984 0.8348 0.6241 0.7826] 2020-10-31 12:43:45 [INFO] [EVAL] Category Acc: [0.9928 0.9301 0.9606 0.8548 0.798 0.8444 0.8575 0.9196 0.9543 0.8187 0.9667 0.8903 0.8053 0.9735 0.9362 0.9432 0.8945 0.718 0.8956] 2020-10-31 12:43:49 [INFO] [EVAL] The model with the best validation mIoU (0.7969) was saved at iter 56000. 2020-10-31 12:44:56 [INFO] [TRAIN] epoch=151, iter=56100/80000, loss=0.1857, lr=0.003378, batch_cost=0.6670, reader_cost=0.0002 | ETA 04:25:40 2020-10-31 12:46:04 [INFO] [TRAIN] epoch=152, iter=56200/80000, loss=0.1668, lr=0.003365, batch_cost=0.6800, reader_cost=0.0100 | ETA 04:29:43 2020-10-31 12:47:12 [INFO] [TRAIN] epoch=152, iter=56300/80000, loss=0.1479, lr=0.003353, batch_cost=0.6806, reader_cost=0.0006 | ETA 04:28:49 2020-10-31 12:48:19 [INFO] [TRAIN] epoch=152, iter=56400/80000, loss=0.1352, lr=0.003340, batch_cost=0.6784, reader_cost=0.0010 | ETA 04:26:50 2020-10-31 12:49:28 [INFO] [TRAIN] epoch=152, iter=56500/80000, loss=0.1518, lr=0.003327, batch_cost=0.6839, reader_cost=0.0006 | ETA 04:27:51 2020-10-31 12:50:37 [INFO] [TRAIN] epoch=153, iter=56600/80000, loss=0.1400, lr=0.003314, batch_cost=0.6888, reader_cost=0.0088 | ETA 04:28:37 2020-10-31 12:51:45 [INFO] [TRAIN] epoch=153, iter=56700/80000, loss=0.1587, lr=0.003302, batch_cost=0.6836, reader_cost=0.0010 | ETA 04:25:28 2020-10-31 12:52:53 [INFO] [TRAIN] epoch=153, iter=56800/80000, loss=0.1491, lr=0.003289, batch_cost=0.6811, reader_cost=0.0006 | ETA 04:23:21 2020-10-31 12:54:01 [INFO] [TRAIN] epoch=153, iter=56900/80000, loss=0.1549, lr=0.003276, batch_cost=0.6791, reader_cost=0.0007 | ETA 04:21:27 2020-10-31 12:55:10 [INFO] [TRAIN] epoch=154, iter=57000/80000, loss=0.1492, lr=0.003264, batch_cost=0.6900, reader_cost=0.0092 | ETA 04:24:30 2020-10-31 12:56:18 [INFO] [TRAIN] epoch=154, iter=57100/80000, loss=0.1566, lr=0.003251, batch_cost=0.6766, reader_cost=0.0005 | ETA 04:18:14 2020-10-31 12:57:26 [INFO] [TRAIN] epoch=154, iter=57200/80000, loss=0.1589, lr=0.003238, batch_cost=0.6779, reader_cost=0.0002 | ETA 04:17:35 2020-10-31 12:58:35 [INFO] [TRAIN] epoch=155, iter=57300/80000, loss=0.1641, lr=0.003225, batch_cost=0.6902, reader_cost=0.0094 | ETA 04:21:07 2020-10-31 12:59:43 [INFO] [TRAIN] epoch=155, iter=57400/80000, loss=0.1594, lr=0.003213, batch_cost=0.6806, reader_cost=0.0005 | ETA 04:16:22 2020-10-31 13:00:51 [INFO] [TRAIN] epoch=155, iter=57500/80000, loss=0.1559, lr=0.003200, batch_cost=0.6879, reader_cost=0.0006 | ETA 04:17:58 2020-10-31 13:02:00 [INFO] [TRAIN] epoch=155, iter=57600/80000, loss=0.1440, lr=0.003187, batch_cost=0.6829, reader_cost=0.0003 | ETA 04:14:57 2020-10-31 13:03:09 [INFO] [TRAIN] epoch=156, iter=57700/80000, loss=0.1331, lr=0.003174, batch_cost=0.6913, reader_cost=0.0097 | ETA 04:16:55 2020-10-31 13:04:17 [INFO] [TRAIN] epoch=156, iter=57800/80000, loss=0.1329, lr=0.003162, batch_cost=0.6790, reader_cost=0.0003 | ETA 04:11:13 2020-10-31 13:05:25 [INFO] [TRAIN] epoch=156, iter=57900/80000, loss=0.1631, lr=0.003149, batch_cost=0.6839, reader_cost=0.0003 | ETA 04:11:53 2020-10-31 13:06:33 [INFO] [TRAIN] epoch=156, iter=58000/80000, loss=0.1342, lr=0.003136, batch_cost=0.6813, reader_cost=0.0006 | ETA 04:09:48 2020-10-31 13:07:42 [INFO] [TRAIN] epoch=157, iter=58100/80000, loss=0.1365, lr=0.003123, batch_cost=0.6882, reader_cost=0.0086 | ETA 04:11:11 2020-10-31 13:08:51 [INFO] [TRAIN] epoch=157, iter=58200/80000, loss=0.1444, lr=0.003110, batch_cost=0.6882, reader_cost=0.0002 | ETA 04:10:03 2020-10-31 13:09:59 [INFO] [TRAIN] epoch=157, iter=58300/80000, loss=0.1460, lr=0.003098, batch_cost=0.6834, reader_cost=0.0002 | ETA 04:07:10 2020-10-31 13:11:07 [INFO] [TRAIN] epoch=157, iter=58400/80000, loss=0.1494, lr=0.003085, batch_cost=0.6805, reader_cost=0.0002 | ETA 04:04:57 2020-10-31 13:12:16 [INFO] [TRAIN] epoch=158, iter=58500/80000, loss=0.1505, lr=0.003072, batch_cost=0.6868, reader_cost=0.0094 | ETA 04:06:05 2020-10-31 13:13:24 [INFO] [TRAIN] epoch=158, iter=58600/80000, loss=0.1402, lr=0.003059, batch_cost=0.6820, reader_cost=0.0003 | ETA 04:03:15 2020-10-31 13:14:32 [INFO] [TRAIN] epoch=158, iter=58700/80000, loss=0.1419, lr=0.003046, batch_cost=0.6778, reader_cost=0.0002 | ETA 04:00:36 2020-10-31 13:15:41 [INFO] [TRAIN] epoch=159, iter=58800/80000, loss=0.1372, lr=0.003033, batch_cost=0.6892, reader_cost=0.0102 | ETA 04:03:31 2020-10-31 13:16:49 [INFO] [TRAIN] epoch=159, iter=58900/80000, loss=0.1510, lr=0.003021, batch_cost=0.6821, reader_cost=0.0005 | ETA 03:59:53 2020-10-31 13:17:58 [INFO] [TRAIN] epoch=159, iter=59000/80000, loss=0.1409, lr=0.003008, batch_cost=0.6839, reader_cost=0.0004 | ETA 03:59:22 2020-10-31 13:19:06 [INFO] [TRAIN] epoch=159, iter=59100/80000, loss=0.1523, lr=0.002995, batch_cost=0.6816, reader_cost=0.0004 | ETA 03:57:24 2020-10-31 13:20:15 [INFO] [TRAIN] epoch=160, iter=59200/80000, loss=0.1377, lr=0.002982, batch_cost=0.6888, reader_cost=0.0096 | ETA 03:58:46 2020-10-31 13:21:23 [INFO] [TRAIN] epoch=160, iter=59300/80000, loss=0.1464, lr=0.002969, batch_cost=0.6840, reader_cost=0.0007 | ETA 03:55:58 2020-10-31 13:22:31 [INFO] [TRAIN] epoch=160, iter=59400/80000, loss=0.1405, lr=0.002956, batch_cost=0.6832, reader_cost=0.0006 | ETA 03:54:33 2020-10-31 13:23:40 [INFO] [TRAIN] epoch=160, iter=59500/80000, loss=0.1312, lr=0.002943, batch_cost=0.6820, reader_cost=0.0010 | ETA 03:53:00 2020-10-31 13:24:48 [INFO] [TRAIN] epoch=161, iter=59600/80000, loss=0.1439, lr=0.002931, batch_cost=0.6866, reader_cost=0.0094 | ETA 03:53:27 2020-10-31 13:25:56 [INFO] [TRAIN] epoch=161, iter=59700/80000, loss=0.1274, lr=0.002918, batch_cost=0.6747, reader_cost=0.0004 | ETA 03:48:16 2020-10-31 13:27:03 [INFO] [TRAIN] epoch=161, iter=59800/80000, loss=0.1420, lr=0.002905, batch_cost=0.6720, reader_cost=0.0002 | ETA 03:46:14 2020-10-31 13:28:12 [INFO] [TRAIN] epoch=162, iter=59900/80000, loss=0.1421, lr=0.002892, batch_cost=0.6891, reader_cost=0.0093 | ETA 03:50:51 2020-10-31 13:29:20 [INFO] [TRAIN] epoch=162, iter=60000/80000, loss=0.1458, lr=0.002879, batch_cost=0.6815, reader_cost=0.0002 | ETA 03:47:10 2020-10-31 13:30:28 [INFO] [TRAIN] epoch=162, iter=60100/80000, loss=0.1452, lr=0.002866, batch_cost=0.6805, reader_cost=0.0004 | ETA 03:45:41 2020-10-31 13:31:36 [INFO] [TRAIN] epoch=162, iter=60200/80000, loss=0.1475, lr=0.002853, batch_cost=0.6833, reader_cost=0.0005 | ETA 03:45:30 2020-10-31 13:32:45 [INFO] [TRAIN] epoch=163, iter=60300/80000, loss=0.1451, lr=0.002840, batch_cost=0.6884, reader_cost=0.0091 | ETA 03:46:00 2020-10-31 13:33:54 [INFO] [TRAIN] epoch=163, iter=60400/80000, loss=0.1346, lr=0.002827, batch_cost=0.6869, reader_cost=0.0005 | ETA 03:44:23 2020-10-31 13:35:02 [INFO] [TRAIN] epoch=163, iter=60500/80000, loss=0.1446, lr=0.002814, batch_cost=0.6793, reader_cost=0.0003 | ETA 03:40:47 2020-10-31 13:36:10 [INFO] [TRAIN] epoch=163, iter=60600/80000, loss=0.1407, lr=0.002801, batch_cost=0.6788, reader_cost=0.0006 | ETA 03:39:27 2020-10-31 13:37:19 [INFO] [TRAIN] epoch=164, iter=60700/80000, loss=0.1295, lr=0.002788, batch_cost=0.6920, reader_cost=0.0107 | ETA 03:42:34 2020-10-31 13:38:27 [INFO] [TRAIN] epoch=164, iter=60800/80000, loss=0.1502, lr=0.002776, batch_cost=0.6812, reader_cost=0.0002 | ETA 03:37:59 2020-10-31 13:39:35 [INFO] [TRAIN] epoch=164, iter=60900/80000, loss=0.1261, lr=0.002763, batch_cost=0.6852, reader_cost=0.0001 | ETA 03:38:06 2020-10-31 13:40:44 [INFO] [TRAIN] epoch=164, iter=61000/80000, loss=0.1508, lr=0.002750, batch_cost=0.6868, reader_cost=0.0007 | ETA 03:37:28 2020-10-31 13:41:53 [INFO] [TRAIN] epoch=165, iter=61100/80000, loss=0.1433, lr=0.002737, batch_cost=0.6866, reader_cost=0.0102 | ETA 03:36:17 2020-10-31 13:43:01 [INFO] [TRAIN] epoch=165, iter=61200/80000, loss=0.1405, lr=0.002724, batch_cost=0.6817, reader_cost=0.0002 | ETA 03:33:35 2020-10-31 13:44:09 [INFO] [TRAIN] epoch=165, iter=61300/80000, loss=0.1473, lr=0.002711, batch_cost=0.6786, reader_cost=0.0006 | ETA 03:31:30 2020-10-31 13:45:17 [INFO] [TRAIN] epoch=166, iter=61400/80000, loss=0.1403, lr=0.002698, batch_cost=0.6801, reader_cost=0.0112 | ETA 03:30:50 2020-10-31 13:46:24 [INFO] [TRAIN] epoch=166, iter=61500/80000, loss=0.1355, lr=0.002685, batch_cost=0.6676, reader_cost=0.0003 | ETA 03:25:51 2020-10-31 13:47:31 [INFO] [TRAIN] epoch=166, iter=61600/80000, loss=0.1623, lr=0.002672, batch_cost=0.6693, reader_cost=0.0002 | ETA 03:25:15 2020-10-31 13:48:38 [INFO] [TRAIN] epoch=166, iter=61700/80000, loss=0.1590, lr=0.002659, batch_cost=0.6718, reader_cost=0.0001 | ETA 03:24:54 2020-10-31 13:49:46 [INFO] [TRAIN] epoch=167, iter=61800/80000, loss=0.1228, lr=0.002646, batch_cost=0.6868, reader_cost=0.0096 | ETA 03:28:19 2020-10-31 13:50:54 [INFO] [TRAIN] epoch=167, iter=61900/80000, loss=0.1418, lr=0.002633, batch_cost=0.6742, reader_cost=0.0003 | ETA 03:23:22 2020-10-31 13:52:02 [INFO] [TRAIN] epoch=167, iter=62000/80000, loss=0.1660, lr=0.002619, batch_cost=0.6767, reader_cost=0.0003 | ETA 03:23:01 2020-10-31 13:53:09 [INFO] [TRAIN] epoch=167, iter=62100/80000, loss=0.1363, lr=0.002606, batch_cost=0.6786, reader_cost=0.0002 | ETA 03:22:26 2020-10-31 13:54:19 [INFO] [TRAIN] epoch=168, iter=62200/80000, loss=0.1468, lr=0.002593, batch_cost=0.6923, reader_cost=0.0101 | ETA 03:25:23 2020-10-31 13:55:27 [INFO] [TRAIN] epoch=168, iter=62300/80000, loss=0.1396, lr=0.002580, batch_cost=0.6811, reader_cost=0.0003 | ETA 03:20:54 2020-10-31 13:56:34 [INFO] [TRAIN] epoch=168, iter=62400/80000, loss=0.1510, lr=0.002567, batch_cost=0.6763, reader_cost=0.0007 | ETA 03:18:22 2020-10-31 13:57:43 [INFO] [TRAIN] epoch=169, iter=62500/80000, loss=0.1588, lr=0.002554, batch_cost=0.6818, reader_cost=0.0086 | ETA 03:18:51 2020-10-31 13:58:51 [INFO] [TRAIN] epoch=169, iter=62600/80000, loss=0.1399, lr=0.002541, batch_cost=0.6815, reader_cost=0.0004 | ETA 03:17:38 2020-10-31 13:59:59 [INFO] [TRAIN] epoch=169, iter=62700/80000, loss=0.1378, lr=0.002528, batch_cost=0.6829, reader_cost=0.0004 | ETA 03:16:54 2020-10-31 14:01:07 [INFO] [TRAIN] epoch=169, iter=62800/80000, loss=0.1383, lr=0.002515, batch_cost=0.6839, reader_cost=0.0009 | ETA 03:16:03 2020-10-31 14:02:16 [INFO] [TRAIN] epoch=170, iter=62900/80000, loss=0.1305, lr=0.002502, batch_cost=0.6899, reader_cost=0.0085 | ETA 03:16:37 2020-10-31 14:03:25 [INFO] [TRAIN] epoch=170, iter=63000/80000, loss=0.1429, lr=0.002489, batch_cost=0.6818, reader_cost=0.0001 | ETA 03:13:11 2020-10-31 14:04:33 [INFO] [TRAIN] epoch=170, iter=63100/80000, loss=0.1266, lr=0.002476, batch_cost=0.6815, reader_cost=0.0002 | ETA 03:11:57 2020-10-31 14:05:41 [INFO] [TRAIN] epoch=170, iter=63200/80000, loss=0.1530, lr=0.002462, batch_cost=0.6801, reader_cost=0.0002 | ETA 03:10:25 2020-10-31 14:06:50 [INFO] [TRAIN] epoch=171, iter=63300/80000, loss=0.1467, lr=0.002449, batch_cost=0.6944, reader_cost=0.0105 | ETA 03:13:16 2020-10-31 14:07:59 [INFO] [TRAIN] epoch=171, iter=63400/80000, loss=0.1289, lr=0.002436, batch_cost=0.6863, reader_cost=0.0006 | ETA 03:09:52 2020-10-31 14:09:08 [INFO] [TRAIN] epoch=171, iter=63500/80000, loss=0.1309, lr=0.002423, batch_cost=0.6877, reader_cost=0.0004 | ETA 03:09:06 2020-10-31 14:10:16 [INFO] [TRAIN] epoch=171, iter=63600/80000, loss=0.1286, lr=0.002410, batch_cost=0.6833, reader_cost=0.0002 | ETA 03:06:46 2020-10-31 14:11:25 [INFO] [TRAIN] epoch=172, iter=63700/80000, loss=0.1238, lr=0.002397, batch_cost=0.6894, reader_cost=0.0093 | ETA 03:07:17 2020-10-31 14:12:34 [INFO] [TRAIN] epoch=172, iter=63800/80000, loss=0.1254, lr=0.002383, batch_cost=0.6885, reader_cost=0.0005 | ETA 03:05:53 2020-10-31 14:13:42 [INFO] [TRAIN] epoch=172, iter=63900/80000, loss=0.1430, lr=0.002370, batch_cost=0.6852, reader_cost=0.0003 | ETA 03:03:52 2020-10-31 14:14:51 [INFO] [TRAIN] epoch=173, iter=64000/80000, loss=0.1340, lr=0.002357, batch_cost=0.6926, reader_cost=0.0092 | ETA 03:04:41 2020-10-31 14:14:59 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-10-31 14:20:41 [INFO] [EVAL] #Images=500 mIoU=0.7998 Acc=0.9642 Kappa=0.9536 2020-10-31 14:20:41 [INFO] [EVAL] Category IoU: [0.983 0.8615 0.9324 0.5757 0.6418 0.6672 0.7277 0.8059 0.9289 0.6549 0.9501 0.8385 0.6508 0.9564 0.8199 0.9009 0.8361 0.6689 0.7958] 2020-10-31 14:20:41 [INFO] [EVAL] Category Acc: [0.9925 0.9236 0.9586 0.8248 0.8237 0.8323 0.8663 0.9086 0.9577 0.8726 0.9677 0.9017 0.8001 0.9752 0.9206 0.95 0.9606 0.8431 0.8791] 2020-10-31 14:20:45 [INFO] [EVAL] The model with the best validation mIoU (0.7998) was saved at iter 64000. 2020-10-31 14:21:52 [INFO] [TRAIN] epoch=173, iter=64100/80000, loss=0.1322, lr=0.002344, batch_cost=0.6695, reader_cost=0.0007 | ETA 02:57:24 2020-10-31 14:23:00 [INFO] [TRAIN] epoch=173, iter=64200/80000, loss=0.1516, lr=0.002331, batch_cost=0.6839, reader_cost=0.0003 | ETA 03:00:06 2020-10-31 14:24:08 [INFO] [TRAIN] epoch=173, iter=64300/80000, loss=0.1363, lr=0.002317, batch_cost=0.6771, reader_cost=0.0006 | ETA 02:57:09 2020-10-31 14:25:16 [INFO] [TRAIN] epoch=174, iter=64400/80000, loss=0.1277, lr=0.002304, batch_cost=0.6871, reader_cost=0.0097 | ETA 02:58:39 2020-10-31 14:26:24 [INFO] [TRAIN] epoch=174, iter=64500/80000, loss=0.1290, lr=0.002291, batch_cost=0.6766, reader_cost=0.0003 | ETA 02:54:46 2020-10-31 14:27:32 [INFO] [TRAIN] epoch=174, iter=64600/80000, loss=0.1410, lr=0.002278, batch_cost=0.6791, reader_cost=0.0002 | ETA 02:54:18 2020-10-31 14:28:40 [INFO] [TRAIN] epoch=174, iter=64700/80000, loss=0.1232, lr=0.002264, batch_cost=0.6804, reader_cost=0.0002 | ETA 02:53:30 2020-10-31 14:29:48 [INFO] [TRAIN] epoch=175, iter=64800/80000, loss=0.1291, lr=0.002251, batch_cost=0.6841, reader_cost=0.0088 | ETA 02:53:18 2020-10-31 14:30:56 [INFO] [TRAIN] epoch=175, iter=64900/80000, loss=0.1320, lr=0.002238, batch_cost=0.6776, reader_cost=0.0003 | ETA 02:50:31 2020-10-31 14:32:04 [INFO] [TRAIN] epoch=175, iter=65000/80000, loss=0.1195, lr=0.002225, batch_cost=0.6824, reader_cost=0.0001 | ETA 02:50:35 2020-10-31 14:33:13 [INFO] [TRAIN] epoch=175, iter=65100/80000, loss=0.1391, lr=0.002211, batch_cost=0.6824, reader_cost=0.0001 | ETA 02:49:27 2020-10-31 14:34:22 [INFO] [TRAIN] epoch=176, iter=65200/80000, loss=0.1431, lr=0.002198, batch_cost=0.6891, reader_cost=0.0101 | ETA 02:49:59 2020-10-31 14:35:29 [INFO] [TRAIN] epoch=176, iter=65300/80000, loss=0.1236, lr=0.002185, batch_cost=0.6789, reader_cost=0.0003 | ETA 02:46:19 2020-10-31 14:36:38 [INFO] [TRAIN] epoch=176, iter=65400/80000, loss=0.1228, lr=0.002171, batch_cost=0.6810, reader_cost=0.0004 | ETA 02:45:42 2020-10-31 14:37:46 [INFO] [TRAIN] epoch=177, iter=65500/80000, loss=0.1308, lr=0.002158, batch_cost=0.6897, reader_cost=0.0091 | ETA 02:46:41 2020-10-31 14:38:54 [INFO] [TRAIN] epoch=177, iter=65600/80000, loss=0.1481, lr=0.002145, batch_cost=0.6794, reader_cost=0.0001 | ETA 02:43:02 2020-10-31 14:40:02 [INFO] [TRAIN] epoch=177, iter=65700/80000, loss=0.1438, lr=0.002131, batch_cost=0.6799, reader_cost=0.0008 | ETA 02:42:02 2020-10-31 14:41:11 [INFO] [TRAIN] epoch=177, iter=65800/80000, loss=0.1320, lr=0.002118, batch_cost=0.6836, reader_cost=0.0004 | ETA 02:41:46 2020-10-31 14:42:20 [INFO] [TRAIN] epoch=178, iter=65900/80000, loss=0.1237, lr=0.002105, batch_cost=0.6878, reader_cost=0.0078 | ETA 02:41:37 2020-10-31 14:43:27 [INFO] [TRAIN] epoch=178, iter=66000/80000, loss=0.1448, lr=0.002091, batch_cost=0.6791, reader_cost=0.0002 | ETA 02:38:27 2020-10-31 14:44:36 [INFO] [TRAIN] epoch=178, iter=66100/80000, loss=0.1357, lr=0.002078, batch_cost=0.6873, reader_cost=0.0007 | ETA 02:39:13 2020-10-31 14:45:44 [INFO] [TRAIN] epoch=178, iter=66200/80000, loss=0.1270, lr=0.002064, batch_cost=0.6766, reader_cost=0.0002 | ETA 02:35:37 2020-10-31 14:46:53 [INFO] [TRAIN] epoch=179, iter=66300/80000, loss=0.1441, lr=0.002051, batch_cost=0.6869, reader_cost=0.0099 | ETA 02:36:50 2020-10-31 14:47:59 [INFO] [TRAIN] epoch=179, iter=66400/80000, loss=0.1317, lr=0.002038, batch_cost=0.6672, reader_cost=0.0002 | ETA 02:31:13 2020-10-31 14:49:06 [INFO] [TRAIN] epoch=179, iter=66500/80000, loss=0.1266, lr=0.002024, batch_cost=0.6700, reader_cost=0.0002 | ETA 02:30:45 2020-10-31 14:50:15 [INFO] [TRAIN] epoch=180, iter=66600/80000, loss=0.1273, lr=0.002011, batch_cost=0.6829, reader_cost=0.0100 | ETA 02:32:31 2020-10-31 14:51:23 [INFO] [TRAIN] epoch=180, iter=66700/80000, loss=0.1355, lr=0.001997, batch_cost=0.6817, reader_cost=0.0007 | ETA 02:31:07 2020-10-31 14:52:31 [INFO] [TRAIN] epoch=180, iter=66800/80000, loss=0.1453, lr=0.001984, batch_cost=0.6818, reader_cost=0.0004 | ETA 02:30:00 2020-10-31 14:53:39 [INFO] [TRAIN] epoch=180, iter=66900/80000, loss=0.1301, lr=0.001970, batch_cost=0.6775, reader_cost=0.0001 | ETA 02:27:55 2020-10-31 14:54:48 [INFO] [TRAIN] epoch=181, iter=67000/80000, loss=0.1269, lr=0.001957, batch_cost=0.6923, reader_cost=0.0093 | ETA 02:30:00 2020-10-31 14:55:56 [INFO] [TRAIN] epoch=181, iter=67100/80000, loss=0.1245, lr=0.001944, batch_cost=0.6764, reader_cost=0.0002 | ETA 02:25:25 2020-10-31 14:57:04 [INFO] [TRAIN] epoch=181, iter=67200/80000, loss=0.1384, lr=0.001930, batch_cost=0.6809, reader_cost=0.0003 | ETA 02:25:15 2020-10-31 14:58:12 [INFO] [TRAIN] epoch=181, iter=67300/80000, loss=0.1229, lr=0.001917, batch_cost=0.6802, reader_cost=0.0005 | ETA 02:23:57 2020-10-31 14:59:21 [INFO] [TRAIN] epoch=182, iter=67400/80000, loss=0.1483, lr=0.001903, batch_cost=0.6926, reader_cost=0.0087 | ETA 02:25:27 2020-10-31 15:00:29 [INFO] [TRAIN] epoch=182, iter=67500/80000, loss=0.1273, lr=0.001889, batch_cost=0.6855, reader_cost=0.0006 | ETA 02:22:48 2020-10-31 15:01:38 [INFO] [TRAIN] epoch=182, iter=67600/80000, loss=0.1281, lr=0.001876, batch_cost=0.6810, reader_cost=0.0004 | ETA 02:20:43 2020-10-31 15:02:46 [INFO] [TRAIN] epoch=182, iter=67700/80000, loss=0.1438, lr=0.001862, batch_cost=0.6829, reader_cost=0.0004 | ETA 02:19:59 2020-10-31 15:03:56 [INFO] [TRAIN] epoch=183, iter=67800/80000, loss=0.1222, lr=0.001849, batch_cost=0.6985, reader_cost=0.0108 | ETA 02:22:01 2020-10-31 15:05:04 [INFO] [TRAIN] epoch=183, iter=67900/80000, loss=0.1368, lr=0.001835, batch_cost=0.6835, reader_cost=0.0005 | ETA 02:17:50 2020-10-31 15:06:13 [INFO] [TRAIN] epoch=183, iter=68000/80000, loss=0.1399, lr=0.001822, batch_cost=0.6847, reader_cost=0.0005 | ETA 02:16:56 2020-10-31 15:07:22 [INFO] [TRAIN] epoch=184, iter=68100/80000, loss=0.1296, lr=0.001808, batch_cost=0.6973, reader_cost=0.0110 | ETA 02:18:17 2020-10-31 15:08:31 [INFO] [TRAIN] epoch=184, iter=68200/80000, loss=0.1230, lr=0.001794, batch_cost=0.6843, reader_cost=0.0007 | ETA 02:14:34 2020-10-31 15:09:39 [INFO] [TRAIN] epoch=184, iter=68300/80000, loss=0.1303, lr=0.001781, batch_cost=0.6856, reader_cost=0.0007 | ETA 02:13:41 2020-10-31 15:10:47 [INFO] [TRAIN] epoch=184, iter=68400/80000, loss=0.1296, lr=0.001767, batch_cost=0.6808, reader_cost=0.0005 | ETA 02:11:37 2020-10-31 15:11:57 [INFO] [TRAIN] epoch=185, iter=68500/80000, loss=0.1339, lr=0.001754, batch_cost=0.6920, reader_cost=0.0098 | ETA 02:12:37 2020-10-31 15:13:05 [INFO] [TRAIN] epoch=185, iter=68600/80000, loss=0.1336, lr=0.001740, batch_cost=0.6805, reader_cost=0.0004 | ETA 02:09:17 2020-10-31 15:14:13 [INFO] [TRAIN] epoch=185, iter=68700/80000, loss=0.1378, lr=0.001726, batch_cost=0.6825, reader_cost=0.0003 | ETA 02:08:31 2020-10-31 15:15:21 [INFO] [TRAIN] epoch=185, iter=68800/80000, loss=0.1394, lr=0.001713, batch_cost=0.6778, reader_cost=0.0003 | ETA 02:06:31 2020-10-31 15:16:30 [INFO] [TRAIN] epoch=186, iter=68900/80000, loss=0.1223, lr=0.001699, batch_cost=0.6893, reader_cost=0.0099 | ETA 02:07:30 2020-10-31 15:17:37 [INFO] [TRAIN] epoch=186, iter=69000/80000, loss=0.1290, lr=0.001685, batch_cost=0.6789, reader_cost=0.0003 | ETA 02:04:27 2020-10-31 15:18:46 [INFO] [TRAIN] epoch=186, iter=69100/80000, loss=0.1312, lr=0.001672, batch_cost=0.6833, reader_cost=0.0002 | ETA 02:04:07 2020-10-31 15:19:55 [INFO] [TRAIN] epoch=187, iter=69200/80000, loss=0.1235, lr=0.001658, batch_cost=0.6896, reader_cost=0.0091 | ETA 02:04:07 2020-10-31 15:21:03 [INFO] [TRAIN] epoch=187, iter=69300/80000, loss=0.1303, lr=0.001644, batch_cost=0.6842, reader_cost=0.0002 | ETA 02:02:01 2020-10-31 15:22:12 [INFO] [TRAIN] epoch=187, iter=69400/80000, loss=0.1433, lr=0.001630, batch_cost=0.6858, reader_cost=0.0007 | ETA 02:01:09 2020-10-31 15:23:20 [INFO] [TRAIN] epoch=187, iter=69500/80000, loss=0.1335, lr=0.001617, batch_cost=0.6784, reader_cost=0.0002 | ETA 01:58:43 2020-10-31 15:24:29 [INFO] [TRAIN] epoch=188, iter=69600/80000, loss=0.1322, lr=0.001603, batch_cost=0.6953, reader_cost=0.0088 | ETA 02:00:30 2020-10-31 15:25:37 [INFO] [TRAIN] epoch=188, iter=69700/80000, loss=0.1289, lr=0.001589, batch_cost=0.6758, reader_cost=0.0003 | ETA 01:56:00 2020-10-31 15:26:45 [INFO] [TRAIN] epoch=188, iter=69800/80000, loss=0.1314, lr=0.001575, batch_cost=0.6800, reader_cost=0.0002 | ETA 01:55:35 2020-10-31 15:27:53 [INFO] [TRAIN] epoch=188, iter=69900/80000, loss=0.1177, lr=0.001561, batch_cost=0.6802, reader_cost=0.0002 | ETA 01:54:30 2020-10-31 15:29:02 [INFO] [TRAIN] epoch=189, iter=70000/80000, loss=0.1281, lr=0.001548, batch_cost=0.6937, reader_cost=0.0089 | ETA 01:55:37 2020-10-31 15:30:10 [INFO] [TRAIN] epoch=189, iter=70100/80000, loss=0.1263, lr=0.001534, batch_cost=0.6837, reader_cost=0.0006 | ETA 01:52:48 2020-10-31 15:31:19 [INFO] [TRAIN] epoch=189, iter=70200/80000, loss=0.1286, lr=0.001520, batch_cost=0.6858, reader_cost=0.0005 | ETA 01:52:00 2020-10-31 15:32:27 [INFO] [TRAIN] epoch=189, iter=70300/80000, loss=0.1378, lr=0.001506, batch_cost=0.6794, reader_cost=0.0008 | ETA 01:49:50 2020-10-31 15:33:36 [INFO] [TRAIN] epoch=190, iter=70400/80000, loss=0.1255, lr=0.001492, batch_cost=0.6906, reader_cost=0.0098 | ETA 01:50:30 2020-10-31 15:34:45 [INFO] [TRAIN] epoch=190, iter=70500/80000, loss=0.1312, lr=0.001478, batch_cost=0.6875, reader_cost=0.0008 | ETA 01:48:51 2020-10-31 15:35:53 [INFO] [TRAIN] epoch=190, iter=70600/80000, loss=0.1243, lr=0.001464, batch_cost=0.6851, reader_cost=0.0005 | ETA 01:47:20 2020-10-31 15:37:02 [INFO] [TRAIN] epoch=191, iter=70700/80000, loss=0.1194, lr=0.001450, batch_cost=0.6912, reader_cost=0.0098 | ETA 01:47:08 2020-10-31 15:38:11 [INFO] [TRAIN] epoch=191, iter=70800/80000, loss=0.1300, lr=0.001436, batch_cost=0.6905, reader_cost=0.0006 | ETA 01:45:52 2020-10-31 15:39:20 [INFO] [TRAIN] epoch=191, iter=70900/80000, loss=0.1247, lr=0.001422, batch_cost=0.6863, reader_cost=0.0007 | ETA 01:44:05 2020-10-31 15:40:29 [INFO] [TRAIN] epoch=191, iter=71000/80000, loss=0.1361, lr=0.001408, batch_cost=0.6859, reader_cost=0.0005 | ETA 01:42:52 2020-10-31 15:41:38 [INFO] [TRAIN] epoch=192, iter=71100/80000, loss=0.1352, lr=0.001394, batch_cost=0.6930, reader_cost=0.0098 | ETA 01:42:47 2020-10-31 15:42:46 [INFO] [TRAIN] epoch=192, iter=71200/80000, loss=0.1332, lr=0.001380, batch_cost=0.6822, reader_cost=0.0002 | ETA 01:40:03 2020-10-31 15:43:55 [INFO] [TRAIN] epoch=192, iter=71300/80000, loss=0.1238, lr=0.001366, batch_cost=0.6842, reader_cost=0.0002 | ETA 01:39:12 2020-10-31 15:45:03 [INFO] [TRAIN] epoch=192, iter=71400/80000, loss=0.1240, lr=0.001352, batch_cost=0.6829, reader_cost=0.0002 | ETA 01:37:52 2020-10-31 15:46:12 [INFO] [TRAIN] epoch=193, iter=71500/80000, loss=0.1211, lr=0.001338, batch_cost=0.6907, reader_cost=0.0092 | ETA 01:37:50 2020-10-31 15:47:20 [INFO] [TRAIN] epoch=193, iter=71600/80000, loss=0.1726, lr=0.001324, batch_cost=0.6776, reader_cost=0.0006 | ETA 01:34:51 2020-10-31 15:48:28 [INFO] [TRAIN] epoch=193, iter=71700/80000, loss=0.1285, lr=0.001310, batch_cost=0.6829, reader_cost=0.0008 | ETA 01:34:28 2020-10-31 15:49:37 [INFO] [TRAIN] epoch=194, iter=71800/80000, loss=0.1323, lr=0.001296, batch_cost=0.6903, reader_cost=0.0102 | ETA 01:34:20 2020-10-31 15:50:44 [INFO] [TRAIN] epoch=194, iter=71900/80000, loss=0.1337, lr=0.001282, batch_cost=0.6664, reader_cost=0.0002 | ETA 01:29:57 2020-10-31 15:51:50 [INFO] [TRAIN] epoch=194, iter=72000/80000, loss=0.1229, lr=0.001268, batch_cost=0.6618, reader_cost=0.0001 | ETA 01:28:14 2020-10-31 15:51:57 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-10-31 15:57:34 [INFO] [EVAL] #Images=500 mIoU=0.8068 Acc=0.9652 Kappa=0.9549 2020-10-31 15:57:34 [INFO] [EVAL] Category IoU: [0.9838 0.8676 0.9341 0.6304 0.6529 0.6752 0.7367 0.8088 0.9289 0.6641 0.9495 0.8392 0.6472 0.9568 0.8466 0.9108 0.8354 0.6639 0.7978] 2020-10-31 15:57:34 [INFO] [EVAL] Category Acc: [0.9941 0.9208 0.9606 0.8651 0.8211 0.835 0.8519 0.9064 0.9587 0.8641 0.966 0.9001 0.8176 0.9757 0.9385 0.9541 0.9533 0.8279 0.8714] 2020-10-31 15:57:38 [INFO] [EVAL] The model with the best validation mIoU (0.8068) was saved at iter 72000. 2020-10-31 15:58:46 [INFO] [TRAIN] epoch=194, iter=72100/80000, loss=0.1245, lr=0.001254, batch_cost=0.6753, reader_cost=0.0003 | ETA 01:28:55 2020-10-31 15:59:55 [INFO] [TRAIN] epoch=195, iter=72200/80000, loss=0.1236, lr=0.001239, batch_cost=0.6893, reader_cost=0.0099 | ETA 01:29:36 2020-10-31 16:01:03 [INFO] [TRAIN] epoch=195, iter=72300/80000, loss=0.1283, lr=0.001225, batch_cost=0.6820, reader_cost=0.0003 | ETA 01:27:31 2020-10-31 16:02:11 [INFO] [TRAIN] epoch=195, iter=72400/80000, loss=0.1182, lr=0.001211, batch_cost=0.6814, reader_cost=0.0006 | ETA 01:26:18 2020-10-31 16:03:19 [INFO] [TRAIN] epoch=195, iter=72500/80000, loss=0.1252, lr=0.001197, batch_cost=0.6779, reader_cost=0.0002 | ETA 01:24:44 2020-10-31 16:04:28 [INFO] [TRAIN] epoch=196, iter=72600/80000, loss=0.1181, lr=0.001183, batch_cost=0.6916, reader_cost=0.0102 | ETA 01:25:17 2020-10-31 16:05:36 [INFO] [TRAIN] epoch=196, iter=72700/80000, loss=0.1200, lr=0.001168, batch_cost=0.6838, reader_cost=0.0003 | ETA 01:23:11 2020-10-31 16:06:44 [INFO] [TRAIN] epoch=196, iter=72800/80000, loss=0.1296, lr=0.001154, batch_cost=0.6752, reader_cost=0.0005 | ETA 01:21:01 2020-10-31 16:07:51 [INFO] [TRAIN] epoch=196, iter=72900/80000, loss=0.1292, lr=0.001140, batch_cost=0.6769, reader_cost=0.0004 | ETA 01:20:05 2020-10-31 16:09:00 [INFO] [TRAIN] epoch=197, iter=73000/80000, loss=0.1445, lr=0.001125, batch_cost=0.6901, reader_cost=0.0089 | ETA 01:20:30 2020-10-31 16:10:09 [INFO] [TRAIN] epoch=197, iter=73100/80000, loss=0.1400, lr=0.001111, batch_cost=0.6818, reader_cost=0.0004 | ETA 01:18:24 2020-10-31 16:11:17 [INFO] [TRAIN] epoch=197, iter=73200/80000, loss=0.1269, lr=0.001097, batch_cost=0.6831, reader_cost=0.0002 | ETA 01:17:25 2020-10-31 16:12:26 [INFO] [TRAIN] epoch=198, iter=73300/80000, loss=0.1135, lr=0.001082, batch_cost=0.6953, reader_cost=0.0080 | ETA 01:17:38 2020-10-31 16:13:35 [INFO] [TRAIN] epoch=198, iter=73400/80000, loss=0.1242, lr=0.001068, batch_cost=0.6854, reader_cost=0.0008 | ETA 01:15:23 2020-10-31 16:14:44 [INFO] [TRAIN] epoch=198, iter=73500/80000, loss=0.1337, lr=0.001053, batch_cost=0.6871, reader_cost=0.0006 | ETA 01:14:26 2020-10-31 16:15:52 [INFO] [TRAIN] epoch=198, iter=73600/80000, loss=0.1348, lr=0.001039, batch_cost=0.6787, reader_cost=0.0003 | ETA 01:12:23 2020-10-31 16:17:01 [INFO] [TRAIN] epoch=199, iter=73700/80000, loss=0.1107, lr=0.001025, batch_cost=0.6914, reader_cost=0.0093 | ETA 01:12:35 2020-10-31 16:18:09 [INFO] [TRAIN] epoch=199, iter=73800/80000, loss=0.1296, lr=0.001010, batch_cost=0.6839, reader_cost=0.0006 | ETA 01:10:39 2020-10-31 16:19:18 [INFO] [TRAIN] epoch=199, iter=73900/80000, loss=0.1230, lr=0.000995, batch_cost=0.6859, reader_cost=0.0006 | ETA 01:09:43 2020-10-31 16:20:27 [INFO] [TRAIN] epoch=199, iter=74000/80000, loss=0.1211, lr=0.000981, batch_cost=0.6911, reader_cost=0.0006 | ETA 01:09:06 2020-10-31 16:21:36 [INFO] [TRAIN] epoch=200, iter=74100/80000, loss=0.1223, lr=0.000966, batch_cost=0.6910, reader_cost=0.0090 | ETA 01:07:56 2020-10-31 16:22:44 [INFO] [TRAIN] epoch=200, iter=74200/80000, loss=0.1316, lr=0.000952, batch_cost=0.6854, reader_cost=0.0005 | ETA 01:06:15 2020-10-31 16:23:53 [INFO] [TRAIN] epoch=200, iter=74300/80000, loss=0.1185, lr=0.000937, batch_cost=0.6813, reader_cost=0.0003 | ETA 01:04:43 2020-10-31 16:25:00 [INFO] [TRAIN] epoch=200, iter=74400/80000, loss=0.1368, lr=0.000922, batch_cost=0.6789, reader_cost=0.0008 | ETA 01:03:21 2020-10-31 16:26:10 [INFO] [TRAIN] epoch=201, iter=74500/80000, loss=0.1262, lr=0.000908, batch_cost=0.6918, reader_cost=0.0089 | ETA 01:03:25 2020-10-31 16:27:18 [INFO] [TRAIN] epoch=201, iter=74600/80000, loss=0.1272, lr=0.000893, batch_cost=0.6788, reader_cost=0.0002 | ETA 01:01:05 2020-10-31 16:28:26 [INFO] [TRAIN] epoch=201, iter=74700/80000, loss=0.1336, lr=0.000878, batch_cost=0.6798, reader_cost=0.0002 | ETA 01:00:02 2020-10-31 16:29:35 [INFO] [TRAIN] epoch=202, iter=74800/80000, loss=0.1232, lr=0.000864, batch_cost=0.6899, reader_cost=0.0093 | ETA 00:59:47 2020-10-31 16:30:43 [INFO] [TRAIN] epoch=202, iter=74900/80000, loss=0.1227, lr=0.000849, batch_cost=0.6806, reader_cost=0.0002 | ETA 00:57:51 2020-10-31 16:31:50 [INFO] [TRAIN] epoch=202, iter=75000/80000, loss=0.1077, lr=0.000834, batch_cost=0.6774, reader_cost=0.0004 | ETA 00:56:27 2020-10-31 16:32:58 [INFO] [TRAIN] epoch=202, iter=75100/80000, loss=0.1347, lr=0.000819, batch_cost=0.6817, reader_cost=0.0003 | ETA 00:55:40 2020-10-31 16:34:08 [INFO] [TRAIN] epoch=203, iter=75200/80000, loss=0.1231, lr=0.000804, batch_cost=0.6924, reader_cost=0.0084 | ETA 00:55:23 2020-10-31 16:35:16 [INFO] [TRAIN] epoch=203, iter=75300/80000, loss=0.1162, lr=0.000789, batch_cost=0.6802, reader_cost=0.0003 | ETA 00:53:17 2020-10-31 16:36:24 [INFO] [TRAIN] epoch=203, iter=75400/80000, loss=0.1265, lr=0.000774, batch_cost=0.6856, reader_cost=0.0004 | ETA 00:52:33 2020-10-31 16:37:33 [INFO] [TRAIN] epoch=203, iter=75500/80000, loss=0.1252, lr=0.000759, batch_cost=0.6874, reader_cost=0.0007 | ETA 00:51:33 2020-10-31 16:38:42 [INFO] [TRAIN] epoch=204, iter=75600/80000, loss=0.1229, lr=0.000744, batch_cost=0.6904, reader_cost=0.0096 | ETA 00:50:37 2020-10-31 16:39:50 [INFO] [TRAIN] epoch=204, iter=75700/80000, loss=0.1188, lr=0.000729, batch_cost=0.6761, reader_cost=0.0003 | ETA 00:48:27 2020-10-31 16:40:58 [INFO] [TRAIN] epoch=204, iter=75800/80000, loss=0.1168, lr=0.000714, batch_cost=0.6845, reader_cost=0.0002 | ETA 00:47:54 2020-10-31 16:42:07 [INFO] [TRAIN] epoch=205, iter=75900/80000, loss=0.1225, lr=0.000699, batch_cost=0.6927, reader_cost=0.0090 | ETA 00:47:20 2020-10-31 16:43:15 [INFO] [TRAIN] epoch=205, iter=76000/80000, loss=0.1280, lr=0.000684, batch_cost=0.6769, reader_cost=0.0002 | ETA 00:45:07 2020-10-31 16:44:23 [INFO] [TRAIN] epoch=205, iter=76100/80000, loss=0.1315, lr=0.000669, batch_cost=0.6794, reader_cost=0.0001 | ETA 00:44:09 2020-10-31 16:45:31 [INFO] [TRAIN] epoch=205, iter=76200/80000, loss=0.1160, lr=0.000654, batch_cost=0.6826, reader_cost=0.0003 | ETA 00:43:13 2020-10-31 16:46:40 [INFO] [TRAIN] epoch=206, iter=76300/80000, loss=0.1246, lr=0.000638, batch_cost=0.6880, reader_cost=0.0092 | ETA 00:42:25 2020-10-31 16:47:48 [INFO] [TRAIN] epoch=206, iter=76400/80000, loss=0.1260, lr=0.000623, batch_cost=0.6805, reader_cost=0.0002 | ETA 00:40:49 2020-10-31 16:48:56 [INFO] [TRAIN] epoch=206, iter=76500/80000, loss=0.1227, lr=0.000608, batch_cost=0.6819, reader_cost=0.0006 | ETA 00:39:46 2020-10-31 16:50:05 [INFO] [TRAIN] epoch=206, iter=76600/80000, loss=0.1208, lr=0.000592, batch_cost=0.6834, reader_cost=0.0002 | ETA 00:38:43 2020-10-31 16:51:14 [INFO] [TRAIN] epoch=207, iter=76700/80000, loss=0.1237, lr=0.000577, batch_cost=0.6893, reader_cost=0.0084 | ETA 00:37:54 2020-10-31 16:52:22 [INFO] [TRAIN] epoch=207, iter=76800/80000, loss=0.1339, lr=0.000561, batch_cost=0.6862, reader_cost=0.0006 | ETA 00:36:35 2020-10-31 16:53:30 [INFO] [TRAIN] epoch=207, iter=76900/80000, loss=0.1223, lr=0.000546, batch_cost=0.6793, reader_cost=0.0002 | ETA 00:35:05 2020-10-31 16:54:38 [INFO] [TRAIN] epoch=207, iter=77000/80000, loss=0.1316, lr=0.000530, batch_cost=0.6826, reader_cost=0.0002 | ETA 00:34:07 2020-10-31 16:55:47 [INFO] [TRAIN] epoch=208, iter=77100/80000, loss=0.1277, lr=0.000515, batch_cost=0.6900, reader_cost=0.0093 | ETA 00:33:21 2020-10-31 16:56:57 [INFO] [TRAIN] epoch=208, iter=77200/80000, loss=0.1231, lr=0.000499, batch_cost=0.6914, reader_cost=0.0006 | ETA 00:32:15 2020-10-31 16:58:05 [INFO] [TRAIN] epoch=208, iter=77300/80000, loss=0.1191, lr=0.000483, batch_cost=0.6852, reader_cost=0.0004 | ETA 00:30:49 2020-10-31 16:59:14 [INFO] [TRAIN] epoch=209, iter=77400/80000, loss=0.1226, lr=0.000468, batch_cost=0.6899, reader_cost=0.0081 | ETA 00:29:53 2020-10-31 17:00:23 [INFO] [TRAIN] epoch=209, iter=77500/80000, loss=0.1320, lr=0.000452, batch_cost=0.6851, reader_cost=0.0003 | ETA 00:28:32 2020-10-31 17:01:31 [INFO] [TRAIN] epoch=209, iter=77600/80000, loss=0.1283, lr=0.000436, batch_cost=0.6789, reader_cost=0.0009 | ETA 00:27:09 2020-10-31 17:02:39 [INFO] [TRAIN] epoch=209, iter=77700/80000, loss=0.1172, lr=0.000420, batch_cost=0.6796, reader_cost=0.0004 | ETA 00:26:03 2020-10-31 17:03:48 [INFO] [TRAIN] epoch=210, iter=77800/80000, loss=0.1228, lr=0.000404, batch_cost=0.6959, reader_cost=0.0093 | ETA 00:25:31 2020-10-31 17:04:57 [INFO] [TRAIN] epoch=210, iter=77900/80000, loss=0.1206, lr=0.000388, batch_cost=0.6841, reader_cost=0.0006 | ETA 00:23:56 2020-10-31 17:06:04 [INFO] [TRAIN] epoch=210, iter=78000/80000, loss=0.1180, lr=0.000371, batch_cost=0.6788, reader_cost=0.0005 | ETA 00:22:37 2020-10-31 17:07:12 [INFO] [TRAIN] epoch=210, iter=78100/80000, loss=0.1142, lr=0.000355, batch_cost=0.6747, reader_cost=0.0004 | ETA 00:21:21 2020-10-31 17:08:21 [INFO] [TRAIN] epoch=211, iter=78200/80000, loss=0.1233, lr=0.000339, batch_cost=0.6906, reader_cost=0.0107 | ETA 00:20:43 2020-10-31 17:09:29 [INFO] [TRAIN] epoch=211, iter=78300/80000, loss=0.1297, lr=0.000322, batch_cost=0.6834, reader_cost=0.0008 | ETA 00:19:21 2020-10-31 17:10:38 [INFO] [TRAIN] epoch=211, iter=78400/80000, loss=0.1210, lr=0.000306, batch_cost=0.6826, reader_cost=0.0005 | ETA 00:18:12 2020-10-31 17:11:47 [INFO] [TRAIN] epoch=212, iter=78500/80000, loss=0.1246, lr=0.000289, batch_cost=0.6909, reader_cost=0.0099 | ETA 00:17:16 2020-10-31 17:12:55 [INFO] [TRAIN] epoch=212, iter=78600/80000, loss=0.1161, lr=0.000272, batch_cost=0.6839, reader_cost=0.0013 | ETA 00:15:57 2020-10-31 17:14:04 [INFO] [TRAIN] epoch=212, iter=78700/80000, loss=0.1267, lr=0.000255, batch_cost=0.6923, reader_cost=0.0012 | ETA 00:14:59 2020-10-31 17:15:13 [INFO] [TRAIN] epoch=212, iter=78800/80000, loss=0.1183, lr=0.000238, batch_cost=0.6830, reader_cost=0.0010 | ETA 00:13:39 2020-10-31 17:16:22 [INFO] [TRAIN] epoch=213, iter=78900/80000, loss=0.1236, lr=0.000221, batch_cost=0.6903, reader_cost=0.0092 | ETA 00:12:39 2020-10-31 17:17:30 [INFO] [TRAIN] epoch=213, iter=79000/80000, loss=0.1169, lr=0.000204, batch_cost=0.6821, reader_cost=0.0003 | ETA 00:11:22 2020-10-31 17:18:38 [INFO] [TRAIN] epoch=213, iter=79100/80000, loss=0.1237, lr=0.000186, batch_cost=0.6848, reader_cost=0.0002 | ETA 00:10:16 2020-10-31 17:19:48 [INFO] [TRAIN] epoch=213, iter=79200/80000, loss=0.1268, lr=0.000169, batch_cost=0.6929, reader_cost=0.0008 | ETA 00:09:14 2020-10-31 17:20:57 [INFO] [TRAIN] epoch=214, iter=79300/80000, loss=0.1156, lr=0.000151, batch_cost=0.6974, reader_cost=0.0089 | ETA 00:08:08 2020-10-31 17:22:05 [INFO] [TRAIN] epoch=214, iter=79400/80000, loss=0.1156, lr=0.000132, batch_cost=0.6802, reader_cost=0.0003 | ETA 00:06:48 2020-10-31 17:23:13 [INFO] [TRAIN] epoch=214, iter=79500/80000, loss=0.1200, lr=0.000114, batch_cost=0.6797, reader_cost=0.0002 | ETA 00:05:39 2020-10-31 17:24:21 [INFO] [TRAIN] epoch=214, iter=79600/80000, loss=0.1258, lr=0.000095, batch_cost=0.6793, reader_cost=0.0006 | ETA 00:04:31 2020-10-31 17:25:29 [INFO] [TRAIN] epoch=215, iter=79700/80000, loss=0.1115, lr=0.000076, batch_cost=0.6826, reader_cost=0.0090 | ETA 00:03:24 2020-10-31 17:26:38 [INFO] [TRAIN] epoch=215, iter=79800/80000, loss=0.1285, lr=0.000056, batch_cost=0.6894, reader_cost=0.0007 | ETA 00:02:17 2020-10-31 17:27:47 [INFO] [TRAIN] epoch=215, iter=79900/80000, loss=0.1340, lr=0.000035, batch_cost=0.6813, reader_cost=0.0004 | ETA 00:01:08 2020-10-31 17:28:55 [INFO] [TRAIN] epoch=216, iter=80000/80000, loss=0.1305, lr=0.000010, batch_cost=0.6863, reader_cost=0.0097 | ETA 00:00:00 2020-10-31 17:29:02 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-10-31 17:34:40 [INFO] [EVAL] #Images=500 mIoU=0.8101 Acc=0.9658 Kappa=0.9556 2020-10-31 17:34:40 [INFO] [EVAL] Category IoU: [0.9845 0.8705 0.935 0.6296 0.642 0.6766 0.7383 0.8119 0.9299 0.6671 0.9511 0.8423 0.6593 0.9578 0.8379 0.9219 0.8592 0.6782 0.7987] 2020-10-31 17:34:40 [INFO] [EVAL] Category Acc: [0.9931 0.9294 0.962 0.8483 0.8254 0.8379 0.8455 0.912 0.9601 0.8258 0.9676 0.9017 0.8074 0.976 0.9364 0.9643 0.9473 0.841 0.8794] 2020-10-31 17:34:43 [INFO] [EVAL] The model with the best validation mIoU (0.8101) was saved at iter 80000.