2020-11-02 17:28:47 [INFO] ------------Environment Information------------- platform: Linux-3.10.0_3-0-0-34-x86_64-with-centos-7.5.1804-Core Python: 3.7.9 (default, Aug 31 2020, 12:42:55) [GCC 7.3.0] Paddle compiled with cuda: True NVCC: Cuda compilation tools, release 10.2, V10.2.89 cudnn: 7.6 GPUs used: 4 CUDA_VISIBLE_DEVICES: 4,5,6,7 GPU: ['GPU 0: Tesla V100-SXM2-16GB', 'GPU 1: Tesla V100-SXM2-16GB', 'GPU 2: Tesla V100-SXM2-16GB', 'GPU 3: Tesla V100-SXM2-16GB', 'GPU 4: Tesla V100-SXM2-16GB', 'GPU 5: Tesla V100-SXM2-16GB', 'GPU 6: Tesla V100-SXM2-16GB', 'GPU 7: Tesla V100-SXM2-16GB'] GCC: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36) PaddlePaddle: 2.0.0-rc0 OpenCV: 4.1.0 ------------------------------------------------ 2020-11-02 17:28:47 [INFO] ---------------Config Information--------------- batch_size: 2 iters: 80000 learning_rate: decay: end_lr: 1.0e-05 power: 0.9 type: poly value: 0.01 loss: coef: - 1 - 0.4 types: - ignore_index: 255 type: CrossEntropyLoss model: align_corners: false backbone: output_stride: 8 pretrained: https://bj.bcebos.com/paddleseg/dygraph/resnet101_vd_ssld.tar.gz type: ResNet101_vd enable_auxiliary_loss: true pretrained: null type: PSPNet optimizer: momentum: 0.9 type: sgd weight_decay: 4.0e-05 train_dataset: dataset_root: data/cityscapes mode: train transforms: - max_scale_factor: 2.0 min_scale_factor: 0.5 scale_step_size: 0.25 type: ResizeStepScaling - crop_size: - 1024 - 512 type: RandomPaddingCrop - type: RandomHorizontalFlip - brightness_range: 0.4 contrast_range: 0.4 saturation_range: 0.4 type: RandomDistort - type: Normalize type: Cityscapes val_dataset: dataset_root: data/cityscapes mode: val transforms: - type: Normalize type: Cityscapes ------------------------------------------------ 2020-11-02 17:28:51 [INFO] Loading pretrained model from https://bj.bcebos.com/paddleseg/dygraph/resnet101_vd_ssld.tar.gz 2020-11-02 17:28:53 [INFO] There are 530/530 variables loaded into ResNet_vd. 2020-11-02 17:30:24 [INFO] [TRAIN] epoch=1, iter=100/80000, loss=2.1714, lr=0.009989, batch_cost=0.8419, reader_cost=0.0157 | ETA 18:41:09 2020-11-02 17:31:44 [INFO] [TRAIN] epoch=1, iter=200/80000, loss=1.5519, lr=0.009978, batch_cost=0.7977, reader_cost=0.0007 | ETA 17:40:53 2020-11-02 17:33:03 [INFO] [TRAIN] epoch=1, iter=300/80000, loss=1.3482, lr=0.009966, batch_cost=0.7981, reader_cost=0.0006 | ETA 17:40:04 2020-11-02 17:34:24 [INFO] [TRAIN] epoch=2, iter=400/80000, loss=1.2207, lr=0.009955, batch_cost=0.8081, reader_cost=0.0106 | ETA 17:52:07 2020-11-02 17:35:45 [INFO] [TRAIN] epoch=2, iter=500/80000, loss=1.1224, lr=0.009944, batch_cost=0.8022, reader_cost=0.0013 | ETA 17:42:57 2020-11-02 17:37:04 [INFO] [TRAIN] epoch=2, iter=600/80000, loss=0.8639, lr=0.009933, batch_cost=0.7989, reader_cost=0.0008 | ETA 17:37:14 2020-11-02 17:38:24 [INFO] [TRAIN] epoch=2, iter=700/80000, loss=0.9100, lr=0.009921, batch_cost=0.8000, reader_cost=0.0012 | ETA 17:37:16 2020-11-02 17:39:46 [INFO] [TRAIN] epoch=3, iter=800/80000, loss=0.8355, lr=0.009910, batch_cost=0.8111, reader_cost=0.0096 | ETA 17:50:41 2020-11-02 17:41:05 [INFO] [TRAIN] epoch=3, iter=900/80000, loss=0.7790, lr=0.009899, batch_cost=0.7982, reader_cost=0.0015 | ETA 17:32:16 2020-11-02 17:42:25 [INFO] [TRAIN] epoch=3, iter=1000/80000, loss=0.6680, lr=0.009888, batch_cost=0.7918, reader_cost=0.0004 | ETA 17:22:32 2020-11-02 17:43:44 [INFO] [TRAIN] epoch=3, iter=1100/80000, loss=0.6630, lr=0.009876, batch_cost=0.7946, reader_cost=0.0005 | ETA 17:24:52 2020-11-02 17:45:05 [INFO] [TRAIN] epoch=4, iter=1200/80000, loss=0.6860, lr=0.009865, batch_cost=0.8102, reader_cost=0.0096 | ETA 17:44:03 2020-11-02 17:46:25 [INFO] [TRAIN] epoch=4, iter=1300/80000, loss=0.6348, lr=0.009854, batch_cost=0.8007, reader_cost=0.0010 | ETA 17:30:18 2020-11-02 17:47:45 [INFO] [TRAIN] epoch=4, iter=1400/80000, loss=0.6445, lr=0.009843, batch_cost=0.7999, reader_cost=0.0015 | ETA 17:27:54 2020-11-02 17:49:06 [INFO] [TRAIN] epoch=5, iter=1500/80000, loss=0.6770, lr=0.009831, batch_cost=0.8095, reader_cost=0.0095 | ETA 17:39:09 2020-11-02 17:50:26 [INFO] [TRAIN] epoch=5, iter=1600/80000, loss=0.5190, lr=0.009820, batch_cost=0.7949, reader_cost=0.0007 | ETA 17:18:40 2020-11-02 17:51:46 [INFO] [TRAIN] epoch=5, iter=1700/80000, loss=0.6820, lr=0.009809, batch_cost=0.8006, reader_cost=0.0009 | ETA 17:24:46 2020-11-02 17:53:06 [INFO] [TRAIN] epoch=5, iter=1800/80000, loss=0.5820, lr=0.009798, batch_cost=0.7994, reader_cost=0.0006 | ETA 17:21:54 2020-11-02 17:54:26 [INFO] [TRAIN] epoch=6, iter=1900/80000, loss=0.5089, lr=0.009786, batch_cost=0.8091, reader_cost=0.0117 | ETA 17:33:12 2020-11-02 17:55:46 [INFO] [TRAIN] epoch=6, iter=2000/80000, loss=0.4530, lr=0.009775, batch_cost=0.7943, reader_cost=0.0007 | ETA 17:12:36 2020-11-02 17:57:05 [INFO] [TRAIN] epoch=6, iter=2100/80000, loss=0.4401, lr=0.009764, batch_cost=0.7944, reader_cost=0.0004 | ETA 17:11:23 2020-11-02 17:58:26 [INFO] [TRAIN] epoch=6, iter=2200/80000, loss=0.4257, lr=0.009753, batch_cost=0.8057, reader_cost=0.0010 | ETA 17:24:44 2020-11-02 17:59:47 [INFO] [TRAIN] epoch=7, iter=2300/80000, loss=0.5357, lr=0.009741, batch_cost=0.8066, reader_cost=0.0110 | ETA 17:24:30 2020-11-02 18:01:06 [INFO] [TRAIN] epoch=7, iter=2400/80000, loss=0.4493, lr=0.009730, batch_cost=0.7931, reader_cost=0.0012 | ETA 17:05:44 2020-11-02 18:02:26 [INFO] [TRAIN] epoch=7, iter=2500/80000, loss=0.4358, lr=0.009719, batch_cost=0.7995, reader_cost=0.0004 | ETA 17:12:41 2020-11-02 18:03:45 [INFO] [TRAIN] epoch=7, iter=2600/80000, loss=0.4221, lr=0.009707, batch_cost=0.7949, reader_cost=0.0015 | ETA 17:05:28 2020-11-02 18:05:06 [INFO] [TRAIN] epoch=8, iter=2700/80000, loss=0.4944, lr=0.009696, batch_cost=0.8046, reader_cost=0.0109 | ETA 17:16:35 2020-11-02 18:06:25 [INFO] [TRAIN] epoch=8, iter=2800/80000, loss=0.4708, lr=0.009685, batch_cost=0.7959, reader_cost=0.0007 | ETA 17:04:02 2020-11-02 18:07:45 [INFO] [TRAIN] epoch=8, iter=2900/80000, loss=0.4058, lr=0.009674, batch_cost=0.7996, reader_cost=0.0006 | ETA 17:07:32 2020-11-02 18:09:06 [INFO] [TRAIN] epoch=9, iter=3000/80000, loss=0.4779, lr=0.009662, batch_cost=0.8115, reader_cost=0.0117 | ETA 17:21:25 2020-11-02 18:10:26 [INFO] [TRAIN] epoch=9, iter=3100/80000, loss=0.4745, lr=0.009651, batch_cost=0.7914, reader_cost=0.0007 | ETA 16:54:17 2020-11-02 18:11:45 [INFO] [TRAIN] epoch=9, iter=3200/80000, loss=0.4611, lr=0.009640, batch_cost=0.7980, reader_cost=0.0006 | ETA 17:01:26 2020-11-02 18:13:06 [INFO] [TRAIN] epoch=9, iter=3300/80000, loss=0.3736, lr=0.009628, batch_cost=0.8010, reader_cost=0.0008 | ETA 17:03:55 2020-11-02 18:14:26 [INFO] [TRAIN] epoch=10, iter=3400/80000, loss=0.3499, lr=0.009617, batch_cost=0.8078, reader_cost=0.0134 | ETA 17:11:18 2020-11-02 18:15:46 [INFO] [TRAIN] epoch=10, iter=3500/80000, loss=0.4635, lr=0.009606, batch_cost=0.7925, reader_cost=0.0007 | ETA 16:50:27 2020-11-02 18:17:05 [INFO] [TRAIN] epoch=10, iter=3600/80000, loss=0.4804, lr=0.009595, batch_cost=0.7983, reader_cost=0.0005 | ETA 16:56:27 2020-11-02 18:18:25 [INFO] [TRAIN] epoch=10, iter=3700/80000, loss=0.5134, lr=0.009583, batch_cost=0.7944, reader_cost=0.0004 | ETA 16:50:15 2020-11-02 18:19:46 [INFO] [TRAIN] epoch=11, iter=3800/80000, loss=0.4466, lr=0.009572, batch_cost=0.8066, reader_cost=0.0106 | ETA 17:04:20 2020-11-02 18:21:05 [INFO] [TRAIN] epoch=11, iter=3900/80000, loss=0.3760, lr=0.009561, batch_cost=0.7975, reader_cost=0.0011 | ETA 16:51:27 2020-11-02 18:22:22 [INFO] [TRAIN] epoch=11, iter=4000/80000, loss=0.3951, lr=0.009549, batch_cost=0.7686, reader_cost=0.0002 | ETA 16:13:33 2020-11-02 18:23:40 [INFO] [TRAIN] epoch=12, iter=4100/80000, loss=0.4133, lr=0.009538, batch_cost=0.7786, reader_cost=0.0100 | ETA 16:24:52 2020-11-02 18:24:59 [INFO] [TRAIN] epoch=12, iter=4200/80000, loss=0.3842, lr=0.009527, batch_cost=0.7872, reader_cost=0.0003 | ETA 16:34:28 2020-11-02 18:26:18 [INFO] [TRAIN] epoch=12, iter=4300/80000, loss=0.3854, lr=0.009516, batch_cost=0.7895, reader_cost=0.0004 | ETA 16:36:05 2020-11-02 18:27:37 [INFO] [TRAIN] epoch=12, iter=4400/80000, loss=0.3176, lr=0.009504, batch_cost=0.7912, reader_cost=0.0006 | ETA 16:36:52 2020-11-02 18:28:57 [INFO] [TRAIN] epoch=13, iter=4500/80000, loss=0.3857, lr=0.009493, batch_cost=0.8012, reader_cost=0.0099 | ETA 16:48:09 2020-11-02 18:30:16 [INFO] [TRAIN] epoch=13, iter=4600/80000, loss=0.4057, lr=0.009482, batch_cost=0.7907, reader_cost=0.0007 | ETA 16:33:35 2020-11-02 18:31:35 [INFO] [TRAIN] epoch=13, iter=4700/80000, loss=0.3794, lr=0.009470, batch_cost=0.7923, reader_cost=0.0002 | ETA 16:34:19 2020-11-02 18:32:54 [INFO] [TRAIN] epoch=13, iter=4800/80000, loss=0.4479, lr=0.009459, batch_cost=0.7854, reader_cost=0.0004 | ETA 16:24:21 2020-11-02 18:34:14 [INFO] [TRAIN] epoch=14, iter=4900/80000, loss=0.3877, lr=0.009448, batch_cost=0.8061, reader_cost=0.0109 | ETA 16:48:56 2020-11-02 18:35:34 [INFO] [TRAIN] epoch=14, iter=5000/80000, loss=0.4208, lr=0.009436, batch_cost=0.7938, reader_cost=0.0002 | ETA 16:32:13 2020-11-02 18:36:53 [INFO] [TRAIN] epoch=14, iter=5100/80000, loss=0.3580, lr=0.009425, batch_cost=0.7896, reader_cost=0.0010 | ETA 16:25:44 2020-11-02 18:38:12 [INFO] [TRAIN] epoch=14, iter=5200/80000, loss=0.3212, lr=0.009414, batch_cost=0.7906, reader_cost=0.0010 | ETA 16:25:34 2020-11-02 18:39:31 [INFO] [TRAIN] epoch=15, iter=5300/80000, loss=0.4081, lr=0.009402, batch_cost=0.7957, reader_cost=0.0103 | ETA 16:30:42 2020-11-02 18:40:50 [INFO] [TRAIN] epoch=15, iter=5400/80000, loss=0.3236, lr=0.009391, batch_cost=0.7835, reader_cost=0.0003 | ETA 16:14:09 2020-11-02 18:42:09 [INFO] [TRAIN] epoch=15, iter=5500/80000, loss=0.3580, lr=0.009380, batch_cost=0.7942, reader_cost=0.0006 | ETA 16:26:08 2020-11-02 18:43:29 [INFO] [TRAIN] epoch=16, iter=5600/80000, loss=0.3169, lr=0.009368, batch_cost=0.7973, reader_cost=0.0093 | ETA 16:28:37 2020-11-02 18:44:48 [INFO] [TRAIN] epoch=16, iter=5700/80000, loss=0.4031, lr=0.009357, batch_cost=0.7950, reader_cost=0.0013 | ETA 16:24:29 2020-11-02 18:46:07 [INFO] [TRAIN] epoch=16, iter=5800/80000, loss=0.3321, lr=0.009346, batch_cost=0.7871, reader_cost=0.0013 | ETA 16:13:20 2020-11-02 18:47:26 [INFO] [TRAIN] epoch=16, iter=5900/80000, loss=0.3644, lr=0.009335, batch_cost=0.7935, reader_cost=0.0008 | ETA 16:20:01 2020-11-02 18:48:46 [INFO] [TRAIN] epoch=17, iter=6000/80000, loss=0.3468, lr=0.009323, batch_cost=0.8001, reader_cost=0.0113 | ETA 16:26:43 2020-11-02 18:50:06 [INFO] [TRAIN] epoch=17, iter=6100/80000, loss=0.3092, lr=0.009312, batch_cost=0.7923, reader_cost=0.0010 | ETA 16:15:52 2020-11-02 18:51:25 [INFO] [TRAIN] epoch=17, iter=6200/80000, loss=0.3393, lr=0.009301, batch_cost=0.7905, reader_cost=0.0009 | ETA 16:12:17 2020-11-02 18:52:44 [INFO] [TRAIN] epoch=17, iter=6300/80000, loss=0.3548, lr=0.009289, batch_cost=0.7913, reader_cost=0.0003 | ETA 16:11:57 2020-11-02 18:54:04 [INFO] [TRAIN] epoch=18, iter=6400/80000, loss=0.3656, lr=0.009278, batch_cost=0.8035, reader_cost=0.0102 | ETA 16:25:40 2020-11-02 18:55:23 [INFO] [TRAIN] epoch=18, iter=6500/80000, loss=0.3299, lr=0.009267, batch_cost=0.7867, reader_cost=0.0007 | ETA 16:03:44 2020-11-02 18:56:42 [INFO] [TRAIN] epoch=18, iter=6600/80000, loss=0.2862, lr=0.009255, batch_cost=0.7922, reader_cost=0.0008 | ETA 16:09:10 2020-11-02 18:58:02 [INFO] [TRAIN] epoch=19, iter=6700/80000, loss=0.3188, lr=0.009244, batch_cost=0.8025, reader_cost=0.0100 | ETA 16:20:26 2020-11-02 18:59:21 [INFO] [TRAIN] epoch=19, iter=6800/80000, loss=0.2967, lr=0.009233, batch_cost=0.7887, reader_cost=0.0007 | ETA 16:02:11 2020-11-02 19:00:41 [INFO] [TRAIN] epoch=19, iter=6900/80000, loss=0.3356, lr=0.009221, batch_cost=0.7941, reader_cost=0.0010 | ETA 16:07:31 2020-11-02 19:02:00 [INFO] [TRAIN] epoch=19, iter=7000/80000, loss=0.3117, lr=0.009210, batch_cost=0.7947, reader_cost=0.0006 | ETA 16:06:54 2020-11-02 19:03:20 [INFO] [TRAIN] epoch=20, iter=7100/80000, loss=0.3360, lr=0.009199, batch_cost=0.8006, reader_cost=0.0093 | ETA 16:12:41 2020-11-02 19:04:39 [INFO] [TRAIN] epoch=20, iter=7200/80000, loss=0.3283, lr=0.009187, batch_cost=0.7903, reader_cost=0.0005 | ETA 15:58:50 2020-11-02 19:05:57 [INFO] [TRAIN] epoch=20, iter=7300/80000, loss=0.3143, lr=0.009176, batch_cost=0.7830, reader_cost=0.0007 | ETA 15:48:46 2020-11-02 19:07:16 [INFO] [TRAIN] epoch=20, iter=7400/80000, loss=0.3213, lr=0.009164, batch_cost=0.7869, reader_cost=0.0008 | ETA 15:52:09 2020-11-02 19:08:36 [INFO] [TRAIN] epoch=21, iter=7500/80000, loss=0.2693, lr=0.009153, batch_cost=0.7999, reader_cost=0.0107 | ETA 16:06:31 2020-11-02 19:09:55 [INFO] [TRAIN] epoch=21, iter=7600/80000, loss=0.3106, lr=0.009142, batch_cost=0.7841, reader_cost=0.0006 | ETA 15:46:09 2020-11-02 19:11:13 [INFO] [TRAIN] epoch=21, iter=7700/80000, loss=0.3136, lr=0.009130, batch_cost=0.7873, reader_cost=0.0007 | ETA 15:48:40 2020-11-02 19:12:32 [INFO] [TRAIN] epoch=21, iter=7800/80000, loss=0.3352, lr=0.009119, batch_cost=0.7905, reader_cost=0.0011 | ETA 15:51:15 2020-11-02 19:13:53 [INFO] [TRAIN] epoch=22, iter=7900/80000, loss=0.3803, lr=0.009108, batch_cost=0.8065, reader_cost=0.0105 | ETA 16:09:10 2020-11-02 19:15:12 [INFO] [TRAIN] epoch=22, iter=8000/80000, loss=0.2844, lr=0.009096, batch_cost=0.7923, reader_cost=0.0011 | ETA 15:50:48 2020-11-02 19:15:21 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-02 19:21:43 [INFO] [EVAL] #Images=500 mIoU=0.6266 Acc=0.9318 Kappa=0.9110 2020-11-02 19:21:43 [INFO] [EVAL] Category IoU: [0.9546 0.7155 0.8806 0.1922 0.5232 0.5458 0.5905 0.7082 0.8875 0.4847 0.8564 0.752 0.469 0.8991 0.418 0.6199 0.2604 0.4215 0.7254] 2020-11-02 19:21:43 [INFO] [EVAL] Category Acc: [0.9669 0.8675 0.9109 0.8329 0.7248 0.7553 0.8164 0.8744 0.9502 0.842 0.9815 0.8314 0.7904 0.9221 0.7787 0.7026 0.3764 0.6854 0.8607] 2020-11-02 19:21:47 [INFO] [EVAL] The model with the best validation mIoU (0.6266) was saved at iter 8000. 2020-11-02 19:23:06 [INFO] [TRAIN] epoch=22, iter=8100/80000, loss=0.2763, lr=0.009085, batch_cost=0.7889, reader_cost=0.0007 | ETA 15:45:25 2020-11-02 19:24:26 [INFO] [TRAIN] epoch=23, iter=8200/80000, loss=0.3422, lr=0.009074, batch_cost=0.7999, reader_cost=0.0096 | ETA 15:57:12 2020-11-02 19:25:46 [INFO] [TRAIN] epoch=23, iter=8300/80000, loss=0.3277, lr=0.009062, batch_cost=0.7918, reader_cost=0.0002 | ETA 15:46:08 2020-11-02 19:27:04 [INFO] [TRAIN] epoch=23, iter=8400/80000, loss=0.2604, lr=0.009051, batch_cost=0.7893, reader_cost=0.0006 | ETA 15:41:51 2020-11-02 19:28:24 [INFO] [TRAIN] epoch=23, iter=8500/80000, loss=0.2929, lr=0.009040, batch_cost=0.7906, reader_cost=0.0006 | ETA 15:42:07 2020-11-02 19:29:44 [INFO] [TRAIN] epoch=24, iter=8600/80000, loss=0.3459, lr=0.009028, batch_cost=0.7998, reader_cost=0.0078 | ETA 15:51:45 2020-11-02 19:31:02 [INFO] [TRAIN] epoch=24, iter=8700/80000, loss=0.3156, lr=0.009017, batch_cost=0.7879, reader_cost=0.0006 | ETA 15:36:18 2020-11-02 19:32:21 [INFO] [TRAIN] epoch=24, iter=8800/80000, loss=0.3455, lr=0.009005, batch_cost=0.7899, reader_cost=0.0011 | ETA 15:37:23 2020-11-02 19:33:40 [INFO] [TRAIN] epoch=24, iter=8900/80000, loss=0.3068, lr=0.008994, batch_cost=0.7863, reader_cost=0.0006 | ETA 15:31:45 2020-11-02 19:35:00 [INFO] [TRAIN] epoch=25, iter=9000/80000, loss=0.3486, lr=0.008983, batch_cost=0.7993, reader_cost=0.0086 | ETA 15:45:53 2020-11-02 19:36:18 [INFO] [TRAIN] epoch=25, iter=9100/80000, loss=0.3104, lr=0.008971, batch_cost=0.7853, reader_cost=0.0008 | ETA 15:27:57 2020-11-02 19:37:37 [INFO] [TRAIN] epoch=25, iter=9200/80000, loss=0.3225, lr=0.008960, batch_cost=0.7882, reader_cost=0.0008 | ETA 15:30:06 2020-11-02 19:38:56 [INFO] [TRAIN] epoch=25, iter=9300/80000, loss=0.3545, lr=0.008949, batch_cost=0.7886, reader_cost=0.0005 | ETA 15:29:15 2020-11-02 19:40:16 [INFO] [TRAIN] epoch=26, iter=9400/80000, loss=0.3157, lr=0.008937, batch_cost=0.7965, reader_cost=0.0086 | ETA 15:37:14 2020-11-02 19:41:35 [INFO] [TRAIN] epoch=26, iter=9500/80000, loss=0.2988, lr=0.008926, batch_cost=0.7905, reader_cost=0.0006 | ETA 15:28:53 2020-11-02 19:42:53 [INFO] [TRAIN] epoch=26, iter=9600/80000, loss=0.2952, lr=0.008914, batch_cost=0.7847, reader_cost=0.0005 | ETA 15:20:42 2020-11-02 19:44:12 [INFO] [TRAIN] epoch=27, iter=9700/80000, loss=0.2801, lr=0.008903, batch_cost=0.7919, reader_cost=0.0086 | ETA 15:27:48 2020-11-02 19:45:31 [INFO] [TRAIN] epoch=27, iter=9800/80000, loss=0.3011, lr=0.008892, batch_cost=0.7898, reader_cost=0.0004 | ETA 15:24:02 2020-11-02 19:46:50 [INFO] [TRAIN] epoch=27, iter=9900/80000, loss=0.3003, lr=0.008880, batch_cost=0.7866, reader_cost=0.0009 | ETA 15:19:02 2020-11-02 19:48:09 [INFO] [TRAIN] epoch=27, iter=10000/80000, loss=0.3310, lr=0.008869, batch_cost=0.7904, reader_cost=0.0007 | ETA 15:22:08 2020-11-02 19:49:28 [INFO] [TRAIN] epoch=28, iter=10100/80000, loss=0.3523, lr=0.008857, batch_cost=0.7916, reader_cost=0.0094 | ETA 15:22:16 2020-11-02 19:50:47 [INFO] [TRAIN] epoch=28, iter=10200/80000, loss=0.2347, lr=0.008846, batch_cost=0.7918, reader_cost=0.0014 | ETA 15:21:09 2020-11-02 19:52:07 [INFO] [TRAIN] epoch=28, iter=10300/80000, loss=0.2516, lr=0.008835, batch_cost=0.7924, reader_cost=0.0018 | ETA 15:20:31 2020-11-02 19:53:26 [INFO] [TRAIN] epoch=28, iter=10400/80000, loss=0.2969, lr=0.008823, batch_cost=0.7888, reader_cost=0.0005 | ETA 15:14:58 2020-11-02 19:54:45 [INFO] [TRAIN] epoch=29, iter=10500/80000, loss=0.2857, lr=0.008812, batch_cost=0.7973, reader_cost=0.0102 | ETA 15:23:34 2020-11-02 19:56:05 [INFO] [TRAIN] epoch=29, iter=10600/80000, loss=0.2637, lr=0.008801, batch_cost=0.7929, reader_cost=0.0010 | ETA 15:17:05 2020-11-02 19:57:24 [INFO] [TRAIN] epoch=29, iter=10700/80000, loss=0.2555, lr=0.008789, batch_cost=0.7924, reader_cost=0.0011 | ETA 15:15:15 2020-11-02 19:58:44 [INFO] [TRAIN] epoch=30, iter=10800/80000, loss=0.3713, lr=0.008778, batch_cost=0.7976, reader_cost=0.0091 | ETA 15:19:53 2020-11-02 20:00:03 [INFO] [TRAIN] epoch=30, iter=10900/80000, loss=0.2472, lr=0.008766, batch_cost=0.7902, reader_cost=0.0007 | ETA 15:10:02 2020-11-02 20:01:21 [INFO] [TRAIN] epoch=30, iter=11000/80000, loss=0.2813, lr=0.008755, batch_cost=0.7861, reader_cost=0.0005 | ETA 15:04:00 2020-11-02 20:02:41 [INFO] [TRAIN] epoch=30, iter=11100/80000, loss=0.2437, lr=0.008743, batch_cost=0.7934, reader_cost=0.0003 | ETA 15:11:03 2020-11-02 20:04:00 [INFO] [TRAIN] epoch=31, iter=11200/80000, loss=0.2988, lr=0.008732, batch_cost=0.7966, reader_cost=0.0092 | ETA 15:13:22 2020-11-02 20:05:19 [INFO] [TRAIN] epoch=31, iter=11300/80000, loss=0.3094, lr=0.008721, batch_cost=0.7905, reader_cost=0.0007 | ETA 15:05:08 2020-11-02 20:06:38 [INFO] [TRAIN] epoch=31, iter=11400/80000, loss=0.3085, lr=0.008709, batch_cost=0.7868, reader_cost=0.0005 | ETA 14:59:32 2020-11-02 20:07:56 [INFO] [TRAIN] epoch=31, iter=11500/80000, loss=0.2878, lr=0.008698, batch_cost=0.7840, reader_cost=0.0009 | ETA 14:55:02 2020-11-02 20:09:17 [INFO] [TRAIN] epoch=32, iter=11600/80000, loss=0.3198, lr=0.008686, batch_cost=0.8010, reader_cost=0.0097 | ETA 15:13:07 2020-11-02 20:10:35 [INFO] [TRAIN] epoch=32, iter=11700/80000, loss=0.3003, lr=0.008675, batch_cost=0.7852, reader_cost=0.0007 | ETA 14:53:46 2020-11-02 20:11:54 [INFO] [TRAIN] epoch=32, iter=11800/80000, loss=0.2740, lr=0.008664, batch_cost=0.7942, reader_cost=0.0012 | ETA 15:02:42 2020-11-02 20:13:12 [INFO] [TRAIN] epoch=32, iter=11900/80000, loss=0.2928, lr=0.008652, batch_cost=0.7712, reader_cost=0.0004 | ETA 14:35:17 2020-11-02 20:14:30 [INFO] [TRAIN] epoch=33, iter=12000/80000, loss=0.3179, lr=0.008641, batch_cost=0.7874, reader_cost=0.0080 | ETA 14:52:26 2020-11-02 20:15:49 [INFO] [TRAIN] epoch=33, iter=12100/80000, loss=0.2485, lr=0.008629, batch_cost=0.7860, reader_cost=0.0007 | ETA 14:49:26 2020-11-02 20:17:08 [INFO] [TRAIN] epoch=33, iter=12200/80000, loss=0.2457, lr=0.008618, batch_cost=0.7906, reader_cost=0.0007 | ETA 14:53:23 2020-11-02 20:18:28 [INFO] [TRAIN] epoch=34, iter=12300/80000, loss=0.2901, lr=0.008606, batch_cost=0.8045, reader_cost=0.0089 | ETA 15:07:46 2020-11-02 20:19:47 [INFO] [TRAIN] epoch=34, iter=12400/80000, loss=0.2744, lr=0.008595, batch_cost=0.7881, reader_cost=0.0008 | ETA 14:47:56 2020-11-02 20:21:06 [INFO] [TRAIN] epoch=34, iter=12500/80000, loss=0.2379, lr=0.008584, batch_cost=0.7868, reader_cost=0.0004 | ETA 14:45:06 2020-11-02 20:22:24 [INFO] [TRAIN] epoch=34, iter=12600/80000, loss=0.2459, lr=0.008572, batch_cost=0.7850, reader_cost=0.0010 | ETA 14:41:49 2020-11-02 20:23:44 [INFO] [TRAIN] epoch=35, iter=12700/80000, loss=0.2546, lr=0.008561, batch_cost=0.7961, reader_cost=0.0085 | ETA 14:52:59 2020-11-02 20:25:03 [INFO] [TRAIN] epoch=35, iter=12800/80000, loss=0.2313, lr=0.008549, batch_cost=0.7924, reader_cost=0.0005 | ETA 14:47:32 2020-11-02 20:26:22 [INFO] [TRAIN] epoch=35, iter=12900/80000, loss=0.2458, lr=0.008538, batch_cost=0.7908, reader_cost=0.0008 | ETA 14:44:19 2020-11-02 20:27:41 [INFO] [TRAIN] epoch=35, iter=13000/80000, loss=0.3211, lr=0.008526, batch_cost=0.7883, reader_cost=0.0009 | ETA 14:40:15 2020-11-02 20:29:01 [INFO] [TRAIN] epoch=36, iter=13100/80000, loss=0.2237, lr=0.008515, batch_cost=0.7985, reader_cost=0.0095 | ETA 14:50:19 2020-11-02 20:30:20 [INFO] [TRAIN] epoch=36, iter=13200/80000, loss=0.2742, lr=0.008504, batch_cost=0.7874, reader_cost=0.0007 | ETA 14:36:36 2020-11-02 20:31:39 [INFO] [TRAIN] epoch=36, iter=13300/80000, loss=0.3026, lr=0.008492, batch_cost=0.7894, reader_cost=0.0014 | ETA 14:37:35 2020-11-02 20:32:58 [INFO] [TRAIN] epoch=37, iter=13400/80000, loss=0.2896, lr=0.008481, batch_cost=0.7942, reader_cost=0.0092 | ETA 14:41:35 2020-11-02 20:34:17 [INFO] [TRAIN] epoch=37, iter=13500/80000, loss=0.2872, lr=0.008469, batch_cost=0.7930, reader_cost=0.0012 | ETA 14:38:57 2020-11-02 20:35:36 [INFO] [TRAIN] epoch=37, iter=13600/80000, loss=0.3113, lr=0.008458, batch_cost=0.7904, reader_cost=0.0010 | ETA 14:34:41 2020-11-02 20:36:55 [INFO] [TRAIN] epoch=37, iter=13700/80000, loss=0.2770, lr=0.008446, batch_cost=0.7893, reader_cost=0.0012 | ETA 14:32:07 2020-11-02 20:38:16 [INFO] [TRAIN] epoch=38, iter=13800/80000, loss=0.3087, lr=0.008435, batch_cost=0.8050, reader_cost=0.0090 | ETA 14:48:09 2020-11-02 20:39:35 [INFO] [TRAIN] epoch=38, iter=13900/80000, loss=0.2585, lr=0.008423, batch_cost=0.7902, reader_cost=0.0016 | ETA 14:30:32 2020-11-02 20:40:54 [INFO] [TRAIN] epoch=38, iter=14000/80000, loss=0.2334, lr=0.008412, batch_cost=0.7944, reader_cost=0.0005 | ETA 14:33:50 2020-11-02 20:42:13 [INFO] [TRAIN] epoch=38, iter=14100/80000, loss=0.3166, lr=0.008400, batch_cost=0.7895, reader_cost=0.0004 | ETA 14:27:08 2020-11-02 20:43:34 [INFO] [TRAIN] epoch=39, iter=14200/80000, loss=0.2902, lr=0.008389, batch_cost=0.8060, reader_cost=0.0088 | ETA 14:43:53 2020-11-02 20:44:53 [INFO] [TRAIN] epoch=39, iter=14300/80000, loss=0.3111, lr=0.008378, batch_cost=0.7888, reader_cost=0.0003 | ETA 14:23:43 2020-11-02 20:46:12 [INFO] [TRAIN] epoch=39, iter=14400/80000, loss=0.2840, lr=0.008366, batch_cost=0.7882, reader_cost=0.0006 | ETA 14:21:48 2020-11-02 20:47:31 [INFO] [TRAIN] epoch=39, iter=14500/80000, loss=0.2315, lr=0.008355, batch_cost=0.7928, reader_cost=0.0012 | ETA 14:25:26 2020-11-02 20:48:51 [INFO] [TRAIN] epoch=40, iter=14600/80000, loss=0.2794, lr=0.008343, batch_cost=0.8011, reader_cost=0.0097 | ETA 14:33:11 2020-11-02 20:50:10 [INFO] [TRAIN] epoch=40, iter=14700/80000, loss=0.2043, lr=0.008332, batch_cost=0.7896, reader_cost=0.0014 | ETA 14:19:20 2020-11-02 20:51:29 [INFO] [TRAIN] epoch=40, iter=14800/80000, loss=0.2463, lr=0.008320, batch_cost=0.7921, reader_cost=0.0004 | ETA 14:20:43 2020-11-02 20:52:49 [INFO] [TRAIN] epoch=41, iter=14900/80000, loss=0.2410, lr=0.008309, batch_cost=0.7970, reader_cost=0.0086 | ETA 14:24:44 2020-11-02 20:54:08 [INFO] [TRAIN] epoch=41, iter=15000/80000, loss=0.2991, lr=0.008297, batch_cost=0.7901, reader_cost=0.0010 | ETA 14:15:57 2020-11-02 20:55:27 [INFO] [TRAIN] epoch=41, iter=15100/80000, loss=0.2473, lr=0.008286, batch_cost=0.7883, reader_cost=0.0005 | ETA 14:12:42 2020-11-02 20:56:46 [INFO] [TRAIN] epoch=41, iter=15200/80000, loss=0.3012, lr=0.008274, batch_cost=0.7911, reader_cost=0.0018 | ETA 14:14:20 2020-11-02 20:58:06 [INFO] [TRAIN] epoch=42, iter=15300/80000, loss=0.2692, lr=0.008263, batch_cost=0.8008, reader_cost=0.0090 | ETA 14:23:33 2020-11-02 20:59:25 [INFO] [TRAIN] epoch=42, iter=15400/80000, loss=0.2376, lr=0.008251, batch_cost=0.7880, reader_cost=0.0012 | ETA 14:08:27 2020-11-02 21:00:44 [INFO] [TRAIN] epoch=42, iter=15500/80000, loss=0.2477, lr=0.008240, batch_cost=0.7926, reader_cost=0.0007 | ETA 14:12:04 2020-11-02 21:02:03 [INFO] [TRAIN] epoch=42, iter=15600/80000, loss=0.2658, lr=0.008228, batch_cost=0.7891, reader_cost=0.0009 | ETA 14:07:01 2020-11-02 21:03:23 [INFO] [TRAIN] epoch=43, iter=15700/80000, loss=0.2971, lr=0.008217, batch_cost=0.7987, reader_cost=0.0085 | ETA 14:15:53 2020-11-02 21:04:42 [INFO] [TRAIN] epoch=43, iter=15800/80000, loss=0.2269, lr=0.008205, batch_cost=0.7919, reader_cost=0.0003 | ETA 14:07:22 2020-11-02 21:06:02 [INFO] [TRAIN] epoch=43, iter=15900/80000, loss=0.2653, lr=0.008194, batch_cost=0.7960, reader_cost=0.0011 | ETA 14:10:21 2020-11-02 21:07:22 [INFO] [TRAIN] epoch=44, iter=16000/80000, loss=0.2717, lr=0.008182, batch_cost=0.8029, reader_cost=0.0098 | ETA 14:16:26 2020-11-02 21:07:31 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-02 21:13:57 [INFO] [EVAL] #Images=500 mIoU=0.7007 Acc=0.9477 Kappa=0.9320 2020-11-02 21:13:57 [INFO] [EVAL] Category IoU: [0.9717 0.8078 0.9011 0.2533 0.5649 0.5952 0.6428 0.7378 0.9031 0.5126 0.9324 0.7937 0.5812 0.9357 0.476 0.7901 0.5943 0.5687 0.7504] 2020-11-02 21:13:57 [INFO] [EVAL] Category Acc: [0.9832 0.9084 0.9496 0.8559 0.7524 0.8066 0.8517 0.9037 0.9204 0.9347 0.9602 0.8835 0.7473 0.9672 0.5067 0.8724 0.917 0.7991 0.8748] 2020-11-02 21:14:02 [INFO] [EVAL] The model with the best validation mIoU (0.7007) was saved at iter 16000. 2020-11-02 21:15:20 [INFO] [TRAIN] epoch=44, iter=16100/80000, loss=0.2470, lr=0.008171, batch_cost=0.7856, reader_cost=0.0005 | ETA 13:56:37 2020-11-02 21:16:39 [INFO] [TRAIN] epoch=44, iter=16200/80000, loss=0.2511, lr=0.008159, batch_cost=0.7881, reader_cost=0.0017 | ETA 13:57:57 2020-11-02 21:17:58 [INFO] [TRAIN] epoch=44, iter=16300/80000, loss=0.2284, lr=0.008148, batch_cost=0.7867, reader_cost=0.0006 | ETA 13:55:12 2020-11-02 21:19:17 [INFO] [TRAIN] epoch=45, iter=16400/80000, loss=0.2606, lr=0.008136, batch_cost=0.7964, reader_cost=0.0090 | ETA 14:04:13 2020-11-02 21:20:36 [INFO] [TRAIN] epoch=45, iter=16500/80000, loss=0.2176, lr=0.008125, batch_cost=0.7871, reader_cost=0.0005 | ETA 13:53:00 2020-11-02 21:21:55 [INFO] [TRAIN] epoch=45, iter=16600/80000, loss=0.2381, lr=0.008113, batch_cost=0.7892, reader_cost=0.0003 | ETA 13:53:52 2020-11-02 21:23:14 [INFO] [TRAIN] epoch=45, iter=16700/80000, loss=0.2951, lr=0.008102, batch_cost=0.7892, reader_cost=0.0004 | ETA 13:52:35 2020-11-02 21:24:34 [INFO] [TRAIN] epoch=46, iter=16800/80000, loss=0.3915, lr=0.008090, batch_cost=0.7967, reader_cost=0.0085 | ETA 13:59:13 2020-11-02 21:25:52 [INFO] [TRAIN] epoch=46, iter=16900/80000, loss=0.2647, lr=0.008079, batch_cost=0.7873, reader_cost=0.0003 | ETA 13:47:56 2020-11-02 21:27:12 [INFO] [TRAIN] epoch=46, iter=17000/80000, loss=0.2880, lr=0.008067, batch_cost=0.7949, reader_cost=0.0006 | ETA 13:54:38 2020-11-02 21:28:31 [INFO] [TRAIN] epoch=46, iter=17100/80000, loss=0.2634, lr=0.008056, batch_cost=0.7912, reader_cost=0.0005 | ETA 13:49:29 2020-11-02 21:29:52 [INFO] [TRAIN] epoch=47, iter=17200/80000, loss=0.3095, lr=0.008044, batch_cost=0.8065, reader_cost=0.0081 | ETA 14:04:10 2020-11-02 21:31:11 [INFO] [TRAIN] epoch=47, iter=17300/80000, loss=0.2377, lr=0.008033, batch_cost=0.7890, reader_cost=0.0012 | ETA 13:44:27 2020-11-02 21:32:29 [INFO] [TRAIN] epoch=47, iter=17400/80000, loss=0.2533, lr=0.008021, batch_cost=0.7870, reader_cost=0.0013 | ETA 13:41:04 2020-11-02 21:33:49 [INFO] [TRAIN] epoch=48, iter=17500/80000, loss=0.2715, lr=0.008010, batch_cost=0.7980, reader_cost=0.0109 | ETA 13:51:15 2020-11-02 21:35:08 [INFO] [TRAIN] epoch=48, iter=17600/80000, loss=0.3092, lr=0.007998, batch_cost=0.7890, reader_cost=0.0012 | ETA 13:40:35 2020-11-02 21:36:26 [INFO] [TRAIN] epoch=48, iter=17700/80000, loss=0.2546, lr=0.007987, batch_cost=0.7837, reader_cost=0.0006 | ETA 13:33:43 2020-11-02 21:37:45 [INFO] [TRAIN] epoch=48, iter=17800/80000, loss=0.2986, lr=0.007975, batch_cost=0.7899, reader_cost=0.0005 | ETA 13:38:50 2020-11-02 21:39:05 [INFO] [TRAIN] epoch=49, iter=17900/80000, loss=0.2397, lr=0.007964, batch_cost=0.8009, reader_cost=0.0095 | ETA 13:48:57 2020-11-02 21:40:24 [INFO] [TRAIN] epoch=49, iter=18000/80000, loss=0.2266, lr=0.007952, batch_cost=0.7898, reader_cost=0.0009 | ETA 13:36:07 2020-11-02 21:41:43 [INFO] [TRAIN] epoch=49, iter=18100/80000, loss=0.2416, lr=0.007941, batch_cost=0.7896, reader_cost=0.0012 | ETA 13:34:34 2020-11-02 21:43:03 [INFO] [TRAIN] epoch=49, iter=18200/80000, loss=0.2492, lr=0.007929, batch_cost=0.7931, reader_cost=0.0013 | ETA 13:36:55 2020-11-02 21:44:23 [INFO] [TRAIN] epoch=50, iter=18300/80000, loss=0.2914, lr=0.007918, batch_cost=0.8016, reader_cost=0.0100 | ETA 13:44:20 2020-11-02 21:45:42 [INFO] [TRAIN] epoch=50, iter=18400/80000, loss=0.2620, lr=0.007906, batch_cost=0.7907, reader_cost=0.0008 | ETA 13:31:44 2020-11-02 21:47:01 [INFO] [TRAIN] epoch=50, iter=18500/80000, loss=0.2308, lr=0.007895, batch_cost=0.7873, reader_cost=0.0008 | ETA 13:26:59 2020-11-02 21:48:19 [INFO] [TRAIN] epoch=50, iter=18600/80000, loss=0.2146, lr=0.007883, batch_cost=0.7876, reader_cost=0.0014 | ETA 13:25:57 2020-11-02 21:49:39 [INFO] [TRAIN] epoch=51, iter=18700/80000, loss=0.2084, lr=0.007871, batch_cost=0.7997, reader_cost=0.0088 | ETA 13:37:01 2020-11-02 21:50:58 [INFO] [TRAIN] epoch=51, iter=18800/80000, loss=0.2439, lr=0.007860, batch_cost=0.7886, reader_cost=0.0008 | ETA 13:24:24 2020-11-02 21:52:17 [INFO] [TRAIN] epoch=51, iter=18900/80000, loss=0.2303, lr=0.007848, batch_cost=0.7891, reader_cost=0.0005 | ETA 13:23:32 2020-11-02 21:53:37 [INFO] [TRAIN] epoch=52, iter=19000/80000, loss=0.2142, lr=0.007837, batch_cost=0.7965, reader_cost=0.0103 | ETA 13:29:43 2020-11-02 21:54:56 [INFO] [TRAIN] epoch=52, iter=19100/80000, loss=0.2340, lr=0.007825, batch_cost=0.7872, reader_cost=0.0006 | ETA 13:18:57 2020-11-02 21:56:15 [INFO] [TRAIN] epoch=52, iter=19200/80000, loss=0.2222, lr=0.007814, batch_cost=0.7899, reader_cost=0.0006 | ETA 13:20:22 2020-11-02 21:57:33 [INFO] [TRAIN] epoch=52, iter=19300/80000, loss=0.2297, lr=0.007802, batch_cost=0.7856, reader_cost=0.0014 | ETA 13:14:45 2020-11-02 21:58:53 [INFO] [TRAIN] epoch=53, iter=19400/80000, loss=0.2051, lr=0.007791, batch_cost=0.7993, reader_cost=0.0096 | ETA 13:27:20 2020-11-02 22:00:12 [INFO] [TRAIN] epoch=53, iter=19500/80000, loss=0.2063, lr=0.007779, batch_cost=0.7916, reader_cost=0.0007 | ETA 13:18:09 2020-11-02 22:01:31 [INFO] [TRAIN] epoch=53, iter=19600/80000, loss=0.2045, lr=0.007768, batch_cost=0.7887, reader_cost=0.0003 | ETA 13:13:54 2020-11-02 22:02:50 [INFO] [TRAIN] epoch=53, iter=19700/80000, loss=0.2424, lr=0.007756, batch_cost=0.7854, reader_cost=0.0009 | ETA 13:09:19 2020-11-02 22:04:08 [INFO] [TRAIN] epoch=54, iter=19800/80000, loss=0.2244, lr=0.007744, batch_cost=0.7797, reader_cost=0.0089 | ETA 13:02:15 2020-11-02 22:05:25 [INFO] [TRAIN] epoch=54, iter=19900/80000, loss=0.2346, lr=0.007733, batch_cost=0.7735, reader_cost=0.0004 | ETA 12:54:47 2020-11-02 22:06:44 [INFO] [TRAIN] epoch=54, iter=20000/80000, loss=0.2157, lr=0.007721, batch_cost=0.7876, reader_cost=0.0003 | ETA 13:07:38 2020-11-02 22:08:04 [INFO] [TRAIN] epoch=55, iter=20100/80000, loss=0.2869, lr=0.007710, batch_cost=0.7992, reader_cost=0.0090 | ETA 13:17:53 2020-11-02 22:09:22 [INFO] [TRAIN] epoch=55, iter=20200/80000, loss=0.2245, lr=0.007698, batch_cost=0.7888, reader_cost=0.0009 | ETA 13:06:08 2020-11-02 22:10:42 [INFO] [TRAIN] epoch=55, iter=20300/80000, loss=0.2327, lr=0.007687, batch_cost=0.7909, reader_cost=0.0008 | ETA 13:06:59 2020-11-02 22:12:01 [INFO] [TRAIN] epoch=55, iter=20400/80000, loss=0.2291, lr=0.007675, batch_cost=0.7902, reader_cost=0.0009 | ETA 13:04:54 2020-11-02 22:13:20 [INFO] [TRAIN] epoch=56, iter=20500/80000, loss=0.2394, lr=0.007663, batch_cost=0.7951, reader_cost=0.0095 | ETA 13:08:29 2020-11-02 22:14:39 [INFO] [TRAIN] epoch=56, iter=20600/80000, loss=0.2262, lr=0.007652, batch_cost=0.7849, reader_cost=0.0008 | ETA 12:57:05 2020-11-02 22:15:57 [INFO] [TRAIN] epoch=56, iter=20700/80000, loss=0.2275, lr=0.007640, batch_cost=0.7822, reader_cost=0.0006 | ETA 12:53:05 2020-11-02 22:17:15 [INFO] [TRAIN] epoch=56, iter=20800/80000, loss=0.2345, lr=0.007629, batch_cost=0.7825, reader_cost=0.0005 | ETA 12:52:02 2020-11-02 22:18:35 [INFO] [TRAIN] epoch=57, iter=20900/80000, loss=0.2155, lr=0.007617, batch_cost=0.7966, reader_cost=0.0088 | ETA 13:04:40 2020-11-02 22:19:54 [INFO] [TRAIN] epoch=57, iter=21000/80000, loss=0.1919, lr=0.007606, batch_cost=0.7912, reader_cost=0.0007 | ETA 12:57:58 2020-11-02 22:21:13 [INFO] [TRAIN] epoch=57, iter=21100/80000, loss=0.1891, lr=0.007594, batch_cost=0.7908, reader_cost=0.0008 | ETA 12:56:17 2020-11-02 22:22:32 [INFO] [TRAIN] epoch=57, iter=21200/80000, loss=0.2101, lr=0.007582, batch_cost=0.7888, reader_cost=0.0008 | ETA 12:53:03 2020-11-02 22:23:52 [INFO] [TRAIN] epoch=58, iter=21300/80000, loss=0.2265, lr=0.007571, batch_cost=0.8019, reader_cost=0.0095 | ETA 13:04:31 2020-11-02 22:25:11 [INFO] [TRAIN] epoch=58, iter=21400/80000, loss=0.2193, lr=0.007559, batch_cost=0.7898, reader_cost=0.0008 | ETA 12:51:24 2020-11-02 22:26:30 [INFO] [TRAIN] epoch=58, iter=21500/80000, loss=0.2306, lr=0.007548, batch_cost=0.7914, reader_cost=0.0008 | ETA 12:51:39 2020-11-02 22:27:50 [INFO] [TRAIN] epoch=59, iter=21600/80000, loss=0.2545, lr=0.007536, batch_cost=0.7980, reader_cost=0.0085 | ETA 12:56:44 2020-11-02 22:29:09 [INFO] [TRAIN] epoch=59, iter=21700/80000, loss=0.2143, lr=0.007524, batch_cost=0.7915, reader_cost=0.0008 | ETA 12:49:02 2020-11-02 22:30:28 [INFO] [TRAIN] epoch=59, iter=21800/80000, loss=0.2081, lr=0.007513, batch_cost=0.7909, reader_cost=0.0013 | ETA 12:47:09 2020-11-02 22:31:47 [INFO] [TRAIN] epoch=59, iter=21900/80000, loss=0.2385, lr=0.007501, batch_cost=0.7859, reader_cost=0.0003 | ETA 12:41:00 2020-11-02 22:33:07 [INFO] [TRAIN] epoch=60, iter=22000/80000, loss=0.3212, lr=0.007490, batch_cost=0.8016, reader_cost=0.0086 | ETA 12:54:52 2020-11-02 22:34:26 [INFO] [TRAIN] epoch=60, iter=22100/80000, loss=0.2148, lr=0.007478, batch_cost=0.7892, reader_cost=0.0008 | ETA 12:41:36 2020-11-02 22:35:45 [INFO] [TRAIN] epoch=60, iter=22200/80000, loss=0.2705, lr=0.007466, batch_cost=0.7909, reader_cost=0.0013 | ETA 12:41:51 2020-11-02 22:37:04 [INFO] [TRAIN] epoch=60, iter=22300/80000, loss=0.2314, lr=0.007455, batch_cost=0.7924, reader_cost=0.0008 | ETA 12:41:59 2020-11-02 22:38:24 [INFO] [TRAIN] epoch=61, iter=22400/80000, loss=0.2143, lr=0.007443, batch_cost=0.8023, reader_cost=0.0093 | ETA 12:50:10 2020-11-02 22:39:43 [INFO] [TRAIN] epoch=61, iter=22500/80000, loss=0.3052, lr=0.007432, batch_cost=0.7888, reader_cost=0.0004 | ETA 12:35:56 2020-11-02 22:41:02 [INFO] [TRAIN] epoch=61, iter=22600/80000, loss=0.2560, lr=0.007420, batch_cost=0.7900, reader_cost=0.0005 | ETA 12:35:44 2020-11-02 22:42:22 [INFO] [TRAIN] epoch=62, iter=22700/80000, loss=0.2483, lr=0.007408, batch_cost=0.8018, reader_cost=0.0096 | ETA 12:45:43 2020-11-02 22:43:41 [INFO] [TRAIN] epoch=62, iter=22800/80000, loss=0.2372, lr=0.007397, batch_cost=0.7866, reader_cost=0.0010 | ETA 12:29:54 2020-11-02 22:45:00 [INFO] [TRAIN] epoch=62, iter=22900/80000, loss=0.2187, lr=0.007385, batch_cost=0.7895, reader_cost=0.0007 | ETA 12:31:19 2020-11-02 22:46:19 [INFO] [TRAIN] epoch=62, iter=23000/80000, loss=0.2534, lr=0.007373, batch_cost=0.7928, reader_cost=0.0009 | ETA 12:33:11 2020-11-02 22:47:39 [INFO] [TRAIN] epoch=63, iter=23100/80000, loss=0.2090, lr=0.007362, batch_cost=0.8003, reader_cost=0.0093 | ETA 12:38:59 2020-11-02 22:48:58 [INFO] [TRAIN] epoch=63, iter=23200/80000, loss=0.2197, lr=0.007350, batch_cost=0.7873, reader_cost=0.0005 | ETA 12:25:19 2020-11-02 22:50:16 [INFO] [TRAIN] epoch=63, iter=23300/80000, loss=0.2409, lr=0.007339, batch_cost=0.7818, reader_cost=0.0005 | ETA 12:18:47 2020-11-02 22:51:35 [INFO] [TRAIN] epoch=63, iter=23400/80000, loss=0.2181, lr=0.007327, batch_cost=0.7844, reader_cost=0.0005 | ETA 12:19:55 2020-11-02 22:52:55 [INFO] [TRAIN] epoch=64, iter=23500/80000, loss=0.2700, lr=0.007315, batch_cost=0.8019, reader_cost=0.0090 | ETA 12:35:05 2020-11-02 22:54:14 [INFO] [TRAIN] epoch=64, iter=23600/80000, loss=0.2239, lr=0.007304, batch_cost=0.7925, reader_cost=0.0012 | ETA 12:24:55 2020-11-02 22:55:33 [INFO] [TRAIN] epoch=64, iter=23700/80000, loss=0.1846, lr=0.007292, batch_cost=0.7887, reader_cost=0.0007 | ETA 12:20:05 2020-11-02 22:56:52 [INFO] [TRAIN] epoch=64, iter=23800/80000, loss=0.1944, lr=0.007280, batch_cost=0.7874, reader_cost=0.0008 | ETA 12:17:30 2020-11-02 22:58:11 [INFO] [TRAIN] epoch=65, iter=23900/80000, loss=0.2746, lr=0.007269, batch_cost=0.7895, reader_cost=0.0089 | ETA 12:18:09 2020-11-02 22:59:28 [INFO] [TRAIN] epoch=65, iter=24000/80000, loss=0.2116, lr=0.007257, batch_cost=0.7688, reader_cost=0.0001 | ETA 11:57:32 2020-11-02 22:59:36 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-02 23:05:53 [INFO] [EVAL] #Images=500 mIoU=0.7482 Acc=0.9538 Kappa=0.9401 2020-11-02 23:05:53 [INFO] [EVAL] Category IoU: [0.9726 0.8102 0.9162 0.5053 0.603 0.6184 0.6876 0.7705 0.9181 0.5781 0.9358 0.8062 0.5999 0.9425 0.6436 0.8219 0.7232 0.6256 0.7369] 2020-11-02 23:05:53 [INFO] [EVAL] Category Acc: [0.9916 0.8765 0.9495 0.8373 0.7868 0.7295 0.8353 0.8831 0.951 0.8736 0.9595 0.8667 0.7204 0.9623 0.9001 0.9407 0.8293 0.7958 0.9156] 2020-11-02 23:05:58 [INFO] [EVAL] The model with the best validation mIoU (0.7482) was saved at iter 24000. 2020-11-02 23:07:17 [INFO] [TRAIN] epoch=65, iter=24100/80000, loss=0.2199, lr=0.007245, batch_cost=0.7891, reader_cost=0.0008 | ETA 12:15:12 2020-11-02 23:08:37 [INFO] [TRAIN] epoch=66, iter=24200/80000, loss=0.2148, lr=0.007234, batch_cost=0.7992, reader_cost=0.0091 | ETA 12:23:17 2020-11-02 23:09:56 [INFO] [TRAIN] epoch=66, iter=24300/80000, loss=0.2296, lr=0.007222, batch_cost=0.7861, reader_cost=0.0005 | ETA 12:09:45 2020-11-02 23:11:15 [INFO] [TRAIN] epoch=66, iter=24400/80000, loss=0.2406, lr=0.007210, batch_cost=0.7892, reader_cost=0.0003 | ETA 12:11:17 2020-11-02 23:12:33 [INFO] [TRAIN] epoch=66, iter=24500/80000, loss=0.2505, lr=0.007199, batch_cost=0.7874, reader_cost=0.0005 | ETA 12:08:22 2020-11-02 23:13:53 [INFO] [TRAIN] epoch=67, iter=24600/80000, loss=0.2272, lr=0.007187, batch_cost=0.7997, reader_cost=0.0080 | ETA 12:18:22 2020-11-02 23:15:12 [INFO] [TRAIN] epoch=67, iter=24700/80000, loss=0.2225, lr=0.007175, batch_cost=0.7883, reader_cost=0.0005 | ETA 12:06:31 2020-11-02 23:16:31 [INFO] [TRAIN] epoch=67, iter=24800/80000, loss=0.2283, lr=0.007164, batch_cost=0.7887, reader_cost=0.0007 | ETA 12:05:35 2020-11-02 23:17:50 [INFO] [TRAIN] epoch=67, iter=24900/80000, loss=0.2165, lr=0.007152, batch_cost=0.7892, reader_cost=0.0012 | ETA 12:04:42 2020-11-02 23:19:10 [INFO] [TRAIN] epoch=68, iter=25000/80000, loss=0.2488, lr=0.007140, batch_cost=0.7969, reader_cost=0.0089 | ETA 12:10:30 2020-11-02 23:20:29 [INFO] [TRAIN] epoch=68, iter=25100/80000, loss=0.2354, lr=0.007129, batch_cost=0.7901, reader_cost=0.0009 | ETA 12:02:56 2020-11-02 23:21:48 [INFO] [TRAIN] epoch=68, iter=25200/80000, loss=0.2321, lr=0.007117, batch_cost=0.7902, reader_cost=0.0006 | ETA 12:01:42 2020-11-02 23:23:08 [INFO] [TRAIN] epoch=69, iter=25300/80000, loss=0.2211, lr=0.007105, batch_cost=0.8038, reader_cost=0.0095 | ETA 12:12:46 2020-11-02 23:24:28 [INFO] [TRAIN] epoch=69, iter=25400/80000, loss=0.2151, lr=0.007094, batch_cost=0.7939, reader_cost=0.0008 | ETA 12:02:26 2020-11-02 23:25:46 [INFO] [TRAIN] epoch=69, iter=25500/80000, loss=0.2090, lr=0.007082, batch_cost=0.7839, reader_cost=0.0003 | ETA 11:52:00 2020-11-02 23:27:05 [INFO] [TRAIN] epoch=69, iter=25600/80000, loss=0.2077, lr=0.007070, batch_cost=0.7909, reader_cost=0.0008 | ETA 11:57:06 2020-11-02 23:28:25 [INFO] [TRAIN] epoch=70, iter=25700/80000, loss=0.2608, lr=0.007059, batch_cost=0.8019, reader_cost=0.0084 | ETA 12:05:40 2020-11-02 23:29:44 [INFO] [TRAIN] epoch=70, iter=25800/80000, loss=0.2138, lr=0.007047, batch_cost=0.7921, reader_cost=0.0007 | ETA 11:55:33 2020-11-02 23:31:04 [INFO] [TRAIN] epoch=70, iter=25900/80000, loss=0.2137, lr=0.007035, batch_cost=0.7932, reader_cost=0.0010 | ETA 11:55:09 2020-11-02 23:32:23 [INFO] [TRAIN] epoch=70, iter=26000/80000, loss=0.2406, lr=0.007024, batch_cost=0.7947, reader_cost=0.0012 | ETA 11:55:13 2020-11-02 23:33:43 [INFO] [TRAIN] epoch=71, iter=26100/80000, loss=0.2038, lr=0.007012, batch_cost=0.8001, reader_cost=0.0094 | ETA 11:58:42 2020-11-02 23:35:02 [INFO] [TRAIN] epoch=71, iter=26200/80000, loss=0.2139, lr=0.007000, batch_cost=0.7858, reader_cost=0.0004 | ETA 11:44:38 2020-11-02 23:36:21 [INFO] [TRAIN] epoch=71, iter=26300/80000, loss=0.2197, lr=0.006989, batch_cost=0.7878, reader_cost=0.0003 | ETA 11:45:04 2020-11-02 23:37:40 [INFO] [TRAIN] epoch=71, iter=26400/80000, loss=0.2283, lr=0.006977, batch_cost=0.7904, reader_cost=0.0003 | ETA 11:46:05 2020-11-02 23:39:00 [INFO] [TRAIN] epoch=72, iter=26500/80000, loss=0.2426, lr=0.006965, batch_cost=0.8016, reader_cost=0.0080 | ETA 11:54:46 2020-11-02 23:40:19 [INFO] [TRAIN] epoch=72, iter=26600/80000, loss=0.1988, lr=0.006954, batch_cost=0.7882, reader_cost=0.0009 | ETA 11:41:28 2020-11-02 23:41:37 [INFO] [TRAIN] epoch=72, iter=26700/80000, loss=0.2476, lr=0.006942, batch_cost=0.7885, reader_cost=0.0005 | ETA 11:40:26 2020-11-02 23:42:57 [INFO] [TRAIN] epoch=73, iter=26800/80000, loss=0.2354, lr=0.006930, batch_cost=0.7973, reader_cost=0.0092 | ETA 11:46:56 2020-11-02 23:44:16 [INFO] [TRAIN] epoch=73, iter=26900/80000, loss=0.2577, lr=0.006918, batch_cost=0.7861, reader_cost=0.0008 | ETA 11:35:43 2020-11-02 23:45:35 [INFO] [TRAIN] epoch=73, iter=27000/80000, loss=0.2232, lr=0.006907, batch_cost=0.7901, reader_cost=0.0020 | ETA 11:37:56 2020-11-02 23:46:54 [INFO] [TRAIN] epoch=73, iter=27100/80000, loss=0.2298, lr=0.006895, batch_cost=0.7895, reader_cost=0.0015 | ETA 11:36:06 2020-11-02 23:48:14 [INFO] [TRAIN] epoch=74, iter=27200/80000, loss=0.2219, lr=0.006883, batch_cost=0.8061, reader_cost=0.0095 | ETA 11:49:21 2020-11-02 23:49:33 [INFO] [TRAIN] epoch=74, iter=27300/80000, loss=0.2242, lr=0.006872, batch_cost=0.7874, reader_cost=0.0008 | ETA 11:31:37 2020-11-02 23:50:52 [INFO] [TRAIN] epoch=74, iter=27400/80000, loss=0.2244, lr=0.006860, batch_cost=0.7895, reader_cost=0.0006 | ETA 11:32:06 2020-11-02 23:52:11 [INFO] [TRAIN] epoch=74, iter=27500/80000, loss=0.2178, lr=0.006848, batch_cost=0.7901, reader_cost=0.0008 | ETA 11:31:22 2020-11-02 23:53:29 [INFO] [TRAIN] epoch=75, iter=27600/80000, loss=0.2314, lr=0.006836, batch_cost=0.7820, reader_cost=0.0102 | ETA 11:22:54 2020-11-02 23:54:46 [INFO] [TRAIN] epoch=75, iter=27700/80000, loss=0.1925, lr=0.006825, batch_cost=0.7677, reader_cost=0.0002 | ETA 11:09:11 2020-11-02 23:56:04 [INFO] [TRAIN] epoch=75, iter=27800/80000, loss=0.2211, lr=0.006813, batch_cost=0.7800, reader_cost=0.0003 | ETA 11:18:35 2020-11-02 23:57:23 [INFO] [TRAIN] epoch=75, iter=27900/80000, loss=0.2267, lr=0.006801, batch_cost=0.7880, reader_cost=0.0008 | ETA 11:24:17 2020-11-02 23:58:43 [INFO] [TRAIN] epoch=76, iter=28000/80000, loss=0.2493, lr=0.006789, batch_cost=0.8018, reader_cost=0.0098 | ETA 11:34:55 2020-11-03 00:00:02 [INFO] [TRAIN] epoch=76, iter=28100/80000, loss=0.2266, lr=0.006778, batch_cost=0.7869, reader_cost=0.0006 | ETA 11:20:41 2020-11-03 00:01:20 [INFO] [TRAIN] epoch=76, iter=28200/80000, loss=0.2025, lr=0.006766, batch_cost=0.7873, reader_cost=0.0004 | ETA 11:19:39 2020-11-03 00:02:40 [INFO] [TRAIN] epoch=77, iter=28300/80000, loss=0.2214, lr=0.006754, batch_cost=0.7955, reader_cost=0.0095 | ETA 11:25:28 2020-11-03 00:03:59 [INFO] [TRAIN] epoch=77, iter=28400/80000, loss=0.2103, lr=0.006743, batch_cost=0.7911, reader_cost=0.0006 | ETA 11:20:21 2020-11-03 00:05:18 [INFO] [TRAIN] epoch=77, iter=28500/80000, loss=0.1995, lr=0.006731, batch_cost=0.7865, reader_cost=0.0004 | ETA 11:15:02 2020-11-03 00:06:37 [INFO] [TRAIN] epoch=77, iter=28600/80000, loss=0.2074, lr=0.006719, batch_cost=0.7933, reader_cost=0.0012 | ETA 11:19:35 2020-11-03 00:07:58 [INFO] [TRAIN] epoch=78, iter=28700/80000, loss=0.1883, lr=0.006707, batch_cost=0.8044, reader_cost=0.0100 | ETA 11:27:46 2020-11-03 00:09:16 [INFO] [TRAIN] epoch=78, iter=28800/80000, loss=0.1994, lr=0.006696, batch_cost=0.7873, reader_cost=0.0009 | ETA 11:11:47 2020-11-03 00:10:35 [INFO] [TRAIN] epoch=78, iter=28900/80000, loss=0.2201, lr=0.006684, batch_cost=0.7890, reader_cost=0.0004 | ETA 11:11:57 2020-11-03 00:11:54 [INFO] [TRAIN] epoch=78, iter=29000/80000, loss=0.1956, lr=0.006672, batch_cost=0.7877, reader_cost=0.0004 | ETA 11:09:30 2020-11-03 00:13:14 [INFO] [TRAIN] epoch=79, iter=29100/80000, loss=0.2209, lr=0.006660, batch_cost=0.8025, reader_cost=0.0093 | ETA 11:20:48 2020-11-03 00:14:33 [INFO] [TRAIN] epoch=79, iter=29200/80000, loss=0.1881, lr=0.006648, batch_cost=0.7893, reader_cost=0.0012 | ETA 11:08:16 2020-11-03 00:15:52 [INFO] [TRAIN] epoch=79, iter=29300/80000, loss=0.2040, lr=0.006637, batch_cost=0.7882, reader_cost=0.0003 | ETA 11:06:02 2020-11-03 00:17:12 [INFO] [TRAIN] epoch=80, iter=29400/80000, loss=0.2246, lr=0.006625, batch_cost=0.7976, reader_cost=0.0096 | ETA 11:12:37 2020-11-03 00:18:31 [INFO] [TRAIN] epoch=80, iter=29500/80000, loss=0.2354, lr=0.006613, batch_cost=0.7915, reader_cost=0.0008 | ETA 11:06:12 2020-11-03 00:19:50 [INFO] [TRAIN] epoch=80, iter=29600/80000, loss=0.2231, lr=0.006601, batch_cost=0.7911, reader_cost=0.0014 | ETA 11:04:32 2020-11-03 00:21:09 [INFO] [TRAIN] epoch=80, iter=29700/80000, loss=0.2102, lr=0.006590, batch_cost=0.7933, reader_cost=0.0009 | ETA 11:05:04 2020-11-03 00:22:29 [INFO] [TRAIN] epoch=81, iter=29800/80000, loss=0.2378, lr=0.006578, batch_cost=0.8007, reader_cost=0.0090 | ETA 11:09:56 2020-11-03 00:23:48 [INFO] [TRAIN] epoch=81, iter=29900/80000, loss=0.2039, lr=0.006566, batch_cost=0.7893, reader_cost=0.0005 | ETA 10:59:04 2020-11-03 00:25:07 [INFO] [TRAIN] epoch=81, iter=30000/80000, loss=0.1915, lr=0.006554, batch_cost=0.7893, reader_cost=0.0004 | ETA 10:57:47 2020-11-03 00:26:26 [INFO] [TRAIN] epoch=81, iter=30100/80000, loss=0.2375, lr=0.006543, batch_cost=0.7897, reader_cost=0.0007 | ETA 10:56:44 2020-11-03 00:27:46 [INFO] [TRAIN] epoch=82, iter=30200/80000, loss=0.1914, lr=0.006531, batch_cost=0.8001, reader_cost=0.0085 | ETA 11:04:02 2020-11-03 00:29:05 [INFO] [TRAIN] epoch=82, iter=30300/80000, loss=0.1988, lr=0.006519, batch_cost=0.7926, reader_cost=0.0010 | ETA 10:56:30 2020-11-03 00:30:25 [INFO] [TRAIN] epoch=82, iter=30400/80000, loss=0.1913, lr=0.006507, batch_cost=0.7927, reader_cost=0.0005 | ETA 10:55:18 2020-11-03 00:31:43 [INFO] [TRAIN] epoch=82, iter=30500/80000, loss=0.1829, lr=0.006495, batch_cost=0.7864, reader_cost=0.0009 | ETA 10:48:48 2020-11-03 00:33:03 [INFO] [TRAIN] epoch=83, iter=30600/80000, loss=0.2016, lr=0.006484, batch_cost=0.7986, reader_cost=0.0074 | ETA 10:57:31 2020-11-03 00:34:22 [INFO] [TRAIN] epoch=83, iter=30700/80000, loss=0.1927, lr=0.006472, batch_cost=0.7895, reader_cost=0.0003 | ETA 10:48:43 2020-11-03 00:35:41 [INFO] [TRAIN] epoch=83, iter=30800/80000, loss=0.1901, lr=0.006460, batch_cost=0.7862, reader_cost=0.0005 | ETA 10:44:39 2020-11-03 00:37:01 [INFO] [TRAIN] epoch=84, iter=30900/80000, loss=0.2085, lr=0.006448, batch_cost=0.8023, reader_cost=0.0089 | ETA 10:56:35 2020-11-03 00:38:20 [INFO] [TRAIN] epoch=84, iter=31000/80000, loss=0.2246, lr=0.006436, batch_cost=0.7911, reader_cost=0.0005 | ETA 10:46:03 2020-11-03 00:39:39 [INFO] [TRAIN] epoch=84, iter=31100/80000, loss=0.1833, lr=0.006425, batch_cost=0.7872, reader_cost=0.0003 | ETA 10:41:31 2020-11-03 00:40:57 [INFO] [TRAIN] epoch=84, iter=31200/80000, loss=0.2067, lr=0.006413, batch_cost=0.7847, reader_cost=0.0003 | ETA 10:38:13 2020-11-03 00:42:18 [INFO] [TRAIN] epoch=85, iter=31300/80000, loss=0.1982, lr=0.006401, batch_cost=0.8022, reader_cost=0.0085 | ETA 10:51:05 2020-11-03 00:43:37 [INFO] [TRAIN] epoch=85, iter=31400/80000, loss=0.1837, lr=0.006389, batch_cost=0.7938, reader_cost=0.0008 | ETA 10:42:59 2020-11-03 00:44:56 [INFO] [TRAIN] epoch=85, iter=31500/80000, loss=0.1968, lr=0.006377, batch_cost=0.7865, reader_cost=0.0004 | ETA 10:35:45 2020-11-03 00:46:15 [INFO] [TRAIN] epoch=85, iter=31600/80000, loss=0.2005, lr=0.006366, batch_cost=0.7916, reader_cost=0.0011 | ETA 10:38:32 2020-11-03 00:47:35 [INFO] [TRAIN] epoch=86, iter=31700/80000, loss=0.2168, lr=0.006354, batch_cost=0.8000, reader_cost=0.0076 | ETA 10:44:00 2020-11-03 00:48:52 [INFO] [TRAIN] epoch=86, iter=31800/80000, loss=0.1991, lr=0.006342, batch_cost=0.7682, reader_cost=0.0002 | ETA 10:17:07 2020-11-03 00:50:09 [INFO] [TRAIN] epoch=86, iter=31900/80000, loss=0.1849, lr=0.006330, batch_cost=0.7742, reader_cost=0.0002 | ETA 10:20:38 2020-11-03 00:51:29 [INFO] [TRAIN] epoch=87, iter=32000/80000, loss=0.1977, lr=0.006318, batch_cost=0.7987, reader_cost=0.0093 | ETA 10:38:56 2020-11-03 00:51:38 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-03 00:58:02 [INFO] [EVAL] #Images=500 mIoU=0.7526 Acc=0.9513 Kappa=0.9369 2020-11-03 00:58:02 [INFO] [EVAL] Category IoU: [0.9619 0.7547 0.9218 0.5152 0.6304 0.6355 0.6874 0.7709 0.9182 0.4824 0.9448 0.8215 0.6322 0.9474 0.7035 0.8258 0.7793 0.5938 0.7734] 2020-11-03 00:58:02 [INFO] [EVAL] Category Acc: [0.9936 0.7872 0.951 0.8925 0.8451 0.8192 0.8698 0.9064 0.9412 0.941 0.9741 0.9105 0.7469 0.9727 0.9586 0.8761 0.9602 0.8646 0.8656] 2020-11-03 00:58:07 [INFO] [EVAL] The model with the best validation mIoU (0.7526) was saved at iter 32000. 2020-11-03 00:59:26 [INFO] [TRAIN] epoch=87, iter=32100/80000, loss=0.1764, lr=0.006306, batch_cost=0.7868, reader_cost=0.0006 | ETA 10:28:07 2020-11-03 01:00:44 [INFO] [TRAIN] epoch=87, iter=32200/80000, loss=0.1795, lr=0.006295, batch_cost=0.7855, reader_cost=0.0006 | ETA 10:25:45 2020-11-03 01:02:03 [INFO] [TRAIN] epoch=87, iter=32300/80000, loss=0.1885, lr=0.006283, batch_cost=0.7866, reader_cost=0.0007 | ETA 10:25:19 2020-11-03 01:03:23 [INFO] [TRAIN] epoch=88, iter=32400/80000, loss=0.2461, lr=0.006271, batch_cost=0.7990, reader_cost=0.0088 | ETA 10:33:51 2020-11-03 01:04:42 [INFO] [TRAIN] epoch=88, iter=32500/80000, loss=0.2109, lr=0.006259, batch_cost=0.7908, reader_cost=0.0006 | ETA 10:26:04 2020-11-03 01:06:01 [INFO] [TRAIN] epoch=88, iter=32600/80000, loss=0.1846, lr=0.006247, batch_cost=0.7900, reader_cost=0.0007 | ETA 10:24:05 2020-11-03 01:07:20 [INFO] [TRAIN] epoch=88, iter=32700/80000, loss=0.2119, lr=0.006235, batch_cost=0.7928, reader_cost=0.0007 | ETA 10:25:01 2020-11-03 01:08:41 [INFO] [TRAIN] epoch=89, iter=32800/80000, loss=0.1900, lr=0.006224, batch_cost=0.8054, reader_cost=0.0088 | ETA 10:33:35 2020-11-03 01:10:00 [INFO] [TRAIN] epoch=89, iter=32900/80000, loss=0.1965, lr=0.006212, batch_cost=0.7947, reader_cost=0.0010 | ETA 10:23:52 2020-11-03 01:11:19 [INFO] [TRAIN] epoch=89, iter=33000/80000, loss=0.1867, lr=0.006200, batch_cost=0.7899, reader_cost=0.0011 | ETA 10:18:43 2020-11-03 01:12:38 [INFO] [TRAIN] epoch=89, iter=33100/80000, loss=0.1976, lr=0.006188, batch_cost=0.7905, reader_cost=0.0006 | ETA 10:17:54 2020-11-03 01:13:59 [INFO] [TRAIN] epoch=90, iter=33200/80000, loss=0.2006, lr=0.006176, batch_cost=0.8059, reader_cost=0.0094 | ETA 10:28:34 2020-11-03 01:15:17 [INFO] [TRAIN] epoch=90, iter=33300/80000, loss=0.1762, lr=0.006164, batch_cost=0.7869, reader_cost=0.0004 | ETA 10:12:26 2020-11-03 01:16:36 [INFO] [TRAIN] epoch=90, iter=33400/80000, loss=0.2031, lr=0.006152, batch_cost=0.7842, reader_cost=0.0008 | ETA 10:09:04 2020-11-03 01:17:55 [INFO] [TRAIN] epoch=91, iter=33500/80000, loss=0.2091, lr=0.006141, batch_cost=0.7954, reader_cost=0.0095 | ETA 10:16:27 2020-11-03 01:19:14 [INFO] [TRAIN] epoch=91, iter=33600/80000, loss=0.2365, lr=0.006129, batch_cost=0.7894, reader_cost=0.0012 | ETA 10:10:29 2020-11-03 01:20:33 [INFO] [TRAIN] epoch=91, iter=33700/80000, loss=0.1856, lr=0.006117, batch_cost=0.7887, reader_cost=0.0006 | ETA 10:08:35 2020-11-03 01:21:52 [INFO] [TRAIN] epoch=91, iter=33800/80000, loss=0.1955, lr=0.006105, batch_cost=0.7828, reader_cost=0.0008 | ETA 10:02:44 2020-11-03 01:23:11 [INFO] [TRAIN] epoch=92, iter=33900/80000, loss=0.2252, lr=0.006093, batch_cost=0.7990, reader_cost=0.0097 | ETA 10:13:53 2020-11-03 01:24:30 [INFO] [TRAIN] epoch=92, iter=34000/80000, loss=0.1872, lr=0.006081, batch_cost=0.7863, reader_cost=0.0002 | ETA 10:02:49 2020-11-03 01:25:49 [INFO] [TRAIN] epoch=92, iter=34100/80000, loss=0.1788, lr=0.006069, batch_cost=0.7846, reader_cost=0.0004 | ETA 10:00:15 2020-11-03 01:27:07 [INFO] [TRAIN] epoch=92, iter=34200/80000, loss=0.2112, lr=0.006057, batch_cost=0.7864, reader_cost=0.0009 | ETA 10:00:19 2020-11-03 01:28:27 [INFO] [TRAIN] epoch=93, iter=34300/80000, loss=0.1880, lr=0.006046, batch_cost=0.7973, reader_cost=0.0088 | ETA 10:07:18 2020-11-03 01:29:46 [INFO] [TRAIN] epoch=93, iter=34400/80000, loss=0.1767, lr=0.006034, batch_cost=0.7899, reader_cost=0.0006 | ETA 10:00:18 2020-11-03 01:31:04 [INFO] [TRAIN] epoch=93, iter=34500/80000, loss=0.1876, lr=0.006022, batch_cost=0.7855, reader_cost=0.0004 | ETA 09:55:40 2020-11-03 01:32:24 [INFO] [TRAIN] epoch=94, iter=34600/80000, loss=0.1769, lr=0.006010, batch_cost=0.7919, reader_cost=0.0095 | ETA 09:59:13 2020-11-03 01:33:42 [INFO] [TRAIN] epoch=94, iter=34700/80000, loss=0.1997, lr=0.005998, batch_cost=0.7836, reader_cost=0.0002 | ETA 09:51:35 2020-11-03 01:35:01 [INFO] [TRAIN] epoch=94, iter=34800/80000, loss=0.1869, lr=0.005986, batch_cost=0.7885, reader_cost=0.0005 | ETA 09:54:02 2020-11-03 01:36:20 [INFO] [TRAIN] epoch=94, iter=34900/80000, loss=0.1765, lr=0.005974, batch_cost=0.7923, reader_cost=0.0005 | ETA 09:55:34 2020-11-03 01:37:40 [INFO] [TRAIN] epoch=95, iter=35000/80000, loss=0.2128, lr=0.005962, batch_cost=0.8016, reader_cost=0.0085 | ETA 10:01:12 2020-11-03 01:38:59 [INFO] [TRAIN] epoch=95, iter=35100/80000, loss=0.1829, lr=0.005950, batch_cost=0.7899, reader_cost=0.0010 | ETA 09:51:04 2020-11-03 01:40:18 [INFO] [TRAIN] epoch=95, iter=35200/80000, loss=0.1741, lr=0.005938, batch_cost=0.7870, reader_cost=0.0004 | ETA 09:47:39 2020-11-03 01:41:37 [INFO] [TRAIN] epoch=95, iter=35300/80000, loss=0.1997, lr=0.005927, batch_cost=0.7857, reader_cost=0.0008 | ETA 09:45:19 2020-11-03 01:42:56 [INFO] [TRAIN] epoch=96, iter=35400/80000, loss=0.1859, lr=0.005915, batch_cost=0.7908, reader_cost=0.0092 | ETA 09:47:51 2020-11-03 01:44:12 [INFO] [TRAIN] epoch=96, iter=35500/80000, loss=0.1970, lr=0.005903, batch_cost=0.7669, reader_cost=0.0002 | ETA 09:28:49 2020-11-03 01:45:29 [INFO] [TRAIN] epoch=96, iter=35600/80000, loss=0.2133, lr=0.005891, batch_cost=0.7707, reader_cost=0.0004 | ETA 09:30:17 2020-11-03 01:46:48 [INFO] [TRAIN] epoch=96, iter=35700/80000, loss=0.1744, lr=0.005879, batch_cost=0.7861, reader_cost=0.0009 | ETA 09:40:24 2020-11-03 01:48:08 [INFO] [TRAIN] epoch=97, iter=35800/80000, loss=0.1838, lr=0.005867, batch_cost=0.7973, reader_cost=0.0089 | ETA 09:47:21 2020-11-03 01:49:27 [INFO] [TRAIN] epoch=97, iter=35900/80000, loss=0.1684, lr=0.005855, batch_cost=0.7892, reader_cost=0.0006 | ETA 09:40:02 2020-11-03 01:50:45 [INFO] [TRAIN] epoch=97, iter=36000/80000, loss=0.1845, lr=0.005843, batch_cost=0.7841, reader_cost=0.0009 | ETA 09:34:58 2020-11-03 01:52:05 [INFO] [TRAIN] epoch=98, iter=36100/80000, loss=0.2051, lr=0.005831, batch_cost=0.8022, reader_cost=0.0101 | ETA 09:46:55 2020-11-03 01:53:24 [INFO] [TRAIN] epoch=98, iter=36200/80000, loss=0.1918, lr=0.005819, batch_cost=0.7907, reader_cost=0.0019 | ETA 09:37:14 2020-11-03 01:54:43 [INFO] [TRAIN] epoch=98, iter=36300/80000, loss=0.1871, lr=0.005807, batch_cost=0.7869, reader_cost=0.0008 | ETA 09:33:07 2020-11-03 01:56:02 [INFO] [TRAIN] epoch=98, iter=36400/80000, loss=0.1903, lr=0.005795, batch_cost=0.7859, reader_cost=0.0009 | ETA 09:31:03 2020-11-03 01:57:21 [INFO] [TRAIN] epoch=99, iter=36500/80000, loss=0.2042, lr=0.005783, batch_cost=0.7930, reader_cost=0.0086 | ETA 09:34:57 2020-11-03 01:58:40 [INFO] [TRAIN] epoch=99, iter=36600/80000, loss=0.1737, lr=0.005771, batch_cost=0.7901, reader_cost=0.0011 | ETA 09:31:32 2020-11-03 01:59:59 [INFO] [TRAIN] epoch=99, iter=36700/80000, loss=0.1749, lr=0.005760, batch_cost=0.7862, reader_cost=0.0005 | ETA 09:27:21 2020-11-03 02:01:18 [INFO] [TRAIN] epoch=99, iter=36800/80000, loss=0.1993, lr=0.005748, batch_cost=0.7921, reader_cost=0.0004 | ETA 09:30:18 2020-11-03 02:02:38 [INFO] [TRAIN] epoch=100, iter=36900/80000, loss=0.1928, lr=0.005736, batch_cost=0.7986, reader_cost=0.0095 | ETA 09:33:39 2020-11-03 02:03:57 [INFO] [TRAIN] epoch=100, iter=37000/80000, loss=0.1657, lr=0.005724, batch_cost=0.7892, reader_cost=0.0006 | ETA 09:25:36 2020-11-03 02:05:15 [INFO] [TRAIN] epoch=100, iter=37100/80000, loss=0.1801, lr=0.005712, batch_cost=0.7859, reader_cost=0.0007 | ETA 09:21:55 2020-11-03 02:06:34 [INFO] [TRAIN] epoch=100, iter=37200/80000, loss=0.2012, lr=0.005700, batch_cost=0.7901, reader_cost=0.0008 | ETA 09:23:35 2020-11-03 02:07:55 [INFO] [TRAIN] epoch=101, iter=37300/80000, loss=0.2051, lr=0.005688, batch_cost=0.8035, reader_cost=0.0091 | ETA 09:31:50 2020-11-03 02:09:13 [INFO] [TRAIN] epoch=101, iter=37400/80000, loss=0.1778, lr=0.005676, batch_cost=0.7868, reader_cost=0.0004 | ETA 09:18:38 2020-11-03 02:10:32 [INFO] [TRAIN] epoch=101, iter=37500/80000, loss=0.2528, lr=0.005664, batch_cost=0.7901, reader_cost=0.0004 | ETA 09:19:41 2020-11-03 02:11:52 [INFO] [TRAIN] epoch=102, iter=37600/80000, loss=0.2800, lr=0.005652, batch_cost=0.8006, reader_cost=0.0092 | ETA 09:25:45 2020-11-03 02:13:11 [INFO] [TRAIN] epoch=102, iter=37700/80000, loss=0.2017, lr=0.005640, batch_cost=0.7863, reader_cost=0.0002 | ETA 09:14:22 2020-11-03 02:14:30 [INFO] [TRAIN] epoch=102, iter=37800/80000, loss=0.1843, lr=0.005628, batch_cost=0.7872, reader_cost=0.0005 | ETA 09:13:39 2020-11-03 02:15:48 [INFO] [TRAIN] epoch=102, iter=37900/80000, loss=0.1739, lr=0.005616, batch_cost=0.7881, reader_cost=0.0006 | ETA 09:12:59 2020-11-03 02:17:08 [INFO] [TRAIN] epoch=103, iter=38000/80000, loss=0.1864, lr=0.005604, batch_cost=0.7947, reader_cost=0.0087 | ETA 09:16:19 2020-11-03 02:18:27 [INFO] [TRAIN] epoch=103, iter=38100/80000, loss=0.2036, lr=0.005592, batch_cost=0.7909, reader_cost=0.0003 | ETA 09:12:17 2020-11-03 02:19:46 [INFO] [TRAIN] epoch=103, iter=38200/80000, loss=0.1807, lr=0.005580, batch_cost=0.7946, reader_cost=0.0008 | ETA 09:13:35 2020-11-03 02:21:06 [INFO] [TRAIN] epoch=103, iter=38300/80000, loss=0.1929, lr=0.005568, batch_cost=0.7927, reader_cost=0.0010 | ETA 09:10:55 2020-11-03 02:22:26 [INFO] [TRAIN] epoch=104, iter=38400/80000, loss=0.1960, lr=0.005556, batch_cost=0.7999, reader_cost=0.0079 | ETA 09:14:34 2020-11-03 02:23:45 [INFO] [TRAIN] epoch=104, iter=38500/80000, loss=0.1715, lr=0.005544, batch_cost=0.7877, reader_cost=0.0007 | ETA 09:04:49 2020-11-03 02:25:03 [INFO] [TRAIN] epoch=104, iter=38600/80000, loss=0.2020, lr=0.005532, batch_cost=0.7869, reader_cost=0.0006 | ETA 09:02:58 2020-11-03 02:26:23 [INFO] [TRAIN] epoch=105, iter=38700/80000, loss=0.2032, lr=0.005520, batch_cost=0.7997, reader_cost=0.0088 | ETA 09:10:26 2020-11-03 02:27:42 [INFO] [TRAIN] epoch=105, iter=38800/80000, loss=0.1895, lr=0.005508, batch_cost=0.7920, reader_cost=0.0011 | ETA 09:03:48 2020-11-03 02:29:02 [INFO] [TRAIN] epoch=105, iter=38900/80000, loss=0.1783, lr=0.005496, batch_cost=0.7924, reader_cost=0.0010 | ETA 09:02:46 2020-11-03 02:30:20 [INFO] [TRAIN] epoch=105, iter=39000/80000, loss=0.1791, lr=0.005484, batch_cost=0.7881, reader_cost=0.0007 | ETA 08:58:33 2020-11-03 02:31:40 [INFO] [TRAIN] epoch=106, iter=39100/80000, loss=0.2063, lr=0.005472, batch_cost=0.7976, reader_cost=0.0087 | ETA 09:03:42 2020-11-03 02:32:59 [INFO] [TRAIN] epoch=106, iter=39200/80000, loss=0.1810, lr=0.005460, batch_cost=0.7842, reader_cost=0.0010 | ETA 08:53:17 2020-11-03 02:34:17 [INFO] [TRAIN] epoch=106, iter=39300/80000, loss=0.1909, lr=0.005448, batch_cost=0.7868, reader_cost=0.0004 | ETA 08:53:42 2020-11-03 02:35:36 [INFO] [TRAIN] epoch=106, iter=39400/80000, loss=0.1876, lr=0.005436, batch_cost=0.7909, reader_cost=0.0004 | ETA 08:55:11 2020-11-03 02:36:56 [INFO] [TRAIN] epoch=107, iter=39500/80000, loss=0.1747, lr=0.005424, batch_cost=0.7994, reader_cost=0.0083 | ETA 08:59:36 2020-11-03 02:38:15 [INFO] [TRAIN] epoch=107, iter=39600/80000, loss=0.1640, lr=0.005412, batch_cost=0.7817, reader_cost=0.0011 | ETA 08:46:22 2020-11-03 02:39:32 [INFO] [TRAIN] epoch=107, iter=39700/80000, loss=0.1868, lr=0.005400, batch_cost=0.7705, reader_cost=0.0007 | ETA 08:37:31 2020-11-03 02:40:49 [INFO] [TRAIN] epoch=107, iter=39800/80000, loss=0.1968, lr=0.005388, batch_cost=0.7742, reader_cost=0.0004 | ETA 08:38:43 2020-11-03 02:42:09 [INFO] [TRAIN] epoch=108, iter=39900/80000, loss=0.1873, lr=0.005376, batch_cost=0.7958, reader_cost=0.0092 | ETA 08:51:53 2020-11-03 02:43:28 [INFO] [TRAIN] epoch=108, iter=40000/80000, loss=0.2051, lr=0.005364, batch_cost=0.7894, reader_cost=0.0004 | ETA 08:46:15 2020-11-03 02:43:37 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-03 02:49:59 [INFO] [EVAL] #Images=500 mIoU=0.7578 Acc=0.9563 Kappa=0.9431 2020-11-03 02:49:59 [INFO] [EVAL] Category IoU: [0.9779 0.8346 0.9156 0.4275 0.6098 0.6335 0.6808 0.7801 0.9174 0.4724 0.9333 0.8158 0.6277 0.9491 0.7389 0.8707 0.8078 0.6332 0.7711] 2020-11-03 02:49:59 [INFO] [EVAL] Category Acc: [0.989 0.9156 0.945 0.8185 0.8157 0.8022 0.8729 0.9074 0.9447 0.9478 0.9522 0.8773 0.7884 0.9721 0.8727 0.948 0.9294 0.8602 0.8824] 2020-11-03 02:50:04 [INFO] [EVAL] The model with the best validation mIoU (0.7578) was saved at iter 40000. 2020-11-03 02:51:23 [INFO] [TRAIN] epoch=108, iter=40100/80000, loss=0.1878, lr=0.005352, batch_cost=0.7851, reader_cost=0.0008 | ETA 08:42:04 2020-11-03 02:52:42 [INFO] [TRAIN] epoch=109, iter=40200/80000, loss=0.1879, lr=0.005340, batch_cost=0.7966, reader_cost=0.0099 | ETA 08:48:23 2020-11-03 02:54:02 [INFO] [TRAIN] epoch=109, iter=40300/80000, loss=0.1810, lr=0.005327, batch_cost=0.7915, reader_cost=0.0004 | ETA 08:43:43 2020-11-03 02:55:21 [INFO] [TRAIN] epoch=109, iter=40400/80000, loss=0.1663, lr=0.005315, batch_cost=0.7900, reader_cost=0.0007 | ETA 08:41:24 2020-11-03 02:56:39 [INFO] [TRAIN] epoch=109, iter=40500/80000, loss=0.1752, lr=0.005303, batch_cost=0.7832, reader_cost=0.0006 | ETA 08:35:34 2020-11-03 02:57:59 [INFO] [TRAIN] epoch=110, iter=40600/80000, loss=0.1730, lr=0.005291, batch_cost=0.8018, reader_cost=0.0082 | ETA 08:46:31 2020-11-03 02:59:18 [INFO] [TRAIN] epoch=110, iter=40700/80000, loss=0.1748, lr=0.005279, batch_cost=0.7897, reader_cost=0.0008 | ETA 08:37:16 2020-11-03 03:00:37 [INFO] [TRAIN] epoch=110, iter=40800/80000, loss=0.1796, lr=0.005267, batch_cost=0.7887, reader_cost=0.0007 | ETA 08:35:17 2020-11-03 03:01:56 [INFO] [TRAIN] epoch=110, iter=40900/80000, loss=0.1833, lr=0.005255, batch_cost=0.7887, reader_cost=0.0004 | ETA 08:33:58 2020-11-03 03:03:16 [INFO] [TRAIN] epoch=111, iter=41000/80000, loss=0.2011, lr=0.005243, batch_cost=0.8027, reader_cost=0.0093 | ETA 08:41:45 2020-11-03 03:04:35 [INFO] [TRAIN] epoch=111, iter=41100/80000, loss=0.1646, lr=0.005231, batch_cost=0.7908, reader_cost=0.0007 | ETA 08:32:43 2020-11-03 03:05:54 [INFO] [TRAIN] epoch=111, iter=41200/80000, loss=0.1692, lr=0.005219, batch_cost=0.7883, reader_cost=0.0012 | ETA 08:29:47 2020-11-03 03:07:14 [INFO] [TRAIN] epoch=112, iter=41300/80000, loss=0.1857, lr=0.005207, batch_cost=0.7979, reader_cost=0.0092 | ETA 08:34:39 2020-11-03 03:08:32 [INFO] [TRAIN] epoch=112, iter=41400/80000, loss=0.1629, lr=0.005195, batch_cost=0.7831, reader_cost=0.0007 | ETA 08:23:48 2020-11-03 03:09:51 [INFO] [TRAIN] epoch=112, iter=41500/80000, loss=0.1677, lr=0.005183, batch_cost=0.7893, reader_cost=0.0007 | ETA 08:26:27 2020-11-03 03:11:10 [INFO] [TRAIN] epoch=112, iter=41600/80000, loss=0.1640, lr=0.005171, batch_cost=0.7884, reader_cost=0.0008 | ETA 08:24:35 2020-11-03 03:12:30 [INFO] [TRAIN] epoch=113, iter=41700/80000, loss=0.2024, lr=0.005158, batch_cost=0.8030, reader_cost=0.0104 | ETA 08:32:35 2020-11-03 03:13:49 [INFO] [TRAIN] epoch=113, iter=41800/80000, loss=0.1714, lr=0.005146, batch_cost=0.7856, reader_cost=0.0003 | ETA 08:20:08 2020-11-03 03:15:08 [INFO] [TRAIN] epoch=113, iter=41900/80000, loss=0.1541, lr=0.005134, batch_cost=0.7888, reader_cost=0.0012 | ETA 08:20:53 2020-11-03 03:16:26 [INFO] [TRAIN] epoch=113, iter=42000/80000, loss=0.1653, lr=0.005122, batch_cost=0.7874, reader_cost=0.0011 | ETA 08:18:39 2020-11-03 03:17:46 [INFO] [TRAIN] epoch=114, iter=42100/80000, loss=0.1950, lr=0.005110, batch_cost=0.7937, reader_cost=0.0091 | ETA 08:21:20 2020-11-03 03:19:05 [INFO] [TRAIN] epoch=114, iter=42200/80000, loss=0.1909, lr=0.005098, batch_cost=0.7908, reader_cost=0.0009 | ETA 08:18:12 2020-11-03 03:20:24 [INFO] [TRAIN] epoch=114, iter=42300/80000, loss=0.1602, lr=0.005086, batch_cost=0.7942, reader_cost=0.0015 | ETA 08:19:00 2020-11-03 03:21:43 [INFO] [TRAIN] epoch=114, iter=42400/80000, loss=0.1595, lr=0.005074, batch_cost=0.7891, reader_cost=0.0015 | ETA 08:14:29 2020-11-03 03:23:03 [INFO] [TRAIN] epoch=115, iter=42500/80000, loss=0.1891, lr=0.005062, batch_cost=0.8030, reader_cost=0.0082 | ETA 08:21:54 2020-11-03 03:24:23 [INFO] [TRAIN] epoch=115, iter=42600/80000, loss=0.1488, lr=0.005049, batch_cost=0.7906, reader_cost=0.0006 | ETA 08:12:49 2020-11-03 03:25:41 [INFO] [TRAIN] epoch=115, iter=42700/80000, loss=0.1704, lr=0.005037, batch_cost=0.7876, reader_cost=0.0015 | ETA 08:09:39 2020-11-03 03:27:02 [INFO] [TRAIN] epoch=116, iter=42800/80000, loss=0.2025, lr=0.005025, batch_cost=0.8021, reader_cost=0.0083 | ETA 08:17:16 2020-11-03 03:28:21 [INFO] [TRAIN] epoch=116, iter=42900/80000, loss=0.1513, lr=0.005013, batch_cost=0.7922, reader_cost=0.0010 | ETA 08:09:48 2020-11-03 03:29:40 [INFO] [TRAIN] epoch=116, iter=43000/80000, loss=0.1653, lr=0.005001, batch_cost=0.7914, reader_cost=0.0003 | ETA 08:08:02 2020-11-03 03:30:59 [INFO] [TRAIN] epoch=116, iter=43100/80000, loss=0.1644, lr=0.004989, batch_cost=0.7904, reader_cost=0.0003 | ETA 08:06:03 2020-11-03 03:32:19 [INFO] [TRAIN] epoch=117, iter=43200/80000, loss=0.1791, lr=0.004977, batch_cost=0.8011, reader_cost=0.0098 | ETA 08:11:19 2020-11-03 03:33:36 [INFO] [TRAIN] epoch=117, iter=43300/80000, loss=0.1956, lr=0.004964, batch_cost=0.7728, reader_cost=0.0006 | ETA 07:52:42 2020-11-03 03:34:53 [INFO] [TRAIN] epoch=117, iter=43400/80000, loss=0.1980, lr=0.004952, batch_cost=0.7652, reader_cost=0.0003 | ETA 07:46:47 2020-11-03 03:36:11 [INFO] [TRAIN] epoch=117, iter=43500/80000, loss=0.1708, lr=0.004940, batch_cost=0.7823, reader_cost=0.0003 | ETA 07:55:52 2020-11-03 03:37:32 [INFO] [TRAIN] epoch=118, iter=43600/80000, loss=0.1825, lr=0.004928, batch_cost=0.8048, reader_cost=0.0088 | ETA 08:08:15 2020-11-03 03:38:51 [INFO] [TRAIN] epoch=118, iter=43700/80000, loss=0.1617, lr=0.004916, batch_cost=0.7914, reader_cost=0.0012 | ETA 07:58:48 2020-11-03 03:40:10 [INFO] [TRAIN] epoch=118, iter=43800/80000, loss=0.1604, lr=0.004904, batch_cost=0.7946, reader_cost=0.0011 | ETA 07:59:23 2020-11-03 03:41:30 [INFO] [TRAIN] epoch=119, iter=43900/80000, loss=0.1675, lr=0.004891, batch_cost=0.8003, reader_cost=0.0085 | ETA 08:01:32 2020-11-03 03:42:49 [INFO] [TRAIN] epoch=119, iter=44000/80000, loss=0.1645, lr=0.004879, batch_cost=0.7905, reader_cost=0.0005 | ETA 07:54:18 2020-11-03 03:44:08 [INFO] [TRAIN] epoch=119, iter=44100/80000, loss=0.1626, lr=0.004867, batch_cost=0.7923, reader_cost=0.0013 | ETA 07:54:02 2020-11-03 03:45:27 [INFO] [TRAIN] epoch=119, iter=44200/80000, loss=0.1808, lr=0.004855, batch_cost=0.7888, reader_cost=0.0006 | ETA 07:50:39 2020-11-03 03:46:48 [INFO] [TRAIN] epoch=120, iter=44300/80000, loss=0.1880, lr=0.004843, batch_cost=0.8021, reader_cost=0.0087 | ETA 07:57:13 2020-11-03 03:48:06 [INFO] [TRAIN] epoch=120, iter=44400/80000, loss=0.1631, lr=0.004831, batch_cost=0.7871, reader_cost=0.0009 | ETA 07:46:59 2020-11-03 03:49:26 [INFO] [TRAIN] epoch=120, iter=44500/80000, loss=0.1624, lr=0.004818, batch_cost=0.7931, reader_cost=0.0013 | ETA 07:49:14 2020-11-03 03:50:45 [INFO] [TRAIN] epoch=120, iter=44600/80000, loss=0.1642, lr=0.004806, batch_cost=0.7918, reader_cost=0.0005 | ETA 07:47:08 2020-11-03 03:52:05 [INFO] [TRAIN] epoch=121, iter=44700/80000, loss=0.1639, lr=0.004794, batch_cost=0.8009, reader_cost=0.0100 | ETA 07:51:10 2020-11-03 03:53:24 [INFO] [TRAIN] epoch=121, iter=44800/80000, loss=0.1600, lr=0.004782, batch_cost=0.7915, reader_cost=0.0010 | ETA 07:44:21 2020-11-03 03:54:43 [INFO] [TRAIN] epoch=121, iter=44900/80000, loss=0.1615, lr=0.004770, batch_cost=0.7862, reader_cost=0.0004 | ETA 07:39:56 2020-11-03 03:56:01 [INFO] [TRAIN] epoch=121, iter=45000/80000, loss=0.1767, lr=0.004757, batch_cost=0.7843, reader_cost=0.0009 | ETA 07:37:31 2020-11-03 03:57:21 [INFO] [TRAIN] epoch=122, iter=45100/80000, loss=0.1789, lr=0.004745, batch_cost=0.7988, reader_cost=0.0089 | ETA 07:44:37 2020-11-03 03:58:39 [INFO] [TRAIN] epoch=122, iter=45200/80000, loss=0.1506, lr=0.004733, batch_cost=0.7853, reader_cost=0.0005 | ETA 07:35:26 2020-11-03 03:59:58 [INFO] [TRAIN] epoch=122, iter=45300/80000, loss=0.1763, lr=0.004721, batch_cost=0.7881, reader_cost=0.0004 | ETA 07:35:45 2020-11-03 04:01:18 [INFO] [TRAIN] epoch=123, iter=45400/80000, loss=0.1817, lr=0.004709, batch_cost=0.7977, reader_cost=0.0085 | ETA 07:40:01 2020-11-03 04:02:37 [INFO] [TRAIN] epoch=123, iter=45500/80000, loss=0.1719, lr=0.004696, batch_cost=0.7909, reader_cost=0.0007 | ETA 07:34:44 2020-11-03 04:03:56 [INFO] [TRAIN] epoch=123, iter=45600/80000, loss=0.1785, lr=0.004684, batch_cost=0.7881, reader_cost=0.0008 | ETA 07:31:50 2020-11-03 04:05:15 [INFO] [TRAIN] epoch=123, iter=45700/80000, loss=0.2024, lr=0.004672, batch_cost=0.7864, reader_cost=0.0007 | ETA 07:29:32 2020-11-03 04:06:34 [INFO] [TRAIN] epoch=124, iter=45800/80000, loss=0.1964, lr=0.004660, batch_cost=0.7985, reader_cost=0.0098 | ETA 07:35:07 2020-11-03 04:07:54 [INFO] [TRAIN] epoch=124, iter=45900/80000, loss=0.1602, lr=0.004647, batch_cost=0.7946, reader_cost=0.0013 | ETA 07:31:34 2020-11-03 04:09:13 [INFO] [TRAIN] epoch=124, iter=46000/80000, loss=0.1562, lr=0.004635, batch_cost=0.7919, reader_cost=0.0012 | ETA 07:28:44 2020-11-03 04:10:32 [INFO] [TRAIN] epoch=124, iter=46100/80000, loss=0.1702, lr=0.004623, batch_cost=0.7926, reader_cost=0.0009 | ETA 07:27:49 2020-11-03 04:11:52 [INFO] [TRAIN] epoch=125, iter=46200/80000, loss=0.1757, lr=0.004611, batch_cost=0.7987, reader_cost=0.0088 | ETA 07:29:55 2020-11-03 04:13:11 [INFO] [TRAIN] epoch=125, iter=46300/80000, loss=0.1797, lr=0.004598, batch_cost=0.7862, reader_cost=0.0006 | ETA 07:21:35 2020-11-03 04:14:30 [INFO] [TRAIN] epoch=125, iter=46400/80000, loss=0.1612, lr=0.004586, batch_cost=0.7882, reader_cost=0.0010 | ETA 07:21:23 2020-11-03 04:15:48 [INFO] [TRAIN] epoch=125, iter=46500/80000, loss=0.1661, lr=0.004574, batch_cost=0.7841, reader_cost=0.0004 | ETA 07:17:47 2020-11-03 04:17:08 [INFO] [TRAIN] epoch=126, iter=46600/80000, loss=0.1898, lr=0.004562, batch_cost=0.8009, reader_cost=0.0081 | ETA 07:25:51 2020-11-03 04:18:27 [INFO] [TRAIN] epoch=126, iter=46700/80000, loss=0.1557, lr=0.004549, batch_cost=0.7890, reader_cost=0.0006 | ETA 07:17:53 2020-11-03 04:19:46 [INFO] [TRAIN] epoch=126, iter=46800/80000, loss=0.1815, lr=0.004537, batch_cost=0.7875, reader_cost=0.0006 | ETA 07:15:43 2020-11-03 04:21:06 [INFO] [TRAIN] epoch=127, iter=46900/80000, loss=0.1696, lr=0.004525, batch_cost=0.8025, reader_cost=0.0089 | ETA 07:22:42 2020-11-03 04:22:25 [INFO] [TRAIN] epoch=127, iter=47000/80000, loss=0.1832, lr=0.004513, batch_cost=0.7918, reader_cost=0.0009 | ETA 07:15:28 2020-11-03 04:23:44 [INFO] [TRAIN] epoch=127, iter=47100/80000, loss=0.1756, lr=0.004500, batch_cost=0.7924, reader_cost=0.0009 | ETA 07:14:31 2020-11-03 04:25:03 [INFO] [TRAIN] epoch=127, iter=47200/80000, loss=0.1763, lr=0.004488, batch_cost=0.7883, reader_cost=0.0006 | ETA 07:10:56 2020-11-03 04:26:24 [INFO] [TRAIN] epoch=128, iter=47300/80000, loss=0.1991, lr=0.004476, batch_cost=0.8029, reader_cost=0.0090 | ETA 07:17:33 2020-11-03 04:27:42 [INFO] [TRAIN] epoch=128, iter=47400/80000, loss=0.1636, lr=0.004463, batch_cost=0.7832, reader_cost=0.0003 | ETA 07:05:31 2020-11-03 04:28:58 [INFO] [TRAIN] epoch=128, iter=47500/80000, loss=0.1515, lr=0.004451, batch_cost=0.7657, reader_cost=0.0002 | ETA 06:54:45 2020-11-03 04:30:16 [INFO] [TRAIN] epoch=128, iter=47600/80000, loss=0.1732, lr=0.004439, batch_cost=0.7713, reader_cost=0.0002 | ETA 06:56:29 2020-11-03 04:31:36 [INFO] [TRAIN] epoch=129, iter=47700/80000, loss=0.2006, lr=0.004427, batch_cost=0.8005, reader_cost=0.0102 | ETA 07:10:56 2020-11-03 04:32:54 [INFO] [TRAIN] epoch=129, iter=47800/80000, loss=0.1698, lr=0.004414, batch_cost=0.7872, reader_cost=0.0010 | ETA 07:02:26 2020-11-03 04:34:13 [INFO] [TRAIN] epoch=129, iter=47900/80000, loss=0.1670, lr=0.004402, batch_cost=0.7849, reader_cost=0.0008 | ETA 06:59:56 2020-11-03 04:35:33 [INFO] [TRAIN] epoch=130, iter=48000/80000, loss=0.1745, lr=0.004390, batch_cost=0.8011, reader_cost=0.0087 | ETA 07:07:15 2020-11-03 04:35:42 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-03 04:42:06 [INFO] [EVAL] #Images=500 mIoU=0.7718 Acc=0.9600 Kappa=0.9480 2020-11-03 04:42:06 [INFO] [EVAL] Category IoU: [0.9813 0.8511 0.9249 0.5395 0.5793 0.6506 0.7172 0.7973 0.9206 0.564 0.9414 0.8206 0.6182 0.9487 0.7542 0.839 0.7733 0.6629 0.7796] 2020-11-03 04:42:06 [INFO] [EVAL] Category Acc: [0.989 0.9346 0.9546 0.8217 0.8383 0.8151 0.8182 0.9038 0.9439 0.9199 0.9741 0.8914 0.7572 0.9676 0.9609 0.9681 0.8961 0.7751 0.8753] 2020-11-03 04:42:11 [INFO] [EVAL] The model with the best validation mIoU (0.7718) was saved at iter 48000. 2020-11-03 04:43:30 [INFO] [TRAIN] epoch=130, iter=48100/80000, loss=0.1595, lr=0.004377, batch_cost=0.7891, reader_cost=0.0008 | ETA 06:59:31 2020-11-03 04:44:49 [INFO] [TRAIN] epoch=130, iter=48200/80000, loss=0.1619, lr=0.004365, batch_cost=0.7897, reader_cost=0.0012 | ETA 06:58:30 2020-11-03 04:46:08 [INFO] [TRAIN] epoch=130, iter=48300/80000, loss=0.1694, lr=0.004353, batch_cost=0.7876, reader_cost=0.0004 | ETA 06:56:06 2020-11-03 04:47:27 [INFO] [TRAIN] epoch=131, iter=48400/80000, loss=0.1711, lr=0.004340, batch_cost=0.7974, reader_cost=0.0084 | ETA 06:59:57 2020-11-03 04:48:46 [INFO] [TRAIN] epoch=131, iter=48500/80000, loss=0.1482, lr=0.004328, batch_cost=0.7909, reader_cost=0.0008 | ETA 06:55:14 2020-11-03 04:50:05 [INFO] [TRAIN] epoch=131, iter=48600/80000, loss=0.1467, lr=0.004316, batch_cost=0.7867, reader_cost=0.0008 | ETA 06:51:42 2020-11-03 04:51:24 [INFO] [TRAIN] epoch=131, iter=48700/80000, loss=0.1527, lr=0.004303, batch_cost=0.7883, reader_cost=0.0005 | ETA 06:51:13 2020-11-03 04:52:44 [INFO] [TRAIN] epoch=132, iter=48800/80000, loss=0.1847, lr=0.004291, batch_cost=0.8013, reader_cost=0.0092 | ETA 06:56:39 2020-11-03 04:54:03 [INFO] [TRAIN] epoch=132, iter=48900/80000, loss=0.1758, lr=0.004279, batch_cost=0.7877, reader_cost=0.0007 | ETA 06:48:17 2020-11-03 04:55:21 [INFO] [TRAIN] epoch=132, iter=49000/80000, loss=0.1367, lr=0.004266, batch_cost=0.7862, reader_cost=0.0008 | ETA 06:46:12 2020-11-03 04:56:41 [INFO] [TRAIN] epoch=132, iter=49100/80000, loss=0.1621, lr=0.004254, batch_cost=0.7939, reader_cost=0.0008 | ETA 06:48:50 2020-11-03 04:58:01 [INFO] [TRAIN] epoch=133, iter=49200/80000, loss=0.1657, lr=0.004241, batch_cost=0.8000, reader_cost=0.0090 | ETA 06:50:40 2020-11-03 04:59:20 [INFO] [TRAIN] epoch=133, iter=49300/80000, loss=0.1538, lr=0.004229, batch_cost=0.7897, reader_cost=0.0009 | ETA 06:44:04 2020-11-03 05:00:39 [INFO] [TRAIN] epoch=133, iter=49400/80000, loss=0.1469, lr=0.004217, batch_cost=0.7922, reader_cost=0.0004 | ETA 06:44:02 2020-11-03 05:01:59 [INFO] [TRAIN] epoch=134, iter=49500/80000, loss=0.1692, lr=0.004204, batch_cost=0.8034, reader_cost=0.0091 | ETA 06:48:22 2020-11-03 05:03:18 [INFO] [TRAIN] epoch=134, iter=49600/80000, loss=0.1658, lr=0.004192, batch_cost=0.7898, reader_cost=0.0003 | ETA 06:40:10 2020-11-03 05:04:37 [INFO] [TRAIN] epoch=134, iter=49700/80000, loss=0.1500, lr=0.004180, batch_cost=0.7858, reader_cost=0.0002 | ETA 06:36:49 2020-11-03 05:05:56 [INFO] [TRAIN] epoch=134, iter=49800/80000, loss=0.1734, lr=0.004167, batch_cost=0.7934, reader_cost=0.0012 | ETA 06:39:20 2020-11-03 05:07:17 [INFO] [TRAIN] epoch=135, iter=49900/80000, loss=0.1713, lr=0.004155, batch_cost=0.8032, reader_cost=0.0094 | ETA 06:42:56 2020-11-03 05:08:35 [INFO] [TRAIN] epoch=135, iter=50000/80000, loss=0.1637, lr=0.004142, batch_cost=0.7874, reader_cost=0.0005 | ETA 06:33:42 2020-11-03 05:09:54 [INFO] [TRAIN] epoch=135, iter=50100/80000, loss=0.1624, lr=0.004130, batch_cost=0.7861, reader_cost=0.0006 | ETA 06:31:43 2020-11-03 05:11:13 [INFO] [TRAIN] epoch=135, iter=50200/80000, loss=0.1978, lr=0.004118, batch_cost=0.7909, reader_cost=0.0006 | ETA 06:32:47 2020-11-03 05:12:32 [INFO] [TRAIN] epoch=136, iter=50300/80000, loss=0.1530, lr=0.004105, batch_cost=0.7939, reader_cost=0.0085 | ETA 06:33:00 2020-11-03 05:13:51 [INFO] [TRAIN] epoch=136, iter=50400/80000, loss=0.1633, lr=0.004093, batch_cost=0.7864, reader_cost=0.0005 | ETA 06:27:56 2020-11-03 05:15:11 [INFO] [TRAIN] epoch=136, iter=50500/80000, loss=0.1524, lr=0.004080, batch_cost=0.7940, reader_cost=0.0007 | ETA 06:30:23 2020-11-03 05:16:31 [INFO] [TRAIN] epoch=137, iter=50600/80000, loss=0.1545, lr=0.004068, batch_cost=0.8038, reader_cost=0.0097 | ETA 06:33:50 2020-11-03 05:17:50 [INFO] [TRAIN] epoch=137, iter=50700/80000, loss=0.1630, lr=0.004056, batch_cost=0.7947, reader_cost=0.0012 | ETA 06:28:04 2020-11-03 05:19:10 [INFO] [TRAIN] epoch=137, iter=50800/80000, loss=0.1792, lr=0.004043, batch_cost=0.7942, reader_cost=0.0012 | ETA 06:26:30 2020-11-03 05:20:29 [INFO] [TRAIN] epoch=137, iter=50900/80000, loss=0.1629, lr=0.004031, batch_cost=0.7942, reader_cost=0.0018 | ETA 06:25:10 2020-11-03 05:21:49 [INFO] [TRAIN] epoch=138, iter=51000/80000, loss=0.1613, lr=0.004018, batch_cost=0.7996, reader_cost=0.0087 | ETA 06:26:29 2020-11-03 05:23:08 [INFO] [TRAIN] epoch=138, iter=51100/80000, loss=0.1569, lr=0.004006, batch_cost=0.7858, reader_cost=0.0011 | ETA 06:18:30 2020-11-03 05:24:25 [INFO] [TRAIN] epoch=138, iter=51200/80000, loss=0.1686, lr=0.003993, batch_cost=0.7758, reader_cost=0.0007 | ETA 06:12:22 2020-11-03 05:25:44 [INFO] [TRAIN] epoch=138, iter=51300/80000, loss=0.1669, lr=0.003981, batch_cost=0.7818, reader_cost=0.0007 | ETA 06:13:56 2020-11-03 05:27:03 [INFO] [TRAIN] epoch=139, iter=51400/80000, loss=0.1632, lr=0.003968, batch_cost=0.7981, reader_cost=0.0087 | ETA 06:20:25 2020-11-03 05:28:22 [INFO] [TRAIN] epoch=139, iter=51500/80000, loss=0.1573, lr=0.003956, batch_cost=0.7911, reader_cost=0.0009 | ETA 06:15:46 2020-11-03 05:29:42 [INFO] [TRAIN] epoch=139, iter=51600/80000, loss=0.1633, lr=0.003944, batch_cost=0.7909, reader_cost=0.0012 | ETA 06:14:21 2020-11-03 05:31:01 [INFO] [TRAIN] epoch=139, iter=51700/80000, loss=0.1809, lr=0.003931, batch_cost=0.7935, reader_cost=0.0012 | ETA 06:14:16 2020-11-03 05:32:21 [INFO] [TRAIN] epoch=140, iter=51800/80000, loss=0.1995, lr=0.003919, batch_cost=0.8023, reader_cost=0.0093 | ETA 06:17:05 2020-11-03 05:33:41 [INFO] [TRAIN] epoch=140, iter=51900/80000, loss=0.1773, lr=0.003906, batch_cost=0.7981, reader_cost=0.0012 | ETA 06:13:45 2020-11-03 05:35:01 [INFO] [TRAIN] epoch=140, iter=52000/80000, loss=0.1591, lr=0.003894, batch_cost=0.7984, reader_cost=0.0013 | ETA 06:12:35 2020-11-03 05:36:21 [INFO] [TRAIN] epoch=141, iter=52100/80000, loss=0.1642, lr=0.003881, batch_cost=0.8045, reader_cost=0.0081 | ETA 06:14:06 2020-11-03 05:37:41 [INFO] [TRAIN] epoch=141, iter=52200/80000, loss=0.1753, lr=0.003869, batch_cost=0.7933, reader_cost=0.0009 | ETA 06:07:34 2020-11-03 05:39:00 [INFO] [TRAIN] epoch=141, iter=52300/80000, loss=0.1506, lr=0.003856, batch_cost=0.7896, reader_cost=0.0003 | ETA 06:04:30 2020-11-03 05:40:18 [INFO] [TRAIN] epoch=141, iter=52400/80000, loss=0.1699, lr=0.003844, batch_cost=0.7891, reader_cost=0.0008 | ETA 06:02:58 2020-11-03 05:41:38 [INFO] [TRAIN] epoch=142, iter=52500/80000, loss=0.1618, lr=0.003831, batch_cost=0.7981, reader_cost=0.0082 | ETA 06:05:48 2020-11-03 05:42:57 [INFO] [TRAIN] epoch=142, iter=52600/80000, loss=0.1508, lr=0.003819, batch_cost=0.7911, reader_cost=0.0013 | ETA 06:01:15 2020-11-03 05:44:16 [INFO] [TRAIN] epoch=142, iter=52700/80000, loss=0.1592, lr=0.003806, batch_cost=0.7878, reader_cost=0.0005 | ETA 05:58:26 2020-11-03 05:45:36 [INFO] [TRAIN] epoch=142, iter=52800/80000, loss=0.1584, lr=0.003794, batch_cost=0.7947, reader_cost=0.0007 | ETA 06:00:16 2020-11-03 05:46:56 [INFO] [TRAIN] epoch=143, iter=52900/80000, loss=0.1624, lr=0.003781, batch_cost=0.7993, reader_cost=0.0080 | ETA 06:01:01 2020-11-03 05:48:15 [INFO] [TRAIN] epoch=143, iter=53000/80000, loss=0.1481, lr=0.003769, batch_cost=0.7939, reader_cost=0.0009 | ETA 05:57:14 2020-11-03 05:49:34 [INFO] [TRAIN] epoch=143, iter=53100/80000, loss=0.1385, lr=0.003756, batch_cost=0.7936, reader_cost=0.0010 | ETA 05:55:47 2020-11-03 05:50:54 [INFO] [TRAIN] epoch=144, iter=53200/80000, loss=0.1771, lr=0.003744, batch_cost=0.7990, reader_cost=0.0082 | ETA 05:56:53 2020-11-03 05:52:13 [INFO] [TRAIN] epoch=144, iter=53300/80000, loss=0.1471, lr=0.003731, batch_cost=0.7930, reader_cost=0.0011 | ETA 05:52:52 2020-11-03 05:53:32 [INFO] [TRAIN] epoch=144, iter=53400/80000, loss=0.1481, lr=0.003718, batch_cost=0.7885, reader_cost=0.0009 | ETA 05:49:33 2020-11-03 05:54:52 [INFO] [TRAIN] epoch=144, iter=53500/80000, loss=0.1806, lr=0.003706, batch_cost=0.7924, reader_cost=0.0011 | ETA 05:49:59 2020-11-03 05:56:12 [INFO] [TRAIN] epoch=145, iter=53600/80000, loss=0.1582, lr=0.003693, batch_cost=0.8008, reader_cost=0.0086 | ETA 05:52:22 2020-11-03 05:57:30 [INFO] [TRAIN] epoch=145, iter=53700/80000, loss=0.1620, lr=0.003681, batch_cost=0.7846, reader_cost=0.0002 | ETA 05:43:54 2020-11-03 05:58:49 [INFO] [TRAIN] epoch=145, iter=53800/80000, loss=0.1676, lr=0.003668, batch_cost=0.7877, reader_cost=0.0009 | ETA 05:43:57 2020-11-03 06:00:08 [INFO] [TRAIN] epoch=145, iter=53900/80000, loss=0.1637, lr=0.003656, batch_cost=0.7919, reader_cost=0.0009 | ETA 05:44:28 2020-11-03 06:01:28 [INFO] [TRAIN] epoch=146, iter=54000/80000, loss=0.1732, lr=0.003643, batch_cost=0.7977, reader_cost=0.0072 | ETA 05:45:39 2020-11-03 06:02:47 [INFO] [TRAIN] epoch=146, iter=54100/80000, loss=0.1528, lr=0.003631, batch_cost=0.7923, reader_cost=0.0005 | ETA 05:41:59 2020-11-03 06:04:06 [INFO] [TRAIN] epoch=146, iter=54200/80000, loss=0.1641, lr=0.003618, batch_cost=0.7884, reader_cost=0.0011 | ETA 05:39:00 2020-11-03 06:05:25 [INFO] [TRAIN] epoch=146, iter=54300/80000, loss=0.1500, lr=0.003605, batch_cost=0.7926, reader_cost=0.0009 | ETA 05:39:29 2020-11-03 06:06:46 [INFO] [TRAIN] epoch=147, iter=54400/80000, loss=0.1753, lr=0.003593, batch_cost=0.8075, reader_cost=0.0085 | ETA 05:44:32 2020-11-03 06:08:06 [INFO] [TRAIN] epoch=147, iter=54500/80000, loss=0.1493, lr=0.003580, batch_cost=0.7958, reader_cost=0.0008 | ETA 05:38:11 2020-11-03 06:09:25 [INFO] [TRAIN] epoch=147, iter=54600/80000, loss=0.1669, lr=0.003568, batch_cost=0.7976, reader_cost=0.0009 | ETA 05:37:39 2020-11-03 06:10:45 [INFO] [TRAIN] epoch=148, iter=54700/80000, loss=0.1539, lr=0.003555, batch_cost=0.7962, reader_cost=0.0087 | ETA 05:35:45 2020-11-03 06:12:04 [INFO] [TRAIN] epoch=148, iter=54800/80000, loss=0.1710, lr=0.003542, batch_cost=0.7924, reader_cost=0.0005 | ETA 05:32:48 2020-11-03 06:13:23 [INFO] [TRAIN] epoch=148, iter=54900/80000, loss=0.1401, lr=0.003530, batch_cost=0.7903, reader_cost=0.0012 | ETA 05:30:37 2020-11-03 06:14:42 [INFO] [TRAIN] epoch=148, iter=55000/80000, loss=0.1461, lr=0.003517, batch_cost=0.7901, reader_cost=0.0008 | ETA 05:29:13 2020-11-03 06:16:02 [INFO] [TRAIN] epoch=149, iter=55100/80000, loss=0.1397, lr=0.003504, batch_cost=0.7997, reader_cost=0.0095 | ETA 05:31:53 2020-11-03 06:17:20 [INFO] [TRAIN] epoch=149, iter=55200/80000, loss=0.1611, lr=0.003492, batch_cost=0.7829, reader_cost=0.0006 | ETA 05:23:35 2020-11-03 06:18:39 [INFO] [TRAIN] epoch=149, iter=55300/80000, loss=0.1607, lr=0.003479, batch_cost=0.7842, reader_cost=0.0008 | ETA 05:22:49 2020-11-03 06:19:56 [INFO] [TRAIN] epoch=149, iter=55400/80000, loss=0.1685, lr=0.003467, batch_cost=0.7740, reader_cost=0.0002 | ETA 05:17:19 2020-11-03 06:21:15 [INFO] [TRAIN] epoch=150, iter=55500/80000, loss=0.1508, lr=0.003454, batch_cost=0.7882, reader_cost=0.0081 | ETA 05:21:52 2020-11-03 06:22:35 [INFO] [TRAIN] epoch=150, iter=55600/80000, loss=0.1376, lr=0.003441, batch_cost=0.7975, reader_cost=0.0013 | ETA 05:24:17 2020-11-03 06:23:54 [INFO] [TRAIN] epoch=150, iter=55700/80000, loss=0.1494, lr=0.003429, batch_cost=0.7965, reader_cost=0.0017 | ETA 05:22:35 2020-11-03 06:25:14 [INFO] [TRAIN] epoch=150, iter=55800/80000, loss=0.1470, lr=0.003416, batch_cost=0.7901, reader_cost=0.0011 | ETA 05:18:41 2020-11-03 06:26:34 [INFO] [TRAIN] epoch=151, iter=55900/80000, loss=0.1597, lr=0.003403, batch_cost=0.8008, reader_cost=0.0102 | ETA 05:21:38 2020-11-03 06:27:52 [INFO] [TRAIN] epoch=151, iter=56000/80000, loss=0.1470, lr=0.003391, batch_cost=0.7888, reader_cost=0.0007 | ETA 05:15:30 2020-11-03 06:28:01 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-03 06:34:27 [INFO] [EVAL] #Images=500 mIoU=0.7880 Acc=0.9601 Kappa=0.9482 2020-11-03 06:34:27 [INFO] [EVAL] Category IoU: [0.9768 0.8246 0.9268 0.56 0.6067 0.6592 0.7303 0.798 0.9264 0.6076 0.9469 0.8282 0.6484 0.9544 0.7693 0.9149 0.8336 0.6715 0.7892] 2020-11-03 06:34:27 [INFO] [EVAL] Category Acc: [0.9889 0.8991 0.9588 0.831 0.8671 0.7911 0.8518 0.9253 0.9519 0.8988 0.9715 0.8825 0.7723 0.9728 0.8607 0.9742 0.9762 0.8404 0.879 ] 2020-11-03 06:34:31 [INFO] [EVAL] The model with the best validation mIoU (0.7880) was saved at iter 56000. 2020-11-03 06:35:50 [INFO] [TRAIN] epoch=151, iter=56100/80000, loss=0.1664, lr=0.003378, batch_cost=0.7842, reader_cost=0.0010 | ETA 05:12:22 2020-11-03 06:37:10 [INFO] [TRAIN] epoch=152, iter=56200/80000, loss=0.1630, lr=0.003365, batch_cost=0.8012, reader_cost=0.0088 | ETA 05:17:49 2020-11-03 06:38:29 [INFO] [TRAIN] epoch=152, iter=56300/80000, loss=0.1595, lr=0.003353, batch_cost=0.7922, reader_cost=0.0010 | ETA 05:12:54 2020-11-03 06:39:48 [INFO] [TRAIN] epoch=152, iter=56400/80000, loss=0.1406, lr=0.003340, batch_cost=0.7917, reader_cost=0.0010 | ETA 05:11:24 2020-11-03 06:41:07 [INFO] [TRAIN] epoch=152, iter=56500/80000, loss=0.1618, lr=0.003327, batch_cost=0.7912, reader_cost=0.0014 | ETA 05:09:52 2020-11-03 06:42:27 [INFO] [TRAIN] epoch=153, iter=56600/80000, loss=0.1616, lr=0.003314, batch_cost=0.7994, reader_cost=0.0092 | ETA 05:11:45 2020-11-03 06:43:47 [INFO] [TRAIN] epoch=153, iter=56700/80000, loss=0.1372, lr=0.003302, batch_cost=0.7934, reader_cost=0.0009 | ETA 05:08:07 2020-11-03 06:45:05 [INFO] [TRAIN] epoch=153, iter=56800/80000, loss=0.1536, lr=0.003289, batch_cost=0.7874, reader_cost=0.0008 | ETA 05:04:27 2020-11-03 06:46:24 [INFO] [TRAIN] epoch=153, iter=56900/80000, loss=0.1556, lr=0.003276, batch_cost=0.7863, reader_cost=0.0010 | ETA 05:02:44 2020-11-03 06:47:44 [INFO] [TRAIN] epoch=154, iter=57000/80000, loss=0.1620, lr=0.003264, batch_cost=0.8036, reader_cost=0.0082 | ETA 05:08:02 2020-11-03 06:49:03 [INFO] [TRAIN] epoch=154, iter=57100/80000, loss=0.1437, lr=0.003251, batch_cost=0.7862, reader_cost=0.0005 | ETA 05:00:03 2020-11-03 06:50:22 [INFO] [TRAIN] epoch=154, iter=57200/80000, loss=0.1463, lr=0.003238, batch_cost=0.7905, reader_cost=0.0006 | ETA 05:00:22 2020-11-03 06:51:42 [INFO] [TRAIN] epoch=155, iter=57300/80000, loss=0.1473, lr=0.003225, batch_cost=0.8000, reader_cost=0.0076 | ETA 05:02:40 2020-11-03 06:53:01 [INFO] [TRAIN] epoch=155, iter=57400/80000, loss=0.1427, lr=0.003213, batch_cost=0.7885, reader_cost=0.0006 | ETA 04:57:01 2020-11-03 06:54:20 [INFO] [TRAIN] epoch=155, iter=57500/80000, loss=0.1390, lr=0.003200, batch_cost=0.7923, reader_cost=0.0006 | ETA 04:57:06 2020-11-03 06:55:39 [INFO] [TRAIN] epoch=155, iter=57600/80000, loss=0.1717, lr=0.003187, batch_cost=0.7907, reader_cost=0.0003 | ETA 04:55:11 2020-11-03 06:56:59 [INFO] [TRAIN] epoch=156, iter=57700/80000, loss=0.1744, lr=0.003174, batch_cost=0.7984, reader_cost=0.0083 | ETA 04:56:44 2020-11-03 06:58:18 [INFO] [TRAIN] epoch=156, iter=57800/80000, loss=0.1555, lr=0.003162, batch_cost=0.7928, reader_cost=0.0009 | ETA 04:53:20 2020-11-03 06:59:37 [INFO] [TRAIN] epoch=156, iter=57900/80000, loss=0.1569, lr=0.003149, batch_cost=0.7886, reader_cost=0.0013 | ETA 04:50:28 2020-11-03 07:00:56 [INFO] [TRAIN] epoch=156, iter=58000/80000, loss=0.1573, lr=0.003136, batch_cost=0.7871, reader_cost=0.0003 | ETA 04:48:35 2020-11-03 07:02:16 [INFO] [TRAIN] epoch=157, iter=58100/80000, loss=0.1530, lr=0.003123, batch_cost=0.8051, reader_cost=0.0090 | ETA 04:53:51 2020-11-03 07:03:35 [INFO] [TRAIN] epoch=157, iter=58200/80000, loss=0.1623, lr=0.003110, batch_cost=0.7874, reader_cost=0.0006 | ETA 04:46:05 2020-11-03 07:04:54 [INFO] [TRAIN] epoch=157, iter=58300/80000, loss=0.1476, lr=0.003098, batch_cost=0.7889, reader_cost=0.0003 | ETA 04:45:20 2020-11-03 07:06:13 [INFO] [TRAIN] epoch=157, iter=58400/80000, loss=0.1359, lr=0.003085, batch_cost=0.7896, reader_cost=0.0005 | ETA 04:44:16 2020-11-03 07:07:33 [INFO] [TRAIN] epoch=158, iter=58500/80000, loss=0.1565, lr=0.003072, batch_cost=0.8019, reader_cost=0.0083 | ETA 04:47:20 2020-11-03 07:08:52 [INFO] [TRAIN] epoch=158, iter=58600/80000, loss=0.1413, lr=0.003059, batch_cost=0.7872, reader_cost=0.0004 | ETA 04:40:46 2020-11-03 07:10:11 [INFO] [TRAIN] epoch=158, iter=58700/80000, loss=0.1439, lr=0.003046, batch_cost=0.7856, reader_cost=0.0005 | ETA 04:38:54 2020-11-03 07:11:31 [INFO] [TRAIN] epoch=159, iter=58800/80000, loss=0.1576, lr=0.003033, batch_cost=0.7998, reader_cost=0.0086 | ETA 04:42:35 2020-11-03 07:12:50 [INFO] [TRAIN] epoch=159, iter=58900/80000, loss=0.1521, lr=0.003021, batch_cost=0.7905, reader_cost=0.0005 | ETA 04:37:59 2020-11-03 07:14:07 [INFO] [TRAIN] epoch=159, iter=59000/80000, loss=0.1478, lr=0.003008, batch_cost=0.7764, reader_cost=0.0003 | ETA 04:31:44 2020-11-03 07:15:24 [INFO] [TRAIN] epoch=159, iter=59100/80000, loss=0.1508, lr=0.002995, batch_cost=0.7718, reader_cost=0.0002 | ETA 04:28:50 2020-11-03 07:16:44 [INFO] [TRAIN] epoch=160, iter=59200/80000, loss=0.1536, lr=0.002982, batch_cost=0.7991, reader_cost=0.0076 | ETA 04:37:01 2020-11-03 07:18:03 [INFO] [TRAIN] epoch=160, iter=59300/80000, loss=0.1393, lr=0.002969, batch_cost=0.7897, reader_cost=0.0008 | ETA 04:32:27 2020-11-03 07:19:22 [INFO] [TRAIN] epoch=160, iter=59400/80000, loss=0.1410, lr=0.002956, batch_cost=0.7919, reader_cost=0.0004 | ETA 04:31:52 2020-11-03 07:20:41 [INFO] [TRAIN] epoch=160, iter=59500/80000, loss=0.1544, lr=0.002943, batch_cost=0.7853, reader_cost=0.0004 | ETA 04:28:17 2020-11-03 07:22:01 [INFO] [TRAIN] epoch=161, iter=59600/80000, loss=0.1580, lr=0.002931, batch_cost=0.7979, reader_cost=0.0089 | ETA 04:31:16 2020-11-03 07:23:19 [INFO] [TRAIN] epoch=161, iter=59700/80000, loss=0.1361, lr=0.002918, batch_cost=0.7860, reader_cost=0.0010 | ETA 04:25:56 2020-11-03 07:24:38 [INFO] [TRAIN] epoch=161, iter=59800/80000, loss=0.1411, lr=0.002905, batch_cost=0.7885, reader_cost=0.0005 | ETA 04:25:27 2020-11-03 07:25:59 [INFO] [TRAIN] epoch=162, iter=59900/80000, loss=0.1423, lr=0.002892, batch_cost=0.8024, reader_cost=0.0080 | ETA 04:28:47 2020-11-03 07:27:17 [INFO] [TRAIN] epoch=162, iter=60000/80000, loss=0.1420, lr=0.002879, batch_cost=0.7870, reader_cost=0.0005 | ETA 04:22:20 2020-11-03 07:28:36 [INFO] [TRAIN] epoch=162, iter=60100/80000, loss=0.1358, lr=0.002866, batch_cost=0.7895, reader_cost=0.0014 | ETA 04:21:51 2020-11-03 07:29:56 [INFO] [TRAIN] epoch=162, iter=60200/80000, loss=0.1226, lr=0.002853, batch_cost=0.7970, reader_cost=0.0012 | ETA 04:23:00 2020-11-03 07:31:16 [INFO] [TRAIN] epoch=163, iter=60300/80000, loss=0.1544, lr=0.002840, batch_cost=0.8011, reader_cost=0.0082 | ETA 04:23:01 2020-11-03 07:32:35 [INFO] [TRAIN] epoch=163, iter=60400/80000, loss=0.1421, lr=0.002827, batch_cost=0.7883, reader_cost=0.0010 | ETA 04:17:31 2020-11-03 07:33:54 [INFO] [TRAIN] epoch=163, iter=60500/80000, loss=0.1470, lr=0.002814, batch_cost=0.7904, reader_cost=0.0004 | ETA 04:16:52 2020-11-03 07:35:13 [INFO] [TRAIN] epoch=163, iter=60600/80000, loss=0.1385, lr=0.002801, batch_cost=0.7885, reader_cost=0.0003 | ETA 04:14:56 2020-11-03 07:36:33 [INFO] [TRAIN] epoch=164, iter=60700/80000, loss=0.1475, lr=0.002788, batch_cost=0.8039, reader_cost=0.0069 | ETA 04:18:34 2020-11-03 07:37:52 [INFO] [TRAIN] epoch=164, iter=60800/80000, loss=0.1621, lr=0.002776, batch_cost=0.7907, reader_cost=0.0004 | ETA 04:13:00 2020-11-03 07:39:11 [INFO] [TRAIN] epoch=164, iter=60900/80000, loss=0.1305, lr=0.002763, batch_cost=0.7889, reader_cost=0.0010 | ETA 04:11:07 2020-11-03 07:40:30 [INFO] [TRAIN] epoch=164, iter=61000/80000, loss=0.1483, lr=0.002750, batch_cost=0.7916, reader_cost=0.0005 | ETA 04:10:39 2020-11-03 07:41:50 [INFO] [TRAIN] epoch=165, iter=61100/80000, loss=0.1673, lr=0.002737, batch_cost=0.7979, reader_cost=0.0073 | ETA 04:11:20 2020-11-03 07:43:09 [INFO] [TRAIN] epoch=165, iter=61200/80000, loss=0.1285, lr=0.002724, batch_cost=0.7895, reader_cost=0.0005 | ETA 04:07:21 2020-11-03 07:44:28 [INFO] [TRAIN] epoch=165, iter=61300/80000, loss=0.1540, lr=0.002711, batch_cost=0.7905, reader_cost=0.0003 | ETA 04:06:22 2020-11-03 07:45:48 [INFO] [TRAIN] epoch=166, iter=61400/80000, loss=0.1609, lr=0.002698, batch_cost=0.7985, reader_cost=0.0084 | ETA 04:07:31 2020-11-03 07:47:07 [INFO] [TRAIN] epoch=166, iter=61500/80000, loss=0.1520, lr=0.002685, batch_cost=0.7899, reader_cost=0.0004 | ETA 04:03:33 2020-11-03 07:48:26 [INFO] [TRAIN] epoch=166, iter=61600/80000, loss=0.1441, lr=0.002672, batch_cost=0.7929, reader_cost=0.0007 | ETA 04:03:09 2020-11-03 07:49:45 [INFO] [TRAIN] epoch=166, iter=61700/80000, loss=0.1370, lr=0.002659, batch_cost=0.7934, reader_cost=0.0010 | ETA 04:01:59 2020-11-03 07:51:05 [INFO] [TRAIN] epoch=167, iter=61800/80000, loss=0.1567, lr=0.002646, batch_cost=0.7953, reader_cost=0.0081 | ETA 04:01:14 2020-11-03 07:52:24 [INFO] [TRAIN] epoch=167, iter=61900/80000, loss=0.1343, lr=0.002633, batch_cost=0.7902, reader_cost=0.0010 | ETA 03:58:22 2020-11-03 07:53:43 [INFO] [TRAIN] epoch=167, iter=62000/80000, loss=0.1381, lr=0.002619, batch_cost=0.7917, reader_cost=0.0008 | ETA 03:57:30 2020-11-03 07:55:02 [INFO] [TRAIN] epoch=167, iter=62100/80000, loss=0.1630, lr=0.002606, batch_cost=0.7900, reader_cost=0.0004 | ETA 03:55:40 2020-11-03 07:56:22 [INFO] [TRAIN] epoch=168, iter=62200/80000, loss=0.1461, lr=0.002593, batch_cost=0.7998, reader_cost=0.0080 | ETA 03:57:16 2020-11-03 07:57:42 [INFO] [TRAIN] epoch=168, iter=62300/80000, loss=0.1456, lr=0.002580, batch_cost=0.7933, reader_cost=0.0010 | ETA 03:54:00 2020-11-03 07:59:00 [INFO] [TRAIN] epoch=168, iter=62400/80000, loss=0.1478, lr=0.002567, batch_cost=0.7899, reader_cost=0.0011 | ETA 03:51:43 2020-11-03 08:00:20 [INFO] [TRAIN] epoch=169, iter=62500/80000, loss=0.1662, lr=0.002554, batch_cost=0.7995, reader_cost=0.0080 | ETA 03:53:10 2020-11-03 08:01:40 [INFO] [TRAIN] epoch=169, iter=62600/80000, loss=0.1479, lr=0.002541, batch_cost=0.7921, reader_cost=0.0004 | ETA 03:49:41 2020-11-03 08:02:58 [INFO] [TRAIN] epoch=169, iter=62700/80000, loss=0.1454, lr=0.002528, batch_cost=0.7864, reader_cost=0.0004 | ETA 03:46:43 2020-11-03 08:04:17 [INFO] [TRAIN] epoch=169, iter=62800/80000, loss=0.1357, lr=0.002515, batch_cost=0.7908, reader_cost=0.0006 | ETA 03:46:41 2020-11-03 08:05:38 [INFO] [TRAIN] epoch=170, iter=62900/80000, loss=0.1338, lr=0.002502, batch_cost=0.8034, reader_cost=0.0090 | ETA 03:48:58 2020-11-03 08:06:57 [INFO] [TRAIN] epoch=170, iter=63000/80000, loss=0.1449, lr=0.002489, batch_cost=0.7924, reader_cost=0.0010 | ETA 03:44:30 2020-11-03 08:08:15 [INFO] [TRAIN] epoch=170, iter=63100/80000, loss=0.1199, lr=0.002476, batch_cost=0.7823, reader_cost=0.0005 | ETA 03:40:21 2020-11-03 08:09:32 [INFO] [TRAIN] epoch=170, iter=63200/80000, loss=0.1376, lr=0.002462, batch_cost=0.7677, reader_cost=0.0004 | ETA 03:34:57 2020-11-03 08:10:50 [INFO] [TRAIN] epoch=171, iter=63300/80000, loss=0.1562, lr=0.002449, batch_cost=0.7816, reader_cost=0.0097 | ETA 03:37:33 2020-11-03 08:12:09 [INFO] [TRAIN] epoch=171, iter=63400/80000, loss=0.1338, lr=0.002436, batch_cost=0.7897, reader_cost=0.0008 | ETA 03:38:29 2020-11-03 08:13:29 [INFO] [TRAIN] epoch=171, iter=63500/80000, loss=0.1420, lr=0.002423, batch_cost=0.7959, reader_cost=0.0011 | ETA 03:38:52 2020-11-03 08:14:47 [INFO] [TRAIN] epoch=171, iter=63600/80000, loss=0.1431, lr=0.002410, batch_cost=0.7862, reader_cost=0.0005 | ETA 03:34:54 2020-11-03 08:16:07 [INFO] [TRAIN] epoch=172, iter=63700/80000, loss=0.1403, lr=0.002397, batch_cost=0.7966, reader_cost=0.0099 | ETA 03:36:23 2020-11-03 08:17:26 [INFO] [TRAIN] epoch=172, iter=63800/80000, loss=0.1239, lr=0.002383, batch_cost=0.7913, reader_cost=0.0003 | ETA 03:33:38 2020-11-03 08:18:45 [INFO] [TRAIN] epoch=172, iter=63900/80000, loss=0.1603, lr=0.002370, batch_cost=0.7906, reader_cost=0.0005 | ETA 03:32:08 2020-11-03 08:20:05 [INFO] [TRAIN] epoch=173, iter=64000/80000, loss=0.1577, lr=0.002357, batch_cost=0.7975, reader_cost=0.0083 | ETA 03:32:40 2020-11-03 08:20:14 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-03 08:26:42 [INFO] [EVAL] #Images=500 mIoU=0.7866 Acc=0.9610 Kappa=0.9494 2020-11-03 08:26:42 [INFO] [EVAL] Category IoU: [0.9789 0.8468 0.9232 0.4915 0.615 0.6608 0.7308 0.7995 0.9276 0.6509 0.9476 0.8303 0.6322 0.9564 0.8412 0.9067 0.7543 0.6659 0.7863] 2020-11-03 08:26:42 [INFO] [EVAL] Category Acc: [0.99 0.9227 0.9509 0.8608 0.8374 0.7937 0.8627 0.9172 0.9563 0.885 0.9702 0.8972 0.78 0.9733 0.9604 0.948 0.9795 0.8584 0.8591] 2020-11-03 08:26:42 [INFO] [EVAL] The model with the best validation mIoU (0.7880) was saved at iter 56000. 2020-11-03 08:28:00 [INFO] [TRAIN] epoch=173, iter=64100/80000, loss=0.1666, lr=0.002344, batch_cost=0.7734, reader_cost=0.0002 | ETA 03:24:56 2020-11-03 08:29:18 [INFO] [TRAIN] epoch=173, iter=64200/80000, loss=0.1557, lr=0.002331, batch_cost=0.7790, reader_cost=0.0002 | ETA 03:25:08 2020-11-03 08:30:36 [INFO] [TRAIN] epoch=173, iter=64300/80000, loss=0.1426, lr=0.002317, batch_cost=0.7881, reader_cost=0.0009 | ETA 03:26:13 2020-11-03 08:31:56 [INFO] [TRAIN] epoch=174, iter=64400/80000, loss=0.1391, lr=0.002304, batch_cost=0.7924, reader_cost=0.0101 | ETA 03:26:00 2020-11-03 08:33:13 [INFO] [TRAIN] epoch=174, iter=64500/80000, loss=0.1419, lr=0.002291, batch_cost=0.7738, reader_cost=0.0003 | ETA 03:19:54 2020-11-03 08:34:31 [INFO] [TRAIN] epoch=174, iter=64600/80000, loss=0.1436, lr=0.002278, batch_cost=0.7836, reader_cost=0.0003 | ETA 03:21:07 2020-11-03 08:35:50 [INFO] [TRAIN] epoch=174, iter=64700/80000, loss=0.1563, lr=0.002264, batch_cost=0.7846, reader_cost=0.0002 | ETA 03:20:04 2020-11-03 08:37:10 [INFO] [TRAIN] epoch=175, iter=64800/80000, loss=0.1651, lr=0.002251, batch_cost=0.8004, reader_cost=0.0093 | ETA 03:22:45 2020-11-03 08:38:28 [INFO] [TRAIN] epoch=175, iter=64900/80000, loss=0.1426, lr=0.002238, batch_cost=0.7817, reader_cost=0.0004 | ETA 03:16:42 2020-11-03 08:39:47 [INFO] [TRAIN] epoch=175, iter=65000/80000, loss=0.1383, lr=0.002225, batch_cost=0.7940, reader_cost=0.0008 | ETA 03:18:30 2020-11-03 08:41:07 [INFO] [TRAIN] epoch=175, iter=65100/80000, loss=0.1280, lr=0.002211, batch_cost=0.7935, reader_cost=0.0008 | ETA 03:17:02 2020-11-03 08:42:27 [INFO] [TRAIN] epoch=176, iter=65200/80000, loss=0.1409, lr=0.002198, batch_cost=0.8022, reader_cost=0.0096 | ETA 03:17:51 2020-11-03 08:43:47 [INFO] [TRAIN] epoch=176, iter=65300/80000, loss=0.1298, lr=0.002185, batch_cost=0.7965, reader_cost=0.0008 | ETA 03:15:08 2020-11-03 08:45:06 [INFO] [TRAIN] epoch=176, iter=65400/80000, loss=0.1461, lr=0.002171, batch_cost=0.7934, reader_cost=0.0010 | ETA 03:13:03 2020-11-03 08:46:26 [INFO] [TRAIN] epoch=177, iter=65500/80000, loss=0.1423, lr=0.002158, batch_cost=0.8025, reader_cost=0.0102 | ETA 03:13:56 2020-11-03 08:47:45 [INFO] [TRAIN] epoch=177, iter=65600/80000, loss=0.1439, lr=0.002145, batch_cost=0.7905, reader_cost=0.0006 | ETA 03:09:42 2020-11-03 08:49:05 [INFO] [TRAIN] epoch=177, iter=65700/80000, loss=0.1289, lr=0.002131, batch_cost=0.7945, reader_cost=0.0006 | ETA 03:09:22 2020-11-03 08:50:24 [INFO] [TRAIN] epoch=177, iter=65800/80000, loss=0.1492, lr=0.002118, batch_cost=0.7921, reader_cost=0.0003 | ETA 03:07:27 2020-11-03 08:51:43 [INFO] [TRAIN] epoch=178, iter=65900/80000, loss=0.1323, lr=0.002105, batch_cost=0.7903, reader_cost=0.0078 | ETA 03:05:43 2020-11-03 08:53:02 [INFO] [TRAIN] epoch=178, iter=66000/80000, loss=0.1361, lr=0.002091, batch_cost=0.7881, reader_cost=0.0002 | ETA 03:03:53 2020-11-03 08:54:21 [INFO] [TRAIN] epoch=178, iter=66100/80000, loss=0.1325, lr=0.002078, batch_cost=0.7901, reader_cost=0.0008 | ETA 03:03:01 2020-11-03 08:55:39 [INFO] [TRAIN] epoch=178, iter=66200/80000, loss=0.1476, lr=0.002064, batch_cost=0.7840, reader_cost=0.0002 | ETA 03:00:19 2020-11-03 08:56:58 [INFO] [TRAIN] epoch=179, iter=66300/80000, loss=0.1515, lr=0.002051, batch_cost=0.7899, reader_cost=0.0087 | ETA 03:00:21 2020-11-03 08:58:17 [INFO] [TRAIN] epoch=179, iter=66400/80000, loss=0.1435, lr=0.002038, batch_cost=0.7865, reader_cost=0.0005 | ETA 02:58:16 2020-11-03 08:59:35 [INFO] [TRAIN] epoch=179, iter=66500/80000, loss=0.1362, lr=0.002024, batch_cost=0.7832, reader_cost=0.0007 | ETA 02:56:13 2020-11-03 09:00:56 [INFO] [TRAIN] epoch=180, iter=66600/80000, loss=0.1453, lr=0.002011, batch_cost=0.8049, reader_cost=0.0080 | ETA 02:59:45 2020-11-03 09:02:14 [INFO] [TRAIN] epoch=180, iter=66700/80000, loss=0.1591, lr=0.001997, batch_cost=0.7835, reader_cost=0.0002 | ETA 02:53:40 2020-11-03 09:03:33 [INFO] [TRAIN] epoch=180, iter=66800/80000, loss=0.1361, lr=0.001984, batch_cost=0.7880, reader_cost=0.0005 | ETA 02:53:21 2020-11-03 09:04:51 [INFO] [TRAIN] epoch=180, iter=66900/80000, loss=0.1566, lr=0.001970, batch_cost=0.7819, reader_cost=0.0003 | ETA 02:50:42 2020-11-03 09:06:11 [INFO] [TRAIN] epoch=181, iter=67000/80000, loss=0.1362, lr=0.001957, batch_cost=0.7964, reader_cost=0.0078 | ETA 02:52:33 2020-11-03 09:07:30 [INFO] [TRAIN] epoch=181, iter=67100/80000, loss=0.1514, lr=0.001944, batch_cost=0.7921, reader_cost=0.0006 | ETA 02:50:17 2020-11-03 09:08:49 [INFO] [TRAIN] epoch=181, iter=67200/80000, loss=0.1500, lr=0.001930, batch_cost=0.7882, reader_cost=0.0003 | ETA 02:48:08 2020-11-03 09:10:07 [INFO] [TRAIN] epoch=181, iter=67300/80000, loss=0.1386, lr=0.001917, batch_cost=0.7844, reader_cost=0.0008 | ETA 02:46:01 2020-11-03 09:11:26 [INFO] [TRAIN] epoch=182, iter=67400/80000, loss=0.1519, lr=0.001903, batch_cost=0.7937, reader_cost=0.0073 | ETA 02:46:40 2020-11-03 09:12:44 [INFO] [TRAIN] epoch=182, iter=67500/80000, loss=0.1387, lr=0.001889, batch_cost=0.7801, reader_cost=0.0003 | ETA 02:42:31 2020-11-03 09:14:03 [INFO] [TRAIN] epoch=182, iter=67600/80000, loss=0.1326, lr=0.001876, batch_cost=0.7840, reader_cost=0.0009 | ETA 02:42:01 2020-11-03 09:15:21 [INFO] [TRAIN] epoch=182, iter=67700/80000, loss=0.1365, lr=0.001862, batch_cost=0.7807, reader_cost=0.0003 | ETA 02:40:02 2020-11-03 09:16:41 [INFO] [TRAIN] epoch=183, iter=67800/80000, loss=0.1379, lr=0.001849, batch_cost=0.7986, reader_cost=0.0094 | ETA 02:42:22 2020-11-03 09:17:59 [INFO] [TRAIN] epoch=183, iter=67900/80000, loss=0.1408, lr=0.001835, batch_cost=0.7818, reader_cost=0.0007 | ETA 02:37:40 2020-11-03 09:19:18 [INFO] [TRAIN] epoch=183, iter=68000/80000, loss=0.1626, lr=0.001822, batch_cost=0.7859, reader_cost=0.0007 | ETA 02:37:10 2020-11-03 09:20:37 [INFO] [TRAIN] epoch=184, iter=68100/80000, loss=0.1358, lr=0.001808, batch_cost=0.7929, reader_cost=0.0086 | ETA 02:37:15 2020-11-03 09:21:56 [INFO] [TRAIN] epoch=184, iter=68200/80000, loss=0.1347, lr=0.001794, batch_cost=0.7886, reader_cost=0.0003 | ETA 02:35:05 2020-11-03 09:23:14 [INFO] [TRAIN] epoch=184, iter=68300/80000, loss=0.1301, lr=0.001781, batch_cost=0.7824, reader_cost=0.0009 | ETA 02:32:34 2020-11-03 09:24:31 [INFO] [TRAIN] epoch=184, iter=68400/80000, loss=0.1462, lr=0.001767, batch_cost=0.7676, reader_cost=0.0004 | ETA 02:28:23 2020-11-03 09:25:48 [INFO] [TRAIN] epoch=185, iter=68500/80000, loss=0.1417, lr=0.001754, batch_cost=0.7771, reader_cost=0.0081 | ETA 02:28:56 2020-11-03 09:27:05 [INFO] [TRAIN] epoch=185, iter=68600/80000, loss=0.1339, lr=0.001740, batch_cost=0.7660, reader_cost=0.0002 | ETA 02:25:32 2020-11-03 09:28:22 [INFO] [TRAIN] epoch=185, iter=68700/80000, loss=0.1346, lr=0.001726, batch_cost=0.7652, reader_cost=0.0002 | ETA 02:24:06 2020-11-03 09:29:39 [INFO] [TRAIN] epoch=185, iter=68800/80000, loss=0.1635, lr=0.001713, batch_cost=0.7791, reader_cost=0.0004 | ETA 02:25:25 2020-11-03 09:30:58 [INFO] [TRAIN] epoch=186, iter=68900/80000, loss=0.1495, lr=0.001699, batch_cost=0.7886, reader_cost=0.0085 | ETA 02:25:53 2020-11-03 09:32:16 [INFO] [TRAIN] epoch=186, iter=69000/80000, loss=0.1272, lr=0.001685, batch_cost=0.7797, reader_cost=0.0009 | ETA 02:22:56 2020-11-03 09:33:35 [INFO] [TRAIN] epoch=186, iter=69100/80000, loss=0.1301, lr=0.001672, batch_cost=0.7830, reader_cost=0.0009 | ETA 02:22:14 2020-11-03 09:34:54 [INFO] [TRAIN] epoch=187, iter=69200/80000, loss=0.1403, lr=0.001658, batch_cost=0.7947, reader_cost=0.0088 | ETA 02:23:03 2020-11-03 09:36:12 [INFO] [TRAIN] epoch=187, iter=69300/80000, loss=0.1466, lr=0.001644, batch_cost=0.7834, reader_cost=0.0005 | ETA 02:19:42 2020-11-03 09:37:30 [INFO] [TRAIN] epoch=187, iter=69400/80000, loss=0.1463, lr=0.001630, batch_cost=0.7771, reader_cost=0.0003 | ETA 02:17:17 2020-11-03 09:38:48 [INFO] [TRAIN] epoch=187, iter=69500/80000, loss=0.1349, lr=0.001617, batch_cost=0.7797, reader_cost=0.0007 | ETA 02:16:26 2020-11-03 09:40:07 [INFO] [TRAIN] epoch=188, iter=69600/80000, loss=0.1384, lr=0.001603, batch_cost=0.7874, reader_cost=0.0086 | ETA 02:16:28 2020-11-03 09:41:25 [INFO] [TRAIN] epoch=188, iter=69700/80000, loss=0.1248, lr=0.001589, batch_cost=0.7786, reader_cost=0.0004 | ETA 02:13:39 2020-11-03 09:42:43 [INFO] [TRAIN] epoch=188, iter=69800/80000, loss=0.1415, lr=0.001575, batch_cost=0.7822, reader_cost=0.0003 | ETA 02:12:58 2020-11-03 09:44:01 [INFO] [TRAIN] epoch=188, iter=69900/80000, loss=0.1521, lr=0.001561, batch_cost=0.7830, reader_cost=0.0003 | ETA 02:11:48 2020-11-03 09:45:21 [INFO] [TRAIN] epoch=189, iter=70000/80000, loss=0.1358, lr=0.001548, batch_cost=0.7971, reader_cost=0.0087 | ETA 02:12:51 2020-11-03 09:46:39 [INFO] [TRAIN] epoch=189, iter=70100/80000, loss=0.1305, lr=0.001534, batch_cost=0.7773, reader_cost=0.0005 | ETA 02:08:15 2020-11-03 09:47:58 [INFO] [TRAIN] epoch=189, iter=70200/80000, loss=0.1462, lr=0.001520, batch_cost=0.7898, reader_cost=0.0005 | ETA 02:08:59 2020-11-03 09:49:16 [INFO] [TRAIN] epoch=189, iter=70300/80000, loss=0.1348, lr=0.001506, batch_cost=0.7842, reader_cost=0.0007 | ETA 02:06:46 2020-11-03 09:50:35 [INFO] [TRAIN] epoch=190, iter=70400/80000, loss=0.1435, lr=0.001492, batch_cost=0.7924, reader_cost=0.0093 | ETA 02:06:47 2020-11-03 09:51:54 [INFO] [TRAIN] epoch=190, iter=70500/80000, loss=0.1203, lr=0.001478, batch_cost=0.7895, reader_cost=0.0004 | ETA 02:04:59 2020-11-03 09:53:13 [INFO] [TRAIN] epoch=190, iter=70600/80000, loss=0.1323, lr=0.001464, batch_cost=0.7914, reader_cost=0.0002 | ETA 02:03:59 2020-11-03 09:54:33 [INFO] [TRAIN] epoch=191, iter=70700/80000, loss=0.1389, lr=0.001450, batch_cost=0.7927, reader_cost=0.0076 | ETA 02:02:51 2020-11-03 09:55:51 [INFO] [TRAIN] epoch=191, iter=70800/80000, loss=0.1339, lr=0.001436, batch_cost=0.7796, reader_cost=0.0002 | ETA 01:59:32 2020-11-03 09:57:09 [INFO] [TRAIN] epoch=191, iter=70900/80000, loss=0.1356, lr=0.001422, batch_cost=0.7885, reader_cost=0.0006 | ETA 01:59:34 2020-11-03 09:58:28 [INFO] [TRAIN] epoch=191, iter=71000/80000, loss=0.1364, lr=0.001408, batch_cost=0.7856, reader_cost=0.0004 | ETA 01:57:50 2020-11-03 09:59:47 [INFO] [TRAIN] epoch=192, iter=71100/80000, loss=0.1373, lr=0.001394, batch_cost=0.7881, reader_cost=0.0092 | ETA 01:56:54 2020-11-03 10:01:06 [INFO] [TRAIN] epoch=192, iter=71200/80000, loss=0.1282, lr=0.001380, batch_cost=0.7869, reader_cost=0.0009 | ETA 01:55:24 2020-11-03 10:02:24 [INFO] [TRAIN] epoch=192, iter=71300/80000, loss=0.1349, lr=0.001366, batch_cost=0.7860, reader_cost=0.0005 | ETA 01:53:58 2020-11-03 10:03:43 [INFO] [TRAIN] epoch=192, iter=71400/80000, loss=0.1310, lr=0.001352, batch_cost=0.7849, reader_cost=0.0004 | ETA 01:52:29 2020-11-03 10:05:01 [INFO] [TRAIN] epoch=193, iter=71500/80000, loss=0.1417, lr=0.001338, batch_cost=0.7873, reader_cost=0.0071 | ETA 01:51:31 2020-11-03 10:06:19 [INFO] [TRAIN] epoch=193, iter=71600/80000, loss=0.1385, lr=0.001324, batch_cost=0.7805, reader_cost=0.0004 | ETA 01:49:16 2020-11-03 10:07:37 [INFO] [TRAIN] epoch=193, iter=71700/80000, loss=0.1339, lr=0.001310, batch_cost=0.7800, reader_cost=0.0007 | ETA 01:47:53 2020-11-03 10:08:57 [INFO] [TRAIN] epoch=194, iter=71800/80000, loss=0.1409, lr=0.001296, batch_cost=0.7931, reader_cost=0.0084 | ETA 01:48:23 2020-11-03 10:10:15 [INFO] [TRAIN] epoch=194, iter=71900/80000, loss=0.1524, lr=0.001282, batch_cost=0.7831, reader_cost=0.0003 | ETA 01:45:42 2020-11-03 10:11:34 [INFO] [TRAIN] epoch=194, iter=72000/80000, loss=0.1195, lr=0.001268, batch_cost=0.7862, reader_cost=0.0003 | ETA 01:44:49 2020-11-03 10:11:43 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-03 10:18:13 [INFO] [EVAL] #Images=500 mIoU=0.8048 Acc=0.9644 Kappa=0.9537 2020-11-03 10:18:13 [INFO] [EVAL] Category IoU: [0.9839 0.8628 0.9317 0.5814 0.6318 0.6666 0.7337 0.8065 0.9291 0.6462 0.9495 0.8356 0.6487 0.9578 0.8665 0.9221 0.8561 0.6895 0.7922] 2020-11-03 10:18:13 [INFO] [EVAL] Category Acc: [0.9905 0.9326 0.9636 0.8747 0.783 0.7841 0.8645 0.9126 0.9558 0.8902 0.9727 0.8993 0.7905 0.978 0.9657 0.9679 0.9593 0.8657 0.8726] 2020-11-03 10:18:18 [INFO] [EVAL] The model with the best validation mIoU (0.8048) was saved at iter 72000. 2020-11-03 10:19:37 [INFO] [TRAIN] epoch=194, iter=72100/80000, loss=0.1217, lr=0.001254, batch_cost=0.7838, reader_cost=0.0002 | ETA 01:43:12 2020-11-03 10:20:56 [INFO] [TRAIN] epoch=195, iter=72200/80000, loss=0.1363, lr=0.001239, batch_cost=0.7921, reader_cost=0.0092 | ETA 01:42:58 2020-11-03 10:22:15 [INFO] [TRAIN] epoch=195, iter=72300/80000, loss=0.1320, lr=0.001225, batch_cost=0.7897, reader_cost=0.0007 | ETA 01:41:20 2020-11-03 10:23:34 [INFO] [TRAIN] epoch=195, iter=72400/80000, loss=0.1208, lr=0.001211, batch_cost=0.7897, reader_cost=0.0003 | ETA 01:40:01 2020-11-03 10:24:53 [INFO] [TRAIN] epoch=195, iter=72500/80000, loss=0.1369, lr=0.001197, batch_cost=0.7910, reader_cost=0.0004 | ETA 01:38:52 2020-11-03 10:26:13 [INFO] [TRAIN] epoch=196, iter=72600/80000, loss=0.1421, lr=0.001183, batch_cost=0.8028, reader_cost=0.0090 | ETA 01:39:00 2020-11-03 10:27:32 [INFO] [TRAIN] epoch=196, iter=72700/80000, loss=0.1169, lr=0.001168, batch_cost=0.7832, reader_cost=0.0005 | ETA 01:35:17 2020-11-03 10:28:50 [INFO] [TRAIN] epoch=196, iter=72800/80000, loss=0.1221, lr=0.001154, batch_cost=0.7883, reader_cost=0.0005 | ETA 01:34:35 2020-11-03 10:30:09 [INFO] [TRAIN] epoch=196, iter=72900/80000, loss=0.1303, lr=0.001140, batch_cost=0.7860, reader_cost=0.0007 | ETA 01:33:00 2020-11-03 10:31:28 [INFO] [TRAIN] epoch=197, iter=73000/80000, loss=0.1488, lr=0.001125, batch_cost=0.7945, reader_cost=0.0079 | ETA 01:32:41 2020-11-03 10:32:47 [INFO] [TRAIN] epoch=197, iter=73100/80000, loss=0.1414, lr=0.001111, batch_cost=0.7806, reader_cost=0.0003 | ETA 01:29:46 2020-11-03 10:34:05 [INFO] [TRAIN] epoch=197, iter=73200/80000, loss=0.1283, lr=0.001097, batch_cost=0.7797, reader_cost=0.0003 | ETA 01:28:22 2020-11-03 10:35:24 [INFO] [TRAIN] epoch=198, iter=73300/80000, loss=0.1207, lr=0.001082, batch_cost=0.7967, reader_cost=0.0086 | ETA 01:28:57 2020-11-03 10:36:42 [INFO] [TRAIN] epoch=198, iter=73400/80000, loss=0.1373, lr=0.001068, batch_cost=0.7814, reader_cost=0.0007 | ETA 01:25:57 2020-11-03 10:38:01 [INFO] [TRAIN] epoch=198, iter=73500/80000, loss=0.1216, lr=0.001053, batch_cost=0.7861, reader_cost=0.0005 | ETA 01:25:09 2020-11-03 10:39:20 [INFO] [TRAIN] epoch=198, iter=73600/80000, loss=0.1257, lr=0.001039, batch_cost=0.7924, reader_cost=0.0010 | ETA 01:24:31 2020-11-03 10:40:40 [INFO] [TRAIN] epoch=199, iter=73700/80000, loss=0.1301, lr=0.001025, batch_cost=0.7968, reader_cost=0.0089 | ETA 01:23:39 2020-11-03 10:41:57 [INFO] [TRAIN] epoch=199, iter=73800/80000, loss=0.1497, lr=0.001010, batch_cost=0.7757, reader_cost=0.0002 | ETA 01:20:09 2020-11-03 10:43:15 [INFO] [TRAIN] epoch=199, iter=73900/80000, loss=0.1290, lr=0.000995, batch_cost=0.7739, reader_cost=0.0003 | ETA 01:18:40 2020-11-03 10:44:32 [INFO] [TRAIN] epoch=199, iter=74000/80000, loss=0.1273, lr=0.000981, batch_cost=0.7689, reader_cost=0.0002 | ETA 01:16:53 2020-11-03 10:45:49 [INFO] [TRAIN] epoch=200, iter=74100/80000, loss=0.1322, lr=0.000966, batch_cost=0.7777, reader_cost=0.0078 | ETA 01:16:28 2020-11-03 10:47:08 [INFO] [TRAIN] epoch=200, iter=74200/80000, loss=0.1381, lr=0.000952, batch_cost=0.7857, reader_cost=0.0006 | ETA 01:15:57 2020-11-03 10:48:27 [INFO] [TRAIN] epoch=200, iter=74300/80000, loss=0.1251, lr=0.000937, batch_cost=0.7874, reader_cost=0.0009 | ETA 01:14:48 2020-11-03 10:49:45 [INFO] [TRAIN] epoch=200, iter=74400/80000, loss=0.1363, lr=0.000922, batch_cost=0.7839, reader_cost=0.0007 | ETA 01:13:09 2020-11-03 10:51:05 [INFO] [TRAIN] epoch=201, iter=74500/80000, loss=0.1422, lr=0.000908, batch_cost=0.7977, reader_cost=0.0071 | ETA 01:13:07 2020-11-03 10:52:23 [INFO] [TRAIN] epoch=201, iter=74600/80000, loss=0.1196, lr=0.000893, batch_cost=0.7828, reader_cost=0.0013 | ETA 01:10:26 2020-11-03 10:53:42 [INFO] [TRAIN] epoch=201, iter=74700/80000, loss=0.1506, lr=0.000878, batch_cost=0.7898, reader_cost=0.0012 | ETA 01:09:45 2020-11-03 10:55:02 [INFO] [TRAIN] epoch=202, iter=74800/80000, loss=0.1414, lr=0.000864, batch_cost=0.7935, reader_cost=0.0094 | ETA 01:08:46 2020-11-03 10:56:20 [INFO] [TRAIN] epoch=202, iter=74900/80000, loss=0.1279, lr=0.000849, batch_cost=0.7880, reader_cost=0.0012 | ETA 01:06:58 2020-11-03 10:57:39 [INFO] [TRAIN] epoch=202, iter=75000/80000, loss=0.1336, lr=0.000834, batch_cost=0.7911, reader_cost=0.0016 | ETA 01:05:55 2020-11-03 10:58:58 [INFO] [TRAIN] epoch=202, iter=75100/80000, loss=0.1319, lr=0.000819, batch_cost=0.7870, reader_cost=0.0009 | ETA 01:04:16 2020-11-03 11:00:18 [INFO] [TRAIN] epoch=203, iter=75200/80000, loss=0.1418, lr=0.000804, batch_cost=0.7934, reader_cost=0.0085 | ETA 01:03:28 2020-11-03 11:01:37 [INFO] [TRAIN] epoch=203, iter=75300/80000, loss=0.1185, lr=0.000789, batch_cost=0.7913, reader_cost=0.0009 | ETA 01:01:59 2020-11-03 11:02:55 [INFO] [TRAIN] epoch=203, iter=75400/80000, loss=0.1225, lr=0.000774, batch_cost=0.7847, reader_cost=0.0010 | ETA 01:00:09 2020-11-03 11:04:15 [INFO] [TRAIN] epoch=203, iter=75500/80000, loss=0.1280, lr=0.000759, batch_cost=0.7984, reader_cost=0.0017 | ETA 00:59:52 2020-11-03 11:05:35 [INFO] [TRAIN] epoch=204, iter=75600/80000, loss=0.1311, lr=0.000744, batch_cost=0.8000, reader_cost=0.0111 | ETA 00:58:40 2020-11-03 11:06:54 [INFO] [TRAIN] epoch=204, iter=75700/80000, loss=0.1246, lr=0.000729, batch_cost=0.7861, reader_cost=0.0010 | ETA 00:56:20 2020-11-03 11:08:13 [INFO] [TRAIN] epoch=204, iter=75800/80000, loss=0.1267, lr=0.000714, batch_cost=0.7925, reader_cost=0.0017 | ETA 00:55:28 2020-11-03 11:09:33 [INFO] [TRAIN] epoch=205, iter=75900/80000, loss=0.1303, lr=0.000699, batch_cost=0.8029, reader_cost=0.0093 | ETA 00:54:51 2020-11-03 11:10:52 [INFO] [TRAIN] epoch=205, iter=76000/80000, loss=0.1442, lr=0.000684, batch_cost=0.7871, reader_cost=0.0012 | ETA 00:52:28 2020-11-03 11:12:10 [INFO] [TRAIN] epoch=205, iter=76100/80000, loss=0.1246, lr=0.000669, batch_cost=0.7822, reader_cost=0.0003 | ETA 00:50:50 2020-11-03 11:13:28 [INFO] [TRAIN] epoch=205, iter=76200/80000, loss=0.1308, lr=0.000654, batch_cost=0.7818, reader_cost=0.0003 | ETA 00:49:30 2020-11-03 11:14:47 [INFO] [TRAIN] epoch=206, iter=76300/80000, loss=0.1315, lr=0.000638, batch_cost=0.7900, reader_cost=0.0088 | ETA 00:48:43 2020-11-03 11:16:05 [INFO] [TRAIN] epoch=206, iter=76400/80000, loss=0.1274, lr=0.000623, batch_cost=0.7771, reader_cost=0.0005 | ETA 00:46:37 2020-11-03 11:17:22 [INFO] [TRAIN] epoch=206, iter=76500/80000, loss=0.1266, lr=0.000608, batch_cost=0.7751, reader_cost=0.0004 | ETA 00:45:12 2020-11-03 11:18:41 [INFO] [TRAIN] epoch=206, iter=76600/80000, loss=0.1308, lr=0.000592, batch_cost=0.7809, reader_cost=0.0006 | ETA 00:44:15 2020-11-03 11:19:59 [INFO] [TRAIN] epoch=207, iter=76700/80000, loss=0.1303, lr=0.000577, batch_cost=0.7855, reader_cost=0.0078 | ETA 00:43:12 2020-11-03 11:21:17 [INFO] [TRAIN] epoch=207, iter=76800/80000, loss=0.1346, lr=0.000561, batch_cost=0.7807, reader_cost=0.0002 | ETA 00:41:38 2020-11-03 11:22:36 [INFO] [TRAIN] epoch=207, iter=76900/80000, loss=0.1250, lr=0.000546, batch_cost=0.7855, reader_cost=0.0005 | ETA 00:40:34 2020-11-03 11:23:55 [INFO] [TRAIN] epoch=207, iter=77000/80000, loss=0.1240, lr=0.000530, batch_cost=0.7905, reader_cost=0.0009 | ETA 00:39:31 2020-11-03 11:25:14 [INFO] [TRAIN] epoch=208, iter=77100/80000, loss=0.1329, lr=0.000515, batch_cost=0.7967, reader_cost=0.0080 | ETA 00:38:30 2020-11-03 11:26:33 [INFO] [TRAIN] epoch=208, iter=77200/80000, loss=0.1151, lr=0.000499, batch_cost=0.7846, reader_cost=0.0002 | ETA 00:36:36 2020-11-03 11:27:51 [INFO] [TRAIN] epoch=208, iter=77300/80000, loss=0.1272, lr=0.000483, batch_cost=0.7831, reader_cost=0.0003 | ETA 00:35:14 2020-11-03 11:29:11 [INFO] [TRAIN] epoch=209, iter=77400/80000, loss=0.1330, lr=0.000468, batch_cost=0.7980, reader_cost=0.0083 | ETA 00:34:34 2020-11-03 11:30:29 [INFO] [TRAIN] epoch=209, iter=77500/80000, loss=0.1305, lr=0.000452, batch_cost=0.7805, reader_cost=0.0002 | ETA 00:32:31 2020-11-03 11:31:47 [INFO] [TRAIN] epoch=209, iter=77600/80000, loss=0.1261, lr=0.000436, batch_cost=0.7792, reader_cost=0.0006 | ETA 00:31:09 2020-11-03 11:33:05 [INFO] [TRAIN] epoch=209, iter=77700/80000, loss=0.1249, lr=0.000420, batch_cost=0.7795, reader_cost=0.0003 | ETA 00:29:52 2020-11-03 11:34:24 [INFO] [TRAIN] epoch=210, iter=77800/80000, loss=0.1238, lr=0.000404, batch_cost=0.7859, reader_cost=0.0091 | ETA 00:28:48 2020-11-03 11:35:42 [INFO] [TRAIN] epoch=210, iter=77900/80000, loss=0.1148, lr=0.000388, batch_cost=0.7814, reader_cost=0.0002 | ETA 00:27:20 2020-11-03 11:37:00 [INFO] [TRAIN] epoch=210, iter=78000/80000, loss=0.1264, lr=0.000371, batch_cost=0.7807, reader_cost=0.0002 | ETA 00:26:01 2020-11-03 11:38:18 [INFO] [TRAIN] epoch=210, iter=78100/80000, loss=0.1315, lr=0.000355, batch_cost=0.7836, reader_cost=0.0007 | ETA 00:24:48 2020-11-03 11:39:37 [INFO] [TRAIN] epoch=211, iter=78200/80000, loss=0.1279, lr=0.000339, batch_cost=0.7890, reader_cost=0.0088 | ETA 00:23:40 2020-11-03 11:40:54 [INFO] [TRAIN] epoch=211, iter=78300/80000, loss=0.1257, lr=0.000322, batch_cost=0.7746, reader_cost=0.0004 | ETA 00:21:56 2020-11-03 11:42:12 [INFO] [TRAIN] epoch=211, iter=78400/80000, loss=0.1204, lr=0.000306, batch_cost=0.7771, reader_cost=0.0004 | ETA 00:20:43 2020-11-03 11:43:31 [INFO] [TRAIN] epoch=212, iter=78500/80000, loss=0.1381, lr=0.000289, batch_cost=0.7873, reader_cost=0.0082 | ETA 00:19:41 2020-11-03 11:44:49 [INFO] [TRAIN] epoch=212, iter=78600/80000, loss=0.1362, lr=0.000272, batch_cost=0.7800, reader_cost=0.0002 | ETA 00:18:12 2020-11-03 11:46:07 [INFO] [TRAIN] epoch=212, iter=78700/80000, loss=0.1219, lr=0.000255, batch_cost=0.7808, reader_cost=0.0002 | ETA 00:16:55 2020-11-03 11:47:25 [INFO] [TRAIN] epoch=212, iter=78800/80000, loss=0.1252, lr=0.000238, batch_cost=0.7779, reader_cost=0.0006 | ETA 00:15:33 2020-11-03 11:48:44 [INFO] [TRAIN] epoch=213, iter=78900/80000, loss=0.1382, lr=0.000221, batch_cost=0.7907, reader_cost=0.0092 | ETA 00:14:29 2020-11-03 11:50:01 [INFO] [TRAIN] epoch=213, iter=79000/80000, loss=0.1316, lr=0.000204, batch_cost=0.7706, reader_cost=0.0003 | ETA 00:12:50 2020-11-03 11:51:19 [INFO] [TRAIN] epoch=213, iter=79100/80000, loss=0.1253, lr=0.000186, batch_cost=0.7795, reader_cost=0.0006 | ETA 00:11:41 2020-11-03 11:52:38 [INFO] [TRAIN] epoch=213, iter=79200/80000, loss=0.1322, lr=0.000169, batch_cost=0.7881, reader_cost=0.0005 | ETA 00:10:30 2020-11-03 11:53:57 [INFO] [TRAIN] epoch=214, iter=79300/80000, loss=0.1299, lr=0.000151, batch_cost=0.7885, reader_cost=0.0088 | ETA 00:09:11 2020-11-03 11:55:14 [INFO] [TRAIN] epoch=214, iter=79400/80000, loss=0.1149, lr=0.000132, batch_cost=0.7782, reader_cost=0.0002 | ETA 00:07:46 2020-11-03 11:56:32 [INFO] [TRAIN] epoch=214, iter=79500/80000, loss=0.1279, lr=0.000114, batch_cost=0.7811, reader_cost=0.0006 | ETA 00:06:30 2020-11-03 11:57:51 [INFO] [TRAIN] epoch=214, iter=79600/80000, loss=0.1226, lr=0.000095, batch_cost=0.7871, reader_cost=0.0008 | ETA 00:05:14 2020-11-03 11:59:09 [INFO] [TRAIN] epoch=215, iter=79700/80000, loss=0.1310, lr=0.000076, batch_cost=0.7779, reader_cost=0.0091 | ETA 00:03:53 2020-11-03 12:00:27 [INFO] [TRAIN] epoch=215, iter=79800/80000, loss=0.1119, lr=0.000056, batch_cost=0.7754, reader_cost=0.0004 | ETA 00:02:35 2020-11-03 12:01:44 [INFO] [TRAIN] epoch=215, iter=79900/80000, loss=0.1310, lr=0.000035, batch_cost=0.7712, reader_cost=0.0003 | ETA 00:01:17 2020-11-03 12:03:02 [INFO] [TRAIN] epoch=216, iter=80000/80000, loss=0.1395, lr=0.000010, batch_cost=0.7867, reader_cost=0.0087 | ETA 00:00:00 2020-11-03 12:03:12 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-03 12:09:48 [INFO] [EVAL] #Images=500 mIoU=0.8027 Acc=0.9643 Kappa=0.9537 2020-11-03 12:09:48 [INFO] [EVAL] Category IoU: [0.9843 0.8652 0.9306 0.5627 0.6263 0.6716 0.7381 0.8067 0.9283 0.622 0.9504 0.8355 0.6455 0.9574 0.848 0.9246 0.8533 0.704 0.7975] 2020-11-03 12:09:48 [INFO] [EVAL] Category Acc: [0.9912 0.9304 0.9588 0.8707 0.8495 0.8102 0.8565 0.9086 0.9548 0.9026 0.9696 0.8957 0.7992 0.9762 0.9563 0.9787 0.9684 0.8533 0.8826] 2020-11-03 12:09:48 [INFO] [EVAL] The model with the best validation mIoU (0.8048) was saved at iter 72000.