----------- Configuration Arguments ----------- gpus: None heter_worker_num: None heter_workers: http_port: None ips: 127.0.0.1 log_dir: log nproc_per_node: None server_num: None servers: training_script: train.py training_script_args: ['--config', 'configs/decoupled_segnet/decoupledsegnet_resnet50_os8_cityscapes_832x832_80k.yml', '--num_workers', '5', '--do_eval', '--save_interval', '2000', '--do_eval', '--use_vdl', '--log_iters', '100'] worker_num: None workers: ------------------------------------------------ launch train in GPU mode which: no nvcc in (/home/chenguowei01/anaconda3/envs/paddle/bin:/home/chenguowei01/anaconda3/condabin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/opt/bin:/home/opt/bin:/usr/local/sbin:/usr/sbin:/opt/bin:/home/opt/bin:/home/chenguowei01/.local/bin:/home/chenguowei01/bin:/opt/bin:/home/opt/bin) 2020-12-25 11:32:08 [INFO] ------------Environment Information------------- platform: Linux-3.10.0_3-0-0-34-x86_64-with-centos-7.5.1804-Core Python: 3.7.9 (default, Aug 31 2020, 12:42:55) [GCC 7.3.0] Paddle compiled with cuda: True NVCC: Cuda compilation tools, release 10.2, V10.2.89 cudnn: 7.6 GPUs used: 4 CUDA_VISIBLE_DEVICES: 3,4,5,6 GPU: ['GPU 0: Tesla V100-SXM2-16GB', 'GPU 1: Tesla V100-SXM2-16GB', 'GPU 2: Tesla V100-SXM2-16GB', 'GPU 3: Tesla V100-SXM2-16GB', 'GPU 4: Tesla V100-SXM2-16GB', 'GPU 5: Tesla V100-SXM2-16GB', 'GPU 6: Tesla V100-SXM2-16GB', 'GPU 7: Tesla V100-SXM2-16GB'] GCC: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39) PaddlePaddle: 2.0.0-rc1 OpenCV: 4.2.0 ------------------------------------------------ 2020-12-25 11:32:08 [INFO] ---------------Config Information--------------- batch_size: 2 iters: 80000 learning_rate: decay: end_lr: 0.0 power: 0.9 type: poly value: 0.01 loss: coef: - 1 - 1 - 25 - 1 types: - ignore_index: 255 type: OhemCrossEntropyLoss - ignore_index: 255 type: RelaxBoundaryLoss - edge_label: true ignore_index: 255 type: BCELoss weight: dynamic - ignore_index: 255 type: OhemEdgeAttentionLoss model: align_corners: false aspp_out_channels: 256 aspp_ratios: - 1 - 12 - 24 - 36 backbone: multi_grid: - 1 - 2 - 4 output_stride: 8 pretrained: https://bj.bcebos.com/paddleseg/dygraph/resnet50_vd_ssld_v2.tar.gz type: ResNet50_vd backbone_indices: - 0 - 3 num_classes: 19 pretrained: null type: DecoupledSegNet optimizer: momentum: 0.9 type: sgd weight_decay: 0.0005 train_dataset: dataset_root: data/cityscapes edge: true mode: train transforms: - max_scale_factor: 2.0 min_scale_factor: 0.75 scale_step_size: 0.25 type: ResizeStepScaling - crop_size: - 832 - 832 type: RandomPaddingCrop - type: RandomHorizontalFlip - brightness_range: 0.4 contrast_range: 0.4 saturation_range: 0.4 type: RandomDistort - type: Normalize type: Cityscapes val_dataset: dataset_root: data/cityscapes mode: val transforms: - type: Normalize type: Cityscapes ------------------------------------------------ W1225 11:32:09.006909 10058 device_context.cc:320] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 10.2, Runtime API Version: 10.2 W1225 11:32:09.006953 10058 device_context.cc:330] device: 0, cuDNN Version: 7.6. 2020-12-25 11:32:13 [INFO] Loading pretrained model from https://bj.bcebos.com/paddleseg/dygraph/resnet50_vd_ssld_v2.tar.gz 2020-12-25 11:32:14 [INFO] There are 275/275 variables loaded into ResNet_vd. W1225 11:32:15.223917 10058 nccl_context.cc:138] Socket connect worker 127.0.0.1:53687 failed, try again after 3 seconds. I1225 11:32:18.224279 10058 nccl_context.cc:179] init nccl context nranks: 4 local rank: 0 gpu id: 0 ring id: 0 2020-12-25 11:35:23 [INFO] [TRAIN] epoch=1, iter=100/80000, loss=19.8288, lr=0.009989, batch_cost=1.8187, reader_cost=0.0608 | ETA 40:21:53 2020-12-25 11:38:17 [INFO] [TRAIN] epoch=1, iter=200/80000, loss=19.4292, lr=0.009978, batch_cost=1.7393, reader_cost=0.0023 | ETA 38:33:19 2020-12-25 11:41:14 [INFO] [TRAIN] epoch=1, iter=300/80000, loss=18.9113, lr=0.009966, batch_cost=1.7669, reader_cost=0.0023 | ETA 39:07:04 2020-12-25 11:44:13 [INFO] [TRAIN] epoch=2, iter=400/80000, loss=18.9913, lr=0.009955, batch_cost=1.7890, reader_cost=0.0531 | ETA 39:33:25 2020-12-25 11:47:07 [INFO] [TRAIN] epoch=2, iter=500/80000, loss=19.0156, lr=0.009944, batch_cost=1.7358, reader_cost=0.0017 | ETA 38:19:58 2020-12-25 11:50:01 [INFO] [TRAIN] epoch=2, iter=600/80000, loss=18.1205, lr=0.009933, batch_cost=1.7379, reader_cost=0.0016 | ETA 38:19:47 2020-12-25 11:52:56 [INFO] [TRAIN] epoch=2, iter=700/80000, loss=18.2738, lr=0.009921, batch_cost=1.7468, reader_cost=0.0019 | ETA 38:28:44 2020-12-25 11:55:55 [INFO] [TRAIN] epoch=3, iter=800/80000, loss=18.2717, lr=0.009910, batch_cost=1.7904, reader_cost=0.0545 | ETA 39:23:17 2020-12-25 11:58:49 [INFO] [TRAIN] epoch=3, iter=900/80000, loss=18.5214, lr=0.009899, batch_cost=1.7381, reader_cost=0.0016 | ETA 38:11:22 2020-12-25 12:01:42 [INFO] [TRAIN] epoch=3, iter=1000/80000, loss=18.2945, lr=0.009888, batch_cost=1.7367, reader_cost=0.0011 | ETA 38:06:41 2020-12-25 12:04:36 [INFO] [TRAIN] epoch=3, iter=1100/80000, loss=18.6045, lr=0.009876, batch_cost=1.7333, reader_cost=0.0010 | ETA 37:59:18 2020-12-25 12:07:34 [INFO] [TRAIN] epoch=4, iter=1200/80000, loss=18.1409, lr=0.009865, batch_cost=1.7818, reader_cost=0.0522 | ETA 39:00:09 2020-12-25 12:10:28 [INFO] [TRAIN] epoch=4, iter=1300/80000, loss=18.0147, lr=0.009854, batch_cost=1.7315, reader_cost=0.0012 | ETA 37:51:07 2020-12-25 12:13:21 [INFO] [TRAIN] epoch=4, iter=1400/80000, loss=17.4767, lr=0.009842, batch_cost=1.7355, reader_cost=0.0014 | ETA 37:53:31 2020-12-25 12:16:20 [INFO] [TRAIN] epoch=5, iter=1500/80000, loss=17.7333, lr=0.009831, batch_cost=1.7882, reader_cost=0.0501 | ETA 38:59:32 2020-12-25 12:19:14 [INFO] [TRAIN] epoch=5, iter=1600/80000, loss=17.8178, lr=0.009820, batch_cost=1.7343, reader_cost=0.0015 | ETA 37:46:12 2020-12-25 12:22:08 [INFO] [TRAIN] epoch=5, iter=1700/80000, loss=17.6539, lr=0.009809, batch_cost=1.7390, reader_cost=0.0011 | ETA 37:49:27 2020-12-25 12:25:01 [INFO] [TRAIN] epoch=5, iter=1800/80000, loss=17.6757, lr=0.009797, batch_cost=1.7357, reader_cost=0.0007 | ETA 37:42:13 2020-12-25 12:28:00 [INFO] [TRAIN] epoch=6, iter=1900/80000, loss=17.3366, lr=0.009786, batch_cost=1.7843, reader_cost=0.0529 | ETA 38:42:30 2020-12-25 12:30:55 [INFO] [TRAIN] epoch=6, iter=2000/80000, loss=17.7173, lr=0.009775, batch_cost=1.7443, reader_cost=0.0017 | ETA 37:47:36 2020-12-25 12:30:55 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 12:31:37 [INFO] [EVAL] #Images=500 mIoU=0.5291 Acc=0.9177 Kappa=0.8933 2020-12-25 12:31:37 [INFO] [EVAL] Class IoU: [0.9608 0.7202 0.8553 0.281 0.3738 0.408 0.3947 0.3952 0.8808 0.4992 0.9047 0.7034 0.4109 0.8854 0.2012 0.4494 0.0629 0.0655 0.6005] 2020-12-25 12:31:37 [INFO] [EVAL] Class Acc: [0.9756 0.8318 0.9222 0.3399 0.4867 0.8756 0.8809 0.9396 0.9548 0.6289 0.9592 0.8169 0.4488 0.9565 0.5499 0.4963 0.0736 0.1862 0.656 ] 2020-12-25 12:31:43 [INFO] [EVAL] The model with the best validation mIoU (0.5291) was saved at iter 2000. 2020-12-25 12:34:36 [INFO] [TRAIN] epoch=6, iter=2100/80000, loss=17.1745, lr=0.009764, batch_cost=1.7265, reader_cost=0.0020 | ETA 37:21:35 2020-12-25 12:37:30 [INFO] [TRAIN] epoch=6, iter=2200/80000, loss=17.7392, lr=0.009752, batch_cost=1.7331, reader_cost=0.0021 | ETA 37:27:13 2020-12-25 12:40:29 [INFO] [TRAIN] epoch=7, iter=2300/80000, loss=17.8225, lr=0.009741, batch_cost=1.7906, reader_cost=0.0536 | ETA 38:38:46 2020-12-25 12:43:23 [INFO] [TRAIN] epoch=7, iter=2400/80000, loss=17.7034, lr=0.009730, batch_cost=1.7358, reader_cost=0.0022 | ETA 37:24:56 2020-12-25 12:46:16 [INFO] [TRAIN] epoch=7, iter=2500/80000, loss=17.1660, lr=0.009718, batch_cost=1.7379, reader_cost=0.0019 | ETA 37:24:48 2020-12-25 12:49:09 [INFO] [TRAIN] epoch=7, iter=2600/80000, loss=17.1926, lr=0.009707, batch_cost=1.7278, reader_cost=0.0016 | ETA 37:08:54 2020-12-25 12:52:09 [INFO] [TRAIN] epoch=8, iter=2700/80000, loss=17.3042, lr=0.009696, batch_cost=1.7907, reader_cost=0.0540 | ETA 38:27:04 2020-12-25 12:55:05 [INFO] [TRAIN] epoch=8, iter=2800/80000, loss=17.4167, lr=0.009685, batch_cost=1.7660, reader_cost=0.0011 | ETA 37:52:13 2020-12-25 12:57:59 [INFO] [TRAIN] epoch=8, iter=2900/80000, loss=17.0469, lr=0.009673, batch_cost=1.7401, reader_cost=0.0008 | ETA 37:16:04 2020-12-25 13:00:58 [INFO] [TRAIN] epoch=9, iter=3000/80000, loss=17.0229, lr=0.009662, batch_cost=1.7859, reader_cost=0.0565 | ETA 38:11:50 2020-12-25 13:03:52 [INFO] [TRAIN] epoch=9, iter=3100/80000, loss=17.3387, lr=0.009651, batch_cost=1.7384, reader_cost=0.0021 | ETA 37:08:04 2020-12-25 13:06:47 [INFO] [TRAIN] epoch=9, iter=3200/80000, loss=16.8470, lr=0.009639, batch_cost=1.7436, reader_cost=0.0025 | ETA 37:11:50 2020-12-25 13:09:41 [INFO] [TRAIN] epoch=9, iter=3300/80000, loss=16.8181, lr=0.009628, batch_cost=1.7407, reader_cost=0.0019 | ETA 37:05:12 2020-12-25 13:12:42 [INFO] [TRAIN] epoch=10, iter=3400/80000, loss=16.9462, lr=0.009617, batch_cost=1.8037, reader_cost=0.0520 | ETA 38:22:46 2020-12-25 13:15:48 [INFO] [TRAIN] epoch=10, iter=3500/80000, loss=17.1548, lr=0.009605, batch_cost=1.8612, reader_cost=0.0028 | ETA 39:32:59 2020-12-25 13:18:46 [INFO] [TRAIN] epoch=10, iter=3600/80000, loss=16.5366, lr=0.009594, batch_cost=1.7793, reader_cost=0.0020 | ETA 37:45:39 2020-12-25 13:21:44 [INFO] [TRAIN] epoch=10, iter=3700/80000, loss=16.9151, lr=0.009583, batch_cost=1.7787, reader_cost=0.0022 | ETA 37:41:53 2020-12-25 13:24:48 [INFO] [TRAIN] epoch=11, iter=3800/80000, loss=16.7339, lr=0.009572, batch_cost=1.8429, reader_cost=0.0533 | ETA 39:00:27 2020-12-25 13:27:45 [INFO] [TRAIN] epoch=11, iter=3900/80000, loss=16.8848, lr=0.009560, batch_cost=1.7673, reader_cost=0.0035 | ETA 37:21:27 2020-12-25 13:30:43 [INFO] [TRAIN] epoch=11, iter=4000/80000, loss=16.4411, lr=0.009549, batch_cost=1.7778, reader_cost=0.0034 | ETA 37:31:49 2020-12-25 13:30:43 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 13:31:25 [INFO] [EVAL] #Images=500 mIoU=0.6614 Acc=0.9426 Kappa=0.9255 2020-12-25 13:31:25 [INFO] [EVAL] Class IoU: [0.9718 0.7855 0.8916 0.3626 0.5099 0.5889 0.6056 0.7199 0.8977 0.5449 0.9153 0.7725 0.5123 0.9188 0.4902 0.6068 0.2966 0.4531 0.7231] 2020-12-25 13:31:25 [INFO] [EVAL] Class Acc: [0.9857 0.8549 0.9279 0.7061 0.6828 0.8095 0.8459 0.8831 0.9609 0.6936 0.9328 0.8671 0.7285 0.9592 0.6096 0.8467 0.8163 0.5656 0.8161] 2020-12-25 13:31:32 [INFO] [EVAL] The model with the best validation mIoU (0.6614) was saved at iter 4000. 2020-12-25 13:34:38 [INFO] [TRAIN] epoch=12, iter=4100/80000, loss=16.8382, lr=0.009538, batch_cost=1.8602, reader_cost=0.0626 | ETA 39:13:08 2020-12-25 13:37:47 [INFO] [TRAIN] epoch=12, iter=4200/80000, loss=16.7085, lr=0.009526, batch_cost=1.8903, reader_cost=0.0035 | ETA 39:48:02 2020-12-25 13:40:55 [INFO] [TRAIN] epoch=12, iter=4300/80000, loss=16.2859, lr=0.009515, batch_cost=1.8813, reader_cost=0.0037 | ETA 39:33:30 2020-12-25 13:44:03 [INFO] [TRAIN] epoch=12, iter=4400/80000, loss=16.2307, lr=0.009504, batch_cost=1.8790, reader_cost=0.0051 | ETA 39:27:31 2020-12-25 13:47:17 [INFO] [TRAIN] epoch=13, iter=4500/80000, loss=16.1677, lr=0.009492, batch_cost=1.9328, reader_cost=0.0571 | ETA 40:32:09 2020-12-25 13:50:24 [INFO] [TRAIN] epoch=13, iter=4600/80000, loss=16.4801, lr=0.009481, batch_cost=1.8762, reader_cost=0.0032 | ETA 39:17:49 2020-12-25 13:53:33 [INFO] [TRAIN] epoch=13, iter=4700/80000, loss=16.2421, lr=0.009470, batch_cost=1.8814, reader_cost=0.0040 | ETA 39:21:06 2020-12-25 13:56:42 [INFO] [TRAIN] epoch=13, iter=4800/80000, loss=16.2420, lr=0.009458, batch_cost=1.8936, reader_cost=0.0045 | ETA 39:33:19 2020-12-25 13:59:58 [INFO] [TRAIN] epoch=14, iter=4900/80000, loss=16.1719, lr=0.009447, batch_cost=1.9543, reader_cost=0.0540 | ETA 40:46:11 2020-12-25 14:03:06 [INFO] [TRAIN] epoch=14, iter=5000/80000, loss=16.2916, lr=0.009436, batch_cost=1.8816, reader_cost=0.0034 | ETA 39:11:56 2020-12-25 14:06:13 [INFO] [TRAIN] epoch=14, iter=5100/80000, loss=16.1448, lr=0.009424, batch_cost=1.8663, reader_cost=0.0036 | ETA 38:49:43 2020-12-25 14:09:20 [INFO] [TRAIN] epoch=14, iter=5200/80000, loss=16.3160, lr=0.009413, batch_cost=1.8667, reader_cost=0.0033 | ETA 38:47:07 2020-12-25 14:12:35 [INFO] [TRAIN] epoch=15, iter=5300/80000, loss=16.2332, lr=0.009402, batch_cost=1.9498, reader_cost=0.0577 | ETA 40:27:27 2020-12-25 14:15:53 [INFO] [TRAIN] epoch=15, iter=5400/80000, loss=16.0777, lr=0.009391, batch_cost=1.9809, reader_cost=0.0043 | ETA 41:02:54 2020-12-25 14:19:18 [INFO] [TRAIN] epoch=15, iter=5500/80000, loss=15.9043, lr=0.009379, batch_cost=2.0506, reader_cost=0.0056 | ETA 42:26:11 2020-12-25 14:22:51 [INFO] [TRAIN] epoch=16, iter=5600/80000, loss=15.8874, lr=0.009368, batch_cost=2.1241, reader_cost=0.0710 | ETA 43:53:52 2020-12-25 14:26:18 [INFO] [TRAIN] epoch=16, iter=5700/80000, loss=16.0966, lr=0.009357, batch_cost=2.0647, reader_cost=0.0070 | ETA 42:36:50 2020-12-25 14:29:43 [INFO] [TRAIN] epoch=16, iter=5800/80000, loss=16.0006, lr=0.009345, batch_cost=2.0565, reader_cost=0.0057 | ETA 42:23:11 2020-12-25 14:33:15 [INFO] [TRAIN] epoch=16, iter=5900/80000, loss=15.9407, lr=0.009334, batch_cost=2.1134, reader_cost=0.0062 | ETA 43:30:00 2020-12-25 14:36:50 [INFO] [TRAIN] epoch=17, iter=6000/80000, loss=16.0291, lr=0.009323, batch_cost=2.1465, reader_cost=0.0708 | ETA 44:07:24 2020-12-25 14:36:50 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 14:37:35 [INFO] [EVAL] #Images=500 mIoU=0.6250 Acc=0.9245 Kappa=0.9023 2020-12-25 14:37:35 [INFO] [EVAL] Class IoU: [0.9608 0.7347 0.8529 0.2943 0.4315 0.5885 0.6302 0.6814 0.9017 0.4919 0.9368 0.7526 0.5568 0.8358 0.4049 0.4989 0.0649 0.5237 0.7323] 2020-12-25 14:37:35 [INFO] [EVAL] Class Acc: [0.9861 0.8746 0.9297 0.85 0.6098 0.7619 0.8627 0.9496 0.9486 0.8452 0.9613 0.8564 0.7012 0.8494 0.7409 0.6772 0.0657 0.7027 0.8302] 2020-12-25 14:37:40 [INFO] [EVAL] The model with the best validation mIoU (0.6614) was saved at iter 4000. 2020-12-25 14:41:10 [INFO] [TRAIN] epoch=17, iter=6100/80000, loss=16.1603, lr=0.009311, batch_cost=2.1000, reader_cost=0.0040 | ETA 43:06:27 2020-12-25 14:44:40 [INFO] [TRAIN] epoch=17, iter=6200/80000, loss=15.6336, lr=0.009300, batch_cost=2.0979, reader_cost=0.0056 | ETA 43:00:23 2020-12-25 14:48:09 [INFO] [TRAIN] epoch=17, iter=6300/80000, loss=15.8203, lr=0.009288, batch_cost=2.0909, reader_cost=0.0054 | ETA 42:48:22 2020-12-25 14:51:43 [INFO] [TRAIN] epoch=18, iter=6400/80000, loss=15.9344, lr=0.009277, batch_cost=2.1389, reader_cost=0.0592 | ETA 43:43:45 2020-12-25 14:55:14 [INFO] [TRAIN] epoch=18, iter=6500/80000, loss=15.9477, lr=0.009266, batch_cost=2.1037, reader_cost=0.0055 | ETA 42:57:04 2020-12-25 14:58:44 [INFO] [TRAIN] epoch=18, iter=6600/80000, loss=15.5966, lr=0.009254, batch_cost=2.0972, reader_cost=0.0055 | ETA 42:45:36 2020-12-25 15:02:24 [INFO] [TRAIN] epoch=19, iter=6700/80000, loss=15.8998, lr=0.009243, batch_cost=2.2025, reader_cost=0.0842 | ETA 44:50:46 2020-12-25 15:05:49 [INFO] [TRAIN] epoch=19, iter=6800/80000, loss=15.6441, lr=0.009232, batch_cost=2.0509, reader_cost=0.0065 | ETA 41:42:02 2020-12-25 15:09:11 [INFO] [TRAIN] epoch=19, iter=6900/80000, loss=15.7160, lr=0.009220, batch_cost=2.0152, reader_cost=0.0048 | ETA 40:55:08 2020-12-25 15:12:33 [INFO] [TRAIN] epoch=19, iter=7000/80000, loss=15.6849, lr=0.009209, batch_cost=2.0198, reader_cost=0.0070 | ETA 40:57:23 2020-12-25 15:16:03 [INFO] [TRAIN] epoch=20, iter=7100/80000, loss=15.6202, lr=0.009198, batch_cost=2.1001, reader_cost=0.0701 | ETA 42:31:38 2020-12-25 15:19:28 [INFO] [TRAIN] epoch=20, iter=7200/80000, loss=15.7425, lr=0.009186, batch_cost=2.0475, reader_cost=0.0065 | ETA 41:24:15 2020-12-25 15:22:53 [INFO] [TRAIN] epoch=20, iter=7300/80000, loss=15.6193, lr=0.009175, batch_cost=2.0422, reader_cost=0.0059 | ETA 41:14:25 2020-12-25 15:26:13 [INFO] [TRAIN] epoch=20, iter=7400/80000, loss=15.5006, lr=0.009164, batch_cost=2.0008, reader_cost=0.0056 | ETA 40:20:56 2020-12-25 15:29:47 [INFO] [TRAIN] epoch=21, iter=7500/80000, loss=15.7080, lr=0.009152, batch_cost=2.1384, reader_cost=0.0689 | ETA 43:03:57 2020-12-25 15:33:12 [INFO] [TRAIN] epoch=21, iter=7600/80000, loss=15.7105, lr=0.009141, batch_cost=2.0419, reader_cost=0.0062 | ETA 41:03:56 2020-12-25 15:36:39 [INFO] [TRAIN] epoch=21, iter=7700/80000, loss=15.4072, lr=0.009130, batch_cost=2.0683, reader_cost=0.0064 | ETA 41:32:17 2020-12-25 15:40:02 [INFO] [TRAIN] epoch=21, iter=7800/80000, loss=15.6684, lr=0.009118, batch_cost=2.0289, reader_cost=0.0075 | ETA 40:41:27 2020-12-25 15:43:32 [INFO] [TRAIN] epoch=22, iter=7900/80000, loss=15.6250, lr=0.009107, batch_cost=2.1011, reader_cost=0.0653 | ETA 42:04:46 2020-12-25 15:46:56 [INFO] [TRAIN] epoch=22, iter=8000/80000, loss=15.4624, lr=0.009095, batch_cost=2.0391, reader_cost=0.0065 | ETA 40:46:56 2020-12-25 15:46:56 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 15:47:40 [INFO] [EVAL] #Images=500 mIoU=0.6641 Acc=0.9390 Kappa=0.9209 2020-12-25 15:47:40 [INFO] [EVAL] Class IoU: [0.9657 0.7473 0.8945 0.2277 0.5228 0.6153 0.6571 0.759 0.884 0.419 0.9391 0.7827 0.5253 0.9274 0.6545 0.6654 0.1849 0.5425 0.7033] 2020-12-25 15:47:40 [INFO] [EVAL] Class Acc: [0.9835 0.8159 0.9384 0.828 0.6539 0.7805 0.7549 0.9023 0.9471 0.5383 0.9627 0.9098 0.722 0.9681 0.7561 0.8579 0.9686 0.7389 0.742 ] 2020-12-25 15:47:47 [INFO] [EVAL] The model with the best validation mIoU (0.6641) was saved at iter 8000. 2020-12-25 15:51:11 [INFO] [TRAIN] epoch=22, iter=8100/80000, loss=15.4772, lr=0.009084, batch_cost=2.0361, reader_cost=0.0053 | ETA 40:39:58 2020-12-25 15:54:45 [INFO] [TRAIN] epoch=23, iter=8200/80000, loss=15.4664, lr=0.009073, batch_cost=2.1412, reader_cost=0.0692 | ETA 42:42:17 2020-12-25 15:58:08 [INFO] [TRAIN] epoch=23, iter=8300/80000, loss=15.4345, lr=0.009061, batch_cost=2.0240, reader_cost=0.0052 | ETA 40:18:43 2020-12-25 16:01:35 [INFO] [TRAIN] epoch=23, iter=8400/80000, loss=15.4459, lr=0.009050, batch_cost=2.0756, reader_cost=0.0063 | ETA 41:16:53 2020-12-25 16:05:05 [INFO] [TRAIN] epoch=23, iter=8500/80000, loss=15.5728, lr=0.009039, batch_cost=2.0936, reader_cost=0.0049 | ETA 41:34:55 2020-12-25 16:08:36 [INFO] [TRAIN] epoch=24, iter=8600/80000, loss=15.4902, lr=0.009027, batch_cost=2.1098, reader_cost=0.0597 | ETA 41:50:38 2020-12-25 16:12:03 [INFO] [TRAIN] epoch=24, iter=8700/80000, loss=15.5023, lr=0.009016, batch_cost=2.0679, reader_cost=0.0057 | ETA 40:57:20 2020-12-25 16:15:29 [INFO] [TRAIN] epoch=24, iter=8800/80000, loss=15.3054, lr=0.009004, batch_cost=2.0574, reader_cost=0.0047 | ETA 40:41:26 2020-12-25 16:18:59 [INFO] [TRAIN] epoch=24, iter=8900/80000, loss=15.6810, lr=0.008993, batch_cost=2.0935, reader_cost=0.0050 | ETA 41:20:47 2020-12-25 16:22:32 [INFO] [TRAIN] epoch=25, iter=9000/80000, loss=15.4184, lr=0.008982, batch_cost=2.1358, reader_cost=0.0663 | ETA 42:07:22 2020-12-25 16:26:00 [INFO] [TRAIN] epoch=25, iter=9100/80000, loss=15.1996, lr=0.008970, batch_cost=2.0706, reader_cost=0.0054 | ETA 40:46:46 2020-12-25 16:29:19 [INFO] [TRAIN] epoch=25, iter=9200/80000, loss=15.2894, lr=0.008959, batch_cost=1.9969, reader_cost=0.0034 | ETA 39:16:17 2020-12-25 16:32:31 [INFO] [TRAIN] epoch=25, iter=9300/80000, loss=15.4038, lr=0.008948, batch_cost=1.9173, reader_cost=0.0032 | ETA 37:39:10 2020-12-25 16:35:51 [INFO] [TRAIN] epoch=26, iter=9400/80000, loss=15.3291, lr=0.008936, batch_cost=2.0001, reader_cost=0.0638 | ETA 39:13:27 2020-12-25 16:39:04 [INFO] [TRAIN] epoch=26, iter=9500/80000, loss=15.2800, lr=0.008925, batch_cost=1.9222, reader_cost=0.0043 | ETA 37:38:35 2020-12-25 16:42:34 [INFO] [TRAIN] epoch=26, iter=9600/80000, loss=15.4091, lr=0.008913, batch_cost=2.1045, reader_cost=0.0045 | ETA 41:09:15 2020-12-25 16:46:13 [INFO] [TRAIN] epoch=27, iter=9700/80000, loss=15.0271, lr=0.008902, batch_cost=2.1812, reader_cost=0.0720 | ETA 42:35:36 2020-12-25 16:49:41 [INFO] [TRAIN] epoch=27, iter=9800/80000, loss=15.3623, lr=0.008891, batch_cost=2.0770, reader_cost=0.0060 | ETA 40:30:06 2020-12-25 16:53:13 [INFO] [TRAIN] epoch=27, iter=9900/80000, loss=15.1699, lr=0.008879, batch_cost=2.1172, reader_cost=0.0049 | ETA 41:13:39 2020-12-25 16:56:44 [INFO] [TRAIN] epoch=27, iter=10000/80000, loss=15.0898, lr=0.008868, batch_cost=2.1087, reader_cost=0.0052 | ETA 41:00:09 2020-12-25 16:56:44 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 16:57:28 [INFO] [EVAL] #Images=500 mIoU=0.6580 Acc=0.9430 Kappa=0.9259 2020-12-25 16:57:28 [INFO] [EVAL] Class IoU: [0.9669 0.7669 0.8987 0.2141 0.4651 0.6172 0.6603 0.7187 0.9042 0.5633 0.9417 0.7647 0.5386 0.9239 0.5729 0.5514 0.475 0.2636 0.6956] 2020-12-25 16:57:28 [INFO] [EVAL] Class Acc: [0.9835 0.8881 0.9287 0.7884 0.7779 0.8042 0.8109 0.9345 0.9396 0.8411 0.9612 0.8908 0.6243 0.9575 0.6232 0.9764 0.6099 0.2738 0.7932] 2020-12-25 16:57:33 [INFO] [EVAL] The model with the best validation mIoU (0.6641) was saved at iter 8000. 2020-12-25 17:01:10 [INFO] [TRAIN] epoch=28, iter=10100/80000, loss=15.3438, lr=0.008856, batch_cost=2.1642, reader_cost=0.0751 | ETA 42:01:16 2020-12-25 17:04:46 [INFO] [TRAIN] epoch=28, iter=10200/80000, loss=15.2907, lr=0.008845, batch_cost=2.1642, reader_cost=0.0081 | ETA 41:57:44 2020-12-25 17:08:20 [INFO] [TRAIN] epoch=28, iter=10300/80000, loss=15.1960, lr=0.008834, batch_cost=2.1373, reader_cost=0.0063 | ETA 41:22:47 2020-12-25 17:11:38 [INFO] [TRAIN] epoch=28, iter=10400/80000, loss=15.3739, lr=0.008822, batch_cost=1.9784, reader_cost=0.0047 | ETA 38:14:59 2020-12-25 17:14:59 [INFO] [TRAIN] epoch=29, iter=10500/80000, loss=15.2689, lr=0.008811, batch_cost=2.0057, reader_cost=0.0543 | ETA 38:43:19 2020-12-25 17:17:58 [INFO] [TRAIN] epoch=29, iter=10600/80000, loss=15.3022, lr=0.008799, batch_cost=1.7865, reader_cost=0.0029 | ETA 34:26:24 2020-12-25 17:20:55 [INFO] [TRAIN] epoch=29, iter=10700/80000, loss=15.1810, lr=0.008788, batch_cost=1.7677, reader_cost=0.0024 | ETA 34:01:43 2020-12-25 17:23:59 [INFO] [TRAIN] epoch=30, iter=10800/80000, loss=15.1495, lr=0.008776, batch_cost=1.8443, reader_cost=0.0565 | ETA 35:27:06 2020-12-25 17:26:56 [INFO] [TRAIN] epoch=30, iter=10900/80000, loss=15.3253, lr=0.008765, batch_cost=1.7623, reader_cost=0.0030 | ETA 33:49:38 2020-12-25 17:30:00 [INFO] [TRAIN] epoch=30, iter=11000/80000, loss=15.1969, lr=0.008754, batch_cost=1.8364, reader_cost=0.0024 | ETA 35:11:49 2020-12-25 17:32:58 [INFO] [TRAIN] epoch=30, iter=11100/80000, loss=15.1504, lr=0.008742, batch_cost=1.7829, reader_cost=0.0031 | ETA 34:07:21 2020-12-25 17:36:02 [INFO] [TRAIN] epoch=31, iter=11200/80000, loss=15.2115, lr=0.008731, batch_cost=1.8415, reader_cost=0.0603 | ETA 35:11:36 2020-12-25 17:38:57 [INFO] [TRAIN] epoch=31, iter=11300/80000, loss=15.2760, lr=0.008719, batch_cost=1.7487, reader_cost=0.0011 | ETA 33:22:14 2020-12-25 17:41:52 [INFO] [TRAIN] epoch=31, iter=11400/80000, loss=15.1245, lr=0.008708, batch_cost=1.7424, reader_cost=0.0006 | ETA 33:12:09 2020-12-25 17:44:48 [INFO] [TRAIN] epoch=31, iter=11500/80000, loss=15.1774, lr=0.008697, batch_cost=1.7604, reader_cost=0.0030 | ETA 33:29:50 2020-12-25 17:47:55 [INFO] [TRAIN] epoch=32, iter=11600/80000, loss=15.3183, lr=0.008685, batch_cost=1.8673, reader_cost=0.0532 | ETA 35:28:41 2020-12-25 17:50:53 [INFO] [TRAIN] epoch=32, iter=11700/80000, loss=15.1128, lr=0.008674, batch_cost=1.7844, reader_cost=0.0029 | ETA 33:51:17 2020-12-25 17:53:52 [INFO] [TRAIN] epoch=32, iter=11800/80000, loss=15.0782, lr=0.008662, batch_cost=1.7834, reader_cost=0.0030 | ETA 33:47:07 2020-12-25 17:56:47 [INFO] [TRAIN] epoch=32, iter=11900/80000, loss=15.2878, lr=0.008651, batch_cost=1.7470, reader_cost=0.0022 | ETA 33:02:48 2020-12-25 17:59:51 [INFO] [TRAIN] epoch=33, iter=12000/80000, loss=15.2593, lr=0.008639, batch_cost=1.8403, reader_cost=0.0521 | ETA 34:45:39 2020-12-25 17:59:51 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 18:00:33 [INFO] [EVAL] #Images=500 mIoU=0.6504 Acc=0.9454 Kappa=0.9289 2020-12-25 18:00:33 [INFO] [EVAL] Class IoU: [0.9712 0.792 0.903 0.3431 0.539 0.573 0.626 0.744 0.9004 0.4429 0.9403 0.762 0.4423 0.9266 0.4603 0.6031 0.361 0.2979 0.7304] 2020-12-25 18:00:33 [INFO] [EVAL] Class Acc: [0.9803 0.9217 0.9405 0.7957 0.6576 0.8533 0.8752 0.9258 0.923 0.8939 0.9778 0.8138 0.76 0.9707 0.7984 0.6562 0.9024 0.7799 0.8314] 2020-12-25 18:00:38 [INFO] [EVAL] The model with the best validation mIoU (0.6641) was saved at iter 8000. 2020-12-25 18:03:43 [INFO] [TRAIN] epoch=33, iter=12100/80000, loss=15.2626, lr=0.008628, batch_cost=1.8455, reader_cost=0.0043 | ETA 34:48:28 2020-12-25 18:06:53 [INFO] [TRAIN] epoch=33, iter=12200/80000, loss=15.2743, lr=0.008617, batch_cost=1.8994, reader_cost=0.0045 | ETA 35:46:22 2020-12-25 18:10:13 [INFO] [TRAIN] epoch=34, iter=12300/80000, loss=15.1562, lr=0.008605, batch_cost=1.9957, reader_cost=0.0578 | ETA 37:31:45 2020-12-25 18:13:28 [INFO] [TRAIN] epoch=34, iter=12400/80000, loss=15.1368, lr=0.008594, batch_cost=1.9542, reader_cost=0.0052 | ETA 36:41:45 2020-12-25 18:16:44 [INFO] [TRAIN] epoch=34, iter=12500/80000, loss=15.2091, lr=0.008582, batch_cost=1.9577, reader_cost=0.0050 | ETA 36:42:28 2020-12-25 18:19:57 [INFO] [TRAIN] epoch=34, iter=12600/80000, loss=15.1750, lr=0.008571, batch_cost=1.9289, reader_cost=0.0052 | ETA 36:06:48 2020-12-25 18:23:16 [INFO] [TRAIN] epoch=35, iter=12700/80000, loss=15.2095, lr=0.008559, batch_cost=1.9882, reader_cost=0.0691 | ETA 37:10:03 2020-12-25 18:26:33 [INFO] [TRAIN] epoch=35, iter=12800/80000, loss=15.2342, lr=0.008548, batch_cost=1.9630, reader_cost=0.0068 | ETA 36:38:32 2020-12-25 18:29:48 [INFO] [TRAIN] epoch=35, iter=12900/80000, loss=14.8973, lr=0.008536, batch_cost=1.9489, reader_cost=0.0065 | ETA 36:19:28 2020-12-25 18:32:58 [INFO] [TRAIN] epoch=35, iter=13000/80000, loss=15.1862, lr=0.008525, batch_cost=1.9013, reader_cost=0.0052 | ETA 35:23:04 2020-12-25 18:36:19 [INFO] [TRAIN] epoch=36, iter=13100/80000, loss=15.1716, lr=0.008514, batch_cost=2.0108, reader_cost=0.0643 | ETA 37:22:00 2020-12-25 18:39:33 [INFO] [TRAIN] epoch=36, iter=13200/80000, loss=15.1077, lr=0.008502, batch_cost=1.9361, reader_cost=0.0048 | ETA 35:55:28 2020-12-25 18:42:46 [INFO] [TRAIN] epoch=36, iter=13300/80000, loss=15.0072, lr=0.008491, batch_cost=1.9218, reader_cost=0.0061 | ETA 35:36:21 2020-12-25 18:46:05 [INFO] [TRAIN] epoch=37, iter=13400/80000, loss=15.1306, lr=0.008479, batch_cost=1.9943, reader_cost=0.0654 | ETA 36:53:42 2020-12-25 18:49:16 [INFO] [TRAIN] epoch=37, iter=13500/80000, loss=15.2848, lr=0.008468, batch_cost=1.9029, reader_cost=0.0051 | ETA 35:09:05 2020-12-25 18:52:25 [INFO] [TRAIN] epoch=37, iter=13600/80000, loss=15.0762, lr=0.008456, batch_cost=1.8907, reader_cost=0.0050 | ETA 34:52:22 2020-12-25 18:55:32 [INFO] [TRAIN] epoch=37, iter=13700/80000, loss=15.2343, lr=0.008445, batch_cost=1.8630, reader_cost=0.0057 | ETA 34:18:39 2020-12-25 18:58:50 [INFO] [TRAIN] epoch=38, iter=13800/80000, loss=15.0482, lr=0.008433, batch_cost=1.9844, reader_cost=0.0564 | ETA 36:29:24 2020-12-25 19:02:02 [INFO] [TRAIN] epoch=38, iter=13900/80000, loss=15.0593, lr=0.008422, batch_cost=1.9183, reader_cost=0.0047 | ETA 35:13:22 2020-12-25 19:05:12 [INFO] [TRAIN] epoch=38, iter=14000/80000, loss=14.9575, lr=0.008410, batch_cost=1.8954, reader_cost=0.0044 | ETA 34:44:55 2020-12-25 19:05:12 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 19:05:55 [INFO] [EVAL] #Images=500 mIoU=0.6824 Acc=0.9460 Kappa=0.9298 2020-12-25 19:05:55 [INFO] [EVAL] Class IoU: [0.9731 0.787 0.8976 0.221 0.5067 0.5635 0.6796 0.7487 0.909 0.6065 0.9352 0.7962 0.5461 0.9043 0.5328 0.5801 0.5791 0.4769 0.7217] 2020-12-25 19:05:55 [INFO] [EVAL] Class Acc: [0.9892 0.8668 0.9225 0.868 0.8458 0.8654 0.8288 0.9207 0.9459 0.7721 0.9716 0.8805 0.7455 0.9237 0.7398 0.9521 0.9677 0.8331 0.8725] 2020-12-25 19:06:03 [INFO] [EVAL] The model with the best validation mIoU (0.6824) was saved at iter 14000. 2020-12-25 19:09:06 [INFO] [TRAIN] epoch=38, iter=14100/80000, loss=15.0986, lr=0.008399, batch_cost=1.8320, reader_cost=0.0050 | ETA 33:32:10 2020-12-25 19:12:13 [INFO] [TRAIN] epoch=39, iter=14200/80000, loss=15.0856, lr=0.008387, batch_cost=1.8674, reader_cost=0.0532 | ETA 34:07:52 2020-12-25 19:15:08 [INFO] [TRAIN] epoch=39, iter=14300/80000, loss=15.0194, lr=0.008376, batch_cost=1.7438, reader_cost=0.0022 | ETA 31:49:30 2020-12-25 19:18:02 [INFO] [TRAIN] epoch=39, iter=14400/80000, loss=14.8208, lr=0.008364, batch_cost=1.7413, reader_cost=0.0016 | ETA 31:43:48 2020-12-25 19:20:55 [INFO] [TRAIN] epoch=39, iter=14500/80000, loss=14.8534, lr=0.008353, batch_cost=1.7261, reader_cost=0.0019 | ETA 31:24:21 2020-12-25 19:23:55 [INFO] [TRAIN] epoch=40, iter=14600/80000, loss=15.0465, lr=0.008342, batch_cost=1.8046, reader_cost=0.0532 | ETA 32:46:59 2020-12-25 19:26:49 [INFO] [TRAIN] epoch=40, iter=14700/80000, loss=15.0198, lr=0.008330, batch_cost=1.7303, reader_cost=0.0015 | ETA 31:23:08 2020-12-25 19:29:44 [INFO] [TRAIN] epoch=40, iter=14800/80000, loss=14.9324, lr=0.008319, batch_cost=1.7502, reader_cost=0.0016 | ETA 31:41:53 2020-12-25 19:32:45 [INFO] [TRAIN] epoch=41, iter=14900/80000, loss=14.9488, lr=0.008307, batch_cost=1.8093, reader_cost=0.0518 | ETA 32:43:06 2020-12-25 19:35:39 [INFO] [TRAIN] epoch=41, iter=15000/80000, loss=15.1120, lr=0.008296, batch_cost=1.7425, reader_cost=0.0022 | ETA 31:27:41 2020-12-25 19:38:33 [INFO] [TRAIN] epoch=41, iter=15100/80000, loss=14.8643, lr=0.008284, batch_cost=1.7379, reader_cost=0.0014 | ETA 31:19:49 2020-12-25 19:41:29 [INFO] [TRAIN] epoch=41, iter=15200/80000, loss=15.0292, lr=0.008273, batch_cost=1.7604, reader_cost=0.0015 | ETA 31:41:11 2020-12-25 19:44:31 [INFO] [TRAIN] epoch=42, iter=15300/80000, loss=15.0705, lr=0.008261, batch_cost=1.8154, reader_cost=0.0522 | ETA 32:37:34 2020-12-25 19:47:25 [INFO] [TRAIN] epoch=42, iter=15400/80000, loss=15.1339, lr=0.008250, batch_cost=1.7353, reader_cost=0.0014 | ETA 31:08:22 2020-12-25 19:50:20 [INFO] [TRAIN] epoch=42, iter=15500/80000, loss=14.9901, lr=0.008238, batch_cost=1.7513, reader_cost=0.0013 | ETA 31:22:39 2020-12-25 19:53:28 [INFO] [TRAIN] epoch=42, iter=15600/80000, loss=15.0548, lr=0.008227, batch_cost=1.8811, reader_cost=0.0027 | ETA 33:39:05 2020-12-25 19:57:04 [INFO] [TRAIN] epoch=43, iter=15700/80000, loss=15.1181, lr=0.008215, batch_cost=2.1547, reader_cost=0.0692 | ETA 38:29:06 2020-12-25 20:00:28 [INFO] [TRAIN] epoch=43, iter=15800/80000, loss=14.9080, lr=0.008204, batch_cost=2.0426, reader_cost=0.0032 | ETA 36:25:34 2020-12-25 20:03:57 [INFO] [TRAIN] epoch=43, iter=15900/80000, loss=14.9187, lr=0.008192, batch_cost=2.0821, reader_cost=0.0058 | ETA 37:04:23 2020-12-25 20:07:27 [INFO] [TRAIN] epoch=44, iter=16000/80000, loss=14.9990, lr=0.008181, batch_cost=2.1035, reader_cost=0.0555 | ETA 37:23:41 2020-12-25 20:07:27 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 20:08:09 [INFO] [EVAL] #Images=500 mIoU=0.6948 Acc=0.9540 Kappa=0.9402 2020-12-25 20:08:09 [INFO] [EVAL] Class IoU: [0.9769 0.827 0.9127 0.4684 0.4701 0.6411 0.6521 0.773 0.9181 0.5928 0.941 0.7942 0.5517 0.9448 0.7295 0.69 0.026 0.5346 0.7574] 2020-12-25 20:08:09 [INFO] [EVAL] Class Acc: [0.9915 0.8815 0.9452 0.8476 0.7476 0.8467 0.8893 0.8971 0.9476 0.8524 0.9586 0.8437 0.8315 0.9715 0.9254 0.7276 0.9925 0.8374 0.8348] 2020-12-25 20:08:16 [INFO] [EVAL] The model with the best validation mIoU (0.6948) was saved at iter 16000. 2020-12-25 20:11:09 [INFO] [TRAIN] epoch=44, iter=16100/80000, loss=15.1101, lr=0.008169, batch_cost=1.7334, reader_cost=0.0017 | ETA 30:46:06 2020-12-25 20:14:03 [INFO] [TRAIN] epoch=44, iter=16200/80000, loss=15.0967, lr=0.008158, batch_cost=1.7321, reader_cost=0.0016 | ETA 30:41:49 2020-12-25 20:16:57 [INFO] [TRAIN] epoch=44, iter=16300/80000, loss=14.8886, lr=0.008146, batch_cost=1.7396, reader_cost=0.0015 | ETA 30:46:54 2020-12-25 20:19:57 [INFO] [TRAIN] epoch=45, iter=16400/80000, loss=15.0078, lr=0.008135, batch_cost=1.8044, reader_cost=0.0504 | ETA 31:52:36 2020-12-25 20:22:51 [INFO] [TRAIN] epoch=45, iter=16500/80000, loss=14.9901, lr=0.008123, batch_cost=1.7373, reader_cost=0.0014 | ETA 30:38:40 2020-12-25 20:25:45 [INFO] [TRAIN] epoch=45, iter=16600/80000, loss=14.8353, lr=0.008112, batch_cost=1.7342, reader_cost=0.0019 | ETA 30:32:28 2020-12-25 20:28:38 [INFO] [TRAIN] epoch=45, iter=16700/80000, loss=15.0370, lr=0.008100, batch_cost=1.7291, reader_cost=0.0027 | ETA 30:24:10 2020-12-25 20:31:40 [INFO] [TRAIN] epoch=46, iter=16800/80000, loss=14.9271, lr=0.008089, batch_cost=1.8194, reader_cost=0.0580 | ETA 31:56:24 2020-12-25 20:34:36 [INFO] [TRAIN] epoch=46, iter=16900/80000, loss=14.9948, lr=0.008077, batch_cost=1.7587, reader_cost=0.0024 | ETA 30:49:32 2020-12-25 20:37:31 [INFO] [TRAIN] epoch=46, iter=17000/80000, loss=14.9487, lr=0.008066, batch_cost=1.7537, reader_cost=0.0017 | ETA 30:41:21 2020-12-25 20:40:26 [INFO] [TRAIN] epoch=46, iter=17100/80000, loss=15.0716, lr=0.008054, batch_cost=1.7439, reader_cost=0.0018 | ETA 30:28:11 2020-12-25 20:43:28 [INFO] [TRAIN] epoch=47, iter=17200/80000, loss=14.9232, lr=0.008042, batch_cost=1.8231, reader_cost=0.0505 | ETA 31:48:10 2020-12-25 20:46:22 [INFO] [TRAIN] epoch=47, iter=17300/80000, loss=14.9309, lr=0.008031, batch_cost=1.7385, reader_cost=0.0015 | ETA 30:16:45 2020-12-25 20:49:16 [INFO] [TRAIN] epoch=47, iter=17400/80000, loss=14.9954, lr=0.008019, batch_cost=1.7320, reader_cost=0.0020 | ETA 30:07:02 2020-12-25 20:52:18 [INFO] [TRAIN] epoch=48, iter=17500/80000, loss=14.9143, lr=0.008008, batch_cost=1.8185, reader_cost=0.0544 | ETA 31:34:17 2020-12-25 20:55:13 [INFO] [TRAIN] epoch=48, iter=17600/80000, loss=14.9517, lr=0.007996, batch_cost=1.7527, reader_cost=0.0023 | ETA 30:22:51 2020-12-25 20:58:08 [INFO] [TRAIN] epoch=48, iter=17700/80000, loss=15.0377, lr=0.007985, batch_cost=1.7443, reader_cost=0.0025 | ETA 30:11:07 2020-12-25 21:01:02 [INFO] [TRAIN] epoch=48, iter=17800/80000, loss=15.0075, lr=0.007973, batch_cost=1.7414, reader_cost=0.0018 | ETA 30:05:14 2020-12-25 21:04:02 [INFO] [TRAIN] epoch=49, iter=17900/80000, loss=15.0407, lr=0.007962, batch_cost=1.7992, reader_cost=0.0508 | ETA 31:02:10 2020-12-25 21:06:56 [INFO] [TRAIN] epoch=49, iter=18000/80000, loss=15.0472, lr=0.007950, batch_cost=1.7357, reader_cost=0.0015 | ETA 29:53:35 2020-12-25 21:06:56 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 21:07:38 [INFO] [EVAL] #Images=500 mIoU=0.7152 Acc=0.9516 Kappa=0.9373 2020-12-25 21:07:38 [INFO] [EVAL] Class IoU: [0.9742 0.8036 0.9127 0.5189 0.5095 0.6438 0.6705 0.7461 0.9162 0.6073 0.9352 0.7909 0.6106 0.9291 0.6149 0.7781 0.4906 0.3986 0.738 ] 2020-12-25 21:07:38 [INFO] [EVAL] Class Acc: [0.9889 0.9008 0.9594 0.6406 0.7848 0.7966 0.8565 0.8727 0.9487 0.7603 0.9513 0.8487 0.7727 0.951 0.9021 0.9033 0.5209 0.6428 0.8069] 2020-12-25 21:07:45 [INFO] [EVAL] The model with the best validation mIoU (0.7152) was saved at iter 18000. 2020-12-25 21:10:38 [INFO] [TRAIN] epoch=49, iter=18100/80000, loss=14.7555, lr=0.007939, batch_cost=1.7307, reader_cost=0.0018 | ETA 29:45:29 2020-12-25 21:13:32 [INFO] [TRAIN] epoch=49, iter=18200/80000, loss=14.9196, lr=0.007927, batch_cost=1.7377, reader_cost=0.0019 | ETA 29:49:51 2020-12-25 21:16:37 [INFO] [TRAIN] epoch=50, iter=18300/80000, loss=14.9455, lr=0.007916, batch_cost=1.8498, reader_cost=0.0613 | ETA 31:42:10 2020-12-25 21:19:33 [INFO] [TRAIN] epoch=50, iter=18400/80000, loss=14.7801, lr=0.007904, batch_cost=1.7614, reader_cost=0.0026 | ETA 30:08:21 2020-12-25 21:22:26 [INFO] [TRAIN] epoch=50, iter=18500/80000, loss=14.9559, lr=0.007892, batch_cost=1.7209, reader_cost=0.0018 | ETA 29:23:55 2020-12-25 21:25:19 [INFO] [TRAIN] epoch=50, iter=18600/80000, loss=14.9205, lr=0.007881, batch_cost=1.7326, reader_cost=0.0016 | ETA 29:33:03 2020-12-25 21:28:21 [INFO] [TRAIN] epoch=51, iter=18700/80000, loss=14.9175, lr=0.007869, batch_cost=1.8185, reader_cost=0.0480 | ETA 30:57:54 2020-12-25 21:31:15 [INFO] [TRAIN] epoch=51, iter=18800/80000, loss=15.0521, lr=0.007858, batch_cost=1.7358, reader_cost=0.0017 | ETA 29:30:31 2020-12-25 21:34:08 [INFO] [TRAIN] epoch=51, iter=18900/80000, loss=14.8385, lr=0.007846, batch_cost=1.7295, reader_cost=0.0020 | ETA 29:21:11 2020-12-25 21:37:07 [INFO] [TRAIN] epoch=52, iter=19000/80000, loss=14.7166, lr=0.007835, batch_cost=1.7954, reader_cost=0.0513 | ETA 30:25:19 2020-12-25 21:40:00 [INFO] [TRAIN] epoch=52, iter=19100/80000, loss=14.8848, lr=0.007823, batch_cost=1.7258, reader_cost=0.0024 | ETA 29:11:40 2020-12-25 21:42:53 [INFO] [TRAIN] epoch=52, iter=19200/80000, loss=14.6887, lr=0.007812, batch_cost=1.7317, reader_cost=0.0025 | ETA 29:14:45 2020-12-25 21:45:45 [INFO] [TRAIN] epoch=52, iter=19300/80000, loss=15.0207, lr=0.007800, batch_cost=1.7195, reader_cost=0.0018 | ETA 28:59:34 2020-12-25 21:48:46 [INFO] [TRAIN] epoch=53, iter=19400/80000, loss=14.8754, lr=0.007788, batch_cost=1.8036, reader_cost=0.0486 | ETA 30:21:37 2020-12-25 21:51:40 [INFO] [TRAIN] epoch=53, iter=19500/80000, loss=15.0204, lr=0.007777, batch_cost=1.7363, reader_cost=0.0017 | ETA 29:10:47 2020-12-25 21:54:33 [INFO] [TRAIN] epoch=53, iter=19600/80000, loss=14.7225, lr=0.007765, batch_cost=1.7355, reader_cost=0.0016 | ETA 29:07:06 2020-12-25 21:57:28 [INFO] [TRAIN] epoch=53, iter=19700/80000, loss=14.9890, lr=0.007754, batch_cost=1.7447, reader_cost=0.0018 | ETA 29:13:26 2020-12-25 22:00:30 [INFO] [TRAIN] epoch=54, iter=19800/80000, loss=14.8422, lr=0.007742, batch_cost=1.8169, reader_cost=0.0513 | ETA 30:22:54 2020-12-25 22:03:23 [INFO] [TRAIN] epoch=54, iter=19900/80000, loss=14.9916, lr=0.007731, batch_cost=1.7324, reader_cost=0.0022 | ETA 28:55:18 2020-12-25 22:06:17 [INFO] [TRAIN] epoch=54, iter=20000/80000, loss=14.7702, lr=0.007719, batch_cost=1.7388, reader_cost=0.0023 | ETA 28:58:47 2020-12-25 22:06:17 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 22:07:00 [INFO] [EVAL] #Images=500 mIoU=0.6804 Acc=0.9248 Kappa=0.9027 2020-12-25 22:07:00 [INFO] [EVAL] Class IoU: [0.9421 0.7545 0.8601 0.3185 0.4496 0.3915 0.6257 0.7591 0.8734 0.4633 0.9419 0.6926 0.5851 0.9192 0.647 0.793 0.6757 0.5118 0.723 ] 2020-12-25 22:07:00 [INFO] [EVAL] Class Acc: [0.9805 0.884 0.901 0.5984 0.5105 0.4754 0.8628 0.9247 0.953 0.8471 0.9646 0.7692 0.8139 0.9642 0.8374 0.8493 0.8008 0.7861 0.7803] 2020-12-25 22:07:04 [INFO] [EVAL] The model with the best validation mIoU (0.7152) was saved at iter 18000. 2020-12-25 22:10:04 [INFO] [TRAIN] epoch=55, iter=20100/80000, loss=14.8773, lr=0.007707, batch_cost=1.7940, reader_cost=0.0507 | ETA 29:51:02 2020-12-25 22:13:00 [INFO] [TRAIN] epoch=55, iter=20200/80000, loss=14.9266, lr=0.007696, batch_cost=1.7650, reader_cost=0.0022 | ETA 29:19:06 2020-12-25 22:15:56 [INFO] [TRAIN] epoch=55, iter=20300/80000, loss=14.7281, lr=0.007684, batch_cost=1.7600, reader_cost=0.0021 | ETA 29:11:10 2020-12-25 22:18:50 [INFO] [TRAIN] epoch=55, iter=20400/80000, loss=14.8771, lr=0.007673, batch_cost=1.7360, reader_cost=0.0024 | ETA 28:44:25 2020-12-25 22:21:52 [INFO] [TRAIN] epoch=56, iter=20500/80000, loss=14.7960, lr=0.007661, batch_cost=1.8128, reader_cost=0.0517 | ETA 29:57:38 2020-12-25 22:24:45 [INFO] [TRAIN] epoch=56, iter=20600/80000, loss=14.8304, lr=0.007650, batch_cost=1.7280, reader_cost=0.0020 | ETA 28:30:40 2020-12-25 22:27:39 [INFO] [TRAIN] epoch=56, iter=20700/80000, loss=14.6952, lr=0.007638, batch_cost=1.7414, reader_cost=0.0021 | ETA 28:41:03 2020-12-25 22:30:34 [INFO] [TRAIN] epoch=56, iter=20800/80000, loss=15.1265, lr=0.007626, batch_cost=1.7498, reader_cost=0.0024 | ETA 28:46:29 2020-12-25 22:33:36 [INFO] [TRAIN] epoch=57, iter=20900/80000, loss=14.9951, lr=0.007615, batch_cost=1.8155, reader_cost=0.0515 | ETA 29:48:16 2020-12-25 22:36:30 [INFO] [TRAIN] epoch=57, iter=21000/80000, loss=14.7203, lr=0.007603, batch_cost=1.7423, reader_cost=0.0013 | ETA 28:33:17 2020-12-25 22:39:23 [INFO] [TRAIN] epoch=57, iter=21100/80000, loss=14.7497, lr=0.007592, batch_cost=1.7320, reader_cost=0.0019 | ETA 28:20:13 2020-12-25 22:42:19 [INFO] [TRAIN] epoch=57, iter=21200/80000, loss=14.8234, lr=0.007580, batch_cost=1.7503, reader_cost=0.0011 | ETA 28:35:15 2020-12-25 22:45:21 [INFO] [TRAIN] epoch=58, iter=21300/80000, loss=14.8460, lr=0.007568, batch_cost=1.8211, reader_cost=0.0498 | ETA 29:41:36 2020-12-25 22:48:14 [INFO] [TRAIN] epoch=58, iter=21400/80000, loss=14.7876, lr=0.007557, batch_cost=1.7283, reader_cost=0.0015 | ETA 28:07:56 2020-12-25 22:51:11 [INFO] [TRAIN] epoch=58, iter=21500/80000, loss=14.7388, lr=0.007545, batch_cost=1.7685, reader_cost=0.0019 | ETA 28:44:19 2020-12-25 22:54:12 [INFO] [TRAIN] epoch=59, iter=21600/80000, loss=14.7134, lr=0.007534, batch_cost=1.8109, reader_cost=0.0495 | ETA 29:22:37 2020-12-25 22:57:05 [INFO] [TRAIN] epoch=59, iter=21700/80000, loss=14.8875, lr=0.007522, batch_cost=1.7290, reader_cost=0.0013 | ETA 28:00:02 2020-12-25 22:59:59 [INFO] [TRAIN] epoch=59, iter=21800/80000, loss=14.7841, lr=0.007510, batch_cost=1.7379, reader_cost=0.0019 | ETA 28:05:48 2020-12-25 23:02:53 [INFO] [TRAIN] epoch=59, iter=21900/80000, loss=14.8668, lr=0.007499, batch_cost=1.7439, reader_cost=0.0020 | ETA 28:08:42 2020-12-25 23:05:55 [INFO] [TRAIN] epoch=60, iter=22000/80000, loss=14.8279, lr=0.007487, batch_cost=1.8183, reader_cost=0.0477 | ETA 29:17:42 2020-12-25 23:05:56 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 23:06:38 [INFO] [EVAL] #Images=500 mIoU=0.7180 Acc=0.9482 Kappa=0.9330 2020-12-25 23:06:38 [INFO] [EVAL] Class IoU: [0.9667 0.7797 0.9092 0.4535 0.5119 0.6466 0.6994 0.7663 0.9116 0.5968 0.9421 0.8059 0.5788 0.9283 0.6546 0.7576 0.4237 0.5495 0.7603] 2020-12-25 23:06:38 [INFO] [EVAL] Class Acc: [0.9934 0.8283 0.9472 0.6791 0.6603 0.7913 0.858 0.9159 0.9563 0.7735 0.9661 0.8669 0.8286 0.9559 0.8188 0.9402 0.4789 0.6419 0.854 ] 2020-12-25 23:06:44 [INFO] [EVAL] The model with the best validation mIoU (0.7180) was saved at iter 22000. 2020-12-25 23:09:38 [INFO] [TRAIN] epoch=60, iter=22100/80000, loss=14.6478, lr=0.007475, batch_cost=1.7366, reader_cost=0.0012 | ETA 27:55:49 2020-12-25 23:12:32 [INFO] [TRAIN] epoch=60, iter=22200/80000, loss=14.7891, lr=0.007464, batch_cost=1.7386, reader_cost=0.0018 | ETA 27:54:51 2020-12-25 23:15:26 [INFO] [TRAIN] epoch=60, iter=22300/80000, loss=14.9463, lr=0.007452, batch_cost=1.7365, reader_cost=0.0018 | ETA 27:49:54 2020-12-25 23:18:26 [INFO] [TRAIN] epoch=61, iter=22400/80000, loss=14.7945, lr=0.007441, batch_cost=1.7977, reader_cost=0.0532 | ETA 28:45:44 2020-12-25 23:21:18 [INFO] [TRAIN] epoch=61, iter=22500/80000, loss=14.6894, lr=0.007429, batch_cost=1.7233, reader_cost=0.0018 | ETA 27:31:32 2020-12-25 23:24:11 [INFO] [TRAIN] epoch=61, iter=22600/80000, loss=14.6926, lr=0.007417, batch_cost=1.7293, reader_cost=0.0015 | ETA 27:34:21 2020-12-25 23:27:12 [INFO] [TRAIN] epoch=62, iter=22700/80000, loss=14.7210, lr=0.007406, batch_cost=1.8015, reader_cost=0.0512 | ETA 28:40:23 2020-12-25 23:30:04 [INFO] [TRAIN] epoch=62, iter=22800/80000, loss=14.8790, lr=0.007394, batch_cost=1.7252, reader_cost=0.0022 | ETA 27:24:44 2020-12-25 23:32:58 [INFO] [TRAIN] epoch=62, iter=22900/80000, loss=14.6698, lr=0.007382, batch_cost=1.7357, reader_cost=0.0019 | ETA 27:31:46 2020-12-25 23:35:53 [INFO] [TRAIN] epoch=62, iter=23000/80000, loss=14.6687, lr=0.007371, batch_cost=1.7491, reader_cost=0.0020 | ETA 27:41:39 2020-12-25 23:38:56 [INFO] [TRAIN] epoch=63, iter=23100/80000, loss=14.7177, lr=0.007359, batch_cost=1.8255, reader_cost=0.0519 | ETA 28:51:10 2020-12-25 23:41:50 [INFO] [TRAIN] epoch=63, iter=23200/80000, loss=14.8613, lr=0.007347, batch_cost=1.7433, reader_cost=0.0023 | ETA 27:30:21 2020-12-25 23:44:44 [INFO] [TRAIN] epoch=63, iter=23300/80000, loss=14.6291, lr=0.007336, batch_cost=1.7370, reader_cost=0.0026 | ETA 27:21:28 2020-12-25 23:47:38 [INFO] [TRAIN] epoch=63, iter=23400/80000, loss=14.5536, lr=0.007324, batch_cost=1.7339, reader_cost=0.0020 | ETA 27:15:39 2020-12-25 23:50:39 [INFO] [TRAIN] epoch=64, iter=23500/80000, loss=14.4989, lr=0.007313, batch_cost=1.8139, reader_cost=0.0508 | ETA 28:28:06 2020-12-25 23:53:33 [INFO] [TRAIN] epoch=64, iter=23600/80000, loss=14.7233, lr=0.007301, batch_cost=1.7351, reader_cost=0.0023 | ETA 27:10:57 2020-12-25 23:56:26 [INFO] [TRAIN] epoch=64, iter=23700/80000, loss=14.4321, lr=0.007289, batch_cost=1.7298, reader_cost=0.0017 | ETA 27:03:05 2020-12-25 23:59:19 [INFO] [TRAIN] epoch=64, iter=23800/80000, loss=14.6838, lr=0.007278, batch_cost=1.7249, reader_cost=0.0021 | ETA 26:55:42 2020-12-26 00:02:20 [INFO] [TRAIN] epoch=65, iter=23900/80000, loss=14.6906, lr=0.007266, batch_cost=1.8066, reader_cost=0.0538 | ETA 28:09:12 2020-12-26 00:05:13 [INFO] [TRAIN] epoch=65, iter=24000/80000, loss=14.6892, lr=0.007254, batch_cost=1.7295, reader_cost=0.0020 | ETA 26:54:11 2020-12-26 00:05:13 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-26 00:05:55 [INFO] [EVAL] #Images=500 mIoU=0.7373 Acc=0.9554 Kappa=0.9422 2020-12-26 00:05:55 [INFO] [EVAL] Class IoU: [0.9795 0.8383 0.9172 0.4325 0.5751 0.6405 0.7047 0.7695 0.9178 0.597 0.9428 0.815 0.6185 0.9329 0.6682 0.7811 0.6372 0.5309 0.7099] 2020-12-26 00:05:55 [INFO] [EVAL] Class Acc: [0.99 0.9226 0.956 0.7757 0.7647 0.8064 0.7934 0.8377 0.9501 0.7522 0.9639 0.9138 0.7156 0.9621 0.7743 0.9722 0.6868 0.7152 0.7533] 2020-12-26 00:06:02 [INFO] [EVAL] The model with the best validation mIoU (0.7373) was saved at iter 24000. 2020-12-26 00:08:56 [INFO] [TRAIN] epoch=65, iter=24100/80000, loss=14.7251, lr=0.007243, batch_cost=1.7388, reader_cost=0.0017 | ETA 26:59:57 2020-12-26 00:11:57 [INFO] [TRAIN] epoch=66, iter=24200/80000, loss=14.5959, lr=0.007231, batch_cost=1.8072, reader_cost=0.0486 | ETA 28:00:40 2020-12-26 00:14:50 [INFO] [TRAIN] epoch=66, iter=24300/80000, loss=14.7331, lr=0.007219, batch_cost=1.7318, reader_cost=0.0019 | ETA 26:47:40 2020-12-26 00:17:44 [INFO] [TRAIN] epoch=66, iter=24400/80000, loss=14.5706, lr=0.007208, batch_cost=1.7390, reader_cost=0.0018 | ETA 26:51:25 2020-12-26 00:20:38 [INFO] [TRAIN] epoch=66, iter=24500/80000, loss=14.6452, lr=0.007196, batch_cost=1.7400, reader_cost=0.0020 | ETA 26:49:27 2020-12-26 00:23:39 [INFO] [TRAIN] epoch=67, iter=24600/80000, loss=14.6268, lr=0.007184, batch_cost=1.8107, reader_cost=0.0523 | ETA 27:51:53 2020-12-26 00:26:36 [INFO] [TRAIN] epoch=67, iter=24700/80000, loss=14.4693, lr=0.007173, batch_cost=1.7635, reader_cost=0.0017 | ETA 27:05:19 2020-12-26 00:29:30 [INFO] [TRAIN] epoch=67, iter=24800/80000, loss=14.3577, lr=0.007161, batch_cost=1.7452, reader_cost=0.0020 | ETA 26:45:35 2020-12-26 00:32:23 [INFO] [TRAIN] epoch=67, iter=24900/80000, loss=14.4918, lr=0.007149, batch_cost=1.7284, reader_cost=0.0014 | ETA 26:27:13 2020-12-26 00:35:24 [INFO] [TRAIN] epoch=68, iter=25000/80000, loss=14.5976, lr=0.007138, batch_cost=1.8063, reader_cost=0.0487 | ETA 27:35:46 2020-12-26 00:38:18 [INFO] [TRAIN] epoch=68, iter=25100/80000, loss=14.7478, lr=0.007126, batch_cost=1.7317, reader_cost=0.0007 | ETA 26:24:32 2020-12-26 00:41:11 [INFO] [TRAIN] epoch=68, iter=25200/80000, loss=14.4431, lr=0.007114, batch_cost=1.7356, reader_cost=0.0011 | ETA 26:25:08 2020-12-26 00:44:16 [INFO] [TRAIN] epoch=69, iter=25300/80000, loss=14.5686, lr=0.007103, batch_cost=1.8449, reader_cost=0.0534 | ETA 28:01:54 2020-12-26 00:47:11 [INFO] [TRAIN] epoch=69, iter=25400/80000, loss=14.7431, lr=0.007091, batch_cost=1.7481, reader_cost=0.0011 | ETA 26:30:47 2020-12-26 00:50:06 [INFO] [TRAIN] epoch=69, iter=25500/80000, loss=14.5782, lr=0.007079, batch_cost=1.7508, reader_cost=0.0011 | ETA 26:30:20 2020-12-26 00:53:01 [INFO] [TRAIN] epoch=69, iter=25600/80000, loss=14.5818, lr=0.007067, batch_cost=1.7493, reader_cost=0.0014 | ETA 26:26:01 2020-12-26 00:56:02 [INFO] [TRAIN] epoch=70, iter=25700/80000, loss=14.5803, lr=0.007056, batch_cost=1.8117, reader_cost=0.0522 | ETA 27:19:34 2020-12-26 00:58:57 [INFO] [TRAIN] epoch=70, iter=25800/80000, loss=14.5652, lr=0.007044, batch_cost=1.7474, reader_cost=0.0006 | ETA 26:18:31 2020-12-26 01:01:52 [INFO] [TRAIN] epoch=70, iter=25900/80000, loss=14.5951, lr=0.007032, batch_cost=1.7418, reader_cost=0.0012 | ETA 26:10:30 2020-12-26 01:04:45 [INFO] [TRAIN] epoch=70, iter=26000/80000, loss=14.4976, lr=0.007021, batch_cost=1.7336, reader_cost=0.0011 | ETA 26:00:13 2020-12-26 01:04:45 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-26 01:05:28 [INFO] [EVAL] #Images=500 mIoU=0.7132 Acc=0.9563 Kappa=0.9433 2020-12-26 01:05:28 [INFO] [EVAL] Class IoU: [0.9791 0.8294 0.9198 0.4718 0.5639 0.6661 0.7188 0.7917 0.9223 0.6295 0.9431 0.8143 0.5818 0.9282 0.5764 0.6959 0.135 0.6126 0.7701] 2020-12-26 01:05:28 [INFO] [EVAL] Class Acc: [0.9861 0.9191 0.9569 0.7726 0.8362 0.8065 0.8456 0.9081 0.9561 0.7997 0.9598 0.869 0.8314 0.9484 0.9119 0.7618 0.8123 0.8389 0.8417] 2020-12-26 01:05:32 [INFO] [EVAL] The model with the best validation mIoU (0.7373) was saved at iter 24000. 2020-12-26 01:08:33 [INFO] [TRAIN] epoch=71, iter=26100/80000, loss=14.5088, lr=0.007009, batch_cost=1.8076, reader_cost=0.0486 | ETA 27:03:48 2020-12-26 01:11:27 [INFO] [TRAIN] epoch=71, iter=26200/80000, loss=14.4731, lr=0.006997, batch_cost=1.7407, reader_cost=0.0014 | ETA 26:00:49 2020-12-26 01:14:25 [INFO] [TRAIN] epoch=71, iter=26300/80000, loss=14.4436, lr=0.006986, batch_cost=1.7775, reader_cost=0.0017 | ETA 26:30:51 2020-12-26 01:17:19 [INFO] [TRAIN] epoch=71, iter=26400/80000, loss=14.6577, lr=0.006974, batch_cost=1.7444, reader_cost=0.0017 | ETA 25:58:17 2020-12-26 01:20:21 [INFO] [TRAIN] epoch=72, iter=26500/80000, loss=14.6511, lr=0.006962, batch_cost=1.8115, reader_cost=0.0520 | ETA 26:55:12 2020-12-26 01:23:14 [INFO] [TRAIN] epoch=72, iter=26600/80000, loss=14.6312, lr=0.006950, batch_cost=1.7325, reader_cost=0.0014 | ETA 25:41:58 2020-12-26 01:26:08 [INFO] [TRAIN] epoch=72, iter=26700/80000, loss=14.3748, lr=0.006939, batch_cost=1.7367, reader_cost=0.0008 | ETA 25:42:44 2020-12-26 01:29:10 [INFO] [TRAIN] epoch=73, iter=26800/80000, loss=14.5181, lr=0.006927, batch_cost=1.8182, reader_cost=0.0475 | ETA 26:52:08 2020-12-26 01:32:03 [INFO] [TRAIN] epoch=73, iter=26900/80000, loss=14.5947, lr=0.006915, batch_cost=1.7308, reader_cost=0.0009 | ETA 25:31:46 2020-12-26 01:34:56 [INFO] [TRAIN] epoch=73, iter=27000/80000, loss=14.4822, lr=0.006904, batch_cost=1.7296, reader_cost=0.0009 | ETA 25:27:50 2020-12-26 01:37:50 [INFO] [TRAIN] epoch=73, iter=27100/80000, loss=14.4855, lr=0.006892, batch_cost=1.7365, reader_cost=0.0013 | ETA 25:30:59 2020-12-26 01:40:51 [INFO] [TRAIN] epoch=74, iter=27200/80000, loss=14.4977, lr=0.006880, batch_cost=1.8071, reader_cost=0.0485 | ETA 26:30:12 2020-12-26 01:43:44 [INFO] [TRAIN] epoch=74, iter=27300/80000, loss=14.5143, lr=0.006868, batch_cost=1.7325, reader_cost=0.0022 | ETA 25:21:44 2020-12-26 01:46:38 [INFO] [TRAIN] epoch=74, iter=27400/80000, loss=14.2790, lr=0.006857, batch_cost=1.7360, reader_cost=0.0013 | ETA 25:21:51 2020-12-26 01:49:31 [INFO] [TRAIN] epoch=74, iter=27500/80000, loss=14.5072, lr=0.006845, batch_cost=1.7313, reader_cost=0.0011 | ETA 25:14:53 2020-12-26 01:52:33 [INFO] [TRAIN] epoch=75, iter=27600/80000, loss=14.6675, lr=0.006833, batch_cost=1.8163, reader_cost=0.0509 | ETA 26:26:11 2020-12-26 01:55:26 [INFO] [TRAIN] epoch=75, iter=27700/80000, loss=14.1986, lr=0.006821, batch_cost=1.7353, reader_cost=0.0010 | ETA 25:12:37 2020-12-26 01:58:21 [INFO] [TRAIN] epoch=75, iter=27800/80000, loss=14.2928, lr=0.006810, batch_cost=1.7427, reader_cost=0.0015 | ETA 25:16:09 2020-12-26 02:01:15 [INFO] [TRAIN] epoch=75, iter=27900/80000, loss=14.5346, lr=0.006798, batch_cost=1.7385, reader_cost=0.0014 | ETA 25:09:34 2020-12-26 02:04:17 [INFO] [TRAIN] epoch=76, iter=28000/80000, loss=14.4097, lr=0.006786, batch_cost=1.8245, reader_cost=0.0488 | ETA 26:21:11 2020-12-26 02:04:17 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-26 02:05:00 [INFO] [EVAL] #Images=500 mIoU=0.7088 Acc=0.9530 Kappa=0.9390 2020-12-26 02:05:00 [INFO] [EVAL] Class IoU: [0.9772 0.8197 0.9165 0.4493 0.4829 0.6494 0.6973 0.7847 0.9146 0.5803 0.9394 0.7958 0.5486 0.942 0.6975 0.7274 0.3286 0.452 0.7639] 2020-12-26 02:05:00 [INFO] [EVAL] Class Acc: [0.9889 0.8907 0.9566 0.6954 0.5591 0.7971 0.8109 0.9151 0.9579 0.8082 0.9592 0.8389 0.8349 0.9667 0.8758 0.8443 0.7121 0.8975 0.8923] 2020-12-26 02:05:04 [INFO] [EVAL] The model with the best validation mIoU (0.7373) was saved at iter 24000. 2020-12-26 02:07:58 [INFO] [TRAIN] epoch=76, iter=28100/80000, loss=14.5022, lr=0.006774, batch_cost=1.7341, reader_cost=0.0012 | ETA 24:59:59 2020-12-26 02:10:50 [INFO] [TRAIN] epoch=76, iter=28200/80000, loss=14.5411, lr=0.006763, batch_cost=1.7278, reader_cost=0.0009 | ETA 24:51:41 2020-12-26 02:13:52 [INFO] [TRAIN] epoch=77, iter=28300/80000, loss=14.5110, lr=0.006751, batch_cost=1.8183, reader_cost=0.0508 | ETA 26:06:45 2020-12-26 02:16:46 [INFO] [TRAIN] epoch=77, iter=28400/80000, loss=14.4616, lr=0.006739, batch_cost=1.7330, reader_cost=0.0018 | ETA 24:50:24 2020-12-26 02:19:39 [INFO] [TRAIN] epoch=77, iter=28500/80000, loss=14.0613, lr=0.006727, batch_cost=1.7354, reader_cost=0.0012 | ETA 24:49:35 2020-12-26 02:22:34 [INFO] [TRAIN] epoch=77, iter=28600/80000, loss=14.4222, lr=0.006716, batch_cost=1.7482, reader_cost=0.0014 | ETA 24:57:36 2020-12-26 02:25:37 [INFO] [TRAIN] epoch=78, iter=28700/80000, loss=14.3717, lr=0.006704, batch_cost=1.8204, reader_cost=0.0521 | ETA 25:56:24 2020-12-26 02:28:31 [INFO] [TRAIN] epoch=78, iter=28800/80000, loss=14.2763, lr=0.006692, batch_cost=1.7384, reader_cost=0.0018 | ETA 24:43:26 2020-12-26 02:31:26 [INFO] [TRAIN] epoch=78, iter=28900/80000, loss=14.2819, lr=0.006680, batch_cost=1.7525, reader_cost=0.0023 | ETA 24:52:31 2020-12-26 02:34:19 [INFO] [TRAIN] epoch=78, iter=29000/80000, loss=14.5522, lr=0.006669, batch_cost=1.7326, reader_cost=0.0019 | ETA 24:32:42 2020-12-26 02:37:22 [INFO] [TRAIN] epoch=79, iter=29100/80000, loss=14.3043, lr=0.006657, batch_cost=1.8228, reader_cost=0.0509 | ETA 25:46:22 2020-12-26 02:40:15 [INFO] [TRAIN] epoch=79, iter=29200/80000, loss=14.2152, lr=0.006645, batch_cost=1.7333, reader_cost=0.0014 | ETA 24:27:33 2020-12-26 02:43:09 [INFO] [TRAIN] epoch=79, iter=29300/80000, loss=14.3258, lr=0.006633, batch_cost=1.7361, reader_cost=0.0015 | ETA 24:26:57 2020-12-26 02:46:11 [INFO] [TRAIN] epoch=80, iter=29400/80000, loss=14.4404, lr=0.006622, batch_cost=1.8163, reader_cost=0.0501 | ETA 25:31:43 2020-12-26 02:49:04 [INFO] [TRAIN] epoch=80, iter=29500/80000, loss=14.2902, lr=0.006610, batch_cost=1.7291, reader_cost=0.0019 | ETA 24:15:18 2020-12-26 02:51:57 [INFO] [TRAIN] epoch=80, iter=29600/80000, loss=14.1162, lr=0.006598, batch_cost=1.7301, reader_cost=0.0021 | ETA 24:13:15 2020-12-26 02:54:50 [INFO] [TRAIN] epoch=80, iter=29700/80000, loss=14.3332, lr=0.006586, batch_cost=1.7320, reader_cost=0.0021 | ETA 24:12:01 2020-12-26 02:57:52 [INFO] [TRAIN] epoch=81, iter=29800/80000, loss=14.3189, lr=0.006574, batch_cost=1.8141, reader_cost=0.0499 | ETA 25:17:46 2020-12-26 03:00:45 [INFO] [TRAIN] epoch=81, iter=29900/80000, loss=14.2311, lr=0.006563, batch_cost=1.7302, reader_cost=0.0018 | ETA 24:04:44 2020-12-26 03:03:39 [INFO] [TRAIN] epoch=81, iter=30000/80000, loss=14.0816, lr=0.006551, batch_cost=1.7409, reader_cost=0.0021 | ETA 24:10:45 2020-12-26 03:03:39 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-26 03:04:22 [INFO] [EVAL] #Images=500 mIoU=0.6890 Acc=0.9456 Kappa=0.9294 2020-12-26 03:04:22 [INFO] [EVAL] Class IoU: [0.9691 0.7751 0.9035 0.3437 0.4927 0.6484 0.6269 0.7705 0.9034 0.528 0.9398 0.7809 0.5469 0.9286 0.5501 0.698 0.4302 0.5047 0.7504] 2020-12-26 03:04:22 [INFO] [EVAL] Class Acc: [0.9825 0.8741 0.9334 0.4659 0.8215 0.7957 0.8645 0.906 0.9663 0.7922 0.9495 0.8966 0.8475 0.954 0.8028 0.7702 0.9011 0.8042 0.8211] 2020-12-26 03:04:26 [INFO] [EVAL] The model with the best validation mIoU (0.7373) was saved at iter 24000. 2020-12-26 03:07:19 [INFO] [TRAIN] epoch=81, iter=30100/80000, loss=14.4004, lr=0.006539, batch_cost=1.7284, reader_cost=0.0012 | ETA 23:57:27 2020-12-26 03:10:21 [INFO] [TRAIN] epoch=82, iter=30200/80000, loss=14.2930, lr=0.006527, batch_cost=1.8179, reader_cost=0.0507 | ETA 25:08:49 2020-12-26 03:13:14 [INFO] [TRAIN] epoch=82, iter=30300/80000, loss=14.1575, lr=0.006515, batch_cost=1.7286, reader_cost=0.0015 | ETA 23:51:53 2020-12-26 03:16:07 [INFO] [TRAIN] epoch=82, iter=30400/80000, loss=14.1870, lr=0.006504, batch_cost=1.7340, reader_cost=0.0014 | ETA 23:53:27 2020-12-26 03:19:01 [INFO] [TRAIN] epoch=82, iter=30500/80000, loss=14.2640, lr=0.006492, batch_cost=1.7294, reader_cost=0.0013 | ETA 23:46:47 2020-12-26 03:22:02 [INFO] [TRAIN] epoch=83, iter=30600/80000, loss=14.0543, lr=0.006480, batch_cost=1.8160, reader_cost=0.0513 | ETA 24:55:08 2020-12-26 03:24:55 [INFO] [TRAIN] epoch=83, iter=30700/80000, loss=14.2348, lr=0.006468, batch_cost=1.7294, reader_cost=0.0017 | ETA 23:40:59 2020-12-26 03:27:49 [INFO] [TRAIN] epoch=83, iter=30800/80000, loss=14.2879, lr=0.006456, batch_cost=1.7368, reader_cost=0.0015 | ETA 23:44:12 2020-12-26 03:30:51 [INFO] [TRAIN] epoch=84, iter=30900/80000, loss=14.2365, lr=0.006445, batch_cost=1.8190, reader_cost=0.0504 | ETA 24:48:31 2020-12-26 03:33:44 [INFO] [TRAIN] epoch=84, iter=31000/80000, loss=14.2849, lr=0.006433, batch_cost=1.7271, reader_cost=0.0017 | ETA 23:30:26 2020-12-26 03:36:38 [INFO] [TRAIN] epoch=84, iter=31100/80000, loss=13.9634, lr=0.006421, batch_cost=1.7344, reader_cost=0.0018 | ETA 23:33:30 2020-12-26 03:39:32 [INFO] [TRAIN] epoch=84, iter=31200/80000, loss=14.1371, lr=0.006409, batch_cost=1.7416, reader_cost=0.0019 | ETA 23:36:28 2020-12-26 03:42:33 [INFO] [TRAIN] epoch=85, iter=31300/80000, loss=14.2905, lr=0.006397, batch_cost=1.8123, reader_cost=0.0502 | ETA 24:31:01 2020-12-26 03:45:26 [INFO] [TRAIN] epoch=85, iter=31400/80000, loss=14.0459, lr=0.006386, batch_cost=1.7279, reader_cost=0.0016 | ETA 23:19:34 2020-12-26 03:48:20 [INFO] [TRAIN] epoch=85, iter=31500/80000, loss=14.1998, lr=0.006374, batch_cost=1.7357, reader_cost=0.0020 | ETA 23:23:03 2020-12-26 03:51:13 [INFO] [TRAIN] epoch=85, iter=31600/80000, loss=14.4493, lr=0.006362, batch_cost=1.7296, reader_cost=0.0016 | ETA 23:15:12 2020-12-26 03:54:15 [INFO] [TRAIN] epoch=86, iter=31700/80000, loss=14.2300, lr=0.006350, batch_cost=1.8165, reader_cost=0.0496 | ETA 24:22:17 2020-12-26 03:57:08 [INFO] [TRAIN] epoch=86, iter=31800/80000, loss=14.0469, lr=0.006338, batch_cost=1.7329, reader_cost=0.0013 | ETA 23:12:06 2020-12-26 04:00:01 [INFO] [TRAIN] epoch=86, iter=31900/80000, loss=13.9624, lr=0.006326, batch_cost=1.7240, reader_cost=0.0016 | ETA 23:02:05 2020-12-26 04:03:02 [INFO] [TRAIN] epoch=87, iter=32000/80000, loss=14.3500, lr=0.006315, batch_cost=1.8104, reader_cost=0.0524 | ETA 24:08:19 2020-12-26 04:03:02 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-26 04:03:44 [INFO] [EVAL] #Images=500 mIoU=0.7288 Acc=0.9531 Kappa=0.9390 2020-12-26 04:03:44 [INFO] [EVAL] Class IoU: [0.9767 0.8181 0.9093 0.3803 0.5739 0.6604 0.6924 0.7825 0.9117 0.6226 0.9377 0.8037 0.6164 0.932 0.6399 0.6687 0.5932 0.5551 0.7724] 2020-12-26 04:03:44 [INFO] [EVAL] Class Acc: [0.9863 0.9095 0.9431 0.8024 0.8265 0.7837 0.8206 0.8966 0.9444 0.8036 0.9797 0.8977 0.7939 0.9659 0.7477 0.9521 0.9361 0.8686 0.856 ] 2020-12-26 04:03:49 [INFO] [EVAL] The model with the best validation mIoU (0.7373) was saved at iter 24000. 2020-12-26 04:06:41 [INFO] [TRAIN] epoch=87, iter=32100/80000, loss=13.7800, lr=0.006303, batch_cost=1.7281, reader_cost=0.0023 | ETA 22:59:34 2020-12-26 04:09:34 [INFO] [TRAIN] epoch=87, iter=32200/80000, loss=14.1696, lr=0.006291, batch_cost=1.7287, reader_cost=0.0015 | ETA 22:57:12 2020-12-26 04:12:29 [INFO] [TRAIN] epoch=87, iter=32300/80000, loss=14.0758, lr=0.006279, batch_cost=1.7429, reader_cost=0.0018 | ETA 23:05:37 2020-12-26 04:15:30 [INFO] [TRAIN] epoch=88, iter=32400/80000, loss=14.0917, lr=0.006267, batch_cost=1.8142, reader_cost=0.0498 | ETA 23:59:15 2020-12-26 04:18:24 [INFO] [TRAIN] epoch=88, iter=32500/80000, loss=13.8781, lr=0.006255, batch_cost=1.7335, reader_cost=0.0018 | ETA 22:52:21 2020-12-26 04:21:17 [INFO] [TRAIN] epoch=88, iter=32600/80000, loss=13.7882, lr=0.006243, batch_cost=1.7312, reader_cost=0.0016 | ETA 22:47:40 2020-12-26 04:24:10 [INFO] [TRAIN] epoch=88, iter=32700/80000, loss=13.9472, lr=0.006232, batch_cost=1.7304, reader_cost=0.0018 | ETA 22:44:09 2020-12-26 04:27:12 [INFO] [TRAIN] epoch=89, iter=32800/80000, loss=14.1505, lr=0.006220, batch_cost=1.8122, reader_cost=0.0487 | ETA 23:45:34 2020-12-26 04:30:05 [INFO] [TRAIN] epoch=89, iter=32900/80000, loss=14.0696, lr=0.006208, batch_cost=1.7269, reader_cost=0.0019 | ETA 22:35:36 2020-12-26 04:32:59 [INFO] [TRAIN] epoch=89, iter=33000/80000, loss=13.8867, lr=0.006196, batch_cost=1.7408, reader_cost=0.0017 | ETA 22:43:36 2020-12-26 04:35:52 [INFO] [TRAIN] epoch=89, iter=33100/80000, loss=13.8502, lr=0.006184, batch_cost=1.7315, reader_cost=0.0015 | ETA 22:33:27 2020-12-26 04:38:53 [INFO] [TRAIN] epoch=90, iter=33200/80000, loss=14.0487, lr=0.006172, batch_cost=1.8099, reader_cost=0.0503 | ETA 23:31:41 2020-12-26 04:41:46 [INFO] [TRAIN] epoch=90, iter=33300/80000, loss=13.8320, lr=0.006160, batch_cost=1.7316, reader_cost=0.0023 | ETA 22:27:44 2020-12-26 04:44:40 [INFO] [TRAIN] epoch=90, iter=33400/80000, loss=13.8420, lr=0.006149, batch_cost=1.7343, reader_cost=0.0023 | ETA 22:26:59 2020-12-26 04:47:42 [INFO] [TRAIN] epoch=91, iter=33500/80000, loss=13.9510, lr=0.006137, batch_cost=1.8219, reader_cost=0.0503 | ETA 23:31:56 2020-12-26 04:50:35 [INFO] [TRAIN] epoch=91, iter=33600/80000, loss=13.7514, lr=0.006125, batch_cost=1.7262, reader_cost=0.0021 | ETA 22:14:55 2020-12-26 04:53:28 [INFO] [TRAIN] epoch=91, iter=33700/80000, loss=13.8451, lr=0.006113, batch_cost=1.7310, reader_cost=0.0024 | ETA 22:15:43 2020-12-26 04:56:22 [INFO] [TRAIN] epoch=91, iter=33800/80000, loss=13.9807, lr=0.006101, batch_cost=1.7339, reader_cost=0.0022 | ETA 22:15:06 2020-12-26 04:59:23 [INFO] [TRAIN] epoch=92, iter=33900/80000, loss=13.7085, lr=0.006089, batch_cost=1.8089, reader_cost=0.0529 | ETA 23:09:49 2020-12-26 05:02:16 [INFO] [TRAIN] epoch=92, iter=34000/80000, loss=13.8681, lr=0.006077, batch_cost=1.7281, reader_cost=0.0020 | ETA 22:04:52 2020-12-26 05:02:16 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-26 05:02:59 [INFO] [EVAL] #Images=500 mIoU=0.7398 Acc=0.9539 Kappa=0.9402 2020-12-26 05:02:59 [INFO] [EVAL] Class IoU: [0.9744 0.8112 0.9174 0.4036 0.545 0.6614 0.7233 0.7951 0.915 0.6 0.9401 0.8143 0.6357 0.9405 0.6696 0.7899 0.6418 0.5311 0.7464] 2020-12-26 05:02:59 [INFO] [EVAL] Class Acc: [0.9844 0.904 0.9469 0.7286 0.7348 0.8183 0.8098 0.895 0.9633 0.7516 0.9559 0.9125 0.7707 0.9709 0.8217 0.8902 0.9012 0.7085 0.7981] 2020-12-26 05:03:05 [INFO] [EVAL] The model with the best validation mIoU (0.7398) was saved at iter 34000. 2020-12-26 05:05:58 [INFO] [TRAIN] epoch=92, iter=34100/80000, loss=13.6688, lr=0.006065, batch_cost=1.7274, reader_cost=0.0020 | ETA 22:01:26 2020-12-26 05:08:51 [INFO] [TRAIN] epoch=92, iter=34200/80000, loss=14.0007, lr=0.006053, batch_cost=1.7348, reader_cost=0.0014 | ETA 22:04:14 2020-12-26 05:11:53 [INFO] [TRAIN] epoch=93, iter=34300/80000, loss=13.8995, lr=0.006042, batch_cost=1.8165, reader_cost=0.0488 | ETA 23:03:32 2020-12-26 05:14:46 [INFO] [TRAIN] epoch=93, iter=34400/80000, loss=13.7053, lr=0.006030, batch_cost=1.7325, reader_cost=0.0015 | ETA 21:56:42 2020-12-26 05:17:39 [INFO] [TRAIN] epoch=93, iter=34500/80000, loss=13.8851, lr=0.006018, batch_cost=1.7289, reader_cost=0.0014 | ETA 21:51:03 2020-12-26 05:20:41 [INFO] [TRAIN] epoch=94, iter=34600/80000, loss=14.3368, lr=0.006006, batch_cost=1.8166, reader_cost=0.0478 | ETA 22:54:34 2020-12-26 05:23:34 [INFO] [TRAIN] epoch=94, iter=34700/80000, loss=13.9311, lr=0.005994, batch_cost=1.7289, reader_cost=0.0013 | ETA 21:45:18 2020-12-26 05:26:28 [INFO] [TRAIN] epoch=94, iter=34800/80000, loss=13.7162, lr=0.005982, batch_cost=1.7327, reader_cost=0.0014 | ETA 21:45:17 2020-12-26 05:29:21 [INFO] [TRAIN] epoch=94, iter=34900/80000, loss=13.7227, lr=0.005970, batch_cost=1.7348, reader_cost=0.0013 | ETA 21:43:59 2020-12-26 05:32:24 [INFO] [TRAIN] epoch=95, iter=35000/80000, loss=13.9321, lr=0.005958, batch_cost=1.8216, reader_cost=0.0488 | ETA 22:46:13 2020-12-26 05:35:17 [INFO] [TRAIN] epoch=95, iter=35100/80000, loss=13.7738, lr=0.005946, batch_cost=1.7282, reader_cost=0.0016 | ETA 21:33:16 2020-12-26 05:38:10 [INFO] [TRAIN] epoch=95, iter=35200/80000, loss=13.6161, lr=0.005934, batch_cost=1.7347, reader_cost=0.0011 | ETA 21:35:16 2020-12-26 05:41:03 [INFO] [TRAIN] epoch=95, iter=35300/80000, loss=13.7336, lr=0.005922, batch_cost=1.7305, reader_cost=0.0017 | ETA 21:29:14 2020-12-26 05:44:05 [INFO] [TRAIN] epoch=96, iter=35400/80000, loss=13.8774, lr=0.005911, batch_cost=1.8122, reader_cost=0.0515 | ETA 22:27:02 2020-12-26 05:46:59 [INFO] [TRAIN] epoch=96, iter=35500/80000, loss=13.7426, lr=0.005899, batch_cost=1.7366, reader_cost=0.0014 | ETA 21:27:59 2020-12-26 05:49:52 [INFO] [TRAIN] epoch=96, iter=35600/80000, loss=13.6586, lr=0.005887, batch_cost=1.7299, reader_cost=0.0012 | ETA 21:20:07 2020-12-26 05:52:45 [INFO] [TRAIN] epoch=96, iter=35700/80000, loss=13.9270, lr=0.005875, batch_cost=1.7276, reader_cost=0.0011 | ETA 21:15:34 2020-12-26 05:55:46 [INFO] [TRAIN] epoch=97, iter=35800/80000, loss=13.5910, lr=0.005863, batch_cost=1.8109, reader_cost=0.0472 | ETA 22:14:03 2020-12-26 05:58:39 [INFO] [TRAIN] epoch=97, iter=35900/80000, loss=13.6750, lr=0.005851, batch_cost=1.7297, reader_cost=0.0014 | ETA 21:11:20 2020-12-26 06:01:33 [INFO] [TRAIN] epoch=97, iter=36000/80000, loss=13.6161, lr=0.005839, batch_cost=1.7359, reader_cost=0.0021 | ETA 21:13:01 2020-12-26 06:01:33 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-26 06:02:16 [INFO] [EVAL] #Images=500 mIoU=0.7643 Acc=0.9543 Kappa=0.9408 2020-12-26 06:02:16 [INFO] [EVAL] Class IoU: [0.9758 0.8214 0.9201 0.5281 0.5897 0.6655 0.7139 0.8 0.9043 0.5123 0.9438 0.8038 0.6511 0.9435 0.7668 0.8694 0.7666 0.5711 0.7748] 2020-12-26 06:02:16 [INFO] [EVAL] Class Acc: [0.9886 0.8889 0.9494 0.7648 0.7752 0.8389 0.835 0.911 0.9678 0.5817 0.9605 0.859 0.7728 0.9754 0.8469 0.9632 0.844 0.6986 0.8592] 2020-12-26 06:02:22 [INFO] [EVAL] The model with the best validation mIoU (0.7643) was saved at iter 36000. 2020-12-26 06:05:25 [INFO] [TRAIN] epoch=98, iter=36100/80000, loss=13.9058, lr=0.005827, batch_cost=1.8252, reader_cost=0.0540 | ETA 22:15:28 2020-12-26 06:08:19 [INFO] [TRAIN] epoch=98, iter=36200/80000, loss=13.5897, lr=0.005815, batch_cost=1.7356, reader_cost=0.0016 | ETA 21:06:59 2020-12-26 06:11:12 [INFO] [TRAIN] epoch=98, iter=36300/80000, loss=13.5777, lr=0.005803, batch_cost=1.7341, reader_cost=0.0021 | ETA 21:03:02 2020-12-26 06:14:06 [INFO] [TRAIN] epoch=98, iter=36400/80000, loss=13.7714, lr=0.005791, batch_cost=1.7354, reader_cost=0.0023 | ETA 21:01:04 2020-12-26 06:17:08 [INFO] [TRAIN] epoch=99, iter=36500/80000, loss=13.6831, lr=0.005779, batch_cost=1.8164, reader_cost=0.0507 | ETA 21:56:53 2020-12-26 06:20:01 [INFO] [TRAIN] epoch=99, iter=36600/80000, loss=13.5890, lr=0.005767, batch_cost=1.7363, reader_cost=0.0018 | ETA 20:55:55 2020-12-26 06:22:55 [INFO] [TRAIN] epoch=99, iter=36700/80000, loss=13.3794, lr=0.005755, batch_cost=1.7367, reader_cost=0.0020 | ETA 20:53:19 2020-12-26 06:25:49 [INFO] [TRAIN] epoch=99, iter=36800/80000, loss=13.8619, lr=0.005743, batch_cost=1.7342, reader_cost=0.0020 | ETA 20:48:37 2020-12-26 06:28:51 [INFO] [TRAIN] epoch=100, iter=36900/80000, loss=13.5921, lr=0.005731, batch_cost=1.8188, reader_cost=0.0501 | ETA 21:46:29 2020-12-26 06:31:45 [INFO] [TRAIN] epoch=100, iter=37000/80000, loss=13.4421, lr=0.005719, batch_cost=1.7353, reader_cost=0.0021 | ETA 20:43:38 2020-12-26 06:34:38 [INFO] [TRAIN] epoch=100, iter=37100/80000, loss=13.6843, lr=0.005707, batch_cost=1.7289, reader_cost=0.0026 | ETA 20:36:09 2020-12-26 06:37:30 [INFO] [TRAIN] epoch=100, iter=37200/80000, loss=13.8718, lr=0.005695, batch_cost=1.7253, reader_cost=0.0024 | ETA 20:30:41 2020-12-26 06:40:32 [INFO] [TRAIN] epoch=101, iter=37300/80000, loss=13.5085, lr=0.005683, batch_cost=1.8175, reader_cost=0.0502 | ETA 21:33:25 2020-12-26 06:43:26 [INFO] [TRAIN] epoch=101, iter=37400/80000, loss=13.5820, lr=0.005671, batch_cost=1.7393, reader_cost=0.0021 | ETA 20:34:53 2020-12-26 06:46:20 [INFO] [TRAIN] epoch=101, iter=37500/80000, loss=13.5065, lr=0.005660, batch_cost=1.7351, reader_cost=0.0015 | ETA 20:29:03 2020-12-26 06:49:22 [INFO] [TRAIN] epoch=102, iter=37600/80000, loss=13.6749, lr=0.005648, batch_cost=1.8146, reader_cost=0.0514 | ETA 21:22:21 2020-12-26 06:52:15 [INFO] [TRAIN] epoch=102, iter=37700/80000, loss=13.5823, lr=0.005636, batch_cost=1.7324, reader_cost=0.0024 | ETA 20:21:19 2020-12-26 06:55:08 [INFO] [TRAIN] epoch=102, iter=37800/80000, loss=13.3927, lr=0.005624, batch_cost=1.7326, reader_cost=0.0017 | ETA 20:18:36 2020-12-26 06:58:02 [INFO] [TRAIN] epoch=102, iter=37900/80000, loss=13.5717, lr=0.005612, batch_cost=1.7360, reader_cost=0.0015 | ETA 20:18:04 2020-12-26 07:01:05 [INFO] [TRAIN] epoch=103, iter=38000/80000, loss=13.5417, lr=0.005600, batch_cost=1.8220, reader_cost=0.0519 | ETA 21:15:25 2020-12-26 07:01:05 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-26 07:01:48 [INFO] [EVAL] #Images=500 mIoU=0.7311 Acc=0.9527 Kappa=0.9388 2020-12-26 07:01:48 [INFO] [EVAL] Class IoU: [0.9774 0.822 0.9178 0.3964 0.5778 0.6573 0.6996 0.7988 0.9076 0.547 0.9413 0.808 0.6353 0.9179 0.5869 0.6386 0.727 0.5649 0.7698] 2020-12-26 07:01:48 [INFO] [EVAL] Class Acc: [0.9945 0.8711 0.9531 0.7981 0.7752 0.8043 0.8344 0.9358 0.9492 0.6674 0.9621 0.8982 0.7752 0.9332 0.8647 0.954 0.9051 0.7505 0.855 ] 2020-12-26 07:01:52 [INFO] [EVAL] The model with the best validation mIoU (0.7643) was saved at iter 36000. 2020-12-26 07:04:44 [INFO] [TRAIN] epoch=103, iter=38100/80000, loss=13.4440, lr=0.005588, batch_cost=1.7267, reader_cost=0.0017 | ETA 20:05:46 2020-12-26 07:07:38 [INFO] [TRAIN] epoch=103, iter=38200/80000, loss=13.4697, lr=0.005576, batch_cost=1.7311, reader_cost=0.0019 | ETA 20:06:00 2020-12-26 07:10:31 [INFO] [TRAIN] epoch=103, iter=38300/80000, loss=13.6195, lr=0.005564, batch_cost=1.7342, reader_cost=0.0017 | ETA 20:05:14 2020-12-26 07:13:33 [INFO] [TRAIN] epoch=104, iter=38400/80000, loss=13.5826, lr=0.005552, batch_cost=1.8173, reader_cost=0.0501 | ETA 20:59:59 2020-12-26 07:16:27 [INFO] [TRAIN] epoch=104, iter=38500/80000, loss=13.3917, lr=0.005540, batch_cost=1.7338, reader_cost=0.0012 | ETA 19:59:10 2020-12-26 07:19:19 [INFO] [TRAIN] epoch=104, iter=38600/80000, loss=13.4172, lr=0.005528, batch_cost=1.7257, reader_cost=0.0017 | ETA 19:50:42 2020-12-26 07:22:22 [INFO] [TRAIN] epoch=105, iter=38700/80000, loss=13.5928, lr=0.005515, batch_cost=1.8233, reader_cost=0.0485 | ETA 20:55:00 2020-12-26 07:25:15 [INFO] [TRAIN] epoch=105, iter=38800/80000, loss=13.6077, lr=0.005503, batch_cost=1.7306, reader_cost=0.0012 | ETA 19:48:21 2020-12-26 07:28:09 [INFO] [TRAIN] epoch=105, iter=38900/80000, loss=13.4021, lr=0.005491, batch_cost=1.7408, reader_cost=0.0020 | ETA 19:52:28 2020-12-26 07:31:04 [INFO] [TRAIN] epoch=105, iter=39000/80000, loss=13.4324, lr=0.005479, batch_cost=1.7427, reader_cost=0.0008 | ETA 19:50:51 2020-12-26 07:34:07 [INFO] [TRAIN] epoch=106, iter=39100/80000, loss=13.3939, lr=0.005467, batch_cost=1.8361, reader_cost=0.0495 | ETA 20:51:35 2020-12-26 07:37:01 [INFO] [TRAIN] epoch=106, iter=39200/80000, loss=13.3530, lr=0.005455, batch_cost=1.7388, reader_cost=0.0015 | ETA 19:42:23 2020-12-26 07:39:55 [INFO] [TRAIN] epoch=106, iter=39300/80000, loss=13.4358, lr=0.005443, batch_cost=1.7316, reader_cost=0.0012 | ETA 19:34:35 2020-12-26 07:42:48 [INFO] [TRAIN] epoch=106, iter=39400/80000, loss=13.5136, lr=0.005431, batch_cost=1.7287, reader_cost=0.0016 | ETA 19:29:47 2020-12-26 07:45:50 [INFO] [TRAIN] epoch=107, iter=39500/80000, loss=13.5031, lr=0.005419, batch_cost=1.8197, reader_cost=0.0525 | ETA 20:28:16 2020-12-26 07:48:43 [INFO] [TRAIN] epoch=107, iter=39600/80000, loss=13.4216, lr=0.005407, batch_cost=1.7334, reader_cost=0.0007 | ETA 19:27:07 2020-12-26 07:51:37 [INFO] [TRAIN] epoch=107, iter=39700/80000, loss=13.0942, lr=0.005395, batch_cost=1.7398, reader_cost=0.0007 | ETA 19:28:35 2020-12-26 07:54:31 [INFO] [TRAIN] epoch=107, iter=39800/80000, loss=13.6308, lr=0.005383, batch_cost=1.7332, reader_cost=0.0008 | ETA 19:21:15 2020-12-26 07:57:33 [INFO] [TRAIN] epoch=108, iter=39900/80000, loss=13.3325, lr=0.005371, batch_cost=1.8219, reader_cost=0.0483 | ETA 20:17:37 2020-12-26 08:00:26 [INFO] [TRAIN] epoch=108, iter=40000/80000, loss=13.4109, lr=0.005359, batch_cost=1.7261, reader_cost=0.0013 | ETA 19:10:43 2020-12-26 08:00:26 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-26 08:01:08 [INFO] [EVAL] #Images=500 mIoU=0.7359 Acc=0.9526 Kappa=0.9386 2020-12-26 08:01:08 [INFO] [EVAL] Class IoU: [0.9751 0.8226 0.912 0.4418 0.5707 0.6091 0.6998 0.7774 0.911 0.6116 0.9274 0.7839 0.6322 0.9352 0.7466 0.7624 0.5363 0.556 0.7706] 2020-12-26 08:01:08 [INFO] [EVAL] Class Acc: [0.9895 0.9109 0.9421 0.8119 0.75 0.7297 0.7852 0.847 0.9613 0.8884 0.9374 0.8472 0.7376 0.9639 0.8765 0.842 0.8152 0.775 0.8498] 2020-12-26 08:01:13 [INFO] [EVAL] The model with the best validation mIoU (0.7643) was saved at iter 36000. 2020-12-26 08:04:06 [INFO] [TRAIN] epoch=108, iter=40100/80000, loss=13.1037, lr=0.005347, batch_cost=1.7318, reader_cost=0.0014 | ETA 19:11:37 2020-12-26 08:07:08 [INFO] [TRAIN] epoch=109, iter=40200/80000, loss=13.2958, lr=0.005335, batch_cost=1.8205, reader_cost=0.0487 | ETA 20:07:37 2020-12-26 08:10:00 [INFO] [TRAIN] epoch=109, iter=40300/80000, loss=13.1997, lr=0.005323, batch_cost=1.7151, reader_cost=0.0013 | ETA 18:54:49 2020-12-26 08:12:52 [INFO] [TRAIN] epoch=109, iter=40400/80000, loss=13.1876, lr=0.005311, batch_cost=1.7272, reader_cost=0.0012 | ETA 18:59:58 2020-12-26 08:15:46 [INFO] [TRAIN] epoch=109, iter=40500/80000, loss=13.3457, lr=0.005299, batch_cost=1.7315, reader_cost=0.0009 | ETA 18:59:54 2020-12-26 08:18:49 [INFO] [TRAIN] epoch=110, iter=40600/80000, loss=13.3292, lr=0.005287, batch_cost=1.8283, reader_cost=0.0505 | ETA 20:00:35 2020-12-26 08:21:43 [INFO] [TRAIN] epoch=110, iter=40700/80000, loss=13.3586, lr=0.005275, batch_cost=1.7440, reader_cost=0.0011 | ETA 19:02:17 2020-12-26 08:24:38 [INFO] [TRAIN] epoch=110, iter=40800/80000, loss=13.0658, lr=0.005262, batch_cost=1.7455, reader_cost=0.0010 | ETA 19:00:23 2020-12-26 08:27:31 [INFO] [TRAIN] epoch=110, iter=40900/80000, loss=13.4369, lr=0.005250, batch_cost=1.7292, reader_cost=0.0007 | ETA 18:46:52 2020-12-26 08:30:33 [INFO] [TRAIN] epoch=111, iter=41000/80000, loss=13.3305, lr=0.005238, batch_cost=1.8178, reader_cost=0.0505 | ETA 19:41:32 2020-12-26 08:33:26 [INFO] [TRAIN] epoch=111, iter=41100/80000, loss=13.3641, lr=0.005226, batch_cost=1.7324, reader_cost=0.0010 | ETA 18:43:09 2020-12-26 08:36:20 [INFO] [TRAIN] epoch=111, iter=41200/80000, loss=13.0553, lr=0.005214, batch_cost=1.7398, reader_cost=0.0015 | ETA 18:45:05 2020-12-26 08:39:22 [INFO] [TRAIN] epoch=112, iter=41300/80000, loss=13.5032, lr=0.005202, batch_cost=1.8158, reader_cost=0.0528 | ETA 19:31:10 2020-12-26 08:42:15 [INFO] [TRAIN] epoch=112, iter=41400/80000, loss=13.0595, lr=0.005190, batch_cost=1.7241, reader_cost=0.0019 | ETA 18:29:11 2020-12-26 08:45:08 [INFO] [TRAIN] epoch=112, iter=41500/80000, loss=13.0310, lr=0.005178, batch_cost=1.7307, reader_cost=0.0016 | ETA 18:30:31 2020-12-26 08:48:01 [INFO] [TRAIN] epoch=112, iter=41600/80000, loss=13.2003, lr=0.005166, batch_cost=1.7333, reader_cost=0.0017 | ETA 18:29:18 2020-12-26 08:51:04 [INFO] [TRAIN] epoch=113, iter=41700/80000, loss=13.4640, lr=0.005154, batch_cost=1.8229, reader_cost=0.0488 | ETA 19:23:37 2020-12-26 08:53:58 [INFO] [TRAIN] epoch=113, iter=41800/80000, loss=13.1611, lr=0.005141, batch_cost=1.7377, reader_cost=0.0019 | ETA 18:26:19 2020-12-26 08:56:52 [INFO] [TRAIN] epoch=113, iter=41900/80000, loss=12.9571, lr=0.005129, batch_cost=1.7418, reader_cost=0.0011 | ETA 18:26:00 2020-12-26 08:59:46 [INFO] [TRAIN] epoch=113, iter=42000/80000, loss=13.2105, lr=0.005117, batch_cost=1.7412, reader_cost=0.0022 | ETA 18:22:45 2020-12-26 08:59:46 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-26 09:00:29 [INFO] [EVAL] #Images=500 mIoU=0.7628 Acc=0.9583 Kappa=0.9459 2020-12-26 09:00:29 [INFO] [EVAL] Class IoU: [0.9793 0.8309 0.9254 0.4465 0.5834 0.6609 0.7163 0.7923 0.9188 0.6024 0.9453 0.8204 0.6586 0.9478 0.7369 0.8172 0.6895 0.6455 0.7752] 2020-12-26 09:00:29 [INFO] [EVAL] Class Acc: [0.9923 0.878 0.9559 0.8815 0.8019 0.7845 0.8859 0.9268 0.9546 0.824 0.9659 0.8998 0.7789 0.9664 0.8653 0.9424 0.7385 0.8581 0.8803] 2020-12-26 09:00:33 [INFO] [EVAL] The model with the best validation mIoU (0.7643) was saved at iter 36000. 2020-12-26 09:03:36 [INFO] [TRAIN] epoch=114, iter=42100/80000, loss=13.2040, lr=0.005105, batch_cost=1.8235, reader_cost=0.0495 | ETA 19:11:49 2020-12-26 09:06:30 [INFO] [TRAIN] epoch=114, iter=42200/80000, loss=12.9765, lr=0.005093, batch_cost=1.7393, reader_cost=0.0013 | ETA 18:15:46 2020-12-26 09:09:24 [INFO] [TRAIN] epoch=114, iter=42300/80000, loss=13.0573, lr=0.005081, batch_cost=1.7408, reader_cost=0.0013 | ETA 18:13:46 2020-12-26 09:12:18 [INFO] [TRAIN] epoch=114, iter=42400/80000, loss=13.1134, lr=0.005069, batch_cost=1.7353, reader_cost=0.0007 | ETA 18:07:28 2020-12-26 09:15:20 [INFO] [TRAIN] epoch=115, iter=42500/80000, loss=12.9133, lr=0.005057, batch_cost=1.8248, reader_cost=0.0497 | ETA 19:00:30 2020-12-26 09:18:14 [INFO] [TRAIN] epoch=115, iter=42600/80000, loss=12.8955, lr=0.005044, batch_cost=1.7388, reader_cost=0.0009 | ETA 18:03:52 2020-12-26 09:21:08 [INFO] [TRAIN] epoch=115, iter=42700/80000, loss=12.8226, lr=0.005032, batch_cost=1.7339, reader_cost=0.0010 | ETA 17:57:54 2020-12-26 09:24:11 [INFO] [TRAIN] epoch=116, iter=42800/80000, loss=13.1884, lr=0.005020, batch_cost=1.8334, reader_cost=0.0512 | ETA 18:56:42 2020-12-26 09:27:05 [INFO] [TRAIN] epoch=116, iter=42900/80000, loss=12.8284, lr=0.005008, batch_cost=1.7373, reader_cost=0.0015 | ETA 17:54:12 2020-12-26 09:29:59 [INFO] [TRAIN] epoch=116, iter=43000/80000, loss=12.8061, lr=0.004996, batch_cost=1.7399, reader_cost=0.0012 | ETA 17:52:54 2020-12-26 09:32:54 [INFO] [TRAIN] epoch=116, iter=43100/80000, loss=12.9637, lr=0.004984, batch_cost=1.7457, reader_cost=0.0010 | ETA 17:53:34 2020-12-26 09:35:57 [INFO] [TRAIN] epoch=117, iter=43200/80000, loss=13.1175, lr=0.004972, batch_cost=1.8253, reader_cost=0.0523 | ETA 18:39:31 2020-12-26 09:38:51 [INFO] [TRAIN] epoch=117, iter=43300/80000, loss=12.8731, lr=0.004959, batch_cost=1.7445, reader_cost=0.0007 | ETA 17:47:04 2020-12-26 09:41:44 [INFO] [TRAIN] epoch=117, iter=43400/80000, loss=12.8112, lr=0.004947, batch_cost=1.7255, reader_cost=0.0007 | ETA 17:32:32 2020-12-26 09:44:37 [INFO] [TRAIN] epoch=117, iter=43500/80000, loss=13.0068, lr=0.004935, batch_cost=1.7321, reader_cost=0.0006 | ETA 17:33:40 2020-12-26 09:47:40 [INFO] [TRAIN] epoch=118, iter=43600/80000, loss=13.0922, lr=0.004923, batch_cost=1.8269, reader_cost=0.0486 | ETA 18:28:20 2020-12-26 09:50:33 [INFO] [TRAIN] epoch=118, iter=43700/80000, loss=12.7942, lr=0.004911, batch_cost=1.7316, reader_cost=0.0012 | ETA 17:27:36 2020-12-26 09:53:27 [INFO] [TRAIN] epoch=118, iter=43800/80000, loss=12.8054, lr=0.004899, batch_cost=1.7315, reader_cost=0.0013 | ETA 17:24:39 2020-12-26 09:56:30 [INFO] [TRAIN] epoch=119, iter=43900/80000, loss=13.0021, lr=0.004886, batch_cost=1.8301, reader_cost=0.0513 | ETA 18:21:07 2020-12-26 09:59:23 [INFO] [TRAIN] epoch=119, iter=44000/80000, loss=12.8563, lr=0.004874, batch_cost=1.7294, reader_cost=0.0014 | ETA 17:17:37 2020-12-26 09:59:23 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-26 10:00:06 [INFO] [EVAL] #Images=500 mIoU=0.7194 Acc=0.9511 Kappa=0.9364 2020-12-26 10:00:06 [INFO] [EVAL] Class IoU: [0.9746 0.7999 0.9133 0.294 0.5365 0.6226 0.6994 0.7929 0.9062 0.5111 0.9437 0.806 0.6203 0.9333 0.6805 0.7854 0.639 0.4717 0.7381] 2020-12-26 10:00:06 [INFO] [EVAL] Class Acc: [0.9835 0.9102 0.9452 0.7396 0.7931 0.8353 0.8558 0.9053 0.9396 0.8849 0.9664 0.8715 0.7295 0.9593 0.7303 0.9522 0.692 0.5622 0.8519] 2020-12-26 10:00:10 [INFO] [EVAL] The model with the best validation mIoU (0.7643) was saved at iter 36000. 2020-12-26 10:03:03 [INFO] [TRAIN] epoch=119, iter=44100/80000, loss=12.8598, lr=0.004862, batch_cost=1.7273, reader_cost=0.0018 | ETA 17:13:30 2020-12-26 10:05:56 [INFO] [TRAIN] epoch=119, iter=44200/80000, loss=13.0276, lr=0.004850, batch_cost=1.7356, reader_cost=0.0013 | ETA 17:15:34 2020-12-26 10:08:59 [INFO] [TRAIN] epoch=120, iter=44300/80000, loss=13.0237, lr=0.004838, batch_cost=1.8271, reader_cost=0.0500 | ETA 18:07:05 2020-12-26 10:11:52 [INFO] [TRAIN] epoch=120, iter=44400/80000, loss=12.7489, lr=0.004825, batch_cost=1.7321, reader_cost=0.0008 | ETA 17:07:41 2020-12-26 10:14:46 [INFO] [TRAIN] epoch=120, iter=44500/80000, loss=12.6373, lr=0.004813, batch_cost=1.7314, reader_cost=0.0010 | ETA 17:04:26 2020-12-26 10:17:39 [INFO] [TRAIN] epoch=120, iter=44600/80000, loss=12.9524, lr=0.004801, batch_cost=1.7286, reader_cost=0.0007 | ETA 16:59:53 2020-12-26 10:20:42 [INFO] [TRAIN] epoch=121, iter=44700/80000, loss=12.7872, lr=0.004789, batch_cost=1.8278, reader_cost=0.0491 | ETA 17:55:22 2020-12-26 10:23:35 [INFO] [TRAIN] epoch=121, iter=44800/80000, loss=12.7976, lr=0.004777, batch_cost=1.7290, reader_cost=0.0009 | ETA 16:54:22 2020-12-26 10:26:28 [INFO] [TRAIN] epoch=121, iter=44900/80000, loss=12.7284, lr=0.004764, batch_cost=1.7289, reader_cost=0.0009 | ETA 16:51:25 2020-12-26 10:29:20 [INFO] [TRAIN] epoch=121, iter=45000/80000, loss=12.9817, lr=0.004752, batch_cost=1.7260, reader_cost=0.0009 | ETA 16:46:49 2020-12-26 10:32:22 [INFO] [TRAIN] epoch=122, iter=45100/80000, loss=12.7119, lr=0.004740, batch_cost=1.8155, reader_cost=0.0495 | ETA 17:36:00 2020-12-26 10:35:15 [INFO] [TRAIN] epoch=122, iter=45200/80000, loss=12.8029, lr=0.004728, batch_cost=1.7314, reader_cost=0.0018 | ETA 16:44:12 2020-12-26 10:38:08 [INFO] [TRAIN] epoch=122, iter=45300/80000, loss=12.7475, lr=0.004715, batch_cost=1.7294, reader_cost=0.0007 | ETA 16:40:08 2020-12-26 10:41:11 [INFO] [TRAIN] epoch=123, iter=45400/80000, loss=12.9284, lr=0.004703, batch_cost=1.8280, reader_cost=0.0495 | ETA 17:34:10 2020-12-26 10:44:04 [INFO] [TRAIN] epoch=123, iter=45500/80000, loss=12.6054, lr=0.004691, batch_cost=1.7300, reader_cost=0.0020 | ETA 16:34:46 2020-12-26 10:46:58 [INFO] [TRAIN] epoch=123, iter=45600/80000, loss=12.5761, lr=0.004679, batch_cost=1.7294, reader_cost=0.0015 | ETA 16:31:30 2020-12-26 10:49:51 [INFO] [TRAIN] epoch=123, iter=45700/80000, loss=12.8669, lr=0.004667, batch_cost=1.7289, reader_cost=0.0007 | ETA 16:28:22 2020-12-26 10:52:54 [INFO] [TRAIN] epoch=124, iter=45800/80000, loss=12.7835, lr=0.004654, batch_cost=1.8315, reader_cost=0.0502 | ETA 17:23:56 2020-12-26 10:55:46 [INFO] [TRAIN] epoch=124, iter=45900/80000, loss=12.8038, lr=0.004642, batch_cost=1.7191, reader_cost=0.0012 | ETA 16:17:01 2020-12-26 10:58:39 [INFO] [TRAIN] epoch=124, iter=46000/80000, loss=12.5864, lr=0.004630, batch_cost=1.7284, reader_cost=0.0014 | ETA 16:19:25 2020-12-26 10:58:39 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-26 10:59:22 [INFO] [EVAL] #Images=500 mIoU=0.7412 Acc=0.9540 Kappa=0.9402 2020-12-26 10:59:22 [INFO] [EVAL] Class IoU: [0.9736 0.821 0.9136 0.3427 0.5331 0.6619 0.7315 0.7808 0.9132 0.6012 0.9483 0.8065 0.6173 0.9504 0.8006 0.7645 0.5328 0.6169 0.7732] 2020-12-26 10:59:22 [INFO] [EVAL] Class Acc: [0.9826 0.9153 0.9439 0.8545 0.7809 0.8267 0.8402 0.9297 0.9569 0.7313 0.9745 0.8657 0.6947 0.975 0.9027 0.9685 0.5582 0.7719 0.8597] 2020-12-26 10:59:26 [INFO] [EVAL] The model with the best validation mIoU (0.7643) was saved at iter 36000. 2020-12-26 11:02:19 [INFO] [TRAIN] epoch=124, iter=46100/80000, loss=12.8017, lr=0.004618, batch_cost=1.7259, reader_cost=0.0012 | ETA 16:15:09 2020-12-26 11:05:21 [INFO] [TRAIN] epoch=125, iter=46200/80000, loss=12.7907, lr=0.004605, batch_cost=1.8221, reader_cost=0.0485 | ETA 17:06:26 2020-12-26 11:08:14 [INFO] [TRAIN] epoch=125, iter=46300/80000, loss=12.7485, lr=0.004593, batch_cost=1.7271, reader_cost=0.0014 | ETA 16:10:02 2020-12-26 11:11:07 [INFO] [TRAIN] epoch=125, iter=46400/80000, loss=12.5926, lr=0.004581, batch_cost=1.7275, reader_cost=0.0019 | ETA 16:07:25 2020-12-26 11:14:00 [INFO] [TRAIN] epoch=125, iter=46500/80000, loss=12.8333, lr=0.004568, batch_cost=1.7347, reader_cost=0.0017 | ETA 16:08:30 2020-12-26 11:17:03 [INFO] [TRAIN] epoch=126, iter=46600/80000, loss=12.5993, lr=0.004556, batch_cost=1.8217, reader_cost=0.0490 | ETA 16:54:04 2020-12-26 11:19:55 [INFO] [TRAIN] epoch=126, iter=46700/80000, loss=12.5620, lr=0.004544, batch_cost=1.7261, reader_cost=0.0019 | ETA 15:57:57 2020-12-26 11:22:48 [INFO] [TRAIN] epoch=126, iter=46800/80000, loss=12.6474, lr=0.004532, batch_cost=1.7284, reader_cost=0.0014 | ETA 15:56:22 2020-12-26 11:25:51 [INFO] [TRAIN] epoch=127, iter=46900/80000, loss=12.7371, lr=0.004519, batch_cost=1.8231, reader_cost=0.0497 | ETA 16:45:45 2020-12-26 11:28:43 [INFO] [TRAIN] epoch=127, iter=47000/80000, loss=12.6635, lr=0.004507, batch_cost=1.7214, reader_cost=0.0015 | ETA 15:46:47 2020-12-26 11:31:36 [INFO] [TRAIN] epoch=127, iter=47100/80000, loss=12.5967, lr=0.004495, batch_cost=1.7280, reader_cost=0.0015 | ETA 15:47:29 2020-12-26 11:34:29 [INFO] [TRAIN] epoch=127, iter=47200/80000, loss=12.7379, lr=0.004482, batch_cost=1.7292, reader_cost=0.0011 | ETA 15:45:16 2020-12-26 11:37:33 [INFO] [TRAIN] epoch=128, iter=47300/80000, loss=12.7053, lr=0.004470, batch_cost=1.8342, reader_cost=0.0524 | ETA 16:39:38 2020-12-26 11:40:25 [INFO] [TRAIN] epoch=128, iter=47400/80000, loss=12.5981, lr=0.004458, batch_cost=1.7257, reader_cost=0.0013 | ETA 15:37:38 2020-12-26 11:43:19 [INFO] [TRAIN] epoch=128, iter=47500/80000, loss=12.4242, lr=0.004446, batch_cost=1.7332, reader_cost=0.0014 | ETA 15:38:48 2020-12-26 11:46:11 [INFO] [TRAIN] epoch=128, iter=47600/80000, loss=12.8093, lr=0.004433, batch_cost=1.7255, reader_cost=0.0009 | ETA 15:31:44 2020-12-26 11:49:16 [INFO] [TRAIN] epoch=129, iter=47700/80000, loss=12.6019, lr=0.004421, batch_cost=1.8459, reader_cost=0.0492 | ETA 16:33:41 2020-12-26 11:52:11 [INFO] [TRAIN] epoch=129, iter=47800/80000, loss=12.5002, lr=0.004409, batch_cost=1.7454, reader_cost=0.0016 | ETA 15:36:40 2020-12-26 11:55:05 [INFO] [TRAIN] epoch=129, iter=47900/80000, loss=12.6624, lr=0.004396, batch_cost=1.7415, reader_cost=0.0017 | ETA 15:31:42 2020-12-26 11:58:09 [INFO] [TRAIN] epoch=130, iter=48000/80000, loss=12.7689, lr=0.004384, batch_cost=1.8359, reader_cost=0.0478 | ETA 16:19:08 2020-12-26 11:58:09 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-26 11:58:52 [INFO] [EVAL] #Images=500 mIoU=0.7805 Acc=0.9605 Kappa=0.9486 2020-12-26 11:58:52 [INFO] [EVAL] Class IoU: [0.9796 0.8404 0.9246 0.4955 0.6233 0.6632 0.7396 0.8042 0.9241 0.6339 0.9443 0.8289 0.6433 0.9508 0.7831 0.8386 0.7781 0.653 0.7808] 2020-12-26 11:58:52 [INFO] [EVAL] Class Acc: [0.9883 0.9259 0.956 0.8888 0.7826 0.878 0.8459 0.8908 0.9497 0.8672 0.9555 0.9175 0.786 0.9705 0.8783 0.9385 0.9429 0.8094 0.8494] 2020-12-26 11:58:59 [INFO] [EVAL] The model with the best validation mIoU (0.7805) was saved at iter 48000. 2020-12-26 12:01:52 [INFO] [TRAIN] epoch=130, iter=48100/80000, loss=12.3700, lr=0.004372, batch_cost=1.7282, reader_cost=0.0020 | ETA 15:18:49 2020-12-26 12:04:47 [INFO] [TRAIN] epoch=130, iter=48200/80000, loss=12.4068, lr=0.004359, batch_cost=1.7517, reader_cost=0.0013 | ETA 15:28:23 2020-12-26 12:07:40 [INFO] [TRAIN] epoch=130, iter=48300/80000, loss=12.5873, lr=0.004347, batch_cost=1.7354, reader_cost=0.0009 | ETA 15:16:52 2020-12-26 12:10:43 [INFO] [TRAIN] epoch=131, iter=48400/80000, loss=12.6193, lr=0.004335, batch_cost=1.8251, reader_cost=0.0506 | ETA 16:01:12 2020-12-26 12:13:36 [INFO] [TRAIN] epoch=131, iter=48500/80000, loss=12.4499, lr=0.004322, batch_cost=1.7249, reader_cost=0.0016 | ETA 15:05:34 2020-12-26 12:16:28 [INFO] [TRAIN] epoch=131, iter=48600/80000, loss=12.3141, lr=0.004310, batch_cost=1.7245, reader_cost=0.0013 | ETA 15:02:29 2020-12-26 12:19:21 [INFO] [TRAIN] epoch=131, iter=48700/80000, loss=12.7663, lr=0.004298, batch_cost=1.7280, reader_cost=0.0015 | ETA 15:01:25 2020-12-26 12:22:24 [INFO] [TRAIN] epoch=132, iter=48800/80000, loss=12.7330, lr=0.004285, batch_cost=1.8296, reader_cost=0.0499 | ETA 15:51:24 2020-12-26 12:25:17 [INFO] [TRAIN] epoch=132, iter=48900/80000, loss=12.4939, lr=0.004273, batch_cost=1.7276, reader_cost=0.0014 | ETA 14:55:29 2020-12-26 12:28:10 [INFO] [TRAIN] epoch=132, iter=49000/80000, loss=12.4269, lr=0.004260, batch_cost=1.7279, reader_cost=0.0015 | ETA 14:52:46 2020-12-26 12:31:03 [INFO] [TRAIN] epoch=132, iter=49100/80000, loss=12.6095, lr=0.004248, batch_cost=1.7306, reader_cost=0.0012 | ETA 14:51:14 2020-12-26 12:34:06 [INFO] [TRAIN] epoch=133, iter=49200/80000, loss=12.4445, lr=0.004236, batch_cost=1.8280, reader_cost=0.0497 | ETA 15:38:21 2020-12-26 12:36:58 [INFO] [TRAIN] epoch=133, iter=49300/80000, loss=12.4351, lr=0.004223, batch_cost=1.7176, reader_cost=0.0014 | ETA 14:38:51 2020-12-26 12:39:51 [INFO] [TRAIN] epoch=133, iter=49400/80000, loss=12.3763, lr=0.004211, batch_cost=1.7304, reader_cost=0.0015 | ETA 14:42:29 2020-12-26 12:42:54 [INFO] [TRAIN] epoch=134, iter=49500/80000, loss=12.5632, lr=0.004199, batch_cost=1.8244, reader_cost=0.0543 | ETA 15:27:23 2020-12-26 12:45:46 [INFO] [TRAIN] epoch=134, iter=49600/80000, loss=12.3312, lr=0.004186, batch_cost=1.7232, reader_cost=0.0010 | ETA 14:33:06 2020-12-26 12:48:39 [INFO] [TRAIN] epoch=134, iter=49700/80000, loss=12.3939, lr=0.004174, batch_cost=1.7248, reader_cost=0.0018 | ETA 14:31:02 2020-12-26 12:51:31 [INFO] [TRAIN] epoch=134, iter=49800/80000, loss=12.5766, lr=0.004161, batch_cost=1.7155, reader_cost=0.0009 | ETA 14:23:26 2020-12-26 12:54:33 [INFO] [TRAIN] epoch=135, iter=49900/80000, loss=12.4638, lr=0.004149, batch_cost=1.8209, reader_cost=0.0490 | ETA 15:13:29 2020-12-26 12:57:25 [INFO] [TRAIN] epoch=135, iter=50000/80000, loss=12.3062, lr=0.004137, batch_cost=1.7224, reader_cost=0.0011 | ETA 14:21:10 2020-12-26 12:57:25 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-26 12:58:08 [INFO] [EVAL] #Images=500 mIoU=0.7620 Acc=0.9560 Kappa=0.9429 2020-12-26 12:58:08 [INFO] [EVAL] Class IoU: [0.9779 0.821 0.9155 0.4691 0.5998 0.658 0.7163 0.796 0.9147 0.5968 0.9473 0.8199 0.6168 0.9464 0.7121 0.8122 0.7624 0.6365 0.7585] 2020-12-26 12:58:08 [INFO] [EVAL] Class Acc: [0.9915 0.8884 0.9567 0.7682 0.8081 0.8181 0.8531 0.9154 0.942 0.8034 0.9668 0.8986 0.715 0.9683 0.8733 0.9275 0.9045 0.8669 0.818 ] 2020-12-26 12:58:12 [INFO] [EVAL] The model with the best validation mIoU (0.7805) was saved at iter 48000. 2020-12-26 13:01:06 [INFO] [TRAIN] epoch=135, iter=50100/80000, loss=12.3141, lr=0.004124, batch_cost=1.7313, reader_cost=0.0014 | ETA 14:22:45 2020-12-26 13:03:59 [INFO] [TRAIN] epoch=135, iter=50200/80000, loss=12.7331, lr=0.004112, batch_cost=1.7285, reader_cost=0.0011 | ETA 14:18:29 2020-12-26 13:07:04 [INFO] [TRAIN] epoch=136, iter=50300/80000, loss=12.4009, lr=0.004099, batch_cost=1.8576, reader_cost=0.0505 | ETA 15:19:30 2020-12-26 13:09:57 [INFO] [TRAIN] epoch=136, iter=50400/80000, loss=12.3732, lr=0.004087, batch_cost=1.7243, reader_cost=0.0019 | ETA 14:10:39 2020-12-26 13:12:51 [INFO] [TRAIN] epoch=136, iter=50500/80000, loss=12.3118, lr=0.004074, batch_cost=1.7341, reader_cost=0.0014 | ETA 14:12:35 2020-12-26 13:15:54 [INFO] [TRAIN] epoch=137, iter=50600/80000, loss=12.6684, lr=0.004062, batch_cost=1.8347, reader_cost=0.0506 | ETA 14:58:59 2020-12-26 13:18:47 [INFO] [TRAIN] epoch=137, iter=50700/80000, loss=12.4042, lr=0.004050, batch_cost=1.7317, reader_cost=0.0014 | ETA 14:05:38 2020-12-26 13:21:43 [INFO] [TRAIN] epoch=137, iter=50800/80000, loss=12.4049, lr=0.004037, batch_cost=1.7576, reader_cost=0.0017 | ETA 14:15:21 2020-12-26 13:24:37 [INFO] [TRAIN] epoch=137, iter=50900/80000, loss=12.4929, lr=0.004025, batch_cost=1.7306, reader_cost=0.0020 | ETA 13:59:21 2020-12-26 13:27:44 [INFO] [TRAIN] epoch=138, iter=51000/80000, loss=12.4257, lr=0.004012, batch_cost=1.8720, reader_cost=0.0494 | ETA 15:04:48 2020-12-26 13:30:39 [INFO] [TRAIN] epoch=138, iter=51100/80000, loss=12.2848, lr=0.004000, batch_cost=1.7496, reader_cost=0.0009 | ETA 14:02:42 2020-12-26 13:33:38 [INFO] [TRAIN] epoch=138, iter=51200/80000, loss=12.2613, lr=0.003987, batch_cost=1.7895, reader_cost=0.0023 | ETA 14:18:57 2020-12-26 13:36:34 [INFO] [TRAIN] epoch=138, iter=51300/80000, loss=12.4412, lr=0.003975, batch_cost=1.7546, reader_cost=0.0013 | ETA 13:59:17 2020-12-26 13:39:40 [INFO] [TRAIN] epoch=139, iter=51400/80000, loss=12.4723, lr=0.003962, batch_cost=1.8648, reader_cost=0.0488 | ETA 14:48:54 2020-12-26 13:42:34 [INFO] [TRAIN] epoch=139, iter=51500/80000, loss=12.4335, lr=0.003950, batch_cost=1.7331, reader_cost=0.0014 | ETA 13:43:14 2020-12-26 13:45:27 [INFO] [TRAIN] epoch=139, iter=51600/80000, loss=12.3517, lr=0.003937, batch_cost=1.7354, reader_cost=0.0011 | ETA 13:41:25 2020-12-26 13:48:22 [INFO] [TRAIN] epoch=139, iter=51700/80000, loss=12.5302, lr=0.003925, batch_cost=1.7432, reader_cost=0.0016 | ETA 13:42:13 2020-12-26 13:51:26 [INFO] [TRAIN] epoch=140, iter=51800/80000, loss=12.4489, lr=0.003913, batch_cost=1.8391, reader_cost=0.0491 | ETA 14:24:23 2020-12-26 13:54:20 [INFO] [TRAIN] epoch=140, iter=51900/80000, loss=12.4170, lr=0.003900, batch_cost=1.7376, reader_cost=0.0016 | ETA 13:33:46 2020-12-26 13:57:13 [INFO] [TRAIN] epoch=140, iter=52000/80000, loss=12.4213, lr=0.003888, batch_cost=1.7344, reader_cost=0.0012 | ETA 13:29:23 2020-12-26 13:57:13 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-26 13:57:57 [INFO] [EVAL] #Images=500 mIoU=0.7566 Acc=0.9615 Kappa=0.9501 2020-12-26 13:57:57 [INFO] [EVAL] Class IoU: [0.9825 0.8558 0.9284 0.537 0.6193 0.6844 0.747 0.8047 0.9267 0.635 0.9515 0.8324 0.6484 0.9476 0.7286 0.7701 0.6198 0.4138 0.7423] 2020-12-26 13:57:57 [INFO] [EVAL] Class Acc: [0.9932 0.9091 0.9578 0.8826 0.8304 0.8138 0.8648 0.8718 0.9576 0.86 0.9704 0.9141 0.744 0.974 0.832 0.7982 0.9248 0.445 0.8874] 2020-12-26 13:58:01 [INFO] [EVAL] The model with the best validation mIoU (0.7805) was saved at iter 48000. 2020-12-26 14:01:04 [INFO] [TRAIN] epoch=141, iter=52100/80000, loss=12.4729, lr=0.003875, batch_cost=1.8348, reader_cost=0.0492 | ETA 14:13:10 2020-12-26 14:03:58 [INFO] [TRAIN] epoch=141, iter=52200/80000, loss=12.1993, lr=0.003863, batch_cost=1.7326, reader_cost=0.0019 | ETA 13:22:47 2020-12-26 14:06:51 [INFO] [TRAIN] epoch=141, iter=52300/80000, loss=12.2654, lr=0.003850, batch_cost=1.7304, reader_cost=0.0018 | ETA 13:18:51 2020-12-26 14:09:44 [INFO] [TRAIN] epoch=141, iter=52400/80000, loss=12.4238, lr=0.003838, batch_cost=1.7319, reader_cost=0.0014 | ETA 13:16:41 2020-12-26 14:12:48 [INFO] [TRAIN] epoch=142, iter=52500/80000, loss=12.4547, lr=0.003825, batch_cost=1.8344, reader_cost=0.0505 | ETA 14:00:46 2020-12-26 14:15:41 [INFO] [TRAIN] epoch=142, iter=52600/80000, loss=12.3110, lr=0.003812, batch_cost=1.7314, reader_cost=0.0012 | ETA 13:10:40 2020-12-26 14:18:34 [INFO] [TRAIN] epoch=142, iter=52700/80000, loss=12.2992, lr=0.003800, batch_cost=1.7288, reader_cost=0.0011 | ETA 13:06:34 2020-12-26 14:21:27 [INFO] [TRAIN] epoch=142, iter=52800/80000, loss=12.4756, lr=0.003787, batch_cost=1.7299, reader_cost=0.0006 | ETA 13:04:12 2020-12-26 14:24:31 [INFO] [TRAIN] epoch=143, iter=52900/80000, loss=12.4676, lr=0.003775, batch_cost=1.8428, reader_cost=0.0492 | ETA 13:52:20 2020-12-26 14:27:25 [INFO] [TRAIN] epoch=143, iter=53000/80000, loss=12.1717, lr=0.003762, batch_cost=1.7369, reader_cost=0.0015 | ETA 13:01:35 2020-12-26 14:30:19 [INFO] [TRAIN] epoch=143, iter=53100/80000, loss=12.3719, lr=0.003750, batch_cost=1.7394, reader_cost=0.0015 | ETA 12:59:50 2020-12-26 14:33:23 [INFO] [TRAIN] epoch=144, iter=53200/80000, loss=12.5540, lr=0.003737, batch_cost=1.8389, reader_cost=0.0486 | ETA 13:41:22 2020-12-26 14:36:16 [INFO] [TRAIN] epoch=144, iter=53300/80000, loss=12.3513, lr=0.003725, batch_cost=1.7272, reader_cost=0.0013 | ETA 12:48:36 2020-12-26 14:39:10 [INFO] [TRAIN] epoch=144, iter=53400/80000, loss=12.2182, lr=0.003712, batch_cost=1.7399, reader_cost=0.0014 | ETA 12:51:21 2020-12-26 14:42:04 [INFO] [TRAIN] epoch=144, iter=53500/80000, loss=12.3210, lr=0.003700, batch_cost=1.7350, reader_cost=0.0015 | ETA 12:46:18 2020-12-26 14:45:09 [INFO] [TRAIN] epoch=145, iter=53600/80000, loss=12.4984, lr=0.003687, batch_cost=1.8445, reader_cost=0.0491 | ETA 13:31:34 2020-12-26 14:48:02 [INFO] [TRAIN] epoch=145, iter=53700/80000, loss=12.3804, lr=0.003674, batch_cost=1.7318, reader_cost=0.0003 | ETA 12:39:05 2020-12-26 14:50:55 [INFO] [TRAIN] epoch=145, iter=53800/80000, loss=12.2014, lr=0.003662, batch_cost=1.7349, reader_cost=0.0003 | ETA 12:37:33 2020-12-26 14:53:49 [INFO] [TRAIN] epoch=145, iter=53900/80000, loss=12.4620, lr=0.003649, batch_cost=1.7341, reader_cost=0.0010 | ETA 12:34:20 2020-12-26 14:56:53 [INFO] [TRAIN] epoch=146, iter=54000/80000, loss=12.3454, lr=0.003637, batch_cost=1.8420, reader_cost=0.0565 | ETA 13:18:11 2020-12-26 14:56:53 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-26 14:57:36 [INFO] [EVAL] #Images=500 mIoU=0.7698 Acc=0.9603 Kappa=0.9484 2020-12-26 14:57:36 [INFO] [EVAL] Class IoU: [0.9799 0.8407 0.9256 0.6051 0.5104 0.661 0.7435 0.8196 0.9229 0.6058 0.9473 0.8335 0.6458 0.9526 0.7367 0.8495 0.6291 0.635 0.7829] 2020-12-26 14:57:36 [INFO] [EVAL] Class Acc: [0.9906 0.9098 0.9546 0.8039 0.8985 0.8506 0.8675 0.9129 0.9496 0.8466 0.9703 0.8925 0.7675 0.97 0.9242 0.8907 0.9239 0.7752 0.8806] 2020-12-26 14:57:40 [INFO] [EVAL] The model with the best validation mIoU (0.7805) was saved at iter 48000. 2020-12-26 15:00:34 [INFO] [TRAIN] epoch=146, iter=54100/80000, loss=12.2699, lr=0.003624, batch_cost=1.7345, reader_cost=0.0010 | ETA 12:28:43 2020-12-26 15:03:31 [INFO] [TRAIN] epoch=146, iter=54200/80000, loss=12.2072, lr=0.003612, batch_cost=1.7673, reader_cost=0.0009 | ETA 12:39:55 2020-12-26 15:06:27 [INFO] [TRAIN] epoch=146, iter=54300/80000, loss=12.5263, lr=0.003599, batch_cost=1.7565, reader_cost=0.0009 | ETA 12:32:21 2020-12-26 15:09:30 [INFO] [TRAIN] epoch=147, iter=54400/80000, loss=12.2394, lr=0.003586, batch_cost=1.8375, reader_cost=0.0494 | ETA 13:03:59 2020-12-26 15:12:24 [INFO] [TRAIN] epoch=147, iter=54500/80000, loss=12.2548, lr=0.003574, batch_cost=1.7308, reader_cost=0.0016 | ETA 12:15:36 2020-12-26 15:15:17 [INFO] [TRAIN] epoch=147, iter=54600/80000, loss=12.2679, lr=0.003561, batch_cost=1.7361, reader_cost=0.0008 | ETA 12:14:55 2020-12-26 15:18:21 [INFO] [TRAIN] epoch=148, iter=54700/80000, loss=12.3787, lr=0.003548, batch_cost=1.8365, reader_cost=0.0504 | ETA 12:54:22 2020-12-26 15:21:16 [INFO] [TRAIN] epoch=148, iter=54800/80000, loss=12.1344, lr=0.003536, batch_cost=1.7504, reader_cost=0.0008 | ETA 12:15:09 2020-12-26 15:24:11 [INFO] [TRAIN] epoch=148, iter=54900/80000, loss=12.2258, lr=0.003523, batch_cost=1.7472, reader_cost=0.0010 | ETA 12:10:54 2020-12-26 15:27:05 [INFO] [TRAIN] epoch=148, iter=55000/80000, loss=12.3691, lr=0.003511, batch_cost=1.7332, reader_cost=0.0007 | ETA 12:02:11 2020-12-26 15:30:08 [INFO] [TRAIN] epoch=149, iter=55100/80000, loss=12.4041, lr=0.003498, batch_cost=1.8344, reader_cost=0.0502 | ETA 12:41:15 2020-12-26 15:33:01 [INFO] [TRAIN] epoch=149, iter=55200/80000, loss=12.2238, lr=0.003485, batch_cost=1.7305, reader_cost=0.0024 | ETA 11:55:15 2020-12-26 15:35:54 [INFO] [TRAIN] epoch=149, iter=55300/80000, loss=12.2176, lr=0.003473, batch_cost=1.7292, reader_cost=0.0024 | ETA 11:51:52 2020-12-26 15:38:48 [INFO] [TRAIN] epoch=149, iter=55400/80000, loss=12.4423, lr=0.003460, batch_cost=1.7308, reader_cost=0.0019 | ETA 11:49:37 2020-12-26 15:41:52 [INFO] [TRAIN] epoch=150, iter=55500/80000, loss=12.2323, lr=0.003447, batch_cost=1.8399, reader_cost=0.0513 | ETA 12:31:17 2020-12-26 15:44:45 [INFO] [TRAIN] epoch=150, iter=55600/80000, loss=12.1537, lr=0.003435, batch_cost=1.7294, reader_cost=0.0019 | ETA 11:43:17 2020-12-26 15:47:38 [INFO] [TRAIN] epoch=150, iter=55700/80000, loss=12.2874, lr=0.003422, batch_cost=1.7321, reader_cost=0.0011 | ETA 11:41:30 2020-12-26 15:50:32 [INFO] [TRAIN] epoch=150, iter=55800/80000, loss=12.4187, lr=0.003409, batch_cost=1.7324, reader_cost=0.0015 | ETA 11:38:44 2020-12-26 15:53:37 [INFO] [TRAIN] epoch=151, iter=55900/80000, loss=12.2214, lr=0.003397, batch_cost=1.8484, reader_cost=0.0524 | ETA 12:22:27 2020-12-26 15:56:30 [INFO] [TRAIN] epoch=151, iter=56000/80000, loss=12.1896, lr=0.003384, batch_cost=1.7305, reader_cost=0.0018 | ETA 11:32:11 2020-12-26 15:56:30 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-26 15:57:13 [INFO] [EVAL] #Images=500 mIoU=0.7778 Acc=0.9612 Kappa=0.9496 2020-12-26 15:57:13 [INFO] [EVAL] Class IoU: [0.9811 0.8519 0.9285 0.5381 0.6073 0.679 0.7449 0.8243 0.9234 0.6335 0.9461 0.8252 0.637 0.9521 0.6138 0.8455 0.7998 0.6638 0.7833] 2020-12-26 15:57:13 [INFO] [EVAL] Class Acc: [0.9897 0.9263 0.9618 0.8261 0.6956 0.8532 0.8678 0.9058 0.9564 0.8334 0.9606 0.9007 0.7272 0.9694 0.933 0.8919 0.8919 0.812 0.8779] 2020-12-26 15:57:17 [INFO] [EVAL] The model with the best validation mIoU (0.7805) was saved at iter 48000. 2020-12-26 16:00:13 [INFO] [TRAIN] epoch=151, iter=56100/80000, loss=12.1393, lr=0.003371, batch_cost=1.7573, reader_cost=0.0021 | ETA 11:40:00 2020-12-26 16:03:21 [INFO] [TRAIN] epoch=152, iter=56200/80000, loss=12.3410, lr=0.003359, batch_cost=1.8810, reader_cost=0.0509 | ETA 12:26:07 2020-12-26 16:06:17 [INFO] [TRAIN] epoch=152, iter=56300/80000, loss=12.1934, lr=0.003346, batch_cost=1.7616, reader_cost=0.0006 | ETA 11:35:50 2020-12-26 16:09:10 [INFO] [TRAIN] epoch=152, iter=56400/80000, loss=12.2015, lr=0.003333, batch_cost=1.7326, reader_cost=0.0014 | ETA 11:21:28 2020-12-26 16:12:06 [INFO] [TRAIN] epoch=152, iter=56500/80000, loss=12.3765, lr=0.003320, batch_cost=1.7585, reader_cost=0.0014 | ETA 11:28:44 2020-12-26 16:15:10 [INFO] [TRAIN] epoch=153, iter=56600/80000, loss=12.3792, lr=0.003308, batch_cost=1.8326, reader_cost=0.0524 | ETA 11:54:41 2020-12-26 16:18:04 [INFO] [TRAIN] epoch=153, iter=56700/80000, loss=12.1574, lr=0.003295, batch_cost=1.7406, reader_cost=0.0014 | ETA 11:15:56 2020-12-26 16:20:57 [INFO] [TRAIN] epoch=153, iter=56800/80000, loss=12.1572, lr=0.003282, batch_cost=1.7328, reader_cost=0.0015 | ETA 11:10:00 2020-12-26 16:23:51 [INFO] [TRAIN] epoch=153, iter=56900/80000, loss=12.1809, lr=0.003270, batch_cost=1.7303, reader_cost=0.0012 | ETA 11:06:09 2020-12-26 16:26:55 [INFO] [TRAIN] epoch=154, iter=57000/80000, loss=12.0891, lr=0.003257, batch_cost=1.8415, reader_cost=0.0505 | ETA 11:45:53 2020-12-26 16:29:49 [INFO] [TRAIN] epoch=154, iter=57100/80000, loss=12.1696, lr=0.003244, batch_cost=1.7386, reader_cost=0.0014 | ETA 11:03:34 2020-12-26 16:32:43 [INFO] [TRAIN] epoch=154, iter=57200/80000, loss=12.1836, lr=0.003231, batch_cost=1.7377, reader_cost=0.0012 | ETA 11:00:19 2020-12-26 16:35:47 [INFO] [TRAIN] epoch=155, iter=57300/80000, loss=12.2013, lr=0.003219, batch_cost=1.8387, reader_cost=0.0502 | ETA 11:35:39 2020-12-26 16:38:40 [INFO] [TRAIN] epoch=155, iter=57400/80000, loss=12.1819, lr=0.003206, batch_cost=1.7319, reader_cost=0.0011 | ETA 10:52:21 2020-12-26 16:41:34 [INFO] [TRAIN] epoch=155, iter=57500/80000, loss=12.1323, lr=0.003193, batch_cost=1.7402, reader_cost=0.0014 | ETA 10:52:34 2020-12-26 16:44:28 [INFO] [TRAIN] epoch=155, iter=57600/80000, loss=12.2070, lr=0.003180, batch_cost=1.7416, reader_cost=0.0015 | ETA 10:50:11 2020-12-26 16:47:33 [INFO] [TRAIN] epoch=156, iter=57700/80000, loss=12.3783, lr=0.003167, batch_cost=1.8415, reader_cost=0.0497 | ETA 11:24:25 2020-12-26 16:50:26 [INFO] [TRAIN] epoch=156, iter=57800/80000, loss=12.1456, lr=0.003155, batch_cost=1.7301, reader_cost=0.0011 | ETA 10:40:07 2020-12-26 16:53:20 [INFO] [TRAIN] epoch=156, iter=57900/80000, loss=12.2731, lr=0.003142, batch_cost=1.7368, reader_cost=0.0008 | ETA 10:39:43 2020-12-26 16:56:13 [INFO] [TRAIN] epoch=156, iter=58000/80000, loss=12.2506, lr=0.003129, batch_cost=1.7338, reader_cost=0.0013 | ETA 10:35:44 2020-12-26 16:56:13 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-26 16:56:56 [INFO] [EVAL] #Images=500 mIoU=0.7056 Acc=0.9325 Kappa=0.9130 2020-12-26 16:56:56 [INFO] [EVAL] Class IoU: [0.9424 0.7875 0.9038 0.3322 0.4363 0.5655 0.678 0.8113 0.877 0.5601 0.9497 0.8128 0.6144 0.7989 0.7054 0.8044 0.612 0.5392 0.6764] 2020-12-26 16:56:56 [INFO] [EVAL] Class Acc: [0.9926 0.8721 0.9432 0.7146 0.5559 0.6564 0.8819 0.9161 0.9352 0.8668 0.9729 0.8732 0.7559 0.815 0.9194 0.9132 0.8921 0.6036 0.7477] 2020-12-26 16:57:00 [INFO] [EVAL] The model with the best validation mIoU (0.7805) was saved at iter 48000. 2020-12-26 17:00:05 [INFO] [TRAIN] epoch=157, iter=58100/80000, loss=12.2290, lr=0.003116, batch_cost=1.8432, reader_cost=0.0481 | ETA 11:12:45 2020-12-26 17:02:59 [INFO] [TRAIN] epoch=157, iter=58200/80000, loss=12.1554, lr=0.003103, batch_cost=1.7358, reader_cost=0.0016 | ETA 10:30:40 2020-12-26 17:05:54 [INFO] [TRAIN] epoch=157, iter=58300/80000, loss=12.0907, lr=0.003091, batch_cost=1.7512, reader_cost=0.0017 | ETA 10:33:21 2020-12-26 17:08:50 [INFO] [TRAIN] epoch=157, iter=58400/80000, loss=12.3122, lr=0.003078, batch_cost=1.7558, reader_cost=0.0010 | ETA 10:32:05 2020-12-26 17:11:55 [INFO] [TRAIN] epoch=158, iter=58500/80000, loss=12.0824, lr=0.003065, batch_cost=1.8528, reader_cost=0.0480 | ETA 11:03:55 2020-12-26 17:14:48 [INFO] [TRAIN] epoch=158, iter=58600/80000, loss=12.1271, lr=0.003052, batch_cost=1.7288, reader_cost=0.0013 | ETA 10:16:36 2020-12-26 17:17:43 [INFO] [TRAIN] epoch=158, iter=58700/80000, loss=12.1814, lr=0.003039, batch_cost=1.7534, reader_cost=0.0014 | ETA 10:22:27 2020-12-26 17:20:49 [INFO] [TRAIN] epoch=159, iter=58800/80000, loss=12.2623, lr=0.003026, batch_cost=1.8563, reader_cost=0.0542 | ETA 10:55:53 2020-12-26 17:23:43 [INFO] [TRAIN] epoch=159, iter=58900/80000, loss=12.0938, lr=0.003014, batch_cost=1.7355, reader_cost=0.0013 | ETA 10:10:19 2020-12-26 17:26:37 [INFO] [TRAIN] epoch=159, iter=59000/80000, loss=12.0841, lr=0.003001, batch_cost=1.7381, reader_cost=0.0015 | ETA 10:08:19 2020-12-26 17:29:30 [INFO] [TRAIN] epoch=159, iter=59100/80000, loss=12.1571, lr=0.002988, batch_cost=1.7297, reader_cost=0.0016 | ETA 10:02:29 2020-12-26 17:32:35 [INFO] [TRAIN] epoch=160, iter=59200/80000, loss=12.2083, lr=0.002975, batch_cost=1.8456, reader_cost=0.0513 | ETA 10:39:48 2020-12-26 17:35:27 [INFO] [TRAIN] epoch=160, iter=59300/80000, loss=12.1346, lr=0.002962, batch_cost=1.7277, reader_cost=0.0013 | ETA 09:56:03 2020-12-26 17:38:21 [INFO] [TRAIN] epoch=160, iter=59400/80000, loss=12.1724, lr=0.002949, batch_cost=1.7352, reader_cost=0.0008 | ETA 09:55:44 2020-12-26 17:41:14 [INFO] [TRAIN] epoch=160, iter=59500/80000, loss=12.3384, lr=0.002936, batch_cost=1.7321, reader_cost=0.0011 | ETA 09:51:48 2020-12-26 17:44:18 [INFO] [TRAIN] epoch=161, iter=59600/80000, loss=12.2331, lr=0.002924, batch_cost=1.8306, reader_cost=0.0508 | ETA 10:22:23 2020-12-26 17:47:10 [INFO] [TRAIN] epoch=161, iter=59700/80000, loss=12.1879, lr=0.002911, batch_cost=1.7271, reader_cost=0.0009 | ETA 09:44:20 2020-12-26 17:50:03 [INFO] [TRAIN] epoch=161, iter=59800/80000, loss=12.1321, lr=0.002898, batch_cost=1.7284, reader_cost=0.0008 | ETA 09:41:54 2020-12-26 17:53:08 [INFO] [TRAIN] epoch=162, iter=59900/80000, loss=12.2944, lr=0.002885, batch_cost=1.8423, reader_cost=0.0495 | ETA 10:17:10 2020-12-26 17:56:00 [INFO] [TRAIN] epoch=162, iter=60000/80000, loss=12.1266, lr=0.002872, batch_cost=1.7240, reader_cost=0.0013 | ETA 09:34:39 2020-12-26 17:56:00 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-26 17:56:43 [INFO] [EVAL] #Images=500 mIoU=0.7922 Acc=0.9609 Kappa=0.9492 2020-12-26 17:56:43 [INFO] [EVAL] Class IoU: [0.9784 0.8352 0.9275 0.5631 0.5948 0.69 0.7445 0.8232 0.9249 0.6385 0.9474 0.8348 0.6528 0.9519 0.7962 0.893 0.8398 0.6464 0.7692] 2020-12-26 17:56:43 [INFO] [EVAL] Class Acc: [0.9921 0.8891 0.9551 0.8496 0.8434 0.8543 0.8594 0.9128 0.9628 0.7356 0.9612 0.9351 0.8291 0.9712 0.9494 0.9629 0.9444 0.8387 0.8086] 2020-12-26 17:56:50 [INFO] [EVAL] The model with the best validation mIoU (0.7922) was saved at iter 60000. 2020-12-26 17:59:43 [INFO] [TRAIN] epoch=162, iter=60100/80000, loss=12.0871, lr=0.002859, batch_cost=1.7338, reader_cost=0.0014 | ETA 09:35:02 2020-12-26 18:02:37 [INFO] [TRAIN] epoch=162, iter=60200/80000, loss=12.1096, lr=0.002846, batch_cost=1.7358, reader_cost=0.0009 | ETA 09:32:48 2020-12-26 18:05:42 [INFO] [TRAIN] epoch=163, iter=60300/80000, loss=12.2795, lr=0.002833, batch_cost=1.8490, reader_cost=0.0489 | ETA 10:07:06 2020-12-26 18:08:38 [INFO] [TRAIN] epoch=163, iter=60400/80000, loss=12.0062, lr=0.002820, batch_cost=1.7632, reader_cost=0.0016 | ETA 09:35:58 2020-12-26 18:11:32 [INFO] [TRAIN] epoch=163, iter=60500/80000, loss=11.9971, lr=0.002807, batch_cost=1.7367, reader_cost=0.0014 | ETA 09:24:26 2020-12-26 18:14:27 [INFO] [TRAIN] epoch=163, iter=60600/80000, loss=12.2462, lr=0.002794, batch_cost=1.7504, reader_cost=0.0008 | ETA 09:25:57 2020-12-26 18:17:33 [INFO] [TRAIN] epoch=164, iter=60700/80000, loss=12.1804, lr=0.002781, batch_cost=1.8538, reader_cost=0.0533 | ETA 09:56:18 2020-12-26 18:20:27 [INFO] [TRAIN] epoch=164, iter=60800/80000, loss=12.1986, lr=0.002768, batch_cost=1.7402, reader_cost=0.0010 | ETA 09:16:52 2020-12-26 18:23:22 [INFO] [TRAIN] epoch=164, iter=60900/80000, loss=12.0841, lr=0.002755, batch_cost=1.7460, reader_cost=0.0017 | ETA 09:15:48 2020-12-26 18:26:15 [INFO] [TRAIN] epoch=164, iter=61000/80000, loss=12.3801, lr=0.002742, batch_cost=1.7373, reader_cost=0.0017 | ETA 09:10:07 2020-12-26 18:29:20 [INFO] [TRAIN] epoch=165, iter=61100/80000, loss=12.1595, lr=0.002729, batch_cost=1.8410, reader_cost=0.0483 | ETA 09:39:54 2020-12-26 18:32:13 [INFO] [TRAIN] epoch=165, iter=61200/80000, loss=12.0142, lr=0.002716, batch_cost=1.7330, reader_cost=0.0008 | ETA 09:03:01 2020-12-26 18:35:06 [INFO] [TRAIN] epoch=165, iter=61300/80000, loss=12.1615, lr=0.002703, batch_cost=1.7316, reader_cost=0.0015 | ETA 08:59:41 2020-12-26 18:38:11 [INFO] [TRAIN] epoch=166, iter=61400/80000, loss=12.2114, lr=0.002690, batch_cost=1.8488, reader_cost=0.0546 | ETA 09:33:08 2020-12-26 18:41:05 [INFO] [TRAIN] epoch=166, iter=61500/80000, loss=12.0381, lr=0.002677, batch_cost=1.7363, reader_cost=0.0016 | ETA 08:55:22 2020-12-26 18:43:59 [INFO] [TRAIN] epoch=166, iter=61600/80000, loss=12.0593, lr=0.002664, batch_cost=1.7348, reader_cost=0.0015 | ETA 08:52:01 2020-12-26 18:46:53 [INFO] [TRAIN] epoch=166, iter=61700/80000, loss=12.2018, lr=0.002651, batch_cost=1.7381, reader_cost=0.0020 | ETA 08:50:07 2020-12-26 18:49:57 [INFO] [TRAIN] epoch=167, iter=61800/80000, loss=12.1528, lr=0.002638, batch_cost=1.8432, reader_cost=0.0481 | ETA 09:19:06 2020-12-26 18:52:51 [INFO] [TRAIN] epoch=167, iter=61900/80000, loss=11.9820, lr=0.002625, batch_cost=1.7362, reader_cost=0.0012 | ETA 08:43:44 2020-12-26 18:55:46 [INFO] [TRAIN] epoch=167, iter=62000/80000, loss=12.0797, lr=0.002612, batch_cost=1.7497, reader_cost=0.0016 | ETA 08:44:55 2020-12-26 18:55:46 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-26 18:56:29 [INFO] [EVAL] #Images=500 mIoU=0.7783 Acc=0.9609 Kappa=0.9493 2020-12-26 18:56:29 [INFO] [EVAL] Class IoU: [0.9805 0.8381 0.9245 0.4791 0.5979 0.6921 0.7474 0.812 0.9281 0.5951 0.9496 0.8312 0.649 0.9535 0.7738 0.8879 0.8062 0.5557 0.7852] 2020-12-26 18:56:29 [INFO] [EVAL] Class Acc: [0.9931 0.8898 0.9508 0.8654 0.7569 0.8583 0.8743 0.8935 0.9617 0.8605 0.9692 0.8862 0.846 0.975 0.877 0.9516 0.9373 0.9216 0.8494] 2020-12-26 18:56:33 [INFO] [EVAL] The model with the best validation mIoU (0.7922) was saved at iter 60000. 2020-12-26 18:59:26 [INFO] [TRAIN] epoch=167, iter=62100/80000, loss=12.3061, lr=0.002599, batch_cost=1.7256, reader_cost=0.0014 | ETA 08:34:47 2020-12-26 19:02:31 [INFO] [TRAIN] epoch=168, iter=62200/80000, loss=12.1174, lr=0.002586, batch_cost=1.8475, reader_cost=0.0494 | ETA 09:08:05 2020-12-26 19:05:24 [INFO] [TRAIN] epoch=168, iter=62300/80000, loss=11.9427, lr=0.002573, batch_cost=1.7302, reader_cost=0.0016 | ETA 08:30:23 2020-12-26 19:08:18 [INFO] [TRAIN] epoch=168, iter=62400/80000, loss=12.1590, lr=0.002560, batch_cost=1.7448, reader_cost=0.0018 | ETA 08:31:48 2020-12-26 19:11:24 [INFO] [TRAIN] epoch=169, iter=62500/80000, loss=12.3009, lr=0.002547, batch_cost=1.8501, reader_cost=0.0483 | ETA 08:59:36 2020-12-26 19:14:17 [INFO] [TRAIN] epoch=169, iter=62600/80000, loss=12.2016, lr=0.002534, batch_cost=1.7298, reader_cost=0.0007 | ETA 08:21:38 2020-12-26 19:17:10 [INFO] [TRAIN] epoch=169, iter=62700/80000, loss=12.1296, lr=0.002520, batch_cost=1.7350, reader_cost=0.0007 | ETA 08:20:14 2020-12-26 19:20:03 [INFO] [TRAIN] epoch=169, iter=62800/80000, loss=12.1978, lr=0.002507, batch_cost=1.7302, reader_cost=0.0017 | ETA 08:15:59 2020-12-26 19:23:08 [INFO] [TRAIN] epoch=170, iter=62900/80000, loss=12.1311, lr=0.002494, batch_cost=1.8416, reader_cost=0.0498 | ETA 08:44:50 2020-12-26 19:26:01 [INFO] [TRAIN] epoch=170, iter=63000/80000, loss=12.0473, lr=0.002481, batch_cost=1.7271, reader_cost=0.0017 | ETA 08:09:20 2020-12-26 19:28:54 [INFO] [TRAIN] epoch=170, iter=63100/80000, loss=11.9726, lr=0.002468, batch_cost=1.7302, reader_cost=0.0014 | ETA 08:07:20 2020-12-26 19:31:47 [INFO] [TRAIN] epoch=170, iter=63200/80000, loss=12.3057, lr=0.002455, batch_cost=1.7318, reader_cost=0.0015 | ETA 08:04:53 2020-12-26 19:34:52 [INFO] [TRAIN] epoch=171, iter=63300/80000, loss=12.2668, lr=0.002442, batch_cost=1.8434, reader_cost=0.0502 | ETA 08:33:05 2020-12-26 19:37:45 [INFO] [TRAIN] epoch=171, iter=63400/80000, loss=12.1098, lr=0.002429, batch_cost=1.7334, reader_cost=0.0012 | ETA 07:59:34 2020-12-26 19:40:39 [INFO] [TRAIN] epoch=171, iter=63500/80000, loss=12.0385, lr=0.002415, batch_cost=1.7411, reader_cost=0.0013 | ETA 07:58:47 2020-12-26 19:43:35 [INFO] [TRAIN] epoch=171, iter=63600/80000, loss=12.3422, lr=0.002402, batch_cost=1.7577, reader_cost=0.0012 | ETA 08:00:25 2020-12-26 19:46:40 [INFO] [TRAIN] epoch=172, iter=63700/80000, loss=11.9186, lr=0.002389, batch_cost=1.8482, reader_cost=0.0480 | ETA 08:22:05 2020-12-26 19:49:35 [INFO] [TRAIN] epoch=172, iter=63800/80000, loss=12.0148, lr=0.002376, batch_cost=1.7436, reader_cost=0.0021 | ETA 07:50:46 2020-12-26 19:52:28 [INFO] [TRAIN] epoch=172, iter=63900/80000, loss=12.1422, lr=0.002363, batch_cost=1.7286, reader_cost=0.0021 | ETA 07:43:50 2020-12-26 19:55:33 [INFO] [TRAIN] epoch=173, iter=64000/80000, loss=12.2394, lr=0.002349, batch_cost=1.8517, reader_cost=0.0517 | ETA 08:13:47 2020-12-26 19:55:33 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-26 19:56:16 [INFO] [EVAL] #Images=500 mIoU=0.7808 Acc=0.9619 Kappa=0.9505 2020-12-26 19:56:16 [INFO] [EVAL] Class IoU: [0.9821 0.8529 0.9259 0.4368 0.6042 0.7002 0.7447 0.8217 0.9271 0.6462 0.9518 0.8389 0.6371 0.9545 0.7479 0.8399 0.7682 0.6628 0.7924] 2020-12-26 19:56:16 [INFO] [EVAL] Class Acc: [0.9931 0.9016 0.9488 0.8245 0.8011 0.8244 0.8873 0.9285 0.9657 0.8565 0.9678 0.9291 0.7045 0.982 0.8071 0.9634 0.8319 0.8193 0.8552] 2020-12-26 19:56:20 [INFO] [EVAL] The model with the best validation mIoU (0.7922) was saved at iter 60000. 2020-12-26 19:59:12 [INFO] [TRAIN] epoch=173, iter=64100/80000, loss=12.0472, lr=0.002336, batch_cost=1.7233, reader_cost=0.0013 | ETA 07:36:41 2020-12-26 20:02:05 [INFO] [TRAIN] epoch=173, iter=64200/80000, loss=11.9922, lr=0.002323, batch_cost=1.7279, reader_cost=0.0015 | ETA 07:35:00 2020-12-26 20:04:58 [INFO] [TRAIN] epoch=173, iter=64300/80000, loss=12.1588, lr=0.002310, batch_cost=1.7271, reader_cost=0.0019 | ETA 07:31:56 2020-12-26 20:08:03 [INFO] [TRAIN] epoch=174, iter=64400/80000, loss=12.1586, lr=0.002296, batch_cost=1.8454, reader_cost=0.0499 | ETA 07:59:47 2020-12-26 20:10:55 [INFO] [TRAIN] epoch=174, iter=64500/80000, loss=11.9284, lr=0.002283, batch_cost=1.7249, reader_cost=0.0011 | ETA 07:25:36 2020-12-26 20:13:47 [INFO] [TRAIN] epoch=174, iter=64600/80000, loss=11.8726, lr=0.002270, batch_cost=1.7188, reader_cost=0.0014 | ETA 07:21:09 2020-12-26 20:16:41 [INFO] [TRAIN] epoch=174, iter=64700/80000, loss=12.1742, lr=0.002257, batch_cost=1.7351, reader_cost=0.0014 | ETA 07:22:27 2020-12-26 20:19:46 [INFO] [TRAIN] epoch=175, iter=64800/80000, loss=12.1630, lr=0.002243, batch_cost=1.8435, reader_cost=0.0505 | ETA 07:47:00 2020-12-26 20:22:38 [INFO] [TRAIN] epoch=175, iter=64900/80000, loss=11.9724, lr=0.002230, batch_cost=1.7273, reader_cost=0.0013 | ETA 07:14:41 2020-12-26 20:25:32 [INFO] [TRAIN] epoch=175, iter=65000/80000, loss=12.1048, lr=0.002217, batch_cost=1.7329, reader_cost=0.0022 | ETA 07:13:13 2020-12-26 20:28:25 [INFO] [TRAIN] epoch=175, iter=65100/80000, loss=12.2963, lr=0.002203, batch_cost=1.7295, reader_cost=0.0011 | ETA 07:09:29 2020-12-26 20:31:30 [INFO] [TRAIN] epoch=176, iter=65200/80000, loss=11.9182, lr=0.002190, batch_cost=1.8470, reader_cost=0.0472 | ETA 07:35:35 2020-12-26 20:34:24 [INFO] [TRAIN] epoch=176, iter=65300/80000, loss=11.9901, lr=0.002177, batch_cost=1.7412, reader_cost=0.0009 | ETA 07:06:35 2020-12-26 20:37:18 [INFO] [TRAIN] epoch=176, iter=65400/80000, loss=12.2106, lr=0.002164, batch_cost=1.7426, reader_cost=0.0009 | ETA 07:04:01 2020-12-26 20:40:24 [INFO] [TRAIN] epoch=177, iter=65500/80000, loss=12.2552, lr=0.002150, batch_cost=1.8602, reader_cost=0.0483 | ETA 07:29:33 2020-12-26 20:43:20 [INFO] [TRAIN] epoch=177, iter=65600/80000, loss=12.0795, lr=0.002137, batch_cost=1.7575, reader_cost=0.0010 | ETA 07:01:48 2020-12-26 20:46:14 [INFO] [TRAIN] epoch=177, iter=65700/80000, loss=11.9386, lr=0.002123, batch_cost=1.7334, reader_cost=0.0013 | ETA 06:53:08 2020-12-26 20:49:09 [INFO] [TRAIN] epoch=177, iter=65800/80000, loss=12.2242, lr=0.002110, batch_cost=1.7537, reader_cost=0.0017 | ETA 06:55:02 2020-12-26 20:52:15 [INFO] [TRAIN] epoch=178, iter=65900/80000, loss=12.0467, lr=0.002097, batch_cost=1.8568, reader_cost=0.0491 | ETA 07:16:21 2020-12-26 20:55:09 [INFO] [TRAIN] epoch=178, iter=66000/80000, loss=12.0928, lr=0.002083, batch_cost=1.7360, reader_cost=0.0015 | ETA 06:45:03 2020-12-26 20:55:09 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-26 20:55:52 [INFO] [EVAL] #Images=500 mIoU=0.7903 Acc=0.9617 Kappa=0.9503 2020-12-26 20:55:52 [INFO] [EVAL] Class IoU: [0.98 0.8362 0.9295 0.4616 0.6138 0.6973 0.7491 0.8245 0.9247 0.6468 0.953 0.8413 0.6698 0.9547 0.7717 0.8839 0.8252 0.655 0.7976] 2020-12-26 20:55:52 [INFO] [EVAL] Class Acc: [0.9888 0.9246 0.9598 0.8033 0.8321 0.8245 0.8545 0.9122 0.9518 0.8478 0.9722 0.9102 0.7726 0.9729 0.9274 0.9254 0.91 0.7428 0.8822] 2020-12-26 20:55:56 [INFO] [EVAL] The model with the best validation mIoU (0.7922) was saved at iter 60000. 2020-12-26 20:58:49 [INFO] [TRAIN] epoch=178, iter=66100/80000, loss=12.0437, lr=0.002070, batch_cost=1.7314, reader_cost=0.0020 | ETA 06:41:06 2020-12-26 21:01:43 [INFO] [TRAIN] epoch=178, iter=66200/80000, loss=12.1868, lr=0.002057, batch_cost=1.7383, reader_cost=0.0014 | ETA 06:39:49 2020-12-26 21:04:51 [INFO] [TRAIN] epoch=179, iter=66300/80000, loss=12.1094, lr=0.002043, batch_cost=1.8732, reader_cost=0.0531 | ETA 07:07:43 2020-12-26 21:07:44 [INFO] [TRAIN] epoch=179, iter=66400/80000, loss=12.0841, lr=0.002030, batch_cost=1.7350, reader_cost=0.0008 | ETA 06:33:15 2020-12-26 21:10:37 [INFO] [TRAIN] epoch=179, iter=66500/80000, loss=11.9860, lr=0.002016, batch_cost=1.7284, reader_cost=0.0011 | ETA 06:28:53 2020-12-26 21:13:43 [INFO] [TRAIN] epoch=180, iter=66600/80000, loss=12.2355, lr=0.002003, batch_cost=1.8510, reader_cost=0.0496 | ETA 06:53:22 2020-12-26 21:16:35 [INFO] [TRAIN] epoch=180, iter=66700/80000, loss=12.0499, lr=0.001989, batch_cost=1.7242, reader_cost=0.0009 | ETA 06:22:11 2020-12-26 21:19:28 [INFO] [TRAIN] epoch=180, iter=66800/80000, loss=12.0352, lr=0.001976, batch_cost=1.7278, reader_cost=0.0008 | ETA 06:20:06 2020-12-26 21:22:21 [INFO] [TRAIN] epoch=180, iter=66900/80000, loss=12.1766, lr=0.001962, batch_cost=1.7260, reader_cost=0.0007 | ETA 06:16:50 2020-12-26 21:25:26 [INFO] [TRAIN] epoch=181, iter=67000/80000, loss=12.1825, lr=0.001949, batch_cost=1.8498, reader_cost=0.0515 | ETA 06:40:47 2020-12-26 21:28:19 [INFO] [TRAIN] epoch=181, iter=67100/80000, loss=11.9684, lr=0.001935, batch_cost=1.7357, reader_cost=0.0011 | ETA 06:13:10 2020-12-26 21:31:12 [INFO] [TRAIN] epoch=181, iter=67200/80000, loss=12.0231, lr=0.001922, batch_cost=1.7245, reader_cost=0.0010 | ETA 06:07:53 2020-12-26 21:34:04 [INFO] [TRAIN] epoch=181, iter=67300/80000, loss=12.1412, lr=0.001908, batch_cost=1.7229, reader_cost=0.0013 | ETA 06:04:41 2020-12-26 21:37:09 [INFO] [TRAIN] epoch=182, iter=67400/80000, loss=12.0938, lr=0.001895, batch_cost=1.8452, reader_cost=0.0487 | ETA 06:27:28 2020-12-26 21:40:02 [INFO] [TRAIN] epoch=182, iter=67500/80000, loss=11.9865, lr=0.001881, batch_cost=1.7259, reader_cost=0.0021 | ETA 05:59:33 2020-12-26 21:42:55 [INFO] [TRAIN] epoch=182, iter=67600/80000, loss=12.0007, lr=0.001868, batch_cost=1.7313, reader_cost=0.0015 | ETA 05:57:48 2020-12-26 21:45:49 [INFO] [TRAIN] epoch=182, iter=67700/80000, loss=12.2574, lr=0.001854, batch_cost=1.7346, reader_cost=0.0013 | ETA 05:55:35 2020-12-26 21:48:54 [INFO] [TRAIN] epoch=183, iter=67800/80000, loss=11.9816, lr=0.001841, batch_cost=1.8548, reader_cost=0.0484 | ETA 06:17:08 2020-12-26 21:51:48 [INFO] [TRAIN] epoch=183, iter=67900/80000, loss=12.0795, lr=0.001827, batch_cost=1.7392, reader_cost=0.0011 | ETA 05:50:44 2020-12-26 21:54:42 [INFO] [TRAIN] epoch=183, iter=68000/80000, loss=11.9896, lr=0.001813, batch_cost=1.7387, reader_cost=0.0012 | ETA 05:47:44 2020-12-26 21:54:42 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-26 21:55:26 [INFO] [EVAL] #Images=500 mIoU=0.7900 Acc=0.9634 Kappa=0.9524 2020-12-26 21:55:26 [INFO] [EVAL] Class IoU: [0.9796 0.8522 0.9347 0.4595 0.6338 0.6967 0.7555 0.8307 0.929 0.6449 0.9544 0.8451 0.6663 0.9577 0.8093 0.8723 0.7073 0.681 0.7995] 2020-12-26 21:55:26 [INFO] [EVAL] Class Acc: [0.9894 0.9072 0.9631 0.7897 0.8276 0.8579 0.857 0.921 0.9599 0.8151 0.9696 0.9048 0.8127 0.9763 0.9108 0.9152 0.9654 0.8253 0.8742] 2020-12-26 21:55:30 [INFO] [EVAL] The model with the best validation mIoU (0.7922) was saved at iter 60000. 2020-12-26 21:58:35 [INFO] [TRAIN] epoch=184, iter=68100/80000, loss=12.0085, lr=0.001800, batch_cost=1.8578, reader_cost=0.0470 | ETA 06:08:27 2020-12-26 22:01:28 [INFO] [TRAIN] epoch=184, iter=68200/80000, loss=11.8959, lr=0.001786, batch_cost=1.7295, reader_cost=0.0016 | ETA 05:40:08 2020-12-26 22:04:22 [INFO] [TRAIN] epoch=184, iter=68300/80000, loss=11.9442, lr=0.001773, batch_cost=1.7299, reader_cost=0.0017 | ETA 05:37:19 2020-12-26 22:07:15 [INFO] [TRAIN] epoch=184, iter=68400/80000, loss=12.0427, lr=0.001759, batch_cost=1.7320, reader_cost=0.0015 | ETA 05:34:51 2020-12-26 22:10:20 [INFO] [TRAIN] epoch=185, iter=68500/80000, loss=12.0605, lr=0.001745, batch_cost=1.8491, reader_cost=0.0469 | ETA 05:54:25 2020-12-26 22:13:13 [INFO] [TRAIN] epoch=185, iter=68600/80000, loss=11.9675, lr=0.001732, batch_cost=1.7322, reader_cost=0.0012 | ETA 05:29:07 2020-12-26 22:16:07 [INFO] [TRAIN] epoch=185, iter=68700/80000, loss=11.9967, lr=0.001718, batch_cost=1.7396, reader_cost=0.0018 | ETA 05:27:37 2020-12-26 22:19:01 [INFO] [TRAIN] epoch=185, iter=68800/80000, loss=12.1997, lr=0.001704, batch_cost=1.7355, reader_cost=0.0018 | ETA 05:23:57 2020-12-26 22:22:06 [INFO] [TRAIN] epoch=186, iter=68900/80000, loss=12.0693, lr=0.001691, batch_cost=1.8473, reader_cost=0.0499 | ETA 05:41:44 2020-12-26 22:24:59 [INFO] [TRAIN] epoch=186, iter=69000/80000, loss=12.0174, lr=0.001677, batch_cost=1.7294, reader_cost=0.0014 | ETA 05:17:03 2020-12-26 22:27:53 [INFO] [TRAIN] epoch=186, iter=69100/80000, loss=12.1188, lr=0.001663, batch_cost=1.7339, reader_cost=0.0015 | ETA 05:14:59 2020-12-26 22:30:59 [INFO] [TRAIN] epoch=187, iter=69200/80000, loss=12.1269, lr=0.001649, batch_cost=1.8605, reader_cost=0.0527 | ETA 05:34:53 2020-12-26 22:33:52 [INFO] [TRAIN] epoch=187, iter=69300/80000, loss=11.9997, lr=0.001636, batch_cost=1.7347, reader_cost=0.0008 | ETA 05:09:20 2020-12-26 22:36:46 [INFO] [TRAIN] epoch=187, iter=69400/80000, loss=12.0671, lr=0.001622, batch_cost=1.7357, reader_cost=0.0005 | ETA 05:06:38 2020-12-26 22:39:40 [INFO] [TRAIN] epoch=187, iter=69500/80000, loss=12.0928, lr=0.001608, batch_cost=1.7405, reader_cost=0.0013 | ETA 05:04:35 2020-12-26 22:42:45 [INFO] [TRAIN] epoch=188, iter=69600/80000, loss=12.1681, lr=0.001594, batch_cost=1.8504, reader_cost=0.0514 | ETA 05:20:44 2020-12-26 22:45:39 [INFO] [TRAIN] epoch=188, iter=69700/80000, loss=11.9609, lr=0.001581, batch_cost=1.7348, reader_cost=0.0021 | ETA 04:57:48 2020-12-26 22:48:33 [INFO] [TRAIN] epoch=188, iter=69800/80000, loss=11.9244, lr=0.001567, batch_cost=1.7333, reader_cost=0.0015 | ETA 04:54:40 2020-12-26 22:51:26 [INFO] [TRAIN] epoch=188, iter=69900/80000, loss=12.2100, lr=0.001553, batch_cost=1.7336, reader_cost=0.0016 | ETA 04:51:49 2020-12-26 22:54:31 [INFO] [TRAIN] epoch=189, iter=70000/80000, loss=12.0018, lr=0.001539, batch_cost=1.8440, reader_cost=0.0496 | ETA 05:07:20 2020-12-26 22:54:31 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-26 22:55:13 [INFO] [EVAL] #Images=500 mIoU=0.7949 Acc=0.9618 Kappa=0.9505 2020-12-26 22:55:13 [INFO] [EVAL] Class IoU: [0.978 0.8275 0.9315 0.5391 0.634 0.6945 0.7537 0.8192 0.9272 0.6007 0.9515 0.8496 0.6731 0.9568 0.8017 0.8893 0.7968 0.6808 0.7984] 2020-12-26 22:55:13 [INFO] [EVAL] Class Acc: [0.9932 0.8706 0.962 0.8991 0.8133 0.8287 0.8448 0.9164 0.956 0.8541 0.9665 0.9147 0.8107 0.9732 0.934 0.9514 0.9352 0.8237 0.8791] 2020-12-26 22:55:20 [INFO] [EVAL] The model with the best validation mIoU (0.7949) was saved at iter 70000. 2020-12-26 22:58:13 [INFO] [TRAIN] epoch=189, iter=70100/80000, loss=11.9708, lr=0.001525, batch_cost=1.7368, reader_cost=0.0018 | ETA 04:46:34 2020-12-26 23:01:08 [INFO] [TRAIN] epoch=189, iter=70200/80000, loss=11.9468, lr=0.001511, batch_cost=1.7446, reader_cost=0.0019 | ETA 04:44:57 2020-12-26 23:04:01 [INFO] [TRAIN] epoch=189, iter=70300/80000, loss=12.1425, lr=0.001497, batch_cost=1.7321, reader_cost=0.0014 | ETA 04:40:01 2020-12-26 23:07:06 [INFO] [TRAIN] epoch=190, iter=70400/80000, loss=12.0353, lr=0.001484, batch_cost=1.8481, reader_cost=0.0507 | ETA 04:55:41 2020-12-26 23:10:00 [INFO] [TRAIN] epoch=190, iter=70500/80000, loss=12.0764, lr=0.001470, batch_cost=1.7336, reader_cost=0.0019 | ETA 04:34:28 2020-12-26 23:12:53 [INFO] [TRAIN] epoch=190, iter=70600/80000, loss=12.0157, lr=0.001456, batch_cost=1.7311, reader_cost=0.0010 | ETA 04:31:12 2020-12-26 23:15:58 [INFO] [TRAIN] epoch=191, iter=70700/80000, loss=12.0823, lr=0.001442, batch_cost=1.8539, reader_cost=0.0481 | ETA 04:47:21 2020-12-26 23:18:52 [INFO] [TRAIN] epoch=191, iter=70800/80000, loss=11.8970, lr=0.001428, batch_cost=1.7310, reader_cost=0.0021 | ETA 04:25:25 2020-12-26 23:21:46 [INFO] [TRAIN] epoch=191, iter=70900/80000, loss=11.9483, lr=0.001414, batch_cost=1.7419, reader_cost=0.0020 | ETA 04:24:11 2020-12-26 23:24:40 [INFO] [TRAIN] epoch=191, iter=71000/80000, loss=12.1032, lr=0.001400, batch_cost=1.7432, reader_cost=0.0019 | ETA 04:21:28 2020-12-26 23:27:46 [INFO] [TRAIN] epoch=192, iter=71100/80000, loss=12.1704, lr=0.001386, batch_cost=1.8530, reader_cost=0.0487 | ETA 04:34:52 2020-12-26 23:30:40 [INFO] [TRAIN] epoch=192, iter=71200/80000, loss=11.9625, lr=0.001372, batch_cost=1.7376, reader_cost=0.0018 | ETA 04:14:50 2020-12-26 23:33:34 [INFO] [TRAIN] epoch=192, iter=71300/80000, loss=12.0747, lr=0.001358, batch_cost=1.7391, reader_cost=0.0010 | ETA 04:12:09 2020-12-26 23:36:27 [INFO] [TRAIN] epoch=192, iter=71400/80000, loss=12.2303, lr=0.001344, batch_cost=1.7321, reader_cost=0.0011 | ETA 04:08:16 2020-12-26 23:39:33 [INFO] [TRAIN] epoch=193, iter=71500/80000, loss=11.9889, lr=0.001330, batch_cost=1.8608, reader_cost=0.0483 | ETA 04:23:36 2020-12-26 23:42:37 [INFO] [TRAIN] epoch=193, iter=71600/80000, loss=11.9475, lr=0.001316, batch_cost=1.8337, reader_cost=0.0016 | ETA 04:16:43 2020-12-26 23:45:42 [INFO] [TRAIN] epoch=193, iter=71700/80000, loss=12.1300, lr=0.001301, batch_cost=1.8495, reader_cost=0.0013 | ETA 04:15:50 2020-12-26 23:49:02 [INFO] [TRAIN] epoch=194, iter=71800/80000, loss=12.1965, lr=0.001287, batch_cost=2.0042, reader_cost=0.0533 | ETA 04:33:54 2020-12-26 23:52:07 [INFO] [TRAIN] epoch=194, iter=71900/80000, loss=12.0079, lr=0.001273, batch_cost=1.8464, reader_cost=0.0020 | ETA 04:09:15 2020-12-26 23:55:12 [INFO] [TRAIN] epoch=194, iter=72000/80000, loss=11.8869, lr=0.001259, batch_cost=1.8484, reader_cost=0.0023 | ETA 04:06:27 2020-12-26 23:55:12 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-26 23:55:56 [INFO] [EVAL] #Images=500 mIoU=0.8014 Acc=0.9640 Kappa=0.9533 2020-12-26 23:55:56 [INFO] [EVAL] Class IoU: [0.9825 0.8575 0.9331 0.5232 0.6467 0.6939 0.7481 0.8127 0.9276 0.6356 0.9525 0.8424 0.6413 0.9564 0.836 0.9051 0.8431 0.6845 0.8041] 2020-12-26 23:55:56 [INFO] [EVAL] Class Acc: [0.991 0.9205 0.9612 0.7988 0.8699 0.8109 0.8765 0.8997 0.9592 0.8288 0.969 0.9082 0.8492 0.9757 0.9351 0.9704 0.9195 0.8314 0.8749] 2020-12-26 23:56:04 [INFO] [EVAL] The model with the best validation mIoU (0.8014) was saved at iter 72000. 2020-12-26 23:59:09 [INFO] [TRAIN] epoch=194, iter=72100/80000, loss=12.1471, lr=0.001245, batch_cost=1.8517, reader_cost=0.0019 | ETA 04:03:48 2020-12-27 00:02:29 [INFO] [TRAIN] epoch=195, iter=72200/80000, loss=12.1758, lr=0.001231, batch_cost=1.9940, reader_cost=0.0520 | ETA 04:19:13 2020-12-27 00:05:34 [INFO] [TRAIN] epoch=195, iter=72300/80000, loss=11.9278, lr=0.001216, batch_cost=1.8545, reader_cost=0.0022 | ETA 03:57:59 2020-12-27 00:08:38 [INFO] [TRAIN] epoch=195, iter=72400/80000, loss=11.8490, lr=0.001202, batch_cost=1.8407, reader_cost=0.0019 | ETA 03:53:09 2020-12-27 00:11:43 [INFO] [TRAIN] epoch=195, iter=72500/80000, loss=12.1792, lr=0.001188, batch_cost=1.8456, reader_cost=0.0013 | ETA 03:50:41 2020-12-27 00:15:02 [INFO] [TRAIN] epoch=196, iter=72600/80000, loss=12.1254, lr=0.001174, batch_cost=1.9906, reader_cost=0.0506 | ETA 04:05:30 2020-12-27 00:18:08 [INFO] [TRAIN] epoch=196, iter=72700/80000, loss=11.9031, lr=0.001159, batch_cost=1.8534, reader_cost=0.0015 | ETA 03:45:30 2020-12-27 00:21:11 [INFO] [TRAIN] epoch=196, iter=72800/80000, loss=11.9360, lr=0.001145, batch_cost=1.8333, reader_cost=0.0015 | ETA 03:39:59 2020-12-27 00:24:16 [INFO] [TRAIN] epoch=196, iter=72900/80000, loss=12.2117, lr=0.001131, batch_cost=1.8516, reader_cost=0.0011 | ETA 03:39:06 2020-12-27 00:27:36 [INFO] [TRAIN] epoch=197, iter=73000/80000, loss=12.0401, lr=0.001117, batch_cost=1.9910, reader_cost=0.0540 | ETA 03:52:16 2020-12-27 00:30:41 [INFO] [TRAIN] epoch=197, iter=73100/80000, loss=11.9938, lr=0.001102, batch_cost=1.8504, reader_cost=0.0017 | ETA 03:32:48 2020-12-27 00:33:46 [INFO] [TRAIN] epoch=197, iter=73200/80000, loss=11.9113, lr=0.001088, batch_cost=1.8522, reader_cost=0.0016 | ETA 03:29:55 2020-12-27 00:37:05 [INFO] [TRAIN] epoch=198, iter=73300/80000, loss=12.0681, lr=0.001073, batch_cost=1.9883, reader_cost=0.0496 | ETA 03:42:01 2020-12-27 00:40:10 [INFO] [TRAIN] epoch=198, iter=73400/80000, loss=11.9592, lr=0.001059, batch_cost=1.8479, reader_cost=0.0020 | ETA 03:23:16 2020-12-27 00:43:17 [INFO] [TRAIN] epoch=198, iter=73500/80000, loss=11.8497, lr=0.001044, batch_cost=1.8659, reader_cost=0.0018 | ETA 03:22:08 2020-12-27 00:46:22 [INFO] [TRAIN] epoch=198, iter=73600/80000, loss=12.0989, lr=0.001030, batch_cost=1.8519, reader_cost=0.0013 | ETA 03:17:31 2020-12-27 00:49:42 [INFO] [TRAIN] epoch=199, iter=73700/80000, loss=12.0665, lr=0.001016, batch_cost=1.9944, reader_cost=0.0548 | ETA 03:29:24 2020-12-27 00:52:47 [INFO] [TRAIN] epoch=199, iter=73800/80000, loss=11.9482, lr=0.001001, batch_cost=1.8540, reader_cost=0.0019 | ETA 03:11:34 2020-12-27 00:55:53 [INFO] [TRAIN] epoch=199, iter=73900/80000, loss=11.8223, lr=0.000986, batch_cost=1.8549, reader_cost=0.0021 | ETA 03:08:34 2020-12-27 00:58:58 [INFO] [TRAIN] epoch=199, iter=74000/80000, loss=12.1380, lr=0.000972, batch_cost=1.8533, reader_cost=0.0024 | ETA 03:05:19 2020-12-27 00:58:58 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-27 00:59:43 [INFO] [EVAL] #Images=500 mIoU=0.8041 Acc=0.9648 Kappa=0.9543 2020-12-27 00:59:43 [INFO] [EVAL] Class IoU: [0.983 0.8618 0.9327 0.5161 0.6302 0.7043 0.7617 0.8369 0.9299 0.6586 0.9529 0.8475 0.6699 0.9572 0.8049 0.9075 0.8474 0.6771 0.7993] 2020-12-27 00:59:43 [INFO] [EVAL] Class Acc: [0.9918 0.9214 0.9605 0.8441 0.8229 0.8666 0.8548 0.9156 0.9581 0.8372 0.9699 0.9106 0.7963 0.9764 0.8942 0.9626 0.9554 0.8041 0.8689] 2020-12-27 00:59:49 [INFO] [EVAL] The model with the best validation mIoU (0.8041) was saved at iter 74000. 2020-12-27 01:03:03 [INFO] [TRAIN] epoch=200, iter=74100/80000, loss=12.0050, lr=0.000957, batch_cost=1.9364, reader_cost=0.0502 | ETA 03:10:24 2020-12-27 01:06:08 [INFO] [TRAIN] epoch=200, iter=74200/80000, loss=11.9629, lr=0.000943, batch_cost=1.8516, reader_cost=0.0016 | ETA 02:58:59 2020-12-27 01:09:15 [INFO] [TRAIN] epoch=200, iter=74300/80000, loss=11.9880, lr=0.000928, batch_cost=1.8629, reader_cost=0.0018 | ETA 02:56:58 2020-12-27 01:12:21 [INFO] [TRAIN] epoch=200, iter=74400/80000, loss=12.1536, lr=0.000913, batch_cost=1.8631, reader_cost=0.0013 | ETA 02:53:53 2020-12-27 01:15:41 [INFO] [TRAIN] epoch=201, iter=74500/80000, loss=11.8712, lr=0.000899, batch_cost=1.9979, reader_cost=0.0494 | ETA 03:03:08 2020-12-27 01:18:46 [INFO] [TRAIN] epoch=201, iter=74600/80000, loss=11.9493, lr=0.000884, batch_cost=1.8519, reader_cost=0.0020 | ETA 02:46:40 2020-12-27 01:21:52 [INFO] [TRAIN] epoch=201, iter=74700/80000, loss=12.0074, lr=0.000869, batch_cost=1.8520, reader_cost=0.0017 | ETA 02:43:35 2020-12-27 01:25:10 [INFO] [TRAIN] epoch=202, iter=74800/80000, loss=12.2191, lr=0.000854, batch_cost=1.9829, reader_cost=0.0508 | ETA 02:51:51 2020-12-27 01:28:16 [INFO] [TRAIN] epoch=202, iter=74900/80000, loss=11.9221, lr=0.000840, batch_cost=1.8539, reader_cost=0.0017 | ETA 02:37:34 2020-12-27 01:31:21 [INFO] [TRAIN] epoch=202, iter=75000/80000, loss=11.7415, lr=0.000825, batch_cost=1.8565, reader_cost=0.0020 | ETA 02:34:42 2020-12-27 01:34:26 [INFO] [TRAIN] epoch=202, iter=75100/80000, loss=12.0579, lr=0.000810, batch_cost=1.8438, reader_cost=0.0015 | ETA 02:30:34 2020-12-27 01:37:44 [INFO] [TRAIN] epoch=203, iter=75200/80000, loss=12.1123, lr=0.000795, batch_cost=1.9809, reader_cost=0.0541 | ETA 02:38:28 2020-12-27 01:40:49 [INFO] [TRAIN] epoch=203, iter=75300/80000, loss=11.8767, lr=0.000780, batch_cost=1.8468, reader_cost=0.0020 | ETA 02:24:40 2020-12-27 01:43:51 [INFO] [TRAIN] epoch=203, iter=75400/80000, loss=11.9753, lr=0.000765, batch_cost=1.8181, reader_cost=0.0016 | ETA 02:19:23 2020-12-27 01:46:57 [INFO] [TRAIN] epoch=203, iter=75500/80000, loss=12.1212, lr=0.000750, batch_cost=1.8563, reader_cost=0.0019 | ETA 02:19:13 2020-12-27 01:50:16 [INFO] [TRAIN] epoch=204, iter=75600/80000, loss=11.8765, lr=0.000735, batch_cost=1.9938, reader_cost=0.0517 | ETA 02:26:12 2020-12-27 01:53:23 [INFO] [TRAIN] epoch=204, iter=75700/80000, loss=11.9918, lr=0.000720, batch_cost=1.8615, reader_cost=0.0021 | ETA 02:13:24 2020-12-27 01:56:28 [INFO] [TRAIN] epoch=204, iter=75800/80000, loss=11.9183, lr=0.000705, batch_cost=1.8573, reader_cost=0.0017 | ETA 02:10:00 2020-12-27 01:59:49 [INFO] [TRAIN] epoch=205, iter=75900/80000, loss=12.0508, lr=0.000690, batch_cost=2.0079, reader_cost=0.0592 | ETA 02:17:12 2020-12-27 02:02:55 [INFO] [TRAIN] epoch=205, iter=76000/80000, loss=11.8422, lr=0.000675, batch_cost=1.8515, reader_cost=0.0019 | ETA 02:03:25 2020-12-27 02:02:55 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-27 02:03:39 [INFO] [EVAL] #Images=500 mIoU=0.8064 Acc=0.9648 Kappa=0.9543 2020-12-27 02:03:39 [INFO] [EVAL] Class IoU: [0.9826 0.8553 0.934 0.5577 0.6575 0.7001 0.7587 0.8307 0.9292 0.6471 0.952 0.8505 0.681 0.958 0.8122 0.8931 0.8101 0.7051 0.8059] 2020-12-27 02:03:39 [INFO] [EVAL] Class Acc: [0.9916 0.9135 0.9627 0.8349 0.8469 0.8403 0.8546 0.9232 0.9581 0.8457 0.9685 0.9112 0.8337 0.9777 0.9203 0.9431 0.9717 0.8351 0.8776] 2020-12-27 02:03:46 [INFO] [EVAL] The model with the best validation mIoU (0.8064) was saved at iter 76000. 2020-12-27 02:06:51 [INFO] [TRAIN] epoch=205, iter=76100/80000, loss=11.8427, lr=0.000660, batch_cost=1.8495, reader_cost=0.0023 | ETA 02:00:12 2020-12-27 02:09:57 [INFO] [TRAIN] epoch=205, iter=76200/80000, loss=12.0140, lr=0.000644, batch_cost=1.8532, reader_cost=0.0011 | ETA 01:57:22 2020-12-27 02:13:15 [INFO] [TRAIN] epoch=206, iter=76300/80000, loss=12.1579, lr=0.000629, batch_cost=1.9833, reader_cost=0.0549 | ETA 02:02:18 2020-12-27 02:16:21 [INFO] [TRAIN] epoch=206, iter=76400/80000, loss=11.9599, lr=0.000614, batch_cost=1.8540, reader_cost=0.0016 | ETA 01:51:14 2020-12-27 02:19:26 [INFO] [TRAIN] epoch=206, iter=76500/80000, loss=11.8612, lr=0.000598, batch_cost=1.8526, reader_cost=0.0015 | ETA 01:48:03 2020-12-27 02:22:33 [INFO] [TRAIN] epoch=206, iter=76600/80000, loss=12.1391, lr=0.000583, batch_cost=1.8668, reader_cost=0.0013 | ETA 01:45:47 2020-12-27 02:25:48 [INFO] [TRAIN] epoch=207, iter=76700/80000, loss=12.0647, lr=0.000568, batch_cost=1.9527, reader_cost=0.0514 | ETA 01:47:23 2020-12-27 02:28:54 [INFO] [TRAIN] epoch=207, iter=76800/80000, loss=11.9577, lr=0.000552, batch_cost=1.8540, reader_cost=0.0015 | ETA 01:38:52 2020-12-27 02:32:00 [INFO] [TRAIN] epoch=207, iter=76900/80000, loss=12.0062, lr=0.000537, batch_cost=1.8574, reader_cost=0.0017 | ETA 01:35:57 2020-12-27 02:35:05 [INFO] [TRAIN] epoch=207, iter=77000/80000, loss=12.1131, lr=0.000521, batch_cost=1.8534, reader_cost=0.0021 | ETA 01:32:40 2020-12-27 02:38:26 [INFO] [TRAIN] epoch=208, iter=77100/80000, loss=11.9712, lr=0.000505, batch_cost=2.0016, reader_cost=0.0525 | ETA 01:36:44 2020-12-27 02:41:31 [INFO] [TRAIN] epoch=208, iter=77200/80000, loss=12.0074, lr=0.000490, batch_cost=1.8524, reader_cost=0.0014 | ETA 01:26:26 2020-12-27 02:44:36 [INFO] [TRAIN] epoch=208, iter=77300/80000, loss=11.9763, lr=0.000474, batch_cost=1.8486, reader_cost=0.0017 | ETA 01:23:11 2020-12-27 02:47:55 [INFO] [TRAIN] epoch=209, iter=77400/80000, loss=11.9442, lr=0.000458, batch_cost=1.9907, reader_cost=0.0539 | ETA 01:26:15 2020-12-27 02:50:59 [INFO] [TRAIN] epoch=209, iter=77500/80000, loss=11.9533, lr=0.000442, batch_cost=1.8381, reader_cost=0.0017 | ETA 01:16:35 2020-12-27 02:54:05 [INFO] [TRAIN] epoch=209, iter=77600/80000, loss=11.8833, lr=0.000426, batch_cost=1.8571, reader_cost=0.0010 | ETA 01:14:17 2020-12-27 02:57:10 [INFO] [TRAIN] epoch=209, iter=77700/80000, loss=11.9457, lr=0.000410, batch_cost=1.8458, reader_cost=0.0018 | ETA 01:10:45 2020-12-27 03:00:29 [INFO] [TRAIN] epoch=210, iter=77800/80000, loss=11.9035, lr=0.000394, batch_cost=1.9871, reader_cost=0.0530 | ETA 01:12:51 2020-12-27 03:03:34 [INFO] [TRAIN] epoch=210, iter=77900/80000, loss=11.8661, lr=0.000378, batch_cost=1.8527, reader_cost=0.0019 | ETA 01:04:50 2020-12-27 03:06:35 [INFO] [TRAIN] epoch=210, iter=78000/80000, loss=11.8584, lr=0.000362, batch_cost=1.8097, reader_cost=0.0014 | ETA 01:00:19 2020-12-27 03:06:35 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-27 03:07:19 [INFO] [EVAL] #Images=500 mIoU=0.8106 Acc=0.9657 Kappa=0.9555 2020-12-27 03:07:19 [INFO] [EVAL] Class IoU: [0.9826 0.8583 0.9356 0.5615 0.6582 0.7102 0.7644 0.8317 0.9316 0.6669 0.9535 0.853 0.6853 0.9599 0.8282 0.905 0.8152 0.6957 0.8038] 2020-12-27 03:07:19 [INFO] [EVAL] Class Acc: [0.9915 0.9178 0.9635 0.8522 0.8296 0.8483 0.8593 0.914 0.9596 0.8453 0.9753 0.911 0.8003 0.9778 0.9349 0.9516 0.9665 0.8634 0.8807] 2020-12-27 03:07:27 [INFO] [EVAL] The model with the best validation mIoU (0.8106) was saved at iter 78000. 2020-12-27 03:10:33 [INFO] [TRAIN] epoch=210, iter=78100/80000, loss=12.0145, lr=0.000345, batch_cost=1.8557, reader_cost=0.0021 | ETA 00:58:45 2020-12-27 03:13:53 [INFO] [TRAIN] epoch=211, iter=78200/80000, loss=12.0119, lr=0.000329, batch_cost=1.9981, reader_cost=0.0545 | ETA 00:59:56 2020-12-27 03:16:58 [INFO] [TRAIN] epoch=211, iter=78300/80000, loss=11.9449, lr=0.000313, batch_cost=1.8478, reader_cost=0.0018 | ETA 00:52:21 2020-12-27 03:20:04 [INFO] [TRAIN] epoch=211, iter=78400/80000, loss=12.0143, lr=0.000296, batch_cost=1.8617, reader_cost=0.0015 | ETA 00:49:38 2020-12-27 03:23:24 [INFO] [TRAIN] epoch=212, iter=78500/80000, loss=12.0345, lr=0.000279, batch_cost=1.9992, reader_cost=0.0554 | ETA 00:49:58 2020-12-27 03:26:30 [INFO] [TRAIN] epoch=212, iter=78600/80000, loss=11.8960, lr=0.000262, batch_cost=1.8562, reader_cost=0.0016 | ETA 00:43:18 2020-12-27 03:29:35 [INFO] [TRAIN] epoch=212, iter=78700/80000, loss=11.9002, lr=0.000246, batch_cost=1.8526, reader_cost=0.0014 | ETA 00:40:08 2020-12-27 03:32:41 [INFO] [TRAIN] epoch=212, iter=78800/80000, loss=12.0949, lr=0.000228, batch_cost=1.8572, reader_cost=0.0016 | ETA 00:37:08 2020-12-27 03:36:00 [INFO] [TRAIN] epoch=213, iter=78900/80000, loss=12.0983, lr=0.000211, batch_cost=1.9863, reader_cost=0.0504 | ETA 00:36:24 2020-12-27 03:39:05 [INFO] [TRAIN] epoch=213, iter=79000/80000, loss=11.9159, lr=0.000194, batch_cost=1.8488, reader_cost=0.0019 | ETA 00:30:48 2020-12-27 03:42:11 [INFO] [TRAIN] epoch=213, iter=79100/80000, loss=11.8771, lr=0.000176, batch_cost=1.8605, reader_cost=0.0020 | ETA 00:27:54 2020-12-27 03:45:16 [INFO] [TRAIN] epoch=213, iter=79200/80000, loss=12.1291, lr=0.000159, batch_cost=1.8476, reader_cost=0.0009 | ETA 00:24:38 2020-12-27 03:48:31 [INFO] [TRAIN] epoch=214, iter=79300/80000, loss=11.9984, lr=0.000141, batch_cost=1.9518, reader_cost=0.0508 | ETA 00:22:46 2020-12-27 03:51:37 [INFO] [TRAIN] epoch=214, iter=79400/80000, loss=11.8608, lr=0.000123, batch_cost=1.8566, reader_cost=0.0019 | ETA 00:18:33 2020-12-27 03:54:42 [INFO] [TRAIN] epoch=214, iter=79500/80000, loss=11.8961, lr=0.000104, batch_cost=1.8519, reader_cost=0.0011 | ETA 00:15:25 2020-12-27 03:57:49 [INFO] [TRAIN] epoch=214, iter=79600/80000, loss=12.1184, lr=0.000085, batch_cost=1.8621, reader_cost=0.0017 | ETA 00:12:24 2020-12-27 04:01:09 [INFO] [TRAIN] epoch=215, iter=79700/80000, loss=11.8703, lr=0.000066, batch_cost=2.0001, reader_cost=0.0507 | ETA 00:10:00 2020-12-27 04:04:15 [INFO] [TRAIN] epoch=215, iter=79800/80000, loss=11.8406, lr=0.000046, batch_cost=1.8575, reader_cost=0.0019 | ETA 00:06:11 2020-12-27 04:07:21 [INFO] [TRAIN] epoch=215, iter=79900/80000, loss=12.0507, lr=0.000025, batch_cost=1.8591, reader_cost=0.0018 | ETA 00:03:05 2020-12-27 04:10:41 [INFO] [TRAIN] epoch=216, iter=80000/80000, loss=12.0494, lr=0.000000, batch_cost=1.9959, reader_cost=0.0509 | ETA 00:00:00 2020-12-27 04:10:41 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-27 04:11:25 [INFO] [EVAL] #Images=500 mIoU=0.8126 Acc=0.9658 Kappa=0.9557 2020-12-27 04:11:25 [INFO] [EVAL] Class IoU: [0.9836 0.8632 0.9346 0.5397 0.6474 0.711 0.7646 0.8353 0.9316 0.6646 0.953 0.8542 0.6814 0.9601 0.8271 0.9173 0.8631 0.7027 0.805 ] 2020-12-27 04:11:25 [INFO] [EVAL] Class Acc: [0.9926 0.9172 0.9616 0.8352 0.8407 0.852 0.8653 0.9216 0.9606 0.8502 0.9694 0.9165 0.8148 0.978 0.9154 0.9661 0.9542 0.8436 0.8751] 2020-12-27 04:11:32 [INFO] [EVAL] The model with the best validation mIoU (0.8126) was saved at iter 80000.