2020-10-31 20:44:04 [INFO] ------------Environment Information------------- platform: Linux-3.10.0_3-0-0-34-x86_64-with-centos-7.5.1804-Core Python: 3.7.9 (default, Aug 31 2020, 12:42:55) [GCC 7.3.0] Paddle compiled with cuda: True NVCC: Cuda compilation tools, release 10.2, V10.2.89 cudnn: 7.6 GPUs used: 4 CUDA_VISIBLE_DEVICES: 4,5,6,7 GPU: ['GPU 0: Tesla V100-SXM2-16GB', 'GPU 1: Tesla V100-SXM2-16GB', 'GPU 2: Tesla V100-SXM2-16GB', 'GPU 3: Tesla V100-SXM2-16GB', 'GPU 4: Tesla V100-SXM2-16GB', 'GPU 5: Tesla V100-SXM2-16GB', 'GPU 6: Tesla V100-SXM2-16GB', 'GPU 7: Tesla V100-SXM2-16GB'] GCC: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36) PaddlePaddle: 2.0.0-rc0 OpenCV: 4.1.0 ------------------------------------------------ 2020-10-31 20:44:05 [INFO] ---------------Config Information--------------- batch_size: 4 iters: 160000 learning_rate: decay: end_lr: 0.0001 power: 0.9 type: poly value: 0.05 loss: coef: - 1.0 - 0.4 types: - ignore_index: 255 type: CrossEntropyLoss model: enable_auxiliary_loss: true num_classes: 19 pretrained: null type: FastSCNN optimizer: momentum: 0.9 type: sgd weight_decay: 4.0e-05 train_dataset: dataset_root: data/cityscapes mode: train transforms: - max_scale_factor: 2.0 min_scale_factor: 0.5 scale_step_size: 0.25 type: ResizeStepScaling - crop_size: - 1024 - 1024 type: RandomPaddingCrop - type: RandomHorizontalFlip - brightness_range: 0.4 contrast_range: 0.4 saturation_range: 0.4 type: RandomDistort - type: Normalize type: Cityscapes val_dataset: dataset_root: data/cityscapes mode: val transforms: - type: Normalize type: Cityscapes ------------------------------------------------ 2020-10-31 20:45:20 [INFO] [TRAIN] epoch=2, iter=200/160000, loss=1.6471, lr=0.049944, batch_cost=0.3286, reader_cost=0.0785 | ETA 14:35:07 2020-10-31 20:46:27 [INFO] [TRAIN] epoch=3, iter=400/160000, loss=1.2469, lr=0.049888, batch_cost=0.3329, reader_cost=0.1652 | ETA 14:45:36 2020-10-31 20:47:34 [INFO] [TRAIN] epoch=4, iter=600/160000, loss=1.1155, lr=0.049832, batch_cost=0.3348, reader_cost=0.1821 | ETA 14:49:28 2020-10-31 20:48:39 [INFO] [TRAIN] epoch=5, iter=800/160000, loss=1.0633, lr=0.049776, batch_cost=0.3277, reader_cost=0.1728 | ETA 14:29:26 2020-10-31 20:49:45 [INFO] [TRAIN] epoch=6, iter=1000/160000, loss=1.0351, lr=0.049720, batch_cost=0.3287, reader_cost=0.1407 | ETA 14:30:55 2020-10-31 20:50:51 [INFO] [TRAIN] epoch=7, iter=1200/160000, loss=0.9695, lr=0.049663, batch_cost=0.3321, reader_cost=0.0977 | ETA 14:38:55 2020-10-31 20:51:58 [INFO] [TRAIN] epoch=8, iter=1400/160000, loss=0.9478, lr=0.049607, batch_cost=0.3314, reader_cost=0.1054 | ETA 14:35:55 2020-10-31 20:53:04 [INFO] [TRAIN] epoch=9, iter=1600/160000, loss=0.8990, lr=0.049551, batch_cost=0.3304, reader_cost=0.0473 | ETA 14:32:19 2020-10-31 20:54:10 [INFO] [TRAIN] epoch=10, iter=1800/160000, loss=0.9022, lr=0.049495, batch_cost=0.3292, reader_cost=0.0888 | ETA 14:28:05 2020-10-31 20:55:15 [INFO] [TRAIN] epoch=11, iter=2000/160000, loss=0.8822, lr=0.049439, batch_cost=0.3282, reader_cost=0.1209 | ETA 14:24:10 2020-10-31 20:56:22 [INFO] [TRAIN] epoch=12, iter=2200/160000, loss=0.8579, lr=0.049382, batch_cost=0.3324, reader_cost=0.1493 | ETA 14:34:07 2020-10-31 20:57:27 [INFO] [TRAIN] epoch=13, iter=2400/160000, loss=0.8254, lr=0.049326, batch_cost=0.3280, reader_cost=0.1590 | ETA 14:21:36 2020-10-31 20:58:35 [INFO] [TRAIN] epoch=14, iter=2600/160000, loss=0.8485, lr=0.049270, batch_cost=0.3356, reader_cost=0.1226 | ETA 14:40:26 2020-10-31 20:59:41 [INFO] [TRAIN] epoch=16, iter=2800/160000, loss=0.8097, lr=0.049214, batch_cost=0.3334, reader_cost=0.0922 | ETA 14:33:28 2020-10-31 21:00:47 [INFO] [TRAIN] epoch=17, iter=3000/160000, loss=0.7554, lr=0.049157, batch_cost=0.3285, reader_cost=0.1574 | ETA 14:19:37 2020-10-31 21:01:52 [INFO] [TRAIN] epoch=18, iter=3200/160000, loss=0.7712, lr=0.049101, batch_cost=0.3262, reader_cost=0.1103 | ETA 14:12:23 2020-10-31 21:02:59 [INFO] [TRAIN] epoch=19, iter=3400/160000, loss=0.7807, lr=0.049045, batch_cost=0.3357, reader_cost=0.0448 | ETA 14:36:06 2020-10-31 21:04:04 [INFO] [TRAIN] epoch=20, iter=3600/160000, loss=0.7728, lr=0.048989, batch_cost=0.3257, reader_cost=0.1019 | ETA 14:08:58 2020-10-31 21:05:10 [INFO] [TRAIN] epoch=21, iter=3800/160000, loss=0.7529, lr=0.048932, batch_cost=0.3261, reader_cost=0.0734 | ETA 14:08:53 2020-10-31 21:06:16 [INFO] [TRAIN] epoch=22, iter=4000/160000, loss=0.7304, lr=0.048876, batch_cost=0.3333, reader_cost=0.0836 | ETA 14:26:39 2020-10-31 21:07:21 [INFO] [TRAIN] epoch=23, iter=4200/160000, loss=0.7626, lr=0.048820, batch_cost=0.3246, reader_cost=0.0872 | ETA 14:02:53 2020-10-31 21:08:28 [INFO] [TRAIN] epoch=24, iter=4400/160000, loss=0.7595, lr=0.048764, batch_cost=0.3358, reader_cost=0.0618 | ETA 14:30:47 2020-10-31 21:09:33 [INFO] [TRAIN] epoch=25, iter=4600/160000, loss=0.7222, lr=0.048707, batch_cost=0.3253, reader_cost=0.0809 | ETA 14:02:33 2020-10-31 21:10:40 [INFO] [TRAIN] epoch=26, iter=4800/160000, loss=0.7059, lr=0.048651, batch_cost=0.3336, reader_cost=0.1799 | ETA 14:22:51 2020-10-31 21:11:46 [INFO] [TRAIN] epoch=27, iter=5000/160000, loss=0.7261, lr=0.048595, batch_cost=0.3278, reader_cost=0.1260 | ETA 14:06:47 2020-10-31 21:12:52 [INFO] [TRAIN] epoch=28, iter=5200/160000, loss=0.7158, lr=0.048538, batch_cost=0.3288, reader_cost=0.1203 | ETA 14:08:23 2020-10-31 21:13:59 [INFO] [TRAIN] epoch=30, iter=5400/160000, loss=0.6981, lr=0.048482, batch_cost=0.3374, reader_cost=0.1793 | ETA 14:29:16 2020-10-31 21:15:05 [INFO] [TRAIN] epoch=31, iter=5600/160000, loss=0.6914, lr=0.048426, batch_cost=0.3302, reader_cost=0.1514 | ETA 14:09:41 2020-10-31 21:16:11 [INFO] [TRAIN] epoch=32, iter=5800/160000, loss=0.6918, lr=0.048369, batch_cost=0.3301, reader_cost=0.1716 | ETA 14:08:24 2020-10-31 21:17:16 [INFO] [TRAIN] epoch=33, iter=6000/160000, loss=0.6759, lr=0.048313, batch_cost=0.3224, reader_cost=0.1108 | ETA 13:47:29 2020-10-31 21:18:22 [INFO] [TRAIN] epoch=34, iter=6200/160000, loss=0.6843, lr=0.048257, batch_cost=0.3330, reader_cost=0.0684 | ETA 14:13:31 2020-10-31 21:19:27 [INFO] [TRAIN] epoch=35, iter=6400/160000, loss=0.6807, lr=0.048200, batch_cost=0.3244, reader_cost=0.1113 | ETA 13:50:35 2020-10-31 21:20:32 [INFO] [TRAIN] epoch=36, iter=6600/160000, loss=0.6881, lr=0.048144, batch_cost=0.3253, reader_cost=0.1690 | ETA 13:51:40 2020-10-31 21:21:37 [INFO] [TRAIN] epoch=37, iter=6800/160000, loss=0.6831, lr=0.048087, batch_cost=0.3253, reader_cost=0.1706 | ETA 13:50:35 2020-10-31 21:22:44 [INFO] [TRAIN] epoch=38, iter=7000/160000, loss=0.6844, lr=0.048031, batch_cost=0.3329, reader_cost=0.1259 | ETA 14:08:50 2020-10-31 21:23:48 [INFO] [TRAIN] epoch=39, iter=7200/160000, loss=0.6819, lr=0.047975, batch_cost=0.3223, reader_cost=0.0322 | ETA 13:40:53 2020-10-31 21:24:55 [INFO] [TRAIN] epoch=40, iter=7400/160000, loss=0.6674, lr=0.047918, batch_cost=0.3322, reader_cost=0.0325 | ETA 14:05:00 2020-10-31 21:26:02 [INFO] [TRAIN] epoch=41, iter=7600/160000, loss=0.6746, lr=0.047862, batch_cost=0.3351, reader_cost=0.0813 | ETA 14:11:03 2020-10-31 21:27:07 [INFO] [TRAIN] epoch=42, iter=7800/160000, loss=0.6652, lr=0.047805, batch_cost=0.3247, reader_cost=0.1204 | ETA 13:43:39 2020-10-31 21:28:15 [INFO] [TRAIN] epoch=44, iter=8000/160000, loss=0.6580, lr=0.047749, batch_cost=0.3400, reader_cost=0.1774 | ETA 14:21:20 2020-10-31 21:28:15 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-10-31 21:31:08 [INFO] [EVAL] #Images=500 mIoU=0.4594 Acc=0.9082 Kappa=0.8797 2020-10-31 21:31:08 [INFO] [EVAL] Category IoU: [0.9476 0.6374 0.8337 0.2445 0.2688 0.3311 0.1834 0.303 0.8709 0.4445 0.9008 0.5379 0.1217 0.8515 0.0543 0.3469 0.2131 0.1283 0.5095] 2020-10-31 21:31:08 [INFO] [EVAL] Category Acc: [0.9616 0.8014 0.8688 0.6416 0.6179 0.7153 0.7829 0.8381 0.928 0.7682 0.9545 0.7127 0.5104 0.9136 0.6399 0.7691 0.3039 0.3241 0.6407] 2020-10-31 21:31:08 [INFO] [EVAL] The model with the best validation mIoU (0.4594) was saved at iter 8000. 2020-10-31 21:32:12 [INFO] [TRAIN] epoch=45, iter=8200/160000, loss=0.6351, lr=0.047693, batch_cost=0.3203, reader_cost=0.1584 | ETA 13:30:19 2020-10-31 21:33:22 [INFO] [TRAIN] epoch=46, iter=8400/160000, loss=0.6372, lr=0.047636, batch_cost=0.3464, reader_cost=0.1989 | ETA 14:35:20 2020-10-31 21:34:33 [INFO] [TRAIN] epoch=47, iter=8600/160000, loss=0.6270, lr=0.047580, batch_cost=0.3572, reader_cost=0.2144 | ETA 15:01:22 2020-10-31 21:35:44 [INFO] [TRAIN] epoch=48, iter=8800/160000, loss=0.6251, lr=0.047523, batch_cost=0.3531, reader_cost=0.2092 | ETA 14:49:41 2020-10-31 21:36:54 [INFO] [TRAIN] epoch=49, iter=9000/160000, loss=0.6596, lr=0.047467, batch_cost=0.3519, reader_cost=0.2085 | ETA 14:45:30 2020-10-31 21:38:03 [INFO] [TRAIN] epoch=50, iter=9200/160000, loss=0.6221, lr=0.047410, batch_cost=0.3460, reader_cost=0.2026 | ETA 14:29:35 2020-10-31 21:39:14 [INFO] [TRAIN] epoch=51, iter=9400/160000, loss=0.6357, lr=0.047354, batch_cost=0.3533, reader_cost=0.2090 | ETA 14:46:46 2020-10-31 21:40:23 [INFO] [TRAIN] epoch=52, iter=9600/160000, loss=0.6253, lr=0.047297, batch_cost=0.3439, reader_cost=0.2038 | ETA 14:22:05 2020-10-31 21:41:32 [INFO] [TRAIN] epoch=53, iter=9800/160000, loss=0.6256, lr=0.047241, batch_cost=0.3474, reader_cost=0.1985 | ETA 14:29:41 2020-10-31 21:42:42 [INFO] [TRAIN] epoch=54, iter=10000/160000, loss=0.6255, lr=0.047184, batch_cost=0.3476, reader_cost=0.1990 | ETA 14:28:58 2020-10-31 21:43:49 [INFO] [TRAIN] epoch=55, iter=10200/160000, loss=0.6375, lr=0.047128, batch_cost=0.3374, reader_cost=0.1920 | ETA 14:02:24 2020-10-31 21:44:58 [INFO] [TRAIN] epoch=56, iter=10400/160000, loss=0.6230, lr=0.047071, batch_cost=0.3451, reader_cost=0.1984 | ETA 14:20:22 2020-10-31 21:46:08 [INFO] [TRAIN] epoch=57, iter=10600/160000, loss=0.6061, lr=0.047015, batch_cost=0.3508, reader_cost=0.2083 | ETA 14:33:25 2020-10-31 21:47:19 [INFO] [TRAIN] epoch=59, iter=10800/160000, loss=0.6306, lr=0.046958, batch_cost=0.3548, reader_cost=0.2081 | ETA 14:42:11 2020-10-31 21:48:31 [INFO] [TRAIN] epoch=60, iter=11000/160000, loss=0.5986, lr=0.046902, batch_cost=0.3582, reader_cost=0.2172 | ETA 14:49:25 2020-10-31 21:49:39 [INFO] [TRAIN] epoch=61, iter=11200/160000, loss=0.6137, lr=0.046845, batch_cost=0.3419, reader_cost=0.1969 | ETA 14:08:00 2020-10-31 21:50:51 [INFO] [TRAIN] epoch=62, iter=11400/160000, loss=0.6226, lr=0.046789, batch_cost=0.3577, reader_cost=0.2141 | ETA 14:45:52 2020-10-31 21:52:01 [INFO] [TRAIN] epoch=63, iter=11600/160000, loss=0.6221, lr=0.046732, batch_cost=0.3491, reader_cost=0.1960 | ETA 14:23:26 2020-10-31 21:53:10 [INFO] [TRAIN] epoch=64, iter=11800/160000, loss=0.6130, lr=0.046676, batch_cost=0.3465, reader_cost=0.1989 | ETA 14:15:55 2020-10-31 21:54:19 [INFO] [TRAIN] epoch=65, iter=12000/160000, loss=0.5976, lr=0.046619, batch_cost=0.3471, reader_cost=0.2000 | ETA 14:16:16 2020-10-31 21:55:27 [INFO] [TRAIN] epoch=66, iter=12200/160000, loss=0.6135, lr=0.046562, batch_cost=0.3383, reader_cost=0.1743 | ETA 13:53:22 2020-10-31 21:56:35 [INFO] [TRAIN] epoch=67, iter=12400/160000, loss=0.5946, lr=0.046506, batch_cost=0.3410, reader_cost=0.1907 | ETA 13:58:50 2020-10-31 21:57:49 [INFO] [TRAIN] epoch=68, iter=12600/160000, loss=0.6214, lr=0.046449, batch_cost=0.3689, reader_cost=0.2179 | ETA 15:06:12 2020-10-31 21:58:59 [INFO] [TRAIN] epoch=69, iter=12800/160000, loss=0.6077, lr=0.046393, batch_cost=0.3491, reader_cost=0.2053 | ETA 14:16:24 2020-10-31 22:00:09 [INFO] [TRAIN] epoch=70, iter=13000/160000, loss=0.5992, lr=0.046336, batch_cost=0.3492, reader_cost=0.2064 | ETA 14:15:38 2020-10-31 22:01:19 [INFO] [TRAIN] epoch=71, iter=13200/160000, loss=0.5889, lr=0.046279, batch_cost=0.3500, reader_cost=0.2102 | ETA 14:16:21 2020-10-31 22:02:29 [INFO] [TRAIN] epoch=73, iter=13400/160000, loss=0.6086, lr=0.046223, batch_cost=0.3499, reader_cost=0.2069 | ETA 14:15:00 2020-10-31 22:03:38 [INFO] [TRAIN] epoch=74, iter=13600/160000, loss=0.5959, lr=0.046166, batch_cost=0.3482, reader_cost=0.2015 | ETA 14:09:32 2020-10-31 22:04:48 [INFO] [TRAIN] epoch=75, iter=13800/160000, loss=0.5712, lr=0.046110, batch_cost=0.3463, reader_cost=0.2076 | ETA 14:03:42 2020-10-31 22:05:58 [INFO] [TRAIN] epoch=76, iter=14000/160000, loss=0.5920, lr=0.046053, batch_cost=0.3532, reader_cost=0.2123 | ETA 14:19:28 2020-10-31 22:07:07 [INFO] [TRAIN] epoch=77, iter=14200/160000, loss=0.5924, lr=0.045996, batch_cost=0.3429, reader_cost=0.1988 | ETA 13:53:12 2020-10-31 22:08:14 [INFO] [TRAIN] epoch=78, iter=14400/160000, loss=0.5794, lr=0.045940, batch_cost=0.3354, reader_cost=0.1886 | ETA 13:33:49 2020-10-31 22:09:23 [INFO] [TRAIN] epoch=79, iter=14600/160000, loss=0.5819, lr=0.045883, batch_cost=0.3456, reader_cost=0.1962 | ETA 13:57:29 2020-10-31 22:10:35 [INFO] [TRAIN] epoch=80, iter=14800/160000, loss=0.5788, lr=0.045826, batch_cost=0.3615, reader_cost=0.2174 | ETA 14:34:55 2020-10-31 22:11:44 [INFO] [TRAIN] epoch=81, iter=15000/160000, loss=0.5661, lr=0.045770, batch_cost=0.3404, reader_cost=0.1982 | ETA 13:42:34 2020-10-31 22:12:56 [INFO] [TRAIN] epoch=82, iter=15200/160000, loss=0.5917, lr=0.045713, batch_cost=0.3610, reader_cost=0.2183 | ETA 14:31:15 2020-10-31 22:14:04 [INFO] [TRAIN] epoch=83, iter=15400/160000, loss=0.5844, lr=0.045656, batch_cost=0.3434, reader_cost=0.1976 | ETA 13:47:34 2020-10-31 22:15:15 [INFO] [TRAIN] epoch=84, iter=15600/160000, loss=0.5779, lr=0.045599, batch_cost=0.3525, reader_cost=0.2088 | ETA 14:08:20 2020-10-31 22:16:25 [INFO] [TRAIN] epoch=85, iter=15800/160000, loss=0.5813, lr=0.045543, batch_cost=0.3482, reader_cost=0.2022 | ETA 13:56:50 2020-10-31 22:17:36 [INFO] [TRAIN] epoch=87, iter=16000/160000, loss=0.5921, lr=0.045486, batch_cost=0.3567, reader_cost=0.2055 | ETA 14:16:09 2020-10-31 22:17:36 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-10-31 22:20:23 [INFO] [EVAL] #Images=500 mIoU=0.5203 Acc=0.9176 Kappa=0.8929 2020-10-31 22:20:23 [INFO] [EVAL] Category IoU: [0.9545 0.6837 0.8576 0.2784 0.3564 0.3754 0.3423 0.4716 0.8785 0.4852 0.9012 0.5822 0.2312 0.8607 0.2873 0.4071 0.24 0.1754 0.5177] 2020-10-31 22:20:23 [INFO] [EVAL] Category Acc: [0.9808 0.7801 0.9013 0.6008 0.6303 0.7353 0.7072 0.8269 0.933 0.7386 0.9219 0.7262 0.6055 0.8979 0.5689 0.7337 0.5145 0.3669 0.636 ] 2020-10-31 22:20:23 [INFO] [EVAL] The model with the best validation mIoU (0.5203) was saved at iter 16000. 2020-10-31 22:21:30 [INFO] [TRAIN] epoch=88, iter=16200/160000, loss=0.5768, lr=0.045429, batch_cost=0.3333, reader_cost=0.1882 | ETA 13:18:50 2020-10-31 22:22:38 [INFO] [TRAIN] epoch=89, iter=16400/160000, loss=0.5745, lr=0.045372, batch_cost=0.3412, reader_cost=0.2036 | ETA 13:36:30 2020-10-31 22:23:47 [INFO] [TRAIN] epoch=90, iter=16600/160000, loss=0.5783, lr=0.045316, batch_cost=0.3440, reader_cost=0.2034 | ETA 13:42:13 2020-10-31 22:24:55 [INFO] [TRAIN] epoch=91, iter=16800/160000, loss=0.5805, lr=0.045259, batch_cost=0.3394, reader_cost=0.2005 | ETA 13:30:02 2020-10-31 22:26:01 [INFO] [TRAIN] epoch=92, iter=17000/160000, loss=0.5594, lr=0.045202, batch_cost=0.3305, reader_cost=0.1819 | ETA 13:07:37 2020-10-31 22:27:10 [INFO] [TRAIN] epoch=93, iter=17200/160000, loss=0.5557, lr=0.045145, batch_cost=0.3461, reader_cost=0.1952 | ETA 13:43:40 2020-10-31 22:28:20 [INFO] [TRAIN] epoch=94, iter=17400/160000, loss=0.5924, lr=0.045089, batch_cost=0.3468, reader_cost=0.2033 | ETA 13:44:20 2020-10-31 22:29:30 [INFO] [TRAIN] epoch=95, iter=17600/160000, loss=0.5581, lr=0.045032, batch_cost=0.3542, reader_cost=0.2149 | ETA 14:00:44 2020-10-31 22:30:42 [INFO] [TRAIN] epoch=96, iter=17800/160000, loss=0.5795, lr=0.044975, batch_cost=0.3565, reader_cost=0.2127 | ETA 14:04:56 2020-10-31 22:31:51 [INFO] [TRAIN] epoch=97, iter=18000/160000, loss=0.5763, lr=0.044918, batch_cost=0.3482, reader_cost=0.2026 | ETA 13:44:09 2020-10-31 22:33:03 [INFO] [TRAIN] epoch=98, iter=18200/160000, loss=0.5576, lr=0.044861, batch_cost=0.3565, reader_cost=0.2137 | ETA 14:02:36 2020-10-31 22:34:11 [INFO] [TRAIN] epoch=99, iter=18400/160000, loss=0.5743, lr=0.044805, batch_cost=0.3427, reader_cost=0.1983 | ETA 13:28:43 2020-10-31 22:35:19 [INFO] [TRAIN] epoch=100, iter=18600/160000, loss=0.5569, lr=0.044748, batch_cost=0.3382, reader_cost=0.1914 | ETA 13:16:59 2020-10-31 22:36:30 [INFO] [TRAIN] epoch=102, iter=18800/160000, loss=0.5703, lr=0.044691, batch_cost=0.3558, reader_cost=0.2123 | ETA 13:57:21 2020-10-31 22:37:40 [INFO] [TRAIN] epoch=103, iter=19000/160000, loss=0.5638, lr=0.044634, batch_cost=0.3492, reader_cost=0.2061 | ETA 13:40:43 2020-10-31 22:38:49 [INFO] [TRAIN] epoch=104, iter=19200/160000, loss=0.5783, lr=0.044577, batch_cost=0.3457, reader_cost=0.2029 | ETA 13:31:09 2020-10-31 22:39:57 [INFO] [TRAIN] epoch=105, iter=19400/160000, loss=0.5774, lr=0.044520, batch_cost=0.3403, reader_cost=0.1968 | ETA 13:17:28 2020-10-31 22:41:07 [INFO] [TRAIN] epoch=106, iter=19600/160000, loss=0.5573, lr=0.044463, batch_cost=0.3483, reader_cost=0.2016 | ETA 13:34:59 2020-10-31 22:42:15 [INFO] [TRAIN] epoch=107, iter=19800/160000, loss=0.5589, lr=0.044407, batch_cost=0.3405, reader_cost=0.1974 | ETA 13:15:33 2020-10-31 22:43:22 [INFO] [TRAIN] epoch=108, iter=20000/160000, loss=0.5567, lr=0.044350, batch_cost=0.3364, reader_cost=0.1894 | ETA 13:04:55 2020-10-31 22:44:30 [INFO] [TRAIN] epoch=109, iter=20200/160000, loss=0.5509, lr=0.044293, batch_cost=0.3384, reader_cost=0.1944 | ETA 13:08:22 2020-10-31 22:45:41 [INFO] [TRAIN] epoch=110, iter=20400/160000, loss=0.5605, lr=0.044236, batch_cost=0.3578, reader_cost=0.2149 | ETA 13:52:30 2020-10-31 22:46:51 [INFO] [TRAIN] epoch=111, iter=20600/160000, loss=0.5748, lr=0.044179, batch_cost=0.3497, reader_cost=0.2079 | ETA 13:32:22 2020-10-31 22:48:00 [INFO] [TRAIN] epoch=112, iter=20800/160000, loss=0.5507, lr=0.044122, batch_cost=0.3445, reader_cost=0.1975 | ETA 13:19:20 2020-10-31 22:49:08 [INFO] [TRAIN] epoch=113, iter=21000/160000, loss=0.5466, lr=0.044065, batch_cost=0.3398, reader_cost=0.1963 | ETA 13:07:05 2020-10-31 22:50:17 [INFO] [TRAIN] epoch=114, iter=21200/160000, loss=0.5599, lr=0.044008, batch_cost=0.3447, reader_cost=0.2007 | ETA 13:17:28 2020-10-31 22:51:27 [INFO] [TRAIN] epoch=116, iter=21400/160000, loss=0.5442, lr=0.043951, batch_cost=0.3516, reader_cost=0.2055 | ETA 13:32:05 2020-10-31 22:52:37 [INFO] [TRAIN] epoch=117, iter=21600/160000, loss=0.5565, lr=0.043894, batch_cost=0.3463, reader_cost=0.1958 | ETA 13:18:41 2020-10-31 22:53:47 [INFO] [TRAIN] epoch=118, iter=21800/160000, loss=0.5545, lr=0.043837, batch_cost=0.3503, reader_cost=0.2038 | ETA 13:26:47 2020-10-31 22:54:55 [INFO] [TRAIN] epoch=119, iter=22000/160000, loss=0.5461, lr=0.043780, batch_cost=0.3408, reader_cost=0.1883 | ETA 13:03:54 2020-10-31 22:56:05 [INFO] [TRAIN] epoch=120, iter=22200/160000, loss=0.5487, lr=0.043723, batch_cost=0.3526, reader_cost=0.1983 | ETA 13:29:48 2020-10-31 22:57:14 [INFO] [TRAIN] epoch=121, iter=22400/160000, loss=0.5671, lr=0.043666, batch_cost=0.3410, reader_cost=0.1971 | ETA 13:02:00 2020-10-31 22:58:23 [INFO] [TRAIN] epoch=122, iter=22600/160000, loss=0.5480, lr=0.043609, batch_cost=0.3460, reader_cost=0.2047 | ETA 13:12:16 2020-10-31 22:59:31 [INFO] [TRAIN] epoch=123, iter=22800/160000, loss=0.5720, lr=0.043552, batch_cost=0.3423, reader_cost=0.1930 | ETA 13:02:42 2020-10-31 23:00:42 [INFO] [TRAIN] epoch=124, iter=23000/160000, loss=0.5379, lr=0.043495, batch_cost=0.3528, reader_cost=0.2095 | ETA 13:25:35 2020-10-31 23:01:51 [INFO] [TRAIN] epoch=125, iter=23200/160000, loss=0.5513, lr=0.043438, batch_cost=0.3476, reader_cost=0.1971 | ETA 13:12:31 2020-10-31 23:03:02 [INFO] [TRAIN] epoch=126, iter=23400/160000, loss=0.5533, lr=0.043381, batch_cost=0.3509, reader_cost=0.1996 | ETA 13:18:51 2020-10-31 23:04:11 [INFO] [TRAIN] epoch=127, iter=23600/160000, loss=0.5602, lr=0.043324, batch_cost=0.3483, reader_cost=0.1995 | ETA 13:11:52 2020-10-31 23:05:20 [INFO] [TRAIN] epoch=128, iter=23800/160000, loss=0.5497, lr=0.043267, batch_cost=0.3461, reader_cost=0.2047 | ETA 13:05:37 2020-10-31 23:06:32 [INFO] [TRAIN] epoch=130, iter=24000/160000, loss=0.5316, lr=0.043210, batch_cost=0.3575, reader_cost=0.2118 | ETA 13:30:21 2020-10-31 23:06:32 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-10-31 23:09:25 [INFO] [EVAL] #Images=500 mIoU=0.5747 Acc=0.9288 Kappa=0.9075 2020-10-31 23:09:25 [INFO] [EVAL] Category IoU: [0.9634 0.7365 0.8723 0.3122 0.3781 0.4212 0.3974 0.5418 0.8913 0.5359 0.9189 0.6283 0.3504 0.8813 0.4334 0.4883 0.2771 0.298 0.593 ] 2020-10-31 23:09:25 [INFO] [EVAL] Category Acc: [0.9833 0.822 0.9224 0.7849 0.7058 0.7622 0.6719 0.8123 0.9286 0.8281 0.9563 0.7185 0.6054 0.9117 0.6058 0.742 0.424 0.4884 0.7173] 2020-10-31 23:09:25 [INFO] [EVAL] The model with the best validation mIoU (0.5747) was saved at iter 24000. 2020-10-31 23:10:31 [INFO] [TRAIN] epoch=131, iter=24200/160000, loss=0.5349, lr=0.043153, batch_cost=0.3315, reader_cost=0.1902 | ETA 12:30:16 2020-10-31 23:11:40 [INFO] [TRAIN] epoch=132, iter=24400/160000, loss=0.5418, lr=0.043096, batch_cost=0.3409, reader_cost=0.1782 | ETA 12:50:21 2020-10-31 23:12:47 [INFO] [TRAIN] epoch=133, iter=24600/160000, loss=0.5387, lr=0.043039, batch_cost=0.3388, reader_cost=0.1804 | ETA 12:44:30 2020-10-31 23:13:56 [INFO] [TRAIN] epoch=134, iter=24800/160000, loss=0.5445, lr=0.042982, batch_cost=0.3431, reader_cost=0.2026 | ETA 12:53:09 2020-10-31 23:15:05 [INFO] [TRAIN] epoch=135, iter=25000/160000, loss=0.5353, lr=0.042925, batch_cost=0.3457, reader_cost=0.2024 | ETA 12:57:50 2020-10-31 23:16:15 [INFO] [TRAIN] epoch=136, iter=25200/160000, loss=0.5471, lr=0.042868, batch_cost=0.3503, reader_cost=0.2060 | ETA 13:07:00 2020-10-31 23:17:24 [INFO] [TRAIN] epoch=137, iter=25400/160000, loss=0.5429, lr=0.042811, batch_cost=0.3440, reader_cost=0.2020 | ETA 12:51:42 2020-10-31 23:18:33 [INFO] [TRAIN] epoch=138, iter=25600/160000, loss=0.5567, lr=0.042754, batch_cost=0.3455, reader_cost=0.1997 | ETA 12:53:52 2020-10-31 23:19:43 [INFO] [TRAIN] epoch=139, iter=25800/160000, loss=0.5484, lr=0.042696, batch_cost=0.3480, reader_cost=0.2051 | ETA 12:58:16 2020-10-31 23:20:51 [INFO] [TRAIN] epoch=140, iter=26000/160000, loss=0.5725, lr=0.042639, batch_cost=0.3399, reader_cost=0.1982 | ETA 12:39:05 2020-10-31 23:21:58 [INFO] [TRAIN] epoch=141, iter=26200/160000, loss=0.5619, lr=0.042582, batch_cost=0.3375, reader_cost=0.1900 | ETA 12:32:38 2020-10-31 23:23:06 [INFO] [TRAIN] epoch=142, iter=26400/160000, loss=0.5387, lr=0.042525, batch_cost=0.3385, reader_cost=0.1918 | ETA 12:33:41 2020-10-31 23:24:16 [INFO] [TRAIN] epoch=144, iter=26600/160000, loss=0.5389, lr=0.042468, batch_cost=0.3491, reader_cost=0.1928 | ETA 12:56:05 2020-10-31 23:25:24 [INFO] [TRAIN] epoch=145, iter=26800/160000, loss=0.5289, lr=0.042411, batch_cost=0.3410, reader_cost=0.1954 | ETA 12:36:57 2020-10-31 23:26:34 [INFO] [TRAIN] epoch=146, iter=27000/160000, loss=0.5215, lr=0.042353, batch_cost=0.3507, reader_cost=0.2001 | ETA 12:57:24 2020-10-31 23:27:44 [INFO] [TRAIN] epoch=147, iter=27200/160000, loss=0.5254, lr=0.042296, batch_cost=0.3512, reader_cost=0.2057 | ETA 12:57:14 2020-10-31 23:28:54 [INFO] [TRAIN] epoch=148, iter=27400/160000, loss=0.5413, lr=0.042239, batch_cost=0.3502, reader_cost=0.2079 | ETA 12:54:02 2020-10-31 23:30:03 [INFO] [TRAIN] epoch=149, iter=27600/160000, loss=0.5145, lr=0.042182, batch_cost=0.3458, reader_cost=0.2045 | ETA 12:43:05 2020-10-31 23:31:13 [INFO] [TRAIN] epoch=150, iter=27800/160000, loss=0.5291, lr=0.042125, batch_cost=0.3465, reader_cost=0.2026 | ETA 12:43:20 2020-10-31 23:32:21 [INFO] [TRAIN] epoch=151, iter=28000/160000, loss=0.5549, lr=0.042067, batch_cost=0.3415, reader_cost=0.1959 | ETA 12:31:21 2020-10-31 23:33:28 [INFO] [TRAIN] epoch=152, iter=28200/160000, loss=0.5310, lr=0.042010, batch_cost=0.3341, reader_cost=0.1902 | ETA 12:13:54 2020-10-31 23:34:36 [INFO] [TRAIN] epoch=153, iter=28400/160000, loss=0.5549, lr=0.041953, batch_cost=0.3423, reader_cost=0.1968 | ETA 12:30:48 2020-10-31 23:35:47 [INFO] [TRAIN] epoch=154, iter=28600/160000, loss=0.5286, lr=0.041896, batch_cost=0.3532, reader_cost=0.2111 | ETA 12:53:32 2020-10-31 23:36:57 [INFO] [TRAIN] epoch=155, iter=28800/160000, loss=0.5408, lr=0.041838, batch_cost=0.3520, reader_cost=0.2066 | ETA 12:49:38 2020-10-31 23:38:07 [INFO] [TRAIN] epoch=156, iter=29000/160000, loss=0.5423, lr=0.041781, batch_cost=0.3474, reader_cost=0.2005 | ETA 12:38:23 2020-10-31 23:39:15 [INFO] [TRAIN] epoch=157, iter=29200/160000, loss=0.5322, lr=0.041724, batch_cost=0.3427, reader_cost=0.1998 | ETA 12:27:04 2020-10-31 23:40:27 [INFO] [TRAIN] epoch=159, iter=29400/160000, loss=0.5324, lr=0.041667, batch_cost=0.3564, reader_cost=0.2090 | ETA 12:55:48 2020-10-31 23:41:37 [INFO] [TRAIN] epoch=160, iter=29600/160000, loss=0.5220, lr=0.041609, batch_cost=0.3498, reader_cost=0.2052 | ETA 12:40:12 2020-10-31 23:42:44 [INFO] [TRAIN] epoch=161, iter=29800/160000, loss=0.5245, lr=0.041552, batch_cost=0.3380, reader_cost=0.1963 | ETA 12:13:31 2020-10-31 23:43:55 [INFO] [TRAIN] epoch=162, iter=30000/160000, loss=0.5163, lr=0.041495, batch_cost=0.3553, reader_cost=0.2090 | ETA 12:49:54 2020-10-31 23:45:05 [INFO] [TRAIN] epoch=163, iter=30200/160000, loss=0.5189, lr=0.041437, batch_cost=0.3492, reader_cost=0.2002 | ETA 12:35:20 2020-10-31 23:46:13 [INFO] [TRAIN] epoch=164, iter=30400/160000, loss=0.5238, lr=0.041380, batch_cost=0.3416, reader_cost=0.1991 | ETA 12:17:44 2020-10-31 23:47:26 [INFO] [TRAIN] epoch=165, iter=30600/160000, loss=0.5305, lr=0.041323, batch_cost=0.3647, reader_cost=0.2199 | ETA 13:06:29 2020-10-31 23:48:34 [INFO] [TRAIN] epoch=166, iter=30800/160000, loss=0.5609, lr=0.041265, batch_cost=0.3369, reader_cost=0.1910 | ETA 12:05:25 2020-10-31 23:49:44 [INFO] [TRAIN] epoch=167, iter=31000/160000, loss=0.5307, lr=0.041208, batch_cost=0.3491, reader_cost=0.2061 | ETA 12:30:29 2020-10-31 23:50:54 [INFO] [TRAIN] epoch=168, iter=31200/160000, loss=0.5209, lr=0.041151, batch_cost=0.3522, reader_cost=0.2094 | ETA 12:36:04 2020-10-31 23:52:03 [INFO] [TRAIN] epoch=169, iter=31400/160000, loss=0.5344, lr=0.041093, batch_cost=0.3445, reader_cost=0.2031 | ETA 12:18:24 2020-10-31 23:53:12 [INFO] [TRAIN] epoch=170, iter=31600/160000, loss=0.5266, lr=0.041036, batch_cost=0.3464, reader_cost=0.2050 | ETA 12:21:14 2020-10-31 23:54:22 [INFO] [TRAIN] epoch=171, iter=31800/160000, loss=0.5163, lr=0.040978, batch_cost=0.3491, reader_cost=0.2086 | ETA 12:25:57 2020-10-31 23:55:33 [INFO] [TRAIN] epoch=173, iter=32000/160000, loss=0.5380, lr=0.040921, batch_cost=0.3558, reader_cost=0.2044 | ETA 12:39:06 2020-10-31 23:55:33 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-10-31 23:58:25 [INFO] [EVAL] #Images=500 mIoU=0.5567 Acc=0.9216 Kappa=0.8983 2020-10-31 23:58:25 [INFO] [EVAL] Category IoU: [0.9563 0.6979 0.8629 0.2562 0.3504 0.4385 0.3134 0.5011 0.8789 0.5213 0.9228 0.6297 0.3565 0.8904 0.4484 0.4436 0.3917 0.1403 0.578 ] 2020-10-31 23:58:25 [INFO] [EVAL] Category Acc: [0.9863 0.7801 0.9125 0.6031 0.4203 0.7191 0.8396 0.8511 0.9171 0.8178 0.9643 0.8007 0.6378 0.9506 0.7224 0.8576 0.5394 0.6137 0.6934] 2020-10-31 23:58:25 [INFO] [EVAL] The model with the best validation mIoU (0.5747) was saved at iter 24000. 2020-10-31 23:59:32 [INFO] [TRAIN] epoch=174, iter=32200/160000, loss=0.5250, lr=0.040864, batch_cost=0.3337, reader_cost=0.1848 | ETA 11:50:43 2020-11-01 00:00:39 [INFO] [TRAIN] epoch=175, iter=32400/160000, loss=0.5162, lr=0.040806, batch_cost=0.3348, reader_cost=0.1944 | ETA 11:52:05 2020-11-01 00:01:49 [INFO] [TRAIN] epoch=176, iter=32600/160000, loss=0.5097, lr=0.040749, batch_cost=0.3515, reader_cost=0.2064 | ETA 12:26:24 2020-11-01 00:02:56 [INFO] [TRAIN] epoch=177, iter=32800/160000, loss=0.5176, lr=0.040691, batch_cost=0.3314, reader_cost=0.1892 | ETA 11:42:34 2020-11-01 00:04:04 [INFO] [TRAIN] epoch=178, iter=33000/160000, loss=0.5282, lr=0.040634, batch_cost=0.3408, reader_cost=0.1995 | ETA 12:01:24 2020-11-01 00:05:12 [INFO] [TRAIN] epoch=179, iter=33200/160000, loss=0.5250, lr=0.040577, batch_cost=0.3411, reader_cost=0.1929 | ETA 12:00:55 2020-11-01 00:06:21 [INFO] [TRAIN] epoch=180, iter=33400/160000, loss=0.5128, lr=0.040519, batch_cost=0.3466, reader_cost=0.2032 | ETA 12:11:22 2020-11-01 00:07:29 [INFO] [TRAIN] epoch=181, iter=33600/160000, loss=0.5293, lr=0.040462, batch_cost=0.3395, reader_cost=0.2000 | ETA 11:55:07 2020-11-01 00:08:38 [INFO] [TRAIN] epoch=182, iter=33800/160000, loss=0.5419, lr=0.040404, batch_cost=0.3420, reader_cost=0.1985 | ETA 11:59:25 2020-11-01 00:09:46 [INFO] [TRAIN] epoch=183, iter=34000/160000, loss=0.5198, lr=0.040347, batch_cost=0.3401, reader_cost=0.1966 | ETA 11:54:14 2020-11-01 00:10:53 [INFO] [TRAIN] epoch=184, iter=34200/160000, loss=0.5170, lr=0.040289, batch_cost=0.3366, reader_cost=0.1879 | ETA 11:45:48 2020-11-01 00:12:00 [INFO] [TRAIN] epoch=185, iter=34400/160000, loss=0.5123, lr=0.040232, batch_cost=0.3352, reader_cost=0.1868 | ETA 11:41:40 2020-11-01 00:13:11 [INFO] [TRAIN] epoch=187, iter=34600/160000, loss=0.5161, lr=0.040174, batch_cost=0.3562, reader_cost=0.1982 | ETA 12:24:29 2020-11-01 00:14:22 [INFO] [TRAIN] epoch=188, iter=34800/160000, loss=0.5114, lr=0.040117, batch_cost=0.3540, reader_cost=0.2113 | ETA 12:18:34 2020-11-01 00:15:30 [INFO] [TRAIN] epoch=189, iter=35000/160000, loss=0.5136, lr=0.040059, batch_cost=0.3407, reader_cost=0.1951 | ETA 11:49:46 2020-11-01 00:16:39 [INFO] [TRAIN] epoch=190, iter=35200/160000, loss=0.5280, lr=0.040001, batch_cost=0.3447, reader_cost=0.1924 | ETA 11:56:55 2020-11-01 00:17:48 [INFO] [TRAIN] epoch=191, iter=35400/160000, loss=0.5250, lr=0.039944, batch_cost=0.3438, reader_cost=0.1775 | ETA 11:53:57 2020-11-01 00:18:57 [INFO] [TRAIN] epoch=192, iter=35600/160000, loss=0.5283, lr=0.039886, batch_cost=0.3472, reader_cost=0.1965 | ETA 11:59:47 2020-11-01 00:20:06 [INFO] [TRAIN] epoch=193, iter=35800/160000, loss=0.5207, lr=0.039829, batch_cost=0.3435, reader_cost=0.1970 | ETA 11:51:08 2020-11-01 00:21:13 [INFO] [TRAIN] epoch=194, iter=36000/160000, loss=0.5336, lr=0.039771, batch_cost=0.3346, reader_cost=0.1848 | ETA 11:31:35 2020-11-01 00:22:20 [INFO] [TRAIN] epoch=195, iter=36200/160000, loss=0.5130, lr=0.039714, batch_cost=0.3348, reader_cost=0.1789 | ETA 11:30:51 2020-11-01 00:23:28 [INFO] [TRAIN] epoch=196, iter=36400/160000, loss=0.5097, lr=0.039656, batch_cost=0.3406, reader_cost=0.1896 | ETA 11:41:40 2020-11-01 00:24:37 [INFO] [TRAIN] epoch=197, iter=36600/160000, loss=0.5191, lr=0.039598, batch_cost=0.3434, reader_cost=0.1968 | ETA 11:46:10 2020-11-01 00:25:44 [INFO] [TRAIN] epoch=198, iter=36800/160000, loss=0.5026, lr=0.039541, batch_cost=0.3382, reader_cost=0.1940 | ETA 11:34:25 2020-11-01 00:26:53 [INFO] [TRAIN] epoch=199, iter=37000/160000, loss=0.4929, lr=0.039483, batch_cost=0.3419, reader_cost=0.1912 | ETA 11:40:48 2020-11-01 00:28:01 [INFO] [TRAIN] epoch=200, iter=37200/160000, loss=0.5100, lr=0.039426, batch_cost=0.3409, reader_cost=0.1952 | ETA 11:37:44 2020-11-01 00:29:15 [INFO] [TRAIN] epoch=202, iter=37400/160000, loss=0.5085, lr=0.039368, batch_cost=0.3692, reader_cost=0.2143 | ETA 12:34:29 2020-11-01 00:30:23 [INFO] [TRAIN] epoch=203, iter=37600/160000, loss=0.5167, lr=0.039310, batch_cost=0.3409, reader_cost=0.1847 | ETA 11:35:28 2020-11-01 00:31:32 [INFO] [TRAIN] epoch=204, iter=37800/160000, loss=0.5146, lr=0.039253, batch_cost=0.3467, reader_cost=0.2030 | ETA 11:46:01 2020-11-01 00:32:39 [INFO] [TRAIN] epoch=205, iter=38000/160000, loss=0.5233, lr=0.039195, batch_cost=0.3356, reader_cost=0.1809 | ETA 11:22:24 2020-11-01 00:33:46 [INFO] [TRAIN] epoch=206, iter=38200/160000, loss=0.5212, lr=0.039137, batch_cost=0.3331, reader_cost=0.1717 | ETA 11:16:11 2020-11-01 00:34:54 [INFO] [TRAIN] epoch=207, iter=38400/160000, loss=0.5180, lr=0.039079, batch_cost=0.3379, reader_cost=0.1856 | ETA 11:24:49 2020-11-01 00:36:02 [INFO] [TRAIN] epoch=208, iter=38600/160000, loss=0.5115, lr=0.039022, batch_cost=0.3416, reader_cost=0.1935 | ETA 11:31:15 2020-11-01 00:37:10 [INFO] [TRAIN] epoch=209, iter=38800/160000, loss=0.5119, lr=0.038964, batch_cost=0.3417, reader_cost=0.1990 | ETA 11:30:14 2020-11-01 00:38:19 [INFO] [TRAIN] epoch=210, iter=39000/160000, loss=0.5059, lr=0.038906, batch_cost=0.3433, reader_cost=0.1973 | ETA 11:32:24 2020-11-01 00:39:27 [INFO] [TRAIN] epoch=211, iter=39200/160000, loss=0.4977, lr=0.038849, batch_cost=0.3411, reader_cost=0.1997 | ETA 11:26:43 2020-11-01 00:40:35 [INFO] [TRAIN] epoch=212, iter=39400/160000, loss=0.5066, lr=0.038791, batch_cost=0.3407, reader_cost=0.1724 | ETA 11:24:47 2020-11-01 00:41:44 [INFO] [TRAIN] epoch=213, iter=39600/160000, loss=0.5149, lr=0.038733, batch_cost=0.3414, reader_cost=0.1886 | ETA 11:24:59 2020-11-01 00:42:53 [INFO] [TRAIN] epoch=214, iter=39800/160000, loss=0.5225, lr=0.038675, batch_cost=0.3490, reader_cost=0.2007 | ETA 11:39:15 2020-11-01 00:44:04 [INFO] [TRAIN] epoch=216, iter=40000/160000, loss=0.5312, lr=0.038618, batch_cost=0.3523, reader_cost=0.2076 | ETA 11:44:35 2020-11-01 00:44:04 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-01 00:47:03 [INFO] [EVAL] #Images=500 mIoU=0.5966 Acc=0.9328 Kappa=0.9127 2020-11-01 00:47:03 [INFO] [EVAL] Category IoU: [0.9634 0.734 0.8796 0.4331 0.4121 0.471 0.4394 0.5928 0.8951 0.5804 0.9203 0.6704 0.392 0.8928 0.4686 0.5281 0.0944 0.3481 0.6198] 2020-11-01 00:47:03 [INFO] [EVAL] Category Acc: [0.9787 0.864 0.9345 0.6962 0.7876 0.6879 0.681 0.7962 0.9284 0.8207 0.9389 0.7513 0.6996 0.9415 0.6076 0.6928 0.6298 0.5842 0.7216] 2020-11-01 00:47:03 [INFO] [EVAL] The model with the best validation mIoU (0.5966) was saved at iter 40000. 2020-11-01 00:48:10 [INFO] [TRAIN] epoch=217, iter=40200/160000, loss=0.5272, lr=0.038560, batch_cost=0.3344, reader_cost=0.1883 | ETA 11:07:45 2020-11-01 00:49:17 [INFO] [TRAIN] epoch=218, iter=40400/160000, loss=0.5134, lr=0.038502, batch_cost=0.3363, reader_cost=0.1879 | ETA 11:10:23 2020-11-01 00:50:26 [INFO] [TRAIN] epoch=219, iter=40600/160000, loss=0.5096, lr=0.038444, batch_cost=0.3438, reader_cost=0.1986 | ETA 11:24:09 2020-11-01 00:51:35 [INFO] [TRAIN] epoch=220, iter=40800/160000, loss=0.5047, lr=0.038386, batch_cost=0.3447, reader_cost=0.1833 | ETA 11:24:45 2020-11-01 00:52:44 [INFO] [TRAIN] epoch=221, iter=41000/160000, loss=0.5003, lr=0.038329, batch_cost=0.3434, reader_cost=0.1978 | ETA 11:21:08 2020-11-01 00:53:54 [INFO] [TRAIN] epoch=222, iter=41200/160000, loss=0.5068, lr=0.038271, batch_cost=0.3498, reader_cost=0.2055 | ETA 11:32:34 2020-11-01 00:55:02 [INFO] [TRAIN] epoch=223, iter=41400/160000, loss=0.5031, lr=0.038213, batch_cost=0.3429, reader_cost=0.1908 | ETA 11:17:42 2020-11-01 00:56:12 [INFO] [TRAIN] epoch=224, iter=41600/160000, loss=0.5083, lr=0.038155, batch_cost=0.3480, reader_cost=0.1994 | ETA 11:26:46 2020-11-01 00:57:22 [INFO] [TRAIN] epoch=225, iter=41800/160000, loss=0.5106, lr=0.038097, batch_cost=0.3498, reader_cost=0.1992 | ETA 11:29:08 2020-11-01 00:58:32 [INFO] [TRAIN] epoch=226, iter=42000/160000, loss=0.5065, lr=0.038039, batch_cost=0.3499, reader_cost=0.2058 | ETA 11:28:09 2020-11-01 00:59:39 [INFO] [TRAIN] epoch=227, iter=42200/160000, loss=0.5020, lr=0.037981, batch_cost=0.3384, reader_cost=0.1951 | ETA 11:04:29 2020-11-01 01:00:47 [INFO] [TRAIN] epoch=228, iter=42400/160000, loss=0.5213, lr=0.037924, batch_cost=0.3381, reader_cost=0.1902 | ETA 11:02:43 2020-11-01 01:01:56 [INFO] [TRAIN] epoch=230, iter=42600/160000, loss=0.4955, lr=0.037866, batch_cost=0.3463, reader_cost=0.1913 | ETA 11:17:30 2020-11-01 01:03:05 [INFO] [TRAIN] epoch=231, iter=42800/160000, loss=0.5106, lr=0.037808, batch_cost=0.3428, reader_cost=0.1944 | ETA 11:09:36 2020-11-01 01:04:14 [INFO] [TRAIN] epoch=232, iter=43000/160000, loss=0.4870, lr=0.037750, batch_cost=0.3451, reader_cost=0.2019 | ETA 11:12:59 2020-11-01 01:05:22 [INFO] [TRAIN] epoch=233, iter=43200/160000, loss=0.4971, lr=0.037692, batch_cost=0.3397, reader_cost=0.1925 | ETA 11:01:19 2020-11-01 01:06:28 [INFO] [TRAIN] epoch=234, iter=43400/160000, loss=0.5153, lr=0.037634, batch_cost=0.3315, reader_cost=0.1767 | ETA 10:44:08 2020-11-01 01:07:35 [INFO] [TRAIN] epoch=235, iter=43600/160000, loss=0.5207, lr=0.037576, batch_cost=0.3337, reader_cost=0.1883 | ETA 10:47:20 2020-11-01 01:08:44 [INFO] [TRAIN] epoch=236, iter=43800/160000, loss=0.4972, lr=0.037518, batch_cost=0.3451, reader_cost=0.2034 | ETA 11:08:24 2020-11-01 01:09:52 [INFO] [TRAIN] epoch=237, iter=44000/160000, loss=0.4959, lr=0.037460, batch_cost=0.3404, reader_cost=0.1918 | ETA 10:58:00 2020-11-01 01:11:03 [INFO] [TRAIN] epoch=238, iter=44200/160000, loss=0.5166, lr=0.037402, batch_cost=0.3555, reader_cost=0.2063 | ETA 11:26:03 2020-11-01 01:12:10 [INFO] [TRAIN] epoch=239, iter=44400/160000, loss=0.5141, lr=0.037344, batch_cost=0.3349, reader_cost=0.1883 | ETA 10:45:11 2020-11-01 01:13:19 [INFO] [TRAIN] epoch=240, iter=44600/160000, loss=0.5026, lr=0.037286, batch_cost=0.3462, reader_cost=0.2007 | ETA 11:05:52 2020-11-01 01:14:27 [INFO] [TRAIN] epoch=241, iter=44800/160000, loss=0.5032, lr=0.037228, batch_cost=0.3377, reader_cost=0.1807 | ETA 10:48:28 2020-11-01 01:15:36 [INFO] [TRAIN] epoch=242, iter=45000/160000, loss=0.5009, lr=0.037170, batch_cost=0.3454, reader_cost=0.1968 | ETA 11:02:01 2020-11-01 01:16:46 [INFO] [TRAIN] epoch=244, iter=45200/160000, loss=0.5156, lr=0.037112, batch_cost=0.3483, reader_cost=0.2042 | ETA 11:06:20 2020-11-01 01:17:53 [INFO] [TRAIN] epoch=245, iter=45400/160000, loss=0.5121, lr=0.037054, batch_cost=0.3354, reader_cost=0.1867 | ETA 10:40:31 2020-11-01 01:19:01 [INFO] [TRAIN] epoch=246, iter=45600/160000, loss=0.5004, lr=0.036996, batch_cost=0.3438, reader_cost=0.2043 | ETA 10:55:35 2020-11-01 01:20:09 [INFO] [TRAIN] epoch=247, iter=45800/160000, loss=0.5261, lr=0.036938, batch_cost=0.3366, reader_cost=0.1905 | ETA 10:40:35 2020-11-01 01:21:17 [INFO] [TRAIN] epoch=248, iter=46000/160000, loss=0.5139, lr=0.036880, batch_cost=0.3438, reader_cost=0.2022 | ETA 10:53:10 2020-11-01 01:22:26 [INFO] [TRAIN] epoch=249, iter=46200/160000, loss=0.5024, lr=0.036822, batch_cost=0.3452, reader_cost=0.1951 | ETA 10:54:41 2020-11-01 01:23:36 [INFO] [TRAIN] epoch=250, iter=46400/160000, loss=0.5077, lr=0.036764, batch_cost=0.3498, reader_cost=0.2102 | ETA 11:02:21 2020-11-01 01:24:45 [INFO] [TRAIN] epoch=251, iter=46600/160000, loss=0.4993, lr=0.036706, batch_cost=0.3419, reader_cost=0.1989 | ETA 10:46:12 2020-11-01 01:25:52 [INFO] [TRAIN] epoch=252, iter=46800/160000, loss=0.5010, lr=0.036648, batch_cost=0.3363, reader_cost=0.1862 | ETA 10:34:25 2020-11-01 01:27:00 [INFO] [TRAIN] epoch=253, iter=47000/160000, loss=0.4867, lr=0.036589, batch_cost=0.3376, reader_cost=0.1837 | ETA 10:35:53 2020-11-01 01:28:10 [INFO] [TRAIN] epoch=254, iter=47200/160000, loss=0.5169, lr=0.036531, batch_cost=0.3495, reader_cost=0.1988 | ETA 10:57:06 2020-11-01 01:29:18 [INFO] [TRAIN] epoch=255, iter=47400/160000, loss=0.4756, lr=0.036473, batch_cost=0.3412, reader_cost=0.2002 | ETA 10:40:15 2020-11-01 01:30:26 [INFO] [TRAIN] epoch=256, iter=47600/160000, loss=0.5001, lr=0.036415, batch_cost=0.3434, reader_cost=0.1995 | ETA 10:43:22 2020-11-01 01:31:35 [INFO] [TRAIN] epoch=257, iter=47800/160000, loss=0.5000, lr=0.036357, batch_cost=0.3441, reader_cost=0.2024 | ETA 10:43:30 2020-11-01 01:32:46 [INFO] [TRAIN] epoch=259, iter=48000/160000, loss=0.4969, lr=0.036299, batch_cost=0.3511, reader_cost=0.2040 | ETA 10:55:26 2020-11-01 01:32:46 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-01 01:35:38 [INFO] [EVAL] #Images=500 mIoU=0.6150 Acc=0.9349 Kappa=0.9151 2020-11-01 01:35:38 [INFO] [EVAL] Category IoU: [0.9631 0.724 0.8847 0.3474 0.4428 0.4707 0.437 0.6022 0.8962 0.548 0.9289 0.679 0.4185 0.9046 0.5285 0.6277 0.4086 0.2614 0.6112] 2020-11-01 01:35:38 [INFO] [EVAL] Category Acc: [0.976 0.8562 0.9285 0.8046 0.7224 0.7584 0.7746 0.8627 0.9222 0.8589 0.9653 0.8092 0.6099 0.9521 0.7905 0.7575 0.7525 0.6878 0.7147] 2020-11-01 01:35:38 [INFO] [EVAL] The model with the best validation mIoU (0.6150) was saved at iter 48000. 2020-11-01 01:36:46 [INFO] [TRAIN] epoch=260, iter=48200/160000, loss=0.5032, lr=0.036240, batch_cost=0.3404, reader_cost=0.1985 | ETA 10:34:19 2020-11-01 01:37:54 [INFO] [TRAIN] epoch=261, iter=48400/160000, loss=0.5006, lr=0.036182, batch_cost=0.3373, reader_cost=0.1896 | ETA 10:27:26 2020-11-01 01:39:01 [INFO] [TRAIN] epoch=262, iter=48600/160000, loss=0.5126, lr=0.036124, batch_cost=0.3395, reader_cost=0.1900 | ETA 10:30:22 2020-11-01 01:40:07 [INFO] [TRAIN] epoch=263, iter=48800/160000, loss=0.4881, lr=0.036066, batch_cost=0.3267, reader_cost=0.1790 | ETA 10:05:32 2020-11-01 01:41:11 [INFO] [TRAIN] epoch=264, iter=49000/160000, loss=0.5102, lr=0.036008, batch_cost=0.3194, reader_cost=0.1662 | ETA 09:50:53 2020-11-01 01:42:14 [INFO] [TRAIN] epoch=265, iter=49200/160000, loss=0.4987, lr=0.035949, batch_cost=0.3148, reader_cost=0.1710 | ETA 09:41:20 2020-11-01 01:43:18 [INFO] [TRAIN] epoch=266, iter=49400/160000, loss=0.5049, lr=0.035891, batch_cost=0.3216, reader_cost=0.1316 | ETA 09:52:51 2020-11-01 01:44:22 [INFO] [TRAIN] epoch=267, iter=49600/160000, loss=0.4902, lr=0.035833, batch_cost=0.3209, reader_cost=0.1507 | ETA 09:50:31 2020-11-01 01:45:30 [INFO] [TRAIN] epoch=268, iter=49800/160000, loss=0.4969, lr=0.035775, batch_cost=0.3400, reader_cost=0.1984 | ETA 10:24:27 2020-11-01 01:46:38 [INFO] [TRAIN] epoch=269, iter=50000/160000, loss=0.4944, lr=0.035716, batch_cost=0.3365, reader_cost=0.1916 | ETA 10:16:57 2020-11-01 01:47:46 [INFO] [TRAIN] epoch=270, iter=50200/160000, loss=0.4936, lr=0.035658, batch_cost=0.3425, reader_cost=0.1986 | ETA 10:26:42 2020-11-01 01:48:53 [INFO] [TRAIN] epoch=271, iter=50400/160000, loss=0.4920, lr=0.035600, batch_cost=0.3346, reader_cost=0.1873 | ETA 10:11:06 2020-11-01 01:50:02 [INFO] [TRAIN] epoch=273, iter=50600/160000, loss=0.5064, lr=0.035541, batch_cost=0.3431, reader_cost=0.1953 | ETA 10:25:35 2020-11-01 01:51:09 [INFO] [TRAIN] epoch=274, iter=50800/160000, loss=0.5036, lr=0.035483, batch_cost=0.3356, reader_cost=0.1764 | ETA 10:10:52 2020-11-01 01:52:16 [INFO] [TRAIN] epoch=275, iter=51000/160000, loss=0.4870, lr=0.035425, batch_cost=0.3350, reader_cost=0.1807 | ETA 10:08:31 2020-11-01 01:53:23 [INFO] [TRAIN] epoch=276, iter=51200/160000, loss=0.4921, lr=0.035366, batch_cost=0.3385, reader_cost=0.1924 | ETA 10:13:53 2020-11-01 01:54:30 [INFO] [TRAIN] epoch=277, iter=51400/160000, loss=0.4854, lr=0.035308, batch_cost=0.3346, reader_cost=0.1838 | ETA 10:05:39 2020-11-01 01:55:37 [INFO] [TRAIN] epoch=278, iter=51600/160000, loss=0.5024, lr=0.035250, batch_cost=0.3341, reader_cost=0.1858 | ETA 10:03:38 2020-11-01 01:56:45 [INFO] [TRAIN] epoch=279, iter=51800/160000, loss=0.4944, lr=0.035191, batch_cost=0.3394, reader_cost=0.1985 | ETA 10:12:03 2020-11-01 01:57:54 [INFO] [TRAIN] epoch=280, iter=52000/160000, loss=0.4873, lr=0.035133, batch_cost=0.3428, reader_cost=0.1980 | ETA 10:17:05 2020-11-01 01:59:02 [INFO] [TRAIN] epoch=281, iter=52200/160000, loss=0.5167, lr=0.035075, batch_cost=0.3426, reader_cost=0.2007 | ETA 10:15:27 2020-11-01 02:00:10 [INFO] [TRAIN] epoch=282, iter=52400/160000, loss=0.4966, lr=0.035016, batch_cost=0.3374, reader_cost=0.1909 | ETA 10:04:59 2020-11-01 02:01:17 [INFO] [TRAIN] epoch=283, iter=52600/160000, loss=0.4878, lr=0.034958, batch_cost=0.3379, reader_cost=0.1953 | ETA 10:04:47 2020-11-01 02:02:24 [INFO] [TRAIN] epoch=284, iter=52800/160000, loss=0.5074, lr=0.034899, batch_cost=0.3345, reader_cost=0.1905 | ETA 09:57:41 2020-11-01 02:03:31 [INFO] [TRAIN] epoch=285, iter=53000/160000, loss=0.5039, lr=0.034841, batch_cost=0.3341, reader_cost=0.1869 | ETA 09:55:44 2020-11-01 02:04:42 [INFO] [TRAIN] epoch=287, iter=53200/160000, loss=0.4919, lr=0.034782, batch_cost=0.3572, reader_cost=0.2058 | ETA 10:35:48 2020-11-01 02:05:51 [INFO] [TRAIN] epoch=288, iter=53400/160000, loss=0.4832, lr=0.034724, batch_cost=0.3440, reader_cost=0.1955 | ETA 10:11:09 2020-11-01 02:06:59 [INFO] [TRAIN] epoch=289, iter=53600/160000, loss=0.4944, lr=0.034666, batch_cost=0.3392, reader_cost=0.1950 | ETA 10:01:29 2020-11-01 02:08:07 [INFO] [TRAIN] epoch=290, iter=53800/160000, loss=0.4935, lr=0.034607, batch_cost=0.3398, reader_cost=0.1975 | ETA 10:01:31 2020-11-01 02:09:15 [INFO] [TRAIN] epoch=291, iter=54000/160000, loss=0.4773, lr=0.034549, batch_cost=0.3412, reader_cost=0.1897 | ETA 10:02:50 2020-11-01 02:10:22 [INFO] [TRAIN] epoch=292, iter=54200/160000, loss=0.4918, lr=0.034490, batch_cost=0.3351, reader_cost=0.1931 | ETA 09:50:54 2020-11-01 02:11:27 [INFO] [TRAIN] epoch=293, iter=54400/160000, loss=0.4924, lr=0.034432, batch_cost=0.3262, reader_cost=0.1816 | ETA 09:34:02 2020-11-01 02:12:34 [INFO] [TRAIN] epoch=294, iter=54600/160000, loss=0.4897, lr=0.034373, batch_cost=0.3342, reader_cost=0.1911 | ETA 09:47:08 2020-11-01 02:13:40 [INFO] [TRAIN] epoch=295, iter=54800/160000, loss=0.4806, lr=0.034315, batch_cost=0.3310, reader_cost=0.1819 | ETA 09:40:20 2020-11-01 02:14:48 [INFO] [TRAIN] epoch=296, iter=55000/160000, loss=0.4988, lr=0.034256, batch_cost=0.3360, reader_cost=0.1914 | ETA 09:47:55 2020-11-01 02:15:52 [INFO] [TRAIN] epoch=297, iter=55200/160000, loss=0.4923, lr=0.034197, batch_cost=0.3240, reader_cost=0.1771 | ETA 09:25:55 2020-11-01 02:16:59 [INFO] [TRAIN] epoch=298, iter=55400/160000, loss=0.4718, lr=0.034139, batch_cost=0.3311, reader_cost=0.1795 | ETA 09:37:14 2020-11-01 02:18:06 [INFO] [TRAIN] epoch=299, iter=55600/160000, loss=0.4994, lr=0.034080, batch_cost=0.3383, reader_cost=0.1900 | ETA 09:48:38 2020-11-01 02:19:15 [INFO] [TRAIN] epoch=300, iter=55800/160000, loss=0.4825, lr=0.034022, batch_cost=0.3438, reader_cost=0.2006 | ETA 09:57:07 2020-11-01 02:20:24 [INFO] [TRAIN] epoch=302, iter=56000/160000, loss=0.4903, lr=0.033963, batch_cost=0.3460, reader_cost=0.1995 | ETA 09:59:41 2020-11-01 02:20:25 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-01 02:23:21 [INFO] [EVAL] #Images=500 mIoU=0.6273 Acc=0.9382 Kappa=0.9195 2020-11-01 02:23:21 [INFO] [EVAL] Category IoU: [0.9659 0.7457 0.8876 0.4152 0.4528 0.48 0.4563 0.6306 0.9015 0.6087 0.9274 0.6881 0.3921 0.9078 0.6343 0.6246 0.2762 0.2918 0.6324] 2020-11-01 02:23:21 [INFO] [EVAL] Category Acc: [0.9782 0.8862 0.9252 0.7817 0.7584 0.7327 0.7311 0.818 0.9391 0.845 0.9562 0.7983 0.7365 0.946 0.8352 0.7557 0.9103 0.6756 0.7155] 2020-11-01 02:23:21 [INFO] [EVAL] The model with the best validation mIoU (0.6273) was saved at iter 56000. 2020-11-01 02:24:26 [INFO] [TRAIN] epoch=303, iter=56200/160000, loss=0.4907, lr=0.033904, batch_cost=0.3241, reader_cost=0.1788 | ETA 09:20:45 2020-11-01 02:25:35 [INFO] [TRAIN] epoch=304, iter=56400/160000, loss=0.4870, lr=0.033846, batch_cost=0.3447, reader_cost=0.2019 | ETA 09:55:10 2020-11-01 02:26:43 [INFO] [TRAIN] epoch=305, iter=56600/160000, loss=0.4995, lr=0.033787, batch_cost=0.3423, reader_cost=0.1998 | ETA 09:49:49 2020-11-01 02:27:51 [INFO] [TRAIN] epoch=306, iter=56800/160000, loss=0.4894, lr=0.033729, batch_cost=0.3358, reader_cost=0.1949 | ETA 09:37:29 2020-11-01 02:28:57 [INFO] [TRAIN] epoch=307, iter=57000/160000, loss=0.4781, lr=0.033670, batch_cost=0.3331, reader_cost=0.1796 | ETA 09:31:45 2020-11-01 02:30:05 [INFO] [TRAIN] epoch=308, iter=57200/160000, loss=0.4901, lr=0.033611, batch_cost=0.3385, reader_cost=0.1936 | ETA 09:39:57 2020-11-01 02:31:14 [INFO] [TRAIN] epoch=309, iter=57400/160000, loss=0.4887, lr=0.033553, batch_cost=0.3455, reader_cost=0.2029 | ETA 09:50:50 2020-11-01 02:32:22 [INFO] [TRAIN] epoch=310, iter=57600/160000, loss=0.4948, lr=0.033494, batch_cost=0.3378, reader_cost=0.1943 | ETA 09:36:31 2020-11-01 02:33:30 [INFO] [TRAIN] epoch=311, iter=57800/160000, loss=0.4759, lr=0.033435, batch_cost=0.3396, reader_cost=0.1950 | ETA 09:38:22 2020-11-01 02:34:36 [INFO] [TRAIN] epoch=312, iter=58000/160000, loss=0.5022, lr=0.033376, batch_cost=0.3341, reader_cost=0.1794 | ETA 09:28:00 2020-11-01 02:35:44 [INFO] [TRAIN] epoch=313, iter=58200/160000, loss=0.4863, lr=0.033318, batch_cost=0.3380, reader_cost=0.1929 | ETA 09:33:30 2020-11-01 02:36:50 [INFO] [TRAIN] epoch=314, iter=58400/160000, loss=0.4853, lr=0.033259, batch_cost=0.3289, reader_cost=0.1780 | ETA 09:16:52 2020-11-01 02:37:59 [INFO] [TRAIN] epoch=316, iter=58600/160000, loss=0.4983, lr=0.033200, batch_cost=0.3456, reader_cost=0.1965 | ETA 09:44:02 2020-11-01 02:39:05 [INFO] [TRAIN] epoch=317, iter=58800/160000, loss=0.4783, lr=0.033141, batch_cost=0.3315, reader_cost=0.1779 | ETA 09:19:05 2020-11-01 02:40:12 [INFO] [TRAIN] epoch=318, iter=59000/160000, loss=0.4787, lr=0.033083, batch_cost=0.3336, reader_cost=0.1865 | ETA 09:21:37 2020-11-01 02:41:20 [INFO] [TRAIN] epoch=319, iter=59200/160000, loss=0.4799, lr=0.033024, batch_cost=0.3424, reader_cost=0.2029 | ETA 09:35:18 2020-11-01 02:42:28 [INFO] [TRAIN] epoch=320, iter=59400/160000, loss=0.4801, lr=0.032965, batch_cost=0.3385, reader_cost=0.1964 | ETA 09:27:28 2020-11-01 02:43:34 [INFO] [TRAIN] epoch=321, iter=59600/160000, loss=0.4800, lr=0.032906, batch_cost=0.3301, reader_cost=0.1853 | ETA 09:12:17 2020-11-01 02:44:42 [INFO] [TRAIN] epoch=322, iter=59800/160000, loss=0.4923, lr=0.032847, batch_cost=0.3385, reader_cost=0.1611 | ETA 09:25:18 2020-11-01 02:45:49 [INFO] [TRAIN] epoch=323, iter=60000/160000, loss=0.4770, lr=0.032789, batch_cost=0.3345, reader_cost=0.1901 | ETA 09:17:29 2020-11-01 02:46:56 [INFO] [TRAIN] epoch=324, iter=60200/160000, loss=0.4870, lr=0.032730, batch_cost=0.3381, reader_cost=0.1938 | ETA 09:22:18 2020-11-01 02:48:05 [INFO] [TRAIN] epoch=325, iter=60400/160000, loss=0.5087, lr=0.032671, batch_cost=0.3428, reader_cost=0.1997 | ETA 09:28:59 2020-11-01 02:49:11 [INFO] [TRAIN] epoch=326, iter=60600/160000, loss=0.5107, lr=0.032612, batch_cost=0.3284, reader_cost=0.1881 | ETA 09:04:05 2020-11-01 02:50:18 [INFO] [TRAIN] epoch=327, iter=60800/160000, loss=0.5031, lr=0.032553, batch_cost=0.3375, reader_cost=0.1832 | ETA 09:18:04 2020-11-01 02:51:27 [INFO] [TRAIN] epoch=328, iter=61000/160000, loss=0.4766, lr=0.032494, batch_cost=0.3446, reader_cost=0.2058 | ETA 09:28:35 2020-11-01 02:52:36 [INFO] [TRAIN] epoch=330, iter=61200/160000, loss=0.4729, lr=0.032435, batch_cost=0.3461, reader_cost=0.1972 | ETA 09:29:51 2020-11-01 02:53:44 [INFO] [TRAIN] epoch=331, iter=61400/160000, loss=0.4980, lr=0.032376, batch_cost=0.3383, reader_cost=0.1969 | ETA 09:15:56 2020-11-01 02:54:50 [INFO] [TRAIN] epoch=332, iter=61600/160000, loss=0.4944, lr=0.032318, batch_cost=0.3306, reader_cost=0.1886 | ETA 09:02:12 2020-11-01 02:55:56 [INFO] [TRAIN] epoch=333, iter=61800/160000, loss=0.4847, lr=0.032259, batch_cost=0.3301, reader_cost=0.1726 | ETA 09:00:18 2020-11-01 02:57:04 [INFO] [TRAIN] epoch=334, iter=62000/160000, loss=0.4866, lr=0.032200, batch_cost=0.3406, reader_cost=0.1896 | ETA 09:16:18 2020-11-01 02:58:13 [INFO] [TRAIN] epoch=335, iter=62200/160000, loss=0.4659, lr=0.032141, batch_cost=0.3422, reader_cost=0.1996 | ETA 09:17:44 2020-11-01 02:59:20 [INFO] [TRAIN] epoch=336, iter=62400/160000, loss=0.4699, lr=0.032082, batch_cost=0.3394, reader_cost=0.1957 | ETA 09:12:03 2020-11-01 03:00:27 [INFO] [TRAIN] epoch=337, iter=62600/160000, loss=0.4758, lr=0.032023, batch_cost=0.3327, reader_cost=0.1881 | ETA 09:00:04 2020-11-01 03:01:35 [INFO] [TRAIN] epoch=338, iter=62800/160000, loss=0.4846, lr=0.031964, batch_cost=0.3380, reader_cost=0.1891 | ETA 09:07:34 2020-11-01 03:02:43 [INFO] [TRAIN] epoch=339, iter=63000/160000, loss=0.4850, lr=0.031905, batch_cost=0.3412, reader_cost=0.1997 | ETA 09:11:34 2020-11-01 03:03:51 [INFO] [TRAIN] epoch=340, iter=63200/160000, loss=0.4821, lr=0.031846, batch_cost=0.3394, reader_cost=0.1976 | ETA 09:07:33 2020-11-01 03:05:00 [INFO] [TRAIN] epoch=341, iter=63400/160000, loss=0.4937, lr=0.031787, batch_cost=0.3450, reader_cost=0.1894 | ETA 09:15:30 2020-11-01 03:06:08 [INFO] [TRAIN] epoch=342, iter=63600/160000, loss=0.4972, lr=0.031728, batch_cost=0.3414, reader_cost=0.1962 | ETA 09:08:31 2020-11-01 03:07:17 [INFO] [TRAIN] epoch=344, iter=63800/160000, loss=0.4810, lr=0.031669, batch_cost=0.3440, reader_cost=0.2021 | ETA 09:11:34 2020-11-01 03:08:23 [INFO] [TRAIN] epoch=345, iter=64000/160000, loss=0.4803, lr=0.031609, batch_cost=0.3325, reader_cost=0.1877 | ETA 08:52:00 2020-11-01 03:08:24 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-01 03:11:25 [INFO] [EVAL] #Images=500 mIoU=0.6537 Acc=0.9402 Kappa=0.9222 2020-11-01 03:11:25 [INFO] [EVAL] Category IoU: [0.9671 0.7527 0.8917 0.4555 0.4523 0.4862 0.4754 0.627 0.9036 0.6012 0.9333 0.6978 0.4369 0.9095 0.5844 0.6716 0.5278 0.4024 0.6433] 2020-11-01 03:11:25 [INFO] [EVAL] Category Acc: [0.9791 0.8931 0.9337 0.8323 0.6896 0.774 0.7942 0.8147 0.9373 0.7971 0.9588 0.7875 0.6982 0.9372 0.8605 0.8227 0.929 0.6441 0.7481] 2020-11-01 03:11:25 [INFO] [EVAL] The model with the best validation mIoU (0.6537) was saved at iter 64000. 2020-11-01 03:12:29 [INFO] [TRAIN] epoch=346, iter=64200/160000, loss=0.4738, lr=0.031550, batch_cost=0.3226, reader_cost=0.1739 | ETA 08:35:03 2020-11-01 03:13:36 [INFO] [TRAIN] epoch=347, iter=64400/160000, loss=0.4792, lr=0.031491, batch_cost=0.3320, reader_cost=0.1900 | ETA 08:48:58 2020-11-01 03:14:44 [INFO] [TRAIN] epoch=348, iter=64600/160000, loss=0.4735, lr=0.031432, batch_cost=0.3413, reader_cost=0.1787 | ETA 09:02:41 2020-11-01 03:15:52 [INFO] [TRAIN] epoch=349, iter=64800/160000, loss=0.4649, lr=0.031373, batch_cost=0.3408, reader_cost=0.1969 | ETA 09:00:39 2020-11-01 03:17:00 [INFO] [TRAIN] epoch=350, iter=65000/160000, loss=0.4768, lr=0.031314, batch_cost=0.3396, reader_cost=0.1899 | ETA 08:57:46 2020-11-01 03:18:08 [INFO] [TRAIN] epoch=351, iter=65200/160000, loss=0.4700, lr=0.031255, batch_cost=0.3389, reader_cost=0.1894 | ETA 08:55:24 2020-11-01 03:19:11 [INFO] [TRAIN] epoch=352, iter=65400/160000, loss=0.4820, lr=0.031196, batch_cost=0.3150, reader_cost=0.1685 | ETA 08:16:38 2020-11-01 03:20:12 [INFO] [TRAIN] epoch=353, iter=65600/160000, loss=0.4889, lr=0.031136, batch_cost=0.3078, reader_cost=0.1625 | ETA 08:04:18 2020-11-01 03:21:16 [INFO] [TRAIN] epoch=354, iter=65800/160000, loss=0.4828, lr=0.031077, batch_cost=0.3176, reader_cost=0.1681 | ETA 08:18:38 2020-11-01 03:22:19 [INFO] [TRAIN] epoch=355, iter=66000/160000, loss=0.5270, lr=0.031018, batch_cost=0.3160, reader_cost=0.1699 | ETA 08:15:01 2020-11-01 03:23:23 [INFO] [TRAIN] epoch=356, iter=66200/160000, loss=0.4879, lr=0.030959, batch_cost=0.3185, reader_cost=0.1771 | ETA 08:17:50 2020-11-01 03:24:25 [INFO] [TRAIN] epoch=357, iter=66400/160000, loss=0.4855, lr=0.030900, batch_cost=0.3126, reader_cost=0.1631 | ETA 08:07:38 2020-11-01 03:25:34 [INFO] [TRAIN] epoch=359, iter=66600/160000, loss=0.4802, lr=0.030840, batch_cost=0.3437, reader_cost=0.1933 | ETA 08:55:06 2020-11-01 03:26:41 [INFO] [TRAIN] epoch=360, iter=66800/160000, loss=0.4714, lr=0.030781, batch_cost=0.3360, reader_cost=0.1869 | ETA 08:41:53 2020-11-01 03:27:45 [INFO] [TRAIN] epoch=361, iter=67000/160000, loss=0.4735, lr=0.030722, batch_cost=0.3190, reader_cost=0.1677 | ETA 08:14:25 2020-11-01 03:28:52 [INFO] [TRAIN] epoch=362, iter=67200/160000, loss=0.4764, lr=0.030663, batch_cost=0.3375, reader_cost=0.1917 | ETA 08:41:55 2020-11-01 03:29:57 [INFO] [TRAIN] epoch=363, iter=67400/160000, loss=0.4944, lr=0.030603, batch_cost=0.3249, reader_cost=0.1851 | ETA 08:21:28 2020-11-01 03:31:04 [INFO] [TRAIN] epoch=364, iter=67600/160000, loss=0.4774, lr=0.030544, batch_cost=0.3305, reader_cost=0.1926 | ETA 08:29:02 2020-11-01 03:32:09 [INFO] [TRAIN] epoch=365, iter=67800/160000, loss=0.4805, lr=0.030485, batch_cost=0.3256, reader_cost=0.1845 | ETA 08:20:17 2020-11-01 03:33:16 [INFO] [TRAIN] epoch=366, iter=68000/160000, loss=0.4717, lr=0.030425, batch_cost=0.3370, reader_cost=0.1927 | ETA 08:36:46 2020-11-01 03:34:24 [INFO] [TRAIN] epoch=367, iter=68200/160000, loss=0.4829, lr=0.030366, batch_cost=0.3395, reader_cost=0.1946 | ETA 08:39:22 2020-11-01 03:35:30 [INFO] [TRAIN] epoch=368, iter=68400/160000, loss=0.4672, lr=0.030307, batch_cost=0.3307, reader_cost=0.1712 | ETA 08:24:55 2020-11-01 03:36:35 [INFO] [TRAIN] epoch=369, iter=68600/160000, loss=0.4736, lr=0.030247, batch_cost=0.3224, reader_cost=0.1709 | ETA 08:11:09 2020-11-01 03:37:40 [INFO] [TRAIN] epoch=370, iter=68800/160000, loss=0.4967, lr=0.030188, batch_cost=0.3288, reader_cost=0.1781 | ETA 08:19:46 2020-11-01 03:38:47 [INFO] [TRAIN] epoch=371, iter=69000/160000, loss=0.4707, lr=0.030129, batch_cost=0.3348, reader_cost=0.1931 | ETA 08:27:47 2020-11-01 03:39:55 [INFO] [TRAIN] epoch=373, iter=69200/160000, loss=0.4599, lr=0.030069, batch_cost=0.3399, reader_cost=0.1854 | ETA 08:34:19 2020-11-01 03:41:02 [INFO] [TRAIN] epoch=374, iter=69400/160000, loss=0.4807, lr=0.030010, batch_cost=0.3336, reader_cost=0.1768 | ETA 08:23:44 2020-11-01 03:42:07 [INFO] [TRAIN] epoch=375, iter=69600/160000, loss=0.4589, lr=0.029950, batch_cost=0.3260, reader_cost=0.1681 | ETA 08:11:07 2020-11-01 03:43:13 [INFO] [TRAIN] epoch=376, iter=69800/160000, loss=0.4827, lr=0.029891, batch_cost=0.3313, reader_cost=0.1867 | ETA 08:18:02 2020-11-01 03:44:19 [INFO] [TRAIN] epoch=377, iter=70000/160000, loss=0.4924, lr=0.029831, batch_cost=0.3254, reader_cost=0.1818 | ETA 08:08:08 2020-11-01 03:45:26 [INFO] [TRAIN] epoch=378, iter=70200/160000, loss=0.4682, lr=0.029772, batch_cost=0.3352, reader_cost=0.1967 | ETA 08:21:42 2020-11-01 03:46:32 [INFO] [TRAIN] epoch=379, iter=70400/160000, loss=0.4723, lr=0.029712, batch_cost=0.3317, reader_cost=0.1856 | ETA 08:15:21 2020-11-01 03:47:39 [INFO] [TRAIN] epoch=380, iter=70600/160000, loss=0.4765, lr=0.029653, batch_cost=0.3328, reader_cost=0.1860 | ETA 08:15:53 2020-11-01 03:48:44 [INFO] [TRAIN] epoch=381, iter=70800/160000, loss=0.4909, lr=0.029593, batch_cost=0.3257, reader_cost=0.1788 | ETA 08:04:09 2020-11-01 03:49:50 [INFO] [TRAIN] epoch=382, iter=71000/160000, loss=0.4807, lr=0.029534, batch_cost=0.3336, reader_cost=0.1865 | ETA 08:14:51 2020-11-01 03:50:55 [INFO] [TRAIN] epoch=383, iter=71200/160000, loss=0.4666, lr=0.029474, batch_cost=0.3207, reader_cost=0.1754 | ETA 07:54:34 2020-11-01 03:52:00 [INFO] [TRAIN] epoch=384, iter=71400/160000, loss=0.4751, lr=0.029415, batch_cost=0.3277, reader_cost=0.1864 | ETA 08:03:55 2020-11-01 03:53:05 [INFO] [TRAIN] epoch=385, iter=71600/160000, loss=0.4897, lr=0.029355, batch_cost=0.3240, reader_cost=0.1715 | ETA 07:57:19 2020-11-01 03:54:12 [INFO] [TRAIN] epoch=387, iter=71800/160000, loss=0.4932, lr=0.029296, batch_cost=0.3339, reader_cost=0.1733 | ETA 08:10:49 2020-11-01 03:55:17 [INFO] [TRAIN] epoch=388, iter=72000/160000, loss=0.4594, lr=0.029236, batch_cost=0.3281, reader_cost=0.1844 | ETA 08:01:11 2020-11-01 03:55:17 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-01 03:58:03 [INFO] [EVAL] #Images=500 mIoU=0.6359 Acc=0.9400 Kappa=0.9219 2020-11-01 03:58:03 [INFO] [EVAL] Category IoU: [0.9707 0.769 0.8918 0.4148 0.4752 0.4897 0.4841 0.6309 0.8988 0.5299 0.9307 0.7018 0.4442 0.9042 0.5339 0.5965 0.5152 0.262 0.6376] 2020-11-01 03:58:03 [INFO] [EVAL] Category Acc: [0.9829 0.885 0.9391 0.8353 0.6598 0.7688 0.7497 0.8412 0.926 0.8385 0.9586 0.8121 0.7007 0.9314 0.7639 0.836 0.857 0.6803 0.7313] 2020-11-01 03:58:03 [INFO] [EVAL] The model with the best validation mIoU (0.6537) was saved at iter 64000. 2020-11-01 03:59:10 [INFO] [TRAIN] epoch=389, iter=72200/160000, loss=0.4715, lr=0.029176, batch_cost=0.3305, reader_cost=0.1843 | ETA 08:03:38 2020-11-01 04:00:16 [INFO] [TRAIN] epoch=390, iter=72400/160000, loss=0.4706, lr=0.029117, batch_cost=0.3299, reader_cost=0.1814 | ETA 08:01:35 2020-11-01 04:01:22 [INFO] [TRAIN] epoch=391, iter=72600/160000, loss=0.4764, lr=0.029057, batch_cost=0.3336, reader_cost=0.1879 | ETA 08:05:54 2020-11-01 04:02:29 [INFO] [TRAIN] epoch=392, iter=72800/160000, loss=0.4744, lr=0.028998, batch_cost=0.3324, reader_cost=0.1810 | ETA 08:03:01 2020-11-01 04:03:34 [INFO] [TRAIN] epoch=393, iter=73000/160000, loss=0.4711, lr=0.028938, batch_cost=0.3253, reader_cost=0.1734 | ETA 07:51:37 2020-11-01 04:04:39 [INFO] [TRAIN] epoch=394, iter=73200/160000, loss=0.4664, lr=0.028878, batch_cost=0.3272, reader_cost=0.1819 | ETA 07:53:17 2020-11-01 04:05:44 [INFO] [TRAIN] epoch=395, iter=73400/160000, loss=0.4555, lr=0.028819, batch_cost=0.3229, reader_cost=0.1805 | ETA 07:46:06 2020-11-01 04:06:47 [INFO] [TRAIN] epoch=396, iter=73600/160000, loss=0.4577, lr=0.028759, batch_cost=0.3179, reader_cost=0.1718 | ETA 07:37:46 2020-11-01 04:07:51 [INFO] [TRAIN] epoch=397, iter=73800/160000, loss=0.4640, lr=0.028699, batch_cost=0.3192, reader_cost=0.1746 | ETA 07:38:34 2020-11-01 04:08:57 [INFO] [TRAIN] epoch=398, iter=74000/160000, loss=0.4657, lr=0.028639, batch_cost=0.3285, reader_cost=0.1796 | ETA 07:50:51 2020-11-01 04:10:03 [INFO] [TRAIN] epoch=399, iter=74200/160000, loss=0.4743, lr=0.028580, batch_cost=0.3284, reader_cost=0.1842 | ETA 07:49:39 2020-11-01 04:11:08 [INFO] [TRAIN] epoch=400, iter=74400/160000, loss=0.4704, lr=0.028520, batch_cost=0.3262, reader_cost=0.1783 | ETA 07:45:24 2020-11-01 04:12:15 [INFO] [TRAIN] epoch=402, iter=74600/160000, loss=0.4856, lr=0.028460, batch_cost=0.3337, reader_cost=0.1825 | ETA 07:54:59 2020-11-01 04:13:20 [INFO] [TRAIN] epoch=403, iter=74800/160000, loss=0.4628, lr=0.028400, batch_cost=0.3290, reader_cost=0.1810 | ETA 07:47:06 2020-11-01 04:14:26 [INFO] [TRAIN] epoch=404, iter=75000/160000, loss=0.4840, lr=0.028341, batch_cost=0.3273, reader_cost=0.1814 | ETA 07:43:42 2020-11-01 04:15:32 [INFO] [TRAIN] epoch=405, iter=75200/160000, loss=0.4786, lr=0.028281, batch_cost=0.3285, reader_cost=0.1728 | ETA 07:44:20 2020-11-01 04:16:39 [INFO] [TRAIN] epoch=406, iter=75400/160000, loss=0.4664, lr=0.028221, batch_cost=0.3374, reader_cost=0.1903 | ETA 07:55:46 2020-11-01 04:17:44 [INFO] [TRAIN] epoch=407, iter=75600/160000, loss=0.4628, lr=0.028161, batch_cost=0.3257, reader_cost=0.1803 | ETA 07:38:11 2020-11-01 04:18:50 [INFO] [TRAIN] epoch=408, iter=75800/160000, loss=0.4792, lr=0.028101, batch_cost=0.3283, reader_cost=0.1850 | ETA 07:40:40 2020-11-01 04:19:54 [INFO] [TRAIN] epoch=409, iter=76000/160000, loss=0.4749, lr=0.028041, batch_cost=0.3220, reader_cost=0.1749 | ETA 07:30:47 2020-11-01 04:20:59 [INFO] [TRAIN] epoch=410, iter=76200/160000, loss=0.4668, lr=0.027982, batch_cost=0.3254, reader_cost=0.1666 | ETA 07:34:26 2020-11-01 04:22:04 [INFO] [TRAIN] epoch=411, iter=76400/160000, loss=0.4648, lr=0.027922, batch_cost=0.3251, reader_cost=0.1726 | ETA 07:32:55 2020-11-01 04:23:09 [INFO] [TRAIN] epoch=412, iter=76600/160000, loss=0.4769, lr=0.027862, batch_cost=0.3219, reader_cost=0.1748 | ETA 07:27:27 2020-11-01 04:24:14 [INFO] [TRAIN] epoch=413, iter=76800/160000, loss=0.4701, lr=0.027802, batch_cost=0.3249, reader_cost=0.1735 | ETA 07:30:35 2020-11-01 04:25:19 [INFO] [TRAIN] epoch=414, iter=77000/160000, loss=0.4618, lr=0.027742, batch_cost=0.3283, reader_cost=0.1840 | ETA 07:34:11 2020-11-01 04:26:27 [INFO] [TRAIN] epoch=416, iter=77200/160000, loss=0.4730, lr=0.027682, batch_cost=0.3393, reader_cost=0.1810 | ETA 07:48:16 2020-11-01 04:27:33 [INFO] [TRAIN] epoch=417, iter=77400/160000, loss=0.4664, lr=0.027622, batch_cost=0.3300, reader_cost=0.1750 | ETA 07:34:18 2020-11-01 04:28:38 [INFO] [TRAIN] epoch=418, iter=77600/160000, loss=0.4552, lr=0.027562, batch_cost=0.3212, reader_cost=0.1723 | ETA 07:21:04 2020-11-01 04:29:43 [INFO] [TRAIN] epoch=419, iter=77800/160000, loss=0.4672, lr=0.027502, batch_cost=0.3287, reader_cost=0.1828 | ETA 07:30:22 2020-11-01 04:30:49 [INFO] [TRAIN] epoch=420, iter=78000/160000, loss=0.4728, lr=0.027442, batch_cost=0.3295, reader_cost=0.1776 | ETA 07:30:22 2020-11-01 04:31:54 [INFO] [TRAIN] epoch=421, iter=78200/160000, loss=0.4829, lr=0.027382, batch_cost=0.3255, reader_cost=0.1814 | ETA 07:23:49 2020-11-01 04:33:00 [INFO] [TRAIN] epoch=422, iter=78400/160000, loss=0.4794, lr=0.027322, batch_cost=0.3284, reader_cost=0.1785 | ETA 07:26:38 2020-11-01 04:34:04 [INFO] [TRAIN] epoch=423, iter=78600/160000, loss=0.4522, lr=0.027262, batch_cost=0.3194, reader_cost=0.1672 | ETA 07:13:16 2020-11-01 04:35:11 [INFO] [TRAIN] epoch=424, iter=78800/160000, loss=0.4779, lr=0.027202, batch_cost=0.3341, reader_cost=0.1930 | ETA 07:32:06 2020-11-01 04:36:16 [INFO] [TRAIN] epoch=425, iter=79000/160000, loss=0.4781, lr=0.027142, batch_cost=0.3249, reader_cost=0.1752 | ETA 07:18:33 2020-11-01 04:37:21 [INFO] [TRAIN] epoch=426, iter=79200/160000, loss=0.4650, lr=0.027082, batch_cost=0.3283, reader_cost=0.1710 | ETA 07:22:10 2020-11-01 04:38:28 [INFO] [TRAIN] epoch=427, iter=79400/160000, loss=0.4525, lr=0.027021, batch_cost=0.3336, reader_cost=0.1838 | ETA 07:28:06 2020-11-01 04:39:35 [INFO] [TRAIN] epoch=428, iter=79600/160000, loss=0.4715, lr=0.026961, batch_cost=0.3327, reader_cost=0.1857 | ETA 07:25:48 2020-11-01 04:40:42 [INFO] [TRAIN] epoch=430, iter=79800/160000, loss=0.4918, lr=0.026901, batch_cost=0.3351, reader_cost=0.1818 | ETA 07:27:57 2020-11-01 04:41:46 [INFO] [TRAIN] epoch=431, iter=80000/160000, loss=0.4696, lr=0.026841, batch_cost=0.3203, reader_cost=0.1736 | ETA 07:07:00 2020-11-01 04:41:46 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-01 04:44:40 [INFO] [EVAL] #Images=500 mIoU=0.6396 Acc=0.9377 Kappa=0.9190 2020-11-01 04:44:40 [INFO] [EVAL] Category IoU: [0.9635 0.7312 0.8938 0.3967 0.4901 0.5045 0.5075 0.626 0.8999 0.5935 0.9346 0.7078 0.4297 0.9044 0.4337 0.6334 0.5374 0.3165 0.6476] 2020-11-01 04:44:40 [INFO] [EVAL] Category Acc: [0.9747 0.8919 0.9488 0.7351 0.6401 0.6837 0.7529 0.848 0.9312 0.7502 0.9575 0.8024 0.7075 0.9365 0.8771 0.8261 0.8721 0.7315 0.7532] 2020-11-01 04:44:40 [INFO] [EVAL] The model with the best validation mIoU (0.6537) was saved at iter 64000. 2020-11-01 04:45:44 [INFO] [TRAIN] epoch=432, iter=80200/160000, loss=0.4562, lr=0.026781, batch_cost=0.3191, reader_cost=0.1748 | ETA 07:04:22 2020-11-01 04:46:49 [INFO] [TRAIN] epoch=433, iter=80400/160000, loss=0.4597, lr=0.026721, batch_cost=0.3252, reader_cost=0.1831 | ETA 07:11:25 2020-11-01 04:47:53 [INFO] [TRAIN] epoch=434, iter=80600/160000, loss=0.4588, lr=0.026660, batch_cost=0.3229, reader_cost=0.1726 | ETA 07:07:21 2020-11-01 04:49:00 [INFO] [TRAIN] epoch=435, iter=80800/160000, loss=0.4809, lr=0.026600, batch_cost=0.3314, reader_cost=0.1888 | ETA 07:17:28 2020-11-01 04:50:06 [INFO] [TRAIN] epoch=436, iter=81000/160000, loss=0.4574, lr=0.026540, batch_cost=0.3336, reader_cost=0.1902 | ETA 07:19:13 2020-11-01 04:51:12 [INFO] [TRAIN] epoch=437, iter=81200/160000, loss=0.4843, lr=0.026480, batch_cost=0.3279, reader_cost=0.1813 | ETA 07:10:38 2020-11-01 04:52:16 [INFO] [TRAIN] epoch=438, iter=81400/160000, loss=0.4826, lr=0.026420, batch_cost=0.3185, reader_cost=0.1614 | ETA 06:57:13 2020-11-01 04:53:21 [INFO] [TRAIN] epoch=439, iter=81600/160000, loss=0.4941, lr=0.026359, batch_cost=0.3284, reader_cost=0.1848 | ETA 07:09:09 2020-11-01 04:54:28 [INFO] [TRAIN] epoch=440, iter=81800/160000, loss=0.4592, lr=0.026299, batch_cost=0.3341, reader_cost=0.1788 | ETA 07:15:22 2020-11-01 04:55:33 [INFO] [TRAIN] epoch=441, iter=82000/160000, loss=0.4664, lr=0.026239, batch_cost=0.3239, reader_cost=0.1679 | ETA 07:01:07 2020-11-01 04:56:41 [INFO] [TRAIN] epoch=442, iter=82200/160000, loss=0.4947, lr=0.026178, batch_cost=0.3380, reader_cost=0.1968 | ETA 07:18:16 2020-11-01 04:57:45 [INFO] [TRAIN] epoch=444, iter=82400/160000, loss=0.4805, lr=0.026118, batch_cost=0.3221, reader_cost=0.1167 | ETA 06:56:31 2020-11-01 04:58:46 [INFO] [TRAIN] epoch=445, iter=82600/160000, loss=0.4821, lr=0.026058, batch_cost=0.3057, reader_cost=0.1527 | ETA 06:34:18 2020-11-01 04:59:49 [INFO] [TRAIN] epoch=446, iter=82800/160000, loss=0.4570, lr=0.025997, batch_cost=0.3144, reader_cost=0.1603 | ETA 06:44:30 2020-11-01 05:00:51 [INFO] [TRAIN] epoch=447, iter=83000/160000, loss=0.4565, lr=0.025937, batch_cost=0.3091, reader_cost=0.1699 | ETA 06:36:39 2020-11-01 05:01:54 [INFO] [TRAIN] epoch=448, iter=83200/160000, loss=0.4690, lr=0.025876, batch_cost=0.3164, reader_cost=0.1729 | ETA 06:44:55 2020-11-01 05:02:55 [INFO] [TRAIN] epoch=449, iter=83400/160000, loss=0.4575, lr=0.025816, batch_cost=0.3038, reader_cost=0.1662 | ETA 06:27:47 2020-11-01 05:03:58 [INFO] [TRAIN] epoch=450, iter=83600/160000, loss=0.4761, lr=0.025756, batch_cost=0.3183, reader_cost=0.1707 | ETA 06:45:21 2020-11-01 05:05:02 [INFO] [TRAIN] epoch=451, iter=83800/160000, loss=0.4678, lr=0.025695, batch_cost=0.3162, reader_cost=0.1684 | ETA 06:41:36 2020-11-01 05:06:05 [INFO] [TRAIN] epoch=452, iter=84000/160000, loss=0.4575, lr=0.025635, batch_cost=0.3177, reader_cost=0.1672 | ETA 06:42:27 2020-11-01 05:07:09 [INFO] [TRAIN] epoch=453, iter=84200/160000, loss=0.4637, lr=0.025574, batch_cost=0.3174, reader_cost=0.1706 | ETA 06:41:01 2020-11-01 05:08:13 [INFO] [TRAIN] epoch=454, iter=84400/160000, loss=0.4569, lr=0.025514, batch_cost=0.3213, reader_cost=0.1721 | ETA 06:44:48 2020-11-01 05:09:18 [INFO] [TRAIN] epoch=455, iter=84600/160000, loss=0.4664, lr=0.025453, batch_cost=0.3260, reader_cost=0.1637 | ETA 06:49:39 2020-11-01 05:10:24 [INFO] [TRAIN] epoch=456, iter=84800/160000, loss=0.4673, lr=0.025393, batch_cost=0.3288, reader_cost=0.1870 | ETA 06:52:05 2020-11-01 05:11:29 [INFO] [TRAIN] epoch=457, iter=85000/160000, loss=0.4633, lr=0.025332, batch_cost=0.3258, reader_cost=0.1787 | ETA 06:47:15 2020-11-01 05:12:35 [INFO] [TRAIN] epoch=459, iter=85200/160000, loss=0.4761, lr=0.025272, batch_cost=0.3303, reader_cost=0.1653 | ETA 06:51:49 2020-11-01 05:13:39 [INFO] [TRAIN] epoch=460, iter=85400/160000, loss=0.4604, lr=0.025211, batch_cost=0.3184, reader_cost=0.1688 | ETA 06:35:49 2020-11-01 05:14:44 [INFO] [TRAIN] epoch=461, iter=85600/160000, loss=0.4632, lr=0.025150, batch_cost=0.3232, reader_cost=0.1796 | ETA 06:40:42 2020-11-01 05:15:49 [INFO] [TRAIN] epoch=462, iter=85800/160000, loss=0.4644, lr=0.025090, batch_cost=0.3289, reader_cost=0.1892 | ETA 06:46:45 2020-11-01 05:16:54 [INFO] [TRAIN] epoch=463, iter=86000/160000, loss=0.4520, lr=0.025029, batch_cost=0.3253, reader_cost=0.1797 | ETA 06:41:09 2020-11-01 05:18:00 [INFO] [TRAIN] epoch=464, iter=86200/160000, loss=0.4605, lr=0.024968, batch_cost=0.3276, reader_cost=0.1733 | ETA 06:43:00 2020-11-01 05:19:05 [INFO] [TRAIN] epoch=465, iter=86400/160000, loss=0.4590, lr=0.024908, batch_cost=0.3263, reader_cost=0.1860 | ETA 06:40:17 2020-11-01 05:20:10 [INFO] [TRAIN] epoch=466, iter=86600/160000, loss=0.4676, lr=0.024847, batch_cost=0.3260, reader_cost=0.1828 | ETA 06:38:51 2020-11-01 05:21:14 [INFO] [TRAIN] epoch=467, iter=86800/160000, loss=0.4603, lr=0.024786, batch_cost=0.3189, reader_cost=0.1759 | ETA 06:29:03 2020-11-01 05:22:19 [INFO] [TRAIN] epoch=468, iter=87000/160000, loss=0.4607, lr=0.024726, batch_cost=0.3232, reader_cost=0.1769 | ETA 06:33:12 2020-11-01 05:23:23 [INFO] [TRAIN] epoch=469, iter=87200/160000, loss=0.4672, lr=0.024665, batch_cost=0.3202, reader_cost=0.1397 | ETA 06:28:32 2020-11-01 05:24:27 [INFO] [TRAIN] epoch=470, iter=87400/160000, loss=0.4680, lr=0.024604, batch_cost=0.3191, reader_cost=0.1768 | ETA 06:26:05 2020-11-01 05:25:29 [INFO] [TRAIN] epoch=471, iter=87600/160000, loss=0.4590, lr=0.024543, batch_cost=0.3127, reader_cost=0.1399 | ETA 06:17:23 2020-11-01 05:26:34 [INFO] [TRAIN] epoch=473, iter=87800/160000, loss=0.4468, lr=0.024483, batch_cost=0.3218, reader_cost=0.1621 | ETA 06:27:12 2020-11-01 05:27:40 [INFO] [TRAIN] epoch=474, iter=88000/160000, loss=0.4556, lr=0.024422, batch_cost=0.3299, reader_cost=0.1831 | ETA 06:35:56 2020-11-01 05:27:40 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-01 05:30:31 [INFO] [EVAL] #Images=500 mIoU=0.6627 Acc=0.9426 Kappa=0.9253 2020-11-01 05:30:31 [INFO] [EVAL] Category IoU: [0.9713 0.7805 0.8944 0.4144 0.4803 0.5149 0.5039 0.6434 0.9052 0.6071 0.9294 0.7143 0.4627 0.9109 0.5579 0.6423 0.5801 0.4314 0.6468] 2020-11-01 05:30:31 [INFO] [EVAL] Category Acc: [0.9834 0.9031 0.9345 0.816 0.6994 0.7423 0.7735 0.8313 0.944 0.7324 0.9588 0.813 0.6809 0.9409 0.8493 0.8187 0.8364 0.6372 0.7437] 2020-11-01 05:30:32 [INFO] [EVAL] The model with the best validation mIoU (0.6627) was saved at iter 88000. 2020-11-01 05:31:34 [INFO] [TRAIN] epoch=475, iter=88200/160000, loss=0.4573, lr=0.024361, batch_cost=0.3105, reader_cost=0.1696 | ETA 06:11:31 2020-11-01 05:32:39 [INFO] [TRAIN] epoch=476, iter=88400/160000, loss=0.4587, lr=0.024300, batch_cost=0.3247, reader_cost=0.1835 | ETA 06:27:28 2020-11-01 05:33:42 [INFO] [TRAIN] epoch=477, iter=88600/160000, loss=0.4660, lr=0.024239, batch_cost=0.3188, reader_cost=0.1723 | ETA 06:19:24 2020-11-01 05:34:47 [INFO] [TRAIN] epoch=478, iter=88800/160000, loss=0.4745, lr=0.024179, batch_cost=0.3249, reader_cost=0.1848 | ETA 06:25:33 2020-11-01 05:35:50 [INFO] [TRAIN] epoch=479, iter=89000/160000, loss=0.4599, lr=0.024118, batch_cost=0.3153, reader_cost=0.1371 | ETA 06:13:06 2020-11-01 05:36:53 [INFO] [TRAIN] epoch=480, iter=89200/160000, loss=0.4626, lr=0.024057, batch_cost=0.3111, reader_cost=0.1506 | ETA 06:07:05 2020-11-01 05:37:57 [INFO] [TRAIN] epoch=481, iter=89400/160000, loss=0.4543, lr=0.023996, batch_cost=0.3214, reader_cost=0.1714 | ETA 06:18:13 2020-11-01 05:38:59 [INFO] [TRAIN] epoch=482, iter=89600/160000, loss=0.4554, lr=0.023935, batch_cost=0.3114, reader_cost=0.1545 | ETA 06:05:22 2020-11-01 05:40:02 [INFO] [TRAIN] epoch=483, iter=89800/160000, loss=0.4541, lr=0.023874, batch_cost=0.3134, reader_cost=0.1576 | ETA 06:06:38 2020-11-01 05:41:06 [INFO] [TRAIN] epoch=484, iter=90000/160000, loss=0.4575, lr=0.023813, batch_cost=0.3215, reader_cost=0.1697 | ETA 06:15:02 2020-11-01 05:42:08 [INFO] [TRAIN] epoch=485, iter=90200/160000, loss=0.4700, lr=0.023752, batch_cost=0.3100, reader_cost=0.1687 | ETA 06:00:34 2020-11-01 05:43:11 [INFO] [TRAIN] epoch=487, iter=90400/160000, loss=0.4589, lr=0.023691, batch_cost=0.3150, reader_cost=0.1391 | ETA 06:05:24 2020-11-01 05:44:16 [INFO] [TRAIN] epoch=488, iter=90600/160000, loss=0.4627, lr=0.023630, batch_cost=0.3241, reader_cost=0.1466 | ETA 06:14:51 2020-11-01 05:45:19 [INFO] [TRAIN] epoch=489, iter=90800/160000, loss=0.4499, lr=0.023569, batch_cost=0.3161, reader_cost=0.1571 | ETA 06:04:35 2020-11-01 05:46:23 [INFO] [TRAIN] epoch=490, iter=91000/160000, loss=0.4548, lr=0.023508, batch_cost=0.3170, reader_cost=0.1567 | ETA 06:04:31 2020-11-01 05:47:24 [INFO] [TRAIN] epoch=491, iter=91200/160000, loss=0.4536, lr=0.023447, batch_cost=0.3090, reader_cost=0.1294 | ETA 05:54:15 2020-11-01 05:48:28 [INFO] [TRAIN] epoch=492, iter=91400/160000, loss=0.4599, lr=0.023386, batch_cost=0.3167, reader_cost=0.0900 | ETA 06:02:05 2020-11-01 05:49:30 [INFO] [TRAIN] epoch=493, iter=91600/160000, loss=0.4539, lr=0.023325, batch_cost=0.3111, reader_cost=0.1225 | ETA 05:54:40 2020-11-01 05:50:32 [INFO] [TRAIN] epoch=494, iter=91800/160000, loss=0.4693, lr=0.023264, batch_cost=0.3109, reader_cost=0.1674 | ETA 05:53:22 2020-11-01 05:51:34 [INFO] [TRAIN] epoch=495, iter=92000/160000, loss=0.4681, lr=0.023202, batch_cost=0.3077, reader_cost=0.1617 | ETA 05:48:40 2020-11-01 05:52:38 [INFO] [TRAIN] epoch=496, iter=92200/160000, loss=0.4542, lr=0.023141, batch_cost=0.3197, reader_cost=0.1658 | ETA 06:01:13 2020-11-01 05:53:41 [INFO] [TRAIN] epoch=497, iter=92400/160000, loss=0.4556, lr=0.023080, batch_cost=0.3185, reader_cost=0.1591 | ETA 05:58:48 2020-11-01 05:54:46 [INFO] [TRAIN] epoch=498, iter=92600/160000, loss=0.4620, lr=0.023019, batch_cost=0.3230, reader_cost=0.1842 | ETA 06:02:49 2020-11-01 05:55:50 [INFO] [TRAIN] epoch=499, iter=92800/160000, loss=0.4592, lr=0.022958, batch_cost=0.3220, reader_cost=0.1721 | ETA 06:00:41 2020-11-01 05:56:53 [INFO] [TRAIN] epoch=500, iter=93000/160000, loss=0.4570, lr=0.022896, batch_cost=0.3135, reader_cost=0.1495 | ETA 05:50:01 2020-11-01 05:57:59 [INFO] [TRAIN] epoch=502, iter=93200/160000, loss=0.4518, lr=0.022835, batch_cost=0.3289, reader_cost=0.1836 | ETA 06:06:07 2020-11-01 05:59:00 [INFO] [TRAIN] epoch=503, iter=93400/160000, loss=0.4614, lr=0.022774, batch_cost=0.3081, reader_cost=0.1557 | ETA 05:41:56 2020-11-01 06:00:05 [INFO] [TRAIN] epoch=504, iter=93600/160000, loss=0.4508, lr=0.022713, batch_cost=0.3224, reader_cost=0.1688 | ETA 05:56:46 2020-11-01 06:01:07 [INFO] [TRAIN] epoch=505, iter=93800/160000, loss=0.4573, lr=0.022651, batch_cost=0.3092, reader_cost=0.1280 | ETA 05:41:09 2020-11-01 06:02:10 [INFO] [TRAIN] epoch=506, iter=94000/160000, loss=0.4547, lr=0.022590, batch_cost=0.3184, reader_cost=0.1400 | ETA 05:50:11 2020-11-01 06:03:15 [INFO] [TRAIN] epoch=507, iter=94200/160000, loss=0.4575, lr=0.022529, batch_cost=0.3228, reader_cost=0.1799 | ETA 05:53:58 2020-11-01 06:04:20 [INFO] [TRAIN] epoch=508, iter=94400/160000, loss=0.4643, lr=0.022467, batch_cost=0.3258, reader_cost=0.1655 | ETA 05:56:09 2020-11-01 06:05:24 [INFO] [TRAIN] epoch=509, iter=94600/160000, loss=0.4497, lr=0.022406, batch_cost=0.3169, reader_cost=0.1707 | ETA 05:45:28 2020-11-01 06:06:25 [INFO] [TRAIN] epoch=510, iter=94800/160000, loss=0.4745, lr=0.022344, batch_cost=0.3080, reader_cost=0.1398 | ETA 05:34:42 2020-11-01 06:07:28 [INFO] [TRAIN] epoch=511, iter=95000/160000, loss=0.4666, lr=0.022283, batch_cost=0.3164, reader_cost=0.1326 | ETA 05:42:45 2020-11-01 06:08:32 [INFO] [TRAIN] epoch=512, iter=95200/160000, loss=0.4577, lr=0.022222, batch_cost=0.3154, reader_cost=0.1745 | ETA 05:40:38 2020-11-01 06:09:36 [INFO] [TRAIN] epoch=513, iter=95400/160000, loss=0.4567, lr=0.022160, batch_cost=0.3210, reader_cost=0.1796 | ETA 05:45:34 2020-11-01 06:10:38 [INFO] [TRAIN] epoch=514, iter=95600/160000, loss=0.4559, lr=0.022099, batch_cost=0.3111, reader_cost=0.1581 | ETA 05:33:57 2020-11-01 06:11:43 [INFO] [TRAIN] epoch=516, iter=95800/160000, loss=0.4631, lr=0.022037, batch_cost=0.3277, reader_cost=0.1881 | ETA 05:50:38 2020-11-01 06:12:48 [INFO] [TRAIN] epoch=517, iter=96000/160000, loss=0.4532, lr=0.021976, batch_cost=0.3220, reader_cost=0.1788 | ETA 05:43:29 2020-11-01 06:12:48 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-01 06:15:38 [INFO] [EVAL] #Images=500 mIoU=0.6187 Acc=0.9334 Kappa=0.9134 2020-11-01 06:15:38 [INFO] [EVAL] Category IoU: [0.96 0.7107 0.8866 0.3121 0.4575 0.4722 0.4823 0.652 0.8933 0.561 0.9303 0.6785 0.4082 0.9076 0.5702 0.5918 0.3332 0.3129 0.6342] 2020-11-01 06:15:38 [INFO] [EVAL] Category Acc: [0.9756 0.8759 0.9403 0.6982 0.5857 0.6784 0.7647 0.8495 0.9228 0.8809 0.9489 0.7727 0.7096 0.9405 0.7965 0.7526 0.9565 0.7916 0.7384] 2020-11-01 06:15:38 [INFO] [EVAL] The model with the best validation mIoU (0.6627) was saved at iter 88000. 2020-11-01 06:16:40 [INFO] [TRAIN] epoch=518, iter=96200/160000, loss=0.4529, lr=0.021914, batch_cost=0.3088, reader_cost=0.1573 | ETA 05:28:21 2020-11-01 06:17:42 [INFO] [TRAIN] epoch=519, iter=96400/160000, loss=0.4495, lr=0.021853, batch_cost=0.3106, reader_cost=0.1718 | ETA 05:29:16 2020-11-01 06:18:46 [INFO] [TRAIN] epoch=520, iter=96600/160000, loss=0.4569, lr=0.021791, batch_cost=0.3183, reader_cost=0.1701 | ETA 05:36:22 2020-11-01 06:19:50 [INFO] [TRAIN] epoch=521, iter=96800/160000, loss=0.4771, lr=0.021729, batch_cost=0.3184, reader_cost=0.1730 | ETA 05:35:23 2020-11-01 06:20:54 [INFO] [TRAIN] epoch=522, iter=97000/160000, loss=0.4435, lr=0.021668, batch_cost=0.3226, reader_cost=0.1829 | ETA 05:38:41 2020-11-01 06:21:56 [INFO] [TRAIN] epoch=523, iter=97200/160000, loss=0.4471, lr=0.021606, batch_cost=0.3096, reader_cost=0.1689 | ETA 05:24:04 2020-11-01 06:22:59 [INFO] [TRAIN] epoch=524, iter=97400/160000, loss=0.4691, lr=0.021544, batch_cost=0.3136, reader_cost=0.1720 | ETA 05:27:08 2020-11-01 06:24:02 [INFO] [TRAIN] epoch=525, iter=97600/160000, loss=0.4621, lr=0.021483, batch_cost=0.3174, reader_cost=0.1743 | ETA 05:30:05 2020-11-01 06:25:06 [INFO] [TRAIN] epoch=526, iter=97800/160000, loss=0.4388, lr=0.021421, batch_cost=0.3196, reader_cost=0.1768 | ETA 05:31:21 2020-11-01 06:26:10 [INFO] [TRAIN] epoch=527, iter=98000/160000, loss=0.4637, lr=0.021359, batch_cost=0.3176, reader_cost=0.1603 | ETA 05:28:09 2020-11-01 06:27:13 [INFO] [TRAIN] epoch=528, iter=98200/160000, loss=0.4538, lr=0.021298, batch_cost=0.3182, reader_cost=0.1729 | ETA 05:27:46 2020-11-01 06:28:17 [INFO] [TRAIN] epoch=530, iter=98400/160000, loss=0.4575, lr=0.021236, batch_cost=0.3172, reader_cost=0.1526 | ETA 05:25:40 2020-11-01 06:29:21 [INFO] [TRAIN] epoch=531, iter=98600/160000, loss=0.4388, lr=0.021174, batch_cost=0.3197, reader_cost=0.1763 | ETA 05:27:09 2020-11-01 06:30:25 [INFO] [TRAIN] epoch=532, iter=98800/160000, loss=0.4406, lr=0.021112, batch_cost=0.3228, reader_cost=0.1797 | ETA 05:29:17 2020-11-01 06:31:29 [INFO] [TRAIN] epoch=533, iter=99000/160000, loss=0.4429, lr=0.021051, batch_cost=0.3161, reader_cost=0.1650 | ETA 05:21:19 2020-11-01 06:32:34 [INFO] [TRAIN] epoch=534, iter=99200/160000, loss=0.4414, lr=0.020989, batch_cost=0.3247, reader_cost=0.1824 | ETA 05:29:02 2020-11-01 06:33:43 [INFO] [TRAIN] epoch=535, iter=99400/160000, loss=0.4586, lr=0.020927, batch_cost=0.3487, reader_cost=0.2064 | ETA 05:52:11 2020-11-01 06:34:47 [INFO] [TRAIN] epoch=536, iter=99600/160000, loss=0.4618, lr=0.020865, batch_cost=0.3196, reader_cost=0.1636 | ETA 05:21:43 2020-11-01 06:35:50 [INFO] [TRAIN] epoch=537, iter=99800/160000, loss=0.4547, lr=0.020803, batch_cost=0.3160, reader_cost=0.1727 | ETA 05:17:05 2020-11-01 06:36:51 [INFO] [TRAIN] epoch=538, iter=100000/160000, loss=0.4494, lr=0.020741, batch_cost=0.3013, reader_cost=0.1399 | ETA 05:01:15 2020-11-01 06:37:51 [INFO] [TRAIN] epoch=539, iter=100200/160000, loss=0.4514, lr=0.020679, batch_cost=0.3028, reader_cost=0.1617 | ETA 05:01:44 2020-11-01 06:38:55 [INFO] [TRAIN] epoch=540, iter=100400/160000, loss=0.4539, lr=0.020617, batch_cost=0.3185, reader_cost=0.1778 | ETA 05:16:24 2020-11-01 06:39:59 [INFO] [TRAIN] epoch=541, iter=100600/160000, loss=0.4497, lr=0.020555, batch_cost=0.3187, reader_cost=0.1538 | ETA 05:15:31 2020-11-01 06:41:02 [INFO] [TRAIN] epoch=542, iter=100800/160000, loss=0.4426, lr=0.020493, batch_cost=0.3174, reader_cost=0.1689 | ETA 05:13:10 2020-11-01 06:42:06 [INFO] [TRAIN] epoch=544, iter=101000/160000, loss=0.4392, lr=0.020431, batch_cost=0.3197, reader_cost=0.1625 | ETA 05:14:19 2020-11-01 06:43:08 [INFO] [TRAIN] epoch=545, iter=101200/160000, loss=0.4560, lr=0.020369, batch_cost=0.3100, reader_cost=0.1569 | ETA 05:03:47 2020-11-01 06:44:11 [INFO] [TRAIN] epoch=546, iter=101400/160000, loss=0.4535, lr=0.020307, batch_cost=0.3135, reader_cost=0.1619 | ETA 05:06:12 2020-11-01 06:45:14 [INFO] [TRAIN] epoch=547, iter=101600/160000, loss=0.4465, lr=0.020245, batch_cost=0.3144, reader_cost=0.1710 | ETA 05:05:58 2020-11-01 06:46:17 [INFO] [TRAIN] epoch=548, iter=101800/160000, loss=0.4546, lr=0.020183, batch_cost=0.3189, reader_cost=0.1783 | ETA 05:09:18 2020-11-01 06:47:21 [INFO] [TRAIN] epoch=549, iter=102000/160000, loss=0.4627, lr=0.020121, batch_cost=0.3153, reader_cost=0.1694 | ETA 05:04:47 2020-11-01 06:48:24 [INFO] [TRAIN] epoch=550, iter=102200/160000, loss=0.4496, lr=0.020059, batch_cost=0.3169, reader_cost=0.1743 | ETA 05:05:16 2020-11-01 06:49:27 [INFO] [TRAIN] epoch=551, iter=102400/160000, loss=0.4464, lr=0.019997, batch_cost=0.3142, reader_cost=0.1745 | ETA 05:01:36 2020-11-01 06:50:30 [INFO] [TRAIN] epoch=552, iter=102600/160000, loss=0.4559, lr=0.019934, batch_cost=0.3158, reader_cost=0.1370 | ETA 05:02:04 2020-11-01 06:51:33 [INFO] [TRAIN] epoch=553, iter=102800/160000, loss=0.4433, lr=0.019872, batch_cost=0.3168, reader_cost=0.1723 | ETA 05:02:01 2020-11-01 06:52:37 [INFO] [TRAIN] epoch=554, iter=103000/160000, loss=0.4345, lr=0.019810, batch_cost=0.3207, reader_cost=0.1798 | ETA 05:04:39 2020-11-01 06:53:41 [INFO] [TRAIN] epoch=555, iter=103200/160000, loss=0.4424, lr=0.019748, batch_cost=0.3165, reader_cost=0.1603 | ETA 04:59:39 2020-11-01 06:54:44 [INFO] [TRAIN] epoch=556, iter=103400/160000, loss=0.4492, lr=0.019685, batch_cost=0.3173, reader_cost=0.1710 | ETA 04:59:17 2020-11-01 06:55:47 [INFO] [TRAIN] epoch=557, iter=103600/160000, loss=0.4396, lr=0.019623, batch_cost=0.3160, reader_cost=0.1760 | ETA 04:57:04 2020-11-01 06:56:53 [INFO] [TRAIN] epoch=559, iter=103800/160000, loss=0.4611, lr=0.019561, batch_cost=0.3275, reader_cost=0.1791 | ETA 05:06:47 2020-11-01 06:57:57 [INFO] [TRAIN] epoch=560, iter=104000/160000, loss=0.4408, lr=0.019499, batch_cost=0.3214, reader_cost=0.1822 | ETA 04:59:55 2020-11-01 06:57:57 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-01 07:00:52 [INFO] [EVAL] #Images=500 mIoU=0.6462 Acc=0.9418 Kappa=0.9242 2020-11-01 07:00:52 [INFO] [EVAL] Category IoU: [0.9687 0.7641 0.8921 0.3777 0.4757 0.5187 0.5135 0.6412 0.9058 0.601 0.9384 0.6953 0.3427 0.9191 0.6306 0.6665 0.3862 0.3806 0.6606] 2020-11-01 07:00:52 [INFO] [EVAL] Category Acc: [0.9822 0.879 0.9213 0.7519 0.7848 0.7546 0.8101 0.874 0.9465 0.838 0.9634 0.7763 0.7668 0.9551 0.7818 0.8132 0.9209 0.7657 0.8216] 2020-11-01 07:00:52 [INFO] [EVAL] The model with the best validation mIoU (0.6627) was saved at iter 88000. 2020-11-01 07:01:53 [INFO] [TRAIN] epoch=561, iter=104200/160000, loss=0.4414, lr=0.019436, batch_cost=0.3071, reader_cost=0.1644 | ETA 04:45:33 2020-11-01 07:02:55 [INFO] [TRAIN] epoch=562, iter=104400/160000, loss=0.4472, lr=0.019374, batch_cost=0.3111, reader_cost=0.1563 | ETA 04:48:14 2020-11-01 07:04:00 [INFO] [TRAIN] epoch=563, iter=104600/160000, loss=0.4353, lr=0.019311, batch_cost=0.3219, reader_cost=0.1761 | ETA 04:57:11 2020-11-01 07:05:04 [INFO] [TRAIN] epoch=564, iter=104800/160000, loss=0.4468, lr=0.019249, batch_cost=0.3232, reader_cost=0.1744 | ETA 04:57:18 2020-11-01 07:06:07 [INFO] [TRAIN] epoch=565, iter=105000/160000, loss=0.4558, lr=0.019186, batch_cost=0.3160, reader_cost=0.1698 | ETA 04:49:42 2020-11-01 07:07:11 [INFO] [TRAIN] epoch=566, iter=105200/160000, loss=0.4488, lr=0.019124, batch_cost=0.3155, reader_cost=0.1395 | ETA 04:48:10 2020-11-01 07:08:17 [INFO] [TRAIN] epoch=567, iter=105400/160000, loss=0.4448, lr=0.019062, batch_cost=0.3312, reader_cost=0.1807 | ETA 05:01:23 2020-11-01 07:09:22 [INFO] [TRAIN] epoch=568, iter=105600/160000, loss=0.4478, lr=0.018999, batch_cost=0.3251, reader_cost=0.1851 | ETA 04:54:44 2020-11-01 07:10:27 [INFO] [TRAIN] epoch=569, iter=105800/160000, loss=0.4427, lr=0.018936, batch_cost=0.3244, reader_cost=0.1784 | ETA 04:53:03 2020-11-01 07:11:31 [INFO] [TRAIN] epoch=570, iter=106000/160000, loss=0.4442, lr=0.018874, batch_cost=0.3235, reader_cost=0.1826 | ETA 04:51:09 2020-11-01 07:12:35 [INFO] [TRAIN] epoch=571, iter=106200/160000, loss=0.4614, lr=0.018811, batch_cost=0.3172, reader_cost=0.1689 | ETA 04:44:26 2020-11-01 07:13:40 [INFO] [TRAIN] epoch=573, iter=106400/160000, loss=0.4575, lr=0.018749, batch_cost=0.3238, reader_cost=0.1798 | ETA 04:49:16 2020-11-01 07:14:43 [INFO] [TRAIN] epoch=574, iter=106600/160000, loss=0.4527, lr=0.018686, batch_cost=0.3161, reader_cost=0.1618 | ETA 04:41:20 2020-11-01 07:15:46 [INFO] [TRAIN] epoch=575, iter=106800/160000, loss=0.4431, lr=0.018623, batch_cost=0.3147, reader_cost=0.1725 | ETA 04:39:02 2020-11-01 07:16:49 [INFO] [TRAIN] epoch=576, iter=107000/160000, loss=0.4397, lr=0.018561, batch_cost=0.3172, reader_cost=0.1730 | ETA 04:40:12 2020-11-01 07:17:52 [INFO] [TRAIN] epoch=577, iter=107200/160000, loss=0.4535, lr=0.018498, batch_cost=0.3120, reader_cost=0.1621 | ETA 04:34:35 2020-11-01 07:18:55 [INFO] [TRAIN] epoch=578, iter=107400/160000, loss=0.4680, lr=0.018435, batch_cost=0.3171, reader_cost=0.1580 | ETA 04:37:59 2020-11-01 07:19:57 [INFO] [TRAIN] epoch=579, iter=107600/160000, loss=0.4524, lr=0.018373, batch_cost=0.3116, reader_cost=0.1569 | ETA 04:32:09 2020-11-01 07:21:02 [INFO] [TRAIN] epoch=580, iter=107800/160000, loss=0.4598, lr=0.018310, batch_cost=0.3213, reader_cost=0.1704 | ETA 04:39:30 2020-11-01 07:22:05 [INFO] [TRAIN] epoch=581, iter=108000/160000, loss=0.4565, lr=0.018247, batch_cost=0.3162, reader_cost=0.1582 | ETA 04:34:04 2020-11-01 07:23:07 [INFO] [TRAIN] epoch=582, iter=108200/160000, loss=0.4520, lr=0.018184, batch_cost=0.3086, reader_cost=0.1323 | ETA 04:26:28 2020-11-01 07:24:11 [INFO] [TRAIN] epoch=583, iter=108400/160000, loss=0.4616, lr=0.018121, batch_cost=0.3212, reader_cost=0.1742 | ETA 04:36:15 2020-11-01 07:25:14 [INFO] [TRAIN] epoch=584, iter=108600/160000, loss=0.4451, lr=0.018058, batch_cost=0.3172, reader_cost=0.1708 | ETA 04:31:45 2020-11-01 07:26:18 [INFO] [TRAIN] epoch=585, iter=108800/160000, loss=0.4438, lr=0.017995, batch_cost=0.3191, reader_cost=0.1720 | ETA 04:32:20 2020-11-01 07:27:25 [INFO] [TRAIN] epoch=587, iter=109000/160000, loss=0.4484, lr=0.017933, batch_cost=0.3356, reader_cost=0.1819 | ETA 04:45:13 2020-11-01 07:28:28 [INFO] [TRAIN] epoch=588, iter=109200/160000, loss=0.4359, lr=0.017870, batch_cost=0.3159, reader_cost=0.1621 | ETA 04:27:25 2020-11-01 07:29:31 [INFO] [TRAIN] epoch=589, iter=109400/160000, loss=0.4357, lr=0.017807, batch_cost=0.3146, reader_cost=0.1659 | ETA 04:25:16 2020-11-01 07:30:35 [INFO] [TRAIN] epoch=590, iter=109600/160000, loss=0.4385, lr=0.017744, batch_cost=0.3190, reader_cost=0.1665 | ETA 04:27:58 2020-11-01 07:31:39 [INFO] [TRAIN] epoch=591, iter=109800/160000, loss=0.4532, lr=0.017681, batch_cost=0.3191, reader_cost=0.1728 | ETA 04:26:56 2020-11-01 07:32:43 [INFO] [TRAIN] epoch=592, iter=110000/160000, loss=0.4475, lr=0.017618, batch_cost=0.3200, reader_cost=0.1765 | ETA 04:26:39 2020-11-01 07:33:47 [INFO] [TRAIN] epoch=593, iter=110200/160000, loss=0.4519, lr=0.017554, batch_cost=0.3203, reader_cost=0.1777 | ETA 04:25:52 2020-11-01 07:34:53 [INFO] [TRAIN] epoch=594, iter=110400/160000, loss=0.4386, lr=0.017491, batch_cost=0.3275, reader_cost=0.1778 | ETA 04:30:45 2020-11-01 07:35:56 [INFO] [TRAIN] epoch=595, iter=110600/160000, loss=0.4469, lr=0.017428, batch_cost=0.3179, reader_cost=0.1685 | ETA 04:21:44 2020-11-01 07:36:59 [INFO] [TRAIN] epoch=596, iter=110800/160000, loss=0.4447, lr=0.017365, batch_cost=0.3134, reader_cost=0.1349 | ETA 04:16:58 2020-11-01 07:38:03 [INFO] [TRAIN] epoch=597, iter=111000/160000, loss=0.4440, lr=0.017302, batch_cost=0.3188, reader_cost=0.1713 | ETA 04:20:19 2020-11-01 07:39:06 [INFO] [TRAIN] epoch=598, iter=111200/160000, loss=0.4515, lr=0.017239, batch_cost=0.3170, reader_cost=0.1667 | ETA 04:17:47 2020-11-01 07:40:09 [INFO] [TRAIN] epoch=599, iter=111400/160000, loss=0.4398, lr=0.017175, batch_cost=0.3149, reader_cost=0.1660 | ETA 04:15:06 2020-11-01 07:41:15 [INFO] [TRAIN] epoch=600, iter=111600/160000, loss=0.4449, lr=0.017112, batch_cost=0.3279, reader_cost=0.1700 | ETA 04:24:32 2020-11-01 07:42:20 [INFO] [TRAIN] epoch=602, iter=111800/160000, loss=0.4359, lr=0.017049, batch_cost=0.3295, reader_cost=0.1671 | ETA 04:24:40 2020-11-01 07:43:26 [INFO] [TRAIN] epoch=603, iter=112000/160000, loss=0.4340, lr=0.016986, batch_cost=0.3285, reader_cost=0.1864 | ETA 04:22:50 2020-11-01 07:43:26 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-01 07:46:19 [INFO] [EVAL] #Images=500 mIoU=0.6593 Acc=0.9423 Kappa=0.9248 2020-11-01 07:46:19 [INFO] [EVAL] Category IoU: [0.9687 0.7564 0.8968 0.3808 0.5149 0.5226 0.5231 0.6621 0.9051 0.614 0.9373 0.7157 0.4557 0.9073 0.4831 0.6693 0.6537 0.3055 0.6548] 2020-11-01 07:46:19 [INFO] [EVAL] Category Acc: [0.9791 0.9061 0.9331 0.8281 0.7394 0.7365 0.8066 0.8498 0.9409 0.7873 0.9625 0.8056 0.7245 0.9376 0.9422 0.9033 0.8792 0.6983 0.7648] 2020-11-01 07:46:19 [INFO] [EVAL] The model with the best validation mIoU (0.6627) was saved at iter 88000. 2020-11-01 07:47:21 [INFO] [TRAIN] epoch=604, iter=112200/160000, loss=0.4429, lr=0.016922, batch_cost=0.3097, reader_cost=0.1608 | ETA 04:06:45 2020-11-01 07:48:24 [INFO] [TRAIN] epoch=605, iter=112400/160000, loss=0.4425, lr=0.016859, batch_cost=0.3114, reader_cost=0.1673 | ETA 04:07:04 2020-11-01 07:49:28 [INFO] [TRAIN] epoch=606, iter=112600/160000, loss=0.4417, lr=0.016796, batch_cost=0.3244, reader_cost=0.1812 | ETA 04:16:18 2020-11-01 07:50:32 [INFO] [TRAIN] epoch=607, iter=112800/160000, loss=0.4278, lr=0.016732, batch_cost=0.3183, reader_cost=0.1767 | ETA 04:10:21 2020-11-01 07:51:37 [INFO] [TRAIN] epoch=608, iter=113000/160000, loss=0.4339, lr=0.016669, batch_cost=0.3261, reader_cost=0.1825 | ETA 04:15:25 2020-11-01 07:52:42 [INFO] [TRAIN] epoch=609, iter=113200/160000, loss=0.4486, lr=0.016605, batch_cost=0.3239, reader_cost=0.1730 | ETA 04:12:38 2020-11-01 07:53:45 [INFO] [TRAIN] epoch=610, iter=113400/160000, loss=0.4358, lr=0.016542, batch_cost=0.3155, reader_cost=0.1748 | ETA 04:05:04 2020-11-01 07:54:50 [INFO] [TRAIN] epoch=611, iter=113600/160000, loss=0.4402, lr=0.016478, batch_cost=0.3257, reader_cost=0.1796 | ETA 04:11:53 2020-11-01 07:55:54 [INFO] [TRAIN] epoch=612, iter=113800/160000, loss=0.4367, lr=0.016415, batch_cost=0.3163, reader_cost=0.1741 | ETA 04:03:31 2020-11-01 07:56:58 [INFO] [TRAIN] epoch=613, iter=114000/160000, loss=0.4405, lr=0.016351, batch_cost=0.3223, reader_cost=0.1776 | ETA 04:07:03 2020-11-01 07:58:03 [INFO] [TRAIN] epoch=614, iter=114200/160000, loss=0.4573, lr=0.016288, batch_cost=0.3231, reader_cost=0.1675 | ETA 04:06:38 2020-11-01 07:59:08 [INFO] [TRAIN] epoch=616, iter=114400/160000, loss=0.4358, lr=0.016224, batch_cost=0.3266, reader_cost=0.1696 | ETA 04:08:11 2020-11-01 08:00:10 [INFO] [TRAIN] epoch=617, iter=114600/160000, loss=0.4398, lr=0.016160, batch_cost=0.3122, reader_cost=0.1516 | ETA 03:56:11 2020-11-01 08:01:13 [INFO] [TRAIN] epoch=618, iter=114800/160000, loss=0.4531, lr=0.016097, batch_cost=0.3145, reader_cost=0.1573 | ETA 03:56:56 2020-11-01 08:02:18 [INFO] [TRAIN] epoch=619, iter=115000/160000, loss=0.4382, lr=0.016033, batch_cost=0.3208, reader_cost=0.1801 | ETA 04:00:34 2020-11-01 08:03:21 [INFO] [TRAIN] epoch=620, iter=115200/160000, loss=0.4333, lr=0.015969, batch_cost=0.3182, reader_cost=0.1580 | ETA 03:57:33 2020-11-01 08:04:25 [INFO] [TRAIN] epoch=621, iter=115400/160000, loss=0.4521, lr=0.015905, batch_cost=0.3191, reader_cost=0.1771 | ETA 03:57:13 2020-11-01 08:05:30 [INFO] [TRAIN] epoch=622, iter=115600/160000, loss=0.4482, lr=0.015841, batch_cost=0.3256, reader_cost=0.1766 | ETA 04:00:55 2020-11-01 08:06:33 [INFO] [TRAIN] epoch=623, iter=115800/160000, loss=0.4348, lr=0.015778, batch_cost=0.3168, reader_cost=0.1759 | ETA 03:53:21 2020-11-01 08:07:38 [INFO] [TRAIN] epoch=624, iter=116000/160000, loss=0.4342, lr=0.015714, batch_cost=0.3208, reader_cost=0.1745 | ETA 03:55:14 2020-11-01 08:08:42 [INFO] [TRAIN] epoch=625, iter=116200/160000, loss=0.4418, lr=0.015650, batch_cost=0.3219, reader_cost=0.1761 | ETA 03:54:58 2020-11-01 08:09:45 [INFO] [TRAIN] epoch=626, iter=116400/160000, loss=0.4454, lr=0.015586, batch_cost=0.3126, reader_cost=0.1339 | ETA 03:47:10 2020-11-01 08:10:50 [INFO] [TRAIN] epoch=627, iter=116600/160000, loss=0.4443, lr=0.015522, batch_cost=0.3248, reader_cost=0.1852 | ETA 03:54:57 2020-11-01 08:11:56 [INFO] [TRAIN] epoch=628, iter=116800/160000, loss=0.4423, lr=0.015458, batch_cost=0.3307, reader_cost=0.1845 | ETA 03:58:04 2020-11-01 08:13:01 [INFO] [TRAIN] epoch=630, iter=117000/160000, loss=0.4275, lr=0.015394, batch_cost=0.3261, reader_cost=0.1758 | ETA 03:53:42 2020-11-01 08:14:05 [INFO] [TRAIN] epoch=631, iter=117200/160000, loss=0.4425, lr=0.015330, batch_cost=0.3205, reader_cost=0.1653 | ETA 03:48:39 2020-11-01 08:15:08 [INFO] [TRAIN] epoch=632, iter=117400/160000, loss=0.4419, lr=0.015266, batch_cost=0.3128, reader_cost=0.1600 | ETA 03:42:05 2020-11-01 08:16:09 [INFO] [TRAIN] epoch=633, iter=117600/160000, loss=0.4510, lr=0.015202, batch_cost=0.3083, reader_cost=0.1635 | ETA 03:37:49 2020-11-01 08:17:13 [INFO] [TRAIN] epoch=634, iter=117800/160000, loss=0.4400, lr=0.015138, batch_cost=0.3167, reader_cost=0.1651 | ETA 03:42:42 2020-11-01 08:18:14 [INFO] [TRAIN] epoch=635, iter=118000/160000, loss=0.4467, lr=0.015074, batch_cost=0.3074, reader_cost=0.1622 | ETA 03:35:11 2020-11-01 08:19:17 [INFO] [TRAIN] epoch=636, iter=118200/160000, loss=0.4374, lr=0.015009, batch_cost=0.3161, reader_cost=0.1736 | ETA 03:40:13 2020-11-01 08:20:19 [INFO] [TRAIN] epoch=637, iter=118400/160000, loss=0.4435, lr=0.014945, batch_cost=0.3065, reader_cost=0.1529 | ETA 03:32:31 2020-11-01 08:21:23 [INFO] [TRAIN] epoch=638, iter=118600/160000, loss=0.4323, lr=0.014881, batch_cost=0.3207, reader_cost=0.1684 | ETA 03:41:15 2020-11-01 08:22:26 [INFO] [TRAIN] epoch=639, iter=118800/160000, loss=0.4399, lr=0.014817, batch_cost=0.3176, reader_cost=0.1697 | ETA 03:38:05 2020-11-01 08:23:28 [INFO] [TRAIN] epoch=640, iter=119000/160000, loss=0.4343, lr=0.014752, batch_cost=0.3089, reader_cost=0.1600 | ETA 03:31:05 2020-11-01 08:24:32 [INFO] [TRAIN] epoch=641, iter=119200/160000, loss=0.4312, lr=0.014688, batch_cost=0.3215, reader_cost=0.1732 | ETA 03:38:36 2020-11-01 08:25:34 [INFO] [TRAIN] epoch=642, iter=119400/160000, loss=0.4361, lr=0.014624, batch_cost=0.3064, reader_cost=0.1463 | ETA 03:27:19 2020-11-01 08:26:38 [INFO] [TRAIN] epoch=644, iter=119600/160000, loss=0.4320, lr=0.014559, batch_cost=0.3207, reader_cost=0.1754 | ETA 03:35:57 2020-11-01 08:27:41 [INFO] [TRAIN] epoch=645, iter=119800/160000, loss=0.4340, lr=0.014495, batch_cost=0.3177, reader_cost=0.1723 | ETA 03:32:53 2020-11-01 08:28:43 [INFO] [TRAIN] epoch=646, iter=120000/160000, loss=0.4325, lr=0.014430, batch_cost=0.3094, reader_cost=0.1535 | ETA 03:26:14 2020-11-01 08:28:43 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-01 08:31:29 [INFO] [EVAL] #Images=500 mIoU=0.6803 Acc=0.9468 Kappa=0.9308 2020-11-01 08:31:29 [INFO] [EVAL] Category IoU: [0.9742 0.7959 0.9022 0.458 0.511 0.5335 0.5554 0.6643 0.909 0.5931 0.9387 0.7269 0.431 0.9226 0.633 0.7021 0.597 0.412 0.6658] 2020-11-01 08:31:29 [INFO] [EVAL] Category Acc: [0.9859 0.8905 0.9403 0.8308 0.689 0.7477 0.7925 0.8489 0.9409 0.8715 0.9651 0.8227 0.7413 0.956 0.8716 0.8636 0.8586 0.6878 0.7702] 2020-11-01 08:31:29 [INFO] [EVAL] The model with the best validation mIoU (0.6803) was saved at iter 120000. 2020-11-01 08:32:30 [INFO] [TRAIN] epoch=647, iter=120200/160000, loss=0.4300, lr=0.014366, batch_cost=0.3080, reader_cost=0.1641 | ETA 03:24:17 2020-11-01 08:33:34 [INFO] [TRAIN] epoch=648, iter=120400/160000, loss=0.4307, lr=0.014301, batch_cost=0.3165, reader_cost=0.1736 | ETA 03:28:54 2020-11-01 08:34:38 [INFO] [TRAIN] epoch=649, iter=120600/160000, loss=0.4436, lr=0.014237, batch_cost=0.3194, reader_cost=0.1772 | ETA 03:29:44 2020-11-01 08:35:40 [INFO] [TRAIN] epoch=650, iter=120800/160000, loss=0.4384, lr=0.014172, batch_cost=0.3131, reader_cost=0.1687 | ETA 03:24:34 2020-11-01 08:36:42 [INFO] [TRAIN] epoch=651, iter=121000/160000, loss=0.4461, lr=0.014108, batch_cost=0.3103, reader_cost=0.1657 | ETA 03:21:42 2020-11-01 08:37:46 [INFO] [TRAIN] epoch=652, iter=121200/160000, loss=0.4300, lr=0.014043, batch_cost=0.3165, reader_cost=0.1643 | ETA 03:24:41 2020-11-01 08:38:50 [INFO] [TRAIN] epoch=653, iter=121400/160000, loss=0.4296, lr=0.013978, batch_cost=0.3213, reader_cost=0.1781 | ETA 03:26:41 2020-11-01 08:39:53 [INFO] [TRAIN] epoch=654, iter=121600/160000, loss=0.4296, lr=0.013913, batch_cost=0.3136, reader_cost=0.1714 | ETA 03:20:41 2020-11-01 08:40:55 [INFO] [TRAIN] epoch=655, iter=121800/160000, loss=0.4283, lr=0.013849, batch_cost=0.3103, reader_cost=0.1628 | ETA 03:17:33 2020-11-01 08:41:59 [INFO] [TRAIN] epoch=656, iter=122000/160000, loss=0.4193, lr=0.013784, batch_cost=0.3229, reader_cost=0.1826 | ETA 03:24:30 2020-11-01 08:43:01 [INFO] [TRAIN] epoch=657, iter=122200/160000, loss=0.4345, lr=0.013719, batch_cost=0.3103, reader_cost=0.1663 | ETA 03:15:31 2020-11-01 08:44:06 [INFO] [TRAIN] epoch=659, iter=122400/160000, loss=0.4451, lr=0.013654, batch_cost=0.3228, reader_cost=0.1634 | ETA 03:22:15 2020-11-01 08:45:10 [INFO] [TRAIN] epoch=660, iter=122600/160000, loss=0.4288, lr=0.013589, batch_cost=0.3185, reader_cost=0.1653 | ETA 03:18:30 2020-11-01 08:46:12 [INFO] [TRAIN] epoch=661, iter=122800/160000, loss=0.4517, lr=0.013524, batch_cost=0.3128, reader_cost=0.1535 | ETA 03:13:55 2020-11-01 08:47:15 [INFO] [TRAIN] epoch=662, iter=123000/160000, loss=0.4274, lr=0.013459, batch_cost=0.3123, reader_cost=0.1701 | ETA 03:12:33 2020-11-01 08:48:18 [INFO] [TRAIN] epoch=663, iter=123200/160000, loss=0.4363, lr=0.013394, batch_cost=0.3165, reader_cost=0.1690 | ETA 03:14:07 2020-11-01 08:49:20 [INFO] [TRAIN] epoch=664, iter=123400/160000, loss=0.4308, lr=0.013329, batch_cost=0.3124, reader_cost=0.1666 | ETA 03:10:32 2020-11-01 08:50:24 [INFO] [TRAIN] epoch=665, iter=123600/160000, loss=0.4333, lr=0.013264, batch_cost=0.3165, reader_cost=0.1739 | ETA 03:12:00 2020-11-01 08:51:28 [INFO] [TRAIN] epoch=666, iter=123800/160000, loss=0.4241, lr=0.013199, batch_cost=0.3222, reader_cost=0.1818 | ETA 03:14:23 2020-11-01 08:52:32 [INFO] [TRAIN] epoch=667, iter=124000/160000, loss=0.4354, lr=0.013134, batch_cost=0.3198, reader_cost=0.1764 | ETA 03:11:53 2020-11-01 08:53:35 [INFO] [TRAIN] epoch=668, iter=124200/160000, loss=0.4341, lr=0.013069, batch_cost=0.3147, reader_cost=0.1624 | ETA 03:07:45 2020-11-01 08:54:37 [INFO] [TRAIN] epoch=669, iter=124400/160000, loss=0.4292, lr=0.013004, batch_cost=0.3124, reader_cost=0.1427 | ETA 03:05:19 2020-11-01 08:55:41 [INFO] [TRAIN] epoch=670, iter=124600/160000, loss=0.4278, lr=0.012938, batch_cost=0.3181, reader_cost=0.1716 | ETA 03:07:40 2020-11-01 08:56:43 [INFO] [TRAIN] epoch=671, iter=124800/160000, loss=0.4266, lr=0.012873, batch_cost=0.3116, reader_cost=0.1602 | ETA 03:02:47 2020-11-01 08:57:47 [INFO] [TRAIN] epoch=673, iter=125000/160000, loss=0.4481, lr=0.012808, batch_cost=0.3173, reader_cost=0.0909 | ETA 03:05:04 2020-11-01 08:58:51 [INFO] [TRAIN] epoch=674, iter=125200/160000, loss=0.4389, lr=0.012742, batch_cost=0.3203, reader_cost=0.1615 | ETA 03:05:45 2020-11-01 08:59:53 [INFO] [TRAIN] epoch=675, iter=125400/160000, loss=0.4300, lr=0.012677, batch_cost=0.3102, reader_cost=0.1534 | ETA 02:58:52 2020-11-01 09:00:57 [INFO] [TRAIN] epoch=676, iter=125600/160000, loss=0.4330, lr=0.012611, batch_cost=0.3179, reader_cost=0.1554 | ETA 03:02:14 2020-11-01 09:02:01 [INFO] [TRAIN] epoch=677, iter=125800/160000, loss=0.4318, lr=0.012546, batch_cost=0.3205, reader_cost=0.1800 | ETA 03:02:41 2020-11-01 09:03:04 [INFO] [TRAIN] epoch=678, iter=126000/160000, loss=0.4334, lr=0.012480, batch_cost=0.3190, reader_cost=0.1665 | ETA 03:00:47 2020-11-01 09:04:07 [INFO] [TRAIN] epoch=679, iter=126200/160000, loss=0.4380, lr=0.012415, batch_cost=0.3132, reader_cost=0.1629 | ETA 02:56:24 2020-11-01 09:05:09 [INFO] [TRAIN] epoch=680, iter=126400/160000, loss=0.4270, lr=0.012349, batch_cost=0.3104, reader_cost=0.1101 | ETA 02:53:48 2020-11-01 09:06:12 [INFO] [TRAIN] epoch=681, iter=126600/160000, loss=0.4301, lr=0.012284, batch_cost=0.3120, reader_cost=0.1390 | ETA 02:53:41 2020-11-01 09:07:17 [INFO] [TRAIN] epoch=682, iter=126800/160000, loss=0.4301, lr=0.012218, batch_cost=0.3258, reader_cost=0.1866 | ETA 03:00:18 2020-11-01 09:08:20 [INFO] [TRAIN] epoch=683, iter=127000/160000, loss=0.4451, lr=0.012152, batch_cost=0.3147, reader_cost=0.1700 | ETA 02:53:04 2020-11-01 09:09:24 [INFO] [TRAIN] epoch=684, iter=127200/160000, loss=0.4328, lr=0.012086, batch_cost=0.3234, reader_cost=0.1661 | ETA 02:56:46 2020-11-01 09:10:28 [INFO] [TRAIN] epoch=685, iter=127400/160000, loss=0.4254, lr=0.012021, batch_cost=0.3172, reader_cost=0.1755 | ETA 02:52:19 2020-11-01 09:11:34 [INFO] [TRAIN] epoch=687, iter=127600/160000, loss=0.4410, lr=0.011955, batch_cost=0.3297, reader_cost=0.1834 | ETA 02:58:03 2020-11-01 09:12:38 [INFO] [TRAIN] epoch=688, iter=127800/160000, loss=0.4440, lr=0.011889, batch_cost=0.3201, reader_cost=0.1708 | ETA 02:51:48 2020-11-01 09:13:41 [INFO] [TRAIN] epoch=689, iter=128000/160000, loss=0.4179, lr=0.011823, batch_cost=0.3157, reader_cost=0.1724 | ETA 02:48:22 2020-11-01 09:13:41 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-01 09:16:31 [INFO] [EVAL] #Images=500 mIoU=0.6815 Acc=0.9468 Kappa=0.9308 2020-11-01 09:16:31 [INFO] [EVAL] Category IoU: [0.9731 0.7929 0.9022 0.4323 0.5034 0.5274 0.5511 0.6769 0.9083 0.6132 0.9382 0.7344 0.4909 0.9237 0.6148 0.6995 0.5752 0.4213 0.6696] 2020-11-01 09:16:31 [INFO] [EVAL] Category Acc: [0.9826 0.9029 0.9351 0.7852 0.7729 0.7942 0.8067 0.8634 0.9427 0.8528 0.9603 0.8455 0.6739 0.9564 0.8411 0.8467 0.786 0.7548 0.7718] 2020-11-01 09:16:31 [INFO] [EVAL] The model with the best validation mIoU (0.6815) was saved at iter 128000. 2020-11-01 09:17:34 [INFO] [TRAIN] epoch=690, iter=128200/160000, loss=0.4211, lr=0.011757, batch_cost=0.3124, reader_cost=0.1694 | ETA 02:45:35 2020-11-01 09:18:36 [INFO] [TRAIN] epoch=691, iter=128400/160000, loss=0.4250, lr=0.011691, batch_cost=0.3119, reader_cost=0.1571 | ETA 02:44:17 2020-11-01 09:19:40 [INFO] [TRAIN] epoch=692, iter=128600/160000, loss=0.4403, lr=0.011625, batch_cost=0.3167, reader_cost=0.1736 | ETA 02:45:43 2020-11-01 09:20:42 [INFO] [TRAIN] epoch=693, iter=128800/160000, loss=0.4272, lr=0.011559, batch_cost=0.3130, reader_cost=0.1613 | ETA 02:42:47 2020-11-01 09:21:46 [INFO] [TRAIN] epoch=694, iter=129000/160000, loss=0.4207, lr=0.011493, batch_cost=0.3193, reader_cost=0.1738 | ETA 02:44:57 2020-11-01 09:22:50 [INFO] [TRAIN] epoch=695, iter=129200/160000, loss=0.4282, lr=0.011427, batch_cost=0.3205, reader_cost=0.1783 | ETA 02:44:32 2020-11-01 09:23:55 [INFO] [TRAIN] epoch=696, iter=129400/160000, loss=0.4414, lr=0.011360, batch_cost=0.3242, reader_cost=0.1779 | ETA 02:45:20 2020-11-01 09:24:59 [INFO] [TRAIN] epoch=697, iter=129600/160000, loss=0.4270, lr=0.011294, batch_cost=0.3181, reader_cost=0.1748 | ETA 02:41:08 2020-11-01 09:26:03 [INFO] [TRAIN] epoch=698, iter=129800/160000, loss=0.4240, lr=0.011228, batch_cost=0.3216, reader_cost=0.1290 | ETA 02:41:51 2020-11-01 09:27:06 [INFO] [TRAIN] epoch=699, iter=130000/160000, loss=0.4293, lr=0.011162, batch_cost=0.3170, reader_cost=0.1730 | ETA 02:38:29 2020-11-01 09:28:10 [INFO] [TRAIN] epoch=700, iter=130200/160000, loss=0.4307, lr=0.011095, batch_cost=0.3179, reader_cost=0.1608 | ETA 02:37:54 2020-11-01 09:29:15 [INFO] [TRAIN] epoch=702, iter=130400/160000, loss=0.4263, lr=0.011029, batch_cost=0.3232, reader_cost=0.1705 | ETA 02:39:26 2020-11-01 09:30:18 [INFO] [TRAIN] epoch=703, iter=130600/160000, loss=0.4318, lr=0.010962, batch_cost=0.3175, reader_cost=0.1457 | ETA 02:35:35 2020-11-01 09:31:22 [INFO] [TRAIN] epoch=704, iter=130800/160000, loss=0.4225, lr=0.010896, batch_cost=0.3190, reader_cost=0.1397 | ETA 02:35:15 2020-11-01 09:32:26 [INFO] [TRAIN] epoch=705, iter=131000/160000, loss=0.4280, lr=0.010829, batch_cost=0.3225, reader_cost=0.1803 | ETA 02:35:52 2020-11-01 09:33:30 [INFO] [TRAIN] epoch=706, iter=131200/160000, loss=0.4329, lr=0.010763, batch_cost=0.3189, reader_cost=0.1731 | ETA 02:33:03 2020-11-01 09:34:34 [INFO] [TRAIN] epoch=707, iter=131400/160000, loss=0.4330, lr=0.010696, batch_cost=0.3184, reader_cost=0.1694 | ETA 02:31:46 2020-11-01 09:35:37 [INFO] [TRAIN] epoch=708, iter=131600/160000, loss=0.4308, lr=0.010629, batch_cost=0.3144, reader_cost=0.1653 | ETA 02:28:48 2020-11-01 09:36:40 [INFO] [TRAIN] epoch=709, iter=131800/160000, loss=0.4239, lr=0.010562, batch_cost=0.3173, reader_cost=0.1692 | ETA 02:29:07 2020-11-01 09:37:43 [INFO] [TRAIN] epoch=710, iter=132000/160000, loss=0.4343, lr=0.010496, batch_cost=0.3128, reader_cost=0.1670 | ETA 02:25:59 2020-11-01 09:38:46 [INFO] [TRAIN] epoch=711, iter=132200/160000, loss=0.4238, lr=0.010429, batch_cost=0.3149, reader_cost=0.1510 | ETA 02:25:55 2020-11-01 09:39:50 [INFO] [TRAIN] epoch=712, iter=132400/160000, loss=0.4326, lr=0.010362, batch_cost=0.3222, reader_cost=0.1802 | ETA 02:28:11 2020-11-01 09:40:54 [INFO] [TRAIN] epoch=713, iter=132600/160000, loss=0.4251, lr=0.010295, batch_cost=0.3204, reader_cost=0.1764 | ETA 02:26:17 2020-11-01 09:41:59 [INFO] [TRAIN] epoch=714, iter=132800/160000, loss=0.4250, lr=0.010228, batch_cost=0.3227, reader_cost=0.1710 | ETA 02:26:16 2020-11-01 09:43:05 [INFO] [TRAIN] epoch=716, iter=133000/160000, loss=0.4250, lr=0.010161, batch_cost=0.3305, reader_cost=0.1750 | ETA 02:28:44 2020-11-01 09:44:09 [INFO] [TRAIN] epoch=717, iter=133200/160000, loss=0.4251, lr=0.010094, batch_cost=0.3211, reader_cost=0.1720 | ETA 02:23:25 2020-11-01 09:45:12 [INFO] [TRAIN] epoch=718, iter=133400/160000, loss=0.4269, lr=0.010027, batch_cost=0.3152, reader_cost=0.1671 | ETA 02:19:43 2020-11-01 09:46:15 [INFO] [TRAIN] epoch=719, iter=133600/160000, loss=0.4244, lr=0.009959, batch_cost=0.3142, reader_cost=0.1698 | ETA 02:18:15 2020-11-01 09:47:17 [INFO] [TRAIN] epoch=720, iter=133800/160000, loss=0.4192, lr=0.009892, batch_cost=0.3124, reader_cost=0.1601 | ETA 02:16:26 2020-11-01 09:48:21 [INFO] [TRAIN] epoch=721, iter=134000/160000, loss=0.4324, lr=0.009825, batch_cost=0.3195, reader_cost=0.1612 | ETA 02:18:27 2020-11-01 09:49:25 [INFO] [TRAIN] epoch=722, iter=134200/160000, loss=0.4257, lr=0.009758, batch_cost=0.3172, reader_cost=0.1708 | ETA 02:16:22 2020-11-01 09:50:28 [INFO] [TRAIN] epoch=723, iter=134400/160000, loss=0.4187, lr=0.009690, batch_cost=0.3137, reader_cost=0.1646 | ETA 02:13:51 2020-11-01 09:51:30 [INFO] [TRAIN] epoch=724, iter=134600/160000, loss=0.4201, lr=0.009623, batch_cost=0.3135, reader_cost=0.1722 | ETA 02:12:42 2020-11-01 09:52:33 [INFO] [TRAIN] epoch=725, iter=134800/160000, loss=0.4318, lr=0.009555, batch_cost=0.3146, reader_cost=0.1585 | ETA 02:12:07 2020-11-01 09:53:37 [INFO] [TRAIN] epoch=726, iter=135000/160000, loss=0.4162, lr=0.009488, batch_cost=0.3197, reader_cost=0.1640 | ETA 02:13:12 2020-11-01 09:54:41 [INFO] [TRAIN] epoch=727, iter=135200/160000, loss=0.4151, lr=0.009420, batch_cost=0.3170, reader_cost=0.1750 | ETA 02:11:02 2020-11-01 09:55:44 [INFO] [TRAIN] epoch=728, iter=135400/160000, loss=0.4319, lr=0.009352, batch_cost=0.3183, reader_cost=0.1676 | ETA 02:10:29 2020-11-01 09:56:48 [INFO] [TRAIN] epoch=730, iter=135600/160000, loss=0.4295, lr=0.009285, batch_cost=0.3202, reader_cost=0.1501 | ETA 02:10:13 2020-11-01 09:57:50 [INFO] [TRAIN] epoch=731, iter=135800/160000, loss=0.4317, lr=0.009217, batch_cost=0.3108, reader_cost=0.1493 | ETA 02:05:21 2020-11-01 09:58:55 [INFO] [TRAIN] epoch=732, iter=136000/160000, loss=0.4215, lr=0.009149, batch_cost=0.3233, reader_cost=0.1735 | ETA 02:09:19 2020-11-01 09:58:55 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-01 10:01:40 [INFO] [EVAL] #Images=500 mIoU=0.6791 Acc=0.9466 Kappa=0.9306 2020-11-01 10:01:40 [INFO] [EVAL] Category IoU: [0.9738 0.797 0.9004 0.4449 0.4983 0.5454 0.5518 0.6772 0.9093 0.5938 0.9379 0.7278 0.4395 0.9244 0.6227 0.699 0.5945 0.3902 0.6742] 2020-11-01 10:01:40 [INFO] [EVAL] Category Acc: [0.9843 0.8975 0.9358 0.7935 0.7504 0.7355 0.7985 0.8639 0.9498 0.7833 0.9589 0.8143 0.7323 0.9577 0.8676 0.8658 0.8951 0.6658 0.7821] 2020-11-01 10:01:40 [INFO] [EVAL] The model with the best validation mIoU (0.6815) was saved at iter 128000. 2020-11-01 10:02:40 [INFO] [TRAIN] epoch=733, iter=136200/160000, loss=0.4281, lr=0.009081, batch_cost=0.3026, reader_cost=0.1593 | ETA 02:00:01 2020-11-01 10:03:43 [INFO] [TRAIN] epoch=734, iter=136400/160000, loss=0.4299, lr=0.009013, batch_cost=0.3124, reader_cost=0.1635 | ETA 02:02:52 2020-11-01 10:04:43 [INFO] [TRAIN] epoch=735, iter=136600/160000, loss=0.4252, lr=0.008945, batch_cost=0.3010, reader_cost=0.1193 | ETA 01:57:22 2020-11-01 10:05:46 [INFO] [TRAIN] epoch=736, iter=136800/160000, loss=0.4286, lr=0.008877, batch_cost=0.3147, reader_cost=0.1096 | ETA 02:01:40 2020-11-01 10:06:49 [INFO] [TRAIN] epoch=737, iter=137000/160000, loss=0.4212, lr=0.008809, batch_cost=0.3126, reader_cost=0.1550 | ETA 01:59:50 2020-11-01 10:07:51 [INFO] [TRAIN] epoch=738, iter=137200/160000, loss=0.4256, lr=0.008741, batch_cost=0.3108, reader_cost=0.1694 | ETA 01:58:05 2020-11-01 10:08:52 [INFO] [TRAIN] epoch=739, iter=137400/160000, loss=0.4278, lr=0.008672, batch_cost=0.3072, reader_cost=0.1656 | ETA 01:55:42 2020-11-01 10:09:53 [INFO] [TRAIN] epoch=740, iter=137600/160000, loss=0.4136, lr=0.008604, batch_cost=0.3057, reader_cost=0.1605 | ETA 01:54:08 2020-11-01 10:10:55 [INFO] [TRAIN] epoch=741, iter=137800/160000, loss=0.4146, lr=0.008536, batch_cost=0.3087, reader_cost=0.1510 | ETA 01:54:13 2020-11-01 10:11:57 [INFO] [TRAIN] epoch=742, iter=138000/160000, loss=0.4201, lr=0.008467, batch_cost=0.3102, reader_cost=0.1583 | ETA 01:53:45 2020-11-01 10:13:01 [INFO] [TRAIN] epoch=744, iter=138200/160000, loss=0.4258, lr=0.008399, batch_cost=0.3181, reader_cost=0.1769 | ETA 01:55:34 2020-11-01 10:14:03 [INFO] [TRAIN] epoch=745, iter=138400/160000, loss=0.4100, lr=0.008330, batch_cost=0.3115, reader_cost=0.1655 | ETA 01:52:08 2020-11-01 10:15:07 [INFO] [TRAIN] epoch=746, iter=138600/160000, loss=0.4112, lr=0.008262, batch_cost=0.3177, reader_cost=0.1720 | ETA 01:53:18 2020-11-01 10:16:08 [INFO] [TRAIN] epoch=747, iter=138800/160000, loss=0.4224, lr=0.008193, batch_cost=0.3095, reader_cost=0.1588 | ETA 01:49:20 2020-11-01 10:17:10 [INFO] [TRAIN] epoch=748, iter=139000/160000, loss=0.4237, lr=0.008124, batch_cost=0.3067, reader_cost=0.1434 | ETA 01:47:20 2020-11-01 10:18:11 [INFO] [TRAIN] epoch=749, iter=139200/160000, loss=0.4143, lr=0.008056, batch_cost=0.3043, reader_cost=0.1565 | ETA 01:45:29 2020-11-01 10:19:12 [INFO] [TRAIN] epoch=750, iter=139400/160000, loss=0.4253, lr=0.007987, batch_cost=0.3076, reader_cost=0.1127 | ETA 01:45:35 2020-11-01 10:20:13 [INFO] [TRAIN] epoch=751, iter=139600/160000, loss=0.4257, lr=0.007918, batch_cost=0.3044, reader_cost=0.1271 | ETA 01:43:28 2020-11-01 10:21:14 [INFO] [TRAIN] epoch=752, iter=139800/160000, loss=0.4359, lr=0.007849, batch_cost=0.3043, reader_cost=0.0979 | ETA 01:42:27 2020-11-01 10:22:17 [INFO] [TRAIN] epoch=753, iter=140000/160000, loss=0.4123, lr=0.007780, batch_cost=0.3130, reader_cost=0.1469 | ETA 01:44:20 2020-11-01 10:23:17 [INFO] [TRAIN] epoch=754, iter=140200/160000, loss=0.4230, lr=0.007710, batch_cost=0.3047, reader_cost=0.1458 | ETA 01:40:33 2020-11-01 10:24:18 [INFO] [TRAIN] epoch=755, iter=140400/160000, loss=0.4183, lr=0.007641, batch_cost=0.3035, reader_cost=0.1488 | ETA 01:39:08 2020-11-01 10:25:20 [INFO] [TRAIN] epoch=756, iter=140600/160000, loss=0.4133, lr=0.007572, batch_cost=0.3075, reader_cost=0.1648 | ETA 01:39:26 2020-11-01 10:26:21 [INFO] [TRAIN] epoch=757, iter=140800/160000, loss=0.4094, lr=0.007503, batch_cost=0.3047, reader_cost=0.0417 | ETA 01:37:30 2020-11-01 10:27:22 [INFO] [TRAIN] epoch=759, iter=141000/160000, loss=0.4203, lr=0.007433, batch_cost=0.3079, reader_cost=0.1229 | ETA 01:37:29 2020-11-01 10:28:25 [INFO] [TRAIN] epoch=760, iter=141200/160000, loss=0.4320, lr=0.007364, batch_cost=0.3118, reader_cost=0.0856 | ETA 01:37:40 2020-11-01 10:29:25 [INFO] [TRAIN] epoch=761, iter=141400/160000, loss=0.4166, lr=0.007294, batch_cost=0.3040, reader_cost=0.1553 | ETA 01:34:13 2020-11-01 10:30:26 [INFO] [TRAIN] epoch=762, iter=141600/160000, loss=0.4218, lr=0.007224, batch_cost=0.3041, reader_cost=0.1330 | ETA 01:33:14 2020-11-01 10:31:28 [INFO] [TRAIN] epoch=763, iter=141800/160000, loss=0.4260, lr=0.007155, batch_cost=0.3091, reader_cost=0.1548 | ETA 01:33:45 2020-11-01 10:32:30 [INFO] [TRAIN] epoch=764, iter=142000/160000, loss=0.4246, lr=0.007085, batch_cost=0.3103, reader_cost=0.1494 | ETA 01:33:04 2020-11-01 10:33:32 [INFO] [TRAIN] epoch=765, iter=142200/160000, loss=0.4193, lr=0.007015, batch_cost=0.3115, reader_cost=0.1449 | ETA 01:32:24 2020-11-01 10:34:34 [INFO] [TRAIN] epoch=766, iter=142400/160000, loss=0.4180, lr=0.006945, batch_cost=0.3073, reader_cost=0.1496 | ETA 01:30:08 2020-11-01 10:35:36 [INFO] [TRAIN] epoch=767, iter=142600/160000, loss=0.4180, lr=0.006875, batch_cost=0.3110, reader_cost=0.1339 | ETA 01:30:12 2020-11-01 10:36:38 [INFO] [TRAIN] epoch=768, iter=142800/160000, loss=0.4224, lr=0.006805, batch_cost=0.3078, reader_cost=0.0939 | ETA 01:28:14 2020-11-01 10:37:39 [INFO] [TRAIN] epoch=769, iter=143000/160000, loss=0.4099, lr=0.006735, batch_cost=0.3077, reader_cost=0.1149 | ETA 01:27:11 2020-11-01 10:38:39 [INFO] [TRAIN] epoch=770, iter=143200/160000, loss=0.4138, lr=0.006664, batch_cost=0.3001, reader_cost=0.1606 | ETA 01:24:01 2020-11-01 10:39:40 [INFO] [TRAIN] epoch=771, iter=143400/160000, loss=0.4223, lr=0.006594, batch_cost=0.3053, reader_cost=0.1647 | ETA 01:24:28 2020-11-01 10:40:44 [INFO] [TRAIN] epoch=773, iter=143600/160000, loss=0.4232, lr=0.006524, batch_cost=0.3193, reader_cost=0.1734 | ETA 01:27:16 2020-11-01 10:41:46 [INFO] [TRAIN] epoch=774, iter=143800/160000, loss=0.4187, lr=0.006453, batch_cost=0.3076, reader_cost=0.1157 | ETA 01:23:02 2020-11-01 10:42:47 [INFO] [TRAIN] epoch=775, iter=144000/160000, loss=0.4173, lr=0.006382, batch_cost=0.3061, reader_cost=0.1156 | ETA 01:21:37 2020-11-01 10:42:47 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-01 10:45:33 [INFO] [EVAL] #Images=500 mIoU=0.6781 Acc=0.9441 Kappa=0.9273 2020-11-01 10:45:33 [INFO] [EVAL] Category IoU: [0.9693 0.7656 0.8957 0.4749 0.4232 0.5386 0.5413 0.6706 0.9098 0.5973 0.9381 0.7279 0.4775 0.9226 0.5966 0.7106 0.6123 0.4326 0.6788] 2020-11-01 10:45:33 [INFO] [EVAL] Category Acc: [0.9799 0.8812 0.9265 0.7281 0.7529 0.7703 0.8134 0.8798 0.9508 0.8501 0.9602 0.8397 0.7118 0.9626 0.8722 0.88 0.8052 0.7467 0.7873] 2020-11-01 10:45:33 [INFO] [EVAL] The model with the best validation mIoU (0.6815) was saved at iter 128000. 2020-11-01 10:46:34 [INFO] [TRAIN] epoch=776, iter=144200/160000, loss=0.4151, lr=0.006312, batch_cost=0.3055, reader_cost=0.1423 | ETA 01:20:27 2020-11-01 10:47:36 [INFO] [TRAIN] epoch=777, iter=144400/160000, loss=0.4173, lr=0.006241, batch_cost=0.3061, reader_cost=0.1387 | ETA 01:19:35 2020-11-01 10:48:38 [INFO] [TRAIN] epoch=778, iter=144600/160000, loss=0.4198, lr=0.006170, batch_cost=0.3099, reader_cost=0.1272 | ETA 01:19:32 2020-11-01 10:49:39 [INFO] [TRAIN] epoch=779, iter=144800/160000, loss=0.4098, lr=0.006099, batch_cost=0.3071, reader_cost=0.1555 | ETA 01:17:48 2020-11-01 10:50:42 [INFO] [TRAIN] epoch=780, iter=145000/160000, loss=0.4181, lr=0.006028, batch_cost=0.3137, reader_cost=0.1710 | ETA 01:18:26 2020-11-01 10:51:44 [INFO] [TRAIN] epoch=781, iter=145200/160000, loss=0.4164, lr=0.005957, batch_cost=0.3100, reader_cost=0.1727 | ETA 01:16:28 2020-11-01 10:52:47 [INFO] [TRAIN] epoch=782, iter=145400/160000, loss=0.4228, lr=0.005885, batch_cost=0.3164, reader_cost=0.1744 | ETA 01:16:59 2020-11-01 10:53:50 [INFO] [TRAIN] epoch=783, iter=145600/160000, loss=0.4171, lr=0.005814, batch_cost=0.3163, reader_cost=0.1755 | ETA 01:15:54 2020-11-01 10:54:52 [INFO] [TRAIN] epoch=784, iter=145800/160000, loss=0.4163, lr=0.005743, batch_cost=0.3101, reader_cost=0.1555 | ETA 01:13:23 2020-11-01 10:55:56 [INFO] [TRAIN] epoch=785, iter=146000/160000, loss=0.4149, lr=0.005671, batch_cost=0.3155, reader_cost=0.1634 | ETA 01:13:36 2020-11-01 10:56:58 [INFO] [TRAIN] epoch=787, iter=146200/160000, loss=0.4273, lr=0.005599, batch_cost=0.3131, reader_cost=0.1692 | ETA 01:12:00 2020-11-01 10:58:01 [INFO] [TRAIN] epoch=788, iter=146400/160000, loss=0.4197, lr=0.005528, batch_cost=0.3128, reader_cost=0.1578 | ETA 01:10:54 2020-11-01 10:59:02 [INFO] [TRAIN] epoch=789, iter=146600/160000, loss=0.4124, lr=0.005456, batch_cost=0.3038, reader_cost=0.1585 | ETA 01:07:51 2020-11-01 11:00:04 [INFO] [TRAIN] epoch=790, iter=146800/160000, loss=0.4242, lr=0.005384, batch_cost=0.3130, reader_cost=0.1700 | ETA 01:08:52 2020-11-01 11:01:06 [INFO] [TRAIN] epoch=791, iter=147000/160000, loss=0.4265, lr=0.005312, batch_cost=0.3092, reader_cost=0.1659 | ETA 01:06:59 2020-11-01 11:02:08 [INFO] [TRAIN] epoch=792, iter=147200/160000, loss=0.4190, lr=0.005239, batch_cost=0.3094, reader_cost=0.1691 | ETA 01:06:00 2020-11-01 11:03:10 [INFO] [TRAIN] epoch=793, iter=147400/160000, loss=0.4197, lr=0.005167, batch_cost=0.3115, reader_cost=0.1723 | ETA 01:05:25 2020-11-01 11:04:11 [INFO] [TRAIN] epoch=794, iter=147600/160000, loss=0.4097, lr=0.005095, batch_cost=0.3054, reader_cost=0.1519 | ETA 01:03:07 2020-11-01 11:05:13 [INFO] [TRAIN] epoch=795, iter=147800/160000, loss=0.4185, lr=0.005022, batch_cost=0.3110, reader_cost=0.1687 | ETA 01:03:13 2020-11-01 11:06:17 [INFO] [TRAIN] epoch=796, iter=148000/160000, loss=0.4187, lr=0.004949, batch_cost=0.3179, reader_cost=0.1614 | ETA 01:03:35 2020-11-01 11:07:20 [INFO] [TRAIN] epoch=797, iter=148200/160000, loss=0.4048, lr=0.004877, batch_cost=0.3149, reader_cost=0.1741 | ETA 01:01:56 2020-11-01 11:08:23 [INFO] [TRAIN] epoch=798, iter=148400/160000, loss=0.4196, lr=0.004804, batch_cost=0.3127, reader_cost=0.1688 | ETA 01:00:27 2020-11-01 11:09:25 [INFO] [TRAIN] epoch=799, iter=148600/160000, loss=0.4155, lr=0.004731, batch_cost=0.3144, reader_cost=0.1693 | ETA 00:59:44 2020-11-01 11:10:27 [INFO] [TRAIN] epoch=800, iter=148800/160000, loss=0.4095, lr=0.004657, batch_cost=0.3096, reader_cost=0.1257 | ETA 00:57:47 2020-11-01 11:11:31 [INFO] [TRAIN] epoch=802, iter=149000/160000, loss=0.4167, lr=0.004584, batch_cost=0.3200, reader_cost=0.1685 | ETA 00:58:40 2020-11-01 11:12:33 [INFO] [TRAIN] epoch=803, iter=149200/160000, loss=0.4158, lr=0.004511, batch_cost=0.3066, reader_cost=0.1110 | ETA 00:55:11 2020-11-01 11:13:35 [INFO] [TRAIN] epoch=804, iter=149400/160000, loss=0.4118, lr=0.004437, batch_cost=0.3119, reader_cost=0.1644 | ETA 00:55:06 2020-11-01 11:14:37 [INFO] [TRAIN] epoch=805, iter=149600/160000, loss=0.4093, lr=0.004363, batch_cost=0.3115, reader_cost=0.1606 | ETA 00:53:59 2020-11-01 11:15:40 [INFO] [TRAIN] epoch=806, iter=149800/160000, loss=0.4186, lr=0.004290, batch_cost=0.3115, reader_cost=0.1721 | ETA 00:52:57 2020-11-01 11:16:42 [INFO] [TRAIN] epoch=807, iter=150000/160000, loss=0.4131, lr=0.004216, batch_cost=0.3124, reader_cost=0.1628 | ETA 00:52:04 2020-11-01 11:17:46 [INFO] [TRAIN] epoch=808, iter=150200/160000, loss=0.4035, lr=0.004141, batch_cost=0.3188, reader_cost=0.1775 | ETA 00:52:03 2020-11-01 11:18:48 [INFO] [TRAIN] epoch=809, iter=150400/160000, loss=0.4242, lr=0.004067, batch_cost=0.3115, reader_cost=0.1660 | ETA 00:49:50 2020-11-01 11:19:52 [INFO] [TRAIN] epoch=810, iter=150600/160000, loss=0.4180, lr=0.003993, batch_cost=0.3204, reader_cost=0.1512 | ETA 00:50:12 2020-11-01 11:20:55 [INFO] [TRAIN] epoch=811, iter=150800/160000, loss=0.4141, lr=0.003918, batch_cost=0.3136, reader_cost=0.1707 | ETA 00:48:04 2020-11-01 11:21:57 [INFO] [TRAIN] epoch=812, iter=151000/160000, loss=0.4045, lr=0.003843, batch_cost=0.3083, reader_cost=0.1516 | ETA 00:46:14 2020-11-01 11:23:00 [INFO] [TRAIN] epoch=813, iter=151200/160000, loss=0.4148, lr=0.003768, batch_cost=0.3169, reader_cost=0.1746 | ETA 00:46:28 2020-11-01 11:24:03 [INFO] [TRAIN] epoch=814, iter=151400/160000, loss=0.4188, lr=0.003693, batch_cost=0.3130, reader_cost=0.1516 | ETA 00:44:51 2020-11-01 11:25:06 [INFO] [TRAIN] epoch=816, iter=151600/160000, loss=0.4143, lr=0.003618, batch_cost=0.3189, reader_cost=0.1180 | ETA 00:44:38 2020-11-01 11:26:09 [INFO] [TRAIN] epoch=817, iter=151800/160000, loss=0.4073, lr=0.003542, batch_cost=0.3119, reader_cost=0.1650 | ETA 00:42:37 2020-11-01 11:27:11 [INFO] [TRAIN] epoch=818, iter=152000/160000, loss=0.4154, lr=0.003467, batch_cost=0.3118, reader_cost=0.1568 | ETA 00:41:34 2020-11-01 11:27:11 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-01 11:29:59 [INFO] [EVAL] #Images=500 mIoU=0.6931 Acc=0.9484 Kappa=0.9330 2020-11-01 11:29:59 [INFO] [EVAL] Category IoU: [0.975 0.8008 0.9038 0.4422 0.5139 0.5511 0.571 0.6858 0.9116 0.6031 0.9381 0.7404 0.4899 0.9253 0.6409 0.7458 0.6411 0.4083 0.6809] 2020-11-01 11:29:59 [INFO] [EVAL] Category Acc: [0.9851 0.8936 0.9429 0.8075 0.7853 0.755 0.7858 0.8444 0.9457 0.8115 0.9575 0.833 0.7061 0.9565 0.8567 0.871 0.8746 0.7409 0.7825] 2020-11-01 11:29:59 [INFO] [EVAL] The model with the best validation mIoU (0.6931) was saved at iter 152000. 2020-11-01 11:30:59 [INFO] [TRAIN] epoch=819, iter=152200/160000, loss=0.4117, lr=0.003391, batch_cost=0.2979, reader_cost=0.1162 | ETA 00:38:43 2020-11-01 11:32:01 [INFO] [TRAIN] epoch=820, iter=152400/160000, loss=0.4111, lr=0.003315, batch_cost=0.3104, reader_cost=0.1141 | ETA 00:39:18 2020-11-01 11:33:03 [INFO] [TRAIN] epoch=821, iter=152600/160000, loss=0.4145, lr=0.003239, batch_cost=0.3070, reader_cost=0.1606 | ETA 00:37:52 2020-11-01 11:34:05 [INFO] [TRAIN] epoch=822, iter=152800/160000, loss=0.4135, lr=0.003162, batch_cost=0.3112, reader_cost=0.1613 | ETA 00:37:20 2020-11-01 11:35:06 [INFO] [TRAIN] epoch=823, iter=153000/160000, loss=0.4179, lr=0.003086, batch_cost=0.3037, reader_cost=0.1428 | ETA 00:35:25 2020-11-01 11:36:08 [INFO] [TRAIN] epoch=824, iter=153200/160000, loss=0.4162, lr=0.003009, batch_cost=0.3127, reader_cost=0.1402 | ETA 00:35:26 2020-11-01 11:37:12 [INFO] [TRAIN] epoch=825, iter=153400/160000, loss=0.4045, lr=0.002932, batch_cost=0.3202, reader_cost=0.1788 | ETA 00:35:13 2020-11-01 11:38:16 [INFO] [TRAIN] epoch=826, iter=153600/160000, loss=0.4100, lr=0.002854, batch_cost=0.3203, reader_cost=0.1822 | ETA 00:34:09 2020-11-01 11:39:20 [INFO] [TRAIN] epoch=827, iter=153800/160000, loss=0.4071, lr=0.002777, batch_cost=0.3206, reader_cost=0.1836 | ETA 00:33:08 2020-11-01 11:40:25 [INFO] [TRAIN] epoch=828, iter=154000/160000, loss=0.4134, lr=0.002699, batch_cost=0.3219, reader_cost=0.1823 | ETA 00:32:11 2020-11-01 11:41:28 [INFO] [TRAIN] epoch=830, iter=154200/160000, loss=0.4110, lr=0.002621, batch_cost=0.3157, reader_cost=0.1712 | ETA 00:30:31 2020-11-01 11:42:32 [INFO] [TRAIN] epoch=831, iter=154400/160000, loss=0.4077, lr=0.002542, batch_cost=0.3192, reader_cost=0.1765 | ETA 00:29:47 2020-11-01 11:43:34 [INFO] [TRAIN] epoch=832, iter=154600/160000, loss=0.4139, lr=0.002464, batch_cost=0.3105, reader_cost=0.1689 | ETA 00:27:56 2020-11-01 11:44:36 [INFO] [TRAIN] epoch=833, iter=154800/160000, loss=0.4002, lr=0.002385, batch_cost=0.3094, reader_cost=0.1530 | ETA 00:26:48 2020-11-01 11:45:37 [INFO] [TRAIN] epoch=834, iter=155000/160000, loss=0.4129, lr=0.002306, batch_cost=0.3088, reader_cost=0.1511 | ETA 00:25:43 2020-11-01 11:46:39 [INFO] [TRAIN] epoch=835, iter=155200/160000, loss=0.4151, lr=0.002226, batch_cost=0.3076, reader_cost=0.1544 | ETA 00:24:36 2020-11-01 11:47:42 [INFO] [TRAIN] epoch=836, iter=155400/160000, loss=0.4047, lr=0.002146, batch_cost=0.3158, reader_cost=0.1713 | ETA 00:24:12 2020-11-01 11:48:44 [INFO] [TRAIN] epoch=837, iter=155600/160000, loss=0.4011, lr=0.002066, batch_cost=0.3087, reader_cost=0.1675 | ETA 00:22:38 2020-11-01 11:49:45 [INFO] [TRAIN] epoch=838, iter=155800/160000, loss=0.4003, lr=0.001985, batch_cost=0.3076, reader_cost=0.1629 | ETA 00:21:32 2020-11-01 11:50:47 [INFO] [TRAIN] epoch=839, iter=156000/160000, loss=0.4105, lr=0.001904, batch_cost=0.3062, reader_cost=0.1599 | ETA 00:20:24 2020-11-01 11:51:48 [INFO] [TRAIN] epoch=840, iter=156200/160000, loss=0.4130, lr=0.001823, batch_cost=0.3082, reader_cost=0.1555 | ETA 00:19:31 2020-11-01 11:52:51 [INFO] [TRAIN] epoch=841, iter=156400/160000, loss=0.4067, lr=0.001741, batch_cost=0.3114, reader_cost=0.1546 | ETA 00:18:40 2020-11-01 11:53:53 [INFO] [TRAIN] epoch=842, iter=156600/160000, loss=0.4023, lr=0.001659, batch_cost=0.3127, reader_cost=0.1690 | ETA 00:17:43 2020-11-01 11:54:56 [INFO] [TRAIN] epoch=844, iter=156800/160000, loss=0.4110, lr=0.001576, batch_cost=0.3132, reader_cost=0.1187 | ETA 00:16:42 2020-11-01 11:55:57 [INFO] [TRAIN] epoch=845, iter=157000/160000, loss=0.4057, lr=0.001493, batch_cost=0.3064, reader_cost=0.1502 | ETA 00:15:19 2020-11-01 11:56:59 [INFO] [TRAIN] epoch=846, iter=157200/160000, loss=0.4033, lr=0.001409, batch_cost=0.3102, reader_cost=0.1641 | ETA 00:14:28 2020-11-01 11:58:01 [INFO] [TRAIN] epoch=847, iter=157400/160000, loss=0.4127, lr=0.001325, batch_cost=0.3079, reader_cost=0.1374 | ETA 00:13:20 2020-11-01 11:59:05 [INFO] [TRAIN] epoch=848, iter=157600/160000, loss=0.4088, lr=0.001240, batch_cost=0.3202, reader_cost=0.1682 | ETA 00:12:48 2020-11-01 12:00:07 [INFO] [TRAIN] epoch=849, iter=157800/160000, loss=0.4092, lr=0.001154, batch_cost=0.3108, reader_cost=0.1623 | ETA 00:11:23 2020-11-01 12:01:11 [INFO] [TRAIN] epoch=850, iter=158000/160000, loss=0.4103, lr=0.001067, batch_cost=0.3213, reader_cost=0.1791 | ETA 00:10:42 2020-11-01 12:02:14 [INFO] [TRAIN] epoch=851, iter=158200/160000, loss=0.4122, lr=0.000980, batch_cost=0.3162, reader_cost=0.1742 | ETA 00:09:29 2020-11-01 12:03:18 [INFO] [TRAIN] epoch=852, iter=158400/160000, loss=0.4069, lr=0.000891, batch_cost=0.3202, reader_cost=0.1758 | ETA 00:08:32 2020-11-01 12:04:21 [INFO] [TRAIN] epoch=853, iter=158600/160000, loss=0.4055, lr=0.000802, batch_cost=0.3116, reader_cost=0.1729 | ETA 00:07:16 2020-11-01 12:05:23 [INFO] [TRAIN] epoch=854, iter=158800/160000, loss=0.4006, lr=0.000711, batch_cost=0.3123, reader_cost=0.1601 | ETA 00:06:14 2020-11-01 12:06:26 [INFO] [TRAIN] epoch=855, iter=159000/160000, loss=0.4056, lr=0.000619, batch_cost=0.3150, reader_cost=0.1655 | ETA 00:05:14 2020-11-01 12:07:29 [INFO] [TRAIN] epoch=856, iter=159200/160000, loss=0.4059, lr=0.000524, batch_cost=0.3151, reader_cost=0.1711 | ETA 00:04:12 2020-11-01 12:08:31 [INFO] [TRAIN] epoch=857, iter=159400/160000, loss=0.3913, lr=0.000428, batch_cost=0.3096, reader_cost=0.1643 | ETA 00:03:05 2020-11-01 12:09:34 [INFO] [TRAIN] epoch=859, iter=159600/160000, loss=0.4109, lr=0.000328, batch_cost=0.3166, reader_cost=0.1774 | ETA 00:02:06 2020-11-01 12:10:37 [INFO] [TRAIN] epoch=860, iter=159800/160000, loss=0.4133, lr=0.000222, batch_cost=0.3117, reader_cost=0.1564 | ETA 00:01:02 2020-11-01 12:11:39 [INFO] [TRAIN] epoch=861, iter=160000/160000, loss=0.3999, lr=0.000101, batch_cost=0.3096, reader_cost=0.1422 | ETA 00:00:00 2020-11-01 12:11:39 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-01 12:14:23 [INFO] [EVAL] #Images=500 mIoU=0.6922 Acc=0.9481 Kappa=0.9326 2020-11-01 12:14:23 [INFO] [EVAL] Category IoU: [0.9742 0.7963 0.9034 0.4268 0.497 0.5533 0.5722 0.6884 0.9118 0.6119 0.9392 0.7416 0.4909 0.9262 0.6625 0.7459 0.6056 0.4251 0.6799] 2020-11-01 12:14:23 [INFO] [EVAL] Category Acc: [0.9845 0.8946 0.9419 0.8289 0.7569 0.7539 0.7822 0.8586 0.9452 0.8149 0.96 0.8327 0.7184 0.9573 0.8744 0.8638 0.8841 0.7231 0.7825] 2020-11-01 12:14:23 [INFO] [EVAL] The model with the best validation mIoU (0.6931) was saved at iter 152000.