2020-12-24 15:36:22 [INFO] ------------Environment Information------------- platform: Linux-3.10.0-1062.18.1.el7.x86_64-x86_64-with-centos-7.7.1908-Core Python: 3.7.4 (default, Aug 13 2019, 20:35:49) [GCC 7.3.0] Paddle compiled with cuda: True NVCC: Cuda compilation tools, release 10.2, V10.2.89 cudnn: 7.6 GPUs used: 4 CUDA_VISIBLE_DEVICES: 0,1,2,3 GPU: ['GPU 0: Tesla V100-SXM2-16GB', 'GPU 1: Tesla V100-SXM2-16GB', 'GPU 2: Tesla V100-SXM2-16GB', 'GPU 3: Tesla V100-SXM2-16GB', 'GPU 4: Tesla V100-SXM2-16GB', 'GPU 5: Tesla V100-SXM2-16GB', 'GPU 6: Tesla V100-SXM2-16GB', 'GPU 7: Tesla V100-SXM2-16GB'] GCC: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39) PaddlePaddle: 2.0.0-rc0 OpenCV: 4.2.0 ------------------------------------------------ 2020-12-24 15:36:22 [INFO] ---------------Config Information--------------- batch_size: 2 iters: 80000 learning_rate: decay: end_lr: 0.0 power: 0.9 type: poly value: 0.01 loss: coef: - 1 - 1 - 25 - 1 types: - ignore_index: 255 type: OhemCrossEntropyLoss - ignore_index: 255 type: RelaxBoundaryLoss - edge_label: true ignore_index: 255 type: BCELoss weight: dynamic - ignore_index: 255 type: OhemEdgeAttentionLoss model: align_corners: false aspp_out_channels: 256 aspp_ratios: - 1 - 12 - 24 - 36 backbone: multi_grid: - 1 - 2 - 4 output_stride: 8 pretrained: https://bj.bcebos.com/paddleseg/dygraph/resnet50_vd_ssld_v2.tar.gz type: ResNet50_vd backbone_indices: - 0 - 3 num_classes: 19 pretrained: null type: DecoupledSegNet optimizer: momentum: 0.9 type: sgd weight_decay: 4.0e-05 train_dataset: dataset_root: data/cityscapes edge: true mode: train transforms: - max_scale_factor: 2.0 min_scale_factor: 0.5 scale_step_size: 0.25 type: ResizeStepScaling - crop_size: - 1024 - 512 type: RandomPaddingCrop - type: RandomHorizontalFlip - brightness_range: 0.4 contrast_range: 0.4 saturation_range: 0.4 type: RandomDistort - type: Normalize type: Cityscapes val_dataset: dataset_root: data/cityscapes mode: val transforms: - type: Normalize type: Cityscapes ------------------------------------------------ 2020-12-24 15:36:26 [INFO] Loading pretrained model from https://bj.bcebos.com/paddleseg/dygraph/resnet50_vd_ssld_v2.tar.gz 2020-12-24 15:36:27 [INFO] There are 275/275 variables loaded into ResNet_vd. 2020-12-24 15:38:44 [INFO] [TRAIN] epoch=1, iter=100/80000, loss=19.9940, lr=0.009989, batch_cost=1.2981, reader_cost=0.0833 | ETA 28:48:35 2020-12-24 15:40:47 [INFO] [TRAIN] epoch=1, iter=200/80000, loss=18.5166, lr=0.009978, batch_cost=1.2272, reader_cost=0.0004 | ETA 27:12:13 2020-12-24 15:42:50 [INFO] [TRAIN] epoch=1, iter=300/80000, loss=18.5074, lr=0.009966, batch_cost=1.2328, reader_cost=0.0003 | ETA 27:17:35 2020-12-24 15:44:58 [INFO] [TRAIN] epoch=2, iter=400/80000, loss=18.7428, lr=0.009955, batch_cost=1.2703, reader_cost=0.0498 | ETA 28:05:18 2020-12-24 15:46:59 [INFO] [TRAIN] epoch=2, iter=500/80000, loss=18.0813, lr=0.009944, batch_cost=1.2133, reader_cost=0.0003 | ETA 26:47:40 2020-12-24 15:49:01 [INFO] [TRAIN] epoch=2, iter=600/80000, loss=17.4069, lr=0.009933, batch_cost=1.2161, reader_cost=0.0021 | ETA 26:49:20 2020-12-24 15:51:02 [INFO] [TRAIN] epoch=2, iter=700/80000, loss=17.6600, lr=0.009921, batch_cost=1.2127, reader_cost=0.0002 | ETA 26:42:46 2020-12-24 15:53:09 [INFO] [TRAIN] epoch=3, iter=800/80000, loss=18.2568, lr=0.009910, batch_cost=1.2593, reader_cost=0.0700 | ETA 27:42:16 2020-12-24 15:55:10 [INFO] [TRAIN] epoch=3, iter=900/80000, loss=17.2942, lr=0.009899, batch_cost=1.2163, reader_cost=0.0015 | ETA 26:43:28 2020-12-24 15:57:13 [INFO] [TRAIN] epoch=3, iter=1000/80000, loss=17.4439, lr=0.009888, batch_cost=1.2239, reader_cost=0.0003 | ETA 26:51:28 2020-12-24 15:59:13 [INFO] [TRAIN] epoch=3, iter=1100/80000, loss=17.7578, lr=0.009876, batch_cost=1.1954, reader_cost=0.0002 | ETA 26:11:56 2020-12-24 16:01:16 [INFO] [TRAIN] epoch=4, iter=1200/80000, loss=18.0485, lr=0.009865, batch_cost=1.2352, reader_cost=0.0419 | ETA 27:02:09 2020-12-24 16:03:18 [INFO] [TRAIN] epoch=4, iter=1300/80000, loss=16.9143, lr=0.009854, batch_cost=1.2168, reader_cost=0.0002 | ETA 26:36:01 2020-12-24 16:05:15 [INFO] [TRAIN] epoch=4, iter=1400/80000, loss=17.3655, lr=0.009842, batch_cost=1.1636, reader_cost=0.0002 | ETA 25:24:21 2020-12-24 16:07:19 [INFO] [TRAIN] epoch=5, iter=1500/80000, loss=17.5689, lr=0.009831, batch_cost=1.2385, reader_cost=0.0429 | ETA 27:00:20 2020-12-24 16:09:20 [INFO] [TRAIN] epoch=5, iter=1600/80000, loss=17.4102, lr=0.009820, batch_cost=1.2038, reader_cost=0.0092 | ETA 26:12:55 2020-12-24 16:11:21 [INFO] [TRAIN] epoch=5, iter=1700/80000, loss=17.2159, lr=0.009809, batch_cost=1.2116, reader_cost=0.0004 | ETA 26:21:04 2020-12-24 16:13:28 [INFO] [TRAIN] epoch=5, iter=1800/80000, loss=17.1905, lr=0.009797, batch_cost=1.2688, reader_cost=0.0002 | ETA 27:33:39 2020-12-24 16:15:31 [INFO] [TRAIN] epoch=6, iter=1900/80000, loss=17.3008, lr=0.009786, batch_cost=1.2214, reader_cost=0.0446 | ETA 26:29:54 2020-12-24 16:17:32 [INFO] [TRAIN] epoch=6, iter=2000/80000, loss=16.8300, lr=0.009775, batch_cost=1.2133, reader_cost=0.0002 | ETA 26:17:19 2020-12-24 16:17:32 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-24 16:18:16 [INFO] [EVAL] #Images=500 mIoU=0.5427 Acc=0.9239 Kappa=0.9009 2020-12-24 16:18:16 [INFO] [EVAL] Class IoU: [0.9505 0.6605 0.8843 0.2427 0.3826 0.5236 0.5624 0.6798 0.8859 0.4876 0.8948 0.7294 0.3844 0.8673 0.0094 0.2953 0.0709 0.1279 0.6721] 2020-12-24 16:18:16 [INFO] [EVAL] Class Acc: [0.9642 0.92 0.9362 0.477 0.47 0.8106 0.7625 0.826 0.9438 0.5993 0.9091 0.8207 0.6599 0.8827 0.4005 0.5499 0.3453 0.5266 0.72 ] 2020-12-24 16:18:23 [INFO] [EVAL] The model with the best validation mIoU (0.5427) was saved at iter 2000. 2020-12-24 16:20:23 [INFO] [TRAIN] epoch=6, iter=2100/80000, loss=16.6396, lr=0.009764, batch_cost=1.2020, reader_cost=0.0002 | ETA 26:00:38 2020-12-24 16:22:24 [INFO] [TRAIN] epoch=6, iter=2200/80000, loss=17.1820, lr=0.009752, batch_cost=1.2092, reader_cost=0.0002 | ETA 26:07:59 2020-12-24 16:24:28 [INFO] [TRAIN] epoch=7, iter=2300/80000, loss=17.4498, lr=0.009741, batch_cost=1.2299, reader_cost=0.0556 | ETA 26:32:43 2020-12-24 16:26:27 [INFO] [TRAIN] epoch=7, iter=2400/80000, loss=16.6353, lr=0.009730, batch_cost=1.1880, reader_cost=0.0016 | ETA 25:36:30 2020-12-24 16:28:28 [INFO] [TRAIN] epoch=7, iter=2500/80000, loss=16.7898, lr=0.009718, batch_cost=1.2116, reader_cost=0.0014 | ETA 26:04:56 2020-12-24 16:30:29 [INFO] [TRAIN] epoch=7, iter=2600/80000, loss=17.2913, lr=0.009707, batch_cost=1.2039, reader_cost=0.0008 | ETA 25:53:03 2020-12-24 16:32:37 [INFO] [TRAIN] epoch=8, iter=2700/80000, loss=17.4123, lr=0.009696, batch_cost=1.2811, reader_cost=0.0421 | ETA 27:30:32 2020-12-24 16:34:42 [INFO] [TRAIN] epoch=8, iter=2800/80000, loss=17.2750, lr=0.009685, batch_cost=1.2432, reader_cost=0.0005 | ETA 26:39:32 2020-12-24 16:36:46 [INFO] [TRAIN] epoch=8, iter=2900/80000, loss=17.0227, lr=0.009673, batch_cost=1.2446, reader_cost=0.0222 | ETA 26:39:16 2020-12-24 16:38:51 [INFO] [TRAIN] epoch=9, iter=3000/80000, loss=17.5270, lr=0.009662, batch_cost=1.2490, reader_cost=0.0429 | ETA 26:42:51 2020-12-24 16:40:52 [INFO] [TRAIN] epoch=9, iter=3100/80000, loss=17.2850, lr=0.009651, batch_cost=1.1997, reader_cost=0.0003 | ETA 25:37:40 2020-12-24 16:42:51 [INFO] [TRAIN] epoch=9, iter=3200/80000, loss=16.9125, lr=0.009639, batch_cost=1.1925, reader_cost=0.0007 | ETA 25:26:24 2020-12-24 16:44:54 [INFO] [TRAIN] epoch=9, iter=3300/80000, loss=16.8127, lr=0.009628, batch_cost=1.2315, reader_cost=0.0003 | ETA 26:14:12 2020-12-24 16:47:00 [INFO] [TRAIN] epoch=10, iter=3400/80000, loss=17.2847, lr=0.009617, batch_cost=1.2521, reader_cost=0.0485 | ETA 26:38:27 2020-12-24 16:48:58 [INFO] [TRAIN] epoch=10, iter=3500/80000, loss=16.7486, lr=0.009605, batch_cost=1.1813, reader_cost=0.0014 | ETA 25:06:09 2020-12-24 16:50:58 [INFO] [TRAIN] epoch=10, iter=3600/80000, loss=16.5657, lr=0.009594, batch_cost=1.2007, reader_cost=0.0002 | ETA 25:28:50 2020-12-24 16:53:01 [INFO] [TRAIN] epoch=10, iter=3700/80000, loss=16.9382, lr=0.009583, batch_cost=1.2204, reader_cost=0.0011 | ETA 25:51:56 2020-12-24 16:55:03 [INFO] [TRAIN] epoch=11, iter=3800/80000, loss=17.1440, lr=0.009572, batch_cost=1.2212, reader_cost=0.0400 | ETA 25:50:55 2020-12-24 16:57:04 [INFO] [TRAIN] epoch=11, iter=3900/80000, loss=16.6871, lr=0.009560, batch_cost=1.2022, reader_cost=0.0183 | ETA 25:24:50 2020-12-24 16:59:08 [INFO] [TRAIN] epoch=11, iter=4000/80000, loss=17.0366, lr=0.009549, batch_cost=1.2391, reader_cost=0.0004 | ETA 26:09:28 2020-12-24 16:59:08 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-24 16:59:52 [INFO] [EVAL] #Images=500 mIoU=0.6208 Acc=0.9383 Kappa=0.9196 2020-12-24 16:59:52 [INFO] [EVAL] Class IoU: [0.961 0.7444 0.8952 0.2718 0.4202 0.5666 0.5105 0.6887 0.9014 0.519 0.9289 0.7639 0.512 0.9082 0.2431 0.5616 0.388 0.3249 0.6854] 2020-12-24 16:59:52 [INFO] [EVAL] Class Acc: [0.9729 0.8657 0.9296 0.8185 0.7198 0.7847 0.885 0.8813 0.9508 0.7394 0.9607 0.8935 0.6592 0.932 0.7452 0.7495 0.5852 0.5252 0.726 ] 2020-12-24 16:59:59 [INFO] [EVAL] The model with the best validation mIoU (0.6208) was saved at iter 4000. 2020-12-24 17:02:07 [INFO] [TRAIN] epoch=12, iter=4100/80000, loss=17.3193, lr=0.009538, batch_cost=1.2744, reader_cost=0.0448 | ETA 26:52:06 2020-12-24 17:04:09 [INFO] [TRAIN] epoch=12, iter=4200/80000, loss=17.4011, lr=0.009526, batch_cost=1.2139, reader_cost=0.0002 | ETA 25:33:29 2020-12-24 17:06:08 [INFO] [TRAIN] epoch=12, iter=4300/80000, loss=16.8604, lr=0.009515, batch_cost=1.1846, reader_cost=0.0012 | ETA 24:54:34 2020-12-24 17:08:06 [INFO] [TRAIN] epoch=12, iter=4400/80000, loss=16.9010, lr=0.009504, batch_cost=1.1825, reader_cost=0.0002 | ETA 24:49:54 2020-12-24 17:10:10 [INFO] [TRAIN] epoch=13, iter=4500/80000, loss=16.9690, lr=0.009492, batch_cost=1.2416, reader_cost=0.0429 | ETA 26:02:18 2020-12-24 17:12:12 [INFO] [TRAIN] epoch=13, iter=4600/80000, loss=16.8477, lr=0.009481, batch_cost=1.2095, reader_cost=0.0017 | ETA 25:19:59 2020-12-24 17:14:12 [INFO] [TRAIN] epoch=13, iter=4700/80000, loss=16.6942, lr=0.009470, batch_cost=1.1974, reader_cost=0.0002 | ETA 25:02:43 2020-12-24 17:16:13 [INFO] [TRAIN] epoch=13, iter=4800/80000, loss=16.7559, lr=0.009458, batch_cost=1.2081, reader_cost=0.0024 | ETA 25:14:08 2020-12-24 17:18:24 [INFO] [TRAIN] epoch=14, iter=4900/80000, loss=17.1376, lr=0.009447, batch_cost=1.3143, reader_cost=0.0439 | ETA 27:25:06 2020-12-24 17:20:30 [INFO] [TRAIN] epoch=14, iter=5000/80000, loss=16.5804, lr=0.009436, batch_cost=1.2596, reader_cost=0.0004 | ETA 26:14:32 2020-12-24 17:22:32 [INFO] [TRAIN] epoch=14, iter=5100/80000, loss=16.4632, lr=0.009424, batch_cost=1.2118, reader_cost=0.0006 | ETA 25:12:42 2020-12-24 17:24:31 [INFO] [TRAIN] epoch=14, iter=5200/80000, loss=16.7134, lr=0.009413, batch_cost=1.1913, reader_cost=0.0002 | ETA 24:45:05 2020-12-24 17:26:41 [INFO] [TRAIN] epoch=15, iter=5300/80000, loss=16.8241, lr=0.009402, batch_cost=1.3026, reader_cost=0.0450 | ETA 27:01:43 2020-12-24 17:28:42 [INFO] [TRAIN] epoch=15, iter=5400/80000, loss=16.4957, lr=0.009391, batch_cost=1.2047, reader_cost=0.0002 | ETA 24:57:52 2020-12-24 17:30:42 [INFO] [TRAIN] epoch=15, iter=5500/80000, loss=16.6918, lr=0.009379, batch_cost=1.1978, reader_cost=0.0006 | ETA 24:47:12 2020-12-24 17:32:53 [INFO] [TRAIN] epoch=16, iter=5600/80000, loss=16.7922, lr=0.009368, batch_cost=1.3020, reader_cost=0.0545 | ETA 26:54:28 2020-12-24 17:34:57 [INFO] [TRAIN] epoch=16, iter=5700/80000, loss=16.5098, lr=0.009357, batch_cost=1.2450, reader_cost=0.0082 | ETA 25:41:46 2020-12-24 17:36:57 [INFO] [TRAIN] epoch=16, iter=5800/80000, loss=16.4977, lr=0.009345, batch_cost=1.1973, reader_cost=0.0002 | ETA 24:40:40 2020-12-24 17:39:00 [INFO] [TRAIN] epoch=16, iter=5900/80000, loss=16.5182, lr=0.009334, batch_cost=1.2203, reader_cost=0.0005 | ETA 25:07:01 2020-12-24 17:41:09 [INFO] [TRAIN] epoch=17, iter=6000/80000, loss=16.9478, lr=0.009323, batch_cost=1.2853, reader_cost=0.0517 | ETA 26:25:13 2020-12-24 17:41:09 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-24 17:41:52 [INFO] [EVAL] #Images=500 mIoU=0.6513 Acc=0.9413 Kappa=0.9235 2020-12-24 17:41:52 [INFO] [EVAL] Class IoU: [0.9602 0.7959 0.8962 0.3288 0.484 0.5939 0.5779 0.7024 0.9003 0.5648 0.8969 0.7805 0.5333 0.917 0.4297 0.4931 0.392 0.422 0.7064] 2020-12-24 17:41:52 [INFO] [EVAL] Class Acc: [0.9706 0.8897 0.9337 0.834 0.7068 0.8143 0.9012 0.9095 0.9462 0.7832 0.9753 0.8827 0.7494 0.9527 0.598 0.8527 0.5336 0.6796 0.7641] 2020-12-24 17:42:00 [INFO] [EVAL] The model with the best validation mIoU (0.6513) was saved at iter 6000. 2020-12-24 17:43:58 [INFO] [TRAIN] epoch=17, iter=6100/80000, loss=16.6890, lr=0.009311, batch_cost=1.1852, reader_cost=0.0004 | ETA 24:19:47 2020-12-24 17:46:01 [INFO] [TRAIN] epoch=17, iter=6200/80000, loss=16.4852, lr=0.009300, batch_cost=1.2276, reader_cost=0.0005 | ETA 25:09:54 2020-12-24 17:47:58 [INFO] [TRAIN] epoch=17, iter=6300/80000, loss=16.7277, lr=0.009288, batch_cost=1.1657, reader_cost=0.0002 | ETA 23:51:53 2020-12-24 17:50:03 [INFO] [TRAIN] epoch=18, iter=6400/80000, loss=17.0379, lr=0.009277, batch_cost=1.2502, reader_cost=0.0389 | ETA 25:33:34 2020-12-24 17:52:07 [INFO] [TRAIN] epoch=18, iter=6500/80000, loss=16.4305, lr=0.009266, batch_cost=1.2330, reader_cost=0.0002 | ETA 25:10:23 2020-12-24 17:54:14 [INFO] [TRAIN] epoch=18, iter=6600/80000, loss=16.6745, lr=0.009254, batch_cost=1.2706, reader_cost=0.0002 | ETA 25:54:22 2020-12-24 17:56:18 [INFO] [TRAIN] epoch=19, iter=6700/80000, loss=17.1008, lr=0.009243, batch_cost=1.2379, reader_cost=0.0423 | ETA 25:12:16 2020-12-24 17:58:19 [INFO] [TRAIN] epoch=19, iter=6800/80000, loss=16.8769, lr=0.009232, batch_cost=1.2064, reader_cost=0.0196 | ETA 24:31:46 2020-12-24 18:00:17 [INFO] [TRAIN] epoch=19, iter=6900/80000, loss=16.8497, lr=0.009220, batch_cost=1.1808, reader_cost=0.0007 | ETA 23:58:36 2020-12-24 18:02:21 [INFO] [TRAIN] epoch=19, iter=7000/80000, loss=16.7133, lr=0.009209, batch_cost=1.2320, reader_cost=0.0002 | ETA 24:58:59 2020-12-24 18:04:29 [INFO] [TRAIN] epoch=20, iter=7100/80000, loss=16.8895, lr=0.009198, batch_cost=1.2816, reader_cost=0.0398 | ETA 25:57:06 2020-12-24 18:06:28 [INFO] [TRAIN] epoch=20, iter=7200/80000, loss=16.4342, lr=0.009186, batch_cost=1.1869, reader_cost=0.0002 | ETA 24:00:07 2020-12-24 18:08:29 [INFO] [TRAIN] epoch=20, iter=7300/80000, loss=16.1909, lr=0.009175, batch_cost=1.2126, reader_cost=0.0002 | ETA 24:29:16 2020-12-24 18:10:28 [INFO] [TRAIN] epoch=20, iter=7400/80000, loss=16.6146, lr=0.009164, batch_cost=1.1854, reader_cost=0.0002 | ETA 23:54:18 2020-12-24 18:12:37 [INFO] [TRAIN] epoch=21, iter=7500/80000, loss=17.0520, lr=0.009152, batch_cost=1.2847, reader_cost=0.0422 | ETA 25:52:21 2020-12-24 18:14:37 [INFO] [TRAIN] epoch=21, iter=7600/80000, loss=16.3354, lr=0.009141, batch_cost=1.1990, reader_cost=0.0006 | ETA 24:06:44 2020-12-24 18:16:39 [INFO] [TRAIN] epoch=21, iter=7700/80000, loss=15.8580, lr=0.009130, batch_cost=1.2212, reader_cost=0.0002 | ETA 24:31:34 2020-12-24 18:18:38 [INFO] [TRAIN] epoch=21, iter=7800/80000, loss=16.6401, lr=0.009118, batch_cost=1.1886, reader_cost=0.0002 | ETA 23:50:13 2020-12-24 18:20:46 [INFO] [TRAIN] epoch=22, iter=7900/80000, loss=17.0453, lr=0.009107, batch_cost=1.2672, reader_cost=0.0382 | ETA 25:22:47 2020-12-24 18:22:42 [INFO] [TRAIN] epoch=22, iter=8000/80000, loss=16.2543, lr=0.009095, batch_cost=1.1587, reader_cost=0.0003 | ETA 23:10:26 2020-12-24 18:22:42 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-24 18:23:25 [INFO] [EVAL] #Images=500 mIoU=0.6772 Acc=0.9453 Kappa=0.9288 2020-12-24 18:23:25 [INFO] [EVAL] Class IoU: [0.9703 0.7711 0.8961 0.3488 0.5243 0.5953 0.6193 0.7521 0.9068 0.5516 0.9301 0.7984 0.5591 0.9182 0.4948 0.5296 0.4163 0.5437 0.7405] 2020-12-24 18:23:25 [INFO] [EVAL] Class Acc: [0.9806 0.9015 0.9211 0.7777 0.7598 0.8386 0.8801 0.9003 0.9524 0.7853 0.9646 0.8876 0.8157 0.9413 0.79 0.8854 0.9055 0.7105 0.8216] 2020-12-24 18:23:33 [INFO] [EVAL] The model with the best validation mIoU (0.6772) was saved at iter 8000. 2020-12-24 18:25:36 [INFO] [TRAIN] epoch=22, iter=8100/80000, loss=16.4684, lr=0.009084, batch_cost=1.2284, reader_cost=0.0004 | ETA 24:31:59 2020-12-24 18:27:39 [INFO] [TRAIN] epoch=23, iter=8200/80000, loss=16.6976, lr=0.009073, batch_cost=1.2275, reader_cost=0.0430 | ETA 24:28:55 2020-12-24 18:29:44 [INFO] [TRAIN] epoch=23, iter=8300/80000, loss=16.6946, lr=0.009061, batch_cost=1.2494, reader_cost=0.0006 | ETA 24:52:59 2020-12-24 18:31:48 [INFO] [TRAIN] epoch=23, iter=8400/80000, loss=16.6411, lr=0.009050, batch_cost=1.2343, reader_cost=0.0080 | ETA 24:32:55 2020-12-24 18:33:45 [INFO] [TRAIN] epoch=23, iter=8500/80000, loss=16.8108, lr=0.009039, batch_cost=1.1693, reader_cost=0.0002 | ETA 23:13:22 2020-12-24 18:35:50 [INFO] [TRAIN] epoch=24, iter=8600/80000, loss=16.7628, lr=0.009027, batch_cost=1.2436, reader_cost=0.0473 | ETA 24:39:50 2020-12-24 18:37:51 [INFO] [TRAIN] epoch=24, iter=8700/80000, loss=16.7028, lr=0.009016, batch_cost=1.2089, reader_cost=0.0002 | ETA 23:56:32 2020-12-24 18:39:49 [INFO] [TRAIN] epoch=24, iter=8800/80000, loss=16.1488, lr=0.009004, batch_cost=1.1797, reader_cost=0.0005 | ETA 23:19:53 2020-12-24 18:41:53 [INFO] [TRAIN] epoch=24, iter=8900/80000, loss=16.7367, lr=0.008993, batch_cost=1.2377, reader_cost=0.0002 | ETA 24:26:37 2020-12-24 18:44:02 [INFO] [TRAIN] epoch=25, iter=9000/80000, loss=17.2225, lr=0.008982, batch_cost=1.2923, reader_cost=0.0388 | ETA 25:29:14 2020-12-24 18:46:00 [INFO] [TRAIN] epoch=25, iter=9100/80000, loss=16.4865, lr=0.008970, batch_cost=1.1732, reader_cost=0.0028 | ETA 23:06:16 2020-12-24 18:48:01 [INFO] [TRAIN] epoch=25, iter=9200/80000, loss=16.4713, lr=0.008959, batch_cost=1.2113, reader_cost=0.0002 | ETA 23:49:22 2020-12-24 18:49:58 [INFO] [TRAIN] epoch=25, iter=9300/80000, loss=16.7602, lr=0.008948, batch_cost=1.1669, reader_cost=0.0004 | ETA 22:55:01 2020-12-24 18:52:11 [INFO] [TRAIN] epoch=26, iter=9400/80000, loss=16.7759, lr=0.008936, batch_cost=1.3260, reader_cost=0.0654 | ETA 26:00:16 2020-12-24 18:54:10 [INFO] [TRAIN] epoch=26, iter=9500/80000, loss=16.5415, lr=0.008925, batch_cost=1.1888, reader_cost=0.0002 | ETA 23:16:53 2020-12-24 18:56:08 [INFO] [TRAIN] epoch=26, iter=9600/80000, loss=16.5671, lr=0.008913, batch_cost=1.1817, reader_cost=0.0005 | ETA 23:06:29 2020-12-24 18:58:18 [INFO] [TRAIN] epoch=27, iter=9700/80000, loss=17.1819, lr=0.008902, batch_cost=1.2886, reader_cost=0.0393 | ETA 25:09:49 2020-12-24 19:00:25 [INFO] [TRAIN] epoch=27, iter=9800/80000, loss=16.6610, lr=0.008891, batch_cost=1.2724, reader_cost=0.0002 | ETA 24:48:42 2020-12-24 19:02:25 [INFO] [TRAIN] epoch=27, iter=9900/80000, loss=16.6401, lr=0.008879, batch_cost=1.2001, reader_cost=0.0002 | ETA 23:22:07 2020-12-24 19:04:29 [INFO] [TRAIN] epoch=27, iter=10000/80000, loss=16.5490, lr=0.008868, batch_cost=1.2369, reader_cost=0.0002 | ETA 24:03:02 2020-12-24 19:04:29 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-24 19:05:12 [INFO] [EVAL] #Images=500 mIoU=0.6539 Acc=0.9436 Kappa=0.9268 2020-12-24 19:05:12 [INFO] [EVAL] Class IoU: [0.9675 0.7705 0.9013 0.3228 0.5356 0.5993 0.6564 0.7312 0.9131 0.5515 0.9391 0.7168 0.4708 0.9233 0.4261 0.523 0.4163 0.3664 0.6924] 2020-12-24 19:05:12 [INFO] [EVAL] Class Acc: [0.9877 0.8584 0.9384 0.7834 0.7275 0.8558 0.8439 0.9218 0.9471 0.797 0.9633 0.8072 0.5119 0.946 0.4639 0.8004 0.7913 0.4625 0.8718] 2020-12-24 19:05:17 [INFO] [EVAL] The model with the best validation mIoU (0.6772) was saved at iter 8000. 2020-12-24 19:07:22 [INFO] [TRAIN] epoch=28, iter=10100/80000, loss=17.0695, lr=0.008856, batch_cost=1.2452, reader_cost=0.0410 | ETA 24:10:36 2020-12-24 19:09:18 [INFO] [TRAIN] epoch=28, iter=10200/80000, loss=16.6378, lr=0.008845, batch_cost=1.1607, reader_cost=0.0003 | ETA 22:30:17 2020-12-24 19:11:20 [INFO] [TRAIN] epoch=28, iter=10300/80000, loss=16.5045, lr=0.008834, batch_cost=1.2156, reader_cost=0.0004 | ETA 23:32:06 2020-12-24 19:13:18 [INFO] [TRAIN] epoch=28, iter=10400/80000, loss=16.9793, lr=0.008822, batch_cost=1.1825, reader_cost=0.0002 | ETA 22:51:39 2020-12-24 19:15:23 [INFO] [TRAIN] epoch=29, iter=10500/80000, loss=16.9723, lr=0.008811, batch_cost=1.2469, reader_cost=0.0432 | ETA 24:04:18 2020-12-24 19:17:24 [INFO] [TRAIN] epoch=29, iter=10600/80000, loss=16.2618, lr=0.008799, batch_cost=1.2021, reader_cost=0.0002 | ETA 23:10:26 2020-12-24 19:19:27 [INFO] [TRAIN] epoch=29, iter=10700/80000, loss=16.4549, lr=0.008788, batch_cost=1.2258, reader_cost=0.0026 | ETA 23:35:45 2020-12-24 19:21:33 [INFO] [TRAIN] epoch=30, iter=10800/80000, loss=16.4737, lr=0.008776, batch_cost=1.2621, reader_cost=0.0516 | ETA 24:15:37 2020-12-24 19:23:33 [INFO] [TRAIN] epoch=30, iter=10900/80000, loss=16.6302, lr=0.008765, batch_cost=1.1963, reader_cost=0.0004 | ETA 22:57:43 2020-12-24 19:25:33 [INFO] [TRAIN] epoch=30, iter=11000/80000, loss=16.4130, lr=0.008754, batch_cost=1.1971, reader_cost=0.0003 | ETA 22:56:41 2020-12-24 19:27:32 [INFO] [TRAIN] epoch=30, iter=11100/80000, loss=16.4237, lr=0.008742, batch_cost=1.1939, reader_cost=0.0006 | ETA 22:50:59 2020-12-24 19:29:42 [INFO] [TRAIN] epoch=31, iter=11200/80000, loss=16.9500, lr=0.008731, batch_cost=1.2966, reader_cost=0.0473 | ETA 24:46:47 2020-12-24 19:31:45 [INFO] [TRAIN] epoch=31, iter=11300/80000, loss=16.4048, lr=0.008719, batch_cost=1.2219, reader_cost=0.0023 | ETA 23:19:06 2020-12-24 19:33:43 [INFO] [TRAIN] epoch=31, iter=11400/80000, loss=16.2604, lr=0.008708, batch_cost=1.1762, reader_cost=0.0005 | ETA 22:24:44 2020-12-24 19:35:45 [INFO] [TRAIN] epoch=31, iter=11500/80000, loss=16.3703, lr=0.008697, batch_cost=1.2186, reader_cost=0.0004 | ETA 23:11:12 2020-12-24 19:37:54 [INFO] [TRAIN] epoch=32, iter=11600/80000, loss=16.9737, lr=0.008685, batch_cost=1.2967, reader_cost=0.0532 | ETA 24:38:11 2020-12-24 19:39:52 [INFO] [TRAIN] epoch=32, iter=11700/80000, loss=16.4833, lr=0.008674, batch_cost=1.1714, reader_cost=0.0002 | ETA 22:13:28 2020-12-24 19:41:54 [INFO] [TRAIN] epoch=32, iter=11800/80000, loss=16.2267, lr=0.008662, batch_cost=1.2209, reader_cost=0.0002 | ETA 23:07:42 2020-12-24 19:43:57 [INFO] [TRAIN] epoch=32, iter=11900/80000, loss=16.9327, lr=0.008651, batch_cost=1.2239, reader_cost=0.0002 | ETA 23:09:07 2020-12-24 19:46:06 [INFO] [TRAIN] epoch=33, iter=12000/80000, loss=16.9654, lr=0.008639, batch_cost=1.2918, reader_cost=0.0370 | ETA 24:24:04 2020-12-24 19:46:06 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-24 19:46:50 [INFO] [EVAL] #Images=500 mIoU=0.6825 Acc=0.9519 Kappa=0.9376 2020-12-24 19:46:50 [INFO] [EVAL] Class IoU: [0.9763 0.8214 0.9127 0.4064 0.5755 0.6263 0.6995 0.7615 0.9168 0.6302 0.9359 0.8099 0.5454 0.9236 0.3925 0.5765 0.205 0.4986 0.7536] 2020-12-24 19:46:50 [INFO] [EVAL] Class Acc: [0.9897 0.8926 0.9562 0.8051 0.7183 0.8471 0.8524 0.9275 0.9433 0.8105 0.9601 0.8776 0.8396 0.9391 0.5622 0.768 0.9754 0.5623 0.8283] 2020-12-24 19:46:57 [INFO] [EVAL] The model with the best validation mIoU (0.6825) was saved at iter 12000. 2020-12-24 19:48:56 [INFO] [TRAIN] epoch=33, iter=12100/80000, loss=16.6074, lr=0.008628, batch_cost=1.1904, reader_cost=0.0002 | ETA 22:27:05 2020-12-24 19:50:54 [INFO] [TRAIN] epoch=33, iter=12200/80000, loss=16.6374, lr=0.008617, batch_cost=1.1777, reader_cost=0.0004 | ETA 22:10:49 2020-12-24 19:53:01 [INFO] [TRAIN] epoch=34, iter=12300/80000, loss=17.0951, lr=0.008605, batch_cost=1.2655, reader_cost=0.0374 | ETA 23:47:53 2020-12-24 19:54:58 [INFO] [TRAIN] epoch=34, iter=12400/80000, loss=16.6546, lr=0.008594, batch_cost=1.1615, reader_cost=0.0010 | ETA 21:48:38 2020-12-24 19:57:02 [INFO] [TRAIN] epoch=34, iter=12500/80000, loss=16.4046, lr=0.008582, batch_cost=1.2447, reader_cost=0.0005 | ETA 23:20:19 2020-12-24 19:59:04 [INFO] [TRAIN] epoch=34, iter=12600/80000, loss=16.2397, lr=0.008571, batch_cost=1.2165, reader_cost=0.0004 | ETA 22:46:32 2020-12-24 20:01:11 [INFO] [TRAIN] epoch=35, iter=12700/80000, loss=16.7361, lr=0.008559, batch_cost=1.2650, reader_cost=0.0430 | ETA 23:38:57 2020-12-24 20:03:15 [INFO] [TRAIN] epoch=35, iter=12800/80000, loss=16.0887, lr=0.008548, batch_cost=1.2373, reader_cost=0.0002 | ETA 23:05:49 2020-12-24 20:05:16 [INFO] [TRAIN] epoch=35, iter=12900/80000, loss=16.0948, lr=0.008536, batch_cost=1.2125, reader_cost=0.0002 | ETA 22:36:00 2020-12-24 20:07:20 [INFO] [TRAIN] epoch=35, iter=13000/80000, loss=16.6350, lr=0.008525, batch_cost=1.2392, reader_cost=0.0321 | ETA 23:03:43 2020-12-24 20:09:26 [INFO] [TRAIN] epoch=36, iter=13100/80000, loss=17.1512, lr=0.008514, batch_cost=1.2486, reader_cost=0.0396 | ETA 23:12:12 2020-12-24 20:11:26 [INFO] [TRAIN] epoch=36, iter=13200/80000, loss=16.1202, lr=0.008502, batch_cost=1.2047, reader_cost=0.0004 | ETA 22:21:10 2020-12-24 20:13:32 [INFO] [TRAIN] epoch=36, iter=13300/80000, loss=16.3890, lr=0.008491, batch_cost=1.2504, reader_cost=0.0014 | ETA 23:10:00 2020-12-24 20:15:40 [INFO] [TRAIN] epoch=37, iter=13400/80000, loss=16.7984, lr=0.008479, batch_cost=1.2815, reader_cost=0.0408 | ETA 23:42:25 2020-12-24 20:17:41 [INFO] [TRAIN] epoch=37, iter=13500/80000, loss=16.7616, lr=0.008468, batch_cost=1.2029, reader_cost=0.0003 | ETA 22:13:13 2020-12-24 20:19:38 [INFO] [TRAIN] epoch=37, iter=13600/80000, loss=16.2786, lr=0.008456, batch_cost=1.1773, reader_cost=0.0005 | ETA 21:42:52 2020-12-24 20:21:40 [INFO] [TRAIN] epoch=37, iter=13700/80000, loss=16.3160, lr=0.008445, batch_cost=1.2123, reader_cost=0.0009 | ETA 22:19:38 2020-12-24 20:23:47 [INFO] [TRAIN] epoch=38, iter=13800/80000, loss=16.8880, lr=0.008433, batch_cost=1.2680, reader_cost=0.0563 | ETA 23:19:03 2020-12-24 20:25:49 [INFO] [TRAIN] epoch=38, iter=13900/80000, loss=16.3420, lr=0.008422, batch_cost=1.2189, reader_cost=0.0004 | ETA 22:22:52 2020-12-24 20:27:47 [INFO] [TRAIN] epoch=38, iter=14000/80000, loss=16.2431, lr=0.008410, batch_cost=1.1786, reader_cost=0.0004 | ETA 21:36:28 2020-12-24 20:27:47 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-24 20:28:30 [INFO] [EVAL] #Images=500 mIoU=0.7387 Acc=0.9556 Kappa=0.9423 2020-12-24 20:28:30 [INFO] [EVAL] Class IoU: [0.9775 0.824 0.9182 0.3588 0.5572 0.6456 0.6932 0.7747 0.9214 0.6015 0.9424 0.8175 0.5982 0.9435 0.6756 0.7907 0.6826 0.5616 0.7509] 2020-12-24 20:28:30 [INFO] [EVAL] Class Acc: [0.9867 0.907 0.9552 0.8835 0.6685 0.8067 0.7669 0.93 0.951 0.8552 0.9649 0.8762 0.6776 0.9739 0.8096 0.9465 0.7757 0.8818 0.8312] 2020-12-24 20:28:38 [INFO] [EVAL] The model with the best validation mIoU (0.7387) was saved at iter 14000. 2020-12-24 20:30:39 [INFO] [TRAIN] epoch=38, iter=14100/80000, loss=16.7135, lr=0.008399, batch_cost=1.2073, reader_cost=0.0004 | ETA 22:06:02 2020-12-24 20:32:47 [INFO] [TRAIN] epoch=39, iter=14200/80000, loss=16.8243, lr=0.008387, batch_cost=1.2816, reader_cost=0.0449 | ETA 23:25:29 2020-12-24 20:34:49 [INFO] [TRAIN] epoch=39, iter=14300/80000, loss=16.3993, lr=0.008376, batch_cost=1.2153, reader_cost=0.0012 | ETA 22:10:45 2020-12-24 20:36:50 [INFO] [TRAIN] epoch=39, iter=14400/80000, loss=15.9284, lr=0.008364, batch_cost=1.2137, reader_cost=0.0004 | ETA 22:06:58 2020-12-24 20:38:52 [INFO] [TRAIN] epoch=39, iter=14500/80000, loss=16.6650, lr=0.008353, batch_cost=1.2192, reader_cost=0.0004 | ETA 22:10:55 2020-12-24 20:41:10 [INFO] [TRAIN] epoch=40, iter=14600/80000, loss=16.8360, lr=0.008342, batch_cost=1.3753, reader_cost=0.0403 | ETA 24:59:02 2020-12-24 20:43:14 [INFO] [TRAIN] epoch=40, iter=14700/80000, loss=16.2023, lr=0.008330, batch_cost=1.2411, reader_cost=0.0003 | ETA 22:30:42 2020-12-24 20:45:23 [INFO] [TRAIN] epoch=40, iter=14800/80000, loss=16.4566, lr=0.008319, batch_cost=1.2812, reader_cost=0.0002 | ETA 23:12:15 2020-12-24 20:47:31 [INFO] [TRAIN] epoch=41, iter=14900/80000, loss=16.6497, lr=0.008307, batch_cost=1.2855, reader_cost=0.0510 | ETA 23:14:43 2020-12-24 20:49:29 [INFO] [TRAIN] epoch=41, iter=15000/80000, loss=16.2211, lr=0.008296, batch_cost=1.1688, reader_cost=0.0004 | ETA 21:06:10 2020-12-24 20:51:31 [INFO] [TRAIN] epoch=41, iter=15100/80000, loss=16.3446, lr=0.008284, batch_cost=1.2185, reader_cost=0.0005 | ETA 21:58:02 2020-12-24 20:53:28 [INFO] [TRAIN] epoch=41, iter=15200/80000, loss=16.6149, lr=0.008273, batch_cost=1.1695, reader_cost=0.0009 | ETA 21:03:02 2020-12-24 20:55:33 [INFO] [TRAIN] epoch=42, iter=15300/80000, loss=16.8466, lr=0.008261, batch_cost=1.2513, reader_cost=0.0426 | ETA 22:29:20 2020-12-24 20:57:31 [INFO] [TRAIN] epoch=42, iter=15400/80000, loss=16.3259, lr=0.008250, batch_cost=1.1771, reader_cost=0.0002 | ETA 21:07:17 2020-12-24 20:59:31 [INFO] [TRAIN] epoch=42, iter=15500/80000, loss=16.1669, lr=0.008238, batch_cost=1.1967, reader_cost=0.0002 | ETA 21:26:26 2020-12-24 21:01:34 [INFO] [TRAIN] epoch=42, iter=15600/80000, loss=16.7996, lr=0.008227, batch_cost=1.2257, reader_cost=0.0002 | ETA 21:55:34 2020-12-24 21:03:40 [INFO] [TRAIN] epoch=43, iter=15700/80000, loss=16.9333, lr=0.008215, batch_cost=1.2581, reader_cost=0.0378 | ETA 22:28:15 2020-12-24 21:05:43 [INFO] [TRAIN] epoch=43, iter=15800/80000, loss=16.3654, lr=0.008204, batch_cost=1.2337, reader_cost=0.0009 | ETA 22:00:03 2020-12-24 21:07:43 [INFO] [TRAIN] epoch=43, iter=15900/80000, loss=16.0891, lr=0.008192, batch_cost=1.1953, reader_cost=0.0002 | ETA 21:16:58 2020-12-24 21:09:53 [INFO] [TRAIN] epoch=44, iter=16000/80000, loss=16.6667, lr=0.008181, batch_cost=1.2945, reader_cost=0.0392 | ETA 23:00:50 2020-12-24 21:09:53 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-24 21:10:36 [INFO] [EVAL] #Images=500 mIoU=0.7334 Acc=0.9548 Kappa=0.9413 2020-12-24 21:10:36 [INFO] [EVAL] Class IoU: [0.975 0.8182 0.9173 0.4877 0.5287 0.639 0.7049 0.775 0.9198 0.6194 0.9419 0.8087 0.5466 0.9443 0.5856 0.788 0.6567 0.516 0.7614] 2020-12-24 21:10:36 [INFO] [EVAL] Class Acc: [0.9844 0.9249 0.956 0.77 0.6633 0.8316 0.8581 0.8487 0.9502 0.8354 0.9677 0.8615 0.8406 0.9671 0.8026 0.906 0.7374 0.9032 0.8233] 2020-12-24 21:10:41 [INFO] [EVAL] The model with the best validation mIoU (0.7387) was saved at iter 14000. 2020-12-24 21:12:43 [INFO] [TRAIN] epoch=44, iter=16100/80000, loss=16.5884, lr=0.008169, batch_cost=1.2184, reader_cost=0.0015 | ETA 21:37:33 2020-12-24 21:14:44 [INFO] [TRAIN] epoch=44, iter=16200/80000, loss=16.2440, lr=0.008158, batch_cost=1.2067, reader_cost=0.0002 | ETA 21:23:05 2020-12-24 21:16:44 [INFO] [TRAIN] epoch=44, iter=16300/80000, loss=16.3278, lr=0.008146, batch_cost=1.1944, reader_cost=0.0002 | ETA 21:08:05 2020-12-24 21:18:53 [INFO] [TRAIN] epoch=45, iter=16400/80000, loss=16.7707, lr=0.008135, batch_cost=1.2875, reader_cost=0.0454 | ETA 22:44:42 2020-12-24 21:20:55 [INFO] [TRAIN] epoch=45, iter=16500/80000, loss=16.1298, lr=0.008123, batch_cost=1.2193, reader_cost=0.0005 | ETA 21:30:26 2020-12-24 21:22:58 [INFO] [TRAIN] epoch=45, iter=16600/80000, loss=16.1610, lr=0.008112, batch_cost=1.2230, reader_cost=0.0003 | ETA 21:32:18 2020-12-24 21:25:01 [INFO] [TRAIN] epoch=45, iter=16700/80000, loss=16.4918, lr=0.008100, batch_cost=1.2293, reader_cost=0.0002 | ETA 21:36:55 2020-12-24 21:27:08 [INFO] [TRAIN] epoch=46, iter=16800/80000, loss=16.8943, lr=0.008089, batch_cost=1.2696, reader_cost=0.0365 | ETA 22:17:19 2020-12-24 21:29:11 [INFO] [TRAIN] epoch=46, iter=16900/80000, loss=16.2572, lr=0.008077, batch_cost=1.2264, reader_cost=0.0089 | ETA 21:29:45 2020-12-24 21:31:12 [INFO] [TRAIN] epoch=46, iter=17000/80000, loss=16.0576, lr=0.008066, batch_cost=1.2125, reader_cost=0.0002 | ETA 21:13:05 2020-12-24 21:33:11 [INFO] [TRAIN] epoch=46, iter=17100/80000, loss=16.8546, lr=0.008054, batch_cost=1.1850, reader_cost=0.0007 | ETA 20:42:19 2020-12-24 21:35:21 [INFO] [TRAIN] epoch=47, iter=17200/80000, loss=16.9837, lr=0.008042, batch_cost=1.2993, reader_cost=0.0400 | ETA 22:39:58 2020-12-24 21:37:20 [INFO] [TRAIN] epoch=47, iter=17300/80000, loss=16.0035, lr=0.008031, batch_cost=1.1873, reader_cost=0.0002 | ETA 20:40:41 2020-12-24 21:39:24 [INFO] [TRAIN] epoch=47, iter=17400/80000, loss=16.4554, lr=0.008019, batch_cost=1.2401, reader_cost=0.0002 | ETA 21:33:47 2020-12-24 21:41:31 [INFO] [TRAIN] epoch=48, iter=17500/80000, loss=16.7139, lr=0.008008, batch_cost=1.2621, reader_cost=0.0442 | ETA 21:54:42 2020-12-24 21:43:28 [INFO] [TRAIN] epoch=48, iter=17600/80000, loss=16.1692, lr=0.007996, batch_cost=1.1712, reader_cost=0.0010 | ETA 20:18:03 2020-12-24 21:45:29 [INFO] [TRAIN] epoch=48, iter=17700/80000, loss=16.6023, lr=0.007985, batch_cost=1.2035, reader_cost=0.0005 | ETA 20:49:39 2020-12-24 21:47:38 [INFO] [TRAIN] epoch=48, iter=17800/80000, loss=16.1334, lr=0.007973, batch_cost=1.2907, reader_cost=0.0002 | ETA 22:18:02 2020-12-24 21:49:45 [INFO] [TRAIN] epoch=49, iter=17900/80000, loss=16.9599, lr=0.007962, batch_cost=1.2717, reader_cost=0.0409 | ETA 21:56:09 2020-12-24 21:51:43 [INFO] [TRAIN] epoch=49, iter=18000/80000, loss=16.3848, lr=0.007950, batch_cost=1.1804, reader_cost=0.0005 | ETA 20:19:45 2020-12-24 21:51:44 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-24 21:52:29 [INFO] [EVAL] #Images=500 mIoU=0.7248 Acc=0.9534 Kappa=0.9396 2020-12-24 21:52:29 [INFO] [EVAL] Class IoU: [0.9797 0.8437 0.9062 0.498 0.553 0.662 0.7123 0.7837 0.9216 0.6374 0.9358 0.7868 0.5937 0.9423 0.2713 0.7188 0.6859 0.5687 0.7709] 2020-12-24 21:52:29 [INFO] [EVAL] Class Acc: [0.9883 0.9168 0.9635 0.6885 0.8552 0.7809 0.8506 0.8779 0.9529 0.8017 0.9692 0.8363 0.7566 0.963 0.3036 0.982 0.7637 0.839 0.872 ] 2020-12-24 21:52:34 [INFO] [EVAL] The model with the best validation mIoU (0.7387) was saved at iter 14000. 2020-12-24 21:54:34 [INFO] [TRAIN] epoch=49, iter=18100/80000, loss=15.9729, lr=0.007939, batch_cost=1.2011, reader_cost=0.0002 | ETA 20:39:10 2020-12-24 21:56:33 [INFO] [TRAIN] epoch=49, iter=18200/80000, loss=16.6984, lr=0.007927, batch_cost=1.1863, reader_cost=0.0002 | ETA 20:21:52 2020-12-24 21:58:39 [INFO] [TRAIN] epoch=50, iter=18300/80000, loss=16.5575, lr=0.007916, batch_cost=1.2583, reader_cost=0.0434 | ETA 21:33:54 2020-12-24 22:00:39 [INFO] [TRAIN] epoch=50, iter=18400/80000, loss=16.1948, lr=0.007904, batch_cost=1.1952, reader_cost=0.0002 | ETA 20:27:05 2020-12-24 22:02:42 [INFO] [TRAIN] epoch=50, iter=18500/80000, loss=16.5065, lr=0.007892, batch_cost=1.2273, reader_cost=0.0007 | ETA 20:58:01 2020-12-24 22:04:37 [INFO] [TRAIN] epoch=50, iter=18600/80000, loss=16.9367, lr=0.007881, batch_cost=1.1475, reader_cost=0.0002 | ETA 19:34:14 2020-12-24 22:06:42 [INFO] [TRAIN] epoch=51, iter=18700/80000, loss=16.7709, lr=0.007869, batch_cost=1.2534, reader_cost=0.0386 | ETA 21:20:33 2020-12-24 22:08:41 [INFO] [TRAIN] epoch=51, iter=18800/80000, loss=16.2783, lr=0.007858, batch_cost=1.1919, reader_cost=0.0002 | ETA 20:15:45 2020-12-24 22:10:40 [INFO] [TRAIN] epoch=51, iter=18900/80000, loss=16.1906, lr=0.007846, batch_cost=1.1830, reader_cost=0.0004 | ETA 20:04:39 2020-12-24 22:12:45 [INFO] [TRAIN] epoch=52, iter=19000/80000, loss=16.8180, lr=0.007835, batch_cost=1.2508, reader_cost=0.0410 | ETA 21:11:35 2020-12-24 22:14:45 [INFO] [TRAIN] epoch=52, iter=19100/80000, loss=16.2168, lr=0.007823, batch_cost=1.1991, reader_cost=0.0003 | ETA 20:17:05 2020-12-24 22:16:51 [INFO] [TRAIN] epoch=52, iter=19200/80000, loss=16.0495, lr=0.007812, batch_cost=1.2503, reader_cost=0.0002 | ETA 21:06:58 2020-12-24 22:18:53 [INFO] [TRAIN] epoch=52, iter=19300/80000, loss=16.2828, lr=0.007800, batch_cost=1.2171, reader_cost=0.0003 | ETA 20:31:18 2020-12-24 22:20:59 [INFO] [TRAIN] epoch=53, iter=19400/80000, loss=16.9349, lr=0.007788, batch_cost=1.2661, reader_cost=0.0420 | ETA 21:18:48 2020-12-24 22:23:06 [INFO] [TRAIN] epoch=53, iter=19500/80000, loss=16.2652, lr=0.007777, batch_cost=1.2677, reader_cost=0.0002 | ETA 21:18:16 2020-12-24 22:25:05 [INFO] [TRAIN] epoch=53, iter=19600/80000, loss=15.9052, lr=0.007765, batch_cost=1.1884, reader_cost=0.0011 | ETA 19:56:18 2020-12-24 22:27:09 [INFO] [TRAIN] epoch=53, iter=19700/80000, loss=16.6198, lr=0.007754, batch_cost=1.2303, reader_cost=0.0010 | ETA 20:36:29 2020-12-24 22:29:15 [INFO] [TRAIN] epoch=54, iter=19800/80000, loss=16.5797, lr=0.007742, batch_cost=1.2623, reader_cost=0.0369 | ETA 21:06:28 2020-12-24 22:31:16 [INFO] [TRAIN] epoch=54, iter=19900/80000, loss=16.2073, lr=0.007731, batch_cost=1.2086, reader_cost=0.0002 | ETA 20:10:35 2020-12-24 22:33:16 [INFO] [TRAIN] epoch=54, iter=20000/80000, loss=16.1315, lr=0.007719, batch_cost=1.2016, reader_cost=0.0002 | ETA 20:01:34 2020-12-24 22:33:17 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-24 22:34:01 [INFO] [EVAL] #Images=500 mIoU=0.7209 Acc=0.9507 Kappa=0.9361 2020-12-24 22:34:01 [INFO] [EVAL] Class IoU: [0.9756 0.8405 0.9014 0.3818 0.5294 0.6521 0.6969 0.7342 0.9147 0.6108 0.9293 0.8157 0.6113 0.9223 0.444 0.7654 0.588 0.611 0.7727] 2020-12-24 22:34:01 [INFO] [EVAL] Class Acc: [0.9891 0.9265 0.9558 0.8245 0.6356 0.8038 0.7749 0.8867 0.9499 0.8022 0.9425 0.8641 0.7354 0.9466 0.4653 0.834 0.812 0.7509 0.8593] 2020-12-24 22:34:05 [INFO] [EVAL] The model with the best validation mIoU (0.7387) was saved at iter 14000. 2020-12-24 22:36:15 [INFO] [TRAIN] epoch=55, iter=20100/80000, loss=16.7947, lr=0.007707, batch_cost=1.2946, reader_cost=0.0579 | ETA 21:32:28 2020-12-24 22:38:18 [INFO] [TRAIN] epoch=55, iter=20200/80000, loss=16.1474, lr=0.007696, batch_cost=1.2262, reader_cost=0.0009 | ETA 20:22:05 2020-12-24 22:40:18 [INFO] [TRAIN] epoch=55, iter=20300/80000, loss=16.1656, lr=0.007684, batch_cost=1.1989, reader_cost=0.0038 | ETA 19:52:55 2020-12-24 22:42:21 [INFO] [TRAIN] epoch=55, iter=20400/80000, loss=16.2164, lr=0.007673, batch_cost=1.2275, reader_cost=0.0002 | ETA 20:19:18 2020-12-24 22:44:29 [INFO] [TRAIN] epoch=56, iter=20500/80000, loss=16.7670, lr=0.007661, batch_cost=1.2839, reader_cost=0.0393 | ETA 21:13:11 2020-12-24 22:46:31 [INFO] [TRAIN] epoch=56, iter=20600/80000, loss=16.2619, lr=0.007650, batch_cost=1.2166, reader_cost=0.0205 | ETA 20:04:24 2020-12-24 22:48:30 [INFO] [TRAIN] epoch=56, iter=20700/80000, loss=16.2376, lr=0.007638, batch_cost=1.1833, reader_cost=0.0002 | ETA 19:29:30 2020-12-24 22:50:27 [INFO] [TRAIN] epoch=56, iter=20800/80000, loss=16.5181, lr=0.007626, batch_cost=1.1680, reader_cost=0.0012 | ETA 19:12:24 2020-12-24 22:52:36 [INFO] [TRAIN] epoch=57, iter=20900/80000, loss=16.7166, lr=0.007615, batch_cost=1.2892, reader_cost=0.0421 | ETA 21:09:54 2020-12-24 22:54:42 [INFO] [TRAIN] epoch=57, iter=21000/80000, loss=15.9429, lr=0.007603, batch_cost=1.2567, reader_cost=0.0009 | ETA 20:35:46 2020-12-24 22:56:44 [INFO] [TRAIN] epoch=57, iter=21100/80000, loss=16.2213, lr=0.007592, batch_cost=1.2142, reader_cost=0.0002 | ETA 19:51:58 2020-12-24 22:58:39 [INFO] [TRAIN] epoch=57, iter=21200/80000, loss=16.6802, lr=0.007580, batch_cost=1.1541, reader_cost=0.0003 | ETA 18:51:01 2020-12-24 23:00:51 [INFO] [TRAIN] epoch=58, iter=21300/80000, loss=16.5321, lr=0.007568, batch_cost=1.3132, reader_cost=0.0441 | ETA 21:24:42 2020-12-24 23:02:50 [INFO] [TRAIN] epoch=58, iter=21400/80000, loss=16.2234, lr=0.007557, batch_cost=1.1886, reader_cost=0.0004 | ETA 19:20:53 2020-12-24 23:04:48 [INFO] [TRAIN] epoch=58, iter=21500/80000, loss=16.3055, lr=0.007545, batch_cost=1.1792, reader_cost=0.0006 | ETA 19:09:44 2020-12-24 23:06:56 [INFO] [TRAIN] epoch=59, iter=21600/80000, loss=16.4872, lr=0.007534, batch_cost=1.2731, reader_cost=0.0359 | ETA 20:39:07 2020-12-24 23:08:57 [INFO] [TRAIN] epoch=59, iter=21700/80000, loss=16.2660, lr=0.007522, batch_cost=1.2084, reader_cost=0.0002 | ETA 19:34:08 2020-12-24 23:10:53 [INFO] [TRAIN] epoch=59, iter=21800/80000, loss=16.1560, lr=0.007510, batch_cost=1.1644, reader_cost=0.0005 | ETA 18:49:30 2020-12-24 23:12:55 [INFO] [TRAIN] epoch=59, iter=21900/80000, loss=16.3725, lr=0.007499, batch_cost=1.2137, reader_cost=0.0002 | ETA 19:35:14 2020-12-24 23:15:06 [INFO] [TRAIN] epoch=60, iter=22000/80000, loss=16.7024, lr=0.007487, batch_cost=1.3044, reader_cost=0.0412 | ETA 21:00:56 2020-12-24 23:15:06 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-24 23:15:50 [INFO] [EVAL] #Images=500 mIoU=0.7700 Acc=0.9597 Kappa=0.9476 2020-12-24 23:15:50 [INFO] [EVAL] Class IoU: [0.979 0.8497 0.9239 0.5169 0.6174 0.6638 0.7202 0.8019 0.9215 0.6401 0.9461 0.8305 0.6274 0.9446 0.7162 0.7765 0.7042 0.6688 0.7809] 2020-12-24 23:15:50 [INFO] [EVAL] Class Acc: [0.9864 0.9319 0.9603 0.817 0.7826 0.8601 0.7997 0.8991 0.9485 0.8305 0.9717 0.9028 0.802 0.9653 0.8677 0.9858 0.777 0.7826 0.8469] 2020-12-24 23:15:57 [INFO] [EVAL] The model with the best validation mIoU (0.7700) was saved at iter 22000. 2020-12-24 23:18:08 [INFO] [TRAIN] epoch=60, iter=22100/80000, loss=15.9310, lr=0.007475, batch_cost=1.3091, reader_cost=0.0002 | ETA 21:03:15 2020-12-24 23:20:17 [INFO] [TRAIN] epoch=60, iter=22200/80000, loss=16.0385, lr=0.007464, batch_cost=1.2807, reader_cost=0.0003 | ETA 20:33:43 2020-12-24 23:22:17 [INFO] [TRAIN] epoch=60, iter=22300/80000, loss=16.4129, lr=0.007452, batch_cost=1.2031, reader_cost=0.0016 | ETA 19:16:58 2020-12-24 23:24:30 [INFO] [TRAIN] epoch=61, iter=22400/80000, loss=16.7020, lr=0.007441, batch_cost=1.3194, reader_cost=0.0374 | ETA 21:06:36 2020-12-24 23:26:30 [INFO] [TRAIN] epoch=61, iter=22500/80000, loss=15.7735, lr=0.007429, batch_cost=1.2032, reader_cost=0.0002 | ETA 19:13:05 2020-12-24 23:28:31 [INFO] [TRAIN] epoch=61, iter=22600/80000, loss=16.2145, lr=0.007417, batch_cost=1.2104, reader_cost=0.0002 | ETA 19:17:55 2020-12-24 23:30:42 [INFO] [TRAIN] epoch=62, iter=22700/80000, loss=16.3593, lr=0.007406, batch_cost=1.3051, reader_cost=0.0593 | ETA 20:46:20 2020-12-24 23:32:41 [INFO] [TRAIN] epoch=62, iter=22800/80000, loss=16.3757, lr=0.007394, batch_cost=1.1919, reader_cost=0.0002 | ETA 18:56:15 2020-12-24 23:34:42 [INFO] [TRAIN] epoch=62, iter=22900/80000, loss=16.0753, lr=0.007382, batch_cost=1.2053, reader_cost=0.0004 | ETA 19:07:00 2020-12-24 23:36:43 [INFO] [TRAIN] epoch=62, iter=23000/80000, loss=16.1243, lr=0.007371, batch_cost=1.2037, reader_cost=0.0002 | ETA 19:03:28 2020-12-24 23:38:52 [INFO] [TRAIN] epoch=63, iter=23100/80000, loss=16.5079, lr=0.007359, batch_cost=1.2919, reader_cost=0.0431 | ETA 20:25:06 2020-12-24 23:40:54 [INFO] [TRAIN] epoch=63, iter=23200/80000, loss=16.0913, lr=0.007347, batch_cost=1.2117, reader_cost=0.0002 | ETA 19:07:06 2020-12-24 23:42:57 [INFO] [TRAIN] epoch=63, iter=23300/80000, loss=15.9893, lr=0.007336, batch_cost=1.2316, reader_cost=0.0002 | ETA 19:23:54 2020-12-24 23:45:00 [INFO] [TRAIN] epoch=63, iter=23400/80000, loss=16.2978, lr=0.007324, batch_cost=1.2293, reader_cost=0.0002 | ETA 19:19:38 2020-12-24 23:47:09 [INFO] [TRAIN] epoch=64, iter=23500/80000, loss=16.2383, lr=0.007313, batch_cost=1.2894, reader_cost=0.0392 | ETA 20:14:11 2020-12-24 23:49:05 [INFO] [TRAIN] epoch=64, iter=23600/80000, loss=16.0637, lr=0.007301, batch_cost=1.1556, reader_cost=0.0027 | ETA 18:06:16 2020-12-24 23:51:02 [INFO] [TRAIN] epoch=64, iter=23700/80000, loss=15.8411, lr=0.007289, batch_cost=1.1717, reader_cost=0.0002 | ETA 18:19:24 2020-12-24 23:53:02 [INFO] [TRAIN] epoch=64, iter=23800/80000, loss=16.4429, lr=0.007278, batch_cost=1.1955, reader_cost=0.0002 | ETA 18:39:48 2020-12-24 23:55:16 [INFO] [TRAIN] epoch=65, iter=23900/80000, loss=16.4759, lr=0.007266, batch_cost=1.3376, reader_cost=0.0590 | ETA 20:50:38 2020-12-24 23:57:21 [INFO] [TRAIN] epoch=65, iter=24000/80000, loss=16.0690, lr=0.007254, batch_cost=1.2442, reader_cost=0.0002 | ETA 19:21:17 2020-12-24 23:57:21 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-24 23:58:05 [INFO] [EVAL] #Images=500 mIoU=0.7552 Acc=0.9572 Kappa=0.9445 2020-12-24 23:58:05 [INFO] [EVAL] Class IoU: [0.9753 0.8131 0.9238 0.4825 0.5864 0.6511 0.7211 0.8041 0.9235 0.6217 0.9427 0.8228 0.6039 0.9504 0.6108 0.8261 0.6902 0.627 0.7724] 2020-12-24 23:58:05 [INFO] [EVAL] Class Acc: [0.9886 0.8883 0.957 0.8048 0.7926 0.8469 0.8327 0.9045 0.9524 0.8382 0.9571 0.9114 0.6822 0.9695 0.8877 0.873 0.7553 0.7515 0.8447] 2020-12-24 23:58:10 [INFO] [EVAL] The model with the best validation mIoU (0.7700) was saved at iter 22000. 2020-12-25 00:00:08 [INFO] [TRAIN] epoch=65, iter=24100/80000, loss=16.2158, lr=0.007243, batch_cost=1.1783, reader_cost=0.0002 | ETA 18:17:45 2020-12-25 00:02:14 [INFO] [TRAIN] epoch=66, iter=24200/80000, loss=16.4752, lr=0.007231, batch_cost=1.2610, reader_cost=0.0540 | ETA 19:32:44 2020-12-25 00:04:11 [INFO] [TRAIN] epoch=66, iter=24300/80000, loss=16.1633, lr=0.007219, batch_cost=1.1707, reader_cost=0.0002 | ETA 18:06:49 2020-12-25 00:06:13 [INFO] [TRAIN] epoch=66, iter=24400/80000, loss=16.1384, lr=0.007208, batch_cost=1.2143, reader_cost=0.0002 | ETA 18:45:17 2020-12-25 00:08:19 [INFO] [TRAIN] epoch=66, iter=24500/80000, loss=16.2378, lr=0.007196, batch_cost=1.2552, reader_cost=0.0002 | ETA 19:21:03 2020-12-25 00:10:30 [INFO] [TRAIN] epoch=67, iter=24600/80000, loss=16.6803, lr=0.007184, batch_cost=1.3084, reader_cost=0.0397 | ETA 20:08:07 2020-12-25 00:12:30 [INFO] [TRAIN] epoch=67, iter=24700/80000, loss=16.0842, lr=0.007173, batch_cost=1.1970, reader_cost=0.0002 | ETA 18:23:14 2020-12-25 00:14:33 [INFO] [TRAIN] epoch=67, iter=24800/80000, loss=15.8558, lr=0.007161, batch_cost=1.2328, reader_cost=0.0004 | ETA 18:54:07 2020-12-25 00:16:38 [INFO] [TRAIN] epoch=67, iter=24900/80000, loss=16.2310, lr=0.007149, batch_cost=1.2509, reader_cost=0.0004 | ETA 19:08:43 2020-12-25 00:18:52 [INFO] [TRAIN] epoch=68, iter=25000/80000, loss=16.5700, lr=0.007138, batch_cost=1.3335, reader_cost=0.0377 | ETA 20:22:25 2020-12-25 00:20:52 [INFO] [TRAIN] epoch=68, iter=25100/80000, loss=16.0124, lr=0.007126, batch_cost=1.1989, reader_cost=0.0002 | ETA 18:17:02 2020-12-25 00:22:50 [INFO] [TRAIN] epoch=68, iter=25200/80000, loss=15.9867, lr=0.007114, batch_cost=1.1831, reader_cost=0.0007 | ETA 18:00:31 2020-12-25 00:24:56 [INFO] [TRAIN] epoch=69, iter=25300/80000, loss=16.5163, lr=0.007103, batch_cost=1.2530, reader_cost=0.0396 | ETA 19:02:20 2020-12-25 00:27:11 [INFO] [TRAIN] epoch=69, iter=25400/80000, loss=16.5138, lr=0.007091, batch_cost=1.3478, reader_cost=0.0002 | ETA 20:26:28 2020-12-25 00:29:10 [INFO] [TRAIN] epoch=69, iter=25500/80000, loss=16.1679, lr=0.007079, batch_cost=1.1930, reader_cost=0.0152 | ETA 18:03:40 2020-12-25 00:31:11 [INFO] [TRAIN] epoch=69, iter=25600/80000, loss=16.0326, lr=0.007067, batch_cost=1.2007, reader_cost=0.0005 | ETA 18:08:36 2020-12-25 00:33:20 [INFO] [TRAIN] epoch=70, iter=25700/80000, loss=16.3109, lr=0.007056, batch_cost=1.2912, reader_cost=0.0361 | ETA 19:28:29 2020-12-25 00:35:20 [INFO] [TRAIN] epoch=70, iter=25800/80000, loss=16.2214, lr=0.007044, batch_cost=1.1986, reader_cost=0.0002 | ETA 18:02:43 2020-12-25 00:37:22 [INFO] [TRAIN] epoch=70, iter=25900/80000, loss=16.3181, lr=0.007032, batch_cost=1.2128, reader_cost=0.0002 | ETA 18:13:31 2020-12-25 00:39:20 [INFO] [TRAIN] epoch=70, iter=26000/80000, loss=16.0477, lr=0.007021, batch_cost=1.1833, reader_cost=0.0003 | ETA 17:45:00 2020-12-25 00:39:20 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 00:40:05 [INFO] [EVAL] #Images=500 mIoU=0.7720 Acc=0.9599 Kappa=0.9479 2020-12-25 00:40:05 [INFO] [EVAL] Class IoU: [0.9812 0.8507 0.9212 0.4689 0.5875 0.662 0.7196 0.8075 0.9219 0.6049 0.9445 0.8358 0.6338 0.9525 0.776 0.8397 0.7471 0.6319 0.7812] 2020-12-25 00:40:05 [INFO] [EVAL] Class Acc: [0.9917 0.9084 0.9484 0.7792 0.8283 0.8586 0.8869 0.9166 0.955 0.828 0.9601 0.9015 0.8544 0.9788 0.8605 0.9446 0.8521 0.7015 0.8516] 2020-12-25 00:40:12 [INFO] [EVAL] The model with the best validation mIoU (0.7720) was saved at iter 26000. 2020-12-25 00:42:25 [INFO] [TRAIN] epoch=71, iter=26100/80000, loss=16.5925, lr=0.007009, batch_cost=1.3333, reader_cost=0.0577 | ETA 19:57:44 2020-12-25 00:44:24 [INFO] [TRAIN] epoch=71, iter=26200/80000, loss=16.0171, lr=0.006997, batch_cost=1.1926, reader_cost=0.0012 | ETA 17:49:21 2020-12-25 00:46:30 [INFO] [TRAIN] epoch=71, iter=26300/80000, loss=15.9723, lr=0.006986, batch_cost=1.2566, reader_cost=0.0021 | ETA 18:44:37 2020-12-25 00:48:32 [INFO] [TRAIN] epoch=71, iter=26400/80000, loss=16.5357, lr=0.006974, batch_cost=1.2184, reader_cost=0.0002 | ETA 18:08:28 2020-12-25 00:50:42 [INFO] [TRAIN] epoch=72, iter=26500/80000, loss=16.5976, lr=0.006962, batch_cost=1.2967, reader_cost=0.0398 | ETA 19:16:14 2020-12-25 00:52:42 [INFO] [TRAIN] epoch=72, iter=26600/80000, loss=15.8959, lr=0.006950, batch_cost=1.1900, reader_cost=0.0002 | ETA 17:39:03 2020-12-25 00:54:42 [INFO] [TRAIN] epoch=72, iter=26700/80000, loss=16.0487, lr=0.006939, batch_cost=1.1959, reader_cost=0.0009 | ETA 17:42:23 2020-12-25 00:56:48 [INFO] [TRAIN] epoch=73, iter=26800/80000, loss=16.7180, lr=0.006927, batch_cost=1.2608, reader_cost=0.0381 | ETA 18:37:56 2020-12-25 00:58:48 [INFO] [TRAIN] epoch=73, iter=26900/80000, loss=16.0107, lr=0.006915, batch_cost=1.1957, reader_cost=0.0016 | ETA 17:38:13 2020-12-25 01:00:48 [INFO] [TRAIN] epoch=73, iter=27000/80000, loss=16.2824, lr=0.006904, batch_cost=1.2009, reader_cost=0.0002 | ETA 17:40:49 2020-12-25 01:02:50 [INFO] [TRAIN] epoch=73, iter=27100/80000, loss=15.8084, lr=0.006892, batch_cost=1.2195, reader_cost=0.0043 | ETA 17:55:09 2020-12-25 01:05:01 [INFO] [TRAIN] epoch=74, iter=27200/80000, loss=16.3761, lr=0.006880, batch_cost=1.3010, reader_cost=0.0406 | ETA 19:04:55 2020-12-25 01:07:00 [INFO] [TRAIN] epoch=74, iter=27300/80000, loss=16.2859, lr=0.006868, batch_cost=1.1922, reader_cost=0.0004 | ETA 17:27:09 2020-12-25 01:09:01 [INFO] [TRAIN] epoch=74, iter=27400/80000, loss=15.8596, lr=0.006857, batch_cost=1.2094, reader_cost=0.0002 | ETA 17:40:14 2020-12-25 01:10:59 [INFO] [TRAIN] epoch=74, iter=27500/80000, loss=16.2804, lr=0.006845, batch_cost=1.1757, reader_cost=0.0002 | ETA 17:08:42 2020-12-25 01:13:09 [INFO] [TRAIN] epoch=75, iter=27600/80000, loss=16.5373, lr=0.006833, batch_cost=1.2970, reader_cost=0.0447 | ETA 18:52:40 2020-12-25 01:15:19 [INFO] [TRAIN] epoch=75, iter=27700/80000, loss=15.9653, lr=0.006821, batch_cost=1.3028, reader_cost=0.0002 | ETA 18:55:36 2020-12-25 01:17:20 [INFO] [TRAIN] epoch=75, iter=27800/80000, loss=16.1166, lr=0.006810, batch_cost=1.2041, reader_cost=0.0003 | ETA 17:27:32 2020-12-25 01:19:21 [INFO] [TRAIN] epoch=75, iter=27900/80000, loss=16.4778, lr=0.006798, batch_cost=1.2092, reader_cost=0.0004 | ETA 17:29:58 2020-12-25 01:21:35 [INFO] [TRAIN] epoch=76, iter=28000/80000, loss=16.3418, lr=0.006786, batch_cost=1.3374, reader_cost=0.0606 | ETA 19:19:03 2020-12-25 01:21:35 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 01:22:19 [INFO] [EVAL] #Images=500 mIoU=0.7748 Acc=0.9599 Kappa=0.9480 2020-12-25 01:22:19 [INFO] [EVAL] Class IoU: [0.9809 0.8442 0.9236 0.5356 0.6158 0.6621 0.7348 0.8115 0.9229 0.5609 0.9455 0.8322 0.6224 0.9506 0.7193 0.8664 0.8101 0.5977 0.7846] 2020-12-25 01:22:19 [INFO] [EVAL] Class Acc: [0.9914 0.906 0.9608 0.8295 0.7832 0.7666 0.8615 0.8929 0.9517 0.9192 0.9602 0.9157 0.7147 0.9741 0.8012 0.9614 0.9412 0.7083 0.8792] 2020-12-25 01:22:26 [INFO] [EVAL] The model with the best validation mIoU (0.7748) was saved at iter 28000. 2020-12-25 01:24:27 [INFO] [TRAIN] epoch=76, iter=28100/80000, loss=16.1054, lr=0.006774, batch_cost=1.2086, reader_cost=0.0006 | ETA 17:25:27 2020-12-25 01:26:26 [INFO] [TRAIN] epoch=76, iter=28200/80000, loss=16.0235, lr=0.006763, batch_cost=1.1892, reader_cost=0.0002 | ETA 17:06:38 2020-12-25 01:28:42 [INFO] [TRAIN] epoch=77, iter=28300/80000, loss=16.6872, lr=0.006751, batch_cost=1.3610, reader_cost=0.0468 | ETA 19:32:41 2020-12-25 01:30:41 [INFO] [TRAIN] epoch=77, iter=28400/80000, loss=16.2656, lr=0.006739, batch_cost=1.1850, reader_cost=0.0030 | ETA 16:59:08 2020-12-25 01:32:40 [INFO] [TRAIN] epoch=77, iter=28500/80000, loss=15.8977, lr=0.006727, batch_cost=1.1888, reader_cost=0.0005 | ETA 17:00:22 2020-12-25 01:34:45 [INFO] [TRAIN] epoch=77, iter=28600/80000, loss=15.9822, lr=0.006716, batch_cost=1.2514, reader_cost=0.0002 | ETA 17:52:02 2020-12-25 01:36:53 [INFO] [TRAIN] epoch=78, iter=28700/80000, loss=16.4793, lr=0.006704, batch_cost=1.2712, reader_cost=0.0406 | ETA 18:06:50 2020-12-25 01:38:50 [INFO] [TRAIN] epoch=78, iter=28800/80000, loss=15.9045, lr=0.006692, batch_cost=1.1748, reader_cost=0.0002 | ETA 16:42:30 2020-12-25 01:40:52 [INFO] [TRAIN] epoch=78, iter=28900/80000, loss=15.7465, lr=0.006680, batch_cost=1.2158, reader_cost=0.0044 | ETA 17:15:25 2020-12-25 01:43:01 [INFO] [TRAIN] epoch=78, iter=29000/80000, loss=16.4699, lr=0.006669, batch_cost=1.2861, reader_cost=0.0013 | ETA 18:13:12 2020-12-25 01:45:08 [INFO] [TRAIN] epoch=79, iter=29100/80000, loss=16.5350, lr=0.006657, batch_cost=1.2669, reader_cost=0.0388 | ETA 17:54:46 2020-12-25 01:47:08 [INFO] [TRAIN] epoch=79, iter=29200/80000, loss=15.7644, lr=0.006645, batch_cost=1.2005, reader_cost=0.0007 | ETA 16:56:23 2020-12-25 01:49:11 [INFO] [TRAIN] epoch=79, iter=29300/80000, loss=15.7410, lr=0.006633, batch_cost=1.2309, reader_cost=0.0002 | ETA 17:20:07 2020-12-25 01:51:23 [INFO] [TRAIN] epoch=80, iter=29400/80000, loss=16.3791, lr=0.006622, batch_cost=1.3125, reader_cost=0.0397 | ETA 18:26:54 2020-12-25 01:53:24 [INFO] [TRAIN] epoch=80, iter=29500/80000, loss=16.0640, lr=0.006610, batch_cost=1.2054, reader_cost=0.0004 | ETA 16:54:34 2020-12-25 01:55:22 [INFO] [TRAIN] epoch=80, iter=29600/80000, loss=15.9933, lr=0.006598, batch_cost=1.1839, reader_cost=0.0002 | ETA 16:34:29 2020-12-25 01:57:23 [INFO] [TRAIN] epoch=80, iter=29700/80000, loss=15.7578, lr=0.006586, batch_cost=1.2052, reader_cost=0.0002 | ETA 16:50:23 2020-12-25 01:59:37 [INFO] [TRAIN] epoch=81, iter=29800/80000, loss=16.2792, lr=0.006574, batch_cost=1.3409, reader_cost=0.0436 | ETA 18:41:55 2020-12-25 02:01:37 [INFO] [TRAIN] epoch=81, iter=29900/80000, loss=15.8484, lr=0.006563, batch_cost=1.1938, reader_cost=0.0004 | ETA 16:36:49 2020-12-25 02:03:39 [INFO] [TRAIN] epoch=81, iter=30000/80000, loss=15.5557, lr=0.006551, batch_cost=1.2177, reader_cost=0.0002 | ETA 16:54:47 2020-12-25 02:03:39 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 02:04:23 [INFO] [EVAL] #Images=500 mIoU=0.7738 Acc=0.9607 Kappa=0.9489 2020-12-25 02:04:23 [INFO] [EVAL] Class IoU: [0.9776 0.8448 0.9281 0.4878 0.5929 0.6851 0.7282 0.801 0.9263 0.6331 0.9438 0.8278 0.5826 0.9535 0.7396 0.871 0.7359 0.6597 0.7829] 2020-12-25 02:04:23 [INFO] [EVAL] Class Acc: [0.9866 0.9113 0.9642 0.7794 0.8707 0.8294 0.8546 0.892 0.9486 0.884 0.9665 0.8825 0.82 0.9742 0.907 0.9351 0.8548 0.8417 0.8909] 2020-12-25 02:04:28 [INFO] [EVAL] The model with the best validation mIoU (0.7748) was saved at iter 28000. 2020-12-25 02:06:29 [INFO] [TRAIN] epoch=81, iter=30100/80000, loss=16.2136, lr=0.006539, batch_cost=1.2098, reader_cost=0.0003 | ETA 16:46:09 2020-12-25 02:08:38 [INFO] [TRAIN] epoch=82, iter=30200/80000, loss=16.2684, lr=0.006527, batch_cost=1.2859, reader_cost=0.0420 | ETA 17:47:16 2020-12-25 02:10:37 [INFO] [TRAIN] epoch=82, iter=30300/80000, loss=15.8047, lr=0.006515, batch_cost=1.1852, reader_cost=0.0044 | ETA 16:21:46 2020-12-25 02:12:43 [INFO] [TRAIN] epoch=82, iter=30400/80000, loss=15.4505, lr=0.006504, batch_cost=1.2630, reader_cost=0.0002 | ETA 17:24:06 2020-12-25 02:14:42 [INFO] [TRAIN] epoch=82, iter=30500/80000, loss=16.3536, lr=0.006492, batch_cost=1.1903, reader_cost=0.0011 | ETA 16:22:00 2020-12-25 02:16:54 [INFO] [TRAIN] epoch=83, iter=30600/80000, loss=16.3049, lr=0.006480, batch_cost=1.3118, reader_cost=0.0403 | ETA 18:00:00 2020-12-25 02:18:52 [INFO] [TRAIN] epoch=83, iter=30700/80000, loss=16.1388, lr=0.006468, batch_cost=1.1845, reader_cost=0.0004 | ETA 16:13:16 2020-12-25 02:20:55 [INFO] [TRAIN] epoch=83, iter=30800/80000, loss=16.1466, lr=0.006456, batch_cost=1.2271, reader_cost=0.0003 | ETA 16:46:15 2020-12-25 02:23:04 [INFO] [TRAIN] epoch=84, iter=30900/80000, loss=16.2086, lr=0.006445, batch_cost=1.2903, reader_cost=0.0384 | ETA 17:35:55 2020-12-25 02:25:03 [INFO] [TRAIN] epoch=84, iter=31000/80000, loss=16.0403, lr=0.006433, batch_cost=1.1852, reader_cost=0.0047 | ETA 16:07:54 2020-12-25 02:27:06 [INFO] [TRAIN] epoch=84, iter=31100/80000, loss=16.1494, lr=0.006421, batch_cost=1.2272, reader_cost=0.0002 | ETA 16:40:08 2020-12-25 02:29:13 [INFO] [TRAIN] epoch=84, iter=31200/80000, loss=16.0623, lr=0.006409, batch_cost=1.2602, reader_cost=0.0003 | ETA 17:04:59 2020-12-25 02:31:32 [INFO] [TRAIN] epoch=85, iter=31300/80000, loss=16.8821, lr=0.006397, batch_cost=1.3904, reader_cost=0.0370 | ETA 18:48:30 2020-12-25 02:33:32 [INFO] [TRAIN] epoch=85, iter=31400/80000, loss=15.9617, lr=0.006386, batch_cost=1.2006, reader_cost=0.0002 | ETA 16:12:27 2020-12-25 02:35:41 [INFO] [TRAIN] epoch=85, iter=31500/80000, loss=16.0624, lr=0.006374, batch_cost=1.2816, reader_cost=0.0065 | ETA 17:15:56 2020-12-25 02:37:41 [INFO] [TRAIN] epoch=85, iter=31600/80000, loss=16.3730, lr=0.006362, batch_cost=1.2049, reader_cost=0.0031 | ETA 16:11:56 2020-12-25 02:39:48 [INFO] [TRAIN] epoch=86, iter=31700/80000, loss=16.3012, lr=0.006350, batch_cost=1.2622, reader_cost=0.0369 | ETA 16:56:05 2020-12-25 02:41:44 [INFO] [TRAIN] epoch=86, iter=31800/80000, loss=15.8058, lr=0.006338, batch_cost=1.1648, reader_cost=0.0002 | ETA 15:35:44 2020-12-25 02:43:51 [INFO] [TRAIN] epoch=86, iter=31900/80000, loss=15.8865, lr=0.006326, batch_cost=1.2639, reader_cost=0.0028 | ETA 16:53:13 2020-12-25 02:46:08 [INFO] [TRAIN] epoch=87, iter=32000/80000, loss=16.2381, lr=0.006315, batch_cost=1.3717, reader_cost=0.0440 | ETA 18:17:23 2020-12-25 02:46:08 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 02:46:52 [INFO] [EVAL] #Images=500 mIoU=0.7699 Acc=0.9596 Kappa=0.9475 2020-12-25 02:46:52 [INFO] [EVAL] Class IoU: [0.9775 0.832 0.9253 0.5179 0.6418 0.6794 0.7266 0.8102 0.9235 0.589 0.9427 0.8367 0.6527 0.9469 0.6359 0.8394 0.737 0.6268 0.7874] 2020-12-25 02:46:52 [INFO] [EVAL] Class Acc: [0.9867 0.9103 0.9608 0.8582 0.805 0.8495 0.8722 0.9168 0.9484 0.9022 0.9575 0.9026 0.8469 0.9649 0.8954 0.9549 0.8053 0.861 0.8476] 2020-12-25 02:46:57 [INFO] [EVAL] The model with the best validation mIoU (0.7748) was saved at iter 28000. 2020-12-25 02:48:54 [INFO] [TRAIN] epoch=87, iter=32100/80000, loss=16.1320, lr=0.006303, batch_cost=1.1696, reader_cost=0.0002 | ETA 15:33:44 2020-12-25 02:50:53 [INFO] [TRAIN] epoch=87, iter=32200/80000, loss=16.0857, lr=0.006291, batch_cost=1.1939, reader_cost=0.0006 | ETA 15:51:07 2020-12-25 02:52:50 [INFO] [TRAIN] epoch=87, iter=32300/80000, loss=15.7757, lr=0.006279, batch_cost=1.1673, reader_cost=0.0015 | ETA 15:28:02 2020-12-25 02:55:03 [INFO] [TRAIN] epoch=88, iter=32400/80000, loss=16.3397, lr=0.006267, batch_cost=1.3225, reader_cost=0.0367 | ETA 17:29:10 2020-12-25 02:57:01 [INFO] [TRAIN] epoch=88, iter=32500/80000, loss=15.8372, lr=0.006255, batch_cost=1.1829, reader_cost=0.0002 | ETA 15:36:28 2020-12-25 02:59:00 [INFO] [TRAIN] epoch=88, iter=32600/80000, loss=15.7373, lr=0.006243, batch_cost=1.1855, reader_cost=0.0002 | ETA 15:36:33 2020-12-25 03:01:01 [INFO] [TRAIN] epoch=88, iter=32700/80000, loss=15.9073, lr=0.006232, batch_cost=1.2057, reader_cost=0.0003 | ETA 15:50:29 2020-12-25 03:03:09 [INFO] [TRAIN] epoch=89, iter=32800/80000, loss=16.5940, lr=0.006220, batch_cost=1.2774, reader_cost=0.0423 | ETA 16:44:51 2020-12-25 03:05:09 [INFO] [TRAIN] epoch=89, iter=32900/80000, loss=16.1381, lr=0.006208, batch_cost=1.2010, reader_cost=0.0007 | ETA 15:42:47 2020-12-25 03:07:11 [INFO] [TRAIN] epoch=89, iter=33000/80000, loss=15.9349, lr=0.006196, batch_cost=1.2152, reader_cost=0.0002 | ETA 15:51:52 2020-12-25 03:09:09 [INFO] [TRAIN] epoch=89, iter=33100/80000, loss=16.1446, lr=0.006184, batch_cost=1.1841, reader_cost=0.0002 | ETA 15:25:32 2020-12-25 03:11:18 [INFO] [TRAIN] epoch=90, iter=33200/80000, loss=16.1399, lr=0.006172, batch_cost=1.2809, reader_cost=0.0520 | ETA 16:39:06 2020-12-25 03:13:15 [INFO] [TRAIN] epoch=90, iter=33300/80000, loss=15.6102, lr=0.006160, batch_cost=1.1723, reader_cost=0.0002 | ETA 15:12:25 2020-12-25 03:15:21 [INFO] [TRAIN] epoch=90, iter=33400/80000, loss=15.8693, lr=0.006149, batch_cost=1.2602, reader_cost=0.0002 | ETA 16:18:43 2020-12-25 03:17:36 [INFO] [TRAIN] epoch=91, iter=33500/80000, loss=16.3675, lr=0.006137, batch_cost=1.3404, reader_cost=0.0459 | ETA 17:18:49 2020-12-25 03:19:35 [INFO] [TRAIN] epoch=91, iter=33600/80000, loss=15.9740, lr=0.006125, batch_cost=1.1871, reader_cost=0.0002 | ETA 15:18:01 2020-12-25 03:21:36 [INFO] [TRAIN] epoch=91, iter=33700/80000, loss=16.1146, lr=0.006113, batch_cost=1.2140, reader_cost=0.0006 | ETA 15:36:48 2020-12-25 03:23:34 [INFO] [TRAIN] epoch=91, iter=33800/80000, loss=15.7636, lr=0.006101, batch_cost=1.1801, reader_cost=0.0004 | ETA 15:08:42 2020-12-25 03:25:45 [INFO] [TRAIN] epoch=92, iter=33900/80000, loss=16.1217, lr=0.006089, batch_cost=1.3022, reader_cost=0.0399 | ETA 16:40:29 2020-12-25 03:27:42 [INFO] [TRAIN] epoch=92, iter=34000/80000, loss=15.9182, lr=0.006077, batch_cost=1.1690, reader_cost=0.0002 | ETA 14:56:13 2020-12-25 03:27:42 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 03:28:26 [INFO] [EVAL] #Images=500 mIoU=0.7761 Acc=0.9614 Kappa=0.9500 2020-12-25 03:28:26 [INFO] [EVAL] Class IoU: [0.9804 0.8419 0.9295 0.5458 0.6282 0.6746 0.733 0.8143 0.9274 0.6019 0.9456 0.8366 0.6664 0.9512 0.6914 0.8069 0.7767 0.6026 0.791 ] 2020-12-25 03:28:26 [INFO] [EVAL] Class Acc: [0.9894 0.9075 0.9616 0.7798 0.8446 0.8364 0.8526 0.9105 0.9575 0.8517 0.9602 0.9106 0.79 0.969 0.9204 0.8598 0.8804 0.8075 0.8654] 2020-12-25 03:28:33 [INFO] [EVAL] The model with the best validation mIoU (0.7761) was saved at iter 34000. 2020-12-25 03:30:30 [INFO] [TRAIN] epoch=92, iter=34100/80000, loss=15.8805, lr=0.006065, batch_cost=1.1671, reader_cost=0.0002 | ETA 14:52:48 2020-12-25 03:32:34 [INFO] [TRAIN] epoch=92, iter=34200/80000, loss=16.1615, lr=0.006053, batch_cost=1.2350, reader_cost=0.0021 | ETA 15:42:41 2020-12-25 03:34:47 [INFO] [TRAIN] epoch=93, iter=34300/80000, loss=16.6497, lr=0.006042, batch_cost=1.3284, reader_cost=0.0415 | ETA 16:51:48 2020-12-25 03:36:46 [INFO] [TRAIN] epoch=93, iter=34400/80000, loss=15.7720, lr=0.006030, batch_cost=1.1849, reader_cost=0.0002 | ETA 15:00:32 2020-12-25 03:38:44 [INFO] [TRAIN] epoch=93, iter=34500/80000, loss=15.9170, lr=0.006018, batch_cost=1.1786, reader_cost=0.0002 | ETA 14:53:44 2020-12-25 03:40:51 [INFO] [TRAIN] epoch=94, iter=34600/80000, loss=16.3409, lr=0.006006, batch_cost=1.2741, reader_cost=0.0387 | ETA 16:04:05 2020-12-25 03:42:52 [INFO] [TRAIN] epoch=94, iter=34700/80000, loss=16.2095, lr=0.005994, batch_cost=1.2043, reader_cost=0.0002 | ETA 15:09:16 2020-12-25 03:44:54 [INFO] [TRAIN] epoch=94, iter=34800/80000, loss=15.8270, lr=0.005982, batch_cost=1.2178, reader_cost=0.0002 | ETA 15:17:25 2020-12-25 03:46:58 [INFO] [TRAIN] epoch=94, iter=34900/80000, loss=15.8535, lr=0.005970, batch_cost=1.2362, reader_cost=0.0015 | ETA 15:29:10 2020-12-25 03:49:12 [INFO] [TRAIN] epoch=95, iter=35000/80000, loss=16.3114, lr=0.005958, batch_cost=1.3373, reader_cost=0.0367 | ETA 16:42:58 2020-12-25 03:51:13 [INFO] [TRAIN] epoch=95, iter=35100/80000, loss=15.9711, lr=0.005946, batch_cost=1.2042, reader_cost=0.0005 | ETA 15:01:07 2020-12-25 03:53:11 [INFO] [TRAIN] epoch=95, iter=35200/80000, loss=15.5336, lr=0.005934, batch_cost=1.1804, reader_cost=0.0013 | ETA 14:41:21 2020-12-25 03:55:14 [INFO] [TRAIN] epoch=95, iter=35300/80000, loss=15.9729, lr=0.005922, batch_cost=1.2263, reader_cost=0.0031 | ETA 15:13:35 2020-12-25 03:57:24 [INFO] [TRAIN] epoch=96, iter=35400/80000, loss=16.4017, lr=0.005911, batch_cost=1.3000, reader_cost=0.0385 | ETA 16:06:20 2020-12-25 03:59:22 [INFO] [TRAIN] epoch=96, iter=35500/80000, loss=15.7566, lr=0.005899, batch_cost=1.1812, reader_cost=0.0006 | ETA 14:36:05 2020-12-25 04:01:23 [INFO] [TRAIN] epoch=96, iter=35600/80000, loss=15.6918, lr=0.005887, batch_cost=1.2043, reader_cost=0.0010 | ETA 14:51:12 2020-12-25 04:03:19 [INFO] [TRAIN] epoch=96, iter=35700/80000, loss=16.0381, lr=0.005875, batch_cost=1.1619, reader_cost=0.0002 | ETA 14:17:50 2020-12-25 04:05:33 [INFO] [TRAIN] epoch=97, iter=35800/80000, loss=16.2404, lr=0.005863, batch_cost=1.3321, reader_cost=0.0432 | ETA 16:21:17 2020-12-25 04:07:31 [INFO] [TRAIN] epoch=97, iter=35900/80000, loss=15.7469, lr=0.005851, batch_cost=1.1774, reader_cost=0.0002 | ETA 14:25:24 2020-12-25 04:09:31 [INFO] [TRAIN] epoch=97, iter=36000/80000, loss=15.7109, lr=0.005839, batch_cost=1.2007, reader_cost=0.0002 | ETA 14:40:31 2020-12-25 04:09:31 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 04:10:15 [INFO] [EVAL] #Images=500 mIoU=0.7804 Acc=0.9610 Kappa=0.9494 2020-12-25 04:10:15 [INFO] [EVAL] Class IoU: [0.981 0.8474 0.9254 0.4458 0.5877 0.6904 0.7394 0.8064 0.9255 0.6368 0.9485 0.8438 0.665 0.9521 0.7639 0.87 0.7411 0.6678 0.7887] 2020-12-25 04:10:15 [INFO] [EVAL] Class Acc: [0.9914 0.9029 0.9536 0.8141 0.8021 0.8117 0.8385 0.9119 0.9623 0.8349 0.9661 0.9123 0.7788 0.9719 0.9111 0.9701 0.8311 0.8353 0.8632] 2020-12-25 04:10:25 [INFO] [EVAL] The model with the best validation mIoU (0.7804) was saved at iter 36000. 2020-12-25 04:12:37 [INFO] [TRAIN] epoch=98, iter=36100/80000, loss=16.5130, lr=0.005827, batch_cost=1.3212, reader_cost=0.0362 | ETA 16:06:41 2020-12-25 04:14:39 [INFO] [TRAIN] epoch=98, iter=36200/80000, loss=15.9344, lr=0.005815, batch_cost=1.2138, reader_cost=0.0002 | ETA 14:46:03 2020-12-25 04:16:37 [INFO] [TRAIN] epoch=98, iter=36300/80000, loss=15.9325, lr=0.005803, batch_cost=1.1835, reader_cost=0.0003 | ETA 14:21:58 2020-12-25 04:18:38 [INFO] [TRAIN] epoch=98, iter=36400/80000, loss=15.9172, lr=0.005791, batch_cost=1.2004, reader_cost=0.0002 | ETA 14:32:17 2020-12-25 04:20:52 [INFO] [TRAIN] epoch=99, iter=36500/80000, loss=16.2342, lr=0.005779, batch_cost=1.3427, reader_cost=0.0385 | ETA 16:13:28 2020-12-25 04:22:54 [INFO] [TRAIN] epoch=99, iter=36600/80000, loss=15.7023, lr=0.005767, batch_cost=1.2148, reader_cost=0.0002 | ETA 14:38:43 2020-12-25 04:24:55 [INFO] [TRAIN] epoch=99, iter=36700/80000, loss=15.7876, lr=0.005755, batch_cost=1.2134, reader_cost=0.0002 | ETA 14:35:40 2020-12-25 04:26:55 [INFO] [TRAIN] epoch=99, iter=36800/80000, loss=16.1879, lr=0.005743, batch_cost=1.1984, reader_cost=0.0002 | ETA 14:22:50 2020-12-25 04:29:07 [INFO] [TRAIN] epoch=100, iter=36900/80000, loss=16.2625, lr=0.005731, batch_cost=1.3168, reader_cost=0.0407 | ETA 15:45:53 2020-12-25 04:31:10 [INFO] [TRAIN] epoch=100, iter=37000/80000, loss=15.8249, lr=0.005719, batch_cost=1.2271, reader_cost=0.0011 | ETA 14:39:24 2020-12-25 04:33:11 [INFO] [TRAIN] epoch=100, iter=37100/80000, loss=15.5641, lr=0.005707, batch_cost=1.2090, reader_cost=0.0007 | ETA 14:24:26 2020-12-25 04:35:09 [INFO] [TRAIN] epoch=100, iter=37200/80000, loss=16.2873, lr=0.005695, batch_cost=1.1711, reader_cost=0.0003 | ETA 13:55:21 2020-12-25 04:37:20 [INFO] [TRAIN] epoch=101, iter=37300/80000, loss=16.0896, lr=0.005683, batch_cost=1.3090, reader_cost=0.0441 | ETA 15:31:32 2020-12-25 04:39:22 [INFO] [TRAIN] epoch=101, iter=37400/80000, loss=15.8357, lr=0.005671, batch_cost=1.2220, reader_cost=0.0037 | ETA 14:27:37 2020-12-25 04:41:21 [INFO] [TRAIN] epoch=101, iter=37500/80000, loss=15.7795, lr=0.005660, batch_cost=1.1856, reader_cost=0.0003 | ETA 13:59:46 2020-12-25 04:43:28 [INFO] [TRAIN] epoch=102, iter=37600/80000, loss=16.4481, lr=0.005648, batch_cost=1.2712, reader_cost=0.0445 | ETA 14:58:19 2020-12-25 04:45:25 [INFO] [TRAIN] epoch=102, iter=37700/80000, loss=15.9832, lr=0.005636, batch_cost=1.1673, reader_cost=0.0011 | ETA 13:42:57 2020-12-25 04:47:23 [INFO] [TRAIN] epoch=102, iter=37800/80000, loss=15.4899, lr=0.005624, batch_cost=1.1760, reader_cost=0.0009 | ETA 13:47:05 2020-12-25 04:49:23 [INFO] [TRAIN] epoch=102, iter=37900/80000, loss=15.9136, lr=0.005612, batch_cost=1.1925, reader_cost=0.0005 | ETA 13:56:42 2020-12-25 04:51:29 [INFO] [TRAIN] epoch=103, iter=38000/80000, loss=16.1124, lr=0.005600, batch_cost=1.2602, reader_cost=0.0395 | ETA 14:42:07 2020-12-25 04:51:29 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 04:52:13 [INFO] [EVAL] #Images=500 mIoU=0.7779 Acc=0.9569 Kappa=0.9441 2020-12-25 04:52:13 [INFO] [EVAL] Class IoU: [0.9696 0.8441 0.9116 0.5365 0.6058 0.6811 0.7351 0.8061 0.9232 0.599 0.9448 0.8398 0.6528 0.9508 0.7329 0.8562 0.7442 0.6575 0.7893] 2020-12-25 04:52:13 [INFO] [EVAL] Class Acc: [0.9901 0.9125 0.9397 0.7905 0.8404 0.8502 0.8535 0.8832 0.9532 0.8709 0.9599 0.9076 0.8205 0.9712 0.9386 0.9552 0.8035 0.7847 0.8632] 2020-12-25 04:52:18 [INFO] [EVAL] The model with the best validation mIoU (0.7804) was saved at iter 36000. 2020-12-25 04:54:17 [INFO] [TRAIN] epoch=103, iter=38100/80000, loss=15.8107, lr=0.005588, batch_cost=1.1920, reader_cost=0.0002 | ETA 13:52:24 2020-12-25 04:56:18 [INFO] [TRAIN] epoch=103, iter=38200/80000, loss=15.7580, lr=0.005576, batch_cost=1.2040, reader_cost=0.0008 | ETA 13:58:45 2020-12-25 04:58:15 [INFO] [TRAIN] epoch=103, iter=38300/80000, loss=16.2588, lr=0.005564, batch_cost=1.1751, reader_cost=0.0002 | ETA 13:36:42 2020-12-25 05:00:35 [INFO] [TRAIN] epoch=104, iter=38400/80000, loss=16.3689, lr=0.005552, batch_cost=1.3952, reader_cost=0.0366 | ETA 16:07:21 2020-12-25 05:02:35 [INFO] [TRAIN] epoch=104, iter=38500/80000, loss=15.4315, lr=0.005540, batch_cost=1.2015, reader_cost=0.0007 | ETA 13:51:02 2020-12-25 05:04:37 [INFO] [TRAIN] epoch=104, iter=38600/80000, loss=15.5359, lr=0.005528, batch_cost=1.2127, reader_cost=0.0019 | ETA 13:56:43 2020-12-25 05:06:50 [INFO] [TRAIN] epoch=105, iter=38700/80000, loss=16.2062, lr=0.005515, batch_cost=1.3316, reader_cost=0.0414 | ETA 15:16:33 2020-12-25 05:08:48 [INFO] [TRAIN] epoch=105, iter=38800/80000, loss=15.7985, lr=0.005503, batch_cost=1.1776, reader_cost=0.0002 | ETA 13:28:39 2020-12-25 05:10:48 [INFO] [TRAIN] epoch=105, iter=38900/80000, loss=15.6613, lr=0.005491, batch_cost=1.1972, reader_cost=0.0002 | ETA 13:40:04 2020-12-25 05:12:53 [INFO] [TRAIN] epoch=105, iter=39000/80000, loss=15.7336, lr=0.005479, batch_cost=1.2484, reader_cost=0.0002 | ETA 14:13:03 2020-12-25 05:15:09 [INFO] [TRAIN] epoch=106, iter=39100/80000, loss=15.9563, lr=0.005467, batch_cost=1.3568, reader_cost=0.0486 | ETA 15:24:52 2020-12-25 05:17:07 [INFO] [TRAIN] epoch=106, iter=39200/80000, loss=15.7028, lr=0.005455, batch_cost=1.1716, reader_cost=0.0004 | ETA 13:16:41 2020-12-25 05:19:12 [INFO] [TRAIN] epoch=106, iter=39300/80000, loss=15.9100, lr=0.005443, batch_cost=1.2513, reader_cost=0.0002 | ETA 14:08:47 2020-12-25 05:21:11 [INFO] [TRAIN] epoch=106, iter=39400/80000, loss=15.9204, lr=0.005431, batch_cost=1.1896, reader_cost=0.0003 | ETA 13:24:59 2020-12-25 05:23:22 [INFO] [TRAIN] epoch=107, iter=39500/80000, loss=16.2999, lr=0.005419, batch_cost=1.3048, reader_cost=0.0550 | ETA 14:40:43 2020-12-25 05:25:24 [INFO] [TRAIN] epoch=107, iter=39600/80000, loss=15.6930, lr=0.005407, batch_cost=1.2228, reader_cost=0.0149 | ETA 13:43:20 2020-12-25 05:27:22 [INFO] [TRAIN] epoch=107, iter=39700/80000, loss=15.5200, lr=0.005395, batch_cost=1.1758, reader_cost=0.0020 | ETA 13:09:44 2020-12-25 05:29:22 [INFO] [TRAIN] epoch=107, iter=39800/80000, loss=16.2102, lr=0.005383, batch_cost=1.1979, reader_cost=0.0002 | ETA 13:22:36 2020-12-25 05:31:31 [INFO] [TRAIN] epoch=108, iter=39900/80000, loss=15.8737, lr=0.005371, batch_cost=1.2867, reader_cost=0.0479 | ETA 14:19:55 2020-12-25 05:33:29 [INFO] [TRAIN] epoch=108, iter=40000/80000, loss=15.7725, lr=0.005359, batch_cost=1.1828, reader_cost=0.0002 | ETA 13:08:30 2020-12-25 05:33:29 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 05:34:13 [INFO] [EVAL] #Images=500 mIoU=0.7880 Acc=0.9627 Kappa=0.9516 2020-12-25 05:34:13 [INFO] [EVAL] Class IoU: [0.981 0.8513 0.9298 0.5393 0.6217 0.6966 0.7421 0.8191 0.9267 0.6262 0.948 0.8395 0.639 0.9575 0.7866 0.8564 0.7419 0.6714 0.7986] 2020-12-25 05:34:13 [INFO] [EVAL] Class Acc: [0.9905 0.9156 0.9614 0.7992 0.8404 0.8243 0.8794 0.9271 0.9513 0.8841 0.9623 0.9197 0.8015 0.9797 0.9293 0.9419 0.8113 0.7789 0.8789] 2020-12-25 05:34:20 [INFO] [EVAL] The model with the best validation mIoU (0.7880) was saved at iter 40000. 2020-12-25 05:36:20 [INFO] [TRAIN] epoch=108, iter=40100/80000, loss=15.7100, lr=0.005347, batch_cost=1.1951, reader_cost=0.0002 | ETA 13:14:44 2020-12-25 05:38:30 [INFO] [TRAIN] epoch=109, iter=40200/80000, loss=15.9936, lr=0.005335, batch_cost=1.2966, reader_cost=0.0410 | ETA 14:20:02 2020-12-25 05:40:29 [INFO] [TRAIN] epoch=109, iter=40300/80000, loss=15.8008, lr=0.005323, batch_cost=1.1893, reader_cost=0.0004 | ETA 13:06:56 2020-12-25 05:42:34 [INFO] [TRAIN] epoch=109, iter=40400/80000, loss=15.7694, lr=0.005311, batch_cost=1.2488, reader_cost=0.0023 | ETA 13:44:11 2020-12-25 05:44:38 [INFO] [TRAIN] epoch=109, iter=40500/80000, loss=15.7698, lr=0.005299, batch_cost=1.2406, reader_cost=0.0002 | ETA 13:36:42 2020-12-25 05:46:48 [INFO] [TRAIN] epoch=110, iter=40600/80000, loss=16.1354, lr=0.005287, batch_cost=1.2916, reader_cost=0.0461 | ETA 14:08:10 2020-12-25 05:48:47 [INFO] [TRAIN] epoch=110, iter=40700/80000, loss=15.6337, lr=0.005275, batch_cost=1.1923, reader_cost=0.0149 | ETA 13:00:58 2020-12-25 05:50:46 [INFO] [TRAIN] epoch=110, iter=40800/80000, loss=15.6805, lr=0.005262, batch_cost=1.1812, reader_cost=0.0170 | ETA 12:51:42 2020-12-25 05:52:47 [INFO] [TRAIN] epoch=110, iter=40900/80000, loss=16.0571, lr=0.005250, batch_cost=1.2110, reader_cost=0.0009 | ETA 13:09:09 2020-12-25 05:54:54 [INFO] [TRAIN] epoch=111, iter=41000/80000, loss=16.1227, lr=0.005238, batch_cost=1.2742, reader_cost=0.0369 | ETA 13:48:13 2020-12-25 05:56:53 [INFO] [TRAIN] epoch=111, iter=41100/80000, loss=15.6349, lr=0.005226, batch_cost=1.1843, reader_cost=0.0002 | ETA 12:47:50 2020-12-25 05:58:52 [INFO] [TRAIN] epoch=111, iter=41200/80000, loss=15.5954, lr=0.005214, batch_cost=1.1904, reader_cost=0.0002 | ETA 12:49:46 2020-12-25 06:01:09 [INFO] [TRAIN] epoch=112, iter=41300/80000, loss=16.1871, lr=0.005202, batch_cost=1.3621, reader_cost=0.0361 | ETA 14:38:31 2020-12-25 06:03:07 [INFO] [TRAIN] epoch=112, iter=41400/80000, loss=15.7011, lr=0.005190, batch_cost=1.1851, reader_cost=0.0002 | ETA 12:42:24 2020-12-25 06:05:09 [INFO] [TRAIN] epoch=112, iter=41500/80000, loss=15.8457, lr=0.005178, batch_cost=1.2116, reader_cost=0.0026 | ETA 12:57:28 2020-12-25 06:07:14 [INFO] [TRAIN] epoch=112, iter=41600/80000, loss=15.4403, lr=0.005166, batch_cost=1.2502, reader_cost=0.0003 | ETA 13:20:06 2020-12-25 06:09:26 [INFO] [TRAIN] epoch=113, iter=41700/80000, loss=16.3599, lr=0.005154, batch_cost=1.3146, reader_cost=0.0429 | ETA 13:59:08 2020-12-25 06:11:26 [INFO] [TRAIN] epoch=113, iter=41800/80000, loss=15.5421, lr=0.005141, batch_cost=1.1971, reader_cost=0.0013 | ETA 12:42:10 2020-12-25 06:13:23 [INFO] [TRAIN] epoch=113, iter=41900/80000, loss=15.5598, lr=0.005129, batch_cost=1.1699, reader_cost=0.0002 | ETA 12:22:51 2020-12-25 06:15:29 [INFO] [TRAIN] epoch=113, iter=42000/80000, loss=15.8160, lr=0.005117, batch_cost=1.2558, reader_cost=0.0002 | ETA 13:15:19 2020-12-25 06:15:29 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 06:16:13 [INFO] [EVAL] #Images=500 mIoU=0.7761 Acc=0.9635 Kappa=0.9527 2020-12-25 06:16:13 [INFO] [EVAL] Class IoU: [0.9833 0.8648 0.9319 0.6027 0.6135 0.6947 0.7446 0.8277 0.9262 0.6174 0.9528 0.8376 0.6191 0.9556 0.6213 0.7986 0.7016 0.657 0.7964] 2020-12-25 06:16:13 [INFO] [EVAL] Class Acc: [0.9919 0.9264 0.9639 0.827 0.8357 0.8436 0.8349 0.9122 0.9502 0.8886 0.9751 0.899 0.8498 0.9722 0.849 0.8248 0.8456 0.7936 0.8769] 2020-12-25 06:16:18 [INFO] [EVAL] The model with the best validation mIoU (0.7880) was saved at iter 40000. 2020-12-25 06:18:33 [INFO] [TRAIN] epoch=114, iter=42100/80000, loss=16.1619, lr=0.005105, batch_cost=1.3552, reader_cost=0.0417 | ETA 14:16:02 2020-12-25 06:20:33 [INFO] [TRAIN] epoch=114, iter=42200/80000, loss=15.6905, lr=0.005093, batch_cost=1.1908, reader_cost=0.0034 | ETA 12:30:13 2020-12-25 06:22:32 [INFO] [TRAIN] epoch=114, iter=42300/80000, loss=15.5204, lr=0.005081, batch_cost=1.1885, reader_cost=0.0002 | ETA 12:26:47 2020-12-25 06:24:40 [INFO] [TRAIN] epoch=114, iter=42400/80000, loss=15.9802, lr=0.005069, batch_cost=1.2819, reader_cost=0.0656 | ETA 13:23:21 2020-12-25 06:26:55 [INFO] [TRAIN] epoch=115, iter=42500/80000, loss=15.8986, lr=0.005057, batch_cost=1.3455, reader_cost=0.0383 | ETA 14:00:56 2020-12-25 06:28:57 [INFO] [TRAIN] epoch=115, iter=42600/80000, loss=15.4292, lr=0.005044, batch_cost=1.2215, reader_cost=0.0005 | ETA 12:41:23 2020-12-25 06:30:56 [INFO] [TRAIN] epoch=115, iter=42700/80000, loss=15.7221, lr=0.005032, batch_cost=1.1877, reader_cost=0.0002 | ETA 12:18:22 2020-12-25 06:33:08 [INFO] [TRAIN] epoch=116, iter=42800/80000, loss=16.2066, lr=0.005020, batch_cost=1.3198, reader_cost=0.0578 | ETA 13:38:15 2020-12-25 06:35:10 [INFO] [TRAIN] epoch=116, iter=42900/80000, loss=15.5676, lr=0.005008, batch_cost=1.2084, reader_cost=0.0002 | ETA 12:27:13 2020-12-25 06:37:17 [INFO] [TRAIN] epoch=116, iter=43000/80000, loss=15.6489, lr=0.004996, batch_cost=1.2742, reader_cost=0.0002 | ETA 13:05:46 2020-12-25 06:39:20 [INFO] [TRAIN] epoch=116, iter=43100/80000, loss=15.5651, lr=0.004984, batch_cost=1.2256, reader_cost=0.0002 | ETA 12:33:45 2020-12-25 06:41:35 [INFO] [TRAIN] epoch=117, iter=43200/80000, loss=16.0959, lr=0.004972, batch_cost=1.3536, reader_cost=0.0715 | ETA 13:50:12 2020-12-25 06:43:36 [INFO] [TRAIN] epoch=117, iter=43300/80000, loss=15.6730, lr=0.004959, batch_cost=1.2033, reader_cost=0.0002 | ETA 12:15:59 2020-12-25 06:45:35 [INFO] [TRAIN] epoch=117, iter=43400/80000, loss=15.5472, lr=0.004947, batch_cost=1.1875, reader_cost=0.0003 | ETA 12:04:22 2020-12-25 06:47:34 [INFO] [TRAIN] epoch=117, iter=43500/80000, loss=16.0447, lr=0.004935, batch_cost=1.1890, reader_cost=0.0008 | ETA 12:03:16 2020-12-25 06:49:45 [INFO] [TRAIN] epoch=118, iter=43600/80000, loss=16.2639, lr=0.004923, batch_cost=1.3039, reader_cost=0.0394 | ETA 13:11:03 2020-12-25 06:51:44 [INFO] [TRAIN] epoch=118, iter=43700/80000, loss=15.6957, lr=0.004911, batch_cost=1.1868, reader_cost=0.0002 | ETA 11:57:59 2020-12-25 06:53:47 [INFO] [TRAIN] epoch=118, iter=43800/80000, loss=15.6234, lr=0.004899, batch_cost=1.2325, reader_cost=0.0002 | ETA 12:23:36 2020-12-25 06:56:00 [INFO] [TRAIN] epoch=119, iter=43900/80000, loss=15.9906, lr=0.004886, batch_cost=1.3255, reader_cost=0.0404 | ETA 13:17:31 2020-12-25 06:57:57 [INFO] [TRAIN] epoch=119, iter=44000/80000, loss=15.8247, lr=0.004874, batch_cost=1.1735, reader_cost=0.0004 | ETA 11:44:06 2020-12-25 06:57:58 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 06:58:42 [INFO] [EVAL] #Images=500 mIoU=0.7848 Acc=0.9636 Kappa=0.9528 2020-12-25 06:58:42 [INFO] [EVAL] Class IoU: [0.9828 0.8705 0.9302 0.5459 0.6274 0.6975 0.74 0.8208 0.9281 0.6658 0.9503 0.8394 0.622 0.952 0.6769 0.8464 0.758 0.6595 0.7979] 2020-12-25 06:58:42 [INFO] [EVAL] Class Acc: [0.9918 0.9269 0.965 0.8114 0.8109 0.8334 0.8404 0.9264 0.9548 0.8442 0.9667 0.8955 0.8248 0.97 0.8871 0.912 0.8268 0.772 0.8705] 2020-12-25 06:58:46 [INFO] [EVAL] The model with the best validation mIoU (0.7880) was saved at iter 40000. 2020-12-25 07:00:48 [INFO] [TRAIN] epoch=119, iter=44100/80000, loss=15.5975, lr=0.004862, batch_cost=1.2135, reader_cost=0.0003 | ETA 12:06:03 2020-12-25 07:02:50 [INFO] [TRAIN] epoch=119, iter=44200/80000, loss=15.5870, lr=0.004850, batch_cost=1.2145, reader_cost=0.0005 | ETA 12:04:40 2020-12-25 07:05:04 [INFO] [TRAIN] epoch=120, iter=44300/80000, loss=16.2913, lr=0.004838, batch_cost=1.3355, reader_cost=0.0467 | ETA 13:14:38 2020-12-25 07:07:03 [INFO] [TRAIN] epoch=120, iter=44400/80000, loss=15.6503, lr=0.004825, batch_cost=1.1914, reader_cost=0.0016 | ETA 11:46:52 2020-12-25 07:09:03 [INFO] [TRAIN] epoch=120, iter=44500/80000, loss=15.4874, lr=0.004813, batch_cost=1.1936, reader_cost=0.0002 | ETA 11:46:11 2020-12-25 07:11:01 [INFO] [TRAIN] epoch=120, iter=44600/80000, loss=16.0067, lr=0.004801, batch_cost=1.1794, reader_cost=0.0014 | ETA 11:35:52 2020-12-25 07:13:11 [INFO] [TRAIN] epoch=121, iter=44700/80000, loss=16.1956, lr=0.004789, batch_cost=1.3051, reader_cost=0.0379 | ETA 12:47:49 2020-12-25 07:15:11 [INFO] [TRAIN] epoch=121, iter=44800/80000, loss=15.7926, lr=0.004777, batch_cost=1.1951, reader_cost=0.0009 | ETA 11:41:07 2020-12-25 07:17:09 [INFO] [TRAIN] epoch=121, iter=44900/80000, loss=15.4990, lr=0.004764, batch_cost=1.1797, reader_cost=0.0014 | ETA 11:30:08 2020-12-25 07:19:08 [INFO] [TRAIN] epoch=121, iter=45000/80000, loss=15.9745, lr=0.004752, batch_cost=1.1817, reader_cost=0.0002 | ETA 11:29:20 2020-12-25 07:21:22 [INFO] [TRAIN] epoch=122, iter=45100/80000, loss=15.9167, lr=0.004740, batch_cost=1.3446, reader_cost=0.0461 | ETA 13:02:05 2020-12-25 07:23:24 [INFO] [TRAIN] epoch=122, iter=45200/80000, loss=15.5162, lr=0.004728, batch_cost=1.2125, reader_cost=0.0002 | ETA 11:43:15 2020-12-25 07:25:24 [INFO] [TRAIN] epoch=122, iter=45300/80000, loss=15.4968, lr=0.004715, batch_cost=1.1922, reader_cost=0.0024 | ETA 11:29:28 2020-12-25 07:27:34 [INFO] [TRAIN] epoch=123, iter=45400/80000, loss=15.9424, lr=0.004703, batch_cost=1.3055, reader_cost=0.0415 | ETA 12:32:51 2020-12-25 07:29:31 [INFO] [TRAIN] epoch=123, iter=45500/80000, loss=15.7584, lr=0.004691, batch_cost=1.1635, reader_cost=0.0018 | ETA 11:09:00 2020-12-25 07:31:29 [INFO] [TRAIN] epoch=123, iter=45600/80000, loss=15.6890, lr=0.004679, batch_cost=1.1790, reader_cost=0.0014 | ETA 11:15:58 2020-12-25 07:33:30 [INFO] [TRAIN] epoch=123, iter=45700/80000, loss=15.6599, lr=0.004667, batch_cost=1.2049, reader_cost=0.0002 | ETA 11:28:47 2020-12-25 07:35:39 [INFO] [TRAIN] epoch=124, iter=45800/80000, loss=16.2557, lr=0.004654, batch_cost=1.2892, reader_cost=0.0395 | ETA 12:14:52 2020-12-25 07:37:42 [INFO] [TRAIN] epoch=124, iter=45900/80000, loss=15.6816, lr=0.004642, batch_cost=1.2283, reader_cost=0.0070 | ETA 11:38:05 2020-12-25 07:39:45 [INFO] [TRAIN] epoch=124, iter=46000/80000, loss=15.5704, lr=0.004630, batch_cost=1.2323, reader_cost=0.0021 | ETA 11:38:16 2020-12-25 07:39:45 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 07:40:30 [INFO] [EVAL] #Images=500 mIoU=0.7741 Acc=0.9586 Kappa=0.9461 2020-12-25 07:40:30 [INFO] [EVAL] Class IoU: [0.9786 0.8442 0.9164 0.4528 0.5811 0.674 0.7269 0.8142 0.9189 0.6338 0.9441 0.8313 0.6286 0.9554 0.7264 0.8226 0.7916 0.6743 0.7936] 2020-12-25 07:40:30 [INFO] [EVAL] Class Acc: [0.9893 0.9241 0.9409 0.8319 0.8322 0.8419 0.8688 0.9269 0.9571 0.7737 0.9656 0.8981 0.8339 0.9788 0.9092 0.9008 0.9042 0.7817 0.8758] 2020-12-25 07:40:35 [INFO] [EVAL] The model with the best validation mIoU (0.7880) was saved at iter 40000. 2020-12-25 07:42:30 [INFO] [TRAIN] epoch=124, iter=46100/80000, loss=15.6948, lr=0.004618, batch_cost=1.1563, reader_cost=0.0002 | ETA 10:53:20 2020-12-25 07:44:40 [INFO] [TRAIN] epoch=125, iter=46200/80000, loss=15.9293, lr=0.004605, batch_cost=1.2899, reader_cost=0.0489 | ETA 12:06:37 2020-12-25 07:46:37 [INFO] [TRAIN] epoch=125, iter=46300/80000, loss=15.5154, lr=0.004593, batch_cost=1.1680, reader_cost=0.0002 | ETA 10:56:00 2020-12-25 07:48:37 [INFO] [TRAIN] epoch=125, iter=46400/80000, loss=15.3802, lr=0.004581, batch_cost=1.2057, reader_cost=0.0002 | ETA 11:15:12 2020-12-25 07:50:37 [INFO] [TRAIN] epoch=125, iter=46500/80000, loss=15.7480, lr=0.004568, batch_cost=1.1908, reader_cost=0.0012 | ETA 11:04:52 2020-12-25 07:52:47 [INFO] [TRAIN] epoch=126, iter=46600/80000, loss=15.8904, lr=0.004556, batch_cost=1.3024, reader_cost=0.0402 | ETA 12:04:59 2020-12-25 07:54:47 [INFO] [TRAIN] epoch=126, iter=46700/80000, loss=15.5050, lr=0.004544, batch_cost=1.1981, reader_cost=0.0130 | ETA 11:04:58 2020-12-25 07:56:49 [INFO] [TRAIN] epoch=126, iter=46800/80000, loss=15.4962, lr=0.004532, batch_cost=1.2130, reader_cost=0.0002 | ETA 11:11:10 2020-12-25 07:59:00 [INFO] [TRAIN] epoch=127, iter=46900/80000, loss=16.1467, lr=0.004519, batch_cost=1.3157, reader_cost=0.0396 | ETA 12:05:49 2020-12-25 08:01:00 [INFO] [TRAIN] epoch=127, iter=47000/80000, loss=15.5640, lr=0.004507, batch_cost=1.1931, reader_cost=0.0005 | ETA 10:56:10 2020-12-25 08:03:00 [INFO] [TRAIN] epoch=127, iter=47100/80000, loss=15.3713, lr=0.004495, batch_cost=1.1929, reader_cost=0.0004 | ETA 10:54:07 2020-12-25 08:04:56 [INFO] [TRAIN] epoch=127, iter=47200/80000, loss=15.7105, lr=0.004482, batch_cost=1.1640, reader_cost=0.0008 | ETA 10:36:20 2020-12-25 08:07:11 [INFO] [TRAIN] epoch=128, iter=47300/80000, loss=16.1867, lr=0.004470, batch_cost=1.3406, reader_cost=0.0383 | ETA 12:10:38 2020-12-25 08:09:07 [INFO] [TRAIN] epoch=128, iter=47400/80000, loss=15.4227, lr=0.004458, batch_cost=1.1647, reader_cost=0.0004 | ETA 10:32:47 2020-12-25 08:11:03 [INFO] [TRAIN] epoch=128, iter=47500/80000, loss=15.1681, lr=0.004446, batch_cost=1.1579, reader_cost=0.0002 | ETA 10:27:11 2020-12-25 08:13:01 [INFO] [TRAIN] epoch=128, iter=47600/80000, loss=15.9243, lr=0.004433, batch_cost=1.1713, reader_cost=0.0002 | ETA 10:32:30 2020-12-25 08:15:15 [INFO] [TRAIN] epoch=129, iter=47700/80000, loss=15.9903, lr=0.004421, batch_cost=1.3387, reader_cost=0.0414 | ETA 12:00:39 2020-12-25 08:17:13 [INFO] [TRAIN] epoch=129, iter=47800/80000, loss=15.2672, lr=0.004409, batch_cost=1.1796, reader_cost=0.0002 | ETA 10:33:04 2020-12-25 08:19:17 [INFO] [TRAIN] epoch=129, iter=47900/80000, loss=15.4979, lr=0.004396, batch_cost=1.2420, reader_cost=0.0005 | ETA 11:04:27 2020-12-25 08:21:31 [INFO] [TRAIN] epoch=130, iter=48000/80000, loss=16.0621, lr=0.004384, batch_cost=1.3391, reader_cost=0.0649 | ETA 11:54:12 2020-12-25 08:21:32 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 08:22:17 [INFO] [EVAL] #Images=500 mIoU=0.7457 Acc=0.9602 Kappa=0.9484 2020-12-25 08:22:17 [INFO] [EVAL] Class IoU: [0.9791 0.8569 0.9244 0.4595 0.6067 0.6892 0.7395 0.821 0.925 0.641 0.9406 0.8427 0.6387 0.9572 0.5992 0.6789 0.3896 0.6846 0.7938] 2020-12-25 08:22:17 [INFO] [EVAL] Class Acc: [0.9881 0.9111 0.9614 0.8475 0.8082 0.8621 0.8132 0.9133 0.9494 0.871 0.9686 0.9116 0.8592 0.9745 0.912 0.7017 0.9103 0.8449 0.8492] 2020-12-25 08:22:21 [INFO] [EVAL] The model with the best validation mIoU (0.7880) was saved at iter 40000. 2020-12-25 08:24:20 [INFO] [TRAIN] epoch=130, iter=48100/80000, loss=15.6646, lr=0.004372, batch_cost=1.1820, reader_cost=0.0002 | ETA 10:28:25 2020-12-25 08:26:26 [INFO] [TRAIN] epoch=130, iter=48200/80000, loss=15.5819, lr=0.004359, batch_cost=1.2599, reader_cost=0.0002 | ETA 11:07:43 2020-12-25 08:28:29 [INFO] [TRAIN] epoch=130, iter=48300/80000, loss=15.5529, lr=0.004347, batch_cost=1.2240, reader_cost=0.0005 | ETA 10:46:41 2020-12-25 08:30:36 [INFO] [TRAIN] epoch=131, iter=48400/80000, loss=16.3525, lr=0.004335, batch_cost=1.2689, reader_cost=0.0421 | ETA 11:08:16 2020-12-25 08:32:36 [INFO] [TRAIN] epoch=131, iter=48500/80000, loss=15.6559, lr=0.004322, batch_cost=1.2048, reader_cost=0.0002 | ETA 10:32:30 2020-12-25 08:34:38 [INFO] [TRAIN] epoch=131, iter=48600/80000, loss=15.4612, lr=0.004310, batch_cost=1.2152, reader_cost=0.0284 | ETA 10:35:57 2020-12-25 08:36:39 [INFO] [TRAIN] epoch=131, iter=48700/80000, loss=15.8361, lr=0.004298, batch_cost=1.2112, reader_cost=0.0008 | ETA 10:31:51 2020-12-25 08:38:56 [INFO] [TRAIN] epoch=132, iter=48800/80000, loss=16.1095, lr=0.004285, batch_cost=1.3651, reader_cost=0.0363 | ETA 11:49:52 2020-12-25 08:40:57 [INFO] [TRAIN] epoch=132, iter=48900/80000, loss=15.8642, lr=0.004273, batch_cost=1.2064, reader_cost=0.0002 | ETA 10:25:20 2020-12-25 08:42:54 [INFO] [TRAIN] epoch=132, iter=49000/80000, loss=15.6264, lr=0.004260, batch_cost=1.1735, reader_cost=0.0051 | ETA 10:06:18 2020-12-25 08:44:51 [INFO] [TRAIN] epoch=132, iter=49100/80000, loss=15.9027, lr=0.004248, batch_cost=1.1640, reader_cost=0.0002 | ETA 09:59:27 2020-12-25 08:47:09 [INFO] [TRAIN] epoch=133, iter=49200/80000, loss=15.9251, lr=0.004236, batch_cost=1.3749, reader_cost=0.0800 | ETA 11:45:45 2020-12-25 08:49:08 [INFO] [TRAIN] epoch=133, iter=49300/80000, loss=15.6211, lr=0.004223, batch_cost=1.1927, reader_cost=0.0002 | ETA 10:10:14 2020-12-25 08:51:10 [INFO] [TRAIN] epoch=133, iter=49400/80000, loss=15.4743, lr=0.004211, batch_cost=1.2158, reader_cost=0.0016 | ETA 10:20:02 2020-12-25 08:53:24 [INFO] [TRAIN] epoch=134, iter=49500/80000, loss=16.1712, lr=0.004199, batch_cost=1.3384, reader_cost=0.0591 | ETA 11:20:21 2020-12-25 08:55:23 [INFO] [TRAIN] epoch=134, iter=49600/80000, loss=15.5282, lr=0.004186, batch_cost=1.1800, reader_cost=0.0002 | ETA 09:57:51 2020-12-25 08:57:21 [INFO] [TRAIN] epoch=134, iter=49700/80000, loss=15.6042, lr=0.004174, batch_cost=1.1855, reader_cost=0.0002 | ETA 09:58:39 2020-12-25 08:59:18 [INFO] [TRAIN] epoch=134, iter=49800/80000, loss=15.7446, lr=0.004161, batch_cost=1.1682, reader_cost=0.0002 | ETA 09:47:58 2020-12-25 09:01:31 [INFO] [TRAIN] epoch=135, iter=49900/80000, loss=15.9406, lr=0.004149, batch_cost=1.3227, reader_cost=0.0475 | ETA 11:03:32 2020-12-25 09:03:35 [INFO] [TRAIN] epoch=135, iter=50000/80000, loss=15.6218, lr=0.004137, batch_cost=1.2349, reader_cost=0.0002 | ETA 10:17:26 2020-12-25 09:03:35 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 09:04:20 [INFO] [EVAL] #Images=500 mIoU=0.7881 Acc=0.9632 Kappa=0.9522 2020-12-25 09:04:20 [INFO] [EVAL] Class IoU: [0.9824 0.8623 0.9288 0.5213 0.6318 0.6899 0.7403 0.821 0.926 0.6503 0.9481 0.8464 0.6622 0.9588 0.7641 0.8264 0.7686 0.6463 0.7986] 2020-12-25 09:04:20 [INFO] [EVAL] Class Acc: [0.9936 0.913 0.962 0.7656 0.8475 0.8659 0.8724 0.9143 0.9477 0.8656 0.9671 0.9024 0.7891 0.9803 0.9254 0.9017 0.8272 0.8752 0.873 ] 2020-12-25 09:04:27 [INFO] [EVAL] The model with the best validation mIoU (0.7881) was saved at iter 50000. 2020-12-25 09:06:30 [INFO] [TRAIN] epoch=135, iter=50100/80000, loss=15.4369, lr=0.004124, batch_cost=1.2225, reader_cost=0.0007 | ETA 10:09:13 2020-12-25 09:08:35 [INFO] [TRAIN] epoch=135, iter=50200/80000, loss=16.0381, lr=0.004112, batch_cost=1.2474, reader_cost=0.0002 | ETA 10:19:31 2020-12-25 09:10:44 [INFO] [TRAIN] epoch=136, iter=50300/80000, loss=15.8187, lr=0.004099, batch_cost=1.2941, reader_cost=0.0457 | ETA 10:40:33 2020-12-25 09:12:48 [INFO] [TRAIN] epoch=136, iter=50400/80000, loss=15.7304, lr=0.004087, batch_cost=1.2394, reader_cost=0.0004 | ETA 10:11:25 2020-12-25 09:14:48 [INFO] [TRAIN] epoch=136, iter=50500/80000, loss=15.4887, lr=0.004074, batch_cost=1.1916, reader_cost=0.0377 | ETA 09:45:52 2020-12-25 09:17:05 [INFO] [TRAIN] epoch=137, iter=50600/80000, loss=16.0381, lr=0.004062, batch_cost=1.3687, reader_cost=0.0450 | ETA 11:10:38 2020-12-25 09:19:05 [INFO] [TRAIN] epoch=137, iter=50700/80000, loss=15.6790, lr=0.004050, batch_cost=1.1961, reader_cost=0.0164 | ETA 09:44:06 2020-12-25 09:21:07 [INFO] [TRAIN] epoch=137, iter=50800/80000, loss=15.6782, lr=0.004037, batch_cost=1.2183, reader_cost=0.0003 | ETA 09:52:54 2020-12-25 09:23:04 [INFO] [TRAIN] epoch=137, iter=50900/80000, loss=15.7557, lr=0.004025, batch_cost=1.1658, reader_cost=0.0013 | ETA 09:25:23 2020-12-25 09:25:14 [INFO] [TRAIN] epoch=138, iter=51000/80000, loss=15.8369, lr=0.004012, batch_cost=1.3016, reader_cost=0.0397 | ETA 10:29:07 2020-12-25 09:27:15 [INFO] [TRAIN] epoch=138, iter=51100/80000, loss=15.5590, lr=0.004000, batch_cost=1.2091, reader_cost=0.0002 | ETA 09:42:22 2020-12-25 09:29:14 [INFO] [TRAIN] epoch=138, iter=51200/80000, loss=15.6659, lr=0.003987, batch_cost=1.1841, reader_cost=0.0002 | ETA 09:28:23 2020-12-25 09:31:22 [INFO] [TRAIN] epoch=138, iter=51300/80000, loss=15.6258, lr=0.003975, batch_cost=1.2813, reader_cost=0.0153 | ETA 10:12:52 2020-12-25 09:33:39 [INFO] [TRAIN] epoch=139, iter=51400/80000, loss=16.1556, lr=0.003962, batch_cost=1.3661, reader_cost=0.0397 | ETA 10:51:11 2020-12-25 09:35:38 [INFO] [TRAIN] epoch=139, iter=51500/80000, loss=15.8637, lr=0.003950, batch_cost=1.1846, reader_cost=0.0009 | ETA 09:22:40 2020-12-25 09:37:44 [INFO] [TRAIN] epoch=139, iter=51600/80000, loss=15.2626, lr=0.003937, batch_cost=1.2615, reader_cost=0.0016 | ETA 09:57:08 2020-12-25 09:39:48 [INFO] [TRAIN] epoch=139, iter=51700/80000, loss=15.8993, lr=0.003925, batch_cost=1.2404, reader_cost=0.0003 | ETA 09:45:03 2020-12-25 09:42:10 [INFO] [TRAIN] epoch=140, iter=51800/80000, loss=15.9623, lr=0.003913, batch_cost=1.4116, reader_cost=0.0393 | ETA 11:03:27 2020-12-25 09:44:14 [INFO] [TRAIN] epoch=140, iter=51900/80000, loss=15.7420, lr=0.003900, batch_cost=1.2362, reader_cost=0.0002 | ETA 09:38:56 2020-12-25 09:46:15 [INFO] [TRAIN] epoch=140, iter=52000/80000, loss=15.5478, lr=0.003888, batch_cost=1.2159, reader_cost=0.0002 | ETA 09:27:24 2020-12-25 09:46:16 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 09:47:02 [INFO] [EVAL] #Images=500 mIoU=0.7958 Acc=0.9637 Kappa=0.9529 2020-12-25 09:47:02 [INFO] [EVAL] Class IoU: [0.9826 0.8598 0.9294 0.4765 0.6307 0.705 0.7525 0.8281 0.9286 0.6325 0.9504 0.8459 0.6522 0.9589 0.8331 0.8793 0.7879 0.686 0.8006] 2020-12-25 09:47:02 [INFO] [EVAL] Class Acc: [0.9916 0.9216 0.957 0.8373 0.8414 0.8463 0.8804 0.908 0.9587 0.8079 0.9656 0.9163 0.8355 0.9776 0.9096 0.9457 0.8418 0.8663 0.8742] 2020-12-25 09:47:09 [INFO] [EVAL] The model with the best validation mIoU (0.7958) was saved at iter 52000. 2020-12-25 09:49:22 [INFO] [TRAIN] epoch=141, iter=52100/80000, loss=16.0013, lr=0.003875, batch_cost=1.3331, reader_cost=0.0388 | ETA 10:19:52 2020-12-25 09:51:20 [INFO] [TRAIN] epoch=141, iter=52200/80000, loss=15.6198, lr=0.003863, batch_cost=1.1715, reader_cost=0.0002 | ETA 09:02:47 2020-12-25 09:53:20 [INFO] [TRAIN] epoch=141, iter=52300/80000, loss=15.7584, lr=0.003850, batch_cost=1.1977, reader_cost=0.0002 | ETA 09:12:57 2020-12-25 09:55:19 [INFO] [TRAIN] epoch=141, iter=52400/80000, loss=15.6906, lr=0.003838, batch_cost=1.1934, reader_cost=0.0004 | ETA 09:08:58 2020-12-25 09:57:34 [INFO] [TRAIN] epoch=142, iter=52500/80000, loss=16.1657, lr=0.003825, batch_cost=1.3416, reader_cost=0.0407 | ETA 10:14:53 2020-12-25 09:59:37 [INFO] [TRAIN] epoch=142, iter=52600/80000, loss=15.6366, lr=0.003812, batch_cost=1.2312, reader_cost=0.0011 | ETA 09:22:15 2020-12-25 10:01:39 [INFO] [TRAIN] epoch=142, iter=52700/80000, loss=15.5838, lr=0.003800, batch_cost=1.2183, reader_cost=0.0009 | ETA 09:14:19 2020-12-25 10:03:49 [INFO] [TRAIN] epoch=142, iter=52800/80000, loss=15.8940, lr=0.003787, batch_cost=1.2971, reader_cost=0.0006 | ETA 09:48:02 2020-12-25 10:06:03 [INFO] [TRAIN] epoch=143, iter=52900/80000, loss=16.0234, lr=0.003775, batch_cost=1.3416, reader_cost=0.0504 | ETA 10:05:58 2020-12-25 10:08:00 [INFO] [TRAIN] epoch=143, iter=53000/80000, loss=15.4445, lr=0.003762, batch_cost=1.1667, reader_cost=0.0005 | ETA 08:45:00 2020-12-25 10:10:01 [INFO] [TRAIN] epoch=143, iter=53100/80000, loss=15.6818, lr=0.003750, batch_cost=1.2013, reader_cost=0.0003 | ETA 08:58:34 2020-12-25 10:12:13 [INFO] [TRAIN] epoch=144, iter=53200/80000, loss=15.8538, lr=0.003737, batch_cost=1.3167, reader_cost=0.0493 | ETA 09:48:08 2020-12-25 10:14:16 [INFO] [TRAIN] epoch=144, iter=53300/80000, loss=16.0143, lr=0.003725, batch_cost=1.2336, reader_cost=0.0010 | ETA 09:08:56 2020-12-25 10:16:17 [INFO] [TRAIN] epoch=144, iter=53400/80000, loss=15.7074, lr=0.003712, batch_cost=1.2030, reader_cost=0.0007 | ETA 08:53:19 2020-12-25 10:18:17 [INFO] [TRAIN] epoch=144, iter=53500/80000, loss=15.6464, lr=0.003700, batch_cost=1.1969, reader_cost=0.0002 | ETA 08:48:39 2020-12-25 10:20:35 [INFO] [TRAIN] epoch=145, iter=53600/80000, loss=15.9764, lr=0.003687, batch_cost=1.3784, reader_cost=0.0421 | ETA 10:06:30 2020-12-25 10:22:43 [INFO] [TRAIN] epoch=145, iter=53700/80000, loss=15.8488, lr=0.003674, batch_cost=1.2843, reader_cost=0.0005 | ETA 09:22:58 2020-12-25 10:24:44 [INFO] [TRAIN] epoch=145, iter=53800/80000, loss=15.6511, lr=0.003662, batch_cost=1.2054, reader_cost=0.0013 | ETA 08:46:22 2020-12-25 10:26:46 [INFO] [TRAIN] epoch=145, iter=53900/80000, loss=15.7864, lr=0.003649, batch_cost=1.2138, reader_cost=0.0002 | ETA 08:47:59 2020-12-25 10:28:58 [INFO] [TRAIN] epoch=146, iter=54000/80000, loss=16.2848, lr=0.003637, batch_cost=1.3201, reader_cost=0.0421 | ETA 09:32:01 2020-12-25 10:28:58 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 10:29:43 [INFO] [EVAL] #Images=500 mIoU=0.7843 Acc=0.9631 Kappa=0.9521 2020-12-25 10:29:43 [INFO] [EVAL] Class IoU: [0.9827 0.8624 0.9303 0.5235 0.6273 0.6922 0.7523 0.8229 0.9281 0.6662 0.9524 0.8389 0.6507 0.9465 0.6419 0.8402 0.7796 0.6712 0.7913] 2020-12-25 10:29:43 [INFO] [EVAL] Class Acc: [0.9902 0.9395 0.9629 0.8062 0.8057 0.8489 0.8586 0.9253 0.954 0.8013 0.9744 0.897 0.7389 0.9641 0.9278 0.9774 0.8345 0.8605 0.8684] 2020-12-25 10:29:48 [INFO] [EVAL] The model with the best validation mIoU (0.7958) was saved at iter 52000. 2020-12-25 10:31:53 [INFO] [TRAIN] epoch=146, iter=54100/80000, loss=15.3831, lr=0.003624, batch_cost=1.2478, reader_cost=0.0006 | ETA 08:58:38 2020-12-25 10:33:57 [INFO] [TRAIN] epoch=146, iter=54200/80000, loss=15.3097, lr=0.003612, batch_cost=1.2344, reader_cost=0.0002 | ETA 08:50:47 2020-12-25 10:35:56 [INFO] [TRAIN] epoch=146, iter=54300/80000, loss=15.9648, lr=0.003599, batch_cost=1.1951, reader_cost=0.0002 | ETA 08:31:53 2020-12-25 10:38:14 [INFO] [TRAIN] epoch=147, iter=54400/80000, loss=16.0945, lr=0.003586, batch_cost=1.3764, reader_cost=0.0510 | ETA 09:47:14 2020-12-25 10:40:16 [INFO] [TRAIN] epoch=147, iter=54500/80000, loss=15.3647, lr=0.003574, batch_cost=1.2165, reader_cost=0.0003 | ETA 08:37:00 2020-12-25 10:42:17 [INFO] [TRAIN] epoch=147, iter=54600/80000, loss=15.4559, lr=0.003561, batch_cost=1.2085, reader_cost=0.0003 | ETA 08:31:35 2020-12-25 10:44:30 [INFO] [TRAIN] epoch=148, iter=54700/80000, loss=16.0285, lr=0.003548, batch_cost=1.3249, reader_cost=0.0370 | ETA 09:18:39 2020-12-25 10:46:28 [INFO] [TRAIN] epoch=148, iter=54800/80000, loss=15.5706, lr=0.003536, batch_cost=1.1784, reader_cost=0.0002 | ETA 08:14:55 2020-12-25 10:48:32 [INFO] [TRAIN] epoch=148, iter=54900/80000, loss=15.6138, lr=0.003523, batch_cost=1.2344, reader_cost=0.0002 | ETA 08:36:24 2020-12-25 10:50:36 [INFO] [TRAIN] epoch=148, iter=55000/80000, loss=15.5478, lr=0.003511, batch_cost=1.2422, reader_cost=0.0003 | ETA 08:37:35 2020-12-25 10:52:52 [INFO] [TRAIN] epoch=149, iter=55100/80000, loss=16.1814, lr=0.003498, batch_cost=1.3507, reader_cost=0.0397 | ETA 09:20:33 2020-12-25 10:54:55 [INFO] [TRAIN] epoch=149, iter=55200/80000, loss=15.4977, lr=0.003485, batch_cost=1.2365, reader_cost=0.0006 | ETA 08:31:05 2020-12-25 10:56:58 [INFO] [TRAIN] epoch=149, iter=55300/80000, loss=15.5242, lr=0.003473, batch_cost=1.2213, reader_cost=0.0005 | ETA 08:22:46 2020-12-25 10:59:02 [INFO] [TRAIN] epoch=149, iter=55400/80000, loss=15.8678, lr=0.003460, batch_cost=1.2375, reader_cost=0.0009 | ETA 08:27:22 2020-12-25 11:01:18 [INFO] [TRAIN] epoch=150, iter=55500/80000, loss=15.9983, lr=0.003447, batch_cost=1.3555, reader_cost=0.0403 | ETA 09:13:29 2020-12-25 11:03:20 [INFO] [TRAIN] epoch=150, iter=55600/80000, loss=15.4834, lr=0.003435, batch_cost=1.2190, reader_cost=0.0006 | ETA 08:15:43 2020-12-25 11:05:21 [INFO] [TRAIN] epoch=150, iter=55700/80000, loss=15.4642, lr=0.003422, batch_cost=1.2125, reader_cost=0.0007 | ETA 08:11:03 2020-12-25 11:07:25 [INFO] [TRAIN] epoch=150, iter=55800/80000, loss=15.8039, lr=0.003409, batch_cost=1.2333, reader_cost=0.0004 | ETA 08:17:26 2020-12-25 11:09:44 [INFO] [TRAIN] epoch=151, iter=55900/80000, loss=15.8297, lr=0.003397, batch_cost=1.3895, reader_cost=0.0553 | ETA 09:18:08 2020-12-25 11:11:48 [INFO] [TRAIN] epoch=151, iter=56000/80000, loss=15.6525, lr=0.003384, batch_cost=1.2383, reader_cost=0.0006 | ETA 08:15:18 2020-12-25 11:11:48 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 11:12:35 [INFO] [EVAL] #Images=500 mIoU=0.7758 Acc=0.9601 Kappa=0.9482 2020-12-25 11:12:35 [INFO] [EVAL] Class IoU: [0.9794 0.8508 0.9243 0.4392 0.6276 0.6735 0.7278 0.8139 0.9231 0.6297 0.9512 0.8387 0.6572 0.9488 0.7308 0.8573 0.7442 0.6295 0.7925] 2020-12-25 11:12:35 [INFO] [EVAL] Class Acc: [0.9908 0.9228 0.9583 0.7694 0.7828 0.7923 0.863 0.923 0.9526 0.8151 0.9687 0.91 0.7896 0.9711 0.7865 0.9759 0.7777 0.7584 0.8975] 2020-12-25 11:12:40 [INFO] [EVAL] The model with the best validation mIoU (0.7958) was saved at iter 52000. 2020-12-25 11:14:42 [INFO] [TRAIN] epoch=151, iter=56100/80000, loss=15.5164, lr=0.003371, batch_cost=1.2175, reader_cost=0.0006 | ETA 08:04:58 2020-12-25 11:16:58 [INFO] [TRAIN] epoch=152, iter=56200/80000, loss=16.1729, lr=0.003359, batch_cost=1.3594, reader_cost=0.0485 | ETA 08:59:12 2020-12-25 11:19:03 [INFO] [TRAIN] epoch=152, iter=56300/80000, loss=15.7805, lr=0.003346, batch_cost=1.2478, reader_cost=0.0002 | ETA 08:12:52 2020-12-25 11:21:07 [INFO] [TRAIN] epoch=152, iter=56400/80000, loss=15.5703, lr=0.003333, batch_cost=1.2330, reader_cost=0.0004 | ETA 08:04:59 2020-12-25 11:23:12 [INFO] [TRAIN] epoch=152, iter=56500/80000, loss=15.5013, lr=0.003320, batch_cost=1.2498, reader_cost=0.0005 | ETA 08:09:29 2020-12-25 11:25:34 [INFO] [TRAIN] epoch=153, iter=56600/80000, loss=16.2556, lr=0.003308, batch_cost=1.4180, reader_cost=0.0508 | ETA 09:13:02 2020-12-25 11:27:39 [INFO] [TRAIN] epoch=153, iter=56700/80000, loss=15.4573, lr=0.003295, batch_cost=1.2432, reader_cost=0.0004 | ETA 08:02:47 2020-12-25 11:29:42 [INFO] [TRAIN] epoch=153, iter=56800/80000, loss=15.5310, lr=0.003282, batch_cost=1.2358, reader_cost=0.0012 | ETA 07:57:49 2020-12-25 11:31:45 [INFO] [TRAIN] epoch=153, iter=56900/80000, loss=15.5617, lr=0.003270, batch_cost=1.2193, reader_cost=0.0012 | ETA 07:49:25 2020-12-25 11:34:02 [INFO] [TRAIN] epoch=154, iter=57000/80000, loss=15.7498, lr=0.003257, batch_cost=1.3695, reader_cost=0.0512 | ETA 08:44:57 2020-12-25 11:36:06 [INFO] [TRAIN] epoch=154, iter=57100/80000, loss=15.3957, lr=0.003244, batch_cost=1.2436, reader_cost=0.0008 | ETA 07:54:38 2020-12-25 11:38:10 [INFO] [TRAIN] epoch=154, iter=57200/80000, loss=15.6222, lr=0.003231, batch_cost=1.2321, reader_cost=0.0006 | ETA 07:48:11 2020-12-25 11:40:27 [INFO] [TRAIN] epoch=155, iter=57300/80000, loss=15.7418, lr=0.003219, batch_cost=1.3663, reader_cost=0.0458 | ETA 08:36:54 2020-12-25 11:42:29 [INFO] [TRAIN] epoch=155, iter=57400/80000, loss=15.4776, lr=0.003206, batch_cost=1.2267, reader_cost=0.0004 | ETA 07:42:03 2020-12-25 11:44:34 [INFO] [TRAIN] epoch=155, iter=57500/80000, loss=15.6582, lr=0.003193, batch_cost=1.2409, reader_cost=0.0008 | ETA 07:45:20 2020-12-25 11:46:37 [INFO] [TRAIN] epoch=155, iter=57600/80000, loss=15.6221, lr=0.003180, batch_cost=1.2311, reader_cost=0.0003 | ETA 07:39:37 2020-12-25 11:48:58 [INFO] [TRAIN] epoch=156, iter=57700/80000, loss=16.1097, lr=0.003167, batch_cost=1.4077, reader_cost=0.0448 | ETA 08:43:11 2020-12-25 11:51:01 [INFO] [TRAIN] epoch=156, iter=57800/80000, loss=15.3616, lr=0.003155, batch_cost=1.2237, reader_cost=0.0007 | ETA 07:32:46 2020-12-25 11:53:05 [INFO] [TRAIN] epoch=156, iter=57900/80000, loss=15.4660, lr=0.003142, batch_cost=1.2361, reader_cost=0.0007 | ETA 07:35:17 2020-12-25 11:55:07 [INFO] [TRAIN] epoch=156, iter=58000/80000, loss=15.5999, lr=0.003129, batch_cost=1.2177, reader_cost=0.0006 | ETA 07:26:29 2020-12-25 11:55:07 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 11:55:52 [INFO] [EVAL] #Images=500 mIoU=0.7959 Acc=0.9633 Kappa=0.9524 2020-12-25 11:55:52 [INFO] [EVAL] Class IoU: [0.9817 0.8568 0.9299 0.4792 0.6444 0.6919 0.7456 0.8179 0.927 0.636 0.9528 0.8502 0.6721 0.9553 0.7989 0.9019 0.7893 0.6931 0.7989] 2020-12-25 11:55:52 [INFO] [EVAL] Class Acc: [0.9941 0.9094 0.9606 0.8683 0.8271 0.8646 0.8662 0.8851 0.9512 0.844 0.9689 0.9021 0.7772 0.9714 0.916 0.9732 0.8363 0.8557 0.8797] 2020-12-25 11:56:00 [INFO] [EVAL] The model with the best validation mIoU (0.7959) was saved at iter 58000. 2020-12-25 11:58:16 [INFO] [TRAIN] epoch=157, iter=58100/80000, loss=15.9317, lr=0.003116, batch_cost=1.3547, reader_cost=0.0411 | ETA 08:14:28 2020-12-25 12:00:18 [INFO] [TRAIN] epoch=157, iter=58200/80000, loss=15.5838, lr=0.003103, batch_cost=1.2189, reader_cost=0.0004 | ETA 07:22:51 2020-12-25 12:02:21 [INFO] [TRAIN] epoch=157, iter=58300/80000, loss=15.3149, lr=0.003091, batch_cost=1.2239, reader_cost=0.0004 | ETA 07:22:38 2020-12-25 12:04:22 [INFO] [TRAIN] epoch=157, iter=58400/80000, loss=15.7915, lr=0.003078, batch_cost=1.2137, reader_cost=0.0002 | ETA 07:16:55 2020-12-25 12:06:39 [INFO] [TRAIN] epoch=158, iter=58500/80000, loss=15.9380, lr=0.003065, batch_cost=1.3678, reader_cost=0.0464 | ETA 08:10:08 2020-12-25 12:08:42 [INFO] [TRAIN] epoch=158, iter=58600/80000, loss=15.4139, lr=0.003052, batch_cost=1.2223, reader_cost=0.0003 | ETA 07:15:57 2020-12-25 12:10:45 [INFO] [TRAIN] epoch=158, iter=58700/80000, loss=15.6016, lr=0.003039, batch_cost=1.2340, reader_cost=0.0005 | ETA 07:18:04 2020-12-25 12:13:08 [INFO] [TRAIN] epoch=159, iter=58800/80000, loss=15.8744, lr=0.003026, batch_cost=1.4250, reader_cost=0.0507 | ETA 08:23:30 2020-12-25 12:15:13 [INFO] [TRAIN] epoch=159, iter=58900/80000, loss=15.4387, lr=0.003014, batch_cost=1.2498, reader_cost=0.0016 | ETA 07:19:31 2020-12-25 12:17:17 [INFO] [TRAIN] epoch=159, iter=59000/80000, loss=15.5706, lr=0.003001, batch_cost=1.2378, reader_cost=0.0007 | ETA 07:13:13 2020-12-25 12:19:21 [INFO] [TRAIN] epoch=159, iter=59100/80000, loss=15.4992, lr=0.002988, batch_cost=1.2385, reader_cost=0.0004 | ETA 07:11:24 2020-12-25 12:21:40 [INFO] [TRAIN] epoch=160, iter=59200/80000, loss=16.1266, lr=0.002975, batch_cost=1.3833, reader_cost=0.0407 | ETA 07:59:32 2020-12-25 12:23:43 [INFO] [TRAIN] epoch=160, iter=59300/80000, loss=15.4135, lr=0.002962, batch_cost=1.2278, reader_cost=0.0011 | ETA 07:03:35 2020-12-25 12:25:47 [INFO] [TRAIN] epoch=160, iter=59400/80000, loss=15.4825, lr=0.002949, batch_cost=1.2345, reader_cost=0.0005 | ETA 07:03:50 2020-12-25 12:27:50 [INFO] [TRAIN] epoch=160, iter=59500/80000, loss=15.7738, lr=0.002936, batch_cost=1.2292, reader_cost=0.0006 | ETA 06:59:59 2020-12-25 12:30:08 [INFO] [TRAIN] epoch=161, iter=59600/80000, loss=15.9118, lr=0.002924, batch_cost=1.3804, reader_cost=0.0401 | ETA 07:49:20 2020-12-25 12:32:11 [INFO] [TRAIN] epoch=161, iter=59700/80000, loss=15.4401, lr=0.002911, batch_cost=1.2285, reader_cost=0.0003 | ETA 06:55:37 2020-12-25 12:34:14 [INFO] [TRAIN] epoch=161, iter=59800/80000, loss=15.4670, lr=0.002898, batch_cost=1.2211, reader_cost=0.0008 | ETA 06:51:06 2020-12-25 12:36:33 [INFO] [TRAIN] epoch=162, iter=59900/80000, loss=15.7990, lr=0.002885, batch_cost=1.3915, reader_cost=0.0508 | ETA 07:46:09 2020-12-25 12:38:36 [INFO] [TRAIN] epoch=162, iter=60000/80000, loss=15.5010, lr=0.002872, batch_cost=1.2281, reader_cost=0.0006 | ETA 06:49:21 2020-12-25 12:38:36 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 12:39:22 [INFO] [EVAL] #Images=500 mIoU=0.8011 Acc=0.9648 Kappa=0.9544 2020-12-25 12:39:22 [INFO] [EVAL] Class IoU: [0.9834 0.867 0.9322 0.533 0.6187 0.7035 0.7554 0.8306 0.9284 0.6644 0.9519 0.8525 0.668 0.9582 0.8472 0.9004 0.7624 0.663 0.8009] 2020-12-25 12:39:22 [INFO] [EVAL] Class Acc: [0.993 0.9181 0.9631 0.8312 0.8485 0.8284 0.8713 0.9228 0.9567 0.8403 0.9654 0.914 0.7861 0.9738 0.9464 0.9678 0.8189 0.844 0.8864] 2020-12-25 12:39:30 [INFO] [EVAL] The model with the best validation mIoU (0.8011) was saved at iter 60000. 2020-12-25 12:41:32 [INFO] [TRAIN] epoch=162, iter=60100/80000, loss=15.6050, lr=0.002859, batch_cost=1.2259, reader_cost=0.0005 | ETA 06:46:36 2020-12-25 12:43:38 [INFO] [TRAIN] epoch=162, iter=60200/80000, loss=15.2841, lr=0.002846, batch_cost=1.2557, reader_cost=0.0010 | ETA 06:54:22 2020-12-25 12:45:57 [INFO] [TRAIN] epoch=163, iter=60300/80000, loss=16.2219, lr=0.002833, batch_cost=1.3815, reader_cost=0.0438 | ETA 07:33:35 2020-12-25 12:47:59 [INFO] [TRAIN] epoch=163, iter=60400/80000, loss=15.4861, lr=0.002820, batch_cost=1.2229, reader_cost=0.0003 | ETA 06:39:29 2020-12-25 12:50:04 [INFO] [TRAIN] epoch=163, iter=60500/80000, loss=15.2913, lr=0.002807, batch_cost=1.2460, reader_cost=0.0004 | ETA 06:44:56 2020-12-25 12:52:07 [INFO] [TRAIN] epoch=163, iter=60600/80000, loss=15.6381, lr=0.002794, batch_cost=1.2226, reader_cost=0.0004 | ETA 06:35:18 2020-12-25 12:54:25 [INFO] [TRAIN] epoch=164, iter=60700/80000, loss=15.9297, lr=0.002781, batch_cost=1.3804, reader_cost=0.0394 | ETA 07:24:01 2020-12-25 12:56:29 [INFO] [TRAIN] epoch=164, iter=60800/80000, loss=15.6274, lr=0.002768, batch_cost=1.2411, reader_cost=0.0008 | ETA 06:37:08 2020-12-25 12:58:32 [INFO] [TRAIN] epoch=164, iter=60900/80000, loss=15.3358, lr=0.002755, batch_cost=1.2244, reader_cost=0.0007 | ETA 06:29:45 2020-12-25 13:00:37 [INFO] [TRAIN] epoch=164, iter=61000/80000, loss=15.8691, lr=0.002742, batch_cost=1.2447, reader_cost=0.0005 | ETA 06:34:08 2020-12-25 13:02:54 [INFO] [TRAIN] epoch=165, iter=61100/80000, loss=15.7305, lr=0.002729, batch_cost=1.3737, reader_cost=0.0472 | ETA 07:12:43 2020-12-25 13:04:57 [INFO] [TRAIN] epoch=165, iter=61200/80000, loss=15.3049, lr=0.002716, batch_cost=1.2273, reader_cost=0.0007 | ETA 06:24:32 2020-12-25 13:07:00 [INFO] [TRAIN] epoch=165, iter=61300/80000, loss=15.5009, lr=0.002703, batch_cost=1.2315, reader_cost=0.0010 | ETA 06:23:48 2020-12-25 13:09:22 [INFO] [TRAIN] epoch=166, iter=61400/80000, loss=15.9085, lr=0.002690, batch_cost=1.4132, reader_cost=0.0427 | ETA 07:18:05 2020-12-25 13:11:25 [INFO] [TRAIN] epoch=166, iter=61500/80000, loss=15.4440, lr=0.002677, batch_cost=1.2289, reader_cost=0.0008 | ETA 06:18:54 2020-12-25 13:13:29 [INFO] [TRAIN] epoch=166, iter=61600/80000, loss=15.6084, lr=0.002664, batch_cost=1.2377, reader_cost=0.0005 | ETA 06:19:33 2020-12-25 13:15:33 [INFO] [TRAIN] epoch=166, iter=61700/80000, loss=15.5008, lr=0.002651, batch_cost=1.2313, reader_cost=0.0011 | ETA 06:15:33 2020-12-25 13:17:53 [INFO] [TRAIN] epoch=167, iter=61800/80000, loss=15.8466, lr=0.002638, batch_cost=1.3964, reader_cost=0.0564 | ETA 07:03:33 2020-12-25 13:19:55 [INFO] [TRAIN] epoch=167, iter=61900/80000, loss=15.2337, lr=0.002625, batch_cost=1.2246, reader_cost=0.0002 | ETA 06:09:25 2020-12-25 13:21:57 [INFO] [TRAIN] epoch=167, iter=62000/80000, loss=15.2784, lr=0.002612, batch_cost=1.2123, reader_cost=0.0002 | ETA 06:03:42 2020-12-25 13:21:57 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 13:22:42 [INFO] [EVAL] #Images=500 mIoU=0.8005 Acc=0.9645 Kappa=0.9539 2020-12-25 13:22:42 [INFO] [EVAL] Class IoU: [0.9846 0.8658 0.9327 0.5504 0.629 0.6964 0.7491 0.8262 0.9255 0.6235 0.951 0.8425 0.6219 0.9575 0.8293 0.9018 0.8259 0.6902 0.8061] 2020-12-25 13:22:42 [INFO] [EVAL] Class Acc: [0.9921 0.9184 0.9576 0.7853 0.8581 0.8446 0.8584 0.919 0.9627 0.8199 0.967 0.8998 0.856 0.9759 0.9377 0.9747 0.8973 0.8209 0.8901] 2020-12-25 13:22:47 [INFO] [EVAL] The model with the best validation mIoU (0.8011) was saved at iter 60000. 2020-12-25 13:24:52 [INFO] [TRAIN] epoch=167, iter=62100/80000, loss=15.8834, lr=0.002599, batch_cost=1.2409, reader_cost=0.0005 | ETA 06:10:12 2020-12-25 13:27:16 [INFO] [TRAIN] epoch=168, iter=62200/80000, loss=15.9768, lr=0.002586, batch_cost=1.4415, reader_cost=0.0535 | ETA 07:07:38 2020-12-25 13:29:21 [INFO] [TRAIN] epoch=168, iter=62300/80000, loss=15.2660, lr=0.002573, batch_cost=1.2466, reader_cost=0.0005 | ETA 06:07:44 2020-12-25 13:31:23 [INFO] [TRAIN] epoch=168, iter=62400/80000, loss=15.2299, lr=0.002560, batch_cost=1.2127, reader_cost=0.0003 | ETA 05:55:42 2020-12-25 13:33:44 [INFO] [TRAIN] epoch=169, iter=62500/80000, loss=15.8828, lr=0.002547, batch_cost=1.4104, reader_cost=0.0428 | ETA 06:51:22 2020-12-25 13:35:46 [INFO] [TRAIN] epoch=169, iter=62600/80000, loss=15.8066, lr=0.002534, batch_cost=1.2174, reader_cost=0.0003 | ETA 05:53:02 2020-12-25 13:37:49 [INFO] [TRAIN] epoch=169, iter=62700/80000, loss=15.4426, lr=0.002520, batch_cost=1.2299, reader_cost=0.0005 | ETA 05:54:36 2020-12-25 13:39:52 [INFO] [TRAIN] epoch=169, iter=62800/80000, loss=15.5367, lr=0.002507, batch_cost=1.2298, reader_cost=0.0004 | ETA 05:52:32 2020-12-25 13:42:15 [INFO] [TRAIN] epoch=170, iter=62900/80000, loss=15.8922, lr=0.002494, batch_cost=1.4186, reader_cost=0.0412 | ETA 06:44:18 2020-12-25 13:44:20 [INFO] [TRAIN] epoch=170, iter=63000/80000, loss=15.5272, lr=0.002481, batch_cost=1.2512, reader_cost=0.0003 | ETA 05:54:30 2020-12-25 13:46:23 [INFO] [TRAIN] epoch=170, iter=63100/80000, loss=15.3510, lr=0.002468, batch_cost=1.2339, reader_cost=0.0005 | ETA 05:47:32 2020-12-25 13:48:26 [INFO] [TRAIN] epoch=170, iter=63200/80000, loss=15.6641, lr=0.002455, batch_cost=1.2264, reader_cost=0.0004 | ETA 05:43:23 2020-12-25 13:50:46 [INFO] [TRAIN] epoch=171, iter=63300/80000, loss=16.1656, lr=0.002442, batch_cost=1.3921, reader_cost=0.0500 | ETA 06:27:28 2020-12-25 13:52:49 [INFO] [TRAIN] epoch=171, iter=63400/80000, loss=15.2128, lr=0.002429, batch_cost=1.2313, reader_cost=0.0004 | ETA 05:40:39 2020-12-25 13:54:54 [INFO] [TRAIN] epoch=171, iter=63500/80000, loss=15.2105, lr=0.002415, batch_cost=1.2483, reader_cost=0.0008 | ETA 05:43:16 2020-12-25 13:57:00 [INFO] [TRAIN] epoch=171, iter=63600/80000, loss=15.8092, lr=0.002402, batch_cost=1.2579, reader_cost=0.0003 | ETA 05:43:49 2020-12-25 13:59:22 [INFO] [TRAIN] epoch=172, iter=63700/80000, loss=15.7223, lr=0.002389, batch_cost=1.4127, reader_cost=0.0460 | ETA 06:23:47 2020-12-25 14:01:24 [INFO] [TRAIN] epoch=172, iter=63800/80000, loss=15.2854, lr=0.002376, batch_cost=1.2263, reader_cost=0.0008 | ETA 05:31:06 2020-12-25 14:03:27 [INFO] [TRAIN] epoch=172, iter=63900/80000, loss=15.3936, lr=0.002363, batch_cost=1.2244, reader_cost=0.0003 | ETA 05:28:33 2020-12-25 14:05:46 [INFO] [TRAIN] epoch=173, iter=64000/80000, loss=15.9206, lr=0.002349, batch_cost=1.3799, reader_cost=0.0490 | ETA 06:07:58 2020-12-25 14:05:46 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 14:06:31 [INFO] [EVAL] #Images=500 mIoU=0.8032 Acc=0.9644 Kappa=0.9539 2020-12-25 14:06:31 [INFO] [EVAL] Class IoU: [0.9823 0.8602 0.9327 0.5561 0.6404 0.7083 0.7456 0.8147 0.9284 0.6475 0.9489 0.8499 0.6603 0.9582 0.8272 0.9014 0.8463 0.6519 0.8011] 2020-12-25 14:06:31 [INFO] [EVAL] Class Acc: [0.9927 0.9141 0.9643 0.8574 0.8551 0.8276 0.8609 0.943 0.956 0.798 0.9634 0.9063 0.8141 0.977 0.9202 0.9633 0.9241 0.7619 0.877 ] 2020-12-25 14:06:39 [INFO] [EVAL] The model with the best validation mIoU (0.8032) was saved at iter 64000. 2020-12-25 14:08:43 [INFO] [TRAIN] epoch=173, iter=64100/80000, loss=15.5071, lr=0.002336, batch_cost=1.2365, reader_cost=0.0006 | ETA 05:27:40 2020-12-25 14:10:48 [INFO] [TRAIN] epoch=173, iter=64200/80000, loss=15.5499, lr=0.002323, batch_cost=1.2464, reader_cost=0.0005 | ETA 05:28:12 2020-12-25 14:12:50 [INFO] [TRAIN] epoch=173, iter=64300/80000, loss=15.3123, lr=0.002310, batch_cost=1.2221, reader_cost=0.0002 | ETA 05:19:46 2020-12-25 14:15:10 [INFO] [TRAIN] epoch=174, iter=64400/80000, loss=15.8463, lr=0.002296, batch_cost=1.3923, reader_cost=0.0458 | ETA 06:01:59 2020-12-25 14:17:16 [INFO] [TRAIN] epoch=174, iter=64500/80000, loss=15.4250, lr=0.002283, batch_cost=1.2639, reader_cost=0.0008 | ETA 05:26:30 2020-12-25 14:19:18 [INFO] [TRAIN] epoch=174, iter=64600/80000, loss=15.2901, lr=0.002270, batch_cost=1.2147, reader_cost=0.0008 | ETA 05:11:46 2020-12-25 14:21:22 [INFO] [TRAIN] epoch=174, iter=64700/80000, loss=15.7314, lr=0.002257, batch_cost=1.2424, reader_cost=0.0006 | ETA 05:16:48 2020-12-25 14:23:42 [INFO] [TRAIN] epoch=175, iter=64800/80000, loss=15.8113, lr=0.002243, batch_cost=1.3991, reader_cost=0.0473 | ETA 05:54:25 2020-12-25 14:25:46 [INFO] [TRAIN] epoch=175, iter=64900/80000, loss=15.4007, lr=0.002230, batch_cost=1.2361, reader_cost=0.0005 | ETA 05:11:04 2020-12-25 14:27:50 [INFO] [TRAIN] epoch=175, iter=65000/80000, loss=15.2324, lr=0.002217, batch_cost=1.2336, reader_cost=0.0005 | ETA 05:08:24 2020-12-25 14:29:53 [INFO] [TRAIN] epoch=175, iter=65100/80000, loss=15.7587, lr=0.002203, batch_cost=1.2326, reader_cost=0.0004 | ETA 05:06:05 2020-12-25 14:32:17 [INFO] [TRAIN] epoch=176, iter=65200/80000, loss=15.6488, lr=0.002190, batch_cost=1.4299, reader_cost=0.0557 | ETA 05:52:42 2020-12-25 14:34:23 [INFO] [TRAIN] epoch=176, iter=65300/80000, loss=15.4294, lr=0.002177, batch_cost=1.2574, reader_cost=0.0007 | ETA 05:08:04 2020-12-25 14:36:26 [INFO] [TRAIN] epoch=176, iter=65400/80000, loss=15.3406, lr=0.002164, batch_cost=1.2287, reader_cost=0.0007 | ETA 04:58:59 2020-12-25 14:38:45 [INFO] [TRAIN] epoch=177, iter=65500/80000, loss=15.7809, lr=0.002150, batch_cost=1.3877, reader_cost=0.0495 | ETA 05:35:21 2020-12-25 14:40:47 [INFO] [TRAIN] epoch=177, iter=65600/80000, loss=15.3091, lr=0.002137, batch_cost=1.2228, reader_cost=0.0019 | ETA 04:53:29 2020-12-25 14:42:53 [INFO] [TRAIN] epoch=177, iter=65700/80000, loss=15.2498, lr=0.002123, batch_cost=1.2541, reader_cost=0.0004 | ETA 04:58:53 2020-12-25 14:44:56 [INFO] [TRAIN] epoch=177, iter=65800/80000, loss=15.6459, lr=0.002110, batch_cost=1.2317, reader_cost=0.0002 | ETA 04:51:30 2020-12-25 14:47:17 [INFO] [TRAIN] epoch=178, iter=65900/80000, loss=15.7425, lr=0.002097, batch_cost=1.4021, reader_cost=0.0490 | ETA 05:29:29 2020-12-25 14:49:20 [INFO] [TRAIN] epoch=178, iter=66000/80000, loss=15.5349, lr=0.002083, batch_cost=1.2316, reader_cost=0.0002 | ETA 04:47:21 2020-12-25 14:49:21 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 14:50:06 [INFO] [EVAL] #Images=500 mIoU=0.8059 Acc=0.9654 Kappa=0.9551 2020-12-25 14:50:06 [INFO] [EVAL] Class IoU: [0.9836 0.869 0.9341 0.5662 0.6553 0.7066 0.7512 0.8277 0.9292 0.6503 0.9522 0.8501 0.6593 0.9591 0.8091 0.8788 0.8213 0.7048 0.8037] 2020-12-25 14:50:06 [INFO] [EVAL] Class Acc: [0.9909 0.9384 0.9672 0.7623 0.8141 0.8452 0.852 0.9027 0.9546 0.8971 0.9675 0.9102 0.8349 0.9779 0.9174 0.9276 0.8938 0.8387 0.8613] 2020-12-25 14:50:14 [INFO] [EVAL] The model with the best validation mIoU (0.8059) was saved at iter 66000. 2020-12-25 14:52:14 [INFO] [TRAIN] epoch=178, iter=66100/80000, loss=15.3822, lr=0.002070, batch_cost=1.1990, reader_cost=0.0005 | ETA 04:37:46 2020-12-25 14:54:19 [INFO] [TRAIN] epoch=178, iter=66200/80000, loss=15.5977, lr=0.002057, batch_cost=1.2473, reader_cost=0.0004 | ETA 04:46:52 2020-12-25 14:56:41 [INFO] [TRAIN] epoch=179, iter=66300/80000, loss=15.8686, lr=0.002043, batch_cost=1.4186, reader_cost=0.0567 | ETA 05:23:54 2020-12-25 14:58:44 [INFO] [TRAIN] epoch=179, iter=66400/80000, loss=15.0892, lr=0.002030, batch_cost=1.2244, reader_cost=0.0004 | ETA 04:37:31 2020-12-25 15:00:48 [INFO] [TRAIN] epoch=179, iter=66500/80000, loss=15.3114, lr=0.002016, batch_cost=1.2421, reader_cost=0.0004 | ETA 04:39:28 2020-12-25 15:03:07 [INFO] [TRAIN] epoch=180, iter=66600/80000, loss=15.7295, lr=0.002003, batch_cost=1.3878, reader_cost=0.0494 | ETA 05:09:55 2020-12-25 15:05:12 [INFO] [TRAIN] epoch=180, iter=66700/80000, loss=15.6257, lr=0.001989, batch_cost=1.2427, reader_cost=0.0002 | ETA 04:35:27 2020-12-25 15:07:15 [INFO] [TRAIN] epoch=180, iter=66800/80000, loss=15.5269, lr=0.001976, batch_cost=1.2245, reader_cost=0.0012 | ETA 04:29:23 2020-12-25 15:09:18 [INFO] [TRAIN] epoch=180, iter=66900/80000, loss=15.4784, lr=0.001962, batch_cost=1.2285, reader_cost=0.0010 | ETA 04:28:13 2020-12-25 15:11:42 [INFO] [TRAIN] epoch=181, iter=67000/80000, loss=15.6769, lr=0.001949, batch_cost=1.4384, reader_cost=0.0556 | ETA 05:11:39 2020-12-25 15:13:45 [INFO] [TRAIN] epoch=181, iter=67100/80000, loss=15.4915, lr=0.001935, batch_cost=1.2247, reader_cost=0.0003 | ETA 04:23:18 2020-12-25 15:15:49 [INFO] [TRAIN] epoch=181, iter=67200/80000, loss=15.3631, lr=0.001922, batch_cost=1.2430, reader_cost=0.0004 | ETA 04:25:10 2020-12-25 15:17:53 [INFO] [TRAIN] epoch=181, iter=67300/80000, loss=15.6110, lr=0.001908, batch_cost=1.2356, reader_cost=0.0005 | ETA 04:21:31 2020-12-25 15:20:14 [INFO] [TRAIN] epoch=182, iter=67400/80000, loss=15.8905, lr=0.001895, batch_cost=1.4130, reader_cost=0.0417 | ETA 04:56:43 2020-12-25 15:22:17 [INFO] [TRAIN] epoch=182, iter=67500/80000, loss=15.5105, lr=0.001881, batch_cost=1.2215, reader_cost=0.0008 | ETA 04:14:28 2020-12-25 15:24:21 [INFO] [TRAIN] epoch=182, iter=67600/80000, loss=15.3225, lr=0.001868, batch_cost=1.2333, reader_cost=0.0004 | ETA 04:14:52 2020-12-25 15:26:22 [INFO] [TRAIN] epoch=182, iter=67700/80000, loss=15.5661, lr=0.001854, batch_cost=1.2150, reader_cost=0.0007 | ETA 04:09:05 2020-12-25 15:28:41 [INFO] [TRAIN] epoch=183, iter=67800/80000, loss=15.5719, lr=0.001841, batch_cost=1.3855, reader_cost=0.0474 | ETA 04:41:42 2020-12-25 15:30:44 [INFO] [TRAIN] epoch=183, iter=67900/80000, loss=15.4314, lr=0.001827, batch_cost=1.2264, reader_cost=0.0005 | ETA 04:07:19 2020-12-25 15:32:46 [INFO] [TRAIN] epoch=183, iter=68000/80000, loss=15.5153, lr=0.001813, batch_cost=1.2183, reader_cost=0.0012 | ETA 04:03:39 2020-12-25 15:32:46 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 15:33:33 [INFO] [EVAL] #Images=500 mIoU=0.8075 Acc=0.9649 Kappa=0.9545 2020-12-25 15:33:33 [INFO] [EVAL] Class IoU: [0.9821 0.8626 0.9327 0.5439 0.6485 0.7087 0.7568 0.8327 0.9295 0.6341 0.9556 0.8518 0.6649 0.9592 0.8298 0.9084 0.8335 0.7009 0.8064] 2020-12-25 15:33:33 [INFO] [EVAL] Class Acc: [0.9921 0.9161 0.9637 0.8265 0.8485 0.8382 0.857 0.9138 0.9547 0.8719 0.9737 0.9134 0.804 0.9771 0.9045 0.9612 0.8887 0.8348 0.8867] 2020-12-25 15:33:40 [INFO] [EVAL] The model with the best validation mIoU (0.8075) was saved at iter 68000. 2020-12-25 15:36:00 [INFO] [TRAIN] epoch=184, iter=68100/80000, loss=15.7836, lr=0.001800, batch_cost=1.3985, reader_cost=0.0434 | ETA 04:37:22 2020-12-25 15:38:04 [INFO] [TRAIN] epoch=184, iter=68200/80000, loss=15.5326, lr=0.001786, batch_cost=1.2355, reader_cost=0.0003 | ETA 04:02:58 2020-12-25 15:40:10 [INFO] [TRAIN] epoch=184, iter=68300/80000, loss=15.3380, lr=0.001773, batch_cost=1.2510, reader_cost=0.0011 | ETA 04:03:56 2020-12-25 15:42:14 [INFO] [TRAIN] epoch=184, iter=68400/80000, loss=15.5000, lr=0.001759, batch_cost=1.2397, reader_cost=0.0002 | ETA 03:59:40 2020-12-25 15:44:33 [INFO] [TRAIN] epoch=185, iter=68500/80000, loss=15.9549, lr=0.001745, batch_cost=1.3876, reader_cost=0.0569 | ETA 04:25:57 2020-12-25 15:46:34 [INFO] [TRAIN] epoch=185, iter=68600/80000, loss=15.4297, lr=0.001732, batch_cost=1.2056, reader_cost=0.0004 | ETA 03:49:04 2020-12-25 15:48:38 [INFO] [TRAIN] epoch=185, iter=68700/80000, loss=15.1900, lr=0.001718, batch_cost=1.2379, reader_cost=0.0010 | ETA 03:53:08 2020-12-25 15:50:42 [INFO] [TRAIN] epoch=185, iter=68800/80000, loss=15.8304, lr=0.001704, batch_cost=1.2377, reader_cost=0.0002 | ETA 03:51:02 2020-12-25 15:53:02 [INFO] [TRAIN] epoch=186, iter=68900/80000, loss=15.9204, lr=0.001691, batch_cost=1.4005, reader_cost=0.0620 | ETA 04:19:05 2020-12-25 15:55:08 [INFO] [TRAIN] epoch=186, iter=69000/80000, loss=15.1969, lr=0.001677, batch_cost=1.2533, reader_cost=0.0002 | ETA 03:49:46 2020-12-25 15:57:13 [INFO] [TRAIN] epoch=186, iter=69100/80000, loss=15.5588, lr=0.001663, batch_cost=1.2515, reader_cost=0.0002 | ETA 03:47:21 2020-12-25 15:59:35 [INFO] [TRAIN] epoch=187, iter=69200/80000, loss=15.7551, lr=0.001649, batch_cost=1.4144, reader_cost=0.0512 | ETA 04:14:35 2020-12-25 16:01:38 [INFO] [TRAIN] epoch=187, iter=69300/80000, loss=15.6215, lr=0.001636, batch_cost=1.2301, reader_cost=0.0003 | ETA 03:39:22 2020-12-25 16:03:41 [INFO] [TRAIN] epoch=187, iter=69400/80000, loss=15.4993, lr=0.001622, batch_cost=1.2295, reader_cost=0.0004 | ETA 03:37:12 2020-12-25 16:05:45 [INFO] [TRAIN] epoch=187, iter=69500/80000, loss=15.3481, lr=0.001608, batch_cost=1.2337, reader_cost=0.0003 | ETA 03:35:54 2020-12-25 16:08:02 [INFO] [TRAIN] epoch=188, iter=69600/80000, loss=15.9192, lr=0.001594, batch_cost=1.3720, reader_cost=0.0424 | ETA 03:57:48 2020-12-25 16:10:03 [INFO] [TRAIN] epoch=188, iter=69700/80000, loss=15.5925, lr=0.001581, batch_cost=1.2082, reader_cost=0.0006 | ETA 03:27:23 2020-12-25 16:12:05 [INFO] [TRAIN] epoch=188, iter=69800/80000, loss=15.2678, lr=0.001567, batch_cost=1.2197, reader_cost=0.0006 | ETA 03:27:20 2020-12-25 16:14:09 [INFO] [TRAIN] epoch=188, iter=69900/80000, loss=15.8015, lr=0.001553, batch_cost=1.2304, reader_cost=0.0005 | ETA 03:27:07 2020-12-25 16:16:29 [INFO] [TRAIN] epoch=189, iter=70000/80000, loss=15.9104, lr=0.001539, batch_cost=1.4049, reader_cost=0.0432 | ETA 03:54:09 2020-12-25 16:16:29 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 16:17:16 [INFO] [EVAL] #Images=500 mIoU=0.8076 Acc=0.9652 Kappa=0.9548 2020-12-25 16:17:16 [INFO] [EVAL] Class IoU: [0.9831 0.8638 0.934 0.5325 0.6527 0.7105 0.7583 0.8258 0.9299 0.6448 0.9543 0.8489 0.6594 0.9594 0.8124 0.9083 0.8632 0.6993 0.8033] 2020-12-25 16:17:16 [INFO] [EVAL] Class Acc: [0.9931 0.9151 0.9664 0.8218 0.8326 0.8232 0.8572 0.9116 0.956 0.8808 0.9711 0.9025 0.8206 0.9759 0.8979 0.9562 0.955 0.8386 0.866 ] 2020-12-25 16:17:23 [INFO] [EVAL] The model with the best validation mIoU (0.8076) was saved at iter 70000. 2020-12-25 16:19:26 [INFO] [TRAIN] epoch=189, iter=70100/80000, loss=15.4233, lr=0.001525, batch_cost=1.2235, reader_cost=0.0006 | ETA 03:21:52 2020-12-25 16:21:29 [INFO] [TRAIN] epoch=189, iter=70200/80000, loss=15.2615, lr=0.001511, batch_cost=1.2302, reader_cost=0.0002 | ETA 03:20:56 2020-12-25 16:23:31 [INFO] [TRAIN] epoch=189, iter=70300/80000, loss=15.8188, lr=0.001497, batch_cost=1.2172, reader_cost=0.0002 | ETA 03:16:46 2020-12-25 16:25:53 [INFO] [TRAIN] epoch=190, iter=70400/80000, loss=15.8648, lr=0.001484, batch_cost=1.4180, reader_cost=0.0514 | ETA 03:46:53 2020-12-25 16:27:56 [INFO] [TRAIN] epoch=190, iter=70500/80000, loss=15.3952, lr=0.001470, batch_cost=1.2277, reader_cost=0.0007 | ETA 03:14:23 2020-12-25 16:29:59 [INFO] [TRAIN] epoch=190, iter=70600/80000, loss=15.2932, lr=0.001456, batch_cost=1.2310, reader_cost=0.0004 | ETA 03:12:51 2020-12-25 16:32:19 [INFO] [TRAIN] epoch=191, iter=70700/80000, loss=15.8191, lr=0.001442, batch_cost=1.3914, reader_cost=0.0532 | ETA 03:35:40 2020-12-25 16:34:21 [INFO] [TRAIN] epoch=191, iter=70800/80000, loss=15.5460, lr=0.001428, batch_cost=1.2162, reader_cost=0.0041 | ETA 03:06:29 2020-12-25 16:36:25 [INFO] [TRAIN] epoch=191, iter=70900/80000, loss=15.3621, lr=0.001414, batch_cost=1.2392, reader_cost=0.0002 | ETA 03:07:56 2020-12-25 16:38:30 [INFO] [TRAIN] epoch=191, iter=71000/80000, loss=15.1792, lr=0.001400, batch_cost=1.2475, reader_cost=0.0008 | ETA 03:07:07 2020-12-25 16:40:51 [INFO] [TRAIN] epoch=192, iter=71100/80000, loss=15.9789, lr=0.001386, batch_cost=1.4133, reader_cost=0.0496 | ETA 03:29:38 2020-12-25 16:42:56 [INFO] [TRAIN] epoch=192, iter=71200/80000, loss=15.4129, lr=0.001372, batch_cost=1.2468, reader_cost=0.0002 | ETA 03:02:52 2020-12-25 16:45:05 [INFO] [TRAIN] epoch=192, iter=71300/80000, loss=15.3012, lr=0.001358, batch_cost=1.2785, reader_cost=0.0005 | ETA 03:05:23 2020-12-25 16:47:08 [INFO] [TRAIN] epoch=192, iter=71400/80000, loss=15.8601, lr=0.001344, batch_cost=1.2335, reader_cost=0.0005 | ETA 02:56:47 2020-12-25 16:49:29 [INFO] [TRAIN] epoch=193, iter=71500/80000, loss=15.8944, lr=0.001330, batch_cost=1.4111, reader_cost=0.0469 | ETA 03:19:54 2020-12-25 16:51:34 [INFO] [TRAIN] epoch=193, iter=71600/80000, loss=15.3170, lr=0.001316, batch_cost=1.2472, reader_cost=0.0006 | ETA 02:54:36 2020-12-25 16:53:37 [INFO] [TRAIN] epoch=193, iter=71700/80000, loss=15.3431, lr=0.001301, batch_cost=1.2229, reader_cost=0.0004 | ETA 02:49:09 2020-12-25 16:56:04 [INFO] [TRAIN] epoch=194, iter=71800/80000, loss=15.7180, lr=0.001287, batch_cost=1.4669, reader_cost=0.0469 | ETA 03:20:28 2020-12-25 16:58:08 [INFO] [TRAIN] epoch=194, iter=71900/80000, loss=15.6149, lr=0.001273, batch_cost=1.2445, reader_cost=0.0004 | ETA 02:48:00 2020-12-25 17:00:13 [INFO] [TRAIN] epoch=194, iter=72000/80000, loss=15.2711, lr=0.001259, batch_cost=1.2433, reader_cost=0.0003 | ETA 02:45:46 2020-12-25 17:00:13 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 17:01:02 [INFO] [EVAL] #Images=500 mIoU=0.8086 Acc=0.9661 Kappa=0.9561 2020-12-25 17:01:02 [INFO] [EVAL] Class IoU: [0.9845 0.8717 0.9347 0.5475 0.6559 0.712 0.7576 0.8326 0.9312 0.6503 0.9518 0.8511 0.6652 0.9603 0.8368 0.8901 0.8255 0.6994 0.8058] 2020-12-25 17:01:02 [INFO] [EVAL] Class Acc: [0.9917 0.9322 0.9642 0.8167 0.8486 0.8423 0.8608 0.9187 0.9578 0.8666 0.9679 0.9109 0.8057 0.9791 0.9231 0.9408 0.8934 0.8361 0.8784] 2020-12-25 17:01:10 [INFO] [EVAL] The model with the best validation mIoU (0.8086) was saved at iter 72000. 2020-12-25 17:03:16 [INFO] [TRAIN] epoch=194, iter=72100/80000, loss=15.5593, lr=0.001245, batch_cost=1.2612, reader_cost=0.0004 | ETA 02:46:03 2020-12-25 17:05:40 [INFO] [TRAIN] epoch=195, iter=72200/80000, loss=15.9362, lr=0.001231, batch_cost=1.4336, reader_cost=0.0486 | ETA 03:06:22 2020-12-25 17:07:43 [INFO] [TRAIN] epoch=195, iter=72300/80000, loss=15.5746, lr=0.001216, batch_cost=1.2352, reader_cost=0.0005 | ETA 02:38:30 2020-12-25 17:09:46 [INFO] [TRAIN] epoch=195, iter=72400/80000, loss=15.3735, lr=0.001202, batch_cost=1.2252, reader_cost=0.0003 | ETA 02:35:11 2020-12-25 17:11:50 [INFO] [TRAIN] epoch=195, iter=72500/80000, loss=15.7500, lr=0.001188, batch_cost=1.2361, reader_cost=0.0003 | ETA 02:34:30 2020-12-25 17:14:12 [INFO] [TRAIN] epoch=196, iter=72600/80000, loss=16.1181, lr=0.001174, batch_cost=1.4115, reader_cost=0.0517 | ETA 02:54:05 2020-12-25 17:16:14 [INFO] [TRAIN] epoch=196, iter=72700/80000, loss=15.3757, lr=0.001159, batch_cost=1.2267, reader_cost=0.0006 | ETA 02:29:14 2020-12-25 17:18:17 [INFO] [TRAIN] epoch=196, iter=72800/80000, loss=15.2618, lr=0.001145, batch_cost=1.2241, reader_cost=0.0006 | ETA 02:26:53 2020-12-25 17:20:23 [INFO] [TRAIN] epoch=196, iter=72900/80000, loss=15.9551, lr=0.001131, batch_cost=1.2568, reader_cost=0.0003 | ETA 02:28:43 2020-12-25 17:22:45 [INFO] [TRAIN] epoch=197, iter=73000/80000, loss=15.9188, lr=0.001117, batch_cost=1.4158, reader_cost=0.0486 | ETA 02:45:10 2020-12-25 17:24:50 [INFO] [TRAIN] epoch=197, iter=73100/80000, loss=15.4940, lr=0.001102, batch_cost=1.2469, reader_cost=0.0003 | ETA 02:23:23 2020-12-25 17:26:50 [INFO] [TRAIN] epoch=197, iter=73200/80000, loss=15.1818, lr=0.001088, batch_cost=1.2042, reader_cost=0.0002 | ETA 02:16:28 2020-12-25 17:29:09 [INFO] [TRAIN] epoch=198, iter=73300/80000, loss=15.7743, lr=0.001073, batch_cost=1.3854, reader_cost=0.0417 | ETA 02:34:42 2020-12-25 17:31:10 [INFO] [TRAIN] epoch=198, iter=73400/80000, loss=15.5916, lr=0.001059, batch_cost=1.2040, reader_cost=0.0005 | ETA 02:12:26 2020-12-25 17:33:10 [INFO] [TRAIN] epoch=198, iter=73500/80000, loss=15.5055, lr=0.001044, batch_cost=1.2003, reader_cost=0.0004 | ETA 02:10:02 2020-12-25 17:35:15 [INFO] [TRAIN] epoch=198, iter=73600/80000, loss=15.3963, lr=0.001030, batch_cost=1.2497, reader_cost=0.0006 | ETA 02:13:18 2020-12-25 17:37:35 [INFO] [TRAIN] epoch=199, iter=73700/80000, loss=15.7401, lr=0.001016, batch_cost=1.3947, reader_cost=0.0412 | ETA 02:26:26 2020-12-25 17:39:39 [INFO] [TRAIN] epoch=199, iter=73800/80000, loss=15.5105, lr=0.001001, batch_cost=1.2331, reader_cost=0.0007 | ETA 02:07:25 2020-12-25 17:41:43 [INFO] [TRAIN] epoch=199, iter=73900/80000, loss=15.4008, lr=0.000986, batch_cost=1.2397, reader_cost=0.0003 | ETA 02:06:02 2020-12-25 17:43:45 [INFO] [TRAIN] epoch=199, iter=74000/80000, loss=15.8633, lr=0.000972, batch_cost=1.2184, reader_cost=0.0003 | ETA 02:01:50 2020-12-25 17:43:45 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 17:44:31 [INFO] [EVAL] #Images=500 mIoU=0.8064 Acc=0.9655 Kappa=0.9552 2020-12-25 17:44:31 [INFO] [EVAL] Class IoU: [0.9834 0.8673 0.9339 0.5331 0.6464 0.7111 0.7583 0.8323 0.9304 0.657 0.954 0.8492 0.6536 0.9602 0.8124 0.8969 0.8336 0.7009 0.8071] 2020-12-25 17:44:31 [INFO] [EVAL] Class Acc: [0.9918 0.9242 0.9622 0.8011 0.8554 0.846 0.872 0.9139 0.9586 0.8524 0.9689 0.9118 0.8142 0.9795 0.9273 0.9443 0.9031 0.84 0.8795] 2020-12-25 17:44:36 [INFO] [EVAL] The model with the best validation mIoU (0.8086) was saved at iter 72000. 2020-12-25 17:47:01 [INFO] [TRAIN] epoch=200, iter=74100/80000, loss=15.7848, lr=0.000957, batch_cost=1.4481, reader_cost=0.0463 | ETA 02:22:23 2020-12-25 17:49:07 [INFO] [TRAIN] epoch=200, iter=74200/80000, loss=15.3737, lr=0.000943, batch_cost=1.2611, reader_cost=0.0004 | ETA 02:01:54 2020-12-25 17:51:10 [INFO] [TRAIN] epoch=200, iter=74300/80000, loss=15.3585, lr=0.000928, batch_cost=1.2256, reader_cost=0.0004 | ETA 01:56:26 2020-12-25 17:53:10 [INFO] [TRAIN] epoch=200, iter=74400/80000, loss=15.7476, lr=0.000913, batch_cost=1.1954, reader_cost=0.0003 | ETA 01:51:34 2020-12-25 17:55:31 [INFO] [TRAIN] epoch=201, iter=74500/80000, loss=15.7024, lr=0.000899, batch_cost=1.4039, reader_cost=0.0450 | ETA 02:08:41 2020-12-25 17:57:36 [INFO] [TRAIN] epoch=201, iter=74600/80000, loss=15.4584, lr=0.000884, batch_cost=1.2530, reader_cost=0.0007 | ETA 01:52:46 2020-12-25 17:59:41 [INFO] [TRAIN] epoch=201, iter=74700/80000, loss=15.4076, lr=0.000869, batch_cost=1.2419, reader_cost=0.0006 | ETA 01:49:42 2020-12-25 18:02:07 [INFO] [TRAIN] epoch=202, iter=74800/80000, loss=16.0107, lr=0.000854, batch_cost=1.4591, reader_cost=0.0426 | ETA 02:06:27 2020-12-25 18:04:10 [INFO] [TRAIN] epoch=202, iter=74900/80000, loss=15.3434, lr=0.000840, batch_cost=1.2206, reader_cost=0.0006 | ETA 01:43:45 2020-12-25 18:06:12 [INFO] [TRAIN] epoch=202, iter=75000/80000, loss=15.1799, lr=0.000825, batch_cost=1.2217, reader_cost=0.0007 | ETA 01:41:48 2020-12-25 18:08:16 [INFO] [TRAIN] epoch=202, iter=75100/80000, loss=15.5636, lr=0.000810, batch_cost=1.2437, reader_cost=0.0008 | ETA 01:41:34 2020-12-25 18:10:36 [INFO] [TRAIN] epoch=203, iter=75200/80000, loss=15.9241, lr=0.000795, batch_cost=1.3942, reader_cost=0.0440 | ETA 01:51:32 2020-12-25 18:12:43 [INFO] [TRAIN] epoch=203, iter=75300/80000, loss=15.2287, lr=0.000780, batch_cost=1.2661, reader_cost=0.0005 | ETA 01:39:10 2020-12-25 18:14:46 [INFO] [TRAIN] epoch=203, iter=75400/80000, loss=15.2619, lr=0.000765, batch_cost=1.2268, reader_cost=0.0004 | ETA 01:34:03 2020-12-25 18:16:51 [INFO] [TRAIN] epoch=203, iter=75500/80000, loss=15.7663, lr=0.000750, batch_cost=1.2449, reader_cost=0.0002 | ETA 01:33:22 2020-12-25 18:19:14 [INFO] [TRAIN] epoch=204, iter=75600/80000, loss=15.7505, lr=0.000735, batch_cost=1.4350, reader_cost=0.0489 | ETA 01:45:13 2020-12-25 18:21:17 [INFO] [TRAIN] epoch=204, iter=75700/80000, loss=15.3093, lr=0.000720, batch_cost=1.2256, reader_cost=0.0002 | ETA 01:27:49 2020-12-25 18:23:21 [INFO] [TRAIN] epoch=204, iter=75800/80000, loss=15.2794, lr=0.000705, batch_cost=1.2357, reader_cost=0.0004 | ETA 01:26:29 2020-12-25 18:25:40 [INFO] [TRAIN] epoch=205, iter=75900/80000, loss=15.7041, lr=0.000690, batch_cost=1.3819, reader_cost=0.0524 | ETA 01:34:25 2020-12-25 18:27:44 [INFO] [TRAIN] epoch=205, iter=76000/80000, loss=15.4429, lr=0.000675, batch_cost=1.2444, reader_cost=0.0005 | ETA 01:22:57 2020-12-25 18:27:44 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 18:28:30 [INFO] [EVAL] #Images=500 mIoU=0.8072 Acc=0.9653 Kappa=0.9550 2020-12-25 18:28:30 [INFO] [EVAL] Class IoU: [0.9826 0.8649 0.9343 0.5522 0.6473 0.7122 0.7603 0.833 0.93 0.6565 0.9523 0.8522 0.6641 0.9582 0.7797 0.9081 0.8306 0.7098 0.8079] 2020-12-25 18:28:30 [INFO] [EVAL] Class Acc: [0.991 0.9272 0.9664 0.8463 0.8406 0.8401 0.8594 0.9166 0.9552 0.8568 0.9668 0.9076 0.8302 0.9743 0.9209 0.9575 0.8897 0.8328 0.8779] 2020-12-25 18:28:35 [INFO] [EVAL] The model with the best validation mIoU (0.8086) was saved at iter 72000. 2020-12-25 18:30:37 [INFO] [TRAIN] epoch=205, iter=76100/80000, loss=15.5199, lr=0.000660, batch_cost=1.2165, reader_cost=0.0005 | ETA 01:19:04 2020-12-25 18:32:39 [INFO] [TRAIN] epoch=205, iter=76200/80000, loss=15.4051, lr=0.000644, batch_cost=1.2235, reader_cost=0.0008 | ETA 01:17:29 2020-12-25 18:35:05 [INFO] [TRAIN] epoch=206, iter=76300/80000, loss=16.0163, lr=0.000629, batch_cost=1.4514, reader_cost=0.0720 | ETA 01:29:30 2020-12-25 18:37:07 [INFO] [TRAIN] epoch=206, iter=76400/80000, loss=15.4981, lr=0.000614, batch_cost=1.2210, reader_cost=0.0004 | ETA 01:13:15 2020-12-25 18:39:10 [INFO] [TRAIN] epoch=206, iter=76500/80000, loss=15.2203, lr=0.000598, batch_cost=1.2292, reader_cost=0.0004 | ETA 01:11:42 2020-12-25 18:41:13 [INFO] [TRAIN] epoch=206, iter=76600/80000, loss=15.4482, lr=0.000583, batch_cost=1.2281, reader_cost=0.0008 | ETA 01:09:35 2020-12-25 18:43:36 [INFO] [TRAIN] epoch=207, iter=76700/80000, loss=15.8465, lr=0.000568, batch_cost=1.4328, reader_cost=0.0751 | ETA 01:18:48 2020-12-25 18:45:40 [INFO] [TRAIN] epoch=207, iter=76800/80000, loss=15.3193, lr=0.000552, batch_cost=1.2357, reader_cost=0.0006 | ETA 01:05:54 2020-12-25 18:47:44 [INFO] [TRAIN] epoch=207, iter=76900/80000, loss=15.1478, lr=0.000537, batch_cost=1.2349, reader_cost=0.0004 | ETA 01:03:48 2020-12-25 18:49:45 [INFO] [TRAIN] epoch=207, iter=77000/80000, loss=15.7290, lr=0.000521, batch_cost=1.2141, reader_cost=0.0002 | ETA 01:00:42 2020-12-25 18:52:05 [INFO] [TRAIN] epoch=208, iter=77100/80000, loss=15.6728, lr=0.000505, batch_cost=1.3990, reader_cost=0.0456 | ETA 01:07:36 2020-12-25 18:54:08 [INFO] [TRAIN] epoch=208, iter=77200/80000, loss=15.4154, lr=0.000490, batch_cost=1.2274, reader_cost=0.0004 | ETA 00:57:16 2020-12-25 18:56:11 [INFO] [TRAIN] epoch=208, iter=77300/80000, loss=15.4615, lr=0.000474, batch_cost=1.2258, reader_cost=0.0003 | ETA 00:55:09 2020-12-25 18:58:33 [INFO] [TRAIN] epoch=209, iter=77400/80000, loss=15.6913, lr=0.000458, batch_cost=1.4129, reader_cost=0.0493 | ETA 01:01:13 2020-12-25 19:00:36 [INFO] [TRAIN] epoch=209, iter=77500/80000, loss=15.3994, lr=0.000442, batch_cost=1.2286, reader_cost=0.0006 | ETA 00:51:11 2020-12-25 19:02:40 [INFO] [TRAIN] epoch=209, iter=77600/80000, loss=15.4554, lr=0.000426, batch_cost=1.2389, reader_cost=0.0004 | ETA 00:49:33 2020-12-25 19:04:43 [INFO] [TRAIN] epoch=209, iter=77700/80000, loss=15.4334, lr=0.000410, batch_cost=1.2308, reader_cost=0.0008 | ETA 00:47:10 2020-12-25 19:07:09 [INFO] [TRAIN] epoch=210, iter=77800/80000, loss=15.7140, lr=0.000394, batch_cost=1.4494, reader_cost=0.0458 | ETA 00:53:08 2020-12-25 19:09:10 [INFO] [TRAIN] epoch=210, iter=77900/80000, loss=15.4625, lr=0.000378, batch_cost=1.2139, reader_cost=0.0009 | ETA 00:42:29 2020-12-25 19:11:14 [INFO] [TRAIN] epoch=210, iter=78000/80000, loss=15.1661, lr=0.000362, batch_cost=1.2321, reader_cost=0.0002 | ETA 00:41:04 2020-12-25 19:11:14 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 19:12:01 [INFO] [EVAL] #Images=500 mIoU=0.8085 Acc=0.9660 Kappa=0.9559 2020-12-25 19:12:01 [INFO] [EVAL] Class IoU: [0.9841 0.8711 0.935 0.5473 0.6529 0.7127 0.7592 0.8331 0.9311 0.6556 0.9533 0.851 0.6667 0.9601 0.7994 0.8946 0.8434 0.7044 0.8071] 2020-12-25 19:12:01 [INFO] [EVAL] Class Acc: [0.9924 0.9285 0.9642 0.8209 0.8445 0.845 0.8655 0.9114 0.9593 0.8388 0.9689 0.9061 0.8076 0.9775 0.9096 0.9375 0.9093 0.8362 0.8796] 2020-12-25 19:12:06 [INFO] [EVAL] The model with the best validation mIoU (0.8086) was saved at iter 72000. 2020-12-25 19:14:10 [INFO] [TRAIN] epoch=210, iter=78100/80000, loss=15.5101, lr=0.000345, batch_cost=1.2345, reader_cost=0.0008 | ETA 00:39:05 2020-12-25 19:16:34 [INFO] [TRAIN] epoch=211, iter=78200/80000, loss=15.8326, lr=0.000329, batch_cost=1.4412, reader_cost=0.0503 | ETA 00:43:14 2020-12-25 19:18:38 [INFO] [TRAIN] epoch=211, iter=78300/80000, loss=15.3203, lr=0.000313, batch_cost=1.2354, reader_cost=0.0004 | ETA 00:35:00 2020-12-25 19:20:41 [INFO] [TRAIN] epoch=211, iter=78400/80000, loss=15.3396, lr=0.000296, batch_cost=1.2220, reader_cost=0.0005 | ETA 00:32:35 2020-12-25 19:23:04 [INFO] [TRAIN] epoch=212, iter=78500/80000, loss=15.6229, lr=0.000279, batch_cost=1.4282, reader_cost=0.0404 | ETA 00:35:42 2020-12-25 19:25:07 [INFO] [TRAIN] epoch=212, iter=78600/80000, loss=15.6774, lr=0.000262, batch_cost=1.2289, reader_cost=0.0003 | ETA 00:28:40 2020-12-25 19:27:13 [INFO] [TRAIN] epoch=212, iter=78700/80000, loss=15.5535, lr=0.000246, batch_cost=1.2593, reader_cost=0.0012 | ETA 00:27:17 2020-12-25 19:29:18 [INFO] [TRAIN] epoch=212, iter=78800/80000, loss=15.5172, lr=0.000228, batch_cost=1.2490, reader_cost=0.0002 | ETA 00:24:58 2020-12-25 19:31:38 [INFO] [TRAIN] epoch=213, iter=78900/80000, loss=15.9778, lr=0.000211, batch_cost=1.4001, reader_cost=0.0447 | ETA 00:25:40 2020-12-25 19:33:41 [INFO] [TRAIN] epoch=213, iter=79000/80000, loss=15.4136, lr=0.000194, batch_cost=1.2190, reader_cost=0.0005 | ETA 00:20:18 2020-12-25 19:35:43 [INFO] [TRAIN] epoch=213, iter=79100/80000, loss=15.4905, lr=0.000176, batch_cost=1.2190, reader_cost=0.0004 | ETA 00:18:17 2020-12-25 19:37:44 [INFO] [TRAIN] epoch=213, iter=79200/80000, loss=15.6486, lr=0.000159, batch_cost=1.2074, reader_cost=0.0008 | ETA 00:16:05 2020-12-25 19:40:11 [INFO] [TRAIN] epoch=214, iter=79300/80000, loss=15.8739, lr=0.000141, batch_cost=1.4661, reader_cost=0.0452 | ETA 00:17:06 2020-12-25 19:42:14 [INFO] [TRAIN] epoch=214, iter=79400/80000, loss=15.3672, lr=0.000123, batch_cost=1.2313, reader_cost=0.0005 | ETA 00:12:18 2020-12-25 19:44:18 [INFO] [TRAIN] epoch=214, iter=79500/80000, loss=15.2399, lr=0.000104, batch_cost=1.2432, reader_cost=0.0006 | ETA 00:10:21 2020-12-25 19:46:22 [INFO] [TRAIN] epoch=214, iter=79600/80000, loss=15.7077, lr=0.000085, batch_cost=1.2283, reader_cost=0.0008 | ETA 00:08:11 2020-12-25 19:48:47 [INFO] [TRAIN] epoch=215, iter=79700/80000, loss=15.6835, lr=0.000066, batch_cost=1.4523, reader_cost=0.0403 | ETA 00:07:15 2020-12-25 19:50:50 [INFO] [TRAIN] epoch=215, iter=79800/80000, loss=15.2225, lr=0.000046, batch_cost=1.2293, reader_cost=0.0005 | ETA 00:04:05 2020-12-25 19:52:53 [INFO] [TRAIN] epoch=215, iter=79900/80000, loss=15.4244, lr=0.000025, batch_cost=1.2289, reader_cost=0.0002 | ETA 00:02:02 2020-12-25 19:55:18 [INFO] [TRAIN] epoch=216, iter=80000/80000, loss=15.7326, lr=0.000000, batch_cost=1.4479, reader_cost=0.0439 | ETA 00:00:00 2020-12-25 19:55:19 [INFO] Start evaluating (total_samples=500, total_iters=125)... 2020-12-25 19:56:05 [INFO] [EVAL] #Images=500 mIoU=0.8085 Acc=0.9660 Kappa=0.9558 2020-12-25 19:56:05 [INFO] [EVAL] Class IoU: [0.9838 0.8706 0.935 0.5571 0.6557 0.7127 0.7591 0.833 0.9309 0.6541 0.9525 0.8524 0.6654 0.9598 0.7916 0.8913 0.8407 0.7076 0.8077] 2020-12-25 19:56:05 [INFO] [EVAL] Class Acc: [0.992 0.9267 0.9649 0.8185 0.8522 0.8466 0.8657 0.9187 0.9579 0.8491 0.9673 0.9111 0.8073 0.9767 0.9251 0.9404 0.9008 0.8419 0.8813] 2020-12-25 19:56:10 [INFO] [EVAL] The model with the best validation mIoU (0.8086) was saved at iter 72000.