2020-11-02 19:10:17 [INFO] ------------Environment Information------------- platform: Linux-3.10.0-1062.18.1.el7.x86_64-x86_64-with-centos-7.7.1908-Core Python: 3.7.9 (default, Aug 31 2020, 12:42:55) [GCC 7.3.0] Paddle compiled with cuda: True NVCC: Cuda compilation tools, release 10.2, V10.2.89 cudnn: 7.6 GPUs used: 4 CUDA_VISIBLE_DEVICES: 0,1,2,3 GPU: ['GPU 0: Tesla V100-SXM2-16GB', 'GPU 1: Tesla V100-SXM2-16GB', 'GPU 2: Tesla V100-SXM2-16GB', 'GPU 3: Tesla V100-SXM2-16GB', 'GPU 4: Tesla V100-SXM2-16GB', 'GPU 5: Tesla V100-SXM2-16GB', 'GPU 6: Tesla V100-SXM2-16GB', 'GPU 7: Tesla V100-SXM2-16GB'] GCC: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39) PaddlePaddle: 2.0.0-rc0 OpenCV: 4.1.1 ------------------------------------------------ 2020-11-02 19:10:17 [INFO] ---------------Config Information--------------- batch_size: 2 iters: 80000 learning_rate: decay: end_lr: 0.0 power: 0.9 type: poly value: 0.01 loss: coef: - 1 types: - ignore_index: 255 type: CrossEntropyLoss model: align_corners: false aspp_out_channels: 256 aspp_ratios: - 1 - 12 - 24 - 36 backbone: multi_grid: - 1 - 2 - 4 output_stride: 8 pretrained: https://bj.bcebos.com/paddleseg/dygraph/resnet50_vd_ssld_v2.tar.gz type: ResNet50_vd backbone_indices: - 3 pretrained: null type: DeepLabV3 optimizer: momentum: 0.9 type: sgd weight_decay: 4.0e-05 train_dataset: dataset_root: data/cityscapes mode: train transforms: - max_scale_factor: 2.0 min_scale_factor: 0.5 scale_step_size: 0.25 type: ResizeStepScaling - crop_size: - 1024 - 512 type: RandomPaddingCrop - type: RandomHorizontalFlip - brightness_range: 0.4 contrast_range: 0.4 saturation_range: 0.4 type: RandomDistort - type: Normalize type: Cityscapes val_dataset: dataset_root: data/cityscapes mode: val transforms: - type: Normalize type: Cityscapes ------------------------------------------------ 2020-11-02 19:10:23 [INFO] Loading pretrained model from https://bj.bcebos.com/paddleseg/dygraph/resnet50_vd_ssld_v2.tar.gz 2020-11-02 19:10:24 [INFO] There are 275/275 variables loaded into ResNet_vd. 2020-11-02 19:12:15 [INFO] [TRAIN] epoch=1, iter=100/80000, loss=0.9921, lr=0.009989, batch_cost=1.0459, reader_cost=0.0167 | ETA 23:12:45 2020-11-02 19:13:54 [INFO] [TRAIN] epoch=1, iter=200/80000, loss=0.6946, lr=0.009978, batch_cost=0.9927, reader_cost=0.0002 | ETA 22:00:16 2020-11-02 19:15:38 [INFO] [TRAIN] epoch=1, iter=300/80000, loss=0.4436, lr=0.009966, batch_cost=1.0412, reader_cost=0.0002 | ETA 23:03:04 2020-11-02 19:17:20 [INFO] [TRAIN] epoch=2, iter=400/80000, loss=0.4713, lr=0.009955, batch_cost=1.0127, reader_cost=0.0119 | ETA 22:23:28 2020-11-02 19:19:00 [INFO] [TRAIN] epoch=2, iter=500/80000, loss=0.4042, lr=0.009944, batch_cost=1.0063, reader_cost=0.0004 | ETA 22:13:21 2020-11-02 19:20:44 [INFO] [TRAIN] epoch=2, iter=600/80000, loss=0.4199, lr=0.009933, batch_cost=1.0392, reader_cost=0.0002 | ETA 22:55:10 2020-11-02 19:22:25 [INFO] [TRAIN] epoch=2, iter=700/80000, loss=0.3527, lr=0.009921, batch_cost=1.0061, reader_cost=0.0002 | ETA 22:09:46 2020-11-02 19:24:09 [INFO] [TRAIN] epoch=3, iter=800/80000, loss=0.3829, lr=0.009910, batch_cost=1.0376, reader_cost=0.0101 | ETA 22:49:40 2020-11-02 19:26:01 [INFO] [TRAIN] epoch=3, iter=900/80000, loss=0.3682, lr=0.009899, batch_cost=1.1276, reader_cost=0.0002 | ETA 24:46:30 2020-11-02 19:27:45 [INFO] [TRAIN] epoch=3, iter=1000/80000, loss=0.2922, lr=0.009888, batch_cost=1.0341, reader_cost=0.0002 | ETA 22:41:34 2020-11-02 19:29:33 [INFO] [TRAIN] epoch=3, iter=1100/80000, loss=0.3302, lr=0.009876, batch_cost=1.0849, reader_cost=0.0002 | ETA 23:46:37 2020-11-02 19:31:14 [INFO] [TRAIN] epoch=4, iter=1200/80000, loss=0.2874, lr=0.009865, batch_cost=1.0074, reader_cost=0.0099 | ETA 22:03:05 2020-11-02 19:32:54 [INFO] [TRAIN] epoch=4, iter=1300/80000, loss=0.2712, lr=0.009854, batch_cost=1.0043, reader_cost=0.0002 | ETA 21:57:16 2020-11-02 19:34:43 [INFO] [TRAIN] epoch=4, iter=1400/80000, loss=0.3088, lr=0.009842, batch_cost=1.0825, reader_cost=0.0002 | ETA 23:38:07 2020-11-02 19:36:25 [INFO] [TRAIN] epoch=5, iter=1500/80000, loss=0.2924, lr=0.009831, batch_cost=1.0221, reader_cost=0.0104 | ETA 22:17:12 2020-11-02 19:38:04 [INFO] [TRAIN] epoch=5, iter=1600/80000, loss=0.2870, lr=0.009820, batch_cost=0.9931, reader_cost=0.0002 | ETA 21:37:35 2020-11-02 19:39:46 [INFO] [TRAIN] epoch=5, iter=1700/80000, loss=0.2912, lr=0.009809, batch_cost=1.0135, reader_cost=0.0002 | ETA 22:02:39 2020-11-02 19:41:29 [INFO] [TRAIN] epoch=5, iter=1800/80000, loss=0.2410, lr=0.009797, batch_cost=1.0351, reader_cost=0.0002 | ETA 22:29:08 2020-11-02 19:43:13 [INFO] [TRAIN] epoch=6, iter=1900/80000, loss=0.2948, lr=0.009786, batch_cost=1.0433, reader_cost=0.0105 | ETA 22:37:59 2020-11-02 19:44:53 [INFO] [TRAIN] epoch=6, iter=2000/80000, loss=0.2454, lr=0.009775, batch_cost=0.9949, reader_cost=0.0002 | ETA 21:33:25 2020-11-02 19:46:34 [INFO] [TRAIN] epoch=6, iter=2100/80000, loss=0.2228, lr=0.009764, batch_cost=1.0145, reader_cost=0.0002 | ETA 21:57:06 2020-11-02 19:48:14 [INFO] [TRAIN] epoch=6, iter=2200/80000, loss=0.2406, lr=0.009752, batch_cost=1.0014, reader_cost=0.0002 | ETA 21:38:31 2020-11-02 19:49:56 [INFO] [TRAIN] epoch=7, iter=2300/80000, loss=0.2588, lr=0.009741, batch_cost=1.0171, reader_cost=0.0123 | ETA 21:57:11 2020-11-02 19:51:42 [INFO] [TRAIN] epoch=7, iter=2400/80000, loss=0.3029, lr=0.009730, batch_cost=1.0543, reader_cost=0.0002 | ETA 22:43:35 2020-11-02 19:53:26 [INFO] [TRAIN] epoch=7, iter=2500/80000, loss=0.2782, lr=0.009718, batch_cost=1.0452, reader_cost=0.0002 | ETA 22:30:06 2020-11-02 19:55:09 [INFO] [TRAIN] epoch=7, iter=2600/80000, loss=0.2439, lr=0.009707, batch_cost=1.0270, reader_cost=0.0004 | ETA 22:04:48 2020-11-02 19:56:51 [INFO] [TRAIN] epoch=8, iter=2700/80000, loss=0.2068, lr=0.009696, batch_cost=1.0222, reader_cost=0.0136 | ETA 21:56:53 2020-11-02 19:58:36 [INFO] [TRAIN] epoch=8, iter=2800/80000, loss=0.2659, lr=0.009685, batch_cost=1.0505, reader_cost=0.0009 | ETA 22:31:41 2020-11-02 20:00:18 [INFO] [TRAIN] epoch=8, iter=2900/80000, loss=0.2061, lr=0.009673, batch_cost=1.0214, reader_cost=0.0002 | ETA 21:52:32 2020-11-02 20:02:02 [INFO] [TRAIN] epoch=9, iter=3000/80000, loss=0.2695, lr=0.009662, batch_cost=1.0407, reader_cost=0.0098 | ETA 22:15:30 2020-11-02 20:03:44 [INFO] [TRAIN] epoch=9, iter=3100/80000, loss=0.2642, lr=0.009651, batch_cost=1.0201, reader_cost=0.0003 | ETA 21:47:28 2020-11-02 20:05:28 [INFO] [TRAIN] epoch=9, iter=3200/80000, loss=0.2628, lr=0.009639, batch_cost=1.0362, reader_cost=0.0002 | ETA 22:06:21 2020-11-02 20:07:11 [INFO] [TRAIN] epoch=9, iter=3300/80000, loss=0.2839, lr=0.009628, batch_cost=1.0321, reader_cost=0.0002 | ETA 21:59:23 2020-11-02 20:08:59 [INFO] [TRAIN] epoch=10, iter=3400/80000, loss=0.2236, lr=0.009617, batch_cost=1.0826, reader_cost=0.0098 | ETA 23:02:10 2020-11-02 20:10:40 [INFO] [TRAIN] epoch=10, iter=3500/80000, loss=0.2254, lr=0.009605, batch_cost=1.0026, reader_cost=0.0005 | ETA 21:18:19 2020-11-02 20:12:19 [INFO] [TRAIN] epoch=10, iter=3600/80000, loss=0.2337, lr=0.009594, batch_cost=0.9903, reader_cost=0.0002 | ETA 21:00:57 2020-11-02 20:14:01 [INFO] [TRAIN] epoch=10, iter=3700/80000, loss=0.2178, lr=0.009583, batch_cost=1.0175, reader_cost=0.0002 | ETA 21:33:54 2020-11-02 20:15:42 [INFO] [TRAIN] epoch=11, iter=3800/80000, loss=0.2007, lr=0.009572, batch_cost=1.0118, reader_cost=0.0101 | ETA 21:25:01 2020-11-02 20:17:23 [INFO] [TRAIN] epoch=11, iter=3900/80000, loss=0.2198, lr=0.009560, batch_cost=1.0094, reader_cost=0.0002 | ETA 21:20:17 2020-11-02 20:19:03 [INFO] [TRAIN] epoch=11, iter=4000/80000, loss=0.1994, lr=0.009549, batch_cost=1.0038, reader_cost=0.0002 | ETA 21:11:26 2020-11-02 20:20:44 [INFO] [TRAIN] epoch=12, iter=4100/80000, loss=0.2253, lr=0.009538, batch_cost=1.0116, reader_cost=0.0117 | ETA 21:19:37 2020-11-02 20:22:24 [INFO] [TRAIN] epoch=12, iter=4200/80000, loss=0.2287, lr=0.009526, batch_cost=1.0004, reader_cost=0.0002 | ETA 21:03:53 2020-11-02 20:24:06 [INFO] [TRAIN] epoch=12, iter=4300/80000, loss=0.2338, lr=0.009515, batch_cost=1.0194, reader_cost=0.0007 | ETA 21:26:11 2020-11-02 20:25:48 [INFO] [TRAIN] epoch=12, iter=4400/80000, loss=0.1941, lr=0.009504, batch_cost=1.0173, reader_cost=0.0002 | ETA 21:21:51 2020-11-02 20:27:34 [INFO] [TRAIN] epoch=13, iter=4500/80000, loss=0.2135, lr=0.009492, batch_cost=1.0621, reader_cost=0.0102 | ETA 22:16:30 2020-11-02 20:29:15 [INFO] [TRAIN] epoch=13, iter=4600/80000, loss=0.2522, lr=0.009481, batch_cost=1.0084, reader_cost=0.0002 | ETA 21:07:16 2020-11-02 20:30:57 [INFO] [TRAIN] epoch=13, iter=4700/80000, loss=0.2155, lr=0.009470, batch_cost=1.0192, reader_cost=0.0003 | ETA 21:19:03 2020-11-02 20:32:41 [INFO] [TRAIN] epoch=13, iter=4800/80000, loss=0.2101, lr=0.009458, batch_cost=1.0388, reader_cost=0.0009 | ETA 21:41:59 2020-11-02 20:34:22 [INFO] [TRAIN] epoch=14, iter=4900/80000, loss=0.2030, lr=0.009447, batch_cost=1.0129, reader_cost=0.0132 | ETA 21:07:47 2020-11-02 20:36:01 [INFO] [TRAIN] epoch=14, iter=5000/80000, loss=0.2136, lr=0.009436, batch_cost=0.9916, reader_cost=0.0002 | ETA 20:39:27 2020-11-02 20:37:40 [INFO] [TRAIN] epoch=14, iter=5100/80000, loss=0.1864, lr=0.009424, batch_cost=0.9899, reader_cost=0.0002 | ETA 20:35:46 2020-11-02 20:39:20 [INFO] [TRAIN] epoch=14, iter=5200/80000, loss=0.2081, lr=0.009413, batch_cost=0.9941, reader_cost=0.0002 | ETA 20:39:15 2020-11-02 20:41:01 [INFO] [TRAIN] epoch=15, iter=5300/80000, loss=0.2333, lr=0.009402, batch_cost=1.0136, reader_cost=0.0127 | ETA 21:01:58 2020-11-02 20:42:43 [INFO] [TRAIN] epoch=15, iter=5400/80000, loss=0.2214, lr=0.009391, batch_cost=1.0182, reader_cost=0.0002 | ETA 21:05:56 2020-11-02 20:44:25 [INFO] [TRAIN] epoch=15, iter=5500/80000, loss=0.1862, lr=0.009379, batch_cost=1.0229, reader_cost=0.0017 | ETA 21:10:09 2020-11-02 20:46:13 [INFO] [TRAIN] epoch=16, iter=5600/80000, loss=0.2225, lr=0.009368, batch_cost=1.0746, reader_cost=0.0103 | ETA 22:12:26 2020-11-02 20:47:56 [INFO] [TRAIN] epoch=16, iter=5700/80000, loss=0.2045, lr=0.009357, batch_cost=1.0322, reader_cost=0.0003 | ETA 21:18:15 2020-11-02 20:49:36 [INFO] [TRAIN] epoch=16, iter=5800/80000, loss=0.1728, lr=0.009345, batch_cost=1.0025, reader_cost=0.0004 | ETA 20:39:48 2020-11-02 20:51:19 [INFO] [TRAIN] epoch=16, iter=5900/80000, loss=0.1913, lr=0.009334, batch_cost=1.0258, reader_cost=0.0002 | ETA 21:06:49 2020-11-02 20:53:00 [INFO] [TRAIN] epoch=17, iter=6000/80000, loss=0.2016, lr=0.009323, batch_cost=1.0106, reader_cost=0.0100 | ETA 20:46:22 2020-11-02 20:54:40 [INFO] [TRAIN] epoch=17, iter=6100/80000, loss=0.2226, lr=0.009311, batch_cost=1.0040, reader_cost=0.0002 | ETA 20:36:33 2020-11-02 20:56:21 [INFO] [TRAIN] epoch=17, iter=6200/80000, loss=0.1927, lr=0.009300, batch_cost=1.0079, reader_cost=0.0002 | ETA 20:39:40 2020-11-02 20:58:01 [INFO] [TRAIN] epoch=17, iter=6300/80000, loss=0.2099, lr=0.009288, batch_cost=1.0041, reader_cost=0.0002 | ETA 20:33:22 2020-11-02 20:59:44 [INFO] [TRAIN] epoch=18, iter=6400/80000, loss=0.2245, lr=0.009277, batch_cost=1.0317, reader_cost=0.0106 | ETA 21:05:36 2020-11-02 21:01:26 [INFO] [TRAIN] epoch=18, iter=6500/80000, loss=0.1847, lr=0.009266, batch_cost=1.0111, reader_cost=0.0006 | ETA 20:38:33 2020-11-02 21:03:05 [INFO] [TRAIN] epoch=18, iter=6600/80000, loss=0.1735, lr=0.009254, batch_cost=0.9983, reader_cost=0.0002 | ETA 20:21:13 2020-11-02 21:04:48 [INFO] [TRAIN] epoch=19, iter=6700/80000, loss=0.1877, lr=0.009243, batch_cost=1.0220, reader_cost=0.0122 | ETA 20:48:29 2020-11-02 21:06:28 [INFO] [TRAIN] epoch=19, iter=6800/80000, loss=0.1780, lr=0.009232, batch_cost=0.9992, reader_cost=0.0002 | ETA 20:19:01 2020-11-02 21:08:10 [INFO] [TRAIN] epoch=19, iter=6900/80000, loss=0.2092, lr=0.009220, batch_cost=1.0199, reader_cost=0.0002 | ETA 20:42:34 2020-11-02 21:09:49 [INFO] [TRAIN] epoch=19, iter=7000/80000, loss=0.1601, lr=0.009209, batch_cost=0.9944, reader_cost=0.0002 | ETA 20:09:48 2020-11-02 21:11:31 [INFO] [TRAIN] epoch=20, iter=7100/80000, loss=0.2073, lr=0.009198, batch_cost=1.0208, reader_cost=0.0101 | ETA 20:40:16 2020-11-02 21:13:12 [INFO] [TRAIN] epoch=20, iter=7200/80000, loss=0.1659, lr=0.009186, batch_cost=1.0129, reader_cost=0.0002 | ETA 20:28:59 2020-11-02 21:14:53 [INFO] [TRAIN] epoch=20, iter=7300/80000, loss=0.1749, lr=0.009175, batch_cost=1.0111, reader_cost=0.0002 | ETA 20:25:05 2020-11-02 21:16:38 [INFO] [TRAIN] epoch=20, iter=7400/80000, loss=0.1799, lr=0.009164, batch_cost=1.0440, reader_cost=0.0002 | ETA 21:03:13 2020-11-02 21:18:19 [INFO] [TRAIN] epoch=21, iter=7500/80000, loss=0.1755, lr=0.009152, batch_cost=1.0142, reader_cost=0.0105 | ETA 20:25:32 2020-11-02 21:20:01 [INFO] [TRAIN] epoch=21, iter=7600/80000, loss=0.1921, lr=0.009141, batch_cost=1.0216, reader_cost=0.0005 | ETA 20:32:44 2020-11-02 21:21:42 [INFO] [TRAIN] epoch=21, iter=7700/80000, loss=0.1534, lr=0.009130, batch_cost=1.0014, reader_cost=0.0002 | ETA 20:06:41 2020-11-02 21:23:23 [INFO] [TRAIN] epoch=21, iter=7800/80000, loss=0.1717, lr=0.009118, batch_cost=1.0104, reader_cost=0.0002 | ETA 20:15:48 2020-11-02 21:25:03 [INFO] [TRAIN] epoch=22, iter=7900/80000, loss=0.1799, lr=0.009107, batch_cost=1.0011, reader_cost=0.0104 | ETA 20:02:59 2020-11-02 21:26:43 [INFO] [TRAIN] epoch=22, iter=8000/80000, loss=0.1823, lr=0.009095, batch_cost=1.0003, reader_cost=0.0002 | ETA 20:00:22 2020-11-02 21:26:47 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-02 21:31:23 [INFO] [EVAL] #Images=500 mIoU=0.6832 Acc=0.9483 Kappa=0.9329 2020-11-02 21:31:23 [INFO] [EVAL] Category IoU: [0.9762 0.8097 0.9079 0.4952 0.5411 0.5828 0.6409 0.7328 0.913 0.5674 0.9344 0.7853 0.5198 0.9045 0.635 0.482 0.2255 0.5929 0.7339] 2020-11-02 21:31:23 [INFO] [EVAL] Category Acc: [0.989 0.8845 0.9428 0.7746 0.6897 0.7641 0.7529 0.8709 0.9554 0.7937 0.9619 0.8616 0.7938 0.9771 0.7665 0.5044 0.9391 0.7365 0.8145] 2020-11-02 21:31:25 [INFO] [EVAL] The model with the best validation mIoU (0.6832) was saved at iter 8000. 2020-11-02 21:33:06 [INFO] [TRAIN] epoch=22, iter=8100/80000, loss=0.1964, lr=0.009084, batch_cost=1.0087, reader_cost=0.0002 | ETA 20:08:43 2020-11-02 21:34:48 [INFO] [TRAIN] epoch=23, iter=8200/80000, loss=0.1857, lr=0.009073, batch_cost=1.0162, reader_cost=0.0132 | ETA 20:16:01 2020-11-02 21:36:27 [INFO] [TRAIN] epoch=23, iter=8300/80000, loss=0.1735, lr=0.009061, batch_cost=0.9911, reader_cost=0.0002 | ETA 19:44:24 2020-11-02 21:38:07 [INFO] [TRAIN] epoch=23, iter=8400/80000, loss=0.1588, lr=0.009050, batch_cost=1.0025, reader_cost=0.0003 | ETA 19:56:15 2020-11-02 21:39:52 [INFO] [TRAIN] epoch=23, iter=8500/80000, loss=0.1751, lr=0.009039, batch_cost=1.0481, reader_cost=0.0002 | ETA 20:48:55 2020-11-02 21:41:38 [INFO] [TRAIN] epoch=24, iter=8600/80000, loss=0.1805, lr=0.009027, batch_cost=1.0560, reader_cost=0.0160 | ETA 20:56:40 2020-11-02 21:43:20 [INFO] [TRAIN] epoch=24, iter=8700/80000, loss=0.1713, lr=0.009016, batch_cost=1.0208, reader_cost=0.0002 | ETA 20:13:04 2020-11-02 21:44:59 [INFO] [TRAIN] epoch=24, iter=8800/80000, loss=0.1592, lr=0.009004, batch_cost=0.9895, reader_cost=0.0002 | ETA 19:34:09 2020-11-02 21:46:40 [INFO] [TRAIN] epoch=24, iter=8900/80000, loss=0.1955, lr=0.008993, batch_cost=1.0159, reader_cost=0.0002 | ETA 20:03:52 2020-11-02 21:48:23 [INFO] [TRAIN] epoch=25, iter=9000/80000, loss=0.1723, lr=0.008982, batch_cost=1.0248, reader_cost=0.0122 | ETA 20:12:42 2020-11-02 21:50:07 [INFO] [TRAIN] epoch=25, iter=9100/80000, loss=0.1700, lr=0.008970, batch_cost=1.0392, reader_cost=0.0003 | ETA 20:28:00 2020-11-02 21:51:47 [INFO] [TRAIN] epoch=25, iter=9200/80000, loss=0.1853, lr=0.008959, batch_cost=0.9999, reader_cost=0.0002 | ETA 19:39:52 2020-11-02 21:53:28 [INFO] [TRAIN] epoch=25, iter=9300/80000, loss=0.1785, lr=0.008948, batch_cost=1.0088, reader_cost=0.0002 | ETA 19:48:39 2020-11-02 21:55:08 [INFO] [TRAIN] epoch=26, iter=9400/80000, loss=0.1520, lr=0.008936, batch_cost=0.9995, reader_cost=0.0131 | ETA 19:36:02 2020-11-02 21:56:48 [INFO] [TRAIN] epoch=26, iter=9500/80000, loss=0.1881, lr=0.008925, batch_cost=1.0087, reader_cost=0.0002 | ETA 19:45:12 2020-11-02 21:58:29 [INFO] [TRAIN] epoch=26, iter=9600/80000, loss=0.1977, lr=0.008913, batch_cost=1.0069, reader_cost=0.0002 | ETA 19:41:22 2020-11-02 22:00:11 [INFO] [TRAIN] epoch=27, iter=9700/80000, loss=0.1703, lr=0.008902, batch_cost=1.0166, reader_cost=0.0143 | ETA 19:51:08 2020-11-02 22:01:51 [INFO] [TRAIN] epoch=27, iter=9800/80000, loss=0.1950, lr=0.008891, batch_cost=1.0055, reader_cost=0.0002 | ETA 19:36:27 2020-11-02 22:03:30 [INFO] [TRAIN] epoch=27, iter=9900/80000, loss=0.1854, lr=0.008879, batch_cost=0.9918, reader_cost=0.0002 | ETA 19:18:46 2020-11-02 22:05:09 [INFO] [TRAIN] epoch=27, iter=10000/80000, loss=0.1701, lr=0.008868, batch_cost=0.9859, reader_cost=0.0002 | ETA 19:10:09 2020-11-02 22:06:50 [INFO] [TRAIN] epoch=28, iter=10100/80000, loss=0.1987, lr=0.008856, batch_cost=1.0101, reader_cost=0.0114 | ETA 19:36:47 2020-11-02 22:08:32 [INFO] [TRAIN] epoch=28, iter=10200/80000, loss=0.1586, lr=0.008845, batch_cost=1.0145, reader_cost=0.0002 | ETA 19:40:08 2020-11-02 22:10:12 [INFO] [TRAIN] epoch=28, iter=10300/80000, loss=0.1638, lr=0.008834, batch_cost=1.0024, reader_cost=0.0002 | ETA 19:24:25 2020-11-02 22:11:52 [INFO] [TRAIN] epoch=28, iter=10400/80000, loss=0.1620, lr=0.008822, batch_cost=1.0036, reader_cost=0.0006 | ETA 19:24:12 2020-11-02 22:13:34 [INFO] [TRAIN] epoch=29, iter=10500/80000, loss=0.1853, lr=0.008811, batch_cost=1.0218, reader_cost=0.0146 | ETA 19:43:32 2020-11-02 22:15:15 [INFO] [TRAIN] epoch=29, iter=10600/80000, loss=0.1699, lr=0.008799, batch_cost=1.0080, reader_cost=0.0002 | ETA 19:25:55 2020-11-02 22:16:57 [INFO] [TRAIN] epoch=29, iter=10700/80000, loss=0.1607, lr=0.008788, batch_cost=1.0140, reader_cost=0.0003 | ETA 19:31:09 2020-11-02 22:18:38 [INFO] [TRAIN] epoch=30, iter=10800/80000, loss=0.1859, lr=0.008776, batch_cost=1.0138, reader_cost=0.0125 | ETA 19:29:14 2020-11-02 22:20:18 [INFO] [TRAIN] epoch=30, iter=10900/80000, loss=0.1708, lr=0.008765, batch_cost=0.9990, reader_cost=0.0004 | ETA 19:10:30 2020-11-02 22:22:00 [INFO] [TRAIN] epoch=30, iter=11000/80000, loss=0.1525, lr=0.008754, batch_cost=1.0181, reader_cost=0.0032 | ETA 19:30:45 2020-11-02 22:23:39 [INFO] [TRAIN] epoch=30, iter=11100/80000, loss=0.1513, lr=0.008742, batch_cost=0.9955, reader_cost=0.0003 | ETA 19:03:10 2020-11-02 22:25:21 [INFO] [TRAIN] epoch=31, iter=11200/80000, loss=0.1810, lr=0.008731, batch_cost=1.0135, reader_cost=0.0163 | ETA 19:22:05 2020-11-02 22:27:00 [INFO] [TRAIN] epoch=31, iter=11300/80000, loss=0.1802, lr=0.008719, batch_cost=0.9958, reader_cost=0.0015 | ETA 19:00:14 2020-11-02 22:28:42 [INFO] [TRAIN] epoch=31, iter=11400/80000, loss=0.1808, lr=0.008708, batch_cost=1.0186, reader_cost=0.0014 | ETA 19:24:37 2020-11-02 22:30:22 [INFO] [TRAIN] epoch=31, iter=11500/80000, loss=0.1687, lr=0.008697, batch_cost=0.9978, reader_cost=0.0002 | ETA 18:59:06 2020-11-02 22:32:10 [INFO] [TRAIN] epoch=32, iter=11600/80000, loss=0.1535, lr=0.008685, batch_cost=1.0776, reader_cost=0.0130 | ETA 20:28:27 2020-11-02 22:33:48 [INFO] [TRAIN] epoch=32, iter=11700/80000, loss=0.1567, lr=0.008674, batch_cost=0.9836, reader_cost=0.0002 | ETA 18:39:36 2020-11-02 22:35:28 [INFO] [TRAIN] epoch=32, iter=11800/80000, loss=0.1571, lr=0.008662, batch_cost=1.0027, reader_cost=0.0002 | ETA 18:59:46 2020-11-02 22:37:08 [INFO] [TRAIN] epoch=32, iter=11900/80000, loss=0.1810, lr=0.008651, batch_cost=1.0033, reader_cost=0.0002 | ETA 18:58:41 2020-11-02 22:38:51 [INFO] [TRAIN] epoch=33, iter=12000/80000, loss=0.1612, lr=0.008639, batch_cost=1.0260, reader_cost=0.0122 | ETA 19:22:45 2020-11-02 22:40:32 [INFO] [TRAIN] epoch=33, iter=12100/80000, loss=0.1598, lr=0.008628, batch_cost=1.0082, reader_cost=0.0002 | ETA 19:00:56 2020-11-02 22:42:11 [INFO] [TRAIN] epoch=33, iter=12200/80000, loss=0.1520, lr=0.008617, batch_cost=0.9947, reader_cost=0.0002 | ETA 18:43:58 2020-11-02 22:43:51 [INFO] [TRAIN] epoch=34, iter=12300/80000, loss=0.1544, lr=0.008605, batch_cost=0.9949, reader_cost=0.0123 | ETA 18:42:31 2020-11-02 22:45:30 [INFO] [TRAIN] epoch=34, iter=12400/80000, loss=0.1673, lr=0.008594, batch_cost=0.9907, reader_cost=0.0002 | ETA 18:36:08 2020-11-02 22:47:10 [INFO] [TRAIN] epoch=34, iter=12500/80000, loss=0.1446, lr=0.008582, batch_cost=0.9999, reader_cost=0.0002 | ETA 18:44:55 2020-11-02 22:48:49 [INFO] [TRAIN] epoch=34, iter=12600/80000, loss=0.1590, lr=0.008571, batch_cost=0.9950, reader_cost=0.0002 | ETA 18:37:42 2020-11-02 22:50:30 [INFO] [TRAIN] epoch=35, iter=12700/80000, loss=0.2078, lr=0.008559, batch_cost=1.0076, reader_cost=0.0127 | ETA 18:50:12 2020-11-02 22:52:11 [INFO] [TRAIN] epoch=35, iter=12800/80000, loss=0.1662, lr=0.008548, batch_cost=1.0040, reader_cost=0.0002 | ETA 18:44:31 2020-11-02 22:53:52 [INFO] [TRAIN] epoch=35, iter=12900/80000, loss=0.1675, lr=0.008536, batch_cost=1.0170, reader_cost=0.0002 | ETA 18:57:23 2020-11-02 22:55:33 [INFO] [TRAIN] epoch=35, iter=13000/80000, loss=0.1567, lr=0.008525, batch_cost=1.0082, reader_cost=0.0002 | ETA 18:45:47 2020-11-02 22:57:16 [INFO] [TRAIN] epoch=36, iter=13100/80000, loss=0.1900, lr=0.008514, batch_cost=1.0270, reader_cost=0.0105 | ETA 19:05:09 2020-11-02 22:59:01 [INFO] [TRAIN] epoch=36, iter=13200/80000, loss=0.1706, lr=0.008502, batch_cost=1.0501, reader_cost=0.0002 | ETA 19:29:08 2020-11-02 23:00:40 [INFO] [TRAIN] epoch=36, iter=13300/80000, loss=0.1616, lr=0.008491, batch_cost=0.9931, reader_cost=0.0002 | ETA 18:23:57 2020-11-02 23:02:28 [INFO] [TRAIN] epoch=37, iter=13400/80000, loss=0.1734, lr=0.008479, batch_cost=1.0832, reader_cost=0.0154 | ETA 20:02:19 2020-11-02 23:04:08 [INFO] [TRAIN] epoch=37, iter=13500/80000, loss=0.1854, lr=0.008468, batch_cost=0.9947, reader_cost=0.0005 | ETA 18:22:26 2020-11-02 23:05:46 [INFO] [TRAIN] epoch=37, iter=13600/80000, loss=0.1700, lr=0.008456, batch_cost=0.9804, reader_cost=0.0002 | ETA 18:04:56 2020-11-02 23:07:26 [INFO] [TRAIN] epoch=37, iter=13700/80000, loss=0.1423, lr=0.008445, batch_cost=1.0042, reader_cost=0.0002 | ETA 18:29:37 2020-11-02 23:09:09 [INFO] [TRAIN] epoch=38, iter=13800/80000, loss=0.1758, lr=0.008433, batch_cost=1.0300, reader_cost=0.0126 | ETA 18:56:27 2020-11-02 23:10:50 [INFO] [TRAIN] epoch=38, iter=13900/80000, loss=0.1622, lr=0.008422, batch_cost=1.0041, reader_cost=0.0002 | ETA 18:26:10 2020-11-02 23:12:31 [INFO] [TRAIN] epoch=38, iter=14000/80000, loss=0.1482, lr=0.008410, batch_cost=1.0070, reader_cost=0.0002 | ETA 18:27:42 2020-11-02 23:14:11 [INFO] [TRAIN] epoch=38, iter=14100/80000, loss=0.1533, lr=0.008399, batch_cost=1.0036, reader_cost=0.0002 | ETA 18:22:16 2020-11-02 23:15:52 [INFO] [TRAIN] epoch=39, iter=14200/80000, loss=0.1601, lr=0.008387, batch_cost=1.0110, reader_cost=0.0122 | ETA 18:28:44 2020-11-02 23:17:34 [INFO] [TRAIN] epoch=39, iter=14300/80000, loss=0.1569, lr=0.008376, batch_cost=1.0179, reader_cost=0.0002 | ETA 18:34:38 2020-11-02 23:19:15 [INFO] [TRAIN] epoch=39, iter=14400/80000, loss=0.1584, lr=0.008364, batch_cost=1.0142, reader_cost=0.0002 | ETA 18:28:49 2020-11-02 23:20:56 [INFO] [TRAIN] epoch=39, iter=14500/80000, loss=0.1551, lr=0.008353, batch_cost=1.0041, reader_cost=0.0002 | ETA 18:16:09 2020-11-02 23:22:40 [INFO] [TRAIN] epoch=40, iter=14600/80000, loss=0.1430, lr=0.008342, batch_cost=1.0489, reader_cost=0.0115 | ETA 19:03:17 2020-11-02 23:24:21 [INFO] [TRAIN] epoch=40, iter=14700/80000, loss=0.1548, lr=0.008330, batch_cost=1.0066, reader_cost=0.0002 | ETA 18:15:29 2020-11-02 23:26:01 [INFO] [TRAIN] epoch=40, iter=14800/80000, loss=0.1677, lr=0.008319, batch_cost=1.0023, reader_cost=0.0002 | ETA 18:09:07 2020-11-02 23:27:42 [INFO] [TRAIN] epoch=41, iter=14900/80000, loss=0.1634, lr=0.008307, batch_cost=1.0032, reader_cost=0.0114 | ETA 18:08:25 2020-11-02 23:29:20 [INFO] [TRAIN] epoch=41, iter=15000/80000, loss=0.1692, lr=0.008296, batch_cost=0.9868, reader_cost=0.0002 | ETA 17:49:02 2020-11-02 23:30:59 [INFO] [TRAIN] epoch=41, iter=15100/80000, loss=0.1568, lr=0.008284, batch_cost=0.9882, reader_cost=0.0002 | ETA 17:48:52 2020-11-02 23:32:38 [INFO] [TRAIN] epoch=41, iter=15200/80000, loss=0.1709, lr=0.008273, batch_cost=0.9920, reader_cost=0.0002 | ETA 17:51:23 2020-11-02 23:34:21 [INFO] [TRAIN] epoch=42, iter=15300/80000, loss=0.1499, lr=0.008261, batch_cost=1.0265, reader_cost=0.0121 | ETA 18:26:56 2020-11-02 23:36:01 [INFO] [TRAIN] epoch=42, iter=15400/80000, loss=0.1583, lr=0.008250, batch_cost=0.9965, reader_cost=0.0002 | ETA 17:52:52 2020-11-02 23:37:42 [INFO] [TRAIN] epoch=42, iter=15500/80000, loss=0.1404, lr=0.008238, batch_cost=1.0142, reader_cost=0.0002 | ETA 18:10:18 2020-11-02 23:39:22 [INFO] [TRAIN] epoch=42, iter=15600/80000, loss=0.1580, lr=0.008227, batch_cost=0.9970, reader_cost=0.0002 | ETA 17:50:09 2020-11-02 23:41:03 [INFO] [TRAIN] epoch=43, iter=15700/80000, loss=0.1467, lr=0.008215, batch_cost=1.0093, reader_cost=0.0123 | ETA 18:01:37 2020-11-02 23:42:43 [INFO] [TRAIN] epoch=43, iter=15800/80000, loss=0.1442, lr=0.008204, batch_cost=1.0030, reader_cost=0.0002 | ETA 17:53:12 2020-11-02 23:44:23 [INFO] [TRAIN] epoch=43, iter=15900/80000, loss=0.1410, lr=0.008192, batch_cost=0.9989, reader_cost=0.0003 | ETA 17:47:08 2020-11-02 23:46:05 [INFO] [TRAIN] epoch=44, iter=16000/80000, loss=0.1535, lr=0.008181, batch_cost=1.0170, reader_cost=0.0148 | ETA 18:04:50 2020-11-02 23:46:09 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-02 23:50:49 [INFO] [EVAL] #Images=500 mIoU=0.7470 Acc=0.9547 Kappa=0.9412 2020-11-02 23:50:49 [INFO] [EVAL] Category IoU: [0.9773 0.8198 0.9156 0.5368 0.5694 0.6102 0.668 0.7633 0.9187 0.6167 0.9418 0.8081 0.5867 0.9398 0.6087 0.7629 0.7798 0.6138 0.7564] 2020-11-02 23:50:49 [INFO] [EVAL] Category Acc: [0.9879 0.9032 0.9564 0.8406 0.6739 0.7551 0.7829 0.8754 0.952 0.8287 0.9737 0.8748 0.8039 0.9744 0.6588 0.97 0.9371 0.8414 0.8301] 2020-11-02 23:50:51 [INFO] [EVAL] The model with the best validation mIoU (0.7470) was saved at iter 16000. 2020-11-02 23:52:31 [INFO] [TRAIN] epoch=44, iter=16100/80000, loss=0.1296, lr=0.008169, batch_cost=1.0064, reader_cost=0.0064 | ETA 17:51:45 2020-11-02 23:54:13 [INFO] [TRAIN] epoch=44, iter=16200/80000, loss=0.1354, lr=0.008158, batch_cost=1.0192, reader_cost=0.0002 | ETA 18:03:43 2020-11-02 23:55:54 [INFO] [TRAIN] epoch=44, iter=16300/80000, loss=0.1352, lr=0.008146, batch_cost=1.0123, reader_cost=0.0025 | ETA 17:54:40 2020-11-02 23:57:36 [INFO] [TRAIN] epoch=45, iter=16400/80000, loss=0.1722, lr=0.008135, batch_cost=1.0113, reader_cost=0.0112 | ETA 17:51:57 2020-11-02 23:59:16 [INFO] [TRAIN] epoch=45, iter=16500/80000, loss=0.1548, lr=0.008123, batch_cost=1.0019, reader_cost=0.0002 | ETA 17:40:18 2020-11-03 00:00:59 [INFO] [TRAIN] epoch=45, iter=16600/80000, loss=0.1380, lr=0.008112, batch_cost=1.0293, reader_cost=0.0002 | ETA 18:07:37 2020-11-03 00:02:39 [INFO] [TRAIN] epoch=45, iter=16700/80000, loss=0.1337, lr=0.008100, batch_cost=1.0074, reader_cost=0.0002 | ETA 17:42:48 2020-11-03 00:04:20 [INFO] [TRAIN] epoch=46, iter=16800/80000, loss=0.1361, lr=0.008089, batch_cost=1.0054, reader_cost=0.0113 | ETA 17:39:01 2020-11-03 00:06:00 [INFO] [TRAIN] epoch=46, iter=16900/80000, loss=0.1431, lr=0.008077, batch_cost=1.0005, reader_cost=0.0002 | ETA 17:32:13 2020-11-03 00:07:41 [INFO] [TRAIN] epoch=46, iter=17000/80000, loss=0.1302, lr=0.008066, batch_cost=1.0048, reader_cost=0.0002 | ETA 17:34:59 2020-11-03 00:09:24 [INFO] [TRAIN] epoch=46, iter=17100/80000, loss=0.1463, lr=0.008054, batch_cost=1.0335, reader_cost=0.0002 | ETA 18:03:27 2020-11-03 00:11:06 [INFO] [TRAIN] epoch=47, iter=17200/80000, loss=0.1516, lr=0.008042, batch_cost=1.0226, reader_cost=0.0110 | ETA 17:50:20 2020-11-03 00:12:48 [INFO] [TRAIN] epoch=47, iter=17300/80000, loss=0.1581, lr=0.008031, batch_cost=1.0179, reader_cost=0.0005 | ETA 17:43:39 2020-11-03 00:14:29 [INFO] [TRAIN] epoch=47, iter=17400/80000, loss=0.1244, lr=0.008019, batch_cost=1.0131, reader_cost=0.0023 | ETA 17:36:58 2020-11-03 00:16:11 [INFO] [TRAIN] epoch=48, iter=17500/80000, loss=0.1692, lr=0.008008, batch_cost=1.0137, reader_cost=0.0146 | ETA 17:35:57 2020-11-03 00:17:50 [INFO] [TRAIN] epoch=48, iter=17600/80000, loss=0.1330, lr=0.007996, batch_cost=0.9920, reader_cost=0.0003 | ETA 17:11:39 2020-11-03 00:19:29 [INFO] [TRAIN] epoch=48, iter=17700/80000, loss=0.1516, lr=0.007985, batch_cost=0.9958, reader_cost=0.0002 | ETA 17:13:55 2020-11-03 00:21:09 [INFO] [TRAIN] epoch=48, iter=17800/80000, loss=0.1359, lr=0.007973, batch_cost=0.9965, reader_cost=0.0002 | ETA 17:12:59 2020-11-03 00:22:53 [INFO] [TRAIN] epoch=49, iter=17900/80000, loss=0.1274, lr=0.007962, batch_cost=1.0357, reader_cost=0.0137 | ETA 17:51:59 2020-11-03 00:24:31 [INFO] [TRAIN] epoch=49, iter=18000/80000, loss=0.1835, lr=0.007950, batch_cost=0.9883, reader_cost=0.0002 | ETA 17:01:16 2020-11-03 00:26:10 [INFO] [TRAIN] epoch=49, iter=18100/80000, loss=0.1298, lr=0.007939, batch_cost=0.9894, reader_cost=0.0002 | ETA 17:00:44 2020-11-03 00:27:54 [INFO] [TRAIN] epoch=49, iter=18200/80000, loss=0.1690, lr=0.007927, batch_cost=1.0349, reader_cost=0.0002 | ETA 17:45:57 2020-11-03 00:29:37 [INFO] [TRAIN] epoch=50, iter=18300/80000, loss=0.1230, lr=0.007916, batch_cost=1.0267, reader_cost=0.0130 | ETA 17:35:49 2020-11-03 00:31:17 [INFO] [TRAIN] epoch=50, iter=18400/80000, loss=0.1402, lr=0.007904, batch_cost=0.9999, reader_cost=0.0002 | ETA 17:06:34 2020-11-03 00:32:56 [INFO] [TRAIN] epoch=50, iter=18500/80000, loss=0.1314, lr=0.007892, batch_cost=0.9978, reader_cost=0.0002 | ETA 17:02:47 2020-11-03 00:34:36 [INFO] [TRAIN] epoch=50, iter=18600/80000, loss=0.1581, lr=0.007881, batch_cost=0.9982, reader_cost=0.0002 | ETA 17:01:29 2020-11-03 00:36:17 [INFO] [TRAIN] epoch=51, iter=18700/80000, loss=0.1355, lr=0.007869, batch_cost=1.0076, reader_cost=0.0119 | ETA 17:09:27 2020-11-03 00:37:57 [INFO] [TRAIN] epoch=51, iter=18800/80000, loss=0.1363, lr=0.007858, batch_cost=0.9996, reader_cost=0.0002 | ETA 16:59:37 2020-11-03 00:39:40 [INFO] [TRAIN] epoch=51, iter=18900/80000, loss=0.1248, lr=0.007846, batch_cost=1.0299, reader_cost=0.0002 | ETA 17:28:48 2020-11-03 00:41:21 [INFO] [TRAIN] epoch=52, iter=19000/80000, loss=0.1343, lr=0.007835, batch_cost=1.0157, reader_cost=0.0118 | ETA 17:12:37 2020-11-03 00:43:02 [INFO] [TRAIN] epoch=52, iter=19100/80000, loss=0.1366, lr=0.007823, batch_cost=1.0010, reader_cost=0.0002 | ETA 16:56:02 2020-11-03 00:44:42 [INFO] [TRAIN] epoch=52, iter=19200/80000, loss=0.1301, lr=0.007812, batch_cost=1.0029, reader_cost=0.0002 | ETA 16:56:14 2020-11-03 00:46:21 [INFO] [TRAIN] epoch=52, iter=19300/80000, loss=0.1284, lr=0.007800, batch_cost=0.9934, reader_cost=0.0002 | ETA 16:45:00 2020-11-03 00:48:06 [INFO] [TRAIN] epoch=53, iter=19400/80000, loss=0.1344, lr=0.007788, batch_cost=1.0439, reader_cost=0.0149 | ETA 17:34:18 2020-11-03 00:49:45 [INFO] [TRAIN] epoch=53, iter=19500/80000, loss=0.1344, lr=0.007777, batch_cost=0.9911, reader_cost=0.0008 | ETA 16:39:22 2020-11-03 00:51:24 [INFO] [TRAIN] epoch=53, iter=19600/80000, loss=0.1359, lr=0.007765, batch_cost=0.9908, reader_cost=0.0001 | ETA 16:37:25 2020-11-03 00:53:06 [INFO] [TRAIN] epoch=53, iter=19700/80000, loss=0.1491, lr=0.007754, batch_cost=1.0181, reader_cost=0.0002 | ETA 17:03:12 2020-11-03 00:54:48 [INFO] [TRAIN] epoch=54, iter=19800/80000, loss=0.1259, lr=0.007742, batch_cost=1.0242, reader_cost=0.0134 | ETA 17:07:37 2020-11-03 00:56:28 [INFO] [TRAIN] epoch=54, iter=19900/80000, loss=0.1376, lr=0.007731, batch_cost=0.9946, reader_cost=0.0011 | ETA 16:36:16 2020-11-03 00:58:10 [INFO] [TRAIN] epoch=54, iter=20000/80000, loss=0.1257, lr=0.007719, batch_cost=1.0273, reader_cost=0.0009 | ETA 17:07:19 2020-11-03 00:59:52 [INFO] [TRAIN] epoch=55, iter=20100/80000, loss=0.1508, lr=0.007707, batch_cost=1.0160, reader_cost=0.0127 | ETA 16:54:20 2020-11-03 01:01:35 [INFO] [TRAIN] epoch=55, iter=20200/80000, loss=0.1411, lr=0.007696, batch_cost=1.0291, reader_cost=0.0009 | ETA 17:05:39 2020-11-03 01:03:19 [INFO] [TRAIN] epoch=55, iter=20300/80000, loss=0.1821, lr=0.007684, batch_cost=1.0467, reader_cost=0.0004 | ETA 17:21:27 2020-11-03 01:05:02 [INFO] [TRAIN] epoch=55, iter=20400/80000, loss=0.1504, lr=0.007673, batch_cost=1.0264, reader_cost=0.0003 | ETA 16:59:36 2020-11-03 01:06:44 [INFO] [TRAIN] epoch=56, iter=20500/80000, loss=0.1572, lr=0.007661, batch_cost=1.0231, reader_cost=0.0127 | ETA 16:54:34 2020-11-03 01:08:26 [INFO] [TRAIN] epoch=56, iter=20600/80000, loss=0.1642, lr=0.007650, batch_cost=1.0145, reader_cost=0.0003 | ETA 16:44:23 2020-11-03 01:10:07 [INFO] [TRAIN] epoch=56, iter=20700/80000, loss=0.1560, lr=0.007638, batch_cost=1.0135, reader_cost=0.0003 | ETA 16:41:43 2020-11-03 01:11:47 [INFO] [TRAIN] epoch=56, iter=20800/80000, loss=0.1749, lr=0.007626, batch_cost=0.9984, reader_cost=0.0002 | ETA 16:25:06 2020-11-03 01:13:28 [INFO] [TRAIN] epoch=57, iter=20900/80000, loss=0.1455, lr=0.007615, batch_cost=1.0052, reader_cost=0.0114 | ETA 16:30:07 2020-11-03 01:15:08 [INFO] [TRAIN] epoch=57, iter=21000/80000, loss=0.1450, lr=0.007603, batch_cost=1.0064, reader_cost=0.0002 | ETA 16:29:37 2020-11-03 01:16:47 [INFO] [TRAIN] epoch=57, iter=21100/80000, loss=0.1157, lr=0.007592, batch_cost=0.9891, reader_cost=0.0002 | ETA 16:10:56 2020-11-03 01:18:28 [INFO] [TRAIN] epoch=57, iter=21200/80000, loss=0.1395, lr=0.007580, batch_cost=1.0113, reader_cost=0.0002 | ETA 16:31:06 2020-11-03 01:20:11 [INFO] [TRAIN] epoch=58, iter=21300/80000, loss=0.1352, lr=0.007568, batch_cost=1.0238, reader_cost=0.0129 | ETA 16:41:39 2020-11-03 01:21:53 [INFO] [TRAIN] epoch=58, iter=21400/80000, loss=0.1338, lr=0.007557, batch_cost=1.0245, reader_cost=0.0002 | ETA 16:40:38 2020-11-03 01:23:31 [INFO] [TRAIN] epoch=58, iter=21500/80000, loss=0.1261, lr=0.007545, batch_cost=0.9833, reader_cost=0.0002 | ETA 15:58:43 2020-11-03 01:25:13 [INFO] [TRAIN] epoch=59, iter=21600/80000, loss=0.1502, lr=0.007534, batch_cost=1.0155, reader_cost=0.0119 | ETA 16:28:26 2020-11-03 01:26:54 [INFO] [TRAIN] epoch=59, iter=21700/80000, loss=0.1499, lr=0.007522, batch_cost=1.0055, reader_cost=0.0002 | ETA 16:17:00 2020-11-03 01:28:35 [INFO] [TRAIN] epoch=59, iter=21800/80000, loss=0.1643, lr=0.007510, batch_cost=1.0116, reader_cost=0.0002 | ETA 16:21:14 2020-11-03 01:30:18 [INFO] [TRAIN] epoch=59, iter=21900/80000, loss=0.1424, lr=0.007499, batch_cost=1.0281, reader_cost=0.0002 | ETA 16:35:34 2020-11-03 01:32:06 [INFO] [TRAIN] epoch=60, iter=22000/80000, loss=0.1564, lr=0.007487, batch_cost=1.0838, reader_cost=0.0159 | ETA 17:27:39 2020-11-03 01:33:45 [INFO] [TRAIN] epoch=60, iter=22100/80000, loss=0.1499, lr=0.007475, batch_cost=0.9959, reader_cost=0.0002 | ETA 16:01:04 2020-11-03 01:35:26 [INFO] [TRAIN] epoch=60, iter=22200/80000, loss=0.1430, lr=0.007464, batch_cost=1.0005, reader_cost=0.0002 | ETA 16:03:48 2020-11-03 01:37:07 [INFO] [TRAIN] epoch=60, iter=22300/80000, loss=0.1438, lr=0.007452, batch_cost=1.0179, reader_cost=0.0002 | ETA 16:18:54 2020-11-03 01:38:51 [INFO] [TRAIN] epoch=61, iter=22400/80000, loss=0.1151, lr=0.007441, batch_cost=1.0351, reader_cost=0.0117 | ETA 16:33:42 2020-11-03 01:40:31 [INFO] [TRAIN] epoch=61, iter=22500/80000, loss=0.1550, lr=0.007429, batch_cost=1.0052, reader_cost=0.0002 | ETA 16:03:17 2020-11-03 01:42:11 [INFO] [TRAIN] epoch=61, iter=22600/80000, loss=0.1128, lr=0.007417, batch_cost=0.9960, reader_cost=0.0002 | ETA 15:52:48 2020-11-03 01:43:56 [INFO] [TRAIN] epoch=62, iter=22700/80000, loss=0.1506, lr=0.007406, batch_cost=1.0511, reader_cost=0.0190 | ETA 16:43:47 2020-11-03 01:45:41 [INFO] [TRAIN] epoch=62, iter=22800/80000, loss=0.1406, lr=0.007394, batch_cost=1.0453, reader_cost=0.0002 | ETA 16:36:30 2020-11-03 01:47:21 [INFO] [TRAIN] epoch=62, iter=22900/80000, loss=0.1165, lr=0.007382, batch_cost=0.9996, reader_cost=0.0002 | ETA 15:51:19 2020-11-03 01:49:01 [INFO] [TRAIN] epoch=62, iter=23000/80000, loss=0.1295, lr=0.007371, batch_cost=0.9995, reader_cost=0.0002 | ETA 15:49:32 2020-11-03 01:50:41 [INFO] [TRAIN] epoch=63, iter=23100/80000, loss=0.1268, lr=0.007359, batch_cost=1.0048, reader_cost=0.0122 | ETA 15:52:52 2020-11-03 01:52:21 [INFO] [TRAIN] epoch=63, iter=23200/80000, loss=0.1774, lr=0.007347, batch_cost=0.9977, reader_cost=0.0002 | ETA 15:44:30 2020-11-03 01:53:59 [INFO] [TRAIN] epoch=63, iter=23300/80000, loss=0.1340, lr=0.007336, batch_cost=0.9807, reader_cost=0.0002 | ETA 15:26:44 2020-11-03 01:55:38 [INFO] [TRAIN] epoch=63, iter=23400/80000, loss=0.1649, lr=0.007324, batch_cost=0.9916, reader_cost=0.0002 | ETA 15:35:24 2020-11-03 01:57:18 [INFO] [TRAIN] epoch=64, iter=23500/80000, loss=0.1181, lr=0.007313, batch_cost=1.0045, reader_cost=0.0127 | ETA 15:45:52 2020-11-03 01:58:58 [INFO] [TRAIN] epoch=64, iter=23600/80000, loss=0.1437, lr=0.007301, batch_cost=0.9928, reader_cost=0.0002 | ETA 15:33:16 2020-11-03 02:00:38 [INFO] [TRAIN] epoch=64, iter=23700/80000, loss=0.1271, lr=0.007289, batch_cost=0.9993, reader_cost=0.0002 | ETA 15:37:42 2020-11-03 02:02:17 [INFO] [TRAIN] epoch=64, iter=23800/80000, loss=0.1391, lr=0.007278, batch_cost=0.9952, reader_cost=0.0002 | ETA 15:32:10 2020-11-03 02:04:00 [INFO] [TRAIN] epoch=65, iter=23900/80000, loss=0.1368, lr=0.007266, batch_cost=1.0235, reader_cost=0.0126 | ETA 15:56:59 2020-11-03 02:05:42 [INFO] [TRAIN] epoch=65, iter=24000/80000, loss=0.1257, lr=0.007254, batch_cost=1.0258, reader_cost=0.0003 | ETA 15:57:27 2020-11-03 02:05:46 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-03 02:10:30 [INFO] [EVAL] #Images=500 mIoU=0.7705 Acc=0.9585 Kappa=0.9462 2020-11-03 02:10:30 [INFO] [EVAL] Category IoU: [0.9805 0.8432 0.9212 0.4995 0.6016 0.628 0.6916 0.7673 0.919 0.6459 0.9414 0.8133 0.6223 0.9493 0.7696 0.8803 0.7427 0.6566 0.7662] 2020-11-03 02:10:30 [INFO] [EVAL] Category Acc: [0.9911 0.9144 0.9575 0.8391 0.7877 0.7619 0.8136 0.8939 0.9493 0.836 0.963 0.8771 0.7241 0.9737 0.9346 0.9473 0.8124 0.8177 0.8705] 2020-11-03 02:10:32 [INFO] [EVAL] The model with the best validation mIoU (0.7705) was saved at iter 24000. 2020-11-03 02:12:10 [INFO] [TRAIN] epoch=65, iter=24100/80000, loss=0.1248, lr=0.007243, batch_cost=0.9826, reader_cost=0.0002 | ETA 15:15:25 2020-11-03 02:13:52 [INFO] [TRAIN] epoch=66, iter=24200/80000, loss=0.1297, lr=0.007231, batch_cost=1.0158, reader_cost=0.0133 | ETA 15:44:42 2020-11-03 02:15:31 [INFO] [TRAIN] epoch=66, iter=24300/80000, loss=0.1388, lr=0.007219, batch_cost=0.9893, reader_cost=0.0002 | ETA 15:18:22 2020-11-03 02:17:12 [INFO] [TRAIN] epoch=66, iter=24400/80000, loss=0.1481, lr=0.007208, batch_cost=1.0124, reader_cost=0.0002 | ETA 15:38:10 2020-11-03 02:18:52 [INFO] [TRAIN] epoch=66, iter=24500/80000, loss=0.1228, lr=0.007196, batch_cost=1.0010, reader_cost=0.0010 | ETA 15:25:56 2020-11-03 02:20:39 [INFO] [TRAIN] epoch=67, iter=24600/80000, loss=0.1443, lr=0.007184, batch_cost=1.0705, reader_cost=0.0140 | ETA 16:28:27 2020-11-03 02:22:19 [INFO] [TRAIN] epoch=67, iter=24700/80000, loss=0.1534, lr=0.007173, batch_cost=0.9989, reader_cost=0.0002 | ETA 15:20:40 2020-11-03 02:23:58 [INFO] [TRAIN] epoch=67, iter=24800/80000, loss=0.1199, lr=0.007161, batch_cost=0.9902, reader_cost=0.0002 | ETA 15:11:00 2020-11-03 02:25:37 [INFO] [TRAIN] epoch=67, iter=24900/80000, loss=0.1401, lr=0.007149, batch_cost=0.9866, reader_cost=0.0002 | ETA 15:05:59 2020-11-03 02:27:18 [INFO] [TRAIN] epoch=68, iter=25000/80000, loss=0.1467, lr=0.007138, batch_cost=1.0109, reader_cost=0.0135 | ETA 15:26:37 2020-11-03 02:28:59 [INFO] [TRAIN] epoch=68, iter=25100/80000, loss=0.1439, lr=0.007126, batch_cost=1.0089, reader_cost=0.0002 | ETA 15:23:05 2020-11-03 02:30:40 [INFO] [TRAIN] epoch=68, iter=25200/80000, loss=0.1408, lr=0.007114, batch_cost=1.0161, reader_cost=0.0002 | ETA 15:28:00 2020-11-03 02:32:22 [INFO] [TRAIN] epoch=69, iter=25300/80000, loss=0.1531, lr=0.007103, batch_cost=1.0215, reader_cost=0.0127 | ETA 15:31:14 2020-11-03 02:34:02 [INFO] [TRAIN] epoch=69, iter=25400/80000, loss=0.1283, lr=0.007091, batch_cost=0.9975, reader_cost=0.0002 | ETA 15:07:45 2020-11-03 02:35:41 [INFO] [TRAIN] epoch=69, iter=25500/80000, loss=0.1306, lr=0.007079, batch_cost=0.9876, reader_cost=0.0002 | ETA 14:57:05 2020-11-03 02:37:19 [INFO] [TRAIN] epoch=69, iter=25600/80000, loss=0.1220, lr=0.007067, batch_cost=0.9846, reader_cost=0.0002 | ETA 14:52:42 2020-11-03 02:39:01 [INFO] [TRAIN] epoch=70, iter=25700/80000, loss=0.1588, lr=0.007056, batch_cost=1.0109, reader_cost=0.0114 | ETA 15:14:50 2020-11-03 02:40:41 [INFO] [TRAIN] epoch=70, iter=25800/80000, loss=0.1332, lr=0.007044, batch_cost=1.0012, reader_cost=0.0015 | ETA 15:04:23 2020-11-03 02:42:20 [INFO] [TRAIN] epoch=70, iter=25900/80000, loss=0.1321, lr=0.007032, batch_cost=0.9974, reader_cost=0.0004 | ETA 14:59:20 2020-11-03 02:44:02 [INFO] [TRAIN] epoch=70, iter=26000/80000, loss=0.1340, lr=0.007021, batch_cost=1.0149, reader_cost=0.0002 | ETA 15:13:22 2020-11-03 02:45:42 [INFO] [TRAIN] epoch=71, iter=26100/80000, loss=0.1213, lr=0.007009, batch_cost=1.0004, reader_cost=0.0121 | ETA 14:58:43 2020-11-03 02:47:21 [INFO] [TRAIN] epoch=71, iter=26200/80000, loss=0.1430, lr=0.006997, batch_cost=0.9934, reader_cost=0.0002 | ETA 14:50:44 2020-11-03 02:49:03 [INFO] [TRAIN] epoch=71, iter=26300/80000, loss=0.1155, lr=0.006986, batch_cost=1.0157, reader_cost=0.0002 | ETA 15:09:05 2020-11-03 02:50:43 [INFO] [TRAIN] epoch=71, iter=26400/80000, loss=0.1433, lr=0.006974, batch_cost=0.9994, reader_cost=0.0002 | ETA 14:52:49 2020-11-03 02:52:25 [INFO] [TRAIN] epoch=72, iter=26500/80000, loss=0.1394, lr=0.006962, batch_cost=1.0222, reader_cost=0.0116 | ETA 15:11:29 2020-11-03 02:54:06 [INFO] [TRAIN] epoch=72, iter=26600/80000, loss=0.1439, lr=0.006950, batch_cost=1.0063, reader_cost=0.0004 | ETA 14:55:36 2020-11-03 02:55:45 [INFO] [TRAIN] epoch=72, iter=26700/80000, loss=0.1247, lr=0.006939, batch_cost=0.9961, reader_cost=0.0002 | ETA 14:44:49 2020-11-03 02:57:28 [INFO] [TRAIN] epoch=73, iter=26800/80000, loss=0.1384, lr=0.006927, batch_cost=1.0233, reader_cost=0.0149 | ETA 15:07:19 2020-11-03 02:59:07 [INFO] [TRAIN] epoch=73, iter=26900/80000, loss=0.1224, lr=0.006915, batch_cost=0.9893, reader_cost=0.0002 | ETA 14:35:33 2020-11-03 03:00:45 [INFO] [TRAIN] epoch=73, iter=27000/80000, loss=0.1587, lr=0.006904, batch_cost=0.9858, reader_cost=0.0003 | ETA 14:30:48 2020-11-03 03:02:24 [INFO] [TRAIN] epoch=73, iter=27100/80000, loss=0.1413, lr=0.006892, batch_cost=0.9936, reader_cost=0.0002 | ETA 14:36:02 2020-11-03 03:04:07 [INFO] [TRAIN] epoch=74, iter=27200/80000, loss=0.1189, lr=0.006880, batch_cost=1.0270, reader_cost=0.0119 | ETA 15:03:47 2020-11-03 03:05:47 [INFO] [TRAIN] epoch=74, iter=27300/80000, loss=0.1427, lr=0.006868, batch_cost=0.9969, reader_cost=0.0002 | ETA 14:35:35 2020-11-03 03:07:27 [INFO] [TRAIN] epoch=74, iter=27400/80000, loss=0.1303, lr=0.006857, batch_cost=1.0017, reader_cost=0.0002 | ETA 14:38:10 2020-11-03 03:09:07 [INFO] [TRAIN] epoch=74, iter=27500/80000, loss=0.1493, lr=0.006845, batch_cost=0.9956, reader_cost=0.0002 | ETA 14:31:07 2020-11-03 03:10:47 [INFO] [TRAIN] epoch=75, iter=27600/80000, loss=0.1343, lr=0.006833, batch_cost=1.0053, reader_cost=0.0124 | ETA 14:37:59 2020-11-03 03:12:27 [INFO] [TRAIN] epoch=75, iter=27700/80000, loss=0.1550, lr=0.006821, batch_cost=0.9970, reader_cost=0.0002 | ETA 14:29:04 2020-11-03 03:14:06 [INFO] [TRAIN] epoch=75, iter=27800/80000, loss=0.1168, lr=0.006810, batch_cost=0.9916, reader_cost=0.0002 | ETA 14:22:42 2020-11-03 03:15:48 [INFO] [TRAIN] epoch=75, iter=27900/80000, loss=0.1489, lr=0.006798, batch_cost=1.0158, reader_cost=0.0002 | ETA 14:42:02 2020-11-03 03:17:28 [INFO] [TRAIN] epoch=76, iter=28000/80000, loss=0.1332, lr=0.006786, batch_cost=1.0039, reader_cost=0.0114 | ETA 14:30:01 2020-11-03 03:19:08 [INFO] [TRAIN] epoch=76, iter=28100/80000, loss=0.1284, lr=0.006774, batch_cost=0.9984, reader_cost=0.0002 | ETA 14:23:37 2020-11-03 03:20:50 [INFO] [TRAIN] epoch=76, iter=28200/80000, loss=0.1174, lr=0.006763, batch_cost=1.0228, reader_cost=0.0002 | ETA 14:42:58 2020-11-03 03:22:32 [INFO] [TRAIN] epoch=77, iter=28300/80000, loss=0.1555, lr=0.006751, batch_cost=1.0194, reader_cost=0.0150 | ETA 14:38:21 2020-11-03 03:24:12 [INFO] [TRAIN] epoch=77, iter=28400/80000, loss=0.1361, lr=0.006739, batch_cost=1.0047, reader_cost=0.0002 | ETA 14:24:04 2020-11-03 03:25:53 [INFO] [TRAIN] epoch=77, iter=28500/80000, loss=0.1310, lr=0.006727, batch_cost=1.0050, reader_cost=0.0003 | ETA 14:22:36 2020-11-03 03:27:33 [INFO] [TRAIN] epoch=77, iter=28600/80000, loss=0.1207, lr=0.006716, batch_cost=1.0009, reader_cost=0.0002 | ETA 14:17:27 2020-11-03 03:29:17 [INFO] [TRAIN] epoch=78, iter=28700/80000, loss=0.1260, lr=0.006704, batch_cost=1.0370, reader_cost=0.0122 | ETA 14:46:37 2020-11-03 03:30:57 [INFO] [TRAIN] epoch=78, iter=28800/80000, loss=0.1418, lr=0.006692, batch_cost=1.0043, reader_cost=0.0002 | ETA 14:17:01 2020-11-03 03:32:37 [INFO] [TRAIN] epoch=78, iter=28900/80000, loss=0.1199, lr=0.006680, batch_cost=0.9937, reader_cost=0.0002 | ETA 14:06:17 2020-11-03 03:34:19 [INFO] [TRAIN] epoch=78, iter=29000/80000, loss=0.1383, lr=0.006669, batch_cost=1.0280, reader_cost=0.0002 | ETA 14:33:47 2020-11-03 03:36:01 [INFO] [TRAIN] epoch=79, iter=29100/80000, loss=0.1277, lr=0.006657, batch_cost=1.0162, reader_cost=0.0118 | ETA 14:22:06 2020-11-03 03:37:42 [INFO] [TRAIN] epoch=79, iter=29200/80000, loss=0.1483, lr=0.006645, batch_cost=1.0063, reader_cost=0.0002 | ETA 14:11:59 2020-11-03 03:39:21 [INFO] [TRAIN] epoch=79, iter=29300/80000, loss=0.1252, lr=0.006633, batch_cost=0.9972, reader_cost=0.0002 | ETA 14:02:36 2020-11-03 03:41:03 [INFO] [TRAIN] epoch=80, iter=29400/80000, loss=0.1394, lr=0.006622, batch_cost=1.0138, reader_cost=0.0131 | ETA 14:14:56 2020-11-03 03:42:44 [INFO] [TRAIN] epoch=80, iter=29500/80000, loss=0.1395, lr=0.006610, batch_cost=1.0166, reader_cost=0.0002 | ETA 14:15:37 2020-11-03 03:44:24 [INFO] [TRAIN] epoch=80, iter=29600/80000, loss=0.1257, lr=0.006598, batch_cost=0.9930, reader_cost=0.0002 | ETA 13:54:09 2020-11-03 03:46:02 [INFO] [TRAIN] epoch=80, iter=29700/80000, loss=0.1265, lr=0.006586, batch_cost=0.9855, reader_cost=0.0001 | ETA 13:46:10 2020-11-03 03:47:43 [INFO] [TRAIN] epoch=81, iter=29800/80000, loss=0.1484, lr=0.006574, batch_cost=1.0108, reader_cost=0.0178 | ETA 14:05:43 2020-11-03 03:49:22 [INFO] [TRAIN] epoch=81, iter=29900/80000, loss=0.1410, lr=0.006563, batch_cost=0.9869, reader_cost=0.0002 | ETA 13:44:04 2020-11-03 03:51:04 [INFO] [TRAIN] epoch=81, iter=30000/80000, loss=0.1222, lr=0.006551, batch_cost=1.0186, reader_cost=0.0002 | ETA 14:08:49 2020-11-03 03:52:43 [INFO] [TRAIN] epoch=81, iter=30100/80000, loss=0.1290, lr=0.006539, batch_cost=0.9956, reader_cost=0.0002 | ETA 13:48:02 2020-11-03 03:54:24 [INFO] [TRAIN] epoch=82, iter=30200/80000, loss=0.1377, lr=0.006527, batch_cost=1.0051, reader_cost=0.0118 | ETA 13:54:13 2020-11-03 03:56:03 [INFO] [TRAIN] epoch=82, iter=30300/80000, loss=0.1453, lr=0.006515, batch_cost=0.9930, reader_cost=0.0002 | ETA 13:42:31 2020-11-03 03:57:52 [INFO] [TRAIN] epoch=82, iter=30400/80000, loss=0.1327, lr=0.006504, batch_cost=1.0861, reader_cost=0.0005 | ETA 14:57:51 2020-11-03 03:59:38 [INFO] [TRAIN] epoch=82, iter=30500/80000, loss=0.1367, lr=0.006492, batch_cost=1.0625, reader_cost=0.0005 | ETA 14:36:34 2020-11-03 04:01:21 [INFO] [TRAIN] epoch=83, iter=30600/80000, loss=0.1285, lr=0.006480, batch_cost=1.0313, reader_cost=0.0133 | ETA 14:09:06 2020-11-03 04:03:01 [INFO] [TRAIN] epoch=83, iter=30700/80000, loss=0.1576, lr=0.006468, batch_cost=0.9936, reader_cost=0.0002 | ETA 13:36:23 2020-11-03 04:04:39 [INFO] [TRAIN] epoch=83, iter=30800/80000, loss=0.1369, lr=0.006456, batch_cost=0.9855, reader_cost=0.0002 | ETA 13:28:06 2020-11-03 04:06:19 [INFO] [TRAIN] epoch=84, iter=30900/80000, loss=0.1415, lr=0.006445, batch_cost=1.0007, reader_cost=0.0120 | ETA 13:38:54 2020-11-03 04:08:02 [INFO] [TRAIN] epoch=84, iter=31000/80000, loss=0.1588, lr=0.006433, batch_cost=1.0321, reader_cost=0.0002 | ETA 14:02:54 2020-11-03 04:09:42 [INFO] [TRAIN] epoch=84, iter=31100/80000, loss=0.1352, lr=0.006421, batch_cost=0.9922, reader_cost=0.0002 | ETA 13:28:39 2020-11-03 04:11:20 [INFO] [TRAIN] epoch=84, iter=31200/80000, loss=0.1351, lr=0.006409, batch_cost=0.9842, reader_cost=0.0002 | ETA 13:20:30 2020-11-03 04:13:01 [INFO] [TRAIN] epoch=85, iter=31300/80000, loss=0.1384, lr=0.006397, batch_cost=1.0047, reader_cost=0.0114 | ETA 13:35:29 2020-11-03 04:14:45 [INFO] [TRAIN] epoch=85, iter=31400/80000, loss=0.1669, lr=0.006386, batch_cost=1.0443, reader_cost=0.0005 | ETA 14:05:51 2020-11-03 04:16:25 [INFO] [TRAIN] epoch=85, iter=31500/80000, loss=0.1384, lr=0.006374, batch_cost=0.9965, reader_cost=0.0002 | ETA 13:25:30 2020-11-03 04:18:04 [INFO] [TRAIN] epoch=85, iter=31600/80000, loss=0.1460, lr=0.006362, batch_cost=0.9948, reader_cost=0.0002 | ETA 13:22:30 2020-11-03 04:19:47 [INFO] [TRAIN] epoch=86, iter=31700/80000, loss=0.1275, lr=0.006350, batch_cost=1.0293, reader_cost=0.0114 | ETA 13:48:32 2020-11-03 04:21:29 [INFO] [TRAIN] epoch=86, iter=31800/80000, loss=0.1515, lr=0.006338, batch_cost=1.0214, reader_cost=0.0002 | ETA 13:40:32 2020-11-03 04:23:11 [INFO] [TRAIN] epoch=86, iter=31900/80000, loss=0.1093, lr=0.006326, batch_cost=1.0135, reader_cost=0.0002 | ETA 13:32:29 2020-11-03 04:24:52 [INFO] [TRAIN] epoch=87, iter=32000/80000, loss=0.1362, lr=0.006315, batch_cost=1.0157, reader_cost=0.0128 | ETA 13:32:31 2020-11-03 04:24:56 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-03 04:29:25 [INFO] [EVAL] #Images=500 mIoU=0.7723 Acc=0.9589 Kappa=0.9466 2020-11-03 04:29:25 [INFO] [EVAL] Category IoU: [0.98 0.8416 0.9218 0.5063 0.6103 0.6287 0.6944 0.7599 0.9204 0.621 0.945 0.8189 0.6278 0.9497 0.8139 0.8892 0.8272 0.5552 0.7622] 2020-11-03 04:29:25 [INFO] [EVAL] Category Acc: [0.9887 0.9156 0.957 0.8445 0.8625 0.746 0.8193 0.9151 0.9484 0.868 0.9685 0.8846 0.7471 0.9747 0.9157 0.9746 0.9399 0.8927 0.8624] 2020-11-03 04:29:27 [INFO] [EVAL] The model with the best validation mIoU (0.7723) was saved at iter 32000. 2020-11-03 04:31:08 [INFO] [TRAIN] epoch=87, iter=32100/80000, loss=0.1612, lr=0.006303, batch_cost=1.0052, reader_cost=0.0003 | ETA 13:22:31 2020-11-03 04:32:48 [INFO] [TRAIN] epoch=87, iter=32200/80000, loss=0.1176, lr=0.006291, batch_cost=1.0043, reader_cost=0.0002 | ETA 13:20:03 2020-11-03 04:34:29 [INFO] [TRAIN] epoch=87, iter=32300/80000, loss=0.1548, lr=0.006279, batch_cost=1.0114, reader_cost=0.0003 | ETA 13:24:02 2020-11-03 04:36:11 [INFO] [TRAIN] epoch=88, iter=32400/80000, loss=0.1272, lr=0.006267, batch_cost=1.0206, reader_cost=0.0131 | ETA 13:29:41 2020-11-03 04:37:52 [INFO] [TRAIN] epoch=88, iter=32500/80000, loss=0.1522, lr=0.006255, batch_cost=1.0096, reader_cost=0.0002 | ETA 13:19:16 2020-11-03 04:39:35 [INFO] [TRAIN] epoch=88, iter=32600/80000, loss=0.1285, lr=0.006243, batch_cost=1.0241, reader_cost=0.0003 | ETA 13:29:01 2020-11-03 04:41:14 [INFO] [TRAIN] epoch=88, iter=32700/80000, loss=0.1343, lr=0.006232, batch_cost=0.9983, reader_cost=0.0014 | ETA 13:06:58 2020-11-03 04:42:55 [INFO] [TRAIN] epoch=89, iter=32800/80000, loss=0.1163, lr=0.006220, batch_cost=1.0082, reader_cost=0.0144 | ETA 13:13:07 2020-11-03 04:44:37 [INFO] [TRAIN] epoch=89, iter=32900/80000, loss=0.1435, lr=0.006208, batch_cost=1.0202, reader_cost=0.0002 | ETA 13:20:53 2020-11-03 04:46:17 [INFO] [TRAIN] epoch=89, iter=33000/80000, loss=0.1223, lr=0.006196, batch_cost=1.0022, reader_cost=0.0002 | ETA 13:05:01 2020-11-03 04:47:58 [INFO] [TRAIN] epoch=89, iter=33100/80000, loss=0.1342, lr=0.006184, batch_cost=1.0083, reader_cost=0.0002 | ETA 13:08:07 2020-11-03 04:49:41 [INFO] [TRAIN] epoch=90, iter=33200/80000, loss=0.1173, lr=0.006172, batch_cost=1.0240, reader_cost=0.0122 | ETA 13:18:41 2020-11-03 04:51:24 [INFO] [TRAIN] epoch=90, iter=33300/80000, loss=0.1214, lr=0.006160, batch_cost=1.0334, reader_cost=0.0002 | ETA 13:24:20 2020-11-03 04:53:04 [INFO] [TRAIN] epoch=90, iter=33400/80000, loss=0.1149, lr=0.006149, batch_cost=1.0005, reader_cost=0.0002 | ETA 12:57:03 2020-11-03 04:54:45 [INFO] [TRAIN] epoch=91, iter=33500/80000, loss=0.1287, lr=0.006137, batch_cost=1.0120, reader_cost=0.0136 | ETA 13:04:17 2020-11-03 04:56:25 [INFO] [TRAIN] epoch=91, iter=33600/80000, loss=0.1302, lr=0.006125, batch_cost=0.9965, reader_cost=0.0002 | ETA 12:50:36 2020-11-03 04:58:06 [INFO] [TRAIN] epoch=91, iter=33700/80000, loss=0.1242, lr=0.006113, batch_cost=1.0106, reader_cost=0.0002 | ETA 12:59:51 2020-11-03 04:59:47 [INFO] [TRAIN] epoch=91, iter=33800/80000, loss=0.1172, lr=0.006101, batch_cost=1.0091, reader_cost=0.0002 | ETA 12:56:58 2020-11-03 05:01:28 [INFO] [TRAIN] epoch=92, iter=33900/80000, loss=0.1119, lr=0.006089, batch_cost=1.0110, reader_cost=0.0120 | ETA 12:56:46 2020-11-03 05:03:08 [INFO] [TRAIN] epoch=92, iter=34000/80000, loss=0.1175, lr=0.006077, batch_cost=1.0042, reader_cost=0.0011 | ETA 12:49:51 2020-11-03 05:04:48 [INFO] [TRAIN] epoch=92, iter=34100/80000, loss=0.1123, lr=0.006065, batch_cost=0.9922, reader_cost=0.0002 | ETA 12:39:03 2020-11-03 05:06:29 [INFO] [TRAIN] epoch=92, iter=34200/80000, loss=0.1255, lr=0.006053, batch_cost=1.0147, reader_cost=0.0002 | ETA 12:54:34 2020-11-03 05:08:10 [INFO] [TRAIN] epoch=93, iter=34300/80000, loss=0.1123, lr=0.006042, batch_cost=1.0052, reader_cost=0.0120 | ETA 12:45:39 2020-11-03 05:09:49 [INFO] [TRAIN] epoch=93, iter=34400/80000, loss=0.1231, lr=0.006030, batch_cost=0.9974, reader_cost=0.0002 | ETA 12:38:01 2020-11-03 05:11:32 [INFO] [TRAIN] epoch=93, iter=34500/80000, loss=0.1125, lr=0.006018, batch_cost=1.0225, reader_cost=0.0002 | ETA 12:55:22 2020-11-03 05:13:15 [INFO] [TRAIN] epoch=94, iter=34600/80000, loss=0.1459, lr=0.006006, batch_cost=1.0337, reader_cost=0.0113 | ETA 13:02:10 2020-11-03 05:14:57 [INFO] [TRAIN] epoch=94, iter=34700/80000, loss=0.1277, lr=0.005994, batch_cost=1.0245, reader_cost=0.0002 | ETA 12:53:29 2020-11-03 05:16:39 [INFO] [TRAIN] epoch=94, iter=34800/80000, loss=0.1205, lr=0.005982, batch_cost=1.0202, reader_cost=0.0002 | ETA 12:48:31 2020-11-03 05:18:20 [INFO] [TRAIN] epoch=94, iter=34900/80000, loss=0.1188, lr=0.005970, batch_cost=1.0086, reader_cost=0.0002 | ETA 12:38:08 2020-11-03 05:20:02 [INFO] [TRAIN] epoch=95, iter=35000/80000, loss=0.1275, lr=0.005958, batch_cost=1.0199, reader_cost=0.0137 | ETA 12:44:56 2020-11-03 05:21:43 [INFO] [TRAIN] epoch=95, iter=35100/80000, loss=0.1325, lr=0.005946, batch_cost=1.0069, reader_cost=0.0002 | ETA 12:33:30 2020-11-03 05:23:21 [INFO] [TRAIN] epoch=95, iter=35200/80000, loss=0.1326, lr=0.005934, batch_cost=0.9796, reader_cost=0.0002 | ETA 12:11:27 2020-11-03 05:25:00 [INFO] [TRAIN] epoch=95, iter=35300/80000, loss=0.1277, lr=0.005922, batch_cost=0.9886, reader_cost=0.0002 | ETA 12:16:28 2020-11-03 05:26:41 [INFO] [TRAIN] epoch=96, iter=35400/80000, loss=0.1242, lr=0.005911, batch_cost=1.0104, reader_cost=0.0113 | ETA 12:31:04 2020-11-03 05:28:21 [INFO] [TRAIN] epoch=96, iter=35500/80000, loss=0.1307, lr=0.005899, batch_cost=1.0030, reader_cost=0.0002 | ETA 12:23:51 2020-11-03 05:30:02 [INFO] [TRAIN] epoch=96, iter=35600/80000, loss=0.1340, lr=0.005887, batch_cost=1.0042, reader_cost=0.0002 | ETA 12:23:07 2020-11-03 05:31:42 [INFO] [TRAIN] epoch=96, iter=35700/80000, loss=0.1244, lr=0.005875, batch_cost=1.0065, reader_cost=0.0002 | ETA 12:23:06 2020-11-03 05:33:24 [INFO] [TRAIN] epoch=97, iter=35800/80000, loss=0.1147, lr=0.005863, batch_cost=1.0165, reader_cost=0.0157 | ETA 12:28:48 2020-11-03 05:35:04 [INFO] [TRAIN] epoch=97, iter=35900/80000, loss=0.1366, lr=0.005851, batch_cost=1.0018, reader_cost=0.0002 | ETA 12:16:21 2020-11-03 05:36:45 [INFO] [TRAIN] epoch=97, iter=36000/80000, loss=0.1186, lr=0.005839, batch_cost=1.0129, reader_cost=0.0002 | ETA 12:22:48 2020-11-03 05:38:26 [INFO] [TRAIN] epoch=98, iter=36100/80000, loss=0.1291, lr=0.005827, batch_cost=1.0007, reader_cost=0.0113 | ETA 12:12:12 2020-11-03 05:40:04 [INFO] [TRAIN] epoch=98, iter=36200/80000, loss=0.1181, lr=0.005815, batch_cost=0.9871, reader_cost=0.0002 | ETA 12:00:35 2020-11-03 05:41:49 [INFO] [TRAIN] epoch=98, iter=36300/80000, loss=0.1323, lr=0.005803, batch_cost=1.0437, reader_cost=0.0002 | ETA 12:40:11 2020-11-03 05:43:28 [INFO] [TRAIN] epoch=98, iter=36400/80000, loss=0.1236, lr=0.005791, batch_cost=0.9913, reader_cost=0.0002 | ETA 12:00:18 2020-11-03 05:45:09 [INFO] [TRAIN] epoch=99, iter=36500/80000, loss=0.1177, lr=0.005779, batch_cost=1.0156, reader_cost=0.0123 | ETA 12:16:17 2020-11-03 05:46:49 [INFO] [TRAIN] epoch=99, iter=36600/80000, loss=0.1246, lr=0.005767, batch_cost=0.9934, reader_cost=0.0002 | ETA 11:58:34 2020-11-03 05:48:28 [INFO] [TRAIN] epoch=99, iter=36700/80000, loss=0.1058, lr=0.005755, batch_cost=0.9981, reader_cost=0.0002 | ETA 12:00:18 2020-11-03 05:50:09 [INFO] [TRAIN] epoch=99, iter=36800/80000, loss=0.1194, lr=0.005743, batch_cost=1.0007, reader_cost=0.0002 | ETA 12:00:28 2020-11-03 05:51:50 [INFO] [TRAIN] epoch=100, iter=36900/80000, loss=0.1231, lr=0.005731, batch_cost=1.0141, reader_cost=0.0147 | ETA 12:08:28 2020-11-03 05:53:30 [INFO] [TRAIN] epoch=100, iter=37000/80000, loss=0.1268, lr=0.005719, batch_cost=0.9998, reader_cost=0.0002 | ETA 11:56:31 2020-11-03 05:55:10 [INFO] [TRAIN] epoch=100, iter=37100/80000, loss=0.0983, lr=0.005707, batch_cost=1.0034, reader_cost=0.0002 | ETA 11:57:26 2020-11-03 05:56:50 [INFO] [TRAIN] epoch=100, iter=37200/80000, loss=0.1331, lr=0.005695, batch_cost=0.9991, reader_cost=0.0002 | ETA 11:52:43 2020-11-03 05:58:32 [INFO] [TRAIN] epoch=101, iter=37300/80000, loss=0.1306, lr=0.005683, batch_cost=1.0231, reader_cost=0.0121 | ETA 12:08:07 2020-11-03 06:00:13 [INFO] [TRAIN] epoch=101, iter=37400/80000, loss=0.1101, lr=0.005671, batch_cost=1.0037, reader_cost=0.0009 | ETA 11:52:36 2020-11-03 06:01:53 [INFO] [TRAIN] epoch=101, iter=37500/80000, loss=0.1202, lr=0.005660, batch_cost=1.0039, reader_cost=0.0002 | ETA 11:51:06 2020-11-03 06:03:34 [INFO] [TRAIN] epoch=102, iter=37600/80000, loss=0.1548, lr=0.005648, batch_cost=1.0031, reader_cost=0.0129 | ETA 11:48:52 2020-11-03 06:05:12 [INFO] [TRAIN] epoch=102, iter=37700/80000, loss=0.1206, lr=0.005636, batch_cost=0.9836, reader_cost=0.0002 | ETA 11:33:27 2020-11-03 06:06:52 [INFO] [TRAIN] epoch=102, iter=37800/80000, loss=0.1261, lr=0.005624, batch_cost=0.9956, reader_cost=0.0002 | ETA 11:40:13 2020-11-03 06:08:35 [INFO] [TRAIN] epoch=102, iter=37900/80000, loss=0.1282, lr=0.005612, batch_cost=1.0340, reader_cost=0.0002 | ETA 12:05:33 2020-11-03 06:10:16 [INFO] [TRAIN] epoch=103, iter=38000/80000, loss=0.1092, lr=0.005600, batch_cost=1.0090, reader_cost=0.0125 | ETA 11:46:19 2020-11-03 06:11:57 [INFO] [TRAIN] epoch=103, iter=38100/80000, loss=0.1463, lr=0.005588, batch_cost=1.0095, reader_cost=0.0002 | ETA 11:44:57 2020-11-03 06:13:35 [INFO] [TRAIN] epoch=103, iter=38200/80000, loss=0.1069, lr=0.005576, batch_cost=0.9852, reader_cost=0.0002 | ETA 11:26:20 2020-11-03 06:15:15 [INFO] [TRAIN] epoch=103, iter=38300/80000, loss=0.1176, lr=0.005564, batch_cost=0.9934, reader_cost=0.0002 | ETA 11:30:25 2020-11-03 06:16:56 [INFO] [TRAIN] epoch=104, iter=38400/80000, loss=0.1199, lr=0.005552, batch_cost=1.0181, reader_cost=0.0131 | ETA 11:45:53 2020-11-03 06:18:37 [INFO] [TRAIN] epoch=104, iter=38500/80000, loss=0.1113, lr=0.005540, batch_cost=1.0100, reader_cost=0.0003 | ETA 11:38:33 2020-11-03 06:20:18 [INFO] [TRAIN] epoch=104, iter=38600/80000, loss=0.1244, lr=0.005528, batch_cost=1.0009, reader_cost=0.0002 | ETA 11:30:37 2020-11-03 06:22:06 [INFO] [TRAIN] epoch=105, iter=38700/80000, loss=0.1206, lr=0.005515, batch_cost=1.0798, reader_cost=0.0116 | ETA 12:23:17 2020-11-03 06:23:46 [INFO] [TRAIN] epoch=105, iter=38800/80000, loss=0.1149, lr=0.005503, batch_cost=1.0081, reader_cost=0.0002 | ETA 11:32:12 2020-11-03 06:25:29 [INFO] [TRAIN] epoch=105, iter=38900/80000, loss=0.1184, lr=0.005491, batch_cost=1.0291, reader_cost=0.0002 | ETA 11:44:56 2020-11-03 06:27:11 [INFO] [TRAIN] epoch=105, iter=39000/80000, loss=0.1195, lr=0.005479, batch_cost=1.0180, reader_cost=0.0003 | ETA 11:35:39 2020-11-03 06:28:54 [INFO] [TRAIN] epoch=106, iter=39100/80000, loss=0.1171, lr=0.005467, batch_cost=1.0256, reader_cost=0.0113 | ETA 11:39:07 2020-11-03 06:30:36 [INFO] [TRAIN] epoch=106, iter=39200/80000, loss=0.1295, lr=0.005455, batch_cost=1.0262, reader_cost=0.0002 | ETA 11:37:47 2020-11-03 06:32:18 [INFO] [TRAIN] epoch=106, iter=39300/80000, loss=0.1280, lr=0.005443, batch_cost=1.0144, reader_cost=0.0003 | ETA 11:28:05 2020-11-03 06:33:58 [INFO] [TRAIN] epoch=106, iter=39400/80000, loss=0.1108, lr=0.005431, batch_cost=1.0013, reader_cost=0.0002 | ETA 11:17:33 2020-11-03 06:35:39 [INFO] [TRAIN] epoch=107, iter=39500/80000, loss=0.1178, lr=0.005419, batch_cost=1.0078, reader_cost=0.0164 | ETA 11:20:14 2020-11-03 06:37:18 [INFO] [TRAIN] epoch=107, iter=39600/80000, loss=0.1211, lr=0.005407, batch_cost=0.9906, reader_cost=0.0002 | ETA 11:06:58 2020-11-03 06:38:57 [INFO] [TRAIN] epoch=107, iter=39700/80000, loss=0.1134, lr=0.005395, batch_cost=0.9891, reader_cost=0.0002 | ETA 11:04:19 2020-11-03 06:40:36 [INFO] [TRAIN] epoch=107, iter=39800/80000, loss=0.1196, lr=0.005383, batch_cost=0.9925, reader_cost=0.0002 | ETA 11:04:59 2020-11-03 06:42:16 [INFO] [TRAIN] epoch=108, iter=39900/80000, loss=0.1140, lr=0.005371, batch_cost=1.0037, reader_cost=0.0115 | ETA 11:10:49 2020-11-03 06:43:55 [INFO] [TRAIN] epoch=108, iter=40000/80000, loss=0.1382, lr=0.005359, batch_cost=0.9897, reader_cost=0.0002 | ETA 10:59:48 2020-11-03 06:43:59 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-03 06:48:25 [INFO] [EVAL] #Images=500 mIoU=0.7782 Acc=0.9605 Kappa=0.9487 2020-11-03 06:48:25 [INFO] [EVAL] Category IoU: [0.9808 0.851 0.9246 0.5052 0.6155 0.6432 0.7057 0.7867 0.9227 0.661 0.943 0.8271 0.6391 0.949 0.7653 0.8677 0.8069 0.6132 0.7778] 2020-11-03 06:48:25 [INFO] [EVAL] Category Acc: [0.9916 0.9209 0.9561 0.8663 0.8098 0.8094 0.8278 0.8963 0.9493 0.8654 0.9645 0.8937 0.7998 0.9672 0.9114 0.9466 0.9011 0.8555 0.8637] 2020-11-03 06:48:27 [INFO] [EVAL] The model with the best validation mIoU (0.7782) was saved at iter 40000. 2020-11-03 06:50:06 [INFO] [TRAIN] epoch=108, iter=40100/80000, loss=0.1141, lr=0.005347, batch_cost=0.9880, reader_cost=0.0003 | ETA 10:57:02 2020-11-03 06:51:47 [INFO] [TRAIN] epoch=109, iter=40200/80000, loss=0.1203, lr=0.005335, batch_cost=1.0084, reader_cost=0.0133 | ETA 11:08:55 2020-11-03 06:53:27 [INFO] [TRAIN] epoch=109, iter=40300/80000, loss=0.1285, lr=0.005323, batch_cost=1.0015, reader_cost=0.0005 | ETA 11:02:40 2020-11-03 06:55:07 [INFO] [TRAIN] epoch=109, iter=40400/80000, loss=0.1294, lr=0.005311, batch_cost=0.9971, reader_cost=0.0002 | ETA 10:58:06 2020-11-03 06:56:48 [INFO] [TRAIN] epoch=109, iter=40500/80000, loss=0.1385, lr=0.005299, batch_cost=1.0155, reader_cost=0.0002 | ETA 11:08:32 2020-11-03 06:58:32 [INFO] [TRAIN] epoch=110, iter=40600/80000, loss=0.1193, lr=0.005287, batch_cost=1.0361, reader_cost=0.0136 | ETA 11:20:23 2020-11-03 07:00:16 [INFO] [TRAIN] epoch=110, iter=40700/80000, loss=0.1255, lr=0.005275, batch_cost=1.0420, reader_cost=0.0002 | ETA 11:22:28 2020-11-03 07:01:56 [INFO] [TRAIN] epoch=110, iter=40800/80000, loss=0.1156, lr=0.005262, batch_cost=1.0009, reader_cost=0.0002 | ETA 10:53:55 2020-11-03 07:03:37 [INFO] [TRAIN] epoch=110, iter=40900/80000, loss=0.1124, lr=0.005250, batch_cost=1.0042, reader_cost=0.0002 | ETA 10:54:23 2020-11-03 07:05:18 [INFO] [TRAIN] epoch=111, iter=41000/80000, loss=0.1164, lr=0.005238, batch_cost=1.0123, reader_cost=0.0123 | ETA 10:57:57 2020-11-03 07:06:58 [INFO] [TRAIN] epoch=111, iter=41100/80000, loss=0.1306, lr=0.005226, batch_cost=0.9989, reader_cost=0.0003 | ETA 10:47:38 2020-11-03 07:08:39 [INFO] [TRAIN] epoch=111, iter=41200/80000, loss=0.1050, lr=0.005214, batch_cost=1.0077, reader_cost=0.0002 | ETA 10:51:39 2020-11-03 07:10:20 [INFO] [TRAIN] epoch=112, iter=41300/80000, loss=0.1213, lr=0.005202, batch_cost=1.0109, reader_cost=0.0156 | ETA 10:52:01 2020-11-03 07:12:00 [INFO] [TRAIN] epoch=112, iter=41400/80000, loss=0.1315, lr=0.005190, batch_cost=1.0055, reader_cost=0.0002 | ETA 10:46:50 2020-11-03 07:13:41 [INFO] [TRAIN] epoch=112, iter=41500/80000, loss=0.1214, lr=0.005178, batch_cost=1.0032, reader_cost=0.0002 | ETA 10:43:42 2020-11-03 07:15:20 [INFO] [TRAIN] epoch=112, iter=41600/80000, loss=0.1055, lr=0.005166, batch_cost=0.9902, reader_cost=0.0003 | ETA 10:33:44 2020-11-03 07:16:59 [INFO] [TRAIN] epoch=113, iter=41700/80000, loss=0.1184, lr=0.005154, batch_cost=0.9994, reader_cost=0.0136 | ETA 10:37:58 2020-11-03 07:18:41 [INFO] [TRAIN] epoch=113, iter=41800/80000, loss=0.1220, lr=0.005141, batch_cost=1.0136, reader_cost=0.0004 | ETA 10:45:19 2020-11-03 07:20:25 [INFO] [TRAIN] epoch=113, iter=41900/80000, loss=0.1114, lr=0.005129, batch_cost=1.0395, reader_cost=0.0002 | ETA 11:00:04 2020-11-03 07:22:05 [INFO] [TRAIN] epoch=113, iter=42000/80000, loss=0.1049, lr=0.005117, batch_cost=1.0063, reader_cost=0.0002 | ETA 10:37:18 2020-11-03 07:23:46 [INFO] [TRAIN] epoch=114, iter=42100/80000, loss=0.1101, lr=0.005105, batch_cost=1.0046, reader_cost=0.0111 | ETA 10:34:34 2020-11-03 07:25:26 [INFO] [TRAIN] epoch=114, iter=42200/80000, loss=0.1199, lr=0.005093, batch_cost=0.9966, reader_cost=0.0005 | ETA 10:27:52 2020-11-03 07:27:05 [INFO] [TRAIN] epoch=114, iter=42300/80000, loss=0.1049, lr=0.005081, batch_cost=0.9970, reader_cost=0.0006 | ETA 10:26:26 2020-11-03 07:28:45 [INFO] [TRAIN] epoch=114, iter=42400/80000, loss=0.1071, lr=0.005069, batch_cost=0.9934, reader_cost=0.0002 | ETA 10:22:32 2020-11-03 07:30:27 [INFO] [TRAIN] epoch=115, iter=42500/80000, loss=0.1177, lr=0.005057, batch_cost=1.0220, reader_cost=0.0141 | ETA 10:38:44 2020-11-03 07:32:09 [INFO] [TRAIN] epoch=115, iter=42600/80000, loss=0.1155, lr=0.005044, batch_cost=1.0206, reader_cost=0.0003 | ETA 10:36:09 2020-11-03 07:33:48 [INFO] [TRAIN] epoch=115, iter=42700/80000, loss=0.1036, lr=0.005032, batch_cost=0.9938, reader_cost=0.0004 | ETA 10:17:48 2020-11-03 07:35:29 [INFO] [TRAIN] epoch=116, iter=42800/80000, loss=0.1316, lr=0.005020, batch_cost=1.0041, reader_cost=0.0131 | ETA 10:22:32 2020-11-03 07:37:07 [INFO] [TRAIN] epoch=116, iter=42900/80000, loss=0.1258, lr=0.005008, batch_cost=0.9837, reader_cost=0.0002 | ETA 10:08:13 2020-11-03 07:38:47 [INFO] [TRAIN] epoch=116, iter=43000/80000, loss=0.1226, lr=0.004996, batch_cost=1.0020, reader_cost=0.0010 | ETA 10:17:54 2020-11-03 07:40:32 [INFO] [TRAIN] epoch=116, iter=43100/80000, loss=0.1083, lr=0.004984, batch_cost=1.0462, reader_cost=0.0002 | ETA 10:43:25 2020-11-03 07:42:14 [INFO] [TRAIN] epoch=117, iter=43200/80000, loss=0.1382, lr=0.004972, batch_cost=1.0256, reader_cost=0.0125 | ETA 10:29:02 2020-11-03 07:43:54 [INFO] [TRAIN] epoch=117, iter=43300/80000, loss=0.1247, lr=0.004959, batch_cost=0.9983, reader_cost=0.0002 | ETA 10:10:39 2020-11-03 07:45:33 [INFO] [TRAIN] epoch=117, iter=43400/80000, loss=0.1366, lr=0.004947, batch_cost=0.9919, reader_cost=0.0002 | ETA 10:05:03 2020-11-03 07:47:17 [INFO] [TRAIN] epoch=117, iter=43500/80000, loss=0.1166, lr=0.004935, batch_cost=1.0370, reader_cost=0.0014 | ETA 10:30:50 2020-11-03 07:49:00 [INFO] [TRAIN] epoch=118, iter=43600/80000, loss=0.1145, lr=0.004923, batch_cost=1.0240, reader_cost=0.0159 | ETA 10:21:12 2020-11-03 07:50:39 [INFO] [TRAIN] epoch=118, iter=43700/80000, loss=0.1110, lr=0.004911, batch_cost=0.9906, reader_cost=0.0002 | ETA 09:59:17 2020-11-03 07:52:17 [INFO] [TRAIN] epoch=118, iter=43800/80000, loss=0.1026, lr=0.004899, batch_cost=0.9882, reader_cost=0.0002 | ETA 09:56:14 2020-11-03 07:53:59 [INFO] [TRAIN] epoch=119, iter=43900/80000, loss=0.1190, lr=0.004886, batch_cost=1.0188, reader_cost=0.0116 | ETA 10:12:59 2020-11-03 07:55:40 [INFO] [TRAIN] epoch=119, iter=44000/80000, loss=0.1039, lr=0.004874, batch_cost=1.0081, reader_cost=0.0002 | ETA 10:04:51 2020-11-03 07:57:21 [INFO] [TRAIN] epoch=119, iter=44100/80000, loss=0.1069, lr=0.004862, batch_cost=1.0111, reader_cost=0.0002 | ETA 10:04:57 2020-11-03 07:59:01 [INFO] [TRAIN] epoch=119, iter=44200/80000, loss=0.1147, lr=0.004850, batch_cost=0.9934, reader_cost=0.0002 | ETA 09:52:42 2020-11-03 08:00:40 [INFO] [TRAIN] epoch=120, iter=44300/80000, loss=0.1242, lr=0.004838, batch_cost=0.9983, reader_cost=0.0143 | ETA 09:53:58 2020-11-03 08:02:21 [INFO] [TRAIN] epoch=120, iter=44400/80000, loss=0.1183, lr=0.004825, batch_cost=1.0085, reader_cost=0.0002 | ETA 09:58:22 2020-11-03 08:04:01 [INFO] [TRAIN] epoch=120, iter=44500/80000, loss=0.1254, lr=0.004813, batch_cost=0.9997, reader_cost=0.0002 | ETA 09:51:28 2020-11-03 08:05:41 [INFO] [TRAIN] epoch=120, iter=44600/80000, loss=0.1072, lr=0.004801, batch_cost=0.9972, reader_cost=0.0002 | ETA 09:48:21 2020-11-03 08:07:21 [INFO] [TRAIN] epoch=121, iter=44700/80000, loss=0.1056, lr=0.004789, batch_cost=1.0027, reader_cost=0.0120 | ETA 09:49:56 2020-11-03 08:09:01 [INFO] [TRAIN] epoch=121, iter=44800/80000, loss=0.1273, lr=0.004777, batch_cost=1.0023, reader_cost=0.0002 | ETA 09:48:00 2020-11-03 08:10:41 [INFO] [TRAIN] epoch=121, iter=44900/80000, loss=0.1010, lr=0.004764, batch_cost=0.9960, reader_cost=0.0002 | ETA 09:42:38 2020-11-03 08:12:25 [INFO] [TRAIN] epoch=121, iter=45000/80000, loss=0.1168, lr=0.004752, batch_cost=1.0402, reader_cost=0.0002 | ETA 10:06:47 2020-11-03 08:14:09 [INFO] [TRAIN] epoch=122, iter=45100/80000, loss=0.1063, lr=0.004740, batch_cost=1.0410, reader_cost=0.0155 | ETA 10:05:32 2020-11-03 08:15:51 [INFO] [TRAIN] epoch=122, iter=45200/80000, loss=0.1407, lr=0.004728, batch_cost=1.0183, reader_cost=0.0002 | ETA 09:50:35 2020-11-03 08:17:30 [INFO] [TRAIN] epoch=122, iter=45300/80000, loss=0.1089, lr=0.004715, batch_cost=0.9894, reader_cost=0.0002 | ETA 09:32:13 2020-11-03 08:19:13 [INFO] [TRAIN] epoch=123, iter=45400/80000, loss=0.1182, lr=0.004703, batch_cost=1.0299, reader_cost=0.0145 | ETA 09:53:52 2020-11-03 08:20:56 [INFO] [TRAIN] epoch=123, iter=45500/80000, loss=0.1070, lr=0.004691, batch_cost=1.0322, reader_cost=0.0002 | ETA 09:53:29 2020-11-03 08:22:37 [INFO] [TRAIN] epoch=123, iter=45600/80000, loss=0.1077, lr=0.004679, batch_cost=1.0032, reader_cost=0.0005 | ETA 09:35:10 2020-11-03 08:24:16 [INFO] [TRAIN] epoch=123, iter=45700/80000, loss=0.1058, lr=0.004667, batch_cost=0.9991, reader_cost=0.0002 | ETA 09:31:08 2020-11-03 08:25:59 [INFO] [TRAIN] epoch=124, iter=45800/80000, loss=0.1111, lr=0.004654, batch_cost=1.0279, reader_cost=0.0125 | ETA 09:45:55 2020-11-03 08:27:41 [INFO] [TRAIN] epoch=124, iter=45900/80000, loss=0.1271, lr=0.004642, batch_cost=1.0133, reader_cost=0.0002 | ETA 09:35:54 2020-11-03 08:29:22 [INFO] [TRAIN] epoch=124, iter=46000/80000, loss=0.1107, lr=0.004630, batch_cost=1.0127, reader_cost=0.0002 | ETA 09:33:50 2020-11-03 08:31:05 [INFO] [TRAIN] epoch=124, iter=46100/80000, loss=0.1223, lr=0.004618, batch_cost=1.0269, reader_cost=0.0002 | ETA 09:40:12 2020-11-03 08:32:45 [INFO] [TRAIN] epoch=125, iter=46200/80000, loss=0.1167, lr=0.004605, batch_cost=1.0025, reader_cost=0.0118 | ETA 09:24:45 2020-11-03 08:34:23 [INFO] [TRAIN] epoch=125, iter=46300/80000, loss=0.1441, lr=0.004593, batch_cost=0.9835, reader_cost=0.0002 | ETA 09:12:23 2020-11-03 08:36:02 [INFO] [TRAIN] epoch=125, iter=46400/80000, loss=0.1050, lr=0.004581, batch_cost=0.9899, reader_cost=0.0003 | ETA 09:14:19 2020-11-03 08:37:42 [INFO] [TRAIN] epoch=125, iter=46500/80000, loss=0.1194, lr=0.004568, batch_cost=0.9987, reader_cost=0.0003 | ETA 09:17:35 2020-11-03 08:39:25 [INFO] [TRAIN] epoch=126, iter=46600/80000, loss=0.1081, lr=0.004556, batch_cost=1.0292, reader_cost=0.0136 | ETA 09:32:54 2020-11-03 08:41:06 [INFO] [TRAIN] epoch=126, iter=46700/80000, loss=0.1260, lr=0.004544, batch_cost=1.0090, reader_cost=0.0002 | ETA 09:19:58 2020-11-03 08:42:47 [INFO] [TRAIN] epoch=126, iter=46800/80000, loss=0.1021, lr=0.004532, batch_cost=1.0069, reader_cost=0.0002 | ETA 09:17:10 2020-11-03 08:44:28 [INFO] [TRAIN] epoch=127, iter=46900/80000, loss=0.1244, lr=0.004519, batch_cost=1.0123, reader_cost=0.0136 | ETA 09:18:26 2020-11-03 08:46:08 [INFO] [TRAIN] epoch=127, iter=47000/80000, loss=0.1513, lr=0.004507, batch_cost=0.9999, reader_cost=0.0002 | ETA 09:09:56 2020-11-03 08:47:48 [INFO] [TRAIN] epoch=127, iter=47100/80000, loss=0.1216, lr=0.004495, batch_cost=1.0030, reader_cost=0.0011 | ETA 09:09:59 2020-11-03 08:49:30 [INFO] [TRAIN] epoch=127, iter=47200/80000, loss=0.1102, lr=0.004482, batch_cost=1.0169, reader_cost=0.0005 | ETA 09:15:55 2020-11-03 08:51:11 [INFO] [TRAIN] epoch=128, iter=47300/80000, loss=0.1142, lr=0.004470, batch_cost=1.0152, reader_cost=0.0127 | ETA 09:13:16 2020-11-03 08:52:54 [INFO] [TRAIN] epoch=128, iter=47400/80000, loss=0.1223, lr=0.004458, batch_cost=1.0306, reader_cost=0.0002 | ETA 09:19:57 2020-11-03 08:54:34 [INFO] [TRAIN] epoch=128, iter=47500/80000, loss=0.0994, lr=0.004446, batch_cost=0.9989, reader_cost=0.0007 | ETA 09:01:04 2020-11-03 08:56:14 [INFO] [TRAIN] epoch=128, iter=47600/80000, loss=0.1165, lr=0.004433, batch_cost=0.9950, reader_cost=0.0002 | ETA 08:57:17 2020-11-03 08:57:54 [INFO] [TRAIN] epoch=129, iter=47700/80000, loss=0.1062, lr=0.004421, batch_cost=1.0026, reader_cost=0.0140 | ETA 08:59:43 2020-11-03 08:59:34 [INFO] [TRAIN] epoch=129, iter=47800/80000, loss=0.1126, lr=0.004409, batch_cost=0.9985, reader_cost=0.0002 | ETA 08:55:51 2020-11-03 09:01:25 [INFO] [TRAIN] epoch=129, iter=47900/80000, loss=0.0966, lr=0.004396, batch_cost=1.1103, reader_cost=0.0004 | ETA 09:54:00 2020-11-03 09:03:10 [INFO] [TRAIN] epoch=130, iter=48000/80000, loss=0.1147, lr=0.004384, batch_cost=1.0498, reader_cost=0.0182 | ETA 09:19:53 2020-11-03 09:03:15 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-03 09:07:47 [INFO] [EVAL] #Images=500 mIoU=0.7882 Acc=0.9605 Kappa=0.9487 2020-11-03 09:07:47 [INFO] [EVAL] Category IoU: [0.981 0.8507 0.9249 0.5295 0.6236 0.645 0.7004 0.7826 0.923 0.6143 0.9379 0.8279 0.6488 0.9486 0.8391 0.9055 0.8306 0.6803 0.7815] 2020-11-03 09:07:47 [INFO] [EVAL] Category Acc: [0.9905 0.9231 0.9579 0.8498 0.8037 0.7849 0.8038 0.9038 0.9507 0.8558 0.9725 0.8959 0.7628 0.9699 0.9313 0.9563 0.9283 0.8077 0.8619] 2020-11-03 09:07:49 [INFO] [EVAL] The model with the best validation mIoU (0.7882) was saved at iter 48000. 2020-11-03 09:09:28 [INFO] [TRAIN] epoch=130, iter=48100/80000, loss=0.1048, lr=0.004372, batch_cost=0.9892, reader_cost=0.0002 | ETA 08:45:53 2020-11-03 09:11:07 [INFO] [TRAIN] epoch=130, iter=48200/80000, loss=0.1114, lr=0.004359, batch_cost=0.9890, reader_cost=0.0002 | ETA 08:44:11 2020-11-03 09:12:47 [INFO] [TRAIN] epoch=130, iter=48300/80000, loss=0.0958, lr=0.004347, batch_cost=1.0010, reader_cost=0.0002 | ETA 08:48:50 2020-11-03 09:14:28 [INFO] [TRAIN] epoch=131, iter=48400/80000, loss=0.1183, lr=0.004335, batch_cost=1.0093, reader_cost=0.0119 | ETA 08:51:32 2020-11-03 09:16:06 [INFO] [TRAIN] epoch=131, iter=48500/80000, loss=0.1116, lr=0.004322, batch_cost=0.9841, reader_cost=0.0002 | ETA 08:36:38 2020-11-03 09:17:46 [INFO] [TRAIN] epoch=131, iter=48600/80000, loss=0.1212, lr=0.004310, batch_cost=0.9975, reader_cost=0.0002 | ETA 08:42:02 2020-11-03 09:19:27 [INFO] [TRAIN] epoch=131, iter=48700/80000, loss=0.1154, lr=0.004298, batch_cost=1.0073, reader_cost=0.0002 | ETA 08:45:29 2020-11-03 09:21:08 [INFO] [TRAIN] epoch=132, iter=48800/80000, loss=0.1079, lr=0.004285, batch_cost=1.0091, reader_cost=0.0144 | ETA 08:44:43 2020-11-03 09:22:48 [INFO] [TRAIN] epoch=132, iter=48900/80000, loss=0.1217, lr=0.004273, batch_cost=0.9976, reader_cost=0.0004 | ETA 08:37:06 2020-11-03 09:24:27 [INFO] [TRAIN] epoch=132, iter=49000/80000, loss=0.0955, lr=0.004260, batch_cost=0.9985, reader_cost=0.0002 | ETA 08:35:54 2020-11-03 09:26:06 [INFO] [TRAIN] epoch=132, iter=49100/80000, loss=0.1099, lr=0.004248, batch_cost=0.9820, reader_cost=0.0002 | ETA 08:25:44 2020-11-03 09:27:47 [INFO] [TRAIN] epoch=133, iter=49200/80000, loss=0.1021, lr=0.004236, batch_cost=1.0141, reader_cost=0.0126 | ETA 08:40:33 2020-11-03 09:29:26 [INFO] [TRAIN] epoch=133, iter=49300/80000, loss=0.1183, lr=0.004223, batch_cost=0.9868, reader_cost=0.0002 | ETA 08:24:56 2020-11-03 09:31:06 [INFO] [TRAIN] epoch=133, iter=49400/80000, loss=0.1013, lr=0.004211, batch_cost=1.0037, reader_cost=0.0003 | ETA 08:31:53 2020-11-03 09:32:46 [INFO] [TRAIN] epoch=134, iter=49500/80000, loss=0.1148, lr=0.004199, batch_cost=0.9999, reader_cost=0.0124 | ETA 08:28:17 2020-11-03 09:34:29 [INFO] [TRAIN] epoch=134, iter=49600/80000, loss=0.1101, lr=0.004186, batch_cost=1.0323, reader_cost=0.0004 | ETA 08:43:03 2020-11-03 09:36:09 [INFO] [TRAIN] epoch=134, iter=49700/80000, loss=0.1143, lr=0.004174, batch_cost=1.0001, reader_cost=0.0002 | ETA 08:25:01 2020-11-03 09:37:49 [INFO] [TRAIN] epoch=134, iter=49800/80000, loss=0.1183, lr=0.004161, batch_cost=0.9970, reader_cost=0.0001 | ETA 08:21:48 2020-11-03 09:39:30 [INFO] [TRAIN] epoch=135, iter=49900/80000, loss=0.1211, lr=0.004149, batch_cost=1.0140, reader_cost=0.0169 | ETA 08:28:42 2020-11-03 09:41:09 [INFO] [TRAIN] epoch=135, iter=50000/80000, loss=0.1055, lr=0.004137, batch_cost=0.9845, reader_cost=0.0002 | ETA 08:12:14 2020-11-03 09:42:48 [INFO] [TRAIN] epoch=135, iter=50100/80000, loss=0.1258, lr=0.004124, batch_cost=0.9925, reader_cost=0.0002 | ETA 08:14:34 2020-11-03 09:44:29 [INFO] [TRAIN] epoch=135, iter=50200/80000, loss=0.1175, lr=0.004112, batch_cost=1.0103, reader_cost=0.0028 | ETA 08:21:47 2020-11-03 09:46:10 [INFO] [TRAIN] epoch=136, iter=50300/80000, loss=0.1031, lr=0.004099, batch_cost=1.0065, reader_cost=0.0121 | ETA 08:18:12 2020-11-03 09:47:51 [INFO] [TRAIN] epoch=136, iter=50400/80000, loss=0.1225, lr=0.004087, batch_cost=1.0121, reader_cost=0.0002 | ETA 08:19:16 2020-11-03 09:49:33 [INFO] [TRAIN] epoch=136, iter=50500/80000, loss=0.0955, lr=0.004074, batch_cost=1.0194, reader_cost=0.0002 | ETA 08:21:12 2020-11-03 09:51:16 [INFO] [TRAIN] epoch=137, iter=50600/80000, loss=0.1123, lr=0.004062, batch_cost=1.0340, reader_cost=0.0196 | ETA 08:26:38 2020-11-03 09:52:55 [INFO] [TRAIN] epoch=137, iter=50700/80000, loss=0.1099, lr=0.004050, batch_cost=0.9911, reader_cost=0.0005 | ETA 08:04:00 2020-11-03 09:54:37 [INFO] [TRAIN] epoch=137, iter=50800/80000, loss=0.1081, lr=0.004037, batch_cost=1.0144, reader_cost=0.0004 | ETA 08:13:41 2020-11-03 09:56:19 [INFO] [TRAIN] epoch=137, iter=50900/80000, loss=0.0902, lr=0.004025, batch_cost=1.0218, reader_cost=0.0002 | ETA 08:15:35 2020-11-03 09:58:01 [INFO] [TRAIN] epoch=138, iter=51000/80000, loss=0.1064, lr=0.004012, batch_cost=1.0233, reader_cost=0.0118 | ETA 08:14:36 2020-11-03 09:59:41 [INFO] [TRAIN] epoch=138, iter=51100/80000, loss=0.1038, lr=0.004000, batch_cost=0.9999, reader_cost=0.0002 | ETA 08:01:37 2020-11-03 10:01:20 [INFO] [TRAIN] epoch=138, iter=51200/80000, loss=0.1120, lr=0.003987, batch_cost=0.9884, reader_cost=0.0002 | ETA 07:54:25 2020-11-03 10:03:00 [INFO] [TRAIN] epoch=138, iter=51300/80000, loss=0.1070, lr=0.003975, batch_cost=0.9974, reader_cost=0.0002 | ETA 07:57:04 2020-11-03 10:04:41 [INFO] [TRAIN] epoch=139, iter=51400/80000, loss=0.1047, lr=0.003962, batch_cost=1.0052, reader_cost=0.0123 | ETA 07:59:09 2020-11-03 10:06:20 [INFO] [TRAIN] epoch=139, iter=51500/80000, loss=0.1193, lr=0.003950, batch_cost=0.9946, reader_cost=0.0002 | ETA 07:52:25 2020-11-03 10:07:59 [INFO] [TRAIN] epoch=139, iter=51600/80000, loss=0.1007, lr=0.003937, batch_cost=0.9949, reader_cost=0.0002 | ETA 07:50:56 2020-11-03 10:09:39 [INFO] [TRAIN] epoch=139, iter=51700/80000, loss=0.1187, lr=0.003925, batch_cost=0.9965, reader_cost=0.0011 | ETA 07:50:00 2020-11-03 10:11:20 [INFO] [TRAIN] epoch=140, iter=51800/80000, loss=0.1115, lr=0.003913, batch_cost=1.0109, reader_cost=0.0138 | ETA 07:55:06 2020-11-03 10:13:01 [INFO] [TRAIN] epoch=140, iter=51900/80000, loss=0.1184, lr=0.003900, batch_cost=1.0089, reader_cost=0.0002 | ETA 07:52:30 2020-11-03 10:14:42 [INFO] [TRAIN] epoch=140, iter=52000/80000, loss=0.1070, lr=0.003888, batch_cost=1.0045, reader_cost=0.0002 | ETA 07:48:45 2020-11-03 10:16:25 [INFO] [TRAIN] epoch=141, iter=52100/80000, loss=0.1057, lr=0.003875, batch_cost=1.0369, reader_cost=0.0129 | ETA 08:02:08 2020-11-03 10:18:07 [INFO] [TRAIN] epoch=141, iter=52200/80000, loss=0.1035, lr=0.003863, batch_cost=1.0154, reader_cost=0.0003 | ETA 07:50:28 2020-11-03 10:19:48 [INFO] [TRAIN] epoch=141, iter=52300/80000, loss=0.1056, lr=0.003850, batch_cost=1.0108, reader_cost=0.0002 | ETA 07:46:39 2020-11-03 10:21:29 [INFO] [TRAIN] epoch=141, iter=52400/80000, loss=0.0915, lr=0.003838, batch_cost=1.0119, reader_cost=0.0002 | ETA 07:45:28 2020-11-03 10:23:10 [INFO] [TRAIN] epoch=142, iter=52500/80000, loss=0.1204, lr=0.003825, batch_cost=1.0111, reader_cost=0.0133 | ETA 07:43:25 2020-11-03 10:24:51 [INFO] [TRAIN] epoch=142, iter=52600/80000, loss=0.1156, lr=0.003812, batch_cost=1.0107, reader_cost=0.0003 | ETA 07:41:33 2020-11-03 10:26:31 [INFO] [TRAIN] epoch=142, iter=52700/80000, loss=0.1108, lr=0.003800, batch_cost=0.9929, reader_cost=0.0002 | ETA 07:31:46 2020-11-03 10:28:11 [INFO] [TRAIN] epoch=142, iter=52800/80000, loss=0.1003, lr=0.003787, batch_cost=1.0009, reader_cost=0.0058 | ETA 07:33:44 2020-11-03 10:29:51 [INFO] [TRAIN] epoch=143, iter=52900/80000, loss=0.1032, lr=0.003775, batch_cost=1.0083, reader_cost=0.0116 | ETA 07:35:24 2020-11-03 10:31:32 [INFO] [TRAIN] epoch=143, iter=53000/80000, loss=0.1074, lr=0.003762, batch_cost=1.0042, reader_cost=0.0002 | ETA 07:31:52 2020-11-03 10:33:12 [INFO] [TRAIN] epoch=143, iter=53100/80000, loss=0.0954, lr=0.003750, batch_cost=0.9966, reader_cost=0.0003 | ETA 07:26:48 2020-11-03 10:34:55 [INFO] [TRAIN] epoch=144, iter=53200/80000, loss=0.1211, lr=0.003737, batch_cost=1.0298, reader_cost=0.0149 | ETA 07:39:59 2020-11-03 10:36:34 [INFO] [TRAIN] epoch=144, iter=53300/80000, loss=0.1073, lr=0.003725, batch_cost=0.9957, reader_cost=0.0002 | ETA 07:23:04 2020-11-03 10:38:18 [INFO] [TRAIN] epoch=144, iter=53400/80000, loss=0.1308, lr=0.003712, batch_cost=1.0356, reader_cost=0.0002 | ETA 07:39:06 2020-11-03 10:39:59 [INFO] [TRAIN] epoch=144, iter=53500/80000, loss=0.1103, lr=0.003700, batch_cost=1.0145, reader_cost=0.0002 | ETA 07:28:04 2020-11-03 10:41:40 [INFO] [TRAIN] epoch=145, iter=53600/80000, loss=0.1072, lr=0.003687, batch_cost=1.0127, reader_cost=0.0172 | ETA 07:25:35 2020-11-03 10:43:21 [INFO] [TRAIN] epoch=145, iter=53700/80000, loss=0.1128, lr=0.003674, batch_cost=1.0089, reader_cost=0.0002 | ETA 07:22:12 2020-11-03 10:45:03 [INFO] [TRAIN] epoch=145, iter=53800/80000, loss=0.1206, lr=0.003662, batch_cost=1.0168, reader_cost=0.0002 | ETA 07:24:01 2020-11-03 10:46:45 [INFO] [TRAIN] epoch=145, iter=53900/80000, loss=0.1043, lr=0.003649, batch_cost=1.0192, reader_cost=0.0005 | ETA 07:23:22 2020-11-03 10:48:27 [INFO] [TRAIN] epoch=146, iter=54000/80000, loss=0.1034, lr=0.003637, batch_cost=1.0237, reader_cost=0.0129 | ETA 07:23:36 2020-11-03 10:50:08 [INFO] [TRAIN] epoch=146, iter=54100/80000, loss=0.1130, lr=0.003624, batch_cost=1.0073, reader_cost=0.0002 | ETA 07:14:48 2020-11-03 10:51:49 [INFO] [TRAIN] epoch=146, iter=54200/80000, loss=0.1102, lr=0.003612, batch_cost=1.0125, reader_cost=0.0002 | ETA 07:15:21 2020-11-03 10:53:29 [INFO] [TRAIN] epoch=146, iter=54300/80000, loss=0.1174, lr=0.003599, batch_cost=0.9956, reader_cost=0.0002 | ETA 07:06:26 2020-11-03 10:55:11 [INFO] [TRAIN] epoch=147, iter=54400/80000, loss=0.1020, lr=0.003586, batch_cost=1.0229, reader_cost=0.0120 | ETA 07:16:25 2020-11-03 10:56:52 [INFO] [TRAIN] epoch=147, iter=54500/80000, loss=0.1069, lr=0.003574, batch_cost=1.0048, reader_cost=0.0005 | ETA 07:07:02 2020-11-03 10:58:32 [INFO] [TRAIN] epoch=147, iter=54600/80000, loss=0.1024, lr=0.003561, batch_cost=0.9992, reader_cost=0.0002 | ETA 07:03:00 2020-11-03 11:00:13 [INFO] [TRAIN] epoch=148, iter=54700/80000, loss=0.1122, lr=0.003548, batch_cost=1.0099, reader_cost=0.0129 | ETA 07:05:50 2020-11-03 11:01:51 [INFO] [TRAIN] epoch=148, iter=54800/80000, loss=0.1085, lr=0.003536, batch_cost=0.9889, reader_cost=0.0002 | ETA 06:55:20 2020-11-03 11:03:33 [INFO] [TRAIN] epoch=148, iter=54900/80000, loss=0.1144, lr=0.003523, batch_cost=1.0145, reader_cost=0.0002 | ETA 07:04:22 2020-11-03 11:05:13 [INFO] [TRAIN] epoch=148, iter=55000/80000, loss=0.0999, lr=0.003511, batch_cost=0.9965, reader_cost=0.0002 | ETA 06:55:12 2020-11-03 11:06:54 [INFO] [TRAIN] epoch=149, iter=55100/80000, loss=0.1044, lr=0.003498, batch_cost=1.0117, reader_cost=0.0159 | ETA 06:59:52 2020-11-03 11:08:35 [INFO] [TRAIN] epoch=149, iter=55200/80000, loss=0.1302, lr=0.003485, batch_cost=1.0078, reader_cost=0.0002 | ETA 06:56:34 2020-11-03 11:10:14 [INFO] [TRAIN] epoch=149, iter=55300/80000, loss=0.0968, lr=0.003473, batch_cost=0.9970, reader_cost=0.0002 | ETA 06:50:24 2020-11-03 11:11:54 [INFO] [TRAIN] epoch=149, iter=55400/80000, loss=0.1165, lr=0.003460, batch_cost=1.0004, reader_cost=0.0002 | ETA 06:50:09 2020-11-03 11:13:38 [INFO] [TRAIN] epoch=150, iter=55500/80000, loss=0.0978, lr=0.003447, batch_cost=1.0345, reader_cost=0.0130 | ETA 07:02:26 2020-11-03 11:15:19 [INFO] [TRAIN] epoch=150, iter=55600/80000, loss=0.1104, lr=0.003435, batch_cost=1.0088, reader_cost=0.0002 | ETA 06:50:14 2020-11-03 11:16:58 [INFO] [TRAIN] epoch=150, iter=55700/80000, loss=0.1052, lr=0.003422, batch_cost=0.9984, reader_cost=0.0002 | ETA 06:44:21 2020-11-03 11:18:38 [INFO] [TRAIN] epoch=150, iter=55800/80000, loss=0.1242, lr=0.003409, batch_cost=0.9930, reader_cost=0.0002 | ETA 06:40:29 2020-11-03 11:20:21 [INFO] [TRAIN] epoch=151, iter=55900/80000, loss=0.1030, lr=0.003397, batch_cost=1.0299, reader_cost=0.0124 | ETA 06:53:40 2020-11-03 11:22:01 [INFO] [TRAIN] epoch=151, iter=56000/80000, loss=0.1099, lr=0.003384, batch_cost=1.0072, reader_cost=0.0002 | ETA 06:42:53 2020-11-03 11:22:06 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-03 11:26:37 [INFO] [EVAL] #Images=500 mIoU=0.7843 Acc=0.9607 Kappa=0.9490 2020-11-03 11:26:37 [INFO] [EVAL] Category IoU: [0.9808 0.8562 0.9241 0.5252 0.6182 0.6534 0.7126 0.7871 0.923 0.6213 0.9434 0.8288 0.6432 0.9538 0.7662 0.8805 0.8307 0.6747 0.7791] 2020-11-03 11:26:37 [INFO] [EVAL] Category Acc: [0.9902 0.9305 0.9555 0.8395 0.8157 0.8013 0.8297 0.9048 0.9555 0.7773 0.9606 0.8979 0.7847 0.9745 0.9357 0.9414 0.9091 0.8491 0.8546] 2020-11-03 11:26:37 [INFO] [EVAL] The model with the best validation mIoU (0.7882) was saved at iter 48000. 2020-11-03 11:28:15 [INFO] [TRAIN] epoch=151, iter=56100/80000, loss=0.1054, lr=0.003371, batch_cost=0.9877, reader_cost=0.0002 | ETA 06:33:26 2020-11-03 11:29:57 [INFO] [TRAIN] epoch=152, iter=56200/80000, loss=0.1110, lr=0.003359, batch_cost=1.0106, reader_cost=0.0152 | ETA 06:40:51 2020-11-03 11:31:35 [INFO] [TRAIN] epoch=152, iter=56300/80000, loss=0.1094, lr=0.003346, batch_cost=0.9885, reader_cost=0.0011 | ETA 06:30:28 2020-11-03 11:33:15 [INFO] [TRAIN] epoch=152, iter=56400/80000, loss=0.1132, lr=0.003333, batch_cost=0.9993, reader_cost=0.0019 | ETA 06:33:03 2020-11-03 11:34:56 [INFO] [TRAIN] epoch=152, iter=56500/80000, loss=0.0973, lr=0.003320, batch_cost=1.0114, reader_cost=0.0002 | ETA 06:36:08 2020-11-03 11:36:37 [INFO] [TRAIN] epoch=153, iter=56600/80000, loss=0.1198, lr=0.003308, batch_cost=1.0052, reader_cost=0.0119 | ETA 06:32:00 2020-11-03 11:38:18 [INFO] [TRAIN] epoch=153, iter=56700/80000, loss=0.1266, lr=0.003295, batch_cost=1.0134, reader_cost=0.0002 | ETA 06:33:31 2020-11-03 11:40:03 [INFO] [TRAIN] epoch=153, iter=56800/80000, loss=0.1043, lr=0.003282, batch_cost=1.0452, reader_cost=0.0002 | ETA 06:44:07 2020-11-03 11:41:44 [INFO] [TRAIN] epoch=153, iter=56900/80000, loss=0.1061, lr=0.003270, batch_cost=1.0112, reader_cost=0.0002 | ETA 06:29:19 2020-11-03 11:43:24 [INFO] [TRAIN] epoch=154, iter=57000/80000, loss=0.0937, lr=0.003257, batch_cost=1.0043, reader_cost=0.0136 | ETA 06:24:59 2020-11-03 11:45:07 [INFO] [TRAIN] epoch=154, iter=57100/80000, loss=0.1123, lr=0.003244, batch_cost=1.0258, reader_cost=0.0002 | ETA 06:31:30 2020-11-03 11:46:46 [INFO] [TRAIN] epoch=154, iter=57200/80000, loss=0.1089, lr=0.003231, batch_cost=0.9900, reader_cost=0.0002 | ETA 06:16:12 2020-11-03 11:48:29 [INFO] [TRAIN] epoch=155, iter=57300/80000, loss=0.1529, lr=0.003219, batch_cost=1.0253, reader_cost=0.0123 | ETA 06:27:55 2020-11-03 11:50:07 [INFO] [TRAIN] epoch=155, iter=57400/80000, loss=0.1145, lr=0.003206, batch_cost=0.9896, reader_cost=0.0002 | ETA 06:12:46 2020-11-03 11:51:47 [INFO] [TRAIN] epoch=155, iter=57500/80000, loss=0.1061, lr=0.003193, batch_cost=0.9994, reader_cost=0.0002 | ETA 06:14:46 2020-11-03 11:53:27 [INFO] [TRAIN] epoch=155, iter=57600/80000, loss=0.0950, lr=0.003180, batch_cost=0.9948, reader_cost=0.0003 | ETA 06:11:23 2020-11-03 11:55:08 [INFO] [TRAIN] epoch=156, iter=57700/80000, loss=0.1046, lr=0.003167, batch_cost=1.0084, reader_cost=0.0126 | ETA 06:14:46 2020-11-03 11:56:49 [INFO] [TRAIN] epoch=156, iter=57800/80000, loss=0.1171, lr=0.003155, batch_cost=1.0150, reader_cost=0.0003 | ETA 06:15:33 2020-11-03 11:58:30 [INFO] [TRAIN] epoch=156, iter=57900/80000, loss=0.1101, lr=0.003142, batch_cost=1.0083, reader_cost=0.0004 | ETA 06:11:24 2020-11-03 12:00:12 [INFO] [TRAIN] epoch=156, iter=58000/80000, loss=0.0981, lr=0.003129, batch_cost=1.0221, reader_cost=0.0002 | ETA 06:14:45 2020-11-03 12:01:55 [INFO] [TRAIN] epoch=157, iter=58100/80000, loss=0.0987, lr=0.003116, batch_cost=1.0225, reader_cost=0.0111 | ETA 06:13:13 2020-11-03 12:03:35 [INFO] [TRAIN] epoch=157, iter=58200/80000, loss=0.1138, lr=0.003103, batch_cost=1.0010, reader_cost=0.0002 | ETA 06:03:42 2020-11-03 12:05:14 [INFO] [TRAIN] epoch=157, iter=58300/80000, loss=0.0965, lr=0.003091, batch_cost=0.9984, reader_cost=0.0002 | ETA 06:01:05 2020-11-03 12:06:57 [INFO] [TRAIN] epoch=157, iter=58400/80000, loss=0.1024, lr=0.003078, batch_cost=1.0211, reader_cost=0.0002 | ETA 06:07:36 2020-11-03 12:08:40 [INFO] [TRAIN] epoch=158, iter=58500/80000, loss=0.1034, lr=0.003065, batch_cost=1.0346, reader_cost=0.0131 | ETA 06:10:43 2020-11-03 12:10:23 [INFO] [TRAIN] epoch=158, iter=58600/80000, loss=0.1160, lr=0.003052, batch_cost=1.0297, reader_cost=0.0009 | ETA 06:07:15 2020-11-03 12:12:03 [INFO] [TRAIN] epoch=158, iter=58700/80000, loss=0.1037, lr=0.003039, batch_cost=1.0034, reader_cost=0.0005 | ETA 05:56:11 2020-11-03 12:13:44 [INFO] [TRAIN] epoch=159, iter=58800/80000, loss=0.1040, lr=0.003026, batch_cost=1.0065, reader_cost=0.0118 | ETA 05:55:36 2020-11-03 12:15:23 [INFO] [TRAIN] epoch=159, iter=58900/80000, loss=0.1037, lr=0.003014, batch_cost=0.9886, reader_cost=0.0002 | ETA 05:47:40 2020-11-03 12:17:03 [INFO] [TRAIN] epoch=159, iter=59000/80000, loss=0.1043, lr=0.003001, batch_cost=0.9973, reader_cost=0.0009 | ETA 05:49:04 2020-11-03 12:18:44 [INFO] [TRAIN] epoch=159, iter=59100/80000, loss=0.0997, lr=0.002988, batch_cost=1.0139, reader_cost=0.0035 | ETA 05:53:10 2020-11-03 12:20:27 [INFO] [TRAIN] epoch=160, iter=59200/80000, loss=0.0974, lr=0.002975, batch_cost=1.0326, reader_cost=0.0123 | ETA 05:57:58 2020-11-03 12:22:08 [INFO] [TRAIN] epoch=160, iter=59300/80000, loss=0.1089, lr=0.002962, batch_cost=1.0046, reader_cost=0.0002 | ETA 05:46:35 2020-11-03 12:23:47 [INFO] [TRAIN] epoch=160, iter=59400/80000, loss=0.1044, lr=0.002949, batch_cost=0.9886, reader_cost=0.0002 | ETA 05:39:24 2020-11-03 12:25:27 [INFO] [TRAIN] epoch=160, iter=59500/80000, loss=0.1075, lr=0.002936, batch_cost=1.0056, reader_cost=0.0002 | ETA 05:43:33 2020-11-03 12:27:11 [INFO] [TRAIN] epoch=161, iter=59600/80000, loss=0.1077, lr=0.002924, batch_cost=1.0339, reader_cost=0.0129 | ETA 05:51:31 2020-11-03 12:28:51 [INFO] [TRAIN] epoch=161, iter=59700/80000, loss=0.1145, lr=0.002911, batch_cost=1.0006, reader_cost=0.0005 | ETA 05:38:32 2020-11-03 12:30:32 [INFO] [TRAIN] epoch=161, iter=59800/80000, loss=0.0913, lr=0.002898, batch_cost=1.0104, reader_cost=0.0002 | ETA 05:40:09 2020-11-03 12:32:16 [INFO] [TRAIN] epoch=162, iter=59900/80000, loss=0.1157, lr=0.002885, batch_cost=1.0464, reader_cost=0.0128 | ETA 05:50:32 2020-11-03 12:33:57 [INFO] [TRAIN] epoch=162, iter=60000/80000, loss=0.0967, lr=0.002872, batch_cost=1.0032, reader_cost=0.0002 | ETA 05:34:24 2020-11-03 12:35:37 [INFO] [TRAIN] epoch=162, iter=60100/80000, loss=0.0952, lr=0.002859, batch_cost=0.9992, reader_cost=0.0031 | ETA 05:31:24 2020-11-03 12:37:17 [INFO] [TRAIN] epoch=162, iter=60200/80000, loss=0.0984, lr=0.002846, batch_cost=1.0035, reader_cost=0.0002 | ETA 05:31:08 2020-11-03 12:38:57 [INFO] [TRAIN] epoch=163, iter=60300/80000, loss=0.1032, lr=0.002833, batch_cost=0.9990, reader_cost=0.0116 | ETA 05:28:00 2020-11-03 12:40:36 [INFO] [TRAIN] epoch=163, iter=60400/80000, loss=0.1010, lr=0.002820, batch_cost=0.9915, reader_cost=0.0002 | ETA 05:23:54 2020-11-03 12:42:15 [INFO] [TRAIN] epoch=163, iter=60500/80000, loss=0.1017, lr=0.002807, batch_cost=0.9855, reader_cost=0.0002 | ETA 05:20:17 2020-11-03 12:43:54 [INFO] [TRAIN] epoch=163, iter=60600/80000, loss=0.1019, lr=0.002794, batch_cost=0.9977, reader_cost=0.0002 | ETA 05:22:35 2020-11-03 12:45:37 [INFO] [TRAIN] epoch=164, iter=60700/80000, loss=0.0950, lr=0.002781, batch_cost=1.0289, reader_cost=0.0160 | ETA 05:30:58 2020-11-03 12:47:17 [INFO] [TRAIN] epoch=164, iter=60800/80000, loss=0.1082, lr=0.002768, batch_cost=1.0011, reader_cost=0.0002 | ETA 05:20:22 2020-11-03 12:48:57 [INFO] [TRAIN] epoch=164, iter=60900/80000, loss=0.0978, lr=0.002755, batch_cost=0.9984, reader_cost=0.0002 | ETA 05:17:48 2020-11-03 12:50:38 [INFO] [TRAIN] epoch=164, iter=61000/80000, loss=0.1186, lr=0.002742, batch_cost=1.0037, reader_cost=0.0002 | ETA 05:17:51 2020-11-03 12:52:18 [INFO] [TRAIN] epoch=165, iter=61100/80000, loss=0.0945, lr=0.002729, batch_cost=1.0080, reader_cost=0.0120 | ETA 05:17:32 2020-11-03 12:53:58 [INFO] [TRAIN] epoch=165, iter=61200/80000, loss=0.1046, lr=0.002716, batch_cost=1.0009, reader_cost=0.0002 | ETA 05:13:36 2020-11-03 12:55:39 [INFO] [TRAIN] epoch=165, iter=61300/80000, loss=0.1034, lr=0.002703, batch_cost=1.0083, reader_cost=0.0002 | ETA 05:14:15 2020-11-03 12:57:20 [INFO] [TRAIN] epoch=166, iter=61400/80000, loss=0.1046, lr=0.002690, batch_cost=1.0113, reader_cost=0.0115 | ETA 05:13:29 2020-11-03 12:59:00 [INFO] [TRAIN] epoch=166, iter=61500/80000, loss=0.0908, lr=0.002677, batch_cost=0.9994, reader_cost=0.0002 | ETA 05:08:08 2020-11-03 13:00:41 [INFO] [TRAIN] epoch=166, iter=61600/80000, loss=0.1064, lr=0.002664, batch_cost=1.0039, reader_cost=0.0002 | ETA 05:07:51 2020-11-03 13:02:23 [INFO] [TRAIN] epoch=166, iter=61700/80000, loss=0.0951, lr=0.002651, batch_cost=1.0218, reader_cost=0.0006 | ETA 05:11:39 2020-11-03 13:04:05 [INFO] [TRAIN] epoch=167, iter=61800/80000, loss=0.0982, lr=0.002638, batch_cost=1.0207, reader_cost=0.0119 | ETA 05:09:36 2020-11-03 13:05:48 [INFO] [TRAIN] epoch=167, iter=61900/80000, loss=0.1177, lr=0.002625, batch_cost=1.0274, reader_cost=0.0002 | ETA 05:09:56 2020-11-03 13:07:30 [INFO] [TRAIN] epoch=167, iter=62000/80000, loss=0.0959, lr=0.002612, batch_cost=1.0181, reader_cost=0.0002 | ETA 05:05:26 2020-11-03 13:09:11 [INFO] [TRAIN] epoch=167, iter=62100/80000, loss=0.1171, lr=0.002599, batch_cost=1.0193, reader_cost=0.0002 | ETA 05:04:04 2020-11-03 13:10:52 [INFO] [TRAIN] epoch=168, iter=62200/80000, loss=0.1383, lr=0.002586, batch_cost=1.0048, reader_cost=0.0127 | ETA 04:58:06 2020-11-03 13:12:32 [INFO] [TRAIN] epoch=168, iter=62300/80000, loss=0.1052, lr=0.002573, batch_cost=1.0020, reader_cost=0.0043 | ETA 04:55:35 2020-11-03 13:14:15 [INFO] [TRAIN] epoch=168, iter=62400/80000, loss=0.0962, lr=0.002560, batch_cost=1.0242, reader_cost=0.0002 | ETA 05:00:25 2020-11-03 13:15:55 [INFO] [TRAIN] epoch=169, iter=62500/80000, loss=0.1089, lr=0.002547, batch_cost=1.0083, reader_cost=0.0130 | ETA 04:54:05 2020-11-03 13:17:38 [INFO] [TRAIN] epoch=169, iter=62600/80000, loss=0.1170, lr=0.002534, batch_cost=1.0211, reader_cost=0.0002 | ETA 04:56:07 2020-11-03 13:19:18 [INFO] [TRAIN] epoch=169, iter=62700/80000, loss=0.1002, lr=0.002520, batch_cost=1.0035, reader_cost=0.0002 | ETA 04:49:20 2020-11-03 13:20:58 [INFO] [TRAIN] epoch=169, iter=62800/80000, loss=0.0912, lr=0.002507, batch_cost=1.0045, reader_cost=0.0004 | ETA 04:47:56 2020-11-03 13:22:44 [INFO] [TRAIN] epoch=170, iter=62900/80000, loss=0.0928, lr=0.002494, batch_cost=1.0526, reader_cost=0.0120 | ETA 04:59:59 2020-11-03 13:24:23 [INFO] [TRAIN] epoch=170, iter=63000/80000, loss=0.1021, lr=0.002481, batch_cost=0.9943, reader_cost=0.0002 | ETA 04:41:42 2020-11-03 13:26:04 [INFO] [TRAIN] epoch=170, iter=63100/80000, loss=0.0924, lr=0.002468, batch_cost=1.0064, reader_cost=0.0004 | ETA 04:43:27 2020-11-03 13:27:48 [INFO] [TRAIN] epoch=170, iter=63200/80000, loss=0.0991, lr=0.002455, batch_cost=1.0470, reader_cost=0.0005 | ETA 04:53:10 2020-11-03 13:29:29 [INFO] [TRAIN] epoch=171, iter=63300/80000, loss=0.1367, lr=0.002442, batch_cost=1.0099, reader_cost=0.0142 | ETA 04:41:06 2020-11-03 13:31:09 [INFO] [TRAIN] epoch=171, iter=63400/80000, loss=0.1203, lr=0.002429, batch_cost=0.9956, reader_cost=0.0002 | ETA 04:35:27 2020-11-03 13:32:53 [INFO] [TRAIN] epoch=171, iter=63500/80000, loss=0.0888, lr=0.002415, batch_cost=1.0415, reader_cost=0.0003 | ETA 04:46:24 2020-11-03 13:34:33 [INFO] [TRAIN] epoch=171, iter=63600/80000, loss=0.1084, lr=0.002402, batch_cost=1.0038, reader_cost=0.0002 | ETA 04:34:22 2020-11-03 13:36:17 [INFO] [TRAIN] epoch=172, iter=63700/80000, loss=0.1012, lr=0.002389, batch_cost=1.0352, reader_cost=0.0148 | ETA 04:41:14 2020-11-03 13:37:57 [INFO] [TRAIN] epoch=172, iter=63800/80000, loss=0.1067, lr=0.002376, batch_cost=1.0038, reader_cost=0.0002 | ETA 04:31:01 2020-11-03 13:39:37 [INFO] [TRAIN] epoch=172, iter=63900/80000, loss=0.0963, lr=0.002363, batch_cost=0.9954, reader_cost=0.0002 | ETA 04:27:05 2020-11-03 13:41:18 [INFO] [TRAIN] epoch=173, iter=64000/80000, loss=0.1148, lr=0.002349, batch_cost=1.0108, reader_cost=0.0136 | ETA 04:29:33 2020-11-03 13:41:22 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-03 13:45:52 [INFO] [EVAL] #Images=500 mIoU=0.7926 Acc=0.9619 Kappa=0.9505 2020-11-03 13:45:52 [INFO] [EVAL] Category IoU: [0.9812 0.8554 0.9266 0.5612 0.6233 0.6557 0.7142 0.7907 0.9254 0.6614 0.9452 0.8314 0.6544 0.9532 0.8258 0.8928 0.8031 0.6767 0.781 ] 2020-11-03 13:45:52 [INFO] [EVAL] Category Acc: [0.9929 0.9129 0.9605 0.8562 0.8397 0.7923 0.8204 0.9077 0.9539 0.8091 0.9655 0.8978 0.7811 0.9726 0.9255 0.9468 0.9667 0.8268 0.8556] 2020-11-03 13:45:54 [INFO] [EVAL] The model with the best validation mIoU (0.7926) was saved at iter 64000. 2020-11-03 13:47:32 [INFO] [TRAIN] epoch=173, iter=64100/80000, loss=0.1194, lr=0.002336, batch_cost=0.9868, reader_cost=0.0002 | ETA 04:21:29 2020-11-03 13:49:13 [INFO] [TRAIN] epoch=173, iter=64200/80000, loss=0.1032, lr=0.002323, batch_cost=1.0111, reader_cost=0.0002 | ETA 04:26:16 2020-11-03 13:50:55 [INFO] [TRAIN] epoch=173, iter=64300/80000, loss=0.0988, lr=0.002310, batch_cost=1.0155, reader_cost=0.0002 | ETA 04:25:42 2020-11-03 13:52:38 [INFO] [TRAIN] epoch=174, iter=64400/80000, loss=0.1033, lr=0.002296, batch_cost=1.0343, reader_cost=0.0127 | ETA 04:28:55 2020-11-03 13:54:19 [INFO] [TRAIN] epoch=174, iter=64500/80000, loss=0.1111, lr=0.002283, batch_cost=1.0011, reader_cost=0.0002 | ETA 04:18:37 2020-11-03 13:55:58 [INFO] [TRAIN] epoch=174, iter=64600/80000, loss=0.0967, lr=0.002270, batch_cost=0.9954, reader_cost=0.0002 | ETA 04:15:29 2020-11-03 13:57:40 [INFO] [TRAIN] epoch=174, iter=64700/80000, loss=0.0979, lr=0.002257, batch_cost=1.0233, reader_cost=0.0002 | ETA 04:20:57 2020-11-03 13:59:23 [INFO] [TRAIN] epoch=175, iter=64800/80000, loss=0.0970, lr=0.002243, batch_cost=1.0231, reader_cost=0.0164 | ETA 04:19:10 2020-11-03 14:01:02 [INFO] [TRAIN] epoch=175, iter=64900/80000, loss=0.1077, lr=0.002230, batch_cost=0.9925, reader_cost=0.0002 | ETA 04:09:46 2020-11-03 14:02:41 [INFO] [TRAIN] epoch=175, iter=65000/80000, loss=0.0896, lr=0.002217, batch_cost=0.9874, reader_cost=0.0002 | ETA 04:06:50 2020-11-03 14:04:21 [INFO] [TRAIN] epoch=175, iter=65100/80000, loss=0.0998, lr=0.002203, batch_cost=1.0002, reader_cost=0.0002 | ETA 04:08:22 2020-11-03 14:06:01 [INFO] [TRAIN] epoch=176, iter=65200/80000, loss=0.1000, lr=0.002190, batch_cost=0.9980, reader_cost=0.0118 | ETA 04:06:10 2020-11-03 14:07:41 [INFO] [TRAIN] epoch=176, iter=65300/80000, loss=0.1273, lr=0.002177, batch_cost=1.0076, reader_cost=0.0002 | ETA 04:06:52 2020-11-03 14:09:22 [INFO] [TRAIN] epoch=176, iter=65400/80000, loss=0.0971, lr=0.002164, batch_cost=1.0091, reader_cost=0.0002 | ETA 04:05:33 2020-11-03 14:11:04 [INFO] [TRAIN] epoch=177, iter=65500/80000, loss=0.1069, lr=0.002150, batch_cost=1.0193, reader_cost=0.0129 | ETA 04:06:19 2020-11-03 14:12:44 [INFO] [TRAIN] epoch=177, iter=65600/80000, loss=0.1091, lr=0.002137, batch_cost=0.9978, reader_cost=0.0002 | ETA 03:59:27 2020-11-03 14:14:26 [INFO] [TRAIN] epoch=177, iter=65700/80000, loss=0.1053, lr=0.002123, batch_cost=1.0188, reader_cost=0.0002 | ETA 04:02:49 2020-11-03 14:16:06 [INFO] [TRAIN] epoch=177, iter=65800/80000, loss=0.0998, lr=0.002110, batch_cost=1.0062, reader_cost=0.0005 | ETA 03:58:07 2020-11-03 14:17:47 [INFO] [TRAIN] epoch=178, iter=65900/80000, loss=0.0992, lr=0.002097, batch_cost=1.0075, reader_cost=0.0127 | ETA 03:56:45 2020-11-03 14:19:27 [INFO] [TRAIN] epoch=178, iter=66000/80000, loss=0.1172, lr=0.002083, batch_cost=0.9989, reader_cost=0.0002 | ETA 03:53:04 2020-11-03 14:21:07 [INFO] [TRAIN] epoch=178, iter=66100/80000, loss=0.0938, lr=0.002070, batch_cost=0.9974, reader_cost=0.0002 | ETA 03:51:04 2020-11-03 14:22:46 [INFO] [TRAIN] epoch=178, iter=66200/80000, loss=0.1017, lr=0.002057, batch_cost=0.9967, reader_cost=0.0002 | ETA 03:49:14 2020-11-03 14:24:28 [INFO] [TRAIN] epoch=179, iter=66300/80000, loss=0.1010, lr=0.002043, batch_cost=1.0131, reader_cost=0.0115 | ETA 03:51:18 2020-11-03 14:26:11 [INFO] [TRAIN] epoch=179, iter=66400/80000, loss=0.1077, lr=0.002030, batch_cost=1.0343, reader_cost=0.0002 | ETA 03:54:26 2020-11-03 14:27:52 [INFO] [TRAIN] epoch=179, iter=66500/80000, loss=0.0980, lr=0.002016, batch_cost=1.0116, reader_cost=0.0002 | ETA 03:47:36 2020-11-03 14:29:34 [INFO] [TRAIN] epoch=180, iter=66600/80000, loss=0.1134, lr=0.002003, batch_cost=1.0207, reader_cost=0.0119 | ETA 03:47:57 2020-11-03 14:31:17 [INFO] [TRAIN] epoch=180, iter=66700/80000, loss=0.1037, lr=0.001989, batch_cost=1.0253, reader_cost=0.0004 | ETA 03:47:15 2020-11-03 14:32:58 [INFO] [TRAIN] epoch=180, iter=66800/80000, loss=0.1073, lr=0.001976, batch_cost=1.0132, reader_cost=0.0002 | ETA 03:42:54 2020-11-03 14:34:41 [INFO] [TRAIN] epoch=180, iter=66900/80000, loss=0.0943, lr=0.001962, batch_cost=1.0309, reader_cost=0.0002 | ETA 03:45:04 2020-11-03 14:36:24 [INFO] [TRAIN] epoch=181, iter=67000/80000, loss=0.0983, lr=0.001949, batch_cost=1.0241, reader_cost=0.0140 | ETA 03:41:53 2020-11-03 14:38:04 [INFO] [TRAIN] epoch=181, iter=67100/80000, loss=0.1054, lr=0.001935, batch_cost=0.9987, reader_cost=0.0002 | ETA 03:34:43 2020-11-03 14:39:44 [INFO] [TRAIN] epoch=181, iter=67200/80000, loss=0.0892, lr=0.001922, batch_cost=1.0021, reader_cost=0.0003 | ETA 03:33:46 2020-11-03 14:41:25 [INFO] [TRAIN] epoch=181, iter=67300/80000, loss=0.0967, lr=0.001908, batch_cost=1.0105, reader_cost=0.0002 | ETA 03:33:52 2020-11-03 14:43:06 [INFO] [TRAIN] epoch=182, iter=67400/80000, loss=0.0932, lr=0.001895, batch_cost=1.0060, reader_cost=0.0134 | ETA 03:31:16 2020-11-03 14:44:50 [INFO] [TRAIN] epoch=182, iter=67500/80000, loss=0.1019, lr=0.001881, batch_cost=1.0428, reader_cost=0.0007 | ETA 03:37:14 2020-11-03 14:46:30 [INFO] [TRAIN] epoch=182, iter=67600/80000, loss=0.0909, lr=0.001868, batch_cost=1.0055, reader_cost=0.0013 | ETA 03:27:48 2020-11-03 14:48:12 [INFO] [TRAIN] epoch=182, iter=67700/80000, loss=0.1040, lr=0.001854, batch_cost=1.0122, reader_cost=0.0002 | ETA 03:27:30 2020-11-03 14:49:54 [INFO] [TRAIN] epoch=183, iter=67800/80000, loss=0.0885, lr=0.001841, batch_cost=1.0226, reader_cost=0.0134 | ETA 03:27:56 2020-11-03 14:51:34 [INFO] [TRAIN] epoch=183, iter=67900/80000, loss=0.1044, lr=0.001827, batch_cost=1.0001, reader_cost=0.0002 | ETA 03:21:41 2020-11-03 14:53:13 [INFO] [TRAIN] epoch=183, iter=68000/80000, loss=0.1045, lr=0.001813, batch_cost=0.9948, reader_cost=0.0002 | ETA 03:18:57 2020-11-03 14:54:57 [INFO] [TRAIN] epoch=184, iter=68100/80000, loss=0.0922, lr=0.001800, batch_cost=1.0358, reader_cost=0.0134 | ETA 03:25:26 2020-11-03 14:56:37 [INFO] [TRAIN] epoch=184, iter=68200/80000, loss=0.0918, lr=0.001786, batch_cost=1.0005, reader_cost=0.0002 | ETA 03:16:45 2020-11-03 14:58:17 [INFO] [TRAIN] epoch=184, iter=68300/80000, loss=0.0956, lr=0.001773, batch_cost=1.0034, reader_cost=0.0002 | ETA 03:15:40 2020-11-03 14:59:57 [INFO] [TRAIN] epoch=184, iter=68400/80000, loss=0.0936, lr=0.001759, batch_cost=1.0006, reader_cost=0.0004 | ETA 03:13:26 2020-11-03 15:01:38 [INFO] [TRAIN] epoch=185, iter=68500/80000, loss=0.1052, lr=0.001745, batch_cost=1.0086, reader_cost=0.0124 | ETA 03:13:18 2020-11-03 15:03:18 [INFO] [TRAIN] epoch=185, iter=68600/80000, loss=0.1032, lr=0.001732, batch_cost=0.9975, reader_cost=0.0002 | ETA 03:09:31 2020-11-03 15:05:01 [INFO] [TRAIN] epoch=185, iter=68700/80000, loss=0.0946, lr=0.001718, batch_cost=1.0308, reader_cost=0.0002 | ETA 03:14:07 2020-11-03 15:06:42 [INFO] [TRAIN] epoch=185, iter=68800/80000, loss=0.1038, lr=0.001704, batch_cost=1.0035, reader_cost=0.0002 | ETA 03:07:18 2020-11-03 15:08:23 [INFO] [TRAIN] epoch=186, iter=68900/80000, loss=0.0964, lr=0.001691, batch_cost=1.0157, reader_cost=0.0146 | ETA 03:07:54 2020-11-03 15:10:08 [INFO] [TRAIN] epoch=186, iter=69000/80000, loss=0.1146, lr=0.001677, batch_cost=1.0483, reader_cost=0.0002 | ETA 03:12:11 2020-11-03 15:11:48 [INFO] [TRAIN] epoch=186, iter=69100/80000, loss=0.1028, lr=0.001663, batch_cost=0.9998, reader_cost=0.0002 | ETA 03:01:37 2020-11-03 15:13:29 [INFO] [TRAIN] epoch=187, iter=69200/80000, loss=0.1027, lr=0.001649, batch_cost=1.0131, reader_cost=0.0141 | ETA 03:02:21 2020-11-03 15:15:27 [INFO] [TRAIN] epoch=187, iter=69300/80000, loss=0.0967, lr=0.001636, batch_cost=1.1777, reader_cost=0.0004 | ETA 03:30:01 2020-11-03 15:17:34 [INFO] [TRAIN] epoch=187, iter=69400/80000, loss=0.1017, lr=0.001622, batch_cost=1.2658, reader_cost=0.0003 | ETA 03:43:37 2020-11-03 15:19:24 [INFO] [TRAIN] epoch=187, iter=69500/80000, loss=0.0951, lr=0.001608, batch_cost=1.1012, reader_cost=0.0002 | ETA 03:12:42 2020-11-03 15:21:07 [INFO] [TRAIN] epoch=188, iter=69600/80000, loss=0.0953, lr=0.001594, batch_cost=1.0330, reader_cost=0.0200 | ETA 02:59:03 2020-11-03 15:22:46 [INFO] [TRAIN] epoch=188, iter=69700/80000, loss=0.1054, lr=0.001581, batch_cost=0.9935, reader_cost=0.0002 | ETA 02:50:32 2020-11-03 15:24:26 [INFO] [TRAIN] epoch=188, iter=69800/80000, loss=0.0921, lr=0.001567, batch_cost=0.9941, reader_cost=0.0002 | ETA 02:48:59 2020-11-03 15:26:06 [INFO] [TRAIN] epoch=188, iter=69900/80000, loss=0.1145, lr=0.001553, batch_cost=1.0028, reader_cost=0.0002 | ETA 02:48:48 2020-11-03 15:27:48 [INFO] [TRAIN] epoch=189, iter=70000/80000, loss=0.0926, lr=0.001539, batch_cost=1.0199, reader_cost=0.0165 | ETA 02:49:59 2020-11-03 15:29:29 [INFO] [TRAIN] epoch=189, iter=70100/80000, loss=0.0989, lr=0.001525, batch_cost=1.0118, reader_cost=0.0002 | ETA 02:46:57 2020-11-03 15:31:13 [INFO] [TRAIN] epoch=189, iter=70200/80000, loss=0.0863, lr=0.001511, batch_cost=1.0396, reader_cost=0.0010 | ETA 02:49:47 2020-11-03 15:32:54 [INFO] [TRAIN] epoch=189, iter=70300/80000, loss=0.1063, lr=0.001497, batch_cost=1.0032, reader_cost=0.0002 | ETA 02:42:11 2020-11-03 15:34:37 [INFO] [TRAIN] epoch=190, iter=70400/80000, loss=0.0959, lr=0.001484, batch_cost=1.0367, reader_cost=0.0125 | ETA 02:45:52 2020-11-03 15:36:18 [INFO] [TRAIN] epoch=190, iter=70500/80000, loss=0.1004, lr=0.001470, batch_cost=1.0078, reader_cost=0.0002 | ETA 02:39:33 2020-11-03 15:38:01 [INFO] [TRAIN] epoch=190, iter=70600/80000, loss=0.0894, lr=0.001456, batch_cost=1.0340, reader_cost=0.0002 | ETA 02:41:59 2020-11-03 15:39:42 [INFO] [TRAIN] epoch=191, iter=70700/80000, loss=0.1069, lr=0.001442, batch_cost=1.0067, reader_cost=0.0129 | ETA 02:36:01 2020-11-03 15:41:23 [INFO] [TRAIN] epoch=191, iter=70800/80000, loss=0.0949, lr=0.001428, batch_cost=1.0057, reader_cost=0.0030 | ETA 02:34:12 2020-11-03 15:43:03 [INFO] [TRAIN] epoch=191, iter=70900/80000, loss=0.1039, lr=0.001414, batch_cost=1.0075, reader_cost=0.0002 | ETA 02:32:47 2020-11-03 15:44:43 [INFO] [TRAIN] epoch=191, iter=71000/80000, loss=0.0935, lr=0.001400, batch_cost=0.9987, reader_cost=0.0002 | ETA 02:29:48 2020-11-03 15:46:25 [INFO] [TRAIN] epoch=192, iter=71100/80000, loss=0.0979, lr=0.001386, batch_cost=1.0198, reader_cost=0.0128 | ETA 02:31:16 2020-11-03 15:48:05 [INFO] [TRAIN] epoch=192, iter=71200/80000, loss=0.1075, lr=0.001372, batch_cost=1.0027, reader_cost=0.0002 | ETA 02:27:03 2020-11-03 15:49:45 [INFO] [TRAIN] epoch=192, iter=71300/80000, loss=0.0891, lr=0.001358, batch_cost=0.9984, reader_cost=0.0002 | ETA 02:24:45 2020-11-03 15:51:26 [INFO] [TRAIN] epoch=192, iter=71400/80000, loss=0.0967, lr=0.001344, batch_cost=1.0065, reader_cost=0.0002 | ETA 02:24:16 2020-11-03 15:53:07 [INFO] [TRAIN] epoch=193, iter=71500/80000, loss=0.0948, lr=0.001330, batch_cost=1.0144, reader_cost=0.0179 | ETA 02:23:42 2020-11-03 15:54:49 [INFO] [TRAIN] epoch=193, iter=71600/80000, loss=0.1028, lr=0.001316, batch_cost=1.0109, reader_cost=0.0002 | ETA 02:21:31 2020-11-03 15:56:28 [INFO] [TRAIN] epoch=193, iter=71700/80000, loss=0.0966, lr=0.001301, batch_cost=0.9902, reader_cost=0.0002 | ETA 02:16:58 2020-11-03 15:58:08 [INFO] [TRAIN] epoch=194, iter=71800/80000, loss=0.1062, lr=0.001287, batch_cost=1.0077, reader_cost=0.0123 | ETA 02:17:43 2020-11-03 15:59:49 [INFO] [TRAIN] epoch=194, iter=71900/80000, loss=0.0991, lr=0.001273, batch_cost=1.0039, reader_cost=0.0002 | ETA 02:15:31 2020-11-03 16:01:31 [INFO] [TRAIN] epoch=194, iter=72000/80000, loss=0.0890, lr=0.001259, batch_cost=1.0258, reader_cost=0.0002 | ETA 02:16:46 2020-11-03 16:01:35 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-03 16:06:06 [INFO] [EVAL] #Images=500 mIoU=0.7956 Acc=0.9625 Kappa=0.9513 2020-11-03 16:06:06 [INFO] [EVAL] Category IoU: [0.9821 0.8593 0.9283 0.5873 0.6242 0.6599 0.7154 0.7785 0.925 0.6521 0.9484 0.8319 0.6437 0.9534 0.8121 0.8875 0.8581 0.6834 0.7851] 2020-11-03 16:06:06 [INFO] [EVAL] Category Acc: [0.9923 0.9208 0.9616 0.8606 0.8362 0.7903 0.8271 0.8861 0.9526 0.8296 0.969 0.9004 0.7717 0.9734 0.949 0.947 0.9474 0.8479 0.8672] 2020-11-03 16:06:09 [INFO] [EVAL] The model with the best validation mIoU (0.7956) was saved at iter 72000. 2020-11-03 16:07:51 [INFO] [TRAIN] epoch=194, iter=72100/80000, loss=0.0978, lr=0.001245, batch_cost=1.0247, reader_cost=0.0003 | ETA 02:14:55 2020-11-03 16:09:38 [INFO] [TRAIN] epoch=195, iter=72200/80000, loss=0.1011, lr=0.001231, batch_cost=1.0656, reader_cost=0.0155 | ETA 02:18:31 2020-11-03 16:11:22 [INFO] [TRAIN] epoch=195, iter=72300/80000, loss=0.0982, lr=0.001216, batch_cost=1.0461, reader_cost=0.0002 | ETA 02:14:14 2020-11-03 16:13:06 [INFO] [TRAIN] epoch=195, iter=72400/80000, loss=0.0929, lr=0.001202, batch_cost=1.0404, reader_cost=0.0002 | ETA 02:11:46 2020-11-03 16:14:48 [INFO] [TRAIN] epoch=195, iter=72500/80000, loss=0.1023, lr=0.001188, batch_cost=1.0111, reader_cost=0.0002 | ETA 02:06:23 2020-11-03 16:16:30 [INFO] [TRAIN] epoch=196, iter=72600/80000, loss=0.0982, lr=0.001174, batch_cost=1.0275, reader_cost=0.0125 | ETA 02:06:43 2020-11-03 16:18:14 [INFO] [TRAIN] epoch=196, iter=72700/80000, loss=0.1085, lr=0.001159, batch_cost=1.0330, reader_cost=0.0006 | ETA 02:05:40 2020-11-03 16:19:56 [INFO] [TRAIN] epoch=196, iter=72800/80000, loss=0.0908, lr=0.001145, batch_cost=1.0236, reader_cost=0.0009 | ETA 02:02:49 2020-11-03 16:21:42 [INFO] [TRAIN] epoch=196, iter=72900/80000, loss=0.1042, lr=0.001131, batch_cost=1.0644, reader_cost=0.0004 | ETA 02:05:57 2020-11-03 16:23:26 [INFO] [TRAIN] epoch=197, iter=73000/80000, loss=0.0962, lr=0.001117, batch_cost=1.0392, reader_cost=0.0152 | ETA 02:01:14 2020-11-03 16:25:15 [INFO] [TRAIN] epoch=197, iter=73100/80000, loss=0.1075, lr=0.001102, batch_cost=1.0863, reader_cost=0.0003 | ETA 02:04:55 2020-11-03 16:27:10 [INFO] [TRAIN] epoch=197, iter=73200/80000, loss=0.0926, lr=0.001088, batch_cost=1.1494, reader_cost=0.0008 | ETA 02:10:15 2020-11-03 16:29:12 [INFO] [TRAIN] epoch=198, iter=73300/80000, loss=0.0960, lr=0.001073, batch_cost=1.2231, reader_cost=0.0602 | ETA 02:16:34 2020-11-03 16:31:09 [INFO] [TRAIN] epoch=198, iter=73400/80000, loss=0.0966, lr=0.001059, batch_cost=1.1695, reader_cost=0.0005 | ETA 02:08:39 2020-11-03 16:33:10 [INFO] [TRAIN] epoch=198, iter=73500/80000, loss=0.0992, lr=0.001044, batch_cost=1.2067, reader_cost=0.0004 | ETA 02:10:43 2020-11-03 16:35:20 [INFO] [TRAIN] epoch=198, iter=73600/80000, loss=0.0885, lr=0.001030, batch_cost=1.2992, reader_cost=0.0004 | ETA 02:18:34 2020-11-03 16:37:33 [INFO] [TRAIN] epoch=199, iter=73700/80000, loss=0.0901, lr=0.001016, batch_cost=1.3341, reader_cost=0.0396 | ETA 02:20:05 2020-11-03 16:39:42 [INFO] [TRAIN] epoch=199, iter=73800/80000, loss=0.1098, lr=0.001001, batch_cost=1.2917, reader_cost=0.0005 | ETA 02:13:28 2020-11-03 16:41:47 [INFO] [TRAIN] epoch=199, iter=73900/80000, loss=0.0915, lr=0.000986, batch_cost=1.2486, reader_cost=0.0005 | ETA 02:06:56 2020-11-03 16:43:49 [INFO] [TRAIN] epoch=199, iter=74000/80000, loss=0.1007, lr=0.000972, batch_cost=1.2209, reader_cost=0.0014 | ETA 02:02:05 2020-11-03 16:45:59 [INFO] [TRAIN] epoch=200, iter=74100/80000, loss=0.0907, lr=0.000957, batch_cost=1.3004, reader_cost=0.0414 | ETA 02:07:52 2020-11-03 16:48:00 [INFO] [TRAIN] epoch=200, iter=74200/80000, loss=0.1017, lr=0.000943, batch_cost=1.2071, reader_cost=0.0004 | ETA 01:56:41 2020-11-03 16:49:53 [INFO] [TRAIN] epoch=200, iter=74300/80000, loss=0.0909, lr=0.000928, batch_cost=1.1310, reader_cost=0.0005 | ETA 01:47:26 2020-11-03 16:51:57 [INFO] [TRAIN] epoch=200, iter=74400/80000, loss=0.1032, lr=0.000913, batch_cost=1.2413, reader_cost=0.0007 | ETA 01:55:51 2020-11-03 16:53:49 [INFO] [TRAIN] epoch=201, iter=74500/80000, loss=0.0966, lr=0.000899, batch_cost=1.1123, reader_cost=0.0425 | ETA 01:41:57 2020-11-03 16:55:31 [INFO] [TRAIN] epoch=201, iter=74600/80000, loss=0.0987, lr=0.000884, batch_cost=1.0256, reader_cost=0.0002 | ETA 01:32:18 2020-11-03 16:57:11 [INFO] [TRAIN] epoch=201, iter=74700/80000, loss=0.0909, lr=0.000869, batch_cost=1.0036, reader_cost=0.0002 | ETA 01:28:38 2020-11-03 16:58:53 [INFO] [TRAIN] epoch=202, iter=74800/80000, loss=0.0946, lr=0.000854, batch_cost=1.0195, reader_cost=0.0133 | ETA 01:28:21 2020-11-03 17:00:42 [INFO] [TRAIN] epoch=202, iter=74900/80000, loss=0.1066, lr=0.000840, batch_cost=1.0860, reader_cost=0.0003 | ETA 01:32:18 2020-11-03 17:02:26 [INFO] [TRAIN] epoch=202, iter=75000/80000, loss=0.0931, lr=0.000825, batch_cost=1.0380, reader_cost=0.0006 | ETA 01:26:29 2020-11-03 17:04:07 [INFO] [TRAIN] epoch=202, iter=75100/80000, loss=0.0967, lr=0.000810, batch_cost=1.0152, reader_cost=0.0002 | ETA 01:22:54 2020-11-03 17:05:51 [INFO] [TRAIN] epoch=203, iter=75200/80000, loss=0.0999, lr=0.000795, batch_cost=1.0365, reader_cost=0.0148 | ETA 01:22:55 2020-11-03 17:07:35 [INFO] [TRAIN] epoch=203, iter=75300/80000, loss=0.0983, lr=0.000780, batch_cost=1.0355, reader_cost=0.0002 | ETA 01:21:06 2020-11-03 17:09:24 [INFO] [TRAIN] epoch=203, iter=75400/80000, loss=0.0903, lr=0.000765, batch_cost=1.0993, reader_cost=0.0004 | ETA 01:24:16 2020-11-03 17:11:06 [INFO] [TRAIN] epoch=203, iter=75500/80000, loss=0.0937, lr=0.000750, batch_cost=1.0172, reader_cost=0.0003 | ETA 01:16:17 2020-11-03 17:12:48 [INFO] [TRAIN] epoch=204, iter=75600/80000, loss=0.0890, lr=0.000735, batch_cost=1.0176, reader_cost=0.0132 | ETA 01:14:37 2020-11-03 17:14:30 [INFO] [TRAIN] epoch=204, iter=75700/80000, loss=0.1059, lr=0.000720, batch_cost=1.0168, reader_cost=0.0002 | ETA 01:12:52 2020-11-03 17:16:13 [INFO] [TRAIN] epoch=204, iter=75800/80000, loss=0.1007, lr=0.000705, batch_cost=1.0328, reader_cost=0.0012 | ETA 01:12:17 2020-11-03 17:18:18 [INFO] [TRAIN] epoch=205, iter=75900/80000, loss=0.0922, lr=0.000690, batch_cost=1.2510, reader_cost=0.0154 | ETA 01:25:29 2020-11-03 17:20:07 [INFO] [TRAIN] epoch=205, iter=76000/80000, loss=0.0939, lr=0.000675, batch_cost=1.0928, reader_cost=0.0029 | ETA 01:12:51 2020-11-03 17:21:57 [INFO] [TRAIN] epoch=205, iter=76100/80000, loss=0.0965, lr=0.000660, batch_cost=1.0960, reader_cost=0.0005 | ETA 01:11:14 2020-11-03 17:23:41 [INFO] [TRAIN] epoch=205, iter=76200/80000, loss=0.0891, lr=0.000644, batch_cost=1.0377, reader_cost=0.0002 | ETA 01:05:43 2020-11-03 17:25:34 [INFO] [TRAIN] epoch=206, iter=76300/80000, loss=0.1052, lr=0.000629, batch_cost=1.1339, reader_cost=0.0174 | ETA 01:09:55 2020-11-03 17:27:28 [INFO] [TRAIN] epoch=206, iter=76400/80000, loss=0.1076, lr=0.000614, batch_cost=1.1388, reader_cost=0.0002 | ETA 01:08:19 2020-11-03 17:29:36 [INFO] [TRAIN] epoch=206, iter=76500/80000, loss=0.0983, lr=0.000598, batch_cost=1.2840, reader_cost=0.0002 | ETA 01:14:54 2020-11-03 17:31:37 [INFO] [TRAIN] epoch=206, iter=76600/80000, loss=0.0960, lr=0.000583, batch_cost=1.2028, reader_cost=0.0002 | ETA 01:08:09 2020-11-03 17:33:45 [INFO] [TRAIN] epoch=207, iter=76700/80000, loss=0.0893, lr=0.000568, batch_cost=1.2797, reader_cost=0.0144 | ETA 01:10:22 2020-11-03 17:35:55 [INFO] [TRAIN] epoch=207, iter=76800/80000, loss=0.1181, lr=0.000552, batch_cost=1.3012, reader_cost=0.0002 | ETA 01:09:23 2020-11-03 17:37:38 [INFO] [TRAIN] epoch=207, iter=76900/80000, loss=0.0962, lr=0.000537, batch_cost=1.0315, reader_cost=0.0009 | ETA 00:53:17 2020-11-03 17:39:20 [INFO] [TRAIN] epoch=207, iter=77000/80000, loss=0.1032, lr=0.000521, batch_cost=1.0251, reader_cost=0.0002 | ETA 00:51:15 2020-11-03 17:41:20 [INFO] [TRAIN] epoch=208, iter=77100/80000, loss=0.0933, lr=0.000505, batch_cost=1.1942, reader_cost=0.0142 | ETA 00:57:43 2020-11-03 17:43:33 [INFO] [TRAIN] epoch=208, iter=77200/80000, loss=0.1016, lr=0.000490, batch_cost=1.3298, reader_cost=0.0002 | ETA 01:02:03 2020-11-03 17:45:44 [INFO] [TRAIN] epoch=208, iter=77300/80000, loss=0.0935, lr=0.000474, batch_cost=1.3150, reader_cost=0.0002 | ETA 00:59:10 2020-11-03 17:47:25 [INFO] [TRAIN] epoch=209, iter=77400/80000, loss=0.0979, lr=0.000458, batch_cost=1.0094, reader_cost=0.0146 | ETA 00:43:44 2020-11-03 17:49:04 [INFO] [TRAIN] epoch=209, iter=77500/80000, loss=0.0991, lr=0.000442, batch_cost=0.9855, reader_cost=0.0002 | ETA 00:41:03 2020-11-03 17:50:43 [INFO] [TRAIN] epoch=209, iter=77600/80000, loss=0.0966, lr=0.000426, batch_cost=0.9907, reader_cost=0.0002 | ETA 00:39:37 2020-11-03 17:52:23 [INFO] [TRAIN] epoch=209, iter=77700/80000, loss=0.0861, lr=0.000410, batch_cost=1.0037, reader_cost=0.0003 | ETA 00:38:28 2020-11-03 17:54:04 [INFO] [TRAIN] epoch=210, iter=77800/80000, loss=0.0904, lr=0.000394, batch_cost=1.0027, reader_cost=0.0150 | ETA 00:36:46 2020-11-03 17:55:43 [INFO] [TRAIN] epoch=210, iter=77900/80000, loss=0.0986, lr=0.000378, batch_cost=0.9911, reader_cost=0.0004 | ETA 00:34:41 2020-11-03 17:57:28 [INFO] [TRAIN] epoch=210, iter=78000/80000, loss=0.0928, lr=0.000362, batch_cost=1.0555, reader_cost=0.0002 | ETA 00:35:10 2020-11-03 17:59:09 [INFO] [TRAIN] epoch=210, iter=78100/80000, loss=0.0876, lr=0.000345, batch_cost=1.0045, reader_cost=0.0002 | ETA 00:31:48 2020-11-03 18:00:51 [INFO] [TRAIN] epoch=211, iter=78200/80000, loss=0.0881, lr=0.000329, batch_cost=1.0214, reader_cost=0.0121 | ETA 00:30:38 2020-11-03 18:02:31 [INFO] [TRAIN] epoch=211, iter=78300/80000, loss=0.1043, lr=0.000313, batch_cost=0.9979, reader_cost=0.0003 | ETA 00:28:16 2020-11-03 18:04:10 [INFO] [TRAIN] epoch=211, iter=78400/80000, loss=0.0863, lr=0.000296, batch_cost=0.9913, reader_cost=0.0002 | ETA 00:26:26 2020-11-03 18:05:50 [INFO] [TRAIN] epoch=212, iter=78500/80000, loss=0.0966, lr=0.000279, batch_cost=1.0063, reader_cost=0.0128 | ETA 00:25:09 2020-11-03 18:07:30 [INFO] [TRAIN] epoch=212, iter=78600/80000, loss=0.0916, lr=0.000262, batch_cost=0.9995, reader_cost=0.0002 | ETA 00:23:19 2020-11-03 18:09:11 [INFO] [TRAIN] epoch=212, iter=78700/80000, loss=0.0985, lr=0.000246, batch_cost=1.0090, reader_cost=0.0002 | ETA 00:21:51 2020-11-03 18:10:51 [INFO] [TRAIN] epoch=212, iter=78800/80000, loss=0.0895, lr=0.000228, batch_cost=1.0022, reader_cost=0.0002 | ETA 00:20:02 2020-11-03 18:12:33 [INFO] [TRAIN] epoch=213, iter=78900/80000, loss=0.1023, lr=0.000211, batch_cost=1.0148, reader_cost=0.0122 | ETA 00:18:36 2020-11-03 18:14:15 [INFO] [TRAIN] epoch=213, iter=79000/80000, loss=0.0980, lr=0.000194, batch_cost=1.0205, reader_cost=0.0002 | ETA 00:17:00 2020-11-03 18:15:56 [INFO] [TRAIN] epoch=213, iter=79100/80000, loss=0.0935, lr=0.000176, batch_cost=1.0113, reader_cost=0.0002 | ETA 00:15:10 2020-11-03 18:17:37 [INFO] [TRAIN] epoch=213, iter=79200/80000, loss=0.0982, lr=0.000159, batch_cost=1.0128, reader_cost=0.0002 | ETA 00:13:30 2020-11-03 18:19:18 [INFO] [TRAIN] epoch=214, iter=79300/80000, loss=0.0917, lr=0.000141, batch_cost=1.0058, reader_cost=0.0131 | ETA 00:11:44 2020-11-03 18:20:58 [INFO] [TRAIN] epoch=214, iter=79400/80000, loss=0.1032, lr=0.000123, batch_cost=1.0049, reader_cost=0.0002 | ETA 00:10:02 2020-11-03 18:22:39 [INFO] [TRAIN] epoch=214, iter=79500/80000, loss=0.0927, lr=0.000104, batch_cost=1.0089, reader_cost=0.0003 | ETA 00:08:24 2020-11-03 18:24:23 [INFO] [TRAIN] epoch=214, iter=79600/80000, loss=0.0985, lr=0.000085, batch_cost=1.0352, reader_cost=0.0002 | ETA 00:06:54 2020-11-03 18:26:04 [INFO] [TRAIN] epoch=215, iter=79700/80000, loss=0.0895, lr=0.000066, batch_cost=1.0085, reader_cost=0.0152 | ETA 00:05:02 2020-11-03 18:27:47 [INFO] [TRAIN] epoch=215, iter=79800/80000, loss=0.0933, lr=0.000046, batch_cost=1.0289, reader_cost=0.0002 | ETA 00:03:25 2020-11-03 18:29:28 [INFO] [TRAIN] epoch=215, iter=79900/80000, loss=0.0950, lr=0.000025, batch_cost=1.0135, reader_cost=0.0002 | ETA 00:01:41 2020-11-03 18:31:11 [INFO] [TRAIN] epoch=216, iter=80000/80000, loss=0.1044, lr=0.000000, batch_cost=1.0318, reader_cost=0.0143 | ETA 00:00:00 2020-11-03 18:31:16 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-03 18:35:44 [INFO] [EVAL] #Images=500 mIoU=0.7990 Acc=0.9633 Kappa=0.9523 2020-11-03 18:35:44 [INFO] [EVAL] Category IoU: [0.983 0.8641 0.9295 0.5896 0.6286 0.6603 0.718 0.7862 0.9258 0.6491 0.948 0.8328 0.6469 0.9543 0.8283 0.9063 0.8564 0.687 0.7874] 2020-11-03 18:35:44 [INFO] [EVAL] Category Acc: [0.9924 0.9234 0.9609 0.8588 0.8319 0.8042 0.8299 0.8987 0.9537 0.8487 0.9678 0.8966 0.7813 0.9738 0.9484 0.9643 0.9415 0.8144 0.8745] 2020-11-03 18:35:46 [INFO] [EVAL] The model with the best validation mIoU (0.7990) was saved at iter 80000.