2020-11-02 18:58:20 [INFO] ------------Environment Information------------- platform: Linux-3.10.0-1062.18.1.el7.x86_64-x86_64-with-centos-7.7.1908-Core Python: 3.7.9 (default, Aug 31 2020, 12:42:55) [GCC 7.3.0] Paddle compiled with cuda: True NVCC: Cuda compilation tools, release 10.2, V10.2.89 cudnn: 7.6 GPUs used: 4 CUDA_VISIBLE_DEVICES: 4,5,6,7 GPU: ['GPU 0: Tesla V100-SXM2-16GB', 'GPU 1: Tesla V100-SXM2-16GB', 'GPU 2: Tesla V100-SXM2-16GB', 'GPU 3: Tesla V100-SXM2-16GB', 'GPU 4: Tesla V100-SXM2-16GB', 'GPU 5: Tesla V100-SXM2-16GB', 'GPU 6: Tesla V100-SXM2-16GB', 'GPU 7: Tesla V100-SXM2-16GB'] GCC: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39) PaddlePaddle: 2.0.0-rc0 OpenCV: 4.1.1 ------------------------------------------------ 2020-11-02 18:58:20 [INFO] ---------------Config Information--------------- batch_size: 2 iters: 80000 learning_rate: decay: end_lr: 0.0 power: 0.9 type: poly value: 0.01 loss: coef: - 1 types: - ignore_index: 255 type: CrossEntropyLoss model: align_corners: false aspp_out_channels: 256 aspp_ratios: - 1 - 12 - 24 - 36 backbone: multi_grid: - 1 - 2 - 4 output_stride: 8 pretrained: https://bj.bcebos.com/paddleseg/dygraph/resnet101_vd_ssld.tar.gz type: ResNet101_vd backbone_indices: - 3 pretrained: null type: DeepLabV3 optimizer: momentum: 0.9 type: sgd weight_decay: 4.0e-05 train_dataset: dataset_root: data/cityscapes mode: train transforms: - max_scale_factor: 2.0 min_scale_factor: 0.5 scale_step_size: 0.25 type: ResizeStepScaling - crop_size: - 1024 - 512 type: RandomPaddingCrop - type: RandomHorizontalFlip - brightness_range: 0.4 contrast_range: 0.4 saturation_range: 0.4 type: RandomDistort - type: Normalize type: Cityscapes val_dataset: dataset_root: data/cityscapes mode: val transforms: - type: Normalize type: Cityscapes ------------------------------------------------ 2020-11-02 18:58:24 [INFO] Loading pretrained model from https://bj.bcebos.com/paddleseg/dygraph/resnet101_vd_ssld.tar.gz 2020-11-02 18:58:26 [INFO] There are 530/530 variables loaded into ResNet_vd. 2020-11-02 19:00:49 [INFO] [TRAIN] epoch=1, iter=100/80000, loss=1.0555, lr=0.009989, batch_cost=1.3736, reader_cost=0.0165 | ETA 30:29:14 2020-11-02 19:03:02 [INFO] [TRAIN] epoch=1, iter=200/80000, loss=0.6641, lr=0.009978, batch_cost=1.3259, reader_cost=0.0003 | ETA 29:23:24 2020-11-02 19:05:11 [INFO] [TRAIN] epoch=1, iter=300/80000, loss=0.6400, lr=0.009966, batch_cost=1.2915, reader_cost=0.0001 | ETA 28:35:32 2020-11-02 19:07:17 [INFO] [TRAIN] epoch=2, iter=400/80000, loss=0.5403, lr=0.009955, batch_cost=1.2531, reader_cost=0.0098 | ETA 27:42:28 2020-11-02 19:09:21 [INFO] [TRAIN] epoch=2, iter=500/80000, loss=0.5283, lr=0.009944, batch_cost=1.2465, reader_cost=0.0003 | ETA 27:31:35 2020-11-02 19:11:29 [INFO] [TRAIN] epoch=2, iter=600/80000, loss=0.3802, lr=0.009933, batch_cost=1.2816, reader_cost=0.0016 | ETA 28:15:57 2020-11-02 19:13:37 [INFO] [TRAIN] epoch=2, iter=700/80000, loss=0.4017, lr=0.009921, batch_cost=1.2719, reader_cost=0.0003 | ETA 28:00:58 2020-11-02 19:15:49 [INFO] [TRAIN] epoch=3, iter=800/80000, loss=0.4192, lr=0.009910, batch_cost=1.3241, reader_cost=0.0102 | ETA 29:07:46 2020-11-02 19:17:56 [INFO] [TRAIN] epoch=3, iter=900/80000, loss=0.3302, lr=0.009899, batch_cost=1.2718, reader_cost=0.0002 | ETA 27:56:36 2020-11-02 19:20:06 [INFO] [TRAIN] epoch=3, iter=1000/80000, loss=0.4376, lr=0.009888, batch_cost=1.2974, reader_cost=0.0001 | ETA 28:28:16 2020-11-02 19:22:13 [INFO] [TRAIN] epoch=3, iter=1100/80000, loss=0.3795, lr=0.009876, batch_cost=1.2665, reader_cost=0.0003 | ETA 27:45:28 2020-11-02 19:24:23 [INFO] [TRAIN] epoch=4, iter=1200/80000, loss=0.3275, lr=0.009865, batch_cost=1.3018, reader_cost=0.0116 | ETA 28:29:39 2020-11-02 19:26:32 [INFO] [TRAIN] epoch=4, iter=1300/80000, loss=0.2889, lr=0.009854, batch_cost=1.2901, reader_cost=0.0002 | ETA 28:12:10 2020-11-02 19:28:47 [INFO] [TRAIN] epoch=4, iter=1400/80000, loss=0.4084, lr=0.009842, batch_cost=1.3553, reader_cost=0.0010 | ETA 29:35:23 2020-11-02 19:30:58 [INFO] [TRAIN] epoch=5, iter=1500/80000, loss=0.3086, lr=0.009831, batch_cost=1.3037, reader_cost=0.0091 | ETA 28:25:38 2020-11-02 19:33:07 [INFO] [TRAIN] epoch=5, iter=1600/80000, loss=0.2626, lr=0.009820, batch_cost=1.2968, reader_cost=0.0003 | ETA 28:14:29 2020-11-02 19:35:19 [INFO] [TRAIN] epoch=5, iter=1700/80000, loss=0.2950, lr=0.009809, batch_cost=1.3178, reader_cost=0.0001 | ETA 28:39:42 2020-11-02 19:37:27 [INFO] [TRAIN] epoch=5, iter=1800/80000, loss=0.3579, lr=0.009797, batch_cost=1.2818, reader_cost=0.0005 | ETA 27:50:34 2020-11-02 19:39:35 [INFO] [TRAIN] epoch=6, iter=1900/80000, loss=0.3800, lr=0.009786, batch_cost=1.2763, reader_cost=0.0112 | ETA 27:41:21 2020-11-02 19:41:41 [INFO] [TRAIN] epoch=6, iter=2000/80000, loss=0.2427, lr=0.009775, batch_cost=1.2653, reader_cost=0.0002 | ETA 27:24:51 2020-11-02 19:43:48 [INFO] [TRAIN] epoch=6, iter=2100/80000, loss=0.2596, lr=0.009764, batch_cost=1.2667, reader_cost=0.0003 | ETA 27:24:38 2020-11-02 19:45:57 [INFO] [TRAIN] epoch=6, iter=2200/80000, loss=0.3210, lr=0.009752, batch_cost=1.2902, reader_cost=0.0002 | ETA 27:53:00 2020-11-02 19:48:05 [INFO] [TRAIN] epoch=7, iter=2300/80000, loss=0.3090, lr=0.009741, batch_cost=1.2824, reader_cost=0.0121 | ETA 27:40:44 2020-11-02 19:50:11 [INFO] [TRAIN] epoch=7, iter=2400/80000, loss=0.2670, lr=0.009730, batch_cost=1.2588, reader_cost=0.0003 | ETA 27:08:04 2020-11-02 19:52:20 [INFO] [TRAIN] epoch=7, iter=2500/80000, loss=0.3179, lr=0.009718, batch_cost=1.2863, reader_cost=0.0001 | ETA 27:41:30 2020-11-02 19:54:29 [INFO] [TRAIN] epoch=7, iter=2600/80000, loss=0.2948, lr=0.009707, batch_cost=1.2963, reader_cost=0.0002 | ETA 27:52:14 2020-11-02 19:56:39 [INFO] [TRAIN] epoch=8, iter=2700/80000, loss=0.2383, lr=0.009696, batch_cost=1.2970, reader_cost=0.0097 | ETA 27:50:54 2020-11-02 19:58:56 [INFO] [TRAIN] epoch=8, iter=2800/80000, loss=0.2120, lr=0.009685, batch_cost=1.3650, reader_cost=0.0002 | ETA 29:16:15 2020-11-02 20:01:05 [INFO] [TRAIN] epoch=8, iter=2900/80000, loss=0.2446, lr=0.009673, batch_cost=1.2944, reader_cost=0.0003 | ETA 27:43:21 2020-11-02 20:03:15 [INFO] [TRAIN] epoch=9, iter=3000/80000, loss=0.2336, lr=0.009662, batch_cost=1.2969, reader_cost=0.0112 | ETA 27:44:20 2020-11-02 20:05:26 [INFO] [TRAIN] epoch=9, iter=3100/80000, loss=0.2188, lr=0.009651, batch_cost=1.3099, reader_cost=0.0002 | ETA 27:58:48 2020-11-02 20:07:31 [INFO] [TRAIN] epoch=9, iter=3200/80000, loss=0.2699, lr=0.009639, batch_cost=1.2527, reader_cost=0.0001 | ETA 26:43:25 2020-11-02 20:09:40 [INFO] [TRAIN] epoch=9, iter=3300/80000, loss=0.2182, lr=0.009628, batch_cost=1.2920, reader_cost=0.0002 | ETA 27:31:35 2020-11-02 20:11:50 [INFO] [TRAIN] epoch=10, iter=3400/80000, loss=0.2414, lr=0.009617, batch_cost=1.2940, reader_cost=0.0115 | ETA 27:32:01 2020-11-02 20:13:57 [INFO] [TRAIN] epoch=10, iter=3500/80000, loss=0.2104, lr=0.009605, batch_cost=1.2764, reader_cost=0.0004 | ETA 27:07:26 2020-11-02 20:16:04 [INFO] [TRAIN] epoch=10, iter=3600/80000, loss=0.2573, lr=0.009594, batch_cost=1.2680, reader_cost=0.0002 | ETA 26:54:36 2020-11-02 20:18:12 [INFO] [TRAIN] epoch=10, iter=3700/80000, loss=0.2446, lr=0.009583, batch_cost=1.2784, reader_cost=0.0001 | ETA 27:05:42 2020-11-02 20:20:21 [INFO] [TRAIN] epoch=11, iter=3800/80000, loss=0.2298, lr=0.009572, batch_cost=1.2927, reader_cost=0.0153 | ETA 27:21:41 2020-11-02 20:22:31 [INFO] [TRAIN] epoch=11, iter=3900/80000, loss=0.2107, lr=0.009560, batch_cost=1.2942, reader_cost=0.0002 | ETA 27:21:32 2020-11-02 20:24:36 [INFO] [TRAIN] epoch=11, iter=4000/80000, loss=0.2228, lr=0.009549, batch_cost=1.2526, reader_cost=0.0001 | ETA 26:26:38 2020-11-02 20:26:41 [INFO] [TRAIN] epoch=12, iter=4100/80000, loss=0.2054, lr=0.009538, batch_cost=1.2472, reader_cost=0.0098 | ETA 26:17:41 2020-11-02 20:28:47 [INFO] [TRAIN] epoch=12, iter=4200/80000, loss=0.2282, lr=0.009526, batch_cost=1.2590, reader_cost=0.0002 | ETA 26:30:32 2020-11-02 20:30:56 [INFO] [TRAIN] epoch=12, iter=4300/80000, loss=0.3095, lr=0.009515, batch_cost=1.2904, reader_cost=0.0001 | ETA 27:08:03 2020-11-02 20:33:06 [INFO] [TRAIN] epoch=12, iter=4400/80000, loss=0.3131, lr=0.009504, batch_cost=1.3063, reader_cost=0.0002 | ETA 27:25:57 2020-11-02 20:35:15 [INFO] [TRAIN] epoch=13, iter=4500/80000, loss=0.2795, lr=0.009492, batch_cost=1.2828, reader_cost=0.0088 | ETA 26:54:10 2020-11-02 20:37:22 [INFO] [TRAIN] epoch=13, iter=4600/80000, loss=0.2339, lr=0.009481, batch_cost=1.2777, reader_cost=0.0002 | ETA 26:45:35 2020-11-02 20:39:27 [INFO] [TRAIN] epoch=13, iter=4700/80000, loss=0.3150, lr=0.009470, batch_cost=1.2517, reader_cost=0.0001 | ETA 26:10:53 2020-11-02 20:41:34 [INFO] [TRAIN] epoch=13, iter=4800/80000, loss=0.2470, lr=0.009458, batch_cost=1.2610, reader_cost=0.0009 | ETA 26:20:28 2020-11-02 20:43:45 [INFO] [TRAIN] epoch=14, iter=4900/80000, loss=0.2285, lr=0.009447, batch_cost=1.3132, reader_cost=0.0089 | ETA 27:23:40 2020-11-02 20:45:59 [INFO] [TRAIN] epoch=14, iter=5000/80000, loss=0.2729, lr=0.009436, batch_cost=1.3399, reader_cost=0.0002 | ETA 27:54:49 2020-11-02 20:48:14 [INFO] [TRAIN] epoch=14, iter=5100/80000, loss=0.2763, lr=0.009424, batch_cost=1.3475, reader_cost=0.0001 | ETA 28:02:10 2020-11-02 20:50:26 [INFO] [TRAIN] epoch=14, iter=5200/80000, loss=0.2437, lr=0.009413, batch_cost=1.3195, reader_cost=0.0002 | ETA 27:25:02 2020-11-02 20:52:36 [INFO] [TRAIN] epoch=15, iter=5300/80000, loss=0.2318, lr=0.009402, batch_cost=1.3013, reader_cost=0.0114 | ETA 27:00:06 2020-11-02 20:54:47 [INFO] [TRAIN] epoch=15, iter=5400/80000, loss=0.1911, lr=0.009391, batch_cost=1.3098, reader_cost=0.0002 | ETA 27:08:28 2020-11-02 20:56:54 [INFO] [TRAIN] epoch=15, iter=5500/80000, loss=0.2417, lr=0.009379, batch_cost=1.2705, reader_cost=0.0001 | ETA 26:17:35 2020-11-02 20:59:01 [INFO] [TRAIN] epoch=16, iter=5600/80000, loss=0.2416, lr=0.009368, batch_cost=1.2703, reader_cost=0.0113 | ETA 26:15:10 2020-11-02 21:01:08 [INFO] [TRAIN] epoch=16, iter=5700/80000, loss=0.1810, lr=0.009357, batch_cost=1.2705, reader_cost=0.0002 | ETA 26:13:21 2020-11-02 21:03:16 [INFO] [TRAIN] epoch=16, iter=5800/80000, loss=0.1872, lr=0.009345, batch_cost=1.2825, reader_cost=0.0001 | ETA 26:26:03 2020-11-02 21:05:26 [INFO] [TRAIN] epoch=16, iter=5900/80000, loss=0.2048, lr=0.009334, batch_cost=1.2983, reader_cost=0.0002 | ETA 26:43:26 2020-11-02 21:07:34 [INFO] [TRAIN] epoch=17, iter=6000/80000, loss=0.1968, lr=0.009323, batch_cost=1.2803, reader_cost=0.0097 | ETA 26:19:05 2020-11-02 21:09:40 [INFO] [TRAIN] epoch=17, iter=6100/80000, loss=0.1879, lr=0.009311, batch_cost=1.2591, reader_cost=0.0001 | ETA 25:50:48 2020-11-02 21:11:45 [INFO] [TRAIN] epoch=17, iter=6200/80000, loss=0.1826, lr=0.009300, batch_cost=1.2554, reader_cost=0.0001 | ETA 25:44:08 2020-11-02 21:13:54 [INFO] [TRAIN] epoch=17, iter=6300/80000, loss=0.2432, lr=0.009288, batch_cost=1.2858, reader_cost=0.0002 | ETA 26:19:25 2020-11-02 21:16:01 [INFO] [TRAIN] epoch=18, iter=6400/80000, loss=0.2225, lr=0.009277, batch_cost=1.2703, reader_cost=0.0102 | ETA 25:58:12 2020-11-02 21:18:09 [INFO] [TRAIN] epoch=18, iter=6500/80000, loss=0.2212, lr=0.009266, batch_cost=1.2786, reader_cost=0.0002 | ETA 26:06:19 2020-11-02 21:20:19 [INFO] [TRAIN] epoch=18, iter=6600/80000, loss=0.1750, lr=0.009254, batch_cost=1.2962, reader_cost=0.0001 | ETA 26:25:41 2020-11-02 21:22:29 [INFO] [TRAIN] epoch=19, iter=6700/80000, loss=0.2047, lr=0.009243, batch_cost=1.3077, reader_cost=0.0110 | ETA 26:37:32 2020-11-02 21:24:38 [INFO] [TRAIN] epoch=19, iter=6800/80000, loss=0.2085, lr=0.009232, batch_cost=1.2855, reader_cost=0.0003 | ETA 26:08:21 2020-11-02 21:26:47 [INFO] [TRAIN] epoch=19, iter=6900/80000, loss=0.2042, lr=0.009220, batch_cost=1.2898, reader_cost=0.0002 | ETA 26:11:20 2020-11-02 21:28:54 [INFO] [TRAIN] epoch=19, iter=7000/80000, loss=0.1935, lr=0.009209, batch_cost=1.2669, reader_cost=0.0001 | ETA 25:41:25 2020-11-02 21:30:59 [INFO] [TRAIN] epoch=20, iter=7100/80000, loss=0.1772, lr=0.009198, batch_cost=1.2536, reader_cost=0.0110 | ETA 25:23:05 2020-11-02 21:33:10 [INFO] [TRAIN] epoch=20, iter=7200/80000, loss=0.1479, lr=0.009186, batch_cost=1.3086, reader_cost=0.0002 | ETA 26:27:42 2020-11-02 21:35:19 [INFO] [TRAIN] epoch=20, iter=7300/80000, loss=0.1830, lr=0.009175, batch_cost=1.2876, reader_cost=0.0004 | ETA 26:00:11 2020-11-02 21:37:25 [INFO] [TRAIN] epoch=20, iter=7400/80000, loss=0.2038, lr=0.009164, batch_cost=1.2644, reader_cost=0.0020 | ETA 25:29:52 2020-11-02 21:39:35 [INFO] [TRAIN] epoch=21, iter=7500/80000, loss=0.1623, lr=0.009152, batch_cost=1.3041, reader_cost=0.0097 | ETA 26:15:49 2020-11-02 21:41:47 [INFO] [TRAIN] epoch=21, iter=7600/80000, loss=0.1543, lr=0.009141, batch_cost=1.3191, reader_cost=0.0002 | ETA 26:31:43 2020-11-02 21:43:53 [INFO] [TRAIN] epoch=21, iter=7700/80000, loss=0.1830, lr=0.009130, batch_cost=1.2569, reader_cost=0.0001 | ETA 25:14:31 2020-11-02 21:46:00 [INFO] [TRAIN] epoch=21, iter=7800/80000, loss=0.1801, lr=0.009118, batch_cost=1.2732, reader_cost=0.0002 | ETA 25:32:08 2020-11-02 21:48:10 [INFO] [TRAIN] epoch=22, iter=7900/80000, loss=0.1790, lr=0.009107, batch_cost=1.2952, reader_cost=0.0126 | ETA 25:56:27 2020-11-02 21:50:20 [INFO] [TRAIN] epoch=22, iter=8000/80000, loss=0.1645, lr=0.009095, batch_cost=1.2987, reader_cost=0.0001 | ETA 25:58:24 2020-11-02 21:50:26 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-02 21:56:12 [INFO] [EVAL] #Images=500 mIoU=0.7062 Acc=0.9509 Kappa=0.9361 2020-11-02 21:56:12 [INFO] [EVAL] Category IoU: [0.9758 0.8125 0.9093 0.4264 0.5779 0.5628 0.6596 0.7419 0.9124 0.5766 0.9318 0.7933 0.5451 0.9288 0.6498 0.7385 0.5325 0.4266 0.715 ] 2020-11-02 21:56:12 [INFO] [EVAL] Category Acc: [0.9893 0.8947 0.938 0.8069 0.7234 0.8269 0.7934 0.8991 0.9434 0.835 0.9736 0.8914 0.7714 0.9708 0.8528 0.836 0.8663 0.517 0.7684] 2020-11-02 21:56:15 [INFO] [EVAL] The model with the best validation mIoU (0.7062) was saved at iter 8000. 2020-11-02 21:58:22 [INFO] [TRAIN] epoch=22, iter=8100/80000, loss=0.2166, lr=0.009084, batch_cost=1.2763, reader_cost=0.0002 | ETA 25:29:23 2020-11-02 22:00:32 [INFO] [TRAIN] epoch=23, iter=8200/80000, loss=0.2139, lr=0.009073, batch_cost=1.2928, reader_cost=0.0125 | ETA 25:47:03 2020-11-02 22:02:41 [INFO] [TRAIN] epoch=23, iter=8300/80000, loss=0.2371, lr=0.009061, batch_cost=1.2941, reader_cost=0.0002 | ETA 25:46:25 2020-11-02 22:04:50 [INFO] [TRAIN] epoch=23, iter=8400/80000, loss=0.1808, lr=0.009050, batch_cost=1.2896, reader_cost=0.0001 | ETA 25:38:52 2020-11-02 22:06:55 [INFO] [TRAIN] epoch=23, iter=8500/80000, loss=0.2601, lr=0.009039, batch_cost=1.2523, reader_cost=0.0002 | ETA 24:52:22 2020-11-02 22:09:04 [INFO] [TRAIN] epoch=24, iter=8600/80000, loss=0.1822, lr=0.009027, batch_cost=1.2910, reader_cost=0.0109 | ETA 25:36:15 2020-11-02 22:11:14 [INFO] [TRAIN] epoch=24, iter=8700/80000, loss=0.1689, lr=0.009016, batch_cost=1.3020, reader_cost=0.0002 | ETA 25:47:09 2020-11-02 22:13:17 [INFO] [TRAIN] epoch=24, iter=8800/80000, loss=0.1996, lr=0.009004, batch_cost=1.2227, reader_cost=0.0001 | ETA 24:10:55 2020-11-02 22:15:21 [INFO] [TRAIN] epoch=24, iter=8900/80000, loss=0.1812, lr=0.008993, batch_cost=1.2444, reader_cost=0.0001 | ETA 24:34:40 2020-11-02 22:17:27 [INFO] [TRAIN] epoch=25, iter=9000/80000, loss=0.1960, lr=0.008982, batch_cost=1.2605, reader_cost=0.0131 | ETA 24:51:32 2020-11-02 22:19:34 [INFO] [TRAIN] epoch=25, iter=9100/80000, loss=0.1646, lr=0.008970, batch_cost=1.2632, reader_cost=0.0002 | ETA 24:52:42 2020-11-02 22:21:38 [INFO] [TRAIN] epoch=25, iter=9200/80000, loss=0.1942, lr=0.008959, batch_cost=1.2437, reader_cost=0.0002 | ETA 24:27:33 2020-11-02 22:23:46 [INFO] [TRAIN] epoch=25, iter=9300/80000, loss=0.2187, lr=0.008948, batch_cost=1.2851, reader_cost=0.0001 | ETA 25:14:14 2020-11-02 22:25:54 [INFO] [TRAIN] epoch=26, iter=9400/80000, loss=0.1769, lr=0.008936, batch_cost=1.2786, reader_cost=0.0106 | ETA 25:04:29 2020-11-02 22:28:02 [INFO] [TRAIN] epoch=26, iter=9500/80000, loss=0.1635, lr=0.008925, batch_cost=1.2767, reader_cost=0.0004 | ETA 25:00:05 2020-11-02 22:30:06 [INFO] [TRAIN] epoch=26, iter=9600/80000, loss=0.1631, lr=0.008913, batch_cost=1.2424, reader_cost=0.0001 | ETA 24:17:44 2020-11-02 22:32:18 [INFO] [TRAIN] epoch=27, iter=9700/80000, loss=0.1790, lr=0.008902, batch_cost=1.3163, reader_cost=0.0088 | ETA 25:42:12 2020-11-02 22:34:26 [INFO] [TRAIN] epoch=27, iter=9800/80000, loss=0.1477, lr=0.008891, batch_cost=1.2784, reader_cost=0.0002 | ETA 24:55:43 2020-11-02 22:36:30 [INFO] [TRAIN] epoch=27, iter=9900/80000, loss=0.1586, lr=0.008879, batch_cost=1.2476, reader_cost=0.0001 | ETA 24:17:40 2020-11-02 22:38:37 [INFO] [TRAIN] epoch=27, iter=10000/80000, loss=0.1880, lr=0.008868, batch_cost=1.2648, reader_cost=0.0002 | ETA 24:35:37 2020-11-02 22:40:42 [INFO] [TRAIN] epoch=28, iter=10100/80000, loss=0.1763, lr=0.008856, batch_cost=1.2512, reader_cost=0.0102 | ETA 24:17:35 2020-11-02 22:42:50 [INFO] [TRAIN] epoch=28, iter=10200/80000, loss=0.1408, lr=0.008845, batch_cost=1.2742, reader_cost=0.0002 | ETA 24:42:18 2020-11-02 22:44:59 [INFO] [TRAIN] epoch=28, iter=10300/80000, loss=0.1729, lr=0.008834, batch_cost=1.2916, reader_cost=0.0002 | ETA 25:00:24 2020-11-02 22:47:04 [INFO] [TRAIN] epoch=28, iter=10400/80000, loss=0.1714, lr=0.008822, batch_cost=1.2514, reader_cost=0.0002 | ETA 24:11:38 2020-11-02 22:49:12 [INFO] [TRAIN] epoch=29, iter=10500/80000, loss=0.1509, lr=0.008811, batch_cost=1.2832, reader_cost=0.0109 | ETA 24:46:24 2020-11-02 22:51:17 [INFO] [TRAIN] epoch=29, iter=10600/80000, loss=0.1566, lr=0.008799, batch_cost=1.2474, reader_cost=0.0001 | ETA 24:02:52 2020-11-02 22:53:23 [INFO] [TRAIN] epoch=29, iter=10700/80000, loss=0.2021, lr=0.008788, batch_cost=1.2644, reader_cost=0.0001 | ETA 24:20:22 2020-11-02 22:55:34 [INFO] [TRAIN] epoch=30, iter=10800/80000, loss=0.1890, lr=0.008776, batch_cost=1.3080, reader_cost=0.0109 | ETA 25:08:32 2020-11-02 22:57:41 [INFO] [TRAIN] epoch=30, iter=10900/80000, loss=0.1603, lr=0.008765, batch_cost=1.2714, reader_cost=0.0010 | ETA 24:24:14 2020-11-02 22:59:48 [INFO] [TRAIN] epoch=30, iter=11000/80000, loss=0.1493, lr=0.008754, batch_cost=1.2693, reader_cost=0.0002 | ETA 24:19:42 2020-11-02 23:01:53 [INFO] [TRAIN] epoch=30, iter=11100/80000, loss=0.1695, lr=0.008742, batch_cost=1.2485, reader_cost=0.0001 | ETA 23:53:39 2020-11-02 23:04:02 [INFO] [TRAIN] epoch=31, iter=11200/80000, loss=0.1961, lr=0.008731, batch_cost=1.2871, reader_cost=0.0107 | ETA 24:35:53 2020-11-02 23:06:09 [INFO] [TRAIN] epoch=31, iter=11300/80000, loss=0.2221, lr=0.008719, batch_cost=1.2685, reader_cost=0.0002 | ETA 24:12:24 2020-11-02 23:08:14 [INFO] [TRAIN] epoch=31, iter=11400/80000, loss=0.1700, lr=0.008708, batch_cost=1.2576, reader_cost=0.0002 | ETA 23:57:50 2020-11-02 23:10:20 [INFO] [TRAIN] epoch=31, iter=11500/80000, loss=0.1787, lr=0.008697, batch_cost=1.2549, reader_cost=0.0002 | ETA 23:52:39 2020-11-02 23:12:27 [INFO] [TRAIN] epoch=32, iter=11600/80000, loss=0.1748, lr=0.008685, batch_cost=1.2748, reader_cost=0.0099 | ETA 24:13:14 2020-11-02 23:14:37 [INFO] [TRAIN] epoch=32, iter=11700/80000, loss=0.1510, lr=0.008674, batch_cost=1.2992, reader_cost=0.0002 | ETA 24:38:54 2020-11-02 23:16:50 [INFO] [TRAIN] epoch=32, iter=11800/80000, loss=0.2176, lr=0.008662, batch_cost=1.3270, reader_cost=0.0001 | ETA 25:08:22 2020-11-02 23:18:57 [INFO] [TRAIN] epoch=32, iter=11900/80000, loss=0.1717, lr=0.008651, batch_cost=1.2687, reader_cost=0.0002 | ETA 24:00:01 2020-11-02 23:21:06 [INFO] [TRAIN] epoch=33, iter=12000/80000, loss=0.1492, lr=0.008639, batch_cost=1.2883, reader_cost=0.0126 | ETA 24:20:01 2020-11-02 23:23:14 [INFO] [TRAIN] epoch=33, iter=12100/80000, loss=0.1410, lr=0.008628, batch_cost=1.2866, reader_cost=0.0004 | ETA 24:16:03 2020-11-02 23:25:24 [INFO] [TRAIN] epoch=33, iter=12200/80000, loss=0.2438, lr=0.008617, batch_cost=1.2987, reader_cost=0.0002 | ETA 24:27:30 2020-11-02 23:27:35 [INFO] [TRAIN] epoch=34, iter=12300/80000, loss=0.2178, lr=0.008605, batch_cost=1.3095, reader_cost=0.0124 | ETA 24:37:35 2020-11-02 23:29:42 [INFO] [TRAIN] epoch=34, iter=12400/80000, loss=0.1670, lr=0.008594, batch_cost=1.2728, reader_cost=0.0003 | ETA 23:54:03 2020-11-02 23:31:49 [INFO] [TRAIN] epoch=34, iter=12500/80000, loss=0.1809, lr=0.008582, batch_cost=1.2636, reader_cost=0.0002 | ETA 23:41:36 2020-11-02 23:33:57 [INFO] [TRAIN] epoch=34, iter=12600/80000, loss=0.1568, lr=0.008571, batch_cost=1.2776, reader_cost=0.0001 | ETA 23:55:07 2020-11-02 23:36:07 [INFO] [TRAIN] epoch=35, iter=12700/80000, loss=0.1438, lr=0.008559, batch_cost=1.3053, reader_cost=0.0115 | ETA 24:24:07 2020-11-02 23:38:13 [INFO] [TRAIN] epoch=35, iter=12800/80000, loss=0.1454, lr=0.008548, batch_cost=1.2637, reader_cost=0.0002 | ETA 23:35:20 2020-11-02 23:40:18 [INFO] [TRAIN] epoch=35, iter=12900/80000, loss=0.1639, lr=0.008536, batch_cost=1.2485, reader_cost=0.0002 | ETA 23:16:12 2020-11-02 23:42:25 [INFO] [TRAIN] epoch=35, iter=13000/80000, loss=0.1710, lr=0.008525, batch_cost=1.2645, reader_cost=0.0001 | ETA 23:32:00 2020-11-02 23:44:32 [INFO] [TRAIN] epoch=36, iter=13100/80000, loss=0.1531, lr=0.008514, batch_cost=1.2735, reader_cost=0.0104 | ETA 23:39:56 2020-11-02 23:46:39 [INFO] [TRAIN] epoch=36, iter=13200/80000, loss=0.1377, lr=0.008502, batch_cost=1.2691, reader_cost=0.0002 | ETA 23:32:57 2020-11-02 23:48:46 [INFO] [TRAIN] epoch=36, iter=13300/80000, loss=0.1627, lr=0.008491, batch_cost=1.2683, reader_cost=0.0001 | ETA 23:29:57 2020-11-02 23:50:54 [INFO] [TRAIN] epoch=37, iter=13400/80000, loss=0.1738, lr=0.008479, batch_cost=1.2762, reader_cost=0.0114 | ETA 23:36:36 2020-11-02 23:53:01 [INFO] [TRAIN] epoch=37, iter=13500/80000, loss=0.1369, lr=0.008468, batch_cost=1.2755, reader_cost=0.0002 | ETA 23:33:37 2020-11-02 23:55:07 [INFO] [TRAIN] epoch=37, iter=13600/80000, loss=0.1487, lr=0.008456, batch_cost=1.2586, reader_cost=0.0002 | ETA 23:12:53 2020-11-02 23:57:12 [INFO] [TRAIN] epoch=37, iter=13700/80000, loss=0.1780, lr=0.008445, batch_cost=1.2458, reader_cost=0.0001 | ETA 22:56:38 2020-11-02 23:59:17 [INFO] [TRAIN] epoch=38, iter=13800/80000, loss=0.1678, lr=0.008433, batch_cost=1.2580, reader_cost=0.0094 | ETA 23:08:00 2020-11-03 00:01:28 [INFO] [TRAIN] epoch=38, iter=13900/80000, loss=0.1307, lr=0.008422, batch_cost=1.3038, reader_cost=0.0001 | ETA 23:56:19 2020-11-03 00:03:34 [INFO] [TRAIN] epoch=38, iter=14000/80000, loss=0.1351, lr=0.008410, batch_cost=1.2623, reader_cost=0.0002 | ETA 23:08:29 2020-11-03 00:05:41 [INFO] [TRAIN] epoch=38, iter=14100/80000, loss=0.1530, lr=0.008399, batch_cost=1.2669, reader_cost=0.0001 | ETA 23:11:30 2020-11-03 00:07:49 [INFO] [TRAIN] epoch=39, iter=14200/80000, loss=0.1623, lr=0.008387, batch_cost=1.2859, reader_cost=0.0098 | ETA 23:30:13 2020-11-03 00:10:01 [INFO] [TRAIN] epoch=39, iter=14300/80000, loss=0.1219, lr=0.008376, batch_cost=1.3208, reader_cost=0.0002 | ETA 24:06:15 2020-11-03 00:12:12 [INFO] [TRAIN] epoch=39, iter=14400/80000, loss=0.1532, lr=0.008364, batch_cost=1.3033, reader_cost=0.0001 | ETA 23:44:56 2020-11-03 00:14:17 [INFO] [TRAIN] epoch=39, iter=14500/80000, loss=0.1374, lr=0.008353, batch_cost=1.2548, reader_cost=0.0001 | ETA 22:49:49 2020-11-03 00:16:26 [INFO] [TRAIN] epoch=40, iter=14600/80000, loss=0.1708, lr=0.008342, batch_cost=1.2848, reader_cost=0.0090 | ETA 23:20:29 2020-11-03 00:18:36 [INFO] [TRAIN] epoch=40, iter=14700/80000, loss=0.1211, lr=0.008330, batch_cost=1.3055, reader_cost=0.0002 | ETA 23:40:47 2020-11-03 00:20:42 [INFO] [TRAIN] epoch=40, iter=14800/80000, loss=0.1682, lr=0.008319, batch_cost=1.2597, reader_cost=0.0002 | ETA 22:48:51 2020-11-03 00:22:50 [INFO] [TRAIN] epoch=41, iter=14900/80000, loss=0.1648, lr=0.008307, batch_cost=1.2831, reader_cost=0.0107 | ETA 23:12:08 2020-11-03 00:24:57 [INFO] [TRAIN] epoch=41, iter=15000/80000, loss=0.1538, lr=0.008296, batch_cost=1.2620, reader_cost=0.0002 | ETA 22:47:08 2020-11-03 00:27:02 [INFO] [TRAIN] epoch=41, iter=15100/80000, loss=0.1346, lr=0.008284, batch_cost=1.2586, reader_cost=0.0001 | ETA 22:41:26 2020-11-03 00:29:08 [INFO] [TRAIN] epoch=41, iter=15200/80000, loss=0.1566, lr=0.008273, batch_cost=1.2576, reader_cost=0.0002 | ETA 22:38:13 2020-11-03 00:31:18 [INFO] [TRAIN] epoch=42, iter=15300/80000, loss=0.1321, lr=0.008261, batch_cost=1.2977, reader_cost=0.0122 | ETA 23:19:21 2020-11-03 00:33:32 [INFO] [TRAIN] epoch=42, iter=15400/80000, loss=0.1395, lr=0.008250, batch_cost=1.3405, reader_cost=0.0003 | ETA 24:03:13 2020-11-03 00:35:37 [INFO] [TRAIN] epoch=42, iter=15500/80000, loss=0.1364, lr=0.008238, batch_cost=1.2534, reader_cost=0.0001 | ETA 22:27:24 2020-11-03 00:37:41 [INFO] [TRAIN] epoch=42, iter=15600/80000, loss=0.1515, lr=0.008227, batch_cost=1.2365, reader_cost=0.0001 | ETA 22:07:08 2020-11-03 00:39:48 [INFO] [TRAIN] epoch=43, iter=15700/80000, loss=0.1692, lr=0.008215, batch_cost=1.2687, reader_cost=0.0088 | ETA 22:39:34 2020-11-03 00:41:52 [INFO] [TRAIN] epoch=43, iter=15800/80000, loss=0.1955, lr=0.008204, batch_cost=1.2417, reader_cost=0.0002 | ETA 22:08:37 2020-11-03 00:43:56 [INFO] [TRAIN] epoch=43, iter=15900/80000, loss=0.2320, lr=0.008192, batch_cost=1.2437, reader_cost=0.0001 | ETA 22:08:43 2020-11-03 00:46:01 [INFO] [TRAIN] epoch=44, iter=16000/80000, loss=0.1598, lr=0.008181, batch_cost=1.2493, reader_cost=0.0116 | ETA 22:12:32 2020-11-03 00:46:07 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-03 00:51:44 [INFO] [EVAL] #Images=500 mIoU=0.7605 Acc=0.9545 Kappa=0.9409 2020-11-03 00:51:44 [INFO] [EVAL] Category IoU: [0.9741 0.8096 0.9182 0.4737 0.5764 0.6243 0.6891 0.7639 0.9167 0.6152 0.9321 0.8125 0.6136 0.9436 0.8251 0.8313 0.7582 0.6261 0.7463] 2020-11-03 00:51:44 [INFO] [EVAL] Category Acc: [0.9868 0.907 0.9586 0.7827 0.7392 0.7594 0.8251 0.8828 0.9459 0.7888 0.9655 0.8933 0.7157 0.9691 0.9283 0.9462 0.8774 0.767 0.8157] 2020-11-03 00:51:48 [INFO] [EVAL] The model with the best validation mIoU (0.7605) was saved at iter 16000. 2020-11-03 00:53:52 [INFO] [TRAIN] epoch=44, iter=16100/80000, loss=0.1222, lr=0.008169, batch_cost=1.2447, reader_cost=0.0002 | ETA 22:05:34 2020-11-03 00:55:59 [INFO] [TRAIN] epoch=44, iter=16200/80000, loss=0.1468, lr=0.008158, batch_cost=1.2684, reader_cost=0.0002 | ETA 22:28:44 2020-11-03 00:58:05 [INFO] [TRAIN] epoch=44, iter=16300/80000, loss=0.1593, lr=0.008146, batch_cost=1.2577, reader_cost=0.0001 | ETA 22:15:16 2020-11-03 01:00:12 [INFO] [TRAIN] epoch=45, iter=16400/80000, loss=0.1600, lr=0.008135, batch_cost=1.2747, reader_cost=0.0107 | ETA 22:31:07 2020-11-03 01:02:29 [INFO] [TRAIN] epoch=45, iter=16500/80000, loss=0.1361, lr=0.008123, batch_cost=1.3687, reader_cost=0.0003 | ETA 24:08:31 2020-11-03 01:04:38 [INFO] [TRAIN] epoch=45, iter=16600/80000, loss=0.1371, lr=0.008112, batch_cost=1.2855, reader_cost=0.0002 | ETA 22:38:18 2020-11-03 01:06:46 [INFO] [TRAIN] epoch=45, iter=16700/80000, loss=0.1659, lr=0.008100, batch_cost=1.2778, reader_cost=0.0002 | ETA 22:28:07 2020-11-03 01:08:59 [INFO] [TRAIN] epoch=46, iter=16800/80000, loss=0.1769, lr=0.008089, batch_cost=1.3365, reader_cost=0.0165 | ETA 23:27:47 2020-11-03 01:11:08 [INFO] [TRAIN] epoch=46, iter=16900/80000, loss=0.1391, lr=0.008077, batch_cost=1.2901, reader_cost=0.0002 | ETA 22:36:43 2020-11-03 01:13:14 [INFO] [TRAIN] epoch=46, iter=17000/80000, loss=0.1388, lr=0.008066, batch_cost=1.2618, reader_cost=0.0002 | ETA 22:04:52 2020-11-03 01:15:23 [INFO] [TRAIN] epoch=46, iter=17100/80000, loss=0.1630, lr=0.008054, batch_cost=1.2815, reader_cost=0.0001 | ETA 22:23:27 2020-11-03 01:17:42 [INFO] [TRAIN] epoch=47, iter=17200/80000, loss=0.1432, lr=0.008042, batch_cost=1.3922, reader_cost=0.0118 | ETA 24:17:08 2020-11-03 01:19:50 [INFO] [TRAIN] epoch=47, iter=17300/80000, loss=0.1358, lr=0.008031, batch_cost=1.2843, reader_cost=0.0002 | ETA 22:22:06 2020-11-03 01:21:57 [INFO] [TRAIN] epoch=47, iter=17400/80000, loss=0.1695, lr=0.008019, batch_cost=1.2679, reader_cost=0.0002 | ETA 22:02:52 2020-11-03 01:24:06 [INFO] [TRAIN] epoch=48, iter=17500/80000, loss=0.1615, lr=0.008008, batch_cost=1.2896, reader_cost=0.0112 | ETA 22:23:18 2020-11-03 01:26:18 [INFO] [TRAIN] epoch=48, iter=17600/80000, loss=0.1486, lr=0.007996, batch_cost=1.3171, reader_cost=0.0002 | ETA 22:49:49 2020-11-03 01:28:28 [INFO] [TRAIN] epoch=48, iter=17700/80000, loss=0.1490, lr=0.007985, batch_cost=1.3014, reader_cost=0.0001 | ETA 22:31:17 2020-11-03 01:30:32 [INFO] [TRAIN] epoch=48, iter=17800/80000, loss=0.1820, lr=0.007973, batch_cost=1.2433, reader_cost=0.0001 | ETA 21:28:54 2020-11-03 01:32:40 [INFO] [TRAIN] epoch=49, iter=17900/80000, loss=0.1550, lr=0.007962, batch_cost=1.2751, reader_cost=0.0103 | ETA 21:59:46 2020-11-03 01:34:48 [INFO] [TRAIN] epoch=49, iter=18000/80000, loss=0.1480, lr=0.007950, batch_cost=1.2856, reader_cost=0.0002 | ETA 22:08:24 2020-11-03 01:37:09 [INFO] [TRAIN] epoch=49, iter=18100/80000, loss=0.1733, lr=0.007939, batch_cost=1.4108, reader_cost=0.0006 | ETA 24:15:31 2020-11-03 01:39:15 [INFO] [TRAIN] epoch=49, iter=18200/80000, loss=0.1474, lr=0.007927, batch_cost=1.2580, reader_cost=0.0002 | ETA 21:35:46 2020-11-03 01:41:28 [INFO] [TRAIN] epoch=50, iter=18300/80000, loss=0.1494, lr=0.007916, batch_cost=1.3270, reader_cost=0.0117 | ETA 22:44:35 2020-11-03 01:43:33 [INFO] [TRAIN] epoch=50, iter=18400/80000, loss=0.1172, lr=0.007904, batch_cost=1.2470, reader_cost=0.0001 | ETA 21:20:12 2020-11-03 01:45:43 [INFO] [TRAIN] epoch=50, iter=18500/80000, loss=0.1611, lr=0.007892, batch_cost=1.3019, reader_cost=0.0002 | ETA 22:14:27 2020-11-03 01:47:52 [INFO] [TRAIN] epoch=50, iter=18600/80000, loss=0.1599, lr=0.007881, batch_cost=1.2890, reader_cost=0.0001 | ETA 21:59:04 2020-11-03 01:49:57 [INFO] [TRAIN] epoch=51, iter=18700/80000, loss=0.1208, lr=0.007869, batch_cost=1.2552, reader_cost=0.0101 | ETA 21:22:22 2020-11-03 01:52:06 [INFO] [TRAIN] epoch=51, iter=18800/80000, loss=0.1488, lr=0.007858, batch_cost=1.2935, reader_cost=0.0009 | ETA 21:59:25 2020-11-03 01:54:14 [INFO] [TRAIN] epoch=51, iter=18900/80000, loss=0.1409, lr=0.007846, batch_cost=1.2761, reader_cost=0.0001 | ETA 21:39:29 2020-11-03 01:56:21 [INFO] [TRAIN] epoch=52, iter=19000/80000, loss=0.1237, lr=0.007835, batch_cost=1.2703, reader_cost=0.0093 | ETA 21:31:30 2020-11-03 01:58:29 [INFO] [TRAIN] epoch=52, iter=19100/80000, loss=0.1315, lr=0.007823, batch_cost=1.2747, reader_cost=0.0002 | ETA 21:33:48 2020-11-03 02:00:33 [INFO] [TRAIN] epoch=52, iter=19200/80000, loss=0.1427, lr=0.007812, batch_cost=1.2446, reader_cost=0.0001 | ETA 21:01:10 2020-11-03 02:02:41 [INFO] [TRAIN] epoch=52, iter=19300/80000, loss=0.1411, lr=0.007800, batch_cost=1.2796, reader_cost=0.0001 | ETA 21:34:31 2020-11-03 02:04:53 [INFO] [TRAIN] epoch=53, iter=19400/80000, loss=0.1448, lr=0.007788, batch_cost=1.3235, reader_cost=0.0136 | ETA 22:16:46 2020-11-03 02:07:00 [INFO] [TRAIN] epoch=53, iter=19500/80000, loss=0.1452, lr=0.007777, batch_cost=1.2645, reader_cost=0.0002 | ETA 21:15:03 2020-11-03 02:09:04 [INFO] [TRAIN] epoch=53, iter=19600/80000, loss=0.1320, lr=0.007765, batch_cost=1.2431, reader_cost=0.0001 | ETA 20:51:22 2020-11-03 02:11:08 [INFO] [TRAIN] epoch=53, iter=19700/80000, loss=0.1540, lr=0.007754, batch_cost=1.2432, reader_cost=0.0002 | ETA 20:49:25 2020-11-03 02:13:17 [INFO] [TRAIN] epoch=54, iter=19800/80000, loss=0.1362, lr=0.007742, batch_cost=1.2871, reader_cost=0.0109 | ETA 21:31:21 2020-11-03 02:15:21 [INFO] [TRAIN] epoch=54, iter=19900/80000, loss=0.1267, lr=0.007731, batch_cost=1.2381, reader_cost=0.0002 | ETA 20:40:07 2020-11-03 02:17:25 [INFO] [TRAIN] epoch=54, iter=20000/80000, loss=0.1433, lr=0.007719, batch_cost=1.2394, reader_cost=0.0001 | ETA 20:39:25 2020-11-03 02:19:30 [INFO] [TRAIN] epoch=55, iter=20100/80000, loss=0.1472, lr=0.007707, batch_cost=1.2524, reader_cost=0.0143 | ETA 20:50:16 2020-11-03 02:21:36 [INFO] [TRAIN] epoch=55, iter=20200/80000, loss=0.1241, lr=0.007696, batch_cost=1.2570, reader_cost=0.0002 | ETA 20:52:46 2020-11-03 02:23:42 [INFO] [TRAIN] epoch=55, iter=20300/80000, loss=0.1372, lr=0.007684, batch_cost=1.2618, reader_cost=0.0009 | ETA 20:55:27 2020-11-03 02:25:47 [INFO] [TRAIN] epoch=55, iter=20400/80000, loss=0.1283, lr=0.007673, batch_cost=1.2499, reader_cost=0.0002 | ETA 20:41:34 2020-11-03 02:27:55 [INFO] [TRAIN] epoch=56, iter=20500/80000, loss=0.1377, lr=0.007661, batch_cost=1.2823, reader_cost=0.0094 | ETA 21:11:36 2020-11-03 02:30:12 [INFO] [TRAIN] epoch=56, iter=20600/80000, loss=0.1200, lr=0.007650, batch_cost=1.3677, reader_cost=0.0004 | ETA 22:34:02 2020-11-03 02:32:23 [INFO] [TRAIN] epoch=56, iter=20700/80000, loss=0.1442, lr=0.007638, batch_cost=1.3105, reader_cost=0.0001 | ETA 21:35:15 2020-11-03 02:34:29 [INFO] [TRAIN] epoch=56, iter=20800/80000, loss=0.1402, lr=0.007626, batch_cost=1.2603, reader_cost=0.0001 | ETA 20:43:29 2020-11-03 02:36:48 [INFO] [TRAIN] epoch=57, iter=20900/80000, loss=0.1375, lr=0.007615, batch_cost=1.3900, reader_cost=0.0095 | ETA 22:49:08 2020-11-03 02:38:53 [INFO] [TRAIN] epoch=57, iter=21000/80000, loss=0.1226, lr=0.007603, batch_cost=1.2481, reader_cost=0.0002 | ETA 20:27:16 2020-11-03 02:40:57 [INFO] [TRAIN] epoch=57, iter=21100/80000, loss=0.1516, lr=0.007592, batch_cost=1.2425, reader_cost=0.0001 | ETA 20:19:41 2020-11-03 02:43:04 [INFO] [TRAIN] epoch=57, iter=21200/80000, loss=0.1349, lr=0.007580, batch_cost=1.2722, reader_cost=0.0001 | ETA 20:46:47 2020-11-03 02:45:10 [INFO] [TRAIN] epoch=58, iter=21300/80000, loss=0.1458, lr=0.007568, batch_cost=1.2596, reader_cost=0.0107 | ETA 20:32:17 2020-11-03 02:47:15 [INFO] [TRAIN] epoch=58, iter=21400/80000, loss=0.1375, lr=0.007557, batch_cost=1.2501, reader_cost=0.0003 | ETA 20:20:56 2020-11-03 02:49:21 [INFO] [TRAIN] epoch=58, iter=21500/80000, loss=0.1499, lr=0.007545, batch_cost=1.2551, reader_cost=0.0002 | ETA 20:23:42 2020-11-03 02:51:28 [INFO] [TRAIN] epoch=59, iter=21600/80000, loss=0.1583, lr=0.007534, batch_cost=1.2712, reader_cost=0.0096 | ETA 20:37:18 2020-11-03 02:53:36 [INFO] [TRAIN] epoch=59, iter=21700/80000, loss=0.1273, lr=0.007522, batch_cost=1.2749, reader_cost=0.0002 | ETA 20:38:47 2020-11-03 02:55:42 [INFO] [TRAIN] epoch=59, iter=21800/80000, loss=0.1365, lr=0.007510, batch_cost=1.2659, reader_cost=0.0002 | ETA 20:27:54 2020-11-03 02:57:49 [INFO] [TRAIN] epoch=59, iter=21900/80000, loss=0.1625, lr=0.007499, batch_cost=1.2706, reader_cost=0.0001 | ETA 20:30:21 2020-11-03 03:00:00 [INFO] [TRAIN] epoch=60, iter=22000/80000, loss=0.1520, lr=0.007487, batch_cost=1.3092, reader_cost=0.0150 | ETA 21:05:35 2020-11-03 03:02:12 [INFO] [TRAIN] epoch=60, iter=22100/80000, loss=0.1333, lr=0.007475, batch_cost=1.3148, reader_cost=0.0002 | ETA 21:08:49 2020-11-03 03:04:26 [INFO] [TRAIN] epoch=60, iter=22200/80000, loss=0.1283, lr=0.007464, batch_cost=1.3441, reader_cost=0.0002 | ETA 21:34:49 2020-11-03 03:06:34 [INFO] [TRAIN] epoch=60, iter=22300/80000, loss=0.1649, lr=0.007452, batch_cost=1.2838, reader_cost=0.0001 | ETA 20:34:36 2020-11-03 03:08:44 [INFO] [TRAIN] epoch=61, iter=22400/80000, loss=0.1232, lr=0.007441, batch_cost=1.2956, reader_cost=0.0113 | ETA 20:43:47 2020-11-03 03:10:51 [INFO] [TRAIN] epoch=61, iter=22500/80000, loss=0.1252, lr=0.007429, batch_cost=1.2701, reader_cost=0.0001 | ETA 20:17:09 2020-11-03 03:12:59 [INFO] [TRAIN] epoch=61, iter=22600/80000, loss=0.1265, lr=0.007417, batch_cost=1.2794, reader_cost=0.0001 | ETA 20:23:57 2020-11-03 03:15:08 [INFO] [TRAIN] epoch=62, iter=22700/80000, loss=0.1618, lr=0.007406, batch_cost=1.2943, reader_cost=0.0094 | ETA 20:36:05 2020-11-03 03:17:15 [INFO] [TRAIN] epoch=62, iter=22800/80000, loss=0.1677, lr=0.007394, batch_cost=1.2617, reader_cost=0.0004 | ETA 20:02:50 2020-11-03 03:19:23 [INFO] [TRAIN] epoch=62, iter=22900/80000, loss=0.1208, lr=0.007382, batch_cost=1.2800, reader_cost=0.0007 | ETA 20:18:09 2020-11-03 03:21:29 [INFO] [TRAIN] epoch=62, iter=23000/80000, loss=0.1525, lr=0.007371, batch_cost=1.2599, reader_cost=0.0002 | ETA 19:56:53 2020-11-03 03:23:33 [INFO] [TRAIN] epoch=63, iter=23100/80000, loss=0.1224, lr=0.007359, batch_cost=1.2464, reader_cost=0.0110 | ETA 19:42:00 2020-11-03 03:25:40 [INFO] [TRAIN] epoch=63, iter=23200/80000, loss=0.1185, lr=0.007347, batch_cost=1.2717, reader_cost=0.0004 | ETA 20:03:52 2020-11-03 03:27:43 [INFO] [TRAIN] epoch=63, iter=23300/80000, loss=0.1074, lr=0.007336, batch_cost=1.2277, reader_cost=0.0001 | ETA 19:20:11 2020-11-03 03:30:00 [INFO] [TRAIN] epoch=63, iter=23400/80000, loss=0.1379, lr=0.007324, batch_cost=1.3660, reader_cost=0.0001 | ETA 21:28:34 2020-11-03 03:32:06 [INFO] [TRAIN] epoch=64, iter=23500/80000, loss=0.1242, lr=0.007313, batch_cost=1.2593, reader_cost=0.0084 | ETA 19:45:51 2020-11-03 03:34:16 [INFO] [TRAIN] epoch=64, iter=23600/80000, loss=0.1210, lr=0.007301, batch_cost=1.3059, reader_cost=0.0002 | ETA 20:27:34 2020-11-03 03:36:26 [INFO] [TRAIN] epoch=64, iter=23700/80000, loss=0.1260, lr=0.007289, batch_cost=1.3009, reader_cost=0.0003 | ETA 20:20:42 2020-11-03 03:38:33 [INFO] [TRAIN] epoch=64, iter=23800/80000, loss=0.1424, lr=0.007278, batch_cost=1.2666, reader_cost=0.0002 | ETA 19:46:23 2020-11-03 03:40:41 [INFO] [TRAIN] epoch=65, iter=23900/80000, loss=0.1357, lr=0.007266, batch_cost=1.2759, reader_cost=0.0097 | ETA 19:52:59 2020-11-03 03:42:47 [INFO] [TRAIN] epoch=65, iter=24000/80000, loss=0.0976, lr=0.007254, batch_cost=1.2611, reader_cost=0.0002 | ETA 19:37:02 2020-11-03 03:42:53 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-03 03:48:29 [INFO] [EVAL] #Images=500 mIoU=0.7663 Acc=0.9593 Kappa=0.9473 2020-11-03 03:48:29 [INFO] [EVAL] Category IoU: [0.9802 0.8458 0.9246 0.5648 0.6415 0.6376 0.707 0.7819 0.9222 0.6539 0.945 0.8238 0.626 0.9455 0.6501 0.732 0.7428 0.6592 0.775 ] 2020-11-03 03:48:29 [INFO] [EVAL] Category Acc: [0.9927 0.9149 0.9611 0.8514 0.8165 0.7714 0.8297 0.8619 0.9518 0.7955 0.965 0.8873 0.7588 0.972 0.9273 0.7593 0.8106 0.8513 0.8642] 2020-11-03 03:48:32 [INFO] [EVAL] The model with the best validation mIoU (0.7663) was saved at iter 24000. 2020-11-03 03:50:38 [INFO] [TRAIN] epoch=65, iter=24100/80000, loss=0.1548, lr=0.007243, batch_cost=1.2582, reader_cost=0.0003 | ETA 19:32:11 2020-11-03 03:52:47 [INFO] [TRAIN] epoch=66, iter=24200/80000, loss=0.1645, lr=0.007231, batch_cost=1.2851, reader_cost=0.0127 | ETA 19:55:07 2020-11-03 03:54:54 [INFO] [TRAIN] epoch=66, iter=24300/80000, loss=0.1207, lr=0.007219, batch_cost=1.2709, reader_cost=0.0007 | ETA 19:39:49 2020-11-03 03:57:02 [INFO] [TRAIN] epoch=66, iter=24400/80000, loss=0.1370, lr=0.007208, batch_cost=1.2789, reader_cost=0.0002 | ETA 19:45:05 2020-11-03 03:59:10 [INFO] [TRAIN] epoch=66, iter=24500/80000, loss=0.1378, lr=0.007196, batch_cost=1.2871, reader_cost=0.0003 | ETA 19:50:35 2020-11-03 04:01:16 [INFO] [TRAIN] epoch=67, iter=24600/80000, loss=0.1484, lr=0.007184, batch_cost=1.2614, reader_cost=0.0130 | ETA 19:24:39 2020-11-03 04:03:38 [INFO] [TRAIN] epoch=67, iter=24700/80000, loss=0.1231, lr=0.007173, batch_cost=1.4210, reader_cost=0.0004 | ETA 21:49:39 2020-11-03 04:05:44 [INFO] [TRAIN] epoch=67, iter=24800/80000, loss=0.1205, lr=0.007161, batch_cost=1.2553, reader_cost=0.0001 | ETA 19:14:54 2020-11-03 04:07:53 [INFO] [TRAIN] epoch=67, iter=24900/80000, loss=0.1381, lr=0.007149, batch_cost=1.2933, reader_cost=0.0004 | ETA 19:47:39 2020-11-03 04:10:00 [INFO] [TRAIN] epoch=68, iter=25000/80000, loss=0.1516, lr=0.007138, batch_cost=1.2704, reader_cost=0.0116 | ETA 19:24:31 2020-11-03 04:12:09 [INFO] [TRAIN] epoch=68, iter=25100/80000, loss=0.1055, lr=0.007126, batch_cost=1.2828, reader_cost=0.0002 | ETA 19:33:45 2020-11-03 04:14:22 [INFO] [TRAIN] epoch=68, iter=25200/80000, loss=0.1388, lr=0.007114, batch_cost=1.3308, reader_cost=0.0001 | ETA 20:15:25 2020-11-03 04:16:30 [INFO] [TRAIN] epoch=69, iter=25300/80000, loss=0.1409, lr=0.007103, batch_cost=1.2804, reader_cost=0.0101 | ETA 19:27:16 2020-11-03 04:18:43 [INFO] [TRAIN] epoch=69, iter=25400/80000, loss=0.1326, lr=0.007091, batch_cost=1.3276, reader_cost=0.0002 | ETA 20:08:04 2020-11-03 04:20:47 [INFO] [TRAIN] epoch=69, iter=25500/80000, loss=0.1419, lr=0.007079, batch_cost=1.2457, reader_cost=0.0002 | ETA 18:51:31 2020-11-03 04:22:56 [INFO] [TRAIN] epoch=69, iter=25600/80000, loss=0.1573, lr=0.007067, batch_cost=1.2843, reader_cost=0.0003 | ETA 19:24:27 2020-11-03 04:25:06 [INFO] [TRAIN] epoch=70, iter=25700/80000, loss=0.1421, lr=0.007056, batch_cost=1.3091, reader_cost=0.0095 | ETA 19:44:45 2020-11-03 04:27:11 [INFO] [TRAIN] epoch=70, iter=25800/80000, loss=0.1187, lr=0.007044, batch_cost=1.2436, reader_cost=0.0002 | ETA 18:43:25 2020-11-03 04:29:15 [INFO] [TRAIN] epoch=70, iter=25900/80000, loss=0.1441, lr=0.007032, batch_cost=1.2390, reader_cost=0.0001 | ETA 18:37:07 2020-11-03 04:31:19 [INFO] [TRAIN] epoch=70, iter=26000/80000, loss=0.1657, lr=0.007021, batch_cost=1.2449, reader_cost=0.0001 | ETA 18:40:22 2020-11-03 04:33:27 [INFO] [TRAIN] epoch=71, iter=26100/80000, loss=0.1466, lr=0.007009, batch_cost=1.2765, reader_cost=0.0114 | ETA 19:06:45 2020-11-03 04:35:37 [INFO] [TRAIN] epoch=71, iter=26200/80000, loss=0.1196, lr=0.006997, batch_cost=1.2998, reader_cost=0.0014 | ETA 19:25:31 2020-11-03 04:37:45 [INFO] [TRAIN] epoch=71, iter=26300/80000, loss=0.1330, lr=0.006986, batch_cost=1.2799, reader_cost=0.0001 | ETA 19:05:32 2020-11-03 04:39:53 [INFO] [TRAIN] epoch=71, iter=26400/80000, loss=0.1597, lr=0.006974, batch_cost=1.2809, reader_cost=0.0005 | ETA 19:04:18 2020-11-03 04:42:06 [INFO] [TRAIN] epoch=72, iter=26500/80000, loss=0.1238, lr=0.006962, batch_cost=1.3273, reader_cost=0.0114 | ETA 19:43:31 2020-11-03 04:44:15 [INFO] [TRAIN] epoch=72, iter=26600/80000, loss=0.1199, lr=0.006950, batch_cost=1.2885, reader_cost=0.0005 | ETA 19:06:47 2020-11-03 04:46:22 [INFO] [TRAIN] epoch=72, iter=26700/80000, loss=0.1416, lr=0.006939, batch_cost=1.2742, reader_cost=0.0003 | ETA 18:51:54 2020-11-03 04:48:32 [INFO] [TRAIN] epoch=73, iter=26800/80000, loss=0.1275, lr=0.006927, batch_cost=1.2982, reader_cost=0.0111 | ETA 19:11:06 2020-11-03 04:50:40 [INFO] [TRAIN] epoch=73, iter=26900/80000, loss=0.1370, lr=0.006915, batch_cost=1.2796, reader_cost=0.0002 | ETA 18:52:26 2020-11-03 04:52:45 [INFO] [TRAIN] epoch=73, iter=27000/80000, loss=0.1651, lr=0.006904, batch_cost=1.2538, reader_cost=0.0003 | ETA 18:27:33 2020-11-03 04:54:52 [INFO] [TRAIN] epoch=73, iter=27100/80000, loss=0.1380, lr=0.006892, batch_cost=1.2727, reader_cost=0.0003 | ETA 18:42:06 2020-11-03 04:57:02 [INFO] [TRAIN] epoch=74, iter=27200/80000, loss=0.1283, lr=0.006880, batch_cost=1.2968, reader_cost=0.0119 | ETA 19:01:11 2020-11-03 04:59:11 [INFO] [TRAIN] epoch=74, iter=27300/80000, loss=0.1271, lr=0.006868, batch_cost=1.2842, reader_cost=0.0002 | ETA 18:47:59 2020-11-03 05:01:19 [INFO] [TRAIN] epoch=74, iter=27400/80000, loss=0.1031, lr=0.006857, batch_cost=1.2835, reader_cost=0.0008 | ETA 18:45:13 2020-11-03 05:03:28 [INFO] [TRAIN] epoch=74, iter=27500/80000, loss=0.1472, lr=0.006845, batch_cost=1.2870, reader_cost=0.0002 | ETA 18:46:06 2020-11-03 05:05:35 [INFO] [TRAIN] epoch=75, iter=27600/80000, loss=0.1210, lr=0.006833, batch_cost=1.2754, reader_cost=0.0096 | ETA 18:33:49 2020-11-03 05:07:52 [INFO] [TRAIN] epoch=75, iter=27700/80000, loss=0.1250, lr=0.006821, batch_cost=1.3710, reader_cost=0.0002 | ETA 19:55:04 2020-11-03 05:10:01 [INFO] [TRAIN] epoch=75, iter=27800/80000, loss=0.1311, lr=0.006810, batch_cost=1.2845, reader_cost=0.0002 | ETA 18:37:29 2020-11-03 05:12:12 [INFO] [TRAIN] epoch=75, iter=27900/80000, loss=0.1356, lr=0.006798, batch_cost=1.3156, reader_cost=0.0002 | ETA 19:02:21 2020-11-03 05:14:24 [INFO] [TRAIN] epoch=76, iter=28000/80000, loss=0.1418, lr=0.006786, batch_cost=1.3145, reader_cost=0.0130 | ETA 18:59:12 2020-11-03 05:16:34 [INFO] [TRAIN] epoch=76, iter=28100/80000, loss=0.1307, lr=0.006774, batch_cost=1.3075, reader_cost=0.0002 | ETA 18:50:56 2020-11-03 05:18:44 [INFO] [TRAIN] epoch=76, iter=28200/80000, loss=0.1351, lr=0.006763, batch_cost=1.2906, reader_cost=0.0001 | ETA 18:34:13 2020-11-03 05:20:50 [INFO] [TRAIN] epoch=77, iter=28300/80000, loss=0.1337, lr=0.006751, batch_cost=1.2662, reader_cost=0.0088 | ETA 18:11:01 2020-11-03 05:22:58 [INFO] [TRAIN] epoch=77, iter=28400/80000, loss=0.1151, lr=0.006739, batch_cost=1.2775, reader_cost=0.0002 | ETA 18:18:37 2020-11-03 05:25:05 [INFO] [TRAIN] epoch=77, iter=28500/80000, loss=0.1336, lr=0.006727, batch_cost=1.2705, reader_cost=0.0002 | ETA 18:10:28 2020-11-03 05:27:10 [INFO] [TRAIN] epoch=77, iter=28600/80000, loss=0.1381, lr=0.006716, batch_cost=1.2497, reader_cost=0.0001 | ETA 17:50:37 2020-11-03 05:29:17 [INFO] [TRAIN] epoch=78, iter=28700/80000, loss=0.1395, lr=0.006704, batch_cost=1.2751, reader_cost=0.0107 | ETA 18:10:12 2020-11-03 05:31:26 [INFO] [TRAIN] epoch=78, iter=28800/80000, loss=0.1222, lr=0.006692, batch_cost=1.2887, reader_cost=0.0002 | ETA 18:19:42 2020-11-03 05:33:34 [INFO] [TRAIN] epoch=78, iter=28900/80000, loss=0.1104, lr=0.006680, batch_cost=1.2736, reader_cost=0.0009 | ETA 18:04:42 2020-11-03 05:35:39 [INFO] [TRAIN] epoch=78, iter=29000/80000, loss=0.1384, lr=0.006669, batch_cost=1.2548, reader_cost=0.0001 | ETA 17:46:32 2020-11-03 05:37:44 [INFO] [TRAIN] epoch=79, iter=29100/80000, loss=0.1299, lr=0.006657, batch_cost=1.2456, reader_cost=0.0091 | ETA 17:36:39 2020-11-03 05:39:49 [INFO] [TRAIN] epoch=79, iter=29200/80000, loss=0.1039, lr=0.006645, batch_cost=1.2560, reader_cost=0.0002 | ETA 17:43:23 2020-11-03 05:41:53 [INFO] [TRAIN] epoch=79, iter=29300/80000, loss=0.1199, lr=0.006633, batch_cost=1.2406, reader_cost=0.0002 | ETA 17:28:18 2020-11-03 05:44:07 [INFO] [TRAIN] epoch=80, iter=29400/80000, loss=0.1348, lr=0.006622, batch_cost=1.3385, reader_cost=0.0104 | ETA 18:48:49 2020-11-03 05:46:15 [INFO] [TRAIN] epoch=80, iter=29500/80000, loss=0.1186, lr=0.006610, batch_cost=1.2744, reader_cost=0.0002 | ETA 17:52:39 2020-11-03 05:48:32 [INFO] [TRAIN] epoch=80, iter=29600/80000, loss=0.1216, lr=0.006598, batch_cost=1.3715, reader_cost=0.0002 | ETA 19:12:02 2020-11-03 05:50:43 [INFO] [TRAIN] epoch=80, iter=29700/80000, loss=0.1138, lr=0.006586, batch_cost=1.3144, reader_cost=0.0001 | ETA 18:21:53 2020-11-03 05:52:51 [INFO] [TRAIN] epoch=81, iter=29800/80000, loss=0.1329, lr=0.006574, batch_cost=1.2760, reader_cost=0.0090 | ETA 17:47:36 2020-11-03 05:55:00 [INFO] [TRAIN] epoch=81, iter=29900/80000, loss=0.1128, lr=0.006563, batch_cost=1.2949, reader_cost=0.0002 | ETA 18:01:13 2020-11-03 05:57:11 [INFO] [TRAIN] epoch=81, iter=30000/80000, loss=0.1003, lr=0.006551, batch_cost=1.3083, reader_cost=0.0002 | ETA 18:10:12 2020-11-03 05:59:18 [INFO] [TRAIN] epoch=81, iter=30100/80000, loss=0.1266, lr=0.006539, batch_cost=1.2655, reader_cost=0.0001 | ETA 17:32:28 2020-11-03 06:01:24 [INFO] [TRAIN] epoch=82, iter=30200/80000, loss=0.1480, lr=0.006527, batch_cost=1.2591, reader_cost=0.0094 | ETA 17:25:03 2020-11-03 06:03:33 [INFO] [TRAIN] epoch=82, iter=30300/80000, loss=0.1054, lr=0.006515, batch_cost=1.2931, reader_cost=0.0003 | ETA 17:51:06 2020-11-03 06:05:40 [INFO] [TRAIN] epoch=82, iter=30400/80000, loss=0.1380, lr=0.006504, batch_cost=1.2747, reader_cost=0.0002 | ETA 17:33:46 2020-11-03 06:07:48 [INFO] [TRAIN] epoch=82, iter=30500/80000, loss=0.1165, lr=0.006492, batch_cost=1.2740, reader_cost=0.0002 | ETA 17:31:03 2020-11-03 06:09:58 [INFO] [TRAIN] epoch=83, iter=30600/80000, loss=0.1152, lr=0.006480, batch_cost=1.3018, reader_cost=0.0091 | ETA 17:51:47 2020-11-03 06:12:06 [INFO] [TRAIN] epoch=83, iter=30700/80000, loss=0.1158, lr=0.006468, batch_cost=1.2791, reader_cost=0.0002 | ETA 17:31:01 2020-11-03 06:14:15 [INFO] [TRAIN] epoch=83, iter=30800/80000, loss=0.1081, lr=0.006456, batch_cost=1.2948, reader_cost=0.0001 | ETA 17:41:44 2020-11-03 06:16:29 [INFO] [TRAIN] epoch=84, iter=30900/80000, loss=0.1189, lr=0.006445, batch_cost=1.3313, reader_cost=0.0128 | ETA 18:09:26 2020-11-03 06:18:45 [INFO] [TRAIN] epoch=84, iter=31000/80000, loss=0.1092, lr=0.006433, batch_cost=1.3683, reader_cost=0.0002 | ETA 18:37:26 2020-11-03 06:20:52 [INFO] [TRAIN] epoch=84, iter=31100/80000, loss=0.1118, lr=0.006421, batch_cost=1.2655, reader_cost=0.0004 | ETA 17:11:23 2020-11-03 06:22:58 [INFO] [TRAIN] epoch=84, iter=31200/80000, loss=0.1175, lr=0.006409, batch_cost=1.2620, reader_cost=0.0001 | ETA 17:06:24 2020-11-03 06:25:09 [INFO] [TRAIN] epoch=85, iter=31300/80000, loss=0.1334, lr=0.006397, batch_cost=1.3093, reader_cost=0.0127 | ETA 17:42:43 2020-11-03 06:27:14 [INFO] [TRAIN] epoch=85, iter=31400/80000, loss=0.1208, lr=0.006386, batch_cost=1.2455, reader_cost=0.0002 | ETA 16:48:50 2020-11-03 06:29:22 [INFO] [TRAIN] epoch=85, iter=31500/80000, loss=0.1290, lr=0.006374, batch_cost=1.2816, reader_cost=0.0002 | ETA 17:15:56 2020-11-03 06:31:31 [INFO] [TRAIN] epoch=85, iter=31600/80000, loss=0.1341, lr=0.006362, batch_cost=1.2887, reader_cost=0.0002 | ETA 17:19:33 2020-11-03 06:33:38 [INFO] [TRAIN] epoch=86, iter=31700/80000, loss=0.1405, lr=0.006350, batch_cost=1.2763, reader_cost=0.0087 | ETA 17:07:25 2020-11-03 06:35:45 [INFO] [TRAIN] epoch=86, iter=31800/80000, loss=0.1108, lr=0.006338, batch_cost=1.2655, reader_cost=0.0009 | ETA 16:56:34 2020-11-03 06:37:51 [INFO] [TRAIN] epoch=86, iter=31900/80000, loss=0.1117, lr=0.006326, batch_cost=1.2603, reader_cost=0.0001 | ETA 16:50:18 2020-11-03 06:39:59 [INFO] [TRAIN] epoch=87, iter=32000/80000, loss=0.1401, lr=0.006315, batch_cost=1.2823, reader_cost=0.0127 | ETA 17:05:49 2020-11-03 06:40:05 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-03 06:45:36 [INFO] [EVAL] #Images=500 mIoU=0.7837 Acc=0.9603 Kappa=0.9485 2020-11-03 06:45:36 [INFO] [EVAL] Category IoU: [0.98 0.8438 0.9247 0.5738 0.6397 0.6455 0.7066 0.7891 0.9232 0.6392 0.9422 0.8295 0.6476 0.947 0.7755 0.8381 0.7936 0.6701 0.7812] 2020-11-03 06:45:36 [INFO] [EVAL] Category Acc: [0.9907 0.9084 0.9549 0.844 0.8368 0.7842 0.7981 0.9021 0.9549 0.8689 0.9666 0.9113 0.7919 0.974 0.847 0.9749 0.8702 0.7991 0.8828] 2020-11-03 06:45:39 [INFO] [EVAL] The model with the best validation mIoU (0.7837) was saved at iter 32000. 2020-11-03 06:47:45 [INFO] [TRAIN] epoch=87, iter=32100/80000, loss=0.1082, lr=0.006303, batch_cost=1.2575, reader_cost=0.0002 | ETA 16:43:53 2020-11-03 06:49:49 [INFO] [TRAIN] epoch=87, iter=32200/80000, loss=0.1215, lr=0.006291, batch_cost=1.2452, reader_cost=0.0001 | ETA 16:31:58 2020-11-03 06:51:53 [INFO] [TRAIN] epoch=87, iter=32300/80000, loss=0.1395, lr=0.006279, batch_cost=1.2385, reader_cost=0.0002 | ETA 16:24:36 2020-11-03 06:54:01 [INFO] [TRAIN] epoch=88, iter=32400/80000, loss=0.1452, lr=0.006267, batch_cost=1.2837, reader_cost=0.0115 | ETA 16:58:23 2020-11-03 06:56:09 [INFO] [TRAIN] epoch=88, iter=32500/80000, loss=0.1025, lr=0.006255, batch_cost=1.2720, reader_cost=0.0002 | ETA 16:46:58 2020-11-03 06:58:26 [INFO] [TRAIN] epoch=88, iter=32600/80000, loss=0.1023, lr=0.006243, batch_cost=1.3768, reader_cost=0.0003 | ETA 18:07:42 2020-11-03 07:00:34 [INFO] [TRAIN] epoch=88, iter=32700/80000, loss=0.1149, lr=0.006232, batch_cost=1.2785, reader_cost=0.0001 | ETA 16:47:50 2020-11-03 07:02:39 [INFO] [TRAIN] epoch=89, iter=32800/80000, loss=0.1180, lr=0.006220, batch_cost=1.2488, reader_cost=0.0103 | ETA 16:22:23 2020-11-03 07:04:46 [INFO] [TRAIN] epoch=89, iter=32900/80000, loss=0.1143, lr=0.006208, batch_cost=1.2661, reader_cost=0.0003 | ETA 16:33:53 2020-11-03 07:06:52 [INFO] [TRAIN] epoch=89, iter=33000/80000, loss=0.1339, lr=0.006196, batch_cost=1.2588, reader_cost=0.0001 | ETA 16:26:03 2020-11-03 07:09:02 [INFO] [TRAIN] epoch=89, iter=33100/80000, loss=0.1459, lr=0.006184, batch_cost=1.2997, reader_cost=0.0002 | ETA 16:55:56 2020-11-03 07:11:10 [INFO] [TRAIN] epoch=90, iter=33200/80000, loss=0.1133, lr=0.006172, batch_cost=1.2867, reader_cost=0.0112 | ETA 16:43:39 2020-11-03 07:13:17 [INFO] [TRAIN] epoch=90, iter=33300/80000, loss=0.1103, lr=0.006160, batch_cost=1.2683, reader_cost=0.0003 | ETA 16:27:08 2020-11-03 07:15:23 [INFO] [TRAIN] epoch=90, iter=33400/80000, loss=0.1404, lr=0.006149, batch_cost=1.2557, reader_cost=0.0013 | ETA 16:15:13 2020-11-03 07:17:29 [INFO] [TRAIN] epoch=91, iter=33500/80000, loss=0.1273, lr=0.006137, batch_cost=1.2647, reader_cost=0.0121 | ETA 16:20:06 2020-11-03 07:19:42 [INFO] [TRAIN] epoch=91, iter=33600/80000, loss=0.1050, lr=0.006125, batch_cost=1.3267, reader_cost=0.0002 | ETA 17:05:58 2020-11-03 07:21:50 [INFO] [TRAIN] epoch=91, iter=33700/80000, loss=0.1064, lr=0.006113, batch_cost=1.2829, reader_cost=0.0006 | ETA 16:29:58 2020-11-03 07:23:56 [INFO] [TRAIN] epoch=91, iter=33800/80000, loss=0.1281, lr=0.006101, batch_cost=1.2575, reader_cost=0.0001 | ETA 16:08:15 2020-11-03 07:26:03 [INFO] [TRAIN] epoch=92, iter=33900/80000, loss=0.1219, lr=0.006089, batch_cost=1.2684, reader_cost=0.0104 | ETA 16:14:35 2020-11-03 07:28:08 [INFO] [TRAIN] epoch=92, iter=34000/80000, loss=0.1105, lr=0.006077, batch_cost=1.2487, reader_cost=0.0002 | ETA 15:57:19 2020-11-03 07:30:11 [INFO] [TRAIN] epoch=92, iter=34100/80000, loss=0.1268, lr=0.006065, batch_cost=1.2307, reader_cost=0.0002 | ETA 15:41:28 2020-11-03 07:32:18 [INFO] [TRAIN] epoch=92, iter=34200/80000, loss=0.1184, lr=0.006053, batch_cost=1.2755, reader_cost=0.0002 | ETA 16:13:37 2020-11-03 07:34:25 [INFO] [TRAIN] epoch=93, iter=34300/80000, loss=0.1267, lr=0.006042, batch_cost=1.2673, reader_cost=0.0146 | ETA 16:05:16 2020-11-03 07:36:32 [INFO] [TRAIN] epoch=93, iter=34400/80000, loss=0.0916, lr=0.006030, batch_cost=1.2697, reader_cost=0.0002 | ETA 16:04:58 2020-11-03 07:38:39 [INFO] [TRAIN] epoch=93, iter=34500/80000, loss=0.1058, lr=0.006018, batch_cost=1.2670, reader_cost=0.0001 | ETA 16:00:49 2020-11-03 07:40:49 [INFO] [TRAIN] epoch=94, iter=34600/80000, loss=0.1206, lr=0.006006, batch_cost=1.3012, reader_cost=0.0091 | ETA 16:24:33 2020-11-03 07:42:55 [INFO] [TRAIN] epoch=94, iter=34700/80000, loss=0.1122, lr=0.005994, batch_cost=1.2604, reader_cost=0.0002 | ETA 15:51:37 2020-11-03 07:45:03 [INFO] [TRAIN] epoch=94, iter=34800/80000, loss=0.1070, lr=0.005982, batch_cost=1.2853, reader_cost=0.0001 | ETA 16:08:17 2020-11-03 07:47:16 [INFO] [TRAIN] epoch=94, iter=34900/80000, loss=0.1146, lr=0.005970, batch_cost=1.3234, reader_cost=0.0002 | ETA 16:34:43 2020-11-03 07:49:23 [INFO] [TRAIN] epoch=95, iter=35000/80000, loss=0.1105, lr=0.005958, batch_cost=1.2719, reader_cost=0.0098 | ETA 15:53:57 2020-11-03 07:51:28 [INFO] [TRAIN] epoch=95, iter=35100/80000, loss=0.1114, lr=0.005946, batch_cost=1.2553, reader_cost=0.0002 | ETA 15:39:22 2020-11-03 07:53:36 [INFO] [TRAIN] epoch=95, iter=35200/80000, loss=0.1073, lr=0.005934, batch_cost=1.2739, reader_cost=0.0005 | ETA 15:51:10 2020-11-03 07:55:46 [INFO] [TRAIN] epoch=95, iter=35300/80000, loss=0.1205, lr=0.005922, batch_cost=1.3027, reader_cost=0.0002 | ETA 16:10:31 2020-11-03 07:57:54 [INFO] [TRAIN] epoch=96, iter=35400/80000, loss=0.1171, lr=0.005911, batch_cost=1.2774, reader_cost=0.0114 | ETA 15:49:30 2020-11-03 08:00:01 [INFO] [TRAIN] epoch=96, iter=35500/80000, loss=0.1127, lr=0.005899, batch_cost=1.2681, reader_cost=0.0003 | ETA 15:40:30 2020-11-03 08:02:06 [INFO] [TRAIN] epoch=96, iter=35600/80000, loss=0.1175, lr=0.005887, batch_cost=1.2507, reader_cost=0.0001 | ETA 15:25:29 2020-11-03 08:04:10 [INFO] [TRAIN] epoch=96, iter=35700/80000, loss=0.1215, lr=0.005875, batch_cost=1.2421, reader_cost=0.0001 | ETA 15:17:02 2020-11-03 08:06:18 [INFO] [TRAIN] epoch=97, iter=35800/80000, loss=0.1093, lr=0.005863, batch_cost=1.2835, reader_cost=0.0125 | ETA 15:45:31 2020-11-03 08:08:25 [INFO] [TRAIN] epoch=97, iter=35900/80000, loss=0.0957, lr=0.005851, batch_cost=1.2651, reader_cost=0.0002 | ETA 15:29:52 2020-11-03 08:10:31 [INFO] [TRAIN] epoch=97, iter=36000/80000, loss=0.1087, lr=0.005839, batch_cost=1.2663, reader_cost=0.0001 | ETA 15:28:35 2020-11-03 08:12:42 [INFO] [TRAIN] epoch=98, iter=36100/80000, loss=0.1308, lr=0.005827, batch_cost=1.3052, reader_cost=0.0136 | ETA 15:54:57 2020-11-03 08:14:50 [INFO] [TRAIN] epoch=98, iter=36200/80000, loss=0.1100, lr=0.005815, batch_cost=1.2806, reader_cost=0.0002 | ETA 15:34:52 2020-11-03 08:17:09 [INFO] [TRAIN] epoch=98, iter=36300/80000, loss=0.1247, lr=0.005803, batch_cost=1.3866, reader_cost=0.0003 | ETA 16:49:56 2020-11-03 08:19:15 [INFO] [TRAIN] epoch=98, iter=36400/80000, loss=0.1420, lr=0.005791, batch_cost=1.2608, reader_cost=0.0001 | ETA 15:16:09 2020-11-03 08:21:33 [INFO] [TRAIN] epoch=99, iter=36500/80000, loss=0.1252, lr=0.005779, batch_cost=1.3876, reader_cost=0.0103 | ETA 16:46:01 2020-11-03 08:23:40 [INFO] [TRAIN] epoch=99, iter=36600/80000, loss=0.1011, lr=0.005767, batch_cost=1.2690, reader_cost=0.0008 | ETA 15:17:56 2020-11-03 08:25:51 [INFO] [TRAIN] epoch=99, iter=36700/80000, loss=0.1273, lr=0.005755, batch_cost=1.3049, reader_cost=0.0002 | ETA 15:41:41 2020-11-03 08:27:58 [INFO] [TRAIN] epoch=99, iter=36800/80000, loss=0.1175, lr=0.005743, batch_cost=1.2709, reader_cost=0.0011 | ETA 15:15:01 2020-11-03 08:30:11 [INFO] [TRAIN] epoch=100, iter=36900/80000, loss=0.1118, lr=0.005731, batch_cost=1.3276, reader_cost=0.0112 | ETA 15:53:37 2020-11-03 08:32:18 [INFO] [TRAIN] epoch=100, iter=37000/80000, loss=0.1042, lr=0.005719, batch_cost=1.2771, reader_cost=0.0002 | ETA 15:15:13 2020-11-03 08:34:24 [INFO] [TRAIN] epoch=100, iter=37100/80000, loss=0.1001, lr=0.005707, batch_cost=1.2541, reader_cost=0.0003 | ETA 14:56:42 2020-11-03 08:36:29 [INFO] [TRAIN] epoch=100, iter=37200/80000, loss=0.1066, lr=0.005695, batch_cost=1.2528, reader_cost=0.0001 | ETA 14:53:40 2020-11-03 08:38:38 [INFO] [TRAIN] epoch=101, iter=37300/80000, loss=0.1121, lr=0.005683, batch_cost=1.2859, reader_cost=0.0108 | ETA 15:15:07 2020-11-03 08:40:43 [INFO] [TRAIN] epoch=101, iter=37400/80000, loss=0.0955, lr=0.005671, batch_cost=1.2507, reader_cost=0.0002 | ETA 14:48:01 2020-11-03 08:42:51 [INFO] [TRAIN] epoch=101, iter=37500/80000, loss=0.1187, lr=0.005660, batch_cost=1.2823, reader_cost=0.0002 | ETA 15:08:17 2020-11-03 08:44:56 [INFO] [TRAIN] epoch=102, iter=37600/80000, loss=0.1229, lr=0.005648, batch_cost=1.2524, reader_cost=0.0109 | ETA 14:45:00 2020-11-03 08:47:00 [INFO] [TRAIN] epoch=102, iter=37700/80000, loss=0.1185, lr=0.005636, batch_cost=1.2424, reader_cost=0.0002 | ETA 14:35:52 2020-11-03 08:49:08 [INFO] [TRAIN] epoch=102, iter=37800/80000, loss=0.1177, lr=0.005624, batch_cost=1.2791, reader_cost=0.0001 | ETA 14:59:38 2020-11-03 08:51:15 [INFO] [TRAIN] epoch=102, iter=37900/80000, loss=0.1300, lr=0.005612, batch_cost=1.2651, reader_cost=0.0001 | ETA 14:47:39 2020-11-03 08:53:25 [INFO] [TRAIN] epoch=103, iter=38000/80000, loss=0.1394, lr=0.005600, batch_cost=1.3048, reader_cost=0.0098 | ETA 15:13:22 2020-11-03 08:55:30 [INFO] [TRAIN] epoch=103, iter=38100/80000, loss=0.1139, lr=0.005588, batch_cost=1.2486, reader_cost=0.0002 | ETA 14:31:58 2020-11-03 08:57:38 [INFO] [TRAIN] epoch=103, iter=38200/80000, loss=0.1204, lr=0.005576, batch_cost=1.2761, reader_cost=0.0002 | ETA 14:49:00 2020-11-03 08:59:41 [INFO] [TRAIN] epoch=103, iter=38300/80000, loss=0.1307, lr=0.005564, batch_cost=1.2332, reader_cost=0.0001 | ETA 14:17:03 2020-11-03 09:01:49 [INFO] [TRAIN] epoch=104, iter=38400/80000, loss=0.1276, lr=0.005552, batch_cost=1.2745, reader_cost=0.0093 | ETA 14:43:39 2020-11-03 09:03:56 [INFO] [TRAIN] epoch=104, iter=38500/80000, loss=0.0966, lr=0.005540, batch_cost=1.2744, reader_cost=0.0002 | ETA 14:41:25 2020-11-03 09:06:04 [INFO] [TRAIN] epoch=104, iter=38600/80000, loss=0.1331, lr=0.005528, batch_cost=1.2757, reader_cost=0.0001 | ETA 14:40:14 2020-11-03 09:08:10 [INFO] [TRAIN] epoch=105, iter=38700/80000, loss=0.1270, lr=0.005515, batch_cost=1.2676, reader_cost=0.0110 | ETA 14:32:32 2020-11-03 09:10:17 [INFO] [TRAIN] epoch=105, iter=38800/80000, loss=0.1258, lr=0.005503, batch_cost=1.2643, reader_cost=0.0002 | ETA 14:28:09 2020-11-03 09:12:27 [INFO] [TRAIN] epoch=105, iter=38900/80000, loss=0.1091, lr=0.005491, batch_cost=1.3034, reader_cost=0.0019 | ETA 14:52:50 2020-11-03 09:14:32 [INFO] [TRAIN] epoch=105, iter=39000/80000, loss=0.1085, lr=0.005479, batch_cost=1.2524, reader_cost=0.0001 | ETA 14:15:46 2020-11-03 09:16:41 [INFO] [TRAIN] epoch=106, iter=39100/80000, loss=0.1120, lr=0.005467, batch_cost=1.2864, reader_cost=0.0103 | ETA 14:36:55 2020-11-03 09:18:50 [INFO] [TRAIN] epoch=106, iter=39200/80000, loss=0.1010, lr=0.005455, batch_cost=1.2845, reader_cost=0.0003 | ETA 14:33:25 2020-11-03 09:20:55 [INFO] [TRAIN] epoch=106, iter=39300/80000, loss=0.1058, lr=0.005443, batch_cost=1.2521, reader_cost=0.0001 | ETA 14:09:22 2020-11-03 09:22:59 [INFO] [TRAIN] epoch=106, iter=39400/80000, loss=0.1054, lr=0.005431, batch_cost=1.2430, reader_cost=0.0002 | ETA 14:01:04 2020-11-03 09:25:07 [INFO] [TRAIN] epoch=107, iter=39500/80000, loss=0.1210, lr=0.005419, batch_cost=1.2767, reader_cost=0.0143 | ETA 14:21:45 2020-11-03 09:27:14 [INFO] [TRAIN] epoch=107, iter=39600/80000, loss=0.0924, lr=0.005407, batch_cost=1.2763, reader_cost=0.0004 | ETA 14:19:22 2020-11-03 09:29:20 [INFO] [TRAIN] epoch=107, iter=39700/80000, loss=0.1056, lr=0.005395, batch_cost=1.2588, reader_cost=0.0001 | ETA 14:05:30 2020-11-03 09:31:25 [INFO] [TRAIN] epoch=107, iter=39800/80000, loss=0.1121, lr=0.005383, batch_cost=1.2470, reader_cost=0.0001 | ETA 13:55:31 2020-11-03 09:33:32 [INFO] [TRAIN] epoch=108, iter=39900/80000, loss=0.1124, lr=0.005371, batch_cost=1.2700, reader_cost=0.0093 | ETA 14:08:47 2020-11-03 09:35:41 [INFO] [TRAIN] epoch=108, iter=40000/80000, loss=0.0933, lr=0.005359, batch_cost=1.2906, reader_cost=0.0005 | ETA 14:20:22 2020-11-03 09:35:47 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-03 09:41:32 [INFO] [EVAL] #Images=500 mIoU=0.7822 Acc=0.9620 Kappa=0.9506 2020-11-03 09:41:32 [INFO] [EVAL] Category IoU: [0.9823 0.8574 0.9262 0.4621 0.6336 0.6559 0.712 0.8075 0.9257 0.6429 0.9469 0.8333 0.6494 0.954 0.8373 0.8513 0.7134 0.6807 0.79 ] 2020-11-03 09:41:32 [INFO] [EVAL] Category Acc: [0.9909 0.9287 0.9546 0.8526 0.8204 0.8224 0.8764 0.9146 0.9546 0.8713 0.9619 0.8901 0.8181 0.9715 0.9128 0.9277 0.8918 0.8408 0.8772] 2020-11-03 09:41:32 [INFO] [EVAL] The model with the best validation mIoU (0.7837) was saved at iter 32000. 2020-11-03 09:43:38 [INFO] [TRAIN] epoch=108, iter=40100/80000, loss=0.1110, lr=0.005347, batch_cost=1.2636, reader_cost=0.0001 | ETA 14:00:18 2020-11-03 09:45:44 [INFO] [TRAIN] epoch=109, iter=40200/80000, loss=0.1082, lr=0.005335, batch_cost=1.2542, reader_cost=0.0101 | ETA 13:51:57 2020-11-03 09:47:49 [INFO] [TRAIN] epoch=109, iter=40300/80000, loss=0.1069, lr=0.005323, batch_cost=1.2538, reader_cost=0.0002 | ETA 13:49:36 2020-11-03 09:49:56 [INFO] [TRAIN] epoch=109, iter=40400/80000, loss=0.1024, lr=0.005311, batch_cost=1.2681, reader_cost=0.0001 | ETA 13:56:55 2020-11-03 09:52:04 [INFO] [TRAIN] epoch=109, iter=40500/80000, loss=0.0948, lr=0.005299, batch_cost=1.2775, reader_cost=0.0002 | ETA 14:01:02 2020-11-03 09:54:10 [INFO] [TRAIN] epoch=110, iter=40600/80000, loss=0.1090, lr=0.005287, batch_cost=1.2623, reader_cost=0.0110 | ETA 13:48:53 2020-11-03 09:56:16 [INFO] [TRAIN] epoch=110, iter=40700/80000, loss=0.0948, lr=0.005275, batch_cost=1.2557, reader_cost=0.0002 | ETA 13:42:29 2020-11-03 09:58:24 [INFO] [TRAIN] epoch=110, iter=40800/80000, loss=0.1110, lr=0.005262, batch_cost=1.2860, reader_cost=0.0003 | ETA 14:00:10 2020-11-03 10:00:32 [INFO] [TRAIN] epoch=110, iter=40900/80000, loss=0.1143, lr=0.005250, batch_cost=1.2750, reader_cost=0.0001 | ETA 13:50:51 2020-11-03 10:02:41 [INFO] [TRAIN] epoch=111, iter=41000/80000, loss=0.1039, lr=0.005238, batch_cost=1.2965, reader_cost=0.0112 | ETA 14:02:43 2020-11-03 10:04:52 [INFO] [TRAIN] epoch=111, iter=41100/80000, loss=0.1046, lr=0.005226, batch_cost=1.3060, reader_cost=0.0006 | ETA 14:06:44 2020-11-03 10:06:57 [INFO] [TRAIN] epoch=111, iter=41200/80000, loss=0.1208, lr=0.005214, batch_cost=1.2518, reader_cost=0.0001 | ETA 13:29:29 2020-11-03 10:09:05 [INFO] [TRAIN] epoch=112, iter=41300/80000, loss=0.1226, lr=0.005202, batch_cost=1.2791, reader_cost=0.0102 | ETA 13:45:00 2020-11-03 10:11:12 [INFO] [TRAIN] epoch=112, iter=41400/80000, loss=0.1047, lr=0.005190, batch_cost=1.2661, reader_cost=0.0001 | ETA 13:34:32 2020-11-03 10:13:20 [INFO] [TRAIN] epoch=112, iter=41500/80000, loss=0.1013, lr=0.005178, batch_cost=1.2884, reader_cost=0.0001 | ETA 13:46:44 2020-11-03 10:15:31 [INFO] [TRAIN] epoch=112, iter=41600/80000, loss=0.0965, lr=0.005166, batch_cost=1.3027, reader_cost=0.0012 | ETA 13:53:42 2020-11-03 10:17:38 [INFO] [TRAIN] epoch=113, iter=41700/80000, loss=0.1247, lr=0.005154, batch_cost=1.2727, reader_cost=0.0118 | ETA 13:32:25 2020-11-03 10:19:44 [INFO] [TRAIN] epoch=113, iter=41800/80000, loss=0.0905, lr=0.005141, batch_cost=1.2554, reader_cost=0.0002 | ETA 13:19:16 2020-11-03 10:21:48 [INFO] [TRAIN] epoch=113, iter=41900/80000, loss=0.1108, lr=0.005129, batch_cost=1.2489, reader_cost=0.0001 | ETA 13:13:03 2020-11-03 10:23:53 [INFO] [TRAIN] epoch=113, iter=42000/80000, loss=0.1028, lr=0.005117, batch_cost=1.2483, reader_cost=0.0001 | ETA 13:10:35 2020-11-03 10:25:59 [INFO] [TRAIN] epoch=114, iter=42100/80000, loss=0.1051, lr=0.005105, batch_cost=1.2586, reader_cost=0.0097 | ETA 13:14:59 2020-11-03 10:28:06 [INFO] [TRAIN] epoch=114, iter=42200/80000, loss=0.1021, lr=0.005093, batch_cost=1.2690, reader_cost=0.0002 | ETA 13:19:28 2020-11-03 10:30:13 [INFO] [TRAIN] epoch=114, iter=42300/80000, loss=0.1120, lr=0.005081, batch_cost=1.2708, reader_cost=0.0002 | ETA 13:18:30 2020-11-03 10:32:18 [INFO] [TRAIN] epoch=114, iter=42400/80000, loss=0.1061, lr=0.005069, batch_cost=1.2466, reader_cost=0.0001 | ETA 13:01:10 2020-11-03 10:34:28 [INFO] [TRAIN] epoch=115, iter=42500/80000, loss=0.1045, lr=0.005057, batch_cost=1.3037, reader_cost=0.0098 | ETA 13:34:48 2020-11-03 10:36:34 [INFO] [TRAIN] epoch=115, iter=42600/80000, loss=0.0846, lr=0.005044, batch_cost=1.2571, reader_cost=0.0003 | ETA 13:03:36 2020-11-03 10:38:50 [INFO] [TRAIN] epoch=115, iter=42700/80000, loss=0.0989, lr=0.005032, batch_cost=1.3582, reader_cost=0.0002 | ETA 14:04:21 2020-11-03 10:41:00 [INFO] [TRAIN] epoch=116, iter=42800/80000, loss=0.1232, lr=0.005020, batch_cost=1.3027, reader_cost=0.0116 | ETA 13:27:40 2020-11-03 10:43:10 [INFO] [TRAIN] epoch=116, iter=42900/80000, loss=0.1030, lr=0.005008, batch_cost=1.2983, reader_cost=0.0001 | ETA 13:22:45 2020-11-03 10:45:17 [INFO] [TRAIN] epoch=116, iter=43000/80000, loss=0.1016, lr=0.004996, batch_cost=1.2699, reader_cost=0.0001 | ETA 13:03:04 2020-11-03 10:47:25 [INFO] [TRAIN] epoch=116, iter=43100/80000, loss=0.1005, lr=0.004984, batch_cost=1.2775, reader_cost=0.0005 | ETA 13:05:40 2020-11-03 10:49:32 [INFO] [TRAIN] epoch=117, iter=43200/80000, loss=0.1239, lr=0.004972, batch_cost=1.2772, reader_cost=0.0102 | ETA 13:03:20 2020-11-03 10:51:39 [INFO] [TRAIN] epoch=117, iter=43300/80000, loss=0.0978, lr=0.004959, batch_cost=1.2701, reader_cost=0.0002 | ETA 12:56:51 2020-11-03 10:53:45 [INFO] [TRAIN] epoch=117, iter=43400/80000, loss=0.1241, lr=0.004947, batch_cost=1.2603, reader_cost=0.0001 | ETA 12:48:48 2020-11-03 10:55:51 [INFO] [TRAIN] epoch=117, iter=43500/80000, loss=0.1024, lr=0.004935, batch_cost=1.2601, reader_cost=0.0002 | ETA 12:46:34 2020-11-03 10:58:00 [INFO] [TRAIN] epoch=118, iter=43600/80000, loss=0.1423, lr=0.004923, batch_cost=1.2838, reader_cost=0.0115 | ETA 12:58:51 2020-11-03 11:00:06 [INFO] [TRAIN] epoch=118, iter=43700/80000, loss=0.1274, lr=0.004911, batch_cost=1.2646, reader_cost=0.0003 | ETA 12:45:05 2020-11-03 11:02:29 [INFO] [TRAIN] epoch=118, iter=43800/80000, loss=0.1135, lr=0.004899, batch_cost=1.4264, reader_cost=0.0006 | ETA 14:20:34 2020-11-03 11:04:39 [INFO] [TRAIN] epoch=119, iter=43900/80000, loss=0.1251, lr=0.004886, batch_cost=1.3030, reader_cost=0.0151 | ETA 13:03:59 2020-11-03 11:06:50 [INFO] [TRAIN] epoch=119, iter=44000/80000, loss=0.1035, lr=0.004874, batch_cost=1.3061, reader_cost=0.0002 | ETA 13:03:40 2020-11-03 11:08:57 [INFO] [TRAIN] epoch=119, iter=44100/80000, loss=0.1057, lr=0.004862, batch_cost=1.2746, reader_cost=0.0002 | ETA 12:42:37 2020-11-03 11:11:05 [INFO] [TRAIN] epoch=119, iter=44200/80000, loss=0.1257, lr=0.004850, batch_cost=1.2797, reader_cost=0.0002 | ETA 12:43:32 2020-11-03 11:13:17 [INFO] [TRAIN] epoch=120, iter=44300/80000, loss=0.1084, lr=0.004838, batch_cost=1.3162, reader_cost=0.0104 | ETA 13:03:07 2020-11-03 11:15:24 [INFO] [TRAIN] epoch=120, iter=44400/80000, loss=0.0938, lr=0.004825, batch_cost=1.2685, reader_cost=0.0003 | ETA 12:32:39 2020-11-03 11:17:32 [INFO] [TRAIN] epoch=120, iter=44500/80000, loss=0.0998, lr=0.004813, batch_cost=1.2782, reader_cost=0.0014 | ETA 12:36:16 2020-11-03 11:19:45 [INFO] [TRAIN] epoch=120, iter=44600/80000, loss=0.1120, lr=0.004801, batch_cost=1.3364, reader_cost=0.0002 | ETA 13:08:27 2020-11-03 11:21:53 [INFO] [TRAIN] epoch=121, iter=44700/80000, loss=0.1037, lr=0.004789, batch_cost=1.2799, reader_cost=0.0119 | ETA 12:32:59 2020-11-03 11:23:56 [INFO] [TRAIN] epoch=121, iter=44800/80000, loss=0.1021, lr=0.004777, batch_cost=1.2242, reader_cost=0.0002 | ETA 11:58:12 2020-11-03 11:26:01 [INFO] [TRAIN] epoch=121, iter=44900/80000, loss=0.0984, lr=0.004764, batch_cost=1.2507, reader_cost=0.0001 | ETA 12:11:38 2020-11-03 11:28:07 [INFO] [TRAIN] epoch=121, iter=45000/80000, loss=0.1057, lr=0.004752, batch_cost=1.2684, reader_cost=0.0001 | ETA 12:19:55 2020-11-03 11:30:20 [INFO] [TRAIN] epoch=122, iter=45100/80000, loss=0.0992, lr=0.004740, batch_cost=1.3246, reader_cost=0.0115 | ETA 12:50:27 2020-11-03 11:32:27 [INFO] [TRAIN] epoch=122, iter=45200/80000, loss=0.1098, lr=0.004728, batch_cost=1.2697, reader_cost=0.0003 | ETA 12:16:26 2020-11-03 11:34:40 [INFO] [TRAIN] epoch=122, iter=45300/80000, loss=0.1237, lr=0.004715, batch_cost=1.3350, reader_cost=0.0004 | ETA 12:52:04 2020-11-03 11:36:48 [INFO] [TRAIN] epoch=123, iter=45400/80000, loss=0.1138, lr=0.004703, batch_cost=1.2735, reader_cost=0.0131 | ETA 12:14:24 2020-11-03 11:38:57 [INFO] [TRAIN] epoch=123, iter=45500/80000, loss=0.0911, lr=0.004691, batch_cost=1.2969, reader_cost=0.0005 | ETA 12:25:41 2020-11-03 11:41:12 [INFO] [TRAIN] epoch=123, iter=45600/80000, loss=0.1049, lr=0.004679, batch_cost=1.3494, reader_cost=0.0002 | ETA 12:53:38 2020-11-03 11:43:18 [INFO] [TRAIN] epoch=123, iter=45700/80000, loss=0.1094, lr=0.004667, batch_cost=1.2594, reader_cost=0.0001 | ETA 11:59:56 2020-11-03 11:45:24 [INFO] [TRAIN] epoch=124, iter=45800/80000, loss=0.1171, lr=0.004654, batch_cost=1.2587, reader_cost=0.0114 | ETA 11:57:26 2020-11-03 11:47:34 [INFO] [TRAIN] epoch=124, iter=45900/80000, loss=0.0970, lr=0.004642, batch_cost=1.2955, reader_cost=0.0001 | ETA 12:16:17 2020-11-03 11:49:42 [INFO] [TRAIN] epoch=124, iter=46000/80000, loss=0.1004, lr=0.004630, batch_cost=1.2846, reader_cost=0.0003 | ETA 12:07:54 2020-11-03 11:51:50 [INFO] [TRAIN] epoch=124, iter=46100/80000, loss=0.1016, lr=0.004618, batch_cost=1.2784, reader_cost=0.0001 | ETA 12:02:17 2020-11-03 11:53:59 [INFO] [TRAIN] epoch=125, iter=46200/80000, loss=0.1140, lr=0.004605, batch_cost=1.2924, reader_cost=0.0146 | ETA 12:08:02 2020-11-03 11:56:07 [INFO] [TRAIN] epoch=125, iter=46300/80000, loss=0.1026, lr=0.004593, batch_cost=1.2757, reader_cost=0.0002 | ETA 11:56:32 2020-11-03 11:58:14 [INFO] [TRAIN] epoch=125, iter=46400/80000, loss=0.0926, lr=0.004581, batch_cost=1.2753, reader_cost=0.0002 | ETA 11:54:09 2020-11-03 12:00:25 [INFO] [TRAIN] epoch=125, iter=46500/80000, loss=0.1124, lr=0.004568, batch_cost=1.3092, reader_cost=0.0002 | ETA 12:10:59 2020-11-03 12:02:37 [INFO] [TRAIN] epoch=126, iter=46600/80000, loss=0.1009, lr=0.004556, batch_cost=1.3133, reader_cost=0.0117 | ETA 12:11:04 2020-11-03 12:04:41 [INFO] [TRAIN] epoch=126, iter=46700/80000, loss=0.1001, lr=0.004544, batch_cost=1.2483, reader_cost=0.0002 | ETA 11:32:47 2020-11-03 12:06:53 [INFO] [TRAIN] epoch=126, iter=46800/80000, loss=0.1064, lr=0.004532, batch_cost=1.3123, reader_cost=0.0002 | ETA 12:06:06 2020-11-03 12:08:59 [INFO] [TRAIN] epoch=127, iter=46900/80000, loss=0.1066, lr=0.004519, batch_cost=1.2622, reader_cost=0.0113 | ETA 11:36:19 2020-11-03 12:11:06 [INFO] [TRAIN] epoch=127, iter=47000/80000, loss=0.1025, lr=0.004507, batch_cost=1.2708, reader_cost=0.0002 | ETA 11:38:57 2020-11-03 12:13:15 [INFO] [TRAIN] epoch=127, iter=47100/80000, loss=0.0971, lr=0.004495, batch_cost=1.2938, reader_cost=0.0006 | ETA 11:49:26 2020-11-03 12:15:21 [INFO] [TRAIN] epoch=127, iter=47200/80000, loss=0.1039, lr=0.004482, batch_cost=1.2583, reader_cost=0.0001 | ETA 11:27:50 2020-11-03 12:17:34 [INFO] [TRAIN] epoch=128, iter=47300/80000, loss=0.1069, lr=0.004470, batch_cost=1.3265, reader_cost=0.0104 | ETA 12:02:56 2020-11-03 12:19:43 [INFO] [TRAIN] epoch=128, iter=47400/80000, loss=0.1072, lr=0.004458, batch_cost=1.2912, reader_cost=0.0005 | ETA 11:41:33 2020-11-03 12:21:51 [INFO] [TRAIN] epoch=128, iter=47500/80000, loss=0.1299, lr=0.004446, batch_cost=1.2763, reader_cost=0.0001 | ETA 11:31:19 2020-11-03 12:24:01 [INFO] [TRAIN] epoch=128, iter=47600/80000, loss=0.1115, lr=0.004433, batch_cost=1.2998, reader_cost=0.0003 | ETA 11:41:52 2020-11-03 12:26:09 [INFO] [TRAIN] epoch=129, iter=47700/80000, loss=0.1034, lr=0.004421, batch_cost=1.2801, reader_cost=0.0152 | ETA 11:29:06 2020-11-03 12:28:14 [INFO] [TRAIN] epoch=129, iter=47800/80000, loss=0.0976, lr=0.004409, batch_cost=1.2521, reader_cost=0.0001 | ETA 11:11:58 2020-11-03 12:30:20 [INFO] [TRAIN] epoch=129, iter=47900/80000, loss=0.1122, lr=0.004396, batch_cost=1.2568, reader_cost=0.0003 | ETA 11:12:22 2020-11-03 12:32:28 [INFO] [TRAIN] epoch=130, iter=48000/80000, loss=0.1122, lr=0.004384, batch_cost=1.2854, reader_cost=0.0165 | ETA 11:25:31 2020-11-03 12:32:34 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-03 12:38:08 [INFO] [EVAL] #Images=500 mIoU=0.7928 Acc=0.9624 Kappa=0.9512 2020-11-03 12:38:08 [INFO] [EVAL] Category IoU: [0.9819 0.8557 0.9289 0.5139 0.6365 0.6649 0.725 0.8094 0.9252 0.6354 0.9496 0.8373 0.6583 0.9534 0.8484 0.8857 0.7922 0.674 0.787 ] 2020-11-03 12:38:08 [INFO] [EVAL] Category Acc: [0.9915 0.922 0.9622 0.8482 0.8072 0.8127 0.8436 0.895 0.9527 0.828 0.9701 0.9045 0.8077 0.9713 0.9315 0.9534 0.8447 0.7696 0.8602] 2020-11-03 12:38:12 [INFO] [EVAL] The model with the best validation mIoU (0.7928) was saved at iter 48000. 2020-11-03 12:40:20 [INFO] [TRAIN] epoch=130, iter=48100/80000, loss=0.0994, lr=0.004372, batch_cost=1.2813, reader_cost=0.0001 | ETA 11:21:13 2020-11-03 12:42:33 [INFO] [TRAIN] epoch=130, iter=48200/80000, loss=0.1034, lr=0.004359, batch_cost=1.3367, reader_cost=0.0002 | ETA 11:48:26 2020-11-03 12:44:41 [INFO] [TRAIN] epoch=130, iter=48300/80000, loss=0.1158, lr=0.004347, batch_cost=1.2787, reader_cost=0.0002 | ETA 11:15:33 2020-11-03 12:46:52 [INFO] [TRAIN] epoch=131, iter=48400/80000, loss=0.1158, lr=0.004335, batch_cost=1.3066, reader_cost=0.0129 | ETA 11:28:08 2020-11-03 12:49:00 [INFO] [TRAIN] epoch=131, iter=48500/80000, loss=0.0922, lr=0.004322, batch_cost=1.2843, reader_cost=0.0002 | ETA 11:14:15 2020-11-03 12:51:10 [INFO] [TRAIN] epoch=131, iter=48600/80000, loss=0.0895, lr=0.004310, batch_cost=1.2940, reader_cost=0.0002 | ETA 11:17:12 2020-11-03 12:53:18 [INFO] [TRAIN] epoch=131, iter=48700/80000, loss=0.1084, lr=0.004298, batch_cost=1.2846, reader_cost=0.0002 | ETA 11:10:08 2020-11-03 12:55:31 [INFO] [TRAIN] epoch=132, iter=48800/80000, loss=0.1003, lr=0.004285, batch_cost=1.3254, reader_cost=0.0116 | ETA 11:29:12 2020-11-03 12:57:39 [INFO] [TRAIN] epoch=132, iter=48900/80000, loss=0.1025, lr=0.004273, batch_cost=1.2813, reader_cost=0.0003 | ETA 11:04:08 2020-11-03 12:59:45 [INFO] [TRAIN] epoch=132, iter=49000/80000, loss=0.0920, lr=0.004260, batch_cost=1.2612, reader_cost=0.0002 | ETA 10:51:36 2020-11-03 13:01:57 [INFO] [TRAIN] epoch=132, iter=49100/80000, loss=0.1145, lr=0.004248, batch_cost=1.3162, reader_cost=0.0001 | ETA 11:17:51 2020-11-03 13:04:04 [INFO] [TRAIN] epoch=133, iter=49200/80000, loss=0.0958, lr=0.004236, batch_cost=1.2698, reader_cost=0.0121 | ETA 10:51:49 2020-11-03 13:06:22 [INFO] [TRAIN] epoch=133, iter=49300/80000, loss=0.0864, lr=0.004223, batch_cost=1.3849, reader_cost=0.0002 | ETA 11:48:36 2020-11-03 13:08:33 [INFO] [TRAIN] epoch=133, iter=49400/80000, loss=0.1062, lr=0.004211, batch_cost=1.3088, reader_cost=0.0003 | ETA 11:07:30 2020-11-03 13:10:45 [INFO] [TRAIN] epoch=134, iter=49500/80000, loss=0.1018, lr=0.004199, batch_cost=1.3151, reader_cost=0.0101 | ETA 11:08:30 2020-11-03 13:12:52 [INFO] [TRAIN] epoch=134, iter=49600/80000, loss=0.0981, lr=0.004186, batch_cost=1.2795, reader_cost=0.0003 | ETA 10:48:16 2020-11-03 13:15:00 [INFO] [TRAIN] epoch=134, iter=49700/80000, loss=0.1055, lr=0.004174, batch_cost=1.2759, reader_cost=0.0002 | ETA 10:44:20 2020-11-03 13:17:06 [INFO] [TRAIN] epoch=134, iter=49800/80000, loss=0.1071, lr=0.004161, batch_cost=1.2558, reader_cost=0.0001 | ETA 10:32:04 2020-11-03 13:19:13 [INFO] [TRAIN] epoch=135, iter=49900/80000, loss=0.1023, lr=0.004149, batch_cost=1.2733, reader_cost=0.0132 | ETA 10:38:46 2020-11-03 13:21:17 [INFO] [TRAIN] epoch=135, iter=50000/80000, loss=0.0914, lr=0.004137, batch_cost=1.2406, reader_cost=0.0002 | ETA 10:20:18 2020-11-03 13:23:27 [INFO] [TRAIN] epoch=135, iter=50100/80000, loss=0.0974, lr=0.004124, batch_cost=1.3000, reader_cost=0.0002 | ETA 10:47:50 2020-11-03 13:25:36 [INFO] [TRAIN] epoch=135, iter=50200/80000, loss=0.1169, lr=0.004112, batch_cost=1.2868, reader_cost=0.0002 | ETA 10:39:05 2020-11-03 13:27:45 [INFO] [TRAIN] epoch=136, iter=50300/80000, loss=0.1006, lr=0.004099, batch_cost=1.2952, reader_cost=0.0133 | ETA 10:41:07 2020-11-03 13:29:52 [INFO] [TRAIN] epoch=136, iter=50400/80000, loss=0.0974, lr=0.004087, batch_cost=1.2646, reader_cost=0.0002 | ETA 10:23:53 2020-11-03 13:31:57 [INFO] [TRAIN] epoch=136, iter=50500/80000, loss=0.0922, lr=0.004074, batch_cost=1.2550, reader_cost=0.0001 | ETA 10:17:02 2020-11-03 13:34:05 [INFO] [TRAIN] epoch=137, iter=50600/80000, loss=0.1101, lr=0.004062, batch_cost=1.2765, reader_cost=0.0098 | ETA 10:25:29 2020-11-03 13:36:09 [INFO] [TRAIN] epoch=137, iter=50700/80000, loss=0.0968, lr=0.004050, batch_cost=1.2422, reader_cost=0.0002 | ETA 10:06:35 2020-11-03 13:38:28 [INFO] [TRAIN] epoch=137, iter=50800/80000, loss=0.0884, lr=0.004037, batch_cost=1.3912, reader_cost=0.0004 | ETA 11:17:04 2020-11-03 13:40:33 [INFO] [TRAIN] epoch=137, iter=50900/80000, loss=0.1077, lr=0.004025, batch_cost=1.2512, reader_cost=0.0002 | ETA 10:06:49 2020-11-03 13:42:40 [INFO] [TRAIN] epoch=138, iter=51000/80000, loss=0.1024, lr=0.004012, batch_cost=1.2681, reader_cost=0.0098 | ETA 10:12:55 2020-11-03 13:44:45 [INFO] [TRAIN] epoch=138, iter=51100/80000, loss=0.0892, lr=0.004000, batch_cost=1.2456, reader_cost=0.0002 | ETA 09:59:56 2020-11-03 13:46:52 [INFO] [TRAIN] epoch=138, iter=51200/80000, loss=0.0906, lr=0.003987, batch_cost=1.2694, reader_cost=0.0001 | ETA 10:09:18 2020-11-03 13:48:56 [INFO] [TRAIN] epoch=138, iter=51300/80000, loss=0.1494, lr=0.003975, batch_cost=1.2415, reader_cost=0.0001 | ETA 09:53:50 2020-11-03 13:51:05 [INFO] [TRAIN] epoch=139, iter=51400/80000, loss=0.1068, lr=0.003962, batch_cost=1.2923, reader_cost=0.0111 | ETA 10:16:00 2020-11-03 13:53:12 [INFO] [TRAIN] epoch=139, iter=51500/80000, loss=0.1047, lr=0.003950, batch_cost=1.2665, reader_cost=0.0002 | ETA 10:01:34 2020-11-03 13:55:16 [INFO] [TRAIN] epoch=139, iter=51600/80000, loss=0.1028, lr=0.003937, batch_cost=1.2475, reader_cost=0.0001 | ETA 09:50:29 2020-11-03 13:57:26 [INFO] [TRAIN] epoch=139, iter=51700/80000, loss=0.1014, lr=0.003925, batch_cost=1.2952, reader_cost=0.0002 | ETA 10:10:53 2020-11-03 13:59:36 [INFO] [TRAIN] epoch=140, iter=51800/80000, loss=0.0948, lr=0.003913, batch_cost=1.2972, reader_cost=0.0111 | ETA 10:09:40 2020-11-03 14:01:43 [INFO] [TRAIN] epoch=140, iter=51900/80000, loss=0.0804, lr=0.003900, batch_cost=1.2729, reader_cost=0.0002 | ETA 09:56:07 2020-11-03 14:03:52 [INFO] [TRAIN] epoch=140, iter=52000/80000, loss=0.0983, lr=0.003888, batch_cost=1.2933, reader_cost=0.0001 | ETA 10:03:31 2020-11-03 14:06:00 [INFO] [TRAIN] epoch=141, iter=52100/80000, loss=0.1012, lr=0.003875, batch_cost=1.2722, reader_cost=0.0122 | ETA 09:51:34 2020-11-03 14:08:06 [INFO] [TRAIN] epoch=141, iter=52200/80000, loss=0.0954, lr=0.003863, batch_cost=1.2622, reader_cost=0.0002 | ETA 09:44:49 2020-11-03 14:10:15 [INFO] [TRAIN] epoch=141, iter=52300/80000, loss=0.1042, lr=0.003850, batch_cost=1.2949, reader_cost=0.0002 | ETA 09:57:47 2020-11-03 14:12:19 [INFO] [TRAIN] epoch=141, iter=52400/80000, loss=0.1021, lr=0.003838, batch_cost=1.2382, reader_cost=0.0002 | ETA 09:29:35 2020-11-03 14:14:34 [INFO] [TRAIN] epoch=142, iter=52500/80000, loss=0.1077, lr=0.003825, batch_cost=1.3490, reader_cost=0.0096 | ETA 10:18:18 2020-11-03 14:16:47 [INFO] [TRAIN] epoch=142, iter=52600/80000, loss=0.0834, lr=0.003812, batch_cost=1.3326, reader_cost=0.0002 | ETA 10:08:32 2020-11-03 14:18:54 [INFO] [TRAIN] epoch=142, iter=52700/80000, loss=0.0911, lr=0.003800, batch_cost=1.2665, reader_cost=0.0001 | ETA 09:36:16 2020-11-03 14:21:01 [INFO] [TRAIN] epoch=142, iter=52800/80000, loss=0.1050, lr=0.003787, batch_cost=1.2703, reader_cost=0.0001 | ETA 09:35:53 2020-11-03 14:23:11 [INFO] [TRAIN] epoch=143, iter=52900/80000, loss=0.1006, lr=0.003775, batch_cost=1.2976, reader_cost=0.0099 | ETA 09:46:05 2020-11-03 14:25:17 [INFO] [TRAIN] epoch=143, iter=53000/80000, loss=0.0936, lr=0.003762, batch_cost=1.2635, reader_cost=0.0002 | ETA 09:28:34 2020-11-03 14:27:28 [INFO] [TRAIN] epoch=143, iter=53100/80000, loss=0.1053, lr=0.003750, batch_cost=1.3066, reader_cost=0.0002 | ETA 09:45:46 2020-11-03 14:29:44 [INFO] [TRAIN] epoch=144, iter=53200/80000, loss=0.1181, lr=0.003737, batch_cost=1.3635, reader_cost=0.0120 | ETA 10:09:02 2020-11-03 14:31:53 [INFO] [TRAIN] epoch=144, iter=53300/80000, loss=0.0879, lr=0.003725, batch_cost=1.2857, reader_cost=0.0002 | ETA 09:32:08 2020-11-03 14:34:00 [INFO] [TRAIN] epoch=144, iter=53400/80000, loss=0.0890, lr=0.003712, batch_cost=1.2765, reader_cost=0.0001 | ETA 09:25:55 2020-11-03 14:36:11 [INFO] [TRAIN] epoch=144, iter=53500/80000, loss=0.1001, lr=0.003700, batch_cost=1.3060, reader_cost=0.0003 | ETA 09:36:48 2020-11-03 14:38:16 [INFO] [TRAIN] epoch=145, iter=53600/80000, loss=0.1080, lr=0.003687, batch_cost=1.2513, reader_cost=0.0107 | ETA 09:10:33 2020-11-03 14:40:22 [INFO] [TRAIN] epoch=145, iter=53700/80000, loss=0.0876, lr=0.003674, batch_cost=1.2639, reader_cost=0.0002 | ETA 09:14:01 2020-11-03 14:42:36 [INFO] [TRAIN] epoch=145, iter=53800/80000, loss=0.0903, lr=0.003662, batch_cost=1.3323, reader_cost=0.0002 | ETA 09:41:46 2020-11-03 14:44:43 [INFO] [TRAIN] epoch=145, iter=53900/80000, loss=0.0914, lr=0.003649, batch_cost=1.2718, reader_cost=0.0012 | ETA 09:13:15 2020-11-03 14:46:50 [INFO] [TRAIN] epoch=146, iter=54000/80000, loss=0.0952, lr=0.003637, batch_cost=1.2677, reader_cost=0.0113 | ETA 09:09:20 2020-11-03 14:49:02 [INFO] [TRAIN] epoch=146, iter=54100/80000, loss=0.0856, lr=0.003624, batch_cost=1.3235, reader_cost=0.0002 | ETA 09:31:19 2020-11-03 14:51:08 [INFO] [TRAIN] epoch=146, iter=54200/80000, loss=0.0878, lr=0.003612, batch_cost=1.2651, reader_cost=0.0001 | ETA 09:03:59 2020-11-03 14:53:15 [INFO] [TRAIN] epoch=146, iter=54300/80000, loss=0.1020, lr=0.003599, batch_cost=1.2648, reader_cost=0.0005 | ETA 09:01:44 2020-11-03 14:55:25 [INFO] [TRAIN] epoch=147, iter=54400/80000, loss=0.1014, lr=0.003586, batch_cost=1.3027, reader_cost=0.0106 | ETA 09:15:48 2020-11-03 14:57:31 [INFO] [TRAIN] epoch=147, iter=54500/80000, loss=0.0798, lr=0.003574, batch_cost=1.2531, reader_cost=0.0002 | ETA 08:52:34 2020-11-03 14:59:36 [INFO] [TRAIN] epoch=147, iter=54600/80000, loss=0.0996, lr=0.003561, batch_cost=1.2520, reader_cost=0.0001 | ETA 08:49:59 2020-11-03 15:01:42 [INFO] [TRAIN] epoch=148, iter=54700/80000, loss=0.1067, lr=0.003548, batch_cost=1.2644, reader_cost=0.0097 | ETA 08:53:08 2020-11-03 15:03:49 [INFO] [TRAIN] epoch=148, iter=54800/80000, loss=0.0952, lr=0.003536, batch_cost=1.2714, reader_cost=0.0002 | ETA 08:53:59 2020-11-03 15:05:56 [INFO] [TRAIN] epoch=148, iter=54900/80000, loss=0.0838, lr=0.003523, batch_cost=1.2630, reader_cost=0.0001 | ETA 08:48:21 2020-11-03 15:08:03 [INFO] [TRAIN] epoch=148, iter=55000/80000, loss=0.0930, lr=0.003511, batch_cost=1.2725, reader_cost=0.0003 | ETA 08:50:12 2020-11-03 15:10:10 [INFO] [TRAIN] epoch=149, iter=55100/80000, loss=0.1013, lr=0.003498, batch_cost=1.2745, reader_cost=0.0126 | ETA 08:48:54 2020-11-03 15:12:18 [INFO] [TRAIN] epoch=149, iter=55200/80000, loss=0.0905, lr=0.003485, batch_cost=1.2762, reader_cost=0.0004 | ETA 08:47:28 2020-11-03 15:14:44 [INFO] [TRAIN] epoch=149, iter=55300/80000, loss=0.0892, lr=0.003473, batch_cost=1.4571, reader_cost=0.0002 | ETA 09:59:49 2020-11-03 15:16:52 [INFO] [TRAIN] epoch=149, iter=55400/80000, loss=0.0998, lr=0.003460, batch_cost=1.2825, reader_cost=0.0002 | ETA 08:45:48 2020-11-03 15:19:05 [INFO] [TRAIN] epoch=150, iter=55500/80000, loss=0.0987, lr=0.003447, batch_cost=1.3296, reader_cost=0.0115 | ETA 09:02:55 2020-11-03 15:21:12 [INFO] [TRAIN] epoch=150, iter=55600/80000, loss=0.0947, lr=0.003435, batch_cost=1.2682, reader_cost=0.0002 | ETA 08:35:45 2020-11-03 15:23:26 [INFO] [TRAIN] epoch=150, iter=55700/80000, loss=0.0987, lr=0.003422, batch_cost=1.3397, reader_cost=0.0008 | ETA 09:02:34 2020-11-03 15:25:31 [INFO] [TRAIN] epoch=150, iter=55800/80000, loss=0.1072, lr=0.003409, batch_cost=1.2585, reader_cost=0.0001 | ETA 08:27:36 2020-11-03 15:27:38 [INFO] [TRAIN] epoch=151, iter=55900/80000, loss=0.0900, lr=0.003397, batch_cost=1.2685, reader_cost=0.0111 | ETA 08:29:30 2020-11-03 15:29:45 [INFO] [TRAIN] epoch=151, iter=56000/80000, loss=0.0858, lr=0.003384, batch_cost=1.2650, reader_cost=0.0003 | ETA 08:25:59 2020-11-03 15:29:51 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-03 15:35:32 [INFO] [EVAL] #Images=500 mIoU=0.8054 Acc=0.9638 Kappa=0.9530 2020-11-03 15:35:32 [INFO] [EVAL] Category IoU: [0.9825 0.8608 0.9304 0.5615 0.6619 0.6666 0.7277 0.8136 0.9264 0.6653 0.9507 0.8375 0.6478 0.9575 0.8703 0.9018 0.8426 0.7015 0.7954] 2020-11-03 15:35:32 [INFO] [EVAL] Category Acc: [0.9908 0.9326 0.9618 0.8849 0.7858 0.8163 0.8677 0.9066 0.9568 0.7974 0.9681 0.9019 0.8175 0.9753 0.9521 0.9726 0.9426 0.8216 0.8828] 2020-11-03 15:35:36 [INFO] [EVAL] The model with the best validation mIoU (0.8054) was saved at iter 56000. 2020-11-03 15:37:46 [INFO] [TRAIN] epoch=151, iter=56100/80000, loss=0.0960, lr=0.003371, batch_cost=1.2995, reader_cost=0.0002 | ETA 08:37:38 2020-11-03 15:39:51 [INFO] [TRAIN] epoch=152, iter=56200/80000, loss=0.1074, lr=0.003359, batch_cost=1.2557, reader_cost=0.0111 | ETA 08:18:06 2020-11-03 15:42:03 [INFO] [TRAIN] epoch=152, iter=56300/80000, loss=0.0911, lr=0.003346, batch_cost=1.3156, reader_cost=0.0002 | ETA 08:39:40 2020-11-03 15:44:16 [INFO] [TRAIN] epoch=152, iter=56400/80000, loss=0.1038, lr=0.003333, batch_cost=1.3256, reader_cost=0.0004 | ETA 08:41:23 2020-11-03 15:46:24 [INFO] [TRAIN] epoch=152, iter=56500/80000, loss=0.0973, lr=0.003320, batch_cost=1.2892, reader_cost=0.0002 | ETA 08:24:55 2020-11-03 15:48:36 [INFO] [TRAIN] epoch=153, iter=56600/80000, loss=0.1124, lr=0.003308, batch_cost=1.3141, reader_cost=0.0104 | ETA 08:32:29 2020-11-03 15:50:45 [INFO] [TRAIN] epoch=153, iter=56700/80000, loss=0.0892, lr=0.003295, batch_cost=1.2955, reader_cost=0.0002 | ETA 08:23:06 2020-11-03 15:52:57 [INFO] [TRAIN] epoch=153, iter=56800/80000, loss=0.0919, lr=0.003282, batch_cost=1.3156, reader_cost=0.0003 | ETA 08:28:41 2020-11-03 15:55:03 [INFO] [TRAIN] epoch=153, iter=56900/80000, loss=0.1074, lr=0.003270, batch_cost=1.2652, reader_cost=0.0002 | ETA 08:07:06 2020-11-03 15:57:11 [INFO] [TRAIN] epoch=154, iter=57000/80000, loss=0.0924, lr=0.003257, batch_cost=1.2798, reader_cost=0.0116 | ETA 08:10:35 2020-11-03 15:59:21 [INFO] [TRAIN] epoch=154, iter=57100/80000, loss=0.0890, lr=0.003244, batch_cost=1.2976, reader_cost=0.0002 | ETA 08:15:14 2020-11-03 16:01:30 [INFO] [TRAIN] epoch=154, iter=57200/80000, loss=0.0877, lr=0.003231, batch_cost=1.2898, reader_cost=0.0001 | ETA 08:10:08 2020-11-03 16:03:35 [INFO] [TRAIN] epoch=155, iter=57300/80000, loss=0.0957, lr=0.003219, batch_cost=1.2500, reader_cost=0.0121 | ETA 07:52:55 2020-11-03 16:05:43 [INFO] [TRAIN] epoch=155, iter=57400/80000, loss=0.0860, lr=0.003206, batch_cost=1.2765, reader_cost=0.0003 | ETA 08:00:49 2020-11-03 16:07:51 [INFO] [TRAIN] epoch=155, iter=57500/80000, loss=0.0952, lr=0.003193, batch_cost=1.2805, reader_cost=0.0001 | ETA 08:00:11 2020-11-03 16:09:58 [INFO] [TRAIN] epoch=155, iter=57600/80000, loss=0.0941, lr=0.003180, batch_cost=1.2696, reader_cost=0.0002 | ETA 07:53:58 2020-11-03 16:12:06 [INFO] [TRAIN] epoch=156, iter=57700/80000, loss=0.0985, lr=0.003167, batch_cost=1.2776, reader_cost=0.0100 | ETA 07:54:51 2020-11-03 16:14:13 [INFO] [TRAIN] epoch=156, iter=57800/80000, loss=0.0886, lr=0.003155, batch_cost=1.2777, reader_cost=0.0002 | ETA 07:52:45 2020-11-03 16:16:20 [INFO] [TRAIN] epoch=156, iter=57900/80000, loss=0.0885, lr=0.003142, batch_cost=1.2629, reader_cost=0.0002 | ETA 07:45:09 2020-11-03 16:18:31 [INFO] [TRAIN] epoch=156, iter=58000/80000, loss=0.0861, lr=0.003129, batch_cost=1.3083, reader_cost=0.0006 | ETA 07:59:43 2020-11-03 16:20:49 [INFO] [TRAIN] epoch=157, iter=58100/80000, loss=0.0945, lr=0.003116, batch_cost=1.3886, reader_cost=0.0165 | ETA 08:26:49 2020-11-03 16:23:01 [INFO] [TRAIN] epoch=157, iter=58200/80000, loss=0.0832, lr=0.003103, batch_cost=1.3132, reader_cost=0.0003 | ETA 07:57:07 2020-11-03 16:25:18 [INFO] [TRAIN] epoch=157, iter=58300/80000, loss=0.0947, lr=0.003091, batch_cost=1.3704, reader_cost=0.0002 | ETA 08:15:37 2020-11-03 16:27:56 [INFO] [TRAIN] epoch=157, iter=58400/80000, loss=0.0953, lr=0.003078, batch_cost=1.5857, reader_cost=0.0007 | ETA 09:30:51 2020-11-03 16:30:39 [INFO] [TRAIN] epoch=158, iter=58500/80000, loss=0.0821, lr=0.003065, batch_cost=1.6271, reader_cost=0.0313 | ETA 09:43:02 2020-11-03 16:33:14 [INFO] [TRAIN] epoch=158, iter=58600/80000, loss=0.0788, lr=0.003052, batch_cost=1.5453, reader_cost=0.0006 | ETA 09:11:08 2020-11-03 16:35:44 [INFO] [TRAIN] epoch=158, iter=58700/80000, loss=0.1059, lr=0.003039, batch_cost=1.5031, reader_cost=0.0003 | ETA 08:53:35 2020-11-03 16:38:28 [INFO] [TRAIN] epoch=159, iter=58800/80000, loss=0.0918, lr=0.003026, batch_cost=1.6455, reader_cost=0.0443 | ETA 09:41:25 2020-11-03 16:41:06 [INFO] [TRAIN] epoch=159, iter=58900/80000, loss=0.0881, lr=0.003014, batch_cost=1.5767, reader_cost=0.0009 | ETA 09:14:27 2020-11-03 16:43:40 [INFO] [TRAIN] epoch=159, iter=59000/80000, loss=0.0852, lr=0.003001, batch_cost=1.5366, reader_cost=0.0003 | ETA 08:57:48 2020-11-03 16:46:18 [INFO] [TRAIN] epoch=159, iter=59100/80000, loss=0.0887, lr=0.002988, batch_cost=1.5809, reader_cost=0.0004 | ETA 09:10:40 2020-11-03 16:48:58 [INFO] [TRAIN] epoch=160, iter=59200/80000, loss=0.1025, lr=0.002975, batch_cost=1.6041, reader_cost=0.0333 | ETA 09:16:04 2020-11-03 16:51:36 [INFO] [TRAIN] epoch=160, iter=59300/80000, loss=0.0801, lr=0.002962, batch_cost=1.5818, reader_cost=0.0003 | ETA 09:05:42 2020-11-03 16:53:55 [INFO] [TRAIN] epoch=160, iter=59400/80000, loss=0.0849, lr=0.002949, batch_cost=1.3854, reader_cost=0.0003 | ETA 07:55:39 2020-11-03 16:56:10 [INFO] [TRAIN] epoch=160, iter=59500/80000, loss=0.0979, lr=0.002936, batch_cost=1.3521, reader_cost=0.0004 | ETA 07:41:57 2020-11-03 16:58:25 [INFO] [TRAIN] epoch=161, iter=59600/80000, loss=0.0910, lr=0.002924, batch_cost=1.3508, reader_cost=0.0157 | ETA 07:39:16 2020-11-03 17:00:36 [INFO] [TRAIN] epoch=161, iter=59700/80000, loss=0.0812, lr=0.002911, batch_cost=1.3051, reader_cost=0.0002 | ETA 07:21:33 2020-11-03 17:02:45 [INFO] [TRAIN] epoch=161, iter=59800/80000, loss=0.0924, lr=0.002898, batch_cost=1.2946, reader_cost=0.0002 | ETA 07:15:50 2020-11-03 17:04:53 [INFO] [TRAIN] epoch=162, iter=59900/80000, loss=0.0937, lr=0.002885, batch_cost=1.2807, reader_cost=0.0130 | ETA 07:09:03 2020-11-03 17:07:10 [INFO] [TRAIN] epoch=162, iter=60000/80000, loss=0.0844, lr=0.002872, batch_cost=1.3703, reader_cost=0.0002 | ETA 07:36:45 2020-11-03 17:09:19 [INFO] [TRAIN] epoch=162, iter=60100/80000, loss=0.0774, lr=0.002859, batch_cost=1.2909, reader_cost=0.0001 | ETA 07:08:08 2020-11-03 17:11:26 [INFO] [TRAIN] epoch=162, iter=60200/80000, loss=0.0928, lr=0.002846, batch_cost=1.2655, reader_cost=0.0001 | ETA 06:57:37 2020-11-03 17:13:37 [INFO] [TRAIN] epoch=163, iter=60300/80000, loss=0.0938, lr=0.002833, batch_cost=1.3124, reader_cost=0.0134 | ETA 07:10:53 2020-11-03 17:15:47 [INFO] [TRAIN] epoch=163, iter=60400/80000, loss=0.0829, lr=0.002820, batch_cost=1.2932, reader_cost=0.0002 | ETA 07:02:27 2020-11-03 17:17:54 [INFO] [TRAIN] epoch=163, iter=60500/80000, loss=0.0837, lr=0.002807, batch_cost=1.2692, reader_cost=0.0001 | ETA 06:52:29 2020-11-03 17:20:02 [INFO] [TRAIN] epoch=163, iter=60600/80000, loss=0.0898, lr=0.002794, batch_cost=1.2879, reader_cost=0.0002 | ETA 06:56:24 2020-11-03 17:22:10 [INFO] [TRAIN] epoch=164, iter=60700/80000, loss=0.1006, lr=0.002781, batch_cost=1.2743, reader_cost=0.0120 | ETA 06:49:53 2020-11-03 17:24:20 [INFO] [TRAIN] epoch=164, iter=60800/80000, loss=0.0938, lr=0.002768, batch_cost=1.3021, reader_cost=0.0013 | ETA 06:56:40 2020-11-03 17:26:29 [INFO] [TRAIN] epoch=164, iter=60900/80000, loss=0.0848, lr=0.002755, batch_cost=1.2857, reader_cost=0.0002 | ETA 06:49:16 2020-11-03 17:28:37 [INFO] [TRAIN] epoch=164, iter=61000/80000, loss=0.1013, lr=0.002742, batch_cost=1.2828, reader_cost=0.0003 | ETA 06:46:12 2020-11-03 17:30:47 [INFO] [TRAIN] epoch=165, iter=61100/80000, loss=0.0849, lr=0.002729, batch_cost=1.3050, reader_cost=0.0125 | ETA 06:51:04 2020-11-03 17:32:54 [INFO] [TRAIN] epoch=165, iter=61200/80000, loss=0.0721, lr=0.002716, batch_cost=1.2671, reader_cost=0.0002 | ETA 06:37:01 2020-11-03 17:35:01 [INFO] [TRAIN] epoch=165, iter=61300/80000, loss=0.1000, lr=0.002703, batch_cost=1.2676, reader_cost=0.0002 | ETA 06:35:04 2020-11-03 17:37:11 [INFO] [TRAIN] epoch=166, iter=61400/80000, loss=0.0926, lr=0.002690, batch_cost=1.3035, reader_cost=0.0107 | ETA 06:44:05 2020-11-03 17:39:18 [INFO] [TRAIN] epoch=166, iter=61500/80000, loss=0.0834, lr=0.002677, batch_cost=1.2642, reader_cost=0.0002 | ETA 06:29:47 2020-11-03 17:41:24 [INFO] [TRAIN] epoch=166, iter=61600/80000, loss=0.0837, lr=0.002664, batch_cost=1.2618, reader_cost=0.0002 | ETA 06:26:57 2020-11-03 17:43:31 [INFO] [TRAIN] epoch=166, iter=61700/80000, loss=0.0901, lr=0.002651, batch_cost=1.2699, reader_cost=0.0002 | ETA 06:27:18 2020-11-03 17:45:36 [INFO] [TRAIN] epoch=167, iter=61800/80000, loss=0.0916, lr=0.002638, batch_cost=1.2525, reader_cost=0.0105 | ETA 06:19:54 2020-11-03 17:47:41 [INFO] [TRAIN] epoch=167, iter=61900/80000, loss=0.0815, lr=0.002625, batch_cost=1.2510, reader_cost=0.0001 | ETA 06:17:22 2020-11-03 17:49:59 [INFO] [TRAIN] epoch=167, iter=62000/80000, loss=0.1021, lr=0.002612, batch_cost=1.3763, reader_cost=0.0002 | ETA 06:52:53 2020-11-03 17:52:08 [INFO] [TRAIN] epoch=167, iter=62100/80000, loss=0.0965, lr=0.002599, batch_cost=1.2951, reader_cost=0.0002 | ETA 06:26:21 2020-11-03 17:54:22 [INFO] [TRAIN] epoch=168, iter=62200/80000, loss=0.0944, lr=0.002586, batch_cost=1.3423, reader_cost=0.0169 | ETA 06:38:13 2020-11-03 17:56:32 [INFO] [TRAIN] epoch=168, iter=62300/80000, loss=0.0778, lr=0.002573, batch_cost=1.2928, reader_cost=0.0002 | ETA 06:21:22 2020-11-03 17:58:45 [INFO] [TRAIN] epoch=168, iter=62400/80000, loss=0.1046, lr=0.002560, batch_cost=1.3318, reader_cost=0.0003 | ETA 06:30:40 2020-11-03 18:00:56 [INFO] [TRAIN] epoch=169, iter=62500/80000, loss=0.0995, lr=0.002547, batch_cost=1.3073, reader_cost=0.0114 | ETA 06:21:17 2020-11-03 18:03:01 [INFO] [TRAIN] epoch=169, iter=62600/80000, loss=0.0911, lr=0.002534, batch_cost=1.2508, reader_cost=0.0003 | ETA 06:02:43 2020-11-03 18:05:11 [INFO] [TRAIN] epoch=169, iter=62700/80000, loss=0.0907, lr=0.002520, batch_cost=1.3007, reader_cost=0.0002 | ETA 06:15:01 2020-11-03 18:07:18 [INFO] [TRAIN] epoch=169, iter=62800/80000, loss=0.0814, lr=0.002507, batch_cost=1.2767, reader_cost=0.0002 | ETA 06:05:59 2020-11-03 18:09:25 [INFO] [TRAIN] epoch=170, iter=62900/80000, loss=0.0917, lr=0.002494, batch_cost=1.2665, reader_cost=0.0103 | ETA 06:00:57 2020-11-03 18:11:31 [INFO] [TRAIN] epoch=170, iter=63000/80000, loss=0.0844, lr=0.002481, batch_cost=1.2595, reader_cost=0.0002 | ETA 05:56:51 2020-11-03 18:13:37 [INFO] [TRAIN] epoch=170, iter=63100/80000, loss=0.0883, lr=0.002468, batch_cost=1.2579, reader_cost=0.0001 | ETA 05:54:18 2020-11-03 18:15:46 [INFO] [TRAIN] epoch=170, iter=63200/80000, loss=0.0969, lr=0.002455, batch_cost=1.2884, reader_cost=0.0001 | ETA 06:00:45 2020-11-03 18:18:02 [INFO] [TRAIN] epoch=171, iter=63300/80000, loss=0.0977, lr=0.002442, batch_cost=1.3652, reader_cost=0.0113 | ETA 06:19:58 2020-11-03 18:20:06 [INFO] [TRAIN] epoch=171, iter=63400/80000, loss=0.0801, lr=0.002429, batch_cost=1.2412, reader_cost=0.0002 | ETA 05:43:23 2020-11-03 18:22:13 [INFO] [TRAIN] epoch=171, iter=63500/80000, loss=0.0822, lr=0.002415, batch_cost=1.2660, reader_cost=0.0001 | ETA 05:48:09 2020-11-03 18:24:27 [INFO] [TRAIN] epoch=171, iter=63600/80000, loss=0.0903, lr=0.002402, batch_cost=1.3422, reader_cost=0.0002 | ETA 06:06:52 2020-11-03 18:26:38 [INFO] [TRAIN] epoch=172, iter=63700/80000, loss=0.0856, lr=0.002389, batch_cost=1.3111, reader_cost=0.0093 | ETA 05:56:11 2020-11-03 18:28:48 [INFO] [TRAIN] epoch=172, iter=63800/80000, loss=0.0720, lr=0.002376, batch_cost=1.3016, reader_cost=0.0002 | ETA 05:51:25 2020-11-03 18:30:57 [INFO] [TRAIN] epoch=172, iter=63900/80000, loss=0.0957, lr=0.002363, batch_cost=1.2896, reader_cost=0.0002 | ETA 05:46:02 2020-11-03 18:33:04 [INFO] [TRAIN] epoch=173, iter=64000/80000, loss=0.0968, lr=0.002349, batch_cost=1.2685, reader_cost=0.0101 | ETA 05:38:16 2020-11-03 18:33:10 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-03 18:38:45 [INFO] [EVAL] #Images=500 mIoU=0.7996 Acc=0.9631 Kappa=0.9521 2020-11-03 18:38:45 [INFO] [EVAL] Category IoU: [0.9818 0.8544 0.9286 0.4502 0.6577 0.6737 0.7328 0.817 0.9283 0.6588 0.9519 0.8393 0.6519 0.9577 0.8418 0.9095 0.8638 0.698 0.7947] 2020-11-03 18:38:45 [INFO] [EVAL] Category Acc: [0.9914 0.9192 0.9581 0.8556 0.8256 0.8261 0.8497 0.9114 0.9568 0.8194 0.9709 0.8985 0.8191 0.9759 0.9188 0.967 0.9429 0.8133 0.8735] 2020-11-03 18:38:45 [INFO] [EVAL] The model with the best validation mIoU (0.8054) was saved at iter 56000. 2020-11-03 18:40:47 [INFO] [TRAIN] epoch=173, iter=64100/80000, loss=0.0906, lr=0.002336, batch_cost=1.2270, reader_cost=0.0001 | ETA 05:25:09 2020-11-03 18:42:50 [INFO] [TRAIN] epoch=173, iter=64200/80000, loss=0.0868, lr=0.002323, batch_cost=1.2255, reader_cost=0.0001 | ETA 05:22:42 2020-11-03 18:44:53 [INFO] [TRAIN] epoch=173, iter=64300/80000, loss=0.0894, lr=0.002310, batch_cost=1.2313, reader_cost=0.0001 | ETA 05:22:11 2020-11-03 18:47:01 [INFO] [TRAIN] epoch=174, iter=64400/80000, loss=0.0882, lr=0.002296, batch_cost=1.2748, reader_cost=0.0134 | ETA 05:31:26 2020-11-03 18:49:04 [INFO] [TRAIN] epoch=174, iter=64500/80000, loss=0.0805, lr=0.002283, batch_cost=1.2315, reader_cost=0.0002 | ETA 05:18:08 2020-11-03 18:51:09 [INFO] [TRAIN] epoch=174, iter=64600/80000, loss=0.0809, lr=0.002270, batch_cost=1.2487, reader_cost=0.0005 | ETA 05:20:29 2020-11-03 18:53:12 [INFO] [TRAIN] epoch=174, iter=64700/80000, loss=0.0866, lr=0.002257, batch_cost=1.2357, reader_cost=0.0001 | ETA 05:15:05 2020-11-03 18:55:16 [INFO] [TRAIN] epoch=175, iter=64800/80000, loss=0.0890, lr=0.002243, batch_cost=1.2412, reader_cost=0.0098 | ETA 05:14:25 2020-11-03 18:57:21 [INFO] [TRAIN] epoch=175, iter=64900/80000, loss=0.0809, lr=0.002230, batch_cost=1.2514, reader_cost=0.0002 | ETA 05:14:56 2020-11-03 18:59:26 [INFO] [TRAIN] epoch=175, iter=65000/80000, loss=0.0882, lr=0.002217, batch_cost=1.2419, reader_cost=0.0001 | ETA 05:10:28 2020-11-03 19:01:29 [INFO] [TRAIN] epoch=175, iter=65100/80000, loss=0.1000, lr=0.002203, batch_cost=1.2330, reader_cost=0.0001 | ETA 05:06:12 2020-11-03 19:03:35 [INFO] [TRAIN] epoch=176, iter=65200/80000, loss=0.0855, lr=0.002190, batch_cost=1.2633, reader_cost=0.0103 | ETA 05:11:36 2020-11-03 19:05:44 [INFO] [TRAIN] epoch=176, iter=65300/80000, loss=0.0762, lr=0.002177, batch_cost=1.2893, reader_cost=0.0002 | ETA 05:15:52 2020-11-03 19:07:49 [INFO] [TRAIN] epoch=176, iter=65400/80000, loss=0.0822, lr=0.002164, batch_cost=1.2474, reader_cost=0.0001 | ETA 05:03:31 2020-11-03 19:09:55 [INFO] [TRAIN] epoch=177, iter=65500/80000, loss=0.0911, lr=0.002150, batch_cost=1.2597, reader_cost=0.0098 | ETA 05:04:25 2020-11-03 19:12:00 [INFO] [TRAIN] epoch=177, iter=65600/80000, loss=0.0874, lr=0.002137, batch_cost=1.2529, reader_cost=0.0002 | ETA 05:00:42 2020-11-03 19:14:04 [INFO] [TRAIN] epoch=177, iter=65700/80000, loss=0.0818, lr=0.002123, batch_cost=1.2385, reader_cost=0.0001 | ETA 04:55:10 2020-11-03 19:16:08 [INFO] [TRAIN] epoch=177, iter=65800/80000, loss=0.0862, lr=0.002110, batch_cost=1.2386, reader_cost=0.0001 | ETA 04:53:07 2020-11-03 19:18:14 [INFO] [TRAIN] epoch=178, iter=65900/80000, loss=0.0878, lr=0.002097, batch_cost=1.2577, reader_cost=0.0081 | ETA 04:55:33 2020-11-03 19:20:17 [INFO] [TRAIN] epoch=178, iter=66000/80000, loss=0.0916, lr=0.002083, batch_cost=1.2323, reader_cost=0.0002 | ETA 04:47:32 2020-11-03 19:22:22 [INFO] [TRAIN] epoch=178, iter=66100/80000, loss=0.0840, lr=0.002070, batch_cost=1.2545, reader_cost=0.0001 | ETA 04:50:37 2020-11-03 19:24:31 [INFO] [TRAIN] epoch=178, iter=66200/80000, loss=0.0893, lr=0.002057, batch_cost=1.2914, reader_cost=0.0001 | ETA 04:57:00 2020-11-03 19:26:37 [INFO] [TRAIN] epoch=179, iter=66300/80000, loss=0.0932, lr=0.002043, batch_cost=1.2538, reader_cost=0.0082 | ETA 04:46:17 2020-11-03 19:28:42 [INFO] [TRAIN] epoch=179, iter=66400/80000, loss=0.0794, lr=0.002030, batch_cost=1.2537, reader_cost=0.0002 | ETA 04:44:10 2020-11-03 19:30:45 [INFO] [TRAIN] epoch=179, iter=66500/80000, loss=0.0861, lr=0.002016, batch_cost=1.2274, reader_cost=0.0001 | ETA 04:36:09 2020-11-03 19:32:58 [INFO] [TRAIN] epoch=180, iter=66600/80000, loss=0.0975, lr=0.002003, batch_cost=1.3345, reader_cost=0.0095 | ETA 04:58:02 2020-11-03 19:35:07 [INFO] [TRAIN] epoch=180, iter=66700/80000, loss=0.0900, lr=0.001989, batch_cost=1.2846, reader_cost=0.0002 | ETA 04:44:45 2020-11-03 19:37:12 [INFO] [TRAIN] epoch=180, iter=66800/80000, loss=0.0952, lr=0.001976, batch_cost=1.2487, reader_cost=0.0017 | ETA 04:34:42 2020-11-03 19:39:16 [INFO] [TRAIN] epoch=180, iter=66900/80000, loss=0.1006, lr=0.001962, batch_cost=1.2471, reader_cost=0.0036 | ETA 04:32:17 2020-11-03 19:41:24 [INFO] [TRAIN] epoch=181, iter=67000/80000, loss=0.0896, lr=0.001949, batch_cost=1.2728, reader_cost=0.0138 | ETA 04:35:46 2020-11-03 19:43:33 [INFO] [TRAIN] epoch=181, iter=67100/80000, loss=0.0833, lr=0.001935, batch_cost=1.2900, reader_cost=0.0003 | ETA 04:37:20 2020-11-03 19:45:37 [INFO] [TRAIN] epoch=181, iter=67200/80000, loss=0.0826, lr=0.001922, batch_cost=1.2410, reader_cost=0.0001 | ETA 04:24:44 2020-11-03 19:47:43 [INFO] [TRAIN] epoch=181, iter=67300/80000, loss=0.0824, lr=0.001908, batch_cost=1.2607, reader_cost=0.0007 | ETA 04:26:50 2020-11-03 19:49:46 [INFO] [TRAIN] epoch=182, iter=67400/80000, loss=0.0927, lr=0.001895, batch_cost=1.2358, reader_cost=0.0098 | ETA 04:19:31 2020-11-03 19:51:50 [INFO] [TRAIN] epoch=182, iter=67500/80000, loss=0.0776, lr=0.001881, batch_cost=1.2321, reader_cost=0.0002 | ETA 04:16:41 2020-11-03 19:53:55 [INFO] [TRAIN] epoch=182, iter=67600/80000, loss=0.0904, lr=0.001868, batch_cost=1.2503, reader_cost=0.0001 | ETA 04:18:23 2020-11-03 19:56:02 [INFO] [TRAIN] epoch=182, iter=67700/80000, loss=0.1010, lr=0.001854, batch_cost=1.2699, reader_cost=0.0001 | ETA 04:20:19 2020-11-03 19:58:09 [INFO] [TRAIN] epoch=183, iter=67800/80000, loss=0.0828, lr=0.001841, batch_cost=1.2752, reader_cost=0.0141 | ETA 04:19:16 2020-11-03 20:00:13 [INFO] [TRAIN] epoch=183, iter=67900/80000, loss=0.0724, lr=0.001827, batch_cost=1.2401, reader_cost=0.0002 | ETA 04:10:04 2020-11-03 20:02:20 [INFO] [TRAIN] epoch=183, iter=68000/80000, loss=0.1217, lr=0.001813, batch_cost=1.2631, reader_cost=0.0001 | ETA 04:12:36 2020-11-03 20:04:27 [INFO] [TRAIN] epoch=184, iter=68100/80000, loss=0.0901, lr=0.001800, batch_cost=1.2723, reader_cost=0.0103 | ETA 04:12:19 2020-11-03 20:06:28 [INFO] [TRAIN] epoch=184, iter=68200/80000, loss=0.0764, lr=0.001786, batch_cost=1.2170, reader_cost=0.0002 | ETA 03:59:20 2020-11-03 20:08:38 [INFO] [TRAIN] epoch=184, iter=68300/80000, loss=0.0761, lr=0.001773, batch_cost=1.2933, reader_cost=0.0001 | ETA 04:12:11 2020-11-03 20:10:43 [INFO] [TRAIN] epoch=184, iter=68400/80000, loss=0.0850, lr=0.001759, batch_cost=1.2558, reader_cost=0.0001 | ETA 04:02:47 2020-11-03 20:12:51 [INFO] [TRAIN] epoch=185, iter=68500/80000, loss=0.0916, lr=0.001745, batch_cost=1.2726, reader_cost=0.0095 | ETA 04:03:55 2020-11-03 20:14:55 [INFO] [TRAIN] epoch=185, iter=68600/80000, loss=0.0795, lr=0.001732, batch_cost=1.2408, reader_cost=0.0002 | ETA 03:55:45 2020-11-03 20:17:00 [INFO] [TRAIN] epoch=185, iter=68700/80000, loss=0.0729, lr=0.001718, batch_cost=1.2565, reader_cost=0.0004 | ETA 03:56:38 2020-11-03 20:19:05 [INFO] [TRAIN] epoch=185, iter=68800/80000, loss=0.0971, lr=0.001704, batch_cost=1.2453, reader_cost=0.0001 | ETA 03:52:27 2020-11-03 20:21:15 [INFO] [TRAIN] epoch=186, iter=68900/80000, loss=0.0865, lr=0.001691, batch_cost=1.2977, reader_cost=0.0096 | ETA 04:00:04 2020-11-03 20:23:24 [INFO] [TRAIN] epoch=186, iter=69000/80000, loss=0.0844, lr=0.001677, batch_cost=1.2970, reader_cost=0.0002 | ETA 03:57:47 2020-11-03 20:25:32 [INFO] [TRAIN] epoch=186, iter=69100/80000, loss=0.0917, lr=0.001663, batch_cost=1.2731, reader_cost=0.0011 | ETA 03:51:17 2020-11-03 20:27:37 [INFO] [TRAIN] epoch=187, iter=69200/80000, loss=0.0872, lr=0.001649, batch_cost=1.2565, reader_cost=0.0105 | ETA 03:46:09 2020-11-03 20:29:44 [INFO] [TRAIN] epoch=187, iter=69300/80000, loss=0.0830, lr=0.001636, batch_cost=1.2630, reader_cost=0.0002 | ETA 03:45:13 2020-11-03 20:31:51 [INFO] [TRAIN] epoch=187, iter=69400/80000, loss=0.0804, lr=0.001622, batch_cost=1.2704, reader_cost=0.0001 | ETA 03:44:26 2020-11-03 20:33:56 [INFO] [TRAIN] epoch=187, iter=69500/80000, loss=0.0846, lr=0.001608, batch_cost=1.2545, reader_cost=0.0001 | ETA 03:39:32 2020-11-03 20:36:04 [INFO] [TRAIN] epoch=188, iter=69600/80000, loss=0.0860, lr=0.001594, batch_cost=1.2740, reader_cost=0.0101 | ETA 03:40:49 2020-11-03 20:38:13 [INFO] [TRAIN] epoch=188, iter=69700/80000, loss=0.0818, lr=0.001581, batch_cost=1.2893, reader_cost=0.0008 | ETA 03:41:19 2020-11-03 20:40:16 [INFO] [TRAIN] epoch=188, iter=69800/80000, loss=0.0760, lr=0.001567, batch_cost=1.2381, reader_cost=0.0003 | ETA 03:30:28 2020-11-03 20:42:22 [INFO] [TRAIN] epoch=188, iter=69900/80000, loss=0.0890, lr=0.001553, batch_cost=1.2581, reader_cost=0.0001 | ETA 03:31:46 2020-11-03 20:44:29 [INFO] [TRAIN] epoch=189, iter=70000/80000, loss=0.0954, lr=0.001539, batch_cost=1.2694, reader_cost=0.0115 | ETA 03:31:33 2020-11-03 20:46:33 [INFO] [TRAIN] epoch=189, iter=70100/80000, loss=0.0766, lr=0.001525, batch_cost=1.2378, reader_cost=0.0002 | ETA 03:24:14 2020-11-03 20:48:37 [INFO] [TRAIN] epoch=189, iter=70200/80000, loss=0.0859, lr=0.001511, batch_cost=1.2456, reader_cost=0.0002 | ETA 03:23:26 2020-11-03 20:50:42 [INFO] [TRAIN] epoch=189, iter=70300/80000, loss=0.0917, lr=0.001497, batch_cost=1.2423, reader_cost=0.0001 | ETA 03:20:50 2020-11-03 20:52:47 [INFO] [TRAIN] epoch=190, iter=70400/80000, loss=0.0793, lr=0.001484, batch_cost=1.2537, reader_cost=0.0094 | ETA 03:20:35 2020-11-03 20:54:52 [INFO] [TRAIN] epoch=190, iter=70500/80000, loss=0.0725, lr=0.001470, batch_cost=1.2487, reader_cost=0.0002 | ETA 03:17:43 2020-11-03 20:56:54 [INFO] [TRAIN] epoch=190, iter=70600/80000, loss=0.0796, lr=0.001456, batch_cost=1.2256, reader_cost=0.0001 | ETA 03:12:00 2020-11-03 20:58:59 [INFO] [TRAIN] epoch=191, iter=70700/80000, loss=0.0873, lr=0.001442, batch_cost=1.2434, reader_cost=0.0104 | ETA 03:12:43 2020-11-03 21:01:04 [INFO] [TRAIN] epoch=191, iter=70800/80000, loss=0.0812, lr=0.001428, batch_cost=1.2511, reader_cost=0.0002 | ETA 03:11:50 2020-11-03 21:03:07 [INFO] [TRAIN] epoch=191, iter=70900/80000, loss=0.0834, lr=0.001414, batch_cost=1.2301, reader_cost=0.0001 | ETA 03:06:34 2020-11-03 21:05:10 [INFO] [TRAIN] epoch=191, iter=71000/80000, loss=0.0855, lr=0.001400, batch_cost=1.2285, reader_cost=0.0001 | ETA 03:04:16 2020-11-03 21:07:14 [INFO] [TRAIN] epoch=192, iter=71100/80000, loss=0.0961, lr=0.001386, batch_cost=1.2414, reader_cost=0.0092 | ETA 03:04:08 2020-11-03 21:09:19 [INFO] [TRAIN] epoch=192, iter=71200/80000, loss=0.0759, lr=0.001372, batch_cost=1.2518, reader_cost=0.0002 | ETA 03:03:35 2020-11-03 21:11:37 [INFO] [TRAIN] epoch=192, iter=71300/80000, loss=0.0836, lr=0.001358, batch_cost=1.3775, reader_cost=0.0013 | ETA 03:19:44 2020-11-03 21:13:41 [INFO] [TRAIN] epoch=192, iter=71400/80000, loss=0.0923, lr=0.001344, batch_cost=1.2406, reader_cost=0.0001 | ETA 02:57:48 2020-11-03 21:15:48 [INFO] [TRAIN] epoch=193, iter=71500/80000, loss=0.0919, lr=0.001330, batch_cost=1.2667, reader_cost=0.0098 | ETA 02:59:26 2020-11-03 21:17:54 [INFO] [TRAIN] epoch=193, iter=71600/80000, loss=0.0811, lr=0.001316, batch_cost=1.2594, reader_cost=0.0002 | ETA 02:56:18 2020-11-03 21:20:01 [INFO] [TRAIN] epoch=193, iter=71700/80000, loss=0.0865, lr=0.001301, batch_cost=1.2746, reader_cost=0.0002 | ETA 02:56:19 2020-11-03 21:22:08 [INFO] [TRAIN] epoch=194, iter=71800/80000, loss=0.0916, lr=0.001287, batch_cost=1.2724, reader_cost=0.0102 | ETA 02:53:53 2020-11-03 21:24:13 [INFO] [TRAIN] epoch=194, iter=71900/80000, loss=0.0877, lr=0.001273, batch_cost=1.2521, reader_cost=0.0002 | ETA 02:49:02 2020-11-03 21:26:17 [INFO] [TRAIN] epoch=194, iter=72000/80000, loss=0.0816, lr=0.001259, batch_cost=1.2366, reader_cost=0.0001 | ETA 02:44:53 2020-11-03 21:26:23 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-03 21:31:59 [INFO] [EVAL] #Images=500 mIoU=0.8052 Acc=0.9644 Kappa=0.9538 2020-11-03 21:31:59 [INFO] [EVAL] Category IoU: [0.9834 0.8641 0.9323 0.5476 0.6547 0.6735 0.736 0.8158 0.9279 0.6468 0.9528 0.8402 0.6615 0.9583 0.8638 0.9093 0.8345 0.7003 0.796 ] 2020-11-03 21:31:59 [INFO] [EVAL] Category Acc: [0.9919 0.9274 0.9643 0.8491 0.8306 0.8012 0.8603 0.9164 0.9559 0.8114 0.9724 0.8944 0.8123 0.9763 0.9512 0.9709 0.9056 0.8175 0.8714] 2020-11-03 21:31:59 [INFO] [EVAL] The model with the best validation mIoU (0.8054) was saved at iter 56000. 2020-11-03 21:34:04 [INFO] [TRAIN] epoch=194, iter=72100/80000, loss=0.0838, lr=0.001245, batch_cost=1.2541, reader_cost=0.0001 | ETA 02:45:07 2020-11-03 21:36:11 [INFO] [TRAIN] epoch=195, iter=72200/80000, loss=0.0870, lr=0.001231, batch_cost=1.2705, reader_cost=0.0130 | ETA 02:45:10 2020-11-03 21:38:20 [INFO] [TRAIN] epoch=195, iter=72300/80000, loss=0.0778, lr=0.001216, batch_cost=1.2810, reader_cost=0.0002 | ETA 02:44:24 2020-11-03 21:40:25 [INFO] [TRAIN] epoch=195, iter=72400/80000, loss=0.0759, lr=0.001202, batch_cost=1.2564, reader_cost=0.0001 | ETA 02:39:08 2020-11-03 21:42:30 [INFO] [TRAIN] epoch=195, iter=72500/80000, loss=0.0880, lr=0.001188, batch_cost=1.2534, reader_cost=0.0002 | ETA 02:36:40 2020-11-03 21:44:39 [INFO] [TRAIN] epoch=196, iter=72600/80000, loss=0.0958, lr=0.001174, batch_cost=1.2852, reader_cost=0.0094 | ETA 02:38:30 2020-11-03 21:46:44 [INFO] [TRAIN] epoch=196, iter=72700/80000, loss=0.0860, lr=0.001159, batch_cost=1.2517, reader_cost=0.0002 | ETA 02:32:17 2020-11-03 21:48:50 [INFO] [TRAIN] epoch=196, iter=72800/80000, loss=0.0849, lr=0.001145, batch_cost=1.2624, reader_cost=0.0001 | ETA 02:31:29 2020-11-03 21:50:58 [INFO] [TRAIN] epoch=196, iter=72900/80000, loss=0.0868, lr=0.001131, batch_cost=1.2752, reader_cost=0.0001 | ETA 02:30:53 2020-11-03 21:53:03 [INFO] [TRAIN] epoch=197, iter=73000/80000, loss=0.0910, lr=0.001117, batch_cost=1.2538, reader_cost=0.0114 | ETA 02:26:16 2020-11-03 21:55:07 [INFO] [TRAIN] epoch=197, iter=73100/80000, loss=0.0785, lr=0.001102, batch_cost=1.2353, reader_cost=0.0002 | ETA 02:22:03 2020-11-03 21:57:10 [INFO] [TRAIN] epoch=197, iter=73200/80000, loss=0.0837, lr=0.001088, batch_cost=1.2326, reader_cost=0.0001 | ETA 02:19:41 2020-11-03 21:59:17 [INFO] [TRAIN] epoch=198, iter=73300/80000, loss=0.0809, lr=0.001073, batch_cost=1.2714, reader_cost=0.0090 | ETA 02:21:58 2020-11-03 22:01:26 [INFO] [TRAIN] epoch=198, iter=73400/80000, loss=0.0819, lr=0.001059, batch_cost=1.2881, reader_cost=0.0002 | ETA 02:21:41 2020-11-03 22:03:33 [INFO] [TRAIN] epoch=198, iter=73500/80000, loss=0.0757, lr=0.001044, batch_cost=1.2676, reader_cost=0.0002 | ETA 02:17:19 2020-11-03 22:05:36 [INFO] [TRAIN] epoch=198, iter=73600/80000, loss=0.0872, lr=0.001030, batch_cost=1.2325, reader_cost=0.0001 | ETA 02:11:27 2020-11-03 22:07:39 [INFO] [TRAIN] epoch=199, iter=73700/80000, loss=0.0844, lr=0.001016, batch_cost=1.2331, reader_cost=0.0085 | ETA 02:09:28 2020-11-03 22:09:42 [INFO] [TRAIN] epoch=199, iter=73800/80000, loss=0.0819, lr=0.001001, batch_cost=1.2281, reader_cost=0.0002 | ETA 02:06:54 2020-11-03 22:11:46 [INFO] [TRAIN] epoch=199, iter=73900/80000, loss=0.0771, lr=0.000986, batch_cost=1.2406, reader_cost=0.0001 | ETA 02:06:07 2020-11-03 22:13:51 [INFO] [TRAIN] epoch=199, iter=74000/80000, loss=0.0960, lr=0.000972, batch_cost=1.2503, reader_cost=0.0001 | ETA 02:05:01 2020-11-03 22:15:59 [INFO] [TRAIN] epoch=200, iter=74100/80000, loss=0.0854, lr=0.000957, batch_cost=1.2722, reader_cost=0.0081 | ETA 02:05:06 2020-11-03 22:18:06 [INFO] [TRAIN] epoch=200, iter=74200/80000, loss=0.0817, lr=0.000943, batch_cost=1.2748, reader_cost=0.0006 | ETA 02:03:13 2020-11-03 22:20:10 [INFO] [TRAIN] epoch=200, iter=74300/80000, loss=0.0858, lr=0.000928, batch_cost=1.2424, reader_cost=0.0001 | ETA 01:58:01 2020-11-03 22:22:16 [INFO] [TRAIN] epoch=200, iter=74400/80000, loss=0.0937, lr=0.000913, batch_cost=1.2564, reader_cost=0.0028 | ETA 01:57:15 2020-11-03 22:24:23 [INFO] [TRAIN] epoch=201, iter=74500/80000, loss=0.0800, lr=0.000899, batch_cost=1.2674, reader_cost=0.0098 | ETA 01:56:10 2020-11-03 22:26:28 [INFO] [TRAIN] epoch=201, iter=74600/80000, loss=0.0737, lr=0.000884, batch_cost=1.2513, reader_cost=0.0003 | ETA 01:52:36 2020-11-03 22:28:32 [INFO] [TRAIN] epoch=201, iter=74700/80000, loss=0.0878, lr=0.000869, batch_cost=1.2417, reader_cost=0.0001 | ETA 01:49:40 2020-11-03 22:30:37 [INFO] [TRAIN] epoch=202, iter=74800/80000, loss=0.0917, lr=0.000854, batch_cost=1.2524, reader_cost=0.0089 | ETA 01:48:32 2020-11-03 22:32:47 [INFO] [TRAIN] epoch=202, iter=74900/80000, loss=0.0805, lr=0.000840, batch_cost=1.2966, reader_cost=0.0002 | ETA 01:50:12 2020-11-03 22:34:52 [INFO] [TRAIN] epoch=202, iter=75000/80000, loss=0.0779, lr=0.000825, batch_cost=1.2513, reader_cost=0.0001 | ETA 01:44:16 2020-11-03 22:36:55 [INFO] [TRAIN] epoch=202, iter=75100/80000, loss=0.0873, lr=0.000810, batch_cost=1.2299, reader_cost=0.0001 | ETA 01:40:26 2020-11-03 22:38:59 [INFO] [TRAIN] epoch=203, iter=75200/80000, loss=0.0887, lr=0.000795, batch_cost=1.2401, reader_cost=0.0137 | ETA 01:39:12 2020-11-03 22:41:02 [INFO] [TRAIN] epoch=203, iter=75300/80000, loss=0.0753, lr=0.000780, batch_cost=1.2319, reader_cost=0.0002 | ETA 01:36:30 2020-11-03 22:43:05 [INFO] [TRAIN] epoch=203, iter=75400/80000, loss=0.0759, lr=0.000765, batch_cost=1.2276, reader_cost=0.0001 | ETA 01:34:07 2020-11-03 22:45:10 [INFO] [TRAIN] epoch=203, iter=75500/80000, loss=0.0940, lr=0.000750, batch_cost=1.2485, reader_cost=0.0001 | ETA 01:33:38 2020-11-03 22:47:16 [INFO] [TRAIN] epoch=204, iter=75600/80000, loss=0.0823, lr=0.000735, batch_cost=1.2643, reader_cost=0.0125 | ETA 01:32:43 2020-11-03 22:49:22 [INFO] [TRAIN] epoch=204, iter=75700/80000, loss=0.0786, lr=0.000720, batch_cost=1.2567, reader_cost=0.0002 | ETA 01:30:03 2020-11-03 22:51:26 [INFO] [TRAIN] epoch=204, iter=75800/80000, loss=0.0809, lr=0.000705, batch_cost=1.2384, reader_cost=0.0001 | ETA 01:26:41 2020-11-03 22:53:32 [INFO] [TRAIN] epoch=205, iter=75900/80000, loss=0.0868, lr=0.000690, batch_cost=1.2593, reader_cost=0.0106 | ETA 01:26:03 2020-11-03 22:55:36 [INFO] [TRAIN] epoch=205, iter=76000/80000, loss=0.0814, lr=0.000675, batch_cost=1.2427, reader_cost=0.0002 | ETA 01:22:50 2020-11-03 22:57:40 [INFO] [TRAIN] epoch=205, iter=76100/80000, loss=0.0831, lr=0.000660, batch_cost=1.2393, reader_cost=0.0001 | ETA 01:20:33 2020-11-03 22:59:46 [INFO] [TRAIN] epoch=205, iter=76200/80000, loss=0.0850, lr=0.000644, batch_cost=1.2592, reader_cost=0.0001 | ETA 01:19:44 2020-11-03 23:01:51 [INFO] [TRAIN] epoch=206, iter=76300/80000, loss=0.0982, lr=0.000629, batch_cost=1.2491, reader_cost=0.0118 | ETA 01:17:01 2020-11-03 23:03:55 [INFO] [TRAIN] epoch=206, iter=76400/80000, loss=0.0810, lr=0.000614, batch_cost=1.2420, reader_cost=0.0001 | ETA 01:14:31 2020-11-03 23:06:02 [INFO] [TRAIN] epoch=206, iter=76500/80000, loss=0.0782, lr=0.000598, batch_cost=1.2666, reader_cost=0.0015 | ETA 01:13:53 2020-11-03 23:08:08 [INFO] [TRAIN] epoch=206, iter=76600/80000, loss=0.0777, lr=0.000583, batch_cost=1.2664, reader_cost=0.0013 | ETA 01:11:45 2020-11-03 23:10:17 [INFO] [TRAIN] epoch=207, iter=76700/80000, loss=0.0892, lr=0.000568, batch_cost=1.2849, reader_cost=0.0097 | ETA 01:10:40 2020-11-03 23:12:19 [INFO] [TRAIN] epoch=207, iter=76800/80000, loss=0.0823, lr=0.000552, batch_cost=1.2256, reader_cost=0.0002 | ETA 01:05:21 2020-11-03 23:14:23 [INFO] [TRAIN] epoch=207, iter=76900/80000, loss=0.0808, lr=0.000537, batch_cost=1.2340, reader_cost=0.0001 | ETA 01:03:45 2020-11-03 23:16:27 [INFO] [TRAIN] epoch=207, iter=77000/80000, loss=0.0860, lr=0.000521, batch_cost=1.2461, reader_cost=0.0001 | ETA 01:02:18 2020-11-03 23:18:34 [INFO] [TRAIN] epoch=208, iter=77100/80000, loss=0.0872, lr=0.000505, batch_cost=1.2648, reader_cost=0.0094 | ETA 01:01:08 2020-11-03 23:20:40 [INFO] [TRAIN] epoch=208, iter=77200/80000, loss=0.0793, lr=0.000490, batch_cost=1.2605, reader_cost=0.0027 | ETA 00:58:49 2020-11-03 23:22:47 [INFO] [TRAIN] epoch=208, iter=77300/80000, loss=0.0913, lr=0.000474, batch_cost=1.2710, reader_cost=0.0001 | ETA 00:57:11 2020-11-03 23:24:53 [INFO] [TRAIN] epoch=209, iter=77400/80000, loss=0.0824, lr=0.000458, batch_cost=1.2607, reader_cost=0.0090 | ETA 00:54:37 2020-11-03 23:26:56 [INFO] [TRAIN] epoch=209, iter=77500/80000, loss=0.0762, lr=0.000442, batch_cost=1.2301, reader_cost=0.0002 | ETA 00:51:15 2020-11-03 23:28:59 [INFO] [TRAIN] epoch=209, iter=77600/80000, loss=0.0803, lr=0.000426, batch_cost=1.2340, reader_cost=0.0001 | ETA 00:49:21 2020-11-03 23:31:06 [INFO] [TRAIN] epoch=209, iter=77700/80000, loss=0.0799, lr=0.000410, batch_cost=1.2685, reader_cost=0.0001 | ETA 00:48:37 2020-11-03 23:33:13 [INFO] [TRAIN] epoch=210, iter=77800/80000, loss=0.0842, lr=0.000394, batch_cost=1.2677, reader_cost=0.0101 | ETA 00:46:28 2020-11-03 23:35:20 [INFO] [TRAIN] epoch=210, iter=77900/80000, loss=0.0780, lr=0.000378, batch_cost=1.2662, reader_cost=0.0004 | ETA 00:44:19 2020-11-03 23:37:25 [INFO] [TRAIN] epoch=210, iter=78000/80000, loss=0.0778, lr=0.000362, batch_cost=1.2492, reader_cost=0.0001 | ETA 00:41:38 2020-11-03 23:39:30 [INFO] [TRAIN] epoch=210, iter=78100/80000, loss=0.0798, lr=0.000345, batch_cost=1.2560, reader_cost=0.0002 | ETA 00:39:46 2020-11-03 23:41:36 [INFO] [TRAIN] epoch=211, iter=78200/80000, loss=0.0788, lr=0.000329, batch_cost=1.2566, reader_cost=0.0130 | ETA 00:37:41 2020-11-03 23:43:42 [INFO] [TRAIN] epoch=211, iter=78300/80000, loss=0.0793, lr=0.000313, batch_cost=1.2573, reader_cost=0.0002 | ETA 00:35:37 2020-11-03 23:45:47 [INFO] [TRAIN] epoch=211, iter=78400/80000, loss=0.0808, lr=0.000296, batch_cost=1.2494, reader_cost=0.0001 | ETA 00:33:19 2020-11-03 23:47:51 [INFO] [TRAIN] epoch=212, iter=78500/80000, loss=0.0799, lr=0.000279, batch_cost=1.2481, reader_cost=0.0097 | ETA 00:31:12 2020-11-03 23:49:55 [INFO] [TRAIN] epoch=212, iter=78600/80000, loss=0.0756, lr=0.000262, batch_cost=1.2349, reader_cost=0.0002 | ETA 00:28:48 2020-11-03 23:51:58 [INFO] [TRAIN] epoch=212, iter=78700/80000, loss=0.0832, lr=0.000246, batch_cost=1.2308, reader_cost=0.0001 | ETA 00:26:39 2020-11-03 23:54:02 [INFO] [TRAIN] epoch=212, iter=78800/80000, loss=0.0817, lr=0.000228, batch_cost=1.2387, reader_cost=0.0001 | ETA 00:24:46 2020-11-03 23:56:05 [INFO] [TRAIN] epoch=213, iter=78900/80000, loss=0.0861, lr=0.000211, batch_cost=1.2345, reader_cost=0.0098 | ETA 00:22:37 2020-11-03 23:58:07 [INFO] [TRAIN] epoch=213, iter=79000/80000, loss=0.0842, lr=0.000194, batch_cost=1.2205, reader_cost=0.0002 | ETA 00:20:20 2020-11-04 00:00:10 [INFO] [TRAIN] epoch=213, iter=79100/80000, loss=0.0779, lr=0.000176, batch_cost=1.2311, reader_cost=0.0001 | ETA 00:18:28 2020-11-04 00:02:15 [INFO] [TRAIN] epoch=213, iter=79200/80000, loss=0.0871, lr=0.000159, batch_cost=1.2472, reader_cost=0.0001 | ETA 00:16:37 2020-11-04 00:04:21 [INFO] [TRAIN] epoch=214, iter=79300/80000, loss=0.0889, lr=0.000141, batch_cost=1.2629, reader_cost=0.0098 | ETA 00:14:44 2020-11-04 00:06:27 [INFO] [TRAIN] epoch=214, iter=79400/80000, loss=0.0772, lr=0.000123, batch_cost=1.2511, reader_cost=0.0006 | ETA 00:12:30 2020-11-04 00:08:30 [INFO] [TRAIN] epoch=214, iter=79500/80000, loss=0.0822, lr=0.000104, batch_cost=1.2365, reader_cost=0.0001 | ETA 00:10:18 2020-11-04 00:10:33 [INFO] [TRAIN] epoch=214, iter=79600/80000, loss=0.0872, lr=0.000085, batch_cost=1.2234, reader_cost=0.0001 | ETA 00:08:09 2020-11-04 00:12:36 [INFO] [TRAIN] epoch=215, iter=79700/80000, loss=0.0856, lr=0.000066, batch_cost=1.2378, reader_cost=0.0120 | ETA 00:06:11 2020-11-04 00:14:39 [INFO] [TRAIN] epoch=215, iter=79800/80000, loss=0.0665, lr=0.000046, batch_cost=1.2284, reader_cost=0.0002 | ETA 00:04:05 2020-11-04 00:16:41 [INFO] [TRAIN] epoch=215, iter=79900/80000, loss=0.0892, lr=0.000025, batch_cost=1.2170, reader_cost=0.0001 | ETA 00:02:01 2020-11-04 00:18:45 [INFO] [TRAIN] epoch=216, iter=80000/80000, loss=0.0895, lr=0.000000, batch_cost=1.2365, reader_cost=0.0092 | ETA 00:00:00 2020-11-04 00:18:51 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-04 00:24:21 [INFO] [EVAL] #Images=500 mIoU=0.8085 Acc=0.9652 Kappa=0.9548 2020-11-04 00:24:21 [INFO] [EVAL] Category IoU: [0.9844 0.8704 0.9323 0.5469 0.6567 0.6761 0.7385 0.8209 0.929 0.6523 0.9534 0.8429 0.6657 0.9588 0.8558 0.9172 0.8612 0.7018 0.7975] 2020-11-04 00:24:21 [INFO] [EVAL] Category Acc: [0.9923 0.9302 0.962 0.8663 0.832 0.8215 0.8504 0.9057 0.957 0.8325 0.9723 0.9029 0.8147 0.9761 0.9421 0.9752 0.9439 0.8131 0.8727] 2020-11-04 00:24:24 [INFO] [EVAL] The model with the best validation mIoU (0.8085) was saved at iter 80000.