2020-11-03 21:14:14 [INFO] ------------Environment Information------------- platform: Linux-3.10.0_3-0-0-34-x86_64-with-centos-7.5.1804-Core Python: 3.7.9 (default, Aug 31 2020, 12:42:55) [GCC 7.3.0] Paddle compiled with cuda: True NVCC: Cuda compilation tools, release 10.2, V10.2.89 cudnn: 7.6 GPUs used: 4 CUDA_VISIBLE_DEVICES: 0,1,2,3 GPU: ['GPU 0: Tesla V100-SXM2-16GB', 'GPU 1: Tesla V100-SXM2-16GB', 'GPU 2: Tesla V100-SXM2-16GB', 'GPU 3: Tesla V100-SXM2-16GB', 'GPU 4: Tesla V100-SXM2-16GB', 'GPU 5: Tesla V100-SXM2-16GB', 'GPU 6: Tesla V100-SXM2-16GB', 'GPU 7: Tesla V100-SXM2-16GB'] GCC: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36) PaddlePaddle: 2.0.0-rc0 OpenCV: 4.1.0 ------------------------------------------------ 2020-11-03 21:14:14 [INFO] ---------------Config Information--------------- batch_size: 2 iters: 80000 learning_rate: decay: end_lr: 0.0 power: 0.9 type: poly value: 0.01 loss: coef: - 1 types: - ignore_index: 255 type: CrossEntropyLoss model: align_corners: false aspp_out_channels: 256 aspp_ratios: - 1 - 12 - 24 - 36 backbone: multi_grid: - 1 - 2 - 4 output_stride: 8 pretrained: https://bj.bcebos.com/paddleseg/dygraph/resnet50_vd_ssld_v2.tar.gz type: ResNet50_vd backbone_indices: - 0 - 3 num_classes: 19 pretrained: null type: DeepLabV3P optimizer: momentum: 0.9 type: sgd weight_decay: 4.0e-05 train_dataset: dataset_root: data/cityscapes mode: train transforms: - max_scale_factor: 2.0 min_scale_factor: 0.5 scale_step_size: 0.25 type: ResizeStepScaling - crop_size: - 1024 - 512 type: RandomPaddingCrop - type: RandomHorizontalFlip - brightness_range: 0.4 contrast_range: 0.4 saturation_range: 0.4 type: RandomDistort - type: Normalize type: Cityscapes val_dataset: dataset_root: data/cityscapes mode: val transforms: - type: Normalize type: Cityscapes ------------------------------------------------ 2020-11-03 21:14:18 [INFO] Loading pretrained model from https://bj.bcebos.com/paddleseg/dygraph/resnet50_vd_ssld_v2.tar.gz 2020-11-03 21:14:19 [INFO] There are 275/275 variables loaded into ResNet_vd. 2020-11-03 21:15:33 [INFO] [TRAIN] epoch=1, iter=100/80000, loss=1.1369, lr=0.009989, batch_cost=0.6772, reader_cost=0.0148 | ETA 15:01:50 2020-11-03 21:16:37 [INFO] [TRAIN] epoch=1, iter=200/80000, loss=0.7951, lr=0.009978, batch_cost=0.6374, reader_cost=0.0014 | ETA 14:07:41 2020-11-03 21:17:40 [INFO] [TRAIN] epoch=1, iter=300/80000, loss=0.5411, lr=0.009966, batch_cost=0.6348, reader_cost=0.0016 | ETA 14:03:12 2020-11-03 21:18:44 [INFO] [TRAIN] epoch=2, iter=400/80000, loss=0.5417, lr=0.009955, batch_cost=0.6407, reader_cost=0.0083 | ETA 14:10:01 2020-11-03 21:19:48 [INFO] [TRAIN] epoch=2, iter=500/80000, loss=0.5208, lr=0.009944, batch_cost=0.6354, reader_cost=0.0009 | ETA 14:01:55 2020-11-03 21:20:51 [INFO] [TRAIN] epoch=2, iter=600/80000, loss=0.3857, lr=0.009933, batch_cost=0.6341, reader_cost=0.0009 | ETA 13:59:05 2020-11-03 21:21:54 [INFO] [TRAIN] epoch=2, iter=700/80000, loss=0.3703, lr=0.009921, batch_cost=0.6328, reader_cost=0.0003 | ETA 13:56:23 2020-11-03 21:22:59 [INFO] [TRAIN] epoch=3, iter=800/80000, loss=0.3368, lr=0.009910, batch_cost=0.6438, reader_cost=0.0086 | ETA 14:09:45 2020-11-03 21:24:02 [INFO] [TRAIN] epoch=3, iter=900/80000, loss=0.4041, lr=0.009899, batch_cost=0.6351, reader_cost=0.0005 | ETA 13:57:18 2020-11-03 21:25:06 [INFO] [TRAIN] epoch=3, iter=1000/80000, loss=0.2905, lr=0.009888, batch_cost=0.6363, reader_cost=0.0009 | ETA 13:57:51 2020-11-03 21:26:09 [INFO] [TRAIN] epoch=3, iter=1100/80000, loss=0.3679, lr=0.009876, batch_cost=0.6336, reader_cost=0.0010 | ETA 13:53:13 2020-11-03 21:27:14 [INFO] [TRAIN] epoch=4, iter=1200/80000, loss=0.2869, lr=0.009865, batch_cost=0.6479, reader_cost=0.0093 | ETA 14:10:53 2020-11-03 21:28:17 [INFO] [TRAIN] epoch=4, iter=1300/80000, loss=0.3051, lr=0.009854, batch_cost=0.6326, reader_cost=0.0005 | ETA 13:49:44 2020-11-03 21:29:20 [INFO] [TRAIN] epoch=4, iter=1400/80000, loss=0.2954, lr=0.009842, batch_cost=0.6263, reader_cost=0.0005 | ETA 13:40:28 2020-11-03 21:30:23 [INFO] [TRAIN] epoch=5, iter=1500/80000, loss=0.2973, lr=0.009831, batch_cost=0.6289, reader_cost=0.0086 | ETA 13:42:48 2020-11-03 21:31:25 [INFO] [TRAIN] epoch=5, iter=1600/80000, loss=0.2669, lr=0.009820, batch_cost=0.6197, reader_cost=0.0002 | ETA 13:29:44 2020-11-03 21:32:28 [INFO] [TRAIN] epoch=5, iter=1700/80000, loss=0.2880, lr=0.009809, batch_cost=0.6326, reader_cost=0.0013 | ETA 13:45:29 2020-11-03 21:33:32 [INFO] [TRAIN] epoch=5, iter=1800/80000, loss=0.3382, lr=0.009797, batch_cost=0.6369, reader_cost=0.0013 | ETA 13:50:04 2020-11-03 21:34:36 [INFO] [TRAIN] epoch=6, iter=1900/80000, loss=0.3054, lr=0.009786, batch_cost=0.6441, reader_cost=0.0088 | ETA 13:58:25 2020-11-03 21:35:40 [INFO] [TRAIN] epoch=6, iter=2000/80000, loss=0.2554, lr=0.009775, batch_cost=0.6366, reader_cost=0.0010 | ETA 13:47:34 2020-11-03 21:36:43 [INFO] [TRAIN] epoch=6, iter=2100/80000, loss=0.2963, lr=0.009764, batch_cost=0.6340, reader_cost=0.0015 | ETA 13:43:09 2020-11-03 21:37:47 [INFO] [TRAIN] epoch=6, iter=2200/80000, loss=0.2262, lr=0.009752, batch_cost=0.6322, reader_cost=0.0006 | ETA 13:39:44 2020-11-03 21:38:51 [INFO] [TRAIN] epoch=7, iter=2300/80000, loss=0.2423, lr=0.009741, batch_cost=0.6430, reader_cost=0.0080 | ETA 13:52:38 2020-11-03 21:39:54 [INFO] [TRAIN] epoch=7, iter=2400/80000, loss=0.2485, lr=0.009730, batch_cost=0.6334, reader_cost=0.0005 | ETA 13:39:12 2020-11-03 21:40:57 [INFO] [TRAIN] epoch=7, iter=2500/80000, loss=0.2629, lr=0.009718, batch_cost=0.6326, reader_cost=0.0007 | ETA 13:37:08 2020-11-03 21:42:01 [INFO] [TRAIN] epoch=7, iter=2600/80000, loss=0.2304, lr=0.009707, batch_cost=0.6345, reader_cost=0.0006 | ETA 13:38:26 2020-11-03 21:43:05 [INFO] [TRAIN] epoch=8, iter=2700/80000, loss=0.2852, lr=0.009696, batch_cost=0.6433, reader_cost=0.0086 | ETA 13:48:44 2020-11-03 21:44:09 [INFO] [TRAIN] epoch=8, iter=2800/80000, loss=0.3089, lr=0.009685, batch_cost=0.6347, reader_cost=0.0005 | ETA 13:36:38 2020-11-03 21:45:12 [INFO] [TRAIN] epoch=8, iter=2900/80000, loss=0.2513, lr=0.009673, batch_cost=0.6350, reader_cost=0.0007 | ETA 13:36:00 2020-11-03 21:46:17 [INFO] [TRAIN] epoch=9, iter=3000/80000, loss=0.2150, lr=0.009662, batch_cost=0.6453, reader_cost=0.0097 | ETA 13:48:11 2020-11-03 21:47:20 [INFO] [TRAIN] epoch=9, iter=3100/80000, loss=0.2357, lr=0.009651, batch_cost=0.6355, reader_cost=0.0015 | ETA 13:34:29 2020-11-03 21:48:24 [INFO] [TRAIN] epoch=9, iter=3200/80000, loss=0.2323, lr=0.009639, batch_cost=0.6371, reader_cost=0.0011 | ETA 13:35:27 2020-11-03 21:49:28 [INFO] [TRAIN] epoch=9, iter=3300/80000, loss=0.2372, lr=0.009628, batch_cost=0.6364, reader_cost=0.0012 | ETA 13:33:34 2020-11-03 21:50:32 [INFO] [TRAIN] epoch=10, iter=3400/80000, loss=0.2250, lr=0.009617, batch_cost=0.6459, reader_cost=0.0095 | ETA 13:44:33 2020-11-03 21:51:36 [INFO] [TRAIN] epoch=10, iter=3500/80000, loss=0.2156, lr=0.009605, batch_cost=0.6373, reader_cost=0.0011 | ETA 13:32:34 2020-11-03 21:52:40 [INFO] [TRAIN] epoch=10, iter=3600/80000, loss=0.2445, lr=0.009594, batch_cost=0.6362, reader_cost=0.0007 | ETA 13:30:08 2020-11-03 21:53:43 [INFO] [TRAIN] epoch=10, iter=3700/80000, loss=0.2282, lr=0.009583, batch_cost=0.6345, reader_cost=0.0013 | ETA 13:26:53 2020-11-03 21:54:48 [INFO] [TRAIN] epoch=11, iter=3800/80000, loss=0.2183, lr=0.009572, batch_cost=0.6466, reader_cost=0.0098 | ETA 13:41:12 2020-11-03 21:55:51 [INFO] [TRAIN] epoch=11, iter=3900/80000, loss=0.2345, lr=0.009560, batch_cost=0.6349, reader_cost=0.0003 | ETA 13:25:15 2020-11-03 21:56:55 [INFO] [TRAIN] epoch=11, iter=4000/80000, loss=0.2356, lr=0.009549, batch_cost=0.6357, reader_cost=0.0008 | ETA 13:25:10 2020-11-03 21:57:59 [INFO] [TRAIN] epoch=12, iter=4100/80000, loss=0.2202, lr=0.009538, batch_cost=0.6445, reader_cost=0.0097 | ETA 13:35:17 2020-11-03 21:59:03 [INFO] [TRAIN] epoch=12, iter=4200/80000, loss=0.2394, lr=0.009526, batch_cost=0.6352, reader_cost=0.0005 | ETA 13:22:26 2020-11-03 22:00:06 [INFO] [TRAIN] epoch=12, iter=4300/80000, loss=0.2523, lr=0.009515, batch_cost=0.6374, reader_cost=0.0006 | ETA 13:24:11 2020-11-03 22:01:10 [INFO] [TRAIN] epoch=12, iter=4400/80000, loss=0.1899, lr=0.009504, batch_cost=0.6349, reader_cost=0.0007 | ETA 13:19:57 2020-11-03 22:02:15 [INFO] [TRAIN] epoch=13, iter=4500/80000, loss=0.1982, lr=0.009492, batch_cost=0.6462, reader_cost=0.0089 | ETA 13:33:08 2020-11-03 22:03:18 [INFO] [TRAIN] epoch=13, iter=4600/80000, loss=0.2245, lr=0.009481, batch_cost=0.6381, reader_cost=0.0007 | ETA 13:21:49 2020-11-03 22:04:22 [INFO] [TRAIN] epoch=13, iter=4700/80000, loss=0.2323, lr=0.009470, batch_cost=0.6370, reader_cost=0.0005 | ETA 13:19:28 2020-11-03 22:05:26 [INFO] [TRAIN] epoch=13, iter=4800/80000, loss=0.2102, lr=0.009458, batch_cost=0.6375, reader_cost=0.0010 | ETA 13:18:57 2020-11-03 22:06:30 [INFO] [TRAIN] epoch=14, iter=4900/80000, loss=0.1950, lr=0.009447, batch_cost=0.6443, reader_cost=0.0090 | ETA 13:26:24 2020-11-03 22:07:34 [INFO] [TRAIN] epoch=14, iter=5000/80000, loss=0.2414, lr=0.009436, batch_cost=0.6357, reader_cost=0.0010 | ETA 13:14:38 2020-11-03 22:08:37 [INFO] [TRAIN] epoch=14, iter=5100/80000, loss=0.2074, lr=0.009424, batch_cost=0.6343, reader_cost=0.0007 | ETA 13:11:50 2020-11-03 22:09:41 [INFO] [TRAIN] epoch=14, iter=5200/80000, loss=0.1983, lr=0.009413, batch_cost=0.6357, reader_cost=0.0005 | ETA 13:12:29 2020-11-03 22:10:45 [INFO] [TRAIN] epoch=15, iter=5300/80000, loss=0.2091, lr=0.009402, batch_cost=0.6446, reader_cost=0.0101 | ETA 13:22:34 2020-11-03 22:11:49 [INFO] [TRAIN] epoch=15, iter=5400/80000, loss=0.2148, lr=0.009391, batch_cost=0.6345, reader_cost=0.0006 | ETA 13:08:54 2020-11-03 22:12:52 [INFO] [TRAIN] epoch=15, iter=5500/80000, loss=0.2085, lr=0.009379, batch_cost=0.6323, reader_cost=0.0004 | ETA 13:05:05 2020-11-03 22:13:55 [INFO] [TRAIN] epoch=16, iter=5600/80000, loss=0.1780, lr=0.009368, batch_cost=0.6267, reader_cost=0.0085 | ETA 12:57:07 2020-11-03 22:14:56 [INFO] [TRAIN] epoch=16, iter=5700/80000, loss=0.1823, lr=0.009357, batch_cost=0.6163, reader_cost=0.0003 | ETA 12:43:13 2020-11-03 22:15:58 [INFO] [TRAIN] epoch=16, iter=5800/80000, loss=0.2101, lr=0.009345, batch_cost=0.6210, reader_cost=0.0003 | ETA 12:48:00 2020-11-03 22:17:02 [INFO] [TRAIN] epoch=16, iter=5900/80000, loss=0.1959, lr=0.009334, batch_cost=0.6317, reader_cost=0.0012 | ETA 13:00:11 2020-11-03 22:18:06 [INFO] [TRAIN] epoch=17, iter=6000/80000, loss=0.1705, lr=0.009323, batch_cost=0.6429, reader_cost=0.0088 | ETA 13:12:53 2020-11-03 22:19:09 [INFO] [TRAIN] epoch=17, iter=6100/80000, loss=0.2110, lr=0.009311, batch_cost=0.6339, reader_cost=0.0008 | ETA 13:00:45 2020-11-03 22:20:13 [INFO] [TRAIN] epoch=17, iter=6200/80000, loss=0.2106, lr=0.009300, batch_cost=0.6345, reader_cost=0.0007 | ETA 13:00:28 2020-11-03 22:21:16 [INFO] [TRAIN] epoch=17, iter=6300/80000, loss=0.2254, lr=0.009288, batch_cost=0.6346, reader_cost=0.0014 | ETA 12:59:30 2020-11-03 22:22:21 [INFO] [TRAIN] epoch=18, iter=6400/80000, loss=0.1948, lr=0.009277, batch_cost=0.6433, reader_cost=0.0098 | ETA 13:09:04 2020-11-03 22:23:24 [INFO] [TRAIN] epoch=18, iter=6500/80000, loss=0.2143, lr=0.009266, batch_cost=0.6329, reader_cost=0.0010 | ETA 12:55:14 2020-11-03 22:24:27 [INFO] [TRAIN] epoch=18, iter=6600/80000, loss=0.1635, lr=0.009254, batch_cost=0.6369, reader_cost=0.0015 | ETA 12:59:08 2020-11-03 22:25:32 [INFO] [TRAIN] epoch=19, iter=6700/80000, loss=0.1767, lr=0.009243, batch_cost=0.6443, reader_cost=0.0105 | ETA 13:07:09 2020-11-03 22:26:35 [INFO] [TRAIN] epoch=19, iter=6800/80000, loss=0.1990, lr=0.009232, batch_cost=0.6328, reader_cost=0.0015 | ETA 12:51:58 2020-11-03 22:27:39 [INFO] [TRAIN] epoch=19, iter=6900/80000, loss=0.2287, lr=0.009220, batch_cost=0.6339, reader_cost=0.0013 | ETA 12:52:14 2020-11-03 22:28:42 [INFO] [TRAIN] epoch=19, iter=7000/80000, loss=0.1860, lr=0.009209, batch_cost=0.6354, reader_cost=0.0013 | ETA 12:53:07 2020-11-03 22:29:47 [INFO] [TRAIN] epoch=20, iter=7100/80000, loss=0.1667, lr=0.009198, batch_cost=0.6489, reader_cost=0.0085 | ETA 13:08:22 2020-11-03 22:30:51 [INFO] [TRAIN] epoch=20, iter=7200/80000, loss=0.1799, lr=0.009186, batch_cost=0.6355, reader_cost=0.0015 | ETA 12:51:05 2020-11-03 22:31:54 [INFO] [TRAIN] epoch=20, iter=7300/80000, loss=0.1726, lr=0.009175, batch_cost=0.6339, reader_cost=0.0007 | ETA 12:48:06 2020-11-03 22:32:57 [INFO] [TRAIN] epoch=20, iter=7400/80000, loss=0.1942, lr=0.009164, batch_cost=0.6339, reader_cost=0.0009 | ETA 12:47:03 2020-11-03 22:34:02 [INFO] [TRAIN] epoch=21, iter=7500/80000, loss=0.1735, lr=0.009152, batch_cost=0.6437, reader_cost=0.0093 | ETA 12:57:46 2020-11-03 22:35:05 [INFO] [TRAIN] epoch=21, iter=7600/80000, loss=0.2098, lr=0.009141, batch_cost=0.6336, reader_cost=0.0014 | ETA 12:44:36 2020-11-03 22:36:09 [INFO] [TRAIN] epoch=21, iter=7700/80000, loss=0.1740, lr=0.009130, batch_cost=0.6366, reader_cost=0.0016 | ETA 12:47:07 2020-11-03 22:37:12 [INFO] [TRAIN] epoch=21, iter=7800/80000, loss=0.1641, lr=0.009118, batch_cost=0.6360, reader_cost=0.0016 | ETA 12:45:17 2020-11-03 22:38:17 [INFO] [TRAIN] epoch=22, iter=7900/80000, loss=0.1748, lr=0.009107, batch_cost=0.6439, reader_cost=0.0103 | ETA 12:53:46 2020-11-03 22:39:20 [INFO] [TRAIN] epoch=22, iter=8000/80000, loss=0.1780, lr=0.009095, batch_cost=0.6337, reader_cost=0.0014 | ETA 12:40:27 2020-11-03 22:39:23 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-03 22:43:46 [INFO] [EVAL] #Images=500 mIoU=0.7053 Acc=0.9501 Kappa=0.9351 2020-11-03 22:43:46 [INFO] [EVAL] Category IoU: [0.9733 0.7984 0.9087 0.4685 0.5512 0.5981 0.6376 0.7374 0.9135 0.5655 0.929 0.7913 0.5657 0.9364 0.5197 0.6472 0.5362 0.5885 0.7353] 2020-11-03 22:43:46 [INFO] [EVAL] Category Acc: [0.9812 0.9307 0.9496 0.7701 0.7864 0.7563 0.7445 0.8685 0.9451 0.8196 0.9499 0.8917 0.6706 0.9678 0.8523 0.6852 0.6921 0.7164 0.8286] 2020-11-03 22:43:47 [INFO] [EVAL] The model with the best validation mIoU (0.7053) was saved at iter 8000. 2020-11-03 22:44:51 [INFO] [TRAIN] epoch=22, iter=8100/80000, loss=0.1699, lr=0.009084, batch_cost=0.6329, reader_cost=0.0013 | ETA 12:38:22 2020-11-03 22:45:55 [INFO] [TRAIN] epoch=23, iter=8200/80000, loss=0.1846, lr=0.009073, batch_cost=0.6449, reader_cost=0.0094 | ETA 12:51:41 2020-11-03 22:46:58 [INFO] [TRAIN] epoch=23, iter=8300/80000, loss=0.1813, lr=0.009061, batch_cost=0.6332, reader_cost=0.0007 | ETA 12:36:44 2020-11-03 22:48:02 [INFO] [TRAIN] epoch=23, iter=8400/80000, loss=0.1593, lr=0.009050, batch_cost=0.6330, reader_cost=0.0006 | ETA 12:35:22 2020-11-03 22:49:05 [INFO] [TRAIN] epoch=23, iter=8500/80000, loss=0.2242, lr=0.009039, batch_cost=0.6309, reader_cost=0.0008 | ETA 12:31:49 2020-11-03 22:50:09 [INFO] [TRAIN] epoch=24, iter=8600/80000, loss=0.1934, lr=0.009027, batch_cost=0.6423, reader_cost=0.0096 | ETA 12:44:22 2020-11-03 22:51:12 [INFO] [TRAIN] epoch=24, iter=8700/80000, loss=0.2046, lr=0.009016, batch_cost=0.6326, reader_cost=0.0007 | ETA 12:31:45 2020-11-03 22:52:16 [INFO] [TRAIN] epoch=24, iter=8800/80000, loss=0.2091, lr=0.009004, batch_cost=0.6386, reader_cost=0.0016 | ETA 12:37:45 2020-11-03 22:53:20 [INFO] [TRAIN] epoch=24, iter=8900/80000, loss=0.1637, lr=0.008993, batch_cost=0.6353, reader_cost=0.0011 | ETA 12:32:49 2020-11-03 22:54:24 [INFO] [TRAIN] epoch=25, iter=9000/80000, loss=0.1644, lr=0.008982, batch_cost=0.6426, reader_cost=0.0092 | ETA 12:40:25 2020-11-03 22:55:28 [INFO] [TRAIN] epoch=25, iter=9100/80000, loss=0.1841, lr=0.008970, batch_cost=0.6364, reader_cost=0.0014 | ETA 12:31:57 2020-11-03 22:56:31 [INFO] [TRAIN] epoch=25, iter=9200/80000, loss=0.1873, lr=0.008959, batch_cost=0.6335, reader_cost=0.0006 | ETA 12:27:30 2020-11-03 22:57:33 [INFO] [TRAIN] epoch=25, iter=9300/80000, loss=0.1673, lr=0.008948, batch_cost=0.6200, reader_cost=0.0005 | ETA 12:10:35 2020-11-03 22:58:35 [INFO] [TRAIN] epoch=26, iter=9400/80000, loss=0.1727, lr=0.008936, batch_cost=0.6254, reader_cost=0.0088 | ETA 12:15:51 2020-11-03 22:59:37 [INFO] [TRAIN] epoch=26, iter=9500/80000, loss=0.1840, lr=0.008925, batch_cost=0.6182, reader_cost=0.0002 | ETA 12:06:23 2020-11-03 23:00:41 [INFO] [TRAIN] epoch=26, iter=9600/80000, loss=0.1838, lr=0.008913, batch_cost=0.6346, reader_cost=0.0006 | ETA 12:24:34 2020-11-03 23:01:45 [INFO] [TRAIN] epoch=27, iter=9700/80000, loss=0.1367, lr=0.008902, batch_cost=0.6421, reader_cost=0.0087 | ETA 12:32:17 2020-11-03 23:02:48 [INFO] [TRAIN] epoch=27, iter=9800/80000, loss=0.1650, lr=0.008891, batch_cost=0.6330, reader_cost=0.0006 | ETA 12:20:36 2020-11-03 23:03:51 [INFO] [TRAIN] epoch=27, iter=9900/80000, loss=0.1849, lr=0.008879, batch_cost=0.6310, reader_cost=0.0006 | ETA 12:17:12 2020-11-03 23:04:54 [INFO] [TRAIN] epoch=27, iter=10000/80000, loss=0.1528, lr=0.008868, batch_cost=0.6297, reader_cost=0.0009 | ETA 12:14:37 2020-11-03 23:05:59 [INFO] [TRAIN] epoch=28, iter=10100/80000, loss=0.1876, lr=0.008856, batch_cost=0.6427, reader_cost=0.0099 | ETA 12:28:47 2020-11-03 23:07:02 [INFO] [TRAIN] epoch=28, iter=10200/80000, loss=0.1723, lr=0.008845, batch_cost=0.6347, reader_cost=0.0013 | ETA 12:18:18 2020-11-03 23:08:06 [INFO] [TRAIN] epoch=28, iter=10300/80000, loss=0.1910, lr=0.008834, batch_cost=0.6353, reader_cost=0.0011 | ETA 12:18:03 2020-11-03 23:09:09 [INFO] [TRAIN] epoch=28, iter=10400/80000, loss=0.1621, lr=0.008822, batch_cost=0.6329, reader_cost=0.0012 | ETA 12:14:07 2020-11-03 23:10:13 [INFO] [TRAIN] epoch=29, iter=10500/80000, loss=0.1420, lr=0.008811, batch_cost=0.6417, reader_cost=0.0082 | ETA 12:23:17 2020-11-03 23:11:16 [INFO] [TRAIN] epoch=29, iter=10600/80000, loss=0.1753, lr=0.008799, batch_cost=0.6336, reader_cost=0.0005 | ETA 12:12:54 2020-11-03 23:12:20 [INFO] [TRAIN] epoch=29, iter=10700/80000, loss=0.1728, lr=0.008788, batch_cost=0.6319, reader_cost=0.0010 | ETA 12:09:50 2020-11-03 23:13:24 [INFO] [TRAIN] epoch=30, iter=10800/80000, loss=0.1657, lr=0.008776, batch_cost=0.6446, reader_cost=0.0095 | ETA 12:23:25 2020-11-03 23:14:28 [INFO] [TRAIN] epoch=30, iter=10900/80000, loss=0.1569, lr=0.008765, batch_cost=0.6369, reader_cost=0.0014 | ETA 12:13:30 2020-11-03 23:15:31 [INFO] [TRAIN] epoch=30, iter=11000/80000, loss=0.1757, lr=0.008754, batch_cost=0.6328, reader_cost=0.0012 | ETA 12:07:40 2020-11-03 23:16:34 [INFO] [TRAIN] epoch=30, iter=11100/80000, loss=0.1898, lr=0.008742, batch_cost=0.6309, reader_cost=0.0004 | ETA 12:04:26 2020-11-03 23:17:39 [INFO] [TRAIN] epoch=31, iter=11200/80000, loss=0.1808, lr=0.008731, batch_cost=0.6439, reader_cost=0.0090 | ETA 12:18:21 2020-11-03 23:18:42 [INFO] [TRAIN] epoch=31, iter=11300/80000, loss=0.1996, lr=0.008719, batch_cost=0.6335, reader_cost=0.0007 | ETA 12:05:24 2020-11-03 23:19:45 [INFO] [TRAIN] epoch=31, iter=11400/80000, loss=0.1915, lr=0.008708, batch_cost=0.6329, reader_cost=0.0008 | ETA 12:03:34 2020-11-03 23:20:48 [INFO] [TRAIN] epoch=31, iter=11500/80000, loss=0.1860, lr=0.008697, batch_cost=0.6289, reader_cost=0.0013 | ETA 11:57:59 2020-11-03 23:21:52 [INFO] [TRAIN] epoch=32, iter=11600/80000, loss=0.1962, lr=0.008685, batch_cost=0.6408, reader_cost=0.0096 | ETA 12:10:29 2020-11-03 23:22:56 [INFO] [TRAIN] epoch=32, iter=11700/80000, loss=0.1862, lr=0.008674, batch_cost=0.6350, reader_cost=0.0015 | ETA 12:02:50 2020-11-03 23:23:59 [INFO] [TRAIN] epoch=32, iter=11800/80000, loss=0.1951, lr=0.008662, batch_cost=0.6362, reader_cost=0.0015 | ETA 12:03:07 2020-11-03 23:25:03 [INFO] [TRAIN] epoch=32, iter=11900/80000, loss=0.1654, lr=0.008651, batch_cost=0.6360, reader_cost=0.0008 | ETA 12:01:48 2020-11-03 23:26:07 [INFO] [TRAIN] epoch=33, iter=12000/80000, loss=0.1837, lr=0.008639, batch_cost=0.6402, reader_cost=0.0093 | ETA 12:05:30 2020-11-03 23:27:10 [INFO] [TRAIN] epoch=33, iter=12100/80000, loss=0.2092, lr=0.008628, batch_cost=0.6311, reader_cost=0.0008 | ETA 11:54:12 2020-11-03 23:28:13 [INFO] [TRAIN] epoch=33, iter=12200/80000, loss=0.1863, lr=0.008617, batch_cost=0.6320, reader_cost=0.0010 | ETA 11:54:09 2020-11-03 23:29:17 [INFO] [TRAIN] epoch=34, iter=12300/80000, loss=0.1632, lr=0.008605, batch_cost=0.6417, reader_cost=0.0086 | ETA 12:04:01 2020-11-03 23:30:20 [INFO] [TRAIN] epoch=34, iter=12400/80000, loss=0.1622, lr=0.008594, batch_cost=0.6305, reader_cost=0.0004 | ETA 11:50:24 2020-11-03 23:31:24 [INFO] [TRAIN] epoch=34, iter=12500/80000, loss=0.1688, lr=0.008582, batch_cost=0.6316, reader_cost=0.0003 | ETA 11:50:33 2020-11-03 23:32:27 [INFO] [TRAIN] epoch=34, iter=12600/80000, loss=0.1397, lr=0.008571, batch_cost=0.6321, reader_cost=0.0005 | ETA 11:50:06 2020-11-03 23:33:31 [INFO] [TRAIN] epoch=35, iter=12700/80000, loss=0.1436, lr=0.008559, batch_cost=0.6422, reader_cost=0.0087 | ETA 12:00:22 2020-11-03 23:34:34 [INFO] [TRAIN] epoch=35, iter=12800/80000, loss=0.1652, lr=0.008548, batch_cost=0.6325, reader_cost=0.0010 | ETA 11:48:23 2020-11-03 23:35:37 [INFO] [TRAIN] epoch=35, iter=12900/80000, loss=0.1619, lr=0.008536, batch_cost=0.6311, reader_cost=0.0007 | ETA 11:45:48 2020-11-03 23:36:41 [INFO] [TRAIN] epoch=35, iter=13000/80000, loss=0.1405, lr=0.008525, batch_cost=0.6328, reader_cost=0.0013 | ETA 11:46:40 2020-11-03 23:37:45 [INFO] [TRAIN] epoch=36, iter=13100/80000, loss=0.1410, lr=0.008514, batch_cost=0.6435, reader_cost=0.0094 | ETA 11:57:28 2020-11-03 23:38:48 [INFO] [TRAIN] epoch=36, iter=13200/80000, loss=0.1623, lr=0.008502, batch_cost=0.6298, reader_cost=0.0004 | ETA 11:41:10 2020-11-03 23:39:52 [INFO] [TRAIN] epoch=36, iter=13300/80000, loss=0.1835, lr=0.008491, batch_cost=0.6357, reader_cost=0.0002 | ETA 11:46:42 2020-11-03 23:40:55 [INFO] [TRAIN] epoch=37, iter=13400/80000, loss=0.1721, lr=0.008479, batch_cost=0.6373, reader_cost=0.0087 | ETA 11:47:23 2020-11-03 23:41:57 [INFO] [TRAIN] epoch=37, iter=13500/80000, loss=0.1669, lr=0.008468, batch_cost=0.6174, reader_cost=0.0002 | ETA 11:24:18 2020-11-03 23:42:59 [INFO] [TRAIN] epoch=37, iter=13600/80000, loss=0.1790, lr=0.008456, batch_cost=0.6195, reader_cost=0.0004 | ETA 11:25:35 2020-11-03 23:44:02 [INFO] [TRAIN] epoch=37, iter=13700/80000, loss=0.1763, lr=0.008445, batch_cost=0.6260, reader_cost=0.0006 | ETA 11:31:46 2020-11-03 23:45:06 [INFO] [TRAIN] epoch=38, iter=13800/80000, loss=0.1397, lr=0.008433, batch_cost=0.6399, reader_cost=0.0088 | ETA 11:46:03 2020-11-03 23:46:09 [INFO] [TRAIN] epoch=38, iter=13900/80000, loss=0.1625, lr=0.008422, batch_cost=0.6320, reader_cost=0.0008 | ETA 11:36:18 2020-11-03 23:47:12 [INFO] [TRAIN] epoch=38, iter=14000/80000, loss=0.1574, lr=0.008410, batch_cost=0.6342, reader_cost=0.0008 | ETA 11:37:34 2020-11-03 23:48:16 [INFO] [TRAIN] epoch=38, iter=14100/80000, loss=0.1420, lr=0.008399, batch_cost=0.6333, reader_cost=0.0011 | ETA 11:35:33 2020-11-03 23:49:20 [INFO] [TRAIN] epoch=39, iter=14200/80000, loss=0.1412, lr=0.008387, batch_cost=0.6407, reader_cost=0.0088 | ETA 11:42:40 2020-11-03 23:50:23 [INFO] [TRAIN] epoch=39, iter=14300/80000, loss=0.1691, lr=0.008376, batch_cost=0.6337, reader_cost=0.0009 | ETA 11:33:56 2020-11-03 23:51:26 [INFO] [TRAIN] epoch=39, iter=14400/80000, loss=0.1436, lr=0.008364, batch_cost=0.6336, reader_cost=0.0011 | ETA 11:32:42 2020-11-03 23:52:29 [INFO] [TRAIN] epoch=39, iter=14500/80000, loss=0.1482, lr=0.008353, batch_cost=0.6315, reader_cost=0.0010 | ETA 11:29:20 2020-11-03 23:53:34 [INFO] [TRAIN] epoch=40, iter=14600/80000, loss=0.1626, lr=0.008342, batch_cost=0.6403, reader_cost=0.0098 | ETA 11:37:54 2020-11-03 23:54:37 [INFO] [TRAIN] epoch=40, iter=14700/80000, loss=0.1641, lr=0.008330, batch_cost=0.6350, reader_cost=0.0010 | ETA 11:31:02 2020-11-03 23:55:40 [INFO] [TRAIN] epoch=40, iter=14800/80000, loss=0.2053, lr=0.008319, batch_cost=0.6324, reader_cost=0.0004 | ETA 11:27:12 2020-11-03 23:56:45 [INFO] [TRAIN] epoch=41, iter=14900/80000, loss=0.1606, lr=0.008307, batch_cost=0.6460, reader_cost=0.0094 | ETA 11:40:57 2020-11-03 23:57:48 [INFO] [TRAIN] epoch=41, iter=15000/80000, loss=0.1917, lr=0.008296, batch_cost=0.6354, reader_cost=0.0007 | ETA 11:28:22 2020-11-03 23:58:52 [INFO] [TRAIN] epoch=41, iter=15100/80000, loss=0.1399, lr=0.008284, batch_cost=0.6351, reader_cost=0.0012 | ETA 11:26:56 2020-11-03 23:59:55 [INFO] [TRAIN] epoch=41, iter=15200/80000, loss=0.1586, lr=0.008273, batch_cost=0.6317, reader_cost=0.0011 | ETA 11:22:12 2020-11-04 00:01:00 [INFO] [TRAIN] epoch=42, iter=15300/80000, loss=0.1397, lr=0.008261, batch_cost=0.6442, reader_cost=0.0109 | ETA 11:34:40 2020-11-04 00:02:03 [INFO] [TRAIN] epoch=42, iter=15400/80000, loss=0.1576, lr=0.008250, batch_cost=0.6362, reader_cost=0.0010 | ETA 11:24:57 2020-11-04 00:03:06 [INFO] [TRAIN] epoch=42, iter=15500/80000, loss=0.1734, lr=0.008238, batch_cost=0.6306, reader_cost=0.0004 | ETA 11:17:56 2020-11-04 00:04:09 [INFO] [TRAIN] epoch=42, iter=15600/80000, loss=0.1485, lr=0.008227, batch_cost=0.6296, reader_cost=0.0004 | ETA 11:15:43 2020-11-04 00:05:13 [INFO] [TRAIN] epoch=43, iter=15700/80000, loss=0.1635, lr=0.008215, batch_cost=0.6426, reader_cost=0.0086 | ETA 11:28:40 2020-11-04 00:06:17 [INFO] [TRAIN] epoch=43, iter=15800/80000, loss=0.1721, lr=0.008204, batch_cost=0.6340, reader_cost=0.0003 | ETA 11:18:22 2020-11-04 00:07:20 [INFO] [TRAIN] epoch=43, iter=15900/80000, loss=0.1661, lr=0.008192, batch_cost=0.6319, reader_cost=0.0007 | ETA 11:15:07 2020-11-04 00:08:24 [INFO] [TRAIN] epoch=44, iter=16000/80000, loss=0.1510, lr=0.008181, batch_cost=0.6430, reader_cost=0.0087 | ETA 11:25:50 2020-11-04 00:08:27 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-04 00:12:45 [INFO] [EVAL] #Images=500 mIoU=0.7474 Acc=0.9535 Kappa=0.9396 2020-11-04 00:12:45 [INFO] [EVAL] Category IoU: [0.974 0.8094 0.9094 0.5367 0.5612 0.6041 0.6626 0.7604 0.9167 0.6168 0.9424 0.8102 0.613 0.9427 0.7513 0.7881 0.6082 0.6355 0.7586] 2020-11-04 00:12:45 [INFO] [EVAL] Category Acc: [0.9894 0.8919 0.9454 0.7235 0.8146 0.8375 0.7994 0.9012 0.9455 0.8102 0.97 0.8904 0.7495 0.9658 0.8535 0.9712 0.7099 0.7758 0.8529] 2020-11-04 00:12:47 [INFO] [EVAL] The model with the best validation mIoU (0.7474) was saved at iter 16000. 2020-11-04 00:13:50 [INFO] [TRAIN] epoch=44, iter=16100/80000, loss=0.1555, lr=0.008169, batch_cost=0.6320, reader_cost=0.0012 | ETA 11:13:05 2020-11-04 00:14:53 [INFO] [TRAIN] epoch=44, iter=16200/80000, loss=0.1441, lr=0.008158, batch_cost=0.6327, reader_cost=0.0010 | ETA 11:12:46 2020-11-04 00:15:57 [INFO] [TRAIN] epoch=44, iter=16300/80000, loss=0.1495, lr=0.008146, batch_cost=0.6349, reader_cost=0.0007 | ETA 11:14:03 2020-11-04 00:17:01 [INFO] [TRAIN] epoch=45, iter=16400/80000, loss=0.1396, lr=0.008135, batch_cost=0.6442, reader_cost=0.0097 | ETA 11:22:50 2020-11-04 00:18:04 [INFO] [TRAIN] epoch=45, iter=16500/80000, loss=0.1647, lr=0.008123, batch_cost=0.6322, reader_cost=0.0005 | ETA 11:09:03 2020-11-04 00:19:07 [INFO] [TRAIN] epoch=45, iter=16600/80000, loss=0.1565, lr=0.008112, batch_cost=0.6298, reader_cost=0.0004 | ETA 11:05:32 2020-11-04 00:20:10 [INFO] [TRAIN] epoch=45, iter=16700/80000, loss=0.1619, lr=0.008100, batch_cost=0.6298, reader_cost=0.0011 | ETA 11:04:23 2020-11-04 00:21:15 [INFO] [TRAIN] epoch=46, iter=16800/80000, loss=0.1719, lr=0.008089, batch_cost=0.6415, reader_cost=0.0087 | ETA 11:15:40 2020-11-04 00:22:18 [INFO] [TRAIN] epoch=46, iter=16900/80000, loss=0.1706, lr=0.008077, batch_cost=0.6296, reader_cost=0.0013 | ETA 11:02:10 2020-11-04 00:23:21 [INFO] [TRAIN] epoch=46, iter=17000/80000, loss=0.1784, lr=0.008066, batch_cost=0.6335, reader_cost=0.0011 | ETA 11:05:13 2020-11-04 00:24:24 [INFO] [TRAIN] epoch=46, iter=17100/80000, loss=0.1401, lr=0.008054, batch_cost=0.6278, reader_cost=0.0006 | ETA 10:58:05 2020-11-04 00:25:26 [INFO] [TRAIN] epoch=47, iter=17200/80000, loss=0.1655, lr=0.008042, batch_cost=0.6258, reader_cost=0.0079 | ETA 10:54:57 2020-11-04 00:26:28 [INFO] [TRAIN] epoch=47, iter=17300/80000, loss=0.1437, lr=0.008031, batch_cost=0.6161, reader_cost=0.0002 | ETA 10:43:49 2020-11-04 00:27:31 [INFO] [TRAIN] epoch=47, iter=17400/80000, loss=0.1755, lr=0.008019, batch_cost=0.6274, reader_cost=0.0007 | ETA 10:54:32 2020-11-04 00:28:35 [INFO] [TRAIN] epoch=48, iter=17500/80000, loss=0.1650, lr=0.008008, batch_cost=0.6457, reader_cost=0.0094 | ETA 11:12:33 2020-11-04 00:29:38 [INFO] [TRAIN] epoch=48, iter=17600/80000, loss=0.1907, lr=0.007996, batch_cost=0.6332, reader_cost=0.0006 | ETA 10:58:31 2020-11-04 00:30:42 [INFO] [TRAIN] epoch=48, iter=17700/80000, loss=0.1645, lr=0.007985, batch_cost=0.6323, reader_cost=0.0003 | ETA 10:56:33 2020-11-04 00:31:45 [INFO] [TRAIN] epoch=48, iter=17800/80000, loss=0.1523, lr=0.007973, batch_cost=0.6313, reader_cost=0.0006 | ETA 10:54:28 2020-11-04 00:32:49 [INFO] [TRAIN] epoch=49, iter=17900/80000, loss=0.1341, lr=0.007962, batch_cost=0.6440, reader_cost=0.0093 | ETA 11:06:32 2020-11-04 00:33:53 [INFO] [TRAIN] epoch=49, iter=18000/80000, loss=0.1525, lr=0.007950, batch_cost=0.6355, reader_cost=0.0014 | ETA 10:56:42 2020-11-04 00:34:56 [INFO] [TRAIN] epoch=49, iter=18100/80000, loss=0.1645, lr=0.007939, batch_cost=0.6340, reader_cost=0.0009 | ETA 10:54:02 2020-11-04 00:36:00 [INFO] [TRAIN] epoch=49, iter=18200/80000, loss=0.1601, lr=0.007927, batch_cost=0.6345, reader_cost=0.0012 | ETA 10:53:33 2020-11-04 00:37:04 [INFO] [TRAIN] epoch=50, iter=18300/80000, loss=0.1505, lr=0.007916, batch_cost=0.6406, reader_cost=0.0101 | ETA 10:58:42 2020-11-04 00:38:07 [INFO] [TRAIN] epoch=50, iter=18400/80000, loss=0.1519, lr=0.007904, batch_cost=0.6309, reader_cost=0.0009 | ETA 10:47:42 2020-11-04 00:39:10 [INFO] [TRAIN] epoch=50, iter=18500/80000, loss=0.1649, lr=0.007892, batch_cost=0.6345, reader_cost=0.0010 | ETA 10:50:19 2020-11-04 00:40:14 [INFO] [TRAIN] epoch=50, iter=18600/80000, loss=0.1484, lr=0.007881, batch_cost=0.6334, reader_cost=0.0011 | ETA 10:48:09 2020-11-04 00:41:18 [INFO] [TRAIN] epoch=51, iter=18700/80000, loss=0.1405, lr=0.007869, batch_cost=0.6430, reader_cost=0.0083 | ETA 10:56:56 2020-11-04 00:42:21 [INFO] [TRAIN] epoch=51, iter=18800/80000, loss=0.1700, lr=0.007858, batch_cost=0.6340, reader_cost=0.0004 | ETA 10:46:39 2020-11-04 00:43:25 [INFO] [TRAIN] epoch=51, iter=18900/80000, loss=0.1629, lr=0.007846, batch_cost=0.6327, reader_cost=0.0004 | ETA 10:44:19 2020-11-04 00:44:29 [INFO] [TRAIN] epoch=52, iter=19000/80000, loss=0.1398, lr=0.007835, batch_cost=0.6401, reader_cost=0.0087 | ETA 10:50:46 2020-11-04 00:45:32 [INFO] [TRAIN] epoch=52, iter=19100/80000, loss=0.1384, lr=0.007823, batch_cost=0.6327, reader_cost=0.0009 | ETA 10:42:10 2020-11-04 00:46:35 [INFO] [TRAIN] epoch=52, iter=19200/80000, loss=0.1587, lr=0.007812, batch_cost=0.6338, reader_cost=0.0006 | ETA 10:42:12 2020-11-04 00:47:39 [INFO] [TRAIN] epoch=52, iter=19300/80000, loss=0.1343, lr=0.007800, batch_cost=0.6342, reader_cost=0.0003 | ETA 10:41:38 2020-11-04 00:48:43 [INFO] [TRAIN] epoch=53, iter=19400/80000, loss=0.1265, lr=0.007788, batch_cost=0.6404, reader_cost=0.0090 | ETA 10:46:45 2020-11-04 00:49:46 [INFO] [TRAIN] epoch=53, iter=19500/80000, loss=0.1654, lr=0.007777, batch_cost=0.6296, reader_cost=0.0011 | ETA 10:34:49 2020-11-04 00:50:49 [INFO] [TRAIN] epoch=53, iter=19600/80000, loss=0.1494, lr=0.007765, batch_cost=0.6321, reader_cost=0.0008 | ETA 10:36:18 2020-11-04 00:51:52 [INFO] [TRAIN] epoch=53, iter=19700/80000, loss=0.1314, lr=0.007754, batch_cost=0.6289, reader_cost=0.0012 | ETA 10:32:04 2020-11-04 00:52:56 [INFO] [TRAIN] epoch=54, iter=19800/80000, loss=0.1335, lr=0.007742, batch_cost=0.6395, reader_cost=0.0094 | ETA 10:41:36 2020-11-04 00:53:59 [INFO] [TRAIN] epoch=54, iter=19900/80000, loss=0.1429, lr=0.007731, batch_cost=0.6351, reader_cost=0.0017 | ETA 10:36:09 2020-11-04 00:55:03 [INFO] [TRAIN] epoch=54, iter=20000/80000, loss=0.1501, lr=0.007719, batch_cost=0.6338, reader_cost=0.0013 | ETA 10:33:50 2020-11-04 00:56:07 [INFO] [TRAIN] epoch=55, iter=20100/80000, loss=0.1396, lr=0.007707, batch_cost=0.6435, reader_cost=0.0095 | ETA 10:42:25 2020-11-04 00:57:10 [INFO] [TRAIN] epoch=55, iter=20200/80000, loss=0.1549, lr=0.007696, batch_cost=0.6338, reader_cost=0.0011 | ETA 10:31:39 2020-11-04 00:58:14 [INFO] [TRAIN] epoch=55, iter=20300/80000, loss=0.1494, lr=0.007684, batch_cost=0.6338, reader_cost=0.0014 | ETA 10:30:40 2020-11-04 00:59:17 [INFO] [TRAIN] epoch=55, iter=20400/80000, loss=0.1330, lr=0.007673, batch_cost=0.6314, reader_cost=0.0011 | ETA 10:27:12 2020-11-04 01:00:21 [INFO] [TRAIN] epoch=56, iter=20500/80000, loss=0.1289, lr=0.007661, batch_cost=0.6382, reader_cost=0.0095 | ETA 10:32:51 2020-11-04 01:01:24 [INFO] [TRAIN] epoch=56, iter=20600/80000, loss=0.1387, lr=0.007650, batch_cost=0.6315, reader_cost=0.0004 | ETA 10:25:08 2020-11-04 01:02:27 [INFO] [TRAIN] epoch=56, iter=20700/80000, loss=0.1483, lr=0.007638, batch_cost=0.6350, reader_cost=0.0004 | ETA 10:27:36 2020-11-04 01:03:31 [INFO] [TRAIN] epoch=56, iter=20800/80000, loss=0.1330, lr=0.007626, batch_cost=0.6332, reader_cost=0.0008 | ETA 10:24:44 2020-11-04 01:04:35 [INFO] [TRAIN] epoch=57, iter=20900/80000, loss=0.1223, lr=0.007615, batch_cost=0.6444, reader_cost=0.0096 | ETA 10:34:41 2020-11-04 01:05:39 [INFO] [TRAIN] epoch=57, iter=21000/80000, loss=0.1470, lr=0.007603, batch_cost=0.6348, reader_cost=0.0007 | ETA 10:24:15 2020-11-04 01:06:42 [INFO] [TRAIN] epoch=57, iter=21100/80000, loss=0.1451, lr=0.007592, batch_cost=0.6335, reader_cost=0.0006 | ETA 10:21:50 2020-11-04 01:07:45 [INFO] [TRAIN] epoch=57, iter=21200/80000, loss=0.1279, lr=0.007580, batch_cost=0.6340, reader_cost=0.0010 | ETA 10:21:20 2020-11-04 01:08:49 [INFO] [TRAIN] epoch=58, iter=21300/80000, loss=0.1343, lr=0.007568, batch_cost=0.6324, reader_cost=0.0092 | ETA 10:18:44 2020-11-04 01:09:50 [INFO] [TRAIN] epoch=58, iter=21400/80000, loss=0.1509, lr=0.007557, batch_cost=0.6180, reader_cost=0.0005 | ETA 10:03:37 2020-11-04 01:10:52 [INFO] [TRAIN] epoch=58, iter=21500/80000, loss=0.1322, lr=0.007545, batch_cost=0.6209, reader_cost=0.0002 | ETA 10:05:23 2020-11-04 01:11:57 [INFO] [TRAIN] epoch=59, iter=21600/80000, loss=0.1293, lr=0.007534, batch_cost=0.6420, reader_cost=0.0093 | ETA 10:24:52 2020-11-04 01:13:00 [INFO] [TRAIN] epoch=59, iter=21700/80000, loss=0.1347, lr=0.007522, batch_cost=0.6319, reader_cost=0.0005 | ETA 10:13:58 2020-11-04 01:14:03 [INFO] [TRAIN] epoch=59, iter=21800/80000, loss=0.1309, lr=0.007510, batch_cost=0.6317, reader_cost=0.0010 | ETA 10:12:45 2020-11-04 01:15:06 [INFO] [TRAIN] epoch=59, iter=21900/80000, loss=0.1449, lr=0.007499, batch_cost=0.6314, reader_cost=0.0008 | ETA 10:11:22 2020-11-04 01:16:11 [INFO] [TRAIN] epoch=60, iter=22000/80000, loss=0.1371, lr=0.007487, batch_cost=0.6449, reader_cost=0.0086 | ETA 10:23:21 2020-11-04 01:17:14 [INFO] [TRAIN] epoch=60, iter=22100/80000, loss=0.1382, lr=0.007475, batch_cost=0.6349, reader_cost=0.0010 | ETA 10:12:43 2020-11-04 01:18:17 [INFO] [TRAIN] epoch=60, iter=22200/80000, loss=0.1497, lr=0.007464, batch_cost=0.6325, reader_cost=0.0004 | ETA 10:09:20 2020-11-04 01:19:21 [INFO] [TRAIN] epoch=60, iter=22300/80000, loss=0.1502, lr=0.007452, batch_cost=0.6335, reader_cost=0.0004 | ETA 10:09:12 2020-11-04 01:20:25 [INFO] [TRAIN] epoch=61, iter=22400/80000, loss=0.1325, lr=0.007441, batch_cost=0.6447, reader_cost=0.0083 | ETA 10:18:53 2020-11-04 01:21:29 [INFO] [TRAIN] epoch=61, iter=22500/80000, loss=0.1497, lr=0.007429, batch_cost=0.6333, reader_cost=0.0010 | ETA 10:06:53 2020-11-04 01:22:32 [INFO] [TRAIN] epoch=61, iter=22600/80000, loss=0.1264, lr=0.007417, batch_cost=0.6307, reader_cost=0.0007 | ETA 10:03:21 2020-11-04 01:23:36 [INFO] [TRAIN] epoch=62, iter=22700/80000, loss=0.1470, lr=0.007406, batch_cost=0.6417, reader_cost=0.0097 | ETA 10:12:50 2020-11-04 01:24:39 [INFO] [TRAIN] epoch=62, iter=22800/80000, loss=0.1474, lr=0.007394, batch_cost=0.6341, reader_cost=0.0008 | ETA 10:04:30 2020-11-04 01:25:43 [INFO] [TRAIN] epoch=62, iter=22900/80000, loss=0.1237, lr=0.007382, batch_cost=0.6356, reader_cost=0.0009 | ETA 10:04:54 2020-11-04 01:26:46 [INFO] [TRAIN] epoch=62, iter=23000/80000, loss=0.1417, lr=0.007371, batch_cost=0.6329, reader_cost=0.0016 | ETA 10:01:12 2020-11-04 01:27:50 [INFO] [TRAIN] epoch=63, iter=23100/80000, loss=0.1418, lr=0.007359, batch_cost=0.6411, reader_cost=0.0092 | ETA 10:07:57 2020-11-04 01:28:54 [INFO] [TRAIN] epoch=63, iter=23200/80000, loss=0.1371, lr=0.007347, batch_cost=0.6342, reader_cost=0.0019 | ETA 10:00:21 2020-11-04 01:29:57 [INFO] [TRAIN] epoch=63, iter=23300/80000, loss=0.1498, lr=0.007336, batch_cost=0.6318, reader_cost=0.0019 | ETA 09:57:01 2020-11-04 01:31:00 [INFO] [TRAIN] epoch=63, iter=23400/80000, loss=0.1231, lr=0.007324, batch_cost=0.6306, reader_cost=0.0012 | ETA 09:54:54 2020-11-04 01:32:04 [INFO] [TRAIN] epoch=64, iter=23500/80000, loss=0.1180, lr=0.007313, batch_cost=0.6444, reader_cost=0.0096 | ETA 10:06:51 2020-11-04 01:33:08 [INFO] [TRAIN] epoch=64, iter=23600/80000, loss=0.1363, lr=0.007301, batch_cost=0.6332, reader_cost=0.0013 | ETA 09:55:13 2020-11-04 01:34:11 [INFO] [TRAIN] epoch=64, iter=23700/80000, loss=0.1372, lr=0.007289, batch_cost=0.6348, reader_cost=0.0010 | ETA 09:55:41 2020-11-04 01:35:14 [INFO] [TRAIN] epoch=64, iter=23800/80000, loss=0.1320, lr=0.007278, batch_cost=0.6299, reader_cost=0.0005 | ETA 09:50:02 2020-11-04 01:36:18 [INFO] [TRAIN] epoch=65, iter=23900/80000, loss=0.1362, lr=0.007266, batch_cost=0.6425, reader_cost=0.0103 | ETA 10:00:42 2020-11-04 01:37:22 [INFO] [TRAIN] epoch=65, iter=24000/80000, loss=0.1282, lr=0.007254, batch_cost=0.6317, reader_cost=0.0014 | ETA 09:49:36 2020-11-04 01:37:24 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-04 01:41:44 [INFO] [EVAL] #Images=500 mIoU=0.7640 Acc=0.9586 Kappa=0.9463 2020-11-04 01:41:44 [INFO] [EVAL] Category IoU: [0.9808 0.8433 0.9191 0.5531 0.6066 0.6432 0.6696 0.7764 0.92 0.5987 0.9459 0.8189 0.6153 0.9469 0.7833 0.8421 0.6467 0.6417 0.7641] 2020-11-04 01:41:44 [INFO] [EVAL] Category Acc: [0.9909 0.9018 0.9545 0.7858 0.8371 0.8075 0.7436 0.8898 0.9509 0.8856 0.9703 0.886 0.7965 0.9751 0.8675 0.9217 0.7046 0.7706 0.8735] 2020-11-04 01:41:45 [INFO] [EVAL] The model with the best validation mIoU (0.7640) was saved at iter 24000. 2020-11-04 01:42:48 [INFO] [TRAIN] epoch=65, iter=24100/80000, loss=0.1606, lr=0.007243, batch_cost=0.6268, reader_cost=0.0007 | ETA 09:44:00 2020-11-04 01:43:52 [INFO] [TRAIN] epoch=66, iter=24200/80000, loss=0.1266, lr=0.007231, batch_cost=0.6376, reader_cost=0.0091 | ETA 09:52:56 2020-11-04 01:44:55 [INFO] [TRAIN] epoch=66, iter=24300/80000, loss=0.1244, lr=0.007219, batch_cost=0.6309, reader_cost=0.0013 | ETA 09:45:39 2020-11-04 01:45:58 [INFO] [TRAIN] epoch=66, iter=24400/80000, loss=0.1374, lr=0.007208, batch_cost=0.6336, reader_cost=0.0009 | ETA 09:47:07 2020-11-04 01:47:02 [INFO] [TRAIN] epoch=66, iter=24500/80000, loss=0.1213, lr=0.007196, batch_cost=0.6371, reader_cost=0.0014 | ETA 09:49:21 2020-11-04 01:48:06 [INFO] [TRAIN] epoch=67, iter=24600/80000, loss=0.1360, lr=0.007184, batch_cost=0.6441, reader_cost=0.0098 | ETA 09:54:41 2020-11-04 01:49:10 [INFO] [TRAIN] epoch=67, iter=24700/80000, loss=0.1397, lr=0.007173, batch_cost=0.6361, reader_cost=0.0003 | ETA 09:46:15 2020-11-04 01:50:13 [INFO] [TRAIN] epoch=67, iter=24800/80000, loss=0.1254, lr=0.007161, batch_cost=0.6321, reader_cost=0.0010 | ETA 09:41:33 2020-11-04 01:51:16 [INFO] [TRAIN] epoch=67, iter=24900/80000, loss=0.1266, lr=0.007149, batch_cost=0.6326, reader_cost=0.0007 | ETA 09:40:55 2020-11-04 01:52:20 [INFO] [TRAIN] epoch=68, iter=25000/80000, loss=0.1235, lr=0.007138, batch_cost=0.6355, reader_cost=0.0092 | ETA 09:42:34 2020-11-04 01:53:22 [INFO] [TRAIN] epoch=68, iter=25100/80000, loss=0.1445, lr=0.007126, batch_cost=0.6242, reader_cost=0.0002 | ETA 09:31:05 2020-11-04 01:54:25 [INFO] [TRAIN] epoch=68, iter=25200/80000, loss=0.1268, lr=0.007114, batch_cost=0.6278, reader_cost=0.0005 | ETA 09:33:25 2020-11-04 01:55:29 [INFO] [TRAIN] epoch=69, iter=25300/80000, loss=0.1168, lr=0.007103, batch_cost=0.6427, reader_cost=0.0091 | ETA 09:45:54 2020-11-04 01:56:33 [INFO] [TRAIN] epoch=69, iter=25400/80000, loss=0.1318, lr=0.007091, batch_cost=0.6336, reader_cost=0.0011 | ETA 09:36:35 2020-11-04 01:57:36 [INFO] [TRAIN] epoch=69, iter=25500/80000, loss=0.1406, lr=0.007079, batch_cost=0.6334, reader_cost=0.0009 | ETA 09:35:17 2020-11-04 01:58:39 [INFO] [TRAIN] epoch=69, iter=25600/80000, loss=0.1477, lr=0.007067, batch_cost=0.6317, reader_cost=0.0006 | ETA 09:32:46 2020-11-04 01:59:43 [INFO] [TRAIN] epoch=70, iter=25700/80000, loss=0.1442, lr=0.007056, batch_cost=0.6434, reader_cost=0.0077 | ETA 09:42:18 2020-11-04 02:00:47 [INFO] [TRAIN] epoch=70, iter=25800/80000, loss=0.1440, lr=0.007044, batch_cost=0.6331, reader_cost=0.0010 | ETA 09:31:52 2020-11-04 02:01:50 [INFO] [TRAIN] epoch=70, iter=25900/80000, loss=0.1388, lr=0.007032, batch_cost=0.6341, reader_cost=0.0017 | ETA 09:31:46 2020-11-04 02:02:53 [INFO] [TRAIN] epoch=70, iter=26000/80000, loss=0.1412, lr=0.007021, batch_cost=0.6331, reader_cost=0.0014 | ETA 09:29:49 2020-11-04 02:03:58 [INFO] [TRAIN] epoch=71, iter=26100/80000, loss=0.1150, lr=0.007009, batch_cost=0.6426, reader_cost=0.0093 | ETA 09:37:18 2020-11-04 02:05:01 [INFO] [TRAIN] epoch=71, iter=26200/80000, loss=0.1333, lr=0.006997, batch_cost=0.6333, reader_cost=0.0009 | ETA 09:27:52 2020-11-04 02:06:04 [INFO] [TRAIN] epoch=71, iter=26300/80000, loss=0.1365, lr=0.006986, batch_cost=0.6333, reader_cost=0.0007 | ETA 09:26:46 2020-11-04 02:07:07 [INFO] [TRAIN] epoch=71, iter=26400/80000, loss=0.1177, lr=0.006974, batch_cost=0.6296, reader_cost=0.0010 | ETA 09:22:24 2020-11-04 02:08:12 [INFO] [TRAIN] epoch=72, iter=26500/80000, loss=0.1300, lr=0.006962, batch_cost=0.6452, reader_cost=0.0086 | ETA 09:35:17 2020-11-04 02:09:15 [INFO] [TRAIN] epoch=72, iter=26600/80000, loss=0.1427, lr=0.006950, batch_cost=0.6355, reader_cost=0.0008 | ETA 09:25:33 2020-11-04 02:10:19 [INFO] [TRAIN] epoch=72, iter=26700/80000, loss=0.1215, lr=0.006939, batch_cost=0.6328, reader_cost=0.0010 | ETA 09:22:05 2020-11-04 02:11:23 [INFO] [TRAIN] epoch=73, iter=26800/80000, loss=0.1137, lr=0.006927, batch_cost=0.6390, reader_cost=0.0092 | ETA 09:26:35 2020-11-04 02:12:26 [INFO] [TRAIN] epoch=73, iter=26900/80000, loss=0.1319, lr=0.006915, batch_cost=0.6312, reader_cost=0.0004 | ETA 09:18:35 2020-11-04 02:13:29 [INFO] [TRAIN] epoch=73, iter=27000/80000, loss=0.1475, lr=0.006904, batch_cost=0.6318, reader_cost=0.0005 | ETA 09:18:03 2020-11-04 02:14:32 [INFO] [TRAIN] epoch=73, iter=27100/80000, loss=0.1125, lr=0.006892, batch_cost=0.6332, reader_cost=0.0011 | ETA 09:18:17 2020-11-04 02:15:36 [INFO] [TRAIN] epoch=74, iter=27200/80000, loss=0.1321, lr=0.006880, batch_cost=0.6406, reader_cost=0.0094 | ETA 09:23:43 2020-11-04 02:16:40 [INFO] [TRAIN] epoch=74, iter=27300/80000, loss=0.1430, lr=0.006868, batch_cost=0.6353, reader_cost=0.0011 | ETA 09:18:01 2020-11-04 02:17:43 [INFO] [TRAIN] epoch=74, iter=27400/80000, loss=0.1212, lr=0.006857, batch_cost=0.6335, reader_cost=0.0010 | ETA 09:15:23 2020-11-04 02:18:46 [INFO] [TRAIN] epoch=74, iter=27500/80000, loss=0.1401, lr=0.006845, batch_cost=0.6301, reader_cost=0.0009 | ETA 09:11:22 2020-11-04 02:19:50 [INFO] [TRAIN] epoch=75, iter=27600/80000, loss=0.1221, lr=0.006833, batch_cost=0.6420, reader_cost=0.0101 | ETA 09:20:41 2020-11-04 02:20:54 [INFO] [TRAIN] epoch=75, iter=27700/80000, loss=0.1352, lr=0.006821, batch_cost=0.6327, reader_cost=0.0011 | ETA 09:11:29 2020-11-04 02:21:57 [INFO] [TRAIN] epoch=75, iter=27800/80000, loss=0.1377, lr=0.006810, batch_cost=0.6359, reader_cost=0.0011 | ETA 09:13:14 2020-11-04 02:23:00 [INFO] [TRAIN] epoch=75, iter=27900/80000, loss=0.1236, lr=0.006798, batch_cost=0.6320, reader_cost=0.0008 | ETA 09:08:47 2020-11-04 02:24:05 [INFO] [TRAIN] epoch=76, iter=28000/80000, loss=0.1252, lr=0.006786, batch_cost=0.6459, reader_cost=0.0079 | ETA 09:19:44 2020-11-04 02:25:08 [INFO] [TRAIN] epoch=76, iter=28100/80000, loss=0.1457, lr=0.006774, batch_cost=0.6337, reader_cost=0.0016 | ETA 09:08:10 2020-11-04 02:26:12 [INFO] [TRAIN] epoch=76, iter=28200/80000, loss=0.1291, lr=0.006763, batch_cost=0.6333, reader_cost=0.0009 | ETA 09:06:42 2020-11-04 02:27:16 [INFO] [TRAIN] epoch=77, iter=28300/80000, loss=0.1237, lr=0.006751, batch_cost=0.6431, reader_cost=0.0095 | ETA 09:14:09 2020-11-04 02:28:19 [INFO] [TRAIN] epoch=77, iter=28400/80000, loss=0.1363, lr=0.006739, batch_cost=0.6322, reader_cost=0.0011 | ETA 09:03:39 2020-11-04 02:29:23 [INFO] [TRAIN] epoch=77, iter=28500/80000, loss=0.1282, lr=0.006727, batch_cost=0.6336, reader_cost=0.0009 | ETA 09:03:52 2020-11-04 02:30:26 [INFO] [TRAIN] epoch=77, iter=28600/80000, loss=0.1117, lr=0.006716, batch_cost=0.6317, reader_cost=0.0007 | ETA 09:01:11 2020-11-04 02:31:30 [INFO] [TRAIN] epoch=78, iter=28700/80000, loss=0.1128, lr=0.006704, batch_cost=0.6399, reader_cost=0.0086 | ETA 09:07:06 2020-11-04 02:32:33 [INFO] [TRAIN] epoch=78, iter=28800/80000, loss=0.1448, lr=0.006692, batch_cost=0.6296, reader_cost=0.0007 | ETA 08:57:13 2020-11-04 02:33:36 [INFO] [TRAIN] epoch=78, iter=28900/80000, loss=0.1255, lr=0.006680, batch_cost=0.6332, reader_cost=0.0010 | ETA 08:59:17 2020-11-04 02:34:39 [INFO] [TRAIN] epoch=78, iter=29000/80000, loss=0.1141, lr=0.006669, batch_cost=0.6314, reader_cost=0.0004 | ETA 08:56:41 2020-11-04 02:35:43 [INFO] [TRAIN] epoch=79, iter=29100/80000, loss=0.1232, lr=0.006657, batch_cost=0.6395, reader_cost=0.0092 | ETA 09:02:30 2020-11-04 02:36:45 [INFO] [TRAIN] epoch=79, iter=29200/80000, loss=0.1237, lr=0.006645, batch_cost=0.6171, reader_cost=0.0002 | ETA 08:42:27 2020-11-04 02:37:46 [INFO] [TRAIN] epoch=79, iter=29300/80000, loss=0.1350, lr=0.006633, batch_cost=0.6158, reader_cost=0.0002 | ETA 08:40:22 2020-11-04 02:38:50 [INFO] [TRAIN] epoch=80, iter=29400/80000, loss=0.1286, lr=0.006622, batch_cost=0.6323, reader_cost=0.0087 | ETA 08:53:14 2020-11-04 02:39:53 [INFO] [TRAIN] epoch=80, iter=29500/80000, loss=0.1238, lr=0.006610, batch_cost=0.6299, reader_cost=0.0006 | ETA 08:50:09 2020-11-04 02:40:56 [INFO] [TRAIN] epoch=80, iter=29600/80000, loss=0.1280, lr=0.006598, batch_cost=0.6297, reader_cost=0.0012 | ETA 08:48:54 2020-11-04 02:41:59 [INFO] [TRAIN] epoch=80, iter=29700/80000, loss=0.1404, lr=0.006586, batch_cost=0.6354, reader_cost=0.0013 | ETA 08:52:39 2020-11-04 02:43:03 [INFO] [TRAIN] epoch=81, iter=29800/80000, loss=0.1514, lr=0.006574, batch_cost=0.6410, reader_cost=0.0090 | ETA 08:56:18 2020-11-04 02:44:06 [INFO] [TRAIN] epoch=81, iter=29900/80000, loss=0.1456, lr=0.006563, batch_cost=0.6303, reader_cost=0.0010 | ETA 08:46:20 2020-11-04 02:45:09 [INFO] [TRAIN] epoch=81, iter=30000/80000, loss=0.1414, lr=0.006551, batch_cost=0.6314, reader_cost=0.0006 | ETA 08:46:12 2020-11-04 02:46:13 [INFO] [TRAIN] epoch=81, iter=30100/80000, loss=0.1401, lr=0.006539, batch_cost=0.6321, reader_cost=0.0006 | ETA 08:45:43 2020-11-04 02:47:17 [INFO] [TRAIN] epoch=82, iter=30200/80000, loss=0.1135, lr=0.006527, batch_cost=0.6442, reader_cost=0.0089 | ETA 08:54:39 2020-11-04 02:48:20 [INFO] [TRAIN] epoch=82, iter=30300/80000, loss=0.1544, lr=0.006515, batch_cost=0.6302, reader_cost=0.0011 | ETA 08:42:00 2020-11-04 02:49:23 [INFO] [TRAIN] epoch=82, iter=30400/80000, loss=0.1397, lr=0.006504, batch_cost=0.6307, reader_cost=0.0006 | ETA 08:41:21 2020-11-04 02:50:27 [INFO] [TRAIN] epoch=82, iter=30500/80000, loss=0.1188, lr=0.006492, batch_cost=0.6336, reader_cost=0.0006 | ETA 08:42:43 2020-11-04 02:51:30 [INFO] [TRAIN] epoch=83, iter=30600/80000, loss=0.1206, lr=0.006480, batch_cost=0.6391, reader_cost=0.0086 | ETA 08:46:13 2020-11-04 02:52:34 [INFO] [TRAIN] epoch=83, iter=30700/80000, loss=0.1433, lr=0.006468, batch_cost=0.6313, reader_cost=0.0011 | ETA 08:38:42 2020-11-04 02:53:37 [INFO] [TRAIN] epoch=83, iter=30800/80000, loss=0.1231, lr=0.006456, batch_cost=0.6329, reader_cost=0.0018 | ETA 08:38:59 2020-11-04 02:54:41 [INFO] [TRAIN] epoch=84, iter=30900/80000, loss=0.1207, lr=0.006445, batch_cost=0.6401, reader_cost=0.0088 | ETA 08:43:49 2020-11-04 02:55:44 [INFO] [TRAIN] epoch=84, iter=31000/80000, loss=0.1261, lr=0.006433, batch_cost=0.6316, reader_cost=0.0003 | ETA 08:35:46 2020-11-04 02:56:47 [INFO] [TRAIN] epoch=84, iter=31100/80000, loss=0.1221, lr=0.006421, batch_cost=0.6304, reader_cost=0.0005 | ETA 08:33:46 2020-11-04 02:57:50 [INFO] [TRAIN] epoch=84, iter=31200/80000, loss=0.1087, lr=0.006409, batch_cost=0.6308, reader_cost=0.0008 | ETA 08:33:01 2020-11-04 02:58:54 [INFO] [TRAIN] epoch=85, iter=31300/80000, loss=0.1252, lr=0.006397, batch_cost=0.6430, reader_cost=0.0095 | ETA 08:41:54 2020-11-04 02:59:58 [INFO] [TRAIN] epoch=85, iter=31400/80000, loss=0.1439, lr=0.006386, batch_cost=0.6342, reader_cost=0.0011 | ETA 08:33:42 2020-11-04 03:01:01 [INFO] [TRAIN] epoch=85, iter=31500/80000, loss=0.1550, lr=0.006374, batch_cost=0.6319, reader_cost=0.0013 | ETA 08:30:47 2020-11-04 03:02:04 [INFO] [TRAIN] epoch=85, iter=31600/80000, loss=0.1258, lr=0.006362, batch_cost=0.6288, reader_cost=0.0004 | ETA 08:27:16 2020-11-04 03:03:08 [INFO] [TRAIN] epoch=86, iter=31700/80000, loss=0.1215, lr=0.006350, batch_cost=0.6421, reader_cost=0.0095 | ETA 08:36:55 2020-11-04 03:04:12 [INFO] [TRAIN] epoch=86, iter=31800/80000, loss=0.1201, lr=0.006338, batch_cost=0.6336, reader_cost=0.0016 | ETA 08:29:01 2020-11-04 03:05:15 [INFO] [TRAIN] epoch=86, iter=31900/80000, loss=0.1096, lr=0.006326, batch_cost=0.6330, reader_cost=0.0011 | ETA 08:27:25 2020-11-04 03:06:19 [INFO] [TRAIN] epoch=87, iter=32000/80000, loss=0.1132, lr=0.006315, batch_cost=0.6445, reader_cost=0.0098 | ETA 08:35:34 2020-11-04 03:06:22 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-04 03:10:39 [INFO] [EVAL] #Images=500 mIoU=0.7764 Acc=0.9588 Kappa=0.9465 2020-11-04 03:10:39 [INFO] [EVAL] Category IoU: [0.9786 0.8332 0.921 0.4805 0.626 0.6526 0.7011 0.7864 0.9217 0.6411 0.9469 0.8245 0.6262 0.9504 0.8248 0.8601 0.738 0.6682 0.77 ] 2020-11-04 03:10:39 [INFO] [EVAL] Category Acc: [0.9886 0.9234 0.9593 0.8627 0.7823 0.7899 0.813 0.8995 0.949 0.7825 0.9728 0.8922 0.7479 0.9722 0.9039 0.9019 0.9283 0.8024 0.8591] 2020-11-04 03:10:40 [INFO] [EVAL] The model with the best validation mIoU (0.7764) was saved at iter 32000. 2020-11-04 03:11:43 [INFO] [TRAIN] epoch=87, iter=32100/80000, loss=0.1191, lr=0.006303, batch_cost=0.6294, reader_cost=0.0004 | ETA 08:22:29 2020-11-04 03:12:46 [INFO] [TRAIN] epoch=87, iter=32200/80000, loss=0.1227, lr=0.006291, batch_cost=0.6310, reader_cost=0.0010 | ETA 08:22:43 2020-11-04 03:13:49 [INFO] [TRAIN] epoch=87, iter=32300/80000, loss=0.1285, lr=0.006279, batch_cost=0.6315, reader_cost=0.0013 | ETA 08:22:01 2020-11-04 03:14:54 [INFO] [TRAIN] epoch=88, iter=32400/80000, loss=0.1204, lr=0.006267, batch_cost=0.6428, reader_cost=0.0097 | ETA 08:29:55 2020-11-04 03:15:57 [INFO] [TRAIN] epoch=88, iter=32500/80000, loss=0.1229, lr=0.006255, batch_cost=0.6320, reader_cost=0.0011 | ETA 08:20:18 2020-11-04 03:17:00 [INFO] [TRAIN] epoch=88, iter=32600/80000, loss=0.1218, lr=0.006243, batch_cost=0.6352, reader_cost=0.0006 | ETA 08:21:48 2020-11-04 03:18:04 [INFO] [TRAIN] epoch=88, iter=32700/80000, loss=0.1135, lr=0.006232, batch_cost=0.6340, reader_cost=0.0004 | ETA 08:19:46 2020-11-04 03:19:08 [INFO] [TRAIN] epoch=89, iter=32800/80000, loss=0.1184, lr=0.006220, batch_cost=0.6414, reader_cost=0.0099 | ETA 08:24:35 2020-11-04 03:20:10 [INFO] [TRAIN] epoch=89, iter=32900/80000, loss=0.1411, lr=0.006208, batch_cost=0.6216, reader_cost=0.0005 | ETA 08:07:59 2020-11-04 03:21:12 [INFO] [TRAIN] epoch=89, iter=33000/80000, loss=0.1485, lr=0.006196, batch_cost=0.6163, reader_cost=0.0002 | ETA 08:02:47 2020-11-04 03:22:13 [INFO] [TRAIN] epoch=89, iter=33100/80000, loss=0.1212, lr=0.006184, batch_cost=0.6158, reader_cost=0.0002 | ETA 08:01:19 2020-11-04 03:23:17 [INFO] [TRAIN] epoch=90, iter=33200/80000, loss=0.1308, lr=0.006172, batch_cost=0.6398, reader_cost=0.0101 | ETA 08:19:02 2020-11-04 03:24:21 [INFO] [TRAIN] epoch=90, iter=33300/80000, loss=0.1221, lr=0.006160, batch_cost=0.6322, reader_cost=0.0010 | ETA 08:12:04 2020-11-04 03:25:24 [INFO] [TRAIN] epoch=90, iter=33400/80000, loss=0.1248, lr=0.006149, batch_cost=0.6327, reader_cost=0.0014 | ETA 08:11:21 2020-11-04 03:26:28 [INFO] [TRAIN] epoch=91, iter=33500/80000, loss=0.1168, lr=0.006137, batch_cost=0.6432, reader_cost=0.0097 | ETA 08:18:27 2020-11-04 03:27:31 [INFO] [TRAIN] epoch=91, iter=33600/80000, loss=0.1385, lr=0.006125, batch_cost=0.6315, reader_cost=0.0008 | ETA 08:08:22 2020-11-04 03:28:34 [INFO] [TRAIN] epoch=91, iter=33700/80000, loss=0.1421, lr=0.006113, batch_cost=0.6284, reader_cost=0.0006 | ETA 08:04:56 2020-11-04 03:29:37 [INFO] [TRAIN] epoch=91, iter=33800/80000, loss=0.1213, lr=0.006101, batch_cost=0.6314, reader_cost=0.0004 | ETA 08:06:10 2020-11-04 03:30:41 [INFO] [TRAIN] epoch=92, iter=33900/80000, loss=0.1217, lr=0.006089, batch_cost=0.6415, reader_cost=0.0093 | ETA 08:12:53 2020-11-04 03:31:44 [INFO] [TRAIN] epoch=92, iter=34000/80000, loss=0.1321, lr=0.006077, batch_cost=0.6295, reader_cost=0.0008 | ETA 08:02:35 2020-11-04 03:32:47 [INFO] [TRAIN] epoch=92, iter=34100/80000, loss=0.1538, lr=0.006065, batch_cost=0.6236, reader_cost=0.0006 | ETA 07:57:05 2020-11-04 03:33:50 [INFO] [TRAIN] epoch=92, iter=34200/80000, loss=0.1220, lr=0.006053, batch_cost=0.6317, reader_cost=0.0004 | ETA 08:02:11 2020-11-04 03:34:54 [INFO] [TRAIN] epoch=93, iter=34300/80000, loss=0.1308, lr=0.006042, batch_cost=0.6413, reader_cost=0.0079 | ETA 08:08:29 2020-11-04 03:35:57 [INFO] [TRAIN] epoch=93, iter=34400/80000, loss=0.1220, lr=0.006030, batch_cost=0.6343, reader_cost=0.0010 | ETA 08:02:06 2020-11-04 03:37:01 [INFO] [TRAIN] epoch=93, iter=34500/80000, loss=0.1188, lr=0.006018, batch_cost=0.6321, reader_cost=0.0007 | ETA 07:59:19 2020-11-04 03:38:05 [INFO] [TRAIN] epoch=94, iter=34600/80000, loss=0.1207, lr=0.006006, batch_cost=0.6390, reader_cost=0.0087 | ETA 08:03:31 2020-11-04 03:39:07 [INFO] [TRAIN] epoch=94, iter=34700/80000, loss=0.1399, lr=0.005994, batch_cost=0.6287, reader_cost=0.0003 | ETA 07:54:37 2020-11-04 03:40:10 [INFO] [TRAIN] epoch=94, iter=34800/80000, loss=0.1421, lr=0.005982, batch_cost=0.6289, reader_cost=0.0010 | ETA 07:53:44 2020-11-04 03:41:14 [INFO] [TRAIN] epoch=94, iter=34900/80000, loss=0.1375, lr=0.005970, batch_cost=0.6326, reader_cost=0.0014 | ETA 07:55:31 2020-11-04 03:42:18 [INFO] [TRAIN] epoch=95, iter=35000/80000, loss=0.1200, lr=0.005958, batch_cost=0.6439, reader_cost=0.0093 | ETA 08:02:56 2020-11-04 03:43:21 [INFO] [TRAIN] epoch=95, iter=35100/80000, loss=0.1139, lr=0.005946, batch_cost=0.6320, reader_cost=0.0009 | ETA 07:52:54 2020-11-04 03:44:25 [INFO] [TRAIN] epoch=95, iter=35200/80000, loss=0.1260, lr=0.005934, batch_cost=0.6359, reader_cost=0.0011 | ETA 07:54:50 2020-11-04 03:45:28 [INFO] [TRAIN] epoch=95, iter=35300/80000, loss=0.1116, lr=0.005922, batch_cost=0.6336, reader_cost=0.0018 | ETA 07:52:00 2020-11-04 03:46:33 [INFO] [TRAIN] epoch=96, iter=35400/80000, loss=0.1108, lr=0.005911, batch_cost=0.6434, reader_cost=0.0103 | ETA 07:58:15 2020-11-04 03:47:36 [INFO] [TRAIN] epoch=96, iter=35500/80000, loss=0.1245, lr=0.005899, batch_cost=0.6338, reader_cost=0.0012 | ETA 07:50:05 2020-11-04 03:48:39 [INFO] [TRAIN] epoch=96, iter=35600/80000, loss=0.1372, lr=0.005887, batch_cost=0.6318, reader_cost=0.0012 | ETA 07:47:32 2020-11-04 03:49:42 [INFO] [TRAIN] epoch=96, iter=35700/80000, loss=0.1181, lr=0.005875, batch_cost=0.6300, reader_cost=0.0006 | ETA 07:45:07 2020-11-04 03:50:46 [INFO] [TRAIN] epoch=97, iter=35800/80000, loss=0.1265, lr=0.005863, batch_cost=0.6407, reader_cost=0.0087 | ETA 07:51:59 2020-11-04 03:51:49 [INFO] [TRAIN] epoch=97, iter=35900/80000, loss=0.1226, lr=0.005851, batch_cost=0.6325, reader_cost=0.0007 | ETA 07:44:54 2020-11-04 03:52:52 [INFO] [TRAIN] epoch=97, iter=36000/80000, loss=0.1254, lr=0.005839, batch_cost=0.6285, reader_cost=0.0007 | ETA 07:40:55 2020-11-04 03:53:56 [INFO] [TRAIN] epoch=98, iter=36100/80000, loss=0.1213, lr=0.005827, batch_cost=0.6391, reader_cost=0.0084 | ETA 07:47:36 2020-11-04 03:54:59 [INFO] [TRAIN] epoch=98, iter=36200/80000, loss=0.1205, lr=0.005815, batch_cost=0.6321, reader_cost=0.0007 | ETA 07:41:27 2020-11-04 03:56:02 [INFO] [TRAIN] epoch=98, iter=36300/80000, loss=0.1343, lr=0.005803, batch_cost=0.6304, reader_cost=0.0005 | ETA 07:39:07 2020-11-04 03:57:06 [INFO] [TRAIN] epoch=98, iter=36400/80000, loss=0.1159, lr=0.005791, batch_cost=0.6320, reader_cost=0.0004 | ETA 07:39:17 2020-11-04 03:58:10 [INFO] [TRAIN] epoch=99, iter=36500/80000, loss=0.1076, lr=0.005779, batch_cost=0.6415, reader_cost=0.0082 | ETA 07:45:04 2020-11-04 03:59:13 [INFO] [TRAIN] epoch=99, iter=36600/80000, loss=0.1244, lr=0.005767, batch_cost=0.6327, reader_cost=0.0007 | ETA 07:37:39 2020-11-04 04:00:17 [INFO] [TRAIN] epoch=99, iter=36700/80000, loss=0.1311, lr=0.005755, batch_cost=0.6352, reader_cost=0.0012 | ETA 07:38:23 2020-11-04 04:01:20 [INFO] [TRAIN] epoch=99, iter=36800/80000, loss=0.1109, lr=0.005743, batch_cost=0.6316, reader_cost=0.0010 | ETA 07:34:46 2020-11-04 04:02:24 [INFO] [TRAIN] epoch=100, iter=36900/80000, loss=0.1224, lr=0.005731, batch_cost=0.6440, reader_cost=0.0097 | ETA 07:42:34 2020-11-04 04:03:27 [INFO] [TRAIN] epoch=100, iter=37000/80000, loss=0.1440, lr=0.005719, batch_cost=0.6314, reader_cost=0.0013 | ETA 07:32:32 2020-11-04 04:04:29 [INFO] [TRAIN] epoch=100, iter=37100/80000, loss=0.1109, lr=0.005707, batch_cost=0.6212, reader_cost=0.0004 | ETA 07:24:09 2020-11-04 04:05:31 [INFO] [TRAIN] epoch=100, iter=37200/80000, loss=0.1138, lr=0.005695, batch_cost=0.6184, reader_cost=0.0002 | ETA 07:21:08 2020-11-04 04:06:34 [INFO] [TRAIN] epoch=101, iter=37300/80000, loss=0.1305, lr=0.005683, batch_cost=0.6321, reader_cost=0.0087 | ETA 07:29:51 2020-11-04 04:07:38 [INFO] [TRAIN] epoch=101, iter=37400/80000, loss=0.1244, lr=0.005671, batch_cost=0.6327, reader_cost=0.0010 | ETA 07:29:14 2020-11-04 04:08:41 [INFO] [TRAIN] epoch=101, iter=37500/80000, loss=0.1100, lr=0.005660, batch_cost=0.6336, reader_cost=0.0012 | ETA 07:28:49 2020-11-04 04:09:45 [INFO] [TRAIN] epoch=102, iter=37600/80000, loss=0.1226, lr=0.005648, batch_cost=0.6431, reader_cost=0.0082 | ETA 07:34:27 2020-11-04 04:10:49 [INFO] [TRAIN] epoch=102, iter=37700/80000, loss=0.1168, lr=0.005636, batch_cost=0.6319, reader_cost=0.0009 | ETA 07:25:30 2020-11-04 04:11:52 [INFO] [TRAIN] epoch=102, iter=37800/80000, loss=0.1310, lr=0.005624, batch_cost=0.6332, reader_cost=0.0007 | ETA 07:25:19 2020-11-04 04:12:55 [INFO] [TRAIN] epoch=102, iter=37900/80000, loss=0.1046, lr=0.005612, batch_cost=0.6319, reader_cost=0.0006 | ETA 07:23:21 2020-11-04 04:13:59 [INFO] [TRAIN] epoch=103, iter=38000/80000, loss=0.1041, lr=0.005600, batch_cost=0.6406, reader_cost=0.0088 | ETA 07:28:25 2020-11-04 04:15:03 [INFO] [TRAIN] epoch=103, iter=38100/80000, loss=0.1289, lr=0.005588, batch_cost=0.6341, reader_cost=0.0008 | ETA 07:22:47 2020-11-04 04:16:06 [INFO] [TRAIN] epoch=103, iter=38200/80000, loss=0.1257, lr=0.005576, batch_cost=0.6311, reader_cost=0.0010 | ETA 07:19:38 2020-11-04 04:17:09 [INFO] [TRAIN] epoch=103, iter=38300/80000, loss=0.1179, lr=0.005564, batch_cost=0.6290, reader_cost=0.0009 | ETA 07:17:08 2020-11-04 04:18:13 [INFO] [TRAIN] epoch=104, iter=38400/80000, loss=0.1127, lr=0.005552, batch_cost=0.6432, reader_cost=0.0098 | ETA 07:25:58 2020-11-04 04:19:16 [INFO] [TRAIN] epoch=104, iter=38500/80000, loss=0.1142, lr=0.005540, batch_cost=0.6348, reader_cost=0.0018 | ETA 07:19:02 2020-11-04 04:20:20 [INFO] [TRAIN] epoch=104, iter=38600/80000, loss=0.1138, lr=0.005528, batch_cost=0.6343, reader_cost=0.0008 | ETA 07:17:38 2020-11-04 04:21:24 [INFO] [TRAIN] epoch=105, iter=38700/80000, loss=0.1016, lr=0.005515, batch_cost=0.6426, reader_cost=0.0095 | ETA 07:22:19 2020-11-04 04:22:27 [INFO] [TRAIN] epoch=105, iter=38800/80000, loss=0.1123, lr=0.005503, batch_cost=0.6318, reader_cost=0.0009 | ETA 07:13:48 2020-11-04 04:23:31 [INFO] [TRAIN] epoch=105, iter=38900/80000, loss=0.1210, lr=0.005491, batch_cost=0.6340, reader_cost=0.0012 | ETA 07:14:17 2020-11-04 04:24:34 [INFO] [TRAIN] epoch=105, iter=39000/80000, loss=0.1154, lr=0.005479, batch_cost=0.6348, reader_cost=0.0005 | ETA 07:13:46 2020-11-04 04:25:38 [INFO] [TRAIN] epoch=106, iter=39100/80000, loss=0.1092, lr=0.005467, batch_cost=0.6436, reader_cost=0.0089 | ETA 07:18:42 2020-11-04 04:26:42 [INFO] [TRAIN] epoch=106, iter=39200/80000, loss=0.1353, lr=0.005455, batch_cost=0.6369, reader_cost=0.0009 | ETA 07:13:04 2020-11-04 04:27:45 [INFO] [TRAIN] epoch=106, iter=39300/80000, loss=0.1464, lr=0.005443, batch_cost=0.6320, reader_cost=0.0016 | ETA 07:08:42 2020-11-04 04:28:49 [INFO] [TRAIN] epoch=106, iter=39400/80000, loss=0.1101, lr=0.005431, batch_cost=0.6324, reader_cost=0.0014 | ETA 07:07:54 2020-11-04 04:29:53 [INFO] [TRAIN] epoch=107, iter=39500/80000, loss=0.1119, lr=0.005419, batch_cost=0.6425, reader_cost=0.0092 | ETA 07:13:42 2020-11-04 04:30:56 [INFO] [TRAIN] epoch=107, iter=39600/80000, loss=0.1438, lr=0.005407, batch_cost=0.6328, reader_cost=0.0016 | ETA 07:06:04 2020-11-04 04:32:00 [INFO] [TRAIN] epoch=107, iter=39700/80000, loss=0.1194, lr=0.005395, batch_cost=0.6359, reader_cost=0.0013 | ETA 07:07:07 2020-11-04 04:33:03 [INFO] [TRAIN] epoch=107, iter=39800/80000, loss=0.1090, lr=0.005383, batch_cost=0.6335, reader_cost=0.0019 | ETA 07:04:25 2020-11-04 04:34:07 [INFO] [TRAIN] epoch=108, iter=39900/80000, loss=0.1170, lr=0.005371, batch_cost=0.6411, reader_cost=0.0102 | ETA 07:08:26 2020-11-04 04:35:11 [INFO] [TRAIN] epoch=108, iter=40000/80000, loss=0.1335, lr=0.005359, batch_cost=0.6343, reader_cost=0.0008 | ETA 07:02:52 2020-11-04 04:35:13 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-04 04:39:34 [INFO] [EVAL] #Images=500 mIoU=0.7855 Acc=0.9615 Kappa=0.9500 2020-11-04 04:39:34 [INFO] [EVAL] Category IoU: [0.9823 0.8549 0.9247 0.5535 0.5982 0.6613 0.7057 0.7943 0.9245 0.639 0.9453 0.8267 0.6282 0.9526 0.8285 0.8925 0.7758 0.663 0.7729] 2020-11-04 04:39:34 [INFO] [EVAL] Category Acc: [0.9898 0.9318 0.9538 0.8054 0.8464 0.8137 0.8376 0.8997 0.9576 0.8482 0.9633 0.8868 0.7907 0.9741 0.8885 0.964 0.8736 0.7595 0.881 ] 2020-11-04 04:39:35 [INFO] [EVAL] The model with the best validation mIoU (0.7855) was saved at iter 40000. 2020-11-04 04:40:38 [INFO] [TRAIN] epoch=108, iter=40100/80000, loss=0.1208, lr=0.005347, batch_cost=0.6308, reader_cost=0.0007 | ETA 06:59:27 2020-11-04 04:41:42 [INFO] [TRAIN] epoch=109, iter=40200/80000, loss=0.1107, lr=0.005335, batch_cost=0.6400, reader_cost=0.0090 | ETA 07:04:31 2020-11-04 04:42:45 [INFO] [TRAIN] epoch=109, iter=40300/80000, loss=0.1073, lr=0.005323, batch_cost=0.6288, reader_cost=0.0005 | ETA 06:56:02 2020-11-04 04:43:49 [INFO] [TRAIN] epoch=109, iter=40400/80000, loss=0.1268, lr=0.005311, batch_cost=0.6360, reader_cost=0.0014 | ETA 06:59:46 2020-11-04 04:44:52 [INFO] [TRAIN] epoch=109, iter=40500/80000, loss=0.1139, lr=0.005299, batch_cost=0.6366, reader_cost=0.0013 | ETA 06:59:07 2020-11-04 04:45:56 [INFO] [TRAIN] epoch=110, iter=40600/80000, loss=0.1074, lr=0.005287, batch_cost=0.6401, reader_cost=0.0087 | ETA 07:00:19 2020-11-04 04:47:00 [INFO] [TRAIN] epoch=110, iter=40700/80000, loss=0.1157, lr=0.005275, batch_cost=0.6339, reader_cost=0.0013 | ETA 06:55:13 2020-11-04 04:48:02 [INFO] [TRAIN] epoch=110, iter=40800/80000, loss=0.1247, lr=0.005262, batch_cost=0.6208, reader_cost=0.0002 | ETA 06:45:35 2020-11-04 04:49:04 [INFO] [TRAIN] epoch=110, iter=40900/80000, loss=0.0961, lr=0.005250, batch_cost=0.6189, reader_cost=0.0002 | ETA 06:43:19 2020-11-04 04:50:07 [INFO] [TRAIN] epoch=111, iter=41000/80000, loss=0.1236, lr=0.005238, batch_cost=0.6298, reader_cost=0.0097 | ETA 06:49:23 2020-11-04 04:51:10 [INFO] [TRAIN] epoch=111, iter=41100/80000, loss=0.1213, lr=0.005226, batch_cost=0.6296, reader_cost=0.0008 | ETA 06:48:10 2020-11-04 04:52:12 [INFO] [TRAIN] epoch=111, iter=41200/80000, loss=0.1207, lr=0.005214, batch_cost=0.6288, reader_cost=0.0009 | ETA 06:46:35 2020-11-04 04:53:16 [INFO] [TRAIN] epoch=112, iter=41300/80000, loss=0.1379, lr=0.005202, batch_cost=0.6388, reader_cost=0.0094 | ETA 06:52:01 2020-11-04 04:54:19 [INFO] [TRAIN] epoch=112, iter=41400/80000, loss=0.1274, lr=0.005190, batch_cost=0.6286, reader_cost=0.0007 | ETA 06:44:25 2020-11-04 04:55:23 [INFO] [TRAIN] epoch=112, iter=41500/80000, loss=0.1280, lr=0.005178, batch_cost=0.6336, reader_cost=0.0004 | ETA 06:46:35 2020-11-04 04:56:26 [INFO] [TRAIN] epoch=112, iter=41600/80000, loss=0.1123, lr=0.005166, batch_cost=0.6363, reader_cost=0.0004 | ETA 06:47:13 2020-11-04 04:57:30 [INFO] [TRAIN] epoch=113, iter=41700/80000, loss=0.1231, lr=0.005154, batch_cost=0.6424, reader_cost=0.0093 | ETA 06:50:05 2020-11-04 04:58:34 [INFO] [TRAIN] epoch=113, iter=41800/80000, loss=0.1226, lr=0.005141, batch_cost=0.6341, reader_cost=0.0011 | ETA 06:43:43 2020-11-04 04:59:37 [INFO] [TRAIN] epoch=113, iter=41900/80000, loss=0.1171, lr=0.005129, batch_cost=0.6322, reader_cost=0.0011 | ETA 06:41:28 2020-11-04 05:00:40 [INFO] [TRAIN] epoch=113, iter=42000/80000, loss=0.1117, lr=0.005117, batch_cost=0.6306, reader_cost=0.0014 | ETA 06:39:22 2020-11-04 05:01:44 [INFO] [TRAIN] epoch=114, iter=42100/80000, loss=0.1007, lr=0.005105, batch_cost=0.6398, reader_cost=0.0097 | ETA 06:44:07 2020-11-04 05:02:47 [INFO] [TRAIN] epoch=114, iter=42200/80000, loss=0.1291, lr=0.005093, batch_cost=0.6331, reader_cost=0.0012 | ETA 06:38:52 2020-11-04 05:03:51 [INFO] [TRAIN] epoch=114, iter=42300/80000, loss=0.1209, lr=0.005081, batch_cost=0.6341, reader_cost=0.0007 | ETA 06:38:26 2020-11-04 05:04:54 [INFO] [TRAIN] epoch=114, iter=42400/80000, loss=0.0993, lr=0.005069, batch_cost=0.6350, reader_cost=0.0006 | ETA 06:37:55 2020-11-04 05:05:59 [INFO] [TRAIN] epoch=115, iter=42500/80000, loss=0.1112, lr=0.005057, batch_cost=0.6456, reader_cost=0.0099 | ETA 06:43:28 2020-11-04 05:07:02 [INFO] [TRAIN] epoch=115, iter=42600/80000, loss=0.1147, lr=0.005044, batch_cost=0.6344, reader_cost=0.0015 | ETA 06:35:25 2020-11-04 05:08:06 [INFO] [TRAIN] epoch=115, iter=42700/80000, loss=0.1164, lr=0.005032, batch_cost=0.6331, reader_cost=0.0016 | ETA 06:33:33 2020-11-04 05:09:10 [INFO] [TRAIN] epoch=116, iter=42800/80000, loss=0.1060, lr=0.005020, batch_cost=0.6396, reader_cost=0.0095 | ETA 06:36:34 2020-11-04 05:10:13 [INFO] [TRAIN] epoch=116, iter=42900/80000, loss=0.1039, lr=0.005008, batch_cost=0.6342, reader_cost=0.0003 | ETA 06:32:10 2020-11-04 05:11:16 [INFO] [TRAIN] epoch=116, iter=43000/80000, loss=0.1377, lr=0.004996, batch_cost=0.6297, reader_cost=0.0004 | ETA 06:28:17 2020-11-04 05:12:19 [INFO] [TRAIN] epoch=116, iter=43100/80000, loss=0.1036, lr=0.004984, batch_cost=0.6307, reader_cost=0.0005 | ETA 06:27:53 2020-11-04 05:13:23 [INFO] [TRAIN] epoch=117, iter=43200/80000, loss=0.1052, lr=0.004972, batch_cost=0.6411, reader_cost=0.0092 | ETA 06:33:11 2020-11-04 05:14:27 [INFO] [TRAIN] epoch=117, iter=43300/80000, loss=0.1216, lr=0.004959, batch_cost=0.6346, reader_cost=0.0010 | ETA 06:28:10 2020-11-04 05:15:30 [INFO] [TRAIN] epoch=117, iter=43400/80000, loss=0.1219, lr=0.004947, batch_cost=0.6330, reader_cost=0.0011 | ETA 06:26:09 2020-11-04 05:16:33 [INFO] [TRAIN] epoch=117, iter=43500/80000, loss=0.1168, lr=0.004935, batch_cost=0.6294, reader_cost=0.0006 | ETA 06:22:52 2020-11-04 05:17:37 [INFO] [TRAIN] epoch=118, iter=43600/80000, loss=0.1092, lr=0.004923, batch_cost=0.6419, reader_cost=0.0093 | ETA 06:29:25 2020-11-04 05:18:40 [INFO] [TRAIN] epoch=118, iter=43700/80000, loss=0.1080, lr=0.004911, batch_cost=0.6328, reader_cost=0.0010 | ETA 06:22:49 2020-11-04 05:19:44 [INFO] [TRAIN] epoch=118, iter=43800/80000, loss=0.1150, lr=0.004899, batch_cost=0.6314, reader_cost=0.0014 | ETA 06:20:55 2020-11-04 05:20:48 [INFO] [TRAIN] epoch=119, iter=43900/80000, loss=0.1057, lr=0.004886, batch_cost=0.6452, reader_cost=0.0095 | ETA 06:28:10 2020-11-04 05:21:51 [INFO] [TRAIN] epoch=119, iter=44000/80000, loss=0.1165, lr=0.004874, batch_cost=0.6334, reader_cost=0.0012 | ETA 06:20:01 2020-11-04 05:22:55 [INFO] [TRAIN] epoch=119, iter=44100/80000, loss=0.1065, lr=0.004862, batch_cost=0.6316, reader_cost=0.0016 | ETA 06:17:53 2020-11-04 05:23:58 [INFO] [TRAIN] epoch=119, iter=44200/80000, loss=0.1074, lr=0.004850, batch_cost=0.6362, reader_cost=0.0018 | ETA 06:19:36 2020-11-04 05:25:02 [INFO] [TRAIN] epoch=120, iter=44300/80000, loss=0.0985, lr=0.004838, batch_cost=0.6411, reader_cost=0.0105 | ETA 06:21:27 2020-11-04 05:26:05 [INFO] [TRAIN] epoch=120, iter=44400/80000, loss=0.1188, lr=0.004825, batch_cost=0.6301, reader_cost=0.0007 | ETA 06:13:53 2020-11-04 05:27:09 [INFO] [TRAIN] epoch=120, iter=44500/80000, loss=0.1120, lr=0.004813, batch_cost=0.6361, reader_cost=0.0017 | ETA 06:16:22 2020-11-04 05:28:12 [INFO] [TRAIN] epoch=120, iter=44600/80000, loss=0.1068, lr=0.004801, batch_cost=0.6315, reader_cost=0.0015 | ETA 06:12:34 2020-11-04 05:29:17 [INFO] [TRAIN] epoch=121, iter=44700/80000, loss=0.1048, lr=0.004789, batch_cost=0.6455, reader_cost=0.0093 | ETA 06:19:45 2020-11-04 05:30:20 [INFO] [TRAIN] epoch=121, iter=44800/80000, loss=0.1281, lr=0.004777, batch_cost=0.6314, reader_cost=0.0005 | ETA 06:10:25 2020-11-04 05:31:22 [INFO] [TRAIN] epoch=121, iter=44900/80000, loss=0.1194, lr=0.004764, batch_cost=0.6248, reader_cost=0.0005 | ETA 06:05:31 2020-11-04 05:32:24 [INFO] [TRAIN] epoch=121, iter=45000/80000, loss=0.1105, lr=0.004752, batch_cost=0.6167, reader_cost=0.0002 | ETA 05:59:43 2020-11-04 05:33:27 [INFO] [TRAIN] epoch=122, iter=45100/80000, loss=0.1049, lr=0.004740, batch_cost=0.6308, reader_cost=0.0095 | ETA 06:06:53 2020-11-04 05:34:30 [INFO] [TRAIN] epoch=122, iter=45200/80000, loss=0.1196, lr=0.004728, batch_cost=0.6326, reader_cost=0.0003 | ETA 06:06:54 2020-11-04 05:35:34 [INFO] [TRAIN] epoch=122, iter=45300/80000, loss=0.1205, lr=0.004715, batch_cost=0.6329, reader_cost=0.0007 | ETA 06:06:00 2020-11-04 05:36:37 [INFO] [TRAIN] epoch=123, iter=45400/80000, loss=0.1091, lr=0.004703, batch_cost=0.6358, reader_cost=0.0089 | ETA 06:06:37 2020-11-04 05:37:39 [INFO] [TRAIN] epoch=123, iter=45500/80000, loss=0.1014, lr=0.004691, batch_cost=0.6221, reader_cost=0.0005 | ETA 05:57:40 2020-11-04 05:38:42 [INFO] [TRAIN] epoch=123, iter=45600/80000, loss=0.1075, lr=0.004679, batch_cost=0.6269, reader_cost=0.0002 | ETA 05:59:25 2020-11-04 05:39:45 [INFO] [TRAIN] epoch=123, iter=45700/80000, loss=0.1196, lr=0.004667, batch_cost=0.6297, reader_cost=0.0002 | ETA 05:59:59 2020-11-04 05:40:48 [INFO] [TRAIN] epoch=124, iter=45800/80000, loss=0.1157, lr=0.004654, batch_cost=0.6333, reader_cost=0.0082 | ETA 06:00:58 2020-11-04 05:41:51 [INFO] [TRAIN] epoch=124, iter=45900/80000, loss=0.1149, lr=0.004642, batch_cost=0.6219, reader_cost=0.0002 | ETA 05:53:28 2020-11-04 05:42:53 [INFO] [TRAIN] epoch=124, iter=46000/80000, loss=0.1161, lr=0.004630, batch_cost=0.6211, reader_cost=0.0002 | ETA 05:51:58 2020-11-04 05:43:55 [INFO] [TRAIN] epoch=124, iter=46100/80000, loss=0.1057, lr=0.004618, batch_cost=0.6198, reader_cost=0.0003 | ETA 05:50:11 2020-11-04 05:44:58 [INFO] [TRAIN] epoch=125, iter=46200/80000, loss=0.1036, lr=0.004605, batch_cost=0.6307, reader_cost=0.0088 | ETA 05:55:17 2020-11-04 05:46:00 [INFO] [TRAIN] epoch=125, iter=46300/80000, loss=0.1133, lr=0.004593, batch_cost=0.6278, reader_cost=0.0005 | ETA 05:52:36 2020-11-04 05:47:03 [INFO] [TRAIN] epoch=125, iter=46400/80000, loss=0.1146, lr=0.004581, batch_cost=0.6274, reader_cost=0.0005 | ETA 05:51:21 2020-11-04 05:48:06 [INFO] [TRAIN] epoch=125, iter=46500/80000, loss=0.0954, lr=0.004568, batch_cost=0.6234, reader_cost=0.0005 | ETA 05:48:03 2020-11-04 05:49:09 [INFO] [TRAIN] epoch=126, iter=46600/80000, loss=0.1079, lr=0.004556, batch_cost=0.6359, reader_cost=0.0085 | ETA 05:53:59 2020-11-04 05:50:12 [INFO] [TRAIN] epoch=126, iter=46700/80000, loss=0.1110, lr=0.004544, batch_cost=0.6284, reader_cost=0.0002 | ETA 05:48:47 2020-11-04 05:51:15 [INFO] [TRAIN] epoch=126, iter=46800/80000, loss=0.0959, lr=0.004532, batch_cost=0.6304, reader_cost=0.0005 | ETA 05:48:50 2020-11-04 05:52:18 [INFO] [TRAIN] epoch=127, iter=46900/80000, loss=0.0961, lr=0.004519, batch_cost=0.6334, reader_cost=0.0097 | ETA 05:49:27 2020-11-04 05:53:22 [INFO] [TRAIN] epoch=127, iter=47000/80000, loss=0.1277, lr=0.004507, batch_cost=0.6312, reader_cost=0.0005 | ETA 05:47:09 2020-11-04 05:54:25 [INFO] [TRAIN] epoch=127, iter=47100/80000, loss=0.1121, lr=0.004495, batch_cost=0.6309, reader_cost=0.0004 | ETA 05:45:55 2020-11-04 05:55:27 [INFO] [TRAIN] epoch=127, iter=47200/80000, loss=0.1097, lr=0.004482, batch_cost=0.6246, reader_cost=0.0007 | ETA 05:41:26 2020-11-04 05:56:30 [INFO] [TRAIN] epoch=128, iter=47300/80000, loss=0.1077, lr=0.004470, batch_cost=0.6310, reader_cost=0.0086 | ETA 05:43:52 2020-11-04 05:57:32 [INFO] [TRAIN] epoch=128, iter=47400/80000, loss=0.1137, lr=0.004458, batch_cost=0.6171, reader_cost=0.0002 | ETA 05:35:18 2020-11-04 05:58:34 [INFO] [TRAIN] epoch=128, iter=47500/80000, loss=0.1142, lr=0.004446, batch_cost=0.6177, reader_cost=0.0002 | ETA 05:34:34 2020-11-04 05:59:35 [INFO] [TRAIN] epoch=128, iter=47600/80000, loss=0.0974, lr=0.004433, batch_cost=0.6168, reader_cost=0.0002 | ETA 05:33:04 2020-11-04 06:00:38 [INFO] [TRAIN] epoch=129, iter=47700/80000, loss=0.1015, lr=0.004421, batch_cost=0.6284, reader_cost=0.0083 | ETA 05:38:16 2020-11-04 06:01:40 [INFO] [TRAIN] epoch=129, iter=47800/80000, loss=0.1015, lr=0.004409, batch_cost=0.6209, reader_cost=0.0002 | ETA 05:33:11 2020-11-04 06:02:42 [INFO] [TRAIN] epoch=129, iter=47900/80000, loss=0.1022, lr=0.004396, batch_cost=0.6199, reader_cost=0.0002 | ETA 05:31:37 2020-11-04 06:03:45 [INFO] [TRAIN] epoch=130, iter=48000/80000, loss=0.1113, lr=0.004384, batch_cost=0.6301, reader_cost=0.0093 | ETA 05:36:02 2020-11-04 06:03:48 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-04 06:08:02 [INFO] [EVAL] #Images=500 mIoU=0.7786 Acc=0.9572 Kappa=0.9443 2020-11-04 06:08:02 [INFO] [EVAL] Category IoU: [0.9702 0.7789 0.9282 0.5316 0.6107 0.6675 0.7111 0.7944 0.923 0.6086 0.9516 0.8181 0.5768 0.9547 0.8432 0.8856 0.7794 0.6764 0.7836] 2020-11-04 06:08:02 [INFO] [EVAL] Category Acc: [0.9772 0.9264 0.962 0.8158 0.8246 0.808 0.8095 0.8875 0.951 0.8246 0.9756 0.8645 0.8342 0.9753 0.9308 0.9577 0.8531 0.8554 0.8752] 2020-11-04 06:08:02 [INFO] [EVAL] The model with the best validation mIoU (0.7855) was saved at iter 40000. 2020-11-04 06:09:05 [INFO] [TRAIN] epoch=130, iter=48100/80000, loss=0.1060, lr=0.004372, batch_cost=0.6293, reader_cost=0.0002 | ETA 05:34:34 2020-11-04 06:10:08 [INFO] [TRAIN] epoch=130, iter=48200/80000, loss=0.1148, lr=0.004359, batch_cost=0.6282, reader_cost=0.0002 | ETA 05:32:57 2020-11-04 06:11:11 [INFO] [TRAIN] epoch=130, iter=48300/80000, loss=0.1084, lr=0.004347, batch_cost=0.6257, reader_cost=0.0003 | ETA 05:30:35 2020-11-04 06:12:14 [INFO] [TRAIN] epoch=131, iter=48400/80000, loss=0.1123, lr=0.004335, batch_cost=0.6347, reader_cost=0.0084 | ETA 05:34:16 2020-11-04 06:13:16 [INFO] [TRAIN] epoch=131, iter=48500/80000, loss=0.1137, lr=0.004322, batch_cost=0.6217, reader_cost=0.0008 | ETA 05:26:23 2020-11-04 06:14:19 [INFO] [TRAIN] epoch=131, iter=48600/80000, loss=0.1124, lr=0.004310, batch_cost=0.6220, reader_cost=0.0002 | ETA 05:25:31 2020-11-04 06:15:21 [INFO] [TRAIN] epoch=131, iter=48700/80000, loss=0.1069, lr=0.004298, batch_cost=0.6208, reader_cost=0.0007 | ETA 05:23:52 2020-11-04 06:16:24 [INFO] [TRAIN] epoch=132, iter=48800/80000, loss=0.0961, lr=0.004285, batch_cost=0.6334, reader_cost=0.0098 | ETA 05:29:22 2020-11-04 06:17:27 [INFO] [TRAIN] epoch=132, iter=48900/80000, loss=0.1133, lr=0.004273, batch_cost=0.6247, reader_cost=0.0007 | ETA 05:23:48 2020-11-04 06:18:29 [INFO] [TRAIN] epoch=132, iter=49000/80000, loss=0.1136, lr=0.004260, batch_cost=0.6201, reader_cost=0.0003 | ETA 05:20:23 2020-11-04 06:19:31 [INFO] [TRAIN] epoch=132, iter=49100/80000, loss=0.1025, lr=0.004248, batch_cost=0.6213, reader_cost=0.0009 | ETA 05:19:58 2020-11-04 06:20:34 [INFO] [TRAIN] epoch=133, iter=49200/80000, loss=0.1078, lr=0.004236, batch_cost=0.6322, reader_cost=0.0082 | ETA 05:24:30 2020-11-04 06:21:36 [INFO] [TRAIN] epoch=133, iter=49300/80000, loss=0.1180, lr=0.004223, batch_cost=0.6247, reader_cost=0.0005 | ETA 05:19:38 2020-11-04 06:22:39 [INFO] [TRAIN] epoch=133, iter=49400/80000, loss=0.0987, lr=0.004211, batch_cost=0.6215, reader_cost=0.0002 | ETA 05:16:56 2020-11-04 06:23:42 [INFO] [TRAIN] epoch=134, iter=49500/80000, loss=0.1008, lr=0.004199, batch_cost=0.6329, reader_cost=0.0085 | ETA 05:21:44 2020-11-04 06:24:45 [INFO] [TRAIN] epoch=134, iter=49600/80000, loss=0.1054, lr=0.004186, batch_cost=0.6291, reader_cost=0.0005 | ETA 05:18:45 2020-11-04 06:25:48 [INFO] [TRAIN] epoch=134, iter=49700/80000, loss=0.1239, lr=0.004174, batch_cost=0.6293, reader_cost=0.0003 | ETA 05:17:48 2020-11-04 06:26:51 [INFO] [TRAIN] epoch=134, iter=49800/80000, loss=0.1063, lr=0.004161, batch_cost=0.6295, reader_cost=0.0008 | ETA 05:16:51 2020-11-04 06:27:54 [INFO] [TRAIN] epoch=135, iter=49900/80000, loss=0.1049, lr=0.004149, batch_cost=0.6316, reader_cost=0.0097 | ETA 05:16:51 2020-11-04 06:28:56 [INFO] [TRAIN] epoch=135, iter=50000/80000, loss=0.1088, lr=0.004137, batch_cost=0.6210, reader_cost=0.0004 | ETA 05:10:29 2020-11-04 06:29:58 [INFO] [TRAIN] epoch=135, iter=50100/80000, loss=0.1149, lr=0.004124, batch_cost=0.6198, reader_cost=0.0002 | ETA 05:08:52 2020-11-04 06:31:00 [INFO] [TRAIN] epoch=135, iter=50200/80000, loss=0.0926, lr=0.004112, batch_cost=0.6181, reader_cost=0.0002 | ETA 05:06:59 2020-11-04 06:32:03 [INFO] [TRAIN] epoch=136, iter=50300/80000, loss=0.1032, lr=0.004099, batch_cost=0.6313, reader_cost=0.0089 | ETA 05:12:31 2020-11-04 06:33:05 [INFO] [TRAIN] epoch=136, iter=50400/80000, loss=0.1191, lr=0.004087, batch_cost=0.6196, reader_cost=0.0002 | ETA 05:05:40 2020-11-04 06:34:07 [INFO] [TRAIN] epoch=136, iter=50500/80000, loss=0.1062, lr=0.004074, batch_cost=0.6216, reader_cost=0.0003 | ETA 05:05:37 2020-11-04 06:35:10 [INFO] [TRAIN] epoch=137, iter=50600/80000, loss=0.1025, lr=0.004062, batch_cost=0.6318, reader_cost=0.0085 | ETA 05:09:36 2020-11-04 06:36:12 [INFO] [TRAIN] epoch=137, iter=50700/80000, loss=0.1164, lr=0.004050, batch_cost=0.6217, reader_cost=0.0007 | ETA 05:03:36 2020-11-04 06:37:14 [INFO] [TRAIN] epoch=137, iter=50800/80000, loss=0.1078, lr=0.004037, batch_cost=0.6219, reader_cost=0.0006 | ETA 05:02:40 2020-11-04 06:38:17 [INFO] [TRAIN] epoch=137, iter=50900/80000, loss=0.1075, lr=0.004025, batch_cost=0.6241, reader_cost=0.0002 | ETA 05:02:41 2020-11-04 06:39:20 [INFO] [TRAIN] epoch=138, iter=51000/80000, loss=0.1012, lr=0.004012, batch_cost=0.6338, reader_cost=0.0103 | ETA 05:06:20 2020-11-04 06:40:22 [INFO] [TRAIN] epoch=138, iter=51100/80000, loss=0.1163, lr=0.004000, batch_cost=0.6172, reader_cost=0.0004 | ETA 04:57:16 2020-11-04 06:41:24 [INFO] [TRAIN] epoch=138, iter=51200/80000, loss=0.1105, lr=0.003987, batch_cost=0.6194, reader_cost=0.0006 | ETA 04:57:17 2020-11-04 06:42:26 [INFO] [TRAIN] epoch=138, iter=51300/80000, loss=0.0992, lr=0.003975, batch_cost=0.6185, reader_cost=0.0004 | ETA 04:55:52 2020-11-04 06:43:28 [INFO] [TRAIN] epoch=139, iter=51400/80000, loss=0.0990, lr=0.003962, batch_cost=0.6254, reader_cost=0.0086 | ETA 04:58:06 2020-11-04 06:44:30 [INFO] [TRAIN] epoch=139, iter=51500/80000, loss=0.1166, lr=0.003950, batch_cost=0.6179, reader_cost=0.0004 | ETA 04:53:30 2020-11-04 06:45:32 [INFO] [TRAIN] epoch=139, iter=51600/80000, loss=0.1100, lr=0.003937, batch_cost=0.6196, reader_cost=0.0003 | ETA 04:53:15 2020-11-04 06:46:34 [INFO] [TRAIN] epoch=139, iter=51700/80000, loss=0.0983, lr=0.003925, batch_cost=0.6190, reader_cost=0.0005 | ETA 04:51:58 2020-11-04 06:47:37 [INFO] [TRAIN] epoch=140, iter=51800/80000, loss=0.1053, lr=0.003913, batch_cost=0.6287, reader_cost=0.0083 | ETA 04:55:28 2020-11-04 06:48:39 [INFO] [TRAIN] epoch=140, iter=51900/80000, loss=0.1158, lr=0.003900, batch_cost=0.6219, reader_cost=0.0002 | ETA 04:51:15 2020-11-04 06:49:41 [INFO] [TRAIN] epoch=140, iter=52000/80000, loss=0.1126, lr=0.003888, batch_cost=0.6224, reader_cost=0.0002 | ETA 04:50:28 2020-11-04 06:50:44 [INFO] [TRAIN] epoch=141, iter=52100/80000, loss=0.1016, lr=0.003875, batch_cost=0.6300, reader_cost=0.0082 | ETA 04:52:55 2020-11-04 06:51:46 [INFO] [TRAIN] epoch=141, iter=52200/80000, loss=0.1030, lr=0.003863, batch_cost=0.6176, reader_cost=0.0002 | ETA 04:46:09 2020-11-04 06:52:48 [INFO] [TRAIN] epoch=141, iter=52300/80000, loss=0.1136, lr=0.003850, batch_cost=0.6174, reader_cost=0.0002 | ETA 04:45:03 2020-11-04 06:53:50 [INFO] [TRAIN] epoch=141, iter=52400/80000, loss=0.0969, lr=0.003838, batch_cost=0.6215, reader_cost=0.0002 | ETA 04:45:52 2020-11-04 06:54:53 [INFO] [TRAIN] epoch=142, iter=52500/80000, loss=0.1083, lr=0.003825, batch_cost=0.6294, reader_cost=0.0091 | ETA 04:48:27 2020-11-04 06:55:55 [INFO] [TRAIN] epoch=142, iter=52600/80000, loss=0.1041, lr=0.003812, batch_cost=0.6223, reader_cost=0.0004 | ETA 04:44:09 2020-11-04 06:56:57 [INFO] [TRAIN] epoch=142, iter=52700/80000, loss=0.1091, lr=0.003800, batch_cost=0.6226, reader_cost=0.0006 | ETA 04:43:17 2020-11-04 06:57:59 [INFO] [TRAIN] epoch=142, iter=52800/80000, loss=0.0893, lr=0.003787, batch_cost=0.6187, reader_cost=0.0005 | ETA 04:40:28 2020-11-04 06:59:02 [INFO] [TRAIN] epoch=143, iter=52900/80000, loss=0.1003, lr=0.003775, batch_cost=0.6272, reader_cost=0.0088 | ETA 04:43:18 2020-11-04 07:00:04 [INFO] [TRAIN] epoch=143, iter=53000/80000, loss=0.1238, lr=0.003762, batch_cost=0.6172, reader_cost=0.0002 | ETA 04:37:44 2020-11-04 07:01:06 [INFO] [TRAIN] epoch=143, iter=53100/80000, loss=0.1040, lr=0.003750, batch_cost=0.6189, reader_cost=0.0003 | ETA 04:37:28 2020-11-04 07:02:09 [INFO] [TRAIN] epoch=144, iter=53200/80000, loss=0.1045, lr=0.003737, batch_cost=0.6323, reader_cost=0.0084 | ETA 04:42:26 2020-11-04 07:03:11 [INFO] [TRAIN] epoch=144, iter=53300/80000, loss=0.1124, lr=0.003725, batch_cost=0.6199, reader_cost=0.0006 | ETA 04:35:50 2020-11-04 07:04:13 [INFO] [TRAIN] epoch=144, iter=53400/80000, loss=0.1031, lr=0.003712, batch_cost=0.6260, reader_cost=0.0004 | ETA 04:37:31 2020-11-04 07:05:16 [INFO] [TRAIN] epoch=144, iter=53500/80000, loss=0.0957, lr=0.003700, batch_cost=0.6306, reader_cost=0.0003 | ETA 04:38:31 2020-11-04 07:06:19 [INFO] [TRAIN] epoch=145, iter=53600/80000, loss=0.0985, lr=0.003687, batch_cost=0.6292, reader_cost=0.0100 | ETA 04:36:51 2020-11-04 07:07:22 [INFO] [TRAIN] epoch=145, iter=53700/80000, loss=0.1081, lr=0.003674, batch_cost=0.6258, reader_cost=0.0003 | ETA 04:34:17 2020-11-04 07:08:25 [INFO] [TRAIN] epoch=145, iter=53800/80000, loss=0.1132, lr=0.003662, batch_cost=0.6335, reader_cost=0.0009 | ETA 04:36:36 2020-11-04 07:09:29 [INFO] [TRAIN] epoch=145, iter=53900/80000, loss=0.0940, lr=0.003649, batch_cost=0.6340, reader_cost=0.0008 | ETA 04:35:48 2020-11-04 07:10:32 [INFO] [TRAIN] epoch=146, iter=54000/80000, loss=0.0964, lr=0.003637, batch_cost=0.6350, reader_cost=0.0085 | ETA 04:35:10 2020-11-04 07:11:35 [INFO] [TRAIN] epoch=146, iter=54100/80000, loss=0.1171, lr=0.003624, batch_cost=0.6270, reader_cost=0.0006 | ETA 04:30:38 2020-11-04 07:12:38 [INFO] [TRAIN] epoch=146, iter=54200/80000, loss=0.1232, lr=0.003612, batch_cost=0.6279, reader_cost=0.0006 | ETA 04:29:59 2020-11-04 07:13:40 [INFO] [TRAIN] epoch=146, iter=54300/80000, loss=0.0947, lr=0.003599, batch_cost=0.6251, reader_cost=0.0007 | ETA 04:27:44 2020-11-04 07:14:44 [INFO] [TRAIN] epoch=147, iter=54400/80000, loss=0.1126, lr=0.003586, batch_cost=0.6355, reader_cost=0.0084 | ETA 04:31:09 2020-11-04 07:15:47 [INFO] [TRAIN] epoch=147, iter=54500/80000, loss=0.1125, lr=0.003574, batch_cost=0.6364, reader_cost=0.0007 | ETA 04:30:29 2020-11-04 07:16:51 [INFO] [TRAIN] epoch=147, iter=54600/80000, loss=0.1071, lr=0.003561, batch_cost=0.6344, reader_cost=0.0003 | ETA 04:28:35 2020-11-04 07:17:55 [INFO] [TRAIN] epoch=148, iter=54700/80000, loss=0.0987, lr=0.003548, batch_cost=0.6374, reader_cost=0.0090 | ETA 04:28:47 2020-11-04 07:18:57 [INFO] [TRAIN] epoch=148, iter=54800/80000, loss=0.1026, lr=0.003536, batch_cost=0.6224, reader_cost=0.0002 | ETA 04:21:23 2020-11-04 07:19:59 [INFO] [TRAIN] epoch=148, iter=54900/80000, loss=0.0993, lr=0.003523, batch_cost=0.6201, reader_cost=0.0002 | ETA 04:19:25 2020-11-04 07:21:01 [INFO] [TRAIN] epoch=148, iter=55000/80000, loss=0.0976, lr=0.003511, batch_cost=0.6220, reader_cost=0.0005 | ETA 04:19:09 2020-11-04 07:22:04 [INFO] [TRAIN] epoch=149, iter=55100/80000, loss=0.1057, lr=0.003498, batch_cost=0.6282, reader_cost=0.0096 | ETA 04:20:41 2020-11-04 07:23:06 [INFO] [TRAIN] epoch=149, iter=55200/80000, loss=0.1010, lr=0.003485, batch_cost=0.6256, reader_cost=0.0003 | ETA 04:18:34 2020-11-04 07:24:08 [INFO] [TRAIN] epoch=149, iter=55300/80000, loss=0.1050, lr=0.003473, batch_cost=0.6198, reader_cost=0.0002 | ETA 04:15:08 2020-11-04 07:25:11 [INFO] [TRAIN] epoch=149, iter=55400/80000, loss=0.1248, lr=0.003460, batch_cost=0.6251, reader_cost=0.0003 | ETA 04:16:17 2020-11-04 07:26:14 [INFO] [TRAIN] epoch=150, iter=55500/80000, loss=0.1009, lr=0.003447, batch_cost=0.6350, reader_cost=0.0088 | ETA 04:19:18 2020-11-04 07:27:17 [INFO] [TRAIN] epoch=150, iter=55600/80000, loss=0.1182, lr=0.003435, batch_cost=0.6236, reader_cost=0.0005 | ETA 04:13:34 2020-11-04 07:28:19 [INFO] [TRAIN] epoch=150, iter=55700/80000, loss=0.1079, lr=0.003422, batch_cost=0.6255, reader_cost=0.0003 | ETA 04:13:20 2020-11-04 07:29:22 [INFO] [TRAIN] epoch=150, iter=55800/80000, loss=0.0938, lr=0.003409, batch_cost=0.6258, reader_cost=0.0005 | ETA 04:12:24 2020-11-04 07:30:25 [INFO] [TRAIN] epoch=151, iter=55900/80000, loss=0.1029, lr=0.003397, batch_cost=0.6306, reader_cost=0.0089 | ETA 04:13:18 2020-11-04 07:31:27 [INFO] [TRAIN] epoch=151, iter=56000/80000, loss=0.1126, lr=0.003384, batch_cost=0.6230, reader_cost=0.0002 | ETA 04:09:12 2020-11-04 07:31:30 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-04 07:35:45 [INFO] [EVAL] #Images=500 mIoU=0.7954 Acc=0.9615 Kappa=0.9500 2020-11-04 07:35:45 [INFO] [EVAL] Category IoU: [0.9798 0.8451 0.9283 0.5845 0.6361 0.6693 0.7189 0.7981 0.9241 0.6249 0.9495 0.8352 0.6434 0.9545 0.8153 0.8904 0.8459 0.6846 0.7839] 2020-11-04 07:35:45 [INFO] [EVAL] Category Acc: [0.992 0.9058 0.9619 0.8347 0.8088 0.8051 0.8365 0.8915 0.9586 0.7509 0.9679 0.8972 0.8056 0.9754 0.8934 0.9333 0.9434 0.8656 0.8583] 2020-11-04 07:35:46 [INFO] [EVAL] The model with the best validation mIoU (0.7954) was saved at iter 56000. 2020-11-04 07:36:48 [INFO] [TRAIN] epoch=151, iter=56100/80000, loss=0.1015, lr=0.003371, batch_cost=0.6209, reader_cost=0.0002 | ETA 04:07:20 2020-11-04 07:37:52 [INFO] [TRAIN] epoch=152, iter=56200/80000, loss=0.0929, lr=0.003359, batch_cost=0.6349, reader_cost=0.0090 | ETA 04:11:50 2020-11-04 07:38:54 [INFO] [TRAIN] epoch=152, iter=56300/80000, loss=0.1089, lr=0.003346, batch_cost=0.6240, reader_cost=0.0003 | ETA 04:06:28 2020-11-04 07:39:56 [INFO] [TRAIN] epoch=152, iter=56400/80000, loss=0.1110, lr=0.003333, batch_cost=0.6192, reader_cost=0.0005 | ETA 04:03:33 2020-11-04 07:40:58 [INFO] [TRAIN] epoch=152, iter=56500/80000, loss=0.1024, lr=0.003320, batch_cost=0.6167, reader_cost=0.0006 | ETA 04:01:31 2020-11-04 07:42:01 [INFO] [TRAIN] epoch=153, iter=56600/80000, loss=0.0992, lr=0.003308, batch_cost=0.6289, reader_cost=0.0106 | ETA 04:05:15 2020-11-04 07:43:03 [INFO] [TRAIN] epoch=153, iter=56700/80000, loss=0.1207, lr=0.003295, batch_cost=0.6273, reader_cost=0.0003 | ETA 04:03:34 2020-11-04 07:44:06 [INFO] [TRAIN] epoch=153, iter=56800/80000, loss=0.1102, lr=0.003282, batch_cost=0.6261, reader_cost=0.0003 | ETA 04:02:05 2020-11-04 07:45:08 [INFO] [TRAIN] epoch=153, iter=56900/80000, loss=0.0840, lr=0.003270, batch_cost=0.6252, reader_cost=0.0004 | ETA 04:00:41 2020-11-04 07:46:11 [INFO] [TRAIN] epoch=154, iter=57000/80000, loss=0.0949, lr=0.003257, batch_cost=0.6306, reader_cost=0.0090 | ETA 04:01:44 2020-11-04 07:47:14 [INFO] [TRAIN] epoch=154, iter=57100/80000, loss=0.1066, lr=0.003244, batch_cost=0.6220, reader_cost=0.0005 | ETA 03:57:23 2020-11-04 07:48:16 [INFO] [TRAIN] epoch=154, iter=57200/80000, loss=0.0986, lr=0.003231, batch_cost=0.6235, reader_cost=0.0008 | ETA 03:56:56 2020-11-04 07:49:20 [INFO] [TRAIN] epoch=155, iter=57300/80000, loss=0.1036, lr=0.003219, batch_cost=0.6359, reader_cost=0.0086 | ETA 04:00:34 2020-11-04 07:50:22 [INFO] [TRAIN] epoch=155, iter=57400/80000, loss=0.1034, lr=0.003206, batch_cost=0.6235, reader_cost=0.0004 | ETA 03:54:51 2020-11-04 07:51:25 [INFO] [TRAIN] epoch=155, iter=57500/80000, loss=0.1091, lr=0.003193, batch_cost=0.6263, reader_cost=0.0008 | ETA 03:54:51 2020-11-04 07:52:27 [INFO] [TRAIN] epoch=155, iter=57600/80000, loss=0.1023, lr=0.003180, batch_cost=0.6275, reader_cost=0.0004 | ETA 03:54:15 2020-11-04 07:53:31 [INFO] [TRAIN] epoch=156, iter=57700/80000, loss=0.1065, lr=0.003167, batch_cost=0.6327, reader_cost=0.0095 | ETA 03:55:08 2020-11-04 07:54:33 [INFO] [TRAIN] epoch=156, iter=57800/80000, loss=0.0947, lr=0.003155, batch_cost=0.6285, reader_cost=0.0005 | ETA 03:52:32 2020-11-04 07:55:36 [INFO] [TRAIN] epoch=156, iter=57900/80000, loss=0.1156, lr=0.003142, batch_cost=0.6255, reader_cost=0.0003 | ETA 03:50:23 2020-11-04 07:56:39 [INFO] [TRAIN] epoch=156, iter=58000/80000, loss=0.0906, lr=0.003129, batch_cost=0.6295, reader_cost=0.0005 | ETA 03:50:49 2020-11-04 07:57:42 [INFO] [TRAIN] epoch=157, iter=58100/80000, loss=0.0906, lr=0.003116, batch_cost=0.6353, reader_cost=0.0089 | ETA 03:51:54 2020-11-04 07:58:45 [INFO] [TRAIN] epoch=157, iter=58200/80000, loss=0.1137, lr=0.003103, batch_cost=0.6298, reader_cost=0.0008 | ETA 03:48:48 2020-11-04 07:59:48 [INFO] [TRAIN] epoch=157, iter=58300/80000, loss=0.0996, lr=0.003091, batch_cost=0.6225, reader_cost=0.0005 | ETA 03:45:09 2020-11-04 08:00:50 [INFO] [TRAIN] epoch=157, iter=58400/80000, loss=0.0892, lr=0.003078, batch_cost=0.6228, reader_cost=0.0007 | ETA 03:44:12 2020-11-04 08:01:53 [INFO] [TRAIN] epoch=158, iter=58500/80000, loss=0.0998, lr=0.003065, batch_cost=0.6328, reader_cost=0.0093 | ETA 03:46:44 2020-11-04 08:02:56 [INFO] [TRAIN] epoch=158, iter=58600/80000, loss=0.1057, lr=0.003052, batch_cost=0.6246, reader_cost=0.0003 | ETA 03:42:47 2020-11-04 08:03:58 [INFO] [TRAIN] epoch=158, iter=58700/80000, loss=0.0992, lr=0.003039, batch_cost=0.6193, reader_cost=0.0002 | ETA 03:39:50 2020-11-04 08:05:01 [INFO] [TRAIN] epoch=159, iter=58800/80000, loss=0.0905, lr=0.003026, batch_cost=0.6310, reader_cost=0.0088 | ETA 03:42:56 2020-11-04 08:06:03 [INFO] [TRAIN] epoch=159, iter=58900/80000, loss=0.0999, lr=0.003014, batch_cost=0.6198, reader_cost=0.0004 | ETA 03:37:57 2020-11-04 08:07:05 [INFO] [TRAIN] epoch=159, iter=59000/80000, loss=0.1022, lr=0.003001, batch_cost=0.6186, reader_cost=0.0005 | ETA 03:36:30 2020-11-04 08:08:07 [INFO] [TRAIN] epoch=159, iter=59100/80000, loss=0.0960, lr=0.002988, batch_cost=0.6231, reader_cost=0.0009 | ETA 03:37:02 2020-11-04 08:09:10 [INFO] [TRAIN] epoch=160, iter=59200/80000, loss=0.0951, lr=0.002975, batch_cost=0.6332, reader_cost=0.0084 | ETA 03:39:29 2020-11-04 08:10:13 [INFO] [TRAIN] epoch=160, iter=59300/80000, loss=0.1030, lr=0.002962, batch_cost=0.6239, reader_cost=0.0007 | ETA 03:35:15 2020-11-04 08:11:15 [INFO] [TRAIN] epoch=160, iter=59400/80000, loss=0.1101, lr=0.002949, batch_cost=0.6194, reader_cost=0.0009 | ETA 03:32:39 2020-11-04 08:12:17 [INFO] [TRAIN] epoch=160, iter=59500/80000, loss=0.0979, lr=0.002936, batch_cost=0.6234, reader_cost=0.0005 | ETA 03:32:58 2020-11-04 08:13:20 [INFO] [TRAIN] epoch=161, iter=59600/80000, loss=0.0995, lr=0.002924, batch_cost=0.6312, reader_cost=0.0089 | ETA 03:34:37 2020-11-04 08:14:23 [INFO] [TRAIN] epoch=161, iter=59700/80000, loss=0.1095, lr=0.002911, batch_cost=0.6275, reader_cost=0.0006 | ETA 03:32:18 2020-11-04 08:15:25 [INFO] [TRAIN] epoch=161, iter=59800/80000, loss=0.0985, lr=0.002898, batch_cost=0.6264, reader_cost=0.0003 | ETA 03:30:53 2020-11-04 08:16:29 [INFO] [TRAIN] epoch=162, iter=59900/80000, loss=0.0997, lr=0.002885, batch_cost=0.6359, reader_cost=0.0089 | ETA 03:33:01 2020-11-04 08:17:32 [INFO] [TRAIN] epoch=162, iter=60000/80000, loss=0.1068, lr=0.002872, batch_cost=0.6288, reader_cost=0.0002 | ETA 03:29:35 2020-11-04 08:18:35 [INFO] [TRAIN] epoch=162, iter=60100/80000, loss=0.0998, lr=0.002859, batch_cost=0.6341, reader_cost=0.0005 | ETA 03:30:19 2020-11-04 08:19:38 [INFO] [TRAIN] epoch=162, iter=60200/80000, loss=0.1005, lr=0.002846, batch_cost=0.6317, reader_cost=0.0010 | ETA 03:28:27 2020-11-04 08:20:42 [INFO] [TRAIN] epoch=163, iter=60300/80000, loss=0.1072, lr=0.002833, batch_cost=0.6377, reader_cost=0.0092 | ETA 03:29:22 2020-11-04 08:21:45 [INFO] [TRAIN] epoch=163, iter=60400/80000, loss=0.1020, lr=0.002820, batch_cost=0.6320, reader_cost=0.0005 | ETA 03:26:27 2020-11-04 08:22:49 [INFO] [TRAIN] epoch=163, iter=60500/80000, loss=0.1046, lr=0.002807, batch_cost=0.6353, reader_cost=0.0008 | ETA 03:26:28 2020-11-04 08:23:52 [INFO] [TRAIN] epoch=163, iter=60600/80000, loss=0.0967, lr=0.002794, batch_cost=0.6289, reader_cost=0.0005 | ETA 03:23:20 2020-11-04 08:24:55 [INFO] [TRAIN] epoch=164, iter=60700/80000, loss=0.0956, lr=0.002781, batch_cost=0.6343, reader_cost=0.0097 | ETA 03:24:02 2020-11-04 08:25:58 [INFO] [TRAIN] epoch=164, iter=60800/80000, loss=0.1085, lr=0.002768, batch_cost=0.6313, reader_cost=0.0004 | ETA 03:22:00 2020-11-04 08:27:02 [INFO] [TRAIN] epoch=164, iter=60900/80000, loss=0.1041, lr=0.002755, batch_cost=0.6325, reader_cost=0.0008 | ETA 03:21:20 2020-11-04 08:28:05 [INFO] [TRAIN] epoch=164, iter=61000/80000, loss=0.0975, lr=0.002742, batch_cost=0.6319, reader_cost=0.0005 | ETA 03:20:06 2020-11-04 08:29:08 [INFO] [TRAIN] epoch=165, iter=61100/80000, loss=0.1052, lr=0.002729, batch_cost=0.6286, reader_cost=0.0085 | ETA 03:18:00 2020-11-04 08:30:10 [INFO] [TRAIN] epoch=165, iter=61200/80000, loss=0.1041, lr=0.002716, batch_cost=0.6220, reader_cost=0.0004 | ETA 03:14:52 2020-11-04 08:31:12 [INFO] [TRAIN] epoch=165, iter=61300/80000, loss=0.1092, lr=0.002703, batch_cost=0.6216, reader_cost=0.0008 | ETA 03:13:44 2020-11-04 08:32:15 [INFO] [TRAIN] epoch=166, iter=61400/80000, loss=0.1029, lr=0.002690, batch_cost=0.6304, reader_cost=0.0085 | ETA 03:15:24 2020-11-04 08:33:18 [INFO] [TRAIN] epoch=166, iter=61500/80000, loss=0.1049, lr=0.002677, batch_cost=0.6264, reader_cost=0.0003 | ETA 03:13:09 2020-11-04 08:34:21 [INFO] [TRAIN] epoch=166, iter=61600/80000, loss=0.1139, lr=0.002664, batch_cost=0.6276, reader_cost=0.0007 | ETA 03:12:27 2020-11-04 08:35:23 [INFO] [TRAIN] epoch=166, iter=61700/80000, loss=0.0961, lr=0.002651, batch_cost=0.6250, reader_cost=0.0005 | ETA 03:10:36 2020-11-04 08:36:27 [INFO] [TRAIN] epoch=167, iter=61800/80000, loss=0.0968, lr=0.002638, batch_cost=0.6348, reader_cost=0.0094 | ETA 03:12:32 2020-11-04 08:37:29 [INFO] [TRAIN] epoch=167, iter=61900/80000, loss=0.0929, lr=0.002625, batch_cost=0.6229, reader_cost=0.0003 | ETA 03:07:54 2020-11-04 08:38:31 [INFO] [TRAIN] epoch=167, iter=62000/80000, loss=0.1036, lr=0.002612, batch_cost=0.6178, reader_cost=0.0002 | ETA 03:05:20 2020-11-04 08:39:33 [INFO] [TRAIN] epoch=167, iter=62100/80000, loss=0.0966, lr=0.002599, batch_cost=0.6192, reader_cost=0.0003 | ETA 03:04:43 2020-11-04 08:40:35 [INFO] [TRAIN] epoch=168, iter=62200/80000, loss=0.0871, lr=0.002586, batch_cost=0.6279, reader_cost=0.0091 | ETA 03:06:15 2020-11-04 08:41:38 [INFO] [TRAIN] epoch=168, iter=62300/80000, loss=0.0960, lr=0.002573, batch_cost=0.6247, reader_cost=0.0001 | ETA 03:04:16 2020-11-04 08:42:40 [INFO] [TRAIN] epoch=168, iter=62400/80000, loss=0.0986, lr=0.002560, batch_cost=0.6244, reader_cost=0.0002 | ETA 03:03:10 2020-11-04 08:43:44 [INFO] [TRAIN] epoch=169, iter=62500/80000, loss=0.1015, lr=0.002547, batch_cost=0.6343, reader_cost=0.0085 | ETA 03:05:00 2020-11-04 08:44:46 [INFO] [TRAIN] epoch=169, iter=62600/80000, loss=0.1047, lr=0.002534, batch_cost=0.6217, reader_cost=0.0002 | ETA 03:00:17 2020-11-04 08:45:48 [INFO] [TRAIN] epoch=169, iter=62700/80000, loss=0.1001, lr=0.002520, batch_cost=0.6266, reader_cost=0.0003 | ETA 03:00:39 2020-11-04 08:46:51 [INFO] [TRAIN] epoch=169, iter=62800/80000, loss=0.0973, lr=0.002507, batch_cost=0.6261, reader_cost=0.0006 | ETA 02:59:29 2020-11-04 08:47:55 [INFO] [TRAIN] epoch=170, iter=62900/80000, loss=0.0895, lr=0.002494, batch_cost=0.6363, reader_cost=0.0108 | ETA 03:01:20 2020-11-04 08:48:58 [INFO] [TRAIN] epoch=170, iter=63000/80000, loss=0.0991, lr=0.002481, batch_cost=0.6285, reader_cost=0.0005 | ETA 02:58:04 2020-11-04 08:50:00 [INFO] [TRAIN] epoch=170, iter=63100/80000, loss=0.0990, lr=0.002468, batch_cost=0.6255, reader_cost=0.0004 | ETA 02:56:10 2020-11-04 08:51:03 [INFO] [TRAIN] epoch=170, iter=63200/80000, loss=0.0928, lr=0.002455, batch_cost=0.6276, reader_cost=0.0003 | ETA 02:55:43 2020-11-04 08:52:06 [INFO] [TRAIN] epoch=171, iter=63300/80000, loss=0.0946, lr=0.002442, batch_cost=0.6311, reader_cost=0.0081 | ETA 02:55:39 2020-11-04 08:53:09 [INFO] [TRAIN] epoch=171, iter=63400/80000, loss=0.1103, lr=0.002429, batch_cost=0.6250, reader_cost=0.0002 | ETA 02:52:55 2020-11-04 08:54:11 [INFO] [TRAIN] epoch=171, iter=63500/80000, loss=0.1006, lr=0.002415, batch_cost=0.6282, reader_cost=0.0002 | ETA 02:52:45 2020-11-04 08:55:14 [INFO] [TRAIN] epoch=171, iter=63600/80000, loss=0.0911, lr=0.002402, batch_cost=0.6274, reader_cost=0.0010 | ETA 02:51:28 2020-11-04 08:56:17 [INFO] [TRAIN] epoch=172, iter=63700/80000, loss=0.0973, lr=0.002389, batch_cost=0.6320, reader_cost=0.0092 | ETA 02:51:41 2020-11-04 08:57:20 [INFO] [TRAIN] epoch=172, iter=63800/80000, loss=0.1030, lr=0.002376, batch_cost=0.6286, reader_cost=0.0008 | ETA 02:49:43 2020-11-04 08:58:23 [INFO] [TRAIN] epoch=172, iter=63900/80000, loss=0.1002, lr=0.002363, batch_cost=0.6250, reader_cost=0.0002 | ETA 02:47:41 2020-11-04 08:59:27 [INFO] [TRAIN] epoch=173, iter=64000/80000, loss=0.0905, lr=0.002349, batch_cost=0.6405, reader_cost=0.0084 | ETA 02:50:47 2020-11-04 08:59:29 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-04 09:03:41 [INFO] [EVAL] #Images=500 mIoU=0.8005 Acc=0.9637 Kappa=0.9529 2020-11-04 09:03:41 [INFO] [EVAL] Category IoU: [0.9833 0.8642 0.9291 0.5529 0.6287 0.6755 0.7191 0.8052 0.9275 0.6536 0.9516 0.8392 0.659 0.9552 0.8284 0.9152 0.847 0.6872 0.7884] 2020-11-04 09:03:41 [INFO] [EVAL] Category Acc: [0.9929 0.9223 0.96 0.8582 0.8405 0.8224 0.8231 0.904 0.956 0.8182 0.9707 0.9012 0.7918 0.9731 0.9276 0.9731 0.9358 0.8205 0.8644] 2020-11-04 09:03:42 [INFO] [EVAL] The model with the best validation mIoU (0.8005) was saved at iter 64000. 2020-11-04 09:04:46 [INFO] [TRAIN] epoch=173, iter=64100/80000, loss=0.1107, lr=0.002336, batch_cost=0.6379, reader_cost=0.0003 | ETA 02:49:02 2020-11-04 09:05:49 [INFO] [TRAIN] epoch=173, iter=64200/80000, loss=0.1002, lr=0.002323, batch_cost=0.6303, reader_cost=0.0007 | ETA 02:45:58 2020-11-04 09:06:52 [INFO] [TRAIN] epoch=173, iter=64300/80000, loss=0.1003, lr=0.002310, batch_cost=0.6272, reader_cost=0.0008 | ETA 02:44:06 2020-11-04 09:07:54 [INFO] [TRAIN] epoch=174, iter=64400/80000, loss=0.0888, lr=0.002296, batch_cost=0.6278, reader_cost=0.0095 | ETA 02:43:13 2020-11-04 09:08:56 [INFO] [TRAIN] epoch=174, iter=64500/80000, loss=0.0946, lr=0.002283, batch_cost=0.6176, reader_cost=0.0002 | ETA 02:39:33 2020-11-04 09:09:58 [INFO] [TRAIN] epoch=174, iter=64600/80000, loss=0.1004, lr=0.002270, batch_cost=0.6196, reader_cost=0.0005 | ETA 02:39:01 2020-11-04 09:11:00 [INFO] [TRAIN] epoch=174, iter=64700/80000, loss=0.1060, lr=0.002257, batch_cost=0.6158, reader_cost=0.0004 | ETA 02:37:02 2020-11-04 09:12:02 [INFO] [TRAIN] epoch=175, iter=64800/80000, loss=0.0964, lr=0.002243, batch_cost=0.6266, reader_cost=0.0093 | ETA 02:38:44 2020-11-04 09:13:04 [INFO] [TRAIN] epoch=175, iter=64900/80000, loss=0.0977, lr=0.002230, batch_cost=0.6190, reader_cost=0.0006 | ETA 02:35:46 2020-11-04 09:14:06 [INFO] [TRAIN] epoch=175, iter=65000/80000, loss=0.0954, lr=0.002217, batch_cost=0.6213, reader_cost=0.0007 | ETA 02:35:19 2020-11-04 09:15:08 [INFO] [TRAIN] epoch=175, iter=65100/80000, loss=0.0873, lr=0.002203, batch_cost=0.6198, reader_cost=0.0003 | ETA 02:33:54 2020-11-04 09:16:11 [INFO] [TRAIN] epoch=176, iter=65200/80000, loss=0.0951, lr=0.002190, batch_cost=0.6309, reader_cost=0.0099 | ETA 02:35:38 2020-11-04 09:17:13 [INFO] [TRAIN] epoch=176, iter=65300/80000, loss=0.0904, lr=0.002177, batch_cost=0.6161, reader_cost=0.0003 | ETA 02:30:55 2020-11-04 09:18:15 [INFO] [TRAIN] epoch=176, iter=65400/80000, loss=0.0888, lr=0.002164, batch_cost=0.6168, reader_cost=0.0002 | ETA 02:30:05 2020-11-04 09:19:17 [INFO] [TRAIN] epoch=177, iter=65500/80000, loss=0.0968, lr=0.002150, batch_cost=0.6267, reader_cost=0.0089 | ETA 02:31:27 2020-11-04 09:20:19 [INFO] [TRAIN] epoch=177, iter=65600/80000, loss=0.1081, lr=0.002137, batch_cost=0.6169, reader_cost=0.0003 | ETA 02:28:02 2020-11-04 09:21:21 [INFO] [TRAIN] epoch=177, iter=65700/80000, loss=0.1011, lr=0.002123, batch_cost=0.6224, reader_cost=0.0004 | ETA 02:28:21 2020-11-04 09:22:24 [INFO] [TRAIN] epoch=177, iter=65800/80000, loss=0.0889, lr=0.002110, batch_cost=0.6214, reader_cost=0.0002 | ETA 02:27:04 2020-11-04 09:23:27 [INFO] [TRAIN] epoch=178, iter=65900/80000, loss=0.0912, lr=0.002097, batch_cost=0.6301, reader_cost=0.0102 | ETA 02:28:04 2020-11-04 09:24:29 [INFO] [TRAIN] epoch=178, iter=66000/80000, loss=0.1103, lr=0.002083, batch_cost=0.6273, reader_cost=0.0002 | ETA 02:26:21 2020-11-04 09:25:32 [INFO] [TRAIN] epoch=178, iter=66100/80000, loss=0.1017, lr=0.002070, batch_cost=0.6268, reader_cost=0.0002 | ETA 02:25:12 2020-11-04 09:26:35 [INFO] [TRAIN] epoch=178, iter=66200/80000, loss=0.0941, lr=0.002057, batch_cost=0.6280, reader_cost=0.0002 | ETA 02:24:26 2020-11-04 09:27:38 [INFO] [TRAIN] epoch=179, iter=66300/80000, loss=0.0941, lr=0.002043, batch_cost=0.6320, reader_cost=0.0093 | ETA 02:24:18 2020-11-04 09:28:40 [INFO] [TRAIN] epoch=179, iter=66400/80000, loss=0.1089, lr=0.002030, batch_cost=0.6240, reader_cost=0.0002 | ETA 02:21:26 2020-11-04 09:29:43 [INFO] [TRAIN] epoch=179, iter=66500/80000, loss=0.0944, lr=0.002016, batch_cost=0.6236, reader_cost=0.0001 | ETA 02:20:19 2020-11-04 09:30:46 [INFO] [TRAIN] epoch=180, iter=66600/80000, loss=0.0942, lr=0.002003, batch_cost=0.6309, reader_cost=0.0094 | ETA 02:20:54 2020-11-04 09:31:48 [INFO] [TRAIN] epoch=180, iter=66700/80000, loss=0.1052, lr=0.001989, batch_cost=0.6180, reader_cost=0.0002 | ETA 02:16:59 2020-11-04 09:32:49 [INFO] [TRAIN] epoch=180, iter=66800/80000, loss=0.1033, lr=0.001976, batch_cost=0.6185, reader_cost=0.0004 | ETA 02:16:03 2020-11-04 09:33:51 [INFO] [TRAIN] epoch=180, iter=66900/80000, loss=0.1031, lr=0.001962, batch_cost=0.6164, reader_cost=0.0003 | ETA 02:14:35 2020-11-04 09:34:54 [INFO] [TRAIN] epoch=181, iter=67000/80000, loss=0.0949, lr=0.001949, batch_cost=0.6272, reader_cost=0.0091 | ETA 02:15:53 2020-11-04 09:35:55 [INFO] [TRAIN] epoch=181, iter=67100/80000, loss=0.0945, lr=0.001935, batch_cost=0.6163, reader_cost=0.0005 | ETA 02:12:30 2020-11-04 09:36:57 [INFO] [TRAIN] epoch=181, iter=67200/80000, loss=0.1043, lr=0.001922, batch_cost=0.6156, reader_cost=0.0002 | ETA 02:11:19 2020-11-04 09:37:59 [INFO] [TRAIN] epoch=181, iter=67300/80000, loss=0.0864, lr=0.001908, batch_cost=0.6149, reader_cost=0.0005 | ETA 02:10:08 2020-11-04 09:39:01 [INFO] [TRAIN] epoch=182, iter=67400/80000, loss=0.0921, lr=0.001895, batch_cost=0.6265, reader_cost=0.0084 | ETA 02:11:34 2020-11-04 09:40:03 [INFO] [TRAIN] epoch=182, iter=67500/80000, loss=0.0991, lr=0.001881, batch_cost=0.6190, reader_cost=0.0004 | ETA 02:08:57 2020-11-04 09:41:05 [INFO] [TRAIN] epoch=182, iter=67600/80000, loss=0.0971, lr=0.001868, batch_cost=0.6210, reader_cost=0.0008 | ETA 02:08:19 2020-11-04 09:42:07 [INFO] [TRAIN] epoch=182, iter=67700/80000, loss=0.0935, lr=0.001854, batch_cost=0.6203, reader_cost=0.0008 | ETA 02:07:09 2020-11-04 09:43:10 [INFO] [TRAIN] epoch=183, iter=67800/80000, loss=0.0966, lr=0.001841, batch_cost=0.6266, reader_cost=0.0094 | ETA 02:07:24 2020-11-04 09:44:11 [INFO] [TRAIN] epoch=183, iter=67900/80000, loss=0.0981, lr=0.001827, batch_cost=0.6158, reader_cost=0.0004 | ETA 02:04:11 2020-11-04 09:45:13 [INFO] [TRAIN] epoch=183, iter=68000/80000, loss=0.0955, lr=0.001813, batch_cost=0.6191, reader_cost=0.0006 | ETA 02:03:49 2020-11-04 09:46:16 [INFO] [TRAIN] epoch=184, iter=68100/80000, loss=0.0870, lr=0.001800, batch_cost=0.6280, reader_cost=0.0097 | ETA 02:04:33 2020-11-04 09:47:18 [INFO] [TRAIN] epoch=184, iter=68200/80000, loss=0.0968, lr=0.001786, batch_cost=0.6180, reader_cost=0.0002 | ETA 02:01:32 2020-11-04 09:48:20 [INFO] [TRAIN] epoch=184, iter=68300/80000, loss=0.0952, lr=0.001773, batch_cost=0.6177, reader_cost=0.0003 | ETA 02:00:26 2020-11-04 09:49:21 [INFO] [TRAIN] epoch=184, iter=68400/80000, loss=0.0941, lr=0.001759, batch_cost=0.6166, reader_cost=0.0004 | ETA 01:59:12 2020-11-04 09:50:25 [INFO] [TRAIN] epoch=185, iter=68500/80000, loss=0.1103, lr=0.001745, batch_cost=0.6315, reader_cost=0.0095 | ETA 02:01:01 2020-11-04 09:51:28 [INFO] [TRAIN] epoch=185, iter=68600/80000, loss=0.0928, lr=0.001732, batch_cost=0.6339, reader_cost=0.0007 | ETA 02:00:26 2020-11-04 09:52:32 [INFO] [TRAIN] epoch=185, iter=68700/80000, loss=0.1022, lr=0.001718, batch_cost=0.6375, reader_cost=0.0006 | ETA 02:00:03 2020-11-04 09:53:34 [INFO] [TRAIN] epoch=185, iter=68800/80000, loss=0.0948, lr=0.001704, batch_cost=0.6281, reader_cost=0.0003 | ETA 01:57:14 2020-11-04 09:54:37 [INFO] [TRAIN] epoch=186, iter=68900/80000, loss=0.0960, lr=0.001691, batch_cost=0.6291, reader_cost=0.0095 | ETA 01:56:22 2020-11-04 09:55:40 [INFO] [TRAIN] epoch=186, iter=69000/80000, loss=0.1008, lr=0.001677, batch_cost=0.6220, reader_cost=0.0005 | ETA 01:54:02 2020-11-04 09:56:41 [INFO] [TRAIN] epoch=186, iter=69100/80000, loss=0.1039, lr=0.001663, batch_cost=0.6188, reader_cost=0.0003 | ETA 01:52:25 2020-11-04 09:57:44 [INFO] [TRAIN] epoch=187, iter=69200/80000, loss=0.0815, lr=0.001649, batch_cost=0.6296, reader_cost=0.0096 | ETA 01:53:19 2020-11-04 09:58:46 [INFO] [TRAIN] epoch=187, iter=69300/80000, loss=0.1055, lr=0.001636, batch_cost=0.6205, reader_cost=0.0004 | ETA 01:50:39 2020-11-04 09:59:49 [INFO] [TRAIN] epoch=187, iter=69400/80000, loss=0.1009, lr=0.001622, batch_cost=0.6251, reader_cost=0.0005 | ETA 01:50:26 2020-11-04 10:00:52 [INFO] [TRAIN] epoch=187, iter=69500/80000, loss=0.0945, lr=0.001608, batch_cost=0.6252, reader_cost=0.0007 | ETA 01:49:25 2020-11-04 10:01:55 [INFO] [TRAIN] epoch=188, iter=69600/80000, loss=0.0887, lr=0.001594, batch_cost=0.6301, reader_cost=0.0099 | ETA 01:49:13 2020-11-04 10:02:57 [INFO] [TRAIN] epoch=188, iter=69700/80000, loss=0.0970, lr=0.001581, batch_cost=0.6203, reader_cost=0.0003 | ETA 01:46:29 2020-11-04 10:03:59 [INFO] [TRAIN] epoch=188, iter=69800/80000, loss=0.1036, lr=0.001567, batch_cost=0.6207, reader_cost=0.0002 | ETA 01:45:30 2020-11-04 10:05:01 [INFO] [TRAIN] epoch=188, iter=69900/80000, loss=0.0924, lr=0.001553, batch_cost=0.6225, reader_cost=0.0002 | ETA 01:44:47 2020-11-04 10:06:04 [INFO] [TRAIN] epoch=189, iter=70000/80000, loss=0.0807, lr=0.001539, batch_cost=0.6311, reader_cost=0.0106 | ETA 01:45:10 2020-11-04 10:07:06 [INFO] [TRAIN] epoch=189, iter=70100/80000, loss=0.0941, lr=0.001525, batch_cost=0.6187, reader_cost=0.0006 | ETA 01:42:05 2020-11-04 10:08:08 [INFO] [TRAIN] epoch=189, iter=70200/80000, loss=0.1067, lr=0.001511, batch_cost=0.6163, reader_cost=0.0004 | ETA 01:40:40 2020-11-04 10:09:09 [INFO] [TRAIN] epoch=189, iter=70300/80000, loss=0.0881, lr=0.001497, batch_cost=0.6169, reader_cost=0.0003 | ETA 01:39:44 2020-11-04 10:10:12 [INFO] [TRAIN] epoch=190, iter=70400/80000, loss=0.0995, lr=0.001484, batch_cost=0.6259, reader_cost=0.0087 | ETA 01:40:08 2020-11-04 10:11:14 [INFO] [TRAIN] epoch=190, iter=70500/80000, loss=0.0949, lr=0.001470, batch_cost=0.6170, reader_cost=0.0003 | ETA 01:37:41 2020-11-04 10:12:15 [INFO] [TRAIN] epoch=190, iter=70600/80000, loss=0.0888, lr=0.001456, batch_cost=0.6149, reader_cost=0.0003 | ETA 01:36:20 2020-11-04 10:13:18 [INFO] [TRAIN] epoch=191, iter=70700/80000, loss=0.0901, lr=0.001442, batch_cost=0.6269, reader_cost=0.0100 | ETA 01:37:10 2020-11-04 10:14:19 [INFO] [TRAIN] epoch=191, iter=70800/80000, loss=0.0961, lr=0.001428, batch_cost=0.6173, reader_cost=0.0003 | ETA 01:34:39 2020-11-04 10:15:22 [INFO] [TRAIN] epoch=191, iter=70900/80000, loss=0.1133, lr=0.001414, batch_cost=0.6209, reader_cost=0.0002 | ETA 01:34:10 2020-11-04 10:16:24 [INFO] [TRAIN] epoch=191, iter=71000/80000, loss=0.0947, lr=0.001400, batch_cost=0.6217, reader_cost=0.0003 | ETA 01:33:15 2020-11-04 10:17:26 [INFO] [TRAIN] epoch=192, iter=71100/80000, loss=0.0920, lr=0.001386, batch_cost=0.6276, reader_cost=0.0094 | ETA 01:33:05 2020-11-04 10:18:28 [INFO] [TRAIN] epoch=192, iter=71200/80000, loss=0.1014, lr=0.001372, batch_cost=0.6153, reader_cost=0.0002 | ETA 01:30:14 2020-11-04 10:19:30 [INFO] [TRAIN] epoch=192, iter=71300/80000, loss=0.1118, lr=0.001358, batch_cost=0.6173, reader_cost=0.0002 | ETA 01:29:30 2020-11-04 10:20:31 [INFO] [TRAIN] epoch=192, iter=71400/80000, loss=0.0883, lr=0.001344, batch_cost=0.6159, reader_cost=0.0003 | ETA 01:28:16 2020-11-04 10:21:34 [INFO] [TRAIN] epoch=193, iter=71500/80000, loss=0.0845, lr=0.001330, batch_cost=0.6265, reader_cost=0.0086 | ETA 01:28:45 2020-11-04 10:22:36 [INFO] [TRAIN] epoch=193, iter=71600/80000, loss=0.1016, lr=0.001316, batch_cost=0.6207, reader_cost=0.0003 | ETA 01:26:53 2020-11-04 10:23:38 [INFO] [TRAIN] epoch=193, iter=71700/80000, loss=0.0989, lr=0.001301, batch_cost=0.6174, reader_cost=0.0002 | ETA 01:25:24 2020-11-04 10:24:40 [INFO] [TRAIN] epoch=194, iter=71800/80000, loss=0.0961, lr=0.001287, batch_cost=0.6257, reader_cost=0.0085 | ETA 01:25:30 2020-11-04 10:25:42 [INFO] [TRAIN] epoch=194, iter=71900/80000, loss=0.0983, lr=0.001273, batch_cost=0.6180, reader_cost=0.0003 | ETA 01:23:26 2020-11-04 10:26:44 [INFO] [TRAIN] epoch=194, iter=72000/80000, loss=0.0869, lr=0.001259, batch_cost=0.6199, reader_cost=0.0002 | ETA 01:22:38 2020-11-04 10:26:47 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-04 10:30:56 [INFO] [EVAL] #Images=500 mIoU=0.8023 Acc=0.9638 Kappa=0.9530 2020-11-04 10:30:56 [INFO] [EVAL] Category IoU: [0.9833 0.8626 0.9293 0.5451 0.635 0.6783 0.7237 0.8053 0.9283 0.6496 0.9508 0.8397 0.6551 0.9569 0.8408 0.9221 0.8613 0.6883 0.7894] 2020-11-04 10:30:56 [INFO] [EVAL] Category Acc: [0.9918 0.9246 0.962 0.821 0.842 0.8012 0.8361 0.8982 0.9568 0.8439 0.968 0.9021 0.8044 0.9756 0.9301 0.9746 0.9383 0.8384 0.8662] 2020-11-04 10:30:57 [INFO] [EVAL] The model with the best validation mIoU (0.8023) was saved at iter 72000. 2020-11-04 10:32:00 [INFO] [TRAIN] epoch=194, iter=72100/80000, loss=0.1013, lr=0.001245, batch_cost=0.6299, reader_cost=0.0002 | ETA 01:22:56 2020-11-04 10:33:04 [INFO] [TRAIN] epoch=195, iter=72200/80000, loss=0.0910, lr=0.001231, batch_cost=0.6319, reader_cost=0.0075 | ETA 01:22:08 2020-11-04 10:34:05 [INFO] [TRAIN] epoch=195, iter=72300/80000, loss=0.1005, lr=0.001216, batch_cost=0.6184, reader_cost=0.0002 | ETA 01:19:21 2020-11-04 10:35:07 [INFO] [TRAIN] epoch=195, iter=72400/80000, loss=0.1019, lr=0.001202, batch_cost=0.6197, reader_cost=0.0002 | ETA 01:18:29 2020-11-04 10:36:09 [INFO] [TRAIN] epoch=195, iter=72500/80000, loss=0.0923, lr=0.001188, batch_cost=0.6187, reader_cost=0.0003 | ETA 01:17:20 2020-11-04 10:37:12 [INFO] [TRAIN] epoch=196, iter=72600/80000, loss=0.0909, lr=0.001174, batch_cost=0.6262, reader_cost=0.0093 | ETA 01:17:14 2020-11-04 10:38:14 [INFO] [TRAIN] epoch=196, iter=72700/80000, loss=0.1013, lr=0.001159, batch_cost=0.6221, reader_cost=0.0002 | ETA 01:15:41 2020-11-04 10:39:17 [INFO] [TRAIN] epoch=196, iter=72800/80000, loss=0.0959, lr=0.001145, batch_cost=0.6256, reader_cost=0.0002 | ETA 01:15:04 2020-11-04 10:40:19 [INFO] [TRAIN] epoch=196, iter=72900/80000, loss=0.0920, lr=0.001131, batch_cost=0.6225, reader_cost=0.0007 | ETA 01:13:40 2020-11-04 10:41:22 [INFO] [TRAIN] epoch=197, iter=73000/80000, loss=0.0937, lr=0.001117, batch_cost=0.6359, reader_cost=0.0095 | ETA 01:14:11 2020-11-04 10:42:25 [INFO] [TRAIN] epoch=197, iter=73100/80000, loss=0.1062, lr=0.001102, batch_cost=0.6267, reader_cost=0.0005 | ETA 01:12:04 2020-11-04 10:43:28 [INFO] [TRAIN] epoch=197, iter=73200/80000, loss=0.0865, lr=0.001088, batch_cost=0.6313, reader_cost=0.0006 | ETA 01:11:32 2020-11-04 10:44:32 [INFO] [TRAIN] epoch=198, iter=73300/80000, loss=0.0827, lr=0.001073, batch_cost=0.6394, reader_cost=0.0088 | ETA 01:11:24 2020-11-04 10:45:34 [INFO] [TRAIN] epoch=198, iter=73400/80000, loss=0.0957, lr=0.001059, batch_cost=0.6168, reader_cost=0.0004 | ETA 01:07:50 2020-11-04 10:46:36 [INFO] [TRAIN] epoch=198, iter=73500/80000, loss=0.1003, lr=0.001044, batch_cost=0.6180, reader_cost=0.0005 | ETA 01:06:56 2020-11-04 10:47:38 [INFO] [TRAIN] epoch=198, iter=73600/80000, loss=0.0911, lr=0.001030, batch_cost=0.6205, reader_cost=0.0005 | ETA 01:06:10 2020-11-04 10:48:41 [INFO] [TRAIN] epoch=199, iter=73700/80000, loss=0.0886, lr=0.001016, batch_cost=0.6283, reader_cost=0.0097 | ETA 01:05:58 2020-11-04 10:49:43 [INFO] [TRAIN] epoch=199, iter=73800/80000, loss=0.0977, lr=0.001001, batch_cost=0.6248, reader_cost=0.0005 | ETA 01:04:33 2020-11-04 10:50:45 [INFO] [TRAIN] epoch=199, iter=73900/80000, loss=0.0947, lr=0.000986, batch_cost=0.6213, reader_cost=0.0006 | ETA 01:03:09 2020-11-04 10:51:47 [INFO] [TRAIN] epoch=199, iter=74000/80000, loss=0.0875, lr=0.000972, batch_cost=0.6226, reader_cost=0.0006 | ETA 01:02:15 2020-11-04 10:52:51 [INFO] [TRAIN] epoch=200, iter=74100/80000, loss=0.0834, lr=0.000957, batch_cost=0.6324, reader_cost=0.0115 | ETA 01:02:11 2020-11-04 10:53:53 [INFO] [TRAIN] epoch=200, iter=74200/80000, loss=0.0975, lr=0.000943, batch_cost=0.6187, reader_cost=0.0002 | ETA 00:59:48 2020-11-04 10:54:54 [INFO] [TRAIN] epoch=200, iter=74300/80000, loss=0.0907, lr=0.000928, batch_cost=0.6184, reader_cost=0.0006 | ETA 00:58:44 2020-11-04 10:55:57 [INFO] [TRAIN] epoch=200, iter=74400/80000, loss=0.0851, lr=0.000913, batch_cost=0.6215, reader_cost=0.0003 | ETA 00:58:00 2020-11-04 10:57:00 [INFO] [TRAIN] epoch=201, iter=74500/80000, loss=0.0954, lr=0.000899, batch_cost=0.6333, reader_cost=0.0091 | ETA 00:58:03 2020-11-04 10:58:03 [INFO] [TRAIN] epoch=201, iter=74600/80000, loss=0.0963, lr=0.000884, batch_cost=0.6298, reader_cost=0.0002 | ETA 00:56:41 2020-11-04 10:59:06 [INFO] [TRAIN] epoch=201, iter=74700/80000, loss=0.0955, lr=0.000869, batch_cost=0.6281, reader_cost=0.0002 | ETA 00:55:28 2020-11-04 11:00:09 [INFO] [TRAIN] epoch=202, iter=74800/80000, loss=0.0897, lr=0.000854, batch_cost=0.6354, reader_cost=0.0084 | ETA 00:55:04 2020-11-04 11:01:12 [INFO] [TRAIN] epoch=202, iter=74900/80000, loss=0.0929, lr=0.000840, batch_cost=0.6318, reader_cost=0.0006 | ETA 00:53:42 2020-11-04 11:02:15 [INFO] [TRAIN] epoch=202, iter=75000/80000, loss=0.0912, lr=0.000825, batch_cost=0.6262, reader_cost=0.0004 | ETA 00:52:11 2020-11-04 11:03:17 [INFO] [TRAIN] epoch=202, iter=75100/80000, loss=0.1015, lr=0.000810, batch_cost=0.6210, reader_cost=0.0003 | ETA 00:50:42 2020-11-04 11:04:20 [INFO] [TRAIN] epoch=203, iter=75200/80000, loss=0.0954, lr=0.000795, batch_cost=0.6301, reader_cost=0.0096 | ETA 00:50:24 2020-11-04 11:05:22 [INFO] [TRAIN] epoch=203, iter=75300/80000, loss=0.0962, lr=0.000780, batch_cost=0.6177, reader_cost=0.0005 | ETA 00:48:23 2020-11-04 11:06:24 [INFO] [TRAIN] epoch=203, iter=75400/80000, loss=0.0992, lr=0.000765, batch_cost=0.6195, reader_cost=0.0003 | ETA 00:47:29 2020-11-04 11:07:26 [INFO] [TRAIN] epoch=203, iter=75500/80000, loss=0.0794, lr=0.000750, batch_cost=0.6175, reader_cost=0.0001 | ETA 00:46:18 2020-11-04 11:08:29 [INFO] [TRAIN] epoch=204, iter=75600/80000, loss=0.0895, lr=0.000735, batch_cost=0.6312, reader_cost=0.0099 | ETA 00:46:17 2020-11-04 11:09:31 [INFO] [TRAIN] epoch=204, iter=75700/80000, loss=0.0959, lr=0.000720, batch_cost=0.6233, reader_cost=0.0002 | ETA 00:44:40 2020-11-04 11:10:34 [INFO] [TRAIN] epoch=204, iter=75800/80000, loss=0.0857, lr=0.000705, batch_cost=0.6258, reader_cost=0.0003 | ETA 00:43:48 2020-11-04 11:11:38 [INFO] [TRAIN] epoch=205, iter=75900/80000, loss=0.0856, lr=0.000690, batch_cost=0.6409, reader_cost=0.0094 | ETA 00:43:47 2020-11-04 11:12:40 [INFO] [TRAIN] epoch=205, iter=76000/80000, loss=0.0933, lr=0.000675, batch_cost=0.6265, reader_cost=0.0005 | ETA 00:41:46 2020-11-04 11:13:43 [INFO] [TRAIN] epoch=205, iter=76100/80000, loss=0.0945, lr=0.000660, batch_cost=0.6272, reader_cost=0.0008 | ETA 00:40:46 2020-11-04 11:14:46 [INFO] [TRAIN] epoch=205, iter=76200/80000, loss=0.0918, lr=0.000644, batch_cost=0.6304, reader_cost=0.0008 | ETA 00:39:55 2020-11-04 11:15:49 [INFO] [TRAIN] epoch=206, iter=76300/80000, loss=0.0956, lr=0.000629, batch_cost=0.6320, reader_cost=0.0089 | ETA 00:38:58 2020-11-04 11:16:52 [INFO] [TRAIN] epoch=206, iter=76400/80000, loss=0.0951, lr=0.000614, batch_cost=0.6256, reader_cost=0.0004 | ETA 00:37:32 2020-11-04 11:17:55 [INFO] [TRAIN] epoch=206, iter=76500/80000, loss=0.0998, lr=0.000598, batch_cost=0.6305, reader_cost=0.0004 | ETA 00:36:46 2020-11-04 11:18:58 [INFO] [TRAIN] epoch=206, iter=76600/80000, loss=0.0861, lr=0.000583, batch_cost=0.6335, reader_cost=0.0005 | ETA 00:35:54 2020-11-04 11:20:02 [INFO] [TRAIN] epoch=207, iter=76700/80000, loss=0.0937, lr=0.000568, batch_cost=0.6347, reader_cost=0.0084 | ETA 00:34:54 2020-11-04 11:21:05 [INFO] [TRAIN] epoch=207, iter=76800/80000, loss=0.0981, lr=0.000552, batch_cost=0.6276, reader_cost=0.0007 | ETA 00:33:28 2020-11-04 11:22:07 [INFO] [TRAIN] epoch=207, iter=76900/80000, loss=0.0986, lr=0.000537, batch_cost=0.6278, reader_cost=0.0007 | ETA 00:32:26 2020-11-04 11:23:11 [INFO] [TRAIN] epoch=207, iter=77000/80000, loss=0.0846, lr=0.000521, batch_cost=0.6320, reader_cost=0.0003 | ETA 00:31:35 2020-11-04 11:24:14 [INFO] [TRAIN] epoch=208, iter=77100/80000, loss=0.0973, lr=0.000505, batch_cost=0.6301, reader_cost=0.0088 | ETA 00:30:27 2020-11-04 11:25:15 [INFO] [TRAIN] epoch=208, iter=77200/80000, loss=0.0865, lr=0.000490, batch_cost=0.6193, reader_cost=0.0008 | ETA 00:28:53 2020-11-04 11:26:18 [INFO] [TRAIN] epoch=208, iter=77300/80000, loss=0.0904, lr=0.000474, batch_cost=0.6229, reader_cost=0.0005 | ETA 00:28:01 2020-11-04 11:27:21 [INFO] [TRAIN] epoch=209, iter=77400/80000, loss=0.0888, lr=0.000458, batch_cost=0.6288, reader_cost=0.0092 | ETA 00:27:14 2020-11-04 11:28:23 [INFO] [TRAIN] epoch=209, iter=77500/80000, loss=0.0966, lr=0.000442, batch_cost=0.6209, reader_cost=0.0006 | ETA 00:25:52 2020-11-04 11:29:25 [INFO] [TRAIN] epoch=209, iter=77600/80000, loss=0.1004, lr=0.000426, batch_cost=0.6220, reader_cost=0.0006 | ETA 00:24:52 2020-11-04 11:30:27 [INFO] [TRAIN] epoch=209, iter=77700/80000, loss=0.0867, lr=0.000410, batch_cost=0.6236, reader_cost=0.0003 | ETA 00:23:54 2020-11-04 11:31:30 [INFO] [TRAIN] epoch=210, iter=77800/80000, loss=0.0825, lr=0.000394, batch_cost=0.6307, reader_cost=0.0101 | ETA 00:23:07 2020-11-04 11:32:33 [INFO] [TRAIN] epoch=210, iter=77900/80000, loss=0.0947, lr=0.000378, batch_cost=0.6296, reader_cost=0.0002 | ETA 00:22:02 2020-11-04 11:33:36 [INFO] [TRAIN] epoch=210, iter=78000/80000, loss=0.1223, lr=0.000362, batch_cost=0.6296, reader_cost=0.0002 | ETA 00:20:59 2020-11-04 11:34:39 [INFO] [TRAIN] epoch=210, iter=78100/80000, loss=0.0738, lr=0.000345, batch_cost=0.6289, reader_cost=0.0002 | ETA 00:19:54 2020-11-04 11:35:42 [INFO] [TRAIN] epoch=211, iter=78200/80000, loss=0.0853, lr=0.000329, batch_cost=0.6296, reader_cost=0.0090 | ETA 00:18:53 2020-11-04 11:36:44 [INFO] [TRAIN] epoch=211, iter=78300/80000, loss=0.1013, lr=0.000313, batch_cost=0.6193, reader_cost=0.0005 | ETA 00:17:32 2020-11-04 11:37:47 [INFO] [TRAIN] epoch=211, iter=78400/80000, loss=0.0920, lr=0.000296, batch_cost=0.6278, reader_cost=0.0010 | ETA 00:16:44 2020-11-04 11:38:51 [INFO] [TRAIN] epoch=212, iter=78500/80000, loss=0.0880, lr=0.000279, batch_cost=0.6401, reader_cost=0.0097 | ETA 00:16:00 2020-11-04 11:39:53 [INFO] [TRAIN] epoch=212, iter=78600/80000, loss=0.0953, lr=0.000262, batch_cost=0.6247, reader_cost=0.0003 | ETA 00:14:34 2020-11-04 11:40:56 [INFO] [TRAIN] epoch=212, iter=78700/80000, loss=0.0940, lr=0.000246, batch_cost=0.6306, reader_cost=0.0002 | ETA 00:13:39 2020-11-04 11:41:59 [INFO] [TRAIN] epoch=212, iter=78800/80000, loss=0.0920, lr=0.000228, batch_cost=0.6303, reader_cost=0.0002 | ETA 00:12:36 2020-11-04 11:43:03 [INFO] [TRAIN] epoch=213, iter=78900/80000, loss=0.0927, lr=0.000211, batch_cost=0.6332, reader_cost=0.0094 | ETA 00:11:36 2020-11-04 11:44:05 [INFO] [TRAIN] epoch=213, iter=79000/80000, loss=0.0982, lr=0.000194, batch_cost=0.6239, reader_cost=0.0003 | ETA 00:10:23 2020-11-04 11:45:08 [INFO] [TRAIN] epoch=213, iter=79100/80000, loss=0.1009, lr=0.000176, batch_cost=0.6247, reader_cost=0.0002 | ETA 00:09:22 2020-11-04 11:46:10 [INFO] [TRAIN] epoch=213, iter=79200/80000, loss=0.1009, lr=0.000159, batch_cost=0.6237, reader_cost=0.0003 | ETA 00:08:18 2020-11-04 11:47:13 [INFO] [TRAIN] epoch=214, iter=79300/80000, loss=0.0887, lr=0.000141, batch_cost=0.6292, reader_cost=0.0085 | ETA 00:07:20 2020-11-04 11:48:15 [INFO] [TRAIN] epoch=214, iter=79400/80000, loss=0.0930, lr=0.000123, batch_cost=0.6204, reader_cost=0.0006 | ETA 00:06:12 2020-11-04 11:49:17 [INFO] [TRAIN] epoch=214, iter=79500/80000, loss=0.0959, lr=0.000104, batch_cost=0.6183, reader_cost=0.0003 | ETA 00:05:09 2020-11-04 11:50:19 [INFO] [TRAIN] epoch=214, iter=79600/80000, loss=0.0900, lr=0.000085, batch_cost=0.6176, reader_cost=0.0002 | ETA 00:04:07 2020-11-04 11:51:22 [INFO] [TRAIN] epoch=215, iter=79700/80000, loss=0.0931, lr=0.000066, batch_cost=0.6311, reader_cost=0.0088 | ETA 00:03:09 2020-11-04 11:52:24 [INFO] [TRAIN] epoch=215, iter=79800/80000, loss=0.0882, lr=0.000046, batch_cost=0.6262, reader_cost=0.0002 | ETA 00:02:05 2020-11-04 11:53:27 [INFO] [TRAIN] epoch=215, iter=79900/80000, loss=0.0998, lr=0.000025, batch_cost=0.6269, reader_cost=0.0002 | ETA 00:01:02 2020-11-04 11:54:30 [INFO] [TRAIN] epoch=216, iter=80000/80000, loss=0.0914, lr=0.000000, batch_cost=0.6351, reader_cost=0.0097 | ETA 00:00:00 2020-11-04 11:54:33 [INFO] Start evaluating (total_samples=500, total_iters=500)... 2020-11-04 11:58:40 [INFO] [EVAL] #Images=500 mIoU=0.8036 Acc=0.9640 Kappa=0.9533 2020-11-04 11:58:40 [INFO] [EVAL] Category IoU: [0.9831 0.8629 0.9303 0.5711 0.6344 0.6815 0.7238 0.8067 0.9275 0.642 0.9516 0.8399 0.6574 0.957 0.8326 0.9176 0.8584 0.6992 0.7915] 2020-11-04 11:58:40 [INFO] [EVAL] Category Acc: [0.9915 0.9249 0.9604 0.8426 0.8562 0.822 0.8325 0.8961 0.9583 0.8218 0.9702 0.9001 0.7918 0.9752 0.9262 0.9623 0.9283 0.8272 0.8747] 2020-11-04 11:58:41 [INFO] [EVAL] The model with the best validation mIoU (0.8036) was saved at iter 80000.