2020-11-26 14:41:24 [INFO] ------------Environment Information------------- platform: Linux-3.10.0_3-0-0-34-x86_64-with-centos-7.5.1804-Core Python: 3.7.9 (default, Aug 31 2020, 12:42:55) [GCC 7.3.0] Paddle compiled with cuda: True NVCC: Cuda compilation tools, release 10.2, V10.2.89 cudnn: 7.6 GPUs used: 4 CUDA_VISIBLE_DEVICES: 0,1,2,3 GPU: ['GPU 0: Tesla V100-SXM2-16GB', 'GPU 1: Tesla V100-SXM2-16GB', 'GPU 2: Tesla V100-SXM2-16GB', 'GPU 3: Tesla V100-SXM2-16GB', 'GPU 4: Tesla V100-SXM2-16GB', 'GPU 5: Tesla V100-SXM2-16GB', 'GPU 6: Tesla V100-SXM2-16GB', 'GPU 7: Tesla V100-SXM2-16GB'] GCC: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39) PaddlePaddle: 2.0.0-rc0 OpenCV: 4.1.0 ------------------------------------------------ 2020-11-26 14:41:24 [INFO] ---------------Config Information--------------- batch_size: 4 iters: 40000 learning_rate: decay: end_lr: 0.0 power: 0.9 type: poly value: 0.01 loss: coef: - 1 types: - ignore_index: 255 type: CrossEntropyLoss model: align_corners: false aspp_out_channels: 256 aspp_ratios: - 1 - 12 - 24 - 36 backbone: multi_grid: - 1 - 2 - 4 output_stride: 8 pretrained: https://bj.bcebos.com/paddleseg/dygraph/resnet50_vd_ssld_v2.tar.gz type: ResNet50_vd backbone_indices: - 3 pretrained: null type: DeepLabV3 optimizer: momentum: 0.9 type: sgd weight_decay: 4.0e-05 train_dataset: dataset_root: data/VOCdevkit/ mode: trainaug transforms: - max_scale_factor: 2.0 min_scale_factor: 0.5 scale_step_size: 0.25 type: ResizeStepScaling - crop_size: - 512 - 512 type: RandomPaddingCrop - type: RandomHorizontalFlip - brightness_range: 0.4 contrast_range: 0.4 saturation_range: 0.4 type: RandomDistort - type: Normalize type: PascalVOC val_dataset: dataset_root: data/VOCdevkit/ mode: val transforms: - target_size: - 512 - 512 type: Padding - type: Normalize type: PascalVOC ------------------------------------------------ 2020-11-26 14:41:28 [INFO] Loading pretrained model from https://bj.bcebos.com/paddleseg/dygraph/resnet50_vd_ssld_v2.tar.gz 2020-11-26 14:41:29 [INFO] There are 275/275 variables loaded into ResNet_vd. 2020-11-26 14:43:21 [INFO] [TRAIN] epoch=1, iter=100/40000, loss=0.9999, lr=0.009978, batch_cost=1.0564, reader_cost=0.0100 | ETA 11:42:31 2020-11-26 14:45:03 [INFO] [TRAIN] epoch=1, iter=200/40000, loss=0.6554, lr=0.009955, batch_cost=1.0186, reader_cost=0.0002 | ETA 11:15:39 2020-11-26 14:46:45 [INFO] [TRAIN] epoch=1, iter=300/40000, loss=0.6415, lr=0.009933, batch_cost=1.0150, reader_cost=0.0001 | ETA 11:11:35 2020-11-26 14:48:26 [INFO] [TRAIN] epoch=1, iter=400/40000, loss=0.5426, lr=0.009910, batch_cost=1.0162, reader_cost=0.0002 | ETA 11:10:39 2020-11-26 14:50:08 [INFO] [TRAIN] epoch=1, iter=500/40000, loss=0.4906, lr=0.009888, batch_cost=1.0188, reader_cost=0.0002 | ETA 11:10:42 2020-11-26 14:51:50 [INFO] [TRAIN] epoch=1, iter=600/40000, loss=0.5364, lr=0.009865, batch_cost=1.0174, reader_cost=0.0002 | ETA 11:08:04 2020-11-26 14:53:32 [INFO] [TRAIN] epoch=2, iter=700/40000, loss=0.4431, lr=0.009843, batch_cost=1.0233, reader_cost=0.0067 | ETA 11:10:16 2020-11-26 14:55:14 [INFO] [TRAIN] epoch=2, iter=800/40000, loss=0.4676, lr=0.009820, batch_cost=1.0158, reader_cost=0.0007 | ETA 11:03:38 2020-11-26 14:56:56 [INFO] [TRAIN] epoch=2, iter=900/40000, loss=0.4215, lr=0.009797, batch_cost=1.0218, reader_cost=0.0011 | ETA 11:05:52 2020-11-26 14:58:38 [INFO] [TRAIN] epoch=2, iter=1000/40000, loss=0.4434, lr=0.009775, batch_cost=1.0182, reader_cost=0.0011 | ETA 11:01:50 2020-11-26 15:00:20 [INFO] [TRAIN] epoch=2, iter=1100/40000, loss=0.4475, lr=0.009752, batch_cost=1.0180, reader_cost=0.0012 | ETA 11:00:00 2020-11-26 15:02:01 [INFO] [TRAIN] epoch=2, iter=1200/40000, loss=0.4237, lr=0.009730, batch_cost=1.0184, reader_cost=0.0007 | ETA 10:58:33 2020-11-26 15:03:44 [INFO] [TRAIN] epoch=2, iter=1300/40000, loss=0.4858, lr=0.009707, batch_cost=1.0220, reader_cost=0.0007 | ETA 10:59:12 2020-11-26 15:05:27 [INFO] [TRAIN] epoch=3, iter=1400/40000, loss=0.4339, lr=0.009685, batch_cost=1.0295, reader_cost=0.0065 | ETA 11:02:20 2020-11-26 15:07:08 [INFO] [TRAIN] epoch=3, iter=1500/40000, loss=0.4183, lr=0.009662, batch_cost=1.0187, reader_cost=0.0010 | ETA 10:53:40 2020-11-26 15:08:50 [INFO] [TRAIN] epoch=3, iter=1600/40000, loss=0.3413, lr=0.009639, batch_cost=1.0128, reader_cost=0.0011 | ETA 10:48:13 2020-11-26 15:10:31 [INFO] [TRAIN] epoch=3, iter=1700/40000, loss=0.3668, lr=0.009617, batch_cost=1.0153, reader_cost=0.0006 | ETA 10:48:06 2020-11-26 15:12:13 [INFO] [TRAIN] epoch=3, iter=1800/40000, loss=0.4054, lr=0.009594, batch_cost=1.0199, reader_cost=0.0010 | ETA 10:49:21 2020-11-26 15:13:55 [INFO] [TRAIN] epoch=3, iter=1900/40000, loss=0.3252, lr=0.009572, batch_cost=1.0198, reader_cost=0.0013 | ETA 10:47:35 2020-11-26 15:15:38 [INFO] [TRAIN] epoch=4, iter=2000/40000, loss=0.3388, lr=0.009549, batch_cost=1.0246, reader_cost=0.0068 | ETA 10:48:56 2020-11-26 15:17:20 [INFO] [TRAIN] epoch=4, iter=2100/40000, loss=0.3694, lr=0.009526, batch_cost=1.0199, reader_cost=0.0008 | ETA 10:44:13 2020-11-26 15:19:01 [INFO] [TRAIN] epoch=4, iter=2200/40000, loss=0.3676, lr=0.009504, batch_cost=1.0125, reader_cost=0.0003 | ETA 10:37:51 2020-11-26 15:20:42 [INFO] [TRAIN] epoch=4, iter=2300/40000, loss=0.3835, lr=0.009481, batch_cost=1.0125, reader_cost=0.0002 | ETA 10:36:11 2020-11-26 15:22:23 [INFO] [TRAIN] epoch=4, iter=2400/40000, loss=0.3578, lr=0.009459, batch_cost=1.0098, reader_cost=0.0002 | ETA 10:32:50 2020-11-26 15:24:05 [INFO] [TRAIN] epoch=4, iter=2500/40000, loss=0.3437, lr=0.009436, batch_cost=1.0135, reader_cost=0.0004 | ETA 10:33:27 2020-11-26 15:25:46 [INFO] [TRAIN] epoch=4, iter=2600/40000, loss=0.3419, lr=0.009413, batch_cost=1.0137, reader_cost=0.0002 | ETA 10:31:53 2020-11-26 15:27:28 [INFO] [TRAIN] epoch=5, iter=2700/40000, loss=0.3350, lr=0.009391, batch_cost=1.0210, reader_cost=0.0061 | ETA 10:34:42 2020-11-26 15:29:09 [INFO] [TRAIN] epoch=5, iter=2800/40000, loss=0.2731, lr=0.009368, batch_cost=1.0150, reader_cost=0.0003 | ETA 10:29:18 2020-11-26 15:30:51 [INFO] [TRAIN] epoch=5, iter=2900/40000, loss=0.3146, lr=0.009345, batch_cost=1.0177, reader_cost=0.0004 | ETA 10:29:17 2020-11-26 15:32:33 [INFO] [TRAIN] epoch=5, iter=3000/40000, loss=0.2963, lr=0.009323, batch_cost=1.0194, reader_cost=0.0004 | ETA 10:28:35 2020-11-26 15:34:15 [INFO] [TRAIN] epoch=5, iter=3100/40000, loss=0.3474, lr=0.009300, batch_cost=1.0157, reader_cost=0.0002 | ETA 10:24:37 2020-11-26 15:35:57 [INFO] [TRAIN] epoch=5, iter=3200/40000, loss=0.3344, lr=0.009277, batch_cost=1.0219, reader_cost=0.0006 | ETA 10:26:47 2020-11-26 15:37:38 [INFO] [TRAIN] epoch=5, iter=3300/40000, loss=0.2687, lr=0.009255, batch_cost=1.0147, reader_cost=0.0002 | ETA 10:20:38 2020-11-26 15:39:21 [INFO] [TRAIN] epoch=6, iter=3400/40000, loss=0.2362, lr=0.009232, batch_cost=1.0249, reader_cost=0.0070 | ETA 10:25:10 2020-11-26 15:41:03 [INFO] [TRAIN] epoch=6, iter=3500/40000, loss=0.2872, lr=0.009209, batch_cost=1.0175, reader_cost=0.0011 | ETA 10:18:59 2020-11-26 15:42:45 [INFO] [TRAIN] epoch=6, iter=3600/40000, loss=0.2572, lr=0.009186, batch_cost=1.0188, reader_cost=0.0009 | ETA 10:18:03 2020-11-26 15:44:27 [INFO] [TRAIN] epoch=6, iter=3700/40000, loss=0.2579, lr=0.009164, batch_cost=1.0218, reader_cost=0.0011 | ETA 10:18:12 2020-11-26 15:46:09 [INFO] [TRAIN] epoch=6, iter=3800/40000, loss=0.2755, lr=0.009141, batch_cost=1.0194, reader_cost=0.0011 | ETA 10:15:00 2020-11-26 15:47:50 [INFO] [TRAIN] epoch=6, iter=3900/40000, loss=0.2650, lr=0.009118, batch_cost=1.0179, reader_cost=0.0011 | ETA 10:12:25 2020-11-26 15:49:33 [INFO] [TRAIN] epoch=7, iter=4000/40000, loss=0.2941, lr=0.009096, batch_cost=1.0257, reader_cost=0.0064 | ETA 10:15:26 2020-11-26 15:49:33 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-26 15:49:54 [INFO] [EVAL] #Images=1449 mIoU=0.7046 Acc=0.9300 Kappa=0.8470 2020-11-26 15:49:54 [INFO] [EVAL] Class IoU: [0.9284 0.7931 0.405 0.853 0.7157 0.6846 0.8529 0.8301 0.8068 0.348 0.8195 0.5712 0.7413 0.8131 0.8072 0.8332 0.535 0.7326 0.361 0.7334 0.6315] 2020-11-26 15:49:54 [INFO] [EVAL] Class Acc: [0.964 0.8412 0.4398 0.9543 0.8163 0.8057 0.9582 0.9436 0.8378 0.4426 0.9388 0.7522 0.9389 0.8731 0.9318 0.9076 0.7318 0.7773 0.7812 0.7651 0.729 ] 2020-11-26 15:49:59 [INFO] [EVAL] The model with the best validation mIoU (0.7046) was saved at iter 4000. 2020-11-26 15:51:42 [INFO] [TRAIN] epoch=7, iter=4100/40000, loss=0.2306, lr=0.009073, batch_cost=1.0236, reader_cost=0.0014 | ETA 10:12:26 2020-11-26 15:53:24 [INFO] [TRAIN] epoch=7, iter=4200/40000, loss=0.2826, lr=0.009050, batch_cost=1.0194, reader_cost=0.0015 | ETA 10:08:13 2020-11-26 15:55:06 [INFO] [TRAIN] epoch=7, iter=4300/40000, loss=0.3030, lr=0.009027, batch_cost=1.0197, reader_cost=0.0014 | ETA 10:06:44 2020-11-26 15:56:48 [INFO] [TRAIN] epoch=7, iter=4400/40000, loss=0.2505, lr=0.009005, batch_cost=1.0222, reader_cost=0.0011 | ETA 10:06:30 2020-11-26 15:58:30 [INFO] [TRAIN] epoch=7, iter=4500/40000, loss=0.2328, lr=0.008982, batch_cost=1.0195, reader_cost=0.0011 | ETA 10:03:12 2020-11-26 16:00:12 [INFO] [TRAIN] epoch=7, iter=4600/40000, loss=0.2669, lr=0.008959, batch_cost=1.0220, reader_cost=0.0011 | ETA 10:02:58 2020-11-26 16:01:55 [INFO] [TRAIN] epoch=8, iter=4700/40000, loss=0.2749, lr=0.008936, batch_cost=1.0299, reader_cost=0.0062 | ETA 10:05:55 2020-11-26 16:03:38 [INFO] [TRAIN] epoch=8, iter=4800/40000, loss=0.2907, lr=0.008913, batch_cost=1.0275, reader_cost=0.0005 | ETA 10:02:47 2020-11-26 16:05:21 [INFO] [TRAIN] epoch=8, iter=4900/40000, loss=0.2878, lr=0.008891, batch_cost=1.0289, reader_cost=0.0006 | ETA 10:01:53 2020-11-26 16:07:03 [INFO] [TRAIN] epoch=8, iter=5000/40000, loss=0.2591, lr=0.008868, batch_cost=1.0253, reader_cost=0.0004 | ETA 09:58:06 2020-11-26 16:08:46 [INFO] [TRAIN] epoch=8, iter=5100/40000, loss=0.2893, lr=0.008845, batch_cost=1.0279, reader_cost=0.0003 | ETA 09:57:53 2020-11-26 16:10:29 [INFO] [TRAIN] epoch=8, iter=5200/40000, loss=0.3013, lr=0.008822, batch_cost=1.0291, reader_cost=0.0007 | ETA 09:56:53 2020-11-26 16:12:12 [INFO] [TRAIN] epoch=9, iter=5300/40000, loss=0.2854, lr=0.008799, batch_cost=1.0322, reader_cost=0.0073 | ETA 09:56:56 2020-11-26 16:13:55 [INFO] [TRAIN] epoch=9, iter=5400/40000, loss=0.2701, lr=0.008777, batch_cost=1.0275, reader_cost=0.0003 | ETA 09:52:32 2020-11-26 16:15:38 [INFO] [TRAIN] epoch=9, iter=5500/40000, loss=0.2468, lr=0.008754, batch_cost=1.0269, reader_cost=0.0007 | ETA 09:50:27 2020-11-26 16:17:20 [INFO] [TRAIN] epoch=9, iter=5600/40000, loss=0.2500, lr=0.008731, batch_cost=1.0249, reader_cost=0.0005 | ETA 09:47:35 2020-11-26 16:19:03 [INFO] [TRAIN] epoch=9, iter=5700/40000, loss=0.2199, lr=0.008708, batch_cost=1.0263, reader_cost=0.0008 | ETA 09:46:42 2020-11-26 16:20:45 [INFO] [TRAIN] epoch=9, iter=5800/40000, loss=0.2433, lr=0.008685, batch_cost=1.0274, reader_cost=0.0006 | ETA 09:45:37 2020-11-26 16:22:28 [INFO] [TRAIN] epoch=9, iter=5900/40000, loss=0.2146, lr=0.008662, batch_cost=1.0264, reader_cost=0.0005 | ETA 09:43:21 2020-11-26 16:24:11 [INFO] [TRAIN] epoch=10, iter=6000/40000, loss=0.2244, lr=0.008639, batch_cost=1.0263, reader_cost=0.0063 | ETA 09:41:32 2020-11-26 16:25:53 [INFO] [TRAIN] epoch=10, iter=6100/40000, loss=0.2174, lr=0.008617, batch_cost=1.0208, reader_cost=0.0009 | ETA 09:36:44 2020-11-26 16:27:36 [INFO] [TRAIN] epoch=10, iter=6200/40000, loss=0.2492, lr=0.008594, batch_cost=1.0287, reader_cost=0.0006 | ETA 09:39:28 2020-11-26 16:29:19 [INFO] [TRAIN] epoch=10, iter=6300/40000, loss=0.2477, lr=0.008571, batch_cost=1.0304, reader_cost=0.0005 | ETA 09:38:45 2020-11-26 16:31:02 [INFO] [TRAIN] epoch=10, iter=6400/40000, loss=0.2246, lr=0.008548, batch_cost=1.0291, reader_cost=0.0003 | ETA 09:36:16 2020-11-26 16:32:44 [INFO] [TRAIN] epoch=10, iter=6500/40000, loss=0.2271, lr=0.008525, batch_cost=1.0252, reader_cost=0.0006 | ETA 09:32:22 2020-11-26 16:34:27 [INFO] [TRAIN] epoch=10, iter=6600/40000, loss=0.2343, lr=0.008502, batch_cost=1.0263, reader_cost=0.0005 | ETA 09:31:19 2020-11-26 16:36:10 [INFO] [TRAIN] epoch=11, iter=6700/40000, loss=0.2461, lr=0.008479, batch_cost=1.0342, reader_cost=0.0061 | ETA 09:33:59 2020-11-26 16:37:53 [INFO] [TRAIN] epoch=11, iter=6800/40000, loss=0.2875, lr=0.008456, batch_cost=1.0271, reader_cost=0.0004 | ETA 09:28:18 2020-11-26 16:39:36 [INFO] [TRAIN] epoch=11, iter=6900/40000, loss=0.2166, lr=0.008433, batch_cost=1.0304, reader_cost=0.0004 | ETA 09:28:25 2020-11-26 16:41:18 [INFO] [TRAIN] epoch=11, iter=7000/40000, loss=0.2270, lr=0.008410, batch_cost=1.0246, reader_cost=0.0002 | ETA 09:23:30 2020-11-26 16:43:01 [INFO] [TRAIN] epoch=11, iter=7100/40000, loss=0.2669, lr=0.008388, batch_cost=1.0227, reader_cost=0.0003 | ETA 09:20:48 2020-11-26 16:44:43 [INFO] [TRAIN] epoch=11, iter=7200/40000, loss=0.3101, lr=0.008365, batch_cost=1.0265, reader_cost=0.0002 | ETA 09:21:08 2020-11-26 16:46:27 [INFO] [TRAIN] epoch=12, iter=7300/40000, loss=0.2720, lr=0.008342, batch_cost=1.0343, reader_cost=0.0066 | ETA 09:23:43 2020-11-26 16:48:10 [INFO] [TRAIN] epoch=12, iter=7400/40000, loss=0.2641, lr=0.008319, batch_cost=1.0304, reader_cost=0.0004 | ETA 09:19:51 2020-11-26 16:49:52 [INFO] [TRAIN] epoch=12, iter=7500/40000, loss=0.2255, lr=0.008296, batch_cost=1.0218, reader_cost=0.0006 | ETA 09:13:29 2020-11-26 16:51:34 [INFO] [TRAIN] epoch=12, iter=7600/40000, loss=0.2017, lr=0.008273, batch_cost=1.0159, reader_cost=0.0002 | ETA 09:08:33 2020-11-26 16:53:16 [INFO] [TRAIN] epoch=12, iter=7700/40000, loss=0.2119, lr=0.008250, batch_cost=1.0219, reader_cost=0.0004 | ETA 09:10:05 2020-11-26 16:54:58 [INFO] [TRAIN] epoch=12, iter=7800/40000, loss=0.2160, lr=0.008227, batch_cost=1.0240, reader_cost=0.0005 | ETA 09:09:34 2020-11-26 16:56:41 [INFO] [TRAIN] epoch=12, iter=7900/40000, loss=0.2132, lr=0.008204, batch_cost=1.0260, reader_cost=0.0003 | ETA 09:08:53 2020-11-26 16:58:24 [INFO] [TRAIN] epoch=13, iter=8000/40000, loss=0.2067, lr=0.008181, batch_cost=1.0300, reader_cost=0.0067 | ETA 09:09:20 2020-11-26 16:58:24 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-26 16:58:46 [INFO] [EVAL] #Images=1449 mIoU=0.7500 Acc=0.9411 Kappa=0.8719 2020-11-26 16:58:46 [INFO] [EVAL] Class IoU: [0.9348 0.8417 0.4061 0.8761 0.5692 0.7643 0.8842 0.7882 0.896 0.3832 0.8832 0.6205 0.8199 0.86 0.8327 0.8374 0.5948 0.8727 0.5758 0.8018 0.7072] 2020-11-26 16:58:46 [INFO] [EVAL] Class Acc: [0.9689 0.8979 0.4202 0.9368 0.6259 0.8375 0.9622 0.8344 0.9475 0.6604 0.9477 0.8396 0.8473 0.8911 0.932 0.9296 0.747 0.9543 0.7774 0.8394 0.845 ] 2020-11-26 16:58:51 [INFO] [EVAL] The model with the best validation mIoU (0.7500) was saved at iter 8000. 2020-11-26 17:00:34 [INFO] [TRAIN] epoch=13, iter=8100/40000, loss=0.1898, lr=0.008158, batch_cost=1.0261, reader_cost=0.0006 | ETA 09:05:33 2020-11-26 17:02:17 [INFO] [TRAIN] epoch=13, iter=8200/40000, loss=0.2114, lr=0.008135, batch_cost=1.0278, reader_cost=0.0003 | ETA 09:04:43 2020-11-26 17:03:59 [INFO] [TRAIN] epoch=13, iter=8300/40000, loss=0.1888, lr=0.008112, batch_cost=1.0266, reader_cost=0.0005 | ETA 09:02:22 2020-11-26 17:05:42 [INFO] [TRAIN] epoch=13, iter=8400/40000, loss=0.2192, lr=0.008089, batch_cost=1.0265, reader_cost=0.0003 | ETA 09:00:38 2020-11-26 17:07:24 [INFO] [TRAIN] epoch=13, iter=8500/40000, loss=0.2417, lr=0.008066, batch_cost=1.0234, reader_cost=0.0004 | ETA 08:57:15 2020-11-26 17:09:07 [INFO] [TRAIN] epoch=14, iter=8600/40000, loss=0.1928, lr=0.008043, batch_cost=1.0276, reader_cost=0.0061 | ETA 08:57:45 2020-11-26 17:10:49 [INFO] [TRAIN] epoch=14, iter=8700/40000, loss=0.2312, lr=0.008020, batch_cost=1.0238, reader_cost=0.0004 | ETA 08:54:06 2020-11-26 17:12:32 [INFO] [TRAIN] epoch=14, iter=8800/40000, loss=0.2083, lr=0.007996, batch_cost=1.0257, reader_cost=0.0002 | ETA 08:53:20 2020-11-26 17:14:15 [INFO] [TRAIN] epoch=14, iter=8900/40000, loss=0.1664, lr=0.007973, batch_cost=1.0268, reader_cost=0.0007 | ETA 08:52:11 2020-11-26 17:15:57 [INFO] [TRAIN] epoch=14, iter=9000/40000, loss=0.1986, lr=0.007950, batch_cost=1.0190, reader_cost=0.0005 | ETA 08:46:28 2020-11-26 17:17:38 [INFO] [TRAIN] epoch=14, iter=9100/40000, loss=0.1850, lr=0.007927, batch_cost=1.0135, reader_cost=0.0002 | ETA 08:41:57 2020-11-26 17:19:20 [INFO] [TRAIN] epoch=14, iter=9200/40000, loss=0.2110, lr=0.007904, batch_cost=1.0210, reader_cost=0.0003 | ETA 08:44:06 2020-11-26 17:21:03 [INFO] [TRAIN] epoch=15, iter=9300/40000, loss=0.2105, lr=0.007881, batch_cost=1.0301, reader_cost=0.0059 | ETA 08:47:02 2020-11-26 17:22:46 [INFO] [TRAIN] epoch=15, iter=9400/40000, loss=0.2099, lr=0.007858, batch_cost=1.0263, reader_cost=0.0006 | ETA 08:43:24 2020-11-26 17:24:29 [INFO] [TRAIN] epoch=15, iter=9500/40000, loss=0.1988, lr=0.007835, batch_cost=1.0298, reader_cost=0.0009 | ETA 08:43:29 2020-11-26 17:26:11 [INFO] [TRAIN] epoch=15, iter=9600/40000, loss=0.2207, lr=0.007812, batch_cost=1.0256, reader_cost=0.0008 | ETA 08:39:37 2020-11-26 17:27:54 [INFO] [TRAIN] epoch=15, iter=9700/40000, loss=0.1966, lr=0.007789, batch_cost=1.0290, reader_cost=0.0008 | ETA 08:39:37 2020-11-26 17:29:37 [INFO] [TRAIN] epoch=15, iter=9800/40000, loss=0.1987, lr=0.007765, batch_cost=1.0298, reader_cost=0.0008 | ETA 08:38:20 2020-11-26 17:31:20 [INFO] [TRAIN] epoch=15, iter=9900/40000, loss=0.1720, lr=0.007742, batch_cost=1.0262, reader_cost=0.0008 | ETA 08:34:48 2020-11-26 17:33:03 [INFO] [TRAIN] epoch=16, iter=10000/40000, loss=0.1915, lr=0.007719, batch_cost=1.0327, reader_cost=0.0068 | ETA 08:36:19 2020-11-26 17:34:46 [INFO] [TRAIN] epoch=16, iter=10100/40000, loss=0.1662, lr=0.007696, batch_cost=1.0273, reader_cost=0.0006 | ETA 08:31:56 2020-11-26 17:36:28 [INFO] [TRAIN] epoch=16, iter=10200/40000, loss=0.2216, lr=0.007673, batch_cost=1.0276, reader_cost=0.0007 | ETA 08:30:22 2020-11-26 17:38:11 [INFO] [TRAIN] epoch=16, iter=10300/40000, loss=0.2053, lr=0.007650, batch_cost=1.0255, reader_cost=0.0005 | ETA 08:27:35 2020-11-26 17:39:54 [INFO] [TRAIN] epoch=16, iter=10400/40000, loss=0.1762, lr=0.007626, batch_cost=1.0265, reader_cost=0.0006 | ETA 08:26:25 2020-11-26 17:41:37 [INFO] [TRAIN] epoch=16, iter=10500/40000, loss=0.2039, lr=0.007603, batch_cost=1.0299, reader_cost=0.0004 | ETA 08:26:22 2020-11-26 17:43:20 [INFO] [TRAIN] epoch=17, iter=10600/40000, loss=0.1981, lr=0.007580, batch_cost=1.0284, reader_cost=0.0073 | ETA 08:23:53 2020-11-26 17:45:02 [INFO] [TRAIN] epoch=17, iter=10700/40000, loss=0.1899, lr=0.007557, batch_cost=1.0248, reader_cost=0.0009 | ETA 08:20:27 2020-11-26 17:46:45 [INFO] [TRAIN] epoch=17, iter=10800/40000, loss=0.1765, lr=0.007534, batch_cost=1.0285, reader_cost=0.0007 | ETA 08:20:32 2020-11-26 17:48:28 [INFO] [TRAIN] epoch=17, iter=10900/40000, loss=0.1834, lr=0.007510, batch_cost=1.0268, reader_cost=0.0006 | ETA 08:18:00 2020-11-26 17:50:11 [INFO] [TRAIN] epoch=17, iter=11000/40000, loss=0.1911, lr=0.007487, batch_cost=1.0296, reader_cost=0.0007 | ETA 08:17:39 2020-11-26 17:51:53 [INFO] [TRAIN] epoch=17, iter=11100/40000, loss=0.2035, lr=0.007464, batch_cost=1.0271, reader_cost=0.0006 | ETA 08:14:42 2020-11-26 17:53:36 [INFO] [TRAIN] epoch=17, iter=11200/40000, loss=0.2273, lr=0.007441, batch_cost=1.0268, reader_cost=0.0009 | ETA 08:12:52 2020-11-26 17:55:19 [INFO] [TRAIN] epoch=18, iter=11300/40000, loss=0.1867, lr=0.007417, batch_cost=1.0351, reader_cost=0.0075 | ETA 08:15:07 2020-11-26 17:57:02 [INFO] [TRAIN] epoch=18, iter=11400/40000, loss=0.1921, lr=0.007394, batch_cost=1.0220, reader_cost=0.0007 | ETA 08:07:09 2020-11-26 17:58:45 [INFO] [TRAIN] epoch=18, iter=11500/40000, loss=0.1633, lr=0.007371, batch_cost=1.0300, reader_cost=0.0004 | ETA 08:09:15 2020-11-26 18:00:27 [INFO] [TRAIN] epoch=18, iter=11600/40000, loss=0.1757, lr=0.007348, batch_cost=1.0268, reader_cost=0.0006 | ETA 08:06:01 2020-11-26 18:02:09 [INFO] [TRAIN] epoch=18, iter=11700/40000, loss=0.1953, lr=0.007324, batch_cost=1.0212, reader_cost=0.0008 | ETA 08:01:40 2020-11-26 18:03:52 [INFO] [TRAIN] epoch=18, iter=11800/40000, loss=0.1777, lr=0.007301, batch_cost=1.0249, reader_cost=0.0005 | ETA 08:01:40 2020-11-26 18:05:35 [INFO] [TRAIN] epoch=19, iter=11900/40000, loss=0.1979, lr=0.007278, batch_cost=1.0322, reader_cost=0.0068 | ETA 08:03:24 2020-11-26 18:07:18 [INFO] [TRAIN] epoch=19, iter=12000/40000, loss=0.1807, lr=0.007254, batch_cost=1.0278, reader_cost=0.0011 | ETA 07:59:39 2020-11-26 18:07:18 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-26 18:07:39 [INFO] [EVAL] #Images=1449 mIoU=0.7621 Acc=0.9447 Kappa=0.8779 2020-11-26 18:07:39 [INFO] [EVAL] Class IoU: [0.94 0.872 0.4247 0.8749 0.7374 0.7838 0.9222 0.8606 0.8956 0.3986 0.8623 0.5446 0.7898 0.852 0.8219 0.8381 0.6266 0.8671 0.5624 0.8493 0.6806] 2020-11-26 18:07:39 [INFO] [EVAL] Class Acc: [0.9652 0.9242 0.448 0.9388 0.8231 0.8766 0.9802 0.9404 0.9411 0.5998 0.9644 0.9049 0.8294 0.9286 0.9129 0.9485 0.8704 0.939 0.6959 0.9219 0.8554] 2020-11-26 18:07:45 [INFO] [EVAL] The model with the best validation mIoU (0.7621) was saved at iter 12000. 2020-11-26 18:09:27 [INFO] [TRAIN] epoch=19, iter=12100/40000, loss=0.1643, lr=0.007231, batch_cost=1.0180, reader_cost=0.0011 | ETA 07:53:21 2020-11-26 18:11:09 [INFO] [TRAIN] epoch=19, iter=12200/40000, loss=0.2183, lr=0.007208, batch_cost=1.0205, reader_cost=0.0004 | ETA 07:52:49 2020-11-26 18:12:51 [INFO] [TRAIN] epoch=19, iter=12300/40000, loss=0.2019, lr=0.007184, batch_cost=1.0232, reader_cost=0.0010 | ETA 07:52:22 2020-11-26 18:14:33 [INFO] [TRAIN] epoch=19, iter=12400/40000, loss=0.1665, lr=0.007161, batch_cost=1.0212, reader_cost=0.0007 | ETA 07:49:46 2020-11-26 18:16:15 [INFO] [TRAIN] epoch=19, iter=12500/40000, loss=0.1811, lr=0.007138, batch_cost=1.0213, reader_cost=0.0008 | ETA 07:48:07 2020-11-26 18:17:58 [INFO] [TRAIN] epoch=20, iter=12600/40000, loss=0.1565, lr=0.007114, batch_cost=1.0291, reader_cost=0.0064 | ETA 07:49:58 2020-11-26 18:19:41 [INFO] [TRAIN] epoch=20, iter=12700/40000, loss=0.1653, lr=0.007091, batch_cost=1.0256, reader_cost=0.0005 | ETA 07:46:38 2020-11-26 18:21:24 [INFO] [TRAIN] epoch=20, iter=12800/40000, loss=0.1917, lr=0.007068, batch_cost=1.0294, reader_cost=0.0008 | ETA 07:46:39 2020-11-26 18:23:06 [INFO] [TRAIN] epoch=20, iter=12900/40000, loss=0.1844, lr=0.007044, batch_cost=1.0267, reader_cost=0.0006 | ETA 07:43:43 2020-11-26 18:24:49 [INFO] [TRAIN] epoch=20, iter=13000/40000, loss=0.1683, lr=0.007021, batch_cost=1.0269, reader_cost=0.0006 | ETA 07:42:06 2020-11-26 18:26:32 [INFO] [TRAIN] epoch=20, iter=13100/40000, loss=0.1556, lr=0.006997, batch_cost=1.0271, reader_cost=0.0008 | ETA 07:40:30 2020-11-26 18:28:14 [INFO] [TRAIN] epoch=20, iter=13200/40000, loss=0.1398, lr=0.006974, batch_cost=1.0253, reader_cost=0.0007 | ETA 07:37:57 2020-11-26 18:29:58 [INFO] [TRAIN] epoch=21, iter=13300/40000, loss=0.1379, lr=0.006951, batch_cost=1.0348, reader_cost=0.0066 | ETA 07:40:28 2020-11-26 18:31:41 [INFO] [TRAIN] epoch=21, iter=13400/40000, loss=0.1520, lr=0.006927, batch_cost=1.0283, reader_cost=0.0005 | ETA 07:35:52 2020-11-26 18:33:23 [INFO] [TRAIN] epoch=21, iter=13500/40000, loss=0.1504, lr=0.006904, batch_cost=1.0212, reader_cost=0.0003 | ETA 07:31:00 2020-11-26 18:35:05 [INFO] [TRAIN] epoch=21, iter=13600/40000, loss=0.1284, lr=0.006880, batch_cost=1.0239, reader_cost=0.0003 | ETA 07:30:30 2020-11-26 18:36:48 [INFO] [TRAIN] epoch=21, iter=13700/40000, loss=0.1352, lr=0.006857, batch_cost=1.0252, reader_cost=0.0004 | ETA 07:29:22 2020-11-26 18:38:31 [INFO] [TRAIN] epoch=21, iter=13800/40000, loss=0.1671, lr=0.006833, batch_cost=1.0298, reader_cost=0.0007 | ETA 07:29:41 2020-11-26 18:40:14 [INFO] [TRAIN] epoch=22, iter=13900/40000, loss=0.1601, lr=0.006810, batch_cost=1.0347, reader_cost=0.0069 | ETA 07:30:04 2020-11-26 18:41:57 [INFO] [TRAIN] epoch=22, iter=14000/40000, loss=0.1603, lr=0.006786, batch_cost=1.0258, reader_cost=0.0006 | ETA 07:24:31 2020-11-26 18:43:40 [INFO] [TRAIN] epoch=22, iter=14100/40000, loss=0.1566, lr=0.006763, batch_cost=1.0299, reader_cost=0.0009 | ETA 07:24:34 2020-11-26 18:45:23 [INFO] [TRAIN] epoch=22, iter=14200/40000, loss=0.1722, lr=0.006739, batch_cost=1.0310, reader_cost=0.0007 | ETA 07:23:20 2020-11-26 18:47:05 [INFO] [TRAIN] epoch=22, iter=14300/40000, loss=0.1340, lr=0.006716, batch_cost=1.0259, reader_cost=0.0007 | ETA 07:19:26 2020-11-26 18:48:48 [INFO] [TRAIN] epoch=22, iter=14400/40000, loss=0.1477, lr=0.006692, batch_cost=1.0273, reader_cost=0.0007 | ETA 07:18:19 2020-11-26 18:50:31 [INFO] [TRAIN] epoch=22, iter=14500/40000, loss=0.1678, lr=0.006669, batch_cost=1.0272, reader_cost=0.0008 | ETA 07:16:32 2020-11-26 18:52:14 [INFO] [TRAIN] epoch=23, iter=14600/40000, loss=0.1271, lr=0.006645, batch_cost=1.0348, reader_cost=0.0066 | ETA 07:18:04 2020-11-26 18:53:57 [INFO] [TRAIN] epoch=23, iter=14700/40000, loss=0.1375, lr=0.006622, batch_cost=1.0245, reader_cost=0.0003 | ETA 07:12:00 2020-11-26 18:55:39 [INFO] [TRAIN] epoch=23, iter=14800/40000, loss=0.1345, lr=0.006598, batch_cost=1.0245, reader_cost=0.0004 | ETA 07:10:17 2020-11-26 18:57:22 [INFO] [TRAIN] epoch=23, iter=14900/40000, loss=0.1204, lr=0.006575, batch_cost=1.0265, reader_cost=0.0004 | ETA 07:09:24 2020-11-26 18:59:04 [INFO] [TRAIN] epoch=23, iter=15000/40000, loss=0.1364, lr=0.006551, batch_cost=1.0272, reader_cost=0.0006 | ETA 07:08:00 2020-11-26 19:00:46 [INFO] [TRAIN] epoch=23, iter=15100/40000, loss=0.1530, lr=0.006527, batch_cost=1.0193, reader_cost=0.0003 | ETA 07:03:00 2020-11-26 19:02:29 [INFO] [TRAIN] epoch=23, iter=15200/40000, loss=0.1503, lr=0.006504, batch_cost=1.0232, reader_cost=0.0009 | ETA 07:02:55 2020-11-26 19:04:12 [INFO] [TRAIN] epoch=24, iter=15300/40000, loss=0.1410, lr=0.006480, batch_cost=1.0336, reader_cost=0.0060 | ETA 07:05:29 2020-11-26 19:05:55 [INFO] [TRAIN] epoch=24, iter=15400/40000, loss=0.1497, lr=0.006457, batch_cost=1.0259, reader_cost=0.0009 | ETA 07:00:37 2020-11-26 19:07:37 [INFO] [TRAIN] epoch=24, iter=15500/40000, loss=0.1430, lr=0.006433, batch_cost=1.0272, reader_cost=0.0005 | ETA 06:59:27 2020-11-26 19:09:20 [INFO] [TRAIN] epoch=24, iter=15600/40000, loss=0.1348, lr=0.006409, batch_cost=1.0230, reader_cost=0.0007 | ETA 06:56:00 2020-11-26 19:11:02 [INFO] [TRAIN] epoch=24, iter=15700/40000, loss=0.1346, lr=0.006386, batch_cost=1.0244, reader_cost=0.0006 | ETA 06:54:52 2020-11-26 19:12:45 [INFO] [TRAIN] epoch=24, iter=15800/40000, loss=0.1598, lr=0.006362, batch_cost=1.0249, reader_cost=0.0006 | ETA 06:53:22 2020-11-26 19:14:27 [INFO] [TRAIN] epoch=25, iter=15900/40000, loss=0.1351, lr=0.006338, batch_cost=1.0267, reader_cost=0.0061 | ETA 06:52:22 2020-11-26 19:16:10 [INFO] [TRAIN] epoch=25, iter=16000/40000, loss=0.1492, lr=0.006315, batch_cost=1.0268, reader_cost=0.0006 | ETA 06:50:42 2020-11-26 19:16:10 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-26 19:16:32 [INFO] [EVAL] #Images=1449 mIoU=0.7501 Acc=0.9443 Kappa=0.8762 2020-11-26 19:16:32 [INFO] [EVAL] Class IoU: [0.9382 0.8291 0.449 0.8952 0.7006 0.7964 0.9015 0.8632 0.8864 0.3803 0.8551 0.546 0.8372 0.8551 0.855 0.8604 0.5074 0.838 0.396 0.8732 0.6882] 2020-11-26 19:16:32 [INFO] [EVAL] Class Acc: [0.9622 0.8504 0.4768 0.9377 0.7778 0.8732 0.9735 0.9188 0.9471 0.7401 0.9631 0.7345 0.9115 0.9208 0.9156 0.9216 0.9155 0.8781 0.8479 0.8995 0.8765] 2020-11-26 19:16:36 [INFO] [EVAL] The model with the best validation mIoU (0.7621) was saved at iter 12000. 2020-11-26 19:18:18 [INFO] [TRAIN] epoch=25, iter=16100/40000, loss=0.1384, lr=0.006291, batch_cost=1.0183, reader_cost=0.0006 | ETA 06:45:36 2020-11-26 19:20:00 [INFO] [TRAIN] epoch=25, iter=16200/40000, loss=0.1500, lr=0.006267, batch_cost=1.0236, reader_cost=0.0006 | ETA 06:46:01 2020-11-26 19:21:43 [INFO] [TRAIN] epoch=25, iter=16300/40000, loss=0.1592, lr=0.006244, batch_cost=1.0243, reader_cost=0.0007 | ETA 06:44:35 2020-11-26 19:23:25 [INFO] [TRAIN] epoch=25, iter=16400/40000, loss=0.1495, lr=0.006220, batch_cost=1.0214, reader_cost=0.0006 | ETA 06:41:44 2020-11-26 19:25:07 [INFO] [TRAIN] epoch=25, iter=16500/40000, loss=0.1470, lr=0.006196, batch_cost=1.0219, reader_cost=0.0006 | ETA 06:40:14 2020-11-26 19:26:49 [INFO] [TRAIN] epoch=26, iter=16600/40000, loss=0.1379, lr=0.006172, batch_cost=1.0201, reader_cost=0.0059 | ETA 06:37:49 2020-11-26 19:28:30 [INFO] [TRAIN] epoch=26, iter=16700/40000, loss=0.1424, lr=0.006149, batch_cost=1.0150, reader_cost=0.0002 | ETA 06:34:10 2020-11-26 19:30:13 [INFO] [TRAIN] epoch=26, iter=16800/40000, loss=0.1331, lr=0.006125, batch_cost=1.0250, reader_cost=0.0006 | ETA 06:36:19 2020-11-26 19:31:55 [INFO] [TRAIN] epoch=26, iter=16900/40000, loss=0.1406, lr=0.006101, batch_cost=1.0251, reader_cost=0.0006 | ETA 06:34:40 2020-11-26 19:33:38 [INFO] [TRAIN] epoch=26, iter=17000/40000, loss=0.1386, lr=0.006077, batch_cost=1.0257, reader_cost=0.0007 | ETA 06:33:10 2020-11-26 19:35:21 [INFO] [TRAIN] epoch=26, iter=17100/40000, loss=0.1400, lr=0.006054, batch_cost=1.0266, reader_cost=0.0005 | ETA 06:31:48 2020-11-26 19:37:04 [INFO] [TRAIN] epoch=27, iter=17200/40000, loss=0.1287, lr=0.006030, batch_cost=1.0344, reader_cost=0.0056 | ETA 06:33:04 2020-11-26 19:38:47 [INFO] [TRAIN] epoch=27, iter=17300/40000, loss=0.1223, lr=0.006006, batch_cost=1.0243, reader_cost=0.0004 | ETA 06:27:31 2020-11-26 19:40:29 [INFO] [TRAIN] epoch=27, iter=17400/40000, loss=0.1385, lr=0.005982, batch_cost=1.0220, reader_cost=0.0007 | ETA 06:24:56 2020-11-26 19:42:11 [INFO] [TRAIN] epoch=27, iter=17500/40000, loss=0.1371, lr=0.005958, batch_cost=1.0218, reader_cost=0.0006 | ETA 06:23:10 2020-11-26 19:43:53 [INFO] [TRAIN] epoch=27, iter=17600/40000, loss=0.1422, lr=0.005935, batch_cost=1.0233, reader_cost=0.0004 | ETA 06:22:00 2020-11-26 19:45:36 [INFO] [TRAIN] epoch=27, iter=17700/40000, loss=0.1287, lr=0.005911, batch_cost=1.0270, reader_cost=0.0008 | ETA 06:21:43 2020-11-26 19:47:19 [INFO] [TRAIN] epoch=27, iter=17800/40000, loss=0.1345, lr=0.005887, batch_cost=1.0279, reader_cost=0.0006 | ETA 06:20:19 2020-11-26 19:49:02 [INFO] [TRAIN] epoch=28, iter=17900/40000, loss=0.1294, lr=0.005863, batch_cost=1.0303, reader_cost=0.0066 | ETA 06:19:30 2020-11-26 19:50:44 [INFO] [TRAIN] epoch=28, iter=18000/40000, loss=0.1195, lr=0.005839, batch_cost=1.0258, reader_cost=0.0006 | ETA 06:16:08 2020-11-26 19:52:26 [INFO] [TRAIN] epoch=28, iter=18100/40000, loss=0.1244, lr=0.005815, batch_cost=1.0164, reader_cost=0.0002 | ETA 06:10:58 2020-11-26 19:54:07 [INFO] [TRAIN] epoch=28, iter=18200/40000, loss=0.1321, lr=0.005791, batch_cost=1.0114, reader_cost=0.0002 | ETA 06:07:28 2020-11-26 19:55:49 [INFO] [TRAIN] epoch=28, iter=18300/40000, loss=0.1345, lr=0.005767, batch_cost=1.0194, reader_cost=0.0003 | ETA 06:08:41 2020-11-26 19:57:32 [INFO] [TRAIN] epoch=28, iter=18400/40000, loss=0.1182, lr=0.005743, batch_cost=1.0245, reader_cost=0.0006 | ETA 06:08:48 2020-11-26 19:59:14 [INFO] [TRAIN] epoch=28, iter=18500/40000, loss=0.1395, lr=0.005720, batch_cost=1.0251, reader_cost=0.0008 | ETA 06:07:19 2020-11-26 20:00:57 [INFO] [TRAIN] epoch=29, iter=18600/40000, loss=0.1242, lr=0.005696, batch_cost=1.0274, reader_cost=0.0064 | ETA 06:06:26 2020-11-26 20:02:39 [INFO] [TRAIN] epoch=29, iter=18700/40000, loss=0.1267, lr=0.005672, batch_cost=1.0230, reader_cost=0.0006 | ETA 06:03:10 2020-11-26 20:04:21 [INFO] [TRAIN] epoch=29, iter=18800/40000, loss=0.1157, lr=0.005648, batch_cost=1.0237, reader_cost=0.0007 | ETA 06:01:42 2020-11-26 20:06:04 [INFO] [TRAIN] epoch=29, iter=18900/40000, loss=0.1294, lr=0.005624, batch_cost=1.0237, reader_cost=0.0007 | ETA 06:00:00 2020-11-26 20:07:46 [INFO] [TRAIN] epoch=29, iter=19000/40000, loss=0.1328, lr=0.005600, batch_cost=1.0203, reader_cost=0.0003 | ETA 05:57:06 2020-11-26 20:09:28 [INFO] [TRAIN] epoch=29, iter=19100/40000, loss=0.1105, lr=0.005576, batch_cost=1.0222, reader_cost=0.0006 | ETA 05:56:04 2020-11-26 20:11:10 [INFO] [TRAIN] epoch=30, iter=19200/40000, loss=0.1252, lr=0.005552, batch_cost=1.0240, reader_cost=0.0061 | ETA 05:54:59 2020-11-26 20:12:53 [INFO] [TRAIN] epoch=30, iter=19300/40000, loss=0.1148, lr=0.005528, batch_cost=1.0238, reader_cost=0.0009 | ETA 05:53:11 2020-11-26 20:14:36 [INFO] [TRAIN] epoch=30, iter=19400/40000, loss=0.1268, lr=0.005504, batch_cost=1.0274, reader_cost=0.0008 | ETA 05:52:44 2020-11-26 20:16:18 [INFO] [TRAIN] epoch=30, iter=19500/40000, loss=0.1227, lr=0.005480, batch_cost=1.0240, reader_cost=0.0006 | ETA 05:49:51 2020-11-26 20:18:00 [INFO] [TRAIN] epoch=30, iter=19600/40000, loss=0.1111, lr=0.005455, batch_cost=1.0169, reader_cost=0.0004 | ETA 05:45:45 2020-11-26 20:19:41 [INFO] [TRAIN] epoch=30, iter=19700/40000, loss=0.1292, lr=0.005431, batch_cost=1.0148, reader_cost=0.0005 | ETA 05:43:21 2020-11-26 20:21:24 [INFO] [TRAIN] epoch=30, iter=19800/40000, loss=0.1261, lr=0.005407, batch_cost=1.0243, reader_cost=0.0005 | ETA 05:44:50 2020-11-26 20:23:06 [INFO] [TRAIN] epoch=31, iter=19900/40000, loss=0.1155, lr=0.005383, batch_cost=1.0270, reader_cost=0.0064 | ETA 05:44:03 2020-11-26 20:24:49 [INFO] [TRAIN] epoch=31, iter=20000/40000, loss=0.1205, lr=0.005359, batch_cost=1.0261, reader_cost=0.0009 | ETA 05:42:01 2020-11-26 20:24:49 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-26 20:25:11 [INFO] [EVAL] #Images=1449 mIoU=0.7864 Acc=0.9529 Kappa=0.8961 2020-11-26 20:25:11 [INFO] [EVAL] Class IoU: [0.9471 0.8573 0.4412 0.8911 0.751 0.7633 0.954 0.901 0.9221 0.4038 0.9229 0.5702 0.8871 0.8957 0.8841 0.8695 0.6288 0.8586 0.5852 0.8791 0.7005] 2020-11-26 20:25:11 [INFO] [EVAL] Class Acc: [0.9693 0.8882 0.4546 0.9351 0.8482 0.893 0.9798 0.9544 0.9664 0.6513 0.9753 0.9412 0.9351 0.9393 0.9441 0.9389 0.8473 0.9139 0.7835 0.9109 0.8249] 2020-11-26 20:25:17 [INFO] [EVAL] The model with the best validation mIoU (0.7864) was saved at iter 20000. 2020-11-26 20:27:00 [INFO] [TRAIN] epoch=31, iter=20100/40000, loss=0.1153, lr=0.005335, batch_cost=1.0264, reader_cost=0.0007 | ETA 05:40:25 2020-11-26 20:28:42 [INFO] [TRAIN] epoch=31, iter=20200/40000, loss=0.1420, lr=0.005311, batch_cost=1.0262, reader_cost=0.0007 | ETA 05:38:39 2020-11-26 20:30:24 [INFO] [TRAIN] epoch=31, iter=20300/40000, loss=0.1168, lr=0.005287, batch_cost=1.0219, reader_cost=0.0003 | ETA 05:35:31 2020-11-26 20:32:06 [INFO] [TRAIN] epoch=31, iter=20400/40000, loss=0.1228, lr=0.005263, batch_cost=1.0182, reader_cost=0.0003 | ETA 05:32:37 2020-11-26 20:33:49 [INFO] [TRAIN] epoch=32, iter=20500/40000, loss=0.1192, lr=0.005238, batch_cost=1.0283, reader_cost=0.0067 | ETA 05:34:10 2020-11-26 20:35:31 [INFO] [TRAIN] epoch=32, iter=20600/40000, loss=0.1112, lr=0.005214, batch_cost=1.0241, reader_cost=0.0004 | ETA 05:31:07 2020-11-26 20:37:14 [INFO] [TRAIN] epoch=32, iter=20700/40000, loss=0.1073, lr=0.005190, batch_cost=1.0251, reader_cost=0.0004 | ETA 05:29:43 2020-11-26 20:38:57 [INFO] [TRAIN] epoch=32, iter=20800/40000, loss=0.1135, lr=0.005166, batch_cost=1.0299, reader_cost=0.0008 | ETA 05:29:33 2020-11-26 20:40:40 [INFO] [TRAIN] epoch=32, iter=20900/40000, loss=0.1074, lr=0.005142, batch_cost=1.0294, reader_cost=0.0010 | ETA 05:27:41 2020-11-26 20:42:23 [INFO] [TRAIN] epoch=32, iter=21000/40000, loss=0.1130, lr=0.005117, batch_cost=1.0315, reader_cost=0.0011 | ETA 05:26:37 2020-11-26 20:44:05 [INFO] [TRAIN] epoch=32, iter=21100/40000, loss=0.1078, lr=0.005093, batch_cost=1.0228, reader_cost=0.0006 | ETA 05:22:10 2020-11-26 20:45:48 [INFO] [TRAIN] epoch=33, iter=21200/40000, loss=0.1128, lr=0.005069, batch_cost=1.0258, reader_cost=0.0067 | ETA 05:21:24 2020-11-26 20:47:31 [INFO] [TRAIN] epoch=33, iter=21300/40000, loss=0.1178, lr=0.005045, batch_cost=1.0298, reader_cost=0.0009 | ETA 05:20:57 2020-11-26 20:49:14 [INFO] [TRAIN] epoch=33, iter=21400/40000, loss=0.1096, lr=0.005020, batch_cost=1.0293, reader_cost=0.0010 | ETA 05:19:04 2020-11-26 20:50:57 [INFO] [TRAIN] epoch=33, iter=21500/40000, loss=0.1146, lr=0.004996, batch_cost=1.0292, reader_cost=0.0011 | ETA 05:17:20 2020-11-26 20:52:40 [INFO] [TRAIN] epoch=33, iter=21600/40000, loss=0.1209, lr=0.004972, batch_cost=1.0290, reader_cost=0.0011 | ETA 05:15:34 2020-11-26 20:54:23 [INFO] [TRAIN] epoch=33, iter=21700/40000, loss=0.1212, lr=0.004947, batch_cost=1.0301, reader_cost=0.0007 | ETA 05:14:11 2020-11-26 20:56:05 [INFO] [TRAIN] epoch=33, iter=21800/40000, loss=0.1156, lr=0.004923, batch_cost=1.0276, reader_cost=0.0011 | ETA 05:11:43 2020-11-26 20:57:49 [INFO] [TRAIN] epoch=34, iter=21900/40000, loss=0.1170, lr=0.004899, batch_cost=1.0328, reader_cost=0.0068 | ETA 05:11:33 2020-11-26 20:59:32 [INFO] [TRAIN] epoch=34, iter=22000/40000, loss=0.1024, lr=0.004874, batch_cost=1.0285, reader_cost=0.0007 | ETA 05:08:33 2020-11-26 21:01:14 [INFO] [TRAIN] epoch=34, iter=22100/40000, loss=0.1138, lr=0.004850, batch_cost=1.0280, reader_cost=0.0005 | ETA 05:06:40 2020-11-26 21:02:57 [INFO] [TRAIN] epoch=34, iter=22200/40000, loss=0.1106, lr=0.004826, batch_cost=1.0274, reader_cost=0.0008 | ETA 05:04:47 2020-11-26 21:04:40 [INFO] [TRAIN] epoch=34, iter=22300/40000, loss=0.1100, lr=0.004801, batch_cost=1.0274, reader_cost=0.0006 | ETA 05:03:04 2020-11-26 21:06:22 [INFO] [TRAIN] epoch=34, iter=22400/40000, loss=0.1245, lr=0.004777, batch_cost=1.0263, reader_cost=0.0009 | ETA 05:01:03 2020-11-26 21:08:06 [INFO] [TRAIN] epoch=35, iter=22500/40000, loss=0.1056, lr=0.004752, batch_cost=1.0360, reader_cost=0.0065 | ETA 05:02:10 2020-11-26 21:09:48 [INFO] [TRAIN] epoch=35, iter=22600/40000, loss=0.1083, lr=0.004728, batch_cost=1.0213, reader_cost=0.0006 | ETA 04:56:10 2020-11-26 21:11:30 [INFO] [TRAIN] epoch=35, iter=22700/40000, loss=0.1009, lr=0.004703, batch_cost=1.0216, reader_cost=0.0002 | ETA 04:54:33 2020-11-26 21:13:13 [INFO] [TRAIN] epoch=35, iter=22800/40000, loss=0.1031, lr=0.004679, batch_cost=1.0245, reader_cost=0.0008 | ETA 04:53:41 2020-11-26 21:14:55 [INFO] [TRAIN] epoch=35, iter=22900/40000, loss=0.1020, lr=0.004654, batch_cost=1.0267, reader_cost=0.0009 | ETA 04:52:37 2020-11-26 21:16:38 [INFO] [TRAIN] epoch=35, iter=23000/40000, loss=0.1068, lr=0.004630, batch_cost=1.0287, reader_cost=0.0010 | ETA 04:51:27 2020-11-26 21:18:21 [INFO] [TRAIN] epoch=35, iter=23100/40000, loss=0.1039, lr=0.004605, batch_cost=1.0281, reader_cost=0.0006 | ETA 04:49:34 2020-11-26 21:20:04 [INFO] [TRAIN] epoch=36, iter=23200/40000, loss=0.0985, lr=0.004581, batch_cost=1.0315, reader_cost=0.0063 | ETA 04:48:48 2020-11-26 21:21:47 [INFO] [TRAIN] epoch=36, iter=23300/40000, loss=0.1070, lr=0.004556, batch_cost=1.0276, reader_cost=0.0006 | ETA 04:46:01 2020-11-26 21:23:29 [INFO] [TRAIN] epoch=36, iter=23400/40000, loss=0.1109, lr=0.004532, batch_cost=1.0236, reader_cost=0.0006 | ETA 04:43:12 2020-11-26 21:25:12 [INFO] [TRAIN] epoch=36, iter=23500/40000, loss=0.1045, lr=0.004507, batch_cost=1.0270, reader_cost=0.0007 | ETA 04:42:26 2020-11-26 21:26:54 [INFO] [TRAIN] epoch=36, iter=23600/40000, loss=0.1080, lr=0.004483, batch_cost=1.0233, reader_cost=0.0005 | ETA 04:39:41 2020-11-26 21:28:37 [INFO] [TRAIN] epoch=36, iter=23700/40000, loss=0.1052, lr=0.004458, batch_cost=1.0279, reader_cost=0.0009 | ETA 04:39:14 2020-11-26 21:30:20 [INFO] [TRAIN] epoch=37, iter=23800/40000, loss=0.1065, lr=0.004433, batch_cost=1.0244, reader_cost=0.0066 | ETA 04:36:35 2020-11-26 21:32:03 [INFO] [TRAIN] epoch=37, iter=23900/40000, loss=0.1119, lr=0.004409, batch_cost=1.0286, reader_cost=0.0006 | ETA 04:36:00 2020-11-26 21:33:45 [INFO] [TRAIN] epoch=37, iter=24000/40000, loss=0.1150, lr=0.004384, batch_cost=1.0233, reader_cost=0.0007 | ETA 04:32:52 2020-11-26 21:33:45 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-26 21:34:08 [INFO] [EVAL] #Images=1449 mIoU=0.7905 Acc=0.9529 Kappa=0.8962 2020-11-26 21:34:08 [INFO] [EVAL] Class IoU: [0.9461 0.8691 0.4408 0.8891 0.7233 0.8309 0.9499 0.8887 0.928 0.4223 0.9255 0.5513 0.8777 0.8922 0.8688 0.8676 0.6105 0.8977 0.5719 0.8999 0.7489] 2020-11-26 21:34:08 [INFO] [EVAL] Class Acc: [0.9691 0.8944 0.4533 0.9372 0.8098 0.9029 0.9723 0.9466 0.9602 0.6863 0.9717 0.9463 0.9144 0.9283 0.9347 0.9402 0.782 0.9496 0.7817 0.9431 0.895 ] 2020-11-26 21:34:14 [INFO] [EVAL] The model with the best validation mIoU (0.7905) was saved at iter 24000. 2020-11-26 21:35:56 [INFO] [TRAIN] epoch=37, iter=24100/40000, loss=0.0981, lr=0.004359, batch_cost=1.0220, reader_cost=0.0004 | ETA 04:30:49 2020-11-26 21:37:37 [INFO] [TRAIN] epoch=37, iter=24200/40000, loss=0.1271, lr=0.004335, batch_cost=1.0172, reader_cost=0.0005 | ETA 04:27:51 2020-11-26 21:39:19 [INFO] [TRAIN] epoch=37, iter=24300/40000, loss=0.1055, lr=0.004310, batch_cost=1.0202, reader_cost=0.0006 | ETA 04:26:57 2020-11-26 21:41:02 [INFO] [TRAIN] epoch=37, iter=24400/40000, loss=0.1021, lr=0.004285, batch_cost=1.0262, reader_cost=0.0008 | ETA 04:26:49 2020-11-26 21:42:45 [INFO] [TRAIN] epoch=38, iter=24500/40000, loss=0.1129, lr=0.004261, batch_cost=1.0278, reader_cost=0.0067 | ETA 04:25:30 2020-11-26 21:44:28 [INFO] [TRAIN] epoch=38, iter=24600/40000, loss=0.1023, lr=0.004236, batch_cost=1.0270, reader_cost=0.0010 | ETA 04:23:35 2020-11-26 21:46:10 [INFO] [TRAIN] epoch=38, iter=24700/40000, loss=0.1037, lr=0.004211, batch_cost=1.0278, reader_cost=0.0011 | ETA 04:22:04 2020-11-26 21:47:53 [INFO] [TRAIN] epoch=38, iter=24800/40000, loss=0.0990, lr=0.004186, batch_cost=1.0216, reader_cost=0.0005 | ETA 04:18:48 2020-11-26 21:49:35 [INFO] [TRAIN] epoch=38, iter=24900/40000, loss=0.0988, lr=0.004162, batch_cost=1.0224, reader_cost=0.0004 | ETA 04:17:18 2020-11-26 21:51:17 [INFO] [TRAIN] epoch=38, iter=25000/40000, loss=0.1009, lr=0.004137, batch_cost=1.0223, reader_cost=0.0007 | ETA 04:15:34 2020-11-26 21:53:00 [INFO] [TRAIN] epoch=38, iter=25100/40000, loss=0.1075, lr=0.004112, batch_cost=1.0292, reader_cost=0.0015 | ETA 04:15:34 2020-11-26 21:54:43 [INFO] [TRAIN] epoch=39, iter=25200/40000, loss=0.0951, lr=0.004087, batch_cost=1.0283, reader_cost=0.0064 | ETA 04:13:38 2020-11-26 21:56:25 [INFO] [TRAIN] epoch=39, iter=25300/40000, loss=0.1059, lr=0.004062, batch_cost=1.0247, reader_cost=0.0004 | ETA 04:11:02 2020-11-26 21:58:08 [INFO] [TRAIN] epoch=39, iter=25400/40000, loss=0.1089, lr=0.004037, batch_cost=1.0271, reader_cost=0.0012 | ETA 04:09:55 2020-11-26 21:59:51 [INFO] [TRAIN] epoch=39, iter=25500/40000, loss=0.1056, lr=0.004012, batch_cost=1.0271, reader_cost=0.0010 | ETA 04:08:13 2020-11-26 22:01:32 [INFO] [TRAIN] epoch=39, iter=25600/40000, loss=0.0933, lr=0.003987, batch_cost=1.0172, reader_cost=0.0003 | ETA 04:04:07 2020-11-26 22:03:14 [INFO] [TRAIN] epoch=39, iter=25700/40000, loss=0.1027, lr=0.003963, batch_cost=1.0208, reader_cost=0.0002 | ETA 04:03:16 2020-11-26 22:04:57 [INFO] [TRAIN] epoch=40, iter=25800/40000, loss=0.1049, lr=0.003938, batch_cost=1.0288, reader_cost=0.0075 | ETA 04:03:28 2020-11-26 22:06:40 [INFO] [TRAIN] epoch=40, iter=25900/40000, loss=0.1017, lr=0.003913, batch_cost=1.0266, reader_cost=0.0012 | ETA 04:01:14 2020-11-26 22:08:23 [INFO] [TRAIN] epoch=40, iter=26000/40000, loss=0.0985, lr=0.003888, batch_cost=1.0293, reader_cost=0.0014 | ETA 04:00:10 2020-11-26 22:10:06 [INFO] [TRAIN] epoch=40, iter=26100/40000, loss=0.1062, lr=0.003863, batch_cost=1.0264, reader_cost=0.0008 | ETA 03:57:46 2020-11-26 22:11:48 [INFO] [TRAIN] epoch=40, iter=26200/40000, loss=0.0966, lr=0.003838, batch_cost=1.0270, reader_cost=0.0010 | ETA 03:56:12 2020-11-26 22:13:31 [INFO] [TRAIN] epoch=40, iter=26300/40000, loss=0.0947, lr=0.003813, batch_cost=1.0247, reader_cost=0.0010 | ETA 03:53:58 2020-11-26 22:15:14 [INFO] [TRAIN] epoch=40, iter=26400/40000, loss=0.1021, lr=0.003788, batch_cost=1.0301, reader_cost=0.0012 | ETA 03:53:29 2020-11-26 22:16:57 [INFO] [TRAIN] epoch=41, iter=26500/40000, loss=0.0980, lr=0.003762, batch_cost=1.0352, reader_cost=0.0071 | ETA 03:52:54 2020-11-26 22:18:40 [INFO] [TRAIN] epoch=41, iter=26600/40000, loss=0.0928, lr=0.003737, batch_cost=1.0300, reader_cost=0.0014 | ETA 03:50:02 2020-11-26 22:20:23 [INFO] [TRAIN] epoch=41, iter=26700/40000, loss=0.1087, lr=0.003712, batch_cost=1.0309, reader_cost=0.0009 | ETA 03:48:30 2020-11-26 22:22:07 [INFO] [TRAIN] epoch=41, iter=26800/40000, loss=0.1000, lr=0.003687, batch_cost=1.0321, reader_cost=0.0015 | ETA 03:47:03 2020-11-26 22:23:50 [INFO] [TRAIN] epoch=41, iter=26900/40000, loss=0.1091, lr=0.003662, batch_cost=1.0332, reader_cost=0.0011 | ETA 03:45:35 2020-11-26 22:25:33 [INFO] [TRAIN] epoch=41, iter=27000/40000, loss=0.0987, lr=0.003637, batch_cost=1.0275, reader_cost=0.0013 | ETA 03:42:37 2020-11-26 22:27:15 [INFO] [TRAIN] epoch=41, iter=27100/40000, loss=0.1011, lr=0.003612, batch_cost=1.0264, reader_cost=0.0008 | ETA 03:40:40 2020-11-26 22:28:58 [INFO] [TRAIN] epoch=42, iter=27200/40000, loss=0.0899, lr=0.003586, batch_cost=1.0242, reader_cost=0.0067 | ETA 03:38:30 2020-11-26 22:30:40 [INFO] [TRAIN] epoch=42, iter=27300/40000, loss=0.0912, lr=0.003561, batch_cost=1.0233, reader_cost=0.0004 | ETA 03:36:35 2020-11-26 22:32:23 [INFO] [TRAIN] epoch=42, iter=27400/40000, loss=0.0953, lr=0.003536, batch_cost=1.0263, reader_cost=0.0011 | ETA 03:35:31 2020-11-26 22:34:06 [INFO] [TRAIN] epoch=42, iter=27500/40000, loss=0.1022, lr=0.003511, batch_cost=1.0289, reader_cost=0.0008 | ETA 03:34:21 2020-11-26 22:35:48 [INFO] [TRAIN] epoch=42, iter=27600/40000, loss=0.0895, lr=0.003485, batch_cost=1.0279, reader_cost=0.0011 | ETA 03:32:26 2020-11-26 22:37:31 [INFO] [TRAIN] epoch=42, iter=27700/40000, loss=0.0983, lr=0.003460, batch_cost=1.0307, reader_cost=0.0011 | ETA 03:31:17 2020-11-26 22:39:15 [INFO] [TRAIN] epoch=43, iter=27800/40000, loss=0.0948, lr=0.003435, batch_cost=1.0337, reader_cost=0.0072 | ETA 03:30:11 2020-11-26 22:40:57 [INFO] [TRAIN] epoch=43, iter=27900/40000, loss=0.0940, lr=0.003409, batch_cost=1.0237, reader_cost=0.0006 | ETA 03:26:26 2020-11-26 22:42:40 [INFO] [TRAIN] epoch=43, iter=28000/40000, loss=0.0945, lr=0.003384, batch_cost=1.0274, reader_cost=0.0009 | ETA 03:25:28 2020-11-26 22:42:40 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-26 22:43:02 [INFO] [EVAL] #Images=1449 mIoU=0.7919 Acc=0.9533 Kappa=0.8960 2020-11-26 22:43:02 [INFO] [EVAL] Class IoU: [0.9459 0.8736 0.4528 0.8992 0.759 0.836 0.9546 0.8981 0.9353 0.3947 0.916 0.5538 0.8906 0.897 0.8717 0.8772 0.6137 0.8964 0.5024 0.9049 0.7565] 2020-11-26 22:43:02 [INFO] [EVAL] Class Acc: [0.9646 0.9141 0.4697 0.9512 0.852 0.9064 0.9825 0.9477 0.9659 0.6354 0.9754 0.9253 0.9369 0.9398 0.9429 0.957 0.8278 0.9466 0.8156 0.958 0.922 ] 2020-11-26 22:43:08 [INFO] [EVAL] The model with the best validation mIoU (0.7919) was saved at iter 28000. 2020-11-26 22:44:50 [INFO] [TRAIN] epoch=43, iter=28100/40000, loss=0.0995, lr=0.003359, batch_cost=1.0246, reader_cost=0.0011 | ETA 03:23:12 2020-11-26 22:46:33 [INFO] [TRAIN] epoch=43, iter=28200/40000, loss=0.1045, lr=0.003333, batch_cost=1.0268, reader_cost=0.0009 | ETA 03:21:56 2020-11-26 22:48:15 [INFO] [TRAIN] epoch=43, iter=28300/40000, loss=0.0970, lr=0.003308, batch_cost=1.0270, reader_cost=0.0009 | ETA 03:20:16 2020-11-26 22:49:58 [INFO] [TRAIN] epoch=43, iter=28400/40000, loss=0.0928, lr=0.003282, batch_cost=1.0239, reader_cost=0.0009 | ETA 03:17:57 2020-11-26 22:51:41 [INFO] [TRAIN] epoch=44, iter=28500/40000, loss=0.0883, lr=0.003257, batch_cost=1.0349, reader_cost=0.0067 | ETA 03:18:21 2020-11-26 22:53:23 [INFO] [TRAIN] epoch=44, iter=28600/40000, loss=0.0927, lr=0.003231, batch_cost=1.0170, reader_cost=0.0002 | ETA 03:13:13 2020-11-26 22:55:04 [INFO] [TRAIN] epoch=44, iter=28700/40000, loss=0.1002, lr=0.003206, batch_cost=1.0109, reader_cost=0.0001 | ETA 03:10:23 2020-11-26 22:56:46 [INFO] [TRAIN] epoch=44, iter=28800/40000, loss=0.0962, lr=0.003180, batch_cost=1.0197, reader_cost=0.0003 | ETA 03:10:21 2020-11-26 22:58:29 [INFO] [TRAIN] epoch=44, iter=28900/40000, loss=0.0946, lr=0.003155, batch_cost=1.0303, reader_cost=0.0006 | ETA 03:10:36 2020-11-26 23:00:12 [INFO] [TRAIN] epoch=44, iter=29000/40000, loss=0.0984, lr=0.003129, batch_cost=1.0260, reader_cost=0.0004 | ETA 03:08:05 2020-11-26 23:01:55 [INFO] [TRAIN] epoch=45, iter=29100/40000, loss=0.0897, lr=0.003104, batch_cost=1.0311, reader_cost=0.0066 | ETA 03:07:18 2020-11-26 23:03:37 [INFO] [TRAIN] epoch=45, iter=29200/40000, loss=0.0860, lr=0.003078, batch_cost=1.0238, reader_cost=0.0009 | ETA 03:04:17 2020-11-26 23:05:20 [INFO] [TRAIN] epoch=45, iter=29300/40000, loss=0.0848, lr=0.003052, batch_cost=1.0228, reader_cost=0.0006 | ETA 03:02:23 2020-11-26 23:07:02 [INFO] [TRAIN] epoch=45, iter=29400/40000, loss=0.0945, lr=0.003027, batch_cost=1.0247, reader_cost=0.0007 | ETA 03:01:01 2020-11-26 23:08:44 [INFO] [TRAIN] epoch=45, iter=29500/40000, loss=0.0840, lr=0.003001, batch_cost=1.0243, reader_cost=0.0006 | ETA 02:59:14 2020-11-26 23:10:27 [INFO] [TRAIN] epoch=45, iter=29600/40000, loss=0.0878, lr=0.002975, batch_cost=1.0212, reader_cost=0.0010 | ETA 02:57:00 2020-11-26 23:12:09 [INFO] [TRAIN] epoch=45, iter=29700/40000, loss=0.0826, lr=0.002949, batch_cost=1.0265, reader_cost=0.0005 | ETA 02:56:13 2020-11-26 23:13:52 [INFO] [TRAIN] epoch=46, iter=29800/40000, loss=0.0852, lr=0.002924, batch_cost=1.0314, reader_cost=0.0066 | ETA 02:55:19 2020-11-26 23:15:35 [INFO] [TRAIN] epoch=46, iter=29900/40000, loss=0.0971, lr=0.002898, batch_cost=1.0240, reader_cost=0.0005 | ETA 02:52:22 2020-11-26 23:17:17 [INFO] [TRAIN] epoch=46, iter=30000/40000, loss=0.0847, lr=0.002872, batch_cost=1.0211, reader_cost=0.0010 | ETA 02:50:11 2020-11-26 23:18:59 [INFO] [TRAIN] epoch=46, iter=30100/40000, loss=0.0886, lr=0.002846, batch_cost=1.0227, reader_cost=0.0008 | ETA 02:48:44 2020-11-26 23:20:40 [INFO] [TRAIN] epoch=46, iter=30200/40000, loss=0.0843, lr=0.002820, batch_cost=1.0121, reader_cost=0.0002 | ETA 02:45:19 2020-11-26 23:22:22 [INFO] [TRAIN] epoch=46, iter=30300/40000, loss=0.0877, lr=0.002794, batch_cost=1.0200, reader_cost=0.0007 | ETA 02:44:53 2020-11-26 23:24:05 [INFO] [TRAIN] epoch=46, iter=30400/40000, loss=0.0919, lr=0.002768, batch_cost=1.0219, reader_cost=0.0007 | ETA 02:43:30 2020-11-26 23:25:48 [INFO] [TRAIN] epoch=47, iter=30500/40000, loss=0.0954, lr=0.002742, batch_cost=1.0303, reader_cost=0.0070 | ETA 02:43:07 2020-11-26 23:27:30 [INFO] [TRAIN] epoch=47, iter=30600/40000, loss=0.0873, lr=0.002716, batch_cost=1.0233, reader_cost=0.0008 | ETA 02:40:18 2020-11-26 23:29:12 [INFO] [TRAIN] epoch=47, iter=30700/40000, loss=0.0834, lr=0.002690, batch_cost=1.0196, reader_cost=0.0008 | ETA 02:38:02 2020-11-26 23:30:54 [INFO] [TRAIN] epoch=47, iter=30800/40000, loss=0.0883, lr=0.002664, batch_cost=1.0224, reader_cost=0.0008 | ETA 02:36:46 2020-11-26 23:32:36 [INFO] [TRAIN] epoch=47, iter=30900/40000, loss=0.0855, lr=0.002638, batch_cost=1.0236, reader_cost=0.0004 | ETA 02:35:14 2020-11-26 23:34:19 [INFO] [TRAIN] epoch=47, iter=31000/40000, loss=0.0872, lr=0.002612, batch_cost=1.0223, reader_cost=0.0006 | ETA 02:33:21 2020-11-26 23:36:02 [INFO] [TRAIN] epoch=48, iter=31100/40000, loss=0.0882, lr=0.002586, batch_cost=1.0359, reader_cost=0.0066 | ETA 02:33:39 2020-11-26 23:37:45 [INFO] [TRAIN] epoch=48, iter=31200/40000, loss=0.0841, lr=0.002560, batch_cost=1.0279, reader_cost=0.0010 | ETA 02:30:45 2020-11-26 23:39:27 [INFO] [TRAIN] epoch=48, iter=31300/40000, loss=0.0861, lr=0.002534, batch_cost=1.0225, reader_cost=0.0009 | ETA 02:28:15 2020-11-26 23:41:10 [INFO] [TRAIN] epoch=48, iter=31400/40000, loss=0.0835, lr=0.002507, batch_cost=1.0253, reader_cost=0.0008 | ETA 02:26:57 2020-11-26 23:42:52 [INFO] [TRAIN] epoch=48, iter=31500/40000, loss=0.0842, lr=0.002481, batch_cost=1.0248, reader_cost=0.0007 | ETA 02:25:10 2020-11-26 23:44:34 [INFO] [TRAIN] epoch=48, iter=31600/40000, loss=0.0910, lr=0.002455, batch_cost=1.0210, reader_cost=0.0006 | ETA 02:22:56 2020-11-26 23:46:16 [INFO] [TRAIN] epoch=48, iter=31700/40000, loss=0.0887, lr=0.002429, batch_cost=1.0163, reader_cost=0.0003 | ETA 02:20:35 2020-11-26 23:47:59 [INFO] [TRAIN] epoch=49, iter=31800/40000, loss=0.0881, lr=0.002402, batch_cost=1.0264, reader_cost=0.0068 | ETA 02:20:16 2020-11-26 23:49:41 [INFO] [TRAIN] epoch=49, iter=31900/40000, loss=0.0866, lr=0.002376, batch_cost=1.0241, reader_cost=0.0003 | ETA 02:18:14 2020-11-26 23:51:24 [INFO] [TRAIN] epoch=49, iter=32000/40000, loss=0.0877, lr=0.002350, batch_cost=1.0251, reader_cost=0.0005 | ETA 02:16:40 2020-11-26 23:51:24 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-26 23:51:46 [INFO] [EVAL] #Images=1449 mIoU=0.7934 Acc=0.9541 Kappa=0.8981 2020-11-26 23:51:46 [INFO] [EVAL] Class IoU: [0.9476 0.8645 0.4505 0.8986 0.7494 0.8331 0.9577 0.8984 0.9349 0.4359 0.8923 0.5771 0.891 0.8966 0.8751 0.8736 0.6173 0.8785 0.5239 0.9088 0.757 ] 2020-11-26 23:51:46 [INFO] [EVAL] Class Acc: [0.9676 0.905 0.4709 0.9436 0.8317 0.8892 0.9825 0.951 0.9596 0.6358 0.9799 0.9182 0.9401 0.9389 0.9399 0.9488 0.8163 0.9258 0.8146 0.9551 0.9354] 2020-11-26 23:51:52 [INFO] [EVAL] The model with the best validation mIoU (0.7934) was saved at iter 32000. 2020-11-26 23:53:34 [INFO] [TRAIN] epoch=49, iter=32100/40000, loss=0.0893, lr=0.002323, batch_cost=1.0275, reader_cost=0.0004 | ETA 02:15:16 2020-11-26 23:55:17 [INFO] [TRAIN] epoch=49, iter=32200/40000, loss=0.0911, lr=0.002297, batch_cost=1.0217, reader_cost=0.0003 | ETA 02:12:49 2020-11-26 23:56:59 [INFO] [TRAIN] epoch=49, iter=32300/40000, loss=0.0877, lr=0.002270, batch_cost=1.0224, reader_cost=0.0005 | ETA 02:11:12 2020-11-26 23:58:42 [INFO] [TRAIN] epoch=50, iter=32400/40000, loss=0.0887, lr=0.002244, batch_cost=1.0282, reader_cost=0.0065 | ETA 02:10:14 2020-11-27 00:00:24 [INFO] [TRAIN] epoch=50, iter=32500/40000, loss=0.0893, lr=0.002217, batch_cost=1.0228, reader_cost=0.0009 | ETA 02:07:51 2020-11-27 00:02:07 [INFO] [TRAIN] epoch=50, iter=32600/40000, loss=0.0940, lr=0.002190, batch_cost=1.0258, reader_cost=0.0008 | ETA 02:06:31 2020-11-27 00:03:49 [INFO] [TRAIN] epoch=50, iter=32700/40000, loss=0.0860, lr=0.002164, batch_cost=1.0251, reader_cost=0.0005 | ETA 02:04:42 2020-11-27 00:05:32 [INFO] [TRAIN] epoch=50, iter=32800/40000, loss=0.0850, lr=0.002137, batch_cost=1.0289, reader_cost=0.0008 | ETA 02:03:27 2020-11-27 00:07:15 [INFO] [TRAIN] epoch=50, iter=32900/40000, loss=0.0863, lr=0.002110, batch_cost=1.0256, reader_cost=0.0007 | ETA 02:01:22 2020-11-27 00:08:57 [INFO] [TRAIN] epoch=50, iter=33000/40000, loss=0.0881, lr=0.002083, batch_cost=1.0258, reader_cost=0.0009 | ETA 01:59:40 2020-11-27 00:10:40 [INFO] [TRAIN] epoch=51, iter=33100/40000, loss=0.0872, lr=0.002057, batch_cost=1.0324, reader_cost=0.0059 | ETA 01:58:43 2020-11-27 00:12:22 [INFO] [TRAIN] epoch=51, iter=33200/40000, loss=0.0901, lr=0.002030, batch_cost=1.0181, reader_cost=0.0004 | ETA 01:55:23 2020-11-27 00:14:04 [INFO] [TRAIN] epoch=51, iter=33300/40000, loss=0.0798, lr=0.002003, batch_cost=1.0183, reader_cost=0.0006 | ETA 01:53:42 2020-11-27 00:15:47 [INFO] [TRAIN] epoch=51, iter=33400/40000, loss=0.0847, lr=0.001976, batch_cost=1.0279, reader_cost=0.0005 | ETA 01:53:04 2020-11-27 00:17:29 [INFO] [TRAIN] epoch=51, iter=33500/40000, loss=0.0861, lr=0.001949, batch_cost=1.0243, reader_cost=0.0009 | ETA 01:50:58 2020-11-27 00:19:12 [INFO] [TRAIN] epoch=51, iter=33600/40000, loss=0.0891, lr=0.001922, batch_cost=1.0232, reader_cost=0.0004 | ETA 01:49:08 2020-11-27 00:20:54 [INFO] [TRAIN] epoch=51, iter=33700/40000, loss=0.0840, lr=0.001895, batch_cost=1.0215, reader_cost=0.0006 | ETA 01:47:15 2020-11-27 00:22:36 [INFO] [TRAIN] epoch=52, iter=33800/40000, loss=0.0794, lr=0.001868, batch_cost=1.0273, reader_cost=0.0066 | ETA 01:46:09 2020-11-27 00:24:19 [INFO] [TRAIN] epoch=52, iter=33900/40000, loss=0.0880, lr=0.001841, batch_cost=1.0243, reader_cost=0.0003 | ETA 01:44:08 2020-11-27 00:26:01 [INFO] [TRAIN] epoch=52, iter=34000/40000, loss=0.0825, lr=0.001814, batch_cost=1.0210, reader_cost=0.0003 | ETA 01:42:05 2020-11-27 00:27:43 [INFO] [TRAIN] epoch=52, iter=34100/40000, loss=0.0781, lr=0.001786, batch_cost=1.0238, reader_cost=0.0003 | ETA 01:40:40 2020-11-27 00:29:26 [INFO] [TRAIN] epoch=52, iter=34200/40000, loss=0.0842, lr=0.001759, batch_cost=1.0232, reader_cost=0.0006 | ETA 01:38:54 2020-11-27 00:31:08 [INFO] [TRAIN] epoch=52, iter=34300/40000, loss=0.0824, lr=0.001732, batch_cost=1.0223, reader_cost=0.0005 | ETA 01:37:07 2020-11-27 00:32:51 [INFO] [TRAIN] epoch=53, iter=34400/40000, loss=0.0898, lr=0.001704, batch_cost=1.0296, reader_cost=0.0063 | ETA 01:36:05 2020-11-27 00:34:33 [INFO] [TRAIN] epoch=53, iter=34500/40000, loss=0.0803, lr=0.001677, batch_cost=1.0195, reader_cost=0.0004 | ETA 01:33:27 2020-11-27 00:36:15 [INFO] [TRAIN] epoch=53, iter=34600/40000, loss=0.0864, lr=0.001650, batch_cost=1.0184, reader_cost=0.0006 | ETA 01:31:39 2020-11-27 00:37:56 [INFO] [TRAIN] epoch=53, iter=34700/40000, loss=0.0862, lr=0.001622, batch_cost=1.0128, reader_cost=0.0002 | ETA 01:29:27 2020-11-27 00:39:38 [INFO] [TRAIN] epoch=53, iter=34800/40000, loss=0.0802, lr=0.001594, batch_cost=1.0183, reader_cost=0.0002 | ETA 01:28:15 2020-11-27 00:41:20 [INFO] [TRAIN] epoch=53, iter=34900/40000, loss=0.0847, lr=0.001567, batch_cost=1.0214, reader_cost=0.0005 | ETA 01:26:49 2020-11-27 00:43:02 [INFO] [TRAIN] epoch=53, iter=35000/40000, loss=0.0893, lr=0.001539, batch_cost=1.0252, reader_cost=0.0008 | ETA 01:25:25 2020-11-27 00:44:45 [INFO] [TRAIN] epoch=54, iter=35100/40000, loss=0.0797, lr=0.001511, batch_cost=1.0266, reader_cost=0.0063 | ETA 01:23:50 2020-11-27 00:46:27 [INFO] [TRAIN] epoch=54, iter=35200/40000, loss=0.0769, lr=0.001484, batch_cost=1.0194, reader_cost=0.0003 | ETA 01:21:33 2020-11-27 00:48:09 [INFO] [TRAIN] epoch=54, iter=35300/40000, loss=0.0821, lr=0.001456, batch_cost=1.0160, reader_cost=0.0005 | ETA 01:19:35 2020-11-27 00:49:51 [INFO] [TRAIN] epoch=54, iter=35400/40000, loss=0.0891, lr=0.001428, batch_cost=1.0217, reader_cost=0.0007 | ETA 01:18:19 2020-11-27 00:51:33 [INFO] [TRAIN] epoch=54, iter=35500/40000, loss=0.0802, lr=0.001400, batch_cost=1.0211, reader_cost=0.0004 | ETA 01:16:34 2020-11-27 00:53:15 [INFO] [TRAIN] epoch=54, iter=35600/40000, loss=0.0930, lr=0.001372, batch_cost=1.0227, reader_cost=0.0003 | ETA 01:14:59 2020-11-27 00:54:58 [INFO] [TRAIN] epoch=55, iter=35700/40000, loss=0.0819, lr=0.001344, batch_cost=1.0254, reader_cost=0.0063 | ETA 01:13:29 2020-11-27 00:56:40 [INFO] [TRAIN] epoch=55, iter=35800/40000, loss=0.0837, lr=0.001316, batch_cost=1.0234, reader_cost=0.0006 | ETA 01:11:38 2020-11-27 00:58:22 [INFO] [TRAIN] epoch=55, iter=35900/40000, loss=0.0836, lr=0.001287, batch_cost=1.0231, reader_cost=0.0005 | ETA 01:09:54 2020-11-27 01:00:05 [INFO] [TRAIN] epoch=55, iter=36000/40000, loss=0.0799, lr=0.001259, batch_cost=1.0245, reader_cost=0.0008 | ETA 01:08:17 2020-11-27 01:00:05 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-27 01:00:27 [INFO] [EVAL] #Images=1449 mIoU=0.7945 Acc=0.9540 Kappa=0.8976 2020-11-27 01:00:27 [INFO] [EVAL] Class IoU: [0.9463 0.8774 0.4467 0.8993 0.757 0.8331 0.9588 0.8943 0.9366 0.399 0.9268 0.5544 0.8861 0.9039 0.8788 0.8762 0.6296 0.8974 0.5192 0.9176 0.7453] 2020-11-27 01:00:27 [INFO] [EVAL] Class Acc: [0.9658 0.917 0.4611 0.9456 0.8649 0.8934 0.9817 0.9459 0.9622 0.6757 0.9762 0.9324 0.9264 0.9419 0.9395 0.9464 0.8068 0.9564 0.8344 0.9592 0.9477] 2020-11-27 01:00:33 [INFO] [EVAL] The model with the best validation mIoU (0.7945) was saved at iter 36000. 2020-11-27 01:02:15 [INFO] [TRAIN] epoch=55, iter=36100/40000, loss=0.0798, lr=0.001231, batch_cost=1.0266, reader_cost=0.0007 | ETA 01:06:43 2020-11-27 01:03:56 [INFO] [TRAIN] epoch=55, iter=36200/40000, loss=0.0852, lr=0.001202, batch_cost=1.0120, reader_cost=0.0005 | ETA 01:04:05 2020-11-27 01:05:39 [INFO] [TRAIN] epoch=55, iter=36300/40000, loss=0.0873, lr=0.001174, batch_cost=1.0215, reader_cost=0.0003 | ETA 01:02:59 2020-11-27 01:07:21 [INFO] [TRAIN] epoch=56, iter=36400/40000, loss=0.0759, lr=0.001145, batch_cost=1.0277, reader_cost=0.0068 | ETA 01:01:39 2020-11-27 01:09:04 [INFO] [TRAIN] epoch=56, iter=36500/40000, loss=0.0815, lr=0.001117, batch_cost=1.0240, reader_cost=0.0005 | ETA 00:59:44 2020-11-27 01:10:46 [INFO] [TRAIN] epoch=56, iter=36600/40000, loss=0.0907, lr=0.001088, batch_cost=1.0259, reader_cost=0.0007 | ETA 00:58:08 2020-11-27 01:12:28 [INFO] [TRAIN] epoch=56, iter=36700/40000, loss=0.0818, lr=0.001059, batch_cost=1.0185, reader_cost=0.0004 | ETA 00:56:01 2020-11-27 01:14:11 [INFO] [TRAIN] epoch=56, iter=36800/40000, loss=0.0751, lr=0.001030, batch_cost=1.0240, reader_cost=0.0005 | ETA 00:54:36 2020-11-27 01:15:53 [INFO] [TRAIN] epoch=56, iter=36900/40000, loss=0.0860, lr=0.001001, batch_cost=1.0262, reader_cost=0.0004 | ETA 00:53:01 2020-11-27 01:17:36 [INFO] [TRAIN] epoch=56, iter=37000/40000, loss=0.0902, lr=0.000972, batch_cost=1.0239, reader_cost=0.0007 | ETA 00:51:11 2020-11-27 01:19:18 [INFO] [TRAIN] epoch=57, iter=37100/40000, loss=0.0817, lr=0.000943, batch_cost=1.0286, reader_cost=0.0067 | ETA 00:49:42 2020-11-27 01:21:01 [INFO] [TRAIN] epoch=57, iter=37200/40000, loss=0.0830, lr=0.000914, batch_cost=1.0217, reader_cost=0.0007 | ETA 00:47:40 2020-11-27 01:22:43 [INFO] [TRAIN] epoch=57, iter=37300/40000, loss=0.0785, lr=0.000884, batch_cost=1.0253, reader_cost=0.0003 | ETA 00:46:08 2020-11-27 01:24:25 [INFO] [TRAIN] epoch=57, iter=37400/40000, loss=0.0867, lr=0.000855, batch_cost=1.0145, reader_cost=0.0004 | ETA 00:43:57 2020-11-27 01:26:07 [INFO] [TRAIN] epoch=57, iter=37500/40000, loss=0.0815, lr=0.000825, batch_cost=1.0197, reader_cost=0.0003 | ETA 00:42:29 2020-11-27 01:27:49 [INFO] [TRAIN] epoch=57, iter=37600/40000, loss=0.0873, lr=0.000795, batch_cost=1.0245, reader_cost=0.0004 | ETA 00:40:58 2020-11-27 01:29:31 [INFO] [TRAIN] epoch=58, iter=37700/40000, loss=0.0806, lr=0.000765, batch_cost=1.0202, reader_cost=0.0061 | ETA 00:39:06 2020-11-27 01:31:13 [INFO] [TRAIN] epoch=58, iter=37800/40000, loss=0.0812, lr=0.000735, batch_cost=1.0164, reader_cost=0.0002 | ETA 00:37:16 2020-11-27 01:32:55 [INFO] [TRAIN] epoch=58, iter=37900/40000, loss=0.0762, lr=0.000705, batch_cost=1.0275, reader_cost=0.0009 | ETA 00:35:57 2020-11-27 01:34:38 [INFO] [TRAIN] epoch=58, iter=38000/40000, loss=0.0931, lr=0.000675, batch_cost=1.0230, reader_cost=0.0004 | ETA 00:34:06 2020-11-27 01:36:20 [INFO] [TRAIN] epoch=58, iter=38100/40000, loss=0.0765, lr=0.000645, batch_cost=1.0236, reader_cost=0.0004 | ETA 00:32:24 2020-11-27 01:38:02 [INFO] [TRAIN] epoch=58, iter=38200/40000, loss=0.0795, lr=0.000614, batch_cost=1.0221, reader_cost=0.0004 | ETA 00:30:39 2020-11-27 01:39:45 [INFO] [TRAIN] epoch=58, iter=38300/40000, loss=0.0831, lr=0.000583, batch_cost=1.0260, reader_cost=0.0003 | ETA 00:29:04 2020-11-27 01:41:28 [INFO] [TRAIN] epoch=59, iter=38400/40000, loss=0.0785, lr=0.000552, batch_cost=1.0296, reader_cost=0.0059 | ETA 00:27:27 2020-11-27 01:43:11 [INFO] [TRAIN] epoch=59, iter=38500/40000, loss=0.0818, lr=0.000521, batch_cost=1.0271, reader_cost=0.0003 | ETA 00:25:40 2020-11-27 01:44:53 [INFO] [TRAIN] epoch=59, iter=38600/40000, loss=0.0816, lr=0.000490, batch_cost=1.0240, reader_cost=0.0005 | ETA 00:23:53 2020-11-27 01:46:35 [INFO] [TRAIN] epoch=59, iter=38700/40000, loss=0.0783, lr=0.000458, batch_cost=1.0243, reader_cost=0.0005 | ETA 00:22:11 2020-11-27 01:48:18 [INFO] [TRAIN] epoch=59, iter=38800/40000, loss=0.0867, lr=0.000426, batch_cost=1.0291, reader_cost=0.0003 | ETA 00:20:34 2020-11-27 01:50:01 [INFO] [TRAIN] epoch=59, iter=38900/40000, loss=0.0842, lr=0.000394, batch_cost=1.0276, reader_cost=0.0004 | ETA 00:18:50 2020-11-27 01:51:44 [INFO] [TRAIN] epoch=60, iter=39000/40000, loss=0.0787, lr=0.000362, batch_cost=1.0333, reader_cost=0.0057 | ETA 00:17:13 2020-11-27 01:53:27 [INFO] [TRAIN] epoch=60, iter=39100/40000, loss=0.0829, lr=0.000329, batch_cost=1.0276, reader_cost=0.0006 | ETA 00:15:24 2020-11-27 01:55:09 [INFO] [TRAIN] epoch=60, iter=39200/40000, loss=0.0842, lr=0.000296, batch_cost=1.0142, reader_cost=0.0003 | ETA 00:13:31 2020-11-27 01:56:50 [INFO] [TRAIN] epoch=60, iter=39300/40000, loss=0.0883, lr=0.000263, batch_cost=1.0175, reader_cost=0.0002 | ETA 00:11:52 2020-11-27 01:58:33 [INFO] [TRAIN] epoch=60, iter=39400/40000, loss=0.0818, lr=0.000229, batch_cost=1.0257, reader_cost=0.0005 | ETA 00:10:15 2020-11-27 02:00:16 [INFO] [TRAIN] epoch=60, iter=39500/40000, loss=0.0793, lr=0.000194, batch_cost=1.0281, reader_cost=0.0009 | ETA 00:08:34 2020-11-27 02:01:58 [INFO] [TRAIN] epoch=60, iter=39600/40000, loss=0.0849, lr=0.000159, batch_cost=1.0267, reader_cost=0.0013 | ETA 00:06:50 2020-11-27 02:03:42 [INFO] [TRAIN] epoch=61, iter=39700/40000, loss=0.0914, lr=0.000123, batch_cost=1.0335, reader_cost=0.0064 | ETA 00:05:10 2020-11-27 02:05:25 [INFO] [TRAIN] epoch=61, iter=39800/40000, loss=0.0805, lr=0.000085, batch_cost=1.0282, reader_cost=0.0003 | ETA 00:03:25 2020-11-27 02:07:07 [INFO] [TRAIN] epoch=61, iter=39900/40000, loss=0.0829, lr=0.000046, batch_cost=1.0270, reader_cost=0.0006 | ETA 00:01:42 2020-11-27 02:08:50 [INFO] [TRAIN] epoch=61, iter=40000/40000, loss=0.0877, lr=0.000001, batch_cost=1.0265, reader_cost=0.0007 | ETA 00:00:00 2020-11-27 02:08:50 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-27 02:09:13 [INFO] [EVAL] #Images=1449 mIoU=0.7976 Acc=0.9551 Kappa=0.9001 2020-11-27 02:09:13 [INFO] [EVAL] Class IoU: [0.9479 0.88 0.4503 0.9006 0.7484 0.837 0.959 0.8965 0.9421 0.417 0.9305 0.5571 0.8974 0.9069 0.8772 0.8782 0.624 0.8988 0.5276 0.9156 0.7567] 2020-11-27 02:09:13 [INFO] [EVAL] Class Acc: [0.9667 0.9198 0.467 0.9451 0.8516 0.8961 0.9825 0.9544 0.9708 0.6728 0.976 0.9298 0.9358 0.9521 0.9447 0.9488 0.8209 0.9467 0.8146 0.9612 0.9342] 2020-11-27 02:09:19 [INFO] [EVAL] The model with the best validation mIoU (0.7976) was saved at iter 40000.