2020-11-24 16:31:20 [INFO] ------------Environment Information------------- platform: Linux-3.10.0_3-0-0-34-x86_64-with-centos-7.5.1804-Core Python: 3.7.9 (default, Aug 31 2020, 12:42:55) [GCC 7.3.0] Paddle compiled with cuda: True NVCC: Cuda compilation tools, release 10.2, V10.2.89 cudnn: 7.6 GPUs used: 4 CUDA_VISIBLE_DEVICES: 0,1,2,3 GPU: ['GPU 0: Tesla V100-SXM2-16GB', 'GPU 1: Tesla V100-SXM2-16GB', 'GPU 2: Tesla V100-SXM2-16GB', 'GPU 3: Tesla V100-SXM2-16GB', 'GPU 4: Tesla V100-SXM2-16GB', 'GPU 5: Tesla V100-SXM2-16GB', 'GPU 6: Tesla V100-SXM2-16GB', 'GPU 7: Tesla V100-SXM2-16GB'] GCC: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39) PaddlePaddle: 2.0.0-rc0 OpenCV: 4.1.0 ------------------------------------------------ 2020-11-24 16:31:21 [INFO] ---------------Config Information--------------- batch_size: 4 iters: 40000 learning_rate: decay: end_lr: 0.0 power: 0.9 type: poly value: 0.01 loss: coef: - 1 - 0.4 types: - ignore_index: 255 type: CrossEntropyLoss model: align_corners: false backbone: output_stride: 8 pretrained: https://bj.bcebos.com/paddleseg/dygraph/resnet50_vd_ssld_v2.tar.gz type: ResNet50_vd backbone_indices: - 2 - 3 enable_auxiliary_loss: true inter_channels: 512 key_value_channels: 256 pretrained: null psp_size: - 1 - 3 - 6 - 8 type: ANN optimizer: momentum: 0.9 type: sgd weight_decay: 4.0e-05 train_dataset: dataset_root: data/VOCdevkit/ mode: trainaug transforms: - max_scale_factor: 2.0 min_scale_factor: 0.5 scale_step_size: 0.25 type: ResizeStepScaling - crop_size: - 512 - 512 type: RandomPaddingCrop - type: RandomHorizontalFlip - brightness_range: 0.4 contrast_range: 0.4 saturation_range: 0.4 type: RandomDistort - type: Normalize type: PascalVOC val_dataset: dataset_root: data/VOCdevkit/ mode: val transforms: - target_size: - 512 - 512 type: Padding - type: Normalize type: PascalVOC ------------------------------------------------ 2020-11-24 16:31:25 [INFO] Loading pretrained model from https://bj.bcebos.com/paddleseg/dygraph/resnet50_vd_ssld_v2.tar.gz 2020-11-24 16:31:27 [INFO] There are 275/275 variables loaded into ResNet_vd. 2020-11-24 16:32:28 [INFO] [TRAIN] epoch=1, iter=100/40000, loss=1.4913, lr=0.009978, batch_cost=0.5482, reader_cost=0.0128 | ETA 06:04:32 2020-11-24 16:33:19 [INFO] [TRAIN] epoch=1, iter=200/40000, loss=1.0972, lr=0.009955, batch_cost=0.5096, reader_cost=0.0007 | ETA 05:38:03 2020-11-24 16:34:10 [INFO] [TRAIN] epoch=1, iter=300/40000, loss=1.0550, lr=0.009933, batch_cost=0.5100, reader_cost=0.0005 | ETA 05:37:27 2020-11-24 16:35:01 [INFO] [TRAIN] epoch=1, iter=400/40000, loss=0.9799, lr=0.009910, batch_cost=0.5128, reader_cost=0.0009 | ETA 05:38:27 2020-11-24 16:35:53 [INFO] [TRAIN] epoch=1, iter=500/40000, loss=0.8463, lr=0.009888, batch_cost=0.5124, reader_cost=0.0015 | ETA 05:37:19 2020-11-24 16:36:44 [INFO] [TRAIN] epoch=1, iter=600/40000, loss=0.8742, lr=0.009865, batch_cost=0.5116, reader_cost=0.0007 | ETA 05:35:55 2020-11-24 16:37:36 [INFO] [TRAIN] epoch=2, iter=700/40000, loss=0.8237, lr=0.009843, batch_cost=0.5240, reader_cost=0.0061 | ETA 05:43:12 2020-11-24 16:38:27 [INFO] [TRAIN] epoch=2, iter=800/40000, loss=0.8151, lr=0.009820, batch_cost=0.5130, reader_cost=0.0008 | ETA 05:35:10 2020-11-24 16:39:19 [INFO] [TRAIN] epoch=2, iter=900/40000, loss=0.7847, lr=0.009797, batch_cost=0.5120, reader_cost=0.0011 | ETA 05:33:39 2020-11-24 16:40:10 [INFO] [TRAIN] epoch=2, iter=1000/40000, loss=0.8060, lr=0.009775, batch_cost=0.5130, reader_cost=0.0011 | ETA 05:33:25 2020-11-24 16:41:01 [INFO] [TRAIN] epoch=2, iter=1100/40000, loss=0.7767, lr=0.009752, batch_cost=0.5124, reader_cost=0.0013 | ETA 05:32:14 2020-11-24 16:41:53 [INFO] [TRAIN] epoch=2, iter=1200/40000, loss=0.6684, lr=0.009730, batch_cost=0.5131, reader_cost=0.0010 | ETA 05:31:47 2020-11-24 16:42:44 [INFO] [TRAIN] epoch=2, iter=1300/40000, loss=0.7168, lr=0.009707, batch_cost=0.5158, reader_cost=0.0012 | ETA 05:32:42 2020-11-24 16:43:36 [INFO] [TRAIN] epoch=3, iter=1400/40000, loss=0.7203, lr=0.009685, batch_cost=0.5198, reader_cost=0.0075 | ETA 05:34:25 2020-11-24 16:44:28 [INFO] [TRAIN] epoch=3, iter=1500/40000, loss=0.7534, lr=0.009662, batch_cost=0.5146, reader_cost=0.0012 | ETA 05:30:11 2020-11-24 16:45:19 [INFO] [TRAIN] epoch=3, iter=1600/40000, loss=0.6163, lr=0.009639, batch_cost=0.5142, reader_cost=0.0013 | ETA 05:29:04 2020-11-24 16:46:10 [INFO] [TRAIN] epoch=3, iter=1700/40000, loss=0.5877, lr=0.009617, batch_cost=0.5147, reader_cost=0.0013 | ETA 05:28:34 2020-11-24 16:47:02 [INFO] [TRAIN] epoch=3, iter=1800/40000, loss=0.7408, lr=0.009594, batch_cost=0.5149, reader_cost=0.0013 | ETA 05:27:48 2020-11-24 16:47:54 [INFO] [TRAIN] epoch=3, iter=1900/40000, loss=0.6015, lr=0.009572, batch_cost=0.5171, reader_cost=0.0009 | ETA 05:28:23 2020-11-24 16:48:46 [INFO] [TRAIN] epoch=4, iter=2000/40000, loss=0.6595, lr=0.009549, batch_cost=0.5219, reader_cost=0.0071 | ETA 05:30:33 2020-11-24 16:49:38 [INFO] [TRAIN] epoch=4, iter=2100/40000, loss=0.6079, lr=0.009526, batch_cost=0.5167, reader_cost=0.0015 | ETA 05:26:24 2020-11-24 16:50:29 [INFO] [TRAIN] epoch=4, iter=2200/40000, loss=0.6254, lr=0.009504, batch_cost=0.5123, reader_cost=0.0012 | ETA 05:22:43 2020-11-24 16:51:20 [INFO] [TRAIN] epoch=4, iter=2300/40000, loss=0.6144, lr=0.009481, batch_cost=0.5125, reader_cost=0.0012 | ETA 05:21:59 2020-11-24 16:52:11 [INFO] [TRAIN] epoch=4, iter=2400/40000, loss=0.5977, lr=0.009459, batch_cost=0.5144, reader_cost=0.0013 | ETA 05:22:22 2020-11-24 16:53:03 [INFO] [TRAIN] epoch=4, iter=2500/40000, loss=0.5871, lr=0.009436, batch_cost=0.5156, reader_cost=0.0015 | ETA 05:22:13 2020-11-24 16:53:54 [INFO] [TRAIN] epoch=4, iter=2600/40000, loss=0.5381, lr=0.009413, batch_cost=0.5145, reader_cost=0.0011 | ETA 05:20:43 2020-11-24 16:54:47 [INFO] [TRAIN] epoch=5, iter=2700/40000, loss=0.6042, lr=0.009391, batch_cost=0.5230, reader_cost=0.0061 | ETA 05:25:07 2020-11-24 16:55:38 [INFO] [TRAIN] epoch=5, iter=2800/40000, loss=0.5419, lr=0.009368, batch_cost=0.5142, reader_cost=0.0013 | ETA 05:18:46 2020-11-24 16:56:29 [INFO] [TRAIN] epoch=5, iter=2900/40000, loss=0.5543, lr=0.009345, batch_cost=0.5117, reader_cost=0.0012 | ETA 05:16:24 2020-11-24 16:57:21 [INFO] [TRAIN] epoch=5, iter=3000/40000, loss=0.5216, lr=0.009323, batch_cost=0.5128, reader_cost=0.0012 | ETA 05:16:11 2020-11-24 16:58:12 [INFO] [TRAIN] epoch=5, iter=3100/40000, loss=0.6758, lr=0.009300, batch_cost=0.5131, reader_cost=0.0015 | ETA 05:15:34 2020-11-24 16:59:04 [INFO] [TRAIN] epoch=5, iter=3200/40000, loss=0.6116, lr=0.009277, batch_cost=0.5180, reader_cost=0.0012 | ETA 05:17:42 2020-11-24 16:59:55 [INFO] [TRAIN] epoch=5, iter=3300/40000, loss=0.4995, lr=0.009255, batch_cost=0.5112, reader_cost=0.0013 | ETA 05:12:42 2020-11-24 17:00:47 [INFO] [TRAIN] epoch=6, iter=3400/40000, loss=0.4878, lr=0.009232, batch_cost=0.5167, reader_cost=0.0062 | ETA 05:15:09 2020-11-24 17:01:38 [INFO] [TRAIN] epoch=6, iter=3500/40000, loss=0.5211, lr=0.009209, batch_cost=0.5171, reader_cost=0.0015 | ETA 05:14:32 2020-11-24 17:02:29 [INFO] [TRAIN] epoch=6, iter=3600/40000, loss=0.5802, lr=0.009186, batch_cost=0.5112, reader_cost=0.0007 | ETA 05:10:06 2020-11-24 17:03:22 [INFO] [TRAIN] epoch=6, iter=3700/40000, loss=0.5861, lr=0.009164, batch_cost=0.5232, reader_cost=0.0010 | ETA 05:16:33 2020-11-24 17:04:13 [INFO] [TRAIN] epoch=6, iter=3800/40000, loss=0.4982, lr=0.009141, batch_cost=0.5171, reader_cost=0.0010 | ETA 05:12:00 2020-11-24 17:05:05 [INFO] [TRAIN] epoch=6, iter=3900/40000, loss=0.4788, lr=0.009118, batch_cost=0.5148, reader_cost=0.0009 | ETA 05:09:42 2020-11-24 17:05:57 [INFO] [TRAIN] epoch=7, iter=4000/40000, loss=0.5944, lr=0.009096, batch_cost=0.5198, reader_cost=0.0063 | ETA 05:11:51 2020-11-24 17:05:57 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-24 17:06:19 [INFO] [EVAL] #Images=1449 mIoU=0.6936 Acc=0.9265 Kappa=0.8398 2020-11-24 17:06:19 [INFO] [EVAL] Class IoU: [0.9274 0.8579 0.3285 0.8712 0.6616 0.7187 0.7757 0.791 0.78 0.3419 0.7863 0.5375 0.7357 0.7977 0.7743 0.8089 0.516 0.7824 0.3559 0.7401 0.6776] 2020-11-24 17:06:19 [INFO] [EVAL] Class Acc: [0.9643 0.9041 0.3375 0.9219 0.7647 0.8114 0.9855 0.864 0.7922 0.438 0.9111 0.889 0.9035 0.9146 0.8348 0.8937 0.6474 0.8377 0.8295 0.7679 0.8777] 2020-11-24 17:06:27 [INFO] [EVAL] The model with the best validation mIoU (0.6936) was saved at iter 4000. 2020-11-24 17:07:20 [INFO] [TRAIN] epoch=7, iter=4100/40000, loss=0.4627, lr=0.009073, batch_cost=0.5224, reader_cost=0.0017 | ETA 05:12:33 2020-11-24 17:08:12 [INFO] [TRAIN] epoch=7, iter=4200/40000, loss=0.5087, lr=0.009050, batch_cost=0.5213, reader_cost=0.0013 | ETA 05:11:03 2020-11-24 17:09:05 [INFO] [TRAIN] epoch=7, iter=4300/40000, loss=0.4807, lr=0.009027, batch_cost=0.5324, reader_cost=0.0012 | ETA 05:16:47 2020-11-24 17:09:57 [INFO] [TRAIN] epoch=7, iter=4400/40000, loss=0.5420, lr=0.009005, batch_cost=0.5238, reader_cost=0.0010 | ETA 05:10:47 2020-11-24 17:10:50 [INFO] [TRAIN] epoch=7, iter=4500/40000, loss=0.5042, lr=0.008982, batch_cost=0.5233, reader_cost=0.0008 | ETA 05:09:36 2020-11-24 17:11:42 [INFO] [TRAIN] epoch=7, iter=4600/40000, loss=0.5509, lr=0.008959, batch_cost=0.5192, reader_cost=0.0012 | ETA 05:06:18 2020-11-24 17:12:34 [INFO] [TRAIN] epoch=8, iter=4700/40000, loss=0.5408, lr=0.008936, batch_cost=0.5198, reader_cost=0.0066 | ETA 05:05:49 2020-11-24 17:13:25 [INFO] [TRAIN] epoch=8, iter=4800/40000, loss=0.5350, lr=0.008913, batch_cost=0.5138, reader_cost=0.0009 | ETA 05:01:25 2020-11-24 17:14:16 [INFO] [TRAIN] epoch=8, iter=4900/40000, loss=0.4685, lr=0.008891, batch_cost=0.5153, reader_cost=0.0012 | ETA 05:01:26 2020-11-24 17:15:08 [INFO] [TRAIN] epoch=8, iter=5000/40000, loss=0.4624, lr=0.008868, batch_cost=0.5136, reader_cost=0.0007 | ETA 04:59:34 2020-11-24 17:16:00 [INFO] [TRAIN] epoch=8, iter=5100/40000, loss=0.4917, lr=0.008845, batch_cost=0.5180, reader_cost=0.0010 | ETA 05:01:18 2020-11-24 17:16:51 [INFO] [TRAIN] epoch=8, iter=5200/40000, loss=0.4309, lr=0.008822, batch_cost=0.5133, reader_cost=0.0009 | ETA 04:57:41 2020-11-24 17:17:43 [INFO] [TRAIN] epoch=9, iter=5300/40000, loss=0.5211, lr=0.008799, batch_cost=0.5210, reader_cost=0.0072 | ETA 05:01:20 2020-11-24 17:18:34 [INFO] [TRAIN] epoch=9, iter=5400/40000, loss=0.4495, lr=0.008777, batch_cost=0.5149, reader_cost=0.0014 | ETA 04:56:55 2020-11-24 17:19:26 [INFO] [TRAIN] epoch=9, iter=5500/40000, loss=0.4450, lr=0.008754, batch_cost=0.5138, reader_cost=0.0007 | ETA 04:55:26 2020-11-24 17:20:16 [INFO] [TRAIN] epoch=9, iter=5600/40000, loss=0.4541, lr=0.008731, batch_cost=0.5014, reader_cost=0.0002 | ETA 04:47:29 2020-11-24 17:21:07 [INFO] [TRAIN] epoch=9, iter=5700/40000, loss=0.4818, lr=0.008708, batch_cost=0.5049, reader_cost=0.0002 | ETA 04:48:36 2020-11-24 17:21:58 [INFO] [TRAIN] epoch=9, iter=5800/40000, loss=0.4732, lr=0.008685, batch_cost=0.5115, reader_cost=0.0006 | ETA 04:51:31 2020-11-24 17:22:49 [INFO] [TRAIN] epoch=9, iter=5900/40000, loss=0.4443, lr=0.008662, batch_cost=0.5139, reader_cost=0.0010 | ETA 04:52:04 2020-11-24 17:23:41 [INFO] [TRAIN] epoch=10, iter=6000/40000, loss=0.4844, lr=0.008639, batch_cost=0.5208, reader_cost=0.0074 | ETA 04:55:07 2020-11-24 17:24:33 [INFO] [TRAIN] epoch=10, iter=6100/40000, loss=0.4057, lr=0.008617, batch_cost=0.5139, reader_cost=0.0011 | ETA 04:50:21 2020-11-24 17:25:25 [INFO] [TRAIN] epoch=10, iter=6200/40000, loss=0.4414, lr=0.008594, batch_cost=0.5219, reader_cost=0.0010 | ETA 04:54:00 2020-11-24 17:26:16 [INFO] [TRAIN] epoch=10, iter=6300/40000, loss=0.4246, lr=0.008571, batch_cost=0.5159, reader_cost=0.0009 | ETA 04:49:45 2020-11-24 17:27:08 [INFO] [TRAIN] epoch=10, iter=6400/40000, loss=0.4651, lr=0.008548, batch_cost=0.5145, reader_cost=0.0012 | ETA 04:48:06 2020-11-24 17:27:59 [INFO] [TRAIN] epoch=10, iter=6500/40000, loss=0.4048, lr=0.008525, batch_cost=0.5149, reader_cost=0.0013 | ETA 04:47:29 2020-11-24 17:28:52 [INFO] [TRAIN] epoch=10, iter=6600/40000, loss=0.4606, lr=0.008502, batch_cost=0.5242, reader_cost=0.0011 | ETA 04:51:47 2020-11-24 17:29:44 [INFO] [TRAIN] epoch=11, iter=6700/40000, loss=0.4260, lr=0.008479, batch_cost=0.5210, reader_cost=0.0076 | ETA 04:49:09 2020-11-24 17:30:35 [INFO] [TRAIN] epoch=11, iter=6800/40000, loss=0.5136, lr=0.008456, batch_cost=0.5145, reader_cost=0.0009 | ETA 04:44:42 2020-11-24 17:31:27 [INFO] [TRAIN] epoch=11, iter=6900/40000, loss=0.4170, lr=0.008433, batch_cost=0.5138, reader_cost=0.0008 | ETA 04:43:25 2020-11-24 17:32:18 [INFO] [TRAIN] epoch=11, iter=7000/40000, loss=0.4557, lr=0.008410, batch_cost=0.5130, reader_cost=0.0008 | ETA 04:42:09 2020-11-24 17:33:10 [INFO] [TRAIN] epoch=11, iter=7100/40000, loss=0.4381, lr=0.008388, batch_cost=0.5186, reader_cost=0.0010 | ETA 04:44:21 2020-11-24 17:34:01 [INFO] [TRAIN] epoch=11, iter=7200/40000, loss=0.3792, lr=0.008365, batch_cost=0.5146, reader_cost=0.0009 | ETA 04:41:19 2020-11-24 17:34:54 [INFO] [TRAIN] epoch=12, iter=7300/40000, loss=0.4239, lr=0.008342, batch_cost=0.5243, reader_cost=0.0072 | ETA 04:45:45 2020-11-24 17:35:45 [INFO] [TRAIN] epoch=12, iter=7400/40000, loss=0.4853, lr=0.008319, batch_cost=0.5144, reader_cost=0.0009 | ETA 04:39:28 2020-11-24 17:36:37 [INFO] [TRAIN] epoch=12, iter=7500/40000, loss=0.3991, lr=0.008296, batch_cost=0.5162, reader_cost=0.0009 | ETA 04:39:36 2020-11-24 17:37:28 [INFO] [TRAIN] epoch=12, iter=7600/40000, loss=0.3667, lr=0.008273, batch_cost=0.5133, reader_cost=0.0012 | ETA 04:37:11 2020-11-24 17:38:20 [INFO] [TRAIN] epoch=12, iter=7700/40000, loss=0.3831, lr=0.008250, batch_cost=0.5145, reader_cost=0.0006 | ETA 04:36:58 2020-11-24 17:39:11 [INFO] [TRAIN] epoch=12, iter=7800/40000, loss=0.3774, lr=0.008227, batch_cost=0.5150, reader_cost=0.0008 | ETA 04:36:22 2020-11-24 17:40:02 [INFO] [TRAIN] epoch=12, iter=7900/40000, loss=0.3732, lr=0.008204, batch_cost=0.5120, reader_cost=0.0004 | ETA 04:33:54 2020-11-24 17:40:54 [INFO] [TRAIN] epoch=13, iter=8000/40000, loss=0.3921, lr=0.008181, batch_cost=0.5196, reader_cost=0.0063 | ETA 04:37:08 2020-11-24 17:40:54 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-24 17:41:16 [INFO] [EVAL] #Images=1449 mIoU=0.7527 Acc=0.9442 Kappa=0.8775 2020-11-24 17:41:16 [INFO] [EVAL] Class IoU: [0.9404 0.9029 0.4197 0.8797 0.6813 0.739 0.9085 0.8273 0.9124 0.4111 0.8376 0.5347 0.861 0.837 0.8462 0.8519 0.5611 0.8461 0.4244 0.857 0.7276] 2020-11-24 17:41:16 [INFO] [EVAL] Class Acc: [0.9681 0.9402 0.444 0.9074 0.7407 0.8393 0.9653 0.867 0.9724 0.5196 0.9393 0.8893 0.8955 0.867 0.9238 0.9246 0.8278 0.9476 0.8382 0.89 0.8295] 2020-11-24 17:41:24 [INFO] [EVAL] The model with the best validation mIoU (0.7527) was saved at iter 8000. 2020-11-24 17:42:15 [INFO] [TRAIN] epoch=13, iter=8100/40000, loss=0.4221, lr=0.008158, batch_cost=0.5126, reader_cost=0.0007 | ETA 04:32:30 2020-11-24 17:43:07 [INFO] [TRAIN] epoch=13, iter=8200/40000, loss=0.4166, lr=0.008135, batch_cost=0.5124, reader_cost=0.0011 | ETA 04:31:34 2020-11-24 17:43:58 [INFO] [TRAIN] epoch=13, iter=8300/40000, loss=0.3951, lr=0.008112, batch_cost=0.5145, reader_cost=0.0005 | ETA 04:31:51 2020-11-24 17:44:49 [INFO] [TRAIN] epoch=13, iter=8400/40000, loss=0.4297, lr=0.008089, batch_cost=0.5144, reader_cost=0.0008 | ETA 04:30:54 2020-11-24 17:45:41 [INFO] [TRAIN] epoch=13, iter=8500/40000, loss=0.4170, lr=0.008066, batch_cost=0.5136, reader_cost=0.0011 | ETA 04:29:38 2020-11-24 17:46:33 [INFO] [TRAIN] epoch=14, iter=8600/40000, loss=0.4376, lr=0.008043, batch_cost=0.5214, reader_cost=0.0069 | ETA 04:32:53 2020-11-24 17:47:25 [INFO] [TRAIN] epoch=14, iter=8700/40000, loss=0.3409, lr=0.008020, batch_cost=0.5158, reader_cost=0.0005 | ETA 04:29:05 2020-11-24 17:48:16 [INFO] [TRAIN] epoch=14, iter=8800/40000, loss=0.3621, lr=0.007996, batch_cost=0.5143, reader_cost=0.0006 | ETA 04:27:27 2020-11-24 17:49:07 [INFO] [TRAIN] epoch=14, iter=8900/40000, loss=0.3668, lr=0.007973, batch_cost=0.5133, reader_cost=0.0008 | ETA 04:26:04 2020-11-24 17:49:59 [INFO] [TRAIN] epoch=14, iter=9000/40000, loss=0.4114, lr=0.007950, batch_cost=0.5153, reader_cost=0.0009 | ETA 04:26:15 2020-11-24 17:50:50 [INFO] [TRAIN] epoch=14, iter=9100/40000, loss=0.3726, lr=0.007927, batch_cost=0.5154, reader_cost=0.0009 | ETA 04:25:26 2020-11-24 17:51:42 [INFO] [TRAIN] epoch=14, iter=9200/40000, loss=0.3839, lr=0.007904, batch_cost=0.5168, reader_cost=0.0011 | ETA 04:25:16 2020-11-24 17:52:34 [INFO] [TRAIN] epoch=15, iter=9300/40000, loss=0.4654, lr=0.007881, batch_cost=0.5179, reader_cost=0.0061 | ETA 04:24:58 2020-11-24 17:53:25 [INFO] [TRAIN] epoch=15, iter=9400/40000, loss=0.3598, lr=0.007858, batch_cost=0.5104, reader_cost=0.0008 | ETA 04:20:19 2020-11-24 17:54:16 [INFO] [TRAIN] epoch=15, iter=9500/40000, loss=0.3730, lr=0.007835, batch_cost=0.5139, reader_cost=0.0013 | ETA 04:21:14 2020-11-24 17:55:07 [INFO] [TRAIN] epoch=15, iter=9600/40000, loss=0.4151, lr=0.007812, batch_cost=0.5094, reader_cost=0.0010 | ETA 04:18:04 2020-11-24 17:55:59 [INFO] [TRAIN] epoch=15, iter=9700/40000, loss=0.3820, lr=0.007789, batch_cost=0.5139, reader_cost=0.0007 | ETA 04:19:31 2020-11-24 17:56:50 [INFO] [TRAIN] epoch=15, iter=9800/40000, loss=0.4262, lr=0.007765, batch_cost=0.5154, reader_cost=0.0006 | ETA 04:19:26 2020-11-24 17:57:42 [INFO] [TRAIN] epoch=15, iter=9900/40000, loss=0.3500, lr=0.007742, batch_cost=0.5134, reader_cost=0.0011 | ETA 04:17:31 2020-11-24 17:58:35 [INFO] [TRAIN] epoch=16, iter=10000/40000, loss=0.3955, lr=0.007719, batch_cost=0.5369, reader_cost=0.0065 | ETA 04:28:26 2020-11-24 17:59:27 [INFO] [TRAIN] epoch=16, iter=10100/40000, loss=0.3411, lr=0.007696, batch_cost=0.5150, reader_cost=0.0010 | ETA 04:16:38 2020-11-24 18:00:18 [INFO] [TRAIN] epoch=16, iter=10200/40000, loss=0.3837, lr=0.007673, batch_cost=0.5120, reader_cost=0.0009 | ETA 04:14:17 2020-11-24 18:01:09 [INFO] [TRAIN] epoch=16, iter=10300/40000, loss=0.3307, lr=0.007650, batch_cost=0.5118, reader_cost=0.0013 | ETA 04:13:20 2020-11-24 18:02:01 [INFO] [TRAIN] epoch=16, iter=10400/40000, loss=0.3869, lr=0.007626, batch_cost=0.5147, reader_cost=0.0009 | ETA 04:13:54 2020-11-24 18:02:52 [INFO] [TRAIN] epoch=16, iter=10500/40000, loss=0.3555, lr=0.007603, batch_cost=0.5137, reader_cost=0.0012 | ETA 04:12:32 2020-11-24 18:03:44 [INFO] [TRAIN] epoch=17, iter=10600/40000, loss=0.3977, lr=0.007580, batch_cost=0.5191, reader_cost=0.0065 | ETA 04:14:21 2020-11-24 18:04:35 [INFO] [TRAIN] epoch=17, iter=10700/40000, loss=0.3115, lr=0.007557, batch_cost=0.5140, reader_cost=0.0011 | ETA 04:10:59 2020-11-24 18:05:27 [INFO] [TRAIN] epoch=17, iter=10800/40000, loss=0.3717, lr=0.007534, batch_cost=0.5152, reader_cost=0.0011 | ETA 04:10:43 2020-11-24 18:06:18 [INFO] [TRAIN] epoch=17, iter=10900/40000, loss=0.3575, lr=0.007510, batch_cost=0.5117, reader_cost=0.0011 | ETA 04:08:09 2020-11-24 18:07:09 [INFO] [TRAIN] epoch=17, iter=11000/40000, loss=0.3484, lr=0.007487, batch_cost=0.5127, reader_cost=0.0013 | ETA 04:07:49 2020-11-24 18:08:01 [INFO] [TRAIN] epoch=17, iter=11100/40000, loss=0.4082, lr=0.007464, batch_cost=0.5145, reader_cost=0.0013 | ETA 04:07:47 2020-11-24 18:08:52 [INFO] [TRAIN] epoch=17, iter=11200/40000, loss=0.3199, lr=0.007441, batch_cost=0.5125, reader_cost=0.0014 | ETA 04:06:00 2020-11-24 18:09:43 [INFO] [TRAIN] epoch=18, iter=11300/40000, loss=0.3406, lr=0.007417, batch_cost=0.5137, reader_cost=0.0067 | ETA 04:05:42 2020-11-24 18:10:34 [INFO] [TRAIN] epoch=18, iter=11400/40000, loss=0.3500, lr=0.007394, batch_cost=0.5030, reader_cost=0.0004 | ETA 03:59:46 2020-11-24 18:11:24 [INFO] [TRAIN] epoch=18, iter=11500/40000, loss=0.3362, lr=0.007371, batch_cost=0.5052, reader_cost=0.0003 | ETA 03:59:57 2020-11-24 18:12:15 [INFO] [TRAIN] epoch=18, iter=11600/40000, loss=0.3715, lr=0.007348, batch_cost=0.5137, reader_cost=0.0013 | ETA 04:03:09 2020-11-24 18:13:07 [INFO] [TRAIN] epoch=18, iter=11700/40000, loss=0.3095, lr=0.007324, batch_cost=0.5147, reader_cost=0.0013 | ETA 04:02:45 2020-11-24 18:13:59 [INFO] [TRAIN] epoch=18, iter=11800/40000, loss=0.3273, lr=0.007301, batch_cost=0.5158, reader_cost=0.0008 | ETA 04:02:25 2020-11-24 18:14:50 [INFO] [TRAIN] epoch=19, iter=11900/40000, loss=0.3331, lr=0.007278, batch_cost=0.5191, reader_cost=0.0068 | ETA 04:03:05 2020-11-24 18:15:42 [INFO] [TRAIN] epoch=19, iter=12000/40000, loss=0.3432, lr=0.007254, batch_cost=0.5116, reader_cost=0.0010 | ETA 03:58:44 2020-11-24 18:15:42 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-24 18:16:04 [INFO] [EVAL] #Images=1449 mIoU=0.7627 Acc=0.9462 Kappa=0.8820 2020-11-24 18:16:04 [INFO] [EVAL] Class IoU: [0.9402 0.8757 0.4275 0.885 0.6939 0.8023 0.946 0.8784 0.9097 0.3338 0.8507 0.5027 0.8633 0.873 0.8429 0.8495 0.6444 0.7612 0.5581 0.9021 0.6768] 2020-11-24 18:16:04 [INFO] [EVAL] Class Acc: [0.9684 0.9071 0.4483 0.9398 0.7763 0.8651 0.9622 0.9697 0.9498 0.7514 0.8814 0.9451 0.9033 0.9469 0.9354 0.9176 0.8534 0.9541 0.6245 0.9616 0.8045] 2020-11-24 18:16:11 [INFO] [EVAL] The model with the best validation mIoU (0.7627) was saved at iter 12000. 2020-11-24 18:17:03 [INFO] [TRAIN] epoch=19, iter=12100/40000, loss=0.3182, lr=0.007231, batch_cost=0.5138, reader_cost=0.0010 | ETA 03:58:54 2020-11-24 18:17:54 [INFO] [TRAIN] epoch=19, iter=12200/40000, loss=0.3103, lr=0.007208, batch_cost=0.5113, reader_cost=0.0009 | ETA 03:56:55 2020-11-24 18:18:45 [INFO] [TRAIN] epoch=19, iter=12300/40000, loss=0.3392, lr=0.007184, batch_cost=0.5117, reader_cost=0.0011 | ETA 03:56:13 2020-11-24 18:19:36 [INFO] [TRAIN] epoch=19, iter=12400/40000, loss=0.3413, lr=0.007161, batch_cost=0.5137, reader_cost=0.0008 | ETA 03:56:18 2020-11-24 18:20:27 [INFO] [TRAIN] epoch=19, iter=12500/40000, loss=0.3065, lr=0.007138, batch_cost=0.5128, reader_cost=0.0006 | ETA 03:55:02 2020-11-24 18:21:20 [INFO] [TRAIN] epoch=20, iter=12600/40000, loss=0.3148, lr=0.007114, batch_cost=0.5225, reader_cost=0.0060 | ETA 03:58:35 2020-11-24 18:22:11 [INFO] [TRAIN] epoch=20, iter=12700/40000, loss=0.3021, lr=0.007091, batch_cost=0.5134, reader_cost=0.0004 | ETA 03:53:34 2020-11-24 18:23:02 [INFO] [TRAIN] epoch=20, iter=12800/40000, loss=0.3346, lr=0.007068, batch_cost=0.5137, reader_cost=0.0003 | ETA 03:52:52 2020-11-24 18:23:54 [INFO] [TRAIN] epoch=20, iter=12900/40000, loss=0.3609, lr=0.007044, batch_cost=0.5153, reader_cost=0.0007 | ETA 03:52:45 2020-11-24 18:24:45 [INFO] [TRAIN] epoch=20, iter=13000/40000, loss=0.3611, lr=0.007021, batch_cost=0.5135, reader_cost=0.0012 | ETA 03:51:05 2020-11-24 18:25:37 [INFO] [TRAIN] epoch=20, iter=13100/40000, loss=0.3108, lr=0.006997, batch_cost=0.5132, reader_cost=0.0010 | ETA 03:50:05 2020-11-24 18:26:28 [INFO] [TRAIN] epoch=20, iter=13200/40000, loss=0.3116, lr=0.006974, batch_cost=0.5134, reader_cost=0.0010 | ETA 03:49:17 2020-11-24 18:27:20 [INFO] [TRAIN] epoch=21, iter=13300/40000, loss=0.3107, lr=0.006951, batch_cost=0.5198, reader_cost=0.0068 | ETA 03:51:17 2020-11-24 18:28:12 [INFO] [TRAIN] epoch=21, iter=13400/40000, loss=0.3378, lr=0.006927, batch_cost=0.5174, reader_cost=0.0009 | ETA 03:49:22 2020-11-24 18:29:03 [INFO] [TRAIN] epoch=21, iter=13500/40000, loss=0.3128, lr=0.006904, batch_cost=0.5147, reader_cost=0.0014 | ETA 03:47:18 2020-11-24 18:29:55 [INFO] [TRAIN] epoch=21, iter=13600/40000, loss=0.3029, lr=0.006880, batch_cost=0.5152, reader_cost=0.0009 | ETA 03:46:41 2020-11-24 18:30:46 [INFO] [TRAIN] epoch=21, iter=13700/40000, loss=0.3041, lr=0.006857, batch_cost=0.5143, reader_cost=0.0011 | ETA 03:45:26 2020-11-24 18:31:37 [INFO] [TRAIN] epoch=21, iter=13800/40000, loss=0.3022, lr=0.006833, batch_cost=0.5134, reader_cost=0.0010 | ETA 03:44:10 2020-11-24 18:32:29 [INFO] [TRAIN] epoch=22, iter=13900/40000, loss=0.3329, lr=0.006810, batch_cost=0.5158, reader_cost=0.0058 | ETA 03:44:21 2020-11-24 18:33:20 [INFO] [TRAIN] epoch=22, iter=14000/40000, loss=0.3491, lr=0.006786, batch_cost=0.5145, reader_cost=0.0007 | ETA 03:42:57 2020-11-24 18:34:12 [INFO] [TRAIN] epoch=22, iter=14100/40000, loss=0.3617, lr=0.006763, batch_cost=0.5135, reader_cost=0.0011 | ETA 03:41:39 2020-11-24 18:35:03 [INFO] [TRAIN] epoch=22, iter=14200/40000, loss=0.3579, lr=0.006739, batch_cost=0.5103, reader_cost=0.0013 | ETA 03:39:24 2020-11-24 18:35:54 [INFO] [TRAIN] epoch=22, iter=14300/40000, loss=0.2851, lr=0.006716, batch_cost=0.5143, reader_cost=0.0009 | ETA 03:40:18 2020-11-24 18:36:46 [INFO] [TRAIN] epoch=22, iter=14400/40000, loss=0.3257, lr=0.006692, batch_cost=0.5135, reader_cost=0.0010 | ETA 03:39:05 2020-11-24 18:37:37 [INFO] [TRAIN] epoch=22, iter=14500/40000, loss=0.3374, lr=0.006669, batch_cost=0.5126, reader_cost=0.0012 | ETA 03:37:51 2020-11-24 18:38:29 [INFO] [TRAIN] epoch=23, iter=14600/40000, loss=0.2912, lr=0.006645, batch_cost=0.5196, reader_cost=0.0064 | ETA 03:39:57 2020-11-24 18:39:20 [INFO] [TRAIN] epoch=23, iter=14700/40000, loss=0.2882, lr=0.006622, batch_cost=0.5157, reader_cost=0.0011 | ETA 03:37:27 2020-11-24 18:40:12 [INFO] [TRAIN] epoch=23, iter=14800/40000, loss=0.3390, lr=0.006598, batch_cost=0.5161, reader_cost=0.0010 | ETA 03:36:46 2020-11-24 18:41:04 [INFO] [TRAIN] epoch=23, iter=14900/40000, loss=0.2976, lr=0.006575, batch_cost=0.5159, reader_cost=0.0009 | ETA 03:35:49 2020-11-24 18:41:55 [INFO] [TRAIN] epoch=23, iter=15000/40000, loss=0.3015, lr=0.006551, batch_cost=0.5158, reader_cost=0.0010 | ETA 03:34:55 2020-11-24 18:42:47 [INFO] [TRAIN] epoch=23, iter=15100/40000, loss=0.3292, lr=0.006527, batch_cost=0.5133, reader_cost=0.0009 | ETA 03:33:01 2020-11-24 18:43:38 [INFO] [TRAIN] epoch=23, iter=15200/40000, loss=0.3244, lr=0.006504, batch_cost=0.5138, reader_cost=0.0011 | ETA 03:32:22 2020-11-24 18:44:30 [INFO] [TRAIN] epoch=24, iter=15300/40000, loss=0.2715, lr=0.006480, batch_cost=0.5216, reader_cost=0.0072 | ETA 03:34:44 2020-11-24 18:45:21 [INFO] [TRAIN] epoch=24, iter=15400/40000, loss=0.2715, lr=0.006457, batch_cost=0.5116, reader_cost=0.0009 | ETA 03:29:46 2020-11-24 18:46:12 [INFO] [TRAIN] epoch=24, iter=15500/40000, loss=0.2909, lr=0.006433, batch_cost=0.5100, reader_cost=0.0006 | ETA 03:28:14 2020-11-24 18:47:04 [INFO] [TRAIN] epoch=24, iter=15600/40000, loss=0.2865, lr=0.006409, batch_cost=0.5143, reader_cost=0.0005 | ETA 03:29:09 2020-11-24 18:47:55 [INFO] [TRAIN] epoch=24, iter=15700/40000, loss=0.3139, lr=0.006386, batch_cost=0.5162, reader_cost=0.0010 | ETA 03:29:03 2020-11-24 18:48:47 [INFO] [TRAIN] epoch=24, iter=15800/40000, loss=0.2660, lr=0.006362, batch_cost=0.5123, reader_cost=0.0012 | ETA 03:26:38 2020-11-24 18:49:39 [INFO] [TRAIN] epoch=25, iter=15900/40000, loss=0.2776, lr=0.006338, batch_cost=0.5205, reader_cost=0.0078 | ETA 03:29:04 2020-11-24 18:50:30 [INFO] [TRAIN] epoch=25, iter=16000/40000, loss=0.2810, lr=0.006315, batch_cost=0.5148, reader_cost=0.0013 | ETA 03:25:54 2020-11-24 18:50:30 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-24 18:50:53 [INFO] [EVAL] #Images=1449 mIoU=0.7687 Acc=0.9485 Kappa=0.8862 2020-11-24 18:50:53 [INFO] [EVAL] Class IoU: [0.943 0.8641 0.4398 0.8833 0.6922 0.7986 0.9229 0.8472 0.934 0.397 0.8639 0.5318 0.909 0.8681 0.8504 0.8754 0.6327 0.8494 0.4015 0.8678 0.7697] 2020-11-24 18:50:53 [INFO] [EVAL] Class Acc: [0.9668 0.8817 0.4598 0.9155 0.797 0.9224 0.9628 0.8967 0.965 0.5477 0.8907 0.9045 0.9454 0.9555 0.9422 0.9356 0.7946 0.9417 0.8236 0.8947 0.9452] 2020-11-24 18:51:01 [INFO] [EVAL] The model with the best validation mIoU (0.7687) was saved at iter 16000. 2020-11-24 18:51:52 [INFO] [TRAIN] epoch=25, iter=16100/40000, loss=0.2520, lr=0.006291, batch_cost=0.5128, reader_cost=0.0014 | ETA 03:24:15 2020-11-24 18:52:44 [INFO] [TRAIN] epoch=25, iter=16200/40000, loss=0.3092, lr=0.006267, batch_cost=0.5152, reader_cost=0.0016 | ETA 03:24:21 2020-11-24 18:53:35 [INFO] [TRAIN] epoch=25, iter=16300/40000, loss=0.3027, lr=0.006244, batch_cost=0.5144, reader_cost=0.0013 | ETA 03:23:10 2020-11-24 18:54:27 [INFO] [TRAIN] epoch=25, iter=16400/40000, loss=0.2919, lr=0.006220, batch_cost=0.5147, reader_cost=0.0012 | ETA 03:22:27 2020-11-24 18:55:19 [INFO] [TRAIN] epoch=25, iter=16500/40000, loss=0.2726, lr=0.006196, batch_cost=0.5192, reader_cost=0.0013 | ETA 03:23:20 2020-11-24 18:56:11 [INFO] [TRAIN] epoch=26, iter=16600/40000, loss=0.3109, lr=0.006172, batch_cost=0.5185, reader_cost=0.0066 | ETA 03:22:12 2020-11-24 18:57:02 [INFO] [TRAIN] epoch=26, iter=16700/40000, loss=0.2807, lr=0.006149, batch_cost=0.5158, reader_cost=0.0011 | ETA 03:20:18 2020-11-24 18:57:53 [INFO] [TRAIN] epoch=26, iter=16800/40000, loss=0.2727, lr=0.006125, batch_cost=0.5140, reader_cost=0.0012 | ETA 03:18:43 2020-11-24 18:58:45 [INFO] [TRAIN] epoch=26, iter=16900/40000, loss=0.3371, lr=0.006101, batch_cost=0.5141, reader_cost=0.0015 | ETA 03:17:56 2020-11-24 18:59:35 [INFO] [TRAIN] epoch=26, iter=17000/40000, loss=0.2873, lr=0.006077, batch_cost=0.5045, reader_cost=0.0005 | ETA 03:13:22 2020-11-24 19:00:26 [INFO] [TRAIN] epoch=26, iter=17100/40000, loss=0.2758, lr=0.006054, batch_cost=0.5029, reader_cost=0.0002 | ETA 03:11:56 2020-11-24 19:01:17 [INFO] [TRAIN] epoch=27, iter=17200/40000, loss=0.2707, lr=0.006030, batch_cost=0.5164, reader_cost=0.0059 | ETA 03:16:13 2020-11-24 19:02:08 [INFO] [TRAIN] epoch=27, iter=17300/40000, loss=0.2720, lr=0.006006, batch_cost=0.5097, reader_cost=0.0004 | ETA 03:12:50 2020-11-24 19:03:00 [INFO] [TRAIN] epoch=27, iter=17400/40000, loss=0.2735, lr=0.005982, batch_cost=0.5136, reader_cost=0.0009 | ETA 03:13:27 2020-11-24 19:03:51 [INFO] [TRAIN] epoch=27, iter=17500/40000, loss=0.2805, lr=0.005958, batch_cost=0.5143, reader_cost=0.0007 | ETA 03:12:51 2020-11-24 19:04:42 [INFO] [TRAIN] epoch=27, iter=17600/40000, loss=0.3022, lr=0.005935, batch_cost=0.5106, reader_cost=0.0003 | ETA 03:10:38 2020-11-24 19:05:33 [INFO] [TRAIN] epoch=27, iter=17700/40000, loss=0.2868, lr=0.005911, batch_cost=0.5133, reader_cost=0.0004 | ETA 03:10:46 2020-11-24 19:06:25 [INFO] [TRAIN] epoch=27, iter=17800/40000, loss=0.2557, lr=0.005887, batch_cost=0.5154, reader_cost=0.0013 | ETA 03:10:42 2020-11-24 19:07:17 [INFO] [TRAIN] epoch=28, iter=17900/40000, loss=0.2687, lr=0.005863, batch_cost=0.5184, reader_cost=0.0066 | ETA 03:10:55 2020-11-24 19:08:08 [INFO] [TRAIN] epoch=28, iter=18000/40000, loss=0.2719, lr=0.005839, batch_cost=0.5165, reader_cost=0.0010 | ETA 03:09:22 2020-11-24 19:09:00 [INFO] [TRAIN] epoch=28, iter=18100/40000, loss=0.2736, lr=0.005815, batch_cost=0.5134, reader_cost=0.0011 | ETA 03:07:22 2020-11-24 19:09:51 [INFO] [TRAIN] epoch=28, iter=18200/40000, loss=0.2781, lr=0.005791, batch_cost=0.5137, reader_cost=0.0012 | ETA 03:06:38 2020-11-24 19:10:43 [INFO] [TRAIN] epoch=28, iter=18300/40000, loss=0.2664, lr=0.005767, batch_cost=0.5144, reader_cost=0.0011 | ETA 03:06:02 2020-11-24 19:11:34 [INFO] [TRAIN] epoch=28, iter=18400/40000, loss=0.2599, lr=0.005743, batch_cost=0.5138, reader_cost=0.0009 | ETA 03:04:58 2020-11-24 19:12:25 [INFO] [TRAIN] epoch=28, iter=18500/40000, loss=0.2543, lr=0.005720, batch_cost=0.5134, reader_cost=0.0012 | ETA 03:03:58 2020-11-24 19:13:17 [INFO] [TRAIN] epoch=29, iter=18600/40000, loss=0.2700, lr=0.005696, batch_cost=0.5181, reader_cost=0.0064 | ETA 03:04:47 2020-11-24 19:14:09 [INFO] [TRAIN] epoch=29, iter=18700/40000, loss=0.2591, lr=0.005672, batch_cost=0.5157, reader_cost=0.0014 | ETA 03:03:05 2020-11-24 19:15:00 [INFO] [TRAIN] epoch=29, iter=18800/40000, loss=0.2670, lr=0.005648, batch_cost=0.5135, reader_cost=0.0013 | ETA 03:01:26 2020-11-24 19:15:52 [INFO] [TRAIN] epoch=29, iter=18900/40000, loss=0.2899, lr=0.005624, batch_cost=0.5170, reader_cost=0.0011 | ETA 03:01:49 2020-11-24 19:16:43 [INFO] [TRAIN] epoch=29, iter=19000/40000, loss=0.2763, lr=0.005600, batch_cost=0.5126, reader_cost=0.0008 | ETA 02:59:24 2020-11-24 19:17:35 [INFO] [TRAIN] epoch=29, iter=19100/40000, loss=0.2556, lr=0.005576, batch_cost=0.5164, reader_cost=0.0019 | ETA 02:59:53 2020-11-24 19:18:27 [INFO] [TRAIN] epoch=30, iter=19200/40000, loss=0.3006, lr=0.005552, batch_cost=0.5209, reader_cost=0.0074 | ETA 03:00:34 2020-11-24 19:19:18 [INFO] [TRAIN] epoch=30, iter=19300/40000, loss=0.2753, lr=0.005528, batch_cost=0.5155, reader_cost=0.0010 | ETA 02:57:49 2020-11-24 19:20:10 [INFO] [TRAIN] epoch=30, iter=19400/40000, loss=0.3047, lr=0.005504, batch_cost=0.5145, reader_cost=0.0013 | ETA 02:56:38 2020-11-24 19:21:01 [INFO] [TRAIN] epoch=30, iter=19500/40000, loss=0.2861, lr=0.005480, batch_cost=0.5146, reader_cost=0.0011 | ETA 02:55:48 2020-11-24 19:21:53 [INFO] [TRAIN] epoch=30, iter=19600/40000, loss=0.2865, lr=0.005455, batch_cost=0.5145, reader_cost=0.0011 | ETA 02:54:55 2020-11-24 19:22:44 [INFO] [TRAIN] epoch=30, iter=19700/40000, loss=0.3005, lr=0.005431, batch_cost=0.5151, reader_cost=0.0013 | ETA 02:54:16 2020-11-24 19:23:36 [INFO] [TRAIN] epoch=30, iter=19800/40000, loss=0.2879, lr=0.005407, batch_cost=0.5205, reader_cost=0.0011 | ETA 02:55:14 2020-11-24 19:24:29 [INFO] [TRAIN] epoch=31, iter=19900/40000, loss=0.2581, lr=0.005383, batch_cost=0.5225, reader_cost=0.0069 | ETA 02:55:03 2020-11-24 19:25:20 [INFO] [TRAIN] epoch=31, iter=20000/40000, loss=0.2560, lr=0.005359, batch_cost=0.5132, reader_cost=0.0011 | ETA 02:51:03 2020-11-24 19:25:20 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-24 19:25:42 [INFO] [EVAL] #Images=1449 mIoU=0.7903 Acc=0.9532 Kappa=0.8967 2020-11-24 19:25:42 [INFO] [EVAL] Class IoU: [0.9473 0.8808 0.4345 0.9092 0.7232 0.8328 0.9458 0.8887 0.9323 0.436 0.9011 0.5933 0.8975 0.9119 0.8647 0.8691 0.6132 0.8414 0.5124 0.8989 0.7624] 2020-11-24 19:25:42 [INFO] [EVAL] Class Acc: [0.9694 0.914 0.4479 0.9485 0.8182 0.9265 0.9797 0.9411 0.9661 0.617 0.937 0.9284 0.9471 0.9618 0.9499 0.9196 0.8147 0.9569 0.7895 0.9382 0.9548] 2020-11-24 19:25:50 [INFO] [EVAL] The model with the best validation mIoU (0.7903) was saved at iter 20000. 2020-11-24 19:26:41 [INFO] [TRAIN] epoch=31, iter=20100/40000, loss=0.2572, lr=0.005335, batch_cost=0.5129, reader_cost=0.0015 | ETA 02:50:07 2020-11-24 19:27:33 [INFO] [TRAIN] epoch=31, iter=20200/40000, loss=0.2795, lr=0.005311, batch_cost=0.5136, reader_cost=0.0010 | ETA 02:49:29 2020-11-24 19:28:24 [INFO] [TRAIN] epoch=31, iter=20300/40000, loss=0.2804, lr=0.005287, batch_cost=0.5125, reader_cost=0.0008 | ETA 02:48:15 2020-11-24 19:29:15 [INFO] [TRAIN] epoch=31, iter=20400/40000, loss=0.2733, lr=0.005263, batch_cost=0.5129, reader_cost=0.0010 | ETA 02:47:32 2020-11-24 19:30:07 [INFO] [TRAIN] epoch=32, iter=20500/40000, loss=0.2817, lr=0.005238, batch_cost=0.5186, reader_cost=0.0060 | ETA 02:48:33 2020-11-24 19:30:58 [INFO] [TRAIN] epoch=32, iter=20600/40000, loss=0.2547, lr=0.005214, batch_cost=0.5091, reader_cost=0.0006 | ETA 02:44:36 2020-11-24 19:31:49 [INFO] [TRAIN] epoch=32, iter=20700/40000, loss=0.2845, lr=0.005190, batch_cost=0.5113, reader_cost=0.0004 | ETA 02:44:27 2020-11-24 19:32:40 [INFO] [TRAIN] epoch=32, iter=20800/40000, loss=0.2670, lr=0.005166, batch_cost=0.5137, reader_cost=0.0011 | ETA 02:44:22 2020-11-24 19:33:32 [INFO] [TRAIN] epoch=32, iter=20900/40000, loss=0.2559, lr=0.005142, batch_cost=0.5128, reader_cost=0.0008 | ETA 02:43:15 2020-11-24 19:34:23 [INFO] [TRAIN] epoch=32, iter=21000/40000, loss=0.2639, lr=0.005117, batch_cost=0.5148, reader_cost=0.0009 | ETA 02:43:01 2020-11-24 19:35:15 [INFO] [TRAIN] epoch=32, iter=21100/40000, loss=0.2601, lr=0.005093, batch_cost=0.5161, reader_cost=0.0007 | ETA 02:42:34 2020-11-24 19:36:07 [INFO] [TRAIN] epoch=33, iter=21200/40000, loss=0.2884, lr=0.005069, batch_cost=0.5212, reader_cost=0.0063 | ETA 02:43:17 2020-11-24 19:36:58 [INFO] [TRAIN] epoch=33, iter=21300/40000, loss=0.2927, lr=0.005045, batch_cost=0.5155, reader_cost=0.0014 | ETA 02:40:40 2020-11-24 19:37:50 [INFO] [TRAIN] epoch=33, iter=21400/40000, loss=0.2652, lr=0.005020, batch_cost=0.5122, reader_cost=0.0012 | ETA 02:38:46 2020-11-24 19:38:41 [INFO] [TRAIN] epoch=33, iter=21500/40000, loss=0.2914, lr=0.004996, batch_cost=0.5122, reader_cost=0.0011 | ETA 02:37:56 2020-11-24 19:39:32 [INFO] [TRAIN] epoch=33, iter=21600/40000, loss=0.2653, lr=0.004972, batch_cost=0.5122, reader_cost=0.0010 | ETA 02:37:05 2020-11-24 19:40:23 [INFO] [TRAIN] epoch=33, iter=21700/40000, loss=0.2620, lr=0.004947, batch_cost=0.5135, reader_cost=0.0012 | ETA 02:36:36 2020-11-24 19:41:15 [INFO] [TRAIN] epoch=33, iter=21800/40000, loss=0.2528, lr=0.004923, batch_cost=0.5138, reader_cost=0.0010 | ETA 02:35:50 2020-11-24 19:42:07 [INFO] [TRAIN] epoch=34, iter=21900/40000, loss=0.2813, lr=0.004899, batch_cost=0.5221, reader_cost=0.0072 | ETA 02:37:29 2020-11-24 19:42:58 [INFO] [TRAIN] epoch=34, iter=22000/40000, loss=0.2647, lr=0.004874, batch_cost=0.5127, reader_cost=0.0009 | ETA 02:33:49 2020-11-24 19:43:50 [INFO] [TRAIN] epoch=34, iter=22100/40000, loss=0.2466, lr=0.004850, batch_cost=0.5132, reader_cost=0.0012 | ETA 02:33:06 2020-11-24 19:44:41 [INFO] [TRAIN] epoch=34, iter=22200/40000, loss=0.2426, lr=0.004826, batch_cost=0.5149, reader_cost=0.0011 | ETA 02:32:44 2020-11-24 19:45:33 [INFO] [TRAIN] epoch=34, iter=22300/40000, loss=0.2724, lr=0.004801, batch_cost=0.5158, reader_cost=0.0013 | ETA 02:32:08 2020-11-24 19:46:24 [INFO] [TRAIN] epoch=34, iter=22400/40000, loss=0.2447, lr=0.004777, batch_cost=0.5155, reader_cost=0.0013 | ETA 02:31:11 2020-11-24 19:47:16 [INFO] [TRAIN] epoch=35, iter=22500/40000, loss=0.2435, lr=0.004752, batch_cost=0.5178, reader_cost=0.0066 | ETA 02:31:01 2020-11-24 19:48:08 [INFO] [TRAIN] epoch=35, iter=22600/40000, loss=0.2545, lr=0.004728, batch_cost=0.5153, reader_cost=0.0007 | ETA 02:29:26 2020-11-24 19:48:59 [INFO] [TRAIN] epoch=35, iter=22700/40000, loss=0.2394, lr=0.004703, batch_cost=0.5160, reader_cost=0.0012 | ETA 02:28:46 2020-11-24 19:49:50 [INFO] [TRAIN] epoch=35, iter=22800/40000, loss=0.2283, lr=0.004679, batch_cost=0.5073, reader_cost=0.0004 | ETA 02:25:24 2020-11-24 19:50:40 [INFO] [TRAIN] epoch=35, iter=22900/40000, loss=0.2350, lr=0.004654, batch_cost=0.5031, reader_cost=0.0002 | ETA 02:23:22 2020-11-24 19:51:31 [INFO] [TRAIN] epoch=35, iter=23000/40000, loss=0.2361, lr=0.004630, batch_cost=0.5104, reader_cost=0.0009 | ETA 02:24:36 2020-11-24 19:52:23 [INFO] [TRAIN] epoch=35, iter=23100/40000, loss=0.2233, lr=0.004605, batch_cost=0.5138, reader_cost=0.0008 | ETA 02:24:43 2020-11-24 19:53:15 [INFO] [TRAIN] epoch=36, iter=23200/40000, loss=0.2195, lr=0.004581, batch_cost=0.5276, reader_cost=0.0063 | ETA 02:27:43 2020-11-24 19:54:07 [INFO] [TRAIN] epoch=36, iter=23300/40000, loss=0.2276, lr=0.004556, batch_cost=0.5132, reader_cost=0.0010 | ETA 02:22:50 2020-11-24 19:54:58 [INFO] [TRAIN] epoch=36, iter=23400/40000, loss=0.2294, lr=0.004532, batch_cost=0.5151, reader_cost=0.0007 | ETA 02:22:30 2020-11-24 19:55:50 [INFO] [TRAIN] epoch=36, iter=23500/40000, loss=0.2237, lr=0.004507, batch_cost=0.5132, reader_cost=0.0011 | ETA 02:21:07 2020-11-24 19:56:41 [INFO] [TRAIN] epoch=36, iter=23600/40000, loss=0.2231, lr=0.004483, batch_cost=0.5143, reader_cost=0.0009 | ETA 02:20:34 2020-11-24 19:57:32 [INFO] [TRAIN] epoch=36, iter=23700/40000, loss=0.2121, lr=0.004458, batch_cost=0.5140, reader_cost=0.0008 | ETA 02:19:38 2020-11-24 19:58:24 [INFO] [TRAIN] epoch=37, iter=23800/40000, loss=0.2590, lr=0.004433, batch_cost=0.5204, reader_cost=0.0074 | ETA 02:20:30 2020-11-24 19:59:16 [INFO] [TRAIN] epoch=37, iter=23900/40000, loss=0.2256, lr=0.004409, batch_cost=0.5126, reader_cost=0.0008 | ETA 02:17:32 2020-11-24 20:00:07 [INFO] [TRAIN] epoch=37, iter=24000/40000, loss=0.2513, lr=0.004384, batch_cost=0.5136, reader_cost=0.0010 | ETA 02:16:56 2020-11-24 20:00:07 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-24 20:00:29 [INFO] [EVAL] #Images=1449 mIoU=0.7977 Acc=0.9544 Kappa=0.8989 2020-11-24 20:00:29 [INFO] [EVAL] Class IoU: [0.9471 0.8708 0.4446 0.9203 0.7503 0.8352 0.9571 0.8929 0.9388 0.4057 0.9211 0.5098 0.8955 0.9119 0.8777 0.873 0.6921 0.895 0.5547 0.9052 0.7529] 2020-11-24 20:00:29 [INFO] [EVAL] Class Acc: [0.9673 0.8963 0.4595 0.9695 0.8551 0.894 0.9763 0.9509 0.9636 0.7072 0.9563 0.9506 0.9382 0.9418 0.9315 0.9487 0.852 0.9524 0.7775 0.9486 0.9263] 2020-11-24 20:00:37 [INFO] [EVAL] The model with the best validation mIoU (0.7977) was saved at iter 24000. 2020-11-24 20:01:29 [INFO] [TRAIN] epoch=37, iter=24100/40000, loss=0.2117, lr=0.004359, batch_cost=0.5124, reader_cost=0.0011 | ETA 02:15:47 2020-11-24 20:02:20 [INFO] [TRAIN] epoch=37, iter=24200/40000, loss=0.2355, lr=0.004335, batch_cost=0.5127, reader_cost=0.0009 | ETA 02:15:00 2020-11-24 20:03:11 [INFO] [TRAIN] epoch=37, iter=24300/40000, loss=0.2467, lr=0.004310, batch_cost=0.5140, reader_cost=0.0006 | ETA 02:14:30 2020-11-24 20:04:03 [INFO] [TRAIN] epoch=37, iter=24400/40000, loss=0.2108, lr=0.004285, batch_cost=0.5132, reader_cost=0.0004 | ETA 02:13:26 2020-11-24 20:04:55 [INFO] [TRAIN] epoch=38, iter=24500/40000, loss=0.2185, lr=0.004261, batch_cost=0.5231, reader_cost=0.0072 | ETA 02:15:07 2020-11-24 20:05:47 [INFO] [TRAIN] epoch=38, iter=24600/40000, loss=0.2029, lr=0.004236, batch_cost=0.5164, reader_cost=0.0014 | ETA 02:12:31 2020-11-24 20:06:38 [INFO] [TRAIN] epoch=38, iter=24700/40000, loss=0.2335, lr=0.004211, batch_cost=0.5135, reader_cost=0.0010 | ETA 02:10:57 2020-11-24 20:07:29 [INFO] [TRAIN] epoch=38, iter=24800/40000, loss=0.2398, lr=0.004186, batch_cost=0.5119, reader_cost=0.0009 | ETA 02:09:41 2020-11-24 20:08:21 [INFO] [TRAIN] epoch=38, iter=24900/40000, loss=0.2421, lr=0.004162, batch_cost=0.5162, reader_cost=0.0013 | ETA 02:09:54 2020-11-24 20:09:12 [INFO] [TRAIN] epoch=38, iter=25000/40000, loss=0.2039, lr=0.004137, batch_cost=0.5140, reader_cost=0.0012 | ETA 02:08:29 2020-11-24 20:10:04 [INFO] [TRAIN] epoch=38, iter=25100/40000, loss=0.2277, lr=0.004112, batch_cost=0.5145, reader_cost=0.0014 | ETA 02:07:46 2020-11-24 20:10:56 [INFO] [TRAIN] epoch=39, iter=25200/40000, loss=0.2467, lr=0.004087, batch_cost=0.5202, reader_cost=0.0064 | ETA 02:08:19 2020-11-24 20:11:47 [INFO] [TRAIN] epoch=39, iter=25300/40000, loss=0.2639, lr=0.004062, batch_cost=0.5154, reader_cost=0.0015 | ETA 02:06:16 2020-11-24 20:12:39 [INFO] [TRAIN] epoch=39, iter=25400/40000, loss=0.2280, lr=0.004037, batch_cost=0.5128, reader_cost=0.0012 | ETA 02:04:46 2020-11-24 20:13:30 [INFO] [TRAIN] epoch=39, iter=25500/40000, loss=0.2467, lr=0.004012, batch_cost=0.5141, reader_cost=0.0009 | ETA 02:04:14 2020-11-24 20:14:22 [INFO] [TRAIN] epoch=39, iter=25600/40000, loss=0.2222, lr=0.003987, batch_cost=0.5156, reader_cost=0.0012 | ETA 02:03:44 2020-11-24 20:15:13 [INFO] [TRAIN] epoch=39, iter=25700/40000, loss=0.2022, lr=0.003963, batch_cost=0.5146, reader_cost=0.0012 | ETA 02:02:38 2020-11-24 20:16:05 [INFO] [TRAIN] epoch=40, iter=25800/40000, loss=0.2228, lr=0.003938, batch_cost=0.5171, reader_cost=0.0070 | ETA 02:02:22 2020-11-24 20:16:56 [INFO] [TRAIN] epoch=40, iter=25900/40000, loss=0.2246, lr=0.003913, batch_cost=0.5142, reader_cost=0.0009 | ETA 02:00:50 2020-11-24 20:17:48 [INFO] [TRAIN] epoch=40, iter=26000/40000, loss=0.2212, lr=0.003888, batch_cost=0.5154, reader_cost=0.0011 | ETA 02:00:16 2020-11-24 20:18:39 [INFO] [TRAIN] epoch=40, iter=26100/40000, loss=0.2483, lr=0.003863, batch_cost=0.5106, reader_cost=0.0014 | ETA 01:58:17 2020-11-24 20:19:31 [INFO] [TRAIN] epoch=40, iter=26200/40000, loss=0.2243, lr=0.003838, batch_cost=0.5218, reader_cost=0.0008 | ETA 02:00:00 2020-11-24 20:20:22 [INFO] [TRAIN] epoch=40, iter=26300/40000, loss=0.2048, lr=0.003813, batch_cost=0.5142, reader_cost=0.0011 | ETA 01:57:24 2020-11-24 20:21:14 [INFO] [TRAIN] epoch=40, iter=26400/40000, loss=0.2173, lr=0.003788, batch_cost=0.5150, reader_cost=0.0014 | ETA 01:56:43 2020-11-24 20:22:06 [INFO] [TRAIN] epoch=41, iter=26500/40000, loss=0.2234, lr=0.003762, batch_cost=0.5221, reader_cost=0.0069 | ETA 01:57:28 2020-11-24 20:22:57 [INFO] [TRAIN] epoch=41, iter=26600/40000, loss=0.2214, lr=0.003737, batch_cost=0.5140, reader_cost=0.0010 | ETA 01:54:47 2020-11-24 20:23:49 [INFO] [TRAIN] epoch=41, iter=26700/40000, loss=0.2220, lr=0.003712, batch_cost=0.5144, reader_cost=0.0011 | ETA 01:54:01 2020-11-24 20:24:40 [INFO] [TRAIN] epoch=41, iter=26800/40000, loss=0.2225, lr=0.003687, batch_cost=0.5141, reader_cost=0.0012 | ETA 01:53:06 2020-11-24 20:25:31 [INFO] [TRAIN] epoch=41, iter=26900/40000, loss=0.2268, lr=0.003662, batch_cost=0.5119, reader_cost=0.0011 | ETA 01:51:46 2020-11-24 20:26:23 [INFO] [TRAIN] epoch=41, iter=27000/40000, loss=0.2236, lr=0.003637, batch_cost=0.5123, reader_cost=0.0013 | ETA 01:50:59 2020-11-24 20:27:14 [INFO] [TRAIN] epoch=41, iter=27100/40000, loss=0.2116, lr=0.003612, batch_cost=0.5113, reader_cost=0.0011 | ETA 01:49:56 2020-11-24 20:28:06 [INFO] [TRAIN] epoch=42, iter=27200/40000, loss=0.2144, lr=0.003586, batch_cost=0.5175, reader_cost=0.0078 | ETA 01:50:24 2020-11-24 20:28:57 [INFO] [TRAIN] epoch=42, iter=27300/40000, loss=0.2240, lr=0.003561, batch_cost=0.5169, reader_cost=0.0010 | ETA 01:49:24 2020-11-24 20:29:49 [INFO] [TRAIN] epoch=42, iter=27400/40000, loss=0.2140, lr=0.003536, batch_cost=0.5122, reader_cost=0.0010 | ETA 01:47:33 2020-11-24 20:30:40 [INFO] [TRAIN] epoch=42, iter=27500/40000, loss=0.2317, lr=0.003511, batch_cost=0.5132, reader_cost=0.0006 | ETA 01:46:55 2020-11-24 20:31:31 [INFO] [TRAIN] epoch=42, iter=27600/40000, loss=0.2110, lr=0.003485, batch_cost=0.5130, reader_cost=0.0011 | ETA 01:46:01 2020-11-24 20:32:22 [INFO] [TRAIN] epoch=42, iter=27700/40000, loss=0.1909, lr=0.003460, batch_cost=0.5097, reader_cost=0.0008 | ETA 01:44:29 2020-11-24 20:33:14 [INFO] [TRAIN] epoch=43, iter=27800/40000, loss=0.2208, lr=0.003435, batch_cost=0.5186, reader_cost=0.0068 | ETA 01:45:27 2020-11-24 20:34:05 [INFO] [TRAIN] epoch=43, iter=27900/40000, loss=0.1991, lr=0.003409, batch_cost=0.5110, reader_cost=0.0008 | ETA 01:43:03 2020-11-24 20:34:56 [INFO] [TRAIN] epoch=43, iter=28000/40000, loss=0.2151, lr=0.003384, batch_cost=0.5131, reader_cost=0.0011 | ETA 01:42:37 2020-11-24 20:34:56 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-24 20:35:19 [INFO] [EVAL] #Images=1449 mIoU=0.7959 Acc=0.9550 Kappa=0.9003 2020-11-24 20:35:19 [INFO] [EVAL] Class IoU: [0.9493 0.8951 0.4454 0.9232 0.7042 0.8435 0.9587 0.8986 0.9359 0.4492 0.9115 0.452 0.9069 0.9102 0.8772 0.873 0.6044 0.9054 0.5783 0.9089 0.7835] 2020-11-24 20:35:19 [INFO] [EVAL] Class Acc: [0.969 0.9247 0.4618 0.9634 0.8546 0.8994 0.979 0.9579 0.96 0.6197 0.9667 0.9706 0.9467 0.95 0.9584 0.9447 0.8555 0.9565 0.7654 0.954 0.9182] 2020-11-24 20:35:24 [INFO] [EVAL] The model with the best validation mIoU (0.7977) was saved at iter 24000. 2020-11-24 20:36:15 [INFO] [TRAIN] epoch=43, iter=28100/40000, loss=0.2147, lr=0.003359, batch_cost=0.5114, reader_cost=0.0012 | ETA 01:41:26 2020-11-24 20:37:06 [INFO] [TRAIN] epoch=43, iter=28200/40000, loss=0.2376, lr=0.003333, batch_cost=0.5117, reader_cost=0.0013 | ETA 01:40:37 2020-11-24 20:37:57 [INFO] [TRAIN] epoch=43, iter=28300/40000, loss=0.2240, lr=0.003308, batch_cost=0.5139, reader_cost=0.0015 | ETA 01:40:12 2020-11-24 20:38:49 [INFO] [TRAIN] epoch=43, iter=28400/40000, loss=0.1898, lr=0.003282, batch_cost=0.5119, reader_cost=0.0013 | ETA 01:38:58 2020-11-24 20:39:40 [INFO] [TRAIN] epoch=44, iter=28500/40000, loss=0.2005, lr=0.003257, batch_cost=0.5107, reader_cost=0.0069 | ETA 01:37:53 2020-11-24 20:40:30 [INFO] [TRAIN] epoch=44, iter=28600/40000, loss=0.2278, lr=0.003231, batch_cost=0.5009, reader_cost=0.0002 | ETA 01:35:10 2020-11-24 20:41:21 [INFO] [TRAIN] epoch=44, iter=28700/40000, loss=0.2012, lr=0.003206, batch_cost=0.5095, reader_cost=0.0003 | ETA 01:35:57 2020-11-24 20:42:12 [INFO] [TRAIN] epoch=44, iter=28800/40000, loss=0.2086, lr=0.003180, batch_cost=0.5142, reader_cost=0.0010 | ETA 01:35:59 2020-11-24 20:43:04 [INFO] [TRAIN] epoch=44, iter=28900/40000, loss=0.2161, lr=0.003155, batch_cost=0.5138, reader_cost=0.0012 | ETA 01:35:03 2020-11-24 20:43:55 [INFO] [TRAIN] epoch=44, iter=29000/40000, loss=0.2045, lr=0.003129, batch_cost=0.5129, reader_cost=0.0010 | ETA 01:34:01 2020-11-24 20:44:47 [INFO] [TRAIN] epoch=45, iter=29100/40000, loss=0.2139, lr=0.003104, batch_cost=0.5201, reader_cost=0.0066 | ETA 01:34:29 2020-11-24 20:45:38 [INFO] [TRAIN] epoch=45, iter=29200/40000, loss=0.1961, lr=0.003078, batch_cost=0.5142, reader_cost=0.0007 | ETA 01:32:33 2020-11-24 20:46:30 [INFO] [TRAIN] epoch=45, iter=29300/40000, loss=0.1993, lr=0.003052, batch_cost=0.5121, reader_cost=0.0012 | ETA 01:31:19 2020-11-24 20:47:21 [INFO] [TRAIN] epoch=45, iter=29400/40000, loss=0.2164, lr=0.003027, batch_cost=0.5136, reader_cost=0.0014 | ETA 01:30:43 2020-11-24 20:48:12 [INFO] [TRAIN] epoch=45, iter=29500/40000, loss=0.2352, lr=0.003001, batch_cost=0.5157, reader_cost=0.0013 | ETA 01:30:15 2020-11-24 20:49:04 [INFO] [TRAIN] epoch=45, iter=29600/40000, loss=0.2092, lr=0.002975, batch_cost=0.5131, reader_cost=0.0015 | ETA 01:28:56 2020-11-24 20:49:55 [INFO] [TRAIN] epoch=45, iter=29700/40000, loss=0.1900, lr=0.002949, batch_cost=0.5100, reader_cost=0.0010 | ETA 01:27:33 2020-11-24 20:50:46 [INFO] [TRAIN] epoch=46, iter=29800/40000, loss=0.1998, lr=0.002924, batch_cost=0.5159, reader_cost=0.0065 | ETA 01:27:41 2020-11-24 20:51:37 [INFO] [TRAIN] epoch=46, iter=29900/40000, loss=0.2123, lr=0.002898, batch_cost=0.5104, reader_cost=0.0005 | ETA 01:25:55 2020-11-24 20:52:28 [INFO] [TRAIN] epoch=46, iter=30000/40000, loss=0.2085, lr=0.002872, batch_cost=0.5103, reader_cost=0.0004 | ETA 01:25:03 2020-11-24 20:53:19 [INFO] [TRAIN] epoch=46, iter=30100/40000, loss=0.2041, lr=0.002846, batch_cost=0.5103, reader_cost=0.0006 | ETA 01:24:12 2020-11-24 20:54:11 [INFO] [TRAIN] epoch=46, iter=30200/40000, loss=0.2075, lr=0.002820, batch_cost=0.5106, reader_cost=0.0006 | ETA 01:23:23 2020-11-24 20:55:02 [INFO] [TRAIN] epoch=46, iter=30300/40000, loss=0.2096, lr=0.002794, batch_cost=0.5125, reader_cost=0.0007 | ETA 01:22:51 2020-11-24 20:55:53 [INFO] [TRAIN] epoch=46, iter=30400/40000, loss=0.2360, lr=0.002768, batch_cost=0.5122, reader_cost=0.0014 | ETA 01:21:57 2020-11-24 20:56:45 [INFO] [TRAIN] epoch=47, iter=30500/40000, loss=0.2158, lr=0.002742, batch_cost=0.5176, reader_cost=0.0060 | ETA 01:21:57 2020-11-24 20:57:36 [INFO] [TRAIN] epoch=47, iter=30600/40000, loss=0.2233, lr=0.002716, batch_cost=0.5075, reader_cost=0.0005 | ETA 01:19:30 2020-11-24 20:58:27 [INFO] [TRAIN] epoch=47, iter=30700/40000, loss=0.2095, lr=0.002690, batch_cost=0.5098, reader_cost=0.0005 | ETA 01:19:01 2020-11-24 20:59:18 [INFO] [TRAIN] epoch=47, iter=30800/40000, loss=0.2083, lr=0.002664, batch_cost=0.5113, reader_cost=0.0004 | ETA 01:18:23 2020-11-24 21:00:09 [INFO] [TRAIN] epoch=47, iter=30900/40000, loss=0.1950, lr=0.002638, batch_cost=0.5114, reader_cost=0.0004 | ETA 01:17:33 2020-11-24 21:01:00 [INFO] [TRAIN] epoch=47, iter=31000/40000, loss=0.2070, lr=0.002612, batch_cost=0.5132, reader_cost=0.0012 | ETA 01:16:58 2020-11-24 21:01:52 [INFO] [TRAIN] epoch=48, iter=31100/40000, loss=0.2052, lr=0.002586, batch_cost=0.5189, reader_cost=0.0064 | ETA 01:16:58 2020-11-24 21:02:44 [INFO] [TRAIN] epoch=48, iter=31200/40000, loss=0.1987, lr=0.002560, batch_cost=0.5157, reader_cost=0.0004 | ETA 01:15:38 2020-11-24 21:03:35 [INFO] [TRAIN] epoch=48, iter=31300/40000, loss=0.1891, lr=0.002534, batch_cost=0.5132, reader_cost=0.0005 | ETA 01:14:25 2020-11-24 21:04:26 [INFO] [TRAIN] epoch=48, iter=31400/40000, loss=0.2116, lr=0.002507, batch_cost=0.5129, reader_cost=0.0004 | ETA 01:13:31 2020-11-24 21:05:17 [INFO] [TRAIN] epoch=48, iter=31500/40000, loss=0.1956, lr=0.002481, batch_cost=0.5121, reader_cost=0.0006 | ETA 01:12:32 2020-11-24 21:06:09 [INFO] [TRAIN] epoch=48, iter=31600/40000, loss=0.2085, lr=0.002455, batch_cost=0.5128, reader_cost=0.0006 | ETA 01:11:47 2020-11-24 21:07:00 [INFO] [TRAIN] epoch=48, iter=31700/40000, loss=0.1913, lr=0.002429, batch_cost=0.5135, reader_cost=0.0005 | ETA 01:11:02 2020-11-24 21:07:52 [INFO] [TRAIN] epoch=49, iter=31800/40000, loss=0.2165, lr=0.002402, batch_cost=0.5182, reader_cost=0.0064 | ETA 01:10:48 2020-11-24 21:08:43 [INFO] [TRAIN] epoch=49, iter=31900/40000, loss=0.1938, lr=0.002376, batch_cost=0.5139, reader_cost=0.0007 | ETA 01:09:22 2020-11-24 21:09:35 [INFO] [TRAIN] epoch=49, iter=32000/40000, loss=0.2021, lr=0.002350, batch_cost=0.5160, reader_cost=0.0005 | ETA 01:08:48 2020-11-24 21:09:35 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-24 21:09:57 [INFO] [EVAL] #Images=1449 mIoU=0.8068 Acc=0.9566 Kappa=0.9036 2020-11-24 21:09:57 [INFO] [EVAL] Class IoU: [0.9496 0.9213 0.4514 0.9151 0.6861 0.8429 0.9568 0.8949 0.9471 0.4958 0.9234 0.5296 0.9177 0.916 0.8772 0.8779 0.7022 0.9137 0.5201 0.9086 0.7946] 2020-11-24 21:09:57 [INFO] [EVAL] Class Acc: [0.9684 0.9522 0.4683 0.9532 0.8195 0.9018 0.9815 0.96 0.9688 0.7036 0.9677 0.9054 0.9482 0.9558 0.942 0.9481 0.8434 0.9609 0.8093 0.9551 0.9512] 2020-11-24 21:10:06 [INFO] [EVAL] The model with the best validation mIoU (0.8068) was saved at iter 32000. 2020-11-24 21:10:57 [INFO] [TRAIN] epoch=49, iter=32100/40000, loss=0.2058, lr=0.002323, batch_cost=0.5127, reader_cost=0.0009 | ETA 01:07:30 2020-11-24 21:11:49 [INFO] [TRAIN] epoch=49, iter=32200/40000, loss=0.2147, lr=0.002297, batch_cost=0.5122, reader_cost=0.0010 | ETA 01:06:35 2020-11-24 21:12:40 [INFO] [TRAIN] epoch=49, iter=32300/40000, loss=0.1871, lr=0.002270, batch_cost=0.5144, reader_cost=0.0006 | ETA 01:06:00 2020-11-24 21:13:32 [INFO] [TRAIN] epoch=50, iter=32400/40000, loss=0.1988, lr=0.002244, batch_cost=0.5194, reader_cost=0.0063 | ETA 01:05:47 2020-11-24 21:14:23 [INFO] [TRAIN] epoch=50, iter=32500/40000, loss=0.1868, lr=0.002217, batch_cost=0.5113, reader_cost=0.0005 | ETA 01:03:55 2020-11-24 21:15:15 [INFO] [TRAIN] epoch=50, iter=32600/40000, loss=0.2220, lr=0.002190, batch_cost=0.5177, reader_cost=0.0008 | ETA 01:03:50 2020-11-24 21:16:06 [INFO] [TRAIN] epoch=50, iter=32700/40000, loss=0.2061, lr=0.002164, batch_cost=0.5106, reader_cost=0.0005 | ETA 01:02:07 2020-11-24 21:16:57 [INFO] [TRAIN] epoch=50, iter=32800/40000, loss=0.2007, lr=0.002137, batch_cost=0.5090, reader_cost=0.0004 | ETA 01:01:04 2020-11-24 21:17:48 [INFO] [TRAIN] epoch=50, iter=32900/40000, loss=0.2036, lr=0.002110, batch_cost=0.5102, reader_cost=0.0008 | ETA 01:00:22 2020-11-24 21:18:39 [INFO] [TRAIN] epoch=50, iter=33000/40000, loss=0.1729, lr=0.002083, batch_cost=0.5097, reader_cost=0.0006 | ETA 00:59:27 2020-11-24 21:19:31 [INFO] [TRAIN] epoch=51, iter=33100/40000, loss=0.1922, lr=0.002057, batch_cost=0.5180, reader_cost=0.0069 | ETA 00:59:34 2020-11-24 21:20:22 [INFO] [TRAIN] epoch=51, iter=33200/40000, loss=0.1963, lr=0.002030, batch_cost=0.5102, reader_cost=0.0005 | ETA 00:57:49 2020-11-24 21:21:13 [INFO] [TRAIN] epoch=51, iter=33300/40000, loss=0.1930, lr=0.002003, batch_cost=0.5128, reader_cost=0.0014 | ETA 00:57:15 2020-11-24 21:22:04 [INFO] [TRAIN] epoch=51, iter=33400/40000, loss=0.1962, lr=0.001976, batch_cost=0.5122, reader_cost=0.0005 | ETA 00:56:20 2020-11-24 21:22:55 [INFO] [TRAIN] epoch=51, iter=33500/40000, loss=0.2044, lr=0.001949, batch_cost=0.5118, reader_cost=0.0003 | ETA 00:55:26 2020-11-24 21:23:47 [INFO] [TRAIN] epoch=51, iter=33600/40000, loss=0.1854, lr=0.001922, batch_cost=0.5122, reader_cost=0.0002 | ETA 00:54:38 2020-11-24 21:24:38 [INFO] [TRAIN] epoch=51, iter=33700/40000, loss=0.1746, lr=0.001895, batch_cost=0.5097, reader_cost=0.0004 | ETA 00:53:30 2020-11-24 21:25:29 [INFO] [TRAIN] epoch=52, iter=33800/40000, loss=0.1885, lr=0.001868, batch_cost=0.5186, reader_cost=0.0065 | ETA 00:53:35 2020-11-24 21:26:21 [INFO] [TRAIN] epoch=52, iter=33900/40000, loss=0.1925, lr=0.001841, batch_cost=0.5122, reader_cost=0.0010 | ETA 00:52:04 2020-11-24 21:27:12 [INFO] [TRAIN] epoch=52, iter=34000/40000, loss=0.1813, lr=0.001814, batch_cost=0.5098, reader_cost=0.0008 | ETA 00:50:58 2020-11-24 21:28:03 [INFO] [TRAIN] epoch=52, iter=34100/40000, loss=0.1770, lr=0.001786, batch_cost=0.5106, reader_cost=0.0007 | ETA 00:50:12 2020-11-24 21:28:54 [INFO] [TRAIN] epoch=52, iter=34200/40000, loss=0.1888, lr=0.001759, batch_cost=0.5111, reader_cost=0.0005 | ETA 00:49:24 2020-11-24 21:29:44 [INFO] [TRAIN] epoch=52, iter=34300/40000, loss=0.1762, lr=0.001732, batch_cost=0.5023, reader_cost=0.0002 | ETA 00:47:43 2020-11-24 21:30:35 [INFO] [TRAIN] epoch=53, iter=34400/40000, loss=0.1901, lr=0.001704, batch_cost=0.5122, reader_cost=0.0067 | ETA 00:47:48 2020-11-24 21:31:26 [INFO] [TRAIN] epoch=53, iter=34500/40000, loss=0.1741, lr=0.001677, batch_cost=0.5106, reader_cost=0.0005 | ETA 00:46:48 2020-11-24 21:32:17 [INFO] [TRAIN] epoch=53, iter=34600/40000, loss=0.1834, lr=0.001650, batch_cost=0.5112, reader_cost=0.0004 | ETA 00:46:00 2020-11-24 21:33:08 [INFO] [TRAIN] epoch=53, iter=34700/40000, loss=0.1821, lr=0.001622, batch_cost=0.5106, reader_cost=0.0005 | ETA 00:45:06 2020-11-24 21:34:00 [INFO] [TRAIN] epoch=53, iter=34800/40000, loss=0.2016, lr=0.001594, batch_cost=0.5130, reader_cost=0.0005 | ETA 00:44:27 2020-11-24 21:34:51 [INFO] [TRAIN] epoch=53, iter=34900/40000, loss=0.1931, lr=0.001567, batch_cost=0.5121, reader_cost=0.0004 | ETA 00:43:31 2020-11-24 21:35:42 [INFO] [TRAIN] epoch=53, iter=35000/40000, loss=0.1849, lr=0.001539, batch_cost=0.5100, reader_cost=0.0003 | ETA 00:42:29 2020-11-24 21:36:34 [INFO] [TRAIN] epoch=54, iter=35100/40000, loss=0.1890, lr=0.001511, batch_cost=0.5178, reader_cost=0.0068 | ETA 00:42:17 2020-11-24 21:37:25 [INFO] [TRAIN] epoch=54, iter=35200/40000, loss=0.1834, lr=0.001484, batch_cost=0.5125, reader_cost=0.0004 | ETA 00:40:59 2020-11-24 21:38:16 [INFO] [TRAIN] epoch=54, iter=35300/40000, loss=0.1839, lr=0.001456, batch_cost=0.5084, reader_cost=0.0004 | ETA 00:39:49 2020-11-24 21:39:07 [INFO] [TRAIN] epoch=54, iter=35400/40000, loss=0.1996, lr=0.001428, batch_cost=0.5086, reader_cost=0.0005 | ETA 00:38:59 2020-11-24 21:39:58 [INFO] [TRAIN] epoch=54, iter=35500/40000, loss=0.1757, lr=0.001400, batch_cost=0.5105, reader_cost=0.0006 | ETA 00:38:17 2020-11-24 21:40:49 [INFO] [TRAIN] epoch=54, iter=35600/40000, loss=0.1900, lr=0.001372, batch_cost=0.5100, reader_cost=0.0006 | ETA 00:37:23 2020-11-24 21:41:40 [INFO] [TRAIN] epoch=55, iter=35700/40000, loss=0.1809, lr=0.001344, batch_cost=0.5158, reader_cost=0.0061 | ETA 00:36:57 2020-11-24 21:42:31 [INFO] [TRAIN] epoch=55, iter=35800/40000, loss=0.1926, lr=0.001316, batch_cost=0.5092, reader_cost=0.0003 | ETA 00:35:38 2020-11-24 21:43:23 [INFO] [TRAIN] epoch=55, iter=35900/40000, loss=0.1902, lr=0.001287, batch_cost=0.5164, reader_cost=0.0004 | ETA 00:35:17 2020-11-24 21:44:14 [INFO] [TRAIN] epoch=55, iter=36000/40000, loss=0.1809, lr=0.001259, batch_cost=0.5085, reader_cost=0.0004 | ETA 00:33:53 2020-11-24 21:44:14 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-24 21:44:36 [INFO] [EVAL] #Images=1449 mIoU=0.8018 Acc=0.9552 Kappa=0.9001 2020-11-24 21:44:36 [INFO] [EVAL] Class IoU: [0.9479 0.9225 0.4471 0.9255 0.7355 0.8455 0.9357 0.8967 0.947 0.4832 0.9185 0.4718 0.9209 0.917 0.8867 0.8767 0.6728 0.9102 0.4853 0.8981 0.7923] 2020-11-24 21:44:36 [INFO] [EVAL] Class Acc: [0.9655 0.961 0.4616 0.9652 0.8107 0.9058 0.9598 0.9508 0.9711 0.7476 0.9671 0.9143 0.9534 0.9553 0.9473 0.9485 0.8746 0.9636 0.8051 0.9664 0.9455] 2020-11-24 21:44:42 [INFO] [EVAL] The model with the best validation mIoU (0.8068) was saved at iter 32000. 2020-11-24 21:45:33 [INFO] [TRAIN] epoch=55, iter=36100/40000, loss=0.1786, lr=0.001231, batch_cost=0.5103, reader_cost=0.0010 | ETA 00:33:09 2020-11-24 21:46:24 [INFO] [TRAIN] epoch=55, iter=36200/40000, loss=0.2031, lr=0.001202, batch_cost=0.5103, reader_cost=0.0016 | ETA 00:32:19 2020-11-24 21:47:15 [INFO] [TRAIN] epoch=55, iter=36300/40000, loss=0.1833, lr=0.001174, batch_cost=0.5075, reader_cost=0.0010 | ETA 00:31:17 2020-11-24 21:48:07 [INFO] [TRAIN] epoch=56, iter=36400/40000, loss=0.1546, lr=0.001145, batch_cost=0.5166, reader_cost=0.0066 | ETA 00:30:59 2020-11-24 21:48:58 [INFO] [TRAIN] epoch=56, iter=36500/40000, loss=0.1668, lr=0.001117, batch_cost=0.5097, reader_cost=0.0009 | ETA 00:29:44 2020-11-24 21:49:49 [INFO] [TRAIN] epoch=56, iter=36600/40000, loss=0.1894, lr=0.001088, batch_cost=0.5093, reader_cost=0.0006 | ETA 00:28:51 2020-11-24 21:50:40 [INFO] [TRAIN] epoch=56, iter=36700/40000, loss=0.1853, lr=0.001059, batch_cost=0.5114, reader_cost=0.0006 | ETA 00:28:07 2020-11-24 21:51:31 [INFO] [TRAIN] epoch=56, iter=36800/40000, loss=0.1817, lr=0.001030, batch_cost=0.5112, reader_cost=0.0012 | ETA 00:27:15 2020-11-24 21:52:22 [INFO] [TRAIN] epoch=56, iter=36900/40000, loss=0.1908, lr=0.001001, batch_cost=0.5130, reader_cost=0.0007 | ETA 00:26:30 2020-11-24 21:53:13 [INFO] [TRAIN] epoch=56, iter=37000/40000, loss=0.1670, lr=0.000972, batch_cost=0.5123, reader_cost=0.0010 | ETA 00:25:36 2020-11-24 21:54:05 [INFO] [TRAIN] epoch=57, iter=37100/40000, loss=0.1896, lr=0.000943, batch_cost=0.5195, reader_cost=0.0066 | ETA 00:25:06 2020-11-24 21:54:57 [INFO] [TRAIN] epoch=57, iter=37200/40000, loss=0.1748, lr=0.000914, batch_cost=0.5155, reader_cost=0.0009 | ETA 00:24:03 2020-11-24 21:55:48 [INFO] [TRAIN] epoch=57, iter=37300/40000, loss=0.1832, lr=0.000884, batch_cost=0.5115, reader_cost=0.0009 | ETA 00:23:00 2020-11-24 21:56:40 [INFO] [TRAIN] epoch=57, iter=37400/40000, loss=0.1939, lr=0.000855, batch_cost=0.5137, reader_cost=0.0008 | ETA 00:22:15 2020-11-24 21:57:31 [INFO] [TRAIN] epoch=57, iter=37500/40000, loss=0.1908, lr=0.000825, batch_cost=0.5124, reader_cost=0.0012 | ETA 00:21:20 2020-11-24 21:58:22 [INFO] [TRAIN] epoch=57, iter=37600/40000, loss=0.1779, lr=0.000795, batch_cost=0.5148, reader_cost=0.0012 | ETA 00:20:35 2020-11-24 21:59:14 [INFO] [TRAIN] epoch=58, iter=37700/40000, loss=0.1934, lr=0.000765, batch_cost=0.5184, reader_cost=0.0070 | ETA 00:19:52 2020-11-24 22:00:05 [INFO] [TRAIN] epoch=58, iter=37800/40000, loss=0.1713, lr=0.000735, batch_cost=0.5137, reader_cost=0.0008 | ETA 00:18:50 2020-11-24 22:00:57 [INFO] [TRAIN] epoch=58, iter=37900/40000, loss=0.1716, lr=0.000705, batch_cost=0.5134, reader_cost=0.0005 | ETA 00:17:58 2020-11-24 22:01:48 [INFO] [TRAIN] epoch=58, iter=38000/40000, loss=0.1879, lr=0.000675, batch_cost=0.5114, reader_cost=0.0004 | ETA 00:17:02 2020-11-24 22:02:39 [INFO] [TRAIN] epoch=58, iter=38100/40000, loss=0.1896, lr=0.000645, batch_cost=0.5132, reader_cost=0.0006 | ETA 00:16:15 2020-11-24 22:03:31 [INFO] [TRAIN] epoch=58, iter=38200/40000, loss=0.1754, lr=0.000614, batch_cost=0.5143, reader_cost=0.0009 | ETA 00:15:25 2020-11-24 22:04:22 [INFO] [TRAIN] epoch=58, iter=38300/40000, loss=0.1650, lr=0.000583, batch_cost=0.5122, reader_cost=0.0007 | ETA 00:14:30 2020-11-24 22:05:14 [INFO] [TRAIN] epoch=59, iter=38400/40000, loss=0.1809, lr=0.000552, batch_cost=0.5162, reader_cost=0.0067 | ETA 00:13:45 2020-11-24 22:06:05 [INFO] [TRAIN] epoch=59, iter=38500/40000, loss=0.1708, lr=0.000521, batch_cost=0.5134, reader_cost=0.0011 | ETA 00:12:50 2020-11-24 22:06:56 [INFO] [TRAIN] epoch=59, iter=38600/40000, loss=0.1911, lr=0.000490, batch_cost=0.5136, reader_cost=0.0005 | ETA 00:11:59 2020-11-24 22:07:47 [INFO] [TRAIN] epoch=59, iter=38700/40000, loss=0.1817, lr=0.000458, batch_cost=0.5117, reader_cost=0.0004 | ETA 00:11:05 2020-11-24 22:08:39 [INFO] [TRAIN] epoch=59, iter=38800/40000, loss=0.1854, lr=0.000426, batch_cost=0.5125, reader_cost=0.0007 | ETA 00:10:14 2020-11-24 22:09:30 [INFO] [TRAIN] epoch=59, iter=38900/40000, loss=0.1811, lr=0.000394, batch_cost=0.5122, reader_cost=0.0003 | ETA 00:09:23 2020-11-24 22:10:22 [INFO] [TRAIN] epoch=60, iter=39000/40000, loss=0.1893, lr=0.000362, batch_cost=0.5219, reader_cost=0.0060 | ETA 00:08:41 2020-11-24 22:11:13 [INFO] [TRAIN] epoch=60, iter=39100/40000, loss=0.1808, lr=0.000329, batch_cost=0.5115, reader_cost=0.0005 | ETA 00:07:40 2020-11-24 22:12:04 [INFO] [TRAIN] epoch=60, iter=39200/40000, loss=0.1754, lr=0.000296, batch_cost=0.5108, reader_cost=0.0004 | ETA 00:06:48 2020-11-24 22:12:56 [INFO] [TRAIN] epoch=60, iter=39300/40000, loss=0.1790, lr=0.000263, batch_cost=0.5151, reader_cost=0.0004 | ETA 00:06:00 2020-11-24 22:13:47 [INFO] [TRAIN] epoch=60, iter=39400/40000, loss=0.1854, lr=0.000229, batch_cost=0.5134, reader_cost=0.0009 | ETA 00:05:08 2020-11-24 22:14:38 [INFO] [TRAIN] epoch=60, iter=39500/40000, loss=0.1861, lr=0.000194, batch_cost=0.5135, reader_cost=0.0006 | ETA 00:04:16 2020-11-24 22:15:30 [INFO] [TRAIN] epoch=60, iter=39600/40000, loss=0.1774, lr=0.000159, batch_cost=0.5121, reader_cost=0.0005 | ETA 00:03:24 2020-11-24 22:16:22 [INFO] [TRAIN] epoch=61, iter=39700/40000, loss=0.1835, lr=0.000123, batch_cost=0.5186, reader_cost=0.0062 | ETA 00:02:35 2020-11-24 22:17:13 [INFO] [TRAIN] epoch=61, iter=39800/40000, loss=0.1695, lr=0.000085, batch_cost=0.5117, reader_cost=0.0006 | ETA 00:01:42 2020-11-24 22:18:04 [INFO] [TRAIN] epoch=61, iter=39900/40000, loss=0.1898, lr=0.000046, batch_cost=0.5146, reader_cost=0.0010 | ETA 00:00:51 2020-11-24 22:18:55 [INFO] [TRAIN] epoch=61, iter=40000/40000, loss=0.1828, lr=0.000001, batch_cost=0.5123, reader_cost=0.0009 | ETA 00:00:00 2020-11-24 22:18:55 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-24 22:19:17 [INFO] [EVAL] #Images=1449 mIoU=0.8082 Acc=0.9568 Kappa=0.9038 2020-11-24 22:19:17 [INFO] [EVAL] Class IoU: [0.9491 0.9206 0.449 0.9276 0.7153 0.8448 0.9564 0.9052 0.9478 0.5066 0.9227 0.5182 0.9217 0.9208 0.8882 0.8775 0.6709 0.9101 0.4996 0.9184 0.8018] 2020-11-24 22:19:17 [INFO] [EVAL] Class Acc: [0.9673 0.9529 0.4647 0.9612 0.8277 0.8986 0.9755 0.9632 0.9708 0.7501 0.9663 0.9071 0.9542 0.9602 0.9454 0.9456 0.8545 0.9524 0.821 0.9631 0.9579] 2020-11-24 22:19:24 [INFO] [EVAL] The model with the best validation mIoU (0.8082) was saved at iter 40000.