2020-11-25 19:52:40 [INFO] ------------Environment Information------------- platform: Linux-3.10.0_3-0-0-34-x86_64-with-centos-7.5.1804-Core Python: 3.7.9 (default, Aug 31 2020, 12:42:55) [GCC 7.3.0] Paddle compiled with cuda: True NVCC: Cuda compilation tools, release 10.2, V10.2.89 cudnn: 7.6 GPUs used: 4 CUDA_VISIBLE_DEVICES: 0,1,2,3 GPU: ['GPU 0: Tesla V100-SXM2-16GB', 'GPU 1: Tesla V100-SXM2-16GB', 'GPU 2: Tesla V100-SXM2-16GB', 'GPU 3: Tesla V100-SXM2-16GB', 'GPU 4: Tesla V100-SXM2-16GB', 'GPU 5: Tesla V100-SXM2-16GB', 'GPU 6: Tesla V100-SXM2-16GB', 'GPU 7: Tesla V100-SXM2-16GB'] GCC: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39) PaddlePaddle: 2.0.0-rc0 OpenCV: 4.1.0 ------------------------------------------------ 2020-11-25 19:52:40 [INFO] ---------------Config Information--------------- batch_size: 4 iters: 40000 learning_rate: decay: end_lr: 0.0 power: 0.9 type: poly value: 0.01 loss: coef: - 1 - 0.4 types: - ignore_index: 255 type: CrossEntropyLoss model: align_corners: false backbone: output_stride: 8 pretrained: https://bj.bcebos.com/paddleseg/dygraph/resnet50_vd_ssld_v2.tar.gz type: ResNet50_vd enable_auxiliary_loss: true pretrained: null type: PSPNet optimizer: momentum: 0.9 type: sgd weight_decay: 4.0e-05 train_dataset: dataset_root: data/VOCdevkit/ mode: trainaug transforms: - max_scale_factor: 2.0 min_scale_factor: 0.5 scale_step_size: 0.25 type: ResizeStepScaling - crop_size: - 512 - 512 type: RandomPaddingCrop - type: RandomHorizontalFlip - brightness_range: 0.4 contrast_range: 0.4 saturation_range: 0.4 type: RandomDistort - type: Normalize type: PascalVOC val_dataset: dataset_root: data/VOCdevkit/ mode: val transforms: - target_size: - 512 - 512 type: Padding - type: Normalize type: PascalVOC ------------------------------------------------ 2020-11-25 19:52:45 [INFO] Loading pretrained model from https://bj.bcebos.com/paddleseg/dygraph/resnet50_vd_ssld_v2.tar.gz 2020-11-25 19:52:47 [INFO] There are 275/275 variables loaded into ResNet_vd. 2020-11-25 19:53:52 [INFO] [TRAIN] epoch=1, iter=100/40000, loss=1.9651, lr=0.009978, batch_cost=0.5845, reader_cost=0.0139 | ETA 06:28:42 2020-11-25 19:54:46 [INFO] [TRAIN] epoch=1, iter=200/40000, loss=1.5565, lr=0.009955, batch_cost=0.5453, reader_cost=0.0015 | ETA 06:01:43 2020-11-25 19:55:47 [INFO] [TRAIN] epoch=1, iter=300/40000, loss=1.2259, lr=0.009933, batch_cost=0.6084, reader_cost=0.0036 | ETA 06:42:33 2020-11-25 19:56:39 [INFO] [TRAIN] epoch=1, iter=400/40000, loss=1.1183, lr=0.009910, batch_cost=0.5222, reader_cost=0.0007 | ETA 05:44:39 2020-11-25 19:57:32 [INFO] [TRAIN] epoch=1, iter=500/40000, loss=0.9893, lr=0.009888, batch_cost=0.5255, reader_cost=0.0009 | ETA 05:45:56 2020-11-25 19:58:24 [INFO] [TRAIN] epoch=1, iter=600/40000, loss=0.9753, lr=0.009865, batch_cost=0.5246, reader_cost=0.0011 | ETA 05:44:31 2020-11-25 19:59:17 [INFO] [TRAIN] epoch=2, iter=700/40000, loss=0.8346, lr=0.009843, batch_cost=0.5287, reader_cost=0.0073 | ETA 05:46:17 2020-11-25 20:00:10 [INFO] [TRAIN] epoch=2, iter=800/40000, loss=0.9362, lr=0.009820, batch_cost=0.5228, reader_cost=0.0008 | ETA 05:41:35 2020-11-25 20:01:02 [INFO] [TRAIN] epoch=2, iter=900/40000, loss=0.8356, lr=0.009797, batch_cost=0.5244, reader_cost=0.0009 | ETA 05:41:43 2020-11-25 20:01:55 [INFO] [TRAIN] epoch=2, iter=1000/40000, loss=0.7539, lr=0.009775, batch_cost=0.5249, reader_cost=0.0009 | ETA 05:41:10 2020-11-25 20:02:47 [INFO] [TRAIN] epoch=2, iter=1100/40000, loss=0.7839, lr=0.009752, batch_cost=0.5216, reader_cost=0.0007 | ETA 05:38:11 2020-11-25 20:03:39 [INFO] [TRAIN] epoch=2, iter=1200/40000, loss=0.7998, lr=0.009730, batch_cost=0.5232, reader_cost=0.0006 | ETA 05:38:21 2020-11-25 20:04:31 [INFO] [TRAIN] epoch=2, iter=1300/40000, loss=0.7581, lr=0.009707, batch_cost=0.5203, reader_cost=0.0009 | ETA 05:35:37 2020-11-25 20:05:24 [INFO] [TRAIN] epoch=3, iter=1400/40000, loss=0.8062, lr=0.009685, batch_cost=0.5278, reader_cost=0.0064 | ETA 05:39:34 2020-11-25 20:06:43 [INFO] [TRAIN] epoch=3, iter=1500/40000, loss=0.7436, lr=0.009662, batch_cost=0.7933, reader_cost=0.0038 | ETA 08:29:03 2020-11-25 20:08:13 [INFO] [TRAIN] epoch=3, iter=1600/40000, loss=0.6507, lr=0.009639, batch_cost=0.8975, reader_cost=0.0046 | ETA 09:34:25 2020-11-25 20:09:13 [INFO] [TRAIN] epoch=3, iter=1700/40000, loss=0.6629, lr=0.009617, batch_cost=0.5982, reader_cost=0.0022 | ETA 06:21:49 2020-11-25 20:10:17 [INFO] [TRAIN] epoch=3, iter=1800/40000, loss=0.7115, lr=0.009594, batch_cost=0.6390, reader_cost=0.0016 | ETA 06:46:50 2020-11-25 20:11:22 [INFO] [TRAIN] epoch=3, iter=1900/40000, loss=0.6665, lr=0.009572, batch_cost=0.6490, reader_cost=0.0023 | ETA 06:52:06 2020-11-25 20:12:22 [INFO] [TRAIN] epoch=4, iter=2000/40000, loss=0.6483, lr=0.009549, batch_cost=0.6058, reader_cost=0.0085 | ETA 06:23:41 2020-11-25 20:13:19 [INFO] [TRAIN] epoch=4, iter=2100/40000, loss=0.6131, lr=0.009526, batch_cost=0.5647, reader_cost=0.0014 | ETA 05:56:43 2020-11-25 20:14:14 [INFO] [TRAIN] epoch=4, iter=2200/40000, loss=0.6205, lr=0.009504, batch_cost=0.5501, reader_cost=0.0017 | ETA 05:46:35 2020-11-25 20:15:06 [INFO] [TRAIN] epoch=4, iter=2300/40000, loss=0.6247, lr=0.009481, batch_cost=0.5265, reader_cost=0.0007 | ETA 05:30:50 2020-11-25 20:15:59 [INFO] [TRAIN] epoch=4, iter=2400/40000, loss=0.6008, lr=0.009459, batch_cost=0.5243, reader_cost=0.0005 | ETA 05:28:33 2020-11-25 20:16:51 [INFO] [TRAIN] epoch=4, iter=2500/40000, loss=0.5786, lr=0.009436, batch_cost=0.5257, reader_cost=0.0005 | ETA 05:28:33 2020-11-25 20:17:44 [INFO] [TRAIN] epoch=4, iter=2600/40000, loss=0.5647, lr=0.009413, batch_cost=0.5282, reader_cost=0.0006 | ETA 05:29:14 2020-11-25 20:18:37 [INFO] [TRAIN] epoch=5, iter=2700/40000, loss=0.5703, lr=0.009391, batch_cost=0.5331, reader_cost=0.0063 | ETA 05:31:25 2020-11-25 20:19:30 [INFO] [TRAIN] epoch=5, iter=2800/40000, loss=0.5508, lr=0.009368, batch_cost=0.5295, reader_cost=0.0005 | ETA 05:28:15 2020-11-25 20:20:23 [INFO] [TRAIN] epoch=5, iter=2900/40000, loss=0.6274, lr=0.009345, batch_cost=0.5302, reader_cost=0.0012 | ETA 05:27:51 2020-11-25 20:21:16 [INFO] [TRAIN] epoch=5, iter=3000/40000, loss=0.5375, lr=0.009323, batch_cost=0.5308, reader_cost=0.0009 | ETA 05:27:19 2020-11-25 20:22:10 [INFO] [TRAIN] epoch=5, iter=3100/40000, loss=0.6133, lr=0.009300, batch_cost=0.5304, reader_cost=0.0008 | ETA 05:26:13 2020-11-25 20:23:02 [INFO] [TRAIN] epoch=5, iter=3200/40000, loss=0.5722, lr=0.009277, batch_cost=0.5287, reader_cost=0.0007 | ETA 05:24:16 2020-11-25 20:23:56 [INFO] [TRAIN] epoch=5, iter=3300/40000, loss=0.5360, lr=0.009255, batch_cost=0.5315, reader_cost=0.0006 | ETA 05:25:07 2020-11-25 20:24:49 [INFO] [TRAIN] epoch=6, iter=3400/40000, loss=0.5168, lr=0.009232, batch_cost=0.5384, reader_cost=0.0065 | ETA 05:28:24 2020-11-25 20:25:42 [INFO] [TRAIN] epoch=6, iter=3500/40000, loss=0.5514, lr=0.009209, batch_cost=0.5278, reader_cost=0.0009 | ETA 05:21:06 2020-11-25 20:26:35 [INFO] [TRAIN] epoch=6, iter=3600/40000, loss=0.5579, lr=0.009186, batch_cost=0.5317, reader_cost=0.0017 | ETA 05:22:35 2020-11-25 20:27:28 [INFO] [TRAIN] epoch=6, iter=3700/40000, loss=0.5355, lr=0.009164, batch_cost=0.5311, reader_cost=0.0015 | ETA 05:21:19 2020-11-25 20:28:22 [INFO] [TRAIN] epoch=6, iter=3800/40000, loss=0.5863, lr=0.009141, batch_cost=0.5316, reader_cost=0.0015 | ETA 05:20:43 2020-11-25 20:29:15 [INFO] [TRAIN] epoch=6, iter=3900/40000, loss=0.5347, lr=0.009118, batch_cost=0.5303, reader_cost=0.0014 | ETA 05:19:02 2020-11-25 20:30:08 [INFO] [TRAIN] epoch=7, iter=4000/40000, loss=0.5219, lr=0.009096, batch_cost=0.5356, reader_cost=0.0068 | ETA 05:21:21 2020-11-25 20:30:08 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-25 20:30:31 [INFO] [EVAL] #Images=1449 mIoU=0.6997 Acc=0.9295 Kappa=0.8470 2020-11-25 20:30:31 [INFO] [EVAL] Class IoU: [0.9264 0.8642 0.3764 0.8629 0.6075 0.7733 0.8294 0.8098 0.8788 0.3516 0.7021 0.5833 0.7878 0.77 0.8266 0.8186 0.4792 0.7083 0.3567 0.8006 0.5802] 2020-11-25 20:30:31 [INFO] [EVAL] Class Acc: [0.9658 0.8987 0.39 0.8951 0.6447 0.9019 0.9749 0.8718 0.9184 0.5311 0.7109 0.84 0.9361 0.8701 0.872 0.884 0.6592 0.7912 0.87 0.8521 0.6271] 2020-11-25 20:30:43 [INFO] [EVAL] The model with the best validation mIoU (0.6997) was saved at iter 4000. 2020-11-25 20:31:36 [INFO] [TRAIN] epoch=7, iter=4100/40000, loss=0.4286, lr=0.009073, batch_cost=0.5282, reader_cost=0.0011 | ETA 05:16:02 2020-11-25 20:32:29 [INFO] [TRAIN] epoch=7, iter=4200/40000, loss=0.5247, lr=0.009050, batch_cost=0.5286, reader_cost=0.0009 | ETA 05:15:25 2020-11-25 20:33:22 [INFO] [TRAIN] epoch=7, iter=4300/40000, loss=0.4926, lr=0.009027, batch_cost=0.5298, reader_cost=0.0005 | ETA 05:15:12 2020-11-25 20:34:15 [INFO] [TRAIN] epoch=7, iter=4400/40000, loss=0.5230, lr=0.009005, batch_cost=0.5265, reader_cost=0.0013 | ETA 05:12:22 2020-11-25 20:35:08 [INFO] [TRAIN] epoch=7, iter=4500/40000, loss=0.4880, lr=0.008982, batch_cost=0.5304, reader_cost=0.0016 | ETA 05:13:48 2020-11-25 20:36:01 [INFO] [TRAIN] epoch=7, iter=4600/40000, loss=0.5828, lr=0.008959, batch_cost=0.5307, reader_cost=0.0015 | ETA 05:13:05 2020-11-25 20:36:54 [INFO] [TRAIN] epoch=8, iter=4700/40000, loss=0.5391, lr=0.008936, batch_cost=0.5358, reader_cost=0.0077 | ETA 05:15:13 2020-11-25 20:37:47 [INFO] [TRAIN] epoch=8, iter=4800/40000, loss=0.5221, lr=0.008913, batch_cost=0.5281, reader_cost=0.0014 | ETA 05:09:48 2020-11-25 20:38:41 [INFO] [TRAIN] epoch=8, iter=4900/40000, loss=0.5252, lr=0.008891, batch_cost=0.5417, reader_cost=0.0021 | ETA 05:16:54 2020-11-25 20:39:34 [INFO] [TRAIN] epoch=8, iter=5000/40000, loss=0.4643, lr=0.008868, batch_cost=0.5283, reader_cost=0.0017 | ETA 05:08:12 2020-11-25 20:40:27 [INFO] [TRAIN] epoch=8, iter=5100/40000, loss=0.4863, lr=0.008845, batch_cost=0.5305, reader_cost=0.0016 | ETA 05:08:33 2020-11-25 20:41:29 [INFO] [TRAIN] epoch=8, iter=5200/40000, loss=0.5001, lr=0.008822, batch_cost=0.6146, reader_cost=0.0030 | ETA 05:56:28 2020-11-25 20:43:03 [INFO] [TRAIN] epoch=9, iter=5300/40000, loss=0.4874, lr=0.008799, batch_cost=0.9390, reader_cost=0.0158 | ETA 09:03:03 2020-11-25 20:44:16 [INFO] [TRAIN] epoch=9, iter=5400/40000, loss=0.5317, lr=0.008777, batch_cost=0.7284, reader_cost=0.0052 | ETA 07:00:02 2020-11-25 20:45:16 [INFO] [TRAIN] epoch=9, iter=5500/40000, loss=0.4377, lr=0.008754, batch_cost=0.6004, reader_cost=0.0020 | ETA 05:45:15 2020-11-25 20:46:22 [INFO] [TRAIN] epoch=9, iter=5600/40000, loss=0.4712, lr=0.008731, batch_cost=0.6687, reader_cost=0.0036 | ETA 06:23:22 2020-11-25 20:47:23 [INFO] [TRAIN] epoch=9, iter=5700/40000, loss=0.5071, lr=0.008708, batch_cost=0.6019, reader_cost=0.0025 | ETA 05:44:05 2020-11-25 20:48:23 [INFO] [TRAIN] epoch=9, iter=5800/40000, loss=0.5508, lr=0.008685, batch_cost=0.6036, reader_cost=0.0027 | ETA 05:44:03 2020-11-25 20:49:20 [INFO] [TRAIN] epoch=9, iter=5900/40000, loss=0.4385, lr=0.008662, batch_cost=0.5703, reader_cost=0.0028 | ETA 05:24:07 2020-11-25 20:50:15 [INFO] [TRAIN] epoch=10, iter=6000/40000, loss=0.4694, lr=0.008639, batch_cost=0.5470, reader_cost=0.0071 | ETA 05:09:58 2020-11-25 20:51:08 [INFO] [TRAIN] epoch=10, iter=6100/40000, loss=0.4571, lr=0.008617, batch_cost=0.5310, reader_cost=0.0016 | ETA 05:00:01 2020-11-25 20:52:01 [INFO] [TRAIN] epoch=10, iter=6200/40000, loss=0.4833, lr=0.008594, batch_cost=0.5315, reader_cost=0.0015 | ETA 04:59:24 2020-11-25 20:52:54 [INFO] [TRAIN] epoch=10, iter=6300/40000, loss=0.4841, lr=0.008571, batch_cost=0.5316, reader_cost=0.0014 | ETA 04:58:36 2020-11-25 20:53:47 [INFO] [TRAIN] epoch=10, iter=6400/40000, loss=0.4325, lr=0.008548, batch_cost=0.5296, reader_cost=0.0015 | ETA 04:56:32 2020-11-25 20:54:40 [INFO] [TRAIN] epoch=10, iter=6500/40000, loss=0.4335, lr=0.008525, batch_cost=0.5304, reader_cost=0.0014 | ETA 04:56:08 2020-11-25 20:55:33 [INFO] [TRAIN] epoch=10, iter=6600/40000, loss=0.4231, lr=0.008502, batch_cost=0.5321, reader_cost=0.0013 | ETA 04:56:11 2020-11-25 20:56:27 [INFO] [TRAIN] epoch=11, iter=6700/40000, loss=0.4332, lr=0.008479, batch_cost=0.5375, reader_cost=0.0070 | ETA 04:58:19 2020-11-25 20:57:21 [INFO] [TRAIN] epoch=11, iter=6800/40000, loss=0.4701, lr=0.008456, batch_cost=0.5333, reader_cost=0.0014 | ETA 04:55:06 2020-11-25 20:58:14 [INFO] [TRAIN] epoch=11, iter=6900/40000, loss=0.3992, lr=0.008433, batch_cost=0.5305, reader_cost=0.0015 | ETA 04:52:39 2020-11-25 20:59:07 [INFO] [TRAIN] epoch=11, iter=7000/40000, loss=0.4379, lr=0.008410, batch_cost=0.5310, reader_cost=0.0014 | ETA 04:52:03 2020-11-25 21:00:00 [INFO] [TRAIN] epoch=11, iter=7100/40000, loss=0.4280, lr=0.008388, batch_cost=0.5309, reader_cost=0.0015 | ETA 04:51:06 2020-11-25 21:00:53 [INFO] [TRAIN] epoch=11, iter=7200/40000, loss=0.4488, lr=0.008365, batch_cost=0.5309, reader_cost=0.0016 | ETA 04:50:14 2020-11-25 21:01:47 [INFO] [TRAIN] epoch=12, iter=7300/40000, loss=0.4071, lr=0.008342, batch_cost=0.5380, reader_cost=0.0079 | ETA 04:53:13 2020-11-25 21:02:40 [INFO] [TRAIN] epoch=12, iter=7400/40000, loss=0.4627, lr=0.008319, batch_cost=0.5317, reader_cost=0.0016 | ETA 04:48:52 2020-11-25 21:03:33 [INFO] [TRAIN] epoch=12, iter=7500/40000, loss=0.4873, lr=0.008296, batch_cost=0.5325, reader_cost=0.0018 | ETA 04:48:26 2020-11-25 21:04:26 [INFO] [TRAIN] epoch=12, iter=7600/40000, loss=0.3543, lr=0.008273, batch_cost=0.5291, reader_cost=0.0014 | ETA 04:45:42 2020-11-25 21:05:19 [INFO] [TRAIN] epoch=12, iter=7700/40000, loss=0.4395, lr=0.008250, batch_cost=0.5301, reader_cost=0.0014 | ETA 04:45:21 2020-11-25 21:06:12 [INFO] [TRAIN] epoch=12, iter=7800/40000, loss=0.4325, lr=0.008227, batch_cost=0.5343, reader_cost=0.0014 | ETA 04:46:44 2020-11-25 21:07:05 [INFO] [TRAIN] epoch=12, iter=7900/40000, loss=0.5107, lr=0.008204, batch_cost=0.5293, reader_cost=0.0015 | ETA 04:43:12 2020-11-25 21:08:00 [INFO] [TRAIN] epoch=13, iter=8000/40000, loss=0.4097, lr=0.008181, batch_cost=0.5469, reader_cost=0.0081 | ETA 04:51:40 2020-11-25 21:08:00 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-25 21:08:24 [INFO] [EVAL] #Images=1449 mIoU=0.7375 Acc=0.9387 Kappa=0.8660 2020-11-25 21:08:24 [INFO] [EVAL] Class IoU: [0.9345 0.8728 0.3884 0.8556 0.7113 0.7933 0.8894 0.8454 0.8837 0.4183 0.8531 0.4888 0.8248 0.79 0.8484 0.8203 0.5884 0.8241 0.4156 0.8444 0.5964] 2020-11-25 21:08:24 [INFO] [EVAL] Class Acc: [0.9669 0.8985 0.4175 0.9194 0.7771 0.8856 0.984 0.9025 0.914 0.567 0.9455 0.9269 0.8882 0.8487 0.9159 0.8692 0.7739 0.9467 0.8506 0.9102 0.635 ] 2020-11-25 21:08:36 [INFO] [EVAL] The model with the best validation mIoU (0.7375) was saved at iter 8000. 2020-11-25 21:09:29 [INFO] [TRAIN] epoch=13, iter=8100/40000, loss=0.3844, lr=0.008158, batch_cost=0.5309, reader_cost=0.0020 | ETA 04:42:16 2020-11-25 21:10:22 [INFO] [TRAIN] epoch=13, iter=8200/40000, loss=0.4278, lr=0.008135, batch_cost=0.5307, reader_cost=0.0017 | ETA 04:41:16 2020-11-25 21:11:15 [INFO] [TRAIN] epoch=13, iter=8300/40000, loss=0.3729, lr=0.008112, batch_cost=0.5309, reader_cost=0.0018 | ETA 04:40:29 2020-11-25 21:12:09 [INFO] [TRAIN] epoch=13, iter=8400/40000, loss=0.4593, lr=0.008089, batch_cost=0.5405, reader_cost=0.0017 | ETA 04:44:38 2020-11-25 21:13:35 [INFO] [TRAIN] epoch=13, iter=8500/40000, loss=0.4158, lr=0.008066, batch_cost=0.8555, reader_cost=0.0064 | ETA 07:29:08 2020-11-25 21:14:58 [INFO] [TRAIN] epoch=14, iter=8600/40000, loss=0.4122, lr=0.008043, batch_cost=0.8379, reader_cost=0.0098 | ETA 07:18:30 2020-11-25 21:15:57 [INFO] [TRAIN] epoch=14, iter=8700/40000, loss=0.3625, lr=0.008020, batch_cost=0.5893, reader_cost=0.0029 | ETA 05:07:26 2020-11-25 21:17:08 [INFO] [TRAIN] epoch=14, iter=8800/40000, loss=0.3853, lr=0.007996, batch_cost=0.7074, reader_cost=0.0034 | ETA 06:07:49 2020-11-25 21:18:08 [INFO] [TRAIN] epoch=14, iter=8900/40000, loss=0.4189, lr=0.007973, batch_cost=0.5978, reader_cost=0.0017 | ETA 05:09:52 2020-11-25 21:19:09 [INFO] [TRAIN] epoch=14, iter=9000/40000, loss=0.3750, lr=0.007950, batch_cost=0.6060, reader_cost=0.0028 | ETA 05:13:07 2020-11-25 21:20:06 [INFO] [TRAIN] epoch=14, iter=9100/40000, loss=0.4115, lr=0.007927, batch_cost=0.5786, reader_cost=0.0023 | ETA 04:57:58 2020-11-25 21:21:00 [INFO] [TRAIN] epoch=14, iter=9200/40000, loss=0.4131, lr=0.007904, batch_cost=0.5377, reader_cost=0.0021 | ETA 04:35:59 2020-11-25 21:21:54 [INFO] [TRAIN] epoch=15, iter=9300/40000, loss=0.4132, lr=0.007881, batch_cost=0.5374, reader_cost=0.0076 | ETA 04:34:58 2020-11-25 21:22:47 [INFO] [TRAIN] epoch=15, iter=9400/40000, loss=0.3803, lr=0.007858, batch_cost=0.5319, reader_cost=0.0015 | ETA 04:31:15 2020-11-25 21:23:40 [INFO] [TRAIN] epoch=15, iter=9500/40000, loss=0.3904, lr=0.007835, batch_cost=0.5329, reader_cost=0.0015 | ETA 04:30:54 2020-11-25 21:24:34 [INFO] [TRAIN] epoch=15, iter=9600/40000, loss=0.4120, lr=0.007812, batch_cost=0.5325, reader_cost=0.0013 | ETA 04:29:47 2020-11-25 21:25:27 [INFO] [TRAIN] epoch=15, iter=9700/40000, loss=0.3915, lr=0.007789, batch_cost=0.5292, reader_cost=0.0015 | ETA 04:27:15 2020-11-25 21:26:20 [INFO] [TRAIN] epoch=15, iter=9800/40000, loss=0.3797, lr=0.007765, batch_cost=0.5307, reader_cost=0.0014 | ETA 04:27:08 2020-11-25 21:27:13 [INFO] [TRAIN] epoch=15, iter=9900/40000, loss=0.3858, lr=0.007742, batch_cost=0.5338, reader_cost=0.0014 | ETA 04:27:48 2020-11-25 21:28:07 [INFO] [TRAIN] epoch=16, iter=10000/40000, loss=0.4096, lr=0.007719, batch_cost=0.5369, reader_cost=0.0067 | ETA 04:28:26 2020-11-25 21:28:59 [INFO] [TRAIN] epoch=16, iter=10100/40000, loss=0.3500, lr=0.007696, batch_cost=0.5236, reader_cost=0.0002 | ETA 04:20:55 2020-11-25 21:29:52 [INFO] [TRAIN] epoch=16, iter=10200/40000, loss=0.3870, lr=0.007673, batch_cost=0.5315, reader_cost=0.0015 | ETA 04:24:00 2020-11-25 21:30:45 [INFO] [TRAIN] epoch=16, iter=10300/40000, loss=0.3629, lr=0.007650, batch_cost=0.5314, reader_cost=0.0015 | ETA 04:23:01 2020-11-25 21:31:39 [INFO] [TRAIN] epoch=16, iter=10400/40000, loss=0.3634, lr=0.007626, batch_cost=0.5335, reader_cost=0.0015 | ETA 04:23:11 2020-11-25 21:32:32 [INFO] [TRAIN] epoch=16, iter=10500/40000, loss=0.3951, lr=0.007603, batch_cost=0.5342, reader_cost=0.0015 | ETA 04:22:39 2020-11-25 21:33:26 [INFO] [TRAIN] epoch=17, iter=10600/40000, loss=0.4413, lr=0.007580, batch_cost=0.5396, reader_cost=0.0074 | ETA 04:24:22 2020-11-25 21:34:19 [INFO] [TRAIN] epoch=17, iter=10700/40000, loss=0.3506, lr=0.007557, batch_cost=0.5322, reader_cost=0.0019 | ETA 04:19:54 2020-11-25 21:35:13 [INFO] [TRAIN] epoch=17, iter=10800/40000, loss=0.4097, lr=0.007534, batch_cost=0.5321, reader_cost=0.0014 | ETA 04:18:56 2020-11-25 21:36:06 [INFO] [TRAIN] epoch=17, iter=10900/40000, loss=0.3452, lr=0.007510, batch_cost=0.5316, reader_cost=0.0018 | ETA 04:17:49 2020-11-25 21:36:59 [INFO] [TRAIN] epoch=17, iter=11000/40000, loss=0.3527, lr=0.007487, batch_cost=0.5333, reader_cost=0.0015 | ETA 04:17:44 2020-11-25 21:37:52 [INFO] [TRAIN] epoch=17, iter=11100/40000, loss=0.3602, lr=0.007464, batch_cost=0.5334, reader_cost=0.0013 | ETA 04:16:54 2020-11-25 21:38:46 [INFO] [TRAIN] epoch=17, iter=11200/40000, loss=0.3883, lr=0.007441, batch_cost=0.5327, reader_cost=0.0014 | ETA 04:15:40 2020-11-25 21:39:40 [INFO] [TRAIN] epoch=18, iter=11300/40000, loss=0.3623, lr=0.007417, batch_cost=0.5452, reader_cost=0.0070 | ETA 04:20:48 2020-11-25 21:40:33 [INFO] [TRAIN] epoch=18, iter=11400/40000, loss=0.3823, lr=0.007394, batch_cost=0.5312, reader_cost=0.0009 | ETA 04:13:12 2020-11-25 21:41:27 [INFO] [TRAIN] epoch=18, iter=11500/40000, loss=0.3897, lr=0.007371, batch_cost=0.5333, reader_cost=0.0013 | ETA 04:13:19 2020-11-25 21:42:20 [INFO] [TRAIN] epoch=18, iter=11600/40000, loss=0.3694, lr=0.007348, batch_cost=0.5332, reader_cost=0.0013 | ETA 04:12:22 2020-11-25 21:43:13 [INFO] [TRAIN] epoch=18, iter=11700/40000, loss=0.3385, lr=0.007324, batch_cost=0.5329, reader_cost=0.0009 | ETA 04:11:20 2020-11-25 21:44:07 [INFO] [TRAIN] epoch=18, iter=11800/40000, loss=0.3621, lr=0.007301, batch_cost=0.5336, reader_cost=0.0016 | ETA 04:10:48 2020-11-25 21:45:00 [INFO] [TRAIN] epoch=19, iter=11900/40000, loss=0.3466, lr=0.007278, batch_cost=0.5375, reader_cost=0.0073 | ETA 04:11:43 2020-11-25 21:45:53 [INFO] [TRAIN] epoch=19, iter=12000/40000, loss=0.3333, lr=0.007254, batch_cost=0.5297, reader_cost=0.0014 | ETA 04:07:12 2020-11-25 21:45:53 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-25 21:46:17 [INFO] [EVAL] #Images=1449 mIoU=0.7525 Acc=0.9438 Kappa=0.8751 2020-11-25 21:46:17 [INFO] [EVAL] Class IoU: [0.937 0.8685 0.425 0.8852 0.6659 0.8183 0.9397 0.8657 0.9157 0.3228 0.8623 0.3671 0.8511 0.9018 0.8251 0.8487 0.597 0.7834 0.5197 0.8594 0.7428] 2020-11-25 21:46:17 [INFO] [EVAL] Class Acc: [0.961 0.9161 0.4487 0.9427 0.7037 0.8779 0.9751 0.9656 0.9434 0.6555 0.8926 0.9583 0.893 0.9418 0.9555 0.9216 0.7364 0.9592 0.713 0.9679 0.9573] 2020-11-25 21:46:28 [INFO] [EVAL] The model with the best validation mIoU (0.7525) was saved at iter 12000. 2020-11-25 21:47:21 [INFO] [TRAIN] epoch=19, iter=12100/40000, loss=0.3513, lr=0.007231, batch_cost=0.5307, reader_cost=0.0015 | ETA 04:06:47 2020-11-25 21:48:15 [INFO] [TRAIN] epoch=19, iter=12200/40000, loss=0.3372, lr=0.007208, batch_cost=0.5313, reader_cost=0.0015 | ETA 04:06:09 2020-11-25 21:49:08 [INFO] [TRAIN] epoch=19, iter=12300/40000, loss=0.3284, lr=0.007184, batch_cost=0.5311, reader_cost=0.0017 | ETA 04:05:12 2020-11-25 21:50:01 [INFO] [TRAIN] epoch=19, iter=12400/40000, loss=0.3324, lr=0.007161, batch_cost=0.5300, reader_cost=0.0013 | ETA 04:03:48 2020-11-25 21:50:54 [INFO] [TRAIN] epoch=19, iter=12500/40000, loss=0.3347, lr=0.007138, batch_cost=0.5331, reader_cost=0.0019 | ETA 04:04:20 2020-11-25 21:51:48 [INFO] [TRAIN] epoch=20, iter=12600/40000, loss=0.3175, lr=0.007114, batch_cost=0.5374, reader_cost=0.0085 | ETA 04:05:24 2020-11-25 21:52:41 [INFO] [TRAIN] epoch=20, iter=12700/40000, loss=0.3478, lr=0.007091, batch_cost=0.5309, reader_cost=0.0012 | ETA 04:01:33 2020-11-25 21:53:34 [INFO] [TRAIN] epoch=20, iter=12800/40000, loss=0.3582, lr=0.007068, batch_cost=0.5316, reader_cost=0.0014 | ETA 04:00:59 2020-11-25 21:54:27 [INFO] [TRAIN] epoch=20, iter=12900/40000, loss=0.3232, lr=0.007044, batch_cost=0.5312, reader_cost=0.0012 | ETA 03:59:55 2020-11-25 21:55:36 [INFO] [TRAIN] epoch=20, iter=13000/40000, loss=0.3409, lr=0.007021, batch_cost=0.6933, reader_cost=0.0047 | ETA 05:11:58 2020-11-25 21:57:07 [INFO] [TRAIN] epoch=20, iter=13100/40000, loss=0.3071, lr=0.006997, batch_cost=0.9037, reader_cost=0.0052 | ETA 06:45:09 2020-11-25 21:58:14 [INFO] [TRAIN] epoch=20, iter=13200/40000, loss=0.3091, lr=0.006974, batch_cost=0.6716, reader_cost=0.0038 | ETA 04:59:58 2020-11-25 21:59:21 [INFO] [TRAIN] epoch=21, iter=13300/40000, loss=0.2813, lr=0.006951, batch_cost=0.6670, reader_cost=0.0100 | ETA 04:56:50 2020-11-25 22:00:26 [INFO] [TRAIN] epoch=21, iter=13400/40000, loss=0.3160, lr=0.006927, batch_cost=0.6534, reader_cost=0.0030 | ETA 04:49:41 2020-11-25 22:01:26 [INFO] [TRAIN] epoch=21, iter=13500/40000, loss=0.3001, lr=0.006904, batch_cost=0.6018, reader_cost=0.0030 | ETA 04:25:48 2020-11-25 22:02:24 [INFO] [TRAIN] epoch=21, iter=13600/40000, loss=0.2826, lr=0.006880, batch_cost=0.5731, reader_cost=0.0027 | ETA 04:12:08 2020-11-25 22:03:19 [INFO] [TRAIN] epoch=21, iter=13700/40000, loss=0.3100, lr=0.006857, batch_cost=0.5566, reader_cost=0.0020 | ETA 04:03:59 2020-11-25 22:04:12 [INFO] [TRAIN] epoch=21, iter=13800/40000, loss=0.3418, lr=0.006833, batch_cost=0.5323, reader_cost=0.0014 | ETA 03:52:25 2020-11-25 22:05:06 [INFO] [TRAIN] epoch=22, iter=13900/40000, loss=0.3196, lr=0.006810, batch_cost=0.5370, reader_cost=0.0073 | ETA 03:53:35 2020-11-25 22:05:59 [INFO] [TRAIN] epoch=22, iter=14000/40000, loss=0.3229, lr=0.006786, batch_cost=0.5319, reader_cost=0.0015 | ETA 03:50:30 2020-11-25 22:06:53 [INFO] [TRAIN] epoch=22, iter=14100/40000, loss=0.3512, lr=0.006763, batch_cost=0.5323, reader_cost=0.0014 | ETA 03:49:47 2020-11-25 22:07:46 [INFO] [TRAIN] epoch=22, iter=14200/40000, loss=0.3124, lr=0.006739, batch_cost=0.5326, reader_cost=0.0016 | ETA 03:48:59 2020-11-25 22:08:40 [INFO] [TRAIN] epoch=22, iter=14300/40000, loss=0.2894, lr=0.006716, batch_cost=0.5386, reader_cost=0.0013 | ETA 03:50:43 2020-11-25 22:09:33 [INFO] [TRAIN] epoch=22, iter=14400/40000, loss=0.3023, lr=0.006692, batch_cost=0.5305, reader_cost=0.0015 | ETA 03:46:20 2020-11-25 22:10:26 [INFO] [TRAIN] epoch=22, iter=14500/40000, loss=0.3345, lr=0.006669, batch_cost=0.5329, reader_cost=0.0014 | ETA 03:46:27 2020-11-25 22:11:20 [INFO] [TRAIN] epoch=23, iter=14600/40000, loss=0.2673, lr=0.006645, batch_cost=0.5372, reader_cost=0.0076 | ETA 03:47:24 2020-11-25 22:12:14 [INFO] [TRAIN] epoch=23, iter=14700/40000, loss=0.3049, lr=0.006622, batch_cost=0.5412, reader_cost=0.0013 | ETA 03:48:13 2020-11-25 22:13:07 [INFO] [TRAIN] epoch=23, iter=14800/40000, loss=0.3310, lr=0.006598, batch_cost=0.5325, reader_cost=0.0017 | ETA 03:43:38 2020-11-25 22:14:02 [INFO] [TRAIN] epoch=23, iter=14900/40000, loss=0.2802, lr=0.006575, batch_cost=0.5533, reader_cost=0.0014 | ETA 03:51:28 2020-11-25 22:14:57 [INFO] [TRAIN] epoch=23, iter=15000/40000, loss=0.3116, lr=0.006551, batch_cost=0.5456, reader_cost=0.0018 | ETA 03:47:20 2020-11-25 22:15:50 [INFO] [TRAIN] epoch=23, iter=15100/40000, loss=0.3276, lr=0.006527, batch_cost=0.5336, reader_cost=0.0018 | ETA 03:41:27 2020-11-25 22:16:44 [INFO] [TRAIN] epoch=23, iter=15200/40000, loss=0.3613, lr=0.006504, batch_cost=0.5327, reader_cost=0.0015 | ETA 03:40:11 2020-11-25 22:17:38 [INFO] [TRAIN] epoch=24, iter=15300/40000, loss=0.3183, lr=0.006480, batch_cost=0.5479, reader_cost=0.0079 | ETA 03:45:33 2020-11-25 22:18:33 [INFO] [TRAIN] epoch=24, iter=15400/40000, loss=0.3338, lr=0.006457, batch_cost=0.5423, reader_cost=0.0015 | ETA 03:42:20 2020-11-25 22:19:27 [INFO] [TRAIN] epoch=24, iter=15500/40000, loss=0.3382, lr=0.006433, batch_cost=0.5418, reader_cost=0.0016 | ETA 03:41:14 2020-11-25 22:20:21 [INFO] [TRAIN] epoch=24, iter=15600/40000, loss=0.3190, lr=0.006409, batch_cost=0.5414, reader_cost=0.0015 | ETA 03:40:10 2020-11-25 22:21:16 [INFO] [TRAIN] epoch=24, iter=15700/40000, loss=0.3224, lr=0.006386, batch_cost=0.5518, reader_cost=0.0012 | ETA 03:43:29 2020-11-25 22:22:09 [INFO] [TRAIN] epoch=24, iter=15800/40000, loss=0.2959, lr=0.006362, batch_cost=0.5321, reader_cost=0.0009 | ETA 03:34:35 2020-11-25 22:23:03 [INFO] [TRAIN] epoch=25, iter=15900/40000, loss=0.2875, lr=0.006338, batch_cost=0.5390, reader_cost=0.0066 | ETA 03:36:29 2020-11-25 22:23:57 [INFO] [TRAIN] epoch=25, iter=16000/40000, loss=0.2890, lr=0.006315, batch_cost=0.5338, reader_cost=0.0014 | ETA 03:33:32 2020-11-25 22:23:57 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-25 22:24:21 [INFO] [EVAL] #Images=1449 mIoU=0.7758 Acc=0.9490 Kappa=0.8863 2020-11-25 22:24:21 [INFO] [EVAL] Class IoU: [0.9437 0.8932 0.4427 0.8683 0.7191 0.8349 0.9285 0.8765 0.8874 0.4276 0.8986 0.5619 0.855 0.8692 0.884 0.8605 0.6343 0.8612 0.3689 0.899 0.7777] 2020-11-25 22:24:21 [INFO] [EVAL] Class Acc: [0.9637 0.9178 0.4823 0.9834 0.8084 0.9114 0.9856 0.9327 0.9234 0.5908 0.9505 0.8803 0.9218 0.969 0.9392 0.9302 0.8581 0.9053 0.7918 0.96 0.9456] 2020-11-25 22:24:31 [INFO] [EVAL] The model with the best validation mIoU (0.7758) was saved at iter 16000. 2020-11-25 22:25:24 [INFO] [TRAIN] epoch=25, iter=16100/40000, loss=0.2928, lr=0.006291, batch_cost=0.5268, reader_cost=0.0014 | ETA 03:29:51 2020-11-25 22:26:17 [INFO] [TRAIN] epoch=25, iter=16200/40000, loss=0.2790, lr=0.006267, batch_cost=0.5291, reader_cost=0.0013 | ETA 03:29:52 2020-11-25 22:27:10 [INFO] [TRAIN] epoch=25, iter=16300/40000, loss=0.2886, lr=0.006244, batch_cost=0.5319, reader_cost=0.0014 | ETA 03:30:05 2020-11-25 22:28:03 [INFO] [TRAIN] epoch=25, iter=16400/40000, loss=0.3320, lr=0.006220, batch_cost=0.5319, reader_cost=0.0014 | ETA 03:29:12 2020-11-25 22:28:57 [INFO] [TRAIN] epoch=25, iter=16500/40000, loss=0.3192, lr=0.006196, batch_cost=0.5324, reader_cost=0.0015 | ETA 03:28:30 2020-11-25 22:29:50 [INFO] [TRAIN] epoch=26, iter=16600/40000, loss=0.3075, lr=0.006172, batch_cost=0.5365, reader_cost=0.0065 | ETA 03:29:14 2020-11-25 22:30:43 [INFO] [TRAIN] epoch=26, iter=16700/40000, loss=0.2863, lr=0.006149, batch_cost=0.5294, reader_cost=0.0014 | ETA 03:25:35 2020-11-25 22:31:36 [INFO] [TRAIN] epoch=26, iter=16800/40000, loss=0.2785, lr=0.006125, batch_cost=0.5300, reader_cost=0.0014 | ETA 03:24:54 2020-11-25 22:32:30 [INFO] [TRAIN] epoch=26, iter=16900/40000, loss=0.3478, lr=0.006101, batch_cost=0.5335, reader_cost=0.0014 | ETA 03:25:24 2020-11-25 22:33:23 [INFO] [TRAIN] epoch=26, iter=17000/40000, loss=0.2879, lr=0.006077, batch_cost=0.5339, reader_cost=0.0012 | ETA 03:24:40 2020-11-25 22:34:16 [INFO] [TRAIN] epoch=26, iter=17100/40000, loss=0.3396, lr=0.006054, batch_cost=0.5320, reader_cost=0.0014 | ETA 03:23:03 2020-11-25 22:35:10 [INFO] [TRAIN] epoch=27, iter=17200/40000, loss=0.2812, lr=0.006030, batch_cost=0.5366, reader_cost=0.0067 | ETA 03:23:54 2020-11-25 22:36:03 [INFO] [TRAIN] epoch=27, iter=17300/40000, loss=0.2949, lr=0.006006, batch_cost=0.5314, reader_cost=0.0015 | ETA 03:21:02 2020-11-25 22:36:56 [INFO] [TRAIN] epoch=27, iter=17400/40000, loss=0.3608, lr=0.005982, batch_cost=0.5333, reader_cost=0.0015 | ETA 03:20:52 2020-11-25 22:37:51 [INFO] [TRAIN] epoch=27, iter=17500/40000, loss=0.2832, lr=0.005958, batch_cost=0.5415, reader_cost=0.0016 | ETA 03:23:04 2020-11-25 22:38:44 [INFO] [TRAIN] epoch=27, iter=17600/40000, loss=0.3006, lr=0.005935, batch_cost=0.5318, reader_cost=0.0012 | ETA 03:18:31 2020-11-25 22:39:37 [INFO] [TRAIN] epoch=27, iter=17700/40000, loss=0.3186, lr=0.005911, batch_cost=0.5295, reader_cost=0.0016 | ETA 03:16:46 2020-11-25 22:40:30 [INFO] [TRAIN] epoch=27, iter=17800/40000, loss=0.2724, lr=0.005887, batch_cost=0.5320, reader_cost=0.0013 | ETA 03:16:51 2020-11-25 22:41:24 [INFO] [TRAIN] epoch=28, iter=17900/40000, loss=0.2887, lr=0.005863, batch_cost=0.5389, reader_cost=0.0073 | ETA 03:18:30 2020-11-25 22:42:17 [INFO] [TRAIN] epoch=28, iter=18000/40000, loss=0.2931, lr=0.005839, batch_cost=0.5300, reader_cost=0.0015 | ETA 03:14:20 2020-11-25 22:43:10 [INFO] [TRAIN] epoch=28, iter=18100/40000, loss=0.2788, lr=0.005815, batch_cost=0.5295, reader_cost=0.0016 | ETA 03:13:16 2020-11-25 22:44:03 [INFO] [TRAIN] epoch=28, iter=18200/40000, loss=0.2955, lr=0.005791, batch_cost=0.5330, reader_cost=0.0015 | ETA 03:13:39 2020-11-25 22:44:56 [INFO] [TRAIN] epoch=28, iter=18300/40000, loss=0.3024, lr=0.005767, batch_cost=0.5333, reader_cost=0.0017 | ETA 03:12:52 2020-11-25 22:45:50 [INFO] [TRAIN] epoch=28, iter=18400/40000, loss=0.2804, lr=0.005743, batch_cost=0.5325, reader_cost=0.0012 | ETA 03:11:41 2020-11-25 22:46:43 [INFO] [TRAIN] epoch=28, iter=18500/40000, loss=0.3028, lr=0.005720, batch_cost=0.5335, reader_cost=0.0017 | ETA 03:11:10 2020-11-25 22:47:37 [INFO] [TRAIN] epoch=29, iter=18600/40000, loss=0.2790, lr=0.005696, batch_cost=0.5378, reader_cost=0.0080 | ETA 03:11:49 2020-11-25 22:48:30 [INFO] [TRAIN] epoch=29, iter=18700/40000, loss=0.2999, lr=0.005672, batch_cost=0.5340, reader_cost=0.0016 | ETA 03:09:34 2020-11-25 22:49:23 [INFO] [TRAIN] epoch=29, iter=18800/40000, loss=0.2542, lr=0.005648, batch_cost=0.5323, reader_cost=0.0012 | ETA 03:08:04 2020-11-25 22:50:17 [INFO] [TRAIN] epoch=29, iter=18900/40000, loss=0.2683, lr=0.005624, batch_cost=0.5342, reader_cost=0.0018 | ETA 03:07:52 2020-11-25 22:51:10 [INFO] [TRAIN] epoch=29, iter=19000/40000, loss=0.2742, lr=0.005600, batch_cost=0.5325, reader_cost=0.0017 | ETA 03:06:22 2020-11-25 22:52:03 [INFO] [TRAIN] epoch=29, iter=19100/40000, loss=0.2802, lr=0.005576, batch_cost=0.5337, reader_cost=0.0011 | ETA 03:05:53 2020-11-25 22:52:57 [INFO] [TRAIN] epoch=30, iter=19200/40000, loss=0.3213, lr=0.005552, batch_cost=0.5394, reader_cost=0.0078 | ETA 03:06:59 2020-11-25 22:53:51 [INFO] [TRAIN] epoch=30, iter=19300/40000, loss=0.2601, lr=0.005528, batch_cost=0.5326, reader_cost=0.0014 | ETA 03:03:45 2020-11-25 22:54:44 [INFO] [TRAIN] epoch=30, iter=19400/40000, loss=0.3164, lr=0.005504, batch_cost=0.5317, reader_cost=0.0016 | ETA 03:02:32 2020-11-25 22:55:37 [INFO] [TRAIN] epoch=30, iter=19500/40000, loss=0.2678, lr=0.005480, batch_cost=0.5291, reader_cost=0.0014 | ETA 03:00:46 2020-11-25 22:56:30 [INFO] [TRAIN] epoch=30, iter=19600/40000, loss=0.2724, lr=0.005455, batch_cost=0.5319, reader_cost=0.0018 | ETA 03:00:50 2020-11-25 22:57:23 [INFO] [TRAIN] epoch=30, iter=19700/40000, loss=0.3059, lr=0.005431, batch_cost=0.5322, reader_cost=0.0016 | ETA 03:00:03 2020-11-25 22:58:16 [INFO] [TRAIN] epoch=30, iter=19800/40000, loss=0.2993, lr=0.005407, batch_cost=0.5333, reader_cost=0.0015 | ETA 02:59:31 2020-11-25 22:59:10 [INFO] [TRAIN] epoch=31, iter=19900/40000, loss=0.2625, lr=0.005383, batch_cost=0.5382, reader_cost=0.0071 | ETA 03:00:18 2020-11-25 23:00:03 [INFO] [TRAIN] epoch=31, iter=20000/40000, loss=0.2629, lr=0.005359, batch_cost=0.5237, reader_cost=0.0007 | ETA 02:54:34 2020-11-25 23:00:03 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-25 23:00:25 [INFO] [EVAL] #Images=1449 mIoU=0.7912 Acc=0.9533 Kappa=0.8960 2020-11-25 23:00:25 [INFO] [EVAL] Class IoU: [0.9474 0.8952 0.4317 0.904 0.7425 0.8272 0.9338 0.881 0.9278 0.4257 0.9072 0.5548 0.8935 0.8788 0.8859 0.8697 0.6599 0.8813 0.5049 0.898 0.7653] 2020-11-25 23:00:25 [INFO] [EVAL] Class Acc: [0.9662 0.9374 0.4512 0.9619 0.8489 0.8834 0.9858 0.9493 0.9601 0.603 0.944 0.9577 0.9408 0.968 0.9266 0.9448 0.8335 0.9553 0.8433 0.9465 0.932 ] 2020-11-25 23:00:39 [INFO] [EVAL] The model with the best validation mIoU (0.7912) was saved at iter 20000. 2020-11-25 23:01:32 [INFO] [TRAIN] epoch=31, iter=20100/40000, loss=0.2444, lr=0.005335, batch_cost=0.5305, reader_cost=0.0014 | ETA 02:55:57 2020-11-25 23:02:25 [INFO] [TRAIN] epoch=31, iter=20200/40000, loss=0.2889, lr=0.005311, batch_cost=0.5299, reader_cost=0.0013 | ETA 02:54:51 2020-11-25 23:03:18 [INFO] [TRAIN] epoch=31, iter=20300/40000, loss=0.2400, lr=0.005287, batch_cost=0.5313, reader_cost=0.0018 | ETA 02:54:27 2020-11-25 23:04:11 [INFO] [TRAIN] epoch=31, iter=20400/40000, loss=0.2957, lr=0.005263, batch_cost=0.5327, reader_cost=0.0018 | ETA 02:54:00 2020-11-25 23:05:05 [INFO] [TRAIN] epoch=32, iter=20500/40000, loss=0.2611, lr=0.005238, batch_cost=0.5430, reader_cost=0.0079 | ETA 02:56:29 2020-11-25 23:05:59 [INFO] [TRAIN] epoch=32, iter=20600/40000, loss=0.2586, lr=0.005214, batch_cost=0.5309, reader_cost=0.0015 | ETA 02:51:39 2020-11-25 23:06:51 [INFO] [TRAIN] epoch=32, iter=20700/40000, loss=0.2820, lr=0.005190, batch_cost=0.5292, reader_cost=0.0016 | ETA 02:50:13 2020-11-25 23:07:44 [INFO] [TRAIN] epoch=32, iter=20800/40000, loss=0.2766, lr=0.005166, batch_cost=0.5266, reader_cost=0.0013 | ETA 02:48:30 2020-11-25 23:08:37 [INFO] [TRAIN] epoch=32, iter=20900/40000, loss=0.2604, lr=0.005142, batch_cost=0.5272, reader_cost=0.0016 | ETA 02:47:49 2020-11-25 23:09:30 [INFO] [TRAIN] epoch=32, iter=21000/40000, loss=0.2690, lr=0.005117, batch_cost=0.5296, reader_cost=0.0016 | ETA 02:47:42 2020-11-25 23:10:23 [INFO] [TRAIN] epoch=32, iter=21100/40000, loss=0.2444, lr=0.005093, batch_cost=0.5329, reader_cost=0.0014 | ETA 02:47:51 2020-11-25 23:11:17 [INFO] [TRAIN] epoch=33, iter=21200/40000, loss=0.2597, lr=0.005069, batch_cost=0.5387, reader_cost=0.0073 | ETA 02:48:47 2020-11-25 23:12:09 [INFO] [TRAIN] epoch=33, iter=21300/40000, loss=0.3105, lr=0.005045, batch_cost=0.5229, reader_cost=0.0005 | ETA 02:42:57 2020-11-25 23:13:02 [INFO] [TRAIN] epoch=33, iter=21400/40000, loss=0.2670, lr=0.005020, batch_cost=0.5278, reader_cost=0.0010 | ETA 02:43:36 2020-11-25 23:13:55 [INFO] [TRAIN] epoch=33, iter=21500/40000, loss=0.2654, lr=0.004996, batch_cost=0.5297, reader_cost=0.0012 | ETA 02:43:19 2020-11-25 23:14:48 [INFO] [TRAIN] epoch=33, iter=21600/40000, loss=0.2606, lr=0.004972, batch_cost=0.5311, reader_cost=0.0017 | ETA 02:42:52 2020-11-25 23:15:41 [INFO] [TRAIN] epoch=33, iter=21700/40000, loss=0.2339, lr=0.004947, batch_cost=0.5299, reader_cost=0.0014 | ETA 02:41:37 2020-11-25 23:16:34 [INFO] [TRAIN] epoch=33, iter=21800/40000, loss=0.2756, lr=0.004923, batch_cost=0.5321, reader_cost=0.0016 | ETA 02:41:23 2020-11-25 23:17:28 [INFO] [TRAIN] epoch=34, iter=21900/40000, loss=0.2699, lr=0.004899, batch_cost=0.5383, reader_cost=0.0078 | ETA 02:42:22 2020-11-25 23:18:22 [INFO] [TRAIN] epoch=34, iter=22000/40000, loss=0.2609, lr=0.004874, batch_cost=0.5345, reader_cost=0.0014 | ETA 02:40:21 2020-11-25 23:19:15 [INFO] [TRAIN] epoch=34, iter=22100/40000, loss=0.2474, lr=0.004850, batch_cost=0.5341, reader_cost=0.0013 | ETA 02:39:20 2020-11-25 23:20:08 [INFO] [TRAIN] epoch=34, iter=22200/40000, loss=0.2503, lr=0.004826, batch_cost=0.5321, reader_cost=0.0011 | ETA 02:37:50 2020-11-25 23:21:01 [INFO] [TRAIN] epoch=34, iter=22300/40000, loss=0.2515, lr=0.004801, batch_cost=0.5318, reader_cost=0.0012 | ETA 02:36:52 2020-11-25 23:21:55 [INFO] [TRAIN] epoch=34, iter=22400/40000, loss=0.2406, lr=0.004777, batch_cost=0.5316, reader_cost=0.0018 | ETA 02:35:55 2020-11-25 23:22:48 [INFO] [TRAIN] epoch=35, iter=22500/40000, loss=0.2432, lr=0.004752, batch_cost=0.5386, reader_cost=0.0076 | ETA 02:37:05 2020-11-25 23:23:42 [INFO] [TRAIN] epoch=35, iter=22600/40000, loss=0.2498, lr=0.004728, batch_cost=0.5322, reader_cost=0.0014 | ETA 02:34:20 2020-11-25 23:24:35 [INFO] [TRAIN] epoch=35, iter=22700/40000, loss=0.2336, lr=0.004703, batch_cost=0.5337, reader_cost=0.0014 | ETA 02:33:52 2020-11-25 23:25:28 [INFO] [TRAIN] epoch=35, iter=22800/40000, loss=0.2375, lr=0.004679, batch_cost=0.5313, reader_cost=0.0011 | ETA 02:32:19 2020-11-25 23:26:22 [INFO] [TRAIN] epoch=35, iter=22900/40000, loss=0.2383, lr=0.004654, batch_cost=0.5341, reader_cost=0.0016 | ETA 02:32:12 2020-11-25 23:27:15 [INFO] [TRAIN] epoch=35, iter=23000/40000, loss=0.2489, lr=0.004630, batch_cost=0.5333, reader_cost=0.0015 | ETA 02:31:06 2020-11-25 23:28:08 [INFO] [TRAIN] epoch=35, iter=23100/40000, loss=0.2606, lr=0.004605, batch_cost=0.5352, reader_cost=0.0015 | ETA 02:30:44 2020-11-25 23:29:02 [INFO] [TRAIN] epoch=36, iter=23200/40000, loss=0.2282, lr=0.004581, batch_cost=0.5388, reader_cost=0.0071 | ETA 02:30:52 2020-11-25 23:29:56 [INFO] [TRAIN] epoch=36, iter=23300/40000, loss=0.2276, lr=0.004556, batch_cost=0.5330, reader_cost=0.0015 | ETA 02:28:21 2020-11-25 23:30:49 [INFO] [TRAIN] epoch=36, iter=23400/40000, loss=0.2457, lr=0.004532, batch_cost=0.5311, reader_cost=0.0016 | ETA 02:26:55 2020-11-25 23:31:42 [INFO] [TRAIN] epoch=36, iter=23500/40000, loss=0.2336, lr=0.004507, batch_cost=0.5307, reader_cost=0.0013 | ETA 02:25:56 2020-11-25 23:32:35 [INFO] [TRAIN] epoch=36, iter=23600/40000, loss=0.2465, lr=0.004483, batch_cost=0.5328, reader_cost=0.0017 | ETA 02:25:37 2020-11-25 23:33:28 [INFO] [TRAIN] epoch=36, iter=23700/40000, loss=0.2484, lr=0.004458, batch_cost=0.5319, reader_cost=0.0018 | ETA 02:24:29 2020-11-25 23:34:23 [INFO] [TRAIN] epoch=37, iter=23800/40000, loss=0.2414, lr=0.004433, batch_cost=0.5464, reader_cost=0.0079 | ETA 02:27:31 2020-11-25 23:35:16 [INFO] [TRAIN] epoch=37, iter=23900/40000, loss=0.2305, lr=0.004409, batch_cost=0.5303, reader_cost=0.0015 | ETA 02:22:17 2020-11-25 23:36:09 [INFO] [TRAIN] epoch=37, iter=24000/40000, loss=0.2521, lr=0.004384, batch_cost=0.5306, reader_cost=0.0017 | ETA 02:21:30 2020-11-25 23:36:09 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-25 23:36:32 [INFO] [EVAL] #Images=1449 mIoU=0.7956 Acc=0.9535 Kappa=0.8971 2020-11-25 23:36:32 [INFO] [EVAL] Class IoU: [0.946 0.9231 0.4214 0.9053 0.7156 0.8404 0.9568 0.8764 0.9159 0.4455 0.9303 0.5916 0.8773 0.9115 0.8555 0.8738 0.6424 0.8947 0.5 0.9055 0.7787] 2020-11-25 23:36:32 [INFO] [EVAL] Class Acc: [0.9676 0.9534 0.441 0.9454 0.7671 0.9142 0.9783 0.9306 0.9369 0.7154 0.9587 0.9477 0.9425 0.9437 0.9203 0.9465 0.8408 0.9387 0.8201 0.9405 0.9142] 2020-11-25 23:36:43 [INFO] [EVAL] The model with the best validation mIoU (0.7956) was saved at iter 24000. 2020-11-25 23:37:37 [INFO] [TRAIN] epoch=37, iter=24100/40000, loss=0.2255, lr=0.004359, batch_cost=0.5317, reader_cost=0.0015 | ETA 02:20:54 2020-11-25 23:38:30 [INFO] [TRAIN] epoch=37, iter=24200/40000, loss=0.2186, lr=0.004335, batch_cost=0.5313, reader_cost=0.0016 | ETA 02:19:54 2020-11-25 23:39:23 [INFO] [TRAIN] epoch=37, iter=24300/40000, loss=0.2429, lr=0.004310, batch_cost=0.5311, reader_cost=0.0015 | ETA 02:18:58 2020-11-25 23:40:16 [INFO] [TRAIN] epoch=37, iter=24400/40000, loss=0.2446, lr=0.004285, batch_cost=0.5322, reader_cost=0.0015 | ETA 02:18:22 2020-11-25 23:41:10 [INFO] [TRAIN] epoch=38, iter=24500/40000, loss=0.2244, lr=0.004261, batch_cost=0.5376, reader_cost=0.0082 | ETA 02:18:52 2020-11-25 23:42:03 [INFO] [TRAIN] epoch=38, iter=24600/40000, loss=0.2366, lr=0.004236, batch_cost=0.5288, reader_cost=0.0019 | ETA 02:15:43 2020-11-25 23:42:56 [INFO] [TRAIN] epoch=38, iter=24700/40000, loss=0.2597, lr=0.004211, batch_cost=0.5305, reader_cost=0.0012 | ETA 02:15:16 2020-11-25 23:43:49 [INFO] [TRAIN] epoch=38, iter=24800/40000, loss=0.2509, lr=0.004186, batch_cost=0.5323, reader_cost=0.0020 | ETA 02:14:51 2020-11-25 23:44:42 [INFO] [TRAIN] epoch=38, iter=24900/40000, loss=0.2281, lr=0.004162, batch_cost=0.5313, reader_cost=0.0014 | ETA 02:13:42 2020-11-25 23:45:35 [INFO] [TRAIN] epoch=38, iter=25000/40000, loss=0.2096, lr=0.004137, batch_cost=0.5288, reader_cost=0.0013 | ETA 02:12:12 2020-11-25 23:46:28 [INFO] [TRAIN] epoch=38, iter=25100/40000, loss=0.2249, lr=0.004112, batch_cost=0.5303, reader_cost=0.0014 | ETA 02:11:42 2020-11-25 23:47:22 [INFO] [TRAIN] epoch=39, iter=25200/40000, loss=0.1998, lr=0.004087, batch_cost=0.5359, reader_cost=0.0076 | ETA 02:12:11 2020-11-25 23:48:15 [INFO] [TRAIN] epoch=39, iter=25300/40000, loss=0.2441, lr=0.004062, batch_cost=0.5298, reader_cost=0.0014 | ETA 02:09:48 2020-11-25 23:49:08 [INFO] [TRAIN] epoch=39, iter=25400/40000, loss=0.2376, lr=0.004037, batch_cost=0.5297, reader_cost=0.0016 | ETA 02:08:53 2020-11-25 23:50:01 [INFO] [TRAIN] epoch=39, iter=25500/40000, loss=0.2322, lr=0.004012, batch_cost=0.5323, reader_cost=0.0017 | ETA 02:08:37 2020-11-25 23:50:54 [INFO] [TRAIN] epoch=39, iter=25600/40000, loss=0.2045, lr=0.003987, batch_cost=0.5323, reader_cost=0.0015 | ETA 02:07:44 2020-11-25 23:51:47 [INFO] [TRAIN] epoch=39, iter=25700/40000, loss=0.2080, lr=0.003963, batch_cost=0.5300, reader_cost=0.0016 | ETA 02:06:18 2020-11-25 23:52:41 [INFO] [TRAIN] epoch=40, iter=25800/40000, loss=0.2274, lr=0.003938, batch_cost=0.5348, reader_cost=0.0070 | ETA 02:06:33 2020-11-25 23:53:34 [INFO] [TRAIN] epoch=40, iter=25900/40000, loss=0.2301, lr=0.003913, batch_cost=0.5304, reader_cost=0.0013 | ETA 02:04:38 2020-11-25 23:54:26 [INFO] [TRAIN] epoch=40, iter=26000/40000, loss=0.2751, lr=0.003888, batch_cost=0.5276, reader_cost=0.0015 | ETA 02:03:06 2020-11-25 23:55:19 [INFO] [TRAIN] epoch=40, iter=26100/40000, loss=0.2236, lr=0.003863, batch_cost=0.5269, reader_cost=0.0017 | ETA 02:02:03 2020-11-25 23:56:12 [INFO] [TRAIN] epoch=40, iter=26200/40000, loss=0.2262, lr=0.003838, batch_cost=0.5281, reader_cost=0.0015 | ETA 02:01:28 2020-11-25 23:57:05 [INFO] [TRAIN] epoch=40, iter=26300/40000, loss=0.2134, lr=0.003813, batch_cost=0.5288, reader_cost=0.0013 | ETA 02:00:45 2020-11-25 23:57:58 [INFO] [TRAIN] epoch=40, iter=26400/40000, loss=0.2282, lr=0.003788, batch_cost=0.5301, reader_cost=0.0014 | ETA 02:00:09 2020-11-25 23:58:51 [INFO] [TRAIN] epoch=41, iter=26500/40000, loss=0.2086, lr=0.003762, batch_cost=0.5338, reader_cost=0.0072 | ETA 02:00:06 2020-11-25 23:59:44 [INFO] [TRAIN] epoch=41, iter=26600/40000, loss=0.2095, lr=0.003737, batch_cost=0.5268, reader_cost=0.0016 | ETA 01:57:38 2020-11-26 00:00:37 [INFO] [TRAIN] epoch=41, iter=26700/40000, loss=0.2367, lr=0.003712, batch_cost=0.5306, reader_cost=0.0013 | ETA 01:57:36 2020-11-26 00:01:29 [INFO] [TRAIN] epoch=41, iter=26800/40000, loss=0.2286, lr=0.003687, batch_cost=0.5206, reader_cost=0.0008 | ETA 01:54:32 2020-11-26 00:02:21 [INFO] [TRAIN] epoch=41, iter=26900/40000, loss=0.2211, lr=0.003662, batch_cost=0.5189, reader_cost=0.0007 | ETA 01:53:17 2020-11-26 00:03:13 [INFO] [TRAIN] epoch=41, iter=27000/40000, loss=0.2338, lr=0.003637, batch_cost=0.5258, reader_cost=0.0012 | ETA 01:53:55 2020-11-26 00:04:06 [INFO] [TRAIN] epoch=41, iter=27100/40000, loss=0.2186, lr=0.003612, batch_cost=0.5289, reader_cost=0.0014 | ETA 01:53:42 2020-11-26 00:05:00 [INFO] [TRAIN] epoch=42, iter=27200/40000, loss=0.2350, lr=0.003586, batch_cost=0.5338, reader_cost=0.0071 | ETA 01:53:52 2020-11-26 00:05:53 [INFO] [TRAIN] epoch=42, iter=27300/40000, loss=0.2290, lr=0.003561, batch_cost=0.5304, reader_cost=0.0017 | ETA 01:52:15 2020-11-26 00:06:46 [INFO] [TRAIN] epoch=42, iter=27400/40000, loss=0.2171, lr=0.003536, batch_cost=0.5295, reader_cost=0.0015 | ETA 01:51:11 2020-11-26 00:07:39 [INFO] [TRAIN] epoch=42, iter=27500/40000, loss=0.2075, lr=0.003511, batch_cost=0.5286, reader_cost=0.0015 | ETA 01:50:08 2020-11-26 00:08:31 [INFO] [TRAIN] epoch=42, iter=27600/40000, loss=0.2158, lr=0.003485, batch_cost=0.5279, reader_cost=0.0014 | ETA 01:49:05 2020-11-26 00:09:24 [INFO] [TRAIN] epoch=42, iter=27700/40000, loss=0.2187, lr=0.003460, batch_cost=0.5277, reader_cost=0.0015 | ETA 01:48:10 2020-11-26 00:10:18 [INFO] [TRAIN] epoch=43, iter=27800/40000, loss=0.2185, lr=0.003435, batch_cost=0.5344, reader_cost=0.0070 | ETA 01:48:39 2020-11-26 00:11:11 [INFO] [TRAIN] epoch=43, iter=27900/40000, loss=0.2038, lr=0.003409, batch_cost=0.5304, reader_cost=0.0014 | ETA 01:46:57 2020-11-26 00:12:04 [INFO] [TRAIN] epoch=43, iter=28000/40000, loss=0.2241, lr=0.003384, batch_cost=0.5326, reader_cost=0.0014 | ETA 01:46:30 2020-11-26 00:12:04 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-26 00:12:28 [INFO] [EVAL] #Images=1449 mIoU=0.7986 Acc=0.9548 Kappa=0.8991 2020-11-26 00:12:28 [INFO] [EVAL] Class IoU: [0.9472 0.9287 0.4426 0.9027 0.7359 0.8492 0.9452 0.8874 0.947 0.4375 0.9018 0.51 0.917 0.8981 0.8716 0.8759 0.6617 0.895 0.532 0.8913 0.7933] 2020-11-26 00:12:28 [INFO] [EVAL] Class Acc: [0.9647 0.9732 0.4617 0.9539 0.8305 0.9135 0.982 0.9439 0.9728 0.7137 0.9655 0.942 0.9457 0.9442 0.9461 0.9476 0.8762 0.9496 0.8234 0.9677 0.9329] 2020-11-26 00:12:39 [INFO] [EVAL] The model with the best validation mIoU (0.7986) was saved at iter 28000. 2020-11-26 00:13:32 [INFO] [TRAIN] epoch=43, iter=28100/40000, loss=0.2196, lr=0.003359, batch_cost=0.5292, reader_cost=0.0013 | ETA 01:44:57 2020-11-26 00:14:25 [INFO] [TRAIN] epoch=43, iter=28200/40000, loss=0.2288, lr=0.003333, batch_cost=0.5331, reader_cost=0.0016 | ETA 01:44:50 2020-11-26 00:15:18 [INFO] [TRAIN] epoch=43, iter=28300/40000, loss=0.2157, lr=0.003308, batch_cost=0.5323, reader_cost=0.0015 | ETA 01:43:47 2020-11-26 00:16:12 [INFO] [TRAIN] epoch=43, iter=28400/40000, loss=0.2106, lr=0.003282, batch_cost=0.5341, reader_cost=0.0016 | ETA 01:43:15 2020-11-26 00:17:05 [INFO] [TRAIN] epoch=44, iter=28500/40000, loss=0.1893, lr=0.003257, batch_cost=0.5372, reader_cost=0.0073 | ETA 01:42:57 2020-11-26 00:17:59 [INFO] [TRAIN] epoch=44, iter=28600/40000, loss=0.2258, lr=0.003231, batch_cost=0.5323, reader_cost=0.0014 | ETA 01:41:07 2020-11-26 00:18:52 [INFO] [TRAIN] epoch=44, iter=28700/40000, loss=0.2119, lr=0.003206, batch_cost=0.5313, reader_cost=0.0015 | ETA 01:40:03 2020-11-26 00:19:45 [INFO] [TRAIN] epoch=44, iter=28800/40000, loss=0.2227, lr=0.003180, batch_cost=0.5321, reader_cost=0.0016 | ETA 01:39:19 2020-11-26 00:20:38 [INFO] [TRAIN] epoch=44, iter=28900/40000, loss=0.2066, lr=0.003155, batch_cost=0.5324, reader_cost=0.0016 | ETA 01:38:29 2020-11-26 00:21:32 [INFO] [TRAIN] epoch=44, iter=29000/40000, loss=0.2356, lr=0.003129, batch_cost=0.5327, reader_cost=0.0015 | ETA 01:37:39 2020-11-26 00:22:25 [INFO] [TRAIN] epoch=45, iter=29100/40000, loss=0.2195, lr=0.003104, batch_cost=0.5380, reader_cost=0.0076 | ETA 01:37:44 2020-11-26 00:23:18 [INFO] [TRAIN] epoch=45, iter=29200/40000, loss=0.2299, lr=0.003078, batch_cost=0.5304, reader_cost=0.0016 | ETA 01:35:28 2020-11-26 00:24:12 [INFO] [TRAIN] epoch=45, iter=29300/40000, loss=0.2063, lr=0.003052, batch_cost=0.5330, reader_cost=0.0016 | ETA 01:35:02 2020-11-26 00:25:05 [INFO] [TRAIN] epoch=45, iter=29400/40000, loss=0.2041, lr=0.003027, batch_cost=0.5329, reader_cost=0.0016 | ETA 01:34:08 2020-11-26 00:25:58 [INFO] [TRAIN] epoch=45, iter=29500/40000, loss=0.2113, lr=0.003001, batch_cost=0.5325, reader_cost=0.0016 | ETA 01:33:11 2020-11-26 00:26:51 [INFO] [TRAIN] epoch=45, iter=29600/40000, loss=0.1938, lr=0.002975, batch_cost=0.5316, reader_cost=0.0013 | ETA 01:32:08 2020-11-26 00:27:44 [INFO] [TRAIN] epoch=45, iter=29700/40000, loss=0.1931, lr=0.002949, batch_cost=0.5304, reader_cost=0.0016 | ETA 01:31:02 2020-11-26 00:28:38 [INFO] [TRAIN] epoch=46, iter=29800/40000, loss=0.1880, lr=0.002924, batch_cost=0.5388, reader_cost=0.0076 | ETA 01:31:35 2020-11-26 00:29:32 [INFO] [TRAIN] epoch=46, iter=29900/40000, loss=0.2146, lr=0.002898, batch_cost=0.5360, reader_cost=0.0014 | ETA 01:30:13 2020-11-26 00:30:25 [INFO] [TRAIN] epoch=46, iter=30000/40000, loss=0.2176, lr=0.002872, batch_cost=0.5302, reader_cost=0.0011 | ETA 01:28:22 2020-11-26 00:31:18 [INFO] [TRAIN] epoch=46, iter=30100/40000, loss=0.2059, lr=0.002846, batch_cost=0.5318, reader_cost=0.0014 | ETA 01:27:45 2020-11-26 00:32:11 [INFO] [TRAIN] epoch=46, iter=30200/40000, loss=0.2104, lr=0.002820, batch_cost=0.5315, reader_cost=0.0014 | ETA 01:26:49 2020-11-26 00:33:05 [INFO] [TRAIN] epoch=46, iter=30300/40000, loss=0.2025, lr=0.002794, batch_cost=0.5322, reader_cost=0.0014 | ETA 01:26:02 2020-11-26 00:33:58 [INFO] [TRAIN] epoch=46, iter=30400/40000, loss=0.2049, lr=0.002768, batch_cost=0.5326, reader_cost=0.0011 | ETA 01:25:13 2020-11-26 00:34:52 [INFO] [TRAIN] epoch=47, iter=30500/40000, loss=0.1932, lr=0.002742, batch_cost=0.5384, reader_cost=0.0077 | ETA 01:25:14 2020-11-26 00:35:45 [INFO] [TRAIN] epoch=47, iter=30600/40000, loss=0.2034, lr=0.002716, batch_cost=0.5326, reader_cost=0.0018 | ETA 01:23:26 2020-11-26 00:36:38 [INFO] [TRAIN] epoch=47, iter=30700/40000, loss=0.2142, lr=0.002690, batch_cost=0.5326, reader_cost=0.0014 | ETA 01:22:32 2020-11-26 00:37:31 [INFO] [TRAIN] epoch=47, iter=30800/40000, loss=0.2042, lr=0.002664, batch_cost=0.5333, reader_cost=0.0012 | ETA 01:21:45 2020-11-26 00:38:25 [INFO] [TRAIN] epoch=47, iter=30900/40000, loss=0.2044, lr=0.002638, batch_cost=0.5320, reader_cost=0.0012 | ETA 01:20:41 2020-11-26 00:39:18 [INFO] [TRAIN] epoch=47, iter=31000/40000, loss=0.2021, lr=0.002612, batch_cost=0.5351, reader_cost=0.0018 | ETA 01:20:15 2020-11-26 00:40:12 [INFO] [TRAIN] epoch=48, iter=31100/40000, loss=0.1867, lr=0.002586, batch_cost=0.5400, reader_cost=0.0072 | ETA 01:20:05 2020-11-26 00:41:06 [INFO] [TRAIN] epoch=48, iter=31200/40000, loss=0.1982, lr=0.002560, batch_cost=0.5346, reader_cost=0.0018 | ETA 01:18:24 2020-11-26 00:41:59 [INFO] [TRAIN] epoch=48, iter=31300/40000, loss=0.2111, lr=0.002534, batch_cost=0.5342, reader_cost=0.0011 | ETA 01:17:27 2020-11-26 00:42:52 [INFO] [TRAIN] epoch=48, iter=31400/40000, loss=0.1871, lr=0.002507, batch_cost=0.5336, reader_cost=0.0016 | ETA 01:16:28 2020-11-26 00:43:45 [INFO] [TRAIN] epoch=48, iter=31500/40000, loss=0.1988, lr=0.002481, batch_cost=0.5305, reader_cost=0.0013 | ETA 01:15:09 2020-11-26 00:44:39 [INFO] [TRAIN] epoch=48, iter=31600/40000, loss=0.2106, lr=0.002455, batch_cost=0.5332, reader_cost=0.0018 | ETA 01:14:38 2020-11-26 00:45:32 [INFO] [TRAIN] epoch=48, iter=31700/40000, loss=0.1950, lr=0.002429, batch_cost=0.5321, reader_cost=0.0016 | ETA 01:13:36 2020-11-26 00:46:26 [INFO] [TRAIN] epoch=49, iter=31800/40000, loss=0.1970, lr=0.002402, batch_cost=0.5392, reader_cost=0.0077 | ETA 01:13:41 2020-11-26 00:47:19 [INFO] [TRAIN] epoch=49, iter=31900/40000, loss=0.2029, lr=0.002376, batch_cost=0.5336, reader_cost=0.0018 | ETA 01:12:02 2020-11-26 00:48:12 [INFO] [TRAIN] epoch=49, iter=32000/40000, loss=0.1968, lr=0.002350, batch_cost=0.5308, reader_cost=0.0013 | ETA 01:10:46 2020-11-26 00:48:12 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-26 00:48:36 [INFO] [EVAL] #Images=1449 mIoU=0.8029 Acc=0.9553 Kappa=0.9007 2020-11-26 00:48:36 [INFO] [EVAL] Class IoU: [0.948 0.9163 0.4426 0.9069 0.7314 0.8526 0.9566 0.8867 0.9333 0.4862 0.9246 0.4817 0.9052 0.9122 0.8753 0.8744 0.6633 0.9013 0.5427 0.9145 0.804 ] 2020-11-26 00:48:36 [INFO] [EVAL] Class Acc: [0.9671 0.943 0.466 0.9596 0.8195 0.9177 0.9842 0.938 0.9631 0.6866 0.9694 0.9384 0.9447 0.9558 0.9459 0.948 0.8437 0.9527 0.7898 0.9669 0.9601] 2020-11-26 00:48:48 [INFO] [EVAL] The model with the best validation mIoU (0.8029) was saved at iter 32000. 2020-11-26 00:49:41 [INFO] [TRAIN] epoch=49, iter=32100/40000, loss=0.2130, lr=0.002323, batch_cost=0.5317, reader_cost=0.0015 | ETA 01:10:00 2020-11-26 00:50:34 [INFO] [TRAIN] epoch=49, iter=32200/40000, loss=0.1920, lr=0.002297, batch_cost=0.5323, reader_cost=0.0018 | ETA 01:09:12 2020-11-26 00:51:28 [INFO] [TRAIN] epoch=49, iter=32300/40000, loss=0.1872, lr=0.002270, batch_cost=0.5313, reader_cost=0.0013 | ETA 01:08:10 2020-11-26 00:52:22 [INFO] [TRAIN] epoch=50, iter=32400/40000, loss=0.1984, lr=0.002244, batch_cost=0.5405, reader_cost=0.0069 | ETA 01:08:27 2020-11-26 00:53:15 [INFO] [TRAIN] epoch=50, iter=32500/40000, loss=0.1867, lr=0.002217, batch_cost=0.5343, reader_cost=0.0008 | ETA 01:06:47 2020-11-26 00:54:08 [INFO] [TRAIN] epoch=50, iter=32600/40000, loss=0.2169, lr=0.002190, batch_cost=0.5343, reader_cost=0.0012 | ETA 01:05:53 2020-11-26 00:55:02 [INFO] [TRAIN] epoch=50, iter=32700/40000, loss=0.2042, lr=0.002164, batch_cost=0.5305, reader_cost=0.0015 | ETA 01:04:32 2020-11-26 00:55:55 [INFO] [TRAIN] epoch=50, iter=32800/40000, loss=0.2003, lr=0.002137, batch_cost=0.5301, reader_cost=0.0015 | ETA 01:03:36 2020-11-26 00:56:48 [INFO] [TRAIN] epoch=50, iter=32900/40000, loss=0.2101, lr=0.002110, batch_cost=0.5310, reader_cost=0.0012 | ETA 01:02:50 2020-11-26 00:57:41 [INFO] [TRAIN] epoch=50, iter=33000/40000, loss=0.1819, lr=0.002083, batch_cost=0.5324, reader_cost=0.0015 | ETA 01:02:06 2020-11-26 00:58:35 [INFO] [TRAIN] epoch=51, iter=33100/40000, loss=0.1946, lr=0.002057, batch_cost=0.5454, reader_cost=0.0072 | ETA 01:02:43 2020-11-26 00:59:28 [INFO] [TRAIN] epoch=51, iter=33200/40000, loss=0.2056, lr=0.002030, batch_cost=0.5302, reader_cost=0.0016 | ETA 01:00:05 2020-11-26 01:00:22 [INFO] [TRAIN] epoch=51, iter=33300/40000, loss=0.1872, lr=0.002003, batch_cost=0.5307, reader_cost=0.0016 | ETA 00:59:15 2020-11-26 01:01:15 [INFO] [TRAIN] epoch=51, iter=33400/40000, loss=0.2019, lr=0.001976, batch_cost=0.5306, reader_cost=0.0015 | ETA 00:58:21 2020-11-26 01:02:08 [INFO] [TRAIN] epoch=51, iter=33500/40000, loss=0.2036, lr=0.001949, batch_cost=0.5327, reader_cost=0.0015 | ETA 00:57:42 2020-11-26 01:03:01 [INFO] [TRAIN] epoch=51, iter=33600/40000, loss=0.2008, lr=0.001922, batch_cost=0.5318, reader_cost=0.0015 | ETA 00:56:43 2020-11-26 01:03:54 [INFO] [TRAIN] epoch=51, iter=33700/40000, loss=0.1783, lr=0.001895, batch_cost=0.5334, reader_cost=0.0015 | ETA 00:56:00 2020-11-26 01:04:48 [INFO] [TRAIN] epoch=52, iter=33800/40000, loss=0.1997, lr=0.001868, batch_cost=0.5391, reader_cost=0.0075 | ETA 00:55:42 2020-11-26 01:05:41 [INFO] [TRAIN] epoch=52, iter=33900/40000, loss=0.1975, lr=0.001841, batch_cost=0.5318, reader_cost=0.0013 | ETA 00:54:04 2020-11-26 01:06:35 [INFO] [TRAIN] epoch=52, iter=34000/40000, loss=0.1889, lr=0.001814, batch_cost=0.5308, reader_cost=0.0015 | ETA 00:53:04 2020-11-26 01:07:28 [INFO] [TRAIN] epoch=52, iter=34100/40000, loss=0.1825, lr=0.001786, batch_cost=0.5316, reader_cost=0.0015 | ETA 00:52:16 2020-11-26 01:08:21 [INFO] [TRAIN] epoch=52, iter=34200/40000, loss=0.1949, lr=0.001759, batch_cost=0.5335, reader_cost=0.0015 | ETA 00:51:34 2020-11-26 01:09:14 [INFO] [TRAIN] epoch=52, iter=34300/40000, loss=0.1819, lr=0.001732, batch_cost=0.5328, reader_cost=0.0017 | ETA 00:50:37 2020-11-26 01:10:08 [INFO] [TRAIN] epoch=53, iter=34400/40000, loss=0.1954, lr=0.001704, batch_cost=0.5373, reader_cost=0.0067 | ETA 00:50:08 2020-11-26 01:11:01 [INFO] [TRAIN] epoch=53, iter=34500/40000, loss=0.1733, lr=0.001677, batch_cost=0.5325, reader_cost=0.0016 | ETA 00:48:48 2020-11-26 01:11:55 [INFO] [TRAIN] epoch=53, iter=34600/40000, loss=0.1812, lr=0.001650, batch_cost=0.5330, reader_cost=0.0012 | ETA 00:47:58 2020-11-26 01:12:48 [INFO] [TRAIN] epoch=53, iter=34700/40000, loss=0.1842, lr=0.001622, batch_cost=0.5353, reader_cost=0.0016 | ETA 00:47:17 2020-11-26 01:13:41 [INFO] [TRAIN] epoch=53, iter=34800/40000, loss=0.1963, lr=0.001594, batch_cost=0.5328, reader_cost=0.0013 | ETA 00:46:10 2020-11-26 01:14:35 [INFO] [TRAIN] epoch=53, iter=34900/40000, loss=0.1860, lr=0.001567, batch_cost=0.5321, reader_cost=0.0014 | ETA 00:45:13 2020-11-26 01:15:28 [INFO] [TRAIN] epoch=53, iter=35000/40000, loss=0.2045, lr=0.001539, batch_cost=0.5333, reader_cost=0.0014 | ETA 00:44:26 2020-11-26 01:16:22 [INFO] [TRAIN] epoch=54, iter=35100/40000, loss=0.1879, lr=0.001511, batch_cost=0.5383, reader_cost=0.0081 | ETA 00:43:57 2020-11-26 01:17:15 [INFO] [TRAIN] epoch=54, iter=35200/40000, loss=0.2027, lr=0.001484, batch_cost=0.5327, reader_cost=0.0016 | ETA 00:42:36 2020-11-26 01:18:09 [INFO] [TRAIN] epoch=54, iter=35300/40000, loss=0.1911, lr=0.001456, batch_cost=0.5338, reader_cost=0.0016 | ETA 00:41:49 2020-11-26 01:19:02 [INFO] [TRAIN] epoch=54, iter=35400/40000, loss=0.1942, lr=0.001428, batch_cost=0.5319, reader_cost=0.0016 | ETA 00:40:46 2020-11-26 01:19:55 [INFO] [TRAIN] epoch=54, iter=35500/40000, loss=0.1749, lr=0.001400, batch_cost=0.5301, reader_cost=0.0014 | ETA 00:39:45 2020-11-26 01:20:48 [INFO] [TRAIN] epoch=54, iter=35600/40000, loss=0.2041, lr=0.001372, batch_cost=0.5328, reader_cost=0.0012 | ETA 00:39:04 2020-11-26 01:21:42 [INFO] [TRAIN] epoch=55, iter=35700/40000, loss=0.2050, lr=0.001344, batch_cost=0.5404, reader_cost=0.0073 | ETA 00:38:43 2020-11-26 01:22:35 [INFO] [TRAIN] epoch=55, iter=35800/40000, loss=0.1851, lr=0.001316, batch_cost=0.5314, reader_cost=0.0013 | ETA 00:37:11 2020-11-26 01:23:28 [INFO] [TRAIN] epoch=55, iter=35900/40000, loss=0.1961, lr=0.001287, batch_cost=0.5321, reader_cost=0.0018 | ETA 00:36:21 2020-11-26 01:24:22 [INFO] [TRAIN] epoch=55, iter=36000/40000, loss=0.1936, lr=0.001259, batch_cost=0.5318, reader_cost=0.0013 | ETA 00:35:27 2020-11-26 01:24:22 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-26 01:24:45 [INFO] [EVAL] #Images=1449 mIoU=0.8014 Acc=0.9546 Kappa=0.8988 2020-11-26 01:24:45 [INFO] [EVAL] Class IoU: [0.9463 0.9133 0.4391 0.9115 0.7323 0.8543 0.9563 0.8971 0.9363 0.4872 0.9257 0.4779 0.9177 0.9125 0.8941 0.8755 0.6592 0.9014 0.4851 0.9109 0.7963] 2020-11-26 01:24:45 [INFO] [EVAL] Class Acc: [0.9648 0.939 0.4558 0.9557 0.8126 0.92 0.9851 0.9588 0.9602 0.7132 0.9739 0.9577 0.9571 0.9556 0.9354 0.9486 0.842 0.9596 0.8062 0.9626 0.9615] 2020-11-26 01:24:53 [INFO] [EVAL] The model with the best validation mIoU (0.8029) was saved at iter 32000. 2020-11-26 01:25:46 [INFO] [TRAIN] epoch=55, iter=36100/40000, loss=0.1745, lr=0.001231, batch_cost=0.5373, reader_cost=0.0017 | ETA 00:34:55 2020-11-26 01:26:40 [INFO] [TRAIN] epoch=55, iter=36200/40000, loss=0.2051, lr=0.001202, batch_cost=0.5318, reader_cost=0.0018 | ETA 00:33:40 2020-11-26 01:27:33 [INFO] [TRAIN] epoch=55, iter=36300/40000, loss=0.1893, lr=0.001174, batch_cost=0.5318, reader_cost=0.0014 | ETA 00:32:47 2020-11-26 01:28:27 [INFO] [TRAIN] epoch=56, iter=36400/40000, loss=0.1578, lr=0.001145, batch_cost=0.5373, reader_cost=0.0074 | ETA 00:32:14 2020-11-26 01:29:20 [INFO] [TRAIN] epoch=56, iter=36500/40000, loss=0.1811, lr=0.001117, batch_cost=0.5348, reader_cost=0.0014 | ETA 00:31:11 2020-11-26 01:30:13 [INFO] [TRAIN] epoch=56, iter=36600/40000, loss=0.2045, lr=0.001088, batch_cost=0.5339, reader_cost=0.0017 | ETA 00:30:15 2020-11-26 01:31:07 [INFO] [TRAIN] epoch=56, iter=36700/40000, loss=0.1836, lr=0.001059, batch_cost=0.5347, reader_cost=0.0013 | ETA 00:29:24 2020-11-26 01:32:00 [INFO] [TRAIN] epoch=56, iter=36800/40000, loss=0.1753, lr=0.001030, batch_cost=0.5315, reader_cost=0.0015 | ETA 00:28:20 2020-11-26 01:32:53 [INFO] [TRAIN] epoch=56, iter=36900/40000, loss=0.1992, lr=0.001001, batch_cost=0.5334, reader_cost=0.0012 | ETA 00:27:33 2020-11-26 01:33:47 [INFO] [TRAIN] epoch=56, iter=37000/40000, loss=0.1705, lr=0.000972, batch_cost=0.5317, reader_cost=0.0016 | ETA 00:26:35 2020-11-26 01:34:40 [INFO] [TRAIN] epoch=57, iter=37100/40000, loss=0.1843, lr=0.000943, batch_cost=0.5391, reader_cost=0.0073 | ETA 00:26:03 2020-11-26 01:35:34 [INFO] [TRAIN] epoch=57, iter=37200/40000, loss=0.1828, lr=0.000914, batch_cost=0.5346, reader_cost=0.0016 | ETA 00:24:56 2020-11-26 01:36:27 [INFO] [TRAIN] epoch=57, iter=37300/40000, loss=0.1821, lr=0.000884, batch_cost=0.5316, reader_cost=0.0015 | ETA 00:23:55 2020-11-26 01:37:20 [INFO] [TRAIN] epoch=57, iter=37400/40000, loss=0.1931, lr=0.000855, batch_cost=0.5315, reader_cost=0.0013 | ETA 00:23:01 2020-11-26 01:38:14 [INFO] [TRAIN] epoch=57, iter=37500/40000, loss=0.1880, lr=0.000825, batch_cost=0.5333, reader_cost=0.0014 | ETA 00:22:13 2020-11-26 01:39:07 [INFO] [TRAIN] epoch=57, iter=37600/40000, loss=0.1877, lr=0.000795, batch_cost=0.5302, reader_cost=0.0018 | ETA 00:21:12 2020-11-26 01:40:00 [INFO] [TRAIN] epoch=58, iter=37700/40000, loss=0.1744, lr=0.000765, batch_cost=0.5373, reader_cost=0.0074 | ETA 00:20:35 2020-11-26 01:40:54 [INFO] [TRAIN] epoch=58, iter=37800/40000, loss=0.1723, lr=0.000735, batch_cost=0.5325, reader_cost=0.0011 | ETA 00:19:31 2020-11-26 01:41:47 [INFO] [TRAIN] epoch=58, iter=37900/40000, loss=0.1845, lr=0.000705, batch_cost=0.5332, reader_cost=0.0011 | ETA 00:18:39 2020-11-26 01:42:40 [INFO] [TRAIN] epoch=58, iter=38000/40000, loss=0.1752, lr=0.000675, batch_cost=0.5321, reader_cost=0.0005 | ETA 00:17:44 2020-11-26 01:43:33 [INFO] [TRAIN] epoch=58, iter=38100/40000, loss=0.1779, lr=0.000645, batch_cost=0.5322, reader_cost=0.0007 | ETA 00:16:51 2020-11-26 01:44:25 [INFO] [TRAIN] epoch=58, iter=38200/40000, loss=0.1788, lr=0.000614, batch_cost=0.5161, reader_cost=0.0002 | ETA 00:15:29 2020-11-26 01:45:16 [INFO] [TRAIN] epoch=58, iter=38300/40000, loss=0.1819, lr=0.000583, batch_cost=0.5123, reader_cost=0.0001 | ETA 00:14:30 2020-11-26 01:46:08 [INFO] [TRAIN] epoch=59, iter=38400/40000, loss=0.1691, lr=0.000552, batch_cost=0.5216, reader_cost=0.0071 | ETA 00:13:54 2020-11-26 01:47:00 [INFO] [TRAIN] epoch=59, iter=38500/40000, loss=0.1866, lr=0.000521, batch_cost=0.5158, reader_cost=0.0005 | ETA 00:12:53 2020-11-26 01:47:52 [INFO] [TRAIN] epoch=59, iter=38600/40000, loss=0.1854, lr=0.000490, batch_cost=0.5175, reader_cost=0.0004 | ETA 00:12:04 2020-11-26 01:48:44 [INFO] [TRAIN] epoch=59, iter=38700/40000, loss=0.1820, lr=0.000458, batch_cost=0.5199, reader_cost=0.0006 | ETA 00:11:15 2020-11-26 01:49:36 [INFO] [TRAIN] epoch=59, iter=38800/40000, loss=0.1853, lr=0.000426, batch_cost=0.5211, reader_cost=0.0002 | ETA 00:10:25 2020-11-26 01:50:27 [INFO] [TRAIN] epoch=59, iter=38900/40000, loss=0.1962, lr=0.000394, batch_cost=0.5144, reader_cost=0.0002 | ETA 00:09:25 2020-11-26 01:51:19 [INFO] [TRAIN] epoch=60, iter=39000/40000, loss=0.1809, lr=0.000362, batch_cost=0.5226, reader_cost=0.0060 | ETA 00:08:42 2020-11-26 01:52:11 [INFO] [TRAIN] epoch=60, iter=39100/40000, loss=0.1778, lr=0.000329, batch_cost=0.5155, reader_cost=0.0004 | ETA 00:07:43 2020-11-26 01:53:03 [INFO] [TRAIN] epoch=60, iter=39200/40000, loss=0.1783, lr=0.000296, batch_cost=0.5174, reader_cost=0.0004 | ETA 00:06:53 2020-11-26 01:53:54 [INFO] [TRAIN] epoch=60, iter=39300/40000, loss=0.1827, lr=0.000263, batch_cost=0.5161, reader_cost=0.0002 | ETA 00:06:01 2020-11-26 01:54:46 [INFO] [TRAIN] epoch=60, iter=39400/40000, loss=0.1856, lr=0.000229, batch_cost=0.5178, reader_cost=0.0001 | ETA 00:05:10 2020-11-26 01:55:38 [INFO] [TRAIN] epoch=60, iter=39500/40000, loss=0.1821, lr=0.000194, batch_cost=0.5167, reader_cost=0.0001 | ETA 00:04:18 2020-11-26 01:56:30 [INFO] [TRAIN] epoch=60, iter=39600/40000, loss=0.1829, lr=0.000159, batch_cost=0.5180, reader_cost=0.0004 | ETA 00:03:27 2020-11-26 01:57:22 [INFO] [TRAIN] epoch=61, iter=39700/40000, loss=0.1782, lr=0.000123, batch_cost=0.5208, reader_cost=0.0064 | ETA 00:02:36 2020-11-26 01:58:13 [INFO] [TRAIN] epoch=61, iter=39800/40000, loss=0.1855, lr=0.000085, batch_cost=0.5169, reader_cost=0.0002 | ETA 00:01:43 2020-11-26 01:59:05 [INFO] [TRAIN] epoch=61, iter=39900/40000, loss=0.1969, lr=0.000046, batch_cost=0.5167, reader_cost=0.0002 | ETA 00:00:51 2020-11-26 01:59:57 [INFO] [TRAIN] epoch=61, iter=40000/40000, loss=0.1807, lr=0.000001, batch_cost=0.5158, reader_cost=0.0001 | ETA 00:00:00 2020-11-26 01:59:57 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-26 02:00:18 [INFO] [EVAL] #Images=1449 mIoU=0.8076 Acc=0.9565 Kappa=0.9033 2020-11-26 02:00:18 [INFO] [EVAL] Class IoU: [0.9486 0.9218 0.441 0.9067 0.7229 0.8576 0.9616 0.8981 0.9393 0.497 0.9249 0.5486 0.9172 0.916 0.8878 0.8783 0.6698 0.8951 0.5065 0.9257 0.7961] 2020-11-26 02:00:18 [INFO] [EVAL] Class Acc: [0.9676 0.95 0.46 0.9496 0.7968 0.9152 0.9826 0.9579 0.9675 0.6997 0.9736 0.9399 0.9494 0.9566 0.947 0.946 0.853 0.9494 0.8294 0.966 0.959 ] 2020-11-26 02:00:29 [INFO] [EVAL] The model with the best validation mIoU (0.8076) was saved at iter 40000.