2020-11-30 03:39:29 [INFO] ------------Environment Information------------- platform: Linux-3.10.0_3-0-0-34-x86_64-with-centos-7.5.1804-Core Python: 3.7.9 (default, Aug 31 2020, 12:42:55) [GCC 7.3.0] Paddle compiled with cuda: True NVCC: Cuda compilation tools, release 10.2, V10.2.89 cudnn: 7.6 GPUs used: 4 CUDA_VISIBLE_DEVICES: 0,1,2,3 GPU: ['GPU 0: Tesla V100-SXM2-16GB', 'GPU 1: Tesla V100-SXM2-16GB', 'GPU 2: Tesla V100-SXM2-16GB', 'GPU 3: Tesla V100-SXM2-16GB', 'GPU 4: Tesla V100-SXM2-16GB', 'GPU 5: Tesla V100-SXM2-16GB', 'GPU 6: Tesla V100-SXM2-16GB', 'GPU 7: Tesla V100-SXM2-16GB'] GCC: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39) PaddlePaddle: 2.0.0-rc0 OpenCV: 4.1.0 ------------------------------------------------ 2020-11-30 03:39:29 [INFO] ---------------Config Information--------------- batch_size: 4 iters: 40000 learning_rate: decay: end_lr: 0.0 power: 0.9 type: poly value: 0.01 loss: coef: - 1 types: - ignore_index: 255 type: CrossEntropyLoss model: align_corners: false aspp_out_channels: 256 aspp_ratios: - 1 - 12 - 24 - 36 backbone: multi_grid: - 1 - 2 - 4 output_stride: 8 pretrained: https://bj.bcebos.com/paddleseg/dygraph/resnet101_vd_ssld.tar.gz type: ResNet101_vd backbone_indices: - 0 - 3 num_classes: 19 pretrained: null type: DeepLabV3P optimizer: momentum: 0.9 type: sgd weight_decay: 4.0e-05 train_dataset: dataset_root: data/VOCdevkit/ mode: trainaug transforms: - max_scale_factor: 2.0 min_scale_factor: 0.5 scale_step_size: 0.25 type: ResizeStepScaling - crop_size: - 512 - 512 type: RandomPaddingCrop - type: RandomHorizontalFlip - brightness_range: 0.4 contrast_range: 0.4 saturation_range: 0.4 type: RandomDistort - type: Normalize type: PascalVOC val_dataset: dataset_root: data/VOCdevkit/ mode: val transforms: - target_size: - 512 - 512 type: Padding - type: Normalize type: PascalVOC ------------------------------------------------ 2020-11-30 03:39:33 [INFO] Loading pretrained model from https://bj.bcebos.com/paddleseg/dygraph/resnet101_vd_ssld.tar.gz 2020-11-30 03:39:35 [INFO] There are 530/530 variables loaded into ResNet_vd. 2020-11-30 03:41:16 [INFO] [TRAIN] epoch=1, iter=100/40000, loss=1.3413, lr=0.009978, batch_cost=0.9501, reader_cost=0.0116 | ETA 10:31:50 2020-11-30 03:42:47 [INFO] [TRAIN] epoch=1, iter=200/40000, loss=0.9239, lr=0.009955, batch_cost=0.9112, reader_cost=0.0004 | ETA 10:04:24 2020-11-30 03:44:18 [INFO] [TRAIN] epoch=1, iter=300/40000, loss=0.9057, lr=0.009933, batch_cost=0.9107, reader_cost=0.0004 | ETA 10:02:35 2020-11-30 03:45:49 [INFO] [TRAIN] epoch=1, iter=400/40000, loss=0.8030, lr=0.009910, batch_cost=0.9102, reader_cost=0.0004 | ETA 10:00:45 2020-11-30 03:47:20 [INFO] [TRAIN] epoch=1, iter=500/40000, loss=0.6847, lr=0.009888, batch_cost=0.9093, reader_cost=0.0007 | ETA 09:58:36 2020-11-30 03:48:51 [INFO] [TRAIN] epoch=1, iter=600/40000, loss=0.7773, lr=0.009865, batch_cost=0.9102, reader_cost=0.0004 | ETA 09:57:40 2020-11-30 03:50:22 [INFO] [TRAIN] epoch=2, iter=700/40000, loss=0.6482, lr=0.009843, batch_cost=0.9129, reader_cost=0.0071 | ETA 09:57:56 2020-11-30 03:51:52 [INFO] [TRAIN] epoch=2, iter=800/40000, loss=0.6663, lr=0.009820, batch_cost=0.8966, reader_cost=0.0002 | ETA 09:45:46 2020-11-30 03:53:22 [INFO] [TRAIN] epoch=2, iter=900/40000, loss=0.6693, lr=0.009797, batch_cost=0.9048, reader_cost=0.0003 | ETA 09:49:38 2020-11-30 03:54:53 [INFO] [TRAIN] epoch=2, iter=1000/40000, loss=0.6573, lr=0.009775, batch_cost=0.9011, reader_cost=0.0003 | ETA 09:45:44 2020-11-30 03:56:23 [INFO] [TRAIN] epoch=2, iter=1100/40000, loss=0.5979, lr=0.009752, batch_cost=0.9018, reader_cost=0.0006 | ETA 09:44:38 2020-11-30 03:57:53 [INFO] [TRAIN] epoch=2, iter=1200/40000, loss=0.6484, lr=0.009730, batch_cost=0.9056, reader_cost=0.0008 | ETA 09:45:38 2020-11-30 03:59:24 [INFO] [TRAIN] epoch=2, iter=1300/40000, loss=0.5757, lr=0.009707, batch_cost=0.9046, reader_cost=0.0009 | ETA 09:43:29 2020-11-30 04:00:54 [INFO] [TRAIN] epoch=3, iter=1400/40000, loss=0.5594, lr=0.009685, batch_cost=0.9040, reader_cost=0.0066 | ETA 09:41:35 2020-11-30 04:02:25 [INFO] [TRAIN] epoch=3, iter=1500/40000, loss=0.5629, lr=0.009662, batch_cost=0.9048, reader_cost=0.0002 | ETA 09:40:35 2020-11-30 04:03:56 [INFO] [TRAIN] epoch=3, iter=1600/40000, loss=0.4968, lr=0.009639, batch_cost=0.9087, reader_cost=0.0003 | ETA 09:41:32 2020-11-30 04:05:26 [INFO] [TRAIN] epoch=3, iter=1700/40000, loss=0.4736, lr=0.009617, batch_cost=0.9031, reader_cost=0.0004 | ETA 09:36:27 2020-11-30 04:06:57 [INFO] [TRAIN] epoch=3, iter=1800/40000, loss=0.5678, lr=0.009594, batch_cost=0.9074, reader_cost=0.0007 | ETA 09:37:43 2020-11-30 04:08:27 [INFO] [TRAIN] epoch=3, iter=1900/40000, loss=0.5266, lr=0.009572, batch_cost=0.9070, reader_cost=0.0005 | ETA 09:35:56 2020-11-30 04:09:58 [INFO] [TRAIN] epoch=4, iter=2000/40000, loss=0.5098, lr=0.009549, batch_cost=0.9084, reader_cost=0.0071 | ETA 09:35:20 2020-11-30 04:11:28 [INFO] [TRAIN] epoch=4, iter=2100/40000, loss=0.4708, lr=0.009526, batch_cost=0.8995, reader_cost=0.0001 | ETA 09:28:10 2020-11-30 04:12:58 [INFO] [TRAIN] epoch=4, iter=2200/40000, loss=0.5456, lr=0.009504, batch_cost=0.8961, reader_cost=0.0002 | ETA 09:24:31 2020-11-30 04:14:27 [INFO] [TRAIN] epoch=4, iter=2300/40000, loss=0.4789, lr=0.009481, batch_cost=0.8938, reader_cost=0.0002 | ETA 09:21:36 2020-11-30 04:15:56 [INFO] [TRAIN] epoch=4, iter=2400/40000, loss=0.4806, lr=0.009459, batch_cost=0.8926, reader_cost=0.0002 | ETA 09:19:22 2020-11-30 04:17:26 [INFO] [TRAIN] epoch=4, iter=2500/40000, loss=0.4643, lr=0.009436, batch_cost=0.8955, reader_cost=0.0002 | ETA 09:19:41 2020-11-30 04:18:56 [INFO] [TRAIN] epoch=4, iter=2600/40000, loss=0.4210, lr=0.009413, batch_cost=0.8983, reader_cost=0.0001 | ETA 09:19:57 2020-11-30 04:20:26 [INFO] [TRAIN] epoch=5, iter=2700/40000, loss=0.4627, lr=0.009391, batch_cost=0.9049, reader_cost=0.0061 | ETA 09:22:31 2020-11-30 04:21:56 [INFO] [TRAIN] epoch=5, iter=2800/40000, loss=0.3931, lr=0.009368, batch_cost=0.8995, reader_cost=0.0005 | ETA 09:17:39 2020-11-30 04:23:26 [INFO] [TRAIN] epoch=5, iter=2900/40000, loss=0.4294, lr=0.009345, batch_cost=0.9004, reader_cost=0.0003 | ETA 09:16:43 2020-11-30 04:24:56 [INFO] [TRAIN] epoch=5, iter=3000/40000, loss=0.3806, lr=0.009323, batch_cost=0.9014, reader_cost=0.0005 | ETA 09:15:50 2020-11-30 04:26:27 [INFO] [TRAIN] epoch=5, iter=3100/40000, loss=0.5247, lr=0.009300, batch_cost=0.9022, reader_cost=0.0003 | ETA 09:14:52 2020-11-30 04:27:57 [INFO] [TRAIN] epoch=5, iter=3200/40000, loss=0.4743, lr=0.009277, batch_cost=0.9066, reader_cost=0.0004 | ETA 09:16:01 2020-11-30 04:29:27 [INFO] [TRAIN] epoch=5, iter=3300/40000, loss=0.4231, lr=0.009255, batch_cost=0.8995, reader_cost=0.0003 | ETA 09:10:11 2020-11-30 04:30:57 [INFO] [TRAIN] epoch=6, iter=3400/40000, loss=0.3807, lr=0.009232, batch_cost=0.9027, reader_cost=0.0074 | ETA 09:10:37 2020-11-30 04:32:27 [INFO] [TRAIN] epoch=6, iter=3500/40000, loss=0.3810, lr=0.009209, batch_cost=0.8973, reader_cost=0.0003 | ETA 09:05:53 2020-11-30 04:33:57 [INFO] [TRAIN] epoch=6, iter=3600/40000, loss=0.4375, lr=0.009186, batch_cost=0.9032, reader_cost=0.0003 | ETA 09:07:57 2020-11-30 04:35:27 [INFO] [TRAIN] epoch=6, iter=3700/40000, loss=0.3866, lr=0.009164, batch_cost=0.8965, reader_cost=0.0003 | ETA 09:02:23 2020-11-30 04:36:57 [INFO] [TRAIN] epoch=6, iter=3800/40000, loss=0.4703, lr=0.009141, batch_cost=0.8974, reader_cost=0.0002 | ETA 09:01:26 2020-11-30 04:38:27 [INFO] [TRAIN] epoch=6, iter=3900/40000, loss=0.3680, lr=0.009118, batch_cost=0.9013, reader_cost=0.0003 | ETA 09:02:17 2020-11-30 04:39:57 [INFO] [TRAIN] epoch=7, iter=4000/40000, loss=0.4628, lr=0.009096, batch_cost=0.9039, reader_cost=0.0068 | ETA 09:02:19 2020-11-30 04:39:57 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-30 04:40:24 [INFO] [EVAL] #Images=1449 mIoU=0.6303 Acc=0.9141 Kappa=0.8124 2020-11-30 04:40:24 [INFO] [EVAL] Class IoU: [0.9157 0.8133 0.3354 0.7553 0.6077 0.565 0.8208 0.782 0.8243 0.2696 0.6229 0.5388 0.6746 0.6454 0.7346 0.7881 0.3128 0.5847 0.3554 0.7353 0.5543] 2020-11-30 04:40:24 [INFO] [EVAL] Class Acc: [0.9572 0.9324 0.3561 0.8846 0.8016 0.6979 0.9042 0.8786 0.8677 0.426 0.7137 0.7102 0.7706 0.8353 0.8527 0.8617 0.8216 0.9703 0.7404 0.7652 0.6977] 2020-11-30 04:40:31 [INFO] [EVAL] The model with the best validation mIoU (0.6303) was saved at iter 4000. 2020-11-30 04:42:01 [INFO] [TRAIN] epoch=7, iter=4100/40000, loss=0.3552, lr=0.009073, batch_cost=0.9022, reader_cost=0.0002 | ETA 08:59:49 2020-11-30 04:43:31 [INFO] [TRAIN] epoch=7, iter=4200/40000, loss=0.4371, lr=0.009050, batch_cost=0.9004, reader_cost=0.0004 | ETA 08:57:14 2020-11-30 04:45:00 [INFO] [TRAIN] epoch=7, iter=4300/40000, loss=0.3824, lr=0.009027, batch_cost=0.8958, reader_cost=0.0005 | ETA 08:53:00 2020-11-30 04:46:30 [INFO] [TRAIN] epoch=7, iter=4400/40000, loss=0.3926, lr=0.009005, batch_cost=0.9006, reader_cost=0.0006 | ETA 08:54:20 2020-11-30 04:48:01 [INFO] [TRAIN] epoch=7, iter=4500/40000, loss=0.3480, lr=0.008982, batch_cost=0.9042, reader_cost=0.0007 | ETA 08:54:57 2020-11-30 04:49:31 [INFO] [TRAIN] epoch=7, iter=4600/40000, loss=0.3814, lr=0.008959, batch_cost=0.9033, reader_cost=0.0005 | ETA 08:52:55 2020-11-30 04:51:01 [INFO] [TRAIN] epoch=8, iter=4700/40000, loss=0.3849, lr=0.008936, batch_cost=0.9012, reader_cost=0.0071 | ETA 08:50:13 2020-11-30 04:52:31 [INFO] [TRAIN] epoch=8, iter=4800/40000, loss=0.4575, lr=0.008913, batch_cost=0.8949, reader_cost=0.0002 | ETA 08:45:01 2020-11-30 04:54:01 [INFO] [TRAIN] epoch=8, iter=4900/40000, loss=0.4093, lr=0.008891, batch_cost=0.8984, reader_cost=0.0004 | ETA 08:45:33 2020-11-30 04:55:31 [INFO] [TRAIN] epoch=8, iter=5000/40000, loss=0.3709, lr=0.008868, batch_cost=0.9014, reader_cost=0.0006 | ETA 08:45:48 2020-11-30 04:57:00 [INFO] [TRAIN] epoch=8, iter=5100/40000, loss=0.4422, lr=0.008845, batch_cost=0.8921, reader_cost=0.0004 | ETA 08:38:54 2020-11-30 04:58:29 [INFO] [TRAIN] epoch=8, iter=5200/40000, loss=0.4010, lr=0.008822, batch_cost=0.8948, reader_cost=0.0002 | ETA 08:38:57 2020-11-30 05:00:00 [INFO] [TRAIN] epoch=9, iter=5300/40000, loss=0.3964, lr=0.008799, batch_cost=0.9045, reader_cost=0.0069 | ETA 08:43:07 2020-11-30 05:01:30 [INFO] [TRAIN] epoch=9, iter=5400/40000, loss=0.3680, lr=0.008777, batch_cost=0.8988, reader_cost=0.0001 | ETA 08:38:19 2020-11-30 05:03:01 [INFO] [TRAIN] epoch=9, iter=5500/40000, loss=0.3267, lr=0.008754, batch_cost=0.9093, reader_cost=0.0001 | ETA 08:42:52 2020-11-30 05:04:31 [INFO] [TRAIN] epoch=9, iter=5600/40000, loss=0.3923, lr=0.008731, batch_cost=0.9075, reader_cost=0.0003 | ETA 08:40:17 2020-11-30 05:06:02 [INFO] [TRAIN] epoch=9, iter=5700/40000, loss=0.3607, lr=0.008708, batch_cost=0.9085, reader_cost=0.0004 | ETA 08:39:20 2020-11-30 05:07:34 [INFO] [TRAIN] epoch=9, iter=5800/40000, loss=0.3499, lr=0.008685, batch_cost=0.9152, reader_cost=0.0004 | ETA 08:41:41 2020-11-30 05:09:05 [INFO] [TRAIN] epoch=9, iter=5900/40000, loss=0.2966, lr=0.008662, batch_cost=0.9122, reader_cost=0.0004 | ETA 08:38:25 2020-11-30 05:10:36 [INFO] [TRAIN] epoch=10, iter=6000/40000, loss=0.3085, lr=0.008639, batch_cost=0.9117, reader_cost=0.0066 | ETA 08:36:38 2020-11-30 05:12:06 [INFO] [TRAIN] epoch=10, iter=6100/40000, loss=0.3056, lr=0.008617, batch_cost=0.9020, reader_cost=0.0005 | ETA 08:29:36 2020-11-30 05:13:37 [INFO] [TRAIN] epoch=10, iter=6200/40000, loss=0.3362, lr=0.008594, batch_cost=0.9079, reader_cost=0.0002 | ETA 08:31:26 2020-11-30 05:15:08 [INFO] [TRAIN] epoch=10, iter=6300/40000, loss=0.3373, lr=0.008571, batch_cost=0.9125, reader_cost=0.0004 | ETA 08:32:32 2020-11-30 05:16:39 [INFO] [TRAIN] epoch=10, iter=6400/40000, loss=0.3413, lr=0.008548, batch_cost=0.9082, reader_cost=0.0003 | ETA 08:28:36 2020-11-30 05:18:11 [INFO] [TRAIN] epoch=10, iter=6500/40000, loss=0.3287, lr=0.008525, batch_cost=0.9136, reader_cost=0.0006 | ETA 08:30:05 2020-11-30 05:19:42 [INFO] [TRAIN] epoch=10, iter=6600/40000, loss=0.3197, lr=0.008502, batch_cost=0.9152, reader_cost=0.0004 | ETA 08:29:27 2020-11-30 05:21:13 [INFO] [TRAIN] epoch=11, iter=6700/40000, loss=0.3156, lr=0.008479, batch_cost=0.9082, reader_cost=0.0066 | ETA 08:24:04 2020-11-30 05:22:43 [INFO] [TRAIN] epoch=11, iter=6800/40000, loss=0.3314, lr=0.008456, batch_cost=0.9015, reader_cost=0.0005 | ETA 08:18:50 2020-11-30 05:24:13 [INFO] [TRAIN] epoch=11, iter=6900/40000, loss=0.2956, lr=0.008433, batch_cost=0.8961, reader_cost=0.0003 | ETA 08:14:21 2020-11-30 05:25:43 [INFO] [TRAIN] epoch=11, iter=7000/40000, loss=0.3515, lr=0.008410, batch_cost=0.9004, reader_cost=0.0006 | ETA 08:15:13 2020-11-30 05:27:13 [INFO] [TRAIN] epoch=11, iter=7100/40000, loss=0.3760, lr=0.008388, batch_cost=0.9012, reader_cost=0.0005 | ETA 08:14:08 2020-11-30 05:28:43 [INFO] [TRAIN] epoch=11, iter=7200/40000, loss=0.3171, lr=0.008365, batch_cost=0.8988, reader_cost=0.0003 | ETA 08:11:21 2020-11-30 05:30:13 [INFO] [TRAIN] epoch=12, iter=7300/40000, loss=0.2926, lr=0.008342, batch_cost=0.9049, reader_cost=0.0066 | ETA 08:13:08 2020-11-30 05:31:43 [INFO] [TRAIN] epoch=12, iter=7400/40000, loss=0.3394, lr=0.008319, batch_cost=0.9001, reader_cost=0.0004 | ETA 08:09:03 2020-11-30 05:33:13 [INFO] [TRAIN] epoch=12, iter=7500/40000, loss=0.2869, lr=0.008296, batch_cost=0.8991, reader_cost=0.0002 | ETA 08:07:01 2020-11-30 05:34:43 [INFO] [TRAIN] epoch=12, iter=7600/40000, loss=0.2862, lr=0.008273, batch_cost=0.8956, reader_cost=0.0003 | ETA 08:03:36 2020-11-30 05:36:14 [INFO] [TRAIN] epoch=12, iter=7700/40000, loss=0.2833, lr=0.008250, batch_cost=0.9089, reader_cost=0.0005 | ETA 08:09:18 2020-11-30 05:37:44 [INFO] [TRAIN] epoch=12, iter=7800/40000, loss=0.2914, lr=0.008227, batch_cost=0.9011, reader_cost=0.0005 | ETA 08:03:35 2020-11-30 05:39:14 [INFO] [TRAIN] epoch=12, iter=7900/40000, loss=0.3092, lr=0.008204, batch_cost=0.8980, reader_cost=0.0005 | ETA 08:00:26 2020-11-30 05:40:44 [INFO] [TRAIN] epoch=13, iter=8000/40000, loss=0.2999, lr=0.008181, batch_cost=0.9030, reader_cost=0.0063 | ETA 08:01:34 2020-11-30 05:40:44 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-30 05:41:11 [INFO] [EVAL] #Images=1449 mIoU=0.6709 Acc=0.9211 Kappa=0.8262 2020-11-30 05:41:11 [INFO] [EVAL] Class IoU: [0.9196 0.8248 0.371 0.8247 0.6456 0.5105 0.7629 0.8085 0.741 0.2953 0.7423 0.54 0.6962 0.8517 0.7713 0.8121 0.4825 0.7989 0.3819 0.7602 0.5479] 2020-11-30 05:41:11 [INFO] [EVAL] Class Acc: [0.9556 0.9311 0.3914 0.8676 0.7115 0.9104 0.7793 0.9354 0.9697 0.4311 0.8831 0.8528 0.7266 0.9269 0.7953 0.9273 0.7074 0.9231 0.8184 0.8354 0.6563] 2020-11-30 05:41:17 [INFO] [EVAL] The model with the best validation mIoU (0.6709) was saved at iter 8000. 2020-11-30 05:42:47 [INFO] [TRAIN] epoch=13, iter=8100/40000, loss=0.2267, lr=0.008158, batch_cost=0.9028, reader_cost=0.0002 | ETA 07:59:59 2020-11-30 05:44:17 [INFO] [TRAIN] epoch=13, iter=8200/40000, loss=0.3120, lr=0.008135, batch_cost=0.8993, reader_cost=0.0002 | ETA 07:56:38 2020-11-30 05:45:47 [INFO] [TRAIN] epoch=13, iter=8300/40000, loss=0.2616, lr=0.008112, batch_cost=0.9007, reader_cost=0.0002 | ETA 07:55:52 2020-11-30 05:47:18 [INFO] [TRAIN] epoch=13, iter=8400/40000, loss=0.3034, lr=0.008089, batch_cost=0.9026, reader_cost=0.0005 | ETA 07:55:21 2020-11-30 05:48:48 [INFO] [TRAIN] epoch=13, iter=8500/40000, loss=0.2636, lr=0.008066, batch_cost=0.9005, reader_cost=0.0002 | ETA 07:52:46 2020-11-30 05:50:18 [INFO] [TRAIN] epoch=14, iter=8600/40000, loss=0.2527, lr=0.008043, batch_cost=0.9056, reader_cost=0.0064 | ETA 07:53:56 2020-11-30 05:51:49 [INFO] [TRAIN] epoch=14, iter=8700/40000, loss=0.2401, lr=0.008020, batch_cost=0.9027, reader_cost=0.0001 | ETA 07:50:54 2020-11-30 05:53:18 [INFO] [TRAIN] epoch=14, iter=8800/40000, loss=0.2581, lr=0.007996, batch_cost=0.8973, reader_cost=0.0001 | ETA 07:46:35 2020-11-30 05:54:48 [INFO] [TRAIN] epoch=14, iter=8900/40000, loss=0.2256, lr=0.007973, batch_cost=0.8960, reader_cost=0.0002 | ETA 07:44:25 2020-11-30 05:56:18 [INFO] [TRAIN] epoch=14, iter=9000/40000, loss=0.2468, lr=0.007950, batch_cost=0.8963, reader_cost=0.0002 | ETA 07:43:05 2020-11-30 05:57:47 [INFO] [TRAIN] epoch=14, iter=9100/40000, loss=0.2870, lr=0.007927, batch_cost=0.8973, reader_cost=0.0002 | ETA 07:42:07 2020-11-30 05:59:17 [INFO] [TRAIN] epoch=14, iter=9200/40000, loss=0.2678, lr=0.007904, batch_cost=0.9013, reader_cost=0.0002 | ETA 07:42:38 2020-11-30 06:00:47 [INFO] [TRAIN] epoch=15, iter=9300/40000, loss=0.3031, lr=0.007881, batch_cost=0.9004, reader_cost=0.0066 | ETA 07:40:42 2020-11-30 06:02:17 [INFO] [TRAIN] epoch=15, iter=9400/40000, loss=0.2442, lr=0.007858, batch_cost=0.8992, reader_cost=0.0002 | ETA 07:38:35 2020-11-30 06:03:48 [INFO] [TRAIN] epoch=15, iter=9500/40000, loss=0.2562, lr=0.007835, batch_cost=0.9026, reader_cost=0.0004 | ETA 07:38:49 2020-11-30 06:05:18 [INFO] [TRAIN] epoch=15, iter=9600/40000, loss=0.2613, lr=0.007812, batch_cost=0.9011, reader_cost=0.0003 | ETA 07:36:34 2020-11-30 06:06:48 [INFO] [TRAIN] epoch=15, iter=9700/40000, loss=0.2562, lr=0.007789, batch_cost=0.9037, reader_cost=0.0002 | ETA 07:36:23 2020-11-30 06:08:19 [INFO] [TRAIN] epoch=15, iter=9800/40000, loss=0.2935, lr=0.007765, batch_cost=0.9045, reader_cost=0.0002 | ETA 07:35:16 2020-11-30 06:09:49 [INFO] [TRAIN] epoch=15, iter=9900/40000, loss=0.2927, lr=0.007742, batch_cost=0.9007, reader_cost=0.0002 | ETA 07:31:51 2020-11-30 06:11:19 [INFO] [TRAIN] epoch=16, iter=10000/40000, loss=0.3254, lr=0.007719, batch_cost=0.9020, reader_cost=0.0062 | ETA 07:30:59 2020-11-30 06:12:49 [INFO] [TRAIN] epoch=16, iter=10100/40000, loss=0.2603, lr=0.007696, batch_cost=0.9028, reader_cost=0.0002 | ETA 07:29:53 2020-11-30 06:14:19 [INFO] [TRAIN] epoch=16, iter=10200/40000, loss=0.3070, lr=0.007673, batch_cost=0.9029, reader_cost=0.0002 | ETA 07:28:26 2020-11-30 06:15:50 [INFO] [TRAIN] epoch=16, iter=10300/40000, loss=0.2595, lr=0.007650, batch_cost=0.9074, reader_cost=0.0002 | ETA 07:29:09 2020-11-30 06:17:20 [INFO] [TRAIN] epoch=16, iter=10400/40000, loss=0.2444, lr=0.007626, batch_cost=0.9007, reader_cost=0.0004 | ETA 07:24:20 2020-11-30 06:18:50 [INFO] [TRAIN] epoch=16, iter=10500/40000, loss=0.2432, lr=0.007603, batch_cost=0.9011, reader_cost=0.0003 | ETA 07:23:02 2020-11-30 06:20:21 [INFO] [TRAIN] epoch=17, iter=10600/40000, loss=0.2776, lr=0.007580, batch_cost=0.9092, reader_cost=0.0068 | ETA 07:25:31 2020-11-30 06:21:51 [INFO] [TRAIN] epoch=17, iter=10700/40000, loss=0.2220, lr=0.007557, batch_cost=0.9019, reader_cost=0.0001 | ETA 07:20:25 2020-11-30 06:23:22 [INFO] [TRAIN] epoch=17, iter=10800/40000, loss=0.2407, lr=0.007534, batch_cost=0.9035, reader_cost=0.0001 | ETA 07:19:41 2020-11-30 06:24:52 [INFO] [TRAIN] epoch=17, iter=10900/40000, loss=0.2047, lr=0.007510, batch_cost=0.9006, reader_cost=0.0002 | ETA 07:16:48 2020-11-30 06:26:22 [INFO] [TRAIN] epoch=17, iter=11000/40000, loss=0.2203, lr=0.007487, batch_cost=0.8995, reader_cost=0.0002 | ETA 07:14:46 2020-11-30 06:27:52 [INFO] [TRAIN] epoch=17, iter=11100/40000, loss=0.2610, lr=0.007464, batch_cost=0.9012, reader_cost=0.0002 | ETA 07:14:05 2020-11-30 06:29:22 [INFO] [TRAIN] epoch=17, iter=11200/40000, loss=0.2298, lr=0.007441, batch_cost=0.8973, reader_cost=0.0002 | ETA 07:10:41 2020-11-30 06:30:52 [INFO] [TRAIN] epoch=18, iter=11300/40000, loss=0.2107, lr=0.007417, batch_cost=0.9019, reader_cost=0.0069 | ETA 07:11:24 2020-11-30 06:32:23 [INFO] [TRAIN] epoch=18, iter=11400/40000, loss=0.2058, lr=0.007394, batch_cost=0.9125, reader_cost=0.0002 | ETA 07:14:56 2020-11-30 06:33:54 [INFO] [TRAIN] epoch=18, iter=11500/40000, loss=0.2466, lr=0.007371, batch_cost=0.9119, reader_cost=0.0003 | ETA 07:13:08 2020-11-30 06:35:25 [INFO] [TRAIN] epoch=18, iter=11600/40000, loss=0.2598, lr=0.007348, batch_cost=0.9091, reader_cost=0.0002 | ETA 07:10:19 2020-11-30 06:36:56 [INFO] [TRAIN] epoch=18, iter=11700/40000, loss=0.2285, lr=0.007324, batch_cost=0.9110, reader_cost=0.0002 | ETA 07:09:40 2020-11-30 06:38:28 [INFO] [TRAIN] epoch=18, iter=11800/40000, loss=0.2465, lr=0.007301, batch_cost=0.9196, reader_cost=0.0004 | ETA 07:12:11 2020-11-30 06:40:00 [INFO] [TRAIN] epoch=19, iter=11900/40000, loss=0.2534, lr=0.007278, batch_cost=0.9203, reader_cost=0.0063 | ETA 07:11:00 2020-11-30 06:41:31 [INFO] [TRAIN] epoch=19, iter=12000/40000, loss=0.2338, lr=0.007254, batch_cost=0.9030, reader_cost=0.0001 | ETA 07:01:25 2020-11-30 06:41:31 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-30 06:41:57 [INFO] [EVAL] #Images=1449 mIoU=0.7410 Acc=0.9384 Kappa=0.8644 2020-11-30 06:41:57 [INFO] [EVAL] Class IoU: [0.9331 0.8863 0.4291 0.8564 0.6464 0.719 0.9167 0.8584 0.8751 0.3884 0.8502 0.5767 0.7927 0.8553 0.826 0.8247 0.5747 0.8463 0.4198 0.824 0.6616] 2020-11-30 06:41:57 [INFO] [EVAL] Class Acc: [0.9634 0.9459 0.453 0.888 0.7649 0.8591 0.9848 0.9214 0.9575 0.6086 0.9067 0.9381 0.8455 0.9183 0.8581 0.9236 0.832 0.9297 0.5889 0.9003 0.7718] 2020-11-30 06:42:03 [INFO] [EVAL] The model with the best validation mIoU (0.7410) was saved at iter 12000. 2020-11-30 06:43:34 [INFO] [TRAIN] epoch=19, iter=12100/40000, loss=0.1849, lr=0.007231, batch_cost=0.9074, reader_cost=0.0008 | ETA 07:01:55 2020-11-30 06:45:05 [INFO] [TRAIN] epoch=19, iter=12200/40000, loss=0.2407, lr=0.007208, batch_cost=0.9087, reader_cost=0.0004 | ETA 07:01:01 2020-11-30 06:46:36 [INFO] [TRAIN] epoch=19, iter=12300/40000, loss=0.2352, lr=0.007184, batch_cost=0.9147, reader_cost=0.0006 | ETA 07:02:18 2020-11-30 06:48:08 [INFO] [TRAIN] epoch=19, iter=12400/40000, loss=0.2026, lr=0.007161, batch_cost=0.9140, reader_cost=0.0008 | ETA 07:00:26 2020-11-30 06:49:39 [INFO] [TRAIN] epoch=19, iter=12500/40000, loss=0.2021, lr=0.007138, batch_cost=0.9129, reader_cost=0.0007 | ETA 06:58:25 2020-11-30 06:51:10 [INFO] [TRAIN] epoch=20, iter=12600/40000, loss=0.1811, lr=0.007114, batch_cost=0.9144, reader_cost=0.0073 | ETA 06:57:34 2020-11-30 06:52:41 [INFO] [TRAIN] epoch=20, iter=12700/40000, loss=0.2123, lr=0.007091, batch_cost=0.9028, reader_cost=0.0002 | ETA 06:50:45 2020-11-30 06:54:11 [INFO] [TRAIN] epoch=20, iter=12800/40000, loss=0.2907, lr=0.007068, batch_cost=0.9072, reader_cost=0.0002 | ETA 06:51:15 2020-11-30 06:55:42 [INFO] [TRAIN] epoch=20, iter=12900/40000, loss=0.2420, lr=0.007044, batch_cost=0.9036, reader_cost=0.0002 | ETA 06:48:06 2020-11-30 06:57:12 [INFO] [TRAIN] epoch=20, iter=13000/40000, loss=0.2415, lr=0.007021, batch_cost=0.9018, reader_cost=0.0002 | ETA 06:45:47 2020-11-30 06:58:43 [INFO] [TRAIN] epoch=20, iter=13100/40000, loss=0.2509, lr=0.006997, batch_cost=0.9079, reader_cost=0.0002 | ETA 06:47:03 2020-11-30 07:00:14 [INFO] [TRAIN] epoch=20, iter=13200/40000, loss=0.2046, lr=0.006974, batch_cost=0.9119, reader_cost=0.0002 | ETA 06:47:18 2020-11-30 07:01:45 [INFO] [TRAIN] epoch=21, iter=13300/40000, loss=0.2234, lr=0.006951, batch_cost=0.9100, reader_cost=0.0064 | ETA 06:44:56 2020-11-30 07:03:15 [INFO] [TRAIN] epoch=21, iter=13400/40000, loss=0.1988, lr=0.006927, batch_cost=0.9004, reader_cost=0.0002 | ETA 06:39:11 2020-11-30 07:04:46 [INFO] [TRAIN] epoch=21, iter=13500/40000, loss=0.2104, lr=0.006904, batch_cost=0.9068, reader_cost=0.0003 | ETA 06:40:29 2020-11-30 07:06:17 [INFO] [TRAIN] epoch=21, iter=13600/40000, loss=0.1752, lr=0.006880, batch_cost=0.9110, reader_cost=0.0005 | ETA 06:40:51 2020-11-30 07:07:47 [INFO] [TRAIN] epoch=21, iter=13700/40000, loss=0.2278, lr=0.006857, batch_cost=0.9064, reader_cost=0.0003 | ETA 06:37:17 2020-11-30 07:09:18 [INFO] [TRAIN] epoch=21, iter=13800/40000, loss=0.2125, lr=0.006833, batch_cost=0.9103, reader_cost=0.0002 | ETA 06:37:30 2020-11-30 07:10:49 [INFO] [TRAIN] epoch=22, iter=13900/40000, loss=0.1961, lr=0.006810, batch_cost=0.9102, reader_cost=0.0071 | ETA 06:35:56 2020-11-30 07:12:19 [INFO] [TRAIN] epoch=22, iter=14000/40000, loss=0.1891, lr=0.006786, batch_cost=0.8994, reader_cost=0.0003 | ETA 06:29:43 2020-11-30 07:13:49 [INFO] [TRAIN] epoch=22, iter=14100/40000, loss=0.2903, lr=0.006763, batch_cost=0.9009, reader_cost=0.0007 | ETA 06:28:53 2020-11-30 07:15:20 [INFO] [TRAIN] epoch=22, iter=14200/40000, loss=0.2009, lr=0.006739, batch_cost=0.9073, reader_cost=0.0007 | ETA 06:30:08 2020-11-30 07:16:50 [INFO] [TRAIN] epoch=22, iter=14300/40000, loss=0.1658, lr=0.006716, batch_cost=0.8997, reader_cost=0.0006 | ETA 06:25:22 2020-11-30 07:18:21 [INFO] [TRAIN] epoch=22, iter=14400/40000, loss=0.1788, lr=0.006692, batch_cost=0.9062, reader_cost=0.0008 | ETA 06:26:39 2020-11-30 07:19:50 [INFO] [TRAIN] epoch=22, iter=14500/40000, loss=0.2019, lr=0.006669, batch_cost=0.8980, reader_cost=0.0004 | ETA 06:21:39 2020-11-30 07:21:21 [INFO] [TRAIN] epoch=23, iter=14600/40000, loss=0.1923, lr=0.006645, batch_cost=0.9091, reader_cost=0.0066 | ETA 06:24:51 2020-11-30 07:22:52 [INFO] [TRAIN] epoch=23, iter=14700/40000, loss=0.1955, lr=0.006622, batch_cost=0.9014, reader_cost=0.0005 | ETA 06:20:05 2020-11-30 07:24:22 [INFO] [TRAIN] epoch=23, iter=14800/40000, loss=0.2273, lr=0.006598, batch_cost=0.9069, reader_cost=0.0007 | ETA 06:20:53 2020-11-30 07:25:53 [INFO] [TRAIN] epoch=23, iter=14900/40000, loss=0.1616, lr=0.006575, batch_cost=0.9083, reader_cost=0.0007 | ETA 06:19:57 2020-11-30 07:27:24 [INFO] [TRAIN] epoch=23, iter=15000/40000, loss=0.1928, lr=0.006551, batch_cost=0.9078, reader_cost=0.0008 | ETA 06:18:14 2020-11-30 07:28:56 [INFO] [TRAIN] epoch=23, iter=15100/40000, loss=0.2007, lr=0.006527, batch_cost=0.9188, reader_cost=0.0014 | ETA 06:21:17 2020-11-30 07:30:27 [INFO] [TRAIN] epoch=23, iter=15200/40000, loss=0.2112, lr=0.006504, batch_cost=0.9152, reader_cost=0.0014 | ETA 06:18:17 2020-11-30 07:31:58 [INFO] [TRAIN] epoch=24, iter=15300/40000, loss=0.1813, lr=0.006480, batch_cost=0.9102, reader_cost=0.0066 | ETA 06:14:42 2020-11-30 07:33:28 [INFO] [TRAIN] epoch=24, iter=15400/40000, loss=0.1609, lr=0.006457, batch_cost=0.9003, reader_cost=0.0005 | ETA 06:09:08 2020-11-30 07:34:58 [INFO] [TRAIN] epoch=24, iter=15500/40000, loss=0.2195, lr=0.006433, batch_cost=0.9005, reader_cost=0.0005 | ETA 06:07:41 2020-11-30 07:36:28 [INFO] [TRAIN] epoch=24, iter=15600/40000, loss=0.1847, lr=0.006409, batch_cost=0.8987, reader_cost=0.0003 | ETA 06:05:27 2020-11-30 07:37:58 [INFO] [TRAIN] epoch=24, iter=15700/40000, loss=0.1763, lr=0.006386, batch_cost=0.9008, reader_cost=0.0004 | ETA 06:04:49 2020-11-30 07:39:29 [INFO] [TRAIN] epoch=24, iter=15800/40000, loss=0.1801, lr=0.006362, batch_cost=0.9033, reader_cost=0.0005 | ETA 06:04:18 2020-11-30 07:40:59 [INFO] [TRAIN] epoch=25, iter=15900/40000, loss=0.1652, lr=0.006338, batch_cost=0.9050, reader_cost=0.0070 | ETA 06:03:29 2020-11-30 07:42:29 [INFO] [TRAIN] epoch=25, iter=16000/40000, loss=0.2095, lr=0.006315, batch_cost=0.9023, reader_cost=0.0002 | ETA 06:00:54 2020-11-30 07:42:29 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-30 07:42:56 [INFO] [EVAL] #Images=1449 mIoU=0.7456 Acc=0.9409 Kappa=0.8700 2020-11-30 07:42:56 [INFO] [EVAL] Class IoU: [0.9353 0.8749 0.4234 0.8509 0.665 0.7677 0.9276 0.8597 0.9055 0.4112 0.8436 0.486 0.8337 0.8507 0.8778 0.8223 0.5711 0.8478 0.3805 0.8881 0.6349] 2020-11-30 07:42:56 [INFO] [EVAL] Class Acc: [0.9646 0.9111 0.4443 0.9124 0.7525 0.8516 0.9541 0.9389 0.9461 0.5494 0.9066 0.8943 0.9183 0.8851 0.9216 0.8724 0.7438 0.9714 0.8275 0.9353 0.8447] 2020-11-30 07:43:02 [INFO] [EVAL] The model with the best validation mIoU (0.7456) was saved at iter 16000. 2020-11-30 07:44:33 [INFO] [TRAIN] epoch=25, iter=16100/40000, loss=0.1972, lr=0.006291, batch_cost=0.9107, reader_cost=0.0002 | ETA 06:02:46 2020-11-30 07:46:04 [INFO] [TRAIN] epoch=25, iter=16200/40000, loss=0.1789, lr=0.006267, batch_cost=0.9057, reader_cost=0.0002 | ETA 05:59:16 2020-11-30 07:47:34 [INFO] [TRAIN] epoch=25, iter=16300/40000, loss=0.1735, lr=0.006244, batch_cost=0.9051, reader_cost=0.0002 | ETA 05:57:30 2020-11-30 07:49:05 [INFO] [TRAIN] epoch=25, iter=16400/40000, loss=0.2231, lr=0.006220, batch_cost=0.9056, reader_cost=0.0002 | ETA 05:56:12 2020-11-30 07:50:36 [INFO] [TRAIN] epoch=25, iter=16500/40000, loss=0.1863, lr=0.006196, batch_cost=0.9086, reader_cost=0.0002 | ETA 05:55:53 2020-11-30 07:52:07 [INFO] [TRAIN] epoch=26, iter=16600/40000, loss=0.2039, lr=0.006172, batch_cost=0.9128, reader_cost=0.0070 | ETA 05:55:58 2020-11-30 07:53:36 [INFO] [TRAIN] epoch=26, iter=16700/40000, loss=0.1941, lr=0.006149, batch_cost=0.8954, reader_cost=0.0002 | ETA 05:47:41 2020-11-30 07:55:07 [INFO] [TRAIN] epoch=26, iter=16800/40000, loss=0.1701, lr=0.006125, batch_cost=0.9007, reader_cost=0.0002 | ETA 05:48:16 2020-11-30 07:56:37 [INFO] [TRAIN] epoch=26, iter=16900/40000, loss=0.2161, lr=0.006101, batch_cost=0.9011, reader_cost=0.0002 | ETA 05:46:56 2020-11-30 07:58:06 [INFO] [TRAIN] epoch=26, iter=17000/40000, loss=0.1799, lr=0.006077, batch_cost=0.8954, reader_cost=0.0002 | ETA 05:43:15 2020-11-30 07:59:36 [INFO] [TRAIN] epoch=26, iter=17100/40000, loss=0.1860, lr=0.006054, batch_cost=0.8939, reader_cost=0.0002 | ETA 05:41:10 2020-11-30 08:01:06 [INFO] [TRAIN] epoch=27, iter=17200/40000, loss=0.1803, lr=0.006030, batch_cost=0.9023, reader_cost=0.0064 | ETA 05:42:52 2020-11-30 08:02:35 [INFO] [TRAIN] epoch=27, iter=17300/40000, loss=0.1653, lr=0.006006, batch_cost=0.8958, reader_cost=0.0002 | ETA 05:38:54 2020-11-30 08:04:06 [INFO] [TRAIN] epoch=27, iter=17400/40000, loss=0.1791, lr=0.005982, batch_cost=0.9022, reader_cost=0.0004 | ETA 05:39:48 2020-11-30 08:05:35 [INFO] [TRAIN] epoch=27, iter=17500/40000, loss=0.1706, lr=0.005958, batch_cost=0.8979, reader_cost=0.0002 | ETA 05:36:43 2020-11-30 08:07:06 [INFO] [TRAIN] epoch=27, iter=17600/40000, loss=0.1623, lr=0.005935, batch_cost=0.9044, reader_cost=0.0004 | ETA 05:37:37 2020-11-30 08:08:36 [INFO] [TRAIN] epoch=27, iter=17700/40000, loss=0.2273, lr=0.005911, batch_cost=0.8985, reader_cost=0.0002 | ETA 05:33:57 2020-11-30 08:10:05 [INFO] [TRAIN] epoch=27, iter=17800/40000, loss=0.1876, lr=0.005887, batch_cost=0.8970, reader_cost=0.0002 | ETA 05:31:52 2020-11-30 08:11:36 [INFO] [TRAIN] epoch=28, iter=17900/40000, loss=0.1847, lr=0.005863, batch_cost=0.9041, reader_cost=0.0069 | ETA 05:33:00 2020-11-30 08:13:07 [INFO] [TRAIN] epoch=28, iter=18000/40000, loss=0.1595, lr=0.005839, batch_cost=0.9072, reader_cost=0.0006 | ETA 05:32:39 2020-11-30 08:14:38 [INFO] [TRAIN] epoch=28, iter=18100/40000, loss=0.1486, lr=0.005815, batch_cost=0.9102, reader_cost=0.0009 | ETA 05:32:12 2020-11-30 08:16:08 [INFO] [TRAIN] epoch=28, iter=18200/40000, loss=0.1910, lr=0.005791, batch_cost=0.9076, reader_cost=0.0011 | ETA 05:29:44 2020-11-30 08:17:40 [INFO] [TRAIN] epoch=28, iter=18300/40000, loss=0.1742, lr=0.005767, batch_cost=0.9123, reader_cost=0.0012 | ETA 05:29:56 2020-11-30 08:19:11 [INFO] [TRAIN] epoch=28, iter=18400/40000, loss=0.1378, lr=0.005743, batch_cost=0.9139, reader_cost=0.0010 | ETA 05:29:00 2020-11-30 08:20:42 [INFO] [TRAIN] epoch=28, iter=18500/40000, loss=0.1730, lr=0.005720, batch_cost=0.9091, reader_cost=0.0008 | ETA 05:25:45 2020-11-30 08:22:12 [INFO] [TRAIN] epoch=29, iter=18600/40000, loss=0.1458, lr=0.005696, batch_cost=0.9020, reader_cost=0.0066 | ETA 05:21:43 2020-11-30 08:23:42 [INFO] [TRAIN] epoch=29, iter=18700/40000, loss=0.1617, lr=0.005672, batch_cost=0.8983, reader_cost=0.0004 | ETA 05:18:53 2020-11-30 08:25:12 [INFO] [TRAIN] epoch=29, iter=18800/40000, loss=0.1488, lr=0.005648, batch_cost=0.9004, reader_cost=0.0009 | ETA 05:18:08 2020-11-30 08:26:42 [INFO] [TRAIN] epoch=29, iter=18900/40000, loss=0.1698, lr=0.005624, batch_cost=0.9009, reader_cost=0.0006 | ETA 05:16:49 2020-11-30 08:28:12 [INFO] [TRAIN] epoch=29, iter=19000/40000, loss=0.1467, lr=0.005600, batch_cost=0.8990, reader_cost=0.0004 | ETA 05:14:38 2020-11-30 08:29:42 [INFO] [TRAIN] epoch=29, iter=19100/40000, loss=0.1604, lr=0.005576, batch_cost=0.8996, reader_cost=0.0007 | ETA 05:13:20 2020-11-30 08:31:12 [INFO] [TRAIN] epoch=30, iter=19200/40000, loss=0.1872, lr=0.005552, batch_cost=0.9041, reader_cost=0.0063 | ETA 05:13:25 2020-11-30 08:32:42 [INFO] [TRAIN] epoch=30, iter=19300/40000, loss=0.1470, lr=0.005528, batch_cost=0.8984, reader_cost=0.0005 | ETA 05:09:56 2020-11-30 08:34:12 [INFO] [TRAIN] epoch=30, iter=19400/40000, loss=0.1656, lr=0.005504, batch_cost=0.9029, reader_cost=0.0006 | ETA 05:09:59 2020-11-30 08:35:43 [INFO] [TRAIN] epoch=30, iter=19500/40000, loss=0.1446, lr=0.005480, batch_cost=0.9046, reader_cost=0.0004 | ETA 05:09:03 2020-11-30 08:37:13 [INFO] [TRAIN] epoch=30, iter=19600/40000, loss=0.1661, lr=0.005455, batch_cost=0.9011, reader_cost=0.0003 | ETA 05:06:23 2020-11-30 08:38:44 [INFO] [TRAIN] epoch=30, iter=19700/40000, loss=0.1532, lr=0.005431, batch_cost=0.9072, reader_cost=0.0006 | ETA 05:06:55 2020-11-30 08:40:14 [INFO] [TRAIN] epoch=30, iter=19800/40000, loss=0.1632, lr=0.005407, batch_cost=0.9053, reader_cost=0.0008 | ETA 05:04:47 2020-11-30 08:41:45 [INFO] [TRAIN] epoch=31, iter=19900/40000, loss=0.1439, lr=0.005383, batch_cost=0.9076, reader_cost=0.0070 | ETA 05:04:03 2020-11-30 08:43:15 [INFO] [TRAIN] epoch=31, iter=20000/40000, loss=0.1239, lr=0.005359, batch_cost=0.8973, reader_cost=0.0004 | ETA 04:59:05 2020-11-30 08:43:15 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-30 08:43:41 [INFO] [EVAL] #Images=1449 mIoU=0.7771 Acc=0.9484 Kappa=0.8865 2020-11-30 08:43:41 [INFO] [EVAL] Class IoU: [0.9418 0.8578 0.4223 0.8849 0.6877 0.7943 0.9277 0.8645 0.899 0.4422 0.9077 0.5975 0.8393 0.9048 0.8873 0.8638 0.5998 0.9055 0.4903 0.8866 0.7143] 2020-11-30 08:43:41 [INFO] [EVAL] Class Acc: [0.9685 0.8909 0.4367 0.9373 0.8122 0.8661 0.9815 0.9291 0.9204 0.6411 0.9669 0.8717 0.9086 0.9473 0.9454 0.9254 0.7996 0.9467 0.7019 0.9469 0.9453] 2020-11-30 08:43:48 [INFO] [EVAL] The model with the best validation mIoU (0.7771) was saved at iter 20000. 2020-11-30 08:45:19 [INFO] [TRAIN] epoch=31, iter=20100/40000, loss=0.1719, lr=0.005335, batch_cost=0.9064, reader_cost=0.0007 | ETA 05:00:37 2020-11-30 08:46:48 [INFO] [TRAIN] epoch=31, iter=20200/40000, loss=0.1542, lr=0.005311, batch_cost=0.8979, reader_cost=0.0005 | ETA 04:56:19 2020-11-30 08:48:18 [INFO] [TRAIN] epoch=31, iter=20300/40000, loss=0.1371, lr=0.005287, batch_cost=0.8946, reader_cost=0.0002 | ETA 04:53:43 2020-11-30 08:49:47 [INFO] [TRAIN] epoch=31, iter=20400/40000, loss=0.1352, lr=0.005263, batch_cost=0.8961, reader_cost=0.0004 | ETA 04:52:44 2020-11-30 08:51:18 [INFO] [TRAIN] epoch=32, iter=20500/40000, loss=0.1463, lr=0.005238, batch_cost=0.9060, reader_cost=0.0074 | ETA 04:54:26 2020-11-30 08:52:47 [INFO] [TRAIN] epoch=32, iter=20600/40000, loss=0.1431, lr=0.005214, batch_cost=0.8939, reader_cost=0.0003 | ETA 04:49:01 2020-11-30 08:54:17 [INFO] [TRAIN] epoch=32, iter=20700/40000, loss=0.1479, lr=0.005190, batch_cost=0.8981, reader_cost=0.0006 | ETA 04:48:53 2020-11-30 08:55:47 [INFO] [TRAIN] epoch=32, iter=20800/40000, loss=0.1621, lr=0.005166, batch_cost=0.8963, reader_cost=0.0006 | ETA 04:46:48 2020-11-30 08:57:17 [INFO] [TRAIN] epoch=32, iter=20900/40000, loss=0.1280, lr=0.005142, batch_cost=0.9042, reader_cost=0.0006 | ETA 04:47:49 2020-11-30 08:58:48 [INFO] [TRAIN] epoch=32, iter=21000/40000, loss=0.1546, lr=0.005117, batch_cost=0.9052, reader_cost=0.0006 | ETA 04:46:38 2020-11-30 09:00:18 [INFO] [TRAIN] epoch=32, iter=21100/40000, loss=0.1288, lr=0.005093, batch_cost=0.9047, reader_cost=0.0005 | ETA 04:44:58 2020-11-30 09:01:50 [INFO] [TRAIN] epoch=33, iter=21200/40000, loss=0.1438, lr=0.005069, batch_cost=0.9126, reader_cost=0.0071 | ETA 04:45:56 2020-11-30 09:03:20 [INFO] [TRAIN] epoch=33, iter=21300/40000, loss=0.1385, lr=0.005045, batch_cost=0.9073, reader_cost=0.0002 | ETA 04:42:46 2020-11-30 09:04:51 [INFO] [TRAIN] epoch=33, iter=21400/40000, loss=0.1341, lr=0.005020, batch_cost=0.9035, reader_cost=0.0002 | ETA 04:40:05 2020-11-30 09:06:22 [INFO] [TRAIN] epoch=33, iter=21500/40000, loss=0.1347, lr=0.004996, batch_cost=0.9092, reader_cost=0.0006 | ETA 04:40:19 2020-11-30 09:07:53 [INFO] [TRAIN] epoch=33, iter=21600/40000, loss=0.1249, lr=0.004972, batch_cost=0.9113, reader_cost=0.0002 | ETA 04:39:28 2020-11-30 09:09:23 [INFO] [TRAIN] epoch=33, iter=21700/40000, loss=0.1275, lr=0.004947, batch_cost=0.9064, reader_cost=0.0002 | ETA 04:36:27 2020-11-30 09:10:55 [INFO] [TRAIN] epoch=33, iter=21800/40000, loss=0.1380, lr=0.004923, batch_cost=0.9125, reader_cost=0.0002 | ETA 04:36:47 2020-11-30 09:12:26 [INFO] [TRAIN] epoch=34, iter=21900/40000, loss=0.1298, lr=0.004899, batch_cost=0.9135, reader_cost=0.0070 | ETA 04:35:35 2020-11-30 09:13:56 [INFO] [TRAIN] epoch=34, iter=22000/40000, loss=0.1292, lr=0.004874, batch_cost=0.9046, reader_cost=0.0002 | ETA 04:31:22 2020-11-30 09:15:28 [INFO] [TRAIN] epoch=34, iter=22100/40000, loss=0.1213, lr=0.004850, batch_cost=0.9120, reader_cost=0.0003 | ETA 04:32:05 2020-11-30 09:16:59 [INFO] [TRAIN] epoch=34, iter=22200/40000, loss=0.1300, lr=0.004826, batch_cost=0.9115, reader_cost=0.0002 | ETA 04:30:25 2020-11-30 09:18:29 [INFO] [TRAIN] epoch=34, iter=22300/40000, loss=0.1356, lr=0.004801, batch_cost=0.9044, reader_cost=0.0002 | ETA 04:26:47 2020-11-30 09:19:59 [INFO] [TRAIN] epoch=34, iter=22400/40000, loss=0.1362, lr=0.004777, batch_cost=0.9007, reader_cost=0.0002 | ETA 04:24:12 2020-11-30 09:21:30 [INFO] [TRAIN] epoch=35, iter=22500/40000, loss=0.1263, lr=0.004752, batch_cost=0.9088, reader_cost=0.0067 | ETA 04:25:03 2020-11-30 09:23:03 [INFO] [TRAIN] epoch=35, iter=22600/40000, loss=0.1267, lr=0.004728, batch_cost=0.9325, reader_cost=0.0002 | ETA 04:30:26 2020-11-30 09:24:33 [INFO] [TRAIN] epoch=35, iter=22700/40000, loss=0.1032, lr=0.004703, batch_cost=0.8976, reader_cost=0.0001 | ETA 04:18:48 2020-11-30 09:26:03 [INFO] [TRAIN] epoch=35, iter=22800/40000, loss=0.1207, lr=0.004679, batch_cost=0.8977, reader_cost=0.0002 | ETA 04:17:20 2020-11-30 09:27:33 [INFO] [TRAIN] epoch=35, iter=22900/40000, loss=0.1129, lr=0.004654, batch_cost=0.9000, reader_cost=0.0002 | ETA 04:16:30 2020-11-30 09:29:03 [INFO] [TRAIN] epoch=35, iter=23000/40000, loss=0.1201, lr=0.004630, batch_cost=0.9042, reader_cost=0.0002 | ETA 04:16:11 2020-11-30 09:30:34 [INFO] [TRAIN] epoch=35, iter=23100/40000, loss=0.1154, lr=0.004605, batch_cost=0.9018, reader_cost=0.0002 | ETA 04:13:59 2020-11-30 09:32:04 [INFO] [TRAIN] epoch=36, iter=23200/40000, loss=0.1364, lr=0.004581, batch_cost=0.9056, reader_cost=0.0070 | ETA 04:13:33 2020-11-30 09:33:33 [INFO] [TRAIN] epoch=36, iter=23300/40000, loss=0.1239, lr=0.004556, batch_cost=0.8925, reader_cost=0.0002 | ETA 04:08:25 2020-11-30 09:35:03 [INFO] [TRAIN] epoch=36, iter=23400/40000, loss=0.1241, lr=0.004532, batch_cost=0.8919, reader_cost=0.0003 | ETA 04:06:46 2020-11-30 09:36:32 [INFO] [TRAIN] epoch=36, iter=23500/40000, loss=0.1035, lr=0.004507, batch_cost=0.8979, reader_cost=0.0004 | ETA 04:06:55 2020-11-30 09:38:02 [INFO] [TRAIN] epoch=36, iter=23600/40000, loss=0.1117, lr=0.004483, batch_cost=0.8971, reader_cost=0.0004 | ETA 04:05:12 2020-11-30 09:39:32 [INFO] [TRAIN] epoch=36, iter=23700/40000, loss=0.1053, lr=0.004458, batch_cost=0.8999, reader_cost=0.0003 | ETA 04:04:27 2020-11-30 09:41:03 [INFO] [TRAIN] epoch=37, iter=23800/40000, loss=0.1157, lr=0.004433, batch_cost=0.9053, reader_cost=0.0067 | ETA 04:04:26 2020-11-30 09:42:32 [INFO] [TRAIN] epoch=37, iter=23900/40000, loss=0.1230, lr=0.004409, batch_cost=0.8937, reader_cost=0.0003 | ETA 03:59:48 2020-11-30 09:44:02 [INFO] [TRAIN] epoch=37, iter=24000/40000, loss=0.1171, lr=0.004384, batch_cost=0.9032, reader_cost=0.0005 | ETA 04:00:51 2020-11-30 09:44:02 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-30 09:44:29 [INFO] [EVAL] #Images=1449 mIoU=0.7769 Acc=0.9502 Kappa=0.8891 2020-11-30 09:44:29 [INFO] [EVAL] Class IoU: [0.9433 0.8579 0.4329 0.8946 0.7383 0.8137 0.9178 0.8897 0.9239 0.366 0.9179 0.5842 0.8632 0.8926 0.8652 0.8636 0.6121 0.9093 0.5083 0.8696 0.6501] 2020-11-30 09:44:29 [INFO] [EVAL] Class Acc: [0.9642 0.8893 0.4518 0.9314 0.8305 0.8748 0.9799 0.9531 0.9527 0.7927 0.9671 0.9454 0.9219 0.9374 0.8869 0.9561 0.7946 0.9602 0.7672 0.9024 0.8595] 2020-11-30 09:44:33 [INFO] [EVAL] The model with the best validation mIoU (0.7771) was saved at iter 20000. 2020-11-30 09:46:03 [INFO] [TRAIN] epoch=37, iter=24100/40000, loss=0.1279, lr=0.004359, batch_cost=0.9028, reader_cost=0.0007 | ETA 03:59:14 2020-11-30 09:47:34 [INFO] [TRAIN] epoch=37, iter=24200/40000, loss=0.1127, lr=0.004335, batch_cost=0.9033, reader_cost=0.0005 | ETA 03:57:51 2020-11-30 09:49:03 [INFO] [TRAIN] epoch=37, iter=24300/40000, loss=0.1414, lr=0.004310, batch_cost=0.8980, reader_cost=0.0008 | ETA 03:54:57 2020-11-30 09:50:33 [INFO] [TRAIN] epoch=37, iter=24400/40000, loss=0.1214, lr=0.004285, batch_cost=0.8971, reader_cost=0.0005 | ETA 03:53:15 2020-11-30 09:52:04 [INFO] [TRAIN] epoch=38, iter=24500/40000, loss=0.1176, lr=0.004261, batch_cost=0.9069, reader_cost=0.0071 | ETA 03:54:16 2020-11-30 09:53:33 [INFO] [TRAIN] epoch=38, iter=24600/40000, loss=0.1211, lr=0.004236, batch_cost=0.8941, reader_cost=0.0006 | ETA 03:49:28 2020-11-30 09:55:03 [INFO] [TRAIN] epoch=38, iter=24700/40000, loss=0.1208, lr=0.004211, batch_cost=0.9002, reader_cost=0.0006 | ETA 03:49:32 2020-11-30 09:56:33 [INFO] [TRAIN] epoch=38, iter=24800/40000, loss=0.1277, lr=0.004186, batch_cost=0.9001, reader_cost=0.0005 | ETA 03:48:01 2020-11-30 09:58:03 [INFO] [TRAIN] epoch=38, iter=24900/40000, loss=0.1145, lr=0.004162, batch_cost=0.8992, reader_cost=0.0006 | ETA 03:46:18 2020-11-30 09:59:33 [INFO] [TRAIN] epoch=38, iter=25000/40000, loss=0.1056, lr=0.004137, batch_cost=0.8973, reader_cost=0.0005 | ETA 03:44:19 2020-11-30 10:01:03 [INFO] [TRAIN] epoch=38, iter=25100/40000, loss=0.1293, lr=0.004112, batch_cost=0.9034, reader_cost=0.0006 | ETA 03:44:21 2020-11-30 10:02:34 [INFO] [TRAIN] epoch=39, iter=25200/40000, loss=0.1097, lr=0.004087, batch_cost=0.9072, reader_cost=0.0063 | ETA 03:43:47 2020-11-30 10:04:03 [INFO] [TRAIN] epoch=39, iter=25300/40000, loss=0.1060, lr=0.004062, batch_cost=0.8951, reader_cost=0.0003 | ETA 03:39:17 2020-11-30 10:05:33 [INFO] [TRAIN] epoch=39, iter=25400/40000, loss=0.1054, lr=0.004037, batch_cost=0.8958, reader_cost=0.0006 | ETA 03:37:58 2020-11-30 10:07:03 [INFO] [TRAIN] epoch=39, iter=25500/40000, loss=0.0985, lr=0.004012, batch_cost=0.8987, reader_cost=0.0004 | ETA 03:37:11 2020-11-30 10:08:33 [INFO] [TRAIN] epoch=39, iter=25600/40000, loss=0.0975, lr=0.003987, batch_cost=0.8991, reader_cost=0.0008 | ETA 03:35:47 2020-11-30 10:10:03 [INFO] [TRAIN] epoch=39, iter=25700/40000, loss=0.1105, lr=0.003963, batch_cost=0.8980, reader_cost=0.0003 | ETA 03:34:01 2020-11-30 10:11:33 [INFO] [TRAIN] epoch=40, iter=25800/40000, loss=0.1321, lr=0.003938, batch_cost=0.9051, reader_cost=0.0070 | ETA 03:34:12 2020-11-30 10:13:03 [INFO] [TRAIN] epoch=40, iter=25900/40000, loss=0.1207, lr=0.003913, batch_cost=0.8979, reader_cost=0.0004 | ETA 03:31:00 2020-11-30 10:14:33 [INFO] [TRAIN] epoch=40, iter=26000/40000, loss=0.1157, lr=0.003888, batch_cost=0.9009, reader_cost=0.0004 | ETA 03:30:12 2020-11-30 10:16:04 [INFO] [TRAIN] epoch=40, iter=26100/40000, loss=0.1109, lr=0.003863, batch_cost=0.9076, reader_cost=0.0007 | ETA 03:30:15 2020-11-30 10:17:34 [INFO] [TRAIN] epoch=40, iter=26200/40000, loss=0.1066, lr=0.003838, batch_cost=0.9039, reader_cost=0.0006 | ETA 03:27:54 2020-11-30 10:19:04 [INFO] [TRAIN] epoch=40, iter=26300/40000, loss=0.0908, lr=0.003813, batch_cost=0.9017, reader_cost=0.0007 | ETA 03:25:53 2020-11-30 10:20:34 [INFO] [TRAIN] epoch=40, iter=26400/40000, loss=0.1046, lr=0.003788, batch_cost=0.9007, reader_cost=0.0003 | ETA 03:24:09 2020-11-30 10:22:05 [INFO] [TRAIN] epoch=41, iter=26500/40000, loss=0.1140, lr=0.003762, batch_cost=0.9071, reader_cost=0.0064 | ETA 03:24:05 2020-11-30 10:23:35 [INFO] [TRAIN] epoch=41, iter=26600/40000, loss=0.1021, lr=0.003737, batch_cost=0.8970, reader_cost=0.0003 | ETA 03:20:19 2020-11-30 10:25:05 [INFO] [TRAIN] epoch=41, iter=26700/40000, loss=0.1151, lr=0.003712, batch_cost=0.9054, reader_cost=0.0002 | ETA 03:20:42 2020-11-30 10:26:36 [INFO] [TRAIN] epoch=41, iter=26800/40000, loss=0.1066, lr=0.003687, batch_cost=0.9046, reader_cost=0.0002 | ETA 03:19:00 2020-11-30 10:28:07 [INFO] [TRAIN] epoch=41, iter=26900/40000, loss=0.0988, lr=0.003662, batch_cost=0.9144, reader_cost=0.0003 | ETA 03:19:38 2020-11-30 10:29:38 [INFO] [TRAIN] epoch=41, iter=27000/40000, loss=0.1195, lr=0.003637, batch_cost=0.9033, reader_cost=0.0005 | ETA 03:15:43 2020-11-30 10:31:08 [INFO] [TRAIN] epoch=41, iter=27100/40000, loss=0.1107, lr=0.003612, batch_cost=0.9041, reader_cost=0.0002 | ETA 03:14:22 2020-11-30 10:32:39 [INFO] [TRAIN] epoch=42, iter=27200/40000, loss=0.1029, lr=0.003586, batch_cost=0.9060, reader_cost=0.0067 | ETA 03:13:16 2020-11-30 10:34:08 [INFO] [TRAIN] epoch=42, iter=27300/40000, loss=0.0971, lr=0.003561, batch_cost=0.8959, reader_cost=0.0002 | ETA 03:09:38 2020-11-30 10:35:38 [INFO] [TRAIN] epoch=42, iter=27400/40000, loss=0.1069, lr=0.003536, batch_cost=0.8975, reader_cost=0.0002 | ETA 03:08:28 2020-11-30 10:37:08 [INFO] [TRAIN] epoch=42, iter=27500/40000, loss=0.1054, lr=0.003511, batch_cost=0.8990, reader_cost=0.0002 | ETA 03:07:18 2020-11-30 10:38:38 [INFO] [TRAIN] epoch=42, iter=27600/40000, loss=0.0950, lr=0.003485, batch_cost=0.8984, reader_cost=0.0002 | ETA 03:05:40 2020-11-30 10:40:08 [INFO] [TRAIN] epoch=42, iter=27700/40000, loss=0.1016, lr=0.003460, batch_cost=0.9043, reader_cost=0.0002 | ETA 03:05:23 2020-11-30 10:41:39 [INFO] [TRAIN] epoch=43, iter=27800/40000, loss=0.1054, lr=0.003435, batch_cost=0.9077, reader_cost=0.0063 | ETA 03:04:33 2020-11-30 10:43:09 [INFO] [TRAIN] epoch=43, iter=27900/40000, loss=0.0937, lr=0.003409, batch_cost=0.9026, reader_cost=0.0003 | ETA 03:02:01 2020-11-30 10:44:39 [INFO] [TRAIN] epoch=43, iter=28000/40000, loss=0.0998, lr=0.003384, batch_cost=0.8963, reader_cost=0.0004 | ETA 02:59:15 2020-11-30 10:44:39 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-30 10:45:06 [INFO] [EVAL] #Images=1449 mIoU=0.7883 Acc=0.9529 Kappa=0.8952 2020-11-30 10:45:06 [INFO] [EVAL] Class IoU: [0.9451 0.9054 0.4401 0.8953 0.6892 0.7993 0.9602 0.8988 0.9391 0.474 0.9012 0.5255 0.889 0.8777 0.8971 0.8825 0.6086 0.9088 0.5343 0.9039 0.6803] 2020-11-30 10:45:06 [INFO] [EVAL] Class Acc: [0.9649 0.9662 0.453 0.9453 0.8547 0.9003 0.9835 0.9684 0.967 0.6666 0.9791 0.9481 0.9436 0.9123 0.9288 0.9549 0.7566 0.9647 0.8231 0.9711 0.8277] 2020-11-30 10:45:12 [INFO] [EVAL] The model with the best validation mIoU (0.7883) was saved at iter 28000. 2020-11-30 10:46:43 [INFO] [TRAIN] epoch=43, iter=28100/40000, loss=0.1004, lr=0.003359, batch_cost=0.9021, reader_cost=0.0006 | ETA 02:58:54 2020-11-30 10:48:13 [INFO] [TRAIN] epoch=43, iter=28200/40000, loss=0.1069, lr=0.003333, batch_cost=0.9026, reader_cost=0.0005 | ETA 02:57:30 2020-11-30 10:49:43 [INFO] [TRAIN] epoch=43, iter=28300/40000, loss=0.1033, lr=0.003308, batch_cost=0.8997, reader_cost=0.0003 | ETA 02:55:26 2020-11-30 10:51:13 [INFO] [TRAIN] epoch=43, iter=28400/40000, loss=0.1002, lr=0.003282, batch_cost=0.8978, reader_cost=0.0005 | ETA 02:53:34 2020-11-30 10:52:43 [INFO] [TRAIN] epoch=44, iter=28500/40000, loss=0.0918, lr=0.003257, batch_cost=0.9061, reader_cost=0.0066 | ETA 02:53:40 2020-11-30 10:54:13 [INFO] [TRAIN] epoch=44, iter=28600/40000, loss=0.0961, lr=0.003231, batch_cost=0.8978, reader_cost=0.0003 | ETA 02:50:35 2020-11-30 10:55:42 [INFO] [TRAIN] epoch=44, iter=28700/40000, loss=0.0942, lr=0.003206, batch_cost=0.8935, reader_cost=0.0002 | ETA 02:48:16 2020-11-30 10:57:12 [INFO] [TRAIN] epoch=44, iter=28800/40000, loss=0.0958, lr=0.003180, batch_cost=0.8998, reader_cost=0.0002 | ETA 02:47:57 2020-11-30 10:58:42 [INFO] [TRAIN] epoch=44, iter=28900/40000, loss=0.0938, lr=0.003155, batch_cost=0.8938, reader_cost=0.0002 | ETA 02:45:21 2020-11-30 11:00:11 [INFO] [TRAIN] epoch=44, iter=29000/40000, loss=0.1015, lr=0.003129, batch_cost=0.8964, reader_cost=0.0002 | ETA 02:44:20 2020-11-30 11:01:42 [INFO] [TRAIN] epoch=45, iter=29100/40000, loss=0.1004, lr=0.003104, batch_cost=0.9033, reader_cost=0.0065 | ETA 02:44:05 2020-11-30 11:03:12 [INFO] [TRAIN] epoch=45, iter=29200/40000, loss=0.0918, lr=0.003078, batch_cost=0.9038, reader_cost=0.0002 | ETA 02:42:40 2020-11-30 11:04:43 [INFO] [TRAIN] epoch=45, iter=29300/40000, loss=0.0876, lr=0.003052, batch_cost=0.9070, reader_cost=0.0002 | ETA 02:41:44 2020-11-30 11:06:13 [INFO] [TRAIN] epoch=45, iter=29400/40000, loss=0.0975, lr=0.003027, batch_cost=0.9025, reader_cost=0.0002 | ETA 02:39:26 2020-11-30 11:07:44 [INFO] [TRAIN] epoch=45, iter=29500/40000, loss=0.0875, lr=0.003001, batch_cost=0.9079, reader_cost=0.0006 | ETA 02:38:53 2020-11-30 11:09:14 [INFO] [TRAIN] epoch=45, iter=29600/40000, loss=0.0869, lr=0.002975, batch_cost=0.9044, reader_cost=0.0002 | ETA 02:36:45 2020-11-30 11:10:45 [INFO] [TRAIN] epoch=45, iter=29700/40000, loss=0.0917, lr=0.002949, batch_cost=0.9034, reader_cost=0.0005 | ETA 02:35:05 2020-11-30 11:12:15 [INFO] [TRAIN] epoch=46, iter=29800/40000, loss=0.1055, lr=0.002924, batch_cost=0.9080, reader_cost=0.0075 | ETA 02:34:21 2020-11-30 11:13:46 [INFO] [TRAIN] epoch=46, iter=29900/40000, loss=0.0987, lr=0.002898, batch_cost=0.9017, reader_cost=0.0002 | ETA 02:31:47 2020-11-30 11:15:16 [INFO] [TRAIN] epoch=46, iter=30000/40000, loss=0.1079, lr=0.002872, batch_cost=0.9080, reader_cost=0.0002 | ETA 02:31:20 2020-11-30 11:16:47 [INFO] [TRAIN] epoch=46, iter=30100/40000, loss=0.0960, lr=0.002846, batch_cost=0.9060, reader_cost=0.0002 | ETA 02:29:29 2020-11-30 11:18:18 [INFO] [TRAIN] epoch=46, iter=30200/40000, loss=0.0856, lr=0.002820, batch_cost=0.9088, reader_cost=0.0005 | ETA 02:28:26 2020-11-30 11:19:49 [INFO] [TRAIN] epoch=46, iter=30300/40000, loss=0.0909, lr=0.002794, batch_cost=0.9066, reader_cost=0.0005 | ETA 02:26:33 2020-11-30 11:21:20 [INFO] [TRAIN] epoch=46, iter=30400/40000, loss=0.0933, lr=0.002768, batch_cost=0.9101, reader_cost=0.0002 | ETA 02:25:36 2020-11-30 11:22:50 [INFO] [TRAIN] epoch=47, iter=30500/40000, loss=0.0897, lr=0.002742, batch_cost=0.9066, reader_cost=0.0067 | ETA 02:23:33 2020-11-30 11:24:21 [INFO] [TRAIN] epoch=47, iter=30600/40000, loss=0.0861, lr=0.002716, batch_cost=0.9061, reader_cost=0.0003 | ETA 02:21:57 2020-11-30 11:25:51 [INFO] [TRAIN] epoch=47, iter=30700/40000, loss=0.0920, lr=0.002690, batch_cost=0.9064, reader_cost=0.0004 | ETA 02:20:29 2020-11-30 11:27:22 [INFO] [TRAIN] epoch=47, iter=30800/40000, loss=0.0885, lr=0.002664, batch_cost=0.9052, reader_cost=0.0007 | ETA 02:18:47 2020-11-30 11:28:53 [INFO] [TRAIN] epoch=47, iter=30900/40000, loss=0.0954, lr=0.002638, batch_cost=0.9069, reader_cost=0.0006 | ETA 02:17:32 2020-11-30 11:30:24 [INFO] [TRAIN] epoch=47, iter=31000/40000, loss=0.0864, lr=0.002612, batch_cost=0.9154, reader_cost=0.0006 | ETA 02:17:18 2020-11-30 11:31:56 [INFO] [TRAIN] epoch=48, iter=31100/40000, loss=0.0915, lr=0.002586, batch_cost=0.9145, reader_cost=0.0075 | ETA 02:15:39 2020-11-30 11:33:25 [INFO] [TRAIN] epoch=48, iter=31200/40000, loss=0.0898, lr=0.002560, batch_cost=0.8981, reader_cost=0.0005 | ETA 02:11:43 2020-11-30 11:34:56 [INFO] [TRAIN] epoch=48, iter=31300/40000, loss=0.0830, lr=0.002534, batch_cost=0.9096, reader_cost=0.0007 | ETA 02:11:53 2020-11-30 11:36:28 [INFO] [TRAIN] epoch=48, iter=31400/40000, loss=0.0867, lr=0.002507, batch_cost=0.9132, reader_cost=0.0007 | ETA 02:10:53 2020-11-30 11:37:59 [INFO] [TRAIN] epoch=48, iter=31500/40000, loss=0.0878, lr=0.002481, batch_cost=0.9161, reader_cost=0.0003 | ETA 02:09:47 2020-11-30 11:39:31 [INFO] [TRAIN] epoch=48, iter=31600/40000, loss=0.0848, lr=0.002455, batch_cost=0.9127, reader_cost=0.0005 | ETA 02:07:46 2020-11-30 11:41:02 [INFO] [TRAIN] epoch=48, iter=31700/40000, loss=0.0928, lr=0.002429, batch_cost=0.9095, reader_cost=0.0002 | ETA 02:05:48 2020-11-30 11:42:33 [INFO] [TRAIN] epoch=49, iter=31800/40000, loss=0.0861, lr=0.002402, batch_cost=0.9119, reader_cost=0.0073 | ETA 02:04:37 2020-11-30 11:44:03 [INFO] [TRAIN] epoch=49, iter=31900/40000, loss=0.0934, lr=0.002376, batch_cost=0.9016, reader_cost=0.0002 | ETA 02:01:43 2020-11-30 11:45:33 [INFO] [TRAIN] epoch=49, iter=32000/40000, loss=0.0907, lr=0.002350, batch_cost=0.9026, reader_cost=0.0002 | ETA 02:00:20 2020-11-30 11:45:33 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-30 11:45:59 [INFO] [EVAL] #Images=1449 mIoU=0.7896 Acc=0.9535 Kappa=0.8969 2020-11-30 11:45:59 [INFO] [EVAL] Class IoU: [0.9469 0.8737 0.4478 0.9023 0.7069 0.7985 0.9581 0.8889 0.9451 0.4304 0.8877 0.5767 0.8848 0.9034 0.8947 0.8813 0.6188 0.8806 0.5437 0.886 0.7244] 2020-11-30 11:45:59 [INFO] [EVAL] Class Acc: [0.9678 0.9096 0.4629 0.9502 0.8036 0.8768 0.9811 0.9594 0.9726 0.6579 0.9787 0.9292 0.9219 0.9367 0.9414 0.9553 0.8059 0.9299 0.7412 0.9622 0.9405] 2020-11-30 11:46:05 [INFO] [EVAL] The model with the best validation mIoU (0.7896) was saved at iter 32000. 2020-11-30 11:47:35 [INFO] [TRAIN] epoch=49, iter=32100/40000, loss=0.0912, lr=0.002323, batch_cost=0.9010, reader_cost=0.0008 | ETA 01:58:38 2020-11-30 11:49:06 [INFO] [TRAIN] epoch=49, iter=32200/40000, loss=0.0894, lr=0.002297, batch_cost=0.9045, reader_cost=0.0003 | ETA 01:57:35 2020-11-30 11:50:36 [INFO] [TRAIN] epoch=49, iter=32300/40000, loss=0.0793, lr=0.002270, batch_cost=0.9000, reader_cost=0.0005 | ETA 01:55:30 2020-11-30 11:52:07 [INFO] [TRAIN] epoch=50, iter=32400/40000, loss=0.0831, lr=0.002244, batch_cost=0.9090, reader_cost=0.0070 | ETA 01:55:08 2020-11-30 11:53:37 [INFO] [TRAIN] epoch=50, iter=32500/40000, loss=0.0892, lr=0.002217, batch_cost=0.9064, reader_cost=0.0004 | ETA 01:53:18 2020-11-30 11:55:07 [INFO] [TRAIN] epoch=50, iter=32600/40000, loss=0.0866, lr=0.002190, batch_cost=0.8960, reader_cost=0.0002 | ETA 01:50:30 2020-11-30 11:56:36 [INFO] [TRAIN] epoch=50, iter=32700/40000, loss=0.0875, lr=0.002164, batch_cost=0.8925, reader_cost=0.0002 | ETA 01:48:35 2020-11-30 11:58:06 [INFO] [TRAIN] epoch=50, iter=32800/40000, loss=0.0836, lr=0.002137, batch_cost=0.8993, reader_cost=0.0007 | ETA 01:47:54 2020-11-30 11:59:37 [INFO] [TRAIN] epoch=50, iter=32900/40000, loss=0.0850, lr=0.002110, batch_cost=0.9041, reader_cost=0.0004 | ETA 01:46:58 2020-11-30 12:01:06 [INFO] [TRAIN] epoch=50, iter=33000/40000, loss=0.0796, lr=0.002083, batch_cost=0.8933, reader_cost=0.0002 | ETA 01:44:12 2020-11-30 12:02:36 [INFO] [TRAIN] epoch=51, iter=33100/40000, loss=0.0906, lr=0.002057, batch_cost=0.9038, reader_cost=0.0073 | ETA 01:43:56 2020-11-30 12:04:07 [INFO] [TRAIN] epoch=51, iter=33200/40000, loss=0.0812, lr=0.002030, batch_cost=0.9068, reader_cost=0.0002 | ETA 01:42:46 2020-11-30 12:05:38 [INFO] [TRAIN] epoch=51, iter=33300/40000, loss=0.0853, lr=0.002003, batch_cost=0.9064, reader_cost=0.0002 | ETA 01:41:12 2020-11-30 12:07:08 [INFO] [TRAIN] epoch=51, iter=33400/40000, loss=0.0802, lr=0.001976, batch_cost=0.9057, reader_cost=0.0001 | ETA 01:39:37 2020-11-30 12:08:39 [INFO] [TRAIN] epoch=51, iter=33500/40000, loss=0.0890, lr=0.001949, batch_cost=0.9128, reader_cost=0.0001 | ETA 01:38:53 2020-11-30 12:10:11 [INFO] [TRAIN] epoch=51, iter=33600/40000, loss=0.0796, lr=0.001922, batch_cost=0.9132, reader_cost=0.0004 | ETA 01:37:24 2020-11-30 12:11:42 [INFO] [TRAIN] epoch=51, iter=33700/40000, loss=0.0838, lr=0.001895, batch_cost=0.9084, reader_cost=0.0002 | ETA 01:35:22 2020-11-30 12:13:12 [INFO] [TRAIN] epoch=52, iter=33800/40000, loss=0.0820, lr=0.001868, batch_cost=0.9090, reader_cost=0.0068 | ETA 01:33:55 2020-11-30 12:14:42 [INFO] [TRAIN] epoch=52, iter=33900/40000, loss=0.0894, lr=0.001841, batch_cost=0.8972, reader_cost=0.0002 | ETA 01:31:12 2020-11-30 12:16:13 [INFO] [TRAIN] epoch=52, iter=34000/40000, loss=0.0941, lr=0.001814, batch_cost=0.9086, reader_cost=0.0005 | ETA 01:30:51 2020-11-30 12:17:44 [INFO] [TRAIN] epoch=52, iter=34100/40000, loss=0.0750, lr=0.001786, batch_cost=0.9110, reader_cost=0.0007 | ETA 01:29:35 2020-11-30 12:19:15 [INFO] [TRAIN] epoch=52, iter=34200/40000, loss=0.0919, lr=0.001759, batch_cost=0.9071, reader_cost=0.0008 | ETA 01:27:41 2020-11-30 12:20:45 [INFO] [TRAIN] epoch=52, iter=34300/40000, loss=0.0816, lr=0.001732, batch_cost=0.9049, reader_cost=0.0005 | ETA 01:25:58 2020-11-30 12:22:17 [INFO] [TRAIN] epoch=53, iter=34400/40000, loss=0.0843, lr=0.001704, batch_cost=0.9139, reader_cost=0.0071 | ETA 01:25:17 2020-11-30 12:23:47 [INFO] [TRAIN] epoch=53, iter=34500/40000, loss=0.0816, lr=0.001677, batch_cost=0.9015, reader_cost=0.0005 | ETA 01:22:38 2020-11-30 12:25:17 [INFO] [TRAIN] epoch=53, iter=34600/40000, loss=0.0884, lr=0.001650, batch_cost=0.8968, reader_cost=0.0004 | ETA 01:20:42 2020-11-30 12:26:47 [INFO] [TRAIN] epoch=53, iter=34700/40000, loss=0.0850, lr=0.001622, batch_cost=0.9020, reader_cost=0.0004 | ETA 01:19:40 2020-11-30 12:28:17 [INFO] [TRAIN] epoch=53, iter=34800/40000, loss=0.0940, lr=0.001594, batch_cost=0.9009, reader_cost=0.0004 | ETA 01:18:04 2020-11-30 12:29:47 [INFO] [TRAIN] epoch=53, iter=34900/40000, loss=0.0813, lr=0.001567, batch_cost=0.9046, reader_cost=0.0002 | ETA 01:16:53 2020-11-30 12:31:18 [INFO] [TRAIN] epoch=53, iter=35000/40000, loss=0.0891, lr=0.001539, batch_cost=0.9025, reader_cost=0.0003 | ETA 01:15:12 2020-11-30 12:32:49 [INFO] [TRAIN] epoch=54, iter=35100/40000, loss=0.0802, lr=0.001511, batch_cost=0.9104, reader_cost=0.0071 | ETA 01:14:20 2020-11-30 12:34:19 [INFO] [TRAIN] epoch=54, iter=35200/40000, loss=0.0829, lr=0.001484, batch_cost=0.9011, reader_cost=0.0007 | ETA 01:12:05 2020-11-30 12:35:49 [INFO] [TRAIN] epoch=54, iter=35300/40000, loss=0.0928, lr=0.001456, batch_cost=0.9067, reader_cost=0.0005 | ETA 01:11:01 2020-11-30 12:37:20 [INFO] [TRAIN] epoch=54, iter=35400/40000, loss=0.0768, lr=0.001428, batch_cost=0.9053, reader_cost=0.0008 | ETA 01:09:24 2020-11-30 12:38:50 [INFO] [TRAIN] epoch=54, iter=35500/40000, loss=0.0773, lr=0.001400, batch_cost=0.9014, reader_cost=0.0007 | ETA 01:07:36 2020-11-30 12:40:21 [INFO] [TRAIN] epoch=54, iter=35600/40000, loss=0.0858, lr=0.001372, batch_cost=0.9064, reader_cost=0.0006 | ETA 01:06:28 2020-11-30 12:41:52 [INFO] [TRAIN] epoch=55, iter=35700/40000, loss=0.0813, lr=0.001344, batch_cost=0.9124, reader_cost=0.0067 | ETA 01:05:23 2020-11-30 12:43:22 [INFO] [TRAIN] epoch=55, iter=35800/40000, loss=0.0832, lr=0.001316, batch_cost=0.9026, reader_cost=0.0003 | ETA 01:03:11 2020-11-30 12:44:53 [INFO] [TRAIN] epoch=55, iter=35900/40000, loss=0.0823, lr=0.001287, batch_cost=0.9076, reader_cost=0.0007 | ETA 01:02:01 2020-11-30 12:46:23 [INFO] [TRAIN] epoch=55, iter=36000/40000, loss=0.0806, lr=0.001259, batch_cost=0.9029, reader_cost=0.0005 | ETA 01:00:11 2020-11-30 12:46:23 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-30 12:46:50 [INFO] [EVAL] #Images=1449 mIoU=0.8037 Acc=0.9562 Kappa=0.9028 2020-11-30 12:46:50 [INFO] [EVAL] Class IoU: [0.9483 0.8855 0.4424 0.9058 0.7141 0.8114 0.9602 0.904 0.9505 0.4764 0.9351 0.6006 0.9083 0.9198 0.8935 0.8859 0.639 0.9111 0.5382 0.9054 0.7429] 2020-11-30 12:46:50 [INFO] [EVAL] Class Acc: [0.9679 0.9262 0.4559 0.9509 0.8133 0.8863 0.9855 0.9656 0.9771 0.6965 0.9722 0.9256 0.9512 0.9508 0.9378 0.9578 0.8025 0.9725 0.8079 0.9678 0.8917] 2020-11-30 12:46:57 [INFO] [EVAL] The model with the best validation mIoU (0.8037) was saved at iter 36000. 2020-11-30 12:48:28 [INFO] [TRAIN] epoch=55, iter=36100/40000, loss=0.0735, lr=0.001231, batch_cost=0.9011, reader_cost=0.0006 | ETA 00:58:34 2020-11-30 12:49:58 [INFO] [TRAIN] epoch=55, iter=36200/40000, loss=0.0803, lr=0.001202, batch_cost=0.9014, reader_cost=0.0008 | ETA 00:57:05 2020-11-30 12:51:27 [INFO] [TRAIN] epoch=55, iter=36300/40000, loss=0.0849, lr=0.001174, batch_cost=0.8950, reader_cost=0.0005 | ETA 00:55:11 2020-11-30 12:52:58 [INFO] [TRAIN] epoch=56, iter=36400/40000, loss=0.0693, lr=0.001145, batch_cost=0.9096, reader_cost=0.0071 | ETA 00:54:34 2020-11-30 12:54:28 [INFO] [TRAIN] epoch=56, iter=36500/40000, loss=0.0869, lr=0.001117, batch_cost=0.9008, reader_cost=0.0003 | ETA 00:52:32 2020-11-30 12:55:58 [INFO] [TRAIN] epoch=56, iter=36600/40000, loss=0.0885, lr=0.001088, batch_cost=0.8957, reader_cost=0.0002 | ETA 00:50:45 2020-11-30 12:57:28 [INFO] [TRAIN] epoch=56, iter=36700/40000, loss=0.0796, lr=0.001059, batch_cost=0.9007, reader_cost=0.0002 | ETA 00:49:32 2020-11-30 12:58:58 [INFO] [TRAIN] epoch=56, iter=36800/40000, loss=0.0726, lr=0.001030, batch_cost=0.9012, reader_cost=0.0002 | ETA 00:48:03 2020-11-30 13:00:28 [INFO] [TRAIN] epoch=56, iter=36900/40000, loss=0.0822, lr=0.001001, batch_cost=0.8988, reader_cost=0.0002 | ETA 00:46:26 2020-11-30 13:01:58 [INFO] [TRAIN] epoch=56, iter=37000/40000, loss=0.0768, lr=0.000972, batch_cost=0.8962, reader_cost=0.0001 | ETA 00:44:48 2020-11-30 13:03:28 [INFO] [TRAIN] epoch=57, iter=37100/40000, loss=0.0834, lr=0.000943, batch_cost=0.9094, reader_cost=0.0067 | ETA 00:43:57 2020-11-30 13:04:59 [INFO] [TRAIN] epoch=57, iter=37200/40000, loss=0.0765, lr=0.000914, batch_cost=0.9031, reader_cost=0.0008 | ETA 00:42:08 2020-11-30 13:06:29 [INFO] [TRAIN] epoch=57, iter=37300/40000, loss=0.0777, lr=0.000884, batch_cost=0.9058, reader_cost=0.0009 | ETA 00:40:45 2020-11-30 13:08:00 [INFO] [TRAIN] epoch=57, iter=37400/40000, loss=0.0765, lr=0.000855, batch_cost=0.9102, reader_cost=0.0007 | ETA 00:39:26 2020-11-30 13:09:31 [INFO] [TRAIN] epoch=57, iter=37500/40000, loss=0.0823, lr=0.000825, batch_cost=0.9042, reader_cost=0.0013 | ETA 00:37:40 2020-11-30 13:11:02 [INFO] [TRAIN] epoch=57, iter=37600/40000, loss=0.0759, lr=0.000795, batch_cost=0.9134, reader_cost=0.0015 | ETA 00:36:32 2020-11-30 13:12:33 [INFO] [TRAIN] epoch=58, iter=37700/40000, loss=0.0819, lr=0.000765, batch_cost=0.9132, reader_cost=0.0065 | ETA 00:35:00 2020-11-30 13:14:04 [INFO] [TRAIN] epoch=58, iter=37800/40000, loss=0.0773, lr=0.000735, batch_cost=0.9059, reader_cost=0.0004 | ETA 00:33:12 2020-11-30 13:15:34 [INFO] [TRAIN] epoch=58, iter=37900/40000, loss=0.0714, lr=0.000705, batch_cost=0.9010, reader_cost=0.0003 | ETA 00:31:32 2020-11-30 13:17:04 [INFO] [TRAIN] epoch=58, iter=38000/40000, loss=0.0817, lr=0.000675, batch_cost=0.8997, reader_cost=0.0004 | ETA 00:29:59 2020-11-30 13:18:36 [INFO] [TRAIN] epoch=58, iter=38100/40000, loss=0.0751, lr=0.000645, batch_cost=0.9141, reader_cost=0.0008 | ETA 00:28:56 2020-11-30 13:20:07 [INFO] [TRAIN] epoch=58, iter=38200/40000, loss=0.0779, lr=0.000614, batch_cost=0.9111, reader_cost=0.0006 | ETA 00:27:19 2020-11-30 13:21:38 [INFO] [TRAIN] epoch=58, iter=38300/40000, loss=0.0883, lr=0.000583, batch_cost=0.9099, reader_cost=0.0003 | ETA 00:25:46 2020-11-30 13:23:09 [INFO] [TRAIN] epoch=59, iter=38400/40000, loss=0.0759, lr=0.000552, batch_cost=0.9115, reader_cost=0.0072 | ETA 00:24:18 2020-11-30 13:24:39 [INFO] [TRAIN] epoch=59, iter=38500/40000, loss=0.0772, lr=0.000521, batch_cost=0.8998, reader_cost=0.0003 | ETA 00:22:29 2020-11-30 13:26:09 [INFO] [TRAIN] epoch=59, iter=38600/40000, loss=0.0795, lr=0.000490, batch_cost=0.8995, reader_cost=0.0004 | ETA 00:20:59 2020-11-30 13:27:39 [INFO] [TRAIN] epoch=59, iter=38700/40000, loss=0.0745, lr=0.000458, batch_cost=0.8980, reader_cost=0.0002 | ETA 00:19:27 2020-11-30 13:29:09 [INFO] [TRAIN] epoch=59, iter=38800/40000, loss=0.0821, lr=0.000426, batch_cost=0.9007, reader_cost=0.0002 | ETA 00:18:00 2020-11-30 13:30:38 [INFO] [TRAIN] epoch=59, iter=38900/40000, loss=0.0778, lr=0.000394, batch_cost=0.8991, reader_cost=0.0002 | ETA 00:16:29 2020-11-30 13:32:09 [INFO] [TRAIN] epoch=60, iter=39000/40000, loss=0.0758, lr=0.000362, batch_cost=0.9061, reader_cost=0.0062 | ETA 00:15:06 2020-11-30 13:33:39 [INFO] [TRAIN] epoch=60, iter=39100/40000, loss=0.0759, lr=0.000329, batch_cost=0.8973, reader_cost=0.0002 | ETA 00:13:27 2020-11-30 13:35:09 [INFO] [TRAIN] epoch=60, iter=39200/40000, loss=0.0862, lr=0.000296, batch_cost=0.9049, reader_cost=0.0002 | ETA 00:12:03 2020-11-30 13:36:39 [INFO] [TRAIN] epoch=60, iter=39300/40000, loss=0.0801, lr=0.000263, batch_cost=0.9015, reader_cost=0.0002 | ETA 00:10:31 2020-11-30 13:38:10 [INFO] [TRAIN] epoch=60, iter=39400/40000, loss=0.0773, lr=0.000229, batch_cost=0.9004, reader_cost=0.0002 | ETA 00:09:00 2020-11-30 13:39:40 [INFO] [TRAIN] epoch=60, iter=39500/40000, loss=0.0751, lr=0.000194, batch_cost=0.9041, reader_cost=0.0002 | ETA 00:07:32 2020-11-30 13:41:10 [INFO] [TRAIN] epoch=60, iter=39600/40000, loss=0.0795, lr=0.000159, batch_cost=0.8963, reader_cost=0.0003 | ETA 00:05:58 2020-11-30 13:42:40 [INFO] [TRAIN] epoch=61, iter=39700/40000, loss=0.0753, lr=0.000123, batch_cost=0.9041, reader_cost=0.0066 | ETA 00:04:31 2020-11-30 13:44:09 [INFO] [TRAIN] epoch=61, iter=39800/40000, loss=0.0731, lr=0.000085, batch_cost=0.8953, reader_cost=0.0002 | ETA 00:02:59 2020-11-30 13:45:40 [INFO] [TRAIN] epoch=61, iter=39900/40000, loss=0.0778, lr=0.000046, batch_cost=0.9039, reader_cost=0.0002 | ETA 00:01:30 2020-11-30 13:47:10 [INFO] [TRAIN] epoch=61, iter=40000/40000, loss=0.0789, lr=0.000001, batch_cost=0.9030, reader_cost=0.0002 | ETA 00:00:00 2020-11-30 13:47:10 [INFO] Start evaluating (total_samples=1449, total_iters=363)... 2020-11-30 13:47:37 [INFO] [EVAL] #Images=1449 mIoU=0.8060 Acc=0.9570 Kappa=0.9048 2020-11-30 13:47:37 [INFO] [EVAL] Class IoU: [0.9493 0.8735 0.4466 0.9076 0.7017 0.8158 0.9589 0.9019 0.9488 0.4978 0.946 0.5738 0.9117 0.9163 0.8997 0.8867 0.6344 0.9185 0.5573 0.9196 0.7598] 2020-11-30 13:47:37 [INFO] [EVAL] Class Acc: [0.9693 0.9027 0.4613 0.9474 0.7979 0.8834 0.9815 0.9698 0.9795 0.6945 0.9777 0.921 0.9455 0.9463 0.943 0.9525 0.7867 0.9669 0.8504 0.9658 0.9187] 2020-11-30 13:47:44 [INFO] [EVAL] The model with the best validation mIoU (0.8060) was saved at iter 40000.