----------- Configuration Arguments ----------- gpus: None heter_worker_num: None heter_workers: http_port: None ips: 127.0.0.1 log_dir: log nproc_per_node: None server_num: None servers: training_script: train.py training_script_args: ['--do_eval', '--use_vdl', '--batch_size', '1', '--learning_rate', '0.01', '--config', 'configs/setr/setr_naive_large_cityscapes_769x769_40k.yml', '--save_interval', '1000', '--num_workers', '6', '--save_dir', 'log/setr_naive_large/exp17/'] worker_num: None workers: ------------------------------------------------ WARNING 2021-05-09 13:37:40,437 launch.py:316] Not found distinct arguments and compiled with cuda. Default use collective mode launch train in GPU mode INFO 2021-05-09 13:37:40,438 launch_utils.py:471] Local start 8 processes. First process distributed environment info (Only For Debug): +=======================================================================================+ | Distributed Envs Value | +---------------------------------------------------------------------------------------+ | PADDLE_TRAINER_ID 0 | | PADDLE_CURRENT_ENDPOINT 127.0.0.1:41283 | | PADDLE_TRAINERS_NUM 8 | | PADDLE_TRAINER_ENDPOINTS ... 0.1:52919,127.0.0.1:35485,127.0.0.1:53822| | FLAGS_selected_gpus 0 | +=======================================================================================+ INFO 2021-05-09 13:37:40,438 launch_utils.py:475] details abouts PADDLE_TRAINER_ENDPOINTS can be found in log/endpoints.log, and detail running logs maybe found in log/workerlog.0 which: no nvcc in (/ssd1/home/wuzewu/miniconda3/envs/paddle2/bin:/ssd1/home/wuzewu/miniconda3/condabin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/opt/bin:/home/opt/bin:/usr/local/sbin:/usr/sbin:/opt/bin:/home/opt/bin:/ssd1/home/wuzewu/.local/bin:/ssd1/home/wuzewu/bin:/opt/bin:/home/opt/bin:/opt/bin:/home/opt/bin:/ssd1/home/wuzewu/.local/bin:/ssd1/home/wuzewu/bin:/opt/bin:/home/opt/bin) 2021-05-09 13:37:42 [INFO] ------------Environment Information------------- platform: Linux-3.10.0-1062.18.1.el7.x86_64-x86_64-with-centos-7.7.1908-Core Python: 3.7.9 (default, Aug 31 2020, 12:42:55) [GCC 7.3.0] Paddle compiled with cuda: True NVCC: Cuda compilation tools, release 10.2, V10.2.89 cudnn: 7.6 GPUs used: 8 CUDA_VISIBLE_DEVICES: 0,1,2,3,4,5,6,7 GPU: ['GPU 0: Tesla V100-SXM2-32GB', 'GPU 1: Tesla V100-SXM2-32GB', 'GPU 2: Tesla V100-SXM2-32GB', 'GPU 3: Tesla V100-SXM2-32GB', 'GPU 4: Tesla V100-SXM2-32GB', 'GPU 5: Tesla V100-SXM2-32GB', 'GPU 6: Tesla V100-SXM2-32GB', 'GPU 7: Tesla V100-SXM2-32GB'] GCC: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39) PaddlePaddle: 2.0.1 OpenCV: 4.4.0 ------------------------------------------------ 2021-05-09 13:37:42 [INFO] ---------------Config Information--------------- batch_size: 1 iters: 40000 loss: coef: - 1 - 0.4 - 0.4 - 0.4 types: - ignore_index: 255 type: CrossEntropyLoss - ignore_index: 255 type: CrossEntropyLoss - ignore_index: 255 type: CrossEntropyLoss - ignore_index: 255 type: CrossEntropyLoss lr_scheduler: end_lr: 0.0001 learning_rate: 0.01 power: 0.9 type: PolynomialDecay model: align_corners: true backbone: pretrained: vit/ViT_large_patch16_384/model.pdparams type: ViT_large_patch16_384 backbone_indices: - 9 - 14 - 19 - 23 head: naive num_classes: 19 type: SegmentationTransformer optimizer: momentum: 0.9 type: sgd weight_decay: 0.0 train_dataset: dataset_root: data/cityscapes mode: train transforms: - max_scale_factor: 2.0 min_scale_factor: 0.25 scale_step_size: 0.25 type: ResizeStepScaling - crop_size: - 769 - 769 type: RandomPaddingCrop - type: RandomHorizontalFlip - brightness_range: 0.5 contrast_range: 0.5 saturation_range: 0.5 type: RandomDistort - type: Normalize type: Cityscapes val_dataset: dataset_root: data/cityscapes mode: val transforms: - target_size: - 2048 - 1024 type: Padding - type: Normalize type: Cityscapes ------------------------------------------------ W0509 13:37:42.591061 2546 device_context.cc:362] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 10.2, Runtime API Version: 10.2 W0509 13:37:42.591140 2546 device_context.cc:372] device: 0, cuDNN Version: 7.6. 2021-05-09 13:37:47 [INFO] Loading pretrained model from vit/ViT_large_patch16_384/model.pdparams 2021-05-09 13:37:58 [INFO] There are 294/294 variables loaded into VisionTransformer. W0509 13:37:58.922026 2546 nccl_context.cc:142] Socket connect worker 127.0.0.1:36498 failed, try again after 3 seconds. I0509 13:38:01.926283 2546 nccl_context.cc:189] init nccl context nranks: 8 local rank: 0 gpu id: 0 ring id: 0 2021-05-09 13:38:14 [INFO] [TRAIN] epoch: 1, iter: 10/40000, loss: 2.8749, lr: 0.009998, batch_cost: 1.0809, reader_cost: 0.26323, ips: 0.9252 samples/sec | ETA 12:00:24 2021-05-09 13:38:22 [INFO] [TRAIN] epoch: 1, iter: 20/40000, loss: 1.6715, lr: 0.009996, batch_cost: 0.7809, reader_cost: 0.00031, ips: 1.2807 samples/sec | ETA 08:40:18 2021-05-09 13:38:30 [INFO] [TRAIN] epoch: 1, iter: 30/40000, loss: 1.6944, lr: 0.009994, batch_cost: 0.7791, reader_cost: 0.00017, ips: 1.2835 samples/sec | ETA 08:39:01 2021-05-09 13:38:37 [INFO] [TRAIN] epoch: 1, iter: 40/40000, loss: 1.4572, lr: 0.009991, batch_cost: 0.7802, reader_cost: 0.00017, ips: 1.2818 samples/sec | ETA 08:39:35 2021-05-09 13:38:45 [INFO] [TRAIN] epoch: 1, iter: 50/40000, loss: 1.6616, lr: 0.009989, batch_cost: 0.7817, reader_cost: 0.00016, ips: 1.2792 samples/sec | ETA 08:40:29 2021-05-09 13:38:53 [INFO] [TRAIN] epoch: 1, iter: 60/40000, loss: 5.1221, lr: 0.009987, batch_cost: 0.7830, reader_cost: 0.00016, ips: 1.2772 samples/sec | ETA 08:41:11 2021-05-09 13:39:01 [INFO] [TRAIN] epoch: 1, iter: 70/40000, loss: 1.9327, lr: 0.009985, batch_cost: 0.7835, reader_cost: 0.00015, ips: 1.2763 samples/sec | ETA 08:41:26 2021-05-09 13:39:09 [INFO] [TRAIN] epoch: 1, iter: 80/40000, loss: 1.9442, lr: 0.009982, batch_cost: 0.7846, reader_cost: 0.00015, ips: 1.2745 samples/sec | ETA 08:42:00 2021-05-09 13:39:17 [INFO] [TRAIN] epoch: 1, iter: 90/40000, loss: 2.0633, lr: 0.009980, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2736 samples/sec | ETA 08:42:15 2021-05-09 13:39:24 [INFO] [TRAIN] epoch: 1, iter: 100/40000, loss: 1.5385, lr: 0.009978, batch_cost: 0.7848, reader_cost: 0.00016, ips: 1.2742 samples/sec | ETA 08:41:53 2021-05-09 13:39:32 [INFO] [TRAIN] epoch: 1, iter: 110/40000, loss: 1.2045, lr: 0.009976, batch_cost: 0.7847, reader_cost: 0.00016, ips: 1.2744 samples/sec | ETA 08:41:40 2021-05-09 13:39:40 [INFO] [TRAIN] epoch: 1, iter: 120/40000, loss: 1.4068, lr: 0.009973, batch_cost: 0.7857, reader_cost: 0.00017, ips: 1.2727 samples/sec | ETA 08:42:14 2021-05-09 13:39:48 [INFO] [TRAIN] epoch: 1, iter: 130/40000, loss: 0.9257, lr: 0.009971, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2723 samples/sec | ETA 08:42:16 2021-05-09 13:39:56 [INFO] [TRAIN] epoch: 1, iter: 140/40000, loss: 1.7320, lr: 0.009969, batch_cost: 0.7859, reader_cost: 0.00014, ips: 1.2724 samples/sec | ETA 08:42:06 2021-05-09 13:40:04 [INFO] [TRAIN] epoch: 1, iter: 150/40000, loss: 1.5376, lr: 0.009967, batch_cost: 0.7873, reader_cost: 0.00016, ips: 1.2702 samples/sec | ETA 08:42:53 2021-05-09 13:40:12 [INFO] [TRAIN] epoch: 1, iter: 160/40000, loss: 0.9056, lr: 0.009965, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2732 samples/sec | ETA 08:41:32 2021-05-09 13:40:19 [INFO] [TRAIN] epoch: 1, iter: 170/40000, loss: 1.4047, lr: 0.009962, batch_cost: 0.7848, reader_cost: 0.00014, ips: 1.2742 samples/sec | ETA 08:40:57 2021-05-09 13:40:27 [INFO] [TRAIN] epoch: 1, iter: 180/40000, loss: 1.6636, lr: 0.009960, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2721 samples/sec | ETA 08:41:41 2021-05-09 13:40:35 [INFO] [TRAIN] epoch: 1, iter: 190/40000, loss: 0.5739, lr: 0.009958, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2708 samples/sec | ETA 08:42:05 2021-05-09 13:40:43 [INFO] [TRAIN] epoch: 1, iter: 200/40000, loss: 1.4374, lr: 0.009956, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 08:41:26 2021-05-09 13:40:51 [INFO] [TRAIN] epoch: 1, iter: 210/40000, loss: 0.9333, lr: 0.009953, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2735 samples/sec | ETA 08:40:44 2021-05-09 13:40:59 [INFO] [TRAIN] epoch: 1, iter: 220/40000, loss: 0.7289, lr: 0.009951, batch_cost: 0.7867, reader_cost: 0.00015, ips: 1.2711 samples/sec | ETA 08:41:36 2021-05-09 13:41:07 [INFO] [TRAIN] epoch: 1, iter: 230/40000, loss: 0.9640, lr: 0.009949, batch_cost: 0.7858, reader_cost: 0.00015, ips: 1.2726 samples/sec | ETA 08:40:50 2021-05-09 13:41:14 [INFO] [TRAIN] epoch: 1, iter: 240/40000, loss: 0.7699, lr: 0.009947, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2735 samples/sec | ETA 08:40:20 2021-05-09 13:41:22 [INFO] [TRAIN] epoch: 1, iter: 250/40000, loss: 1.1600, lr: 0.009945, batch_cost: 0.7851, reader_cost: 0.00015, ips: 1.2738 samples/sec | ETA 08:40:05 2021-05-09 13:41:30 [INFO] [TRAIN] epoch: 1, iter: 260/40000, loss: 1.0631, lr: 0.009942, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 08:41:02 2021-05-09 13:41:38 [INFO] [TRAIN] epoch: 1, iter: 270/40000, loss: 0.5687, lr: 0.009940, batch_cost: 0.7851, reader_cost: 0.00016, ips: 1.2738 samples/sec | ETA 08:39:50 2021-05-09 13:41:46 [INFO] [TRAIN] epoch: 1, iter: 280/40000, loss: 1.6009, lr: 0.009938, batch_cost: 0.7847, reader_cost: 0.00017, ips: 1.2743 samples/sec | ETA 08:39:29 2021-05-09 13:41:54 [INFO] [TRAIN] epoch: 1, iter: 290/40000, loss: 1.4207, lr: 0.009936, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2729 samples/sec | ETA 08:39:56 2021-05-09 13:42:02 [INFO] [TRAIN] epoch: 1, iter: 300/40000, loss: 0.9085, lr: 0.009933, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2727 samples/sec | ETA 08:39:54 2021-05-09 13:42:09 [INFO] [TRAIN] epoch: 1, iter: 310/40000, loss: 1.1426, lr: 0.009931, batch_cost: 0.7852, reader_cost: 0.00017, ips: 1.2736 samples/sec | ETA 08:39:23 2021-05-09 13:42:17 [INFO] [TRAIN] epoch: 1, iter: 320/40000, loss: 1.0704, lr: 0.009929, batch_cost: 0.7848, reader_cost: 0.00017, ips: 1.2743 samples/sec | ETA 08:38:59 2021-05-09 13:42:25 [INFO] [TRAIN] epoch: 1, iter: 330/40000, loss: 1.1947, lr: 0.009927, batch_cost: 0.7851, reader_cost: 0.00016, ips: 1.2737 samples/sec | ETA 08:39:06 2021-05-09 13:42:33 [INFO] [TRAIN] epoch: 1, iter: 340/40000, loss: 0.9796, lr: 0.009924, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 08:39:29 2021-05-09 13:42:41 [INFO] [TRAIN] epoch: 1, iter: 350/40000, loss: 0.9218, lr: 0.009922, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2717 samples/sec | ETA 08:39:38 2021-05-09 13:42:49 [INFO] [TRAIN] epoch: 1, iter: 360/40000, loss: 1.0766, lr: 0.009920, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2727 samples/sec | ETA 08:39:06 2021-05-09 13:42:57 [INFO] [TRAIN] epoch: 1, iter: 370/40000, loss: 1.0755, lr: 0.009918, batch_cost: 0.7850, reader_cost: 0.00029, ips: 1.2740 samples/sec | ETA 08:38:27 2021-05-09 13:43:07 [INFO] [TRAIN] epoch: 2, iter: 380/40000, loss: 0.9299, lr: 0.009916, batch_cost: 1.0766, reader_cost: 0.27939, ips: 0.9289 samples/sec | ETA 11:50:54 2021-05-09 13:43:15 [INFO] [TRAIN] epoch: 2, iter: 390/40000, loss: 1.4119, lr: 0.009913, batch_cost: 0.7905, reader_cost: 0.00033, ips: 1.2650 samples/sec | ETA 08:41:52 2021-05-09 13:43:23 [INFO] [TRAIN] epoch: 2, iter: 400/40000, loss: 0.8895, lr: 0.009911, batch_cost: 0.7849, reader_cost: 0.00016, ips: 1.2740 samples/sec | ETA 08:38:03 2021-05-09 13:43:31 [INFO] [TRAIN] epoch: 2, iter: 410/40000, loss: 0.8064, lr: 0.009909, batch_cost: 0.7854, reader_cost: 0.00017, ips: 1.2732 samples/sec | ETA 08:38:14 2021-05-09 13:43:39 [INFO] [TRAIN] epoch: 2, iter: 420/40000, loss: 0.9860, lr: 0.009907, batch_cost: 0.7854, reader_cost: 0.00015, ips: 1.2732 samples/sec | ETA 08:38:06 2021-05-09 13:43:47 [INFO] [TRAIN] epoch: 2, iter: 430/40000, loss: 0.8935, lr: 0.009904, batch_cost: 0.7850, reader_cost: 0.00017, ips: 1.2739 samples/sec | ETA 08:37:40 2021-05-09 13:43:54 [INFO] [TRAIN] epoch: 2, iter: 440/40000, loss: 0.9854, lr: 0.009902, batch_cost: 0.7865, reader_cost: 0.00018, ips: 1.2715 samples/sec | ETA 08:38:33 2021-05-09 13:44:02 [INFO] [TRAIN] epoch: 2, iter: 450/40000, loss: 0.4145, lr: 0.009900, batch_cost: 0.7867, reader_cost: 0.00017, ips: 1.2711 samples/sec | ETA 08:38:35 2021-05-09 13:44:10 [INFO] [TRAIN] epoch: 2, iter: 460/40000, loss: 0.5945, lr: 0.009898, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2713 samples/sec | ETA 08:38:21 2021-05-09 13:44:18 [INFO] [TRAIN] epoch: 2, iter: 470/40000, loss: 1.4059, lr: 0.009895, batch_cost: 0.7844, reader_cost: 0.00016, ips: 1.2749 samples/sec | ETA 08:36:46 2021-05-09 13:44:26 [INFO] [TRAIN] epoch: 2, iter: 480/40000, loss: 0.5037, lr: 0.009893, batch_cost: 0.7857, reader_cost: 0.00017, ips: 1.2727 samples/sec | ETA 08:37:32 2021-05-09 13:44:34 [INFO] [TRAIN] epoch: 2, iter: 490/40000, loss: 0.5721, lr: 0.009891, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2720 samples/sec | ETA 08:37:40 2021-05-09 13:44:42 [INFO] [TRAIN] epoch: 2, iter: 500/40000, loss: 0.4739, lr: 0.009889, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 08:37:40 2021-05-09 13:44:50 [INFO] [TRAIN] epoch: 2, iter: 510/40000, loss: 1.1487, lr: 0.009887, batch_cost: 0.7884, reader_cost: 0.00015, ips: 1.2684 samples/sec | ETA 08:38:54 2021-05-09 13:44:57 [INFO] [TRAIN] epoch: 2, iter: 520/40000, loss: 0.6326, lr: 0.009884, batch_cost: 0.7873, reader_cost: 0.00017, ips: 1.2701 samples/sec | ETA 08:38:03 2021-05-09 13:45:05 [INFO] [TRAIN] epoch: 2, iter: 530/40000, loss: 0.6623, lr: 0.009882, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2716 samples/sec | ETA 08:37:19 2021-05-09 13:45:13 [INFO] [TRAIN] epoch: 2, iter: 540/40000, loss: 0.6102, lr: 0.009880, batch_cost: 0.7851, reader_cost: 0.00015, ips: 1.2737 samples/sec | ETA 08:36:21 2021-05-09 13:45:21 [INFO] [TRAIN] epoch: 2, iter: 550/40000, loss: 0.6544, lr: 0.009878, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2715 samples/sec | ETA 08:37:06 2021-05-09 13:45:29 [INFO] [TRAIN] epoch: 2, iter: 560/40000, loss: 1.1813, lr: 0.009875, batch_cost: 0.7851, reader_cost: 0.00014, ips: 1.2738 samples/sec | ETA 08:36:02 2021-05-09 13:45:37 [INFO] [TRAIN] epoch: 2, iter: 570/40000, loss: 0.3908, lr: 0.009873, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2729 samples/sec | ETA 08:36:16 2021-05-09 13:45:45 [INFO] [TRAIN] epoch: 2, iter: 580/40000, loss: 0.7230, lr: 0.009871, batch_cost: 0.7849, reader_cost: 0.00015, ips: 1.2741 samples/sec | ETA 08:35:40 2021-05-09 13:45:52 [INFO] [TRAIN] epoch: 2, iter: 590/40000, loss: 0.7734, lr: 0.009869, batch_cost: 0.7854, reader_cost: 0.00015, ips: 1.2733 samples/sec | ETA 08:35:51 2021-05-09 13:46:00 [INFO] [TRAIN] epoch: 2, iter: 600/40000, loss: 0.5652, lr: 0.009866, batch_cost: 0.7858, reader_cost: 0.00015, ips: 1.2726 samples/sec | ETA 08:35:59 2021-05-09 13:46:08 [INFO] [TRAIN] epoch: 2, iter: 610/40000, loss: 0.7547, lr: 0.009864, batch_cost: 0.7849, reader_cost: 0.00016, ips: 1.2740 samples/sec | ETA 08:35:19 2021-05-09 13:46:16 [INFO] [TRAIN] epoch: 2, iter: 620/40000, loss: 0.5793, lr: 0.009862, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2728 samples/sec | ETA 08:35:39 2021-05-09 13:46:24 [INFO] [TRAIN] epoch: 2, iter: 630/40000, loss: 0.4734, lr: 0.009860, batch_cost: 0.7841, reader_cost: 0.00015, ips: 1.2754 samples/sec | ETA 08:34:28 2021-05-09 13:46:32 [INFO] [TRAIN] epoch: 2, iter: 640/40000, loss: 0.5132, lr: 0.009858, batch_cost: 0.7849, reader_cost: 0.00015, ips: 1.2741 samples/sec | ETA 08:34:52 2021-05-09 13:46:39 [INFO] [TRAIN] epoch: 2, iter: 650/40000, loss: 0.7726, lr: 0.009855, batch_cost: 0.7835, reader_cost: 0.00015, ips: 1.2762 samples/sec | ETA 08:33:52 2021-05-09 13:46:47 [INFO] [TRAIN] epoch: 2, iter: 660/40000, loss: 0.6526, lr: 0.009853, batch_cost: 0.7845, reader_cost: 0.00016, ips: 1.2747 samples/sec | ETA 08:34:22 2021-05-09 13:46:55 [INFO] [TRAIN] epoch: 2, iter: 670/40000, loss: 0.7485, lr: 0.009851, batch_cost: 0.7831, reader_cost: 0.00017, ips: 1.2769 samples/sec | ETA 08:33:20 2021-05-09 13:47:03 [INFO] [TRAIN] epoch: 2, iter: 680/40000, loss: 1.0143, lr: 0.009849, batch_cost: 0.7849, reader_cost: 0.00016, ips: 1.2741 samples/sec | ETA 08:34:20 2021-05-09 13:47:11 [INFO] [TRAIN] epoch: 2, iter: 690/40000, loss: 0.8462, lr: 0.009846, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2719 samples/sec | ETA 08:35:07 2021-05-09 13:47:19 [INFO] [TRAIN] epoch: 2, iter: 700/40000, loss: 0.8969, lr: 0.009844, batch_cost: 0.7850, reader_cost: 0.00016, ips: 1.2739 samples/sec | ETA 08:34:09 2021-05-09 13:47:27 [INFO] [TRAIN] epoch: 2, iter: 710/40000, loss: 0.6752, lr: 0.009842, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 08:35:00 2021-05-09 13:47:34 [INFO] [TRAIN] epoch: 2, iter: 720/40000, loss: 0.6503, lr: 0.009840, batch_cost: 0.7843, reader_cost: 0.00015, ips: 1.2751 samples/sec | ETA 08:33:25 2021-05-09 13:47:42 [INFO] [TRAIN] epoch: 2, iter: 730/40000, loss: 1.3158, lr: 0.009837, batch_cost: 0.7845, reader_cost: 0.00015, ips: 1.2747 samples/sec | ETA 08:33:27 2021-05-09 13:47:50 [INFO] [TRAIN] epoch: 2, iter: 740/40000, loss: 0.6861, lr: 0.009835, batch_cost: 0.7849, reader_cost: 0.00012, ips: 1.2741 samples/sec | ETA 08:33:33 2021-05-09 13:48:01 [INFO] [TRAIN] epoch: 3, iter: 750/40000, loss: 0.7218, lr: 0.009833, batch_cost: 1.0689, reader_cost: 0.21815, ips: 0.9355 samples/sec | ETA 11:39:14 2021-05-09 13:48:09 [INFO] [TRAIN] epoch: 3, iter: 760/40000, loss: 0.8491, lr: 0.009831, batch_cost: 0.7984, reader_cost: 0.00031, ips: 1.2525 samples/sec | ETA 08:42:10 2021-05-09 13:48:17 [INFO] [TRAIN] epoch: 3, iter: 770/40000, loss: 0.8339, lr: 0.009829, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2706 samples/sec | ETA 08:34:35 2021-05-09 13:48:25 [INFO] [TRAIN] epoch: 3, iter: 780/40000, loss: 0.6234, lr: 0.009826, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2714 samples/sec | ETA 08:34:08 2021-05-09 13:48:32 [INFO] [TRAIN] epoch: 3, iter: 790/40000, loss: 1.0308, lr: 0.009824, batch_cost: 0.7853, reader_cost: 0.00015, ips: 1.2733 samples/sec | ETA 08:33:12 2021-05-09 13:48:40 [INFO] [TRAIN] epoch: 3, iter: 800/40000, loss: 0.7354, lr: 0.009822, batch_cost: 0.7857, reader_cost: 0.00017, ips: 1.2728 samples/sec | ETA 08:33:18 2021-05-09 13:48:48 [INFO] [TRAIN] epoch: 3, iter: 810/40000, loss: 0.5475, lr: 0.009820, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2732 samples/sec | ETA 08:33:00 2021-05-09 13:48:56 [INFO] [TRAIN] epoch: 3, iter: 820/40000, loss: 0.3746, lr: 0.009817, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2732 samples/sec | ETA 08:32:53 2021-05-09 13:49:04 [INFO] [TRAIN] epoch: 3, iter: 830/40000, loss: 0.6041, lr: 0.009815, batch_cost: 0.7847, reader_cost: 0.00015, ips: 1.2744 samples/sec | ETA 08:32:16 2021-05-09 13:49:12 [INFO] [TRAIN] epoch: 3, iter: 840/40000, loss: 0.8224, lr: 0.009813, batch_cost: 0.7873, reader_cost: 0.00015, ips: 1.2702 samples/sec | ETA 08:33:50 2021-05-09 13:49:20 [INFO] [TRAIN] epoch: 3, iter: 850/40000, loss: 0.5944, lr: 0.009811, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2735 samples/sec | ETA 08:32:21 2021-05-09 13:49:27 [INFO] [TRAIN] epoch: 3, iter: 860/40000, loss: 0.4803, lr: 0.009808, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2736 samples/sec | ETA 08:32:11 2021-05-09 13:49:35 [INFO] [TRAIN] epoch: 3, iter: 870/40000, loss: 0.4437, lr: 0.009806, batch_cost: 0.7874, reader_cost: 0.00016, ips: 1.2701 samples/sec | ETA 08:33:29 2021-05-09 13:49:43 [INFO] [TRAIN] epoch: 3, iter: 880/40000, loss: 0.6650, lr: 0.009804, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2729 samples/sec | ETA 08:32:12 2021-05-09 13:49:51 [INFO] [TRAIN] epoch: 3, iter: 890/40000, loss: 0.6477, lr: 0.009802, batch_cost: 0.7850, reader_cost: 0.00015, ips: 1.2738 samples/sec | ETA 08:31:42 2021-05-09 13:49:59 [INFO] [TRAIN] epoch: 3, iter: 900/40000, loss: 0.7812, lr: 0.009800, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2722 samples/sec | ETA 08:32:14 2021-05-09 13:50:07 [INFO] [TRAIN] epoch: 3, iter: 910/40000, loss: 0.4996, lr: 0.009797, batch_cost: 0.7843, reader_cost: 0.00016, ips: 1.2750 samples/sec | ETA 08:30:58 2021-05-09 13:50:15 [INFO] [TRAIN] epoch: 3, iter: 920/40000, loss: 0.3502, lr: 0.009795, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2736 samples/sec | ETA 08:31:24 2021-05-09 13:50:22 [INFO] [TRAIN] epoch: 3, iter: 930/40000, loss: 0.3321, lr: 0.009793, batch_cost: 0.7853, reader_cost: 0.00014, ips: 1.2733 samples/sec | ETA 08:31:22 2021-05-09 13:50:30 [INFO] [TRAIN] epoch: 3, iter: 940/40000, loss: 0.2595, lr: 0.009791, batch_cost: 0.7849, reader_cost: 0.00014, ips: 1.2741 samples/sec | ETA 08:30:57 2021-05-09 13:50:38 [INFO] [TRAIN] epoch: 3, iter: 950/40000, loss: 0.4187, lr: 0.009788, batch_cost: 0.7854, reader_cost: 0.00017, ips: 1.2732 samples/sec | ETA 08:31:10 2021-05-09 13:50:46 [INFO] [TRAIN] epoch: 3, iter: 960/40000, loss: 0.8463, lr: 0.009786, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 08:31:05 2021-05-09 13:50:54 [INFO] [TRAIN] epoch: 3, iter: 970/40000, loss: 0.5222, lr: 0.009784, batch_cost: 0.7849, reader_cost: 0.00015, ips: 1.2740 samples/sec | ETA 08:30:35 2021-05-09 13:51:02 [INFO] [TRAIN] epoch: 3, iter: 980/40000, loss: 0.7302, lr: 0.009782, batch_cost: 0.7845, reader_cost: 0.00015, ips: 1.2748 samples/sec | ETA 08:30:09 2021-05-09 13:51:09 [INFO] [TRAIN] epoch: 3, iter: 990/40000, loss: 0.5693, lr: 0.009779, batch_cost: 0.7838, reader_cost: 0.00016, ips: 1.2758 samples/sec | ETA 08:29:37 2021-05-09 13:51:17 [INFO] [TRAIN] epoch: 3, iter: 1000/40000, loss: 0.5945, lr: 0.009777, batch_cost: 0.7853, reader_cost: 0.00017, ips: 1.2733 samples/sec | ETA 08:30:28 2021-05-09 13:51:17 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... /ssd1/home/wuzewu/miniconda3/envs/paddle2/lib/python3.7/site-packages/paddle/fluid/dygraph/math_op_patch.py:238: UserWarning: The dtype of left and right variables are not the same, left dtype is VarType.INT32, but right dtype is VarType.BOOL, the right dtype will convert to VarType.INT32 format(lhs_dtype, rhs_dtype, lhs_dtype)) /ssd1/home/wuzewu/miniconda3/envs/paddle2/lib/python3.7/site-packages/paddle/fluid/dygraph/math_op_patch.py:238: UserWarning: The dtype of left and right variables are not the same, left dtype is VarType.INT64, but right dtype is VarType.BOOL, the right dtype will convert to VarType.INT64 format(lhs_dtype, rhs_dtype, lhs_dtype)) 2021-05-09 13:54:46 [INFO] [EVAL] #Images: 500 mIoU: 0.5614 Acc: 0.9272 Kappa: 0.9053 2021-05-09 13:54:46 [INFO] [EVAL] Class IoU: [0.9631 0.7073 0.8748 0.4019 0.3794 0.3682 0.4308 0.5982 0.8861 0.5341 0.9016 0.662 0.1773 0.8681 0.4082 0.498 0.0022 0.3919 0.6139] 2021-05-09 13:54:46 [INFO] [EVAL] Class Acc: [0.9792 0.8211 0.9375 0.9483 0.7394 0.6366 0.6256 0.7851 0.9233 0.7432 0.9359 0.7297 0.7176 0.8849 0.8905 0.6918 0.1401 0.5023 0.7414] 2021-05-09 13:55:27 [INFO] [EVAL] The model with the best validation mIoU (0.5614) was saved at iter 1000. 2021-05-09 13:55:35 [INFO] [TRAIN] epoch: 3, iter: 1010/40000, loss: 0.7650, lr: 0.009775, batch_cost: 0.7807, reader_cost: 0.00040, ips: 1.2809 samples/sec | ETA 08:27:20 2021-05-09 13:55:42 [INFO] [TRAIN] epoch: 3, iter: 1020/40000, loss: 0.4420, lr: 0.009773, batch_cost: 0.7827, reader_cost: 0.00017, ips: 1.2776 samples/sec | ETA 08:28:31 2021-05-09 13:55:50 [INFO] [TRAIN] epoch: 3, iter: 1030/40000, loss: 0.9833, lr: 0.009770, batch_cost: 0.7830, reader_cost: 0.00016, ips: 1.2772 samples/sec | ETA 08:28:32 2021-05-09 13:56:00 [INFO] [TRAIN] epoch: 3, iter: 1040/40000, loss: 0.6252, lr: 0.009768, batch_cost: 0.7848, reader_cost: 0.00017, ips: 1.2743 samples/sec | ETA 08:29:34 2021-05-09 13:56:08 [INFO] [TRAIN] epoch: 3, iter: 1050/40000, loss: 1.0400, lr: 0.009766, batch_cost: 0.7848, reader_cost: 0.00016, ips: 1.2742 samples/sec | ETA 08:29:27 2021-05-09 13:56:16 [INFO] [TRAIN] epoch: 3, iter: 1060/40000, loss: 0.6310, lr: 0.009764, batch_cost: 0.7879, reader_cost: 0.00016, ips: 1.2692 samples/sec | ETA 08:31:20 2021-05-09 13:56:23 [INFO] [TRAIN] epoch: 3, iter: 1070/40000, loss: 0.5206, lr: 0.009762, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2724 samples/sec | ETA 08:29:56 2021-05-09 13:56:31 [INFO] [TRAIN] epoch: 3, iter: 1080/40000, loss: 0.3146, lr: 0.009759, batch_cost: 0.7852, reader_cost: 0.00017, ips: 1.2736 samples/sec | ETA 08:29:19 2021-05-09 13:56:39 [INFO] [TRAIN] epoch: 3, iter: 1090/40000, loss: 0.5785, lr: 0.009757, batch_cost: 0.7851, reader_cost: 0.00016, ips: 1.2737 samples/sec | ETA 08:29:08 2021-05-09 13:56:47 [INFO] [TRAIN] epoch: 3, iter: 1100/40000, loss: 0.5278, lr: 0.009755, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2717 samples/sec | ETA 08:29:48 2021-05-09 13:56:55 [INFO] [TRAIN] epoch: 3, iter: 1110/40000, loss: 0.7081, lr: 0.009753, batch_cost: 0.7864, reader_cost: 0.00012, ips: 1.2716 samples/sec | ETA 08:29:43 2021-05-09 13:57:06 [INFO] [TRAIN] epoch: 4, iter: 1120/40000, loss: 0.7601, lr: 0.009750, batch_cost: 1.0862, reader_cost: 0.23053, ips: 0.9206 samples/sec | ETA 11:43:51 2021-05-09 13:57:14 [INFO] [TRAIN] epoch: 4, iter: 1130/40000, loss: 1.0942, lr: 0.009748, batch_cost: 0.8021, reader_cost: 0.00032, ips: 1.2467 samples/sec | ETA 08:39:38 2021-05-09 13:57:22 [INFO] [TRAIN] epoch: 4, iter: 1140/40000, loss: 1.0691, lr: 0.009746, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2719 samples/sec | ETA 08:29:13 2021-05-09 13:57:30 [INFO] [TRAIN] epoch: 4, iter: 1150/40000, loss: 0.6761, lr: 0.009744, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2713 samples/sec | ETA 08:29:18 2021-05-09 13:57:37 [INFO] [TRAIN] epoch: 4, iter: 1160/40000, loss: 1.0425, lr: 0.009741, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 08:28:37 2021-05-09 13:57:45 [INFO] [TRAIN] epoch: 4, iter: 1170/40000, loss: 0.6614, lr: 0.009739, batch_cost: 0.7874, reader_cost: 0.00017, ips: 1.2701 samples/sec | ETA 08:29:33 2021-05-09 13:57:53 [INFO] [TRAIN] epoch: 4, iter: 1180/40000, loss: 0.4814, lr: 0.009737, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 08:28:13 2021-05-09 13:58:01 [INFO] [TRAIN] epoch: 4, iter: 1190/40000, loss: 0.5660, lr: 0.009735, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2720 samples/sec | ETA 08:28:31 2021-05-09 13:58:09 [INFO] [TRAIN] epoch: 4, iter: 1200/40000, loss: 0.5838, lr: 0.009733, batch_cost: 0.7854, reader_cost: 0.00017, ips: 1.2732 samples/sec | ETA 08:27:53 2021-05-09 13:58:17 [INFO] [TRAIN] epoch: 4, iter: 1210/40000, loss: 0.5981, lr: 0.009730, batch_cost: 0.7867, reader_cost: 0.00014, ips: 1.2712 samples/sec | ETA 08:28:34 2021-05-09 13:58:25 [INFO] [TRAIN] epoch: 4, iter: 1220/40000, loss: 0.5554, lr: 0.009728, batch_cost: 0.7867, reader_cost: 0.00015, ips: 1.2712 samples/sec | ETA 08:28:26 2021-05-09 13:58:32 [INFO] [TRAIN] epoch: 4, iter: 1230/40000, loss: 0.5917, lr: 0.009726, batch_cost: 0.7871, reader_cost: 0.00017, ips: 1.2705 samples/sec | ETA 08:28:35 2021-05-09 13:58:40 [INFO] [TRAIN] epoch: 4, iter: 1240/40000, loss: 0.3234, lr: 0.009724, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 08:28:02 2021-05-09 13:58:48 [INFO] [TRAIN] epoch: 4, iter: 1250/40000, loss: 0.4820, lr: 0.009721, batch_cost: 0.7849, reader_cost: 0.00018, ips: 1.2741 samples/sec | ETA 08:26:53 2021-05-09 13:58:56 [INFO] [TRAIN] epoch: 4, iter: 1260/40000, loss: 0.4522, lr: 0.009719, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2709 samples/sec | ETA 08:28:01 2021-05-09 13:59:04 [INFO] [TRAIN] epoch: 4, iter: 1270/40000, loss: 1.0576, lr: 0.009717, batch_cost: 0.7872, reader_cost: 0.00015, ips: 1.2704 samples/sec | ETA 08:28:07 2021-05-09 13:59:12 [INFO] [TRAIN] epoch: 4, iter: 1280/40000, loss: 0.6060, lr: 0.009715, batch_cost: 0.7880, reader_cost: 0.00015, ips: 1.2691 samples/sec | ETA 08:28:30 2021-05-09 13:59:20 [INFO] [TRAIN] epoch: 4, iter: 1290/40000, loss: 0.3883, lr: 0.009712, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 08:27:32 2021-05-09 13:59:27 [INFO] [TRAIN] epoch: 4, iter: 1300/40000, loss: 0.7953, lr: 0.009710, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2728 samples/sec | ETA 08:26:45 2021-05-09 13:59:35 [INFO] [TRAIN] epoch: 4, iter: 1310/40000, loss: 0.1478, lr: 0.009708, batch_cost: 0.7850, reader_cost: 0.00016, ips: 1.2739 samples/sec | ETA 08:26:12 2021-05-09 13:59:43 [INFO] [TRAIN] epoch: 4, iter: 1320/40000, loss: 0.3679, lr: 0.009706, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2708 samples/sec | ETA 08:27:18 2021-05-09 13:59:51 [INFO] [TRAIN] epoch: 4, iter: 1330/40000, loss: 0.6830, lr: 0.009703, batch_cost: 0.7872, reader_cost: 0.00015, ips: 1.2703 samples/sec | ETA 08:27:21 2021-05-09 13:59:59 [INFO] [TRAIN] epoch: 4, iter: 1340/40000, loss: 0.4461, lr: 0.009701, batch_cost: 0.7847, reader_cost: 0.00016, ips: 1.2744 samples/sec | ETA 08:25:35 2021-05-09 14:00:07 [INFO] [TRAIN] epoch: 4, iter: 1350/40000, loss: 0.4590, lr: 0.009699, batch_cost: 0.7870, reader_cost: 0.00017, ips: 1.2707 samples/sec | ETA 08:26:57 2021-05-09 14:00:15 [INFO] [TRAIN] epoch: 4, iter: 1360/40000, loss: 0.4651, lr: 0.009697, batch_cost: 0.7849, reader_cost: 0.00016, ips: 1.2741 samples/sec | ETA 08:25:28 2021-05-09 14:00:23 [INFO] [TRAIN] epoch: 4, iter: 1370/40000, loss: 0.4701, lr: 0.009695, batch_cost: 0.7878, reader_cost: 0.00015, ips: 1.2694 samples/sec | ETA 08:27:11 2021-05-09 14:00:30 [INFO] [TRAIN] epoch: 4, iter: 1380/40000, loss: 0.3945, lr: 0.009692, batch_cost: 0.7876, reader_cost: 0.00016, ips: 1.2696 samples/sec | ETA 08:26:58 2021-05-09 14:00:38 [INFO] [TRAIN] epoch: 4, iter: 1390/40000, loss: 0.6418, lr: 0.009690, batch_cost: 0.7853, reader_cost: 0.00017, ips: 1.2734 samples/sec | ETA 08:25:19 2021-05-09 14:00:46 [INFO] [TRAIN] epoch: 4, iter: 1400/40000, loss: 0.8923, lr: 0.009688, batch_cost: 0.7853, reader_cost: 0.00017, ips: 1.2734 samples/sec | ETA 08:25:13 2021-05-09 14:00:54 [INFO] [TRAIN] epoch: 4, iter: 1410/40000, loss: 0.5084, lr: 0.009686, batch_cost: 0.7863, reader_cost: 0.00017, ips: 1.2718 samples/sec | ETA 08:25:41 2021-05-09 14:01:02 [INFO] [TRAIN] epoch: 4, iter: 1420/40000, loss: 0.8277, lr: 0.009683, batch_cost: 0.7851, reader_cost: 0.00015, ips: 1.2738 samples/sec | ETA 08:24:48 2021-05-09 14:01:10 [INFO] [TRAIN] epoch: 4, iter: 1430/40000, loss: 0.7445, lr: 0.009681, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2709 samples/sec | ETA 08:25:48 2021-05-09 14:01:18 [INFO] [TRAIN] epoch: 4, iter: 1440/40000, loss: 0.7451, lr: 0.009679, batch_cost: 0.7867, reader_cost: 0.00015, ips: 1.2711 samples/sec | ETA 08:25:36 2021-05-09 14:01:25 [INFO] [TRAIN] epoch: 4, iter: 1450/40000, loss: 0.2493, lr: 0.009677, batch_cost: 0.7869, reader_cost: 0.00015, ips: 1.2708 samples/sec | ETA 08:25:36 2021-05-09 14:01:33 [INFO] [TRAIN] epoch: 4, iter: 1460/40000, loss: 0.5377, lr: 0.009674, batch_cost: 0.7852, reader_cost: 0.00014, ips: 1.2736 samples/sec | ETA 08:24:20 2021-05-09 14:01:41 [INFO] [TRAIN] epoch: 4, iter: 1470/40000, loss: 0.6925, lr: 0.009672, batch_cost: 0.7862, reader_cost: 0.00014, ips: 1.2720 samples/sec | ETA 08:24:50 2021-05-09 14:01:49 [INFO] [TRAIN] epoch: 4, iter: 1480/40000, loss: 0.5497, lr: 0.009670, batch_cost: 0.7850, reader_cost: 0.00012, ips: 1.2738 samples/sec | ETA 08:23:59 2021-05-09 14:02:00 [INFO] [TRAIN] epoch: 5, iter: 1490/40000, loss: 0.6462, lr: 0.009668, batch_cost: 1.0664, reader_cost: 0.22526, ips: 0.9378 samples/sec | ETA 11:24:25 2021-05-09 14:02:08 [INFO] [TRAIN] epoch: 5, iter: 1500/40000, loss: 0.5342, lr: 0.009665, batch_cost: 0.7962, reader_cost: 0.00031, ips: 1.2560 samples/sec | ETA 08:30:53 2021-05-09 14:02:15 [INFO] [TRAIN] epoch: 5, iter: 1510/40000, loss: 0.7697, lr: 0.009663, batch_cost: 0.7848, reader_cost: 0.00017, ips: 1.2742 samples/sec | ETA 08:23:26 2021-05-09 14:02:23 [INFO] [TRAIN] epoch: 5, iter: 1520/40000, loss: 0.6811, lr: 0.009661, batch_cost: 0.7871, reader_cost: 0.00017, ips: 1.2704 samples/sec | ETA 08:24:48 2021-05-09 14:02:31 [INFO] [TRAIN] epoch: 5, iter: 1530/40000, loss: 0.7882, lr: 0.009659, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2729 samples/sec | ETA 08:23:43 2021-05-09 14:02:39 [INFO] [TRAIN] epoch: 5, iter: 1540/40000, loss: 0.6788, lr: 0.009657, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2730 samples/sec | ETA 08:23:32 2021-05-09 14:02:47 [INFO] [TRAIN] epoch: 5, iter: 1550/40000, loss: 0.4597, lr: 0.009654, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 08:23:46 2021-05-09 14:02:55 [INFO] [TRAIN] epoch: 5, iter: 1560/40000, loss: 0.2727, lr: 0.009652, batch_cost: 0.7869, reader_cost: 0.00017, ips: 1.2708 samples/sec | ETA 08:24:09 2021-05-09 14:03:03 [INFO] [TRAIN] epoch: 5, iter: 1570/40000, loss: 0.6362, lr: 0.009650, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2721 samples/sec | ETA 08:23:28 2021-05-09 14:03:11 [INFO] [TRAIN] epoch: 5, iter: 1580/40000, loss: 0.5945, lr: 0.009648, batch_cost: 0.7871, reader_cost: 0.00015, ips: 1.2704 samples/sec | ETA 08:24:01 2021-05-09 14:03:18 [INFO] [TRAIN] epoch: 5, iter: 1590/40000, loss: 0.5805, lr: 0.009645, batch_cost: 0.7862, reader_cost: 0.00014, ips: 1.2719 samples/sec | ETA 08:23:19 2021-05-09 14:03:26 [INFO] [TRAIN] epoch: 5, iter: 1600/40000, loss: 0.4602, lr: 0.009643, batch_cost: 0.7851, reader_cost: 0.00017, ips: 1.2738 samples/sec | ETA 08:22:27 2021-05-09 14:03:34 [INFO] [TRAIN] epoch: 5, iter: 1610/40000, loss: 0.2368, lr: 0.009641, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2725 samples/sec | ETA 08:22:49 2021-05-09 14:03:42 [INFO] [TRAIN] epoch: 5, iter: 1620/40000, loss: 0.5052, lr: 0.009639, batch_cost: 0.7882, reader_cost: 0.00016, ips: 1.2687 samples/sec | ETA 08:24:10 2021-05-09 14:03:50 [INFO] [TRAIN] epoch: 5, iter: 1630/40000, loss: 0.5007, lr: 0.009636, batch_cost: 0.7858, reader_cost: 0.00015, ips: 1.2726 samples/sec | ETA 08:22:31 2021-05-09 14:03:58 [INFO] [TRAIN] epoch: 5, iter: 1640/40000, loss: 0.5320, lr: 0.009634, batch_cost: 0.7876, reader_cost: 0.00015, ips: 1.2697 samples/sec | ETA 08:23:31 2021-05-09 14:04:06 [INFO] [TRAIN] epoch: 5, iter: 1650/40000, loss: 0.5258, lr: 0.009632, batch_cost: 0.7853, reader_cost: 0.00014, ips: 1.2735 samples/sec | ETA 08:21:54 2021-05-09 14:04:13 [INFO] [TRAIN] epoch: 5, iter: 1660/40000, loss: 0.4187, lr: 0.009630, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2718 samples/sec | ETA 08:22:26 2021-05-09 14:04:21 [INFO] [TRAIN] epoch: 5, iter: 1670/40000, loss: 0.5642, lr: 0.009627, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 08:22:24 2021-05-09 14:04:29 [INFO] [TRAIN] epoch: 5, iter: 1680/40000, loss: 0.5262, lr: 0.009625, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2718 samples/sec | ETA 08:22:10 2021-05-09 14:04:37 [INFO] [TRAIN] epoch: 5, iter: 1690/40000, loss: 0.2161, lr: 0.009623, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2729 samples/sec | ETA 08:21:36 2021-05-09 14:04:45 [INFO] [TRAIN] epoch: 5, iter: 1700/40000, loss: 0.7001, lr: 0.009621, batch_cost: 0.7851, reader_cost: 0.00014, ips: 1.2737 samples/sec | ETA 08:21:09 2021-05-09 14:04:53 [INFO] [TRAIN] epoch: 5, iter: 1710/40000, loss: 0.3666, lr: 0.009618, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2714 samples/sec | ETA 08:21:55 2021-05-09 14:05:01 [INFO] [TRAIN] epoch: 5, iter: 1720/40000, loss: 0.6229, lr: 0.009616, batch_cost: 0.7857, reader_cost: 0.00017, ips: 1.2728 samples/sec | ETA 08:21:15 2021-05-09 14:05:08 [INFO] [TRAIN] epoch: 5, iter: 1730/40000, loss: 0.5268, lr: 0.009614, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2731 samples/sec | ETA 08:20:59 2021-05-09 14:05:16 [INFO] [TRAIN] epoch: 5, iter: 1740/40000, loss: 0.4558, lr: 0.009612, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2719 samples/sec | ETA 08:21:20 2021-05-09 14:05:24 [INFO] [TRAIN] epoch: 5, iter: 1750/40000, loss: 0.3190, lr: 0.009610, batch_cost: 0.7853, reader_cost: 0.00014, ips: 1.2734 samples/sec | ETA 08:20:38 2021-05-09 14:05:32 [INFO] [TRAIN] epoch: 5, iter: 1760/40000, loss: 0.4842, lr: 0.009607, batch_cost: 0.7858, reader_cost: 0.00014, ips: 1.2725 samples/sec | ETA 08:20:50 2021-05-09 14:05:40 [INFO] [TRAIN] epoch: 5, iter: 1770/40000, loss: 0.8168, lr: 0.009605, batch_cost: 0.7846, reader_cost: 0.00016, ips: 1.2745 samples/sec | ETA 08:19:55 2021-05-09 14:05:48 [INFO] [TRAIN] epoch: 5, iter: 1780/40000, loss: 0.8482, lr: 0.009603, batch_cost: 0.7877, reader_cost: 0.00015, ips: 1.2695 samples/sec | ETA 08:21:45 2021-05-09 14:05:56 [INFO] [TRAIN] epoch: 5, iter: 1790/40000, loss: 0.9387, lr: 0.009601, batch_cost: 0.7843, reader_cost: 0.00015, ips: 1.2750 samples/sec | ETA 08:19:29 2021-05-09 14:06:03 [INFO] [TRAIN] epoch: 5, iter: 1800/40000, loss: 0.6949, lr: 0.009598, batch_cost: 0.7848, reader_cost: 0.00015, ips: 1.2742 samples/sec | ETA 08:19:40 2021-05-09 14:06:11 [INFO] [TRAIN] epoch: 5, iter: 1810/40000, loss: 0.4234, lr: 0.009596, batch_cost: 0.7865, reader_cost: 0.00014, ips: 1.2715 samples/sec | ETA 08:20:35 2021-05-09 14:06:19 [INFO] [TRAIN] epoch: 5, iter: 1820/40000, loss: 0.4750, lr: 0.009594, batch_cost: 0.7846, reader_cost: 0.00014, ips: 1.2745 samples/sec | ETA 08:19:17 2021-05-09 14:06:27 [INFO] [TRAIN] epoch: 5, iter: 1830/40000, loss: 1.1616, lr: 0.009592, batch_cost: 0.7852, reader_cost: 0.00017, ips: 1.2736 samples/sec | ETA 08:19:29 2021-05-09 14:06:35 [INFO] [TRAIN] epoch: 5, iter: 1840/40000, loss: 0.6714, lr: 0.009589, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2728 samples/sec | ETA 08:19:40 2021-05-09 14:06:43 [INFO] [TRAIN] epoch: 5, iter: 1850/40000, loss: 0.8141, lr: 0.009587, batch_cost: 0.7860, reader_cost: 0.00017, ips: 1.2723 samples/sec | ETA 08:19:45 2021-05-09 14:06:51 [INFO] [TRAIN] epoch: 5, iter: 1860/40000, loss: 0.8025, lr: 0.009585, batch_cost: 0.7841, reader_cost: 0.00011, ips: 1.2754 samples/sec | ETA 08:18:23 2021-05-09 14:07:01 [INFO] [TRAIN] epoch: 6, iter: 1870/40000, loss: 0.5304, lr: 0.009583, batch_cost: 1.0850, reader_cost: 0.24868, ips: 0.9216 samples/sec | ETA 11:29:31 2021-05-09 14:07:09 [INFO] [TRAIN] epoch: 6, iter: 1880/40000, loss: 0.8618, lr: 0.009580, batch_cost: 0.7874, reader_cost: 0.00033, ips: 1.2701 samples/sec | ETA 08:20:13 2021-05-09 14:07:17 [INFO] [TRAIN] epoch: 6, iter: 1890/40000, loss: 1.0165, lr: 0.009578, batch_cost: 0.7853, reader_cost: 0.00015, ips: 1.2733 samples/sec | ETA 08:18:49 2021-05-09 14:07:25 [INFO] [TRAIN] epoch: 6, iter: 1900/40000, loss: 0.7382, lr: 0.009576, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2720 samples/sec | ETA 08:19:12 2021-05-09 14:07:33 [INFO] [TRAIN] epoch: 6, iter: 1910/40000, loss: 1.0482, lr: 0.009574, batch_cost: 0.7874, reader_cost: 0.00016, ips: 1.2701 samples/sec | ETA 08:19:50 2021-05-09 14:07:41 [INFO] [TRAIN] epoch: 6, iter: 1920/40000, loss: 0.5612, lr: 0.009571, batch_cost: 0.7860, reader_cost: 0.00017, ips: 1.2723 samples/sec | ETA 08:18:50 2021-05-09 14:07:49 [INFO] [TRAIN] epoch: 6, iter: 1930/40000, loss: 0.4244, lr: 0.009569, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2726 samples/sec | ETA 08:18:34 2021-05-09 14:07:56 [INFO] [TRAIN] epoch: 6, iter: 1940/40000, loss: 0.4103, lr: 0.009567, batch_cost: 0.7878, reader_cost: 0.00016, ips: 1.2693 samples/sec | ETA 08:19:45 2021-05-09 14:08:04 [INFO] [TRAIN] epoch: 6, iter: 1950/40000, loss: 0.5231, lr: 0.009565, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 08:18:07 2021-05-09 14:08:12 [INFO] [TRAIN] epoch: 6, iter: 1960/40000, loss: 0.3635, lr: 0.009563, batch_cost: 0.7846, reader_cost: 0.00016, ips: 1.2745 samples/sec | ETA 08:17:27 2021-05-09 14:08:20 [INFO] [TRAIN] epoch: 6, iter: 1970/40000, loss: 0.6916, lr: 0.009560, batch_cost: 0.7851, reader_cost: 0.00014, ips: 1.2737 samples/sec | ETA 08:17:37 2021-05-09 14:08:28 [INFO] [TRAIN] epoch: 6, iter: 1980/40000, loss: 0.2045, lr: 0.009558, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2725 samples/sec | ETA 08:17:58 2021-05-09 14:08:36 [INFO] [TRAIN] epoch: 6, iter: 1990/40000, loss: 0.4836, lr: 0.009556, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2711 samples/sec | ETA 08:18:23 2021-05-09 14:08:44 [INFO] [TRAIN] epoch: 6, iter: 2000/40000, loss: 0.5452, lr: 0.009554, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2726 samples/sec | ETA 08:17:39 2021-05-09 14:08:44 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-09 14:12:12 [INFO] [EVAL] #Images: 500 mIoU: 0.6300 Acc: 0.9319 Kappa: 0.9112 2021-05-09 14:12:12 [INFO] [EVAL] Class IoU: [0.964 0.7312 0.8752 0.5175 0.4279 0.3184 0.4052 0.5933 0.8865 0.5547 0.918 0.6912 0.4158 0.8911 0.5823 0.6073 0.6136 0.349 0.6281] 2021-05-09 14:12:12 [INFO] [EVAL] Class Acc: [0.9762 0.8662 0.9063 0.7742 0.7018 0.7451 0.7316 0.8639 0.9427 0.6986 0.9592 0.833 0.6097 0.9442 0.673 0.8368 0.749 0.8496 0.7381] 2021-05-09 14:12:59 [INFO] [EVAL] The model with the best validation mIoU (0.6300) was saved at iter 2000. 2021-05-09 14:13:07 [INFO] [TRAIN] epoch: 6, iter: 2010/40000, loss: 0.3924, lr: 0.009551, batch_cost: 0.7833, reader_cost: 0.00024, ips: 1.2767 samples/sec | ETA 08:15:56 2021-05-09 14:13:15 [INFO] [TRAIN] epoch: 6, iter: 2020/40000, loss: 0.6039, lr: 0.009549, batch_cost: 0.7838, reader_cost: 0.00016, ips: 1.2758 samples/sec | ETA 08:16:08 2021-05-09 14:13:23 [INFO] [TRAIN] epoch: 6, iter: 2030/40000, loss: 0.5498, lr: 0.009547, batch_cost: 0.7849, reader_cost: 0.00015, ips: 1.2741 samples/sec | ETA 08:16:40 2021-05-09 14:13:31 [INFO] [TRAIN] epoch: 6, iter: 2040/40000, loss: 0.4134, lr: 0.009545, batch_cost: 0.7856, reader_cost: 0.00017, ips: 1.2729 samples/sec | ETA 08:17:02 2021-05-09 14:13:39 [INFO] [TRAIN] epoch: 6, iter: 2050/40000, loss: 0.3705, lr: 0.009542, batch_cost: 0.7863, reader_cost: 0.00017, ips: 1.2718 samples/sec | ETA 08:17:19 2021-05-09 14:13:47 [INFO] [TRAIN] epoch: 6, iter: 2060/40000, loss: 0.7891, lr: 0.009540, batch_cost: 0.7863, reader_cost: 0.00017, ips: 1.2718 samples/sec | ETA 08:17:11 2021-05-09 14:13:54 [INFO] [TRAIN] epoch: 6, iter: 2070/40000, loss: 0.6447, lr: 0.009538, batch_cost: 0.7849, reader_cost: 0.00016, ips: 1.2741 samples/sec | ETA 08:16:10 2021-05-09 14:14:02 [INFO] [TRAIN] epoch: 6, iter: 2080/40000, loss: 0.5160, lr: 0.009536, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 08:16:47 2021-05-09 14:14:10 [INFO] [TRAIN] epoch: 6, iter: 2090/40000, loss: 0.3282, lr: 0.009533, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2718 samples/sec | ETA 08:16:46 2021-05-09 14:14:18 [INFO] [TRAIN] epoch: 6, iter: 2100/40000, loss: 0.4034, lr: 0.009531, batch_cost: 0.7858, reader_cost: 0.00015, ips: 1.2726 samples/sec | ETA 08:16:20 2021-05-09 14:14:26 [INFO] [TRAIN] epoch: 6, iter: 2110/40000, loss: 0.3782, lr: 0.009529, batch_cost: 0.7877, reader_cost: 0.00015, ips: 1.2695 samples/sec | ETA 08:17:25 2021-05-09 14:14:34 [INFO] [TRAIN] epoch: 6, iter: 2120/40000, loss: 0.3821, lr: 0.009527, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2717 samples/sec | ETA 08:16:28 2021-05-09 14:14:42 [INFO] [TRAIN] epoch: 6, iter: 2130/40000, loss: 0.5359, lr: 0.009524, batch_cost: 0.7871, reader_cost: 0.00015, ips: 1.2704 samples/sec | ETA 08:16:48 2021-05-09 14:14:49 [INFO] [TRAIN] epoch: 6, iter: 2140/40000, loss: 0.6344, lr: 0.009522, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 08:16:22 2021-05-09 14:14:57 [INFO] [TRAIN] epoch: 6, iter: 2150/40000, loss: 0.6232, lr: 0.009520, batch_cost: 0.7863, reader_cost: 0.00017, ips: 1.2718 samples/sec | ETA 08:16:01 2021-05-09 14:15:05 [INFO] [TRAIN] epoch: 6, iter: 2160/40000, loss: 0.9161, lr: 0.009518, batch_cost: 0.7877, reader_cost: 0.00016, ips: 1.2696 samples/sec | ETA 08:16:45 2021-05-09 14:15:13 [INFO] [TRAIN] epoch: 6, iter: 2170/40000, loss: 1.1412, lr: 0.009516, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2714 samples/sec | ETA 08:15:55 2021-05-09 14:15:21 [INFO] [TRAIN] epoch: 6, iter: 2180/40000, loss: 0.5831, lr: 0.009513, batch_cost: 0.7867, reader_cost: 0.00015, ips: 1.2711 samples/sec | ETA 08:15:52 2021-05-09 14:15:29 [INFO] [TRAIN] epoch: 6, iter: 2190/40000, loss: 0.2726, lr: 0.009511, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2719 samples/sec | ETA 08:15:27 2021-05-09 14:15:37 [INFO] [TRAIN] epoch: 6, iter: 2200/40000, loss: 0.5937, lr: 0.009509, batch_cost: 0.7882, reader_cost: 0.00017, ips: 1.2688 samples/sec | ETA 08:16:32 2021-05-09 14:15:45 [INFO] [TRAIN] epoch: 6, iter: 2210/40000, loss: 0.8585, lr: 0.009507, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 08:15:06 2021-05-09 14:15:52 [INFO] [TRAIN] epoch: 6, iter: 2220/40000, loss: 0.7916, lr: 0.009504, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 08:15:20 2021-05-09 14:16:00 [INFO] [TRAIN] epoch: 6, iter: 2230/40000, loss: 0.5556, lr: 0.009502, batch_cost: 0.7846, reader_cost: 0.00027, ips: 1.2746 samples/sec | ETA 08:13:52 2021-05-09 14:16:11 [INFO] [TRAIN] epoch: 7, iter: 2240/40000, loss: 1.0088, lr: 0.009500, batch_cost: 1.0818, reader_cost: 0.27150, ips: 0.9244 samples/sec | ETA 11:20:49 2021-05-09 14:16:19 [INFO] [TRAIN] epoch: 7, iter: 2250/40000, loss: 0.7107, lr: 0.009498, batch_cost: 0.7922, reader_cost: 0.00033, ips: 1.2624 samples/sec | ETA 08:18:24 2021-05-09 14:16:27 [INFO] [TRAIN] epoch: 7, iter: 2260/40000, loss: 0.6774, lr: 0.009495, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2716 samples/sec | ETA 08:14:38 2021-05-09 14:16:35 [INFO] [TRAIN] epoch: 7, iter: 2270/40000, loss: 0.8200, lr: 0.009493, batch_cost: 0.7875, reader_cost: 0.00015, ips: 1.2699 samples/sec | ETA 08:15:11 2021-05-09 14:16:43 [INFO] [TRAIN] epoch: 7, iter: 2280/40000, loss: 0.9484, lr: 0.009491, batch_cost: 0.7872, reader_cost: 0.00014, ips: 1.2703 samples/sec | ETA 08:14:54 2021-05-09 14:16:50 [INFO] [TRAIN] epoch: 7, iter: 2290/40000, loss: 0.4687, lr: 0.009489, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2710 samples/sec | ETA 08:14:28 2021-05-09 14:16:58 [INFO] [TRAIN] epoch: 7, iter: 2300/40000, loss: 1.0702, lr: 0.009486, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2726 samples/sec | ETA 08:13:43 2021-05-09 14:17:06 [INFO] [TRAIN] epoch: 7, iter: 2310/40000, loss: 0.4358, lr: 0.009484, batch_cost: 0.7875, reader_cost: 0.00016, ips: 1.2698 samples/sec | ETA 08:14:40 2021-05-09 14:17:14 [INFO] [TRAIN] epoch: 7, iter: 2320/40000, loss: 0.5636, lr: 0.009482, batch_cost: 0.7871, reader_cost: 0.00015, ips: 1.2706 samples/sec | ETA 08:14:16 2021-05-09 14:17:22 [INFO] [TRAIN] epoch: 7, iter: 2330/40000, loss: 0.5715, lr: 0.009480, batch_cost: 0.7892, reader_cost: 0.00016, ips: 1.2671 samples/sec | ETA 08:15:28 2021-05-09 14:17:30 [INFO] [TRAIN] epoch: 7, iter: 2340/40000, loss: 0.5374, lr: 0.009477, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2709 samples/sec | ETA 08:13:53 2021-05-09 14:17:38 [INFO] [TRAIN] epoch: 7, iter: 2350/40000, loss: 0.3108, lr: 0.009475, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2723 samples/sec | ETA 08:13:12 2021-05-09 14:17:46 [INFO] [TRAIN] epoch: 7, iter: 2360/40000, loss: 0.3102, lr: 0.009473, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2714 samples/sec | ETA 08:13:25 2021-05-09 14:17:53 [INFO] [TRAIN] epoch: 7, iter: 2370/40000, loss: 0.9714, lr: 0.009471, batch_cost: 0.7873, reader_cost: 0.00015, ips: 1.2702 samples/sec | ETA 08:13:44 2021-05-09 14:18:01 [INFO] [TRAIN] epoch: 7, iter: 2380/40000, loss: 0.4351, lr: 0.009468, batch_cost: 0.7873, reader_cost: 0.00016, ips: 1.2702 samples/sec | ETA 08:13:37 2021-05-09 14:18:09 [INFO] [TRAIN] epoch: 7, iter: 2390/40000, loss: 0.4801, lr: 0.009466, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2725 samples/sec | ETA 08:12:34 2021-05-09 14:18:17 [INFO] [TRAIN] epoch: 7, iter: 2400/40000, loss: 0.3911, lr: 0.009464, batch_cost: 0.7871, reader_cost: 0.00015, ips: 1.2704 samples/sec | ETA 08:13:16 2021-05-09 14:18:25 [INFO] [TRAIN] epoch: 7, iter: 2410/40000, loss: 0.9081, lr: 0.009462, batch_cost: 0.7848, reader_cost: 0.00016, ips: 1.2742 samples/sec | ETA 08:11:41 2021-05-09 14:18:33 [INFO] [TRAIN] epoch: 7, iter: 2420/40000, loss: 0.5077, lr: 0.009460, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2722 samples/sec | ETA 08:12:20 2021-05-09 14:18:41 [INFO] [TRAIN] epoch: 7, iter: 2430/40000, loss: 0.2905, lr: 0.009457, batch_cost: 0.7864, reader_cost: 0.00017, ips: 1.2716 samples/sec | ETA 08:12:24 2021-05-09 14:18:48 [INFO] [TRAIN] epoch: 7, iter: 2440/40000, loss: 0.4304, lr: 0.009455, batch_cost: 0.7854, reader_cost: 0.00018, ips: 1.2732 samples/sec | ETA 08:11:41 2021-05-09 14:18:56 [INFO] [TRAIN] epoch: 7, iter: 2450/40000, loss: 0.5300, lr: 0.009453, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2736 samples/sec | ETA 08:11:22 2021-05-09 14:19:04 [INFO] [TRAIN] epoch: 7, iter: 2460/40000, loss: 0.3442, lr: 0.009451, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2718 samples/sec | ETA 08:11:57 2021-05-09 14:19:12 [INFO] [TRAIN] epoch: 7, iter: 2470/40000, loss: 0.6420, lr: 0.009448, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2730 samples/sec | ETA 08:11:21 2021-05-09 14:19:20 [INFO] [TRAIN] epoch: 7, iter: 2480/40000, loss: 0.4464, lr: 0.009446, batch_cost: 0.7848, reader_cost: 0.00014, ips: 1.2742 samples/sec | ETA 08:10:45 2021-05-09 14:19:28 [INFO] [TRAIN] epoch: 7, iter: 2490/40000, loss: 0.6211, lr: 0.009444, batch_cost: 0.7864, reader_cost: 0.00017, ips: 1.2717 samples/sec | ETA 08:11:36 2021-05-09 14:19:36 [INFO] [TRAIN] epoch: 7, iter: 2500/40000, loss: 1.2459, lr: 0.009442, batch_cost: 0.7851, reader_cost: 0.00015, ips: 1.2736 samples/sec | ETA 08:10:42 2021-05-09 14:19:43 [INFO] [TRAIN] epoch: 7, iter: 2510/40000, loss: 0.6742, lr: 0.009439, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2720 samples/sec | ETA 08:11:13 2021-05-09 14:19:51 [INFO] [TRAIN] epoch: 7, iter: 2520/40000, loss: 0.9163, lr: 0.009437, batch_cost: 0.7859, reader_cost: 0.00014, ips: 1.2724 samples/sec | ETA 08:10:55 2021-05-09 14:19:59 [INFO] [TRAIN] epoch: 7, iter: 2530/40000, loss: 0.5894, lr: 0.009435, batch_cost: 0.7848, reader_cost: 0.00015, ips: 1.2742 samples/sec | ETA 08:10:07 2021-05-09 14:20:07 [INFO] [TRAIN] epoch: 7, iter: 2540/40000, loss: 0.6547, lr: 0.009433, batch_cost: 0.7848, reader_cost: 0.00016, ips: 1.2742 samples/sec | ETA 08:09:59 2021-05-09 14:20:15 [INFO] [TRAIN] epoch: 7, iter: 2550/40000, loss: 0.5763, lr: 0.009430, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2729 samples/sec | ETA 08:10:20 2021-05-09 14:20:23 [INFO] [TRAIN] epoch: 7, iter: 2560/40000, loss: 0.3519, lr: 0.009428, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2713 samples/sec | ETA 08:10:49 2021-05-09 14:20:31 [INFO] [TRAIN] epoch: 7, iter: 2570/40000, loss: 0.4251, lr: 0.009426, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2723 samples/sec | ETA 08:10:20 2021-05-09 14:20:38 [INFO] [TRAIN] epoch: 7, iter: 2580/40000, loss: 0.3408, lr: 0.009424, batch_cost: 0.7862, reader_cost: 0.00017, ips: 1.2719 samples/sec | ETA 08:10:20 2021-05-09 14:20:46 [INFO] [TRAIN] epoch: 7, iter: 2590/40000, loss: 0.7738, lr: 0.009421, batch_cost: 0.7876, reader_cost: 0.00015, ips: 1.2697 samples/sec | ETA 08:11:04 2021-05-09 14:20:54 [INFO] [TRAIN] epoch: 7, iter: 2600/40000, loss: 0.5566, lr: 0.009419, batch_cost: 0.7845, reader_cost: 0.00012, ips: 1.2748 samples/sec | ETA 08:08:58 2021-05-09 14:21:05 [INFO] [TRAIN] epoch: 8, iter: 2610/40000, loss: 0.4720, lr: 0.009417, batch_cost: 1.0693, reader_cost: 0.27934, ips: 0.9352 samples/sec | ETA 11:06:19 2021-05-09 14:21:13 [INFO] [TRAIN] epoch: 8, iter: 2620/40000, loss: 0.4451, lr: 0.009415, batch_cost: 0.8017, reader_cost: 0.00037, ips: 1.2473 samples/sec | ETA 08:19:28 2021-05-09 14:21:21 [INFO] [TRAIN] epoch: 8, iter: 2630/40000, loss: 0.5359, lr: 0.009412, batch_cost: 0.7894, reader_cost: 0.00015, ips: 1.2669 samples/sec | ETA 08:11:38 2021-05-09 14:21:29 [INFO] [TRAIN] epoch: 8, iter: 2640/40000, loss: 0.5374, lr: 0.009410, batch_cost: 0.7869, reader_cost: 0.00015, ips: 1.2709 samples/sec | ETA 08:09:56 2021-05-09 14:21:37 [INFO] [TRAIN] epoch: 8, iter: 2650/40000, loss: 0.6592, lr: 0.009408, batch_cost: 0.7853, reader_cost: 0.00015, ips: 1.2734 samples/sec | ETA 08:08:51 2021-05-09 14:21:44 [INFO] [TRAIN] epoch: 8, iter: 2660/40000, loss: 0.5184, lr: 0.009406, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2709 samples/sec | ETA 08:09:39 2021-05-09 14:21:52 [INFO] [TRAIN] epoch: 8, iter: 2670/40000, loss: 0.3500, lr: 0.009403, batch_cost: 0.7867, reader_cost: 0.00015, ips: 1.2712 samples/sec | ETA 08:09:27 2021-05-09 14:22:00 [INFO] [TRAIN] epoch: 8, iter: 2680/40000, loss: 0.1858, lr: 0.009401, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2719 samples/sec | ETA 08:09:01 2021-05-09 14:22:08 [INFO] [TRAIN] epoch: 8, iter: 2690/40000, loss: 0.4873, lr: 0.009399, batch_cost: 0.7873, reader_cost: 0.00016, ips: 1.2701 samples/sec | ETA 08:09:35 2021-05-09 14:22:16 [INFO] [TRAIN] epoch: 8, iter: 2700/40000, loss: 0.4489, lr: 0.009397, batch_cost: 0.7884, reader_cost: 0.00015, ips: 1.2683 samples/sec | ETA 08:10:08 2021-05-09 14:22:24 [INFO] [TRAIN] epoch: 8, iter: 2710/40000, loss: 0.5206, lr: 0.009394, batch_cost: 0.7878, reader_cost: 0.00015, ips: 1.2694 samples/sec | ETA 08:09:35 2021-05-09 14:22:32 [INFO] [TRAIN] epoch: 8, iter: 2720/40000, loss: 0.4850, lr: 0.009392, batch_cost: 0.7876, reader_cost: 0.00015, ips: 1.2697 samples/sec | ETA 08:09:21 2021-05-09 14:22:40 [INFO] [TRAIN] epoch: 8, iter: 2730/40000, loss: 0.3045, lr: 0.009390, batch_cost: 0.7877, reader_cost: 0.00015, ips: 1.2696 samples/sec | ETA 08:09:15 2021-05-09 14:22:47 [INFO] [TRAIN] epoch: 8, iter: 2740/40000, loss: 0.6978, lr: 0.009388, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 08:07:57 2021-05-09 14:22:55 [INFO] [TRAIN] epoch: 8, iter: 2750/40000, loss: 0.3669, lr: 0.009386, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2715 samples/sec | ETA 08:08:16 2021-05-09 14:23:03 [INFO] [TRAIN] epoch: 8, iter: 2760/40000, loss: 0.6330, lr: 0.009383, batch_cost: 0.7865, reader_cost: 0.00014, ips: 1.2715 samples/sec | ETA 08:08:09 2021-05-09 14:23:11 [INFO] [TRAIN] epoch: 8, iter: 2770/40000, loss: 0.3510, lr: 0.009381, batch_cost: 0.7859, reader_cost: 0.00014, ips: 1.2724 samples/sec | ETA 08:07:39 2021-05-09 14:23:19 [INFO] [TRAIN] epoch: 8, iter: 2780/40000, loss: 0.4625, lr: 0.009379, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2724 samples/sec | ETA 08:07:31 2021-05-09 14:23:27 [INFO] [TRAIN] epoch: 8, iter: 2790/40000, loss: 0.3949, lr: 0.009377, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2714 samples/sec | ETA 08:07:47 2021-05-09 14:23:35 [INFO] [TRAIN] epoch: 8, iter: 2800/40000, loss: 0.2152, lr: 0.009374, batch_cost: 0.7878, reader_cost: 0.00016, ips: 1.2693 samples/sec | ETA 08:08:26 2021-05-09 14:23:42 [INFO] [TRAIN] epoch: 8, iter: 2810/40000, loss: 0.5139, lr: 0.009372, batch_cost: 0.7886, reader_cost: 0.00016, ips: 1.2680 samples/sec | ETA 08:08:49 2021-05-09 14:23:50 [INFO] [TRAIN] epoch: 8, iter: 2820/40000, loss: 0.4664, lr: 0.009370, batch_cost: 0.7873, reader_cost: 0.00017, ips: 1.2702 samples/sec | ETA 08:07:51 2021-05-09 14:23:58 [INFO] [TRAIN] epoch: 8, iter: 2830/40000, loss: 0.5709, lr: 0.009368, batch_cost: 0.7860, reader_cost: 0.00018, ips: 1.2722 samples/sec | ETA 08:06:56 2021-05-09 14:24:06 [INFO] [TRAIN] epoch: 8, iter: 2840/40000, loss: 0.3426, lr: 0.009365, batch_cost: 0.7876, reader_cost: 0.00016, ips: 1.2697 samples/sec | ETA 08:07:47 2021-05-09 14:24:14 [INFO] [TRAIN] epoch: 8, iter: 2850/40000, loss: 0.2520, lr: 0.009363, batch_cost: 0.7877, reader_cost: 0.00016, ips: 1.2696 samples/sec | ETA 08:07:41 2021-05-09 14:24:22 [INFO] [TRAIN] epoch: 8, iter: 2860/40000, loss: 0.4227, lr: 0.009361, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2715 samples/sec | ETA 08:06:49 2021-05-09 14:24:30 [INFO] [TRAIN] epoch: 8, iter: 2870/40000, loss: 0.2635, lr: 0.009359, batch_cost: 0.7879, reader_cost: 0.00015, ips: 1.2692 samples/sec | ETA 08:07:35 2021-05-09 14:24:38 [INFO] [TRAIN] epoch: 8, iter: 2880/40000, loss: 0.6418, lr: 0.009356, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2728 samples/sec | ETA 08:06:03 2021-05-09 14:24:45 [INFO] [TRAIN] epoch: 8, iter: 2890/40000, loss: 0.6047, lr: 0.009354, batch_cost: 0.7849, reader_cost: 0.00015, ips: 1.2741 samples/sec | ETA 08:05:27 2021-05-09 14:24:53 [INFO] [TRAIN] epoch: 8, iter: 2900/40000, loss: 0.6065, lr: 0.009352, batch_cost: 0.7841, reader_cost: 0.00015, ips: 1.2754 samples/sec | ETA 08:04:49 2021-05-09 14:25:01 [INFO] [TRAIN] epoch: 8, iter: 2910/40000, loss: 0.7323, lr: 0.009350, batch_cost: 0.7849, reader_cost: 0.00015, ips: 1.2740 samples/sec | ETA 08:05:12 2021-05-09 14:25:09 [INFO] [TRAIN] epoch: 8, iter: 2920/40000, loss: 0.6732, lr: 0.009347, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 08:05:46 2021-05-09 14:25:17 [INFO] [TRAIN] epoch: 8, iter: 2930/40000, loss: 0.3930, lr: 0.009345, batch_cost: 0.7874, reader_cost: 0.00017, ips: 1.2700 samples/sec | ETA 08:06:28 2021-05-09 14:25:25 [INFO] [TRAIN] epoch: 8, iter: 2940/40000, loss: 0.2709, lr: 0.009343, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2727 samples/sec | ETA 08:05:19 2021-05-09 14:25:33 [INFO] [TRAIN] epoch: 8, iter: 2950/40000, loss: 0.4383, lr: 0.009341, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2728 samples/sec | ETA 08:05:08 2021-05-09 14:25:40 [INFO] [TRAIN] epoch: 8, iter: 2960/40000, loss: 0.4697, lr: 0.009338, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 08:05:27 2021-05-09 14:25:48 [INFO] [TRAIN] epoch: 8, iter: 2970/40000, loss: 0.4517, lr: 0.009336, batch_cost: 0.7858, reader_cost: 0.00012, ips: 1.2726 samples/sec | ETA 08:04:58 2021-05-09 14:25:59 [INFO] [TRAIN] epoch: 9, iter: 2980/40000, loss: 0.4631, lr: 0.009334, batch_cost: 1.0745, reader_cost: 0.27857, ips: 0.9306 samples/sec | ETA 11:02:59 2021-05-09 14:26:07 [INFO] [TRAIN] epoch: 9, iter: 2990/40000, loss: 0.4707, lr: 0.009332, batch_cost: 0.8003, reader_cost: 0.00031, ips: 1.2495 samples/sec | ETA 08:13:40 2021-05-09 14:26:15 [INFO] [TRAIN] epoch: 9, iter: 3000/40000, loss: 0.7055, lr: 0.009329, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2728 samples/sec | ETA 08:04:30 2021-05-09 14:26:15 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-09 14:29:44 [INFO] [EVAL] #Images: 500 mIoU: 0.6681 Acc: 0.9383 Kappa: 0.9199 2021-05-09 14:29:44 [INFO] [EVAL] Class IoU: [0.9667 0.7421 0.8918 0.5382 0.4802 0.3664 0.504 0.6146 0.8962 0.5082 0.9309 0.6963 0.4549 0.9127 0.7431 0.7778 0.5623 0.4692 0.6381] 2021-05-09 14:29:44 [INFO] [EVAL] Class Acc: [0.9856 0.8221 0.9361 0.8234 0.7082 0.7475 0.7842 0.9033 0.9282 0.9076 0.9651 0.7913 0.5283 0.9404 0.9329 0.8816 0.7662 0.7505 0.7467] 2021-05-09 14:30:31 [INFO] [EVAL] The model with the best validation mIoU (0.6681) was saved at iter 3000. 2021-05-09 14:30:39 [INFO] [TRAIN] epoch: 9, iter: 3010/40000, loss: 0.5368, lr: 0.009327, batch_cost: 0.7813, reader_cost: 0.00083, ips: 1.2798 samples/sec | ETA 08:01:41 2021-05-09 14:30:47 [INFO] [TRAIN] epoch: 9, iter: 3020/40000, loss: 0.6160, lr: 0.009325, batch_cost: 0.7848, reader_cost: 0.00031, ips: 1.2742 samples/sec | ETA 08:03:42 2021-05-09 14:30:55 [INFO] [TRAIN] epoch: 9, iter: 3030/40000, loss: 0.5704, lr: 0.009323, batch_cost: 0.7842, reader_cost: 0.00016, ips: 1.2752 samples/sec | ETA 08:03:11 2021-05-09 14:31:02 [INFO] [TRAIN] epoch: 9, iter: 3040/40000, loss: 0.4144, lr: 0.009320, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2720 samples/sec | ETA 08:04:17 2021-05-09 14:31:10 [INFO] [TRAIN] epoch: 9, iter: 3050/40000, loss: 0.2794, lr: 0.009318, batch_cost: 0.7854, reader_cost: 0.00015, ips: 1.2733 samples/sec | ETA 08:03:39 2021-05-09 14:31:18 [INFO] [TRAIN] epoch: 9, iter: 3060/40000, loss: 0.5407, lr: 0.009316, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2735 samples/sec | ETA 08:03:26 2021-05-09 14:31:26 [INFO] [TRAIN] epoch: 9, iter: 3070/40000, loss: 0.7348, lr: 0.009314, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2717 samples/sec | ETA 08:03:59 2021-05-09 14:31:34 [INFO] [TRAIN] epoch: 9, iter: 3080/40000, loss: 0.5149, lr: 0.009311, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2707 samples/sec | ETA 08:04:15 2021-05-09 14:31:42 [INFO] [TRAIN] epoch: 9, iter: 3090/40000, loss: 0.4589, lr: 0.009309, batch_cost: 0.7869, reader_cost: 0.00018, ips: 1.2708 samples/sec | ETA 08:04:04 2021-05-09 14:31:50 [INFO] [TRAIN] epoch: 9, iter: 3100/40000, loss: 0.2387, lr: 0.009307, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 08:03:41 2021-05-09 14:31:58 [INFO] [TRAIN] epoch: 9, iter: 3110/40000, loss: 0.3357, lr: 0.009305, batch_cost: 0.7860, reader_cost: 0.00017, ips: 1.2723 samples/sec | ETA 08:03:15 2021-05-09 14:32:05 [INFO] [TRAIN] epoch: 9, iter: 3120/40000, loss: 0.6563, lr: 0.009302, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2720 samples/sec | ETA 08:03:13 2021-05-09 14:32:13 [INFO] [TRAIN] epoch: 9, iter: 3130/40000, loss: 0.5536, lr: 0.009300, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2704 samples/sec | ETA 08:03:42 2021-05-09 14:32:21 [INFO] [TRAIN] epoch: 9, iter: 3140/40000, loss: 0.6966, lr: 0.009298, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2716 samples/sec | ETA 08:03:07 2021-05-09 14:32:29 [INFO] [TRAIN] epoch: 9, iter: 3150/40000, loss: 0.3643, lr: 0.009296, batch_cost: 0.7869, reader_cost: 0.00015, ips: 1.2708 samples/sec | ETA 08:03:16 2021-05-09 14:32:37 [INFO] [TRAIN] epoch: 9, iter: 3160/40000, loss: 0.5260, lr: 0.009293, batch_cost: 0.7849, reader_cost: 0.00015, ips: 1.2740 samples/sec | ETA 08:01:56 2021-05-09 14:32:45 [INFO] [TRAIN] epoch: 9, iter: 3170/40000, loss: 0.2204, lr: 0.009291, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2705 samples/sec | ETA 08:03:08 2021-05-09 14:32:53 [INFO] [TRAIN] epoch: 9, iter: 3180/40000, loss: 0.2266, lr: 0.009289, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2715 samples/sec | ETA 08:02:37 2021-05-09 14:33:00 [INFO] [TRAIN] epoch: 9, iter: 3190/40000, loss: 0.4853, lr: 0.009287, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 08:02:36 2021-05-09 14:33:08 [INFO] [TRAIN] epoch: 9, iter: 3200/40000, loss: 0.2433, lr: 0.009284, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2709 samples/sec | ETA 08:02:35 2021-05-09 14:33:16 [INFO] [TRAIN] epoch: 9, iter: 3210/40000, loss: 0.3420, lr: 0.009282, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2735 samples/sec | ETA 08:01:28 2021-05-09 14:33:24 [INFO] [TRAIN] epoch: 9, iter: 3220/40000, loss: 0.3107, lr: 0.009280, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2732 samples/sec | ETA 08:01:27 2021-05-09 14:33:32 [INFO] [TRAIN] epoch: 9, iter: 3230/40000, loss: 0.3598, lr: 0.009278, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2735 samples/sec | ETA 08:01:12 2021-05-09 14:33:40 [INFO] [TRAIN] epoch: 9, iter: 3240/40000, loss: 0.4263, lr: 0.009276, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2723 samples/sec | ETA 08:01:31 2021-05-09 14:33:48 [INFO] [TRAIN] epoch: 9, iter: 3250/40000, loss: 0.4837, lr: 0.009273, batch_cost: 0.7852, reader_cost: 0.00017, ips: 1.2736 samples/sec | ETA 08:00:55 2021-05-09 14:33:55 [INFO] [TRAIN] epoch: 9, iter: 3260/40000, loss: 0.6922, lr: 0.009271, batch_cost: 0.7844, reader_cost: 0.00017, ips: 1.2749 samples/sec | ETA 08:00:17 2021-05-09 14:34:03 [INFO] [TRAIN] epoch: 9, iter: 3270/40000, loss: 0.5102, lr: 0.009269, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2720 samples/sec | ETA 08:01:16 2021-05-09 14:34:11 [INFO] [TRAIN] epoch: 9, iter: 3280/40000, loss: 0.8820, lr: 0.009267, batch_cost: 0.7874, reader_cost: 0.00016, ips: 1.2700 samples/sec | ETA 08:01:53 2021-05-09 14:34:19 [INFO] [TRAIN] epoch: 9, iter: 3290/40000, loss: 0.5642, lr: 0.009264, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2720 samples/sec | ETA 08:01:01 2021-05-09 14:34:27 [INFO] [TRAIN] epoch: 9, iter: 3300/40000, loss: 0.4809, lr: 0.009262, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 08:00:47 2021-05-09 14:34:35 [INFO] [TRAIN] epoch: 9, iter: 3310/40000, loss: 0.1605, lr: 0.009260, batch_cost: 0.7845, reader_cost: 0.00017, ips: 1.2747 samples/sec | ETA 07:59:43 2021-05-09 14:34:43 [INFO] [TRAIN] epoch: 9, iter: 3320/40000, loss: 0.4980, lr: 0.009258, batch_cost: 0.7839, reader_cost: 0.00017, ips: 1.2756 samples/sec | ETA 07:59:15 2021-05-09 14:34:50 [INFO] [TRAIN] epoch: 9, iter: 3330/40000, loss: 0.6131, lr: 0.009255, batch_cost: 0.7864, reader_cost: 0.00017, ips: 1.2716 samples/sec | ETA 08:00:37 2021-05-09 14:34:58 [INFO] [TRAIN] epoch: 9, iter: 3340/40000, loss: 0.5588, lr: 0.009253, batch_cost: 0.7841, reader_cost: 0.00014, ips: 1.2753 samples/sec | ETA 07:59:05 2021-05-09 14:35:09 [INFO] [TRAIN] epoch: 10, iter: 3350/40000, loss: 0.6274, lr: 0.009251, batch_cost: 1.0802, reader_cost: 0.24992, ips: 0.9258 samples/sec | ETA 10:59:48 2021-05-09 14:35:17 [INFO] [TRAIN] epoch: 10, iter: 3360/40000, loss: 0.4312, lr: 0.009249, batch_cost: 0.7946, reader_cost: 0.00032, ips: 1.2585 samples/sec | ETA 08:05:14 2021-05-09 14:35:25 [INFO] [TRAIN] epoch: 10, iter: 3370/40000, loss: 0.7884, lr: 0.009246, batch_cost: 0.7848, reader_cost: 0.00015, ips: 1.2743 samples/sec | ETA 07:59:06 2021-05-09 14:35:33 [INFO] [TRAIN] epoch: 10, iter: 3380/40000, loss: 0.5260, lr: 0.009244, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 07:59:59 2021-05-09 14:35:41 [INFO] [TRAIN] epoch: 10, iter: 3390/40000, loss: 0.6775, lr: 0.009242, batch_cost: 0.7876, reader_cost: 0.00017, ips: 1.2697 samples/sec | ETA 08:00:33 2021-05-09 14:35:48 [INFO] [TRAIN] epoch: 10, iter: 3400/40000, loss: 0.7094, lr: 0.009240, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2709 samples/sec | ETA 07:59:57 2021-05-09 14:35:56 [INFO] [TRAIN] epoch: 10, iter: 3410/40000, loss: 0.5364, lr: 0.009237, batch_cost: 0.7865, reader_cost: 0.00017, ips: 1.2715 samples/sec | ETA 07:59:37 2021-05-09 14:36:04 [INFO] [TRAIN] epoch: 10, iter: 3420/40000, loss: 0.1731, lr: 0.009235, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2727 samples/sec | ETA 07:59:01 2021-05-09 14:36:12 [INFO] [TRAIN] epoch: 10, iter: 3430/40000, loss: 0.4574, lr: 0.009233, batch_cost: 0.7877, reader_cost: 0.00016, ips: 1.2695 samples/sec | ETA 08:00:07 2021-05-09 14:36:20 [INFO] [TRAIN] epoch: 10, iter: 3440/40000, loss: 0.4341, lr: 0.009231, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2714 samples/sec | ETA 07:59:16 2021-05-09 14:36:28 [INFO] [TRAIN] epoch: 10, iter: 3450/40000, loss: 0.3660, lr: 0.009228, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2710 samples/sec | ETA 07:59:16 2021-05-09 14:36:36 [INFO] [TRAIN] epoch: 10, iter: 3460/40000, loss: 0.5031, lr: 0.009226, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 07:58:41 2021-05-09 14:36:44 [INFO] [TRAIN] epoch: 10, iter: 3470/40000, loss: 0.2219, lr: 0.009224, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2729 samples/sec | ETA 07:58:17 2021-05-09 14:36:51 [INFO] [TRAIN] epoch: 10, iter: 3480/40000, loss: 0.4717, lr: 0.009222, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2716 samples/sec | ETA 07:58:40 2021-05-09 14:36:59 [INFO] [TRAIN] epoch: 10, iter: 3490/40000, loss: 0.6864, lr: 0.009219, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2723 samples/sec | ETA 07:58:15 2021-05-09 14:37:07 [INFO] [TRAIN] epoch: 10, iter: 3500/40000, loss: 0.5344, lr: 0.009217, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 07:58:13 2021-05-09 14:37:15 [INFO] [TRAIN] epoch: 10, iter: 3510/40000, loss: 0.2980, lr: 0.009215, batch_cost: 0.7852, reader_cost: 0.00017, ips: 1.2736 samples/sec | ETA 07:57:31 2021-05-09 14:37:23 [INFO] [TRAIN] epoch: 10, iter: 3520/40000, loss: 0.2877, lr: 0.009213, batch_cost: 0.7851, reader_cost: 0.00016, ips: 1.2738 samples/sec | ETA 07:57:19 2021-05-09 14:37:31 [INFO] [TRAIN] epoch: 10, iter: 3530/40000, loss: 0.5081, lr: 0.009210, batch_cost: 0.7867, reader_cost: 0.00015, ips: 1.2711 samples/sec | ETA 07:58:11 2021-05-09 14:37:39 [INFO] [TRAIN] epoch: 10, iter: 3540/40000, loss: 0.1596, lr: 0.009208, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2717 samples/sec | ETA 07:57:51 2021-05-09 14:37:46 [INFO] [TRAIN] epoch: 10, iter: 3550/40000, loss: 0.3567, lr: 0.009206, batch_cost: 0.7869, reader_cost: 0.00017, ips: 1.2708 samples/sec | ETA 07:58:02 2021-05-09 14:37:54 [INFO] [TRAIN] epoch: 10, iter: 3560/40000, loss: 0.5944, lr: 0.009204, batch_cost: 0.7845, reader_cost: 0.00017, ips: 1.2747 samples/sec | ETA 07:56:26 2021-05-09 14:38:02 [INFO] [TRAIN] epoch: 10, iter: 3570/40000, loss: 0.6219, lr: 0.009201, batch_cost: 0.7867, reader_cost: 0.00017, ips: 1.2711 samples/sec | ETA 07:57:40 2021-05-09 14:38:10 [INFO] [TRAIN] epoch: 10, iter: 3580/40000, loss: 0.5383, lr: 0.009199, batch_cost: 0.7856, reader_cost: 0.00017, ips: 1.2729 samples/sec | ETA 07:56:51 2021-05-09 14:38:18 [INFO] [TRAIN] epoch: 10, iter: 3590/40000, loss: 0.4138, lr: 0.009197, batch_cost: 0.7843, reader_cost: 0.00014, ips: 1.2750 samples/sec | ETA 07:55:56 2021-05-09 14:38:26 [INFO] [TRAIN] epoch: 10, iter: 3600/40000, loss: 0.5052, lr: 0.009195, batch_cost: 0.7876, reader_cost: 0.00015, ips: 1.2697 samples/sec | ETA 07:57:48 2021-05-09 14:38:34 [INFO] [TRAIN] epoch: 10, iter: 3610/40000, loss: 0.3796, lr: 0.009192, batch_cost: 0.7876, reader_cost: 0.00017, ips: 1.2697 samples/sec | ETA 07:57:40 2021-05-09 14:38:41 [INFO] [TRAIN] epoch: 10, iter: 3620/40000, loss: 0.4430, lr: 0.009190, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2714 samples/sec | ETA 07:56:53 2021-05-09 14:38:49 [INFO] [TRAIN] epoch: 10, iter: 3630/40000, loss: 0.8895, lr: 0.009188, batch_cost: 0.7864, reader_cost: 0.00018, ips: 1.2716 samples/sec | ETA 07:56:41 2021-05-09 14:38:57 [INFO] [TRAIN] epoch: 10, iter: 3640/40000, loss: 0.5266, lr: 0.009186, batch_cost: 0.7879, reader_cost: 0.00016, ips: 1.2692 samples/sec | ETA 07:57:28 2021-05-09 14:39:05 [INFO] [TRAIN] epoch: 10, iter: 3650/40000, loss: 0.7596, lr: 0.009183, batch_cost: 0.7839, reader_cost: 0.00017, ips: 1.2757 samples/sec | ETA 07:54:55 2021-05-09 14:39:13 [INFO] [TRAIN] epoch: 10, iter: 3660/40000, loss: 0.7682, lr: 0.009181, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 07:56:07 2021-05-09 14:39:21 [INFO] [TRAIN] epoch: 10, iter: 3670/40000, loss: 0.6509, lr: 0.009179, batch_cost: 0.7856, reader_cost: 0.00017, ips: 1.2729 samples/sec | ETA 07:55:40 2021-05-09 14:39:29 [INFO] [TRAIN] epoch: 10, iter: 3680/40000, loss: 0.2877, lr: 0.009177, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2728 samples/sec | ETA 07:55:36 2021-05-09 14:39:36 [INFO] [TRAIN] epoch: 10, iter: 3690/40000, loss: 0.4644, lr: 0.009174, batch_cost: 0.7861, reader_cost: 0.00017, ips: 1.2720 samples/sec | ETA 07:55:44 2021-05-09 14:39:44 [INFO] [TRAIN] epoch: 10, iter: 3700/40000, loss: 0.3506, lr: 0.009172, batch_cost: 0.7893, reader_cost: 0.00018, ips: 1.2669 samples/sec | ETA 07:57:31 2021-05-09 14:39:52 [INFO] [TRAIN] epoch: 10, iter: 3710/40000, loss: 0.5321, lr: 0.009170, batch_cost: 0.7851, reader_cost: 0.00015, ips: 1.2737 samples/sec | ETA 07:54:52 2021-05-09 14:40:00 [INFO] [TRAIN] epoch: 10, iter: 3720/40000, loss: 0.5655, lr: 0.009168, batch_cost: 0.7850, reader_cost: 0.00009, ips: 1.2739 samples/sec | ETA 07:54:39 2021-05-09 14:40:11 [INFO] [TRAIN] epoch: 11, iter: 3730/40000, loss: 0.5409, lr: 0.009165, batch_cost: 1.0843, reader_cost: 0.22829, ips: 0.9222 samples/sec | ETA 10:55:29 2021-05-09 14:40:19 [INFO] [TRAIN] epoch: 11, iter: 3740/40000, loss: 0.5412, lr: 0.009163, batch_cost: 0.7867, reader_cost: 0.00033, ips: 1.2712 samples/sec | ETA 07:55:24 2021-05-09 14:40:27 [INFO] [TRAIN] epoch: 11, iter: 3750/40000, loss: 0.3846, lr: 0.009161, batch_cost: 0.7859, reader_cost: 0.00017, ips: 1.2724 samples/sec | ETA 07:54:49 2021-05-09 14:40:35 [INFO] [TRAIN] epoch: 11, iter: 3760/40000, loss: 0.5830, lr: 0.009159, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2717 samples/sec | ETA 07:54:57 2021-05-09 14:40:42 [INFO] [TRAIN] epoch: 11, iter: 3770/40000, loss: 0.6687, lr: 0.009156, batch_cost: 0.7856, reader_cost: 0.00017, ips: 1.2730 samples/sec | ETA 07:54:21 2021-05-09 14:40:50 [INFO] [TRAIN] epoch: 11, iter: 3780/40000, loss: 0.7949, lr: 0.009154, batch_cost: 0.7867, reader_cost: 0.00017, ips: 1.2712 samples/sec | ETA 07:54:53 2021-05-09 14:40:58 [INFO] [TRAIN] epoch: 11, iter: 3790/40000, loss: 0.4477, lr: 0.009152, batch_cost: 0.7864, reader_cost: 0.00017, ips: 1.2717 samples/sec | ETA 07:54:34 2021-05-09 14:41:06 [INFO] [TRAIN] epoch: 11, iter: 3800/40000, loss: 0.3692, lr: 0.009150, batch_cost: 0.7874, reader_cost: 0.00015, ips: 1.2699 samples/sec | ETA 07:55:05 2021-05-09 14:41:14 [INFO] [TRAIN] epoch: 11, iter: 3810/40000, loss: 0.5132, lr: 0.009147, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 07:54:28 2021-05-09 14:41:22 [INFO] [TRAIN] epoch: 11, iter: 3820/40000, loss: 0.3300, lr: 0.009145, batch_cost: 0.7875, reader_cost: 0.00016, ips: 1.2699 samples/sec | ETA 07:54:50 2021-05-09 14:41:30 [INFO] [TRAIN] epoch: 11, iter: 3830/40000, loss: 0.4279, lr: 0.009143, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2709 samples/sec | ETA 07:54:19 2021-05-09 14:41:37 [INFO] [TRAIN] epoch: 11, iter: 3840/40000, loss: 0.1957, lr: 0.009141, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 07:53:57 2021-05-09 14:41:45 [INFO] [TRAIN] epoch: 11, iter: 3850/40000, loss: 0.4118, lr: 0.009138, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2709 samples/sec | ETA 07:54:04 2021-05-09 14:41:53 [INFO] [TRAIN] epoch: 11, iter: 3860/40000, loss: 0.6832, lr: 0.009136, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2725 samples/sec | ETA 07:53:21 2021-05-09 14:42:01 [INFO] [TRAIN] epoch: 11, iter: 3870/40000, loss: 0.3193, lr: 0.009134, batch_cost: 0.7865, reader_cost: 0.00017, ips: 1.2714 samples/sec | ETA 07:53:36 2021-05-09 14:42:09 [INFO] [TRAIN] epoch: 11, iter: 3880/40000, loss: 0.4522, lr: 0.009132, batch_cost: 0.7867, reader_cost: 0.00017, ips: 1.2712 samples/sec | ETA 07:53:34 2021-05-09 14:42:17 [INFO] [TRAIN] epoch: 11, iter: 3890/40000, loss: 0.3752, lr: 0.009129, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2726 samples/sec | ETA 07:52:55 2021-05-09 14:42:25 [INFO] [TRAIN] epoch: 11, iter: 3900/40000, loss: 0.4118, lr: 0.009127, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2726 samples/sec | ETA 07:52:46 2021-05-09 14:42:33 [INFO] [TRAIN] epoch: 11, iter: 3910/40000, loss: 0.2345, lr: 0.009125, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2707 samples/sec | ETA 07:53:22 2021-05-09 14:42:40 [INFO] [TRAIN] epoch: 11, iter: 3920/40000, loss: 0.1439, lr: 0.009123, batch_cost: 0.7862, reader_cost: 0.00017, ips: 1.2720 samples/sec | ETA 07:52:45 2021-05-09 14:42:48 [INFO] [TRAIN] epoch: 11, iter: 3930/40000, loss: 0.4531, lr: 0.009120, batch_cost: 0.7849, reader_cost: 0.00016, ips: 1.2740 samples/sec | ETA 07:51:51 2021-05-09 14:42:56 [INFO] [TRAIN] epoch: 11, iter: 3940/40000, loss: 0.4749, lr: 0.009118, batch_cost: 0.7858, reader_cost: 0.00019, ips: 1.2726 samples/sec | ETA 07:52:15 2021-05-09 14:43:04 [INFO] [TRAIN] epoch: 11, iter: 3950/40000, loss: 0.4181, lr: 0.009116, batch_cost: 0.7869, reader_cost: 0.00017, ips: 1.2708 samples/sec | ETA 07:52:47 2021-05-09 14:43:12 [INFO] [TRAIN] epoch: 11, iter: 3960/40000, loss: 0.4179, lr: 0.009114, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2709 samples/sec | ETA 07:52:37 2021-05-09 14:43:20 [INFO] [TRAIN] epoch: 11, iter: 3970/40000, loss: 0.3397, lr: 0.009111, batch_cost: 0.7838, reader_cost: 0.00015, ips: 1.2758 samples/sec | ETA 07:50:41 2021-05-09 14:43:28 [INFO] [TRAIN] epoch: 11, iter: 3980/40000, loss: 0.3634, lr: 0.009109, batch_cost: 0.7847, reader_cost: 0.00015, ips: 1.2743 samples/sec | ETA 07:51:06 2021-05-09 14:43:35 [INFO] [TRAIN] epoch: 11, iter: 3990/40000, loss: 0.3160, lr: 0.009107, batch_cost: 0.7843, reader_cost: 0.00016, ips: 1.2750 samples/sec | ETA 07:50:43 2021-05-09 14:43:43 [INFO] [TRAIN] epoch: 11, iter: 4000/40000, loss: 0.5222, lr: 0.009105, batch_cost: 0.7848, reader_cost: 0.00016, ips: 1.2743 samples/sec | ETA 07:50:51 2021-05-09 14:43:43 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-09 14:47:18 [INFO] [EVAL] #Images: 500 mIoU: 0.6851 Acc: 0.9433 Kappa: 0.9263 2021-05-09 14:47:18 [INFO] [EVAL] Class IoU: [0.9711 0.7801 0.8966 0.5674 0.4486 0.423 0.5205 0.6366 0.8985 0.6148 0.9295 0.7256 0.4528 0.9164 0.7351 0.8126 0.5731 0.4524 0.6627] 2021-05-09 14:47:18 [INFO] [EVAL] Class Acc: [0.9833 0.9054 0.9364 0.8224 0.8588 0.6863 0.7433 0.8838 0.9309 0.783 0.9566 0.8254 0.7583 0.9426 0.9093 0.8859 0.7204 0.6071 0.7805] 2021-05-09 14:48:07 [INFO] [EVAL] The model with the best validation mIoU (0.6851) was saved at iter 4000. 2021-05-09 14:48:15 [INFO] [TRAIN] epoch: 11, iter: 4010/40000, loss: 0.5075, lr: 0.009102, batch_cost: 0.7825, reader_cost: 0.00024, ips: 1.2779 samples/sec | ETA 07:49:22 2021-05-09 14:48:22 [INFO] [TRAIN] epoch: 11, iter: 4020/40000, loss: 0.6169, lr: 0.009100, batch_cost: 0.7830, reader_cost: 0.00016, ips: 1.2771 samples/sec | ETA 07:49:33 2021-05-09 14:48:30 [INFO] [TRAIN] epoch: 11, iter: 4030/40000, loss: 0.4756, lr: 0.009098, batch_cost: 0.7838, reader_cost: 0.00015, ips: 1.2759 samples/sec | ETA 07:49:51 2021-05-09 14:48:38 [INFO] [TRAIN] epoch: 11, iter: 4040/40000, loss: 0.2935, lr: 0.009096, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2717 samples/sec | ETA 07:51:16 2021-05-09 14:48:46 [INFO] [TRAIN] epoch: 11, iter: 4050/40000, loss: 0.2259, lr: 0.009093, batch_cost: 0.7872, reader_cost: 0.00014, ips: 1.2704 samples/sec | ETA 07:51:39 2021-05-09 14:48:54 [INFO] [TRAIN] epoch: 11, iter: 4060/40000, loss: 0.4378, lr: 0.009091, batch_cost: 0.7847, reader_cost: 0.00015, ips: 1.2743 samples/sec | ETA 07:50:03 2021-05-09 14:49:02 [INFO] [TRAIN] epoch: 11, iter: 4070/40000, loss: 0.4441, lr: 0.009089, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2720 samples/sec | ETA 07:50:46 2021-05-09 14:49:10 [INFO] [TRAIN] epoch: 11, iter: 4080/40000, loss: 0.5237, lr: 0.009087, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2718 samples/sec | ETA 07:50:43 2021-05-09 14:49:17 [INFO] [TRAIN] epoch: 11, iter: 4090/40000, loss: 0.5238, lr: 0.009084, batch_cost: 0.7837, reader_cost: 0.00010, ips: 1.2761 samples/sec | ETA 07:49:01 2021-05-09 14:49:28 [INFO] [TRAIN] epoch: 12, iter: 4100/40000, loss: 0.3892, lr: 0.009082, batch_cost: 1.0952, reader_cost: 0.25190, ips: 0.9131 samples/sec | ETA 10:55:17 2021-05-09 14:49:36 [INFO] [TRAIN] epoch: 12, iter: 4110/40000, loss: 0.9183, lr: 0.009080, batch_cost: 0.7959, reader_cost: 0.00036, ips: 1.2565 samples/sec | ETA 07:56:03 2021-05-09 14:49:44 [INFO] [TRAIN] epoch: 12, iter: 4120/40000, loss: 0.5211, lr: 0.009078, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2727 samples/sec | ETA 07:49:52 2021-05-09 14:49:52 [INFO] [TRAIN] epoch: 12, iter: 4130/40000, loss: 0.4596, lr: 0.009075, batch_cost: 0.7876, reader_cost: 0.00015, ips: 1.2697 samples/sec | ETA 07:50:49 2021-05-09 14:50:00 [INFO] [TRAIN] epoch: 12, iter: 4140/40000, loss: 0.7045, lr: 0.009073, batch_cost: 0.7870, reader_cost: 0.00015, ips: 1.2706 samples/sec | ETA 07:50:22 2021-05-09 14:50:08 [INFO] [TRAIN] epoch: 12, iter: 4150/40000, loss: 0.5059, lr: 0.009071, batch_cost: 0.7880, reader_cost: 0.00017, ips: 1.2690 samples/sec | ETA 07:50:50 2021-05-09 14:50:16 [INFO] [TRAIN] epoch: 12, iter: 4160/40000, loss: 0.3292, lr: 0.009069, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2709 samples/sec | ETA 07:50:01 2021-05-09 14:50:24 [INFO] [TRAIN] epoch: 12, iter: 4170/40000, loss: 0.2635, lr: 0.009066, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2713 samples/sec | ETA 07:49:44 2021-05-09 14:50:31 [INFO] [TRAIN] epoch: 12, iter: 4180/40000, loss: 0.3869, lr: 0.009064, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2721 samples/sec | ETA 07:49:17 2021-05-09 14:50:39 [INFO] [TRAIN] epoch: 12, iter: 4190/40000, loss: 0.3956, lr: 0.009062, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2721 samples/sec | ETA 07:49:10 2021-05-09 14:50:47 [INFO] [TRAIN] epoch: 12, iter: 4200/40000, loss: 0.4143, lr: 0.009060, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2721 samples/sec | ETA 07:49:02 2021-05-09 14:50:55 [INFO] [TRAIN] epoch: 12, iter: 4210/40000, loss: 0.2144, lr: 0.009057, batch_cost: 0.7877, reader_cost: 0.00015, ips: 1.2695 samples/sec | ETA 07:49:52 2021-05-09 14:51:03 [INFO] [TRAIN] epoch: 12, iter: 4220/40000, loss: 0.2751, lr: 0.009055, batch_cost: 0.7879, reader_cost: 0.00016, ips: 1.2692 samples/sec | ETA 07:49:50 2021-05-09 14:51:11 [INFO] [TRAIN] epoch: 12, iter: 4230/40000, loss: 0.3671, lr: 0.009053, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 07:48:39 2021-05-09 14:51:19 [INFO] [TRAIN] epoch: 12, iter: 4240/40000, loss: 0.4466, lr: 0.009051, batch_cost: 0.7873, reader_cost: 0.00017, ips: 1.2702 samples/sec | ETA 07:49:12 2021-05-09 14:51:26 [INFO] [TRAIN] epoch: 12, iter: 4250/40000, loss: 0.4413, lr: 0.009048, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 07:48:20 2021-05-09 14:51:34 [INFO] [TRAIN] epoch: 12, iter: 4260/40000, loss: 0.3666, lr: 0.009046, batch_cost: 0.7863, reader_cost: 0.00018, ips: 1.2718 samples/sec | ETA 07:48:22 2021-05-09 14:51:42 [INFO] [TRAIN] epoch: 12, iter: 4270/40000, loss: 0.3967, lr: 0.009044, batch_cost: 0.7864, reader_cost: 0.00018, ips: 1.2717 samples/sec | ETA 07:48:16 2021-05-09 14:51:50 [INFO] [TRAIN] epoch: 12, iter: 4280/40000, loss: 0.3038, lr: 0.009042, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2730 samples/sec | ETA 07:47:40 2021-05-09 14:51:58 [INFO] [TRAIN] epoch: 12, iter: 4290/40000, loss: 0.1596, lr: 0.009039, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2730 samples/sec | ETA 07:47:31 2021-05-09 14:52:06 [INFO] [TRAIN] epoch: 12, iter: 4300/40000, loss: 0.4171, lr: 0.009037, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 07:47:43 2021-05-09 14:52:14 [INFO] [TRAIN] epoch: 12, iter: 4310/40000, loss: 0.4507, lr: 0.009035, batch_cost: 0.7901, reader_cost: 0.00015, ips: 1.2657 samples/sec | ETA 07:49:57 2021-05-09 14:52:22 [INFO] [TRAIN] epoch: 12, iter: 4320/40000, loss: 0.3350, lr: 0.009033, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2724 samples/sec | ETA 07:47:21 2021-05-09 14:52:29 [INFO] [TRAIN] epoch: 12, iter: 4330/40000, loss: 0.3389, lr: 0.009030, batch_cost: 0.7848, reader_cost: 0.00015, ips: 1.2741 samples/sec | ETA 07:46:35 2021-05-09 14:52:37 [INFO] [TRAIN] epoch: 12, iter: 4340/40000, loss: 0.3817, lr: 0.009028, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2728 samples/sec | ETA 07:46:57 2021-05-09 14:52:45 [INFO] [TRAIN] epoch: 12, iter: 4350/40000, loss: 0.4669, lr: 0.009026, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 07:47:02 2021-05-09 14:52:53 [INFO] [TRAIN] epoch: 12, iter: 4360/40000, loss: 0.2294, lr: 0.009024, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2714 samples/sec | ETA 07:47:12 2021-05-09 14:53:01 [INFO] [TRAIN] epoch: 12, iter: 4370/40000, loss: 0.5320, lr: 0.009021, batch_cost: 0.7864, reader_cost: 0.00017, ips: 1.2716 samples/sec | ETA 07:46:59 2021-05-09 14:53:09 [INFO] [TRAIN] epoch: 12, iter: 4380/40000, loss: 0.4361, lr: 0.009019, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2723 samples/sec | ETA 07:46:36 2021-05-09 14:53:17 [INFO] [TRAIN] epoch: 12, iter: 4390/40000, loss: 0.6068, lr: 0.009017, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2725 samples/sec | ETA 07:46:24 2021-05-09 14:53:24 [INFO] [TRAIN] epoch: 12, iter: 4400/40000, loss: 0.8289, lr: 0.009015, batch_cost: 0.7849, reader_cost: 0.00015, ips: 1.2740 samples/sec | ETA 07:45:43 2021-05-09 14:53:32 [INFO] [TRAIN] epoch: 12, iter: 4410/40000, loss: 0.5748, lr: 0.009012, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2722 samples/sec | ETA 07:46:15 2021-05-09 14:53:40 [INFO] [TRAIN] epoch: 12, iter: 4420/40000, loss: 0.3702, lr: 0.009010, batch_cost: 0.7860, reader_cost: 0.00017, ips: 1.2723 samples/sec | ETA 07:46:05 2021-05-09 14:53:48 [INFO] [TRAIN] epoch: 12, iter: 4430/40000, loss: 0.3696, lr: 0.009008, batch_cost: 0.7870, reader_cost: 0.00018, ips: 1.2707 samples/sec | ETA 07:46:33 2021-05-09 14:53:56 [INFO] [TRAIN] epoch: 12, iter: 4440/40000, loss: 0.4294, lr: 0.009005, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2725 samples/sec | ETA 07:45:45 2021-05-09 14:54:04 [INFO] [TRAIN] epoch: 12, iter: 4450/40000, loss: 0.7788, lr: 0.009003, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2734 samples/sec | ETA 07:45:17 2021-05-09 14:54:12 [INFO] [TRAIN] epoch: 12, iter: 4460/40000, loss: 0.5791, lr: 0.009001, batch_cost: 0.7867, reader_cost: 0.00011, ips: 1.2711 samples/sec | ETA 07:45:59 2021-05-09 14:54:23 [INFO] [TRAIN] epoch: 13, iter: 4470/40000, loss: 0.5570, lr: 0.008999, batch_cost: 1.0951, reader_cost: 0.23058, ips: 0.9132 samples/sec | ETA 10:48:29 2021-05-09 14:54:30 [INFO] [TRAIN] epoch: 13, iter: 4480/40000, loss: 0.4238, lr: 0.008996, batch_cost: 0.7932, reader_cost: 0.00035, ips: 1.2608 samples/sec | ETA 07:49:33 2021-05-09 14:54:38 [INFO] [TRAIN] epoch: 13, iter: 4490/40000, loss: 0.5278, lr: 0.008994, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2726 samples/sec | ETA 07:45:04 2021-05-09 14:54:46 [INFO] [TRAIN] epoch: 13, iter: 4500/40000, loss: 0.4816, lr: 0.008992, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 07:44:59 2021-05-09 14:54:54 [INFO] [TRAIN] epoch: 13, iter: 4510/40000, loss: 0.5482, lr: 0.008990, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2725 samples/sec | ETA 07:44:50 2021-05-09 14:55:02 [INFO] [TRAIN] epoch: 13, iter: 4520/40000, loss: 0.4562, lr: 0.008987, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 07:45:11 2021-05-09 14:55:10 [INFO] [TRAIN] epoch: 13, iter: 4530/40000, loss: 0.5074, lr: 0.008985, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2734 samples/sec | ETA 07:44:14 2021-05-09 14:55:18 [INFO] [TRAIN] epoch: 13, iter: 4540/40000, loss: 0.2530, lr: 0.008983, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2703 samples/sec | ETA 07:45:14 2021-05-09 14:55:26 [INFO] [TRAIN] epoch: 13, iter: 4550/40000, loss: 0.5207, lr: 0.008981, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 07:44:41 2021-05-09 14:55:33 [INFO] [TRAIN] epoch: 13, iter: 4560/40000, loss: 0.3818, lr: 0.008978, batch_cost: 0.7880, reader_cost: 0.00016, ips: 1.2690 samples/sec | ETA 07:45:28 2021-05-09 14:55:41 [INFO] [TRAIN] epoch: 13, iter: 4570/40000, loss: 0.4550, lr: 0.008976, batch_cost: 0.7849, reader_cost: 0.00017, ips: 1.2740 samples/sec | ETA 07:43:30 2021-05-09 14:55:49 [INFO] [TRAIN] epoch: 13, iter: 4580/40000, loss: 0.5744, lr: 0.008974, batch_cost: 0.7879, reader_cost: 0.00016, ips: 1.2692 samples/sec | ETA 07:45:06 2021-05-09 14:55:57 [INFO] [TRAIN] epoch: 13, iter: 4590/40000, loss: 0.2703, lr: 0.008972, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2707 samples/sec | ETA 07:44:26 2021-05-09 14:56:05 [INFO] [TRAIN] epoch: 13, iter: 4600/40000, loss: 0.6756, lr: 0.008969, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2705 samples/sec | ETA 07:44:22 2021-05-09 14:56:13 [INFO] [TRAIN] epoch: 13, iter: 4610/40000, loss: 0.4364, lr: 0.008967, batch_cost: 0.7873, reader_cost: 0.00015, ips: 1.2702 samples/sec | ETA 07:44:22 2021-05-09 14:56:21 [INFO] [TRAIN] epoch: 13, iter: 4620/40000, loss: 0.5318, lr: 0.008965, batch_cost: 0.7843, reader_cost: 0.00015, ips: 1.2751 samples/sec | ETA 07:42:27 2021-05-09 14:56:28 [INFO] [TRAIN] epoch: 13, iter: 4630/40000, loss: 0.2458, lr: 0.008963, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2720 samples/sec | ETA 07:43:26 2021-05-09 14:56:36 [INFO] [TRAIN] epoch: 13, iter: 4640/40000, loss: 0.3940, lr: 0.008960, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2703 samples/sec | ETA 07:43:56 2021-05-09 14:56:44 [INFO] [TRAIN] epoch: 13, iter: 4650/40000, loss: 0.3187, lr: 0.008958, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 07:42:54 2021-05-09 14:56:52 [INFO] [TRAIN] epoch: 13, iter: 4660/40000, loss: 0.0742, lr: 0.008956, batch_cost: 0.7866, reader_cost: 0.00017, ips: 1.2713 samples/sec | ETA 07:43:19 2021-05-09 14:57:00 [INFO] [TRAIN] epoch: 13, iter: 4670/40000, loss: 0.4479, lr: 0.008954, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2733 samples/sec | ETA 07:42:26 2021-05-09 14:57:08 [INFO] [TRAIN] epoch: 13, iter: 4680/40000, loss: 0.3899, lr: 0.008951, batch_cost: 0.7869, reader_cost: 0.00015, ips: 1.2708 samples/sec | ETA 07:43:12 2021-05-09 14:57:16 [INFO] [TRAIN] epoch: 13, iter: 4690/40000, loss: 0.3825, lr: 0.008949, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2722 samples/sec | ETA 07:42:33 2021-05-09 14:57:23 [INFO] [TRAIN] epoch: 13, iter: 4700/40000, loss: 0.3667, lr: 0.008947, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2725 samples/sec | ETA 07:42:21 2021-05-09 14:57:31 [INFO] [TRAIN] epoch: 13, iter: 4710/40000, loss: 0.2355, lr: 0.008945, batch_cost: 0.7848, reader_cost: 0.00017, ips: 1.2742 samples/sec | ETA 07:41:35 2021-05-09 14:57:39 [INFO] [TRAIN] epoch: 13, iter: 4720/40000, loss: 0.4855, lr: 0.008942, batch_cost: 0.7864, reader_cost: 0.00017, ips: 1.2716 samples/sec | ETA 07:42:24 2021-05-09 14:57:47 [INFO] [TRAIN] epoch: 13, iter: 4730/40000, loss: 0.3663, lr: 0.008940, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2705 samples/sec | ETA 07:42:41 2021-05-09 14:57:55 [INFO] [TRAIN] epoch: 13, iter: 4740/40000, loss: 0.4853, lr: 0.008938, batch_cost: 0.7838, reader_cost: 0.00015, ips: 1.2759 samples/sec | ETA 07:40:35 2021-05-09 14:58:03 [INFO] [TRAIN] epoch: 13, iter: 4750/40000, loss: 0.6422, lr: 0.008936, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2729 samples/sec | ETA 07:41:33 2021-05-09 14:58:11 [INFO] [TRAIN] epoch: 13, iter: 4760/40000, loss: 0.5490, lr: 0.008933, batch_cost: 0.7845, reader_cost: 0.00015, ips: 1.2746 samples/sec | ETA 07:40:47 2021-05-09 14:58:18 [INFO] [TRAIN] epoch: 13, iter: 4770/40000, loss: 0.7009, lr: 0.008931, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2729 samples/sec | ETA 07:41:16 2021-05-09 14:58:26 [INFO] [TRAIN] epoch: 13, iter: 4780/40000, loss: 0.5525, lr: 0.008929, batch_cost: 0.7855, reader_cost: 0.00017, ips: 1.2731 samples/sec | ETA 07:41:05 2021-05-09 14:58:34 [INFO] [TRAIN] epoch: 13, iter: 4790/40000, loss: 0.5413, lr: 0.008927, batch_cost: 0.7859, reader_cost: 0.00017, ips: 1.2724 samples/sec | ETA 07:41:11 2021-05-09 14:58:42 [INFO] [TRAIN] epoch: 13, iter: 4800/40000, loss: 0.1908, lr: 0.008924, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2736 samples/sec | ETA 07:40:38 2021-05-09 14:58:50 [INFO] [TRAIN] epoch: 13, iter: 4810/40000, loss: 0.4789, lr: 0.008922, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2704 samples/sec | ETA 07:41:40 2021-05-09 14:58:58 [INFO] [TRAIN] epoch: 13, iter: 4820/40000, loss: 0.4702, lr: 0.008920, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2709 samples/sec | ETA 07:41:21 2021-05-09 14:59:06 [INFO] [TRAIN] epoch: 13, iter: 4830/40000, loss: 0.3463, lr: 0.008918, batch_cost: 0.7853, reader_cost: 0.00012, ips: 1.2734 samples/sec | ETA 07:40:19 2021-05-09 14:59:17 [INFO] [TRAIN] epoch: 14, iter: 4840/40000, loss: 0.6192, lr: 0.008915, batch_cost: 1.0886, reader_cost: 0.29975, ips: 0.9186 samples/sec | ETA 10:37:54 2021-05-09 14:59:24 [INFO] [TRAIN] epoch: 14, iter: 4850/40000, loss: 0.5014, lr: 0.008913, batch_cost: 0.7941, reader_cost: 0.00033, ips: 1.2593 samples/sec | ETA 07:45:12 2021-05-09 14:59:32 [INFO] [TRAIN] epoch: 14, iter: 4860/40000, loss: 0.6025, lr: 0.008911, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2734 samples/sec | ETA 07:39:54 2021-05-09 14:59:40 [INFO] [TRAIN] epoch: 14, iter: 4870/40000, loss: 0.4122, lr: 0.008909, batch_cost: 0.7856, reader_cost: 0.00017, ips: 1.2729 samples/sec | ETA 07:39:57 2021-05-09 14:59:48 [INFO] [TRAIN] epoch: 14, iter: 4880/40000, loss: 0.4853, lr: 0.008906, batch_cost: 0.7858, reader_cost: 0.00017, ips: 1.2726 samples/sec | ETA 07:39:58 2021-05-09 14:59:56 [INFO] [TRAIN] epoch: 14, iter: 4890/40000, loss: 0.5919, lr: 0.008904, batch_cost: 0.7871, reader_cost: 0.00017, ips: 1.2705 samples/sec | ETA 07:40:34 2021-05-09 15:00:04 [INFO] [TRAIN] epoch: 14, iter: 4900/40000, loss: 0.4737, lr: 0.008902, batch_cost: 0.7871, reader_cost: 0.00017, ips: 1.2705 samples/sec | ETA 07:40:27 2021-05-09 15:00:12 [INFO] [TRAIN] epoch: 14, iter: 4910/40000, loss: 0.2226, lr: 0.008899, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2713 samples/sec | ETA 07:40:01 2021-05-09 15:00:20 [INFO] [TRAIN] epoch: 14, iter: 4920/40000, loss: 0.5555, lr: 0.008897, batch_cost: 0.7873, reader_cost: 0.00016, ips: 1.2702 samples/sec | ETA 07:40:17 2021-05-09 15:00:27 [INFO] [TRAIN] epoch: 14, iter: 4930/40000, loss: 0.3276, lr: 0.008895, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2707 samples/sec | ETA 07:39:57 2021-05-09 15:00:35 [INFO] [TRAIN] epoch: 14, iter: 4940/40000, loss: 0.3849, lr: 0.008893, batch_cost: 0.7881, reader_cost: 0.00016, ips: 1.2689 samples/sec | ETA 07:40:30 2021-05-09 15:00:43 [INFO] [TRAIN] epoch: 14, iter: 4950/40000, loss: 0.4051, lr: 0.008890, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2708 samples/sec | ETA 07:39:40 2021-05-09 15:00:51 [INFO] [TRAIN] epoch: 14, iter: 4960/40000, loss: 0.1888, lr: 0.008888, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2720 samples/sec | ETA 07:39:07 2021-05-09 15:00:59 [INFO] [TRAIN] epoch: 14, iter: 4970/40000, loss: 0.3456, lr: 0.008886, batch_cost: 0.7863, reader_cost: 0.00014, ips: 1.2718 samples/sec | ETA 07:39:03 2021-05-09 15:01:07 [INFO] [TRAIN] epoch: 14, iter: 4980/40000, loss: 0.6301, lr: 0.008884, batch_cost: 0.7883, reader_cost: 0.00015, ips: 1.2685 samples/sec | ETA 07:40:06 2021-05-09 15:01:15 [INFO] [TRAIN] epoch: 14, iter: 4990/40000, loss: 0.4570, lr: 0.008881, batch_cost: 0.7873, reader_cost: 0.00015, ips: 1.2701 samples/sec | ETA 07:39:23 2021-05-09 15:01:22 [INFO] [TRAIN] epoch: 14, iter: 5000/40000, loss: 0.4079, lr: 0.008879, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2730 samples/sec | ETA 07:38:14 2021-05-09 15:01:23 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-09 15:04:53 [INFO] [EVAL] #Images: 500 mIoU: 0.6957 Acc: 0.9460 Kappa: 0.9299 2021-05-09 15:04:53 [INFO] [EVAL] Class IoU: [0.973 0.793 0.9005 0.5806 0.5004 0.4062 0.5293 0.6643 0.9058 0.6114 0.9334 0.7324 0.4841 0.9259 0.8241 0.8026 0.5503 0.4517 0.6494] 2021-05-09 15:04:53 [INFO] [EVAL] Class Acc: [0.9921 0.8405 0.9314 0.8396 0.7222 0.7654 0.8099 0.8772 0.9454 0.8477 0.9592 0.8175 0.6984 0.9564 0.9356 0.8569 0.9259 0.5756 0.8594] 2021-05-09 15:05:40 [INFO] [EVAL] The model with the best validation mIoU (0.6957) was saved at iter 5000. 2021-05-09 15:05:48 [INFO] [TRAIN] epoch: 14, iter: 5010/40000, loss: 0.2490, lr: 0.008877, batch_cost: 0.7829, reader_cost: 0.00026, ips: 1.2774 samples/sec | ETA 07:36:32 2021-05-09 15:05:56 [INFO] [TRAIN] epoch: 14, iter: 5020/40000, loss: 0.5072, lr: 0.008875, batch_cost: 0.7835, reader_cost: 0.00032, ips: 1.2763 samples/sec | ETA 07:36:47 2021-05-09 15:06:04 [INFO] [TRAIN] epoch: 14, iter: 5030/40000, loss: 0.1707, lr: 0.008872, batch_cost: 0.7853, reader_cost: 0.00015, ips: 1.2733 samples/sec | ETA 07:37:43 2021-05-09 15:06:12 [INFO] [TRAIN] epoch: 14, iter: 5040/40000, loss: 0.2629, lr: 0.008870, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2709 samples/sec | ETA 07:38:28 2021-05-09 15:06:19 [INFO] [TRAIN] epoch: 14, iter: 5050/40000, loss: 0.5075, lr: 0.008868, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2721 samples/sec | ETA 07:37:55 2021-05-09 15:06:27 [INFO] [TRAIN] epoch: 14, iter: 5060/40000, loss: 0.2640, lr: 0.008866, batch_cost: 0.7858, reader_cost: 0.00015, ips: 1.2725 samples/sec | ETA 07:37:37 2021-05-09 15:06:35 [INFO] [TRAIN] epoch: 14, iter: 5070/40000, loss: 0.3596, lr: 0.008863, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2724 samples/sec | ETA 07:37:32 2021-05-09 15:06:43 [INFO] [TRAIN] epoch: 14, iter: 5080/40000, loss: 0.4450, lr: 0.008861, batch_cost: 0.7879, reader_cost: 0.00016, ips: 1.2693 samples/sec | ETA 07:38:32 2021-05-09 15:06:51 [INFO] [TRAIN] epoch: 14, iter: 5090/40000, loss: 0.3038, lr: 0.008859, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 07:37:10 2021-05-09 15:06:59 [INFO] [TRAIN] epoch: 14, iter: 5100/40000, loss: 0.2431, lr: 0.008857, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2729 samples/sec | ETA 07:36:58 2021-05-09 15:07:07 [INFO] [TRAIN] epoch: 14, iter: 5110/40000, loss: 0.4360, lr: 0.008854, batch_cost: 0.7872, reader_cost: 0.00015, ips: 1.2704 samples/sec | ETA 07:37:44 2021-05-09 15:07:15 [INFO] [TRAIN] epoch: 14, iter: 5120/40000, loss: 0.5288, lr: 0.008852, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2716 samples/sec | ETA 07:37:09 2021-05-09 15:07:22 [INFO] [TRAIN] epoch: 14, iter: 5130/40000, loss: 0.4628, lr: 0.008850, batch_cost: 0.7857, reader_cost: 0.00014, ips: 1.2728 samples/sec | ETA 07:36:36 2021-05-09 15:07:30 [INFO] [TRAIN] epoch: 14, iter: 5140/40000, loss: 0.6580, lr: 0.008848, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2719 samples/sec | ETA 07:36:48 2021-05-09 15:07:38 [INFO] [TRAIN] epoch: 14, iter: 5150/40000, loss: 0.8718, lr: 0.008845, batch_cost: 0.7887, reader_cost: 0.00018, ips: 1.2680 samples/sec | ETA 07:38:04 2021-05-09 15:07:46 [INFO] [TRAIN] epoch: 14, iter: 5160/40000, loss: 0.4528, lr: 0.008843, batch_cost: 0.7866, reader_cost: 0.00019, ips: 1.2712 samples/sec | ETA 07:36:46 2021-05-09 15:07:54 [INFO] [TRAIN] epoch: 14, iter: 5170/40000, loss: 0.1337, lr: 0.008841, batch_cost: 0.7862, reader_cost: 0.00018, ips: 1.2720 samples/sec | ETA 07:36:22 2021-05-09 15:08:02 [INFO] [TRAIN] epoch: 14, iter: 5180/40000, loss: 0.3621, lr: 0.008839, batch_cost: 0.7883, reader_cost: 0.00017, ips: 1.2686 samples/sec | ETA 07:37:27 2021-05-09 15:08:10 [INFO] [TRAIN] epoch: 14, iter: 5190/40000, loss: 0.3687, lr: 0.008836, batch_cost: 0.7867, reader_cost: 0.00017, ips: 1.2712 samples/sec | ETA 07:36:23 2021-05-09 15:08:17 [INFO] [TRAIN] epoch: 14, iter: 5200/40000, loss: 0.4451, lr: 0.008834, batch_cost: 0.7857, reader_cost: 0.00014, ips: 1.2727 samples/sec | ETA 07:35:43 2021-05-09 15:08:28 [INFO] [TRAIN] epoch: 15, iter: 5210/40000, loss: 0.7190, lr: 0.008832, batch_cost: 1.0925, reader_cost: 0.24760, ips: 0.9153 samples/sec | ETA 10:33:28 2021-05-09 15:08:36 [INFO] [TRAIN] epoch: 15, iter: 5220/40000, loss: 0.3137, lr: 0.008829, batch_cost: 0.8002, reader_cost: 0.00034, ips: 1.2497 samples/sec | ETA 07:43:51 2021-05-09 15:08:44 [INFO] [TRAIN] epoch: 15, iter: 5230/40000, loss: 0.7341, lr: 0.008827, batch_cost: 0.7846, reader_cost: 0.00017, ips: 1.2745 samples/sec | ETA 07:34:41 2021-05-09 15:08:52 [INFO] [TRAIN] epoch: 15, iter: 5240/40000, loss: 0.5173, lr: 0.008825, batch_cost: 0.7867, reader_cost: 0.00018, ips: 1.2711 samples/sec | ETA 07:35:47 2021-05-09 15:09:00 [INFO] [TRAIN] epoch: 15, iter: 5250/40000, loss: 0.5894, lr: 0.008823, batch_cost: 0.7842, reader_cost: 0.00016, ips: 1.2753 samples/sec | ETA 07:34:09 2021-05-09 15:09:08 [INFO] [TRAIN] epoch: 15, iter: 5260/40000, loss: 0.6496, lr: 0.008820, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2710 samples/sec | ETA 07:35:33 2021-05-09 15:09:16 [INFO] [TRAIN] epoch: 15, iter: 5270/40000, loss: 0.6802, lr: 0.008818, batch_cost: 0.7861, reader_cost: 0.00017, ips: 1.2720 samples/sec | ETA 07:35:02 2021-05-09 15:09:24 [INFO] [TRAIN] epoch: 15, iter: 5280/40000, loss: 0.3436, lr: 0.008816, batch_cost: 0.7856, reader_cost: 0.00014, ips: 1.2729 samples/sec | ETA 07:34:36 2021-05-09 15:09:31 [INFO] [TRAIN] epoch: 15, iter: 5290/40000, loss: 0.5701, lr: 0.008814, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2725 samples/sec | ETA 07:34:36 2021-05-09 15:09:39 [INFO] [TRAIN] epoch: 15, iter: 5300/40000, loss: 0.4767, lr: 0.008811, batch_cost: 0.7855, reader_cost: 0.00018, ips: 1.2731 samples/sec | ETA 07:34:15 2021-05-09 15:09:47 [INFO] [TRAIN] epoch: 15, iter: 5310/40000, loss: 0.4527, lr: 0.008809, batch_cost: 0.7857, reader_cost: 0.00017, ips: 1.2727 samples/sec | ETA 07:34:16 2021-05-09 15:09:55 [INFO] [TRAIN] epoch: 15, iter: 5320/40000, loss: 0.4188, lr: 0.008807, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2732 samples/sec | ETA 07:33:57 2021-05-09 15:10:03 [INFO] [TRAIN] epoch: 15, iter: 5330/40000, loss: 0.1623, lr: 0.008805, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 07:34:08 2021-05-09 15:10:11 [INFO] [TRAIN] epoch: 15, iter: 5340/40000, loss: 0.5103, lr: 0.008802, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 07:34:07 2021-05-09 15:10:19 [INFO] [TRAIN] epoch: 15, iter: 5350/40000, loss: 0.5537, lr: 0.008800, batch_cost: 0.7849, reader_cost: 0.00016, ips: 1.2740 samples/sec | ETA 07:33:18 2021-05-09 15:10:26 [INFO] [TRAIN] epoch: 15, iter: 5360/40000, loss: 0.4130, lr: 0.008798, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2714 samples/sec | ETA 07:34:06 2021-05-09 15:10:34 [INFO] [TRAIN] epoch: 15, iter: 5370/40000, loss: 0.4250, lr: 0.008796, batch_cost: 0.7847, reader_cost: 0.00016, ips: 1.2744 samples/sec | ETA 07:32:54 2021-05-09 15:10:42 [INFO] [TRAIN] epoch: 15, iter: 5380/40000, loss: 0.4078, lr: 0.008793, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2711 samples/sec | ETA 07:33:55 2021-05-09 15:10:50 [INFO] [TRAIN] epoch: 15, iter: 5390/40000, loss: 0.5142, lr: 0.008791, batch_cost: 0.7858, reader_cost: 0.00017, ips: 1.2727 samples/sec | ETA 07:33:15 2021-05-09 15:10:58 [INFO] [TRAIN] epoch: 15, iter: 5400/40000, loss: 0.2274, lr: 0.008789, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2724 samples/sec | ETA 07:33:12 2021-05-09 15:11:06 [INFO] [TRAIN] epoch: 15, iter: 5410/40000, loss: 0.2930, lr: 0.008787, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2735 samples/sec | ETA 07:32:40 2021-05-09 15:11:14 [INFO] [TRAIN] epoch: 15, iter: 5420/40000, loss: 0.4629, lr: 0.008784, batch_cost: 0.7853, reader_cost: 0.00015, ips: 1.2734 samples/sec | ETA 07:32:36 2021-05-09 15:11:21 [INFO] [TRAIN] epoch: 15, iter: 5430/40000, loss: 0.3654, lr: 0.008782, batch_cost: 0.7854, reader_cost: 0.00015, ips: 1.2733 samples/sec | ETA 07:32:30 2021-05-09 15:11:29 [INFO] [TRAIN] epoch: 15, iter: 5440/40000, loss: 0.5002, lr: 0.008780, batch_cost: 0.7841, reader_cost: 0.00016, ips: 1.2754 samples/sec | ETA 07:31:37 2021-05-09 15:11:37 [INFO] [TRAIN] epoch: 15, iter: 5450/40000, loss: 0.2987, lr: 0.008778, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2735 samples/sec | ETA 07:32:09 2021-05-09 15:11:45 [INFO] [TRAIN] epoch: 15, iter: 5460/40000, loss: 0.3190, lr: 0.008775, batch_cost: 0.7876, reader_cost: 0.00016, ips: 1.2696 samples/sec | ETA 07:33:24 2021-05-09 15:11:53 [INFO] [TRAIN] epoch: 15, iter: 5470/40000, loss: 0.4060, lr: 0.008773, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 07:32:22 2021-05-09 15:12:01 [INFO] [TRAIN] epoch: 15, iter: 5480/40000, loss: 0.3893, lr: 0.008771, batch_cost: 0.7861, reader_cost: 0.00017, ips: 1.2722 samples/sec | ETA 07:32:14 2021-05-09 15:12:09 [INFO] [TRAIN] epoch: 15, iter: 5490/40000, loss: 0.6099, lr: 0.008768, batch_cost: 0.7869, reader_cost: 0.00018, ips: 1.2708 samples/sec | ETA 07:32:36 2021-05-09 15:12:16 [INFO] [TRAIN] epoch: 15, iter: 5500/40000, loss: 0.4705, lr: 0.008766, batch_cost: 0.7883, reader_cost: 0.00017, ips: 1.2685 samples/sec | ETA 07:33:17 2021-05-09 15:12:24 [INFO] [TRAIN] epoch: 15, iter: 5510/40000, loss: 0.6106, lr: 0.008764, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2718 samples/sec | ETA 07:31:59 2021-05-09 15:12:32 [INFO] [TRAIN] epoch: 15, iter: 5520/40000, loss: 0.5052, lr: 0.008762, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 07:32:03 2021-05-09 15:12:40 [INFO] [TRAIN] epoch: 15, iter: 5530/40000, loss: 0.2975, lr: 0.008759, batch_cost: 0.7859, reader_cost: 0.00017, ips: 1.2725 samples/sec | ETA 07:31:28 2021-05-09 15:12:48 [INFO] [TRAIN] epoch: 15, iter: 5540/40000, loss: 0.1807, lr: 0.008757, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2723 samples/sec | ETA 07:31:24 2021-05-09 15:12:56 [INFO] [TRAIN] epoch: 15, iter: 5550/40000, loss: 0.2729, lr: 0.008755, batch_cost: 0.7864, reader_cost: 0.00017, ips: 1.2716 samples/sec | ETA 07:31:31 2021-05-09 15:13:04 [INFO] [TRAIN] epoch: 15, iter: 5560/40000, loss: 0.3130, lr: 0.008753, batch_cost: 0.7849, reader_cost: 0.00016, ips: 1.2741 samples/sec | ETA 07:30:30 2021-05-09 15:13:11 [INFO] [TRAIN] epoch: 15, iter: 5570/40000, loss: 0.3347, lr: 0.008750, batch_cost: 0.7867, reader_cost: 0.00015, ips: 1.2712 samples/sec | ETA 07:31:25 2021-05-09 15:13:19 [INFO] [TRAIN] epoch: 15, iter: 5580/40000, loss: 0.5410, lr: 0.008748, batch_cost: 0.7864, reader_cost: 0.00010, ips: 1.2716 samples/sec | ETA 07:31:07 2021-05-09 15:13:30 [INFO] [TRAIN] epoch: 16, iter: 5590/40000, loss: 0.3493, lr: 0.008746, batch_cost: 1.0921, reader_cost: 0.23162, ips: 0.9156 samples/sec | ETA 10:26:20 2021-05-09 15:13:38 [INFO] [TRAIN] epoch: 16, iter: 5600/40000, loss: 0.5405, lr: 0.008744, batch_cost: 0.7896, reader_cost: 0.00031, ips: 1.2665 samples/sec | ETA 07:32:41 2021-05-09 15:13:46 [INFO] [TRAIN] epoch: 16, iter: 5610/40000, loss: 0.3343, lr: 0.008741, batch_cost: 0.7875, reader_cost: 0.00018, ips: 1.2698 samples/sec | ETA 07:31:23 2021-05-09 15:13:54 [INFO] [TRAIN] epoch: 16, iter: 5620/40000, loss: 0.5764, lr: 0.008739, batch_cost: 0.7868, reader_cost: 0.00017, ips: 1.2710 samples/sec | ETA 07:30:49 2021-05-09 15:14:02 [INFO] [TRAIN] epoch: 16, iter: 5630/40000, loss: 0.6289, lr: 0.008737, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 07:29:57 2021-05-09 15:14:10 [INFO] [TRAIN] epoch: 16, iter: 5640/40000, loss: 1.0962, lr: 0.008735, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 07:30:08 2021-05-09 15:14:17 [INFO] [TRAIN] epoch: 16, iter: 5650/40000, loss: 0.2301, lr: 0.008732, batch_cost: 0.7873, reader_cost: 0.00015, ips: 1.2702 samples/sec | ETA 07:30:43 2021-05-09 15:14:25 [INFO] [TRAIN] epoch: 16, iter: 5660/40000, loss: 0.5186, lr: 0.008730, batch_cost: 0.7883, reader_cost: 0.00016, ips: 1.2686 samples/sec | ETA 07:31:08 2021-05-09 15:14:33 [INFO] [TRAIN] epoch: 16, iter: 5670/40000, loss: 0.4354, lr: 0.008728, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2729 samples/sec | ETA 07:29:30 2021-05-09 15:14:41 [INFO] [TRAIN] epoch: 16, iter: 5680/40000, loss: 0.2952, lr: 0.008726, batch_cost: 0.7851, reader_cost: 0.00017, ips: 1.2736 samples/sec | ETA 07:29:06 2021-05-09 15:14:49 [INFO] [TRAIN] epoch: 16, iter: 5690/40000, loss: 0.3227, lr: 0.008723, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2711 samples/sec | ETA 07:29:52 2021-05-09 15:14:57 [INFO] [TRAIN] epoch: 16, iter: 5700/40000, loss: 0.1573, lr: 0.008721, batch_cost: 0.7868, reader_cost: 0.00017, ips: 1.2709 samples/sec | ETA 07:29:47 2021-05-09 15:15:05 [INFO] [TRAIN] epoch: 16, iter: 5710/40000, loss: 0.2752, lr: 0.008719, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 07:28:53 2021-05-09 15:15:13 [INFO] [TRAIN] epoch: 16, iter: 5720/40000, loss: 0.4363, lr: 0.008716, batch_cost: 0.7844, reader_cost: 0.00016, ips: 1.2749 samples/sec | ETA 07:28:08 2021-05-09 15:15:20 [INFO] [TRAIN] epoch: 16, iter: 5730/40000, loss: 0.4202, lr: 0.008714, batch_cost: 0.7840, reader_cost: 0.00016, ips: 1.2755 samples/sec | ETA 07:27:48 2021-05-09 15:15:28 [INFO] [TRAIN] epoch: 16, iter: 5740/40000, loss: 0.3063, lr: 0.008712, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 07:28:50 2021-05-09 15:15:36 [INFO] [TRAIN] epoch: 16, iter: 5750/40000, loss: 0.3057, lr: 0.008710, batch_cost: 0.7862, reader_cost: 0.00017, ips: 1.2720 samples/sec | ETA 07:28:47 2021-05-09 15:15:44 [INFO] [TRAIN] epoch: 16, iter: 5760/40000, loss: 0.3605, lr: 0.008707, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2714 samples/sec | ETA 07:28:49 2021-05-09 15:15:52 [INFO] [TRAIN] epoch: 16, iter: 5770/40000, loss: 0.7345, lr: 0.008705, batch_cost: 0.7842, reader_cost: 0.00015, ips: 1.2752 samples/sec | ETA 07:27:22 2021-05-09 15:16:00 [INFO] [TRAIN] epoch: 16, iter: 5780/40000, loss: 0.1159, lr: 0.008703, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2734 samples/sec | ETA 07:27:53 2021-05-09 15:16:08 [INFO] [TRAIN] epoch: 16, iter: 5790/40000, loss: 0.4795, lr: 0.008701, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2735 samples/sec | ETA 07:27:43 2021-05-09 15:16:15 [INFO] [TRAIN] epoch: 16, iter: 5800/40000, loss: 0.3909, lr: 0.008698, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2736 samples/sec | ETA 07:27:33 2021-05-09 15:16:23 [INFO] [TRAIN] epoch: 16, iter: 5810/40000, loss: 0.5323, lr: 0.008696, batch_cost: 0.7852, reader_cost: 0.00014, ips: 1.2736 samples/sec | ETA 07:27:24 2021-05-09 15:16:31 [INFO] [TRAIN] epoch: 16, iter: 5820/40000, loss: 0.3969, lr: 0.008694, batch_cost: 0.7847, reader_cost: 0.00015, ips: 1.2744 samples/sec | ETA 07:27:00 2021-05-09 15:16:39 [INFO] [TRAIN] epoch: 16, iter: 5830/40000, loss: 0.4040, lr: 0.008692, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2720 samples/sec | ETA 07:27:43 2021-05-09 15:16:47 [INFO] [TRAIN] epoch: 16, iter: 5840/40000, loss: 0.2476, lr: 0.008689, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2719 samples/sec | ETA 07:27:38 2021-05-09 15:16:55 [INFO] [TRAIN] epoch: 16, iter: 5850/40000, loss: 0.3179, lr: 0.008687, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2723 samples/sec | ETA 07:27:22 2021-05-09 15:17:03 [INFO] [TRAIN] epoch: 16, iter: 5860/40000, loss: 0.6068, lr: 0.008685, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2730 samples/sec | ETA 07:26:59 2021-05-09 15:17:10 [INFO] [TRAIN] epoch: 16, iter: 5870/40000, loss: 0.3965, lr: 0.008683, batch_cost: 0.7855, reader_cost: 0.00017, ips: 1.2731 samples/sec | ETA 07:26:48 2021-05-09 15:17:18 [INFO] [TRAIN] epoch: 16, iter: 5880/40000, loss: 0.4404, lr: 0.008680, batch_cost: 0.7850, reader_cost: 0.00015, ips: 1.2739 samples/sec | ETA 07:26:23 2021-05-09 15:17:26 [INFO] [TRAIN] epoch: 16, iter: 5890/40000, loss: 0.6186, lr: 0.008678, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2735 samples/sec | ETA 07:26:24 2021-05-09 15:17:34 [INFO] [TRAIN] epoch: 16, iter: 5900/40000, loss: 0.4693, lr: 0.008676, batch_cost: 0.7864, reader_cost: 0.00017, ips: 1.2716 samples/sec | ETA 07:26:56 2021-05-09 15:17:42 [INFO] [TRAIN] epoch: 16, iter: 5910/40000, loss: 0.3992, lr: 0.008673, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2715 samples/sec | ETA 07:26:50 2021-05-09 15:17:50 [INFO] [TRAIN] epoch: 16, iter: 5920/40000, loss: 0.3350, lr: 0.008671, batch_cost: 0.7878, reader_cost: 0.00016, ips: 1.2693 samples/sec | ETA 07:27:28 2021-05-09 15:17:58 [INFO] [TRAIN] epoch: 16, iter: 5930/40000, loss: 0.3867, lr: 0.008669, batch_cost: 0.7866, reader_cost: 0.00017, ips: 1.2713 samples/sec | ETA 07:26:38 2021-05-09 15:18:05 [INFO] [TRAIN] epoch: 16, iter: 5940/40000, loss: 0.4733, lr: 0.008667, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2731 samples/sec | ETA 07:25:54 2021-05-09 15:18:13 [INFO] [TRAIN] epoch: 16, iter: 5950/40000, loss: 0.5153, lr: 0.008664, batch_cost: 0.7845, reader_cost: 0.00025, ips: 1.2747 samples/sec | ETA 07:25:11 2021-05-09 15:18:24 [INFO] [TRAIN] epoch: 17, iter: 5960/40000, loss: 0.5975, lr: 0.008662, batch_cost: 1.0883, reader_cost: 0.23321, ips: 0.9188 samples/sec | ETA 10:17:27 2021-05-09 15:18:32 [INFO] [TRAIN] epoch: 17, iter: 5970/40000, loss: 0.4128, lr: 0.008660, batch_cost: 0.7911, reader_cost: 0.00034, ips: 1.2641 samples/sec | ETA 07:28:40 2021-05-09 15:18:40 [INFO] [TRAIN] epoch: 17, iter: 5980/40000, loss: 0.6989, lr: 0.008658, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2706 samples/sec | ETA 07:26:14 2021-05-09 15:18:48 [INFO] [TRAIN] epoch: 17, iter: 5990/40000, loss: 0.4174, lr: 0.008655, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 07:25:47 2021-05-09 15:18:56 [INFO] [TRAIN] epoch: 17, iter: 6000/40000, loss: 0.6053, lr: 0.008653, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 07:25:45 2021-05-09 15:18:56 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-09 15:22:26 [INFO] [EVAL] #Images: 500 mIoU: 0.7286 Acc: 0.9488 Kappa: 0.9336 2021-05-09 15:22:26 [INFO] [EVAL] Class IoU: [0.9741 0.8006 0.9078 0.6013 0.5935 0.4477 0.5487 0.6629 0.9065 0.6091 0.9361 0.7447 0.5226 0.9271 0.8256 0.8557 0.7406 0.5404 0.6982] 2021-05-09 15:22:26 [INFO] [EVAL] Class Acc: [0.9943 0.8471 0.9468 0.8642 0.7484 0.6944 0.7804 0.8905 0.9391 0.8584 0.9654 0.8381 0.6885 0.9526 0.9381 0.924 0.9007 0.8494 0.8238] 2021-05-09 15:23:14 [INFO] [EVAL] The model with the best validation mIoU (0.7286) was saved at iter 6000. 2021-05-09 15:23:22 [INFO] [TRAIN] epoch: 17, iter: 6010/40000, loss: 0.2803, lr: 0.008651, batch_cost: 0.7811, reader_cost: 0.00043, ips: 1.2803 samples/sec | ETA 07:22:29 2021-05-09 15:23:30 [INFO] [TRAIN] epoch: 17, iter: 6020/40000, loss: 0.2814, lr: 0.008649, batch_cost: 0.7830, reader_cost: 0.00016, ips: 1.2772 samples/sec | ETA 07:23:25 2021-05-09 15:23:38 [INFO] [TRAIN] epoch: 17, iter: 6030/40000, loss: 0.3490, lr: 0.008646, batch_cost: 0.7859, reader_cost: 0.00018, ips: 1.2725 samples/sec | ETA 07:24:56 2021-05-09 15:23:46 [INFO] [TRAIN] epoch: 17, iter: 6040/40000, loss: 0.3398, lr: 0.008644, batch_cost: 0.7848, reader_cost: 0.00017, ips: 1.2742 samples/sec | ETA 07:24:11 2021-05-09 15:23:54 [INFO] [TRAIN] epoch: 17, iter: 6050/40000, loss: 0.4180, lr: 0.008642, batch_cost: 0.7868, reader_cost: 0.00018, ips: 1.2709 samples/sec | ETA 07:25:12 2021-05-09 15:24:01 [INFO] [TRAIN] epoch: 17, iter: 6060/40000, loss: 0.4081, lr: 0.008640, batch_cost: 0.7854, reader_cost: 0.00017, ips: 1.2733 samples/sec | ETA 07:24:15 2021-05-09 15:24:09 [INFO] [TRAIN] epoch: 17, iter: 6070/40000, loss: 0.1876, lr: 0.008637, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2704 samples/sec | ETA 07:25:08 2021-05-09 15:24:17 [INFO] [TRAIN] epoch: 17, iter: 6080/40000, loss: 0.3196, lr: 0.008635, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 07:24:37 2021-05-09 15:24:25 [INFO] [TRAIN] epoch: 17, iter: 6090/40000, loss: 0.3880, lr: 0.008633, batch_cost: 0.7880, reader_cost: 0.00016, ips: 1.2690 samples/sec | ETA 07:25:21 2021-05-09 15:24:33 [INFO] [TRAIN] epoch: 17, iter: 6100/40000, loss: 0.3717, lr: 0.008630, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2714 samples/sec | ETA 07:24:22 2021-05-09 15:24:41 [INFO] [TRAIN] epoch: 17, iter: 6110/40000, loss: 0.3583, lr: 0.008628, batch_cost: 0.7862, reader_cost: 0.00017, ips: 1.2719 samples/sec | ETA 07:24:05 2021-05-09 15:24:49 [INFO] [TRAIN] epoch: 17, iter: 6120/40000, loss: 0.2527, lr: 0.008626, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 07:23:52 2021-05-09 15:24:57 [INFO] [TRAIN] epoch: 17, iter: 6130/40000, loss: 0.5492, lr: 0.008624, batch_cost: 0.7889, reader_cost: 0.00015, ips: 1.2676 samples/sec | ETA 07:25:20 2021-05-09 15:25:04 [INFO] [TRAIN] epoch: 17, iter: 6140/40000, loss: 0.4367, lr: 0.008621, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 07:23:48 2021-05-09 15:25:12 [INFO] [TRAIN] epoch: 17, iter: 6150/40000, loss: 0.2408, lr: 0.008619, batch_cost: 0.7862, reader_cost: 0.00017, ips: 1.2719 samples/sec | ETA 07:23:32 2021-05-09 15:25:20 [INFO] [TRAIN] epoch: 17, iter: 6160/40000, loss: 0.4937, lr: 0.008617, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2716 samples/sec | ETA 07:23:31 2021-05-09 15:25:28 [INFO] [TRAIN] epoch: 17, iter: 6170/40000, loss: 0.3564, lr: 0.008615, batch_cost: 0.7851, reader_cost: 0.00018, ips: 1.2737 samples/sec | ETA 07:22:40 2021-05-09 15:25:36 [INFO] [TRAIN] epoch: 17, iter: 6180/40000, loss: 0.3120, lr: 0.008612, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2709 samples/sec | ETA 07:23:31 2021-05-09 15:25:44 [INFO] [TRAIN] epoch: 17, iter: 6190/40000, loss: 0.3648, lr: 0.008610, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2724 samples/sec | ETA 07:22:52 2021-05-09 15:25:52 [INFO] [TRAIN] epoch: 17, iter: 6200/40000, loss: 0.2270, lr: 0.008608, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2733 samples/sec | ETA 07:22:24 2021-05-09 15:25:59 [INFO] [TRAIN] epoch: 17, iter: 6210/40000, loss: 0.3477, lr: 0.008606, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2721 samples/sec | ETA 07:22:41 2021-05-09 15:26:07 [INFO] [TRAIN] epoch: 17, iter: 6220/40000, loss: 0.3048, lr: 0.008603, batch_cost: 0.7863, reader_cost: 0.00017, ips: 1.2717 samples/sec | ETA 07:22:42 2021-05-09 15:26:15 [INFO] [TRAIN] epoch: 17, iter: 6230/40000, loss: 0.5015, lr: 0.008601, batch_cost: 0.7846, reader_cost: 0.00016, ips: 1.2746 samples/sec | ETA 07:21:34 2021-05-09 15:26:23 [INFO] [TRAIN] epoch: 17, iter: 6240/40000, loss: 0.4704, lr: 0.008599, batch_cost: 0.7845, reader_cost: 0.00015, ips: 1.2748 samples/sec | ETA 07:21:23 2021-05-09 15:26:31 [INFO] [TRAIN] epoch: 17, iter: 6250/40000, loss: 0.4421, lr: 0.008596, batch_cost: 0.7854, reader_cost: 0.00015, ips: 1.2733 samples/sec | ETA 07:21:46 2021-05-09 15:26:39 [INFO] [TRAIN] epoch: 17, iter: 6260/40000, loss: 0.7429, lr: 0.008594, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2718 samples/sec | ETA 07:22:09 2021-05-09 15:26:47 [INFO] [TRAIN] epoch: 17, iter: 6270/40000, loss: 0.3743, lr: 0.008592, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2720 samples/sec | ETA 07:21:58 2021-05-09 15:26:54 [INFO] [TRAIN] epoch: 17, iter: 6280/40000, loss: 0.1504, lr: 0.008590, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2711 samples/sec | ETA 07:22:08 2021-05-09 15:27:02 [INFO] [TRAIN] epoch: 17, iter: 6290/40000, loss: 0.2957, lr: 0.008587, batch_cost: 0.7875, reader_cost: 0.00016, ips: 1.2698 samples/sec | ETA 07:22:28 2021-05-09 15:27:10 [INFO] [TRAIN] epoch: 17, iter: 6300/40000, loss: 0.3361, lr: 0.008585, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2713 samples/sec | ETA 07:21:49 2021-05-09 15:27:18 [INFO] [TRAIN] epoch: 17, iter: 6310/40000, loss: 0.4230, lr: 0.008583, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2730 samples/sec | ETA 07:21:05 2021-05-09 15:27:26 [INFO] [TRAIN] epoch: 17, iter: 6320/40000, loss: 0.3316, lr: 0.008581, batch_cost: 0.7842, reader_cost: 0.00013, ips: 1.2752 samples/sec | ETA 07:20:12 2021-05-09 15:27:37 [INFO] [TRAIN] epoch: 18, iter: 6330/40000, loss: 0.3661, lr: 0.008578, batch_cost: 1.0990, reader_cost: 0.27638, ips: 0.9099 samples/sec | ETA 10:16:44 2021-05-09 15:27:45 [INFO] [TRAIN] epoch: 18, iter: 6340/40000, loss: 0.3858, lr: 0.008576, batch_cost: 0.8010, reader_cost: 0.00032, ips: 1.2485 samples/sec | ETA 07:29:20 2021-05-09 15:27:53 [INFO] [TRAIN] epoch: 18, iter: 6350/40000, loss: 0.5324, lr: 0.008574, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2730 samples/sec | ETA 07:20:34 2021-05-09 15:28:01 [INFO] [TRAIN] epoch: 18, iter: 6360/40000, loss: 0.4674, lr: 0.008572, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2725 samples/sec | ETA 07:20:37 2021-05-09 15:28:08 [INFO] [TRAIN] epoch: 18, iter: 6370/40000, loss: 0.4267, lr: 0.008569, batch_cost: 0.7845, reader_cost: 0.00017, ips: 1.2747 samples/sec | ETA 07:19:42 2021-05-09 15:28:16 [INFO] [TRAIN] epoch: 18, iter: 6380/40000, loss: 0.4776, lr: 0.008567, batch_cost: 0.7877, reader_cost: 0.00016, ips: 1.2695 samples/sec | ETA 07:21:23 2021-05-09 15:28:24 [INFO] [TRAIN] epoch: 18, iter: 6390/40000, loss: 0.5405, lr: 0.008565, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2706 samples/sec | ETA 07:20:52 2021-05-09 15:28:32 [INFO] [TRAIN] epoch: 18, iter: 6400/40000, loss: 0.1531, lr: 0.008562, batch_cost: 0.7870, reader_cost: 0.00015, ips: 1.2707 samples/sec | ETA 07:20:42 2021-05-09 15:28:40 [INFO] [TRAIN] epoch: 18, iter: 6410/40000, loss: 0.5010, lr: 0.008560, batch_cost: 0.7856, reader_cost: 0.00018, ips: 1.2730 samples/sec | ETA 07:19:46 2021-05-09 15:28:48 [INFO] [TRAIN] epoch: 18, iter: 6420/40000, loss: 0.4034, lr: 0.008558, batch_cost: 0.7871, reader_cost: 0.00015, ips: 1.2705 samples/sec | ETA 07:20:31 2021-05-09 15:28:56 [INFO] [TRAIN] epoch: 18, iter: 6430/40000, loss: 0.3619, lr: 0.008556, batch_cost: 0.7874, reader_cost: 0.00016, ips: 1.2700 samples/sec | ETA 07:20:33 2021-05-09 15:29:03 [INFO] [TRAIN] epoch: 18, iter: 6440/40000, loss: 0.3055, lr: 0.008553, batch_cost: 0.7870, reader_cost: 0.00015, ips: 1.2707 samples/sec | ETA 07:20:11 2021-05-09 15:29:11 [INFO] [TRAIN] epoch: 18, iter: 6450/40000, loss: 0.2290, lr: 0.008551, batch_cost: 0.7878, reader_cost: 0.00014, ips: 1.2693 samples/sec | ETA 07:20:30 2021-05-09 15:29:19 [INFO] [TRAIN] epoch: 18, iter: 6460/40000, loss: 0.5259, lr: 0.008549, batch_cost: 0.7867, reader_cost: 0.00015, ips: 1.2711 samples/sec | ETA 07:19:47 2021-05-09 15:29:27 [INFO] [TRAIN] epoch: 18, iter: 6470/40000, loss: 0.6451, lr: 0.008547, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 07:19:11 2021-05-09 15:29:35 [INFO] [TRAIN] epoch: 18, iter: 6480/40000, loss: 0.3037, lr: 0.008544, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2736 samples/sec | ETA 07:18:38 2021-05-09 15:29:43 [INFO] [TRAIN] epoch: 18, iter: 6490/40000, loss: 0.2847, lr: 0.008542, batch_cost: 0.7883, reader_cost: 0.00016, ips: 1.2686 samples/sec | ETA 07:20:15 2021-05-09 15:29:51 [INFO] [TRAIN] epoch: 18, iter: 6500/40000, loss: 0.3628, lr: 0.008540, batch_cost: 0.7854, reader_cost: 0.00017, ips: 1.2732 samples/sec | ETA 07:18:31 2021-05-09 15:29:59 [INFO] [TRAIN] epoch: 18, iter: 6510/40000, loss: 0.2306, lr: 0.008538, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2703 samples/sec | ETA 07:19:24 2021-05-09 15:30:06 [INFO] [TRAIN] epoch: 18, iter: 6520/40000, loss: 0.2125, lr: 0.008535, batch_cost: 0.7869, reader_cost: 0.00017, ips: 1.2708 samples/sec | ETA 07:19:05 2021-05-09 15:30:14 [INFO] [TRAIN] epoch: 18, iter: 6530/40000, loss: 0.2859, lr: 0.008533, batch_cost: 0.7880, reader_cost: 0.00016, ips: 1.2691 samples/sec | ETA 07:19:32 2021-05-09 15:30:22 [INFO] [TRAIN] epoch: 18, iter: 6540/40000, loss: 0.3450, lr: 0.008531, batch_cost: 0.7843, reader_cost: 0.00015, ips: 1.2750 samples/sec | ETA 07:17:22 2021-05-09 15:30:30 [INFO] [TRAIN] epoch: 18, iter: 6550/40000, loss: 0.2506, lr: 0.008528, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2728 samples/sec | ETA 07:17:59 2021-05-09 15:30:38 [INFO] [TRAIN] epoch: 18, iter: 6560/40000, loss: 0.3937, lr: 0.008526, batch_cost: 0.7840, reader_cost: 0.00015, ips: 1.2756 samples/sec | ETA 07:16:55 2021-05-09 15:30:46 [INFO] [TRAIN] epoch: 18, iter: 6570/40000, loss: 0.5074, lr: 0.008524, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2716 samples/sec | ETA 07:18:10 2021-05-09 15:30:54 [INFO] [TRAIN] epoch: 18, iter: 6580/40000, loss: 0.2190, lr: 0.008522, batch_cost: 0.7859, reader_cost: 0.00017, ips: 1.2724 samples/sec | ETA 07:17:45 2021-05-09 15:31:01 [INFO] [TRAIN] epoch: 18, iter: 6590/40000, loss: 0.2577, lr: 0.008519, batch_cost: 0.7846, reader_cost: 0.00016, ips: 1.2745 samples/sec | ETA 07:16:53 2021-05-09 15:31:09 [INFO] [TRAIN] epoch: 18, iter: 6600/40000, loss: 0.3628, lr: 0.008517, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 07:17:15 2021-05-09 15:31:17 [INFO] [TRAIN] epoch: 18, iter: 6610/40000, loss: 0.5725, lr: 0.008515, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2720 samples/sec | ETA 07:17:29 2021-05-09 15:31:25 [INFO] [TRAIN] epoch: 18, iter: 6620/40000, loss: 0.3821, lr: 0.008513, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 07:17:28 2021-05-09 15:31:33 [INFO] [TRAIN] epoch: 18, iter: 6630/40000, loss: 0.7169, lr: 0.008510, batch_cost: 0.7866, reader_cost: 0.00017, ips: 1.2713 samples/sec | ETA 07:17:28 2021-05-09 15:31:41 [INFO] [TRAIN] epoch: 18, iter: 6640/40000, loss: 0.8310, lr: 0.008508, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 07:17:13 2021-05-09 15:31:49 [INFO] [TRAIN] epoch: 18, iter: 6650/40000, loss: 0.7601, lr: 0.008506, batch_cost: 0.7880, reader_cost: 0.00016, ips: 1.2691 samples/sec | ETA 07:17:58 2021-05-09 15:31:56 [INFO] [TRAIN] epoch: 18, iter: 6660/40000, loss: 0.3454, lr: 0.008504, batch_cost: 0.7849, reader_cost: 0.00014, ips: 1.2740 samples/sec | ETA 07:16:09 2021-05-09 15:32:04 [INFO] [TRAIN] epoch: 18, iter: 6670/40000, loss: 0.3714, lr: 0.008501, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2713 samples/sec | ETA 07:16:58 2021-05-09 15:32:12 [INFO] [TRAIN] epoch: 18, iter: 6680/40000, loss: 0.4525, lr: 0.008499, batch_cost: 0.7853, reader_cost: 0.00015, ips: 1.2733 samples/sec | ETA 07:16:07 2021-05-09 15:32:20 [INFO] [TRAIN] epoch: 18, iter: 6690/40000, loss: 0.4643, lr: 0.008497, batch_cost: 0.7852, reader_cost: 0.00013, ips: 1.2735 samples/sec | ETA 07:15:56 2021-05-09 15:32:31 [INFO] [TRAIN] epoch: 19, iter: 6700/40000, loss: 0.6050, lr: 0.008494, batch_cost: 1.0922, reader_cost: 0.25513, ips: 0.9156 samples/sec | ETA 10:06:11 2021-05-09 15:32:39 [INFO] [TRAIN] epoch: 19, iter: 6710/40000, loss: 0.5712, lr: 0.008492, batch_cost: 0.8014, reader_cost: 0.00036, ips: 1.2478 samples/sec | ETA 07:24:39 2021-05-09 15:32:47 [INFO] [TRAIN] epoch: 19, iter: 6720/40000, loss: 0.6568, lr: 0.008490, batch_cost: 0.7862, reader_cost: 0.00018, ips: 1.2719 samples/sec | ETA 07:16:05 2021-05-09 15:32:55 [INFO] [TRAIN] epoch: 19, iter: 6730/40000, loss: 0.3337, lr: 0.008488, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2715 samples/sec | ETA 07:16:05 2021-05-09 15:33:03 [INFO] [TRAIN] epoch: 19, iter: 6740/40000, loss: 0.4054, lr: 0.008485, batch_cost: 0.7870, reader_cost: 0.00015, ips: 1.2707 samples/sec | ETA 07:16:14 2021-05-09 15:33:10 [INFO] [TRAIN] epoch: 19, iter: 6750/40000, loss: 0.5964, lr: 0.008483, batch_cost: 0.7878, reader_cost: 0.00015, ips: 1.2694 samples/sec | ETA 07:16:33 2021-05-09 15:33:18 [INFO] [TRAIN] epoch: 19, iter: 6760/40000, loss: 0.3857, lr: 0.008481, batch_cost: 0.7869, reader_cost: 0.00015, ips: 1.2708 samples/sec | ETA 07:15:57 2021-05-09 15:33:26 [INFO] [TRAIN] epoch: 19, iter: 6770/40000, loss: 0.1742, lr: 0.008479, batch_cost: 0.7870, reader_cost: 0.00017, ips: 1.2706 samples/sec | ETA 07:15:53 2021-05-09 15:33:34 [INFO] [TRAIN] epoch: 19, iter: 6780/40000, loss: 0.4450, lr: 0.008476, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2704 samples/sec | ETA 07:15:49 2021-05-09 15:33:42 [INFO] [TRAIN] epoch: 19, iter: 6790/40000, loss: 0.4472, lr: 0.008474, batch_cost: 0.7870, reader_cost: 0.00018, ips: 1.2706 samples/sec | ETA 07:15:37 2021-05-09 15:33:50 [INFO] [TRAIN] epoch: 19, iter: 6800/40000, loss: 0.3733, lr: 0.008472, batch_cost: 0.7864, reader_cost: 0.00017, ips: 1.2716 samples/sec | ETA 07:15:08 2021-05-09 15:33:58 [INFO] [TRAIN] epoch: 19, iter: 6810/40000, loss: 0.3386, lr: 0.008469, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 07:14:51 2021-05-09 15:34:06 [INFO] [TRAIN] epoch: 19, iter: 6820/40000, loss: 0.7395, lr: 0.008467, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2725 samples/sec | ETA 07:14:35 2021-05-09 15:34:13 [INFO] [TRAIN] epoch: 19, iter: 6830/40000, loss: 0.3952, lr: 0.008465, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 07:14:42 2021-05-09 15:34:21 [INFO] [TRAIN] epoch: 19, iter: 6840/40000, loss: 0.5531, lr: 0.008463, batch_cost: 0.7870, reader_cost: 0.00015, ips: 1.2707 samples/sec | ETA 07:14:55 2021-05-09 15:34:29 [INFO] [TRAIN] epoch: 19, iter: 6850/40000, loss: 0.3975, lr: 0.008460, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2736 samples/sec | ETA 07:13:48 2021-05-09 15:34:37 [INFO] [TRAIN] epoch: 19, iter: 6860/40000, loss: 0.3364, lr: 0.008458, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 07:13:58 2021-05-09 15:34:45 [INFO] [TRAIN] epoch: 19, iter: 6870/40000, loss: 0.8281, lr: 0.008456, batch_cost: 0.7876, reader_cost: 0.00016, ips: 1.2696 samples/sec | ETA 07:14:54 2021-05-09 15:34:53 [INFO] [TRAIN] epoch: 19, iter: 6880/40000, loss: 0.2743, lr: 0.008454, batch_cost: 0.7858, reader_cost: 0.00015, ips: 1.2726 samples/sec | ETA 07:13:44 2021-05-09 15:35:01 [INFO] [TRAIN] epoch: 19, iter: 6890/40000, loss: 0.3716, lr: 0.008451, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2716 samples/sec | ETA 07:13:58 2021-05-09 15:35:08 [INFO] [TRAIN] epoch: 19, iter: 6900/40000, loss: 0.2631, lr: 0.008449, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 07:13:48 2021-05-09 15:35:16 [INFO] [TRAIN] epoch: 19, iter: 6910/40000, loss: 0.4221, lr: 0.008447, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2735 samples/sec | ETA 07:13:02 2021-05-09 15:35:24 [INFO] [TRAIN] epoch: 19, iter: 6920/40000, loss: 0.2400, lr: 0.008445, batch_cost: 0.7846, reader_cost: 0.00016, ips: 1.2745 samples/sec | ETA 07:12:34 2021-05-09 15:35:32 [INFO] [TRAIN] epoch: 19, iter: 6930/40000, loss: 0.3389, lr: 0.008442, batch_cost: 0.7836, reader_cost: 0.00016, ips: 1.2762 samples/sec | ETA 07:11:53 2021-05-09 15:35:40 [INFO] [TRAIN] epoch: 19, iter: 6940/40000, loss: 0.4647, lr: 0.008440, batch_cost: 0.7865, reader_cost: 0.00018, ips: 1.2714 samples/sec | ETA 07:13:22 2021-05-09 15:35:48 [INFO] [TRAIN] epoch: 19, iter: 6950/40000, loss: 0.4576, lr: 0.008438, batch_cost: 0.7876, reader_cost: 0.00017, ips: 1.2697 samples/sec | ETA 07:13:50 2021-05-09 15:35:56 [INFO] [TRAIN] epoch: 19, iter: 6960/40000, loss: 0.2914, lr: 0.008435, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2723 samples/sec | ETA 07:12:48 2021-05-09 15:36:03 [INFO] [TRAIN] epoch: 19, iter: 6970/40000, loss: 0.4133, lr: 0.008433, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 07:12:38 2021-05-09 15:36:11 [INFO] [TRAIN] epoch: 19, iter: 6980/40000, loss: 0.5640, lr: 0.008431, batch_cost: 0.7875, reader_cost: 0.00016, ips: 1.2698 samples/sec | ETA 07:13:24 2021-05-09 15:36:19 [INFO] [TRAIN] epoch: 19, iter: 6990/40000, loss: 0.3642, lr: 0.008429, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2735 samples/sec | ETA 07:11:59 2021-05-09 15:36:27 [INFO] [TRAIN] epoch: 19, iter: 7000/40000, loss: 0.7789, lr: 0.008426, batch_cost: 0.7853, reader_cost: 0.00017, ips: 1.2733 samples/sec | ETA 07:11:55 2021-05-09 15:36:27 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-09 15:39:57 [INFO] [EVAL] #Images: 500 mIoU: 0.7076 Acc: 0.9477 Kappa: 0.9322 2021-05-09 15:39:57 [INFO] [EVAL] Class IoU: [0.9752 0.8072 0.9053 0.553 0.5833 0.4176 0.5532 0.6713 0.9046 0.6249 0.9347 0.7392 0.5182 0.9167 0.6753 0.8207 0.6389 0.5198 0.6847] 2021-05-09 15:39:57 [INFO] [EVAL] Class Acc: [0.9927 0.8611 0.9463 0.934 0.7695 0.756 0.713 0.8667 0.9371 0.8442 0.9623 0.8392 0.6925 0.9371 0.7256 0.8761 0.9193 0.6911 0.8011] 2021-05-09 15:40:25 [INFO] [EVAL] The model with the best validation mIoU (0.7286) was saved at iter 6000. 2021-05-09 15:40:33 [INFO] [TRAIN] epoch: 19, iter: 7010/40000, loss: 0.6382, lr: 0.008424, batch_cost: 0.7823, reader_cost: 0.00024, ips: 1.2783 samples/sec | ETA 07:10:07 2021-05-09 15:40:41 [INFO] [TRAIN] epoch: 19, iter: 7020/40000, loss: 0.3467, lr: 0.008422, batch_cost: 0.7851, reader_cost: 0.00016, ips: 1.2737 samples/sec | ETA 07:11:32 2021-05-09 15:40:49 [INFO] [TRAIN] epoch: 19, iter: 7030/40000, loss: 0.1483, lr: 0.008420, batch_cost: 0.7843, reader_cost: 0.00016, ips: 1.2750 samples/sec | ETA 07:10:59 2021-05-09 15:40:57 [INFO] [TRAIN] epoch: 19, iter: 7040/40000, loss: 0.3458, lr: 0.008417, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2729 samples/sec | ETA 07:11:33 2021-05-09 15:41:05 [INFO] [TRAIN] epoch: 19, iter: 7050/40000, loss: 0.5601, lr: 0.008415, batch_cost: 0.7850, reader_cost: 0.00017, ips: 1.2738 samples/sec | ETA 07:11:06 2021-05-09 15:41:13 [INFO] [TRAIN] epoch: 19, iter: 7060/40000, loss: 0.5189, lr: 0.008413, batch_cost: 0.7882, reader_cost: 0.00014, ips: 1.2688 samples/sec | ETA 07:12:41 2021-05-09 15:41:24 [INFO] [TRAIN] epoch: 20, iter: 7070/40000, loss: 0.5255, lr: 0.008410, batch_cost: 1.0958, reader_cost: 0.23276, ips: 0.9125 samples/sec | ETA 10:01:26 2021-05-09 15:41:31 [INFO] [TRAIN] epoch: 20, iter: 7080/40000, loss: 0.4707, lr: 0.008408, batch_cost: 0.7988, reader_cost: 0.00035, ips: 1.2519 samples/sec | ETA 07:18:16 2021-05-09 15:41:39 [INFO] [TRAIN] epoch: 20, iter: 7090/40000, loss: 0.8237, lr: 0.008406, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2730 samples/sec | ETA 07:10:53 2021-05-09 15:41:47 [INFO] [TRAIN] epoch: 20, iter: 7100/40000, loss: 0.3369, lr: 0.008404, batch_cost: 0.7868, reader_cost: 0.00017, ips: 1.2709 samples/sec | ETA 07:11:27 2021-05-09 15:41:55 [INFO] [TRAIN] epoch: 20, iter: 7110/40000, loss: 0.5523, lr: 0.008401, batch_cost: 0.7875, reader_cost: 0.00018, ips: 1.2698 samples/sec | ETA 07:11:41 2021-05-09 15:42:03 [INFO] [TRAIN] epoch: 20, iter: 7120/40000, loss: 0.5984, lr: 0.008399, batch_cost: 0.7870, reader_cost: 0.00018, ips: 1.2707 samples/sec | ETA 07:11:16 2021-05-09 15:42:11 [INFO] [TRAIN] epoch: 20, iter: 7130/40000, loss: 0.3227, lr: 0.008397, batch_cost: 0.7873, reader_cost: 0.00017, ips: 1.2701 samples/sec | ETA 07:11:19 2021-05-09 15:42:19 [INFO] [TRAIN] epoch: 20, iter: 7140/40000, loss: 0.1570, lr: 0.008395, batch_cost: 0.7864, reader_cost: 0.00017, ips: 1.2717 samples/sec | ETA 07:10:39 2021-05-09 15:42:27 [INFO] [TRAIN] epoch: 20, iter: 7150/40000, loss: 0.3721, lr: 0.008392, batch_cost: 0.7862, reader_cost: 0.00017, ips: 1.2719 samples/sec | ETA 07:10:28 2021-05-09 15:42:34 [INFO] [TRAIN] epoch: 20, iter: 7160/40000, loss: 0.3647, lr: 0.008390, batch_cost: 0.7880, reader_cost: 0.00017, ips: 1.2690 samples/sec | ETA 07:11:17 2021-05-09 15:42:42 [INFO] [TRAIN] epoch: 20, iter: 7170/40000, loss: 0.3348, lr: 0.008388, batch_cost: 0.7860, reader_cost: 0.00017, ips: 1.2723 samples/sec | ETA 07:10:02 2021-05-09 15:42:50 [INFO] [TRAIN] epoch: 20, iter: 7180/40000, loss: 0.3717, lr: 0.008385, batch_cost: 0.7850, reader_cost: 0.00015, ips: 1.2739 samples/sec | ETA 07:09:23 2021-05-09 15:42:58 [INFO] [TRAIN] epoch: 20, iter: 7190/40000, loss: 0.2678, lr: 0.008383, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2732 samples/sec | ETA 07:09:30 2021-05-09 15:43:06 [INFO] [TRAIN] epoch: 20, iter: 7200/40000, loss: 0.3589, lr: 0.008381, batch_cost: 0.7875, reader_cost: 0.00017, ips: 1.2698 samples/sec | ETA 07:10:31 2021-05-09 15:43:14 [INFO] [TRAIN] epoch: 20, iter: 7210/40000, loss: 0.4643, lr: 0.008379, batch_cost: 0.7849, reader_cost: 0.00015, ips: 1.2740 samples/sec | ETA 07:08:56 2021-05-09 15:43:22 [INFO] [TRAIN] epoch: 20, iter: 7220/40000, loss: 0.2737, lr: 0.008376, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 07:09:16 2021-05-09 15:43:29 [INFO] [TRAIN] epoch: 20, iter: 7230/40000, loss: 0.3510, lr: 0.008374, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2721 samples/sec | ETA 07:09:21 2021-05-09 15:43:37 [INFO] [TRAIN] epoch: 20, iter: 7240/40000, loss: 0.3734, lr: 0.008372, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2726 samples/sec | ETA 07:09:03 2021-05-09 15:43:45 [INFO] [TRAIN] epoch: 20, iter: 7250/40000, loss: 0.4393, lr: 0.008370, batch_cost: 0.7867, reader_cost: 0.00018, ips: 1.2712 samples/sec | ETA 07:09:23 2021-05-09 15:43:53 [INFO] [TRAIN] epoch: 20, iter: 7260/40000, loss: 0.1867, lr: 0.008367, batch_cost: 0.7847, reader_cost: 0.00018, ips: 1.2744 samples/sec | ETA 07:08:11 2021-05-09 15:44:01 [INFO] [TRAIN] epoch: 20, iter: 7270/40000, loss: 0.1842, lr: 0.008365, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2736 samples/sec | ETA 07:08:19 2021-05-09 15:44:09 [INFO] [TRAIN] epoch: 20, iter: 7280/40000, loss: 0.4736, lr: 0.008363, batch_cost: 0.7852, reader_cost: 0.00017, ips: 1.2736 samples/sec | ETA 07:08:11 2021-05-09 15:44:17 [INFO] [TRAIN] epoch: 20, iter: 7290/40000, loss: 0.2676, lr: 0.008360, batch_cost: 0.7877, reader_cost: 0.00018, ips: 1.2696 samples/sec | ETA 07:09:24 2021-05-09 15:44:24 [INFO] [TRAIN] epoch: 20, iter: 7300/40000, loss: 0.3032, lr: 0.008358, batch_cost: 0.7845, reader_cost: 0.00017, ips: 1.2747 samples/sec | ETA 07:07:32 2021-05-09 15:44:32 [INFO] [TRAIN] epoch: 20, iter: 7310/40000, loss: 0.3289, lr: 0.008356, batch_cost: 0.7860, reader_cost: 0.00017, ips: 1.2723 samples/sec | ETA 07:08:14 2021-05-09 15:44:40 [INFO] [TRAIN] epoch: 20, iter: 7320/40000, loss: 0.3056, lr: 0.008354, batch_cost: 0.7853, reader_cost: 0.00017, ips: 1.2734 samples/sec | ETA 07:07:42 2021-05-09 15:44:48 [INFO] [TRAIN] epoch: 20, iter: 7330/40000, loss: 0.3568, lr: 0.008351, batch_cost: 0.7854, reader_cost: 0.00015, ips: 1.2732 samples/sec | ETA 07:07:39 2021-05-09 15:44:56 [INFO] [TRAIN] epoch: 20, iter: 7340/40000, loss: 0.3454, lr: 0.008349, batch_cost: 0.7860, reader_cost: 0.00017, ips: 1.2723 samples/sec | ETA 07:07:50 2021-05-09 15:45:04 [INFO] [TRAIN] epoch: 20, iter: 7350/40000, loss: 0.7595, lr: 0.008347, batch_cost: 0.7839, reader_cost: 0.00016, ips: 1.2757 samples/sec | ETA 07:06:33 2021-05-09 15:45:12 [INFO] [TRAIN] epoch: 20, iter: 7360/40000, loss: 0.4302, lr: 0.008345, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 07:07:31 2021-05-09 15:45:19 [INFO] [TRAIN] epoch: 20, iter: 7370/40000, loss: 0.7296, lr: 0.008342, batch_cost: 0.7848, reader_cost: 0.00016, ips: 1.2742 samples/sec | ETA 07:06:48 2021-05-09 15:45:27 [INFO] [TRAIN] epoch: 20, iter: 7380/40000, loss: 0.6805, lr: 0.008340, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2732 samples/sec | ETA 07:07:00 2021-05-09 15:45:35 [INFO] [TRAIN] epoch: 20, iter: 7390/40000, loss: 0.4021, lr: 0.008338, batch_cost: 0.7856, reader_cost: 0.00017, ips: 1.2729 samples/sec | ETA 07:06:58 2021-05-09 15:45:43 [INFO] [TRAIN] epoch: 20, iter: 7400/40000, loss: 0.2030, lr: 0.008335, batch_cost: 0.7854, reader_cost: 0.00017, ips: 1.2733 samples/sec | ETA 07:06:42 2021-05-09 15:45:51 [INFO] [TRAIN] epoch: 20, iter: 7410/40000, loss: 0.2969, lr: 0.008333, batch_cost: 0.7850, reader_cost: 0.00015, ips: 1.2739 samples/sec | ETA 07:06:22 2021-05-09 15:45:59 [INFO] [TRAIN] epoch: 20, iter: 7420/40000, loss: 0.4087, lr: 0.008331, batch_cost: 0.7852, reader_cost: 0.00017, ips: 1.2735 samples/sec | ETA 07:06:22 2021-05-09 15:46:07 [INFO] [TRAIN] epoch: 20, iter: 7430/40000, loss: 0.7415, lr: 0.008329, batch_cost: 0.7845, reader_cost: 0.00016, ips: 1.2746 samples/sec | ETA 07:05:52 2021-05-09 15:46:14 [INFO] [TRAIN] epoch: 20, iter: 7440/40000, loss: 0.5530, lr: 0.008326, batch_cost: 0.7852, reader_cost: 0.00013, ips: 1.2736 samples/sec | ETA 07:06:05 2021-05-09 15:46:25 [INFO] [TRAIN] epoch: 21, iter: 7450/40000, loss: 0.2183, lr: 0.008324, batch_cost: 1.1040, reader_cost: 0.25646, ips: 0.9058 samples/sec | ETA 09:58:56 2021-05-09 15:46:33 [INFO] [TRAIN] epoch: 21, iter: 7460/40000, loss: 0.4875, lr: 0.008322, batch_cost: 0.7888, reader_cost: 0.00032, ips: 1.2678 samples/sec | ETA 07:07:47 2021-05-09 15:46:41 [INFO] [TRAIN] epoch: 21, iter: 7470/40000, loss: 0.3824, lr: 0.008320, batch_cost: 0.7859, reader_cost: 0.00017, ips: 1.2724 samples/sec | ETA 07:06:06 2021-05-09 15:46:49 [INFO] [TRAIN] epoch: 21, iter: 7480/40000, loss: 0.5304, lr: 0.008317, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2733 samples/sec | ETA 07:05:39 2021-05-09 15:46:57 [INFO] [TRAIN] epoch: 21, iter: 7490/40000, loss: 0.5493, lr: 0.008315, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2718 samples/sec | ETA 07:06:02 2021-05-09 15:47:05 [INFO] [TRAIN] epoch: 21, iter: 7500/40000, loss: 0.5328, lr: 0.008313, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2729 samples/sec | ETA 07:05:31 2021-05-09 15:47:13 [INFO] [TRAIN] epoch: 21, iter: 7510/40000, loss: 0.2709, lr: 0.008310, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 07:05:38 2021-05-09 15:47:20 [INFO] [TRAIN] epoch: 21, iter: 7520/40000, loss: 0.4024, lr: 0.008308, batch_cost: 0.7843, reader_cost: 0.00016, ips: 1.2750 samples/sec | ETA 07:04:35 2021-05-09 15:47:28 [INFO] [TRAIN] epoch: 21, iter: 7530/40000, loss: 0.4251, lr: 0.008306, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2729 samples/sec | ETA 07:05:09 2021-05-09 15:47:36 [INFO] [TRAIN] epoch: 21, iter: 7540/40000, loss: 0.4658, lr: 0.008304, batch_cost: 0.7860, reader_cost: 0.00014, ips: 1.2723 samples/sec | ETA 07:05:12 2021-05-09 15:47:44 [INFO] [TRAIN] epoch: 21, iter: 7550/40000, loss: 0.4202, lr: 0.008301, batch_cost: 0.7853, reader_cost: 0.00014, ips: 1.2735 samples/sec | ETA 07:04:41 2021-05-09 15:47:52 [INFO] [TRAIN] epoch: 21, iter: 7560/40000, loss: 0.1761, lr: 0.008299, batch_cost: 0.7864, reader_cost: 0.00014, ips: 1.2717 samples/sec | ETA 07:05:10 2021-05-09 15:48:00 [INFO] [TRAIN] epoch: 21, iter: 7570/40000, loss: 0.3120, lr: 0.008297, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2713 samples/sec | ETA 07:05:09 2021-05-09 15:48:08 [INFO] [TRAIN] epoch: 21, iter: 7580/40000, loss: 0.4360, lr: 0.008295, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2728 samples/sec | ETA 07:04:32 2021-05-09 15:48:15 [INFO] [TRAIN] epoch: 21, iter: 7590/40000, loss: 0.3451, lr: 0.008292, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2716 samples/sec | ETA 07:04:47 2021-05-09 15:48:23 [INFO] [TRAIN] epoch: 21, iter: 7600/40000, loss: 0.3189, lr: 0.008290, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2730 samples/sec | ETA 07:04:11 2021-05-09 15:48:31 [INFO] [TRAIN] epoch: 21, iter: 7610/40000, loss: 0.1882, lr: 0.008288, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2735 samples/sec | ETA 07:03:53 2021-05-09 15:48:39 [INFO] [TRAIN] epoch: 21, iter: 7620/40000, loss: 0.2743, lr: 0.008285, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2724 samples/sec | ETA 07:04:08 2021-05-09 15:48:47 [INFO] [TRAIN] epoch: 21, iter: 7630/40000, loss: 0.1265, lr: 0.008283, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2709 samples/sec | ETA 07:04:30 2021-05-09 15:48:55 [INFO] [TRAIN] epoch: 21, iter: 7640/40000, loss: 0.2812, lr: 0.008281, batch_cost: 0.7875, reader_cost: 0.00016, ips: 1.2699 samples/sec | ETA 07:04:42 2021-05-09 15:49:03 [INFO] [TRAIN] epoch: 21, iter: 7650/40000, loss: 0.3427, lr: 0.008279, batch_cost: 0.7853, reader_cost: 0.00015, ips: 1.2735 samples/sec | ETA 07:03:23 2021-05-09 15:49:11 [INFO] [TRAIN] epoch: 21, iter: 7660/40000, loss: 0.3018, lr: 0.008276, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2731 samples/sec | ETA 07:03:21 2021-05-09 15:49:18 [INFO] [TRAIN] epoch: 21, iter: 7670/40000, loss: 0.4958, lr: 0.008274, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2730 samples/sec | ETA 07:03:16 2021-05-09 15:49:26 [INFO] [TRAIN] epoch: 21, iter: 7680/40000, loss: 0.3542, lr: 0.008272, batch_cost: 0.7859, reader_cost: 0.00014, ips: 1.2725 samples/sec | ETA 07:03:19 2021-05-09 15:49:34 [INFO] [TRAIN] epoch: 21, iter: 7690/40000, loss: 0.3146, lr: 0.008270, batch_cost: 0.7855, reader_cost: 0.00014, ips: 1.2730 samples/sec | ETA 07:03:00 2021-05-09 15:49:42 [INFO] [TRAIN] epoch: 21, iter: 7700/40000, loss: 0.2505, lr: 0.008267, batch_cost: 0.7851, reader_cost: 0.00015, ips: 1.2737 samples/sec | ETA 07:02:39 2021-05-09 15:49:50 [INFO] [TRAIN] epoch: 21, iter: 7710/40000, loss: 0.3142, lr: 0.008265, batch_cost: 0.7851, reader_cost: 0.00015, ips: 1.2737 samples/sec | ETA 07:02:31 2021-05-09 15:49:58 [INFO] [TRAIN] epoch: 21, iter: 7720/40000, loss: 0.4877, lr: 0.008263, batch_cost: 0.7844, reader_cost: 0.00015, ips: 1.2749 samples/sec | ETA 07:02:00 2021-05-09 15:50:05 [INFO] [TRAIN] epoch: 21, iter: 7730/40000, loss: 0.4262, lr: 0.008260, batch_cost: 0.7860, reader_cost: 0.00017, ips: 1.2723 samples/sec | ETA 07:02:43 2021-05-09 15:50:13 [INFO] [TRAIN] epoch: 21, iter: 7740/40000, loss: 0.5193, lr: 0.008258, batch_cost: 0.7848, reader_cost: 0.00016, ips: 1.2741 samples/sec | ETA 07:01:59 2021-05-09 15:50:21 [INFO] [TRAIN] epoch: 21, iter: 7750/40000, loss: 0.6321, lr: 0.008256, batch_cost: 0.7850, reader_cost: 0.00016, ips: 1.2739 samples/sec | ETA 07:01:56 2021-05-09 15:50:29 [INFO] [TRAIN] epoch: 21, iter: 7760/40000, loss: 0.3950, lr: 0.008254, batch_cost: 0.7858, reader_cost: 0.00015, ips: 1.2725 samples/sec | ETA 07:02:15 2021-05-09 15:50:37 [INFO] [TRAIN] epoch: 21, iter: 7770/40000, loss: 0.1317, lr: 0.008251, batch_cost: 0.7837, reader_cost: 0.00015, ips: 1.2760 samples/sec | ETA 07:00:59 2021-05-09 15:50:45 [INFO] [TRAIN] epoch: 21, iter: 7780/40000, loss: 0.2916, lr: 0.008249, batch_cost: 0.7849, reader_cost: 0.00016, ips: 1.2740 samples/sec | ETA 07:01:29 2021-05-09 15:50:53 [INFO] [TRAIN] epoch: 21, iter: 7790/40000, loss: 0.3438, lr: 0.008247, batch_cost: 0.7866, reader_cost: 0.00017, ips: 1.2713 samples/sec | ETA 07:02:15 2021-05-09 15:51:00 [INFO] [TRAIN] epoch: 21, iter: 7800/40000, loss: 0.3667, lr: 0.008244, batch_cost: 0.7851, reader_cost: 0.00015, ips: 1.2737 samples/sec | ETA 07:01:20 2021-05-09 15:51:08 [INFO] [TRAIN] epoch: 21, iter: 7810/40000, loss: 0.4588, lr: 0.008242, batch_cost: 0.7843, reader_cost: 0.00025, ips: 1.2750 samples/sec | ETA 07:00:47 2021-05-09 15:51:19 [INFO] [TRAIN] epoch: 22, iter: 7820/40000, loss: 0.3596, lr: 0.008240, batch_cost: 1.1007, reader_cost: 0.24635, ips: 0.9085 samples/sec | ETA 09:50:21 2021-05-09 15:51:27 [INFO] [TRAIN] epoch: 22, iter: 7830/40000, loss: 0.6859, lr: 0.008238, batch_cost: 0.7932, reader_cost: 0.00033, ips: 1.2608 samples/sec | ETA 07:05:15 2021-05-09 15:51:35 [INFO] [TRAIN] epoch: 22, iter: 7840/40000, loss: 0.3732, lr: 0.008235, batch_cost: 0.7847, reader_cost: 0.00017, ips: 1.2743 samples/sec | ETA 07:00:37 2021-05-09 15:51:43 [INFO] [TRAIN] epoch: 22, iter: 7850/40000, loss: 0.4542, lr: 0.008233, batch_cost: 0.7846, reader_cost: 0.00016, ips: 1.2745 samples/sec | ETA 07:00:25 2021-05-09 15:51:51 [INFO] [TRAIN] epoch: 22, iter: 7860/40000, loss: 0.6288, lr: 0.008231, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2720 samples/sec | ETA 07:01:06 2021-05-09 15:51:59 [INFO] [TRAIN] epoch: 22, iter: 7870/40000, loss: 0.4485, lr: 0.008229, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2727 samples/sec | ETA 07:00:46 2021-05-09 15:52:07 [INFO] [TRAIN] epoch: 22, iter: 7880/40000, loss: 0.2376, lr: 0.008226, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2724 samples/sec | ETA 07:00:43 2021-05-09 15:52:14 [INFO] [TRAIN] epoch: 22, iter: 7890/40000, loss: 0.1787, lr: 0.008224, batch_cost: 0.7872, reader_cost: 0.00015, ips: 1.2703 samples/sec | ETA 07:01:17 2021-05-09 15:52:22 [INFO] [TRAIN] epoch: 22, iter: 7900/40000, loss: 0.3788, lr: 0.008222, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2719 samples/sec | ETA 07:00:37 2021-05-09 15:52:30 [INFO] [TRAIN] epoch: 22, iter: 7910/40000, loss: 0.2400, lr: 0.008219, batch_cost: 0.7878, reader_cost: 0.00015, ips: 1.2694 samples/sec | ETA 07:01:18 2021-05-09 15:52:38 [INFO] [TRAIN] epoch: 22, iter: 7920/40000, loss: 0.3428, lr: 0.008217, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2724 samples/sec | ETA 07:00:12 2021-05-09 15:52:46 [INFO] [TRAIN] epoch: 22, iter: 7930/40000, loss: 0.1745, lr: 0.008215, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2709 samples/sec | ETA 07:00:33 2021-05-09 15:52:54 [INFO] [TRAIN] epoch: 22, iter: 7940/40000, loss: 0.2557, lr: 0.008213, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2728 samples/sec | ETA 06:59:47 2021-05-09 15:53:02 [INFO] [TRAIN] epoch: 22, iter: 7950/40000, loss: 0.4052, lr: 0.008210, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2733 samples/sec | ETA 06:59:31 2021-05-09 15:53:09 [INFO] [TRAIN] epoch: 22, iter: 7960/40000, loss: 0.2837, lr: 0.008208, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2707 samples/sec | ETA 07:00:14 2021-05-09 15:53:17 [INFO] [TRAIN] epoch: 22, iter: 7970/40000, loss: 0.3055, lr: 0.008206, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2733 samples/sec | ETA 06:59:15 2021-05-09 15:53:25 [INFO] [TRAIN] epoch: 22, iter: 7980/40000, loss: 0.2951, lr: 0.008203, batch_cost: 0.7846, reader_cost: 0.00017, ips: 1.2746 samples/sec | ETA 06:58:41 2021-05-09 15:53:33 [INFO] [TRAIN] epoch: 22, iter: 7990/40000, loss: 0.3577, lr: 0.008201, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2717 samples/sec | ETA 06:59:30 2021-05-09 15:53:41 [INFO] [TRAIN] epoch: 22, iter: 8000/40000, loss: 0.5401, lr: 0.008199, batch_cost: 0.7871, reader_cost: 0.00018, ips: 1.2705 samples/sec | ETA 06:59:46 2021-05-09 15:53:41 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-09 15:57:11 [INFO] [EVAL] #Images: 500 mIoU: 0.7299 Acc: 0.9507 Kappa: 0.9359 2021-05-09 15:57:11 [INFO] [EVAL] Class IoU: [0.9772 0.8167 0.9079 0.5463 0.5792 0.4447 0.5643 0.6817 0.9085 0.6481 0.9345 0.7488 0.5065 0.929 0.8207 0.8507 0.7347 0.5683 0.7007] 2021-05-09 15:57:11 [INFO] [EVAL] Class Acc: [0.9867 0.9236 0.9352 0.8548 0.7683 0.7459 0.8003 0.8877 0.9511 0.8063 0.9646 0.831 0.7882 0.9517 0.9089 0.9251 0.8561 0.8249 0.7899] 2021-05-09 15:58:01 [INFO] [EVAL] The model with the best validation mIoU (0.7299) was saved at iter 8000. 2021-05-09 15:58:09 [INFO] [TRAIN] epoch: 22, iter: 8010/40000, loss: 0.3320, lr: 0.008197, batch_cost: 0.7835, reader_cost: 0.00043, ips: 1.2763 samples/sec | ETA 06:57:45 2021-05-09 15:58:17 [INFO] [TRAIN] epoch: 22, iter: 8020/40000, loss: 0.3669, lr: 0.008194, batch_cost: 0.7843, reader_cost: 0.00037, ips: 1.2750 samples/sec | ETA 06:58:03 2021-05-09 15:58:25 [INFO] [TRAIN] epoch: 22, iter: 8030/40000, loss: 0.4320, lr: 0.008192, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2735 samples/sec | ETA 06:58:23 2021-05-09 15:58:32 [INFO] [TRAIN] epoch: 22, iter: 8040/40000, loss: 0.2117, lr: 0.008190, batch_cost: 0.7834, reader_cost: 0.00016, ips: 1.2766 samples/sec | ETA 06:57:15 2021-05-09 15:58:40 [INFO] [TRAIN] epoch: 22, iter: 8050/40000, loss: 0.3378, lr: 0.008188, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 06:58:15 2021-05-09 15:58:48 [INFO] [TRAIN] epoch: 22, iter: 8060/40000, loss: 0.2699, lr: 0.008185, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 06:58:22 2021-05-09 15:58:56 [INFO] [TRAIN] epoch: 22, iter: 8070/40000, loss: 0.3160, lr: 0.008183, batch_cost: 0.7867, reader_cost: 0.00017, ips: 1.2712 samples/sec | ETA 06:58:37 2021-05-09 15:59:04 [INFO] [TRAIN] epoch: 22, iter: 8080/40000, loss: 0.2229, lr: 0.008181, batch_cost: 0.7871, reader_cost: 0.00017, ips: 1.2704 samples/sec | ETA 06:58:45 2021-05-09 15:59:12 [INFO] [TRAIN] epoch: 22, iter: 8090/40000, loss: 0.4510, lr: 0.008178, batch_cost: 0.7858, reader_cost: 0.00017, ips: 1.2727 samples/sec | ETA 06:57:53 2021-05-09 15:59:20 [INFO] [TRAIN] epoch: 22, iter: 8100/40000, loss: 0.5083, lr: 0.008176, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2709 samples/sec | ETA 06:58:20 2021-05-09 15:59:27 [INFO] [TRAIN] epoch: 22, iter: 8110/40000, loss: 0.4217, lr: 0.008174, batch_cost: 0.7874, reader_cost: 0.00016, ips: 1.2700 samples/sec | ETA 06:58:30 2021-05-09 15:59:35 [INFO] [TRAIN] epoch: 22, iter: 8120/40000, loss: 0.6206, lr: 0.008172, batch_cost: 0.7863, reader_cost: 0.00017, ips: 1.2718 samples/sec | ETA 06:57:46 2021-05-09 15:59:43 [INFO] [TRAIN] epoch: 22, iter: 8130/40000, loss: 0.5197, lr: 0.008169, batch_cost: 0.7879, reader_cost: 0.00017, ips: 1.2692 samples/sec | ETA 06:58:29 2021-05-09 15:59:51 [INFO] [TRAIN] epoch: 22, iter: 8140/40000, loss: 0.4982, lr: 0.008167, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2708 samples/sec | ETA 06:57:51 2021-05-09 15:59:59 [INFO] [TRAIN] epoch: 22, iter: 8150/40000, loss: 0.2162, lr: 0.008165, batch_cost: 0.7887, reader_cost: 0.00016, ips: 1.2680 samples/sec | ETA 06:58:38 2021-05-09 16:00:07 [INFO] [TRAIN] epoch: 22, iter: 8160/40000, loss: 0.3776, lr: 0.008162, batch_cost: 0.7866, reader_cost: 0.00017, ips: 1.2713 samples/sec | ETA 06:57:26 2021-05-09 16:00:15 [INFO] [TRAIN] epoch: 22, iter: 8170/40000, loss: 0.4852, lr: 0.008160, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2708 samples/sec | ETA 06:57:26 2021-05-09 16:00:23 [INFO] [TRAIN] epoch: 22, iter: 8180/40000, loss: 0.6985, lr: 0.008158, batch_cost: 0.7852, reader_cost: 0.00011, ips: 1.2735 samples/sec | ETA 06:56:26 2021-05-09 16:00:33 [INFO] [TRAIN] epoch: 23, iter: 8190/40000, loss: 0.3525, lr: 0.008156, batch_cost: 1.0780, reader_cost: 0.22410, ips: 0.9277 samples/sec | ETA 09:31:30 2021-05-09 16:00:41 [INFO] [TRAIN] epoch: 23, iter: 8200/40000, loss: 0.4216, lr: 0.008153, batch_cost: 0.7983, reader_cost: 0.00035, ips: 1.2526 samples/sec | ETA 07:03:06 2021-05-09 16:00:49 [INFO] [TRAIN] epoch: 23, iter: 8210/40000, loss: 0.6088, lr: 0.008151, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2723 samples/sec | ETA 06:56:26 2021-05-09 16:00:57 [INFO] [TRAIN] epoch: 23, iter: 8220/40000, loss: 0.4256, lr: 0.008149, batch_cost: 0.7847, reader_cost: 0.00016, ips: 1.2744 samples/sec | ETA 06:55:37 2021-05-09 16:01:05 [INFO] [TRAIN] epoch: 23, iter: 8230/40000, loss: 0.5892, lr: 0.008147, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2729 samples/sec | ETA 06:55:59 2021-05-09 16:01:13 [INFO] [TRAIN] epoch: 23, iter: 8240/40000, loss: 0.8024, lr: 0.008144, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2720 samples/sec | ETA 06:56:08 2021-05-09 16:01:21 [INFO] [TRAIN] epoch: 23, iter: 8250/40000, loss: 0.4184, lr: 0.008142, batch_cost: 0.7868, reader_cost: 0.00017, ips: 1.2709 samples/sec | ETA 06:56:22 2021-05-09 16:01:29 [INFO] [TRAIN] epoch: 23, iter: 8260/40000, loss: 0.2977, lr: 0.008140, batch_cost: 0.7881, reader_cost: 0.00016, ips: 1.2688 samples/sec | ETA 06:56:54 2021-05-09 16:01:36 [INFO] [TRAIN] epoch: 23, iter: 8270/40000, loss: 0.3828, lr: 0.008137, batch_cost: 0.7860, reader_cost: 0.00018, ips: 1.2723 samples/sec | ETA 06:55:39 2021-05-09 16:01:44 [INFO] [TRAIN] epoch: 23, iter: 8280/40000, loss: 0.2803, lr: 0.008135, batch_cost: 0.7877, reader_cost: 0.00017, ips: 1.2696 samples/sec | ETA 06:56:24 2021-05-09 16:01:52 [INFO] [TRAIN] epoch: 23, iter: 8290/40000, loss: 0.4151, lr: 0.008133, batch_cost: 0.7874, reader_cost: 0.00016, ips: 1.2701 samples/sec | ETA 06:56:06 2021-05-09 16:02:00 [INFO] [TRAIN] epoch: 23, iter: 8300/40000, loss: 0.2983, lr: 0.008131, batch_cost: 0.7886, reader_cost: 0.00016, ips: 1.2680 samples/sec | ETA 06:56:39 2021-05-09 16:02:08 [INFO] [TRAIN] epoch: 23, iter: 8310/40000, loss: 0.2021, lr: 0.008128, batch_cost: 0.7876, reader_cost: 0.00017, ips: 1.2696 samples/sec | ETA 06:56:00 2021-05-09 16:02:16 [INFO] [TRAIN] epoch: 23, iter: 8320/40000, loss: 0.3315, lr: 0.008126, batch_cost: 0.7870, reader_cost: 0.00015, ips: 1.2707 samples/sec | ETA 06:55:30 2021-05-09 16:02:24 [INFO] [TRAIN] epoch: 23, iter: 8330/40000, loss: 0.4084, lr: 0.008124, batch_cost: 0.7879, reader_cost: 0.00016, ips: 1.2692 samples/sec | ETA 06:55:52 2021-05-09 16:02:32 [INFO] [TRAIN] epoch: 23, iter: 8340/40000, loss: 0.2564, lr: 0.008121, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 06:54:46 2021-05-09 16:02:39 [INFO] [TRAIN] epoch: 23, iter: 8350/40000, loss: 0.4019, lr: 0.008119, batch_cost: 0.7872, reader_cost: 0.00018, ips: 1.2704 samples/sec | ETA 06:55:13 2021-05-09 16:02:47 [INFO] [TRAIN] epoch: 23, iter: 8360/40000, loss: 0.3397, lr: 0.008117, batch_cost: 0.7843, reader_cost: 0.00015, ips: 1.2750 samples/sec | ETA 06:53:36 2021-05-09 16:02:55 [INFO] [TRAIN] epoch: 23, iter: 8370/40000, loss: 0.4087, lr: 0.008115, batch_cost: 0.7856, reader_cost: 0.00014, ips: 1.2729 samples/sec | ETA 06:54:09 2021-05-09 16:03:03 [INFO] [TRAIN] epoch: 23, iter: 8380/40000, loss: 0.1248, lr: 0.008112, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2703 samples/sec | ETA 06:54:51 2021-05-09 16:03:11 [INFO] [TRAIN] epoch: 23, iter: 8390/40000, loss: 0.2664, lr: 0.008110, batch_cost: 0.7870, reader_cost: 0.00015, ips: 1.2707 samples/sec | ETA 06:54:35 2021-05-09 16:03:19 [INFO] [TRAIN] epoch: 23, iter: 8400/40000, loss: 0.2973, lr: 0.008108, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2735 samples/sec | ETA 06:53:33 2021-05-09 16:03:27 [INFO] [TRAIN] epoch: 23, iter: 8410/40000, loss: 0.3084, lr: 0.008105, batch_cost: 0.7865, reader_cost: 0.00017, ips: 1.2715 samples/sec | ETA 06:54:05 2021-05-09 16:03:34 [INFO] [TRAIN] epoch: 23, iter: 8420/40000, loss: 0.3702, lr: 0.008103, batch_cost: 0.7872, reader_cost: 0.00021, ips: 1.2704 samples/sec | ETA 06:54:19 2021-05-09 16:03:42 [INFO] [TRAIN] epoch: 23, iter: 8430/40000, loss: 0.4581, lr: 0.008101, batch_cost: 0.7879, reader_cost: 0.00019, ips: 1.2692 samples/sec | ETA 06:54:33 2021-05-09 16:03:50 [INFO] [TRAIN] epoch: 23, iter: 8440/40000, loss: 0.3512, lr: 0.008099, batch_cost: 0.7884, reader_cost: 0.00018, ips: 1.2683 samples/sec | ETA 06:54:43 2021-05-09 16:03:58 [INFO] [TRAIN] epoch: 23, iter: 8450/40000, loss: 0.2330, lr: 0.008096, batch_cost: 0.7867, reader_cost: 0.00018, ips: 1.2711 samples/sec | ETA 06:53:40 2021-05-09 16:04:06 [INFO] [TRAIN] epoch: 23, iter: 8460/40000, loss: 0.4011, lr: 0.008094, batch_cost: 0.7866, reader_cost: 0.00017, ips: 1.2714 samples/sec | ETA 06:53:27 2021-05-09 16:04:14 [INFO] [TRAIN] epoch: 23, iter: 8470/40000, loss: 0.6035, lr: 0.008092, batch_cost: 0.7868, reader_cost: 0.00017, ips: 1.2709 samples/sec | ETA 06:53:29 2021-05-09 16:04:22 [INFO] [TRAIN] epoch: 23, iter: 8480/40000, loss: 0.4675, lr: 0.008090, batch_cost: 0.7870, reader_cost: 0.00017, ips: 1.2707 samples/sec | ETA 06:53:25 2021-05-09 16:04:30 [INFO] [TRAIN] epoch: 23, iter: 8490/40000, loss: 0.6756, lr: 0.008087, batch_cost: 0.7861, reader_cost: 0.00018, ips: 1.2721 samples/sec | ETA 06:52:49 2021-05-09 16:04:37 [INFO] [TRAIN] epoch: 23, iter: 8500/40000, loss: 0.4514, lr: 0.008085, batch_cost: 0.7868, reader_cost: 0.00017, ips: 1.2710 samples/sec | ETA 06:53:04 2021-05-09 16:04:45 [INFO] [TRAIN] epoch: 23, iter: 8510/40000, loss: 0.2419, lr: 0.008083, batch_cost: 0.7876, reader_cost: 0.00017, ips: 1.2696 samples/sec | ETA 06:53:22 2021-05-09 16:04:53 [INFO] [TRAIN] epoch: 23, iter: 8520/40000, loss: 0.1566, lr: 0.008080, batch_cost: 0.7878, reader_cost: 0.00017, ips: 1.2694 samples/sec | ETA 06:53:19 2021-05-09 16:05:01 [INFO] [TRAIN] epoch: 23, iter: 8530/40000, loss: 0.4056, lr: 0.008078, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2719 samples/sec | ETA 06:52:22 2021-05-09 16:05:09 [INFO] [TRAIN] epoch: 23, iter: 8540/40000, loss: 0.6240, lr: 0.008076, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2706 samples/sec | ETA 06:52:40 2021-05-09 16:05:17 [INFO] [TRAIN] epoch: 23, iter: 8550/40000, loss: 0.4741, lr: 0.008074, batch_cost: 0.7863, reader_cost: 0.00014, ips: 1.2718 samples/sec | ETA 06:52:09 2021-05-09 16:05:28 [INFO] [TRAIN] epoch: 24, iter: 8560/40000, loss: 0.2704, lr: 0.008071, batch_cost: 1.0971, reader_cost: 0.23486, ips: 0.9115 samples/sec | ETA 09:34:52 2021-05-09 16:05:36 [INFO] [TRAIN] epoch: 24, iter: 8570/40000, loss: 0.4687, lr: 0.008069, batch_cost: 0.7933, reader_cost: 0.00033, ips: 1.2606 samples/sec | ETA 06:55:32 2021-05-09 16:05:44 [INFO] [TRAIN] epoch: 24, iter: 8580/40000, loss: 0.6431, lr: 0.008067, batch_cost: 0.7876, reader_cost: 0.00015, ips: 1.2697 samples/sec | ETA 06:52:25 2021-05-09 16:05:51 [INFO] [TRAIN] epoch: 24, iter: 8590/40000, loss: 0.3306, lr: 0.008064, batch_cost: 0.7878, reader_cost: 0.00015, ips: 1.2693 samples/sec | ETA 06:52:25 2021-05-09 16:05:59 [INFO] [TRAIN] epoch: 24, iter: 8600/40000, loss: 0.5258, lr: 0.008062, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 06:51:34 2021-05-09 16:06:07 [INFO] [TRAIN] epoch: 24, iter: 8610/40000, loss: 0.5014, lr: 0.008060, batch_cost: 0.7873, reader_cost: 0.00015, ips: 1.2701 samples/sec | ETA 06:51:54 2021-05-09 16:06:15 [INFO] [TRAIN] epoch: 24, iter: 8620/40000, loss: 0.4663, lr: 0.008058, batch_cost: 0.7890, reader_cost: 0.00015, ips: 1.2674 samples/sec | ETA 06:52:39 2021-05-09 16:06:23 [INFO] [TRAIN] epoch: 24, iter: 8630/40000, loss: 0.1455, lr: 0.008055, batch_cost: 0.7850, reader_cost: 0.00014, ips: 1.2739 samples/sec | ETA 06:50:24 2021-05-09 16:06:31 [INFO] [TRAIN] epoch: 24, iter: 8640/40000, loss: 0.4134, lr: 0.008053, batch_cost: 0.7871, reader_cost: 0.00014, ips: 1.2705 samples/sec | ETA 06:51:22 2021-05-09 16:06:39 [INFO] [TRAIN] epoch: 24, iter: 8650/40000, loss: 0.4347, lr: 0.008051, batch_cost: 0.7883, reader_cost: 0.00015, ips: 1.2686 samples/sec | ETA 06:51:52 2021-05-09 16:06:47 [INFO] [TRAIN] epoch: 24, iter: 8660/40000, loss: 0.3202, lr: 0.008048, batch_cost: 0.7875, reader_cost: 0.00017, ips: 1.2699 samples/sec | ETA 06:51:19 2021-05-09 16:06:54 [INFO] [TRAIN] epoch: 24, iter: 8670/40000, loss: 0.3357, lr: 0.008046, batch_cost: 0.7877, reader_cost: 0.00015, ips: 1.2695 samples/sec | ETA 06:51:19 2021-05-09 16:07:02 [INFO] [TRAIN] epoch: 24, iter: 8680/40000, loss: 0.1689, lr: 0.008044, batch_cost: 0.7881, reader_cost: 0.00014, ips: 1.2689 samples/sec | ETA 06:51:23 2021-05-09 16:07:10 [INFO] [TRAIN] epoch: 24, iter: 8690/40000, loss: 0.3243, lr: 0.008042, batch_cost: 0.7878, reader_cost: 0.00015, ips: 1.2693 samples/sec | ETA 06:51:07 2021-05-09 16:07:18 [INFO] [TRAIN] epoch: 24, iter: 8700/40000, loss: 0.7764, lr: 0.008039, batch_cost: 0.7867, reader_cost: 0.00015, ips: 1.2711 samples/sec | ETA 06:50:24 2021-05-09 16:07:26 [INFO] [TRAIN] epoch: 24, iter: 8710/40000, loss: 0.4840, lr: 0.008037, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2723 samples/sec | ETA 06:49:53 2021-05-09 16:07:34 [INFO] [TRAIN] epoch: 24, iter: 8720/40000, loss: 0.3503, lr: 0.008035, batch_cost: 0.7860, reader_cost: 0.00017, ips: 1.2722 samples/sec | ETA 06:49:46 2021-05-09 16:07:42 [INFO] [TRAIN] epoch: 24, iter: 8730/40000, loss: 0.2588, lr: 0.008032, batch_cost: 0.7880, reader_cost: 0.00019, ips: 1.2690 samples/sec | ETA 06:50:41 2021-05-09 16:07:49 [INFO] [TRAIN] epoch: 24, iter: 8740/40000, loss: 0.5280, lr: 0.008030, batch_cost: 0.7865, reader_cost: 0.00018, ips: 1.2715 samples/sec | ETA 06:49:45 2021-05-09 16:07:57 [INFO] [TRAIN] epoch: 24, iter: 8750/40000, loss: 0.1906, lr: 0.008028, batch_cost: 0.7862, reader_cost: 0.00017, ips: 1.2720 samples/sec | ETA 06:49:28 2021-05-09 16:08:05 [INFO] [TRAIN] epoch: 24, iter: 8760/40000, loss: 0.2564, lr: 0.008026, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2709 samples/sec | ETA 06:49:40 2021-05-09 16:08:13 [INFO] [TRAIN] epoch: 24, iter: 8770/40000, loss: 0.3209, lr: 0.008023, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2723 samples/sec | ETA 06:49:07 2021-05-09 16:08:21 [INFO] [TRAIN] epoch: 24, iter: 8780/40000, loss: 0.5250, lr: 0.008021, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2726 samples/sec | ETA 06:48:52 2021-05-09 16:08:29 [INFO] [TRAIN] epoch: 24, iter: 8790/40000, loss: 0.3360, lr: 0.008019, batch_cost: 0.7865, reader_cost: 0.00018, ips: 1.2714 samples/sec | ETA 06:49:06 2021-05-09 16:08:37 [INFO] [TRAIN] epoch: 24, iter: 8800/40000, loss: 0.4232, lr: 0.008016, batch_cost: 0.7854, reader_cost: 0.00018, ips: 1.2733 samples/sec | ETA 06:48:24 2021-05-09 16:08:45 [INFO] [TRAIN] epoch: 24, iter: 8810/40000, loss: 0.3840, lr: 0.008014, batch_cost: 0.7868, reader_cost: 0.00017, ips: 1.2710 samples/sec | ETA 06:49:00 2021-05-09 16:08:52 [INFO] [TRAIN] epoch: 24, iter: 8820/40000, loss: 0.1816, lr: 0.008012, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2727 samples/sec | ETA 06:48:18 2021-05-09 16:09:00 [INFO] [TRAIN] epoch: 24, iter: 8830/40000, loss: 0.3658, lr: 0.008010, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2721 samples/sec | ETA 06:48:23 2021-05-09 16:09:08 [INFO] [TRAIN] epoch: 24, iter: 8840/40000, loss: 0.5394, lr: 0.008007, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2708 samples/sec | ETA 06:48:40 2021-05-09 16:09:16 [INFO] [TRAIN] epoch: 24, iter: 8850/40000, loss: 0.3786, lr: 0.008005, batch_cost: 0.7861, reader_cost: 0.00017, ips: 1.2721 samples/sec | ETA 06:48:07 2021-05-09 16:09:24 [INFO] [TRAIN] epoch: 24, iter: 8860/40000, loss: 0.5997, lr: 0.008003, batch_cost: 0.7849, reader_cost: 0.00017, ips: 1.2740 samples/sec | ETA 06:47:22 2021-05-09 16:09:32 [INFO] [TRAIN] epoch: 24, iter: 8870/40000, loss: 0.4358, lr: 0.008001, batch_cost: 0.7845, reader_cost: 0.00016, ips: 1.2747 samples/sec | ETA 06:47:01 2021-05-09 16:09:40 [INFO] [TRAIN] epoch: 24, iter: 8880/40000, loss: 0.3682, lr: 0.007998, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2732 samples/sec | ETA 06:47:22 2021-05-09 16:09:47 [INFO] [TRAIN] epoch: 24, iter: 8890/40000, loss: 0.1758, lr: 0.007996, batch_cost: 0.7882, reader_cost: 0.00017, ips: 1.2688 samples/sec | ETA 06:48:39 2021-05-09 16:09:55 [INFO] [TRAIN] epoch: 24, iter: 8900/40000, loss: 0.2444, lr: 0.007994, batch_cost: 0.7869, reader_cost: 0.00015, ips: 1.2708 samples/sec | ETA 06:47:53 2021-05-09 16:10:03 [INFO] [TRAIN] epoch: 24, iter: 8910/40000, loss: 0.4406, lr: 0.007991, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2717 samples/sec | ETA 06:47:27 2021-05-09 16:10:11 [INFO] [TRAIN] epoch: 24, iter: 8920/40000, loss: 0.4545, lr: 0.007989, batch_cost: 0.7845, reader_cost: 0.00013, ips: 1.2748 samples/sec | ETA 06:46:20 2021-05-09 16:10:22 [INFO] [TRAIN] epoch: 25, iter: 8930/40000, loss: 0.2899, lr: 0.007987, batch_cost: 1.0987, reader_cost: 0.25230, ips: 0.9102 samples/sec | ETA 09:28:55 2021-05-09 16:10:30 [INFO] [TRAIN] epoch: 25, iter: 8940/40000, loss: 0.2811, lr: 0.007985, batch_cost: 0.8014, reader_cost: 0.00034, ips: 1.2478 samples/sec | ETA 06:54:51 2021-05-09 16:10:38 [INFO] [TRAIN] epoch: 25, iter: 8950/40000, loss: 0.6596, lr: 0.007982, batch_cost: 0.7856, reader_cost: 0.00017, ips: 1.2730 samples/sec | ETA 06:46:32 2021-05-09 16:10:46 [INFO] [TRAIN] epoch: 25, iter: 8960/40000, loss: 0.6189, lr: 0.007980, batch_cost: 0.7853, reader_cost: 0.00015, ips: 1.2734 samples/sec | ETA 06:46:15 2021-05-09 16:10:54 [INFO] [TRAIN] epoch: 25, iter: 8970/40000, loss: 0.4513, lr: 0.007978, batch_cost: 0.7858, reader_cost: 0.00017, ips: 1.2726 samples/sec | ETA 06:46:23 2021-05-09 16:11:01 [INFO] [TRAIN] epoch: 25, iter: 8980/40000, loss: 0.6742, lr: 0.007975, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2725 samples/sec | ETA 06:46:17 2021-05-09 16:11:09 [INFO] [TRAIN] epoch: 25, iter: 8990/40000, loss: 0.4001, lr: 0.007973, batch_cost: 0.7850, reader_cost: 0.00015, ips: 1.2739 samples/sec | ETA 06:45:41 2021-05-09 16:11:17 [INFO] [TRAIN] epoch: 25, iter: 9000/40000, loss: 0.1497, lr: 0.007971, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 06:46:25 2021-05-09 16:11:17 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-09 16:14:48 [INFO] [EVAL] #Images: 500 mIoU: 0.7278 Acc: 0.9513 Kappa: 0.9368 2021-05-09 16:14:48 [INFO] [EVAL] Class IoU: [0.9771 0.819 0.9115 0.623 0.58 0.4257 0.5633 0.6806 0.908 0.6427 0.9382 0.7517 0.519 0.9327 0.7535 0.8329 0.7067 0.574 0.6882] 2021-05-09 16:14:48 [INFO] [EVAL] Class Acc: [0.9898 0.8928 0.9438 0.8701 0.7868 0.7826 0.7994 0.8717 0.9396 0.8459 0.9657 0.8374 0.7675 0.9646 0.8781 0.8765 0.9302 0.7151 0.7592] 2021-05-09 16:15:16 [INFO] [EVAL] The model with the best validation mIoU (0.7299) was saved at iter 8000. 2021-05-09 16:15:24 [INFO] [TRAIN] epoch: 25, iter: 9010/40000, loss: 0.3239, lr: 0.007969, batch_cost: 0.7797, reader_cost: 0.00024, ips: 1.2825 samples/sec | ETA 06:42:42 2021-05-09 16:15:33 [INFO] [TRAIN] epoch: 25, iter: 9020/40000, loss: 0.4027, lr: 0.007966, batch_cost: 0.7834, reader_cost: 0.00027, ips: 1.2764 samples/sec | ETA 06:44:31 2021-05-09 16:15:41 [INFO] [TRAIN] epoch: 25, iter: 9030/40000, loss: 0.2908, lr: 0.007964, batch_cost: 0.7851, reader_cost: 0.00018, ips: 1.2738 samples/sec | ETA 06:45:13 2021-05-09 16:15:49 [INFO] [TRAIN] epoch: 25, iter: 9040/40000, loss: 0.3214, lr: 0.007962, batch_cost: 0.7840, reader_cost: 0.00016, ips: 1.2755 samples/sec | ETA 06:44:33 2021-05-09 16:15:56 [INFO] [TRAIN] epoch: 25, iter: 9050/40000, loss: 0.1716, lr: 0.007959, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2710 samples/sec | ETA 06:45:50 2021-05-09 16:16:04 [INFO] [TRAIN] epoch: 25, iter: 9060/40000, loss: 0.4908, lr: 0.007957, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2718 samples/sec | ETA 06:45:28 2021-05-09 16:16:12 [INFO] [TRAIN] epoch: 25, iter: 9070/40000, loss: 0.2791, lr: 0.007955, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2715 samples/sec | ETA 06:45:26 2021-05-09 16:16:20 [INFO] [TRAIN] epoch: 25, iter: 9080/40000, loss: 0.5049, lr: 0.007953, batch_cost: 0.7849, reader_cost: 0.00015, ips: 1.2740 samples/sec | ETA 06:44:30 2021-05-09 16:16:28 [INFO] [TRAIN] epoch: 25, iter: 9090/40000, loss: 0.3266, lr: 0.007950, batch_cost: 0.7850, reader_cost: 0.00016, ips: 1.2740 samples/sec | ETA 06:44:22 2021-05-09 16:16:36 [INFO] [TRAIN] epoch: 25, iter: 9100/40000, loss: 0.2923, lr: 0.007948, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2729 samples/sec | ETA 06:44:35 2021-05-09 16:16:44 [INFO] [TRAIN] epoch: 25, iter: 9110/40000, loss: 0.5547, lr: 0.007946, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2720 samples/sec | ETA 06:44:43 2021-05-09 16:16:51 [INFO] [TRAIN] epoch: 25, iter: 9120/40000, loss: 0.2340, lr: 0.007943, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 06:44:28 2021-05-09 16:16:59 [INFO] [TRAIN] epoch: 25, iter: 9130/40000, loss: 0.2107, lr: 0.007941, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2736 samples/sec | ETA 06:43:58 2021-05-09 16:17:07 [INFO] [TRAIN] epoch: 25, iter: 9140/40000, loss: 0.4899, lr: 0.007939, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2711 samples/sec | ETA 06:44:37 2021-05-09 16:17:15 [INFO] [TRAIN] epoch: 25, iter: 9150/40000, loss: 0.2814, lr: 0.007937, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2733 samples/sec | ETA 06:43:49 2021-05-09 16:17:23 [INFO] [TRAIN] epoch: 25, iter: 9160/40000, loss: 0.3505, lr: 0.007934, batch_cost: 0.7843, reader_cost: 0.00016, ips: 1.2751 samples/sec | ETA 06:43:06 2021-05-09 16:17:31 [INFO] [TRAIN] epoch: 25, iter: 9170/40000, loss: 0.2343, lr: 0.007932, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 06:43:36 2021-05-09 16:17:39 [INFO] [TRAIN] epoch: 25, iter: 9180/40000, loss: 0.2693, lr: 0.007930, batch_cost: 0.7852, reader_cost: 0.00017, ips: 1.2736 samples/sec | ETA 06:43:18 2021-05-09 16:17:46 [INFO] [TRAIN] epoch: 25, iter: 9190/40000, loss: 0.3252, lr: 0.007927, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2732 samples/sec | ETA 06:43:19 2021-05-09 16:17:54 [INFO] [TRAIN] epoch: 25, iter: 9200/40000, loss: 0.4522, lr: 0.007925, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 06:43:30 2021-05-09 16:18:02 [INFO] [TRAIN] epoch: 25, iter: 9210/40000, loss: 0.5036, lr: 0.007923, batch_cost: 0.7867, reader_cost: 0.00015, ips: 1.2711 samples/sec | ETA 06:43:43 2021-05-09 16:18:10 [INFO] [TRAIN] epoch: 25, iter: 9220/40000, loss: 0.4797, lr: 0.007921, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2723 samples/sec | ETA 06:43:12 2021-05-09 16:18:18 [INFO] [TRAIN] epoch: 25, iter: 9230/40000, loss: 0.5497, lr: 0.007918, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2730 samples/sec | ETA 06:42:50 2021-05-09 16:18:26 [INFO] [TRAIN] epoch: 25, iter: 9240/40000, loss: 0.5194, lr: 0.007916, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2710 samples/sec | ETA 06:43:22 2021-05-09 16:18:34 [INFO] [TRAIN] epoch: 25, iter: 9250/40000, loss: 0.6589, lr: 0.007914, batch_cost: 0.7833, reader_cost: 0.00015, ips: 1.2766 samples/sec | ETA 06:41:27 2021-05-09 16:18:41 [INFO] [TRAIN] epoch: 25, iter: 9260/40000, loss: 0.1859, lr: 0.007911, batch_cost: 0.7853, reader_cost: 0.00017, ips: 1.2735 samples/sec | ETA 06:42:19 2021-05-09 16:18:49 [INFO] [TRAIN] epoch: 25, iter: 9270/40000, loss: 0.3539, lr: 0.007909, batch_cost: 0.7856, reader_cost: 0.00017, ips: 1.2729 samples/sec | ETA 06:42:21 2021-05-09 16:18:57 [INFO] [TRAIN] epoch: 25, iter: 9280/40000, loss: 0.3752, lr: 0.007907, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2719 samples/sec | ETA 06:42:33 2021-05-09 16:19:05 [INFO] [TRAIN] epoch: 25, iter: 9290/40000, loss: 0.3580, lr: 0.007905, batch_cost: 0.7845, reader_cost: 0.00016, ips: 1.2747 samples/sec | ETA 06:41:31 2021-05-09 16:19:13 [INFO] [TRAIN] epoch: 25, iter: 9300/40000, loss: 0.5165, lr: 0.007902, batch_cost: 0.7845, reader_cost: 0.00010, ips: 1.2746 samples/sec | ETA 06:41:25 2021-05-09 16:19:24 [INFO] [TRAIN] epoch: 26, iter: 9310/40000, loss: 0.4117, lr: 0.007900, batch_cost: 1.0973, reader_cost: 0.25195, ips: 0.9114 samples/sec | ETA 09:21:15 2021-05-09 16:19:32 [INFO] [TRAIN] epoch: 26, iter: 9320/40000, loss: 0.4131, lr: 0.007898, batch_cost: 0.7873, reader_cost: 0.00032, ips: 1.2701 samples/sec | ETA 06:42:35 2021-05-09 16:19:40 [INFO] [TRAIN] epoch: 26, iter: 9330/40000, loss: 0.2716, lr: 0.007895, batch_cost: 0.7857, reader_cost: 0.00017, ips: 1.2728 samples/sec | ETA 06:41:36 2021-05-09 16:19:47 [INFO] [TRAIN] epoch: 26, iter: 9340/40000, loss: 0.4300, lr: 0.007893, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 06:41:58 2021-05-09 16:19:55 [INFO] [TRAIN] epoch: 26, iter: 9350/40000, loss: 0.5447, lr: 0.007891, batch_cost: 0.7843, reader_cost: 0.00017, ips: 1.2749 samples/sec | ETA 06:40:40 2021-05-09 16:20:03 [INFO] [TRAIN] epoch: 26, iter: 9360/40000, loss: 0.3892, lr: 0.007889, batch_cost: 0.7859, reader_cost: 0.00017, ips: 1.2725 samples/sec | ETA 06:41:18 2021-05-09 16:20:11 [INFO] [TRAIN] epoch: 26, iter: 9370/40000, loss: 0.2374, lr: 0.007886, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 06:41:28 2021-05-09 16:20:19 [INFO] [TRAIN] epoch: 26, iter: 9380/40000, loss: 0.2764, lr: 0.007884, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2730 samples/sec | ETA 06:40:52 2021-05-09 16:20:27 [INFO] [TRAIN] epoch: 26, iter: 9390/40000, loss: 0.4181, lr: 0.007882, batch_cost: 0.7874, reader_cost: 0.00016, ips: 1.2699 samples/sec | ETA 06:41:43 2021-05-09 16:20:35 [INFO] [TRAIN] epoch: 26, iter: 9400/40000, loss: 0.2316, lr: 0.007879, batch_cost: 0.7852, reader_cost: 0.00018, ips: 1.2735 samples/sec | ETA 06:40:27 2021-05-09 16:20:42 [INFO] [TRAIN] epoch: 26, iter: 9410/40000, loss: 0.3596, lr: 0.007877, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 06:40:58 2021-05-09 16:20:50 [INFO] [TRAIN] epoch: 26, iter: 9420/40000, loss: 0.1406, lr: 0.007875, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2716 samples/sec | ETA 06:40:48 2021-05-09 16:20:58 [INFO] [TRAIN] epoch: 26, iter: 9430/40000, loss: 0.3251, lr: 0.007872, batch_cost: 0.7866, reader_cost: 0.00017, ips: 1.2713 samples/sec | ETA 06:40:45 2021-05-09 16:21:06 [INFO] [TRAIN] epoch: 26, iter: 9440/40000, loss: 0.6091, lr: 0.007870, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2730 samples/sec | ETA 06:40:06 2021-05-09 16:21:14 [INFO] [TRAIN] epoch: 26, iter: 9450/40000, loss: 0.3925, lr: 0.007868, batch_cost: 0.7853, reader_cost: 0.00015, ips: 1.2733 samples/sec | ETA 06:39:52 2021-05-09 16:21:22 [INFO] [TRAIN] epoch: 26, iter: 9460/40000, loss: 0.3439, lr: 0.007866, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2726 samples/sec | ETA 06:39:58 2021-05-09 16:21:30 [INFO] [TRAIN] epoch: 26, iter: 9470/40000, loss: 0.2814, lr: 0.007863, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2710 samples/sec | ETA 06:40:19 2021-05-09 16:21:37 [INFO] [TRAIN] epoch: 26, iter: 9480/40000, loss: 0.2572, lr: 0.007861, batch_cost: 0.7885, reader_cost: 0.00017, ips: 1.2683 samples/sec | ETA 06:41:03 2021-05-09 16:21:45 [INFO] [TRAIN] epoch: 26, iter: 9490/40000, loss: 0.2202, lr: 0.007859, batch_cost: 0.7881, reader_cost: 0.00017, ips: 1.2689 samples/sec | ETA 06:40:44 2021-05-09 16:21:53 [INFO] [TRAIN] epoch: 26, iter: 9500/40000, loss: 0.1283, lr: 0.007856, batch_cost: 0.7847, reader_cost: 0.00016, ips: 1.2743 samples/sec | ETA 06:38:54 2021-05-09 16:22:01 [INFO] [TRAIN] epoch: 26, iter: 9510/40000, loss: 0.3738, lr: 0.007854, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2730 samples/sec | ETA 06:39:10 2021-05-09 16:22:09 [INFO] [TRAIN] epoch: 26, iter: 9520/40000, loss: 0.2913, lr: 0.007852, batch_cost: 0.7858, reader_cost: 0.00015, ips: 1.2726 samples/sec | ETA 06:39:11 2021-05-09 16:22:17 [INFO] [TRAIN] epoch: 26, iter: 9530/40000, loss: 0.2490, lr: 0.007850, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 06:39:23 2021-05-09 16:22:25 [INFO] [TRAIN] epoch: 26, iter: 9540/40000, loss: 0.4570, lr: 0.007847, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2705 samples/sec | ETA 06:39:35 2021-05-09 16:22:32 [INFO] [TRAIN] epoch: 26, iter: 9550/40000, loss: 0.2560, lr: 0.007845, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2714 samples/sec | ETA 06:39:10 2021-05-09 16:22:40 [INFO] [TRAIN] epoch: 26, iter: 9560/40000, loss: 0.3784, lr: 0.007843, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2718 samples/sec | ETA 06:38:54 2021-05-09 16:22:48 [INFO] [TRAIN] epoch: 26, iter: 9570/40000, loss: 0.3643, lr: 0.007840, batch_cost: 0.7864, reader_cost: 0.00017, ips: 1.2715 samples/sec | ETA 06:38:51 2021-05-09 16:22:56 [INFO] [TRAIN] epoch: 26, iter: 9580/40000, loss: 0.5217, lr: 0.007838, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2718 samples/sec | ETA 06:38:38 2021-05-09 16:23:04 [INFO] [TRAIN] epoch: 26, iter: 9590/40000, loss: 0.4499, lr: 0.007836, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2711 samples/sec | ETA 06:38:45 2021-05-09 16:23:12 [INFO] [TRAIN] epoch: 26, iter: 9600/40000, loss: 0.4747, lr: 0.007834, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2706 samples/sec | ETA 06:38:46 2021-05-09 16:23:20 [INFO] [TRAIN] epoch: 26, iter: 9610/40000, loss: 0.5250, lr: 0.007831, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2721 samples/sec | ETA 06:38:09 2021-05-09 16:23:28 [INFO] [TRAIN] epoch: 26, iter: 9620/40000, loss: 0.3890, lr: 0.007829, batch_cost: 0.7853, reader_cost: 0.00017, ips: 1.2735 samples/sec | ETA 06:37:36 2021-05-09 16:23:35 [INFO] [TRAIN] epoch: 26, iter: 9630/40000, loss: 0.1177, lr: 0.007827, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2728 samples/sec | ETA 06:37:40 2021-05-09 16:23:43 [INFO] [TRAIN] epoch: 26, iter: 9640/40000, loss: 0.3318, lr: 0.007824, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 06:38:02 2021-05-09 16:23:51 [INFO] [TRAIN] epoch: 26, iter: 9650/40000, loss: 0.3105, lr: 0.007822, batch_cost: 0.7867, reader_cost: 0.00015, ips: 1.2711 samples/sec | ETA 06:37:57 2021-05-09 16:23:59 [INFO] [TRAIN] epoch: 26, iter: 9660/40000, loss: 0.2590, lr: 0.007820, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2716 samples/sec | ETA 06:37:39 2021-05-09 16:24:07 [INFO] [TRAIN] epoch: 26, iter: 9670/40000, loss: 0.4813, lr: 0.007818, batch_cost: 0.7839, reader_cost: 0.00023, ips: 1.2756 samples/sec | ETA 06:36:16 2021-05-09 16:24:18 [INFO] [TRAIN] epoch: 27, iter: 9680/40000, loss: 0.2751, lr: 0.007815, batch_cost: 1.1083, reader_cost: 0.26068, ips: 0.9023 samples/sec | ETA 09:20:03 2021-05-09 16:24:26 [INFO] [TRAIN] epoch: 27, iter: 9690/40000, loss: 0.5004, lr: 0.007813, batch_cost: 0.7959, reader_cost: 0.00034, ips: 1.2564 samples/sec | ETA 06:42:03 2021-05-09 16:24:34 [INFO] [TRAIN] epoch: 27, iter: 9700/40000, loss: 0.3567, lr: 0.007811, batch_cost: 0.7857, reader_cost: 0.00017, ips: 1.2728 samples/sec | ETA 06:36:46 2021-05-09 16:24:42 [INFO] [TRAIN] epoch: 27, iter: 9710/40000, loss: 0.5244, lr: 0.007808, batch_cost: 0.7859, reader_cost: 0.00017, ips: 1.2724 samples/sec | ETA 06:36:45 2021-05-09 16:24:49 [INFO] [TRAIN] epoch: 27, iter: 9720/40000, loss: 0.4400, lr: 0.007806, batch_cost: 0.7857, reader_cost: 0.00017, ips: 1.2728 samples/sec | ETA 06:36:30 2021-05-09 16:24:57 [INFO] [TRAIN] epoch: 27, iter: 9730/40000, loss: 0.3941, lr: 0.007804, batch_cost: 0.7878, reader_cost: 0.00016, ips: 1.2693 samples/sec | ETA 06:37:27 2021-05-09 16:25:05 [INFO] [TRAIN] epoch: 27, iter: 9740/40000, loss: 0.4949, lr: 0.007802, batch_cost: 0.7871, reader_cost: 0.00017, ips: 1.2706 samples/sec | ETA 06:36:56 2021-05-09 16:25:13 [INFO] [TRAIN] epoch: 27, iter: 9750/40000, loss: 0.1941, lr: 0.007799, batch_cost: 0.7866, reader_cost: 0.00018, ips: 1.2714 samples/sec | ETA 06:36:33 2021-05-09 16:25:21 [INFO] [TRAIN] epoch: 27, iter: 9760/40000, loss: 0.4463, lr: 0.007797, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 06:36:06 2021-05-09 16:25:29 [INFO] [TRAIN] epoch: 27, iter: 9770/40000, loss: 0.2201, lr: 0.007795, batch_cost: 0.7873, reader_cost: 0.00017, ips: 1.2702 samples/sec | ETA 06:36:39 2021-05-09 16:25:37 [INFO] [TRAIN] epoch: 27, iter: 9780/40000, loss: 0.3444, lr: 0.007792, batch_cost: 0.7866, reader_cost: 0.00018, ips: 1.2713 samples/sec | ETA 06:36:10 2021-05-09 16:25:45 [INFO] [TRAIN] epoch: 27, iter: 9790/40000, loss: 0.2264, lr: 0.007790, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 06:35:55 2021-05-09 16:25:52 [INFO] [TRAIN] epoch: 27, iter: 9800/40000, loss: 0.2424, lr: 0.007788, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2709 samples/sec | ETA 06:36:03 2021-05-09 16:26:00 [INFO] [TRAIN] epoch: 27, iter: 9810/40000, loss: 0.2902, lr: 0.007785, batch_cost: 0.7874, reader_cost: 0.00016, ips: 1.2700 samples/sec | ETA 06:36:11 2021-05-09 16:26:08 [INFO] [TRAIN] epoch: 27, iter: 9820/40000, loss: 0.3187, lr: 0.007783, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2711 samples/sec | ETA 06:35:43 2021-05-09 16:26:16 [INFO] [TRAIN] epoch: 27, iter: 9830/40000, loss: 0.2984, lr: 0.007781, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2703 samples/sec | ETA 06:35:50 2021-05-09 16:26:24 [INFO] [TRAIN] epoch: 27, iter: 9840/40000, loss: 0.3295, lr: 0.007779, batch_cost: 0.7846, reader_cost: 0.00015, ips: 1.2746 samples/sec | ETA 06:34:22 2021-05-09 16:26:32 [INFO] [TRAIN] epoch: 27, iter: 9850/40000, loss: 0.3023, lr: 0.007776, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2719 samples/sec | ETA 06:35:03 2021-05-09 16:26:40 [INFO] [TRAIN] epoch: 27, iter: 9860/40000, loss: 0.4485, lr: 0.007774, batch_cost: 0.7853, reader_cost: 0.00015, ips: 1.2734 samples/sec | ETA 06:34:28 2021-05-09 16:26:47 [INFO] [TRAIN] epoch: 27, iter: 9870/40000, loss: 0.1881, lr: 0.007772, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2735 samples/sec | ETA 06:34:19 2021-05-09 16:26:55 [INFO] [TRAIN] epoch: 27, iter: 9880/40000, loss: 0.3301, lr: 0.007769, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2719 samples/sec | ETA 06:34:40 2021-05-09 16:27:03 [INFO] [TRAIN] epoch: 27, iter: 9890/40000, loss: 0.4081, lr: 0.007767, batch_cost: 0.7859, reader_cost: 0.00014, ips: 1.2724 samples/sec | ETA 06:34:24 2021-05-09 16:27:11 [INFO] [TRAIN] epoch: 27, iter: 9900/40000, loss: 0.2937, lr: 0.007765, batch_cost: 0.7856, reader_cost: 0.00014, ips: 1.2730 samples/sec | ETA 06:34:05 2021-05-09 16:27:19 [INFO] [TRAIN] epoch: 27, iter: 9910/40000, loss: 0.2876, lr: 0.007763, batch_cost: 0.7858, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 06:34:03 2021-05-09 16:27:27 [INFO] [TRAIN] epoch: 27, iter: 9920/40000, loss: 0.2217, lr: 0.007760, batch_cost: 0.7867, reader_cost: 0.00017, ips: 1.2712 samples/sec | ETA 06:34:23 2021-05-09 16:27:35 [INFO] [TRAIN] epoch: 27, iter: 9930/40000, loss: 0.3040, lr: 0.007758, batch_cost: 0.7855, reader_cost: 0.00017, ips: 1.2731 samples/sec | ETA 06:33:38 2021-05-09 16:27:42 [INFO] [TRAIN] epoch: 27, iter: 9940/40000, loss: 0.2924, lr: 0.007756, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2733 samples/sec | ETA 06:33:28 2021-05-09 16:27:50 [INFO] [TRAIN] epoch: 27, iter: 9950/40000, loss: 0.4337, lr: 0.007753, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2719 samples/sec | ETA 06:33:46 2021-05-09 16:27:58 [INFO] [TRAIN] epoch: 27, iter: 9960/40000, loss: 0.5764, lr: 0.007751, batch_cost: 0.7844, reader_cost: 0.00016, ips: 1.2748 samples/sec | ETA 06:32:44 2021-05-09 16:28:06 [INFO] [TRAIN] epoch: 27, iter: 9970/40000, loss: 0.3915, lr: 0.007749, batch_cost: 0.7853, reader_cost: 0.00015, ips: 1.2735 samples/sec | ETA 06:33:01 2021-05-09 16:28:14 [INFO] [TRAIN] epoch: 27, iter: 9980/40000, loss: 0.5306, lr: 0.007747, batch_cost: 0.7869, reader_cost: 0.00015, ips: 1.2709 samples/sec | ETA 06:33:41 2021-05-09 16:28:22 [INFO] [TRAIN] epoch: 27, iter: 9990/40000, loss: 0.3403, lr: 0.007744, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 06:33:00 2021-05-09 16:28:30 [INFO] [TRAIN] epoch: 27, iter: 10000/40000, loss: 0.2604, lr: 0.007742, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2728 samples/sec | ETA 06:32:49 2021-05-09 16:28:30 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-09 16:32:00 [INFO] [EVAL] #Images: 500 mIoU: 0.7379 Acc: 0.9507 Kappa: 0.9360 2021-05-09 16:32:00 [INFO] [EVAL] Class IoU: [0.9762 0.8183 0.9083 0.5624 0.5908 0.4146 0.5685 0.6904 0.9047 0.6277 0.9376 0.753 0.5385 0.9318 0.8474 0.8781 0.7817 0.5764 0.7146] 2021-05-09 16:32:00 [INFO] [EVAL] Class Acc: [0.9934 0.8724 0.9475 0.8189 0.8439 0.8019 0.8142 0.8691 0.9256 0.8936 0.9674 0.8234 0.7569 0.9522 0.9504 0.9567 0.9364 0.8579 0.833 ] 2021-05-09 16:32:49 [INFO] [EVAL] The model with the best validation mIoU (0.7379) was saved at iter 10000. 2021-05-09 16:32:56 [INFO] [TRAIN] epoch: 27, iter: 10010/40000, loss: 0.4071, lr: 0.007740, batch_cost: 0.7810, reader_cost: 0.00066, ips: 1.2804 samples/sec | ETA 06:30:21 2021-05-09 16:33:04 [INFO] [TRAIN] epoch: 27, iter: 10020/40000, loss: 0.2521, lr: 0.007737, batch_cost: 0.7845, reader_cost: 0.00015, ips: 1.2746 samples/sec | ETA 06:32:00 2021-05-09 16:33:12 [INFO] [TRAIN] epoch: 27, iter: 10030/40000, loss: 0.3680, lr: 0.007735, batch_cost: 0.7846, reader_cost: 0.00015, ips: 1.2745 samples/sec | ETA 06:31:54 2021-05-09 16:33:20 [INFO] [TRAIN] epoch: 27, iter: 10040/40000, loss: 0.3171, lr: 0.007733, batch_cost: 0.7859, reader_cost: 0.00012, ips: 1.2723 samples/sec | ETA 06:32:27 2021-05-09 16:33:31 [INFO] [TRAIN] epoch: 28, iter: 10050/40000, loss: 0.3879, lr: 0.007730, batch_cost: 1.0951, reader_cost: 0.28560, ips: 0.9132 samples/sec | ETA 09:06:36 2021-05-09 16:33:39 [INFO] [TRAIN] epoch: 28, iter: 10060/40000, loss: 0.4378, lr: 0.007728, batch_cost: 0.8020, reader_cost: 0.00035, ips: 1.2469 samples/sec | ETA 06:40:10 2021-05-09 16:33:47 [INFO] [TRAIN] epoch: 28, iter: 10070/40000, loss: 0.5572, lr: 0.007726, batch_cost: 0.7866, reader_cost: 0.00018, ips: 1.2714 samples/sec | ETA 06:32:21 2021-05-09 16:33:55 [INFO] [TRAIN] epoch: 28, iter: 10080/40000, loss: 0.4108, lr: 0.007724, batch_cost: 0.7867, reader_cost: 0.00017, ips: 1.2712 samples/sec | ETA 06:32:17 2021-05-09 16:34:03 [INFO] [TRAIN] epoch: 28, iter: 10090/40000, loss: 0.4330, lr: 0.007721, batch_cost: 0.7862, reader_cost: 0.00017, ips: 1.2719 samples/sec | ETA 06:31:55 2021-05-09 16:34:10 [INFO] [TRAIN] epoch: 28, iter: 10100/40000, loss: 0.4428, lr: 0.007719, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2716 samples/sec | ETA 06:31:54 2021-05-09 16:34:18 [INFO] [TRAIN] epoch: 28, iter: 10110/40000, loss: 0.2313, lr: 0.007717, batch_cost: 0.7855, reader_cost: 0.00017, ips: 1.2731 samples/sec | ETA 06:31:18 2021-05-09 16:34:26 [INFO] [TRAIN] epoch: 28, iter: 10120/40000, loss: 0.2145, lr: 0.007714, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 06:31:26 2021-05-09 16:34:34 [INFO] [TRAIN] epoch: 28, iter: 10130/40000, loss: 0.3928, lr: 0.007712, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2727 samples/sec | ETA 06:31:10 2021-05-09 16:34:42 [INFO] [TRAIN] epoch: 28, iter: 10140/40000, loss: 0.2438, lr: 0.007710, batch_cost: 0.7851, reader_cost: 0.00017, ips: 1.2736 samples/sec | ETA 06:30:44 2021-05-09 16:34:50 [INFO] [TRAIN] epoch: 28, iter: 10150/40000, loss: 0.4061, lr: 0.007708, batch_cost: 0.7850, reader_cost: 0.00017, ips: 1.2739 samples/sec | ETA 06:30:31 2021-05-09 16:34:58 [INFO] [TRAIN] epoch: 28, iter: 10160/40000, loss: 0.2604, lr: 0.007705, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2720 samples/sec | ETA 06:30:59 2021-05-09 16:35:05 [INFO] [TRAIN] epoch: 28, iter: 10170/40000, loss: 0.1698, lr: 0.007703, batch_cost: 0.7860, reader_cost: 0.00017, ips: 1.2722 samples/sec | ETA 06:30:47 2021-05-09 16:35:13 [INFO] [TRAIN] epoch: 28, iter: 10180/40000, loss: 0.2719, lr: 0.007701, batch_cost: 0.7847, reader_cost: 0.00018, ips: 1.2743 samples/sec | ETA 06:30:01 2021-05-09 16:35:21 [INFO] [TRAIN] epoch: 28, iter: 10190/40000, loss: 0.5526, lr: 0.007698, batch_cost: 0.7860, reader_cost: 0.00018, ips: 1.2722 samples/sec | ETA 06:30:31 2021-05-09 16:35:29 [INFO] [TRAIN] epoch: 28, iter: 10200/40000, loss: 0.4854, lr: 0.007696, batch_cost: 0.7861, reader_cost: 0.00018, ips: 1.2721 samples/sec | ETA 06:30:26 2021-05-09 16:35:37 [INFO] [TRAIN] epoch: 28, iter: 10210/40000, loss: 0.1796, lr: 0.007694, batch_cost: 0.7866, reader_cost: 0.00019, ips: 1.2712 samples/sec | ETA 06:30:34 2021-05-09 16:35:45 [INFO] [TRAIN] epoch: 28, iter: 10220/40000, loss: 0.2782, lr: 0.007691, batch_cost: 0.7879, reader_cost: 0.00018, ips: 1.2691 samples/sec | ETA 06:31:04 2021-05-09 16:35:53 [INFO] [TRAIN] epoch: 28, iter: 10230/40000, loss: 0.3140, lr: 0.007689, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2723 samples/sec | ETA 06:29:58 2021-05-09 16:36:00 [INFO] [TRAIN] epoch: 28, iter: 10240/40000, loss: 0.2489, lr: 0.007687, batch_cost: 0.7847, reader_cost: 0.00015, ips: 1.2744 samples/sec | ETA 06:29:12 2021-05-09 16:36:08 [INFO] [TRAIN] epoch: 28, iter: 10250/40000, loss: 0.4558, lr: 0.007685, batch_cost: 0.7873, reader_cost: 0.00016, ips: 1.2701 samples/sec | ETA 06:30:22 2021-05-09 16:36:16 [INFO] [TRAIN] epoch: 28, iter: 10260/40000, loss: 0.3271, lr: 0.007682, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2711 samples/sec | ETA 06:29:57 2021-05-09 16:36:24 [INFO] [TRAIN] epoch: 28, iter: 10270/40000, loss: 0.7145, lr: 0.007680, batch_cost: 0.7871, reader_cost: 0.00015, ips: 1.2705 samples/sec | ETA 06:30:01 2021-05-09 16:36:32 [INFO] [TRAIN] epoch: 28, iter: 10280/40000, loss: 0.2594, lr: 0.007678, batch_cost: 0.7844, reader_cost: 0.00015, ips: 1.2749 samples/sec | ETA 06:28:31 2021-05-09 16:36:40 [INFO] [TRAIN] epoch: 28, iter: 10290/40000, loss: 0.2010, lr: 0.007675, batch_cost: 0.7847, reader_cost: 0.00014, ips: 1.2743 samples/sec | ETA 06:28:34 2021-05-09 16:36:48 [INFO] [TRAIN] epoch: 28, iter: 10300/40000, loss: 0.2655, lr: 0.007673, batch_cost: 0.7861, reader_cost: 0.00017, ips: 1.2721 samples/sec | ETA 06:29:06 2021-05-09 16:36:56 [INFO] [TRAIN] epoch: 28, iter: 10310/40000, loss: 0.3139, lr: 0.007671, batch_cost: 0.7854, reader_cost: 0.00015, ips: 1.2733 samples/sec | ETA 06:28:37 2021-05-09 16:37:03 [INFO] [TRAIN] epoch: 28, iter: 10320/40000, loss: 0.4180, lr: 0.007669, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2717 samples/sec | ETA 06:28:58 2021-05-09 16:37:11 [INFO] [TRAIN] epoch: 28, iter: 10330/40000, loss: 0.5556, lr: 0.007666, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2725 samples/sec | ETA 06:28:36 2021-05-09 16:37:19 [INFO] [TRAIN] epoch: 28, iter: 10340/40000, loss: 0.4512, lr: 0.007664, batch_cost: 0.7876, reader_cost: 0.00017, ips: 1.2697 samples/sec | ETA 06:29:20 2021-05-09 16:37:27 [INFO] [TRAIN] epoch: 28, iter: 10350/40000, loss: 0.6006, lr: 0.007662, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2735 samples/sec | ETA 06:28:02 2021-05-09 16:37:35 [INFO] [TRAIN] epoch: 28, iter: 10360/40000, loss: 0.4719, lr: 0.007659, batch_cost: 0.7877, reader_cost: 0.00015, ips: 1.2695 samples/sec | ETA 06:29:07 2021-05-09 16:37:43 [INFO] [TRAIN] epoch: 28, iter: 10370/40000, loss: 0.3008, lr: 0.007657, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2720 samples/sec | ETA 06:28:13 2021-05-09 16:37:51 [INFO] [TRAIN] epoch: 28, iter: 10380/40000, loss: 0.1790, lr: 0.007655, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2716 samples/sec | ETA 06:28:14 2021-05-09 16:37:58 [INFO] [TRAIN] epoch: 28, iter: 10390/40000, loss: 0.3945, lr: 0.007652, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2714 samples/sec | ETA 06:28:08 2021-05-09 16:38:06 [INFO] [TRAIN] epoch: 28, iter: 10400/40000, loss: 0.4056, lr: 0.007650, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2710 samples/sec | ETA 06:28:09 2021-05-09 16:38:14 [INFO] [TRAIN] epoch: 28, iter: 10410/40000, loss: 0.4779, lr: 0.007648, batch_cost: 0.7862, reader_cost: 0.00012, ips: 1.2720 samples/sec | ETA 06:27:43 2021-05-09 16:38:25 [INFO] [TRAIN] epoch: 29, iter: 10420/40000, loss: 0.4847, lr: 0.007646, batch_cost: 1.1031, reader_cost: 0.28383, ips: 0.9065 samples/sec | ETA 09:03:50 2021-05-09 16:38:33 [INFO] [TRAIN] epoch: 29, iter: 10430/40000, loss: 0.4790, lr: 0.007643, batch_cost: 0.7916, reader_cost: 0.00036, ips: 1.2632 samples/sec | ETA 06:30:07 2021-05-09 16:38:41 [INFO] [TRAIN] epoch: 29, iter: 10440/40000, loss: 0.5705, lr: 0.007641, batch_cost: 0.7873, reader_cost: 0.00016, ips: 1.2701 samples/sec | ETA 06:27:53 2021-05-09 16:38:49 [INFO] [TRAIN] epoch: 29, iter: 10450/40000, loss: 0.5580, lr: 0.007639, batch_cost: 0.7863, reader_cost: 0.00017, ips: 1.2717 samples/sec | ETA 06:27:16 2021-05-09 16:38:57 [INFO] [TRAIN] epoch: 29, iter: 10460/40000, loss: 0.4686, lr: 0.007636, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2713 samples/sec | ETA 06:27:16 2021-05-09 16:39:05 [INFO] [TRAIN] epoch: 29, iter: 10470/40000, loss: 0.4886, lr: 0.007634, batch_cost: 0.7871, reader_cost: 0.00015, ips: 1.2705 samples/sec | ETA 06:27:22 2021-05-09 16:39:12 [INFO] [TRAIN] epoch: 29, iter: 10480/40000, loss: 0.4426, lr: 0.007632, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2712 samples/sec | ETA 06:27:01 2021-05-09 16:39:20 [INFO] [TRAIN] epoch: 29, iter: 10490/40000, loss: 0.1684, lr: 0.007630, batch_cost: 0.7878, reader_cost: 0.00016, ips: 1.2694 samples/sec | ETA 06:27:27 2021-05-09 16:39:28 [INFO] [TRAIN] epoch: 29, iter: 10500/40000, loss: 0.3059, lr: 0.007627, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2715 samples/sec | ETA 06:26:41 2021-05-09 16:39:36 [INFO] [TRAIN] epoch: 29, iter: 10510/40000, loss: 0.3009, lr: 0.007625, batch_cost: 0.7859, reader_cost: 0.00018, ips: 1.2724 samples/sec | ETA 06:26:16 2021-05-09 16:39:44 [INFO] [TRAIN] epoch: 29, iter: 10520/40000, loss: 0.2524, lr: 0.007623, batch_cost: 0.7872, reader_cost: 0.00018, ips: 1.2704 samples/sec | ETA 06:26:46 2021-05-09 16:39:52 [INFO] [TRAIN] epoch: 29, iter: 10530/40000, loss: 0.3379, lr: 0.007620, batch_cost: 0.7873, reader_cost: 0.00017, ips: 1.2702 samples/sec | ETA 06:26:41 2021-05-09 16:40:00 [INFO] [TRAIN] epoch: 29, iter: 10540/40000, loss: 0.1267, lr: 0.007618, batch_cost: 0.7872, reader_cost: 0.00017, ips: 1.2704 samples/sec | ETA 06:26:30 2021-05-09 16:40:08 [INFO] [TRAIN] epoch: 29, iter: 10550/40000, loss: 0.3643, lr: 0.007616, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 06:25:50 2021-05-09 16:40:15 [INFO] [TRAIN] epoch: 29, iter: 10560/40000, loss: 0.3550, lr: 0.007613, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 06:25:31 2021-05-09 16:40:23 [INFO] [TRAIN] epoch: 29, iter: 10570/40000, loss: 0.3807, lr: 0.007611, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2709 samples/sec | ETA 06:25:56 2021-05-09 16:40:31 [INFO] [TRAIN] epoch: 29, iter: 10580/40000, loss: 0.3164, lr: 0.007609, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2714 samples/sec | ETA 06:25:39 2021-05-09 16:40:39 [INFO] [TRAIN] epoch: 29, iter: 10590/40000, loss: 0.1663, lr: 0.007607, batch_cost: 0.7876, reader_cost: 0.00016, ips: 1.2696 samples/sec | ETA 06:26:03 2021-05-09 16:40:47 [INFO] [TRAIN] epoch: 29, iter: 10600/40000, loss: 0.2283, lr: 0.007604, batch_cost: 0.7876, reader_cost: 0.00016, ips: 1.2697 samples/sec | ETA 06:25:54 2021-05-09 16:40:55 [INFO] [TRAIN] epoch: 29, iter: 10610/40000, loss: 0.1129, lr: 0.007602, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 06:25:20 2021-05-09 16:41:03 [INFO] [TRAIN] epoch: 29, iter: 10620/40000, loss: 0.2490, lr: 0.007600, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2727 samples/sec | ETA 06:24:44 2021-05-09 16:41:10 [INFO] [TRAIN] epoch: 29, iter: 10630/40000, loss: 0.4273, lr: 0.007597, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 06:24:47 2021-05-09 16:41:18 [INFO] [TRAIN] epoch: 29, iter: 10640/40000, loss: 0.1655, lr: 0.007595, batch_cost: 0.7836, reader_cost: 0.00017, ips: 1.2762 samples/sec | ETA 06:23:26 2021-05-09 16:41:26 [INFO] [TRAIN] epoch: 29, iter: 10650/40000, loss: 0.3685, lr: 0.007593, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2730 samples/sec | ETA 06:24:16 2021-05-09 16:41:34 [INFO] [TRAIN] epoch: 29, iter: 10660/40000, loss: 0.3034, lr: 0.007590, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2725 samples/sec | ETA 06:24:16 2021-05-09 16:41:42 [INFO] [TRAIN] epoch: 29, iter: 10670/40000, loss: 0.2300, lr: 0.007588, batch_cost: 0.7885, reader_cost: 0.00017, ips: 1.2683 samples/sec | ETA 06:25:26 2021-05-09 16:41:50 [INFO] [TRAIN] epoch: 29, iter: 10680/40000, loss: 0.2706, lr: 0.007586, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2728 samples/sec | ETA 06:23:56 2021-05-09 16:41:58 [INFO] [TRAIN] epoch: 29, iter: 10690/40000, loss: 0.3351, lr: 0.007584, batch_cost: 0.7864, reader_cost: 0.00017, ips: 1.2716 samples/sec | ETA 06:24:09 2021-05-09 16:42:05 [INFO] [TRAIN] epoch: 29, iter: 10700/40000, loss: 0.5103, lr: 0.007581, batch_cost: 0.7851, reader_cost: 0.00017, ips: 1.2737 samples/sec | ETA 06:23:24 2021-05-09 16:42:13 [INFO] [TRAIN] epoch: 29, iter: 10710/40000, loss: 0.3881, lr: 0.007579, batch_cost: 0.7856, reader_cost: 0.00018, ips: 1.2729 samples/sec | ETA 06:23:30 2021-05-09 16:42:21 [INFO] [TRAIN] epoch: 29, iter: 10720/40000, loss: 0.6188, lr: 0.007577, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 06:23:35 2021-05-09 16:42:29 [INFO] [TRAIN] epoch: 29, iter: 10730/40000, loss: 0.4384, lr: 0.007574, batch_cost: 0.7852, reader_cost: 0.00017, ips: 1.2736 samples/sec | ETA 06:23:02 2021-05-09 16:42:37 [INFO] [TRAIN] epoch: 29, iter: 10740/40000, loss: 0.3025, lr: 0.007572, batch_cost: 0.7845, reader_cost: 0.00015, ips: 1.2747 samples/sec | ETA 06:22:33 2021-05-09 16:42:45 [INFO] [TRAIN] epoch: 29, iter: 10750/40000, loss: 0.2072, lr: 0.007570, batch_cost: 0.7876, reader_cost: 0.00017, ips: 1.2696 samples/sec | ETA 06:23:58 2021-05-09 16:42:53 [INFO] [TRAIN] epoch: 29, iter: 10760/40000, loss: 0.3754, lr: 0.007567, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2715 samples/sec | ETA 06:23:15 2021-05-09 16:43:01 [INFO] [TRAIN] epoch: 29, iter: 10770/40000, loss: 0.3343, lr: 0.007565, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2723 samples/sec | ETA 06:22:53 2021-05-09 16:43:08 [INFO] [TRAIN] epoch: 29, iter: 10780/40000, loss: 0.3257, lr: 0.007563, batch_cost: 0.7859, reader_cost: 0.00014, ips: 1.2725 samples/sec | ETA 06:22:43 2021-05-09 16:43:19 [INFO] [TRAIN] epoch: 30, iter: 10790/40000, loss: 0.4358, lr: 0.007561, batch_cost: 1.0843, reader_cost: 0.23919, ips: 0.9223 samples/sec | ETA 08:47:51 2021-05-09 16:43:27 [INFO] [TRAIN] epoch: 30, iter: 10800/40000, loss: 0.1585, lr: 0.007558, batch_cost: 0.8019, reader_cost: 0.00035, ips: 1.2471 samples/sec | ETA 06:30:14 2021-05-09 16:43:35 [INFO] [TRAIN] epoch: 30, iter: 10810/40000, loss: 0.5404, lr: 0.007556, batch_cost: 0.7854, reader_cost: 0.00017, ips: 1.2732 samples/sec | ETA 06:22:05 2021-05-09 16:43:43 [INFO] [TRAIN] epoch: 30, iter: 10820/40000, loss: 0.3652, lr: 0.007554, batch_cost: 0.7868, reader_cost: 0.00018, ips: 1.2710 samples/sec | ETA 06:22:38 2021-05-09 16:43:51 [INFO] [TRAIN] epoch: 30, iter: 10830/40000, loss: 0.3960, lr: 0.007551, batch_cost: 0.7872, reader_cost: 0.00017, ips: 1.2704 samples/sec | ETA 06:22:41 2021-05-09 16:43:59 [INFO] [TRAIN] epoch: 30, iter: 10840/40000, loss: 0.6748, lr: 0.007549, batch_cost: 0.7867, reader_cost: 0.00017, ips: 1.2711 samples/sec | ETA 06:22:20 2021-05-09 16:44:07 [INFO] [TRAIN] epoch: 30, iter: 10850/40000, loss: 0.7523, lr: 0.007547, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2724 samples/sec | ETA 06:21:49 2021-05-09 16:44:14 [INFO] [TRAIN] epoch: 30, iter: 10860/40000, loss: 0.2878, lr: 0.007544, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2728 samples/sec | ETA 06:21:34 2021-05-09 16:44:22 [INFO] [TRAIN] epoch: 30, iter: 10870/40000, loss: 0.3466, lr: 0.007542, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2723 samples/sec | ETA 06:21:34 2021-05-09 16:44:30 [INFO] [TRAIN] epoch: 30, iter: 10880/40000, loss: 0.3795, lr: 0.007540, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2730 samples/sec | ETA 06:21:14 2021-05-09 16:44:38 [INFO] [TRAIN] epoch: 30, iter: 10890/40000, loss: 0.2497, lr: 0.007538, batch_cost: 0.7871, reader_cost: 0.00015, ips: 1.2704 samples/sec | ETA 06:21:53 2021-05-09 16:44:46 [INFO] [TRAIN] epoch: 30, iter: 10900/40000, loss: 0.3161, lr: 0.007535, batch_cost: 0.7850, reader_cost: 0.00017, ips: 1.2739 samples/sec | ETA 06:20:43 2021-05-09 16:44:54 [INFO] [TRAIN] epoch: 30, iter: 10910/40000, loss: 0.1551, lr: 0.007533, batch_cost: 0.7867, reader_cost: 0.00017, ips: 1.2711 samples/sec | ETA 06:21:24 2021-05-09 16:45:02 [INFO] [TRAIN] epoch: 30, iter: 10920/40000, loss: 0.3985, lr: 0.007531, batch_cost: 0.7886, reader_cost: 0.00017, ips: 1.2680 samples/sec | ETA 06:22:13 2021-05-09 16:45:09 [INFO] [TRAIN] epoch: 30, iter: 10930/40000, loss: 0.3426, lr: 0.007528, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2721 samples/sec | ETA 06:20:52 2021-05-09 16:45:17 [INFO] [TRAIN] epoch: 30, iter: 10940/40000, loss: 0.3533, lr: 0.007526, batch_cost: 0.7858, reader_cost: 0.00015, ips: 1.2726 samples/sec | ETA 06:20:35 2021-05-09 16:45:25 [INFO] [TRAIN] epoch: 30, iter: 10950/40000, loss: 0.3106, lr: 0.007524, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2735 samples/sec | ETA 06:20:10 2021-05-09 16:45:33 [INFO] [TRAIN] epoch: 30, iter: 10960/40000, loss: 0.3062, lr: 0.007521, batch_cost: 0.7858, reader_cost: 0.00015, ips: 1.2725 samples/sec | ETA 06:20:20 2021-05-09 16:45:41 [INFO] [TRAIN] epoch: 30, iter: 10970/40000, loss: 0.3432, lr: 0.007519, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 06:20:03 2021-05-09 16:45:49 [INFO] [TRAIN] epoch: 30, iter: 10980/40000, loss: 0.1461, lr: 0.007517, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 06:20:23 2021-05-09 16:45:57 [INFO] [TRAIN] epoch: 30, iter: 10990/40000, loss: 0.1852, lr: 0.007515, batch_cost: 0.7853, reader_cost: 0.00017, ips: 1.2735 samples/sec | ETA 06:19:40 2021-05-09 16:46:04 [INFO] [TRAIN] epoch: 30, iter: 11000/40000, loss: 0.3528, lr: 0.007512, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 06:20:07 2021-05-09 16:46:05 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-09 16:49:35 [INFO] [EVAL] #Images: 500 mIoU: 0.7461 Acc: 0.9530 Kappa: 0.9390 2021-05-09 16:49:35 [INFO] [EVAL] Class IoU: [0.9767 0.8206 0.9166 0.6268 0.6045 0.4657 0.5865 0.6996 0.9104 0.6537 0.9372 0.7568 0.5162 0.9361 0.8206 0.8662 0.7559 0.6111 0.7149] 2021-05-09 16:49:35 [INFO] [EVAL] Class Acc: [0.9911 0.8813 0.9517 0.8405 0.7669 0.7372 0.7919 0.8675 0.946 0.8236 0.9579 0.8353 0.7944 0.9624 0.9165 0.9255 0.8875 0.8031 0.8372] 2021-05-09 16:50:23 [INFO] [EVAL] The model with the best validation mIoU (0.7461) was saved at iter 11000. 2021-05-09 16:50:31 [INFO] [TRAIN] epoch: 30, iter: 11010/40000, loss: 0.2381, lr: 0.007510, batch_cost: 0.7845, reader_cost: 0.00042, ips: 1.2747 samples/sec | ETA 06:19:03 2021-05-09 16:50:39 [INFO] [TRAIN] epoch: 30, iter: 11020/40000, loss: 0.2956, lr: 0.007508, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2723 samples/sec | ETA 06:19:37 2021-05-09 16:50:47 [INFO] [TRAIN] epoch: 30, iter: 11030/40000, loss: 0.2453, lr: 0.007505, batch_cost: 0.7843, reader_cost: 0.00016, ips: 1.2750 samples/sec | ETA 06:18:41 2021-05-09 16:50:55 [INFO] [TRAIN] epoch: 30, iter: 11040/40000, loss: 0.2021, lr: 0.007503, batch_cost: 0.7872, reader_cost: 0.00017, ips: 1.2703 samples/sec | ETA 06:19:57 2021-05-09 16:51:03 [INFO] [TRAIN] epoch: 30, iter: 11050/40000, loss: 0.1783, lr: 0.007501, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2714 samples/sec | ETA 06:19:30 2021-05-09 16:51:11 [INFO] [TRAIN] epoch: 30, iter: 11060/40000, loss: 0.2534, lr: 0.007498, batch_cost: 0.7860, reader_cost: 0.00017, ips: 1.2723 samples/sec | ETA 06:19:06 2021-05-09 16:51:18 [INFO] [TRAIN] epoch: 30, iter: 11070/40000, loss: 0.6499, lr: 0.007496, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2704 samples/sec | ETA 06:19:32 2021-05-09 16:51:26 [INFO] [TRAIN] epoch: 30, iter: 11080/40000, loss: 0.6556, lr: 0.007494, batch_cost: 0.7861, reader_cost: 0.00017, ips: 1.2721 samples/sec | ETA 06:18:54 2021-05-09 16:51:34 [INFO] [TRAIN] epoch: 30, iter: 11090/40000, loss: 0.4908, lr: 0.007492, batch_cost: 0.7871, reader_cost: 0.00015, ips: 1.2704 samples/sec | ETA 06:19:15 2021-05-09 16:51:42 [INFO] [TRAIN] epoch: 30, iter: 11100/40000, loss: 0.4037, lr: 0.007489, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2704 samples/sec | ETA 06:19:08 2021-05-09 16:51:50 [INFO] [TRAIN] epoch: 30, iter: 11110/40000, loss: 0.3811, lr: 0.007487, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2706 samples/sec | ETA 06:18:56 2021-05-09 16:51:58 [INFO] [TRAIN] epoch: 30, iter: 11120/40000, loss: 0.1893, lr: 0.007485, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2713 samples/sec | ETA 06:18:36 2021-05-09 16:52:06 [INFO] [TRAIN] epoch: 30, iter: 11130/40000, loss: 0.3390, lr: 0.007482, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2713 samples/sec | ETA 06:18:28 2021-05-09 16:52:13 [INFO] [TRAIN] epoch: 30, iter: 11140/40000, loss: 0.3732, lr: 0.007480, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2707 samples/sec | ETA 06:18:31 2021-05-09 16:52:21 [INFO] [TRAIN] epoch: 30, iter: 11150/40000, loss: 0.3469, lr: 0.007478, batch_cost: 0.7870, reader_cost: 0.00015, ips: 1.2706 samples/sec | ETA 06:18:25 2021-05-09 16:52:29 [INFO] [TRAIN] epoch: 30, iter: 11160/40000, loss: 0.3579, lr: 0.007475, batch_cost: 0.7827, reader_cost: 0.00009, ips: 1.2776 samples/sec | ETA 06:16:13 2021-05-09 16:52:40 [INFO] [TRAIN] epoch: 31, iter: 11170/40000, loss: 0.2098, lr: 0.007473, batch_cost: 1.1029, reader_cost: 0.23855, ips: 0.9067 samples/sec | ETA 08:49:56 2021-05-09 16:52:48 [INFO] [TRAIN] epoch: 31, iter: 11180/40000, loss: 0.5163, lr: 0.007471, batch_cost: 0.7900, reader_cost: 0.00032, ips: 1.2659 samples/sec | ETA 06:19:27 2021-05-09 16:52:56 [INFO] [TRAIN] epoch: 31, iter: 11190/40000, loss: 0.4065, lr: 0.007469, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2720 samples/sec | ETA 06:17:30 2021-05-09 16:53:04 [INFO] [TRAIN] epoch: 31, iter: 11200/40000, loss: 0.5505, lr: 0.007466, batch_cost: 0.7858, reader_cost: 0.00015, ips: 1.2726 samples/sec | ETA 06:17:10 2021-05-09 16:53:12 [INFO] [TRAIN] epoch: 31, iter: 11210/40000, loss: 0.5271, lr: 0.007464, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2721 samples/sec | ETA 06:17:12 2021-05-09 16:53:20 [INFO] [TRAIN] epoch: 31, iter: 11220/40000, loss: 0.3404, lr: 0.007462, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2731 samples/sec | ETA 06:16:47 2021-05-09 16:53:27 [INFO] [TRAIN] epoch: 31, iter: 11230/40000, loss: 0.1936, lr: 0.007459, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2728 samples/sec | ETA 06:16:43 2021-05-09 16:53:35 [INFO] [TRAIN] epoch: 31, iter: 11240/40000, loss: 0.3382, lr: 0.007457, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2735 samples/sec | ETA 06:16:23 2021-05-09 16:53:43 [INFO] [TRAIN] epoch: 31, iter: 11250/40000, loss: 0.3628, lr: 0.007455, batch_cost: 0.7872, reader_cost: 0.00015, ips: 1.2704 samples/sec | ETA 06:17:10 2021-05-09 16:53:51 [INFO] [TRAIN] epoch: 31, iter: 11260/40000, loss: 0.2452, lr: 0.007452, batch_cost: 0.7856, reader_cost: 0.00014, ips: 1.2729 samples/sec | ETA 06:16:18 2021-05-09 16:53:59 [INFO] [TRAIN] epoch: 31, iter: 11270/40000, loss: 0.2648, lr: 0.007450, batch_cost: 0.7875, reader_cost: 0.00015, ips: 1.2698 samples/sec | ETA 06:17:04 2021-05-09 16:54:07 [INFO] [TRAIN] epoch: 31, iter: 11280/40000, loss: 0.1507, lr: 0.007448, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 06:16:32 2021-05-09 16:54:15 [INFO] [TRAIN] epoch: 31, iter: 11290/40000, loss: 0.3014, lr: 0.007446, batch_cost: 0.7870, reader_cost: 0.00015, ips: 1.2707 samples/sec | ETA 06:16:33 2021-05-09 16:54:22 [INFO] [TRAIN] epoch: 31, iter: 11300/40000, loss: 0.4980, lr: 0.007443, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2709 samples/sec | ETA 06:16:21 2021-05-09 16:54:30 [INFO] [TRAIN] epoch: 31, iter: 11310/40000, loss: 0.3918, lr: 0.007441, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2732 samples/sec | ETA 06:15:34 2021-05-09 16:54:38 [INFO] [TRAIN] epoch: 31, iter: 11320/40000, loss: 0.3313, lr: 0.007439, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2726 samples/sec | ETA 06:15:36 2021-05-09 16:54:46 [INFO] [TRAIN] epoch: 31, iter: 11330/40000, loss: 0.3194, lr: 0.007436, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2719 samples/sec | ETA 06:15:41 2021-05-09 16:54:54 [INFO] [TRAIN] epoch: 31, iter: 11340/40000, loss: 0.3498, lr: 0.007434, batch_cost: 0.7853, reader_cost: 0.00015, ips: 1.2734 samples/sec | ETA 06:15:06 2021-05-09 16:55:02 [INFO] [TRAIN] epoch: 31, iter: 11350/40000, loss: 0.1429, lr: 0.007432, batch_cost: 0.7859, reader_cost: 0.00014, ips: 1.2724 samples/sec | ETA 06:15:16 2021-05-09 16:55:10 [INFO] [TRAIN] epoch: 31, iter: 11360/40000, loss: 0.1512, lr: 0.007429, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2711 samples/sec | ETA 06:15:32 2021-05-09 16:55:17 [INFO] [TRAIN] epoch: 31, iter: 11370/40000, loss: 0.3529, lr: 0.007427, batch_cost: 0.7845, reader_cost: 0.00015, ips: 1.2748 samples/sec | ETA 06:14:18 2021-05-09 16:55:25 [INFO] [TRAIN] epoch: 31, iter: 11380/40000, loss: 0.2419, lr: 0.007425, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2728 samples/sec | ETA 06:14:45 2021-05-09 16:55:33 [INFO] [TRAIN] epoch: 31, iter: 11390/40000, loss: 0.2459, lr: 0.007423, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2725 samples/sec | ETA 06:14:43 2021-05-09 16:55:41 [INFO] [TRAIN] epoch: 31, iter: 11400/40000, loss: 0.3536, lr: 0.007420, batch_cost: 0.7838, reader_cost: 0.00017, ips: 1.2758 samples/sec | ETA 06:13:37 2021-05-09 16:55:49 [INFO] [TRAIN] epoch: 31, iter: 11410/40000, loss: 0.3658, lr: 0.007418, batch_cost: 0.7849, reader_cost: 0.00016, ips: 1.2740 samples/sec | ETA 06:14:01 2021-05-09 16:55:57 [INFO] [TRAIN] epoch: 31, iter: 11420/40000, loss: 0.2769, lr: 0.007416, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2735 samples/sec | ETA 06:14:01 2021-05-09 16:56:05 [INFO] [TRAIN] epoch: 31, iter: 11430/40000, loss: 0.4128, lr: 0.007413, batch_cost: 0.7847, reader_cost: 0.00018, ips: 1.2744 samples/sec | ETA 06:13:38 2021-05-09 16:56:12 [INFO] [TRAIN] epoch: 31, iter: 11440/40000, loss: 0.5061, lr: 0.007411, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2729 samples/sec | ETA 06:13:57 2021-05-09 16:56:20 [INFO] [TRAIN] epoch: 31, iter: 11450/40000, loss: 0.4129, lr: 0.007409, batch_cost: 0.7834, reader_cost: 0.00016, ips: 1.2765 samples/sec | ETA 06:12:45 2021-05-09 16:56:28 [INFO] [TRAIN] epoch: 31, iter: 11460/40000, loss: 0.3939, lr: 0.007406, batch_cost: 0.7847, reader_cost: 0.00015, ips: 1.2744 samples/sec | ETA 06:13:15 2021-05-09 16:56:36 [INFO] [TRAIN] epoch: 31, iter: 11470/40000, loss: 0.3610, lr: 0.007404, batch_cost: 0.7835, reader_cost: 0.00016, ips: 1.2764 samples/sec | ETA 06:12:32 2021-05-09 16:56:44 [INFO] [TRAIN] epoch: 31, iter: 11480/40000, loss: 0.4128, lr: 0.007402, batch_cost: 0.7847, reader_cost: 0.00016, ips: 1.2743 samples/sec | ETA 06:13:00 2021-05-09 16:56:52 [INFO] [TRAIN] epoch: 31, iter: 11490/40000, loss: 0.1258, lr: 0.007399, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2729 samples/sec | ETA 06:13:17 2021-05-09 16:56:59 [INFO] [TRAIN] epoch: 31, iter: 11500/40000, loss: 0.3003, lr: 0.007397, batch_cost: 0.7854, reader_cost: 0.00015, ips: 1.2732 samples/sec | ETA 06:13:04 2021-05-09 16:57:07 [INFO] [TRAIN] epoch: 31, iter: 11510/40000, loss: 0.3069, lr: 0.007395, batch_cost: 0.7842, reader_cost: 0.00018, ips: 1.2753 samples/sec | ETA 06:12:20 2021-05-09 16:57:15 [INFO] [TRAIN] epoch: 31, iter: 11520/40000, loss: 0.3662, lr: 0.007393, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2726 samples/sec | ETA 06:12:59 2021-05-09 16:57:23 [INFO] [TRAIN] epoch: 31, iter: 11530/40000, loss: 0.4296, lr: 0.007390, batch_cost: 0.7831, reader_cost: 0.00024, ips: 1.2771 samples/sec | ETA 06:11:33 2021-05-09 16:57:34 [INFO] [TRAIN] epoch: 32, iter: 11540/40000, loss: 0.3215, lr: 0.007388, batch_cost: 1.0925, reader_cost: 0.25264, ips: 0.9154 samples/sec | ETA 08:38:11 2021-05-09 16:57:42 [INFO] [TRAIN] epoch: 32, iter: 11550/40000, loss: 0.4371, lr: 0.007386, batch_cost: 0.7910, reader_cost: 0.00032, ips: 1.2642 samples/sec | ETA 06:15:03 2021-05-09 16:57:50 [INFO] [TRAIN] epoch: 32, iter: 11560/40000, loss: 0.4889, lr: 0.007383, batch_cost: 0.7847, reader_cost: 0.00015, ips: 1.2743 samples/sec | ETA 06:11:57 2021-05-09 16:57:58 [INFO] [TRAIN] epoch: 32, iter: 11570/40000, loss: 0.4890, lr: 0.007381, batch_cost: 0.7852, reader_cost: 0.00017, ips: 1.2735 samples/sec | ETA 06:12:03 2021-05-09 16:58:05 [INFO] [TRAIN] epoch: 32, iter: 11580/40000, loss: 0.4986, lr: 0.007379, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 06:12:36 2021-05-09 16:58:13 [INFO] [TRAIN] epoch: 32, iter: 11590/40000, loss: 0.3198, lr: 0.007376, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2709 samples/sec | ETA 06:12:34 2021-05-09 16:58:21 [INFO] [TRAIN] epoch: 32, iter: 11600/40000, loss: 0.2152, lr: 0.007374, batch_cost: 0.7851, reader_cost: 0.00016, ips: 1.2738 samples/sec | ETA 06:11:36 2021-05-09 16:58:29 [INFO] [TRAIN] epoch: 32, iter: 11610/40000, loss: 0.1587, lr: 0.007372, batch_cost: 0.7869, reader_cost: 0.00014, ips: 1.2708 samples/sec | ETA 06:12:21 2021-05-09 16:58:37 [INFO] [TRAIN] epoch: 32, iter: 11620/40000, loss: 0.3266, lr: 0.007370, batch_cost: 0.7850, reader_cost: 0.00017, ips: 1.2738 samples/sec | ETA 06:11:18 2021-05-09 16:58:45 [INFO] [TRAIN] epoch: 32, iter: 11630/40000, loss: 0.2497, lr: 0.007367, batch_cost: 0.7866, reader_cost: 0.00018, ips: 1.2712 samples/sec | ETA 06:11:56 2021-05-09 16:58:53 [INFO] [TRAIN] epoch: 32, iter: 11640/40000, loss: 0.4101, lr: 0.007365, batch_cost: 0.7847, reader_cost: 0.00017, ips: 1.2743 samples/sec | ETA 06:10:55 2021-05-09 16:59:00 [INFO] [TRAIN] epoch: 32, iter: 11650/40000, loss: 0.1999, lr: 0.007363, batch_cost: 0.7856, reader_cost: 0.00017, ips: 1.2729 samples/sec | ETA 06:11:11 2021-05-09 16:59:08 [INFO] [TRAIN] epoch: 32, iter: 11660/40000, loss: 0.2385, lr: 0.007360, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2735 samples/sec | ETA 06:10:52 2021-05-09 16:59:16 [INFO] [TRAIN] epoch: 32, iter: 11670/40000, loss: 0.2666, lr: 0.007358, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2736 samples/sec | ETA 06:10:43 2021-05-09 16:59:24 [INFO] [TRAIN] epoch: 32, iter: 11680/40000, loss: 0.2795, lr: 0.007356, batch_cost: 0.7858, reader_cost: 0.00015, ips: 1.2725 samples/sec | ETA 06:10:55 2021-05-09 16:59:32 [INFO] [TRAIN] epoch: 32, iter: 11690/40000, loss: 0.3698, lr: 0.007353, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2725 samples/sec | ETA 06:10:47 2021-05-09 16:59:40 [INFO] [TRAIN] epoch: 32, iter: 11700/40000, loss: 0.3337, lr: 0.007351, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2730 samples/sec | ETA 06:10:30 2021-05-09 16:59:48 [INFO] [TRAIN] epoch: 32, iter: 11710/40000, loss: 0.3193, lr: 0.007349, batch_cost: 0.7851, reader_cost: 0.00017, ips: 1.2736 samples/sec | ETA 06:10:11 2021-05-09 16:59:55 [INFO] [TRAIN] epoch: 32, iter: 11720/40000, loss: 0.2207, lr: 0.007346, batch_cost: 0.7849, reader_cost: 0.00017, ips: 1.2740 samples/sec | ETA 06:09:58 2021-05-09 17:00:03 [INFO] [TRAIN] epoch: 32, iter: 11730/40000, loss: 0.0296, lr: 0.007344, batch_cost: 0.7847, reader_cost: 0.00016, ips: 1.2744 samples/sec | ETA 06:09:43 2021-05-09 17:00:11 [INFO] [TRAIN] epoch: 32, iter: 11740/40000, loss: 0.4396, lr: 0.007342, batch_cost: 0.7843, reader_cost: 0.00015, ips: 1.2750 samples/sec | ETA 06:09:25 2021-05-09 17:00:19 [INFO] [TRAIN] epoch: 32, iter: 11750/40000, loss: 0.2872, lr: 0.007340, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2723 samples/sec | ETA 06:10:03 2021-05-09 17:00:27 [INFO] [TRAIN] epoch: 32, iter: 11760/40000, loss: 0.2450, lr: 0.007337, batch_cost: 0.7841, reader_cost: 0.00016, ips: 1.2753 samples/sec | ETA 06:09:04 2021-05-09 17:00:35 [INFO] [TRAIN] epoch: 32, iter: 11770/40000, loss: 0.2659, lr: 0.007335, batch_cost: 0.7858, reader_cost: 0.00017, ips: 1.2726 samples/sec | ETA 06:09:43 2021-05-09 17:00:43 [INFO] [TRAIN] epoch: 32, iter: 11780/40000, loss: 0.2825, lr: 0.007333, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2732 samples/sec | ETA 06:09:24 2021-05-09 17:00:50 [INFO] [TRAIN] epoch: 32, iter: 11790/40000, loss: 0.3032, lr: 0.007330, batch_cost: 0.7851, reader_cost: 0.00016, ips: 1.2737 samples/sec | ETA 06:09:08 2021-05-09 17:00:58 [INFO] [TRAIN] epoch: 32, iter: 11800/40000, loss: 0.1556, lr: 0.007328, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2723 samples/sec | ETA 06:09:25 2021-05-09 17:01:06 [INFO] [TRAIN] epoch: 32, iter: 11810/40000, loss: 0.3984, lr: 0.007326, batch_cost: 0.7838, reader_cost: 0.00016, ips: 1.2759 samples/sec | ETA 06:08:14 2021-05-09 17:01:14 [INFO] [TRAIN] epoch: 32, iter: 11820/40000, loss: 0.4925, lr: 0.007323, batch_cost: 0.7842, reader_cost: 0.00016, ips: 1.2752 samples/sec | ETA 06:08:18 2021-05-09 17:01:22 [INFO] [TRAIN] epoch: 32, iter: 11830/40000, loss: 0.4407, lr: 0.007321, batch_cost: 0.7841, reader_cost: 0.00016, ips: 1.2753 samples/sec | ETA 06:08:08 2021-05-09 17:01:30 [INFO] [TRAIN] epoch: 32, iter: 11840/40000, loss: 0.4795, lr: 0.007319, batch_cost: 0.7847, reader_cost: 0.00016, ips: 1.2744 samples/sec | ETA 06:08:16 2021-05-09 17:01:37 [INFO] [TRAIN] epoch: 32, iter: 11850/40000, loss: 0.5227, lr: 0.007316, batch_cost: 0.7832, reader_cost: 0.00015, ips: 1.2768 samples/sec | ETA 06:07:27 2021-05-09 17:01:45 [INFO] [TRAIN] epoch: 32, iter: 11860/40000, loss: 0.2585, lr: 0.007314, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2728 samples/sec | ETA 06:08:28 2021-05-09 17:01:53 [INFO] [TRAIN] epoch: 32, iter: 11870/40000, loss: 0.2269, lr: 0.007312, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2734 samples/sec | ETA 06:08:09 2021-05-09 17:02:01 [INFO] [TRAIN] epoch: 32, iter: 11880/40000, loss: 0.2770, lr: 0.007310, batch_cost: 0.7849, reader_cost: 0.00015, ips: 1.2741 samples/sec | ETA 06:07:50 2021-05-09 17:02:09 [INFO] [TRAIN] epoch: 32, iter: 11890/40000, loss: 0.3904, lr: 0.007307, batch_cost: 0.7849, reader_cost: 0.00016, ips: 1.2741 samples/sec | ETA 06:07:42 2021-05-09 17:02:17 [INFO] [TRAIN] epoch: 32, iter: 11900/40000, loss: 0.4221, lr: 0.007305, batch_cost: 0.7857, reader_cost: 0.00011, ips: 1.2727 samples/sec | ETA 06:07:58 2021-05-09 17:02:28 [INFO] [TRAIN] epoch: 33, iter: 11910/40000, loss: 0.3631, lr: 0.007303, batch_cost: 1.1049, reader_cost: 0.31140, ips: 0.9051 samples/sec | ETA 08:37:15 2021-05-09 17:02:36 [INFO] [TRAIN] epoch: 33, iter: 11920/40000, loss: 0.3755, lr: 0.007300, batch_cost: 0.7957, reader_cost: 0.00033, ips: 1.2568 samples/sec | ETA 06:12:23 2021-05-09 17:02:44 [INFO] [TRAIN] epoch: 33, iter: 11930/40000, loss: 0.6324, lr: 0.007298, batch_cost: 0.7846, reader_cost: 0.00017, ips: 1.2745 samples/sec | ETA 06:07:03 2021-05-09 17:02:51 [INFO] [TRAIN] epoch: 33, iter: 11940/40000, loss: 0.4845, lr: 0.007296, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2703 samples/sec | ETA 06:08:08 2021-05-09 17:02:59 [INFO] [TRAIN] epoch: 33, iter: 11950/40000, loss: 0.4484, lr: 0.007293, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2718 samples/sec | ETA 06:07:34 2021-05-09 17:03:07 [INFO] [TRAIN] epoch: 33, iter: 11960/40000, loss: 0.4238, lr: 0.007291, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 06:07:04 2021-05-09 17:03:15 [INFO] [TRAIN] epoch: 33, iter: 11970/40000, loss: 0.4078, lr: 0.007289, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2713 samples/sec | ETA 06:07:28 2021-05-09 17:03:23 [INFO] [TRAIN] epoch: 33, iter: 11980/40000, loss: 0.6412, lr: 0.007286, batch_cost: 0.7871, reader_cost: 0.00015, ips: 1.2704 samples/sec | ETA 06:07:35 2021-05-09 17:03:31 [INFO] [TRAIN] epoch: 33, iter: 11990/40000, loss: 0.3122, lr: 0.007284, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2722 samples/sec | ETA 06:06:56 2021-05-09 17:03:39 [INFO] [TRAIN] epoch: 33, iter: 12000/40000, loss: 0.2419, lr: 0.007282, batch_cost: 0.7868, reader_cost: 0.00017, ips: 1.2710 samples/sec | ETA 06:07:09 2021-05-09 17:03:39 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-09 17:07:10 [INFO] [EVAL] #Images: 500 mIoU: 0.7439 Acc: 0.9527 Kappa: 0.9385 2021-05-09 17:07:10 [INFO] [EVAL] Class IoU: [0.9783 0.8268 0.9123 0.6216 0.5878 0.4238 0.5853 0.6983 0.9103 0.653 0.9351 0.7606 0.5502 0.9345 0.818 0.8582 0.7553 0.6166 0.7076] 2021-05-09 17:07:10 [INFO] [EVAL] Class Acc: [0.9886 0.9117 0.9443 0.8392 0.7522 0.7763 0.7504 0.8867 0.9441 0.8443 0.9549 0.8563 0.6882 0.9642 0.9176 0.9276 0.8588 0.826 0.8127] 2021-05-09 17:07:38 [INFO] [EVAL] The model with the best validation mIoU (0.7461) was saved at iter 11000. 2021-05-09 17:07:46 [INFO] [TRAIN] epoch: 33, iter: 12010/40000, loss: 0.3788, lr: 0.007280, batch_cost: 0.7822, reader_cost: 0.00026, ips: 1.2785 samples/sec | ETA 06:04:52 2021-05-09 17:07:54 [INFO] [TRAIN] epoch: 33, iter: 12020/40000, loss: 0.2497, lr: 0.007277, batch_cost: 0.7845, reader_cost: 0.00017, ips: 1.2747 samples/sec | ETA 06:05:50 2021-05-09 17:08:02 [INFO] [TRAIN] epoch: 33, iter: 12030/40000, loss: 0.4636, lr: 0.007275, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2736 samples/sec | ETA 06:06:02 2021-05-09 17:08:11 [INFO] [TRAIN] epoch: 33, iter: 12040/40000, loss: 0.2874, lr: 0.007273, batch_cost: 0.7845, reader_cost: 0.00016, ips: 1.2747 samples/sec | ETA 06:05:34 2021-05-09 17:08:19 [INFO] [TRAIN] epoch: 33, iter: 12050/40000, loss: 0.3121, lr: 0.007270, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 06:06:09 2021-05-09 17:08:27 [INFO] [TRAIN] epoch: 33, iter: 12060/40000, loss: 0.3363, lr: 0.007268, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2703 samples/sec | ETA 06:06:34 2021-05-09 17:08:35 [INFO] [TRAIN] epoch: 33, iter: 12070/40000, loss: 0.3027, lr: 0.007266, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 06:05:45 2021-05-09 17:08:43 [INFO] [TRAIN] epoch: 33, iter: 12080/40000, loss: 0.3321, lr: 0.007263, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2723 samples/sec | ETA 06:05:44 2021-05-09 17:08:51 [INFO] [TRAIN] epoch: 33, iter: 12090/40000, loss: 0.4037, lr: 0.007261, batch_cost: 0.7874, reader_cost: 0.00016, ips: 1.2700 samples/sec | ETA 06:06:15 2021-05-09 17:08:59 [INFO] [TRAIN] epoch: 33, iter: 12100/40000, loss: 0.2222, lr: 0.007259, batch_cost: 0.7879, reader_cost: 0.00017, ips: 1.2692 samples/sec | ETA 06:06:22 2021-05-09 17:09:06 [INFO] [TRAIN] epoch: 33, iter: 12110/40000, loss: 0.3191, lr: 0.007256, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2708 samples/sec | ETA 06:05:47 2021-05-09 17:09:14 [INFO] [TRAIN] epoch: 33, iter: 12120/40000, loss: 0.3708, lr: 0.007254, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 06:05:22 2021-05-09 17:09:22 [INFO] [TRAIN] epoch: 33, iter: 12130/40000, loss: 0.2888, lr: 0.007252, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2730 samples/sec | ETA 06:04:53 2021-05-09 17:09:30 [INFO] [TRAIN] epoch: 33, iter: 12140/40000, loss: 0.2814, lr: 0.007250, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2703 samples/sec | ETA 06:05:32 2021-05-09 17:09:38 [INFO] [TRAIN] epoch: 33, iter: 12150/40000, loss: 0.1691, lr: 0.007247, batch_cost: 0.7879, reader_cost: 0.00018, ips: 1.2692 samples/sec | ETA 06:05:43 2021-05-09 17:09:46 [INFO] [TRAIN] epoch: 33, iter: 12160/40000, loss: 0.3996, lr: 0.007245, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 06:04:44 2021-05-09 17:09:54 [INFO] [TRAIN] epoch: 33, iter: 12170/40000, loss: 0.2042, lr: 0.007243, batch_cost: 0.7854, reader_cost: 0.00015, ips: 1.2733 samples/sec | ETA 06:04:17 2021-05-09 17:10:01 [INFO] [TRAIN] epoch: 33, iter: 12180/40000, loss: 0.3400, lr: 0.007240, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2715 samples/sec | ETA 06:04:39 2021-05-09 17:10:09 [INFO] [TRAIN] epoch: 33, iter: 12190/40000, loss: 0.4989, lr: 0.007238, batch_cost: 0.7871, reader_cost: 0.00017, ips: 1.2704 samples/sec | ETA 06:04:50 2021-05-09 17:10:17 [INFO] [TRAIN] epoch: 33, iter: 12200/40000, loss: 0.4449, lr: 0.007236, batch_cost: 0.7889, reader_cost: 0.00017, ips: 1.2676 samples/sec | ETA 06:05:31 2021-05-09 17:10:25 [INFO] [TRAIN] epoch: 33, iter: 12210/40000, loss: 0.5801, lr: 0.007233, batch_cost: 0.7852, reader_cost: 0.00017, ips: 1.2736 samples/sec | ETA 06:03:39 2021-05-09 17:10:33 [INFO] [TRAIN] epoch: 33, iter: 12220/40000, loss: 0.3692, lr: 0.007231, batch_cost: 0.7869, reader_cost: 0.00017, ips: 1.2708 samples/sec | ETA 06:04:20 2021-05-09 17:10:41 [INFO] [TRAIN] epoch: 33, iter: 12230/40000, loss: 0.8679, lr: 0.007229, batch_cost: 0.7876, reader_cost: 0.00018, ips: 1.2698 samples/sec | ETA 06:04:30 2021-05-09 17:10:49 [INFO] [TRAIN] epoch: 33, iter: 12240/40000, loss: 0.2049, lr: 0.007226, batch_cost: 0.7868, reader_cost: 0.00017, ips: 1.2709 samples/sec | ETA 06:04:02 2021-05-09 17:10:57 [INFO] [TRAIN] epoch: 33, iter: 12250/40000, loss: 0.5718, lr: 0.007224, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2723 samples/sec | ETA 06:03:31 2021-05-09 17:11:04 [INFO] [TRAIN] epoch: 33, iter: 12260/40000, loss: 0.5561, lr: 0.007222, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2718 samples/sec | ETA 06:03:31 2021-05-09 17:11:12 [INFO] [TRAIN] epoch: 33, iter: 12270/40000, loss: 0.5918, lr: 0.007220, batch_cost: 0.7853, reader_cost: 0.00012, ips: 1.2734 samples/sec | ETA 06:02:56 2021-05-09 17:11:23 [INFO] [TRAIN] epoch: 34, iter: 12280/40000, loss: 0.6078, lr: 0.007217, batch_cost: 1.0875, reader_cost: 0.25374, ips: 0.9195 samples/sec | ETA 08:22:25 2021-05-09 17:11:31 [INFO] [TRAIN] epoch: 34, iter: 12290/40000, loss: 0.2869, lr: 0.007215, batch_cost: 0.7970, reader_cost: 0.00034, ips: 1.2547 samples/sec | ETA 06:08:04 2021-05-09 17:11:39 [INFO] [TRAIN] epoch: 34, iter: 12300/40000, loss: 0.6949, lr: 0.007213, batch_cost: 0.7866, reader_cost: 0.00018, ips: 1.2713 samples/sec | ETA 06:03:08 2021-05-09 17:11:47 [INFO] [TRAIN] epoch: 34, iter: 12310/40000, loss: 0.3253, lr: 0.007210, batch_cost: 0.7881, reader_cost: 0.00018, ips: 1.2688 samples/sec | ETA 06:03:43 2021-05-09 17:11:55 [INFO] [TRAIN] epoch: 34, iter: 12320/40000, loss: 0.4866, lr: 0.007208, batch_cost: 0.7881, reader_cost: 0.00014, ips: 1.2688 samples/sec | ETA 06:03:35 2021-05-09 17:12:03 [INFO] [TRAIN] epoch: 34, iter: 12330/40000, loss: 0.3915, lr: 0.007206, batch_cost: 0.7849, reader_cost: 0.00016, ips: 1.2740 samples/sec | ETA 06:01:58 2021-05-09 17:12:10 [INFO] [TRAIN] epoch: 34, iter: 12340/40000, loss: 0.3765, lr: 0.007203, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2720 samples/sec | ETA 06:02:25 2021-05-09 17:12:18 [INFO] [TRAIN] epoch: 34, iter: 12350/40000, loss: 0.1966, lr: 0.007201, batch_cost: 0.7862, reader_cost: 0.00017, ips: 1.2720 samples/sec | ETA 06:02:17 2021-05-09 17:12:26 [INFO] [TRAIN] epoch: 34, iter: 12360/40000, loss: 0.4375, lr: 0.007199, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2710 samples/sec | ETA 06:02:27 2021-05-09 17:12:34 [INFO] [TRAIN] epoch: 34, iter: 12370/40000, loss: 0.2260, lr: 0.007196, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2727 samples/sec | ETA 06:01:49 2021-05-09 17:12:42 [INFO] [TRAIN] epoch: 34, iter: 12380/40000, loss: 0.2636, lr: 0.007194, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 06:01:52 2021-05-09 17:12:50 [INFO] [TRAIN] epoch: 34, iter: 12390/40000, loss: 0.3870, lr: 0.007192, batch_cost: 0.7845, reader_cost: 0.00015, ips: 1.2748 samples/sec | ETA 06:00:59 2021-05-09 17:12:58 [INFO] [TRAIN] epoch: 34, iter: 12400/40000, loss: 0.1703, lr: 0.007189, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2715 samples/sec | ETA 06:01:46 2021-05-09 17:13:06 [INFO] [TRAIN] epoch: 34, iter: 12410/40000, loss: 0.3130, lr: 0.007187, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2707 samples/sec | ETA 06:01:51 2021-05-09 17:13:13 [INFO] [TRAIN] epoch: 34, iter: 12420/40000, loss: 0.3097, lr: 0.007185, batch_cost: 0.7850, reader_cost: 0.00017, ips: 1.2739 samples/sec | ETA 06:00:50 2021-05-09 17:13:21 [INFO] [TRAIN] epoch: 34, iter: 12430/40000, loss: 0.4788, lr: 0.007183, batch_cost: 0.7851, reader_cost: 0.00016, ips: 1.2737 samples/sec | ETA 06:00:46 2021-05-09 17:13:29 [INFO] [TRAIN] epoch: 34, iter: 12440/40000, loss: 0.3410, lr: 0.007180, batch_cost: 0.7850, reader_cost: 0.00017, ips: 1.2739 samples/sec | ETA 06:00:34 2021-05-09 17:13:37 [INFO] [TRAIN] epoch: 34, iter: 12450/40000, loss: 0.1805, lr: 0.007178, batch_cost: 0.7854, reader_cost: 0.00015, ips: 1.2732 samples/sec | ETA 06:00:39 2021-05-09 17:13:45 [INFO] [TRAIN] epoch: 34, iter: 12460/40000, loss: 0.3754, lr: 0.007176, batch_cost: 0.7883, reader_cost: 0.00015, ips: 1.2686 samples/sec | ETA 06:01:48 2021-05-09 17:13:53 [INFO] [TRAIN] epoch: 34, iter: 12470/40000, loss: 0.2761, lr: 0.007173, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 06:00:35 2021-05-09 17:14:01 [INFO] [TRAIN] epoch: 34, iter: 12480/40000, loss: 0.1606, lr: 0.007171, batch_cost: 0.7871, reader_cost: 0.00015, ips: 1.2705 samples/sec | ETA 06:01:00 2021-05-09 17:14:08 [INFO] [TRAIN] epoch: 34, iter: 12490/40000, loss: 0.3368, lr: 0.007169, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2729 samples/sec | ETA 06:00:11 2021-05-09 17:14:16 [INFO] [TRAIN] epoch: 34, iter: 12500/40000, loss: 0.2073, lr: 0.007166, batch_cost: 0.7850, reader_cost: 0.00016, ips: 1.2739 samples/sec | ETA 05:59:47 2021-05-09 17:14:24 [INFO] [TRAIN] epoch: 34, iter: 12510/40000, loss: 0.2340, lr: 0.007164, batch_cost: 0.7848, reader_cost: 0.00015, ips: 1.2742 samples/sec | ETA 05:59:33 2021-05-09 17:14:32 [INFO] [TRAIN] epoch: 34, iter: 12520/40000, loss: 0.2564, lr: 0.007162, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 05:59:45 2021-05-09 17:14:40 [INFO] [TRAIN] epoch: 34, iter: 12530/40000, loss: 0.2773, lr: 0.007159, batch_cost: 0.7855, reader_cost: 0.00017, ips: 1.2730 samples/sec | ETA 05:59:38 2021-05-09 17:14:48 [INFO] [TRAIN] epoch: 34, iter: 12540/40000, loss: 0.1608, lr: 0.007157, batch_cost: 0.7872, reader_cost: 0.00017, ips: 1.2703 samples/sec | ETA 06:00:17 2021-05-09 17:14:56 [INFO] [TRAIN] epoch: 34, iter: 12550/40000, loss: 0.2951, lr: 0.007155, batch_cost: 0.7875, reader_cost: 0.00015, ips: 1.2699 samples/sec | ETA 06:00:15 2021-05-09 17:15:03 [INFO] [TRAIN] epoch: 34, iter: 12560/40000, loss: 0.5274, lr: 0.007152, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2735 samples/sec | ETA 05:59:06 2021-05-09 17:15:11 [INFO] [TRAIN] epoch: 34, iter: 12570/40000, loss: 0.3168, lr: 0.007150, batch_cost: 0.7837, reader_cost: 0.00016, ips: 1.2761 samples/sec | ETA 05:58:15 2021-05-09 17:15:19 [INFO] [TRAIN] epoch: 34, iter: 12580/40000, loss: 0.6534, lr: 0.007148, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2719 samples/sec | ETA 05:59:17 2021-05-09 17:15:27 [INFO] [TRAIN] epoch: 34, iter: 12590/40000, loss: 0.4080, lr: 0.007146, batch_cost: 0.7867, reader_cost: 0.00015, ips: 1.2711 samples/sec | ETA 05:59:24 2021-05-09 17:15:35 [INFO] [TRAIN] epoch: 34, iter: 12600/40000, loss: 0.3786, lr: 0.007143, batch_cost: 0.7848, reader_cost: 0.00016, ips: 1.2743 samples/sec | ETA 05:58:22 2021-05-09 17:15:43 [INFO] [TRAIN] epoch: 34, iter: 12610/40000, loss: 0.1083, lr: 0.007141, batch_cost: 0.7842, reader_cost: 0.00018, ips: 1.2752 samples/sec | ETA 05:57:59 2021-05-09 17:15:51 [INFO] [TRAIN] epoch: 34, iter: 12620/40000, loss: 0.4010, lr: 0.007139, batch_cost: 0.7859, reader_cost: 0.00017, ips: 1.2724 samples/sec | ETA 05:58:38 2021-05-09 17:15:58 [INFO] [TRAIN] epoch: 34, iter: 12630/40000, loss: 0.5854, lr: 0.007136, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2725 samples/sec | ETA 05:58:29 2021-05-09 17:16:06 [INFO] [TRAIN] epoch: 34, iter: 12640/40000, loss: 0.5546, lr: 0.007134, batch_cost: 0.7869, reader_cost: 0.00014, ips: 1.2708 samples/sec | ETA 05:58:49 2021-05-09 17:16:17 [INFO] [TRAIN] epoch: 35, iter: 12650/40000, loss: 0.8742, lr: 0.007132, batch_cost: 1.0882, reader_cost: 0.22620, ips: 0.9190 samples/sec | ETA 08:16:01 2021-05-09 17:16:25 [INFO] [TRAIN] epoch: 35, iter: 12660/40000, loss: 0.2971, lr: 0.007129, batch_cost: 0.8012, reader_cost: 0.00034, ips: 1.2481 samples/sec | ETA 06:05:04 2021-05-09 17:16:33 [INFO] [TRAIN] epoch: 35, iter: 12670/40000, loss: 0.7779, lr: 0.007127, batch_cost: 0.7834, reader_cost: 0.00016, ips: 1.2765 samples/sec | ETA 05:56:50 2021-05-09 17:16:41 [INFO] [TRAIN] epoch: 35, iter: 12680/40000, loss: 0.3231, lr: 0.007125, batch_cost: 0.7879, reader_cost: 0.00017, ips: 1.2693 samples/sec | ETA 05:58:44 2021-05-09 17:16:49 [INFO] [TRAIN] epoch: 35, iter: 12690/40000, loss: 0.4734, lr: 0.007122, batch_cost: 0.7885, reader_cost: 0.00016, ips: 1.2683 samples/sec | ETA 05:58:53 2021-05-09 17:16:57 [INFO] [TRAIN] epoch: 35, iter: 12700/40000, loss: 0.5171, lr: 0.007120, batch_cost: 0.7876, reader_cost: 0.00015, ips: 1.2697 samples/sec | ETA 05:58:21 2021-05-09 17:17:04 [INFO] [TRAIN] epoch: 35, iter: 12710/40000, loss: 0.3838, lr: 0.007118, batch_cost: 0.7872, reader_cost: 0.00015, ips: 1.2703 samples/sec | ETA 05:58:03 2021-05-09 17:17:12 [INFO] [TRAIN] epoch: 35, iter: 12720/40000, loss: 0.1514, lr: 0.007115, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2728 samples/sec | ETA 05:57:12 2021-05-09 17:17:20 [INFO] [TRAIN] epoch: 35, iter: 12730/40000, loss: 0.3934, lr: 0.007113, batch_cost: 0.7877, reader_cost: 0.00015, ips: 1.2695 samples/sec | ETA 05:58:00 2021-05-09 17:17:28 [INFO] [TRAIN] epoch: 35, iter: 12740/40000, loss: 0.4134, lr: 0.007111, batch_cost: 0.7856, reader_cost: 0.00017, ips: 1.2729 samples/sec | ETA 05:56:55 2021-05-09 17:17:36 [INFO] [TRAIN] epoch: 35, iter: 12750/40000, loss: 0.2280, lr: 0.007109, batch_cost: 0.7862, reader_cost: 0.00017, ips: 1.2720 samples/sec | ETA 05:57:03 2021-05-09 17:17:44 [INFO] [TRAIN] epoch: 35, iter: 12760/40000, loss: 0.2720, lr: 0.007106, batch_cost: 0.7872, reader_cost: 0.00017, ips: 1.2703 samples/sec | ETA 05:57:23 2021-05-09 17:17:52 [INFO] [TRAIN] epoch: 35, iter: 12770/40000, loss: 0.1235, lr: 0.007104, batch_cost: 0.7858, reader_cost: 0.00015, ips: 1.2726 samples/sec | ETA 05:56:36 2021-05-09 17:18:00 [INFO] [TRAIN] epoch: 35, iter: 12780/40000, loss: 0.3006, lr: 0.007102, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 05:56:33 2021-05-09 17:18:07 [INFO] [TRAIN] epoch: 35, iter: 12790/40000, loss: 0.4056, lr: 0.007099, batch_cost: 0.7871, reader_cost: 0.00017, ips: 1.2704 samples/sec | ETA 05:56:58 2021-05-09 17:18:15 [INFO] [TRAIN] epoch: 35, iter: 12800/40000, loss: 0.4077, lr: 0.007097, batch_cost: 0.7854, reader_cost: 0.00017, ips: 1.2733 samples/sec | ETA 05:56:02 2021-05-09 17:18:23 [INFO] [TRAIN] epoch: 35, iter: 12810/40000, loss: 0.2248, lr: 0.007095, batch_cost: 0.7840, reader_cost: 0.00016, ips: 1.2755 samples/sec | ETA 05:55:16 2021-05-09 17:18:31 [INFO] [TRAIN] epoch: 35, iter: 12820/40000, loss: 0.3469, lr: 0.007092, batch_cost: 0.7870, reader_cost: 0.00015, ips: 1.2707 samples/sec | ETA 05:56:29 2021-05-09 17:18:39 [INFO] [TRAIN] epoch: 35, iter: 12830/40000, loss: 0.4431, lr: 0.007090, batch_cost: 0.7847, reader_cost: 0.00015, ips: 1.2744 samples/sec | ETA 05:55:19 2021-05-09 17:18:47 [INFO] [TRAIN] epoch: 35, iter: 12840/40000, loss: 0.2127, lr: 0.007088, batch_cost: 0.7874, reader_cost: 0.00015, ips: 1.2700 samples/sec | ETA 05:56:25 2021-05-09 17:18:55 [INFO] [TRAIN] epoch: 35, iter: 12850/40000, loss: 0.2688, lr: 0.007085, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2724 samples/sec | ETA 05:55:36 2021-05-09 17:19:02 [INFO] [TRAIN] epoch: 35, iter: 12860/40000, loss: 0.4412, lr: 0.007083, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2711 samples/sec | ETA 05:55:52 2021-05-09 17:19:10 [INFO] [TRAIN] epoch: 35, iter: 12870/40000, loss: 0.3251, lr: 0.007081, batch_cost: 0.7861, reader_cost: 0.00014, ips: 1.2722 samples/sec | ETA 05:55:25 2021-05-09 17:19:18 [INFO] [TRAIN] epoch: 35, iter: 12880/40000, loss: 0.3232, lr: 0.007078, batch_cost: 0.7838, reader_cost: 0.00014, ips: 1.2758 samples/sec | ETA 05:54:17 2021-05-09 17:19:26 [INFO] [TRAIN] epoch: 35, iter: 12890/40000, loss: 0.2923, lr: 0.007076, batch_cost: 0.7875, reader_cost: 0.00015, ips: 1.2698 samples/sec | ETA 05:55:50 2021-05-09 17:19:34 [INFO] [TRAIN] epoch: 35, iter: 12900/40000, loss: 0.2745, lr: 0.007074, batch_cost: 0.7877, reader_cost: 0.00015, ips: 1.2695 samples/sec | ETA 05:55:47 2021-05-09 17:19:42 [INFO] [TRAIN] epoch: 35, iter: 12910/40000, loss: 0.5010, lr: 0.007071, batch_cost: 0.7884, reader_cost: 0.00015, ips: 1.2684 samples/sec | ETA 05:55:57 2021-05-09 17:19:50 [INFO] [TRAIN] epoch: 35, iter: 12920/40000, loss: 0.3230, lr: 0.007069, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2720 samples/sec | ETA 05:54:48 2021-05-09 17:19:57 [INFO] [TRAIN] epoch: 35, iter: 12930/40000, loss: 0.4179, lr: 0.007067, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2732 samples/sec | ETA 05:54:21 2021-05-09 17:20:05 [INFO] [TRAIN] epoch: 35, iter: 12940/40000, loss: 0.3333, lr: 0.007065, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 05:54:38 2021-05-09 17:20:13 [INFO] [TRAIN] epoch: 35, iter: 12950/40000, loss: 0.4557, lr: 0.007062, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2734 samples/sec | ETA 05:54:02 2021-05-09 17:20:21 [INFO] [TRAIN] epoch: 35, iter: 12960/40000, loss: 0.5934, lr: 0.007060, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2714 samples/sec | ETA 05:54:27 2021-05-09 17:20:29 [INFO] [TRAIN] epoch: 35, iter: 12970/40000, loss: 0.5867, lr: 0.007058, batch_cost: 0.7861, reader_cost: 0.00017, ips: 1.2721 samples/sec | ETA 05:54:07 2021-05-09 17:20:37 [INFO] [TRAIN] epoch: 35, iter: 12980/40000, loss: 0.1840, lr: 0.007055, batch_cost: 0.7850, reader_cost: 0.00017, ips: 1.2739 samples/sec | ETA 05:53:30 2021-05-09 17:20:45 [INFO] [TRAIN] epoch: 35, iter: 12990/40000, loss: 0.2398, lr: 0.007053, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2723 samples/sec | ETA 05:53:49 2021-05-09 17:20:53 [INFO] [TRAIN] epoch: 35, iter: 13000/40000, loss: 0.2979, lr: 0.007051, batch_cost: 0.7860, reader_cost: 0.00017, ips: 1.2723 samples/sec | ETA 05:53:40 2021-05-09 17:20:53 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-09 17:24:24 [INFO] [EVAL] #Images: 500 mIoU: 0.7277 Acc: 0.9496 Kappa: 0.9346 2021-05-09 17:24:24 [INFO] [EVAL] Class IoU: [0.975 0.8048 0.9043 0.5982 0.5747 0.4349 0.574 0.6739 0.9087 0.6389 0.9335 0.7515 0.5361 0.9327 0.7632 0.8343 0.6814 0.5941 0.7122] 2021-05-09 17:24:24 [INFO] [EVAL] Class Acc: [0.99 0.8686 0.9304 0.8789 0.7952 0.7627 0.8033 0.899 0.9517 0.8715 0.9574 0.8399 0.7564 0.9631 0.9494 0.883 0.8669 0.8223 0.8146] 2021-05-09 17:24:52 [INFO] [EVAL] The model with the best validation mIoU (0.7461) was saved at iter 11000. 2021-05-09 17:25:00 [INFO] [TRAIN] epoch: 35, iter: 13010/40000, loss: 0.3851, lr: 0.007048, batch_cost: 0.7835, reader_cost: 0.00023, ips: 1.2763 samples/sec | ETA 05:52:27 2021-05-09 17:25:07 [INFO] [TRAIN] epoch: 35, iter: 13020/40000, loss: 0.4940, lr: 0.007046, batch_cost: 0.7847, reader_cost: 0.00029, ips: 1.2743 samples/sec | ETA 05:52:52 2021-05-09 17:25:19 [INFO] [TRAIN] epoch: 36, iter: 13030/40000, loss: 0.1831, lr: 0.007044, batch_cost: 1.1166, reader_cost: 0.24565, ips: 0.8956 samples/sec | ETA 08:21:53 2021-05-09 17:25:27 [INFO] [TRAIN] epoch: 36, iter: 13040/40000, loss: 0.5915, lr: 0.007041, batch_cost: 0.7926, reader_cost: 0.00032, ips: 1.2616 samples/sec | ETA 05:56:09 2021-05-09 17:25:34 [INFO] [TRAIN] epoch: 36, iter: 13050/40000, loss: 0.4119, lr: 0.007039, batch_cost: 0.7863, reader_cost: 0.00017, ips: 1.2717 samples/sec | ETA 05:53:11 2021-05-09 17:25:42 [INFO] [TRAIN] epoch: 36, iter: 13060/40000, loss: 0.5048, lr: 0.007037, batch_cost: 0.7868, reader_cost: 0.00017, ips: 1.2710 samples/sec | ETA 05:53:15 2021-05-09 17:25:50 [INFO] [TRAIN] epoch: 36, iter: 13070/40000, loss: 0.5564, lr: 0.007034, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2730 samples/sec | ETA 05:52:34 2021-05-09 17:25:58 [INFO] [TRAIN] epoch: 36, iter: 13080/40000, loss: 0.4234, lr: 0.007032, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2715 samples/sec | ETA 05:52:51 2021-05-09 17:26:06 [INFO] [TRAIN] epoch: 36, iter: 13090/40000, loss: 0.1837, lr: 0.007030, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2710 samples/sec | ETA 05:52:51 2021-05-09 17:26:14 [INFO] [TRAIN] epoch: 36, iter: 13100/40000, loss: 0.3181, lr: 0.007027, batch_cost: 0.7865, reader_cost: 0.00017, ips: 1.2715 samples/sec | ETA 05:52:36 2021-05-09 17:26:22 [INFO] [TRAIN] epoch: 36, iter: 13110/40000, loss: 0.4000, lr: 0.007025, batch_cost: 0.7869, reader_cost: 0.00015, ips: 1.2708 samples/sec | ETA 05:52:40 2021-05-09 17:26:29 [INFO] [TRAIN] epoch: 36, iter: 13120/40000, loss: 0.2642, lr: 0.007023, batch_cost: 0.7858, reader_cost: 0.00015, ips: 1.2725 samples/sec | ETA 05:52:03 2021-05-09 17:26:37 [INFO] [TRAIN] epoch: 36, iter: 13130/40000, loss: 0.3761, lr: 0.007020, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2705 samples/sec | ETA 05:52:28 2021-05-09 17:26:45 [INFO] [TRAIN] epoch: 36, iter: 13140/40000, loss: 0.1366, lr: 0.007018, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 05:52:00 2021-05-09 17:26:53 [INFO] [TRAIN] epoch: 36, iter: 13150/40000, loss: 0.2920, lr: 0.007016, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2710 samples/sec | ETA 05:52:05 2021-05-09 17:27:01 [INFO] [TRAIN] epoch: 36, iter: 13160/40000, loss: 0.4299, lr: 0.007014, batch_cost: 0.7844, reader_cost: 0.00018, ips: 1.2748 samples/sec | ETA 05:50:53 2021-05-09 17:27:09 [INFO] [TRAIN] epoch: 36, iter: 13170/40000, loss: 0.3818, lr: 0.007011, batch_cost: 0.7863, reader_cost: 0.00017, ips: 1.2718 samples/sec | ETA 05:51:35 2021-05-09 17:27:17 [INFO] [TRAIN] epoch: 36, iter: 13180/40000, loss: 0.3199, lr: 0.007009, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2703 samples/sec | ETA 05:51:53 2021-05-09 17:27:25 [INFO] [TRAIN] epoch: 36, iter: 13190/40000, loss: 0.2016, lr: 0.007007, batch_cost: 0.7874, reader_cost: 0.00016, ips: 1.2700 samples/sec | ETA 05:51:49 2021-05-09 17:27:32 [INFO] [TRAIN] epoch: 36, iter: 13200/40000, loss: 0.2418, lr: 0.007004, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2709 samples/sec | ETA 05:51:27 2021-05-09 17:27:40 [INFO] [TRAIN] epoch: 36, iter: 13210/40000, loss: 0.2010, lr: 0.007002, batch_cost: 0.7846, reader_cost: 0.00018, ips: 1.2745 samples/sec | ETA 05:50:19 2021-05-09 17:27:48 [INFO] [TRAIN] epoch: 36, iter: 13220/40000, loss: 0.1916, lr: 0.007000, batch_cost: 0.7857, reader_cost: 0.00017, ips: 1.2728 samples/sec | ETA 05:50:40 2021-05-09 17:27:56 [INFO] [TRAIN] epoch: 36, iter: 13230/40000, loss: 0.3962, lr: 0.006997, batch_cost: 0.7859, reader_cost: 0.00018, ips: 1.2724 samples/sec | ETA 05:50:39 2021-05-09 17:28:04 [INFO] [TRAIN] epoch: 36, iter: 13240/40000, loss: 0.2634, lr: 0.006995, batch_cost: 0.7874, reader_cost: 0.00017, ips: 1.2700 samples/sec | ETA 05:51:11 2021-05-09 17:28:12 [INFO] [TRAIN] epoch: 36, iter: 13250/40000, loss: 0.2513, lr: 0.006993, batch_cost: 0.7862, reader_cost: 0.00017, ips: 1.2719 samples/sec | ETA 05:50:31 2021-05-09 17:28:20 [INFO] [TRAIN] epoch: 36, iter: 13260/40000, loss: 0.3580, lr: 0.006990, batch_cost: 0.7853, reader_cost: 0.00017, ips: 1.2734 samples/sec | ETA 05:49:59 2021-05-09 17:28:27 [INFO] [TRAIN] epoch: 36, iter: 13270/40000, loss: 0.3594, lr: 0.006988, batch_cost: 0.7861, reader_cost: 0.00017, ips: 1.2722 samples/sec | ETA 05:50:11 2021-05-09 17:28:35 [INFO] [TRAIN] epoch: 36, iter: 13280/40000, loss: 0.2562, lr: 0.006986, batch_cost: 0.7860, reader_cost: 0.00017, ips: 1.2723 samples/sec | ETA 05:50:00 2021-05-09 17:28:43 [INFO] [TRAIN] epoch: 36, iter: 13290/40000, loss: 0.3175, lr: 0.006983, batch_cost: 0.7854, reader_cost: 0.00019, ips: 1.2733 samples/sec | ETA 05:49:37 2021-05-09 17:28:51 [INFO] [TRAIN] epoch: 36, iter: 13300/40000, loss: 0.5614, lr: 0.006981, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2726 samples/sec | ETA 05:49:40 2021-05-09 17:28:59 [INFO] [TRAIN] epoch: 36, iter: 13310/40000, loss: 0.4166, lr: 0.006979, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2710 samples/sec | ETA 05:49:58 2021-05-09 17:29:07 [INFO] [TRAIN] epoch: 36, iter: 13320/40000, loss: 0.4464, lr: 0.006976, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 05:49:16 2021-05-09 17:29:15 [INFO] [TRAIN] epoch: 36, iter: 13330/40000, loss: 0.5821, lr: 0.006974, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2716 samples/sec | ETA 05:49:33 2021-05-09 17:29:22 [INFO] [TRAIN] epoch: 36, iter: 13340/40000, loss: 0.4536, lr: 0.006972, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 05:49:08 2021-05-09 17:29:30 [INFO] [TRAIN] epoch: 36, iter: 13350/40000, loss: 0.1654, lr: 0.006969, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2730 samples/sec | ETA 05:48:55 2021-05-09 17:29:38 [INFO] [TRAIN] epoch: 36, iter: 13360/40000, loss: 0.3090, lr: 0.006967, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2728 samples/sec | ETA 05:48:49 2021-05-09 17:29:46 [INFO] [TRAIN] epoch: 36, iter: 13370/40000, loss: 0.3131, lr: 0.006965, batch_cost: 0.7876, reader_cost: 0.00015, ips: 1.2696 samples/sec | ETA 05:49:34 2021-05-09 17:29:54 [INFO] [TRAIN] epoch: 36, iter: 13380/40000, loss: 0.4677, lr: 0.006963, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2704 samples/sec | ETA 05:49:14 2021-05-09 17:30:02 [INFO] [TRAIN] epoch: 36, iter: 13390/40000, loss: 0.3523, lr: 0.006960, batch_cost: 0.7852, reader_cost: 0.00025, ips: 1.2736 samples/sec | ETA 05:48:14 2021-05-09 17:30:13 [INFO] [TRAIN] epoch: 37, iter: 13400/40000, loss: 0.3150, lr: 0.006958, batch_cost: 1.0878, reader_cost: 0.24098, ips: 0.9193 samples/sec | ETA 08:02:14 2021-05-09 17:30:21 [INFO] [TRAIN] epoch: 37, iter: 13410/40000, loss: 0.4504, lr: 0.006956, batch_cost: 0.7903, reader_cost: 0.00032, ips: 1.2653 samples/sec | ETA 05:50:14 2021-05-09 17:30:28 [INFO] [TRAIN] epoch: 37, iter: 13420/40000, loss: 0.4164, lr: 0.006953, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2735 samples/sec | ETA 05:47:51 2021-05-09 17:30:36 [INFO] [TRAIN] epoch: 37, iter: 13430/40000, loss: 0.5900, lr: 0.006951, batch_cost: 0.7857, reader_cost: 0.00017, ips: 1.2728 samples/sec | ETA 05:47:55 2021-05-09 17:30:44 [INFO] [TRAIN] epoch: 37, iter: 13440/40000, loss: 0.5144, lr: 0.006949, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2735 samples/sec | ETA 05:47:35 2021-05-09 17:30:52 [INFO] [TRAIN] epoch: 37, iter: 13450/40000, loss: 0.4190, lr: 0.006946, batch_cost: 0.7849, reader_cost: 0.00014, ips: 1.2740 samples/sec | ETA 05:47:20 2021-05-09 17:31:00 [INFO] [TRAIN] epoch: 37, iter: 13460/40000, loss: 0.3044, lr: 0.006944, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 05:47:58 2021-05-09 17:31:08 [INFO] [TRAIN] epoch: 37, iter: 13470/40000, loss: 0.2238, lr: 0.006942, batch_cost: 0.7864, reader_cost: 0.00018, ips: 1.2716 samples/sec | ETA 05:47:42 2021-05-09 17:31:16 [INFO] [TRAIN] epoch: 37, iter: 13480/40000, loss: 0.3155, lr: 0.006939, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 05:47:28 2021-05-09 17:31:23 [INFO] [TRAIN] epoch: 37, iter: 13490/40000, loss: 0.2221, lr: 0.006937, batch_cost: 0.7873, reader_cost: 0.00015, ips: 1.2702 samples/sec | ETA 05:47:50 2021-05-09 17:31:31 [INFO] [TRAIN] epoch: 37, iter: 13500/40000, loss: 0.3100, lr: 0.006935, batch_cost: 0.7871, reader_cost: 0.00015, ips: 1.2705 samples/sec | ETA 05:47:37 2021-05-09 17:31:39 [INFO] [TRAIN] epoch: 37, iter: 13510/40000, loss: 0.1700, lr: 0.006932, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2734 samples/sec | ETA 05:46:43 2021-05-09 17:31:47 [INFO] [TRAIN] epoch: 37, iter: 13520/40000, loss: 0.2551, lr: 0.006930, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2713 samples/sec | ETA 05:47:08 2021-05-09 17:31:55 [INFO] [TRAIN] epoch: 37, iter: 13530/40000, loss: 0.3144, lr: 0.006928, batch_cost: 0.7869, reader_cost: 0.00017, ips: 1.2708 samples/sec | ETA 05:47:09 2021-05-09 17:32:03 [INFO] [TRAIN] epoch: 37, iter: 13540/40000, loss: 0.4594, lr: 0.006925, batch_cost: 0.7869, reader_cost: 0.00018, ips: 1.2708 samples/sec | ETA 05:47:02 2021-05-09 17:32:11 [INFO] [TRAIN] epoch: 37, iter: 13550/40000, loss: 0.3682, lr: 0.006923, batch_cost: 0.7856, reader_cost: 0.00017, ips: 1.2729 samples/sec | ETA 05:46:20 2021-05-09 17:32:18 [INFO] [TRAIN] epoch: 37, iter: 13560/40000, loss: 0.2524, lr: 0.006921, batch_cost: 0.7843, reader_cost: 0.00015, ips: 1.2750 samples/sec | ETA 05:45:37 2021-05-09 17:32:26 [INFO] [TRAIN] epoch: 37, iter: 13570/40000, loss: 0.4058, lr: 0.006918, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2720 samples/sec | ETA 05:46:18 2021-05-09 17:32:34 [INFO] [TRAIN] epoch: 37, iter: 13580/40000, loss: 0.3408, lr: 0.006916, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2723 samples/sec | ETA 05:46:05 2021-05-09 17:32:42 [INFO] [TRAIN] epoch: 37, iter: 13590/40000, loss: 0.1404, lr: 0.006914, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2736 samples/sec | ETA 05:45:37 2021-05-09 17:32:50 [INFO] [TRAIN] epoch: 37, iter: 13600/40000, loss: 0.3341, lr: 0.006911, batch_cost: 0.7854, reader_cost: 0.00015, ips: 1.2733 samples/sec | ETA 05:45:33 2021-05-09 17:32:58 [INFO] [TRAIN] epoch: 37, iter: 13610/40000, loss: 0.4627, lr: 0.006909, batch_cost: 0.7856, reader_cost: 0.00017, ips: 1.2729 samples/sec | ETA 05:45:32 2021-05-09 17:33:06 [INFO] [TRAIN] epoch: 37, iter: 13620/40000, loss: 0.3060, lr: 0.006907, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2735 samples/sec | ETA 05:45:14 2021-05-09 17:33:13 [INFO] [TRAIN] epoch: 37, iter: 13630/40000, loss: 0.2785, lr: 0.006904, batch_cost: 0.7847, reader_cost: 0.00016, ips: 1.2744 samples/sec | ETA 05:44:51 2021-05-09 17:33:21 [INFO] [TRAIN] epoch: 37, iter: 13640/40000, loss: 0.2156, lr: 0.006902, batch_cost: 0.7842, reader_cost: 0.00016, ips: 1.2751 samples/sec | ETA 05:44:32 2021-05-09 17:33:29 [INFO] [TRAIN] epoch: 37, iter: 13650/40000, loss: 0.4094, lr: 0.006900, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2721 samples/sec | ETA 05:45:13 2021-05-09 17:33:37 [INFO] [TRAIN] epoch: 37, iter: 13660/40000, loss: 0.2420, lr: 0.006898, batch_cost: 0.7865, reader_cost: 0.00018, ips: 1.2714 samples/sec | ETA 05:45:16 2021-05-09 17:33:45 [INFO] [TRAIN] epoch: 37, iter: 13670/40000, loss: 0.3310, lr: 0.006895, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2718 samples/sec | ETA 05:45:02 2021-05-09 17:33:53 [INFO] [TRAIN] epoch: 37, iter: 13680/40000, loss: 0.4274, lr: 0.006893, batch_cost: 0.7851, reader_cost: 0.00015, ips: 1.2737 samples/sec | ETA 05:44:23 2021-05-09 17:34:01 [INFO] [TRAIN] epoch: 37, iter: 13690/40000, loss: 0.3446, lr: 0.006891, batch_cost: 0.7871, reader_cost: 0.00015, ips: 1.2705 samples/sec | ETA 05:45:07 2021-05-09 17:34:08 [INFO] [TRAIN] epoch: 37, iter: 13700/40000, loss: 0.4806, lr: 0.006888, batch_cost: 0.7846, reader_cost: 0.00015, ips: 1.2745 samples/sec | ETA 05:43:55 2021-05-09 17:34:16 [INFO] [TRAIN] epoch: 37, iter: 13710/40000, loss: 0.2595, lr: 0.006886, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2715 samples/sec | ETA 05:44:36 2021-05-09 17:34:24 [INFO] [TRAIN] epoch: 37, iter: 13720/40000, loss: 0.1456, lr: 0.006884, batch_cost: 0.7874, reader_cost: 0.00015, ips: 1.2699 samples/sec | ETA 05:44:53 2021-05-09 17:34:32 [INFO] [TRAIN] epoch: 37, iter: 13730/40000, loss: 0.2565, lr: 0.006881, batch_cost: 0.7849, reader_cost: 0.00015, ips: 1.2740 samples/sec | ETA 05:43:40 2021-05-09 17:34:40 [INFO] [TRAIN] epoch: 37, iter: 13740/40000, loss: 0.2156, lr: 0.006879, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2718 samples/sec | ETA 05:44:08 2021-05-09 17:34:48 [INFO] [TRAIN] epoch: 37, iter: 13750/40000, loss: 0.3276, lr: 0.006877, batch_cost: 0.7869, reader_cost: 0.00015, ips: 1.2707 samples/sec | ETA 05:44:17 2021-05-09 17:34:56 [INFO] [TRAIN] epoch: 37, iter: 13760/40000, loss: 0.3748, lr: 0.006874, batch_cost: 0.7851, reader_cost: 0.00011, ips: 1.2737 samples/sec | ETA 05:43:20 2021-05-09 17:35:07 [INFO] [TRAIN] epoch: 38, iter: 13770/40000, loss: 0.3836, lr: 0.006872, batch_cost: 1.1059, reader_cost: 0.27015, ips: 0.9043 samples/sec | ETA 08:03:26 2021-05-09 17:35:15 [INFO] [TRAIN] epoch: 38, iter: 13780/40000, loss: 0.4975, lr: 0.006870, batch_cost: 0.7981, reader_cost: 0.00034, ips: 1.2530 samples/sec | ETA 05:48:45 2021-05-09 17:35:22 [INFO] [TRAIN] epoch: 38, iter: 13790/40000, loss: 0.4101, lr: 0.006867, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2715 samples/sec | ETA 05:43:33 2021-05-09 17:35:30 [INFO] [TRAIN] epoch: 38, iter: 13800/40000, loss: 0.3855, lr: 0.006865, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2708 samples/sec | ETA 05:43:36 2021-05-09 17:35:38 [INFO] [TRAIN] epoch: 38, iter: 13810/40000, loss: 0.3988, lr: 0.006863, batch_cost: 0.7848, reader_cost: 0.00016, ips: 1.2741 samples/sec | ETA 05:42:34 2021-05-09 17:35:46 [INFO] [TRAIN] epoch: 38, iter: 13820/40000, loss: 0.4471, lr: 0.006860, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2718 samples/sec | ETA 05:43:05 2021-05-09 17:35:54 [INFO] [TRAIN] epoch: 38, iter: 13830/40000, loss: 0.3839, lr: 0.006858, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2734 samples/sec | ETA 05:42:32 2021-05-09 17:36:02 [INFO] [TRAIN] epoch: 38, iter: 13840/40000, loss: 0.1909, lr: 0.006856, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 05:42:59 2021-05-09 17:36:10 [INFO] [TRAIN] epoch: 38, iter: 13850/40000, loss: 0.3226, lr: 0.006853, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2729 samples/sec | ETA 05:42:23 2021-05-09 17:36:18 [INFO] [TRAIN] epoch: 38, iter: 13860/40000, loss: 0.3253, lr: 0.006851, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2718 samples/sec | ETA 05:42:33 2021-05-09 17:36:25 [INFO] [TRAIN] epoch: 38, iter: 13870/40000, loss: 0.2846, lr: 0.006849, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2715 samples/sec | ETA 05:42:31 2021-05-09 17:36:33 [INFO] [TRAIN] epoch: 38, iter: 13880/40000, loss: 0.1968, lr: 0.006846, batch_cost: 0.7868, reader_cost: 0.00017, ips: 1.2709 samples/sec | ETA 05:42:31 2021-05-09 17:36:41 [INFO] [TRAIN] epoch: 38, iter: 13890/40000, loss: 0.1912, lr: 0.006844, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2729 samples/sec | ETA 05:41:52 2021-05-09 17:36:49 [INFO] [TRAIN] epoch: 38, iter: 13900/40000, loss: 0.3604, lr: 0.006842, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2718 samples/sec | ETA 05:42:02 2021-05-09 17:36:57 [INFO] [TRAIN] epoch: 38, iter: 13910/40000, loss: 0.3915, lr: 0.006839, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2715 samples/sec | ETA 05:41:58 2021-05-09 17:37:05 [INFO] [TRAIN] epoch: 38, iter: 13920/40000, loss: 0.2646, lr: 0.006837, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 05:41:39 2021-05-09 17:37:13 [INFO] [TRAIN] epoch: 38, iter: 13930/40000, loss: 0.3689, lr: 0.006835, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2713 samples/sec | ETA 05:41:47 2021-05-09 17:37:20 [INFO] [TRAIN] epoch: 38, iter: 13940/40000, loss: 0.2272, lr: 0.006832, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2708 samples/sec | ETA 05:41:47 2021-05-09 17:37:28 [INFO] [TRAIN] epoch: 38, iter: 13950/40000, loss: 0.2575, lr: 0.006830, batch_cost: 0.7867, reader_cost: 0.00017, ips: 1.2711 samples/sec | ETA 05:41:33 2021-05-09 17:37:36 [INFO] [TRAIN] epoch: 38, iter: 13960/40000, loss: 0.0820, lr: 0.006828, batch_cost: 0.7871, reader_cost: 0.00019, ips: 1.2705 samples/sec | ETA 05:41:35 2021-05-09 17:37:44 [INFO] [TRAIN] epoch: 38, iter: 13970/40000, loss: 0.2346, lr: 0.006825, batch_cost: 0.7878, reader_cost: 0.00019, ips: 1.2694 samples/sec | ETA 05:41:45 2021-05-09 17:37:52 [INFO] [TRAIN] epoch: 38, iter: 13980/40000, loss: 0.3651, lr: 0.006823, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2728 samples/sec | ETA 05:40:43 2021-05-09 17:38:00 [INFO] [TRAIN] epoch: 38, iter: 13990/40000, loss: 0.2494, lr: 0.006821, batch_cost: 0.7842, reader_cost: 0.00016, ips: 1.2752 samples/sec | ETA 05:39:56 2021-05-09 17:38:08 [INFO] [TRAIN] epoch: 38, iter: 14000/40000, loss: 0.3820, lr: 0.006818, batch_cost: 0.7871, reader_cost: 0.00017, ips: 1.2704 samples/sec | ETA 05:41:05 2021-05-09 17:38:08 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-09 17:41:39 [INFO] [EVAL] #Images: 500 mIoU: 0.7446 Acc: 0.9527 Kappa: 0.9386 2021-05-09 17:41:39 [INFO] [EVAL] Class IoU: [0.9763 0.8192 0.9128 0.6301 0.6134 0.4656 0.589 0.6828 0.9124 0.651 0.9378 0.7646 0.5639 0.9349 0.8158 0.854 0.6915 0.6195 0.7129] 2021-05-09 17:41:39 [INFO] [EVAL] Class Acc: [0.9935 0.8714 0.94 0.8363 0.7977 0.7518 0.8031 0.9098 0.9531 0.844 0.9672 0.8658 0.7448 0.9634 0.8779 0.8987 0.8308 0.8358 0.7844] 2021-05-09 17:42:07 [INFO] [EVAL] The model with the best validation mIoU (0.7461) was saved at iter 11000. 2021-05-09 17:42:15 [INFO] [TRAIN] epoch: 38, iter: 14010/40000, loss: 0.2802, lr: 0.006816, batch_cost: 0.7845, reader_cost: 0.00045, ips: 1.2748 samples/sec | ETA 05:39:47 2021-05-09 17:42:23 [INFO] [TRAIN] epoch: 38, iter: 14020/40000, loss: 0.3744, lr: 0.006814, batch_cost: 0.7845, reader_cost: 0.00016, ips: 1.2747 samples/sec | ETA 05:39:41 2021-05-09 17:42:31 [INFO] [TRAIN] epoch: 38, iter: 14030/40000, loss: 0.2071, lr: 0.006812, batch_cost: 0.7843, reader_cost: 0.00017, ips: 1.2750 samples/sec | ETA 05:39:28 2021-05-09 17:42:41 [INFO] [TRAIN] epoch: 38, iter: 14040/40000, loss: 0.3450, lr: 0.006809, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 05:39:51 2021-05-09 17:42:48 [INFO] [TRAIN] epoch: 38, iter: 14050/40000, loss: 0.4995, lr: 0.006807, batch_cost: 0.7846, reader_cost: 0.00015, ips: 1.2745 samples/sec | ETA 05:39:21 2021-05-09 17:42:56 [INFO] [TRAIN] epoch: 38, iter: 14060/40000, loss: 0.3084, lr: 0.006805, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2719 samples/sec | ETA 05:39:54 2021-05-09 17:43:04 [INFO] [TRAIN] epoch: 38, iter: 14070/40000, loss: 0.5659, lr: 0.006802, batch_cost: 0.7851, reader_cost: 0.00016, ips: 1.2737 samples/sec | ETA 05:39:18 2021-05-09 17:43:12 [INFO] [TRAIN] epoch: 38, iter: 14080/40000, loss: 0.5634, lr: 0.006800, batch_cost: 0.7873, reader_cost: 0.00015, ips: 1.2702 samples/sec | ETA 05:40:06 2021-05-09 17:43:20 [INFO] [TRAIN] epoch: 38, iter: 14090/40000, loss: 0.2288, lr: 0.006798, batch_cost: 0.7845, reader_cost: 0.00016, ips: 1.2747 samples/sec | ETA 05:38:45 2021-05-09 17:43:28 [INFO] [TRAIN] epoch: 38, iter: 14100/40000, loss: 0.1950, lr: 0.006795, batch_cost: 0.7863, reader_cost: 0.00017, ips: 1.2718 samples/sec | ETA 05:39:24 2021-05-09 17:43:36 [INFO] [TRAIN] epoch: 38, iter: 14110/40000, loss: 0.2966, lr: 0.006793, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2707 samples/sec | ETA 05:39:35 2021-05-09 17:43:43 [INFO] [TRAIN] epoch: 38, iter: 14120/40000, loss: 0.4084, lr: 0.006791, batch_cost: 0.7870, reader_cost: 0.00015, ips: 1.2707 samples/sec | ETA 05:39:27 2021-05-09 17:43:51 [INFO] [TRAIN] epoch: 38, iter: 14130/40000, loss: 0.4237, lr: 0.006788, batch_cost: 0.7855, reader_cost: 0.00012, ips: 1.2731 samples/sec | ETA 05:38:40 2021-05-09 17:44:02 [INFO] [TRAIN] epoch: 39, iter: 14140/40000, loss: 0.4071, lr: 0.006786, batch_cost: 1.1085, reader_cost: 0.26143, ips: 0.9021 samples/sec | ETA 07:57:45 2021-05-09 17:44:10 [INFO] [TRAIN] epoch: 39, iter: 14150/40000, loss: 0.4803, lr: 0.006784, batch_cost: 0.7974, reader_cost: 0.00033, ips: 1.2541 samples/sec | ETA 05:43:32 2021-05-09 17:44:18 [INFO] [TRAIN] epoch: 39, iter: 14160/40000, loss: 0.6060, lr: 0.006781, batch_cost: 0.7877, reader_cost: 0.00016, ips: 1.2695 samples/sec | ETA 05:39:14 2021-05-09 17:44:26 [INFO] [TRAIN] epoch: 39, iter: 14170/40000, loss: 0.2351, lr: 0.006779, batch_cost: 0.7881, reader_cost: 0.00015, ips: 1.2689 samples/sec | ETA 05:39:16 2021-05-09 17:44:34 [INFO] [TRAIN] epoch: 39, iter: 14180/40000, loss: 0.4616, lr: 0.006777, batch_cost: 0.7867, reader_cost: 0.00015, ips: 1.2711 samples/sec | ETA 05:38:32 2021-05-09 17:44:42 [INFO] [TRAIN] epoch: 39, iter: 14190/40000, loss: 0.4528, lr: 0.006774, batch_cost: 0.7869, reader_cost: 0.00015, ips: 1.2709 samples/sec | ETA 05:38:28 2021-05-09 17:44:50 [INFO] [TRAIN] epoch: 39, iter: 14200/40000, loss: 0.3713, lr: 0.006772, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2710 samples/sec | ETA 05:38:18 2021-05-09 17:44:58 [INFO] [TRAIN] epoch: 39, iter: 14210/40000, loss: 0.2954, lr: 0.006770, batch_cost: 0.7854, reader_cost: 0.00017, ips: 1.2732 samples/sec | ETA 05:37:36 2021-05-09 17:45:05 [INFO] [TRAIN] epoch: 39, iter: 14220/40000, loss: 0.3121, lr: 0.006767, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2713 samples/sec | ETA 05:37:58 2021-05-09 17:45:13 [INFO] [TRAIN] epoch: 39, iter: 14230/40000, loss: 0.4337, lr: 0.006765, batch_cost: 0.7872, reader_cost: 0.00015, ips: 1.2703 samples/sec | ETA 05:38:06 2021-05-09 17:45:21 [INFO] [TRAIN] epoch: 39, iter: 14240/40000, loss: 0.3148, lr: 0.006763, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2716 samples/sec | ETA 05:37:38 2021-05-09 17:45:29 [INFO] [TRAIN] epoch: 39, iter: 14250/40000, loss: 0.2893, lr: 0.006760, batch_cost: 0.7871, reader_cost: 0.00015, ips: 1.2704 samples/sec | ETA 05:37:49 2021-05-09 17:45:37 [INFO] [TRAIN] epoch: 39, iter: 14260/40000, loss: 0.1460, lr: 0.006758, batch_cost: 0.7871, reader_cost: 0.00015, ips: 1.2705 samples/sec | ETA 05:37:40 2021-05-09 17:45:45 [INFO] [TRAIN] epoch: 39, iter: 14270/40000, loss: 0.3771, lr: 0.006756, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2703 samples/sec | ETA 05:37:35 2021-05-09 17:45:53 [INFO] [TRAIN] epoch: 39, iter: 14280/40000, loss: 0.3661, lr: 0.006753, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2714 samples/sec | ETA 05:37:09 2021-05-09 17:46:01 [INFO] [TRAIN] epoch: 39, iter: 14290/40000, loss: 0.2596, lr: 0.006751, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2730 samples/sec | ETA 05:36:35 2021-05-09 17:46:08 [INFO] [TRAIN] epoch: 39, iter: 14300/40000, loss: 0.2957, lr: 0.006749, batch_cost: 0.7851, reader_cost: 0.00016, ips: 1.2737 samples/sec | ETA 05:36:17 2021-05-09 17:46:16 [INFO] [TRAIN] epoch: 39, iter: 14310/40000, loss: 0.2138, lr: 0.006746, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2707 samples/sec | ETA 05:36:57 2021-05-09 17:46:24 [INFO] [TRAIN] epoch: 39, iter: 14320/40000, loss: 0.2727, lr: 0.006744, batch_cost: 0.7847, reader_cost: 0.00016, ips: 1.2744 samples/sec | ETA 05:35:50 2021-05-09 17:46:32 [INFO] [TRAIN] epoch: 39, iter: 14330/40000, loss: 0.1431, lr: 0.006742, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2703 samples/sec | ETA 05:36:47 2021-05-09 17:46:40 [INFO] [TRAIN] epoch: 39, iter: 14340/40000, loss: 0.2464, lr: 0.006739, batch_cost: 0.7855, reader_cost: 0.00017, ips: 1.2731 samples/sec | ETA 05:35:56 2021-05-09 17:46:48 [INFO] [TRAIN] epoch: 39, iter: 14350/40000, loss: 0.3332, lr: 0.006737, batch_cost: 0.7842, reader_cost: 0.00015, ips: 1.2752 samples/sec | ETA 05:35:14 2021-05-09 17:46:56 [INFO] [TRAIN] epoch: 39, iter: 14360/40000, loss: 0.2932, lr: 0.006735, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2709 samples/sec | ETA 05:36:14 2021-05-09 17:47:03 [INFO] [TRAIN] epoch: 39, iter: 14370/40000, loss: 0.2717, lr: 0.006732, batch_cost: 0.7849, reader_cost: 0.00015, ips: 1.2740 samples/sec | ETA 05:35:17 2021-05-09 17:47:11 [INFO] [TRAIN] epoch: 39, iter: 14380/40000, loss: 0.2231, lr: 0.006730, batch_cost: 0.7845, reader_cost: 0.00017, ips: 1.2746 samples/sec | ETA 05:35:00 2021-05-09 17:47:19 [INFO] [TRAIN] epoch: 39, iter: 14390/40000, loss: 0.3103, lr: 0.006728, batch_cost: 0.7847, reader_cost: 0.00015, ips: 1.2743 samples/sec | ETA 05:34:56 2021-05-09 17:47:27 [INFO] [TRAIN] epoch: 39, iter: 14400/40000, loss: 0.1507, lr: 0.006725, batch_cost: 0.7854, reader_cost: 0.00014, ips: 1.2732 samples/sec | ETA 05:35:06 2021-05-09 17:47:35 [INFO] [TRAIN] epoch: 39, iter: 14410/40000, loss: 0.3589, lr: 0.006723, batch_cost: 0.7858, reader_cost: 0.00019, ips: 1.2726 samples/sec | ETA 05:35:09 2021-05-09 17:47:43 [INFO] [TRAIN] epoch: 39, iter: 14420/40000, loss: 0.5260, lr: 0.006721, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2720 samples/sec | ETA 05:35:10 2021-05-09 17:47:51 [INFO] [TRAIN] epoch: 39, iter: 14430/40000, loss: 0.3521, lr: 0.006718, batch_cost: 0.7853, reader_cost: 0.00017, ips: 1.2733 samples/sec | ETA 05:34:41 2021-05-09 17:47:58 [INFO] [TRAIN] epoch: 39, iter: 14440/40000, loss: 0.5207, lr: 0.006716, batch_cost: 0.7835, reader_cost: 0.00018, ips: 1.2763 samples/sec | ETA 05:33:47 2021-05-09 17:48:06 [INFO] [TRAIN] epoch: 39, iter: 14450/40000, loss: 0.5151, lr: 0.006714, batch_cost: 0.7856, reader_cost: 0.00017, ips: 1.2730 samples/sec | ETA 05:34:30 2021-05-09 17:48:14 [INFO] [TRAIN] epoch: 39, iter: 14460/40000, loss: 0.2942, lr: 0.006711, batch_cost: 0.7852, reader_cost: 0.00017, ips: 1.2736 samples/sec | ETA 05:34:13 2021-05-09 17:48:22 [INFO] [TRAIN] epoch: 39, iter: 14470/40000, loss: 0.1182, lr: 0.006709, batch_cost: 0.7854, reader_cost: 0.00015, ips: 1.2732 samples/sec | ETA 05:34:11 2021-05-09 17:48:30 [INFO] [TRAIN] epoch: 39, iter: 14480/40000, loss: 0.2505, lr: 0.006707, batch_cost: 0.7882, reader_cost: 0.00016, ips: 1.2687 samples/sec | ETA 05:35:15 2021-05-09 17:48:38 [INFO] [TRAIN] epoch: 39, iter: 14490/40000, loss: 0.3665, lr: 0.006704, batch_cost: 0.7872, reader_cost: 0.00017, ips: 1.2703 samples/sec | ETA 05:34:41 2021-05-09 17:48:46 [INFO] [TRAIN] epoch: 39, iter: 14500/40000, loss: 0.2812, lr: 0.006702, batch_cost: 0.7877, reader_cost: 0.00017, ips: 1.2694 samples/sec | ETA 05:34:47 2021-05-09 17:48:57 [INFO] [TRAIN] epoch: 40, iter: 14510/40000, loss: 0.3816, lr: 0.006700, batch_cost: 1.1094, reader_cost: 0.23546, ips: 0.9014 samples/sec | ETA 07:51:18 2021-05-09 17:49:05 [INFO] [TRAIN] epoch: 40, iter: 14520/40000, loss: 0.2902, lr: 0.006697, batch_cost: 0.7948, reader_cost: 0.00033, ips: 1.2581 samples/sec | ETA 05:37:32 2021-05-09 17:49:12 [INFO] [TRAIN] epoch: 40, iter: 14530/40000, loss: 0.6379, lr: 0.006695, batch_cost: 0.7868, reader_cost: 0.00017, ips: 1.2710 samples/sec | ETA 05:33:59 2021-05-09 17:49:20 [INFO] [TRAIN] epoch: 40, iter: 14540/40000, loss: 0.2606, lr: 0.006693, batch_cost: 0.7869, reader_cost: 0.00015, ips: 1.2708 samples/sec | ETA 05:33:54 2021-05-09 17:49:28 [INFO] [TRAIN] epoch: 40, iter: 14550/40000, loss: 0.5042, lr: 0.006690, batch_cost: 0.7873, reader_cost: 0.00015, ips: 1.2702 samples/sec | ETA 05:33:55 2021-05-09 17:49:36 [INFO] [TRAIN] epoch: 40, iter: 14560/40000, loss: 0.6006, lr: 0.006688, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2716 samples/sec | ETA 05:33:25 2021-05-09 17:49:44 [INFO] [TRAIN] epoch: 40, iter: 14570/40000, loss: 0.4138, lr: 0.006686, batch_cost: 0.7877, reader_cost: 0.00016, ips: 1.2695 samples/sec | ETA 05:33:52 2021-05-09 17:49:52 [INFO] [TRAIN] epoch: 40, iter: 14580/40000, loss: 0.1732, lr: 0.006683, batch_cost: 0.7877, reader_cost: 0.00015, ips: 1.2695 samples/sec | ETA 05:33:44 2021-05-09 17:50:00 [INFO] [TRAIN] epoch: 40, iter: 14590/40000, loss: 0.3756, lr: 0.006681, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2704 samples/sec | ETA 05:33:21 2021-05-09 17:50:08 [INFO] [TRAIN] epoch: 40, iter: 14600/40000, loss: 0.3389, lr: 0.006679, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 05:32:47 2021-05-09 17:50:15 [INFO] [TRAIN] epoch: 40, iter: 14610/40000, loss: 0.2206, lr: 0.006676, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2732 samples/sec | ETA 05:32:21 2021-05-09 17:50:23 [INFO] [TRAIN] epoch: 40, iter: 14620/40000, loss: 0.3241, lr: 0.006674, batch_cost: 0.7864, reader_cost: 0.00014, ips: 1.2716 samples/sec | ETA 05:32:39 2021-05-09 17:50:31 [INFO] [TRAIN] epoch: 40, iter: 14630/40000, loss: 0.1476, lr: 0.006672, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2716 samples/sec | ETA 05:32:30 2021-05-09 17:50:39 [INFO] [TRAIN] epoch: 40, iter: 14640/40000, loss: 0.4059, lr: 0.006669, batch_cost: 0.7860, reader_cost: 0.00014, ips: 1.2723 samples/sec | ETA 05:32:12 2021-05-09 17:50:47 [INFO] [TRAIN] epoch: 40, iter: 14650/40000, loss: 0.3074, lr: 0.006667, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2714 samples/sec | ETA 05:32:19 2021-05-09 17:50:55 [INFO] [TRAIN] epoch: 40, iter: 14660/40000, loss: 0.2879, lr: 0.006665, batch_cost: 0.7875, reader_cost: 0.00016, ips: 1.2699 samples/sec | ETA 05:32:34 2021-05-09 17:51:03 [INFO] [TRAIN] epoch: 40, iter: 14670/40000, loss: 0.2432, lr: 0.006662, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 05:31:57 2021-05-09 17:51:10 [INFO] [TRAIN] epoch: 40, iter: 14680/40000, loss: 0.2076, lr: 0.006660, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 05:31:50 2021-05-09 17:51:18 [INFO] [TRAIN] epoch: 40, iter: 14690/40000, loss: 0.2694, lr: 0.006658, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2718 samples/sec | ETA 05:31:41 2021-05-09 17:51:26 [INFO] [TRAIN] epoch: 40, iter: 14700/40000, loss: 0.1380, lr: 0.006655, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 05:31:37 2021-05-09 17:51:34 [INFO] [TRAIN] epoch: 40, iter: 14710/40000, loss: 0.2028, lr: 0.006653, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2727 samples/sec | ETA 05:31:11 2021-05-09 17:51:42 [INFO] [TRAIN] epoch: 40, iter: 14720/40000, loss: 0.4330, lr: 0.006651, batch_cost: 0.7861, reader_cost: 0.00017, ips: 1.2721 samples/sec | ETA 05:31:12 2021-05-09 17:51:50 [INFO] [TRAIN] epoch: 40, iter: 14730/40000, loss: 0.2898, lr: 0.006648, batch_cost: 0.7846, reader_cost: 0.00016, ips: 1.2745 samples/sec | ETA 05:30:26 2021-05-09 17:51:58 [INFO] [TRAIN] epoch: 40, iter: 14740/40000, loss: 0.2716, lr: 0.006646, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2735 samples/sec | ETA 05:30:35 2021-05-09 17:52:05 [INFO] [TRAIN] epoch: 40, iter: 14750/40000, loss: 0.2949, lr: 0.006644, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 05:30:56 2021-05-09 17:52:13 [INFO] [TRAIN] epoch: 40, iter: 14760/40000, loss: 0.2579, lr: 0.006641, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2715 samples/sec | ETA 05:30:49 2021-05-09 17:52:21 [INFO] [TRAIN] epoch: 40, iter: 14770/40000, loss: 0.1841, lr: 0.006639, batch_cost: 0.7886, reader_cost: 0.00017, ips: 1.2681 samples/sec | ETA 05:31:36 2021-05-09 17:52:29 [INFO] [TRAIN] epoch: 40, iter: 14780/40000, loss: 0.8428, lr: 0.006637, batch_cost: 0.7867, reader_cost: 0.00015, ips: 1.2711 samples/sec | ETA 05:30:40 2021-05-09 17:52:37 [INFO] [TRAIN] epoch: 40, iter: 14790/40000, loss: 0.5513, lr: 0.006634, batch_cost: 0.7869, reader_cost: 0.00017, ips: 1.2708 samples/sec | ETA 05:30:37 2021-05-09 17:52:45 [INFO] [TRAIN] epoch: 40, iter: 14800/40000, loss: 0.2839, lr: 0.006632, batch_cost: 0.7875, reader_cost: 0.00017, ips: 1.2698 samples/sec | ETA 05:30:45 2021-05-09 17:52:53 [INFO] [TRAIN] epoch: 40, iter: 14810/40000, loss: 0.4993, lr: 0.006630, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2719 samples/sec | ETA 05:30:04 2021-05-09 17:53:01 [INFO] [TRAIN] epoch: 40, iter: 14820/40000, loss: 0.3253, lr: 0.006627, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2704 samples/sec | ETA 05:30:19 2021-05-09 17:53:08 [INFO] [TRAIN] epoch: 40, iter: 14830/40000, loss: 0.2054, lr: 0.006625, batch_cost: 0.7886, reader_cost: 0.00015, ips: 1.2681 samples/sec | ETA 05:30:47 2021-05-09 17:53:16 [INFO] [TRAIN] epoch: 40, iter: 14840/40000, loss: 0.1499, lr: 0.006623, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2703 samples/sec | ETA 05:30:07 2021-05-09 17:53:24 [INFO] [TRAIN] epoch: 40, iter: 14850/40000, loss: 0.1737, lr: 0.006621, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2707 samples/sec | ETA 05:29:51 2021-05-09 17:53:32 [INFO] [TRAIN] epoch: 40, iter: 14860/40000, loss: 0.2827, lr: 0.006618, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2705 samples/sec | ETA 05:29:47 2021-05-09 17:53:40 [INFO] [TRAIN] epoch: 40, iter: 14870/40000, loss: 0.3893, lr: 0.006616, batch_cost: 0.7871, reader_cost: 0.00014, ips: 1.2705 samples/sec | ETA 05:29:39 2021-05-09 17:53:48 [INFO] [TRAIN] epoch: 40, iter: 14880/40000, loss: 0.4317, lr: 0.006614, batch_cost: 0.7859, reader_cost: 0.00010, ips: 1.2724 samples/sec | ETA 05:29:01 2021-05-09 17:53:59 [INFO] [TRAIN] epoch: 41, iter: 14890/40000, loss: 0.1883, lr: 0.006611, batch_cost: 1.0846, reader_cost: 0.23197, ips: 0.9220 samples/sec | ETA 07:33:55 2021-05-09 17:54:07 [INFO] [TRAIN] epoch: 41, iter: 14900/40000, loss: 0.4439, lr: 0.006609, batch_cost: 0.7907, reader_cost: 0.00032, ips: 1.2648 samples/sec | ETA 05:30:45 2021-05-09 17:54:14 [INFO] [TRAIN] epoch: 41, iter: 14910/40000, loss: 0.4527, lr: 0.006606, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 05:28:56 2021-05-09 17:54:22 [INFO] [TRAIN] epoch: 41, iter: 14920/40000, loss: 0.5499, lr: 0.006604, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 05:28:49 2021-05-09 17:54:30 [INFO] [TRAIN] epoch: 41, iter: 14930/40000, loss: 0.4971, lr: 0.006602, batch_cost: 0.7878, reader_cost: 0.00017, ips: 1.2693 samples/sec | ETA 05:29:10 2021-05-09 17:54:38 [INFO] [TRAIN] epoch: 41, iter: 14940/40000, loss: 0.2977, lr: 0.006599, batch_cost: 0.7858, reader_cost: 0.00017, ips: 1.2726 samples/sec | ETA 05:28:12 2021-05-09 17:54:46 [INFO] [TRAIN] epoch: 41, iter: 14950/40000, loss: 0.2323, lr: 0.006597, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2720 samples/sec | ETA 05:28:12 2021-05-09 17:54:54 [INFO] [TRAIN] epoch: 41, iter: 14960/40000, loss: 0.3323, lr: 0.006595, batch_cost: 0.7873, reader_cost: 0.00017, ips: 1.2702 samples/sec | ETA 05:28:33 2021-05-09 17:55:02 [INFO] [TRAIN] epoch: 41, iter: 14970/40000, loss: 0.3934, lr: 0.006592, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2733 samples/sec | ETA 05:27:37 2021-05-09 17:55:10 [INFO] [TRAIN] epoch: 41, iter: 14980/40000, loss: 0.2521, lr: 0.006590, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 05:27:55 2021-05-09 17:55:17 [INFO] [TRAIN] epoch: 41, iter: 14990/40000, loss: 0.3426, lr: 0.006588, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2721 samples/sec | ETA 05:27:41 2021-05-09 17:55:25 [INFO] [TRAIN] epoch: 41, iter: 15000/40000, loss: 0.1150, lr: 0.006585, batch_cost: 0.7880, reader_cost: 0.00018, ips: 1.2691 samples/sec | ETA 05:28:19 2021-05-09 17:55:25 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-09 17:58:57 [INFO] [EVAL] #Images: 500 mIoU: 0.7531 Acc: 0.9545 Kappa: 0.9410 2021-05-09 17:58:57 [INFO] [EVAL] Class IoU: [0.9794 0.834 0.9153 0.6298 0.6033 0.4762 0.6001 0.7143 0.9126 0.6437 0.9368 0.7656 0.5564 0.9371 0.8282 0.8845 0.7449 0.6237 0.7226] 2021-05-09 17:58:57 [INFO] [EVAL] Class Acc: [0.9904 0.9048 0.9472 0.8223 0.8432 0.7389 0.7674 0.8693 0.9493 0.8324 0.959 0.8499 0.7557 0.9616 0.9289 0.9479 0.8574 0.8335 0.8332] 2021-05-09 17:59:45 [INFO] [EVAL] The model with the best validation mIoU (0.7531) was saved at iter 15000. 2021-05-09 17:59:53 [INFO] [TRAIN] epoch: 41, iter: 15010/40000, loss: 0.3063, lr: 0.006583, batch_cost: 0.7819, reader_cost: 0.00028, ips: 1.2790 samples/sec | ETA 05:25:38 2021-05-09 18:00:00 [INFO] [TRAIN] epoch: 41, iter: 15020/40000, loss: 0.2991, lr: 0.006581, batch_cost: 0.7838, reader_cost: 0.00016, ips: 1.2758 samples/sec | ETA 05:26:19 2021-05-09 18:00:08 [INFO] [TRAIN] epoch: 41, iter: 15030/40000, loss: 0.4055, lr: 0.006578, batch_cost: 0.7847, reader_cost: 0.00014, ips: 1.2743 samples/sec | ETA 05:26:35 2021-05-09 18:00:16 [INFO] [TRAIN] epoch: 41, iter: 15040/40000, loss: 0.2590, lr: 0.006576, batch_cost: 0.7844, reader_cost: 0.00015, ips: 1.2749 samples/sec | ETA 05:26:18 2021-05-09 18:00:24 [INFO] [TRAIN] epoch: 41, iter: 15050/40000, loss: 0.2697, lr: 0.006574, batch_cost: 0.7838, reader_cost: 0.00015, ips: 1.2758 samples/sec | ETA 05:25:55 2021-05-09 18:00:32 [INFO] [TRAIN] epoch: 41, iter: 15060/40000, loss: 0.1755, lr: 0.006571, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2723 samples/sec | ETA 05:26:42 2021-05-09 18:00:40 [INFO] [TRAIN] epoch: 41, iter: 15070/40000, loss: 0.2811, lr: 0.006569, batch_cost: 0.7857, reader_cost: 0.00017, ips: 1.2728 samples/sec | ETA 05:26:27 2021-05-09 18:00:47 [INFO] [TRAIN] epoch: 41, iter: 15080/40000, loss: 0.2367, lr: 0.006567, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2720 samples/sec | ETA 05:26:31 2021-05-09 18:00:55 [INFO] [TRAIN] epoch: 41, iter: 15090/40000, loss: 0.3978, lr: 0.006564, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2711 samples/sec | ETA 05:26:37 2021-05-09 18:01:03 [INFO] [TRAIN] epoch: 41, iter: 15100/40000, loss: 0.2780, lr: 0.006562, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2706 samples/sec | ETA 05:26:37 2021-05-09 18:01:11 [INFO] [TRAIN] epoch: 41, iter: 15110/40000, loss: 0.3255, lr: 0.006560, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2706 samples/sec | ETA 05:26:29 2021-05-09 18:01:19 [INFO] [TRAIN] epoch: 41, iter: 15120/40000, loss: 0.2902, lr: 0.006557, batch_cost: 0.7851, reader_cost: 0.00016, ips: 1.2738 samples/sec | ETA 05:25:32 2021-05-09 18:01:27 [INFO] [TRAIN] epoch: 41, iter: 15130/40000, loss: 0.2962, lr: 0.006555, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 05:25:49 2021-05-09 18:01:35 [INFO] [TRAIN] epoch: 41, iter: 15140/40000, loss: 0.1442, lr: 0.006553, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2711 samples/sec | ETA 05:25:58 2021-05-09 18:01:43 [INFO] [TRAIN] epoch: 41, iter: 15150/40000, loss: 0.2237, lr: 0.006550, batch_cost: 0.7872, reader_cost: 0.00017, ips: 1.2704 samples/sec | ETA 05:26:00 2021-05-09 18:01:50 [INFO] [TRAIN] epoch: 41, iter: 15160/40000, loss: 0.4186, lr: 0.006548, batch_cost: 0.7851, reader_cost: 0.00015, ips: 1.2737 samples/sec | ETA 05:25:02 2021-05-09 18:01:58 [INFO] [TRAIN] epoch: 41, iter: 15170/40000, loss: 0.4714, lr: 0.006546, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2705 samples/sec | ETA 05:25:42 2021-05-09 18:02:06 [INFO] [TRAIN] epoch: 41, iter: 15180/40000, loss: 0.4819, lr: 0.006543, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 05:25:17 2021-05-09 18:02:14 [INFO] [TRAIN] epoch: 41, iter: 15190/40000, loss: 0.5603, lr: 0.006541, batch_cost: 0.7878, reader_cost: 0.00016, ips: 1.2694 samples/sec | ETA 05:25:44 2021-05-09 18:02:22 [INFO] [TRAIN] epoch: 41, iter: 15200/40000, loss: 0.2987, lr: 0.006539, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2720 samples/sec | ETA 05:24:56 2021-05-09 18:02:30 [INFO] [TRAIN] epoch: 41, iter: 15210/40000, loss: 0.1709, lr: 0.006536, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2731 samples/sec | ETA 05:24:31 2021-05-09 18:02:38 [INFO] [TRAIN] epoch: 41, iter: 15220/40000, loss: 0.3326, lr: 0.006534, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2714 samples/sec | ETA 05:24:49 2021-05-09 18:02:45 [INFO] [TRAIN] epoch: 41, iter: 15230/40000, loss: 0.3457, lr: 0.006532, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2718 samples/sec | ETA 05:24:36 2021-05-09 18:02:53 [INFO] [TRAIN] epoch: 41, iter: 15240/40000, loss: 0.3848, lr: 0.006529, batch_cost: 0.7871, reader_cost: 0.00015, ips: 1.2704 samples/sec | ETA 05:24:49 2021-05-09 18:03:01 [INFO] [TRAIN] epoch: 41, iter: 15250/40000, loss: 0.3456, lr: 0.006527, batch_cost: 0.7851, reader_cost: 0.00010, ips: 1.2736 samples/sec | ETA 05:23:52 2021-05-09 18:03:12 [INFO] [TRAIN] epoch: 42, iter: 15260/40000, loss: 0.4053, lr: 0.006525, batch_cost: 1.1021, reader_cost: 0.26588, ips: 0.9073 samples/sec | ETA 07:34:26 2021-05-09 18:03:20 [INFO] [TRAIN] epoch: 42, iter: 15270/40000, loss: 0.5651, lr: 0.006522, batch_cost: 0.7964, reader_cost: 0.00032, ips: 1.2557 samples/sec | ETA 05:28:14 2021-05-09 18:03:28 [INFO] [TRAIN] epoch: 42, iter: 15280/40000, loss: 0.4450, lr: 0.006520, batch_cost: 0.7863, reader_cost: 0.00017, ips: 1.2717 samples/sec | ETA 05:23:58 2021-05-09 18:03:36 [INFO] [TRAIN] epoch: 42, iter: 15290/40000, loss: 0.3810, lr: 0.006518, batch_cost: 0.7865, reader_cost: 0.00017, ips: 1.2714 samples/sec | ETA 05:23:55 2021-05-09 18:03:44 [INFO] [TRAIN] epoch: 42, iter: 15300/40000, loss: 0.4230, lr: 0.006515, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2718 samples/sec | ETA 05:23:40 2021-05-09 18:03:52 [INFO] [TRAIN] epoch: 42, iter: 15310/40000, loss: 0.3056, lr: 0.006513, batch_cost: 0.7867, reader_cost: 0.00015, ips: 1.2711 samples/sec | ETA 05:23:43 2021-05-09 18:03:59 [INFO] [TRAIN] epoch: 42, iter: 15320/40000, loss: 0.2472, lr: 0.006511, batch_cost: 0.7857, reader_cost: 0.00014, ips: 1.2727 samples/sec | ETA 05:23:12 2021-05-09 18:04:07 [INFO] [TRAIN] epoch: 42, iter: 15330/40000, loss: 0.2001, lr: 0.006508, batch_cost: 0.7872, reader_cost: 0.00015, ips: 1.2703 samples/sec | ETA 05:23:40 2021-05-09 18:04:15 [INFO] [TRAIN] epoch: 42, iter: 15340/40000, loss: 0.3547, lr: 0.006506, batch_cost: 0.7853, reader_cost: 0.00015, ips: 1.2734 samples/sec | ETA 05:22:45 2021-05-09 18:04:23 [INFO] [TRAIN] epoch: 42, iter: 15350/40000, loss: 0.3078, lr: 0.006504, batch_cost: 0.7874, reader_cost: 0.00015, ips: 1.2701 samples/sec | ETA 05:23:28 2021-05-09 18:04:31 [INFO] [TRAIN] epoch: 42, iter: 15360/40000, loss: 0.3069, lr: 0.006501, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2716 samples/sec | ETA 05:22:57 2021-05-09 18:04:39 [INFO] [TRAIN] epoch: 42, iter: 15370/40000, loss: 0.2083, lr: 0.006499, batch_cost: 0.7867, reader_cost: 0.00017, ips: 1.2712 samples/sec | ETA 05:22:55 2021-05-09 18:04:47 [INFO] [TRAIN] epoch: 42, iter: 15380/40000, loss: 0.2834, lr: 0.006497, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2732 samples/sec | ETA 05:22:17 2021-05-09 18:04:55 [INFO] [TRAIN] epoch: 42, iter: 15390/40000, loss: 0.4943, lr: 0.006494, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 05:22:25 2021-05-09 18:05:02 [INFO] [TRAIN] epoch: 42, iter: 15400/40000, loss: 0.3148, lr: 0.006492, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 05:22:02 2021-05-09 18:05:10 [INFO] [TRAIN] epoch: 42, iter: 15410/40000, loss: 0.5428, lr: 0.006490, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2733 samples/sec | ETA 05:21:52 2021-05-09 18:05:18 [INFO] [TRAIN] epoch: 42, iter: 15420/40000, loss: 0.3018, lr: 0.006487, batch_cost: 0.7855, reader_cost: 0.00017, ips: 1.2730 samples/sec | ETA 05:21:48 2021-05-09 18:05:26 [INFO] [TRAIN] epoch: 42, iter: 15430/40000, loss: 0.2838, lr: 0.006485, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2721 samples/sec | ETA 05:21:54 2021-05-09 18:05:34 [INFO] [TRAIN] epoch: 42, iter: 15440/40000, loss: 0.3328, lr: 0.006483, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2712 samples/sec | ETA 05:21:59 2021-05-09 18:05:42 [INFO] [TRAIN] epoch: 42, iter: 15450/40000, loss: 0.0917, lr: 0.006480, batch_cost: 0.7853, reader_cost: 0.00017, ips: 1.2734 samples/sec | ETA 05:21:19 2021-05-09 18:05:50 [INFO] [TRAIN] epoch: 42, iter: 15460/40000, loss: 0.2660, lr: 0.006478, batch_cost: 0.7851, reader_cost: 0.00017, ips: 1.2736 samples/sec | ETA 05:21:07 2021-05-09 18:05:57 [INFO] [TRAIN] epoch: 42, iter: 15470/40000, loss: 0.3722, lr: 0.006476, batch_cost: 0.7847, reader_cost: 0.00017, ips: 1.2744 samples/sec | ETA 05:20:48 2021-05-09 18:06:05 [INFO] [TRAIN] epoch: 42, iter: 15480/40000, loss: 0.2360, lr: 0.006473, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 05:21:00 2021-05-09 18:06:13 [INFO] [TRAIN] epoch: 42, iter: 15490/40000, loss: 0.2429, lr: 0.006471, batch_cost: 0.7837, reader_cost: 0.00016, ips: 1.2760 samples/sec | ETA 05:20:08 2021-05-09 18:06:21 [INFO] [TRAIN] epoch: 42, iter: 15500/40000, loss: 0.2221, lr: 0.006469, batch_cost: 0.7844, reader_cost: 0.00016, ips: 1.2749 samples/sec | ETA 05:20:17 2021-05-09 18:06:29 [INFO] [TRAIN] epoch: 42, iter: 15510/40000, loss: 0.3216, lr: 0.006466, batch_cost: 0.7847, reader_cost: 0.00016, ips: 1.2743 samples/sec | ETA 05:20:17 2021-05-09 18:06:37 [INFO] [TRAIN] epoch: 42, iter: 15520/40000, loss: 0.1743, lr: 0.006464, batch_cost: 0.7860, reader_cost: 0.00014, ips: 1.2722 samples/sec | ETA 05:20:42 2021-05-09 18:06:44 [INFO] [TRAIN] epoch: 42, iter: 15530/40000, loss: 0.3903, lr: 0.006462, batch_cost: 0.7867, reader_cost: 0.00017, ips: 1.2711 samples/sec | ETA 05:20:51 2021-05-09 18:06:52 [INFO] [TRAIN] epoch: 42, iter: 15540/40000, loss: 0.5011, lr: 0.006459, batch_cost: 0.7851, reader_cost: 0.00015, ips: 1.2738 samples/sec | ETA 05:20:02 2021-05-09 18:07:00 [INFO] [TRAIN] epoch: 42, iter: 15550/40000, loss: 0.3500, lr: 0.006457, batch_cost: 0.7845, reader_cost: 0.00015, ips: 1.2747 samples/sec | ETA 05:19:41 2021-05-09 18:07:08 [INFO] [TRAIN] epoch: 42, iter: 15560/40000, loss: 0.5904, lr: 0.006455, batch_cost: 0.7848, reader_cost: 0.00016, ips: 1.2742 samples/sec | ETA 05:19:40 2021-05-09 18:07:16 [INFO] [TRAIN] epoch: 42, iter: 15570/40000, loss: 0.3875, lr: 0.006452, batch_cost: 0.7849, reader_cost: 0.00017, ips: 1.2741 samples/sec | ETA 05:19:34 2021-05-09 18:07:24 [INFO] [TRAIN] epoch: 42, iter: 15580/40000, loss: 0.1506, lr: 0.006450, batch_cost: 0.7879, reader_cost: 0.00016, ips: 1.2692 samples/sec | ETA 05:20:40 2021-05-09 18:07:32 [INFO] [TRAIN] epoch: 42, iter: 15590/40000, loss: 0.2645, lr: 0.006448, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2722 samples/sec | ETA 05:19:47 2021-05-09 18:07:39 [INFO] [TRAIN] epoch: 42, iter: 15600/40000, loss: 0.3979, lr: 0.006445, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 05:19:39 2021-05-09 18:07:47 [INFO] [TRAIN] epoch: 42, iter: 15610/40000, loss: 0.3445, lr: 0.006443, batch_cost: 0.7871, reader_cost: 0.00018, ips: 1.2705 samples/sec | ETA 05:19:57 2021-05-09 18:07:55 [INFO] [TRAIN] epoch: 42, iter: 15620/40000, loss: 0.3927, lr: 0.006441, batch_cost: 0.7843, reader_cost: 0.00012, ips: 1.2750 samples/sec | ETA 05:18:42 2021-05-09 18:08:06 [INFO] [TRAIN] epoch: 43, iter: 15630/40000, loss: 0.2720, lr: 0.006438, batch_cost: 1.0905, reader_cost: 0.26584, ips: 0.9170 samples/sec | ETA 07:22:56 2021-05-09 18:08:14 [INFO] [TRAIN] epoch: 43, iter: 15640/40000, loss: 0.3804, lr: 0.006436, batch_cost: 0.7955, reader_cost: 0.00035, ips: 1.2571 samples/sec | ETA 05:22:57 2021-05-09 18:08:22 [INFO] [TRAIN] epoch: 43, iter: 15650/40000, loss: 0.5846, lr: 0.006434, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2718 samples/sec | ETA 05:19:05 2021-05-09 18:08:30 [INFO] [TRAIN] epoch: 43, iter: 15660/40000, loss: 0.3979, lr: 0.006431, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2718 samples/sec | ETA 05:18:58 2021-05-09 18:08:38 [INFO] [TRAIN] epoch: 43, iter: 15670/40000, loss: 0.4729, lr: 0.006429, batch_cost: 0.7861, reader_cost: 0.00017, ips: 1.2722 samples/sec | ETA 05:18:44 2021-05-09 18:08:46 [INFO] [TRAIN] epoch: 43, iter: 15680/40000, loss: 0.4918, lr: 0.006427, batch_cost: 0.7864, reader_cost: 0.00017, ips: 1.2717 samples/sec | ETA 05:18:44 2021-05-09 18:08:53 [INFO] [TRAIN] epoch: 43, iter: 15690/40000, loss: 0.2835, lr: 0.006424, batch_cost: 0.7866, reader_cost: 0.00017, ips: 1.2713 samples/sec | ETA 05:18:42 2021-05-09 18:09:01 [INFO] [TRAIN] epoch: 43, iter: 15700/40000, loss: 0.1439, lr: 0.006422, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 05:18:36 2021-05-09 18:09:09 [INFO] [TRAIN] epoch: 43, iter: 15710/40000, loss: 0.3705, lr: 0.006419, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2705 samples/sec | ETA 05:18:37 2021-05-09 18:09:17 [INFO] [TRAIN] epoch: 43, iter: 15720/40000, loss: 0.2351, lr: 0.006417, batch_cost: 0.7882, reader_cost: 0.00016, ips: 1.2687 samples/sec | ETA 05:18:57 2021-05-09 18:09:25 [INFO] [TRAIN] epoch: 43, iter: 15730/40000, loss: 0.3587, lr: 0.006415, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 05:18:08 2021-05-09 18:09:33 [INFO] [TRAIN] epoch: 43, iter: 15740/40000, loss: 0.2613, lr: 0.006412, batch_cost: 0.7869, reader_cost: 0.00017, ips: 1.2707 samples/sec | ETA 05:18:11 2021-05-09 18:09:41 [INFO] [TRAIN] epoch: 43, iter: 15750/40000, loss: 0.1772, lr: 0.006410, batch_cost: 0.7853, reader_cost: 0.00018, ips: 1.2734 samples/sec | ETA 05:17:23 2021-05-09 18:09:48 [INFO] [TRAIN] epoch: 43, iter: 15760/40000, loss: 0.3448, lr: 0.006408, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2718 samples/sec | ETA 05:17:39 2021-05-09 18:09:56 [INFO] [TRAIN] epoch: 43, iter: 15770/40000, loss: 1.1140, lr: 0.006405, batch_cost: 0.7855, reader_cost: 0.00017, ips: 1.2730 samples/sec | ETA 05:17:13 2021-05-09 18:10:04 [INFO] [TRAIN] epoch: 43, iter: 15780/40000, loss: 0.4231, lr: 0.006403, batch_cost: 0.7846, reader_cost: 0.00015, ips: 1.2745 samples/sec | ETA 05:16:43 2021-05-09 18:10:12 [INFO] [TRAIN] epoch: 43, iter: 15790/40000, loss: 0.3546, lr: 0.006401, batch_cost: 0.7866, reader_cost: 0.00018, ips: 1.2714 samples/sec | ETA 05:17:22 2021-05-09 18:10:20 [INFO] [TRAIN] epoch: 43, iter: 15800/40000, loss: 0.2535, lr: 0.006398, batch_cost: 0.7865, reader_cost: 0.00017, ips: 1.2715 samples/sec | ETA 05:17:12 2021-05-09 18:10:28 [INFO] [TRAIN] epoch: 43, iter: 15810/40000, loss: 0.3043, lr: 0.006396, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2709 samples/sec | ETA 05:17:13 2021-05-09 18:10:36 [INFO] [TRAIN] epoch: 43, iter: 15820/40000, loss: 0.1827, lr: 0.006394, batch_cost: 0.7855, reader_cost: 0.00017, ips: 1.2731 samples/sec | ETA 05:16:33 2021-05-09 18:10:43 [INFO] [TRAIN] epoch: 43, iter: 15830/40000, loss: 0.3152, lr: 0.006391, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2729 samples/sec | ETA 05:16:27 2021-05-09 18:10:51 [INFO] [TRAIN] epoch: 43, iter: 15840/40000, loss: 0.3581, lr: 0.006389, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 05:16:27 2021-05-09 18:10:59 [INFO] [TRAIN] epoch: 43, iter: 15850/40000, loss: 0.2348, lr: 0.006387, batch_cost: 0.7864, reader_cost: 0.00017, ips: 1.2716 samples/sec | ETA 05:16:31 2021-05-09 18:11:07 [INFO] [TRAIN] epoch: 43, iter: 15860/40000, loss: 0.3176, lr: 0.006384, batch_cost: 0.7856, reader_cost: 0.00017, ips: 1.2729 samples/sec | ETA 05:16:05 2021-05-09 18:11:15 [INFO] [TRAIN] epoch: 43, iter: 15870/40000, loss: 0.1798, lr: 0.006382, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2736 samples/sec | ETA 05:15:46 2021-05-09 18:11:23 [INFO] [TRAIN] epoch: 43, iter: 15880/40000, loss: 0.2765, lr: 0.006380, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2720 samples/sec | ETA 05:16:02 2021-05-09 18:11:31 [INFO] [TRAIN] epoch: 43, iter: 15890/40000, loss: 0.1895, lr: 0.006377, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2730 samples/sec | ETA 05:15:40 2021-05-09 18:11:38 [INFO] [TRAIN] epoch: 43, iter: 15900/40000, loss: 0.3929, lr: 0.006375, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2717 samples/sec | ETA 05:15:51 2021-05-09 18:11:46 [INFO] [TRAIN] epoch: 43, iter: 15910/40000, loss: 0.5715, lr: 0.006373, batch_cost: 0.7847, reader_cost: 0.00015, ips: 1.2743 samples/sec | ETA 05:15:03 2021-05-09 18:11:54 [INFO] [TRAIN] epoch: 43, iter: 15920/40000, loss: 0.3955, lr: 0.006370, batch_cost: 0.7859, reader_cost: 0.00017, ips: 1.2725 samples/sec | ETA 05:15:23 2021-05-09 18:12:02 [INFO] [TRAIN] epoch: 43, iter: 15930/40000, loss: 0.5196, lr: 0.006368, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2711 samples/sec | ETA 05:15:36 2021-05-09 18:12:10 [INFO] [TRAIN] epoch: 43, iter: 15940/40000, loss: 0.2960, lr: 0.006366, batch_cost: 0.7877, reader_cost: 0.00015, ips: 1.2695 samples/sec | ETA 05:15:52 2021-05-09 18:12:18 [INFO] [TRAIN] epoch: 43, iter: 15950/40000, loss: 0.2479, lr: 0.006363, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2719 samples/sec | ETA 05:15:09 2021-05-09 18:12:26 [INFO] [TRAIN] epoch: 43, iter: 15960/40000, loss: 0.1353, lr: 0.006361, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2716 samples/sec | ETA 05:15:05 2021-05-09 18:12:34 [INFO] [TRAIN] epoch: 43, iter: 15970/40000, loss: 0.4331, lr: 0.006359, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2728 samples/sec | ETA 05:14:39 2021-05-09 18:12:41 [INFO] [TRAIN] epoch: 43, iter: 15980/40000, loss: 0.3554, lr: 0.006356, batch_cost: 0.7866, reader_cost: 0.00018, ips: 1.2713 samples/sec | ETA 05:14:53 2021-05-09 18:12:49 [INFO] [TRAIN] epoch: 43, iter: 15990/40000, loss: 0.3473, lr: 0.006354, batch_cost: 0.7842, reader_cost: 0.00014, ips: 1.2751 samples/sec | ETA 05:13:49 2021-05-09 18:13:00 [INFO] [TRAIN] epoch: 44, iter: 16000/40000, loss: 0.2888, lr: 0.006352, batch_cost: 1.0869, reader_cost: 0.23661, ips: 0.9200 samples/sec | ETA 07:14:46 2021-05-09 18:13:00 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-09 18:16:34 [INFO] [EVAL] #Images: 500 mIoU: 0.7582 Acc: 0.9548 Kappa: 0.9413 2021-05-09 18:16:34 [INFO] [EVAL] Class IoU: [0.9785 0.8308 0.9161 0.6458 0.6156 0.4791 0.5973 0.7073 0.9134 0.6552 0.9377 0.7691 0.5739 0.9368 0.8515 0.8851 0.7699 0.6193 0.7232] 2021-05-09 18:16:34 [INFO] [EVAL] Class Acc: [0.9885 0.9144 0.9512 0.8412 0.8016 0.7468 0.7882 0.8631 0.9475 0.8365 0.9608 0.85 0.7429 0.9584 0.9308 0.9457 0.9125 0.8365 0.8314] 2021-05-09 18:17:25 [INFO] [EVAL] The model with the best validation mIoU (0.7582) was saved at iter 16000. 2021-05-09 18:17:33 [INFO] [TRAIN] epoch: 44, iter: 16010/40000, loss: 0.2573, lr: 0.006349, batch_cost: 0.7925, reader_cost: 0.00025, ips: 1.2618 samples/sec | ETA 05:16:52 2021-05-09 18:17:41 [INFO] [TRAIN] epoch: 44, iter: 16020/40000, loss: 0.5345, lr: 0.006347, batch_cost: 0.7844, reader_cost: 0.00017, ips: 1.2748 samples/sec | ETA 05:13:30 2021-05-09 18:17:49 [INFO] [TRAIN] epoch: 44, iter: 16030/40000, loss: 0.2866, lr: 0.006345, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 05:14:02 2021-05-09 18:17:57 [INFO] [TRAIN] epoch: 44, iter: 16040/40000, loss: 0.5382, lr: 0.006342, batch_cost: 0.7870, reader_cost: 0.00018, ips: 1.2707 samples/sec | ETA 05:14:16 2021-05-09 18:18:04 [INFO] [TRAIN] epoch: 44, iter: 16050/40000, loss: 0.5203, lr: 0.006340, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2735 samples/sec | ETA 05:13:26 2021-05-09 18:18:12 [INFO] [TRAIN] epoch: 44, iter: 16060/40000, loss: 0.3017, lr: 0.006337, batch_cost: 0.7857, reader_cost: 0.00014, ips: 1.2728 samples/sec | ETA 05:13:28 2021-05-09 18:18:20 [INFO] [TRAIN] epoch: 44, iter: 16070/40000, loss: 0.0632, lr: 0.006335, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2719 samples/sec | ETA 05:13:33 2021-05-09 18:18:28 [INFO] [TRAIN] epoch: 44, iter: 16080/40000, loss: 0.2741, lr: 0.006333, batch_cost: 0.7875, reader_cost: 0.00016, ips: 1.2699 samples/sec | ETA 05:13:56 2021-05-09 18:18:36 [INFO] [TRAIN] epoch: 44, iter: 16090/40000, loss: 0.3031, lr: 0.006330, batch_cost: 0.7883, reader_cost: 0.00015, ips: 1.2685 samples/sec | ETA 05:14:08 2021-05-09 18:18:44 [INFO] [TRAIN] epoch: 44, iter: 16100/40000, loss: 0.2989, lr: 0.006328, batch_cost: 0.7877, reader_cost: 0.00015, ips: 1.2696 samples/sec | ETA 05:13:45 2021-05-09 18:18:52 [INFO] [TRAIN] epoch: 44, iter: 16110/40000, loss: 0.3437, lr: 0.006326, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2705 samples/sec | ETA 05:13:24 2021-05-09 18:18:59 [INFO] [TRAIN] epoch: 44, iter: 16120/40000, loss: 0.1194, lr: 0.006323, batch_cost: 0.7879, reader_cost: 0.00016, ips: 1.2693 samples/sec | ETA 05:13:34 2021-05-09 18:19:07 [INFO] [TRAIN] epoch: 44, iter: 16130/40000, loss: 0.3153, lr: 0.006321, batch_cost: 0.7877, reader_cost: 0.00015, ips: 1.2695 samples/sec | ETA 05:13:22 2021-05-09 18:19:15 [INFO] [TRAIN] epoch: 44, iter: 16140/40000, loss: 0.5082, lr: 0.006319, batch_cost: 0.7876, reader_cost: 0.00015, ips: 1.2697 samples/sec | ETA 05:13:11 2021-05-09 18:19:23 [INFO] [TRAIN] epoch: 44, iter: 16150/40000, loss: 0.3646, lr: 0.006316, batch_cost: 0.7891, reader_cost: 0.00015, ips: 1.2673 samples/sec | ETA 05:13:39 2021-05-09 18:19:31 [INFO] [TRAIN] epoch: 44, iter: 16160/40000, loss: 0.2810, lr: 0.006314, batch_cost: 0.7851, reader_cost: 0.00015, ips: 1.2737 samples/sec | ETA 05:11:57 2021-05-09 18:19:39 [INFO] [TRAIN] epoch: 44, iter: 16170/40000, loss: 0.2272, lr: 0.006312, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2709 samples/sec | ETA 05:12:30 2021-05-09 18:19:47 [INFO] [TRAIN] epoch: 44, iter: 16180/40000, loss: 0.2998, lr: 0.006309, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2722 samples/sec | ETA 05:12:03 2021-05-09 18:19:55 [INFO] [TRAIN] epoch: 44, iter: 16190/40000, loss: 0.2524, lr: 0.006307, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2720 samples/sec | ETA 05:11:58 2021-05-09 18:20:02 [INFO] [TRAIN] epoch: 44, iter: 16200/40000, loss: 0.1929, lr: 0.006305, batch_cost: 0.7877, reader_cost: 0.00014, ips: 1.2695 samples/sec | ETA 05:12:27 2021-05-09 18:20:10 [INFO] [TRAIN] epoch: 44, iter: 16210/40000, loss: 0.3303, lr: 0.006302, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2728 samples/sec | ETA 05:11:31 2021-05-09 18:20:18 [INFO] [TRAIN] epoch: 44, iter: 16220/40000, loss: 0.8977, lr: 0.006300, batch_cost: 0.7874, reader_cost: 0.00015, ips: 1.2700 samples/sec | ETA 05:12:03 2021-05-09 18:20:26 [INFO] [TRAIN] epoch: 44, iter: 16230/40000, loss: 0.2513, lr: 0.006298, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2728 samples/sec | ETA 05:11:15 2021-05-09 18:20:34 [INFO] [TRAIN] epoch: 44, iter: 16240/40000, loss: 0.5340, lr: 0.006295, batch_cost: 0.7858, reader_cost: 0.00014, ips: 1.2725 samples/sec | ETA 05:11:11 2021-05-09 18:20:42 [INFO] [TRAIN] epoch: 44, iter: 16250/40000, loss: 0.2112, lr: 0.006293, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2730 samples/sec | ETA 05:10:56 2021-05-09 18:20:50 [INFO] [TRAIN] epoch: 44, iter: 16260/40000, loss: 0.1941, lr: 0.006291, batch_cost: 0.7849, reader_cost: 0.00016, ips: 1.2740 samples/sec | ETA 05:10:34 2021-05-09 18:20:57 [INFO] [TRAIN] epoch: 44, iter: 16270/40000, loss: 0.4097, lr: 0.006288, batch_cost: 0.7871, reader_cost: 0.00017, ips: 1.2705 samples/sec | ETA 05:11:17 2021-05-09 18:21:05 [INFO] [TRAIN] epoch: 44, iter: 16280/40000, loss: 0.4282, lr: 0.006286, batch_cost: 0.7853, reader_cost: 0.00017, ips: 1.2734 samples/sec | ETA 05:10:26 2021-05-09 18:21:13 [INFO] [TRAIN] epoch: 44, iter: 16290/40000, loss: 0.2682, lr: 0.006284, batch_cost: 0.7876, reader_cost: 0.00017, ips: 1.2697 samples/sec | ETA 05:11:14 2021-05-09 18:21:21 [INFO] [TRAIN] epoch: 44, iter: 16300/40000, loss: 0.5998, lr: 0.006281, batch_cost: 0.7852, reader_cost: 0.00017, ips: 1.2736 samples/sec | ETA 05:10:08 2021-05-09 18:21:29 [INFO] [TRAIN] epoch: 44, iter: 16310/40000, loss: 0.7017, lr: 0.006279, batch_cost: 0.7846, reader_cost: 0.00017, ips: 1.2745 samples/sec | ETA 05:09:47 2021-05-09 18:21:37 [INFO] [TRAIN] epoch: 44, iter: 16320/40000, loss: 0.5791, lr: 0.006276, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2729 samples/sec | ETA 05:10:03 2021-05-09 18:21:45 [INFO] [TRAIN] epoch: 44, iter: 16330/40000, loss: 0.0856, lr: 0.006274, batch_cost: 0.7854, reader_cost: 0.00015, ips: 1.2732 samples/sec | ETA 05:09:50 2021-05-09 18:21:52 [INFO] [TRAIN] epoch: 44, iter: 16340/40000, loss: 0.2682, lr: 0.006272, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2710 samples/sec | ETA 05:10:15 2021-05-09 18:22:00 [INFO] [TRAIN] epoch: 44, iter: 16350/40000, loss: 0.3133, lr: 0.006269, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2706 samples/sec | ETA 05:10:13 2021-05-09 18:22:08 [INFO] [TRAIN] epoch: 44, iter: 16360/40000, loss: 0.3143, lr: 0.006267, batch_cost: 0.7844, reader_cost: 0.00014, ips: 1.2749 samples/sec | ETA 05:09:02 2021-05-09 18:22:19 [INFO] [TRAIN] epoch: 45, iter: 16370/40000, loss: 0.4252, lr: 0.006265, batch_cost: 1.1149, reader_cost: 0.29695, ips: 0.8969 samples/sec | ETA 07:19:05 2021-05-09 18:22:27 [INFO] [TRAIN] epoch: 45, iter: 16380/40000, loss: 0.2705, lr: 0.006262, batch_cost: 0.7981, reader_cost: 0.00031, ips: 1.2530 samples/sec | ETA 05:14:11 2021-05-09 18:22:35 [INFO] [TRAIN] epoch: 45, iter: 16390/40000, loss: 0.6312, lr: 0.006260, batch_cost: 0.7856, reader_cost: 0.00017, ips: 1.2730 samples/sec | ETA 05:09:06 2021-05-09 18:22:43 [INFO] [TRAIN] epoch: 45, iter: 16400/40000, loss: 0.2251, lr: 0.006258, batch_cost: 0.7871, reader_cost: 0.00015, ips: 1.2705 samples/sec | ETA 05:09:35 2021-05-09 18:22:51 [INFO] [TRAIN] epoch: 45, iter: 16410/40000, loss: 0.4879, lr: 0.006255, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2728 samples/sec | ETA 05:08:53 2021-05-09 18:22:59 [INFO] [TRAIN] epoch: 45, iter: 16420/40000, loss: 0.4660, lr: 0.006253, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2736 samples/sec | ETA 05:08:34 2021-05-09 18:23:07 [INFO] [TRAIN] epoch: 45, iter: 16430/40000, loss: 0.5148, lr: 0.006251, batch_cost: 0.7858, reader_cost: 0.00017, ips: 1.2727 samples/sec | ETA 05:08:40 2021-05-09 18:23:14 [INFO] [TRAIN] epoch: 45, iter: 16440/40000, loss: 0.2753, lr: 0.006248, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2716 samples/sec | ETA 05:08:48 2021-05-09 18:23:22 [INFO] [TRAIN] epoch: 45, iter: 16450/40000, loss: 0.2878, lr: 0.006246, batch_cost: 0.7862, reader_cost: 0.00017, ips: 1.2719 samples/sec | ETA 05:08:35 2021-05-09 18:23:30 [INFO] [TRAIN] epoch: 45, iter: 16460/40000, loss: 0.4266, lr: 0.006244, batch_cost: 0.7867, reader_cost: 0.00017, ips: 1.2711 samples/sec | ETA 05:08:40 2021-05-09 18:23:38 [INFO] [TRAIN] epoch: 45, iter: 16470/40000, loss: 0.2426, lr: 0.006241, batch_cost: 0.7864, reader_cost: 0.00018, ips: 1.2716 samples/sec | ETA 05:08:23 2021-05-09 18:23:46 [INFO] [TRAIN] epoch: 45, iter: 16480/40000, loss: 0.3174, lr: 0.006239, batch_cost: 0.7863, reader_cost: 0.00017, ips: 1.2718 samples/sec | ETA 05:08:13 2021-05-09 18:23:54 [INFO] [TRAIN] epoch: 45, iter: 16490/40000, loss: 0.1921, lr: 0.006237, batch_cost: 0.7876, reader_cost: 0.00018, ips: 1.2696 samples/sec | ETA 05:08:37 2021-05-09 18:24:02 [INFO] [TRAIN] epoch: 45, iter: 16500/40000, loss: 0.3535, lr: 0.006234, batch_cost: 0.7872, reader_cost: 0.00015, ips: 1.2704 samples/sec | ETA 05:08:18 2021-05-09 18:24:10 [INFO] [TRAIN] epoch: 45, iter: 16510/40000, loss: 0.3163, lr: 0.006232, batch_cost: 0.7859, reader_cost: 0.00017, ips: 1.2725 samples/sec | ETA 05:07:39 2021-05-09 18:24:17 [INFO] [TRAIN] epoch: 45, iter: 16520/40000, loss: 0.3093, lr: 0.006230, batch_cost: 0.7867, reader_cost: 0.00015, ips: 1.2712 samples/sec | ETA 05:07:50 2021-05-09 18:24:25 [INFO] [TRAIN] epoch: 45, iter: 16530/40000, loss: 0.2280, lr: 0.006227, batch_cost: 0.7876, reader_cost: 0.00015, ips: 1.2697 samples/sec | ETA 05:08:04 2021-05-09 18:24:33 [INFO] [TRAIN] epoch: 45, iter: 16540/40000, loss: 0.1419, lr: 0.006225, batch_cost: 0.7887, reader_cost: 0.00015, ips: 1.2679 samples/sec | ETA 05:08:22 2021-05-09 18:24:41 [INFO] [TRAIN] epoch: 45, iter: 16550/40000, loss: 0.4153, lr: 0.006222, batch_cost: 0.7870, reader_cost: 0.00015, ips: 1.2706 samples/sec | ETA 05:07:35 2021-05-09 18:24:49 [INFO] [TRAIN] epoch: 45, iter: 16560/40000, loss: 0.1349, lr: 0.006220, batch_cost: 0.7887, reader_cost: 0.00017, ips: 1.2680 samples/sec | ETA 05:08:06 2021-05-09 18:24:57 [INFO] [TRAIN] epoch: 45, iter: 16570/40000, loss: 0.1632, lr: 0.006218, batch_cost: 0.7878, reader_cost: 0.00016, ips: 1.2694 samples/sec | ETA 05:07:38 2021-05-09 18:25:05 [INFO] [TRAIN] epoch: 45, iter: 16580/40000, loss: 0.3576, lr: 0.006215, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2719 samples/sec | ETA 05:06:53 2021-05-09 18:25:13 [INFO] [TRAIN] epoch: 45, iter: 16590/40000, loss: 0.1686, lr: 0.006213, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2703 samples/sec | ETA 05:07:09 2021-05-09 18:25:20 [INFO] [TRAIN] epoch: 45, iter: 16600/40000, loss: 0.2506, lr: 0.006211, batch_cost: 0.7851, reader_cost: 0.00016, ips: 1.2738 samples/sec | ETA 05:06:10 2021-05-09 18:25:28 [INFO] [TRAIN] epoch: 45, iter: 16610/40000, loss: 0.2797, lr: 0.006208, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 05:06:17 2021-05-09 18:25:36 [INFO] [TRAIN] epoch: 45, iter: 16620/40000, loss: 0.3145, lr: 0.006206, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 05:06:24 2021-05-09 18:25:44 [INFO] [TRAIN] epoch: 45, iter: 16630/40000, loss: 0.4234, lr: 0.006204, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2711 samples/sec | ETA 05:06:25 2021-05-09 18:25:52 [INFO] [TRAIN] epoch: 45, iter: 16640/40000, loss: 0.2888, lr: 0.006201, batch_cost: 0.7851, reader_cost: 0.00015, ips: 1.2738 samples/sec | ETA 05:05:38 2021-05-09 18:26:00 [INFO] [TRAIN] epoch: 45, iter: 16650/40000, loss: 0.4635, lr: 0.006199, batch_cost: 0.7838, reader_cost: 0.00015, ips: 1.2759 samples/sec | ETA 05:05:00 2021-05-09 18:26:08 [INFO] [TRAIN] epoch: 45, iter: 16660/40000, loss: 0.4369, lr: 0.006197, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2730 samples/sec | ETA 05:05:35 2021-05-09 18:26:15 [INFO] [TRAIN] epoch: 45, iter: 16670/40000, loss: 0.4740, lr: 0.006194, batch_cost: 0.7851, reader_cost: 0.00016, ips: 1.2737 samples/sec | ETA 05:05:16 2021-05-09 18:26:23 [INFO] [TRAIN] epoch: 45, iter: 16680/40000, loss: 0.3904, lr: 0.006192, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 05:05:22 2021-05-09 18:26:31 [INFO] [TRAIN] epoch: 45, iter: 16690/40000, loss: 0.3710, lr: 0.006190, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2736 samples/sec | ETA 05:05:02 2021-05-09 18:26:39 [INFO] [TRAIN] epoch: 45, iter: 16700/40000, loss: 0.1702, lr: 0.006187, batch_cost: 0.7876, reader_cost: 0.00017, ips: 1.2696 samples/sec | ETA 05:05:51 2021-05-09 18:26:47 [INFO] [TRAIN] epoch: 45, iter: 16710/40000, loss: 0.4296, lr: 0.006185, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2719 samples/sec | ETA 05:05:11 2021-05-09 18:26:55 [INFO] [TRAIN] epoch: 45, iter: 16720/40000, loss: 0.3432, lr: 0.006183, batch_cost: 0.7850, reader_cost: 0.00016, ips: 1.2738 samples/sec | ETA 05:04:35 2021-05-09 18:27:03 [INFO] [TRAIN] epoch: 45, iter: 16730/40000, loss: 0.4601, lr: 0.006180, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2731 samples/sec | ETA 05:04:38 2021-05-09 18:27:10 [INFO] [TRAIN] epoch: 45, iter: 16740/40000, loss: 0.3411, lr: 0.006178, batch_cost: 0.7842, reader_cost: 0.00010, ips: 1.2752 samples/sec | ETA 05:04:00 2021-05-09 18:27:21 [INFO] [TRAIN] epoch: 46, iter: 16750/40000, loss: 0.2900, lr: 0.006175, batch_cost: 1.1024, reader_cost: 0.26067, ips: 0.9071 samples/sec | ETA 07:07:10 2021-05-09 18:27:29 [INFO] [TRAIN] epoch: 46, iter: 16760/40000, loss: 0.6415, lr: 0.006173, batch_cost: 0.7909, reader_cost: 0.00033, ips: 1.2644 samples/sec | ETA 05:06:20 2021-05-09 18:27:37 [INFO] [TRAIN] epoch: 46, iter: 16770/40000, loss: 0.2589, lr: 0.006171, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 05:04:21 2021-05-09 18:27:45 [INFO] [TRAIN] epoch: 46, iter: 16780/40000, loss: 0.4231, lr: 0.006168, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2707 samples/sec | ETA 05:04:33 2021-05-09 18:27:53 [INFO] [TRAIN] epoch: 46, iter: 16790/40000, loss: 0.4921, lr: 0.006166, batch_cost: 0.7873, reader_cost: 0.00017, ips: 1.2702 samples/sec | ETA 05:04:33 2021-05-09 18:28:01 [INFO] [TRAIN] epoch: 46, iter: 16800/40000, loss: 0.6323, lr: 0.006164, batch_cost: 0.7877, reader_cost: 0.00016, ips: 1.2695 samples/sec | ETA 05:04:35 2021-05-09 18:28:09 [INFO] [TRAIN] epoch: 46, iter: 16810/40000, loss: 0.1873, lr: 0.006161, batch_cost: 0.7884, reader_cost: 0.00015, ips: 1.2684 samples/sec | ETA 05:04:42 2021-05-09 18:28:17 [INFO] [TRAIN] epoch: 46, iter: 16820/40000, loss: 0.2400, lr: 0.006159, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2710 samples/sec | ETA 05:03:58 2021-05-09 18:28:24 [INFO] [TRAIN] epoch: 46, iter: 16830/40000, loss: 0.3930, lr: 0.006157, batch_cost: 0.7882, reader_cost: 0.00016, ips: 1.2688 samples/sec | ETA 05:04:21 2021-05-09 18:28:32 [INFO] [TRAIN] epoch: 46, iter: 16840/40000, loss: 0.2042, lr: 0.006154, batch_cost: 0.7880, reader_cost: 0.00016, ips: 1.2691 samples/sec | ETA 05:04:09 2021-05-09 18:28:40 [INFO] [TRAIN] epoch: 46, iter: 16850/40000, loss: 0.3608, lr: 0.006152, batch_cost: 0.7853, reader_cost: 0.00014, ips: 1.2734 samples/sec | ETA 05:02:59 2021-05-09 18:28:48 [INFO] [TRAIN] epoch: 46, iter: 16860/40000, loss: 0.1383, lr: 0.006150, batch_cost: 0.7880, reader_cost: 0.00015, ips: 1.2691 samples/sec | ETA 05:03:53 2021-05-09 18:28:56 [INFO] [TRAIN] epoch: 46, iter: 16870/40000, loss: 0.4147, lr: 0.006147, batch_cost: 0.7882, reader_cost: 0.00015, ips: 1.2687 samples/sec | ETA 05:03:51 2021-05-09 18:29:04 [INFO] [TRAIN] epoch: 46, iter: 16880/40000, loss: 0.3960, lr: 0.006145, batch_cost: 0.7866, reader_cost: 0.00014, ips: 1.2713 samples/sec | ETA 05:03:05 2021-05-09 18:29:12 [INFO] [TRAIN] epoch: 46, iter: 16890/40000, loss: 0.3967, lr: 0.006143, batch_cost: 0.7870, reader_cost: 0.00015, ips: 1.2706 samples/sec | ETA 05:03:07 2021-05-09 18:29:20 [INFO] [TRAIN] epoch: 46, iter: 16900/40000, loss: 0.2251, lr: 0.006140, batch_cost: 0.7871, reader_cost: 0.00015, ips: 1.2705 samples/sec | ETA 05:03:02 2021-05-09 18:29:27 [INFO] [TRAIN] epoch: 46, iter: 16910/40000, loss: 0.2461, lr: 0.006138, batch_cost: 0.7845, reader_cost: 0.00015, ips: 1.2747 samples/sec | ETA 05:01:53 2021-05-09 18:29:35 [INFO] [TRAIN] epoch: 46, iter: 16920/40000, loss: 0.2956, lr: 0.006135, batch_cost: 0.7891, reader_cost: 0.00017, ips: 1.2673 samples/sec | ETA 05:03:32 2021-05-09 18:29:43 [INFO] [TRAIN] epoch: 46, iter: 16930/40000, loss: 0.1830, lr: 0.006133, batch_cost: 0.7870, reader_cost: 0.00015, ips: 1.2706 samples/sec | ETA 05:02:36 2021-05-09 18:29:51 [INFO] [TRAIN] epoch: 46, iter: 16940/40000, loss: 0.1105, lr: 0.006131, batch_cost: 0.7851, reader_cost: 0.00015, ips: 1.2737 samples/sec | ETA 05:01:45 2021-05-09 18:29:59 [INFO] [TRAIN] epoch: 46, iter: 16950/40000, loss: 0.3294, lr: 0.006128, batch_cost: 0.7867, reader_cost: 0.00015, ips: 1.2712 samples/sec | ETA 05:02:12 2021-05-09 18:30:07 [INFO] [TRAIN] epoch: 46, iter: 16960/40000, loss: 0.2231, lr: 0.006126, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2709 samples/sec | ETA 05:02:09 2021-05-09 18:30:15 [INFO] [TRAIN] epoch: 46, iter: 16970/40000, loss: 0.1818, lr: 0.006124, batch_cost: 0.7874, reader_cost: 0.00015, ips: 1.2701 samples/sec | ETA 05:02:13 2021-05-09 18:30:22 [INFO] [TRAIN] epoch: 46, iter: 16980/40000, loss: 0.3866, lr: 0.006121, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2727 samples/sec | ETA 05:01:28 2021-05-09 18:30:30 [INFO] [TRAIN] epoch: 46, iter: 16990/40000, loss: 0.2368, lr: 0.006119, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2723 samples/sec | ETA 05:01:25 2021-05-09 18:30:38 [INFO] [TRAIN] epoch: 46, iter: 17000/40000, loss: 0.2597, lr: 0.006117, batch_cost: 0.7858, reader_cost: 0.00015, ips: 1.2726 samples/sec | ETA 05:01:13 2021-05-09 18:30:38 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-09 18:34:10 [INFO] [EVAL] #Images: 500 mIoU: 0.7526 Acc: 0.9544 Kappa: 0.9409 2021-05-09 18:34:10 [INFO] [EVAL] Class IoU: [0.9791 0.8335 0.9161 0.6214 0.6045 0.4824 0.6041 0.7112 0.912 0.6303 0.9395 0.7651 0.5551 0.9384 0.7991 0.8848 0.7934 0.6066 0.7225] 2021-05-09 18:34:10 [INFO] [EVAL] Class Acc: [0.9919 0.8979 0.9546 0.8825 0.7506 0.739 0.7976 0.8707 0.9415 0.8689 0.9635 0.8361 0.7801 0.9606 0.8648 0.9487 0.8889 0.8156 0.8231] 2021-05-09 18:34:38 [INFO] [EVAL] The model with the best validation mIoU (0.7582) was saved at iter 16000. 2021-05-09 18:34:46 [INFO] [TRAIN] epoch: 46, iter: 17010/40000, loss: 0.2347, lr: 0.006114, batch_cost: 0.7833, reader_cost: 0.00025, ips: 1.2766 samples/sec | ETA 05:00:08 2021-05-09 18:34:54 [INFO] [TRAIN] epoch: 46, iter: 17020/40000, loss: 0.4157, lr: 0.006112, batch_cost: 0.7837, reader_cost: 0.00016, ips: 1.2760 samples/sec | ETA 05:00:09 2021-05-09 18:35:02 [INFO] [TRAIN] epoch: 46, iter: 17030/40000, loss: 0.3547, lr: 0.006110, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 05:00:42 2021-05-09 18:35:10 [INFO] [TRAIN] epoch: 46, iter: 17040/40000, loss: 0.4255, lr: 0.006107, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2716 samples/sec | ETA 05:00:56 2021-05-09 18:35:17 [INFO] [TRAIN] epoch: 46, iter: 17050/40000, loss: 0.6109, lr: 0.006105, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2726 samples/sec | ETA 05:00:34 2021-05-09 18:35:25 [INFO] [TRAIN] epoch: 46, iter: 17060/40000, loss: 0.5456, lr: 0.006103, batch_cost: 0.7875, reader_cost: 0.00017, ips: 1.2699 samples/sec | ETA 05:01:04 2021-05-09 18:35:33 [INFO] [TRAIN] epoch: 46, iter: 17070/40000, loss: 0.0846, lr: 0.006100, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2720 samples/sec | ETA 05:00:26 2021-05-09 18:35:41 [INFO] [TRAIN] epoch: 46, iter: 17080/40000, loss: 0.3336, lr: 0.006098, batch_cost: 0.7863, reader_cost: 0.00017, ips: 1.2718 samples/sec | ETA 05:00:21 2021-05-09 18:35:49 [INFO] [TRAIN] epoch: 46, iter: 17090/40000, loss: 0.3235, lr: 0.006095, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2728 samples/sec | ETA 04:59:59 2021-05-09 18:35:57 [INFO] [TRAIN] epoch: 46, iter: 17100/40000, loss: 0.4172, lr: 0.006093, batch_cost: 0.7853, reader_cost: 0.00015, ips: 1.2734 samples/sec | ETA 04:59:43 2021-05-09 18:36:05 [INFO] [TRAIN] epoch: 46, iter: 17110/40000, loss: 0.3414, lr: 0.006091, batch_cost: 0.7848, reader_cost: 0.00010, ips: 1.2742 samples/sec | ETA 04:59:24 2021-05-09 18:36:16 [INFO] [TRAIN] epoch: 47, iter: 17120/40000, loss: 0.7039, lr: 0.006088, batch_cost: 1.0985, reader_cost: 0.26231, ips: 0.9104 samples/sec | ETA 06:58:52 2021-05-09 18:36:23 [INFO] [TRAIN] epoch: 47, iter: 17130/40000, loss: 0.5396, lr: 0.006086, batch_cost: 0.7940, reader_cost: 0.00032, ips: 1.2595 samples/sec | ETA 05:02:37 2021-05-09 18:36:31 [INFO] [TRAIN] epoch: 47, iter: 17140/40000, loss: 0.4263, lr: 0.006084, batch_cost: 0.7863, reader_cost: 0.00017, ips: 1.2718 samples/sec | ETA 04:59:34 2021-05-09 18:36:39 [INFO] [TRAIN] epoch: 47, iter: 17150/40000, loss: 0.3359, lr: 0.006081, batch_cost: 0.7870, reader_cost: 0.00018, ips: 1.2706 samples/sec | ETA 04:59:43 2021-05-09 18:36:47 [INFO] [TRAIN] epoch: 47, iter: 17160/40000, loss: 0.5578, lr: 0.006079, batch_cost: 0.7861, reader_cost: 0.00017, ips: 1.2720 samples/sec | ETA 04:59:15 2021-05-09 18:36:55 [INFO] [TRAIN] epoch: 47, iter: 17170/40000, loss: 0.3608, lr: 0.006077, batch_cost: 0.7869, reader_cost: 0.00015, ips: 1.2707 samples/sec | ETA 04:59:25 2021-05-09 18:37:03 [INFO] [TRAIN] epoch: 47, iter: 17180/40000, loss: 0.1778, lr: 0.006074, batch_cost: 0.7872, reader_cost: 0.00017, ips: 1.2704 samples/sec | ETA 04:59:23 2021-05-09 18:37:11 [INFO] [TRAIN] epoch: 47, iter: 17190/40000, loss: 0.1229, lr: 0.006072, batch_cost: 0.7853, reader_cost: 0.00015, ips: 1.2734 samples/sec | ETA 04:58:32 2021-05-09 18:37:19 [INFO] [TRAIN] epoch: 47, iter: 17200/40000, loss: 0.3515, lr: 0.006070, batch_cost: 0.7858, reader_cost: 0.00015, ips: 1.2726 samples/sec | ETA 04:58:35 2021-05-09 18:37:26 [INFO] [TRAIN] epoch: 47, iter: 17210/40000, loss: 0.3982, lr: 0.006067, batch_cost: 0.7874, reader_cost: 0.00015, ips: 1.2700 samples/sec | ETA 04:59:05 2021-05-09 18:37:34 [INFO] [TRAIN] epoch: 47, iter: 17220/40000, loss: 0.2432, lr: 0.006065, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 04:58:33 2021-05-09 18:37:42 [INFO] [TRAIN] epoch: 47, iter: 17230/40000, loss: 0.2253, lr: 0.006062, batch_cost: 0.7860, reader_cost: 0.00018, ips: 1.2723 samples/sec | ETA 04:58:17 2021-05-09 18:37:50 [INFO] [TRAIN] epoch: 47, iter: 17240/40000, loss: 0.2421, lr: 0.006060, batch_cost: 0.7855, reader_cost: 0.00017, ips: 1.2731 samples/sec | ETA 04:57:56 2021-05-09 18:37:58 [INFO] [TRAIN] epoch: 47, iter: 17250/40000, loss: 0.4425, lr: 0.006058, batch_cost: 0.7880, reader_cost: 0.00016, ips: 1.2691 samples/sec | ETA 04:58:46 2021-05-09 18:38:06 [INFO] [TRAIN] epoch: 47, iter: 17260/40000, loss: 0.3790, lr: 0.006055, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2727 samples/sec | ETA 04:57:46 2021-05-09 18:38:14 [INFO] [TRAIN] epoch: 47, iter: 17270/40000, loss: 0.2713, lr: 0.006053, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 04:57:47 2021-05-09 18:38:21 [INFO] [TRAIN] epoch: 47, iter: 17280/40000, loss: 0.2051, lr: 0.006051, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2732 samples/sec | ETA 04:57:25 2021-05-09 18:38:29 [INFO] [TRAIN] epoch: 47, iter: 17290/40000, loss: 0.2220, lr: 0.006048, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2736 samples/sec | ETA 04:57:10 2021-05-09 18:38:37 [INFO] [TRAIN] epoch: 47, iter: 17300/40000, loss: 0.2163, lr: 0.006046, batch_cost: 0.7870, reader_cost: 0.00017, ips: 1.2706 samples/sec | ETA 04:57:45 2021-05-09 18:38:45 [INFO] [TRAIN] epoch: 47, iter: 17310/40000, loss: 0.1389, lr: 0.006044, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 04:57:15 2021-05-09 18:38:53 [INFO] [TRAIN] epoch: 47, iter: 17320/40000, loss: 0.2407, lr: 0.006041, batch_cost: 0.7864, reader_cost: 0.00017, ips: 1.2716 samples/sec | ETA 04:57:15 2021-05-09 18:39:01 [INFO] [TRAIN] epoch: 47, iter: 17330/40000, loss: 0.2858, lr: 0.006039, batch_cost: 0.7847, reader_cost: 0.00017, ips: 1.2743 samples/sec | ETA 04:56:29 2021-05-09 18:39:09 [INFO] [TRAIN] epoch: 47, iter: 17340/40000, loss: 0.1785, lr: 0.006037, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2703 samples/sec | ETA 04:57:18 2021-05-09 18:39:16 [INFO] [TRAIN] epoch: 47, iter: 17350/40000, loss: 0.2760, lr: 0.006034, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 04:56:30 2021-05-09 18:39:24 [INFO] [TRAIN] epoch: 47, iter: 17360/40000, loss: 0.2403, lr: 0.006032, batch_cost: 0.7847, reader_cost: 0.00016, ips: 1.2744 samples/sec | ETA 04:56:04 2021-05-09 18:39:32 [INFO] [TRAIN] epoch: 47, iter: 17370/40000, loss: 0.3593, lr: 0.006029, batch_cost: 0.7845, reader_cost: 0.00016, ips: 1.2747 samples/sec | ETA 04:55:52 2021-05-09 18:39:40 [INFO] [TRAIN] epoch: 47, iter: 17380/40000, loss: 0.1902, lr: 0.006027, batch_cost: 0.7844, reader_cost: 0.00016, ips: 1.2748 samples/sec | ETA 04:55:44 2021-05-09 18:39:48 [INFO] [TRAIN] epoch: 47, iter: 17390/40000, loss: 0.4419, lr: 0.006025, batch_cost: 0.7853, reader_cost: 0.00015, ips: 1.2735 samples/sec | ETA 04:55:54 2021-05-09 18:39:56 [INFO] [TRAIN] epoch: 47, iter: 17400/40000, loss: 0.4593, lr: 0.006022, batch_cost: 0.7844, reader_cost: 0.00016, ips: 1.2748 samples/sec | ETA 04:55:28 2021-05-09 18:40:04 [INFO] [TRAIN] epoch: 47, iter: 17410/40000, loss: 0.4368, lr: 0.006020, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 04:55:53 2021-05-09 18:40:11 [INFO] [TRAIN] epoch: 47, iter: 17420/40000, loss: 0.4697, lr: 0.006018, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2708 samples/sec | ETA 04:56:07 2021-05-09 18:40:19 [INFO] [TRAIN] epoch: 47, iter: 17430/40000, loss: 0.4694, lr: 0.006015, batch_cost: 0.7848, reader_cost: 0.00016, ips: 1.2742 samples/sec | ETA 04:55:13 2021-05-09 18:40:27 [INFO] [TRAIN] epoch: 47, iter: 17440/40000, loss: 0.2955, lr: 0.006013, batch_cost: 0.7852, reader_cost: 0.00018, ips: 1.2735 samples/sec | ETA 04:55:14 2021-05-09 18:40:35 [INFO] [TRAIN] epoch: 47, iter: 17450/40000, loss: 0.2085, lr: 0.006011, batch_cost: 0.7842, reader_cost: 0.00017, ips: 1.2751 samples/sec | ETA 04:54:44 2021-05-09 18:40:43 [INFO] [TRAIN] epoch: 47, iter: 17460/40000, loss: 0.2455, lr: 0.006008, batch_cost: 0.7856, reader_cost: 0.00019, ips: 1.2728 samples/sec | ETA 04:55:08 2021-05-09 18:40:51 [INFO] [TRAIN] epoch: 47, iter: 17470/40000, loss: 0.4181, lr: 0.006006, batch_cost: 0.7843, reader_cost: 0.00017, ips: 1.2750 samples/sec | ETA 04:54:30 2021-05-09 18:40:58 [INFO] [TRAIN] epoch: 47, iter: 17480/40000, loss: 0.3211, lr: 0.006004, batch_cost: 0.7844, reader_cost: 0.00012, ips: 1.2748 samples/sec | ETA 04:54:24 2021-05-09 18:41:09 [INFO] [TRAIN] epoch: 48, iter: 17490/40000, loss: 0.2914, lr: 0.006001, batch_cost: 1.0889, reader_cost: 0.25695, ips: 0.9183 samples/sec | ETA 06:48:32 2021-05-09 18:41:17 [INFO] [TRAIN] epoch: 48, iter: 17500/40000, loss: 0.2863, lr: 0.005999, batch_cost: 0.7964, reader_cost: 0.00033, ips: 1.2557 samples/sec | ETA 04:58:38 2021-05-09 18:41:25 [INFO] [TRAIN] epoch: 48, iter: 17510/40000, loss: 0.5145, lr: 0.005996, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2718 samples/sec | ETA 04:54:43 2021-05-09 18:41:33 [INFO] [TRAIN] epoch: 48, iter: 17520/40000, loss: 0.4064, lr: 0.005994, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2718 samples/sec | ETA 04:54:35 2021-05-09 18:41:41 [INFO] [TRAIN] epoch: 48, iter: 17530/40000, loss: 0.4675, lr: 0.005992, batch_cost: 0.7854, reader_cost: 0.00018, ips: 1.2733 samples/sec | ETA 04:54:07 2021-05-09 18:41:49 [INFO] [TRAIN] epoch: 48, iter: 17540/40000, loss: 0.4673, lr: 0.005989, batch_cost: 0.7874, reader_cost: 0.00016, ips: 1.2699 samples/sec | ETA 04:54:45 2021-05-09 18:41:57 [INFO] [TRAIN] epoch: 48, iter: 17550/40000, loss: 0.3125, lr: 0.005987, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2733 samples/sec | ETA 04:53:50 2021-05-09 18:42:05 [INFO] [TRAIN] epoch: 48, iter: 17560/40000, loss: 0.1808, lr: 0.005985, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 04:53:51 2021-05-09 18:42:12 [INFO] [TRAIN] epoch: 48, iter: 17570/40000, loss: 0.3657, lr: 0.005982, batch_cost: 0.7851, reader_cost: 0.00015, ips: 1.2737 samples/sec | ETA 04:53:30 2021-05-09 18:42:20 [INFO] [TRAIN] epoch: 48, iter: 17580/40000, loss: 0.3383, lr: 0.005980, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2720 samples/sec | ETA 04:53:45 2021-05-09 18:42:28 [INFO] [TRAIN] epoch: 48, iter: 17590/40000, loss: 0.3074, lr: 0.005978, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 04:53:28 2021-05-09 18:42:36 [INFO] [TRAIN] epoch: 48, iter: 17600/40000, loss: 0.2213, lr: 0.005975, batch_cost: 0.7857, reader_cost: 0.00014, ips: 1.2728 samples/sec | ETA 04:53:18 2021-05-09 18:42:44 [INFO] [TRAIN] epoch: 48, iter: 17610/40000, loss: 0.2437, lr: 0.005973, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2720 samples/sec | ETA 04:53:21 2021-05-09 18:42:52 [INFO] [TRAIN] epoch: 48, iter: 17620/40000, loss: 0.4137, lr: 0.005970, batch_cost: 0.7842, reader_cost: 0.00015, ips: 1.2752 samples/sec | ETA 04:52:30 2021-05-09 18:43:00 [INFO] [TRAIN] epoch: 48, iter: 17630/40000, loss: 0.3062, lr: 0.005968, batch_cost: 0.7881, reader_cost: 0.00016, ips: 1.2689 samples/sec | ETA 04:53:48 2021-05-09 18:43:07 [INFO] [TRAIN] epoch: 48, iter: 17640/40000, loss: 0.2800, lr: 0.005966, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2729 samples/sec | ETA 04:52:45 2021-05-09 18:43:15 [INFO] [TRAIN] epoch: 48, iter: 17650/40000, loss: 0.2550, lr: 0.005963, batch_cost: 0.7850, reader_cost: 0.00017, ips: 1.2739 samples/sec | ETA 04:52:25 2021-05-09 18:43:23 [INFO] [TRAIN] epoch: 48, iter: 17660/40000, loss: 0.1651, lr: 0.005961, batch_cost: 0.7862, reader_cost: 0.00017, ips: 1.2719 samples/sec | ETA 04:52:44 2021-05-09 18:43:31 [INFO] [TRAIN] epoch: 48, iter: 17670/40000, loss: 0.2735, lr: 0.005959, batch_cost: 0.7847, reader_cost: 0.00016, ips: 1.2744 samples/sec | ETA 04:52:01 2021-05-09 18:43:39 [INFO] [TRAIN] epoch: 48, iter: 17680/40000, loss: 0.1021, lr: 0.005956, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2715 samples/sec | ETA 04:52:33 2021-05-09 18:43:47 [INFO] [TRAIN] epoch: 48, iter: 17690/40000, loss: 0.3200, lr: 0.005954, batch_cost: 0.7870, reader_cost: 0.00015, ips: 1.2706 samples/sec | ETA 04:52:38 2021-05-09 18:43:55 [INFO] [TRAIN] epoch: 48, iter: 17700/40000, loss: 0.2803, lr: 0.005952, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2728 samples/sec | ETA 04:52:00 2021-05-09 18:44:02 [INFO] [TRAIN] epoch: 48, iter: 17710/40000, loss: 0.2588, lr: 0.005949, batch_cost: 0.7840, reader_cost: 0.00015, ips: 1.2755 samples/sec | ETA 04:51:15 2021-05-09 18:44:10 [INFO] [TRAIN] epoch: 48, iter: 17720/40000, loss: 0.3851, lr: 0.005947, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 04:51:52 2021-05-09 18:44:18 [INFO] [TRAIN] epoch: 48, iter: 17730/40000, loss: 0.2153, lr: 0.005944, batch_cost: 0.7860, reader_cost: 0.00014, ips: 1.2722 samples/sec | ETA 04:51:44 2021-05-09 18:44:26 [INFO] [TRAIN] epoch: 48, iter: 17740/40000, loss: 0.2708, lr: 0.005942, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2729 samples/sec | ETA 04:51:28 2021-05-09 18:44:34 [INFO] [TRAIN] epoch: 48, iter: 17750/40000, loss: 0.1270, lr: 0.005940, batch_cost: 0.7854, reader_cost: 0.00017, ips: 1.2733 samples/sec | ETA 04:51:14 2021-05-09 18:44:42 [INFO] [TRAIN] epoch: 48, iter: 17760/40000, loss: 0.3562, lr: 0.005937, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2728 samples/sec | ETA 04:51:13 2021-05-09 18:44:50 [INFO] [TRAIN] epoch: 48, iter: 17770/40000, loss: 0.4515, lr: 0.005935, batch_cost: 0.7849, reader_cost: 0.00017, ips: 1.2740 samples/sec | ETA 04:50:48 2021-05-09 18:44:57 [INFO] [TRAIN] epoch: 48, iter: 17780/40000, loss: 0.3693, lr: 0.005933, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2723 samples/sec | ETA 04:51:04 2021-05-09 18:45:05 [INFO] [TRAIN] epoch: 48, iter: 17790/40000, loss: 0.4593, lr: 0.005930, batch_cost: 0.7844, reader_cost: 0.00016, ips: 1.2748 samples/sec | ETA 04:50:22 2021-05-09 18:45:13 [INFO] [TRAIN] epoch: 48, iter: 17800/40000, loss: 0.3452, lr: 0.005928, batch_cost: 0.7841, reader_cost: 0.00016, ips: 1.2753 samples/sec | ETA 04:50:07 2021-05-09 18:45:21 [INFO] [TRAIN] epoch: 48, iter: 17810/40000, loss: 0.2826, lr: 0.005926, batch_cost: 0.7850, reader_cost: 0.00016, ips: 1.2738 samples/sec | ETA 04:50:19 2021-05-09 18:45:29 [INFO] [TRAIN] epoch: 48, iter: 17820/40000, loss: 0.1685, lr: 0.005923, batch_cost: 0.7851, reader_cost: 0.00015, ips: 1.2737 samples/sec | ETA 04:50:13 2021-05-09 18:45:37 [INFO] [TRAIN] epoch: 48, iter: 17830/40000, loss: 0.2895, lr: 0.005921, batch_cost: 0.7846, reader_cost: 0.00017, ips: 1.2746 samples/sec | ETA 04:49:54 2021-05-09 18:45:44 [INFO] [TRAIN] epoch: 48, iter: 17840/40000, loss: 0.3771, lr: 0.005919, batch_cost: 0.7846, reader_cost: 0.00016, ips: 1.2745 samples/sec | ETA 04:49:47 2021-05-09 18:45:52 [INFO] [TRAIN] epoch: 48, iter: 17850/40000, loss: 0.3271, lr: 0.005916, batch_cost: 0.7853, reader_cost: 0.00013, ips: 1.2734 samples/sec | ETA 04:49:54 2021-05-09 18:46:03 [INFO] [TRAIN] epoch: 49, iter: 17860/40000, loss: 0.2676, lr: 0.005914, batch_cost: 1.1024, reader_cost: 0.24832, ips: 0.9071 samples/sec | ETA 06:46:47 2021-05-09 18:46:11 [INFO] [TRAIN] epoch: 49, iter: 17870/40000, loss: 0.3167, lr: 0.005911, batch_cost: 0.7948, reader_cost: 0.00033, ips: 1.2582 samples/sec | ETA 04:53:08 2021-05-09 18:46:19 [INFO] [TRAIN] epoch: 49, iter: 17880/40000, loss: 0.5458, lr: 0.005909, batch_cost: 0.7848, reader_cost: 0.00016, ips: 1.2742 samples/sec | ETA 04:49:19 2021-05-09 18:46:27 [INFO] [TRAIN] epoch: 49, iter: 17890/40000, loss: 0.2252, lr: 0.005907, batch_cost: 0.7861, reader_cost: 0.00017, ips: 1.2720 samples/sec | ETA 04:49:41 2021-05-09 18:46:35 [INFO] [TRAIN] epoch: 49, iter: 17900/40000, loss: 0.4838, lr: 0.005904, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2736 samples/sec | ETA 04:49:12 2021-05-09 18:46:43 [INFO] [TRAIN] epoch: 49, iter: 17910/40000, loss: 0.4291, lr: 0.005902, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 04:49:36 2021-05-09 18:46:51 [INFO] [TRAIN] epoch: 49, iter: 17920/40000, loss: 0.3287, lr: 0.005900, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2733 samples/sec | ETA 04:49:01 2021-05-09 18:46:58 [INFO] [TRAIN] epoch: 49, iter: 17930/40000, loss: 0.1026, lr: 0.005897, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2730 samples/sec | ETA 04:48:56 2021-05-09 18:47:06 [INFO] [TRAIN] epoch: 49, iter: 17940/40000, loss: 0.2578, lr: 0.005895, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2735 samples/sec | ETA 04:48:41 2021-05-09 18:47:14 [INFO] [TRAIN] epoch: 49, iter: 17950/40000, loss: 0.3299, lr: 0.005893, batch_cost: 0.7869, reader_cost: 0.00015, ips: 1.2707 samples/sec | ETA 04:49:12 2021-05-09 18:47:22 [INFO] [TRAIN] epoch: 49, iter: 17960/40000, loss: 0.2449, lr: 0.005890, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2725 samples/sec | ETA 04:48:40 2021-05-09 18:47:30 [INFO] [TRAIN] epoch: 49, iter: 17970/40000, loss: 0.2999, lr: 0.005888, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2734 samples/sec | ETA 04:48:20 2021-05-09 18:47:38 [INFO] [TRAIN] epoch: 49, iter: 17980/40000, loss: 0.1159, lr: 0.005885, batch_cost: 0.7874, reader_cost: 0.00015, ips: 1.2699 samples/sec | ETA 04:48:59 2021-05-09 18:47:46 [INFO] [TRAIN] epoch: 49, iter: 17990/40000, loss: 0.4011, lr: 0.005883, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2715 samples/sec | ETA 04:48:30 2021-05-09 18:47:53 [INFO] [TRAIN] epoch: 49, iter: 18000/40000, loss: 0.2795, lr: 0.005881, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 04:48:00 2021-05-09 18:47:54 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-09 18:51:25 [INFO] [EVAL] #Images: 500 mIoU: 0.7549 Acc: 0.9551 Kappa: 0.9417 2021-05-09 18:51:25 [INFO] [EVAL] Class IoU: [0.9801 0.8397 0.9145 0.6299 0.5935 0.4841 0.6048 0.7142 0.9131 0.6386 0.9398 0.7724 0.5674 0.9398 0.8311 0.8777 0.762 0.6154 0.7255] 2021-05-09 18:51:25 [INFO] [EVAL] Class Acc: [0.9918 0.908 0.9411 0.8798 0.8174 0.7565 0.8202 0.8757 0.9502 0.8373 0.9691 0.8602 0.7561 0.9654 0.9009 0.937 0.9125 0.7844 0.8165] 2021-05-09 18:51:53 [INFO] [EVAL] The model with the best validation mIoU (0.7582) was saved at iter 16000. 2021-05-09 18:52:01 [INFO] [TRAIN] epoch: 49, iter: 18010/40000, loss: 0.3689, lr: 0.005878, batch_cost: 0.7847, reader_cost: 0.00023, ips: 1.2744 samples/sec | ETA 04:47:35 2021-05-09 18:52:10 [INFO] [TRAIN] epoch: 49, iter: 18020/40000, loss: 0.3313, lr: 0.005876, batch_cost: 0.7849, reader_cost: 0.00017, ips: 1.2741 samples/sec | ETA 04:47:31 2021-05-09 18:52:17 [INFO] [TRAIN] epoch: 49, iter: 18030/40000, loss: 0.2986, lr: 0.005874, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2729 samples/sec | ETA 04:47:39 2021-05-09 18:52:25 [INFO] [TRAIN] epoch: 49, iter: 18040/40000, loss: 0.2673, lr: 0.005871, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 04:47:47 2021-05-09 18:52:33 [INFO] [TRAIN] epoch: 49, iter: 18050/40000, loss: 0.1214, lr: 0.005869, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2714 samples/sec | ETA 04:47:44 2021-05-09 18:52:41 [INFO] [TRAIN] epoch: 49, iter: 18060/40000, loss: 0.2146, lr: 0.005866, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2704 samples/sec | ETA 04:47:49 2021-05-09 18:52:49 [INFO] [TRAIN] epoch: 49, iter: 18070/40000, loss: 0.2914, lr: 0.005864, batch_cost: 0.7874, reader_cost: 0.00015, ips: 1.2699 samples/sec | ETA 04:47:48 2021-05-09 18:52:57 [INFO] [TRAIN] epoch: 49, iter: 18080/40000, loss: 0.1695, lr: 0.005862, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2720 samples/sec | ETA 04:47:12 2021-05-09 18:53:05 [INFO] [TRAIN] epoch: 49, iter: 18090/40000, loss: 0.3029, lr: 0.005859, batch_cost: 0.7869, reader_cost: 0.00015, ips: 1.2709 samples/sec | ETA 04:47:20 2021-05-09 18:53:13 [INFO] [TRAIN] epoch: 49, iter: 18100/40000, loss: 0.1764, lr: 0.005857, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2725 samples/sec | ETA 04:46:50 2021-05-09 18:53:20 [INFO] [TRAIN] epoch: 49, iter: 18110/40000, loss: 0.2698, lr: 0.005855, batch_cost: 0.7880, reader_cost: 0.00016, ips: 1.2691 samples/sec | ETA 04:47:28 2021-05-09 18:53:28 [INFO] [TRAIN] epoch: 49, iter: 18120/40000, loss: 0.1684, lr: 0.005852, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2709 samples/sec | ETA 04:46:55 2021-05-09 18:53:36 [INFO] [TRAIN] epoch: 49, iter: 18130/40000, loss: 0.3220, lr: 0.005850, batch_cost: 0.7874, reader_cost: 0.00018, ips: 1.2700 samples/sec | ETA 04:47:00 2021-05-09 18:53:44 [INFO] [TRAIN] epoch: 49, iter: 18140/40000, loss: 0.4153, lr: 0.005848, batch_cost: 0.7879, reader_cost: 0.00015, ips: 1.2692 samples/sec | ETA 04:47:03 2021-05-09 18:53:52 [INFO] [TRAIN] epoch: 49, iter: 18150/40000, loss: 0.2701, lr: 0.005845, batch_cost: 0.7866, reader_cost: 0.00017, ips: 1.2713 samples/sec | ETA 04:46:26 2021-05-09 18:54:00 [INFO] [TRAIN] epoch: 49, iter: 18160/40000, loss: 0.5602, lr: 0.005843, batch_cost: 0.7861, reader_cost: 0.00017, ips: 1.2721 samples/sec | ETA 04:46:08 2021-05-09 18:54:08 [INFO] [TRAIN] epoch: 49, iter: 18170/40000, loss: 0.4975, lr: 0.005840, batch_cost: 0.7871, reader_cost: 0.00015, ips: 1.2705 samples/sec | ETA 04:46:21 2021-05-09 18:54:15 [INFO] [TRAIN] epoch: 49, iter: 18180/40000, loss: 0.3988, lr: 0.005838, batch_cost: 0.7873, reader_cost: 0.00014, ips: 1.2702 samples/sec | ETA 04:46:18 2021-05-09 18:54:23 [INFO] [TRAIN] epoch: 49, iter: 18190/40000, loss: 0.1807, lr: 0.005836, batch_cost: 0.7873, reader_cost: 0.00015, ips: 1.2702 samples/sec | ETA 04:46:10 2021-05-09 18:54:31 [INFO] [TRAIN] epoch: 49, iter: 18200/40000, loss: 0.2205, lr: 0.005833, batch_cost: 0.7884, reader_cost: 0.00018, ips: 1.2683 samples/sec | ETA 04:46:27 2021-05-09 18:54:39 [INFO] [TRAIN] epoch: 49, iter: 18210/40000, loss: 0.2259, lr: 0.005831, batch_cost: 0.7869, reader_cost: 0.00015, ips: 1.2708 samples/sec | ETA 04:45:46 2021-05-09 18:54:47 [INFO] [TRAIN] epoch: 49, iter: 18220/40000, loss: 0.3600, lr: 0.005829, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 04:45:16 2021-05-09 18:54:58 [INFO] [TRAIN] epoch: 50, iter: 18230/40000, loss: 0.4834, lr: 0.005826, batch_cost: 1.0909, reader_cost: 0.29096, ips: 0.9167 samples/sec | ETA 06:35:48 2021-05-09 18:55:06 [INFO] [TRAIN] epoch: 50, iter: 18240/40000, loss: 0.2328, lr: 0.005824, batch_cost: 0.7941, reader_cost: 0.00034, ips: 1.2594 samples/sec | ETA 04:47:58 2021-05-09 18:55:14 [INFO] [TRAIN] epoch: 50, iter: 18250/40000, loss: 0.6324, lr: 0.005822, batch_cost: 0.7875, reader_cost: 0.00016, ips: 1.2699 samples/sec | ETA 04:45:27 2021-05-09 18:55:22 [INFO] [TRAIN] epoch: 50, iter: 18260/40000, loss: 0.3210, lr: 0.005819, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2720 samples/sec | ETA 04:44:51 2021-05-09 18:55:29 [INFO] [TRAIN] epoch: 50, iter: 18270/40000, loss: 0.3088, lr: 0.005817, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2720 samples/sec | ETA 04:44:42 2021-05-09 18:55:37 [INFO] [TRAIN] epoch: 50, iter: 18280/40000, loss: 0.4665, lr: 0.005814, batch_cost: 0.7876, reader_cost: 0.00019, ips: 1.2697 samples/sec | ETA 04:45:06 2021-05-09 18:55:45 [INFO] [TRAIN] epoch: 50, iter: 18290/40000, loss: 0.2862, lr: 0.005812, batch_cost: 0.7877, reader_cost: 0.00016, ips: 1.2695 samples/sec | ETA 04:45:01 2021-05-09 18:55:53 [INFO] [TRAIN] epoch: 50, iter: 18300/40000, loss: 0.1591, lr: 0.005810, batch_cost: 0.7873, reader_cost: 0.00016, ips: 1.2702 samples/sec | ETA 04:44:44 2021-05-09 18:56:01 [INFO] [TRAIN] epoch: 50, iter: 18310/40000, loss: 0.3011, lr: 0.005807, batch_cost: 0.7876, reader_cost: 0.00015, ips: 1.2697 samples/sec | ETA 04:44:43 2021-05-09 18:56:09 [INFO] [TRAIN] epoch: 50, iter: 18320/40000, loss: 0.2485, lr: 0.005805, batch_cost: 0.7870, reader_cost: 0.00018, ips: 1.2706 samples/sec | ETA 04:44:22 2021-05-09 18:56:17 [INFO] [TRAIN] epoch: 50, iter: 18330/40000, loss: 0.2020, lr: 0.005803, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2729 samples/sec | ETA 04:43:44 2021-05-09 18:56:25 [INFO] [TRAIN] epoch: 50, iter: 18340/40000, loss: 0.3584, lr: 0.005800, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 04:43:42 2021-05-09 18:56:32 [INFO] [TRAIN] epoch: 50, iter: 18350/40000, loss: 0.1743, lr: 0.005798, batch_cost: 0.7875, reader_cost: 0.00015, ips: 1.2698 samples/sec | ETA 04:44:09 2021-05-09 18:56:40 [INFO] [TRAIN] epoch: 50, iter: 18360/40000, loss: 0.2757, lr: 0.005795, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2723 samples/sec | ETA 04:43:28 2021-05-09 18:56:48 [INFO] [TRAIN] epoch: 50, iter: 18370/40000, loss: 0.5447, lr: 0.005793, batch_cost: 0.7863, reader_cost: 0.00017, ips: 1.2718 samples/sec | ETA 04:43:26 2021-05-09 18:56:56 [INFO] [TRAIN] epoch: 50, iter: 18380/40000, loss: 0.4957, lr: 0.005791, batch_cost: 0.7873, reader_cost: 0.00015, ips: 1.2702 samples/sec | ETA 04:43:41 2021-05-09 18:57:04 [INFO] [TRAIN] epoch: 50, iter: 18390/40000, loss: 0.2846, lr: 0.005788, batch_cost: 0.7869, reader_cost: 0.00017, ips: 1.2707 samples/sec | ETA 04:43:25 2021-05-09 18:57:12 [INFO] [TRAIN] epoch: 50, iter: 18400/40000, loss: 0.2738, lr: 0.005786, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2731 samples/sec | ETA 04:42:46 2021-05-09 18:57:20 [INFO] [TRAIN] epoch: 50, iter: 18410/40000, loss: 0.3306, lr: 0.005784, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2734 samples/sec | ETA 04:42:35 2021-05-09 18:57:27 [INFO] [TRAIN] epoch: 50, iter: 18420/40000, loss: 0.1567, lr: 0.005781, batch_cost: 0.7880, reader_cost: 0.00015, ips: 1.2690 samples/sec | ETA 04:43:25 2021-05-09 18:57:35 [INFO] [TRAIN] epoch: 50, iter: 18430/40000, loss: 0.2739, lr: 0.005779, batch_cost: 0.7858, reader_cost: 0.00015, ips: 1.2726 samples/sec | ETA 04:42:30 2021-05-09 18:57:43 [INFO] [TRAIN] epoch: 50, iter: 18440/40000, loss: 0.3881, lr: 0.005777, batch_cost: 0.7878, reader_cost: 0.00018, ips: 1.2693 samples/sec | ETA 04:43:05 2021-05-09 18:57:51 [INFO] [TRAIN] epoch: 50, iter: 18450/40000, loss: 0.1912, lr: 0.005774, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2720 samples/sec | ETA 04:42:21 2021-05-09 18:57:59 [INFO] [TRAIN] epoch: 50, iter: 18460/40000, loss: 0.2854, lr: 0.005772, batch_cost: 0.7874, reader_cost: 0.00016, ips: 1.2700 samples/sec | ETA 04:42:40 2021-05-09 18:58:07 [INFO] [TRAIN] epoch: 50, iter: 18470/40000, loss: 0.2648, lr: 0.005769, batch_cost: 0.7857, reader_cost: 0.00017, ips: 1.2728 samples/sec | ETA 04:41:55 2021-05-09 18:58:15 [INFO] [TRAIN] epoch: 50, iter: 18480/40000, loss: 0.2121, lr: 0.005767, batch_cost: 0.7874, reader_cost: 0.00017, ips: 1.2700 samples/sec | ETA 04:42:24 2021-05-09 18:58:23 [INFO] [TRAIN] epoch: 50, iter: 18490/40000, loss: 0.1639, lr: 0.005765, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2713 samples/sec | ETA 04:41:59 2021-05-09 18:58:30 [INFO] [TRAIN] epoch: 50, iter: 18500/40000, loss: 0.4096, lr: 0.005762, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2714 samples/sec | ETA 04:41:51 2021-05-09 18:58:38 [INFO] [TRAIN] epoch: 50, iter: 18510/40000, loss: 0.4080, lr: 0.005760, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2714 samples/sec | ETA 04:41:43 2021-05-09 18:58:46 [INFO] [TRAIN] epoch: 50, iter: 18520/40000, loss: 0.2956, lr: 0.005758, batch_cost: 0.7858, reader_cost: 0.00017, ips: 1.2726 samples/sec | ETA 04:41:18 2021-05-09 18:58:54 [INFO] [TRAIN] epoch: 50, iter: 18530/40000, loss: 0.5305, lr: 0.005755, batch_cost: 0.7865, reader_cost: 0.00018, ips: 1.2715 samples/sec | ETA 04:41:25 2021-05-09 18:59:02 [INFO] [TRAIN] epoch: 50, iter: 18540/40000, loss: 0.5815, lr: 0.005753, batch_cost: 0.7855, reader_cost: 0.00018, ips: 1.2731 samples/sec | ETA 04:40:56 2021-05-09 18:59:10 [INFO] [TRAIN] epoch: 50, iter: 18550/40000, loss: 0.3235, lr: 0.005750, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2724 samples/sec | ETA 04:40:58 2021-05-09 18:59:18 [INFO] [TRAIN] epoch: 50, iter: 18560/40000, loss: 0.1149, lr: 0.005748, batch_cost: 0.7885, reader_cost: 0.00017, ips: 1.2682 samples/sec | ETA 04:41:45 2021-05-09 18:59:25 [INFO] [TRAIN] epoch: 50, iter: 18570/40000, loss: 0.2982, lr: 0.005746, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2721 samples/sec | ETA 04:40:45 2021-05-09 18:59:33 [INFO] [TRAIN] epoch: 50, iter: 18580/40000, loss: 0.3175, lr: 0.005743, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 04:40:34 2021-05-09 18:59:41 [INFO] [TRAIN] epoch: 50, iter: 18590/40000, loss: 0.3293, lr: 0.005741, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2703 samples/sec | ETA 04:40:54 2021-05-09 18:59:49 [INFO] [TRAIN] epoch: 50, iter: 18600/40000, loss: 0.4771, lr: 0.005739, batch_cost: 0.7864, reader_cost: 0.00011, ips: 1.2716 samples/sec | ETA 04:40:28 2021-05-09 19:00:00 [INFO] [TRAIN] epoch: 51, iter: 18610/40000, loss: 0.2376, lr: 0.005736, batch_cost: 1.1032, reader_cost: 0.21194, ips: 0.9064 samples/sec | ETA 06:33:17 2021-05-09 19:00:08 [INFO] [TRAIN] epoch: 51, iter: 18620/40000, loss: 0.4967, lr: 0.005734, batch_cost: 0.7909, reader_cost: 0.00030, ips: 1.2645 samples/sec | ETA 04:41:48 2021-05-09 19:00:16 [INFO] [TRAIN] epoch: 51, iter: 18630/40000, loss: 0.2531, lr: 0.005731, batch_cost: 0.7868, reader_cost: 0.00017, ips: 1.2709 samples/sec | ETA 04:40:14 2021-05-09 19:00:24 [INFO] [TRAIN] epoch: 51, iter: 18640/40000, loss: 0.4829, lr: 0.005729, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2704 samples/sec | ETA 04:40:14 2021-05-09 19:00:32 [INFO] [TRAIN] epoch: 51, iter: 18650/40000, loss: 0.4953, lr: 0.005727, batch_cost: 0.7873, reader_cost: 0.00015, ips: 1.2701 samples/sec | ETA 04:40:09 2021-05-09 19:00:39 [INFO] [TRAIN] epoch: 51, iter: 18660/40000, loss: 0.3527, lr: 0.005724, batch_cost: 0.7867, reader_cost: 0.00017, ips: 1.2711 samples/sec | ETA 04:39:49 2021-05-09 19:00:47 [INFO] [TRAIN] epoch: 51, iter: 18670/40000, loss: 0.1912, lr: 0.005722, batch_cost: 0.7857, reader_cost: 0.00018, ips: 1.2728 samples/sec | ETA 04:39:18 2021-05-09 19:00:55 [INFO] [TRAIN] epoch: 51, iter: 18680/40000, loss: 0.2920, lr: 0.005720, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 04:39:19 2021-05-09 19:01:03 [INFO] [TRAIN] epoch: 51, iter: 18690/40000, loss: 0.3529, lr: 0.005717, batch_cost: 0.7879, reader_cost: 0.00016, ips: 1.2692 samples/sec | ETA 04:39:49 2021-05-09 19:01:11 [INFO] [TRAIN] epoch: 51, iter: 18700/40000, loss: 0.2416, lr: 0.005715, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2703 samples/sec | ETA 04:39:28 2021-05-09 19:01:19 [INFO] [TRAIN] epoch: 51, iter: 18710/40000, loss: 0.3908, lr: 0.005713, batch_cost: 0.7872, reader_cost: 0.00015, ips: 1.2704 samples/sec | ETA 04:39:19 2021-05-09 19:01:27 [INFO] [TRAIN] epoch: 51, iter: 18720/40000, loss: 0.1192, lr: 0.005710, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2706 samples/sec | ETA 04:39:07 2021-05-09 19:01:35 [INFO] [TRAIN] epoch: 51, iter: 18730/40000, loss: 0.3010, lr: 0.005708, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 04:38:35 2021-05-09 19:01:42 [INFO] [TRAIN] epoch: 51, iter: 18740/40000, loss: 0.3422, lr: 0.005705, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2710 samples/sec | ETA 04:38:47 2021-05-09 19:01:50 [INFO] [TRAIN] epoch: 51, iter: 18750/40000, loss: 0.2336, lr: 0.005703, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2725 samples/sec | ETA 04:38:19 2021-05-09 19:01:58 [INFO] [TRAIN] epoch: 51, iter: 18760/40000, loss: 0.2513, lr: 0.005701, batch_cost: 0.7846, reader_cost: 0.00017, ips: 1.2746 samples/sec | ETA 04:37:44 2021-05-09 19:02:06 [INFO] [TRAIN] epoch: 51, iter: 18770/40000, loss: 0.2411, lr: 0.005698, batch_cost: 0.7852, reader_cost: 0.00017, ips: 1.2736 samples/sec | ETA 04:37:48 2021-05-09 19:02:14 [INFO] [TRAIN] epoch: 51, iter: 18780/40000, loss: 0.3198, lr: 0.005696, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 04:37:59 2021-05-09 19:02:22 [INFO] [TRAIN] epoch: 51, iter: 18790/40000, loss: 0.2548, lr: 0.005694, batch_cost: 0.7851, reader_cost: 0.00016, ips: 1.2738 samples/sec | ETA 04:37:31 2021-05-09 19:02:30 [INFO] [TRAIN] epoch: 51, iter: 18800/40000, loss: 0.1456, lr: 0.005691, batch_cost: 0.7875, reader_cost: 0.00015, ips: 1.2699 samples/sec | ETA 04:38:14 2021-05-09 19:02:37 [INFO] [TRAIN] epoch: 51, iter: 18810/40000, loss: 0.3370, lr: 0.005689, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2708 samples/sec | ETA 04:37:55 2021-05-09 19:02:45 [INFO] [TRAIN] epoch: 51, iter: 18820/40000, loss: 0.2065, lr: 0.005686, batch_cost: 0.7854, reader_cost: 0.00017, ips: 1.2733 samples/sec | ETA 04:37:14 2021-05-09 19:02:53 [INFO] [TRAIN] epoch: 51, iter: 18830/40000, loss: 0.2103, lr: 0.005684, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2725 samples/sec | ETA 04:37:16 2021-05-09 19:03:01 [INFO] [TRAIN] epoch: 51, iter: 18840/40000, loss: 0.3061, lr: 0.005682, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2736 samples/sec | ETA 04:36:54 2021-05-09 19:03:09 [INFO] [TRAIN] epoch: 51, iter: 18850/40000, loss: 0.1695, lr: 0.005679, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2731 samples/sec | ETA 04:36:52 2021-05-09 19:03:17 [INFO] [TRAIN] epoch: 51, iter: 18860/40000, loss: 0.2357, lr: 0.005677, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 04:37:03 2021-05-09 19:03:25 [INFO] [TRAIN] epoch: 51, iter: 18870/40000, loss: 0.2528, lr: 0.005675, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 04:36:46 2021-05-09 19:03:32 [INFO] [TRAIN] epoch: 51, iter: 18880/40000, loss: 0.3165, lr: 0.005672, batch_cost: 0.7858, reader_cost: 0.00015, ips: 1.2726 samples/sec | ETA 04:36:35 2021-05-09 19:03:40 [INFO] [TRAIN] epoch: 51, iter: 18890/40000, loss: 0.3589, lr: 0.005670, batch_cost: 0.7880, reader_cost: 0.00018, ips: 1.2691 samples/sec | ETA 04:37:14 2021-05-09 19:03:48 [INFO] [TRAIN] epoch: 51, iter: 18900/40000, loss: 0.5029, lr: 0.005667, batch_cost: 0.7855, reader_cost: 0.00017, ips: 1.2731 samples/sec | ETA 04:36:14 2021-05-09 19:03:56 [INFO] [TRAIN] epoch: 51, iter: 18910/40000, loss: 0.4765, lr: 0.005665, batch_cost: 0.7875, reader_cost: 0.00015, ips: 1.2698 samples/sec | ETA 04:36:49 2021-05-09 19:04:04 [INFO] [TRAIN] epoch: 51, iter: 18920/40000, loss: 0.2625, lr: 0.005663, batch_cost: 0.7853, reader_cost: 0.00015, ips: 1.2734 samples/sec | ETA 04:35:54 2021-05-09 19:04:12 [INFO] [TRAIN] epoch: 51, iter: 18930/40000, loss: 0.1878, lr: 0.005660, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2724 samples/sec | ETA 04:35:59 2021-05-09 19:04:20 [INFO] [TRAIN] epoch: 51, iter: 18940/40000, loss: 0.2427, lr: 0.005658, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2719 samples/sec | ETA 04:35:57 2021-05-09 19:04:27 [INFO] [TRAIN] epoch: 51, iter: 18950/40000, loss: 0.2320, lr: 0.005656, batch_cost: 0.7859, reader_cost: 0.00017, ips: 1.2724 samples/sec | ETA 04:35:44 2021-05-09 19:04:35 [INFO] [TRAIN] epoch: 51, iter: 18960/40000, loss: 0.3904, lr: 0.005653, batch_cost: 0.7870, reader_cost: 0.00015, ips: 1.2707 samples/sec | ETA 04:35:57 2021-05-09 19:04:43 [INFO] [TRAIN] epoch: 51, iter: 18970/40000, loss: 0.3602, lr: 0.005651, batch_cost: 0.7869, reader_cost: 0.00024, ips: 1.2708 samples/sec | ETA 04:35:49 2021-05-09 19:04:54 [INFO] [TRAIN] epoch: 52, iter: 18980/40000, loss: 0.5093, lr: 0.005648, batch_cost: 1.0895, reader_cost: 0.25925, ips: 0.9179 samples/sec | ETA 06:21:40 2021-05-09 19:05:02 [INFO] [TRAIN] epoch: 52, iter: 18990/40000, loss: 0.4824, lr: 0.005646, batch_cost: 0.7951, reader_cost: 0.00034, ips: 1.2578 samples/sec | ETA 04:38:24 2021-05-09 19:05:10 [INFO] [TRAIN] epoch: 52, iter: 19000/40000, loss: 0.3563, lr: 0.005644, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2730 samples/sec | ETA 04:34:57 2021-05-09 19:05:10 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-09 19:08:40 [INFO] [EVAL] #Images: 500 mIoU: 0.7398 Acc: 0.9522 Kappa: 0.9379 2021-05-09 19:08:41 [INFO] [EVAL] Class IoU: [0.9768 0.8186 0.9128 0.6008 0.606 0.4663 0.5839 0.6978 0.9084 0.6105 0.9384 0.7599 0.5018 0.936 0.793 0.8721 0.7594 0.597 0.7174] 2021-05-09 19:08:41 [INFO] [EVAL] Class Acc: [0.9899 0.9 0.941 0.8948 0.7917 0.754 0.8423 0.8985 0.9449 0.778 0.9711 0.8376 0.8181 0.9631 0.8525 0.9358 0.8984 0.7621 0.816 ] 2021-05-09 19:09:09 [INFO] [EVAL] The model with the best validation mIoU (0.7582) was saved at iter 16000. 2021-05-09 19:09:16 [INFO] [TRAIN] epoch: 52, iter: 19010/40000, loss: 0.3111, lr: 0.005641, batch_cost: 0.7841, reader_cost: 0.00023, ips: 1.2753 samples/sec | ETA 04:34:18 2021-05-09 19:09:24 [INFO] [TRAIN] epoch: 52, iter: 19020/40000, loss: 0.5272, lr: 0.005639, batch_cost: 0.7846, reader_cost: 0.00032, ips: 1.2745 samples/sec | ETA 04:34:20 2021-05-09 19:09:32 [INFO] [TRAIN] epoch: 52, iter: 19030/40000, loss: 0.3057, lr: 0.005637, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2733 samples/sec | ETA 04:34:28 2021-05-09 19:09:42 [INFO] [TRAIN] epoch: 52, iter: 19040/40000, loss: 0.2395, lr: 0.005634, batch_cost: 0.7848, reader_cost: 0.00017, ips: 1.2742 samples/sec | ETA 04:34:08 2021-05-09 19:09:50 [INFO] [TRAIN] epoch: 52, iter: 19050/40000, loss: 0.1718, lr: 0.005632, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2735 samples/sec | ETA 04:34:10 2021-05-09 19:09:58 [INFO] [TRAIN] epoch: 52, iter: 19060/40000, loss: 0.3194, lr: 0.005629, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2705 samples/sec | ETA 04:34:41 2021-05-09 19:10:06 [INFO] [TRAIN] epoch: 52, iter: 19070/40000, loss: 0.3697, lr: 0.005627, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2736 samples/sec | ETA 04:33:53 2021-05-09 19:10:14 [INFO] [TRAIN] epoch: 52, iter: 19080/40000, loss: 0.3031, lr: 0.005625, batch_cost: 0.7875, reader_cost: 0.00015, ips: 1.2698 samples/sec | ETA 04:34:35 2021-05-09 19:10:22 [INFO] [TRAIN] epoch: 52, iter: 19090/40000, loss: 0.1436, lr: 0.005622, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2713 samples/sec | ETA 04:34:07 2021-05-09 19:10:30 [INFO] [TRAIN] epoch: 52, iter: 19100/40000, loss: 0.2385, lr: 0.005620, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2716 samples/sec | ETA 04:33:55 2021-05-09 19:10:37 [INFO] [TRAIN] epoch: 52, iter: 19110/40000, loss: 0.5857, lr: 0.005618, batch_cost: 0.7857, reader_cost: 0.00017, ips: 1.2727 samples/sec | ETA 04:33:34 2021-05-09 19:10:45 [INFO] [TRAIN] epoch: 52, iter: 19120/40000, loss: 0.1878, lr: 0.005615, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 04:33:32 2021-05-09 19:10:53 [INFO] [TRAIN] epoch: 52, iter: 19130/40000, loss: 0.3148, lr: 0.005613, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2723 samples/sec | ETA 04:33:23 2021-05-09 19:11:01 [INFO] [TRAIN] epoch: 52, iter: 19140/40000, loss: 0.2744, lr: 0.005610, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2705 samples/sec | ETA 04:33:38 2021-05-09 19:11:09 [INFO] [TRAIN] epoch: 52, iter: 19150/40000, loss: 0.3386, lr: 0.005608, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2728 samples/sec | ETA 04:33:00 2021-05-09 19:11:17 [INFO] [TRAIN] epoch: 52, iter: 19160/40000, loss: 0.1768, lr: 0.005606, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2735 samples/sec | ETA 04:32:44 2021-05-09 19:11:25 [INFO] [TRAIN] epoch: 52, iter: 19170/40000, loss: 0.0477, lr: 0.005603, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2706 samples/sec | ETA 04:33:14 2021-05-09 19:11:32 [INFO] [TRAIN] epoch: 52, iter: 19180/40000, loss: 0.2752, lr: 0.005601, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 04:32:46 2021-05-09 19:11:40 [INFO] [TRAIN] epoch: 52, iter: 19190/40000, loss: 0.2779, lr: 0.005599, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2730 samples/sec | ETA 04:32:27 2021-05-09 19:11:48 [INFO] [TRAIN] epoch: 52, iter: 19200/40000, loss: 0.2311, lr: 0.005596, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2713 samples/sec | ETA 04:32:41 2021-05-09 19:11:56 [INFO] [TRAIN] epoch: 52, iter: 19210/40000, loss: 0.3473, lr: 0.005594, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 04:32:10 2021-05-09 19:12:04 [INFO] [TRAIN] epoch: 52, iter: 19220/40000, loss: 0.2225, lr: 0.005591, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2734 samples/sec | ETA 04:31:58 2021-05-09 19:12:12 [INFO] [TRAIN] epoch: 52, iter: 19230/40000, loss: 0.2731, lr: 0.005589, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2733 samples/sec | ETA 04:31:51 2021-05-09 19:12:20 [INFO] [TRAIN] epoch: 52, iter: 19240/40000, loss: 0.1819, lr: 0.005587, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 04:31:57 2021-05-09 19:12:27 [INFO] [TRAIN] epoch: 52, iter: 19250/40000, loss: 0.3770, lr: 0.005584, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 04:31:51 2021-05-09 19:12:35 [INFO] [TRAIN] epoch: 52, iter: 19260/40000, loss: 0.4771, lr: 0.005582, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 04:31:36 2021-05-09 19:12:43 [INFO] [TRAIN] epoch: 52, iter: 19270/40000, loss: 0.4814, lr: 0.005579, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2718 samples/sec | ETA 04:31:39 2021-05-09 19:12:51 [INFO] [TRAIN] epoch: 52, iter: 19280/40000, loss: 0.5327, lr: 0.005577, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2723 samples/sec | ETA 04:31:25 2021-05-09 19:12:59 [INFO] [TRAIN] epoch: 52, iter: 19290/40000, loss: 0.4483, lr: 0.005575, batch_cost: 0.7848, reader_cost: 0.00014, ips: 1.2742 samples/sec | ETA 04:30:53 2021-05-09 19:13:07 [INFO] [TRAIN] epoch: 52, iter: 19300/40000, loss: 0.1521, lr: 0.005572, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2736 samples/sec | ETA 04:30:53 2021-05-09 19:13:15 [INFO] [TRAIN] epoch: 52, iter: 19310/40000, loss: 0.2708, lr: 0.005570, batch_cost: 0.7857, reader_cost: 0.00014, ips: 1.2728 samples/sec | ETA 04:30:55 2021-05-09 19:13:22 [INFO] [TRAIN] epoch: 52, iter: 19320/40000, loss: 0.2862, lr: 0.005568, batch_cost: 0.7841, reader_cost: 0.00015, ips: 1.2753 samples/sec | ETA 04:30:16 2021-05-09 19:13:30 [INFO] [TRAIN] epoch: 52, iter: 19330/40000, loss: 0.5154, lr: 0.005565, batch_cost: 0.7850, reader_cost: 0.00015, ips: 1.2739 samples/sec | ETA 04:30:25 2021-05-09 19:13:38 [INFO] [TRAIN] epoch: 52, iter: 19340/40000, loss: 0.3150, lr: 0.005563, batch_cost: 0.7859, reader_cost: 0.00011, ips: 1.2725 samples/sec | ETA 04:30:35 2021-05-09 19:13:49 [INFO] [TRAIN] epoch: 53, iter: 19350/40000, loss: 0.3877, lr: 0.005560, batch_cost: 1.1031, reader_cost: 0.25422, ips: 0.9066 samples/sec | ETA 06:19:38 2021-05-09 19:13:57 [INFO] [TRAIN] epoch: 53, iter: 19360/40000, loss: 0.2602, lr: 0.005558, batch_cost: 0.7967, reader_cost: 0.00033, ips: 1.2552 samples/sec | ETA 04:34:03 2021-05-09 19:14:05 [INFO] [TRAIN] epoch: 53, iter: 19370/40000, loss: 0.4828, lr: 0.005556, batch_cost: 0.7849, reader_cost: 0.00015, ips: 1.2741 samples/sec | ETA 04:29:52 2021-05-09 19:14:13 [INFO] [TRAIN] epoch: 53, iter: 19380/40000, loss: 0.4030, lr: 0.005553, batch_cost: 0.7879, reader_cost: 0.00015, ips: 1.2692 samples/sec | ETA 04:30:46 2021-05-09 19:14:21 [INFO] [TRAIN] epoch: 53, iter: 19390/40000, loss: 0.4366, lr: 0.005551, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2720 samples/sec | ETA 04:30:03 2021-05-09 19:14:29 [INFO] [TRAIN] epoch: 53, iter: 19400/40000, loss: 0.3215, lr: 0.005549, batch_cost: 0.7869, reader_cost: 0.00017, ips: 1.2708 samples/sec | ETA 04:30:10 2021-05-09 19:14:37 [INFO] [TRAIN] epoch: 53, iter: 19410/40000, loss: 0.3499, lr: 0.005546, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2712 samples/sec | ETA 04:29:56 2021-05-09 19:14:44 [INFO] [TRAIN] epoch: 53, iter: 19420/40000, loss: 0.1823, lr: 0.005544, batch_cost: 0.7866, reader_cost: 0.00017, ips: 1.2713 samples/sec | ETA 04:29:48 2021-05-09 19:14:52 [INFO] [TRAIN] epoch: 53, iter: 19430/40000, loss: 0.3090, lr: 0.005541, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2730 samples/sec | ETA 04:29:18 2021-05-09 19:15:00 [INFO] [TRAIN] epoch: 53, iter: 19440/40000, loss: 0.2311, lr: 0.005539, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2716 samples/sec | ETA 04:29:29 2021-05-09 19:15:08 [INFO] [TRAIN] epoch: 53, iter: 19450/40000, loss: 0.3362, lr: 0.005537, batch_cost: 0.7869, reader_cost: 0.00015, ips: 1.2708 samples/sec | ETA 04:29:30 2021-05-09 19:15:16 [INFO] [TRAIN] epoch: 53, iter: 19460/40000, loss: 0.1991, lr: 0.005534, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2723 samples/sec | ETA 04:29:04 2021-05-09 19:15:24 [INFO] [TRAIN] epoch: 53, iter: 19470/40000, loss: 0.1801, lr: 0.005532, batch_cost: 0.7874, reader_cost: 0.00015, ips: 1.2700 samples/sec | ETA 04:29:25 2021-05-09 19:15:32 [INFO] [TRAIN] epoch: 53, iter: 19480/40000, loss: 0.4525, lr: 0.005530, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2728 samples/sec | ETA 04:28:41 2021-05-09 19:15:39 [INFO] [TRAIN] epoch: 53, iter: 19490/40000, loss: 0.2779, lr: 0.005527, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2729 samples/sec | ETA 04:28:32 2021-05-09 19:15:47 [INFO] [TRAIN] epoch: 53, iter: 19500/40000, loss: 0.2560, lr: 0.005525, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2711 samples/sec | ETA 04:28:47 2021-05-09 19:15:55 [INFO] [TRAIN] epoch: 53, iter: 19510/40000, loss: 0.2416, lr: 0.005522, batch_cost: 0.7881, reader_cost: 0.00015, ips: 1.2689 samples/sec | ETA 04:29:08 2021-05-09 19:16:03 [INFO] [TRAIN] epoch: 53, iter: 19520/40000, loss: 0.1915, lr: 0.005520, batch_cost: 0.7875, reader_cost: 0.00015, ips: 1.2698 samples/sec | ETA 04:28:48 2021-05-09 19:16:11 [INFO] [TRAIN] epoch: 53, iter: 19530/40000, loss: 0.2545, lr: 0.005518, batch_cost: 0.7872, reader_cost: 0.00015, ips: 1.2703 samples/sec | ETA 04:28:34 2021-05-09 19:16:19 [INFO] [TRAIN] epoch: 53, iter: 19540/40000, loss: 0.1184, lr: 0.005515, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2729 samples/sec | ETA 04:27:53 2021-05-09 19:16:27 [INFO] [TRAIN] epoch: 53, iter: 19550/40000, loss: 0.3181, lr: 0.005513, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2725 samples/sec | ETA 04:27:50 2021-05-09 19:16:34 [INFO] [TRAIN] epoch: 53, iter: 19560/40000, loss: 0.3078, lr: 0.005510, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2726 samples/sec | ETA 04:27:41 2021-05-09 19:16:42 [INFO] [TRAIN] epoch: 53, iter: 19570/40000, loss: 0.2408, lr: 0.005508, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2719 samples/sec | ETA 04:27:42 2021-05-09 19:16:50 [INFO] [TRAIN] epoch: 53, iter: 19580/40000, loss: 0.3345, lr: 0.005506, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2723 samples/sec | ETA 04:27:29 2021-05-09 19:16:58 [INFO] [TRAIN] epoch: 53, iter: 19590/40000, loss: 0.1676, lr: 0.005503, batch_cost: 0.7872, reader_cost: 0.00015, ips: 1.2703 samples/sec | ETA 04:27:47 2021-05-09 19:17:06 [INFO] [TRAIN] epoch: 53, iter: 19600/40000, loss: 0.2549, lr: 0.005501, batch_cost: 0.7858, reader_cost: 0.00015, ips: 1.2725 samples/sec | ETA 04:27:11 2021-05-09 19:17:14 [INFO] [TRAIN] epoch: 53, iter: 19610/40000, loss: 0.2218, lr: 0.005499, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2728 samples/sec | ETA 04:26:59 2021-05-09 19:17:22 [INFO] [TRAIN] epoch: 53, iter: 19620/40000, loss: 0.3585, lr: 0.005496, batch_cost: 0.7854, reader_cost: 0.00015, ips: 1.2732 samples/sec | ETA 04:26:46 2021-05-09 19:17:30 [INFO] [TRAIN] epoch: 53, iter: 19630/40000, loss: 0.5030, lr: 0.005494, batch_cost: 0.7850, reader_cost: 0.00016, ips: 1.2739 samples/sec | ETA 04:26:29 2021-05-09 19:17:37 [INFO] [TRAIN] epoch: 53, iter: 19640/40000, loss: 0.3629, lr: 0.005491, batch_cost: 0.7859, reader_cost: 0.00018, ips: 1.2725 samples/sec | ETA 04:26:40 2021-05-09 19:17:45 [INFO] [TRAIN] epoch: 53, iter: 19650/40000, loss: 0.5544, lr: 0.005489, batch_cost: 0.7844, reader_cost: 0.00017, ips: 1.2748 samples/sec | ETA 04:26:03 2021-05-09 19:17:53 [INFO] [TRAIN] epoch: 53, iter: 19660/40000, loss: 0.3799, lr: 0.005487, batch_cost: 0.7862, reader_cost: 0.00017, ips: 1.2720 samples/sec | ETA 04:26:30 2021-05-09 19:18:01 [INFO] [TRAIN] epoch: 53, iter: 19670/40000, loss: 0.2940, lr: 0.005484, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2730 samples/sec | ETA 04:26:10 2021-05-09 19:18:09 [INFO] [TRAIN] epoch: 53, iter: 19680/40000, loss: 0.1470, lr: 0.005482, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2714 samples/sec | ETA 04:26:21 2021-05-09 19:18:17 [INFO] [TRAIN] epoch: 53, iter: 19690/40000, loss: 0.2706, lr: 0.005479, batch_cost: 0.7864, reader_cost: 0.00017, ips: 1.2716 samples/sec | ETA 04:26:12 2021-05-09 19:18:25 [INFO] [TRAIN] epoch: 53, iter: 19700/40000, loss: 0.3555, lr: 0.005477, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2712 samples/sec | ETA 04:26:08 2021-05-09 19:18:32 [INFO] [TRAIN] epoch: 53, iter: 19710/40000, loss: 0.2974, lr: 0.005475, batch_cost: 0.7854, reader_cost: 0.00012, ips: 1.2732 samples/sec | ETA 04:25:36 2021-05-09 19:18:43 [INFO] [TRAIN] epoch: 54, iter: 19720/40000, loss: 0.3724, lr: 0.005472, batch_cost: 1.1011, reader_cost: 0.29998, ips: 0.9082 samples/sec | ETA 06:12:09 2021-05-09 19:18:51 [INFO] [TRAIN] epoch: 54, iter: 19730/40000, loss: 0.2619, lr: 0.005470, batch_cost: 0.7977, reader_cost: 0.00031, ips: 1.2536 samples/sec | ETA 04:29:28 2021-05-09 19:18:59 [INFO] [TRAIN] epoch: 54, iter: 19740/40000, loss: 0.5518, lr: 0.005468, batch_cost: 0.7851, reader_cost: 0.00017, ips: 1.2737 samples/sec | ETA 04:25:05 2021-05-09 19:19:07 [INFO] [TRAIN] epoch: 54, iter: 19750/40000, loss: 0.2843, lr: 0.005465, batch_cost: 0.7883, reader_cost: 0.00015, ips: 1.2685 samples/sec | ETA 04:26:03 2021-05-09 19:19:15 [INFO] [TRAIN] epoch: 54, iter: 19760/40000, loss: 0.3862, lr: 0.005463, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2704 samples/sec | ETA 04:25:32 2021-05-09 19:19:23 [INFO] [TRAIN] epoch: 54, iter: 19770/40000, loss: 0.6163, lr: 0.005460, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2728 samples/sec | ETA 04:24:53 2021-05-09 19:19:31 [INFO] [TRAIN] epoch: 54, iter: 19780/40000, loss: 0.8011, lr: 0.005458, batch_cost: 0.7861, reader_cost: 0.00018, ips: 1.2720 samples/sec | ETA 04:24:55 2021-05-09 19:19:39 [INFO] [TRAIN] epoch: 54, iter: 19790/40000, loss: 0.2075, lr: 0.005456, batch_cost: 0.7877, reader_cost: 0.00018, ips: 1.2696 samples/sec | ETA 04:25:18 2021-05-09 19:19:46 [INFO] [TRAIN] epoch: 54, iter: 19800/40000, loss: 0.4160, lr: 0.005453, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 04:24:34 2021-05-09 19:19:54 [INFO] [TRAIN] epoch: 54, iter: 19810/40000, loss: 0.4271, lr: 0.005451, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2732 samples/sec | ETA 04:24:18 2021-05-09 19:20:02 [INFO] [TRAIN] epoch: 54, iter: 19820/40000, loss: 0.3491, lr: 0.005448, batch_cost: 0.7872, reader_cost: 0.00017, ips: 1.2703 samples/sec | ETA 04:24:45 2021-05-09 19:20:10 [INFO] [TRAIN] epoch: 54, iter: 19830/40000, loss: 0.3727, lr: 0.005446, batch_cost: 0.7861, reader_cost: 0.00017, ips: 1.2722 samples/sec | ETA 04:24:14 2021-05-09 19:20:18 [INFO] [TRAIN] epoch: 54, iter: 19840/40000, loss: 0.1572, lr: 0.005444, batch_cost: 0.7849, reader_cost: 0.00017, ips: 1.2740 samples/sec | ETA 04:23:44 2021-05-09 19:20:26 [INFO] [TRAIN] epoch: 54, iter: 19850/40000, loss: 0.4062, lr: 0.005441, batch_cost: 0.7849, reader_cost: 0.00017, ips: 1.2740 samples/sec | ETA 04:23:36 2021-05-09 19:20:34 [INFO] [TRAIN] epoch: 54, iter: 19860/40000, loss: 0.3883, lr: 0.005439, batch_cost: 0.7849, reader_cost: 0.00016, ips: 1.2740 samples/sec | ETA 04:23:28 2021-05-09 19:20:41 [INFO] [TRAIN] epoch: 54, iter: 19870/40000, loss: 0.4877, lr: 0.005437, batch_cost: 0.7840, reader_cost: 0.00015, ips: 1.2755 samples/sec | ETA 04:23:01 2021-05-09 19:20:49 [INFO] [TRAIN] epoch: 54, iter: 19880/40000, loss: 0.2470, lr: 0.005434, batch_cost: 0.7849, reader_cost: 0.00015, ips: 1.2741 samples/sec | ETA 04:23:11 2021-05-09 19:20:57 [INFO] [TRAIN] epoch: 54, iter: 19890/40000, loss: 0.2227, lr: 0.005432, batch_cost: 0.7844, reader_cost: 0.00016, ips: 1.2749 samples/sec | ETA 04:22:53 2021-05-09 19:21:05 [INFO] [TRAIN] epoch: 54, iter: 19900/40000, loss: 0.5681, lr: 0.005429, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2728 samples/sec | ETA 04:23:12 2021-05-09 19:21:13 [INFO] [TRAIN] epoch: 54, iter: 19910/40000, loss: 0.1282, lr: 0.005427, batch_cost: 0.7849, reader_cost: 0.00016, ips: 1.2740 samples/sec | ETA 04:22:48 2021-05-09 19:21:21 [INFO] [TRAIN] epoch: 54, iter: 19920/40000, loss: 0.1927, lr: 0.005425, batch_cost: 0.7860, reader_cost: 0.00017, ips: 1.2722 samples/sec | ETA 04:23:03 2021-05-09 19:21:29 [INFO] [TRAIN] epoch: 54, iter: 19930/40000, loss: 0.2431, lr: 0.005422, batch_cost: 0.7850, reader_cost: 0.00016, ips: 1.2739 samples/sec | ETA 04:22:35 2021-05-09 19:21:36 [INFO] [TRAIN] epoch: 54, iter: 19940/40000, loss: 0.2772, lr: 0.005420, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 04:22:57 2021-05-09 19:21:44 [INFO] [TRAIN] epoch: 54, iter: 19950/40000, loss: 0.3583, lr: 0.005417, batch_cost: 0.7851, reader_cost: 0.00015, ips: 1.2737 samples/sec | ETA 04:22:21 2021-05-09 19:21:52 [INFO] [TRAIN] epoch: 54, iter: 19960/40000, loss: 0.3055, lr: 0.005415, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2721 samples/sec | ETA 04:22:33 2021-05-09 19:22:00 [INFO] [TRAIN] epoch: 54, iter: 19970/40000, loss: 0.2138, lr: 0.005413, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2729 samples/sec | ETA 04:22:16 2021-05-09 19:22:08 [INFO] [TRAIN] epoch: 54, iter: 19980/40000, loss: 0.3436, lr: 0.005410, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2719 samples/sec | ETA 04:22:20 2021-05-09 19:22:16 [INFO] [TRAIN] epoch: 54, iter: 19990/40000, loss: 0.3614, lr: 0.005408, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2736 samples/sec | ETA 04:21:51 2021-05-09 19:22:24 [INFO] [TRAIN] epoch: 54, iter: 20000/40000, loss: 0.6349, lr: 0.005406, batch_cost: 0.7847, reader_cost: 0.00016, ips: 1.2744 samples/sec | ETA 04:21:33 2021-05-09 19:22:24 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-09 19:25:55 [INFO] [EVAL] #Images: 500 mIoU: 0.7461 Acc: 0.9543 Kappa: 0.9406 2021-05-09 19:25:55 [INFO] [EVAL] Class IoU: [0.9806 0.8428 0.9149 0.6178 0.5824 0.4302 0.589 0.6935 0.9111 0.6166 0.9377 0.7634 0.5551 0.9361 0.8193 0.876 0.7725 0.6155 0.7211] 2021-05-09 19:25:55 [INFO] [EVAL] Class Acc: [0.9916 0.9132 0.9434 0.7811 0.7818 0.8075 0.8385 0.9049 0.9407 0.9162 0.9614 0.8383 0.7649 0.9626 0.9004 0.9441 0.9272 0.8519 0.8222] 2021-05-09 19:26:23 [INFO] [EVAL] The model with the best validation mIoU (0.7582) was saved at iter 16000. 2021-05-09 19:26:31 [INFO] [TRAIN] epoch: 54, iter: 20010/40000, loss: 0.3269, lr: 0.005403, batch_cost: 0.7807, reader_cost: 0.00023, ips: 1.2810 samples/sec | ETA 04:20:05 2021-05-09 19:26:39 [INFO] [TRAIN] epoch: 54, iter: 20020/40000, loss: 0.6177, lr: 0.005401, batch_cost: 0.7844, reader_cost: 0.00018, ips: 1.2749 samples/sec | ETA 04:21:12 2021-05-09 19:26:47 [INFO] [TRAIN] epoch: 54, iter: 20030/40000, loss: 0.4206, lr: 0.005398, batch_cost: 0.7844, reader_cost: 0.00015, ips: 1.2749 samples/sec | ETA 04:21:03 2021-05-09 19:26:55 [INFO] [TRAIN] epoch: 54, iter: 20040/40000, loss: 0.2736, lr: 0.005396, batch_cost: 0.7843, reader_cost: 0.00017, ips: 1.2751 samples/sec | ETA 04:20:54 2021-05-09 19:27:03 [INFO] [TRAIN] epoch: 54, iter: 20050/40000, loss: 0.1852, lr: 0.005394, batch_cost: 0.7882, reader_cost: 0.00015, ips: 1.2687 samples/sec | ETA 04:22:04 2021-05-09 19:27:11 [INFO] [TRAIN] epoch: 54, iter: 20060/40000, loss: 0.2880, lr: 0.005391, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 04:21:19 2021-05-09 19:27:18 [INFO] [TRAIN] epoch: 54, iter: 20070/40000, loss: 0.2589, lr: 0.005389, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2713 samples/sec | ETA 04:21:17 2021-05-09 19:27:26 [INFO] [TRAIN] epoch: 54, iter: 20080/40000, loss: 0.3760, lr: 0.005386, batch_cost: 0.7852, reader_cost: 0.00013, ips: 1.2736 samples/sec | ETA 04:20:41 2021-05-09 19:27:37 [INFO] [TRAIN] epoch: 55, iter: 20090/40000, loss: 0.4409, lr: 0.005384, batch_cost: 1.0940, reader_cost: 0.26440, ips: 0.9141 samples/sec | ETA 06:03:01 2021-05-09 19:27:45 [INFO] [TRAIN] epoch: 55, iter: 20100/40000, loss: 0.2864, lr: 0.005382, batch_cost: 0.7992, reader_cost: 0.00034, ips: 1.2513 samples/sec | ETA 04:25:03 2021-05-09 19:27:53 [INFO] [TRAIN] epoch: 55, iter: 20110/40000, loss: 0.5261, lr: 0.005379, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2715 samples/sec | ETA 04:20:42 2021-05-09 19:28:01 [INFO] [TRAIN] epoch: 55, iter: 20120/40000, loss: 0.1886, lr: 0.005377, batch_cost: 0.7865, reader_cost: 0.00017, ips: 1.2714 samples/sec | ETA 04:20:35 2021-05-09 19:28:09 [INFO] [TRAIN] epoch: 55, iter: 20130/40000, loss: 0.4195, lr: 0.005374, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2710 samples/sec | ETA 04:20:33 2021-05-09 19:28:17 [INFO] [TRAIN] epoch: 55, iter: 20140/40000, loss: 0.4981, lr: 0.005372, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2730 samples/sec | ETA 04:20:00 2021-05-09 19:28:24 [INFO] [TRAIN] epoch: 55, iter: 20150/40000, loss: 0.3850, lr: 0.005370, batch_cost: 0.7857, reader_cost: 0.00014, ips: 1.2727 samples/sec | ETA 04:19:57 2021-05-09 19:28:32 [INFO] [TRAIN] epoch: 55, iter: 20160/40000, loss: 0.1318, lr: 0.005367, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2715 samples/sec | ETA 04:20:03 2021-05-09 19:28:40 [INFO] [TRAIN] epoch: 55, iter: 20170/40000, loss: 0.2918, lr: 0.005365, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 04:19:41 2021-05-09 19:28:48 [INFO] [TRAIN] epoch: 55, iter: 20180/40000, loss: 0.3578, lr: 0.005363, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2716 samples/sec | ETA 04:19:47 2021-05-09 19:28:56 [INFO] [TRAIN] epoch: 55, iter: 20190/40000, loss: 0.2347, lr: 0.005360, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2715 samples/sec | ETA 04:19:40 2021-05-09 19:29:04 [INFO] [TRAIN] epoch: 55, iter: 20200/40000, loss: 0.2926, lr: 0.005358, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2726 samples/sec | ETA 04:19:18 2021-05-09 19:29:12 [INFO] [TRAIN] epoch: 55, iter: 20210/40000, loss: 0.2205, lr: 0.005355, batch_cost: 0.7874, reader_cost: 0.00015, ips: 1.2700 samples/sec | ETA 04:19:42 2021-05-09 19:29:20 [INFO] [TRAIN] epoch: 55, iter: 20220/40000, loss: 0.3172, lr: 0.005353, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2713 samples/sec | ETA 04:19:18 2021-05-09 19:29:27 [INFO] [TRAIN] epoch: 55, iter: 20230/40000, loss: 0.3027, lr: 0.005351, batch_cost: 0.7876, reader_cost: 0.00016, ips: 1.2697 samples/sec | ETA 04:19:30 2021-05-09 19:29:35 [INFO] [TRAIN] epoch: 55, iter: 20240/40000, loss: 0.3616, lr: 0.005348, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 04:19:04 2021-05-09 19:29:43 [INFO] [TRAIN] epoch: 55, iter: 20250/40000, loss: 0.2014, lr: 0.005346, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2731 samples/sec | ETA 04:18:33 2021-05-09 19:29:51 [INFO] [TRAIN] epoch: 55, iter: 20260/40000, loss: 0.1877, lr: 0.005343, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2721 samples/sec | ETA 04:18:37 2021-05-09 19:29:59 [INFO] [TRAIN] epoch: 55, iter: 20270/40000, loss: 0.3661, lr: 0.005341, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2729 samples/sec | ETA 04:18:20 2021-05-09 19:30:07 [INFO] [TRAIN] epoch: 55, iter: 20280/40000, loss: 0.1472, lr: 0.005339, batch_cost: 0.7853, reader_cost: 0.00018, ips: 1.2735 samples/sec | ETA 04:18:05 2021-05-09 19:30:15 [INFO] [TRAIN] epoch: 55, iter: 20290/40000, loss: 0.2188, lr: 0.005336, batch_cost: 0.7876, reader_cost: 0.00015, ips: 1.2696 samples/sec | ETA 04:18:44 2021-05-09 19:30:22 [INFO] [TRAIN] epoch: 55, iter: 20300/40000, loss: 0.3618, lr: 0.005334, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 04:18:04 2021-05-09 19:30:30 [INFO] [TRAIN] epoch: 55, iter: 20310/40000, loss: 0.1824, lr: 0.005331, batch_cost: 0.7879, reader_cost: 0.00017, ips: 1.2692 samples/sec | ETA 04:18:33 2021-05-09 19:30:38 [INFO] [TRAIN] epoch: 55, iter: 20320/40000, loss: 0.2126, lr: 0.005329, batch_cost: 0.7849, reader_cost: 0.00018, ips: 1.2740 samples/sec | ETA 04:17:26 2021-05-09 19:30:46 [INFO] [TRAIN] epoch: 55, iter: 20330/40000, loss: 0.2461, lr: 0.005327, batch_cost: 0.7843, reader_cost: 0.00018, ips: 1.2750 samples/sec | ETA 04:17:07 2021-05-09 19:30:54 [INFO] [TRAIN] epoch: 55, iter: 20340/40000, loss: 0.3085, lr: 0.005324, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2735 samples/sec | ETA 04:17:17 2021-05-09 19:31:02 [INFO] [TRAIN] epoch: 55, iter: 20350/40000, loss: 0.1900, lr: 0.005322, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2730 samples/sec | ETA 04:17:16 2021-05-09 19:31:10 [INFO] [TRAIN] epoch: 55, iter: 20360/40000, loss: 0.3322, lr: 0.005319, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2722 samples/sec | ETA 04:17:17 2021-05-09 19:31:17 [INFO] [TRAIN] epoch: 55, iter: 20370/40000, loss: 0.5058, lr: 0.005317, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2705 samples/sec | ETA 04:17:30 2021-05-09 19:31:25 [INFO] [TRAIN] epoch: 55, iter: 20380/40000, loss: 0.2637, lr: 0.005315, batch_cost: 0.7846, reader_cost: 0.00015, ips: 1.2745 samples/sec | ETA 04:16:33 2021-05-09 19:31:33 [INFO] [TRAIN] epoch: 55, iter: 20390/40000, loss: 0.4544, lr: 0.005312, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 04:16:51 2021-05-09 19:31:41 [INFO] [TRAIN] epoch: 55, iter: 20400/40000, loss: 0.7498, lr: 0.005310, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2722 samples/sec | ETA 04:16:46 2021-05-09 19:31:49 [INFO] [TRAIN] epoch: 55, iter: 20410/40000, loss: 0.4448, lr: 0.005308, batch_cost: 0.7840, reader_cost: 0.00016, ips: 1.2755 samples/sec | ETA 04:15:58 2021-05-09 19:31:57 [INFO] [TRAIN] epoch: 55, iter: 20420/40000, loss: 0.1573, lr: 0.005305, batch_cost: 0.7845, reader_cost: 0.00017, ips: 1.2746 samples/sec | ETA 04:16:01 2021-05-09 19:32:05 [INFO] [TRAIN] epoch: 55, iter: 20430/40000, loss: 0.2494, lr: 0.005303, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 04:16:11 2021-05-09 19:32:12 [INFO] [TRAIN] epoch: 55, iter: 20440/40000, loss: 0.4280, lr: 0.005300, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2705 samples/sec | ETA 04:16:35 2021-05-09 19:32:20 [INFO] [TRAIN] epoch: 55, iter: 20450/40000, loss: 0.3211, lr: 0.005298, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2718 samples/sec | ETA 04:16:12 2021-05-09 19:32:28 [INFO] [TRAIN] epoch: 55, iter: 20460/40000, loss: 0.3444, lr: 0.005296, batch_cost: 0.7839, reader_cost: 0.00010, ips: 1.2757 samples/sec | ETA 04:15:16 2021-05-09 19:32:39 [INFO] [TRAIN] epoch: 56, iter: 20470/40000, loss: 0.1986, lr: 0.005293, batch_cost: 1.1013, reader_cost: 0.24619, ips: 0.9080 samples/sec | ETA 05:58:29 2021-05-09 19:32:47 [INFO] [TRAIN] epoch: 56, iter: 20480/40000, loss: 0.5163, lr: 0.005291, batch_cost: 0.7900, reader_cost: 0.00033, ips: 1.2658 samples/sec | ETA 04:17:01 2021-05-09 19:32:55 [INFO] [TRAIN] epoch: 56, iter: 20490/40000, loss: 0.2534, lr: 0.005288, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2718 samples/sec | ETA 04:15:40 2021-05-09 19:33:03 [INFO] [TRAIN] epoch: 56, iter: 20500/40000, loss: 0.3952, lr: 0.005286, batch_cost: 0.7853, reader_cost: 0.00017, ips: 1.2734 samples/sec | ETA 04:15:13 2021-05-09 19:33:11 [INFO] [TRAIN] epoch: 56, iter: 20510/40000, loss: 0.4578, lr: 0.005284, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2726 samples/sec | ETA 04:15:15 2021-05-09 19:33:19 [INFO] [TRAIN] epoch: 56, iter: 20520/40000, loss: 0.3032, lr: 0.005281, batch_cost: 0.7875, reader_cost: 0.00015, ips: 1.2698 samples/sec | ETA 04:15:41 2021-05-09 19:33:26 [INFO] [TRAIN] epoch: 56, iter: 20530/40000, loss: 0.1452, lr: 0.005279, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2713 samples/sec | ETA 04:15:14 2021-05-09 19:33:34 [INFO] [TRAIN] epoch: 56, iter: 20540/40000, loss: 0.2387, lr: 0.005276, batch_cost: 0.7846, reader_cost: 0.00016, ips: 1.2745 samples/sec | ETA 04:14:28 2021-05-09 19:33:42 [INFO] [TRAIN] epoch: 56, iter: 20550/40000, loss: 0.3417, lr: 0.005274, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 04:14:56 2021-05-09 19:33:50 [INFO] [TRAIN] epoch: 56, iter: 20560/40000, loss: 0.2026, lr: 0.005272, batch_cost: 0.7875, reader_cost: 0.00015, ips: 1.2699 samples/sec | ETA 04:15:08 2021-05-09 19:33:58 [INFO] [TRAIN] epoch: 56, iter: 20570/40000, loss: 0.3786, lr: 0.005269, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2710 samples/sec | ETA 04:14:47 2021-05-09 19:34:06 [INFO] [TRAIN] epoch: 56, iter: 20580/40000, loss: 0.1738, lr: 0.005267, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2709 samples/sec | ETA 04:14:40 2021-05-09 19:34:14 [INFO] [TRAIN] epoch: 56, iter: 20590/40000, loss: 0.2906, lr: 0.005264, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2720 samples/sec | ETA 04:14:19 2021-05-09 19:34:21 [INFO] [TRAIN] epoch: 56, iter: 20600/40000, loss: 0.2984, lr: 0.005262, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2705 samples/sec | ETA 04:14:29 2021-05-09 19:34:29 [INFO] [TRAIN] epoch: 56, iter: 20610/40000, loss: 0.2583, lr: 0.005260, batch_cost: 0.7870, reader_cost: 0.00015, ips: 1.2707 samples/sec | ETA 04:14:19 2021-05-09 19:34:37 [INFO] [TRAIN] epoch: 56, iter: 20620/40000, loss: 0.2128, lr: 0.005257, batch_cost: 0.7874, reader_cost: 0.00019, ips: 1.2700 samples/sec | ETA 04:14:20 2021-05-09 19:34:45 [INFO] [TRAIN] epoch: 56, iter: 20630/40000, loss: 0.3306, lr: 0.005255, batch_cost: 0.7848, reader_cost: 0.00016, ips: 1.2742 samples/sec | ETA 04:13:21 2021-05-09 19:34:53 [INFO] [TRAIN] epoch: 56, iter: 20640/40000, loss: 0.2067, lr: 0.005252, batch_cost: 0.7866, reader_cost: 0.00017, ips: 1.2713 samples/sec | ETA 04:13:47 2021-05-09 19:35:01 [INFO] [TRAIN] epoch: 56, iter: 20650/40000, loss: 0.2626, lr: 0.005250, batch_cost: 0.7856, reader_cost: 0.00017, ips: 1.2728 samples/sec | ETA 04:13:22 2021-05-09 19:35:09 [INFO] [TRAIN] epoch: 56, iter: 20660/40000, loss: 0.1553, lr: 0.005248, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2709 samples/sec | ETA 04:13:38 2021-05-09 19:35:17 [INFO] [TRAIN] epoch: 56, iter: 20670/40000, loss: 0.3507, lr: 0.005245, batch_cost: 0.7876, reader_cost: 0.00016, ips: 1.2697 samples/sec | ETA 04:13:43 2021-05-09 19:35:24 [INFO] [TRAIN] epoch: 56, iter: 20680/40000, loss: 0.3622, lr: 0.005243, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2713 samples/sec | ETA 04:13:16 2021-05-09 19:35:32 [INFO] [TRAIN] epoch: 56, iter: 20690/40000, loss: 0.2345, lr: 0.005241, batch_cost: 0.7860, reader_cost: 0.00017, ips: 1.2723 samples/sec | ETA 04:12:57 2021-05-09 19:35:40 [INFO] [TRAIN] epoch: 56, iter: 20700/40000, loss: 0.2599, lr: 0.005238, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2713 samples/sec | ETA 04:13:01 2021-05-09 19:35:48 [INFO] [TRAIN] epoch: 56, iter: 20710/40000, loss: 0.2300, lr: 0.005236, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2729 samples/sec | ETA 04:12:34 2021-05-09 19:35:56 [INFO] [TRAIN] epoch: 56, iter: 20720/40000, loss: 0.3808, lr: 0.005233, batch_cost: 0.7865, reader_cost: 0.00017, ips: 1.2715 samples/sec | ETA 04:12:43 2021-05-09 19:36:04 [INFO] [TRAIN] epoch: 56, iter: 20730/40000, loss: 0.2047, lr: 0.005231, batch_cost: 0.7866, reader_cost: 0.00017, ips: 1.2713 samples/sec | ETA 04:12:37 2021-05-09 19:36:12 [INFO] [TRAIN] epoch: 56, iter: 20740/40000, loss: 0.3495, lr: 0.005229, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 04:12:30 2021-05-09 19:36:19 [INFO] [TRAIN] epoch: 56, iter: 20750/40000, loss: 0.4395, lr: 0.005226, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 04:12:12 2021-05-09 19:36:27 [INFO] [TRAIN] epoch: 56, iter: 20760/40000, loss: 0.5132, lr: 0.005224, batch_cost: 0.7853, reader_cost: 0.00015, ips: 1.2734 samples/sec | ETA 04:11:48 2021-05-09 19:36:35 [INFO] [TRAIN] epoch: 56, iter: 20770/40000, loss: 0.5426, lr: 0.005221, batch_cost: 0.7858, reader_cost: 0.00017, ips: 1.2726 samples/sec | ETA 04:11:50 2021-05-09 19:36:43 [INFO] [TRAIN] epoch: 56, iter: 20780/40000, loss: 0.2765, lr: 0.005219, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2727 samples/sec | ETA 04:11:42 2021-05-09 19:36:51 [INFO] [TRAIN] epoch: 56, iter: 20790/40000, loss: 0.1261, lr: 0.005217, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2733 samples/sec | ETA 04:11:26 2021-05-09 19:36:59 [INFO] [TRAIN] epoch: 56, iter: 20800/40000, loss: 0.2828, lr: 0.005214, batch_cost: 0.7871, reader_cost: 0.00018, ips: 1.2705 samples/sec | ETA 04:11:52 2021-05-09 19:37:07 [INFO] [TRAIN] epoch: 56, iter: 20810/40000, loss: 0.1897, lr: 0.005212, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 04:11:32 2021-05-09 19:37:14 [INFO] [TRAIN] epoch: 56, iter: 20820/40000, loss: 0.4505, lr: 0.005209, batch_cost: 0.7863, reader_cost: 0.00018, ips: 1.2718 samples/sec | ETA 04:11:21 2021-05-09 19:37:22 [INFO] [TRAIN] epoch: 56, iter: 20830/40000, loss: 0.3686, lr: 0.005207, batch_cost: 0.7863, reader_cost: 0.00031, ips: 1.2717 samples/sec | ETA 04:11:13 2021-05-09 19:37:33 [INFO] [TRAIN] epoch: 57, iter: 20840/40000, loss: 0.3682, lr: 0.005205, batch_cost: 1.0951, reader_cost: 0.25640, ips: 0.9131 samples/sec | ETA 05:49:42 2021-05-09 19:37:41 [INFO] [TRAIN] epoch: 57, iter: 20850/40000, loss: 0.3679, lr: 0.005202, batch_cost: 0.7919, reader_cost: 0.00030, ips: 1.2628 samples/sec | ETA 04:12:44 2021-05-09 19:37:49 [INFO] [TRAIN] epoch: 57, iter: 20860/40000, loss: 0.4019, lr: 0.005200, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2717 samples/sec | ETA 04:10:51 2021-05-09 19:37:57 [INFO] [TRAIN] epoch: 57, iter: 20870/40000, loss: 0.4684, lr: 0.005197, batch_cost: 0.7862, reader_cost: 0.00017, ips: 1.2719 samples/sec | ETA 04:10:40 2021-05-09 19:38:05 [INFO] [TRAIN] epoch: 57, iter: 20880/40000, loss: 0.4164, lr: 0.005195, batch_cost: 0.7863, reader_cost: 0.00017, ips: 1.2718 samples/sec | ETA 04:10:34 2021-05-09 19:38:13 [INFO] [TRAIN] epoch: 57, iter: 20890/40000, loss: 0.3493, lr: 0.005193, batch_cost: 0.7857, reader_cost: 0.00017, ips: 1.2728 samples/sec | ETA 04:10:14 2021-05-09 19:38:21 [INFO] [TRAIN] epoch: 57, iter: 20900/40000, loss: 0.2028, lr: 0.005190, batch_cost: 0.7873, reader_cost: 0.00016, ips: 1.2702 samples/sec | ETA 04:10:37 2021-05-09 19:38:28 [INFO] [TRAIN] epoch: 57, iter: 20910/40000, loss: 0.1789, lr: 0.005188, batch_cost: 0.7873, reader_cost: 0.00017, ips: 1.2701 samples/sec | ETA 04:10:29 2021-05-09 19:38:36 [INFO] [TRAIN] epoch: 57, iter: 20920/40000, loss: 0.3437, lr: 0.005185, batch_cost: 0.7862, reader_cost: 0.00017, ips: 1.2720 samples/sec | ETA 04:10:00 2021-05-09 19:38:44 [INFO] [TRAIN] epoch: 57, iter: 20930/40000, loss: 0.2098, lr: 0.005183, batch_cost: 0.7885, reader_cost: 0.00015, ips: 1.2682 samples/sec | ETA 04:10:37 2021-05-09 19:38:52 [INFO] [TRAIN] epoch: 57, iter: 20940/40000, loss: 0.2866, lr: 0.005181, batch_cost: 0.7844, reader_cost: 0.00016, ips: 1.2749 samples/sec | ETA 04:09:09 2021-05-09 19:39:00 [INFO] [TRAIN] epoch: 57, iter: 20950/40000, loss: 0.1811, lr: 0.005178, batch_cost: 0.7863, reader_cost: 0.00017, ips: 1.2717 samples/sec | ETA 04:09:39 2021-05-09 19:39:08 [INFO] [TRAIN] epoch: 57, iter: 20960/40000, loss: 0.2517, lr: 0.005176, batch_cost: 0.7876, reader_cost: 0.00015, ips: 1.2696 samples/sec | ETA 04:09:56 2021-05-09 19:39:16 [INFO] [TRAIN] epoch: 57, iter: 20970/40000, loss: 0.3042, lr: 0.005173, batch_cost: 0.7876, reader_cost: 0.00015, ips: 1.2698 samples/sec | ETA 04:09:47 2021-05-09 19:39:23 [INFO] [TRAIN] epoch: 57, iter: 20980/40000, loss: 0.3909, lr: 0.005171, batch_cost: 0.7874, reader_cost: 0.00015, ips: 1.2701 samples/sec | ETA 04:09:35 2021-05-09 19:39:31 [INFO] [TRAIN] epoch: 57, iter: 20990/40000, loss: 0.2785, lr: 0.005169, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2722 samples/sec | ETA 04:09:02 2021-05-09 19:39:39 [INFO] [TRAIN] epoch: 57, iter: 21000/40000, loss: 0.2949, lr: 0.005166, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2736 samples/sec | ETA 04:08:37 2021-05-09 19:39:39 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-09 19:43:10 [INFO] [EVAL] #Images: 500 mIoU: 0.7601 Acc: 0.9562 Kappa: 0.9431 2021-05-09 19:43:10 [INFO] [EVAL] Class IoU: [0.9809 0.8453 0.9178 0.6277 0.6122 0.4783 0.6124 0.7126 0.914 0.6584 0.939 0.7718 0.5687 0.94 0.8338 0.8988 0.8116 0.5965 0.7229] 2021-05-09 19:43:10 [INFO] [EVAL] Class Acc: [0.9912 0.9157 0.9489 0.8101 0.8267 0.7673 0.806 0.8814 0.9455 0.8575 0.9619 0.8536 0.7489 0.9622 0.9507 0.9627 0.924 0.7537 0.8478] 2021-05-09 19:43:59 [INFO] [EVAL] The model with the best validation mIoU (0.7601) was saved at iter 21000. 2021-05-09 19:44:07 [INFO] [TRAIN] epoch: 57, iter: 21010/40000, loss: 0.2165, lr: 0.005164, batch_cost: 0.7821, reader_cost: 0.00025, ips: 1.2785 samples/sec | ETA 04:07:32 2021-05-09 19:44:15 [INFO] [TRAIN] epoch: 57, iter: 21020/40000, loss: 0.2161, lr: 0.005161, batch_cost: 0.7830, reader_cost: 0.00033, ips: 1.2772 samples/sec | ETA 04:07:40 2021-05-09 19:44:22 [INFO] [TRAIN] epoch: 57, iter: 21030/40000, loss: 0.0756, lr: 0.005159, batch_cost: 0.7840, reader_cost: 0.00016, ips: 1.2755 samples/sec | ETA 04:07:52 2021-05-09 19:44:30 [INFO] [TRAIN] epoch: 57, iter: 21040/40000, loss: 0.3338, lr: 0.005157, batch_cost: 0.7855, reader_cost: 0.00014, ips: 1.2731 samples/sec | ETA 04:08:12 2021-05-09 19:44:38 [INFO] [TRAIN] epoch: 57, iter: 21050/40000, loss: 0.2734, lr: 0.005154, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2727 samples/sec | ETA 04:08:09 2021-05-09 19:44:46 [INFO] [TRAIN] epoch: 57, iter: 21060/40000, loss: 0.1518, lr: 0.005152, batch_cost: 0.7870, reader_cost: 0.00017, ips: 1.2706 samples/sec | ETA 04:08:26 2021-05-09 19:44:54 [INFO] [TRAIN] epoch: 57, iter: 21070/40000, loss: 0.2478, lr: 0.005149, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 04:08:00 2021-05-09 19:45:02 [INFO] [TRAIN] epoch: 57, iter: 21080/40000, loss: 0.1685, lr: 0.005147, batch_cost: 0.7852, reader_cost: 0.00017, ips: 1.2735 samples/sec | ETA 04:07:36 2021-05-09 19:45:10 [INFO] [TRAIN] epoch: 57, iter: 21090/40000, loss: 0.2928, lr: 0.005145, batch_cost: 0.7876, reader_cost: 0.00014, ips: 1.2697 samples/sec | ETA 04:08:12 2021-05-09 19:45:18 [INFO] [TRAIN] epoch: 57, iter: 21100/40000, loss: 0.1400, lr: 0.005142, batch_cost: 0.7872, reader_cost: 0.00015, ips: 1.2703 samples/sec | ETA 04:07:57 2021-05-09 19:45:25 [INFO] [TRAIN] epoch: 57, iter: 21110/40000, loss: 0.4069, lr: 0.005140, batch_cost: 0.7872, reader_cost: 0.00015, ips: 1.2703 samples/sec | ETA 04:07:50 2021-05-09 19:45:33 [INFO] [TRAIN] epoch: 57, iter: 21120/40000, loss: 0.5894, lr: 0.005137, batch_cost: 0.7874, reader_cost: 0.00014, ips: 1.2701 samples/sec | ETA 04:07:45 2021-05-09 19:45:41 [INFO] [TRAIN] epoch: 57, iter: 21130/40000, loss: 0.4010, lr: 0.005135, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2714 samples/sec | ETA 04:07:22 2021-05-09 19:45:49 [INFO] [TRAIN] epoch: 57, iter: 21140/40000, loss: 0.5976, lr: 0.005133, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2718 samples/sec | ETA 04:07:09 2021-05-09 19:45:57 [INFO] [TRAIN] epoch: 57, iter: 21150/40000, loss: 0.6133, lr: 0.005130, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2725 samples/sec | ETA 04:06:52 2021-05-09 19:46:05 [INFO] [TRAIN] epoch: 57, iter: 21160/40000, loss: 0.1997, lr: 0.005128, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2706 samples/sec | ETA 04:07:07 2021-05-09 19:46:13 [INFO] [TRAIN] epoch: 57, iter: 21170/40000, loss: 0.2261, lr: 0.005125, batch_cost: 0.7876, reader_cost: 0.00016, ips: 1.2697 samples/sec | ETA 04:07:10 2021-05-09 19:46:20 [INFO] [TRAIN] epoch: 57, iter: 21180/40000, loss: 0.2679, lr: 0.005123, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 04:06:41 2021-05-09 19:46:28 [INFO] [TRAIN] epoch: 57, iter: 21190/40000, loss: 0.3487, lr: 0.005121, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2723 samples/sec | ETA 04:06:24 2021-05-09 19:46:36 [INFO] [TRAIN] epoch: 57, iter: 21200/40000, loss: 0.3763, lr: 0.005118, batch_cost: 0.7834, reader_cost: 0.00013, ips: 1.2765 samples/sec | ETA 04:05:28 2021-05-09 19:46:47 [INFO] [TRAIN] epoch: 58, iter: 21210/40000, loss: 0.4760, lr: 0.005116, batch_cost: 1.0895, reader_cost: 0.22706, ips: 0.9178 samples/sec | ETA 05:41:12 2021-05-09 19:46:55 [INFO] [TRAIN] epoch: 58, iter: 21220/40000, loss: 0.4097, lr: 0.005113, batch_cost: 0.7995, reader_cost: 0.00032, ips: 1.2508 samples/sec | ETA 04:10:14 2021-05-09 19:47:03 [INFO] [TRAIN] epoch: 58, iter: 21230/40000, loss: 0.4908, lr: 0.005111, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2718 samples/sec | ETA 04:05:58 2021-05-09 19:47:11 [INFO] [TRAIN] epoch: 58, iter: 21240/40000, loss: 0.3091, lr: 0.005109, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2728 samples/sec | ETA 04:05:39 2021-05-09 19:47:19 [INFO] [TRAIN] epoch: 58, iter: 21250/40000, loss: 0.4144, lr: 0.005106, batch_cost: 0.7875, reader_cost: 0.00016, ips: 1.2698 samples/sec | ETA 04:06:06 2021-05-09 19:47:27 [INFO] [TRAIN] epoch: 58, iter: 21260/40000, loss: 0.4186, lr: 0.005104, batch_cost: 0.7870, reader_cost: 0.00015, ips: 1.2706 samples/sec | ETA 04:05:48 2021-05-09 19:47:34 [INFO] [TRAIN] epoch: 58, iter: 21270/40000, loss: 0.3083, lr: 0.005101, batch_cost: 0.7878, reader_cost: 0.00015, ips: 1.2694 samples/sec | ETA 04:05:55 2021-05-09 19:47:42 [INFO] [TRAIN] epoch: 58, iter: 21280/40000, loss: 0.1276, lr: 0.005099, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2714 samples/sec | ETA 04:05:24 2021-05-09 19:47:50 [INFO] [TRAIN] epoch: 58, iter: 21290/40000, loss: 0.4580, lr: 0.005097, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2723 samples/sec | ETA 04:05:05 2021-05-09 19:47:58 [INFO] [TRAIN] epoch: 58, iter: 21300/40000, loss: 0.2239, lr: 0.005094, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2710 samples/sec | ETA 04:05:12 2021-05-09 19:48:06 [INFO] [TRAIN] epoch: 58, iter: 21310/40000, loss: 0.2774, lr: 0.005092, batch_cost: 0.7872, reader_cost: 0.00015, ips: 1.2704 samples/sec | ETA 04:05:12 2021-05-09 19:48:14 [INFO] [TRAIN] epoch: 58, iter: 21320/40000, loss: 0.2451, lr: 0.005089, batch_cost: 0.7884, reader_cost: 0.00018, ips: 1.2683 samples/sec | ETA 04:05:27 2021-05-09 19:48:22 [INFO] [TRAIN] epoch: 58, iter: 21330/40000, loss: 0.1732, lr: 0.005087, batch_cost: 0.7869, reader_cost: 0.00017, ips: 1.2709 samples/sec | ETA 04:04:50 2021-05-09 19:48:30 [INFO] [TRAIN] epoch: 58, iter: 21340/40000, loss: 0.3395, lr: 0.005085, batch_cost: 0.7876, reader_cost: 0.00015, ips: 1.2696 samples/sec | ETA 04:04:56 2021-05-09 19:48:37 [INFO] [TRAIN] epoch: 58, iter: 21350/40000, loss: 0.4790, lr: 0.005082, batch_cost: 0.7876, reader_cost: 0.00017, ips: 1.2697 samples/sec | ETA 04:04:48 2021-05-09 19:48:45 [INFO] [TRAIN] epoch: 58, iter: 21360/40000, loss: 0.3621, lr: 0.005080, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 04:04:19 2021-05-09 19:48:53 [INFO] [TRAIN] epoch: 58, iter: 21370/40000, loss: 0.2182, lr: 0.005077, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2721 samples/sec | ETA 04:04:05 2021-05-09 19:49:01 [INFO] [TRAIN] epoch: 58, iter: 21380/40000, loss: 0.1587, lr: 0.005075, batch_cost: 0.7865, reader_cost: 0.00014, ips: 1.2715 samples/sec | ETA 04:04:04 2021-05-09 19:49:09 [INFO] [TRAIN] epoch: 58, iter: 21390/40000, loss: 0.2117, lr: 0.005072, batch_cost: 0.7864, reader_cost: 0.00014, ips: 1.2716 samples/sec | ETA 04:03:54 2021-05-09 19:49:17 [INFO] [TRAIN] epoch: 58, iter: 21400/40000, loss: 0.0898, lr: 0.005070, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2725 samples/sec | ETA 04:03:36 2021-05-09 19:49:25 [INFO] [TRAIN] epoch: 58, iter: 21410/40000, loss: 0.2653, lr: 0.005068, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2730 samples/sec | ETA 04:03:23 2021-05-09 19:49:32 [INFO] [TRAIN] epoch: 58, iter: 21420/40000, loss: 0.3071, lr: 0.005065, batch_cost: 0.7874, reader_cost: 0.00015, ips: 1.2700 samples/sec | ETA 04:03:49 2021-05-09 19:49:40 [INFO] [TRAIN] epoch: 58, iter: 21430/40000, loss: 0.4473, lr: 0.005063, batch_cost: 0.7859, reader_cost: 0.00017, ips: 1.2724 samples/sec | ETA 04:03:14 2021-05-09 19:49:48 [INFO] [TRAIN] epoch: 58, iter: 21440/40000, loss: 0.3140, lr: 0.005060, batch_cost: 0.7838, reader_cost: 0.00016, ips: 1.2758 samples/sec | ETA 04:02:28 2021-05-09 19:49:56 [INFO] [TRAIN] epoch: 58, iter: 21450/40000, loss: 0.2518, lr: 0.005058, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2733 samples/sec | ETA 04:02:48 2021-05-09 19:50:04 [INFO] [TRAIN] epoch: 58, iter: 21460/40000, loss: 0.2361, lr: 0.005056, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 04:02:43 2021-05-09 19:50:12 [INFO] [TRAIN] epoch: 58, iter: 21470/40000, loss: 0.2989, lr: 0.005053, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 04:02:56 2021-05-09 19:50:20 [INFO] [TRAIN] epoch: 58, iter: 21480/40000, loss: 0.3358, lr: 0.005051, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2736 samples/sec | ETA 04:02:21 2021-05-09 19:50:27 [INFO] [TRAIN] epoch: 58, iter: 21490/40000, loss: 0.4063, lr: 0.005048, batch_cost: 0.7875, reader_cost: 0.00015, ips: 1.2699 samples/sec | ETA 04:02:55 2021-05-09 19:50:35 [INFO] [TRAIN] epoch: 58, iter: 21500/40000, loss: 0.3595, lr: 0.005046, batch_cost: 0.7879, reader_cost: 0.00016, ips: 1.2691 samples/sec | ETA 04:02:57 2021-05-09 19:50:43 [INFO] [TRAIN] epoch: 58, iter: 21510/40000, loss: 0.5231, lr: 0.005044, batch_cost: 0.7856, reader_cost: 0.00017, ips: 1.2728 samples/sec | ETA 04:02:06 2021-05-09 19:50:51 [INFO] [TRAIN] epoch: 58, iter: 21520/40000, loss: 0.2885, lr: 0.005041, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2718 samples/sec | ETA 04:02:10 2021-05-09 19:50:59 [INFO] [TRAIN] epoch: 58, iter: 21530/40000, loss: 0.3138, lr: 0.005039, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2736 samples/sec | ETA 04:01:41 2021-05-09 19:51:07 [INFO] [TRAIN] epoch: 58, iter: 21540/40000, loss: 0.1103, lr: 0.005036, batch_cost: 0.7843, reader_cost: 0.00016, ips: 1.2750 samples/sec | ETA 04:01:18 2021-05-09 19:51:15 [INFO] [TRAIN] epoch: 58, iter: 21550/40000, loss: 0.1992, lr: 0.005034, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2718 samples/sec | ETA 04:01:46 2021-05-09 19:51:22 [INFO] [TRAIN] epoch: 58, iter: 21560/40000, loss: 0.3074, lr: 0.005032, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2721 samples/sec | ETA 04:01:35 2021-05-09 19:51:30 [INFO] [TRAIN] epoch: 58, iter: 21570/40000, loss: 0.2982, lr: 0.005029, batch_cost: 0.7837, reader_cost: 0.00013, ips: 1.2760 samples/sec | ETA 04:00:43 2021-05-09 19:51:41 [INFO] [TRAIN] epoch: 59, iter: 21580/40000, loss: 0.2653, lr: 0.005027, batch_cost: 1.0911, reader_cost: 0.29795, ips: 0.9165 samples/sec | ETA 05:34:58 2021-05-09 19:51:49 [INFO] [TRAIN] epoch: 59, iter: 21590/40000, loss: 0.2524, lr: 0.005024, batch_cost: 0.7921, reader_cost: 0.00034, ips: 1.2625 samples/sec | ETA 04:03:02 2021-05-09 19:51:57 [INFO] [TRAIN] epoch: 59, iter: 21600/40000, loss: 0.4097, lr: 0.005022, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2734 samples/sec | ETA 04:00:49 2021-05-09 19:52:05 [INFO] [TRAIN] epoch: 59, iter: 21610/40000, loss: 0.4727, lr: 0.005020, batch_cost: 0.7874, reader_cost: 0.00016, ips: 1.2701 samples/sec | ETA 04:01:19 2021-05-09 19:52:13 [INFO] [TRAIN] epoch: 59, iter: 21620/40000, loss: 0.3629, lr: 0.005017, batch_cost: 0.7848, reader_cost: 0.00016, ips: 1.2743 samples/sec | ETA 04:00:23 2021-05-09 19:52:21 [INFO] [TRAIN] epoch: 59, iter: 21630/40000, loss: 0.3797, lr: 0.005015, batch_cost: 0.7875, reader_cost: 0.00018, ips: 1.2698 samples/sec | ETA 04:01:06 2021-05-09 19:52:28 [INFO] [TRAIN] epoch: 59, iter: 21640/40000, loss: 0.1998, lr: 0.005012, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2716 samples/sec | ETA 04:00:38 2021-05-09 19:52:36 [INFO] [TRAIN] epoch: 59, iter: 21650/40000, loss: 0.0968, lr: 0.005010, batch_cost: 0.7862, reader_cost: 0.00017, ips: 1.2720 samples/sec | ETA 04:00:25 2021-05-09 19:52:44 [INFO] [TRAIN] epoch: 59, iter: 21660/40000, loss: 0.3050, lr: 0.005008, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2710 samples/sec | ETA 04:00:29 2021-05-09 19:52:52 [INFO] [TRAIN] epoch: 59, iter: 21670/40000, loss: 0.2828, lr: 0.005005, batch_cost: 0.7886, reader_cost: 0.00015, ips: 1.2680 samples/sec | ETA 04:00:55 2021-05-09 19:53:00 [INFO] [TRAIN] epoch: 59, iter: 21680/40000, loss: 0.2857, lr: 0.005003, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2720 samples/sec | ETA 04:00:03 2021-05-09 19:53:08 [INFO] [TRAIN] epoch: 59, iter: 21690/40000, loss: 0.3071, lr: 0.005000, batch_cost: 0.7881, reader_cost: 0.00016, ips: 1.2689 samples/sec | ETA 04:00:29 2021-05-09 19:53:16 [INFO] [TRAIN] epoch: 59, iter: 21700/40000, loss: 0.1096, lr: 0.004998, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2710 samples/sec | ETA 03:59:57 2021-05-09 19:53:24 [INFO] [TRAIN] epoch: 59, iter: 21710/40000, loss: 0.3526, lr: 0.004995, batch_cost: 0.7875, reader_cost: 0.00015, ips: 1.2699 samples/sec | ETA 04:00:03 2021-05-09 19:53:31 [INFO] [TRAIN] epoch: 59, iter: 21720/40000, loss: 0.3192, lr: 0.004993, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2714 samples/sec | ETA 03:59:38 2021-05-09 19:53:39 [INFO] [TRAIN] epoch: 59, iter: 21730/40000, loss: 0.3191, lr: 0.004991, batch_cost: 0.7864, reader_cost: 0.00017, ips: 1.2716 samples/sec | ETA 03:59:28 2021-05-09 19:53:47 [INFO] [TRAIN] epoch: 59, iter: 21740/40000, loss: 0.3022, lr: 0.004988, batch_cost: 0.7870, reader_cost: 0.00018, ips: 1.2707 samples/sec | ETA 03:59:30 2021-05-09 19:53:55 [INFO] [TRAIN] epoch: 59, iter: 21750/40000, loss: 0.2685, lr: 0.004986, batch_cost: 0.7876, reader_cost: 0.00017, ips: 1.2696 samples/sec | ETA 03:59:34 2021-05-09 19:54:03 [INFO] [TRAIN] epoch: 59, iter: 21760/40000, loss: 0.2893, lr: 0.004983, batch_cost: 0.7858, reader_cost: 0.00015, ips: 1.2725 samples/sec | ETA 03:58:53 2021-05-09 19:54:11 [INFO] [TRAIN] epoch: 59, iter: 21770/40000, loss: 0.2890, lr: 0.004981, batch_cost: 0.7851, reader_cost: 0.00016, ips: 1.2737 samples/sec | ETA 03:58:32 2021-05-09 19:54:19 [INFO] [TRAIN] epoch: 59, iter: 21780/40000, loss: 0.1516, lr: 0.004979, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2733 samples/sec | ETA 03:58:29 2021-05-09 19:54:26 [INFO] [TRAIN] epoch: 59, iter: 21790/40000, loss: 0.3136, lr: 0.004976, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2714 samples/sec | ETA 03:58:42 2021-05-09 19:54:34 [INFO] [TRAIN] epoch: 59, iter: 21800/40000, loss: 0.1700, lr: 0.004974, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2719 samples/sec | ETA 03:58:28 2021-05-09 19:54:42 [INFO] [TRAIN] epoch: 59, iter: 21810/40000, loss: 0.2900, lr: 0.004971, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 03:58:19 2021-05-09 19:54:50 [INFO] [TRAIN] epoch: 59, iter: 21820/40000, loss: 0.1451, lr: 0.004969, batch_cost: 0.7838, reader_cost: 0.00016, ips: 1.2758 samples/sec | ETA 03:57:29 2021-05-09 19:54:58 [INFO] [TRAIN] epoch: 59, iter: 21830/40000, loss: 0.2398, lr: 0.004967, batch_cost: 0.7873, reader_cost: 0.00016, ips: 1.2702 samples/sec | ETA 03:58:25 2021-05-09 19:55:06 [INFO] [TRAIN] epoch: 59, iter: 21840/40000, loss: 0.2008, lr: 0.004964, batch_cost: 0.7851, reader_cost: 0.00015, ips: 1.2737 samples/sec | ETA 03:57:37 2021-05-09 19:55:14 [INFO] [TRAIN] epoch: 59, iter: 21850/40000, loss: 0.3763, lr: 0.004962, batch_cost: 0.7851, reader_cost: 0.00016, ips: 1.2737 samples/sec | ETA 03:57:29 2021-05-09 19:55:21 [INFO] [TRAIN] epoch: 59, iter: 21860/40000, loss: 0.4546, lr: 0.004959, batch_cost: 0.7875, reader_cost: 0.00016, ips: 1.2698 samples/sec | ETA 03:58:05 2021-05-09 19:55:29 [INFO] [TRAIN] epoch: 59, iter: 21870/40000, loss: 0.3409, lr: 0.004957, batch_cost: 0.7860, reader_cost: 0.00017, ips: 1.2722 samples/sec | ETA 03:57:30 2021-05-09 19:55:37 [INFO] [TRAIN] epoch: 59, iter: 21880/40000, loss: 0.5472, lr: 0.004955, batch_cost: 0.7864, reader_cost: 0.00018, ips: 1.2717 samples/sec | ETA 03:57:28 2021-05-09 19:55:45 [INFO] [TRAIN] epoch: 59, iter: 21890/40000, loss: 0.3535, lr: 0.004952, batch_cost: 0.7846, reader_cost: 0.00017, ips: 1.2745 samples/sec | ETA 03:56:49 2021-05-09 19:55:53 [INFO] [TRAIN] epoch: 59, iter: 21900/40000, loss: 0.3931, lr: 0.004950, batch_cost: 0.7856, reader_cost: 0.00018, ips: 1.2730 samples/sec | ETA 03:56:58 2021-05-09 19:56:01 [INFO] [TRAIN] epoch: 59, iter: 21910/40000, loss: 0.1557, lr: 0.004947, batch_cost: 0.7866, reader_cost: 0.00017, ips: 1.2713 samples/sec | ETA 03:57:09 2021-05-09 19:56:09 [INFO] [TRAIN] epoch: 59, iter: 21920/40000, loss: 0.2940, lr: 0.004945, batch_cost: 0.7864, reader_cost: 0.00017, ips: 1.2716 samples/sec | ETA 03:56:58 2021-05-09 19:56:16 [INFO] [TRAIN] epoch: 59, iter: 21930/40000, loss: 0.3327, lr: 0.004942, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2713 samples/sec | ETA 03:56:53 2021-05-09 19:56:24 [INFO] [TRAIN] epoch: 59, iter: 21940/40000, loss: 0.3833, lr: 0.004940, batch_cost: 0.7839, reader_cost: 0.00014, ips: 1.2757 samples/sec | ETA 03:55:56 2021-05-09 19:56:35 [INFO] [TRAIN] epoch: 60, iter: 21950/40000, loss: 0.3475, lr: 0.004938, batch_cost: 1.0924, reader_cost: 0.28199, ips: 0.9154 samples/sec | ETA 05:28:37 2021-05-09 19:56:43 [INFO] [TRAIN] epoch: 60, iter: 21960/40000, loss: 0.2209, lr: 0.004935, batch_cost: 0.7969, reader_cost: 0.00033, ips: 1.2548 samples/sec | ETA 03:59:36 2021-05-09 19:56:51 [INFO] [TRAIN] epoch: 60, iter: 21970/40000, loss: 0.4878, lr: 0.004933, batch_cost: 0.7849, reader_cost: 0.00016, ips: 1.2741 samples/sec | ETA 03:55:50 2021-05-09 19:56:59 [INFO] [TRAIN] epoch: 60, iter: 21980/40000, loss: 0.2400, lr: 0.004930, batch_cost: 0.7849, reader_cost: 0.00015, ips: 1.2741 samples/sec | ETA 03:55:43 2021-05-09 19:57:07 [INFO] [TRAIN] epoch: 60, iter: 21990/40000, loss: 0.5326, lr: 0.004928, batch_cost: 0.7858, reader_cost: 0.00017, ips: 1.2725 samples/sec | ETA 03:55:52 2021-05-09 19:57:15 [INFO] [TRAIN] epoch: 60, iter: 22000/40000, loss: 0.4463, lr: 0.004926, batch_cost: 0.7851, reader_cost: 0.00016, ips: 1.2737 samples/sec | ETA 03:55:32 2021-05-09 19:57:15 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-09 20:00:46 [INFO] [EVAL] #Images: 500 mIoU: 0.7469 Acc: 0.9533 Kappa: 0.9395 2021-05-09 20:00:46 [INFO] [EVAL] Class IoU: [0.9761 0.8216 0.9155 0.6288 0.6142 0.4837 0.6031 0.7126 0.9127 0.6416 0.9397 0.7674 0.5773 0.9367 0.8254 0.8718 0.6759 0.5707 0.7159] 2021-05-09 20:00:46 [INFO] [EVAL] Class Acc: [0.9947 0.8655 0.9476 0.7988 0.8247 0.738 0.8133 0.8805 0.9483 0.8054 0.9658 0.8632 0.7687 0.9608 0.9227 0.9585 0.9877 0.7569 0.8281] 2021-05-09 20:01:14 [INFO] [EVAL] The model with the best validation mIoU (0.7601) was saved at iter 21000. 2021-05-09 20:01:22 [INFO] [TRAIN] epoch: 60, iter: 22010/40000, loss: 0.4373, lr: 0.004923, batch_cost: 0.7833, reader_cost: 0.00059, ips: 1.2767 samples/sec | ETA 03:54:51 2021-05-09 20:01:30 [INFO] [TRAIN] epoch: 60, iter: 22020/40000, loss: 0.1647, lr: 0.004921, batch_cost: 0.7851, reader_cost: 0.00033, ips: 1.2737 samples/sec | ETA 03:55:16 2021-05-09 20:01:38 [INFO] [TRAIN] epoch: 60, iter: 22030/40000, loss: 0.2322, lr: 0.004918, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2735 samples/sec | ETA 03:55:10 2021-05-09 20:01:46 [INFO] [TRAIN] epoch: 60, iter: 22040/40000, loss: 0.3675, lr: 0.004916, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 03:55:25 2021-05-09 20:01:53 [INFO] [TRAIN] epoch: 60, iter: 22050/40000, loss: 0.2352, lr: 0.004914, batch_cost: 0.7876, reader_cost: 0.00014, ips: 1.2697 samples/sec | ETA 03:55:37 2021-05-09 20:02:01 [INFO] [TRAIN] epoch: 60, iter: 22060/40000, loss: 0.2956, lr: 0.004911, batch_cost: 0.7847, reader_cost: 0.00015, ips: 1.2743 samples/sec | ETA 03:54:37 2021-05-09 20:02:09 [INFO] [TRAIN] epoch: 60, iter: 22070/40000, loss: 0.1361, lr: 0.004909, batch_cost: 0.7854, reader_cost: 0.00017, ips: 1.2733 samples/sec | ETA 03:54:41 2021-05-09 20:02:17 [INFO] [TRAIN] epoch: 60, iter: 22080/40000, loss: 0.2742, lr: 0.004906, batch_cost: 0.7878, reader_cost: 0.00016, ips: 1.2694 samples/sec | ETA 03:55:17 2021-05-09 20:02:25 [INFO] [TRAIN] epoch: 60, iter: 22090/40000, loss: 0.2489, lr: 0.004904, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2719 samples/sec | ETA 03:54:41 2021-05-09 20:02:33 [INFO] [TRAIN] epoch: 60, iter: 22100/40000, loss: 0.3536, lr: 0.004901, batch_cost: 0.7861, reader_cost: 0.00014, ips: 1.2720 samples/sec | ETA 03:54:31 2021-05-09 20:02:41 [INFO] [TRAIN] epoch: 60, iter: 22110/40000, loss: 0.2714, lr: 0.004899, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2713 samples/sec | ETA 03:54:32 2021-05-09 20:02:49 [INFO] [TRAIN] epoch: 60, iter: 22120/40000, loss: 0.2506, lr: 0.004897, batch_cost: 0.7858, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 03:54:09 2021-05-09 20:02:56 [INFO] [TRAIN] epoch: 60, iter: 22130/40000, loss: 0.1465, lr: 0.004894, batch_cost: 0.7867, reader_cost: 0.00015, ips: 1.2711 samples/sec | ETA 03:54:18 2021-05-09 20:03:04 [INFO] [TRAIN] epoch: 60, iter: 22140/40000, loss: 0.1477, lr: 0.004892, batch_cost: 0.7867, reader_cost: 0.00015, ips: 1.2711 samples/sec | ETA 03:54:10 2021-05-09 20:03:12 [INFO] [TRAIN] epoch: 60, iter: 22150/40000, loss: 0.2181, lr: 0.004889, batch_cost: 0.7853, reader_cost: 0.00015, ips: 1.2735 samples/sec | ETA 03:53:36 2021-05-09 20:03:20 [INFO] [TRAIN] epoch: 60, iter: 22160/40000, loss: 0.3595, lr: 0.004887, batch_cost: 0.7867, reader_cost: 0.00015, ips: 1.2712 samples/sec | ETA 03:53:53 2021-05-09 20:03:28 [INFO] [TRAIN] epoch: 60, iter: 22170/40000, loss: 0.2753, lr: 0.004885, batch_cost: 0.7874, reader_cost: 0.00015, ips: 1.2699 samples/sec | ETA 03:54:00 2021-05-09 20:03:36 [INFO] [TRAIN] epoch: 60, iter: 22180/40000, loss: 0.3005, lr: 0.004882, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2719 samples/sec | ETA 03:53:30 2021-05-09 20:03:44 [INFO] [TRAIN] epoch: 60, iter: 22190/40000, loss: 0.3253, lr: 0.004880, batch_cost: 0.7853, reader_cost: 0.00015, ips: 1.2734 samples/sec | ETA 03:53:05 2021-05-09 20:03:51 [INFO] [TRAIN] epoch: 60, iter: 22200/40000, loss: 0.2261, lr: 0.004877, batch_cost: 0.7875, reader_cost: 0.00016, ips: 1.2698 samples/sec | ETA 03:53:38 2021-05-09 20:03:59 [INFO] [TRAIN] epoch: 60, iter: 22210/40000, loss: 0.2371, lr: 0.004875, batch_cost: 0.7878, reader_cost: 0.00016, ips: 1.2694 samples/sec | ETA 03:53:34 2021-05-09 20:04:07 [INFO] [TRAIN] epoch: 60, iter: 22220/40000, loss: 0.3775, lr: 0.004872, batch_cost: 0.7874, reader_cost: 0.00016, ips: 1.2700 samples/sec | ETA 03:53:19 2021-05-09 20:04:15 [INFO] [TRAIN] epoch: 60, iter: 22230/40000, loss: 0.4066, lr: 0.004870, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2724 samples/sec | ETA 03:52:45 2021-05-09 20:04:23 [INFO] [TRAIN] epoch: 60, iter: 22240/40000, loss: 0.3575, lr: 0.004868, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2712 samples/sec | ETA 03:52:50 2021-05-09 20:04:31 [INFO] [TRAIN] epoch: 60, iter: 22250/40000, loss: 0.4305, lr: 0.004865, batch_cost: 0.7851, reader_cost: 0.00016, ips: 1.2737 samples/sec | ETA 03:52:15 2021-05-09 20:04:39 [INFO] [TRAIN] epoch: 60, iter: 22260/40000, loss: 0.5029, lr: 0.004863, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2725 samples/sec | ETA 03:52:21 2021-05-09 20:04:46 [INFO] [TRAIN] epoch: 60, iter: 22270/40000, loss: 0.3946, lr: 0.004860, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2723 samples/sec | ETA 03:52:15 2021-05-09 20:04:54 [INFO] [TRAIN] epoch: 60, iter: 22280/40000, loss: 0.2676, lr: 0.004858, batch_cost: 0.7844, reader_cost: 0.00017, ips: 1.2749 samples/sec | ETA 03:51:39 2021-05-09 20:05:02 [INFO] [TRAIN] epoch: 60, iter: 22290/40000, loss: 0.2566, lr: 0.004856, batch_cost: 0.7877, reader_cost: 0.00018, ips: 1.2696 samples/sec | ETA 03:52:29 2021-05-09 20:05:10 [INFO] [TRAIN] epoch: 60, iter: 22300/40000, loss: 0.2635, lr: 0.004853, batch_cost: 0.7878, reader_cost: 0.00018, ips: 1.2694 samples/sec | ETA 03:52:23 2021-05-09 20:05:18 [INFO] [TRAIN] epoch: 60, iter: 22310/40000, loss: 0.5022, lr: 0.004851, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2730 samples/sec | ETA 03:51:35 2021-05-09 20:05:26 [INFO] [TRAIN] epoch: 60, iter: 22320/40000, loss: 0.3418, lr: 0.004848, batch_cost: 0.7850, reader_cost: 0.00009, ips: 1.2740 samples/sec | ETA 03:51:18 2021-05-09 20:05:37 [INFO] [TRAIN] epoch: 61, iter: 22330/40000, loss: 0.2119, lr: 0.004846, batch_cost: 1.1107, reader_cost: 0.29941, ips: 0.9003 samples/sec | ETA 05:27:06 2021-05-09 20:05:45 [INFO] [TRAIN] epoch: 61, iter: 22340/40000, loss: 0.3611, lr: 0.004843, batch_cost: 0.7929, reader_cost: 0.00032, ips: 1.2613 samples/sec | ETA 03:53:21 2021-05-09 20:05:53 [INFO] [TRAIN] epoch: 61, iter: 22350/40000, loss: 0.2659, lr: 0.004841, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2707 samples/sec | ETA 03:51:29 2021-05-09 20:06:01 [INFO] [TRAIN] epoch: 61, iter: 22360/40000, loss: 0.4743, lr: 0.004839, batch_cost: 0.7860, reader_cost: 0.00017, ips: 1.2723 samples/sec | ETA 03:51:04 2021-05-09 20:06:08 [INFO] [TRAIN] epoch: 61, iter: 22370/40000, loss: 0.4755, lr: 0.004836, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2713 samples/sec | ETA 03:51:07 2021-05-09 20:06:16 [INFO] [TRAIN] epoch: 61, iter: 22380/40000, loss: 0.4304, lr: 0.004834, batch_cost: 0.7874, reader_cost: 0.00016, ips: 1.2700 samples/sec | ETA 03:51:14 2021-05-09 20:06:24 [INFO] [TRAIN] epoch: 61, iter: 22390/40000, loss: 0.2005, lr: 0.004831, batch_cost: 0.7868, reader_cost: 0.00017, ips: 1.2710 samples/sec | ETA 03:50:55 2021-05-09 20:06:32 [INFO] [TRAIN] epoch: 61, iter: 22400/40000, loss: 0.2331, lr: 0.004829, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2729 samples/sec | ETA 03:50:27 2021-05-09 20:06:40 [INFO] [TRAIN] epoch: 61, iter: 22410/40000, loss: 0.3688, lr: 0.004827, batch_cost: 0.7854, reader_cost: 0.00015, ips: 1.2733 samples/sec | ETA 03:50:14 2021-05-09 20:06:48 [INFO] [TRAIN] epoch: 61, iter: 22420/40000, loss: 0.2214, lr: 0.004824, batch_cost: 0.7877, reader_cost: 0.00016, ips: 1.2695 samples/sec | ETA 03:50:47 2021-05-09 20:06:56 [INFO] [TRAIN] epoch: 61, iter: 22430/40000, loss: 0.3219, lr: 0.004822, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2733 samples/sec | ETA 03:49:59 2021-05-09 20:07:03 [INFO] [TRAIN] epoch: 61, iter: 22440/40000, loss: 0.1531, lr: 0.004819, batch_cost: 0.7873, reader_cost: 0.00017, ips: 1.2701 samples/sec | ETA 03:50:25 2021-05-09 20:07:11 [INFO] [TRAIN] epoch: 61, iter: 22450/40000, loss: 0.3325, lr: 0.004817, batch_cost: 0.7875, reader_cost: 0.00015, ips: 1.2699 samples/sec | ETA 03:50:19 2021-05-09 20:07:19 [INFO] [TRAIN] epoch: 61, iter: 22460/40000, loss: 0.3240, lr: 0.004814, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2719 samples/sec | ETA 03:49:50 2021-05-09 20:07:27 [INFO] [TRAIN] epoch: 61, iter: 22470/40000, loss: 0.3178, lr: 0.004812, batch_cost: 0.7879, reader_cost: 0.00016, ips: 1.2692 samples/sec | ETA 03:50:11 2021-05-09 20:07:35 [INFO] [TRAIN] epoch: 61, iter: 22480/40000, loss: 0.3345, lr: 0.004810, batch_cost: 0.7873, reader_cost: 0.00015, ips: 1.2702 samples/sec | ETA 03:49:53 2021-05-09 20:07:43 [INFO] [TRAIN] epoch: 61, iter: 22490/40000, loss: 0.2157, lr: 0.004807, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2721 samples/sec | ETA 03:49:24 2021-05-09 20:07:51 [INFO] [TRAIN] epoch: 61, iter: 22500/40000, loss: 0.2531, lr: 0.004805, batch_cost: 0.7883, reader_cost: 0.00015, ips: 1.2685 samples/sec | ETA 03:49:55 2021-05-09 20:07:59 [INFO] [TRAIN] epoch: 61, iter: 22510/40000, loss: 0.1951, lr: 0.004802, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2705 samples/sec | ETA 03:49:26 2021-05-09 20:08:06 [INFO] [TRAIN] epoch: 61, iter: 22520/40000, loss: 0.2492, lr: 0.004800, batch_cost: 0.7870, reader_cost: 0.00015, ips: 1.2706 samples/sec | ETA 03:49:17 2021-05-09 20:08:14 [INFO] [TRAIN] epoch: 61, iter: 22530/40000, loss: 0.3839, lr: 0.004798, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2735 samples/sec | ETA 03:48:38 2021-05-09 20:08:22 [INFO] [TRAIN] epoch: 61, iter: 22540/40000, loss: 0.2739, lr: 0.004795, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 03:48:38 2021-05-09 20:08:30 [INFO] [TRAIN] epoch: 61, iter: 22550/40000, loss: 0.2235, lr: 0.004793, batch_cost: 0.7880, reader_cost: 0.00015, ips: 1.2690 samples/sec | ETA 03:49:11 2021-05-09 20:08:38 [INFO] [TRAIN] epoch: 61, iter: 22560/40000, loss: 0.2866, lr: 0.004790, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2728 samples/sec | ETA 03:48:22 2021-05-09 20:08:46 [INFO] [TRAIN] epoch: 61, iter: 22570/40000, loss: 0.1638, lr: 0.004788, batch_cost: 0.7851, reader_cost: 0.00016, ips: 1.2737 samples/sec | ETA 03:48:04 2021-05-09 20:08:54 [INFO] [TRAIN] epoch: 61, iter: 22580/40000, loss: 0.1868, lr: 0.004785, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2731 samples/sec | ETA 03:48:03 2021-05-09 20:09:01 [INFO] [TRAIN] epoch: 61, iter: 22590/40000, loss: 0.1927, lr: 0.004783, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2717 samples/sec | ETA 03:48:10 2021-05-09 20:09:09 [INFO] [TRAIN] epoch: 61, iter: 22600/40000, loss: 0.4393, lr: 0.004781, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2706 samples/sec | ETA 03:48:14 2021-05-09 20:09:17 [INFO] [TRAIN] epoch: 61, iter: 22610/40000, loss: 0.5692, lr: 0.004778, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2721 samples/sec | ETA 03:47:50 2021-05-09 20:09:25 [INFO] [TRAIN] epoch: 61, iter: 22620/40000, loss: 0.4062, lr: 0.004776, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2714 samples/sec | ETA 03:47:50 2021-05-09 20:09:33 [INFO] [TRAIN] epoch: 61, iter: 22630/40000, loss: 0.4112, lr: 0.004773, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2706 samples/sec | ETA 03:47:50 2021-05-09 20:09:41 [INFO] [TRAIN] epoch: 61, iter: 22640/40000, loss: 0.3123, lr: 0.004771, batch_cost: 0.7860, reader_cost: 0.00017, ips: 1.2722 samples/sec | ETA 03:47:25 2021-05-09 20:09:49 [INFO] [TRAIN] epoch: 61, iter: 22650/40000, loss: 0.1535, lr: 0.004768, batch_cost: 0.7853, reader_cost: 0.00015, ips: 1.2734 samples/sec | ETA 03:47:04 2021-05-09 20:09:57 [INFO] [TRAIN] epoch: 61, iter: 22660/40000, loss: 0.2145, lr: 0.004766, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2715 samples/sec | ETA 03:47:17 2021-05-09 20:10:04 [INFO] [TRAIN] epoch: 61, iter: 22670/40000, loss: 0.3087, lr: 0.004764, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2735 samples/sec | ETA 03:46:48 2021-05-09 20:10:12 [INFO] [TRAIN] epoch: 61, iter: 22680/40000, loss: 0.2992, lr: 0.004761, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 03:46:53 2021-05-09 20:10:20 [INFO] [TRAIN] epoch: 61, iter: 22690/40000, loss: 0.3633, lr: 0.004759, batch_cost: 0.7841, reader_cost: 0.00025, ips: 1.2754 samples/sec | ETA 03:46:12 2021-05-09 20:10:31 [INFO] [TRAIN] epoch: 62, iter: 22700/40000, loss: 0.1811, lr: 0.004756, batch_cost: 1.0823, reader_cost: 0.24364, ips: 0.9240 samples/sec | ETA 05:12:03 2021-05-09 20:10:39 [INFO] [TRAIN] epoch: 62, iter: 22710/40000, loss: 0.4799, lr: 0.004754, batch_cost: 0.7947, reader_cost: 0.00034, ips: 1.2584 samples/sec | ETA 03:48:59 2021-05-09 20:10:47 [INFO] [TRAIN] epoch: 62, iter: 22720/40000, loss: 0.3311, lr: 0.004752, batch_cost: 0.7877, reader_cost: 0.00017, ips: 1.2695 samples/sec | ETA 03:46:51 2021-05-09 20:10:55 [INFO] [TRAIN] epoch: 62, iter: 22730/40000, loss: 0.3543, lr: 0.004749, batch_cost: 0.7870, reader_cost: 0.00018, ips: 1.2706 samples/sec | ETA 03:46:31 2021-05-09 20:11:02 [INFO] [TRAIN] epoch: 62, iter: 22740/40000, loss: 0.4220, lr: 0.004747, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2706 samples/sec | ETA 03:46:23 2021-05-09 20:11:10 [INFO] [TRAIN] epoch: 62, iter: 22750/40000, loss: 0.2905, lr: 0.004744, batch_cost: 0.7871, reader_cost: 0.00017, ips: 1.2705 samples/sec | ETA 03:46:16 2021-05-09 20:11:18 [INFO] [TRAIN] epoch: 62, iter: 22760/40000, loss: 0.2209, lr: 0.004742, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2718 samples/sec | ETA 03:45:55 2021-05-09 20:11:26 [INFO] [TRAIN] epoch: 62, iter: 22770/40000, loss: 0.1862, lr: 0.004739, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2718 samples/sec | ETA 03:45:47 2021-05-09 20:11:34 [INFO] [TRAIN] epoch: 62, iter: 22780/40000, loss: 0.3186, lr: 0.004737, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2716 samples/sec | ETA 03:45:42 2021-05-09 20:11:42 [INFO] [TRAIN] epoch: 62, iter: 22790/40000, loss: 0.2428, lr: 0.004735, batch_cost: 0.7881, reader_cost: 0.00014, ips: 1.2689 samples/sec | ETA 03:46:03 2021-05-09 20:11:50 [INFO] [TRAIN] epoch: 62, iter: 22800/40000, loss: 0.2898, lr: 0.004732, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2713 samples/sec | ETA 03:45:29 2021-05-09 20:11:58 [INFO] [TRAIN] epoch: 62, iter: 22810/40000, loss: 0.2154, lr: 0.004730, batch_cost: 0.7877, reader_cost: 0.00016, ips: 1.2695 samples/sec | ETA 03:45:40 2021-05-09 20:12:05 [INFO] [TRAIN] epoch: 62, iter: 22820/40000, loss: 0.2159, lr: 0.004727, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2722 samples/sec | ETA 03:45:03 2021-05-09 20:12:13 [INFO] [TRAIN] epoch: 62, iter: 22830/40000, loss: 0.4013, lr: 0.004725, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2716 samples/sec | ETA 03:45:02 2021-05-09 20:12:21 [INFO] [TRAIN] epoch: 62, iter: 22840/40000, loss: 0.2482, lr: 0.004722, batch_cost: 0.7884, reader_cost: 0.00015, ips: 1.2684 samples/sec | ETA 03:45:28 2021-05-09 20:12:29 [INFO] [TRAIN] epoch: 62, iter: 22850/40000, loss: 0.2949, lr: 0.004720, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 03:44:38 2021-05-09 20:12:37 [INFO] [TRAIN] epoch: 62, iter: 22860/40000, loss: 0.2205, lr: 0.004718, batch_cost: 0.7851, reader_cost: 0.00016, ips: 1.2738 samples/sec | ETA 03:44:16 2021-05-09 20:12:45 [INFO] [TRAIN] epoch: 62, iter: 22870/40000, loss: 0.2336, lr: 0.004715, batch_cost: 0.7875, reader_cost: 0.00015, ips: 1.2698 samples/sec | ETA 03:44:50 2021-05-09 20:12:53 [INFO] [TRAIN] epoch: 62, iter: 22880/40000, loss: 0.1879, lr: 0.004713, batch_cost: 0.7889, reader_cost: 0.00016, ips: 1.2675 samples/sec | ETA 03:45:06 2021-05-09 20:13:01 [INFO] [TRAIN] epoch: 62, iter: 22890/40000, loss: 0.1730, lr: 0.004710, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2719 samples/sec | ETA 03:44:12 2021-05-09 20:13:08 [INFO] [TRAIN] epoch: 62, iter: 22900/40000, loss: 0.2621, lr: 0.004708, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2716 samples/sec | ETA 03:44:07 2021-05-09 20:13:16 [INFO] [TRAIN] epoch: 62, iter: 22910/40000, loss: 0.2619, lr: 0.004705, batch_cost: 0.7854, reader_cost: 0.00017, ips: 1.2732 samples/sec | ETA 03:43:42 2021-05-09 20:13:24 [INFO] [TRAIN] epoch: 62, iter: 22920/40000, loss: 0.3552, lr: 0.004703, batch_cost: 0.7860, reader_cost: 0.00017, ips: 1.2722 samples/sec | ETA 03:43:45 2021-05-09 20:13:32 [INFO] [TRAIN] epoch: 62, iter: 22930/40000, loss: 0.2579, lr: 0.004701, batch_cost: 0.7856, reader_cost: 0.00017, ips: 1.2730 samples/sec | ETA 03:43:29 2021-05-09 20:13:40 [INFO] [TRAIN] epoch: 62, iter: 22940/40000, loss: 0.1643, lr: 0.004698, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 03:43:20 2021-05-09 20:13:48 [INFO] [TRAIN] epoch: 62, iter: 22950/40000, loss: 0.2594, lr: 0.004696, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2725 samples/sec | ETA 03:43:18 2021-05-09 20:13:56 [INFO] [TRAIN] epoch: 62, iter: 22960/40000, loss: 0.2014, lr: 0.004693, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2714 samples/sec | ETA 03:43:22 2021-05-09 20:14:03 [INFO] [TRAIN] epoch: 62, iter: 22970/40000, loss: 0.3604, lr: 0.004691, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2723 samples/sec | ETA 03:43:04 2021-05-09 20:14:11 [INFO] [TRAIN] epoch: 62, iter: 22980/40000, loss: 0.4520, lr: 0.004688, batch_cost: 0.7867, reader_cost: 0.00014, ips: 1.2712 samples/sec | ETA 03:43:09 2021-05-09 20:14:19 [INFO] [TRAIN] epoch: 62, iter: 22990/40000, loss: 0.3840, lr: 0.004686, batch_cost: 0.7849, reader_cost: 0.00017, ips: 1.2741 samples/sec | ETA 03:42:30 2021-05-09 20:14:27 [INFO] [TRAIN] epoch: 62, iter: 23000/40000, loss: 0.5145, lr: 0.004684, batch_cost: 0.7858, reader_cost: 0.00017, ips: 1.2725 samples/sec | ETA 03:42:39 2021-05-09 20:14:27 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-09 20:17:57 [INFO] [EVAL] #Images: 500 mIoU: 0.7466 Acc: 0.9534 Kappa: 0.9396 2021-05-09 20:17:57 [INFO] [EVAL] Class IoU: [0.9792 0.836 0.9139 0.5788 0.6141 0.4663 0.6005 0.7209 0.9104 0.6266 0.9376 0.7644 0.5441 0.9378 0.8342 0.8791 0.7733 0.5521 0.7162] 2021-05-09 20:17:57 [INFO] [EVAL] Class Acc: [0.9917 0.8936 0.9476 0.67 0.7907 0.769 0.8113 0.8846 0.9536 0.7757 0.9628 0.854 0.7659 0.9638 0.9194 0.936 0.8963 0.7011 0.8392] 2021-05-09 20:18:25 [INFO] [EVAL] The model with the best validation mIoU (0.7601) was saved at iter 21000. 2021-05-09 20:18:33 [INFO] [TRAIN] epoch: 62, iter: 23010/40000, loss: 0.3280, lr: 0.004681, batch_cost: 0.7825, reader_cost: 0.00024, ips: 1.2779 samples/sec | ETA 03:41:34 2021-05-09 20:18:41 [INFO] [TRAIN] epoch: 62, iter: 23020/40000, loss: 0.2016, lr: 0.004679, batch_cost: 0.7860, reader_cost: 0.00018, ips: 1.2723 samples/sec | ETA 03:42:25 2021-05-09 20:18:49 [INFO] [TRAIN] epoch: 62, iter: 23030/40000, loss: 0.2301, lr: 0.004676, batch_cost: 0.7847, reader_cost: 0.00018, ips: 1.2743 samples/sec | ETA 03:41:56 2021-05-09 20:18:56 [INFO] [TRAIN] epoch: 62, iter: 23040/40000, loss: 0.1681, lr: 0.004674, batch_cost: 0.7846, reader_cost: 0.00016, ips: 1.2745 samples/sec | ETA 03:41:46 2021-05-09 20:19:04 [INFO] [TRAIN] epoch: 62, iter: 23050/40000, loss: 0.3703, lr: 0.004671, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 03:42:11 2021-05-09 20:19:12 [INFO] [TRAIN] epoch: 62, iter: 23060/40000, loss: 0.3120, lr: 0.004669, batch_cost: 0.7850, reader_cost: 0.00012, ips: 1.2738 samples/sec | ETA 03:41:38 2021-05-09 20:19:23 [INFO] [TRAIN] epoch: 63, iter: 23070/40000, loss: 0.2810, lr: 0.004667, batch_cost: 1.0923, reader_cost: 0.25132, ips: 0.9155 samples/sec | ETA 05:08:12 2021-05-09 20:19:31 [INFO] [TRAIN] epoch: 63, iter: 23080/40000, loss: 0.4336, lr: 0.004664, batch_cost: 0.7998, reader_cost: 0.00035, ips: 1.2504 samples/sec | ETA 03:45:32 2021-05-09 20:19:39 [INFO] [TRAIN] epoch: 63, iter: 23090/40000, loss: 0.3603, lr: 0.004662, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2725 samples/sec | ETA 03:41:28 2021-05-09 20:19:47 [INFO] [TRAIN] epoch: 63, iter: 23100/40000, loss: 0.3961, lr: 0.004659, batch_cost: 0.7886, reader_cost: 0.00017, ips: 1.2681 samples/sec | ETA 03:42:06 2021-05-09 20:19:55 [INFO] [TRAIN] epoch: 63, iter: 23110/40000, loss: 0.4112, lr: 0.004657, batch_cost: 0.7877, reader_cost: 0.00015, ips: 1.2695 samples/sec | ETA 03:41:44 2021-05-09 20:20:03 [INFO] [TRAIN] epoch: 63, iter: 23120/40000, loss: 0.3068, lr: 0.004654, batch_cost: 0.7868, reader_cost: 0.00014, ips: 1.2710 samples/sec | ETA 03:41:20 2021-05-09 20:20:10 [INFO] [TRAIN] epoch: 63, iter: 23130/40000, loss: 0.3694, lr: 0.004652, batch_cost: 0.7886, reader_cost: 0.00015, ips: 1.2680 samples/sec | ETA 03:41:44 2021-05-09 20:20:18 [INFO] [TRAIN] epoch: 63, iter: 23140/40000, loss: 0.2377, lr: 0.004650, batch_cost: 0.7864, reader_cost: 0.00017, ips: 1.2716 samples/sec | ETA 03:40:58 2021-05-09 20:20:26 [INFO] [TRAIN] epoch: 63, iter: 23150/40000, loss: 0.3381, lr: 0.004647, batch_cost: 0.7851, reader_cost: 0.00017, ips: 1.2737 samples/sec | ETA 03:40:29 2021-05-09 20:20:34 [INFO] [TRAIN] epoch: 63, iter: 23160/40000, loss: 0.2586, lr: 0.004645, batch_cost: 0.7870, reader_cost: 0.00015, ips: 1.2706 samples/sec | ETA 03:40:53 2021-05-09 20:20:42 [INFO] [TRAIN] epoch: 63, iter: 23170/40000, loss: 0.2344, lr: 0.004642, batch_cost: 0.7876, reader_cost: 0.00015, ips: 1.2697 samples/sec | ETA 03:40:55 2021-05-09 20:20:50 [INFO] [TRAIN] epoch: 63, iter: 23180/40000, loss: 0.1853, lr: 0.004640, batch_cost: 0.7875, reader_cost: 0.00016, ips: 1.2699 samples/sec | ETA 03:40:45 2021-05-09 20:20:58 [INFO] [TRAIN] epoch: 63, iter: 23190/40000, loss: 0.1699, lr: 0.004637, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2709 samples/sec | ETA 03:40:26 2021-05-09 20:21:06 [INFO] [TRAIN] epoch: 63, iter: 23200/40000, loss: 0.3028, lr: 0.004635, batch_cost: 0.7867, reader_cost: 0.00015, ips: 1.2711 samples/sec | ETA 03:40:16 2021-05-09 20:21:13 [INFO] [TRAIN] epoch: 63, iter: 23210/40000, loss: 0.2820, lr: 0.004633, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 03:40:08 2021-05-09 20:21:21 [INFO] [TRAIN] epoch: 63, iter: 23220/40000, loss: 0.3070, lr: 0.004630, batch_cost: 0.7857, reader_cost: 0.00017, ips: 1.2728 samples/sec | ETA 03:39:43 2021-05-09 20:21:29 [INFO] [TRAIN] epoch: 63, iter: 23230/40000, loss: 0.2781, lr: 0.004628, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 03:39:47 2021-05-09 20:21:37 [INFO] [TRAIN] epoch: 63, iter: 23240/40000, loss: 0.1971, lr: 0.004625, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2724 samples/sec | ETA 03:39:31 2021-05-09 20:21:45 [INFO] [TRAIN] epoch: 63, iter: 23250/40000, loss: 0.2456, lr: 0.004623, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2718 samples/sec | ETA 03:39:29 2021-05-09 20:21:53 [INFO] [TRAIN] epoch: 63, iter: 23260/40000, loss: 0.0438, lr: 0.004620, batch_cost: 0.7870, reader_cost: 0.00017, ips: 1.2707 samples/sec | ETA 03:39:33 2021-05-09 20:22:01 [INFO] [TRAIN] epoch: 63, iter: 23270/40000, loss: 0.2341, lr: 0.004618, batch_cost: 0.7854, reader_cost: 0.00015, ips: 1.2732 samples/sec | ETA 03:38:59 2021-05-09 20:22:08 [INFO] [TRAIN] epoch: 63, iter: 23280/40000, loss: 0.2865, lr: 0.004616, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 03:39:00 2021-05-09 20:22:16 [INFO] [TRAIN] epoch: 63, iter: 23290/40000, loss: 0.1956, lr: 0.004613, batch_cost: 0.7874, reader_cost: 0.00016, ips: 1.2700 samples/sec | ETA 03:39:17 2021-05-09 20:22:24 [INFO] [TRAIN] epoch: 63, iter: 23300/40000, loss: 0.2574, lr: 0.004611, batch_cost: 0.7853, reader_cost: 0.00015, ips: 1.2733 samples/sec | ETA 03:38:35 2021-05-09 20:22:32 [INFO] [TRAIN] epoch: 63, iter: 23310/40000, loss: 0.2000, lr: 0.004608, batch_cost: 0.7853, reader_cost: 0.00017, ips: 1.2734 samples/sec | ETA 03:38:26 2021-05-09 20:22:40 [INFO] [TRAIN] epoch: 63, iter: 23320/40000, loss: 0.2421, lr: 0.004606, batch_cost: 0.7857, reader_cost: 0.00018, ips: 1.2728 samples/sec | ETA 03:38:25 2021-05-09 20:22:48 [INFO] [TRAIN] epoch: 63, iter: 23330/40000, loss: 0.1832, lr: 0.004603, batch_cost: 0.7855, reader_cost: 0.00017, ips: 1.2731 samples/sec | ETA 03:38:14 2021-05-09 20:22:56 [INFO] [TRAIN] epoch: 63, iter: 23340/40000, loss: 0.3971, lr: 0.004601, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2711 samples/sec | ETA 03:38:26 2021-05-09 20:23:03 [INFO] [TRAIN] epoch: 63, iter: 23350/40000, loss: 0.4997, lr: 0.004599, batch_cost: 0.7879, reader_cost: 0.00017, ips: 1.2692 samples/sec | ETA 03:38:38 2021-05-09 20:23:11 [INFO] [TRAIN] epoch: 63, iter: 23360/40000, loss: 0.2649, lr: 0.004596, batch_cost: 0.7876, reader_cost: 0.00017, ips: 1.2696 samples/sec | ETA 03:38:26 2021-05-09 20:23:19 [INFO] [TRAIN] epoch: 63, iter: 23370/40000, loss: 0.5832, lr: 0.004594, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 03:37:51 2021-05-09 20:23:27 [INFO] [TRAIN] epoch: 63, iter: 23380/40000, loss: 0.3559, lr: 0.004591, batch_cost: 0.7860, reader_cost: 0.00017, ips: 1.2722 samples/sec | ETA 03:37:43 2021-05-09 20:23:35 [INFO] [TRAIN] epoch: 63, iter: 23390/40000, loss: 0.2764, lr: 0.004589, batch_cost: 0.7873, reader_cost: 0.00017, ips: 1.2702 samples/sec | ETA 03:37:56 2021-05-09 20:23:43 [INFO] [TRAIN] epoch: 63, iter: 23400/40000, loss: 0.1719, lr: 0.004586, batch_cost: 0.7857, reader_cost: 0.00017, ips: 1.2728 samples/sec | ETA 03:37:22 2021-05-09 20:23:51 [INFO] [TRAIN] epoch: 63, iter: 23410/40000, loss: 0.2246, lr: 0.004584, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2733 samples/sec | ETA 03:37:08 2021-05-09 20:23:59 [INFO] [TRAIN] epoch: 63, iter: 23420/40000, loss: 0.2560, lr: 0.004582, batch_cost: 0.7879, reader_cost: 0.00018, ips: 1.2693 samples/sec | ETA 03:37:42 2021-05-09 20:24:06 [INFO] [TRAIN] epoch: 63, iter: 23430/40000, loss: 0.2782, lr: 0.004579, batch_cost: 0.7862, reader_cost: 0.00014, ips: 1.2720 samples/sec | ETA 03:37:06 2021-05-09 20:24:17 [INFO] [TRAIN] epoch: 64, iter: 23440/40000, loss: 0.3283, lr: 0.004577, batch_cost: 1.0890, reader_cost: 0.27449, ips: 0.9183 samples/sec | ETA 05:00:33 2021-05-09 20:24:25 [INFO] [TRAIN] epoch: 64, iter: 23450/40000, loss: 0.2088, lr: 0.004574, batch_cost: 0.7923, reader_cost: 0.00033, ips: 1.2622 samples/sec | ETA 03:38:31 2021-05-09 20:24:33 [INFO] [TRAIN] epoch: 64, iter: 23460/40000, loss: 0.4674, lr: 0.004572, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2732 samples/sec | ETA 03:36:30 2021-05-09 20:24:41 [INFO] [TRAIN] epoch: 64, iter: 23470/40000, loss: 0.2962, lr: 0.004569, batch_cost: 0.7861, reader_cost: 0.00017, ips: 1.2721 samples/sec | ETA 03:36:34 2021-05-09 20:24:49 [INFO] [TRAIN] epoch: 64, iter: 23480/40000, loss: 0.4344, lr: 0.004567, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2706 samples/sec | ETA 03:36:42 2021-05-09 20:24:57 [INFO] [TRAIN] epoch: 64, iter: 23490/40000, loss: 0.4452, lr: 0.004565, batch_cost: 0.7870, reader_cost: 0.00015, ips: 1.2706 samples/sec | ETA 03:36:33 2021-05-09 20:25:05 [INFO] [TRAIN] epoch: 64, iter: 23500/40000, loss: 0.2047, lr: 0.004562, batch_cost: 0.7873, reader_cost: 0.00015, ips: 1.2701 samples/sec | ETA 03:36:31 2021-05-09 20:25:12 [INFO] [TRAIN] epoch: 64, iter: 23510/40000, loss: 0.1567, lr: 0.004560, batch_cost: 0.7897, reader_cost: 0.00015, ips: 1.2662 samples/sec | ETA 03:37:02 2021-05-09 20:25:20 [INFO] [TRAIN] epoch: 64, iter: 23520/40000, loss: 0.2842, lr: 0.004557, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2728 samples/sec | ETA 03:35:48 2021-05-09 20:25:28 [INFO] [TRAIN] epoch: 64, iter: 23530/40000, loss: 0.2747, lr: 0.004555, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2711 samples/sec | ETA 03:35:57 2021-05-09 20:25:36 [INFO] [TRAIN] epoch: 64, iter: 23540/40000, loss: 0.2208, lr: 0.004552, batch_cost: 0.7862, reader_cost: 0.00017, ips: 1.2719 samples/sec | ETA 03:35:41 2021-05-09 20:25:44 [INFO] [TRAIN] epoch: 64, iter: 23550/40000, loss: 0.3439, lr: 0.004550, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 03:35:28 2021-05-09 20:25:52 [INFO] [TRAIN] epoch: 64, iter: 23560/40000, loss: 0.1740, lr: 0.004548, batch_cost: 0.7875, reader_cost: 0.00015, ips: 1.2698 samples/sec | ETA 03:35:46 2021-05-09 20:26:00 [INFO] [TRAIN] epoch: 64, iter: 23570/40000, loss: 0.2548, lr: 0.004545, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2707 samples/sec | ETA 03:35:30 2021-05-09 20:26:08 [INFO] [TRAIN] epoch: 64, iter: 23580/40000, loss: 0.3155, lr: 0.004543, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2710 samples/sec | ETA 03:35:18 2021-05-09 20:26:15 [INFO] [TRAIN] epoch: 64, iter: 23590/40000, loss: 0.2896, lr: 0.004540, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2714 samples/sec | ETA 03:35:07 2021-05-09 20:26:23 [INFO] [TRAIN] epoch: 64, iter: 23600/40000, loss: 0.2254, lr: 0.004538, batch_cost: 0.7888, reader_cost: 0.00015, ips: 1.2678 samples/sec | ETA 03:35:35 2021-05-09 20:26:31 [INFO] [TRAIN] epoch: 64, iter: 23610/40000, loss: 0.1516, lr: 0.004535, batch_cost: 0.7880, reader_cost: 0.00016, ips: 1.2691 samples/sec | ETA 03:35:14 2021-05-09 20:26:39 [INFO] [TRAIN] epoch: 64, iter: 23620/40000, loss: 0.3520, lr: 0.004533, batch_cost: 0.7849, reader_cost: 0.00018, ips: 1.2740 samples/sec | ETA 03:34:17 2021-05-09 20:26:47 [INFO] [TRAIN] epoch: 64, iter: 23630/40000, loss: 0.1379, lr: 0.004530, batch_cost: 0.7868, reader_cost: 0.00017, ips: 1.2710 samples/sec | ETA 03:34:39 2021-05-09 20:26:55 [INFO] [TRAIN] epoch: 64, iter: 23640/40000, loss: 0.2049, lr: 0.004528, batch_cost: 0.7874, reader_cost: 0.00017, ips: 1.2700 samples/sec | ETA 03:34:41 2021-05-09 20:27:03 [INFO] [TRAIN] epoch: 64, iter: 23650/40000, loss: 0.3091, lr: 0.004526, batch_cost: 0.7865, reader_cost: 0.00017, ips: 1.2714 samples/sec | ETA 03:34:19 2021-05-09 20:27:10 [INFO] [TRAIN] epoch: 64, iter: 23660/40000, loss: 0.1645, lr: 0.004523, batch_cost: 0.7850, reader_cost: 0.00016, ips: 1.2739 samples/sec | ETA 03:33:47 2021-05-09 20:27:18 [INFO] [TRAIN] epoch: 64, iter: 23670/40000, loss: 0.2309, lr: 0.004521, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2728 samples/sec | ETA 03:33:50 2021-05-09 20:27:26 [INFO] [TRAIN] epoch: 64, iter: 23680/40000, loss: 0.2102, lr: 0.004518, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 03:33:45 2021-05-09 20:27:34 [INFO] [TRAIN] epoch: 64, iter: 23690/40000, loss: 0.2272, lr: 0.004516, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2708 samples/sec | ETA 03:33:54 2021-05-09 20:27:42 [INFO] [TRAIN] epoch: 64, iter: 23700/40000, loss: 0.1138, lr: 0.004513, batch_cost: 0.7863, reader_cost: 0.00017, ips: 1.2717 samples/sec | ETA 03:33:37 2021-05-09 20:27:50 [INFO] [TRAIN] epoch: 64, iter: 23710/40000, loss: 0.3250, lr: 0.004511, batch_cost: 0.7858, reader_cost: 0.00014, ips: 1.2725 samples/sec | ETA 03:33:21 2021-05-09 20:27:58 [INFO] [TRAIN] epoch: 64, iter: 23720/40000, loss: 0.4507, lr: 0.004509, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2719 samples/sec | ETA 03:33:20 2021-05-09 20:28:06 [INFO] [TRAIN] epoch: 64, iter: 23730/40000, loss: 0.2892, lr: 0.004506, batch_cost: 0.7876, reader_cost: 0.00015, ips: 1.2697 samples/sec | ETA 03:33:34 2021-05-09 20:28:13 [INFO] [TRAIN] epoch: 64, iter: 23740/40000, loss: 0.4987, lr: 0.004504, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2727 samples/sec | ETA 03:32:56 2021-05-09 20:28:21 [INFO] [TRAIN] epoch: 64, iter: 23750/40000, loss: 0.3571, lr: 0.004501, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2716 samples/sec | ETA 03:32:59 2021-05-09 20:28:29 [INFO] [TRAIN] epoch: 64, iter: 23760/40000, loss: 0.3130, lr: 0.004499, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2709 samples/sec | ETA 03:32:58 2021-05-09 20:28:37 [INFO] [TRAIN] epoch: 64, iter: 23770/40000, loss: 0.1248, lr: 0.004496, batch_cost: 0.7872, reader_cost: 0.00018, ips: 1.2703 samples/sec | ETA 03:32:56 2021-05-09 20:28:45 [INFO] [TRAIN] epoch: 64, iter: 23780/40000, loss: 0.2180, lr: 0.004494, batch_cost: 0.7886, reader_cost: 0.00019, ips: 1.2680 samples/sec | ETA 03:33:11 2021-05-09 20:28:53 [INFO] [TRAIN] epoch: 64, iter: 23790/40000, loss: 0.3943, lr: 0.004491, batch_cost: 0.7877, reader_cost: 0.00016, ips: 1.2695 samples/sec | ETA 03:32:48 2021-05-09 20:29:01 [INFO] [TRAIN] epoch: 64, iter: 23800/40000, loss: 0.2813, lr: 0.004489, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2716 samples/sec | ETA 03:32:19 2021-05-09 20:29:12 [INFO] [TRAIN] epoch: 65, iter: 23810/40000, loss: 0.3614, lr: 0.004487, batch_cost: 1.1094, reader_cost: 0.30908, ips: 0.9014 samples/sec | ETA 04:59:20 2021-05-09 20:29:20 [INFO] [TRAIN] epoch: 65, iter: 23820/40000, loss: 0.3029, lr: 0.004484, batch_cost: 0.8001, reader_cost: 0.00035, ips: 1.2498 samples/sec | ETA 03:35:45 2021-05-09 20:29:28 [INFO] [TRAIN] epoch: 65, iter: 23830/40000, loss: 0.5324, lr: 0.004482, batch_cost: 0.7887, reader_cost: 0.00015, ips: 1.2679 samples/sec | ETA 03:32:33 2021-05-09 20:29:35 [INFO] [TRAIN] epoch: 65, iter: 23840/40000, loss: 0.2099, lr: 0.004479, batch_cost: 0.7887, reader_cost: 0.00017, ips: 1.2679 samples/sec | ETA 03:32:25 2021-05-09 20:29:43 [INFO] [TRAIN] epoch: 65, iter: 23850/40000, loss: 0.4675, lr: 0.004477, batch_cost: 0.7883, reader_cost: 0.00016, ips: 1.2686 samples/sec | ETA 03:32:10 2021-05-09 20:29:51 [INFO] [TRAIN] epoch: 65, iter: 23860/40000, loss: 0.3421, lr: 0.004474, batch_cost: 0.7873, reader_cost: 0.00016, ips: 1.2701 samples/sec | ETA 03:31:47 2021-05-09 20:29:59 [INFO] [TRAIN] epoch: 65, iter: 23870/40000, loss: 0.2245, lr: 0.004472, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2728 samples/sec | ETA 03:31:13 2021-05-09 20:30:07 [INFO] [TRAIN] epoch: 65, iter: 23880/40000, loss: 0.1747, lr: 0.004470, batch_cost: 0.7881, reader_cost: 0.00016, ips: 1.2688 samples/sec | ETA 03:31:44 2021-05-09 20:30:15 [INFO] [TRAIN] epoch: 65, iter: 23890/40000, loss: 0.3039, lr: 0.004467, batch_cost: 0.7860, reader_cost: 0.00017, ips: 1.2723 samples/sec | ETA 03:31:01 2021-05-09 20:30:23 [INFO] [TRAIN] epoch: 65, iter: 23900/40000, loss: 0.4560, lr: 0.004465, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2723 samples/sec | ETA 03:30:53 2021-05-09 20:30:31 [INFO] [TRAIN] epoch: 65, iter: 23910/40000, loss: 0.2484, lr: 0.004462, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2723 samples/sec | ETA 03:30:46 2021-05-09 20:30:38 [INFO] [TRAIN] epoch: 65, iter: 23920/40000, loss: 0.2367, lr: 0.004460, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 03:30:31 2021-05-09 20:30:46 [INFO] [TRAIN] epoch: 65, iter: 23930/40000, loss: 0.0908, lr: 0.004457, batch_cost: 0.7875, reader_cost: 0.00017, ips: 1.2698 samples/sec | ETA 03:30:55 2021-05-09 20:30:54 [INFO] [TRAIN] epoch: 65, iter: 23940/40000, loss: 0.2967, lr: 0.004455, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2706 samples/sec | ETA 03:30:39 2021-05-09 20:31:02 [INFO] [TRAIN] epoch: 65, iter: 23950/40000, loss: 0.3344, lr: 0.004452, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2704 samples/sec | ETA 03:30:34 2021-05-09 20:31:10 [INFO] [TRAIN] epoch: 65, iter: 23960/40000, loss: 0.3292, lr: 0.004450, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2714 samples/sec | ETA 03:30:16 2021-05-09 20:31:18 [INFO] [TRAIN] epoch: 65, iter: 23970/40000, loss: 0.2730, lr: 0.004448, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2710 samples/sec | ETA 03:30:12 2021-05-09 20:31:26 [INFO] [TRAIN] epoch: 65, iter: 23980/40000, loss: 0.2182, lr: 0.004445, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2716 samples/sec | ETA 03:29:58 2021-05-09 20:31:33 [INFO] [TRAIN] epoch: 65, iter: 23990/40000, loss: 0.2701, lr: 0.004443, batch_cost: 0.7855, reader_cost: 0.00018, ips: 1.2731 samples/sec | ETA 03:29:35 2021-05-09 20:31:41 [INFO] [TRAIN] epoch: 65, iter: 24000/40000, loss: 0.2232, lr: 0.004440, batch_cost: 0.7880, reader_cost: 0.00016, ips: 1.2691 samples/sec | ETA 03:30:07 2021-05-09 20:31:41 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-09 20:35:13 [INFO] [EVAL] #Images: 500 mIoU: 0.7600 Acc: 0.9560 Kappa: 0.9429 2021-05-09 20:35:13 [INFO] [EVAL] Class IoU: [0.9804 0.8428 0.9186 0.6296 0.6034 0.4975 0.6163 0.7223 0.9139 0.667 0.9402 0.7722 0.574 0.9386 0.8412 0.8662 0.7802 0.6073 0.7273] 2021-05-09 20:35:13 [INFO] [EVAL] Class Acc: [0.9917 0.9179 0.9542 0.8183 0.8163 0.7321 0.8124 0.8688 0.9444 0.8212 0.9638 0.8473 0.7909 0.9611 0.9141 0.9618 0.8616 0.7811 0.8317] 2021-05-09 20:35:41 [INFO] [EVAL] The model with the best validation mIoU (0.7601) was saved at iter 21000. 2021-05-09 20:35:49 [INFO] [TRAIN] epoch: 65, iter: 24010/40000, loss: 0.2147, lr: 0.004438, batch_cost: 0.7843, reader_cost: 0.00050, ips: 1.2751 samples/sec | ETA 03:29:00 2021-05-09 20:35:58 [INFO] [TRAIN] epoch: 65, iter: 24020/40000, loss: 0.2965, lr: 0.004435, batch_cost: 0.7833, reader_cost: 0.00017, ips: 1.2766 samples/sec | ETA 03:28:37 2021-05-09 20:36:05 [INFO] [TRAIN] epoch: 65, iter: 24030/40000, loss: 0.1918, lr: 0.004433, batch_cost: 0.7849, reader_cost: 0.00018, ips: 1.2741 samples/sec | ETA 03:28:54 2021-05-09 20:36:13 [INFO] [TRAIN] epoch: 65, iter: 24040/40000, loss: 0.2918, lr: 0.004430, batch_cost: 0.7873, reader_cost: 0.00016, ips: 1.2702 samples/sec | ETA 03:29:24 2021-05-09 20:36:21 [INFO] [TRAIN] epoch: 65, iter: 24050/40000, loss: 0.3092, lr: 0.004428, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2713 samples/sec | ETA 03:29:06 2021-05-09 20:36:29 [INFO] [TRAIN] epoch: 65, iter: 24060/40000, loss: 0.2165, lr: 0.004426, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2729 samples/sec | ETA 03:28:42 2021-05-09 20:36:37 [INFO] [TRAIN] epoch: 65, iter: 24070/40000, loss: 0.1567, lr: 0.004423, batch_cost: 0.7860, reader_cost: 0.00017, ips: 1.2723 samples/sec | ETA 03:28:40 2021-05-09 20:36:45 [INFO] [TRAIN] epoch: 65, iter: 24080/40000, loss: 0.2876, lr: 0.004421, batch_cost: 0.7853, reader_cost: 0.00017, ips: 1.2733 samples/sec | ETA 03:28:22 2021-05-09 20:36:53 [INFO] [TRAIN] epoch: 65, iter: 24090/40000, loss: 0.3460, lr: 0.004418, batch_cost: 0.7870, reader_cost: 0.00017, ips: 1.2707 samples/sec | ETA 03:28:40 2021-05-09 20:37:00 [INFO] [TRAIN] epoch: 65, iter: 24100/40000, loss: 0.4030, lr: 0.004416, batch_cost: 0.7880, reader_cost: 0.00016, ips: 1.2690 samples/sec | ETA 03:28:49 2021-05-09 20:37:08 [INFO] [TRAIN] epoch: 65, iter: 24110/40000, loss: 0.5420, lr: 0.004413, batch_cost: 0.7885, reader_cost: 0.00015, ips: 1.2683 samples/sec | ETA 03:28:48 2021-05-09 20:37:16 [INFO] [TRAIN] epoch: 65, iter: 24120/40000, loss: 0.3878, lr: 0.004411, batch_cost: 0.7872, reader_cost: 0.00015, ips: 1.2704 samples/sec | ETA 03:28:20 2021-05-09 20:37:24 [INFO] [TRAIN] epoch: 65, iter: 24130/40000, loss: 0.2486, lr: 0.004408, batch_cost: 0.7879, reader_cost: 0.00015, ips: 1.2693 samples/sec | ETA 03:28:23 2021-05-09 20:37:32 [INFO] [TRAIN] epoch: 65, iter: 24140/40000, loss: 0.0909, lr: 0.004406, batch_cost: 0.7869, reader_cost: 0.00015, ips: 1.2708 samples/sec | ETA 03:28:00 2021-05-09 20:37:40 [INFO] [TRAIN] epoch: 65, iter: 24150/40000, loss: 0.1927, lr: 0.004404, batch_cost: 0.7879, reader_cost: 0.00016, ips: 1.2692 samples/sec | ETA 03:28:07 2021-05-09 20:37:48 [INFO] [TRAIN] epoch: 65, iter: 24160/40000, loss: 0.3148, lr: 0.004401, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2721 samples/sec | ETA 03:27:32 2021-05-09 20:37:56 [INFO] [TRAIN] epoch: 65, iter: 24170/40000, loss: 0.3223, lr: 0.004399, batch_cost: 0.7870, reader_cost: 0.00014, ips: 1.2706 samples/sec | ETA 03:27:38 2021-05-09 20:38:03 [INFO] [TRAIN] epoch: 65, iter: 24180/40000, loss: 0.2502, lr: 0.004396, batch_cost: 0.7846, reader_cost: 0.00009, ips: 1.2745 samples/sec | ETA 03:26:52 2021-05-09 20:38:15 [INFO] [TRAIN] epoch: 66, iter: 24190/40000, loss: 0.1434, lr: 0.004394, batch_cost: 1.1146, reader_cost: 0.27642, ips: 0.8971 samples/sec | ETA 04:53:42 2021-05-09 20:38:22 [INFO] [TRAIN] epoch: 66, iter: 24200/40000, loss: 0.5359, lr: 0.004391, batch_cost: 0.7879, reader_cost: 0.00032, ips: 1.2692 samples/sec | ETA 03:27:28 2021-05-09 20:38:30 [INFO] [TRAIN] epoch: 66, iter: 24210/40000, loss: 0.2328, lr: 0.004389, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2730 samples/sec | ETA 03:26:43 2021-05-09 20:38:38 [INFO] [TRAIN] epoch: 66, iter: 24220/40000, loss: 0.3731, lr: 0.004386, batch_cost: 0.7876, reader_cost: 0.00015, ips: 1.2697 samples/sec | ETA 03:27:07 2021-05-09 20:38:46 [INFO] [TRAIN] epoch: 66, iter: 24230/40000, loss: 0.4917, lr: 0.004384, batch_cost: 0.7879, reader_cost: 0.00017, ips: 1.2691 samples/sec | ETA 03:27:05 2021-05-09 20:38:54 [INFO] [TRAIN] epoch: 66, iter: 24240/40000, loss: 0.4027, lr: 0.004382, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 03:26:25 2021-05-09 20:39:02 [INFO] [TRAIN] epoch: 66, iter: 24250/40000, loss: 0.1969, lr: 0.004379, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2707 samples/sec | ETA 03:26:34 2021-05-09 20:39:10 [INFO] [TRAIN] epoch: 66, iter: 24260/40000, loss: 0.2369, lr: 0.004377, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 03:26:03 2021-05-09 20:39:18 [INFO] [TRAIN] epoch: 66, iter: 24270/40000, loss: 0.3549, lr: 0.004374, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 03:26:10 2021-05-09 20:39:25 [INFO] [TRAIN] epoch: 66, iter: 24280/40000, loss: 0.2897, lr: 0.004372, batch_cost: 0.7869, reader_cost: 0.00015, ips: 1.2708 samples/sec | ETA 03:26:10 2021-05-09 20:39:33 [INFO] [TRAIN] epoch: 66, iter: 24290/40000, loss: 0.3650, lr: 0.004369, batch_cost: 0.7854, reader_cost: 0.00017, ips: 1.2732 samples/sec | ETA 03:25:38 2021-05-09 20:39:41 [INFO] [TRAIN] epoch: 66, iter: 24300/40000, loss: 0.1504, lr: 0.004367, batch_cost: 0.7858, reader_cost: 0.00017, ips: 1.2727 samples/sec | ETA 03:25:36 2021-05-09 20:39:49 [INFO] [TRAIN] epoch: 66, iter: 24310/40000, loss: 0.2755, lr: 0.004364, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 03:25:37 2021-05-09 20:39:57 [INFO] [TRAIN] epoch: 66, iter: 24320/40000, loss: 0.3782, lr: 0.004362, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2708 samples/sec | ETA 03:25:38 2021-05-09 20:40:05 [INFO] [TRAIN] epoch: 66, iter: 24330/40000, loss: 0.2957, lr: 0.004360, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2717 samples/sec | ETA 03:25:22 2021-05-09 20:40:13 [INFO] [TRAIN] epoch: 66, iter: 24340/40000, loss: 0.3700, lr: 0.004357, batch_cost: 0.7835, reader_cost: 0.00015, ips: 1.2764 samples/sec | ETA 03:24:29 2021-05-09 20:40:20 [INFO] [TRAIN] epoch: 66, iter: 24350/40000, loss: 0.1570, lr: 0.004355, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2727 samples/sec | ETA 03:24:56 2021-05-09 20:40:28 [INFO] [TRAIN] epoch: 66, iter: 24360/40000, loss: 0.1979, lr: 0.004352, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2729 samples/sec | ETA 03:24:47 2021-05-09 20:40:36 [INFO] [TRAIN] epoch: 66, iter: 24370/40000, loss: 0.1712, lr: 0.004350, batch_cost: 0.7858, reader_cost: 0.00017, ips: 1.2725 samples/sec | ETA 03:24:42 2021-05-09 20:40:44 [INFO] [TRAIN] epoch: 66, iter: 24380/40000, loss: 0.2473, lr: 0.004347, batch_cost: 0.7869, reader_cost: 0.00017, ips: 1.2709 samples/sec | ETA 03:24:50 2021-05-09 20:40:52 [INFO] [TRAIN] epoch: 66, iter: 24390/40000, loss: 0.3268, lr: 0.004345, batch_cost: 0.7868, reader_cost: 0.00014, ips: 1.2709 samples/sec | ETA 03:24:42 2021-05-09 20:41:00 [INFO] [TRAIN] epoch: 66, iter: 24400/40000, loss: 0.2463, lr: 0.004342, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2722 samples/sec | ETA 03:24:22 2021-05-09 20:41:08 [INFO] [TRAIN] epoch: 66, iter: 24410/40000, loss: 0.2503, lr: 0.004340, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2719 samples/sec | ETA 03:24:17 2021-05-09 20:41:15 [INFO] [TRAIN] epoch: 66, iter: 24420/40000, loss: 0.2544, lr: 0.004338, batch_cost: 0.7833, reader_cost: 0.00016, ips: 1.2766 samples/sec | ETA 03:23:24 2021-05-09 20:41:23 [INFO] [TRAIN] epoch: 66, iter: 24430/40000, loss: 0.1475, lr: 0.004335, batch_cost: 0.7845, reader_cost: 0.00017, ips: 1.2748 samples/sec | ETA 03:23:33 2021-05-09 20:41:31 [INFO] [TRAIN] epoch: 66, iter: 24440/40000, loss: 0.1987, lr: 0.004333, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2730 samples/sec | ETA 03:23:42 2021-05-09 20:41:39 [INFO] [TRAIN] epoch: 66, iter: 24450/40000, loss: 0.2574, lr: 0.004330, batch_cost: 0.7863, reader_cost: 0.00018, ips: 1.2718 samples/sec | ETA 03:23:46 2021-05-09 20:41:47 [INFO] [TRAIN] epoch: 66, iter: 24460/40000, loss: 0.4030, lr: 0.004328, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2716 samples/sec | ETA 03:23:41 2021-05-09 20:41:55 [INFO] [TRAIN] epoch: 66, iter: 24470/40000, loss: 0.3425, lr: 0.004325, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2715 samples/sec | ETA 03:23:34 2021-05-09 20:42:03 [INFO] [TRAIN] epoch: 66, iter: 24480/40000, loss: 0.4044, lr: 0.004323, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2716 samples/sec | ETA 03:23:25 2021-05-09 20:42:10 [INFO] [TRAIN] epoch: 66, iter: 24490/40000, loss: 0.4930, lr: 0.004320, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2726 samples/sec | ETA 03:23:07 2021-05-09 20:42:18 [INFO] [TRAIN] epoch: 66, iter: 24500/40000, loss: 0.3087, lr: 0.004318, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2722 samples/sec | ETA 03:23:03 2021-05-09 20:42:26 [INFO] [TRAIN] epoch: 66, iter: 24510/40000, loss: 0.1187, lr: 0.004316, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2723 samples/sec | ETA 03:22:55 2021-05-09 20:42:34 [INFO] [TRAIN] epoch: 66, iter: 24520/40000, loss: 0.1911, lr: 0.004313, batch_cost: 0.7848, reader_cost: 0.00017, ips: 1.2742 samples/sec | ETA 03:22:28 2021-05-09 20:42:42 [INFO] [TRAIN] epoch: 66, iter: 24530/40000, loss: 0.2256, lr: 0.004311, batch_cost: 0.7852, reader_cost: 0.00018, ips: 1.2735 samples/sec | ETA 03:22:27 2021-05-09 20:42:50 [INFO] [TRAIN] epoch: 66, iter: 24540/40000, loss: 0.2728, lr: 0.004308, batch_cost: 0.7844, reader_cost: 0.00017, ips: 1.2749 samples/sec | ETA 03:22:06 2021-05-09 20:42:58 [INFO] [TRAIN] epoch: 66, iter: 24550/40000, loss: 0.2589, lr: 0.004306, batch_cost: 0.7836, reader_cost: 0.00025, ips: 1.2762 samples/sec | ETA 03:21:45 2021-05-09 20:43:08 [INFO] [TRAIN] epoch: 67, iter: 24560/40000, loss: 0.2788, lr: 0.004303, batch_cost: 1.0925, reader_cost: 0.28834, ips: 0.9153 samples/sec | ETA 04:41:08 2021-05-09 20:43:16 [INFO] [TRAIN] epoch: 67, iter: 24570/40000, loss: 0.5048, lr: 0.004301, batch_cost: 0.7905, reader_cost: 0.00036, ips: 1.2651 samples/sec | ETA 03:23:16 2021-05-09 20:43:24 [INFO] [TRAIN] epoch: 67, iter: 24580/40000, loss: 0.3136, lr: 0.004298, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2704 samples/sec | ETA 03:22:17 2021-05-09 20:43:32 [INFO] [TRAIN] epoch: 67, iter: 24590/40000, loss: 0.4426, lr: 0.004296, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2732 samples/sec | ETA 03:21:42 2021-05-09 20:43:40 [INFO] [TRAIN] epoch: 67, iter: 24600/40000, loss: 0.4960, lr: 0.004293, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2714 samples/sec | ETA 03:21:52 2021-05-09 20:43:48 [INFO] [TRAIN] epoch: 67, iter: 24610/40000, loss: 0.2581, lr: 0.004291, batch_cost: 0.7870, reader_cost: 0.00015, ips: 1.2707 samples/sec | ETA 03:21:51 2021-05-09 20:43:56 [INFO] [TRAIN] epoch: 67, iter: 24620/40000, loss: 0.1893, lr: 0.004289, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2710 samples/sec | ETA 03:21:40 2021-05-09 20:44:04 [INFO] [TRAIN] epoch: 67, iter: 24630/40000, loss: 0.2277, lr: 0.004286, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2721 samples/sec | ETA 03:21:22 2021-05-09 20:44:11 [INFO] [TRAIN] epoch: 67, iter: 24640/40000, loss: 0.4230, lr: 0.004284, batch_cost: 0.7885, reader_cost: 0.00017, ips: 1.2683 samples/sec | ETA 03:21:50 2021-05-09 20:44:19 [INFO] [TRAIN] epoch: 67, iter: 24650/40000, loss: 0.1797, lr: 0.004281, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2704 samples/sec | ETA 03:21:22 2021-05-09 20:44:27 [INFO] [TRAIN] epoch: 67, iter: 24660/40000, loss: 0.3446, lr: 0.004279, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2705 samples/sec | ETA 03:21:13 2021-05-09 20:44:35 [INFO] [TRAIN] epoch: 67, iter: 24670/40000, loss: 0.2931, lr: 0.004276, batch_cost: 0.7870, reader_cost: 0.00015, ips: 1.2707 samples/sec | ETA 03:21:04 2021-05-09 20:44:43 [INFO] [TRAIN] epoch: 67, iter: 24680/40000, loss: 0.2065, lr: 0.004274, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2728 samples/sec | ETA 03:20:36 2021-05-09 20:44:51 [INFO] [TRAIN] epoch: 67, iter: 24690/40000, loss: 0.3651, lr: 0.004271, batch_cost: 0.7878, reader_cost: 0.00015, ips: 1.2693 samples/sec | ETA 03:21:01 2021-05-09 20:44:59 [INFO] [TRAIN] epoch: 67, iter: 24700/40000, loss: 0.2313, lr: 0.004269, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2723 samples/sec | ETA 03:20:25 2021-05-09 20:45:07 [INFO] [TRAIN] epoch: 67, iter: 24710/40000, loss: 0.2686, lr: 0.004267, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 03:20:19 2021-05-09 20:45:14 [INFO] [TRAIN] epoch: 67, iter: 24720/40000, loss: 0.2481, lr: 0.004264, batch_cost: 0.7861, reader_cost: 0.00014, ips: 1.2721 samples/sec | ETA 03:20:11 2021-05-09 20:45:22 [INFO] [TRAIN] epoch: 67, iter: 24730/40000, loss: 0.2106, lr: 0.004262, batch_cost: 0.7880, reader_cost: 0.00015, ips: 1.2691 samples/sec | ETA 03:20:32 2021-05-09 20:45:30 [INFO] [TRAIN] epoch: 67, iter: 24740/40000, loss: 0.1845, lr: 0.004259, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2724 samples/sec | ETA 03:19:53 2021-05-09 20:45:38 [INFO] [TRAIN] epoch: 67, iter: 24750/40000, loss: 0.1268, lr: 0.004257, batch_cost: 0.7868, reader_cost: 0.00017, ips: 1.2710 samples/sec | ETA 03:19:58 2021-05-09 20:45:46 [INFO] [TRAIN] epoch: 67, iter: 24760/40000, loss: 0.2707, lr: 0.004254, batch_cost: 0.7850, reader_cost: 0.00016, ips: 1.2739 samples/sec | ETA 03:19:23 2021-05-09 20:45:54 [INFO] [TRAIN] epoch: 67, iter: 24770/40000, loss: 0.2081, lr: 0.004252, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2716 samples/sec | ETA 03:19:37 2021-05-09 20:46:02 [INFO] [TRAIN] epoch: 67, iter: 24780/40000, loss: 0.1417, lr: 0.004249, batch_cost: 0.7858, reader_cost: 0.00015, ips: 1.2726 samples/sec | ETA 03:19:19 2021-05-09 20:46:09 [INFO] [TRAIN] epoch: 67, iter: 24790/40000, loss: 0.2811, lr: 0.004247, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2722 samples/sec | ETA 03:19:16 2021-05-09 20:46:17 [INFO] [TRAIN] epoch: 67, iter: 24800/40000, loss: 0.1832, lr: 0.004244, batch_cost: 0.7846, reader_cost: 0.00015, ips: 1.2746 samples/sec | ETA 03:18:45 2021-05-09 20:46:25 [INFO] [TRAIN] epoch: 67, iter: 24810/40000, loss: 0.3270, lr: 0.004242, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2711 samples/sec | ETA 03:19:10 2021-05-09 20:46:33 [INFO] [TRAIN] epoch: 67, iter: 24820/40000, loss: 0.3103, lr: 0.004240, batch_cost: 0.7873, reader_cost: 0.00017, ips: 1.2701 samples/sec | ETA 03:19:11 2021-05-09 20:46:41 [INFO] [TRAIN] epoch: 67, iter: 24830/40000, loss: 0.3872, lr: 0.004237, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2723 samples/sec | ETA 03:18:43 2021-05-09 20:46:49 [INFO] [TRAIN] epoch: 67, iter: 24840/40000, loss: 0.3967, lr: 0.004235, batch_cost: 0.7848, reader_cost: 0.00016, ips: 1.2742 samples/sec | ETA 03:18:17 2021-05-09 20:46:57 [INFO] [TRAIN] epoch: 67, iter: 24850/40000, loss: 0.3006, lr: 0.004232, batch_cost: 0.7841, reader_cost: 0.00015, ips: 1.2753 samples/sec | ETA 03:17:59 2021-05-09 20:47:04 [INFO] [TRAIN] epoch: 67, iter: 24860/40000, loss: 0.5223, lr: 0.004230, batch_cost: 0.7858, reader_cost: 0.00015, ips: 1.2726 samples/sec | ETA 03:18:16 2021-05-09 20:47:12 [INFO] [TRAIN] epoch: 67, iter: 24870/40000, loss: 0.2412, lr: 0.004227, batch_cost: 0.7842, reader_cost: 0.00015, ips: 1.2751 samples/sec | ETA 03:17:45 2021-05-09 20:47:20 [INFO] [TRAIN] epoch: 67, iter: 24880/40000, loss: 0.2071, lr: 0.004225, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 03:18:00 2021-05-09 20:47:28 [INFO] [TRAIN] epoch: 67, iter: 24890/40000, loss: 0.2329, lr: 0.004222, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2704 samples/sec | ETA 03:18:14 2021-05-09 20:47:36 [INFO] [TRAIN] epoch: 67, iter: 24900/40000, loss: 0.2840, lr: 0.004220, batch_cost: 0.7869, reader_cost: 0.00017, ips: 1.2708 samples/sec | ETA 03:18:02 2021-05-09 20:47:44 [INFO] [TRAIN] epoch: 67, iter: 24910/40000, loss: 0.2613, lr: 0.004217, batch_cost: 0.7841, reader_cost: 0.00017, ips: 1.2753 samples/sec | ETA 03:17:12 2021-05-09 20:47:52 [INFO] [TRAIN] epoch: 67, iter: 24920/40000, loss: 0.2804, lr: 0.004215, batch_cost: 0.7854, reader_cost: 0.00011, ips: 1.2733 samples/sec | ETA 03:17:23 2021-05-09 20:48:02 [INFO] [TRAIN] epoch: 68, iter: 24930/40000, loss: 0.1865, lr: 0.004213, batch_cost: 1.0800, reader_cost: 0.24259, ips: 0.9259 samples/sec | ETA 04:31:15 2021-05-09 20:48:10 [INFO] [TRAIN] epoch: 68, iter: 24940/40000, loss: 0.3426, lr: 0.004210, batch_cost: 0.7937, reader_cost: 0.00035, ips: 1.2600 samples/sec | ETA 03:19:12 2021-05-09 20:48:18 [INFO] [TRAIN] epoch: 68, iter: 24950/40000, loss: 0.3999, lr: 0.004208, batch_cost: 0.7876, reader_cost: 0.00016, ips: 1.2696 samples/sec | ETA 03:17:33 2021-05-09 20:48:26 [INFO] [TRAIN] epoch: 68, iter: 24960/40000, loss: 0.3939, lr: 0.004205, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2716 samples/sec | ETA 03:17:07 2021-05-09 20:48:34 [INFO] [TRAIN] epoch: 68, iter: 24970/40000, loss: 0.4441, lr: 0.004203, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2705 samples/sec | ETA 03:17:10 2021-05-09 20:48:42 [INFO] [TRAIN] epoch: 68, iter: 24980/40000, loss: 0.3777, lr: 0.004200, batch_cost: 0.7886, reader_cost: 0.00016, ips: 1.2680 samples/sec | ETA 03:17:25 2021-05-09 20:48:50 [INFO] [TRAIN] epoch: 68, iter: 24990/40000, loss: 0.2730, lr: 0.004198, batch_cost: 0.7879, reader_cost: 0.00017, ips: 1.2692 samples/sec | ETA 03:17:06 2021-05-09 20:48:58 [INFO] [TRAIN] epoch: 68, iter: 25000/40000, loss: 0.1382, lr: 0.004195, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 03:16:22 2021-05-09 20:48:58 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-09 20:52:29 [INFO] [EVAL] #Images: 500 mIoU: 0.7635 Acc: 0.9565 Kappa: 0.9435 2021-05-09 20:52:29 [INFO] [EVAL] Class IoU: [0.9812 0.8464 0.9187 0.6311 0.617 0.4824 0.6214 0.7255 0.9145 0.6483 0.9397 0.7777 0.5844 0.9402 0.8198 0.8873 0.8084 0.6302 0.7331] 2021-05-09 20:52:29 [INFO] [EVAL] Class Acc: [0.9923 0.9047 0.9546 0.7845 0.8016 0.7661 0.7937 0.8704 0.9434 0.8558 0.9643 0.8674 0.7622 0.9651 0.9635 0.9502 0.912 0.7862 0.8315] 2021-05-09 20:53:19 [INFO] [EVAL] The model with the best validation mIoU (0.7635) was saved at iter 25000. 2021-05-09 20:53:26 [INFO] [TRAIN] epoch: 68, iter: 25010/40000, loss: 0.2889, lr: 0.004193, batch_cost: 0.7838, reader_cost: 0.00046, ips: 1.2758 samples/sec | ETA 03:15:49 2021-05-09 20:53:34 [INFO] [TRAIN] epoch: 68, iter: 25020/40000, loss: 0.2453, lr: 0.004190, batch_cost: 0.7842, reader_cost: 0.00031, ips: 1.2752 samples/sec | ETA 03:15:47 2021-05-09 20:53:42 [INFO] [TRAIN] epoch: 68, iter: 25030/40000, loss: 0.2682, lr: 0.004188, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2732 samples/sec | ETA 03:15:57 2021-05-09 20:53:50 [INFO] [TRAIN] epoch: 68, iter: 25040/40000, loss: 0.1997, lr: 0.004186, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2717 samples/sec | ETA 03:16:03 2021-05-09 20:53:58 [INFO] [TRAIN] epoch: 68, iter: 25050/40000, loss: 0.2053, lr: 0.004183, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2720 samples/sec | ETA 03:15:53 2021-05-09 20:54:06 [INFO] [TRAIN] epoch: 68, iter: 25060/40000, loss: 0.4031, lr: 0.004181, batch_cost: 0.7880, reader_cost: 0.00015, ips: 1.2691 samples/sec | ETA 03:16:12 2021-05-09 20:54:14 [INFO] [TRAIN] epoch: 68, iter: 25070/40000, loss: 0.2893, lr: 0.004178, batch_cost: 0.7875, reader_cost: 0.00015, ips: 1.2699 samples/sec | ETA 03:15:56 2021-05-09 20:54:21 [INFO] [TRAIN] epoch: 68, iter: 25080/40000, loss: 0.2993, lr: 0.004176, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2703 samples/sec | ETA 03:15:45 2021-05-09 20:54:29 [INFO] [TRAIN] epoch: 68, iter: 25090/40000, loss: 0.2715, lr: 0.004173, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2716 samples/sec | ETA 03:15:25 2021-05-09 20:54:37 [INFO] [TRAIN] epoch: 68, iter: 25100/40000, loss: 0.3532, lr: 0.004171, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2713 samples/sec | ETA 03:15:19 2021-05-09 20:54:45 [INFO] [TRAIN] epoch: 68, iter: 25110/40000, loss: 0.3293, lr: 0.004168, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2710 samples/sec | ETA 03:15:15 2021-05-09 20:54:53 [INFO] [TRAIN] epoch: 68, iter: 25120/40000, loss: 0.1552, lr: 0.004166, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2714 samples/sec | ETA 03:15:03 2021-05-09 20:55:01 [INFO] [TRAIN] epoch: 68, iter: 25130/40000, loss: 0.2419, lr: 0.004163, batch_cost: 0.7871, reader_cost: 0.00015, ips: 1.2705 samples/sec | ETA 03:15:04 2021-05-09 20:55:09 [INFO] [TRAIN] epoch: 68, iter: 25140/40000, loss: 0.3187, lr: 0.004161, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2723 samples/sec | ETA 03:14:39 2021-05-09 20:55:16 [INFO] [TRAIN] epoch: 68, iter: 25150/40000, loss: 0.2160, lr: 0.004158, batch_cost: 0.7870, reader_cost: 0.00015, ips: 1.2707 samples/sec | ETA 03:14:46 2021-05-09 20:55:24 [INFO] [TRAIN] epoch: 68, iter: 25160/40000, loss: 0.2427, lr: 0.004156, batch_cost: 0.7873, reader_cost: 0.00017, ips: 1.2702 samples/sec | ETA 03:14:42 2021-05-09 20:55:32 [INFO] [TRAIN] epoch: 68, iter: 25170/40000, loss: 0.2355, lr: 0.004154, batch_cost: 0.7869, reader_cost: 0.00018, ips: 1.2709 samples/sec | ETA 03:14:29 2021-05-09 20:55:40 [INFO] [TRAIN] epoch: 68, iter: 25180/40000, loss: 0.2021, lr: 0.004151, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2709 samples/sec | ETA 03:14:20 2021-05-09 20:55:48 [INFO] [TRAIN] epoch: 68, iter: 25190/40000, loss: 0.2152, lr: 0.004149, batch_cost: 0.7876, reader_cost: 0.00016, ips: 1.2696 samples/sec | ETA 03:14:24 2021-05-09 20:55:56 [INFO] [TRAIN] epoch: 68, iter: 25200/40000, loss: 0.2456, lr: 0.004146, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2735 samples/sec | ETA 03:13:41 2021-05-09 20:56:04 [INFO] [TRAIN] epoch: 68, iter: 25210/40000, loss: 0.2596, lr: 0.004144, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2724 samples/sec | ETA 03:13:43 2021-05-09 20:56:12 [INFO] [TRAIN] epoch: 68, iter: 25220/40000, loss: 0.3604, lr: 0.004141, batch_cost: 0.7856, reader_cost: 0.00017, ips: 1.2729 samples/sec | ETA 03:13:30 2021-05-09 20:56:19 [INFO] [TRAIN] epoch: 68, iter: 25230/40000, loss: 0.4938, lr: 0.004139, batch_cost: 0.7856, reader_cost: 0.00017, ips: 1.2729 samples/sec | ETA 03:13:23 2021-05-09 20:56:27 [INFO] [TRAIN] epoch: 68, iter: 25240/40000, loss: 0.4558, lr: 0.004136, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2709 samples/sec | ETA 03:13:33 2021-05-09 20:56:35 [INFO] [TRAIN] epoch: 68, iter: 25250/40000, loss: 0.4534, lr: 0.004134, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2726 samples/sec | ETA 03:13:10 2021-05-09 20:56:43 [INFO] [TRAIN] epoch: 68, iter: 25260/40000, loss: 0.1883, lr: 0.004131, batch_cost: 0.7870, reader_cost: 0.00015, ips: 1.2706 samples/sec | ETA 03:13:20 2021-05-09 20:56:51 [INFO] [TRAIN] epoch: 68, iter: 25270/40000, loss: 0.2249, lr: 0.004129, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2721 samples/sec | ETA 03:12:59 2021-05-09 20:56:59 [INFO] [TRAIN] epoch: 68, iter: 25280/40000, loss: 0.3379, lr: 0.004126, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2731 samples/sec | ETA 03:12:42 2021-05-09 20:57:07 [INFO] [TRAIN] epoch: 68, iter: 25290/40000, loss: 0.3297, lr: 0.004124, batch_cost: 0.7850, reader_cost: 0.00011, ips: 1.2738 samples/sec | ETA 03:12:28 2021-05-09 20:57:18 [INFO] [TRAIN] epoch: 69, iter: 25300/40000, loss: 0.2554, lr: 0.004122, batch_cost: 1.1013, reader_cost: 0.23724, ips: 0.9080 samples/sec | ETA 04:29:48 2021-05-09 20:57:26 [INFO] [TRAIN] epoch: 69, iter: 25310/40000, loss: 0.3583, lr: 0.004119, batch_cost: 0.7967, reader_cost: 0.00031, ips: 1.2552 samples/sec | ETA 03:15:03 2021-05-09 20:57:33 [INFO] [TRAIN] epoch: 69, iter: 25320/40000, loss: 0.5643, lr: 0.004117, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2733 samples/sec | ETA 03:12:09 2021-05-09 20:57:41 [INFO] [TRAIN] epoch: 69, iter: 25330/40000, loss: 0.2199, lr: 0.004114, batch_cost: 0.7873, reader_cost: 0.00018, ips: 1.2701 samples/sec | ETA 03:12:30 2021-05-09 20:57:49 [INFO] [TRAIN] epoch: 69, iter: 25340/40000, loss: 0.3721, lr: 0.004112, batch_cost: 0.7863, reader_cost: 0.00017, ips: 1.2718 samples/sec | ETA 03:12:06 2021-05-09 20:57:57 [INFO] [TRAIN] epoch: 69, iter: 25350/40000, loss: 0.3738, lr: 0.004109, batch_cost: 0.7856, reader_cost: 0.00017, ips: 1.2729 samples/sec | ETA 03:11:49 2021-05-09 20:58:05 [INFO] [TRAIN] epoch: 69, iter: 25360/40000, loss: 0.3081, lr: 0.004107, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2710 samples/sec | ETA 03:11:58 2021-05-09 20:58:13 [INFO] [TRAIN] epoch: 69, iter: 25370/40000, loss: 0.1035, lr: 0.004104, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 03:11:31 2021-05-09 20:58:21 [INFO] [TRAIN] epoch: 69, iter: 25380/40000, loss: 0.2871, lr: 0.004102, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2706 samples/sec | ETA 03:11:46 2021-05-09 20:58:28 [INFO] [TRAIN] epoch: 69, iter: 25390/40000, loss: 0.2581, lr: 0.004099, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2703 samples/sec | ETA 03:11:41 2021-05-09 20:58:36 [INFO] [TRAIN] epoch: 69, iter: 25400/40000, loss: 0.2830, lr: 0.004097, batch_cost: 0.7856, reader_cost: 0.00018, ips: 1.2729 samples/sec | ETA 03:11:10 2021-05-09 20:58:44 [INFO] [TRAIN] epoch: 69, iter: 25410/40000, loss: 0.2691, lr: 0.004094, batch_cost: 0.7859, reader_cost: 0.00018, ips: 1.2725 samples/sec | ETA 03:11:05 2021-05-09 20:58:52 [INFO] [TRAIN] epoch: 69, iter: 25420/40000, loss: 0.1351, lr: 0.004092, batch_cost: 0.7879, reader_cost: 0.00017, ips: 1.2692 samples/sec | ETA 03:11:27 2021-05-09 20:59:00 [INFO] [TRAIN] epoch: 69, iter: 25430/40000, loss: 0.2939, lr: 0.004090, batch_cost: 0.7873, reader_cost: 0.00017, ips: 1.2701 samples/sec | ETA 03:11:11 2021-05-09 20:59:08 [INFO] [TRAIN] epoch: 69, iter: 25440/40000, loss: 0.2505, lr: 0.004087, batch_cost: 0.7882, reader_cost: 0.00016, ips: 1.2688 samples/sec | ETA 03:11:15 2021-05-09 20:59:16 [INFO] [TRAIN] epoch: 69, iter: 25450/40000, loss: 0.3444, lr: 0.004085, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2714 samples/sec | ETA 03:10:44 2021-05-09 20:59:24 [INFO] [TRAIN] epoch: 69, iter: 25460/40000, loss: 0.2378, lr: 0.004082, batch_cost: 0.7859, reader_cost: 0.00017, ips: 1.2725 samples/sec | ETA 03:10:26 2021-05-09 20:59:31 [INFO] [TRAIN] epoch: 69, iter: 25470/40000, loss: 0.2495, lr: 0.004080, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2704 samples/sec | ETA 03:10:37 2021-05-09 20:59:39 [INFO] [TRAIN] epoch: 69, iter: 25480/40000, loss: 0.2772, lr: 0.004077, batch_cost: 0.7857, reader_cost: 0.00017, ips: 1.2727 samples/sec | ETA 03:10:08 2021-05-09 20:59:47 [INFO] [TRAIN] epoch: 69, iter: 25490/40000, loss: 0.1927, lr: 0.004075, batch_cost: 0.7876, reader_cost: 0.00017, ips: 1.2698 samples/sec | ETA 03:10:27 2021-05-09 20:59:55 [INFO] [TRAIN] epoch: 69, iter: 25500/40000, loss: 0.1633, lr: 0.004072, batch_cost: 0.7874, reader_cost: 0.00016, ips: 1.2699 samples/sec | ETA 03:10:17 2021-05-09 21:00:03 [INFO] [TRAIN] epoch: 69, iter: 25510/40000, loss: 0.2984, lr: 0.004070, batch_cost: 0.7851, reader_cost: 0.00016, ips: 1.2737 samples/sec | ETA 03:09:36 2021-05-09 21:00:11 [INFO] [TRAIN] epoch: 69, iter: 25520/40000, loss: 0.1607, lr: 0.004067, batch_cost: 0.7857, reader_cost: 0.00017, ips: 1.2727 samples/sec | ETA 03:09:37 2021-05-09 21:00:19 [INFO] [TRAIN] epoch: 69, iter: 25530/40000, loss: 0.1948, lr: 0.004065, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2718 samples/sec | ETA 03:09:37 2021-05-09 21:00:26 [INFO] [TRAIN] epoch: 69, iter: 25540/40000, loss: 0.2208, lr: 0.004062, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2711 samples/sec | ETA 03:09:36 2021-05-09 21:00:34 [INFO] [TRAIN] epoch: 69, iter: 25550/40000, loss: 0.2242, lr: 0.004060, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2720 samples/sec | ETA 03:09:19 2021-05-09 21:00:42 [INFO] [TRAIN] epoch: 69, iter: 25560/40000, loss: 0.1400, lr: 0.004057, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2720 samples/sec | ETA 03:09:11 2021-05-09 21:00:50 [INFO] [TRAIN] epoch: 69, iter: 25570/40000, loss: 0.3346, lr: 0.004055, batch_cost: 0.7879, reader_cost: 0.00016, ips: 1.2692 samples/sec | ETA 03:09:29 2021-05-09 21:00:58 [INFO] [TRAIN] epoch: 69, iter: 25580/40000, loss: 0.5179, lr: 0.004053, batch_cost: 0.7874, reader_cost: 0.00016, ips: 1.2700 samples/sec | ETA 03:09:14 2021-05-09 21:01:06 [INFO] [TRAIN] epoch: 69, iter: 25590/40000, loss: 0.4095, lr: 0.004050, batch_cost: 0.7873, reader_cost: 0.00016, ips: 1.2702 samples/sec | ETA 03:09:05 2021-05-09 21:01:14 [INFO] [TRAIN] epoch: 69, iter: 25600/40000, loss: 0.4314, lr: 0.004048, batch_cost: 0.7850, reader_cost: 0.00017, ips: 1.2739 samples/sec | ETA 03:08:23 2021-05-09 21:01:22 [INFO] [TRAIN] epoch: 69, iter: 25610/40000, loss: 0.5171, lr: 0.004045, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2714 samples/sec | ETA 03:08:38 2021-05-09 21:01:29 [INFO] [TRAIN] epoch: 69, iter: 25620/40000, loss: 0.1841, lr: 0.004043, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2709 samples/sec | ETA 03:08:34 2021-05-09 21:01:37 [INFO] [TRAIN] epoch: 69, iter: 25630/40000, loss: 0.1308, lr: 0.004040, batch_cost: 0.7864, reader_cost: 0.00017, ips: 1.2716 samples/sec | ETA 03:08:20 2021-05-09 21:01:45 [INFO] [TRAIN] epoch: 69, iter: 25640/40000, loss: 0.2782, lr: 0.004038, batch_cost: 0.7865, reader_cost: 0.00017, ips: 1.2715 samples/sec | ETA 03:08:13 2021-05-09 21:01:53 [INFO] [TRAIN] epoch: 69, iter: 25650/40000, loss: 0.2284, lr: 0.004035, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2726 samples/sec | ETA 03:07:56 2021-05-09 21:02:01 [INFO] [TRAIN] epoch: 69, iter: 25660/40000, loss: 0.3312, lr: 0.004033, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2725 samples/sec | ETA 03:07:49 2021-05-09 21:02:12 [INFO] [TRAIN] epoch: 70, iter: 25670/40000, loss: 0.3070, lr: 0.004030, batch_cost: 1.0775, reader_cost: 0.24257, ips: 0.9281 samples/sec | ETA 04:17:20 2021-05-09 21:02:20 [INFO] [TRAIN] epoch: 70, iter: 25680/40000, loss: 0.2921, lr: 0.004028, batch_cost: 0.7923, reader_cost: 0.00032, ips: 1.2622 samples/sec | ETA 03:09:05 2021-05-09 21:02:27 [INFO] [TRAIN] epoch: 70, iter: 25690/40000, loss: 0.4887, lr: 0.004025, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2731 samples/sec | ETA 03:07:20 2021-05-09 21:02:35 [INFO] [TRAIN] epoch: 70, iter: 25700/40000, loss: 0.2613, lr: 0.004023, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 03:07:26 2021-05-09 21:02:43 [INFO] [TRAIN] epoch: 70, iter: 25710/40000, loss: 0.3341, lr: 0.004020, batch_cost: 0.7870, reader_cost: 0.00015, ips: 1.2706 samples/sec | ETA 03:07:26 2021-05-09 21:02:51 [INFO] [TRAIN] epoch: 70, iter: 25720/40000, loss: 0.4437, lr: 0.004018, batch_cost: 0.7869, reader_cost: 0.00015, ips: 1.2708 samples/sec | ETA 03:07:16 2021-05-09 21:02:59 [INFO] [TRAIN] epoch: 70, iter: 25730/40000, loss: 0.2268, lr: 0.004016, batch_cost: 0.7860, reader_cost: 0.00014, ips: 1.2722 samples/sec | ETA 03:06:56 2021-05-09 21:03:07 [INFO] [TRAIN] epoch: 70, iter: 25740/40000, loss: 0.1159, lr: 0.004013, batch_cost: 0.7869, reader_cost: 0.00015, ips: 1.2708 samples/sec | ETA 03:07:01 2021-05-09 21:03:15 [INFO] [TRAIN] epoch: 70, iter: 25750/40000, loss: 0.2118, lr: 0.004011, batch_cost: 0.7889, reader_cost: 0.00016, ips: 1.2677 samples/sec | ETA 03:07:21 2021-05-09 21:03:22 [INFO] [TRAIN] epoch: 70, iter: 25760/40000, loss: 0.3655, lr: 0.004008, batch_cost: 0.7874, reader_cost: 0.00015, ips: 1.2701 samples/sec | ETA 03:06:51 2021-05-09 21:03:30 [INFO] [TRAIN] epoch: 70, iter: 25770/40000, loss: 0.2541, lr: 0.004006, batch_cost: 0.7865, reader_cost: 0.00017, ips: 1.2714 samples/sec | ETA 03:06:32 2021-05-09 21:03:38 [INFO] [TRAIN] epoch: 70, iter: 25780/40000, loss: 0.3003, lr: 0.004003, batch_cost: 0.7854, reader_cost: 0.00018, ips: 1.2732 samples/sec | ETA 03:06:08 2021-05-09 21:03:46 [INFO] [TRAIN] epoch: 70, iter: 25790/40000, loss: 0.1377, lr: 0.004001, batch_cost: 0.7872, reader_cost: 0.00018, ips: 1.2703 samples/sec | ETA 03:06:26 2021-05-09 21:03:54 [INFO] [TRAIN] epoch: 70, iter: 25800/40000, loss: 0.2541, lr: 0.003998, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2708 samples/sec | ETA 03:06:14 2021-05-09 21:04:02 [INFO] [TRAIN] epoch: 70, iter: 25810/40000, loss: 0.2171, lr: 0.003996, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2714 samples/sec | ETA 03:06:01 2021-05-09 21:04:10 [INFO] [TRAIN] epoch: 70, iter: 25820/40000, loss: 0.2729, lr: 0.003993, batch_cost: 0.7861, reader_cost: 0.00018, ips: 1.2721 samples/sec | ETA 03:05:47 2021-05-09 21:04:18 [INFO] [TRAIN] epoch: 70, iter: 25830/40000, loss: 0.3062, lr: 0.003991, batch_cost: 0.7859, reader_cost: 0.00019, ips: 1.2725 samples/sec | ETA 03:05:35 2021-05-09 21:04:25 [INFO] [TRAIN] epoch: 70, iter: 25840/40000, loss: 0.2253, lr: 0.003988, batch_cost: 0.7867, reader_cost: 0.00017, ips: 1.2712 samples/sec | ETA 03:05:39 2021-05-09 21:04:33 [INFO] [TRAIN] epoch: 70, iter: 25850/40000, loss: 0.2935, lr: 0.003986, batch_cost: 0.7869, reader_cost: 0.00017, ips: 1.2709 samples/sec | ETA 03:05:33 2021-05-09 21:04:41 [INFO] [TRAIN] epoch: 70, iter: 25860/40000, loss: 0.1971, lr: 0.003983, batch_cost: 0.7871, reader_cost: 0.00019, ips: 1.2705 samples/sec | ETA 03:05:29 2021-05-09 21:04:49 [INFO] [TRAIN] epoch: 70, iter: 25870/40000, loss: 0.1522, lr: 0.003981, batch_cost: 0.7864, reader_cost: 0.00018, ips: 1.2717 samples/sec | ETA 03:05:11 2021-05-09 21:04:57 [INFO] [TRAIN] epoch: 70, iter: 25880/40000, loss: 0.3515, lr: 0.003978, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 03:04:56 2021-05-09 21:05:05 [INFO] [TRAIN] epoch: 70, iter: 25890/40000, loss: 0.3002, lr: 0.003976, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 03:04:51 2021-05-09 21:05:13 [INFO] [TRAIN] epoch: 70, iter: 25900/40000, loss: 0.2224, lr: 0.003974, batch_cost: 0.7851, reader_cost: 0.00015, ips: 1.2737 samples/sec | ETA 03:04:30 2021-05-09 21:05:20 [INFO] [TRAIN] epoch: 70, iter: 25910/40000, loss: 0.1797, lr: 0.003971, batch_cost: 0.7854, reader_cost: 0.00015, ips: 1.2732 samples/sec | ETA 03:04:26 2021-05-09 21:05:28 [INFO] [TRAIN] epoch: 70, iter: 25920/40000, loss: 0.1931, lr: 0.003969, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2718 samples/sec | ETA 03:04:30 2021-05-09 21:05:36 [INFO] [TRAIN] epoch: 70, iter: 25930/40000, loss: 0.1959, lr: 0.003966, batch_cost: 0.7873, reader_cost: 0.00018, ips: 1.2702 samples/sec | ETA 03:04:37 2021-05-09 21:05:44 [INFO] [TRAIN] epoch: 70, iter: 25940/40000, loss: 0.3660, lr: 0.003964, batch_cost: 0.7854, reader_cost: 0.00017, ips: 1.2732 samples/sec | ETA 03:04:03 2021-05-09 21:05:52 [INFO] [TRAIN] epoch: 70, iter: 25950/40000, loss: 0.4337, lr: 0.003961, batch_cost: 0.7843, reader_cost: 0.00015, ips: 1.2751 samples/sec | ETA 03:03:38 2021-05-09 21:06:00 [INFO] [TRAIN] epoch: 70, iter: 25960/40000, loss: 0.3454, lr: 0.003959, batch_cost: 0.7847, reader_cost: 0.00015, ips: 1.2743 samples/sec | ETA 03:03:37 2021-05-09 21:06:08 [INFO] [TRAIN] epoch: 70, iter: 25970/40000, loss: 0.3845, lr: 0.003956, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2734 samples/sec | ETA 03:03:37 2021-05-09 21:06:15 [INFO] [TRAIN] epoch: 70, iter: 25980/40000, loss: 0.3847, lr: 0.003954, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2725 samples/sec | ETA 03:03:37 2021-05-09 21:06:23 [INFO] [TRAIN] epoch: 70, iter: 25990/40000, loss: 0.2107, lr: 0.003951, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2710 samples/sec | ETA 03:03:42 2021-05-09 21:06:31 [INFO] [TRAIN] epoch: 70, iter: 26000/40000, loss: 0.0839, lr: 0.003949, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 03:03:32 2021-05-09 21:06:31 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-09 21:10:02 [INFO] [EVAL] #Images: 500 mIoU: 0.7640 Acc: 0.9562 Kappa: 0.9431 2021-05-09 21:10:02 [INFO] [EVAL] Class IoU: [0.9798 0.8405 0.9186 0.6156 0.6191 0.5067 0.6119 0.7268 0.9155 0.6672 0.9408 0.7767 0.5851 0.9409 0.8301 0.8813 0.7921 0.6348 0.7323] 2021-05-09 21:10:02 [INFO] [EVAL] Class Acc: [0.9923 0.9035 0.9551 0.8167 0.7965 0.7127 0.8406 0.8775 0.9463 0.8518 0.965 0.8656 0.7585 0.9666 0.9193 0.9367 0.938 0.8329 0.8235] 2021-05-09 21:10:50 [INFO] [EVAL] The model with the best validation mIoU (0.7640) was saved at iter 26000. 2021-05-09 21:10:58 [INFO] [TRAIN] epoch: 70, iter: 26010/40000, loss: 0.2042, lr: 0.003946, batch_cost: 0.7818, reader_cost: 0.00048, ips: 1.2791 samples/sec | ETA 03:02:17 2021-05-09 21:11:06 [INFO] [TRAIN] epoch: 70, iter: 26020/40000, loss: 0.2291, lr: 0.003944, batch_cost: 0.7873, reader_cost: 0.00035, ips: 1.2702 samples/sec | ETA 03:03:26 2021-05-09 21:11:14 [INFO] [TRAIN] epoch: 70, iter: 26030/40000, loss: 0.3089, lr: 0.003941, batch_cost: 0.7848, reader_cost: 0.00015, ips: 1.2742 samples/sec | ETA 03:02:43 2021-05-09 21:11:21 [INFO] [TRAIN] epoch: 70, iter: 26040/40000, loss: 0.3739, lr: 0.003939, batch_cost: 0.7853, reader_cost: 0.00009, ips: 1.2734 samples/sec | ETA 03:02:42 2021-05-09 21:11:32 [INFO] [TRAIN] epoch: 71, iter: 26050/40000, loss: 0.2276, lr: 0.003936, batch_cost: 1.1045, reader_cost: 0.28125, ips: 0.9054 samples/sec | ETA 04:16:47 2021-05-09 21:11:40 [INFO] [TRAIN] epoch: 71, iter: 26060/40000, loss: 0.3122, lr: 0.003934, batch_cost: 0.7898, reader_cost: 0.00033, ips: 1.2662 samples/sec | ETA 03:03:29 2021-05-09 21:11:48 [INFO] [TRAIN] epoch: 71, iter: 26070/40000, loss: 0.2380, lr: 0.003931, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2709 samples/sec | ETA 03:02:40 2021-05-09 21:11:56 [INFO] [TRAIN] epoch: 71, iter: 26080/40000, loss: 0.2674, lr: 0.003929, batch_cost: 0.7882, reader_cost: 0.00016, ips: 1.2687 samples/sec | ETA 03:02:52 2021-05-09 21:12:04 [INFO] [TRAIN] epoch: 71, iter: 26090/40000, loss: 0.4857, lr: 0.003927, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2704 samples/sec | ETA 03:02:29 2021-05-09 21:12:12 [INFO] [TRAIN] epoch: 71, iter: 26100/40000, loss: 0.4175, lr: 0.003924, batch_cost: 0.7878, reader_cost: 0.00014, ips: 1.2694 samples/sec | ETA 03:02:30 2021-05-09 21:12:20 [INFO] [TRAIN] epoch: 71, iter: 26110/40000, loss: 0.1483, lr: 0.003922, batch_cost: 0.7876, reader_cost: 0.00014, ips: 1.2697 samples/sec | ETA 03:02:19 2021-05-09 21:12:28 [INFO] [TRAIN] epoch: 71, iter: 26120/40000, loss: 0.2641, lr: 0.003919, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2714 samples/sec | ETA 03:01:57 2021-05-09 21:12:35 [INFO] [TRAIN] epoch: 71, iter: 26130/40000, loss: 0.3564, lr: 0.003917, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2735 samples/sec | ETA 03:01:31 2021-05-09 21:12:43 [INFO] [TRAIN] epoch: 71, iter: 26140/40000, loss: 0.1992, lr: 0.003914, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2704 samples/sec | ETA 03:01:49 2021-05-09 21:12:51 [INFO] [TRAIN] epoch: 71, iter: 26150/40000, loss: 0.3161, lr: 0.003912, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2713 samples/sec | ETA 03:01:34 2021-05-09 21:12:59 [INFO] [TRAIN] epoch: 71, iter: 26160/40000, loss: 0.1046, lr: 0.003909, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2716 samples/sec | ETA 03:01:23 2021-05-09 21:13:07 [INFO] [TRAIN] epoch: 71, iter: 26170/40000, loss: 0.2789, lr: 0.003907, batch_cost: 0.7872, reader_cost: 0.00015, ips: 1.2703 samples/sec | ETA 03:01:27 2021-05-09 21:13:15 [INFO] [TRAIN] epoch: 71, iter: 26180/40000, loss: 0.3515, lr: 0.003904, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2714 samples/sec | ETA 03:01:10 2021-05-09 21:13:23 [INFO] [TRAIN] epoch: 71, iter: 26190/40000, loss: 0.3166, lr: 0.003902, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2704 samples/sec | ETA 03:01:10 2021-05-09 21:13:30 [INFO] [TRAIN] epoch: 71, iter: 26200/40000, loss: 0.2530, lr: 0.003899, batch_cost: 0.7862, reader_cost: 0.00017, ips: 1.2720 samples/sec | ETA 03:00:49 2021-05-09 21:13:38 [INFO] [TRAIN] epoch: 71, iter: 26210/40000, loss: 0.2482, lr: 0.003897, batch_cost: 0.7850, reader_cost: 0.00016, ips: 1.2738 samples/sec | ETA 03:00:25 2021-05-09 21:13:46 [INFO] [TRAIN] epoch: 71, iter: 26220/40000, loss: 0.2583, lr: 0.003894, batch_cost: 0.7861, reader_cost: 0.00017, ips: 1.2721 samples/sec | ETA 03:00:32 2021-05-09 21:13:54 [INFO] [TRAIN] epoch: 71, iter: 26230/40000, loss: 0.1313, lr: 0.003892, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 03:00:29 2021-05-09 21:14:02 [INFO] [TRAIN] epoch: 71, iter: 26240/40000, loss: 0.1716, lr: 0.003889, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2705 samples/sec | ETA 03:00:30 2021-05-09 21:14:10 [INFO] [TRAIN] epoch: 71, iter: 26250/40000, loss: 0.3308, lr: 0.003887, batch_cost: 0.7845, reader_cost: 0.00015, ips: 1.2748 samples/sec | ETA 02:59:46 2021-05-09 21:14:18 [INFO] [TRAIN] epoch: 71, iter: 26260/40000, loss: 0.2281, lr: 0.003884, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2729 samples/sec | ETA 02:59:54 2021-05-09 21:14:25 [INFO] [TRAIN] epoch: 71, iter: 26270/40000, loss: 0.1356, lr: 0.003882, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 02:59:58 2021-05-09 21:14:33 [INFO] [TRAIN] epoch: 71, iter: 26280/40000, loss: 0.2382, lr: 0.003879, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2707 samples/sec | ETA 02:59:57 2021-05-09 21:14:41 [INFO] [TRAIN] epoch: 71, iter: 26290/40000, loss: 0.1950, lr: 0.003877, batch_cost: 0.7855, reader_cost: 0.00017, ips: 1.2731 samples/sec | ETA 02:59:29 2021-05-09 21:14:49 [INFO] [TRAIN] epoch: 71, iter: 26300/40000, loss: 0.2848, lr: 0.003874, batch_cost: 0.7854, reader_cost: 0.00015, ips: 1.2733 samples/sec | ETA 02:59:19 2021-05-09 21:14:57 [INFO] [TRAIN] epoch: 71, iter: 26310/40000, loss: 0.2136, lr: 0.003872, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 02:59:12 2021-05-09 21:15:05 [INFO] [TRAIN] epoch: 71, iter: 26320/40000, loss: 0.4420, lr: 0.003870, batch_cost: 0.7848, reader_cost: 0.00016, ips: 1.2742 samples/sec | ETA 02:58:56 2021-05-09 21:15:13 [INFO] [TRAIN] epoch: 71, iter: 26330/40000, loss: 0.3224, lr: 0.003867, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2723 samples/sec | ETA 02:59:04 2021-05-09 21:15:21 [INFO] [TRAIN] epoch: 71, iter: 26340/40000, loss: 0.2868, lr: 0.003865, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2729 samples/sec | ETA 02:58:51 2021-05-09 21:15:28 [INFO] [TRAIN] epoch: 71, iter: 26350/40000, loss: 0.4872, lr: 0.003862, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2722 samples/sec | ETA 02:58:49 2021-05-09 21:15:36 [INFO] [TRAIN] epoch: 71, iter: 26360/40000, loss: 0.3577, lr: 0.003860, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2720 samples/sec | ETA 02:58:42 2021-05-09 21:15:44 [INFO] [TRAIN] epoch: 71, iter: 26370/40000, loss: 0.1000, lr: 0.003857, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2723 samples/sec | ETA 02:58:33 2021-05-09 21:15:52 [INFO] [TRAIN] epoch: 71, iter: 26380/40000, loss: 0.2939, lr: 0.003855, batch_cost: 0.7874, reader_cost: 0.00016, ips: 1.2700 samples/sec | ETA 02:58:44 2021-05-09 21:16:00 [INFO] [TRAIN] epoch: 71, iter: 26390/40000, loss: 0.2840, lr: 0.003852, batch_cost: 0.7860, reader_cost: 0.00017, ips: 1.2722 samples/sec | ETA 02:58:17 2021-05-09 21:16:08 [INFO] [TRAIN] epoch: 71, iter: 26400/40000, loss: 0.3331, lr: 0.003850, batch_cost: 0.7848, reader_cost: 0.00017, ips: 1.2742 samples/sec | ETA 02:57:53 2021-05-09 21:16:16 [INFO] [TRAIN] epoch: 71, iter: 26410/40000, loss: 0.3633, lr: 0.003847, batch_cost: 0.7865, reader_cost: 0.00025, ips: 1.2714 samples/sec | ETA 02:58:09 2021-05-09 21:16:27 [INFO] [TRAIN] epoch: 72, iter: 26420/40000, loss: 0.3015, lr: 0.003845, batch_cost: 1.1005, reader_cost: 0.26219, ips: 0.9087 samples/sec | ETA 04:09:04 2021-05-09 21:16:34 [INFO] [TRAIN] epoch: 72, iter: 26430/40000, loss: 0.3697, lr: 0.003842, batch_cost: 0.7946, reader_cost: 0.00034, ips: 1.2585 samples/sec | ETA 02:59:42 2021-05-09 21:16:42 [INFO] [TRAIN] epoch: 72, iter: 26440/40000, loss: 0.3261, lr: 0.003840, batch_cost: 0.7853, reader_cost: 0.00017, ips: 1.2734 samples/sec | ETA 02:57:28 2021-05-09 21:16:50 [INFO] [TRAIN] epoch: 72, iter: 26450/40000, loss: 0.5038, lr: 0.003837, batch_cost: 0.7853, reader_cost: 0.00018, ips: 1.2735 samples/sec | ETA 02:57:20 2021-05-09 21:16:58 [INFO] [TRAIN] epoch: 72, iter: 26460/40000, loss: 0.5567, lr: 0.003835, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2728 samples/sec | ETA 02:57:18 2021-05-09 21:17:06 [INFO] [TRAIN] epoch: 72, iter: 26470/40000, loss: 0.3533, lr: 0.003832, batch_cost: 0.7878, reader_cost: 0.00016, ips: 1.2694 samples/sec | ETA 02:57:38 2021-05-09 21:17:14 [INFO] [TRAIN] epoch: 72, iter: 26480/40000, loss: 0.2093, lr: 0.003830, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2725 samples/sec | ETA 02:57:04 2021-05-09 21:17:22 [INFO] [TRAIN] epoch: 72, iter: 26490/40000, loss: 0.2663, lr: 0.003827, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2720 samples/sec | ETA 02:57:01 2021-05-09 21:17:30 [INFO] [TRAIN] epoch: 72, iter: 26500/40000, loss: 0.3577, lr: 0.003825, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2710 samples/sec | ETA 02:57:01 2021-05-09 21:17:37 [INFO] [TRAIN] epoch: 72, iter: 26510/40000, loss: 0.2115, lr: 0.003822, batch_cost: 0.7877, reader_cost: 0.00015, ips: 1.2695 samples/sec | ETA 02:57:06 2021-05-09 21:17:45 [INFO] [TRAIN] epoch: 72, iter: 26520/40000, loss: 0.3195, lr: 0.003820, batch_cost: 0.7874, reader_cost: 0.00015, ips: 1.2700 samples/sec | ETA 02:56:54 2021-05-09 21:17:53 [INFO] [TRAIN] epoch: 72, iter: 26530/40000, loss: 0.1447, lr: 0.003817, batch_cost: 0.7877, reader_cost: 0.00015, ips: 1.2695 samples/sec | ETA 02:56:50 2021-05-09 21:18:01 [INFO] [TRAIN] epoch: 72, iter: 26540/40000, loss: 0.2805, lr: 0.003815, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2713 samples/sec | ETA 02:56:27 2021-05-09 21:18:09 [INFO] [TRAIN] epoch: 72, iter: 26550/40000, loss: 0.2926, lr: 0.003812, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2717 samples/sec | ETA 02:56:16 2021-05-09 21:18:17 [INFO] [TRAIN] epoch: 72, iter: 26560/40000, loss: 0.3897, lr: 0.003810, batch_cost: 0.7869, reader_cost: 0.00015, ips: 1.2708 samples/sec | ETA 02:56:15 2021-05-09 21:18:25 [INFO] [TRAIN] epoch: 72, iter: 26570/40000, loss: 0.2317, lr: 0.003807, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2706 samples/sec | ETA 02:56:09 2021-05-09 21:18:32 [INFO] [TRAIN] epoch: 72, iter: 26580/40000, loss: 0.2078, lr: 0.003805, batch_cost: 0.7864, reader_cost: 0.00014, ips: 1.2716 samples/sec | ETA 02:55:53 2021-05-09 21:18:40 [INFO] [TRAIN] epoch: 72, iter: 26590/40000, loss: 0.2835, lr: 0.003803, batch_cost: 0.7862, reader_cost: 0.00018, ips: 1.2719 samples/sec | ETA 02:55:43 2021-05-09 21:18:48 [INFO] [TRAIN] epoch: 72, iter: 26600/40000, loss: 0.3483, lr: 0.003800, batch_cost: 0.7872, reader_cost: 0.00018, ips: 1.2703 samples/sec | ETA 02:55:48 2021-05-09 21:18:56 [INFO] [TRAIN] epoch: 72, iter: 26610/40000, loss: 0.1750, lr: 0.003798, batch_cost: 0.7867, reader_cost: 0.00017, ips: 1.2712 samples/sec | ETA 02:55:33 2021-05-09 21:19:04 [INFO] [TRAIN] epoch: 72, iter: 26620/40000, loss: 0.3385, lr: 0.003795, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 02:55:17 2021-05-09 21:19:12 [INFO] [TRAIN] epoch: 72, iter: 26630/40000, loss: 0.2710, lr: 0.003793, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2711 samples/sec | ETA 02:55:18 2021-05-09 21:19:20 [INFO] [TRAIN] epoch: 72, iter: 26640/40000, loss: 0.1775, lr: 0.003790, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2703 samples/sec | ETA 02:55:17 2021-05-09 21:19:28 [INFO] [TRAIN] epoch: 72, iter: 26650/40000, loss: 0.2415, lr: 0.003788, batch_cost: 0.7850, reader_cost: 0.00015, ips: 1.2739 samples/sec | ETA 02:54:39 2021-05-09 21:19:35 [INFO] [TRAIN] epoch: 72, iter: 26660/40000, loss: 0.1930, lr: 0.003785, batch_cost: 0.7878, reader_cost: 0.00017, ips: 1.2693 samples/sec | ETA 02:55:09 2021-05-09 21:19:43 [INFO] [TRAIN] epoch: 72, iter: 26670/40000, loss: 0.2219, lr: 0.003783, batch_cost: 0.7891, reader_cost: 0.00016, ips: 1.2672 samples/sec | ETA 02:55:19 2021-05-09 21:19:51 [INFO] [TRAIN] epoch: 72, iter: 26680/40000, loss: 0.2118, lr: 0.003780, batch_cost: 0.7859, reader_cost: 0.00017, ips: 1.2724 samples/sec | ETA 02:54:28 2021-05-09 21:19:59 [INFO] [TRAIN] epoch: 72, iter: 26690/40000, loss: 0.3830, lr: 0.003778, batch_cost: 0.7874, reader_cost: 0.00017, ips: 1.2701 samples/sec | ETA 02:54:39 2021-05-09 21:20:07 [INFO] [TRAIN] epoch: 72, iter: 26700/40000, loss: 0.4237, lr: 0.003775, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 02:54:22 2021-05-09 21:20:15 [INFO] [TRAIN] epoch: 72, iter: 26710/40000, loss: 0.3816, lr: 0.003773, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2729 samples/sec | ETA 02:54:01 2021-05-09 21:20:23 [INFO] [TRAIN] epoch: 72, iter: 26720/40000, loss: 0.4420, lr: 0.003770, batch_cost: 0.7840, reader_cost: 0.00015, ips: 1.2755 samples/sec | ETA 02:53:31 2021-05-09 21:20:30 [INFO] [TRAIN] epoch: 72, iter: 26730/40000, loss: 0.3236, lr: 0.003768, batch_cost: 0.7883, reader_cost: 0.00016, ips: 1.2685 samples/sec | ETA 02:54:20 2021-05-09 21:20:38 [INFO] [TRAIN] epoch: 72, iter: 26740/40000, loss: 0.2443, lr: 0.003765, batch_cost: 0.7868, reader_cost: 0.00017, ips: 1.2710 samples/sec | ETA 02:53:52 2021-05-09 21:20:46 [INFO] [TRAIN] epoch: 72, iter: 26750/40000, loss: 0.2285, lr: 0.003763, batch_cost: 0.7867, reader_cost: 0.00017, ips: 1.2711 samples/sec | ETA 02:53:43 2021-05-09 21:20:54 [INFO] [TRAIN] epoch: 72, iter: 26760/40000, loss: 0.1732, lr: 0.003760, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2734 samples/sec | ETA 02:53:16 2021-05-09 21:21:02 [INFO] [TRAIN] epoch: 72, iter: 26770/40000, loss: 0.2732, lr: 0.003758, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2734 samples/sec | ETA 02:53:09 2021-05-09 21:21:10 [INFO] [TRAIN] epoch: 72, iter: 26780/40000, loss: 0.4001, lr: 0.003755, batch_cost: 0.7850, reader_cost: 0.00011, ips: 1.2739 samples/sec | ETA 02:52:57 2021-05-09 21:21:21 [INFO] [TRAIN] epoch: 73, iter: 26790/40000, loss: 0.3925, lr: 0.003753, batch_cost: 1.0929, reader_cost: 0.25789, ips: 0.9150 samples/sec | ETA 04:00:37 2021-05-09 21:21:29 [INFO] [TRAIN] epoch: 73, iter: 26800/40000, loss: 0.3116, lr: 0.003750, batch_cost: 0.7938, reader_cost: 0.00036, ips: 1.2597 samples/sec | ETA 02:54:38 2021-05-09 21:21:37 [INFO] [TRAIN] epoch: 73, iter: 26810/40000, loss: 0.4477, lr: 0.003748, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2720 samples/sec | ETA 02:52:49 2021-05-09 21:21:44 [INFO] [TRAIN] epoch: 73, iter: 26820/40000, loss: 0.3134, lr: 0.003745, batch_cost: 0.7873, reader_cost: 0.00015, ips: 1.2701 samples/sec | ETA 02:52:56 2021-05-09 21:21:52 [INFO] [TRAIN] epoch: 73, iter: 26830/40000, loss: 0.4212, lr: 0.003743, batch_cost: 0.7881, reader_cost: 0.00017, ips: 1.2689 samples/sec | ETA 02:52:58 2021-05-09 21:22:00 [INFO] [TRAIN] epoch: 73, iter: 26840/40000, loss: 0.3249, lr: 0.003740, batch_cost: 0.7854, reader_cost: 0.00015, ips: 1.2733 samples/sec | ETA 02:52:15 2021-05-09 21:22:08 [INFO] [TRAIN] epoch: 73, iter: 26850/40000, loss: 0.3680, lr: 0.003738, batch_cost: 0.7883, reader_cost: 0.00016, ips: 1.2686 samples/sec | ETA 02:52:45 2021-05-09 21:22:16 [INFO] [TRAIN] epoch: 73, iter: 26860/40000, loss: 0.1378, lr: 0.003735, batch_cost: 0.7879, reader_cost: 0.00016, ips: 1.2692 samples/sec | ETA 02:52:33 2021-05-09 21:22:24 [INFO] [TRAIN] epoch: 73, iter: 26870/40000, loss: 0.3264, lr: 0.003733, batch_cost: 0.7870, reader_cost: 0.00015, ips: 1.2707 samples/sec | ETA 02:52:13 2021-05-09 21:22:32 [INFO] [TRAIN] epoch: 73, iter: 26880/40000, loss: 0.2736, lr: 0.003730, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2718 samples/sec | ETA 02:51:56 2021-05-09 21:22:40 [INFO] [TRAIN] epoch: 73, iter: 26890/40000, loss: 0.2655, lr: 0.003728, batch_cost: 0.7875, reader_cost: 0.00015, ips: 1.2698 samples/sec | ETA 02:52:04 2021-05-09 21:22:47 [INFO] [TRAIN] epoch: 73, iter: 26900/40000, loss: 0.1867, lr: 0.003725, batch_cost: 0.7882, reader_cost: 0.00018, ips: 1.2687 samples/sec | ETA 02:52:05 2021-05-09 21:22:55 [INFO] [TRAIN] epoch: 73, iter: 26910/40000, loss: 0.1324, lr: 0.003723, batch_cost: 0.7871, reader_cost: 0.00017, ips: 1.2705 samples/sec | ETA 02:51:42 2021-05-09 21:23:03 [INFO] [TRAIN] epoch: 73, iter: 26920/40000, loss: 0.4586, lr: 0.003720, batch_cost: 0.7886, reader_cost: 0.00017, ips: 1.2680 samples/sec | ETA 02:51:55 2021-05-09 21:23:11 [INFO] [TRAIN] epoch: 73, iter: 26930/40000, loss: 0.3105, lr: 0.003718, batch_cost: 0.7872, reader_cost: 0.00018, ips: 1.2703 samples/sec | ETA 02:51:29 2021-05-09 21:23:19 [INFO] [TRAIN] epoch: 73, iter: 26940/40000, loss: 0.3049, lr: 0.003715, batch_cost: 0.7875, reader_cost: 0.00015, ips: 1.2699 samples/sec | ETA 02:51:24 2021-05-09 21:23:27 [INFO] [TRAIN] epoch: 73, iter: 26950/40000, loss: 0.2353, lr: 0.003713, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 02:51:06 2021-05-09 21:23:35 [INFO] [TRAIN] epoch: 73, iter: 26960/40000, loss: 0.2597, lr: 0.003710, batch_cost: 0.7881, reader_cost: 0.00018, ips: 1.2689 samples/sec | ETA 02:51:16 2021-05-09 21:23:43 [INFO] [TRAIN] epoch: 73, iter: 26970/40000, loss: 0.2442, lr: 0.003708, batch_cost: 0.7874, reader_cost: 0.00019, ips: 1.2700 samples/sec | ETA 02:51:00 2021-05-09 21:23:50 [INFO] [TRAIN] epoch: 73, iter: 26980/40000, loss: 0.0808, lr: 0.003705, batch_cost: 0.7859, reader_cost: 0.00019, ips: 1.2725 samples/sec | ETA 02:50:32 2021-05-09 21:23:58 [INFO] [TRAIN] epoch: 73, iter: 26990/40000, loss: 0.3589, lr: 0.003703, batch_cost: 0.7878, reader_cost: 0.00018, ips: 1.2693 samples/sec | ETA 02:50:49 2021-05-09 21:24:06 [INFO] [TRAIN] epoch: 73, iter: 27000/40000, loss: 0.3073, lr: 0.003700, batch_cost: 0.7870, reader_cost: 0.00018, ips: 1.2707 samples/sec | ETA 02:50:30 2021-05-09 21:24:06 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-09 21:27:37 [INFO] [EVAL] #Images: 500 mIoU: 0.7684 Acc: 0.9568 Kappa: 0.9440 2021-05-09 21:27:37 [INFO] [EVAL] Class IoU: [0.9803 0.8464 0.92 0.6461 0.6315 0.502 0.6221 0.727 0.9151 0.6583 0.9403 0.7754 0.5571 0.9416 0.8645 0.8955 0.8125 0.6346 0.7289] 2021-05-09 21:27:37 [INFO] [EVAL] Class Acc: [0.9937 0.9054 0.9536 0.8182 0.8019 0.7343 0.8079 0.8745 0.9489 0.8196 0.9615 0.86 0.7805 0.963 0.9585 0.9455 0.8991 0.8108 0.8338] 2021-05-09 21:28:26 [INFO] [EVAL] The model with the best validation mIoU (0.7684) was saved at iter 27000. 2021-05-09 21:28:34 [INFO] [TRAIN] epoch: 73, iter: 27010/40000, loss: 0.1616, lr: 0.003698, batch_cost: 0.7842, reader_cost: 0.00025, ips: 1.2752 samples/sec | ETA 02:49:46 2021-05-09 21:28:42 [INFO] [TRAIN] epoch: 73, iter: 27020/40000, loss: 0.2646, lr: 0.003695, batch_cost: 0.7838, reader_cost: 0.00032, ips: 1.2758 samples/sec | ETA 02:49:33 2021-05-09 21:28:50 [INFO] [TRAIN] epoch: 73, iter: 27030/40000, loss: 0.1896, lr: 0.003693, batch_cost: 0.7831, reader_cost: 0.00015, ips: 1.2770 samples/sec | ETA 02:49:16 2021-05-09 21:28:58 [INFO] [TRAIN] epoch: 73, iter: 27040/40000, loss: 0.2597, lr: 0.003691, batch_cost: 0.7851, reader_cost: 0.00015, ips: 1.2738 samples/sec | ETA 02:49:34 2021-05-09 21:29:06 [INFO] [TRAIN] epoch: 73, iter: 27050/40000, loss: 0.1863, lr: 0.003688, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2729 samples/sec | ETA 02:49:33 2021-05-09 21:29:14 [INFO] [TRAIN] epoch: 73, iter: 27060/40000, loss: 0.3354, lr: 0.003686, batch_cost: 0.7881, reader_cost: 0.00016, ips: 1.2689 samples/sec | ETA 02:49:57 2021-05-09 21:29:21 [INFO] [TRAIN] epoch: 73, iter: 27070/40000, loss: 0.3957, lr: 0.003683, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2729 samples/sec | ETA 02:49:17 2021-05-09 21:29:29 [INFO] [TRAIN] epoch: 73, iter: 27080/40000, loss: 0.2312, lr: 0.003681, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2716 samples/sec | ETA 02:49:20 2021-05-09 21:29:37 [INFO] [TRAIN] epoch: 73, iter: 27090/40000, loss: 0.3542, lr: 0.003678, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2704 samples/sec | ETA 02:49:22 2021-05-09 21:29:45 [INFO] [TRAIN] epoch: 73, iter: 27100/40000, loss: 0.2856, lr: 0.003676, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2729 samples/sec | ETA 02:48:54 2021-05-09 21:29:53 [INFO] [TRAIN] epoch: 73, iter: 27110/40000, loss: 0.2169, lr: 0.003673, batch_cost: 0.7869, reader_cost: 0.00015, ips: 1.2708 samples/sec | ETA 02:49:02 2021-05-09 21:30:01 [INFO] [TRAIN] epoch: 73, iter: 27120/40000, loss: 0.1673, lr: 0.003671, batch_cost: 0.7873, reader_cost: 0.00015, ips: 1.2701 samples/sec | ETA 02:49:01 2021-05-09 21:30:09 [INFO] [TRAIN] epoch: 73, iter: 27130/40000, loss: 0.2684, lr: 0.003668, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2716 samples/sec | ETA 02:48:41 2021-05-09 21:30:17 [INFO] [TRAIN] epoch: 73, iter: 27140/40000, loss: 0.2724, lr: 0.003666, batch_cost: 0.7875, reader_cost: 0.00016, ips: 1.2699 samples/sec | ETA 02:48:46 2021-05-09 21:30:24 [INFO] [TRAIN] epoch: 73, iter: 27150/40000, loss: 0.3246, lr: 0.003663, batch_cost: 0.7851, reader_cost: 0.00013, ips: 1.2737 samples/sec | ETA 02:48:08 2021-05-09 21:30:35 [INFO] [TRAIN] epoch: 74, iter: 27160/40000, loss: 0.2795, lr: 0.003661, batch_cost: 1.1027, reader_cost: 0.25443, ips: 0.9068 samples/sec | ETA 03:55:58 2021-05-09 21:30:43 [INFO] [TRAIN] epoch: 74, iter: 27170/40000, loss: 0.2437, lr: 0.003658, batch_cost: 0.7968, reader_cost: 0.00032, ips: 1.2551 samples/sec | ETA 02:50:22 2021-05-09 21:30:51 [INFO] [TRAIN] epoch: 74, iter: 27180/40000, loss: 0.4648, lr: 0.003656, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2720 samples/sec | ETA 02:47:58 2021-05-09 21:30:59 [INFO] [TRAIN] epoch: 74, iter: 27190/40000, loss: 0.3463, lr: 0.003653, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 02:47:54 2021-05-09 21:31:07 [INFO] [TRAIN] epoch: 74, iter: 27200/40000, loss: 0.3709, lr: 0.003651, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2711 samples/sec | ETA 02:47:50 2021-05-09 21:31:15 [INFO] [TRAIN] epoch: 74, iter: 27210/40000, loss: 0.4728, lr: 0.003648, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2713 samples/sec | ETA 02:47:40 2021-05-09 21:31:23 [INFO] [TRAIN] epoch: 74, iter: 27220/40000, loss: 0.3613, lr: 0.003646, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2727 samples/sec | ETA 02:47:21 2021-05-09 21:31:31 [INFO] [TRAIN] epoch: 74, iter: 27230/40000, loss: 0.1227, lr: 0.003643, batch_cost: 0.7858, reader_cost: 0.00017, ips: 1.2727 samples/sec | ETA 02:47:14 2021-05-09 21:31:38 [INFO] [TRAIN] epoch: 74, iter: 27240/40000, loss: 0.3357, lr: 0.003641, batch_cost: 0.7863, reader_cost: 0.00018, ips: 1.2717 samples/sec | ETA 02:47:13 2021-05-09 21:31:46 [INFO] [TRAIN] epoch: 74, iter: 27250/40000, loss: 0.3043, lr: 0.003638, batch_cost: 0.7871, reader_cost: 0.00015, ips: 1.2705 samples/sec | ETA 02:47:15 2021-05-09 21:31:54 [INFO] [TRAIN] epoch: 74, iter: 27260/40000, loss: 0.3062, lr: 0.003636, batch_cost: 0.7879, reader_cost: 0.00016, ips: 1.2692 samples/sec | ETA 02:47:17 2021-05-09 21:32:02 [INFO] [TRAIN] epoch: 74, iter: 27270/40000, loss: 0.3963, lr: 0.003633, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 02:46:49 2021-05-09 21:32:10 [INFO] [TRAIN] epoch: 74, iter: 27280/40000, loss: 0.2846, lr: 0.003631, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2733 samples/sec | ETA 02:46:29 2021-05-09 21:32:18 [INFO] [TRAIN] epoch: 74, iter: 27290/40000, loss: 0.3329, lr: 0.003628, batch_cost: 0.7886, reader_cost: 0.00016, ips: 1.2681 samples/sec | ETA 02:47:02 2021-05-09 21:32:26 [INFO] [TRAIN] epoch: 74, iter: 27300/40000, loss: 0.2396, lr: 0.003626, batch_cost: 0.7876, reader_cost: 0.00015, ips: 1.2698 samples/sec | ETA 02:46:41 2021-05-09 21:32:33 [INFO] [TRAIN] epoch: 74, iter: 27310/40000, loss: 0.2658, lr: 0.003623, batch_cost: 0.7845, reader_cost: 0.00016, ips: 1.2747 samples/sec | ETA 02:45:54 2021-05-09 21:32:41 [INFO] [TRAIN] epoch: 74, iter: 27320/40000, loss: 0.2094, lr: 0.003621, batch_cost: 0.7873, reader_cost: 0.00016, ips: 1.2701 samples/sec | ETA 02:46:23 2021-05-09 21:32:49 [INFO] [TRAIN] epoch: 74, iter: 27330/40000, loss: 0.1672, lr: 0.003618, batch_cost: 0.7858, reader_cost: 0.00018, ips: 1.2726 samples/sec | ETA 02:45:55 2021-05-09 21:32:57 [INFO] [TRAIN] epoch: 74, iter: 27340/40000, loss: 0.2390, lr: 0.003616, batch_cost: 0.7873, reader_cost: 0.00016, ips: 1.2702 samples/sec | ETA 02:46:06 2021-05-09 21:33:05 [INFO] [TRAIN] epoch: 74, iter: 27350/40000, loss: 0.1125, lr: 0.003613, batch_cost: 0.7876, reader_cost: 0.00016, ips: 1.2696 samples/sec | ETA 02:46:03 2021-05-09 21:33:13 [INFO] [TRAIN] epoch: 74, iter: 27360/40000, loss: 0.2404, lr: 0.003611, batch_cost: 0.7852, reader_cost: 0.00014, ips: 1.2736 samples/sec | ETA 02:45:24 2021-05-09 21:33:21 [INFO] [TRAIN] epoch: 74, iter: 27370/40000, loss: 0.3115, lr: 0.003608, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2721 samples/sec | ETA 02:45:28 2021-05-09 21:33:29 [INFO] [TRAIN] epoch: 74, iter: 27380/40000, loss: 0.1966, lr: 0.003606, batch_cost: 0.7854, reader_cost: 0.00015, ips: 1.2732 samples/sec | ETA 02:45:12 2021-05-09 21:33:36 [INFO] [TRAIN] epoch: 74, iter: 27390/40000, loss: 0.2519, lr: 0.003603, batch_cost: 0.7865, reader_cost: 0.00017, ips: 1.2715 samples/sec | ETA 02:45:17 2021-05-09 21:33:44 [INFO] [TRAIN] epoch: 74, iter: 27400/40000, loss: 0.2078, lr: 0.003601, batch_cost: 0.7845, reader_cost: 0.00017, ips: 1.2748 samples/sec | ETA 02:44:44 2021-05-09 21:33:52 [INFO] [TRAIN] epoch: 74, iter: 27410/40000, loss: 0.2089, lr: 0.003598, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2735 samples/sec | ETA 02:44:46 2021-05-09 21:34:00 [INFO] [TRAIN] epoch: 74, iter: 27420/40000, loss: 0.8266, lr: 0.003596, batch_cost: 0.7869, reader_cost: 0.00015, ips: 1.2708 samples/sec | ETA 02:44:58 2021-05-09 21:34:08 [INFO] [TRAIN] epoch: 74, iter: 27430/40000, loss: 0.3766, lr: 0.003593, batch_cost: 0.7869, reader_cost: 0.00015, ips: 1.2708 samples/sec | ETA 02:44:51 2021-05-09 21:34:16 [INFO] [TRAIN] epoch: 74, iter: 27440/40000, loss: 0.4755, lr: 0.003591, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2718 samples/sec | ETA 02:44:35 2021-05-09 21:34:24 [INFO] [TRAIN] epoch: 74, iter: 27450/40000, loss: 0.2382, lr: 0.003588, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2724 samples/sec | ETA 02:44:23 2021-05-09 21:34:31 [INFO] [TRAIN] epoch: 74, iter: 27460/40000, loss: 0.4570, lr: 0.003586, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 02:44:22 2021-05-09 21:34:39 [INFO] [TRAIN] epoch: 74, iter: 27470/40000, loss: 0.2705, lr: 0.003583, batch_cost: 0.7856, reader_cost: 0.00017, ips: 1.2728 samples/sec | ETA 02:44:04 2021-05-09 21:34:47 [INFO] [TRAIN] epoch: 74, iter: 27480/40000, loss: 0.3438, lr: 0.003581, batch_cost: 0.7866, reader_cost: 0.00017, ips: 1.2713 samples/sec | ETA 02:44:08 2021-05-09 21:34:55 [INFO] [TRAIN] epoch: 74, iter: 27490/40000, loss: 0.1101, lr: 0.003578, batch_cost: 0.7874, reader_cost: 0.00015, ips: 1.2700 samples/sec | ETA 02:44:10 2021-05-09 21:35:03 [INFO] [TRAIN] epoch: 74, iter: 27500/40000, loss: 0.2680, lr: 0.003576, batch_cost: 0.7883, reader_cost: 0.00016, ips: 1.2685 samples/sec | ETA 02:44:14 2021-05-09 21:35:11 [INFO] [TRAIN] epoch: 74, iter: 27510/40000, loss: 0.3105, lr: 0.003573, batch_cost: 0.7849, reader_cost: 0.00014, ips: 1.2741 samples/sec | ETA 02:43:23 2021-05-09 21:35:19 [INFO] [TRAIN] epoch: 74, iter: 27520/40000, loss: 0.3026, lr: 0.003571, batch_cost: 0.7855, reader_cost: 0.00014, ips: 1.2731 samples/sec | ETA 02:43:22 2021-05-09 21:35:29 [INFO] [TRAIN] epoch: 75, iter: 27530/40000, loss: 0.3381, lr: 0.003568, batch_cost: 1.0890, reader_cost: 0.26487, ips: 0.9182 samples/sec | ETA 03:46:20 2021-05-09 21:35:38 [INFO] [TRAIN] epoch: 75, iter: 27540/40000, loss: 0.3458, lr: 0.003566, batch_cost: 0.8005, reader_cost: 0.00035, ips: 1.2492 samples/sec | ETA 02:46:14 2021-05-09 21:35:45 [INFO] [TRAIN] epoch: 75, iter: 27550/40000, loss: 0.4193, lr: 0.003563, batch_cost: 0.7862, reader_cost: 0.00017, ips: 1.2720 samples/sec | ETA 02:43:08 2021-05-09 21:35:53 [INFO] [TRAIN] epoch: 75, iter: 27560/40000, loss: 0.2622, lr: 0.003561, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2726 samples/sec | ETA 02:42:55 2021-05-09 21:36:01 [INFO] [TRAIN] epoch: 75, iter: 27570/40000, loss: 0.4025, lr: 0.003558, batch_cost: 0.7850, reader_cost: 0.00015, ips: 1.2739 samples/sec | ETA 02:42:37 2021-05-09 21:36:09 [INFO] [TRAIN] epoch: 75, iter: 27580/40000, loss: 0.4902, lr: 0.003556, batch_cost: 0.7843, reader_cost: 0.00016, ips: 1.2750 samples/sec | ETA 02:42:20 2021-05-09 21:36:17 [INFO] [TRAIN] epoch: 75, iter: 27590/40000, loss: 0.2199, lr: 0.003553, batch_cost: 0.7867, reader_cost: 0.00015, ips: 1.2712 samples/sec | ETA 02:42:42 2021-05-09 21:36:25 [INFO] [TRAIN] epoch: 75, iter: 27600/40000, loss: 0.1444, lr: 0.003551, batch_cost: 0.7878, reader_cost: 0.00015, ips: 1.2694 samples/sec | ETA 02:42:48 2021-05-09 21:36:33 [INFO] [TRAIN] epoch: 75, iter: 27610/40000, loss: 0.1929, lr: 0.003548, batch_cost: 0.7867, reader_cost: 0.00017, ips: 1.2711 samples/sec | ETA 02:42:27 2021-05-09 21:36:40 [INFO] [TRAIN] epoch: 75, iter: 27620/40000, loss: 0.3769, lr: 0.003546, batch_cost: 0.7872, reader_cost: 0.00017, ips: 1.2703 samples/sec | ETA 02:42:26 2021-05-09 21:36:48 [INFO] [TRAIN] epoch: 75, iter: 27630/40000, loss: 0.2948, lr: 0.003543, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2714 samples/sec | ETA 02:42:09 2021-05-09 21:36:56 [INFO] [TRAIN] epoch: 75, iter: 27640/40000, loss: 0.2292, lr: 0.003541, batch_cost: 0.7872, reader_cost: 0.00015, ips: 1.2703 samples/sec | ETA 02:42:10 2021-05-09 21:37:04 [INFO] [TRAIN] epoch: 75, iter: 27650/40000, loss: 0.1301, lr: 0.003538, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2710 samples/sec | ETA 02:41:56 2021-05-09 21:37:12 [INFO] [TRAIN] epoch: 75, iter: 27660/40000, loss: 0.2778, lr: 0.003536, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2723 samples/sec | ETA 02:41:38 2021-05-09 21:37:20 [INFO] [TRAIN] epoch: 75, iter: 27670/40000, loss: 0.3611, lr: 0.003533, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2709 samples/sec | ETA 02:41:41 2021-05-09 21:37:28 [INFO] [TRAIN] epoch: 75, iter: 27680/40000, loss: 0.2682, lr: 0.003531, batch_cost: 0.7876, reader_cost: 0.00016, ips: 1.2697 samples/sec | ETA 02:41:43 2021-05-09 21:37:35 [INFO] [TRAIN] epoch: 75, iter: 27690/40000, loss: 0.2457, lr: 0.003528, batch_cost: 0.7862, reader_cost: 0.00017, ips: 1.2720 samples/sec | ETA 02:41:18 2021-05-09 21:37:43 [INFO] [TRAIN] epoch: 75, iter: 27700/40000, loss: 0.2219, lr: 0.003526, batch_cost: 0.7869, reader_cost: 0.00017, ips: 1.2708 samples/sec | ETA 02:41:18 2021-05-09 21:37:51 [INFO] [TRAIN] epoch: 75, iter: 27710/40000, loss: 0.1685, lr: 0.003523, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2706 samples/sec | ETA 02:41:12 2021-05-09 21:37:59 [INFO] [TRAIN] epoch: 75, iter: 27720/40000, loss: 0.1845, lr: 0.003521, batch_cost: 0.7867, reader_cost: 0.00017, ips: 1.2712 samples/sec | ETA 02:41:00 2021-05-09 21:38:07 [INFO] [TRAIN] epoch: 75, iter: 27730/40000, loss: 0.0780, lr: 0.003518, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 02:40:48 2021-05-09 21:38:15 [INFO] [TRAIN] epoch: 75, iter: 27740/40000, loss: 0.2325, lr: 0.003516, batch_cost: 0.7843, reader_cost: 0.00015, ips: 1.2750 samples/sec | ETA 02:40:15 2021-05-09 21:38:23 [INFO] [TRAIN] epoch: 75, iter: 27750/40000, loss: 0.1732, lr: 0.003513, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2723 samples/sec | ETA 02:40:28 2021-05-09 21:38:31 [INFO] [TRAIN] epoch: 75, iter: 27760/40000, loss: 0.2307, lr: 0.003510, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2731 samples/sec | ETA 02:40:13 2021-05-09 21:38:38 [INFO] [TRAIN] epoch: 75, iter: 27770/40000, loss: 0.2411, lr: 0.003508, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2724 samples/sec | ETA 02:40:11 2021-05-09 21:38:46 [INFO] [TRAIN] epoch: 75, iter: 27780/40000, loss: 0.1766, lr: 0.003505, batch_cost: 0.7851, reader_cost: 0.00016, ips: 1.2737 samples/sec | ETA 02:39:54 2021-05-09 21:38:54 [INFO] [TRAIN] epoch: 75, iter: 27790/40000, loss: 0.1983, lr: 0.003503, batch_cost: 0.7866, reader_cost: 0.00017, ips: 1.2714 samples/sec | ETA 02:40:03 2021-05-09 21:39:02 [INFO] [TRAIN] epoch: 75, iter: 27800/40000, loss: 0.2996, lr: 0.003500, batch_cost: 0.7869, reader_cost: 0.00017, ips: 1.2708 samples/sec | ETA 02:40:00 2021-05-09 21:39:10 [INFO] [TRAIN] epoch: 75, iter: 27810/40000, loss: 0.3713, lr: 0.003498, batch_cost: 0.7858, reader_cost: 0.00017, ips: 1.2726 samples/sec | ETA 02:39:39 2021-05-09 21:39:18 [INFO] [TRAIN] epoch: 75, iter: 27820/40000, loss: 0.2611, lr: 0.003495, batch_cost: 0.7847, reader_cost: 0.00015, ips: 1.2743 samples/sec | ETA 02:39:18 2021-05-09 21:39:26 [INFO] [TRAIN] epoch: 75, iter: 27830/40000, loss: 0.6435, lr: 0.003493, batch_cost: 0.7853, reader_cost: 0.00015, ips: 1.2734 samples/sec | ETA 02:39:17 2021-05-09 21:39:33 [INFO] [TRAIN] epoch: 75, iter: 27840/40000, loss: 0.5026, lr: 0.003490, batch_cost: 0.7851, reader_cost: 0.00015, ips: 1.2737 samples/sec | ETA 02:39:07 2021-05-09 21:39:41 [INFO] [TRAIN] epoch: 75, iter: 27850/40000, loss: 0.3553, lr: 0.003488, batch_cost: 0.7877, reader_cost: 0.00015, ips: 1.2695 samples/sec | ETA 02:39:31 2021-05-09 21:39:49 [INFO] [TRAIN] epoch: 75, iter: 27860/40000, loss: 0.1589, lr: 0.003485, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2720 samples/sec | ETA 02:39:04 2021-05-09 21:39:57 [INFO] [TRAIN] epoch: 75, iter: 27870/40000, loss: 0.2252, lr: 0.003483, batch_cost: 0.7849, reader_cost: 0.00016, ips: 1.2741 samples/sec | ETA 02:38:40 2021-05-09 21:40:05 [INFO] [TRAIN] epoch: 75, iter: 27880/40000, loss: 0.3012, lr: 0.003480, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2723 samples/sec | ETA 02:38:46 2021-05-09 21:40:13 [INFO] [TRAIN] epoch: 75, iter: 27890/40000, loss: 0.3504, lr: 0.003478, batch_cost: 0.7847, reader_cost: 0.00015, ips: 1.2743 samples/sec | ETA 02:38:23 2021-05-09 21:40:21 [INFO] [TRAIN] epoch: 75, iter: 27900/40000, loss: 0.3088, lr: 0.003475, batch_cost: 0.7847, reader_cost: 0.00010, ips: 1.2744 samples/sec | ETA 02:38:14 2021-05-09 21:40:32 [INFO] [TRAIN] epoch: 76, iter: 27910/40000, loss: 0.2355, lr: 0.003473, batch_cost: 1.1084, reader_cost: 0.29218, ips: 0.9022 samples/sec | ETA 03:43:20 2021-05-09 21:40:40 [INFO] [TRAIN] epoch: 76, iter: 27920/40000, loss: 0.4516, lr: 0.003470, batch_cost: 0.7904, reader_cost: 0.00034, ips: 1.2652 samples/sec | ETA 02:39:07 2021-05-09 21:40:47 [INFO] [TRAIN] epoch: 76, iter: 27930/40000, loss: 0.2619, lr: 0.003468, batch_cost: 0.7869, reader_cost: 0.00015, ips: 1.2708 samples/sec | ETA 02:38:17 2021-05-09 21:40:55 [INFO] [TRAIN] epoch: 76, iter: 27940/40000, loss: 0.4346, lr: 0.003465, batch_cost: 0.7847, reader_cost: 0.00015, ips: 1.2744 samples/sec | ETA 02:37:43 2021-05-09 21:41:03 [INFO] [TRAIN] epoch: 76, iter: 27950/40000, loss: 0.4450, lr: 0.003463, batch_cost: 0.7858, reader_cost: 0.00015, ips: 1.2726 samples/sec | ETA 02:37:48 2021-05-09 21:41:11 [INFO] [TRAIN] epoch: 76, iter: 27960/40000, loss: 0.2921, lr: 0.003460, batch_cost: 0.7856, reader_cost: 0.00014, ips: 1.2730 samples/sec | ETA 02:37:38 2021-05-09 21:41:19 [INFO] [TRAIN] epoch: 76, iter: 27970/40000, loss: 0.1479, lr: 0.003458, batch_cost: 0.7872, reader_cost: 0.00015, ips: 1.2703 samples/sec | ETA 02:37:50 2021-05-09 21:41:27 [INFO] [TRAIN] epoch: 76, iter: 27980/40000, loss: 0.2155, lr: 0.003455, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2730 samples/sec | ETA 02:37:22 2021-05-09 21:41:35 [INFO] [TRAIN] epoch: 76, iter: 27990/40000, loss: 0.2875, lr: 0.003453, batch_cost: 0.7855, reader_cost: 0.00018, ips: 1.2730 samples/sec | ETA 02:37:14 2021-05-09 21:41:42 [INFO] [TRAIN] epoch: 76, iter: 28000/40000, loss: 0.1990, lr: 0.003450, batch_cost: 0.7876, reader_cost: 0.00016, ips: 1.2697 samples/sec | ETA 02:37:31 2021-05-09 21:41:42 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-09 21:45:13 [INFO] [EVAL] #Images: 500 mIoU: 0.7601 Acc: 0.9564 Kappa: 0.9434 2021-05-09 21:45:13 [INFO] [EVAL] Class IoU: [0.9801 0.8427 0.918 0.6342 0.6215 0.483 0.6206 0.7261 0.9154 0.6615 0.9404 0.7757 0.5609 0.9417 0.8341 0.8751 0.739 0.6403 0.7318] 2021-05-09 21:45:13 [INFO] [EVAL] Class Acc: [0.9913 0.9098 0.9485 0.8431 0.8355 0.7724 0.82 0.8958 0.946 0.8382 0.9692 0.8587 0.7847 0.9654 0.9462 0.9579 0.7903 0.8082 0.8228] 2021-05-09 21:45:42 [INFO] [EVAL] The model with the best validation mIoU (0.7684) was saved at iter 27000. 2021-05-09 21:45:50 [INFO] [TRAIN] epoch: 76, iter: 28010/40000, loss: 0.3012, lr: 0.003448, batch_cost: 0.7812, reader_cost: 0.00024, ips: 1.2801 samples/sec | ETA 02:36:06 2021-05-09 21:45:58 [INFO] [TRAIN] epoch: 76, iter: 28020/40000, loss: 0.1786, lr: 0.003445, batch_cost: 0.7850, reader_cost: 0.00019, ips: 1.2739 samples/sec | ETA 02:36:44 2021-05-09 21:46:05 [INFO] [TRAIN] epoch: 76, iter: 28030/40000, loss: 0.2904, lr: 0.003443, batch_cost: 0.7856, reader_cost: 0.00017, ips: 1.2729 samples/sec | ETA 02:36:43 2021-05-09 21:46:13 [INFO] [TRAIN] epoch: 76, iter: 28040/40000, loss: 0.3234, lr: 0.003440, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2711 samples/sec | ETA 02:36:49 2021-05-09 21:46:21 [INFO] [TRAIN] epoch: 76, iter: 28050/40000, loss: 0.2781, lr: 0.003438, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2733 samples/sec | ETA 02:36:25 2021-05-09 21:46:29 [INFO] [TRAIN] epoch: 76, iter: 28060/40000, loss: 0.3000, lr: 0.003435, batch_cost: 0.7848, reader_cost: 0.00016, ips: 1.2742 samples/sec | ETA 02:36:10 2021-05-09 21:46:37 [INFO] [TRAIN] epoch: 76, iter: 28070/40000, loss: 0.2350, lr: 0.003433, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2727 samples/sec | ETA 02:36:13 2021-05-09 21:46:45 [INFO] [TRAIN] epoch: 76, iter: 28080/40000, loss: 0.2824, lr: 0.003430, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2725 samples/sec | ETA 02:36:07 2021-05-09 21:46:53 [INFO] [TRAIN] epoch: 76, iter: 28090/40000, loss: 0.1697, lr: 0.003428, batch_cost: 0.7874, reader_cost: 0.00015, ips: 1.2699 samples/sec | ETA 02:36:18 2021-05-09 21:47:00 [INFO] [TRAIN] epoch: 76, iter: 28100/40000, loss: 0.1062, lr: 0.003425, batch_cost: 0.7836, reader_cost: 0.00015, ips: 1.2761 samples/sec | ETA 02:35:24 2021-05-09 21:47:08 [INFO] [TRAIN] epoch: 76, iter: 28110/40000, loss: 0.3205, lr: 0.003423, batch_cost: 0.7841, reader_cost: 0.00015, ips: 1.2753 samples/sec | ETA 02:35:23 2021-05-09 21:47:16 [INFO] [TRAIN] epoch: 76, iter: 28120/40000, loss: 0.2962, lr: 0.003420, batch_cost: 0.7870, reader_cost: 0.00015, ips: 1.2706 samples/sec | ETA 02:35:49 2021-05-09 21:47:24 [INFO] [TRAIN] epoch: 76, iter: 28130/40000, loss: 0.2072, lr: 0.003418, batch_cost: 0.7871, reader_cost: 0.00015, ips: 1.2705 samples/sec | ETA 02:35:42 2021-05-09 21:47:32 [INFO] [TRAIN] epoch: 76, iter: 28140/40000, loss: 0.2716, lr: 0.003415, batch_cost: 0.7859, reader_cost: 0.00017, ips: 1.2724 samples/sec | ETA 02:35:20 2021-05-09 21:47:40 [INFO] [TRAIN] epoch: 76, iter: 28150/40000, loss: 0.1803, lr: 0.003413, batch_cost: 0.7867, reader_cost: 0.00019, ips: 1.2711 samples/sec | ETA 02:35:22 2021-05-09 21:47:48 [INFO] [TRAIN] epoch: 76, iter: 28160/40000, loss: 0.2255, lr: 0.003410, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2717 samples/sec | ETA 02:35:10 2021-05-09 21:47:55 [INFO] [TRAIN] epoch: 76, iter: 28170/40000, loss: 0.2785, lr: 0.003408, batch_cost: 0.7846, reader_cost: 0.00016, ips: 1.2746 samples/sec | ETA 02:34:41 2021-05-09 21:48:03 [INFO] [TRAIN] epoch: 76, iter: 28180/40000, loss: 0.3124, lr: 0.003405, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 02:34:47 2021-05-09 21:48:11 [INFO] [TRAIN] epoch: 76, iter: 28190/40000, loss: 0.3174, lr: 0.003402, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2712 samples/sec | ETA 02:34:50 2021-05-09 21:48:19 [INFO] [TRAIN] epoch: 76, iter: 28200/40000, loss: 0.4607, lr: 0.003400, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 02:34:40 2021-05-09 21:48:27 [INFO] [TRAIN] epoch: 76, iter: 28210/40000, loss: 0.4045, lr: 0.003397, batch_cost: 0.7842, reader_cost: 0.00016, ips: 1.2751 samples/sec | ETA 02:34:06 2021-05-09 21:48:35 [INFO] [TRAIN] epoch: 76, iter: 28220/40000, loss: 0.2325, lr: 0.003395, batch_cost: 0.7844, reader_cost: 0.00015, ips: 1.2748 samples/sec | ETA 02:34:00 2021-05-09 21:48:43 [INFO] [TRAIN] epoch: 76, iter: 28230/40000, loss: 0.1343, lr: 0.003392, batch_cost: 0.7872, reader_cost: 0.00017, ips: 1.2703 samples/sec | ETA 02:34:25 2021-05-09 21:48:50 [INFO] [TRAIN] epoch: 76, iter: 28240/40000, loss: 0.2854, lr: 0.003390, batch_cost: 0.7854, reader_cost: 0.00017, ips: 1.2732 samples/sec | ETA 02:33:56 2021-05-09 21:48:58 [INFO] [TRAIN] epoch: 76, iter: 28250/40000, loss: 0.2013, lr: 0.003387, batch_cost: 0.7869, reader_cost: 0.00017, ips: 1.2709 samples/sec | ETA 02:34:05 2021-05-09 21:49:06 [INFO] [TRAIN] epoch: 76, iter: 28260/40000, loss: 0.3345, lr: 0.003385, batch_cost: 0.7869, reader_cost: 0.00015, ips: 1.2708 samples/sec | ETA 02:33:58 2021-05-09 21:49:14 [INFO] [TRAIN] epoch: 76, iter: 28270/40000, loss: 0.2719, lr: 0.003382, batch_cost: 0.7863, reader_cost: 0.00010, ips: 1.2718 samples/sec | ETA 02:33:42 2021-05-09 21:49:25 [INFO] [TRAIN] epoch: 77, iter: 28280/40000, loss: 0.3289, lr: 0.003380, batch_cost: 1.1060, reader_cost: 0.26354, ips: 0.9041 samples/sec | ETA 03:36:02 2021-05-09 21:49:33 [INFO] [TRAIN] epoch: 77, iter: 28290/40000, loss: 0.3339, lr: 0.003377, batch_cost: 0.7934, reader_cost: 0.00033, ips: 1.2603 samples/sec | ETA 02:34:51 2021-05-09 21:49:41 [INFO] [TRAIN] epoch: 77, iter: 28300/40000, loss: 0.4526, lr: 0.003375, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2735 samples/sec | ETA 02:33:07 2021-05-09 21:49:49 [INFO] [TRAIN] epoch: 77, iter: 28310/40000, loss: 0.3654, lr: 0.003372, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2735 samples/sec | ETA 02:32:59 2021-05-09 21:49:57 [INFO] [TRAIN] epoch: 77, iter: 28320/40000, loss: 0.3562, lr: 0.003370, batch_cost: 0.7856, reader_cost: 0.00017, ips: 1.2729 samples/sec | ETA 02:32:56 2021-05-09 21:50:04 [INFO] [TRAIN] epoch: 77, iter: 28330/40000, loss: 0.3532, lr: 0.003367, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 02:32:46 2021-05-09 21:50:12 [INFO] [TRAIN] epoch: 77, iter: 28340/40000, loss: 0.2117, lr: 0.003365, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2703 samples/sec | ETA 02:32:59 2021-05-09 21:50:20 [INFO] [TRAIN] epoch: 77, iter: 28350/40000, loss: 0.2361, lr: 0.003362, batch_cost: 0.7854, reader_cost: 0.00015, ips: 1.2732 samples/sec | ETA 02:32:30 2021-05-09 21:50:28 [INFO] [TRAIN] epoch: 77, iter: 28360/40000, loss: 0.3236, lr: 0.003360, batch_cost: 0.7847, reader_cost: 0.00014, ips: 1.2744 samples/sec | ETA 02:32:13 2021-05-09 21:50:36 [INFO] [TRAIN] epoch: 77, iter: 28370/40000, loss: 0.2713, lr: 0.003357, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2724 samples/sec | ETA 02:32:20 2021-05-09 21:50:44 [INFO] [TRAIN] epoch: 77, iter: 28380/40000, loss: 0.4566, lr: 0.003355, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 02:32:07 2021-05-09 21:50:52 [INFO] [TRAIN] epoch: 77, iter: 28390/40000, loss: 0.1959, lr: 0.003352, batch_cost: 0.7873, reader_cost: 0.00016, ips: 1.2701 samples/sec | ETA 02:32:20 2021-05-09 21:50:59 [INFO] [TRAIN] epoch: 77, iter: 28400/40000, loss: 0.1833, lr: 0.003350, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2723 samples/sec | ETA 02:31:57 2021-05-09 21:51:07 [INFO] [TRAIN] epoch: 77, iter: 28410/40000, loss: 0.3913, lr: 0.003347, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2713 samples/sec | ETA 02:31:56 2021-05-09 21:51:15 [INFO] [TRAIN] epoch: 77, iter: 28420/40000, loss: 0.2517, lr: 0.003345, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2729 samples/sec | ETA 02:31:37 2021-05-09 21:51:23 [INFO] [TRAIN] epoch: 77, iter: 28430/40000, loss: 0.3350, lr: 0.003342, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2730 samples/sec | ETA 02:31:29 2021-05-09 21:51:31 [INFO] [TRAIN] epoch: 77, iter: 28440/40000, loss: 0.1921, lr: 0.003339, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2719 samples/sec | ETA 02:31:28 2021-05-09 21:51:39 [INFO] [TRAIN] epoch: 77, iter: 28450/40000, loss: 0.2228, lr: 0.003337, batch_cost: 0.7859, reader_cost: 0.00017, ips: 1.2724 samples/sec | ETA 02:31:17 2021-05-09 21:51:47 [INFO] [TRAIN] epoch: 77, iter: 28460/40000, loss: 0.1303, lr: 0.003334, batch_cost: 0.7852, reader_cost: 0.00017, ips: 1.2735 samples/sec | ETA 02:31:01 2021-05-09 21:51:54 [INFO] [TRAIN] epoch: 77, iter: 28470/40000, loss: 0.0713, lr: 0.003332, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2727 samples/sec | ETA 02:30:59 2021-05-09 21:52:02 [INFO] [TRAIN] epoch: 77, iter: 28480/40000, loss: 0.3191, lr: 0.003329, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2709 samples/sec | ETA 02:31:04 2021-05-09 21:52:10 [INFO] [TRAIN] epoch: 77, iter: 28490/40000, loss: 0.2441, lr: 0.003327, batch_cost: 0.7854, reader_cost: 0.00015, ips: 1.2732 samples/sec | ETA 02:30:40 2021-05-09 21:52:18 [INFO] [TRAIN] epoch: 77, iter: 28500/40000, loss: 0.1660, lr: 0.003324, batch_cost: 0.7832, reader_cost: 0.00015, ips: 1.2768 samples/sec | ETA 02:30:06 2021-05-09 21:52:26 [INFO] [TRAIN] epoch: 77, iter: 28510/40000, loss: 0.2605, lr: 0.003322, batch_cost: 0.7842, reader_cost: 0.00017, ips: 1.2752 samples/sec | ETA 02:30:10 2021-05-09 21:52:34 [INFO] [TRAIN] epoch: 77, iter: 28520/40000, loss: 0.2109, lr: 0.003319, batch_cost: 0.7842, reader_cost: 0.00016, ips: 1.2751 samples/sec | ETA 02:30:03 2021-05-09 21:52:42 [INFO] [TRAIN] epoch: 77, iter: 28530/40000, loss: 0.3044, lr: 0.003317, batch_cost: 0.7850, reader_cost: 0.00017, ips: 1.2740 samples/sec | ETA 02:30:03 2021-05-09 21:52:49 [INFO] [TRAIN] epoch: 77, iter: 28540/40000, loss: 0.2978, lr: 0.003314, batch_cost: 0.7846, reader_cost: 0.00016, ips: 1.2745 samples/sec | ETA 02:29:51 2021-05-09 21:52:57 [INFO] [TRAIN] epoch: 77, iter: 28550/40000, loss: 0.3381, lr: 0.003312, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2714 samples/sec | ETA 02:30:05 2021-05-09 21:53:05 [INFO] [TRAIN] epoch: 77, iter: 28560/40000, loss: 0.4167, lr: 0.003309, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2716 samples/sec | ETA 02:29:56 2021-05-09 21:53:13 [INFO] [TRAIN] epoch: 77, iter: 28570/40000, loss: 0.3466, lr: 0.003307, batch_cost: 0.7862, reader_cost: 0.00017, ips: 1.2719 samples/sec | ETA 02:29:46 2021-05-09 21:53:21 [INFO] [TRAIN] epoch: 77, iter: 28580/40000, loss: 0.4200, lr: 0.003304, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2729 samples/sec | ETA 02:29:31 2021-05-09 21:53:29 [INFO] [TRAIN] epoch: 77, iter: 28590/40000, loss: 0.3547, lr: 0.003302, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 02:29:25 2021-05-09 21:53:37 [INFO] [TRAIN] epoch: 77, iter: 28600/40000, loss: 0.1405, lr: 0.003299, batch_cost: 0.7869, reader_cost: 0.00015, ips: 1.2708 samples/sec | ETA 02:29:30 2021-05-09 21:53:44 [INFO] [TRAIN] epoch: 77, iter: 28610/40000, loss: 0.1987, lr: 0.003297, batch_cost: 0.7872, reader_cost: 0.00017, ips: 1.2703 samples/sec | ETA 02:29:26 2021-05-09 21:53:52 [INFO] [TRAIN] epoch: 77, iter: 28620/40000, loss: 0.2472, lr: 0.003294, batch_cost: 0.7863, reader_cost: 0.00018, ips: 1.2718 samples/sec | ETA 02:29:07 2021-05-09 21:54:00 [INFO] [TRAIN] epoch: 77, iter: 28630/40000, loss: 0.3224, lr: 0.003292, batch_cost: 0.7877, reader_cost: 0.00019, ips: 1.2695 samples/sec | ETA 02:29:16 2021-05-09 21:54:08 [INFO] [TRAIN] epoch: 77, iter: 28640/40000, loss: 0.2509, lr: 0.003289, batch_cost: 0.7844, reader_cost: 0.00011, ips: 1.2748 samples/sec | ETA 02:28:31 2021-05-09 21:54:19 [INFO] [TRAIN] epoch: 78, iter: 28650/40000, loss: 0.2516, lr: 0.003286, batch_cost: 1.0883, reader_cost: 0.28041, ips: 0.9188 samples/sec | ETA 03:25:52 2021-05-09 21:54:27 [INFO] [TRAIN] epoch: 78, iter: 28660/40000, loss: 0.2954, lr: 0.003284, batch_cost: 0.7999, reader_cost: 0.00035, ips: 1.2502 samples/sec | ETA 02:31:10 2021-05-09 21:54:35 [INFO] [TRAIN] epoch: 78, iter: 28670/40000, loss: 0.3474, lr: 0.003281, batch_cost: 0.7853, reader_cost: 0.00017, ips: 1.2734 samples/sec | ETA 02:28:17 2021-05-09 21:54:43 [INFO] [TRAIN] epoch: 78, iter: 28680/40000, loss: 0.2879, lr: 0.003279, batch_cost: 0.7876, reader_cost: 0.00016, ips: 1.2697 samples/sec | ETA 02:28:35 2021-05-09 21:54:51 [INFO] [TRAIN] epoch: 78, iter: 28690/40000, loss: 0.4628, lr: 0.003276, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 02:28:03 2021-05-09 21:54:58 [INFO] [TRAIN] epoch: 78, iter: 28700/40000, loss: 0.3010, lr: 0.003274, batch_cost: 0.7864, reader_cost: 0.00019, ips: 1.2716 samples/sec | ETA 02:28:06 2021-05-09 21:55:06 [INFO] [TRAIN] epoch: 78, iter: 28710/40000, loss: 0.2039, lr: 0.003271, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2706 samples/sec | ETA 02:28:05 2021-05-09 21:55:14 [INFO] [TRAIN] epoch: 78, iter: 28720/40000, loss: 0.1159, lr: 0.003269, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 02:27:53 2021-05-09 21:55:22 [INFO] [TRAIN] epoch: 78, iter: 28730/40000, loss: 0.3063, lr: 0.003266, batch_cost: 0.7853, reader_cost: 0.00015, ips: 1.2734 samples/sec | ETA 02:27:30 2021-05-09 21:55:30 [INFO] [TRAIN] epoch: 78, iter: 28740/40000, loss: 0.2643, lr: 0.003264, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2726 samples/sec | ETA 02:27:27 2021-05-09 21:55:38 [INFO] [TRAIN] epoch: 78, iter: 28750/40000, loss: 0.2273, lr: 0.003261, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2703 samples/sec | ETA 02:27:36 2021-05-09 21:55:46 [INFO] [TRAIN] epoch: 78, iter: 28760/40000, loss: 0.2357, lr: 0.003259, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2728 samples/sec | ETA 02:27:11 2021-05-09 21:55:53 [INFO] [TRAIN] epoch: 78, iter: 28770/40000, loss: 0.1319, lr: 0.003256, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2713 samples/sec | ETA 02:27:13 2021-05-09 21:56:01 [INFO] [TRAIN] epoch: 78, iter: 28780/40000, loss: 0.3688, lr: 0.003254, batch_cost: 0.7875, reader_cost: 0.00016, ips: 1.2698 samples/sec | ETA 02:27:15 2021-05-09 21:56:09 [INFO] [TRAIN] epoch: 78, iter: 28790/40000, loss: 0.2389, lr: 0.003251, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2729 samples/sec | ETA 02:26:46 2021-05-09 21:56:17 [INFO] [TRAIN] epoch: 78, iter: 28800/40000, loss: 0.2238, lr: 0.003249, batch_cost: 0.7849, reader_cost: 0.00015, ips: 1.2741 samples/sec | ETA 02:26:30 2021-05-09 21:56:25 [INFO] [TRAIN] epoch: 78, iter: 28810/40000, loss: 0.2711, lr: 0.003246, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 02:26:40 2021-05-09 21:56:33 [INFO] [TRAIN] epoch: 78, iter: 28820/40000, loss: 0.1271, lr: 0.003244, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2723 samples/sec | ETA 02:26:27 2021-05-09 21:56:41 [INFO] [TRAIN] epoch: 78, iter: 28830/40000, loss: 0.2993, lr: 0.003241, batch_cost: 0.7862, reader_cost: 0.00017, ips: 1.2720 samples/sec | ETA 02:26:21 2021-05-09 21:56:48 [INFO] [TRAIN] epoch: 78, iter: 28840/40000, loss: 0.1000, lr: 0.003238, batch_cost: 0.7875, reader_cost: 0.00017, ips: 1.2698 samples/sec | ETA 02:26:28 2021-05-09 21:56:56 [INFO] [TRAIN] epoch: 78, iter: 28850/40000, loss: 0.2455, lr: 0.003236, batch_cost: 0.7879, reader_cost: 0.00017, ips: 1.2692 samples/sec | ETA 02:26:25 2021-05-09 21:57:04 [INFO] [TRAIN] epoch: 78, iter: 28860/40000, loss: 0.2947, lr: 0.003233, batch_cost: 0.7860, reader_cost: 0.00017, ips: 1.2723 samples/sec | ETA 02:25:55 2021-05-09 21:57:12 [INFO] [TRAIN] epoch: 78, iter: 28870/40000, loss: 0.1660, lr: 0.003231, batch_cost: 0.7894, reader_cost: 0.00016, ips: 1.2669 samples/sec | ETA 02:26:25 2021-05-09 21:57:20 [INFO] [TRAIN] epoch: 78, iter: 28880/40000, loss: 0.2441, lr: 0.003228, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 02:25:47 2021-05-09 21:57:28 [INFO] [TRAIN] epoch: 78, iter: 28890/40000, loss: 0.3456, lr: 0.003226, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2717 samples/sec | ETA 02:25:36 2021-05-09 21:57:36 [INFO] [TRAIN] epoch: 78, iter: 28900/40000, loss: 0.2385, lr: 0.003223, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2714 samples/sec | ETA 02:25:30 2021-05-09 21:57:44 [INFO] [TRAIN] epoch: 78, iter: 28910/40000, loss: 0.1853, lr: 0.003221, batch_cost: 0.7860, reader_cost: 0.00017, ips: 1.2723 samples/sec | ETA 02:25:16 2021-05-09 21:57:51 [INFO] [TRAIN] epoch: 78, iter: 28920/40000, loss: 0.2938, lr: 0.003218, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2730 samples/sec | ETA 02:25:03 2021-05-09 21:57:59 [INFO] [TRAIN] epoch: 78, iter: 28930/40000, loss: 0.6006, lr: 0.003216, batch_cost: 0.7865, reader_cost: 0.00017, ips: 1.2714 samples/sec | ETA 02:25:07 2021-05-09 21:58:07 [INFO] [TRAIN] epoch: 78, iter: 28940/40000, loss: 0.3636, lr: 0.003213, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2725 samples/sec | ETA 02:24:51 2021-05-09 21:58:15 [INFO] [TRAIN] epoch: 78, iter: 28950/40000, loss: 0.4467, lr: 0.003211, batch_cost: 0.7868, reader_cost: 0.00017, ips: 1.2709 samples/sec | ETA 02:24:54 2021-05-09 21:58:23 [INFO] [TRAIN] epoch: 78, iter: 28960/40000, loss: 0.3633, lr: 0.003208, batch_cost: 0.7855, reader_cost: 0.00019, ips: 1.2731 samples/sec | ETA 02:24:31 2021-05-09 21:58:31 [INFO] [TRAIN] epoch: 78, iter: 28970/40000, loss: 0.1967, lr: 0.003206, batch_cost: 0.7866, reader_cost: 0.00014, ips: 1.2713 samples/sec | ETA 02:24:35 2021-05-09 21:58:39 [INFO] [TRAIN] epoch: 78, iter: 28980/40000, loss: 0.1614, lr: 0.003203, batch_cost: 0.7877, reader_cost: 0.00015, ips: 1.2695 samples/sec | ETA 02:24:40 2021-05-09 21:58:46 [INFO] [TRAIN] epoch: 78, iter: 28990/40000, loss: 0.1913, lr: 0.003200, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2727 samples/sec | ETA 02:24:10 2021-05-09 21:58:54 [INFO] [TRAIN] epoch: 78, iter: 29000/40000, loss: 0.2229, lr: 0.003198, batch_cost: 0.7851, reader_cost: 0.00017, ips: 1.2738 samples/sec | ETA 02:23:55 2021-05-09 21:58:54 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-09 22:02:25 [INFO] [EVAL] #Images: 500 mIoU: 0.7684 Acc: 0.9569 Kappa: 0.9440 2021-05-09 22:02:25 [INFO] [EVAL] Class IoU: [0.9799 0.8408 0.9202 0.6366 0.6288 0.5002 0.6264 0.7287 0.9157 0.668 0.9412 0.7791 0.5767 0.9425 0.8389 0.8942 0.8083 0.6378 0.7355] 2021-05-09 22:02:25 [INFO] [EVAL] Class Acc: [0.9918 0.9051 0.9544 0.823 0.8008 0.7533 0.8015 0.8979 0.9458 0.8474 0.9649 0.8579 0.79 0.967 0.926 0.9475 0.9182 0.8789 0.8344] 2021-05-09 22:03:15 [INFO] [EVAL] The model with the best validation mIoU (0.7684) was saved at iter 29000. 2021-05-09 22:03:23 [INFO] [TRAIN] epoch: 78, iter: 29010/40000, loss: 0.3084, lr: 0.003195, batch_cost: 0.7833, reader_cost: 0.00025, ips: 1.2767 samples/sec | ETA 02:23:28 2021-05-09 22:03:34 [INFO] [TRAIN] epoch: 79, iter: 29020/40000, loss: 0.3496, lr: 0.003193, batch_cost: 1.0976, reader_cost: 0.30185, ips: 0.9111 samples/sec | ETA 03:20:51 2021-05-09 22:03:42 [INFO] [TRAIN] epoch: 79, iter: 29030/40000, loss: 0.2516, lr: 0.003190, batch_cost: 0.8048, reader_cost: 0.00033, ips: 1.2425 samples/sec | ETA 02:27:09 2021-05-09 22:03:49 [INFO] [TRAIN] epoch: 79, iter: 29040/40000, loss: 0.5888, lr: 0.003188, batch_cost: 0.7848, reader_cost: 0.00017, ips: 1.2743 samples/sec | ETA 02:23:21 2021-05-09 22:03:57 [INFO] [TRAIN] epoch: 79, iter: 29050/40000, loss: 0.2618, lr: 0.003185, batch_cost: 0.7874, reader_cost: 0.00016, ips: 1.2701 samples/sec | ETA 02:23:41 2021-05-09 22:04:05 [INFO] [TRAIN] epoch: 79, iter: 29060/40000, loss: 0.3361, lr: 0.003183, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2720 samples/sec | ETA 02:23:20 2021-05-09 22:04:13 [INFO] [TRAIN] epoch: 79, iter: 29070/40000, loss: 0.4368, lr: 0.003180, batch_cost: 0.7871, reader_cost: 0.00015, ips: 1.2705 samples/sec | ETA 02:23:22 2021-05-09 22:04:21 [INFO] [TRAIN] epoch: 79, iter: 29080/40000, loss: 0.2155, lr: 0.003178, batch_cost: 0.7847, reader_cost: 0.00015, ips: 1.2744 samples/sec | ETA 02:22:48 2021-05-09 22:04:29 [INFO] [TRAIN] epoch: 79, iter: 29090/40000, loss: 0.1399, lr: 0.003175, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2726 samples/sec | ETA 02:22:52 2021-05-09 22:04:37 [INFO] [TRAIN] epoch: 79, iter: 29100/40000, loss: 0.3142, lr: 0.003173, batch_cost: 0.7862, reader_cost: 0.00018, ips: 1.2720 samples/sec | ETA 02:22:49 2021-05-09 22:04:45 [INFO] [TRAIN] epoch: 79, iter: 29110/40000, loss: 0.2581, lr: 0.003170, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2720 samples/sec | ETA 02:22:41 2021-05-09 22:04:52 [INFO] [TRAIN] epoch: 79, iter: 29120/40000, loss: 0.2375, lr: 0.003167, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2721 samples/sec | ETA 02:22:33 2021-05-09 22:05:00 [INFO] [TRAIN] epoch: 79, iter: 29130/40000, loss: 0.3046, lr: 0.003165, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2723 samples/sec | ETA 02:22:23 2021-05-09 22:05:08 [INFO] [TRAIN] epoch: 79, iter: 29140/40000, loss: 0.0981, lr: 0.003162, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 02:22:15 2021-05-09 22:05:16 [INFO] [TRAIN] epoch: 79, iter: 29150/40000, loss: 0.3223, lr: 0.003160, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2727 samples/sec | ETA 02:22:05 2021-05-09 22:05:24 [INFO] [TRAIN] epoch: 79, iter: 29160/40000, loss: 0.2328, lr: 0.003157, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2728 samples/sec | ETA 02:21:56 2021-05-09 22:05:32 [INFO] [TRAIN] epoch: 79, iter: 29170/40000, loss: 0.2661, lr: 0.003155, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2730 samples/sec | ETA 02:21:47 2021-05-09 22:05:40 [INFO] [TRAIN] epoch: 79, iter: 29180/40000, loss: 0.2557, lr: 0.003152, batch_cost: 0.7873, reader_cost: 0.00018, ips: 1.2702 samples/sec | ETA 02:21:58 2021-05-09 22:05:47 [INFO] [TRAIN] epoch: 79, iter: 29190/40000, loss: 0.2024, lr: 0.003150, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2726 samples/sec | ETA 02:21:34 2021-05-09 22:05:55 [INFO] [TRAIN] epoch: 79, iter: 29200/40000, loss: 0.3729, lr: 0.003147, batch_cost: 0.7866, reader_cost: 0.00017, ips: 1.2713 samples/sec | ETA 02:21:35 2021-05-09 22:06:03 [INFO] [TRAIN] epoch: 79, iter: 29210/40000, loss: 0.1234, lr: 0.003145, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2735 samples/sec | ETA 02:21:12 2021-05-09 22:06:11 [INFO] [TRAIN] epoch: 79, iter: 29220/40000, loss: 0.2315, lr: 0.003142, batch_cost: 0.7849, reader_cost: 0.00017, ips: 1.2740 samples/sec | ETA 02:21:01 2021-05-09 22:06:19 [INFO] [TRAIN] epoch: 79, iter: 29230/40000, loss: 0.3260, lr: 0.003140, batch_cost: 0.7845, reader_cost: 0.00016, ips: 1.2747 samples/sec | ETA 02:20:48 2021-05-09 22:06:27 [INFO] [TRAIN] epoch: 79, iter: 29240/40000, loss: 0.2142, lr: 0.003137, batch_cost: 0.7862, reader_cost: 0.00017, ips: 1.2720 samples/sec | ETA 02:20:59 2021-05-09 22:06:35 [INFO] [TRAIN] epoch: 79, iter: 29250/40000, loss: 0.2312, lr: 0.003134, batch_cost: 0.7844, reader_cost: 0.00017, ips: 1.2748 samples/sec | ETA 02:20:32 2021-05-09 22:06:42 [INFO] [TRAIN] epoch: 79, iter: 29260/40000, loss: 0.1686, lr: 0.003132, batch_cost: 0.7863, reader_cost: 0.00017, ips: 1.2718 samples/sec | ETA 02:20:44 2021-05-09 22:06:50 [INFO] [TRAIN] epoch: 79, iter: 29270/40000, loss: 0.2210, lr: 0.003129, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2725 samples/sec | ETA 02:20:32 2021-05-09 22:06:58 [INFO] [TRAIN] epoch: 79, iter: 29280/40000, loss: 0.1063, lr: 0.003127, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2730 samples/sec | ETA 02:20:20 2021-05-09 22:07:06 [INFO] [TRAIN] epoch: 79, iter: 29290/40000, loss: 0.3052, lr: 0.003124, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2705 samples/sec | ETA 02:20:29 2021-05-09 22:07:14 [INFO] [TRAIN] epoch: 79, iter: 29300/40000, loss: 0.5021, lr: 0.003122, batch_cost: 0.7853, reader_cost: 0.00017, ips: 1.2735 samples/sec | ETA 02:20:02 2021-05-09 22:07:22 [INFO] [TRAIN] epoch: 79, iter: 29310/40000, loss: 0.3163, lr: 0.003119, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2718 samples/sec | ETA 02:20:05 2021-05-09 22:07:30 [INFO] [TRAIN] epoch: 79, iter: 29320/40000, loss: 0.5573, lr: 0.003117, batch_cost: 0.7856, reader_cost: 0.00018, ips: 1.2729 samples/sec | ETA 02:19:50 2021-05-09 22:07:37 [INFO] [TRAIN] epoch: 79, iter: 29330/40000, loss: 0.3988, lr: 0.003114, batch_cost: 0.7855, reader_cost: 0.00017, ips: 1.2730 samples/sec | ETA 02:19:41 2021-05-09 22:07:45 [INFO] [TRAIN] epoch: 79, iter: 29340/40000, loss: 0.2721, lr: 0.003112, batch_cost: 0.7852, reader_cost: 0.00017, ips: 1.2735 samples/sec | ETA 02:19:30 2021-05-09 22:07:53 [INFO] [TRAIN] epoch: 79, iter: 29350/40000, loss: 0.0933, lr: 0.003109, batch_cost: 0.7849, reader_cost: 0.00016, ips: 1.2740 samples/sec | ETA 02:19:19 2021-05-09 22:08:01 [INFO] [TRAIN] epoch: 79, iter: 29360/40000, loss: 0.1951, lr: 0.003107, batch_cost: 0.7857, reader_cost: 0.00017, ips: 1.2728 samples/sec | ETA 02:19:19 2021-05-09 22:08:09 [INFO] [TRAIN] epoch: 79, iter: 29370/40000, loss: 0.3199, lr: 0.003104, batch_cost: 0.7873, reader_cost: 0.00015, ips: 1.2701 samples/sec | ETA 02:19:29 2021-05-09 22:08:17 [INFO] [TRAIN] epoch: 79, iter: 29380/40000, loss: 0.2907, lr: 0.003101, batch_cost: 0.7867, reader_cost: 0.00015, ips: 1.2711 samples/sec | ETA 02:19:14 2021-05-09 22:08:28 [INFO] [TRAIN] epoch: 80, iter: 29390/40000, loss: 0.3720, lr: 0.003099, batch_cost: 1.1169, reader_cost: 0.23904, ips: 0.8953 samples/sec | ETA 03:17:30 2021-05-09 22:08:36 [INFO] [TRAIN] epoch: 80, iter: 29400/40000, loss: 0.1428, lr: 0.003096, batch_cost: 0.8028, reader_cost: 0.00033, ips: 1.2457 samples/sec | ETA 02:21:49 2021-05-09 22:08:44 [INFO] [TRAIN] epoch: 80, iter: 29410/40000, loss: 0.5262, lr: 0.003094, batch_cost: 0.7873, reader_cost: 0.00016, ips: 1.2701 samples/sec | ETA 02:18:57 2021-05-09 22:08:52 [INFO] [TRAIN] epoch: 80, iter: 29420/40000, loss: 0.3170, lr: 0.003091, batch_cost: 0.7882, reader_cost: 0.00016, ips: 1.2687 samples/sec | ETA 02:18:59 2021-05-09 22:09:00 [INFO] [TRAIN] epoch: 80, iter: 29430/40000, loss: 0.4123, lr: 0.003089, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2736 samples/sec | ETA 02:18:19 2021-05-09 22:09:07 [INFO] [TRAIN] epoch: 80, iter: 29440/40000, loss: 0.4750, lr: 0.003086, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 02:18:25 2021-05-09 22:09:15 [INFO] [TRAIN] epoch: 80, iter: 29450/40000, loss: 0.3354, lr: 0.003084, batch_cost: 0.7899, reader_cost: 0.00017, ips: 1.2661 samples/sec | ETA 02:18:52 2021-05-09 22:09:23 [INFO] [TRAIN] epoch: 80, iter: 29460/40000, loss: 0.1200, lr: 0.003081, batch_cost: 0.7877, reader_cost: 0.00016, ips: 1.2695 samples/sec | ETA 02:18:22 2021-05-09 22:09:31 [INFO] [TRAIN] epoch: 80, iter: 29470/40000, loss: 0.3686, lr: 0.003079, batch_cost: 0.7843, reader_cost: 0.00016, ips: 1.2751 samples/sec | ETA 02:17:38 2021-05-09 22:09:39 [INFO] [TRAIN] epoch: 80, iter: 29480/40000, loss: 0.2573, lr: 0.003076, batch_cost: 0.7851, reader_cost: 0.00016, ips: 1.2738 samples/sec | ETA 02:17:38 2021-05-09 22:09:47 [INFO] [TRAIN] epoch: 80, iter: 29490/40000, loss: 0.2030, lr: 0.003073, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2705 samples/sec | ETA 02:17:52 2021-05-09 22:09:55 [INFO] [TRAIN] epoch: 80, iter: 29500/40000, loss: 0.2938, lr: 0.003071, batch_cost: 0.7876, reader_cost: 0.00017, ips: 1.2697 samples/sec | ETA 02:17:49 2021-05-09 22:10:02 [INFO] [TRAIN] epoch: 80, iter: 29510/40000, loss: 0.1213, lr: 0.003068, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2703 samples/sec | ETA 02:17:37 2021-05-09 22:10:10 [INFO] [TRAIN] epoch: 80, iter: 29520/40000, loss: 0.3182, lr: 0.003066, batch_cost: 0.7863, reader_cost: 0.00017, ips: 1.2717 samples/sec | ETA 02:17:20 2021-05-09 22:10:18 [INFO] [TRAIN] epoch: 80, iter: 29530/40000, loss: 0.3787, lr: 0.003063, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2728 samples/sec | ETA 02:17:06 2021-05-09 22:10:26 [INFO] [TRAIN] epoch: 80, iter: 29540/40000, loss: 0.2667, lr: 0.003061, batch_cost: 0.7855, reader_cost: 0.00017, ips: 1.2730 samples/sec | ETA 02:16:56 2021-05-09 22:10:34 [INFO] [TRAIN] epoch: 80, iter: 29550/40000, loss: 0.2953, lr: 0.003058, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2735 samples/sec | ETA 02:16:45 2021-05-09 22:10:42 [INFO] [TRAIN] epoch: 80, iter: 29560/40000, loss: 0.2373, lr: 0.003056, batch_cost: 0.7868, reader_cost: 0.00017, ips: 1.2709 samples/sec | ETA 02:16:54 2021-05-09 22:10:50 [INFO] [TRAIN] epoch: 80, iter: 29570/40000, loss: 0.2123, lr: 0.003053, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2711 samples/sec | ETA 02:16:45 2021-05-09 22:10:57 [INFO] [TRAIN] epoch: 80, iter: 29580/40000, loss: 0.1272, lr: 0.003051, batch_cost: 0.7861, reader_cost: 0.00017, ips: 1.2720 samples/sec | ETA 02:16:31 2021-05-09 22:11:05 [INFO] [TRAIN] epoch: 80, iter: 29590/40000, loss: 0.1664, lr: 0.003048, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 02:16:16 2021-05-09 22:11:13 [INFO] [TRAIN] epoch: 80, iter: 29600/40000, loss: 0.2753, lr: 0.003045, batch_cost: 0.7877, reader_cost: 0.00015, ips: 1.2696 samples/sec | ETA 02:16:31 2021-05-09 22:11:21 [INFO] [TRAIN] epoch: 80, iter: 29610/40000, loss: 0.1533, lr: 0.003043, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2733 samples/sec | ETA 02:15:59 2021-05-09 22:11:29 [INFO] [TRAIN] epoch: 80, iter: 29620/40000, loss: 0.2584, lr: 0.003040, batch_cost: 0.7853, reader_cost: 0.00017, ips: 1.2734 samples/sec | ETA 02:15:51 2021-05-09 22:11:37 [INFO] [TRAIN] epoch: 80, iter: 29630/40000, loss: 0.2374, lr: 0.003038, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2710 samples/sec | ETA 02:15:59 2021-05-09 22:11:45 [INFO] [TRAIN] epoch: 80, iter: 29640/40000, loss: 0.1852, lr: 0.003035, batch_cost: 0.7855, reader_cost: 0.00014, ips: 1.2730 samples/sec | ETA 02:15:37 2021-05-09 22:11:53 [INFO] [TRAIN] epoch: 80, iter: 29650/40000, loss: 0.1341, lr: 0.003033, batch_cost: 0.7853, reader_cost: 0.00014, ips: 1.2734 samples/sec | ETA 02:15:27 2021-05-09 22:12:00 [INFO] [TRAIN] epoch: 80, iter: 29660/40000, loss: 0.3238, lr: 0.003030, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2728 samples/sec | ETA 02:15:23 2021-05-09 22:12:08 [INFO] [TRAIN] epoch: 80, iter: 29670/40000, loss: 0.3815, lr: 0.003028, batch_cost: 0.7848, reader_cost: 0.00015, ips: 1.2742 samples/sec | ETA 02:15:07 2021-05-09 22:12:16 [INFO] [TRAIN] epoch: 80, iter: 29680/40000, loss: 0.2524, lr: 0.003025, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 02:15:08 2021-05-09 22:12:24 [INFO] [TRAIN] epoch: 80, iter: 29690/40000, loss: 0.5085, lr: 0.003022, batch_cost: 0.7847, reader_cost: 0.00014, ips: 1.2744 samples/sec | ETA 02:14:50 2021-05-09 22:12:32 [INFO] [TRAIN] epoch: 80, iter: 29700/40000, loss: 0.4578, lr: 0.003020, batch_cost: 0.7854, reader_cost: 0.00015, ips: 1.2732 samples/sec | ETA 02:14:49 2021-05-09 22:12:40 [INFO] [TRAIN] epoch: 80, iter: 29710/40000, loss: 0.2619, lr: 0.003017, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2710 samples/sec | ETA 02:14:56 2021-05-09 22:12:48 [INFO] [TRAIN] epoch: 80, iter: 29720/40000, loss: 0.1028, lr: 0.003015, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2724 samples/sec | ETA 02:14:39 2021-05-09 22:12:55 [INFO] [TRAIN] epoch: 80, iter: 29730/40000, loss: 0.3368, lr: 0.003012, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2704 samples/sec | ETA 02:14:43 2021-05-09 22:13:03 [INFO] [TRAIN] epoch: 80, iter: 29740/40000, loss: 0.2102, lr: 0.003010, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2726 samples/sec | ETA 02:14:22 2021-05-09 22:13:11 [INFO] [TRAIN] epoch: 80, iter: 29750/40000, loss: 0.3009, lr: 0.003007, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2731 samples/sec | ETA 02:14:10 2021-05-09 22:13:19 [INFO] [TRAIN] epoch: 80, iter: 29760/40000, loss: 0.3972, lr: 0.003005, batch_cost: 0.7845, reader_cost: 0.00010, ips: 1.2747 samples/sec | ETA 02:13:53 2021-05-09 22:13:30 [INFO] [TRAIN] epoch: 81, iter: 29770/40000, loss: 0.3735, lr: 0.003002, batch_cost: 1.0979, reader_cost: 0.28307, ips: 0.9108 samples/sec | ETA 03:07:11 2021-05-09 22:13:38 [INFO] [TRAIN] epoch: 81, iter: 29780/40000, loss: 0.5151, lr: 0.003000, batch_cost: 0.7891, reader_cost: 0.00034, ips: 1.2673 samples/sec | ETA 02:14:24 2021-05-09 22:13:46 [INFO] [TRAIN] epoch: 81, iter: 29790/40000, loss: 0.2290, lr: 0.002997, batch_cost: 0.7859, reader_cost: 0.00017, ips: 1.2725 samples/sec | ETA 02:13:43 2021-05-09 22:13:54 [INFO] [TRAIN] epoch: 81, iter: 29800/40000, loss: 0.5040, lr: 0.002994, batch_cost: 0.7845, reader_cost: 0.00015, ips: 1.2747 samples/sec | ETA 02:13:21 2021-05-09 22:14:01 [INFO] [TRAIN] epoch: 81, iter: 29810/40000, loss: 0.4827, lr: 0.002992, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2710 samples/sec | ETA 02:13:37 2021-05-09 22:14:09 [INFO] [TRAIN] epoch: 81, iter: 29820/40000, loss: 0.2348, lr: 0.002989, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2732 samples/sec | ETA 02:13:15 2021-05-09 22:14:17 [INFO] [TRAIN] epoch: 81, iter: 29830/40000, loss: 0.1823, lr: 0.002987, batch_cost: 0.7868, reader_cost: 0.00017, ips: 1.2710 samples/sec | ETA 02:13:21 2021-05-09 22:14:25 [INFO] [TRAIN] epoch: 81, iter: 29840/40000, loss: 0.2434, lr: 0.002984, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2728 samples/sec | ETA 02:13:02 2021-05-09 22:14:33 [INFO] [TRAIN] epoch: 81, iter: 29850/40000, loss: 0.3057, lr: 0.002982, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2730 samples/sec | ETA 02:12:53 2021-05-09 22:14:41 [INFO] [TRAIN] epoch: 81, iter: 29860/40000, loss: 0.1934, lr: 0.002979, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2731 samples/sec | ETA 02:12:44 2021-05-09 22:14:49 [INFO] [TRAIN] epoch: 81, iter: 29870/40000, loss: 0.2906, lr: 0.002977, batch_cost: 0.7858, reader_cost: 0.00015, ips: 1.2726 samples/sec | ETA 02:12:39 2021-05-09 22:14:56 [INFO] [TRAIN] epoch: 81, iter: 29880/40000, loss: 0.1374, lr: 0.002974, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 02:12:33 2021-05-09 22:15:04 [INFO] [TRAIN] epoch: 81, iter: 29890/40000, loss: 0.2169, lr: 0.002971, batch_cost: 0.7855, reader_cost: 0.00017, ips: 1.2732 samples/sec | ETA 02:12:20 2021-05-09 22:15:12 [INFO] [TRAIN] epoch: 81, iter: 29900/40000, loss: 0.3568, lr: 0.002969, batch_cost: 0.7846, reader_cost: 0.00016, ips: 1.2746 samples/sec | ETA 02:12:04 2021-05-09 22:15:20 [INFO] [TRAIN] epoch: 81, iter: 29910/40000, loss: 0.2987, lr: 0.002966, batch_cost: 0.7844, reader_cost: 0.00015, ips: 1.2748 samples/sec | ETA 02:11:54 2021-05-09 22:15:28 [INFO] [TRAIN] epoch: 81, iter: 29920/40000, loss: 0.2845, lr: 0.002964, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2731 samples/sec | ETA 02:11:57 2021-05-09 22:15:36 [INFO] [TRAIN] epoch: 81, iter: 29930/40000, loss: 0.1662, lr: 0.002961, batch_cost: 0.7854, reader_cost: 0.00018, ips: 1.2732 samples/sec | ETA 02:11:48 2021-05-09 22:15:44 [INFO] [TRAIN] epoch: 81, iter: 29940/40000, loss: 0.2117, lr: 0.002959, batch_cost: 0.7849, reader_cost: 0.00018, ips: 1.2740 samples/sec | ETA 02:11:36 2021-05-09 22:15:51 [INFO] [TRAIN] epoch: 81, iter: 29950/40000, loss: 0.1560, lr: 0.002956, batch_cost: 0.7849, reader_cost: 0.00016, ips: 1.2741 samples/sec | ETA 02:11:28 2021-05-09 22:15:59 [INFO] [TRAIN] epoch: 81, iter: 29960/40000, loss: 0.1052, lr: 0.002954, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2728 samples/sec | ETA 02:11:28 2021-05-09 22:16:07 [INFO] [TRAIN] epoch: 81, iter: 29970/40000, loss: 0.3797, lr: 0.002951, batch_cost: 0.7851, reader_cost: 0.00017, ips: 1.2738 samples/sec | ETA 02:11:14 2021-05-09 22:16:15 [INFO] [TRAIN] epoch: 81, iter: 29980/40000, loss: 0.2617, lr: 0.002948, batch_cost: 0.7866, reader_cost: 0.00017, ips: 1.2712 samples/sec | ETA 02:11:22 2021-05-09 22:16:23 [INFO] [TRAIN] epoch: 81, iter: 29990/40000, loss: 0.2048, lr: 0.002946, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2729 samples/sec | ETA 02:11:04 2021-05-09 22:16:31 [INFO] [TRAIN] epoch: 81, iter: 30000/40000, loss: 0.2738, lr: 0.002943, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2728 samples/sec | ETA 02:10:56 2021-05-09 22:16:31 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-09 22:20:02 [INFO] [EVAL] #Images: 500 mIoU: 0.7695 Acc: 0.9571 Kappa: 0.9444 2021-05-09 22:20:02 [INFO] [EVAL] Class IoU: [0.9804 0.8451 0.9204 0.6492 0.6213 0.5031 0.6299 0.733 0.9157 0.6644 0.9419 0.7755 0.5649 0.9413 0.844 0.8932 0.8132 0.6502 0.734 ] 2021-05-09 22:20:02 [INFO] [EVAL] Class Acc: [0.9917 0.9148 0.9538 0.8262 0.8136 0.7516 0.8192 0.8857 0.9459 0.836 0.9704 0.8475 0.7904 0.9635 0.9222 0.9414 0.9072 0.8513 0.8486] 2021-05-09 22:20:50 [INFO] [EVAL] The model with the best validation mIoU (0.7695) was saved at iter 30000. 2021-05-09 22:20:58 [INFO] [TRAIN] epoch: 81, iter: 30010/40000, loss: 0.1718, lr: 0.002941, batch_cost: 0.7796, reader_cost: 0.00066, ips: 1.2828 samples/sec | ETA 02:09:47 2021-05-09 22:21:06 [INFO] [TRAIN] epoch: 81, iter: 30020/40000, loss: 0.2224, lr: 0.002938, batch_cost: 0.7840, reader_cost: 0.00035, ips: 1.2755 samples/sec | ETA 02:10:24 2021-05-09 22:21:14 [INFO] [TRAIN] epoch: 81, iter: 30030/40000, loss: 0.2795, lr: 0.002936, batch_cost: 0.7848, reader_cost: 0.00016, ips: 1.2741 samples/sec | ETA 02:10:24 2021-05-09 22:21:22 [INFO] [TRAIN] epoch: 81, iter: 30040/40000, loss: 0.3717, lr: 0.002933, batch_cost: 0.7847, reader_cost: 0.00016, ips: 1.2743 samples/sec | ETA 02:10:15 2021-05-09 22:21:30 [INFO] [TRAIN] epoch: 81, iter: 30050/40000, loss: 0.3994, lr: 0.002930, batch_cost: 0.7864, reader_cost: 0.00017, ips: 1.2716 samples/sec | ETA 02:10:24 2021-05-09 22:21:38 [INFO] [TRAIN] epoch: 81, iter: 30060/40000, loss: 0.3093, lr: 0.002928, batch_cost: 0.7866, reader_cost: 0.00018, ips: 1.2713 samples/sec | ETA 02:10:18 2021-05-09 22:21:45 [INFO] [TRAIN] epoch: 81, iter: 30070/40000, loss: 0.4544, lr: 0.002925, batch_cost: 0.7851, reader_cost: 0.00016, ips: 1.2738 samples/sec | ETA 02:09:55 2021-05-09 22:21:53 [INFO] [TRAIN] epoch: 81, iter: 30080/40000, loss: 0.5962, lr: 0.002923, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2720 samples/sec | ETA 02:09:58 2021-05-09 22:22:01 [INFO] [TRAIN] epoch: 81, iter: 30090/40000, loss: 0.0949, lr: 0.002920, batch_cost: 0.7875, reader_cost: 0.00016, ips: 1.2699 samples/sec | ETA 02:10:03 2021-05-09 22:22:09 [INFO] [TRAIN] epoch: 81, iter: 30100/40000, loss: 0.2289, lr: 0.002918, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2726 samples/sec | ETA 02:09:39 2021-05-09 22:22:17 [INFO] [TRAIN] epoch: 81, iter: 30110/40000, loss: 0.2332, lr: 0.002915, batch_cost: 0.7877, reader_cost: 0.00017, ips: 1.2696 samples/sec | ETA 02:09:50 2021-05-09 22:22:25 [INFO] [TRAIN] epoch: 81, iter: 30120/40000, loss: 0.2604, lr: 0.002913, batch_cost: 0.7878, reader_cost: 0.00016, ips: 1.2694 samples/sec | ETA 02:09:43 2021-05-09 22:22:33 [INFO] [TRAIN] epoch: 81, iter: 30130/40000, loss: 0.3145, lr: 0.002910, batch_cost: 0.7842, reader_cost: 0.00029, ips: 1.2751 samples/sec | ETA 02:09:00 2021-05-09 22:22:44 [INFO] [TRAIN] epoch: 82, iter: 30140/40000, loss: 0.3025, lr: 0.002907, batch_cost: 1.1058, reader_cost: 0.25774, ips: 0.9043 samples/sec | ETA 03:01:43 2021-05-09 22:22:52 [INFO] [TRAIN] epoch: 82, iter: 30150/40000, loss: 0.3666, lr: 0.002905, batch_cost: 0.7935, reader_cost: 0.00030, ips: 1.2603 samples/sec | ETA 02:10:15 2021-05-09 22:22:59 [INFO] [TRAIN] epoch: 82, iter: 30160/40000, loss: 0.3102, lr: 0.002902, batch_cost: 0.7837, reader_cost: 0.00016, ips: 1.2760 samples/sec | ETA 02:08:31 2021-05-09 22:23:07 [INFO] [TRAIN] epoch: 82, iter: 30170/40000, loss: 0.3600, lr: 0.002900, batch_cost: 0.7847, reader_cost: 0.00015, ips: 1.2744 samples/sec | ETA 02:08:33 2021-05-09 22:23:15 [INFO] [TRAIN] epoch: 82, iter: 30180/40000, loss: 0.4030, lr: 0.002897, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2720 samples/sec | ETA 02:08:39 2021-05-09 22:23:23 [INFO] [TRAIN] epoch: 82, iter: 30190/40000, loss: 0.3116, lr: 0.002895, batch_cost: 0.7880, reader_cost: 0.00015, ips: 1.2690 samples/sec | ETA 02:08:50 2021-05-09 22:23:31 [INFO] [TRAIN] epoch: 82, iter: 30200/40000, loss: 0.2534, lr: 0.002892, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 02:08:28 2021-05-09 22:23:39 [INFO] [TRAIN] epoch: 82, iter: 30210/40000, loss: 0.1906, lr: 0.002889, batch_cost: 0.7861, reader_cost: 0.00018, ips: 1.2722 samples/sec | ETA 02:08:15 2021-05-09 22:23:47 [INFO] [TRAIN] epoch: 82, iter: 30220/40000, loss: 0.4231, lr: 0.002887, batch_cost: 0.7884, reader_cost: 0.00016, ips: 1.2684 samples/sec | ETA 02:08:30 2021-05-09 22:23:54 [INFO] [TRAIN] epoch: 82, iter: 30230/40000, loss: 0.2118, lr: 0.002884, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2724 samples/sec | ETA 02:07:58 2021-05-09 22:24:02 [INFO] [TRAIN] epoch: 82, iter: 30240/40000, loss: 0.2922, lr: 0.002882, batch_cost: 0.7850, reader_cost: 0.00015, ips: 1.2739 samples/sec | ETA 02:07:41 2021-05-09 22:24:10 [INFO] [TRAIN] epoch: 82, iter: 30250/40000, loss: 0.1482, lr: 0.002879, batch_cost: 0.7885, reader_cost: 0.00016, ips: 1.2682 samples/sec | ETA 02:08:07 2021-05-09 22:24:18 [INFO] [TRAIN] epoch: 82, iter: 30260/40000, loss: 0.2548, lr: 0.002877, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2719 samples/sec | ETA 02:07:38 2021-05-09 22:24:26 [INFO] [TRAIN] epoch: 82, iter: 30270/40000, loss: 0.2831, lr: 0.002874, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2710 samples/sec | ETA 02:07:35 2021-05-09 22:24:34 [INFO] [TRAIN] epoch: 82, iter: 30280/40000, loss: 0.1960, lr: 0.002872, batch_cost: 0.7850, reader_cost: 0.00015, ips: 1.2739 samples/sec | ETA 02:07:10 2021-05-09 22:24:42 [INFO] [TRAIN] epoch: 82, iter: 30290/40000, loss: 0.2394, lr: 0.002869, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2713 samples/sec | ETA 02:07:17 2021-05-09 22:24:50 [INFO] [TRAIN] epoch: 82, iter: 30300/40000, loss: 0.2257, lr: 0.002866, batch_cost: 0.7853, reader_cost: 0.00014, ips: 1.2735 samples/sec | ETA 02:06:57 2021-05-09 22:24:57 [INFO] [TRAIN] epoch: 82, iter: 30310/40000, loss: 0.2139, lr: 0.002864, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2703 samples/sec | ETA 02:07:08 2021-05-09 22:25:05 [INFO] [TRAIN] epoch: 82, iter: 30320/40000, loss: 0.1754, lr: 0.002861, batch_cost: 0.7886, reader_cost: 0.00015, ips: 1.2680 samples/sec | ETA 02:07:14 2021-05-09 22:25:13 [INFO] [TRAIN] epoch: 82, iter: 30330/40000, loss: 0.1382, lr: 0.002859, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2735 samples/sec | ETA 02:06:33 2021-05-09 22:25:21 [INFO] [TRAIN] epoch: 82, iter: 30340/40000, loss: 0.3072, lr: 0.002856, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2720 samples/sec | ETA 02:06:34 2021-05-09 22:25:29 [INFO] [TRAIN] epoch: 82, iter: 30350/40000, loss: 0.3418, lr: 0.002854, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2720 samples/sec | ETA 02:06:26 2021-05-09 22:25:37 [INFO] [TRAIN] epoch: 82, iter: 30360/40000, loss: 0.2038, lr: 0.002851, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2729 samples/sec | ETA 02:06:13 2021-05-09 22:25:45 [INFO] [TRAIN] epoch: 82, iter: 30370/40000, loss: 0.2576, lr: 0.002848, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2705 samples/sec | ETA 02:06:19 2021-05-09 22:25:52 [INFO] [TRAIN] epoch: 82, iter: 30380/40000, loss: 0.1429, lr: 0.002846, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2720 samples/sec | ETA 02:06:02 2021-05-09 22:26:00 [INFO] [TRAIN] epoch: 82, iter: 30390/40000, loss: 0.2508, lr: 0.002843, batch_cost: 0.7848, reader_cost: 0.00016, ips: 1.2742 samples/sec | ETA 02:05:41 2021-05-09 22:26:08 [INFO] [TRAIN] epoch: 82, iter: 30400/40000, loss: 0.2135, lr: 0.002841, batch_cost: 0.7877, reader_cost: 0.00015, ips: 1.2696 samples/sec | ETA 02:06:01 2021-05-09 22:26:16 [INFO] [TRAIN] epoch: 82, iter: 30410/40000, loss: 0.3631, lr: 0.002838, batch_cost: 0.7851, reader_cost: 0.00016, ips: 1.2737 samples/sec | ETA 02:05:29 2021-05-09 22:26:24 [INFO] [TRAIN] epoch: 82, iter: 30420/40000, loss: 0.3331, lr: 0.002836, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2730 samples/sec | ETA 02:05:25 2021-05-09 22:26:32 [INFO] [TRAIN] epoch: 82, iter: 30430/40000, loss: 0.3897, lr: 0.002833, batch_cost: 0.7850, reader_cost: 0.00016, ips: 1.2738 samples/sec | ETA 02:05:12 2021-05-09 22:26:40 [INFO] [TRAIN] epoch: 82, iter: 30440/40000, loss: 0.3791, lr: 0.002830, batch_cost: 0.7862, reader_cost: 0.00017, ips: 1.2719 samples/sec | ETA 02:05:16 2021-05-09 22:26:47 [INFO] [TRAIN] epoch: 82, iter: 30450/40000, loss: 0.2816, lr: 0.002828, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2719 samples/sec | ETA 02:05:08 2021-05-09 22:26:55 [INFO] [TRAIN] epoch: 82, iter: 30460/40000, loss: 0.1419, lr: 0.002825, batch_cost: 0.7884, reader_cost: 0.00016, ips: 1.2685 samples/sec | ETA 02:05:20 2021-05-09 22:27:03 [INFO] [TRAIN] epoch: 82, iter: 30470/40000, loss: 0.1941, lr: 0.002823, batch_cost: 0.7874, reader_cost: 0.00017, ips: 1.2700 samples/sec | ETA 02:05:03 2021-05-09 22:27:11 [INFO] [TRAIN] epoch: 82, iter: 30480/40000, loss: 0.2100, lr: 0.002820, batch_cost: 0.7889, reader_cost: 0.00016, ips: 1.2676 samples/sec | ETA 02:05:10 2021-05-09 22:27:19 [INFO] [TRAIN] epoch: 82, iter: 30490/40000, loss: 0.3029, lr: 0.002818, batch_cost: 0.7875, reader_cost: 0.00015, ips: 1.2699 samples/sec | ETA 02:04:48 2021-05-09 22:27:27 [INFO] [TRAIN] epoch: 82, iter: 30500/40000, loss: 0.3002, lr: 0.002815, batch_cost: 0.7879, reader_cost: 0.00012, ips: 1.2691 samples/sec | ETA 02:04:45 2021-05-09 22:27:38 [INFO] [TRAIN] epoch: 83, iter: 30510/40000, loss: 0.2805, lr: 0.002812, batch_cost: 1.0863, reader_cost: 0.27368, ips: 0.9206 samples/sec | ETA 02:51:48 2021-05-09 22:27:46 [INFO] [TRAIN] epoch: 83, iter: 30520/40000, loss: 0.3049, lr: 0.002810, batch_cost: 0.7912, reader_cost: 0.00034, ips: 1.2638 samples/sec | ETA 02:05:01 2021-05-09 22:27:54 [INFO] [TRAIN] epoch: 83, iter: 30530/40000, loss: 0.4130, lr: 0.002807, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 02:04:07 2021-05-09 22:28:01 [INFO] [TRAIN] epoch: 83, iter: 30540/40000, loss: 0.2989, lr: 0.002805, batch_cost: 0.7874, reader_cost: 0.00016, ips: 1.2700 samples/sec | ETA 02:04:08 2021-05-09 22:28:09 [INFO] [TRAIN] epoch: 83, iter: 30550/40000, loss: 0.4257, lr: 0.002802, batch_cost: 0.7872, reader_cost: 0.00015, ips: 1.2703 samples/sec | ETA 02:03:58 2021-05-09 22:28:17 [INFO] [TRAIN] epoch: 83, iter: 30560/40000, loss: 0.3023, lr: 0.002800, batch_cost: 0.7875, reader_cost: 0.00015, ips: 1.2699 samples/sec | ETA 02:03:53 2021-05-09 22:28:25 [INFO] [TRAIN] epoch: 83, iter: 30570/40000, loss: 0.3971, lr: 0.002797, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2714 samples/sec | ETA 02:03:37 2021-05-09 22:28:33 [INFO] [TRAIN] epoch: 83, iter: 30580/40000, loss: 0.1356, lr: 0.002794, batch_cost: 0.7851, reader_cost: 0.00015, ips: 1.2737 samples/sec | ETA 02:03:15 2021-05-09 22:28:41 [INFO] [TRAIN] epoch: 83, iter: 30590/40000, loss: 0.2998, lr: 0.002792, batch_cost: 0.7846, reader_cost: 0.00016, ips: 1.2745 samples/sec | ETA 02:03:03 2021-05-09 22:28:49 [INFO] [TRAIN] epoch: 83, iter: 30600/40000, loss: 0.3620, lr: 0.002789, batch_cost: 0.7875, reader_cost: 0.00015, ips: 1.2698 samples/sec | ETA 02:03:22 2021-05-09 22:28:56 [INFO] [TRAIN] epoch: 83, iter: 30610/40000, loss: 0.3018, lr: 0.002787, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2713 samples/sec | ETA 02:03:05 2021-05-09 22:29:04 [INFO] [TRAIN] epoch: 83, iter: 30620/40000, loss: 0.1993, lr: 0.002784, batch_cost: 0.7875, reader_cost: 0.00014, ips: 1.2699 samples/sec | ETA 02:03:06 2021-05-09 22:29:12 [INFO] [TRAIN] epoch: 83, iter: 30630/40000, loss: 0.1471, lr: 0.002782, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2728 samples/sec | ETA 02:02:41 2021-05-09 22:29:20 [INFO] [TRAIN] epoch: 83, iter: 30640/40000, loss: 0.3008, lr: 0.002779, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2736 samples/sec | ETA 02:02:29 2021-05-09 22:29:28 [INFO] [TRAIN] epoch: 83, iter: 30650/40000, loss: 0.2347, lr: 0.002776, batch_cost: 0.7859, reader_cost: 0.00014, ips: 1.2725 samples/sec | ETA 02:02:27 2021-05-09 22:29:36 [INFO] [TRAIN] epoch: 83, iter: 30660/40000, loss: 0.3662, lr: 0.002774, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2730 samples/sec | ETA 02:02:17 2021-05-09 22:29:44 [INFO] [TRAIN] epoch: 83, iter: 30670/40000, loss: 0.2851, lr: 0.002771, batch_cost: 0.7860, reader_cost: 0.00014, ips: 1.2722 samples/sec | ETA 02:02:13 2021-05-09 22:29:51 [INFO] [TRAIN] epoch: 83, iter: 30680/40000, loss: 0.3130, lr: 0.002769, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2730 samples/sec | ETA 02:02:01 2021-05-09 22:29:59 [INFO] [TRAIN] epoch: 83, iter: 30690/40000, loss: 0.2932, lr: 0.002766, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 02:02:00 2021-05-09 22:30:07 [INFO] [TRAIN] epoch: 83, iter: 30700/40000, loss: 0.0554, lr: 0.002764, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2730 samples/sec | ETA 02:01:45 2021-05-09 22:30:15 [INFO] [TRAIN] epoch: 83, iter: 30710/40000, loss: 0.2802, lr: 0.002761, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2715 samples/sec | ETA 02:01:46 2021-05-09 22:30:23 [INFO] [TRAIN] epoch: 83, iter: 30720/40000, loss: 0.2945, lr: 0.002758, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 02:01:40 2021-05-09 22:30:31 [INFO] [TRAIN] epoch: 83, iter: 30730/40000, loss: 0.1667, lr: 0.002756, batch_cost: 0.7869, reader_cost: 0.00017, ips: 1.2709 samples/sec | ETA 02:01:34 2021-05-09 22:30:39 [INFO] [TRAIN] epoch: 83, iter: 30740/40000, loss: 0.2670, lr: 0.002753, batch_cost: 0.7858, reader_cost: 0.00017, ips: 1.2726 samples/sec | ETA 02:01:16 2021-05-09 22:30:46 [INFO] [TRAIN] epoch: 83, iter: 30750/40000, loss: 0.1641, lr: 0.002751, batch_cost: 0.7855, reader_cost: 0.00017, ips: 1.2730 samples/sec | ETA 02:01:06 2021-05-09 22:30:54 [INFO] [TRAIN] epoch: 83, iter: 30760/40000, loss: 0.2478, lr: 0.002748, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2719 samples/sec | ETA 02:01:04 2021-05-09 22:31:02 [INFO] [TRAIN] epoch: 83, iter: 30770/40000, loss: 0.1892, lr: 0.002745, batch_cost: 0.7867, reader_cost: 0.00017, ips: 1.2712 samples/sec | ETA 02:01:01 2021-05-09 22:31:10 [INFO] [TRAIN] epoch: 83, iter: 30780/40000, loss: 0.2553, lr: 0.002743, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2714 samples/sec | ETA 02:00:52 2021-05-09 22:31:18 [INFO] [TRAIN] epoch: 83, iter: 30790/40000, loss: 0.4174, lr: 0.002740, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2732 samples/sec | ETA 02:00:33 2021-05-09 22:31:26 [INFO] [TRAIN] epoch: 83, iter: 30800/40000, loss: 0.3686, lr: 0.002738, batch_cost: 0.7879, reader_cost: 0.00017, ips: 1.2693 samples/sec | ETA 02:00:48 2021-05-09 22:31:34 [INFO] [TRAIN] epoch: 83, iter: 30810/40000, loss: 0.4828, lr: 0.002735, batch_cost: 0.7850, reader_cost: 0.00016, ips: 1.2740 samples/sec | ETA 02:00:13 2021-05-09 22:31:42 [INFO] [TRAIN] epoch: 83, iter: 30820/40000, loss: 0.4216, lr: 0.002733, batch_cost: 0.7873, reader_cost: 0.00016, ips: 1.2701 samples/sec | ETA 02:00:27 2021-05-09 22:31:49 [INFO] [TRAIN] epoch: 83, iter: 30830/40000, loss: 0.1265, lr: 0.002730, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2735 samples/sec | ETA 02:00:00 2021-05-09 22:31:57 [INFO] [TRAIN] epoch: 83, iter: 30840/40000, loss: 0.1694, lr: 0.002727, batch_cost: 0.7869, reader_cost: 0.00019, ips: 1.2708 samples/sec | ETA 02:00:08 2021-05-09 22:32:05 [INFO] [TRAIN] epoch: 83, iter: 30850/40000, loss: 0.3247, lr: 0.002725, batch_cost: 0.7852, reader_cost: 0.00018, ips: 1.2735 samples/sec | ETA 01:59:44 2021-05-09 22:32:13 [INFO] [TRAIN] epoch: 83, iter: 30860/40000, loss: 0.2269, lr: 0.002722, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 01:59:44 2021-05-09 22:32:21 [INFO] [TRAIN] epoch: 83, iter: 30870/40000, loss: 0.2889, lr: 0.002720, batch_cost: 0.7840, reader_cost: 0.00012, ips: 1.2756 samples/sec | ETA 01:59:17 2021-05-09 22:32:32 [INFO] [TRAIN] epoch: 84, iter: 30880/40000, loss: 0.2569, lr: 0.002717, batch_cost: 1.0835, reader_cost: 0.28092, ips: 0.9229 samples/sec | ETA 02:44:41 2021-05-09 22:32:40 [INFO] [TRAIN] epoch: 84, iter: 30890/40000, loss: 0.2956, lr: 0.002715, batch_cost: 0.7999, reader_cost: 0.00033, ips: 1.2501 samples/sec | ETA 02:01:27 2021-05-09 22:32:48 [INFO] [TRAIN] epoch: 84, iter: 30900/40000, loss: 0.4057, lr: 0.002712, batch_cost: 0.7865, reader_cost: 0.00017, ips: 1.2714 samples/sec | ETA 01:59:17 2021-05-09 22:32:55 [INFO] [TRAIN] epoch: 84, iter: 30910/40000, loss: 0.2064, lr: 0.002709, batch_cost: 0.7858, reader_cost: 0.00017, ips: 1.2725 samples/sec | ETA 01:59:03 2021-05-09 22:33:03 [INFO] [TRAIN] epoch: 84, iter: 30920/40000, loss: 0.3997, lr: 0.002707, batch_cost: 0.7851, reader_cost: 0.00017, ips: 1.2737 samples/sec | ETA 01:58:49 2021-05-09 22:33:11 [INFO] [TRAIN] epoch: 84, iter: 30930/40000, loss: 0.4395, lr: 0.002704, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 01:58:52 2021-05-09 22:33:19 [INFO] [TRAIN] epoch: 84, iter: 30940/40000, loss: 0.3172, lr: 0.002702, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2719 samples/sec | ETA 01:58:43 2021-05-09 22:33:27 [INFO] [TRAIN] epoch: 84, iter: 30950/40000, loss: 0.1461, lr: 0.002699, batch_cost: 0.7863, reader_cost: 0.00017, ips: 1.2717 samples/sec | ETA 01:58:36 2021-05-09 22:33:35 [INFO] [TRAIN] epoch: 84, iter: 30960/40000, loss: 0.2644, lr: 0.002696, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2725 samples/sec | ETA 01:58:24 2021-05-09 22:33:43 [INFO] [TRAIN] epoch: 84, iter: 30970/40000, loss: 0.3022, lr: 0.002694, batch_cost: 0.7864, reader_cost: 0.00017, ips: 1.2717 samples/sec | ETA 01:58:20 2021-05-09 22:33:50 [INFO] [TRAIN] epoch: 84, iter: 30980/40000, loss: 0.2503, lr: 0.002691, batch_cost: 0.7869, reader_cost: 0.00018, ips: 1.2708 samples/sec | ETA 01:58:18 2021-05-09 22:33:58 [INFO] [TRAIN] epoch: 84, iter: 30990/40000, loss: 0.3751, lr: 0.002689, batch_cost: 0.7866, reader_cost: 0.00018, ips: 1.2713 samples/sec | ETA 01:58:07 2021-05-09 22:34:06 [INFO] [TRAIN] epoch: 84, iter: 31000/40000, loss: 0.1463, lr: 0.002686, batch_cost: 0.7847, reader_cost: 0.00016, ips: 1.2743 samples/sec | ETA 01:57:42 2021-05-09 22:34:06 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-09 22:37:36 [INFO] [EVAL] #Images: 500 mIoU: 0.7681 Acc: 0.9570 Kappa: 0.9442 2021-05-09 22:37:36 [INFO] [EVAL] Class IoU: [0.9801 0.8419 0.9205 0.6461 0.6253 0.4927 0.6296 0.7323 0.9163 0.6641 0.9394 0.7771 0.5643 0.9423 0.843 0.8922 0.8068 0.6429 0.7369] 2021-05-09 22:37:36 [INFO] [EVAL] Class Acc: [0.992 0.9071 0.953 0.8059 0.8162 0.7678 0.7857 0.8861 0.9475 0.8626 0.9585 0.8559 0.8158 0.9643 0.9349 0.955 0.8886 0.8273 0.8356] 2021-05-09 22:38:04 [INFO] [EVAL] The model with the best validation mIoU (0.7695) was saved at iter 30000. 2021-05-09 22:38:12 [INFO] [TRAIN] epoch: 84, iter: 31010/40000, loss: 0.2762, lr: 0.002683, batch_cost: 0.7830, reader_cost: 0.00025, ips: 1.2771 samples/sec | ETA 01:57:19 2021-05-09 22:38:20 [INFO] [TRAIN] epoch: 84, iter: 31020/40000, loss: 0.2891, lr: 0.002681, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2728 samples/sec | ETA 01:57:35 2021-05-09 22:38:28 [INFO] [TRAIN] epoch: 84, iter: 31030/40000, loss: 0.2298, lr: 0.002678, batch_cost: 0.7850, reader_cost: 0.00016, ips: 1.2739 samples/sec | ETA 01:57:21 2021-05-09 22:38:36 [INFO] [TRAIN] epoch: 84, iter: 31040/40000, loss: 0.1616, lr: 0.002676, batch_cost: 0.7848, reader_cost: 0.00017, ips: 1.2742 samples/sec | ETA 01:57:11 2021-05-09 22:38:43 [INFO] [TRAIN] epoch: 84, iter: 31050/40000, loss: 0.1165, lr: 0.002673, batch_cost: 0.7883, reader_cost: 0.00017, ips: 1.2685 samples/sec | ETA 01:57:35 2021-05-09 22:38:51 [INFO] [TRAIN] epoch: 84, iter: 31060/40000, loss: 0.2934, lr: 0.002671, batch_cost: 0.7860, reader_cost: 0.00018, ips: 1.2723 samples/sec | ETA 01:57:06 2021-05-09 22:38:59 [INFO] [TRAIN] epoch: 84, iter: 31070/40000, loss: 0.0823, lr: 0.002668, batch_cost: 0.7847, reader_cost: 0.00017, ips: 1.2744 samples/sec | ETA 01:56:47 2021-05-09 22:39:07 [INFO] [TRAIN] epoch: 84, iter: 31080/40000, loss: 0.1962, lr: 0.002665, batch_cost: 0.7874, reader_cost: 0.00016, ips: 1.2700 samples/sec | ETA 01:57:03 2021-05-09 22:39:15 [INFO] [TRAIN] epoch: 84, iter: 31090/40000, loss: 0.2728, lr: 0.002663, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2728 samples/sec | ETA 01:56:40 2021-05-09 22:39:23 [INFO] [TRAIN] epoch: 84, iter: 31100/40000, loss: 0.2234, lr: 0.002660, batch_cost: 0.7875, reader_cost: 0.00016, ips: 1.2698 samples/sec | ETA 01:56:49 2021-05-09 22:39:31 [INFO] [TRAIN] epoch: 84, iter: 31110/40000, loss: 0.2871, lr: 0.002658, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 01:56:22 2021-05-09 22:39:39 [INFO] [TRAIN] epoch: 84, iter: 31120/40000, loss: 0.2339, lr: 0.002655, batch_cost: 0.7867, reader_cost: 0.00018, ips: 1.2711 samples/sec | ETA 01:56:25 2021-05-09 22:39:46 [INFO] [TRAIN] epoch: 84, iter: 31130/40000, loss: 0.2293, lr: 0.002652, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2723 samples/sec | ETA 01:56:11 2021-05-09 22:39:54 [INFO] [TRAIN] epoch: 84, iter: 31140/40000, loss: 0.2633, lr: 0.002650, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2709 samples/sec | ETA 01:56:11 2021-05-09 22:40:02 [INFO] [TRAIN] epoch: 84, iter: 31150/40000, loss: 0.3510, lr: 0.002647, batch_cost: 0.7873, reader_cost: 0.00018, ips: 1.2701 samples/sec | ETA 01:56:07 2021-05-09 22:40:10 [INFO] [TRAIN] epoch: 84, iter: 31160/40000, loss: 0.2954, lr: 0.002645, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2732 samples/sec | ETA 01:55:43 2021-05-09 22:40:18 [INFO] [TRAIN] epoch: 84, iter: 31170/40000, loss: 0.2287, lr: 0.002642, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 01:55:46 2021-05-09 22:40:26 [INFO] [TRAIN] epoch: 84, iter: 31180/40000, loss: 0.5196, lr: 0.002639, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2707 samples/sec | ETA 01:55:40 2021-05-09 22:40:34 [INFO] [TRAIN] epoch: 84, iter: 31190/40000, loss: 0.3974, lr: 0.002637, batch_cost: 0.7862, reader_cost: 0.00017, ips: 1.2719 samples/sec | ETA 01:55:26 2021-05-09 22:40:41 [INFO] [TRAIN] epoch: 84, iter: 31200/40000, loss: 0.2670, lr: 0.002634, batch_cost: 0.7853, reader_cost: 0.00017, ips: 1.2734 samples/sec | ETA 01:55:10 2021-05-09 22:40:49 [INFO] [TRAIN] epoch: 84, iter: 31210/40000, loss: 0.1295, lr: 0.002632, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2731 samples/sec | ETA 01:55:04 2021-05-09 22:40:57 [INFO] [TRAIN] epoch: 84, iter: 31220/40000, loss: 0.2633, lr: 0.002629, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2726 samples/sec | ETA 01:54:59 2021-05-09 22:41:05 [INFO] [TRAIN] epoch: 84, iter: 31230/40000, loss: 0.2861, lr: 0.002627, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2707 samples/sec | ETA 01:55:01 2021-05-09 22:41:13 [INFO] [TRAIN] epoch: 84, iter: 31240/40000, loss: 0.2694, lr: 0.002624, batch_cost: 0.7850, reader_cost: 0.00013, ips: 1.2738 samples/sec | ETA 01:54:36 2021-05-09 22:41:24 [INFO] [TRAIN] epoch: 85, iter: 31250/40000, loss: 0.4416, lr: 0.002621, batch_cost: 1.0946, reader_cost: 0.28002, ips: 0.9136 samples/sec | ETA 02:39:37 2021-05-09 22:41:32 [INFO] [TRAIN] epoch: 85, iter: 31260/40000, loss: 0.3133, lr: 0.002619, batch_cost: 0.7968, reader_cost: 0.00034, ips: 1.2551 samples/sec | ETA 01:56:03 2021-05-09 22:41:40 [INFO] [TRAIN] epoch: 85, iter: 31270/40000, loss: 0.5786, lr: 0.002616, batch_cost: 0.7861, reader_cost: 0.00014, ips: 1.2720 samples/sec | ETA 01:54:23 2021-05-09 22:41:47 [INFO] [TRAIN] epoch: 85, iter: 31280/40000, loss: 0.2645, lr: 0.002614, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2718 samples/sec | ETA 01:54:16 2021-05-09 22:41:55 [INFO] [TRAIN] epoch: 85, iter: 31290/40000, loss: 0.4580, lr: 0.002611, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2735 samples/sec | ETA 01:53:59 2021-05-09 22:42:03 [INFO] [TRAIN] epoch: 85, iter: 31300/40000, loss: 0.4380, lr: 0.002608, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2731 samples/sec | ETA 01:53:53 2021-05-09 22:42:11 [INFO] [TRAIN] epoch: 85, iter: 31310/40000, loss: 0.1820, lr: 0.002606, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 01:53:50 2021-05-09 22:42:19 [INFO] [TRAIN] epoch: 85, iter: 31320/40000, loss: 0.1527, lr: 0.002603, batch_cost: 0.7858, reader_cost: 0.00017, ips: 1.2726 samples/sec | ETA 01:53:40 2021-05-09 22:42:27 [INFO] [TRAIN] epoch: 85, iter: 31330/40000, loss: 0.3239, lr: 0.002601, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2704 samples/sec | ETA 01:53:44 2021-05-09 22:42:35 [INFO] [TRAIN] epoch: 85, iter: 31340/40000, loss: 0.3031, lr: 0.002598, batch_cost: 0.7877, reader_cost: 0.00017, ips: 1.2696 samples/sec | ETA 01:53:41 2021-05-09 22:42:43 [INFO] [TRAIN] epoch: 85, iter: 31350/40000, loss: 0.2338, lr: 0.002595, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2714 samples/sec | ETA 01:53:23 2021-05-09 22:42:50 [INFO] [TRAIN] epoch: 85, iter: 31360/40000, loss: 0.2918, lr: 0.002593, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2719 samples/sec | ETA 01:53:13 2021-05-09 22:42:58 [INFO] [TRAIN] epoch: 85, iter: 31370/40000, loss: 0.0868, lr: 0.002590, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2732 samples/sec | ETA 01:52:58 2021-05-09 22:43:06 [INFO] [TRAIN] epoch: 85, iter: 31380/40000, loss: 0.2126, lr: 0.002588, batch_cost: 0.7880, reader_cost: 0.00016, ips: 1.2690 samples/sec | ETA 01:53:12 2021-05-09 22:43:14 [INFO] [TRAIN] epoch: 85, iter: 31390/40000, loss: 0.2950, lr: 0.002585, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2710 samples/sec | ETA 01:52:54 2021-05-09 22:43:22 [INFO] [TRAIN] epoch: 85, iter: 31400/40000, loss: 0.3854, lr: 0.002582, batch_cost: 0.7860, reader_cost: 0.00017, ips: 1.2723 samples/sec | ETA 01:52:39 2021-05-09 22:43:30 [INFO] [TRAIN] epoch: 85, iter: 31410/40000, loss: 0.2582, lr: 0.002580, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 01:52:36 2021-05-09 22:43:38 [INFO] [TRAIN] epoch: 85, iter: 31420/40000, loss: 0.1983, lr: 0.002577, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 01:52:28 2021-05-09 22:43:45 [INFO] [TRAIN] epoch: 85, iter: 31430/40000, loss: 0.1869, lr: 0.002575, batch_cost: 0.7871, reader_cost: 0.00017, ips: 1.2705 samples/sec | ETA 01:52:25 2021-05-09 22:43:53 [INFO] [TRAIN] epoch: 85, iter: 31440/40000, loss: 0.1653, lr: 0.002572, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2708 samples/sec | ETA 01:52:15 2021-05-09 22:44:01 [INFO] [TRAIN] epoch: 85, iter: 31450/40000, loss: 0.1603, lr: 0.002569, batch_cost: 0.7877, reader_cost: 0.00016, ips: 1.2696 samples/sec | ETA 01:52:14 2021-05-09 22:44:09 [INFO] [TRAIN] epoch: 85, iter: 31460/40000, loss: 0.2858, lr: 0.002567, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 01:51:56 2021-05-09 22:44:17 [INFO] [TRAIN] epoch: 85, iter: 31470/40000, loss: 0.2593, lr: 0.002564, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2736 samples/sec | ETA 01:51:37 2021-05-09 22:44:25 [INFO] [TRAIN] epoch: 85, iter: 31480/40000, loss: 0.2615, lr: 0.002562, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2735 samples/sec | ETA 01:51:30 2021-05-09 22:44:33 [INFO] [TRAIN] epoch: 85, iter: 31490/40000, loss: 0.2410, lr: 0.002559, batch_cost: 0.7865, reader_cost: 0.00017, ips: 1.2715 samples/sec | ETA 01:51:32 2021-05-09 22:44:41 [INFO] [TRAIN] epoch: 85, iter: 31500/40000, loss: 0.1575, lr: 0.002556, batch_cost: 0.7855, reader_cost: 0.00017, ips: 1.2730 samples/sec | ETA 01:51:17 2021-05-09 22:44:48 [INFO] [TRAIN] epoch: 85, iter: 31510/40000, loss: 0.1964, lr: 0.002554, batch_cost: 0.7858, reader_cost: 0.00017, ips: 1.2726 samples/sec | ETA 01:51:11 2021-05-09 22:44:56 [INFO] [TRAIN] epoch: 85, iter: 31520/40000, loss: 0.2992, lr: 0.002551, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2728 samples/sec | ETA 01:51:02 2021-05-09 22:45:04 [INFO] [TRAIN] epoch: 85, iter: 31530/40000, loss: 0.4543, lr: 0.002549, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2717 samples/sec | ETA 01:51:00 2021-05-09 22:45:12 [INFO] [TRAIN] epoch: 85, iter: 31540/40000, loss: 0.3643, lr: 0.002546, batch_cost: 0.7864, reader_cost: 0.00014, ips: 1.2716 samples/sec | ETA 01:50:53 2021-05-09 22:45:20 [INFO] [TRAIN] epoch: 85, iter: 31550/40000, loss: 0.4177, lr: 0.002543, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 01:50:41 2021-05-09 22:45:28 [INFO] [TRAIN] epoch: 85, iter: 31560/40000, loss: 0.3617, lr: 0.002541, batch_cost: 0.7853, reader_cost: 0.00017, ips: 1.2735 samples/sec | ETA 01:50:27 2021-05-09 22:45:36 [INFO] [TRAIN] epoch: 85, iter: 31570/40000, loss: 0.3586, lr: 0.002538, batch_cost: 0.7863, reader_cost: 0.00017, ips: 1.2717 samples/sec | ETA 01:50:28 2021-05-09 22:45:43 [INFO] [TRAIN] epoch: 85, iter: 31580/40000, loss: 0.1601, lr: 0.002536, batch_cost: 0.7874, reader_cost: 0.00017, ips: 1.2699 samples/sec | ETA 01:50:30 2021-05-09 22:45:51 [INFO] [TRAIN] epoch: 85, iter: 31590/40000, loss: 0.2285, lr: 0.002533, batch_cost: 0.7843, reader_cost: 0.00017, ips: 1.2750 samples/sec | ETA 01:49:55 2021-05-09 22:45:59 [INFO] [TRAIN] epoch: 85, iter: 31600/40000, loss: 0.2914, lr: 0.002530, batch_cost: 0.7875, reader_cost: 0.00015, ips: 1.2698 samples/sec | ETA 01:50:15 2021-05-09 22:46:07 [INFO] [TRAIN] epoch: 85, iter: 31610/40000, loss: 0.2689, lr: 0.002528, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2724 samples/sec | ETA 01:49:53 2021-05-09 22:46:15 [INFO] [TRAIN] epoch: 85, iter: 31620/40000, loss: 0.3214, lr: 0.002525, batch_cost: 0.7859, reader_cost: 0.00010, ips: 1.2724 samples/sec | ETA 01:49:46 2021-05-09 22:46:26 [INFO] [TRAIN] epoch: 86, iter: 31630/40000, loss: 0.1934, lr: 0.002523, batch_cost: 1.1116, reader_cost: 0.26125, ips: 0.8996 samples/sec | ETA 02:35:03 2021-05-09 22:46:34 [INFO] [TRAIN] epoch: 86, iter: 31640/40000, loss: 0.4360, lr: 0.002520, batch_cost: 0.7913, reader_cost: 0.00033, ips: 1.2638 samples/sec | ETA 01:50:15 2021-05-09 22:46:42 [INFO] [TRAIN] epoch: 86, iter: 31650/40000, loss: 0.3210, lr: 0.002517, batch_cost: 0.7869, reader_cost: 0.00015, ips: 1.2709 samples/sec | ETA 01:49:30 2021-05-09 22:46:50 [INFO] [TRAIN] epoch: 86, iter: 31660/40000, loss: 0.3721, lr: 0.002515, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2716 samples/sec | ETA 01:49:18 2021-05-09 22:46:57 [INFO] [TRAIN] epoch: 86, iter: 31670/40000, loss: 0.3312, lr: 0.002512, batch_cost: 0.7847, reader_cost: 0.00018, ips: 1.2744 samples/sec | ETA 01:48:56 2021-05-09 22:47:05 [INFO] [TRAIN] epoch: 86, iter: 31680/40000, loss: 0.4117, lr: 0.002510, batch_cost: 0.7847, reader_cost: 0.00016, ips: 1.2744 samples/sec | ETA 01:48:48 2021-05-09 22:47:13 [INFO] [TRAIN] epoch: 86, iter: 31690/40000, loss: 0.1401, lr: 0.002507, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2719 samples/sec | ETA 01:48:53 2021-05-09 22:47:21 [INFO] [TRAIN] epoch: 86, iter: 31700/40000, loss: 0.2365, lr: 0.002504, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2713 samples/sec | ETA 01:48:48 2021-05-09 22:47:29 [INFO] [TRAIN] epoch: 86, iter: 31710/40000, loss: 0.3366, lr: 0.002502, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2723 samples/sec | ETA 01:48:35 2021-05-09 22:47:37 [INFO] [TRAIN] epoch: 86, iter: 31720/40000, loss: 0.1596, lr: 0.002499, batch_cost: 0.7868, reader_cost: 0.00018, ips: 1.2709 samples/sec | ETA 01:48:34 2021-05-09 22:47:45 [INFO] [TRAIN] epoch: 86, iter: 31730/40000, loss: 0.2895, lr: 0.002497, batch_cost: 0.7841, reader_cost: 0.00017, ips: 1.2753 samples/sec | ETA 01:48:04 2021-05-09 22:47:52 [INFO] [TRAIN] epoch: 86, iter: 31740/40000, loss: 0.1393, lr: 0.002494, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2711 samples/sec | ETA 01:48:18 2021-05-09 22:48:00 [INFO] [TRAIN] epoch: 86, iter: 31750/40000, loss: 0.2379, lr: 0.002491, batch_cost: 0.7846, reader_cost: 0.00016, ips: 1.2746 samples/sec | ETA 01:47:52 2021-05-09 22:48:08 [INFO] [TRAIN] epoch: 86, iter: 31760/40000, loss: 0.2635, lr: 0.002489, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 01:47:57 2021-05-09 22:48:16 [INFO] [TRAIN] epoch: 86, iter: 31770/40000, loss: 0.3072, lr: 0.002486, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 01:47:49 2021-05-09 22:48:24 [INFO] [TRAIN] epoch: 86, iter: 31780/40000, loss: 0.2492, lr: 0.002483, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2736 samples/sec | ETA 01:47:34 2021-05-09 22:48:32 [INFO] [TRAIN] epoch: 86, iter: 31790/40000, loss: 0.1600, lr: 0.002481, batch_cost: 0.7838, reader_cost: 0.00016, ips: 1.2759 samples/sec | ETA 01:47:14 2021-05-09 22:48:40 [INFO] [TRAIN] epoch: 86, iter: 31800/40000, loss: 0.2163, lr: 0.002478, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2727 samples/sec | ETA 01:47:23 2021-05-09 22:48:47 [INFO] [TRAIN] epoch: 86, iter: 31810/40000, loss: 0.1424, lr: 0.002476, batch_cost: 0.7846, reader_cost: 0.00015, ips: 1.2745 samples/sec | ETA 01:47:05 2021-05-09 22:48:55 [INFO] [TRAIN] epoch: 86, iter: 31820/40000, loss: 0.1647, lr: 0.002473, batch_cost: 0.7846, reader_cost: 0.00017, ips: 1.2745 samples/sec | ETA 01:46:58 2021-05-09 22:49:03 [INFO] [TRAIN] epoch: 86, iter: 31830/40000, loss: 0.2716, lr: 0.002470, batch_cost: 0.7846, reader_cost: 0.00016, ips: 1.2746 samples/sec | ETA 01:46:49 2021-05-09 22:49:11 [INFO] [TRAIN] epoch: 86, iter: 31840/40000, loss: 0.2298, lr: 0.002468, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2727 samples/sec | ETA 01:46:51 2021-05-09 22:49:19 [INFO] [TRAIN] epoch: 86, iter: 31850/40000, loss: 0.2756, lr: 0.002465, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 01:46:46 2021-05-09 22:49:27 [INFO] [TRAIN] epoch: 86, iter: 31860/40000, loss: 0.2438, lr: 0.002463, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 01:46:35 2021-05-09 22:49:35 [INFO] [TRAIN] epoch: 86, iter: 31870/40000, loss: 0.1215, lr: 0.002460, batch_cost: 0.7862, reader_cost: 0.00017, ips: 1.2719 samples/sec | ETA 01:46:32 2021-05-09 22:49:42 [INFO] [TRAIN] epoch: 86, iter: 31880/40000, loss: 0.2236, lr: 0.002457, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2723 samples/sec | ETA 01:46:22 2021-05-09 22:49:50 [INFO] [TRAIN] epoch: 86, iter: 31890/40000, loss: 0.2437, lr: 0.002455, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 01:46:15 2021-05-09 22:49:58 [INFO] [TRAIN] epoch: 86, iter: 31900/40000, loss: 0.3938, lr: 0.002452, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2715 samples/sec | ETA 01:46:10 2021-05-09 22:50:06 [INFO] [TRAIN] epoch: 86, iter: 31910/40000, loss: 0.3999, lr: 0.002450, batch_cost: 0.7874, reader_cost: 0.00015, ips: 1.2700 samples/sec | ETA 01:46:10 2021-05-09 22:50:14 [INFO] [TRAIN] epoch: 86, iter: 31920/40000, loss: 0.3754, lr: 0.002447, batch_cost: 0.7859, reader_cost: 0.00014, ips: 1.2725 samples/sec | ETA 01:45:49 2021-05-09 22:50:22 [INFO] [TRAIN] epoch: 86, iter: 31930/40000, loss: 0.5476, lr: 0.002444, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2731 samples/sec | ETA 01:45:38 2021-05-09 22:50:30 [INFO] [TRAIN] epoch: 86, iter: 31940/40000, loss: 0.3953, lr: 0.002442, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2713 samples/sec | ETA 01:45:40 2021-05-09 22:50:37 [INFO] [TRAIN] epoch: 86, iter: 31950/40000, loss: 0.2009, lr: 0.002439, batch_cost: 0.7864, reader_cost: 0.00019, ips: 1.2717 samples/sec | ETA 01:45:30 2021-05-09 22:50:45 [INFO] [TRAIN] epoch: 86, iter: 31960/40000, loss: 0.1834, lr: 0.002436, batch_cost: 0.7872, reader_cost: 0.00018, ips: 1.2704 samples/sec | ETA 01:45:28 2021-05-09 22:50:53 [INFO] [TRAIN] epoch: 86, iter: 31970/40000, loss: 0.2662, lr: 0.002434, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2723 samples/sec | ETA 01:45:11 2021-05-09 22:51:01 [INFO] [TRAIN] epoch: 86, iter: 31980/40000, loss: 0.3230, lr: 0.002431, batch_cost: 0.7844, reader_cost: 0.00016, ips: 1.2748 samples/sec | ETA 01:44:51 2021-05-09 22:51:09 [INFO] [TRAIN] epoch: 86, iter: 31990/40000, loss: 0.3632, lr: 0.002429, batch_cost: 0.7856, reader_cost: 0.00031, ips: 1.2730 samples/sec | ETA 01:44:52 2021-05-09 22:51:20 [INFO] [TRAIN] epoch: 87, iter: 32000/40000, loss: 0.2294, lr: 0.002426, batch_cost: 1.0905, reader_cost: 0.24448, ips: 0.9170 samples/sec | ETA 02:25:24 2021-05-09 22:51:20 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-09 22:54:54 [INFO] [EVAL] #Images: 500 mIoU: 0.7658 Acc: 0.9567 Kappa: 0.9438 2021-05-09 22:54:54 [INFO] [EVAL] Class IoU: [0.9801 0.8419 0.9198 0.6292 0.6202 0.495 0.6246 0.7246 0.9154 0.6475 0.9417 0.7781 0.5805 0.9416 0.8335 0.8998 0.8075 0.6364 0.7339] 2021-05-09 22:54:54 [INFO] [EVAL] Class Acc: [0.9922 0.9034 0.9503 0.8529 0.8206 0.7667 0.8203 0.8816 0.9467 0.8709 0.9662 0.8617 0.7606 0.9615 0.9449 0.9609 0.8829 0.7844 0.8397] 2021-05-09 22:55:22 [INFO] [EVAL] The model with the best validation mIoU (0.7695) was saved at iter 30000. 2021-05-09 22:55:30 [INFO] [TRAIN] epoch: 87, iter: 32010/40000, loss: 0.4165, lr: 0.002423, batch_cost: 0.7848, reader_cost: 0.00060, ips: 1.2741 samples/sec | ETA 01:44:30 2021-05-09 22:55:38 [INFO] [TRAIN] epoch: 87, iter: 32020/40000, loss: 0.3165, lr: 0.002421, batch_cost: 0.7847, reader_cost: 0.00017, ips: 1.2743 samples/sec | ETA 01:44:22 2021-05-09 22:55:46 [INFO] [TRAIN] epoch: 87, iter: 32030/40000, loss: 0.3130, lr: 0.002418, batch_cost: 0.7850, reader_cost: 0.00016, ips: 1.2739 samples/sec | ETA 01:44:16 2021-05-09 22:55:57 [INFO] [TRAIN] epoch: 87, iter: 32040/40000, loss: 0.3458, lr: 0.002416, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 01:44:19 2021-05-09 22:56:05 [INFO] [TRAIN] epoch: 87, iter: 32050/40000, loss: 0.2865, lr: 0.002413, batch_cost: 0.7844, reader_cost: 0.00016, ips: 1.2748 samples/sec | ETA 01:43:56 2021-05-09 22:56:13 [INFO] [TRAIN] epoch: 87, iter: 32060/40000, loss: 0.1949, lr: 0.002410, batch_cost: 0.7883, reader_cost: 0.00016, ips: 1.2685 samples/sec | ETA 01:44:19 2021-05-09 22:56:21 [INFO] [TRAIN] epoch: 87, iter: 32070/40000, loss: 0.2115, lr: 0.002408, batch_cost: 0.7862, reader_cost: 0.00017, ips: 1.2720 samples/sec | ETA 01:43:54 2021-05-09 22:56:29 [INFO] [TRAIN] epoch: 87, iter: 32080/40000, loss: 0.3280, lr: 0.002405, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2726 samples/sec | ETA 01:43:43 2021-05-09 22:56:36 [INFO] [TRAIN] epoch: 87, iter: 32090/40000, loss: 0.1963, lr: 0.002402, batch_cost: 0.7876, reader_cost: 0.00017, ips: 1.2696 samples/sec | ETA 01:43:50 2021-05-09 22:56:44 [INFO] [TRAIN] epoch: 87, iter: 32100/40000, loss: 0.2527, lr: 0.002400, batch_cost: 0.7873, reader_cost: 0.00016, ips: 1.2702 samples/sec | ETA 01:43:39 2021-05-09 22:56:52 [INFO] [TRAIN] epoch: 87, iter: 32110/40000, loss: 0.1559, lr: 0.002397, batch_cost: 0.7881, reader_cost: 0.00016, ips: 1.2689 samples/sec | ETA 01:43:38 2021-05-09 22:57:00 [INFO] [TRAIN] epoch: 87, iter: 32120/40000, loss: 0.1928, lr: 0.002395, batch_cost: 0.7853, reader_cost: 0.00017, ips: 1.2735 samples/sec | ETA 01:43:07 2021-05-09 22:57:08 [INFO] [TRAIN] epoch: 87, iter: 32130/40000, loss: 0.3382, lr: 0.002392, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2713 samples/sec | ETA 01:43:10 2021-05-09 22:57:16 [INFO] [TRAIN] epoch: 87, iter: 32140/40000, loss: 0.2931, lr: 0.002389, batch_cost: 0.7875, reader_cost: 0.00016, ips: 1.2699 samples/sec | ETA 01:43:09 2021-05-09 22:57:24 [INFO] [TRAIN] epoch: 87, iter: 32150/40000, loss: 0.3563, lr: 0.002387, batch_cost: 0.7851, reader_cost: 0.00016, ips: 1.2737 samples/sec | ETA 01:42:43 2021-05-09 22:57:31 [INFO] [TRAIN] epoch: 87, iter: 32160/40000, loss: 0.2532, lr: 0.002384, batch_cost: 0.7850, reader_cost: 0.00015, ips: 1.2739 samples/sec | ETA 01:42:34 2021-05-09 22:57:39 [INFO] [TRAIN] epoch: 87, iter: 32170/40000, loss: 0.3069, lr: 0.002381, batch_cost: 0.7867, reader_cost: 0.00017, ips: 1.2711 samples/sec | ETA 01:42:40 2021-05-09 22:57:47 [INFO] [TRAIN] epoch: 87, iter: 32180/40000, loss: 0.3314, lr: 0.002379, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2719 samples/sec | ETA 01:42:28 2021-05-09 22:57:55 [INFO] [TRAIN] epoch: 87, iter: 32190/40000, loss: 0.0349, lr: 0.002376, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2726 samples/sec | ETA 01:42:17 2021-05-09 22:58:03 [INFO] [TRAIN] epoch: 87, iter: 32200/40000, loss: 0.2565, lr: 0.002374, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 01:42:14 2021-05-09 22:58:11 [INFO] [TRAIN] epoch: 87, iter: 32210/40000, loss: 0.2190, lr: 0.002371, batch_cost: 0.7850, reader_cost: 0.00017, ips: 1.2740 samples/sec | ETA 01:41:54 2021-05-09 22:58:19 [INFO] [TRAIN] epoch: 87, iter: 32220/40000, loss: 0.2338, lr: 0.002368, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2710 samples/sec | ETA 01:42:01 2021-05-09 22:58:27 [INFO] [TRAIN] epoch: 87, iter: 32230/40000, loss: 0.2135, lr: 0.002366, batch_cost: 0.7850, reader_cost: 0.00016, ips: 1.2739 samples/sec | ETA 01:41:39 2021-05-09 22:58:34 [INFO] [TRAIN] epoch: 87, iter: 32240/40000, loss: 0.1396, lr: 0.002363, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2718 samples/sec | ETA 01:41:41 2021-05-09 22:58:42 [INFO] [TRAIN] epoch: 87, iter: 32250/40000, loss: 0.2140, lr: 0.002360, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 01:41:29 2021-05-09 22:58:50 [INFO] [TRAIN] epoch: 87, iter: 32260/40000, loss: 0.2123, lr: 0.002358, batch_cost: 0.7858, reader_cost: 0.00014, ips: 1.2725 samples/sec | ETA 01:41:22 2021-05-09 22:58:58 [INFO] [TRAIN] epoch: 87, iter: 32270/40000, loss: 0.4831, lr: 0.002355, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2716 samples/sec | ETA 01:41:19 2021-05-09 22:59:06 [INFO] [TRAIN] epoch: 87, iter: 32280/40000, loss: 0.3869, lr: 0.002353, batch_cost: 0.7844, reader_cost: 0.00016, ips: 1.2748 samples/sec | ETA 01:40:55 2021-05-09 22:59:14 [INFO] [TRAIN] epoch: 87, iter: 32290/40000, loss: 0.2898, lr: 0.002350, batch_cost: 0.7845, reader_cost: 0.00017, ips: 1.2747 samples/sec | ETA 01:40:48 2021-05-09 22:59:21 [INFO] [TRAIN] epoch: 87, iter: 32300/40000, loss: 0.4675, lr: 0.002347, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2733 samples/sec | ETA 01:40:47 2021-05-09 22:59:29 [INFO] [TRAIN] epoch: 87, iter: 32310/40000, loss: 0.3148, lr: 0.002345, batch_cost: 0.7869, reader_cost: 0.00015, ips: 1.2708 samples/sec | ETA 01:40:51 2021-05-09 22:59:37 [INFO] [TRAIN] epoch: 87, iter: 32320/40000, loss: 0.2038, lr: 0.002342, batch_cost: 0.7851, reader_cost: 0.00016, ips: 1.2738 samples/sec | ETA 01:40:29 2021-05-09 22:59:45 [INFO] [TRAIN] epoch: 87, iter: 32330/40000, loss: 0.1899, lr: 0.002339, batch_cost: 0.7859, reader_cost: 0.00017, ips: 1.2724 samples/sec | ETA 01:40:27 2021-05-09 22:59:53 [INFO] [TRAIN] epoch: 87, iter: 32340/40000, loss: 0.2384, lr: 0.002337, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2731 samples/sec | ETA 01:40:16 2021-05-09 23:00:01 [INFO] [TRAIN] epoch: 87, iter: 32350/40000, loss: 0.3186, lr: 0.002334, batch_cost: 0.7844, reader_cost: 0.00015, ips: 1.2748 samples/sec | ETA 01:40:01 2021-05-09 23:00:09 [INFO] [TRAIN] epoch: 87, iter: 32360/40000, loss: 0.2473, lr: 0.002332, batch_cost: 0.7851, reader_cost: 0.00013, ips: 1.2736 samples/sec | ETA 01:39:58 2021-05-09 23:00:20 [INFO] [TRAIN] epoch: 88, iter: 32370/40000, loss: 0.2885, lr: 0.002329, batch_cost: 1.1004, reader_cost: 0.29083, ips: 0.9088 samples/sec | ETA 02:19:55 2021-05-09 23:00:28 [INFO] [TRAIN] epoch: 88, iter: 32380/40000, loss: 0.3529, lr: 0.002326, batch_cost: 0.7958, reader_cost: 0.00034, ips: 1.2565 samples/sec | ETA 01:41:04 2021-05-09 23:00:35 [INFO] [TRAIN] epoch: 88, iter: 32390/40000, loss: 0.3682, lr: 0.002324, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2705 samples/sec | ETA 01:39:49 2021-05-09 23:00:43 [INFO] [TRAIN] epoch: 88, iter: 32400/40000, loss: 0.3041, lr: 0.002321, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2713 samples/sec | ETA 01:39:37 2021-05-09 23:00:51 [INFO] [TRAIN] epoch: 88, iter: 32410/40000, loss: 0.3618, lr: 0.002318, batch_cost: 0.7858, reader_cost: 0.00018, ips: 1.2726 samples/sec | ETA 01:39:24 2021-05-09 23:00:59 [INFO] [TRAIN] epoch: 88, iter: 32420/40000, loss: 0.3878, lr: 0.002316, batch_cost: 0.7862, reader_cost: 0.00017, ips: 1.2719 samples/sec | ETA 01:39:19 2021-05-09 23:01:07 [INFO] [TRAIN] epoch: 88, iter: 32430/40000, loss: 0.2178, lr: 0.002313, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2714 samples/sec | ETA 01:39:13 2021-05-09 23:01:15 [INFO] [TRAIN] epoch: 88, iter: 32440/40000, loss: 0.2010, lr: 0.002311, batch_cost: 0.7875, reader_cost: 0.00016, ips: 1.2698 samples/sec | ETA 01:39:13 2021-05-09 23:01:23 [INFO] [TRAIN] epoch: 88, iter: 32450/40000, loss: 0.3014, lr: 0.002308, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2716 samples/sec | ETA 01:38:57 2021-05-09 23:01:31 [INFO] [TRAIN] epoch: 88, iter: 32460/40000, loss: 0.2848, lr: 0.002305, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 01:38:47 2021-05-09 23:01:38 [INFO] [TRAIN] epoch: 88, iter: 32470/40000, loss: 0.2599, lr: 0.002303, batch_cost: 0.7867, reader_cost: 0.00018, ips: 1.2712 samples/sec | ETA 01:38:43 2021-05-09 23:01:46 [INFO] [TRAIN] epoch: 88, iter: 32480/40000, loss: 0.2161, lr: 0.002300, batch_cost: 0.7871, reader_cost: 0.00018, ips: 1.2706 samples/sec | ETA 01:38:38 2021-05-09 23:01:54 [INFO] [TRAIN] epoch: 88, iter: 32490/40000, loss: 0.1724, lr: 0.002297, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2721 samples/sec | ETA 01:38:23 2021-05-09 23:02:02 [INFO] [TRAIN] epoch: 88, iter: 32500/40000, loss: 0.2971, lr: 0.002295, batch_cost: 0.7863, reader_cost: 0.00017, ips: 1.2717 samples/sec | ETA 01:38:17 2021-05-09 23:02:10 [INFO] [TRAIN] epoch: 88, iter: 32510/40000, loss: 0.3052, lr: 0.002292, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 01:38:09 2021-05-09 23:02:18 [INFO] [TRAIN] epoch: 88, iter: 32520/40000, loss: 0.2089, lr: 0.002289, batch_cost: 0.7855, reader_cost: 0.00017, ips: 1.2731 samples/sec | ETA 01:37:55 2021-05-09 23:02:26 [INFO] [TRAIN] epoch: 88, iter: 32530/40000, loss: 0.2283, lr: 0.002287, batch_cost: 0.7844, reader_cost: 0.00016, ips: 1.2749 samples/sec | ETA 01:37:39 2021-05-09 23:02:33 [INFO] [TRAIN] epoch: 88, iter: 32540/40000, loss: 0.2314, lr: 0.002284, batch_cost: 0.7851, reader_cost: 0.00016, ips: 1.2737 samples/sec | ETA 01:37:36 2021-05-09 23:02:41 [INFO] [TRAIN] epoch: 88, iter: 32550/40000, loss: 0.2773, lr: 0.002282, batch_cost: 0.7850, reader_cost: 0.00016, ips: 1.2740 samples/sec | ETA 01:37:27 2021-05-09 23:02:49 [INFO] [TRAIN] epoch: 88, iter: 32560/40000, loss: 0.0517, lr: 0.002279, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2719 samples/sec | ETA 01:37:29 2021-05-09 23:02:57 [INFO] [TRAIN] epoch: 88, iter: 32570/40000, loss: 0.1983, lr: 0.002276, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2703 samples/sec | ETA 01:37:29 2021-05-09 23:03:05 [INFO] [TRAIN] epoch: 88, iter: 32580/40000, loss: 0.2692, lr: 0.002274, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2731 samples/sec | ETA 01:37:08 2021-05-09 23:03:13 [INFO] [TRAIN] epoch: 88, iter: 32590/40000, loss: 0.1292, lr: 0.002271, batch_cost: 0.7842, reader_cost: 0.00016, ips: 1.2752 samples/sec | ETA 01:36:50 2021-05-09 23:03:21 [INFO] [TRAIN] epoch: 88, iter: 32600/40000, loss: 0.2689, lr: 0.002268, batch_cost: 0.7839, reader_cost: 0.00017, ips: 1.2757 samples/sec | ETA 01:36:40 2021-05-09 23:03:28 [INFO] [TRAIN] epoch: 88, iter: 32610/40000, loss: 0.1639, lr: 0.002266, batch_cost: 0.7849, reader_cost: 0.00016, ips: 1.2741 samples/sec | ETA 01:36:40 2021-05-09 23:03:36 [INFO] [TRAIN] epoch: 88, iter: 32620/40000, loss: 0.3102, lr: 0.002263, batch_cost: 0.7860, reader_cost: 0.00018, ips: 1.2723 samples/sec | ETA 01:36:40 2021-05-09 23:03:44 [INFO] [TRAIN] epoch: 88, iter: 32630/40000, loss: 0.1577, lr: 0.002261, batch_cost: 0.7857, reader_cost: 0.00014, ips: 1.2727 samples/sec | ETA 01:36:30 2021-05-09 23:03:52 [INFO] [TRAIN] epoch: 88, iter: 32640/40000, loss: 0.2802, lr: 0.002258, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2734 samples/sec | ETA 01:36:19 2021-05-09 23:04:00 [INFO] [TRAIN] epoch: 88, iter: 32650/40000, loss: 0.4426, lr: 0.002255, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2730 samples/sec | ETA 01:36:13 2021-05-09 23:04:08 [INFO] [TRAIN] epoch: 88, iter: 32660/40000, loss: 0.2969, lr: 0.002253, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2721 samples/sec | ETA 01:36:10 2021-05-09 23:04:16 [INFO] [TRAIN] epoch: 88, iter: 32670/40000, loss: 0.5372, lr: 0.002250, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2711 samples/sec | ETA 01:36:06 2021-05-09 23:04:23 [INFO] [TRAIN] epoch: 88, iter: 32680/40000, loss: 0.3463, lr: 0.002247, batch_cost: 0.7872, reader_cost: 0.00015, ips: 1.2704 samples/sec | ETA 01:36:02 2021-05-09 23:04:31 [INFO] [TRAIN] epoch: 88, iter: 32690/40000, loss: 0.2160, lr: 0.002245, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2719 samples/sec | ETA 01:35:47 2021-05-09 23:04:39 [INFO] [TRAIN] epoch: 88, iter: 32700/40000, loss: 0.1061, lr: 0.002242, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2722 samples/sec | ETA 01:35:37 2021-05-09 23:04:47 [INFO] [TRAIN] epoch: 88, iter: 32710/40000, loss: 0.2491, lr: 0.002239, batch_cost: 0.7874, reader_cost: 0.00016, ips: 1.2700 samples/sec | ETA 01:35:40 2021-05-09 23:04:55 [INFO] [TRAIN] epoch: 88, iter: 32720/40000, loss: 0.2473, lr: 0.002237, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2735 samples/sec | ETA 01:35:16 2021-05-09 23:05:03 [INFO] [TRAIN] epoch: 88, iter: 32730/40000, loss: 0.2984, lr: 0.002234, batch_cost: 0.7869, reader_cost: 0.00013, ips: 1.2708 samples/sec | ETA 01:35:20 2021-05-09 23:05:14 [INFO] [TRAIN] epoch: 89, iter: 32740/40000, loss: 0.3128, lr: 0.002231, batch_cost: 1.0838, reader_cost: 0.23199, ips: 0.9227 samples/sec | ETA 02:11:08 2021-05-09 23:05:22 [INFO] [TRAIN] epoch: 89, iter: 32750/40000, loss: 0.2531, lr: 0.002229, batch_cost: 0.7971, reader_cost: 0.00031, ips: 1.2545 samples/sec | ETA 01:36:19 2021-05-09 23:05:29 [INFO] [TRAIN] epoch: 89, iter: 32760/40000, loss: 0.4902, lr: 0.002226, batch_cost: 0.7850, reader_cost: 0.00016, ips: 1.2739 samples/sec | ETA 01:34:43 2021-05-09 23:05:37 [INFO] [TRAIN] epoch: 89, iter: 32770/40000, loss: 0.3446, lr: 0.002224, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2709 samples/sec | ETA 01:34:49 2021-05-09 23:05:45 [INFO] [TRAIN] epoch: 89, iter: 32780/40000, loss: 0.3718, lr: 0.002221, batch_cost: 0.7871, reader_cost: 0.00015, ips: 1.2705 samples/sec | ETA 01:34:42 2021-05-09 23:05:53 [INFO] [TRAIN] epoch: 89, iter: 32790/40000, loss: 0.4220, lr: 0.002218, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2717 samples/sec | ETA 01:34:29 2021-05-09 23:06:01 [INFO] [TRAIN] epoch: 89, iter: 32800/40000, loss: 0.3659, lr: 0.002216, batch_cost: 0.7884, reader_cost: 0.00015, ips: 1.2683 samples/sec | ETA 01:34:36 2021-05-09 23:06:09 [INFO] [TRAIN] epoch: 89, iter: 32810/40000, loss: 0.1113, lr: 0.002213, batch_cost: 0.7871, reader_cost: 0.00015, ips: 1.2705 samples/sec | ETA 01:34:19 2021-05-09 23:06:17 [INFO] [TRAIN] epoch: 89, iter: 32820/40000, loss: 0.3241, lr: 0.002210, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2718 samples/sec | ETA 01:34:05 2021-05-09 23:06:24 [INFO] [TRAIN] epoch: 89, iter: 32830/40000, loss: 0.2028, lr: 0.002208, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2714 samples/sec | ETA 01:33:59 2021-05-09 23:06:32 [INFO] [TRAIN] epoch: 89, iter: 32840/40000, loss: 0.2665, lr: 0.002205, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2725 samples/sec | ETA 01:33:46 2021-05-09 23:06:40 [INFO] [TRAIN] epoch: 89, iter: 32850/40000, loss: 0.2682, lr: 0.002202, batch_cost: 0.7865, reader_cost: 0.00017, ips: 1.2715 samples/sec | ETA 01:33:43 2021-05-09 23:06:48 [INFO] [TRAIN] epoch: 89, iter: 32860/40000, loss: 0.1188, lr: 0.002200, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 01:33:32 2021-05-09 23:06:56 [INFO] [TRAIN] epoch: 89, iter: 32870/40000, loss: 0.2681, lr: 0.002197, batch_cost: 0.7875, reader_cost: 0.00016, ips: 1.2699 samples/sec | ETA 01:33:34 2021-05-09 23:07:04 [INFO] [TRAIN] epoch: 89, iter: 32880/40000, loss: 0.3415, lr: 0.002194, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2717 samples/sec | ETA 01:33:18 2021-05-09 23:07:12 [INFO] [TRAIN] epoch: 89, iter: 32890/40000, loss: 0.2222, lr: 0.002192, batch_cost: 0.7897, reader_cost: 0.00016, ips: 1.2663 samples/sec | ETA 01:33:34 2021-05-09 23:07:20 [INFO] [TRAIN] epoch: 89, iter: 32900/40000, loss: 0.2270, lr: 0.002189, batch_cost: 0.7868, reader_cost: 0.00017, ips: 1.2710 samples/sec | ETA 01:33:06 2021-05-09 23:07:27 [INFO] [TRAIN] epoch: 89, iter: 32910/40000, loss: 0.1510, lr: 0.002186, batch_cost: 0.7868, reader_cost: 0.00018, ips: 1.2710 samples/sec | ETA 01:32:58 2021-05-09 23:07:35 [INFO] [TRAIN] epoch: 89, iter: 32920/40000, loss: 0.2821, lr: 0.002184, batch_cost: 0.7872, reader_cost: 0.00017, ips: 1.2703 samples/sec | ETA 01:32:53 2021-05-09 23:07:43 [INFO] [TRAIN] epoch: 89, iter: 32930/40000, loss: 0.1332, lr: 0.002181, batch_cost: 0.7869, reader_cost: 0.00017, ips: 1.2709 samples/sec | ETA 01:32:43 2021-05-09 23:07:51 [INFO] [TRAIN] epoch: 89, iter: 32940/40000, loss: 0.2159, lr: 0.002179, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2728 samples/sec | ETA 01:32:26 2021-05-09 23:07:59 [INFO] [TRAIN] epoch: 89, iter: 32950/40000, loss: 0.2757, lr: 0.002176, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 01:32:17 2021-05-09 23:08:07 [INFO] [TRAIN] epoch: 89, iter: 32960/40000, loss: 0.1477, lr: 0.002173, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2733 samples/sec | ETA 01:32:08 2021-05-09 23:08:15 [INFO] [TRAIN] epoch: 89, iter: 32970/40000, loss: 0.3548, lr: 0.002171, batch_cost: 0.7846, reader_cost: 0.00016, ips: 1.2745 samples/sec | ETA 01:31:55 2021-05-09 23:08:22 [INFO] [TRAIN] epoch: 89, iter: 32980/40000, loss: 0.1816, lr: 0.002168, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2717 samples/sec | ETA 01:32:00 2021-05-09 23:08:30 [INFO] [TRAIN] epoch: 89, iter: 32990/40000, loss: 0.2721, lr: 0.002165, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2728 samples/sec | ETA 01:31:47 2021-05-09 23:08:38 [INFO] [TRAIN] epoch: 89, iter: 33000/40000, loss: 0.0956, lr: 0.002163, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 01:31:42 2021-05-09 23:08:38 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-09 23:12:10 [INFO] [EVAL] #Images: 500 mIoU: 0.7665 Acc: 0.9566 Kappa: 0.9437 2021-05-09 23:12:10 [INFO] [EVAL] Class IoU: [0.9795 0.839 0.9212 0.6232 0.6181 0.4954 0.6272 0.731 0.9154 0.6575 0.9419 0.7799 0.5719 0.9428 0.8415 0.893 0.8056 0.6432 0.7369] 2021-05-09 23:12:10 [INFO] [EVAL] Class Acc: [0.9921 0.9022 0.9543 0.8476 0.7621 0.7689 0.8116 0.8682 0.9458 0.8356 0.9644 0.8639 0.8042 0.9649 0.9512 0.965 0.8892 0.8247 0.8348] 2021-05-09 23:12:37 [INFO] [EVAL] The model with the best validation mIoU (0.7695) was saved at iter 30000. 2021-05-09 23:12:45 [INFO] [TRAIN] epoch: 89, iter: 33010/40000, loss: 0.2913, lr: 0.002160, batch_cost: 0.7840, reader_cost: 0.00026, ips: 1.2755 samples/sec | ETA 01:31:20 2021-05-09 23:12:53 [INFO] [TRAIN] epoch: 89, iter: 33020/40000, loss: 0.4118, lr: 0.002157, batch_cost: 0.7847, reader_cost: 0.00017, ips: 1.2744 samples/sec | ETA 01:31:16 2021-05-09 23:13:01 [INFO] [TRAIN] epoch: 89, iter: 33030/40000, loss: 0.2930, lr: 0.002155, batch_cost: 0.7850, reader_cost: 0.00016, ips: 1.2738 samples/sec | ETA 01:31:11 2021-05-09 23:13:09 [INFO] [TRAIN] epoch: 89, iter: 33040/40000, loss: 0.4930, lr: 0.002152, batch_cost: 0.7845, reader_cost: 0.00016, ips: 1.2747 samples/sec | ETA 01:31:00 2021-05-09 23:13:17 [INFO] [TRAIN] epoch: 89, iter: 33050/40000, loss: 0.3486, lr: 0.002149, batch_cost: 0.7874, reader_cost: 0.00016, ips: 1.2700 samples/sec | ETA 01:31:12 2021-05-09 23:13:25 [INFO] [TRAIN] epoch: 89, iter: 33060/40000, loss: 0.3780, lr: 0.002147, batch_cost: 0.7858, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 01:30:53 2021-05-09 23:13:33 [INFO] [TRAIN] epoch: 89, iter: 33070/40000, loss: 0.1655, lr: 0.002144, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2716 samples/sec | ETA 01:30:49 2021-05-09 23:13:41 [INFO] [TRAIN] epoch: 89, iter: 33080/40000, loss: 0.2815, lr: 0.002141, batch_cost: 0.7844, reader_cost: 0.00017, ips: 1.2748 samples/sec | ETA 01:30:28 2021-05-09 23:13:49 [INFO] [TRAIN] epoch: 89, iter: 33090/40000, loss: 0.2231, lr: 0.002139, batch_cost: 0.7859, reader_cost: 0.00018, ips: 1.2724 samples/sec | ETA 01:30:30 2021-05-09 23:13:56 [INFO] [TRAIN] epoch: 89, iter: 33100/40000, loss: 0.2335, lr: 0.002136, batch_cost: 0.7861, reader_cost: 0.00014, ips: 1.2721 samples/sec | ETA 01:30:24 2021-05-09 23:14:08 [INFO] [TRAIN] epoch: 90, iter: 33110/40000, loss: 0.2899, lr: 0.002133, batch_cost: 1.1058, reader_cost: 0.29163, ips: 0.9043 samples/sec | ETA 02:06:58 2021-05-09 23:14:16 [INFO] [TRAIN] epoch: 90, iter: 33120/40000, loss: 0.3151, lr: 0.002131, batch_cost: 0.8024, reader_cost: 0.00032, ips: 1.2463 samples/sec | ETA 01:32:00 2021-05-09 23:14:23 [INFO] [TRAIN] epoch: 90, iter: 33130/40000, loss: 0.4915, lr: 0.002128, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2734 samples/sec | ETA 01:29:54 2021-05-09 23:14:31 [INFO] [TRAIN] epoch: 90, iter: 33140/40000, loss: 0.1846, lr: 0.002125, batch_cost: 0.7859, reader_cost: 0.00017, ips: 1.2724 samples/sec | ETA 01:29:51 2021-05-09 23:14:39 [INFO] [TRAIN] epoch: 90, iter: 33150/40000, loss: 0.3966, lr: 0.002123, batch_cost: 0.7866, reader_cost: 0.00017, ips: 1.2713 samples/sec | ETA 01:29:48 2021-05-09 23:14:47 [INFO] [TRAIN] epoch: 90, iter: 33160/40000, loss: 0.4421, lr: 0.002120, batch_cost: 0.7868, reader_cost: 0.00018, ips: 1.2709 samples/sec | ETA 01:29:42 2021-05-09 23:14:55 [INFO] [TRAIN] epoch: 90, iter: 33170/40000, loss: 0.2066, lr: 0.002118, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2715 samples/sec | ETA 01:29:31 2021-05-09 23:15:03 [INFO] [TRAIN] epoch: 90, iter: 33180/40000, loss: 0.1308, lr: 0.002115, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2720 samples/sec | ETA 01:29:21 2021-05-09 23:15:11 [INFO] [TRAIN] epoch: 90, iter: 33190/40000, loss: 0.2240, lr: 0.002112, batch_cost: 0.7842, reader_cost: 0.00015, ips: 1.2753 samples/sec | ETA 01:29:00 2021-05-09 23:15:18 [INFO] [TRAIN] epoch: 90, iter: 33200/40000, loss: 0.2540, lr: 0.002110, batch_cost: 0.7872, reader_cost: 0.00015, ips: 1.2703 samples/sec | ETA 01:29:13 2021-05-09 23:15:26 [INFO] [TRAIN] epoch: 90, iter: 33210/40000, loss: 0.1738, lr: 0.002107, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2733 samples/sec | ETA 01:28:52 2021-05-09 23:15:34 [INFO] [TRAIN] epoch: 90, iter: 33220/40000, loss: 0.3224, lr: 0.002104, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2719 samples/sec | ETA 01:28:50 2021-05-09 23:15:42 [INFO] [TRAIN] epoch: 90, iter: 33230/40000, loss: 0.1770, lr: 0.002102, batch_cost: 0.7858, reader_cost: 0.00017, ips: 1.2726 samples/sec | ETA 01:28:39 2021-05-09 23:15:50 [INFO] [TRAIN] epoch: 90, iter: 33240/40000, loss: 0.2543, lr: 0.002099, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 01:28:29 2021-05-09 23:15:58 [INFO] [TRAIN] epoch: 90, iter: 33250/40000, loss: 0.2830, lr: 0.002096, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2723 samples/sec | ETA 01:28:25 2021-05-09 23:16:06 [INFO] [TRAIN] epoch: 90, iter: 33260/40000, loss: 0.3028, lr: 0.002094, batch_cost: 0.7862, reader_cost: 0.00017, ips: 1.2720 samples/sec | ETA 01:28:18 2021-05-09 23:16:13 [INFO] [TRAIN] epoch: 90, iter: 33270/40000, loss: 0.1845, lr: 0.002091, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2721 samples/sec | ETA 01:28:10 2021-05-09 23:16:21 [INFO] [TRAIN] epoch: 90, iter: 33280/40000, loss: 0.2513, lr: 0.002088, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2725 samples/sec | ETA 01:28:00 2021-05-09 23:16:29 [INFO] [TRAIN] epoch: 90, iter: 33290/40000, loss: 0.3032, lr: 0.002086, batch_cost: 0.7867, reader_cost: 0.00015, ips: 1.2711 samples/sec | ETA 01:27:59 2021-05-09 23:16:37 [INFO] [TRAIN] epoch: 90, iter: 33300/40000, loss: 0.1243, lr: 0.002083, batch_cost: 0.7849, reader_cost: 0.00016, ips: 1.2741 samples/sec | ETA 01:27:38 2021-05-09 23:16:45 [INFO] [TRAIN] epoch: 90, iter: 33310/40000, loss: 0.1716, lr: 0.002080, batch_cost: 0.7865, reader_cost: 0.00017, ips: 1.2714 samples/sec | ETA 01:27:41 2021-05-09 23:16:53 [INFO] [TRAIN] epoch: 90, iter: 33320/40000, loss: 0.3470, lr: 0.002078, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2706 samples/sec | ETA 01:27:37 2021-05-09 23:17:01 [INFO] [TRAIN] epoch: 90, iter: 33330/40000, loss: 0.1019, lr: 0.002075, batch_cost: 0.7845, reader_cost: 0.00016, ips: 1.2747 samples/sec | ETA 01:27:12 2021-05-09 23:17:08 [INFO] [TRAIN] epoch: 90, iter: 33340/40000, loss: 0.2775, lr: 0.002072, batch_cost: 0.7849, reader_cost: 0.00016, ips: 1.2740 samples/sec | ETA 01:27:07 2021-05-09 23:17:16 [INFO] [TRAIN] epoch: 90, iter: 33350/40000, loss: 0.1925, lr: 0.002070, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2717 samples/sec | ETA 01:27:09 2021-05-09 23:17:24 [INFO] [TRAIN] epoch: 90, iter: 33360/40000, loss: 0.2411, lr: 0.002067, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 01:26:57 2021-05-09 23:17:32 [INFO] [TRAIN] epoch: 90, iter: 33370/40000, loss: 0.1616, lr: 0.002064, batch_cost: 0.7872, reader_cost: 0.00017, ips: 1.2703 samples/sec | ETA 01:26:59 2021-05-09 23:17:40 [INFO] [TRAIN] epoch: 90, iter: 33380/40000, loss: 0.4901, lr: 0.002062, batch_cost: 0.7853, reader_cost: 0.00018, ips: 1.2734 samples/sec | ETA 01:26:38 2021-05-09 23:17:48 [INFO] [TRAIN] epoch: 90, iter: 33390/40000, loss: 0.3668, lr: 0.002059, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2736 samples/sec | ETA 01:26:29 2021-05-09 23:17:56 [INFO] [TRAIN] epoch: 90, iter: 33400/40000, loss: 0.3075, lr: 0.002056, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2719 samples/sec | ETA 01:26:29 2021-05-09 23:18:04 [INFO] [TRAIN] epoch: 90, iter: 33410/40000, loss: 0.5045, lr: 0.002054, batch_cost: 0.7860, reader_cost: 0.00017, ips: 1.2723 samples/sec | ETA 01:26:19 2021-05-09 23:18:11 [INFO] [TRAIN] epoch: 90, iter: 33420/40000, loss: 0.4763, lr: 0.002051, batch_cost: 0.7845, reader_cost: 0.00016, ips: 1.2747 samples/sec | ETA 01:26:02 2021-05-09 23:18:19 [INFO] [TRAIN] epoch: 90, iter: 33430/40000, loss: 0.4186, lr: 0.002048, batch_cost: 0.7846, reader_cost: 0.00015, ips: 1.2746 samples/sec | ETA 01:25:54 2021-05-09 23:18:27 [INFO] [TRAIN] epoch: 90, iter: 33440/40000, loss: 0.0962, lr: 0.002046, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2714 samples/sec | ETA 01:25:59 2021-05-09 23:18:35 [INFO] [TRAIN] epoch: 90, iter: 33450/40000, loss: 0.2781, lr: 0.002043, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2703 samples/sec | ETA 01:25:56 2021-05-09 23:18:43 [INFO] [TRAIN] epoch: 90, iter: 33460/40000, loss: 0.2716, lr: 0.002040, batch_cost: 0.7857, reader_cost: 0.00017, ips: 1.2727 samples/sec | ETA 01:25:38 2021-05-09 23:18:51 [INFO] [TRAIN] epoch: 90, iter: 33470/40000, loss: 0.3282, lr: 0.002038, batch_cost: 0.7848, reader_cost: 0.00016, ips: 1.2742 samples/sec | ETA 01:25:24 2021-05-09 23:18:59 [INFO] [TRAIN] epoch: 90, iter: 33480/40000, loss: 0.3441, lr: 0.002035, batch_cost: 0.7846, reader_cost: 0.00011, ips: 1.2745 samples/sec | ETA 01:25:15 2021-05-09 23:19:09 [INFO] [TRAIN] epoch: 91, iter: 33490/40000, loss: 0.1754, lr: 0.002032, batch_cost: 1.0925, reader_cost: 0.27812, ips: 0.9154 samples/sec | ETA 01:58:31 2021-05-09 23:19:17 [INFO] [TRAIN] epoch: 91, iter: 33500/40000, loss: 0.4272, lr: 0.002030, batch_cost: 0.7920, reader_cost: 0.00032, ips: 1.2627 samples/sec | ETA 01:25:47 2021-05-09 23:19:25 [INFO] [TRAIN] epoch: 91, iter: 33510/40000, loss: 0.2348, lr: 0.002027, batch_cost: 0.7864, reader_cost: 0.00017, ips: 1.2717 samples/sec | ETA 01:25:03 2021-05-09 23:19:33 [INFO] [TRAIN] epoch: 91, iter: 33520/40000, loss: 0.3472, lr: 0.002024, batch_cost: 0.7862, reader_cost: 0.00017, ips: 1.2720 samples/sec | ETA 01:24:54 2021-05-09 23:19:41 [INFO] [TRAIN] epoch: 91, iter: 33530/40000, loss: 0.4542, lr: 0.002022, batch_cost: 0.7858, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 01:24:43 2021-05-09 23:19:49 [INFO] [TRAIN] epoch: 91, iter: 33540/40000, loss: 0.2144, lr: 0.002019, batch_cost: 0.7866, reader_cost: 0.00014, ips: 1.2713 samples/sec | ETA 01:24:41 2021-05-09 23:19:57 [INFO] [TRAIN] epoch: 91, iter: 33550/40000, loss: 0.1664, lr: 0.002016, batch_cost: 0.7873, reader_cost: 0.00015, ips: 1.2702 samples/sec | ETA 01:24:37 2021-05-09 23:20:05 [INFO] [TRAIN] epoch: 91, iter: 33560/40000, loss: 0.1712, lr: 0.002014, batch_cost: 0.7849, reader_cost: 0.00015, ips: 1.2740 samples/sec | ETA 01:24:14 2021-05-09 23:20:12 [INFO] [TRAIN] epoch: 91, iter: 33570/40000, loss: 0.3269, lr: 0.002011, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2713 samples/sec | ETA 01:24:17 2021-05-09 23:20:20 [INFO] [TRAIN] epoch: 91, iter: 33580/40000, loss: 0.1563, lr: 0.002008, batch_cost: 0.7842, reader_cost: 0.00015, ips: 1.2751 samples/sec | ETA 01:23:54 2021-05-09 23:20:28 [INFO] [TRAIN] epoch: 91, iter: 33590/40000, loss: 0.2690, lr: 0.002006, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 01:23:54 2021-05-09 23:20:36 [INFO] [TRAIN] epoch: 91, iter: 33600/40000, loss: 0.1116, lr: 0.002003, batch_cost: 0.7840, reader_cost: 0.00018, ips: 1.2755 samples/sec | ETA 01:23:37 2021-05-09 23:20:44 [INFO] [TRAIN] epoch: 91, iter: 33610/40000, loss: 0.2473, lr: 0.002000, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2711 samples/sec | ETA 01:23:47 2021-05-09 23:20:52 [INFO] [TRAIN] epoch: 91, iter: 33620/40000, loss: 0.3403, lr: 0.001997, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2720 samples/sec | ETA 01:23:35 2021-05-09 23:21:00 [INFO] [TRAIN] epoch: 91, iter: 33630/40000, loss: 0.2906, lr: 0.001995, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2721 samples/sec | ETA 01:23:27 2021-05-09 23:21:07 [INFO] [TRAIN] epoch: 91, iter: 33640/40000, loss: 0.2411, lr: 0.001992, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 01:23:18 2021-05-09 23:21:15 [INFO] [TRAIN] epoch: 91, iter: 33650/40000, loss: 0.2426, lr: 0.001989, batch_cost: 0.7851, reader_cost: 0.00015, ips: 1.2737 samples/sec | ETA 01:23:05 2021-05-09 23:21:23 [INFO] [TRAIN] epoch: 91, iter: 33660/40000, loss: 0.2455, lr: 0.001987, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2729 samples/sec | ETA 01:23:00 2021-05-09 23:21:31 [INFO] [TRAIN] epoch: 91, iter: 33670/40000, loss: 0.1354, lr: 0.001984, batch_cost: 0.7851, reader_cost: 0.00016, ips: 1.2737 samples/sec | ETA 01:22:49 2021-05-09 23:21:39 [INFO] [TRAIN] epoch: 91, iter: 33680/40000, loss: 0.0887, lr: 0.001981, batch_cost: 0.7855, reader_cost: 0.00017, ips: 1.2731 samples/sec | ETA 01:22:44 2021-05-09 23:21:47 [INFO] [TRAIN] epoch: 91, iter: 33690/40000, loss: 0.2506, lr: 0.001979, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2735 samples/sec | ETA 01:22:34 2021-05-09 23:21:55 [INFO] [TRAIN] epoch: 91, iter: 33700/40000, loss: 0.2504, lr: 0.001976, batch_cost: 0.7854, reader_cost: 0.00017, ips: 1.2732 samples/sec | ETA 01:22:28 2021-05-09 23:22:02 [INFO] [TRAIN] epoch: 91, iter: 33710/40000, loss: 0.1975, lr: 0.001973, batch_cost: 0.7866, reader_cost: 0.00017, ips: 1.2713 samples/sec | ETA 01:22:27 2021-05-09 23:22:10 [INFO] [TRAIN] epoch: 91, iter: 33720/40000, loss: 0.2176, lr: 0.001971, batch_cost: 0.7846, reader_cost: 0.00015, ips: 1.2746 samples/sec | ETA 01:22:07 2021-05-09 23:22:18 [INFO] [TRAIN] epoch: 91, iter: 33730/40000, loss: 0.1949, lr: 0.001968, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 01:22:10 2021-05-09 23:22:26 [INFO] [TRAIN] epoch: 91, iter: 33740/40000, loss: 0.3280, lr: 0.001965, batch_cost: 0.7842, reader_cost: 0.00016, ips: 1.2752 samples/sec | ETA 01:21:49 2021-05-09 23:22:34 [INFO] [TRAIN] epoch: 91, iter: 33750/40000, loss: 0.2367, lr: 0.001963, batch_cost: 0.7851, reader_cost: 0.00016, ips: 1.2737 samples/sec | ETA 01:21:46 2021-05-09 23:22:42 [INFO] [TRAIN] epoch: 91, iter: 33760/40000, loss: 0.4101, lr: 0.001960, batch_cost: 0.7850, reader_cost: 0.00017, ips: 1.2738 samples/sec | ETA 01:21:38 2021-05-09 23:22:49 [INFO] [TRAIN] epoch: 91, iter: 33770/40000, loss: 0.5094, lr: 0.001957, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2735 samples/sec | ETA 01:21:32 2021-05-09 23:22:57 [INFO] [TRAIN] epoch: 91, iter: 33780/40000, loss: 0.4146, lr: 0.001955, batch_cost: 0.7848, reader_cost: 0.00016, ips: 1.2743 samples/sec | ETA 01:21:21 2021-05-09 23:23:05 [INFO] [TRAIN] epoch: 91, iter: 33790/40000, loss: 0.5233, lr: 0.001952, batch_cost: 0.7857, reader_cost: 0.00017, ips: 1.2728 samples/sec | ETA 01:21:18 2021-05-09 23:23:13 [INFO] [TRAIN] epoch: 91, iter: 33800/40000, loss: 0.2234, lr: 0.001949, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2707 samples/sec | ETA 01:21:19 2021-05-09 23:23:21 [INFO] [TRAIN] epoch: 91, iter: 33810/40000, loss: 0.1112, lr: 0.001947, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2727 samples/sec | ETA 01:21:03 2021-05-09 23:23:29 [INFO] [TRAIN] epoch: 91, iter: 33820/40000, loss: 0.2894, lr: 0.001944, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 01:20:55 2021-05-09 23:23:37 [INFO] [TRAIN] epoch: 91, iter: 33830/40000, loss: 0.2268, lr: 0.001941, batch_cost: 0.7853, reader_cost: 0.00015, ips: 1.2733 samples/sec | ETA 01:20:45 2021-05-09 23:23:44 [INFO] [TRAIN] epoch: 91, iter: 33840/40000, loss: 0.3088, lr: 0.001939, batch_cost: 0.7864, reader_cost: 0.00014, ips: 1.2717 samples/sec | ETA 01:20:43 2021-05-09 23:23:52 [INFO] [TRAIN] epoch: 91, iter: 33850/40000, loss: 0.3355, lr: 0.001936, batch_cost: 0.7851, reader_cost: 0.00029, ips: 1.2738 samples/sec | ETA 01:20:28 2021-05-09 23:24:04 [INFO] [TRAIN] epoch: 92, iter: 33860/40000, loss: 0.2752, lr: 0.001933, batch_cost: 1.1173, reader_cost: 0.22347, ips: 0.8950 samples/sec | ETA 01:54:20 2021-05-09 23:24:11 [INFO] [TRAIN] epoch: 92, iter: 33870/40000, loss: 0.3641, lr: 0.001930, batch_cost: 0.7938, reader_cost: 0.00033, ips: 1.2598 samples/sec | ETA 01:21:05 2021-05-09 23:24:19 [INFO] [TRAIN] epoch: 92, iter: 33880/40000, loss: 0.2246, lr: 0.001928, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2717 samples/sec | ETA 01:20:12 2021-05-09 23:24:27 [INFO] [TRAIN] epoch: 92, iter: 33890/40000, loss: 0.2679, lr: 0.001925, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2716 samples/sec | ETA 01:20:04 2021-05-09 23:24:35 [INFO] [TRAIN] epoch: 92, iter: 33900/40000, loss: 0.4787, lr: 0.001922, batch_cost: 0.7846, reader_cost: 0.00015, ips: 1.2745 samples/sec | ETA 01:19:46 2021-05-09 23:24:43 [INFO] [TRAIN] epoch: 92, iter: 33910/40000, loss: 0.4075, lr: 0.001920, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2736 samples/sec | ETA 01:19:41 2021-05-09 23:24:51 [INFO] [TRAIN] epoch: 92, iter: 33920/40000, loss: 0.2906, lr: 0.001917, batch_cost: 0.7873, reader_cost: 0.00015, ips: 1.2701 samples/sec | ETA 01:19:46 2021-05-09 23:24:59 [INFO] [TRAIN] epoch: 92, iter: 33930/40000, loss: 0.2386, lr: 0.001914, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2714 samples/sec | ETA 01:19:34 2021-05-09 23:25:06 [INFO] [TRAIN] epoch: 92, iter: 33940/40000, loss: 0.3544, lr: 0.001912, batch_cost: 0.7858, reader_cost: 0.00017, ips: 1.2727 samples/sec | ETA 01:19:21 2021-05-09 23:25:14 [INFO] [TRAIN] epoch: 92, iter: 33950/40000, loss: 0.1540, lr: 0.001909, batch_cost: 0.7875, reader_cost: 0.00016, ips: 1.2698 samples/sec | ETA 01:19:24 2021-05-09 23:25:22 [INFO] [TRAIN] epoch: 92, iter: 33960/40000, loss: 0.2873, lr: 0.001906, batch_cost: 0.7878, reader_cost: 0.00016, ips: 1.2694 samples/sec | ETA 01:19:18 2021-05-09 23:25:30 [INFO] [TRAIN] epoch: 92, iter: 33970/40000, loss: 0.1307, lr: 0.001904, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2718 samples/sec | ETA 01:19:01 2021-05-09 23:25:38 [INFO] [TRAIN] epoch: 92, iter: 33980/40000, loss: 0.2121, lr: 0.001901, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2719 samples/sec | ETA 01:18:52 2021-05-09 23:25:46 [INFO] [TRAIN] epoch: 92, iter: 33990/40000, loss: 0.2777, lr: 0.001898, batch_cost: 0.7849, reader_cost: 0.00015, ips: 1.2740 samples/sec | ETA 01:18:37 2021-05-09 23:25:54 [INFO] [TRAIN] epoch: 92, iter: 34000/40000, loss: 0.2055, lr: 0.001895, batch_cost: 0.7869, reader_cost: 0.00015, ips: 1.2708 samples/sec | ETA 01:18:41 2021-05-09 23:25:54 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-09 23:29:24 [INFO] [EVAL] #Images: 500 mIoU: 0.7689 Acc: 0.9572 Kappa: 0.9444 2021-05-09 23:29:25 [INFO] [EVAL] Class IoU: [0.9803 0.8424 0.9215 0.6391 0.629 0.508 0.6271 0.727 0.9159 0.6544 0.9416 0.7815 0.583 0.9423 0.8518 0.899 0.8164 0.6182 0.731 ] 2021-05-09 23:29:25 [INFO] [EVAL] Class Acc: [0.9918 0.9101 0.9542 0.8119 0.8206 0.7372 0.8182 0.8746 0.9492 0.8359 0.9624 0.8651 0.7805 0.9636 0.9504 0.9559 0.9035 0.7522 0.8468] 2021-05-09 23:29:53 [INFO] [EVAL] The model with the best validation mIoU (0.7695) was saved at iter 30000. 2021-05-09 23:30:00 [INFO] [TRAIN] epoch: 92, iter: 34010/40000, loss: 0.2345, lr: 0.001893, batch_cost: 0.7810, reader_cost: 0.00067, ips: 1.2804 samples/sec | ETA 01:17:58 2021-05-09 23:30:11 [INFO] [TRAIN] epoch: 92, iter: 34020/40000, loss: 0.1986, lr: 0.001890, batch_cost: 0.7837, reader_cost: 0.00017, ips: 1.2760 samples/sec | ETA 01:18:06 2021-05-09 23:30:19 [INFO] [TRAIN] epoch: 92, iter: 34030/40000, loss: 0.2291, lr: 0.001887, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2733 samples/sec | ETA 01:18:08 2021-05-09 23:30:27 [INFO] [TRAIN] epoch: 92, iter: 34040/40000, loss: 0.2349, lr: 0.001885, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2713 samples/sec | ETA 01:18:08 2021-05-09 23:30:35 [INFO] [TRAIN] epoch: 92, iter: 34050/40000, loss: 0.1212, lr: 0.001882, batch_cost: 0.7886, reader_cost: 0.00017, ips: 1.2681 samples/sec | ETA 01:18:12 2021-05-09 23:30:43 [INFO] [TRAIN] epoch: 92, iter: 34060/40000, loss: 0.2729, lr: 0.001879, batch_cost: 0.7864, reader_cost: 0.00017, ips: 1.2717 samples/sec | ETA 01:17:50 2021-05-09 23:30:51 [INFO] [TRAIN] epoch: 92, iter: 34070/40000, loss: 0.2339, lr: 0.001877, batch_cost: 0.7880, reader_cost: 0.00016, ips: 1.2690 samples/sec | ETA 01:17:53 2021-05-09 23:30:58 [INFO] [TRAIN] epoch: 92, iter: 34080/40000, loss: 0.1556, lr: 0.001874, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2706 samples/sec | ETA 01:17:39 2021-05-09 23:31:06 [INFO] [TRAIN] epoch: 92, iter: 34090/40000, loss: 0.2308, lr: 0.001871, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2708 samples/sec | ETA 01:17:30 2021-05-09 23:31:14 [INFO] [TRAIN] epoch: 92, iter: 34100/40000, loss: 0.1321, lr: 0.001869, batch_cost: 0.7860, reader_cost: 0.00014, ips: 1.2722 samples/sec | ETA 01:17:17 2021-05-09 23:31:22 [INFO] [TRAIN] epoch: 92, iter: 34110/40000, loss: 0.2671, lr: 0.001866, batch_cost: 0.7867, reader_cost: 0.00015, ips: 1.2712 samples/sec | ETA 01:17:13 2021-05-09 23:31:30 [INFO] [TRAIN] epoch: 92, iter: 34120/40000, loss: 0.1798, lr: 0.001863, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 01:17:04 2021-05-09 23:31:38 [INFO] [TRAIN] epoch: 92, iter: 34130/40000, loss: 0.2903, lr: 0.001860, batch_cost: 0.7875, reader_cost: 0.00015, ips: 1.2698 samples/sec | ETA 01:17:02 2021-05-09 23:31:46 [INFO] [TRAIN] epoch: 92, iter: 34140/40000, loss: 0.4056, lr: 0.001858, batch_cost: 0.7874, reader_cost: 0.00015, ips: 1.2701 samples/sec | ETA 01:16:53 2021-05-09 23:31:54 [INFO] [TRAIN] epoch: 92, iter: 34150/40000, loss: 0.3361, lr: 0.001855, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 01:16:40 2021-05-09 23:32:01 [INFO] [TRAIN] epoch: 92, iter: 34160/40000, loss: 0.4277, lr: 0.001852, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2719 samples/sec | ETA 01:16:31 2021-05-09 23:32:09 [INFO] [TRAIN] epoch: 92, iter: 34170/40000, loss: 0.3596, lr: 0.001850, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 01:16:22 2021-05-09 23:32:17 [INFO] [TRAIN] epoch: 92, iter: 34180/40000, loss: 0.1576, lr: 0.001847, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2716 samples/sec | ETA 01:16:16 2021-05-09 23:32:25 [INFO] [TRAIN] epoch: 92, iter: 34190/40000, loss: 0.1972, lr: 0.001844, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2711 samples/sec | ETA 01:16:10 2021-05-09 23:32:33 [INFO] [TRAIN] epoch: 92, iter: 34200/40000, loss: 0.1981, lr: 0.001842, batch_cost: 0.7877, reader_cost: 0.00016, ips: 1.2695 samples/sec | ETA 01:16:08 2021-05-09 23:32:41 [INFO] [TRAIN] epoch: 92, iter: 34210/40000, loss: 0.3125, lr: 0.001839, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 01:15:49 2021-05-09 23:32:49 [INFO] [TRAIN] epoch: 92, iter: 34220/40000, loss: 0.2411, lr: 0.001836, batch_cost: 0.7846, reader_cost: 0.00010, ips: 1.2745 samples/sec | ETA 01:15:35 2021-05-09 23:33:00 [INFO] [TRAIN] epoch: 93, iter: 34230/40000, loss: 0.3520, lr: 0.001833, batch_cost: 1.0998, reader_cost: 0.27487, ips: 0.9092 samples/sec | ETA 01:45:46 2021-05-09 23:33:08 [INFO] [TRAIN] epoch: 93, iter: 34240/40000, loss: 0.2991, lr: 0.001831, batch_cost: 0.7959, reader_cost: 0.00035, ips: 1.2564 samples/sec | ETA 01:16:24 2021-05-09 23:33:15 [INFO] [TRAIN] epoch: 93, iter: 34250/40000, loss: 0.3376, lr: 0.001828, batch_cost: 0.7873, reader_cost: 0.00017, ips: 1.2702 samples/sec | ETA 01:15:27 2021-05-09 23:33:23 [INFO] [TRAIN] epoch: 93, iter: 34260/40000, loss: 0.3334, lr: 0.001825, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2720 samples/sec | ETA 01:15:12 2021-05-09 23:33:31 [INFO] [TRAIN] epoch: 93, iter: 34270/40000, loss: 0.4315, lr: 0.001823, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 01:15:07 2021-05-09 23:33:39 [INFO] [TRAIN] epoch: 93, iter: 34280/40000, loss: 0.3005, lr: 0.001820, batch_cost: 0.7861, reader_cost: 0.00017, ips: 1.2721 samples/sec | ETA 01:14:56 2021-05-09 23:33:47 [INFO] [TRAIN] epoch: 93, iter: 34290/40000, loss: 0.2345, lr: 0.001817, batch_cost: 0.7871, reader_cost: 0.00015, ips: 1.2706 samples/sec | ETA 01:14:54 2021-05-09 23:33:55 [INFO] [TRAIN] epoch: 93, iter: 34300/40000, loss: 0.1075, lr: 0.001814, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2714 samples/sec | ETA 01:14:43 2021-05-09 23:34:03 [INFO] [TRAIN] epoch: 93, iter: 34310/40000, loss: 0.2661, lr: 0.001812, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2723 samples/sec | ETA 01:14:32 2021-05-09 23:34:10 [INFO] [TRAIN] epoch: 93, iter: 34320/40000, loss: 0.2907, lr: 0.001809, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 01:14:27 2021-05-09 23:34:18 [INFO] [TRAIN] epoch: 93, iter: 34330/40000, loss: 0.2710, lr: 0.001806, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2721 samples/sec | ETA 01:14:17 2021-05-09 23:34:26 [INFO] [TRAIN] epoch: 93, iter: 34340/40000, loss: 0.2689, lr: 0.001804, batch_cost: 0.7846, reader_cost: 0.00016, ips: 1.2745 samples/sec | ETA 01:14:00 2021-05-09 23:34:34 [INFO] [TRAIN] epoch: 93, iter: 34350/40000, loss: 0.1792, lr: 0.001801, batch_cost: 0.7848, reader_cost: 0.00016, ips: 1.2742 samples/sec | ETA 01:13:54 2021-05-09 23:34:42 [INFO] [TRAIN] epoch: 93, iter: 34360/40000, loss: 0.2801, lr: 0.001798, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2725 samples/sec | ETA 01:13:52 2021-05-09 23:34:50 [INFO] [TRAIN] epoch: 93, iter: 34370/40000, loss: 0.2238, lr: 0.001796, batch_cost: 0.7847, reader_cost: 0.00015, ips: 1.2745 samples/sec | ETA 01:13:37 2021-05-09 23:34:58 [INFO] [TRAIN] epoch: 93, iter: 34380/40000, loss: 0.2747, lr: 0.001793, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2724 samples/sec | ETA 01:13:36 2021-05-09 23:35:05 [INFO] [TRAIN] epoch: 93, iter: 34390/40000, loss: 0.1593, lr: 0.001790, batch_cost: 0.7843, reader_cost: 0.00015, ips: 1.2750 samples/sec | ETA 01:13:20 2021-05-09 23:35:13 [INFO] [TRAIN] epoch: 93, iter: 34400/40000, loss: 0.1642, lr: 0.001787, batch_cost: 0.7865, reader_cost: 0.00014, ips: 1.2715 samples/sec | ETA 01:13:24 2021-05-09 23:35:21 [INFO] [TRAIN] epoch: 93, iter: 34410/40000, loss: 0.2885, lr: 0.001785, batch_cost: 0.7853, reader_cost: 0.00015, ips: 1.2734 samples/sec | ETA 01:13:09 2021-05-09 23:35:29 [INFO] [TRAIN] epoch: 93, iter: 34420/40000, loss: 0.1034, lr: 0.001782, batch_cost: 0.7842, reader_cost: 0.00015, ips: 1.2752 samples/sec | ETA 01:12:55 2021-05-09 23:35:37 [INFO] [TRAIN] epoch: 93, iter: 34430/40000, loss: 0.1858, lr: 0.001779, batch_cost: 0.7840, reader_cost: 0.00016, ips: 1.2755 samples/sec | ETA 01:12:46 2021-05-09 23:35:45 [INFO] [TRAIN] epoch: 93, iter: 34440/40000, loss: 0.3285, lr: 0.001777, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2722 samples/sec | ETA 01:12:50 2021-05-09 23:35:53 [INFO] [TRAIN] epoch: 93, iter: 34450/40000, loss: 0.1941, lr: 0.001774, batch_cost: 0.7849, reader_cost: 0.00016, ips: 1.2740 samples/sec | ETA 01:12:36 2021-05-09 23:36:00 [INFO] [TRAIN] epoch: 93, iter: 34460/40000, loss: 0.2228, lr: 0.001771, batch_cost: 0.7853, reader_cost: 0.00017, ips: 1.2734 samples/sec | ETA 01:12:30 2021-05-09 23:36:08 [INFO] [TRAIN] epoch: 93, iter: 34470/40000, loss: 0.1313, lr: 0.001768, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2732 samples/sec | ETA 01:12:23 2021-05-09 23:36:16 [INFO] [TRAIN] epoch: 93, iter: 34480/40000, loss: 0.2436, lr: 0.001766, batch_cost: 0.7852, reader_cost: 0.00017, ips: 1.2736 samples/sec | ETA 01:12:14 2021-05-09 23:36:24 [INFO] [TRAIN] epoch: 93, iter: 34490/40000, loss: 0.1322, lr: 0.001763, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2709 samples/sec | ETA 01:12:15 2021-05-09 23:36:32 [INFO] [TRAIN] epoch: 93, iter: 34500/40000, loss: 0.3061, lr: 0.001760, batch_cost: 0.7867, reader_cost: 0.00015, ips: 1.2712 samples/sec | ETA 01:12:06 2021-05-09 23:36:40 [INFO] [TRAIN] epoch: 93, iter: 34510/40000, loss: 0.3551, lr: 0.001758, batch_cost: 0.7853, reader_cost: 0.00017, ips: 1.2734 samples/sec | ETA 01:11:51 2021-05-09 23:36:48 [INFO] [TRAIN] epoch: 93, iter: 34520/40000, loss: 0.2869, lr: 0.001755, batch_cost: 0.7861, reader_cost: 0.00017, ips: 1.2721 samples/sec | ETA 01:11:47 2021-05-09 23:36:55 [INFO] [TRAIN] epoch: 93, iter: 34530/40000, loss: 0.4749, lr: 0.001752, batch_cost: 0.7850, reader_cost: 0.00016, ips: 1.2739 samples/sec | ETA 01:11:33 2021-05-09 23:37:03 [INFO] [TRAIN] epoch: 93, iter: 34540/40000, loss: 0.4304, lr: 0.001749, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2736 samples/sec | ETA 01:11:27 2021-05-09 23:37:11 [INFO] [TRAIN] epoch: 93, iter: 34550/40000, loss: 0.3005, lr: 0.001747, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2720 samples/sec | ETA 01:11:24 2021-05-09 23:37:19 [INFO] [TRAIN] epoch: 93, iter: 34560/40000, loss: 0.1815, lr: 0.001744, batch_cost: 0.7876, reader_cost: 0.00017, ips: 1.2697 samples/sec | ETA 01:11:24 2021-05-09 23:37:27 [INFO] [TRAIN] epoch: 93, iter: 34570/40000, loss: 0.2048, lr: 0.001741, batch_cost: 0.7849, reader_cost: 0.00017, ips: 1.2740 samples/sec | ETA 01:11:02 2021-05-09 23:37:35 [INFO] [TRAIN] epoch: 93, iter: 34580/40000, loss: 0.3255, lr: 0.001739, batch_cost: 0.7874, reader_cost: 0.00016, ips: 1.2700 samples/sec | ETA 01:11:07 2021-05-09 23:37:43 [INFO] [TRAIN] epoch: 93, iter: 34590/40000, loss: 0.3520, lr: 0.001736, batch_cost: 0.7866, reader_cost: 0.00013, ips: 1.2713 samples/sec | ETA 01:10:55 2021-05-09 23:37:54 [INFO] [TRAIN] epoch: 94, iter: 34600/40000, loss: 0.3962, lr: 0.001733, batch_cost: 1.0996, reader_cost: 0.23947, ips: 0.9094 samples/sec | ETA 01:38:57 2021-05-09 23:38:02 [INFO] [TRAIN] epoch: 94, iter: 34610/40000, loss: 0.2267, lr: 0.001730, batch_cost: 0.7977, reader_cost: 0.00033, ips: 1.2536 samples/sec | ETA 01:11:39 2021-05-09 23:38:09 [INFO] [TRAIN] epoch: 94, iter: 34620/40000, loss: 0.4802, lr: 0.001728, batch_cost: 0.7861, reader_cost: 0.00014, ips: 1.2720 samples/sec | ETA 01:10:29 2021-05-09 23:38:17 [INFO] [TRAIN] epoch: 94, iter: 34630/40000, loss: 0.2376, lr: 0.001725, batch_cost: 0.7872, reader_cost: 0.00015, ips: 1.2704 samples/sec | ETA 01:10:27 2021-05-09 23:38:25 [INFO] [TRAIN] epoch: 94, iter: 34640/40000, loss: 0.3833, lr: 0.001722, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2732 samples/sec | ETA 01:10:09 2021-05-09 23:38:33 [INFO] [TRAIN] epoch: 94, iter: 34650/40000, loss: 0.4103, lr: 0.001719, batch_cost: 0.7861, reader_cost: 0.00017, ips: 1.2721 samples/sec | ETA 01:10:05 2021-05-09 23:38:41 [INFO] [TRAIN] epoch: 94, iter: 34660/40000, loss: 0.3073, lr: 0.001717, batch_cost: 0.7872, reader_cost: 0.00017, ips: 1.2704 samples/sec | ETA 01:10:03 2021-05-09 23:38:49 [INFO] [TRAIN] epoch: 94, iter: 34670/40000, loss: 0.1492, lr: 0.001714, batch_cost: 0.7862, reader_cost: 0.00018, ips: 1.2719 samples/sec | ETA 01:09:50 2021-05-09 23:38:57 [INFO] [TRAIN] epoch: 94, iter: 34680/40000, loss: 0.3277, lr: 0.001711, batch_cost: 0.7864, reader_cost: 0.00017, ips: 1.2716 samples/sec | ETA 01:09:43 2021-05-09 23:39:05 [INFO] [TRAIN] epoch: 94, iter: 34690/40000, loss: 0.3205, lr: 0.001709, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2719 samples/sec | ETA 01:09:34 2021-05-09 23:39:12 [INFO] [TRAIN] epoch: 94, iter: 34700/40000, loss: 0.2382, lr: 0.001706, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2718 samples/sec | ETA 01:09:27 2021-05-09 23:39:20 [INFO] [TRAIN] epoch: 94, iter: 34710/40000, loss: 0.3220, lr: 0.001703, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2734 samples/sec | ETA 01:09:14 2021-05-09 23:39:28 [INFO] [TRAIN] epoch: 94, iter: 34720/40000, loss: 0.1526, lr: 0.001700, batch_cost: 0.7859, reader_cost: 0.00017, ips: 1.2724 samples/sec | ETA 01:09:09 2021-05-09 23:39:36 [INFO] [TRAIN] epoch: 94, iter: 34730/40000, loss: 0.2903, lr: 0.001698, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 01:09:02 2021-05-09 23:39:44 [INFO] [TRAIN] epoch: 94, iter: 34740/40000, loss: 0.3363, lr: 0.001695, batch_cost: 0.7875, reader_cost: 0.00015, ips: 1.2699 samples/sec | ETA 01:09:02 2021-05-09 23:39:52 [INFO] [TRAIN] epoch: 94, iter: 34750/40000, loss: 0.2636, lr: 0.001692, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2719 samples/sec | ETA 01:08:47 2021-05-09 23:40:00 [INFO] [TRAIN] epoch: 94, iter: 34760/40000, loss: 0.1960, lr: 0.001689, batch_cost: 0.7858, reader_cost: 0.00015, ips: 1.2726 samples/sec | ETA 01:08:37 2021-05-09 23:40:07 [INFO] [TRAIN] epoch: 94, iter: 34770/40000, loss: 0.1981, lr: 0.001687, batch_cost: 0.7872, reader_cost: 0.00015, ips: 1.2704 samples/sec | ETA 01:08:36 2021-05-09 23:40:15 [INFO] [TRAIN] epoch: 94, iter: 34780/40000, loss: 0.2232, lr: 0.001684, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2711 samples/sec | ETA 01:08:26 2021-05-09 23:40:23 [INFO] [TRAIN] epoch: 94, iter: 34790/40000, loss: 0.1689, lr: 0.001681, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2726 samples/sec | ETA 01:08:14 2021-05-09 23:40:31 [INFO] [TRAIN] epoch: 94, iter: 34800/40000, loss: 0.1867, lr: 0.001679, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2723 samples/sec | ETA 01:08:07 2021-05-09 23:40:39 [INFO] [TRAIN] epoch: 94, iter: 34810/40000, loss: 0.3203, lr: 0.001676, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2733 samples/sec | ETA 01:07:55 2021-05-09 23:40:47 [INFO] [TRAIN] epoch: 94, iter: 34820/40000, loss: 0.1919, lr: 0.001673, batch_cost: 0.7882, reader_cost: 0.00016, ips: 1.2687 samples/sec | ETA 01:08:02 2021-05-09 23:40:55 [INFO] [TRAIN] epoch: 94, iter: 34830/40000, loss: 0.2407, lr: 0.001670, batch_cost: 0.7853, reader_cost: 0.00015, ips: 1.2734 samples/sec | ETA 01:07:39 2021-05-09 23:41:02 [INFO] [TRAIN] epoch: 94, iter: 34840/40000, loss: 0.1526, lr: 0.001668, batch_cost: 0.7841, reader_cost: 0.00014, ips: 1.2753 samples/sec | ETA 01:07:26 2021-05-09 23:41:10 [INFO] [TRAIN] epoch: 94, iter: 34850/40000, loss: 0.2865, lr: 0.001665, batch_cost: 0.7835, reader_cost: 0.00015, ips: 1.2763 samples/sec | ETA 01:07:15 2021-05-09 23:41:18 [INFO] [TRAIN] epoch: 94, iter: 34860/40000, loss: 0.1954, lr: 0.001662, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2722 samples/sec | ETA 01:07:20 2021-05-09 23:41:26 [INFO] [TRAIN] epoch: 94, iter: 34870/40000, loss: 0.3114, lr: 0.001659, batch_cost: 0.7841, reader_cost: 0.00015, ips: 1.2754 samples/sec | ETA 01:07:02 2021-05-09 23:41:34 [INFO] [TRAIN] epoch: 94, iter: 34880/40000, loss: 0.4036, lr: 0.001657, batch_cost: 0.7848, reader_cost: 0.00015, ips: 1.2743 samples/sec | ETA 01:06:57 2021-05-09 23:41:42 [INFO] [TRAIN] epoch: 94, iter: 34890/40000, loss: 0.2540, lr: 0.001654, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2731 samples/sec | ETA 01:06:53 2021-05-09 23:41:50 [INFO] [TRAIN] epoch: 94, iter: 34900/40000, loss: 0.4512, lr: 0.001651, batch_cost: 0.7849, reader_cost: 0.00016, ips: 1.2740 samples/sec | ETA 01:06:43 2021-05-09 23:41:57 [INFO] [TRAIN] epoch: 94, iter: 34910/40000, loss: 0.2413, lr: 0.001648, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2736 samples/sec | ETA 01:06:36 2021-05-09 23:42:05 [INFO] [TRAIN] epoch: 94, iter: 34920/40000, loss: 0.3174, lr: 0.001646, batch_cost: 0.7868, reader_cost: 0.00017, ips: 1.2710 samples/sec | ETA 01:06:36 2021-05-09 23:42:13 [INFO] [TRAIN] epoch: 94, iter: 34930/40000, loss: 0.1127, lr: 0.001643, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2704 samples/sec | ETA 01:06:30 2021-05-09 23:42:21 [INFO] [TRAIN] epoch: 94, iter: 34940/40000, loss: 0.1843, lr: 0.001640, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2717 samples/sec | ETA 01:06:19 2021-05-09 23:42:29 [INFO] [TRAIN] epoch: 94, iter: 34950/40000, loss: 0.2321, lr: 0.001638, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2707 samples/sec | ETA 01:06:14 2021-05-09 23:42:37 [INFO] [TRAIN] epoch: 94, iter: 34960/40000, loss: 0.3543, lr: 0.001635, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2720 samples/sec | ETA 01:06:02 2021-05-09 23:42:48 [INFO] [TRAIN] epoch: 95, iter: 34970/40000, loss: 0.2823, lr: 0.001632, batch_cost: 1.0940, reader_cost: 0.25069, ips: 0.9141 samples/sec | ETA 01:31:42 2021-05-09 23:42:56 [INFO] [TRAIN] epoch: 95, iter: 34980/40000, loss: 0.1632, lr: 0.001629, batch_cost: 0.7995, reader_cost: 0.00036, ips: 1.2508 samples/sec | ETA 01:06:53 2021-05-09 23:43:04 [INFO] [TRAIN] epoch: 95, iter: 34990/40000, loss: 0.5352, lr: 0.001627, batch_cost: 0.7878, reader_cost: 0.00015, ips: 1.2694 samples/sec | ETA 01:05:46 2021-05-09 23:43:11 [INFO] [TRAIN] epoch: 95, iter: 35000/40000, loss: 0.2163, lr: 0.001624, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 01:05:29 2021-05-09 23:43:11 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-09 23:46:43 [INFO] [EVAL] #Images: 500 mIoU: 0.7586 Acc: 0.9570 Kappa: 0.9443 2021-05-09 23:46:43 [INFO] [EVAL] Class IoU: [0.9807 0.8449 0.9212 0.622 0.6306 0.4985 0.627 0.7297 0.9154 0.6558 0.9421 0.7833 0.5968 0.9427 0.8541 0.8535 0.6411 0.6373 0.7363] 2021-05-09 23:46:43 [INFO] [EVAL] Class Acc: [0.9926 0.9059 0.9551 0.8294 0.7953 0.7651 0.813 0.8819 0.9448 0.8424 0.9648 0.8677 0.7531 0.9639 0.9571 0.893 0.9577 0.8024 0.8426] 2021-05-09 23:47:11 [INFO] [EVAL] The model with the best validation mIoU (0.7695) was saved at iter 30000. 2021-05-09 23:47:19 [INFO] [TRAIN] epoch: 95, iter: 35010/40000, loss: 0.3856, lr: 0.001621, batch_cost: 0.7811, reader_cost: 0.00076, ips: 1.2802 samples/sec | ETA 01:04:57 2021-05-09 23:47:27 [INFO] [TRAIN] epoch: 95, iter: 35020/40000, loss: 0.4066, lr: 0.001618, batch_cost: 0.7832, reader_cost: 0.00015, ips: 1.2768 samples/sec | ETA 01:05:00 2021-05-09 23:47:34 [INFO] [TRAIN] epoch: 95, iter: 35030/40000, loss: 0.2328, lr: 0.001616, batch_cost: 0.7848, reader_cost: 0.00016, ips: 1.2742 samples/sec | ETA 01:05:00 2021-05-09 23:47:44 [INFO] [TRAIN] epoch: 95, iter: 35040/40000, loss: 0.1471, lr: 0.001613, batch_cost: 0.7851, reader_cost: 0.00018, ips: 1.2738 samples/sec | ETA 01:04:53 2021-05-09 23:47:52 [INFO] [TRAIN] epoch: 95, iter: 35050/40000, loss: 0.2843, lr: 0.001610, batch_cost: 0.7850, reader_cost: 0.00016, ips: 1.2738 samples/sec | ETA 01:04:45 2021-05-09 23:48:00 [INFO] [TRAIN] epoch: 95, iter: 35060/40000, loss: 0.2252, lr: 0.001607, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2717 samples/sec | ETA 01:04:44 2021-05-09 23:48:08 [INFO] [TRAIN] epoch: 95, iter: 35070/40000, loss: 0.2077, lr: 0.001605, batch_cost: 0.7853, reader_cost: 0.00018, ips: 1.2734 samples/sec | ETA 01:04:31 2021-05-09 23:48:16 [INFO] [TRAIN] epoch: 95, iter: 35080/40000, loss: 0.3404, lr: 0.001602, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2713 samples/sec | ETA 01:04:30 2021-05-09 23:48:23 [INFO] [TRAIN] epoch: 95, iter: 35090/40000, loss: 0.1087, lr: 0.001599, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2715 samples/sec | ETA 01:04:21 2021-05-09 23:48:31 [INFO] [TRAIN] epoch: 95, iter: 35100/40000, loss: 0.2614, lr: 0.001596, batch_cost: 0.7873, reader_cost: 0.00015, ips: 1.2701 samples/sec | ETA 01:04:17 2021-05-09 23:48:39 [INFO] [TRAIN] epoch: 95, iter: 35110/40000, loss: 0.2439, lr: 0.001594, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2707 samples/sec | ETA 01:04:08 2021-05-09 23:48:47 [INFO] [TRAIN] epoch: 95, iter: 35120/40000, loss: 0.2297, lr: 0.001591, batch_cost: 0.7876, reader_cost: 0.00016, ips: 1.2697 samples/sec | ETA 01:04:03 2021-05-09 23:48:55 [INFO] [TRAIN] epoch: 95, iter: 35130/40000, loss: 0.2760, lr: 0.001588, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2732 samples/sec | ETA 01:03:44 2021-05-09 23:49:03 [INFO] [TRAIN] epoch: 95, iter: 35140/40000, loss: 0.1927, lr: 0.001585, batch_cost: 0.7881, reader_cost: 0.00015, ips: 1.2688 samples/sec | ETA 01:03:50 2021-05-09 23:49:11 [INFO] [TRAIN] epoch: 95, iter: 35150/40000, loss: 0.3284, lr: 0.001583, batch_cost: 0.7854, reader_cost: 0.00015, ips: 1.2732 samples/sec | ETA 01:03:29 2021-05-09 23:49:19 [INFO] [TRAIN] epoch: 95, iter: 35160/40000, loss: 0.1179, lr: 0.001580, batch_cost: 0.7860, reader_cost: 0.00014, ips: 1.2722 samples/sec | ETA 01:03:24 2021-05-09 23:49:26 [INFO] [TRAIN] epoch: 95, iter: 35170/40000, loss: 0.1730, lr: 0.001577, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 01:03:15 2021-05-09 23:49:34 [INFO] [TRAIN] epoch: 95, iter: 35180/40000, loss: 0.2737, lr: 0.001574, batch_cost: 0.7848, reader_cost: 0.00016, ips: 1.2741 samples/sec | ETA 01:03:02 2021-05-09 23:49:42 [INFO] [TRAIN] epoch: 95, iter: 35190/40000, loss: 0.1733, lr: 0.001572, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2720 samples/sec | ETA 01:03:01 2021-05-09 23:49:50 [INFO] [TRAIN] epoch: 95, iter: 35200/40000, loss: 0.2638, lr: 0.001569, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2719 samples/sec | ETA 01:02:53 2021-05-09 23:49:58 [INFO] [TRAIN] epoch: 95, iter: 35210/40000, loss: 0.2355, lr: 0.001566, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2734 samples/sec | ETA 01:02:41 2021-05-09 23:50:06 [INFO] [TRAIN] epoch: 95, iter: 35220/40000, loss: 0.1909, lr: 0.001563, batch_cost: 0.7858, reader_cost: 0.00017, ips: 1.2726 samples/sec | ETA 01:02:36 2021-05-09 23:50:14 [INFO] [TRAIN] epoch: 95, iter: 35230/40000, loss: 0.2118, lr: 0.001561, batch_cost: 0.7858, reader_cost: 0.00014, ips: 1.2725 samples/sec | ETA 01:02:28 2021-05-09 23:50:21 [INFO] [TRAIN] epoch: 95, iter: 35240/40000, loss: 0.2292, lr: 0.001558, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2717 samples/sec | ETA 01:02:23 2021-05-09 23:50:29 [INFO] [TRAIN] epoch: 95, iter: 35250/40000, loss: 0.3636, lr: 0.001555, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2723 samples/sec | ETA 01:02:13 2021-05-09 23:50:37 [INFO] [TRAIN] epoch: 95, iter: 35260/40000, loss: 0.2622, lr: 0.001552, batch_cost: 0.7850, reader_cost: 0.00016, ips: 1.2738 samples/sec | ETA 01:02:01 2021-05-09 23:50:45 [INFO] [TRAIN] epoch: 95, iter: 35270/40000, loss: 0.3423, lr: 0.001550, batch_cost: 0.7847, reader_cost: 0.00015, ips: 1.2744 samples/sec | ETA 01:01:51 2021-05-09 23:50:53 [INFO] [TRAIN] epoch: 95, iter: 35280/40000, loss: 0.4348, lr: 0.001547, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2725 samples/sec | ETA 01:01:49 2021-05-09 23:51:01 [INFO] [TRAIN] epoch: 95, iter: 35290/40000, loss: 0.2299, lr: 0.001544, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2725 samples/sec | ETA 01:01:41 2021-05-09 23:51:09 [INFO] [TRAIN] epoch: 95, iter: 35300/40000, loss: 0.1065, lr: 0.001541, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2722 samples/sec | ETA 01:01:34 2021-05-09 23:51:16 [INFO] [TRAIN] epoch: 95, iter: 35310/40000, loss: 0.3130, lr: 0.001539, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2732 samples/sec | ETA 01:01:23 2021-05-09 23:51:24 [INFO] [TRAIN] epoch: 95, iter: 35320/40000, loss: 0.2614, lr: 0.001536, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 01:01:18 2021-05-09 23:51:32 [INFO] [TRAIN] epoch: 95, iter: 35330/40000, loss: 0.3009, lr: 0.001533, batch_cost: 0.7855, reader_cost: 0.00014, ips: 1.2731 samples/sec | ETA 01:01:08 2021-05-09 23:51:40 [INFO] [TRAIN] epoch: 95, iter: 35340/40000, loss: 0.3892, lr: 0.001530, batch_cost: 0.7834, reader_cost: 0.00009, ips: 1.2764 samples/sec | ETA 01:00:50 2021-05-09 23:51:51 [INFO] [TRAIN] epoch: 96, iter: 35350/40000, loss: 0.2209, lr: 0.001527, batch_cost: 1.1066, reader_cost: 0.31015, ips: 0.9037 samples/sec | ETA 01:25:45 2021-05-09 23:51:59 [INFO] [TRAIN] epoch: 96, iter: 35360/40000, loss: 0.4021, lr: 0.001525, batch_cost: 0.7895, reader_cost: 0.00034, ips: 1.2667 samples/sec | ETA 01:01:03 2021-05-09 23:52:07 [INFO] [TRAIN] epoch: 96, iter: 35370/40000, loss: 0.2820, lr: 0.001522, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2703 samples/sec | ETA 01:00:44 2021-05-09 23:52:15 [INFO] [TRAIN] epoch: 96, iter: 35380/40000, loss: 0.4132, lr: 0.001519, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 01:00:32 2021-05-09 23:52:23 [INFO] [TRAIN] epoch: 96, iter: 35390/40000, loss: 0.4551, lr: 0.001516, batch_cost: 0.7854, reader_cost: 0.00015, ips: 1.2732 samples/sec | ETA 01:00:20 2021-05-09 23:52:30 [INFO] [TRAIN] epoch: 96, iter: 35400/40000, loss: 0.2316, lr: 0.001514, batch_cost: 0.7875, reader_cost: 0.00016, ips: 1.2699 samples/sec | ETA 01:00:22 2021-05-09 23:52:38 [INFO] [TRAIN] epoch: 96, iter: 35410/40000, loss: 0.2297, lr: 0.001511, batch_cost: 0.7870, reader_cost: 0.00017, ips: 1.2707 samples/sec | ETA 01:00:12 2021-05-09 23:52:46 [INFO] [TRAIN] epoch: 96, iter: 35420/40000, loss: 0.1908, lr: 0.001508, batch_cost: 0.7849, reader_cost: 0.00016, ips: 1.2740 samples/sec | ETA 00:59:55 2021-05-09 23:52:54 [INFO] [TRAIN] epoch: 96, iter: 35430/40000, loss: 0.3010, lr: 0.001505, batch_cost: 0.7857, reader_cost: 0.00017, ips: 1.2727 samples/sec | ETA 00:59:50 2021-05-09 23:53:02 [INFO] [TRAIN] epoch: 96, iter: 35440/40000, loss: 0.1595, lr: 0.001503, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 00:59:47 2021-05-09 23:53:10 [INFO] [TRAIN] epoch: 96, iter: 35450/40000, loss: 0.2544, lr: 0.001500, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2708 samples/sec | ETA 00:59:40 2021-05-09 23:53:18 [INFO] [TRAIN] epoch: 96, iter: 35460/40000, loss: 0.1422, lr: 0.001497, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2720 samples/sec | ETA 00:59:29 2021-05-09 23:53:25 [INFO] [TRAIN] epoch: 96, iter: 35470/40000, loss: 0.2861, lr: 0.001494, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2718 samples/sec | ETA 00:59:21 2021-05-09 23:53:33 [INFO] [TRAIN] epoch: 96, iter: 35480/40000, loss: 0.3747, lr: 0.001492, batch_cost: 0.7859, reader_cost: 0.00017, ips: 1.2724 samples/sec | ETA 00:59:12 2021-05-09 23:53:41 [INFO] [TRAIN] epoch: 96, iter: 35490/40000, loss: 0.2629, lr: 0.001489, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2732 samples/sec | ETA 00:59:02 2021-05-09 23:53:49 [INFO] [TRAIN] epoch: 96, iter: 35500/40000, loss: 0.2818, lr: 0.001486, batch_cost: 0.7858, reader_cost: 0.00017, ips: 1.2726 samples/sec | ETA 00:58:56 2021-05-09 23:53:57 [INFO] [TRAIN] epoch: 96, iter: 35510/40000, loss: 0.2199, lr: 0.001483, batch_cost: 0.7872, reader_cost: 0.00017, ips: 1.2704 samples/sec | ETA 00:58:54 2021-05-09 23:54:05 [INFO] [TRAIN] epoch: 96, iter: 35520/40000, loss: 0.1811, lr: 0.001480, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 00:58:41 2021-05-09 23:54:13 [INFO] [TRAIN] epoch: 96, iter: 35530/40000, loss: 0.1377, lr: 0.001478, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2718 samples/sec | ETA 00:58:34 2021-05-09 23:54:20 [INFO] [TRAIN] epoch: 96, iter: 35540/40000, loss: 0.1613, lr: 0.001475, batch_cost: 0.7858, reader_cost: 0.00015, ips: 1.2726 samples/sec | ETA 00:58:24 2021-05-09 23:54:28 [INFO] [TRAIN] epoch: 96, iter: 35550/40000, loss: 0.2855, lr: 0.001472, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2716 samples/sec | ETA 00:58:19 2021-05-09 23:54:36 [INFO] [TRAIN] epoch: 96, iter: 35560/40000, loss: 0.1856, lr: 0.001469, batch_cost: 0.7845, reader_cost: 0.00016, ips: 1.2747 samples/sec | ETA 00:58:03 2021-05-09 23:54:44 [INFO] [TRAIN] epoch: 96, iter: 35570/40000, loss: 0.2087, lr: 0.001467, batch_cost: 0.7836, reader_cost: 0.00015, ips: 1.2761 samples/sec | ETA 00:57:51 2021-05-09 23:54:52 [INFO] [TRAIN] epoch: 96, iter: 35580/40000, loss: 0.2224, lr: 0.001464, batch_cost: 0.7846, reader_cost: 0.00015, ips: 1.2745 samples/sec | ETA 00:57:48 2021-05-09 23:55:00 [INFO] [TRAIN] epoch: 96, iter: 35590/40000, loss: 0.1341, lr: 0.001461, batch_cost: 0.7873, reader_cost: 0.00015, ips: 1.2702 samples/sec | ETA 00:57:51 2021-05-09 23:55:08 [INFO] [TRAIN] epoch: 96, iter: 35600/40000, loss: 0.3099, lr: 0.001458, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2729 samples/sec | ETA 00:57:36 2021-05-09 23:55:15 [INFO] [TRAIN] epoch: 96, iter: 35610/40000, loss: 0.2495, lr: 0.001455, batch_cost: 0.7862, reader_cost: 0.00017, ips: 1.2719 samples/sec | ETA 00:57:31 2021-05-09 23:55:23 [INFO] [TRAIN] epoch: 96, iter: 35620/40000, loss: 0.3550, lr: 0.001453, batch_cost: 0.7848, reader_cost: 0.00015, ips: 1.2742 samples/sec | ETA 00:57:17 2021-05-09 23:55:31 [INFO] [TRAIN] epoch: 96, iter: 35630/40000, loss: 0.4118, lr: 0.001450, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2703 samples/sec | ETA 00:57:20 2021-05-09 23:55:39 [INFO] [TRAIN] epoch: 96, iter: 35640/40000, loss: 0.4114, lr: 0.001447, batch_cost: 0.7855, reader_cost: 0.00017, ips: 1.2731 samples/sec | ETA 00:57:04 2021-05-09 23:55:47 [INFO] [TRAIN] epoch: 96, iter: 35650/40000, loss: 0.5059, lr: 0.001444, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2712 samples/sec | ETA 00:57:01 2021-05-09 23:55:55 [INFO] [TRAIN] epoch: 96, iter: 35660/40000, loss: 0.3210, lr: 0.001442, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2730 samples/sec | ETA 00:56:49 2021-05-09 23:56:03 [INFO] [TRAIN] epoch: 96, iter: 35670/40000, loss: 0.1207, lr: 0.001439, batch_cost: 0.7870, reader_cost: 0.00015, ips: 1.2707 samples/sec | ETA 00:56:47 2021-05-09 23:56:10 [INFO] [TRAIN] epoch: 96, iter: 35680/40000, loss: 0.1902, lr: 0.001436, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 00:56:35 2021-05-09 23:56:18 [INFO] [TRAIN] epoch: 96, iter: 35690/40000, loss: 0.2289, lr: 0.001433, batch_cost: 0.7851, reader_cost: 0.00016, ips: 1.2738 samples/sec | ETA 00:56:23 2021-05-09 23:56:26 [INFO] [TRAIN] epoch: 96, iter: 35700/40000, loss: 0.2935, lr: 0.001430, batch_cost: 0.7838, reader_cost: 0.00015, ips: 1.2759 samples/sec | ETA 00:56:10 2021-05-09 23:56:34 [INFO] [TRAIN] epoch: 96, iter: 35710/40000, loss: 0.3168, lr: 0.001428, batch_cost: 0.7851, reader_cost: 0.00028, ips: 1.2737 samples/sec | ETA 00:56:08 2021-05-09 23:56:45 [INFO] [TRAIN] epoch: 97, iter: 35720/40000, loss: 0.2719, lr: 0.001425, batch_cost: 1.1140, reader_cost: 0.27308, ips: 0.8977 samples/sec | ETA 01:19:27 2021-05-09 23:56:53 [INFO] [TRAIN] epoch: 97, iter: 35730/40000, loss: 0.3895, lr: 0.001422, batch_cost: 0.7943, reader_cost: 0.00034, ips: 1.2590 samples/sec | ETA 00:56:31 2021-05-09 23:57:01 [INFO] [TRAIN] epoch: 97, iter: 35740/40000, loss: 0.4608, lr: 0.001419, batch_cost: 0.7849, reader_cost: 0.00017, ips: 1.2741 samples/sec | ETA 00:55:43 2021-05-09 23:57:09 [INFO] [TRAIN] epoch: 97, iter: 35750/40000, loss: 0.4023, lr: 0.001417, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2729 samples/sec | ETA 00:55:38 2021-05-09 23:57:17 [INFO] [TRAIN] epoch: 97, iter: 35760/40000, loss: 0.4232, lr: 0.001414, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 00:55:32 2021-05-09 23:57:25 [INFO] [TRAIN] epoch: 97, iter: 35770/40000, loss: 0.3077, lr: 0.001411, batch_cost: 0.7850, reader_cost: 0.00016, ips: 1.2738 samples/sec | ETA 00:55:20 2021-05-09 23:57:32 [INFO] [TRAIN] epoch: 97, iter: 35780/40000, loss: 0.1831, lr: 0.001408, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 00:55:18 2021-05-09 23:57:40 [INFO] [TRAIN] epoch: 97, iter: 35790/40000, loss: 0.1247, lr: 0.001405, batch_cost: 0.7877, reader_cost: 0.00015, ips: 1.2696 samples/sec | ETA 00:55:16 2021-05-09 23:57:48 [INFO] [TRAIN] epoch: 97, iter: 35800/40000, loss: 0.3410, lr: 0.001403, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2705 samples/sec | ETA 00:55:05 2021-05-09 23:57:56 [INFO] [TRAIN] epoch: 97, iter: 35810/40000, loss: 0.1827, lr: 0.001400, batch_cost: 0.7851, reader_cost: 0.00016, ips: 1.2737 samples/sec | ETA 00:54:49 2021-05-09 23:58:04 [INFO] [TRAIN] epoch: 97, iter: 35820/40000, loss: 0.2965, lr: 0.001397, batch_cost: 0.7841, reader_cost: 0.00015, ips: 1.2753 samples/sec | ETA 00:54:37 2021-05-09 23:58:12 [INFO] [TRAIN] epoch: 97, iter: 35830/40000, loss: 0.1422, lr: 0.001394, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2715 samples/sec | ETA 00:54:39 2021-05-09 23:58:20 [INFO] [TRAIN] epoch: 97, iter: 35840/40000, loss: 0.1747, lr: 0.001391, batch_cost: 0.7860, reader_cost: 0.00017, ips: 1.2723 samples/sec | ETA 00:54:29 2021-05-09 23:58:27 [INFO] [TRAIN] epoch: 97, iter: 35850/40000, loss: 0.3152, lr: 0.001389, batch_cost: 0.7856, reader_cost: 0.00014, ips: 1.2729 samples/sec | ETA 00:54:20 2021-05-09 23:58:35 [INFO] [TRAIN] epoch: 97, iter: 35860/40000, loss: 0.2632, lr: 0.001386, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2709 samples/sec | ETA 00:54:17 2021-05-09 23:58:43 [INFO] [TRAIN] epoch: 97, iter: 35870/40000, loss: 0.2074, lr: 0.001383, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2715 samples/sec | ETA 00:54:08 2021-05-09 23:58:51 [INFO] [TRAIN] epoch: 97, iter: 35880/40000, loss: 0.1363, lr: 0.001380, batch_cost: 0.7849, reader_cost: 0.00015, ips: 1.2741 samples/sec | ETA 00:53:53 2021-05-09 23:58:59 [INFO] [TRAIN] epoch: 97, iter: 35890/40000, loss: 0.3046, lr: 0.001377, batch_cost: 0.7868, reader_cost: 0.00017, ips: 1.2710 samples/sec | ETA 00:53:53 2021-05-09 23:59:07 [INFO] [TRAIN] epoch: 97, iter: 35900/40000, loss: 0.1819, lr: 0.001375, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 00:53:41 2021-05-09 23:59:15 [INFO] [TRAIN] epoch: 97, iter: 35910/40000, loss: 0.0788, lr: 0.001372, batch_cost: 0.7851, reader_cost: 0.00016, ips: 1.2738 samples/sec | ETA 00:53:30 2021-05-09 23:59:22 [INFO] [TRAIN] epoch: 97, iter: 35920/40000, loss: 0.2320, lr: 0.001369, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2736 samples/sec | ETA 00:53:23 2021-05-09 23:59:30 [INFO] [TRAIN] epoch: 97, iter: 35930/40000, loss: 0.3717, lr: 0.001366, batch_cost: 0.7845, reader_cost: 0.00016, ips: 1.2747 samples/sec | ETA 00:53:12 2021-05-09 23:59:38 [INFO] [TRAIN] epoch: 97, iter: 35940/40000, loss: 0.1536, lr: 0.001363, batch_cost: 0.7858, reader_cost: 0.00018, ips: 1.2726 samples/sec | ETA 00:53:10 2021-05-09 23:59:46 [INFO] [TRAIN] epoch: 97, iter: 35950/40000, loss: 0.2260, lr: 0.001361, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2736 samples/sec | ETA 00:52:59 2021-05-09 23:59:54 [INFO] [TRAIN] epoch: 97, iter: 35960/40000, loss: 0.1476, lr: 0.001358, batch_cost: 0.7847, reader_cost: 0.00017, ips: 1.2743 samples/sec | ETA 00:52:50 2021-05-10 00:00:02 [INFO] [TRAIN] epoch: 97, iter: 35970/40000, loss: 0.2280, lr: 0.001355, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2730 samples/sec | ETA 00:52:45 2021-05-10 00:00:10 [INFO] [TRAIN] epoch: 97, iter: 35980/40000, loss: 0.1232, lr: 0.001352, batch_cost: 0.7845, reader_cost: 0.00016, ips: 1.2747 samples/sec | ETA 00:52:33 2021-05-10 00:00:17 [INFO] [TRAIN] epoch: 97, iter: 35990/40000, loss: 0.3436, lr: 0.001349, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2709 samples/sec | ETA 00:52:35 2021-05-10 00:00:25 [INFO] [TRAIN] epoch: 97, iter: 36000/40000, loss: 0.3981, lr: 0.001347, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 00:52:25 2021-05-10 00:00:25 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-10 00:03:56 [INFO] [EVAL] #Images: 500 mIoU: 0.7691 Acc: 0.9576 Kappa: 0.9449 2021-05-10 00:03:56 [INFO] [EVAL] Class IoU: [0.9812 0.8471 0.9209 0.6228 0.6278 0.5007 0.631 0.7296 0.9163 0.6537 0.942 0.7834 0.5893 0.9439 0.8534 0.8951 0.796 0.6416 0.7374] 2021-05-10 00:03:56 [INFO] [EVAL] Class Acc: [0.9908 0.9207 0.9509 0.7973 0.8205 0.7685 0.8069 0.8883 0.9493 0.8311 0.9647 0.871 0.7815 0.9672 0.9305 0.9481 0.8944 0.7999 0.8473] 2021-05-10 00:04:25 [INFO] [EVAL] The model with the best validation mIoU (0.7695) was saved at iter 30000. 2021-05-10 00:04:32 [INFO] [TRAIN] epoch: 97, iter: 36010/40000, loss: 0.3764, lr: 0.001344, batch_cost: 0.7823, reader_cost: 0.00030, ips: 1.2783 samples/sec | ETA 00:52:01 2021-05-10 00:04:40 [INFO] [TRAIN] epoch: 97, iter: 36020/40000, loss: 0.4504, lr: 0.001341, batch_cost: 0.7839, reader_cost: 0.00032, ips: 1.2757 samples/sec | ETA 00:51:59 2021-05-10 00:04:48 [INFO] [TRAIN] epoch: 97, iter: 36030/40000, loss: 0.2153, lr: 0.001338, batch_cost: 0.7840, reader_cost: 0.00017, ips: 1.2756 samples/sec | ETA 00:51:52 2021-05-10 00:04:56 [INFO] [TRAIN] epoch: 97, iter: 36040/40000, loss: 0.1720, lr: 0.001335, batch_cost: 0.7856, reader_cost: 0.00017, ips: 1.2729 samples/sec | ETA 00:51:50 2021-05-10 00:05:04 [INFO] [TRAIN] epoch: 97, iter: 36050/40000, loss: 0.1695, lr: 0.001333, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2703 samples/sec | ETA 00:51:49 2021-05-10 00:05:12 [INFO] [TRAIN] epoch: 97, iter: 36060/40000, loss: 0.2147, lr: 0.001330, batch_cost: 0.7879, reader_cost: 0.00015, ips: 1.2692 samples/sec | ETA 00:51:44 2021-05-10 00:05:19 [INFO] [TRAIN] epoch: 97, iter: 36070/40000, loss: 0.3042, lr: 0.001327, batch_cost: 0.7858, reader_cost: 0.00015, ips: 1.2725 samples/sec | ETA 00:51:28 2021-05-10 00:05:27 [INFO] [TRAIN] epoch: 97, iter: 36080/40000, loss: 0.3281, lr: 0.001324, batch_cost: 0.7850, reader_cost: 0.00010, ips: 1.2739 samples/sec | ETA 00:51:17 2021-05-10 00:05:38 [INFO] [TRAIN] epoch: 98, iter: 36090/40000, loss: 0.3154, lr: 0.001321, batch_cost: 1.0799, reader_cost: 0.25151, ips: 0.9260 samples/sec | ETA 01:10:22 2021-05-10 00:05:46 [INFO] [TRAIN] epoch: 98, iter: 36100/40000, loss: 0.3217, lr: 0.001319, batch_cost: 0.8002, reader_cost: 0.00032, ips: 1.2497 samples/sec | ETA 00:52:00 2021-05-10 00:05:54 [INFO] [TRAIN] epoch: 98, iter: 36110/40000, loss: 0.3576, lr: 0.001316, batch_cost: 0.7881, reader_cost: 0.00015, ips: 1.2688 samples/sec | ETA 00:51:05 2021-05-10 00:06:02 [INFO] [TRAIN] epoch: 98, iter: 36120/40000, loss: 0.3199, lr: 0.001313, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2705 samples/sec | ETA 00:50:53 2021-05-10 00:06:10 [INFO] [TRAIN] epoch: 98, iter: 36130/40000, loss: 0.3470, lr: 0.001310, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2723 samples/sec | ETA 00:50:41 2021-05-10 00:06:18 [INFO] [TRAIN] epoch: 98, iter: 36140/40000, loss: 0.2637, lr: 0.001307, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2716 samples/sec | ETA 00:50:35 2021-05-10 00:06:25 [INFO] [TRAIN] epoch: 98, iter: 36150/40000, loss: 0.2540, lr: 0.001304, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2710 samples/sec | ETA 00:50:29 2021-05-10 00:06:33 [INFO] [TRAIN] epoch: 98, iter: 36160/40000, loss: 0.1598, lr: 0.001302, batch_cost: 0.7873, reader_cost: 0.00014, ips: 1.2702 samples/sec | ETA 00:50:23 2021-05-10 00:06:41 [INFO] [TRAIN] epoch: 98, iter: 36170/40000, loss: 0.2638, lr: 0.001299, batch_cost: 0.7849, reader_cost: 0.00016, ips: 1.2741 samples/sec | ETA 00:50:06 2021-05-10 00:06:49 [INFO] [TRAIN] epoch: 98, iter: 36180/40000, loss: 0.2614, lr: 0.001296, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2732 samples/sec | ETA 00:50:00 2021-05-10 00:06:57 [INFO] [TRAIN] epoch: 98, iter: 36190/40000, loss: 0.3098, lr: 0.001293, batch_cost: 0.7852, reader_cost: 0.00015, ips: 1.2736 samples/sec | ETA 00:49:51 2021-05-10 00:07:05 [INFO] [TRAIN] epoch: 98, iter: 36200/40000, loss: 0.2405, lr: 0.001290, batch_cost: 0.7853, reader_cost: 0.00015, ips: 1.2734 samples/sec | ETA 00:49:44 2021-05-10 00:07:13 [INFO] [TRAIN] epoch: 98, iter: 36210/40000, loss: 0.1437, lr: 0.001288, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2725 samples/sec | ETA 00:49:38 2021-05-10 00:07:21 [INFO] [TRAIN] epoch: 98, iter: 36220/40000, loss: 0.2823, lr: 0.001285, batch_cost: 0.7875, reader_cost: 0.00015, ips: 1.2698 samples/sec | ETA 00:49:36 2021-05-10 00:07:28 [INFO] [TRAIN] epoch: 98, iter: 36230/40000, loss: 0.3670, lr: 0.001282, batch_cost: 0.7847, reader_cost: 0.00017, ips: 1.2744 samples/sec | ETA 00:49:18 2021-05-10 00:07:36 [INFO] [TRAIN] epoch: 98, iter: 36240/40000, loss: 0.2724, lr: 0.001279, batch_cost: 0.7852, reader_cost: 0.00017, ips: 1.2735 samples/sec | ETA 00:49:12 2021-05-10 00:07:44 [INFO] [TRAIN] epoch: 98, iter: 36250/40000, loss: 0.1667, lr: 0.001276, batch_cost: 0.7853, reader_cost: 0.00017, ips: 1.2734 samples/sec | ETA 00:49:04 2021-05-10 00:07:52 [INFO] [TRAIN] epoch: 98, iter: 36260/40000, loss: 0.2781, lr: 0.001273, batch_cost: 0.7869, reader_cost: 0.00017, ips: 1.2708 samples/sec | ETA 00:49:03 2021-05-10 00:08:00 [INFO] [TRAIN] epoch: 98, iter: 36270/40000, loss: 0.2493, lr: 0.001271, batch_cost: 0.7852, reader_cost: 0.00017, ips: 1.2736 samples/sec | ETA 00:48:48 2021-05-10 00:08:08 [INFO] [TRAIN] epoch: 98, iter: 36280/40000, loss: 0.1182, lr: 0.001268, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2734 samples/sec | ETA 00:48:41 2021-05-10 00:08:15 [INFO] [TRAIN] epoch: 98, iter: 36290/40000, loss: 0.2209, lr: 0.001265, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2710 samples/sec | ETA 00:48:38 2021-05-10 00:08:23 [INFO] [TRAIN] epoch: 98, iter: 36300/40000, loss: 0.3137, lr: 0.001262, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2733 samples/sec | ETA 00:48:25 2021-05-10 00:08:31 [INFO] [TRAIN] epoch: 98, iter: 36310/40000, loss: 0.1817, lr: 0.001259, batch_cost: 0.7873, reader_cost: 0.00016, ips: 1.2702 samples/sec | ETA 00:48:25 2021-05-10 00:08:39 [INFO] [TRAIN] epoch: 98, iter: 36320/40000, loss: 0.2789, lr: 0.001257, batch_cost: 0.7859, reader_cost: 0.00018, ips: 1.2725 samples/sec | ETA 00:48:12 2021-05-10 00:08:47 [INFO] [TRAIN] epoch: 98, iter: 36330/40000, loss: 0.1119, lr: 0.001254, batch_cost: 0.7872, reader_cost: 0.00018, ips: 1.2703 samples/sec | ETA 00:48:09 2021-05-10 00:08:55 [INFO] [TRAIN] epoch: 98, iter: 36340/40000, loss: 0.2527, lr: 0.001251, batch_cost: 0.7861, reader_cost: 0.00018, ips: 1.2721 samples/sec | ETA 00:47:57 2021-05-10 00:09:03 [INFO] [TRAIN] epoch: 98, iter: 36350/40000, loss: 0.1676, lr: 0.001248, batch_cost: 0.7863, reader_cost: 0.00017, ips: 1.2718 samples/sec | ETA 00:47:49 2021-05-10 00:09:11 [INFO] [TRAIN] epoch: 98, iter: 36360/40000, loss: 0.3287, lr: 0.001245, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2727 samples/sec | ETA 00:47:39 2021-05-10 00:09:18 [INFO] [TRAIN] epoch: 98, iter: 36370/40000, loss: 0.4763, lr: 0.001242, batch_cost: 0.7841, reader_cost: 0.00016, ips: 1.2754 samples/sec | ETA 00:47:26 2021-05-10 00:09:26 [INFO] [TRAIN] epoch: 98, iter: 36380/40000, loss: 0.3172, lr: 0.001240, batch_cost: 0.7865, reader_cost: 0.00017, ips: 1.2715 samples/sec | ETA 00:47:26 2021-05-10 00:09:34 [INFO] [TRAIN] epoch: 98, iter: 36390/40000, loss: 0.5034, lr: 0.001237, batch_cost: 0.7853, reader_cost: 0.00017, ips: 1.2734 samples/sec | ETA 00:47:14 2021-05-10 00:09:42 [INFO] [TRAIN] epoch: 98, iter: 36400/40000, loss: 0.3643, lr: 0.001234, batch_cost: 0.7884, reader_cost: 0.00018, ips: 1.2684 samples/sec | ETA 00:47:18 2021-05-10 00:09:50 [INFO] [TRAIN] epoch: 98, iter: 36410/40000, loss: 0.6272, lr: 0.001231, batch_cost: 0.7884, reader_cost: 0.00016, ips: 1.2685 samples/sec | ETA 00:47:10 2021-05-10 00:09:58 [INFO] [TRAIN] epoch: 98, iter: 36420/40000, loss: 0.1383, lr: 0.001228, batch_cost: 0.7877, reader_cost: 0.00017, ips: 1.2695 samples/sec | ETA 00:47:00 2021-05-10 00:10:06 [INFO] [TRAIN] epoch: 98, iter: 36430/40000, loss: 0.1718, lr: 0.001225, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2703 samples/sec | ETA 00:46:50 2021-05-10 00:10:14 [INFO] [TRAIN] epoch: 98, iter: 36440/40000, loss: 0.2992, lr: 0.001223, batch_cost: 0.7893, reader_cost: 0.00016, ips: 1.2670 samples/sec | ETA 00:46:49 2021-05-10 00:10:21 [INFO] [TRAIN] epoch: 98, iter: 36450/40000, loss: 0.3261, lr: 0.001220, batch_cost: 0.7849, reader_cost: 0.00013, ips: 1.2740 samples/sec | ETA 00:46:26 2021-05-10 00:10:32 [INFO] [TRAIN] epoch: 99, iter: 36460/40000, loss: 0.2978, lr: 0.001217, batch_cost: 1.0780, reader_cost: 0.25839, ips: 0.9276 samples/sec | ETA 01:03:36 2021-05-10 00:10:40 [INFO] [TRAIN] epoch: 99, iter: 36470/40000, loss: 0.2747, lr: 0.001214, batch_cost: 0.7965, reader_cost: 0.00038, ips: 1.2556 samples/sec | ETA 00:46:51 2021-05-10 00:10:48 [INFO] [TRAIN] epoch: 99, iter: 36480/40000, loss: 0.4135, lr: 0.001211, batch_cost: 0.7850, reader_cost: 0.00017, ips: 1.2739 samples/sec | ETA 00:46:03 2021-05-10 00:10:56 [INFO] [TRAIN] epoch: 99, iter: 36490/40000, loss: 0.2283, lr: 0.001208, batch_cost: 0.7867, reader_cost: 0.00017, ips: 1.2711 samples/sec | ETA 00:46:01 2021-05-10 00:11:04 [INFO] [TRAIN] epoch: 99, iter: 36500/40000, loss: 0.3984, lr: 0.001205, batch_cost: 0.7871, reader_cost: 0.00017, ips: 1.2706 samples/sec | ETA 00:45:54 2021-05-10 00:11:12 [INFO] [TRAIN] epoch: 99, iter: 36510/40000, loss: 0.3574, lr: 0.001203, batch_cost: 0.7874, reader_cost: 0.00016, ips: 1.2701 samples/sec | ETA 00:45:47 2021-05-10 00:11:19 [INFO] [TRAIN] epoch: 99, iter: 36520/40000, loss: 0.2596, lr: 0.001200, batch_cost: 0.7869, reader_cost: 0.00017, ips: 1.2708 samples/sec | ETA 00:45:38 2021-05-10 00:11:27 [INFO] [TRAIN] epoch: 99, iter: 36530/40000, loss: 0.1498, lr: 0.001197, batch_cost: 0.7849, reader_cost: 0.00015, ips: 1.2741 samples/sec | ETA 00:45:23 2021-05-10 00:11:35 [INFO] [TRAIN] epoch: 99, iter: 36540/40000, loss: 0.3282, lr: 0.001194, batch_cost: 0.7863, reader_cost: 0.00018, ips: 1.2718 samples/sec | ETA 00:45:20 2021-05-10 00:11:43 [INFO] [TRAIN] epoch: 99, iter: 36550/40000, loss: 0.3310, lr: 0.001191, batch_cost: 0.7865, reader_cost: 0.00018, ips: 1.2715 samples/sec | ETA 00:45:13 2021-05-10 00:11:51 [INFO] [TRAIN] epoch: 99, iter: 36560/40000, loss: 0.2030, lr: 0.001188, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 00:45:05 2021-05-10 00:11:59 [INFO] [TRAIN] epoch: 99, iter: 36570/40000, loss: 0.3010, lr: 0.001186, batch_cost: 0.7870, reader_cost: 0.00017, ips: 1.2707 samples/sec | ETA 00:44:59 2021-05-10 00:12:07 [INFO] [TRAIN] epoch: 99, iter: 36580/40000, loss: 0.1026, lr: 0.001183, batch_cost: 0.7871, reader_cost: 0.00017, ips: 1.2705 samples/sec | ETA 00:44:51 2021-05-10 00:12:15 [INFO] [TRAIN] epoch: 99, iter: 36590/40000, loss: 0.2079, lr: 0.001180, batch_cost: 0.7886, reader_cost: 0.00016, ips: 1.2681 samples/sec | ETA 00:44:49 2021-05-10 00:12:22 [INFO] [TRAIN] epoch: 99, iter: 36600/40000, loss: 0.2653, lr: 0.001177, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2703 samples/sec | ETA 00:44:36 2021-05-10 00:12:30 [INFO] [TRAIN] epoch: 99, iter: 36610/40000, loss: 0.3216, lr: 0.001174, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2710 samples/sec | ETA 00:44:27 2021-05-10 00:12:38 [INFO] [TRAIN] epoch: 99, iter: 36620/40000, loss: 0.2214, lr: 0.001171, batch_cost: 0.7861, reader_cost: 0.00017, ips: 1.2722 samples/sec | ETA 00:44:16 2021-05-10 00:12:46 [INFO] [TRAIN] epoch: 99, iter: 36630/40000, loss: 0.1692, lr: 0.001168, batch_cost: 0.7848, reader_cost: 0.00017, ips: 1.2742 samples/sec | ETA 00:44:04 2021-05-10 00:12:54 [INFO] [TRAIN] epoch: 99, iter: 36640/40000, loss: 0.1902, lr: 0.001166, batch_cost: 0.7867, reader_cost: 0.00014, ips: 1.2711 samples/sec | ETA 00:44:03 2021-05-10 00:13:02 [INFO] [TRAIN] epoch: 99, iter: 36650/40000, loss: 0.0738, lr: 0.001163, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 00:43:54 2021-05-10 00:13:10 [INFO] [TRAIN] epoch: 99, iter: 36660/40000, loss: 0.1723, lr: 0.001160, batch_cost: 0.7857, reader_cost: 0.00017, ips: 1.2728 samples/sec | ETA 00:43:44 2021-05-10 00:13:17 [INFO] [TRAIN] epoch: 99, iter: 36670/40000, loss: 0.2823, lr: 0.001157, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2728 samples/sec | ETA 00:43:36 2021-05-10 00:13:25 [INFO] [TRAIN] epoch: 99, iter: 36680/40000, loss: 0.2564, lr: 0.001154, batch_cost: 0.7871, reader_cost: 0.00015, ips: 1.2705 samples/sec | ETA 00:43:33 2021-05-10 00:13:33 [INFO] [TRAIN] epoch: 99, iter: 36690/40000, loss: 0.2639, lr: 0.001151, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2731 samples/sec | ETA 00:43:19 2021-05-10 00:13:41 [INFO] [TRAIN] epoch: 99, iter: 36700/40000, loss: 0.1860, lr: 0.001148, batch_cost: 0.7845, reader_cost: 0.00016, ips: 1.2747 samples/sec | ETA 00:43:08 2021-05-10 00:13:49 [INFO] [TRAIN] epoch: 99, iter: 36710/40000, loss: 0.2193, lr: 0.001146, batch_cost: 0.7860, reader_cost: 0.00018, ips: 1.2722 samples/sec | ETA 00:43:06 2021-05-10 00:13:57 [INFO] [TRAIN] epoch: 99, iter: 36720/40000, loss: 0.1642, lr: 0.001143, batch_cost: 0.7859, reader_cost: 0.00017, ips: 1.2725 samples/sec | ETA 00:42:57 2021-05-10 00:14:05 [INFO] [TRAIN] epoch: 99, iter: 36730/40000, loss: 0.2880, lr: 0.001140, batch_cost: 0.7854, reader_cost: 0.00018, ips: 1.2732 samples/sec | ETA 00:42:48 2021-05-10 00:14:12 [INFO] [TRAIN] epoch: 99, iter: 36740/40000, loss: 0.4299, lr: 0.001137, batch_cost: 0.7872, reader_cost: 0.00015, ips: 1.2703 samples/sec | ETA 00:42:46 2021-05-10 00:14:20 [INFO] [TRAIN] epoch: 99, iter: 36750/40000, loss: 0.2865, lr: 0.001134, batch_cost: 0.7852, reader_cost: 0.00017, ips: 1.2735 samples/sec | ETA 00:42:31 2021-05-10 00:14:28 [INFO] [TRAIN] epoch: 99, iter: 36760/40000, loss: 0.5540, lr: 0.001131, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2711 samples/sec | ETA 00:42:28 2021-05-10 00:14:36 [INFO] [TRAIN] epoch: 99, iter: 36770/40000, loss: 0.3931, lr: 0.001128, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 00:42:19 2021-05-10 00:14:44 [INFO] [TRAIN] epoch: 99, iter: 36780/40000, loss: 0.1977, lr: 0.001126, batch_cost: 0.7859, reader_cost: 0.00017, ips: 1.2724 samples/sec | ETA 00:42:10 2021-05-10 00:14:52 [INFO] [TRAIN] epoch: 99, iter: 36790/40000, loss: 0.0973, lr: 0.001123, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 00:42:01 2021-05-10 00:15:00 [INFO] [TRAIN] epoch: 99, iter: 36800/40000, loss: 0.1987, lr: 0.001120, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2714 samples/sec | ETA 00:41:56 2021-05-10 00:15:07 [INFO] [TRAIN] epoch: 99, iter: 36810/40000, loss: 0.2941, lr: 0.001117, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2722 samples/sec | ETA 00:41:47 2021-05-10 00:15:15 [INFO] [TRAIN] epoch: 99, iter: 36820/40000, loss: 0.3499, lr: 0.001114, batch_cost: 0.7866, reader_cost: 0.00014, ips: 1.2713 samples/sec | ETA 00:41:41 2021-05-10 00:15:26 [INFO] [TRAIN] epoch: 100, iter: 36830/40000, loss: 0.3095, lr: 0.001111, batch_cost: 1.0953, reader_cost: 0.25236, ips: 0.9130 samples/sec | ETA 00:57:52 2021-05-10 00:15:34 [INFO] [TRAIN] epoch: 100, iter: 36840/40000, loss: 0.2403, lr: 0.001108, batch_cost: 0.7985, reader_cost: 0.00033, ips: 1.2524 samples/sec | ETA 00:42:03 2021-05-10 00:15:42 [INFO] [TRAIN] epoch: 100, iter: 36850/40000, loss: 0.5446, lr: 0.001106, batch_cost: 0.7852, reader_cost: 0.00017, ips: 1.2735 samples/sec | ETA 00:41:13 2021-05-10 00:15:50 [INFO] [TRAIN] epoch: 100, iter: 36860/40000, loss: 0.2303, lr: 0.001103, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2729 samples/sec | ETA 00:41:06 2021-05-10 00:15:58 [INFO] [TRAIN] epoch: 100, iter: 36870/40000, loss: 0.2971, lr: 0.001100, batch_cost: 0.7870, reader_cost: 0.00015, ips: 1.2706 samples/sec | ETA 00:41:03 2021-05-10 00:16:06 [INFO] [TRAIN] epoch: 100, iter: 36880/40000, loss: 0.4292, lr: 0.001097, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2710 samples/sec | ETA 00:40:54 2021-05-10 00:16:14 [INFO] [TRAIN] epoch: 100, iter: 36890/40000, loss: 0.2227, lr: 0.001094, batch_cost: 0.7861, reader_cost: 0.00017, ips: 1.2721 samples/sec | ETA 00:40:44 2021-05-10 00:16:21 [INFO] [TRAIN] epoch: 100, iter: 36900/40000, loss: 0.1133, lr: 0.001091, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 00:40:35 2021-05-10 00:16:29 [INFO] [TRAIN] epoch: 100, iter: 36910/40000, loss: 0.2663, lr: 0.001088, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 00:40:27 2021-05-10 00:16:37 [INFO] [TRAIN] epoch: 100, iter: 36920/40000, loss: 0.2707, lr: 0.001085, batch_cost: 0.7875, reader_cost: 0.00015, ips: 1.2698 samples/sec | ETA 00:40:25 2021-05-10 00:16:45 [INFO] [TRAIN] epoch: 100, iter: 36930/40000, loss: 0.1748, lr: 0.001083, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2710 samples/sec | ETA 00:40:15 2021-05-10 00:16:53 [INFO] [TRAIN] epoch: 100, iter: 36940/40000, loss: 0.2913, lr: 0.001080, batch_cost: 0.7860, reader_cost: 0.00017, ips: 1.2722 samples/sec | ETA 00:40:05 2021-05-10 00:17:01 [INFO] [TRAIN] epoch: 100, iter: 36950/40000, loss: 0.1024, lr: 0.001077, batch_cost: 0.7855, reader_cost: 0.00017, ips: 1.2730 samples/sec | ETA 00:39:55 2021-05-10 00:17:09 [INFO] [TRAIN] epoch: 100, iter: 36960/40000, loss: 0.2514, lr: 0.001074, batch_cost: 0.7844, reader_cost: 0.00016, ips: 1.2748 samples/sec | ETA 00:39:44 2021-05-10 00:17:16 [INFO] [TRAIN] epoch: 100, iter: 36970/40000, loss: 0.4099, lr: 0.001071, batch_cost: 0.7847, reader_cost: 0.00016, ips: 1.2744 samples/sec | ETA 00:39:37 2021-05-10 00:17:24 [INFO] [TRAIN] epoch: 100, iter: 36980/40000, loss: 0.2871, lr: 0.001068, batch_cost: 0.7843, reader_cost: 0.00016, ips: 1.2750 samples/sec | ETA 00:39:28 2021-05-10 00:17:32 [INFO] [TRAIN] epoch: 100, iter: 36990/40000, loss: 0.2657, lr: 0.001065, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2732 samples/sec | ETA 00:39:24 2021-05-10 00:17:40 [INFO] [TRAIN] epoch: 100, iter: 37000/40000, loss: 0.1574, lr: 0.001062, batch_cost: 0.7874, reader_cost: 0.00018, ips: 1.2699 samples/sec | ETA 00:39:22 2021-05-10 00:17:40 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-10 00:21:10 [INFO] [EVAL] #Images: 500 mIoU: 0.7708 Acc: 0.9576 Kappa: 0.9450 2021-05-10 00:21:10 [INFO] [EVAL] Class IoU: [0.981 0.8465 0.9215 0.6298 0.6236 0.5026 0.6315 0.7302 0.9166 0.6531 0.9415 0.7828 0.5876 0.9443 0.859 0.898 0.8099 0.6474 0.7385] 2021-05-10 00:21:10 [INFO] [EVAL] Class Acc: [0.9924 0.9078 0.9515 0.8149 0.8207 0.7619 0.8066 0.8784 0.9507 0.8394 0.9632 0.8629 0.7779 0.9671 0.9343 0.9655 0.8797 0.819 0.8401] 2021-05-10 00:21:58 [INFO] [EVAL] The model with the best validation mIoU (0.7708) was saved at iter 37000. 2021-05-10 00:22:06 [INFO] [TRAIN] epoch: 100, iter: 37010/40000, loss: 0.2537, lr: 0.001059, batch_cost: 0.7867, reader_cost: 0.00046, ips: 1.2711 samples/sec | ETA 00:39:12 2021-05-10 00:22:14 [INFO] [TRAIN] epoch: 100, iter: 37020/40000, loss: 0.1520, lr: 0.001057, batch_cost: 0.7840, reader_cost: 0.00036, ips: 1.2755 samples/sec | ETA 00:38:56 2021-05-10 00:22:22 [INFO] [TRAIN] epoch: 100, iter: 37030/40000, loss: 0.1415, lr: 0.001054, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2713 samples/sec | ETA 00:38:56 2021-05-10 00:22:30 [INFO] [TRAIN] epoch: 100, iter: 37040/40000, loss: 0.2754, lr: 0.001051, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2710 samples/sec | ETA 00:38:48 2021-05-10 00:22:37 [INFO] [TRAIN] epoch: 100, iter: 37050/40000, loss: 0.2614, lr: 0.001048, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 00:38:40 2021-05-10 00:22:45 [INFO] [TRAIN] epoch: 100, iter: 37060/40000, loss: 0.2629, lr: 0.001045, batch_cost: 0.7871, reader_cost: 0.00017, ips: 1.2706 samples/sec | ETA 00:38:33 2021-05-10 00:22:53 [INFO] [TRAIN] epoch: 100, iter: 37070/40000, loss: 0.1993, lr: 0.001042, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 00:38:23 2021-05-10 00:23:01 [INFO] [TRAIN] epoch: 100, iter: 37080/40000, loss: 0.2363, lr: 0.001039, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 00:38:14 2021-05-10 00:23:09 [INFO] [TRAIN] epoch: 100, iter: 37090/40000, loss: 0.0872, lr: 0.001036, batch_cost: 0.7837, reader_cost: 0.00016, ips: 1.2760 samples/sec | ETA 00:38:00 2021-05-10 00:23:17 [INFO] [TRAIN] epoch: 100, iter: 37100/40000, loss: 0.2699, lr: 0.001033, batch_cost: 0.7861, reader_cost: 0.00017, ips: 1.2722 samples/sec | ETA 00:37:59 2021-05-10 00:23:25 [INFO] [TRAIN] epoch: 100, iter: 37110/40000, loss: 0.3285, lr: 0.001031, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2720 samples/sec | ETA 00:37:52 2021-05-10 00:23:32 [INFO] [TRAIN] epoch: 100, iter: 37120/40000, loss: 0.3350, lr: 0.001028, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2711 samples/sec | ETA 00:37:45 2021-05-10 00:23:40 [INFO] [TRAIN] epoch: 100, iter: 37130/40000, loss: 0.3937, lr: 0.001025, batch_cost: 0.7862, reader_cost: 0.00019, ips: 1.2719 samples/sec | ETA 00:37:36 2021-05-10 00:23:48 [INFO] [TRAIN] epoch: 100, iter: 37140/40000, loss: 0.3776, lr: 0.001022, batch_cost: 0.7854, reader_cost: 0.00018, ips: 1.2732 samples/sec | ETA 00:37:26 2021-05-10 00:23:56 [INFO] [TRAIN] epoch: 100, iter: 37150/40000, loss: 0.3583, lr: 0.001019, batch_cost: 0.7851, reader_cost: 0.00018, ips: 1.2737 samples/sec | ETA 00:37:17 2021-05-10 00:24:04 [INFO] [TRAIN] epoch: 100, iter: 37160/40000, loss: 0.1095, lr: 0.001016, batch_cost: 0.7882, reader_cost: 0.00017, ips: 1.2687 samples/sec | ETA 00:37:18 2021-05-10 00:24:12 [INFO] [TRAIN] epoch: 100, iter: 37170/40000, loss: 0.2164, lr: 0.001013, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2736 samples/sec | ETA 00:37:02 2021-05-10 00:24:20 [INFO] [TRAIN] epoch: 100, iter: 37180/40000, loss: 0.2338, lr: 0.001010, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2716 samples/sec | ETA 00:36:57 2021-05-10 00:24:27 [INFO] [TRAIN] epoch: 100, iter: 37190/40000, loss: 0.2917, lr: 0.001007, batch_cost: 0.7862, reader_cost: 0.00014, ips: 1.2719 samples/sec | ETA 00:36:49 2021-05-10 00:24:35 [INFO] [TRAIN] epoch: 100, iter: 37200/40000, loss: 0.3941, lr: 0.001004, batch_cost: 0.7844, reader_cost: 0.00010, ips: 1.2749 samples/sec | ETA 00:36:36 2021-05-10 00:24:46 [INFO] [TRAIN] epoch: 101, iter: 37210/40000, loss: 0.2086, lr: 0.001001, batch_cost: 1.0907, reader_cost: 0.24312, ips: 0.9169 samples/sec | ETA 00:50:42 2021-05-10 00:24:54 [INFO] [TRAIN] epoch: 101, iter: 37220/40000, loss: 0.4297, lr: 0.000999, batch_cost: 0.7884, reader_cost: 0.00034, ips: 1.2684 samples/sec | ETA 00:36:31 2021-05-10 00:25:02 [INFO] [TRAIN] epoch: 101, iter: 37230/40000, loss: 0.2817, lr: 0.000996, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2736 samples/sec | ETA 00:36:15 2021-05-10 00:25:10 [INFO] [TRAIN] epoch: 101, iter: 37240/40000, loss: 0.4144, lr: 0.000993, batch_cost: 0.7871, reader_cost: 0.00015, ips: 1.2705 samples/sec | ETA 00:36:12 2021-05-10 00:25:18 [INFO] [TRAIN] epoch: 101, iter: 37250/40000, loss: 0.4415, lr: 0.000990, batch_cost: 0.7848, reader_cost: 0.00016, ips: 1.2742 samples/sec | ETA 00:35:58 2021-05-10 00:25:26 [INFO] [TRAIN] epoch: 101, iter: 37260/40000, loss: 0.2379, lr: 0.000987, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2731 samples/sec | ETA 00:35:52 2021-05-10 00:25:33 [INFO] [TRAIN] epoch: 101, iter: 37270/40000, loss: 0.3897, lr: 0.000984, batch_cost: 0.7872, reader_cost: 0.00015, ips: 1.2703 samples/sec | ETA 00:35:49 2021-05-10 00:25:41 [INFO] [TRAIN] epoch: 101, iter: 37280/40000, loss: 0.2058, lr: 0.000981, batch_cost: 0.7861, reader_cost: 0.00018, ips: 1.2722 samples/sec | ETA 00:35:38 2021-05-10 00:25:49 [INFO] [TRAIN] epoch: 101, iter: 37290/40000, loss: 0.2799, lr: 0.000978, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2729 samples/sec | ETA 00:35:28 2021-05-10 00:25:57 [INFO] [TRAIN] epoch: 101, iter: 37300/40000, loss: 0.2086, lr: 0.000975, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 00:35:21 2021-05-10 00:26:05 [INFO] [TRAIN] epoch: 101, iter: 37310/40000, loss: 0.2873, lr: 0.000972, batch_cost: 0.7846, reader_cost: 0.00015, ips: 1.2746 samples/sec | ETA 00:35:10 2021-05-10 00:26:13 [INFO] [TRAIN] epoch: 101, iter: 37320/40000, loss: 0.1061, lr: 0.000969, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2715 samples/sec | ETA 00:35:07 2021-05-10 00:26:21 [INFO] [TRAIN] epoch: 101, iter: 37330/40000, loss: 0.2651, lr: 0.000967, batch_cost: 0.7848, reader_cost: 0.00015, ips: 1.2743 samples/sec | ETA 00:34:55 2021-05-10 00:26:28 [INFO] [TRAIN] epoch: 101, iter: 37340/40000, loss: 0.2599, lr: 0.000964, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2728 samples/sec | ETA 00:34:49 2021-05-10 00:26:36 [INFO] [TRAIN] epoch: 101, iter: 37350/40000, loss: 0.3685, lr: 0.000961, batch_cost: 0.7863, reader_cost: 0.00017, ips: 1.2718 samples/sec | ETA 00:34:43 2021-05-10 00:26:44 [INFO] [TRAIN] epoch: 101, iter: 37360/40000, loss: 0.2435, lr: 0.000958, batch_cost: 0.7850, reader_cost: 0.00017, ips: 1.2739 samples/sec | ETA 00:34:32 2021-05-10 00:26:52 [INFO] [TRAIN] epoch: 101, iter: 37370/40000, loss: 0.1444, lr: 0.000955, batch_cost: 0.7849, reader_cost: 0.00016, ips: 1.2740 samples/sec | ETA 00:34:24 2021-05-10 00:27:00 [INFO] [TRAIN] epoch: 101, iter: 37380/40000, loss: 0.1581, lr: 0.000952, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 00:34:19 2021-05-10 00:27:08 [INFO] [TRAIN] epoch: 101, iter: 37390/40000, loss: 0.2194, lr: 0.000949, batch_cost: 0.7850, reader_cost: 0.00017, ips: 1.2739 samples/sec | ETA 00:34:08 2021-05-10 00:27:16 [INFO] [TRAIN] epoch: 101, iter: 37400/40000, loss: 0.1106, lr: 0.000946, batch_cost: 0.7865, reader_cost: 0.00017, ips: 1.2714 samples/sec | ETA 00:34:04 2021-05-10 00:27:23 [INFO] [TRAIN] epoch: 101, iter: 37410/40000, loss: 0.3479, lr: 0.000943, batch_cost: 0.7842, reader_cost: 0.00016, ips: 1.2751 samples/sec | ETA 00:33:51 2021-05-10 00:27:31 [INFO] [TRAIN] epoch: 101, iter: 37420/40000, loss: 0.1986, lr: 0.000940, batch_cost: 0.7858, reader_cost: 0.00017, ips: 1.2726 samples/sec | ETA 00:33:47 2021-05-10 00:27:39 [INFO] [TRAIN] epoch: 101, iter: 37430/40000, loss: 0.1890, lr: 0.000937, batch_cost: 0.7858, reader_cost: 0.00019, ips: 1.2725 samples/sec | ETA 00:33:39 2021-05-10 00:27:47 [INFO] [TRAIN] epoch: 101, iter: 37440/40000, loss: 0.2292, lr: 0.000934, batch_cost: 0.7865, reader_cost: 0.00018, ips: 1.2715 samples/sec | ETA 00:33:33 2021-05-10 00:27:55 [INFO] [TRAIN] epoch: 101, iter: 37450/40000, loss: 0.2161, lr: 0.000931, batch_cost: 0.7863, reader_cost: 0.00019, ips: 1.2717 samples/sec | ETA 00:33:25 2021-05-10 00:28:03 [INFO] [TRAIN] epoch: 101, iter: 37460/40000, loss: 0.1946, lr: 0.000928, batch_cost: 0.7862, reader_cost: 0.00019, ips: 1.2719 samples/sec | ETA 00:33:16 2021-05-10 00:28:11 [INFO] [TRAIN] epoch: 101, iter: 37470/40000, loss: 0.2918, lr: 0.000926, batch_cost: 0.7858, reader_cost: 0.00017, ips: 1.2727 samples/sec | ETA 00:33:07 2021-05-10 00:28:18 [INFO] [TRAIN] epoch: 101, iter: 37480/40000, loss: 0.3550, lr: 0.000923, batch_cost: 0.7877, reader_cost: 0.00016, ips: 1.2695 samples/sec | ETA 00:33:05 2021-05-10 00:28:26 [INFO] [TRAIN] epoch: 101, iter: 37490/40000, loss: 0.3330, lr: 0.000920, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2718 samples/sec | ETA 00:32:53 2021-05-10 00:28:34 [INFO] [TRAIN] epoch: 101, iter: 37500/40000, loss: 0.3304, lr: 0.000917, batch_cost: 0.7854, reader_cost: 0.00017, ips: 1.2732 samples/sec | ETA 00:32:43 2021-05-10 00:28:42 [INFO] [TRAIN] epoch: 101, iter: 37510/40000, loss: 0.4755, lr: 0.000914, batch_cost: 0.7846, reader_cost: 0.00017, ips: 1.2745 samples/sec | ETA 00:32:33 2021-05-10 00:28:50 [INFO] [TRAIN] epoch: 101, iter: 37520/40000, loss: 0.5728, lr: 0.000911, batch_cost: 0.7865, reader_cost: 0.00017, ips: 1.2715 samples/sec | ETA 00:32:30 2021-05-10 00:28:58 [INFO] [TRAIN] epoch: 101, iter: 37530/40000, loss: 0.2237, lr: 0.000908, batch_cost: 0.7863, reader_cost: 0.00017, ips: 1.2718 samples/sec | ETA 00:32:22 2021-05-10 00:29:06 [INFO] [TRAIN] epoch: 101, iter: 37540/40000, loss: 0.1997, lr: 0.000905, batch_cost: 0.7879, reader_cost: 0.00016, ips: 1.2692 samples/sec | ETA 00:32:18 2021-05-10 00:29:13 [INFO] [TRAIN] epoch: 101, iter: 37550/40000, loss: 0.2763, lr: 0.000902, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2717 samples/sec | ETA 00:32:06 2021-05-10 00:29:21 [INFO] [TRAIN] epoch: 101, iter: 37560/40000, loss: 0.2593, lr: 0.000899, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2731 samples/sec | ETA 00:31:56 2021-05-10 00:29:29 [INFO] [TRAIN] epoch: 101, iter: 37570/40000, loss: 0.2897, lr: 0.000896, batch_cost: 0.7842, reader_cost: 0.00023, ips: 1.2752 samples/sec | ETA 00:31:45 2021-05-10 00:29:40 [INFO] [TRAIN] epoch: 102, iter: 37580/40000, loss: 0.2509, lr: 0.000893, batch_cost: 1.1113, reader_cost: 0.29334, ips: 0.8999 samples/sec | ETA 00:44:49 2021-05-10 00:29:48 [INFO] [TRAIN] epoch: 102, iter: 37590/40000, loss: 0.3573, lr: 0.000890, batch_cost: 0.7914, reader_cost: 0.00034, ips: 1.2636 samples/sec | ETA 00:31:47 2021-05-10 00:29:56 [INFO] [TRAIN] epoch: 102, iter: 37600/40000, loss: 0.3606, lr: 0.000887, batch_cost: 0.7866, reader_cost: 0.00017, ips: 1.2712 samples/sec | ETA 00:31:27 2021-05-10 00:30:04 [INFO] [TRAIN] epoch: 102, iter: 37610/40000, loss: 0.2984, lr: 0.000884, batch_cost: 0.7877, reader_cost: 0.00016, ips: 1.2696 samples/sec | ETA 00:31:22 2021-05-10 00:30:12 [INFO] [TRAIN] epoch: 102, iter: 37620/40000, loss: 0.4833, lr: 0.000881, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2707 samples/sec | ETA 00:31:12 2021-05-10 00:30:20 [INFO] [TRAIN] epoch: 102, iter: 37630/40000, loss: 0.2741, lr: 0.000878, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 00:31:03 2021-05-10 00:30:28 [INFO] [TRAIN] epoch: 102, iter: 37640/40000, loss: 0.1645, lr: 0.000875, batch_cost: 0.7866, reader_cost: 0.00017, ips: 1.2713 samples/sec | ETA 00:30:56 2021-05-10 00:30:35 [INFO] [TRAIN] epoch: 102, iter: 37650/40000, loss: 0.1914, lr: 0.000873, batch_cost: 0.7864, reader_cost: 0.00017, ips: 1.2716 samples/sec | ETA 00:30:47 2021-05-10 00:30:43 [INFO] [TRAIN] epoch: 102, iter: 37660/40000, loss: 0.3048, lr: 0.000870, batch_cost: 0.7872, reader_cost: 0.00017, ips: 1.2704 samples/sec | ETA 00:30:41 2021-05-10 00:30:51 [INFO] [TRAIN] epoch: 102, iter: 37670/40000, loss: 0.2456, lr: 0.000867, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2726 samples/sec | ETA 00:30:30 2021-05-10 00:30:59 [INFO] [TRAIN] epoch: 102, iter: 37680/40000, loss: 0.2675, lr: 0.000864, batch_cost: 0.7861, reader_cost: 0.00017, ips: 1.2722 samples/sec | ETA 00:30:23 2021-05-10 00:31:07 [INFO] [TRAIN] epoch: 102, iter: 37690/40000, loss: 0.1849, lr: 0.000861, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2727 samples/sec | ETA 00:30:15 2021-05-10 00:31:15 [INFO] [TRAIN] epoch: 102, iter: 37700/40000, loss: 0.1852, lr: 0.000858, batch_cost: 0.7875, reader_cost: 0.00016, ips: 1.2698 samples/sec | ETA 00:30:11 2021-05-10 00:31:23 [INFO] [TRAIN] epoch: 102, iter: 37710/40000, loss: 0.2668, lr: 0.000855, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2724 samples/sec | ETA 00:29:59 2021-05-10 00:31:30 [INFO] [TRAIN] epoch: 102, iter: 37720/40000, loss: 0.2208, lr: 0.000852, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2725 samples/sec | ETA 00:29:51 2021-05-10 00:31:38 [INFO] [TRAIN] epoch: 102, iter: 37730/40000, loss: 0.3816, lr: 0.000849, batch_cost: 0.7850, reader_cost: 0.00016, ips: 1.2739 samples/sec | ETA 00:29:41 2021-05-10 00:31:46 [INFO] [TRAIN] epoch: 102, iter: 37740/40000, loss: 0.3259, lr: 0.000846, batch_cost: 0.7850, reader_cost: 0.00018, ips: 1.2739 samples/sec | ETA 00:29:34 2021-05-10 00:31:54 [INFO] [TRAIN] epoch: 102, iter: 37750/40000, loss: 0.1737, lr: 0.000843, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 00:29:28 2021-05-10 00:32:02 [INFO] [TRAIN] epoch: 102, iter: 37760/40000, loss: 0.2368, lr: 0.000840, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2725 samples/sec | ETA 00:29:20 2021-05-10 00:32:10 [INFO] [TRAIN] epoch: 102, iter: 37770/40000, loss: 0.0848, lr: 0.000837, batch_cost: 0.7841, reader_cost: 0.00016, ips: 1.2754 samples/sec | ETA 00:29:08 2021-05-10 00:32:18 [INFO] [TRAIN] epoch: 102, iter: 37780/40000, loss: 0.2873, lr: 0.000834, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 00:29:03 2021-05-10 00:32:25 [INFO] [TRAIN] epoch: 102, iter: 37790/40000, loss: 0.3321, lr: 0.000831, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2730 samples/sec | ETA 00:28:56 2021-05-10 00:32:33 [INFO] [TRAIN] epoch: 102, iter: 37800/40000, loss: 0.2466, lr: 0.000828, batch_cost: 0.7850, reader_cost: 0.00017, ips: 1.2738 samples/sec | ETA 00:28:47 2021-05-10 00:32:41 [INFO] [TRAIN] epoch: 102, iter: 37810/40000, loss: 0.2204, lr: 0.000825, batch_cost: 0.7847, reader_cost: 0.00016, ips: 1.2743 samples/sec | ETA 00:28:38 2021-05-10 00:32:49 [INFO] [TRAIN] epoch: 102, iter: 37820/40000, loss: 0.1972, lr: 0.000822, batch_cost: 0.7846, reader_cost: 0.00016, ips: 1.2745 samples/sec | ETA 00:28:30 2021-05-10 00:32:57 [INFO] [TRAIN] epoch: 102, iter: 37830/40000, loss: 0.1949, lr: 0.000819, batch_cost: 0.7843, reader_cost: 0.00017, ips: 1.2750 samples/sec | ETA 00:28:21 2021-05-10 00:33:05 [INFO] [TRAIN] epoch: 102, iter: 37840/40000, loss: 0.1739, lr: 0.000816, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2731 samples/sec | ETA 00:28:16 2021-05-10 00:33:12 [INFO] [TRAIN] epoch: 102, iter: 37850/40000, loss: 0.3531, lr: 0.000813, batch_cost: 0.7846, reader_cost: 0.00015, ips: 1.2746 samples/sec | ETA 00:28:06 2021-05-10 00:33:20 [INFO] [TRAIN] epoch: 102, iter: 37860/40000, loss: 0.3019, lr: 0.000810, batch_cost: 0.7843, reader_cost: 0.00017, ips: 1.2750 samples/sec | ETA 00:27:58 2021-05-10 00:33:28 [INFO] [TRAIN] epoch: 102, iter: 37870/40000, loss: 0.3240, lr: 0.000807, batch_cost: 0.7855, reader_cost: 0.00017, ips: 1.2731 samples/sec | ETA 00:27:53 2021-05-10 00:33:36 [INFO] [TRAIN] epoch: 102, iter: 37880/40000, loss: 0.3876, lr: 0.000804, batch_cost: 0.7864, reader_cost: 0.00017, ips: 1.2716 samples/sec | ETA 00:27:47 2021-05-10 00:33:44 [INFO] [TRAIN] epoch: 102, iter: 37890/40000, loss: 0.2275, lr: 0.000801, batch_cost: 0.7853, reader_cost: 0.00015, ips: 1.2734 samples/sec | ETA 00:27:36 2021-05-10 00:33:52 [INFO] [TRAIN] epoch: 102, iter: 37900/40000, loss: 0.1668, lr: 0.000798, batch_cost: 0.7853, reader_cost: 0.00018, ips: 1.2734 samples/sec | ETA 00:27:29 2021-05-10 00:34:00 [INFO] [TRAIN] epoch: 102, iter: 37910/40000, loss: 0.1846, lr: 0.000795, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2736 samples/sec | ETA 00:27:20 2021-05-10 00:34:07 [INFO] [TRAIN] epoch: 102, iter: 37920/40000, loss: 0.2253, lr: 0.000792, batch_cost: 0.7849, reader_cost: 0.00017, ips: 1.2740 samples/sec | ETA 00:27:12 2021-05-10 00:34:15 [INFO] [TRAIN] epoch: 102, iter: 37930/40000, loss: 0.2604, lr: 0.000789, batch_cost: 0.7860, reader_cost: 0.00017, ips: 1.2723 samples/sec | ETA 00:27:06 2021-05-10 00:34:23 [INFO] [TRAIN] epoch: 102, iter: 37940/40000, loss: 0.3351, lr: 0.000786, batch_cost: 0.7853, reader_cost: 0.00011, ips: 1.2735 samples/sec | ETA 00:26:57 2021-05-10 00:34:34 [INFO] [TRAIN] epoch: 103, iter: 37950/40000, loss: 0.6528, lr: 0.000783, batch_cost: 1.0998, reader_cost: 0.24701, ips: 0.9093 samples/sec | ETA 00:37:34 2021-05-10 00:34:42 [INFO] [TRAIN] epoch: 103, iter: 37960/40000, loss: 0.3307, lr: 0.000780, batch_cost: 0.8002, reader_cost: 0.00034, ips: 1.2496 samples/sec | ETA 00:27:12 2021-05-10 00:34:50 [INFO] [TRAIN] epoch: 103, iter: 37970/40000, loss: 0.3730, lr: 0.000777, batch_cost: 0.7848, reader_cost: 0.00015, ips: 1.2742 samples/sec | ETA 00:26:33 2021-05-10 00:34:58 [INFO] [TRAIN] epoch: 103, iter: 37980/40000, loss: 0.2926, lr: 0.000774, batch_cost: 0.7854, reader_cost: 0.00017, ips: 1.2733 samples/sec | ETA 00:26:26 2021-05-10 00:35:06 [INFO] [TRAIN] epoch: 103, iter: 37990/40000, loss: 0.2687, lr: 0.000771, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2727 samples/sec | ETA 00:26:19 2021-05-10 00:35:14 [INFO] [TRAIN] epoch: 103, iter: 38000/40000, loss: 0.2752, lr: 0.000768, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2718 samples/sec | ETA 00:26:12 2021-05-10 00:35:14 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-10 00:38:44 [INFO] [EVAL] #Images: 500 mIoU: 0.7703 Acc: 0.9576 Kappa: 0.9450 2021-05-10 00:38:44 [INFO] [EVAL] Class IoU: [0.9812 0.8472 0.9214 0.6307 0.6275 0.5038 0.6329 0.7314 0.9163 0.6426 0.9417 0.7821 0.5942 0.9428 0.852 0.8982 0.8118 0.6418 0.7363] 2021-05-10 00:38:44 [INFO] [EVAL] Class Acc: [0.992 0.914 0.953 0.8145 0.8275 0.7594 0.8082 0.8888 0.9474 0.8669 0.9645 0.8576 0.7671 0.9628 0.9452 0.962 0.886 0.8043 0.8406] 2021-05-10 00:39:13 [INFO] [EVAL] The model with the best validation mIoU (0.7708) was saved at iter 37000. 2021-05-10 00:39:20 [INFO] [TRAIN] epoch: 103, iter: 38010/40000, loss: 0.2781, lr: 0.000765, batch_cost: 0.7826, reader_cost: 0.00028, ips: 1.2778 samples/sec | ETA 00:25:57 2021-05-10 00:39:28 [INFO] [TRAIN] epoch: 103, iter: 38020/40000, loss: 0.1493, lr: 0.000762, batch_cost: 0.7838, reader_cost: 0.00016, ips: 1.2759 samples/sec | ETA 00:25:51 2021-05-10 00:39:36 [INFO] [TRAIN] epoch: 103, iter: 38030/40000, loss: 0.3821, lr: 0.000759, batch_cost: 0.7847, reader_cost: 0.00017, ips: 1.2744 samples/sec | ETA 00:25:45 2021-05-10 00:39:44 [INFO] [TRAIN] epoch: 103, iter: 38040/40000, loss: 0.2711, lr: 0.000756, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2736 samples/sec | ETA 00:25:38 2021-05-10 00:39:52 [INFO] [TRAIN] epoch: 103, iter: 38050/40000, loss: 0.2806, lr: 0.000753, batch_cost: 0.7855, reader_cost: 0.00015, ips: 1.2730 samples/sec | ETA 00:25:31 2021-05-10 00:40:00 [INFO] [TRAIN] epoch: 103, iter: 38060/40000, loss: 0.2070, lr: 0.000750, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2723 samples/sec | ETA 00:25:24 2021-05-10 00:40:07 [INFO] [TRAIN] epoch: 103, iter: 38070/40000, loss: 0.1437, lr: 0.000747, batch_cost: 0.7876, reader_cost: 0.00015, ips: 1.2696 samples/sec | ETA 00:25:20 2021-05-10 00:40:15 [INFO] [TRAIN] epoch: 103, iter: 38080/40000, loss: 0.3374, lr: 0.000744, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2712 samples/sec | ETA 00:25:10 2021-05-10 00:40:23 [INFO] [TRAIN] epoch: 103, iter: 38090/40000, loss: 0.2408, lr: 0.000741, batch_cost: 0.7862, reader_cost: 0.00015, ips: 1.2720 samples/sec | ETA 00:25:01 2021-05-10 00:40:31 [INFO] [TRAIN] epoch: 103, iter: 38100/40000, loss: 0.2384, lr: 0.000738, batch_cost: 0.7843, reader_cost: 0.00014, ips: 1.2750 samples/sec | ETA 00:24:50 2021-05-10 00:40:39 [INFO] [TRAIN] epoch: 103, iter: 38110/40000, loss: 0.1910, lr: 0.000735, batch_cost: 0.7872, reader_cost: 0.00015, ips: 1.2704 samples/sec | ETA 00:24:47 2021-05-10 00:40:47 [INFO] [TRAIN] epoch: 103, iter: 38120/40000, loss: 0.2203, lr: 0.000732, batch_cost: 0.7878, reader_cost: 0.00015, ips: 1.2693 samples/sec | ETA 00:24:41 2021-05-10 00:40:55 [INFO] [TRAIN] epoch: 103, iter: 38130/40000, loss: 0.1975, lr: 0.000729, batch_cost: 0.7875, reader_cost: 0.00015, ips: 1.2698 samples/sec | ETA 00:24:32 2021-05-10 00:41:03 [INFO] [TRAIN] epoch: 103, iter: 38140/40000, loss: 0.0918, lr: 0.000726, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2728 samples/sec | ETA 00:24:21 2021-05-10 00:41:10 [INFO] [TRAIN] epoch: 103, iter: 38150/40000, loss: 0.1971, lr: 0.000723, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2735 samples/sec | ETA 00:24:12 2021-05-10 00:41:18 [INFO] [TRAIN] epoch: 103, iter: 38160/40000, loss: 0.2873, lr: 0.000720, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 00:24:06 2021-05-10 00:41:26 [INFO] [TRAIN] epoch: 103, iter: 38170/40000, loss: 0.1675, lr: 0.000717, batch_cost: 0.7872, reader_cost: 0.00016, ips: 1.2704 samples/sec | ETA 00:24:00 2021-05-10 00:41:34 [INFO] [TRAIN] epoch: 103, iter: 38180/40000, loss: 0.2084, lr: 0.000714, batch_cost: 0.7849, reader_cost: 0.00015, ips: 1.2741 samples/sec | ETA 00:23:48 2021-05-10 00:41:42 [INFO] [TRAIN] epoch: 103, iter: 38190/40000, loss: 0.1888, lr: 0.000711, batch_cost: 0.7864, reader_cost: 0.00017, ips: 1.2716 samples/sec | ETA 00:23:43 2021-05-10 00:41:50 [INFO] [TRAIN] epoch: 103, iter: 38200/40000, loss: 0.2406, lr: 0.000708, batch_cost: 0.7850, reader_cost: 0.00015, ips: 1.2739 samples/sec | ETA 00:23:32 2021-05-10 00:41:58 [INFO] [TRAIN] epoch: 103, iter: 38210/40000, loss: 0.1889, lr: 0.000705, batch_cost: 0.7864, reader_cost: 0.00015, ips: 1.2716 samples/sec | ETA 00:23:27 2021-05-10 00:42:05 [INFO] [TRAIN] epoch: 103, iter: 38220/40000, loss: 0.2641, lr: 0.000702, batch_cost: 0.7845, reader_cost: 0.00016, ips: 1.2747 samples/sec | ETA 00:23:16 2021-05-10 00:42:13 [INFO] [TRAIN] epoch: 103, iter: 38230/40000, loss: 0.4112, lr: 0.000699, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2713 samples/sec | ETA 00:23:12 2021-05-10 00:42:21 [INFO] [TRAIN] epoch: 103, iter: 38240/40000, loss: 0.3197, lr: 0.000696, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 00:23:04 2021-05-10 00:42:29 [INFO] [TRAIN] epoch: 103, iter: 38250/40000, loss: 0.4137, lr: 0.000693, batch_cost: 0.7847, reader_cost: 0.00015, ips: 1.2744 samples/sec | ETA 00:22:53 2021-05-10 00:42:37 [INFO] [TRAIN] epoch: 103, iter: 38260/40000, loss: 0.3792, lr: 0.000690, batch_cost: 0.7844, reader_cost: 0.00016, ips: 1.2749 samples/sec | ETA 00:22:44 2021-05-10 00:42:45 [INFO] [TRAIN] epoch: 103, iter: 38270/40000, loss: 0.1919, lr: 0.000686, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2726 samples/sec | ETA 00:22:39 2021-05-10 00:42:53 [INFO] [TRAIN] epoch: 103, iter: 38280/40000, loss: 0.1428, lr: 0.000683, batch_cost: 0.7846, reader_cost: 0.00015, ips: 1.2745 samples/sec | ETA 00:22:29 2021-05-10 00:43:00 [INFO] [TRAIN] epoch: 103, iter: 38290/40000, loss: 0.1760, lr: 0.000680, batch_cost: 0.7875, reader_cost: 0.00017, ips: 1.2698 samples/sec | ETA 00:22:26 2021-05-10 00:43:08 [INFO] [TRAIN] epoch: 103, iter: 38300/40000, loss: 0.2821, lr: 0.000677, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2719 samples/sec | ETA 00:22:16 2021-05-10 00:43:16 [INFO] [TRAIN] epoch: 103, iter: 38310/40000, loss: 0.3245, lr: 0.000674, batch_cost: 0.7846, reader_cost: 0.00013, ips: 1.2746 samples/sec | ETA 00:22:05 2021-05-10 00:43:27 [INFO] [TRAIN] epoch: 104, iter: 38320/40000, loss: 0.3926, lr: 0.000671, batch_cost: 1.0906, reader_cost: 0.23697, ips: 0.9169 samples/sec | ETA 00:30:32 2021-05-10 00:43:35 [INFO] [TRAIN] epoch: 104, iter: 38330/40000, loss: 0.2698, lr: 0.000668, batch_cost: 0.7948, reader_cost: 0.00032, ips: 1.2581 samples/sec | ETA 00:22:07 2021-05-10 00:43:43 [INFO] [TRAIN] epoch: 104, iter: 38340/40000, loss: 0.5225, lr: 0.000665, batch_cost: 0.7847, reader_cost: 0.00017, ips: 1.2744 samples/sec | ETA 00:21:42 2021-05-10 00:43:51 [INFO] [TRAIN] epoch: 104, iter: 38350/40000, loss: 0.2177, lr: 0.000662, batch_cost: 0.7869, reader_cost: 0.00018, ips: 1.2709 samples/sec | ETA 00:21:38 2021-05-10 00:43:59 [INFO] [TRAIN] epoch: 104, iter: 38360/40000, loss: 0.3378, lr: 0.000659, batch_cost: 0.7882, reader_cost: 0.00017, ips: 1.2688 samples/sec | ETA 00:21:32 2021-05-10 00:44:06 [INFO] [TRAIN] epoch: 104, iter: 38370/40000, loss: 0.3135, lr: 0.000656, batch_cost: 0.7859, reader_cost: 0.00017, ips: 1.2724 samples/sec | ETA 00:21:20 2021-05-10 00:44:14 [INFO] [TRAIN] epoch: 104, iter: 38380/40000, loss: 0.2313, lr: 0.000653, batch_cost: 0.7863, reader_cost: 0.00017, ips: 1.2718 samples/sec | ETA 00:21:13 2021-05-10 00:44:22 [INFO] [TRAIN] epoch: 104, iter: 38390/40000, loss: 0.1333, lr: 0.000650, batch_cost: 0.7873, reader_cost: 0.00016, ips: 1.2702 samples/sec | ETA 00:21:07 2021-05-10 00:44:30 [INFO] [TRAIN] epoch: 104, iter: 38400/40000, loss: 0.2641, lr: 0.000647, batch_cost: 0.7850, reader_cost: 0.00017, ips: 1.2739 samples/sec | ETA 00:20:55 2021-05-10 00:44:38 [INFO] [TRAIN] epoch: 104, iter: 38410/40000, loss: 0.3061, lr: 0.000644, batch_cost: 0.7884, reader_cost: 0.00016, ips: 1.2684 samples/sec | ETA 00:20:53 2021-05-10 00:44:46 [INFO] [TRAIN] epoch: 104, iter: 38420/40000, loss: 0.1969, lr: 0.000641, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2713 samples/sec | ETA 00:20:42 2021-05-10 00:44:54 [INFO] [TRAIN] epoch: 104, iter: 38430/40000, loss: 0.2904, lr: 0.000637, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2730 samples/sec | ETA 00:20:33 2021-05-10 00:45:01 [INFO] [TRAIN] epoch: 104, iter: 38440/40000, loss: 0.1710, lr: 0.000634, batch_cost: 0.7868, reader_cost: 0.00016, ips: 1.2709 samples/sec | ETA 00:20:27 2021-05-10 00:45:09 [INFO] [TRAIN] epoch: 104, iter: 38450/40000, loss: 0.2118, lr: 0.000631, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2710 samples/sec | ETA 00:20:19 2021-05-10 00:45:17 [INFO] [TRAIN] epoch: 104, iter: 38460/40000, loss: 0.2276, lr: 0.000628, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2735 samples/sec | ETA 00:20:09 2021-05-10 00:45:25 [INFO] [TRAIN] epoch: 104, iter: 38470/40000, loss: 0.2790, lr: 0.000625, batch_cost: 0.7858, reader_cost: 0.00015, ips: 1.2725 samples/sec | ETA 00:20:02 2021-05-10 00:45:33 [INFO] [TRAIN] epoch: 104, iter: 38480/40000, loss: 0.2348, lr: 0.000622, batch_cost: 0.7839, reader_cost: 0.00015, ips: 1.2757 samples/sec | ETA 00:19:51 2021-05-10 00:45:41 [INFO] [TRAIN] epoch: 104, iter: 38490/40000, loss: 0.2438, lr: 0.000619, batch_cost: 0.7867, reader_cost: 0.00016, ips: 1.2711 samples/sec | ETA 00:19:47 2021-05-10 00:45:49 [INFO] [TRAIN] epoch: 104, iter: 38500/40000, loss: 0.2894, lr: 0.000616, batch_cost: 0.7880, reader_cost: 0.00016, ips: 1.2690 samples/sec | ETA 00:19:42 2021-05-10 00:45:57 [INFO] [TRAIN] epoch: 104, iter: 38510/40000, loss: 0.1153, lr: 0.000613, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2717 samples/sec | ETA 00:19:31 2021-05-10 00:46:04 [INFO] [TRAIN] epoch: 104, iter: 38520/40000, loss: 0.1805, lr: 0.000610, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2730 samples/sec | ETA 00:19:22 2021-05-10 00:46:12 [INFO] [TRAIN] epoch: 104, iter: 38530/40000, loss: 0.3059, lr: 0.000607, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2727 samples/sec | ETA 00:19:15 2021-05-10 00:46:20 [INFO] [TRAIN] epoch: 104, iter: 38540/40000, loss: 0.1586, lr: 0.000603, batch_cost: 0.7847, reader_cost: 0.00015, ips: 1.2743 samples/sec | ETA 00:19:05 2021-05-10 00:46:28 [INFO] [TRAIN] epoch: 104, iter: 38550/40000, loss: 0.2027, lr: 0.000600, batch_cost: 0.7850, reader_cost: 0.00016, ips: 1.2738 samples/sec | ETA 00:18:58 2021-05-10 00:46:36 [INFO] [TRAIN] epoch: 104, iter: 38560/40000, loss: 0.1476, lr: 0.000597, batch_cost: 0.7847, reader_cost: 0.00015, ips: 1.2744 samples/sec | ETA 00:18:49 2021-05-10 00:46:44 [INFO] [TRAIN] epoch: 104, iter: 38570/40000, loss: 0.2887, lr: 0.000594, batch_cost: 0.7849, reader_cost: 0.00017, ips: 1.2740 samples/sec | ETA 00:18:42 2021-05-10 00:46:52 [INFO] [TRAIN] epoch: 104, iter: 38580/40000, loss: 0.1076, lr: 0.000591, batch_cost: 0.7869, reader_cost: 0.00017, ips: 1.2709 samples/sec | ETA 00:18:37 2021-05-10 00:46:59 [INFO] [TRAIN] epoch: 104, iter: 38590/40000, loss: 0.3398, lr: 0.000588, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 00:18:28 2021-05-10 00:47:07 [INFO] [TRAIN] epoch: 104, iter: 38600/40000, loss: 0.4055, lr: 0.000585, batch_cost: 0.7845, reader_cost: 0.00016, ips: 1.2746 samples/sec | ETA 00:18:18 2021-05-10 00:47:15 [INFO] [TRAIN] epoch: 104, iter: 38610/40000, loss: 0.3180, lr: 0.000582, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2728 samples/sec | ETA 00:18:12 2021-05-10 00:47:23 [INFO] [TRAIN] epoch: 104, iter: 38620/40000, loss: 0.4799, lr: 0.000579, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2720 samples/sec | ETA 00:18:04 2021-05-10 00:47:31 [INFO] [TRAIN] epoch: 104, iter: 38630/40000, loss: 0.4993, lr: 0.000575, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2724 samples/sec | ETA 00:17:56 2021-05-10 00:47:39 [INFO] [TRAIN] epoch: 104, iter: 38640/40000, loss: 0.1831, lr: 0.000572, batch_cost: 0.7847, reader_cost: 0.00018, ips: 1.2743 samples/sec | ETA 00:17:47 2021-05-10 00:47:46 [INFO] [TRAIN] epoch: 104, iter: 38650/40000, loss: 0.1066, lr: 0.000569, batch_cost: 0.7856, reader_cost: 0.00017, ips: 1.2729 samples/sec | ETA 00:17:40 2021-05-10 00:47:54 [INFO] [TRAIN] epoch: 104, iter: 38660/40000, loss: 0.2744, lr: 0.000566, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2724 samples/sec | ETA 00:17:33 2021-05-10 00:48:02 [INFO] [TRAIN] epoch: 104, iter: 38670/40000, loss: 0.2683, lr: 0.000563, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2724 samples/sec | ETA 00:17:25 2021-05-10 00:48:10 [INFO] [TRAIN] epoch: 104, iter: 38680/40000, loss: 0.3330, lr: 0.000560, batch_cost: 0.7864, reader_cost: 0.00013, ips: 1.2716 samples/sec | ETA 00:17:18 2021-05-10 00:48:21 [INFO] [TRAIN] epoch: 105, iter: 38690/40000, loss: 0.2619, lr: 0.000557, batch_cost: 1.0849, reader_cost: 0.22355, ips: 0.9217 samples/sec | ETA 00:23:41 2021-05-10 00:48:29 [INFO] [TRAIN] epoch: 105, iter: 38700/40000, loss: 0.1980, lr: 0.000554, batch_cost: 0.7975, reader_cost: 0.00033, ips: 1.2540 samples/sec | ETA 00:17:16 2021-05-10 00:48:37 [INFO] [TRAIN] epoch: 105, iter: 38710/40000, loss: 0.5502, lr: 0.000550, batch_cost: 0.7853, reader_cost: 0.00017, ips: 1.2735 samples/sec | ETA 00:16:52 2021-05-10 00:48:45 [INFO] [TRAIN] epoch: 105, iter: 38720/40000, loss: 0.1835, lr: 0.000547, batch_cost: 0.7863, reader_cost: 0.00018, ips: 1.2717 samples/sec | ETA 00:16:46 2021-05-10 00:48:52 [INFO] [TRAIN] epoch: 105, iter: 38730/40000, loss: 0.2965, lr: 0.000544, batch_cost: 0.7858, reader_cost: 0.00018, ips: 1.2725 samples/sec | ETA 00:16:38 2021-05-10 00:49:00 [INFO] [TRAIN] epoch: 105, iter: 38740/40000, loss: 0.4532, lr: 0.000541, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2716 samples/sec | ETA 00:16:30 2021-05-10 00:49:08 [INFO] [TRAIN] epoch: 105, iter: 38750/40000, loss: 0.2353, lr: 0.000538, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2714 samples/sec | ETA 00:16:23 2021-05-10 00:49:16 [INFO] [TRAIN] epoch: 105, iter: 38760/40000, loss: 0.0940, lr: 0.000535, batch_cost: 0.7878, reader_cost: 0.00017, ips: 1.2693 samples/sec | ETA 00:16:16 2021-05-10 00:49:24 [INFO] [TRAIN] epoch: 105, iter: 38770/40000, loss: 0.2297, lr: 0.000532, batch_cost: 0.7848, reader_cost: 0.00016, ips: 1.2742 samples/sec | ETA 00:16:05 2021-05-10 00:49:32 [INFO] [TRAIN] epoch: 105, iter: 38780/40000, loss: 0.2605, lr: 0.000528, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2734 samples/sec | ETA 00:15:58 2021-05-10 00:49:40 [INFO] [TRAIN] epoch: 105, iter: 38790/40000, loss: 0.2518, lr: 0.000525, batch_cost: 0.7855, reader_cost: 0.00018, ips: 1.2731 samples/sec | ETA 00:15:50 2021-05-10 00:49:47 [INFO] [TRAIN] epoch: 105, iter: 38800/40000, loss: 0.3015, lr: 0.000522, batch_cost: 0.7842, reader_cost: 0.00017, ips: 1.2751 samples/sec | ETA 00:15:41 2021-05-10 00:49:55 [INFO] [TRAIN] epoch: 105, iter: 38810/40000, loss: 0.1178, lr: 0.000519, batch_cost: 0.7849, reader_cost: 0.00017, ips: 1.2741 samples/sec | ETA 00:15:33 2021-05-10 00:50:03 [INFO] [TRAIN] epoch: 105, iter: 38820/40000, loss: 0.3881, lr: 0.000516, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2713 samples/sec | ETA 00:15:28 2021-05-10 00:50:11 [INFO] [TRAIN] epoch: 105, iter: 38830/40000, loss: 0.3350, lr: 0.000513, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2729 samples/sec | ETA 00:15:19 2021-05-10 00:50:19 [INFO] [TRAIN] epoch: 105, iter: 38840/40000, loss: 0.1745, lr: 0.000509, batch_cost: 0.7856, reader_cost: 0.00015, ips: 1.2729 samples/sec | ETA 00:15:11 2021-05-10 00:50:27 [INFO] [TRAIN] epoch: 105, iter: 38850/40000, loss: 0.2312, lr: 0.000506, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2726 samples/sec | ETA 00:15:03 2021-05-10 00:50:35 [INFO] [TRAIN] epoch: 105, iter: 38860/40000, loss: 0.1474, lr: 0.000503, batch_cost: 0.7849, reader_cost: 0.00015, ips: 1.2740 samples/sec | ETA 00:14:54 2021-05-10 00:50:42 [INFO] [TRAIN] epoch: 105, iter: 38870/40000, loss: 0.3881, lr: 0.000500, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2713 samples/sec | ETA 00:14:48 2021-05-10 00:50:50 [INFO] [TRAIN] epoch: 105, iter: 38880/40000, loss: 0.1273, lr: 0.000497, batch_cost: 0.7853, reader_cost: 0.00014, ips: 1.2733 samples/sec | ETA 00:14:39 2021-05-10 00:50:58 [INFO] [TRAIN] epoch: 105, iter: 38890/40000, loss: 0.1728, lr: 0.000493, batch_cost: 0.7852, reader_cost: 0.00014, ips: 1.2735 samples/sec | ETA 00:14:31 2021-05-10 00:51:06 [INFO] [TRAIN] epoch: 105, iter: 38900/40000, loss: 0.3691, lr: 0.000490, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2723 samples/sec | ETA 00:14:24 2021-05-10 00:51:14 [INFO] [TRAIN] epoch: 105, iter: 38910/40000, loss: 0.1539, lr: 0.000487, batch_cost: 0.7863, reader_cost: 0.00018, ips: 1.2717 samples/sec | ETA 00:14:17 2021-05-10 00:51:22 [INFO] [TRAIN] epoch: 105, iter: 38920/40000, loss: 0.2012, lr: 0.000484, batch_cost: 0.7854, reader_cost: 0.00017, ips: 1.2732 samples/sec | ETA 00:14:08 2021-05-10 00:51:30 [INFO] [TRAIN] epoch: 105, iter: 38930/40000, loss: 0.1661, lr: 0.000481, batch_cost: 0.7861, reader_cost: 0.00018, ips: 1.2721 samples/sec | ETA 00:14:01 2021-05-10 00:51:38 [INFO] [TRAIN] epoch: 105, iter: 38940/40000, loss: 0.2238, lr: 0.000478, batch_cost: 0.7870, reader_cost: 0.00018, ips: 1.2707 samples/sec | ETA 00:13:54 2021-05-10 00:51:45 [INFO] [TRAIN] epoch: 105, iter: 38950/40000, loss: 0.1267, lr: 0.000474, batch_cost: 0.7851, reader_cost: 0.00016, ips: 1.2738 samples/sec | ETA 00:13:44 2021-05-10 00:51:53 [INFO] [TRAIN] epoch: 105, iter: 38960/40000, loss: 0.2549, lr: 0.000471, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2732 samples/sec | ETA 00:13:36 2021-05-10 00:52:01 [INFO] [TRAIN] epoch: 105, iter: 38970/40000, loss: 0.4307, lr: 0.000468, batch_cost: 0.7847, reader_cost: 0.00016, ips: 1.2744 samples/sec | ETA 00:13:28 2021-05-10 00:52:09 [INFO] [TRAIN] epoch: 105, iter: 38980/40000, loss: 0.2856, lr: 0.000465, batch_cost: 0.7861, reader_cost: 0.00017, ips: 1.2722 samples/sec | ETA 00:13:21 2021-05-10 00:52:17 [INFO] [TRAIN] epoch: 105, iter: 38990/40000, loss: 0.4726, lr: 0.000461, batch_cost: 0.7869, reader_cost: 0.00017, ips: 1.2709 samples/sec | ETA 00:13:14 2021-05-10 00:52:25 [INFO] [TRAIN] epoch: 105, iter: 39000/40000, loss: 0.4181, lr: 0.000458, batch_cost: 0.7871, reader_cost: 0.00016, ips: 1.2705 samples/sec | ETA 00:13:07 2021-05-10 00:52:25 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-10 00:55:55 [INFO] [EVAL] #Images: 500 mIoU: 0.7730 Acc: 0.9579 Kappa: 0.9454 2021-05-10 00:55:55 [INFO] [EVAL] Class IoU: [0.9814 0.8487 0.9223 0.6344 0.6308 0.5045 0.6321 0.7338 0.9167 0.6549 0.9412 0.7811 0.5857 0.9443 0.8596 0.8978 0.8232 0.6548 0.7388] 2021-05-10 00:55:55 [INFO] [EVAL] Class Acc: [0.9919 0.9153 0.9541 0.8177 0.8104 0.7572 0.8104 0.8814 0.9496 0.8381 0.962 0.8544 0.7829 0.967 0.9387 0.9563 0.9022 0.8209 0.8414] 2021-05-10 00:56:43 [INFO] [EVAL] The model with the best validation mIoU (0.7730) was saved at iter 39000. 2021-05-10 00:56:51 [INFO] [TRAIN] epoch: 105, iter: 39010/40000, loss: 0.2928, lr: 0.000455, batch_cost: 0.7804, reader_cost: 0.00027, ips: 1.2814 samples/sec | ETA 00:12:52 2021-05-10 00:56:59 [INFO] [TRAIN] epoch: 105, iter: 39020/40000, loss: 0.1462, lr: 0.000452, batch_cost: 0.7821, reader_cost: 0.00029, ips: 1.2786 samples/sec | ETA 00:12:46 2021-05-10 00:57:06 [INFO] [TRAIN] epoch: 105, iter: 39030/40000, loss: 0.3144, lr: 0.000449, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 00:12:42 2021-05-10 00:57:14 [INFO] [TRAIN] epoch: 105, iter: 39040/40000, loss: 0.2205, lr: 0.000445, batch_cost: 0.7837, reader_cost: 0.00016, ips: 1.2761 samples/sec | ETA 00:12:32 2021-05-10 00:57:22 [INFO] [TRAIN] epoch: 105, iter: 39050/40000, loss: 0.3002, lr: 0.000442, batch_cost: 0.7853, reader_cost: 0.00014, ips: 1.2734 samples/sec | ETA 00:12:26 2021-05-10 00:57:30 [INFO] [TRAIN] epoch: 105, iter: 39060/40000, loss: 0.3057, lr: 0.000439, batch_cost: 0.7845, reader_cost: 0.00010, ips: 1.2748 samples/sec | ETA 00:12:17 2021-05-10 00:57:41 [INFO] [TRAIN] epoch: 106, iter: 39070/40000, loss: 0.2178, lr: 0.000436, batch_cost: 1.0979, reader_cost: 0.24314, ips: 0.9108 samples/sec | ETA 00:17:01 2021-05-10 00:57:49 [INFO] [TRAIN] epoch: 106, iter: 39080/40000, loss: 0.4099, lr: 0.000432, batch_cost: 0.7863, reader_cost: 0.00033, ips: 1.2717 samples/sec | ETA 00:12:03 2021-05-10 00:57:57 [INFO] [TRAIN] epoch: 106, iter: 39090/40000, loss: 0.1796, lr: 0.000429, batch_cost: 0.7871, reader_cost: 0.00017, ips: 1.2705 samples/sec | ETA 00:11:56 2021-05-10 00:58:05 [INFO] [TRAIN] epoch: 106, iter: 39100/40000, loss: 0.3555, lr: 0.000426, batch_cost: 0.7856, reader_cost: 0.00016, ips: 1.2729 samples/sec | ETA 00:11:47 2021-05-10 00:58:12 [INFO] [TRAIN] epoch: 106, iter: 39110/40000, loss: 0.4852, lr: 0.000423, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2722 samples/sec | ETA 00:11:39 2021-05-10 00:58:20 [INFO] [TRAIN] epoch: 106, iter: 39120/40000, loss: 0.2528, lr: 0.000419, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2725 samples/sec | ETA 00:11:31 2021-05-10 00:58:28 [INFO] [TRAIN] epoch: 106, iter: 39130/40000, loss: 0.1591, lr: 0.000416, batch_cost: 0.7847, reader_cost: 0.00017, ips: 1.2743 samples/sec | ETA 00:11:22 2021-05-10 00:58:36 [INFO] [TRAIN] epoch: 106, iter: 39140/40000, loss: 0.1970, lr: 0.000413, batch_cost: 0.7855, reader_cost: 0.00017, ips: 1.2731 samples/sec | ETA 00:11:15 2021-05-10 00:58:44 [INFO] [TRAIN] epoch: 106, iter: 39150/40000, loss: 0.3580, lr: 0.000410, batch_cost: 0.7861, reader_cost: 0.00017, ips: 1.2721 samples/sec | ETA 00:11:08 2021-05-10 00:58:52 [INFO] [TRAIN] epoch: 106, iter: 39160/40000, loss: 0.2238, lr: 0.000406, batch_cost: 0.7850, reader_cost: 0.00015, ips: 1.2740 samples/sec | ETA 00:10:59 2021-05-10 00:58:59 [INFO] [TRAIN] epoch: 106, iter: 39170/40000, loss: 0.2383, lr: 0.000403, batch_cost: 0.7849, reader_cost: 0.00017, ips: 1.2740 samples/sec | ETA 00:10:51 2021-05-10 00:59:07 [INFO] [TRAIN] epoch: 106, iter: 39180/40000, loss: 0.1187, lr: 0.000400, batch_cost: 0.7865, reader_cost: 0.00017, ips: 1.2715 samples/sec | ETA 00:10:44 2021-05-10 00:59:15 [INFO] [TRAIN] epoch: 106, iter: 39190/40000, loss: 0.2512, lr: 0.000396, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2710 samples/sec | ETA 00:10:37 2021-05-10 00:59:23 [INFO] [TRAIN] epoch: 106, iter: 39200/40000, loss: 0.3208, lr: 0.000393, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2722 samples/sec | ETA 00:10:28 2021-05-10 00:59:31 [INFO] [TRAIN] epoch: 106, iter: 39210/40000, loss: 0.2955, lr: 0.000390, batch_cost: 0.7849, reader_cost: 0.00018, ips: 1.2741 samples/sec | ETA 00:10:20 2021-05-10 00:59:39 [INFO] [TRAIN] epoch: 106, iter: 39220/40000, loss: 0.2837, lr: 0.000387, batch_cost: 0.7859, reader_cost: 0.00017, ips: 1.2725 samples/sec | ETA 00:10:12 2021-05-10 00:59:47 [INFO] [TRAIN] epoch: 106, iter: 39230/40000, loss: 0.2426, lr: 0.000383, batch_cost: 0.7868, reader_cost: 0.00015, ips: 1.2710 samples/sec | ETA 00:10:05 2021-05-10 00:59:55 [INFO] [TRAIN] epoch: 106, iter: 39240/40000, loss: 0.3651, lr: 0.000380, batch_cost: 0.7854, reader_cost: 0.00015, ips: 1.2732 samples/sec | ETA 00:09:56 2021-05-10 01:00:02 [INFO] [TRAIN] epoch: 106, iter: 39250/40000, loss: 0.1127, lr: 0.000377, batch_cost: 0.7882, reader_cost: 0.00016, ips: 1.2688 samples/sec | ETA 00:09:51 2021-05-10 01:00:10 [INFO] [TRAIN] epoch: 106, iter: 39260/40000, loss: 0.0675, lr: 0.000373, batch_cost: 0.7860, reader_cost: 0.00016, ips: 1.2723 samples/sec | ETA 00:09:41 2021-05-10 01:00:18 [INFO] [TRAIN] epoch: 106, iter: 39270/40000, loss: 0.3593, lr: 0.000370, batch_cost: 0.7847, reader_cost: 0.00016, ips: 1.2744 samples/sec | ETA 00:09:32 2021-05-10 01:00:26 [INFO] [TRAIN] epoch: 106, iter: 39280/40000, loss: 0.2312, lr: 0.000367, batch_cost: 0.7866, reader_cost: 0.00016, ips: 1.2712 samples/sec | ETA 00:09:26 2021-05-10 01:00:34 [INFO] [TRAIN] epoch: 106, iter: 39290/40000, loss: 0.2317, lr: 0.000363, batch_cost: 0.7849, reader_cost: 0.00016, ips: 1.2740 samples/sec | ETA 00:09:17 2021-05-10 01:00:42 [INFO] [TRAIN] epoch: 106, iter: 39300/40000, loss: 0.2481, lr: 0.000360, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 00:09:10 2021-05-10 01:00:50 [INFO] [TRAIN] epoch: 106, iter: 39310/40000, loss: 0.1593, lr: 0.000357, batch_cost: 0.7849, reader_cost: 0.00016, ips: 1.2741 samples/sec | ETA 00:09:01 2021-05-10 01:00:57 [INFO] [TRAIN] epoch: 106, iter: 39320/40000, loss: 0.1679, lr: 0.000353, batch_cost: 0.7858, reader_cost: 0.00015, ips: 1.2726 samples/sec | ETA 00:08:54 2021-05-10 01:01:05 [INFO] [TRAIN] epoch: 106, iter: 39330/40000, loss: 0.1937, lr: 0.000350, batch_cost: 0.7839, reader_cost: 0.00016, ips: 1.2756 samples/sec | ETA 00:08:45 2021-05-10 01:01:13 [INFO] [TRAIN] epoch: 106, iter: 39340/40000, loss: 0.4617, lr: 0.000347, batch_cost: 0.7877, reader_cost: 0.00015, ips: 1.2695 samples/sec | ETA 00:08:39 2021-05-10 01:01:21 [INFO] [TRAIN] epoch: 106, iter: 39350/40000, loss: 0.3766, lr: 0.000343, batch_cost: 0.7847, reader_cost: 0.00015, ips: 1.2744 samples/sec | ETA 00:08:30 2021-05-10 01:01:29 [INFO] [TRAIN] epoch: 106, iter: 39360/40000, loss: 0.3897, lr: 0.000340, batch_cost: 0.7858, reader_cost: 0.00015, ips: 1.2725 samples/sec | ETA 00:08:22 2021-05-10 01:01:37 [INFO] [TRAIN] epoch: 106, iter: 39370/40000, loss: 0.4415, lr: 0.000336, batch_cost: 0.7843, reader_cost: 0.00017, ips: 1.2750 samples/sec | ETA 00:08:14 2021-05-10 01:01:45 [INFO] [TRAIN] epoch: 106, iter: 39380/40000, loss: 0.2930, lr: 0.000333, batch_cost: 0.7863, reader_cost: 0.00015, ips: 1.2717 samples/sec | ETA 00:08:07 2021-05-10 01:01:52 [INFO] [TRAIN] epoch: 106, iter: 39390/40000, loss: 0.2126, lr: 0.000330, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2733 samples/sec | ETA 00:07:59 2021-05-10 01:02:00 [INFO] [TRAIN] epoch: 106, iter: 39400/40000, loss: 0.2204, lr: 0.000326, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2733 samples/sec | ETA 00:07:51 2021-05-10 01:02:08 [INFO] [TRAIN] epoch: 106, iter: 39410/40000, loss: 0.3366, lr: 0.000323, batch_cost: 0.7861, reader_cost: 0.00017, ips: 1.2720 samples/sec | ETA 00:07:43 2021-05-10 01:02:16 [INFO] [TRAIN] epoch: 106, iter: 39420/40000, loss: 0.3130, lr: 0.000320, batch_cost: 0.7869, reader_cost: 0.00016, ips: 1.2709 samples/sec | ETA 00:07:36 2021-05-10 01:02:24 [INFO] [TRAIN] epoch: 106, iter: 39430/40000, loss: 0.2531, lr: 0.000316, batch_cost: 0.7852, reader_cost: 0.00027, ips: 1.2736 samples/sec | ETA 00:07:27 2021-05-10 01:02:35 [INFO] [TRAIN] epoch: 107, iter: 39440/40000, loss: 0.2550, lr: 0.000313, batch_cost: 1.1050, reader_cost: 0.24247, ips: 0.9050 samples/sec | ETA 00:10:18 2021-05-10 01:02:43 [INFO] [TRAIN] epoch: 107, iter: 39450/40000, loss: 0.3148, lr: 0.000309, batch_cost: 0.7952, reader_cost: 0.00031, ips: 1.2575 samples/sec | ETA 00:07:17 2021-05-10 01:02:51 [INFO] [TRAIN] epoch: 107, iter: 39460/40000, loss: 0.4089, lr: 0.000306, batch_cost: 0.7855, reader_cost: 0.00017, ips: 1.2730 samples/sec | ETA 00:07:04 2021-05-10 01:02:59 [INFO] [TRAIN] epoch: 107, iter: 39470/40000, loss: 0.3013, lr: 0.000302, batch_cost: 0.7862, reader_cost: 0.00016, ips: 1.2719 samples/sec | ETA 00:06:56 2021-05-10 01:03:06 [INFO] [TRAIN] epoch: 107, iter: 39480/40000, loss: 0.3993, lr: 0.000299, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2724 samples/sec | ETA 00:06:48 2021-05-10 01:03:14 [INFO] [TRAIN] epoch: 107, iter: 39490/40000, loss: 0.2922, lr: 0.000296, batch_cost: 0.7881, reader_cost: 0.00015, ips: 1.2690 samples/sec | ETA 00:06:41 2021-05-10 01:03:22 [INFO] [TRAIN] epoch: 107, iter: 39500/40000, loss: 0.2807, lr: 0.000292, batch_cost: 0.7878, reader_cost: 0.00015, ips: 1.2693 samples/sec | ETA 00:06:33 2021-05-10 01:03:30 [INFO] [TRAIN] epoch: 107, iter: 39510/40000, loss: 0.1251, lr: 0.000289, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2724 samples/sec | ETA 00:06:25 2021-05-10 01:03:38 [INFO] [TRAIN] epoch: 107, iter: 39520/40000, loss: 0.3170, lr: 0.000285, batch_cost: 0.7877, reader_cost: 0.00015, ips: 1.2695 samples/sec | ETA 00:06:18 2021-05-10 01:03:46 [INFO] [TRAIN] epoch: 107, iter: 39530/40000, loss: 0.2084, lr: 0.000282, batch_cost: 0.7866, reader_cost: 0.00018, ips: 1.2712 samples/sec | ETA 00:06:09 2021-05-10 01:03:54 [INFO] [TRAIN] epoch: 107, iter: 39540/40000, loss: 0.2486, lr: 0.000278, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2722 samples/sec | ETA 00:06:01 2021-05-10 01:04:01 [INFO] [TRAIN] epoch: 107, iter: 39550/40000, loss: 0.1830, lr: 0.000275, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2736 samples/sec | ETA 00:05:53 2021-05-10 01:04:09 [INFO] [TRAIN] epoch: 107, iter: 39560/40000, loss: 0.2420, lr: 0.000271, batch_cost: 0.7870, reader_cost: 0.00016, ips: 1.2707 samples/sec | ETA 00:05:46 2021-05-10 01:04:17 [INFO] [TRAIN] epoch: 107, iter: 39570/40000, loss: 0.3055, lr: 0.000268, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2721 samples/sec | ETA 00:05:38 2021-05-10 01:04:25 [INFO] [TRAIN] epoch: 107, iter: 39580/40000, loss: 0.2225, lr: 0.000264, batch_cost: 0.7853, reader_cost: 0.00016, ips: 1.2735 samples/sec | ETA 00:05:29 2021-05-10 01:04:33 [INFO] [TRAIN] epoch: 107, iter: 39590/40000, loss: 0.2349, lr: 0.000261, batch_cost: 0.7851, reader_cost: 0.00015, ips: 1.2738 samples/sec | ETA 00:05:21 2021-05-10 01:04:41 [INFO] [TRAIN] epoch: 107, iter: 39600/40000, loss: 0.1903, lr: 0.000257, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 00:05:14 2021-05-10 01:04:49 [INFO] [TRAIN] epoch: 107, iter: 39610/40000, loss: 0.1909, lr: 0.000254, batch_cost: 0.7871, reader_cost: 0.00015, ips: 1.2705 samples/sec | ETA 00:05:06 2021-05-10 01:04:57 [INFO] [TRAIN] epoch: 107, iter: 39620/40000, loss: 0.2320, lr: 0.000250, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 00:04:58 2021-05-10 01:05:04 [INFO] [TRAIN] epoch: 107, iter: 39630/40000, loss: 0.0813, lr: 0.000247, batch_cost: 0.7867, reader_cost: 0.00015, ips: 1.2711 samples/sec | ETA 00:04:51 2021-05-10 01:05:12 [INFO] [TRAIN] epoch: 107, iter: 39640/40000, loss: 0.2305, lr: 0.000243, batch_cost: 0.7854, reader_cost: 0.00015, ips: 1.2732 samples/sec | ETA 00:04:42 2021-05-10 01:05:20 [INFO] [TRAIN] epoch: 107, iter: 39650/40000, loss: 0.2738, lr: 0.000239, batch_cost: 0.7848, reader_cost: 0.00014, ips: 1.2742 samples/sec | ETA 00:04:34 2021-05-10 01:05:28 [INFO] [TRAIN] epoch: 107, iter: 39660/40000, loss: 0.4533, lr: 0.000236, batch_cost: 0.7850, reader_cost: 0.00015, ips: 1.2740 samples/sec | ETA 00:04:26 2021-05-10 01:05:36 [INFO] [TRAIN] epoch: 107, iter: 39670/40000, loss: 0.2009, lr: 0.000232, batch_cost: 0.7845, reader_cost: 0.00015, ips: 1.2747 samples/sec | ETA 00:04:18 2021-05-10 01:05:44 [INFO] [TRAIN] epoch: 107, iter: 39680/40000, loss: 0.1309, lr: 0.000229, batch_cost: 0.7846, reader_cost: 0.00015, ips: 1.2745 samples/sec | ETA 00:04:11 2021-05-10 01:05:51 [INFO] [TRAIN] epoch: 107, iter: 39690/40000, loss: 0.2305, lr: 0.000225, batch_cost: 0.7860, reader_cost: 0.00015, ips: 1.2723 samples/sec | ETA 00:04:03 2021-05-10 01:05:59 [INFO] [TRAIN] epoch: 107, iter: 39700/40000, loss: 0.1754, lr: 0.000221, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2714 samples/sec | ETA 00:03:55 2021-05-10 01:06:07 [INFO] [TRAIN] epoch: 107, iter: 39710/40000, loss: 0.3082, lr: 0.000218, batch_cost: 0.7854, reader_cost: 0.00016, ips: 1.2732 samples/sec | ETA 00:03:47 2021-05-10 01:06:15 [INFO] [TRAIN] epoch: 107, iter: 39720/40000, loss: 0.3840, lr: 0.000214, batch_cost: 0.7863, reader_cost: 0.00016, ips: 1.2718 samples/sec | ETA 00:03:40 2021-05-10 01:06:23 [INFO] [TRAIN] epoch: 107, iter: 39730/40000, loss: 0.3031, lr: 0.000211, batch_cost: 0.7855, reader_cost: 0.00016, ips: 1.2731 samples/sec | ETA 00:03:32 2021-05-10 01:06:31 [INFO] [TRAIN] epoch: 107, iter: 39740/40000, loss: 0.3738, lr: 0.000207, batch_cost: 0.7861, reader_cost: 0.00016, ips: 1.2720 samples/sec | ETA 00:03:24 2021-05-10 01:06:39 [INFO] [TRAIN] epoch: 107, iter: 39750/40000, loss: 0.4077, lr: 0.000203, batch_cost: 0.7837, reader_cost: 0.00016, ips: 1.2760 samples/sec | ETA 00:03:15 2021-05-10 01:06:46 [INFO] [TRAIN] epoch: 107, iter: 39760/40000, loss: 0.1153, lr: 0.000199, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 00:03:08 2021-05-10 01:06:54 [INFO] [TRAIN] epoch: 107, iter: 39770/40000, loss: 0.1781, lr: 0.000196, batch_cost: 0.7859, reader_cost: 0.00015, ips: 1.2723 samples/sec | ETA 00:03:00 2021-05-10 01:07:02 [INFO] [TRAIN] epoch: 107, iter: 39780/40000, loss: 0.2257, lr: 0.000192, batch_cost: 0.7859, reader_cost: 0.00016, ips: 1.2724 samples/sec | ETA 00:02:52 2021-05-10 01:07:10 [INFO] [TRAIN] epoch: 107, iter: 39790/40000, loss: 0.2526, lr: 0.000188, batch_cost: 0.7865, reader_cost: 0.00015, ips: 1.2715 samples/sec | ETA 00:02:45 2021-05-10 01:07:18 [INFO] [TRAIN] epoch: 107, iter: 39800/40000, loss: 0.3361, lr: 0.000184, batch_cost: 0.7847, reader_cost: 0.00012, ips: 1.2744 samples/sec | ETA 00:02:36 2021-05-10 01:07:29 [INFO] [TRAIN] epoch: 108, iter: 39810/40000, loss: 0.3469, lr: 0.000181, batch_cost: 1.0802, reader_cost: 0.24811, ips: 0.9257 samples/sec | ETA 00:03:25 2021-05-10 01:07:37 [INFO] [TRAIN] epoch: 108, iter: 39820/40000, loss: 0.3311, lr: 0.000177, batch_cost: 0.7959, reader_cost: 0.00035, ips: 1.2564 samples/sec | ETA 00:02:23 2021-05-10 01:07:45 [INFO] [TRAIN] epoch: 108, iter: 39830/40000, loss: 0.3406, lr: 0.000173, batch_cost: 0.7859, reader_cost: 0.00019, ips: 1.2725 samples/sec | ETA 00:02:13 2021-05-10 01:07:52 [INFO] [TRAIN] epoch: 108, iter: 39840/40000, loss: 0.3092, lr: 0.000169, batch_cost: 0.7873, reader_cost: 0.00016, ips: 1.2702 samples/sec | ETA 00:02:05 2021-05-10 01:08:00 [INFO] [TRAIN] epoch: 108, iter: 39850/40000, loss: 0.3528, lr: 0.000165, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 00:01:57 2021-05-10 01:08:08 [INFO] [TRAIN] epoch: 108, iter: 39860/40000, loss: 0.3197, lr: 0.000161, batch_cost: 0.7865, reader_cost: 0.00016, ips: 1.2715 samples/sec | ETA 00:01:50 2021-05-10 01:08:16 [INFO] [TRAIN] epoch: 108, iter: 39870/40000, loss: 0.2512, lr: 0.000157, batch_cost: 0.7858, reader_cost: 0.00016, ips: 1.2726 samples/sec | ETA 00:01:42 2021-05-10 01:08:24 [INFO] [TRAIN] epoch: 108, iter: 39880/40000, loss: 0.1380, lr: 0.000153, batch_cost: 0.7864, reader_cost: 0.00016, ips: 1.2717 samples/sec | ETA 00:01:34 2021-05-10 01:08:32 [INFO] [TRAIN] epoch: 108, iter: 39890/40000, loss: 0.2251, lr: 0.000149, batch_cost: 0.7861, reader_cost: 0.00015, ips: 1.2720 samples/sec | ETA 00:01:26 2021-05-10 01:08:40 [INFO] [TRAIN] epoch: 108, iter: 39900/40000, loss: 0.2215, lr: 0.000145, batch_cost: 0.7857, reader_cost: 0.00016, ips: 1.2728 samples/sec | ETA 00:01:18 2021-05-10 01:08:47 [INFO] [TRAIN] epoch: 108, iter: 39910/40000, loss: 0.2637, lr: 0.000141, batch_cost: 0.7870, reader_cost: 0.00017, ips: 1.2706 samples/sec | ETA 00:01:10 2021-05-10 01:08:55 [INFO] [TRAIN] epoch: 108, iter: 39920/40000, loss: 0.2626, lr: 0.000137, batch_cost: 0.7872, reader_cost: 0.00015, ips: 1.2702 samples/sec | ETA 00:01:02 2021-05-10 01:09:03 [INFO] [TRAIN] epoch: 108, iter: 39930/40000, loss: 0.2174, lr: 0.000133, batch_cost: 0.7871, reader_cost: 0.00014, ips: 1.2705 samples/sec | ETA 00:00:55 2021-05-10 01:09:11 [INFO] [TRAIN] epoch: 108, iter: 39940/40000, loss: 0.3223, lr: 0.000129, batch_cost: 0.7875, reader_cost: 0.00015, ips: 1.2699 samples/sec | ETA 00:00:47 2021-05-10 01:09:19 [INFO] [TRAIN] epoch: 108, iter: 39950/40000, loss: 0.2240, lr: 0.000125, batch_cost: 0.7857, reader_cost: 0.00015, ips: 1.2727 samples/sec | ETA 00:00:39 2021-05-10 01:09:27 [INFO] [TRAIN] epoch: 108, iter: 39960/40000, loss: 0.2985, lr: 0.000120, batch_cost: 0.7866, reader_cost: 0.00015, ips: 1.2713 samples/sec | ETA 00:00:31 2021-05-10 01:09:35 [INFO] [TRAIN] epoch: 108, iter: 39970/40000, loss: 0.2273, lr: 0.000116, batch_cost: 0.7844, reader_cost: 0.00016, ips: 1.2748 samples/sec | ETA 00:00:23 2021-05-10 01:09:43 [INFO] [TRAIN] epoch: 108, iter: 39980/40000, loss: 0.1670, lr: 0.000111, batch_cost: 0.7865, reader_cost: 0.00017, ips: 1.2714 samples/sec | ETA 00:00:15 2021-05-10 01:09:50 [INFO] [TRAIN] epoch: 108, iter: 39990/40000, loss: 0.1660, lr: 0.000106, batch_cost: 0.7852, reader_cost: 0.00016, ips: 1.2735 samples/sec | ETA 00:00:07 2021-05-10 01:09:58 [INFO] [TRAIN] epoch: 108, iter: 40000/40000, loss: 0.0497, lr: 0.000101, batch_cost: 0.7854, reader_cost: 0.00015, ips: 1.2732 samples/sec | ETA 00:00:00 2021-05-10 01:09:58 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-10 01:13:30 [INFO] [EVAL] #Images: 500 mIoU: 0.7729 Acc: 0.9573 Kappa: 0.9447 2021-05-10 01:13:30 [INFO] [EVAL] Class IoU: [0.9798 0.8398 0.9224 0.6377 0.6313 0.5067 0.6343 0.733 0.9167 0.6503 0.9418 0.7832 0.5901 0.9441 0.8579 0.8991 0.8251 0.6537 0.7382] 2021-05-10 01:13:30 [INFO] [EVAL] Class Acc: [0.9926 0.8997 0.9551 0.818 0.8175 0.7571 0.7998 0.8829 0.9479 0.8503 0.9641 0.8605 0.7766 0.9659 0.9441 0.9555 0.913 0.8171 0.8341] 2021-05-10 01:14:04 [INFO] [EVAL] The model with the best validation mIoU (0.7730) was saved at iter 39000. 's flops has been counted 's flops has been counted Cannot find suitable count function for . Treat it as zero FLOPs. 's flops has been counted Cannot find suitable count function for . Treat it as zero FLOPs. Cannot find suitable count function for . Treat it as zero FLOPs. Customize Function has been applied to /ssd1/home/wuzewu/miniconda3/envs/paddle2/lib/python3.7/site-packages/paddle/fluid/dygraph/math_op_patch.py:238: UserWarning: The dtype of left and right variables are not the same, left dtype is VarType.FP32, but right dtype is VarType.INT32, the right dtype will convert to VarType.FP32 format(lhs_dtype, rhs_dtype, lhs_dtype)) Total Flops: -203727126528 Total Params: 303367187 INFO 2021-05-10 01:14:08,863 launch.py:240] Local processes completed.