----------- Configuration Arguments ----------- gpus: None heter_worker_num: None heter_workers: http_port: None ips: 127.0.0.1 log_dir: log nproc_per_node: None server_num: None servers: training_script: train.py training_script_args: ['--do_eval', '--use_vdl', '--batch_size', '1', '--learning_rate', '0.01', '--config', 'configs/setr/setr_pup_large_cityscapes_769x769_40k.yml', '--save_interval', '1000', '--num_workers', '6', '--save_dir', 'log/setr_pup_large/exp11/'] worker_num: None workers: ------------------------------------------------ WARNING 2021-05-06 11:20:37,575 launch.py:316] Not found distinct arguments and compiled with cuda. Default use collective mode launch train in GPU mode INFO 2021-05-06 11:20:37,576 launch_utils.py:471] Local start 8 processes. First process distributed environment info (Only For Debug): +=======================================================================================+ | Distributed Envs Value | +---------------------------------------------------------------------------------------+ | PADDLE_TRAINER_ID 0 | | PADDLE_CURRENT_ENDPOINT 127.0.0.1:33920 | | PADDLE_TRAINERS_NUM 8 | | PADDLE_TRAINER_ENDPOINTS ... 0.1:54899,127.0.0.1:35768,127.0.0.1:48987| | FLAGS_selected_gpus 0 | +=======================================================================================+ INFO 2021-05-06 11:20:37,576 launch_utils.py:475] details abouts PADDLE_TRAINER_ENDPOINTS can be found in log/endpoints.log, and detail running logs maybe found in log/workerlog.0 which: no nvcc in (/ssd1/home/wuzewu/miniconda3/envs/paddle2/bin:/ssd1/home/wuzewu/miniconda3/condabin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/opt/bin:/home/opt/bin:/usr/local/sbin:/usr/sbin:/opt/bin:/home/opt/bin:/ssd1/home/wuzewu/.local/bin:/ssd1/home/wuzewu/bin:/opt/bin:/home/opt/bin:/opt/bin:/home/opt/bin:/ssd1/home/wuzewu/.local/bin:/ssd1/home/wuzewu/bin:/opt/bin:/home/opt/bin) 2021-05-06 11:20:39 [INFO] ------------Environment Information------------- platform: Linux-3.10.0-1062.18.1.el7.x86_64-x86_64-with-centos-7.7.1908-Core Python: 3.7.9 (default, Aug 31 2020, 12:42:55) [GCC 7.3.0] Paddle compiled with cuda: True NVCC: Cuda compilation tools, release 10.2, V10.2.89 cudnn: 7.6 GPUs used: 8 CUDA_VISIBLE_DEVICES: 0,1,2,3,4,5,6,7 GPU: ['GPU 0: Tesla V100-SXM2-32GB', 'GPU 1: Tesla V100-SXM2-32GB', 'GPU 2: Tesla V100-SXM2-32GB', 'GPU 3: Tesla V100-SXM2-32GB', 'GPU 4: Tesla V100-SXM2-32GB', 'GPU 5: Tesla V100-SXM2-32GB', 'GPU 6: Tesla V100-SXM2-32GB', 'GPU 7: Tesla V100-SXM2-32GB'] GCC: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39) PaddlePaddle: 2.0.1 OpenCV: 4.4.0 ------------------------------------------------ 2021-05-06 11:20:39 [INFO] ---------------Config Information--------------- batch_size: 1 iters: 40000 loss: coef: - 1 - 0.4 - 0.4 - 0.4 - 0.4 types: - ignore_index: 255 type: CrossEntropyLoss - ignore_index: 255 type: CrossEntropyLoss - ignore_index: 255 type: CrossEntropyLoss - ignore_index: 255 type: CrossEntropyLoss - ignore_index: 255 type: CrossEntropyLoss lr_scheduler: end_lr: 0.0001 learning_rate: 0.01 power: 0.9 type: PolynomialDecay model: align_corners: true backbone: pretrained: vit/ViT_large_patch16_384/model.pdparams type: ViT_large_patch16_384 backbone_indices: - 9 - 14 - 19 - 23 head: pup num_classes: 19 type: SegmentationTransformer optimizer: momentum: 0.9 type: sgd weight_decay: 4.0e-05 train_dataset: dataset_root: data/cityscapes mode: train transforms: - max_scale_factor: 2.0 min_scale_factor: 0.25 scale_step_size: 0.25 type: ResizeStepScaling - crop_size: - 769 - 769 type: RandomPaddingCrop - type: RandomHorizontalFlip - brightness_range: 0.5 contrast_range: 0.5 saturation_range: 0.5 type: RandomDistort - type: Normalize type: Cityscapes val_dataset: dataset_root: data/cityscapes mode: val transforms: - target_size: - 2048 - 1024 type: Padding - type: Normalize type: Cityscapes ------------------------------------------------ W0506 11:20:39.424291 16674 device_context.cc:362] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 10.2, Runtime API Version: 10.2 W0506 11:20:39.424342 16674 device_context.cc:372] device: 0, cuDNN Version: 7.6. 2021-05-06 11:20:43 [INFO] Loading pretrained model from vit/ViT_large_patch16_384/model.pdparams 2021-05-06 11:20:54 [INFO] There are 294/294 variables loaded into VisionTransformer. W0506 11:20:54.905038 16674 nccl_context.cc:142] Socket connect worker 127.0.0.1:33059 failed, try again after 3 seconds. I0506 11:20:57.905522 16674 nccl_context.cc:189] init nccl context nranks: 8 local rank: 0 gpu id: 0 ring id: 0 2021-05-06 11:21:11 [INFO] [TRAIN] epoch: 1, iter: 10/40000, loss: 4.9753, lr: 0.009998, batch_cost: 1.1755, reader_cost: 0.21297, ips: 0.8507 samples/sec | ETA 13:03:27 2021-05-06 11:21:19 [INFO] [TRAIN] epoch: 1, iter: 20/40000, loss: 6.4355, lr: 0.009996, batch_cost: 0.8318, reader_cost: 0.00034, ips: 1.2022 samples/sec | ETA 09:14:15 2021-05-06 11:21:28 [INFO] [TRAIN] epoch: 1, iter: 30/40000, loss: 6.1834, lr: 0.009994, batch_cost: 0.8303, reader_cost: 0.00015, ips: 1.2043 samples/sec | ETA 09:13:09 2021-05-06 11:21:36 [INFO] [TRAIN] epoch: 1, iter: 40/40000, loss: 3.8547, lr: 0.009991, batch_cost: 0.8322, reader_cost: 0.00016, ips: 1.2016 samples/sec | ETA 09:14:14 2021-05-06 11:21:44 [INFO] [TRAIN] epoch: 1, iter: 50/40000, loss: 4.4163, lr: 0.009989, batch_cost: 0.8340, reader_cost: 0.00014, ips: 1.1991 samples/sec | ETA 09:15:16 2021-05-06 11:21:53 [INFO] [TRAIN] epoch: 1, iter: 60/40000, loss: 5.1870, lr: 0.009987, batch_cost: 0.8337, reader_cost: 0.00014, ips: 1.1995 samples/sec | ETA 09:14:56 2021-05-06 11:22:01 [INFO] [TRAIN] epoch: 1, iter: 70/40000, loss: 4.5324, lr: 0.009985, batch_cost: 0.8383, reader_cost: 0.00015, ips: 1.1928 samples/sec | ETA 09:17:54 2021-05-06 11:22:09 [INFO] [TRAIN] epoch: 1, iter: 80/40000, loss: 5.1388, lr: 0.009982, batch_cost: 0.8368, reader_cost: 0.00014, ips: 1.1950 samples/sec | ETA 09:16:46 2021-05-06 11:22:18 [INFO] [TRAIN] epoch: 1, iter: 90/40000, loss: 6.0569, lr: 0.009980, batch_cost: 0.8375, reader_cost: 0.00014, ips: 1.1940 samples/sec | ETA 09:17:05 2021-05-06 11:22:26 [INFO] [TRAIN] epoch: 1, iter: 100/40000, loss: 2.6992, lr: 0.009978, batch_cost: 0.8376, reader_cost: 0.00015, ips: 1.1939 samples/sec | ETA 09:17:00 2021-05-06 11:22:34 [INFO] [TRAIN] epoch: 1, iter: 110/40000, loss: 4.1018, lr: 0.009976, batch_cost: 0.8377, reader_cost: 0.00015, ips: 1.1938 samples/sec | ETA 09:16:54 2021-05-06 11:22:43 [INFO] [TRAIN] epoch: 1, iter: 120/40000, loss: 4.8946, lr: 0.009973, batch_cost: 0.8372, reader_cost: 0.00016, ips: 1.1944 samples/sec | ETA 09:16:28 2021-05-06 11:22:51 [INFO] [TRAIN] epoch: 1, iter: 130/40000, loss: 3.8572, lr: 0.009971, batch_cost: 0.8368, reader_cost: 0.00015, ips: 1.1951 samples/sec | ETA 09:16:01 2021-05-06 11:23:00 [INFO] [TRAIN] epoch: 1, iter: 140/40000, loss: 4.0018, lr: 0.009969, batch_cost: 0.8393, reader_cost: 0.00014, ips: 1.1914 samples/sec | ETA 09:17:35 2021-05-06 11:23:08 [INFO] [TRAIN] epoch: 1, iter: 150/40000, loss: 4.9929, lr: 0.009967, batch_cost: 0.8379, reader_cost: 0.00014, ips: 1.1935 samples/sec | ETA 09:16:29 2021-05-06 11:23:16 [INFO] [TRAIN] epoch: 1, iter: 160/40000, loss: 3.6596, lr: 0.009965, batch_cost: 0.8373, reader_cost: 0.00016, ips: 1.1943 samples/sec | ETA 09:15:59 2021-05-06 11:23:25 [INFO] [TRAIN] epoch: 1, iter: 170/40000, loss: 3.6618, lr: 0.009962, batch_cost: 0.8388, reader_cost: 0.00015, ips: 1.1922 samples/sec | ETA 09:16:48 2021-05-06 11:23:33 [INFO] [TRAIN] epoch: 1, iter: 180/40000, loss: 4.5246, lr: 0.009960, batch_cost: 0.8372, reader_cost: 0.00015, ips: 1.1944 samples/sec | ETA 09:15:39 2021-05-06 11:23:41 [INFO] [TRAIN] epoch: 1, iter: 190/40000, loss: 4.3497, lr: 0.009958, batch_cost: 0.8371, reader_cost: 0.00016, ips: 1.1946 samples/sec | ETA 09:15:23 2021-05-06 11:23:50 [INFO] [TRAIN] epoch: 1, iter: 200/40000, loss: 5.0912, lr: 0.009956, batch_cost: 0.8390, reader_cost: 0.00016, ips: 1.1919 samples/sec | ETA 09:16:33 2021-05-06 11:23:58 [INFO] [TRAIN] epoch: 1, iter: 210/40000, loss: 3.4211, lr: 0.009953, batch_cost: 0.8390, reader_cost: 0.00016, ips: 1.1919 samples/sec | ETA 09:16:22 2021-05-06 11:24:07 [INFO] [TRAIN] epoch: 1, iter: 220/40000, loss: 3.4079, lr: 0.009951, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1914 samples/sec | ETA 09:16:29 2021-05-06 11:24:15 [INFO] [TRAIN] epoch: 1, iter: 230/40000, loss: 3.1011, lr: 0.009949, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 09:17:00 2021-05-06 11:24:23 [INFO] [TRAIN] epoch: 1, iter: 240/40000, loss: 2.5969, lr: 0.009947, batch_cost: 0.8388, reader_cost: 0.00014, ips: 1.1922 samples/sec | ETA 09:15:50 2021-05-06 11:24:32 [INFO] [TRAIN] epoch: 1, iter: 250/40000, loss: 2.5664, lr: 0.009945, batch_cost: 0.8378, reader_cost: 0.00014, ips: 1.1936 samples/sec | ETA 09:15:02 2021-05-06 11:24:40 [INFO] [TRAIN] epoch: 1, iter: 260/40000, loss: 2.5907, lr: 0.009942, batch_cost: 0.8389, reader_cost: 0.00015, ips: 1.1920 samples/sec | ETA 09:15:39 2021-05-06 11:24:49 [INFO] [TRAIN] epoch: 1, iter: 270/40000, loss: 2.3824, lr: 0.009940, batch_cost: 0.8383, reader_cost: 0.00015, ips: 1.1929 samples/sec | ETA 09:15:05 2021-05-06 11:24:57 [INFO] [TRAIN] epoch: 1, iter: 280/40000, loss: 4.7335, lr: 0.009938, batch_cost: 0.8371, reader_cost: 0.00014, ips: 1.1946 samples/sec | ETA 09:14:09 2021-05-06 11:25:05 [INFO] [TRAIN] epoch: 1, iter: 290/40000, loss: 3.8717, lr: 0.009936, batch_cost: 0.8394, reader_cost: 0.00016, ips: 1.1913 samples/sec | ETA 09:15:33 2021-05-06 11:25:14 [INFO] [TRAIN] epoch: 1, iter: 300/40000, loss: 2.9522, lr: 0.009933, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 09:15:40 2021-05-06 11:25:22 [INFO] [TRAIN] epoch: 1, iter: 310/40000, loss: 2.8792, lr: 0.009931, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 09:16:23 2021-05-06 11:25:31 [INFO] [TRAIN] epoch: 1, iter: 320/40000, loss: 3.2020, lr: 0.009929, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 09:15:23 2021-05-06 11:25:39 [INFO] [TRAIN] epoch: 1, iter: 330/40000, loss: 2.3632, lr: 0.009927, batch_cost: 0.8391, reader_cost: 0.00016, ips: 1.1917 samples/sec | ETA 09:14:48 2021-05-06 11:25:47 [INFO] [TRAIN] epoch: 1, iter: 340/40000, loss: 2.8249, lr: 0.009924, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1903 samples/sec | ETA 09:15:17 2021-05-06 11:25:56 [INFO] [TRAIN] epoch: 1, iter: 350/40000, loss: 2.7946, lr: 0.009922, batch_cost: 0.8400, reader_cost: 0.00014, ips: 1.1905 samples/sec | ETA 09:15:06 2021-05-06 11:26:04 [INFO] [TRAIN] epoch: 1, iter: 360/40000, loss: 2.5891, lr: 0.009920, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1917 samples/sec | ETA 09:14:22 2021-05-06 11:26:12 [INFO] [TRAIN] epoch: 1, iter: 370/40000, loss: 2.5736, lr: 0.009918, batch_cost: 0.8371, reader_cost: 0.00028, ips: 1.1946 samples/sec | ETA 09:12:54 2021-05-06 11:26:24 [INFO] [TRAIN] epoch: 2, iter: 380/40000, loss: 3.0126, lr: 0.009916, batch_cost: 1.1522, reader_cost: 0.18344, ips: 0.8679 samples/sec | ETA 12:40:49 2021-05-06 11:26:32 [INFO] [TRAIN] epoch: 2, iter: 390/40000, loss: 2.9945, lr: 0.009913, batch_cost: 0.8410, reader_cost: 0.00031, ips: 1.1890 samples/sec | ETA 09:15:13 2021-05-06 11:26:41 [INFO] [TRAIN] epoch: 2, iter: 400/40000, loss: 2.4269, lr: 0.009911, batch_cost: 0.8373, reader_cost: 0.00015, ips: 1.1944 samples/sec | ETA 09:12:35 2021-05-06 11:26:49 [INFO] [TRAIN] epoch: 2, iter: 410/40000, loss: 2.3694, lr: 0.009909, batch_cost: 0.8396, reader_cost: 0.00014, ips: 1.1911 samples/sec | ETA 09:13:57 2021-05-06 11:26:58 [INFO] [TRAIN] epoch: 2, iter: 420/40000, loss: 2.5952, lr: 0.009907, batch_cost: 0.8392, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 09:13:37 2021-05-06 11:27:06 [INFO] [TRAIN] epoch: 2, iter: 430/40000, loss: 2.8804, lr: 0.009904, batch_cost: 0.8386, reader_cost: 0.00015, ips: 1.1924 samples/sec | ETA 09:13:05 2021-05-06 11:27:14 [INFO] [TRAIN] epoch: 2, iter: 440/40000, loss: 2.9594, lr: 0.009902, batch_cost: 0.8376, reader_cost: 0.00015, ips: 1.1938 samples/sec | ETA 09:12:16 2021-05-06 11:27:23 [INFO] [TRAIN] epoch: 2, iter: 450/40000, loss: 2.5706, lr: 0.009900, batch_cost: 0.8385, reader_cost: 0.00015, ips: 1.1926 samples/sec | ETA 09:12:43 2021-05-06 11:27:31 [INFO] [TRAIN] epoch: 2, iter: 460/40000, loss: 1.9711, lr: 0.009898, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 09:13:45 2021-05-06 11:27:40 [INFO] [TRAIN] epoch: 2, iter: 470/40000, loss: 3.2776, lr: 0.009895, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 09:13:47 2021-05-06 11:27:48 [INFO] [TRAIN] epoch: 2, iter: 480/40000, loss: 1.3437, lr: 0.009893, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1888 samples/sec | ETA 09:14:02 2021-05-06 11:27:56 [INFO] [TRAIN] epoch: 2, iter: 490/40000, loss: 3.6743, lr: 0.009891, batch_cost: 0.8413, reader_cost: 0.00016, ips: 1.1886 samples/sec | ETA 09:14:00 2021-05-06 11:28:05 [INFO] [TRAIN] epoch: 2, iter: 500/40000, loss: 2.3886, lr: 0.009889, batch_cost: 0.8410, reader_cost: 0.00016, ips: 1.1891 samples/sec | ETA 09:13:38 2021-05-06 11:28:13 [INFO] [TRAIN] epoch: 2, iter: 510/40000, loss: 1.9915, lr: 0.009887, batch_cost: 0.8397, reader_cost: 0.00016, ips: 1.1909 samples/sec | ETA 09:12:41 2021-05-06 11:28:22 [INFO] [TRAIN] epoch: 2, iter: 520/40000, loss: 2.0350, lr: 0.009884, batch_cost: 0.8418, reader_cost: 0.00016, ips: 1.1879 samples/sec | ETA 09:13:54 2021-05-06 11:28:30 [INFO] [TRAIN] epoch: 2, iter: 530/40000, loss: 1.6839, lr: 0.009882, batch_cost: 0.8413, reader_cost: 0.00016, ips: 1.1886 samples/sec | ETA 09:13:27 2021-05-06 11:28:38 [INFO] [TRAIN] epoch: 2, iter: 540/40000, loss: 1.7324, lr: 0.009880, batch_cost: 0.8417, reader_cost: 0.00017, ips: 1.1880 samples/sec | ETA 09:13:34 2021-05-06 11:28:47 [INFO] [TRAIN] epoch: 2, iter: 550/40000, loss: 2.5879, lr: 0.009878, batch_cost: 0.8430, reader_cost: 0.00017, ips: 1.1863 samples/sec | ETA 09:14:15 2021-05-06 11:28:55 [INFO] [TRAIN] epoch: 2, iter: 560/40000, loss: 3.7594, lr: 0.009875, batch_cost: 0.8402, reader_cost: 0.00017, ips: 1.1902 samples/sec | ETA 09:12:18 2021-05-06 11:29:04 [INFO] [TRAIN] epoch: 2, iter: 570/40000, loss: 2.5010, lr: 0.009873, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1917 samples/sec | ETA 09:11:26 2021-05-06 11:29:12 [INFO] [TRAIN] epoch: 2, iter: 580/40000, loss: 2.4545, lr: 0.009871, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 09:12:36 2021-05-06 11:29:20 [INFO] [TRAIN] epoch: 2, iter: 590/40000, loss: 3.3104, lr: 0.009869, batch_cost: 0.8388, reader_cost: 0.00015, ips: 1.1922 samples/sec | ETA 09:10:56 2021-05-06 11:29:29 [INFO] [TRAIN] epoch: 2, iter: 600/40000, loss: 1.7378, lr: 0.009866, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 09:11:30 2021-05-06 11:29:37 [INFO] [TRAIN] epoch: 2, iter: 610/40000, loss: 2.5403, lr: 0.009864, batch_cost: 0.8395, reader_cost: 0.00016, ips: 1.1911 samples/sec | ETA 09:11:09 2021-05-06 11:29:46 [INFO] [TRAIN] epoch: 2, iter: 620/40000, loss: 1.4491, lr: 0.009862, batch_cost: 0.8398, reader_cost: 0.00016, ips: 1.1907 samples/sec | ETA 09:11:13 2021-05-06 11:29:54 [INFO] [TRAIN] epoch: 2, iter: 630/40000, loss: 1.4715, lr: 0.009860, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 09:11:20 2021-05-06 11:30:02 [INFO] [TRAIN] epoch: 2, iter: 640/40000, loss: 1.5364, lr: 0.009858, batch_cost: 0.8387, reader_cost: 0.00015, ips: 1.1923 samples/sec | ETA 09:10:11 2021-05-06 11:30:11 [INFO] [TRAIN] epoch: 2, iter: 650/40000, loss: 1.8482, lr: 0.009855, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1914 samples/sec | ETA 09:10:29 2021-05-06 11:30:19 [INFO] [TRAIN] epoch: 2, iter: 660/40000, loss: 1.5075, lr: 0.009853, batch_cost: 0.8396, reader_cost: 0.00016, ips: 1.1911 samples/sec | ETA 09:10:28 2021-05-06 11:30:28 [INFO] [TRAIN] epoch: 2, iter: 670/40000, loss: 2.9318, lr: 0.009851, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 09:10:54 2021-05-06 11:30:36 [INFO] [TRAIN] epoch: 2, iter: 680/40000, loss: 1.8555, lr: 0.009849, batch_cost: 0.8396, reader_cost: 0.00016, ips: 1.1911 samples/sec | ETA 09:10:11 2021-05-06 11:30:44 [INFO] [TRAIN] epoch: 2, iter: 690/40000, loss: 2.1029, lr: 0.009846, batch_cost: 0.8398, reader_cost: 0.00014, ips: 1.1908 samples/sec | ETA 09:10:11 2021-05-06 11:30:53 [INFO] [TRAIN] epoch: 2, iter: 700/40000, loss: 1.9306, lr: 0.009844, batch_cost: 0.8389, reader_cost: 0.00014, ips: 1.1921 samples/sec | ETA 09:09:27 2021-05-06 11:31:01 [INFO] [TRAIN] epoch: 2, iter: 710/40000, loss: 2.2506, lr: 0.009842, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1903 samples/sec | ETA 09:10:09 2021-05-06 11:31:10 [INFO] [TRAIN] epoch: 2, iter: 720/40000, loss: 2.4760, lr: 0.009840, batch_cost: 0.8381, reader_cost: 0.00014, ips: 1.1932 samples/sec | ETA 09:08:39 2021-05-06 11:31:18 [INFO] [TRAIN] epoch: 2, iter: 730/40000, loss: 2.0416, lr: 0.009837, batch_cost: 0.8391, reader_cost: 0.00014, ips: 1.1917 samples/sec | ETA 09:09:13 2021-05-06 11:31:26 [INFO] [TRAIN] epoch: 2, iter: 740/40000, loss: 1.7031, lr: 0.009835, batch_cost: 0.8374, reader_cost: 0.00012, ips: 1.1942 samples/sec | ETA 09:07:55 2021-05-06 11:31:38 [INFO] [TRAIN] epoch: 3, iter: 750/40000, loss: 2.7194, lr: 0.009833, batch_cost: 1.1327, reader_cost: 0.18447, ips: 0.8829 samples/sec | ETA 12:20:57 2021-05-06 11:31:46 [INFO] [TRAIN] epoch: 3, iter: 760/40000, loss: 1.9431, lr: 0.009831, batch_cost: 0.8392, reader_cost: 0.00032, ips: 1.1916 samples/sec | ETA 09:08:49 2021-05-06 11:31:54 [INFO] [TRAIN] epoch: 3, iter: 770/40000, loss: 1.4777, lr: 0.009829, batch_cost: 0.8381, reader_cost: 0.00015, ips: 1.1932 samples/sec | ETA 09:07:59 2021-05-06 11:32:03 [INFO] [TRAIN] epoch: 3, iter: 780/40000, loss: 1.6626, lr: 0.009826, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 09:09:44 2021-05-06 11:32:11 [INFO] [TRAIN] epoch: 3, iter: 790/40000, loss: 1.3146, lr: 0.009824, batch_cost: 0.8431, reader_cost: 0.00015, ips: 1.1861 samples/sec | ETA 09:10:57 2021-05-06 11:32:20 [INFO] [TRAIN] epoch: 3, iter: 800/40000, loss: 2.2029, lr: 0.009822, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 09:09:04 2021-05-06 11:32:28 [INFO] [TRAIN] epoch: 3, iter: 810/40000, loss: 1.8239, lr: 0.009820, batch_cost: 0.8408, reader_cost: 0.00016, ips: 1.1894 samples/sec | ETA 09:09:09 2021-05-06 11:32:37 [INFO] [TRAIN] epoch: 3, iter: 820/40000, loss: 1.9769, lr: 0.009817, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 09:08:20 2021-05-06 11:32:45 [INFO] [TRAIN] epoch: 3, iter: 830/40000, loss: 2.0018, lr: 0.009815, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 09:08:21 2021-05-06 11:32:53 [INFO] [TRAIN] epoch: 3, iter: 840/40000, loss: 2.3874, lr: 0.009813, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 09:08:20 2021-05-06 11:33:02 [INFO] [TRAIN] epoch: 3, iter: 850/40000, loss: 1.1590, lr: 0.009811, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 09:07:38 2021-05-06 11:33:10 [INFO] [TRAIN] epoch: 3, iter: 860/40000, loss: 1.2109, lr: 0.009808, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1887 samples/sec | ETA 09:08:45 2021-05-06 11:33:19 [INFO] [TRAIN] epoch: 3, iter: 870/40000, loss: 1.6696, lr: 0.009806, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1911 samples/sec | ETA 09:07:33 2021-05-06 11:33:27 [INFO] [TRAIN] epoch: 3, iter: 880/40000, loss: 1.6334, lr: 0.009804, batch_cost: 0.8378, reader_cost: 0.00014, ips: 1.1937 samples/sec | ETA 09:06:13 2021-05-06 11:33:35 [INFO] [TRAIN] epoch: 3, iter: 890/40000, loss: 2.3196, lr: 0.009802, batch_cost: 0.8384, reader_cost: 0.00014, ips: 1.1928 samples/sec | ETA 09:06:28 2021-05-06 11:33:44 [INFO] [TRAIN] epoch: 3, iter: 900/40000, loss: 2.0968, lr: 0.009800, batch_cost: 0.8377, reader_cost: 0.00015, ips: 1.1937 samples/sec | ETA 09:05:55 2021-05-06 11:33:52 [INFO] [TRAIN] epoch: 3, iter: 910/40000, loss: 1.4074, lr: 0.009797, batch_cost: 0.8389, reader_cost: 0.00016, ips: 1.1920 samples/sec | ETA 09:06:32 2021-05-06 11:34:00 [INFO] [TRAIN] epoch: 3, iter: 920/40000, loss: 1.1969, lr: 0.009795, batch_cost: 0.8382, reader_cost: 0.00015, ips: 1.1931 samples/sec | ETA 09:05:54 2021-05-06 11:34:09 [INFO] [TRAIN] epoch: 3, iter: 930/40000, loss: 1.0078, lr: 0.009793, batch_cost: 0.8380, reader_cost: 0.00015, ips: 1.1934 samples/sec | ETA 09:05:38 2021-05-06 11:34:17 [INFO] [TRAIN] epoch: 3, iter: 940/40000, loss: 2.0676, lr: 0.009791, batch_cost: 0.8392, reader_cost: 0.00014, ips: 1.1915 samples/sec | ETA 09:06:20 2021-05-06 11:34:26 [INFO] [TRAIN] epoch: 3, iter: 950/40000, loss: 1.7430, lr: 0.009788, batch_cost: 0.8408, reader_cost: 0.00016, ips: 1.1894 samples/sec | ETA 09:07:12 2021-05-06 11:34:34 [INFO] [TRAIN] epoch: 3, iter: 960/40000, loss: 1.7717, lr: 0.009786, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1918 samples/sec | ETA 09:05:58 2021-05-06 11:34:42 [INFO] [TRAIN] epoch: 3, iter: 970/40000, loss: 0.9032, lr: 0.009784, batch_cost: 0.8389, reader_cost: 0.00015, ips: 1.1920 samples/sec | ETA 09:05:42 2021-05-06 11:34:51 [INFO] [TRAIN] epoch: 3, iter: 980/40000, loss: 1.3303, lr: 0.009782, batch_cost: 0.8382, reader_cost: 0.00015, ips: 1.1930 samples/sec | ETA 09:05:07 2021-05-06 11:34:59 [INFO] [TRAIN] epoch: 3, iter: 990/40000, loss: 0.9279, lr: 0.009779, batch_cost: 0.8385, reader_cost: 0.00014, ips: 1.1926 samples/sec | ETA 09:05:11 2021-05-06 11:35:08 [INFO] [TRAIN] epoch: 3, iter: 1000/40000, loss: 1.3107, lr: 0.009777, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 09:05:52 2021-05-06 11:35:08 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... /ssd1/home/wuzewu/miniconda3/envs/paddle2/lib/python3.7/site-packages/paddle/fluid/dygraph/math_op_patch.py:238: UserWarning: The dtype of left and right variables are not the same, left dtype is VarType.INT32, but right dtype is VarType.BOOL, the right dtype will convert to VarType.INT32 format(lhs_dtype, rhs_dtype, lhs_dtype)) /ssd1/home/wuzewu/miniconda3/envs/paddle2/lib/python3.7/site-packages/paddle/fluid/dygraph/math_op_patch.py:238: UserWarning: The dtype of left and right variables are not the same, left dtype is VarType.INT64, but right dtype is VarType.BOOL, the right dtype will convert to VarType.INT64 format(lhs_dtype, rhs_dtype, lhs_dtype)) 2021-05-06 11:38:44 [INFO] [EVAL] #Images: 500 mIoU: 0.2760 Acc: 0.8604 Kappa: 0.8149 2021-05-06 11:38:44 [INFO] [EVAL] Class IoU: [0.9138 0.4116 0.7508 0.0039 0. 0. 0. 0. 0.8165 0.3417 0.8569 0.3936 0. 0.754 0. 0. 0. 0. 0.0012] 2021-05-06 11:38:44 [INFO] [EVAL] Class Acc: [0.9297 0.7524 0.774 0.5588 0. 0. 0. 0. 0.9207 0.4322 0.8948 0.5123 0. 0.8469 0. 0. 0. 0. 0.6083] 2021-05-06 11:39:26 [INFO] [EVAL] The model with the best validation mIoU (0.2760) was saved at iter 1000. 2021-05-06 11:39:34 [INFO] [TRAIN] epoch: 3, iter: 1010/40000, loss: 1.3331, lr: 0.009775, batch_cost: 0.8353, reader_cost: 0.00034, ips: 1.1972 samples/sec | ETA 09:02:48 2021-05-06 11:39:43 [INFO] [TRAIN] epoch: 3, iter: 1020/40000, loss: 1.1838, lr: 0.009773, batch_cost: 0.8362, reader_cost: 0.00015, ips: 1.1959 samples/sec | ETA 09:03:14 2021-05-06 11:39:51 [INFO] [TRAIN] epoch: 3, iter: 1030/40000, loss: 1.3877, lr: 0.009770, batch_cost: 0.8378, reader_cost: 0.00015, ips: 1.1936 samples/sec | ETA 09:04:10 2021-05-06 11:40:01 [INFO] [TRAIN] epoch: 3, iter: 1040/40000, loss: 1.3641, lr: 0.009768, batch_cost: 0.8397, reader_cost: 0.00016, ips: 1.1908 samples/sec | ETA 09:05:16 2021-05-06 11:40:09 [INFO] [TRAIN] epoch: 3, iter: 1050/40000, loss: 1.5336, lr: 0.009766, batch_cost: 0.8385, reader_cost: 0.00015, ips: 1.1926 samples/sec | ETA 09:04:20 2021-05-06 11:40:18 [INFO] [TRAIN] epoch: 3, iter: 1060/40000, loss: 1.1915, lr: 0.009764, batch_cost: 0.8386, reader_cost: 0.00015, ips: 1.1925 samples/sec | ETA 09:04:15 2021-05-06 11:40:26 [INFO] [TRAIN] epoch: 3, iter: 1070/40000, loss: 1.4751, lr: 0.009762, batch_cost: 0.8384, reader_cost: 0.00016, ips: 1.1928 samples/sec | ETA 09:03:58 2021-05-06 11:40:34 [INFO] [TRAIN] epoch: 3, iter: 1080/40000, loss: 0.8842, lr: 0.009759, batch_cost: 0.8425, reader_cost: 0.00015, ips: 1.1869 samples/sec | ETA 09:06:31 2021-05-06 11:40:43 [INFO] [TRAIN] epoch: 3, iter: 1090/40000, loss: 1.2821, lr: 0.009757, batch_cost: 0.8409, reader_cost: 0.00016, ips: 1.1892 samples/sec | ETA 09:05:19 2021-05-06 11:40:51 [INFO] [TRAIN] epoch: 3, iter: 1100/40000, loss: 1.0553, lr: 0.009755, batch_cost: 0.8401, reader_cost: 0.00016, ips: 1.1903 samples/sec | ETA 09:04:41 2021-05-06 11:41:00 [INFO] [TRAIN] epoch: 3, iter: 1110/40000, loss: 1.3501, lr: 0.009753, batch_cost: 0.8379, reader_cost: 0.00011, ips: 1.1934 samples/sec | ETA 09:03:07 2021-05-06 11:41:11 [INFO] [TRAIN] epoch: 4, iter: 1120/40000, loss: 1.6751, lr: 0.009750, batch_cost: 1.1697, reader_cost: 0.17836, ips: 0.8549 samples/sec | ETA 12:37:58 2021-05-06 11:41:20 [INFO] [TRAIN] epoch: 4, iter: 1130/40000, loss: 1.3532, lr: 0.009748, batch_cost: 0.8448, reader_cost: 0.00031, ips: 1.1838 samples/sec | ETA 09:07:15 2021-05-06 11:41:28 [INFO] [TRAIN] epoch: 4, iter: 1140/40000, loss: 1.3481, lr: 0.009746, batch_cost: 0.8396, reader_cost: 0.00014, ips: 1.1911 samples/sec | ETA 09:03:45 2021-05-06 11:41:37 [INFO] [TRAIN] epoch: 4, iter: 1150/40000, loss: 1.0911, lr: 0.009744, batch_cost: 0.8417, reader_cost: 0.00014, ips: 1.1880 samples/sec | ETA 09:05:00 2021-05-06 11:41:45 [INFO] [TRAIN] epoch: 4, iter: 1160/40000, loss: 1.0456, lr: 0.009741, batch_cost: 0.8411, reader_cost: 0.00016, ips: 1.1889 samples/sec | ETA 09:04:30 2021-05-06 11:41:53 [INFO] [TRAIN] epoch: 4, iter: 1170/40000, loss: 1.0214, lr: 0.009739, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1885 samples/sec | ETA 09:04:30 2021-05-06 11:42:02 [INFO] [TRAIN] epoch: 4, iter: 1180/40000, loss: 1.7959, lr: 0.009737, batch_cost: 0.8385, reader_cost: 0.00014, ips: 1.1927 samples/sec | ETA 09:02:29 2021-05-06 11:42:10 [INFO] [TRAIN] epoch: 4, iter: 1190/40000, loss: 1.8642, lr: 0.009735, batch_cost: 0.8396, reader_cost: 0.00014, ips: 1.1910 samples/sec | ETA 09:03:04 2021-05-06 11:42:19 [INFO] [TRAIN] epoch: 4, iter: 1200/40000, loss: 1.0954, lr: 0.009733, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 09:03:31 2021-05-06 11:42:27 [INFO] [TRAIN] epoch: 4, iter: 1210/40000, loss: 0.7696, lr: 0.009730, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 09:03:08 2021-05-06 11:42:35 [INFO] [TRAIN] epoch: 4, iter: 1220/40000, loss: 1.0573, lr: 0.009728, batch_cost: 0.8413, reader_cost: 0.00016, ips: 1.1887 samples/sec | ETA 09:03:44 2021-05-06 11:42:44 [INFO] [TRAIN] epoch: 4, iter: 1230/40000, loss: 0.8422, lr: 0.009726, batch_cost: 0.8404, reader_cost: 0.00016, ips: 1.1899 samples/sec | ETA 09:03:03 2021-05-06 11:42:52 [INFO] [TRAIN] epoch: 4, iter: 1240/40000, loss: 1.0083, lr: 0.009724, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1913 samples/sec | ETA 09:02:16 2021-05-06 11:43:01 [INFO] [TRAIN] epoch: 4, iter: 1250/40000, loss: 0.9766, lr: 0.009721, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 09:02:16 2021-05-06 11:43:09 [INFO] [TRAIN] epoch: 4, iter: 1260/40000, loss: 1.2203, lr: 0.009719, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1917 samples/sec | ETA 09:01:48 2021-05-06 11:43:17 [INFO] [TRAIN] epoch: 4, iter: 1270/40000, loss: 1.9600, lr: 0.009717, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 09:02:46 2021-05-06 11:43:26 [INFO] [TRAIN] epoch: 4, iter: 1280/40000, loss: 0.9619, lr: 0.009715, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1887 samples/sec | ETA 09:02:52 2021-05-06 11:43:34 [INFO] [TRAIN] epoch: 4, iter: 1290/40000, loss: 1.0784, lr: 0.009712, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 09:02:27 2021-05-06 11:43:43 [INFO] [TRAIN] epoch: 4, iter: 1300/40000, loss: 1.2463, lr: 0.009710, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 09:02:27 2021-05-06 11:43:51 [INFO] [TRAIN] epoch: 4, iter: 1310/40000, loss: 1.1394, lr: 0.009708, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 09:01:46 2021-05-06 11:43:59 [INFO] [TRAIN] epoch: 4, iter: 1320/40000, loss: 1.2463, lr: 0.009706, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 09:01:57 2021-05-06 11:44:08 [INFO] [TRAIN] epoch: 4, iter: 1330/40000, loss: 1.5807, lr: 0.009703, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 09:01:07 2021-05-06 11:44:16 [INFO] [TRAIN] epoch: 4, iter: 1340/40000, loss: 0.5825, lr: 0.009701, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 09:01:10 2021-05-06 11:44:25 [INFO] [TRAIN] epoch: 4, iter: 1350/40000, loss: 1.0366, lr: 0.009699, batch_cost: 0.8406, reader_cost: 0.00014, ips: 1.1896 samples/sec | ETA 09:01:30 2021-05-06 11:44:33 [INFO] [TRAIN] epoch: 4, iter: 1360/40000, loss: 1.2362, lr: 0.009697, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 09:01:01 2021-05-06 11:44:41 [INFO] [TRAIN] epoch: 4, iter: 1370/40000, loss: 1.1310, lr: 0.009695, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 09:01:13 2021-05-06 11:44:50 [INFO] [TRAIN] epoch: 4, iter: 1380/40000, loss: 1.1113, lr: 0.009692, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 09:00:40 2021-05-06 11:44:58 [INFO] [TRAIN] epoch: 4, iter: 1390/40000, loss: 0.8779, lr: 0.009690, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 09:00:21 2021-05-06 11:45:07 [INFO] [TRAIN] epoch: 4, iter: 1400/40000, loss: 1.0368, lr: 0.009688, batch_cost: 0.8402, reader_cost: 0.00016, ips: 1.1903 samples/sec | ETA 09:00:29 2021-05-06 11:45:15 [INFO] [TRAIN] epoch: 4, iter: 1410/40000, loss: 1.0248, lr: 0.009686, batch_cost: 0.8431, reader_cost: 0.00016, ips: 1.1861 samples/sec | ETA 09:02:14 2021-05-06 11:45:24 [INFO] [TRAIN] epoch: 4, iter: 1420/40000, loss: 0.5951, lr: 0.009683, batch_cost: 0.8395, reader_cost: 0.00016, ips: 1.1912 samples/sec | ETA 08:59:48 2021-05-06 11:45:32 [INFO] [TRAIN] epoch: 4, iter: 1430/40000, loss: 0.8589, lr: 0.009681, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1888 samples/sec | ETA 09:00:45 2021-05-06 11:45:40 [INFO] [TRAIN] epoch: 4, iter: 1440/40000, loss: 1.2752, lr: 0.009679, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 08:59:58 2021-05-06 11:45:49 [INFO] [TRAIN] epoch: 4, iter: 1450/40000, loss: 0.7455, lr: 0.009677, batch_cost: 0.8408, reader_cost: 0.00014, ips: 1.1894 samples/sec | ETA 09:00:11 2021-05-06 11:45:57 [INFO] [TRAIN] epoch: 4, iter: 1460/40000, loss: 1.0169, lr: 0.009674, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 08:59:29 2021-05-06 11:46:06 [INFO] [TRAIN] epoch: 4, iter: 1470/40000, loss: 0.9206, lr: 0.009672, batch_cost: 0.8404, reader_cost: 0.00014, ips: 1.1900 samples/sec | ETA 08:59:38 2021-05-06 11:46:14 [INFO] [TRAIN] epoch: 4, iter: 1480/40000, loss: 0.7028, lr: 0.009670, batch_cost: 0.8392, reader_cost: 0.00013, ips: 1.1916 samples/sec | ETA 08:58:47 2021-05-06 11:46:25 [INFO] [TRAIN] epoch: 5, iter: 1490/40000, loss: 0.9866, lr: 0.009668, batch_cost: 1.1448, reader_cost: 0.20859, ips: 0.8735 samples/sec | ETA 12:14:44 2021-05-06 11:46:34 [INFO] [TRAIN] epoch: 5, iter: 1500/40000, loss: 0.7746, lr: 0.009665, batch_cost: 0.8653, reader_cost: 0.00031, ips: 1.1557 samples/sec | ETA 09:15:13 2021-05-06 11:46:42 [INFO] [TRAIN] epoch: 5, iter: 1510/40000, loss: 0.8728, lr: 0.009663, batch_cost: 0.8413, reader_cost: 0.00014, ips: 1.1886 samples/sec | ETA 08:59:42 2021-05-06 11:46:51 [INFO] [TRAIN] epoch: 5, iter: 1520/40000, loss: 0.8364, lr: 0.009661, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 08:59:01 2021-05-06 11:46:59 [INFO] [TRAIN] epoch: 5, iter: 1530/40000, loss: 1.1632, lr: 0.009659, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 08:58:47 2021-05-06 11:47:08 [INFO] [TRAIN] epoch: 5, iter: 1540/40000, loss: 1.2748, lr: 0.009657, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 08:58:36 2021-05-06 11:47:16 [INFO] [TRAIN] epoch: 5, iter: 1550/40000, loss: 1.2239, lr: 0.009654, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 08:58:26 2021-05-06 11:47:24 [INFO] [TRAIN] epoch: 5, iter: 1560/40000, loss: 1.2652, lr: 0.009652, batch_cost: 0.8396, reader_cost: 0.00014, ips: 1.1910 samples/sec | ETA 08:57:54 2021-05-06 11:47:33 [INFO] [TRAIN] epoch: 5, iter: 1570/40000, loss: 1.0156, lr: 0.009650, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1913 samples/sec | ETA 08:57:39 2021-05-06 11:47:41 [INFO] [TRAIN] epoch: 5, iter: 1580/40000, loss: 0.8743, lr: 0.009648, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 08:58:20 2021-05-06 11:47:50 [INFO] [TRAIN] epoch: 5, iter: 1590/40000, loss: 0.5582, lr: 0.009645, batch_cost: 0.8407, reader_cost: 0.00014, ips: 1.1895 samples/sec | ETA 08:58:11 2021-05-06 11:47:58 [INFO] [TRAIN] epoch: 5, iter: 1600/40000, loss: 0.7296, lr: 0.009643, batch_cost: 0.8411, reader_cost: 0.00014, ips: 1.1889 samples/sec | ETA 08:58:17 2021-05-06 11:48:06 [INFO] [TRAIN] epoch: 5, iter: 1610/40000, loss: 0.8392, lr: 0.009641, batch_cost: 0.8399, reader_cost: 0.00014, ips: 1.1906 samples/sec | ETA 08:57:23 2021-05-06 11:48:15 [INFO] [TRAIN] epoch: 5, iter: 1620/40000, loss: 1.0792, lr: 0.009639, batch_cost: 0.8396, reader_cost: 0.00014, ips: 1.1911 samples/sec | ETA 08:57:03 2021-05-06 11:48:23 [INFO] [TRAIN] epoch: 5, iter: 1630/40000, loss: 0.9713, lr: 0.009636, batch_cost: 0.8379, reader_cost: 0.00015, ips: 1.1934 samples/sec | ETA 08:55:51 2021-05-06 11:48:32 [INFO] [TRAIN] epoch: 5, iter: 1640/40000, loss: 1.1437, lr: 0.009634, batch_cost: 0.8395, reader_cost: 0.00016, ips: 1.1912 samples/sec | ETA 08:56:42 2021-05-06 11:48:40 [INFO] [TRAIN] epoch: 5, iter: 1650/40000, loss: 0.9916, lr: 0.009632, batch_cost: 0.8397, reader_cost: 0.00016, ips: 1.1909 samples/sec | ETA 08:56:43 2021-05-06 11:48:48 [INFO] [TRAIN] epoch: 5, iter: 1660/40000, loss: 0.7305, lr: 0.009630, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 08:56:57 2021-05-06 11:48:57 [INFO] [TRAIN] epoch: 5, iter: 1670/40000, loss: 0.8623, lr: 0.009627, batch_cost: 0.8402, reader_cost: 0.00014, ips: 1.1902 samples/sec | ETA 08:56:43 2021-05-06 11:49:05 [INFO] [TRAIN] epoch: 5, iter: 1680/40000, loss: 1.1408, lr: 0.009625, batch_cost: 0.8388, reader_cost: 0.00014, ips: 1.1921 samples/sec | ETA 08:55:44 2021-05-06 11:49:14 [INFO] [TRAIN] epoch: 5, iter: 1690/40000, loss: 0.6840, lr: 0.009623, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 08:56:20 2021-05-06 11:49:22 [INFO] [TRAIN] epoch: 5, iter: 1700/40000, loss: 1.0572, lr: 0.009621, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1880 samples/sec | ETA 08:57:19 2021-05-06 11:49:30 [INFO] [TRAIN] epoch: 5, iter: 1710/40000, loss: 0.8797, lr: 0.009618, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1918 samples/sec | ETA 08:55:29 2021-05-06 11:49:39 [INFO] [TRAIN] epoch: 5, iter: 1720/40000, loss: 0.9606, lr: 0.009616, batch_cost: 0.8403, reader_cost: 0.00014, ips: 1.1900 samples/sec | ETA 08:56:07 2021-05-06 11:49:47 [INFO] [TRAIN] epoch: 5, iter: 1730/40000, loss: 0.6480, lr: 0.009614, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1904 samples/sec | ETA 08:55:49 2021-05-06 11:49:56 [INFO] [TRAIN] epoch: 5, iter: 1740/40000, loss: 1.0408, lr: 0.009612, batch_cost: 0.8392, reader_cost: 0.00014, ips: 1.1916 samples/sec | ETA 08:55:07 2021-05-06 11:50:04 [INFO] [TRAIN] epoch: 5, iter: 1750/40000, loss: 0.8351, lr: 0.009610, batch_cost: 0.8417, reader_cost: 0.00017, ips: 1.1881 samples/sec | ETA 08:56:33 2021-05-06 11:50:12 [INFO] [TRAIN] epoch: 5, iter: 1760/40000, loss: 0.8098, lr: 0.009607, batch_cost: 0.8391, reader_cost: 0.00014, ips: 1.1918 samples/sec | ETA 08:54:47 2021-05-06 11:50:21 [INFO] [TRAIN] epoch: 5, iter: 1770/40000, loss: 1.2897, lr: 0.009605, batch_cost: 0.8405, reader_cost: 0.00014, ips: 1.1898 samples/sec | ETA 08:55:30 2021-05-06 11:50:29 [INFO] [TRAIN] epoch: 5, iter: 1780/40000, loss: 1.0153, lr: 0.009603, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1887 samples/sec | ETA 08:55:53 2021-05-06 11:50:38 [INFO] [TRAIN] epoch: 5, iter: 1790/40000, loss: 0.8964, lr: 0.009601, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 08:55:38 2021-05-06 11:50:46 [INFO] [TRAIN] epoch: 5, iter: 1800/40000, loss: 0.5368, lr: 0.009598, batch_cost: 0.8403, reader_cost: 0.00016, ips: 1.1900 samples/sec | ETA 08:55:01 2021-05-06 11:50:55 [INFO] [TRAIN] epoch: 5, iter: 1810/40000, loss: 0.8049, lr: 0.009596, batch_cost: 0.8412, reader_cost: 0.00014, ips: 1.1888 samples/sec | ETA 08:55:25 2021-05-06 11:51:03 [INFO] [TRAIN] epoch: 5, iter: 1820/40000, loss: 0.8241, lr: 0.009594, batch_cost: 0.8422, reader_cost: 0.00015, ips: 1.1873 samples/sec | ETA 08:55:56 2021-05-06 11:51:11 [INFO] [TRAIN] epoch: 5, iter: 1830/40000, loss: 0.7716, lr: 0.009592, batch_cost: 0.8423, reader_cost: 0.00016, ips: 1.1872 samples/sec | ETA 08:55:50 2021-05-06 11:51:20 [INFO] [TRAIN] epoch: 5, iter: 1840/40000, loss: 0.6935, lr: 0.009589, batch_cost: 0.8428, reader_cost: 0.00016, ips: 1.1865 samples/sec | ETA 08:56:02 2021-05-06 11:51:28 [INFO] [TRAIN] epoch: 5, iter: 1850/40000, loss: 1.0280, lr: 0.009587, batch_cost: 0.8392, reader_cost: 0.00015, ips: 1.1917 samples/sec | ETA 08:53:34 2021-05-06 11:51:37 [INFO] [TRAIN] epoch: 5, iter: 1860/40000, loss: 0.8472, lr: 0.009585, batch_cost: 0.8392, reader_cost: 0.00010, ips: 1.1916 samples/sec | ETA 08:53:28 2021-05-06 11:51:48 [INFO] [TRAIN] epoch: 6, iter: 1870/40000, loss: 1.0463, lr: 0.009583, batch_cost: 1.1502, reader_cost: 0.20407, ips: 0.8694 samples/sec | ETA 12:10:58 2021-05-06 11:51:57 [INFO] [TRAIN] epoch: 6, iter: 1880/40000, loss: 0.7151, lr: 0.009580, batch_cost: 0.8426, reader_cost: 0.00030, ips: 1.1868 samples/sec | ETA 08:55:19 2021-05-06 11:52:05 [INFO] [TRAIN] epoch: 6, iter: 1890/40000, loss: 0.7567, lr: 0.009578, batch_cost: 0.8399, reader_cost: 0.00014, ips: 1.1906 samples/sec | ETA 08:53:29 2021-05-06 11:52:13 [INFO] [TRAIN] epoch: 6, iter: 1900/40000, loss: 0.5990, lr: 0.009576, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1882 samples/sec | ETA 08:54:25 2021-05-06 11:52:22 [INFO] [TRAIN] epoch: 6, iter: 1910/40000, loss: 1.3917, lr: 0.009574, batch_cost: 0.8419, reader_cost: 0.00015, ips: 1.1877 samples/sec | ETA 08:54:29 2021-05-06 11:52:30 [INFO] [TRAIN] epoch: 6, iter: 1920/40000, loss: 1.2663, lr: 0.009571, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1885 samples/sec | ETA 08:53:59 2021-05-06 11:52:39 [INFO] [TRAIN] epoch: 6, iter: 1930/40000, loss: 0.9902, lr: 0.009569, batch_cost: 0.8402, reader_cost: 0.00014, ips: 1.1902 samples/sec | ETA 08:53:06 2021-05-06 11:52:47 [INFO] [TRAIN] epoch: 6, iter: 1940/40000, loss: 0.9924, lr: 0.009567, batch_cost: 0.8390, reader_cost: 0.00014, ips: 1.1919 samples/sec | ETA 08:52:12 2021-05-06 11:52:55 [INFO] [TRAIN] epoch: 6, iter: 1950/40000, loss: 0.6114, lr: 0.009565, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 08:53:13 2021-05-06 11:53:04 [INFO] [TRAIN] epoch: 6, iter: 1960/40000, loss: 0.2804, lr: 0.009563, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1884 samples/sec | ETA 08:53:28 2021-05-06 11:53:12 [INFO] [TRAIN] epoch: 6, iter: 1970/40000, loss: 0.7181, lr: 0.009560, batch_cost: 0.8404, reader_cost: 0.00014, ips: 1.1899 samples/sec | ETA 08:52:39 2021-05-06 11:53:21 [INFO] [TRAIN] epoch: 6, iter: 1980/40000, loss: 0.7046, lr: 0.009558, batch_cost: 0.8405, reader_cost: 0.00014, ips: 1.1898 samples/sec | ETA 08:52:34 2021-05-06 11:53:29 [INFO] [TRAIN] epoch: 6, iter: 1990/40000, loss: 0.8213, lr: 0.009556, batch_cost: 0.8405, reader_cost: 0.00014, ips: 1.1897 samples/sec | ETA 08:52:29 2021-05-06 11:53:37 [INFO] [TRAIN] epoch: 6, iter: 2000/40000, loss: 0.9138, lr: 0.009554, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 08:51:58 2021-05-06 11:53:37 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 11:57:15 [INFO] [EVAL] #Images: 500 mIoU: 0.4926 Acc: 0.9205 Kappa: 0.8968 2021-05-06 11:57:15 [INFO] [EVAL] Class IoU: [0.9635 0.724 0.8753 0.4586 0.375 0.3209 0.0079 0.4836 0.8789 0.4647 0.9025 0.6533 0.1641 0.8719 0.3322 0. 0.1378 0.1797 0.5651] 2021-05-06 11:57:15 [INFO] [EVAL] Class Acc: [0.9798 0.8443 0.9396 0.7721 0.4858 0.6916 0.6441 0.6686 0.9318 0.5581 0.9275 0.767 0.6187 0.9093 0.5667 0. 0.1423 0.5515 0.6219] 2021-05-06 11:58:05 [INFO] [EVAL] The model with the best validation mIoU (0.4926) was saved at iter 2000. 2021-05-06 11:58:13 [INFO] [TRAIN] epoch: 6, iter: 2010/40000, loss: 0.6930, lr: 0.009551, batch_cost: 0.8367, reader_cost: 0.00021, ips: 1.1952 samples/sec | ETA 08:49:44 2021-05-06 11:58:22 [INFO] [TRAIN] epoch: 6, iter: 2020/40000, loss: 0.5732, lr: 0.009549, batch_cost: 0.8366, reader_cost: 0.00015, ips: 1.1954 samples/sec | ETA 08:49:32 2021-05-06 11:58:30 [INFO] [TRAIN] epoch: 6, iter: 2030/40000, loss: 1.0686, lr: 0.009547, batch_cost: 0.8387, reader_cost: 0.00014, ips: 1.1924 samples/sec | ETA 08:50:44 2021-05-06 11:58:38 [INFO] [TRAIN] epoch: 6, iter: 2040/40000, loss: 0.7534, lr: 0.009545, batch_cost: 0.8398, reader_cost: 0.00014, ips: 1.1908 samples/sec | ETA 08:51:18 2021-05-06 11:58:47 [INFO] [TRAIN] epoch: 6, iter: 2050/40000, loss: 0.9560, lr: 0.009542, batch_cost: 0.8396, reader_cost: 0.00014, ips: 1.1910 samples/sec | ETA 08:51:02 2021-05-06 11:58:55 [INFO] [TRAIN] epoch: 6, iter: 2060/40000, loss: 0.7935, lr: 0.009540, batch_cost: 0.8410, reader_cost: 0.00013, ips: 1.1890 samples/sec | ETA 08:51:49 2021-05-06 11:59:04 [INFO] [TRAIN] epoch: 6, iter: 2070/40000, loss: 0.5369, lr: 0.009538, batch_cost: 0.8409, reader_cost: 0.00014, ips: 1.1892 samples/sec | ETA 08:51:35 2021-05-06 11:59:12 [INFO] [TRAIN] epoch: 6, iter: 2080/40000, loss: 0.8029, lr: 0.009536, batch_cost: 0.8417, reader_cost: 0.00016, ips: 1.1881 samples/sec | ETA 08:51:55 2021-05-06 11:59:20 [INFO] [TRAIN] epoch: 6, iter: 2090/40000, loss: 0.4616, lr: 0.009533, batch_cost: 0.8415, reader_cost: 0.00014, ips: 1.1883 samples/sec | ETA 08:51:42 2021-05-06 11:59:29 [INFO] [TRAIN] epoch: 6, iter: 2100/40000, loss: 0.6750, lr: 0.009531, batch_cost: 0.8418, reader_cost: 0.00014, ips: 1.1880 samples/sec | ETA 08:51:42 2021-05-06 11:59:37 [INFO] [TRAIN] epoch: 6, iter: 2110/40000, loss: 0.4937, lr: 0.009529, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 08:50:44 2021-05-06 11:59:46 [INFO] [TRAIN] epoch: 6, iter: 2120/40000, loss: 0.5994, lr: 0.009527, batch_cost: 0.8424, reader_cost: 0.00015, ips: 1.1870 samples/sec | ETA 08:51:51 2021-05-06 11:59:54 [INFO] [TRAIN] epoch: 6, iter: 2130/40000, loss: 0.4893, lr: 0.009524, batch_cost: 0.8410, reader_cost: 0.00014, ips: 1.1891 samples/sec | ETA 08:50:47 2021-05-06 12:00:03 [INFO] [TRAIN] epoch: 6, iter: 2140/40000, loss: 0.7376, lr: 0.009522, batch_cost: 0.8422, reader_cost: 0.00014, ips: 1.1874 samples/sec | ETA 08:51:24 2021-05-06 12:00:11 [INFO] [TRAIN] epoch: 6, iter: 2150/40000, loss: 0.7215, lr: 0.009520, batch_cost: 0.8431, reader_cost: 0.00015, ips: 1.1861 samples/sec | ETA 08:51:52 2021-05-06 12:00:19 [INFO] [TRAIN] epoch: 6, iter: 2160/40000, loss: 1.2724, lr: 0.009518, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1912 samples/sec | ETA 08:49:26 2021-05-06 12:00:28 [INFO] [TRAIN] epoch: 6, iter: 2170/40000, loss: 0.5777, lr: 0.009516, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 08:49:31 2021-05-06 12:00:36 [INFO] [TRAIN] epoch: 6, iter: 2180/40000, loss: 0.7236, lr: 0.009513, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 08:49:27 2021-05-06 12:00:45 [INFO] [TRAIN] epoch: 6, iter: 2190/40000, loss: 0.7325, lr: 0.009511, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 08:49:40 2021-05-06 12:00:53 [INFO] [TRAIN] epoch: 6, iter: 2200/40000, loss: 0.7639, lr: 0.009509, batch_cost: 0.8412, reader_cost: 0.00014, ips: 1.1888 samples/sec | ETA 08:49:56 2021-05-06 12:01:01 [INFO] [TRAIN] epoch: 6, iter: 2210/40000, loss: 1.5782, lr: 0.009507, batch_cost: 0.8410, reader_cost: 0.00014, ips: 1.1890 samples/sec | ETA 08:49:42 2021-05-06 12:01:10 [INFO] [TRAIN] epoch: 6, iter: 2220/40000, loss: 0.9357, lr: 0.009504, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1904 samples/sec | ETA 08:48:57 2021-05-06 12:01:18 [INFO] [TRAIN] epoch: 6, iter: 2230/40000, loss: 0.9193, lr: 0.009502, batch_cost: 0.8392, reader_cost: 0.00028, ips: 1.1916 samples/sec | ETA 08:48:17 2021-05-06 12:01:30 [INFO] [TRAIN] epoch: 7, iter: 2240/40000, loss: 1.3228, lr: 0.009500, batch_cost: 1.1660, reader_cost: 0.20619, ips: 0.8576 samples/sec | ETA 12:13:47 2021-05-06 12:01:38 [INFO] [TRAIN] epoch: 7, iter: 2250/40000, loss: 0.8448, lr: 0.009498, batch_cost: 0.8413, reader_cost: 0.00016, ips: 1.1887 samples/sec | ETA 08:49:18 2021-05-06 12:01:47 [INFO] [TRAIN] epoch: 7, iter: 2260/40000, loss: 0.6075, lr: 0.009495, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 08:48:52 2021-05-06 12:01:55 [INFO] [TRAIN] epoch: 7, iter: 2270/40000, loss: 0.8451, lr: 0.009493, batch_cost: 0.8404, reader_cost: 0.00014, ips: 1.1899 samples/sec | ETA 08:48:29 2021-05-06 12:02:03 [INFO] [TRAIN] epoch: 7, iter: 2280/40000, loss: 0.7873, lr: 0.009491, batch_cost: 0.8403, reader_cost: 0.00014, ips: 1.1900 samples/sec | ETA 08:48:17 2021-05-06 12:02:12 [INFO] [TRAIN] epoch: 7, iter: 2290/40000, loss: 1.1124, lr: 0.009489, batch_cost: 0.8440, reader_cost: 0.00015, ips: 1.1849 samples/sec | ETA 08:50:25 2021-05-06 12:02:20 [INFO] [TRAIN] epoch: 7, iter: 2300/40000, loss: 1.1010, lr: 0.009486, batch_cost: 0.8422, reader_cost: 0.00015, ips: 1.1874 samples/sec | ETA 08:49:10 2021-05-06 12:02:29 [INFO] [TRAIN] epoch: 7, iter: 2310/40000, loss: 1.2013, lr: 0.009484, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 08:47:49 2021-05-06 12:02:37 [INFO] [TRAIN] epoch: 7, iter: 2320/40000, loss: 0.7287, lr: 0.009482, batch_cost: 0.8408, reader_cost: 0.00016, ips: 1.1893 samples/sec | ETA 08:48:02 2021-05-06 12:02:46 [INFO] [TRAIN] epoch: 7, iter: 2330/40000, loss: 0.2585, lr: 0.009480, batch_cost: 0.8403, reader_cost: 0.00014, ips: 1.1900 samples/sec | ETA 08:47:35 2021-05-06 12:02:54 [INFO] [TRAIN] epoch: 7, iter: 2340/40000, loss: 0.5249, lr: 0.009477, batch_cost: 0.8410, reader_cost: 0.00014, ips: 1.1891 samples/sec | ETA 08:47:52 2021-05-06 12:03:02 [INFO] [TRAIN] epoch: 7, iter: 2350/40000, loss: 0.6325, lr: 0.009475, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1887 samples/sec | ETA 08:47:52 2021-05-06 12:03:11 [INFO] [TRAIN] epoch: 7, iter: 2360/40000, loss: 0.9351, lr: 0.009473, batch_cost: 0.8388, reader_cost: 0.00015, ips: 1.1922 samples/sec | ETA 08:46:13 2021-05-06 12:03:19 [INFO] [TRAIN] epoch: 7, iter: 2370/40000, loss: 1.1411, lr: 0.009471, batch_cost: 0.8392, reader_cost: 0.00014, ips: 1.1916 samples/sec | ETA 08:46:19 2021-05-06 12:03:28 [INFO] [TRAIN] epoch: 7, iter: 2380/40000, loss: 0.7427, lr: 0.009468, batch_cost: 0.8407, reader_cost: 0.00014, ips: 1.1894 samples/sec | ETA 08:47:08 2021-05-06 12:03:36 [INFO] [TRAIN] epoch: 7, iter: 2390/40000, loss: 0.7429, lr: 0.009466, batch_cost: 0.8404, reader_cost: 0.00014, ips: 1.1899 samples/sec | ETA 08:46:46 2021-05-06 12:03:44 [INFO] [TRAIN] epoch: 7, iter: 2400/40000, loss: 0.6269, lr: 0.009464, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 08:45:56 2021-05-06 12:03:53 [INFO] [TRAIN] epoch: 7, iter: 2410/40000, loss: 0.9017, lr: 0.009462, batch_cost: 0.8403, reader_cost: 0.00016, ips: 1.1901 samples/sec | ETA 08:46:26 2021-05-06 12:04:01 [INFO] [TRAIN] epoch: 7, iter: 2420/40000, loss: 0.9758, lr: 0.009460, batch_cost: 0.8391, reader_cost: 0.00014, ips: 1.1917 samples/sec | ETA 08:45:34 2021-05-06 12:04:10 [INFO] [TRAIN] epoch: 7, iter: 2430/40000, loss: 2.1740, lr: 0.009457, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1917 samples/sec | ETA 08:45:26 2021-05-06 12:04:18 [INFO] [TRAIN] epoch: 7, iter: 2440/40000, loss: 0.7697, lr: 0.009455, batch_cost: 0.8397, reader_cost: 0.00014, ips: 1.1909 samples/sec | ETA 08:45:39 2021-05-06 12:04:26 [INFO] [TRAIN] epoch: 7, iter: 2450/40000, loss: 0.9248, lr: 0.009453, batch_cost: 0.8402, reader_cost: 0.00014, ips: 1.1902 samples/sec | ETA 08:45:49 2021-05-06 12:04:35 [INFO] [TRAIN] epoch: 7, iter: 2460/40000, loss: 0.4856, lr: 0.009451, batch_cost: 0.8402, reader_cost: 0.00014, ips: 1.1901 samples/sec | ETA 08:45:42 2021-05-06 12:04:43 [INFO] [TRAIN] epoch: 7, iter: 2470/40000, loss: 0.6414, lr: 0.009448, batch_cost: 0.8407, reader_cost: 0.00014, ips: 1.1895 samples/sec | ETA 08:45:50 2021-05-06 12:04:52 [INFO] [TRAIN] epoch: 7, iter: 2480/40000, loss: 0.4089, lr: 0.009446, batch_cost: 0.8406, reader_cost: 0.00014, ips: 1.1897 samples/sec | ETA 08:45:38 2021-05-06 12:05:00 [INFO] [TRAIN] epoch: 7, iter: 2490/40000, loss: 0.6936, lr: 0.009444, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 08:45:31 2021-05-06 12:05:08 [INFO] [TRAIN] epoch: 7, iter: 2500/40000, loss: 0.5991, lr: 0.009442, batch_cost: 0.8390, reader_cost: 0.00015, ips: 1.1919 samples/sec | ETA 08:44:22 2021-05-06 12:05:17 [INFO] [TRAIN] epoch: 7, iter: 2510/40000, loss: 0.5967, lr: 0.009439, batch_cost: 0.8393, reader_cost: 0.00014, ips: 1.1915 samples/sec | ETA 08:44:25 2021-05-06 12:05:25 [INFO] [TRAIN] epoch: 7, iter: 2520/40000, loss: 0.6639, lr: 0.009437, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 08:44:37 2021-05-06 12:05:34 [INFO] [TRAIN] epoch: 7, iter: 2530/40000, loss: 0.6142, lr: 0.009435, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 08:45:06 2021-05-06 12:05:42 [INFO] [TRAIN] epoch: 7, iter: 2540/40000, loss: 0.5851, lr: 0.009433, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 08:44:31 2021-05-06 12:05:50 [INFO] [TRAIN] epoch: 7, iter: 2550/40000, loss: 0.5889, lr: 0.009430, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1882 samples/sec | ETA 08:45:18 2021-05-06 12:05:59 [INFO] [TRAIN] epoch: 7, iter: 2560/40000, loss: 0.9029, lr: 0.009428, batch_cost: 0.8407, reader_cost: 0.00017, ips: 1.1895 samples/sec | ETA 08:44:35 2021-05-06 12:06:07 [INFO] [TRAIN] epoch: 7, iter: 2570/40000, loss: 0.5971, lr: 0.009426, batch_cost: 0.8417, reader_cost: 0.00017, ips: 1.1881 samples/sec | ETA 08:45:03 2021-05-06 12:06:16 [INFO] [TRAIN] epoch: 7, iter: 2580/40000, loss: 0.7703, lr: 0.009424, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 08:44:04 2021-05-06 12:06:24 [INFO] [TRAIN] epoch: 7, iter: 2590/40000, loss: 0.7745, lr: 0.009421, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 08:43:34 2021-05-06 12:06:32 [INFO] [TRAIN] epoch: 7, iter: 2600/40000, loss: 0.8963, lr: 0.009419, batch_cost: 0.8397, reader_cost: 0.00011, ips: 1.1909 samples/sec | ETA 08:43:23 2021-05-06 12:06:44 [INFO] [TRAIN] epoch: 8, iter: 2610/40000, loss: 1.1321, lr: 0.009417, batch_cost: 1.1498, reader_cost: 0.20422, ips: 0.8697 samples/sec | ETA 11:56:31 2021-05-06 12:06:52 [INFO] [TRAIN] epoch: 8, iter: 2620/40000, loss: 0.7332, lr: 0.009415, batch_cost: 0.8436, reader_cost: 0.00031, ips: 1.1854 samples/sec | ETA 08:45:32 2021-05-06 12:07:01 [INFO] [TRAIN] epoch: 8, iter: 2630/40000, loss: 0.7121, lr: 0.009412, batch_cost: 0.8417, reader_cost: 0.00014, ips: 1.1881 samples/sec | ETA 08:44:13 2021-05-06 12:07:09 [INFO] [TRAIN] epoch: 8, iter: 2640/40000, loss: 0.5656, lr: 0.009410, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 08:43:02 2021-05-06 12:07:18 [INFO] [TRAIN] epoch: 8, iter: 2650/40000, loss: 1.1385, lr: 0.009408, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 08:43:27 2021-05-06 12:07:26 [INFO] [TRAIN] epoch: 8, iter: 2660/40000, loss: 0.9980, lr: 0.009406, batch_cost: 0.8408, reader_cost: 0.00016, ips: 1.1893 samples/sec | ETA 08:43:15 2021-05-06 12:07:34 [INFO] [TRAIN] epoch: 8, iter: 2670/40000, loss: 0.7411, lr: 0.009403, batch_cost: 0.8405, reader_cost: 0.00014, ips: 1.1898 samples/sec | ETA 08:42:55 2021-05-06 12:07:43 [INFO] [TRAIN] epoch: 8, iter: 2680/40000, loss: 0.8535, lr: 0.009401, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1904 samples/sec | ETA 08:42:31 2021-05-06 12:07:51 [INFO] [TRAIN] epoch: 8, iter: 2690/40000, loss: 0.6267, lr: 0.009399, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 08:41:54 2021-05-06 12:08:00 [INFO] [TRAIN] epoch: 8, iter: 2700/40000, loss: 0.3088, lr: 0.009397, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 08:42:23 2021-05-06 12:08:08 [INFO] [TRAIN] epoch: 8, iter: 2710/40000, loss: 0.6330, lr: 0.009394, batch_cost: 0.8417, reader_cost: 0.00014, ips: 1.1881 samples/sec | ETA 08:43:05 2021-05-06 12:08:16 [INFO] [TRAIN] epoch: 8, iter: 2720/40000, loss: 0.7525, lr: 0.009392, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1885 samples/sec | ETA 08:42:47 2021-05-06 12:08:25 [INFO] [TRAIN] epoch: 8, iter: 2730/40000, loss: 0.9984, lr: 0.009390, batch_cost: 0.8411, reader_cost: 0.00016, ips: 1.1890 samples/sec | ETA 08:42:26 2021-05-06 12:08:33 [INFO] [TRAIN] epoch: 8, iter: 2740/40000, loss: 1.0599, lr: 0.009388, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 08:41:39 2021-05-06 12:08:42 [INFO] [TRAIN] epoch: 8, iter: 2750/40000, loss: 0.9776, lr: 0.009386, batch_cost: 0.8405, reader_cost: 0.00014, ips: 1.1897 samples/sec | ETA 08:41:49 2021-05-06 12:08:50 [INFO] [TRAIN] epoch: 8, iter: 2760/40000, loss: 0.7000, lr: 0.009383, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 08:41:08 2021-05-06 12:08:58 [INFO] [TRAIN] epoch: 8, iter: 2770/40000, loss: 0.7149, lr: 0.009381, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 08:41:29 2021-05-06 12:09:07 [INFO] [TRAIN] epoch: 8, iter: 2780/40000, loss: 0.9845, lr: 0.009379, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1879 samples/sec | ETA 08:42:13 2021-05-06 12:09:15 [INFO] [TRAIN] epoch: 8, iter: 2790/40000, loss: 1.7578, lr: 0.009377, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 08:41:14 2021-05-06 12:09:24 [INFO] [TRAIN] epoch: 8, iter: 2800/40000, loss: 1.1652, lr: 0.009374, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 08:40:53 2021-05-06 12:09:32 [INFO] [TRAIN] epoch: 8, iter: 2810/40000, loss: 0.8483, lr: 0.009372, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 08:41:07 2021-05-06 12:09:40 [INFO] [TRAIN] epoch: 8, iter: 2820/40000, loss: 1.0458, lr: 0.009370, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 08:40:31 2021-05-06 12:09:49 [INFO] [TRAIN] epoch: 8, iter: 2830/40000, loss: 0.6448, lr: 0.009368, batch_cost: 0.8392, reader_cost: 0.00014, ips: 1.1916 samples/sec | ETA 08:39:53 2021-05-06 12:09:57 [INFO] [TRAIN] epoch: 8, iter: 2840/40000, loss: 0.7064, lr: 0.009365, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1904 samples/sec | ETA 08:40:16 2021-05-06 12:10:06 [INFO] [TRAIN] epoch: 8, iter: 2850/40000, loss: 0.4911, lr: 0.009363, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 08:40:26 2021-05-06 12:10:14 [INFO] [TRAIN] epoch: 8, iter: 2860/40000, loss: 0.5677, lr: 0.009361, batch_cost: 0.8417, reader_cost: 0.00014, ips: 1.1881 samples/sec | ETA 08:41:00 2021-05-06 12:10:22 [INFO] [TRAIN] epoch: 8, iter: 2870/40000, loss: 0.6093, lr: 0.009359, batch_cost: 0.8414, reader_cost: 0.00014, ips: 1.1885 samples/sec | ETA 08:40:41 2021-05-06 12:10:31 [INFO] [TRAIN] epoch: 8, iter: 2880/40000, loss: 0.5455, lr: 0.009356, batch_cost: 0.8427, reader_cost: 0.00014, ips: 1.1866 samples/sec | ETA 08:41:21 2021-05-06 12:10:39 [INFO] [TRAIN] epoch: 8, iter: 2890/40000, loss: 0.4797, lr: 0.009354, batch_cost: 0.8397, reader_cost: 0.00017, ips: 1.1908 samples/sec | ETA 08:39:22 2021-05-06 12:10:48 [INFO] [TRAIN] epoch: 8, iter: 2900/40000, loss: 1.0153, lr: 0.009352, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 08:39:45 2021-05-06 12:10:56 [INFO] [TRAIN] epoch: 8, iter: 2910/40000, loss: 0.5763, lr: 0.009350, batch_cost: 0.8400, reader_cost: 0.00016, ips: 1.1905 samples/sec | ETA 08:39:16 2021-05-06 12:11:05 [INFO] [TRAIN] epoch: 8, iter: 2920/40000, loss: 0.6527, lr: 0.009347, batch_cost: 0.8405, reader_cost: 0.00014, ips: 1.1898 samples/sec | ETA 08:39:24 2021-05-06 12:11:13 [INFO] [TRAIN] epoch: 8, iter: 2930/40000, loss: 1.1591, lr: 0.009345, batch_cost: 0.8419, reader_cost: 0.00015, ips: 1.1878 samples/sec | ETA 08:40:08 2021-05-06 12:11:21 [INFO] [TRAIN] epoch: 8, iter: 2940/40000, loss: 0.5515, lr: 0.009343, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 08:39:03 2021-05-06 12:11:30 [INFO] [TRAIN] epoch: 8, iter: 2950/40000, loss: 0.9053, lr: 0.009341, batch_cost: 0.8399, reader_cost: 0.00016, ips: 1.1907 samples/sec | ETA 08:38:36 2021-05-06 12:11:38 [INFO] [TRAIN] epoch: 8, iter: 2960/40000, loss: 1.0224, lr: 0.009338, batch_cost: 0.8394, reader_cost: 0.00016, ips: 1.1913 samples/sec | ETA 08:38:11 2021-05-06 12:11:47 [INFO] [TRAIN] epoch: 8, iter: 2970/40000, loss: 0.6749, lr: 0.009336, batch_cost: 0.8391, reader_cost: 0.00012, ips: 1.1918 samples/sec | ETA 08:37:50 2021-05-06 12:11:58 [INFO] [TRAIN] epoch: 9, iter: 2980/40000, loss: 0.8124, lr: 0.009334, batch_cost: 1.1520, reader_cost: 0.20026, ips: 0.8680 samples/sec | ETA 11:50:47 2021-05-06 12:12:07 [INFO] [TRAIN] epoch: 9, iter: 2990/40000, loss: 0.5968, lr: 0.009332, batch_cost: 0.8514, reader_cost: 0.00032, ips: 1.1745 samples/sec | ETA 08:45:11 2021-05-06 12:12:15 [INFO] [TRAIN] epoch: 9, iter: 3000/40000, loss: 0.6934, lr: 0.009329, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 08:38:32 2021-05-06 12:12:15 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 12:15:52 [INFO] [EVAL] #Images: 500 mIoU: 0.5707 Acc: 0.9296 Kappa: 0.9088 2021-05-06 12:15:52 [INFO] [EVAL] Class IoU: [0.9615 0.7393 0.8863 0.4871 0.4525 0.3466 0.3514 0.5315 0.8863 0.5308 0.9212 0.6754 0.3188 0.8948 0.4726 0.0468 0.3647 0.3489 0.6265] 2021-05-06 12:15:52 [INFO] [EVAL] Class Acc: [0.9933 0.7924 0.9348 0.7468 0.6009 0.688 0.7976 0.9108 0.9174 0.7786 0.9647 0.7693 0.6741 0.9227 0.5449 0.9151 0.4039 0.7619 0.7244] 2021-05-06 12:16:40 [INFO] [EVAL] The model with the best validation mIoU (0.5707) was saved at iter 3000. 2021-05-06 12:16:48 [INFO] [TRAIN] epoch: 9, iter: 3010/40000, loss: 0.5446, lr: 0.009327, batch_cost: 0.8348, reader_cost: 0.00023, ips: 1.1979 samples/sec | ETA 08:34:40 2021-05-06 12:16:57 [INFO] [TRAIN] epoch: 9, iter: 3020/40000, loss: 0.5652, lr: 0.009325, batch_cost: 0.8361, reader_cost: 0.00014, ips: 1.1960 samples/sec | ETA 08:35:20 2021-05-06 12:17:05 [INFO] [TRAIN] epoch: 9, iter: 3030/40000, loss: 0.7425, lr: 0.009323, batch_cost: 0.8413, reader_cost: 0.00014, ips: 1.1887 samples/sec | ETA 08:38:21 2021-05-06 12:17:13 [INFO] [TRAIN] epoch: 9, iter: 3040/40000, loss: 0.8348, lr: 0.009320, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1912 samples/sec | ETA 08:37:08 2021-05-06 12:17:22 [INFO] [TRAIN] epoch: 9, iter: 3050/40000, loss: 1.0233, lr: 0.009318, batch_cost: 0.8390, reader_cost: 0.00015, ips: 1.1918 samples/sec | ETA 08:36:42 2021-05-06 12:17:30 [INFO] [TRAIN] epoch: 9, iter: 3060/40000, loss: 0.8926, lr: 0.009316, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 08:37:00 2021-05-06 12:17:39 [INFO] [TRAIN] epoch: 9, iter: 3070/40000, loss: 0.6632, lr: 0.009314, batch_cost: 0.8408, reader_cost: 0.00014, ips: 1.1893 samples/sec | ETA 08:37:31 2021-05-06 12:17:47 [INFO] [TRAIN] epoch: 9, iter: 3080/40000, loss: 0.5060, lr: 0.009311, batch_cost: 0.8392, reader_cost: 0.00014, ips: 1.1916 samples/sec | ETA 08:36:23 2021-05-06 12:17:55 [INFO] [TRAIN] epoch: 9, iter: 3090/40000, loss: 0.5966, lr: 0.009309, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1904 samples/sec | ETA 08:36:46 2021-05-06 12:18:04 [INFO] [TRAIN] epoch: 9, iter: 3100/40000, loss: 0.7462, lr: 0.009307, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1884 samples/sec | ETA 08:37:29 2021-05-06 12:18:12 [INFO] [TRAIN] epoch: 9, iter: 3110/40000, loss: 0.7275, lr: 0.009305, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 08:36:35 2021-05-06 12:18:21 [INFO] [TRAIN] epoch: 9, iter: 3120/40000, loss: 0.7697, lr: 0.009302, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 08:36:18 2021-05-06 12:18:29 [INFO] [TRAIN] epoch: 9, iter: 3130/40000, loss: 0.6153, lr: 0.009300, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 08:36:42 2021-05-06 12:18:37 [INFO] [TRAIN] epoch: 9, iter: 3140/40000, loss: 0.8690, lr: 0.009298, batch_cost: 0.8403, reader_cost: 0.00016, ips: 1.1901 samples/sec | ETA 08:36:12 2021-05-06 12:18:46 [INFO] [TRAIN] epoch: 9, iter: 3150/40000, loss: 0.5563, lr: 0.009296, batch_cost: 0.8416, reader_cost: 0.00017, ips: 1.1882 samples/sec | ETA 08:36:52 2021-05-06 12:18:54 [INFO] [TRAIN] epoch: 9, iter: 3160/40000, loss: 0.5888, lr: 0.009293, batch_cost: 0.8397, reader_cost: 0.00016, ips: 1.1909 samples/sec | ETA 08:35:35 2021-05-06 12:19:03 [INFO] [TRAIN] epoch: 9, iter: 3170/40000, loss: 0.8974, lr: 0.009291, batch_cost: 0.8408, reader_cost: 0.00014, ips: 1.1893 samples/sec | ETA 08:36:08 2021-05-06 12:19:11 [INFO] [TRAIN] epoch: 9, iter: 3180/40000, loss: 0.5704, lr: 0.009289, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 08:35:53 2021-05-06 12:19:20 [INFO] [TRAIN] epoch: 9, iter: 3190/40000, loss: 0.7804, lr: 0.009287, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 08:35:29 2021-05-06 12:19:28 [INFO] [TRAIN] epoch: 9, iter: 3200/40000, loss: 0.3259, lr: 0.009284, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1886 samples/sec | ETA 08:36:01 2021-05-06 12:19:36 [INFO] [TRAIN] epoch: 9, iter: 3210/40000, loss: 0.4193, lr: 0.009282, batch_cost: 0.8431, reader_cost: 0.00015, ips: 1.1862 samples/sec | ETA 08:36:56 2021-05-06 12:19:45 [INFO] [TRAIN] epoch: 9, iter: 3220/40000, loss: 0.4172, lr: 0.009280, batch_cost: 0.8420, reader_cost: 0.00015, ips: 1.1876 samples/sec | ETA 08:36:09 2021-05-06 12:19:53 [INFO] [TRAIN] epoch: 9, iter: 3230/40000, loss: 0.4868, lr: 0.009278, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 08:34:54 2021-05-06 12:20:02 [INFO] [TRAIN] epoch: 9, iter: 3240/40000, loss: 0.6803, lr: 0.009276, batch_cost: 0.8415, reader_cost: 0.00016, ips: 1.1883 samples/sec | ETA 08:35:35 2021-05-06 12:20:10 [INFO] [TRAIN] epoch: 9, iter: 3250/40000, loss: 0.5237, lr: 0.009273, batch_cost: 0.8415, reader_cost: 0.00016, ips: 1.1883 samples/sec | ETA 08:35:26 2021-05-06 12:20:18 [INFO] [TRAIN] epoch: 9, iter: 3260/40000, loss: 0.8563, lr: 0.009271, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 08:34:44 2021-05-06 12:20:27 [INFO] [TRAIN] epoch: 9, iter: 3270/40000, loss: 0.9796, lr: 0.009269, batch_cost: 0.8422, reader_cost: 0.00014, ips: 1.1873 samples/sec | ETA 08:35:35 2021-05-06 12:20:35 [INFO] [TRAIN] epoch: 9, iter: 3280/40000, loss: 0.4686, lr: 0.009267, batch_cost: 0.8392, reader_cost: 0.00014, ips: 1.1916 samples/sec | ETA 08:33:35 2021-05-06 12:20:44 [INFO] [TRAIN] epoch: 9, iter: 3290/40000, loss: 0.6415, lr: 0.009264, batch_cost: 0.8392, reader_cost: 0.00015, ips: 1.1916 samples/sec | ETA 08:33:28 2021-05-06 12:20:52 [INFO] [TRAIN] epoch: 9, iter: 3300/40000, loss: 1.0351, lr: 0.009262, batch_cost: 0.8389, reader_cost: 0.00015, ips: 1.1920 samples/sec | ETA 08:33:08 2021-05-06 12:21:00 [INFO] [TRAIN] epoch: 9, iter: 3310/40000, loss: 0.5135, lr: 0.009260, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 08:33:53 2021-05-06 12:21:09 [INFO] [TRAIN] epoch: 9, iter: 3320/40000, loss: 0.6540, lr: 0.009258, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 08:33:21 2021-05-06 12:21:17 [INFO] [TRAIN] epoch: 9, iter: 3330/40000, loss: 0.9298, lr: 0.009255, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 08:33:35 2021-05-06 12:21:26 [INFO] [TRAIN] epoch: 9, iter: 3340/40000, loss: 0.7776, lr: 0.009253, batch_cost: 0.8380, reader_cost: 0.00013, ips: 1.1933 samples/sec | ETA 08:32:02 2021-05-06 12:21:37 [INFO] [TRAIN] epoch: 10, iter: 3350/40000, loss: 0.8014, lr: 0.009251, batch_cost: 1.1470, reader_cost: 0.19756, ips: 0.8719 samples/sec | ETA 11:40:35 2021-05-06 12:21:46 [INFO] [TRAIN] epoch: 10, iter: 3360/40000, loss: 0.7243, lr: 0.009249, batch_cost: 0.8651, reader_cost: 0.00032, ips: 1.1559 samples/sec | ETA 08:48:18 2021-05-06 12:21:54 [INFO] [TRAIN] epoch: 10, iter: 3370/40000, loss: 0.5408, lr: 0.009246, batch_cost: 0.8401, reader_cost: 0.00016, ips: 1.1903 samples/sec | ETA 08:32:52 2021-05-06 12:22:03 [INFO] [TRAIN] epoch: 10, iter: 3380/40000, loss: 0.4688, lr: 0.009244, batch_cost: 0.8385, reader_cost: 0.00015, ips: 1.1926 samples/sec | ETA 08:31:44 2021-05-06 12:22:11 [INFO] [TRAIN] epoch: 10, iter: 3390/40000, loss: 0.4057, lr: 0.009242, batch_cost: 0.8431, reader_cost: 0.00014, ips: 1.1861 samples/sec | ETA 08:34:25 2021-05-06 12:22:19 [INFO] [TRAIN] epoch: 10, iter: 3400/40000, loss: 0.8331, lr: 0.009240, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 08:32:26 2021-05-06 12:22:28 [INFO] [TRAIN] epoch: 10, iter: 3410/40000, loss: 0.8331, lr: 0.009237, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 08:32:35 2021-05-06 12:22:36 [INFO] [TRAIN] epoch: 10, iter: 3420/40000, loss: 0.7445, lr: 0.009235, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1913 samples/sec | ETA 08:31:45 2021-05-06 12:22:45 [INFO] [TRAIN] epoch: 10, iter: 3430/40000, loss: 0.6615, lr: 0.009233, batch_cost: 0.8385, reader_cost: 0.00015, ips: 1.1927 samples/sec | ETA 08:31:02 2021-05-06 12:22:53 [INFO] [TRAIN] epoch: 10, iter: 3440/40000, loss: 0.4748, lr: 0.009231, batch_cost: 0.8420, reader_cost: 0.00015, ips: 1.1877 samples/sec | ETA 08:33:03 2021-05-06 12:23:01 [INFO] [TRAIN] epoch: 10, iter: 3450/40000, loss: 0.2697, lr: 0.009228, batch_cost: 0.8400, reader_cost: 0.00014, ips: 1.1904 samples/sec | ETA 08:31:42 2021-05-06 12:23:10 [INFO] [TRAIN] epoch: 10, iter: 3460/40000, loss: 0.5402, lr: 0.009226, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 08:32:13 2021-05-06 12:23:18 [INFO] [TRAIN] epoch: 10, iter: 3470/40000, loss: 0.5897, lr: 0.009224, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 08:32:02 2021-05-06 12:23:27 [INFO] [TRAIN] epoch: 10, iter: 3480/40000, loss: 0.8796, lr: 0.009222, batch_cost: 0.8393, reader_cost: 0.00014, ips: 1.1915 samples/sec | ETA 08:30:51 2021-05-06 12:23:35 [INFO] [TRAIN] epoch: 10, iter: 3490/40000, loss: 0.7714, lr: 0.009219, batch_cost: 0.8409, reader_cost: 0.00014, ips: 1.1892 samples/sec | ETA 08:31:40 2021-05-06 12:23:43 [INFO] [TRAIN] epoch: 10, iter: 3500/40000, loss: 0.6098, lr: 0.009217, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 08:31:37 2021-05-06 12:23:52 [INFO] [TRAIN] epoch: 10, iter: 3510/40000, loss: 0.3667, lr: 0.009215, batch_cost: 0.8395, reader_cost: 0.00016, ips: 1.1911 samples/sec | ETA 08:30:34 2021-05-06 12:24:00 [INFO] [TRAIN] epoch: 10, iter: 3520/40000, loss: 0.6151, lr: 0.009213, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1903 samples/sec | ETA 08:30:47 2021-05-06 12:24:09 [INFO] [TRAIN] epoch: 10, iter: 3530/40000, loss: 0.7221, lr: 0.009210, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 08:30:33 2021-05-06 12:24:17 [INFO] [TRAIN] epoch: 10, iter: 3540/40000, loss: 0.7500, lr: 0.009208, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 08:30:46 2021-05-06 12:24:25 [INFO] [TRAIN] epoch: 10, iter: 3550/40000, loss: 0.4490, lr: 0.009206, batch_cost: 0.8419, reader_cost: 0.00014, ips: 1.1878 samples/sec | ETA 08:31:27 2021-05-06 12:24:34 [INFO] [TRAIN] epoch: 10, iter: 3560/40000, loss: 0.8353, lr: 0.009204, batch_cost: 0.8427, reader_cost: 0.00015, ips: 1.1866 samples/sec | ETA 08:31:49 2021-05-06 12:24:42 [INFO] [TRAIN] epoch: 10, iter: 3570/40000, loss: 0.8414, lr: 0.009201, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1888 samples/sec | ETA 08:30:44 2021-05-06 12:24:51 [INFO] [TRAIN] epoch: 10, iter: 3580/40000, loss: 0.4220, lr: 0.009199, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1885 samples/sec | ETA 08:30:44 2021-05-06 12:24:59 [INFO] [TRAIN] epoch: 10, iter: 3590/40000, loss: 0.5136, lr: 0.009197, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 08:29:40 2021-05-06 12:25:07 [INFO] [TRAIN] epoch: 10, iter: 3600/40000, loss: 0.6477, lr: 0.009195, batch_cost: 0.8394, reader_cost: 0.00016, ips: 1.1914 samples/sec | ETA 08:29:13 2021-05-06 12:25:16 [INFO] [TRAIN] epoch: 10, iter: 3610/40000, loss: 0.5792, lr: 0.009192, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1911 samples/sec | ETA 08:29:11 2021-05-06 12:25:24 [INFO] [TRAIN] epoch: 10, iter: 3620/40000, loss: 0.5825, lr: 0.009190, batch_cost: 0.8420, reader_cost: 0.00016, ips: 1.1877 samples/sec | ETA 08:30:31 2021-05-06 12:25:33 [INFO] [TRAIN] epoch: 10, iter: 3630/40000, loss: 0.7767, lr: 0.009188, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 08:29:50 2021-05-06 12:25:41 [INFO] [TRAIN] epoch: 10, iter: 3640/40000, loss: 0.6667, lr: 0.009186, batch_cost: 0.8411, reader_cost: 0.00016, ips: 1.1889 samples/sec | ETA 08:29:42 2021-05-06 12:25:50 [INFO] [TRAIN] epoch: 10, iter: 3650/40000, loss: 0.4290, lr: 0.009183, batch_cost: 0.8400, reader_cost: 0.00014, ips: 1.1905 samples/sec | ETA 08:28:53 2021-05-06 12:25:58 [INFO] [TRAIN] epoch: 10, iter: 3660/40000, loss: 0.4395, lr: 0.009181, batch_cost: 0.8387, reader_cost: 0.00014, ips: 1.1923 samples/sec | ETA 08:27:58 2021-05-06 12:26:06 [INFO] [TRAIN] epoch: 10, iter: 3670/40000, loss: 0.5916, lr: 0.009179, batch_cost: 0.8410, reader_cost: 0.00014, ips: 1.1891 samples/sec | ETA 08:29:13 2021-05-06 12:26:15 [INFO] [TRAIN] epoch: 10, iter: 3680/40000, loss: 0.7676, lr: 0.009177, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1904 samples/sec | ETA 08:28:31 2021-05-06 12:26:23 [INFO] [TRAIN] epoch: 10, iter: 3690/40000, loss: 1.0064, lr: 0.009174, batch_cost: 0.8416, reader_cost: 0.00014, ips: 1.1882 samples/sec | ETA 08:29:18 2021-05-06 12:26:32 [INFO] [TRAIN] epoch: 10, iter: 3700/40000, loss: 0.5635, lr: 0.009172, batch_cost: 0.8412, reader_cost: 0.00014, ips: 1.1888 samples/sec | ETA 08:28:54 2021-05-06 12:26:40 [INFO] [TRAIN] epoch: 10, iter: 3710/40000, loss: 0.6275, lr: 0.009170, batch_cost: 0.8401, reader_cost: 0.00013, ips: 1.1903 samples/sec | ETA 08:28:08 2021-05-06 12:26:48 [INFO] [TRAIN] epoch: 10, iter: 3720/40000, loss: 0.6953, lr: 0.009168, batch_cost: 0.8387, reader_cost: 0.00009, ips: 1.1923 samples/sec | ETA 08:27:07 2021-05-06 12:27:00 [INFO] [TRAIN] epoch: 11, iter: 3730/40000, loss: 0.8506, lr: 0.009165, batch_cost: 1.1647, reader_cost: 0.21028, ips: 0.8586 samples/sec | ETA 11:44:02 2021-05-06 12:27:08 [INFO] [TRAIN] epoch: 11, iter: 3740/40000, loss: 0.6906, lr: 0.009163, batch_cost: 0.8412, reader_cost: 0.00031, ips: 1.1888 samples/sec | ETA 08:28:21 2021-05-06 12:27:17 [INFO] [TRAIN] epoch: 11, iter: 3750/40000, loss: 0.6212, lr: 0.009161, batch_cost: 0.8417, reader_cost: 0.00016, ips: 1.1880 samples/sec | ETA 08:28:32 2021-05-06 12:27:25 [INFO] [TRAIN] epoch: 11, iter: 3760/40000, loss: 0.2989, lr: 0.009159, batch_cost: 0.8401, reader_cost: 0.00017, ips: 1.1903 samples/sec | ETA 08:27:24 2021-05-06 12:27:34 [INFO] [TRAIN] epoch: 11, iter: 3770/40000, loss: 0.6348, lr: 0.009156, batch_cost: 0.8419, reader_cost: 0.00016, ips: 1.1878 samples/sec | ETA 08:28:22 2021-05-06 12:27:42 [INFO] [TRAIN] epoch: 11, iter: 3780/40000, loss: 0.7661, lr: 0.009154, batch_cost: 0.8420, reader_cost: 0.00015, ips: 1.1877 samples/sec | ETA 08:28:17 2021-05-06 12:27:50 [INFO] [TRAIN] epoch: 11, iter: 3790/40000, loss: 0.8904, lr: 0.009152, batch_cost: 0.8412, reader_cost: 0.00016, ips: 1.1887 samples/sec | ETA 08:27:41 2021-05-06 12:27:59 [INFO] [TRAIN] epoch: 11, iter: 3800/40000, loss: 0.8763, lr: 0.009150, batch_cost: 0.8420, reader_cost: 0.00016, ips: 1.1877 samples/sec | ETA 08:27:59 2021-05-06 12:28:07 [INFO] [TRAIN] epoch: 11, iter: 3810/40000, loss: 0.7071, lr: 0.009147, batch_cost: 0.8430, reader_cost: 0.00015, ips: 1.1862 samples/sec | ETA 08:28:29 2021-05-06 12:28:16 [INFO] [TRAIN] epoch: 11, iter: 3820/40000, loss: 0.2610, lr: 0.009145, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 08:26:41 2021-05-06 12:28:24 [INFO] [TRAIN] epoch: 11, iter: 3830/40000, loss: 0.5013, lr: 0.009143, batch_cost: 0.8402, reader_cost: 0.00014, ips: 1.1901 samples/sec | ETA 08:26:31 2021-05-06 12:28:33 [INFO] [TRAIN] epoch: 11, iter: 3840/40000, loss: 0.6027, lr: 0.009141, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 08:26:38 2021-05-06 12:28:41 [INFO] [TRAIN] epoch: 11, iter: 3850/40000, loss: 0.6876, lr: 0.009138, batch_cost: 0.8401, reader_cost: 0.00016, ips: 1.1903 samples/sec | ETA 08:26:09 2021-05-06 12:28:49 [INFO] [TRAIN] epoch: 11, iter: 3860/40000, loss: 0.9718, lr: 0.009136, batch_cost: 0.8391, reader_cost: 0.00016, ips: 1.1918 samples/sec | ETA 08:25:23 2021-05-06 12:28:58 [INFO] [TRAIN] epoch: 11, iter: 3870/40000, loss: 0.5261, lr: 0.009134, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 08:25:48 2021-05-06 12:29:06 [INFO] [TRAIN] epoch: 11, iter: 3880/40000, loss: 0.8196, lr: 0.009132, batch_cost: 0.8393, reader_cost: 0.00014, ips: 1.1914 samples/sec | ETA 08:25:16 2021-05-06 12:29:15 [INFO] [TRAIN] epoch: 11, iter: 3890/40000, loss: 0.4749, lr: 0.009129, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1887 samples/sec | ETA 08:26:17 2021-05-06 12:29:23 [INFO] [TRAIN] epoch: 11, iter: 3900/40000, loss: 0.8096, lr: 0.009127, batch_cost: 0.8399, reader_cost: 0.00016, ips: 1.1907 samples/sec | ETA 08:25:18 2021-05-06 12:29:31 [INFO] [TRAIN] epoch: 11, iter: 3910/40000, loss: 0.6750, lr: 0.009125, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1883 samples/sec | ETA 08:26:11 2021-05-06 12:29:40 [INFO] [TRAIN] epoch: 11, iter: 3920/40000, loss: 0.7470, lr: 0.009123, batch_cost: 0.8399, reader_cost: 0.00016, ips: 1.1906 samples/sec | ETA 08:25:05 2021-05-06 12:29:48 [INFO] [TRAIN] epoch: 11, iter: 3930/40000, loss: 0.6307, lr: 0.009120, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 08:25:15 2021-05-06 12:29:57 [INFO] [TRAIN] epoch: 11, iter: 3940/40000, loss: 0.6379, lr: 0.009118, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 08:24:44 2021-05-06 12:30:05 [INFO] [TRAIN] epoch: 11, iter: 3950/40000, loss: 0.2541, lr: 0.009116, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 08:25:20 2021-05-06 12:30:13 [INFO] [TRAIN] epoch: 11, iter: 3960/40000, loss: 0.6903, lr: 0.009114, batch_cost: 0.8411, reader_cost: 0.00017, ips: 1.1890 samples/sec | ETA 08:25:12 2021-05-06 12:30:22 [INFO] [TRAIN] epoch: 11, iter: 3970/40000, loss: 0.3567, lr: 0.009111, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 08:25:05 2021-05-06 12:30:30 [INFO] [TRAIN] epoch: 11, iter: 3980/40000, loss: 0.6857, lr: 0.009109, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 08:24:41 2021-05-06 12:30:39 [INFO] [TRAIN] epoch: 11, iter: 3990/40000, loss: 0.4051, lr: 0.009107, batch_cost: 0.8417, reader_cost: 0.00016, ips: 1.1881 samples/sec | ETA 08:25:10 2021-05-06 12:30:47 [INFO] [TRAIN] epoch: 11, iter: 4000/40000, loss: 0.4403, lr: 0.009105, batch_cost: 0.8405, reader_cost: 0.00017, ips: 1.1898 samples/sec | ETA 08:24:17 2021-05-06 12:30:47 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 12:34:24 [INFO] [EVAL] #Images: 500 mIoU: 0.6182 Acc: 0.9350 Kappa: 0.9154 2021-05-06 12:34:24 [INFO] [EVAL] Class IoU: [0.9689 0.7572 0.8781 0.4455 0.4591 0.3592 0.3842 0.6022 0.8936 0.5605 0.9174 0.6939 0.3812 0.8956 0.5448 0.5656 0.3986 0.3985 0.6421] 2021-05-06 12:34:24 [INFO] [EVAL] Class Acc: [0.9874 0.8343 0.9044 0.7916 0.7728 0.7729 0.7968 0.8642 0.9474 0.7832 0.9729 0.8483 0.6967 0.9204 0.95 0.7617 0.4322 0.8018 0.7465] 2021-05-06 12:35:11 [INFO] [EVAL] The model with the best validation mIoU (0.6182) was saved at iter 4000. 2021-05-06 12:35:20 [INFO] [TRAIN] epoch: 11, iter: 4010/40000, loss: 0.6406, lr: 0.009102, batch_cost: 0.8378, reader_cost: 0.00026, ips: 1.1935 samples/sec | ETA 08:22:34 2021-05-06 12:35:28 [INFO] [TRAIN] epoch: 11, iter: 4020/40000, loss: 0.6063, lr: 0.009100, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 08:23:33 2021-05-06 12:35:36 [INFO] [TRAIN] epoch: 11, iter: 4030/40000, loss: 0.4049, lr: 0.009098, batch_cost: 0.8386, reader_cost: 0.00015, ips: 1.1924 samples/sec | ETA 08:22:45 2021-05-06 12:35:45 [INFO] [TRAIN] epoch: 11, iter: 4040/40000, loss: 0.5771, lr: 0.009096, batch_cost: 0.8394, reader_cost: 0.00014, ips: 1.1913 samples/sec | ETA 08:23:04 2021-05-06 12:35:53 [INFO] [TRAIN] epoch: 11, iter: 4050/40000, loss: 0.5663, lr: 0.009093, batch_cost: 0.8395, reader_cost: 0.00014, ips: 1.1912 samples/sec | ETA 08:23:00 2021-05-06 12:36:02 [INFO] [TRAIN] epoch: 11, iter: 4060/40000, loss: 0.6282, lr: 0.009091, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 08:23:38 2021-05-06 12:36:10 [INFO] [TRAIN] epoch: 11, iter: 4070/40000, loss: 0.5156, lr: 0.009089, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 08:23:38 2021-05-06 12:36:18 [INFO] [TRAIN] epoch: 11, iter: 4080/40000, loss: 0.5908, lr: 0.009087, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1884 samples/sec | ETA 08:23:46 2021-05-06 12:36:27 [INFO] [TRAIN] epoch: 11, iter: 4090/40000, loss: 0.6240, lr: 0.009084, batch_cost: 0.8398, reader_cost: 0.00029, ips: 1.1907 samples/sec | ETA 08:22:38 2021-05-06 12:36:39 [INFO] [TRAIN] epoch: 12, iter: 4100/40000, loss: 0.8733, lr: 0.009082, batch_cost: 1.2003, reader_cost: 0.19375, ips: 0.8331 samples/sec | ETA 11:58:11 2021-05-06 12:36:47 [INFO] [TRAIN] epoch: 12, iter: 4110/40000, loss: 0.7438, lr: 0.009080, batch_cost: 0.8413, reader_cost: 0.00032, ips: 1.1887 samples/sec | ETA 08:23:13 2021-05-06 12:36:56 [INFO] [TRAIN] epoch: 12, iter: 4120/40000, loss: 0.9460, lr: 0.009078, batch_cost: 0.8421, reader_cost: 0.00015, ips: 1.1875 samples/sec | ETA 08:23:33 2021-05-06 12:37:04 [INFO] [TRAIN] epoch: 12, iter: 4130/40000, loss: 0.4601, lr: 0.009075, batch_cost: 0.8411, reader_cost: 0.00016, ips: 1.1890 samples/sec | ETA 08:22:48 2021-05-06 12:37:12 [INFO] [TRAIN] epoch: 12, iter: 4140/40000, loss: 0.4991, lr: 0.009073, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1882 samples/sec | ETA 08:23:00 2021-05-06 12:37:21 [INFO] [TRAIN] epoch: 12, iter: 4150/40000, loss: 0.8571, lr: 0.009071, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 08:21:42 2021-05-06 12:37:29 [INFO] [TRAIN] epoch: 12, iter: 4160/40000, loss: 0.8276, lr: 0.009069, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 08:21:38 2021-05-06 12:37:38 [INFO] [TRAIN] epoch: 12, iter: 4170/40000, loss: 0.8012, lr: 0.009066, batch_cost: 0.8390, reader_cost: 0.00015, ips: 1.1918 samples/sec | ETA 08:21:02 2021-05-06 12:37:46 [INFO] [TRAIN] epoch: 12, iter: 4180/40000, loss: 0.4778, lr: 0.009064, batch_cost: 0.8398, reader_cost: 0.00016, ips: 1.1907 samples/sec | ETA 08:21:22 2021-05-06 12:37:54 [INFO] [TRAIN] epoch: 12, iter: 4190/40000, loss: 0.2609, lr: 0.009062, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1882 samples/sec | ETA 08:22:17 2021-05-06 12:38:03 [INFO] [TRAIN] epoch: 12, iter: 4200/40000, loss: 0.4318, lr: 0.009060, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 08:21:21 2021-05-06 12:38:11 [INFO] [TRAIN] epoch: 12, iter: 4210/40000, loss: 0.4218, lr: 0.009057, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 08:21:23 2021-05-06 12:38:20 [INFO] [TRAIN] epoch: 12, iter: 4220/40000, loss: 0.5256, lr: 0.009055, batch_cost: 0.8400, reader_cost: 0.00014, ips: 1.1905 samples/sec | ETA 08:20:55 2021-05-06 12:38:28 [INFO] [TRAIN] epoch: 12, iter: 4230/40000, loss: 0.9695, lr: 0.009053, batch_cost: 0.8383, reader_cost: 0.00015, ips: 1.1928 samples/sec | ETA 08:19:47 2021-05-06 12:38:36 [INFO] [TRAIN] epoch: 12, iter: 4240/40000, loss: 0.6720, lr: 0.009051, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 08:20:34 2021-05-06 12:38:45 [INFO] [TRAIN] epoch: 12, iter: 4250/40000, loss: 0.4960, lr: 0.009048, batch_cost: 0.8389, reader_cost: 0.00015, ips: 1.1920 samples/sec | ETA 08:19:50 2021-05-06 12:38:53 [INFO] [TRAIN] epoch: 12, iter: 4260/40000, loss: 0.4498, lr: 0.009046, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1911 samples/sec | ETA 08:20:04 2021-05-06 12:39:02 [INFO] [TRAIN] epoch: 12, iter: 4270/40000, loss: 0.5434, lr: 0.009044, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 08:20:40 2021-05-06 12:39:10 [INFO] [TRAIN] epoch: 12, iter: 4280/40000, loss: 0.9349, lr: 0.009042, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 08:20:28 2021-05-06 12:39:18 [INFO] [TRAIN] epoch: 12, iter: 4290/40000, loss: 0.8242, lr: 0.009039, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 08:20:35 2021-05-06 12:39:27 [INFO] [TRAIN] epoch: 12, iter: 4300/40000, loss: 0.5274, lr: 0.009037, batch_cost: 0.8430, reader_cost: 0.00015, ips: 1.1863 samples/sec | ETA 08:21:33 2021-05-06 12:39:35 [INFO] [TRAIN] epoch: 12, iter: 4310/40000, loss: 0.8923, lr: 0.009035, batch_cost: 0.8416, reader_cost: 0.00014, ips: 1.1882 samples/sec | ETA 08:20:36 2021-05-06 12:39:44 [INFO] [TRAIN] epoch: 12, iter: 4320/40000, loss: 0.2698, lr: 0.009033, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 08:20:06 2021-05-06 12:39:52 [INFO] [TRAIN] epoch: 12, iter: 4330/40000, loss: 0.8119, lr: 0.009030, batch_cost: 0.8408, reader_cost: 0.00014, ips: 1.1893 samples/sec | ETA 08:19:52 2021-05-06 12:40:01 [INFO] [TRAIN] epoch: 12, iter: 4340/40000, loss: 0.3946, lr: 0.009028, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 08:18:48 2021-05-06 12:40:09 [INFO] [TRAIN] epoch: 12, iter: 4350/40000, loss: 0.5607, lr: 0.009026, batch_cost: 0.8408, reader_cost: 0.00016, ips: 1.1893 samples/sec | ETA 08:19:34 2021-05-06 12:40:17 [INFO] [TRAIN] epoch: 12, iter: 4360/40000, loss: 0.4343, lr: 0.009024, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1883 samples/sec | ETA 08:19:51 2021-05-06 12:40:26 [INFO] [TRAIN] epoch: 12, iter: 4370/40000, loss: 0.4226, lr: 0.009021, batch_cost: 0.8407, reader_cost: 0.00017, ips: 1.1895 samples/sec | ETA 08:19:13 2021-05-06 12:40:34 [INFO] [TRAIN] epoch: 12, iter: 4380/40000, loss: 0.5030, lr: 0.009019, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1886 samples/sec | ETA 08:19:27 2021-05-06 12:40:43 [INFO] [TRAIN] epoch: 12, iter: 4390/40000, loss: 0.6272, lr: 0.009017, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 08:18:44 2021-05-06 12:40:51 [INFO] [TRAIN] epoch: 12, iter: 4400/40000, loss: 0.5467, lr: 0.009015, batch_cost: 0.8400, reader_cost: 0.00014, ips: 1.1905 samples/sec | ETA 08:18:24 2021-05-06 12:40:59 [INFO] [TRAIN] epoch: 12, iter: 4410/40000, loss: 0.4212, lr: 0.009012, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1904 samples/sec | ETA 08:18:17 2021-05-06 12:41:08 [INFO] [TRAIN] epoch: 12, iter: 4420/40000, loss: 0.6941, lr: 0.009010, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 08:18:16 2021-05-06 12:41:16 [INFO] [TRAIN] epoch: 12, iter: 4430/40000, loss: 0.7012, lr: 0.009008, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 08:18:13 2021-05-06 12:41:25 [INFO] [TRAIN] epoch: 12, iter: 4440/40000, loss: 1.2822, lr: 0.009005, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1883 samples/sec | ETA 08:18:44 2021-05-06 12:41:33 [INFO] [TRAIN] epoch: 12, iter: 4450/40000, loss: 0.9510, lr: 0.009003, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 08:17:37 2021-05-06 12:41:41 [INFO] [TRAIN] epoch: 12, iter: 4460/40000, loss: 0.7784, lr: 0.009001, batch_cost: 0.8390, reader_cost: 0.00012, ips: 1.1919 samples/sec | ETA 08:16:56 2021-05-06 12:41:53 [INFO] [TRAIN] epoch: 13, iter: 4470/40000, loss: 1.0168, lr: 0.008999, batch_cost: 1.1377, reader_cost: 0.20157, ips: 0.8790 samples/sec | ETA 11:13:41 2021-05-06 12:42:01 [INFO] [TRAIN] epoch: 13, iter: 4480/40000, loss: 0.4667, lr: 0.008996, batch_cost: 0.8406, reader_cost: 0.00033, ips: 1.1897 samples/sec | ETA 08:17:36 2021-05-06 12:42:10 [INFO] [TRAIN] epoch: 13, iter: 4490/40000, loss: 0.5626, lr: 0.008994, batch_cost: 0.8406, reader_cost: 0.00016, ips: 1.1896 samples/sec | ETA 08:17:30 2021-05-06 12:42:18 [INFO] [TRAIN] epoch: 13, iter: 4500/40000, loss: 0.3340, lr: 0.008992, batch_cost: 0.8414, reader_cost: 0.00016, ips: 1.1884 samples/sec | ETA 08:17:50 2021-05-06 12:42:26 [INFO] [TRAIN] epoch: 13, iter: 4510/40000, loss: 0.3955, lr: 0.008990, batch_cost: 0.8412, reader_cost: 0.00018, ips: 1.1888 samples/sec | ETA 08:17:33 2021-05-06 12:42:35 [INFO] [TRAIN] epoch: 13, iter: 4520/40000, loss: 0.9583, lr: 0.008987, batch_cost: 0.8436, reader_cost: 0.00015, ips: 1.1854 samples/sec | ETA 08:18:50 2021-05-06 12:42:43 [INFO] [TRAIN] epoch: 13, iter: 4530/40000, loss: 0.7908, lr: 0.008985, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 08:16:47 2021-05-06 12:42:52 [INFO] [TRAIN] epoch: 13, iter: 4540/40000, loss: 0.8850, lr: 0.008983, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 08:16:37 2021-05-06 12:43:00 [INFO] [TRAIN] epoch: 13, iter: 4550/40000, loss: 0.6500, lr: 0.008981, batch_cost: 0.8391, reader_cost: 0.00014, ips: 1.1917 samples/sec | ETA 08:15:47 2021-05-06 12:43:08 [INFO] [TRAIN] epoch: 13, iter: 4560/40000, loss: 0.3079, lr: 0.008978, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 08:15:58 2021-05-06 12:43:17 [INFO] [TRAIN] epoch: 13, iter: 4570/40000, loss: 0.5341, lr: 0.008976, batch_cost: 0.8420, reader_cost: 0.00014, ips: 1.1877 samples/sec | ETA 08:17:11 2021-05-06 12:43:25 [INFO] [TRAIN] epoch: 13, iter: 4580/40000, loss: 0.7418, lr: 0.008974, batch_cost: 0.8405, reader_cost: 0.00016, ips: 1.1897 samples/sec | ETA 08:16:12 2021-05-06 12:43:34 [INFO] [TRAIN] epoch: 13, iter: 4590/40000, loss: 0.8065, lr: 0.008972, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 08:16:06 2021-05-06 12:43:42 [INFO] [TRAIN] epoch: 13, iter: 4600/40000, loss: 0.8931, lr: 0.008969, batch_cost: 0.8413, reader_cost: 0.00016, ips: 1.1887 samples/sec | ETA 08:16:21 2021-05-06 12:43:51 [INFO] [TRAIN] epoch: 13, iter: 4610/40000, loss: 0.6389, lr: 0.008967, batch_cost: 0.8419, reader_cost: 0.00016, ips: 1.1878 samples/sec | ETA 08:16:35 2021-05-06 12:43:59 [INFO] [TRAIN] epoch: 13, iter: 4620/40000, loss: 0.4954, lr: 0.008965, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 08:15:48 2021-05-06 12:44:07 [INFO] [TRAIN] epoch: 13, iter: 4630/40000, loss: 0.6242, lr: 0.008963, batch_cost: 0.8408, reader_cost: 0.00016, ips: 1.1893 samples/sec | ETA 08:15:39 2021-05-06 12:44:16 [INFO] [TRAIN] epoch: 13, iter: 4640/40000, loss: 0.5156, lr: 0.008960, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 08:15:03 2021-05-06 12:44:24 [INFO] [TRAIN] epoch: 13, iter: 4650/40000, loss: 0.3986, lr: 0.008958, batch_cost: 0.8414, reader_cost: 0.00016, ips: 1.1884 samples/sec | ETA 08:15:44 2021-05-06 12:44:33 [INFO] [TRAIN] epoch: 13, iter: 4660/40000, loss: 0.9415, lr: 0.008956, batch_cost: 0.8383, reader_cost: 0.00015, ips: 1.1928 samples/sec | ETA 08:13:47 2021-05-06 12:44:41 [INFO] [TRAIN] epoch: 13, iter: 4670/40000, loss: 0.5972, lr: 0.008954, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1881 samples/sec | ETA 08:15:37 2021-05-06 12:44:49 [INFO] [TRAIN] epoch: 13, iter: 4680/40000, loss: 0.5200, lr: 0.008951, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1886 samples/sec | ETA 08:15:16 2021-05-06 12:44:58 [INFO] [TRAIN] epoch: 13, iter: 4690/40000, loss: 0.6302, lr: 0.008949, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 08:14:34 2021-05-06 12:45:06 [INFO] [TRAIN] epoch: 13, iter: 4700/40000, loss: 0.4929, lr: 0.008947, batch_cost: 0.8410, reader_cost: 0.00014, ips: 1.1890 samples/sec | ETA 08:14:47 2021-05-06 12:45:15 [INFO] [TRAIN] epoch: 13, iter: 4710/40000, loss: 0.3616, lr: 0.008945, batch_cost: 0.8408, reader_cost: 0.00017, ips: 1.1893 samples/sec | ETA 08:14:33 2021-05-06 12:45:23 [INFO] [TRAIN] epoch: 13, iter: 4720/40000, loss: 0.6215, lr: 0.008942, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 08:13:53 2021-05-06 12:45:31 [INFO] [TRAIN] epoch: 13, iter: 4730/40000, loss: 0.4217, lr: 0.008940, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 08:14:03 2021-05-06 12:45:40 [INFO] [TRAIN] epoch: 13, iter: 4740/40000, loss: 0.6951, lr: 0.008938, batch_cost: 0.8403, reader_cost: 0.00014, ips: 1.1900 samples/sec | ETA 08:13:50 2021-05-06 12:45:48 [INFO] [TRAIN] epoch: 13, iter: 4750/40000, loss: 0.4723, lr: 0.008936, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 08:13:23 2021-05-06 12:45:57 [INFO] [TRAIN] epoch: 13, iter: 4760/40000, loss: 1.5615, lr: 0.008933, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 08:13:18 2021-05-06 12:46:05 [INFO] [TRAIN] epoch: 13, iter: 4770/40000, loss: 0.4981, lr: 0.008931, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 08:13:00 2021-05-06 12:46:13 [INFO] [TRAIN] epoch: 13, iter: 4780/40000, loss: 0.5481, lr: 0.008929, batch_cost: 0.8420, reader_cost: 0.00015, ips: 1.1877 samples/sec | ETA 08:14:14 2021-05-06 12:46:22 [INFO] [TRAIN] epoch: 13, iter: 4790/40000, loss: 0.8379, lr: 0.008927, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 08:13:09 2021-05-06 12:46:30 [INFO] [TRAIN] epoch: 13, iter: 4800/40000, loss: 0.5020, lr: 0.008924, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1882 samples/sec | ETA 08:13:45 2021-05-06 12:46:39 [INFO] [TRAIN] epoch: 13, iter: 4810/40000, loss: 0.6155, lr: 0.008922, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 08:13:03 2021-05-06 12:46:47 [INFO] [TRAIN] epoch: 13, iter: 4820/40000, loss: 0.8116, lr: 0.008920, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 08:12:51 2021-05-06 12:46:55 [INFO] [TRAIN] epoch: 13, iter: 4830/40000, loss: 0.5366, lr: 0.008918, batch_cost: 0.8383, reader_cost: 0.00012, ips: 1.1929 samples/sec | ETA 08:11:23 2021-05-06 12:47:07 [INFO] [TRAIN] epoch: 14, iter: 4840/40000, loss: 0.7546, lr: 0.008915, batch_cost: 1.1377, reader_cost: 0.18875, ips: 0.8790 samples/sec | ETA 11:06:42 2021-05-06 12:47:15 [INFO] [TRAIN] epoch: 14, iter: 4850/40000, loss: 0.6735, lr: 0.008913, batch_cost: 0.8573, reader_cost: 0.00030, ips: 1.1665 samples/sec | ETA 08:22:13 2021-05-06 12:47:24 [INFO] [TRAIN] epoch: 14, iter: 4860/40000, loss: 0.5425, lr: 0.008911, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1914 samples/sec | ETA 08:11:34 2021-05-06 12:47:32 [INFO] [TRAIN] epoch: 14, iter: 4870/40000, loss: 0.5639, lr: 0.008909, batch_cost: 0.8423, reader_cost: 0.00015, ips: 1.1872 samples/sec | ETA 08:13:10 2021-05-06 12:47:41 [INFO] [TRAIN] epoch: 14, iter: 4880/40000, loss: 0.5776, lr: 0.008906, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 08:12:08 2021-05-06 12:47:49 [INFO] [TRAIN] epoch: 14, iter: 4890/40000, loss: 0.6286, lr: 0.008904, batch_cost: 0.8406, reader_cost: 0.00014, ips: 1.1896 samples/sec | ETA 08:11:53 2021-05-06 12:47:57 [INFO] [TRAIN] epoch: 14, iter: 4900/40000, loss: 0.6870, lr: 0.008902, batch_cost: 0.8398, reader_cost: 0.00014, ips: 1.1908 samples/sec | ETA 08:11:16 2021-05-06 12:48:06 [INFO] [TRAIN] epoch: 14, iter: 4910/40000, loss: 0.8827, lr: 0.008899, batch_cost: 0.8393, reader_cost: 0.00016, ips: 1.1914 samples/sec | ETA 08:10:52 2021-05-06 12:48:14 [INFO] [TRAIN] epoch: 14, iter: 4920/40000, loss: 0.9875, lr: 0.008897, batch_cost: 0.8411, reader_cost: 0.00014, ips: 1.1889 samples/sec | ETA 08:11:46 2021-05-06 12:48:23 [INFO] [TRAIN] epoch: 14, iter: 4930/40000, loss: 0.4986, lr: 0.008895, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 08:11:25 2021-05-06 12:48:31 [INFO] [TRAIN] epoch: 14, iter: 4940/40000, loss: 0.4123, lr: 0.008893, batch_cost: 0.8408, reader_cost: 0.00014, ips: 1.1894 samples/sec | ETA 08:11:17 2021-05-06 12:48:39 [INFO] [TRAIN] epoch: 14, iter: 4950/40000, loss: 0.5864, lr: 0.008890, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 08:10:55 2021-05-06 12:48:48 [INFO] [TRAIN] epoch: 14, iter: 4960/40000, loss: 0.7795, lr: 0.008888, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 08:11:09 2021-05-06 12:48:56 [INFO] [TRAIN] epoch: 14, iter: 4970/40000, loss: 0.5531, lr: 0.008886, batch_cost: 0.8389, reader_cost: 0.00016, ips: 1.1920 samples/sec | ETA 08:09:47 2021-05-06 12:49:05 [INFO] [TRAIN] epoch: 14, iter: 4980/40000, loss: 0.7266, lr: 0.008884, batch_cost: 0.8369, reader_cost: 0.00015, ips: 1.1949 samples/sec | ETA 08:08:28 2021-05-06 12:49:13 [INFO] [TRAIN] epoch: 14, iter: 4990/40000, loss: 0.5446, lr: 0.008881, batch_cost: 0.8405, reader_cost: 0.00014, ips: 1.1897 samples/sec | ETA 08:10:27 2021-05-06 12:49:21 [INFO] [TRAIN] epoch: 14, iter: 5000/40000, loss: 0.7079, lr: 0.008879, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 08:09:56 2021-05-06 12:49:21 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 12:52:58 [INFO] [EVAL] #Images: 500 mIoU: 0.6238 Acc: 0.9397 Kappa: 0.9215 2021-05-06 12:52:58 [INFO] [EVAL] Class IoU: [0.9717 0.7812 0.8898 0.5521 0.437 0.4052 0.4691 0.6111 0.8994 0.5903 0.93 0.6858 0.2567 0.9023 0.481 0.598 0.4929 0.2459 0.6526] 2021-05-06 12:52:58 [INFO] [EVAL] Class Acc: [0.9849 0.8964 0.9163 0.7294 0.7707 0.7403 0.6313 0.8918 0.9542 0.7766 0.9577 0.7747 0.6958 0.9238 0.9155 0.7917 0.5511 0.9066 0.7727] 2021-05-06 12:53:46 [INFO] [EVAL] The model with the best validation mIoU (0.6238) was saved at iter 5000. 2021-05-06 12:53:54 [INFO] [TRAIN] epoch: 14, iter: 5010/40000, loss: 0.4335, lr: 0.008877, batch_cost: 0.8361, reader_cost: 0.00023, ips: 1.1961 samples/sec | ETA 08:07:34 2021-05-06 12:54:02 [INFO] [TRAIN] epoch: 14, iter: 5020/40000, loss: 1.7515, lr: 0.008875, batch_cost: 0.8377, reader_cost: 0.00015, ips: 1.1938 samples/sec | ETA 08:08:21 2021-05-06 12:54:11 [INFO] [TRAIN] epoch: 14, iter: 5030/40000, loss: 0.6626, lr: 0.008872, batch_cost: 0.8384, reader_cost: 0.00015, ips: 1.1928 samples/sec | ETA 08:08:38 2021-05-06 12:54:19 [INFO] [TRAIN] epoch: 14, iter: 5040/40000, loss: 0.6790, lr: 0.008870, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 08:09:47 2021-05-06 12:54:28 [INFO] [TRAIN] epoch: 14, iter: 5050/40000, loss: 0.8266, lr: 0.008868, batch_cost: 0.8408, reader_cost: 0.00014, ips: 1.1894 samples/sec | ETA 08:09:44 2021-05-06 12:54:36 [INFO] [TRAIN] epoch: 14, iter: 5060/40000, loss: 0.5101, lr: 0.008866, batch_cost: 0.8400, reader_cost: 0.00014, ips: 1.1905 samples/sec | ETA 08:09:09 2021-05-06 12:54:44 [INFO] [TRAIN] epoch: 14, iter: 5070/40000, loss: 0.4673, lr: 0.008863, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 08:09:27 2021-05-06 12:54:53 [INFO] [TRAIN] epoch: 14, iter: 5080/40000, loss: 0.5904, lr: 0.008861, batch_cost: 0.8423, reader_cost: 0.00014, ips: 1.1873 samples/sec | ETA 08:10:11 2021-05-06 12:55:01 [INFO] [TRAIN] epoch: 14, iter: 5090/40000, loss: 0.3690, lr: 0.008859, batch_cost: 0.8403, reader_cost: 0.00014, ips: 1.1900 samples/sec | ETA 08:08:55 2021-05-06 12:55:10 [INFO] [TRAIN] epoch: 14, iter: 5100/40000, loss: 0.5722, lr: 0.008857, batch_cost: 0.8400, reader_cost: 0.00014, ips: 1.1904 samples/sec | ETA 08:08:36 2021-05-06 12:55:18 [INFO] [TRAIN] epoch: 14, iter: 5110/40000, loss: 0.4119, lr: 0.008854, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 08:08:22 2021-05-06 12:55:27 [INFO] [TRAIN] epoch: 14, iter: 5120/40000, loss: 0.4707, lr: 0.008852, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1888 samples/sec | ETA 08:08:59 2021-05-06 12:55:35 [INFO] [TRAIN] epoch: 14, iter: 5130/40000, loss: 0.7727, lr: 0.008850, batch_cost: 0.8397, reader_cost: 0.00014, ips: 1.1909 samples/sec | ETA 08:08:01 2021-05-06 12:55:43 [INFO] [TRAIN] epoch: 14, iter: 5140/40000, loss: 0.4340, lr: 0.008848, batch_cost: 0.8407, reader_cost: 0.00017, ips: 1.1895 samples/sec | ETA 08:08:26 2021-05-06 12:55:52 [INFO] [TRAIN] epoch: 14, iter: 5150/40000, loss: 0.5764, lr: 0.008845, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1880 samples/sec | ETA 08:08:55 2021-05-06 12:56:00 [INFO] [TRAIN] epoch: 14, iter: 5160/40000, loss: 0.7520, lr: 0.008843, batch_cost: 0.8408, reader_cost: 0.00014, ips: 1.1894 samples/sec | ETA 08:08:12 2021-05-06 12:56:09 [INFO] [TRAIN] epoch: 14, iter: 5170/40000, loss: 0.4814, lr: 0.008841, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 08:08:10 2021-05-06 12:56:17 [INFO] [TRAIN] epoch: 14, iter: 5180/40000, loss: 0.4707, lr: 0.008839, batch_cost: 0.8420, reader_cost: 0.00015, ips: 1.1877 samples/sec | ETA 08:08:37 2021-05-06 12:56:25 [INFO] [TRAIN] epoch: 14, iter: 5190/40000, loss: 0.4931, lr: 0.008836, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 08:07:06 2021-05-06 12:56:34 [INFO] [TRAIN] epoch: 14, iter: 5200/40000, loss: 0.6266, lr: 0.008834, batch_cost: 0.8396, reader_cost: 0.00013, ips: 1.1910 samples/sec | ETA 08:06:59 2021-05-06 12:56:45 [INFO] [TRAIN] epoch: 15, iter: 5210/40000, loss: 0.6388, lr: 0.008832, batch_cost: 1.1488, reader_cost: 0.21444, ips: 0.8705 samples/sec | ETA 11:06:06 2021-05-06 12:56:54 [INFO] [TRAIN] epoch: 15, iter: 5220/40000, loss: 0.6116, lr: 0.008829, batch_cost: 0.8548, reader_cost: 0.00033, ips: 1.1699 samples/sec | ETA 08:15:29 2021-05-06 12:57:02 [INFO] [TRAIN] epoch: 15, iter: 5230/40000, loss: 0.8478, lr: 0.008827, batch_cost: 0.8419, reader_cost: 0.00014, ips: 1.1879 samples/sec | ETA 08:07:51 2021-05-06 12:57:11 [INFO] [TRAIN] epoch: 15, iter: 5240/40000, loss: 0.5896, lr: 0.008825, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 08:06:58 2021-05-06 12:57:19 [INFO] [TRAIN] epoch: 15, iter: 5250/40000, loss: 0.3982, lr: 0.008823, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 08:06:44 2021-05-06 12:57:27 [INFO] [TRAIN] epoch: 15, iter: 5260/40000, loss: 0.4412, lr: 0.008820, batch_cost: 0.8424, reader_cost: 0.00015, ips: 1.1871 samples/sec | ETA 08:07:43 2021-05-06 12:57:36 [INFO] [TRAIN] epoch: 15, iter: 5270/40000, loss: 0.7981, lr: 0.008818, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 08:06:51 2021-05-06 12:57:44 [INFO] [TRAIN] epoch: 15, iter: 5280/40000, loss: 0.6900, lr: 0.008816, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1882 samples/sec | ETA 08:07:00 2021-05-06 12:57:53 [INFO] [TRAIN] epoch: 15, iter: 5290/40000, loss: 0.6499, lr: 0.008814, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1914 samples/sec | ETA 08:05:34 2021-05-06 12:58:01 [INFO] [TRAIN] epoch: 15, iter: 5300/40000, loss: 0.5525, lr: 0.008811, batch_cost: 0.8388, reader_cost: 0.00015, ips: 1.1921 samples/sec | ETA 08:05:07 2021-05-06 12:58:09 [INFO] [TRAIN] epoch: 15, iter: 5310/40000, loss: 0.4857, lr: 0.008809, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 08:05:45 2021-05-06 12:58:18 [INFO] [TRAIN] epoch: 15, iter: 5320/40000, loss: 0.4833, lr: 0.008807, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1884 samples/sec | ETA 08:06:23 2021-05-06 12:58:26 [INFO] [TRAIN] epoch: 15, iter: 5330/40000, loss: 0.4511, lr: 0.008805, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 08:05:52 2021-05-06 12:58:35 [INFO] [TRAIN] epoch: 15, iter: 5340/40000, loss: 0.9494, lr: 0.008802, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 08:05:47 2021-05-06 12:58:43 [INFO] [TRAIN] epoch: 15, iter: 5350/40000, loss: 0.6860, lr: 0.008800, batch_cost: 0.8392, reader_cost: 0.00015, ips: 1.1916 samples/sec | ETA 08:04:37 2021-05-06 12:58:51 [INFO] [TRAIN] epoch: 15, iter: 5360/40000, loss: 0.5243, lr: 0.008798, batch_cost: 0.8394, reader_cost: 0.00014, ips: 1.1914 samples/sec | ETA 08:04:35 2021-05-06 12:59:00 [INFO] [TRAIN] epoch: 15, iter: 5370/40000, loss: 0.4685, lr: 0.008796, batch_cost: 0.8386, reader_cost: 0.00014, ips: 1.1924 samples/sec | ETA 08:04:01 2021-05-06 12:59:08 [INFO] [TRAIN] epoch: 15, iter: 5380/40000, loss: 0.4999, lr: 0.008793, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 08:05:10 2021-05-06 12:59:17 [INFO] [TRAIN] epoch: 15, iter: 5390/40000, loss: 0.8435, lr: 0.008791, batch_cost: 0.8421, reader_cost: 0.00014, ips: 1.1876 samples/sec | ETA 08:05:43 2021-05-06 12:59:25 [INFO] [TRAIN] epoch: 15, iter: 5400/40000, loss: 0.5942, lr: 0.008789, batch_cost: 0.8428, reader_cost: 0.00015, ips: 1.1865 samples/sec | ETA 08:06:01 2021-05-06 12:59:34 [INFO] [TRAIN] epoch: 15, iter: 5410/40000, loss: 0.4804, lr: 0.008787, batch_cost: 0.8400, reader_cost: 0.00014, ips: 1.1904 samples/sec | ETA 08:04:16 2021-05-06 12:59:42 [INFO] [TRAIN] epoch: 15, iter: 5420/40000, loss: 0.7250, lr: 0.008784, batch_cost: 0.8421, reader_cost: 0.00015, ips: 1.1875 samples/sec | ETA 08:05:21 2021-05-06 12:59:50 [INFO] [TRAIN] epoch: 15, iter: 5430/40000, loss: 0.3947, lr: 0.008782, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 08:04:18 2021-05-06 12:59:59 [INFO] [TRAIN] epoch: 15, iter: 5440/40000, loss: 0.4545, lr: 0.008780, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 08:03:39 2021-05-06 13:00:07 [INFO] [TRAIN] epoch: 15, iter: 5450/40000, loss: 0.3590, lr: 0.008778, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 08:03:41 2021-05-06 13:00:16 [INFO] [TRAIN] epoch: 15, iter: 5460/40000, loss: 0.5967, lr: 0.008775, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 08:03:26 2021-05-06 13:00:24 [INFO] [TRAIN] epoch: 15, iter: 5470/40000, loss: 0.5512, lr: 0.008773, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 08:03:52 2021-05-06 13:00:32 [INFO] [TRAIN] epoch: 15, iter: 5480/40000, loss: 0.3128, lr: 0.008771, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1879 samples/sec | ETA 08:04:19 2021-05-06 13:00:41 [INFO] [TRAIN] epoch: 15, iter: 5490/40000, loss: 0.5037, lr: 0.008768, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 08:02:42 2021-05-06 13:00:49 [INFO] [TRAIN] epoch: 15, iter: 5500/40000, loss: 0.8527, lr: 0.008766, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1911 samples/sec | ETA 08:02:44 2021-05-06 13:00:58 [INFO] [TRAIN] epoch: 15, iter: 5510/40000, loss: 0.4363, lr: 0.008764, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 08:02:47 2021-05-06 13:01:06 [INFO] [TRAIN] epoch: 15, iter: 5520/40000, loss: 0.6034, lr: 0.008762, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 08:02:35 2021-05-06 13:01:14 [INFO] [TRAIN] epoch: 15, iter: 5530/40000, loss: 0.7525, lr: 0.008759, batch_cost: 0.8398, reader_cost: 0.00016, ips: 1.1907 samples/sec | ETA 08:02:28 2021-05-06 13:01:23 [INFO] [TRAIN] epoch: 15, iter: 5540/40000, loss: 0.4719, lr: 0.008757, batch_cost: 0.8396, reader_cost: 0.00016, ips: 1.1911 samples/sec | ETA 08:02:11 2021-05-06 13:01:31 [INFO] [TRAIN] epoch: 15, iter: 5550/40000, loss: 0.4866, lr: 0.008755, batch_cost: 0.8411, reader_cost: 0.00017, ips: 1.1889 samples/sec | ETA 08:02:57 2021-05-06 13:01:40 [INFO] [TRAIN] epoch: 15, iter: 5560/40000, loss: 0.4909, lr: 0.008753, batch_cost: 0.8436, reader_cost: 0.00016, ips: 1.1853 samples/sec | ETA 08:04:14 2021-05-06 13:01:48 [INFO] [TRAIN] epoch: 15, iter: 5570/40000, loss: 0.3987, lr: 0.008750, batch_cost: 0.8398, reader_cost: 0.00014, ips: 1.1907 samples/sec | ETA 08:01:54 2021-05-06 13:01:56 [INFO] [TRAIN] epoch: 15, iter: 5580/40000, loss: 0.6696, lr: 0.008748, batch_cost: 0.8386, reader_cost: 0.00009, ips: 1.1924 samples/sec | ETA 08:01:05 2021-05-06 13:02:08 [INFO] [TRAIN] epoch: 16, iter: 5590/40000, loss: 0.6705, lr: 0.008746, batch_cost: 1.1706, reader_cost: 0.20926, ips: 0.8543 samples/sec | ETA 11:11:18 2021-05-06 13:02:17 [INFO] [TRAIN] epoch: 16, iter: 5600/40000, loss: 0.4592, lr: 0.008744, batch_cost: 0.8443, reader_cost: 0.00029, ips: 1.1844 samples/sec | ETA 08:04:03 2021-05-06 13:02:25 [INFO] [TRAIN] epoch: 16, iter: 5610/40000, loss: 0.5658, lr: 0.008741, batch_cost: 0.8423, reader_cost: 0.00014, ips: 1.1873 samples/sec | ETA 08:02:45 2021-05-06 13:02:33 [INFO] [TRAIN] epoch: 16, iter: 5620/40000, loss: 0.5120, lr: 0.008739, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1887 samples/sec | ETA 08:02:01 2021-05-06 13:02:42 [INFO] [TRAIN] epoch: 16, iter: 5630/40000, loss: 1.1221, lr: 0.008737, batch_cost: 0.8422, reader_cost: 0.00015, ips: 1.1874 samples/sec | ETA 08:02:25 2021-05-06 13:02:50 [INFO] [TRAIN] epoch: 16, iter: 5640/40000, loss: 0.6365, lr: 0.008735, batch_cost: 0.8423, reader_cost: 0.00015, ips: 1.1872 samples/sec | ETA 08:02:22 2021-05-06 13:02:59 [INFO] [TRAIN] epoch: 16, iter: 5650/40000, loss: 0.5505, lr: 0.008732, batch_cost: 0.8401, reader_cost: 0.00016, ips: 1.1904 samples/sec | ETA 08:00:56 2021-05-06 13:03:07 [INFO] [TRAIN] epoch: 16, iter: 5660/40000, loss: 0.9743, lr: 0.008730, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 08:00:35 2021-05-06 13:03:15 [INFO] [TRAIN] epoch: 16, iter: 5670/40000, loss: 0.5547, lr: 0.008728, batch_cost: 0.8398, reader_cost: 0.00014, ips: 1.1907 samples/sec | ETA 08:00:30 2021-05-06 13:03:24 [INFO] [TRAIN] epoch: 16, iter: 5680/40000, loss: 0.1903, lr: 0.008726, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 08:01:02 2021-05-06 13:03:32 [INFO] [TRAIN] epoch: 16, iter: 5690/40000, loss: 0.3732, lr: 0.008723, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 08:00:18 2021-05-06 13:03:41 [INFO] [TRAIN] epoch: 16, iter: 5700/40000, loss: 0.4393, lr: 0.008721, batch_cost: 0.8397, reader_cost: 0.00016, ips: 1.1910 samples/sec | ETA 08:00:00 2021-05-06 13:03:49 [INFO] [TRAIN] epoch: 16, iter: 5710/40000, loss: 0.5577, lr: 0.008719, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 08:00:38 2021-05-06 13:03:57 [INFO] [TRAIN] epoch: 16, iter: 5720/40000, loss: 0.7815, lr: 0.008716, batch_cost: 0.8398, reader_cost: 0.00016, ips: 1.1907 samples/sec | ETA 07:59:49 2021-05-06 13:04:06 [INFO] [TRAIN] epoch: 16, iter: 5730/40000, loss: 0.7989, lr: 0.008714, batch_cost: 0.8411, reader_cost: 0.00016, ips: 1.1889 samples/sec | ETA 08:00:23 2021-05-06 13:04:14 [INFO] [TRAIN] epoch: 16, iter: 5740/40000, loss: 0.4125, lr: 0.008712, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1903 samples/sec | ETA 07:59:42 2021-05-06 13:04:23 [INFO] [TRAIN] epoch: 16, iter: 5750/40000, loss: 0.4631, lr: 0.008710, batch_cost: 0.8403, reader_cost: 0.00014, ips: 1.1900 samples/sec | ETA 07:59:41 2021-05-06 13:04:31 [INFO] [TRAIN] epoch: 16, iter: 5760/40000, loss: 0.5821, lr: 0.008707, batch_cost: 0.8414, reader_cost: 0.00014, ips: 1.1884 samples/sec | ETA 08:00:11 2021-05-06 13:04:39 [INFO] [TRAIN] epoch: 16, iter: 5770/40000, loss: 0.8018, lr: 0.008705, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 07:59:39 2021-05-06 13:04:48 [INFO] [TRAIN] epoch: 16, iter: 5780/40000, loss: 0.9282, lr: 0.008703, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 07:59:06 2021-05-06 13:04:56 [INFO] [TRAIN] epoch: 16, iter: 5790/40000, loss: 0.7936, lr: 0.008701, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 07:59:16 2021-05-06 13:05:05 [INFO] [TRAIN] epoch: 16, iter: 5800/40000, loss: 0.6609, lr: 0.008698, batch_cost: 0.8422, reader_cost: 0.00015, ips: 1.1874 samples/sec | ETA 08:00:03 2021-05-06 13:05:13 [INFO] [TRAIN] epoch: 16, iter: 5810/40000, loss: 0.3473, lr: 0.008696, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 07:59:08 2021-05-06 13:05:22 [INFO] [TRAIN] epoch: 16, iter: 5820/40000, loss: 0.4998, lr: 0.008694, batch_cost: 0.8398, reader_cost: 0.00016, ips: 1.1907 samples/sec | ETA 07:58:25 2021-05-06 13:05:30 [INFO] [TRAIN] epoch: 16, iter: 5830/40000, loss: 0.3361, lr: 0.008692, batch_cost: 0.8407, reader_cost: 0.00016, ips: 1.1895 samples/sec | ETA 07:58:46 2021-05-06 13:05:38 [INFO] [TRAIN] epoch: 16, iter: 5840/40000, loss: 0.4970, lr: 0.008689, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 07:58:32 2021-05-06 13:05:47 [INFO] [TRAIN] epoch: 16, iter: 5850/40000, loss: 0.4799, lr: 0.008687, batch_cost: 0.8427, reader_cost: 0.00015, ips: 1.1867 samples/sec | ETA 07:59:38 2021-05-06 13:05:55 [INFO] [TRAIN] epoch: 16, iter: 5860/40000, loss: 0.5466, lr: 0.008685, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1884 samples/sec | ETA 07:58:46 2021-05-06 13:06:04 [INFO] [TRAIN] epoch: 16, iter: 5870/40000, loss: 0.4710, lr: 0.008683, batch_cost: 0.8414, reader_cost: 0.00017, ips: 1.1885 samples/sec | ETA 07:58:36 2021-05-06 13:06:12 [INFO] [TRAIN] epoch: 16, iter: 5880/40000, loss: 0.5444, lr: 0.008680, batch_cost: 0.8396, reader_cost: 0.00016, ips: 1.1911 samples/sec | ETA 07:57:26 2021-05-06 13:06:20 [INFO] [TRAIN] epoch: 16, iter: 5890/40000, loss: 0.5301, lr: 0.008678, batch_cost: 0.8385, reader_cost: 0.00016, ips: 1.1926 samples/sec | ETA 07:56:42 2021-05-06 13:06:29 [INFO] [TRAIN] epoch: 16, iter: 5900/40000, loss: 0.7074, lr: 0.008676, batch_cost: 0.8386, reader_cost: 0.00015, ips: 1.1924 samples/sec | ETA 07:56:37 2021-05-06 13:06:37 [INFO] [TRAIN] epoch: 16, iter: 5910/40000, loss: 0.8476, lr: 0.008673, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 07:57:31 2021-05-06 13:06:46 [INFO] [TRAIN] epoch: 16, iter: 5920/40000, loss: 0.5417, lr: 0.008671, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1885 samples/sec | ETA 07:57:54 2021-05-06 13:06:54 [INFO] [TRAIN] epoch: 16, iter: 5930/40000, loss: 0.6189, lr: 0.008669, batch_cost: 0.8427, reader_cost: 0.00016, ips: 1.1866 samples/sec | ETA 07:58:31 2021-05-06 13:07:02 [INFO] [TRAIN] epoch: 16, iter: 5940/40000, loss: 0.5228, lr: 0.008667, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 07:57:14 2021-05-06 13:07:11 [INFO] [TRAIN] epoch: 16, iter: 5950/40000, loss: 0.5008, lr: 0.008664, batch_cost: 0.8384, reader_cost: 0.00031, ips: 1.1928 samples/sec | ETA 07:55:46 2021-05-06 13:07:23 [INFO] [TRAIN] epoch: 17, iter: 5960/40000, loss: 0.5949, lr: 0.008662, batch_cost: 1.1787, reader_cost: 0.19521, ips: 0.8484 samples/sec | ETA 11:08:44 2021-05-06 13:07:31 [INFO] [TRAIN] epoch: 17, iter: 5970/40000, loss: 0.3820, lr: 0.008660, batch_cost: 0.8415, reader_cost: 0.00032, ips: 1.1884 samples/sec | ETA 07:57:15 2021-05-06 13:07:39 [INFO] [TRAIN] epoch: 17, iter: 5980/40000, loss: 0.5487, lr: 0.008658, batch_cost: 0.8408, reader_cost: 0.00016, ips: 1.1893 samples/sec | ETA 07:56:44 2021-05-06 13:07:48 [INFO] [TRAIN] epoch: 17, iter: 5990/40000, loss: 0.2957, lr: 0.008655, batch_cost: 0.8402, reader_cost: 0.00014, ips: 1.1902 samples/sec | ETA 07:56:14 2021-05-06 13:07:56 [INFO] [TRAIN] epoch: 17, iter: 6000/40000, loss: 0.4195, lr: 0.008653, batch_cost: 0.8397, reader_cost: 0.00014, ips: 1.1908 samples/sec | ETA 07:55:51 2021-05-06 13:07:56 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 13:11:32 [INFO] [EVAL] #Images: 500 mIoU: 0.6940 Acc: 0.9459 Kappa: 0.9298 2021-05-06 13:11:32 [INFO] [EVAL] Class IoU: [0.9734 0.7988 0.9014 0.5494 0.5205 0.4471 0.4948 0.6519 0.9053 0.6209 0.9298 0.7364 0.4668 0.9244 0.7571 0.7799 0.588 0.4729 0.6676] 2021-05-06 13:11:32 [INFO] [EVAL] Class Acc: [0.9917 0.8677 0.9328 0.8553 0.6773 0.7113 0.748 0.8695 0.9494 0.771 0.963 0.8296 0.6379 0.9578 0.9183 0.9247 0.683 0.8659 0.7769] 2021-05-06 13:12:22 [INFO] [EVAL] The model with the best validation mIoU (0.6940) was saved at iter 6000. 2021-05-06 13:12:31 [INFO] [TRAIN] epoch: 17, iter: 6010/40000, loss: 0.6470, lr: 0.008651, batch_cost: 0.8371, reader_cost: 0.00024, ips: 1.1946 samples/sec | ETA 07:54:13 2021-05-06 13:12:39 [INFO] [TRAIN] epoch: 17, iter: 6020/40000, loss: 0.8081, lr: 0.008649, batch_cost: 0.8384, reader_cost: 0.00015, ips: 1.1928 samples/sec | ETA 07:54:48 2021-05-06 13:12:47 [INFO] [TRAIN] epoch: 17, iter: 6030/40000, loss: 0.9330, lr: 0.008646, batch_cost: 0.8387, reader_cost: 0.00015, ips: 1.1923 samples/sec | ETA 07:54:52 2021-05-06 13:12:56 [INFO] [TRAIN] epoch: 17, iter: 6040/40000, loss: 0.5182, lr: 0.008644, batch_cost: 0.8402, reader_cost: 0.00014, ips: 1.1902 samples/sec | ETA 07:55:31 2021-05-06 13:13:04 [INFO] [TRAIN] epoch: 17, iter: 6050/40000, loss: 0.4018, lr: 0.008642, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 07:55:25 2021-05-06 13:13:13 [INFO] [TRAIN] epoch: 17, iter: 6060/40000, loss: 0.4686, lr: 0.008640, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 07:55:48 2021-05-06 13:13:21 [INFO] [TRAIN] epoch: 17, iter: 6070/40000, loss: 0.4975, lr: 0.008637, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1904 samples/sec | ETA 07:55:03 2021-05-06 13:13:29 [INFO] [TRAIN] epoch: 17, iter: 6080/40000, loss: 0.6569, lr: 0.008635, batch_cost: 0.8395, reader_cost: 0.00014, ips: 1.1912 samples/sec | ETA 07:54:36 2021-05-06 13:13:38 [INFO] [TRAIN] epoch: 17, iter: 6090/40000, loss: 0.7868, lr: 0.008633, batch_cost: 0.8386, reader_cost: 0.00013, ips: 1.1924 samples/sec | ETA 07:53:57 2021-05-06 13:13:46 [INFO] [TRAIN] epoch: 17, iter: 6100/40000, loss: 0.9480, lr: 0.008630, batch_cost: 0.8407, reader_cost: 0.00014, ips: 1.1895 samples/sec | ETA 07:55:00 2021-05-06 13:13:55 [INFO] [TRAIN] epoch: 17, iter: 6110/40000, loss: 0.4872, lr: 0.008628, batch_cost: 0.8413, reader_cost: 0.00014, ips: 1.1887 samples/sec | ETA 07:55:10 2021-05-06 13:14:03 [INFO] [TRAIN] epoch: 17, iter: 6120/40000, loss: 0.5085, lr: 0.008626, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1904 samples/sec | ETA 07:54:22 2021-05-06 13:14:11 [INFO] [TRAIN] epoch: 17, iter: 6130/40000, loss: 0.7536, lr: 0.008624, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1884 samples/sec | ETA 07:55:00 2021-05-06 13:14:20 [INFO] [TRAIN] epoch: 17, iter: 6140/40000, loss: 0.8683, lr: 0.008621, batch_cost: 0.8402, reader_cost: 0.00014, ips: 1.1902 samples/sec | ETA 07:54:08 2021-05-06 13:14:28 [INFO] [TRAIN] epoch: 17, iter: 6150/40000, loss: 0.6998, lr: 0.008619, batch_cost: 0.8390, reader_cost: 0.00014, ips: 1.1919 samples/sec | ETA 07:53:21 2021-05-06 13:14:37 [INFO] [TRAIN] epoch: 17, iter: 6160/40000, loss: 0.7251, lr: 0.008617, batch_cost: 0.8412, reader_cost: 0.00014, ips: 1.1887 samples/sec | ETA 07:54:26 2021-05-06 13:14:45 [INFO] [TRAIN] epoch: 17, iter: 6170/40000, loss: 0.4474, lr: 0.008615, batch_cost: 0.8408, reader_cost: 0.00014, ips: 1.1894 samples/sec | ETA 07:54:03 2021-05-06 13:14:53 [INFO] [TRAIN] epoch: 17, iter: 6180/40000, loss: 0.3640, lr: 0.008612, batch_cost: 0.8394, reader_cost: 0.00014, ips: 1.1914 samples/sec | ETA 07:53:07 2021-05-06 13:15:02 [INFO] [TRAIN] epoch: 17, iter: 6190/40000, loss: 0.5073, lr: 0.008610, batch_cost: 0.8409, reader_cost: 0.00014, ips: 1.1892 samples/sec | ETA 07:53:51 2021-05-06 13:15:10 [INFO] [TRAIN] epoch: 17, iter: 6200/40000, loss: 0.4100, lr: 0.008608, batch_cost: 0.8411, reader_cost: 0.00014, ips: 1.1890 samples/sec | ETA 07:53:47 2021-05-06 13:15:19 [INFO] [TRAIN] epoch: 17, iter: 6210/40000, loss: 0.4611, lr: 0.008606, batch_cost: 0.8403, reader_cost: 0.00014, ips: 1.1901 samples/sec | ETA 07:53:13 2021-05-06 13:15:27 [INFO] [TRAIN] epoch: 17, iter: 6220/40000, loss: 0.3473, lr: 0.008603, batch_cost: 0.8404, reader_cost: 0.00016, ips: 1.1899 samples/sec | ETA 07:53:08 2021-05-06 13:15:35 [INFO] [TRAIN] epoch: 17, iter: 6230/40000, loss: 0.9961, lr: 0.008601, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 07:52:57 2021-05-06 13:15:44 [INFO] [TRAIN] epoch: 17, iter: 6240/40000, loss: 0.6047, lr: 0.008599, batch_cost: 0.8405, reader_cost: 0.00014, ips: 1.1897 samples/sec | ETA 07:52:56 2021-05-06 13:15:52 [INFO] [TRAIN] epoch: 17, iter: 6250/40000, loss: 0.5178, lr: 0.008596, batch_cost: 0.8402, reader_cost: 0.00014, ips: 1.1902 samples/sec | ETA 07:52:36 2021-05-06 13:16:01 [INFO] [TRAIN] epoch: 17, iter: 6260/40000, loss: 0.5213, lr: 0.008594, batch_cost: 0.8392, reader_cost: 0.00014, ips: 1.1917 samples/sec | ETA 07:51:53 2021-05-06 13:16:09 [INFO] [TRAIN] epoch: 17, iter: 6270/40000, loss: 0.5764, lr: 0.008592, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 07:52:27 2021-05-06 13:16:17 [INFO] [TRAIN] epoch: 17, iter: 6280/40000, loss: 0.6103, lr: 0.008590, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 07:52:03 2021-05-06 13:16:26 [INFO] [TRAIN] epoch: 17, iter: 6290/40000, loss: 0.4305, lr: 0.008587, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1886 samples/sec | ETA 07:52:41 2021-05-06 13:16:34 [INFO] [TRAIN] epoch: 17, iter: 6300/40000, loss: 0.6263, lr: 0.008585, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1881 samples/sec | ETA 07:52:44 2021-05-06 13:16:43 [INFO] [TRAIN] epoch: 17, iter: 6310/40000, loss: 0.8009, lr: 0.008583, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 07:51:35 2021-05-06 13:16:51 [INFO] [TRAIN] epoch: 17, iter: 6320/40000, loss: 1.8368, lr: 0.008581, batch_cost: 0.8369, reader_cost: 0.00010, ips: 1.1948 samples/sec | ETA 07:49:48 2021-05-06 13:17:03 [INFO] [TRAIN] epoch: 18, iter: 6330/40000, loss: 2.2559, lr: 0.008578, batch_cost: 1.1906, reader_cost: 0.21475, ips: 0.8399 samples/sec | ETA 11:08:08 2021-05-06 13:17:11 [INFO] [TRAIN] epoch: 18, iter: 6340/40000, loss: 0.9139, lr: 0.008576, batch_cost: 0.8400, reader_cost: 0.00031, ips: 1.1904 samples/sec | ETA 07:51:15 2021-05-06 13:17:20 [INFO] [TRAIN] epoch: 18, iter: 6350/40000, loss: 0.8468, lr: 0.008574, batch_cost: 0.8425, reader_cost: 0.00016, ips: 1.1869 samples/sec | ETA 07:52:30 2021-05-06 13:17:28 [INFO] [TRAIN] epoch: 18, iter: 6360/40000, loss: 0.5876, lr: 0.008572, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 07:51:28 2021-05-06 13:17:37 [INFO] [TRAIN] epoch: 18, iter: 6370/40000, loss: 0.5514, lr: 0.008569, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1912 samples/sec | ETA 07:50:32 2021-05-06 13:17:45 [INFO] [TRAIN] epoch: 18, iter: 6380/40000, loss: 0.9548, lr: 0.008567, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1885 samples/sec | ETA 07:51:27 2021-05-06 13:17:53 [INFO] [TRAIN] epoch: 18, iter: 6390/40000, loss: 0.7497, lr: 0.008565, batch_cost: 0.8403, reader_cost: 0.00016, ips: 1.1900 samples/sec | ETA 07:50:43 2021-05-06 13:18:02 [INFO] [TRAIN] epoch: 18, iter: 6400/40000, loss: 0.8090, lr: 0.008562, batch_cost: 0.8392, reader_cost: 0.00014, ips: 1.1916 samples/sec | ETA 07:49:56 2021-05-06 13:18:10 [INFO] [TRAIN] epoch: 18, iter: 6410/40000, loss: 0.5980, lr: 0.008560, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1882 samples/sec | ETA 07:51:10 2021-05-06 13:18:19 [INFO] [TRAIN] epoch: 18, iter: 6420/40000, loss: 0.5946, lr: 0.008558, batch_cost: 0.8410, reader_cost: 0.00016, ips: 1.1891 samples/sec | ETA 07:50:40 2021-05-06 13:18:27 [INFO] [TRAIN] epoch: 18, iter: 6430/40000, loss: 0.3972, lr: 0.008556, batch_cost: 0.8430, reader_cost: 0.00015, ips: 1.1862 samples/sec | ETA 07:51:40 2021-05-06 13:18:35 [INFO] [TRAIN] epoch: 18, iter: 6440/40000, loss: 0.5006, lr: 0.008553, batch_cost: 0.8409, reader_cost: 0.00016, ips: 1.1892 samples/sec | ETA 07:50:19 2021-05-06 13:18:44 [INFO] [TRAIN] epoch: 18, iter: 6450/40000, loss: 0.8420, lr: 0.008551, batch_cost: 0.8420, reader_cost: 0.00014, ips: 1.1877 samples/sec | ETA 07:50:48 2021-05-06 13:18:52 [INFO] [TRAIN] epoch: 18, iter: 6460/40000, loss: 0.7286, lr: 0.008549, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 07:49:45 2021-05-06 13:19:01 [INFO] [TRAIN] epoch: 18, iter: 6470/40000, loss: 0.7607, lr: 0.008547, batch_cost: 0.8391, reader_cost: 0.00014, ips: 1.1917 samples/sec | ETA 07:48:55 2021-05-06 13:19:09 [INFO] [TRAIN] epoch: 18, iter: 6480/40000, loss: 0.4480, lr: 0.008544, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 07:49:54 2021-05-06 13:19:18 [INFO] [TRAIN] epoch: 18, iter: 6490/40000, loss: 0.5505, lr: 0.008542, batch_cost: 0.8414, reader_cost: 0.00016, ips: 1.1885 samples/sec | ETA 07:49:54 2021-05-06 13:19:26 [INFO] [TRAIN] epoch: 18, iter: 6500/40000, loss: 0.7080, lr: 0.008540, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 07:49:06 2021-05-06 13:19:34 [INFO] [TRAIN] epoch: 18, iter: 6510/40000, loss: 0.6188, lr: 0.008538, batch_cost: 0.8407, reader_cost: 0.00016, ips: 1.1895 samples/sec | ETA 07:49:14 2021-05-06 13:19:43 [INFO] [TRAIN] epoch: 18, iter: 6520/40000, loss: 0.7654, lr: 0.008535, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 07:48:52 2021-05-06 13:19:51 [INFO] [TRAIN] epoch: 18, iter: 6530/40000, loss: 0.4519, lr: 0.008533, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1917 samples/sec | ETA 07:48:05 2021-05-06 13:20:00 [INFO] [TRAIN] epoch: 18, iter: 6540/40000, loss: 0.4887, lr: 0.008531, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 07:48:42 2021-05-06 13:20:08 [INFO] [TRAIN] epoch: 18, iter: 6550/40000, loss: 0.2290, lr: 0.008528, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1885 samples/sec | ETA 07:49:03 2021-05-06 13:20:16 [INFO] [TRAIN] epoch: 18, iter: 6560/40000, loss: 0.4242, lr: 0.008526, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1879 samples/sec | ETA 07:49:09 2021-05-06 13:20:25 [INFO] [TRAIN] epoch: 18, iter: 6570/40000, loss: 0.4409, lr: 0.008524, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1903 samples/sec | ETA 07:48:04 2021-05-06 13:20:33 [INFO] [TRAIN] epoch: 18, iter: 6580/40000, loss: 0.3492, lr: 0.008522, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 07:47:52 2021-05-06 13:20:42 [INFO] [TRAIN] epoch: 18, iter: 6590/40000, loss: 0.3828, lr: 0.008519, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1882 samples/sec | ETA 07:48:37 2021-05-06 13:20:50 [INFO] [TRAIN] epoch: 18, iter: 6600/40000, loss: 0.2439, lr: 0.008517, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 07:47:49 2021-05-06 13:20:58 [INFO] [TRAIN] epoch: 18, iter: 6610/40000, loss: 0.4450, lr: 0.008515, batch_cost: 0.8406, reader_cost: 0.00016, ips: 1.1897 samples/sec | ETA 07:47:46 2021-05-06 13:21:07 [INFO] [TRAIN] epoch: 18, iter: 6620/40000, loss: 0.3717, lr: 0.008513, batch_cost: 0.8407, reader_cost: 0.00016, ips: 1.1895 samples/sec | ETA 07:47:42 2021-05-06 13:21:15 [INFO] [TRAIN] epoch: 18, iter: 6630/40000, loss: 0.3851, lr: 0.008510, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1882 samples/sec | ETA 07:48:04 2021-05-06 13:21:24 [INFO] [TRAIN] epoch: 18, iter: 6640/40000, loss: 0.5033, lr: 0.008508, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 07:46:56 2021-05-06 13:21:32 [INFO] [TRAIN] epoch: 18, iter: 6650/40000, loss: 0.6789, lr: 0.008506, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1913 samples/sec | ETA 07:46:35 2021-05-06 13:21:40 [INFO] [TRAIN] epoch: 18, iter: 6660/40000, loss: 0.3645, lr: 0.008504, batch_cost: 0.8408, reader_cost: 0.00016, ips: 1.1893 samples/sec | ETA 07:47:12 2021-05-06 13:21:49 [INFO] [TRAIN] epoch: 18, iter: 6670/40000, loss: 0.6405, lr: 0.008501, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 07:46:39 2021-05-06 13:21:57 [INFO] [TRAIN] epoch: 18, iter: 6680/40000, loss: 0.7373, lr: 0.008499, batch_cost: 0.8414, reader_cost: 0.00014, ips: 1.1885 samples/sec | ETA 07:47:14 2021-05-06 13:22:06 [INFO] [TRAIN] epoch: 18, iter: 6690/40000, loss: 0.5992, lr: 0.008497, batch_cost: 0.8403, reader_cost: 0.00012, ips: 1.1901 samples/sec | ETA 07:46:29 2021-05-06 13:22:17 [INFO] [TRAIN] epoch: 19, iter: 6700/40000, loss: 0.4861, lr: 0.008494, batch_cost: 1.1728, reader_cost: 0.19570, ips: 0.8527 samples/sec | ETA 10:50:52 2021-05-06 13:22:26 [INFO] [TRAIN] epoch: 19, iter: 6710/40000, loss: 0.5535, lr: 0.008492, batch_cost: 0.8525, reader_cost: 0.00033, ips: 1.1731 samples/sec | ETA 07:52:58 2021-05-06 13:22:34 [INFO] [TRAIN] epoch: 19, iter: 6720/40000, loss: 0.4652, lr: 0.008490, batch_cost: 0.8428, reader_cost: 0.00015, ips: 1.1865 samples/sec | ETA 07:47:28 2021-05-06 13:22:43 [INFO] [TRAIN] epoch: 19, iter: 6730/40000, loss: 0.3599, lr: 0.008488, batch_cost: 0.8419, reader_cost: 0.00015, ips: 1.1877 samples/sec | ETA 07:46:51 2021-05-06 13:22:51 [INFO] [TRAIN] epoch: 19, iter: 6740/40000, loss: 0.2840, lr: 0.008485, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1879 samples/sec | ETA 07:46:38 2021-05-06 13:23:00 [INFO] [TRAIN] epoch: 19, iter: 6750/40000, loss: 0.7996, lr: 0.008483, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1879 samples/sec | ETA 07:46:31 2021-05-06 13:23:08 [INFO] [TRAIN] epoch: 19, iter: 6760/40000, loss: 0.5601, lr: 0.008481, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 07:45:09 2021-05-06 13:23:16 [INFO] [TRAIN] epoch: 19, iter: 6770/40000, loss: 0.9267, lr: 0.008479, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 07:45:41 2021-05-06 13:23:25 [INFO] [TRAIN] epoch: 19, iter: 6780/40000, loss: 0.5502, lr: 0.008476, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 07:45:15 2021-05-06 13:23:33 [INFO] [TRAIN] epoch: 19, iter: 6790/40000, loss: 0.5389, lr: 0.008474, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 07:45:29 2021-05-06 13:23:42 [INFO] [TRAIN] epoch: 19, iter: 6800/40000, loss: 0.3849, lr: 0.008472, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 07:45:11 2021-05-06 13:23:50 [INFO] [TRAIN] epoch: 19, iter: 6810/40000, loss: 0.4255, lr: 0.008469, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 07:44:54 2021-05-06 13:23:58 [INFO] [TRAIN] epoch: 19, iter: 6820/40000, loss: 0.6371, lr: 0.008467, batch_cost: 0.8421, reader_cost: 0.00016, ips: 1.1875 samples/sec | ETA 07:45:40 2021-05-06 13:24:07 [INFO] [TRAIN] epoch: 19, iter: 6830/40000, loss: 0.7652, lr: 0.008465, batch_cost: 0.8396, reader_cost: 0.00017, ips: 1.1911 samples/sec | ETA 07:44:09 2021-05-06 13:24:15 [INFO] [TRAIN] epoch: 19, iter: 6840/40000, loss: 1.5242, lr: 0.008463, batch_cost: 0.8393, reader_cost: 0.00014, ips: 1.1915 samples/sec | ETA 07:43:50 2021-05-06 13:24:24 [INFO] [TRAIN] epoch: 19, iter: 6850/40000, loss: 0.4706, lr: 0.008460, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 07:43:59 2021-05-06 13:24:32 [INFO] [TRAIN] epoch: 19, iter: 6860/40000, loss: 0.5381, lr: 0.008458, batch_cost: 0.8410, reader_cost: 0.00014, ips: 1.1891 samples/sec | ETA 07:44:30 2021-05-06 13:24:40 [INFO] [TRAIN] epoch: 19, iter: 6870/40000, loss: 0.6402, lr: 0.008456, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 07:44:08 2021-05-06 13:24:49 [INFO] [TRAIN] epoch: 19, iter: 6880/40000, loss: 0.4535, lr: 0.008454, batch_cost: 0.8422, reader_cost: 0.00016, ips: 1.1874 samples/sec | ETA 07:44:52 2021-05-06 13:24:57 [INFO] [TRAIN] epoch: 19, iter: 6890/40000, loss: 0.8019, lr: 0.008451, batch_cost: 0.8405, reader_cost: 0.00017, ips: 1.1898 samples/sec | ETA 07:43:49 2021-05-06 13:25:06 [INFO] [TRAIN] epoch: 19, iter: 6900/40000, loss: 0.7127, lr: 0.008449, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 07:43:57 2021-05-06 13:25:14 [INFO] [TRAIN] epoch: 19, iter: 6910/40000, loss: 0.4963, lr: 0.008447, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1883 samples/sec | ETA 07:44:07 2021-05-06 13:25:23 [INFO] [TRAIN] epoch: 19, iter: 6920/40000, loss: 0.5283, lr: 0.008445, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 07:43:27 2021-05-06 13:25:31 [INFO] [TRAIN] epoch: 19, iter: 6930/40000, loss: 0.2581, lr: 0.008442, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1885 samples/sec | ETA 07:43:45 2021-05-06 13:25:39 [INFO] [TRAIN] epoch: 19, iter: 6940/40000, loss: 0.5149, lr: 0.008440, batch_cost: 0.8414, reader_cost: 0.00016, ips: 1.1884 samples/sec | ETA 07:43:37 2021-05-06 13:25:48 [INFO] [TRAIN] epoch: 19, iter: 6950/40000, loss: 0.4654, lr: 0.008438, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1882 samples/sec | ETA 07:43:35 2021-05-06 13:25:56 [INFO] [TRAIN] epoch: 19, iter: 6960/40000, loss: 0.6213, lr: 0.008435, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 07:43:00 2021-05-06 13:26:05 [INFO] [TRAIN] epoch: 19, iter: 6970/40000, loss: 0.4808, lr: 0.008433, batch_cost: 0.8430, reader_cost: 0.00016, ips: 1.1862 samples/sec | ETA 07:44:04 2021-05-06 13:26:13 [INFO] [TRAIN] epoch: 19, iter: 6980/40000, loss: 0.4907, lr: 0.008431, batch_cost: 0.8433, reader_cost: 0.00015, ips: 1.1858 samples/sec | ETA 07:44:05 2021-05-06 13:26:21 [INFO] [TRAIN] epoch: 19, iter: 6990/40000, loss: 0.4767, lr: 0.008429, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1879 samples/sec | ETA 07:43:09 2021-05-06 13:26:30 [INFO] [TRAIN] epoch: 19, iter: 7000/40000, loss: 0.5069, lr: 0.008426, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1912 samples/sec | ETA 07:41:42 2021-05-06 13:26:30 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 13:30:06 [INFO] [EVAL] #Images: 500 mIoU: 0.6961 Acc: 0.9439 Kappa: 0.9272 2021-05-06 13:30:06 [INFO] [EVAL] Class IoU: [0.9687 0.7685 0.9017 0.5493 0.5364 0.448 0.5158 0.6545 0.9029 0.6076 0.9327 0.7401 0.4706 0.92 0.7436 0.781 0.6194 0.4978 0.6669] 2021-05-06 13:30:06 [INFO] [EVAL] Class Acc: [0.9821 0.874 0.9469 0.7865 0.7186 0.7114 0.7094 0.7731 0.9403 0.8013 0.9552 0.8396 0.6895 0.9582 0.8425 0.8425 0.8048 0.6878 0.7951] 2021-05-06 13:30:56 [INFO] [EVAL] The model with the best validation mIoU (0.6961) was saved at iter 7000. 2021-05-06 13:31:04 [INFO] [TRAIN] epoch: 19, iter: 7010/40000, loss: 0.4385, lr: 0.008424, batch_cost: 0.8369, reader_cost: 0.00044, ips: 1.1948 samples/sec | ETA 07:40:10 2021-05-06 13:31:13 [INFO] [TRAIN] epoch: 19, iter: 7020/40000, loss: 0.5712, lr: 0.008422, batch_cost: 0.8372, reader_cost: 0.00015, ips: 1.1945 samples/sec | ETA 07:40:09 2021-05-06 13:31:21 [INFO] [TRAIN] epoch: 19, iter: 7030/40000, loss: 0.3954, lr: 0.008420, batch_cost: 0.8389, reader_cost: 0.00015, ips: 1.1920 samples/sec | ETA 07:40:59 2021-05-06 13:31:30 [INFO] [TRAIN] epoch: 19, iter: 7040/40000, loss: 0.3782, lr: 0.008417, batch_cost: 0.8412, reader_cost: 0.00014, ips: 1.1888 samples/sec | ETA 07:42:06 2021-05-06 13:31:38 [INFO] [TRAIN] epoch: 19, iter: 7050/40000, loss: 0.4797, lr: 0.008415, batch_cost: 0.8403, reader_cost: 0.00016, ips: 1.1901 samples/sec | ETA 07:41:27 2021-05-06 13:31:46 [INFO] [TRAIN] epoch: 19, iter: 7060/40000, loss: 0.6663, lr: 0.008413, batch_cost: 0.8409, reader_cost: 0.00013, ips: 1.1892 samples/sec | ETA 07:41:39 2021-05-06 13:31:58 [INFO] [TRAIN] epoch: 20, iter: 7070/40000, loss: 0.5263, lr: 0.008410, batch_cost: 1.1622, reader_cost: 0.21243, ips: 0.8604 samples/sec | ETA 10:37:51 2021-05-06 13:32:07 [INFO] [TRAIN] epoch: 20, iter: 7080/40000, loss: 0.6228, lr: 0.008408, batch_cost: 0.8573, reader_cost: 0.00032, ips: 1.1665 samples/sec | ETA 07:50:22 2021-05-06 13:32:15 [INFO] [TRAIN] epoch: 20, iter: 7090/40000, loss: 0.7686, lr: 0.008406, batch_cost: 0.8411, reader_cost: 0.00014, ips: 1.1889 samples/sec | ETA 07:41:20 2021-05-06 13:32:23 [INFO] [TRAIN] epoch: 20, iter: 7100/40000, loss: 0.4434, lr: 0.008404, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 07:40:51 2021-05-06 13:32:32 [INFO] [TRAIN] epoch: 20, iter: 7110/40000, loss: 0.3366, lr: 0.008401, batch_cost: 0.8425, reader_cost: 0.00015, ips: 1.1870 samples/sec | ETA 07:41:49 2021-05-06 13:32:40 [INFO] [TRAIN] epoch: 20, iter: 7120/40000, loss: 0.4527, lr: 0.008399, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1884 samples/sec | ETA 07:41:06 2021-05-06 13:32:49 [INFO] [TRAIN] epoch: 20, iter: 7130/40000, loss: 0.6521, lr: 0.008397, batch_cost: 0.8431, reader_cost: 0.00015, ips: 1.1861 samples/sec | ETA 07:41:51 2021-05-06 13:32:57 [INFO] [TRAIN] epoch: 20, iter: 7140/40000, loss: 0.6144, lr: 0.008395, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 07:40:02 2021-05-06 13:33:06 [INFO] [TRAIN] epoch: 20, iter: 7150/40000, loss: 0.6151, lr: 0.008392, batch_cost: 0.8419, reader_cost: 0.00015, ips: 1.1878 samples/sec | ETA 07:40:56 2021-05-06 13:33:14 [INFO] [TRAIN] epoch: 20, iter: 7160/40000, loss: 0.4037, lr: 0.008390, batch_cost: 0.8430, reader_cost: 0.00016, ips: 1.1862 samples/sec | ETA 07:41:25 2021-05-06 13:33:22 [INFO] [TRAIN] epoch: 20, iter: 7170/40000, loss: 0.6042, lr: 0.008388, batch_cost: 0.8426, reader_cost: 0.00015, ips: 1.1868 samples/sec | ETA 07:41:02 2021-05-06 13:33:31 [INFO] [TRAIN] epoch: 20, iter: 7180/40000, loss: 0.3609, lr: 0.008385, batch_cost: 0.8400, reader_cost: 0.00014, ips: 1.1904 samples/sec | ETA 07:39:29 2021-05-06 13:33:39 [INFO] [TRAIN] epoch: 20, iter: 7190/40000, loss: 0.6151, lr: 0.008383, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1882 samples/sec | ETA 07:40:12 2021-05-06 13:33:48 [INFO] [TRAIN] epoch: 20, iter: 7200/40000, loss: 0.6567, lr: 0.008381, batch_cost: 0.8409, reader_cost: 0.00014, ips: 1.1892 samples/sec | ETA 07:39:41 2021-05-06 13:33:56 [INFO] [TRAIN] epoch: 20, iter: 7210/40000, loss: 0.6681, lr: 0.008379, batch_cost: 0.8398, reader_cost: 0.00014, ips: 1.1907 samples/sec | ETA 07:38:58 2021-05-06 13:34:04 [INFO] [TRAIN] epoch: 20, iter: 7220/40000, loss: 0.4094, lr: 0.008376, batch_cost: 0.8407, reader_cost: 0.00016, ips: 1.1895 samples/sec | ETA 07:39:17 2021-05-06 13:34:13 [INFO] [TRAIN] epoch: 20, iter: 7230/40000, loss: 0.4808, lr: 0.008374, batch_cost: 0.8404, reader_cost: 0.00017, ips: 1.1899 samples/sec | ETA 07:38:59 2021-05-06 13:34:21 [INFO] [TRAIN] epoch: 20, iter: 7240/40000, loss: 0.5808, lr: 0.008372, batch_cost: 0.8404, reader_cost: 0.00017, ips: 1.1898 samples/sec | ETA 07:38:52 2021-05-06 13:34:30 [INFO] [TRAIN] epoch: 20, iter: 7250/40000, loss: 0.9889, lr: 0.008370, batch_cost: 0.8383, reader_cost: 0.00017, ips: 1.1929 samples/sec | ETA 07:37:34 2021-05-06 13:34:38 [INFO] [TRAIN] epoch: 20, iter: 7260/40000, loss: 1.1249, lr: 0.008367, batch_cost: 0.8408, reader_cost: 0.00017, ips: 1.1894 samples/sec | ETA 07:38:46 2021-05-06 13:34:46 [INFO] [TRAIN] epoch: 20, iter: 7270/40000, loss: 0.8905, lr: 0.008365, batch_cost: 0.8400, reader_cost: 0.00014, ips: 1.1905 samples/sec | ETA 07:38:11 2021-05-06 13:34:55 [INFO] [TRAIN] epoch: 20, iter: 7280/40000, loss: 0.6424, lr: 0.008363, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 07:38:24 2021-05-06 13:35:03 [INFO] [TRAIN] epoch: 20, iter: 7290/40000, loss: 0.6504, lr: 0.008360, batch_cost: 0.8407, reader_cost: 0.00016, ips: 1.1895 samples/sec | ETA 07:38:19 2021-05-06 13:35:12 [INFO] [TRAIN] epoch: 20, iter: 7300/40000, loss: 0.4532, lr: 0.008358, batch_cost: 0.8412, reader_cost: 0.00016, ips: 1.1887 samples/sec | ETA 07:38:28 2021-05-06 13:35:20 [INFO] [TRAIN] epoch: 20, iter: 7310/40000, loss: 0.5130, lr: 0.008356, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1884 samples/sec | ETA 07:38:28 2021-05-06 13:35:29 [INFO] [TRAIN] epoch: 20, iter: 7320/40000, loss: 0.2691, lr: 0.008354, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 07:37:45 2021-05-06 13:35:37 [INFO] [TRAIN] epoch: 20, iter: 7330/40000, loss: 0.5329, lr: 0.008351, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 07:37:50 2021-05-06 13:35:45 [INFO] [TRAIN] epoch: 20, iter: 7340/40000, loss: 0.3817, lr: 0.008349, batch_cost: 0.8423, reader_cost: 0.00015, ips: 1.1872 samples/sec | ETA 07:38:30 2021-05-06 13:35:54 [INFO] [TRAIN] epoch: 20, iter: 7350/40000, loss: 0.4691, lr: 0.008347, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 07:37:35 2021-05-06 13:36:02 [INFO] [TRAIN] epoch: 20, iter: 7360/40000, loss: 0.5886, lr: 0.008345, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 07:37:34 2021-05-06 13:36:11 [INFO] [TRAIN] epoch: 20, iter: 7370/40000, loss: 0.4911, lr: 0.008342, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 07:37:06 2021-05-06 13:36:19 [INFO] [TRAIN] epoch: 20, iter: 7380/40000, loss: 0.4079, lr: 0.008340, batch_cost: 0.8400, reader_cost: 0.00016, ips: 1.1904 samples/sec | ETA 07:36:42 2021-05-06 13:36:27 [INFO] [TRAIN] epoch: 20, iter: 7390/40000, loss: 0.6889, lr: 0.008338, batch_cost: 0.8405, reader_cost: 0.00016, ips: 1.1897 samples/sec | ETA 07:36:49 2021-05-06 13:36:36 [INFO] [TRAIN] epoch: 20, iter: 7400/40000, loss: 0.3663, lr: 0.008335, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 07:36:28 2021-05-06 13:36:44 [INFO] [TRAIN] epoch: 20, iter: 7410/40000, loss: 0.4509, lr: 0.008333, batch_cost: 0.8394, reader_cost: 0.00014, ips: 1.1913 samples/sec | ETA 07:35:57 2021-05-06 13:36:53 [INFO] [TRAIN] epoch: 20, iter: 7420/40000, loss: 0.5071, lr: 0.008331, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1914 samples/sec | ETA 07:35:45 2021-05-06 13:37:01 [INFO] [TRAIN] epoch: 20, iter: 7430/40000, loss: 0.6814, lr: 0.008329, batch_cost: 0.8394, reader_cost: 0.00014, ips: 1.1913 samples/sec | ETA 07:35:38 2021-05-06 13:37:09 [INFO] [TRAIN] epoch: 20, iter: 7440/40000, loss: 0.8057, lr: 0.008326, batch_cost: 0.8386, reader_cost: 0.00010, ips: 1.1925 samples/sec | ETA 07:35:04 2021-05-06 13:37:21 [INFO] [TRAIN] epoch: 21, iter: 7450/40000, loss: 0.6355, lr: 0.008324, batch_cost: 1.1488, reader_cost: 0.21298, ips: 0.8705 samples/sec | ETA 10:23:11 2021-05-06 13:37:29 [INFO] [TRAIN] epoch: 21, iter: 7460/40000, loss: 0.4490, lr: 0.008322, batch_cost: 0.8409, reader_cost: 0.00030, ips: 1.1893 samples/sec | ETA 07:36:01 2021-05-06 13:37:38 [INFO] [TRAIN] epoch: 21, iter: 7470/40000, loss: 0.5467, lr: 0.008320, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1884 samples/sec | ETA 07:36:12 2021-05-06 13:37:46 [INFO] [TRAIN] epoch: 21, iter: 7480/40000, loss: 0.2430, lr: 0.008317, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 07:35:53 2021-05-06 13:37:54 [INFO] [TRAIN] epoch: 21, iter: 7490/40000, loss: 0.6769, lr: 0.008315, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 07:34:59 2021-05-06 13:38:03 [INFO] [TRAIN] epoch: 21, iter: 7500/40000, loss: 0.8104, lr: 0.008313, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1917 samples/sec | ETA 07:34:32 2021-05-06 13:38:11 [INFO] [TRAIN] epoch: 21, iter: 7510/40000, loss: 0.6622, lr: 0.008310, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 07:35:27 2021-05-06 13:38:20 [INFO] [TRAIN] epoch: 21, iter: 7520/40000, loss: 1.0250, lr: 0.008308, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 07:34:35 2021-05-06 13:38:28 [INFO] [TRAIN] epoch: 21, iter: 7530/40000, loss: 0.5585, lr: 0.008306, batch_cost: 0.8400, reader_cost: 0.00016, ips: 1.1905 samples/sec | ETA 07:34:34 2021-05-06 13:38:36 [INFO] [TRAIN] epoch: 21, iter: 7540/40000, loss: 0.2696, lr: 0.008304, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 07:34:25 2021-05-06 13:38:45 [INFO] [TRAIN] epoch: 21, iter: 7550/40000, loss: 0.4452, lr: 0.008301, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 07:34:38 2021-05-06 13:38:53 [INFO] [TRAIN] epoch: 21, iter: 7560/40000, loss: 0.4509, lr: 0.008299, batch_cost: 0.8405, reader_cost: 0.00017, ips: 1.1897 samples/sec | ETA 07:34:27 2021-05-06 13:39:02 [INFO] [TRAIN] epoch: 21, iter: 7570/40000, loss: 0.5052, lr: 0.008297, batch_cost: 0.8389, reader_cost: 0.00015, ips: 1.1920 samples/sec | ETA 07:33:26 2021-05-06 13:39:10 [INFO] [TRAIN] epoch: 21, iter: 7580/40000, loss: 0.7579, lr: 0.008295, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1913 samples/sec | ETA 07:33:35 2021-05-06 13:39:18 [INFO] [TRAIN] epoch: 21, iter: 7590/40000, loss: 0.6743, lr: 0.008292, batch_cost: 0.8408, reader_cost: 0.00016, ips: 1.1894 samples/sec | ETA 07:34:09 2021-05-06 13:39:27 [INFO] [TRAIN] epoch: 21, iter: 7600/40000, loss: 0.4546, lr: 0.008290, batch_cost: 0.8391, reader_cost: 0.00016, ips: 1.1917 samples/sec | ETA 07:33:07 2021-05-06 13:39:35 [INFO] [TRAIN] epoch: 21, iter: 7610/40000, loss: 0.6537, lr: 0.008288, batch_cost: 0.8399, reader_cost: 0.00016, ips: 1.1906 samples/sec | ETA 07:33:25 2021-05-06 13:39:44 [INFO] [TRAIN] epoch: 21, iter: 7620/40000, loss: 0.8445, lr: 0.008285, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1903 samples/sec | ETA 07:33:23 2021-05-06 13:39:52 [INFO] [TRAIN] epoch: 21, iter: 7630/40000, loss: 1.4369, lr: 0.008283, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 07:33:10 2021-05-06 13:40:00 [INFO] [TRAIN] epoch: 21, iter: 7640/40000, loss: 0.6586, lr: 0.008281, batch_cost: 0.8394, reader_cost: 0.00014, ips: 1.1913 samples/sec | ETA 07:32:44 2021-05-06 13:40:09 [INFO] [TRAIN] epoch: 21, iter: 7650/40000, loss: 0.5410, lr: 0.008279, batch_cost: 0.8396, reader_cost: 0.00016, ips: 1.1910 samples/sec | ETA 07:32:42 2021-05-06 13:40:17 [INFO] [TRAIN] epoch: 21, iter: 7660/40000, loss: 0.8041, lr: 0.008276, batch_cost: 0.8409, reader_cost: 0.00014, ips: 1.1892 samples/sec | ETA 07:33:13 2021-05-06 13:40:26 [INFO] [TRAIN] epoch: 21, iter: 7670/40000, loss: 0.7853, lr: 0.008274, batch_cost: 0.8401, reader_cost: 0.00016, ips: 1.1904 samples/sec | ETA 07:32:39 2021-05-06 13:40:34 [INFO] [TRAIN] epoch: 21, iter: 7680/40000, loss: 0.5074, lr: 0.008272, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 07:32:57 2021-05-06 13:40:42 [INFO] [TRAIN] epoch: 21, iter: 7690/40000, loss: 0.4301, lr: 0.008270, batch_cost: 0.8407, reader_cost: 0.00016, ips: 1.1895 samples/sec | ETA 07:32:42 2021-05-06 13:40:51 [INFO] [TRAIN] epoch: 21, iter: 7700/40000, loss: 0.4965, lr: 0.008267, batch_cost: 0.8407, reader_cost: 0.00014, ips: 1.1895 samples/sec | ETA 07:32:34 2021-05-06 13:40:59 [INFO] [TRAIN] epoch: 21, iter: 7710/40000, loss: 0.4057, lr: 0.008265, batch_cost: 0.8388, reader_cost: 0.00015, ips: 1.1921 samples/sec | ETA 07:31:25 2021-05-06 13:41:08 [INFO] [TRAIN] epoch: 21, iter: 7720/40000, loss: 0.4103, lr: 0.008263, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1912 samples/sec | ETA 07:31:39 2021-05-06 13:41:16 [INFO] [TRAIN] epoch: 21, iter: 7730/40000, loss: 0.5492, lr: 0.008260, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1913 samples/sec | ETA 07:31:27 2021-05-06 13:41:24 [INFO] [TRAIN] epoch: 21, iter: 7740/40000, loss: 0.5568, lr: 0.008258, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 07:32:02 2021-05-06 13:41:33 [INFO] [TRAIN] epoch: 21, iter: 7750/40000, loss: 0.4946, lr: 0.008256, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 07:31:37 2021-05-06 13:41:41 [INFO] [TRAIN] epoch: 21, iter: 7760/40000, loss: 0.6262, lr: 0.008254, batch_cost: 0.8416, reader_cost: 0.00014, ips: 1.1883 samples/sec | ETA 07:32:11 2021-05-06 13:41:50 [INFO] [TRAIN] epoch: 21, iter: 7770/40000, loss: 0.4929, lr: 0.008251, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 07:31:41 2021-05-06 13:41:58 [INFO] [TRAIN] epoch: 21, iter: 7780/40000, loss: 0.4744, lr: 0.008249, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1914 samples/sec | ETA 07:30:44 2021-05-06 13:42:07 [INFO] [TRAIN] epoch: 21, iter: 7790/40000, loss: 0.7083, lr: 0.008247, batch_cost: 0.8387, reader_cost: 0.00016, ips: 1.1923 samples/sec | ETA 07:30:16 2021-05-06 13:42:15 [INFO] [TRAIN] epoch: 21, iter: 7800/40000, loss: 0.5737, lr: 0.008244, batch_cost: 0.8398, reader_cost: 0.00016, ips: 1.1907 samples/sec | ETA 07:30:42 2021-05-06 13:42:23 [INFO] [TRAIN] epoch: 21, iter: 7810/40000, loss: 0.7599, lr: 0.008242, batch_cost: 0.8383, reader_cost: 0.00025, ips: 1.1929 samples/sec | ETA 07:29:45 2021-05-06 13:42:35 [INFO] [TRAIN] epoch: 22, iter: 7820/40000, loss: 0.6564, lr: 0.008240, batch_cost: 1.1879, reader_cost: 0.20913, ips: 0.8418 samples/sec | ETA 10:37:06 2021-05-06 13:42:44 [INFO] [TRAIN] epoch: 22, iter: 7830/40000, loss: 0.5210, lr: 0.008238, batch_cost: 0.8430, reader_cost: 0.00031, ips: 1.1862 samples/sec | ETA 07:32:00 2021-05-06 13:42:52 [INFO] [TRAIN] epoch: 22, iter: 7840/40000, loss: 0.5885, lr: 0.008235, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 07:30:31 2021-05-06 13:43:00 [INFO] [TRAIN] epoch: 22, iter: 7850/40000, loss: 0.3876, lr: 0.008233, batch_cost: 0.8419, reader_cost: 0.00015, ips: 1.1877 samples/sec | ETA 07:31:08 2021-05-06 13:43:09 [INFO] [TRAIN] epoch: 22, iter: 7860/40000, loss: 0.7190, lr: 0.008231, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 07:30:09 2021-05-06 13:43:17 [INFO] [TRAIN] epoch: 22, iter: 7870/40000, loss: 0.6916, lr: 0.008229, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 07:30:25 2021-05-06 13:43:26 [INFO] [TRAIN] epoch: 22, iter: 7880/40000, loss: 0.5232, lr: 0.008226, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 07:29:39 2021-05-06 13:43:34 [INFO] [TRAIN] epoch: 22, iter: 7890/40000, loss: 0.9111, lr: 0.008224, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 07:29:43 2021-05-06 13:43:42 [INFO] [TRAIN] epoch: 22, iter: 7900/40000, loss: 0.5213, lr: 0.008222, batch_cost: 0.8419, reader_cost: 0.00016, ips: 1.1878 samples/sec | ETA 07:30:24 2021-05-06 13:43:51 [INFO] [TRAIN] epoch: 22, iter: 7910/40000, loss: 0.2764, lr: 0.008219, batch_cost: 0.8412, reader_cost: 0.00017, ips: 1.1887 samples/sec | ETA 07:29:54 2021-05-06 13:43:59 [INFO] [TRAIN] epoch: 22, iter: 7920/40000, loss: 0.2814, lr: 0.008217, batch_cost: 0.8400, reader_cost: 0.00016, ips: 1.1905 samples/sec | ETA 07:29:05 2021-05-06 13:44:08 [INFO] [TRAIN] epoch: 22, iter: 7930/40000, loss: 0.4498, lr: 0.008215, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 07:29:22 2021-05-06 13:44:16 [INFO] [TRAIN] epoch: 22, iter: 7940/40000, loss: 0.7779, lr: 0.008213, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1918 samples/sec | ETA 07:28:21 2021-05-06 13:44:24 [INFO] [TRAIN] epoch: 22, iter: 7950/40000, loss: 0.6993, lr: 0.008210, batch_cost: 0.8389, reader_cost: 0.00015, ips: 1.1921 samples/sec | ETA 07:28:06 2021-05-06 13:44:33 [INFO] [TRAIN] epoch: 22, iter: 7960/40000, loss: 0.4295, lr: 0.008208, batch_cost: 0.8400, reader_cost: 0.00016, ips: 1.1905 samples/sec | ETA 07:28:33 2021-05-06 13:44:41 [INFO] [TRAIN] epoch: 22, iter: 7970/40000, loss: 0.2948, lr: 0.008206, batch_cost: 0.8387, reader_cost: 0.00016, ips: 1.1923 samples/sec | ETA 07:27:45 2021-05-06 13:44:50 [INFO] [TRAIN] epoch: 22, iter: 7980/40000, loss: 0.4878, lr: 0.008203, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 07:28:34 2021-05-06 13:44:58 [INFO] [TRAIN] epoch: 22, iter: 7990/40000, loss: 0.5354, lr: 0.008201, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 07:28:20 2021-05-06 13:45:06 [INFO] [TRAIN] epoch: 22, iter: 8000/40000, loss: 1.0027, lr: 0.008199, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 07:27:53 2021-05-06 13:45:06 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 13:48:42 [INFO] [EVAL] #Images: 500 mIoU: 0.6623 Acc: 0.9467 Kappa: 0.9308 2021-05-06 13:48:42 [INFO] [EVAL] Class IoU: [0.9749 0.8058 0.9044 0.5478 0.5179 0.4705 0.5111 0.6743 0.9048 0.6011 0.9324 0.7357 0.4347 0.9264 0.7666 0.6776 0.0022 0.5174 0.6772] 2021-05-06 13:48:42 [INFO] [EVAL] Class Acc: [0.9865 0.9041 0.9467 0.7893 0.6933 0.7102 0.7223 0.8549 0.9345 0.8606 0.9709 0.8073 0.7453 0.9591 0.8946 0.7158 0.7024 0.7073 0.7992] 2021-05-06 13:49:11 [INFO] [EVAL] The model with the best validation mIoU (0.6961) was saved at iter 7000. 2021-05-06 13:49:19 [INFO] [TRAIN] epoch: 22, iter: 8010/40000, loss: 0.5674, lr: 0.008197, batch_cost: 0.8385, reader_cost: 0.00040, ips: 1.1927 samples/sec | ETA 07:27:02 2021-05-06 13:49:27 [INFO] [TRAIN] epoch: 22, iter: 8020/40000, loss: 0.6482, lr: 0.008194, batch_cost: 0.8389, reader_cost: 0.00016, ips: 1.1920 samples/sec | ETA 07:27:08 2021-05-06 13:49:36 [INFO] [TRAIN] epoch: 22, iter: 8030/40000, loss: 0.6024, lr: 0.008192, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 07:27:39 2021-05-06 13:49:44 [INFO] [TRAIN] epoch: 22, iter: 8040/40000, loss: 0.2714, lr: 0.008190, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 07:28:00 2021-05-06 13:49:53 [INFO] [TRAIN] epoch: 22, iter: 8050/40000, loss: 0.4243, lr: 0.008188, batch_cost: 0.8407, reader_cost: 0.00014, ips: 1.1895 samples/sec | ETA 07:27:40 2021-05-06 13:50:01 [INFO] [TRAIN] epoch: 22, iter: 8060/40000, loss: 0.4170, lr: 0.008185, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 07:27:44 2021-05-06 13:50:09 [INFO] [TRAIN] epoch: 22, iter: 8070/40000, loss: 0.3551, lr: 0.008183, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1883 samples/sec | ETA 07:27:51 2021-05-06 13:50:18 [INFO] [TRAIN] epoch: 22, iter: 8080/40000, loss: 0.7088, lr: 0.008181, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 07:27:16 2021-05-06 13:50:26 [INFO] [TRAIN] epoch: 22, iter: 8090/40000, loss: 0.3794, lr: 0.008178, batch_cost: 0.8420, reader_cost: 0.00015, ips: 1.1876 samples/sec | ETA 07:27:49 2021-05-06 13:50:35 [INFO] [TRAIN] epoch: 22, iter: 8100/40000, loss: 0.5157, lr: 0.008176, batch_cost: 0.8405, reader_cost: 0.00017, ips: 1.1898 samples/sec | ETA 07:26:50 2021-05-06 13:50:43 [INFO] [TRAIN] epoch: 22, iter: 8110/40000, loss: 0.6832, lr: 0.008174, batch_cost: 0.8414, reader_cost: 0.00016, ips: 1.1886 samples/sec | ETA 07:27:10 2021-05-06 13:50:51 [INFO] [TRAIN] epoch: 22, iter: 8120/40000, loss: 0.4443, lr: 0.008172, batch_cost: 0.8423, reader_cost: 0.00015, ips: 1.1872 samples/sec | ETA 07:27:33 2021-05-06 13:51:00 [INFO] [TRAIN] epoch: 22, iter: 8130/40000, loss: 0.5299, lr: 0.008169, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1914 samples/sec | ETA 07:25:50 2021-05-06 13:51:08 [INFO] [TRAIN] epoch: 22, iter: 8140/40000, loss: 0.7364, lr: 0.008167, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 07:25:50 2021-05-06 13:51:17 [INFO] [TRAIN] epoch: 22, iter: 8150/40000, loss: 0.4984, lr: 0.008165, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 07:26:18 2021-05-06 13:51:25 [INFO] [TRAIN] epoch: 22, iter: 8160/40000, loss: 0.5828, lr: 0.008162, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 07:26:04 2021-05-06 13:51:33 [INFO] [TRAIN] epoch: 22, iter: 8170/40000, loss: 0.9538, lr: 0.008160, batch_cost: 0.8393, reader_cost: 0.00014, ips: 1.1915 samples/sec | ETA 07:25:14 2021-05-06 13:51:42 [INFO] [TRAIN] epoch: 22, iter: 8180/40000, loss: 0.7742, lr: 0.008158, batch_cost: 0.8372, reader_cost: 0.00011, ips: 1.1945 samples/sec | ETA 07:23:59 2021-05-06 13:51:54 [INFO] [TRAIN] epoch: 23, iter: 8190/40000, loss: 0.5558, lr: 0.008156, batch_cost: 1.1769, reader_cost: 0.20060, ips: 0.8497 samples/sec | ETA 10:23:57 2021-05-06 13:52:02 [INFO] [TRAIN] epoch: 23, iter: 8200/40000, loss: 0.5135, lr: 0.008153, batch_cost: 0.8400, reader_cost: 0.00031, ips: 1.1905 samples/sec | ETA 07:25:11 2021-05-06 13:52:10 [INFO] [TRAIN] epoch: 23, iter: 8210/40000, loss: 0.5615, lr: 0.008151, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1913 samples/sec | ETA 07:24:44 2021-05-06 13:52:19 [INFO] [TRAIN] epoch: 23, iter: 8220/40000, loss: 0.3666, lr: 0.008149, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 07:25:12 2021-05-06 13:52:27 [INFO] [TRAIN] epoch: 23, iter: 8230/40000, loss: 0.4582, lr: 0.008147, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 07:25:13 2021-05-06 13:52:36 [INFO] [TRAIN] epoch: 23, iter: 8240/40000, loss: 0.9152, lr: 0.008144, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1879 samples/sec | ETA 07:25:36 2021-05-06 13:52:44 [INFO] [TRAIN] epoch: 23, iter: 8250/40000, loss: 0.5356, lr: 0.008142, batch_cost: 0.8403, reader_cost: 0.00016, ips: 1.1900 samples/sec | ETA 07:24:40 2021-05-06 13:52:52 [INFO] [TRAIN] epoch: 23, iter: 8260/40000, loss: 0.6662, lr: 0.008140, batch_cost: 0.8401, reader_cost: 0.00016, ips: 1.1903 samples/sec | ETA 07:24:24 2021-05-06 13:53:01 [INFO] [TRAIN] epoch: 23, iter: 8270/40000, loss: 0.5649, lr: 0.008137, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1913 samples/sec | ETA 07:23:53 2021-05-06 13:53:09 [INFO] [TRAIN] epoch: 23, iter: 8280/40000, loss: 0.2297, lr: 0.008135, batch_cost: 0.8418, reader_cost: 0.00016, ips: 1.1880 samples/sec | ETA 07:25:01 2021-05-06 13:53:18 [INFO] [TRAIN] epoch: 23, iter: 8290/40000, loss: 0.4861, lr: 0.008133, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 07:24:22 2021-05-06 13:53:26 [INFO] [TRAIN] epoch: 23, iter: 8300/40000, loss: 0.4929, lr: 0.008131, batch_cost: 0.8425, reader_cost: 0.00016, ips: 1.1870 samples/sec | ETA 07:25:06 2021-05-06 13:53:35 [INFO] [TRAIN] epoch: 23, iter: 8310/40000, loss: 0.5703, lr: 0.008128, batch_cost: 0.8423, reader_cost: 0.00015, ips: 1.1873 samples/sec | ETA 07:24:51 2021-05-06 13:53:43 [INFO] [TRAIN] epoch: 23, iter: 8320/40000, loss: 0.8038, lr: 0.008126, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 07:23:35 2021-05-06 13:53:51 [INFO] [TRAIN] epoch: 23, iter: 8330/40000, loss: 0.7413, lr: 0.008124, batch_cost: 0.8392, reader_cost: 0.00015, ips: 1.1916 samples/sec | ETA 07:22:58 2021-05-06 13:54:00 [INFO] [TRAIN] epoch: 23, iter: 8340/40000, loss: 0.2436, lr: 0.008121, batch_cost: 0.8390, reader_cost: 0.00015, ips: 1.1919 samples/sec | ETA 07:22:43 2021-05-06 13:54:08 [INFO] [TRAIN] epoch: 23, iter: 8350/40000, loss: 0.5464, lr: 0.008119, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1888 samples/sec | ETA 07:23:42 2021-05-06 13:54:17 [INFO] [TRAIN] epoch: 23, iter: 8360/40000, loss: 0.3977, lr: 0.008117, batch_cost: 0.8408, reader_cost: 0.00017, ips: 1.1893 samples/sec | ETA 07:23:24 2021-05-06 13:54:25 [INFO] [TRAIN] epoch: 23, iter: 8370/40000, loss: 0.9416, lr: 0.008115, batch_cost: 0.8406, reader_cost: 0.00014, ips: 1.1897 samples/sec | ETA 07:23:07 2021-05-06 13:54:33 [INFO] [TRAIN] epoch: 23, iter: 8380/40000, loss: 0.7002, lr: 0.008112, batch_cost: 0.8399, reader_cost: 0.00016, ips: 1.1906 samples/sec | ETA 07:22:37 2021-05-06 13:54:42 [INFO] [TRAIN] epoch: 23, iter: 8390/40000, loss: 0.4246, lr: 0.008110, batch_cost: 0.8418, reader_cost: 0.00016, ips: 1.1880 samples/sec | ETA 07:23:28 2021-05-06 13:54:50 [INFO] [TRAIN] epoch: 23, iter: 8400/40000, loss: 0.3945, lr: 0.008108, batch_cost: 0.8423, reader_cost: 0.00016, ips: 1.1873 samples/sec | ETA 07:23:35 2021-05-06 13:54:59 [INFO] [TRAIN] epoch: 23, iter: 8410/40000, loss: 0.6072, lr: 0.008105, batch_cost: 0.8403, reader_cost: 0.00016, ips: 1.1901 samples/sec | ETA 07:22:24 2021-05-06 13:55:07 [INFO] [TRAIN] epoch: 23, iter: 8420/40000, loss: 0.5440, lr: 0.008103, batch_cost: 0.8405, reader_cost: 0.00016, ips: 1.1898 samples/sec | ETA 07:22:23 2021-05-06 13:55:15 [INFO] [TRAIN] epoch: 23, iter: 8430/40000, loss: 0.3999, lr: 0.008101, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 07:22:27 2021-05-06 13:55:24 [INFO] [TRAIN] epoch: 23, iter: 8440/40000, loss: 0.3919, lr: 0.008099, batch_cost: 0.8447, reader_cost: 0.00015, ips: 1.1839 samples/sec | ETA 07:24:17 2021-05-06 13:55:32 [INFO] [TRAIN] epoch: 23, iter: 8450/40000, loss: 0.4659, lr: 0.008096, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 07:22:09 2021-05-06 13:55:41 [INFO] [TRAIN] epoch: 23, iter: 8460/40000, loss: 0.5352, lr: 0.008094, batch_cost: 0.8428, reader_cost: 0.00016, ips: 1.1865 samples/sec | ETA 07:23:01 2021-05-06 13:55:49 [INFO] [TRAIN] epoch: 23, iter: 8470/40000, loss: 0.5542, lr: 0.008092, batch_cost: 0.8418, reader_cost: 0.00016, ips: 1.1879 samples/sec | ETA 07:22:23 2021-05-06 13:55:58 [INFO] [TRAIN] epoch: 23, iter: 8480/40000, loss: 0.4796, lr: 0.008090, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 07:21:42 2021-05-06 13:56:06 [INFO] [TRAIN] epoch: 23, iter: 8490/40000, loss: 0.3736, lr: 0.008087, batch_cost: 0.8392, reader_cost: 0.00015, ips: 1.1916 samples/sec | ETA 07:20:42 2021-05-06 13:56:14 [INFO] [TRAIN] epoch: 23, iter: 8500/40000, loss: 0.3529, lr: 0.008085, batch_cost: 0.8394, reader_cost: 0.00016, ips: 1.1913 samples/sec | ETA 07:20:42 2021-05-06 13:56:23 [INFO] [TRAIN] epoch: 23, iter: 8510/40000, loss: 0.6458, lr: 0.008083, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 07:21:26 2021-05-06 13:56:31 [INFO] [TRAIN] epoch: 23, iter: 8520/40000, loss: 0.3257, lr: 0.008080, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 07:20:35 2021-05-06 13:56:40 [INFO] [TRAIN] epoch: 23, iter: 8530/40000, loss: 0.7324, lr: 0.008078, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1913 samples/sec | ETA 07:20:16 2021-05-06 13:56:48 [INFO] [TRAIN] epoch: 23, iter: 8540/40000, loss: 0.4394, lr: 0.008076, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 07:20:50 2021-05-06 13:56:56 [INFO] [TRAIN] epoch: 23, iter: 8550/40000, loss: 0.6232, lr: 0.008074, batch_cost: 0.8397, reader_cost: 0.00012, ips: 1.1909 samples/sec | ETA 07:20:08 2021-05-06 13:57:08 [INFO] [TRAIN] epoch: 24, iter: 8560/40000, loss: 0.5637, lr: 0.008071, batch_cost: 1.1677, reader_cost: 0.21128, ips: 0.8564 samples/sec | ETA 10:11:52 2021-05-06 13:57:16 [INFO] [TRAIN] epoch: 24, iter: 8570/40000, loss: 0.6968, lr: 0.008069, batch_cost: 0.8486, reader_cost: 0.00032, ips: 1.1784 samples/sec | ETA 07:24:31 2021-05-06 13:57:25 [INFO] [TRAIN] epoch: 24, iter: 8580/40000, loss: 0.5650, lr: 0.008067, batch_cost: 0.8419, reader_cost: 0.00017, ips: 1.1878 samples/sec | ETA 07:20:51 2021-05-06 13:57:33 [INFO] [TRAIN] epoch: 24, iter: 8590/40000, loss: 0.3896, lr: 0.008064, batch_cost: 0.8429, reader_cost: 0.00015, ips: 1.1864 samples/sec | ETA 07:21:15 2021-05-06 13:57:42 [INFO] [TRAIN] epoch: 24, iter: 8600/40000, loss: 0.3762, lr: 0.008062, batch_cost: 0.8401, reader_cost: 0.00016, ips: 1.1903 samples/sec | ETA 07:19:40 2021-05-06 13:57:50 [INFO] [TRAIN] epoch: 24, iter: 8610/40000, loss: 0.5996, lr: 0.008060, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1879 samples/sec | ETA 07:20:25 2021-05-06 13:57:59 [INFO] [TRAIN] epoch: 24, iter: 8620/40000, loss: 0.8379, lr: 0.008058, batch_cost: 0.8405, reader_cost: 0.00016, ips: 1.1898 samples/sec | ETA 07:19:34 2021-05-06 13:58:07 [INFO] [TRAIN] epoch: 24, iter: 8630/40000, loss: 0.6094, lr: 0.008055, batch_cost: 0.8397, reader_cost: 0.00016, ips: 1.1909 samples/sec | ETA 07:19:00 2021-05-06 13:58:15 [INFO] [TRAIN] epoch: 24, iter: 8640/40000, loss: 0.6017, lr: 0.008053, batch_cost: 0.8398, reader_cost: 0.00016, ips: 1.1908 samples/sec | ETA 07:18:55 2021-05-06 13:58:24 [INFO] [TRAIN] epoch: 24, iter: 8650/40000, loss: 0.4277, lr: 0.008051, batch_cost: 0.8401, reader_cost: 0.00016, ips: 1.1904 samples/sec | ETA 07:18:56 2021-05-06 13:58:32 [INFO] [TRAIN] epoch: 24, iter: 8660/40000, loss: 0.3394, lr: 0.008048, batch_cost: 0.8404, reader_cost: 0.00014, ips: 1.1899 samples/sec | ETA 07:18:58 2021-05-06 13:58:41 [INFO] [TRAIN] epoch: 24, iter: 8670/40000, loss: 0.3632, lr: 0.008046, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 07:19:09 2021-05-06 13:58:49 [INFO] [TRAIN] epoch: 24, iter: 8680/40000, loss: 0.5567, lr: 0.008044, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1888 samples/sec | ETA 07:19:06 2021-05-06 13:58:57 [INFO] [TRAIN] epoch: 24, iter: 8690/40000, loss: 0.5726, lr: 0.008042, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 07:18:16 2021-05-06 13:59:06 [INFO] [TRAIN] epoch: 24, iter: 8700/40000, loss: 0.7033, lr: 0.008039, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 07:18:17 2021-05-06 13:59:14 [INFO] [TRAIN] epoch: 24, iter: 8710/40000, loss: 0.5793, lr: 0.008037, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1882 samples/sec | ETA 07:18:53 2021-05-06 13:59:23 [INFO] [TRAIN] epoch: 24, iter: 8720/40000, loss: 0.5409, lr: 0.008035, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 07:18:19 2021-05-06 13:59:31 [INFO] [TRAIN] epoch: 24, iter: 8730/40000, loss: 0.4846, lr: 0.008032, batch_cost: 0.8433, reader_cost: 0.00015, ips: 1.1858 samples/sec | ETA 07:19:31 2021-05-06 13:59:39 [INFO] [TRAIN] epoch: 24, iter: 8740/40000, loss: 0.7082, lr: 0.008030, batch_cost: 0.8416, reader_cost: 0.00016, ips: 1.1881 samples/sec | ETA 07:18:29 2021-05-06 13:59:48 [INFO] [TRAIN] epoch: 24, iter: 8750/40000, loss: 0.7392, lr: 0.008028, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 07:17:48 2021-05-06 13:59:56 [INFO] [TRAIN] epoch: 24, iter: 8760/40000, loss: 0.3922, lr: 0.008026, batch_cost: 0.8419, reader_cost: 0.00015, ips: 1.1878 samples/sec | ETA 07:18:21 2021-05-06 14:00:05 [INFO] [TRAIN] epoch: 24, iter: 8770/40000, loss: 0.4144, lr: 0.008023, batch_cost: 0.8427, reader_cost: 0.00014, ips: 1.1866 samples/sec | ETA 07:18:39 2021-05-06 14:00:13 [INFO] [TRAIN] epoch: 24, iter: 8780/40000, loss: 0.3445, lr: 0.008021, batch_cost: 0.8412, reader_cost: 0.00016, ips: 1.1888 samples/sec | ETA 07:17:42 2021-05-06 14:00:22 [INFO] [TRAIN] epoch: 24, iter: 8790/40000, loss: 0.2783, lr: 0.008019, batch_cost: 0.8409, reader_cost: 0.00017, ips: 1.1891 samples/sec | ETA 07:17:25 2021-05-06 14:00:30 [INFO] [TRAIN] epoch: 24, iter: 8800/40000, loss: 0.4360, lr: 0.008016, batch_cost: 0.8407, reader_cost: 0.00016, ips: 1.1895 samples/sec | ETA 07:17:10 2021-05-06 14:00:38 [INFO] [TRAIN] epoch: 24, iter: 8810/40000, loss: 0.4700, lr: 0.008014, batch_cost: 0.8421, reader_cost: 0.00015, ips: 1.1876 samples/sec | ETA 07:17:43 2021-05-06 14:00:47 [INFO] [TRAIN] epoch: 24, iter: 8820/40000, loss: 0.4323, lr: 0.008012, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 07:17:01 2021-05-06 14:00:55 [INFO] [TRAIN] epoch: 24, iter: 8830/40000, loss: 0.5760, lr: 0.008010, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 07:16:18 2021-05-06 14:01:04 [INFO] [TRAIN] epoch: 24, iter: 8840/40000, loss: 0.4800, lr: 0.008007, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1882 samples/sec | ETA 07:17:04 2021-05-06 14:01:12 [INFO] [TRAIN] epoch: 24, iter: 8850/40000, loss: 0.7816, lr: 0.008005, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 07:16:34 2021-05-06 14:01:20 [INFO] [TRAIN] epoch: 24, iter: 8860/40000, loss: 0.3821, lr: 0.008003, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 07:16:29 2021-05-06 14:01:29 [INFO] [TRAIN] epoch: 24, iter: 8870/40000, loss: 0.4343, lr: 0.008001, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 07:16:22 2021-05-06 14:01:37 [INFO] [TRAIN] epoch: 24, iter: 8880/40000, loss: 0.5972, lr: 0.007998, batch_cost: 0.8409, reader_cost: 0.00016, ips: 1.1893 samples/sec | ETA 07:16:07 2021-05-06 14:01:46 [INFO] [TRAIN] epoch: 24, iter: 8890/40000, loss: 0.5645, lr: 0.007996, batch_cost: 0.8428, reader_cost: 0.00015, ips: 1.1866 samples/sec | ETA 07:16:58 2021-05-06 14:01:54 [INFO] [TRAIN] epoch: 24, iter: 8900/40000, loss: 0.4451, lr: 0.007994, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 07:15:36 2021-05-06 14:02:02 [INFO] [TRAIN] epoch: 24, iter: 8910/40000, loss: 0.5226, lr: 0.007991, batch_cost: 0.8419, reader_cost: 0.00015, ips: 1.1878 samples/sec | ETA 07:16:15 2021-05-06 14:02:11 [INFO] [TRAIN] epoch: 24, iter: 8920/40000, loss: 0.5883, lr: 0.007989, batch_cost: 0.8384, reader_cost: 0.00012, ips: 1.1927 samples/sec | ETA 07:14:17 2021-05-06 14:02:22 [INFO] [TRAIN] epoch: 25, iter: 8930/40000, loss: 0.4750, lr: 0.007987, batch_cost: 1.1468, reader_cost: 0.20826, ips: 0.8720 samples/sec | ETA 09:53:50 2021-05-06 14:02:31 [INFO] [TRAIN] epoch: 25, iter: 8940/40000, loss: 0.5455, lr: 0.007985, batch_cost: 0.8651, reader_cost: 0.00033, ips: 1.1559 samples/sec | ETA 07:27:50 2021-05-06 14:02:39 [INFO] [TRAIN] epoch: 25, iter: 8950/40000, loss: 0.6215, lr: 0.007982, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1881 samples/sec | ETA 07:15:34 2021-05-06 14:02:48 [INFO] [TRAIN] epoch: 25, iter: 8960/40000, loss: 0.5503, lr: 0.007980, batch_cost: 0.8417, reader_cost: 0.00016, ips: 1.1881 samples/sec | ETA 07:15:26 2021-05-06 14:02:56 [INFO] [TRAIN] epoch: 25, iter: 8970/40000, loss: 1.0310, lr: 0.007978, batch_cost: 0.8408, reader_cost: 0.00016, ips: 1.1893 samples/sec | ETA 07:14:51 2021-05-06 14:03:05 [INFO] [TRAIN] epoch: 25, iter: 8980/40000, loss: 0.8328, lr: 0.007975, batch_cost: 0.8427, reader_cost: 0.00016, ips: 1.1867 samples/sec | ETA 07:15:40 2021-05-06 14:03:13 [INFO] [TRAIN] epoch: 25, iter: 8990/40000, loss: 0.8136, lr: 0.007973, batch_cost: 0.8424, reader_cost: 0.00015, ips: 1.1871 samples/sec | ETA 07:15:22 2021-05-06 14:03:21 [INFO] [TRAIN] epoch: 25, iter: 9000/40000, loss: 0.5945, lr: 0.007971, batch_cost: 0.8396, reader_cost: 0.00016, ips: 1.1911 samples/sec | ETA 07:13:46 2021-05-06 14:03:22 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 14:06:59 [INFO] [EVAL] #Images: 500 mIoU: 0.7059 Acc: 0.9476 Kappa: 0.9319 2021-05-06 14:06:59 [INFO] [EVAL] Class IoU: [0.9727 0.7976 0.9065 0.5776 0.5166 0.4204 0.4989 0.6672 0.9105 0.617 0.9344 0.7359 0.4509 0.9225 0.7671 0.8442 0.6665 0.5264 0.6794] 2021-05-06 14:06:59 [INFO] [EVAL] Class Acc: [0.9865 0.8934 0.9401 0.6786 0.7522 0.7866 0.7894 0.8773 0.9488 0.8025 0.9625 0.826 0.7309 0.9425 0.9327 0.8984 0.781 0.7219 0.8045] 2021-05-06 14:07:48 [INFO] [EVAL] The model with the best validation mIoU (0.7059) was saved at iter 9000. 2021-05-06 14:07:57 [INFO] [TRAIN] epoch: 25, iter: 9010/40000, loss: 0.7150, lr: 0.007969, batch_cost: 0.8364, reader_cost: 0.00049, ips: 1.1956 samples/sec | ETA 07:12:00 2021-05-06 14:08:05 [INFO] [TRAIN] epoch: 25, iter: 9020/40000, loss: 0.4309, lr: 0.007966, batch_cost: 0.8364, reader_cost: 0.00015, ips: 1.1956 samples/sec | ETA 07:11:51 2021-05-06 14:08:13 [INFO] [TRAIN] epoch: 25, iter: 9030/40000, loss: 0.2989, lr: 0.007964, batch_cost: 0.8388, reader_cost: 0.00015, ips: 1.1921 samples/sec | ETA 07:12:58 2021-05-06 14:08:22 [INFO] [TRAIN] epoch: 25, iter: 9040/40000, loss: 0.4361, lr: 0.007962, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 07:13:05 2021-05-06 14:08:30 [INFO] [TRAIN] epoch: 25, iter: 9050/40000, loss: 0.4924, lr: 0.007959, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1886 samples/sec | ETA 07:13:58 2021-05-06 14:08:39 [INFO] [TRAIN] epoch: 25, iter: 9060/40000, loss: 0.5502, lr: 0.007957, batch_cost: 0.8407, reader_cost: 0.00017, ips: 1.1895 samples/sec | ETA 07:13:31 2021-05-06 14:08:47 [INFO] [TRAIN] epoch: 25, iter: 9070/40000, loss: 0.4814, lr: 0.007955, batch_cost: 0.8421, reader_cost: 0.00016, ips: 1.1876 samples/sec | ETA 07:14:04 2021-05-06 14:08:55 [INFO] [TRAIN] epoch: 25, iter: 9080/40000, loss: 0.6094, lr: 0.007953, batch_cost: 0.8432, reader_cost: 0.00016, ips: 1.1860 samples/sec | ETA 07:14:31 2021-05-06 14:09:04 [INFO] [TRAIN] epoch: 25, iter: 9090/40000, loss: 0.3962, lr: 0.007950, batch_cost: 0.8418, reader_cost: 0.00016, ips: 1.1879 samples/sec | ETA 07:13:40 2021-05-06 14:09:12 [INFO] [TRAIN] epoch: 25, iter: 9100/40000, loss: 0.4628, lr: 0.007948, batch_cost: 0.8410, reader_cost: 0.00016, ips: 1.1890 samples/sec | ETA 07:13:07 2021-05-06 14:09:21 [INFO] [TRAIN] epoch: 25, iter: 9110/40000, loss: 0.5313, lr: 0.007946, batch_cost: 0.8407, reader_cost: 0.00014, ips: 1.1894 samples/sec | ETA 07:12:50 2021-05-06 14:09:29 [INFO] [TRAIN] epoch: 25, iter: 9120/40000, loss: 0.6490, lr: 0.007943, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 07:12:09 2021-05-06 14:09:38 [INFO] [TRAIN] epoch: 25, iter: 9130/40000, loss: 0.4970, lr: 0.007941, batch_cost: 0.8426, reader_cost: 0.00015, ips: 1.1868 samples/sec | ETA 07:13:30 2021-05-06 14:09:46 [INFO] [TRAIN] epoch: 25, iter: 9140/40000, loss: 0.5903, lr: 0.007939, batch_cost: 0.8430, reader_cost: 0.00015, ips: 1.1863 samples/sec | ETA 07:13:34 2021-05-06 14:09:54 [INFO] [TRAIN] epoch: 25, iter: 9150/40000, loss: 0.5079, lr: 0.007937, batch_cost: 0.8420, reader_cost: 0.00016, ips: 1.1876 samples/sec | ETA 07:12:56 2021-05-06 14:10:03 [INFO] [TRAIN] epoch: 25, iter: 9160/40000, loss: 0.3748, lr: 0.007934, batch_cost: 0.8432, reader_cost: 0.00015, ips: 1.1859 samples/sec | ETA 07:13:25 2021-05-06 14:10:11 [INFO] [TRAIN] epoch: 25, iter: 9170/40000, loss: 0.3300, lr: 0.007932, batch_cost: 0.8426, reader_cost: 0.00015, ips: 1.1868 samples/sec | ETA 07:12:56 2021-05-06 14:10:20 [INFO] [TRAIN] epoch: 25, iter: 9180/40000, loss: 0.3032, lr: 0.007930, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 07:12:01 2021-05-06 14:10:28 [INFO] [TRAIN] epoch: 25, iter: 9190/40000, loss: 0.5016, lr: 0.007927, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1881 samples/sec | ETA 07:12:12 2021-05-06 14:10:37 [INFO] [TRAIN] epoch: 25, iter: 9200/40000, loss: 0.4012, lr: 0.007925, batch_cost: 0.8426, reader_cost: 0.00015, ips: 1.1867 samples/sec | ETA 07:12:33 2021-05-06 14:10:45 [INFO] [TRAIN] epoch: 25, iter: 9210/40000, loss: 0.3696, lr: 0.007923, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1880 samples/sec | ETA 07:11:57 2021-05-06 14:10:53 [INFO] [TRAIN] epoch: 25, iter: 9220/40000, loss: 0.6363, lr: 0.007921, batch_cost: 0.8422, reader_cost: 0.00015, ips: 1.1873 samples/sec | ETA 07:12:03 2021-05-06 14:11:02 [INFO] [TRAIN] epoch: 25, iter: 9230/40000, loss: 0.4888, lr: 0.007918, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 07:11:17 2021-05-06 14:11:10 [INFO] [TRAIN] epoch: 25, iter: 9240/40000, loss: 0.3488, lr: 0.007916, batch_cost: 0.8411, reader_cost: 0.00014, ips: 1.1889 samples/sec | ETA 07:11:12 2021-05-06 14:11:19 [INFO] [TRAIN] epoch: 25, iter: 9250/40000, loss: 0.6345, lr: 0.007914, batch_cost: 0.8399, reader_cost: 0.00014, ips: 1.1906 samples/sec | ETA 07:10:26 2021-05-06 14:11:27 [INFO] [TRAIN] epoch: 25, iter: 9260/40000, loss: 0.5147, lr: 0.007911, batch_cost: 0.8428, reader_cost: 0.00016, ips: 1.1865 samples/sec | ETA 07:11:47 2021-05-06 14:11:35 [INFO] [TRAIN] epoch: 25, iter: 9270/40000, loss: 0.4592, lr: 0.007909, batch_cost: 0.8423, reader_cost: 0.00015, ips: 1.1872 samples/sec | ETA 07:11:24 2021-05-06 14:11:44 [INFO] [TRAIN] epoch: 25, iter: 9280/40000, loss: 0.4459, lr: 0.007907, batch_cost: 0.8421, reader_cost: 0.00016, ips: 1.1875 samples/sec | ETA 07:11:10 2021-05-06 14:11:52 [INFO] [TRAIN] epoch: 25, iter: 9290/40000, loss: 0.4552, lr: 0.007905, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 07:10:26 2021-05-06 14:12:01 [INFO] [TRAIN] epoch: 25, iter: 9300/40000, loss: 0.5765, lr: 0.007902, batch_cost: 0.8401, reader_cost: 0.00011, ips: 1.1903 samples/sec | ETA 07:09:52 2021-05-06 14:12:13 [INFO] [TRAIN] epoch: 26, iter: 9310/40000, loss: 0.6707, lr: 0.007900, batch_cost: 1.1921, reader_cost: 0.23325, ips: 0.8388 samples/sec | ETA 10:09:46 2021-05-06 14:12:21 [INFO] [TRAIN] epoch: 26, iter: 9320/40000, loss: 0.4099, lr: 0.007898, batch_cost: 0.8398, reader_cost: 0.00030, ips: 1.1907 samples/sec | ETA 07:09:26 2021-05-06 14:12:29 [INFO] [TRAIN] epoch: 26, iter: 9330/40000, loss: 0.7969, lr: 0.007895, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1887 samples/sec | ETA 07:10:02 2021-05-06 14:12:38 [INFO] [TRAIN] epoch: 26, iter: 9340/40000, loss: 0.4131, lr: 0.007893, batch_cost: 0.8417, reader_cost: 0.00016, ips: 1.1881 samples/sec | ETA 07:10:05 2021-05-06 14:12:46 [INFO] [TRAIN] epoch: 26, iter: 9350/40000, loss: 0.5533, lr: 0.007891, batch_cost: 0.8422, reader_cost: 0.00015, ips: 1.1873 samples/sec | ETA 07:10:14 2021-05-06 14:12:55 [INFO] [TRAIN] epoch: 26, iter: 9360/40000, loss: 0.7718, lr: 0.007889, batch_cost: 0.8419, reader_cost: 0.00016, ips: 1.1877 samples/sec | ETA 07:09:56 2021-05-06 14:13:03 [INFO] [TRAIN] epoch: 26, iter: 9370/40000, loss: 0.6188, lr: 0.007886, batch_cost: 0.8419, reader_cost: 0.00015, ips: 1.1878 samples/sec | ETA 07:09:48 2021-05-06 14:13:12 [INFO] [TRAIN] epoch: 26, iter: 9380/40000, loss: 0.5941, lr: 0.007884, batch_cost: 0.8413, reader_cost: 0.00014, ips: 1.1887 samples/sec | ETA 07:09:20 2021-05-06 14:13:20 [INFO] [TRAIN] epoch: 26, iter: 9390/40000, loss: 0.4920, lr: 0.007882, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 07:08:56 2021-05-06 14:13:28 [INFO] [TRAIN] epoch: 26, iter: 9400/40000, loss: 0.1834, lr: 0.007879, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 07:08:46 2021-05-06 14:13:37 [INFO] [TRAIN] epoch: 26, iter: 9410/40000, loss: 0.3633, lr: 0.007877, batch_cost: 0.8402, reader_cost: 0.00014, ips: 1.1902 samples/sec | ETA 07:08:20 2021-05-06 14:13:45 [INFO] [TRAIN] epoch: 26, iter: 9420/40000, loss: 0.5373, lr: 0.007875, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 07:08:27 2021-05-06 14:13:54 [INFO] [TRAIN] epoch: 26, iter: 9430/40000, loss: 0.5233, lr: 0.007872, batch_cost: 0.8403, reader_cost: 0.00014, ips: 1.1901 samples/sec | ETA 07:08:06 2021-05-06 14:14:02 [INFO] [TRAIN] epoch: 26, iter: 9440/40000, loss: 0.7222, lr: 0.007870, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 07:08:20 2021-05-06 14:14:10 [INFO] [TRAIN] epoch: 26, iter: 9450/40000, loss: 0.6870, lr: 0.007868, batch_cost: 0.8405, reader_cost: 0.00014, ips: 1.1897 samples/sec | ETA 07:07:58 2021-05-06 14:14:19 [INFO] [TRAIN] epoch: 26, iter: 9460/40000, loss: 0.5082, lr: 0.007866, batch_cost: 0.8421, reader_cost: 0.00014, ips: 1.1875 samples/sec | ETA 07:08:38 2021-05-06 14:14:27 [INFO] [TRAIN] epoch: 26, iter: 9470/40000, loss: 0.4971, lr: 0.007863, batch_cost: 0.8416, reader_cost: 0.00014, ips: 1.1883 samples/sec | ETA 07:08:12 2021-05-06 14:14:36 [INFO] [TRAIN] epoch: 26, iter: 9480/40000, loss: 0.3284, lr: 0.007861, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 07:07:31 2021-05-06 14:14:44 [INFO] [TRAIN] epoch: 26, iter: 9490/40000, loss: 0.8986, lr: 0.007859, batch_cost: 0.8414, reader_cost: 0.00017, ips: 1.1885 samples/sec | ETA 07:07:51 2021-05-06 14:14:52 [INFO] [TRAIN] epoch: 26, iter: 9500/40000, loss: 0.5374, lr: 0.007856, batch_cost: 0.8408, reader_cost: 0.00016, ips: 1.1894 samples/sec | ETA 07:07:24 2021-05-06 14:15:01 [INFO] [TRAIN] epoch: 26, iter: 9510/40000, loss: 0.5988, lr: 0.007854, batch_cost: 0.8415, reader_cost: 0.00016, ips: 1.1884 samples/sec | ETA 07:07:35 2021-05-06 14:15:09 [INFO] [TRAIN] epoch: 26, iter: 9520/40000, loss: 0.4269, lr: 0.007852, batch_cost: 0.8408, reader_cost: 0.00014, ips: 1.1893 samples/sec | ETA 07:07:08 2021-05-06 14:15:18 [INFO] [TRAIN] epoch: 26, iter: 9530/40000, loss: 0.2585, lr: 0.007850, batch_cost: 0.8399, reader_cost: 0.00014, ips: 1.1906 samples/sec | ETA 07:06:32 2021-05-06 14:15:26 [INFO] [TRAIN] epoch: 26, iter: 9540/40000, loss: 0.5366, lr: 0.007847, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 07:06:53 2021-05-06 14:15:34 [INFO] [TRAIN] epoch: 26, iter: 9550/40000, loss: 0.3585, lr: 0.007845, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 07:06:45 2021-05-06 14:15:43 [INFO] [TRAIN] epoch: 26, iter: 9560/40000, loss: 0.4576, lr: 0.007843, batch_cost: 0.8410, reader_cost: 0.00014, ips: 1.1890 samples/sec | ETA 07:06:40 2021-05-06 14:15:51 [INFO] [TRAIN] epoch: 26, iter: 9570/40000, loss: 0.5326, lr: 0.007840, batch_cost: 0.8394, reader_cost: 0.00014, ips: 1.1914 samples/sec | ETA 07:05:42 2021-05-06 14:16:00 [INFO] [TRAIN] epoch: 26, iter: 9580/40000, loss: 0.4878, lr: 0.007838, batch_cost: 0.8394, reader_cost: 0.00016, ips: 1.1913 samples/sec | ETA 07:05:34 2021-05-06 14:16:08 [INFO] [TRAIN] epoch: 26, iter: 9590/40000, loss: 0.5689, lr: 0.007836, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 07:05:41 2021-05-06 14:16:16 [INFO] [TRAIN] epoch: 26, iter: 9600/40000, loss: 0.4899, lr: 0.007834, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1887 samples/sec | ETA 07:06:14 2021-05-06 14:16:25 [INFO] [TRAIN] epoch: 26, iter: 9610/40000, loss: 0.3599, lr: 0.007831, batch_cost: 0.8424, reader_cost: 0.00015, ips: 1.1871 samples/sec | ETA 07:06:39 2021-05-06 14:16:33 [INFO] [TRAIN] epoch: 26, iter: 9620/40000, loss: 0.6469, lr: 0.007829, batch_cost: 0.8434, reader_cost: 0.00014, ips: 1.1857 samples/sec | ETA 07:07:02 2021-05-06 14:16:42 [INFO] [TRAIN] epoch: 26, iter: 9630/40000, loss: 0.5380, lr: 0.007827, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 07:05:44 2021-05-06 14:16:50 [INFO] [TRAIN] epoch: 26, iter: 9640/40000, loss: 0.4225, lr: 0.007824, batch_cost: 0.8421, reader_cost: 0.00016, ips: 1.1875 samples/sec | ETA 07:06:05 2021-05-06 14:16:59 [INFO] [TRAIN] epoch: 26, iter: 9650/40000, loss: 0.3786, lr: 0.007822, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 07:05:23 2021-05-06 14:17:07 [INFO] [TRAIN] epoch: 26, iter: 9660/40000, loss: 0.3002, lr: 0.007820, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 07:04:59 2021-05-06 14:17:15 [INFO] [TRAIN] epoch: 26, iter: 9670/40000, loss: 0.4989, lr: 0.007818, batch_cost: 0.8393, reader_cost: 0.00027, ips: 1.1914 samples/sec | ETA 07:04:16 2021-05-06 14:17:27 [INFO] [TRAIN] epoch: 27, iter: 9680/40000, loss: 0.5289, lr: 0.007815, batch_cost: 1.1833, reader_cost: 0.20507, ips: 0.8451 samples/sec | ETA 09:57:58 2021-05-06 14:17:36 [INFO] [TRAIN] epoch: 27, iter: 9690/40000, loss: 0.4656, lr: 0.007813, batch_cost: 0.8431, reader_cost: 0.00031, ips: 1.1860 samples/sec | ETA 07:05:55 2021-05-06 14:17:44 [INFO] [TRAIN] epoch: 27, iter: 9700/40000, loss: 0.4540, lr: 0.007811, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 07:04:30 2021-05-06 14:17:52 [INFO] [TRAIN] epoch: 27, iter: 9710/40000, loss: 0.2601, lr: 0.007808, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1884 samples/sec | ETA 07:04:47 2021-05-06 14:18:01 [INFO] [TRAIN] epoch: 27, iter: 9720/40000, loss: 0.2941, lr: 0.007806, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 07:03:54 2021-05-06 14:18:09 [INFO] [TRAIN] epoch: 27, iter: 9730/40000, loss: 0.5121, lr: 0.007804, batch_cost: 0.8424, reader_cost: 0.00014, ips: 1.1870 samples/sec | ETA 07:05:00 2021-05-06 14:18:18 [INFO] [TRAIN] epoch: 27, iter: 9740/40000, loss: 0.5278, lr: 0.007802, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1918 samples/sec | ETA 07:03:11 2021-05-06 14:18:26 [INFO] [TRAIN] epoch: 27, iter: 9750/40000, loss: 0.6684, lr: 0.007799, batch_cost: 0.8389, reader_cost: 0.00016, ips: 1.1920 samples/sec | ETA 07:02:56 2021-05-06 14:18:34 [INFO] [TRAIN] epoch: 27, iter: 9760/40000, loss: 0.4518, lr: 0.007797, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 07:03:17 2021-05-06 14:18:43 [INFO] [TRAIN] epoch: 27, iter: 9770/40000, loss: 0.1662, lr: 0.007795, batch_cost: 0.8397, reader_cost: 0.00016, ips: 1.1908 samples/sec | ETA 07:03:05 2021-05-06 14:18:51 [INFO] [TRAIN] epoch: 27, iter: 9780/40000, loss: 0.3755, lr: 0.007792, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 07:03:20 2021-05-06 14:19:00 [INFO] [TRAIN] epoch: 27, iter: 9790/40000, loss: 0.3171, lr: 0.007790, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1881 samples/sec | ETA 07:03:46 2021-05-06 14:19:08 [INFO] [TRAIN] epoch: 27, iter: 9800/40000, loss: 0.6984, lr: 0.007788, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1918 samples/sec | ETA 07:02:20 2021-05-06 14:19:16 [INFO] [TRAIN] epoch: 27, iter: 9810/40000, loss: 0.6297, lr: 0.007785, batch_cost: 0.8406, reader_cost: 0.00016, ips: 1.1896 samples/sec | ETA 07:02:58 2021-05-06 14:19:25 [INFO] [TRAIN] epoch: 27, iter: 9820/40000, loss: 0.4631, lr: 0.007783, batch_cost: 0.8408, reader_cost: 0.00016, ips: 1.1893 samples/sec | ETA 07:02:55 2021-05-06 14:19:33 [INFO] [TRAIN] epoch: 27, iter: 9830/40000, loss: 0.2029, lr: 0.007781, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1883 samples/sec | ETA 07:03:10 2021-05-06 14:19:42 [INFO] [TRAIN] epoch: 27, iter: 9840/40000, loss: 0.4557, lr: 0.007779, batch_cost: 0.8393, reader_cost: 0.00017, ips: 1.1915 samples/sec | ETA 07:01:51 2021-05-06 14:19:50 [INFO] [TRAIN] epoch: 27, iter: 9850/40000, loss: 0.3531, lr: 0.007776, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1883 samples/sec | ETA 07:02:53 2021-05-06 14:19:59 [INFO] [TRAIN] epoch: 27, iter: 9860/40000, loss: 0.6536, lr: 0.007774, batch_cost: 0.8410, reader_cost: 0.00016, ips: 1.1891 samples/sec | ETA 07:02:27 2021-05-06 14:20:07 [INFO] [TRAIN] epoch: 27, iter: 9870/40000, loss: 0.6310, lr: 0.007772, batch_cost: 0.8407, reader_cost: 0.00016, ips: 1.1895 samples/sec | ETA 07:02:09 2021-05-06 14:20:15 [INFO] [TRAIN] epoch: 27, iter: 9880/40000, loss: 0.4879, lr: 0.007769, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1886 samples/sec | ETA 07:02:21 2021-05-06 14:20:24 [INFO] [TRAIN] epoch: 27, iter: 9890/40000, loss: 0.4521, lr: 0.007767, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1886 samples/sec | ETA 07:02:12 2021-05-06 14:20:32 [INFO] [TRAIN] epoch: 27, iter: 9900/40000, loss: 0.2727, lr: 0.007765, batch_cost: 0.8402, reader_cost: 0.00016, ips: 1.1902 samples/sec | ETA 07:01:28 2021-05-06 14:20:41 [INFO] [TRAIN] epoch: 27, iter: 9910/40000, loss: 0.4546, lr: 0.007763, batch_cost: 0.8402, reader_cost: 0.00016, ips: 1.1901 samples/sec | ETA 07:01:22 2021-05-06 14:20:49 [INFO] [TRAIN] epoch: 27, iter: 9920/40000, loss: 0.3863, lr: 0.007760, batch_cost: 0.8409, reader_cost: 0.00014, ips: 1.1892 samples/sec | ETA 07:01:34 2021-05-06 14:20:57 [INFO] [TRAIN] epoch: 27, iter: 9930/40000, loss: 0.6062, lr: 0.007758, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 07:01:05 2021-05-06 14:21:06 [INFO] [TRAIN] epoch: 27, iter: 9940/40000, loss: 0.3359, lr: 0.007756, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1883 samples/sec | ETA 07:01:36 2021-05-06 14:21:14 [INFO] [TRAIN] epoch: 27, iter: 9950/40000, loss: 0.6552, lr: 0.007753, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 07:00:48 2021-05-06 14:21:23 [INFO] [TRAIN] epoch: 27, iter: 9960/40000, loss: 0.5872, lr: 0.007751, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1881 samples/sec | ETA 07:01:23 2021-05-06 14:21:31 [INFO] [TRAIN] epoch: 27, iter: 9970/40000, loss: 0.5129, lr: 0.007749, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 07:00:15 2021-05-06 14:21:39 [INFO] [TRAIN] epoch: 27, iter: 9980/40000, loss: 0.4172, lr: 0.007747, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 07:00:19 2021-05-06 14:21:48 [INFO] [TRAIN] epoch: 27, iter: 9990/40000, loss: 0.3479, lr: 0.007744, batch_cost: 0.8393, reader_cost: 0.00014, ips: 1.1915 samples/sec | ETA 06:59:47 2021-05-06 14:21:56 [INFO] [TRAIN] epoch: 27, iter: 10000/40000, loss: 0.6452, lr: 0.007742, batch_cost: 0.8391, reader_cost: 0.00014, ips: 1.1917 samples/sec | ETA 06:59:33 2021-05-06 14:21:56 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 14:25:34 [INFO] [EVAL] #Images: 500 mIoU: 0.7133 Acc: 0.9492 Kappa: 0.9340 2021-05-06 14:25:34 [INFO] [EVAL] Class IoU: [0.976 0.8167 0.9034 0.6132 0.5355 0.4366 0.5336 0.6561 0.9104 0.626 0.9361 0.7445 0.5119 0.9253 0.7393 0.8266 0.6683 0.5184 0.6754] 2021-05-06 14:25:34 [INFO] [EVAL] Class Acc: [0.9939 0.8926 0.9282 0.8238 0.7901 0.7621 0.7187 0.8935 0.9482 0.846 0.9654 0.8476 0.6928 0.9469 0.8707 0.941 0.75 0.8633 0.7709] 2021-05-06 14:26:24 [INFO] [EVAL] The model with the best validation mIoU (0.7133) was saved at iter 10000. 2021-05-06 14:26:32 [INFO] [TRAIN] epoch: 27, iter: 10010/40000, loss: 0.3915, lr: 0.007740, batch_cost: 0.8385, reader_cost: 0.00025, ips: 1.1926 samples/sec | ETA 06:59:06 2021-05-06 14:26:41 [INFO] [TRAIN] epoch: 27, iter: 10020/40000, loss: 0.4011, lr: 0.007737, batch_cost: 0.8375, reader_cost: 0.00016, ips: 1.1940 samples/sec | ETA 06:58:29 2021-05-06 14:26:49 [INFO] [TRAIN] epoch: 27, iter: 10030/40000, loss: 0.4219, lr: 0.007735, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1913 samples/sec | ETA 06:59:16 2021-05-06 14:26:57 [INFO] [TRAIN] epoch: 27, iter: 10040/40000, loss: 0.4692, lr: 0.007733, batch_cost: 0.8385, reader_cost: 0.00010, ips: 1.1925 samples/sec | ETA 06:58:42 2021-05-06 14:27:09 [INFO] [TRAIN] epoch: 28, iter: 10050/40000, loss: 0.6605, lr: 0.007730, batch_cost: 1.1940, reader_cost: 0.21090, ips: 0.8375 samples/sec | ETA 09:56:00 2021-05-06 14:27:18 [INFO] [TRAIN] epoch: 28, iter: 10060/40000, loss: 0.5611, lr: 0.007728, batch_cost: 0.8414, reader_cost: 0.00032, ips: 1.1884 samples/sec | ETA 06:59:52 2021-05-06 14:27:26 [INFO] [TRAIN] epoch: 28, iter: 10070/40000, loss: 0.5979, lr: 0.007726, batch_cost: 0.8414, reader_cost: 0.00016, ips: 1.1885 samples/sec | ETA 06:59:42 2021-05-06 14:27:35 [INFO] [TRAIN] epoch: 28, iter: 10080/40000, loss: 0.3112, lr: 0.007724, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 06:59:21 2021-05-06 14:27:43 [INFO] [TRAIN] epoch: 28, iter: 10090/40000, loss: 0.3898, lr: 0.007721, batch_cost: 0.8422, reader_cost: 0.00015, ips: 1.1874 samples/sec | ETA 06:59:50 2021-05-06 14:27:51 [INFO] [TRAIN] epoch: 28, iter: 10100/40000, loss: 0.5907, lr: 0.007719, batch_cost: 0.8413, reader_cost: 0.00014, ips: 1.1886 samples/sec | ETA 06:59:14 2021-05-06 14:28:00 [INFO] [TRAIN] epoch: 28, iter: 10110/40000, loss: 0.6086, lr: 0.007717, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 06:58:06 2021-05-06 14:28:08 [INFO] [TRAIN] epoch: 28, iter: 10120/40000, loss: 0.6702, lr: 0.007714, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 06:58:21 2021-05-06 14:28:17 [INFO] [TRAIN] epoch: 28, iter: 10130/40000, loss: 0.5283, lr: 0.007712, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1903 samples/sec | ETA 06:58:14 2021-05-06 14:28:25 [INFO] [TRAIN] epoch: 28, iter: 10140/40000, loss: 0.7128, lr: 0.007710, batch_cost: 0.8387, reader_cost: 0.00014, ips: 1.1924 samples/sec | ETA 06:57:22 2021-05-06 14:28:33 [INFO] [TRAIN] epoch: 28, iter: 10150/40000, loss: 0.5277, lr: 0.007708, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1912 samples/sec | ETA 06:57:39 2021-05-06 14:28:42 [INFO] [TRAIN] epoch: 28, iter: 10160/40000, loss: 0.6786, lr: 0.007705, batch_cost: 0.8387, reader_cost: 0.00017, ips: 1.1923 samples/sec | ETA 06:57:08 2021-05-06 14:28:50 [INFO] [TRAIN] epoch: 28, iter: 10170/40000, loss: 0.7444, lr: 0.007703, batch_cost: 0.8410, reader_cost: 0.00016, ips: 1.1890 samples/sec | ETA 06:58:08 2021-05-06 14:28:59 [INFO] [TRAIN] epoch: 28, iter: 10180/40000, loss: 0.6876, lr: 0.007701, batch_cost: 0.8388, reader_cost: 0.00015, ips: 1.1922 samples/sec | ETA 06:56:52 2021-05-06 14:29:07 [INFO] [TRAIN] epoch: 28, iter: 10190/40000, loss: 1.0764, lr: 0.007698, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 06:57:34 2021-05-06 14:29:15 [INFO] [TRAIN] epoch: 28, iter: 10200/40000, loss: 0.3834, lr: 0.007696, batch_cost: 0.8396, reader_cost: 0.00016, ips: 1.1910 samples/sec | ETA 06:57:00 2021-05-06 14:29:24 [INFO] [TRAIN] epoch: 28, iter: 10210/40000, loss: 0.4855, lr: 0.007694, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 06:57:28 2021-05-06 14:29:32 [INFO] [TRAIN] epoch: 28, iter: 10220/40000, loss: 0.4314, lr: 0.007691, batch_cost: 0.8408, reader_cost: 0.00016, ips: 1.1894 samples/sec | ETA 06:57:18 2021-05-06 14:29:41 [INFO] [TRAIN] epoch: 28, iter: 10230/40000, loss: 1.0152, lr: 0.007689, batch_cost: 0.8387, reader_cost: 0.00014, ips: 1.1923 samples/sec | ETA 06:56:08 2021-05-06 14:29:49 [INFO] [TRAIN] epoch: 28, iter: 10240/40000, loss: 1.0662, lr: 0.007687, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1914 samples/sec | ETA 06:56:18 2021-05-06 14:29:57 [INFO] [TRAIN] epoch: 28, iter: 10250/40000, loss: 0.7006, lr: 0.007685, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1879 samples/sec | ETA 06:57:24 2021-05-06 14:30:06 [INFO] [TRAIN] epoch: 28, iter: 10260/40000, loss: 0.5965, lr: 0.007682, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1914 samples/sec | ETA 06:56:02 2021-05-06 14:30:14 [INFO] [TRAIN] epoch: 28, iter: 10270/40000, loss: 0.4371, lr: 0.007680, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 06:56:28 2021-05-06 14:30:23 [INFO] [TRAIN] epoch: 28, iter: 10280/40000, loss: 0.3393, lr: 0.007678, batch_cost: 0.8423, reader_cost: 0.00015, ips: 1.1872 samples/sec | ETA 06:57:14 2021-05-06 14:30:31 [INFO] [TRAIN] epoch: 28, iter: 10290/40000, loss: 0.3745, lr: 0.007675, batch_cost: 0.8405, reader_cost: 0.00014, ips: 1.1897 samples/sec | ETA 06:56:12 2021-05-06 14:30:39 [INFO] [TRAIN] epoch: 28, iter: 10300/40000, loss: 0.4814, lr: 0.007673, batch_cost: 0.8412, reader_cost: 0.00017, ips: 1.1888 samples/sec | ETA 06:56:23 2021-05-06 14:30:48 [INFO] [TRAIN] epoch: 28, iter: 10310/40000, loss: 0.4170, lr: 0.007671, batch_cost: 0.8392, reader_cost: 0.00016, ips: 1.1916 samples/sec | ETA 06:55:17 2021-05-06 14:30:56 [INFO] [TRAIN] epoch: 28, iter: 10320/40000, loss: 0.3772, lr: 0.007669, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 06:55:41 2021-05-06 14:31:05 [INFO] [TRAIN] epoch: 28, iter: 10330/40000, loss: 0.4283, lr: 0.007666, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 06:55:44 2021-05-06 14:31:13 [INFO] [TRAIN] epoch: 28, iter: 10340/40000, loss: 0.5861, lr: 0.007664, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 06:55:31 2021-05-06 14:31:21 [INFO] [TRAIN] epoch: 28, iter: 10350/40000, loss: 0.3857, lr: 0.007662, batch_cost: 0.8399, reader_cost: 0.00014, ips: 1.1906 samples/sec | ETA 06:55:02 2021-05-06 14:31:30 [INFO] [TRAIN] epoch: 28, iter: 10360/40000, loss: 0.5048, lr: 0.007659, batch_cost: 0.8402, reader_cost: 0.00016, ips: 1.1901 samples/sec | ETA 06:55:04 2021-05-06 14:31:38 [INFO] [TRAIN] epoch: 28, iter: 10370/40000, loss: 0.5636, lr: 0.007657, batch_cost: 0.8402, reader_cost: 0.00014, ips: 1.1901 samples/sec | ETA 06:54:56 2021-05-06 14:31:47 [INFO] [TRAIN] epoch: 28, iter: 10380/40000, loss: 0.3413, lr: 0.007655, batch_cost: 0.8427, reader_cost: 0.00015, ips: 1.1867 samples/sec | ETA 06:55:59 2021-05-06 14:31:55 [INFO] [TRAIN] epoch: 28, iter: 10390/40000, loss: 0.4762, lr: 0.007652, batch_cost: 0.8411, reader_cost: 0.00014, ips: 1.1890 samples/sec | ETA 06:55:03 2021-05-06 14:32:04 [INFO] [TRAIN] epoch: 28, iter: 10400/40000, loss: 0.6034, lr: 0.007650, batch_cost: 0.8409, reader_cost: 0.00014, ips: 1.1892 samples/sec | ETA 06:54:49 2021-05-06 14:32:12 [INFO] [TRAIN] epoch: 28, iter: 10410/40000, loss: 0.5900, lr: 0.007648, batch_cost: 0.8400, reader_cost: 0.00012, ips: 1.1904 samples/sec | ETA 06:54:16 2021-05-06 14:32:23 [INFO] [TRAIN] epoch: 29, iter: 10420/40000, loss: 0.4878, lr: 0.007646, batch_cost: 1.1280, reader_cost: 0.20936, ips: 0.8865 samples/sec | ETA 09:16:06 2021-05-06 14:32:32 [INFO] [TRAIN] epoch: 29, iter: 10430/40000, loss: 0.4156, lr: 0.007643, batch_cost: 0.8448, reader_cost: 0.00034, ips: 1.1838 samples/sec | ETA 06:56:19 2021-05-06 14:32:40 [INFO] [TRAIN] epoch: 29, iter: 10440/40000, loss: 0.4361, lr: 0.007641, batch_cost: 0.8418, reader_cost: 0.00016, ips: 1.1879 samples/sec | ETA 06:54:43 2021-05-06 14:32:49 [INFO] [TRAIN] epoch: 29, iter: 10450/40000, loss: 0.4716, lr: 0.007639, batch_cost: 0.8430, reader_cost: 0.00015, ips: 1.1862 samples/sec | ETA 06:55:10 2021-05-06 14:32:57 [INFO] [TRAIN] epoch: 29, iter: 10460/40000, loss: 0.3785, lr: 0.007636, batch_cost: 0.8410, reader_cost: 0.00014, ips: 1.1891 samples/sec | ETA 06:54:01 2021-05-06 14:33:05 [INFO] [TRAIN] epoch: 29, iter: 10470/40000, loss: 0.8596, lr: 0.007634, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1879 samples/sec | ETA 06:54:18 2021-05-06 14:33:14 [INFO] [TRAIN] epoch: 29, iter: 10480/40000, loss: 0.6198, lr: 0.007632, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1904 samples/sec | ETA 06:53:18 2021-05-06 14:33:22 [INFO] [TRAIN] epoch: 29, iter: 10490/40000, loss: 0.7002, lr: 0.007630, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1880 samples/sec | ETA 06:53:59 2021-05-06 14:33:31 [INFO] [TRAIN] epoch: 29, iter: 10500/40000, loss: 0.4687, lr: 0.007627, batch_cost: 0.8421, reader_cost: 0.00015, ips: 1.1875 samples/sec | ETA 06:54:02 2021-05-06 14:33:39 [INFO] [TRAIN] epoch: 29, iter: 10510/40000, loss: 0.2996, lr: 0.007625, batch_cost: 0.8418, reader_cost: 0.00016, ips: 1.1879 samples/sec | ETA 06:53:44 2021-05-06 14:33:47 [INFO] [TRAIN] epoch: 29, iter: 10520/40000, loss: 0.1960, lr: 0.007623, batch_cost: 0.8411, reader_cost: 0.00016, ips: 1.1890 samples/sec | ETA 06:53:14 2021-05-06 14:33:56 [INFO] [TRAIN] epoch: 29, iter: 10530/40000, loss: 0.3809, lr: 0.007620, batch_cost: 0.8411, reader_cost: 0.00016, ips: 1.1890 samples/sec | ETA 06:53:06 2021-05-06 14:34:04 [INFO] [TRAIN] epoch: 29, iter: 10540/40000, loss: 0.5025, lr: 0.007618, batch_cost: 0.8420, reader_cost: 0.00015, ips: 1.1876 samples/sec | ETA 06:53:25 2021-05-06 14:34:13 [INFO] [TRAIN] epoch: 29, iter: 10550/40000, loss: 0.6023, lr: 0.007616, batch_cost: 0.8423, reader_cost: 0.00015, ips: 1.1873 samples/sec | ETA 06:53:24 2021-05-06 14:34:21 [INFO] [TRAIN] epoch: 29, iter: 10560/40000, loss: 0.6154, lr: 0.007613, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1879 samples/sec | ETA 06:53:02 2021-05-06 14:34:30 [INFO] [TRAIN] epoch: 29, iter: 10570/40000, loss: 0.3921, lr: 0.007611, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 06:52:34 2021-05-06 14:34:38 [INFO] [TRAIN] epoch: 29, iter: 10580/40000, loss: 0.4802, lr: 0.007609, batch_cost: 0.8415, reader_cost: 0.00014, ips: 1.1883 samples/sec | ETA 06:52:38 2021-05-06 14:34:46 [INFO] [TRAIN] epoch: 29, iter: 10590/40000, loss: 0.2824, lr: 0.007607, batch_cost: 0.8410, reader_cost: 0.00014, ips: 1.1890 samples/sec | ETA 06:52:15 2021-05-06 14:34:55 [INFO] [TRAIN] epoch: 29, iter: 10600/40000, loss: 0.3393, lr: 0.007604, batch_cost: 0.8405, reader_cost: 0.00014, ips: 1.1898 samples/sec | ETA 06:51:51 2021-05-06 14:35:03 [INFO] [TRAIN] epoch: 29, iter: 10610/40000, loss: 0.8111, lr: 0.007602, batch_cost: 0.8388, reader_cost: 0.00014, ips: 1.1922 samples/sec | ETA 06:50:52 2021-05-06 14:35:12 [INFO] [TRAIN] epoch: 29, iter: 10620/40000, loss: 0.4168, lr: 0.007600, batch_cost: 0.8390, reader_cost: 0.00014, ips: 1.1918 samples/sec | ETA 06:50:51 2021-05-06 14:35:20 [INFO] [TRAIN] epoch: 29, iter: 10630/40000, loss: 0.4410, lr: 0.007597, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1903 samples/sec | ETA 06:51:14 2021-05-06 14:35:28 [INFO] [TRAIN] epoch: 29, iter: 10640/40000, loss: 0.3270, lr: 0.007595, batch_cost: 0.8406, reader_cost: 0.00016, ips: 1.1897 samples/sec | ETA 06:51:18 2021-05-06 14:35:37 [INFO] [TRAIN] epoch: 29, iter: 10650/40000, loss: 0.3656, lr: 0.007593, batch_cost: 0.8407, reader_cost: 0.00014, ips: 1.1895 samples/sec | ETA 06:51:14 2021-05-06 14:35:45 [INFO] [TRAIN] epoch: 29, iter: 10660/40000, loss: 0.3728, lr: 0.007590, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 06:50:55 2021-05-06 14:35:54 [INFO] [TRAIN] epoch: 29, iter: 10670/40000, loss: 0.3541, lr: 0.007588, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1912 samples/sec | ETA 06:50:21 2021-05-06 14:36:02 [INFO] [TRAIN] epoch: 29, iter: 10680/40000, loss: 0.4472, lr: 0.007586, batch_cost: 0.8405, reader_cost: 0.00016, ips: 1.1897 samples/sec | ETA 06:50:44 2021-05-06 14:36:10 [INFO] [TRAIN] epoch: 29, iter: 10690/40000, loss: 0.2688, lr: 0.007584, batch_cost: 0.8398, reader_cost: 0.00016, ips: 1.1908 samples/sec | ETA 06:50:13 2021-05-06 14:36:19 [INFO] [TRAIN] epoch: 29, iter: 10700/40000, loss: 0.3616, lr: 0.007581, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 06:50:33 2021-05-06 14:36:27 [INFO] [TRAIN] epoch: 29, iter: 10710/40000, loss: 0.4515, lr: 0.007579, batch_cost: 0.8424, reader_cost: 0.00015, ips: 1.1871 samples/sec | ETA 06:51:13 2021-05-06 14:36:36 [INFO] [TRAIN] epoch: 29, iter: 10720/40000, loss: 0.4845, lr: 0.007577, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 06:50:22 2021-05-06 14:36:44 [INFO] [TRAIN] epoch: 29, iter: 10730/40000, loss: 0.4083, lr: 0.007574, batch_cost: 0.8414, reader_cost: 0.00016, ips: 1.1885 samples/sec | ETA 06:50:27 2021-05-06 14:36:52 [INFO] [TRAIN] epoch: 29, iter: 10740/40000, loss: 0.6288, lr: 0.007572, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 06:49:50 2021-05-06 14:37:01 [INFO] [TRAIN] epoch: 29, iter: 10750/40000, loss: 0.4530, lr: 0.007570, batch_cost: 0.8421, reader_cost: 0.00016, ips: 1.1876 samples/sec | ETA 06:50:30 2021-05-06 14:37:09 [INFO] [TRAIN] epoch: 29, iter: 10760/40000, loss: 0.5241, lr: 0.007567, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1887 samples/sec | ETA 06:49:58 2021-05-06 14:37:18 [INFO] [TRAIN] epoch: 29, iter: 10770/40000, loss: 0.3621, lr: 0.007565, batch_cost: 0.8418, reader_cost: 0.00016, ips: 1.1879 samples/sec | ETA 06:50:05 2021-05-06 14:37:26 [INFO] [TRAIN] epoch: 29, iter: 10780/40000, loss: 0.5576, lr: 0.007563, batch_cost: 0.8392, reader_cost: 0.00014, ips: 1.1916 samples/sec | ETA 06:48:42 2021-05-06 14:37:37 [INFO] [TRAIN] epoch: 30, iter: 10790/40000, loss: 0.4511, lr: 0.007561, batch_cost: 1.1397, reader_cost: 0.21665, ips: 0.8774 samples/sec | ETA 09:14:51 2021-05-06 14:37:46 [INFO] [TRAIN] epoch: 30, iter: 10800/40000, loss: 0.4955, lr: 0.007558, batch_cost: 0.8583, reader_cost: 0.00032, ips: 1.1650 samples/sec | ETA 06:57:43 2021-05-06 14:37:54 [INFO] [TRAIN] epoch: 30, iter: 10810/40000, loss: 0.6742, lr: 0.007556, batch_cost: 0.8416, reader_cost: 0.00014, ips: 1.1882 samples/sec | ETA 06:49:26 2021-05-06 14:38:03 [INFO] [TRAIN] epoch: 30, iter: 10820/40000, loss: 0.4176, lr: 0.007554, batch_cost: 0.8404, reader_cost: 0.00014, ips: 1.1899 samples/sec | ETA 06:48:43 2021-05-06 14:38:11 [INFO] [TRAIN] epoch: 30, iter: 10830/40000, loss: 0.3230, lr: 0.007551, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 06:48:28 2021-05-06 14:38:20 [INFO] [TRAIN] epoch: 30, iter: 10840/40000, loss: 0.6387, lr: 0.007549, batch_cost: 0.8419, reader_cost: 0.00016, ips: 1.1878 samples/sec | ETA 06:49:10 2021-05-06 14:38:28 [INFO] [TRAIN] epoch: 30, iter: 10850/40000, loss: 0.7701, lr: 0.007547, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 06:48:09 2021-05-06 14:38:37 [INFO] [TRAIN] epoch: 30, iter: 10860/40000, loss: 0.6218, lr: 0.007544, batch_cost: 0.8419, reader_cost: 0.00016, ips: 1.1878 samples/sec | ETA 06:48:52 2021-05-06 14:38:45 [INFO] [TRAIN] epoch: 30, iter: 10870/40000, loss: 0.5541, lr: 0.007542, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1887 samples/sec | ETA 06:48:24 2021-05-06 14:38:53 [INFO] [TRAIN] epoch: 30, iter: 10880/40000, loss: 0.4301, lr: 0.007540, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 06:48:03 2021-05-06 14:39:02 [INFO] [TRAIN] epoch: 30, iter: 10890/40000, loss: 0.2236, lr: 0.007538, batch_cost: 0.8419, reader_cost: 0.00016, ips: 1.1878 samples/sec | ETA 06:48:27 2021-05-06 14:39:10 [INFO] [TRAIN] epoch: 30, iter: 10900/40000, loss: 0.3468, lr: 0.007535, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 06:47:30 2021-05-06 14:39:19 [INFO] [TRAIN] epoch: 30, iter: 10910/40000, loss: 0.5702, lr: 0.007533, batch_cost: 0.8418, reader_cost: 0.00016, ips: 1.1879 samples/sec | ETA 06:48:08 2021-05-06 14:39:27 [INFO] [TRAIN] epoch: 30, iter: 10920/40000, loss: 0.9628, lr: 0.007531, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 06:47:14 2021-05-06 14:39:35 [INFO] [TRAIN] epoch: 30, iter: 10930/40000, loss: 0.6132, lr: 0.007528, batch_cost: 0.8382, reader_cost: 0.00016, ips: 1.1930 samples/sec | ETA 06:46:06 2021-05-06 14:39:44 [INFO] [TRAIN] epoch: 30, iter: 10940/40000, loss: 0.5451, lr: 0.007526, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 06:46:39 2021-05-06 14:39:52 [INFO] [TRAIN] epoch: 30, iter: 10950/40000, loss: 0.4392, lr: 0.007524, batch_cost: 0.8413, reader_cost: 0.00014, ips: 1.1886 samples/sec | ETA 06:47:19 2021-05-06 14:40:01 [INFO] [TRAIN] epoch: 30, iter: 10960/40000, loss: 0.4158, lr: 0.007521, batch_cost: 0.8395, reader_cost: 0.00014, ips: 1.1912 samples/sec | ETA 06:46:19 2021-05-06 14:40:09 [INFO] [TRAIN] epoch: 30, iter: 10970/40000, loss: 0.3398, lr: 0.007519, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1882 samples/sec | ETA 06:47:11 2021-05-06 14:40:17 [INFO] [TRAIN] epoch: 30, iter: 10980/40000, loss: 0.6197, lr: 0.007517, batch_cost: 0.8405, reader_cost: 0.00016, ips: 1.1898 samples/sec | ETA 06:46:30 2021-05-06 14:40:26 [INFO] [TRAIN] epoch: 30, iter: 10990/40000, loss: 0.5918, lr: 0.007515, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1884 samples/sec | ETA 06:46:50 2021-05-06 14:40:34 [INFO] [TRAIN] epoch: 30, iter: 11000/40000, loss: 0.4916, lr: 0.007512, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1886 samples/sec | ETA 06:46:38 2021-05-06 14:40:34 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 14:44:10 [INFO] [EVAL] #Images: 500 mIoU: 0.7039 Acc: 0.9472 Kappa: 0.9316 2021-05-06 14:44:10 [INFO] [EVAL] Class IoU: [0.9737 0.8005 0.9032 0.5791 0.5542 0.4759 0.5239 0.6677 0.905 0.6151 0.9243 0.7466 0.4846 0.9255 0.7673 0.7358 0.563 0.5393 0.69 ] 2021-05-06 14:44:10 [INFO] [EVAL] Class Acc: [0.9916 0.8573 0.9393 0.8348 0.7258 0.7513 0.7695 0.8994 0.9471 0.7648 0.9476 0.8621 0.7786 0.9539 0.8759 0.9463 0.625 0.8119 0.7878] 2021-05-06 14:44:39 [INFO] [EVAL] The model with the best validation mIoU (0.7133) was saved at iter 10000. 2021-05-06 14:44:47 [INFO] [TRAIN] epoch: 30, iter: 11010/40000, loss: 0.3358, lr: 0.007510, batch_cost: 0.8369, reader_cost: 0.00057, ips: 1.1948 samples/sec | ETA 06:44:22 2021-05-06 14:44:56 [INFO] [TRAIN] epoch: 30, iter: 11020/40000, loss: 0.3687, lr: 0.007508, batch_cost: 0.8396, reader_cost: 0.00016, ips: 1.1911 samples/sec | ETA 06:45:30 2021-05-06 14:45:04 [INFO] [TRAIN] epoch: 30, iter: 11030/40000, loss: 0.3967, lr: 0.007505, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1911 samples/sec | ETA 06:45:21 2021-05-06 14:45:14 [INFO] [TRAIN] epoch: 30, iter: 11040/40000, loss: 0.3724, lr: 0.007503, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 06:45:58 2021-05-06 14:45:22 [INFO] [TRAIN] epoch: 30, iter: 11050/40000, loss: 0.4695, lr: 0.007501, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 06:45:50 2021-05-06 14:45:31 [INFO] [TRAIN] epoch: 30, iter: 11060/40000, loss: 0.2828, lr: 0.007498, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1880 samples/sec | ETA 06:46:00 2021-05-06 14:45:39 [INFO] [TRAIN] epoch: 30, iter: 11070/40000, loss: 0.3841, lr: 0.007496, batch_cost: 0.8424, reader_cost: 0.00015, ips: 1.1871 samples/sec | ETA 06:46:10 2021-05-06 14:45:47 [INFO] [TRAIN] epoch: 30, iter: 11080/40000, loss: 0.5779, lr: 0.007494, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1887 samples/sec | ETA 06:45:29 2021-05-06 14:45:56 [INFO] [TRAIN] epoch: 30, iter: 11090/40000, loss: 0.5284, lr: 0.007492, batch_cost: 0.8392, reader_cost: 0.00013, ips: 1.1917 samples/sec | ETA 06:44:20 2021-05-06 14:46:04 [INFO] [TRAIN] epoch: 30, iter: 11100/40000, loss: 0.3636, lr: 0.007489, batch_cost: 0.8398, reader_cost: 0.00014, ips: 1.1907 samples/sec | ETA 06:44:31 2021-05-06 14:46:13 [INFO] [TRAIN] epoch: 30, iter: 11110/40000, loss: 0.5410, lr: 0.007487, batch_cost: 0.8404, reader_cost: 0.00014, ips: 1.1898 samples/sec | ETA 06:44:40 2021-05-06 14:46:21 [INFO] [TRAIN] epoch: 30, iter: 11120/40000, loss: 0.4424, lr: 0.007485, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 06:44:14 2021-05-06 14:46:29 [INFO] [TRAIN] epoch: 30, iter: 11130/40000, loss: 0.4329, lr: 0.007482, batch_cost: 0.8397, reader_cost: 0.00014, ips: 1.1909 samples/sec | ETA 06:44:01 2021-05-06 14:46:38 [INFO] [TRAIN] epoch: 30, iter: 11140/40000, loss: 0.3496, lr: 0.007480, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 06:44:29 2021-05-06 14:46:46 [INFO] [TRAIN] epoch: 30, iter: 11150/40000, loss: 0.5510, lr: 0.007478, batch_cost: 0.8408, reader_cost: 0.00014, ips: 1.1894 samples/sec | ETA 06:44:15 2021-05-06 14:46:55 [INFO] [TRAIN] epoch: 30, iter: 11160/40000, loss: 0.5398, lr: 0.007475, batch_cost: 0.8387, reader_cost: 0.00009, ips: 1.1924 samples/sec | ETA 06:43:07 2021-05-06 14:47:06 [INFO] [TRAIN] epoch: 31, iter: 11170/40000, loss: 0.6243, lr: 0.007473, batch_cost: 1.1759, reader_cost: 0.22332, ips: 0.8504 samples/sec | ETA 09:25:00 2021-05-06 14:47:15 [INFO] [TRAIN] epoch: 31, iter: 11180/40000, loss: 0.4780, lr: 0.007471, batch_cost: 0.8420, reader_cost: 0.00030, ips: 1.1876 samples/sec | ETA 06:44:26 2021-05-06 14:47:23 [INFO] [TRAIN] epoch: 31, iter: 11190/40000, loss: 0.5270, lr: 0.007469, batch_cost: 0.8406, reader_cost: 0.00014, ips: 1.1896 samples/sec | ETA 06:43:37 2021-05-06 14:47:32 [INFO] [TRAIN] epoch: 31, iter: 11200/40000, loss: 0.4959, lr: 0.007466, batch_cost: 0.8407, reader_cost: 0.00014, ips: 1.1895 samples/sec | ETA 06:43:31 2021-05-06 14:47:40 [INFO] [TRAIN] epoch: 31, iter: 11210/40000, loss: 0.3737, lr: 0.007464, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 06:43:27 2021-05-06 14:47:48 [INFO] [TRAIN] epoch: 31, iter: 11220/40000, loss: 0.7720, lr: 0.007462, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1914 samples/sec | ETA 06:42:35 2021-05-06 14:47:57 [INFO] [TRAIN] epoch: 31, iter: 11230/40000, loss: 0.5645, lr: 0.007459, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1917 samples/sec | ETA 06:42:21 2021-05-06 14:48:05 [INFO] [TRAIN] epoch: 31, iter: 11240/40000, loss: 0.7017, lr: 0.007457, batch_cost: 0.8390, reader_cost: 0.00015, ips: 1.1919 samples/sec | ETA 06:42:10 2021-05-06 14:48:14 [INFO] [TRAIN] epoch: 31, iter: 11250/40000, loss: 0.4428, lr: 0.007455, batch_cost: 0.8392, reader_cost: 0.00016, ips: 1.1917 samples/sec | ETA 06:42:05 2021-05-06 14:48:22 [INFO] [TRAIN] epoch: 31, iter: 11260/40000, loss: 0.1734, lr: 0.007452, batch_cost: 0.8412, reader_cost: 0.00016, ips: 1.1888 samples/sec | ETA 06:42:55 2021-05-06 14:48:30 [INFO] [TRAIN] epoch: 31, iter: 11270/40000, loss: 0.2839, lr: 0.007450, batch_cost: 0.8389, reader_cost: 0.00016, ips: 1.1920 samples/sec | ETA 06:41:41 2021-05-06 14:48:39 [INFO] [TRAIN] epoch: 31, iter: 11280/40000, loss: 0.4497, lr: 0.007448, batch_cost: 0.8389, reader_cost: 0.00015, ips: 1.1920 samples/sec | ETA 06:41:32 2021-05-06 14:48:47 [INFO] [TRAIN] epoch: 31, iter: 11290/40000, loss: 0.5482, lr: 0.007446, batch_cost: 0.8382, reader_cost: 0.00014, ips: 1.1930 samples/sec | ETA 06:41:04 2021-05-06 14:48:56 [INFO] [TRAIN] epoch: 31, iter: 11300/40000, loss: 0.7741, lr: 0.007443, batch_cost: 0.8412, reader_cost: 0.00014, ips: 1.1887 samples/sec | ETA 06:42:23 2021-05-06 14:49:04 [INFO] [TRAIN] epoch: 31, iter: 11310/40000, loss: 0.5103, lr: 0.007441, batch_cost: 0.8390, reader_cost: 0.00014, ips: 1.1919 samples/sec | ETA 06:41:10 2021-05-06 14:49:12 [INFO] [TRAIN] epoch: 31, iter: 11320/40000, loss: 0.3500, lr: 0.007439, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 06:41:33 2021-05-06 14:49:21 [INFO] [TRAIN] epoch: 31, iter: 11330/40000, loss: 0.3651, lr: 0.007436, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 06:41:19 2021-05-06 14:49:29 [INFO] [TRAIN] epoch: 31, iter: 11340/40000, loss: 0.3884, lr: 0.007434, batch_cost: 0.8399, reader_cost: 0.00016, ips: 1.1906 samples/sec | ETA 06:41:11 2021-05-06 14:49:38 [INFO] [TRAIN] epoch: 31, iter: 11350/40000, loss: 0.4957, lr: 0.007432, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1880 samples/sec | ETA 06:41:55 2021-05-06 14:49:46 [INFO] [TRAIN] epoch: 31, iter: 11360/40000, loss: 1.4500, lr: 0.007429, batch_cost: 0.8412, reader_cost: 0.00016, ips: 1.1887 samples/sec | ETA 06:41:32 2021-05-06 14:49:54 [INFO] [TRAIN] epoch: 31, iter: 11370/40000, loss: 0.4956, lr: 0.007427, batch_cost: 0.8392, reader_cost: 0.00015, ips: 1.1916 samples/sec | ETA 06:40:27 2021-05-06 14:50:03 [INFO] [TRAIN] epoch: 31, iter: 11380/40000, loss: 0.5214, lr: 0.007425, batch_cost: 0.8412, reader_cost: 0.00016, ips: 1.1888 samples/sec | ETA 06:41:13 2021-05-06 14:50:11 [INFO] [TRAIN] epoch: 31, iter: 11390/40000, loss: 0.2529, lr: 0.007423, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 06:40:44 2021-05-06 14:50:20 [INFO] [TRAIN] epoch: 31, iter: 11400/40000, loss: 0.4320, lr: 0.007420, batch_cost: 0.8412, reader_cost: 0.00016, ips: 1.1888 samples/sec | ETA 06:40:58 2021-05-06 14:50:28 [INFO] [TRAIN] epoch: 31, iter: 11410/40000, loss: 0.3060, lr: 0.007418, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 06:40:17 2021-05-06 14:50:36 [INFO] [TRAIN] epoch: 31, iter: 11420/40000, loss: 0.3735, lr: 0.007416, batch_cost: 0.8400, reader_cost: 0.00016, ips: 1.1904 samples/sec | ETA 06:40:08 2021-05-06 14:50:45 [INFO] [TRAIN] epoch: 31, iter: 11430/40000, loss: 0.3898, lr: 0.007413, batch_cost: 0.8404, reader_cost: 0.00016, ips: 1.1899 samples/sec | ETA 06:40:09 2021-05-06 14:50:53 [INFO] [TRAIN] epoch: 31, iter: 11440/40000, loss: 0.6094, lr: 0.007411, batch_cost: 0.8400, reader_cost: 0.00016, ips: 1.1905 samples/sec | ETA 06:39:49 2021-05-06 14:51:02 [INFO] [TRAIN] epoch: 31, iter: 11450/40000, loss: 0.5018, lr: 0.007409, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 06:39:44 2021-05-06 14:51:10 [INFO] [TRAIN] epoch: 31, iter: 11460/40000, loss: 0.3215, lr: 0.007406, batch_cost: 0.8398, reader_cost: 0.00016, ips: 1.1908 samples/sec | ETA 06:39:26 2021-05-06 14:51:18 [INFO] [TRAIN] epoch: 31, iter: 11470/40000, loss: 0.3266, lr: 0.007404, batch_cost: 0.8399, reader_cost: 0.00014, ips: 1.1905 samples/sec | ETA 06:39:23 2021-05-06 14:51:27 [INFO] [TRAIN] epoch: 31, iter: 11480/40000, loss: 0.5209, lr: 0.007402, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1885 samples/sec | ETA 06:39:57 2021-05-06 14:51:35 [INFO] [TRAIN] epoch: 31, iter: 11490/40000, loss: 0.4105, lr: 0.007399, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1912 samples/sec | ETA 06:38:53 2021-05-06 14:51:44 [INFO] [TRAIN] epoch: 31, iter: 11500/40000, loss: 0.6548, lr: 0.007397, batch_cost: 0.8404, reader_cost: 0.00014, ips: 1.1899 samples/sec | ETA 06:39:11 2021-05-06 14:51:52 [INFO] [TRAIN] epoch: 31, iter: 11510/40000, loss: 0.3119, lr: 0.007395, batch_cost: 0.8410, reader_cost: 0.00014, ips: 1.1891 samples/sec | ETA 06:39:19 2021-05-06 14:52:00 [INFO] [TRAIN] epoch: 31, iter: 11520/40000, loss: 0.4256, lr: 0.007393, batch_cost: 0.8415, reader_cost: 0.00014, ips: 1.1884 samples/sec | ETA 06:39:24 2021-05-06 14:52:09 [INFO] [TRAIN] epoch: 31, iter: 11530/40000, loss: 0.6930, lr: 0.007390, batch_cost: 0.8389, reader_cost: 0.00028, ips: 1.1920 samples/sec | ETA 06:38:04 2021-05-06 14:52:20 [INFO] [TRAIN] epoch: 32, iter: 11540/40000, loss: 0.5327, lr: 0.007388, batch_cost: 1.1442, reader_cost: 0.21369, ips: 0.8740 samples/sec | ETA 09:02:43 2021-05-06 14:52:29 [INFO] [TRAIN] epoch: 32, iter: 11550/40000, loss: 0.5882, lr: 0.007386, batch_cost: 0.8424, reader_cost: 0.00031, ips: 1.1871 samples/sec | ETA 06:39:25 2021-05-06 14:52:37 [INFO] [TRAIN] epoch: 32, iter: 11560/40000, loss: 0.5101, lr: 0.007383, batch_cost: 0.8415, reader_cost: 0.00016, ips: 1.1883 samples/sec | ETA 06:38:53 2021-05-06 14:52:46 [INFO] [TRAIN] epoch: 32, iter: 11570/40000, loss: 0.3645, lr: 0.007381, batch_cost: 0.8410, reader_cost: 0.00016, ips: 1.1890 samples/sec | ETA 06:38:30 2021-05-06 14:52:54 [INFO] [TRAIN] epoch: 32, iter: 11580/40000, loss: 0.4769, lr: 0.007379, batch_cost: 0.8412, reader_cost: 0.00016, ips: 1.1887 samples/sec | ETA 06:38:28 2021-05-06 14:53:02 [INFO] [TRAIN] epoch: 32, iter: 11590/40000, loss: 0.6332, lr: 0.007376, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 06:38:05 2021-05-06 14:53:11 [INFO] [TRAIN] epoch: 32, iter: 11600/40000, loss: 0.4637, lr: 0.007374, batch_cost: 0.8406, reader_cost: 0.00014, ips: 1.1896 samples/sec | ETA 06:37:53 2021-05-06 14:53:19 [INFO] [TRAIN] epoch: 32, iter: 11610/40000, loss: 0.5995, lr: 0.007372, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 06:37:53 2021-05-06 14:53:28 [INFO] [TRAIN] epoch: 32, iter: 11620/40000, loss: 0.3458, lr: 0.007370, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 06:37:32 2021-05-06 14:53:36 [INFO] [TRAIN] epoch: 32, iter: 11630/40000, loss: 0.1913, lr: 0.007367, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1879 samples/sec | ETA 06:38:03 2021-05-06 14:53:44 [INFO] [TRAIN] epoch: 32, iter: 11640/40000, loss: 0.4223, lr: 0.007365, batch_cost: 0.8411, reader_cost: 0.00017, ips: 1.1889 samples/sec | ETA 06:37:33 2021-05-06 14:53:53 [INFO] [TRAIN] epoch: 32, iter: 11650/40000, loss: 0.3720, lr: 0.007363, batch_cost: 0.8422, reader_cost: 0.00016, ips: 1.1874 samples/sec | ETA 06:37:56 2021-05-06 14:54:01 [INFO] [TRAIN] epoch: 32, iter: 11660/40000, loss: 0.6041, lr: 0.007360, batch_cost: 0.8408, reader_cost: 0.00016, ips: 1.1893 samples/sec | ETA 06:37:08 2021-05-06 14:54:10 [INFO] [TRAIN] epoch: 32, iter: 11670/40000, loss: 0.6044, lr: 0.007358, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 06:36:16 2021-05-06 14:54:18 [INFO] [TRAIN] epoch: 32, iter: 11680/40000, loss: 0.4427, lr: 0.007356, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 06:36:47 2021-05-06 14:54:26 [INFO] [TRAIN] epoch: 32, iter: 11690/40000, loss: 0.3060, lr: 0.007353, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1885 samples/sec | ETA 06:37:00 2021-05-06 14:54:35 [INFO] [TRAIN] epoch: 32, iter: 11700/40000, loss: 0.4282, lr: 0.007351, batch_cost: 0.8413, reader_cost: 0.00016, ips: 1.1886 samples/sec | ETA 06:36:49 2021-05-06 14:54:43 [INFO] [TRAIN] epoch: 32, iter: 11710/40000, loss: 0.4999, lr: 0.007349, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1882 samples/sec | ETA 06:36:48 2021-05-06 14:54:52 [INFO] [TRAIN] epoch: 32, iter: 11720/40000, loss: 0.6110, lr: 0.007346, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 06:35:54 2021-05-06 14:55:00 [INFO] [TRAIN] epoch: 32, iter: 11730/40000, loss: 0.6669, lr: 0.007344, batch_cost: 0.8406, reader_cost: 0.00014, ips: 1.1896 samples/sec | ETA 06:36:03 2021-05-06 14:55:08 [INFO] [TRAIN] epoch: 32, iter: 11740/40000, loss: 0.5670, lr: 0.007342, batch_cost: 0.8400, reader_cost: 0.00016, ips: 1.1905 samples/sec | ETA 06:35:38 2021-05-06 14:55:17 [INFO] [TRAIN] epoch: 32, iter: 11750/40000, loss: 0.4386, lr: 0.007340, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 06:35:36 2021-05-06 14:55:25 [INFO] [TRAIN] epoch: 32, iter: 11760/40000, loss: 0.3653, lr: 0.007337, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 06:35:31 2021-05-06 14:55:34 [INFO] [TRAIN] epoch: 32, iter: 11770/40000, loss: 0.4098, lr: 0.007335, batch_cost: 0.8402, reader_cost: 0.00017, ips: 1.1902 samples/sec | ETA 06:35:18 2021-05-06 14:55:42 [INFO] [TRAIN] epoch: 32, iter: 11780/40000, loss: 0.6718, lr: 0.007333, batch_cost: 0.8419, reader_cost: 0.00016, ips: 1.1877 samples/sec | ETA 06:35:59 2021-05-06 14:55:50 [INFO] [TRAIN] epoch: 32, iter: 11790/40000, loss: 0.4258, lr: 0.007330, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 06:34:36 2021-05-06 14:55:59 [INFO] [TRAIN] epoch: 32, iter: 11800/40000, loss: 0.5099, lr: 0.007328, batch_cost: 0.8401, reader_cost: 0.00017, ips: 1.1903 samples/sec | ETA 06:34:51 2021-05-06 14:56:07 [INFO] [TRAIN] epoch: 32, iter: 11810/40000, loss: 0.4388, lr: 0.007326, batch_cost: 0.8401, reader_cost: 0.00016, ips: 1.1903 samples/sec | ETA 06:34:43 2021-05-06 14:56:16 [INFO] [TRAIN] epoch: 32, iter: 11820/40000, loss: 0.4307, lr: 0.007323, batch_cost: 0.8422, reader_cost: 0.00016, ips: 1.1874 samples/sec | ETA 06:35:31 2021-05-06 14:56:24 [INFO] [TRAIN] epoch: 32, iter: 11830/40000, loss: 0.5510, lr: 0.007321, batch_cost: 0.8402, reader_cost: 0.00016, ips: 1.1903 samples/sec | ETA 06:34:27 2021-05-06 14:56:33 [INFO] [TRAIN] epoch: 32, iter: 11840/40000, loss: 0.3446, lr: 0.007319, batch_cost: 0.8390, reader_cost: 0.00015, ips: 1.1918 samples/sec | ETA 06:33:47 2021-05-06 14:56:41 [INFO] [TRAIN] epoch: 32, iter: 11850/40000, loss: 0.4002, lr: 0.007316, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 06:34:11 2021-05-06 14:56:49 [INFO] [TRAIN] epoch: 32, iter: 11860/40000, loss: 0.5240, lr: 0.007314, batch_cost: 0.8410, reader_cost: 0.00016, ips: 1.1890 samples/sec | ETA 06:34:27 2021-05-06 14:56:58 [INFO] [TRAIN] epoch: 32, iter: 11870/40000, loss: 0.4107, lr: 0.007312, batch_cost: 0.8427, reader_cost: 0.00016, ips: 1.1867 samples/sec | ETA 06:35:04 2021-05-06 14:57:06 [INFO] [TRAIN] epoch: 32, iter: 11880/40000, loss: 0.4821, lr: 0.007310, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 06:33:36 2021-05-06 14:57:15 [INFO] [TRAIN] epoch: 32, iter: 11890/40000, loss: 0.4636, lr: 0.007307, batch_cost: 0.8404, reader_cost: 0.00016, ips: 1.1899 samples/sec | ETA 06:33:44 2021-05-06 14:57:23 [INFO] [TRAIN] epoch: 32, iter: 11900/40000, loss: 0.4637, lr: 0.007305, batch_cost: 0.8383, reader_cost: 0.00011, ips: 1.1929 samples/sec | ETA 06:32:35 2021-05-06 14:57:35 [INFO] [TRAIN] epoch: 33, iter: 11910/40000, loss: 0.6511, lr: 0.007303, batch_cost: 1.1640, reader_cost: 0.20504, ips: 0.8591 samples/sec | ETA 09:04:56 2021-05-06 14:57:43 [INFO] [TRAIN] epoch: 33, iter: 11920/40000, loss: 0.4864, lr: 0.007300, batch_cost: 0.8416, reader_cost: 0.00016, ips: 1.1881 samples/sec | ETA 06:33:53 2021-05-06 14:57:51 [INFO] [TRAIN] epoch: 33, iter: 11930/40000, loss: 0.6540, lr: 0.007298, batch_cost: 0.8444, reader_cost: 0.00015, ips: 1.1843 samples/sec | ETA 06:35:01 2021-05-06 14:58:00 [INFO] [TRAIN] epoch: 33, iter: 11940/40000, loss: 0.4004, lr: 0.007296, batch_cost: 0.8416, reader_cost: 0.00014, ips: 1.1882 samples/sec | ETA 06:33:35 2021-05-06 14:58:08 [INFO] [TRAIN] epoch: 33, iter: 11950/40000, loss: 0.4629, lr: 0.007293, batch_cost: 0.8412, reader_cost: 0.00014, ips: 1.1888 samples/sec | ETA 06:33:14 2021-05-06 14:58:17 [INFO] [TRAIN] epoch: 33, iter: 11960/40000, loss: 0.5139, lr: 0.007291, batch_cost: 0.8434, reader_cost: 0.00015, ips: 1.1856 samples/sec | ETA 06:34:09 2021-05-06 14:58:25 [INFO] [TRAIN] epoch: 33, iter: 11970/40000, loss: 0.5939, lr: 0.007289, batch_cost: 0.8420, reader_cost: 0.00015, ips: 1.1877 samples/sec | ETA 06:33:20 2021-05-06 14:58:34 [INFO] [TRAIN] epoch: 33, iter: 11980/40000, loss: 0.6093, lr: 0.007286, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 06:32:17 2021-05-06 14:58:42 [INFO] [TRAIN] epoch: 33, iter: 11990/40000, loss: 0.4168, lr: 0.007284, batch_cost: 0.8433, reader_cost: 0.00017, ips: 1.1858 samples/sec | ETA 06:33:40 2021-05-06 14:58:50 [INFO] [TRAIN] epoch: 33, iter: 12000/40000, loss: 0.2404, lr: 0.007282, batch_cost: 0.8422, reader_cost: 0.00015, ips: 1.1873 samples/sec | ETA 06:33:02 2021-05-06 14:58:50 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 15:02:28 [INFO] [EVAL] #Images: 500 mIoU: 0.7302 Acc: 0.9515 Kappa: 0.9370 2021-05-06 15:02:28 [INFO] [EVAL] Class IoU: [0.9764 0.8204 0.9093 0.5817 0.5811 0.4871 0.5593 0.7025 0.9095 0.6477 0.9356 0.7556 0.4974 0.9318 0.7925 0.8152 0.7205 0.5533 0.6958] 2021-05-06 15:02:28 [INFO] [EVAL] Class Acc: [0.9906 0.893 0.9459 0.87 0.8017 0.7464 0.7703 0.8748 0.9399 0.8651 0.9538 0.8372 0.7412 0.9575 0.8776 0.9488 0.8196 0.7841 0.8173] 2021-05-06 15:03:20 [INFO] [EVAL] The model with the best validation mIoU (0.7302) was saved at iter 12000. 2021-05-06 15:03:28 [INFO] [TRAIN] epoch: 33, iter: 12010/40000, loss: 0.4509, lr: 0.007280, batch_cost: 0.8384, reader_cost: 0.00023, ips: 1.1928 samples/sec | ETA 06:31:06 2021-05-06 15:03:37 [INFO] [TRAIN] epoch: 33, iter: 12020/40000, loss: 0.3944, lr: 0.007277, batch_cost: 0.8382, reader_cost: 0.00015, ips: 1.1930 samples/sec | ETA 06:30:52 2021-05-06 15:03:45 [INFO] [TRAIN] epoch: 33, iter: 12030/40000, loss: 0.4481, lr: 0.007275, batch_cost: 0.8388, reader_cost: 0.00017, ips: 1.1922 samples/sec | ETA 06:31:00 2021-05-06 15:03:53 [INFO] [TRAIN] epoch: 33, iter: 12040/40000, loss: 0.5557, lr: 0.007273, batch_cost: 0.8394, reader_cost: 0.00016, ips: 1.1914 samples/sec | ETA 06:31:09 2021-05-06 15:04:02 [INFO] [TRAIN] epoch: 33, iter: 12050/40000, loss: 0.6045, lr: 0.007270, batch_cost: 0.8407, reader_cost: 0.00016, ips: 1.1895 samples/sec | ETA 06:31:37 2021-05-06 15:04:10 [INFO] [TRAIN] epoch: 33, iter: 12060/40000, loss: 0.3597, lr: 0.007268, batch_cost: 0.8422, reader_cost: 0.00015, ips: 1.1874 samples/sec | ETA 06:32:11 2021-05-06 15:04:19 [INFO] [TRAIN] epoch: 33, iter: 12070/40000, loss: 0.5152, lr: 0.007266, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 06:31:06 2021-05-06 15:04:27 [INFO] [TRAIN] epoch: 33, iter: 12080/40000, loss: 0.4689, lr: 0.007263, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 06:31:14 2021-05-06 15:04:35 [INFO] [TRAIN] epoch: 33, iter: 12090/40000, loss: 0.4711, lr: 0.007261, batch_cost: 0.8418, reader_cost: 0.00018, ips: 1.1879 samples/sec | ETA 06:31:35 2021-05-06 15:04:44 [INFO] [TRAIN] epoch: 33, iter: 12100/40000, loss: 0.6636, lr: 0.007259, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 06:30:51 2021-05-06 15:04:52 [INFO] [TRAIN] epoch: 33, iter: 12110/40000, loss: 0.7952, lr: 0.007256, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 06:30:39 2021-05-06 15:05:01 [INFO] [TRAIN] epoch: 33, iter: 12120/40000, loss: 0.7087, lr: 0.007254, batch_cost: 0.8394, reader_cost: 0.00014, ips: 1.1914 samples/sec | ETA 06:30:01 2021-05-06 15:05:09 [INFO] [TRAIN] epoch: 33, iter: 12130/40000, loss: 0.3137, lr: 0.007252, batch_cost: 0.8413, reader_cost: 0.00014, ips: 1.1887 samples/sec | ETA 06:30:46 2021-05-06 15:05:17 [INFO] [TRAIN] epoch: 33, iter: 12140/40000, loss: 0.4542, lr: 0.007250, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 06:29:57 2021-05-06 15:05:26 [INFO] [TRAIN] epoch: 33, iter: 12150/40000, loss: 0.2321, lr: 0.007247, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 06:30:24 2021-05-06 15:05:34 [INFO] [TRAIN] epoch: 33, iter: 12160/40000, loss: 0.4310, lr: 0.007245, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1903 samples/sec | ETA 06:29:49 2021-05-06 15:05:43 [INFO] [TRAIN] epoch: 33, iter: 12170/40000, loss: 0.4419, lr: 0.007243, batch_cost: 0.8406, reader_cost: 0.00016, ips: 1.1897 samples/sec | ETA 06:29:52 2021-05-06 15:05:51 [INFO] [TRAIN] epoch: 33, iter: 12180/40000, loss: 1.6248, lr: 0.007240, batch_cost: 0.8392, reader_cost: 0.00014, ips: 1.1916 samples/sec | ETA 06:29:06 2021-05-06 15:05:59 [INFO] [TRAIN] epoch: 33, iter: 12190/40000, loss: 0.5457, lr: 0.007238, batch_cost: 0.8387, reader_cost: 0.00014, ips: 1.1923 samples/sec | ETA 06:28:44 2021-05-06 15:06:08 [INFO] [TRAIN] epoch: 33, iter: 12200/40000, loss: 0.6931, lr: 0.007236, batch_cost: 0.8383, reader_cost: 0.00016, ips: 1.1928 samples/sec | ETA 06:28:25 2021-05-06 15:06:16 [INFO] [TRAIN] epoch: 33, iter: 12210/40000, loss: 0.3745, lr: 0.007233, batch_cost: 0.8391, reader_cost: 0.00014, ips: 1.1918 samples/sec | ETA 06:28:37 2021-05-06 15:06:25 [INFO] [TRAIN] epoch: 33, iter: 12220/40000, loss: 0.3551, lr: 0.007231, batch_cost: 0.8392, reader_cost: 0.00015, ips: 1.1916 samples/sec | ETA 06:28:32 2021-05-06 15:06:33 [INFO] [TRAIN] epoch: 33, iter: 12230/40000, loss: 0.6923, lr: 0.007229, batch_cost: 0.8387, reader_cost: 0.00016, ips: 1.1923 samples/sec | ETA 06:28:11 2021-05-06 15:06:41 [INFO] [TRAIN] epoch: 33, iter: 12240/40000, loss: 0.3893, lr: 0.007226, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1883 samples/sec | ETA 06:29:21 2021-05-06 15:06:50 [INFO] [TRAIN] epoch: 33, iter: 12250/40000, loss: 0.6012, lr: 0.007224, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 06:28:57 2021-05-06 15:06:58 [INFO] [TRAIN] epoch: 33, iter: 12260/40000, loss: 0.4572, lr: 0.007222, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 06:28:20 2021-05-06 15:07:07 [INFO] [TRAIN] epoch: 33, iter: 12270/40000, loss: 0.6406, lr: 0.007220, batch_cost: 0.8393, reader_cost: 0.00012, ips: 1.1915 samples/sec | ETA 06:27:53 2021-05-06 15:07:18 [INFO] [TRAIN] epoch: 34, iter: 12280/40000, loss: 0.4876, lr: 0.007217, batch_cost: 1.1651, reader_cost: 0.23559, ips: 0.8583 samples/sec | ETA 08:58:17 2021-05-06 15:07:27 [INFO] [TRAIN] epoch: 34, iter: 12290/40000, loss: 0.3667, lr: 0.007215, batch_cost: 0.8449, reader_cost: 0.00032, ips: 1.1836 samples/sec | ETA 06:30:10 2021-05-06 15:07:35 [INFO] [TRAIN] epoch: 34, iter: 12300/40000, loss: 0.5918, lr: 0.007213, batch_cost: 0.8421, reader_cost: 0.00015, ips: 1.1875 samples/sec | ETA 06:28:46 2021-05-06 15:07:44 [INFO] [TRAIN] epoch: 34, iter: 12310/40000, loss: 0.4372, lr: 0.007210, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 06:27:56 2021-05-06 15:07:52 [INFO] [TRAIN] epoch: 34, iter: 12320/40000, loss: 0.3403, lr: 0.007208, batch_cost: 0.8408, reader_cost: 0.00016, ips: 1.1894 samples/sec | ETA 06:27:52 2021-05-06 15:08:00 [INFO] [TRAIN] epoch: 34, iter: 12330/40000, loss: 0.3934, lr: 0.007206, batch_cost: 0.8427, reader_cost: 0.00015, ips: 1.1866 samples/sec | ETA 06:28:38 2021-05-06 15:08:09 [INFO] [TRAIN] epoch: 34, iter: 12340/40000, loss: 0.7481, lr: 0.007203, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 06:27:09 2021-05-06 15:08:17 [INFO] [TRAIN] epoch: 34, iter: 12350/40000, loss: 0.5915, lr: 0.007201, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1886 samples/sec | ETA 06:27:42 2021-05-06 15:08:26 [INFO] [TRAIN] epoch: 34, iter: 12360/40000, loss: 0.6319, lr: 0.007199, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 06:27:16 2021-05-06 15:08:34 [INFO] [TRAIN] epoch: 34, iter: 12370/40000, loss: 0.3097, lr: 0.007196, batch_cost: 0.8397, reader_cost: 0.00016, ips: 1.1910 samples/sec | ETA 06:26:39 2021-05-06 15:08:42 [INFO] [TRAIN] epoch: 34, iter: 12380/40000, loss: 0.2670, lr: 0.007194, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1914 samples/sec | ETA 06:26:22 2021-05-06 15:08:51 [INFO] [TRAIN] epoch: 34, iter: 12390/40000, loss: 0.4018, lr: 0.007192, batch_cost: 0.8407, reader_cost: 0.00016, ips: 1.1894 samples/sec | ETA 06:26:52 2021-05-06 15:08:59 [INFO] [TRAIN] epoch: 34, iter: 12400/40000, loss: 0.5118, lr: 0.007189, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1917 samples/sec | ETA 06:25:59 2021-05-06 15:09:08 [INFO] [TRAIN] epoch: 34, iter: 12410/40000, loss: 0.5355, lr: 0.007187, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1911 samples/sec | ETA 06:26:03 2021-05-06 15:09:16 [INFO] [TRAIN] epoch: 34, iter: 12420/40000, loss: 0.5432, lr: 0.007185, batch_cost: 0.8406, reader_cost: 0.00016, ips: 1.1897 samples/sec | ETA 06:26:23 2021-05-06 15:09:24 [INFO] [TRAIN] epoch: 34, iter: 12430/40000, loss: 0.4286, lr: 0.007183, batch_cost: 0.8388, reader_cost: 0.00015, ips: 1.1922 samples/sec | ETA 06:25:25 2021-05-06 15:09:33 [INFO] [TRAIN] epoch: 34, iter: 12440/40000, loss: 0.3702, lr: 0.007180, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 06:25:54 2021-05-06 15:09:41 [INFO] [TRAIN] epoch: 34, iter: 12450/40000, loss: 0.3193, lr: 0.007178, batch_cost: 0.8399, reader_cost: 0.00014, ips: 1.1907 samples/sec | ETA 06:25:37 2021-05-06 15:09:50 [INFO] [TRAIN] epoch: 34, iter: 12460/40000, loss: 0.6021, lr: 0.007176, batch_cost: 0.8388, reader_cost: 0.00014, ips: 1.1921 samples/sec | ETA 06:25:01 2021-05-06 15:09:58 [INFO] [TRAIN] epoch: 34, iter: 12470/40000, loss: 0.7653, lr: 0.007173, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 06:25:25 2021-05-06 15:10:06 [INFO] [TRAIN] epoch: 34, iter: 12480/40000, loss: 0.2939, lr: 0.007171, batch_cost: 0.8408, reader_cost: 0.00017, ips: 1.1894 samples/sec | ETA 06:25:38 2021-05-06 15:10:15 [INFO] [TRAIN] epoch: 34, iter: 12490/40000, loss: 0.4242, lr: 0.007169, batch_cost: 0.8422, reader_cost: 0.00014, ips: 1.1874 samples/sec | ETA 06:26:08 2021-05-06 15:10:23 [INFO] [TRAIN] epoch: 34, iter: 12500/40000, loss: 0.2596, lr: 0.007166, batch_cost: 0.8408, reader_cost: 0.00014, ips: 1.1894 samples/sec | ETA 06:25:20 2021-05-06 15:10:32 [INFO] [TRAIN] epoch: 34, iter: 12510/40000, loss: 0.2388, lr: 0.007164, batch_cost: 0.8414, reader_cost: 0.00014, ips: 1.1885 samples/sec | ETA 06:25:29 2021-05-06 15:10:40 [INFO] [TRAIN] epoch: 34, iter: 12520/40000, loss: 0.3343, lr: 0.007162, batch_cost: 0.8403, reader_cost: 0.00014, ips: 1.1901 samples/sec | ETA 06:24:50 2021-05-06 15:10:48 [INFO] [TRAIN] epoch: 34, iter: 12530/40000, loss: 0.3890, lr: 0.007159, batch_cost: 0.8404, reader_cost: 0.00014, ips: 1.1899 samples/sec | ETA 06:24:45 2021-05-06 15:10:57 [INFO] [TRAIN] epoch: 34, iter: 12540/40000, loss: 0.3614, lr: 0.007157, batch_cost: 0.8411, reader_cost: 0.00014, ips: 1.1889 samples/sec | ETA 06:24:56 2021-05-06 15:11:05 [INFO] [TRAIN] epoch: 34, iter: 12550/40000, loss: 0.2531, lr: 0.007155, batch_cost: 0.8405, reader_cost: 0.00014, ips: 1.1897 samples/sec | ETA 06:24:32 2021-05-06 15:11:14 [INFO] [TRAIN] epoch: 34, iter: 12560/40000, loss: 0.4852, lr: 0.007152, batch_cost: 0.8413, reader_cost: 0.00014, ips: 1.1887 samples/sec | ETA 06:24:44 2021-05-06 15:11:22 [INFO] [TRAIN] epoch: 34, iter: 12570/40000, loss: 0.7802, lr: 0.007150, batch_cost: 0.8410, reader_cost: 0.00016, ips: 1.1890 samples/sec | ETA 06:24:29 2021-05-06 15:11:30 [INFO] [TRAIN] epoch: 34, iter: 12580/40000, loss: 0.3924, lr: 0.007148, batch_cost: 0.8420, reader_cost: 0.00015, ips: 1.1876 samples/sec | ETA 06:24:47 2021-05-06 15:11:39 [INFO] [TRAIN] epoch: 34, iter: 12590/40000, loss: 0.3285, lr: 0.007146, batch_cost: 0.8406, reader_cost: 0.00016, ips: 1.1896 samples/sec | ETA 06:24:00 2021-05-06 15:11:47 [INFO] [TRAIN] epoch: 34, iter: 12600/40000, loss: 0.6732, lr: 0.007143, batch_cost: 0.8395, reader_cost: 0.00014, ips: 1.1912 samples/sec | ETA 06:23:21 2021-05-06 15:11:56 [INFO] [TRAIN] epoch: 34, iter: 12610/40000, loss: 0.4018, lr: 0.007141, batch_cost: 0.8415, reader_cost: 0.00017, ips: 1.1884 samples/sec | ETA 06:24:08 2021-05-06 15:12:04 [INFO] [TRAIN] epoch: 34, iter: 12620/40000, loss: 0.3652, lr: 0.007139, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 06:23:28 2021-05-06 15:12:13 [INFO] [TRAIN] epoch: 34, iter: 12630/40000, loss: 0.3294, lr: 0.007136, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 06:23:02 2021-05-06 15:12:21 [INFO] [TRAIN] epoch: 34, iter: 12640/40000, loss: 0.4367, lr: 0.007134, batch_cost: 0.8400, reader_cost: 0.00013, ips: 1.1905 samples/sec | ETA 06:23:02 2021-05-06 15:12:32 [INFO] [TRAIN] epoch: 35, iter: 12650/40000, loss: 0.4311, lr: 0.007132, batch_cost: 1.1430, reader_cost: 0.20259, ips: 0.8749 samples/sec | ETA 08:40:59 2021-05-06 15:12:41 [INFO] [TRAIN] epoch: 35, iter: 12660/40000, loss: 0.3335, lr: 0.007129, batch_cost: 0.8575, reader_cost: 0.00033, ips: 1.1662 samples/sec | ETA 06:30:43 2021-05-06 15:12:49 [INFO] [TRAIN] epoch: 35, iter: 12670/40000, loss: 0.4606, lr: 0.007127, batch_cost: 0.8418, reader_cost: 0.00014, ips: 1.1880 samples/sec | ETA 06:23:25 2021-05-06 15:12:58 [INFO] [TRAIN] epoch: 35, iter: 12680/40000, loss: 0.5243, lr: 0.007125, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 06:22:49 2021-05-06 15:13:06 [INFO] [TRAIN] epoch: 35, iter: 12690/40000, loss: 0.2554, lr: 0.007122, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 06:22:44 2021-05-06 15:13:15 [INFO] [TRAIN] epoch: 35, iter: 12700/40000, loss: 0.3953, lr: 0.007120, batch_cost: 0.8438, reader_cost: 0.00014, ips: 1.1851 samples/sec | ETA 06:23:55 2021-05-06 15:13:23 [INFO] [TRAIN] epoch: 35, iter: 12710/40000, loss: 0.8055, lr: 0.007118, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1887 samples/sec | ETA 06:22:38 2021-05-06 15:13:31 [INFO] [TRAIN] epoch: 35, iter: 12720/40000, loss: 0.3900, lr: 0.007115, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 06:22:07 2021-05-06 15:13:40 [INFO] [TRAIN] epoch: 35, iter: 12730/40000, loss: 0.5717, lr: 0.007113, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 06:22:09 2021-05-06 15:13:48 [INFO] [TRAIN] epoch: 35, iter: 12740/40000, loss: 0.3978, lr: 0.007111, batch_cost: 0.8414, reader_cost: 0.00014, ips: 1.1884 samples/sec | ETA 06:22:17 2021-05-06 15:13:57 [INFO] [TRAIN] epoch: 35, iter: 12750/40000, loss: 0.2358, lr: 0.007109, batch_cost: 0.8398, reader_cost: 0.00014, ips: 1.1908 samples/sec | ETA 06:21:24 2021-05-06 15:14:05 [INFO] [TRAIN] epoch: 35, iter: 12760/40000, loss: 0.2958, lr: 0.007106, batch_cost: 0.8396, reader_cost: 0.00014, ips: 1.1910 samples/sec | ETA 06:21:11 2021-05-06 15:14:13 [INFO] [TRAIN] epoch: 35, iter: 12770/40000, loss: 0.4699, lr: 0.007104, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 06:21:44 2021-05-06 15:14:22 [INFO] [TRAIN] epoch: 35, iter: 12780/40000, loss: 0.5065, lr: 0.007102, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1912 samples/sec | ETA 06:20:50 2021-05-06 15:14:30 [INFO] [TRAIN] epoch: 35, iter: 12790/40000, loss: 0.5743, lr: 0.007099, batch_cost: 0.8387, reader_cost: 0.00015, ips: 1.1923 samples/sec | ETA 06:20:21 2021-05-06 15:14:39 [INFO] [TRAIN] epoch: 35, iter: 12800/40000, loss: 0.3613, lr: 0.007097, batch_cost: 0.8397, reader_cost: 0.00016, ips: 1.1908 samples/sec | ETA 06:20:41 2021-05-06 15:14:47 [INFO] [TRAIN] epoch: 35, iter: 12810/40000, loss: 0.2623, lr: 0.007095, batch_cost: 0.8426, reader_cost: 0.00016, ips: 1.1868 samples/sec | ETA 06:21:49 2021-05-06 15:14:55 [INFO] [TRAIN] epoch: 35, iter: 12820/40000, loss: 0.3498, lr: 0.007092, batch_cost: 0.8394, reader_cost: 0.00016, ips: 1.1914 samples/sec | ETA 06:20:13 2021-05-06 15:15:04 [INFO] [TRAIN] epoch: 35, iter: 12830/40000, loss: 0.9969, lr: 0.007090, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 06:20:44 2021-05-06 15:15:12 [INFO] [TRAIN] epoch: 35, iter: 12840/40000, loss: 0.7395, lr: 0.007088, batch_cost: 0.8388, reader_cost: 0.00016, ips: 1.1921 samples/sec | ETA 06:19:43 2021-05-06 15:15:21 [INFO] [TRAIN] epoch: 35, iter: 12850/40000, loss: 0.4555, lr: 0.007085, batch_cost: 0.8415, reader_cost: 0.00016, ips: 1.1883 samples/sec | ETA 06:20:47 2021-05-06 15:15:29 [INFO] [TRAIN] epoch: 35, iter: 12860/40000, loss: 0.5036, lr: 0.007083, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 06:19:46 2021-05-06 15:15:37 [INFO] [TRAIN] epoch: 35, iter: 12870/40000, loss: 0.4273, lr: 0.007081, batch_cost: 0.8423, reader_cost: 0.00016, ips: 1.1873 samples/sec | ETA 06:20:50 2021-05-06 15:15:46 [INFO] [TRAIN] epoch: 35, iter: 12880/40000, loss: 0.2016, lr: 0.007078, batch_cost: 0.8421, reader_cost: 0.00015, ips: 1.1875 samples/sec | ETA 06:20:38 2021-05-06 15:15:54 [INFO] [TRAIN] epoch: 35, iter: 12890/40000, loss: 0.3635, lr: 0.007076, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 06:19:54 2021-05-06 15:16:03 [INFO] [TRAIN] epoch: 35, iter: 12900/40000, loss: 0.2537, lr: 0.007074, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 06:19:04 2021-05-06 15:16:11 [INFO] [TRAIN] epoch: 35, iter: 12910/40000, loss: 0.5070, lr: 0.007071, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 06:19:09 2021-05-06 15:16:20 [INFO] [TRAIN] epoch: 35, iter: 12920/40000, loss: 0.4359, lr: 0.007069, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 06:19:36 2021-05-06 15:16:28 [INFO] [TRAIN] epoch: 35, iter: 12930/40000, loss: 0.3658, lr: 0.007067, batch_cost: 0.8413, reader_cost: 0.00016, ips: 1.1886 samples/sec | ETA 06:19:33 2021-05-06 15:16:36 [INFO] [TRAIN] epoch: 35, iter: 12940/40000, loss: 0.4373, lr: 0.007065, batch_cost: 0.8409, reader_cost: 0.00016, ips: 1.1892 samples/sec | ETA 06:19:15 2021-05-06 15:16:45 [INFO] [TRAIN] epoch: 35, iter: 12950/40000, loss: 0.3745, lr: 0.007062, batch_cost: 0.8412, reader_cost: 0.00016, ips: 1.1887 samples/sec | ETA 06:19:15 2021-05-06 15:16:53 [INFO] [TRAIN] epoch: 35, iter: 12960/40000, loss: 0.4050, lr: 0.007060, batch_cost: 0.8421, reader_cost: 0.00015, ips: 1.1875 samples/sec | ETA 06:19:30 2021-05-06 15:17:02 [INFO] [TRAIN] epoch: 35, iter: 12970/40000, loss: 0.5125, lr: 0.007058, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1913 samples/sec | ETA 06:18:10 2021-05-06 15:17:10 [INFO] [TRAIN] epoch: 35, iter: 12980/40000, loss: 0.4566, lr: 0.007055, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1883 samples/sec | ETA 06:18:58 2021-05-06 15:17:18 [INFO] [TRAIN] epoch: 35, iter: 12990/40000, loss: 0.3485, lr: 0.007053, batch_cost: 0.8412, reader_cost: 0.00016, ips: 1.1887 samples/sec | ETA 06:18:42 2021-05-06 15:17:27 [INFO] [TRAIN] epoch: 35, iter: 13000/40000, loss: 0.3312, lr: 0.007051, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 06:17:56 2021-05-06 15:17:27 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 15:21:04 [INFO] [EVAL] #Images: 500 mIoU: 0.7162 Acc: 0.9443 Kappa: 0.9281 2021-05-06 15:21:04 [INFO] [EVAL] Class IoU: [0.9603 0.749 0.9103 0.5294 0.5411 0.4763 0.5538 0.6933 0.9105 0.6478 0.9316 0.7557 0.52 0.9245 0.7738 0.7916 0.6906 0.5504 0.6971] 2021-05-06 15:21:04 [INFO] [EVAL] Class Acc: [0.9965 0.7831 0.9445 0.8796 0.6702 0.7788 0.7642 0.8613 0.9489 0.8498 0.9467 0.8481 0.7153 0.9426 0.87 0.9523 0.7835 0.8498 0.7988] 2021-05-06 15:21:33 [INFO] [EVAL] The model with the best validation mIoU (0.7302) was saved at iter 12000. 2021-05-06 15:21:41 [INFO] [TRAIN] epoch: 35, iter: 13010/40000, loss: 0.5054, lr: 0.007048, batch_cost: 0.8353, reader_cost: 0.00029, ips: 1.1972 samples/sec | ETA 06:15:43 2021-05-06 15:21:50 [INFO] [TRAIN] epoch: 35, iter: 13020/40000, loss: 0.5150, lr: 0.007046, batch_cost: 0.8358, reader_cost: 0.00010, ips: 1.1964 samples/sec | ETA 06:15:51 2021-05-06 15:22:02 [INFO] [TRAIN] epoch: 36, iter: 13030/40000, loss: 0.4585, lr: 0.007044, batch_cost: 1.1926, reader_cost: 0.22053, ips: 0.8385 samples/sec | ETA 08:56:05 2021-05-06 15:22:10 [INFO] [TRAIN] epoch: 36, iter: 13040/40000, loss: 0.4120, lr: 0.007041, batch_cost: 0.8405, reader_cost: 0.00031, ips: 1.1898 samples/sec | ETA 06:17:39 2021-05-06 15:22:18 [INFO] [TRAIN] epoch: 36, iter: 13050/40000, loss: 0.6196, lr: 0.007039, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1886 samples/sec | ETA 06:17:52 2021-05-06 15:22:27 [INFO] [TRAIN] epoch: 36, iter: 13060/40000, loss: 0.2681, lr: 0.007037, batch_cost: 0.8419, reader_cost: 0.00015, ips: 1.1877 samples/sec | ETA 06:18:01 2021-05-06 15:22:35 [INFO] [TRAIN] epoch: 36, iter: 13070/40000, loss: 0.5396, lr: 0.007034, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1888 samples/sec | ETA 06:17:32 2021-05-06 15:22:44 [INFO] [TRAIN] epoch: 36, iter: 13080/40000, loss: 0.5878, lr: 0.007032, batch_cost: 0.8425, reader_cost: 0.00015, ips: 1.1869 samples/sec | ETA 06:18:00 2021-05-06 15:22:52 [INFO] [TRAIN] epoch: 36, iter: 13090/40000, loss: 0.5303, lr: 0.007030, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 06:16:52 2021-05-06 15:23:00 [INFO] [TRAIN] epoch: 36, iter: 13100/40000, loss: 0.5682, lr: 0.007027, batch_cost: 0.8389, reader_cost: 0.00016, ips: 1.1921 samples/sec | ETA 06:16:05 2021-05-06 15:23:09 [INFO] [TRAIN] epoch: 36, iter: 13110/40000, loss: 0.5064, lr: 0.007025, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 06:16:43 2021-05-06 15:23:17 [INFO] [TRAIN] epoch: 36, iter: 13120/40000, loss: 0.1574, lr: 0.007023, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 06:16:22 2021-05-06 15:23:26 [INFO] [TRAIN] epoch: 36, iter: 13130/40000, loss: 0.4184, lr: 0.007020, batch_cost: 0.8399, reader_cost: 0.00016, ips: 1.1907 samples/sec | ETA 06:16:07 2021-05-06 15:23:34 [INFO] [TRAIN] epoch: 36, iter: 13140/40000, loss: 0.4461, lr: 0.007018, batch_cost: 0.8417, reader_cost: 0.00016, ips: 1.1881 samples/sec | ETA 06:16:47 2021-05-06 15:23:43 [INFO] [TRAIN] epoch: 36, iter: 13150/40000, loss: 0.5015, lr: 0.007016, batch_cost: 0.8406, reader_cost: 0.00018, ips: 1.1896 samples/sec | ETA 06:16:10 2021-05-06 15:23:51 [INFO] [TRAIN] epoch: 36, iter: 13160/40000, loss: 0.6304, lr: 0.007014, batch_cost: 0.8404, reader_cost: 0.00018, ips: 1.1898 samples/sec | ETA 06:15:57 2021-05-06 15:23:59 [INFO] [TRAIN] epoch: 36, iter: 13170/40000, loss: 0.5680, lr: 0.007011, batch_cost: 0.8412, reader_cost: 0.00016, ips: 1.1888 samples/sec | ETA 06:16:08 2021-05-06 15:24:08 [INFO] [TRAIN] epoch: 36, iter: 13180/40000, loss: 0.3218, lr: 0.007009, batch_cost: 0.8394, reader_cost: 0.00014, ips: 1.1913 samples/sec | ETA 06:15:12 2021-05-06 15:24:16 [INFO] [TRAIN] epoch: 36, iter: 13190/40000, loss: 0.4287, lr: 0.007007, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 06:15:11 2021-05-06 15:24:25 [INFO] [TRAIN] epoch: 36, iter: 13200/40000, loss: 0.3885, lr: 0.007004, batch_cost: 0.8382, reader_cost: 0.00014, ips: 1.1930 samples/sec | ETA 06:14:24 2021-05-06 15:24:33 [INFO] [TRAIN] epoch: 36, iter: 13210/40000, loss: 0.4854, lr: 0.007002, batch_cost: 0.8395, reader_cost: 0.00014, ips: 1.1912 samples/sec | ETA 06:14:49 2021-05-06 15:24:41 [INFO] [TRAIN] epoch: 36, iter: 13220/40000, loss: 0.7806, lr: 0.007000, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 06:14:50 2021-05-06 15:24:50 [INFO] [TRAIN] epoch: 36, iter: 13230/40000, loss: 0.5772, lr: 0.006997, batch_cost: 0.8398, reader_cost: 0.00014, ips: 1.1907 samples/sec | ETA 06:14:42 2021-05-06 15:24:58 [INFO] [TRAIN] epoch: 36, iter: 13240/40000, loss: 0.3825, lr: 0.006995, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 06:15:03 2021-05-06 15:25:07 [INFO] [TRAIN] epoch: 36, iter: 13250/40000, loss: 0.2903, lr: 0.006993, batch_cost: 0.8394, reader_cost: 0.00016, ips: 1.1914 samples/sec | ETA 06:14:13 2021-05-06 15:25:15 [INFO] [TRAIN] epoch: 36, iter: 13260/40000, loss: 0.5044, lr: 0.006990, batch_cost: 0.8400, reader_cost: 0.00014, ips: 1.1905 samples/sec | ETA 06:14:21 2021-05-06 15:25:23 [INFO] [TRAIN] epoch: 36, iter: 13270/40000, loss: 0.3260, lr: 0.006988, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1884 samples/sec | ETA 06:14:52 2021-05-06 15:25:32 [INFO] [TRAIN] epoch: 36, iter: 13280/40000, loss: 0.3303, lr: 0.006986, batch_cost: 0.8396, reader_cost: 0.00016, ips: 1.1910 samples/sec | ETA 06:13:55 2021-05-06 15:25:40 [INFO] [TRAIN] epoch: 36, iter: 13290/40000, loss: 0.3229, lr: 0.006983, batch_cost: 0.8400, reader_cost: 0.00016, ips: 1.1904 samples/sec | ETA 06:13:57 2021-05-06 15:25:49 [INFO] [TRAIN] epoch: 36, iter: 13300/40000, loss: 0.6424, lr: 0.006981, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 06:13:56 2021-05-06 15:25:57 [INFO] [TRAIN] epoch: 36, iter: 13310/40000, loss: 0.4434, lr: 0.006979, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 06:13:59 2021-05-06 15:26:05 [INFO] [TRAIN] epoch: 36, iter: 13320/40000, loss: 0.4449, lr: 0.006976, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 06:13:11 2021-05-06 15:26:14 [INFO] [TRAIN] epoch: 36, iter: 13330/40000, loss: 0.2975, lr: 0.006974, batch_cost: 0.8392, reader_cost: 0.00015, ips: 1.1916 samples/sec | ETA 06:13:01 2021-05-06 15:26:22 [INFO] [TRAIN] epoch: 36, iter: 13340/40000, loss: 0.6078, lr: 0.006972, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 06:13:10 2021-05-06 15:26:31 [INFO] [TRAIN] epoch: 36, iter: 13350/40000, loss: 0.5093, lr: 0.006969, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 06:13:27 2021-05-06 15:26:39 [INFO] [TRAIN] epoch: 36, iter: 13360/40000, loss: 0.3379, lr: 0.006967, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 06:13:27 2021-05-06 15:26:47 [INFO] [TRAIN] epoch: 36, iter: 13370/40000, loss: 0.7969, lr: 0.006965, batch_cost: 0.8410, reader_cost: 0.00014, ips: 1.1891 samples/sec | ETA 06:13:14 2021-05-06 15:26:56 [INFO] [TRAIN] epoch: 36, iter: 13380/40000, loss: 0.4491, lr: 0.006963, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 06:13:06 2021-05-06 15:27:04 [INFO] [TRAIN] epoch: 36, iter: 13390/40000, loss: 0.3529, lr: 0.006960, batch_cost: 0.8394, reader_cost: 0.00028, ips: 1.1913 samples/sec | ETA 06:12:16 2021-05-06 15:27:16 [INFO] [TRAIN] epoch: 37, iter: 13400/40000, loss: 0.5763, lr: 0.006958, batch_cost: 1.1732, reader_cost: 0.21445, ips: 0.8524 samples/sec | ETA 08:40:05 2021-05-06 15:27:24 [INFO] [TRAIN] epoch: 37, iter: 13410/40000, loss: 0.4464, lr: 0.006956, batch_cost: 0.8412, reader_cost: 0.00032, ips: 1.1888 samples/sec | ETA 06:12:46 2021-05-06 15:27:33 [INFO] [TRAIN] epoch: 37, iter: 13420/40000, loss: 0.4936, lr: 0.006953, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 06:12:21 2021-05-06 15:27:41 [INFO] [TRAIN] epoch: 37, iter: 13430/40000, loss: 0.3954, lr: 0.006951, batch_cost: 0.8398, reader_cost: 0.00016, ips: 1.1907 samples/sec | ETA 06:11:53 2021-05-06 15:27:50 [INFO] [TRAIN] epoch: 37, iter: 13440/40000, loss: 0.5163, lr: 0.006949, batch_cost: 0.8400, reader_cost: 0.00016, ips: 1.1905 samples/sec | ETA 06:11:50 2021-05-06 15:27:58 [INFO] [TRAIN] epoch: 37, iter: 13450/40000, loss: 0.5424, lr: 0.006946, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 06:12:08 2021-05-06 15:28:06 [INFO] [TRAIN] epoch: 37, iter: 13460/40000, loss: 0.9525, lr: 0.006944, batch_cost: 0.8418, reader_cost: 0.00016, ips: 1.1880 samples/sec | ETA 06:12:20 2021-05-06 15:28:15 [INFO] [TRAIN] epoch: 37, iter: 13470/40000, loss: 1.1123, lr: 0.006942, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 06:11:27 2021-05-06 15:28:23 [INFO] [TRAIN] epoch: 37, iter: 13480/40000, loss: 0.5885, lr: 0.006939, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1885 samples/sec | ETA 06:11:53 2021-05-06 15:28:32 [INFO] [TRAIN] epoch: 37, iter: 13490/40000, loss: 0.2714, lr: 0.006937, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 06:11:36 2021-05-06 15:28:40 [INFO] [TRAIN] epoch: 37, iter: 13500/40000, loss: 0.3546, lr: 0.006935, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 06:10:41 2021-05-06 15:28:48 [INFO] [TRAIN] epoch: 37, iter: 13510/40000, loss: 0.5223, lr: 0.006932, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1918 samples/sec | ETA 06:10:27 2021-05-06 15:28:57 [INFO] [TRAIN] epoch: 37, iter: 13520/40000, loss: 0.5377, lr: 0.006930, batch_cost: 0.8409, reader_cost: 0.00016, ips: 1.1892 samples/sec | ETA 06:11:07 2021-05-06 15:29:05 [INFO] [TRAIN] epoch: 37, iter: 13530/40000, loss: 0.7399, lr: 0.006928, batch_cost: 0.8406, reader_cost: 0.00014, ips: 1.1896 samples/sec | ETA 06:10:51 2021-05-06 15:29:14 [INFO] [TRAIN] epoch: 37, iter: 13540/40000, loss: 0.6546, lr: 0.006925, batch_cost: 0.8423, reader_cost: 0.00014, ips: 1.1873 samples/sec | ETA 06:11:26 2021-05-06 15:29:22 [INFO] [TRAIN] epoch: 37, iter: 13550/40000, loss: 0.5682, lr: 0.006923, batch_cost: 0.8420, reader_cost: 0.00015, ips: 1.1877 samples/sec | ETA 06:11:10 2021-05-06 15:29:30 [INFO] [TRAIN] epoch: 37, iter: 13560/40000, loss: 0.4034, lr: 0.006921, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1880 samples/sec | ETA 06:10:55 2021-05-06 15:29:39 [INFO] [TRAIN] epoch: 37, iter: 13570/40000, loss: 0.5212, lr: 0.006918, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 06:10:19 2021-05-06 15:29:47 [INFO] [TRAIN] epoch: 37, iter: 13580/40000, loss: 0.4050, lr: 0.006916, batch_cost: 0.8398, reader_cost: 0.00014, ips: 1.1908 samples/sec | ETA 06:09:46 2021-05-06 15:29:56 [INFO] [TRAIN] epoch: 37, iter: 13590/40000, loss: 0.3813, lr: 0.006914, batch_cost: 0.8405, reader_cost: 0.00014, ips: 1.1898 samples/sec | ETA 06:09:56 2021-05-06 15:30:04 [INFO] [TRAIN] epoch: 37, iter: 13600/40000, loss: 0.4756, lr: 0.006911, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1879 samples/sec | ETA 06:10:23 2021-05-06 15:30:12 [INFO] [TRAIN] epoch: 37, iter: 13610/40000, loss: 0.5554, lr: 0.006909, batch_cost: 0.8428, reader_cost: 0.00015, ips: 1.1865 samples/sec | ETA 06:10:41 2021-05-06 15:30:21 [INFO] [TRAIN] epoch: 37, iter: 13620/40000, loss: 0.2096, lr: 0.006907, batch_cost: 0.8405, reader_cost: 0.00014, ips: 1.1897 samples/sec | ETA 06:09:33 2021-05-06 15:30:29 [INFO] [TRAIN] epoch: 37, iter: 13630/40000, loss: 0.3241, lr: 0.006904, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1882 samples/sec | ETA 06:09:52 2021-05-06 15:30:38 [INFO] [TRAIN] epoch: 37, iter: 13640/40000, loss: 0.4196, lr: 0.006902, batch_cost: 0.8407, reader_cost: 0.00016, ips: 1.1895 samples/sec | ETA 06:09:20 2021-05-06 15:30:46 [INFO] [TRAIN] epoch: 37, iter: 13650/40000, loss: 0.4171, lr: 0.006900, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 06:08:47 2021-05-06 15:30:55 [INFO] [TRAIN] epoch: 37, iter: 13660/40000, loss: 0.4415, lr: 0.006898, batch_cost: 0.8399, reader_cost: 0.00016, ips: 1.1907 samples/sec | ETA 06:08:42 2021-05-06 15:31:03 [INFO] [TRAIN] epoch: 37, iter: 13670/40000, loss: 0.2665, lr: 0.006895, batch_cost: 0.8400, reader_cost: 0.00016, ips: 1.1905 samples/sec | ETA 06:08:37 2021-05-06 15:31:11 [INFO] [TRAIN] epoch: 37, iter: 13680/40000, loss: 0.4273, lr: 0.006893, batch_cost: 0.8429, reader_cost: 0.00014, ips: 1.1864 samples/sec | ETA 06:09:45 2021-05-06 15:31:20 [INFO] [TRAIN] epoch: 37, iter: 13690/40000, loss: 0.4906, lr: 0.006891, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 06:08:35 2021-05-06 15:31:28 [INFO] [TRAIN] epoch: 37, iter: 13700/40000, loss: 0.3493, lr: 0.006888, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 06:08:31 2021-05-06 15:31:37 [INFO] [TRAIN] epoch: 37, iter: 13710/40000, loss: 0.3209, lr: 0.006886, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1881 samples/sec | ETA 06:08:47 2021-05-06 15:31:45 [INFO] [TRAIN] epoch: 37, iter: 13720/40000, loss: 0.6922, lr: 0.006884, batch_cost: 0.8411, reader_cost: 0.00017, ips: 1.1889 samples/sec | ETA 06:08:24 2021-05-06 15:31:53 [INFO] [TRAIN] epoch: 37, iter: 13730/40000, loss: 0.4727, lr: 0.006881, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 06:08:07 2021-05-06 15:32:02 [INFO] [TRAIN] epoch: 37, iter: 13740/40000, loss: 0.4301, lr: 0.006879, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 06:07:49 2021-05-06 15:32:10 [INFO] [TRAIN] epoch: 37, iter: 13750/40000, loss: 0.3517, lr: 0.006877, batch_cost: 0.8392, reader_cost: 0.00016, ips: 1.1916 samples/sec | ETA 06:07:09 2021-05-06 15:32:19 [INFO] [TRAIN] epoch: 37, iter: 13760/40000, loss: 0.4010, lr: 0.006874, batch_cost: 0.8378, reader_cost: 0.00011, ips: 1.1936 samples/sec | ETA 06:06:24 2021-05-06 15:32:30 [INFO] [TRAIN] epoch: 38, iter: 13770/40000, loss: 0.6896, lr: 0.006872, batch_cost: 1.1728, reader_cost: 0.20906, ips: 0.8526 samples/sec | ETA 08:32:43 2021-05-06 15:32:39 [INFO] [TRAIN] epoch: 38, iter: 13780/40000, loss: 0.4149, lr: 0.006870, batch_cost: 0.8410, reader_cost: 0.00033, ips: 1.1891 samples/sec | ETA 06:07:30 2021-05-06 15:32:47 [INFO] [TRAIN] epoch: 38, iter: 13790/40000, loss: 0.4644, lr: 0.006867, batch_cost: 0.8417, reader_cost: 0.00016, ips: 1.1881 samples/sec | ETA 06:07:40 2021-05-06 15:32:56 [INFO] [TRAIN] epoch: 38, iter: 13800/40000, loss: 0.4016, lr: 0.006865, batch_cost: 0.8405, reader_cost: 0.00017, ips: 1.1898 samples/sec | ETA 06:06:59 2021-05-06 15:33:04 [INFO] [TRAIN] epoch: 38, iter: 13810/40000, loss: 0.3873, lr: 0.006863, batch_cost: 0.8399, reader_cost: 0.00016, ips: 1.1906 samples/sec | ETA 06:06:37 2021-05-06 15:33:12 [INFO] [TRAIN] epoch: 38, iter: 13820/40000, loss: 0.7787, lr: 0.006860, batch_cost: 0.8418, reader_cost: 0.00016, ips: 1.1879 samples/sec | ETA 06:07:18 2021-05-06 15:33:21 [INFO] [TRAIN] epoch: 38, iter: 13830/40000, loss: 0.6009, lr: 0.006858, batch_cost: 0.8423, reader_cost: 0.00016, ips: 1.1872 samples/sec | ETA 06:07:24 2021-05-06 15:33:29 [INFO] [TRAIN] epoch: 38, iter: 13840/40000, loss: 0.6398, lr: 0.006856, batch_cost: 0.8413, reader_cost: 0.00016, ips: 1.1886 samples/sec | ETA 06:06:49 2021-05-06 15:33:38 [INFO] [TRAIN] epoch: 38, iter: 13850/40000, loss: 0.4607, lr: 0.006853, batch_cost: 0.8432, reader_cost: 0.00017, ips: 1.1860 samples/sec | ETA 06:07:29 2021-05-06 15:33:46 [INFO] [TRAIN] epoch: 38, iter: 13860/40000, loss: 0.2313, lr: 0.006851, batch_cost: 0.8424, reader_cost: 0.00016, ips: 1.1871 samples/sec | ETA 06:07:00 2021-05-06 15:33:54 [INFO] [TRAIN] epoch: 38, iter: 13870/40000, loss: 0.3253, lr: 0.006849, batch_cost: 0.8426, reader_cost: 0.00015, ips: 1.1868 samples/sec | ETA 06:06:57 2021-05-06 15:34:03 [INFO] [TRAIN] epoch: 38, iter: 13880/40000, loss: 0.3086, lr: 0.006846, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 06:05:46 2021-05-06 15:34:11 [INFO] [TRAIN] epoch: 38, iter: 13890/40000, loss: 0.3619, lr: 0.006844, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1911 samples/sec | ETA 06:05:20 2021-05-06 15:34:20 [INFO] [TRAIN] epoch: 38, iter: 13900/40000, loss: 0.6654, lr: 0.006842, batch_cost: 0.8400, reader_cost: 0.00016, ips: 1.1904 samples/sec | ETA 06:05:25 2021-05-06 15:34:28 [INFO] [TRAIN] epoch: 38, iter: 13910/40000, loss: 0.5209, lr: 0.006839, batch_cost: 0.8413, reader_cost: 0.00016, ips: 1.1887 samples/sec | ETA 06:05:48 2021-05-06 15:34:37 [INFO] [TRAIN] epoch: 38, iter: 13920/40000, loss: 0.3904, lr: 0.006837, batch_cost: 0.8419, reader_cost: 0.00015, ips: 1.1877 samples/sec | ETA 06:05:57 2021-05-06 15:34:45 [INFO] [TRAIN] epoch: 38, iter: 13930/40000, loss: 0.4615, lr: 0.006835, batch_cost: 0.8422, reader_cost: 0.00015, ips: 1.1874 samples/sec | ETA 06:05:55 2021-05-06 15:34:53 [INFO] [TRAIN] epoch: 38, iter: 13940/40000, loss: 0.3150, lr: 0.006832, batch_cost: 0.8414, reader_cost: 0.00016, ips: 1.1885 samples/sec | ETA 06:05:26 2021-05-06 15:35:02 [INFO] [TRAIN] epoch: 38, iter: 13950/40000, loss: 0.3404, lr: 0.006830, batch_cost: 0.8402, reader_cost: 0.00014, ips: 1.1902 samples/sec | ETA 06:04:46 2021-05-06 15:35:10 [INFO] [TRAIN] epoch: 38, iter: 13960/40000, loss: 0.4794, lr: 0.006828, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 06:04:54 2021-05-06 15:35:19 [INFO] [TRAIN] epoch: 38, iter: 13970/40000, loss: 0.3950, lr: 0.006825, batch_cost: 0.8419, reader_cost: 0.00015, ips: 1.1878 samples/sec | ETA 06:05:13 2021-05-06 15:35:27 [INFO] [TRAIN] epoch: 38, iter: 13980/40000, loss: 0.5783, lr: 0.006823, batch_cost: 0.8429, reader_cost: 0.00015, ips: 1.1864 samples/sec | ETA 06:05:31 2021-05-06 15:35:35 [INFO] [TRAIN] epoch: 38, iter: 13990/40000, loss: 0.2250, lr: 0.006821, batch_cost: 0.8416, reader_cost: 0.00016, ips: 1.1883 samples/sec | ETA 06:04:48 2021-05-06 15:35:44 [INFO] [TRAIN] epoch: 38, iter: 14000/40000, loss: 0.4693, lr: 0.006818, batch_cost: 0.8429, reader_cost: 0.00015, ips: 1.1864 samples/sec | ETA 06:05:14 2021-05-06 15:35:44 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 15:39:20 [INFO] [EVAL] #Images: 500 mIoU: 0.7206 Acc: 0.9514 Kappa: 0.9369 2021-05-06 15:39:20 [INFO] [EVAL] Class IoU: [0.9778 0.817 0.9115 0.6186 0.5578 0.469 0.5659 0.6912 0.9126 0.6325 0.9382 0.7577 0.4537 0.9321 0.705 0.7848 0.6704 0.5969 0.6991] 2021-05-06 15:39:20 [INFO] [EVAL] Class Acc: [0.9879 0.9267 0.9443 0.7853 0.6663 0.7788 0.7389 0.8948 0.9516 0.7481 0.959 0.8455 0.8111 0.9623 0.8032 0.9337 0.737 0.8229 0.7916] 2021-05-06 15:39:49 [INFO] [EVAL] The model with the best validation mIoU (0.7302) was saved at iter 12000. 2021-05-06 15:39:57 [INFO] [TRAIN] epoch: 38, iter: 14010/40000, loss: 0.3905, lr: 0.006816, batch_cost: 0.8378, reader_cost: 0.00043, ips: 1.1936 samples/sec | ETA 06:02:54 2021-05-06 15:40:05 [INFO] [TRAIN] epoch: 38, iter: 14020/40000, loss: 0.3742, lr: 0.006814, batch_cost: 0.8389, reader_cost: 0.00014, ips: 1.1920 samples/sec | ETA 06:03:14 2021-05-06 15:40:14 [INFO] [TRAIN] epoch: 38, iter: 14030/40000, loss: 0.3195, lr: 0.006812, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1881 samples/sec | ETA 06:04:17 2021-05-06 15:40:22 [INFO] [TRAIN] epoch: 38, iter: 14040/40000, loss: 1.1272, lr: 0.006809, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 06:03:45 2021-05-06 15:40:31 [INFO] [TRAIN] epoch: 38, iter: 14050/40000, loss: 0.6819, lr: 0.006807, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1881 samples/sec | ETA 06:04:01 2021-05-06 15:40:39 [INFO] [TRAIN] epoch: 38, iter: 14060/40000, loss: 0.6614, lr: 0.006805, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1886 samples/sec | ETA 06:03:43 2021-05-06 15:40:47 [INFO] [TRAIN] epoch: 38, iter: 14070/40000, loss: 0.3917, lr: 0.006802, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 06:03:02 2021-05-06 15:40:56 [INFO] [TRAIN] epoch: 38, iter: 14080/40000, loss: 0.5732, lr: 0.006800, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1887 samples/sec | ETA 06:03:26 2021-05-06 15:41:04 [INFO] [TRAIN] epoch: 38, iter: 14090/40000, loss: 0.5894, lr: 0.006798, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 06:03:01 2021-05-06 15:41:13 [INFO] [TRAIN] epoch: 38, iter: 14100/40000, loss: 0.4174, lr: 0.006795, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 06:03:00 2021-05-06 15:41:21 [INFO] [TRAIN] epoch: 38, iter: 14110/40000, loss: 0.4558, lr: 0.006793, batch_cost: 0.8403, reader_cost: 0.00016, ips: 1.1901 samples/sec | ETA 06:02:34 2021-05-06 15:41:29 [INFO] [TRAIN] epoch: 38, iter: 14120/40000, loss: 0.3927, lr: 0.006791, batch_cost: 0.8395, reader_cost: 0.00014, ips: 1.1912 samples/sec | ETA 06:02:06 2021-05-06 15:41:38 [INFO] [TRAIN] epoch: 38, iter: 14130/40000, loss: 0.5771, lr: 0.006788, batch_cost: 0.8383, reader_cost: 0.00012, ips: 1.1930 samples/sec | ETA 06:01:25 2021-05-06 15:41:50 [INFO] [TRAIN] epoch: 39, iter: 14140/40000, loss: 0.6478, lr: 0.006786, batch_cost: 1.1694, reader_cost: 0.20811, ips: 0.8551 samples/sec | ETA 08:24:01 2021-05-06 15:41:58 [INFO] [TRAIN] epoch: 39, iter: 14150/40000, loss: 0.5410, lr: 0.006784, batch_cost: 0.8507, reader_cost: 0.00030, ips: 1.1755 samples/sec | ETA 06:06:30 2021-05-06 15:42:06 [INFO] [TRAIN] epoch: 39, iter: 14160/40000, loss: 0.4989, lr: 0.006781, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 06:01:52 2021-05-06 15:42:15 [INFO] [TRAIN] epoch: 39, iter: 14170/40000, loss: 0.3730, lr: 0.006779, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1881 samples/sec | ETA 06:02:19 2021-05-06 15:42:23 [INFO] [TRAIN] epoch: 39, iter: 14180/40000, loss: 0.2929, lr: 0.006777, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 06:01:46 2021-05-06 15:42:32 [INFO] [TRAIN] epoch: 39, iter: 14190/40000, loss: 0.4345, lr: 0.006774, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 06:01:42 2021-05-06 15:42:40 [INFO] [TRAIN] epoch: 39, iter: 14200/40000, loss: 0.6316, lr: 0.006772, batch_cost: 0.8409, reader_cost: 0.00017, ips: 1.1892 samples/sec | ETA 06:01:34 2021-05-06 15:42:48 [INFO] [TRAIN] epoch: 39, iter: 14210/40000, loss: 0.7984, lr: 0.006770, batch_cost: 0.8411, reader_cost: 0.00016, ips: 1.1889 samples/sec | ETA 06:01:32 2021-05-06 15:42:57 [INFO] [TRAIN] epoch: 39, iter: 14220/40000, loss: 0.4337, lr: 0.006767, batch_cost: 0.8408, reader_cost: 0.00014, ips: 1.1893 samples/sec | ETA 06:01:16 2021-05-06 15:43:05 [INFO] [TRAIN] epoch: 39, iter: 14230/40000, loss: 0.4625, lr: 0.006765, batch_cost: 0.8423, reader_cost: 0.00015, ips: 1.1872 samples/sec | ETA 06:01:47 2021-05-06 15:43:14 [INFO] [TRAIN] epoch: 39, iter: 14240/40000, loss: 0.3289, lr: 0.006763, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 06:00:38 2021-05-06 15:43:22 [INFO] [TRAIN] epoch: 39, iter: 14250/40000, loss: 0.3526, lr: 0.006760, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 06:00:57 2021-05-06 15:43:31 [INFO] [TRAIN] epoch: 39, iter: 14260/40000, loss: 0.5504, lr: 0.006758, batch_cost: 0.8391, reader_cost: 0.00014, ips: 1.1918 samples/sec | ETA 05:59:57 2021-05-06 15:43:39 [INFO] [TRAIN] epoch: 39, iter: 14270/40000, loss: 0.6064, lr: 0.006756, batch_cost: 0.8403, reader_cost: 0.00016, ips: 1.1901 samples/sec | ETA 06:00:19 2021-05-06 15:43:47 [INFO] [TRAIN] epoch: 39, iter: 14280/40000, loss: 0.5574, lr: 0.006753, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 06:00:06 2021-05-06 15:43:56 [INFO] [TRAIN] epoch: 39, iter: 14290/40000, loss: 0.3621, lr: 0.006751, batch_cost: 0.8383, reader_cost: 0.00015, ips: 1.1929 samples/sec | ETA 05:59:12 2021-05-06 15:44:04 [INFO] [TRAIN] epoch: 39, iter: 14300/40000, loss: 0.4107, lr: 0.006749, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1914 samples/sec | ETA 05:59:30 2021-05-06 15:44:12 [INFO] [TRAIN] epoch: 39, iter: 14310/40000, loss: 0.3182, lr: 0.006746, batch_cost: 0.8385, reader_cost: 0.00014, ips: 1.1927 samples/sec | ETA 05:58:59 2021-05-06 15:44:21 [INFO] [TRAIN] epoch: 39, iter: 14320/40000, loss: 0.7416, lr: 0.006744, batch_cost: 0.8388, reader_cost: 0.00017, ips: 1.1922 samples/sec | ETA 05:59:00 2021-05-06 15:44:29 [INFO] [TRAIN] epoch: 39, iter: 14330/40000, loss: 1.0398, lr: 0.006742, batch_cost: 0.8379, reader_cost: 0.00015, ips: 1.1935 samples/sec | ETA 05:58:29 2021-05-06 15:44:38 [INFO] [TRAIN] epoch: 39, iter: 14340/40000, loss: 0.5487, lr: 0.006739, batch_cost: 0.8404, reader_cost: 0.00017, ips: 1.1899 samples/sec | ETA 05:59:25 2021-05-06 15:44:46 [INFO] [TRAIN] epoch: 39, iter: 14350/40000, loss: 0.4385, lr: 0.006737, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 05:59:21 2021-05-06 15:44:54 [INFO] [TRAIN] epoch: 39, iter: 14360/40000, loss: 0.2697, lr: 0.006735, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 05:59:08 2021-05-06 15:45:03 [INFO] [TRAIN] epoch: 39, iter: 14370/40000, loss: 0.3474, lr: 0.006732, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 05:58:41 2021-05-06 15:45:11 [INFO] [TRAIN] epoch: 39, iter: 14380/40000, loss: 0.3420, lr: 0.006730, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 05:58:21 2021-05-06 15:45:20 [INFO] [TRAIN] epoch: 39, iter: 14390/40000, loss: 0.3574, lr: 0.006728, batch_cost: 0.8418, reader_cost: 0.00014, ips: 1.1880 samples/sec | ETA 05:59:17 2021-05-06 15:45:28 [INFO] [TRAIN] epoch: 39, iter: 14400/40000, loss: 0.4264, lr: 0.006725, batch_cost: 0.8403, reader_cost: 0.00014, ips: 1.1900 samples/sec | ETA 05:58:31 2021-05-06 15:45:36 [INFO] [TRAIN] epoch: 39, iter: 14410/40000, loss: 0.5588, lr: 0.006723, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 05:58:33 2021-05-06 15:45:45 [INFO] [TRAIN] epoch: 39, iter: 14420/40000, loss: 0.3907, lr: 0.006721, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 05:58:28 2021-05-06 15:45:53 [INFO] [TRAIN] epoch: 39, iter: 14430/40000, loss: 0.5029, lr: 0.006718, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 05:58:20 2021-05-06 15:46:02 [INFO] [TRAIN] epoch: 39, iter: 14440/40000, loss: 0.3531, lr: 0.006716, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 05:58:01 2021-05-06 15:46:10 [INFO] [TRAIN] epoch: 39, iter: 14450/40000, loss: 0.3632, lr: 0.006714, batch_cost: 0.8410, reader_cost: 0.00014, ips: 1.1890 samples/sec | ETA 05:58:08 2021-05-06 15:46:19 [INFO] [TRAIN] epoch: 39, iter: 14460/40000, loss: 0.6487, lr: 0.006711, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 05:57:33 2021-05-06 15:46:27 [INFO] [TRAIN] epoch: 39, iter: 14470/40000, loss: 0.3162, lr: 0.006709, batch_cost: 0.8404, reader_cost: 0.00016, ips: 1.1899 samples/sec | ETA 05:57:35 2021-05-06 15:46:35 [INFO] [TRAIN] epoch: 39, iter: 14480/40000, loss: 0.5155, lr: 0.006707, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 05:56:58 2021-05-06 15:46:44 [INFO] [TRAIN] epoch: 39, iter: 14490/40000, loss: 0.4768, lr: 0.006704, batch_cost: 0.8401, reader_cost: 0.00016, ips: 1.1903 samples/sec | ETA 05:57:11 2021-05-06 15:46:52 [INFO] [TRAIN] epoch: 39, iter: 14500/40000, loss: 0.3709, lr: 0.006702, batch_cost: 0.8398, reader_cost: 0.00014, ips: 1.1908 samples/sec | ETA 05:56:53 2021-05-06 15:47:04 [INFO] [TRAIN] epoch: 40, iter: 14510/40000, loss: 0.4388, lr: 0.006700, batch_cost: 1.1681, reader_cost: 0.22234, ips: 0.8561 samples/sec | ETA 08:16:14 2021-05-06 15:47:13 [INFO] [TRAIN] epoch: 40, iter: 14520/40000, loss: 0.4148, lr: 0.006697, batch_cost: 0.8733, reader_cost: 0.00032, ips: 1.1451 samples/sec | ETA 06:10:50 2021-05-06 15:47:21 [INFO] [TRAIN] epoch: 40, iter: 14530/40000, loss: 0.5323, lr: 0.006695, batch_cost: 0.8417, reader_cost: 0.00016, ips: 1.1880 samples/sec | ETA 05:57:19 2021-05-06 15:47:29 [INFO] [TRAIN] epoch: 40, iter: 14540/40000, loss: 0.4168, lr: 0.006693, batch_cost: 0.8404, reader_cost: 0.00016, ips: 1.1899 samples/sec | ETA 05:56:36 2021-05-06 15:47:38 [INFO] [TRAIN] epoch: 40, iter: 14550/40000, loss: 0.3198, lr: 0.006690, batch_cost: 0.8399, reader_cost: 0.00016, ips: 1.1906 samples/sec | ETA 05:56:15 2021-05-06 15:47:46 [INFO] [TRAIN] epoch: 40, iter: 14560/40000, loss: 0.4566, lr: 0.006688, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 05:56:17 2021-05-06 15:47:55 [INFO] [TRAIN] epoch: 40, iter: 14570/40000, loss: 0.6841, lr: 0.006686, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 05:56:21 2021-05-06 15:48:03 [INFO] [TRAIN] epoch: 40, iter: 14580/40000, loss: 0.5382, lr: 0.006683, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1880 samples/sec | ETA 05:56:37 2021-05-06 15:48:11 [INFO] [TRAIN] epoch: 40, iter: 14590/40000, loss: 0.5891, lr: 0.006681, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1885 samples/sec | ETA 05:56:20 2021-05-06 15:48:20 [INFO] [TRAIN] epoch: 40, iter: 14600/40000, loss: 0.4617, lr: 0.006679, batch_cost: 0.8411, reader_cost: 0.00014, ips: 1.1889 samples/sec | ETA 05:56:04 2021-05-06 15:48:28 [INFO] [TRAIN] epoch: 40, iter: 14610/40000, loss: 0.2355, lr: 0.006676, batch_cost: 0.8399, reader_cost: 0.00014, ips: 1.1906 samples/sec | ETA 05:55:24 2021-05-06 15:48:37 [INFO] [TRAIN] epoch: 40, iter: 14620/40000, loss: 0.4276, lr: 0.006674, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 05:55:26 2021-05-06 15:48:45 [INFO] [TRAIN] epoch: 40, iter: 14630/40000, loss: 0.7132, lr: 0.006672, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 05:55:16 2021-05-06 15:48:53 [INFO] [TRAIN] epoch: 40, iter: 14640/40000, loss: 0.7054, lr: 0.006669, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 05:54:44 2021-05-06 15:49:02 [INFO] [TRAIN] epoch: 40, iter: 14650/40000, loss: 0.5986, lr: 0.006667, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 05:54:46 2021-05-06 15:49:10 [INFO] [TRAIN] epoch: 40, iter: 14660/40000, loss: 0.4066, lr: 0.006665, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 05:54:43 2021-05-06 15:49:19 [INFO] [TRAIN] epoch: 40, iter: 14670/40000, loss: 0.3849, lr: 0.006662, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 05:54:42 2021-05-06 15:49:27 [INFO] [TRAIN] epoch: 40, iter: 14680/40000, loss: 0.3306, lr: 0.006660, batch_cost: 0.8403, reader_cost: 0.00016, ips: 1.1901 samples/sec | ETA 05:54:35 2021-05-06 15:49:35 [INFO] [TRAIN] epoch: 40, iter: 14690/40000, loss: 0.3135, lr: 0.006658, batch_cost: 0.8410, reader_cost: 0.00014, ips: 1.1890 samples/sec | ETA 05:54:46 2021-05-06 15:49:44 [INFO] [TRAIN] epoch: 40, iter: 14700/40000, loss: 0.6713, lr: 0.006655, batch_cost: 0.8407, reader_cost: 0.00016, ips: 1.1895 samples/sec | ETA 05:54:30 2021-05-06 15:49:52 [INFO] [TRAIN] epoch: 40, iter: 14710/40000, loss: 0.4524, lr: 0.006653, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 05:54:18 2021-05-06 15:50:01 [INFO] [TRAIN] epoch: 40, iter: 14720/40000, loss: 0.6307, lr: 0.006651, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1884 samples/sec | ETA 05:54:31 2021-05-06 15:50:09 [INFO] [TRAIN] epoch: 40, iter: 14730/40000, loss: 0.5393, lr: 0.006648, batch_cost: 0.8390, reader_cost: 0.00015, ips: 1.1919 samples/sec | ETA 05:53:21 2021-05-06 15:50:17 [INFO] [TRAIN] epoch: 40, iter: 14740/40000, loss: 0.2771, lr: 0.006646, batch_cost: 0.8425, reader_cost: 0.00016, ips: 1.1869 samples/sec | ETA 05:54:42 2021-05-06 15:50:26 [INFO] [TRAIN] epoch: 40, iter: 14750/40000, loss: 0.5389, lr: 0.006644, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 05:53:42 2021-05-06 15:50:34 [INFO] [TRAIN] epoch: 40, iter: 14760/40000, loss: 0.3344, lr: 0.006641, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 05:53:41 2021-05-06 15:50:43 [INFO] [TRAIN] epoch: 40, iter: 14770/40000, loss: 0.4865, lr: 0.006639, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 05:53:18 2021-05-06 15:50:51 [INFO] [TRAIN] epoch: 40, iter: 14780/40000, loss: 0.3180, lr: 0.006637, batch_cost: 0.8397, reader_cost: 0.00016, ips: 1.1909 samples/sec | ETA 05:52:56 2021-05-06 15:51:00 [INFO] [TRAIN] epoch: 40, iter: 14790/40000, loss: 0.3392, lr: 0.006634, batch_cost: 0.8407, reader_cost: 0.00016, ips: 1.1895 samples/sec | ETA 05:53:13 2021-05-06 15:51:08 [INFO] [TRAIN] epoch: 40, iter: 14800/40000, loss: 0.5166, lr: 0.006632, batch_cost: 0.8406, reader_cost: 0.00016, ips: 1.1897 samples/sec | ETA 05:53:02 2021-05-06 15:51:16 [INFO] [TRAIN] epoch: 40, iter: 14810/40000, loss: 0.4851, lr: 0.006630, batch_cost: 0.8412, reader_cost: 0.00016, ips: 1.1888 samples/sec | ETA 05:53:08 2021-05-06 15:51:25 [INFO] [TRAIN] epoch: 40, iter: 14820/40000, loss: 0.3048, lr: 0.006627, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1885 samples/sec | ETA 05:53:06 2021-05-06 15:51:33 [INFO] [TRAIN] epoch: 40, iter: 14830/40000, loss: 0.4434, lr: 0.006625, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 05:52:46 2021-05-06 15:51:42 [INFO] [TRAIN] epoch: 40, iter: 14840/40000, loss: 0.3682, lr: 0.006623, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 05:52:39 2021-05-06 15:51:50 [INFO] [TRAIN] epoch: 40, iter: 14850/40000, loss: 0.3003, lr: 0.006621, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1883 samples/sec | ETA 05:52:45 2021-05-06 15:51:58 [INFO] [TRAIN] epoch: 40, iter: 14860/40000, loss: 0.4248, lr: 0.006618, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 05:52:23 2021-05-06 15:52:07 [INFO] [TRAIN] epoch: 40, iter: 14870/40000, loss: 0.4523, lr: 0.006616, batch_cost: 0.8404, reader_cost: 0.00014, ips: 1.1898 samples/sec | ETA 05:52:00 2021-05-06 15:52:15 [INFO] [TRAIN] epoch: 40, iter: 14880/40000, loss: 0.6219, lr: 0.006614, batch_cost: 0.8401, reader_cost: 0.00010, ips: 1.1903 samples/sec | ETA 05:51:43 2021-05-06 15:52:27 [INFO] [TRAIN] epoch: 41, iter: 14890/40000, loss: 0.4971, lr: 0.006611, batch_cost: 1.1458, reader_cost: 0.20996, ips: 0.8727 samples/sec | ETA 07:59:32 2021-05-06 15:52:35 [INFO] [TRAIN] epoch: 41, iter: 14900/40000, loss: 0.4906, lr: 0.006609, batch_cost: 0.8421, reader_cost: 0.00029, ips: 1.1875 samples/sec | ETA 05:52:17 2021-05-06 15:52:44 [INFO] [TRAIN] epoch: 41, iter: 14910/40000, loss: 0.5482, lr: 0.006606, batch_cost: 0.8434, reader_cost: 0.00015, ips: 1.1857 samples/sec | ETA 05:52:40 2021-05-06 15:52:52 [INFO] [TRAIN] epoch: 41, iter: 14920/40000, loss: 0.2965, lr: 0.006604, batch_cost: 0.8417, reader_cost: 0.00014, ips: 1.1881 samples/sec | ETA 05:51:48 2021-05-06 15:53:00 [INFO] [TRAIN] epoch: 41, iter: 14930/40000, loss: 0.3062, lr: 0.006602, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1887 samples/sec | ETA 05:51:31 2021-05-06 15:53:09 [INFO] [TRAIN] epoch: 41, iter: 14940/40000, loss: 0.6582, lr: 0.006599, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1883 samples/sec | ETA 05:51:28 2021-05-06 15:53:17 [INFO] [TRAIN] epoch: 41, iter: 14950/40000, loss: 0.5079, lr: 0.006597, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 05:50:52 2021-05-06 15:53:26 [INFO] [TRAIN] epoch: 41, iter: 14960/40000, loss: 0.6399, lr: 0.006595, batch_cost: 0.8410, reader_cost: 0.00016, ips: 1.1890 samples/sec | ETA 05:50:59 2021-05-06 15:53:34 [INFO] [TRAIN] epoch: 41, iter: 14970/40000, loss: 0.4854, lr: 0.006592, batch_cost: 0.8390, reader_cost: 0.00015, ips: 1.1919 samples/sec | ETA 05:50:00 2021-05-06 15:53:42 [INFO] [TRAIN] epoch: 41, iter: 14980/40000, loss: 0.1643, lr: 0.006590, batch_cost: 0.8398, reader_cost: 0.00017, ips: 1.1907 samples/sec | ETA 05:50:12 2021-05-06 15:53:51 [INFO] [TRAIN] epoch: 41, iter: 14990/40000, loss: 0.3604, lr: 0.006588, batch_cost: 0.8410, reader_cost: 0.00016, ips: 1.1891 samples/sec | ETA 05:50:33 2021-05-06 15:53:59 [INFO] [TRAIN] epoch: 41, iter: 15000/40000, loss: 0.4377, lr: 0.006585, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 05:49:56 2021-05-06 15:53:59 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 15:57:36 [INFO] [EVAL] #Images: 500 mIoU: 0.7373 Acc: 0.9534 Kappa: 0.9396 2021-05-06 15:57:36 [INFO] [EVAL] Class IoU: [0.9794 0.8343 0.9143 0.6059 0.5692 0.4974 0.5701 0.7038 0.9127 0.651 0.9403 0.7546 0.4842 0.9305 0.7542 0.817 0.7788 0.6025 0.7089] 2021-05-06 15:57:36 [INFO] [EVAL] Class Acc: [0.9898 0.9146 0.9519 0.7164 0.823 0.7626 0.7877 0.8923 0.9513 0.7724 0.9621 0.817 0.7958 0.9499 0.8704 0.9582 0.9062 0.8231 0.7938] 2021-05-06 15:58:28 [INFO] [EVAL] The model with the best validation mIoU (0.7373) was saved at iter 15000. 2021-05-06 15:58:36 [INFO] [TRAIN] epoch: 41, iter: 15010/40000, loss: 0.4727, lr: 0.006583, batch_cost: 0.8370, reader_cost: 0.00048, ips: 1.1947 samples/sec | ETA 05:48:37 2021-05-06 15:58:44 [INFO] [TRAIN] epoch: 41, iter: 15020/40000, loss: 0.5174, lr: 0.006581, batch_cost: 0.8376, reader_cost: 0.00016, ips: 1.1939 samples/sec | ETA 05:48:42 2021-05-06 15:58:53 [INFO] [TRAIN] epoch: 41, iter: 15030/40000, loss: 0.4827, lr: 0.006578, batch_cost: 0.8402, reader_cost: 0.00016, ips: 1.1901 samples/sec | ETA 05:49:40 2021-05-06 15:59:01 [INFO] [TRAIN] epoch: 41, iter: 15040/40000, loss: 0.2428, lr: 0.006576, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 05:49:25 2021-05-06 15:59:10 [INFO] [TRAIN] epoch: 41, iter: 15050/40000, loss: 0.3459, lr: 0.006574, batch_cost: 0.8411, reader_cost: 0.00014, ips: 1.1890 samples/sec | ETA 05:49:44 2021-05-06 15:59:18 [INFO] [TRAIN] epoch: 41, iter: 15060/40000, loss: 0.4251, lr: 0.006571, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 05:49:35 2021-05-06 15:59:27 [INFO] [TRAIN] epoch: 41, iter: 15070/40000, loss: 0.5006, lr: 0.006569, batch_cost: 0.8419, reader_cost: 0.00016, ips: 1.1878 samples/sec | ETA 05:49:47 2021-05-06 15:59:35 [INFO] [TRAIN] epoch: 41, iter: 15080/40000, loss: 0.4865, lr: 0.006567, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1888 samples/sec | ETA 05:49:21 2021-05-06 15:59:43 [INFO] [TRAIN] epoch: 41, iter: 15090/40000, loss: 0.5499, lr: 0.006564, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 05:48:52 2021-05-06 15:59:52 [INFO] [TRAIN] epoch: 41, iter: 15100/40000, loss: 0.3539, lr: 0.006562, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 05:48:50 2021-05-06 16:00:00 [INFO] [TRAIN] epoch: 41, iter: 15110/40000, loss: 0.3578, lr: 0.006560, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1911 samples/sec | ETA 05:48:16 2021-05-06 16:00:09 [INFO] [TRAIN] epoch: 41, iter: 15120/40000, loss: 0.3551, lr: 0.006557, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 05:48:21 2021-05-06 16:00:17 [INFO] [TRAIN] epoch: 41, iter: 15130/40000, loss: 0.2475, lr: 0.006555, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 05:48:24 2021-05-06 16:00:25 [INFO] [TRAIN] epoch: 41, iter: 15140/40000, loss: 0.2867, lr: 0.006553, batch_cost: 0.8394, reader_cost: 0.00014, ips: 1.1913 samples/sec | ETA 05:47:47 2021-05-06 16:00:34 [INFO] [TRAIN] epoch: 41, iter: 15150/40000, loss: 0.3670, lr: 0.006550, batch_cost: 0.8413, reader_cost: 0.00014, ips: 1.1886 samples/sec | ETA 05:48:26 2021-05-06 16:00:42 [INFO] [TRAIN] epoch: 41, iter: 15160/40000, loss: 0.2798, lr: 0.006548, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1880 samples/sec | ETA 05:48:29 2021-05-06 16:00:51 [INFO] [TRAIN] epoch: 41, iter: 15170/40000, loss: 0.5215, lr: 0.006546, batch_cost: 0.8389, reader_cost: 0.00014, ips: 1.1921 samples/sec | ETA 05:47:08 2021-05-06 16:00:59 [INFO] [TRAIN] epoch: 41, iter: 15180/40000, loss: 0.4299, lr: 0.006543, batch_cost: 0.8390, reader_cost: 0.00014, ips: 1.1919 samples/sec | ETA 05:47:04 2021-05-06 16:01:07 [INFO] [TRAIN] epoch: 41, iter: 15190/40000, loss: 0.2956, lr: 0.006541, batch_cost: 0.8391, reader_cost: 0.00013, ips: 1.1918 samples/sec | ETA 05:46:57 2021-05-06 16:01:16 [INFO] [TRAIN] epoch: 41, iter: 15200/40000, loss: 0.4337, lr: 0.006539, batch_cost: 0.8397, reader_cost: 0.00014, ips: 1.1909 samples/sec | ETA 05:47:04 2021-05-06 16:01:24 [INFO] [TRAIN] epoch: 41, iter: 15210/40000, loss: 0.4163, lr: 0.006536, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1903 samples/sec | ETA 05:47:06 2021-05-06 16:01:33 [INFO] [TRAIN] epoch: 41, iter: 15220/40000, loss: 0.3556, lr: 0.006534, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1883 samples/sec | ETA 05:47:32 2021-05-06 16:01:41 [INFO] [TRAIN] epoch: 41, iter: 15230/40000, loss: 0.4457, lr: 0.006532, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 05:46:52 2021-05-06 16:01:49 [INFO] [TRAIN] epoch: 41, iter: 15240/40000, loss: 0.4345, lr: 0.006529, batch_cost: 0.8420, reader_cost: 0.00015, ips: 1.1877 samples/sec | ETA 05:47:27 2021-05-06 16:01:58 [INFO] [TRAIN] epoch: 41, iter: 15250/40000, loss: 0.4972, lr: 0.006527, batch_cost: 0.8368, reader_cost: 0.00010, ips: 1.1950 samples/sec | ETA 05:45:10 2021-05-06 16:02:10 [INFO] [TRAIN] epoch: 42, iter: 15260/40000, loss: 0.5118, lr: 0.006525, batch_cost: 1.1871, reader_cost: 0.23327, ips: 0.8424 samples/sec | ETA 08:09:29 2021-05-06 16:02:18 [INFO] [TRAIN] epoch: 42, iter: 15270/40000, loss: 0.3536, lr: 0.006522, batch_cost: 0.8437, reader_cost: 0.00032, ips: 1.1852 samples/sec | ETA 05:47:45 2021-05-06 16:02:27 [INFO] [TRAIN] epoch: 42, iter: 15280/40000, loss: 0.4831, lr: 0.006520, batch_cost: 0.8424, reader_cost: 0.00016, ips: 1.1871 samples/sec | ETA 05:47:03 2021-05-06 16:02:35 [INFO] [TRAIN] epoch: 42, iter: 15290/40000, loss: 0.3697, lr: 0.006518, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 05:46:11 2021-05-06 16:02:43 [INFO] [TRAIN] epoch: 42, iter: 15300/40000, loss: 0.3016, lr: 0.006515, batch_cost: 0.8430, reader_cost: 0.00015, ips: 1.1862 samples/sec | ETA 05:47:02 2021-05-06 16:02:52 [INFO] [TRAIN] epoch: 42, iter: 15310/40000, loss: 0.6023, lr: 0.006513, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 05:46:06 2021-05-06 16:03:00 [INFO] [TRAIN] epoch: 42, iter: 15320/40000, loss: 0.5579, lr: 0.006511, batch_cost: 0.8421, reader_cost: 0.00015, ips: 1.1876 samples/sec | ETA 05:46:21 2021-05-06 16:03:09 [INFO] [TRAIN] epoch: 42, iter: 15330/40000, loss: 0.5395, lr: 0.006508, batch_cost: 0.8384, reader_cost: 0.00014, ips: 1.1928 samples/sec | ETA 05:44:42 2021-05-06 16:03:17 [INFO] [TRAIN] epoch: 42, iter: 15340/40000, loss: 0.3428, lr: 0.006506, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1911 samples/sec | ETA 05:45:03 2021-05-06 16:03:25 [INFO] [TRAIN] epoch: 42, iter: 15350/40000, loss: 0.2239, lr: 0.006504, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1914 samples/sec | ETA 05:44:50 2021-05-06 16:03:34 [INFO] [TRAIN] epoch: 42, iter: 15360/40000, loss: 0.2629, lr: 0.006501, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 05:44:48 2021-05-06 16:03:42 [INFO] [TRAIN] epoch: 42, iter: 15370/40000, loss: 0.4551, lr: 0.006499, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1884 samples/sec | ETA 05:45:25 2021-05-06 16:03:51 [INFO] [TRAIN] epoch: 42, iter: 15380/40000, loss: 0.4805, lr: 0.006497, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 05:44:36 2021-05-06 16:03:59 [INFO] [TRAIN] epoch: 42, iter: 15390/40000, loss: 0.7474, lr: 0.006494, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 05:44:15 2021-05-06 16:04:07 [INFO] [TRAIN] epoch: 42, iter: 15400/40000, loss: 0.4676, lr: 0.006492, batch_cost: 0.8403, reader_cost: 0.00014, ips: 1.1900 samples/sec | ETA 05:44:32 2021-05-06 16:04:16 [INFO] [TRAIN] epoch: 42, iter: 15410/40000, loss: 0.3459, lr: 0.006490, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1881 samples/sec | ETA 05:44:56 2021-05-06 16:04:24 [INFO] [TRAIN] epoch: 42, iter: 15420/40000, loss: 0.4170, lr: 0.006487, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 05:44:21 2021-05-06 16:04:33 [INFO] [TRAIN] epoch: 42, iter: 15430/40000, loss: 0.2900, lr: 0.006485, batch_cost: 0.8407, reader_cost: 0.00016, ips: 1.1895 samples/sec | ETA 05:44:16 2021-05-06 16:04:41 [INFO] [TRAIN] epoch: 42, iter: 15440/40000, loss: 0.5074, lr: 0.006483, batch_cost: 0.8406, reader_cost: 0.00017, ips: 1.1896 samples/sec | ETA 05:44:06 2021-05-06 16:04:49 [INFO] [TRAIN] epoch: 42, iter: 15450/40000, loss: 0.8764, lr: 0.006480, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1912 samples/sec | ETA 05:43:29 2021-05-06 16:04:58 [INFO] [TRAIN] epoch: 42, iter: 15460/40000, loss: 0.3296, lr: 0.006478, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 05:43:45 2021-05-06 16:05:06 [INFO] [TRAIN] epoch: 42, iter: 15470/40000, loss: 0.5776, lr: 0.006476, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 05:43:19 2021-05-06 16:05:15 [INFO] [TRAIN] epoch: 42, iter: 15480/40000, loss: 0.2205, lr: 0.006473, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 05:42:59 2021-05-06 16:05:23 [INFO] [TRAIN] epoch: 42, iter: 15490/40000, loss: 0.3339, lr: 0.006471, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 05:43:17 2021-05-06 16:05:31 [INFO] [TRAIN] epoch: 42, iter: 15500/40000, loss: 0.3123, lr: 0.006469, batch_cost: 0.8399, reader_cost: 0.00016, ips: 1.1907 samples/sec | ETA 05:42:56 2021-05-06 16:05:40 [INFO] [TRAIN] epoch: 42, iter: 15510/40000, loss: 0.3638, lr: 0.006466, batch_cost: 0.8405, reader_cost: 0.00016, ips: 1.1898 samples/sec | ETA 05:43:03 2021-05-06 16:05:48 [INFO] [TRAIN] epoch: 42, iter: 15520/40000, loss: 0.3402, lr: 0.006464, batch_cost: 0.8392, reader_cost: 0.00015, ips: 1.1916 samples/sec | ETA 05:42:23 2021-05-06 16:05:57 [INFO] [TRAIN] epoch: 42, iter: 15530/40000, loss: 0.2841, lr: 0.006462, batch_cost: 0.8384, reader_cost: 0.00016, ips: 1.1928 samples/sec | ETA 05:41:54 2021-05-06 16:06:05 [INFO] [TRAIN] epoch: 42, iter: 15540/40000, loss: 0.4702, lr: 0.006459, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 05:42:36 2021-05-06 16:06:13 [INFO] [TRAIN] epoch: 42, iter: 15550/40000, loss: 0.4445, lr: 0.006457, batch_cost: 0.8394, reader_cost: 0.00014, ips: 1.1913 samples/sec | ETA 05:42:03 2021-05-06 16:06:22 [INFO] [TRAIN] epoch: 42, iter: 15560/40000, loss: 0.4214, lr: 0.006455, batch_cost: 0.8392, reader_cost: 0.00014, ips: 1.1916 samples/sec | ETA 05:41:50 2021-05-06 16:06:30 [INFO] [TRAIN] epoch: 42, iter: 15570/40000, loss: 0.3627, lr: 0.006452, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 05:42:07 2021-05-06 16:06:39 [INFO] [TRAIN] epoch: 42, iter: 15580/40000, loss: 0.5239, lr: 0.006450, batch_cost: 0.8402, reader_cost: 0.00016, ips: 1.1901 samples/sec | ETA 05:41:58 2021-05-06 16:06:47 [INFO] [TRAIN] epoch: 42, iter: 15590/40000, loss: 0.3015, lr: 0.006448, batch_cost: 0.8402, reader_cost: 0.00016, ips: 1.1902 samples/sec | ETA 05:41:49 2021-05-06 16:06:55 [INFO] [TRAIN] epoch: 42, iter: 15600/40000, loss: 0.6418, lr: 0.006445, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 05:41:18 2021-05-06 16:07:04 [INFO] [TRAIN] epoch: 42, iter: 15610/40000, loss: 0.4240, lr: 0.006443, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1885 samples/sec | ETA 05:42:01 2021-05-06 16:07:12 [INFO] [TRAIN] epoch: 42, iter: 15620/40000, loss: 0.4152, lr: 0.006441, batch_cost: 0.8399, reader_cost: 0.00011, ips: 1.1906 samples/sec | ETA 05:41:17 2021-05-06 16:07:24 [INFO] [TRAIN] epoch: 43, iter: 15630/40000, loss: 0.4287, lr: 0.006438, batch_cost: 1.1597, reader_cost: 0.20381, ips: 0.8623 samples/sec | ETA 07:51:02 2021-05-06 16:07:32 [INFO] [TRAIN] epoch: 43, iter: 15640/40000, loss: 0.3519, lr: 0.006436, batch_cost: 0.8423, reader_cost: 0.00031, ips: 1.1873 samples/sec | ETA 05:41:57 2021-05-06 16:07:41 [INFO] [TRAIN] epoch: 43, iter: 15650/40000, loss: 0.6683, lr: 0.006434, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 05:41:04 2021-05-06 16:07:49 [INFO] [TRAIN] epoch: 43, iter: 15660/40000, loss: 0.4234, lr: 0.006431, batch_cost: 0.8425, reader_cost: 0.00015, ips: 1.1870 samples/sec | ETA 05:41:45 2021-05-06 16:07:57 [INFO] [TRAIN] epoch: 43, iter: 15670/40000, loss: 0.3676, lr: 0.006429, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1911 samples/sec | ETA 05:40:25 2021-05-06 16:08:06 [INFO] [TRAIN] epoch: 43, iter: 15680/40000, loss: 0.9451, lr: 0.006427, batch_cost: 0.8453, reader_cost: 0.00015, ips: 1.1831 samples/sec | ETA 05:42:36 2021-05-06 16:08:14 [INFO] [TRAIN] epoch: 43, iter: 15690/40000, loss: 0.5419, lr: 0.006424, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1880 samples/sec | ETA 05:41:02 2021-05-06 16:08:23 [INFO] [TRAIN] epoch: 43, iter: 15700/40000, loss: 0.6825, lr: 0.006422, batch_cost: 0.8414, reader_cost: 0.00014, ips: 1.1885 samples/sec | ETA 05:40:45 2021-05-06 16:08:31 [INFO] [TRAIN] epoch: 43, iter: 15710/40000, loss: 0.4607, lr: 0.006419, batch_cost: 0.8412, reader_cost: 0.00014, ips: 1.1888 samples/sec | ETA 05:40:31 2021-05-06 16:08:40 [INFO] [TRAIN] epoch: 43, iter: 15720/40000, loss: 0.2867, lr: 0.006417, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 05:40:06 2021-05-06 16:08:48 [INFO] [TRAIN] epoch: 43, iter: 15730/40000, loss: 0.5617, lr: 0.006415, batch_cost: 0.8419, reader_cost: 0.00015, ips: 1.1877 samples/sec | ETA 05:40:33 2021-05-06 16:08:56 [INFO] [TRAIN] epoch: 43, iter: 15740/40000, loss: 0.3516, lr: 0.006412, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1883 samples/sec | ETA 05:40:15 2021-05-06 16:09:05 [INFO] [TRAIN] epoch: 43, iter: 15750/40000, loss: 0.3360, lr: 0.006410, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1911 samples/sec | ETA 05:39:18 2021-05-06 16:09:13 [INFO] [TRAIN] epoch: 43, iter: 15760/40000, loss: 0.6222, lr: 0.006408, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 05:39:18 2021-05-06 16:09:22 [INFO] [TRAIN] epoch: 43, iter: 15770/40000, loss: 0.6472, lr: 0.006405, batch_cost: 0.8426, reader_cost: 0.00014, ips: 1.1867 samples/sec | ETA 05:40:17 2021-05-06 16:09:30 [INFO] [TRAIN] epoch: 43, iter: 15780/40000, loss: 0.3468, lr: 0.006403, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1883 samples/sec | ETA 05:39:41 2021-05-06 16:09:38 [INFO] [TRAIN] epoch: 43, iter: 15790/40000, loss: 0.4042, lr: 0.006401, batch_cost: 0.8417, reader_cost: 0.00014, ips: 1.1881 samples/sec | ETA 05:39:37 2021-05-06 16:09:47 [INFO] [TRAIN] epoch: 43, iter: 15800/40000, loss: 0.3949, lr: 0.006398, batch_cost: 0.8416, reader_cost: 0.00014, ips: 1.1882 samples/sec | ETA 05:39:27 2021-05-06 16:09:55 [INFO] [TRAIN] epoch: 43, iter: 15810/40000, loss: 0.7790, lr: 0.006396, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 05:38:48 2021-05-06 16:10:04 [INFO] [TRAIN] epoch: 43, iter: 15820/40000, loss: 0.7282, lr: 0.006394, batch_cost: 0.8412, reader_cost: 0.00014, ips: 1.1887 samples/sec | ETA 05:39:01 2021-05-06 16:10:12 [INFO] [TRAIN] epoch: 43, iter: 15830/40000, loss: 0.4766, lr: 0.006391, batch_cost: 0.8422, reader_cost: 0.00017, ips: 1.1873 samples/sec | ETA 05:39:16 2021-05-06 16:10:20 [INFO] [TRAIN] epoch: 43, iter: 15840/40000, loss: 0.4978, lr: 0.006389, batch_cost: 0.8403, reader_cost: 0.00017, ips: 1.1900 samples/sec | ETA 05:38:22 2021-05-06 16:10:29 [INFO] [TRAIN] epoch: 43, iter: 15850/40000, loss: 0.3655, lr: 0.006387, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 05:38:18 2021-05-06 16:10:37 [INFO] [TRAIN] epoch: 43, iter: 15860/40000, loss: 0.2855, lr: 0.006384, batch_cost: 0.8428, reader_cost: 0.00015, ips: 1.1865 samples/sec | ETA 05:39:06 2021-05-06 16:10:46 [INFO] [TRAIN] epoch: 43, iter: 15870/40000, loss: 0.2357, lr: 0.006382, batch_cost: 0.8424, reader_cost: 0.00014, ips: 1.1871 samples/sec | ETA 05:38:46 2021-05-06 16:10:54 [INFO] [TRAIN] epoch: 43, iter: 15880/40000, loss: 0.3599, lr: 0.006380, batch_cost: 0.8422, reader_cost: 0.00015, ips: 1.1874 samples/sec | ETA 05:38:34 2021-05-06 16:11:03 [INFO] [TRAIN] epoch: 43, iter: 15890/40000, loss: 0.3302, lr: 0.006377, batch_cost: 0.8398, reader_cost: 0.00014, ips: 1.1908 samples/sec | ETA 05:37:26 2021-05-06 16:11:11 [INFO] [TRAIN] epoch: 43, iter: 15900/40000, loss: 0.5500, lr: 0.006375, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1888 samples/sec | ETA 05:37:53 2021-05-06 16:11:19 [INFO] [TRAIN] epoch: 43, iter: 15910/40000, loss: 0.3754, lr: 0.006373, batch_cost: 0.8404, reader_cost: 0.00014, ips: 1.1899 samples/sec | ETA 05:37:25 2021-05-06 16:11:28 [INFO] [TRAIN] epoch: 43, iter: 15920/40000, loss: 0.4346, lr: 0.006370, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1888 samples/sec | ETA 05:37:35 2021-05-06 16:11:36 [INFO] [TRAIN] epoch: 43, iter: 15930/40000, loss: 0.3660, lr: 0.006368, batch_cost: 0.8420, reader_cost: 0.00015, ips: 1.1877 samples/sec | ETA 05:37:46 2021-05-06 16:11:45 [INFO] [TRAIN] epoch: 43, iter: 15940/40000, loss: 0.2770, lr: 0.006366, batch_cost: 0.8422, reader_cost: 0.00016, ips: 1.1874 samples/sec | ETA 05:37:43 2021-05-06 16:11:53 [INFO] [TRAIN] epoch: 43, iter: 15950/40000, loss: 0.4275, lr: 0.006363, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 05:37:00 2021-05-06 16:12:01 [INFO] [TRAIN] epoch: 43, iter: 15960/40000, loss: 0.2817, lr: 0.006361, batch_cost: 0.8425, reader_cost: 0.00015, ips: 1.1870 samples/sec | ETA 05:37:32 2021-05-06 16:12:10 [INFO] [TRAIN] epoch: 43, iter: 15970/40000, loss: 0.4308, lr: 0.006359, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1879 samples/sec | ETA 05:37:09 2021-05-06 16:12:18 [INFO] [TRAIN] epoch: 43, iter: 15980/40000, loss: 0.4772, lr: 0.006356, batch_cost: 0.8419, reader_cost: 0.00015, ips: 1.1878 samples/sec | ETA 05:37:02 2021-05-06 16:12:27 [INFO] [TRAIN] epoch: 43, iter: 15990/40000, loss: 0.4352, lr: 0.006354, batch_cost: 0.8390, reader_cost: 0.00012, ips: 1.1919 samples/sec | ETA 05:35:43 2021-05-06 16:12:38 [INFO] [TRAIN] epoch: 44, iter: 16000/40000, loss: 0.4254, lr: 0.006352, batch_cost: 1.1467, reader_cost: 0.22335, ips: 0.8720 samples/sec | ETA 07:38:41 2021-05-06 16:12:38 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 16:16:17 [INFO] [EVAL] #Images: 500 mIoU: 0.7096 Acc: 0.9489 Kappa: 0.9337 2021-05-06 16:16:17 [INFO] [EVAL] Class IoU: [0.9746 0.8129 0.9082 0.5722 0.545 0.4601 0.5421 0.6816 0.9072 0.6248 0.9371 0.7446 0.5179 0.9222 0.7361 0.7111 0.6153 0.5785 0.6916] 2021-05-06 16:16:17 [INFO] [EVAL] Class Acc: [0.989 0.9028 0.94 0.8578 0.7192 0.7787 0.7473 0.9039 0.9447 0.7847 0.9557 0.8224 0.7566 0.9463 0.829 0.9683 0.6651 0.798 0.8118] 2021-05-06 16:16:46 [INFO] [EVAL] The model with the best validation mIoU (0.7373) was saved at iter 15000. 2021-05-06 16:16:55 [INFO] [TRAIN] epoch: 44, iter: 16010/40000, loss: 0.4945, lr: 0.006349, batch_cost: 0.8411, reader_cost: 0.00022, ips: 1.1890 samples/sec | ETA 05:36:17 2021-05-06 16:17:03 [INFO] [TRAIN] epoch: 44, iter: 16020/40000, loss: 0.4305, lr: 0.006347, batch_cost: 0.8393, reader_cost: 0.00017, ips: 1.1915 samples/sec | ETA 05:35:26 2021-05-06 16:17:12 [INFO] [TRAIN] epoch: 44, iter: 16030/40000, loss: 0.5119, lr: 0.006345, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1887 samples/sec | ETA 05:36:05 2021-05-06 16:17:20 [INFO] [TRAIN] epoch: 44, iter: 16040/40000, loss: 0.4816, lr: 0.006342, batch_cost: 0.8404, reader_cost: 0.00016, ips: 1.1900 samples/sec | ETA 05:35:34 2021-05-06 16:17:28 [INFO] [TRAIN] epoch: 44, iter: 16050/40000, loss: 0.7239, lr: 0.006340, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1885 samples/sec | ETA 05:35:50 2021-05-06 16:17:37 [INFO] [TRAIN] epoch: 44, iter: 16060/40000, loss: 0.4462, lr: 0.006337, batch_cost: 0.8404, reader_cost: 0.00014, ips: 1.1899 samples/sec | ETA 05:35:18 2021-05-06 16:17:45 [INFO] [TRAIN] epoch: 44, iter: 16070/40000, loss: 0.5242, lr: 0.006335, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 05:35:01 2021-05-06 16:17:54 [INFO] [TRAIN] epoch: 44, iter: 16080/40000, loss: 0.4444, lr: 0.006333, batch_cost: 0.8420, reader_cost: 0.00015, ips: 1.1876 samples/sec | ETA 05:35:41 2021-05-06 16:18:02 [INFO] [TRAIN] epoch: 44, iter: 16090/40000, loss: 0.3698, lr: 0.006330, batch_cost: 0.8424, reader_cost: 0.00015, ips: 1.1870 samples/sec | ETA 05:35:42 2021-05-06 16:18:10 [INFO] [TRAIN] epoch: 44, iter: 16100/40000, loss: 0.3131, lr: 0.006328, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1882 samples/sec | ETA 05:35:14 2021-05-06 16:18:19 [INFO] [TRAIN] epoch: 44, iter: 16110/40000, loss: 0.4056, lr: 0.006326, batch_cost: 0.8448, reader_cost: 0.00015, ips: 1.1837 samples/sec | ETA 05:36:22 2021-05-06 16:18:27 [INFO] [TRAIN] epoch: 44, iter: 16120/40000, loss: 0.4013, lr: 0.006323, batch_cost: 0.8419, reader_cost: 0.00014, ips: 1.1879 samples/sec | ETA 05:35:03 2021-05-06 16:18:36 [INFO] [TRAIN] epoch: 44, iter: 16130/40000, loss: 0.5544, lr: 0.006321, batch_cost: 0.8401, reader_cost: 0.00016, ips: 1.1903 samples/sec | ETA 05:34:13 2021-05-06 16:18:44 [INFO] [TRAIN] epoch: 44, iter: 16140/40000, loss: 0.6286, lr: 0.006319, batch_cost: 0.8416, reader_cost: 0.00016, ips: 1.1882 samples/sec | ETA 05:34:40 2021-05-06 16:18:53 [INFO] [TRAIN] epoch: 44, iter: 16150/40000, loss: 0.3308, lr: 0.006316, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 05:34:02 2021-05-06 16:19:01 [INFO] [TRAIN] epoch: 44, iter: 16160/40000, loss: 0.4103, lr: 0.006314, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1888 samples/sec | ETA 05:34:14 2021-05-06 16:19:09 [INFO] [TRAIN] epoch: 44, iter: 16170/40000, loss: 0.3946, lr: 0.006312, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1879 samples/sec | ETA 05:34:20 2021-05-06 16:19:18 [INFO] [TRAIN] epoch: 44, iter: 16180/40000, loss: 0.4650, lr: 0.006309, batch_cost: 0.8426, reader_cost: 0.00014, ips: 1.1868 samples/sec | ETA 05:34:30 2021-05-06 16:19:26 [INFO] [TRAIN] epoch: 44, iter: 16190/40000, loss: 0.8368, lr: 0.006307, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1886 samples/sec | ETA 05:33:52 2021-05-06 16:19:35 [INFO] [TRAIN] epoch: 44, iter: 16200/40000, loss: 0.4270, lr: 0.006305, batch_cost: 0.8425, reader_cost: 0.00017, ips: 1.1870 samples/sec | ETA 05:34:10 2021-05-06 16:19:43 [INFO] [TRAIN] epoch: 44, iter: 16210/40000, loss: 0.4388, lr: 0.006302, batch_cost: 0.8421, reader_cost: 0.00015, ips: 1.1875 samples/sec | ETA 05:33:53 2021-05-06 16:19:51 [INFO] [TRAIN] epoch: 44, iter: 16220/40000, loss: 0.3123, lr: 0.006300, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 05:33:10 2021-05-06 16:20:00 [INFO] [TRAIN] epoch: 44, iter: 16230/40000, loss: 0.2040, lr: 0.006298, batch_cost: 0.8420, reader_cost: 0.00015, ips: 1.1877 samples/sec | ETA 05:33:33 2021-05-06 16:20:08 [INFO] [TRAIN] epoch: 44, iter: 16240/40000, loss: 0.2240, lr: 0.006295, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 05:32:49 2021-05-06 16:20:17 [INFO] [TRAIN] epoch: 44, iter: 16250/40000, loss: 0.2429, lr: 0.006293, batch_cost: 0.8398, reader_cost: 0.00016, ips: 1.1908 samples/sec | ETA 05:32:25 2021-05-06 16:20:25 [INFO] [TRAIN] epoch: 44, iter: 16260/40000, loss: 0.3191, lr: 0.006291, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 05:32:36 2021-05-06 16:20:34 [INFO] [TRAIN] epoch: 44, iter: 16270/40000, loss: 0.2799, lr: 0.006288, batch_cost: 0.8389, reader_cost: 0.00014, ips: 1.1921 samples/sec | ETA 05:31:46 2021-05-06 16:20:42 [INFO] [TRAIN] epoch: 44, iter: 16280/40000, loss: 0.3798, lr: 0.006286, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1887 samples/sec | ETA 05:32:33 2021-05-06 16:20:50 [INFO] [TRAIN] epoch: 44, iter: 16290/40000, loss: 0.4636, lr: 0.006284, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 05:32:20 2021-05-06 16:20:59 [INFO] [TRAIN] epoch: 44, iter: 16300/40000, loss: 0.3724, lr: 0.006281, batch_cost: 0.8383, reader_cost: 0.00015, ips: 1.1928 samples/sec | ETA 05:31:08 2021-05-06 16:21:07 [INFO] [TRAIN] epoch: 44, iter: 16310/40000, loss: 0.3572, lr: 0.006279, batch_cost: 0.8388, reader_cost: 0.00015, ips: 1.1922 samples/sec | ETA 05:31:11 2021-05-06 16:21:16 [INFO] [TRAIN] epoch: 44, iter: 16320/40000, loss: 0.6765, lr: 0.006276, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 05:31:45 2021-05-06 16:21:24 [INFO] [TRAIN] epoch: 44, iter: 16330/40000, loss: 0.3709, lr: 0.006274, batch_cost: 0.8412, reader_cost: 0.00014, ips: 1.1887 samples/sec | ETA 05:31:52 2021-05-06 16:21:32 [INFO] [TRAIN] epoch: 44, iter: 16340/40000, loss: 0.4772, lr: 0.006272, batch_cost: 0.8406, reader_cost: 0.00014, ips: 1.1896 samples/sec | ETA 05:31:28 2021-05-06 16:21:41 [INFO] [TRAIN] epoch: 44, iter: 16350/40000, loss: 0.5118, lr: 0.006269, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1887 samples/sec | ETA 05:31:35 2021-05-06 16:21:49 [INFO] [TRAIN] epoch: 44, iter: 16360/40000, loss: 0.4480, lr: 0.006267, batch_cost: 0.8397, reader_cost: 0.00013, ips: 1.1909 samples/sec | ETA 05:30:50 2021-05-06 16:22:01 [INFO] [TRAIN] epoch: 45, iter: 16370/40000, loss: 0.3980, lr: 0.006265, batch_cost: 1.1562, reader_cost: 0.24187, ips: 0.8649 samples/sec | ETA 07:35:20 2021-05-06 16:22:09 [INFO] [TRAIN] epoch: 45, iter: 16380/40000, loss: 0.4739, lr: 0.006262, batch_cost: 0.8633, reader_cost: 0.00036, ips: 1.1584 samples/sec | ETA 05:39:50 2021-05-06 16:22:18 [INFO] [TRAIN] epoch: 45, iter: 16390/40000, loss: 0.4953, lr: 0.006260, batch_cost: 0.8422, reader_cost: 0.00015, ips: 1.1874 samples/sec | ETA 05:31:23 2021-05-06 16:22:26 [INFO] [TRAIN] epoch: 45, iter: 16400/40000, loss: 0.3965, lr: 0.006258, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 05:30:37 2021-05-06 16:22:35 [INFO] [TRAIN] epoch: 45, iter: 16410/40000, loss: 0.3880, lr: 0.006255, batch_cost: 0.8396, reader_cost: 0.00016, ips: 1.1911 samples/sec | ETA 05:30:05 2021-05-06 16:22:43 [INFO] [TRAIN] epoch: 45, iter: 16420/40000, loss: 0.6063, lr: 0.006253, batch_cost: 0.8442, reader_cost: 0.00017, ips: 1.1846 samples/sec | ETA 05:31:45 2021-05-06 16:22:51 [INFO] [TRAIN] epoch: 45, iter: 16430/40000, loss: 0.6280, lr: 0.006251, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1885 samples/sec | ETA 05:30:31 2021-05-06 16:23:00 [INFO] [TRAIN] epoch: 45, iter: 16440/40000, loss: 0.7353, lr: 0.006248, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1914 samples/sec | ETA 05:29:35 2021-05-06 16:23:08 [INFO] [TRAIN] epoch: 45, iter: 16450/40000, loss: 0.4249, lr: 0.006246, batch_cost: 0.8395, reader_cost: 0.00014, ips: 1.1911 samples/sec | ETA 05:29:31 2021-05-06 16:23:17 [INFO] [TRAIN] epoch: 45, iter: 16460/40000, loss: 0.4168, lr: 0.006244, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1913 samples/sec | ETA 05:29:20 2021-05-06 16:23:25 [INFO] [TRAIN] epoch: 45, iter: 16470/40000, loss: 0.2511, lr: 0.006241, batch_cost: 0.8410, reader_cost: 0.00016, ips: 1.1890 samples/sec | ETA 05:29:49 2021-05-06 16:23:33 [INFO] [TRAIN] epoch: 45, iter: 16480/40000, loss: 0.3342, lr: 0.006239, batch_cost: 0.8398, reader_cost: 0.00017, ips: 1.1908 samples/sec | ETA 05:29:11 2021-05-06 16:23:42 [INFO] [TRAIN] epoch: 45, iter: 16490/40000, loss: 0.6017, lr: 0.006237, batch_cost: 0.8402, reader_cost: 0.00016, ips: 1.1902 samples/sec | ETA 05:29:12 2021-05-06 16:23:50 [INFO] [TRAIN] epoch: 45, iter: 16500/40000, loss: 0.5344, lr: 0.006234, batch_cost: 0.8416, reader_cost: 0.00016, ips: 1.1882 samples/sec | ETA 05:29:37 2021-05-06 16:23:59 [INFO] [TRAIN] epoch: 45, iter: 16510/40000, loss: 0.6492, lr: 0.006232, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 05:28:34 2021-05-06 16:24:07 [INFO] [TRAIN] epoch: 45, iter: 16520/40000, loss: 0.4148, lr: 0.006230, batch_cost: 0.8386, reader_cost: 0.00015, ips: 1.1925 samples/sec | ETA 05:28:09 2021-05-06 16:24:15 [INFO] [TRAIN] epoch: 45, iter: 16530/40000, loss: 0.2810, lr: 0.006227, batch_cost: 0.8388, reader_cost: 0.00015, ips: 1.1922 samples/sec | ETA 05:28:07 2021-05-06 16:24:24 [INFO] [TRAIN] epoch: 45, iter: 16540/40000, loss: 0.2838, lr: 0.006225, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 05:28:29 2021-05-06 16:24:32 [INFO] [TRAIN] epoch: 45, iter: 16550/40000, loss: 0.7054, lr: 0.006222, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 05:28:18 2021-05-06 16:24:41 [INFO] [TRAIN] epoch: 45, iter: 16560/40000, loss: 0.5830, lr: 0.006220, batch_cost: 0.8391, reader_cost: 0.00014, ips: 1.1918 samples/sec | ETA 05:27:48 2021-05-06 16:24:49 [INFO] [TRAIN] epoch: 45, iter: 16570/40000, loss: 0.4639, lr: 0.006218, batch_cost: 0.8419, reader_cost: 0.00016, ips: 1.1878 samples/sec | ETA 05:28:46 2021-05-06 16:24:57 [INFO] [TRAIN] epoch: 45, iter: 16580/40000, loss: 0.3842, lr: 0.006215, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 05:28:05 2021-05-06 16:25:06 [INFO] [TRAIN] epoch: 45, iter: 16590/40000, loss: 0.3320, lr: 0.006213, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 05:28:05 2021-05-06 16:25:14 [INFO] [TRAIN] epoch: 45, iter: 16600/40000, loss: 0.2619, lr: 0.006211, batch_cost: 0.8419, reader_cost: 0.00016, ips: 1.1878 samples/sec | ETA 05:28:19 2021-05-06 16:25:23 [INFO] [TRAIN] epoch: 45, iter: 16610/40000, loss: 0.3493, lr: 0.006208, batch_cost: 0.8409, reader_cost: 0.00016, ips: 1.1892 samples/sec | ETA 05:27:48 2021-05-06 16:25:31 [INFO] [TRAIN] epoch: 45, iter: 16620/40000, loss: 0.4802, lr: 0.006206, batch_cost: 0.8402, reader_cost: 0.00016, ips: 1.1902 samples/sec | ETA 05:27:24 2021-05-06 16:25:39 [INFO] [TRAIN] epoch: 45, iter: 16630/40000, loss: 0.3990, lr: 0.006204, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 05:27:09 2021-05-06 16:25:48 [INFO] [TRAIN] epoch: 45, iter: 16640/40000, loss: 0.3042, lr: 0.006201, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 05:26:54 2021-05-06 16:25:56 [INFO] [TRAIN] epoch: 45, iter: 16650/40000, loss: 0.3946, lr: 0.006199, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 05:27:12 2021-05-06 16:26:05 [INFO] [TRAIN] epoch: 45, iter: 16660/40000, loss: 0.5327, lr: 0.006197, batch_cost: 0.8414, reader_cost: 0.00016, ips: 1.1885 samples/sec | ETA 05:27:17 2021-05-06 16:26:13 [INFO] [TRAIN] epoch: 45, iter: 16670/40000, loss: 0.4434, lr: 0.006194, batch_cost: 0.8396, reader_cost: 0.00016, ips: 1.1911 samples/sec | ETA 05:26:27 2021-05-06 16:26:21 [INFO] [TRAIN] epoch: 45, iter: 16680/40000, loss: 0.3055, lr: 0.006192, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1912 samples/sec | ETA 05:26:16 2021-05-06 16:26:30 [INFO] [TRAIN] epoch: 45, iter: 16690/40000, loss: 0.5186, lr: 0.006190, batch_cost: 0.8405, reader_cost: 0.00014, ips: 1.1898 samples/sec | ETA 05:26:32 2021-05-06 16:26:38 [INFO] [TRAIN] epoch: 45, iter: 16700/40000, loss: 0.3553, lr: 0.006187, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 05:26:32 2021-05-06 16:26:47 [INFO] [TRAIN] epoch: 45, iter: 16710/40000, loss: 0.4102, lr: 0.006185, batch_cost: 0.8402, reader_cost: 0.00016, ips: 1.1902 samples/sec | ETA 05:26:07 2021-05-06 16:26:55 [INFO] [TRAIN] epoch: 45, iter: 16720/40000, loss: 0.3000, lr: 0.006183, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 05:26:15 2021-05-06 16:27:03 [INFO] [TRAIN] epoch: 45, iter: 16730/40000, loss: 0.5082, lr: 0.006180, batch_cost: 0.8391, reader_cost: 0.00014, ips: 1.1918 samples/sec | ETA 05:25:25 2021-05-06 16:27:12 [INFO] [TRAIN] epoch: 45, iter: 16740/40000, loss: 0.3865, lr: 0.006178, batch_cost: 0.8393, reader_cost: 0.00009, ips: 1.1914 samples/sec | ETA 05:25:22 2021-05-06 16:27:23 [INFO] [TRAIN] epoch: 46, iter: 16750/40000, loss: 0.4980, lr: 0.006175, batch_cost: 1.1549, reader_cost: 0.21103, ips: 0.8659 samples/sec | ETA 07:27:30 2021-05-06 16:27:32 [INFO] [TRAIN] epoch: 46, iter: 16760/40000, loss: 0.4237, lr: 0.006173, batch_cost: 0.8426, reader_cost: 0.00031, ips: 1.1869 samples/sec | ETA 05:26:21 2021-05-06 16:27:40 [INFO] [TRAIN] epoch: 46, iter: 16770/40000, loss: 0.4341, lr: 0.006171, batch_cost: 0.8419, reader_cost: 0.00015, ips: 1.1877 samples/sec | ETA 05:25:57 2021-05-06 16:27:49 [INFO] [TRAIN] epoch: 46, iter: 16780/40000, loss: 0.3570, lr: 0.006168, batch_cost: 0.8404, reader_cost: 0.00014, ips: 1.1899 samples/sec | ETA 05:25:14 2021-05-06 16:27:57 [INFO] [TRAIN] epoch: 46, iter: 16790/40000, loss: 0.5167, lr: 0.006166, batch_cost: 0.8428, reader_cost: 0.00015, ips: 1.1865 samples/sec | ETA 05:26:01 2021-05-06 16:28:06 [INFO] [TRAIN] epoch: 46, iter: 16800/40000, loss: 0.7161, lr: 0.006164, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 05:24:42 2021-05-06 16:28:14 [INFO] [TRAIN] epoch: 46, iter: 16810/40000, loss: 0.5862, lr: 0.006161, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 05:24:39 2021-05-06 16:28:22 [INFO] [TRAIN] epoch: 46, iter: 16820/40000, loss: 0.6028, lr: 0.006159, batch_cost: 0.8406, reader_cost: 0.00016, ips: 1.1896 samples/sec | ETA 05:24:45 2021-05-06 16:28:31 [INFO] [TRAIN] epoch: 46, iter: 16830/40000, loss: 0.4391, lr: 0.006157, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 05:24:34 2021-05-06 16:28:39 [INFO] [TRAIN] epoch: 46, iter: 16840/40000, loss: 0.1377, lr: 0.006154, batch_cost: 0.8393, reader_cost: 0.00016, ips: 1.1914 samples/sec | ETA 05:23:58 2021-05-06 16:28:48 [INFO] [TRAIN] epoch: 46, iter: 16850/40000, loss: 0.3901, lr: 0.006152, batch_cost: 0.8399, reader_cost: 0.00016, ips: 1.1906 samples/sec | ETA 05:24:03 2021-05-06 16:28:56 [INFO] [TRAIN] epoch: 46, iter: 16860/40000, loss: 0.4002, lr: 0.006150, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 05:24:07 2021-05-06 16:29:04 [INFO] [TRAIN] epoch: 46, iter: 16870/40000, loss: 0.5475, lr: 0.006147, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 05:23:50 2021-05-06 16:29:13 [INFO] [TRAIN] epoch: 46, iter: 16880/40000, loss: 0.5248, lr: 0.006145, batch_cost: 0.8396, reader_cost: 0.00016, ips: 1.1910 samples/sec | ETA 05:23:32 2021-05-06 16:29:21 [INFO] [TRAIN] epoch: 46, iter: 16890/40000, loss: 0.4752, lr: 0.006143, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1912 samples/sec | ETA 05:23:20 2021-05-06 16:29:30 [INFO] [TRAIN] epoch: 46, iter: 16900/40000, loss: 0.3270, lr: 0.006140, batch_cost: 0.8413, reader_cost: 0.00014, ips: 1.1886 samples/sec | ETA 05:23:54 2021-05-06 16:29:38 [INFO] [TRAIN] epoch: 46, iter: 16910/40000, loss: 0.3494, lr: 0.006138, batch_cost: 0.8390, reader_cost: 0.00015, ips: 1.1919 samples/sec | ETA 05:22:51 2021-05-06 16:29:46 [INFO] [TRAIN] epoch: 46, iter: 16920/40000, loss: 0.7553, lr: 0.006135, batch_cost: 0.8400, reader_cost: 0.00014, ips: 1.1904 samples/sec | ETA 05:23:07 2021-05-06 16:29:55 [INFO] [TRAIN] epoch: 46, iter: 16930/40000, loss: 0.5164, lr: 0.006133, batch_cost: 0.8403, reader_cost: 0.00014, ips: 1.1900 samples/sec | ETA 05:23:06 2021-05-06 16:30:03 [INFO] [TRAIN] epoch: 46, iter: 16940/40000, loss: 0.8364, lr: 0.006131, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 05:22:43 2021-05-06 16:30:12 [INFO] [TRAIN] epoch: 46, iter: 16950/40000, loss: 0.3962, lr: 0.006128, batch_cost: 0.8408, reader_cost: 0.00014, ips: 1.1893 samples/sec | ETA 05:23:01 2021-05-06 16:30:20 [INFO] [TRAIN] epoch: 46, iter: 16960/40000, loss: 0.5370, lr: 0.006126, batch_cost: 0.8398, reader_cost: 0.00016, ips: 1.1908 samples/sec | ETA 05:22:28 2021-05-06 16:30:28 [INFO] [TRAIN] epoch: 46, iter: 16970/40000, loss: 0.2184, lr: 0.006124, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 05:22:22 2021-05-06 16:30:37 [INFO] [TRAIN] epoch: 46, iter: 16980/40000, loss: 0.4830, lr: 0.006121, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 05:22:37 2021-05-06 16:30:45 [INFO] [TRAIN] epoch: 46, iter: 16990/40000, loss: 0.3055, lr: 0.006119, batch_cost: 0.8407, reader_cost: 0.00014, ips: 1.1896 samples/sec | ETA 05:22:23 2021-05-06 16:30:54 [INFO] [TRAIN] epoch: 46, iter: 17000/40000, loss: 0.4048, lr: 0.006117, batch_cost: 0.8385, reader_cost: 0.00014, ips: 1.1926 samples/sec | ETA 05:21:26 2021-05-06 16:30:54 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 16:34:30 [INFO] [EVAL] #Images: 500 mIoU: 0.7317 Acc: 0.9484 Kappa: 0.9329 2021-05-06 16:34:30 [INFO] [EVAL] Class IoU: [0.9779 0.8249 0.895 0.5592 0.5619 0.4494 0.5557 0.7056 0.8971 0.625 0.9367 0.76 0.5412 0.932 0.7517 0.8616 0.7567 0.6039 0.7078] 2021-05-06 16:34:30 [INFO] [EVAL] Class Acc: [0.9896 0.9081 0.9111 0.9007 0.7767 0.7387 0.7188 0.8514 0.9719 0.7326 0.9755 0.8545 0.7733 0.9565 0.8901 0.9085 0.8658 0.8056 0.8142] 2021-05-06 16:34:59 [INFO] [EVAL] The model with the best validation mIoU (0.7373) was saved at iter 15000. 2021-05-06 16:35:07 [INFO] [TRAIN] epoch: 46, iter: 17010/40000, loss: 0.3176, lr: 0.006114, batch_cost: 0.8389, reader_cost: 0.00023, ips: 1.1921 samples/sec | ETA 05:21:25 2021-05-06 16:35:16 [INFO] [TRAIN] epoch: 46, iter: 17020/40000, loss: 0.3860, lr: 0.006112, batch_cost: 0.8378, reader_cost: 0.00015, ips: 1.1936 samples/sec | ETA 05:20:53 2021-05-06 16:35:24 [INFO] [TRAIN] epoch: 46, iter: 17030/40000, loss: 0.4998, lr: 0.006110, batch_cost: 0.8381, reader_cost: 0.00016, ips: 1.1932 samples/sec | ETA 05:20:50 2021-05-06 16:35:32 [INFO] [TRAIN] epoch: 46, iter: 17040/40000, loss: 0.5453, lr: 0.006107, batch_cost: 0.8375, reader_cost: 0.00016, ips: 1.1940 samples/sec | ETA 05:20:29 2021-05-06 16:35:41 [INFO] [TRAIN] epoch: 46, iter: 17050/40000, loss: 0.3861, lr: 0.006105, batch_cost: 0.8381, reader_cost: 0.00015, ips: 1.1932 samples/sec | ETA 05:20:34 2021-05-06 16:35:49 [INFO] [TRAIN] epoch: 46, iter: 17060/40000, loss: 0.4650, lr: 0.006103, batch_cost: 0.8382, reader_cost: 0.00015, ips: 1.1930 samples/sec | ETA 05:20:28 2021-05-06 16:35:58 [INFO] [TRAIN] epoch: 46, iter: 17070/40000, loss: 0.4904, lr: 0.006100, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1911 samples/sec | ETA 05:20:51 2021-05-06 16:36:06 [INFO] [TRAIN] epoch: 46, iter: 17080/40000, loss: 0.4904, lr: 0.006098, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1914 samples/sec | ETA 05:20:37 2021-05-06 16:36:14 [INFO] [TRAIN] epoch: 46, iter: 17090/40000, loss: 0.5191, lr: 0.006095, batch_cost: 0.8419, reader_cost: 0.00016, ips: 1.1877 samples/sec | ETA 05:21:28 2021-05-06 16:36:23 [INFO] [TRAIN] epoch: 46, iter: 17100/40000, loss: 0.5249, lr: 0.006093, batch_cost: 0.8411, reader_cost: 0.00016, ips: 1.1889 samples/sec | ETA 05:21:02 2021-05-06 16:36:31 [INFO] [TRAIN] epoch: 46, iter: 17110/40000, loss: 0.4400, lr: 0.006091, batch_cost: 0.8383, reader_cost: 0.00009, ips: 1.1929 samples/sec | ETA 05:19:48 2021-05-06 16:36:43 [INFO] [TRAIN] epoch: 47, iter: 17120/40000, loss: 0.6537, lr: 0.006088, batch_cost: 1.1802, reader_cost: 0.21122, ips: 0.8473 samples/sec | ETA 07:30:02 2021-05-06 16:36:51 [INFO] [TRAIN] epoch: 47, iter: 17130/40000, loss: 0.4586, lr: 0.006086, batch_cost: 0.8427, reader_cost: 0.00029, ips: 1.1867 samples/sec | ETA 05:21:12 2021-05-06 16:37:00 [INFO] [TRAIN] epoch: 47, iter: 17140/40000, loss: 0.5353, lr: 0.006084, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1918 samples/sec | ETA 05:19:41 2021-05-06 16:37:08 [INFO] [TRAIN] epoch: 47, iter: 17150/40000, loss: 0.4258, lr: 0.006081, batch_cost: 0.8396, reader_cost: 0.00017, ips: 1.1910 samples/sec | ETA 05:19:45 2021-05-06 16:37:17 [INFO] [TRAIN] epoch: 47, iter: 17160/40000, loss: 0.4841, lr: 0.006079, batch_cost: 0.8408, reader_cost: 0.00016, ips: 1.1894 samples/sec | ETA 05:20:02 2021-05-06 16:37:25 [INFO] [TRAIN] epoch: 47, iter: 17170/40000, loss: 0.9331, lr: 0.006077, batch_cost: 0.8405, reader_cost: 0.00014, ips: 1.1897 samples/sec | ETA 05:19:49 2021-05-06 16:37:33 [INFO] [TRAIN] epoch: 47, iter: 17180/40000, loss: 0.5116, lr: 0.006074, batch_cost: 0.8388, reader_cost: 0.00015, ips: 1.1922 samples/sec | ETA 05:19:00 2021-05-06 16:37:42 [INFO] [TRAIN] epoch: 47, iter: 17190/40000, loss: 0.6082, lr: 0.006072, batch_cost: 0.8383, reader_cost: 0.00016, ips: 1.1929 samples/sec | ETA 05:18:41 2021-05-06 16:37:50 [INFO] [TRAIN] epoch: 47, iter: 17200/40000, loss: 0.4090, lr: 0.006070, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 05:19:31 2021-05-06 16:37:59 [INFO] [TRAIN] epoch: 47, iter: 17210/40000, loss: 0.1923, lr: 0.006067, batch_cost: 0.8399, reader_cost: 0.00016, ips: 1.1906 samples/sec | ETA 05:19:01 2021-05-06 16:38:07 [INFO] [TRAIN] epoch: 47, iter: 17220/40000, loss: 0.2436, lr: 0.006065, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1888 samples/sec | ETA 05:19:21 2021-05-06 16:38:16 [INFO] [TRAIN] epoch: 47, iter: 17230/40000, loss: 0.4530, lr: 0.006062, batch_cost: 0.8424, reader_cost: 0.00016, ips: 1.1871 samples/sec | ETA 05:19:41 2021-05-06 16:38:24 [INFO] [TRAIN] epoch: 47, iter: 17240/40000, loss: 0.5207, lr: 0.006060, batch_cost: 0.8407, reader_cost: 0.00014, ips: 1.1895 samples/sec | ETA 05:18:54 2021-05-06 16:38:32 [INFO] [TRAIN] epoch: 47, iter: 17250/40000, loss: 0.5895, lr: 0.006058, batch_cost: 0.8396, reader_cost: 0.00017, ips: 1.1910 samples/sec | ETA 05:18:21 2021-05-06 16:38:41 [INFO] [TRAIN] epoch: 47, iter: 17260/40000, loss: 0.4074, lr: 0.006055, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 05:18:05 2021-05-06 16:38:49 [INFO] [TRAIN] epoch: 47, iter: 17270/40000, loss: 0.2878, lr: 0.006053, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 05:18:37 2021-05-06 16:38:58 [INFO] [TRAIN] epoch: 47, iter: 17280/40000, loss: 0.3417, lr: 0.006051, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 05:17:48 2021-05-06 16:39:06 [INFO] [TRAIN] epoch: 47, iter: 17290/40000, loss: 0.3060, lr: 0.006048, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 05:18:11 2021-05-06 16:39:14 [INFO] [TRAIN] epoch: 47, iter: 17300/40000, loss: 0.5517, lr: 0.006046, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 05:17:59 2021-05-06 16:39:23 [INFO] [TRAIN] epoch: 47, iter: 17310/40000, loss: 0.6165, lr: 0.006044, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 05:17:45 2021-05-06 16:39:31 [INFO] [TRAIN] epoch: 47, iter: 17320/40000, loss: 0.2688, lr: 0.006041, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1887 samples/sec | ETA 05:17:59 2021-05-06 16:39:40 [INFO] [TRAIN] epoch: 47, iter: 17330/40000, loss: 0.4838, lr: 0.006039, batch_cost: 0.8411, reader_cost: 0.00016, ips: 1.1890 samples/sec | ETA 05:17:47 2021-05-06 16:39:48 [INFO] [TRAIN] epoch: 47, iter: 17340/40000, loss: 0.2524, lr: 0.006037, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 05:17:07 2021-05-06 16:39:56 [INFO] [TRAIN] epoch: 47, iter: 17350/40000, loss: 0.3696, lr: 0.006034, batch_cost: 0.8408, reader_cost: 0.00014, ips: 1.1893 samples/sec | ETA 05:17:24 2021-05-06 16:40:05 [INFO] [TRAIN] epoch: 47, iter: 17360/40000, loss: 0.3055, lr: 0.006032, batch_cost: 0.8404, reader_cost: 0.00016, ips: 1.1899 samples/sec | ETA 05:17:06 2021-05-06 16:40:13 [INFO] [TRAIN] epoch: 47, iter: 17370/40000, loss: 0.4610, lr: 0.006029, batch_cost: 0.8404, reader_cost: 0.00014, ips: 1.1899 samples/sec | ETA 05:16:58 2021-05-06 16:40:22 [INFO] [TRAIN] epoch: 47, iter: 17380/40000, loss: 0.3531, lr: 0.006027, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 05:16:33 2021-05-06 16:40:30 [INFO] [TRAIN] epoch: 47, iter: 17390/40000, loss: 0.4856, lr: 0.006025, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1880 samples/sec | ETA 05:17:11 2021-05-06 16:40:38 [INFO] [TRAIN] epoch: 47, iter: 17400/40000, loss: 0.4076, lr: 0.006022, batch_cost: 0.8412, reader_cost: 0.00018, ips: 1.1887 samples/sec | ETA 05:16:52 2021-05-06 16:40:47 [INFO] [TRAIN] epoch: 47, iter: 17410/40000, loss: 0.4946, lr: 0.006020, batch_cost: 0.8430, reader_cost: 0.00014, ips: 1.1863 samples/sec | ETA 05:17:22 2021-05-06 16:40:55 [INFO] [TRAIN] epoch: 47, iter: 17420/40000, loss: 0.3161, lr: 0.006018, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1904 samples/sec | ETA 05:16:08 2021-05-06 16:41:04 [INFO] [TRAIN] epoch: 47, iter: 17430/40000, loss: 0.3777, lr: 0.006015, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 05:16:18 2021-05-06 16:41:12 [INFO] [TRAIN] epoch: 47, iter: 17440/40000, loss: 0.5311, lr: 0.006013, batch_cost: 0.8409, reader_cost: 0.00014, ips: 1.1892 samples/sec | ETA 05:16:10 2021-05-06 16:41:20 [INFO] [TRAIN] epoch: 47, iter: 17450/40000, loss: 0.4042, lr: 0.006011, batch_cost: 0.8408, reader_cost: 0.00014, ips: 1.1894 samples/sec | ETA 05:15:59 2021-05-06 16:41:29 [INFO] [TRAIN] epoch: 47, iter: 17460/40000, loss: 0.4172, lr: 0.006008, batch_cost: 0.8425, reader_cost: 0.00014, ips: 1.1869 samples/sec | ETA 05:16:30 2021-05-06 16:41:37 [INFO] [TRAIN] epoch: 47, iter: 17470/40000, loss: 0.4677, lr: 0.006006, batch_cost: 0.8411, reader_cost: 0.00016, ips: 1.1889 samples/sec | ETA 05:15:50 2021-05-06 16:41:46 [INFO] [TRAIN] epoch: 47, iter: 17480/40000, loss: 0.4303, lr: 0.006004, batch_cost: 0.8389, reader_cost: 0.00010, ips: 1.1921 samples/sec | ETA 05:14:51 2021-05-06 16:41:57 [INFO] [TRAIN] epoch: 48, iter: 17490/40000, loss: 0.5291, lr: 0.006001, batch_cost: 1.1764, reader_cost: 0.22529, ips: 0.8501 samples/sec | ETA 07:21:19 2021-05-06 16:42:06 [INFO] [TRAIN] epoch: 48, iter: 17500/40000, loss: 0.3568, lr: 0.005999, batch_cost: 0.8402, reader_cost: 0.00031, ips: 1.1901 samples/sec | ETA 05:15:05 2021-05-06 16:42:14 [INFO] [TRAIN] epoch: 48, iter: 17510/40000, loss: 0.5186, lr: 0.005996, batch_cost: 0.8415, reader_cost: 0.00016, ips: 1.1883 samples/sec | ETA 05:15:25 2021-05-06 16:42:23 [INFO] [TRAIN] epoch: 48, iter: 17520/40000, loss: 0.5463, lr: 0.005994, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 05:14:58 2021-05-06 16:42:31 [INFO] [TRAIN] epoch: 48, iter: 17530/40000, loss: 0.3873, lr: 0.005992, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 05:14:27 2021-05-06 16:42:40 [INFO] [TRAIN] epoch: 48, iter: 17540/40000, loss: 0.7814, lr: 0.005989, batch_cost: 0.8421, reader_cost: 0.00015, ips: 1.1876 samples/sec | ETA 05:15:12 2021-05-06 16:42:48 [INFO] [TRAIN] epoch: 48, iter: 17550/40000, loss: 0.5039, lr: 0.005987, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 05:14:20 2021-05-06 16:42:56 [INFO] [TRAIN] epoch: 48, iter: 17560/40000, loss: 0.6279, lr: 0.005985, batch_cost: 0.8387, reader_cost: 0.00015, ips: 1.1923 samples/sec | ETA 05:13:40 2021-05-06 16:43:05 [INFO] [TRAIN] epoch: 48, iter: 17570/40000, loss: 0.5291, lr: 0.005982, batch_cost: 0.8387, reader_cost: 0.00015, ips: 1.1923 samples/sec | ETA 05:13:31 2021-05-06 16:43:13 [INFO] [TRAIN] epoch: 48, iter: 17580/40000, loss: 0.2931, lr: 0.005980, batch_cost: 0.8377, reader_cost: 0.00015, ips: 1.1938 samples/sec | ETA 05:13:00 2021-05-06 16:43:21 [INFO] [TRAIN] epoch: 48, iter: 17590/40000, loss: 0.3448, lr: 0.005978, batch_cost: 0.8398, reader_cost: 0.00014, ips: 1.1907 samples/sec | ETA 05:13:40 2021-05-06 16:43:30 [INFO] [TRAIN] epoch: 48, iter: 17600/40000, loss: 0.3323, lr: 0.005975, batch_cost: 0.8393, reader_cost: 0.00016, ips: 1.1915 samples/sec | ETA 05:13:20 2021-05-06 16:43:38 [INFO] [TRAIN] epoch: 48, iter: 17610/40000, loss: 0.4375, lr: 0.005973, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 05:13:38 2021-05-06 16:43:47 [INFO] [TRAIN] epoch: 48, iter: 17620/40000, loss: 0.7213, lr: 0.005970, batch_cost: 0.8400, reader_cost: 0.00014, ips: 1.1904 samples/sec | ETA 05:13:20 2021-05-06 16:43:55 [INFO] [TRAIN] epoch: 48, iter: 17630/40000, loss: 0.5187, lr: 0.005968, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 05:13:15 2021-05-06 16:44:03 [INFO] [TRAIN] epoch: 48, iter: 17640/40000, loss: 0.3213, lr: 0.005966, batch_cost: 0.8418, reader_cost: 0.00017, ips: 1.1880 samples/sec | ETA 05:13:42 2021-05-06 16:44:12 [INFO] [TRAIN] epoch: 48, iter: 17650/40000, loss: 0.4368, lr: 0.005963, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 05:13:12 2021-05-06 16:44:20 [INFO] [TRAIN] epoch: 48, iter: 17660/40000, loss: 0.3580, lr: 0.005961, batch_cost: 0.8413, reader_cost: 0.00016, ips: 1.1886 samples/sec | ETA 05:13:14 2021-05-06 16:44:29 [INFO] [TRAIN] epoch: 48, iter: 17670/40000, loss: 0.4116, lr: 0.005959, batch_cost: 0.8403, reader_cost: 0.00016, ips: 1.1900 samples/sec | ETA 05:12:44 2021-05-06 16:44:37 [INFO] [TRAIN] epoch: 48, iter: 17680/40000, loss: 0.5196, lr: 0.005956, batch_cost: 0.8410, reader_cost: 0.00016, ips: 1.1891 samples/sec | ETA 05:12:50 2021-05-06 16:44:46 [INFO] [TRAIN] epoch: 48, iter: 17690/40000, loss: 0.3688, lr: 0.005954, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1885 samples/sec | ETA 05:12:51 2021-05-06 16:44:54 [INFO] [TRAIN] epoch: 48, iter: 17700/40000, loss: 0.4492, lr: 0.005952, batch_cost: 0.8407, reader_cost: 0.00016, ips: 1.1895 samples/sec | ETA 05:12:27 2021-05-06 16:45:02 [INFO] [TRAIN] epoch: 48, iter: 17710/40000, loss: 0.2624, lr: 0.005949, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1914 samples/sec | ETA 05:11:49 2021-05-06 16:45:11 [INFO] [TRAIN] epoch: 48, iter: 17720/40000, loss: 0.3726, lr: 0.005947, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 05:11:57 2021-05-06 16:45:19 [INFO] [TRAIN] epoch: 48, iter: 17730/40000, loss: 0.3484, lr: 0.005944, batch_cost: 0.8426, reader_cost: 0.00017, ips: 1.1867 samples/sec | ETA 05:12:45 2021-05-06 16:45:28 [INFO] [TRAIN] epoch: 48, iter: 17740/40000, loss: 0.3039, lr: 0.005942, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 05:11:51 2021-05-06 16:45:36 [INFO] [TRAIN] epoch: 48, iter: 17750/40000, loss: 0.2918, lr: 0.005940, batch_cost: 0.8418, reader_cost: 0.00016, ips: 1.1880 samples/sec | ETA 05:12:09 2021-05-06 16:45:44 [INFO] [TRAIN] epoch: 48, iter: 17760/40000, loss: 0.4526, lr: 0.005937, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 05:11:38 2021-05-06 16:45:53 [INFO] [TRAIN] epoch: 48, iter: 17770/40000, loss: 0.4479, lr: 0.005935, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 05:11:18 2021-05-06 16:46:01 [INFO] [TRAIN] epoch: 48, iter: 17780/40000, loss: 0.5443, lr: 0.005933, batch_cost: 0.8402, reader_cost: 0.00016, ips: 1.1902 samples/sec | ETA 05:11:09 2021-05-06 16:46:10 [INFO] [TRAIN] epoch: 48, iter: 17790/40000, loss: 0.3424, lr: 0.005930, batch_cost: 0.8409, reader_cost: 0.00014, ips: 1.1893 samples/sec | ETA 05:11:15 2021-05-06 16:46:18 [INFO] [TRAIN] epoch: 48, iter: 17800/40000, loss: 0.4748, lr: 0.005928, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1885 samples/sec | ETA 05:11:18 2021-05-06 16:46:26 [INFO] [TRAIN] epoch: 48, iter: 17810/40000, loss: 0.5149, lr: 0.005926, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 05:10:39 2021-05-06 16:46:35 [INFO] [TRAIN] epoch: 48, iter: 17820/40000, loss: 0.3134, lr: 0.005923, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 05:10:48 2021-05-06 16:46:43 [INFO] [TRAIN] epoch: 48, iter: 17830/40000, loss: 0.4504, lr: 0.005921, batch_cost: 0.8425, reader_cost: 0.00018, ips: 1.1869 samples/sec | ETA 05:11:19 2021-05-06 16:46:52 [INFO] [TRAIN] epoch: 48, iter: 17840/40000, loss: 0.3183, lr: 0.005919, batch_cost: 0.8427, reader_cost: 0.00017, ips: 1.1867 samples/sec | ETA 05:11:13 2021-05-06 16:47:00 [INFO] [TRAIN] epoch: 48, iter: 17850/40000, loss: 0.3784, lr: 0.005916, batch_cost: 0.8388, reader_cost: 0.00012, ips: 1.1921 samples/sec | ETA 05:09:40 2021-05-06 16:47:12 [INFO] [TRAIN] epoch: 49, iter: 17860/40000, loss: 0.3893, lr: 0.005914, batch_cost: 1.1505, reader_cost: 0.22978, ips: 0.8692 samples/sec | ETA 07:04:31 2021-05-06 16:47:20 [INFO] [TRAIN] epoch: 49, iter: 17870/40000, loss: 0.4385, lr: 0.005911, batch_cost: 0.8574, reader_cost: 0.00032, ips: 1.1663 samples/sec | ETA 05:16:15 2021-05-06 16:47:29 [INFO] [TRAIN] epoch: 49, iter: 17880/40000, loss: 0.5020, lr: 0.005909, batch_cost: 0.8401, reader_cost: 0.00016, ips: 1.1903 samples/sec | ETA 05:09:43 2021-05-06 16:47:37 [INFO] [TRAIN] epoch: 49, iter: 17890/40000, loss: 0.4602, lr: 0.005907, batch_cost: 0.8408, reader_cost: 0.00017, ips: 1.1893 samples/sec | ETA 05:09:51 2021-05-06 16:47:45 [INFO] [TRAIN] epoch: 49, iter: 17900/40000, loss: 0.3214, lr: 0.005904, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 05:09:36 2021-05-06 16:47:54 [INFO] [TRAIN] epoch: 49, iter: 17910/40000, loss: 0.5169, lr: 0.005902, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 05:09:34 2021-05-06 16:48:02 [INFO] [TRAIN] epoch: 49, iter: 17920/40000, loss: 0.5899, lr: 0.005900, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1884 samples/sec | ETA 05:09:39 2021-05-06 16:48:11 [INFO] [TRAIN] epoch: 49, iter: 17930/40000, loss: 0.7062, lr: 0.005897, batch_cost: 0.8408, reader_cost: 0.00016, ips: 1.1894 samples/sec | ETA 05:09:16 2021-05-06 16:48:19 [INFO] [TRAIN] epoch: 49, iter: 17940/40000, loss: 0.4788, lr: 0.005895, batch_cost: 0.8420, reader_cost: 0.00017, ips: 1.1876 samples/sec | ETA 05:09:34 2021-05-06 16:48:27 [INFO] [TRAIN] epoch: 49, iter: 17950/40000, loss: 0.2660, lr: 0.005893, batch_cost: 0.8427, reader_cost: 0.00016, ips: 1.1867 samples/sec | ETA 05:09:41 2021-05-06 16:48:36 [INFO] [TRAIN] epoch: 49, iter: 17960/40000, loss: 0.2583, lr: 0.005890, batch_cost: 0.8427, reader_cost: 0.00017, ips: 1.1867 samples/sec | ETA 05:09:33 2021-05-06 16:48:44 [INFO] [TRAIN] epoch: 49, iter: 17970/40000, loss: 0.2664, lr: 0.005888, batch_cost: 0.8412, reader_cost: 0.00017, ips: 1.1887 samples/sec | ETA 05:08:52 2021-05-06 16:48:53 [INFO] [TRAIN] epoch: 49, iter: 17980/40000, loss: 0.3360, lr: 0.005885, batch_cost: 0.8409, reader_cost: 0.00017, ips: 1.1892 samples/sec | ETA 05:08:36 2021-05-06 16:49:01 [INFO] [TRAIN] epoch: 49, iter: 17990/40000, loss: 0.7071, lr: 0.005883, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 05:08:03 2021-05-06 16:49:10 [INFO] [TRAIN] epoch: 49, iter: 18000/40000, loss: 0.5227, lr: 0.005881, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 05:08:20 2021-05-06 16:49:10 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 16:52:46 [INFO] [EVAL] #Images: 500 mIoU: 0.7085 Acc: 0.9512 Kappa: 0.9365 2021-05-06 16:52:46 [INFO] [EVAL] Class IoU: [0.9792 0.8391 0.9075 0.604 0.462 0.463 0.5232 0.6625 0.9107 0.6514 0.9396 0.7379 0.4914 0.929 0.6268 0.8515 0.6704 0.5604 0.6516] 2021-05-06 16:52:46 [INFO] [EVAL] Class Acc: [0.9907 0.9134 0.9329 0.7071 0.8968 0.7784 0.8466 0.9089 0.9421 0.8658 0.9707 0.8601 0.7455 0.9614 0.6655 0.905 0.9384 0.828 0.8828] 2021-05-06 16:53:15 [INFO] [EVAL] The model with the best validation mIoU (0.7373) was saved at iter 15000. 2021-05-06 16:53:23 [INFO] [TRAIN] epoch: 49, iter: 18010/40000, loss: 0.4032, lr: 0.005878, batch_cost: 0.8384, reader_cost: 0.00016, ips: 1.1927 samples/sec | ETA 05:07:17 2021-05-06 16:53:31 [INFO] [TRAIN] epoch: 49, iter: 18020/40000, loss: 0.5295, lr: 0.005876, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 05:07:36 2021-05-06 16:53:40 [INFO] [TRAIN] epoch: 49, iter: 18030/40000, loss: 0.3991, lr: 0.005874, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 05:07:26 2021-05-06 16:53:48 [INFO] [TRAIN] epoch: 49, iter: 18040/40000, loss: 0.3830, lr: 0.005871, batch_cost: 0.8397, reader_cost: 0.00014, ips: 1.1909 samples/sec | ETA 05:07:20 2021-05-06 16:53:57 [INFO] [TRAIN] epoch: 49, iter: 18050/40000, loss: 0.5904, lr: 0.005869, batch_cost: 0.8399, reader_cost: 0.00016, ips: 1.1907 samples/sec | ETA 05:07:14 2021-05-06 16:54:05 [INFO] [TRAIN] epoch: 49, iter: 18060/40000, loss: 0.6515, lr: 0.005866, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1885 samples/sec | ETA 05:07:40 2021-05-06 16:54:13 [INFO] [TRAIN] epoch: 49, iter: 18070/40000, loss: 0.3849, lr: 0.005864, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1880 samples/sec | ETA 05:07:39 2021-05-06 16:54:22 [INFO] [TRAIN] epoch: 49, iter: 18080/40000, loss: 0.2443, lr: 0.005862, batch_cost: 0.8405, reader_cost: 0.00016, ips: 1.1898 samples/sec | ETA 05:07:02 2021-05-06 16:54:30 [INFO] [TRAIN] epoch: 49, iter: 18090/40000, loss: 0.3150, lr: 0.005859, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1886 samples/sec | ETA 05:07:14 2021-05-06 16:54:39 [INFO] [TRAIN] epoch: 49, iter: 18100/40000, loss: 0.2731, lr: 0.005857, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 05:06:55 2021-05-06 16:54:47 [INFO] [TRAIN] epoch: 49, iter: 18110/40000, loss: 0.3113, lr: 0.005855, batch_cost: 0.8409, reader_cost: 0.00014, ips: 1.1893 samples/sec | ETA 05:06:46 2021-05-06 16:54:56 [INFO] [TRAIN] epoch: 49, iter: 18120/40000, loss: 0.3172, lr: 0.005852, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 05:06:25 2021-05-06 16:55:04 [INFO] [TRAIN] epoch: 49, iter: 18130/40000, loss: 0.3157, lr: 0.005850, batch_cost: 0.8396, reader_cost: 0.00014, ips: 1.1910 samples/sec | ETA 05:06:02 2021-05-06 16:55:12 [INFO] [TRAIN] epoch: 49, iter: 18140/40000, loss: 0.2951, lr: 0.005848, batch_cost: 0.8424, reader_cost: 0.00014, ips: 1.1871 samples/sec | ETA 05:06:55 2021-05-06 16:55:21 [INFO] [TRAIN] epoch: 49, iter: 18150/40000, loss: 0.4699, lr: 0.005845, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 05:05:57 2021-05-06 16:55:29 [INFO] [TRAIN] epoch: 49, iter: 18160/40000, loss: 0.4743, lr: 0.005843, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 05:05:53 2021-05-06 16:55:38 [INFO] [TRAIN] epoch: 49, iter: 18170/40000, loss: 0.4391, lr: 0.005840, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 05:05:29 2021-05-06 16:55:46 [INFO] [TRAIN] epoch: 49, iter: 18180/40000, loss: 0.6794, lr: 0.005838, batch_cost: 0.8404, reader_cost: 0.00014, ips: 1.1899 samples/sec | ETA 05:05:38 2021-05-06 16:55:54 [INFO] [TRAIN] epoch: 49, iter: 18190/40000, loss: 0.6906, lr: 0.005836, batch_cost: 0.8410, reader_cost: 0.00014, ips: 1.1891 samples/sec | ETA 05:05:41 2021-05-06 16:56:03 [INFO] [TRAIN] epoch: 49, iter: 18200/40000, loss: 0.4867, lr: 0.005833, batch_cost: 0.8396, reader_cost: 0.00014, ips: 1.1911 samples/sec | ETA 05:05:02 2021-05-06 16:56:11 [INFO] [TRAIN] epoch: 49, iter: 18210/40000, loss: 0.3147, lr: 0.005831, batch_cost: 0.8413, reader_cost: 0.00016, ips: 1.1886 samples/sec | ETA 05:05:32 2021-05-06 16:56:20 [INFO] [TRAIN] epoch: 49, iter: 18220/40000, loss: 0.5341, lr: 0.005829, batch_cost: 0.8388, reader_cost: 0.00013, ips: 1.1922 samples/sec | ETA 05:04:28 2021-05-06 16:56:31 [INFO] [TRAIN] epoch: 50, iter: 18230/40000, loss: 0.5535, lr: 0.005826, batch_cost: 1.1445, reader_cost: 0.21769, ips: 0.8738 samples/sec | ETA 06:55:15 2021-05-06 16:56:40 [INFO] [TRAIN] epoch: 50, iter: 18240/40000, loss: 0.3980, lr: 0.005824, batch_cost: 0.8547, reader_cost: 0.00033, ips: 1.1701 samples/sec | ETA 05:09:57 2021-05-06 16:56:48 [INFO] [TRAIN] epoch: 50, iter: 18250/40000, loss: 0.6573, lr: 0.005822, batch_cost: 0.8419, reader_cost: 0.00015, ips: 1.1877 samples/sec | ETA 05:05:12 2021-05-06 16:56:56 [INFO] [TRAIN] epoch: 50, iter: 18260/40000, loss: 0.5142, lr: 0.005819, batch_cost: 0.8416, reader_cost: 0.00019, ips: 1.1882 samples/sec | ETA 05:04:57 2021-05-06 16:57:05 [INFO] [TRAIN] epoch: 50, iter: 18270/40000, loss: 0.2395, lr: 0.005817, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 05:04:18 2021-05-06 16:57:13 [INFO] [TRAIN] epoch: 50, iter: 18280/40000, loss: 0.4954, lr: 0.005814, batch_cost: 0.8389, reader_cost: 0.00014, ips: 1.1920 samples/sec | ETA 05:03:41 2021-05-06 16:57:22 [INFO] [TRAIN] epoch: 50, iter: 18290/40000, loss: 0.4671, lr: 0.005812, batch_cost: 0.8403, reader_cost: 0.00016, ips: 1.1901 samples/sec | ETA 05:04:02 2021-05-06 16:57:30 [INFO] [TRAIN] epoch: 50, iter: 18300/40000, loss: 0.5706, lr: 0.005810, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 05:04:01 2021-05-06 16:57:38 [INFO] [TRAIN] epoch: 50, iter: 18310/40000, loss: 0.5681, lr: 0.005807, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 05:03:52 2021-05-06 16:57:47 [INFO] [TRAIN] epoch: 50, iter: 18320/40000, loss: 0.3300, lr: 0.005805, batch_cost: 0.8412, reader_cost: 0.00016, ips: 1.1888 samples/sec | ETA 05:03:56 2021-05-06 16:57:55 [INFO] [TRAIN] epoch: 50, iter: 18330/40000, loss: 0.2465, lr: 0.005803, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 05:03:14 2021-05-06 16:58:04 [INFO] [TRAIN] epoch: 50, iter: 18340/40000, loss: 0.3554, lr: 0.005800, batch_cost: 0.8395, reader_cost: 0.00016, ips: 1.1911 samples/sec | ETA 05:03:04 2021-05-06 16:58:12 [INFO] [TRAIN] epoch: 50, iter: 18350/40000, loss: 0.4675, lr: 0.005798, batch_cost: 0.8427, reader_cost: 0.00015, ips: 1.1867 samples/sec | ETA 05:04:03 2021-05-06 16:58:20 [INFO] [TRAIN] epoch: 50, iter: 18360/40000, loss: 0.5032, lr: 0.005795, batch_cost: 0.8394, reader_cost: 0.00016, ips: 1.1913 samples/sec | ETA 05:02:44 2021-05-06 16:58:29 [INFO] [TRAIN] epoch: 50, iter: 18370/40000, loss: 0.4586, lr: 0.005793, batch_cost: 0.8384, reader_cost: 0.00015, ips: 1.1927 samples/sec | ETA 05:02:14 2021-05-06 16:58:37 [INFO] [TRAIN] epoch: 50, iter: 18380/40000, loss: 0.4782, lr: 0.005791, batch_cost: 0.8394, reader_cost: 0.00016, ips: 1.1913 samples/sec | ETA 05:02:28 2021-05-06 16:58:46 [INFO] [TRAIN] epoch: 50, iter: 18390/40000, loss: 0.2902, lr: 0.005788, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 05:02:28 2021-05-06 16:58:54 [INFO] [TRAIN] epoch: 50, iter: 18400/40000, loss: 0.4113, lr: 0.005786, batch_cost: 0.8389, reader_cost: 0.00014, ips: 1.1920 samples/sec | ETA 05:02:00 2021-05-06 16:59:02 [INFO] [TRAIN] epoch: 50, iter: 18410/40000, loss: 0.5498, lr: 0.005784, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1903 samples/sec | ETA 05:02:17 2021-05-06 16:59:11 [INFO] [TRAIN] epoch: 50, iter: 18420/40000, loss: 0.6377, lr: 0.005781, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1918 samples/sec | ETA 05:01:46 2021-05-06 16:59:19 [INFO] [TRAIN] epoch: 50, iter: 18430/40000, loss: 0.4601, lr: 0.005779, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 05:02:00 2021-05-06 16:59:28 [INFO] [TRAIN] epoch: 50, iter: 18440/40000, loss: 0.3927, lr: 0.005777, batch_cost: 0.8415, reader_cost: 0.00014, ips: 1.1883 samples/sec | ETA 05:02:23 2021-05-06 16:59:36 [INFO] [TRAIN] epoch: 50, iter: 18450/40000, loss: 0.3219, lr: 0.005774, batch_cost: 0.8402, reader_cost: 0.00016, ips: 1.1902 samples/sec | ETA 05:01:46 2021-05-06 16:59:44 [INFO] [TRAIN] epoch: 50, iter: 18460/40000, loss: 0.2958, lr: 0.005772, batch_cost: 0.8409, reader_cost: 0.00014, ips: 1.1892 samples/sec | ETA 05:01:53 2021-05-06 16:59:53 [INFO] [TRAIN] epoch: 50, iter: 18470/40000, loss: 0.3552, lr: 0.005769, batch_cost: 0.8421, reader_cost: 0.00015, ips: 1.1875 samples/sec | ETA 05:02:11 2021-05-06 17:00:01 [INFO] [TRAIN] epoch: 50, iter: 18480/40000, loss: 0.2041, lr: 0.005767, batch_cost: 0.8404, reader_cost: 0.00014, ips: 1.1899 samples/sec | ETA 05:01:25 2021-05-06 17:00:10 [INFO] [TRAIN] epoch: 50, iter: 18490/40000, loss: 0.3708, lr: 0.005765, batch_cost: 0.8411, reader_cost: 0.00016, ips: 1.1889 samples/sec | ETA 05:01:32 2021-05-06 17:00:18 [INFO] [TRAIN] epoch: 50, iter: 18500/40000, loss: 0.3695, lr: 0.005762, batch_cost: 0.8423, reader_cost: 0.00015, ips: 1.1872 samples/sec | ETA 05:01:50 2021-05-06 17:00:26 [INFO] [TRAIN] epoch: 50, iter: 18510/40000, loss: 0.2564, lr: 0.005760, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 05:01:05 2021-05-06 17:00:35 [INFO] [TRAIN] epoch: 50, iter: 18520/40000, loss: 0.5986, lr: 0.005758, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 05:00:58 2021-05-06 17:00:43 [INFO] [TRAIN] epoch: 50, iter: 18530/40000, loss: 0.3866, lr: 0.005755, batch_cost: 0.8419, reader_cost: 0.00016, ips: 1.1878 samples/sec | ETA 05:01:14 2021-05-06 17:00:52 [INFO] [TRAIN] epoch: 50, iter: 18540/40000, loss: 0.3639, lr: 0.005753, batch_cost: 0.8430, reader_cost: 0.00016, ips: 1.1862 samples/sec | ETA 05:01:31 2021-05-06 17:01:00 [INFO] [TRAIN] epoch: 50, iter: 18550/40000, loss: 0.5543, lr: 0.005750, batch_cost: 0.8405, reader_cost: 0.00016, ips: 1.1898 samples/sec | ETA 05:00:28 2021-05-06 17:01:09 [INFO] [TRAIN] epoch: 50, iter: 18560/40000, loss: 0.3183, lr: 0.005748, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 05:00:23 2021-05-06 17:01:17 [INFO] [TRAIN] epoch: 50, iter: 18570/40000, loss: 0.3836, lr: 0.005746, batch_cost: 0.8419, reader_cost: 0.00015, ips: 1.1878 samples/sec | ETA 05:00:41 2021-05-06 17:01:25 [INFO] [TRAIN] epoch: 50, iter: 18580/40000, loss: 0.3049, lr: 0.005743, batch_cost: 0.8420, reader_cost: 0.00016, ips: 1.1877 samples/sec | ETA 05:00:34 2021-05-06 17:01:34 [INFO] [TRAIN] epoch: 50, iter: 18590/40000, loss: 0.4416, lr: 0.005741, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 04:59:58 2021-05-06 17:01:42 [INFO] [TRAIN] epoch: 50, iter: 18600/40000, loss: 0.5527, lr: 0.005739, batch_cost: 0.8390, reader_cost: 0.00010, ips: 1.1918 samples/sec | ETA 04:59:15 2021-05-06 17:01:54 [INFO] [TRAIN] epoch: 51, iter: 18610/40000, loss: 0.5509, lr: 0.005736, batch_cost: 1.1682, reader_cost: 0.22865, ips: 0.8560 samples/sec | ETA 06:56:27 2021-05-06 17:02:02 [INFO] [TRAIN] epoch: 51, iter: 18620/40000, loss: 0.5642, lr: 0.005734, batch_cost: 0.8431, reader_cost: 0.00032, ips: 1.1861 samples/sec | ETA 05:00:24 2021-05-06 17:02:11 [INFO] [TRAIN] epoch: 51, iter: 18630/40000, loss: 0.4166, lr: 0.005731, batch_cost: 0.8409, reader_cost: 0.00016, ips: 1.1892 samples/sec | ETA 04:59:30 2021-05-06 17:02:19 [INFO] [TRAIN] epoch: 51, iter: 18640/40000, loss: 0.2648, lr: 0.005729, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 04:59:15 2021-05-06 17:02:28 [INFO] [TRAIN] epoch: 51, iter: 18650/40000, loss: 0.3973, lr: 0.005727, batch_cost: 0.8424, reader_cost: 0.00015, ips: 1.1872 samples/sec | ETA 04:59:44 2021-05-06 17:02:36 [INFO] [TRAIN] epoch: 51, iter: 18660/40000, loss: 0.5978, lr: 0.005724, batch_cost: 0.8413, reader_cost: 0.00014, ips: 1.1886 samples/sec | ETA 04:59:13 2021-05-06 17:02:44 [INFO] [TRAIN] epoch: 51, iter: 18670/40000, loss: 0.4038, lr: 0.005722, batch_cost: 0.8420, reader_cost: 0.00015, ips: 1.1877 samples/sec | ETA 04:59:19 2021-05-06 17:02:53 [INFO] [TRAIN] epoch: 51, iter: 18680/40000, loss: 0.5895, lr: 0.005720, batch_cost: 0.8395, reader_cost: 0.00016, ips: 1.1913 samples/sec | ETA 04:58:17 2021-05-06 17:03:01 [INFO] [TRAIN] epoch: 51, iter: 18690/40000, loss: 0.3761, lr: 0.005717, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1880 samples/sec | ETA 04:58:58 2021-05-06 17:03:10 [INFO] [TRAIN] epoch: 51, iter: 18700/40000, loss: 0.2021, lr: 0.005715, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1885 samples/sec | ETA 04:58:42 2021-05-06 17:03:18 [INFO] [TRAIN] epoch: 51, iter: 18710/40000, loss: 0.3858, lr: 0.005713, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 04:58:14 2021-05-06 17:03:26 [INFO] [TRAIN] epoch: 51, iter: 18720/40000, loss: 0.4258, lr: 0.005710, batch_cost: 0.8427, reader_cost: 0.00015, ips: 1.1867 samples/sec | ETA 04:58:52 2021-05-06 17:03:35 [INFO] [TRAIN] epoch: 51, iter: 18730/40000, loss: 0.4554, lr: 0.005708, batch_cost: 0.8402, reader_cost: 0.00016, ips: 1.1902 samples/sec | ETA 04:57:50 2021-05-06 17:03:43 [INFO] [TRAIN] epoch: 51, iter: 18740/40000, loss: 0.5264, lr: 0.005705, batch_cost: 0.8390, reader_cost: 0.00016, ips: 1.1919 samples/sec | ETA 04:57:17 2021-05-06 17:03:52 [INFO] [TRAIN] epoch: 51, iter: 18750/40000, loss: 0.3850, lr: 0.005703, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 04:57:35 2021-05-06 17:04:00 [INFO] [TRAIN] epoch: 51, iter: 18760/40000, loss: 0.4816, lr: 0.005701, batch_cost: 0.8396, reader_cost: 0.00014, ips: 1.1910 samples/sec | ETA 04:57:13 2021-05-06 17:04:08 [INFO] [TRAIN] epoch: 51, iter: 18770/40000, loss: 0.3194, lr: 0.005698, batch_cost: 0.8413, reader_cost: 0.00014, ips: 1.1887 samples/sec | ETA 04:57:40 2021-05-06 17:04:17 [INFO] [TRAIN] epoch: 51, iter: 18780/40000, loss: 0.3472, lr: 0.005696, batch_cost: 0.8422, reader_cost: 0.00015, ips: 1.1874 samples/sec | ETA 04:57:51 2021-05-06 17:04:25 [INFO] [TRAIN] epoch: 51, iter: 18790/40000, loss: 1.3099, lr: 0.005694, batch_cost: 0.8425, reader_cost: 0.00016, ips: 1.1870 samples/sec | ETA 04:57:48 2021-05-06 17:04:34 [INFO] [TRAIN] epoch: 51, iter: 18800/40000, loss: 0.7173, lr: 0.005691, batch_cost: 0.8415, reader_cost: 0.00016, ips: 1.1884 samples/sec | ETA 04:57:19 2021-05-06 17:04:42 [INFO] [TRAIN] epoch: 51, iter: 18810/40000, loss: 0.4175, lr: 0.005689, batch_cost: 0.8405, reader_cost: 0.00017, ips: 1.1897 samples/sec | ETA 04:56:50 2021-05-06 17:04:51 [INFO] [TRAIN] epoch: 51, iter: 18820/40000, loss: 0.3028, lr: 0.005686, batch_cost: 0.8411, reader_cost: 0.00014, ips: 1.1889 samples/sec | ETA 04:56:54 2021-05-06 17:04:59 [INFO] [TRAIN] epoch: 51, iter: 18830/40000, loss: 0.3372, lr: 0.005684, batch_cost: 0.8396, reader_cost: 0.00016, ips: 1.1911 samples/sec | ETA 04:56:13 2021-05-06 17:05:07 [INFO] [TRAIN] epoch: 51, iter: 18840/40000, loss: 0.4197, lr: 0.005682, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 04:56:31 2021-05-06 17:05:16 [INFO] [TRAIN] epoch: 51, iter: 18850/40000, loss: 0.4091, lr: 0.005679, batch_cost: 0.8407, reader_cost: 0.00014, ips: 1.1894 samples/sec | ETA 04:56:21 2021-05-06 17:05:24 [INFO] [TRAIN] epoch: 51, iter: 18860/40000, loss: 0.3850, lr: 0.005677, batch_cost: 0.8423, reader_cost: 0.00014, ips: 1.1872 samples/sec | ETA 04:56:47 2021-05-06 17:05:33 [INFO] [TRAIN] epoch: 51, iter: 18870/40000, loss: 0.4258, lr: 0.005675, batch_cost: 0.8416, reader_cost: 0.00014, ips: 1.1882 samples/sec | ETA 04:56:22 2021-05-06 17:05:41 [INFO] [TRAIN] epoch: 51, iter: 18880/40000, loss: 0.5260, lr: 0.005672, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 04:55:45 2021-05-06 17:05:49 [INFO] [TRAIN] epoch: 51, iter: 18890/40000, loss: 0.3606, lr: 0.005670, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 04:55:31 2021-05-06 17:05:58 [INFO] [TRAIN] epoch: 51, iter: 18900/40000, loss: 0.3921, lr: 0.005667, batch_cost: 0.8410, reader_cost: 0.00016, ips: 1.1891 samples/sec | ETA 04:55:44 2021-05-06 17:06:06 [INFO] [TRAIN] epoch: 51, iter: 18910/40000, loss: 0.2874, lr: 0.005665, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 04:55:07 2021-05-06 17:06:15 [INFO] [TRAIN] epoch: 51, iter: 18920/40000, loss: 0.3434, lr: 0.005663, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 04:55:04 2021-05-06 17:06:23 [INFO] [TRAIN] epoch: 51, iter: 18930/40000, loss: 0.5003, lr: 0.005660, batch_cost: 0.8406, reader_cost: 0.00014, ips: 1.1896 samples/sec | ETA 04:55:11 2021-05-06 17:06:31 [INFO] [TRAIN] epoch: 51, iter: 18940/40000, loss: 0.3416, lr: 0.005658, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 04:55:12 2021-05-06 17:06:40 [INFO] [TRAIN] epoch: 51, iter: 18950/40000, loss: 0.3091, lr: 0.005656, batch_cost: 0.8407, reader_cost: 0.00016, ips: 1.1895 samples/sec | ETA 04:54:56 2021-05-06 17:06:48 [INFO] [TRAIN] epoch: 51, iter: 18960/40000, loss: 0.3934, lr: 0.005653, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 04:54:35 2021-05-06 17:06:57 [INFO] [TRAIN] epoch: 51, iter: 18970/40000, loss: 0.3349, lr: 0.005651, batch_cost: 0.8375, reader_cost: 0.00029, ips: 1.1940 samples/sec | ETA 04:53:33 2021-05-06 17:07:08 [INFO] [TRAIN] epoch: 52, iter: 18980/40000, loss: 0.4989, lr: 0.005648, batch_cost: 1.1769, reader_cost: 0.18896, ips: 0.8497 samples/sec | ETA 06:52:19 2021-05-06 17:07:17 [INFO] [TRAIN] epoch: 52, iter: 18990/40000, loss: 0.5347, lr: 0.005646, batch_cost: 0.8409, reader_cost: 0.00033, ips: 1.1892 samples/sec | ETA 04:54:27 2021-05-06 17:07:25 [INFO] [TRAIN] epoch: 52, iter: 19000/40000, loss: 0.3913, lr: 0.005644, batch_cost: 0.8392, reader_cost: 0.00015, ips: 1.1916 samples/sec | ETA 04:53:43 2021-05-06 17:07:25 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 17:11:02 [INFO] [EVAL] #Images: 500 mIoU: 0.7393 Acc: 0.9536 Kappa: 0.9396 2021-05-06 17:11:02 [INFO] [EVAL] Class IoU: [0.9791 0.8293 0.9119 0.5487 0.518 0.5139 0.5883 0.7098 0.9142 0.6523 0.9419 0.7664 0.527 0.9365 0.7574 0.8837 0.7593 0.6016 0.7065] 2021-05-06 17:11:02 [INFO] [EVAL] Class Acc: [0.9876 0.9372 0.944 0.7649 0.8436 0.7308 0.7813 0.8773 0.9433 0.8676 0.9666 0.8647 0.743 0.9619 0.8294 0.9415 0.8637 0.7671 0.837 ] 2021-05-06 17:11:51 [INFO] [EVAL] The model with the best validation mIoU (0.7393) was saved at iter 19000. 2021-05-06 17:12:00 [INFO] [TRAIN] epoch: 52, iter: 19010/40000, loss: 0.3621, lr: 0.005641, batch_cost: 0.8375, reader_cost: 0.00016, ips: 1.1940 samples/sec | ETA 04:52:59 2021-05-06 17:12:08 [INFO] [TRAIN] epoch: 52, iter: 19020/40000, loss: 0.3339, lr: 0.005639, batch_cost: 0.8381, reader_cost: 0.00015, ips: 1.1931 samples/sec | ETA 04:53:04 2021-05-06 17:12:17 [INFO] [TRAIN] epoch: 52, iter: 19030/40000, loss: 0.4629, lr: 0.005637, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 04:53:29 2021-05-06 17:12:25 [INFO] [TRAIN] epoch: 52, iter: 19040/40000, loss: 0.4237, lr: 0.005634, batch_cost: 0.8399, reader_cost: 0.00016, ips: 1.1907 samples/sec | ETA 04:53:23 2021-05-06 17:12:33 [INFO] [TRAIN] epoch: 52, iter: 19050/40000, loss: 0.7196, lr: 0.005632, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 04:53:22 2021-05-06 17:12:42 [INFO] [TRAIN] epoch: 52, iter: 19060/40000, loss: 0.3371, lr: 0.005629, batch_cost: 0.8413, reader_cost: 0.00016, ips: 1.1887 samples/sec | ETA 04:53:36 2021-05-06 17:12:50 [INFO] [TRAIN] epoch: 52, iter: 19070/40000, loss: 0.3549, lr: 0.005627, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 04:53:09 2021-05-06 17:12:59 [INFO] [TRAIN] epoch: 52, iter: 19080/40000, loss: 0.3036, lr: 0.005625, batch_cost: 0.8410, reader_cost: 0.00016, ips: 1.1891 samples/sec | ETA 04:53:13 2021-05-06 17:13:07 [INFO] [TRAIN] epoch: 52, iter: 19090/40000, loss: 0.4503, lr: 0.005622, batch_cost: 0.8421, reader_cost: 0.00015, ips: 1.1875 samples/sec | ETA 04:53:28 2021-05-06 17:13:15 [INFO] [TRAIN] epoch: 52, iter: 19100/40000, loss: 0.4111, lr: 0.005620, batch_cost: 0.8429, reader_cost: 0.00015, ips: 1.1864 samples/sec | ETA 04:53:35 2021-05-06 17:13:24 [INFO] [TRAIN] epoch: 52, iter: 19110/40000, loss: 0.6991, lr: 0.005618, batch_cost: 0.8415, reader_cost: 0.00016, ips: 1.1883 samples/sec | ETA 04:52:59 2021-05-06 17:13:32 [INFO] [TRAIN] epoch: 52, iter: 19120/40000, loss: 0.2633, lr: 0.005615, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 04:52:35 2021-05-06 17:13:41 [INFO] [TRAIN] epoch: 52, iter: 19130/40000, loss: 0.3127, lr: 0.005613, batch_cost: 0.8418, reader_cost: 0.00017, ips: 1.1879 samples/sec | ETA 04:52:48 2021-05-06 17:13:49 [INFO] [TRAIN] epoch: 52, iter: 19140/40000, loss: 0.3956, lr: 0.005610, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1881 samples/sec | ETA 04:52:36 2021-05-06 17:13:57 [INFO] [TRAIN] epoch: 52, iter: 19150/40000, loss: 0.3792, lr: 0.005608, batch_cost: 0.8400, reader_cost: 0.00014, ips: 1.1905 samples/sec | ETA 04:51:52 2021-05-06 17:14:06 [INFO] [TRAIN] epoch: 52, iter: 19160/40000, loss: 0.5482, lr: 0.005606, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1914 samples/sec | ETA 04:51:32 2021-05-06 17:14:14 [INFO] [TRAIN] epoch: 52, iter: 19170/40000, loss: 0.5743, lr: 0.005603, batch_cost: 0.8400, reader_cost: 0.00016, ips: 1.1905 samples/sec | ETA 04:51:37 2021-05-06 17:14:23 [INFO] [TRAIN] epoch: 52, iter: 19180/40000, loss: 0.3554, lr: 0.005601, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 04:51:44 2021-05-06 17:14:31 [INFO] [TRAIN] epoch: 52, iter: 19190/40000, loss: 0.5237, lr: 0.005599, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1911 samples/sec | ETA 04:51:11 2021-05-06 17:14:39 [INFO] [TRAIN] epoch: 52, iter: 19200/40000, loss: 0.2271, lr: 0.005596, batch_cost: 0.8409, reader_cost: 0.00016, ips: 1.1892 samples/sec | ETA 04:51:30 2021-05-06 17:14:48 [INFO] [TRAIN] epoch: 52, iter: 19210/40000, loss: 0.4140, lr: 0.005594, batch_cost: 0.8405, reader_cost: 0.00016, ips: 1.1898 samples/sec | ETA 04:51:13 2021-05-06 17:14:56 [INFO] [TRAIN] epoch: 52, iter: 19220/40000, loss: 0.3012, lr: 0.005591, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 04:51:01 2021-05-06 17:15:05 [INFO] [TRAIN] epoch: 52, iter: 19230/40000, loss: 0.4011, lr: 0.005589, batch_cost: 0.8406, reader_cost: 0.00014, ips: 1.1896 samples/sec | ETA 04:50:59 2021-05-06 17:15:13 [INFO] [TRAIN] epoch: 52, iter: 19240/40000, loss: 0.3464, lr: 0.005587, batch_cost: 0.8393, reader_cost: 0.00014, ips: 1.1915 samples/sec | ETA 04:50:23 2021-05-06 17:15:21 [INFO] [TRAIN] epoch: 52, iter: 19250/40000, loss: 1.5015, lr: 0.005584, batch_cost: 0.8400, reader_cost: 0.00014, ips: 1.1905 samples/sec | ETA 04:50:29 2021-05-06 17:15:30 [INFO] [TRAIN] epoch: 52, iter: 19260/40000, loss: 0.5323, lr: 0.005582, batch_cost: 0.8412, reader_cost: 0.00014, ips: 1.1888 samples/sec | ETA 04:50:45 2021-05-06 17:15:38 [INFO] [TRAIN] epoch: 52, iter: 19270/40000, loss: 0.4691, lr: 0.005579, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1903 samples/sec | ETA 04:50:15 2021-05-06 17:15:47 [INFO] [TRAIN] epoch: 52, iter: 19280/40000, loss: 0.2924, lr: 0.005577, batch_cost: 0.8418, reader_cost: 0.00014, ips: 1.1880 samples/sec | ETA 04:50:41 2021-05-06 17:15:55 [INFO] [TRAIN] epoch: 52, iter: 19290/40000, loss: 0.2917, lr: 0.005575, batch_cost: 0.8401, reader_cost: 0.00016, ips: 1.1903 samples/sec | ETA 04:49:58 2021-05-06 17:16:04 [INFO] [TRAIN] epoch: 52, iter: 19300/40000, loss: 0.4817, lr: 0.005572, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 04:50:00 2021-05-06 17:16:12 [INFO] [TRAIN] epoch: 52, iter: 19310/40000, loss: 0.4210, lr: 0.005570, batch_cost: 0.8408, reader_cost: 0.00016, ips: 1.1893 samples/sec | ETA 04:49:56 2021-05-06 17:16:20 [INFO] [TRAIN] epoch: 52, iter: 19320/40000, loss: 0.4293, lr: 0.005568, batch_cost: 0.8419, reader_cost: 0.00014, ips: 1.1878 samples/sec | ETA 04:50:10 2021-05-06 17:16:29 [INFO] [TRAIN] epoch: 52, iter: 19330/40000, loss: 0.4604, lr: 0.005565, batch_cost: 0.8403, reader_cost: 0.00014, ips: 1.1900 samples/sec | ETA 04:49:29 2021-05-06 17:16:37 [INFO] [TRAIN] epoch: 52, iter: 19340/40000, loss: 0.4124, lr: 0.005563, batch_cost: 0.8405, reader_cost: 0.00012, ips: 1.1897 samples/sec | ETA 04:49:25 2021-05-06 17:16:49 [INFO] [TRAIN] epoch: 53, iter: 19350/40000, loss: 0.6732, lr: 0.005560, batch_cost: 1.1856, reader_cost: 0.21486, ips: 0.8434 samples/sec | ETA 06:48:03 2021-05-06 17:16:57 [INFO] [TRAIN] epoch: 53, iter: 19360/40000, loss: 0.2684, lr: 0.005558, batch_cost: 0.8424, reader_cost: 0.00030, ips: 1.1871 samples/sec | ETA 04:49:47 2021-05-06 17:17:06 [INFO] [TRAIN] epoch: 53, iter: 19370/40000, loss: 0.4449, lr: 0.005556, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 04:48:59 2021-05-06 17:17:14 [INFO] [TRAIN] epoch: 53, iter: 19380/40000, loss: 0.3020, lr: 0.005553, batch_cost: 0.8447, reader_cost: 0.00015, ips: 1.1838 samples/sec | ETA 04:50:17 2021-05-06 17:17:23 [INFO] [TRAIN] epoch: 53, iter: 19390/40000, loss: 0.3991, lr: 0.005551, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1879 samples/sec | ETA 04:49:09 2021-05-06 17:17:31 [INFO] [TRAIN] epoch: 53, iter: 19400/40000, loss: 0.3906, lr: 0.005549, batch_cost: 0.8426, reader_cost: 0.00015, ips: 1.1868 samples/sec | ETA 04:49:17 2021-05-06 17:17:40 [INFO] [TRAIN] epoch: 53, iter: 19410/40000, loss: 0.4691, lr: 0.005546, batch_cost: 0.8423, reader_cost: 0.00017, ips: 1.1873 samples/sec | ETA 04:49:02 2021-05-06 17:17:48 [INFO] [TRAIN] epoch: 53, iter: 19420/40000, loss: 0.6989, lr: 0.005544, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 04:48:14 2021-05-06 17:17:56 [INFO] [TRAIN] epoch: 53, iter: 19430/40000, loss: 0.4488, lr: 0.005541, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 04:48:21 2021-05-06 17:18:05 [INFO] [TRAIN] epoch: 53, iter: 19440/40000, loss: 0.2601, lr: 0.005539, batch_cost: 0.8407, reader_cost: 0.00014, ips: 1.1895 samples/sec | ETA 04:48:04 2021-05-06 17:18:13 [INFO] [TRAIN] epoch: 53, iter: 19450/40000, loss: 0.3286, lr: 0.005537, batch_cost: 0.8427, reader_cost: 0.00015, ips: 1.1866 samples/sec | ETA 04:48:37 2021-05-06 17:18:22 [INFO] [TRAIN] epoch: 53, iter: 19460/40000, loss: 0.3068, lr: 0.005534, batch_cost: 0.8402, reader_cost: 0.00016, ips: 1.1903 samples/sec | ETA 04:47:36 2021-05-06 17:18:30 [INFO] [TRAIN] epoch: 53, iter: 19470/40000, loss: 0.3688, lr: 0.005532, batch_cost: 0.8390, reader_cost: 0.00015, ips: 1.1920 samples/sec | ETA 04:47:03 2021-05-06 17:18:38 [INFO] [TRAIN] epoch: 53, iter: 19480/40000, loss: 0.6400, lr: 0.005530, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 04:47:14 2021-05-06 17:18:47 [INFO] [TRAIN] epoch: 53, iter: 19490/40000, loss: 0.5511, lr: 0.005527, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 04:47:22 2021-05-06 17:18:55 [INFO] [TRAIN] epoch: 53, iter: 19500/40000, loss: 0.2583, lr: 0.005525, batch_cost: 0.8411, reader_cost: 0.00017, ips: 1.1889 samples/sec | ETA 04:47:23 2021-05-06 17:19:04 [INFO] [TRAIN] epoch: 53, iter: 19510/40000, loss: 0.4300, lr: 0.005522, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 04:47:12 2021-05-06 17:19:12 [INFO] [TRAIN] epoch: 53, iter: 19520/40000, loss: 0.4608, lr: 0.005520, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 04:46:58 2021-05-06 17:19:20 [INFO] [TRAIN] epoch: 53, iter: 19530/40000, loss: 0.3773, lr: 0.005518, batch_cost: 0.8409, reader_cost: 0.00016, ips: 1.1893 samples/sec | ETA 04:46:52 2021-05-06 17:19:29 [INFO] [TRAIN] epoch: 53, iter: 19540/40000, loss: 0.4520, lr: 0.005515, batch_cost: 0.8419, reader_cost: 0.00015, ips: 1.1878 samples/sec | ETA 04:47:05 2021-05-06 17:19:37 [INFO] [TRAIN] epoch: 53, iter: 19550/40000, loss: 0.3637, lr: 0.005513, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 04:46:14 2021-05-06 17:19:46 [INFO] [TRAIN] epoch: 53, iter: 19560/40000, loss: 0.4306, lr: 0.005510, batch_cost: 0.8443, reader_cost: 0.00015, ips: 1.1844 samples/sec | ETA 04:47:37 2021-05-06 17:19:54 [INFO] [TRAIN] epoch: 53, iter: 19570/40000, loss: 0.3308, lr: 0.005508, batch_cost: 0.8406, reader_cost: 0.00017, ips: 1.1896 samples/sec | ETA 04:46:13 2021-05-06 17:20:03 [INFO] [TRAIN] epoch: 53, iter: 19580/40000, loss: 0.3649, lr: 0.005506, batch_cost: 0.8414, reader_cost: 0.00016, ips: 1.1885 samples/sec | ETA 04:46:21 2021-05-06 17:20:11 [INFO] [TRAIN] epoch: 53, iter: 19590/40000, loss: 0.2959, lr: 0.005503, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1880 samples/sec | ETA 04:46:19 2021-05-06 17:20:19 [INFO] [TRAIN] epoch: 53, iter: 19600/40000, loss: 0.4571, lr: 0.005501, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1881 samples/sec | ETA 04:46:10 2021-05-06 17:20:28 [INFO] [TRAIN] epoch: 53, iter: 19610/40000, loss: 0.3592, lr: 0.005499, batch_cost: 0.8424, reader_cost: 0.00015, ips: 1.1871 samples/sec | ETA 04:46:16 2021-05-06 17:20:36 [INFO] [TRAIN] epoch: 53, iter: 19620/40000, loss: 0.2390, lr: 0.005496, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1881 samples/sec | ETA 04:45:53 2021-05-06 17:20:45 [INFO] [TRAIN] epoch: 53, iter: 19630/40000, loss: 0.3495, lr: 0.005494, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1879 samples/sec | ETA 04:45:47 2021-05-06 17:20:53 [INFO] [TRAIN] epoch: 53, iter: 19640/40000, loss: 0.4318, lr: 0.005491, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 04:45:22 2021-05-06 17:21:01 [INFO] [TRAIN] epoch: 53, iter: 19650/40000, loss: 0.3225, lr: 0.005489, batch_cost: 0.8410, reader_cost: 0.00014, ips: 1.1890 samples/sec | ETA 04:45:15 2021-05-06 17:21:10 [INFO] [TRAIN] epoch: 53, iter: 19660/40000, loss: 0.2609, lr: 0.005487, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 04:44:41 2021-05-06 17:21:18 [INFO] [TRAIN] epoch: 53, iter: 19670/40000, loss: 0.7054, lr: 0.005484, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 04:44:47 2021-05-06 17:21:27 [INFO] [TRAIN] epoch: 53, iter: 19680/40000, loss: 0.3526, lr: 0.005482, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 04:44:30 2021-05-06 17:21:35 [INFO] [TRAIN] epoch: 53, iter: 19690/40000, loss: 0.4191, lr: 0.005479, batch_cost: 0.8392, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 04:44:05 2021-05-06 17:21:43 [INFO] [TRAIN] epoch: 53, iter: 19700/40000, loss: 0.4346, lr: 0.005477, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1917 samples/sec | ETA 04:43:54 2021-05-06 17:21:52 [INFO] [TRAIN] epoch: 53, iter: 19710/40000, loss: 0.3349, lr: 0.005475, batch_cost: 0.8392, reader_cost: 0.00012, ips: 1.1916 samples/sec | ETA 04:43:48 2021-05-06 17:22:03 [INFO] [TRAIN] epoch: 54, iter: 19720/40000, loss: 0.5222, lr: 0.005472, batch_cost: 1.1497, reader_cost: 0.21474, ips: 0.8698 samples/sec | ETA 06:28:36 2021-05-06 17:22:12 [INFO] [TRAIN] epoch: 54, iter: 19730/40000, loss: 0.3907, lr: 0.005470, batch_cost: 0.8479, reader_cost: 0.00031, ips: 1.1794 samples/sec | ETA 04:46:26 2021-05-06 17:22:20 [INFO] [TRAIN] epoch: 54, iter: 19740/40000, loss: 0.4122, lr: 0.005468, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 04:43:46 2021-05-06 17:22:29 [INFO] [TRAIN] epoch: 54, iter: 19750/40000, loss: 0.4159, lr: 0.005465, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 04:43:33 2021-05-06 17:22:37 [INFO] [TRAIN] epoch: 54, iter: 19760/40000, loss: 0.3213, lr: 0.005463, batch_cost: 0.8425, reader_cost: 0.00017, ips: 1.1870 samples/sec | ETA 04:44:11 2021-05-06 17:22:45 [INFO] [TRAIN] epoch: 54, iter: 19770/40000, loss: 0.4067, lr: 0.005460, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 04:43:35 2021-05-06 17:22:54 [INFO] [TRAIN] epoch: 54, iter: 19780/40000, loss: 0.5465, lr: 0.005458, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 04:43:20 2021-05-06 17:23:02 [INFO] [TRAIN] epoch: 54, iter: 19790/40000, loss: 0.4929, lr: 0.005456, batch_cost: 0.8378, reader_cost: 0.00016, ips: 1.1936 samples/sec | ETA 04:42:11 2021-05-06 17:23:11 [INFO] [TRAIN] epoch: 54, iter: 19800/40000, loss: 0.4494, lr: 0.005453, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 04:42:55 2021-05-06 17:23:19 [INFO] [TRAIN] epoch: 54, iter: 19810/40000, loss: 0.3055, lr: 0.005451, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 04:42:55 2021-05-06 17:23:27 [INFO] [TRAIN] epoch: 54, iter: 19820/40000, loss: 0.2976, lr: 0.005448, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1881 samples/sec | ETA 04:43:04 2021-05-06 17:23:36 [INFO] [TRAIN] epoch: 54, iter: 19830/40000, loss: 0.3994, lr: 0.005446, batch_cost: 0.8405, reader_cost: 0.00017, ips: 1.1897 samples/sec | ETA 04:42:33 2021-05-06 17:23:44 [INFO] [TRAIN] epoch: 54, iter: 19840/40000, loss: 0.4274, lr: 0.005444, batch_cost: 0.8415, reader_cost: 0.00014, ips: 1.1884 samples/sec | ETA 04:42:44 2021-05-06 17:23:53 [INFO] [TRAIN] epoch: 54, iter: 19850/40000, loss: 0.4587, lr: 0.005441, batch_cost: 0.8414, reader_cost: 0.00014, ips: 1.1885 samples/sec | ETA 04:42:33 2021-05-06 17:24:01 [INFO] [TRAIN] epoch: 54, iter: 19860/40000, loss: 0.4208, lr: 0.005439, batch_cost: 0.8405, reader_cost: 0.00014, ips: 1.1898 samples/sec | ETA 04:42:07 2021-05-06 17:24:10 [INFO] [TRAIN] epoch: 54, iter: 19870/40000, loss: 0.3275, lr: 0.005437, batch_cost: 0.8399, reader_cost: 0.00016, ips: 1.1906 samples/sec | ETA 04:41:47 2021-05-06 17:24:18 [INFO] [TRAIN] epoch: 54, iter: 19880/40000, loss: 0.3331, lr: 0.005434, batch_cost: 0.8402, reader_cost: 0.00016, ips: 1.1902 samples/sec | ETA 04:41:45 2021-05-06 17:24:26 [INFO] [TRAIN] epoch: 54, iter: 19890/40000, loss: 0.3667, lr: 0.005432, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 04:41:48 2021-05-06 17:24:35 [INFO] [TRAIN] epoch: 54, iter: 19900/40000, loss: 0.4946, lr: 0.005429, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 04:41:29 2021-05-06 17:24:43 [INFO] [TRAIN] epoch: 54, iter: 19910/40000, loss: 0.7708, lr: 0.005427, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1914 samples/sec | ETA 04:41:01 2021-05-06 17:24:52 [INFO] [TRAIN] epoch: 54, iter: 19920/40000, loss: 0.3226, lr: 0.005425, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 04:41:16 2021-05-06 17:25:00 [INFO] [TRAIN] epoch: 54, iter: 19930/40000, loss: 0.3389, lr: 0.005422, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 04:41:07 2021-05-06 17:25:08 [INFO] [TRAIN] epoch: 54, iter: 19940/40000, loss: 0.1997, lr: 0.005420, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1883 samples/sec | ETA 04:41:21 2021-05-06 17:25:17 [INFO] [TRAIN] epoch: 54, iter: 19950/40000, loss: 0.2613, lr: 0.005417, batch_cost: 0.8412, reader_cost: 0.00016, ips: 1.1888 samples/sec | ETA 04:41:05 2021-05-06 17:25:25 [INFO] [TRAIN] epoch: 54, iter: 19960/40000, loss: 0.4100, lr: 0.005415, batch_cost: 0.8412, reader_cost: 0.00016, ips: 1.1888 samples/sec | ETA 04:40:57 2021-05-06 17:25:34 [INFO] [TRAIN] epoch: 54, iter: 19970/40000, loss: 0.2821, lr: 0.005413, batch_cost: 0.8396, reader_cost: 0.00016, ips: 1.1911 samples/sec | ETA 04:40:16 2021-05-06 17:25:42 [INFO] [TRAIN] epoch: 54, iter: 19980/40000, loss: 0.3953, lr: 0.005410, batch_cost: 0.8402, reader_cost: 0.00016, ips: 1.1902 samples/sec | ETA 04:40:21 2021-05-06 17:25:50 [INFO] [TRAIN] epoch: 54, iter: 19990/40000, loss: 0.3031, lr: 0.005408, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1886 samples/sec | ETA 04:40:34 2021-05-06 17:25:59 [INFO] [TRAIN] epoch: 54, iter: 20000/40000, loss: 0.3425, lr: 0.005406, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 04:40:11 2021-05-06 17:25:59 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 17:29:35 [INFO] [EVAL] #Images: 500 mIoU: 0.7511 Acc: 0.9545 Kappa: 0.9410 2021-05-06 17:29:35 [INFO] [EVAL] Class IoU: [0.9783 0.817 0.9184 0.6441 0.6219 0.5094 0.5949 0.7163 0.9163 0.6086 0.9386 0.7734 0.5673 0.9352 0.7455 0.8866 0.7733 0.6126 0.7139] 2021-05-06 17:29:35 [INFO] [EVAL] Class Acc: [0.9917 0.8627 0.959 0.8424 0.7995 0.7691 0.7915 0.8504 0.9485 0.839 0.9637 0.8632 0.7432 0.958 0.9055 0.9365 0.8909 0.7291 0.812 ] 2021-05-06 17:30:24 [INFO] [EVAL] The model with the best validation mIoU (0.7511) was saved at iter 20000. 2021-05-06 17:30:32 [INFO] [TRAIN] epoch: 54, iter: 20010/40000, loss: 0.3521, lr: 0.005403, batch_cost: 0.8374, reader_cost: 0.00030, ips: 1.1941 samples/sec | ETA 04:39:00 2021-05-06 17:30:41 [INFO] [TRAIN] epoch: 54, iter: 20020/40000, loss: 0.2801, lr: 0.005401, batch_cost: 0.8369, reader_cost: 0.00016, ips: 1.1949 samples/sec | ETA 04:38:40 2021-05-06 17:30:49 [INFO] [TRAIN] epoch: 54, iter: 20030/40000, loss: 0.4081, lr: 0.005398, batch_cost: 0.8372, reader_cost: 0.00015, ips: 1.1944 samples/sec | ETA 04:38:39 2021-05-06 17:30:58 [INFO] [TRAIN] epoch: 54, iter: 20040/40000, loss: 0.4655, lr: 0.005396, batch_cost: 0.8388, reader_cost: 0.00015, ips: 1.1922 samples/sec | ETA 04:39:02 2021-05-06 17:31:06 [INFO] [TRAIN] epoch: 54, iter: 20050/40000, loss: 0.4098, lr: 0.005394, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1904 samples/sec | ETA 04:39:19 2021-05-06 17:31:14 [INFO] [TRAIN] epoch: 54, iter: 20060/40000, loss: 0.4408, lr: 0.005391, batch_cost: 0.8395, reader_cost: 0.00014, ips: 1.1912 samples/sec | ETA 04:38:59 2021-05-06 17:31:23 [INFO] [TRAIN] epoch: 54, iter: 20070/40000, loss: 0.3408, lr: 0.005389, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1911 samples/sec | ETA 04:38:52 2021-05-06 17:31:31 [INFO] [TRAIN] epoch: 54, iter: 20080/40000, loss: 0.4315, lr: 0.005386, batch_cost: 0.8400, reader_cost: 0.00014, ips: 1.1904 samples/sec | ETA 04:38:53 2021-05-06 17:31:43 [INFO] [TRAIN] epoch: 55, iter: 20090/40000, loss: 0.4697, lr: 0.005384, batch_cost: 1.1671, reader_cost: 0.22720, ips: 0.8568 samples/sec | ETA 06:27:16 2021-05-06 17:31:51 [INFO] [TRAIN] epoch: 55, iter: 20100/40000, loss: 0.5267, lr: 0.005382, batch_cost: 0.8612, reader_cost: 0.00032, ips: 1.1612 samples/sec | ETA 04:45:37 2021-05-06 17:32:00 [INFO] [TRAIN] epoch: 55, iter: 20110/40000, loss: 0.4030, lr: 0.005379, batch_cost: 0.8410, reader_cost: 0.00016, ips: 1.1891 samples/sec | ETA 04:38:46 2021-05-06 17:32:08 [INFO] [TRAIN] epoch: 55, iter: 20120/40000, loss: 0.3311, lr: 0.005377, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1887 samples/sec | ETA 04:38:43 2021-05-06 17:32:17 [INFO] [TRAIN] epoch: 55, iter: 20130/40000, loss: 0.2370, lr: 0.005374, batch_cost: 0.8419, reader_cost: 0.00015, ips: 1.1878 samples/sec | ETA 04:38:49 2021-05-06 17:32:25 [INFO] [TRAIN] epoch: 55, iter: 20140/40000, loss: 0.5190, lr: 0.005372, batch_cost: 0.8422, reader_cost: 0.00015, ips: 1.1874 samples/sec | ETA 04:38:45 2021-05-06 17:32:34 [INFO] [TRAIN] epoch: 55, iter: 20150/40000, loss: 0.6067, lr: 0.005370, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 04:37:55 2021-05-06 17:32:42 [INFO] [TRAIN] epoch: 55, iter: 20160/40000, loss: 0.4830, lr: 0.005367, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 04:37:40 2021-05-06 17:32:50 [INFO] [TRAIN] epoch: 55, iter: 20170/40000, loss: 0.4706, lr: 0.005365, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1918 samples/sec | ETA 04:37:19 2021-05-06 17:32:59 [INFO] [TRAIN] epoch: 55, iter: 20180/40000, loss: 0.3676, lr: 0.005363, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 04:37:25 2021-05-06 17:33:07 [INFO] [TRAIN] epoch: 55, iter: 20190/40000, loss: 0.1835, lr: 0.005360, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 04:37:33 2021-05-06 17:33:16 [INFO] [TRAIN] epoch: 55, iter: 20200/40000, loss: 0.3014, lr: 0.005358, batch_cost: 0.8409, reader_cost: 0.00016, ips: 1.1892 samples/sec | ETA 04:37:29 2021-05-06 17:33:24 [INFO] [TRAIN] epoch: 55, iter: 20210/40000, loss: 0.3834, lr: 0.005355, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 04:37:17 2021-05-06 17:33:32 [INFO] [TRAIN] epoch: 55, iter: 20220/40000, loss: 0.4051, lr: 0.005353, batch_cost: 0.8403, reader_cost: 0.00016, ips: 1.1900 samples/sec | ETA 04:37:01 2021-05-06 17:33:41 [INFO] [TRAIN] epoch: 55, iter: 20230/40000, loss: 0.5056, lr: 0.005351, batch_cost: 0.8386, reader_cost: 0.00016, ips: 1.1925 samples/sec | ETA 04:36:18 2021-05-06 17:33:49 [INFO] [TRAIN] epoch: 55, iter: 20240/40000, loss: 0.4206, lr: 0.005348, batch_cost: 0.8407, reader_cost: 0.00016, ips: 1.1895 samples/sec | ETA 04:36:51 2021-05-06 17:33:58 [INFO] [TRAIN] epoch: 55, iter: 20250/40000, loss: 0.2860, lr: 0.005346, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1913 samples/sec | ETA 04:36:18 2021-05-06 17:34:06 [INFO] [TRAIN] epoch: 55, iter: 20260/40000, loss: 0.4535, lr: 0.005343, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 04:36:23 2021-05-06 17:34:14 [INFO] [TRAIN] epoch: 55, iter: 20270/40000, loss: 0.4940, lr: 0.005341, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 04:36:07 2021-05-06 17:34:23 [INFO] [TRAIN] epoch: 55, iter: 20280/40000, loss: 1.1927, lr: 0.005339, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 04:36:06 2021-05-06 17:34:31 [INFO] [TRAIN] epoch: 55, iter: 20290/40000, loss: 0.4173, lr: 0.005336, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 04:35:53 2021-05-06 17:34:40 [INFO] [TRAIN] epoch: 55, iter: 20300/40000, loss: 0.5338, lr: 0.005334, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 04:36:03 2021-05-06 17:34:48 [INFO] [TRAIN] epoch: 55, iter: 20310/40000, loss: 0.3339, lr: 0.005331, batch_cost: 0.8396, reader_cost: 0.00016, ips: 1.1911 samples/sec | ETA 04:35:31 2021-05-06 17:34:56 [INFO] [TRAIN] epoch: 55, iter: 20320/40000, loss: 0.1981, lr: 0.005329, batch_cost: 0.8420, reader_cost: 0.00016, ips: 1.1877 samples/sec | ETA 04:36:09 2021-05-06 17:35:05 [INFO] [TRAIN] epoch: 55, iter: 20330/40000, loss: 0.2494, lr: 0.005327, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 04:35:26 2021-05-06 17:35:13 [INFO] [TRAIN] epoch: 55, iter: 20340/40000, loss: 0.3316, lr: 0.005324, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 04:35:15 2021-05-06 17:35:22 [INFO] [TRAIN] epoch: 55, iter: 20350/40000, loss: 0.3774, lr: 0.005322, batch_cost: 0.8394, reader_cost: 0.00014, ips: 1.1913 samples/sec | ETA 04:34:54 2021-05-06 17:35:30 [INFO] [TRAIN] epoch: 55, iter: 20360/40000, loss: 0.3478, lr: 0.005319, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1918 samples/sec | ETA 04:34:38 2021-05-06 17:35:38 [INFO] [TRAIN] epoch: 55, iter: 20370/40000, loss: 0.4351, lr: 0.005317, batch_cost: 0.8411, reader_cost: 0.00014, ips: 1.1890 samples/sec | ETA 04:35:10 2021-05-06 17:35:47 [INFO] [TRAIN] epoch: 55, iter: 20380/40000, loss: 0.3508, lr: 0.005315, batch_cost: 0.8395, reader_cost: 0.00014, ips: 1.1912 samples/sec | ETA 04:34:31 2021-05-06 17:35:55 [INFO] [TRAIN] epoch: 55, iter: 20390/40000, loss: 0.3347, lr: 0.005312, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 04:34:25 2021-05-06 17:36:04 [INFO] [TRAIN] epoch: 55, iter: 20400/40000, loss: 0.2843, lr: 0.005310, batch_cost: 0.8405, reader_cost: 0.00016, ips: 1.1897 samples/sec | ETA 04:34:34 2021-05-06 17:36:12 [INFO] [TRAIN] epoch: 55, iter: 20410/40000, loss: 0.4635, lr: 0.005308, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 04:34:01 2021-05-06 17:36:20 [INFO] [TRAIN] epoch: 55, iter: 20420/40000, loss: 0.3913, lr: 0.005305, batch_cost: 0.8393, reader_cost: 0.00016, ips: 1.1915 samples/sec | ETA 04:33:52 2021-05-06 17:36:29 [INFO] [TRAIN] epoch: 55, iter: 20430/40000, loss: 0.3562, lr: 0.005303, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 04:34:17 2021-05-06 17:36:37 [INFO] [TRAIN] epoch: 55, iter: 20440/40000, loss: 0.4300, lr: 0.005300, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 04:34:00 2021-05-06 17:36:46 [INFO] [TRAIN] epoch: 55, iter: 20450/40000, loss: 0.3338, lr: 0.005298, batch_cost: 0.8403, reader_cost: 0.00014, ips: 1.1900 samples/sec | ETA 04:33:48 2021-05-06 17:36:54 [INFO] [TRAIN] epoch: 55, iter: 20460/40000, loss: 0.3170, lr: 0.005296, batch_cost: 0.8393, reader_cost: 0.00009, ips: 1.1915 samples/sec | ETA 04:33:19 2021-05-06 17:37:06 [INFO] [TRAIN] epoch: 56, iter: 20470/40000, loss: 0.5100, lr: 0.005293, batch_cost: 1.1663, reader_cost: 0.20011, ips: 0.8574 samples/sec | ETA 06:19:36 2021-05-06 17:37:14 [INFO] [TRAIN] epoch: 56, iter: 20480/40000, loss: 0.4088, lr: 0.005291, batch_cost: 0.8416, reader_cost: 0.00032, ips: 1.1883 samples/sec | ETA 04:33:47 2021-05-06 17:37:22 [INFO] [TRAIN] epoch: 56, iter: 20490/40000, loss: 0.3621, lr: 0.005288, batch_cost: 0.8425, reader_cost: 0.00016, ips: 1.1869 samples/sec | ETA 04:33:57 2021-05-06 17:37:31 [INFO] [TRAIN] epoch: 56, iter: 20500/40000, loss: 0.2223, lr: 0.005286, batch_cost: 0.8415, reader_cost: 0.00016, ips: 1.1884 samples/sec | ETA 04:33:29 2021-05-06 17:37:39 [INFO] [TRAIN] epoch: 56, iter: 20510/40000, loss: 0.5073, lr: 0.005284, batch_cost: 0.8424, reader_cost: 0.00016, ips: 1.1871 samples/sec | ETA 04:33:38 2021-05-06 17:37:48 [INFO] [TRAIN] epoch: 56, iter: 20520/40000, loss: 0.5794, lr: 0.005281, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1887 samples/sec | ETA 04:33:07 2021-05-06 17:37:56 [INFO] [TRAIN] epoch: 56, iter: 20530/40000, loss: 0.4337, lr: 0.005279, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 04:32:37 2021-05-06 17:38:04 [INFO] [TRAIN] epoch: 56, iter: 20540/40000, loss: 0.6272, lr: 0.005276, batch_cost: 0.8389, reader_cost: 0.00015, ips: 1.1921 samples/sec | ETA 04:32:04 2021-05-06 17:38:13 [INFO] [TRAIN] epoch: 56, iter: 20550/40000, loss: 0.4210, lr: 0.005274, batch_cost: 0.8426, reader_cost: 0.00015, ips: 1.1868 samples/sec | ETA 04:33:08 2021-05-06 17:38:21 [INFO] [TRAIN] epoch: 56, iter: 20560/40000, loss: 0.2685, lr: 0.005272, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 04:32:21 2021-05-06 17:38:30 [INFO] [TRAIN] epoch: 56, iter: 20570/40000, loss: 0.3480, lr: 0.005269, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 04:32:04 2021-05-06 17:38:38 [INFO] [TRAIN] epoch: 56, iter: 20580/40000, loss: 0.5127, lr: 0.005267, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 04:32:08 2021-05-06 17:38:47 [INFO] [TRAIN] epoch: 56, iter: 20590/40000, loss: 0.4762, lr: 0.005264, batch_cost: 0.8415, reader_cost: 0.00016, ips: 1.1884 samples/sec | ETA 04:32:12 2021-05-06 17:38:55 [INFO] [TRAIN] epoch: 56, iter: 20600/40000, loss: 0.5288, lr: 0.005262, batch_cost: 0.8401, reader_cost: 0.00016, ips: 1.1903 samples/sec | ETA 04:31:38 2021-05-06 17:39:03 [INFO] [TRAIN] epoch: 56, iter: 20610/40000, loss: 0.3692, lr: 0.005260, batch_cost: 0.8416, reader_cost: 0.00016, ips: 1.1882 samples/sec | ETA 04:31:58 2021-05-06 17:39:12 [INFO] [TRAIN] epoch: 56, iter: 20620/40000, loss: 0.2694, lr: 0.005257, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 04:31:36 2021-05-06 17:39:20 [INFO] [TRAIN] epoch: 56, iter: 20630/40000, loss: 0.5441, lr: 0.005255, batch_cost: 0.8394, reader_cost: 0.00016, ips: 1.1913 samples/sec | ETA 04:30:59 2021-05-06 17:39:29 [INFO] [TRAIN] epoch: 56, iter: 20640/40000, loss: 0.2888, lr: 0.005252, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 04:31:09 2021-05-06 17:39:37 [INFO] [TRAIN] epoch: 56, iter: 20650/40000, loss: 0.6177, lr: 0.005250, batch_cost: 0.8411, reader_cost: 0.00016, ips: 1.1889 samples/sec | ETA 04:31:14 2021-05-06 17:39:45 [INFO] [TRAIN] epoch: 56, iter: 20660/40000, loss: 0.4026, lr: 0.005248, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 04:30:40 2021-05-06 17:39:54 [INFO] [TRAIN] epoch: 56, iter: 20670/40000, loss: 0.5394, lr: 0.005245, batch_cost: 0.8442, reader_cost: 0.00014, ips: 1.1846 samples/sec | ETA 04:31:57 2021-05-06 17:40:02 [INFO] [TRAIN] epoch: 56, iter: 20680/40000, loss: 0.4741, lr: 0.005243, batch_cost: 0.8415, reader_cost: 0.00016, ips: 1.1883 samples/sec | ETA 04:30:58 2021-05-06 17:40:11 [INFO] [TRAIN] epoch: 56, iter: 20690/40000, loss: 0.2375, lr: 0.005241, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 04:30:13 2021-05-06 17:40:19 [INFO] [TRAIN] epoch: 56, iter: 20700/40000, loss: 0.3126, lr: 0.005238, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 04:30:29 2021-05-06 17:40:27 [INFO] [TRAIN] epoch: 56, iter: 20710/40000, loss: 0.2376, lr: 0.005236, batch_cost: 0.8399, reader_cost: 0.00014, ips: 1.1907 samples/sec | ETA 04:30:01 2021-05-06 17:40:36 [INFO] [TRAIN] epoch: 56, iter: 20720/40000, loss: 0.4270, lr: 0.005233, batch_cost: 0.8426, reader_cost: 0.00017, ips: 1.1869 samples/sec | ETA 04:30:44 2021-05-06 17:40:44 [INFO] [TRAIN] epoch: 56, iter: 20730/40000, loss: 0.3052, lr: 0.005231, batch_cost: 0.8426, reader_cost: 0.00016, ips: 1.1868 samples/sec | ETA 04:30:36 2021-05-06 17:40:53 [INFO] [TRAIN] epoch: 56, iter: 20740/40000, loss: 0.2998, lr: 0.005229, batch_cost: 0.8425, reader_cost: 0.00015, ips: 1.1869 samples/sec | ETA 04:30:27 2021-05-06 17:41:01 [INFO] [TRAIN] epoch: 56, iter: 20750/40000, loss: 0.3990, lr: 0.005226, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 04:29:27 2021-05-06 17:41:10 [INFO] [TRAIN] epoch: 56, iter: 20760/40000, loss: 0.3941, lr: 0.005224, batch_cost: 0.8434, reader_cost: 0.00014, ips: 1.1857 samples/sec | ETA 04:30:26 2021-05-06 17:41:18 [INFO] [TRAIN] epoch: 56, iter: 20770/40000, loss: 0.3916, lr: 0.005221, batch_cost: 0.8411, reader_cost: 0.00016, ips: 1.1889 samples/sec | ETA 04:29:34 2021-05-06 17:41:26 [INFO] [TRAIN] epoch: 56, iter: 20780/40000, loss: 0.4102, lr: 0.005219, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 04:29:09 2021-05-06 17:41:35 [INFO] [TRAIN] epoch: 56, iter: 20790/40000, loss: 0.3403, lr: 0.005217, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 04:29:12 2021-05-06 17:41:43 [INFO] [TRAIN] epoch: 56, iter: 20800/40000, loss: 0.3659, lr: 0.005214, batch_cost: 0.8405, reader_cost: 0.00014, ips: 1.1898 samples/sec | ETA 04:28:57 2021-05-06 17:41:52 [INFO] [TRAIN] epoch: 56, iter: 20810/40000, loss: 0.2695, lr: 0.005212, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 04:28:32 2021-05-06 17:42:00 [INFO] [TRAIN] epoch: 56, iter: 20820/40000, loss: 0.4883, lr: 0.005209, batch_cost: 0.8398, reader_cost: 0.00014, ips: 1.1907 samples/sec | ETA 04:28:27 2021-05-06 17:42:08 [INFO] [TRAIN] epoch: 56, iter: 20830/40000, loss: 0.3886, lr: 0.005207, batch_cost: 0.8391, reader_cost: 0.00028, ips: 1.1918 samples/sec | ETA 04:28:04 2021-05-06 17:42:20 [INFO] [TRAIN] epoch: 57, iter: 20840/40000, loss: 0.5053, lr: 0.005205, batch_cost: 1.1612, reader_cost: 0.21886, ips: 0.8612 samples/sec | ETA 06:10:49 2021-05-06 17:42:28 [INFO] [TRAIN] epoch: 57, iter: 20850/40000, loss: 0.3403, lr: 0.005202, batch_cost: 0.8399, reader_cost: 0.00031, ips: 1.1906 samples/sec | ETA 04:28:04 2021-05-06 17:42:37 [INFO] [TRAIN] epoch: 57, iter: 20860/40000, loss: 0.5000, lr: 0.005200, batch_cost: 0.8417, reader_cost: 0.00014, ips: 1.1881 samples/sec | ETA 04:28:30 2021-05-06 17:42:45 [INFO] [TRAIN] epoch: 57, iter: 20870/40000, loss: 0.3516, lr: 0.005197, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 04:28:00 2021-05-06 17:42:54 [INFO] [TRAIN] epoch: 57, iter: 20880/40000, loss: 0.5243, lr: 0.005195, batch_cost: 0.8408, reader_cost: 0.00017, ips: 1.1893 samples/sec | ETA 04:27:56 2021-05-06 17:43:02 [INFO] [TRAIN] epoch: 57, iter: 20890/40000, loss: 0.8196, lr: 0.005193, batch_cost: 0.8412, reader_cost: 0.00014, ips: 1.1888 samples/sec | ETA 04:27:54 2021-05-06 17:43:10 [INFO] [TRAIN] epoch: 57, iter: 20900/40000, loss: 0.4818, lr: 0.005190, batch_cost: 0.8415, reader_cost: 0.00016, ips: 1.1883 samples/sec | ETA 04:27:53 2021-05-06 17:43:19 [INFO] [TRAIN] epoch: 57, iter: 20910/40000, loss: 0.5563, lr: 0.005188, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1880 samples/sec | ETA 04:27:49 2021-05-06 17:43:27 [INFO] [TRAIN] epoch: 57, iter: 20920/40000, loss: 0.4365, lr: 0.005185, batch_cost: 0.8424, reader_cost: 0.00015, ips: 1.1871 samples/sec | ETA 04:27:53 2021-05-06 17:43:36 [INFO] [TRAIN] epoch: 57, iter: 20930/40000, loss: 0.2057, lr: 0.005183, batch_cost: 0.8426, reader_cost: 0.00015, ips: 1.1868 samples/sec | ETA 04:27:49 2021-05-06 17:43:44 [INFO] [TRAIN] epoch: 57, iter: 20940/40000, loss: 0.3049, lr: 0.005181, batch_cost: 0.8433, reader_cost: 0.00016, ips: 1.1858 samples/sec | ETA 04:27:53 2021-05-06 17:43:53 [INFO] [TRAIN] epoch: 57, iter: 20950/40000, loss: 0.3186, lr: 0.005178, batch_cost: 0.8419, reader_cost: 0.00015, ips: 1.1877 samples/sec | ETA 04:27:19 2021-05-06 17:44:01 [INFO] [TRAIN] epoch: 57, iter: 20960/40000, loss: 0.5656, lr: 0.005176, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1881 samples/sec | ETA 04:27:05 2021-05-06 17:44:09 [INFO] [TRAIN] epoch: 57, iter: 20970/40000, loss: 0.4903, lr: 0.005173, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1904 samples/sec | ETA 04:26:26 2021-05-06 17:44:18 [INFO] [TRAIN] epoch: 57, iter: 20980/40000, loss: 0.4276, lr: 0.005171, batch_cost: 0.8411, reader_cost: 0.00014, ips: 1.1889 samples/sec | ETA 04:26:37 2021-05-06 17:44:26 [INFO] [TRAIN] epoch: 57, iter: 20990/40000, loss: 0.2359, lr: 0.005169, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 04:26:19 2021-05-06 17:44:35 [INFO] [TRAIN] epoch: 57, iter: 21000/40000, loss: 0.3882, lr: 0.005166, batch_cost: 0.8418, reader_cost: 0.00014, ips: 1.1879 samples/sec | ETA 04:26:33 2021-05-06 17:44:35 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 17:48:14 [INFO] [EVAL] #Images: 500 mIoU: 0.7439 Acc: 0.9541 Kappa: 0.9405 2021-05-06 17:48:14 [INFO] [EVAL] Class IoU: [0.9781 0.8275 0.9152 0.613 0.5834 0.4933 0.5751 0.7063 0.9125 0.647 0.9395 0.7627 0.4975 0.9405 0.8098 0.8779 0.749 0.5987 0.706 ] 2021-05-06 17:48:14 [INFO] [EVAL] Class Acc: [0.9923 0.8801 0.95 0.8507 0.824 0.797 0.8109 0.8959 0.9377 0.8532 0.9641 0.8388 0.75 0.9679 0.9145 0.9244 0.8845 0.7599 0.8613] 2021-05-06 17:48:43 [INFO] [EVAL] The model with the best validation mIoU (0.7511) was saved at iter 20000. 2021-05-06 17:48:51 [INFO] [TRAIN] epoch: 57, iter: 21010/40000, loss: 0.3545, lr: 0.005164, batch_cost: 0.8384, reader_cost: 0.00017, ips: 1.1927 samples/sec | ETA 04:25:21 2021-05-06 17:49:00 [INFO] [TRAIN] epoch: 57, iter: 21020/40000, loss: 0.7021, lr: 0.005161, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 04:25:42 2021-05-06 17:49:08 [INFO] [TRAIN] epoch: 57, iter: 21030/40000, loss: 0.9054, lr: 0.005159, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 04:25:35 2021-05-06 17:49:17 [INFO] [TRAIN] epoch: 57, iter: 21040/40000, loss: 0.5094, lr: 0.005157, batch_cost: 0.8409, reader_cost: 0.00014, ips: 1.1891 samples/sec | ETA 04:25:44 2021-05-06 17:49:25 [INFO] [TRAIN] epoch: 57, iter: 21050/40000, loss: 0.4860, lr: 0.005154, batch_cost: 0.8413, reader_cost: 0.00014, ips: 1.1886 samples/sec | ETA 04:25:42 2021-05-06 17:49:33 [INFO] [TRAIN] epoch: 57, iter: 21060/40000, loss: 0.2202, lr: 0.005152, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 04:25:16 2021-05-06 17:49:42 [INFO] [TRAIN] epoch: 57, iter: 21070/40000, loss: 0.2655, lr: 0.005149, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 04:25:21 2021-05-06 17:49:50 [INFO] [TRAIN] epoch: 57, iter: 21080/40000, loss: 0.3219, lr: 0.005147, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1884 samples/sec | ETA 04:25:20 2021-05-06 17:49:59 [INFO] [TRAIN] epoch: 57, iter: 21090/40000, loss: 0.4320, lr: 0.005145, batch_cost: 0.8414, reader_cost: 0.00017, ips: 1.1885 samples/sec | ETA 04:25:10 2021-05-06 17:50:07 [INFO] [TRAIN] epoch: 57, iter: 21100/40000, loss: 0.2481, lr: 0.005142, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 04:24:39 2021-05-06 17:50:15 [INFO] [TRAIN] epoch: 57, iter: 21110/40000, loss: 0.2370, lr: 0.005140, batch_cost: 0.8399, reader_cost: 0.00014, ips: 1.1906 samples/sec | ETA 04:24:25 2021-05-06 17:50:24 [INFO] [TRAIN] epoch: 57, iter: 21120/40000, loss: 0.3876, lr: 0.005137, batch_cost: 0.8412, reader_cost: 0.00014, ips: 1.1888 samples/sec | ETA 04:24:41 2021-05-06 17:50:32 [INFO] [TRAIN] epoch: 57, iter: 21130/40000, loss: 0.3749, lr: 0.005135, batch_cost: 0.8393, reader_cost: 0.00014, ips: 1.1914 samples/sec | ETA 04:23:58 2021-05-06 17:50:41 [INFO] [TRAIN] epoch: 57, iter: 21140/40000, loss: 0.3637, lr: 0.005133, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 04:24:08 2021-05-06 17:50:49 [INFO] [TRAIN] epoch: 57, iter: 21150/40000, loss: 0.4527, lr: 0.005130, batch_cost: 0.8408, reader_cost: 0.00016, ips: 1.1894 samples/sec | ETA 04:24:08 2021-05-06 17:50:57 [INFO] [TRAIN] epoch: 57, iter: 21160/40000, loss: 0.4211, lr: 0.005128, batch_cost: 0.8414, reader_cost: 0.00016, ips: 1.1886 samples/sec | ETA 04:24:11 2021-05-06 17:51:06 [INFO] [TRAIN] epoch: 57, iter: 21170/40000, loss: 0.3252, lr: 0.005125, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1882 samples/sec | ETA 04:24:06 2021-05-06 17:51:14 [INFO] [TRAIN] epoch: 57, iter: 21180/40000, loss: 0.5097, lr: 0.005123, batch_cost: 0.8408, reader_cost: 0.00014, ips: 1.1893 samples/sec | ETA 04:23:43 2021-05-06 17:51:23 [INFO] [TRAIN] epoch: 57, iter: 21190/40000, loss: 0.3189, lr: 0.005121, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 04:23:18 2021-05-06 17:51:31 [INFO] [TRAIN] epoch: 57, iter: 21200/40000, loss: 0.5125, lr: 0.005118, batch_cost: 0.8380, reader_cost: 0.00010, ips: 1.1933 samples/sec | ETA 04:22:35 2021-05-06 17:51:43 [INFO] [TRAIN] epoch: 58, iter: 21210/40000, loss: 0.4755, lr: 0.005116, batch_cost: 1.1912, reader_cost: 0.22354, ips: 0.8395 samples/sec | ETA 06:13:03 2021-05-06 17:51:51 [INFO] [TRAIN] epoch: 58, iter: 21220/40000, loss: 0.3361, lr: 0.005113, batch_cost: 0.8406, reader_cost: 0.00033, ips: 1.1896 samples/sec | ETA 04:23:06 2021-05-06 17:52:00 [INFO] [TRAIN] epoch: 58, iter: 21230/40000, loss: 0.3703, lr: 0.005111, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 04:23:00 2021-05-06 17:52:08 [INFO] [TRAIN] epoch: 58, iter: 21240/40000, loss: 0.2753, lr: 0.005109, batch_cost: 0.8420, reader_cost: 0.00015, ips: 1.1876 samples/sec | ETA 04:23:15 2021-05-06 17:52:17 [INFO] [TRAIN] epoch: 58, iter: 21250/40000, loss: 0.3322, lr: 0.005106, batch_cost: 0.8402, reader_cost: 0.00016, ips: 1.1901 samples/sec | ETA 04:22:34 2021-05-06 17:52:25 [INFO] [TRAIN] epoch: 58, iter: 21260/40000, loss: 0.5220, lr: 0.005104, batch_cost: 0.8428, reader_cost: 0.00016, ips: 1.1865 samples/sec | ETA 04:23:14 2021-05-06 17:52:33 [INFO] [TRAIN] epoch: 58, iter: 21270/40000, loss: 0.5138, lr: 0.005101, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1882 samples/sec | ETA 04:22:43 2021-05-06 17:52:42 [INFO] [TRAIN] epoch: 58, iter: 21280/40000, loss: 0.5921, lr: 0.005099, batch_cost: 0.8388, reader_cost: 0.00016, ips: 1.1922 samples/sec | ETA 04:21:41 2021-05-06 17:52:50 [INFO] [TRAIN] epoch: 58, iter: 21290/40000, loss: 0.5082, lr: 0.005097, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 04:22:06 2021-05-06 17:52:59 [INFO] [TRAIN] epoch: 58, iter: 21300/40000, loss: 0.1279, lr: 0.005094, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 04:22:00 2021-05-06 17:53:07 [INFO] [TRAIN] epoch: 58, iter: 21310/40000, loss: 0.2867, lr: 0.005092, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1911 samples/sec | ETA 04:21:31 2021-05-06 17:53:15 [INFO] [TRAIN] epoch: 58, iter: 21320/40000, loss: 0.3971, lr: 0.005089, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1914 samples/sec | ETA 04:21:19 2021-05-06 17:53:24 [INFO] [TRAIN] epoch: 58, iter: 21330/40000, loss: 0.3520, lr: 0.005087, batch_cost: 0.8428, reader_cost: 0.00018, ips: 1.1865 samples/sec | ETA 04:22:15 2021-05-06 17:53:32 [INFO] [TRAIN] epoch: 58, iter: 21340/40000, loss: 0.5324, lr: 0.005085, batch_cost: 0.8396, reader_cost: 0.00016, ips: 1.1910 samples/sec | ETA 04:21:07 2021-05-06 17:53:41 [INFO] [TRAIN] epoch: 58, iter: 21350/40000, loss: 0.7028, lr: 0.005082, batch_cost: 0.8409, reader_cost: 0.00016, ips: 1.1893 samples/sec | ETA 04:21:22 2021-05-06 17:53:49 [INFO] [TRAIN] epoch: 58, iter: 21360/40000, loss: 0.3619, lr: 0.005080, batch_cost: 0.8421, reader_cost: 0.00017, ips: 1.1876 samples/sec | ETA 04:21:36 2021-05-06 17:53:57 [INFO] [TRAIN] epoch: 58, iter: 21370/40000, loss: 0.3341, lr: 0.005077, batch_cost: 0.8398, reader_cost: 0.00016, ips: 1.1907 samples/sec | ETA 04:20:46 2021-05-06 17:54:06 [INFO] [TRAIN] epoch: 58, iter: 21380/40000, loss: 0.3388, lr: 0.005075, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 04:20:55 2021-05-06 17:54:14 [INFO] [TRAIN] epoch: 58, iter: 21390/40000, loss: 0.4281, lr: 0.005072, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 04:20:18 2021-05-06 17:54:23 [INFO] [TRAIN] epoch: 58, iter: 21400/40000, loss: 0.4992, lr: 0.005070, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 04:20:18 2021-05-06 17:54:31 [INFO] [TRAIN] epoch: 58, iter: 21410/40000, loss: 0.4556, lr: 0.005068, batch_cost: 0.8400, reader_cost: 0.00014, ips: 1.1905 samples/sec | ETA 04:20:15 2021-05-06 17:54:40 [INFO] [TRAIN] epoch: 58, iter: 21420/40000, loss: 0.2880, lr: 0.005065, batch_cost: 0.8421, reader_cost: 0.00015, ips: 1.1874 samples/sec | ETA 04:20:46 2021-05-06 17:54:48 [INFO] [TRAIN] epoch: 58, iter: 21430/40000, loss: 0.2591, lr: 0.005063, batch_cost: 0.8399, reader_cost: 0.00014, ips: 1.1907 samples/sec | ETA 04:19:56 2021-05-06 17:54:56 [INFO] [TRAIN] epoch: 58, iter: 21440/40000, loss: 0.2960, lr: 0.005060, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 04:19:49 2021-05-06 17:55:05 [INFO] [TRAIN] epoch: 58, iter: 21450/40000, loss: 0.2907, lr: 0.005058, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 04:19:56 2021-05-06 17:55:13 [INFO] [TRAIN] epoch: 58, iter: 21460/40000, loss: 0.2945, lr: 0.005056, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 04:19:48 2021-05-06 17:55:22 [INFO] [TRAIN] epoch: 58, iter: 21470/40000, loss: 0.2745, lr: 0.005053, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 04:19:40 2021-05-06 17:55:30 [INFO] [TRAIN] epoch: 58, iter: 21480/40000, loss: 0.2905, lr: 0.005051, batch_cost: 0.8421, reader_cost: 0.00015, ips: 1.1875 samples/sec | ETA 04:19:55 2021-05-06 17:55:38 [INFO] [TRAIN] epoch: 58, iter: 21490/40000, loss: 0.3329, lr: 0.005048, batch_cost: 0.8419, reader_cost: 0.00015, ips: 1.1878 samples/sec | ETA 04:19:42 2021-05-06 17:55:47 [INFO] [TRAIN] epoch: 58, iter: 21500/40000, loss: 0.4305, lr: 0.005046, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 04:19:10 2021-05-06 17:55:55 [INFO] [TRAIN] epoch: 58, iter: 21510/40000, loss: 0.3295, lr: 0.005044, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 04:18:58 2021-05-06 17:56:04 [INFO] [TRAIN] epoch: 58, iter: 21520/40000, loss: 0.2287, lr: 0.005041, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1881 samples/sec | ETA 04:19:13 2021-05-06 17:56:12 [INFO] [TRAIN] epoch: 58, iter: 21530/40000, loss: 0.4582, lr: 0.005039, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 04:18:47 2021-05-06 17:56:20 [INFO] [TRAIN] epoch: 58, iter: 21540/40000, loss: 0.2808, lr: 0.005036, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1880 samples/sec | ETA 04:18:58 2021-05-06 17:56:29 [INFO] [TRAIN] epoch: 58, iter: 21550/40000, loss: 0.3992, lr: 0.005034, batch_cost: 0.8410, reader_cost: 0.00014, ips: 1.1891 samples/sec | ETA 04:18:35 2021-05-06 17:56:37 [INFO] [TRAIN] epoch: 58, iter: 21560/40000, loss: 0.2964, lr: 0.005032, batch_cost: 0.8405, reader_cost: 0.00014, ips: 1.1897 samples/sec | ETA 04:18:19 2021-05-06 17:56:46 [INFO] [TRAIN] epoch: 58, iter: 21570/40000, loss: 0.3984, lr: 0.005029, batch_cost: 0.8394, reader_cost: 0.00011, ips: 1.1913 samples/sec | ETA 04:17:50 2021-05-06 17:56:58 [INFO] [TRAIN] epoch: 59, iter: 21580/40000, loss: 0.4214, lr: 0.005027, batch_cost: 1.1886, reader_cost: 0.21158, ips: 0.8413 samples/sec | ETA 06:04:53 2021-05-06 17:57:06 [INFO] [TRAIN] epoch: 59, iter: 21590/40000, loss: 0.4224, lr: 0.005024, batch_cost: 0.8522, reader_cost: 0.00031, ips: 1.1734 samples/sec | ETA 04:21:28 2021-05-06 17:57:14 [INFO] [TRAIN] epoch: 59, iter: 21600/40000, loss: 0.4208, lr: 0.005022, batch_cost: 0.8390, reader_cost: 0.00015, ips: 1.1919 samples/sec | ETA 04:17:17 2021-05-06 17:57:23 [INFO] [TRAIN] epoch: 59, iter: 21610/40000, loss: 0.3754, lr: 0.005020, batch_cost: 0.8404, reader_cost: 0.00014, ips: 1.1900 samples/sec | ETA 04:17:34 2021-05-06 17:57:31 [INFO] [TRAIN] epoch: 59, iter: 21620/40000, loss: 0.2705, lr: 0.005017, batch_cost: 0.8426, reader_cost: 0.00015, ips: 1.1868 samples/sec | ETA 04:18:07 2021-05-06 17:57:40 [INFO] [TRAIN] epoch: 59, iter: 21630/40000, loss: 0.4544, lr: 0.005015, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1880 samples/sec | ETA 04:17:43 2021-05-06 17:57:48 [INFO] [TRAIN] epoch: 59, iter: 21640/40000, loss: 0.3813, lr: 0.005012, batch_cost: 0.8404, reader_cost: 0.00016, ips: 1.1900 samples/sec | ETA 04:17:09 2021-05-06 17:57:57 [INFO] [TRAIN] epoch: 59, iter: 21650/40000, loss: 0.6045, lr: 0.005010, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1881 samples/sec | ETA 04:17:24 2021-05-06 17:58:05 [INFO] [TRAIN] epoch: 59, iter: 21660/40000, loss: 0.4759, lr: 0.005008, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 04:16:54 2021-05-06 17:58:13 [INFO] [TRAIN] epoch: 59, iter: 21670/40000, loss: 0.2814, lr: 0.005005, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1880 samples/sec | ETA 04:17:09 2021-05-06 17:58:22 [INFO] [TRAIN] epoch: 59, iter: 21680/40000, loss: 0.2471, lr: 0.005003, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 04:16:47 2021-05-06 17:58:30 [INFO] [TRAIN] epoch: 59, iter: 21690/40000, loss: 0.2895, lr: 0.005000, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 04:16:35 2021-05-06 17:58:39 [INFO] [TRAIN] epoch: 59, iter: 21700/40000, loss: 0.4576, lr: 0.004998, batch_cost: 0.8413, reader_cost: 0.00016, ips: 1.1886 samples/sec | ETA 04:16:36 2021-05-06 17:58:47 [INFO] [TRAIN] epoch: 59, iter: 21710/40000, loss: 0.5004, lr: 0.004995, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1913 samples/sec | ETA 04:15:52 2021-05-06 17:58:55 [INFO] [TRAIN] epoch: 59, iter: 21720/40000, loss: 0.4627, lr: 0.004993, batch_cost: 0.8405, reader_cost: 0.00016, ips: 1.1897 samples/sec | ETA 04:16:05 2021-05-06 17:59:04 [INFO] [TRAIN] epoch: 59, iter: 21730/40000, loss: 0.2628, lr: 0.004991, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1912 samples/sec | ETA 04:15:36 2021-05-06 17:59:12 [INFO] [TRAIN] epoch: 59, iter: 21740/40000, loss: 0.4107, lr: 0.004988, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1881 samples/sec | ETA 04:16:09 2021-05-06 17:59:21 [INFO] [TRAIN] epoch: 59, iter: 21750/40000, loss: 0.3391, lr: 0.004986, batch_cost: 0.8388, reader_cost: 0.00014, ips: 1.1922 samples/sec | ETA 04:15:08 2021-05-06 17:59:29 [INFO] [TRAIN] epoch: 59, iter: 21760/40000, loss: 0.7126, lr: 0.004983, batch_cost: 0.8385, reader_cost: 0.00015, ips: 1.1926 samples/sec | ETA 04:14:54 2021-05-06 17:59:37 [INFO] [TRAIN] epoch: 59, iter: 21770/40000, loss: 0.5357, lr: 0.004981, batch_cost: 0.8417, reader_cost: 0.00016, ips: 1.1881 samples/sec | ETA 04:15:44 2021-05-06 17:59:46 [INFO] [TRAIN] epoch: 59, iter: 21780/40000, loss: 0.2998, lr: 0.004979, batch_cost: 0.8403, reader_cost: 0.00016, ips: 1.1900 samples/sec | ETA 04:15:10 2021-05-06 17:59:54 [INFO] [TRAIN] epoch: 59, iter: 21790/40000, loss: 0.3045, lr: 0.004976, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 04:14:59 2021-05-06 18:00:03 [INFO] [TRAIN] epoch: 59, iter: 21800/40000, loss: 0.2787, lr: 0.004974, batch_cost: 0.8419, reader_cost: 0.00015, ips: 1.1877 samples/sec | ETA 04:15:23 2021-05-06 18:00:11 [INFO] [TRAIN] epoch: 59, iter: 21810/40000, loss: 0.1852, lr: 0.004971, batch_cost: 0.8426, reader_cost: 0.00015, ips: 1.1868 samples/sec | ETA 04:15:26 2021-05-06 18:00:19 [INFO] [TRAIN] epoch: 59, iter: 21820/40000, loss: 0.2813, lr: 0.004969, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 04:14:43 2021-05-06 18:00:28 [INFO] [TRAIN] epoch: 59, iter: 21830/40000, loss: 0.3316, lr: 0.004967, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 04:14:41 2021-05-06 18:00:36 [INFO] [TRAIN] epoch: 59, iter: 21840/40000, loss: 0.3901, lr: 0.004964, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 04:14:29 2021-05-06 18:00:45 [INFO] [TRAIN] epoch: 59, iter: 21850/40000, loss: 0.3412, lr: 0.004962, batch_cost: 0.8407, reader_cost: 0.00014, ips: 1.1895 samples/sec | ETA 04:14:19 2021-05-06 18:00:53 [INFO] [TRAIN] epoch: 59, iter: 21860/40000, loss: 0.2801, lr: 0.004959, batch_cost: 0.8413, reader_cost: 0.00014, ips: 1.1886 samples/sec | ETA 04:14:21 2021-05-06 18:01:01 [INFO] [TRAIN] epoch: 59, iter: 21870/40000, loss: 0.4288, lr: 0.004957, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1886 samples/sec | ETA 04:14:13 2021-05-06 18:01:10 [INFO] [TRAIN] epoch: 59, iter: 21880/40000, loss: 0.2890, lr: 0.004955, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 04:13:59 2021-05-06 18:01:18 [INFO] [TRAIN] epoch: 59, iter: 21890/40000, loss: 0.2553, lr: 0.004952, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 04:13:28 2021-05-06 18:01:27 [INFO] [TRAIN] epoch: 59, iter: 21900/40000, loss: 0.5391, lr: 0.004950, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 04:13:26 2021-05-06 18:01:35 [INFO] [TRAIN] epoch: 59, iter: 21910/40000, loss: 0.3426, lr: 0.004947, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 04:13:13 2021-05-06 18:01:44 [INFO] [TRAIN] epoch: 59, iter: 21920/40000, loss: 0.4008, lr: 0.004945, batch_cost: 0.8392, reader_cost: 0.00016, ips: 1.1917 samples/sec | ETA 04:12:52 2021-05-06 18:01:52 [INFO] [TRAIN] epoch: 59, iter: 21930/40000, loss: 0.3285, lr: 0.004942, batch_cost: 0.8420, reader_cost: 0.00016, ips: 1.1876 samples/sec | ETA 04:13:35 2021-05-06 18:02:00 [INFO] [TRAIN] epoch: 59, iter: 21940/40000, loss: 0.4705, lr: 0.004940, batch_cost: 0.8411, reader_cost: 0.00014, ips: 1.1889 samples/sec | ETA 04:13:10 2021-05-06 18:02:12 [INFO] [TRAIN] epoch: 60, iter: 21950/40000, loss: 0.4448, lr: 0.004938, batch_cost: 1.1465, reader_cost: 0.21270, ips: 0.8722 samples/sec | ETA 05:44:54 2021-05-06 18:02:20 [INFO] [TRAIN] epoch: 60, iter: 21960/40000, loss: 0.5546, lr: 0.004935, batch_cost: 0.8527, reader_cost: 0.00031, ips: 1.1727 samples/sec | ETA 04:16:23 2021-05-06 18:02:29 [INFO] [TRAIN] epoch: 60, iter: 21970/40000, loss: 0.3673, lr: 0.004933, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1882 samples/sec | ETA 04:12:54 2021-05-06 18:02:37 [INFO] [TRAIN] epoch: 60, iter: 21980/40000, loss: 0.3124, lr: 0.004930, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 04:12:20 2021-05-06 18:02:46 [INFO] [TRAIN] epoch: 60, iter: 21990/40000, loss: 0.3191, lr: 0.004928, batch_cost: 0.8420, reader_cost: 0.00015, ips: 1.1877 samples/sec | ETA 04:12:43 2021-05-06 18:02:54 [INFO] [TRAIN] epoch: 60, iter: 22000/40000, loss: 0.5234, lr: 0.004926, batch_cost: 0.8422, reader_cost: 0.00016, ips: 1.1873 samples/sec | ETA 04:12:40 2021-05-06 18:02:54 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 18:06:32 [INFO] [EVAL] #Images: 500 mIoU: 0.7598 Acc: 0.9565 Kappa: 0.9435 2021-05-06 18:06:32 [INFO] [EVAL] Class IoU: [0.9803 0.8427 0.9178 0.6486 0.6074 0.4944 0.5772 0.716 0.9173 0.6494 0.9436 0.7726 0.5855 0.9397 0.8198 0.8875 0.7982 0.617 0.7207] 2021-05-06 18:06:32 [INFO] [EVAL] Class Acc: [0.9911 0.9009 0.9487 0.8522 0.7451 0.8101 0.8313 0.8995 0.9487 0.8419 0.9647 0.8849 0.7563 0.9679 0.8971 0.9465 0.9094 0.8252 0.8377] 2021-05-06 18:07:23 [INFO] [EVAL] The model with the best validation mIoU (0.7598) was saved at iter 22000. 2021-05-06 18:07:31 [INFO] [TRAIN] epoch: 60, iter: 22010/40000, loss: 0.5234, lr: 0.004923, batch_cost: 0.8359, reader_cost: 0.00025, ips: 1.1963 samples/sec | ETA 04:10:38 2021-05-06 18:07:39 [INFO] [TRAIN] epoch: 60, iter: 22020/40000, loss: 0.4800, lr: 0.004921, batch_cost: 0.8371, reader_cost: 0.00014, ips: 1.1946 samples/sec | ETA 04:10:50 2021-05-06 18:07:48 [INFO] [TRAIN] epoch: 60, iter: 22030/40000, loss: 0.4970, lr: 0.004918, batch_cost: 0.8383, reader_cost: 0.00014, ips: 1.1929 samples/sec | ETA 04:11:03 2021-05-06 18:07:56 [INFO] [TRAIN] epoch: 60, iter: 22040/40000, loss: 0.3465, lr: 0.004916, batch_cost: 0.8400, reader_cost: 0.00014, ips: 1.1905 samples/sec | ETA 04:11:26 2021-05-06 18:08:05 [INFO] [TRAIN] epoch: 60, iter: 22050/40000, loss: 0.1648, lr: 0.004914, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 04:11:31 2021-05-06 18:08:13 [INFO] [TRAIN] epoch: 60, iter: 22060/40000, loss: 0.2972, lr: 0.004911, batch_cost: 0.8435, reader_cost: 0.00015, ips: 1.1855 samples/sec | ETA 04:12:13 2021-05-06 18:08:21 [INFO] [TRAIN] epoch: 60, iter: 22070/40000, loss: 0.4976, lr: 0.004909, batch_cost: 0.8393, reader_cost: 0.00014, ips: 1.1915 samples/sec | ETA 04:10:48 2021-05-06 18:08:30 [INFO] [TRAIN] epoch: 60, iter: 22080/40000, loss: 0.4771, lr: 0.004906, batch_cost: 0.8390, reader_cost: 0.00016, ips: 1.1920 samples/sec | ETA 04:10:34 2021-05-06 18:08:38 [INFO] [TRAIN] epoch: 60, iter: 22090/40000, loss: 0.5144, lr: 0.004904, batch_cost: 0.8396, reader_cost: 0.00014, ips: 1.1910 samples/sec | ETA 04:10:37 2021-05-06 18:08:47 [INFO] [TRAIN] epoch: 60, iter: 22100/40000, loss: 0.4976, lr: 0.004901, batch_cost: 0.8422, reader_cost: 0.00014, ips: 1.1874 samples/sec | ETA 04:11:14 2021-05-06 18:08:55 [INFO] [TRAIN] epoch: 60, iter: 22110/40000, loss: 0.3413, lr: 0.004899, batch_cost: 0.8424, reader_cost: 0.00014, ips: 1.1871 samples/sec | ETA 04:11:10 2021-05-06 18:09:03 [INFO] [TRAIN] epoch: 60, iter: 22120/40000, loss: 0.3579, lr: 0.004897, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1903 samples/sec | ETA 04:10:20 2021-05-06 18:09:12 [INFO] [TRAIN] epoch: 60, iter: 22130/40000, loss: 0.2493, lr: 0.004894, batch_cost: 0.8423, reader_cost: 0.00014, ips: 1.1872 samples/sec | ETA 04:10:52 2021-05-06 18:09:20 [INFO] [TRAIN] epoch: 60, iter: 22140/40000, loss: 0.5796, lr: 0.004892, batch_cost: 0.8396, reader_cost: 0.00014, ips: 1.1911 samples/sec | ETA 04:09:54 2021-05-06 18:09:29 [INFO] [TRAIN] epoch: 60, iter: 22150/40000, loss: 0.4719, lr: 0.004889, batch_cost: 0.8413, reader_cost: 0.00014, ips: 1.1886 samples/sec | ETA 04:10:17 2021-05-06 18:09:37 [INFO] [TRAIN] epoch: 60, iter: 22160/40000, loss: 0.4398, lr: 0.004887, batch_cost: 0.8396, reader_cost: 0.00014, ips: 1.1910 samples/sec | ETA 04:09:38 2021-05-06 18:09:45 [INFO] [TRAIN] epoch: 60, iter: 22170/40000, loss: 0.4376, lr: 0.004885, batch_cost: 0.8403, reader_cost: 0.00014, ips: 1.1901 samples/sec | ETA 04:09:42 2021-05-06 18:09:54 [INFO] [TRAIN] epoch: 60, iter: 22180/40000, loss: 0.2353, lr: 0.004882, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1887 samples/sec | ETA 04:09:50 2021-05-06 18:10:02 [INFO] [TRAIN] epoch: 60, iter: 22190/40000, loss: 0.3391, lr: 0.004880, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 04:09:16 2021-05-06 18:10:11 [INFO] [TRAIN] epoch: 60, iter: 22200/40000, loss: 0.2529, lr: 0.004877, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 04:09:11 2021-05-06 18:10:19 [INFO] [TRAIN] epoch: 60, iter: 22210/40000, loss: 0.3116, lr: 0.004875, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 04:09:20 2021-05-06 18:10:27 [INFO] [TRAIN] epoch: 60, iter: 22220/40000, loss: 0.3371, lr: 0.004872, batch_cost: 0.8406, reader_cost: 0.00016, ips: 1.1897 samples/sec | ETA 04:09:05 2021-05-06 18:10:36 [INFO] [TRAIN] epoch: 60, iter: 22230/40000, loss: 0.3394, lr: 0.004870, batch_cost: 0.8423, reader_cost: 0.00015, ips: 1.1872 samples/sec | ETA 04:09:27 2021-05-06 18:10:44 [INFO] [TRAIN] epoch: 60, iter: 22240/40000, loss: 0.3005, lr: 0.004868, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1917 samples/sec | ETA 04:08:23 2021-05-06 18:10:53 [INFO] [TRAIN] epoch: 60, iter: 22250/40000, loss: 0.3578, lr: 0.004865, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 04:08:38 2021-05-06 18:11:01 [INFO] [TRAIN] epoch: 60, iter: 22260/40000, loss: 0.3643, lr: 0.004863, batch_cost: 0.8385, reader_cost: 0.00014, ips: 1.1925 samples/sec | ETA 04:07:55 2021-05-06 18:11:10 [INFO] [TRAIN] epoch: 60, iter: 22270/40000, loss: 0.6931, lr: 0.004860, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 04:08:18 2021-05-06 18:11:18 [INFO] [TRAIN] epoch: 60, iter: 22280/40000, loss: 0.4118, lr: 0.004858, batch_cost: 0.8407, reader_cost: 0.00014, ips: 1.1894 samples/sec | ETA 04:08:18 2021-05-06 18:11:26 [INFO] [TRAIN] epoch: 60, iter: 22290/40000, loss: 0.4264, lr: 0.004856, batch_cost: 0.8399, reader_cost: 0.00016, ips: 1.1906 samples/sec | ETA 04:07:54 2021-05-06 18:11:35 [INFO] [TRAIN] epoch: 60, iter: 22300/40000, loss: 0.3172, lr: 0.004853, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 04:07:57 2021-05-06 18:11:43 [INFO] [TRAIN] epoch: 60, iter: 22310/40000, loss: 0.4476, lr: 0.004851, batch_cost: 0.8415, reader_cost: 0.00014, ips: 1.1884 samples/sec | ETA 04:08:05 2021-05-06 18:11:52 [INFO] [TRAIN] epoch: 60, iter: 22320/40000, loss: 0.2881, lr: 0.004848, batch_cost: 0.8390, reader_cost: 0.00009, ips: 1.1919 samples/sec | ETA 04:07:14 2021-05-06 18:12:04 [INFO] [TRAIN] epoch: 61, iter: 22330/40000, loss: 0.4713, lr: 0.004846, batch_cost: 1.2124, reader_cost: 0.19257, ips: 0.8248 samples/sec | ETA 05:57:02 2021-05-06 18:12:12 [INFO] [TRAIN] epoch: 61, iter: 22340/40000, loss: 0.3309, lr: 0.004843, batch_cost: 0.8430, reader_cost: 0.00031, ips: 1.1863 samples/sec | ETA 04:08:06 2021-05-06 18:12:20 [INFO] [TRAIN] epoch: 61, iter: 22350/40000, loss: 0.4631, lr: 0.004841, batch_cost: 0.8385, reader_cost: 0.00015, ips: 1.1926 samples/sec | ETA 04:06:40 2021-05-06 18:12:29 [INFO] [TRAIN] epoch: 61, iter: 22360/40000, loss: 0.2469, lr: 0.004839, batch_cost: 0.8415, reader_cost: 0.00016, ips: 1.1883 samples/sec | ETA 04:07:24 2021-05-06 18:12:37 [INFO] [TRAIN] epoch: 61, iter: 22370/40000, loss: 0.3511, lr: 0.004836, batch_cost: 0.8434, reader_cost: 0.00016, ips: 1.1857 samples/sec | ETA 04:07:48 2021-05-06 18:12:46 [INFO] [TRAIN] epoch: 61, iter: 22380/40000, loss: 0.4168, lr: 0.004834, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1888 samples/sec | ETA 04:07:01 2021-05-06 18:12:54 [INFO] [TRAIN] epoch: 61, iter: 22390/40000, loss: 0.4238, lr: 0.004831, batch_cost: 0.8397, reader_cost: 0.00016, ips: 1.1908 samples/sec | ETA 04:06:27 2021-05-06 18:13:03 [INFO] [TRAIN] epoch: 61, iter: 22400/40000, loss: 0.5096, lr: 0.004829, batch_cost: 0.8394, reader_cost: 0.00014, ips: 1.1913 samples/sec | ETA 04:06:13 2021-05-06 18:13:11 [INFO] [TRAIN] epoch: 61, iter: 22410/40000, loss: 0.5293, lr: 0.004827, batch_cost: 0.8406, reader_cost: 0.00014, ips: 1.1896 samples/sec | ETA 04:06:26 2021-05-06 18:13:19 [INFO] [TRAIN] epoch: 61, iter: 22420/40000, loss: 0.1200, lr: 0.004824, batch_cost: 0.8424, reader_cost: 0.00015, ips: 1.1871 samples/sec | ETA 04:06:48 2021-05-06 18:13:28 [INFO] [TRAIN] epoch: 61, iter: 22430/40000, loss: 0.3372, lr: 0.004822, batch_cost: 0.8409, reader_cost: 0.00017, ips: 1.1892 samples/sec | ETA 04:06:14 2021-05-06 18:13:36 [INFO] [TRAIN] epoch: 61, iter: 22440/40000, loss: 0.3881, lr: 0.004819, batch_cost: 0.8424, reader_cost: 0.00016, ips: 1.1871 samples/sec | ETA 04:06:32 2021-05-06 18:13:45 [INFO] [TRAIN] epoch: 61, iter: 22450/40000, loss: 0.4490, lr: 0.004817, batch_cost: 0.8409, reader_cost: 0.00016, ips: 1.1892 samples/sec | ETA 04:05:57 2021-05-06 18:13:53 [INFO] [TRAIN] epoch: 61, iter: 22460/40000, loss: 0.5675, lr: 0.004814, batch_cost: 0.8407, reader_cost: 0.00014, ips: 1.1895 samples/sec | ETA 04:05:45 2021-05-06 18:14:01 [INFO] [TRAIN] epoch: 61, iter: 22470/40000, loss: 0.4135, lr: 0.004812, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1879 samples/sec | ETA 04:05:57 2021-05-06 18:14:10 [INFO] [TRAIN] epoch: 61, iter: 22480/40000, loss: 0.3109, lr: 0.004810, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1887 samples/sec | ETA 04:05:38 2021-05-06 18:14:18 [INFO] [TRAIN] epoch: 61, iter: 22490/40000, loss: 0.4289, lr: 0.004807, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 04:05:03 2021-05-06 18:14:27 [INFO] [TRAIN] epoch: 61, iter: 22500/40000, loss: 0.4601, lr: 0.004805, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1914 samples/sec | ETA 04:04:48 2021-05-06 18:14:35 [INFO] [TRAIN] epoch: 61, iter: 22510/40000, loss: 0.6562, lr: 0.004802, batch_cost: 0.8404, reader_cost: 0.00016, ips: 1.1900 samples/sec | ETA 04:04:58 2021-05-06 18:14:43 [INFO] [TRAIN] epoch: 61, iter: 22520/40000, loss: 0.5667, lr: 0.004800, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 04:04:58 2021-05-06 18:14:52 [INFO] [TRAIN] epoch: 61, iter: 22530/40000, loss: 0.4357, lr: 0.004798, batch_cost: 0.8396, reader_cost: 0.00014, ips: 1.1910 samples/sec | ETA 04:04:28 2021-05-06 18:15:00 [INFO] [TRAIN] epoch: 61, iter: 22540/40000, loss: 0.2342, lr: 0.004795, batch_cost: 0.8403, reader_cost: 0.00014, ips: 1.1901 samples/sec | ETA 04:04:31 2021-05-06 18:15:09 [INFO] [TRAIN] epoch: 61, iter: 22550/40000, loss: 0.2015, lr: 0.004793, batch_cost: 0.8392, reader_cost: 0.00014, ips: 1.1917 samples/sec | ETA 04:04:03 2021-05-06 18:15:17 [INFO] [TRAIN] epoch: 61, iter: 22560/40000, loss: 0.2976, lr: 0.004790, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 04:04:19 2021-05-06 18:15:25 [INFO] [TRAIN] epoch: 61, iter: 22570/40000, loss: 0.2384, lr: 0.004788, batch_cost: 0.8405, reader_cost: 0.00016, ips: 1.1897 samples/sec | ETA 04:04:10 2021-05-06 18:15:34 [INFO] [TRAIN] epoch: 61, iter: 22580/40000, loss: 0.3219, lr: 0.004785, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 04:04:00 2021-05-06 18:15:42 [INFO] [TRAIN] epoch: 61, iter: 22590/40000, loss: 0.2384, lr: 0.004783, batch_cost: 0.8420, reader_cost: 0.00015, ips: 1.1876 samples/sec | ETA 04:04:19 2021-05-06 18:15:51 [INFO] [TRAIN] epoch: 61, iter: 22600/40000, loss: 0.3462, lr: 0.004781, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 04:03:39 2021-05-06 18:15:59 [INFO] [TRAIN] epoch: 61, iter: 22610/40000, loss: 0.5598, lr: 0.004778, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1882 samples/sec | ETA 04:03:56 2021-05-06 18:16:07 [INFO] [TRAIN] epoch: 61, iter: 22620/40000, loss: 0.3940, lr: 0.004776, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 04:03:13 2021-05-06 18:16:16 [INFO] [TRAIN] epoch: 61, iter: 22630/40000, loss: 0.2299, lr: 0.004773, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 04:03:16 2021-05-06 18:16:24 [INFO] [TRAIN] epoch: 61, iter: 22640/40000, loss: 0.4112, lr: 0.004771, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 04:03:13 2021-05-06 18:16:33 [INFO] [TRAIN] epoch: 61, iter: 22650/40000, loss: 0.4996, lr: 0.004768, batch_cost: 0.8421, reader_cost: 0.00015, ips: 1.1875 samples/sec | ETA 04:03:30 2021-05-06 18:16:41 [INFO] [TRAIN] epoch: 61, iter: 22660/40000, loss: 0.2872, lr: 0.004766, batch_cost: 0.8442, reader_cost: 0.00014, ips: 1.1846 samples/sec | ETA 04:03:57 2021-05-06 18:16:50 [INFO] [TRAIN] epoch: 61, iter: 22670/40000, loss: 0.5021, lr: 0.004764, batch_cost: 0.8421, reader_cost: 0.00015, ips: 1.1875 samples/sec | ETA 04:03:13 2021-05-06 18:16:58 [INFO] [TRAIN] epoch: 61, iter: 22680/40000, loss: 0.3264, lr: 0.004761, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1886 samples/sec | ETA 04:02:52 2021-05-06 18:17:06 [INFO] [TRAIN] epoch: 61, iter: 22690/40000, loss: 0.2777, lr: 0.004759, batch_cost: 0.8378, reader_cost: 0.00027, ips: 1.1936 samples/sec | ETA 04:01:42 2021-05-06 18:17:18 [INFO] [TRAIN] epoch: 62, iter: 22700/40000, loss: 0.5326, lr: 0.004756, batch_cost: 1.1595, reader_cost: 0.21716, ips: 0.8624 samples/sec | ETA 05:34:19 2021-05-06 18:17:26 [INFO] [TRAIN] epoch: 62, iter: 22710/40000, loss: 0.3451, lr: 0.004754, batch_cost: 0.8408, reader_cost: 0.00032, ips: 1.1893 samples/sec | ETA 04:02:17 2021-05-06 18:17:35 [INFO] [TRAIN] epoch: 62, iter: 22720/40000, loss: 0.3889, lr: 0.004752, batch_cost: 0.8416, reader_cost: 0.00016, ips: 1.1882 samples/sec | ETA 04:02:22 2021-05-06 18:17:43 [INFO] [TRAIN] epoch: 62, iter: 22730/40000, loss: 0.3015, lr: 0.004749, batch_cost: 0.8412, reader_cost: 0.00016, ips: 1.1887 samples/sec | ETA 04:02:07 2021-05-06 18:17:52 [INFO] [TRAIN] epoch: 62, iter: 22740/40000, loss: 0.3847, lr: 0.004747, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 04:01:35 2021-05-06 18:18:00 [INFO] [TRAIN] epoch: 62, iter: 22750/40000, loss: 0.4165, lr: 0.004744, batch_cost: 0.8433, reader_cost: 0.00015, ips: 1.1858 samples/sec | ETA 04:02:26 2021-05-06 18:18:08 [INFO] [TRAIN] epoch: 62, iter: 22760/40000, loss: 0.7175, lr: 0.004742, batch_cost: 0.8417, reader_cost: 0.00016, ips: 1.1880 samples/sec | ETA 04:01:51 2021-05-06 18:18:17 [INFO] [TRAIN] epoch: 62, iter: 22770/40000, loss: 0.6663, lr: 0.004739, batch_cost: 0.8395, reader_cost: 0.00014, ips: 1.1912 samples/sec | ETA 04:01:04 2021-05-06 18:18:25 [INFO] [TRAIN] epoch: 62, iter: 22780/40000, loss: 0.3598, lr: 0.004737, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1886 samples/sec | ETA 04:01:27 2021-05-06 18:18:34 [INFO] [TRAIN] epoch: 62, iter: 22790/40000, loss: 0.1431, lr: 0.004735, batch_cost: 0.8398, reader_cost: 0.00014, ips: 1.1908 samples/sec | ETA 04:00:52 2021-05-06 18:18:42 [INFO] [TRAIN] epoch: 62, iter: 22800/40000, loss: 0.3385, lr: 0.004732, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 04:00:51 2021-05-06 18:18:50 [INFO] [TRAIN] epoch: 62, iter: 22810/40000, loss: 0.3426, lr: 0.004730, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1888 samples/sec | ETA 04:00:59 2021-05-06 18:18:59 [INFO] [TRAIN] epoch: 62, iter: 22820/40000, loss: 0.3736, lr: 0.004727, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 04:00:35 2021-05-06 18:19:07 [INFO] [TRAIN] epoch: 62, iter: 22830/40000, loss: 0.5742, lr: 0.004725, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 04:00:32 2021-05-06 18:19:16 [INFO] [TRAIN] epoch: 62, iter: 22840/40000, loss: 0.4329, lr: 0.004722, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 04:00:30 2021-05-06 18:19:24 [INFO] [TRAIN] epoch: 62, iter: 22850/40000, loss: 0.3359, lr: 0.004720, batch_cost: 0.8436, reader_cost: 0.00015, ips: 1.1854 samples/sec | ETA 04:01:07 2021-05-06 18:19:33 [INFO] [TRAIN] epoch: 62, iter: 22860/40000, loss: 0.4250, lr: 0.004718, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1882 samples/sec | ETA 04:00:25 2021-05-06 18:19:41 [INFO] [TRAIN] epoch: 62, iter: 22870/40000, loss: 0.2587, lr: 0.004715, batch_cost: 0.8432, reader_cost: 0.00016, ips: 1.1860 samples/sec | ETA 04:00:43 2021-05-06 18:19:49 [INFO] [TRAIN] epoch: 62, iter: 22880/40000, loss: 0.4368, lr: 0.004713, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 03:59:49 2021-05-06 18:19:58 [INFO] [TRAIN] epoch: 62, iter: 22890/40000, loss: 0.4585, lr: 0.004710, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 03:59:43 2021-05-06 18:20:06 [INFO] [TRAIN] epoch: 62, iter: 22900/40000, loss: 0.3628, lr: 0.004708, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1887 samples/sec | ETA 03:59:44 2021-05-06 18:20:15 [INFO] [TRAIN] epoch: 62, iter: 22910/40000, loss: 0.4660, lr: 0.004705, batch_cost: 0.8407, reader_cost: 0.00016, ips: 1.1895 samples/sec | ETA 03:59:27 2021-05-06 18:20:23 [INFO] [TRAIN] epoch: 62, iter: 22920/40000, loss: 0.2912, lr: 0.004703, batch_cost: 0.8405, reader_cost: 0.00016, ips: 1.1898 samples/sec | ETA 03:59:15 2021-05-06 18:20:31 [INFO] [TRAIN] epoch: 62, iter: 22930/40000, loss: 0.3271, lr: 0.004701, batch_cost: 0.8409, reader_cost: 0.00016, ips: 1.1892 samples/sec | ETA 03:59:14 2021-05-06 18:20:40 [INFO] [TRAIN] epoch: 62, iter: 22940/40000, loss: 0.2541, lr: 0.004698, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1881 samples/sec | ETA 03:59:18 2021-05-06 18:20:48 [INFO] [TRAIN] epoch: 62, iter: 22950/40000, loss: 0.3856, lr: 0.004696, batch_cost: 0.8406, reader_cost: 0.00016, ips: 1.1897 samples/sec | ETA 03:58:51 2021-05-06 18:20:57 [INFO] [TRAIN] epoch: 62, iter: 22960/40000, loss: 0.3691, lr: 0.004693, batch_cost: 0.8417, reader_cost: 0.00014, ips: 1.1880 samples/sec | ETA 03:59:02 2021-05-06 18:21:05 [INFO] [TRAIN] epoch: 62, iter: 22970/40000, loss: 0.2393, lr: 0.004691, batch_cost: 0.8431, reader_cost: 0.00014, ips: 1.1862 samples/sec | ETA 03:59:17 2021-05-06 18:21:14 [INFO] [TRAIN] epoch: 62, iter: 22980/40000, loss: 0.3408, lr: 0.004688, batch_cost: 0.8420, reader_cost: 0.00016, ips: 1.1877 samples/sec | ETA 03:58:50 2021-05-06 18:21:22 [INFO] [TRAIN] epoch: 62, iter: 22990/40000, loss: 0.4281, lr: 0.004686, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1880 samples/sec | ETA 03:58:38 2021-05-06 18:21:30 [INFO] [TRAIN] epoch: 62, iter: 23000/40000, loss: 0.2688, lr: 0.004684, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1885 samples/sec | ETA 03:58:23 2021-05-06 18:21:30 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 18:25:07 [INFO] [EVAL] #Images: 500 mIoU: 0.7587 Acc: 0.9561 Kappa: 0.9431 2021-05-06 18:25:07 [INFO] [EVAL] Class IoU: [0.9805 0.8455 0.9174 0.6376 0.6039 0.5234 0.6016 0.7266 0.9156 0.6548 0.9399 0.7751 0.5753 0.9375 0.7948 0.889 0.7712 0.6114 0.7133] 2021-05-06 18:25:07 [INFO] [EVAL] Class Acc: [0.993 0.903 0.9558 0.8535 0.7752 0.7541 0.74 0.8596 0.9484 0.8045 0.9576 0.8715 0.766 0.9598 0.8773 0.9416 0.8815 0.7702 0.828 ] 2021-05-06 18:25:36 [INFO] [EVAL] The model with the best validation mIoU (0.7598) was saved at iter 22000. 2021-05-06 18:25:44 [INFO] [TRAIN] epoch: 62, iter: 23010/40000, loss: 0.3355, lr: 0.004681, batch_cost: 0.8381, reader_cost: 0.00025, ips: 1.1931 samples/sec | ETA 03:57:19 2021-05-06 18:25:53 [INFO] [TRAIN] epoch: 62, iter: 23020/40000, loss: 0.4718, lr: 0.004679, batch_cost: 0.8384, reader_cost: 0.00015, ips: 1.1928 samples/sec | ETA 03:57:15 2021-05-06 18:26:02 [INFO] [TRAIN] epoch: 62, iter: 23030/40000, loss: 0.2885, lr: 0.004676, batch_cost: 0.8407, reader_cost: 0.00017, ips: 1.1894 samples/sec | ETA 03:57:47 2021-05-06 18:26:10 [INFO] [TRAIN] epoch: 62, iter: 23040/40000, loss: 0.2724, lr: 0.004674, batch_cost: 0.8406, reader_cost: 0.00014, ips: 1.1896 samples/sec | ETA 03:57:36 2021-05-06 18:26:18 [INFO] [TRAIN] epoch: 62, iter: 23050/40000, loss: 0.3776, lr: 0.004671, batch_cost: 0.8387, reader_cost: 0.00015, ips: 1.1923 samples/sec | ETA 03:56:55 2021-05-06 18:26:27 [INFO] [TRAIN] epoch: 62, iter: 23060/40000, loss: 0.3483, lr: 0.004669, batch_cost: 0.8388, reader_cost: 0.00011, ips: 1.1922 samples/sec | ETA 03:56:48 2021-05-06 18:26:39 [INFO] [TRAIN] epoch: 63, iter: 23070/40000, loss: 0.5153, lr: 0.004667, batch_cost: 1.1660, reader_cost: 0.22314, ips: 0.8576 samples/sec | ETA 05:29:00 2021-05-06 18:26:47 [INFO] [TRAIN] epoch: 63, iter: 23080/40000, loss: 0.3060, lr: 0.004664, batch_cost: 0.8420, reader_cost: 0.00034, ips: 1.1876 samples/sec | ETA 03:57:26 2021-05-06 18:26:55 [INFO] [TRAIN] epoch: 63, iter: 23090/40000, loss: 0.4351, lr: 0.004662, batch_cost: 0.8424, reader_cost: 0.00016, ips: 1.1871 samples/sec | ETA 03:57:24 2021-05-06 18:27:04 [INFO] [TRAIN] epoch: 63, iter: 23100/40000, loss: 0.3853, lr: 0.004659, batch_cost: 0.8426, reader_cost: 0.00015, ips: 1.1868 samples/sec | ETA 03:57:20 2021-05-06 18:27:12 [INFO] [TRAIN] epoch: 63, iter: 23110/40000, loss: 0.3543, lr: 0.004657, batch_cost: 0.8427, reader_cost: 0.00015, ips: 1.1867 samples/sec | ETA 03:57:12 2021-05-06 18:27:21 [INFO] [TRAIN] epoch: 63, iter: 23120/40000, loss: 0.3623, lr: 0.004654, batch_cost: 0.8413, reader_cost: 0.00017, ips: 1.1887 samples/sec | ETA 03:56:40 2021-05-06 18:27:29 [INFO] [TRAIN] epoch: 63, iter: 23130/40000, loss: 0.4584, lr: 0.004652, batch_cost: 0.8398, reader_cost: 0.00017, ips: 1.1908 samples/sec | ETA 03:56:07 2021-05-06 18:27:37 [INFO] [TRAIN] epoch: 63, iter: 23140/40000, loss: 0.6207, lr: 0.004650, batch_cost: 0.8411, reader_cost: 0.00016, ips: 1.1889 samples/sec | ETA 03:56:21 2021-05-06 18:27:46 [INFO] [TRAIN] epoch: 63, iter: 23150/40000, loss: 0.3704, lr: 0.004647, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1912 samples/sec | ETA 03:55:44 2021-05-06 18:27:54 [INFO] [TRAIN] epoch: 63, iter: 23160/40000, loss: 0.1898, lr: 0.004645, batch_cost: 0.8399, reader_cost: 0.00014, ips: 1.1907 samples/sec | ETA 03:55:43 2021-05-06 18:28:03 [INFO] [TRAIN] epoch: 63, iter: 23170/40000, loss: 0.2313, lr: 0.004642, batch_cost: 0.8405, reader_cost: 0.00014, ips: 1.1898 samples/sec | ETA 03:55:45 2021-05-06 18:28:11 [INFO] [TRAIN] epoch: 63, iter: 23180/40000, loss: 0.2514, lr: 0.004640, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 03:55:38 2021-05-06 18:28:19 [INFO] [TRAIN] epoch: 63, iter: 23190/40000, loss: 0.4712, lr: 0.004637, batch_cost: 0.8423, reader_cost: 0.00016, ips: 1.1872 samples/sec | ETA 03:55:59 2021-05-06 18:28:28 [INFO] [TRAIN] epoch: 63, iter: 23200/40000, loss: 0.4949, lr: 0.004635, batch_cost: 0.8395, reader_cost: 0.00016, ips: 1.1912 samples/sec | ETA 03:55:03 2021-05-06 18:28:36 [INFO] [TRAIN] epoch: 63, iter: 23210/40000, loss: 0.4434, lr: 0.004633, batch_cost: 0.8402, reader_cost: 0.00016, ips: 1.1902 samples/sec | ETA 03:55:06 2021-05-06 18:28:45 [INFO] [TRAIN] epoch: 63, iter: 23220/40000, loss: 0.3761, lr: 0.004630, batch_cost: 0.8407, reader_cost: 0.00017, ips: 1.1894 samples/sec | ETA 03:55:07 2021-05-06 18:28:53 [INFO] [TRAIN] epoch: 63, iter: 23230/40000, loss: 0.2854, lr: 0.004628, batch_cost: 0.8384, reader_cost: 0.00016, ips: 1.1928 samples/sec | ETA 03:54:19 2021-05-06 18:29:01 [INFO] [TRAIN] epoch: 63, iter: 23240/40000, loss: 0.3720, lr: 0.004625, batch_cost: 0.8397, reader_cost: 0.00016, ips: 1.1908 samples/sec | ETA 03:54:34 2021-05-06 18:29:10 [INFO] [TRAIN] epoch: 63, iter: 23250/40000, loss: 0.5088, lr: 0.004623, batch_cost: 0.8388, reader_cost: 0.00017, ips: 1.1922 samples/sec | ETA 03:54:09 2021-05-06 18:29:18 [INFO] [TRAIN] epoch: 63, iter: 23260/40000, loss: 0.6298, lr: 0.004620, batch_cost: 0.8400, reader_cost: 0.00016, ips: 1.1905 samples/sec | ETA 03:54:21 2021-05-06 18:29:27 [INFO] [TRAIN] epoch: 63, iter: 23270/40000, loss: 0.4183, lr: 0.004618, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 03:54:30 2021-05-06 18:29:35 [INFO] [TRAIN] epoch: 63, iter: 23280/40000, loss: 0.4457, lr: 0.004616, batch_cost: 0.8418, reader_cost: 0.00016, ips: 1.1879 samples/sec | ETA 03:54:35 2021-05-06 18:29:43 [INFO] [TRAIN] epoch: 63, iter: 23290/40000, loss: 0.2227, lr: 0.004613, batch_cost: 0.8398, reader_cost: 0.00016, ips: 1.1908 samples/sec | ETA 03:53:52 2021-05-06 18:29:52 [INFO] [TRAIN] epoch: 63, iter: 23300/40000, loss: 0.3150, lr: 0.004611, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 03:53:49 2021-05-06 18:30:00 [INFO] [TRAIN] epoch: 63, iter: 23310/40000, loss: 0.2224, lr: 0.004608, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 03:53:54 2021-05-06 18:30:09 [INFO] [TRAIN] epoch: 63, iter: 23320/40000, loss: 0.3782, lr: 0.004606, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1918 samples/sec | ETA 03:53:15 2021-05-06 18:30:17 [INFO] [TRAIN] epoch: 63, iter: 23330/40000, loss: 0.4712, lr: 0.004603, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 03:53:18 2021-05-06 18:30:26 [INFO] [TRAIN] epoch: 63, iter: 23340/40000, loss: 0.2993, lr: 0.004601, batch_cost: 0.8414, reader_cost: 0.00016, ips: 1.1885 samples/sec | ETA 03:53:38 2021-05-06 18:30:34 [INFO] [TRAIN] epoch: 63, iter: 23350/40000, loss: 0.4684, lr: 0.004599, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 03:53:12 2021-05-06 18:30:42 [INFO] [TRAIN] epoch: 63, iter: 23360/40000, loss: 0.3768, lr: 0.004596, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 03:53:09 2021-05-06 18:30:51 [INFO] [TRAIN] epoch: 63, iter: 23370/40000, loss: 0.4138, lr: 0.004594, batch_cost: 0.8389, reader_cost: 0.00015, ips: 1.1921 samples/sec | ETA 03:52:30 2021-05-06 18:30:59 [INFO] [TRAIN] epoch: 63, iter: 23380/40000, loss: 0.2915, lr: 0.004591, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 03:52:48 2021-05-06 18:31:08 [INFO] [TRAIN] epoch: 63, iter: 23390/40000, loss: 0.5106, lr: 0.004589, batch_cost: 0.8407, reader_cost: 0.00016, ips: 1.1894 samples/sec | ETA 03:52:44 2021-05-06 18:31:16 [INFO] [TRAIN] epoch: 63, iter: 23400/40000, loss: 0.3856, lr: 0.004586, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1888 samples/sec | ETA 03:52:44 2021-05-06 18:31:24 [INFO] [TRAIN] epoch: 63, iter: 23410/40000, loss: 0.4459, lr: 0.004584, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1912 samples/sec | ETA 03:52:07 2021-05-06 18:31:33 [INFO] [TRAIN] epoch: 63, iter: 23420/40000, loss: 0.3959, lr: 0.004582, batch_cost: 0.8395, reader_cost: 0.00014, ips: 1.1912 samples/sec | ETA 03:51:59 2021-05-06 18:31:41 [INFO] [TRAIN] epoch: 63, iter: 23430/40000, loss: 0.4337, lr: 0.004579, batch_cost: 0.8379, reader_cost: 0.00011, ips: 1.1935 samples/sec | ETA 03:51:23 2021-05-06 18:31:53 [INFO] [TRAIN] epoch: 64, iter: 23440/40000, loss: 0.5004, lr: 0.004577, batch_cost: 1.1564, reader_cost: 0.23423, ips: 0.8647 samples/sec | ETA 05:19:10 2021-05-06 18:32:01 [INFO] [TRAIN] epoch: 64, iter: 23450/40000, loss: 0.3771, lr: 0.004574, batch_cost: 0.8508, reader_cost: 0.00031, ips: 1.1754 samples/sec | ETA 03:54:40 2021-05-06 18:32:10 [INFO] [TRAIN] epoch: 64, iter: 23460/40000, loss: 0.4208, lr: 0.004572, batch_cost: 0.8410, reader_cost: 0.00016, ips: 1.1891 samples/sec | ETA 03:51:50 2021-05-06 18:32:18 [INFO] [TRAIN] epoch: 64, iter: 23470/40000, loss: 0.2834, lr: 0.004569, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 03:51:23 2021-05-06 18:32:26 [INFO] [TRAIN] epoch: 64, iter: 23480/40000, loss: 0.3182, lr: 0.004567, batch_cost: 0.8422, reader_cost: 0.00015, ips: 1.1874 samples/sec | ETA 03:51:53 2021-05-06 18:32:35 [INFO] [TRAIN] epoch: 64, iter: 23490/40000, loss: 0.4997, lr: 0.004565, batch_cost: 0.8419, reader_cost: 0.00015, ips: 1.1877 samples/sec | ETA 03:51:40 2021-05-06 18:32:43 [INFO] [TRAIN] epoch: 64, iter: 23500/40000, loss: 0.4568, lr: 0.004562, batch_cost: 0.8427, reader_cost: 0.00015, ips: 1.1866 samples/sec | ETA 03:51:44 2021-05-06 18:32:52 [INFO] [TRAIN] epoch: 64, iter: 23510/40000, loss: 0.5262, lr: 0.004560, batch_cost: 0.8411, reader_cost: 0.00014, ips: 1.1889 samples/sec | ETA 03:51:10 2021-05-06 18:33:00 [INFO] [TRAIN] epoch: 64, iter: 23520/40000, loss: 0.3875, lr: 0.004557, batch_cost: 0.8404, reader_cost: 0.00014, ips: 1.1899 samples/sec | ETA 03:50:49 2021-05-06 18:33:08 [INFO] [TRAIN] epoch: 64, iter: 23530/40000, loss: 0.2398, lr: 0.004555, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1882 samples/sec | ETA 03:51:01 2021-05-06 18:33:17 [INFO] [TRAIN] epoch: 64, iter: 23540/40000, loss: 0.2640, lr: 0.004552, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 03:50:22 2021-05-06 18:33:25 [INFO] [TRAIN] epoch: 64, iter: 23550/40000, loss: 0.3620, lr: 0.004550, batch_cost: 0.8391, reader_cost: 0.00014, ips: 1.1917 samples/sec | ETA 03:50:03 2021-05-06 18:33:34 [INFO] [TRAIN] epoch: 64, iter: 23560/40000, loss: 0.4562, lr: 0.004548, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1886 samples/sec | ETA 03:50:31 2021-05-06 18:33:42 [INFO] [TRAIN] epoch: 64, iter: 23570/40000, loss: 0.4666, lr: 0.004545, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 03:50:19 2021-05-06 18:33:51 [INFO] [TRAIN] epoch: 64, iter: 23580/40000, loss: 0.5531, lr: 0.004543, batch_cost: 0.8398, reader_cost: 0.00016, ips: 1.1908 samples/sec | ETA 03:49:49 2021-05-06 18:33:59 [INFO] [TRAIN] epoch: 64, iter: 23590/40000, loss: 0.2326, lr: 0.004540, batch_cost: 0.8421, reader_cost: 0.00014, ips: 1.1874 samples/sec | ETA 03:50:19 2021-05-06 18:34:07 [INFO] [TRAIN] epoch: 64, iter: 23600/40000, loss: 0.3056, lr: 0.004538, batch_cost: 0.8401, reader_cost: 0.00017, ips: 1.1904 samples/sec | ETA 03:49:37 2021-05-06 18:34:16 [INFO] [TRAIN] epoch: 64, iter: 23610/40000, loss: 0.2977, lr: 0.004535, batch_cost: 0.8415, reader_cost: 0.00016, ips: 1.1883 samples/sec | ETA 03:49:52 2021-05-06 18:34:24 [INFO] [TRAIN] epoch: 64, iter: 23620/40000, loss: 1.3504, lr: 0.004533, batch_cost: 0.8406, reader_cost: 0.00016, ips: 1.1897 samples/sec | ETA 03:49:28 2021-05-06 18:34:33 [INFO] [TRAIN] epoch: 64, iter: 23630/40000, loss: 0.5021, lr: 0.004530, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1914 samples/sec | ETA 03:49:00 2021-05-06 18:34:41 [INFO] [TRAIN] epoch: 64, iter: 23640/40000, loss: 0.3325, lr: 0.004528, batch_cost: 0.8419, reader_cost: 0.00016, ips: 1.1877 samples/sec | ETA 03:49:34 2021-05-06 18:34:49 [INFO] [TRAIN] epoch: 64, iter: 23650/40000, loss: 0.3426, lr: 0.004526, batch_cost: 0.8402, reader_cost: 0.00014, ips: 1.1902 samples/sec | ETA 03:48:57 2021-05-06 18:34:58 [INFO] [TRAIN] epoch: 64, iter: 23660/40000, loss: 0.2282, lr: 0.004523, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 03:49:00 2021-05-06 18:35:06 [INFO] [TRAIN] epoch: 64, iter: 23670/40000, loss: 0.2725, lr: 0.004521, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 03:48:38 2021-05-06 18:35:15 [INFO] [TRAIN] epoch: 64, iter: 23680/40000, loss: 0.2768, lr: 0.004518, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 03:48:42 2021-05-06 18:35:23 [INFO] [TRAIN] epoch: 64, iter: 23690/40000, loss: 0.2361, lr: 0.004516, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 03:48:25 2021-05-06 18:35:31 [INFO] [TRAIN] epoch: 64, iter: 23700/40000, loss: 0.3092, lr: 0.004513, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 03:48:28 2021-05-06 18:35:40 [INFO] [TRAIN] epoch: 64, iter: 23710/40000, loss: 0.2745, lr: 0.004511, batch_cost: 0.8392, reader_cost: 0.00016, ips: 1.1916 samples/sec | ETA 03:47:50 2021-05-06 18:35:48 [INFO] [TRAIN] epoch: 64, iter: 23720/40000, loss: 0.3500, lr: 0.004509, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 03:47:59 2021-05-06 18:35:57 [INFO] [TRAIN] epoch: 64, iter: 23730/40000, loss: 0.4146, lr: 0.004506, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 03:47:53 2021-05-06 18:36:05 [INFO] [TRAIN] epoch: 64, iter: 23740/40000, loss: 0.2678, lr: 0.004504, batch_cost: 0.8397, reader_cost: 0.00016, ips: 1.1909 samples/sec | ETA 03:47:33 2021-05-06 18:36:13 [INFO] [TRAIN] epoch: 64, iter: 23750/40000, loss: 0.3093, lr: 0.004501, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1918 samples/sec | ETA 03:47:14 2021-05-06 18:36:22 [INFO] [TRAIN] epoch: 64, iter: 23760/40000, loss: 0.4897, lr: 0.004499, batch_cost: 0.8389, reader_cost: 0.00015, ips: 1.1920 samples/sec | ETA 03:47:04 2021-05-06 18:36:30 [INFO] [TRAIN] epoch: 64, iter: 23770/40000, loss: 0.2761, lr: 0.004496, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 03:47:27 2021-05-06 18:36:39 [INFO] [TRAIN] epoch: 64, iter: 23780/40000, loss: 0.3751, lr: 0.004494, batch_cost: 0.8407, reader_cost: 0.00017, ips: 1.1895 samples/sec | ETA 03:47:15 2021-05-06 18:36:47 [INFO] [TRAIN] epoch: 64, iter: 23790/40000, loss: 0.3647, lr: 0.004491, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1912 samples/sec | ETA 03:46:48 2021-05-06 18:36:55 [INFO] [TRAIN] epoch: 64, iter: 23800/40000, loss: 0.3430, lr: 0.004489, batch_cost: 0.8399, reader_cost: 0.00013, ips: 1.1906 samples/sec | ETA 03:46:46 2021-05-06 18:37:07 [INFO] [TRAIN] epoch: 65, iter: 23810/40000, loss: 0.4120, lr: 0.004487, batch_cost: 1.1502, reader_cost: 0.20715, ips: 0.8694 samples/sec | ETA 05:10:21 2021-05-06 18:37:16 [INFO] [TRAIN] epoch: 65, iter: 23820/40000, loss: 0.4414, lr: 0.004484, batch_cost: 0.8608, reader_cost: 0.00033, ips: 1.1618 samples/sec | ETA 03:52:07 2021-05-06 18:37:24 [INFO] [TRAIN] epoch: 65, iter: 23830/40000, loss: 0.5149, lr: 0.004482, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 03:46:29 2021-05-06 18:37:32 [INFO] [TRAIN] epoch: 65, iter: 23840/40000, loss: 0.3726, lr: 0.004479, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1888 samples/sec | ETA 03:46:33 2021-05-06 18:37:41 [INFO] [TRAIN] epoch: 65, iter: 23850/40000, loss: 0.2777, lr: 0.004477, batch_cost: 0.8408, reader_cost: 0.00016, ips: 1.1893 samples/sec | ETA 03:46:19 2021-05-06 18:37:49 [INFO] [TRAIN] epoch: 65, iter: 23860/40000, loss: 0.3931, lr: 0.004474, batch_cost: 0.8426, reader_cost: 0.00015, ips: 1.1868 samples/sec | ETA 03:46:39 2021-05-06 18:37:58 [INFO] [TRAIN] epoch: 65, iter: 23870/40000, loss: 0.4592, lr: 0.004472, batch_cost: 0.8399, reader_cost: 0.00016, ips: 1.1906 samples/sec | ETA 03:45:48 2021-05-06 18:38:06 [INFO] [TRAIN] epoch: 65, iter: 23880/40000, loss: 0.5222, lr: 0.004470, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1917 samples/sec | ETA 03:45:26 2021-05-06 18:38:14 [INFO] [TRAIN] epoch: 65, iter: 23890/40000, loss: 0.5398, lr: 0.004467, batch_cost: 0.8397, reader_cost: 0.00014, ips: 1.1909 samples/sec | ETA 03:45:27 2021-05-06 18:38:23 [INFO] [TRAIN] epoch: 65, iter: 23900/40000, loss: 0.3951, lr: 0.004465, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 03:45:25 2021-05-06 18:38:31 [INFO] [TRAIN] epoch: 65, iter: 23910/40000, loss: 0.2145, lr: 0.004462, batch_cost: 0.8405, reader_cost: 0.00014, ips: 1.1898 samples/sec | ETA 03:45:23 2021-05-06 18:38:40 [INFO] [TRAIN] epoch: 65, iter: 23920/40000, loss: 0.2244, lr: 0.004460, batch_cost: 0.8393, reader_cost: 0.00014, ips: 1.1915 samples/sec | ETA 03:44:55 2021-05-06 18:38:48 [INFO] [TRAIN] epoch: 65, iter: 23930/40000, loss: 0.4967, lr: 0.004457, batch_cost: 0.8414, reader_cost: 0.00017, ips: 1.1886 samples/sec | ETA 03:45:20 2021-05-06 18:38:56 [INFO] [TRAIN] epoch: 65, iter: 23940/40000, loss: 0.4723, lr: 0.004455, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 03:44:39 2021-05-06 18:39:05 [INFO] [TRAIN] epoch: 65, iter: 23950/40000, loss: 0.5134, lr: 0.004452, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1912 samples/sec | ETA 03:44:34 2021-05-06 18:39:13 [INFO] [TRAIN] epoch: 65, iter: 23960/40000, loss: 0.4689, lr: 0.004450, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 03:44:49 2021-05-06 18:39:22 [INFO] [TRAIN] epoch: 65, iter: 23970/40000, loss: 0.5257, lr: 0.004448, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 03:44:43 2021-05-06 18:39:30 [INFO] [TRAIN] epoch: 65, iter: 23980/40000, loss: 0.2958, lr: 0.004445, batch_cost: 0.8383, reader_cost: 0.00016, ips: 1.1929 samples/sec | ETA 03:43:48 2021-05-06 18:39:38 [INFO] [TRAIN] epoch: 65, iter: 23990/40000, loss: 0.3035, lr: 0.004443, batch_cost: 0.8395, reader_cost: 0.00016, ips: 1.1911 samples/sec | ETA 03:44:01 2021-05-06 18:39:47 [INFO] [TRAIN] epoch: 65, iter: 24000/40000, loss: 0.5366, lr: 0.004440, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1904 samples/sec | ETA 03:44:00 2021-05-06 18:39:47 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 18:43:24 [INFO] [EVAL] #Images: 500 mIoU: 0.7462 Acc: 0.9564 Kappa: 0.9434 2021-05-06 18:43:24 [INFO] [EVAL] Class IoU: [0.9818 0.8491 0.9166 0.6532 0.5889 0.5034 0.6016 0.7233 0.9176 0.6714 0.9416 0.7779 0.5489 0.9401 0.7882 0.8312 0.6302 0.6087 0.7049] 2021-05-06 18:43:24 [INFO] [EVAL] Class Acc: [0.9924 0.9246 0.9444 0.8457 0.7382 0.7975 0.7658 0.8743 0.951 0.8243 0.9733 0.8613 0.6407 0.9671 0.9449 0.9125 0.9495 0.7863 0.8363] 2021-05-06 18:43:54 [INFO] [EVAL] The model with the best validation mIoU (0.7598) was saved at iter 22000. 2021-05-06 18:44:02 [INFO] [TRAIN] epoch: 65, iter: 24010/40000, loss: 0.5527, lr: 0.004438, batch_cost: 0.8358, reader_cost: 0.00022, ips: 1.1965 samples/sec | ETA 03:42:43 2021-05-06 18:44:10 [INFO] [TRAIN] epoch: 65, iter: 24020/40000, loss: 0.3577, lr: 0.004435, batch_cost: 0.8368, reader_cost: 0.00016, ips: 1.1950 samples/sec | ETA 03:42:51 2021-05-06 18:44:19 [INFO] [TRAIN] epoch: 65, iter: 24030/40000, loss: 0.3751, lr: 0.004433, batch_cost: 0.8385, reader_cost: 0.00016, ips: 1.1926 samples/sec | ETA 03:43:10 2021-05-06 18:44:27 [INFO] [TRAIN] epoch: 65, iter: 24040/40000, loss: 0.2956, lr: 0.004430, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 03:43:33 2021-05-06 18:44:35 [INFO] [TRAIN] epoch: 65, iter: 24050/40000, loss: 0.3665, lr: 0.004428, batch_cost: 0.8390, reader_cost: 0.00015, ips: 1.1919 samples/sec | ETA 03:43:01 2021-05-06 18:44:44 [INFO] [TRAIN] epoch: 65, iter: 24060/40000, loss: 0.2645, lr: 0.004426, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 03:43:06 2021-05-06 18:44:52 [INFO] [TRAIN] epoch: 65, iter: 24070/40000, loss: 0.3869, lr: 0.004423, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 03:43:06 2021-05-06 18:45:01 [INFO] [TRAIN] epoch: 65, iter: 24080/40000, loss: 0.3144, lr: 0.004421, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 03:42:55 2021-05-06 18:45:09 [INFO] [TRAIN] epoch: 65, iter: 24090/40000, loss: 0.4174, lr: 0.004418, batch_cost: 0.8384, reader_cost: 0.00014, ips: 1.1927 samples/sec | ETA 03:42:19 2021-05-06 18:45:17 [INFO] [TRAIN] epoch: 65, iter: 24100/40000, loss: 0.5218, lr: 0.004416, batch_cost: 0.8412, reader_cost: 0.00014, ips: 1.1887 samples/sec | ETA 03:42:55 2021-05-06 18:45:26 [INFO] [TRAIN] epoch: 65, iter: 24110/40000, loss: 0.6177, lr: 0.004413, batch_cost: 0.8407, reader_cost: 0.00014, ips: 1.1894 samples/sec | ETA 03:42:39 2021-05-06 18:45:34 [INFO] [TRAIN] epoch: 65, iter: 24120/40000, loss: 0.3312, lr: 0.004411, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 03:42:30 2021-05-06 18:45:43 [INFO] [TRAIN] epoch: 65, iter: 24130/40000, loss: 0.5268, lr: 0.004408, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 03:42:27 2021-05-06 18:45:51 [INFO] [TRAIN] epoch: 65, iter: 24140/40000, loss: 0.2875, lr: 0.004406, batch_cost: 0.8427, reader_cost: 0.00014, ips: 1.1867 samples/sec | ETA 03:42:44 2021-05-06 18:46:00 [INFO] [TRAIN] epoch: 65, iter: 24150/40000, loss: 0.3406, lr: 0.004404, batch_cost: 0.8412, reader_cost: 0.00014, ips: 1.1888 samples/sec | ETA 03:42:13 2021-05-06 18:46:08 [INFO] [TRAIN] epoch: 65, iter: 24160/40000, loss: 0.4576, lr: 0.004401, batch_cost: 0.8424, reader_cost: 0.00015, ips: 1.1871 samples/sec | ETA 03:42:23 2021-05-06 18:46:16 [INFO] [TRAIN] epoch: 65, iter: 24170/40000, loss: 0.3865, lr: 0.004399, batch_cost: 0.8397, reader_cost: 0.00014, ips: 1.1908 samples/sec | ETA 03:41:33 2021-05-06 18:46:25 [INFO] [TRAIN] epoch: 65, iter: 24180/40000, loss: 0.2769, lr: 0.004396, batch_cost: 0.8385, reader_cost: 0.00009, ips: 1.1926 samples/sec | ETA 03:41:04 2021-05-06 18:46:37 [INFO] [TRAIN] epoch: 66, iter: 24190/40000, loss: 0.4367, lr: 0.004394, batch_cost: 1.1870, reader_cost: 0.23364, ips: 0.8425 samples/sec | ETA 05:12:46 2021-05-06 18:46:45 [INFO] [TRAIN] epoch: 66, iter: 24200/40000, loss: 0.3765, lr: 0.004391, batch_cost: 0.8415, reader_cost: 0.00032, ips: 1.1883 samples/sec | ETA 03:41:36 2021-05-06 18:46:53 [INFO] [TRAIN] epoch: 66, iter: 24210/40000, loss: 0.3736, lr: 0.004389, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1881 samples/sec | ETA 03:41:30 2021-05-06 18:47:02 [INFO] [TRAIN] epoch: 66, iter: 24220/40000, loss: 0.2192, lr: 0.004386, batch_cost: 0.8410, reader_cost: 0.00014, ips: 1.1891 samples/sec | ETA 03:41:10 2021-05-06 18:47:10 [INFO] [TRAIN] epoch: 66, iter: 24230/40000, loss: 0.4589, lr: 0.004384, batch_cost: 0.8431, reader_cost: 0.00015, ips: 1.1860 samples/sec | ETA 03:41:36 2021-05-06 18:47:19 [INFO] [TRAIN] epoch: 66, iter: 24240/40000, loss: 0.7215, lr: 0.004382, batch_cost: 0.8407, reader_cost: 0.00016, ips: 1.1894 samples/sec | ETA 03:40:50 2021-05-06 18:47:27 [INFO] [TRAIN] epoch: 66, iter: 24250/40000, loss: 0.4067, lr: 0.004379, batch_cost: 0.8413, reader_cost: 0.00017, ips: 1.1886 samples/sec | ETA 03:40:51 2021-05-06 18:47:35 [INFO] [TRAIN] epoch: 66, iter: 24260/40000, loss: 0.5045, lr: 0.004377, batch_cost: 0.8383, reader_cost: 0.00015, ips: 1.1929 samples/sec | ETA 03:39:55 2021-05-06 18:47:44 [INFO] [TRAIN] epoch: 66, iter: 24270/40000, loss: 0.3760, lr: 0.004374, batch_cost: 0.8427, reader_cost: 0.00017, ips: 1.1867 samples/sec | ETA 03:40:55 2021-05-06 18:47:52 [INFO] [TRAIN] epoch: 66, iter: 24280/40000, loss: 0.1762, lr: 0.004372, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1913 samples/sec | ETA 03:39:55 2021-05-06 18:48:01 [INFO] [TRAIN] epoch: 66, iter: 24290/40000, loss: 0.3253, lr: 0.004369, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 03:40:03 2021-05-06 18:48:09 [INFO] [TRAIN] epoch: 66, iter: 24300/40000, loss: 0.4438, lr: 0.004367, batch_cost: 0.8424, reader_cost: 0.00015, ips: 1.1871 samples/sec | ETA 03:40:25 2021-05-06 18:48:18 [INFO] [TRAIN] epoch: 66, iter: 24310/40000, loss: 0.3802, lr: 0.004364, batch_cost: 0.8431, reader_cost: 0.00015, ips: 1.1861 samples/sec | ETA 03:40:28 2021-05-06 18:48:26 [INFO] [TRAIN] epoch: 66, iter: 24320/40000, loss: 0.4741, lr: 0.004362, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1883 samples/sec | ETA 03:39:55 2021-05-06 18:48:34 [INFO] [TRAIN] epoch: 66, iter: 24330/40000, loss: 0.4331, lr: 0.004360, batch_cost: 0.8422, reader_cost: 0.00015, ips: 1.1873 samples/sec | ETA 03:39:57 2021-05-06 18:48:43 [INFO] [TRAIN] epoch: 66, iter: 24340/40000, loss: 0.4448, lr: 0.004357, batch_cost: 0.8412, reader_cost: 0.00016, ips: 1.1887 samples/sec | ETA 03:39:33 2021-05-06 18:48:51 [INFO] [TRAIN] epoch: 66, iter: 24350/40000, loss: 0.4455, lr: 0.004355, batch_cost: 0.8409, reader_cost: 0.00016, ips: 1.1892 samples/sec | ETA 03:39:19 2021-05-06 18:49:00 [INFO] [TRAIN] epoch: 66, iter: 24360/40000, loss: 0.3980, lr: 0.004352, batch_cost: 0.8405, reader_cost: 0.00016, ips: 1.1897 samples/sec | ETA 03:39:06 2021-05-06 18:49:08 [INFO] [TRAIN] epoch: 66, iter: 24370/40000, loss: 1.0654, lr: 0.004350, batch_cost: 0.8432, reader_cost: 0.00016, ips: 1.1860 samples/sec | ETA 03:39:38 2021-05-06 18:49:16 [INFO] [TRAIN] epoch: 66, iter: 24380/40000, loss: 0.6581, lr: 0.004347, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 03:38:46 2021-05-06 18:49:25 [INFO] [TRAIN] epoch: 66, iter: 24390/40000, loss: 0.5276, lr: 0.004345, batch_cost: 0.8423, reader_cost: 0.00015, ips: 1.1872 samples/sec | ETA 03:39:08 2021-05-06 18:49:33 [INFO] [TRAIN] epoch: 66, iter: 24400/40000, loss: 0.3479, lr: 0.004342, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1885 samples/sec | ETA 03:38:46 2021-05-06 18:49:42 [INFO] [TRAIN] epoch: 66, iter: 24410/40000, loss: 0.2556, lr: 0.004340, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 03:38:24 2021-05-06 18:49:50 [INFO] [TRAIN] epoch: 66, iter: 24420/40000, loss: 0.3287, lr: 0.004338, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1888 samples/sec | ETA 03:38:26 2021-05-06 18:49:59 [INFO] [TRAIN] epoch: 66, iter: 24430/40000, loss: 0.2038, lr: 0.004335, batch_cost: 0.8426, reader_cost: 0.00015, ips: 1.1869 samples/sec | ETA 03:38:38 2021-05-06 18:50:07 [INFO] [TRAIN] epoch: 66, iter: 24440/40000, loss: 0.3159, lr: 0.004333, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 03:38:07 2021-05-06 18:50:15 [INFO] [TRAIN] epoch: 66, iter: 24450/40000, loss: 0.3785, lr: 0.004330, batch_cost: 0.8411, reader_cost: 0.00014, ips: 1.1889 samples/sec | ETA 03:37:59 2021-05-06 18:50:24 [INFO] [TRAIN] epoch: 66, iter: 24460/40000, loss: 0.2923, lr: 0.004328, batch_cost: 0.8404, reader_cost: 0.00014, ips: 1.1899 samples/sec | ETA 03:37:39 2021-05-06 18:50:32 [INFO] [TRAIN] epoch: 66, iter: 24470/40000, loss: 0.4148, lr: 0.004325, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1879 samples/sec | ETA 03:37:53 2021-05-06 18:50:41 [INFO] [TRAIN] epoch: 66, iter: 24480/40000, loss: 0.3994, lr: 0.004323, batch_cost: 0.8412, reader_cost: 0.00016, ips: 1.1887 samples/sec | ETA 03:37:35 2021-05-06 18:50:49 [INFO] [TRAIN] epoch: 66, iter: 24490/40000, loss: 0.2714, lr: 0.004320, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 03:37:14 2021-05-06 18:50:57 [INFO] [TRAIN] epoch: 66, iter: 24500/40000, loss: 0.3621, lr: 0.004318, batch_cost: 0.8390, reader_cost: 0.00015, ips: 1.1919 samples/sec | ETA 03:36:44 2021-05-06 18:51:06 [INFO] [TRAIN] epoch: 66, iter: 24510/40000, loss: 0.4536, lr: 0.004316, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 03:36:50 2021-05-06 18:51:14 [INFO] [TRAIN] epoch: 66, iter: 24520/40000, loss: 0.2929, lr: 0.004313, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1885 samples/sec | ETA 03:37:04 2021-05-06 18:51:23 [INFO] [TRAIN] epoch: 66, iter: 24530/40000, loss: 0.4763, lr: 0.004311, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1883 samples/sec | ETA 03:36:58 2021-05-06 18:51:31 [INFO] [TRAIN] epoch: 66, iter: 24540/40000, loss: 0.2829, lr: 0.004308, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1884 samples/sec | ETA 03:36:49 2021-05-06 18:51:39 [INFO] [TRAIN] epoch: 66, iter: 24550/40000, loss: 0.3045, lr: 0.004306, batch_cost: 0.8374, reader_cost: 0.00027, ips: 1.1942 samples/sec | ETA 03:35:37 2021-05-06 18:51:51 [INFO] [TRAIN] epoch: 67, iter: 24560/40000, loss: 0.5093, lr: 0.004303, batch_cost: 1.1508, reader_cost: 0.23238, ips: 0.8690 samples/sec | ETA 04:56:08 2021-05-06 18:51:59 [INFO] [TRAIN] epoch: 67, iter: 24570/40000, loss: 0.3119, lr: 0.004301, batch_cost: 0.8398, reader_cost: 0.00030, ips: 1.1907 samples/sec | ETA 03:35:58 2021-05-06 18:52:08 [INFO] [TRAIN] epoch: 67, iter: 24580/40000, loss: 0.3501, lr: 0.004298, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 03:36:00 2021-05-06 18:52:16 [INFO] [TRAIN] epoch: 67, iter: 24590/40000, loss: 0.3197, lr: 0.004296, batch_cost: 0.8403, reader_cost: 0.00014, ips: 1.1900 samples/sec | ETA 03:35:49 2021-05-06 18:52:25 [INFO] [TRAIN] epoch: 67, iter: 24600/40000, loss: 0.4155, lr: 0.004293, batch_cost: 0.8412, reader_cost: 0.00014, ips: 1.1888 samples/sec | ETA 03:35:54 2021-05-06 18:52:33 [INFO] [TRAIN] epoch: 67, iter: 24610/40000, loss: 0.3813, lr: 0.004291, batch_cost: 0.8424, reader_cost: 0.00015, ips: 1.1870 samples/sec | ETA 03:36:05 2021-05-06 18:52:41 [INFO] [TRAIN] epoch: 67, iter: 24620/40000, loss: 0.4040, lr: 0.004289, batch_cost: 0.8421, reader_cost: 0.00016, ips: 1.1875 samples/sec | ETA 03:35:51 2021-05-06 18:52:50 [INFO] [TRAIN] epoch: 67, iter: 24630/40000, loss: 0.6008, lr: 0.004286, batch_cost: 0.8380, reader_cost: 0.00015, ips: 1.1933 samples/sec | ETA 03:34:40 2021-05-06 18:52:58 [INFO] [TRAIN] epoch: 67, iter: 24640/40000, loss: 0.5152, lr: 0.004284, batch_cost: 0.8418, reader_cost: 0.00014, ips: 1.1879 samples/sec | ETA 03:35:30 2021-05-06 18:53:07 [INFO] [TRAIN] epoch: 67, iter: 24650/40000, loss: 0.1869, lr: 0.004281, batch_cost: 0.8410, reader_cost: 0.00016, ips: 1.1890 samples/sec | ETA 03:35:09 2021-05-06 18:53:15 [INFO] [TRAIN] epoch: 67, iter: 24660/40000, loss: 0.3313, lr: 0.004279, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 03:34:47 2021-05-06 18:53:23 [INFO] [TRAIN] epoch: 67, iter: 24670/40000, loss: 0.3323, lr: 0.004276, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 03:34:47 2021-05-06 18:53:32 [INFO] [TRAIN] epoch: 67, iter: 24680/40000, loss: 0.3527, lr: 0.004274, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 03:34:27 2021-05-06 18:53:40 [INFO] [TRAIN] epoch: 67, iter: 24690/40000, loss: 0.5763, lr: 0.004271, batch_cost: 0.8389, reader_cost: 0.00014, ips: 1.1920 samples/sec | ETA 03:34:03 2021-05-06 18:53:49 [INFO] [TRAIN] epoch: 67, iter: 24700/40000, loss: 0.4282, lr: 0.004269, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 03:34:28 2021-05-06 18:53:57 [INFO] [TRAIN] epoch: 67, iter: 24710/40000, loss: 0.2971, lr: 0.004267, batch_cost: 0.8416, reader_cost: 0.00016, ips: 1.1882 samples/sec | ETA 03:34:27 2021-05-06 18:54:05 [INFO] [TRAIN] epoch: 67, iter: 24720/40000, loss: 0.3147, lr: 0.004264, batch_cost: 0.8418, reader_cost: 0.00016, ips: 1.1879 samples/sec | ETA 03:34:23 2021-05-06 18:54:14 [INFO] [TRAIN] epoch: 67, iter: 24730/40000, loss: 0.3482, lr: 0.004262, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 03:33:45 2021-05-06 18:54:22 [INFO] [TRAIN] epoch: 67, iter: 24740/40000, loss: 0.4970, lr: 0.004259, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1888 samples/sec | ETA 03:33:56 2021-05-06 18:54:31 [INFO] [TRAIN] epoch: 67, iter: 24750/40000, loss: 0.5289, lr: 0.004257, batch_cost: 0.8399, reader_cost: 0.00016, ips: 1.1907 samples/sec | ETA 03:33:27 2021-05-06 18:54:39 [INFO] [TRAIN] epoch: 67, iter: 24760/40000, loss: 0.3220, lr: 0.004254, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 03:33:36 2021-05-06 18:54:48 [INFO] [TRAIN] epoch: 67, iter: 24770/40000, loss: 0.3676, lr: 0.004252, batch_cost: 0.8411, reader_cost: 0.00014, ips: 1.1889 samples/sec | ETA 03:33:29 2021-05-06 18:54:56 [INFO] [TRAIN] epoch: 67, iter: 24780/40000, loss: 0.2232, lr: 0.004249, batch_cost: 0.8422, reader_cost: 0.00015, ips: 1.1873 samples/sec | ETA 03:33:38 2021-05-06 18:55:04 [INFO] [TRAIN] epoch: 67, iter: 24790/40000, loss: 0.4074, lr: 0.004247, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 03:33:06 2021-05-06 18:55:13 [INFO] [TRAIN] epoch: 67, iter: 24800/40000, loss: 0.2765, lr: 0.004244, batch_cost: 0.8421, reader_cost: 0.00016, ips: 1.1875 samples/sec | ETA 03:33:20 2021-05-06 18:55:21 [INFO] [TRAIN] epoch: 67, iter: 24810/40000, loss: 0.4306, lr: 0.004242, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 03:32:45 2021-05-06 18:55:30 [INFO] [TRAIN] epoch: 67, iter: 24820/40000, loss: 0.2845, lr: 0.004240, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 03:32:39 2021-05-06 18:55:38 [INFO] [TRAIN] epoch: 67, iter: 24830/40000, loss: 0.2241, lr: 0.004237, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 03:32:19 2021-05-06 18:55:46 [INFO] [TRAIN] epoch: 67, iter: 24840/40000, loss: 0.2935, lr: 0.004235, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 03:32:13 2021-05-06 18:55:55 [INFO] [TRAIN] epoch: 67, iter: 24850/40000, loss: 0.3654, lr: 0.004232, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 03:32:18 2021-05-06 18:56:03 [INFO] [TRAIN] epoch: 67, iter: 24860/40000, loss: 0.3767, lr: 0.004230, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 03:32:06 2021-05-06 18:56:12 [INFO] [TRAIN] epoch: 67, iter: 24870/40000, loss: 0.2467, lr: 0.004227, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 03:32:06 2021-05-06 18:56:20 [INFO] [TRAIN] epoch: 67, iter: 24880/40000, loss: 0.4633, lr: 0.004225, batch_cost: 0.8390, reader_cost: 0.00015, ips: 1.1919 samples/sec | ETA 03:31:25 2021-05-06 18:56:28 [INFO] [TRAIN] epoch: 67, iter: 24890/40000, loss: 0.3207, lr: 0.004222, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 03:31:41 2021-05-06 18:56:37 [INFO] [TRAIN] epoch: 67, iter: 24900/40000, loss: 0.4017, lr: 0.004220, batch_cost: 0.8419, reader_cost: 0.00015, ips: 1.1878 samples/sec | ETA 03:31:52 2021-05-06 18:56:45 [INFO] [TRAIN] epoch: 67, iter: 24910/40000, loss: 0.2448, lr: 0.004217, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 03:31:13 2021-05-06 18:56:54 [INFO] [TRAIN] epoch: 67, iter: 24920/40000, loss: 0.3504, lr: 0.004215, batch_cost: 0.8384, reader_cost: 0.00011, ips: 1.1928 samples/sec | ETA 03:30:42 2021-05-06 18:57:05 [INFO] [TRAIN] epoch: 68, iter: 24930/40000, loss: 0.4129, lr: 0.004213, batch_cost: 1.1864, reader_cost: 0.20949, ips: 0.8429 samples/sec | ETA 04:57:58 2021-05-06 18:57:14 [INFO] [TRAIN] epoch: 68, iter: 24940/40000, loss: 0.3419, lr: 0.004210, batch_cost: 0.8412, reader_cost: 0.00031, ips: 1.1888 samples/sec | ETA 03:31:07 2021-05-06 18:57:22 [INFO] [TRAIN] epoch: 68, iter: 24950/40000, loss: 0.3786, lr: 0.004208, batch_cost: 0.8422, reader_cost: 0.00015, ips: 1.1874 samples/sec | ETA 03:31:14 2021-05-06 18:57:31 [INFO] [TRAIN] epoch: 68, iter: 24960/40000, loss: 0.3939, lr: 0.004205, batch_cost: 0.8419, reader_cost: 0.00017, ips: 1.1878 samples/sec | ETA 03:31:01 2021-05-06 18:57:39 [INFO] [TRAIN] epoch: 68, iter: 24970/40000, loss: 0.2972, lr: 0.004203, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1886 samples/sec | ETA 03:30:44 2021-05-06 18:57:48 [INFO] [TRAIN] epoch: 68, iter: 24980/40000, loss: 0.5548, lr: 0.004200, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1884 samples/sec | ETA 03:30:39 2021-05-06 18:57:56 [INFO] [TRAIN] epoch: 68, iter: 24990/40000, loss: 0.4483, lr: 0.004198, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 03:30:18 2021-05-06 18:58:04 [INFO] [TRAIN] epoch: 68, iter: 25000/40000, loss: 0.6234, lr: 0.004195, batch_cost: 0.8414, reader_cost: 0.00016, ips: 1.1885 samples/sec | ETA 03:30:20 2021-05-06 18:58:04 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 19:01:42 [INFO] [EVAL] #Images: 500 mIoU: 0.7412 Acc: 0.9547 Kappa: 0.9413 2021-05-06 19:01:42 [INFO] [EVAL] Class IoU: [0.9803 0.8388 0.9157 0.6541 0.5621 0.5139 0.5981 0.7186 0.9154 0.6204 0.9391 0.766 0.5482 0.9377 0.7654 0.8636 0.6769 0.6059 0.6628] 2021-05-06 19:01:42 [INFO] [EVAL] Class Acc: [0.9918 0.8905 0.9442 0.837 0.796 0.7638 0.7725 0.8693 0.9596 0.8515 0.9584 0.8607 0.7707 0.9722 0.8188 0.9131 0.8794 0.7581 0.7316] 2021-05-06 19:02:12 [INFO] [EVAL] The model with the best validation mIoU (0.7598) was saved at iter 22000. 2021-05-06 19:02:20 [INFO] [TRAIN] epoch: 68, iter: 25010/40000, loss: 0.4037, lr: 0.004193, batch_cost: 0.8355, reader_cost: 0.00035, ips: 1.1968 samples/sec | ETA 03:28:44 2021-05-06 19:02:28 [INFO] [TRAIN] epoch: 68, iter: 25020/40000, loss: 0.2164, lr: 0.004190, batch_cost: 0.8380, reader_cost: 0.00015, ips: 1.1933 samples/sec | ETA 03:29:13 2021-05-06 19:02:37 [INFO] [TRAIN] epoch: 68, iter: 25030/40000, loss: 0.2708, lr: 0.004188, batch_cost: 0.8406, reader_cost: 0.00016, ips: 1.1896 samples/sec | ETA 03:29:43 2021-05-06 19:02:45 [INFO] [TRAIN] epoch: 68, iter: 25040/40000, loss: 0.3120, lr: 0.004186, batch_cost: 0.8424, reader_cost: 0.00016, ips: 1.1871 samples/sec | ETA 03:30:02 2021-05-06 19:02:54 [INFO] [TRAIN] epoch: 68, iter: 25050/40000, loss: 0.4592, lr: 0.004183, batch_cost: 0.8416, reader_cost: 0.00016, ips: 1.1882 samples/sec | ETA 03:29:42 2021-05-06 19:03:02 [INFO] [TRAIN] epoch: 68, iter: 25060/40000, loss: 0.5792, lr: 0.004181, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 03:29:20 2021-05-06 19:03:10 [INFO] [TRAIN] epoch: 68, iter: 25070/40000, loss: 0.3959, lr: 0.004178, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1884 samples/sec | ETA 03:29:23 2021-05-06 19:03:19 [INFO] [TRAIN] epoch: 68, iter: 25080/40000, loss: 0.2687, lr: 0.004176, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 03:28:53 2021-05-06 19:03:27 [INFO] [TRAIN] epoch: 68, iter: 25090/40000, loss: 0.3905, lr: 0.004173, batch_cost: 0.8426, reader_cost: 0.00015, ips: 1.1868 samples/sec | ETA 03:29:22 2021-05-06 19:03:36 [INFO] [TRAIN] epoch: 68, iter: 25100/40000, loss: 0.3705, lr: 0.004171, batch_cost: 0.8419, reader_cost: 0.00016, ips: 1.1878 samples/sec | ETA 03:29:04 2021-05-06 19:03:44 [INFO] [TRAIN] epoch: 68, iter: 25110/40000, loss: 0.9822, lr: 0.004168, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 03:28:34 2021-05-06 19:03:52 [INFO] [TRAIN] epoch: 68, iter: 25120/40000, loss: 0.4391, lr: 0.004166, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 03:28:21 2021-05-06 19:04:01 [INFO] [TRAIN] epoch: 68, iter: 25130/40000, loss: 0.4526, lr: 0.004163, batch_cost: 0.8408, reader_cost: 0.00016, ips: 1.1893 samples/sec | ETA 03:28:23 2021-05-06 19:04:09 [INFO] [TRAIN] epoch: 68, iter: 25140/40000, loss: 0.4160, lr: 0.004161, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 03:28:13 2021-05-06 19:04:18 [INFO] [TRAIN] epoch: 68, iter: 25150/40000, loss: 0.3435, lr: 0.004158, batch_cost: 0.8406, reader_cost: 0.00016, ips: 1.1897 samples/sec | ETA 03:28:02 2021-05-06 19:04:26 [INFO] [TRAIN] epoch: 68, iter: 25160/40000, loss: 0.3707, lr: 0.004156, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 03:27:44 2021-05-06 19:04:34 [INFO] [TRAIN] epoch: 68, iter: 25170/40000, loss: 0.2193, lr: 0.004154, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 03:27:33 2021-05-06 19:04:43 [INFO] [TRAIN] epoch: 68, iter: 25180/40000, loss: 0.2751, lr: 0.004151, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 03:27:38 2021-05-06 19:04:51 [INFO] [TRAIN] epoch: 68, iter: 25190/40000, loss: 0.3687, lr: 0.004149, batch_cost: 0.8384, reader_cost: 0.00015, ips: 1.1928 samples/sec | ETA 03:26:56 2021-05-06 19:05:00 [INFO] [TRAIN] epoch: 68, iter: 25200/40000, loss: 0.1679, lr: 0.004146, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1918 samples/sec | ETA 03:26:58 2021-05-06 19:05:08 [INFO] [TRAIN] epoch: 68, iter: 25210/40000, loss: 0.2312, lr: 0.004144, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 03:27:12 2021-05-06 19:05:16 [INFO] [TRAIN] epoch: 68, iter: 25220/40000, loss: 0.5172, lr: 0.004141, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 03:27:10 2021-05-06 19:05:25 [INFO] [TRAIN] epoch: 68, iter: 25230/40000, loss: 0.2770, lr: 0.004139, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 03:26:43 2021-05-06 19:05:33 [INFO] [TRAIN] epoch: 68, iter: 25240/40000, loss: 0.4012, lr: 0.004136, batch_cost: 0.8398, reader_cost: 0.00014, ips: 1.1908 samples/sec | ETA 03:26:35 2021-05-06 19:05:42 [INFO] [TRAIN] epoch: 68, iter: 25250/40000, loss: 0.5958, lr: 0.004134, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 03:26:36 2021-05-06 19:05:50 [INFO] [TRAIN] epoch: 68, iter: 25260/40000, loss: 0.2618, lr: 0.004131, batch_cost: 0.8403, reader_cost: 0.00014, ips: 1.1900 samples/sec | ETA 03:26:26 2021-05-06 19:05:58 [INFO] [TRAIN] epoch: 68, iter: 25270/40000, loss: 0.3098, lr: 0.004129, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 03:26:23 2021-05-06 19:06:07 [INFO] [TRAIN] epoch: 68, iter: 25280/40000, loss: 0.3643, lr: 0.004126, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 03:26:09 2021-05-06 19:06:15 [INFO] [TRAIN] epoch: 68, iter: 25290/40000, loss: 0.4411, lr: 0.004124, batch_cost: 0.8401, reader_cost: 0.00012, ips: 1.1903 samples/sec | ETA 03:25:57 2021-05-06 19:06:27 [INFO] [TRAIN] epoch: 69, iter: 25300/40000, loss: 0.3466, lr: 0.004122, batch_cost: 1.1489, reader_cost: 0.21263, ips: 0.8704 samples/sec | ETA 04:41:29 2021-05-06 19:06:35 [INFO] [TRAIN] epoch: 69, iter: 25310/40000, loss: 0.7195, lr: 0.004119, batch_cost: 0.8503, reader_cost: 0.00032, ips: 1.1760 samples/sec | ETA 03:28:11 2021-05-06 19:06:44 [INFO] [TRAIN] epoch: 69, iter: 25320/40000, loss: 0.5686, lr: 0.004117, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 03:25:36 2021-05-06 19:06:52 [INFO] [TRAIN] epoch: 69, iter: 25330/40000, loss: 0.4169, lr: 0.004114, batch_cost: 0.8397, reader_cost: 0.00016, ips: 1.1910 samples/sec | ETA 03:25:17 2021-05-06 19:07:00 [INFO] [TRAIN] epoch: 69, iter: 25340/40000, loss: 0.3234, lr: 0.004112, batch_cost: 0.8398, reader_cost: 0.00016, ips: 1.1908 samples/sec | ETA 03:25:11 2021-05-06 19:07:09 [INFO] [TRAIN] epoch: 69, iter: 25350/40000, loss: 0.5401, lr: 0.004109, batch_cost: 0.8449, reader_cost: 0.00016, ips: 1.1836 samples/sec | ETA 03:26:17 2021-05-06 19:07:17 [INFO] [TRAIN] epoch: 69, iter: 25360/40000, loss: 0.5215, lr: 0.004107, batch_cost: 0.8382, reader_cost: 0.00015, ips: 1.1930 samples/sec | ETA 03:24:31 2021-05-06 19:07:26 [INFO] [TRAIN] epoch: 69, iter: 25370/40000, loss: 0.5767, lr: 0.004104, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 03:24:58 2021-05-06 19:07:34 [INFO] [TRAIN] epoch: 69, iter: 25380/40000, loss: 0.4251, lr: 0.004102, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1914 samples/sec | ETA 03:24:30 2021-05-06 19:07:42 [INFO] [TRAIN] epoch: 69, iter: 25390/40000, loss: 0.2654, lr: 0.004099, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 03:24:28 2021-05-06 19:07:51 [INFO] [TRAIN] epoch: 69, iter: 25400/40000, loss: 0.2549, lr: 0.004097, batch_cost: 0.8408, reader_cost: 0.00016, ips: 1.1894 samples/sec | ETA 03:24:35 2021-05-06 19:07:59 [INFO] [TRAIN] epoch: 69, iter: 25410/40000, loss: 0.3178, lr: 0.004094, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 03:24:29 2021-05-06 19:08:08 [INFO] [TRAIN] epoch: 69, iter: 25420/40000, loss: 0.3766, lr: 0.004092, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 03:24:17 2021-05-06 19:08:16 [INFO] [TRAIN] epoch: 69, iter: 25430/40000, loss: 0.4542, lr: 0.004090, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1918 samples/sec | ETA 03:23:45 2021-05-06 19:08:24 [INFO] [TRAIN] epoch: 69, iter: 25440/40000, loss: 0.4477, lr: 0.004087, batch_cost: 0.8386, reader_cost: 0.00015, ips: 1.1924 samples/sec | ETA 03:23:30 2021-05-06 19:08:33 [INFO] [TRAIN] epoch: 69, iter: 25450/40000, loss: 0.3522, lr: 0.004085, batch_cost: 0.8398, reader_cost: 0.00014, ips: 1.1908 samples/sec | ETA 03:23:39 2021-05-06 19:08:41 [INFO] [TRAIN] epoch: 69, iter: 25460/40000, loss: 0.3728, lr: 0.004082, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 03:23:32 2021-05-06 19:08:50 [INFO] [TRAIN] epoch: 69, iter: 25470/40000, loss: 0.2941, lr: 0.004080, batch_cost: 0.8388, reader_cost: 0.00015, ips: 1.1922 samples/sec | ETA 03:23:07 2021-05-06 19:08:58 [INFO] [TRAIN] epoch: 69, iter: 25480/40000, loss: 0.5744, lr: 0.004077, batch_cost: 0.8405, reader_cost: 0.00017, ips: 1.1898 samples/sec | ETA 03:23:23 2021-05-06 19:09:06 [INFO] [TRAIN] epoch: 69, iter: 25490/40000, loss: 0.5134, lr: 0.004075, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 03:23:02 2021-05-06 19:09:15 [INFO] [TRAIN] epoch: 69, iter: 25500/40000, loss: 0.3302, lr: 0.004072, batch_cost: 0.8397, reader_cost: 0.00014, ips: 1.1908 samples/sec | ETA 03:22:56 2021-05-06 19:09:23 [INFO] [TRAIN] epoch: 69, iter: 25510/40000, loss: 0.4218, lr: 0.004070, batch_cost: 0.8423, reader_cost: 0.00016, ips: 1.1873 samples/sec | ETA 03:23:24 2021-05-06 19:09:32 [INFO] [TRAIN] epoch: 69, iter: 25520/40000, loss: 0.2145, lr: 0.004067, batch_cost: 0.8418, reader_cost: 0.00016, ips: 1.1880 samples/sec | ETA 03:23:08 2021-05-06 19:09:40 [INFO] [TRAIN] epoch: 69, iter: 25530/40000, loss: 0.1568, lr: 0.004065, batch_cost: 0.8400, reader_cost: 0.00014, ips: 1.1905 samples/sec | ETA 03:22:34 2021-05-06 19:09:49 [INFO] [TRAIN] epoch: 69, iter: 25540/40000, loss: 0.2437, lr: 0.004062, batch_cost: 0.8392, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 03:22:15 2021-05-06 19:09:57 [INFO] [TRAIN] epoch: 69, iter: 25550/40000, loss: 0.2758, lr: 0.004060, batch_cost: 0.8398, reader_cost: 0.00014, ips: 1.1908 samples/sec | ETA 03:22:14 2021-05-06 19:10:05 [INFO] [TRAIN] epoch: 69, iter: 25560/40000, loss: 0.3165, lr: 0.004057, batch_cost: 0.8419, reader_cost: 0.00015, ips: 1.1878 samples/sec | ETA 03:22:36 2021-05-06 19:10:14 [INFO] [TRAIN] epoch: 69, iter: 25570/40000, loss: 0.3117, lr: 0.004055, batch_cost: 0.8388, reader_cost: 0.00014, ips: 1.1922 samples/sec | ETA 03:21:43 2021-05-06 19:10:22 [INFO] [TRAIN] epoch: 69, iter: 25580/40000, loss: 0.5290, lr: 0.004053, batch_cost: 0.8401, reader_cost: 0.00016, ips: 1.1903 samples/sec | ETA 03:21:54 2021-05-06 19:10:31 [INFO] [TRAIN] epoch: 69, iter: 25590/40000, loss: 0.4607, lr: 0.004050, batch_cost: 0.8392, reader_cost: 0.00016, ips: 1.1915 samples/sec | ETA 03:21:33 2021-05-06 19:10:39 [INFO] [TRAIN] epoch: 69, iter: 25600/40000, loss: 0.2066, lr: 0.004048, batch_cost: 0.8389, reader_cost: 0.00014, ips: 1.1920 samples/sec | ETA 03:21:20 2021-05-06 19:10:47 [INFO] [TRAIN] epoch: 69, iter: 25610/40000, loss: 0.3566, lr: 0.004045, batch_cost: 0.8397, reader_cost: 0.00014, ips: 1.1909 samples/sec | ETA 03:21:23 2021-05-06 19:10:56 [INFO] [TRAIN] epoch: 69, iter: 25620/40000, loss: 0.4465, lr: 0.004043, batch_cost: 0.8393, reader_cost: 0.00014, ips: 1.1915 samples/sec | ETA 03:21:08 2021-05-06 19:11:04 [INFO] [TRAIN] epoch: 69, iter: 25630/40000, loss: 0.3431, lr: 0.004040, batch_cost: 0.8406, reader_cost: 0.00016, ips: 1.1896 samples/sec | ETA 03:21:19 2021-05-06 19:11:13 [INFO] [TRAIN] epoch: 69, iter: 25640/40000, loss: 0.5658, lr: 0.004038, batch_cost: 0.8411, reader_cost: 0.00016, ips: 1.1890 samples/sec | ETA 03:21:17 2021-05-06 19:11:21 [INFO] [TRAIN] epoch: 69, iter: 25650/40000, loss: 0.3158, lr: 0.004035, batch_cost: 0.8395, reader_cost: 0.00016, ips: 1.1912 samples/sec | ETA 03:20:47 2021-05-06 19:11:29 [INFO] [TRAIN] epoch: 69, iter: 25660/40000, loss: 0.3960, lr: 0.004033, batch_cost: 0.8399, reader_cost: 0.00014, ips: 1.1906 samples/sec | ETA 03:20:44 2021-05-06 19:11:41 [INFO] [TRAIN] epoch: 70, iter: 25670/40000, loss: 0.3763, lr: 0.004030, batch_cost: 1.1658, reader_cost: 0.23739, ips: 0.8578 samples/sec | ETA 04:38:25 2021-05-06 19:11:50 [INFO] [TRAIN] epoch: 70, iter: 25680/40000, loss: 0.4617, lr: 0.004028, batch_cost: 0.8601, reader_cost: 0.00032, ips: 1.1626 samples/sec | ETA 03:25:17 2021-05-06 19:11:58 [INFO] [TRAIN] epoch: 70, iter: 25690/40000, loss: 0.3700, lr: 0.004025, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 03:20:25 2021-05-06 19:12:06 [INFO] [TRAIN] epoch: 70, iter: 25700/40000, loss: 0.4432, lr: 0.004023, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1885 samples/sec | ETA 03:20:32 2021-05-06 19:12:15 [INFO] [TRAIN] epoch: 70, iter: 25710/40000, loss: 0.1980, lr: 0.004020, batch_cost: 0.8408, reader_cost: 0.00014, ips: 1.1893 samples/sec | ETA 03:20:15 2021-05-06 19:12:23 [INFO] [TRAIN] epoch: 70, iter: 25720/40000, loss: 0.4269, lr: 0.004018, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 03:20:05 2021-05-06 19:12:32 [INFO] [TRAIN] epoch: 70, iter: 25730/40000, loss: 0.5016, lr: 0.004016, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 03:19:51 2021-05-06 19:12:40 [INFO] [TRAIN] epoch: 70, iter: 25740/40000, loss: 0.4014, lr: 0.004013, batch_cost: 0.8381, reader_cost: 0.00015, ips: 1.1932 samples/sec | ETA 03:19:11 2021-05-06 19:12:48 [INFO] [TRAIN] epoch: 70, iter: 25750/40000, loss: 0.3843, lr: 0.004011, batch_cost: 0.8387, reader_cost: 0.00015, ips: 1.1923 samples/sec | ETA 03:19:11 2021-05-06 19:12:57 [INFO] [TRAIN] epoch: 70, iter: 25760/40000, loss: 0.3251, lr: 0.004008, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1914 samples/sec | ETA 03:19:12 2021-05-06 19:13:05 [INFO] [TRAIN] epoch: 70, iter: 25770/40000, loss: 0.1756, lr: 0.004006, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 03:19:16 2021-05-06 19:13:14 [INFO] [TRAIN] epoch: 70, iter: 25780/40000, loss: 0.3013, lr: 0.004003, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1881 samples/sec | ETA 03:19:29 2021-05-06 19:13:22 [INFO] [TRAIN] epoch: 70, iter: 25790/40000, loss: 0.3058, lr: 0.004001, batch_cost: 0.8389, reader_cost: 0.00014, ips: 1.1921 samples/sec | ETA 03:18:40 2021-05-06 19:13:30 [INFO] [TRAIN] epoch: 70, iter: 25800/40000, loss: 0.6183, lr: 0.003998, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1888 samples/sec | ETA 03:19:04 2021-05-06 19:13:39 [INFO] [TRAIN] epoch: 70, iter: 25810/40000, loss: 0.4131, lr: 0.003996, batch_cost: 0.8386, reader_cost: 0.00014, ips: 1.1925 samples/sec | ETA 03:18:19 2021-05-06 19:13:47 [INFO] [TRAIN] epoch: 70, iter: 25820/40000, loss: 0.3510, lr: 0.003993, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 03:18:29 2021-05-06 19:13:56 [INFO] [TRAIN] epoch: 70, iter: 25830/40000, loss: 0.2988, lr: 0.003991, batch_cost: 0.8396, reader_cost: 0.00014, ips: 1.1911 samples/sec | ETA 03:18:16 2021-05-06 19:14:04 [INFO] [TRAIN] epoch: 70, iter: 25840/40000, loss: 0.3310, lr: 0.003988, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 03:18:13 2021-05-06 19:14:12 [INFO] [TRAIN] epoch: 70, iter: 25850/40000, loss: 0.3399, lr: 0.003986, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 03:18:19 2021-05-06 19:14:21 [INFO] [TRAIN] epoch: 70, iter: 25860/40000, loss: 0.8265, lr: 0.003983, batch_cost: 0.8392, reader_cost: 0.00016, ips: 1.1916 samples/sec | ETA 03:17:46 2021-05-06 19:14:29 [INFO] [TRAIN] epoch: 70, iter: 25870/40000, loss: 0.4302, lr: 0.003981, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 03:17:49 2021-05-06 19:14:38 [INFO] [TRAIN] epoch: 70, iter: 25880/40000, loss: 0.3705, lr: 0.003978, batch_cost: 0.8411, reader_cost: 0.00017, ips: 1.1890 samples/sec | ETA 03:17:55 2021-05-06 19:14:46 [INFO] [TRAIN] epoch: 70, iter: 25890/40000, loss: 0.3979, lr: 0.003976, batch_cost: 0.8400, reader_cost: 0.00017, ips: 1.1905 samples/sec | ETA 03:17:32 2021-05-06 19:14:54 [INFO] [TRAIN] epoch: 70, iter: 25900/40000, loss: 0.1353, lr: 0.003974, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 03:17:24 2021-05-06 19:15:03 [INFO] [TRAIN] epoch: 70, iter: 25910/40000, loss: 0.2703, lr: 0.003971, batch_cost: 0.8408, reader_cost: 0.00014, ips: 1.1893 samples/sec | ETA 03:17:27 2021-05-06 19:15:11 [INFO] [TRAIN] epoch: 70, iter: 25920/40000, loss: 0.2048, lr: 0.003969, batch_cost: 0.8408, reader_cost: 0.00014, ips: 1.1894 samples/sec | ETA 03:17:18 2021-05-06 19:15:20 [INFO] [TRAIN] epoch: 70, iter: 25930/40000, loss: 0.3915, lr: 0.003966, batch_cost: 0.8421, reader_cost: 0.00015, ips: 1.1876 samples/sec | ETA 03:17:27 2021-05-06 19:15:28 [INFO] [TRAIN] epoch: 70, iter: 25940/40000, loss: 0.2577, lr: 0.003964, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1913 samples/sec | ETA 03:16:41 2021-05-06 19:15:36 [INFO] [TRAIN] epoch: 70, iter: 25950/40000, loss: 0.3297, lr: 0.003961, batch_cost: 0.8405, reader_cost: 0.00016, ips: 1.1898 samples/sec | ETA 03:16:49 2021-05-06 19:15:45 [INFO] [TRAIN] epoch: 70, iter: 25960/40000, loss: 0.4807, lr: 0.003959, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1886 samples/sec | ETA 03:16:52 2021-05-06 19:15:53 [INFO] [TRAIN] epoch: 70, iter: 25970/40000, loss: 0.3505, lr: 0.003956, batch_cost: 0.8388, reader_cost: 0.00014, ips: 1.1921 samples/sec | ETA 03:16:08 2021-05-06 19:16:02 [INFO] [TRAIN] epoch: 70, iter: 25980/40000, loss: 0.2295, lr: 0.003954, batch_cost: 0.8399, reader_cost: 0.00014, ips: 1.1906 samples/sec | ETA 03:16:15 2021-05-06 19:16:10 [INFO] [TRAIN] epoch: 70, iter: 25990/40000, loss: 0.4896, lr: 0.003951, batch_cost: 0.8396, reader_cost: 0.00016, ips: 1.1911 samples/sec | ETA 03:16:02 2021-05-06 19:16:18 [INFO] [TRAIN] epoch: 70, iter: 26000/40000, loss: 0.3720, lr: 0.003949, batch_cost: 0.8411, reader_cost: 0.00017, ips: 1.1889 samples/sec | ETA 03:16:15 2021-05-06 19:16:18 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 19:19:55 [INFO] [EVAL] #Images: 500 mIoU: 0.7536 Acc: 0.9564 Kappa: 0.9435 2021-05-06 19:19:55 [INFO] [EVAL] Class IoU: [0.9813 0.8464 0.918 0.6277 0.6084 0.5273 0.6049 0.7146 0.9155 0.6518 0.9442 0.7773 0.5814 0.9376 0.7807 0.8736 0.7313 0.5925 0.7031] 2021-05-06 19:19:55 [INFO] [EVAL] Class Acc: [0.9922 0.9184 0.9542 0.899 0.7985 0.7625 0.7527 0.9005 0.9441 0.8165 0.9667 0.8595 0.7057 0.9619 0.8747 0.9272 0.9187 0.7355 0.7901] 2021-05-06 19:20:24 [INFO] [EVAL] The model with the best validation mIoU (0.7598) was saved at iter 22000. 2021-05-06 19:20:33 [INFO] [TRAIN] epoch: 70, iter: 26010/40000, loss: 0.2988, lr: 0.003946, batch_cost: 0.8365, reader_cost: 0.00034, ips: 1.1955 samples/sec | ETA 03:15:02 2021-05-06 19:20:41 [INFO] [TRAIN] epoch: 70, iter: 26020/40000, loss: 0.3013, lr: 0.003944, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1918 samples/sec | ETA 03:15:30 2021-05-06 19:20:49 [INFO] [TRAIN] epoch: 70, iter: 26030/40000, loss: 0.4284, lr: 0.003941, batch_cost: 0.8394, reader_cost: 0.00014, ips: 1.1913 samples/sec | ETA 03:15:26 2021-05-06 19:20:58 [INFO] [TRAIN] epoch: 70, iter: 26040/40000, loss: 0.3253, lr: 0.003939, batch_cost: 0.8381, reader_cost: 0.00009, ips: 1.1932 samples/sec | ETA 03:14:59 2021-05-06 19:21:10 [INFO] [TRAIN] epoch: 71, iter: 26050/40000, loss: 0.4865, lr: 0.003936, batch_cost: 1.1952, reader_cost: 0.23685, ips: 0.8367 samples/sec | ETA 04:37:53 2021-05-06 19:21:18 [INFO] [TRAIN] epoch: 71, iter: 26060/40000, loss: 0.3354, lr: 0.003934, batch_cost: 0.8389, reader_cost: 0.00031, ips: 1.1920 samples/sec | ETA 03:14:54 2021-05-06 19:21:26 [INFO] [TRAIN] epoch: 71, iter: 26070/40000, loss: 0.4021, lr: 0.003931, batch_cost: 0.8396, reader_cost: 0.00014, ips: 1.1911 samples/sec | ETA 03:14:55 2021-05-06 19:21:35 [INFO] [TRAIN] epoch: 71, iter: 26080/40000, loss: 0.2255, lr: 0.003929, batch_cost: 0.8386, reader_cost: 0.00015, ips: 1.1925 samples/sec | ETA 03:14:33 2021-05-06 19:21:43 [INFO] [TRAIN] epoch: 71, iter: 26090/40000, loss: 0.5169, lr: 0.003927, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 03:14:57 2021-05-06 19:21:52 [INFO] [TRAIN] epoch: 71, iter: 26100/40000, loss: 0.5754, lr: 0.003924, batch_cost: 0.8408, reader_cost: 0.00014, ips: 1.1894 samples/sec | ETA 03:14:47 2021-05-06 19:22:00 [INFO] [TRAIN] epoch: 71, iter: 26110/40000, loss: 0.4741, lr: 0.003922, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1886 samples/sec | ETA 03:14:46 2021-05-06 19:22:08 [INFO] [TRAIN] epoch: 71, iter: 26120/40000, loss: 0.5712, lr: 0.003919, batch_cost: 0.8392, reader_cost: 0.00015, ips: 1.1917 samples/sec | ETA 03:14:07 2021-05-06 19:22:17 [INFO] [TRAIN] epoch: 71, iter: 26130/40000, loss: 0.4805, lr: 0.003917, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 03:14:25 2021-05-06 19:22:25 [INFO] [TRAIN] epoch: 71, iter: 26140/40000, loss: 0.1322, lr: 0.003914, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 03:13:58 2021-05-06 19:22:34 [INFO] [TRAIN] epoch: 71, iter: 26150/40000, loss: 0.2676, lr: 0.003912, batch_cost: 0.8392, reader_cost: 0.00014, ips: 1.1916 samples/sec | ETA 03:13:43 2021-05-06 19:22:42 [INFO] [TRAIN] epoch: 71, iter: 26160/40000, loss: 0.3152, lr: 0.003909, batch_cost: 0.8389, reader_cost: 0.00014, ips: 1.1920 samples/sec | ETA 03:13:30 2021-05-06 19:22:50 [INFO] [TRAIN] epoch: 71, iter: 26170/40000, loss: 0.3978, lr: 0.003907, batch_cost: 0.8395, reader_cost: 0.00014, ips: 1.1912 samples/sec | ETA 03:13:30 2021-05-06 19:22:59 [INFO] [TRAIN] epoch: 71, iter: 26180/40000, loss: 0.5254, lr: 0.003904, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 03:13:37 2021-05-06 19:23:07 [INFO] [TRAIN] epoch: 71, iter: 26190/40000, loss: 0.5006, lr: 0.003902, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1880 samples/sec | ETA 03:13:44 2021-05-06 19:23:16 [INFO] [TRAIN] epoch: 71, iter: 26200/40000, loss: 0.2712, lr: 0.003899, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 03:13:11 2021-05-06 19:23:24 [INFO] [TRAIN] epoch: 71, iter: 26210/40000, loss: 0.3919, lr: 0.003897, batch_cost: 0.8399, reader_cost: 0.00016, ips: 1.1907 samples/sec | ETA 03:13:01 2021-05-06 19:23:32 [INFO] [TRAIN] epoch: 71, iter: 26220/40000, loss: 0.4669, lr: 0.003894, batch_cost: 0.8389, reader_cost: 0.00015, ips: 1.1921 samples/sec | ETA 03:12:39 2021-05-06 19:23:41 [INFO] [TRAIN] epoch: 71, iter: 26230/40000, loss: 0.4920, lr: 0.003892, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 03:12:48 2021-05-06 19:23:49 [INFO] [TRAIN] epoch: 71, iter: 26240/40000, loss: 0.4214, lr: 0.003889, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 03:12:38 2021-05-06 19:23:58 [INFO] [TRAIN] epoch: 71, iter: 26250/40000, loss: 0.4169, lr: 0.003887, batch_cost: 0.8408, reader_cost: 0.00016, ips: 1.1893 samples/sec | ETA 03:12:41 2021-05-06 19:24:06 [INFO] [TRAIN] epoch: 71, iter: 26260/40000, loss: 0.3719, lr: 0.003884, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1886 samples/sec | ETA 03:12:40 2021-05-06 19:24:15 [INFO] [TRAIN] epoch: 71, iter: 26270/40000, loss: 0.2043, lr: 0.003882, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1883 samples/sec | ETA 03:12:33 2021-05-06 19:24:23 [INFO] [TRAIN] epoch: 71, iter: 26280/40000, loss: 0.2949, lr: 0.003879, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 03:12:01 2021-05-06 19:24:31 [INFO] [TRAIN] epoch: 71, iter: 26290/40000, loss: 0.3025, lr: 0.003877, batch_cost: 0.8403, reader_cost: 0.00014, ips: 1.1901 samples/sec | ETA 03:11:59 2021-05-06 19:24:40 [INFO] [TRAIN] epoch: 71, iter: 26300/40000, loss: 0.3375, lr: 0.003874, batch_cost: 0.8384, reader_cost: 0.00015, ips: 1.1927 samples/sec | ETA 03:11:26 2021-05-06 19:24:48 [INFO] [TRAIN] epoch: 71, iter: 26310/40000, loss: 0.2962, lr: 0.003872, batch_cost: 0.8414, reader_cost: 0.00014, ips: 1.1885 samples/sec | ETA 03:11:59 2021-05-06 19:24:57 [INFO] [TRAIN] epoch: 71, iter: 26320/40000, loss: 0.3927, lr: 0.003870, batch_cost: 0.8392, reader_cost: 0.00015, ips: 1.1916 samples/sec | ETA 03:11:20 2021-05-06 19:25:05 [INFO] [TRAIN] epoch: 71, iter: 26330/40000, loss: 0.3340, lr: 0.003867, batch_cost: 0.8409, reader_cost: 0.00017, ips: 1.1893 samples/sec | ETA 03:11:34 2021-05-06 19:25:13 [INFO] [TRAIN] epoch: 71, iter: 26340/40000, loss: 0.3103, lr: 0.003865, batch_cost: 0.8417, reader_cost: 0.00016, ips: 1.1881 samples/sec | ETA 03:11:36 2021-05-06 19:25:22 [INFO] [TRAIN] epoch: 71, iter: 26350/40000, loss: 0.2772, lr: 0.003862, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 03:11:09 2021-05-06 19:25:30 [INFO] [TRAIN] epoch: 71, iter: 26360/40000, loss: 0.3681, lr: 0.003860, batch_cost: 0.8404, reader_cost: 0.00016, ips: 1.1899 samples/sec | ETA 03:11:02 2021-05-06 19:25:39 [INFO] [TRAIN] epoch: 71, iter: 26370/40000, loss: 0.4170, lr: 0.003857, batch_cost: 0.8400, reader_cost: 0.00016, ips: 1.1904 samples/sec | ETA 03:10:49 2021-05-06 19:25:47 [INFO] [TRAIN] epoch: 71, iter: 26380/40000, loss: 0.3406, lr: 0.003855, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 03:10:37 2021-05-06 19:25:55 [INFO] [TRAIN] epoch: 71, iter: 26390/40000, loss: 0.3784, lr: 0.003852, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 03:10:41 2021-05-06 19:26:04 [INFO] [TRAIN] epoch: 71, iter: 26400/40000, loss: 0.4034, lr: 0.003850, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 03:10:23 2021-05-06 19:26:12 [INFO] [TRAIN] epoch: 71, iter: 26410/40000, loss: 0.3503, lr: 0.003847, batch_cost: 0.8390, reader_cost: 0.00031, ips: 1.1918 samples/sec | ETA 03:10:02 2021-05-06 19:26:24 [INFO] [TRAIN] epoch: 72, iter: 26420/40000, loss: 0.4491, lr: 0.003845, batch_cost: 1.1624, reader_cost: 0.22346, ips: 0.8603 samples/sec | ETA 04:23:05 2021-05-06 19:26:32 [INFO] [TRAIN] epoch: 72, iter: 26430/40000, loss: 0.3131, lr: 0.003842, batch_cost: 0.8403, reader_cost: 0.00033, ips: 1.1901 samples/sec | ETA 03:10:02 2021-05-06 19:26:41 [INFO] [TRAIN] epoch: 72, iter: 26440/40000, loss: 0.3820, lr: 0.003840, batch_cost: 0.8424, reader_cost: 0.00015, ips: 1.1871 samples/sec | ETA 03:10:22 2021-05-06 19:26:49 [INFO] [TRAIN] epoch: 72, iter: 26450/40000, loss: 0.3744, lr: 0.003837, batch_cost: 0.8411, reader_cost: 0.00014, ips: 1.1889 samples/sec | ETA 03:09:57 2021-05-06 19:26:57 [INFO] [TRAIN] epoch: 72, iter: 26460/40000, loss: 0.4101, lr: 0.003835, batch_cost: 0.8389, reader_cost: 0.00015, ips: 1.1920 samples/sec | ETA 03:09:19 2021-05-06 19:27:06 [INFO] [TRAIN] epoch: 72, iter: 26470/40000, loss: 0.5241, lr: 0.003832, batch_cost: 0.8425, reader_cost: 0.00014, ips: 1.1869 samples/sec | ETA 03:09:59 2021-05-06 19:27:14 [INFO] [TRAIN] epoch: 72, iter: 26480/40000, loss: 0.5558, lr: 0.003830, batch_cost: 0.8396, reader_cost: 0.00014, ips: 1.1910 samples/sec | ETA 03:09:11 2021-05-06 19:27:23 [INFO] [TRAIN] epoch: 72, iter: 26490/40000, loss: 0.7700, lr: 0.003827, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 03:09:10 2021-05-06 19:27:31 [INFO] [TRAIN] epoch: 72, iter: 26500/40000, loss: 0.3622, lr: 0.003825, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 03:09:12 2021-05-06 19:27:39 [INFO] [TRAIN] epoch: 72, iter: 26510/40000, loss: 0.1627, lr: 0.003822, batch_cost: 0.8416, reader_cost: 0.00016, ips: 1.1882 samples/sec | ETA 03:09:12 2021-05-06 19:27:48 [INFO] [TRAIN] epoch: 72, iter: 26520/40000, loss: 0.2977, lr: 0.003820, batch_cost: 0.8404, reader_cost: 0.00014, ips: 1.1900 samples/sec | ETA 03:08:48 2021-05-06 19:27:56 [INFO] [TRAIN] epoch: 72, iter: 26530/40000, loss: 0.3759, lr: 0.003817, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 03:08:46 2021-05-06 19:28:05 [INFO] [TRAIN] epoch: 72, iter: 26540/40000, loss: 0.5375, lr: 0.003815, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 03:08:28 2021-05-06 19:28:13 [INFO] [TRAIN] epoch: 72, iter: 26550/40000, loss: 0.5206, lr: 0.003812, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 03:08:27 2021-05-06 19:28:21 [INFO] [TRAIN] epoch: 72, iter: 26560/40000, loss: 0.4838, lr: 0.003810, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1912 samples/sec | ETA 03:08:03 2021-05-06 19:28:30 [INFO] [TRAIN] epoch: 72, iter: 26570/40000, loss: 0.2385, lr: 0.003807, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 03:08:08 2021-05-06 19:28:38 [INFO] [TRAIN] epoch: 72, iter: 26580/40000, loss: 0.3046, lr: 0.003805, batch_cost: 0.8397, reader_cost: 0.00016, ips: 1.1909 samples/sec | ETA 03:07:48 2021-05-06 19:28:47 [INFO] [TRAIN] epoch: 72, iter: 26590/40000, loss: 0.7744, lr: 0.003803, batch_cost: 0.8404, reader_cost: 0.00017, ips: 1.1900 samples/sec | ETA 03:07:49 2021-05-06 19:28:55 [INFO] [TRAIN] epoch: 72, iter: 26600/40000, loss: 0.6115, lr: 0.003800, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 03:07:36 2021-05-06 19:29:04 [INFO] [TRAIN] epoch: 72, iter: 26610/40000, loss: 0.6949, lr: 0.003798, batch_cost: 0.8398, reader_cost: 0.00016, ips: 1.1908 samples/sec | ETA 03:07:24 2021-05-06 19:29:12 [INFO] [TRAIN] epoch: 72, iter: 26620/40000, loss: 0.3803, lr: 0.003795, batch_cost: 0.8434, reader_cost: 0.00016, ips: 1.1857 samples/sec | ETA 03:08:04 2021-05-06 19:29:20 [INFO] [TRAIN] epoch: 72, iter: 26630/40000, loss: 0.3048, lr: 0.003793, batch_cost: 0.8397, reader_cost: 0.00014, ips: 1.1909 samples/sec | ETA 03:07:06 2021-05-06 19:29:29 [INFO] [TRAIN] epoch: 72, iter: 26640/40000, loss: 0.2001, lr: 0.003790, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 03:07:05 2021-05-06 19:29:37 [INFO] [TRAIN] epoch: 72, iter: 26650/40000, loss: 0.2420, lr: 0.003788, batch_cost: 0.8416, reader_cost: 0.00016, ips: 1.1882 samples/sec | ETA 03:07:15 2021-05-06 19:29:46 [INFO] [TRAIN] epoch: 72, iter: 26660/40000, loss: 0.3025, lr: 0.003785, batch_cost: 0.8407, reader_cost: 0.00014, ips: 1.1895 samples/sec | ETA 03:06:54 2021-05-06 19:29:54 [INFO] [TRAIN] epoch: 72, iter: 26670/40000, loss: 0.3105, lr: 0.003783, batch_cost: 0.8407, reader_cost: 0.00014, ips: 1.1895 samples/sec | ETA 03:06:46 2021-05-06 19:30:02 [INFO] [TRAIN] epoch: 72, iter: 26680/40000, loss: 0.2870, lr: 0.003780, batch_cost: 0.8405, reader_cost: 0.00016, ips: 1.1897 samples/sec | ETA 03:06:35 2021-05-06 19:30:11 [INFO] [TRAIN] epoch: 72, iter: 26690/40000, loss: 0.2949, lr: 0.003778, batch_cost: 0.8402, reader_cost: 0.00014, ips: 1.1901 samples/sec | ETA 03:06:23 2021-05-06 19:30:19 [INFO] [TRAIN] epoch: 72, iter: 26700/40000, loss: 0.4092, lr: 0.003775, batch_cost: 0.8390, reader_cost: 0.00014, ips: 1.1918 samples/sec | ETA 03:05:59 2021-05-06 19:30:28 [INFO] [TRAIN] epoch: 72, iter: 26710/40000, loss: 0.4544, lr: 0.003773, batch_cost: 0.8395, reader_cost: 0.00014, ips: 1.1912 samples/sec | ETA 03:05:57 2021-05-06 19:30:36 [INFO] [TRAIN] epoch: 72, iter: 26720/40000, loss: 0.2473, lr: 0.003770, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 03:06:02 2021-05-06 19:30:44 [INFO] [TRAIN] epoch: 72, iter: 26730/40000, loss: 0.4260, lr: 0.003768, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1881 samples/sec | ETA 03:06:09 2021-05-06 19:30:53 [INFO] [TRAIN] epoch: 72, iter: 26740/40000, loss: 0.4913, lr: 0.003765, batch_cost: 0.8394, reader_cost: 0.00016, ips: 1.1914 samples/sec | ETA 03:05:29 2021-05-06 19:31:01 [INFO] [TRAIN] epoch: 72, iter: 26750/40000, loss: 0.3954, lr: 0.003763, batch_cost: 0.8391, reader_cost: 0.00014, ips: 1.1917 samples/sec | ETA 03:05:18 2021-05-06 19:31:10 [INFO] [TRAIN] epoch: 72, iter: 26760/40000, loss: 0.2764, lr: 0.003760, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 03:05:23 2021-05-06 19:31:18 [INFO] [TRAIN] epoch: 72, iter: 26770/40000, loss: 0.3589, lr: 0.003758, batch_cost: 0.8421, reader_cost: 0.00014, ips: 1.1876 samples/sec | ETA 03:05:40 2021-05-06 19:31:26 [INFO] [TRAIN] epoch: 72, iter: 26780/40000, loss: 0.3440, lr: 0.003755, batch_cost: 0.8400, reader_cost: 0.00010, ips: 1.1905 samples/sec | ETA 03:05:04 2021-05-06 19:31:38 [INFO] [TRAIN] epoch: 73, iter: 26790/40000, loss: 0.5917, lr: 0.003753, batch_cost: 1.1786, reader_cost: 0.21361, ips: 0.8485 samples/sec | ETA 04:19:29 2021-05-06 19:31:47 [INFO] [TRAIN] epoch: 73, iter: 26800/40000, loss: 0.3649, lr: 0.003750, batch_cost: 0.8414, reader_cost: 0.00032, ips: 1.1885 samples/sec | ETA 03:05:06 2021-05-06 19:31:55 [INFO] [TRAIN] epoch: 73, iter: 26810/40000, loss: 0.5492, lr: 0.003748, batch_cost: 0.8412, reader_cost: 0.00014, ips: 1.1888 samples/sec | ETA 03:04:55 2021-05-06 19:32:03 [INFO] [TRAIN] epoch: 73, iter: 26820/40000, loss: 0.6160, lr: 0.003745, batch_cost: 0.8421, reader_cost: 0.00016, ips: 1.1875 samples/sec | ETA 03:04:59 2021-05-06 19:32:12 [INFO] [TRAIN] epoch: 73, iter: 26830/40000, loss: 0.3941, lr: 0.003743, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 03:04:27 2021-05-06 19:32:20 [INFO] [TRAIN] epoch: 73, iter: 26840/40000, loss: 0.3857, lr: 0.003740, batch_cost: 0.8432, reader_cost: 0.00014, ips: 1.1859 samples/sec | ETA 03:04:57 2021-05-06 19:32:29 [INFO] [TRAIN] epoch: 73, iter: 26850/40000, loss: 0.4739, lr: 0.003738, batch_cost: 0.8400, reader_cost: 0.00014, ips: 1.1905 samples/sec | ETA 03:04:05 2021-05-06 19:32:37 [INFO] [TRAIN] epoch: 73, iter: 26860/40000, loss: 0.6097, lr: 0.003735, batch_cost: 0.8416, reader_cost: 0.00014, ips: 1.1882 samples/sec | ETA 03:04:18 2021-05-06 19:32:46 [INFO] [TRAIN] epoch: 73, iter: 26870/40000, loss: 0.4849, lr: 0.003733, batch_cost: 0.8425, reader_cost: 0.00015, ips: 1.1870 samples/sec | ETA 03:04:21 2021-05-06 19:32:54 [INFO] [TRAIN] epoch: 73, iter: 26880/40000, loss: 0.2756, lr: 0.003730, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 03:03:38 2021-05-06 19:33:02 [INFO] [TRAIN] epoch: 73, iter: 26890/40000, loss: 0.2681, lr: 0.003728, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1887 samples/sec | ETA 03:03:48 2021-05-06 19:33:11 [INFO] [TRAIN] epoch: 73, iter: 26900/40000, loss: 0.3223, lr: 0.003725, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 03:03:33 2021-05-06 19:33:19 [INFO] [TRAIN] epoch: 73, iter: 26910/40000, loss: 0.2839, lr: 0.003723, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1912 samples/sec | ETA 03:03:09 2021-05-06 19:33:28 [INFO] [TRAIN] epoch: 73, iter: 26920/40000, loss: 0.5669, lr: 0.003720, batch_cost: 0.8390, reader_cost: 0.00015, ips: 1.1919 samples/sec | ETA 03:02:53 2021-05-06 19:33:36 [INFO] [TRAIN] epoch: 73, iter: 26930/40000, loss: 0.5205, lr: 0.003718, batch_cost: 0.8399, reader_cost: 0.00014, ips: 1.1906 samples/sec | ETA 03:02:57 2021-05-06 19:33:44 [INFO] [TRAIN] epoch: 73, iter: 26940/40000, loss: 0.2485, lr: 0.003715, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 03:02:53 2021-05-06 19:33:53 [INFO] [TRAIN] epoch: 73, iter: 26950/40000, loss: 0.3341, lr: 0.003713, batch_cost: 0.8407, reader_cost: 0.00016, ips: 1.1894 samples/sec | ETA 03:02:51 2021-05-06 19:34:01 [INFO] [TRAIN] epoch: 73, iter: 26960/40000, loss: 0.3132, lr: 0.003710, batch_cost: 0.8395, reader_cost: 0.00014, ips: 1.1912 samples/sec | ETA 03:02:27 2021-05-06 19:34:10 [INFO] [TRAIN] epoch: 73, iter: 26970/40000, loss: 0.4813, lr: 0.003708, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 03:02:28 2021-05-06 19:34:18 [INFO] [TRAIN] epoch: 73, iter: 26980/40000, loss: 0.6922, lr: 0.003705, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 03:02:07 2021-05-06 19:34:26 [INFO] [TRAIN] epoch: 73, iter: 26990/40000, loss: 0.3898, lr: 0.003703, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 03:02:08 2021-05-06 19:34:35 [INFO] [TRAIN] epoch: 73, iter: 27000/40000, loss: 0.4345, lr: 0.003700, batch_cost: 0.8415, reader_cost: 0.00014, ips: 1.1884 samples/sec | ETA 03:02:19 2021-05-06 19:34:35 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 19:38:12 [INFO] [EVAL] #Images: 500 mIoU: 0.7336 Acc: 0.9552 Kappa: 0.9420 2021-05-06 19:38:12 [INFO] [EVAL] Class IoU: [0.979 0.8348 0.919 0.6533 0.5847 0.5388 0.597 0.7124 0.9168 0.6425 0.9416 0.775 0.5382 0.9392 0.7769 0.8725 0.7726 0.2707 0.6724] 2021-05-06 19:38:12 [INFO] [EVAL] Class Acc: [0.993 0.8998 0.9522 0.8218 0.7354 0.7495 0.7867 0.8548 0.9541 0.8089 0.9628 0.8519 0.7783 0.9646 0.9548 0.9281 0.8727 0.7129 0.7577] 2021-05-06 19:38:41 [INFO] [EVAL] The model with the best validation mIoU (0.7598) was saved at iter 22000. 2021-05-06 19:38:50 [INFO] [TRAIN] epoch: 73, iter: 27010/40000, loss: 0.1863, lr: 0.003698, batch_cost: 0.8394, reader_cost: 0.00025, ips: 1.1914 samples/sec | ETA 03:01:43 2021-05-06 19:38:58 [INFO] [TRAIN] epoch: 73, iter: 27020/40000, loss: 0.3118, lr: 0.003695, batch_cost: 0.8405, reader_cost: 0.00016, ips: 1.1898 samples/sec | ETA 03:01:49 2021-05-06 19:39:06 [INFO] [TRAIN] epoch: 73, iter: 27030/40000, loss: 0.2815, lr: 0.003693, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 03:01:40 2021-05-06 19:39:16 [INFO] [TRAIN] epoch: 73, iter: 27040/40000, loss: 0.3311, lr: 0.003691, batch_cost: 0.8387, reader_cost: 0.00017, ips: 1.1923 samples/sec | ETA 03:01:09 2021-05-06 19:39:25 [INFO] [TRAIN] epoch: 73, iter: 27050/40000, loss: 0.3092, lr: 0.003688, batch_cost: 0.8415, reader_cost: 0.00016, ips: 1.1883 samples/sec | ETA 03:01:37 2021-05-06 19:39:33 [INFO] [TRAIN] epoch: 73, iter: 27060/40000, loss: 0.3141, lr: 0.003686, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 03:01:20 2021-05-06 19:39:42 [INFO] [TRAIN] epoch: 73, iter: 27070/40000, loss: 0.2934, lr: 0.003683, batch_cost: 0.8421, reader_cost: 0.00015, ips: 1.1875 samples/sec | ETA 03:01:28 2021-05-06 19:39:50 [INFO] [TRAIN] epoch: 73, iter: 27080/40000, loss: 0.3130, lr: 0.003681, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 03:00:59 2021-05-06 19:39:58 [INFO] [TRAIN] epoch: 73, iter: 27090/40000, loss: 0.2014, lr: 0.003678, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 03:00:57 2021-05-06 19:40:07 [INFO] [TRAIN] epoch: 73, iter: 27100/40000, loss: 0.2829, lr: 0.003676, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 03:00:48 2021-05-06 19:40:15 [INFO] [TRAIN] epoch: 73, iter: 27110/40000, loss: 0.3621, lr: 0.003673, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 03:00:23 2021-05-06 19:40:24 [INFO] [TRAIN] epoch: 73, iter: 27120/40000, loss: 0.2661, lr: 0.003671, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1888 samples/sec | ETA 03:00:34 2021-05-06 19:40:32 [INFO] [TRAIN] epoch: 73, iter: 27130/40000, loss: 0.4138, lr: 0.003668, batch_cost: 0.8390, reader_cost: 0.00015, ips: 1.1918 samples/sec | ETA 02:59:58 2021-05-06 19:40:40 [INFO] [TRAIN] epoch: 73, iter: 27140/40000, loss: 0.3133, lr: 0.003666, batch_cost: 0.8416, reader_cost: 0.00014, ips: 1.1883 samples/sec | ETA 03:00:22 2021-05-06 19:40:49 [INFO] [TRAIN] epoch: 73, iter: 27150/40000, loss: 0.4203, lr: 0.003663, batch_cost: 0.8394, reader_cost: 0.00012, ips: 1.1914 samples/sec | ETA 02:59:45 2021-05-06 19:41:01 [INFO] [TRAIN] epoch: 74, iter: 27160/40000, loss: 0.3078, lr: 0.003661, batch_cost: 1.1749, reader_cost: 0.22901, ips: 0.8511 samples/sec | ETA 04:11:25 2021-05-06 19:41:09 [INFO] [TRAIN] epoch: 74, iter: 27170/40000, loss: 0.3555, lr: 0.003658, batch_cost: 0.8511, reader_cost: 0.00034, ips: 1.1750 samples/sec | ETA 03:01:59 2021-05-06 19:41:17 [INFO] [TRAIN] epoch: 74, iter: 27180/40000, loss: 0.3221, lr: 0.003656, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1880 samples/sec | ETA 02:59:51 2021-05-06 19:41:26 [INFO] [TRAIN] epoch: 74, iter: 27190/40000, loss: 0.3982, lr: 0.003653, batch_cost: 0.8407, reader_cost: 0.00016, ips: 1.1894 samples/sec | ETA 02:59:29 2021-05-06 19:41:34 [INFO] [TRAIN] epoch: 74, iter: 27200/40000, loss: 0.2910, lr: 0.003651, batch_cost: 0.8405, reader_cost: 0.00014, ips: 1.1897 samples/sec | ETA 02:59:18 2021-05-06 19:41:43 [INFO] [TRAIN] epoch: 74, iter: 27210/40000, loss: 0.4678, lr: 0.003648, batch_cost: 0.8429, reader_cost: 0.00014, ips: 1.1864 samples/sec | ETA 02:59:40 2021-05-06 19:41:51 [INFO] [TRAIN] epoch: 74, iter: 27220/40000, loss: 0.6042, lr: 0.003646, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1918 samples/sec | ETA 02:58:43 2021-05-06 19:42:00 [INFO] [TRAIN] epoch: 74, iter: 27230/40000, loss: 0.5350, lr: 0.003643, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 02:58:43 2021-05-06 19:42:08 [INFO] [TRAIN] epoch: 74, iter: 27240/40000, loss: 0.4818, lr: 0.003641, batch_cost: 0.8407, reader_cost: 0.00017, ips: 1.1895 samples/sec | ETA 02:58:46 2021-05-06 19:42:16 [INFO] [TRAIN] epoch: 74, iter: 27250/40000, loss: 0.2946, lr: 0.003638, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 02:58:43 2021-05-06 19:42:25 [INFO] [TRAIN] epoch: 74, iter: 27260/40000, loss: 0.2786, lr: 0.003636, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 02:58:18 2021-05-06 19:42:33 [INFO] [TRAIN] epoch: 74, iter: 27270/40000, loss: 0.3513, lr: 0.003633, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1904 samples/sec | ETA 02:58:13 2021-05-06 19:42:42 [INFO] [TRAIN] epoch: 74, iter: 27280/40000, loss: 0.2740, lr: 0.003631, batch_cost: 0.8407, reader_cost: 0.00014, ips: 1.1895 samples/sec | ETA 02:58:13 2021-05-06 19:42:50 [INFO] [TRAIN] epoch: 74, iter: 27290/40000, loss: 0.4690, lr: 0.003628, batch_cost: 0.8392, reader_cost: 0.00015, ips: 1.1917 samples/sec | ETA 02:57:45 2021-05-06 19:42:58 [INFO] [TRAIN] epoch: 74, iter: 27300/40000, loss: 0.4880, lr: 0.003626, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1904 samples/sec | ETA 02:57:49 2021-05-06 19:43:07 [INFO] [TRAIN] epoch: 74, iter: 27310/40000, loss: 0.2390, lr: 0.003623, batch_cost: 0.8381, reader_cost: 0.00014, ips: 1.1932 samples/sec | ETA 02:57:15 2021-05-06 19:43:15 [INFO] [TRAIN] epoch: 74, iter: 27320/40000, loss: 0.3590, lr: 0.003621, batch_cost: 0.8393, reader_cost: 0.00014, ips: 1.1915 samples/sec | ETA 02:57:22 2021-05-06 19:43:24 [INFO] [TRAIN] epoch: 74, iter: 27330/40000, loss: 0.3083, lr: 0.003618, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 02:57:26 2021-05-06 19:43:32 [INFO] [TRAIN] epoch: 74, iter: 27340/40000, loss: 0.4343, lr: 0.003616, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 02:57:11 2021-05-06 19:43:40 [INFO] [TRAIN] epoch: 74, iter: 27350/40000, loss: 0.4829, lr: 0.003613, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 02:57:13 2021-05-06 19:43:49 [INFO] [TRAIN] epoch: 74, iter: 27360/40000, loss: 0.3954, lr: 0.003611, batch_cost: 0.8398, reader_cost: 0.00016, ips: 1.1908 samples/sec | ETA 02:56:54 2021-05-06 19:43:57 [INFO] [TRAIN] epoch: 74, iter: 27370/40000, loss: 0.3603, lr: 0.003608, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 02:57:00 2021-05-06 19:44:06 [INFO] [TRAIN] epoch: 74, iter: 27380/40000, loss: 0.2400, lr: 0.003606, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 02:56:46 2021-05-06 19:44:14 [INFO] [TRAIN] epoch: 74, iter: 27390/40000, loss: 0.2392, lr: 0.003603, batch_cost: 0.8407, reader_cost: 0.00014, ips: 1.1895 samples/sec | ETA 02:56:41 2021-05-06 19:44:22 [INFO] [TRAIN] epoch: 74, iter: 27400/40000, loss: 0.2354, lr: 0.003601, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 02:56:31 2021-05-06 19:44:31 [INFO] [TRAIN] epoch: 74, iter: 27410/40000, loss: 0.2725, lr: 0.003598, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1911 samples/sec | ETA 02:56:10 2021-05-06 19:44:39 [INFO] [TRAIN] epoch: 74, iter: 27420/40000, loss: 0.3539, lr: 0.003596, batch_cost: 0.8398, reader_cost: 0.00014, ips: 1.1908 samples/sec | ETA 02:56:04 2021-05-06 19:44:48 [INFO] [TRAIN] epoch: 74, iter: 27430/40000, loss: 0.2600, lr: 0.003593, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 02:56:01 2021-05-06 19:44:56 [INFO] [TRAIN] epoch: 74, iter: 27440/40000, loss: 0.5401, lr: 0.003591, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 02:55:48 2021-05-06 19:45:04 [INFO] [TRAIN] epoch: 74, iter: 27450/40000, loss: 0.3868, lr: 0.003588, batch_cost: 0.8394, reader_cost: 0.00016, ips: 1.1913 samples/sec | ETA 02:55:34 2021-05-06 19:45:13 [INFO] [TRAIN] epoch: 74, iter: 27460/40000, loss: 0.2256, lr: 0.003586, batch_cost: 0.8386, reader_cost: 0.00015, ips: 1.1924 samples/sec | ETA 02:55:16 2021-05-06 19:45:21 [INFO] [TRAIN] epoch: 74, iter: 27470/40000, loss: 0.2819, lr: 0.003583, batch_cost: 0.8415, reader_cost: 0.00016, ips: 1.1884 samples/sec | ETA 02:55:43 2021-05-06 19:45:30 [INFO] [TRAIN] epoch: 74, iter: 27480/40000, loss: 0.4564, lr: 0.003581, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 02:55:22 2021-05-06 19:45:38 [INFO] [TRAIN] epoch: 74, iter: 27490/40000, loss: 0.2804, lr: 0.003578, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1887 samples/sec | ETA 02:55:24 2021-05-06 19:45:46 [INFO] [TRAIN] epoch: 74, iter: 27500/40000, loss: 0.4145, lr: 0.003576, batch_cost: 0.8409, reader_cost: 0.00016, ips: 1.1893 samples/sec | ETA 02:55:10 2021-05-06 19:45:55 [INFO] [TRAIN] epoch: 74, iter: 27510/40000, loss: 0.3982, lr: 0.003573, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1912 samples/sec | ETA 02:54:45 2021-05-06 19:46:03 [INFO] [TRAIN] epoch: 74, iter: 27520/40000, loss: 0.3741, lr: 0.003571, batch_cost: 0.8400, reader_cost: 0.00013, ips: 1.1904 samples/sec | ETA 02:54:43 2021-05-06 19:46:15 [INFO] [TRAIN] epoch: 75, iter: 27530/40000, loss: 0.2320, lr: 0.003568, batch_cost: 1.1368, reader_cost: 0.21065, ips: 0.8797 samples/sec | ETA 03:56:15 2021-05-06 19:46:23 [INFO] [TRAIN] epoch: 75, iter: 27540/40000, loss: 0.5377, lr: 0.003566, batch_cost: 0.8634, reader_cost: 0.00032, ips: 1.1582 samples/sec | ETA 02:59:18 2021-05-06 19:46:32 [INFO] [TRAIN] epoch: 75, iter: 27550/40000, loss: 0.4357, lr: 0.003563, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 02:54:29 2021-05-06 19:46:40 [INFO] [TRAIN] epoch: 75, iter: 27560/40000, loss: 0.3874, lr: 0.003561, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1886 samples/sec | ETA 02:54:26 2021-05-06 19:46:48 [INFO] [TRAIN] epoch: 75, iter: 27570/40000, loss: 0.2300, lr: 0.003558, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1903 samples/sec | ETA 02:54:02 2021-05-06 19:46:57 [INFO] [TRAIN] epoch: 75, iter: 27580/40000, loss: 0.5129, lr: 0.003556, batch_cost: 0.8421, reader_cost: 0.00017, ips: 1.1876 samples/sec | ETA 02:54:18 2021-05-06 19:47:05 [INFO] [TRAIN] epoch: 75, iter: 27590/40000, loss: 0.3968, lr: 0.003553, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 02:53:54 2021-05-06 19:47:14 [INFO] [TRAIN] epoch: 75, iter: 27600/40000, loss: 0.5419, lr: 0.003551, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 02:53:31 2021-05-06 19:47:22 [INFO] [TRAIN] epoch: 75, iter: 27610/40000, loss: 0.3691, lr: 0.003548, batch_cost: 0.8408, reader_cost: 0.00016, ips: 1.1893 samples/sec | ETA 02:53:37 2021-05-06 19:47:30 [INFO] [TRAIN] epoch: 75, iter: 27620/40000, loss: 0.3589, lr: 0.003546, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1911 samples/sec | ETA 02:53:14 2021-05-06 19:47:39 [INFO] [TRAIN] epoch: 75, iter: 27630/40000, loss: 0.2467, lr: 0.003543, batch_cost: 0.8398, reader_cost: 0.00017, ips: 1.1908 samples/sec | ETA 02:53:08 2021-05-06 19:47:47 [INFO] [TRAIN] epoch: 75, iter: 27640/40000, loss: 0.2469, lr: 0.003541, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1884 samples/sec | ETA 02:53:20 2021-05-06 19:47:56 [INFO] [TRAIN] epoch: 75, iter: 27650/40000, loss: 0.5362, lr: 0.003538, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 02:52:53 2021-05-06 19:48:04 [INFO] [TRAIN] epoch: 75, iter: 27660/40000, loss: 0.5250, lr: 0.003536, batch_cost: 0.8397, reader_cost: 0.00014, ips: 1.1910 samples/sec | ETA 02:52:41 2021-05-06 19:48:12 [INFO] [TRAIN] epoch: 75, iter: 27670/40000, loss: 0.5300, lr: 0.003533, batch_cost: 0.8388, reader_cost: 0.00016, ips: 1.1921 samples/sec | ETA 02:52:22 2021-05-06 19:48:21 [INFO] [TRAIN] epoch: 75, iter: 27680/40000, loss: 0.3755, lr: 0.003531, batch_cost: 0.8413, reader_cost: 0.00016, ips: 1.1887 samples/sec | ETA 02:52:44 2021-05-06 19:48:29 [INFO] [TRAIN] epoch: 75, iter: 27690/40000, loss: 0.2937, lr: 0.003528, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 02:52:25 2021-05-06 19:48:38 [INFO] [TRAIN] epoch: 75, iter: 27700/40000, loss: 0.3498, lr: 0.003526, batch_cost: 0.8405, reader_cost: 0.00016, ips: 1.1898 samples/sec | ETA 02:52:17 2021-05-06 19:48:46 [INFO] [TRAIN] epoch: 75, iter: 27710/40000, loss: 0.5715, lr: 0.003523, batch_cost: 0.8410, reader_cost: 0.00016, ips: 1.1890 samples/sec | ETA 02:52:16 2021-05-06 19:48:54 [INFO] [TRAIN] epoch: 75, iter: 27720/40000, loss: 0.6739, lr: 0.003521, batch_cost: 0.8375, reader_cost: 0.00015, ips: 1.1940 samples/sec | ETA 02:51:25 2021-05-06 19:49:03 [INFO] [TRAIN] epoch: 75, iter: 27730/40000, loss: 0.2648, lr: 0.003518, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 02:51:50 2021-05-06 19:49:11 [INFO] [TRAIN] epoch: 75, iter: 27740/40000, loss: 0.2766, lr: 0.003516, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 02:51:35 2021-05-06 19:49:20 [INFO] [TRAIN] epoch: 75, iter: 27750/40000, loss: 0.3595, lr: 0.003513, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 02:51:28 2021-05-06 19:49:28 [INFO] [TRAIN] epoch: 75, iter: 27760/40000, loss: 0.3146, lr: 0.003510, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 02:51:20 2021-05-06 19:49:36 [INFO] [TRAIN] epoch: 75, iter: 27770/40000, loss: 0.3337, lr: 0.003508, batch_cost: 0.8421, reader_cost: 0.00015, ips: 1.1876 samples/sec | ETA 02:51:38 2021-05-06 19:49:45 [INFO] [TRAIN] epoch: 75, iter: 27780/40000, loss: 0.2299, lr: 0.003505, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1903 samples/sec | ETA 02:51:06 2021-05-06 19:49:53 [INFO] [TRAIN] epoch: 75, iter: 27790/40000, loss: 0.2949, lr: 0.003503, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 02:51:01 2021-05-06 19:50:02 [INFO] [TRAIN] epoch: 75, iter: 27800/40000, loss: 0.3191, lr: 0.003500, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1882 samples/sec | ETA 02:51:07 2021-05-06 19:50:10 [INFO] [TRAIN] epoch: 75, iter: 27810/40000, loss: 0.2624, lr: 0.003498, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1887 samples/sec | ETA 02:50:55 2021-05-06 19:50:18 [INFO] [TRAIN] epoch: 75, iter: 27820/40000, loss: 0.3639, lr: 0.003495, batch_cost: 0.8394, reader_cost: 0.00014, ips: 1.1913 samples/sec | ETA 02:50:24 2021-05-06 19:50:27 [INFO] [TRAIN] epoch: 75, iter: 27830/40000, loss: 0.3479, lr: 0.003493, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 02:50:13 2021-05-06 19:50:35 [INFO] [TRAIN] epoch: 75, iter: 27840/40000, loss: 0.3554, lr: 0.003490, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 02:50:16 2021-05-06 19:50:44 [INFO] [TRAIN] epoch: 75, iter: 27850/40000, loss: 0.5245, lr: 0.003488, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 02:50:05 2021-05-06 19:50:52 [INFO] [TRAIN] epoch: 75, iter: 27860/40000, loss: 0.3807, lr: 0.003485, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 02:50:04 2021-05-06 19:51:01 [INFO] [TRAIN] epoch: 75, iter: 27870/40000, loss: 0.3359, lr: 0.003483, batch_cost: 0.8423, reader_cost: 0.00015, ips: 1.1872 samples/sec | ETA 02:50:17 2021-05-06 19:51:09 [INFO] [TRAIN] epoch: 75, iter: 27880/40000, loss: 0.3221, lr: 0.003480, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 02:49:37 2021-05-06 19:51:17 [INFO] [TRAIN] epoch: 75, iter: 27890/40000, loss: 0.3468, lr: 0.003478, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 02:49:32 2021-05-06 19:51:26 [INFO] [TRAIN] epoch: 75, iter: 27900/40000, loss: 0.3302, lr: 0.003475, batch_cost: 0.8389, reader_cost: 0.00011, ips: 1.1921 samples/sec | ETA 02:49:10 2021-05-06 19:51:37 [INFO] [TRAIN] epoch: 76, iter: 27910/40000, loss: 0.5172, lr: 0.003473, batch_cost: 1.1300, reader_cost: 0.21204, ips: 0.8849 samples/sec | ETA 03:47:42 2021-05-06 19:51:45 [INFO] [TRAIN] epoch: 76, iter: 27920/40000, loss: 0.3914, lr: 0.003470, batch_cost: 0.8414, reader_cost: 0.00031, ips: 1.1884 samples/sec | ETA 02:49:24 2021-05-06 19:51:54 [INFO] [TRAIN] epoch: 76, iter: 27930/40000, loss: 0.4920, lr: 0.003468, batch_cost: 0.8410, reader_cost: 0.00014, ips: 1.1890 samples/sec | ETA 02:49:11 2021-05-06 19:52:02 [INFO] [TRAIN] epoch: 76, iter: 27940/40000, loss: 0.2558, lr: 0.003465, batch_cost: 0.8408, reader_cost: 0.00014, ips: 1.1894 samples/sec | ETA 02:48:59 2021-05-06 19:52:11 [INFO] [TRAIN] epoch: 76, iter: 27950/40000, loss: 0.3977, lr: 0.003463, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 02:48:40 2021-05-06 19:52:19 [INFO] [TRAIN] epoch: 76, iter: 27960/40000, loss: 0.6372, lr: 0.003460, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1879 samples/sec | ETA 02:48:55 2021-05-06 19:52:27 [INFO] [TRAIN] epoch: 76, iter: 27970/40000, loss: 0.4253, lr: 0.003458, batch_cost: 0.8422, reader_cost: 0.00015, ips: 1.1874 samples/sec | ETA 02:48:51 2021-05-06 19:52:36 [INFO] [TRAIN] epoch: 76, iter: 27980/40000, loss: 0.4079, lr: 0.003455, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 02:48:25 2021-05-06 19:52:44 [INFO] [TRAIN] epoch: 76, iter: 27990/40000, loss: 0.3158, lr: 0.003453, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1887 samples/sec | ETA 02:48:23 2021-05-06 19:52:53 [INFO] [TRAIN] epoch: 76, iter: 28000/40000, loss: 0.1122, lr: 0.003450, batch_cost: 0.8386, reader_cost: 0.00016, ips: 1.1924 samples/sec | ETA 02:47:43 2021-05-06 19:52:53 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 19:56:30 [INFO] [EVAL] #Images: 500 mIoU: 0.7621 Acc: 0.9573 Kappa: 0.9446 2021-05-06 19:56:30 [INFO] [EVAL] Class IoU: [0.9804 0.843 0.9208 0.6567 0.5978 0.5412 0.6074 0.7261 0.9193 0.6443 0.9452 0.784 0.5708 0.9403 0.8024 0.8958 0.8086 0.5886 0.7074] 2021-05-06 19:56:30 [INFO] [EVAL] Class Acc: [0.9904 0.9218 0.9524 0.8593 0.7245 0.7716 0.8061 0.8891 0.9515 0.876 0.9712 0.8718 0.7207 0.965 0.9195 0.9594 0.9112 0.8818 0.7836] 2021-05-06 19:57:20 [INFO] [EVAL] The model with the best validation mIoU (0.7621) was saved at iter 28000. 2021-05-06 19:57:28 [INFO] [TRAIN] epoch: 76, iter: 28010/40000, loss: 0.2934, lr: 0.003448, batch_cost: 0.8371, reader_cost: 0.00029, ips: 1.1945 samples/sec | ETA 02:47:17 2021-05-06 19:57:37 [INFO] [TRAIN] epoch: 76, iter: 28020/40000, loss: 0.4432, lr: 0.003445, batch_cost: 0.8386, reader_cost: 0.00015, ips: 1.1925 samples/sec | ETA 02:47:26 2021-05-06 19:57:45 [INFO] [TRAIN] epoch: 76, iter: 28030/40000, loss: 0.4056, lr: 0.003443, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 02:47:33 2021-05-06 19:57:53 [INFO] [TRAIN] epoch: 76, iter: 28040/40000, loss: 0.5727, lr: 0.003440, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 02:47:24 2021-05-06 19:58:02 [INFO] [TRAIN] epoch: 76, iter: 28050/40000, loss: 0.3726, lr: 0.003438, batch_cost: 0.8423, reader_cost: 0.00015, ips: 1.1873 samples/sec | ETA 02:47:44 2021-05-06 19:58:10 [INFO] [TRAIN] epoch: 76, iter: 28060/40000, loss: 0.3311, lr: 0.003435, batch_cost: 0.8400, reader_cost: 0.00016, ips: 1.1905 samples/sec | ETA 02:47:09 2021-05-06 19:58:19 [INFO] [TRAIN] epoch: 76, iter: 28070/40000, loss: 0.3687, lr: 0.003433, batch_cost: 0.8407, reader_cost: 0.00017, ips: 1.1895 samples/sec | ETA 02:47:09 2021-05-06 19:58:27 [INFO] [TRAIN] epoch: 76, iter: 28080/40000, loss: 0.4213, lr: 0.003430, batch_cost: 0.8400, reader_cost: 0.00017, ips: 1.1904 samples/sec | ETA 02:46:53 2021-05-06 19:58:35 [INFO] [TRAIN] epoch: 76, iter: 28090/40000, loss: 0.5108, lr: 0.003428, batch_cost: 0.8412, reader_cost: 0.00017, ips: 1.1888 samples/sec | ETA 02:46:58 2021-05-06 19:58:44 [INFO] [TRAIN] epoch: 76, iter: 28100/40000, loss: 0.3639, lr: 0.003425, batch_cost: 0.8414, reader_cost: 0.00016, ips: 1.1884 samples/sec | ETA 02:46:53 2021-05-06 19:58:52 [INFO] [TRAIN] epoch: 76, iter: 28110/40000, loss: 0.4061, lr: 0.003423, batch_cost: 0.8397, reader_cost: 0.00014, ips: 1.1908 samples/sec | ETA 02:46:24 2021-05-06 19:59:01 [INFO] [TRAIN] epoch: 76, iter: 28120/40000, loss: 0.3198, lr: 0.003420, batch_cost: 0.8413, reader_cost: 0.00014, ips: 1.1887 samples/sec | ETA 02:46:34 2021-05-06 19:59:09 [INFO] [TRAIN] epoch: 76, iter: 28130/40000, loss: 0.2381, lr: 0.003418, batch_cost: 0.8400, reader_cost: 0.00014, ips: 1.1905 samples/sec | ETA 02:46:10 2021-05-06 19:59:18 [INFO] [TRAIN] epoch: 76, iter: 28140/40000, loss: 0.3180, lr: 0.003415, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 02:46:01 2021-05-06 19:59:26 [INFO] [TRAIN] epoch: 76, iter: 28150/40000, loss: 0.2500, lr: 0.003413, batch_cost: 0.8400, reader_cost: 0.00016, ips: 1.1905 samples/sec | ETA 02:45:53 2021-05-06 19:59:34 [INFO] [TRAIN] epoch: 76, iter: 28160/40000, loss: 0.3102, lr: 0.003410, batch_cost: 0.8380, reader_cost: 0.00015, ips: 1.1933 samples/sec | ETA 02:45:21 2021-05-06 19:59:43 [INFO] [TRAIN] epoch: 76, iter: 28170/40000, loss: 0.2836, lr: 0.003408, batch_cost: 0.8399, reader_cost: 0.00014, ips: 1.1906 samples/sec | ETA 02:45:35 2021-05-06 19:59:51 [INFO] [TRAIN] epoch: 76, iter: 28180/40000, loss: 0.3134, lr: 0.003405, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 02:45:30 2021-05-06 20:00:00 [INFO] [TRAIN] epoch: 76, iter: 28190/40000, loss: 0.2886, lr: 0.003402, batch_cost: 0.8403, reader_cost: 0.00014, ips: 1.1900 samples/sec | ETA 02:45:24 2021-05-06 20:00:08 [INFO] [TRAIN] epoch: 76, iter: 28200/40000, loss: 0.4360, lr: 0.003400, batch_cost: 0.8399, reader_cost: 0.00014, ips: 1.1906 samples/sec | ETA 02:45:10 2021-05-06 20:00:16 [INFO] [TRAIN] epoch: 76, iter: 28210/40000, loss: 0.2709, lr: 0.003397, batch_cost: 0.8389, reader_cost: 0.00015, ips: 1.1920 samples/sec | ETA 02:44:51 2021-05-06 20:00:25 [INFO] [TRAIN] epoch: 76, iter: 28220/40000, loss: 0.3225, lr: 0.003395, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 02:45:03 2021-05-06 20:00:33 [INFO] [TRAIN] epoch: 76, iter: 28230/40000, loss: 0.3472, lr: 0.003392, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 02:44:53 2021-05-06 20:00:41 [INFO] [TRAIN] epoch: 76, iter: 28240/40000, loss: 0.3217, lr: 0.003390, batch_cost: 0.8382, reader_cost: 0.00014, ips: 1.1930 samples/sec | ETA 02:44:17 2021-05-06 20:00:50 [INFO] [TRAIN] epoch: 76, iter: 28250/40000, loss: 0.3195, lr: 0.003387, batch_cost: 0.8410, reader_cost: 0.00014, ips: 1.1890 samples/sec | ETA 02:44:42 2021-05-06 20:00:58 [INFO] [TRAIN] epoch: 76, iter: 28260/40000, loss: 0.3410, lr: 0.003385, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 02:44:24 2021-05-06 20:01:07 [INFO] [TRAIN] epoch: 76, iter: 28270/40000, loss: 0.3170, lr: 0.003382, batch_cost: 0.8398, reader_cost: 0.00029, ips: 1.1908 samples/sec | ETA 02:44:10 2021-05-06 20:01:19 [INFO] [TRAIN] epoch: 77, iter: 28280/40000, loss: 0.4716, lr: 0.003380, batch_cost: 1.1874, reader_cost: 0.20503, ips: 0.8422 samples/sec | ETA 03:51:55 2021-05-06 20:01:27 [INFO] [TRAIN] epoch: 77, iter: 28290/40000, loss: 0.2867, lr: 0.003377, batch_cost: 0.8422, reader_cost: 0.00034, ips: 1.1873 samples/sec | ETA 02:44:22 2021-05-06 20:01:35 [INFO] [TRAIN] epoch: 77, iter: 28300/40000, loss: 0.4516, lr: 0.003375, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 02:43:57 2021-05-06 20:01:44 [INFO] [TRAIN] epoch: 77, iter: 28310/40000, loss: 0.2445, lr: 0.003372, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1882 samples/sec | ETA 02:43:58 2021-05-06 20:01:52 [INFO] [TRAIN] epoch: 77, iter: 28320/40000, loss: 0.4516, lr: 0.003370, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1911 samples/sec | ETA 02:43:26 2021-05-06 20:02:01 [INFO] [TRAIN] epoch: 77, iter: 28330/40000, loss: 0.5531, lr: 0.003367, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1881 samples/sec | ETA 02:43:42 2021-05-06 20:02:09 [INFO] [TRAIN] epoch: 77, iter: 28340/40000, loss: 0.3553, lr: 0.003365, batch_cost: 0.8407, reader_cost: 0.00016, ips: 1.1895 samples/sec | ETA 02:43:22 2021-05-06 20:02:17 [INFO] [TRAIN] epoch: 77, iter: 28350/40000, loss: 0.5967, lr: 0.003362, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1912 samples/sec | ETA 02:43:00 2021-05-06 20:02:26 [INFO] [TRAIN] epoch: 77, iter: 28360/40000, loss: 0.3184, lr: 0.003360, batch_cost: 0.8415, reader_cost: 0.00016, ips: 1.1884 samples/sec | ETA 02:43:14 2021-05-06 20:02:34 [INFO] [TRAIN] epoch: 77, iter: 28370/40000, loss: 0.2443, lr: 0.003357, batch_cost: 0.8404, reader_cost: 0.00014, ips: 1.1898 samples/sec | ETA 02:42:54 2021-05-06 20:02:43 [INFO] [TRAIN] epoch: 77, iter: 28380/40000, loss: 0.4495, lr: 0.003355, batch_cost: 0.8397, reader_cost: 0.00014, ips: 1.1909 samples/sec | ETA 02:42:37 2021-05-06 20:02:51 [INFO] [TRAIN] epoch: 77, iter: 28390/40000, loss: 0.3345, lr: 0.003352, batch_cost: 0.8388, reader_cost: 0.00015, ips: 1.1922 samples/sec | ETA 02:42:18 2021-05-06 20:02:59 [INFO] [TRAIN] epoch: 77, iter: 28400/40000, loss: 0.3470, lr: 0.003350, batch_cost: 0.8389, reader_cost: 0.00015, ips: 1.1920 samples/sec | ETA 02:42:11 2021-05-06 20:03:08 [INFO] [TRAIN] epoch: 77, iter: 28410/40000, loss: 0.5834, lr: 0.003347, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 02:42:20 2021-05-06 20:03:16 [INFO] [TRAIN] epoch: 77, iter: 28420/40000, loss: 0.3434, lr: 0.003345, batch_cost: 0.8410, reader_cost: 0.00014, ips: 1.1891 samples/sec | ETA 02:42:18 2021-05-06 20:03:25 [INFO] [TRAIN] epoch: 77, iter: 28430/40000, loss: 0.2617, lr: 0.003342, batch_cost: 0.8420, reader_cost: 0.00015, ips: 1.1877 samples/sec | ETA 02:42:21 2021-05-06 20:03:33 [INFO] [TRAIN] epoch: 77, iter: 28440/40000, loss: 0.2611, lr: 0.003339, batch_cost: 0.8395, reader_cost: 0.00016, ips: 1.1911 samples/sec | ETA 02:41:45 2021-05-06 20:03:41 [INFO] [TRAIN] epoch: 77, iter: 28450/40000, loss: 0.2540, lr: 0.003337, batch_cost: 0.8402, reader_cost: 0.00017, ips: 1.1902 samples/sec | ETA 02:41:44 2021-05-06 20:03:50 [INFO] [TRAIN] epoch: 77, iter: 28460/40000, loss: 0.2183, lr: 0.003334, batch_cost: 0.8395, reader_cost: 0.00016, ips: 1.1911 samples/sec | ETA 02:41:28 2021-05-06 20:03:58 [INFO] [TRAIN] epoch: 77, iter: 28470/40000, loss: 0.5369, lr: 0.003332, batch_cost: 0.8397, reader_cost: 0.00016, ips: 1.1908 samples/sec | ETA 02:41:22 2021-05-06 20:04:07 [INFO] [TRAIN] epoch: 77, iter: 28480/40000, loss: 0.4706, lr: 0.003329, batch_cost: 0.8398, reader_cost: 0.00016, ips: 1.1908 samples/sec | ETA 02:41:14 2021-05-06 20:04:15 [INFO] [TRAIN] epoch: 77, iter: 28490/40000, loss: 0.3233, lr: 0.003327, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 02:41:19 2021-05-06 20:04:23 [INFO] [TRAIN] epoch: 77, iter: 28500/40000, loss: 0.2365, lr: 0.003324, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 02:41:11 2021-05-06 20:04:32 [INFO] [TRAIN] epoch: 77, iter: 28510/40000, loss: 0.3292, lr: 0.003322, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 02:40:57 2021-05-06 20:04:40 [INFO] [TRAIN] epoch: 77, iter: 28520/40000, loss: 0.2402, lr: 0.003319, batch_cost: 0.8394, reader_cost: 0.00016, ips: 1.1913 samples/sec | ETA 02:40:36 2021-05-06 20:04:49 [INFO] [TRAIN] epoch: 77, iter: 28530/40000, loss: 0.3829, lr: 0.003317, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1917 samples/sec | ETA 02:40:24 2021-05-06 20:04:57 [INFO] [TRAIN] epoch: 77, iter: 28540/40000, loss: 0.3033, lr: 0.003314, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 02:40:38 2021-05-06 20:05:05 [INFO] [TRAIN] epoch: 77, iter: 28550/40000, loss: 0.2849, lr: 0.003312, batch_cost: 0.8387, reader_cost: 0.00015, ips: 1.1923 samples/sec | ETA 02:40:03 2021-05-06 20:05:14 [INFO] [TRAIN] epoch: 77, iter: 28560/40000, loss: 0.3200, lr: 0.003309, batch_cost: 0.8383, reader_cost: 0.00014, ips: 1.1929 samples/sec | ETA 02:39:50 2021-05-06 20:05:22 [INFO] [TRAIN] epoch: 77, iter: 28570/40000, loss: 0.3741, lr: 0.003307, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 02:39:58 2021-05-06 20:05:31 [INFO] [TRAIN] epoch: 77, iter: 28580/40000, loss: 0.2294, lr: 0.003304, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 02:39:53 2021-05-06 20:05:39 [INFO] [TRAIN] epoch: 77, iter: 28590/40000, loss: 0.3456, lr: 0.003302, batch_cost: 0.8405, reader_cost: 0.00016, ips: 1.1898 samples/sec | ETA 02:39:50 2021-05-06 20:05:47 [INFO] [TRAIN] epoch: 77, iter: 28600/40000, loss: 0.4109, lr: 0.003299, batch_cost: 0.8409, reader_cost: 0.00016, ips: 1.1892 samples/sec | ETA 02:39:46 2021-05-06 20:05:56 [INFO] [TRAIN] epoch: 77, iter: 28610/40000, loss: 0.3067, lr: 0.003297, batch_cost: 0.8413, reader_cost: 0.00016, ips: 1.1887 samples/sec | ETA 02:39:42 2021-05-06 20:06:04 [INFO] [TRAIN] epoch: 77, iter: 28620/40000, loss: 0.4342, lr: 0.003294, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1911 samples/sec | ETA 02:39:14 2021-05-06 20:06:13 [INFO] [TRAIN] epoch: 77, iter: 28630/40000, loss: 0.3313, lr: 0.003292, batch_cost: 0.8394, reader_cost: 0.00016, ips: 1.1914 samples/sec | ETA 02:39:03 2021-05-06 20:06:21 [INFO] [TRAIN] epoch: 77, iter: 28640/40000, loss: 0.2827, lr: 0.003289, batch_cost: 0.8385, reader_cost: 0.00011, ips: 1.1926 samples/sec | ETA 02:38:45 2021-05-06 20:06:33 [INFO] [TRAIN] epoch: 78, iter: 28650/40000, loss: 0.4445, lr: 0.003286, batch_cost: 1.1451, reader_cost: 0.21099, ips: 0.8733 samples/sec | ETA 03:36:37 2021-05-06 20:06:41 [INFO] [TRAIN] epoch: 78, iter: 28660/40000, loss: 0.3233, lr: 0.003284, batch_cost: 0.8407, reader_cost: 0.00034, ips: 1.1895 samples/sec | ETA 02:38:53 2021-05-06 20:06:49 [INFO] [TRAIN] epoch: 78, iter: 28670/40000, loss: 0.3659, lr: 0.003281, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1912 samples/sec | ETA 02:38:31 2021-05-06 20:06:58 [INFO] [TRAIN] epoch: 78, iter: 28680/40000, loss: 0.3708, lr: 0.003279, batch_cost: 0.8392, reader_cost: 0.00015, ips: 1.1916 samples/sec | ETA 02:38:19 2021-05-06 20:07:06 [INFO] [TRAIN] epoch: 78, iter: 28690/40000, loss: 0.3877, lr: 0.003276, batch_cost: 0.8406, reader_cost: 0.00016, ips: 1.1897 samples/sec | ETA 02:38:26 2021-05-06 20:07:15 [INFO] [TRAIN] epoch: 78, iter: 28700/40000, loss: 0.3146, lr: 0.003274, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1888 samples/sec | ETA 02:38:25 2021-05-06 20:07:23 [INFO] [TRAIN] epoch: 78, iter: 28710/40000, loss: 0.2611, lr: 0.003271, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 02:38:12 2021-05-06 20:07:31 [INFO] [TRAIN] epoch: 78, iter: 28720/40000, loss: 0.5224, lr: 0.003269, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 02:38:07 2021-05-06 20:07:40 [INFO] [TRAIN] epoch: 78, iter: 28730/40000, loss: 0.4555, lr: 0.003266, batch_cost: 0.8427, reader_cost: 0.00016, ips: 1.1866 samples/sec | ETA 02:38:17 2021-05-06 20:07:48 [INFO] [TRAIN] epoch: 78, iter: 28740/40000, loss: 0.1744, lr: 0.003264, batch_cost: 0.8409, reader_cost: 0.00017, ips: 1.1892 samples/sec | ETA 02:37:48 2021-05-06 20:07:57 [INFO] [TRAIN] epoch: 78, iter: 28750/40000, loss: 0.2267, lr: 0.003261, batch_cost: 0.8409, reader_cost: 0.00016, ips: 1.1893 samples/sec | ETA 02:37:39 2021-05-06 20:08:05 [INFO] [TRAIN] epoch: 78, iter: 28760/40000, loss: 0.3332, lr: 0.003259, batch_cost: 0.8412, reader_cost: 0.00017, ips: 1.1888 samples/sec | ETA 02:37:35 2021-05-06 20:08:13 [INFO] [TRAIN] epoch: 78, iter: 28770/40000, loss: 0.7836, lr: 0.003256, batch_cost: 0.8420, reader_cost: 0.00015, ips: 1.1876 samples/sec | ETA 02:37:35 2021-05-06 20:08:22 [INFO] [TRAIN] epoch: 78, iter: 28780/40000, loss: 0.5253, lr: 0.003254, batch_cost: 0.8388, reader_cost: 0.00015, ips: 1.1922 samples/sec | ETA 02:36:50 2021-05-06 20:08:30 [INFO] [TRAIN] epoch: 78, iter: 28790/40000, loss: 0.4045, lr: 0.003251, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 02:36:57 2021-05-06 20:08:39 [INFO] [TRAIN] epoch: 78, iter: 28800/40000, loss: 0.2750, lr: 0.003249, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1884 samples/sec | ETA 02:37:04 2021-05-06 20:08:47 [INFO] [TRAIN] epoch: 78, iter: 28810/40000, loss: 0.3846, lr: 0.003246, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 02:36:51 2021-05-06 20:08:55 [INFO] [TRAIN] epoch: 78, iter: 28820/40000, loss: 0.2580, lr: 0.003244, batch_cost: 0.8432, reader_cost: 0.00016, ips: 1.1860 samples/sec | ETA 02:37:06 2021-05-06 20:09:04 [INFO] [TRAIN] epoch: 78, iter: 28830/40000, loss: 0.3092, lr: 0.003241, batch_cost: 0.8422, reader_cost: 0.00015, ips: 1.1873 samples/sec | ETA 02:36:47 2021-05-06 20:09:12 [INFO] [TRAIN] epoch: 78, iter: 28840/40000, loss: 0.5587, lr: 0.003238, batch_cost: 0.8408, reader_cost: 0.00014, ips: 1.1893 samples/sec | ETA 02:36:23 2021-05-06 20:09:21 [INFO] [TRAIN] epoch: 78, iter: 28850/40000, loss: 0.4120, lr: 0.003236, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 02:36:06 2021-05-06 20:09:29 [INFO] [TRAIN] epoch: 78, iter: 28860/40000, loss: 0.3367, lr: 0.003233, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 02:36:08 2021-05-06 20:09:38 [INFO] [TRAIN] epoch: 78, iter: 28870/40000, loss: 0.1892, lr: 0.003231, batch_cost: 0.8420, reader_cost: 0.00016, ips: 1.1877 samples/sec | ETA 02:36:11 2021-05-06 20:09:46 [INFO] [TRAIN] epoch: 78, iter: 28880/40000, loss: 0.2724, lr: 0.003228, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 02:35:45 2021-05-06 20:09:54 [INFO] [TRAIN] epoch: 78, iter: 28890/40000, loss: 0.1903, lr: 0.003226, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1888 samples/sec | ETA 02:35:45 2021-05-06 20:10:03 [INFO] [TRAIN] epoch: 78, iter: 28900/40000, loss: 0.2934, lr: 0.003223, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 02:35:36 2021-05-06 20:10:11 [INFO] [TRAIN] epoch: 78, iter: 28910/40000, loss: 0.2633, lr: 0.003221, batch_cost: 0.8422, reader_cost: 0.00015, ips: 1.1873 samples/sec | ETA 02:35:40 2021-05-06 20:10:20 [INFO] [TRAIN] epoch: 78, iter: 28920/40000, loss: 0.6713, lr: 0.003218, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 02:35:12 2021-05-06 20:10:28 [INFO] [TRAIN] epoch: 78, iter: 28930/40000, loss: 0.3908, lr: 0.003216, batch_cost: 0.8416, reader_cost: 0.00016, ips: 1.1883 samples/sec | ETA 02:35:15 2021-05-06 20:10:36 [INFO] [TRAIN] epoch: 78, iter: 28940/40000, loss: 0.4781, lr: 0.003213, batch_cost: 0.8396, reader_cost: 0.00014, ips: 1.1911 samples/sec | ETA 02:34:45 2021-05-06 20:10:45 [INFO] [TRAIN] epoch: 78, iter: 28950/40000, loss: 0.2342, lr: 0.003211, batch_cost: 0.8407, reader_cost: 0.00014, ips: 1.1895 samples/sec | ETA 02:34:49 2021-05-06 20:10:53 [INFO] [TRAIN] epoch: 78, iter: 28960/40000, loss: 0.3018, lr: 0.003208, batch_cost: 0.8409, reader_cost: 0.00014, ips: 1.1891 samples/sec | ETA 02:34:43 2021-05-06 20:11:02 [INFO] [TRAIN] epoch: 78, iter: 28970/40000, loss: 0.4054, lr: 0.003206, batch_cost: 0.8413, reader_cost: 0.00014, ips: 1.1887 samples/sec | ETA 02:34:39 2021-05-06 20:11:10 [INFO] [TRAIN] epoch: 78, iter: 28980/40000, loss: 0.3203, lr: 0.003203, batch_cost: 0.8418, reader_cost: 0.00014, ips: 1.1879 samples/sec | ETA 02:34:36 2021-05-06 20:11:18 [INFO] [TRAIN] epoch: 78, iter: 28990/40000, loss: 0.4243, lr: 0.003200, batch_cost: 0.8413, reader_cost: 0.00014, ips: 1.1887 samples/sec | ETA 02:34:22 2021-05-06 20:11:27 [INFO] [TRAIN] epoch: 78, iter: 29000/40000, loss: 0.2648, lr: 0.003198, batch_cost: 0.8418, reader_cost: 0.00014, ips: 1.1879 samples/sec | ETA 02:34:20 2021-05-06 20:11:27 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 20:15:05 [INFO] [EVAL] #Images: 500 mIoU: 0.7451 Acc: 0.9562 Kappa: 0.9431 2021-05-06 20:15:05 [INFO] [EVAL] Class IoU: [0.9804 0.8457 0.9171 0.6517 0.5756 0.5294 0.6013 0.7346 0.9163 0.6524 0.9418 0.7623 0.47 0.9339 0.7713 0.8887 0.7692 0.5042 0.7107] 2021-05-06 20:15:05 [INFO] [EVAL] Class Acc: [0.9916 0.9152 0.9468 0.8799 0.8711 0.8143 0.8316 0.8857 0.9491 0.7677 0.9634 0.8361 0.776 0.9542 0.817 0.9415 0.9384 0.9099 0.8735] 2021-05-06 20:15:34 [INFO] [EVAL] The model with the best validation mIoU (0.7621) was saved at iter 28000. 2021-05-06 20:15:42 [INFO] [TRAIN] epoch: 78, iter: 29010/40000, loss: 0.3628, lr: 0.003195, batch_cost: 0.8362, reader_cost: 0.00028, ips: 1.1959 samples/sec | ETA 02:33:10 2021-05-06 20:15:54 [INFO] [TRAIN] epoch: 79, iter: 29020/40000, loss: 0.6227, lr: 0.003193, batch_cost: 1.1897, reader_cost: 0.20970, ips: 0.8406 samples/sec | ETA 03:37:42 2021-05-06 20:16:02 [INFO] [TRAIN] epoch: 79, iter: 29030/40000, loss: 0.3537, lr: 0.003190, batch_cost: 0.8476, reader_cost: 0.00032, ips: 1.1798 samples/sec | ETA 02:34:58 2021-05-06 20:16:11 [INFO] [TRAIN] epoch: 79, iter: 29040/40000, loss: 0.3820, lr: 0.003188, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1914 samples/sec | ETA 02:33:19 2021-05-06 20:16:19 [INFO] [TRAIN] epoch: 79, iter: 29050/40000, loss: 0.3390, lr: 0.003185, batch_cost: 0.8394, reader_cost: 0.00014, ips: 1.1913 samples/sec | ETA 02:33:11 2021-05-06 20:16:28 [INFO] [TRAIN] epoch: 79, iter: 29060/40000, loss: 0.2733, lr: 0.003183, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 02:33:10 2021-05-06 20:16:36 [INFO] [TRAIN] epoch: 79, iter: 29070/40000, loss: 0.4039, lr: 0.003180, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1880 samples/sec | ETA 02:33:20 2021-05-06 20:16:44 [INFO] [TRAIN] epoch: 79, iter: 29080/40000, loss: 0.5072, lr: 0.003178, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 02:32:49 2021-05-06 20:16:53 [INFO] [TRAIN] epoch: 79, iter: 29090/40000, loss: 0.5344, lr: 0.003175, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 02:32:47 2021-05-06 20:17:01 [INFO] [TRAIN] epoch: 79, iter: 29100/40000, loss: 0.3981, lr: 0.003173, batch_cost: 0.8398, reader_cost: 0.00014, ips: 1.1908 samples/sec | ETA 02:32:33 2021-05-06 20:17:10 [INFO] [TRAIN] epoch: 79, iter: 29110/40000, loss: 0.2580, lr: 0.003170, batch_cost: 0.8418, reader_cost: 0.00014, ips: 1.1879 samples/sec | ETA 02:32:47 2021-05-06 20:17:18 [INFO] [TRAIN] epoch: 79, iter: 29120/40000, loss: 0.1846, lr: 0.003167, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 02:32:22 2021-05-06 20:17:27 [INFO] [TRAIN] epoch: 79, iter: 29130/40000, loss: 0.2985, lr: 0.003165, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 02:32:17 2021-05-06 20:17:35 [INFO] [TRAIN] epoch: 79, iter: 29140/40000, loss: 0.3355, lr: 0.003162, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1882 samples/sec | ETA 02:32:19 2021-05-06 20:17:43 [INFO] [TRAIN] epoch: 79, iter: 29150/40000, loss: 0.4936, lr: 0.003160, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1913 samples/sec | ETA 02:31:47 2021-05-06 20:17:52 [INFO] [TRAIN] epoch: 79, iter: 29160/40000, loss: 0.4450, lr: 0.003157, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 02:31:48 2021-05-06 20:18:00 [INFO] [TRAIN] epoch: 79, iter: 29170/40000, loss: 0.2562, lr: 0.003155, batch_cost: 0.8381, reader_cost: 0.00014, ips: 1.1932 samples/sec | ETA 02:31:16 2021-05-06 20:18:09 [INFO] [TRAIN] epoch: 79, iter: 29180/40000, loss: 0.2763, lr: 0.003152, batch_cost: 0.8420, reader_cost: 0.00015, ips: 1.1877 samples/sec | ETA 02:31:50 2021-05-06 20:18:17 [INFO] [TRAIN] epoch: 79, iter: 29190/40000, loss: 0.3353, lr: 0.003150, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 02:31:29 2021-05-06 20:18:25 [INFO] [TRAIN] epoch: 79, iter: 29200/40000, loss: 0.6468, lr: 0.003147, batch_cost: 0.8388, reader_cost: 0.00014, ips: 1.1921 samples/sec | ETA 02:30:59 2021-05-06 20:18:34 [INFO] [TRAIN] epoch: 79, iter: 29210/40000, loss: 0.5952, lr: 0.003145, batch_cost: 0.8392, reader_cost: 0.00015, ips: 1.1917 samples/sec | ETA 02:30:54 2021-05-06 20:18:42 [INFO] [TRAIN] epoch: 79, iter: 29220/40000, loss: 0.3417, lr: 0.003142, batch_cost: 0.8417, reader_cost: 0.00016, ips: 1.1881 samples/sec | ETA 02:31:13 2021-05-06 20:18:51 [INFO] [TRAIN] epoch: 79, iter: 29230/40000, loss: 0.3721, lr: 0.003140, batch_cost: 0.8417, reader_cost: 0.00016, ips: 1.1880 samples/sec | ETA 02:31:05 2021-05-06 20:18:59 [INFO] [TRAIN] epoch: 79, iter: 29240/40000, loss: 0.2183, lr: 0.003137, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 02:30:39 2021-05-06 20:19:07 [INFO] [TRAIN] epoch: 79, iter: 29250/40000, loss: 0.2387, lr: 0.003134, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1903 samples/sec | ETA 02:30:31 2021-05-06 20:19:16 [INFO] [TRAIN] epoch: 79, iter: 29260/40000, loss: 0.2510, lr: 0.003132, batch_cost: 0.8406, reader_cost: 0.00014, ips: 1.1897 samples/sec | ETA 02:30:27 2021-05-06 20:19:24 [INFO] [TRAIN] epoch: 79, iter: 29270/40000, loss: 0.2281, lr: 0.003129, batch_cost: 0.8394, reader_cost: 0.00014, ips: 1.1914 samples/sec | ETA 02:30:06 2021-05-06 20:19:33 [INFO] [TRAIN] epoch: 79, iter: 29280/40000, loss: 0.2873, lr: 0.003127, batch_cost: 0.8398, reader_cost: 0.00014, ips: 1.1908 samples/sec | ETA 02:30:02 2021-05-06 20:19:41 [INFO] [TRAIN] epoch: 79, iter: 29290/40000, loss: 0.2604, lr: 0.003124, batch_cost: 0.8400, reader_cost: 0.00014, ips: 1.1905 samples/sec | ETA 02:29:56 2021-05-06 20:19:49 [INFO] [TRAIN] epoch: 79, iter: 29300/40000, loss: 0.3520, lr: 0.003122, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1886 samples/sec | ETA 02:30:02 2021-05-06 20:19:58 [INFO] [TRAIN] epoch: 79, iter: 29310/40000, loss: 0.3633, lr: 0.003119, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 02:29:48 2021-05-06 20:20:06 [INFO] [TRAIN] epoch: 79, iter: 29320/40000, loss: 0.2671, lr: 0.003117, batch_cost: 0.8403, reader_cost: 0.00016, ips: 1.1900 samples/sec | ETA 02:29:34 2021-05-06 20:20:15 [INFO] [TRAIN] epoch: 79, iter: 29330/40000, loss: 0.3215, lr: 0.003114, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 02:29:27 2021-05-06 20:20:23 [INFO] [TRAIN] epoch: 79, iter: 29340/40000, loss: 0.4891, lr: 0.003112, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1880 samples/sec | ETA 02:29:32 2021-05-06 20:20:31 [INFO] [TRAIN] epoch: 79, iter: 29350/40000, loss: 0.2662, lr: 0.003109, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 02:29:04 2021-05-06 20:20:40 [INFO] [TRAIN] epoch: 79, iter: 29360/40000, loss: 0.3507, lr: 0.003107, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 02:29:01 2021-05-06 20:20:48 [INFO] [TRAIN] epoch: 79, iter: 29370/40000, loss: 0.4134, lr: 0.003104, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 02:29:00 2021-05-06 20:20:57 [INFO] [TRAIN] epoch: 79, iter: 29380/40000, loss: 0.3171, lr: 0.003101, batch_cost: 0.8399, reader_cost: 0.00013, ips: 1.1906 samples/sec | ETA 02:28:39 2021-05-06 20:21:08 [INFO] [TRAIN] epoch: 80, iter: 29390/40000, loss: 0.4223, lr: 0.003099, batch_cost: 1.1476, reader_cost: 0.21714, ips: 0.8714 samples/sec | ETA 03:22:56 2021-05-06 20:21:17 [INFO] [TRAIN] epoch: 80, iter: 29400/40000, loss: 0.3600, lr: 0.003096, batch_cost: 0.8544, reader_cost: 0.00032, ips: 1.1705 samples/sec | ETA 02:30:56 2021-05-06 20:21:25 [INFO] [TRAIN] epoch: 80, iter: 29410/40000, loss: 0.7050, lr: 0.003094, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1886 samples/sec | ETA 02:28:29 2021-05-06 20:21:33 [INFO] [TRAIN] epoch: 80, iter: 29420/40000, loss: 0.2721, lr: 0.003091, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 02:28:06 2021-05-06 20:21:42 [INFO] [TRAIN] epoch: 80, iter: 29430/40000, loss: 0.2630, lr: 0.003089, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 02:28:10 2021-05-06 20:21:50 [INFO] [TRAIN] epoch: 80, iter: 29440/40000, loss: 0.5055, lr: 0.003086, batch_cost: 0.8435, reader_cost: 0.00016, ips: 1.1856 samples/sec | ETA 02:28:27 2021-05-06 20:21:59 [INFO] [TRAIN] epoch: 80, iter: 29450/40000, loss: 0.4331, lr: 0.003084, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 02:27:49 2021-05-06 20:22:07 [INFO] [TRAIN] epoch: 80, iter: 29460/40000, loss: 0.4289, lr: 0.003081, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1884 samples/sec | ETA 02:27:48 2021-05-06 20:22:16 [INFO] [TRAIN] epoch: 80, iter: 29470/40000, loss: 0.5099, lr: 0.003079, batch_cost: 0.8394, reader_cost: 0.00014, ips: 1.1913 samples/sec | ETA 02:27:18 2021-05-06 20:22:24 [INFO] [TRAIN] epoch: 80, iter: 29480/40000, loss: 0.2867, lr: 0.003076, batch_cost: 0.8414, reader_cost: 0.00016, ips: 1.1885 samples/sec | ETA 02:27:31 2021-05-06 20:22:32 [INFO] [TRAIN] epoch: 80, iter: 29490/40000, loss: 0.2448, lr: 0.003073, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 02:27:07 2021-05-06 20:22:41 [INFO] [TRAIN] epoch: 80, iter: 29500/40000, loss: 0.2826, lr: 0.003071, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1911 samples/sec | ETA 02:26:55 2021-05-06 20:22:49 [INFO] [TRAIN] epoch: 80, iter: 29510/40000, loss: 0.4028, lr: 0.003068, batch_cost: 0.8412, reader_cost: 0.00014, ips: 1.1887 samples/sec | ETA 02:27:04 2021-05-06 20:22:58 [INFO] [TRAIN] epoch: 80, iter: 29520/40000, loss: 0.5183, lr: 0.003066, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1918 samples/sec | ETA 02:26:33 2021-05-06 20:23:06 [INFO] [TRAIN] epoch: 80, iter: 29530/40000, loss: 0.5834, lr: 0.003063, batch_cost: 0.8394, reader_cost: 0.00016, ips: 1.1913 samples/sec | ETA 02:26:28 2021-05-06 20:23:14 [INFO] [TRAIN] epoch: 80, iter: 29540/40000, loss: 0.3104, lr: 0.003061, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1887 samples/sec | ETA 02:26:39 2021-05-06 20:23:23 [INFO] [TRAIN] epoch: 80, iter: 29550/40000, loss: 0.3114, lr: 0.003058, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1881 samples/sec | ETA 02:26:35 2021-05-06 20:23:31 [INFO] [TRAIN] epoch: 80, iter: 29560/40000, loss: 0.3242, lr: 0.003056, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1913 samples/sec | ETA 02:26:03 2021-05-06 20:23:40 [INFO] [TRAIN] epoch: 80, iter: 29570/40000, loss: 0.4437, lr: 0.003053, batch_cost: 0.8400, reader_cost: 0.00016, ips: 1.1905 samples/sec | ETA 02:26:00 2021-05-06 20:23:48 [INFO] [TRAIN] epoch: 80, iter: 29580/40000, loss: 0.5375, lr: 0.003051, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 02:25:57 2021-05-06 20:23:56 [INFO] [TRAIN] epoch: 80, iter: 29590/40000, loss: 0.4616, lr: 0.003048, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 02:25:43 2021-05-06 20:24:05 [INFO] [TRAIN] epoch: 80, iter: 29600/40000, loss: 0.4053, lr: 0.003045, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 02:25:34 2021-05-06 20:24:13 [INFO] [TRAIN] epoch: 80, iter: 29610/40000, loss: 0.2398, lr: 0.003043, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 02:25:29 2021-05-06 20:24:22 [INFO] [TRAIN] epoch: 80, iter: 29620/40000, loss: 0.2438, lr: 0.003040, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 02:25:28 2021-05-06 20:24:30 [INFO] [TRAIN] epoch: 80, iter: 29630/40000, loss: 0.2784, lr: 0.003038, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 02:25:21 2021-05-06 20:24:38 [INFO] [TRAIN] epoch: 80, iter: 29640/40000, loss: 0.2342, lr: 0.003035, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 02:25:04 2021-05-06 20:24:47 [INFO] [TRAIN] epoch: 80, iter: 29650/40000, loss: 0.2871, lr: 0.003033, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 02:24:53 2021-05-06 20:24:55 [INFO] [TRAIN] epoch: 80, iter: 29660/40000, loss: 0.2745, lr: 0.003030, batch_cost: 0.8411, reader_cost: 0.00014, ips: 1.1889 samples/sec | ETA 02:24:56 2021-05-06 20:25:04 [INFO] [TRAIN] epoch: 80, iter: 29670/40000, loss: 0.2545, lr: 0.003028, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 02:24:30 2021-05-06 20:25:12 [INFO] [TRAIN] epoch: 80, iter: 29680/40000, loss: 0.3863, lr: 0.003025, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 02:24:26 2021-05-06 20:25:20 [INFO] [TRAIN] epoch: 80, iter: 29690/40000, loss: 0.3370, lr: 0.003022, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1913 samples/sec | ETA 02:24:14 2021-05-06 20:25:29 [INFO] [TRAIN] epoch: 80, iter: 29700/40000, loss: 0.2710, lr: 0.003020, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 02:24:23 2021-05-06 20:25:37 [INFO] [TRAIN] epoch: 80, iter: 29710/40000, loss: 0.4191, lr: 0.003017, batch_cost: 0.8411, reader_cost: 0.00016, ips: 1.1889 samples/sec | ETA 02:24:15 2021-05-06 20:25:46 [INFO] [TRAIN] epoch: 80, iter: 29720/40000, loss: 0.3243, lr: 0.003015, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 02:24:00 2021-05-06 20:25:54 [INFO] [TRAIN] epoch: 80, iter: 29730/40000, loss: 0.3318, lr: 0.003012, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 02:23:49 2021-05-06 20:26:02 [INFO] [TRAIN] epoch: 80, iter: 29740/40000, loss: 0.3031, lr: 0.003010, batch_cost: 0.8399, reader_cost: 0.00016, ips: 1.1907 samples/sec | ETA 02:23:37 2021-05-06 20:26:11 [INFO] [TRAIN] epoch: 80, iter: 29750/40000, loss: 0.3330, lr: 0.003007, batch_cost: 0.8387, reader_cost: 0.00015, ips: 1.1923 samples/sec | ETA 02:23:16 2021-05-06 20:26:19 [INFO] [TRAIN] epoch: 80, iter: 29760/40000, loss: 0.3708, lr: 0.003005, batch_cost: 0.8394, reader_cost: 0.00010, ips: 1.1913 samples/sec | ETA 02:23:15 2021-05-06 20:26:31 [INFO] [TRAIN] epoch: 81, iter: 29770/40000, loss: 0.4624, lr: 0.003002, batch_cost: 1.1744, reader_cost: 0.21819, ips: 0.8515 samples/sec | ETA 03:20:13 2021-05-06 20:26:39 [INFO] [TRAIN] epoch: 81, iter: 29780/40000, loss: 0.4298, lr: 0.003000, batch_cost: 0.8407, reader_cost: 0.00031, ips: 1.1894 samples/sec | ETA 02:23:12 2021-05-06 20:26:48 [INFO] [TRAIN] epoch: 81, iter: 29790/40000, loss: 0.3132, lr: 0.002997, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1912 samples/sec | ETA 02:22:50 2021-05-06 20:26:56 [INFO] [TRAIN] epoch: 81, iter: 29800/40000, loss: 0.2704, lr: 0.002994, batch_cost: 0.8397, reader_cost: 0.00016, ips: 1.1909 samples/sec | ETA 02:22:44 2021-05-06 20:27:05 [INFO] [TRAIN] epoch: 81, iter: 29810/40000, loss: 0.2583, lr: 0.002992, batch_cost: 0.8426, reader_cost: 0.00015, ips: 1.1868 samples/sec | ETA 02:23:05 2021-05-06 20:27:13 [INFO] [TRAIN] epoch: 81, iter: 29820/40000, loss: 0.5286, lr: 0.002989, batch_cost: 0.8413, reader_cost: 0.00016, ips: 1.1886 samples/sec | ETA 02:22:44 2021-05-06 20:27:21 [INFO] [TRAIN] epoch: 81, iter: 29830/40000, loss: 0.4187, lr: 0.002987, batch_cost: 0.8402, reader_cost: 0.00016, ips: 1.1902 samples/sec | ETA 02:22:24 2021-05-06 20:27:30 [INFO] [TRAIN] epoch: 81, iter: 29840/40000, loss: 0.5047, lr: 0.002984, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1913 samples/sec | ETA 02:22:08 2021-05-06 20:27:38 [INFO] [TRAIN] epoch: 81, iter: 29850/40000, loss: 0.3900, lr: 0.002982, batch_cost: 0.8423, reader_cost: 0.00017, ips: 1.1872 samples/sec | ETA 02:22:29 2021-05-06 20:27:47 [INFO] [TRAIN] epoch: 81, iter: 29860/40000, loss: 0.1512, lr: 0.002979, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1885 samples/sec | ETA 02:22:11 2021-05-06 20:27:55 [INFO] [TRAIN] epoch: 81, iter: 29870/40000, loss: 0.2899, lr: 0.002977, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 02:21:54 2021-05-06 20:28:03 [INFO] [TRAIN] epoch: 81, iter: 29880/40000, loss: 0.4855, lr: 0.002974, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1914 samples/sec | ETA 02:21:34 2021-05-06 20:28:12 [INFO] [TRAIN] epoch: 81, iter: 29890/40000, loss: 0.3754, lr: 0.002971, batch_cost: 0.8392, reader_cost: 0.00015, ips: 1.1916 samples/sec | ETA 02:21:24 2021-05-06 20:28:20 [INFO] [TRAIN] epoch: 81, iter: 29900/40000, loss: 0.5950, lr: 0.002969, batch_cost: 0.8390, reader_cost: 0.00015, ips: 1.1919 samples/sec | ETA 02:21:14 2021-05-06 20:28:29 [INFO] [TRAIN] epoch: 81, iter: 29910/40000, loss: 0.4320, lr: 0.002966, batch_cost: 0.8402, reader_cost: 0.00016, ips: 1.1902 samples/sec | ETA 02:21:17 2021-05-06 20:28:37 [INFO] [TRAIN] epoch: 81, iter: 29920/40000, loss: 0.3210, lr: 0.002964, batch_cost: 0.8424, reader_cost: 0.00015, ips: 1.1871 samples/sec | ETA 02:21:31 2021-05-06 20:28:45 [INFO] [TRAIN] epoch: 81, iter: 29930/40000, loss: 0.3044, lr: 0.002961, batch_cost: 0.8402, reader_cost: 0.00014, ips: 1.1902 samples/sec | ETA 02:21:01 2021-05-06 20:28:54 [INFO] [TRAIN] epoch: 81, iter: 29940/40000, loss: 0.5626, lr: 0.002959, batch_cost: 0.8409, reader_cost: 0.00014, ips: 1.1892 samples/sec | ETA 02:20:59 2021-05-06 20:29:02 [INFO] [TRAIN] epoch: 81, iter: 29950/40000, loss: 0.5085, lr: 0.002956, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 02:20:50 2021-05-06 20:29:11 [INFO] [TRAIN] epoch: 81, iter: 29960/40000, loss: 0.4193, lr: 0.002954, batch_cost: 0.8397, reader_cost: 0.00014, ips: 1.1909 samples/sec | ETA 02:20:30 2021-05-06 20:29:19 [INFO] [TRAIN] epoch: 81, iter: 29970/40000, loss: 0.4128, lr: 0.002951, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 02:20:32 2021-05-06 20:29:27 [INFO] [TRAIN] epoch: 81, iter: 29980/40000, loss: 0.2694, lr: 0.002948, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 02:20:18 2021-05-06 20:29:36 [INFO] [TRAIN] epoch: 81, iter: 29990/40000, loss: 0.1919, lr: 0.002946, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1884 samples/sec | ETA 02:20:23 2021-05-06 20:29:44 [INFO] [TRAIN] epoch: 81, iter: 30000/40000, loss: 0.3003, lr: 0.002943, batch_cost: 0.8399, reader_cost: 0.00017, ips: 1.1906 samples/sec | ETA 02:19:59 2021-05-06 20:29:44 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 20:33:21 [INFO] [EVAL] #Images: 500 mIoU: 0.7628 Acc: 0.9572 Kappa: 0.9445 2021-05-06 20:33:21 [INFO] [EVAL] Class IoU: [0.9805 0.8446 0.9174 0.6342 0.5925 0.527 0.6144 0.7308 0.9192 0.6556 0.9446 0.785 0.5757 0.942 0.7851 0.8899 0.7932 0.6345 0.7266] 2021-05-06 20:33:21 [INFO] [EVAL] Class Acc: [0.9921 0.8988 0.9445 0.8929 0.7926 0.8115 0.7742 0.8913 0.9536 0.8553 0.9739 0.8651 0.7681 0.9678 0.8772 0.94 0.9044 0.7903 0.8402] 2021-05-06 20:34:11 [INFO] [EVAL] The model with the best validation mIoU (0.7628) was saved at iter 30000. 2021-05-06 20:34:19 [INFO] [TRAIN] epoch: 81, iter: 30010/40000, loss: 0.2013, lr: 0.002941, batch_cost: 0.8368, reader_cost: 0.00023, ips: 1.1950 samples/sec | ETA 02:19:19 2021-05-06 20:34:28 [INFO] [TRAIN] epoch: 81, iter: 30020/40000, loss: 0.3316, lr: 0.002938, batch_cost: 0.8371, reader_cost: 0.00015, ips: 1.1946 samples/sec | ETA 02:19:14 2021-05-06 20:34:36 [INFO] [TRAIN] epoch: 81, iter: 30030/40000, loss: 0.3033, lr: 0.002936, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 02:19:33 2021-05-06 20:34:44 [INFO] [TRAIN] epoch: 81, iter: 30040/40000, loss: 0.2698, lr: 0.002933, batch_cost: 0.8389, reader_cost: 0.00015, ips: 1.1921 samples/sec | ETA 02:19:15 2021-05-06 20:34:53 [INFO] [TRAIN] epoch: 81, iter: 30050/40000, loss: 0.4042, lr: 0.002930, batch_cost: 0.8383, reader_cost: 0.00015, ips: 1.1928 samples/sec | ETA 02:19:01 2021-05-06 20:35:01 [INFO] [TRAIN] epoch: 81, iter: 30060/40000, loss: 0.2770, lr: 0.002928, batch_cost: 0.8392, reader_cost: 0.00016, ips: 1.1916 samples/sec | ETA 02:19:01 2021-05-06 20:35:09 [INFO] [TRAIN] epoch: 81, iter: 30070/40000, loss: 0.3007, lr: 0.002925, batch_cost: 0.8390, reader_cost: 0.00014, ips: 1.1920 samples/sec | ETA 02:18:50 2021-05-06 20:35:18 [INFO] [TRAIN] epoch: 81, iter: 30080/40000, loss: 0.3174, lr: 0.002923, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 02:19:02 2021-05-06 20:35:26 [INFO] [TRAIN] epoch: 81, iter: 30090/40000, loss: 0.3095, lr: 0.002920, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1882 samples/sec | ETA 02:19:00 2021-05-06 20:35:35 [INFO] [TRAIN] epoch: 81, iter: 30100/40000, loss: 0.3392, lr: 0.002918, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 02:18:46 2021-05-06 20:35:43 [INFO] [TRAIN] epoch: 81, iter: 30110/40000, loss: 0.3691, lr: 0.002915, batch_cost: 0.8426, reader_cost: 0.00015, ips: 1.1868 samples/sec | ETA 02:18:53 2021-05-06 20:35:52 [INFO] [TRAIN] epoch: 81, iter: 30120/40000, loss: 0.2889, lr: 0.002913, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 02:18:21 2021-05-06 20:36:00 [INFO] [TRAIN] epoch: 81, iter: 30130/40000, loss: 0.3076, lr: 0.002910, batch_cost: 0.8382, reader_cost: 0.00010, ips: 1.1930 samples/sec | ETA 02:17:53 2021-05-06 20:36:12 [INFO] [TRAIN] epoch: 82, iter: 30140/40000, loss: 0.5347, lr: 0.002907, batch_cost: 1.1858, reader_cost: 0.21979, ips: 0.8433 samples/sec | ETA 03:14:51 2021-05-06 20:36:20 [INFO] [TRAIN] epoch: 82, iter: 30150/40000, loss: 0.3654, lr: 0.002905, batch_cost: 0.8414, reader_cost: 0.00032, ips: 1.1885 samples/sec | ETA 02:18:07 2021-05-06 20:36:29 [INFO] [TRAIN] epoch: 82, iter: 30160/40000, loss: 0.3437, lr: 0.002902, batch_cost: 0.8415, reader_cost: 0.00016, ips: 1.1884 samples/sec | ETA 02:18:00 2021-05-06 20:36:37 [INFO] [TRAIN] epoch: 82, iter: 30170/40000, loss: 0.2548, lr: 0.002900, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 02:17:42 2021-05-06 20:36:45 [INFO] [TRAIN] epoch: 82, iter: 30180/40000, loss: 0.2941, lr: 0.002897, batch_cost: 0.8430, reader_cost: 0.00016, ips: 1.1862 samples/sec | ETA 02:17:58 2021-05-06 20:36:54 [INFO] [TRAIN] epoch: 82, iter: 30190/40000, loss: 0.3862, lr: 0.002895, batch_cost: 0.8419, reader_cost: 0.00016, ips: 1.1878 samples/sec | ETA 02:17:38 2021-05-06 20:37:02 [INFO] [TRAIN] epoch: 82, iter: 30200/40000, loss: 0.3945, lr: 0.002892, batch_cost: 0.8405, reader_cost: 0.00016, ips: 1.1897 samples/sec | ETA 02:17:17 2021-05-06 20:37:11 [INFO] [TRAIN] epoch: 82, iter: 30210/40000, loss: 0.4574, lr: 0.002889, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 02:17:07 2021-05-06 20:37:19 [INFO] [TRAIN] epoch: 82, iter: 30220/40000, loss: 0.4022, lr: 0.002887, batch_cost: 0.8421, reader_cost: 0.00016, ips: 1.1875 samples/sec | ETA 02:17:15 2021-05-06 20:37:28 [INFO] [TRAIN] epoch: 82, iter: 30230/40000, loss: 0.2749, lr: 0.002884, batch_cost: 0.8407, reader_cost: 0.00016, ips: 1.1895 samples/sec | ETA 02:16:53 2021-05-06 20:37:36 [INFO] [TRAIN] epoch: 82, iter: 30240/40000, loss: 0.2691, lr: 0.002882, batch_cost: 0.8417, reader_cost: 0.00016, ips: 1.1881 samples/sec | ETA 02:16:55 2021-05-06 20:37:44 [INFO] [TRAIN] epoch: 82, iter: 30250/40000, loss: 0.3578, lr: 0.002879, batch_cost: 0.8417, reader_cost: 0.00017, ips: 1.1881 samples/sec | ETA 02:16:46 2021-05-06 20:37:53 [INFO] [TRAIN] epoch: 82, iter: 30260/40000, loss: 0.3917, lr: 0.002877, batch_cost: 0.8404, reader_cost: 0.00016, ips: 1.1900 samples/sec | ETA 02:16:25 2021-05-06 20:38:01 [INFO] [TRAIN] epoch: 82, iter: 30270/40000, loss: 0.4713, lr: 0.002874, batch_cost: 0.8401, reader_cost: 0.00016, ips: 1.1903 samples/sec | ETA 02:16:14 2021-05-06 20:38:10 [INFO] [TRAIN] epoch: 82, iter: 30280/40000, loss: 0.3666, lr: 0.002872, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 02:16:03 2021-05-06 20:38:18 [INFO] [TRAIN] epoch: 82, iter: 30290/40000, loss: 0.2549, lr: 0.002869, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 02:16:04 2021-05-06 20:38:26 [INFO] [TRAIN] epoch: 82, iter: 30300/40000, loss: 0.3766, lr: 0.002866, batch_cost: 0.8433, reader_cost: 0.00014, ips: 1.1858 samples/sec | ETA 02:16:19 2021-05-06 20:38:35 [INFO] [TRAIN] epoch: 82, iter: 30310/40000, loss: 0.4578, lr: 0.002864, batch_cost: 0.8422, reader_cost: 0.00015, ips: 1.1874 samples/sec | ETA 02:16:00 2021-05-06 20:38:43 [INFO] [TRAIN] epoch: 82, iter: 30320/40000, loss: 0.3323, lr: 0.002861, batch_cost: 0.8399, reader_cost: 0.00016, ips: 1.1906 samples/sec | ETA 02:15:30 2021-05-06 20:38:52 [INFO] [TRAIN] epoch: 82, iter: 30330/40000, loss: 0.4044, lr: 0.002859, batch_cost: 0.8417, reader_cost: 0.00016, ips: 1.1880 samples/sec | ETA 02:15:39 2021-05-06 20:39:00 [INFO] [TRAIN] epoch: 82, iter: 30340/40000, loss: 0.3019, lr: 0.002856, batch_cost: 0.8405, reader_cost: 0.00016, ips: 1.1897 samples/sec | ETA 02:15:19 2021-05-06 20:39:08 [INFO] [TRAIN] epoch: 82, iter: 30350/40000, loss: 0.3973, lr: 0.002854, batch_cost: 0.8420, reader_cost: 0.00015, ips: 1.1876 samples/sec | ETA 02:15:25 2021-05-06 20:39:17 [INFO] [TRAIN] epoch: 82, iter: 30360/40000, loss: 0.2016, lr: 0.002851, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 02:14:57 2021-05-06 20:39:25 [INFO] [TRAIN] epoch: 82, iter: 30370/40000, loss: 0.3150, lr: 0.002848, batch_cost: 0.8389, reader_cost: 0.00016, ips: 1.1920 samples/sec | ETA 02:14:39 2021-05-06 20:39:34 [INFO] [TRAIN] epoch: 82, iter: 30380/40000, loss: 0.1830, lr: 0.002846, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 02:14:41 2021-05-06 20:39:42 [INFO] [TRAIN] epoch: 82, iter: 30390/40000, loss: 0.3285, lr: 0.002843, batch_cost: 0.8434, reader_cost: 0.00016, ips: 1.1856 samples/sec | ETA 02:15:05 2021-05-06 20:39:51 [INFO] [TRAIN] epoch: 82, iter: 30400/40000, loss: 0.2896, lr: 0.002841, batch_cost: 0.8429, reader_cost: 0.00016, ips: 1.1863 samples/sec | ETA 02:14:52 2021-05-06 20:39:59 [INFO] [TRAIN] epoch: 82, iter: 30410/40000, loss: 0.2388, lr: 0.002838, batch_cost: 0.8418, reader_cost: 0.00016, ips: 1.1880 samples/sec | ETA 02:14:32 2021-05-06 20:40:07 [INFO] [TRAIN] epoch: 82, iter: 30420/40000, loss: 0.3103, lr: 0.002836, batch_cost: 0.8414, reader_cost: 0.00016, ips: 1.1885 samples/sec | ETA 02:14:20 2021-05-06 20:40:16 [INFO] [TRAIN] epoch: 82, iter: 30430/40000, loss: 0.3994, lr: 0.002833, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 02:14:08 2021-05-06 20:40:24 [INFO] [TRAIN] epoch: 82, iter: 30440/40000, loss: 0.3364, lr: 0.002830, batch_cost: 0.8395, reader_cost: 0.00016, ips: 1.1912 samples/sec | ETA 02:13:45 2021-05-06 20:40:33 [INFO] [TRAIN] epoch: 82, iter: 30450/40000, loss: 0.3674, lr: 0.002828, batch_cost: 0.8411, reader_cost: 0.00016, ips: 1.1889 samples/sec | ETA 02:13:52 2021-05-06 20:40:41 [INFO] [TRAIN] epoch: 82, iter: 30460/40000, loss: 0.5169, lr: 0.002825, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 02:13:40 2021-05-06 20:40:49 [INFO] [TRAIN] epoch: 82, iter: 30470/40000, loss: 0.2776, lr: 0.002823, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1881 samples/sec | ETA 02:13:41 2021-05-06 20:40:58 [INFO] [TRAIN] epoch: 82, iter: 30480/40000, loss: 0.3035, lr: 0.002820, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 02:13:16 2021-05-06 20:41:06 [INFO] [TRAIN] epoch: 82, iter: 30490/40000, loss: 0.2904, lr: 0.002818, batch_cost: 0.8400, reader_cost: 0.00014, ips: 1.1904 samples/sec | ETA 02:13:08 2021-05-06 20:41:15 [INFO] [TRAIN] epoch: 82, iter: 30500/40000, loss: 0.4087, lr: 0.002815, batch_cost: 0.8376, reader_cost: 0.00012, ips: 1.1939 samples/sec | ETA 02:12:37 2021-05-06 20:41:26 [INFO] [TRAIN] epoch: 83, iter: 30510/40000, loss: 0.4594, lr: 0.002812, batch_cost: 1.1508, reader_cost: 0.19794, ips: 0.8690 samples/sec | ETA 03:02:00 2021-05-06 20:41:35 [INFO] [TRAIN] epoch: 83, iter: 30520/40000, loss: 0.3971, lr: 0.002810, batch_cost: 0.8408, reader_cost: 0.00032, ips: 1.1893 samples/sec | ETA 02:12:51 2021-05-06 20:41:43 [INFO] [TRAIN] epoch: 83, iter: 30530/40000, loss: 0.4162, lr: 0.002807, batch_cost: 0.8414, reader_cost: 0.00017, ips: 1.1884 samples/sec | ETA 02:12:48 2021-05-06 20:41:51 [INFO] [TRAIN] epoch: 83, iter: 30540/40000, loss: 0.2614, lr: 0.002805, batch_cost: 0.8419, reader_cost: 0.00015, ips: 1.1878 samples/sec | ETA 02:12:44 2021-05-06 20:42:00 [INFO] [TRAIN] epoch: 83, iter: 30550/40000, loss: 0.3003, lr: 0.002802, batch_cost: 0.8414, reader_cost: 0.00016, ips: 1.1885 samples/sec | ETA 02:12:31 2021-05-06 20:42:08 [INFO] [TRAIN] epoch: 83, iter: 30560/40000, loss: 0.4327, lr: 0.002800, batch_cost: 0.8437, reader_cost: 0.00016, ips: 1.1853 samples/sec | ETA 02:12:44 2021-05-06 20:42:17 [INFO] [TRAIN] epoch: 83, iter: 30570/40000, loss: 0.4119, lr: 0.002797, batch_cost: 0.8405, reader_cost: 0.00016, ips: 1.1898 samples/sec | ETA 02:12:05 2021-05-06 20:42:25 [INFO] [TRAIN] epoch: 83, iter: 30580/40000, loss: 0.4073, lr: 0.002794, batch_cost: 0.8397, reader_cost: 0.00014, ips: 1.1909 samples/sec | ETA 02:11:50 2021-05-06 20:42:33 [INFO] [TRAIN] epoch: 83, iter: 30590/40000, loss: 0.3485, lr: 0.002792, batch_cost: 0.8411, reader_cost: 0.00014, ips: 1.1889 samples/sec | ETA 02:11:54 2021-05-06 20:42:42 [INFO] [TRAIN] epoch: 83, iter: 30600/40000, loss: 0.2334, lr: 0.002789, batch_cost: 0.8415, reader_cost: 0.00016, ips: 1.1883 samples/sec | ETA 02:11:50 2021-05-06 20:42:50 [INFO] [TRAIN] epoch: 83, iter: 30610/40000, loss: 0.3206, lr: 0.002787, batch_cost: 0.8428, reader_cost: 0.00016, ips: 1.1865 samples/sec | ETA 02:11:53 2021-05-06 20:42:59 [INFO] [TRAIN] epoch: 83, iter: 30620/40000, loss: 0.3913, lr: 0.002784, batch_cost: 0.8425, reader_cost: 0.00015, ips: 1.1869 samples/sec | ETA 02:11:42 2021-05-06 20:43:07 [INFO] [TRAIN] epoch: 83, iter: 30630/40000, loss: 0.3909, lr: 0.002782, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1881 samples/sec | ETA 02:11:26 2021-05-06 20:43:16 [INFO] [TRAIN] epoch: 83, iter: 30640/40000, loss: 0.5626, lr: 0.002779, batch_cost: 0.8426, reader_cost: 0.00016, ips: 1.1868 samples/sec | ETA 02:11:26 2021-05-06 20:43:24 [INFO] [TRAIN] epoch: 83, iter: 30650/40000, loss: 0.3940, lr: 0.002776, batch_cost: 0.8423, reader_cost: 0.00015, ips: 1.1872 samples/sec | ETA 02:11:15 2021-05-06 20:43:32 [INFO] [TRAIN] epoch: 83, iter: 30660/40000, loss: 0.2743, lr: 0.002774, batch_cost: 0.8435, reader_cost: 0.00015, ips: 1.1855 samples/sec | ETA 02:11:18 2021-05-06 20:43:41 [INFO] [TRAIN] epoch: 83, iter: 30670/40000, loss: 0.4543, lr: 0.002771, batch_cost: 0.8412, reader_cost: 0.00017, ips: 1.1888 samples/sec | ETA 02:10:48 2021-05-06 20:43:49 [INFO] [TRAIN] epoch: 83, iter: 30680/40000, loss: 0.3898, lr: 0.002769, batch_cost: 0.8414, reader_cost: 0.00016, ips: 1.1885 samples/sec | ETA 02:10:42 2021-05-06 20:43:58 [INFO] [TRAIN] epoch: 83, iter: 30690/40000, loss: 0.2926, lr: 0.002766, batch_cost: 0.8415, reader_cost: 0.00016, ips: 1.1884 samples/sec | ETA 02:10:34 2021-05-06 20:44:06 [INFO] [TRAIN] epoch: 83, iter: 30700/40000, loss: 0.5715, lr: 0.002764, batch_cost: 0.8410, reader_cost: 0.00016, ips: 1.1890 samples/sec | ETA 02:10:21 2021-05-06 20:44:14 [INFO] [TRAIN] epoch: 83, iter: 30710/40000, loss: 0.4643, lr: 0.002761, batch_cost: 0.8419, reader_cost: 0.00015, ips: 1.1878 samples/sec | ETA 02:10:20 2021-05-06 20:44:23 [INFO] [TRAIN] epoch: 83, iter: 30720/40000, loss: 0.3098, lr: 0.002758, batch_cost: 0.8418, reader_cost: 0.00014, ips: 1.1879 samples/sec | ETA 02:10:11 2021-05-06 20:44:31 [INFO] [TRAIN] epoch: 83, iter: 30730/40000, loss: 0.1677, lr: 0.002756, batch_cost: 0.8425, reader_cost: 0.00015, ips: 1.1870 samples/sec | ETA 02:10:09 2021-05-06 20:44:40 [INFO] [TRAIN] epoch: 83, iter: 30740/40000, loss: 0.2497, lr: 0.002753, batch_cost: 0.8412, reader_cost: 0.00014, ips: 1.1888 samples/sec | ETA 02:09:49 2021-05-06 20:44:48 [INFO] [TRAIN] epoch: 83, iter: 30750/40000, loss: 0.2234, lr: 0.002751, batch_cost: 0.8427, reader_cost: 0.00016, ips: 1.1867 samples/sec | ETA 02:09:54 2021-05-06 20:44:57 [INFO] [TRAIN] epoch: 83, iter: 30760/40000, loss: 0.3114, lr: 0.002748, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 02:09:19 2021-05-06 20:45:05 [INFO] [TRAIN] epoch: 83, iter: 30770/40000, loss: 0.3263, lr: 0.002745, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 02:09:16 2021-05-06 20:45:13 [INFO] [TRAIN] epoch: 83, iter: 30780/40000, loss: 0.2813, lr: 0.002743, batch_cost: 0.8394, reader_cost: 0.00014, ips: 1.1914 samples/sec | ETA 02:08:59 2021-05-06 20:45:22 [INFO] [TRAIN] epoch: 83, iter: 30790/40000, loss: 0.3212, lr: 0.002740, batch_cost: 0.8412, reader_cost: 0.00016, ips: 1.1887 samples/sec | ETA 02:09:07 2021-05-06 20:45:30 [INFO] [TRAIN] epoch: 83, iter: 30800/40000, loss: 0.4350, lr: 0.002738, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 02:08:48 2021-05-06 20:45:39 [INFO] [TRAIN] epoch: 83, iter: 30810/40000, loss: 0.2455, lr: 0.002735, batch_cost: 0.8395, reader_cost: 0.00016, ips: 1.1913 samples/sec | ETA 02:08:34 2021-05-06 20:45:47 [INFO] [TRAIN] epoch: 83, iter: 30820/40000, loss: 0.2731, lr: 0.002733, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 02:08:41 2021-05-06 20:45:55 [INFO] [TRAIN] epoch: 83, iter: 30830/40000, loss: 0.3833, lr: 0.002730, batch_cost: 0.8384, reader_cost: 0.00014, ips: 1.1927 samples/sec | ETA 02:08:08 2021-05-06 20:46:04 [INFO] [TRAIN] epoch: 83, iter: 30840/40000, loss: 0.2915, lr: 0.002727, batch_cost: 0.8405, reader_cost: 0.00014, ips: 1.1898 samples/sec | ETA 02:08:18 2021-05-06 20:46:12 [INFO] [TRAIN] epoch: 83, iter: 30850/40000, loss: 0.3996, lr: 0.002725, batch_cost: 0.8400, reader_cost: 0.00014, ips: 1.1905 samples/sec | ETA 02:08:05 2021-05-06 20:46:21 [INFO] [TRAIN] epoch: 83, iter: 30860/40000, loss: 0.2500, lr: 0.002722, batch_cost: 0.8385, reader_cost: 0.00016, ips: 1.1926 samples/sec | ETA 02:07:44 2021-05-06 20:46:29 [INFO] [TRAIN] epoch: 83, iter: 30870/40000, loss: 0.3563, lr: 0.002720, batch_cost: 0.8385, reader_cost: 0.00012, ips: 1.1926 samples/sec | ETA 02:07:35 2021-05-06 20:46:40 [INFO] [TRAIN] epoch: 84, iter: 30880/40000, loss: 0.3723, lr: 0.002717, batch_cost: 1.1386, reader_cost: 0.20566, ips: 0.8783 samples/sec | ETA 02:53:03 2021-05-06 20:46:49 [INFO] [TRAIN] epoch: 84, iter: 30890/40000, loss: 0.3362, lr: 0.002715, batch_cost: 0.8516, reader_cost: 0.00030, ips: 1.1742 samples/sec | ETA 02:09:18 2021-05-06 20:46:57 [INFO] [TRAIN] epoch: 84, iter: 30900/40000, loss: 0.2932, lr: 0.002712, batch_cost: 0.8391, reader_cost: 0.00016, ips: 1.1917 samples/sec | ETA 02:07:16 2021-05-06 20:47:06 [INFO] [TRAIN] epoch: 84, iter: 30910/40000, loss: 0.2817, lr: 0.002709, batch_cost: 0.8394, reader_cost: 0.00016, ips: 1.1914 samples/sec | ETA 02:07:09 2021-05-06 20:47:14 [INFO] [TRAIN] epoch: 84, iter: 30920/40000, loss: 0.2704, lr: 0.002707, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 02:07:00 2021-05-06 20:47:22 [INFO] [TRAIN] epoch: 84, iter: 30930/40000, loss: 0.4838, lr: 0.002704, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1888 samples/sec | ETA 02:07:09 2021-05-06 20:47:31 [INFO] [TRAIN] epoch: 84, iter: 30940/40000, loss: 0.4292, lr: 0.002702, batch_cost: 0.8411, reader_cost: 0.00014, ips: 1.1889 samples/sec | ETA 02:07:00 2021-05-06 20:47:39 [INFO] [TRAIN] epoch: 84, iter: 30950/40000, loss: 0.5243, lr: 0.002699, batch_cost: 0.8433, reader_cost: 0.00015, ips: 1.1859 samples/sec | ETA 02:07:11 2021-05-06 20:47:48 [INFO] [TRAIN] epoch: 84, iter: 30960/40000, loss: 0.3608, lr: 0.002696, batch_cost: 0.8413, reader_cost: 0.00016, ips: 1.1886 samples/sec | ETA 02:06:45 2021-05-06 20:47:56 [INFO] [TRAIN] epoch: 84, iter: 30970/40000, loss: 0.3179, lr: 0.002694, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 02:06:34 2021-05-06 20:48:05 [INFO] [TRAIN] epoch: 84, iter: 30980/40000, loss: 0.2120, lr: 0.002691, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1883 samples/sec | ETA 02:06:30 2021-05-06 20:48:13 [INFO] [TRAIN] epoch: 84, iter: 30990/40000, loss: 0.4513, lr: 0.002689, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1880 samples/sec | ETA 02:06:23 2021-05-06 20:48:21 [INFO] [TRAIN] epoch: 84, iter: 31000/40000, loss: 0.4154, lr: 0.002686, batch_cost: 0.8416, reader_cost: 0.00016, ips: 1.1883 samples/sec | ETA 02:06:14 2021-05-06 20:48:21 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 20:51:58 [INFO] [EVAL] #Images: 500 mIoU: 0.7678 Acc: 0.9583 Kappa: 0.9459 2021-05-06 20:51:58 [INFO] [EVAL] Class IoU: [0.981 0.8475 0.9224 0.6468 0.6037 0.5557 0.6074 0.7373 0.9192 0.6543 0.9444 0.7823 0.5425 0.9435 0.8259 0.8966 0.8043 0.6425 0.7305] 2021-05-06 20:51:58 [INFO] [EVAL] Class Acc: [0.9916 0.9152 0.9556 0.8744 0.7433 0.7811 0.8308 0.89 0.9499 0.8155 0.9637 0.8667 0.8001 0.9672 0.9183 0.9505 0.9137 0.8492 0.8386] 2021-05-06 20:52:49 [INFO] [EVAL] The model with the best validation mIoU (0.7678) was saved at iter 31000. 2021-05-06 20:52:58 [INFO] [TRAIN] epoch: 84, iter: 31010/40000, loss: 0.3540, lr: 0.002683, batch_cost: 0.8351, reader_cost: 0.00034, ips: 1.1974 samples/sec | ETA 02:05:07 2021-05-06 20:53:06 [INFO] [TRAIN] epoch: 84, iter: 31020/40000, loss: 0.4283, lr: 0.002681, batch_cost: 0.8369, reader_cost: 0.00015, ips: 1.1949 samples/sec | ETA 02:05:15 2021-05-06 20:53:15 [INFO] [TRAIN] epoch: 84, iter: 31030/40000, loss: 0.2842, lr: 0.002678, batch_cost: 0.8380, reader_cost: 0.00015, ips: 1.1934 samples/sec | ETA 02:05:16 2021-05-06 20:53:23 [INFO] [TRAIN] epoch: 84, iter: 31040/40000, loss: 0.2924, lr: 0.002676, batch_cost: 0.8383, reader_cost: 0.00015, ips: 1.1928 samples/sec | ETA 02:05:11 2021-05-06 20:53:31 [INFO] [TRAIN] epoch: 84, iter: 31050/40000, loss: 0.2290, lr: 0.002673, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 02:05:15 2021-05-06 20:53:40 [INFO] [TRAIN] epoch: 84, iter: 31060/40000, loss: 0.3623, lr: 0.002671, batch_cost: 0.8392, reader_cost: 0.00015, ips: 1.1916 samples/sec | ETA 02:05:02 2021-05-06 20:53:48 [INFO] [TRAIN] epoch: 84, iter: 31070/40000, loss: 0.4607, lr: 0.002668, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 02:05:02 2021-05-06 20:53:57 [INFO] [TRAIN] epoch: 84, iter: 31080/40000, loss: 0.3714, lr: 0.002665, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 02:05:00 2021-05-06 20:54:05 [INFO] [TRAIN] epoch: 84, iter: 31090/40000, loss: 0.2998, lr: 0.002663, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 02:04:44 2021-05-06 20:54:13 [INFO] [TRAIN] epoch: 84, iter: 31100/40000, loss: 0.4143, lr: 0.002660, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1888 samples/sec | ETA 02:04:46 2021-05-06 20:54:22 [INFO] [TRAIN] epoch: 84, iter: 31110/40000, loss: 0.2920, lr: 0.002658, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 02:04:31 2021-05-06 20:54:30 [INFO] [TRAIN] epoch: 84, iter: 31120/40000, loss: 0.2949, lr: 0.002655, batch_cost: 0.8419, reader_cost: 0.00014, ips: 1.1878 samples/sec | ETA 02:04:36 2021-05-06 20:54:39 [INFO] [TRAIN] epoch: 84, iter: 31130/40000, loss: 0.2347, lr: 0.002652, batch_cost: 0.8418, reader_cost: 0.00014, ips: 1.1879 samples/sec | ETA 02:04:26 2021-05-06 20:54:47 [INFO] [TRAIN] epoch: 84, iter: 31140/40000, loss: 0.3705, lr: 0.002650, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1880 samples/sec | ETA 02:04:17 2021-05-06 20:54:55 [INFO] [TRAIN] epoch: 84, iter: 31150/40000, loss: 0.3079, lr: 0.002647, batch_cost: 0.8398, reader_cost: 0.00016, ips: 1.1908 samples/sec | ETA 02:03:51 2021-05-06 20:55:04 [INFO] [TRAIN] epoch: 84, iter: 31160/40000, loss: 0.5660, lr: 0.002645, batch_cost: 0.8407, reader_cost: 0.00016, ips: 1.1896 samples/sec | ETA 02:03:51 2021-05-06 20:55:12 [INFO] [TRAIN] epoch: 84, iter: 31170/40000, loss: 0.2873, lr: 0.002642, batch_cost: 0.8398, reader_cost: 0.00016, ips: 1.1907 samples/sec | ETA 02:03:35 2021-05-06 20:55:21 [INFO] [TRAIN] epoch: 84, iter: 31180/40000, loss: 0.2884, lr: 0.002639, batch_cost: 0.8389, reader_cost: 0.00016, ips: 1.1921 samples/sec | ETA 02:03:18 2021-05-06 20:55:29 [INFO] [TRAIN] epoch: 84, iter: 31190/40000, loss: 0.3208, lr: 0.002637, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 02:03:24 2021-05-06 20:55:37 [INFO] [TRAIN] epoch: 84, iter: 31200/40000, loss: 0.4609, lr: 0.002634, batch_cost: 0.8388, reader_cost: 0.00015, ips: 1.1921 samples/sec | ETA 02:03:01 2021-05-06 20:55:46 [INFO] [TRAIN] epoch: 84, iter: 31210/40000, loss: 0.3288, lr: 0.002632, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 02:03:05 2021-05-06 20:55:54 [INFO] [TRAIN] epoch: 84, iter: 31220/40000, loss: 0.3755, lr: 0.002629, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1917 samples/sec | ETA 02:02:47 2021-05-06 20:56:03 [INFO] [TRAIN] epoch: 84, iter: 31230/40000, loss: 0.3671, lr: 0.002627, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1913 samples/sec | ETA 02:02:41 2021-05-06 20:56:11 [INFO] [TRAIN] epoch: 84, iter: 31240/40000, loss: 0.4312, lr: 0.002624, batch_cost: 0.8378, reader_cost: 0.00013, ips: 1.1936 samples/sec | ETA 02:02:19 2021-05-06 20:56:23 [INFO] [TRAIN] epoch: 85, iter: 31250/40000, loss: 0.4069, lr: 0.002621, batch_cost: 1.1561, reader_cost: 0.22372, ips: 0.8650 samples/sec | ETA 02:48:35 2021-05-06 20:56:31 [INFO] [TRAIN] epoch: 85, iter: 31260/40000, loss: 0.4416, lr: 0.002619, batch_cost: 0.8640, reader_cost: 0.00032, ips: 1.1574 samples/sec | ETA 02:05:51 2021-05-06 20:56:40 [INFO] [TRAIN] epoch: 85, iter: 31270/40000, loss: 0.4314, lr: 0.002616, batch_cost: 0.8400, reader_cost: 0.00016, ips: 1.1905 samples/sec | ETA 02:02:13 2021-05-06 20:56:48 [INFO] [TRAIN] epoch: 85, iter: 31280/40000, loss: 0.4452, lr: 0.002614, batch_cost: 0.8394, reader_cost: 0.00016, ips: 1.1914 samples/sec | ETA 02:01:59 2021-05-06 20:56:56 [INFO] [TRAIN] epoch: 85, iter: 31290/40000, loss: 0.2812, lr: 0.002611, batch_cost: 0.8397, reader_cost: 0.00014, ips: 1.1909 samples/sec | ETA 02:01:53 2021-05-06 20:57:05 [INFO] [TRAIN] epoch: 85, iter: 31300/40000, loss: 0.3146, lr: 0.002608, batch_cost: 0.8409, reader_cost: 0.00014, ips: 1.1892 samples/sec | ETA 02:01:55 2021-05-06 20:57:13 [INFO] [TRAIN] epoch: 85, iter: 31310/40000, loss: 0.4273, lr: 0.002606, batch_cost: 0.8416, reader_cost: 0.00016, ips: 1.1881 samples/sec | ETA 02:01:53 2021-05-06 20:57:22 [INFO] [TRAIN] epoch: 85, iter: 31320/40000, loss: 0.7063, lr: 0.002603, batch_cost: 0.8402, reader_cost: 0.00014, ips: 1.1902 samples/sec | ETA 02:01:33 2021-05-06 20:57:30 [INFO] [TRAIN] epoch: 85, iter: 31330/40000, loss: 0.4673, lr: 0.002601, batch_cost: 0.8384, reader_cost: 0.00016, ips: 1.1927 samples/sec | ETA 02:01:09 2021-05-06 20:57:38 [INFO] [TRAIN] epoch: 85, iter: 31340/40000, loss: 0.2829, lr: 0.002598, batch_cost: 0.8419, reader_cost: 0.00016, ips: 1.1878 samples/sec | ETA 02:01:30 2021-05-06 20:57:47 [INFO] [TRAIN] epoch: 85, iter: 31350/40000, loss: 0.2455, lr: 0.002595, batch_cost: 0.8412, reader_cost: 0.00014, ips: 1.1888 samples/sec | ETA 02:01:16 2021-05-06 20:57:55 [INFO] [TRAIN] epoch: 85, iter: 31360/40000, loss: 0.2789, lr: 0.002593, batch_cost: 0.8395, reader_cost: 0.00014, ips: 1.1912 samples/sec | ETA 02:00:52 2021-05-06 20:58:04 [INFO] [TRAIN] epoch: 85, iter: 31370/40000, loss: 0.4059, lr: 0.002590, batch_cost: 0.8415, reader_cost: 0.00014, ips: 1.1884 samples/sec | ETA 02:01:01 2021-05-06 20:58:12 [INFO] [TRAIN] epoch: 85, iter: 31380/40000, loss: 0.4145, lr: 0.002588, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 02:00:42 2021-05-06 20:58:20 [INFO] [TRAIN] epoch: 85, iter: 31390/40000, loss: 0.5510, lr: 0.002585, batch_cost: 0.8393, reader_cost: 0.00014, ips: 1.1915 samples/sec | ETA 02:00:26 2021-05-06 20:58:29 [INFO] [TRAIN] epoch: 85, iter: 31400/40000, loss: 0.3950, lr: 0.002582, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1882 samples/sec | ETA 02:00:38 2021-05-06 20:58:37 [INFO] [TRAIN] epoch: 85, iter: 31410/40000, loss: 0.3179, lr: 0.002580, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 02:00:18 2021-05-06 20:58:46 [INFO] [TRAIN] epoch: 85, iter: 31420/40000, loss: 0.2695, lr: 0.002577, batch_cost: 0.8384, reader_cost: 0.00015, ips: 1.1928 samples/sec | ETA 01:59:53 2021-05-06 20:58:54 [INFO] [TRAIN] epoch: 85, iter: 31430/40000, loss: 0.4644, lr: 0.002575, batch_cost: 0.8393, reader_cost: 0.00014, ips: 1.1915 samples/sec | ETA 01:59:52 2021-05-06 20:59:02 [INFO] [TRAIN] epoch: 85, iter: 31440/40000, loss: 0.6012, lr: 0.002572, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 01:59:54 2021-05-06 20:59:11 [INFO] [TRAIN] epoch: 85, iter: 31450/40000, loss: 0.3081, lr: 0.002569, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 01:59:44 2021-05-06 20:59:19 [INFO] [TRAIN] epoch: 85, iter: 31460/40000, loss: 0.4494, lr: 0.002567, batch_cost: 0.8405, reader_cost: 0.00016, ips: 1.1898 samples/sec | ETA 01:59:37 2021-05-06 20:59:28 [INFO] [TRAIN] epoch: 85, iter: 31470/40000, loss: 0.3577, lr: 0.002564, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 01:59:25 2021-05-06 20:59:36 [INFO] [TRAIN] epoch: 85, iter: 31480/40000, loss: 0.2335, lr: 0.002562, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1912 samples/sec | ETA 01:59:12 2021-05-06 20:59:44 [INFO] [TRAIN] epoch: 85, iter: 31490/40000, loss: 0.3149, lr: 0.002559, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1888 samples/sec | ETA 01:59:18 2021-05-06 20:59:53 [INFO] [TRAIN] epoch: 85, iter: 31500/40000, loss: 0.2292, lr: 0.002556, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 01:58:58 2021-05-06 21:00:01 [INFO] [TRAIN] epoch: 85, iter: 31510/40000, loss: 0.3726, lr: 0.002554, batch_cost: 0.8388, reader_cost: 0.00015, ips: 1.1921 samples/sec | ETA 01:58:41 2021-05-06 21:00:10 [INFO] [TRAIN] epoch: 85, iter: 31520/40000, loss: 0.2614, lr: 0.002551, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 01:58:50 2021-05-06 21:00:18 [INFO] [TRAIN] epoch: 85, iter: 31530/40000, loss: 0.3012, lr: 0.002549, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 01:58:37 2021-05-06 21:00:26 [INFO] [TRAIN] epoch: 85, iter: 31540/40000, loss: 0.3911, lr: 0.002546, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 01:58:26 2021-05-06 21:00:35 [INFO] [TRAIN] epoch: 85, iter: 31550/40000, loss: 0.3269, lr: 0.002543, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 01:58:21 2021-05-06 21:00:43 [INFO] [TRAIN] epoch: 85, iter: 31560/40000, loss: 0.2238, lr: 0.002541, batch_cost: 0.8398, reader_cost: 0.00014, ips: 1.1907 samples/sec | ETA 01:58:08 2021-05-06 21:00:52 [INFO] [TRAIN] epoch: 85, iter: 31570/40000, loss: 0.3637, lr: 0.002538, batch_cost: 0.8393, reader_cost: 0.00014, ips: 1.1915 samples/sec | ETA 01:57:54 2021-05-06 21:01:00 [INFO] [TRAIN] epoch: 85, iter: 31580/40000, loss: 0.2606, lr: 0.002536, batch_cost: 0.8392, reader_cost: 0.00015, ips: 1.1916 samples/sec | ETA 01:57:46 2021-05-06 21:01:08 [INFO] [TRAIN] epoch: 85, iter: 31590/40000, loss: 0.3761, lr: 0.002533, batch_cost: 0.8399, reader_cost: 0.00016, ips: 1.1906 samples/sec | ETA 01:57:43 2021-05-06 21:01:17 [INFO] [TRAIN] epoch: 85, iter: 31600/40000, loss: 0.3838, lr: 0.002530, batch_cost: 0.8422, reader_cost: 0.00015, ips: 1.1874 samples/sec | ETA 01:57:54 2021-05-06 21:01:25 [INFO] [TRAIN] epoch: 85, iter: 31610/40000, loss: 0.3170, lr: 0.002528, batch_cost: 0.8395, reader_cost: 0.00013, ips: 1.1912 samples/sec | ETA 01:57:23 2021-05-06 21:01:34 [INFO] [TRAIN] epoch: 85, iter: 31620/40000, loss: 0.2117, lr: 0.002525, batch_cost: 0.8386, reader_cost: 0.00009, ips: 1.1925 samples/sec | ETA 01:57:07 2021-05-06 21:01:45 [INFO] [TRAIN] epoch: 86, iter: 31630/40000, loss: 0.4541, lr: 0.002523, batch_cost: 1.1790, reader_cost: 0.21206, ips: 0.8482 samples/sec | ETA 02:44:28 2021-05-06 21:01:54 [INFO] [TRAIN] epoch: 86, iter: 31640/40000, loss: 0.2971, lr: 0.002520, batch_cost: 0.8398, reader_cost: 0.00030, ips: 1.1907 samples/sec | ETA 01:57:01 2021-05-06 21:02:02 [INFO] [TRAIN] epoch: 86, iter: 31650/40000, loss: 0.4422, lr: 0.002517, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1917 samples/sec | ETA 01:56:46 2021-05-06 21:02:11 [INFO] [TRAIN] epoch: 86, iter: 31660/40000, loss: 0.1991, lr: 0.002515, batch_cost: 0.8387, reader_cost: 0.00015, ips: 1.1923 samples/sec | ETA 01:56:34 2021-05-06 21:02:19 [INFO] [TRAIN] epoch: 86, iter: 31670/40000, loss: 0.3018, lr: 0.002512, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 01:56:42 2021-05-06 21:02:27 [INFO] [TRAIN] epoch: 86, iter: 31680/40000, loss: 0.2914, lr: 0.002510, batch_cost: 0.8386, reader_cost: 0.00016, ips: 1.1924 samples/sec | ETA 01:56:17 2021-05-06 21:02:36 [INFO] [TRAIN] epoch: 86, iter: 31690/40000, loss: 0.3740, lr: 0.002507, batch_cost: 0.8386, reader_cost: 0.00015, ips: 1.1925 samples/sec | ETA 01:56:08 2021-05-06 21:02:44 [INFO] [TRAIN] epoch: 86, iter: 31700/40000, loss: 0.4665, lr: 0.002504, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1913 samples/sec | ETA 01:56:07 2021-05-06 21:02:53 [INFO] [TRAIN] epoch: 86, iter: 31710/40000, loss: 0.3735, lr: 0.002502, batch_cost: 0.8406, reader_cost: 0.00016, ips: 1.1897 samples/sec | ETA 01:56:08 2021-05-06 21:03:01 [INFO] [TRAIN] epoch: 86, iter: 31720/40000, loss: 0.1210, lr: 0.002499, batch_cost: 0.8396, reader_cost: 0.00016, ips: 1.1910 samples/sec | ETA 01:55:52 2021-05-06 21:03:09 [INFO] [TRAIN] epoch: 86, iter: 31730/40000, loss: 0.2981, lr: 0.002497, batch_cost: 0.8390, reader_cost: 0.00016, ips: 1.1919 samples/sec | ETA 01:55:38 2021-05-06 21:03:18 [INFO] [TRAIN] epoch: 86, iter: 31740/40000, loss: 0.4404, lr: 0.002494, batch_cost: 0.8389, reader_cost: 0.00015, ips: 1.1920 samples/sec | ETA 01:55:29 2021-05-06 21:03:26 [INFO] [TRAIN] epoch: 86, iter: 31750/40000, loss: 0.4250, lr: 0.002491, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 01:55:34 2021-05-06 21:03:35 [INFO] [TRAIN] epoch: 86, iter: 31760/40000, loss: 0.5019, lr: 0.002489, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 01:55:23 2021-05-06 21:03:43 [INFO] [TRAIN] epoch: 86, iter: 31770/40000, loss: 0.4005, lr: 0.002486, batch_cost: 0.8410, reader_cost: 0.00016, ips: 1.1891 samples/sec | ETA 01:55:21 2021-05-06 21:03:51 [INFO] [TRAIN] epoch: 86, iter: 31780/40000, loss: 0.2992, lr: 0.002483, batch_cost: 0.8403, reader_cost: 0.00016, ips: 1.1901 samples/sec | ETA 01:55:06 2021-05-06 21:04:00 [INFO] [TRAIN] epoch: 86, iter: 31790/40000, loss: 0.2900, lr: 0.002481, batch_cost: 0.8395, reader_cost: 0.00014, ips: 1.1912 samples/sec | ETA 01:54:52 2021-05-06 21:04:08 [INFO] [TRAIN] epoch: 86, iter: 31800/40000, loss: 0.3135, lr: 0.002478, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1913 samples/sec | ETA 01:54:43 2021-05-06 21:04:17 [INFO] [TRAIN] epoch: 86, iter: 31810/40000, loss: 0.4798, lr: 0.002476, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1911 samples/sec | ETA 01:54:36 2021-05-06 21:04:25 [INFO] [TRAIN] epoch: 86, iter: 31820/40000, loss: 0.4048, lr: 0.002473, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 01:54:27 2021-05-06 21:04:33 [INFO] [TRAIN] epoch: 86, iter: 31830/40000, loss: 0.4077, lr: 0.002470, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1888 samples/sec | ETA 01:54:32 2021-05-06 21:04:42 [INFO] [TRAIN] epoch: 86, iter: 31840/40000, loss: 0.3427, lr: 0.002468, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 01:54:21 2021-05-06 21:04:50 [INFO] [TRAIN] epoch: 86, iter: 31850/40000, loss: 0.3186, lr: 0.002465, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 01:54:15 2021-05-06 21:04:59 [INFO] [TRAIN] epoch: 86, iter: 31860/40000, loss: 0.2878, lr: 0.002463, batch_cost: 0.8396, reader_cost: 0.00016, ips: 1.1911 samples/sec | ETA 01:53:54 2021-05-06 21:05:07 [INFO] [TRAIN] epoch: 86, iter: 31870/40000, loss: 0.1666, lr: 0.002460, batch_cost: 0.8387, reader_cost: 0.00016, ips: 1.1923 samples/sec | ETA 01:53:38 2021-05-06 21:05:15 [INFO] [TRAIN] epoch: 86, iter: 31880/40000, loss: 0.3181, lr: 0.002457, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1911 samples/sec | ETA 01:53:37 2021-05-06 21:05:24 [INFO] [TRAIN] epoch: 86, iter: 31890/40000, loss: 0.3100, lr: 0.002455, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 01:53:30 2021-05-06 21:05:32 [INFO] [TRAIN] epoch: 86, iter: 31900/40000, loss: 0.2397, lr: 0.002452, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 01:53:20 2021-05-06 21:05:41 [INFO] [TRAIN] epoch: 86, iter: 31910/40000, loss: 0.3758, lr: 0.002450, batch_cost: 0.8403, reader_cost: 0.00016, ips: 1.1901 samples/sec | ETA 01:53:17 2021-05-06 21:05:49 [INFO] [TRAIN] epoch: 86, iter: 31920/40000, loss: 0.2992, lr: 0.002447, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 01:53:13 2021-05-06 21:05:57 [INFO] [TRAIN] epoch: 86, iter: 31930/40000, loss: 0.2665, lr: 0.002444, batch_cost: 0.8413, reader_cost: 0.00014, ips: 1.1886 samples/sec | ETA 01:53:09 2021-05-06 21:06:06 [INFO] [TRAIN] epoch: 86, iter: 31940/40000, loss: 0.3505, lr: 0.002442, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 01:52:59 2021-05-06 21:06:14 [INFO] [TRAIN] epoch: 86, iter: 31950/40000, loss: 0.4231, lr: 0.002439, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 01:52:49 2021-05-06 21:06:23 [INFO] [TRAIN] epoch: 86, iter: 31960/40000, loss: 0.3423, lr: 0.002436, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1883 samples/sec | ETA 01:52:45 2021-05-06 21:06:31 [INFO] [TRAIN] epoch: 86, iter: 31970/40000, loss: 0.3674, lr: 0.002434, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 01:52:25 2021-05-06 21:06:39 [INFO] [TRAIN] epoch: 86, iter: 31980/40000, loss: 0.3441, lr: 0.002431, batch_cost: 0.8402, reader_cost: 0.00014, ips: 1.1901 samples/sec | ETA 01:52:18 2021-05-06 21:06:48 [INFO] [TRAIN] epoch: 86, iter: 31990/40000, loss: 0.3568, lr: 0.002429, batch_cost: 0.8383, reader_cost: 0.00029, ips: 1.1929 samples/sec | ETA 01:51:54 2021-05-06 21:07:00 [INFO] [TRAIN] epoch: 87, iter: 32000/40000, loss: 0.4387, lr: 0.002426, batch_cost: 1.1702, reader_cost: 0.21799, ips: 0.8546 samples/sec | ETA 02:36:01 2021-05-06 21:07:00 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 21:10:39 [INFO] [EVAL] #Images: 500 mIoU: 0.7713 Acc: 0.9587 Kappa: 0.9464 2021-05-06 21:10:39 [INFO] [EVAL] Class IoU: [0.982 0.8527 0.9208 0.6574 0.6109 0.5634 0.6214 0.7324 0.9191 0.6534 0.9447 0.791 0.598 0.943 0.8009 0.8889 0.8097 0.6336 0.7308] 2021-05-06 21:10:39 [INFO] [EVAL] Class Acc: [0.991 0.9271 0.9504 0.8801 0.8139 0.7561 0.7888 0.8964 0.9536 0.8492 0.9652 0.8884 0.7211 0.9652 0.9484 0.9644 0.9103 0.7557 0.8244] 2021-05-06 21:11:28 [INFO] [EVAL] The model with the best validation mIoU (0.7713) was saved at iter 32000. 2021-05-06 21:11:37 [INFO] [TRAIN] epoch: 87, iter: 32010/40000, loss: 0.3476, lr: 0.002423, batch_cost: 0.8392, reader_cost: 0.00025, ips: 1.1917 samples/sec | ETA 01:51:44 2021-05-06 21:11:45 [INFO] [TRAIN] epoch: 87, iter: 32020/40000, loss: 0.4995, lr: 0.002421, batch_cost: 0.8381, reader_cost: 0.00014, ips: 1.1932 samples/sec | ETA 01:51:27 2021-05-06 21:11:53 [INFO] [TRAIN] epoch: 87, iter: 32030/40000, loss: 0.2339, lr: 0.002418, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 01:51:32 2021-05-06 21:12:02 [INFO] [TRAIN] epoch: 87, iter: 32040/40000, loss: 0.3230, lr: 0.002416, batch_cost: 0.8392, reader_cost: 0.00014, ips: 1.1916 samples/sec | ETA 01:51:19 2021-05-06 21:12:10 [INFO] [TRAIN] epoch: 87, iter: 32050/40000, loss: 0.4263, lr: 0.002413, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1881 samples/sec | ETA 01:51:31 2021-05-06 21:12:19 [INFO] [TRAIN] epoch: 87, iter: 32060/40000, loss: 0.3413, lr: 0.002410, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 01:51:09 2021-05-06 21:12:27 [INFO] [TRAIN] epoch: 87, iter: 32070/40000, loss: 0.4977, lr: 0.002408, batch_cost: 0.8395, reader_cost: 0.00014, ips: 1.1912 samples/sec | ETA 01:50:57 2021-05-06 21:12:35 [INFO] [TRAIN] epoch: 87, iter: 32080/40000, loss: 0.3699, lr: 0.002405, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1884 samples/sec | ETA 01:51:04 2021-05-06 21:12:44 [INFO] [TRAIN] epoch: 87, iter: 32090/40000, loss: 0.1357, lr: 0.002402, batch_cost: 0.8410, reader_cost: 0.00014, ips: 1.1891 samples/sec | ETA 01:50:52 2021-05-06 21:12:52 [INFO] [TRAIN] epoch: 87, iter: 32100/40000, loss: 0.2173, lr: 0.002400, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1903 samples/sec | ETA 01:50:36 2021-05-06 21:13:01 [INFO] [TRAIN] epoch: 87, iter: 32110/40000, loss: 0.2951, lr: 0.002397, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 01:50:27 2021-05-06 21:13:09 [INFO] [TRAIN] epoch: 87, iter: 32120/40000, loss: 0.3117, lr: 0.002395, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 01:50:24 2021-05-06 21:13:17 [INFO] [TRAIN] epoch: 87, iter: 32130/40000, loss: 0.5465, lr: 0.002392, batch_cost: 0.8403, reader_cost: 0.00014, ips: 1.1901 samples/sec | ETA 01:50:12 2021-05-06 21:13:26 [INFO] [TRAIN] epoch: 87, iter: 32140/40000, loss: 0.4177, lr: 0.002389, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1882 samples/sec | ETA 01:50:15 2021-05-06 21:13:34 [INFO] [TRAIN] epoch: 87, iter: 32150/40000, loss: 0.2896, lr: 0.002387, batch_cost: 0.8405, reader_cost: 0.00014, ips: 1.1898 samples/sec | ETA 01:49:57 2021-05-06 21:13:43 [INFO] [TRAIN] epoch: 87, iter: 32160/40000, loss: 0.3832, lr: 0.002384, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 01:49:44 2021-05-06 21:13:51 [INFO] [TRAIN] epoch: 87, iter: 32170/40000, loss: 0.3557, lr: 0.002381, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 01:49:40 2021-05-06 21:14:00 [INFO] [TRAIN] epoch: 87, iter: 32180/40000, loss: 0.6209, lr: 0.002379, batch_cost: 0.8400, reader_cost: 0.00014, ips: 1.1905 samples/sec | ETA 01:49:28 2021-05-06 21:14:08 [INFO] [TRAIN] epoch: 87, iter: 32190/40000, loss: 0.4226, lr: 0.002376, batch_cost: 0.8387, reader_cost: 0.00015, ips: 1.1923 samples/sec | ETA 01:49:10 2021-05-06 21:14:16 [INFO] [TRAIN] epoch: 87, iter: 32200/40000, loss: 0.3259, lr: 0.002374, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 01:49:08 2021-05-06 21:14:25 [INFO] [TRAIN] epoch: 87, iter: 32210/40000, loss: 0.3172, lr: 0.002371, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 01:49:09 2021-05-06 21:14:33 [INFO] [TRAIN] epoch: 87, iter: 32220/40000, loss: 0.1854, lr: 0.002368, batch_cost: 0.8393, reader_cost: 0.00014, ips: 1.1915 samples/sec | ETA 01:48:49 2021-05-06 21:14:41 [INFO] [TRAIN] epoch: 87, iter: 32230/40000, loss: 0.2611, lr: 0.002366, batch_cost: 0.8397, reader_cost: 0.00014, ips: 1.1909 samples/sec | ETA 01:48:44 2021-05-06 21:14:50 [INFO] [TRAIN] epoch: 87, iter: 32240/40000, loss: 0.2005, lr: 0.002363, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1886 samples/sec | ETA 01:48:48 2021-05-06 21:14:58 [INFO] [TRAIN] epoch: 87, iter: 32250/40000, loss: 0.3059, lr: 0.002360, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 01:48:30 2021-05-06 21:15:07 [INFO] [TRAIN] epoch: 87, iter: 32260/40000, loss: 0.3220, lr: 0.002358, batch_cost: 0.8387, reader_cost: 0.00014, ips: 1.1923 samples/sec | ETA 01:48:11 2021-05-06 21:15:15 [INFO] [TRAIN] epoch: 87, iter: 32270/40000, loss: 0.3370, lr: 0.002355, batch_cost: 0.8391, reader_cost: 0.00014, ips: 1.1918 samples/sec | ETA 01:48:05 2021-05-06 21:15:23 [INFO] [TRAIN] epoch: 87, iter: 32280/40000, loss: 0.3416, lr: 0.002353, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1903 samples/sec | ETA 01:48:05 2021-05-06 21:15:32 [INFO] [TRAIN] epoch: 87, iter: 32290/40000, loss: 0.3342, lr: 0.002350, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 01:47:54 2021-05-06 21:15:40 [INFO] [TRAIN] epoch: 87, iter: 32300/40000, loss: 0.2916, lr: 0.002347, batch_cost: 0.8402, reader_cost: 0.00016, ips: 1.1902 samples/sec | ETA 01:47:49 2021-05-06 21:15:49 [INFO] [TRAIN] epoch: 87, iter: 32310/40000, loss: 0.2745, lr: 0.002345, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 01:47:36 2021-05-06 21:15:57 [INFO] [TRAIN] epoch: 87, iter: 32320/40000, loss: 0.4291, lr: 0.002342, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1883 samples/sec | ETA 01:47:42 2021-05-06 21:16:06 [INFO] [TRAIN] epoch: 87, iter: 32330/40000, loss: 0.2572, lr: 0.002339, batch_cost: 0.8425, reader_cost: 0.00014, ips: 1.1870 samples/sec | ETA 01:47:41 2021-05-06 21:16:14 [INFO] [TRAIN] epoch: 87, iter: 32340/40000, loss: 0.3804, lr: 0.002337, batch_cost: 0.8416, reader_cost: 0.00014, ips: 1.1882 samples/sec | ETA 01:47:26 2021-05-06 21:16:22 [INFO] [TRAIN] epoch: 87, iter: 32350/40000, loss: 0.3201, lr: 0.002334, batch_cost: 0.8398, reader_cost: 0.00014, ips: 1.1908 samples/sec | ETA 01:47:04 2021-05-06 21:16:31 [INFO] [TRAIN] epoch: 87, iter: 32360/40000, loss: 0.3246, lr: 0.002332, batch_cost: 0.8368, reader_cost: 0.00010, ips: 1.1950 samples/sec | ETA 01:46:33 2021-05-06 21:16:42 [INFO] [TRAIN] epoch: 88, iter: 32370/40000, loss: 0.4185, lr: 0.002329, batch_cost: 1.1692, reader_cost: 0.21022, ips: 0.8553 samples/sec | ETA 02:28:40 2021-05-06 21:16:51 [INFO] [TRAIN] epoch: 88, iter: 32380/40000, loss: 0.4006, lr: 0.002326, batch_cost: 0.8405, reader_cost: 0.00031, ips: 1.1898 samples/sec | ETA 01:46:44 2021-05-06 21:16:59 [INFO] [TRAIN] epoch: 88, iter: 32390/40000, loss: 0.3625, lr: 0.002324, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 01:46:37 2021-05-06 21:17:08 [INFO] [TRAIN] epoch: 88, iter: 32400/40000, loss: 0.2457, lr: 0.002321, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1888 samples/sec | ETA 01:46:32 2021-05-06 21:17:16 [INFO] [TRAIN] epoch: 88, iter: 32410/40000, loss: 0.2891, lr: 0.002318, batch_cost: 0.8422, reader_cost: 0.00015, ips: 1.1873 samples/sec | ETA 01:46:32 2021-05-06 21:17:24 [INFO] [TRAIN] epoch: 88, iter: 32420/40000, loss: 0.5610, lr: 0.002316, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1887 samples/sec | ETA 01:46:16 2021-05-06 21:17:33 [INFO] [TRAIN] epoch: 88, iter: 32430/40000, loss: 0.3321, lr: 0.002313, batch_cost: 0.8395, reader_cost: 0.00014, ips: 1.1912 samples/sec | ETA 01:45:54 2021-05-06 21:17:41 [INFO] [TRAIN] epoch: 88, iter: 32440/40000, loss: 0.5457, lr: 0.002311, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 01:45:48 2021-05-06 21:17:50 [INFO] [TRAIN] epoch: 88, iter: 32450/40000, loss: 0.4227, lr: 0.002308, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 01:45:46 2021-05-06 21:17:58 [INFO] [TRAIN] epoch: 88, iter: 32460/40000, loss: 0.2339, lr: 0.002305, batch_cost: 0.8394, reader_cost: 0.00016, ips: 1.1913 samples/sec | ETA 01:45:29 2021-05-06 21:18:06 [INFO] [TRAIN] epoch: 88, iter: 32470/40000, loss: 0.2408, lr: 0.002303, batch_cost: 0.8431, reader_cost: 0.00015, ips: 1.1860 samples/sec | ETA 01:45:48 2021-05-06 21:18:15 [INFO] [TRAIN] epoch: 88, iter: 32480/40000, loss: 0.3440, lr: 0.002300, batch_cost: 0.8419, reader_cost: 0.00015, ips: 1.1877 samples/sec | ETA 01:45:31 2021-05-06 21:18:23 [INFO] [TRAIN] epoch: 88, iter: 32490/40000, loss: 0.3703, lr: 0.002297, batch_cost: 0.8394, reader_cost: 0.00016, ips: 1.1914 samples/sec | ETA 01:45:03 2021-05-06 21:18:32 [INFO] [TRAIN] epoch: 88, iter: 32500/40000, loss: 0.5404, lr: 0.002295, batch_cost: 0.8395, reader_cost: 0.00017, ips: 1.1912 samples/sec | ETA 01:44:56 2021-05-06 21:18:40 [INFO] [TRAIN] epoch: 88, iter: 32510/40000, loss: 0.4065, lr: 0.002292, batch_cost: 0.8407, reader_cost: 0.00016, ips: 1.1895 samples/sec | ETA 01:44:56 2021-05-06 21:18:49 [INFO] [TRAIN] epoch: 88, iter: 32520/40000, loss: 0.1797, lr: 0.002289, batch_cost: 0.8392, reader_cost: 0.00017, ips: 1.1915 samples/sec | ETA 01:44:37 2021-05-06 21:18:57 [INFO] [TRAIN] epoch: 88, iter: 32530/40000, loss: 0.3560, lr: 0.002287, batch_cost: 0.8406, reader_cost: 0.00018, ips: 1.1896 samples/sec | ETA 01:44:39 2021-05-06 21:19:05 [INFO] [TRAIN] epoch: 88, iter: 32540/40000, loss: 0.3121, lr: 0.002284, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 01:44:23 2021-05-06 21:19:14 [INFO] [TRAIN] epoch: 88, iter: 32550/40000, loss: 0.7204, lr: 0.002282, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 01:44:21 2021-05-06 21:19:22 [INFO] [TRAIN] epoch: 88, iter: 32560/40000, loss: 0.6855, lr: 0.002279, batch_cost: 0.8388, reader_cost: 0.00015, ips: 1.1921 samples/sec | ETA 01:44:00 2021-05-06 21:19:31 [INFO] [TRAIN] epoch: 88, iter: 32570/40000, loss: 0.3549, lr: 0.002276, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 01:44:05 2021-05-06 21:19:39 [INFO] [TRAIN] epoch: 88, iter: 32580/40000, loss: 0.2551, lr: 0.002274, batch_cost: 0.8414, reader_cost: 0.00016, ips: 1.1885 samples/sec | ETA 01:44:03 2021-05-06 21:19:47 [INFO] [TRAIN] epoch: 88, iter: 32590/40000, loss: 0.2062, lr: 0.002271, batch_cost: 0.8408, reader_cost: 0.00014, ips: 1.1893 samples/sec | ETA 01:43:50 2021-05-06 21:19:56 [INFO] [TRAIN] epoch: 88, iter: 32600/40000, loss: 0.3274, lr: 0.002268, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1918 samples/sec | ETA 01:43:29 2021-05-06 21:20:04 [INFO] [TRAIN] epoch: 88, iter: 32610/40000, loss: 0.4283, lr: 0.002266, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 01:43:34 2021-05-06 21:20:13 [INFO] [TRAIN] epoch: 88, iter: 32620/40000, loss: 0.3159, lr: 0.002263, batch_cost: 0.8398, reader_cost: 0.00014, ips: 1.1908 samples/sec | ETA 01:43:17 2021-05-06 21:20:21 [INFO] [TRAIN] epoch: 88, iter: 32630/40000, loss: 0.3533, lr: 0.002261, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 01:43:18 2021-05-06 21:20:29 [INFO] [TRAIN] epoch: 88, iter: 32640/40000, loss: 0.2453, lr: 0.002258, batch_cost: 0.8397, reader_cost: 0.00016, ips: 1.1909 samples/sec | ETA 01:43:00 2021-05-06 21:20:38 [INFO] [TRAIN] epoch: 88, iter: 32650/40000, loss: 0.2842, lr: 0.002255, batch_cost: 0.8390, reader_cost: 0.00016, ips: 1.1918 samples/sec | ETA 01:42:46 2021-05-06 21:20:46 [INFO] [TRAIN] epoch: 88, iter: 32660/40000, loss: 0.3444, lr: 0.002253, batch_cost: 0.8396, reader_cost: 0.00014, ips: 1.1910 samples/sec | ETA 01:42:42 2021-05-06 21:20:55 [INFO] [TRAIN] epoch: 88, iter: 32670/40000, loss: 0.3425, lr: 0.002250, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1917 samples/sec | ETA 01:42:30 2021-05-06 21:21:03 [INFO] [TRAIN] epoch: 88, iter: 32680/40000, loss: 0.2256, lr: 0.002247, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 01:42:25 2021-05-06 21:21:11 [INFO] [TRAIN] epoch: 88, iter: 32690/40000, loss: 0.4763, lr: 0.002245, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 01:42:22 2021-05-06 21:21:20 [INFO] [TRAIN] epoch: 88, iter: 32700/40000, loss: 0.2578, lr: 0.002242, batch_cost: 0.8421, reader_cost: 0.00015, ips: 1.1875 samples/sec | ETA 01:42:27 2021-05-06 21:21:28 [INFO] [TRAIN] epoch: 88, iter: 32710/40000, loss: 0.3289, lr: 0.002239, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 01:42:11 2021-05-06 21:21:37 [INFO] [TRAIN] epoch: 88, iter: 32720/40000, loss: 0.2466, lr: 0.002237, batch_cost: 0.8420, reader_cost: 0.00015, ips: 1.1877 samples/sec | ETA 01:42:09 2021-05-06 21:21:45 [INFO] [TRAIN] epoch: 88, iter: 32730/40000, loss: 0.3762, lr: 0.002234, batch_cost: 0.8384, reader_cost: 0.00011, ips: 1.1927 samples/sec | ETA 01:41:35 2021-05-06 21:21:56 [INFO] [TRAIN] epoch: 89, iter: 32740/40000, loss: 0.3395, lr: 0.002231, batch_cost: 1.1420, reader_cost: 0.20998, ips: 0.8757 samples/sec | ETA 02:18:10 2021-05-06 21:22:05 [INFO] [TRAIN] epoch: 89, iter: 32750/40000, loss: 0.2945, lr: 0.002229, batch_cost: 0.8499, reader_cost: 0.00031, ips: 1.1766 samples/sec | ETA 01:42:41 2021-05-06 21:22:13 [INFO] [TRAIN] epoch: 89, iter: 32760/40000, loss: 0.3454, lr: 0.002226, batch_cost: 0.8417, reader_cost: 0.00016, ips: 1.1880 samples/sec | ETA 01:41:34 2021-05-06 21:22:22 [INFO] [TRAIN] epoch: 89, iter: 32770/40000, loss: 0.2374, lr: 0.002224, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 01:41:16 2021-05-06 21:22:30 [INFO] [TRAIN] epoch: 89, iter: 32780/40000, loss: 0.2967, lr: 0.002221, batch_cost: 0.8405, reader_cost: 0.00016, ips: 1.1897 samples/sec | ETA 01:41:08 2021-05-06 21:22:39 [INFO] [TRAIN] epoch: 89, iter: 32790/40000, loss: 0.4503, lr: 0.002218, batch_cost: 0.8420, reader_cost: 0.00016, ips: 1.1877 samples/sec | ETA 01:41:10 2021-05-06 21:22:47 [INFO] [TRAIN] epoch: 89, iter: 32800/40000, loss: 0.5815, lr: 0.002216, batch_cost: 0.8395, reader_cost: 0.00016, ips: 1.1911 samples/sec | ETA 01:40:44 2021-05-06 21:22:55 [INFO] [TRAIN] epoch: 89, iter: 32810/40000, loss: 0.4983, lr: 0.002213, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 01:40:43 2021-05-06 21:23:04 [INFO] [TRAIN] epoch: 89, iter: 32820/40000, loss: 0.4201, lr: 0.002210, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 01:40:30 2021-05-06 21:23:12 [INFO] [TRAIN] epoch: 89, iter: 32830/40000, loss: 0.2021, lr: 0.002208, batch_cost: 0.8415, reader_cost: 0.00014, ips: 1.1884 samples/sec | ETA 01:40:33 2021-05-06 21:23:21 [INFO] [TRAIN] epoch: 89, iter: 32840/40000, loss: 0.2346, lr: 0.002205, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 01:40:16 2021-05-06 21:23:29 [INFO] [TRAIN] epoch: 89, iter: 32850/40000, loss: 0.2922, lr: 0.002202, batch_cost: 0.8422, reader_cost: 0.00016, ips: 1.1874 samples/sec | ETA 01:40:21 2021-05-06 21:23:37 [INFO] [TRAIN] epoch: 89, iter: 32860/40000, loss: 0.4253, lr: 0.002200, batch_cost: 0.8421, reader_cost: 0.00016, ips: 1.1876 samples/sec | ETA 01:40:12 2021-05-06 21:23:46 [INFO] [TRAIN] epoch: 89, iter: 32870/40000, loss: 0.4538, lr: 0.002197, batch_cost: 0.8407, reader_cost: 0.00016, ips: 1.1895 samples/sec | ETA 01:39:54 2021-05-06 21:23:54 [INFO] [TRAIN] epoch: 89, iter: 32880/40000, loss: 0.4910, lr: 0.002194, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 01:39:39 2021-05-06 21:24:03 [INFO] [TRAIN] epoch: 89, iter: 32890/40000, loss: 0.2626, lr: 0.002192, batch_cost: 0.8415, reader_cost: 0.00016, ips: 1.1883 samples/sec | ETA 01:39:43 2021-05-06 21:24:11 [INFO] [TRAIN] epoch: 89, iter: 32900/40000, loss: 0.3415, lr: 0.002189, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 01:39:24 2021-05-06 21:24:19 [INFO] [TRAIN] epoch: 89, iter: 32910/40000, loss: 0.2712, lr: 0.002186, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 01:39:14 2021-05-06 21:24:28 [INFO] [TRAIN] epoch: 89, iter: 32920/40000, loss: 0.5387, lr: 0.002184, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 01:39:12 2021-05-06 21:24:36 [INFO] [TRAIN] epoch: 89, iter: 32930/40000, loss: 0.5509, lr: 0.002181, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 01:38:58 2021-05-06 21:24:45 [INFO] [TRAIN] epoch: 89, iter: 32940/40000, loss: 0.3547, lr: 0.002179, batch_cost: 0.8406, reader_cost: 0.00017, ips: 1.1896 samples/sec | ETA 01:38:54 2021-05-06 21:24:53 [INFO] [TRAIN] epoch: 89, iter: 32950/40000, loss: 0.3174, lr: 0.002176, batch_cost: 0.8437, reader_cost: 0.00015, ips: 1.1852 samples/sec | ETA 01:39:08 2021-05-06 21:25:01 [INFO] [TRAIN] epoch: 89, iter: 32960/40000, loss: 0.1370, lr: 0.002173, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 01:38:35 2021-05-06 21:25:10 [INFO] [TRAIN] epoch: 89, iter: 32970/40000, loss: 0.3609, lr: 0.002171, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 01:38:23 2021-05-06 21:25:18 [INFO] [TRAIN] epoch: 89, iter: 32980/40000, loss: 0.2160, lr: 0.002168, batch_cost: 0.8396, reader_cost: 0.00014, ips: 1.1911 samples/sec | ETA 01:38:13 2021-05-06 21:25:27 [INFO] [TRAIN] epoch: 89, iter: 32990/40000, loss: 0.3022, lr: 0.002165, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 01:38:13 2021-05-06 21:25:35 [INFO] [TRAIN] epoch: 89, iter: 33000/40000, loss: 0.3202, lr: 0.002163, batch_cost: 0.8394, reader_cost: 0.00016, ips: 1.1913 samples/sec | ETA 01:37:55 2021-05-06 21:25:35 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 21:29:12 [INFO] [EVAL] #Images: 500 mIoU: 0.7697 Acc: 0.9585 Kappa: 0.9461 2021-05-06 21:29:12 [INFO] [EVAL] Class IoU: [0.981 0.8492 0.9211 0.6743 0.6011 0.546 0.6208 0.7301 0.9197 0.6657 0.9457 0.7875 0.5796 0.9412 0.7865 0.9003 0.7997 0.6424 0.7329] 2021-05-06 21:29:12 [INFO] [EVAL] Class Acc: [0.991 0.9179 0.9522 0.8498 0.8401 0.8006 0.7824 0.8848 0.9505 0.8375 0.9703 0.8718 0.7513 0.9636 0.8729 0.9568 0.902 0.789 0.8346] 2021-05-06 21:29:40 [INFO] [EVAL] The model with the best validation mIoU (0.7713) was saved at iter 32000. 2021-05-06 21:29:49 [INFO] [TRAIN] epoch: 89, iter: 33010/40000, loss: 0.2726, lr: 0.002160, batch_cost: 0.8363, reader_cost: 0.00025, ips: 1.1957 samples/sec | ETA 01:37:25 2021-05-06 21:29:57 [INFO] [TRAIN] epoch: 89, iter: 33020/40000, loss: 0.2983, lr: 0.002157, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1917 samples/sec | ETA 01:37:37 2021-05-06 21:30:06 [INFO] [TRAIN] epoch: 89, iter: 33030/40000, loss: 0.6160, lr: 0.002155, batch_cost: 0.8397, reader_cost: 0.00017, ips: 1.1910 samples/sec | ETA 01:37:32 2021-05-06 21:30:14 [INFO] [TRAIN] epoch: 89, iter: 33040/40000, loss: 0.3133, lr: 0.002152, batch_cost: 0.8404, reader_cost: 0.00016, ips: 1.1899 samples/sec | ETA 01:37:29 2021-05-06 21:30:22 [INFO] [TRAIN] epoch: 89, iter: 33050/40000, loss: 0.2221, lr: 0.002149, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1887 samples/sec | ETA 01:37:26 2021-05-06 21:30:31 [INFO] [TRAIN] epoch: 89, iter: 33060/40000, loss: 0.6565, lr: 0.002147, batch_cost: 0.8399, reader_cost: 0.00014, ips: 1.1906 samples/sec | ETA 01:37:09 2021-05-06 21:30:39 [INFO] [TRAIN] epoch: 89, iter: 33070/40000, loss: 0.2477, lr: 0.002144, batch_cost: 0.8407, reader_cost: 0.00017, ips: 1.1895 samples/sec | ETA 01:37:06 2021-05-06 21:30:48 [INFO] [TRAIN] epoch: 89, iter: 33080/40000, loss: 0.3782, lr: 0.002141, batch_cost: 0.8410, reader_cost: 0.00016, ips: 1.1890 samples/sec | ETA 01:36:59 2021-05-06 21:30:56 [INFO] [TRAIN] epoch: 89, iter: 33090/40000, loss: 0.2734, lr: 0.002139, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 01:36:50 2021-05-06 21:31:04 [INFO] [TRAIN] epoch: 89, iter: 33100/40000, loss: 0.3548, lr: 0.002136, batch_cost: 0.8398, reader_cost: 0.00012, ips: 1.1907 samples/sec | ETA 01:36:34 2021-05-06 21:31:16 [INFO] [TRAIN] epoch: 90, iter: 33110/40000, loss: 0.4598, lr: 0.002133, batch_cost: 1.1723, reader_cost: 0.21782, ips: 0.8530 samples/sec | ETA 02:14:37 2021-05-06 21:31:25 [INFO] [TRAIN] epoch: 90, iter: 33120/40000, loss: 0.4641, lr: 0.002131, batch_cost: 0.8715, reader_cost: 0.00033, ips: 1.1474 samples/sec | ETA 01:39:56 2021-05-06 21:31:33 [INFO] [TRAIN] epoch: 90, iter: 33130/40000, loss: 0.4175, lr: 0.002128, batch_cost: 0.8437, reader_cost: 0.00016, ips: 1.1852 samples/sec | ETA 01:36:36 2021-05-06 21:31:42 [INFO] [TRAIN] epoch: 90, iter: 33140/40000, loss: 0.3397, lr: 0.002125, batch_cost: 0.8424, reader_cost: 0.00016, ips: 1.1871 samples/sec | ETA 01:36:18 2021-05-06 21:31:50 [INFO] [TRAIN] epoch: 90, iter: 33150/40000, loss: 0.2525, lr: 0.002123, batch_cost: 0.8392, reader_cost: 0.00014, ips: 1.1917 samples/sec | ETA 01:35:48 2021-05-06 21:31:58 [INFO] [TRAIN] epoch: 90, iter: 33160/40000, loss: 0.5153, lr: 0.002120, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 01:35:53 2021-05-06 21:32:07 [INFO] [TRAIN] epoch: 90, iter: 33170/40000, loss: 0.4612, lr: 0.002118, batch_cost: 0.8419, reader_cost: 0.00015, ips: 1.1878 samples/sec | ETA 01:35:49 2021-05-06 21:32:15 [INFO] [TRAIN] epoch: 90, iter: 33180/40000, loss: 0.3947, lr: 0.002115, batch_cost: 0.8391, reader_cost: 0.00014, ips: 1.1918 samples/sec | ETA 01:35:22 2021-05-06 21:32:24 [INFO] [TRAIN] epoch: 90, iter: 33190/40000, loss: 0.4826, lr: 0.002112, batch_cost: 0.8384, reader_cost: 0.00015, ips: 1.1928 samples/sec | ETA 01:35:09 2021-05-06 21:32:32 [INFO] [TRAIN] epoch: 90, iter: 33200/40000, loss: 0.2855, lr: 0.002110, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 01:35:11 2021-05-06 21:32:41 [INFO] [TRAIN] epoch: 90, iter: 33210/40000, loss: 0.2002, lr: 0.002107, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 01:35:10 2021-05-06 21:32:49 [INFO] [TRAIN] epoch: 90, iter: 33220/40000, loss: 0.3108, lr: 0.002104, batch_cost: 0.8425, reader_cost: 0.00016, ips: 1.1869 samples/sec | ETA 01:35:12 2021-05-06 21:32:57 [INFO] [TRAIN] epoch: 90, iter: 33230/40000, loss: 0.3758, lr: 0.002102, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 01:34:49 2021-05-06 21:33:06 [INFO] [TRAIN] epoch: 90, iter: 33240/40000, loss: 0.4687, lr: 0.002099, batch_cost: 0.8398, reader_cost: 0.00014, ips: 1.1908 samples/sec | ETA 01:34:36 2021-05-06 21:33:14 [INFO] [TRAIN] epoch: 90, iter: 33250/40000, loss: 0.5267, lr: 0.002096, batch_cost: 0.8398, reader_cost: 0.00014, ips: 1.1908 samples/sec | ETA 01:34:28 2021-05-06 21:33:23 [INFO] [TRAIN] epoch: 90, iter: 33260/40000, loss: 0.3391, lr: 0.002094, batch_cost: 0.8422, reader_cost: 0.00014, ips: 1.1874 samples/sec | ETA 01:34:36 2021-05-06 21:33:31 [INFO] [TRAIN] epoch: 90, iter: 33270/40000, loss: 0.2382, lr: 0.002091, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 01:34:18 2021-05-06 21:33:39 [INFO] [TRAIN] epoch: 90, iter: 33280/40000, loss: 0.3401, lr: 0.002088, batch_cost: 0.8411, reader_cost: 0.00016, ips: 1.1890 samples/sec | ETA 01:34:11 2021-05-06 21:33:48 [INFO] [TRAIN] epoch: 90, iter: 33290/40000, loss: 0.4490, lr: 0.002086, batch_cost: 0.8403, reader_cost: 0.00016, ips: 1.1901 samples/sec | ETA 01:33:58 2021-05-06 21:33:56 [INFO] [TRAIN] epoch: 90, iter: 33300/40000, loss: 0.4553, lr: 0.002083, batch_cost: 0.8386, reader_cost: 0.00015, ips: 1.1925 samples/sec | ETA 01:33:38 2021-05-06 21:34:05 [INFO] [TRAIN] epoch: 90, iter: 33310/40000, loss: 0.3885, lr: 0.002080, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 01:33:41 2021-05-06 21:34:13 [INFO] [TRAIN] epoch: 90, iter: 33320/40000, loss: 0.4558, lr: 0.002078, batch_cost: 0.8411, reader_cost: 0.00016, ips: 1.1890 samples/sec | ETA 01:33:38 2021-05-06 21:34:21 [INFO] [TRAIN] epoch: 90, iter: 33330/40000, loss: 0.2959, lr: 0.002075, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 01:33:25 2021-05-06 21:34:30 [INFO] [TRAIN] epoch: 90, iter: 33340/40000, loss: 0.1861, lr: 0.002072, batch_cost: 0.8420, reader_cost: 0.00016, ips: 1.1877 samples/sec | ETA 01:33:27 2021-05-06 21:34:38 [INFO] [TRAIN] epoch: 90, iter: 33350/40000, loss: 0.2957, lr: 0.002070, batch_cost: 0.8399, reader_cost: 0.00016, ips: 1.1907 samples/sec | ETA 01:33:05 2021-05-06 21:34:47 [INFO] [TRAIN] epoch: 90, iter: 33360/40000, loss: 0.2625, lr: 0.002067, batch_cost: 0.8417, reader_cost: 0.00015, ips: 1.1881 samples/sec | ETA 01:33:08 2021-05-06 21:34:55 [INFO] [TRAIN] epoch: 90, iter: 33370/40000, loss: 0.3008, lr: 0.002064, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1888 samples/sec | ETA 01:32:57 2021-05-06 21:35:03 [INFO] [TRAIN] epoch: 90, iter: 33380/40000, loss: 0.3140, lr: 0.002062, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1913 samples/sec | ETA 01:32:36 2021-05-06 21:35:12 [INFO] [TRAIN] epoch: 90, iter: 33390/40000, loss: 0.2293, lr: 0.002059, batch_cost: 0.8409, reader_cost: 0.00014, ips: 1.1892 samples/sec | ETA 01:32:38 2021-05-06 21:35:20 [INFO] [TRAIN] epoch: 90, iter: 33400/40000, loss: 0.3412, lr: 0.002056, batch_cost: 0.8415, reader_cost: 0.00014, ips: 1.1883 samples/sec | ETA 01:32:34 2021-05-06 21:35:29 [INFO] [TRAIN] epoch: 90, iter: 33410/40000, loss: 0.3666, lr: 0.002054, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 01:32:16 2021-05-06 21:35:37 [INFO] [TRAIN] epoch: 90, iter: 33420/40000, loss: 0.3008, lr: 0.002051, batch_cost: 0.8402, reader_cost: 0.00016, ips: 1.1903 samples/sec | ETA 01:32:08 2021-05-06 21:35:45 [INFO] [TRAIN] epoch: 90, iter: 33430/40000, loss: 0.4919, lr: 0.002048, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 01:32:06 2021-05-06 21:35:54 [INFO] [TRAIN] epoch: 90, iter: 33440/40000, loss: 0.2363, lr: 0.002046, batch_cost: 0.8405, reader_cost: 0.00016, ips: 1.1897 samples/sec | ETA 01:31:53 2021-05-06 21:36:02 [INFO] [TRAIN] epoch: 90, iter: 33450/40000, loss: 0.3380, lr: 0.002043, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 01:31:41 2021-05-06 21:36:11 [INFO] [TRAIN] epoch: 90, iter: 33460/40000, loss: 0.2923, lr: 0.002040, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1918 samples/sec | ETA 01:31:27 2021-05-06 21:36:19 [INFO] [TRAIN] epoch: 90, iter: 33470/40000, loss: 0.3464, lr: 0.002038, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 01:31:20 2021-05-06 21:36:27 [INFO] [TRAIN] epoch: 90, iter: 33480/40000, loss: 0.3419, lr: 0.002035, batch_cost: 0.8398, reader_cost: 0.00010, ips: 1.1907 samples/sec | ETA 01:31:15 2021-05-06 21:36:39 [INFO] [TRAIN] epoch: 91, iter: 33490/40000, loss: 0.4072, lr: 0.002032, batch_cost: 1.1364, reader_cost: 0.18755, ips: 0.8800 samples/sec | ETA 02:03:17 2021-05-06 21:36:47 [INFO] [TRAIN] epoch: 91, iter: 33500/40000, loss: 0.4119, lr: 0.002030, batch_cost: 0.8420, reader_cost: 0.00031, ips: 1.1876 samples/sec | ETA 01:31:13 2021-05-06 21:36:56 [INFO] [TRAIN] epoch: 91, iter: 33510/40000, loss: 0.3540, lr: 0.002027, batch_cost: 0.8418, reader_cost: 0.00016, ips: 1.1879 samples/sec | ETA 01:31:03 2021-05-06 21:37:04 [INFO] [TRAIN] epoch: 91, iter: 33520/40000, loss: 0.1954, lr: 0.002024, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 01:30:48 2021-05-06 21:37:12 [INFO] [TRAIN] epoch: 91, iter: 33530/40000, loss: 0.4260, lr: 0.002022, batch_cost: 0.8414, reader_cost: 0.00014, ips: 1.1885 samples/sec | ETA 01:30:43 2021-05-06 21:37:21 [INFO] [TRAIN] epoch: 91, iter: 33540/40000, loss: 0.4305, lr: 0.002019, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 01:30:32 2021-05-06 21:37:29 [INFO] [TRAIN] epoch: 91, iter: 33550/40000, loss: 0.4414, lr: 0.002016, batch_cost: 0.8414, reader_cost: 0.00016, ips: 1.1885 samples/sec | ETA 01:30:27 2021-05-06 21:37:38 [INFO] [TRAIN] epoch: 91, iter: 33560/40000, loss: 0.3844, lr: 0.002014, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1914 samples/sec | ETA 01:30:05 2021-05-06 21:37:46 [INFO] [TRAIN] epoch: 91, iter: 33570/40000, loss: 0.4056, lr: 0.002011, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 01:30:08 2021-05-06 21:37:55 [INFO] [TRAIN] epoch: 91, iter: 33580/40000, loss: 0.1289, lr: 0.002008, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 01:29:55 2021-05-06 21:38:03 [INFO] [TRAIN] epoch: 91, iter: 33590/40000, loss: 0.2347, lr: 0.002006, batch_cost: 0.8392, reader_cost: 0.00015, ips: 1.1916 samples/sec | ETA 01:29:39 2021-05-06 21:38:11 [INFO] [TRAIN] epoch: 91, iter: 33600/40000, loss: 0.3830, lr: 0.002003, batch_cost: 0.8406, reader_cost: 0.00016, ips: 1.1897 samples/sec | ETA 01:29:39 2021-05-06 21:38:20 [INFO] [TRAIN] epoch: 91, iter: 33610/40000, loss: 0.4613, lr: 0.002000, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 01:29:26 2021-05-06 21:38:28 [INFO] [TRAIN] epoch: 91, iter: 33620/40000, loss: 0.4964, lr: 0.001997, batch_cost: 0.8394, reader_cost: 0.00016, ips: 1.1914 samples/sec | ETA 01:29:15 2021-05-06 21:38:37 [INFO] [TRAIN] epoch: 91, iter: 33630/40000, loss: 0.4853, lr: 0.001995, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 01:29:13 2021-05-06 21:38:45 [INFO] [TRAIN] epoch: 91, iter: 33640/40000, loss: 0.2890, lr: 0.001992, batch_cost: 0.8403, reader_cost: 0.00014, ips: 1.1900 samples/sec | ETA 01:29:04 2021-05-06 21:38:53 [INFO] [TRAIN] epoch: 91, iter: 33650/40000, loss: 0.3464, lr: 0.001989, batch_cost: 0.8378, reader_cost: 0.00015, ips: 1.1936 samples/sec | ETA 01:28:40 2021-05-06 21:39:02 [INFO] [TRAIN] epoch: 91, iter: 33660/40000, loss: 0.2755, lr: 0.001987, batch_cost: 0.8386, reader_cost: 0.00015, ips: 1.1924 samples/sec | ETA 01:28:36 2021-05-06 21:39:10 [INFO] [TRAIN] epoch: 91, iter: 33670/40000, loss: 0.4239, lr: 0.001984, batch_cost: 0.8387, reader_cost: 0.00014, ips: 1.1923 samples/sec | ETA 01:28:29 2021-05-06 21:39:18 [INFO] [TRAIN] epoch: 91, iter: 33680/40000, loss: 0.5386, lr: 0.001981, batch_cost: 0.8409, reader_cost: 0.00014, ips: 1.1892 samples/sec | ETA 01:28:34 2021-05-06 21:39:27 [INFO] [TRAIN] epoch: 91, iter: 33690/40000, loss: 0.2888, lr: 0.001979, batch_cost: 0.8389, reader_cost: 0.00014, ips: 1.1921 samples/sec | ETA 01:28:13 2021-05-06 21:39:35 [INFO] [TRAIN] epoch: 91, iter: 33700/40000, loss: 0.2727, lr: 0.001976, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 01:28:14 2021-05-06 21:39:44 [INFO] [TRAIN] epoch: 91, iter: 33710/40000, loss: 0.1570, lr: 0.001973, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 01:28:03 2021-05-06 21:39:52 [INFO] [TRAIN] epoch: 91, iter: 33720/40000, loss: 0.2902, lr: 0.001971, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1887 samples/sec | ETA 01:28:02 2021-05-06 21:40:00 [INFO] [TRAIN] epoch: 91, iter: 33730/40000, loss: 0.2903, lr: 0.001968, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 01:27:48 2021-05-06 21:40:09 [INFO] [TRAIN] epoch: 91, iter: 33740/40000, loss: 0.3265, lr: 0.001965, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 01:27:39 2021-05-06 21:40:17 [INFO] [TRAIN] epoch: 91, iter: 33750/40000, loss: 0.2894, lr: 0.001963, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 01:27:30 2021-05-06 21:40:26 [INFO] [TRAIN] epoch: 91, iter: 33760/40000, loss: 0.2419, lr: 0.001960, batch_cost: 0.8417, reader_cost: 0.00016, ips: 1.1880 samples/sec | ETA 01:27:32 2021-05-06 21:40:34 [INFO] [TRAIN] epoch: 91, iter: 33770/40000, loss: 0.3730, lr: 0.001957, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 01:27:13 2021-05-06 21:40:43 [INFO] [TRAIN] epoch: 91, iter: 33780/40000, loss: 0.5421, lr: 0.001955, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 01:27:08 2021-05-06 21:40:51 [INFO] [TRAIN] epoch: 91, iter: 33790/40000, loss: 0.2829, lr: 0.001952, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1918 samples/sec | ETA 01:26:50 2021-05-06 21:40:59 [INFO] [TRAIN] epoch: 91, iter: 33800/40000, loss: 0.3128, lr: 0.001949, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1904 samples/sec | ETA 01:26:48 2021-05-06 21:41:08 [INFO] [TRAIN] epoch: 91, iter: 33810/40000, loss: 0.2926, lr: 0.001947, batch_cost: 0.8411, reader_cost: 0.00014, ips: 1.1889 samples/sec | ETA 01:26:46 2021-05-06 21:41:16 [INFO] [TRAIN] epoch: 91, iter: 33820/40000, loss: 0.3391, lr: 0.001944, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 01:26:33 2021-05-06 21:41:25 [INFO] [TRAIN] epoch: 91, iter: 33830/40000, loss: 0.2522, lr: 0.001941, batch_cost: 0.8400, reader_cost: 0.00014, ips: 1.1905 samples/sec | ETA 01:26:22 2021-05-06 21:41:33 [INFO] [TRAIN] epoch: 91, iter: 33840/40000, loss: 0.3737, lr: 0.001939, batch_cost: 0.8410, reader_cost: 0.00016, ips: 1.1891 samples/sec | ETA 01:26:20 2021-05-06 21:41:41 [INFO] [TRAIN] epoch: 91, iter: 33850/40000, loss: 0.3558, lr: 0.001936, batch_cost: 0.8385, reader_cost: 0.00028, ips: 1.1926 samples/sec | ETA 01:25:56 2021-05-06 21:41:53 [INFO] [TRAIN] epoch: 92, iter: 33860/40000, loss: 0.3989, lr: 0.001933, batch_cost: 1.1567, reader_cost: 0.20185, ips: 0.8645 samples/sec | ETA 01:58:22 2021-05-06 21:42:01 [INFO] [TRAIN] epoch: 92, iter: 33870/40000, loss: 0.3127, lr: 0.001930, batch_cost: 0.8426, reader_cost: 0.00032, ips: 1.1869 samples/sec | ETA 01:26:04 2021-05-06 21:42:10 [INFO] [TRAIN] epoch: 92, iter: 33880/40000, loss: 0.2942, lr: 0.001928, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 01:25:45 2021-05-06 21:42:18 [INFO] [TRAIN] epoch: 92, iter: 33890/40000, loss: 0.2419, lr: 0.001925, batch_cost: 0.8414, reader_cost: 0.00015, ips: 1.1885 samples/sec | ETA 01:25:40 2021-05-06 21:42:27 [INFO] [TRAIN] epoch: 92, iter: 33900/40000, loss: 0.3113, lr: 0.001922, batch_cost: 0.8398, reader_cost: 0.00016, ips: 1.1908 samples/sec | ETA 01:25:22 2021-05-06 21:42:35 [INFO] [TRAIN] epoch: 92, iter: 33910/40000, loss: 0.4599, lr: 0.001920, batch_cost: 0.8417, reader_cost: 0.00014, ips: 1.1880 samples/sec | ETA 01:25:26 2021-05-06 21:42:43 [INFO] [TRAIN] epoch: 92, iter: 33920/40000, loss: 0.5045, lr: 0.001917, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 01:25:12 2021-05-06 21:42:52 [INFO] [TRAIN] epoch: 92, iter: 33930/40000, loss: 0.5689, lr: 0.001914, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 01:25:03 2021-05-06 21:43:00 [INFO] [TRAIN] epoch: 92, iter: 33940/40000, loss: 0.3226, lr: 0.001912, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 01:24:54 2021-05-06 21:43:09 [INFO] [TRAIN] epoch: 92, iter: 33950/40000, loss: 0.2365, lr: 0.001909, batch_cost: 0.8424, reader_cost: 0.00015, ips: 1.1871 samples/sec | ETA 01:24:56 2021-05-06 21:43:17 [INFO] [TRAIN] epoch: 92, iter: 33960/40000, loss: 0.2729, lr: 0.001906, batch_cost: 0.8407, reader_cost: 0.00016, ips: 1.1895 samples/sec | ETA 01:24:37 2021-05-06 21:43:25 [INFO] [TRAIN] epoch: 92, iter: 33970/40000, loss: 0.3112, lr: 0.001904, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 01:24:25 2021-05-06 21:43:34 [INFO] [TRAIN] epoch: 92, iter: 33980/40000, loss: 0.3642, lr: 0.001901, batch_cost: 0.8392, reader_cost: 0.00016, ips: 1.1916 samples/sec | ETA 01:24:11 2021-05-06 21:43:42 [INFO] [TRAIN] epoch: 92, iter: 33990/40000, loss: 0.5171, lr: 0.001898, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 01:24:07 2021-05-06 21:43:51 [INFO] [TRAIN] epoch: 92, iter: 34000/40000, loss: 0.2744, lr: 0.001895, batch_cost: 0.8414, reader_cost: 0.00017, ips: 1.1885 samples/sec | ETA 01:24:08 2021-05-06 21:43:51 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 21:47:28 [INFO] [EVAL] #Images: 500 mIoU: 0.7708 Acc: 0.9585 Kappa: 0.9461 2021-05-06 21:47:28 [INFO] [EVAL] Class IoU: [0.9809 0.8484 0.9215 0.674 0.6026 0.5599 0.6216 0.7346 0.9184 0.6479 0.9457 0.7894 0.5934 0.9434 0.801 0.8996 0.7804 0.6494 0.7327] 2021-05-06 21:47:28 [INFO] [EVAL] Class Acc: [0.9923 0.9155 0.9545 0.873 0.8496 0.7948 0.8123 0.8776 0.9465 0.8041 0.9675 0.8753 0.7302 0.9672 0.8799 0.94 0.8978 0.7844 0.8363] 2021-05-06 21:47:56 [INFO] [EVAL] The model with the best validation mIoU (0.7713) was saved at iter 32000. 2021-05-06 21:48:05 [INFO] [TRAIN] epoch: 92, iter: 34010/40000, loss: 0.2471, lr: 0.001893, batch_cost: 0.8363, reader_cost: 0.00040, ips: 1.1957 samples/sec | ETA 01:23:29 2021-05-06 21:48:17 [INFO] [TRAIN] epoch: 92, iter: 34020/40000, loss: 0.2877, lr: 0.001890, batch_cost: 0.8390, reader_cost: 0.00016, ips: 1.1919 samples/sec | ETA 01:23:36 2021-05-06 21:48:26 [INFO] [TRAIN] epoch: 92, iter: 34030/40000, loss: 0.3283, lr: 0.001887, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 01:23:34 2021-05-06 21:48:34 [INFO] [TRAIN] epoch: 92, iter: 34040/40000, loss: 0.4929, lr: 0.001885, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 01:23:27 2021-05-06 21:48:43 [INFO] [TRAIN] epoch: 92, iter: 34050/40000, loss: 0.5335, lr: 0.001882, batch_cost: 0.8409, reader_cost: 0.00016, ips: 1.1892 samples/sec | ETA 01:23:23 2021-05-06 21:48:51 [INFO] [TRAIN] epoch: 92, iter: 34060/40000, loss: 0.3850, lr: 0.001879, batch_cost: 0.8405, reader_cost: 0.00017, ips: 1.1898 samples/sec | ETA 01:23:12 2021-05-06 21:48:59 [INFO] [TRAIN] epoch: 92, iter: 34070/40000, loss: 0.2648, lr: 0.001877, batch_cost: 0.8421, reader_cost: 0.00015, ips: 1.1875 samples/sec | ETA 01:23:13 2021-05-06 21:49:08 [INFO] [TRAIN] epoch: 92, iter: 34080/40000, loss: 0.1877, lr: 0.001874, batch_cost: 0.8399, reader_cost: 0.00014, ips: 1.1906 samples/sec | ETA 01:22:52 2021-05-06 21:49:16 [INFO] [TRAIN] epoch: 92, iter: 34090/40000, loss: 0.2748, lr: 0.001871, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1918 samples/sec | ETA 01:22:38 2021-05-06 21:49:25 [INFO] [TRAIN] epoch: 92, iter: 34100/40000, loss: 0.1576, lr: 0.001869, batch_cost: 0.8424, reader_cost: 0.00015, ips: 1.1871 samples/sec | ETA 01:22:50 2021-05-06 21:49:33 [INFO] [TRAIN] epoch: 92, iter: 34110/40000, loss: 0.3538, lr: 0.001866, batch_cost: 0.8409, reader_cost: 0.00017, ips: 1.1892 samples/sec | ETA 01:22:32 2021-05-06 21:49:41 [INFO] [TRAIN] epoch: 92, iter: 34120/40000, loss: 0.2913, lr: 0.001863, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 01:22:17 2021-05-06 21:49:50 [INFO] [TRAIN] epoch: 92, iter: 34130/40000, loss: 0.2412, lr: 0.001860, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 01:22:15 2021-05-06 21:49:58 [INFO] [TRAIN] epoch: 92, iter: 34140/40000, loss: 0.3374, lr: 0.001858, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 01:22:00 2021-05-06 21:50:07 [INFO] [TRAIN] epoch: 92, iter: 34150/40000, loss: 0.4567, lr: 0.001855, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 01:21:54 2021-05-06 21:50:15 [INFO] [TRAIN] epoch: 92, iter: 34160/40000, loss: 0.2843, lr: 0.001852, batch_cost: 0.8388, reader_cost: 0.00015, ips: 1.1922 samples/sec | ETA 01:21:38 2021-05-06 21:50:23 [INFO] [TRAIN] epoch: 92, iter: 34170/40000, loss: 0.3584, lr: 0.001850, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1914 samples/sec | ETA 01:21:33 2021-05-06 21:50:32 [INFO] [TRAIN] epoch: 92, iter: 34180/40000, loss: 0.3242, lr: 0.001847, batch_cost: 0.8403, reader_cost: 0.00016, ips: 1.1901 samples/sec | ETA 01:21:30 2021-05-06 21:50:40 [INFO] [TRAIN] epoch: 92, iter: 34190/40000, loss: 0.3306, lr: 0.001844, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1882 samples/sec | ETA 01:21:29 2021-05-06 21:50:49 [INFO] [TRAIN] epoch: 92, iter: 34200/40000, loss: 0.4184, lr: 0.001842, batch_cost: 0.8397, reader_cost: 0.00016, ips: 1.1909 samples/sec | ETA 01:21:10 2021-05-06 21:50:57 [INFO] [TRAIN] epoch: 92, iter: 34210/40000, loss: 0.3452, lr: 0.001839, batch_cost: 0.8390, reader_cost: 0.00015, ips: 1.1920 samples/sec | ETA 01:20:57 2021-05-06 21:51:05 [INFO] [TRAIN] epoch: 92, iter: 34220/40000, loss: 0.2609, lr: 0.001836, batch_cost: 0.8365, reader_cost: 0.00011, ips: 1.1955 samples/sec | ETA 01:20:34 2021-05-06 21:51:17 [INFO] [TRAIN] epoch: 93, iter: 34230/40000, loss: 0.3744, lr: 0.001833, batch_cost: 1.1991, reader_cost: 0.23820, ips: 0.8340 samples/sec | ETA 01:55:18 2021-05-06 21:51:26 [INFO] [TRAIN] epoch: 93, iter: 34240/40000, loss: 0.2952, lr: 0.001831, batch_cost: 0.8410, reader_cost: 0.00031, ips: 1.1891 samples/sec | ETA 01:20:43 2021-05-06 21:51:34 [INFO] [TRAIN] epoch: 93, iter: 34250/40000, loss: 0.3615, lr: 0.001828, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 01:20:33 2021-05-06 21:51:43 [INFO] [TRAIN] epoch: 93, iter: 34260/40000, loss: 0.4920, lr: 0.001825, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 01:20:21 2021-05-06 21:51:51 [INFO] [TRAIN] epoch: 93, iter: 34270/40000, loss: 0.3091, lr: 0.001823, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 01:20:15 2021-05-06 21:51:59 [INFO] [TRAIN] epoch: 93, iter: 34280/40000, loss: 0.3319, lr: 0.001820, batch_cost: 0.8437, reader_cost: 0.00015, ips: 1.1852 samples/sec | ETA 01:20:26 2021-05-06 21:52:08 [INFO] [TRAIN] epoch: 93, iter: 34290/40000, loss: 0.5120, lr: 0.001817, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1882 samples/sec | ETA 01:20:05 2021-05-06 21:52:16 [INFO] [TRAIN] epoch: 93, iter: 34300/40000, loss: 0.4452, lr: 0.001814, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1883 samples/sec | ETA 01:19:56 2021-05-06 21:52:25 [INFO] [TRAIN] epoch: 93, iter: 34310/40000, loss: 0.3530, lr: 0.001812, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 01:19:45 2021-05-06 21:52:33 [INFO] [TRAIN] epoch: 93, iter: 34320/40000, loss: 0.1947, lr: 0.001809, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 01:19:36 2021-05-06 21:52:42 [INFO] [TRAIN] epoch: 93, iter: 34330/40000, loss: 0.3149, lr: 0.001806, batch_cost: 0.8419, reader_cost: 0.00017, ips: 1.1878 samples/sec | ETA 01:19:33 2021-05-06 21:52:50 [INFO] [TRAIN] epoch: 93, iter: 34340/40000, loss: 0.3745, lr: 0.001804, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 01:19:17 2021-05-06 21:52:58 [INFO] [TRAIN] epoch: 93, iter: 34350/40000, loss: 0.3799, lr: 0.001801, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1903 samples/sec | ETA 01:19:06 2021-05-06 21:53:07 [INFO] [TRAIN] epoch: 93, iter: 34360/40000, loss: 0.4935, lr: 0.001798, batch_cost: 0.8403, reader_cost: 0.00014, ips: 1.1901 samples/sec | ETA 01:18:59 2021-05-06 21:53:15 [INFO] [TRAIN] epoch: 93, iter: 34370/40000, loss: 0.3598, lr: 0.001796, batch_cost: 0.8402, reader_cost: 0.00016, ips: 1.1902 samples/sec | ETA 01:18:50 2021-05-06 21:53:24 [INFO] [TRAIN] epoch: 93, iter: 34380/40000, loss: 0.2129, lr: 0.001793, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 01:18:43 2021-05-06 21:53:32 [INFO] [TRAIN] epoch: 93, iter: 34390/40000, loss: 0.2579, lr: 0.001790, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 01:18:30 2021-05-06 21:53:40 [INFO] [TRAIN] epoch: 93, iter: 34400/40000, loss: 0.2854, lr: 0.001787, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1888 samples/sec | ETA 01:18:30 2021-05-06 21:53:49 [INFO] [TRAIN] epoch: 93, iter: 34410/40000, loss: 0.3465, lr: 0.001785, batch_cost: 0.8436, reader_cost: 0.00014, ips: 1.1853 samples/sec | ETA 01:18:36 2021-05-06 21:53:57 [INFO] [TRAIN] epoch: 93, iter: 34420/40000, loss: 0.5430, lr: 0.001782, batch_cost: 0.8412, reader_cost: 0.00017, ips: 1.1887 samples/sec | ETA 01:18:14 2021-05-06 21:54:06 [INFO] [TRAIN] epoch: 93, iter: 34430/40000, loss: 0.3411, lr: 0.001779, batch_cost: 0.8419, reader_cost: 0.00014, ips: 1.1877 samples/sec | ETA 01:18:09 2021-05-06 21:54:14 [INFO] [TRAIN] epoch: 93, iter: 34440/40000, loss: 0.2820, lr: 0.001777, batch_cost: 0.8398, reader_cost: 0.00014, ips: 1.1907 samples/sec | ETA 01:17:49 2021-05-06 21:54:22 [INFO] [TRAIN] epoch: 93, iter: 34450/40000, loss: 0.2108, lr: 0.001774, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 01:17:43 2021-05-06 21:54:31 [INFO] [TRAIN] epoch: 93, iter: 34460/40000, loss: 0.4527, lr: 0.001771, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 01:17:29 2021-05-06 21:54:39 [INFO] [TRAIN] epoch: 93, iter: 34470/40000, loss: 0.2377, lr: 0.001768, batch_cost: 0.8395, reader_cost: 0.00014, ips: 1.1911 samples/sec | ETA 01:17:22 2021-05-06 21:54:48 [INFO] [TRAIN] epoch: 93, iter: 34480/40000, loss: 0.3595, lr: 0.001766, batch_cost: 0.8400, reader_cost: 0.00014, ips: 1.1904 samples/sec | ETA 01:17:16 2021-05-06 21:54:56 [INFO] [TRAIN] epoch: 93, iter: 34490/40000, loss: 0.2749, lr: 0.001763, batch_cost: 0.8403, reader_cost: 0.00014, ips: 1.1901 samples/sec | ETA 01:17:09 2021-05-06 21:55:04 [INFO] [TRAIN] epoch: 93, iter: 34500/40000, loss: 0.2461, lr: 0.001760, batch_cost: 0.8392, reader_cost: 0.00014, ips: 1.1916 samples/sec | ETA 01:16:55 2021-05-06 21:55:13 [INFO] [TRAIN] epoch: 93, iter: 34510/40000, loss: 0.2858, lr: 0.001758, batch_cost: 0.8413, reader_cost: 0.00014, ips: 1.1886 samples/sec | ETA 01:16:58 2021-05-06 21:55:21 [INFO] [TRAIN] epoch: 93, iter: 34520/40000, loss: 0.3337, lr: 0.001755, batch_cost: 0.8397, reader_cost: 0.00014, ips: 1.1908 samples/sec | ETA 01:16:41 2021-05-06 21:55:30 [INFO] [TRAIN] epoch: 93, iter: 34530/40000, loss: 0.2562, lr: 0.001752, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1886 samples/sec | ETA 01:16:42 2021-05-06 21:55:38 [INFO] [TRAIN] epoch: 93, iter: 34540/40000, loss: 0.3295, lr: 0.001749, batch_cost: 0.8395, reader_cost: 0.00014, ips: 1.1912 samples/sec | ETA 01:16:23 2021-05-06 21:55:46 [INFO] [TRAIN] epoch: 93, iter: 34550/40000, loss: 0.5393, lr: 0.001747, batch_cost: 0.8393, reader_cost: 0.00014, ips: 1.1914 samples/sec | ETA 01:16:14 2021-05-06 21:55:55 [INFO] [TRAIN] epoch: 93, iter: 34560/40000, loss: 0.3173, lr: 0.001744, batch_cost: 0.8414, reader_cost: 0.00014, ips: 1.1884 samples/sec | ETA 01:16:17 2021-05-06 21:56:03 [INFO] [TRAIN] epoch: 93, iter: 34570/40000, loss: 0.4004, lr: 0.001741, batch_cost: 0.8397, reader_cost: 0.00014, ips: 1.1909 samples/sec | ETA 01:15:59 2021-05-06 21:56:12 [INFO] [TRAIN] epoch: 93, iter: 34580/40000, loss: 0.4076, lr: 0.001739, batch_cost: 0.8404, reader_cost: 0.00014, ips: 1.1899 samples/sec | ETA 01:15:55 2021-05-06 21:56:20 [INFO] [TRAIN] epoch: 93, iter: 34590/40000, loss: 0.3867, lr: 0.001736, batch_cost: 0.8389, reader_cost: 0.00012, ips: 1.1921 samples/sec | ETA 01:15:38 2021-05-06 21:56:32 [INFO] [TRAIN] epoch: 94, iter: 34600/40000, loss: 0.3603, lr: 0.001733, batch_cost: 1.1589, reader_cost: 0.23330, ips: 0.8629 samples/sec | ETA 01:44:18 2021-05-06 21:56:40 [INFO] [TRAIN] epoch: 94, iter: 34610/40000, loss: 0.2874, lr: 0.001730, batch_cost: 0.8437, reader_cost: 0.00034, ips: 1.1853 samples/sec | ETA 01:15:47 2021-05-06 21:56:49 [INFO] [TRAIN] epoch: 94, iter: 34620/40000, loss: 0.3654, lr: 0.001728, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 01:15:18 2021-05-06 21:56:57 [INFO] [TRAIN] epoch: 94, iter: 34630/40000, loss: 0.3689, lr: 0.001725, batch_cost: 0.8390, reader_cost: 0.00015, ips: 1.1919 samples/sec | ETA 01:15:05 2021-05-06 21:57:05 [INFO] [TRAIN] epoch: 94, iter: 34640/40000, loss: 0.3520, lr: 0.001722, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1914 samples/sec | ETA 01:14:58 2021-05-06 21:57:14 [INFO] [TRAIN] epoch: 94, iter: 34650/40000, loss: 0.3729, lr: 0.001719, batch_cost: 0.8415, reader_cost: 0.00016, ips: 1.1883 samples/sec | ETA 01:15:02 2021-05-06 21:57:22 [INFO] [TRAIN] epoch: 94, iter: 34660/40000, loss: 0.5222, lr: 0.001717, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 01:14:46 2021-05-06 21:57:31 [INFO] [TRAIN] epoch: 94, iter: 34670/40000, loss: 0.5076, lr: 0.001714, batch_cost: 0.8410, reader_cost: 0.00016, ips: 1.1890 samples/sec | ETA 01:14:42 2021-05-06 21:57:39 [INFO] [TRAIN] epoch: 94, iter: 34680/40000, loss: 0.3756, lr: 0.001711, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1917 samples/sec | ETA 01:14:24 2021-05-06 21:57:47 [INFO] [TRAIN] epoch: 94, iter: 34690/40000, loss: 0.4234, lr: 0.001709, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 01:14:23 2021-05-06 21:57:56 [INFO] [TRAIN] epoch: 94, iter: 34700/40000, loss: 0.2088, lr: 0.001706, batch_cost: 0.8380, reader_cost: 0.00014, ips: 1.1933 samples/sec | ETA 01:14:01 2021-05-06 21:58:04 [INFO] [TRAIN] epoch: 94, iter: 34710/40000, loss: 0.3598, lr: 0.001703, batch_cost: 0.8381, reader_cost: 0.00015, ips: 1.1931 samples/sec | ETA 01:13:53 2021-05-06 21:58:12 [INFO] [TRAIN] epoch: 94, iter: 34720/40000, loss: 0.3529, lr: 0.001700, batch_cost: 0.8396, reader_cost: 0.00014, ips: 1.1911 samples/sec | ETA 01:13:52 2021-05-06 21:58:21 [INFO] [TRAIN] epoch: 94, iter: 34730/40000, loss: 0.5310, lr: 0.001698, batch_cost: 0.8392, reader_cost: 0.00015, ips: 1.1917 samples/sec | ETA 01:13:42 2021-05-06 21:58:29 [INFO] [TRAIN] epoch: 94, iter: 34740/40000, loss: 0.4120, lr: 0.001695, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 01:13:38 2021-05-06 21:58:38 [INFO] [TRAIN] epoch: 94, iter: 34750/40000, loss: 0.2968, lr: 0.001692, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1917 samples/sec | ETA 01:13:25 2021-05-06 21:58:46 [INFO] [TRAIN] epoch: 94, iter: 34760/40000, loss: 0.3207, lr: 0.001689, batch_cost: 0.8407, reader_cost: 0.00016, ips: 1.1895 samples/sec | ETA 01:13:25 2021-05-06 21:58:54 [INFO] [TRAIN] epoch: 94, iter: 34770/40000, loss: 0.2489, lr: 0.001687, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1918 samples/sec | ETA 01:13:08 2021-05-06 21:59:03 [INFO] [TRAIN] epoch: 94, iter: 34780/40000, loss: 0.5834, lr: 0.001684, batch_cost: 0.8379, reader_cost: 0.00014, ips: 1.1935 samples/sec | ETA 01:12:53 2021-05-06 21:59:11 [INFO] [TRAIN] epoch: 94, iter: 34790/40000, loss: 0.4022, lr: 0.001681, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 01:13:01 2021-05-06 21:59:20 [INFO] [TRAIN] epoch: 94, iter: 34800/40000, loss: 0.4326, lr: 0.001679, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 01:12:53 2021-05-06 21:59:28 [INFO] [TRAIN] epoch: 94, iter: 34810/40000, loss: 0.3311, lr: 0.001676, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 01:12:40 2021-05-06 21:59:36 [INFO] [TRAIN] epoch: 94, iter: 34820/40000, loss: 0.3119, lr: 0.001673, batch_cost: 0.8411, reader_cost: 0.00017, ips: 1.1889 samples/sec | ETA 01:12:36 2021-05-06 21:59:45 [INFO] [TRAIN] epoch: 94, iter: 34830/40000, loss: 0.1774, lr: 0.001670, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 01:12:27 2021-05-06 21:59:53 [INFO] [TRAIN] epoch: 94, iter: 34840/40000, loss: 0.2101, lr: 0.001668, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 01:12:18 2021-05-06 22:00:02 [INFO] [TRAIN] epoch: 94, iter: 34850/40000, loss: 0.3164, lr: 0.001665, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 01:12:06 2021-05-06 22:00:10 [INFO] [TRAIN] epoch: 94, iter: 34860/40000, loss: 0.3059, lr: 0.001662, batch_cost: 0.8404, reader_cost: 0.00016, ips: 1.1899 samples/sec | ETA 01:11:59 2021-05-06 22:00:19 [INFO] [TRAIN] epoch: 94, iter: 34870/40000, loss: 0.2395, lr: 0.001659, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 01:11:47 2021-05-06 22:00:27 [INFO] [TRAIN] epoch: 94, iter: 34880/40000, loss: 0.4180, lr: 0.001657, batch_cost: 0.8387, reader_cost: 0.00015, ips: 1.1924 samples/sec | ETA 01:11:34 2021-05-06 22:00:35 [INFO] [TRAIN] epoch: 94, iter: 34890/40000, loss: 0.3993, lr: 0.001654, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 01:11:33 2021-05-06 22:00:44 [INFO] [TRAIN] epoch: 94, iter: 34900/40000, loss: 0.3171, lr: 0.001651, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 01:11:27 2021-05-06 22:00:52 [INFO] [TRAIN] epoch: 94, iter: 34910/40000, loss: 0.1929, lr: 0.001648, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1914 samples/sec | ETA 01:11:12 2021-05-06 22:01:00 [INFO] [TRAIN] epoch: 94, iter: 34920/40000, loss: 0.5467, lr: 0.001646, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 01:11:03 2021-05-06 22:01:09 [INFO] [TRAIN] epoch: 94, iter: 34930/40000, loss: 0.2234, lr: 0.001643, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1879 samples/sec | ETA 01:11:07 2021-05-06 22:01:17 [INFO] [TRAIN] epoch: 94, iter: 34940/40000, loss: 0.2922, lr: 0.001640, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 01:10:49 2021-05-06 22:01:26 [INFO] [TRAIN] epoch: 94, iter: 34950/40000, loss: 0.3134, lr: 0.001638, batch_cost: 0.8400, reader_cost: 0.00016, ips: 1.1904 samples/sec | ETA 01:10:42 2021-05-06 22:01:34 [INFO] [TRAIN] epoch: 94, iter: 34960/40000, loss: 0.4123, lr: 0.001635, batch_cost: 0.8381, reader_cost: 0.00013, ips: 1.1932 samples/sec | ETA 01:10:23 2021-05-06 22:01:45 [INFO] [TRAIN] epoch: 95, iter: 34970/40000, loss: 0.2815, lr: 0.001632, batch_cost: 1.1314, reader_cost: 0.21224, ips: 0.8838 samples/sec | ETA 01:34:51 2021-05-06 22:01:54 [INFO] [TRAIN] epoch: 95, iter: 34980/40000, loss: 0.4106, lr: 0.001629, batch_cost: 0.8625, reader_cost: 0.00031, ips: 1.1595 samples/sec | ETA 01:12:09 2021-05-06 22:02:02 [INFO] [TRAIN] epoch: 95, iter: 34990/40000, loss: 0.3760, lr: 0.001627, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 01:10:13 2021-05-06 22:02:11 [INFO] [TRAIN] epoch: 95, iter: 35000/40000, loss: 0.2977, lr: 0.001624, batch_cost: 0.8421, reader_cost: 0.00015, ips: 1.1875 samples/sec | ETA 01:10:10 2021-05-06 22:02:11 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 22:05:48 [INFO] [EVAL] #Images: 500 mIoU: 0.7744 Acc: 0.9593 Kappa: 0.9472 2021-05-06 22:05:48 [INFO] [EVAL] Class IoU: [0.9819 0.8537 0.9224 0.6744 0.6227 0.5626 0.6216 0.7347 0.9203 0.6637 0.947 0.7877 0.5897 0.9437 0.7879 0.9037 0.8104 0.6524 0.7336] 2021-05-06 22:05:48 [INFO] [EVAL] Class Acc: [0.9915 0.9209 0.9552 0.8695 0.8047 0.8006 0.8018 0.8927 0.9498 0.8234 0.9734 0.8574 0.7362 0.9679 0.8789 0.9558 0.9314 0.8021 0.848 ] 2021-05-06 22:06:40 [INFO] [EVAL] The model with the best validation mIoU (0.7744) was saved at iter 35000. 2021-05-06 22:06:48 [INFO] [TRAIN] epoch: 95, iter: 35010/40000, loss: 0.3100, lr: 0.001621, batch_cost: 0.8356, reader_cost: 0.00025, ips: 1.1968 samples/sec | ETA 01:09:29 2021-05-06 22:06:57 [INFO] [TRAIN] epoch: 95, iter: 35020/40000, loss: 0.4166, lr: 0.001618, batch_cost: 0.8377, reader_cost: 0.00015, ips: 1.1937 samples/sec | ETA 01:09:31 2021-05-06 22:07:05 [INFO] [TRAIN] epoch: 95, iter: 35030/40000, loss: 0.4618, lr: 0.001616, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1886 samples/sec | ETA 01:09:41 2021-05-06 22:07:13 [INFO] [TRAIN] epoch: 95, iter: 35040/40000, loss: 0.4959, lr: 0.001613, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 01:09:30 2021-05-06 22:07:22 [INFO] [TRAIN] epoch: 95, iter: 35050/40000, loss: 0.4728, lr: 0.001610, batch_cost: 0.8383, reader_cost: 0.00015, ips: 1.1928 samples/sec | ETA 01:09:09 2021-05-06 22:07:30 [INFO] [TRAIN] epoch: 95, iter: 35060/40000, loss: 0.2972, lr: 0.001607, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1912 samples/sec | ETA 01:09:07 2021-05-06 22:07:39 [INFO] [TRAIN] epoch: 95, iter: 35070/40000, loss: 0.1291, lr: 0.001605, batch_cost: 0.8425, reader_cost: 0.00015, ips: 1.1870 samples/sec | ETA 01:09:13 2021-05-06 22:07:47 [INFO] [TRAIN] epoch: 95, iter: 35080/40000, loss: 0.3204, lr: 0.001602, batch_cost: 0.8420, reader_cost: 0.00016, ips: 1.1877 samples/sec | ETA 01:09:02 2021-05-06 22:07:55 [INFO] [TRAIN] epoch: 95, iter: 35090/40000, loss: 0.3859, lr: 0.001599, batch_cost: 0.8409, reader_cost: 0.00016, ips: 1.1891 samples/sec | ETA 01:08:49 2021-05-06 22:08:04 [INFO] [TRAIN] epoch: 95, iter: 35100/40000, loss: 0.4236, lr: 0.001596, batch_cost: 0.8398, reader_cost: 0.00014, ips: 1.1908 samples/sec | ETA 01:08:34 2021-05-06 22:08:12 [INFO] [TRAIN] epoch: 95, iter: 35110/40000, loss: 0.4308, lr: 0.001594, batch_cost: 0.8396, reader_cost: 0.00016, ips: 1.1910 samples/sec | ETA 01:08:25 2021-05-06 22:08:21 [INFO] [TRAIN] epoch: 95, iter: 35120/40000, loss: 0.2404, lr: 0.001591, batch_cost: 0.8409, reader_cost: 0.00017, ips: 1.1891 samples/sec | ETA 01:08:23 2021-05-06 22:08:29 [INFO] [TRAIN] epoch: 95, iter: 35130/40000, loss: 0.2982, lr: 0.001588, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 01:08:12 2021-05-06 22:08:37 [INFO] [TRAIN] epoch: 95, iter: 35140/40000, loss: 0.2579, lr: 0.001585, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 01:08:03 2021-05-06 22:08:46 [INFO] [TRAIN] epoch: 95, iter: 35150/40000, loss: 0.2895, lr: 0.001583, batch_cost: 0.8414, reader_cost: 0.00016, ips: 1.1886 samples/sec | ETA 01:08:00 2021-05-06 22:08:54 [INFO] [TRAIN] epoch: 95, iter: 35160/40000, loss: 0.4838, lr: 0.001580, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 01:07:50 2021-05-06 22:09:03 [INFO] [TRAIN] epoch: 95, iter: 35170/40000, loss: 0.2791, lr: 0.001577, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 01:07:36 2021-05-06 22:09:11 [INFO] [TRAIN] epoch: 95, iter: 35180/40000, loss: 0.2926, lr: 0.001574, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 01:07:29 2021-05-06 22:09:20 [INFO] [TRAIN] epoch: 95, iter: 35190/40000, loss: 0.2411, lr: 0.001572, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1911 samples/sec | ETA 01:07:18 2021-05-06 22:09:28 [INFO] [TRAIN] epoch: 95, iter: 35200/40000, loss: 0.1999, lr: 0.001569, batch_cost: 0.8402, reader_cost: 0.00014, ips: 1.1902 samples/sec | ETA 01:07:12 2021-05-06 22:09:36 [INFO] [TRAIN] epoch: 95, iter: 35210/40000, loss: 0.2632, lr: 0.001566, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 01:07:08 2021-05-06 22:09:45 [INFO] [TRAIN] epoch: 95, iter: 35220/40000, loss: 0.1939, lr: 0.001563, batch_cost: 0.8404, reader_cost: 0.00016, ips: 1.1898 samples/sec | ETA 01:06:57 2021-05-06 22:09:53 [INFO] [TRAIN] epoch: 95, iter: 35230/40000, loss: 0.3483, lr: 0.001561, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 01:06:45 2021-05-06 22:10:02 [INFO] [TRAIN] epoch: 95, iter: 35240/40000, loss: 0.2490, lr: 0.001558, batch_cost: 0.8400, reader_cost: 0.00014, ips: 1.1904 samples/sec | ETA 01:06:38 2021-05-06 22:10:10 [INFO] [TRAIN] epoch: 95, iter: 35250/40000, loss: 0.2955, lr: 0.001555, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1917 samples/sec | ETA 01:06:25 2021-05-06 22:10:18 [INFO] [TRAIN] epoch: 95, iter: 35260/40000, loss: 0.3895, lr: 0.001552, batch_cost: 0.8398, reader_cost: 0.00016, ips: 1.1907 samples/sec | ETA 01:06:20 2021-05-06 22:10:27 [INFO] [TRAIN] epoch: 95, iter: 35270/40000, loss: 0.3058, lr: 0.001550, batch_cost: 0.8396, reader_cost: 0.00014, ips: 1.1910 samples/sec | ETA 01:06:11 2021-05-06 22:10:35 [INFO] [TRAIN] epoch: 95, iter: 35280/40000, loss: 0.3080, lr: 0.001547, batch_cost: 0.8387, reader_cost: 0.00015, ips: 1.1924 samples/sec | ETA 01:05:58 2021-05-06 22:10:43 [INFO] [TRAIN] epoch: 95, iter: 35290/40000, loss: 0.3940, lr: 0.001544, batch_cost: 0.8384, reader_cost: 0.00014, ips: 1.1927 samples/sec | ETA 01:05:49 2021-05-06 22:10:52 [INFO] [TRAIN] epoch: 95, iter: 35300/40000, loss: 0.3107, lr: 0.001541, batch_cost: 0.8396, reader_cost: 0.00014, ips: 1.1910 samples/sec | ETA 01:05:46 2021-05-06 22:11:00 [INFO] [TRAIN] epoch: 95, iter: 35310/40000, loss: 0.3373, lr: 0.001539, batch_cost: 0.8405, reader_cost: 0.00016, ips: 1.1898 samples/sec | ETA 01:05:41 2021-05-06 22:11:09 [INFO] [TRAIN] epoch: 95, iter: 35320/40000, loss: 0.3015, lr: 0.001536, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 01:05:30 2021-05-06 22:11:17 [INFO] [TRAIN] epoch: 95, iter: 35330/40000, loss: 0.3447, lr: 0.001533, batch_cost: 0.8390, reader_cost: 0.00014, ips: 1.1920 samples/sec | ETA 01:05:17 2021-05-06 22:11:25 [INFO] [TRAIN] epoch: 95, iter: 35340/40000, loss: 0.3944, lr: 0.001530, batch_cost: 0.8381, reader_cost: 0.00009, ips: 1.1932 samples/sec | ETA 01:05:05 2021-05-06 22:11:37 [INFO] [TRAIN] epoch: 96, iter: 35350/40000, loss: 0.4650, lr: 0.001527, batch_cost: 1.1692, reader_cost: 0.19907, ips: 0.8553 samples/sec | ETA 01:30:36 2021-05-06 22:11:46 [INFO] [TRAIN] epoch: 96, iter: 35360/40000, loss: 0.3132, lr: 0.001525, batch_cost: 0.8416, reader_cost: 0.00030, ips: 1.1882 samples/sec | ETA 01:05:05 2021-05-06 22:11:54 [INFO] [TRAIN] epoch: 96, iter: 35370/40000, loss: 0.3361, lr: 0.001522, batch_cost: 0.8383, reader_cost: 0.00015, ips: 1.1929 samples/sec | ETA 01:04:41 2021-05-06 22:12:02 [INFO] [TRAIN] epoch: 96, iter: 35380/40000, loss: 0.2648, lr: 0.001519, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 01:04:39 2021-05-06 22:12:11 [INFO] [TRAIN] epoch: 96, iter: 35390/40000, loss: 0.6323, lr: 0.001516, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1888 samples/sec | ETA 01:04:38 2021-05-06 22:12:19 [INFO] [TRAIN] epoch: 96, iter: 35400/40000, loss: 0.4611, lr: 0.001514, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 01:04:27 2021-05-06 22:12:28 [INFO] [TRAIN] epoch: 96, iter: 35410/40000, loss: 0.4777, lr: 0.001511, batch_cost: 0.8406, reader_cost: 0.00016, ips: 1.1896 samples/sec | ETA 01:04:18 2021-05-06 22:12:36 [INFO] [TRAIN] epoch: 96, iter: 35420/40000, loss: 0.5486, lr: 0.001508, batch_cost: 0.8388, reader_cost: 0.00014, ips: 1.1922 samples/sec | ETA 01:04:01 2021-05-06 22:12:44 [INFO] [TRAIN] epoch: 96, iter: 35430/40000, loss: 0.4079, lr: 0.001505, batch_cost: 0.8425, reader_cost: 0.00014, ips: 1.1869 samples/sec | ETA 01:04:10 2021-05-06 22:12:53 [INFO] [TRAIN] epoch: 96, iter: 35440/40000, loss: 0.0995, lr: 0.001503, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1879 samples/sec | ETA 01:03:58 2021-05-06 22:13:01 [INFO] [TRAIN] epoch: 96, iter: 35450/40000, loss: 0.2392, lr: 0.001500, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 01:03:47 2021-05-06 22:13:10 [INFO] [TRAIN] epoch: 96, iter: 35460/40000, loss: 0.3821, lr: 0.001497, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 01:03:33 2021-05-06 22:13:18 [INFO] [TRAIN] epoch: 96, iter: 35470/40000, loss: 0.4115, lr: 0.001494, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 01:03:23 2021-05-06 22:13:26 [INFO] [TRAIN] epoch: 96, iter: 35480/40000, loss: 0.5354, lr: 0.001492, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 01:03:16 2021-05-06 22:13:35 [INFO] [TRAIN] epoch: 96, iter: 35490/40000, loss: 0.4149, lr: 0.001489, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1882 samples/sec | ETA 01:03:15 2021-05-06 22:13:43 [INFO] [TRAIN] epoch: 96, iter: 35500/40000, loss: 0.3066, lr: 0.001486, batch_cost: 0.8396, reader_cost: 0.00016, ips: 1.1910 samples/sec | ETA 01:02:58 2021-05-06 22:13:52 [INFO] [TRAIN] epoch: 96, iter: 35510/40000, loss: 0.3741, lr: 0.001483, batch_cost: 0.8401, reader_cost: 0.00016, ips: 1.1904 samples/sec | ETA 01:02:51 2021-05-06 22:14:00 [INFO] [TRAIN] epoch: 96, iter: 35520/40000, loss: 0.3327, lr: 0.001480, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 01:02:42 2021-05-06 22:14:08 [INFO] [TRAIN] epoch: 96, iter: 35530/40000, loss: 0.4477, lr: 0.001478, batch_cost: 0.8389, reader_cost: 0.00015, ips: 1.1920 samples/sec | ETA 01:02:29 2021-05-06 22:14:17 [INFO] [TRAIN] epoch: 96, iter: 35540/40000, loss: 0.6394, lr: 0.001475, batch_cost: 0.8388, reader_cost: 0.00017, ips: 1.1922 samples/sec | ETA 01:02:21 2021-05-06 22:14:25 [INFO] [TRAIN] epoch: 96, iter: 35550/40000, loss: 0.3645, lr: 0.001472, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 01:02:16 2021-05-06 22:14:34 [INFO] [TRAIN] epoch: 96, iter: 35560/40000, loss: 0.3112, lr: 0.001469, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 01:02:14 2021-05-06 22:14:42 [INFO] [TRAIN] epoch: 96, iter: 35570/40000, loss: 0.1711, lr: 0.001467, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 01:02:05 2021-05-06 22:14:50 [INFO] [TRAIN] epoch: 96, iter: 35580/40000, loss: 0.2424, lr: 0.001464, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 01:01:54 2021-05-06 22:14:59 [INFO] [TRAIN] epoch: 96, iter: 35590/40000, loss: 0.2036, lr: 0.001461, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 01:01:46 2021-05-06 22:15:07 [INFO] [TRAIN] epoch: 96, iter: 35600/40000, loss: 0.4093, lr: 0.001458, batch_cost: 0.8399, reader_cost: 0.00014, ips: 1.1906 samples/sec | ETA 01:01:35 2021-05-06 22:15:16 [INFO] [TRAIN] epoch: 96, iter: 35610/40000, loss: 0.3046, lr: 0.001455, batch_cost: 0.8414, reader_cost: 0.00014, ips: 1.1885 samples/sec | ETA 01:01:33 2021-05-06 22:15:24 [INFO] [TRAIN] epoch: 96, iter: 35620/40000, loss: 0.3095, lr: 0.001453, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1914 samples/sec | ETA 01:01:16 2021-05-06 22:15:32 [INFO] [TRAIN] epoch: 96, iter: 35630/40000, loss: 0.4731, lr: 0.001450, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 01:01:11 2021-05-06 22:15:41 [INFO] [TRAIN] epoch: 96, iter: 35640/40000, loss: 0.3127, lr: 0.001447, batch_cost: 0.8415, reader_cost: 0.00016, ips: 1.1883 samples/sec | ETA 01:01:09 2021-05-06 22:15:49 [INFO] [TRAIN] epoch: 96, iter: 35650/40000, loss: 0.2717, lr: 0.001444, batch_cost: 0.8401, reader_cost: 0.00014, ips: 1.1904 samples/sec | ETA 01:00:54 2021-05-06 22:15:58 [INFO] [TRAIN] epoch: 96, iter: 35660/40000, loss: 0.3371, lr: 0.001442, batch_cost: 0.8380, reader_cost: 0.00015, ips: 1.1933 samples/sec | ETA 01:00:37 2021-05-06 22:16:06 [INFO] [TRAIN] epoch: 96, iter: 35670/40000, loss: 0.3472, lr: 0.001439, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 01:00:40 2021-05-06 22:16:14 [INFO] [TRAIN] epoch: 96, iter: 35680/40000, loss: 0.2581, lr: 0.001436, batch_cost: 0.8402, reader_cost: 0.00014, ips: 1.1902 samples/sec | ETA 01:00:29 2021-05-06 22:16:23 [INFO] [TRAIN] epoch: 96, iter: 35690/40000, loss: 0.3450, lr: 0.001433, batch_cost: 0.8403, reader_cost: 0.00014, ips: 1.1901 samples/sec | ETA 01:00:21 2021-05-06 22:16:31 [INFO] [TRAIN] epoch: 96, iter: 35700/40000, loss: 0.3158, lr: 0.001430, batch_cost: 0.8399, reader_cost: 0.00014, ips: 1.1906 samples/sec | ETA 01:00:11 2021-05-06 22:16:40 [INFO] [TRAIN] epoch: 96, iter: 35710/40000, loss: 0.2979, lr: 0.001428, batch_cost: 0.8379, reader_cost: 0.00030, ips: 1.1934 samples/sec | ETA 00:59:54 2021-05-06 22:16:51 [INFO] [TRAIN] epoch: 97, iter: 35720/40000, loss: 0.6749, lr: 0.001425, batch_cost: 1.1496, reader_cost: 0.21031, ips: 0.8699 samples/sec | ETA 01:22:00 2021-05-06 22:17:00 [INFO] [TRAIN] epoch: 97, iter: 35730/40000, loss: 0.3082, lr: 0.001422, batch_cost: 0.8405, reader_cost: 0.00033, ips: 1.1897 samples/sec | ETA 00:59:49 2021-05-06 22:17:08 [INFO] [TRAIN] epoch: 97, iter: 35740/40000, loss: 0.3413, lr: 0.001419, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 00:59:41 2021-05-06 22:17:16 [INFO] [TRAIN] epoch: 97, iter: 35750/40000, loss: 0.2969, lr: 0.001417, batch_cost: 0.8425, reader_cost: 0.00015, ips: 1.1870 samples/sec | ETA 00:59:40 2021-05-06 22:17:25 [INFO] [TRAIN] epoch: 97, iter: 35760/40000, loss: 0.4281, lr: 0.001414, batch_cost: 0.8401, reader_cost: 0.00016, ips: 1.1903 samples/sec | ETA 00:59:22 2021-05-06 22:17:33 [INFO] [TRAIN] epoch: 97, iter: 35770/40000, loss: 0.5061, lr: 0.001411, batch_cost: 0.8438, reader_cost: 0.00016, ips: 1.1851 samples/sec | ETA 00:59:29 2021-05-06 22:17:42 [INFO] [TRAIN] epoch: 97, iter: 35780/40000, loss: 0.4401, lr: 0.001408, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 00:59:06 2021-05-06 22:17:50 [INFO] [TRAIN] epoch: 97, iter: 35790/40000, loss: 0.5302, lr: 0.001405, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1886 samples/sec | ETA 00:59:02 2021-05-06 22:17:58 [INFO] [TRAIN] epoch: 97, iter: 35800/40000, loss: 0.3929, lr: 0.001403, batch_cost: 0.8401, reader_cost: 0.00016, ips: 1.1904 samples/sec | ETA 00:58:48 2021-05-06 22:18:07 [INFO] [TRAIN] epoch: 97, iter: 35810/40000, loss: 0.1359, lr: 0.001400, batch_cost: 0.8423, reader_cost: 0.00015, ips: 1.1872 samples/sec | ETA 00:58:49 2021-05-06 22:18:15 [INFO] [TRAIN] epoch: 97, iter: 35820/40000, loss: 0.2747, lr: 0.001397, batch_cost: 0.8396, reader_cost: 0.00016, ips: 1.1911 samples/sec | ETA 00:58:29 2021-05-06 22:18:24 [INFO] [TRAIN] epoch: 97, iter: 35830/40000, loss: 0.2391, lr: 0.001394, batch_cost: 0.8393, reader_cost: 0.00015, ips: 1.1914 samples/sec | ETA 00:58:19 2021-05-06 22:18:32 [INFO] [TRAIN] epoch: 97, iter: 35840/40000, loss: 0.2875, lr: 0.001391, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 00:58:14 2021-05-06 22:18:40 [INFO] [TRAIN] epoch: 97, iter: 35850/40000, loss: 0.4763, lr: 0.001389, batch_cost: 0.8412, reader_cost: 0.00016, ips: 1.1888 samples/sec | ETA 00:58:10 2021-05-06 22:18:49 [INFO] [TRAIN] epoch: 97, iter: 35860/40000, loss: 0.4520, lr: 0.001386, batch_cost: 0.8403, reader_cost: 0.00014, ips: 1.1900 samples/sec | ETA 00:57:58 2021-05-06 22:18:57 [INFO] [TRAIN] epoch: 97, iter: 35870/40000, loss: 0.1610, lr: 0.001383, batch_cost: 0.8387, reader_cost: 0.00015, ips: 1.1923 samples/sec | ETA 00:57:43 2021-05-06 22:19:06 [INFO] [TRAIN] epoch: 97, iter: 35880/40000, loss: 0.3156, lr: 0.001380, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 00:57:41 2021-05-06 22:19:14 [INFO] [TRAIN] epoch: 97, iter: 35890/40000, loss: 0.3653, lr: 0.001377, batch_cost: 0.8398, reader_cost: 0.00014, ips: 1.1907 samples/sec | ETA 00:57:31 2021-05-06 22:19:22 [INFO] [TRAIN] epoch: 97, iter: 35900/40000, loss: 0.4359, lr: 0.001375, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1918 samples/sec | ETA 00:57:20 2021-05-06 22:19:31 [INFO] [TRAIN] epoch: 97, iter: 35910/40000, loss: 0.4625, lr: 0.001372, batch_cost: 0.8400, reader_cost: 0.00016, ips: 1.1904 samples/sec | ETA 00:57:15 2021-05-06 22:19:39 [INFO] [TRAIN] epoch: 97, iter: 35920/40000, loss: 0.3374, lr: 0.001369, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 00:57:10 2021-05-06 22:19:48 [INFO] [TRAIN] epoch: 97, iter: 35930/40000, loss: 0.3530, lr: 0.001366, batch_cost: 0.8406, reader_cost: 0.00014, ips: 1.1897 samples/sec | ETA 00:57:01 2021-05-06 22:19:56 [INFO] [TRAIN] epoch: 97, iter: 35940/40000, loss: 0.2364, lr: 0.001363, batch_cost: 0.8401, reader_cost: 0.00016, ips: 1.1903 samples/sec | ETA 00:56:50 2021-05-06 22:20:04 [INFO] [TRAIN] epoch: 97, iter: 35950/40000, loss: 0.2774, lr: 0.001361, batch_cost: 0.8420, reader_cost: 0.00015, ips: 1.1876 samples/sec | ETA 00:56:50 2021-05-06 22:20:13 [INFO] [TRAIN] epoch: 97, iter: 35960/40000, loss: 0.2079, lr: 0.001358, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1883 samples/sec | ETA 00:56:39 2021-05-06 22:20:21 [INFO] [TRAIN] epoch: 97, iter: 35970/40000, loss: 0.3407, lr: 0.001355, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 00:56:26 2021-05-06 22:20:30 [INFO] [TRAIN] epoch: 97, iter: 35980/40000, loss: 0.2182, lr: 0.001352, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 00:56:16 2021-05-06 22:20:38 [INFO] [TRAIN] epoch: 97, iter: 35990/40000, loss: 0.2685, lr: 0.001349, batch_cost: 0.8414, reader_cost: 0.00014, ips: 1.1885 samples/sec | ETA 00:56:14 2021-05-06 22:20:47 [INFO] [TRAIN] epoch: 97, iter: 36000/40000, loss: 0.3034, lr: 0.001347, batch_cost: 0.8404, reader_cost: 0.00016, ips: 1.1899 samples/sec | ETA 00:56:01 2021-05-06 22:20:47 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 22:24:24 [INFO] [EVAL] #Images: 500 mIoU: 0.7752 Acc: 0.9592 Kappa: 0.9471 2021-05-06 22:24:24 [INFO] [EVAL] Class IoU: [0.9814 0.8527 0.9219 0.6498 0.6102 0.5603 0.6277 0.7397 0.9205 0.6732 0.9457 0.7906 0.5987 0.9453 0.823 0.9061 0.7963 0.647 0.7379] 2021-05-06 22:24:24 [INFO] [EVAL] Class Acc: [0.9921 0.9153 0.9525 0.8413 0.8182 0.8101 0.7839 0.8903 0.952 0.8294 0.9658 0.875 0.7773 0.9682 0.9297 0.9529 0.9116 0.7982 0.8316] 2021-05-06 22:25:16 [INFO] [EVAL] The model with the best validation mIoU (0.7752) was saved at iter 36000. 2021-05-06 22:25:24 [INFO] [TRAIN] epoch: 97, iter: 36010/40000, loss: 0.3675, lr: 0.001344, batch_cost: 0.8386, reader_cost: 0.00042, ips: 1.1924 samples/sec | ETA 00:55:46 2021-05-06 22:25:33 [INFO] [TRAIN] epoch: 97, iter: 36020/40000, loss: 0.2426, lr: 0.001341, batch_cost: 0.8375, reader_cost: 0.00014, ips: 1.1940 samples/sec | ETA 00:55:33 2021-05-06 22:25:41 [INFO] [TRAIN] epoch: 97, iter: 36030/40000, loss: 0.2664, lr: 0.001338, batch_cost: 0.8402, reader_cost: 0.00014, ips: 1.1902 samples/sec | ETA 00:55:35 2021-05-06 22:25:50 [INFO] [TRAIN] epoch: 97, iter: 36040/40000, loss: 0.3769, lr: 0.001335, batch_cost: 0.8405, reader_cost: 0.00014, ips: 1.1898 samples/sec | ETA 00:55:28 2021-05-06 22:25:58 [INFO] [TRAIN] epoch: 97, iter: 36050/40000, loss: 0.2460, lr: 0.001333, batch_cost: 0.8422, reader_cost: 0.00015, ips: 1.1874 samples/sec | ETA 00:55:26 2021-05-06 22:26:07 [INFO] [TRAIN] epoch: 97, iter: 36060/40000, loss: 0.2608, lr: 0.001330, batch_cost: 0.8400, reader_cost: 0.00016, ips: 1.1905 samples/sec | ETA 00:55:09 2021-05-06 22:26:15 [INFO] [TRAIN] epoch: 97, iter: 36070/40000, loss: 0.3305, lr: 0.001327, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 00:55:05 2021-05-06 22:26:23 [INFO] [TRAIN] epoch: 97, iter: 36080/40000, loss: 0.3483, lr: 0.001324, batch_cost: 0.8424, reader_cost: 0.00010, ips: 1.1870 samples/sec | ETA 00:55:02 2021-05-06 22:26:35 [INFO] [TRAIN] epoch: 98, iter: 36090/40000, loss: 0.3870, lr: 0.001321, batch_cost: 1.2027, reader_cost: 0.20046, ips: 0.8315 samples/sec | ETA 01:18:22 2021-05-06 22:26:44 [INFO] [TRAIN] epoch: 98, iter: 36100/40000, loss: 0.3872, lr: 0.001319, batch_cost: 0.8430, reader_cost: 0.00031, ips: 1.1862 samples/sec | ETA 00:54:47 2021-05-06 22:26:52 [INFO] [TRAIN] epoch: 98, iter: 36110/40000, loss: 0.2788, lr: 0.001316, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 00:54:32 2021-05-06 22:27:01 [INFO] [TRAIN] epoch: 98, iter: 36120/40000, loss: 0.2284, lr: 0.001313, batch_cost: 0.8418, reader_cost: 0.00014, ips: 1.1880 samples/sec | ETA 00:54:26 2021-05-06 22:27:09 [INFO] [TRAIN] epoch: 98, iter: 36130/40000, loss: 0.2823, lr: 0.001310, batch_cost: 0.8394, reader_cost: 0.00014, ips: 1.1913 samples/sec | ETA 00:54:08 2021-05-06 22:27:17 [INFO] [TRAIN] epoch: 98, iter: 36140/40000, loss: 0.4791, lr: 0.001307, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1882 samples/sec | ETA 00:54:08 2021-05-06 22:27:26 [INFO] [TRAIN] epoch: 98, iter: 36150/40000, loss: 0.3390, lr: 0.001304, batch_cost: 0.8402, reader_cost: 0.00016, ips: 1.1902 samples/sec | ETA 00:53:54 2021-05-06 22:27:34 [INFO] [TRAIN] epoch: 98, iter: 36160/40000, loss: 0.6386, lr: 0.001302, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 00:53:45 2021-05-06 22:27:43 [INFO] [TRAIN] epoch: 98, iter: 36170/40000, loss: 0.3904, lr: 0.001299, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1884 samples/sec | ETA 00:53:42 2021-05-06 22:27:51 [INFO] [TRAIN] epoch: 98, iter: 36180/40000, loss: 0.2347, lr: 0.001296, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 00:53:29 2021-05-06 22:27:59 [INFO] [TRAIN] epoch: 98, iter: 36190/40000, loss: 0.3094, lr: 0.001293, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 00:53:19 2021-05-06 22:28:08 [INFO] [TRAIN] epoch: 98, iter: 36200/40000, loss: 0.3242, lr: 0.001290, batch_cost: 0.8403, reader_cost: 0.00016, ips: 1.1901 samples/sec | ETA 00:53:13 2021-05-06 22:28:16 [INFO] [TRAIN] epoch: 98, iter: 36210/40000, loss: 0.3197, lr: 0.001288, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 00:53:05 2021-05-06 22:28:25 [INFO] [TRAIN] epoch: 98, iter: 36220/40000, loss: 0.5286, lr: 0.001285, batch_cost: 0.8390, reader_cost: 0.00016, ips: 1.1919 samples/sec | ETA 00:52:51 2021-05-06 22:28:33 [INFO] [TRAIN] epoch: 98, iter: 36230/40000, loss: 0.5893, lr: 0.001282, batch_cost: 0.8406, reader_cost: 0.00016, ips: 1.1896 samples/sec | ETA 00:52:49 2021-05-06 22:28:41 [INFO] [TRAIN] epoch: 98, iter: 36240/40000, loss: 0.3189, lr: 0.001279, batch_cost: 0.8396, reader_cost: 0.00016, ips: 1.1910 samples/sec | ETA 00:52:36 2021-05-06 22:28:50 [INFO] [TRAIN] epoch: 98, iter: 36250/40000, loss: 0.2746, lr: 0.001276, batch_cost: 0.8397, reader_cost: 0.00017, ips: 1.1910 samples/sec | ETA 00:52:28 2021-05-06 22:28:58 [INFO] [TRAIN] epoch: 98, iter: 36260/40000, loss: 0.3213, lr: 0.001273, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 00:52:25 2021-05-06 22:29:07 [INFO] [TRAIN] epoch: 98, iter: 36270/40000, loss: 0.4334, lr: 0.001271, batch_cost: 0.8400, reader_cost: 0.00016, ips: 1.1904 samples/sec | ETA 00:52:13 2021-05-06 22:29:15 [INFO] [TRAIN] epoch: 98, iter: 36280/40000, loss: 0.6649, lr: 0.001268, batch_cost: 0.8392, reader_cost: 0.00014, ips: 1.1916 samples/sec | ETA 00:52:01 2021-05-06 22:29:23 [INFO] [TRAIN] epoch: 98, iter: 36290/40000, loss: 0.3296, lr: 0.001265, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 00:51:56 2021-05-06 22:29:32 [INFO] [TRAIN] epoch: 98, iter: 36300/40000, loss: 0.3300, lr: 0.001262, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 00:51:50 2021-05-06 22:29:40 [INFO] [TRAIN] epoch: 98, iter: 36310/40000, loss: 0.2148, lr: 0.001259, batch_cost: 0.8387, reader_cost: 0.00015, ips: 1.1923 samples/sec | ETA 00:51:34 2021-05-06 22:29:49 [INFO] [TRAIN] epoch: 98, iter: 36320/40000, loss: 0.2790, lr: 0.001257, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1911 samples/sec | ETA 00:51:29 2021-05-06 22:29:57 [INFO] [TRAIN] epoch: 98, iter: 36330/40000, loss: 0.2084, lr: 0.001254, batch_cost: 0.8383, reader_cost: 0.00015, ips: 1.1929 samples/sec | ETA 00:51:16 2021-05-06 22:30:05 [INFO] [TRAIN] epoch: 98, iter: 36340/40000, loss: 0.2790, lr: 0.001251, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 00:51:12 2021-05-06 22:30:14 [INFO] [TRAIN] epoch: 98, iter: 36350/40000, loss: 0.3054, lr: 0.001248, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 00:51:05 2021-05-06 22:30:22 [INFO] [TRAIN] epoch: 98, iter: 36360/40000, loss: 0.3195, lr: 0.001245, batch_cost: 0.8387, reader_cost: 0.00015, ips: 1.1923 samples/sec | ETA 00:50:52 2021-05-06 22:30:31 [INFO] [TRAIN] epoch: 98, iter: 36370/40000, loss: 0.3235, lr: 0.001242, batch_cost: 0.8389, reader_cost: 0.00017, ips: 1.1920 samples/sec | ETA 00:50:45 2021-05-06 22:30:39 [INFO] [TRAIN] epoch: 98, iter: 36380/40000, loss: 0.4237, lr: 0.001240, batch_cost: 0.8404, reader_cost: 0.00016, ips: 1.1899 samples/sec | ETA 00:50:42 2021-05-06 22:30:47 [INFO] [TRAIN] epoch: 98, iter: 36390/40000, loss: 0.2416, lr: 0.001237, batch_cost: 0.8404, reader_cost: 0.00019, ips: 1.1899 samples/sec | ETA 00:50:33 2021-05-06 22:30:56 [INFO] [TRAIN] epoch: 98, iter: 36400/40000, loss: 0.2651, lr: 0.001234, batch_cost: 0.8393, reader_cost: 0.00016, ips: 1.1914 samples/sec | ETA 00:50:21 2021-05-06 22:31:04 [INFO] [TRAIN] epoch: 98, iter: 36410/40000, loss: 0.4355, lr: 0.001231, batch_cost: 0.8393, reader_cost: 0.00016, ips: 1.1915 samples/sec | ETA 00:50:13 2021-05-06 22:31:13 [INFO] [TRAIN] epoch: 98, iter: 36420/40000, loss: 0.3054, lr: 0.001228, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 00:50:08 2021-05-06 22:31:21 [INFO] [TRAIN] epoch: 98, iter: 36430/40000, loss: 0.3317, lr: 0.001225, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 00:49:58 2021-05-06 22:31:29 [INFO] [TRAIN] epoch: 98, iter: 36440/40000, loss: 0.3017, lr: 0.001223, batch_cost: 0.8392, reader_cost: 0.00015, ips: 1.1916 samples/sec | ETA 00:49:47 2021-05-06 22:31:38 [INFO] [TRAIN] epoch: 98, iter: 36450/40000, loss: 0.4148, lr: 0.001220, batch_cost: 0.8374, reader_cost: 0.00013, ips: 1.1942 samples/sec | ETA 00:49:32 2021-05-06 22:31:49 [INFO] [TRAIN] epoch: 99, iter: 36460/40000, loss: 0.3484, lr: 0.001217, batch_cost: 1.1344, reader_cost: 0.20907, ips: 0.8815 samples/sec | ETA 01:06:55 2021-05-06 22:31:58 [INFO] [TRAIN] epoch: 99, iter: 36470/40000, loss: 0.3209, lr: 0.001214, batch_cost: 0.8482, reader_cost: 0.00033, ips: 1.1790 samples/sec | ETA 00:49:54 2021-05-06 22:32:06 [INFO] [TRAIN] epoch: 99, iter: 36480/40000, loss: 0.3787, lr: 0.001211, batch_cost: 0.8394, reader_cost: 0.00014, ips: 1.1913 samples/sec | ETA 00:49:14 2021-05-06 22:32:14 [INFO] [TRAIN] epoch: 99, iter: 36490/40000, loss: 0.2645, lr: 0.001208, batch_cost: 0.8383, reader_cost: 0.00015, ips: 1.1928 samples/sec | ETA 00:49:02 2021-05-06 22:32:23 [INFO] [TRAIN] epoch: 99, iter: 36500/40000, loss: 0.2561, lr: 0.001205, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1918 samples/sec | ETA 00:48:56 2021-05-06 22:32:31 [INFO] [TRAIN] epoch: 99, iter: 36510/40000, loss: 0.5376, lr: 0.001203, batch_cost: 0.8416, reader_cost: 0.00014, ips: 1.1883 samples/sec | ETA 00:48:57 2021-05-06 22:32:40 [INFO] [TRAIN] epoch: 99, iter: 36520/40000, loss: 0.4324, lr: 0.001200, batch_cost: 0.8407, reader_cost: 0.00016, ips: 1.1895 samples/sec | ETA 00:48:45 2021-05-06 22:32:48 [INFO] [TRAIN] epoch: 99, iter: 36530/40000, loss: 0.3953, lr: 0.001197, batch_cost: 0.8417, reader_cost: 0.00014, ips: 1.1881 samples/sec | ETA 00:48:40 2021-05-06 22:32:56 [INFO] [TRAIN] epoch: 99, iter: 36540/40000, loss: 0.4101, lr: 0.001194, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 00:48:27 2021-05-06 22:33:05 [INFO] [TRAIN] epoch: 99, iter: 36550/40000, loss: 0.2841, lr: 0.001191, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 00:48:16 2021-05-06 22:33:13 [INFO] [TRAIN] epoch: 99, iter: 36560/40000, loss: 0.1851, lr: 0.001188, batch_cost: 0.8400, reader_cost: 0.00014, ips: 1.1904 samples/sec | ETA 00:48:09 2021-05-06 22:33:22 [INFO] [TRAIN] epoch: 99, iter: 36570/40000, loss: 0.3086, lr: 0.001186, batch_cost: 0.8402, reader_cost: 0.00014, ips: 1.1901 samples/sec | ETA 00:48:02 2021-05-06 22:33:30 [INFO] [TRAIN] epoch: 99, iter: 36580/40000, loss: 0.3112, lr: 0.001183, batch_cost: 0.8400, reader_cost: 0.00014, ips: 1.1905 samples/sec | ETA 00:47:52 2021-05-06 22:33:38 [INFO] [TRAIN] epoch: 99, iter: 36590/40000, loss: 0.4256, lr: 0.001180, batch_cost: 0.8414, reader_cost: 0.00017, ips: 1.1885 samples/sec | ETA 00:47:49 2021-05-06 22:33:47 [INFO] [TRAIN] epoch: 99, iter: 36600/40000, loss: 0.4388, lr: 0.001177, batch_cost: 0.8393, reader_cost: 0.00014, ips: 1.1915 samples/sec | ETA 00:47:33 2021-05-06 22:33:55 [INFO] [TRAIN] epoch: 99, iter: 36610/40000, loss: 0.3131, lr: 0.001174, batch_cost: 0.8409, reader_cost: 0.00014, ips: 1.1891 samples/sec | ETA 00:47:30 2021-05-06 22:34:04 [INFO] [TRAIN] epoch: 99, iter: 36620/40000, loss: 0.3215, lr: 0.001171, batch_cost: 0.8396, reader_cost: 0.00016, ips: 1.1910 samples/sec | ETA 00:47:17 2021-05-06 22:34:12 [INFO] [TRAIN] epoch: 99, iter: 36630/40000, loss: 0.2575, lr: 0.001168, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 00:47:12 2021-05-06 22:34:20 [INFO] [TRAIN] epoch: 99, iter: 36640/40000, loss: 0.3734, lr: 0.001166, batch_cost: 0.8406, reader_cost: 0.00016, ips: 1.1896 samples/sec | ETA 00:47:04 2021-05-06 22:34:29 [INFO] [TRAIN] epoch: 99, iter: 36650/40000, loss: 0.4600, lr: 0.001163, batch_cost: 0.8402, reader_cost: 0.00014, ips: 1.1902 samples/sec | ETA 00:46:54 2021-05-06 22:34:37 [INFO] [TRAIN] epoch: 99, iter: 36660/40000, loss: 0.3236, lr: 0.001160, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1887 samples/sec | ETA 00:46:49 2021-05-06 22:34:46 [INFO] [TRAIN] epoch: 99, iter: 36670/40000, loss: 0.3142, lr: 0.001157, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 00:46:38 2021-05-06 22:34:54 [INFO] [TRAIN] epoch: 99, iter: 36680/40000, loss: 0.2383, lr: 0.001154, batch_cost: 0.8387, reader_cost: 0.00014, ips: 1.1923 samples/sec | ETA 00:46:24 2021-05-06 22:35:02 [INFO] [TRAIN] epoch: 99, iter: 36690/40000, loss: 0.2234, lr: 0.001151, batch_cost: 0.8393, reader_cost: 0.00014, ips: 1.1915 samples/sec | ETA 00:46:17 2021-05-06 22:35:11 [INFO] [TRAIN] epoch: 99, iter: 36700/40000, loss: 0.2214, lr: 0.001148, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1913 samples/sec | ETA 00:46:10 2021-05-06 22:35:19 [INFO] [TRAIN] epoch: 99, iter: 36710/40000, loss: 0.2928, lr: 0.001146, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 00:46:06 2021-05-06 22:35:28 [INFO] [TRAIN] epoch: 99, iter: 36720/40000, loss: 0.2777, lr: 0.001143, batch_cost: 0.8408, reader_cost: 0.00016, ips: 1.1893 samples/sec | ETA 00:45:57 2021-05-06 22:35:36 [INFO] [TRAIN] epoch: 99, iter: 36730/40000, loss: 0.3299, lr: 0.001140, batch_cost: 0.8399, reader_cost: 0.00016, ips: 1.1906 samples/sec | ETA 00:45:46 2021-05-06 22:35:44 [INFO] [TRAIN] epoch: 99, iter: 36740/40000, loss: 0.3166, lr: 0.001137, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 00:45:40 2021-05-06 22:35:53 [INFO] [TRAIN] epoch: 99, iter: 36750/40000, loss: 0.4256, lr: 0.001134, batch_cost: 0.8390, reader_cost: 0.00015, ips: 1.1919 samples/sec | ETA 00:45:26 2021-05-06 22:36:01 [INFO] [TRAIN] epoch: 99, iter: 36760/40000, loss: 0.2906, lr: 0.001131, batch_cost: 0.8389, reader_cost: 0.00015, ips: 1.1920 samples/sec | ETA 00:45:18 2021-05-06 22:36:10 [INFO] [TRAIN] epoch: 99, iter: 36770/40000, loss: 0.2773, lr: 0.001128, batch_cost: 0.8406, reader_cost: 0.00016, ips: 1.1897 samples/sec | ETA 00:45:15 2021-05-06 22:36:18 [INFO] [TRAIN] epoch: 99, iter: 36780/40000, loss: 0.3720, lr: 0.001126, batch_cost: 0.8397, reader_cost: 0.00014, ips: 1.1909 samples/sec | ETA 00:45:03 2021-05-06 22:36:26 [INFO] [TRAIN] epoch: 99, iter: 36790/40000, loss: 0.3007, lr: 0.001123, batch_cost: 0.8416, reader_cost: 0.00015, ips: 1.1882 samples/sec | ETA 00:45:01 2021-05-06 22:36:35 [INFO] [TRAIN] epoch: 99, iter: 36800/40000, loss: 0.3107, lr: 0.001120, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1888 samples/sec | ETA 00:44:51 2021-05-06 22:36:43 [INFO] [TRAIN] epoch: 99, iter: 36810/40000, loss: 0.3352, lr: 0.001117, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 00:44:40 2021-05-06 22:36:52 [INFO] [TRAIN] epoch: 99, iter: 36820/40000, loss: 0.4233, lr: 0.001114, batch_cost: 0.8385, reader_cost: 0.00014, ips: 1.1926 samples/sec | ETA 00:44:26 2021-05-06 22:37:03 [INFO] [TRAIN] epoch: 100, iter: 36830/40000, loss: 0.3403, lr: 0.001111, batch_cost: 1.1248, reader_cost: 0.21310, ips: 0.8891 samples/sec | ETA 00:59:25 2021-05-06 22:37:12 [INFO] [TRAIN] epoch: 100, iter: 36840/40000, loss: 0.4102, lr: 0.001108, batch_cost: 0.8645, reader_cost: 0.00033, ips: 1.1567 samples/sec | ETA 00:45:31 2021-05-06 22:37:20 [INFO] [TRAIN] epoch: 100, iter: 36850/40000, loss: 0.4321, lr: 0.001106, batch_cost: 0.8413, reader_cost: 0.00016, ips: 1.1886 samples/sec | ETA 00:44:10 2021-05-06 22:37:28 [INFO] [TRAIN] epoch: 100, iter: 36860/40000, loss: 0.2870, lr: 0.001103, batch_cost: 0.8422, reader_cost: 0.00016, ips: 1.1873 samples/sec | ETA 00:44:04 2021-05-06 22:37:37 [INFO] [TRAIN] epoch: 100, iter: 36870/40000, loss: 0.1662, lr: 0.001100, batch_cost: 0.8400, reader_cost: 0.00014, ips: 1.1904 samples/sec | ETA 00:43:49 2021-05-06 22:37:45 [INFO] [TRAIN] epoch: 100, iter: 36880/40000, loss: 0.5307, lr: 0.001097, batch_cost: 0.8412, reader_cost: 0.00016, ips: 1.1888 samples/sec | ETA 00:43:44 2021-05-06 22:37:54 [INFO] [TRAIN] epoch: 100, iter: 36890/40000, loss: 0.4103, lr: 0.001094, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 00:43:35 2021-05-06 22:38:02 [INFO] [TRAIN] epoch: 100, iter: 36900/40000, loss: 0.4550, lr: 0.001091, batch_cost: 0.8396, reader_cost: 0.00014, ips: 1.1911 samples/sec | ETA 00:43:22 2021-05-06 22:38:10 [INFO] [TRAIN] epoch: 100, iter: 36910/40000, loss: 0.3889, lr: 0.001088, batch_cost: 0.8397, reader_cost: 0.00016, ips: 1.1909 samples/sec | ETA 00:43:14 2021-05-06 22:38:19 [INFO] [TRAIN] epoch: 100, iter: 36920/40000, loss: 0.2713, lr: 0.001085, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 00:43:08 2021-05-06 22:38:27 [INFO] [TRAIN] epoch: 100, iter: 36930/40000, loss: 0.1745, lr: 0.001083, batch_cost: 0.8407, reader_cost: 0.00014, ips: 1.1894 samples/sec | ETA 00:43:01 2021-05-06 22:38:36 [INFO] [TRAIN] epoch: 100, iter: 36940/40000, loss: 0.2958, lr: 0.001080, batch_cost: 0.8423, reader_cost: 0.00015, ips: 1.1872 samples/sec | ETA 00:42:57 2021-05-06 22:38:44 [INFO] [TRAIN] epoch: 100, iter: 36950/40000, loss: 0.4422, lr: 0.001077, batch_cost: 0.8410, reader_cost: 0.00014, ips: 1.1890 samples/sec | ETA 00:42:45 2021-05-06 22:38:52 [INFO] [TRAIN] epoch: 100, iter: 36960/40000, loss: 0.4362, lr: 0.001074, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 00:42:34 2021-05-06 22:39:01 [INFO] [TRAIN] epoch: 100, iter: 36970/40000, loss: 0.5909, lr: 0.001071, batch_cost: 0.8392, reader_cost: 0.00015, ips: 1.1915 samples/sec | ETA 00:42:22 2021-05-06 22:39:09 [INFO] [TRAIN] epoch: 100, iter: 36980/40000, loss: 0.4258, lr: 0.001068, batch_cost: 0.8398, reader_cost: 0.00014, ips: 1.1908 samples/sec | ETA 00:42:16 2021-05-06 22:39:18 [INFO] [TRAIN] epoch: 100, iter: 36990/40000, loss: 0.3282, lr: 0.001065, batch_cost: 0.8401, reader_cost: 0.00016, ips: 1.1903 samples/sec | ETA 00:42:08 2021-05-06 22:39:26 [INFO] [TRAIN] epoch: 100, iter: 37000/40000, loss: 0.2356, lr: 0.001062, batch_cost: 0.8398, reader_cost: 0.00014, ips: 1.1907 samples/sec | ETA 00:41:59 2021-05-06 22:39:26 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 22:43:03 [INFO] [EVAL] #Images: 500 mIoU: 0.7717 Acc: 0.9589 Kappa: 0.9467 2021-05-06 22:43:03 [INFO] [EVAL] Class IoU: [0.9817 0.8517 0.9219 0.6506 0.6127 0.5694 0.6201 0.7393 0.9189 0.6366 0.9446 0.7867 0.5717 0.9455 0.823 0.9017 0.803 0.6473 0.7356] 2021-05-06 22:43:03 [INFO] [EVAL] Class Acc: [0.9919 0.9157 0.9535 0.8625 0.8014 0.7851 0.8025 0.8726 0.9497 0.8831 0.9653 0.8777 0.7475 0.9676 0.935 0.9449 0.926 0.8117 0.8321] 2021-05-06 22:43:32 [INFO] [EVAL] The model with the best validation mIoU (0.7752) was saved at iter 36000. 2021-05-06 22:43:40 [INFO] [TRAIN] epoch: 100, iter: 37010/40000, loss: 0.3750, lr: 0.001059, batch_cost: 0.8386, reader_cost: 0.00042, ips: 1.1924 samples/sec | ETA 00:41:47 2021-05-06 22:43:49 [INFO] [TRAIN] epoch: 100, iter: 37020/40000, loss: 0.5183, lr: 0.001057, batch_cost: 0.8375, reader_cost: 0.00015, ips: 1.1940 samples/sec | ETA 00:41:35 2021-05-06 22:43:57 [INFO] [TRAIN] epoch: 100, iter: 37030/40000, loss: 0.2633, lr: 0.001054, batch_cost: 0.8398, reader_cost: 0.00016, ips: 1.1908 samples/sec | ETA 00:41:34 2021-05-06 22:44:06 [INFO] [TRAIN] epoch: 100, iter: 37040/40000, loss: 0.3001, lr: 0.001051, batch_cost: 0.8412, reader_cost: 0.00016, ips: 1.1887 samples/sec | ETA 00:41:30 2021-05-06 22:44:14 [INFO] [TRAIN] epoch: 100, iter: 37050/40000, loss: 0.3216, lr: 0.001048, batch_cost: 0.8411, reader_cost: 0.00016, ips: 1.1889 samples/sec | ETA 00:41:21 2021-05-06 22:44:22 [INFO] [TRAIN] epoch: 100, iter: 37060/40000, loss: 0.2366, lr: 0.001045, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1879 samples/sec | ETA 00:41:14 2021-05-06 22:44:31 [INFO] [TRAIN] epoch: 100, iter: 37070/40000, loss: 0.2237, lr: 0.001042, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 00:41:02 2021-05-06 22:44:39 [INFO] [TRAIN] epoch: 100, iter: 37080/40000, loss: 0.2516, lr: 0.001039, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 00:40:52 2021-05-06 22:44:48 [INFO] [TRAIN] epoch: 100, iter: 37090/40000, loss: 0.2790, lr: 0.001036, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 00:40:47 2021-05-06 22:44:56 [INFO] [TRAIN] epoch: 100, iter: 37100/40000, loss: 0.2679, lr: 0.001033, batch_cost: 0.8389, reader_cost: 0.00015, ips: 1.1921 samples/sec | ETA 00:40:32 2021-05-06 22:45:04 [INFO] [TRAIN] epoch: 100, iter: 37110/40000, loss: 0.2042, lr: 0.001031, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1917 samples/sec | ETA 00:40:25 2021-05-06 22:45:13 [INFO] [TRAIN] epoch: 100, iter: 37120/40000, loss: 0.3040, lr: 0.001028, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 00:40:20 2021-05-06 22:45:21 [INFO] [TRAIN] epoch: 100, iter: 37130/40000, loss: 0.2967, lr: 0.001025, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 00:40:10 2021-05-06 22:45:30 [INFO] [TRAIN] epoch: 100, iter: 37140/40000, loss: 0.2177, lr: 0.001022, batch_cost: 0.8394, reader_cost: 0.00014, ips: 1.1914 samples/sec | ETA 00:40:00 2021-05-06 22:45:38 [INFO] [TRAIN] epoch: 100, iter: 37150/40000, loss: 0.4396, lr: 0.001019, batch_cost: 0.8388, reader_cost: 0.00016, ips: 1.1922 samples/sec | ETA 00:39:50 2021-05-06 22:45:46 [INFO] [TRAIN] epoch: 100, iter: 37160/40000, loss: 0.2964, lr: 0.001016, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 00:39:48 2021-05-06 22:45:55 [INFO] [TRAIN] epoch: 100, iter: 37170/40000, loss: 0.3139, lr: 0.001013, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 00:39:38 2021-05-06 22:46:03 [INFO] [TRAIN] epoch: 100, iter: 37180/40000, loss: 0.3628, lr: 0.001010, batch_cost: 0.8388, reader_cost: 0.00014, ips: 1.1921 samples/sec | ETA 00:39:25 2021-05-06 22:46:12 [INFO] [TRAIN] epoch: 100, iter: 37190/40000, loss: 0.3071, lr: 0.001007, batch_cost: 0.8398, reader_cost: 0.00014, ips: 1.1907 samples/sec | ETA 00:39:19 2021-05-06 22:46:20 [INFO] [TRAIN] epoch: 100, iter: 37200/40000, loss: 0.3844, lr: 0.001004, batch_cost: 0.8379, reader_cost: 0.00010, ips: 1.1934 samples/sec | ETA 00:39:06 2021-05-06 22:46:32 [INFO] [TRAIN] epoch: 101, iter: 37210/40000, loss: 0.4054, lr: 0.001001, batch_cost: 1.1892, reader_cost: 0.21505, ips: 0.8409 samples/sec | ETA 00:55:17 2021-05-06 22:46:40 [INFO] [TRAIN] epoch: 101, iter: 37220/40000, loss: 0.3845, lr: 0.000999, batch_cost: 0.8392, reader_cost: 0.00031, ips: 1.1916 samples/sec | ETA 00:38:52 2021-05-06 22:46:49 [INFO] [TRAIN] epoch: 101, iter: 37230/40000, loss: 0.3750, lr: 0.000996, batch_cost: 0.8388, reader_cost: 0.00016, ips: 1.1922 samples/sec | ETA 00:38:43 2021-05-06 22:46:57 [INFO] [TRAIN] epoch: 101, iter: 37240/40000, loss: 0.3467, lr: 0.000993, batch_cost: 0.8392, reader_cost: 0.00015, ips: 1.1916 samples/sec | ETA 00:38:36 2021-05-06 22:47:05 [INFO] [TRAIN] epoch: 101, iter: 37250/40000, loss: 0.4636, lr: 0.000990, batch_cost: 0.8389, reader_cost: 0.00015, ips: 1.1921 samples/sec | ETA 00:38:26 2021-05-06 22:47:14 [INFO] [TRAIN] epoch: 101, iter: 37260/40000, loss: 0.4831, lr: 0.000987, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 00:38:24 2021-05-06 22:47:22 [INFO] [TRAIN] epoch: 101, iter: 37270/40000, loss: 0.4642, lr: 0.000984, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 00:38:14 2021-05-06 22:47:31 [INFO] [TRAIN] epoch: 101, iter: 37280/40000, loss: 0.4799, lr: 0.000981, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 00:38:05 2021-05-06 22:47:39 [INFO] [TRAIN] epoch: 101, iter: 37290/40000, loss: 0.3197, lr: 0.000978, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 00:37:56 2021-05-06 22:47:47 [INFO] [TRAIN] epoch: 101, iter: 37300/40000, loss: 0.1530, lr: 0.000975, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 00:37:49 2021-05-06 22:47:56 [INFO] [TRAIN] epoch: 101, iter: 37310/40000, loss: 0.2719, lr: 0.000972, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 00:37:42 2021-05-06 22:48:04 [INFO] [TRAIN] epoch: 101, iter: 37320/40000, loss: 0.2659, lr: 0.000969, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1888 samples/sec | ETA 00:37:34 2021-05-06 22:48:13 [INFO] [TRAIN] epoch: 101, iter: 37330/40000, loss: 0.4280, lr: 0.000967, batch_cost: 0.8392, reader_cost: 0.00015, ips: 1.1916 samples/sec | ETA 00:37:20 2021-05-06 22:48:21 [INFO] [TRAIN] epoch: 101, iter: 37340/40000, loss: 0.3948, lr: 0.000964, batch_cost: 0.8386, reader_cost: 0.00015, ips: 1.1924 samples/sec | ETA 00:37:10 2021-05-06 22:48:29 [INFO] [TRAIN] epoch: 101, iter: 37350/40000, loss: 0.3696, lr: 0.000961, batch_cost: 0.8405, reader_cost: 0.00014, ips: 1.1897 samples/sec | ETA 00:37:07 2021-05-06 22:48:38 [INFO] [TRAIN] epoch: 101, iter: 37360/40000, loss: 0.2729, lr: 0.000958, batch_cost: 0.8410, reader_cost: 0.00014, ips: 1.1891 samples/sec | ETA 00:37:00 2021-05-06 22:48:46 [INFO] [TRAIN] epoch: 101, iter: 37370/40000, loss: 0.3028, lr: 0.000955, batch_cost: 0.8428, reader_cost: 0.00016, ips: 1.1865 samples/sec | ETA 00:36:56 2021-05-06 22:48:55 [INFO] [TRAIN] epoch: 101, iter: 37380/40000, loss: 0.2965, lr: 0.000952, batch_cost: 0.8409, reader_cost: 0.00016, ips: 1.1892 samples/sec | ETA 00:36:43 2021-05-06 22:49:03 [INFO] [TRAIN] epoch: 101, iter: 37390/40000, loss: 0.5696, lr: 0.000949, batch_cost: 0.8382, reader_cost: 0.00015, ips: 1.1930 samples/sec | ETA 00:36:27 2021-05-06 22:49:12 [INFO] [TRAIN] epoch: 101, iter: 37400/40000, loss: 0.5624, lr: 0.000946, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 00:36:25 2021-05-06 22:49:20 [INFO] [TRAIN] epoch: 101, iter: 37410/40000, loss: 0.3480, lr: 0.000943, batch_cost: 0.8392, reader_cost: 0.00015, ips: 1.1916 samples/sec | ETA 00:36:13 2021-05-06 22:49:28 [INFO] [TRAIN] epoch: 101, iter: 37420/40000, loss: 0.2182, lr: 0.000940, batch_cost: 0.8389, reader_cost: 0.00015, ips: 1.1920 samples/sec | ETA 00:36:04 2021-05-06 22:49:37 [INFO] [TRAIN] epoch: 101, iter: 37430/40000, loss: 0.1466, lr: 0.000937, batch_cost: 0.8393, reader_cost: 0.00014, ips: 1.1914 samples/sec | ETA 00:35:57 2021-05-06 22:49:45 [INFO] [TRAIN] epoch: 101, iter: 37440/40000, loss: 0.2779, lr: 0.000934, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 00:35:51 2021-05-06 22:49:53 [INFO] [TRAIN] epoch: 101, iter: 37450/40000, loss: 0.2047, lr: 0.000931, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 00:35:42 2021-05-06 22:50:02 [INFO] [TRAIN] epoch: 101, iter: 37460/40000, loss: 0.2423, lr: 0.000928, batch_cost: 0.8392, reader_cost: 0.00015, ips: 1.1916 samples/sec | ETA 00:35:31 2021-05-06 22:50:10 [INFO] [TRAIN] epoch: 101, iter: 37470/40000, loss: 0.3427, lr: 0.000926, batch_cost: 0.8410, reader_cost: 0.00016, ips: 1.1891 samples/sec | ETA 00:35:27 2021-05-06 22:50:19 [INFO] [TRAIN] epoch: 101, iter: 37480/40000, loss: 0.2865, lr: 0.000923, batch_cost: 0.8388, reader_cost: 0.00014, ips: 1.1921 samples/sec | ETA 00:35:13 2021-05-06 22:50:27 [INFO] [TRAIN] epoch: 101, iter: 37490/40000, loss: 0.3358, lr: 0.000920, batch_cost: 0.8394, reader_cost: 0.00014, ips: 1.1913 samples/sec | ETA 00:35:06 2021-05-06 22:50:35 [INFO] [TRAIN] epoch: 101, iter: 37500/40000, loss: 0.2978, lr: 0.000917, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1918 samples/sec | ETA 00:34:57 2021-05-06 22:50:44 [INFO] [TRAIN] epoch: 101, iter: 37510/40000, loss: 0.3078, lr: 0.000914, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 00:34:50 2021-05-06 22:50:52 [INFO] [TRAIN] epoch: 101, iter: 37520/40000, loss: 0.4284, lr: 0.000911, batch_cost: 0.8405, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 00:34:44 2021-05-06 22:51:01 [INFO] [TRAIN] epoch: 101, iter: 37530/40000, loss: 0.3005, lr: 0.000908, batch_cost: 0.8389, reader_cost: 0.00015, ips: 1.1920 samples/sec | ETA 00:34:32 2021-05-06 22:51:09 [INFO] [TRAIN] epoch: 101, iter: 37540/40000, loss: 0.2972, lr: 0.000905, batch_cost: 0.8395, reader_cost: 0.00016, ips: 1.1912 samples/sec | ETA 00:34:25 2021-05-06 22:51:17 [INFO] [TRAIN] epoch: 101, iter: 37550/40000, loss: 0.4606, lr: 0.000902, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 00:34:17 2021-05-06 22:51:26 [INFO] [TRAIN] epoch: 101, iter: 37560/40000, loss: 0.3122, lr: 0.000899, batch_cost: 0.8398, reader_cost: 0.00016, ips: 1.1907 samples/sec | ETA 00:34:09 2021-05-06 22:51:34 [INFO] [TRAIN] epoch: 101, iter: 37570/40000, loss: 0.3411, lr: 0.000896, batch_cost: 0.8394, reader_cost: 0.00028, ips: 1.1914 samples/sec | ETA 00:33:59 2021-05-06 22:51:46 [INFO] [TRAIN] epoch: 102, iter: 37580/40000, loss: 0.4204, lr: 0.000893, batch_cost: 1.1496, reader_cost: 0.18807, ips: 0.8699 samples/sec | ETA 00:46:22 2021-05-06 22:51:54 [INFO] [TRAIN] epoch: 102, iter: 37590/40000, loss: 0.3056, lr: 0.000890, batch_cost: 0.8392, reader_cost: 0.00033, ips: 1.1917 samples/sec | ETA 00:33:42 2021-05-06 22:52:03 [INFO] [TRAIN] epoch: 102, iter: 37600/40000, loss: 0.3360, lr: 0.000887, batch_cost: 0.8388, reader_cost: 0.00015, ips: 1.1922 samples/sec | ETA 00:33:33 2021-05-06 22:52:11 [INFO] [TRAIN] epoch: 102, iter: 37610/40000, loss: 0.2314, lr: 0.000884, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 00:33:27 2021-05-06 22:52:19 [INFO] [TRAIN] epoch: 102, iter: 37620/40000, loss: 0.3733, lr: 0.000881, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 00:33:19 2021-05-06 22:52:28 [INFO] [TRAIN] epoch: 102, iter: 37630/40000, loss: 0.4986, lr: 0.000878, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 00:33:12 2021-05-06 22:52:36 [INFO] [TRAIN] epoch: 102, iter: 37640/40000, loss: 0.3536, lr: 0.000875, batch_cost: 0.8385, reader_cost: 0.00015, ips: 1.1926 samples/sec | ETA 00:32:58 2021-05-06 22:52:45 [INFO] [TRAIN] epoch: 102, iter: 37650/40000, loss: 0.5410, lr: 0.000873, batch_cost: 0.8388, reader_cost: 0.00015, ips: 1.1922 samples/sec | ETA 00:32:51 2021-05-06 22:52:53 [INFO] [TRAIN] epoch: 102, iter: 37660/40000, loss: 0.3610, lr: 0.000870, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 00:32:45 2021-05-06 22:53:01 [INFO] [TRAIN] epoch: 102, iter: 37670/40000, loss: 0.1652, lr: 0.000867, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1887 samples/sec | ETA 00:32:40 2021-05-06 22:53:10 [INFO] [TRAIN] epoch: 102, iter: 37680/40000, loss: 0.2591, lr: 0.000864, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 00:32:28 2021-05-06 22:53:18 [INFO] [TRAIN] epoch: 102, iter: 37690/40000, loss: 0.3834, lr: 0.000861, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 00:32:22 2021-05-06 22:53:27 [INFO] [TRAIN] epoch: 102, iter: 37700/40000, loss: 0.3592, lr: 0.000858, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1890 samples/sec | ETA 00:32:14 2021-05-06 22:53:35 [INFO] [TRAIN] epoch: 102, iter: 37710/40000, loss: 0.4513, lr: 0.000855, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 00:32:02 2021-05-06 22:53:43 [INFO] [TRAIN] epoch: 102, iter: 37720/40000, loss: 0.3869, lr: 0.000852, batch_cost: 0.8414, reader_cost: 0.00016, ips: 1.1884 samples/sec | ETA 00:31:58 2021-05-06 22:53:52 [INFO] [TRAIN] epoch: 102, iter: 37730/40000, loss: 0.4191, lr: 0.000849, batch_cost: 0.8400, reader_cost: 0.00016, ips: 1.1904 samples/sec | ETA 00:31:46 2021-05-06 22:54:00 [INFO] [TRAIN] epoch: 102, iter: 37740/40000, loss: 0.3483, lr: 0.000846, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 00:31:38 2021-05-06 22:54:09 [INFO] [TRAIN] epoch: 102, iter: 37750/40000, loss: 0.2657, lr: 0.000843, batch_cost: 0.8407, reader_cost: 0.00017, ips: 1.1895 samples/sec | ETA 00:31:31 2021-05-06 22:54:17 [INFO] [TRAIN] epoch: 102, iter: 37760/40000, loss: 0.3204, lr: 0.000840, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 00:31:21 2021-05-06 22:54:25 [INFO] [TRAIN] epoch: 102, iter: 37770/40000, loss: 0.4975, lr: 0.000837, batch_cost: 0.8395, reader_cost: 0.00014, ips: 1.1912 samples/sec | ETA 00:31:12 2021-05-06 22:54:34 [INFO] [TRAIN] epoch: 102, iter: 37780/40000, loss: 0.3079, lr: 0.000834, batch_cost: 0.8410, reader_cost: 0.00014, ips: 1.1891 samples/sec | ETA 00:31:07 2021-05-06 22:54:42 [INFO] [TRAIN] epoch: 102, iter: 37790/40000, loss: 0.3760, lr: 0.000831, batch_cost: 0.8401, reader_cost: 0.00016, ips: 1.1903 samples/sec | ETA 00:30:56 2021-05-06 22:54:51 [INFO] [TRAIN] epoch: 102, iter: 37800/40000, loss: 0.1823, lr: 0.000828, batch_cost: 0.8396, reader_cost: 0.00016, ips: 1.1911 samples/sec | ETA 00:30:47 2021-05-06 22:54:59 [INFO] [TRAIN] epoch: 102, iter: 37810/40000, loss: 0.2552, lr: 0.000825, batch_cost: 0.8387, reader_cost: 0.00014, ips: 1.1923 samples/sec | ETA 00:30:36 2021-05-06 22:55:07 [INFO] [TRAIN] epoch: 102, iter: 37820/40000, loss: 0.1928, lr: 0.000822, batch_cost: 0.8389, reader_cost: 0.00014, ips: 1.1921 samples/sec | ETA 00:30:28 2021-05-06 22:55:16 [INFO] [TRAIN] epoch: 102, iter: 37830/40000, loss: 0.2888, lr: 0.000819, batch_cost: 0.8394, reader_cost: 0.00014, ips: 1.1913 samples/sec | ETA 00:30:21 2021-05-06 22:55:24 [INFO] [TRAIN] epoch: 102, iter: 37840/40000, loss: 0.3152, lr: 0.000816, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 00:30:16 2021-05-06 22:55:33 [INFO] [TRAIN] epoch: 102, iter: 37850/40000, loss: 0.2894, lr: 0.000813, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 00:30:05 2021-05-06 22:55:41 [INFO] [TRAIN] epoch: 102, iter: 37860/40000, loss: 0.3056, lr: 0.000810, batch_cost: 0.8410, reader_cost: 0.00014, ips: 1.1890 samples/sec | ETA 00:29:59 2021-05-06 22:55:49 [INFO] [TRAIN] epoch: 102, iter: 37870/40000, loss: 0.3941, lr: 0.000807, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1913 samples/sec | ETA 00:29:48 2021-05-06 22:55:58 [INFO] [TRAIN] epoch: 102, iter: 37880/40000, loss: 0.2152, lr: 0.000804, batch_cost: 0.8384, reader_cost: 0.00015, ips: 1.1928 samples/sec | ETA 00:29:37 2021-05-06 22:56:06 [INFO] [TRAIN] epoch: 102, iter: 37890/40000, loss: 0.3298, lr: 0.000801, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 00:29:31 2021-05-06 22:56:15 [INFO] [TRAIN] epoch: 102, iter: 37900/40000, loss: 0.3685, lr: 0.000798, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 00:29:23 2021-05-06 22:56:23 [INFO] [TRAIN] epoch: 102, iter: 37910/40000, loss: 0.2701, lr: 0.000795, batch_cost: 0.8411, reader_cost: 0.00016, ips: 1.1889 samples/sec | ETA 00:29:17 2021-05-06 22:56:31 [INFO] [TRAIN] epoch: 102, iter: 37920/40000, loss: 0.3365, lr: 0.000792, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1912 samples/sec | ETA 00:29:06 2021-05-06 22:56:40 [INFO] [TRAIN] epoch: 102, iter: 37930/40000, loss: 0.3152, lr: 0.000789, batch_cost: 0.8401, reader_cost: 0.00017, ips: 1.1903 samples/sec | ETA 00:28:59 2021-05-06 22:56:48 [INFO] [TRAIN] epoch: 102, iter: 37940/40000, loss: 0.3241, lr: 0.000786, batch_cost: 0.8376, reader_cost: 0.00012, ips: 1.1939 samples/sec | ETA 00:28:45 2021-05-06 22:57:00 [INFO] [TRAIN] epoch: 103, iter: 37950/40000, loss: 0.4925, lr: 0.000783, batch_cost: 1.1592, reader_cost: 0.21532, ips: 0.8626 samples/sec | ETA 00:39:36 2021-05-06 22:57:08 [INFO] [TRAIN] epoch: 103, iter: 37960/40000, loss: 0.3446, lr: 0.000780, batch_cost: 0.8421, reader_cost: 0.00030, ips: 1.1875 samples/sec | ETA 00:28:37 2021-05-06 22:57:17 [INFO] [TRAIN] epoch: 103, iter: 37970/40000, loss: 0.3177, lr: 0.000777, batch_cost: 0.8378, reader_cost: 0.00014, ips: 1.1936 samples/sec | ETA 00:28:20 2021-05-06 22:57:25 [INFO] [TRAIN] epoch: 103, iter: 37980/40000, loss: 0.3032, lr: 0.000774, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1912 samples/sec | ETA 00:28:15 2021-05-06 22:57:33 [INFO] [TRAIN] epoch: 103, iter: 37990/40000, loss: 0.3331, lr: 0.000771, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 00:28:08 2021-05-06 22:57:42 [INFO] [TRAIN] epoch: 103, iter: 38000/40000, loss: 0.3378, lr: 0.000768, batch_cost: 0.8404, reader_cost: 0.00016, ips: 1.1899 samples/sec | ETA 00:28:00 2021-05-06 22:57:42 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 23:01:20 [INFO] [EVAL] #Images: 500 mIoU: 0.7780 Acc: 0.9593 Kappa: 0.9472 2021-05-06 23:01:20 [INFO] [EVAL] Class IoU: [0.9808 0.8482 0.923 0.6722 0.6245 0.5724 0.6273 0.7423 0.9207 0.6597 0.9456 0.7901 0.5883 0.9451 0.8223 0.9088 0.8193 0.6533 0.7373] 2021-05-06 23:01:20 [INFO] [EVAL] Class Acc: [0.992 0.913 0.9558 0.8561 0.8305 0.7985 0.778 0.8775 0.9512 0.8434 0.9646 0.8679 0.7535 0.9658 0.9307 0.957 0.9237 0.8068 0.8298] 2021-05-06 23:02:10 [INFO] [EVAL] The model with the best validation mIoU (0.7780) was saved at iter 38000. 2021-05-06 23:02:18 [INFO] [TRAIN] epoch: 103, iter: 38010/40000, loss: 0.3873, lr: 0.000765, batch_cost: 0.8353, reader_cost: 0.00023, ips: 1.1971 samples/sec | ETA 00:27:42 2021-05-06 23:02:27 [INFO] [TRAIN] epoch: 103, iter: 38020/40000, loss: 0.5580, lr: 0.000762, batch_cost: 0.8357, reader_cost: 0.00015, ips: 1.1967 samples/sec | ETA 00:27:34 2021-05-06 23:02:35 [INFO] [TRAIN] epoch: 103, iter: 38030/40000, loss: 0.4779, lr: 0.000759, batch_cost: 0.8388, reader_cost: 0.00015, ips: 1.1921 samples/sec | ETA 00:27:32 2021-05-06 23:02:43 [INFO] [TRAIN] epoch: 103, iter: 38040/40000, loss: 0.1730, lr: 0.000756, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 00:27:27 2021-05-06 23:02:52 [INFO] [TRAIN] epoch: 103, iter: 38050/40000, loss: 0.2649, lr: 0.000753, batch_cost: 0.8404, reader_cost: 0.00016, ips: 1.1899 samples/sec | ETA 00:27:18 2021-05-06 23:03:00 [INFO] [TRAIN] epoch: 103, iter: 38060/40000, loss: 0.3153, lr: 0.000750, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 00:27:09 2021-05-06 23:03:09 [INFO] [TRAIN] epoch: 103, iter: 38070/40000, loss: 0.3246, lr: 0.000747, batch_cost: 0.8405, reader_cost: 0.00014, ips: 1.1897 samples/sec | ETA 00:27:02 2021-05-06 23:03:17 [INFO] [TRAIN] epoch: 103, iter: 38080/40000, loss: 0.5020, lr: 0.000744, batch_cost: 0.8399, reader_cost: 0.00014, ips: 1.1907 samples/sec | ETA 00:26:52 2021-05-06 23:03:25 [INFO] [TRAIN] epoch: 103, iter: 38090/40000, loss: 0.4022, lr: 0.000741, batch_cost: 0.8398, reader_cost: 0.00014, ips: 1.1907 samples/sec | ETA 00:26:44 2021-05-06 23:03:34 [INFO] [TRAIN] epoch: 103, iter: 38100/40000, loss: 0.1847, lr: 0.000738, batch_cost: 0.8412, reader_cost: 0.00014, ips: 1.1888 samples/sec | ETA 00:26:38 2021-05-06 23:03:42 [INFO] [TRAIN] epoch: 103, iter: 38110/40000, loss: 0.3557, lr: 0.000735, batch_cost: 0.8418, reader_cost: 0.00014, ips: 1.1880 samples/sec | ETA 00:26:30 2021-05-06 23:03:51 [INFO] [TRAIN] epoch: 103, iter: 38120/40000, loss: 0.3641, lr: 0.000732, batch_cost: 0.8392, reader_cost: 0.00014, ips: 1.1916 samples/sec | ETA 00:26:17 2021-05-06 23:03:59 [INFO] [TRAIN] epoch: 103, iter: 38130/40000, loss: 0.4484, lr: 0.000729, batch_cost: 0.8411, reader_cost: 0.00014, ips: 1.1889 samples/sec | ETA 00:26:12 2021-05-06 23:04:07 [INFO] [TRAIN] epoch: 103, iter: 38140/40000, loss: 0.5049, lr: 0.000726, batch_cost: 0.8404, reader_cost: 0.00014, ips: 1.1899 samples/sec | ETA 00:26:03 2021-05-06 23:04:16 [INFO] [TRAIN] epoch: 103, iter: 38150/40000, loss: 0.3354, lr: 0.000723, batch_cost: 0.8424, reader_cost: 0.00014, ips: 1.1871 samples/sec | ETA 00:25:58 2021-05-06 23:04:24 [INFO] [TRAIN] epoch: 103, iter: 38160/40000, loss: 0.2887, lr: 0.000720, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 00:25:45 2021-05-06 23:04:33 [INFO] [TRAIN] epoch: 103, iter: 38170/40000, loss: 0.1472, lr: 0.000717, batch_cost: 0.8401, reader_cost: 0.00016, ips: 1.1903 samples/sec | ETA 00:25:37 2021-05-06 23:04:41 [INFO] [TRAIN] epoch: 103, iter: 38180/40000, loss: 0.2583, lr: 0.000714, batch_cost: 0.8420, reader_cost: 0.00016, ips: 1.1876 samples/sec | ETA 00:25:32 2021-05-06 23:04:49 [INFO] [TRAIN] epoch: 103, iter: 38190/40000, loss: 0.3096, lr: 0.000711, batch_cost: 0.8400, reader_cost: 0.00014, ips: 1.1905 samples/sec | ETA 00:25:20 2021-05-06 23:04:58 [INFO] [TRAIN] epoch: 103, iter: 38200/40000, loss: 0.2508, lr: 0.000708, batch_cost: 0.8383, reader_cost: 0.00015, ips: 1.1929 samples/sec | ETA 00:25:08 2021-05-06 23:05:06 [INFO] [TRAIN] epoch: 103, iter: 38210/40000, loss: 0.2801, lr: 0.000705, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 00:25:02 2021-05-06 23:05:15 [INFO] [TRAIN] epoch: 103, iter: 38220/40000, loss: 0.1535, lr: 0.000702, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 00:24:56 2021-05-06 23:05:23 [INFO] [TRAIN] epoch: 103, iter: 38230/40000, loss: 0.3250, lr: 0.000699, batch_cost: 0.8403, reader_cost: 0.00014, ips: 1.1901 samples/sec | ETA 00:24:47 2021-05-06 23:05:31 [INFO] [TRAIN] epoch: 103, iter: 38240/40000, loss: 0.3204, lr: 0.000696, batch_cost: 0.8387, reader_cost: 0.00014, ips: 1.1923 samples/sec | ETA 00:24:36 2021-05-06 23:05:40 [INFO] [TRAIN] epoch: 103, iter: 38250/40000, loss: 0.2653, lr: 0.000693, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1897 samples/sec | ETA 00:24:31 2021-05-06 23:05:48 [INFO] [TRAIN] epoch: 103, iter: 38260/40000, loss: 0.2914, lr: 0.000690, batch_cost: 0.8394, reader_cost: 0.00014, ips: 1.1913 samples/sec | ETA 00:24:20 2021-05-06 23:05:57 [INFO] [TRAIN] epoch: 103, iter: 38270/40000, loss: 0.3927, lr: 0.000686, batch_cost: 0.8397, reader_cost: 0.00014, ips: 1.1909 samples/sec | ETA 00:24:12 2021-05-06 23:06:05 [INFO] [TRAIN] epoch: 103, iter: 38280/40000, loss: 0.2469, lr: 0.000683, batch_cost: 0.8404, reader_cost: 0.00014, ips: 1.1899 samples/sec | ETA 00:24:05 2021-05-06 23:06:13 [INFO] [TRAIN] epoch: 103, iter: 38290/40000, loss: 0.2931, lr: 0.000680, batch_cost: 0.8413, reader_cost: 0.00014, ips: 1.1887 samples/sec | ETA 00:23:58 2021-05-06 23:06:22 [INFO] [TRAIN] epoch: 103, iter: 38300/40000, loss: 0.2337, lr: 0.000677, batch_cost: 0.8392, reader_cost: 0.00017, ips: 1.1916 samples/sec | ETA 00:23:46 2021-05-06 23:06:30 [INFO] [TRAIN] epoch: 103, iter: 38310/40000, loss: 0.3745, lr: 0.000674, batch_cost: 0.8382, reader_cost: 0.00012, ips: 1.1930 samples/sec | ETA 00:23:36 2021-05-06 23:06:42 [INFO] [TRAIN] epoch: 104, iter: 38320/40000, loss: 0.4194, lr: 0.000671, batch_cost: 1.1743, reader_cost: 0.20930, ips: 0.8515 samples/sec | ETA 00:32:52 2021-05-06 23:06:50 [INFO] [TRAIN] epoch: 104, iter: 38330/40000, loss: 0.3624, lr: 0.000668, batch_cost: 0.8466, reader_cost: 0.00032, ips: 1.1812 samples/sec | ETA 00:23:33 2021-05-06 23:06:59 [INFO] [TRAIN] epoch: 104, iter: 38340/40000, loss: 0.3610, lr: 0.000665, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1912 samples/sec | ETA 00:23:13 2021-05-06 23:07:07 [INFO] [TRAIN] epoch: 104, iter: 38350/40000, loss: 0.3192, lr: 0.000662, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 00:23:06 2021-05-06 23:07:16 [INFO] [TRAIN] epoch: 104, iter: 38360/40000, loss: 0.2651, lr: 0.000659, batch_cost: 0.8413, reader_cost: 0.00016, ips: 1.1887 samples/sec | ETA 00:22:59 2021-05-06 23:07:24 [INFO] [TRAIN] epoch: 104, iter: 38370/40000, loss: 0.3996, lr: 0.000656, batch_cost: 0.8412, reader_cost: 0.00016, ips: 1.1887 samples/sec | ETA 00:22:51 2021-05-06 23:07:32 [INFO] [TRAIN] epoch: 104, iter: 38380/40000, loss: 0.3850, lr: 0.000653, batch_cost: 0.8405, reader_cost: 0.00016, ips: 1.1898 samples/sec | ETA 00:22:41 2021-05-06 23:07:41 [INFO] [TRAIN] epoch: 104, iter: 38390/40000, loss: 0.4237, lr: 0.000650, batch_cost: 0.8406, reader_cost: 0.00016, ips: 1.1896 samples/sec | ETA 00:22:33 2021-05-06 23:07:49 [INFO] [TRAIN] epoch: 104, iter: 38400/40000, loss: 0.3722, lr: 0.000647, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1886 samples/sec | ETA 00:22:26 2021-05-06 23:07:58 [INFO] [TRAIN] epoch: 104, iter: 38410/40000, loss: 0.2862, lr: 0.000644, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 00:22:16 2021-05-06 23:08:06 [INFO] [TRAIN] epoch: 104, iter: 38420/40000, loss: 0.1652, lr: 0.000641, batch_cost: 0.8412, reader_cost: 0.00014, ips: 1.1887 samples/sec | ETA 00:22:09 2021-05-06 23:08:14 [INFO] [TRAIN] epoch: 104, iter: 38430/40000, loss: 0.3006, lr: 0.000637, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 00:21:58 2021-05-06 23:08:23 [INFO] [TRAIN] epoch: 104, iter: 38440/40000, loss: 0.3599, lr: 0.000634, batch_cost: 0.8398, reader_cost: 0.00014, ips: 1.1907 samples/sec | ETA 00:21:50 2021-05-06 23:08:31 [INFO] [TRAIN] epoch: 104, iter: 38450/40000, loss: 0.3870, lr: 0.000631, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1883 samples/sec | ETA 00:21:44 2021-05-06 23:08:40 [INFO] [TRAIN] epoch: 104, iter: 38460/40000, loss: 0.4073, lr: 0.000628, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 00:21:33 2021-05-06 23:08:48 [INFO] [TRAIN] epoch: 104, iter: 38470/40000, loss: 0.2568, lr: 0.000625, batch_cost: 0.8420, reader_cost: 0.00015, ips: 1.1876 samples/sec | ETA 00:21:28 2021-05-06 23:08:57 [INFO] [TRAIN] epoch: 104, iter: 38480/40000, loss: 0.3549, lr: 0.000622, batch_cost: 0.8400, reader_cost: 0.00016, ips: 1.1904 samples/sec | ETA 00:21:16 2021-05-06 23:09:05 [INFO] [TRAIN] epoch: 104, iter: 38490/40000, loss: 0.2991, lr: 0.000619, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 00:21:09 2021-05-06 23:09:13 [INFO] [TRAIN] epoch: 104, iter: 38500/40000, loss: 0.3473, lr: 0.000616, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 00:20:59 2021-05-06 23:09:22 [INFO] [TRAIN] epoch: 104, iter: 38510/40000, loss: 0.4883, lr: 0.000613, batch_cost: 0.8391, reader_cost: 0.00016, ips: 1.1918 samples/sec | ETA 00:20:50 2021-05-06 23:09:30 [INFO] [TRAIN] epoch: 104, iter: 38520/40000, loss: 0.2868, lr: 0.000610, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 00:20:43 2021-05-06 23:09:39 [INFO] [TRAIN] epoch: 104, iter: 38530/40000, loss: 0.2821, lr: 0.000607, batch_cost: 0.8387, reader_cost: 0.00015, ips: 1.1924 samples/sec | ETA 00:20:32 2021-05-06 23:09:47 [INFO] [TRAIN] epoch: 104, iter: 38540/40000, loss: 0.2420, lr: 0.000603, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 00:20:25 2021-05-06 23:09:55 [INFO] [TRAIN] epoch: 104, iter: 38550/40000, loss: 0.2001, lr: 0.000600, batch_cost: 0.8410, reader_cost: 0.00014, ips: 1.1891 samples/sec | ETA 00:20:19 2021-05-06 23:10:04 [INFO] [TRAIN] epoch: 104, iter: 38560/40000, loss: 0.2173, lr: 0.000597, batch_cost: 0.8410, reader_cost: 0.00014, ips: 1.1891 samples/sec | ETA 00:20:10 2021-05-06 23:10:12 [INFO] [TRAIN] epoch: 104, iter: 38570/40000, loss: 0.2990, lr: 0.000594, batch_cost: 0.8412, reader_cost: 0.00016, ips: 1.1887 samples/sec | ETA 00:20:02 2021-05-06 23:10:21 [INFO] [TRAIN] epoch: 104, iter: 38580/40000, loss: 0.2578, lr: 0.000591, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 00:19:53 2021-05-06 23:10:29 [INFO] [TRAIN] epoch: 104, iter: 38590/40000, loss: 0.2963, lr: 0.000588, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 00:19:44 2021-05-06 23:10:37 [INFO] [TRAIN] epoch: 104, iter: 38600/40000, loss: 0.2936, lr: 0.000585, batch_cost: 0.8397, reader_cost: 0.00016, ips: 1.1909 samples/sec | ETA 00:19:35 2021-05-06 23:10:46 [INFO] [TRAIN] epoch: 104, iter: 38610/40000, loss: 0.4911, lr: 0.000582, batch_cost: 0.8426, reader_cost: 0.00016, ips: 1.1868 samples/sec | ETA 00:19:31 2021-05-06 23:10:54 [INFO] [TRAIN] epoch: 104, iter: 38620/40000, loss: 0.2445, lr: 0.000579, batch_cost: 0.8414, reader_cost: 0.00016, ips: 1.1885 samples/sec | ETA 00:19:21 2021-05-06 23:11:03 [INFO] [TRAIN] epoch: 104, iter: 38630/40000, loss: 0.3486, lr: 0.000575, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1918 samples/sec | ETA 00:19:09 2021-05-06 23:11:11 [INFO] [TRAIN] epoch: 104, iter: 38640/40000, loss: 0.3312, lr: 0.000572, batch_cost: 0.8387, reader_cost: 0.00015, ips: 1.1923 samples/sec | ETA 00:19:00 2021-05-06 23:11:19 [INFO] [TRAIN] epoch: 104, iter: 38650/40000, loss: 0.2568, lr: 0.000569, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 00:18:54 2021-05-06 23:11:28 [INFO] [TRAIN] epoch: 104, iter: 38660/40000, loss: 0.3168, lr: 0.000566, batch_cost: 0.8393, reader_cost: 0.00016, ips: 1.1914 samples/sec | ETA 00:18:44 2021-05-06 23:11:36 [INFO] [TRAIN] epoch: 104, iter: 38670/40000, loss: 0.2837, lr: 0.000563, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1914 samples/sec | ETA 00:18:36 2021-05-06 23:11:45 [INFO] [TRAIN] epoch: 104, iter: 38680/40000, loss: 0.4208, lr: 0.000560, batch_cost: 0.8389, reader_cost: 0.00013, ips: 1.1920 samples/sec | ETA 00:18:27 2021-05-06 23:11:56 [INFO] [TRAIN] epoch: 105, iter: 38690/40000, loss: 0.3489, lr: 0.000557, batch_cost: 1.1375, reader_cost: 0.20939, ips: 0.8791 samples/sec | ETA 00:24:50 2021-05-06 23:12:05 [INFO] [TRAIN] epoch: 105, iter: 38700/40000, loss: 0.3226, lr: 0.000554, batch_cost: 0.8590, reader_cost: 0.00034, ips: 1.1641 samples/sec | ETA 00:18:36 2021-05-06 23:12:13 [INFO] [TRAIN] epoch: 105, iter: 38710/40000, loss: 0.4475, lr: 0.000550, batch_cost: 0.8418, reader_cost: 0.00016, ips: 1.1879 samples/sec | ETA 00:18:05 2021-05-06 23:12:21 [INFO] [TRAIN] epoch: 105, iter: 38720/40000, loss: 0.3543, lr: 0.000547, batch_cost: 0.8415, reader_cost: 0.00014, ips: 1.1884 samples/sec | ETA 00:17:57 2021-05-06 23:12:30 [INFO] [TRAIN] epoch: 105, iter: 38730/40000, loss: 0.2205, lr: 0.000544, batch_cost: 0.8386, reader_cost: 0.00015, ips: 1.1925 samples/sec | ETA 00:17:45 2021-05-06 23:12:38 [INFO] [TRAIN] epoch: 105, iter: 38740/40000, loss: 0.4863, lr: 0.000541, batch_cost: 0.8417, reader_cost: 0.00017, ips: 1.1881 samples/sec | ETA 00:17:40 2021-05-06 23:12:47 [INFO] [TRAIN] epoch: 105, iter: 38750/40000, loss: 0.4464, lr: 0.000538, batch_cost: 0.8394, reader_cost: 0.00016, ips: 1.1914 samples/sec | ETA 00:17:29 2021-05-06 23:12:55 [INFO] [TRAIN] epoch: 105, iter: 38760/40000, loss: 0.3686, lr: 0.000535, batch_cost: 0.8390, reader_cost: 0.00016, ips: 1.1919 samples/sec | ETA 00:17:20 2021-05-06 23:13:03 [INFO] [TRAIN] epoch: 105, iter: 38770/40000, loss: 0.3498, lr: 0.000532, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 00:17:14 2021-05-06 23:13:12 [INFO] [TRAIN] epoch: 105, iter: 38780/40000, loss: 0.2482, lr: 0.000528, batch_cost: 0.8385, reader_cost: 0.00015, ips: 1.1926 samples/sec | ETA 00:17:02 2021-05-06 23:13:20 [INFO] [TRAIN] epoch: 105, iter: 38790/40000, loss: 0.2007, lr: 0.000525, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 00:16:56 2021-05-06 23:13:29 [INFO] [TRAIN] epoch: 105, iter: 38800/40000, loss: 0.2728, lr: 0.000522, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 00:16:48 2021-05-06 23:13:37 [INFO] [TRAIN] epoch: 105, iter: 38810/40000, loss: 0.3507, lr: 0.000519, batch_cost: 0.8398, reader_cost: 0.00016, ips: 1.1907 samples/sec | ETA 00:16:39 2021-05-06 23:13:45 [INFO] [TRAIN] epoch: 105, iter: 38820/40000, loss: 0.4782, lr: 0.000516, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 00:16:30 2021-05-06 23:13:54 [INFO] [TRAIN] epoch: 105, iter: 38830/40000, loss: 0.4570, lr: 0.000513, batch_cost: 0.8399, reader_cost: 0.00015, ips: 1.1906 samples/sec | ETA 00:16:22 2021-05-06 23:14:02 [INFO] [TRAIN] epoch: 105, iter: 38840/40000, loss: 0.2680, lr: 0.000509, batch_cost: 0.8393, reader_cost: 0.00016, ips: 1.1915 samples/sec | ETA 00:16:13 2021-05-06 23:14:11 [INFO] [TRAIN] epoch: 105, iter: 38850/40000, loss: 0.2500, lr: 0.000506, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 00:16:06 2021-05-06 23:14:19 [INFO] [TRAIN] epoch: 105, iter: 38860/40000, loss: 0.2322, lr: 0.000503, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1883 samples/sec | ETA 00:15:59 2021-05-06 23:14:27 [INFO] [TRAIN] epoch: 105, iter: 38870/40000, loss: 0.4386, lr: 0.000500, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 00:15:49 2021-05-06 23:14:36 [INFO] [TRAIN] epoch: 105, iter: 38880/40000, loss: 0.6208, lr: 0.000497, batch_cost: 0.8403, reader_cost: 0.00016, ips: 1.1900 samples/sec | ETA 00:15:41 2021-05-06 23:14:44 [INFO] [TRAIN] epoch: 105, iter: 38890/40000, loss: 0.2796, lr: 0.000493, batch_cost: 0.8438, reader_cost: 0.00015, ips: 1.1851 samples/sec | ETA 00:15:36 2021-05-06 23:14:53 [INFO] [TRAIN] epoch: 105, iter: 38900/40000, loss: 0.4427, lr: 0.000490, batch_cost: 0.8415, reader_cost: 0.00015, ips: 1.1883 samples/sec | ETA 00:15:25 2021-05-06 23:15:01 [INFO] [TRAIN] epoch: 105, iter: 38910/40000, loss: 0.2339, lr: 0.000487, batch_cost: 0.8418, reader_cost: 0.00015, ips: 1.1879 samples/sec | ETA 00:15:17 2021-05-06 23:15:09 [INFO] [TRAIN] epoch: 105, iter: 38920/40000, loss: 0.2095, lr: 0.000484, batch_cost: 0.8403, reader_cost: 0.00014, ips: 1.1901 samples/sec | ETA 00:15:07 2021-05-06 23:15:18 [INFO] [TRAIN] epoch: 105, iter: 38930/40000, loss: 0.2319, lr: 0.000481, batch_cost: 0.8407, reader_cost: 0.00014, ips: 1.1894 samples/sec | ETA 00:14:59 2021-05-06 23:15:26 [INFO] [TRAIN] epoch: 105, iter: 38940/40000, loss: 0.2094, lr: 0.000478, batch_cost: 0.8408, reader_cost: 0.00014, ips: 1.1893 samples/sec | ETA 00:14:51 2021-05-06 23:15:35 [INFO] [TRAIN] epoch: 105, iter: 38950/40000, loss: 0.2973, lr: 0.000474, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 00:14:41 2021-05-06 23:15:43 [INFO] [TRAIN] epoch: 105, iter: 38960/40000, loss: 0.2884, lr: 0.000471, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1888 samples/sec | ETA 00:14:34 2021-05-06 23:15:51 [INFO] [TRAIN] epoch: 105, iter: 38970/40000, loss: 0.2069, lr: 0.000468, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 00:14:25 2021-05-06 23:16:00 [INFO] [TRAIN] epoch: 105, iter: 38980/40000, loss: 0.3436, lr: 0.000465, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1901 samples/sec | ETA 00:14:17 2021-05-06 23:16:08 [INFO] [TRAIN] epoch: 105, iter: 38990/40000, loss: 0.3057, lr: 0.000461, batch_cost: 0.8413, reader_cost: 0.00015, ips: 1.1887 samples/sec | ETA 00:14:09 2021-05-06 23:16:17 [INFO] [TRAIN] epoch: 105, iter: 39000/40000, loss: 0.2212, lr: 0.000458, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1912 samples/sec | ETA 00:13:59 2021-05-06 23:16:17 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 23:19:53 [INFO] [EVAL] #Images: 500 mIoU: 0.7776 Acc: 0.9596 Kappa: 0.9476 2021-05-06 23:19:53 [INFO] [EVAL] Class IoU: [0.9812 0.8519 0.9232 0.6728 0.6238 0.5714 0.6287 0.7437 0.9215 0.6755 0.9466 0.7906 0.5836 0.9449 0.8191 0.9039 0.8045 0.6494 0.7385] 2021-05-06 23:19:53 [INFO] [EVAL] Class Acc: [0.9925 0.9187 0.9545 0.8612 0.8054 0.802 0.7746 0.8827 0.9529 0.8265 0.967 0.8731 0.7613 0.9656 0.9201 0.9477 0.9331 0.8283 0.8311] 2021-05-06 23:20:22 [INFO] [EVAL] The model with the best validation mIoU (0.7780) was saved at iter 38000. 2021-05-06 23:20:30 [INFO] [TRAIN] epoch: 105, iter: 39010/40000, loss: 0.3899, lr: 0.000455, batch_cost: 0.8372, reader_cost: 0.00016, ips: 1.1944 samples/sec | ETA 00:13:48 2021-05-06 23:20:42 [INFO] [TRAIN] epoch: 105, iter: 39020/40000, loss: 0.3465, lr: 0.000452, batch_cost: 0.8365, reader_cost: 0.00016, ips: 1.1955 samples/sec | ETA 00:13:39 2021-05-06 23:20:50 [INFO] [TRAIN] epoch: 105, iter: 39030/40000, loss: 0.3651, lr: 0.000449, batch_cost: 0.8387, reader_cost: 0.00015, ips: 1.1923 samples/sec | ETA 00:13:33 2021-05-06 23:20:58 [INFO] [TRAIN] epoch: 105, iter: 39040/40000, loss: 0.2992, lr: 0.000445, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 00:13:27 2021-05-06 23:21:07 [INFO] [TRAIN] epoch: 105, iter: 39050/40000, loss: 0.3345, lr: 0.000442, batch_cost: 0.8395, reader_cost: 0.00014, ips: 1.1911 samples/sec | ETA 00:13:17 2021-05-06 23:21:15 [INFO] [TRAIN] epoch: 105, iter: 39060/40000, loss: 0.3149, lr: 0.000439, batch_cost: 0.8382, reader_cost: 0.00010, ips: 1.1930 samples/sec | ETA 00:13:07 2021-05-06 23:21:27 [INFO] [TRAIN] epoch: 106, iter: 39070/40000, loss: 0.4100, lr: 0.000436, batch_cost: 1.1922, reader_cost: 0.21364, ips: 0.8388 samples/sec | ETA 00:18:28 2021-05-06 23:21:35 [INFO] [TRAIN] epoch: 106, iter: 39080/40000, loss: 0.2757, lr: 0.000432, batch_cost: 0.8430, reader_cost: 0.00032, ips: 1.1863 samples/sec | ETA 00:12:55 2021-05-06 23:21:44 [INFO] [TRAIN] epoch: 106, iter: 39090/40000, loss: 0.3131, lr: 0.000429, batch_cost: 0.8431, reader_cost: 0.00015, ips: 1.1862 samples/sec | ETA 00:12:47 2021-05-06 23:21:52 [INFO] [TRAIN] epoch: 106, iter: 39100/40000, loss: 0.1648, lr: 0.000426, batch_cost: 0.8405, reader_cost: 0.00017, ips: 1.1898 samples/sec | ETA 00:12:36 2021-05-06 23:22:01 [INFO] [TRAIN] epoch: 106, iter: 39110/40000, loss: 0.5381, lr: 0.000423, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1913 samples/sec | ETA 00:12:27 2021-05-06 23:22:09 [INFO] [TRAIN] epoch: 106, iter: 39120/40000, loss: 0.4490, lr: 0.000419, batch_cost: 0.8393, reader_cost: 0.00014, ips: 1.1915 samples/sec | ETA 00:12:18 2021-05-06 23:22:17 [INFO] [TRAIN] epoch: 106, iter: 39130/40000, loss: 0.4717, lr: 0.000416, batch_cost: 0.8424, reader_cost: 0.00014, ips: 1.1872 samples/sec | ETA 00:12:12 2021-05-06 23:22:26 [INFO] [TRAIN] epoch: 106, iter: 39140/40000, loss: 0.4651, lr: 0.000413, batch_cost: 0.8397, reader_cost: 0.00014, ips: 1.1909 samples/sec | ETA 00:12:02 2021-05-06 23:22:34 [INFO] [TRAIN] epoch: 106, iter: 39150/40000, loss: 0.3747, lr: 0.000410, batch_cost: 0.8398, reader_cost: 0.00015, ips: 1.1907 samples/sec | ETA 00:11:53 2021-05-06 23:22:43 [INFO] [TRAIN] epoch: 106, iter: 39160/40000, loss: 0.1346, lr: 0.000406, batch_cost: 0.8397, reader_cost: 0.00014, ips: 1.1909 samples/sec | ETA 00:11:45 2021-05-06 23:22:51 [INFO] [TRAIN] epoch: 106, iter: 39170/40000, loss: 0.2671, lr: 0.000403, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 00:11:37 2021-05-06 23:22:59 [INFO] [TRAIN] epoch: 106, iter: 39180/40000, loss: 0.3542, lr: 0.000400, batch_cost: 0.8392, reader_cost: 0.00016, ips: 1.1916 samples/sec | ETA 00:11:28 2021-05-06 23:23:08 [INFO] [TRAIN] epoch: 106, iter: 39190/40000, loss: 0.4524, lr: 0.000396, batch_cost: 0.8381, reader_cost: 0.00015, ips: 1.1931 samples/sec | ETA 00:11:18 2021-05-06 23:23:16 [INFO] [TRAIN] epoch: 106, iter: 39200/40000, loss: 0.4836, lr: 0.000393, batch_cost: 0.8386, reader_cost: 0.00014, ips: 1.1924 samples/sec | ETA 00:11:10 2021-05-06 23:23:25 [INFO] [TRAIN] epoch: 106, iter: 39210/40000, loss: 0.4461, lr: 0.000390, batch_cost: 0.8403, reader_cost: 0.00014, ips: 1.1901 samples/sec | ETA 00:11:03 2021-05-06 23:23:33 [INFO] [TRAIN] epoch: 106, iter: 39220/40000, loss: 0.2169, lr: 0.000387, batch_cost: 0.8402, reader_cost: 0.00014, ips: 1.1902 samples/sec | ETA 00:10:55 2021-05-06 23:23:41 [INFO] [TRAIN] epoch: 106, iter: 39230/40000, loss: 0.2995, lr: 0.000383, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 00:10:46 2021-05-06 23:23:50 [INFO] [TRAIN] epoch: 106, iter: 39240/40000, loss: 0.3113, lr: 0.000380, batch_cost: 0.8399, reader_cost: 0.00016, ips: 1.1906 samples/sec | ETA 00:10:38 2021-05-06 23:23:58 [INFO] [TRAIN] epoch: 106, iter: 39250/40000, loss: 0.4243, lr: 0.000377, batch_cost: 0.8385, reader_cost: 0.00015, ips: 1.1926 samples/sec | ETA 00:10:28 2021-05-06 23:24:07 [INFO] [TRAIN] epoch: 106, iter: 39260/40000, loss: 0.3926, lr: 0.000373, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 00:10:21 2021-05-06 23:24:15 [INFO] [TRAIN] epoch: 106, iter: 39270/40000, loss: 0.3138, lr: 0.000370, batch_cost: 0.8404, reader_cost: 0.00016, ips: 1.1899 samples/sec | ETA 00:10:13 2021-05-06 23:24:23 [INFO] [TRAIN] epoch: 106, iter: 39280/40000, loss: 0.2928, lr: 0.000367, batch_cost: 0.8412, reader_cost: 0.00015, ips: 1.1888 samples/sec | ETA 00:10:05 2021-05-06 23:24:32 [INFO] [TRAIN] epoch: 106, iter: 39290/40000, loss: 0.1520, lr: 0.000363, batch_cost: 0.8407, reader_cost: 0.00016, ips: 1.1895 samples/sec | ETA 00:09:56 2021-05-06 23:24:40 [INFO] [TRAIN] epoch: 106, iter: 39300/40000, loss: 0.3128, lr: 0.000360, batch_cost: 0.8420, reader_cost: 0.00017, ips: 1.1877 samples/sec | ETA 00:09:49 2021-05-06 23:24:49 [INFO] [TRAIN] epoch: 106, iter: 39310/40000, loss: 0.2036, lr: 0.000357, batch_cost: 0.8404, reader_cost: 0.00016, ips: 1.1899 samples/sec | ETA 00:09:39 2021-05-06 23:24:57 [INFO] [TRAIN] epoch: 106, iter: 39320/40000, loss: 0.2872, lr: 0.000353, batch_cost: 0.8396, reader_cost: 0.00016, ips: 1.1911 samples/sec | ETA 00:09:30 2021-05-06 23:25:05 [INFO] [TRAIN] epoch: 106, iter: 39330/40000, loss: 0.2442, lr: 0.000350, batch_cost: 0.8403, reader_cost: 0.00016, ips: 1.1901 samples/sec | ETA 00:09:22 2021-05-06 23:25:14 [INFO] [TRAIN] epoch: 106, iter: 39340/40000, loss: 0.2465, lr: 0.000347, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1911 samples/sec | ETA 00:09:14 2021-05-06 23:25:22 [INFO] [TRAIN] epoch: 106, iter: 39350/40000, loss: 0.3642, lr: 0.000343, batch_cost: 0.8402, reader_cost: 0.00015, ips: 1.1902 samples/sec | ETA 00:09:06 2021-05-06 23:25:31 [INFO] [TRAIN] epoch: 106, iter: 39360/40000, loss: 0.2778, lr: 0.000340, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1908 samples/sec | ETA 00:08:57 2021-05-06 23:25:39 [INFO] [TRAIN] epoch: 106, iter: 39370/40000, loss: 0.3372, lr: 0.000336, batch_cost: 0.8386, reader_cost: 0.00015, ips: 1.1924 samples/sec | ETA 00:08:48 2021-05-06 23:25:47 [INFO] [TRAIN] epoch: 106, iter: 39380/40000, loss: 0.3548, lr: 0.000333, batch_cost: 0.8390, reader_cost: 0.00015, ips: 1.1919 samples/sec | ETA 00:08:40 2021-05-06 23:25:56 [INFO] [TRAIN] epoch: 106, iter: 39390/40000, loss: 0.2695, lr: 0.000330, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1898 samples/sec | ETA 00:08:32 2021-05-06 23:26:04 [INFO] [TRAIN] epoch: 106, iter: 39400/40000, loss: 0.3204, lr: 0.000326, batch_cost: 0.8390, reader_cost: 0.00014, ips: 1.1919 samples/sec | ETA 00:08:23 2021-05-06 23:26:13 [INFO] [TRAIN] epoch: 106, iter: 39410/40000, loss: 0.4494, lr: 0.000323, batch_cost: 0.8391, reader_cost: 0.00015, ips: 1.1918 samples/sec | ETA 00:08:15 2021-05-06 23:26:21 [INFO] [TRAIN] epoch: 106, iter: 39420/40000, loss: 0.3048, lr: 0.000320, batch_cost: 0.8394, reader_cost: 0.00015, ips: 1.1914 samples/sec | ETA 00:08:06 2021-05-06 23:26:29 [INFO] [TRAIN] epoch: 106, iter: 39430/40000, loss: 0.2922, lr: 0.000316, batch_cost: 0.8379, reader_cost: 0.00028, ips: 1.1934 samples/sec | ETA 00:07:57 2021-05-06 23:26:41 [INFO] [TRAIN] epoch: 107, iter: 39440/40000, loss: 0.4245, lr: 0.000313, batch_cost: 1.1749, reader_cost: 0.21508, ips: 0.8512 samples/sec | ETA 00:10:57 2021-05-06 23:26:50 [INFO] [TRAIN] epoch: 107, iter: 39450/40000, loss: 0.2918, lr: 0.000309, batch_cost: 0.8411, reader_cost: 0.00030, ips: 1.1889 samples/sec | ETA 00:07:42 2021-05-06 23:26:58 [INFO] [TRAIN] epoch: 107, iter: 39460/40000, loss: 0.3745, lr: 0.000306, batch_cost: 0.8402, reader_cost: 0.00016, ips: 1.1901 samples/sec | ETA 00:07:33 2021-05-06 23:27:06 [INFO] [TRAIN] epoch: 107, iter: 39470/40000, loss: 0.2234, lr: 0.000302, batch_cost: 0.8417, reader_cost: 0.00016, ips: 1.1881 samples/sec | ETA 00:07:26 2021-05-06 23:27:15 [INFO] [TRAIN] epoch: 107, iter: 39480/40000, loss: 0.3940, lr: 0.000299, batch_cost: 0.8404, reader_cost: 0.00015, ips: 1.1899 samples/sec | ETA 00:07:17 2021-05-06 23:27:23 [INFO] [TRAIN] epoch: 107, iter: 39490/40000, loss: 0.4825, lr: 0.000296, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 00:07:08 2021-05-06 23:27:32 [INFO] [TRAIN] epoch: 107, iter: 39500/40000, loss: 0.5300, lr: 0.000292, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1892 samples/sec | ETA 00:07:00 2021-05-06 23:27:40 [INFO] [TRAIN] epoch: 107, iter: 39510/40000, loss: 0.4595, lr: 0.000289, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 00:06:51 2021-05-06 23:27:48 [INFO] [TRAIN] epoch: 107, iter: 39520/40000, loss: 0.3133, lr: 0.000285, batch_cost: 0.8402, reader_cost: 0.00014, ips: 1.1901 samples/sec | ETA 00:06:43 2021-05-06 23:27:57 [INFO] [TRAIN] epoch: 107, iter: 39530/40000, loss: 0.1362, lr: 0.000282, batch_cost: 0.8406, reader_cost: 0.00015, ips: 1.1896 samples/sec | ETA 00:06:35 2021-05-06 23:28:05 [INFO] [TRAIN] epoch: 107, iter: 39540/40000, loss: 0.2118, lr: 0.000278, batch_cost: 0.8399, reader_cost: 0.00016, ips: 1.1907 samples/sec | ETA 00:06:26 2021-05-06 23:28:14 [INFO] [TRAIN] epoch: 107, iter: 39550/40000, loss: 0.3077, lr: 0.000275, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 00:06:18 2021-05-06 23:28:22 [INFO] [TRAIN] epoch: 107, iter: 39560/40000, loss: 0.3856, lr: 0.000271, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1905 samples/sec | ETA 00:06:09 2021-05-06 23:28:30 [INFO] [TRAIN] epoch: 107, iter: 39570/40000, loss: 0.5005, lr: 0.000268, batch_cost: 0.8401, reader_cost: 0.00016, ips: 1.1904 samples/sec | ETA 00:06:01 2021-05-06 23:28:39 [INFO] [TRAIN] epoch: 107, iter: 39580/40000, loss: 0.3237, lr: 0.000264, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 00:05:52 2021-05-06 23:28:47 [INFO] [TRAIN] epoch: 107, iter: 39590/40000, loss: 0.2231, lr: 0.000261, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1895 samples/sec | ETA 00:05:44 2021-05-06 23:28:56 [INFO] [TRAIN] epoch: 107, iter: 39600/40000, loss: 0.2518, lr: 0.000257, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 00:05:36 2021-05-06 23:29:04 [INFO] [TRAIN] epoch: 107, iter: 39610/40000, loss: 0.3770, lr: 0.000254, batch_cost: 0.8408, reader_cost: 0.00014, ips: 1.1894 samples/sec | ETA 00:05:27 2021-05-06 23:29:12 [INFO] [TRAIN] epoch: 107, iter: 39620/40000, loss: 0.7165, lr: 0.000250, batch_cost: 0.8397, reader_cost: 0.00014, ips: 1.1910 samples/sec | ETA 00:05:19 2021-05-06 23:29:21 [INFO] [TRAIN] epoch: 107, iter: 39630/40000, loss: 0.4402, lr: 0.000247, batch_cost: 0.8389, reader_cost: 0.00014, ips: 1.1920 samples/sec | ETA 00:05:10 2021-05-06 23:29:29 [INFO] [TRAIN] epoch: 107, iter: 39640/40000, loss: 0.3207, lr: 0.000243, batch_cost: 0.8400, reader_cost: 0.00015, ips: 1.1904 samples/sec | ETA 00:05:02 2021-05-06 23:29:38 [INFO] [TRAIN] epoch: 107, iter: 39650/40000, loss: 0.3513, lr: 0.000239, batch_cost: 0.8394, reader_cost: 0.00017, ips: 1.1913 samples/sec | ETA 00:04:53 2021-05-06 23:29:46 [INFO] [TRAIN] epoch: 107, iter: 39660/40000, loss: 0.1783, lr: 0.000236, batch_cost: 0.8407, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 00:04:45 2021-05-06 23:29:54 [INFO] [TRAIN] epoch: 107, iter: 39670/40000, loss: 0.2120, lr: 0.000232, batch_cost: 0.8405, reader_cost: 0.00016, ips: 1.1897 samples/sec | ETA 00:04:37 2021-05-06 23:30:03 [INFO] [TRAIN] epoch: 107, iter: 39680/40000, loss: 0.1664, lr: 0.000229, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 00:04:28 2021-05-06 23:30:11 [INFO] [TRAIN] epoch: 107, iter: 39690/40000, loss: 0.2622, lr: 0.000225, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1911 samples/sec | ETA 00:04:20 2021-05-06 23:30:20 [INFO] [TRAIN] epoch: 107, iter: 39700/40000, loss: 0.2710, lr: 0.000221, batch_cost: 0.8403, reader_cost: 0.00015, ips: 1.1900 samples/sec | ETA 00:04:12 2021-05-06 23:30:28 [INFO] [TRAIN] epoch: 107, iter: 39710/40000, loss: 0.2396, lr: 0.000218, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1910 samples/sec | ETA 00:04:03 2021-05-06 23:30:36 [INFO] [TRAIN] epoch: 107, iter: 39720/40000, loss: 0.2273, lr: 0.000214, batch_cost: 0.8408, reader_cost: 0.00014, ips: 1.1893 samples/sec | ETA 00:03:55 2021-05-06 23:30:45 [INFO] [TRAIN] epoch: 107, iter: 39730/40000, loss: 0.3175, lr: 0.000211, batch_cost: 0.8385, reader_cost: 0.00015, ips: 1.1927 samples/sec | ETA 00:03:46 2021-05-06 23:30:53 [INFO] [TRAIN] epoch: 107, iter: 39740/40000, loss: 0.2242, lr: 0.000207, batch_cost: 0.8385, reader_cost: 0.00014, ips: 1.1926 samples/sec | ETA 00:03:38 2021-05-06 23:31:02 [INFO] [TRAIN] epoch: 107, iter: 39750/40000, loss: 0.3442, lr: 0.000203, batch_cost: 0.8400, reader_cost: 0.00014, ips: 1.1905 samples/sec | ETA 00:03:29 2021-05-06 23:31:10 [INFO] [TRAIN] epoch: 107, iter: 39760/40000, loss: 0.2868, lr: 0.000199, batch_cost: 0.8385, reader_cost: 0.00014, ips: 1.1927 samples/sec | ETA 00:03:21 2021-05-06 23:31:18 [INFO] [TRAIN] epoch: 107, iter: 39770/40000, loss: 0.2981, lr: 0.000196, batch_cost: 0.8409, reader_cost: 0.00015, ips: 1.1893 samples/sec | ETA 00:03:13 2021-05-06 23:31:27 [INFO] [TRAIN] epoch: 107, iter: 39780/40000, loss: 0.3628, lr: 0.000192, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 00:03:05 2021-05-06 23:31:35 [INFO] [TRAIN] epoch: 107, iter: 39790/40000, loss: 0.2461, lr: 0.000188, batch_cost: 0.8390, reader_cost: 0.00015, ips: 1.1919 samples/sec | ETA 00:02:56 2021-05-06 23:31:44 [INFO] [TRAIN] epoch: 107, iter: 39800/40000, loss: 0.2937, lr: 0.000184, batch_cost: 0.8380, reader_cost: 0.00011, ips: 1.1933 samples/sec | ETA 00:02:47 2021-05-06 23:31:55 [INFO] [TRAIN] epoch: 108, iter: 39810/40000, loss: 0.4582, lr: 0.000181, batch_cost: 1.1730, reader_cost: 0.20825, ips: 0.8525 samples/sec | ETA 00:03:42 2021-05-06 23:32:04 [INFO] [TRAIN] epoch: 108, iter: 39820/40000, loss: 0.2593, lr: 0.000177, batch_cost: 0.8406, reader_cost: 0.00030, ips: 1.1896 samples/sec | ETA 00:02:31 2021-05-06 23:32:12 [INFO] [TRAIN] epoch: 108, iter: 39830/40000, loss: 0.2837, lr: 0.000173, batch_cost: 0.8410, reader_cost: 0.00015, ips: 1.1891 samples/sec | ETA 00:02:22 2021-05-06 23:32:21 [INFO] [TRAIN] epoch: 108, iter: 39840/40000, loss: 0.4627, lr: 0.000169, batch_cost: 0.8402, reader_cost: 0.00014, ips: 1.1902 samples/sec | ETA 00:02:14 2021-05-06 23:32:29 [INFO] [TRAIN] epoch: 108, iter: 39850/40000, loss: 0.2306, lr: 0.000165, batch_cost: 0.8411, reader_cost: 0.00015, ips: 1.1889 samples/sec | ETA 00:02:06 2021-05-06 23:32:37 [INFO] [TRAIN] epoch: 108, iter: 39860/40000, loss: 0.3311, lr: 0.000161, batch_cost: 0.8432, reader_cost: 0.00015, ips: 1.1860 samples/sec | ETA 00:01:58 2021-05-06 23:32:46 [INFO] [TRAIN] epoch: 108, iter: 39870/40000, loss: 0.4357, lr: 0.000157, batch_cost: 0.8395, reader_cost: 0.00015, ips: 1.1912 samples/sec | ETA 00:01:49 2021-05-06 23:32:54 [INFO] [TRAIN] epoch: 108, iter: 39880/40000, loss: 0.5642, lr: 0.000153, batch_cost: 0.8379, reader_cost: 0.00015, ips: 1.1934 samples/sec | ETA 00:01:40 2021-05-06 23:33:03 [INFO] [TRAIN] epoch: 108, iter: 39890/40000, loss: 0.3046, lr: 0.000149, batch_cost: 0.8396, reader_cost: 0.00015, ips: 1.1911 samples/sec | ETA 00:01:32 2021-05-06 23:33:11 [INFO] [TRAIN] epoch: 108, iter: 39900/40000, loss: 0.1582, lr: 0.000145, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 00:01:23 2021-05-06 23:33:19 [INFO] [TRAIN] epoch: 108, iter: 39910/40000, loss: 0.2521, lr: 0.000141, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 00:01:15 2021-05-06 23:33:28 [INFO] [TRAIN] epoch: 108, iter: 39920/40000, loss: 0.4890, lr: 0.000137, batch_cost: 0.8408, reader_cost: 0.00015, ips: 1.1894 samples/sec | ETA 00:01:07 2021-05-06 23:33:36 [INFO] [TRAIN] epoch: 108, iter: 39930/40000, loss: 0.3595, lr: 0.000133, batch_cost: 0.8390, reader_cost: 0.00016, ips: 1.1919 samples/sec | ETA 00:00:58 2021-05-06 23:33:45 [INFO] [TRAIN] epoch: 108, iter: 39940/40000, loss: 0.5085, lr: 0.000129, batch_cost: 0.8389, reader_cost: 0.00017, ips: 1.1920 samples/sec | ETA 00:00:50 2021-05-06 23:33:53 [INFO] [TRAIN] epoch: 108, iter: 39950/40000, loss: 0.3599, lr: 0.000125, batch_cost: 0.8399, reader_cost: 0.00016, ips: 1.1906 samples/sec | ETA 00:00:41 2021-05-06 23:34:01 [INFO] [TRAIN] epoch: 108, iter: 39960/40000, loss: 0.2718, lr: 0.000120, batch_cost: 0.8394, reader_cost: 0.00014, ips: 1.1913 samples/sec | ETA 00:00:33 2021-05-06 23:34:10 [INFO] [TRAIN] epoch: 108, iter: 39970/40000, loss: 0.3287, lr: 0.000116, batch_cost: 0.8397, reader_cost: 0.00015, ips: 1.1909 samples/sec | ETA 00:00:25 2021-05-06 23:34:18 [INFO] [TRAIN] epoch: 108, iter: 39980/40000, loss: 0.2796, lr: 0.000111, batch_cost: 0.8400, reader_cost: 0.00016, ips: 1.1905 samples/sec | ETA 00:00:16 2021-05-06 23:34:27 [INFO] [TRAIN] epoch: 108, iter: 39990/40000, loss: 0.4327, lr: 0.000106, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 00:00:08 2021-05-06 23:34:35 [INFO] [TRAIN] epoch: 108, iter: 40000/40000, loss: 0.4211, lr: 0.000101, batch_cost: 0.8401, reader_cost: 0.00015, ips: 1.1903 samples/sec | ETA 00:00:00 2021-05-06 23:34:35 [INFO] Start evaluating (total_samples: 500, total_iters: 63)... 2021-05-06 23:38:15 [INFO] [EVAL] #Images: 500 mIoU: 0.7808 Acc: 0.9599 Kappa: 0.9480 2021-05-06 23:38:15 [INFO] [EVAL] Class IoU: [0.9815 0.853 0.9239 0.6772 0.6217 0.5777 0.632 0.7455 0.9216 0.6712 0.9468 0.7926 0.599 0.9454 0.8215 0.9096 0.8212 0.6543 0.7394] 2021-05-06 23:38:15 [INFO] [EVAL] Class Acc: [0.9927 0.9168 0.9567 0.8606 0.8169 0.791 0.7843 0.8666 0.9527 0.829 0.968 0.867 0.7594 0.9669 0.9182 0.9588 0.925 0.7937 0.8343] 2021-05-06 23:39:15 [INFO] [EVAL] The model with the best validation mIoU (0.7808) was saved at iter 40000. 's flops has been counted 's flops has been counted Cannot find suitable count function for . Treat it as zero FLOPs. 's flops has been counted Cannot find suitable count function for . Treat it as zero FLOPs. Cannot find suitable count function for . Treat it as zero FLOPs. Customize Function has been applied to Cannot find suitable count function for . Treat it as zero FLOPs. /ssd1/home/wuzewu/miniconda3/envs/paddle2/lib/python3.7/site-packages/paddle/fluid/dygraph/math_op_patch.py:238: UserWarning: The dtype of left and right variables are not the same, left dtype is VarType.FP32, but right dtype is VarType.INT32, the right dtype will convert to VarType.FP32 format(lhs_dtype, rhs_dtype, lhs_dtype)) Total Flops: -83885703168 Total Params: 307237651 INFO 2021-05-06 23:39:20,044 launch.py:240] Local processes completed.