LAUNCH INFO 2022-07-29 18:54:15,932 ----------- Configuration ---------------------- LAUNCH INFO 2022-07-29 18:54:15,933 devices: None LAUNCH INFO 2022-07-29 18:54:15,933 elastic_level: -1 LAUNCH INFO 2022-07-29 18:54:15,933 elastic_timeout: 30 LAUNCH INFO 2022-07-29 18:54:15,933 gloo_port: 6767 LAUNCH INFO 2022-07-29 18:54:15,933 host: None LAUNCH INFO 2022-07-29 18:54:15,933 job_id: default LAUNCH INFO 2022-07-29 18:54:15,933 legacy: False LAUNCH INFO 2022-07-29 18:54:15,933 log_dir: log LAUNCH INFO 2022-07-29 18:54:15,933 log_level: INFO LAUNCH INFO 2022-07-29 18:54:15,933 master: None LAUNCH INFO 2022-07-29 18:54:15,933 max_restart: 3 LAUNCH INFO 2022-07-29 18:54:15,933 nnodes: 1 LAUNCH INFO 2022-07-29 18:54:15,933 nproc_per_node: None LAUNCH INFO 2022-07-29 18:54:15,933 rank: -1 LAUNCH INFO 2022-07-29 18:54:15,933 run_mode: collective LAUNCH INFO 2022-07-29 18:54:15,933 server_num: None LAUNCH INFO 2022-07-29 18:54:15,933 servers: LAUNCH INFO 2022-07-29 18:54:15,933 trainer_num: None LAUNCH INFO 2022-07-29 18:54:15,933 trainers: LAUNCH INFO 2022-07-29 18:54:15,933 training_script: tools/train.py LAUNCH INFO 2022-07-29 18:54:15,933 training_script_args: ['--config', 'configs/smoke/smoke_dla34_no_dcn_iter70000.yml', '--num_workers', '3', '--log_interval', '50', '--save_interval', '5000'] LAUNCH INFO 2022-07-29 18:54:15,933 with_gloo: 0 LAUNCH INFO 2022-07-29 18:54:15,933 -------------------------------------------------- LAUNCH INFO 2022-07-29 18:54:16,027 Job: default, mode collective, replicas 1[1:1], elastic False LAUNCH INFO 2022-07-29 18:54:16,028 Run Pod: mwzpon, replicas 4, status ready LAUNCH INFO 2022-07-29 18:54:16,066 Watching Pod: mwzpon, replicas 4, status running 2022-07-29 18:54:24,744 - INFO - ------------Environment Information------------- platform: Linux-3.10.0-1062.18.1.el7.x86_64-x86_64-with-centos-7.7.1908-Core gcc (GCC) 9.1.0 Python - 3.6.13 |Anaconda, Inc.| (default, Jun 4 2021, 14:25:59) [GCC 7.5.0] Science Toolkits: cv2 - 4.6.0 numpy - 1.19.5 numba - 0.53.1 pandas - 1.1.5 pillow - 8.4.0 skimage - 0.17.2 PaddlePaddle: paddle(gpu) - 2.3.0 paddle3d - 0.5.0 paddleseg - 2.5.0 FLAGS_cudnn_deterministic - Not set. FLAGS_cudnn_exhaustive_search - Not set. CUDA: cudnn - 7605 nvcc - Cuda compilation tools, release 10.2, V10.2.89 GPUs: GPU 0: Tesla V100-SXM2-32GB GPU 1: Tesla V100-SXM2-32GB GPU 2: Tesla V100-SXM2-32GB GPU 3: Tesla V100-SXM2-32GB ------------------------------------------------ 2022-07-29 18:54:24,749 - INFO - ---------------Config Information--------------- batch_size: 8 iters: 70000 lr_scheduler: learning_rate: 0.000125 milestones: - 36000 - 55000 type: MultiStepDecay model: backbone: pretrained: https://bj.bcebos.com/paddle3d/pretrained/imagenet/dla34.pdparams type: DLA34 depth_ref: - 28.01 - 16.32 dim_ref: - - 3.88 - 1.63 - 1.53 - - 1.78 - 1.7 - 0.58 - - 0.88 - 1.73 - 0.67 head: in_channels: 64 norm_type: gn num_chanels: 256 num_classes: 3 reg_channels: - 1 - 2 - 3 - 2 - 2 type: SMOKEPredictor max_detection: 50 pred_2d: true type: SMOKE optimizer: type: Adam train_dataset: dataset_root: datasets/KITTI mode: train transforms: - reader: pillow to_chw: false type: LoadImage - input_size: - 1280 - 384 mode: train num_classes: 3 type: Gt2SmokeTarget - mean: - 0.485 - 0.456 - 0.406 std: - 0.229 - 0.224 - 0.225 type: Normalize type: KittiMonoDataset val_dataset: dataset_root: datasets/KITTI mode: val transforms: - reader: pillow to_chw: false type: LoadImage - input_size: - 1280 - 384 mode: val num_classes: 3 type: Gt2SmokeTarget - mean: - 0.485 - 0.456 - 0.406 std: - 0.229 - 0.224 - 0.225 type: Normalize type: KittiMonoDataset ------------------------------------------------ W0729 18:54:24.752116 13400 gpu_context.cc:278] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 10.2, Runtime API Version: 10.2 W0729 18:54:24.752157 13400 gpu_context.cc:306] device: 0, cuDNN Version: 7.6. 2022-07-29 18:54:47,973 - WARNING - There is a file with the same name locally, we directly load the local file 2022-07-29 18:54:48,495 - INFO - There are 189/189 variables loaded into DLA. I0729 18:54:48.564275 13400 nccl_context.cc:83] init nccl context nranks: 4 local rank: 0 gpu id: 0 ring id: 0 /home/wuzewu/miniconda3/envs/paddle3d/lib/python3.6/site-packages/paddle/fluid/dygraph/math_op_patch.py:278: UserWarning: The dtype of left and right variables are not the same, left dtype is paddle.float32, but right dtype is paddle.int32, the right dtype will convert to paddle.float32 format(lhs_dtype, rhs_dtype, lhs_dtype)) 2022-07-29 18:55:14,741 - INFO - [TRAIN] epoch=1/604, iter=50/70000, loss=12.130205, lr=0.000125 | ETA 06:21:27 2022-07-29 18:55:32,088 - INFO - [TRAIN] epoch=1/604, iter=100/70000, loss=8.908664, lr=0.000125 | ETA 08:01:33 2022-07-29 18:55:50,650 - INFO - [TRAIN] epoch=2/604, iter=150/70000, loss=8.775135, lr=0.000125 | ETA 06:02:46 2022-07-29 18:56:09,497 - INFO - [TRAIN] epoch=2/604, iter=200/70000, loss=7.691983, lr=0.000125 | ETA 13:34:00 2022-07-29 18:56:28,023 - INFO - [TRAIN] epoch=3/604, iter=250/70000, loss=7.214788, lr=0.000125 | ETA 06:20:28 2022-07-29 18:56:45,544 - INFO - [TRAIN] epoch=3/604, iter=300/70000, loss=6.873077, lr=0.000125 | ETA 06:03:26 2022-07-29 18:57:03,714 - INFO - [TRAIN] epoch=4/604, iter=350/70000, loss=7.055847, lr=0.000125 | ETA 14:07:43 2022-07-29 18:57:20,730 - INFO - [TRAIN] epoch=4/604, iter=400/70000, loss=5.973123, lr=0.000125 | ETA 06:12:12 2022-07-29 18:57:39,027 - INFO - [TRAIN] epoch=4/604, iter=450/70000, loss=5.684747, lr=0.000125 | ETA 06:11:28 2022-07-29 18:57:58,847 - INFO - [TRAIN] epoch=5/604, iter=500/70000, loss=5.627173, lr=0.000125 | ETA 06:35:25 2022-07-29 18:58:16,170 - INFO - [TRAIN] epoch=5/604, iter=550/70000, loss=5.482371, lr=0.000125 | ETA 06:16:46 2022-07-29 18:58:36,092 - INFO - [TRAIN] epoch=6/604, iter=600/70000, loss=5.067576, lr=0.000125 | ETA 08:00:32 2022-07-29 18:58:53,951 - INFO - [TRAIN] epoch=6/604, iter=650/70000, loss=5.652784, lr=0.000125 | ETA 06:09:59 2022-07-29 18:59:12,551 - INFO - [TRAIN] epoch=7/604, iter=700/70000, loss=5.295916, lr=0.000125 | ETA 07:57:21 2022-07-29 18:59:29,953 - INFO - [TRAIN] epoch=7/604, iter=750/70000, loss=5.223449, lr=0.000125 | ETA 06:22:38 2022-07-29 18:59:47,265 - INFO - [TRAIN] epoch=7/604, iter=800/70000, loss=5.408122, lr=0.000125 | ETA 06:14:54 2022-07-29 19:00:05,959 - INFO - [TRAIN] epoch=8/604, iter=850/70000, loss=5.017084, lr=0.000125 | ETA 06:09:13 2022-07-29 19:00:23,335 - INFO - [TRAIN] epoch=8/604, iter=900/70000, loss=4.584700, lr=0.000125 | ETA 06:01:03 2022-07-29 19:00:42,971 - INFO - [TRAIN] epoch=9/604, iter=950/70000, loss=5.074500, lr=0.000125 | ETA 07:08:08 2022-07-29 19:01:00,986 - INFO - [TRAIN] epoch=9/604, iter=1000/70000, loss=4.643615, lr=0.000125 | ETA 05:58:24 2022-07-29 19:01:19,725 - INFO - [TRAIN] epoch=10/604, iter=1050/70000, loss=4.977087, lr=0.000125 | ETA 06:39:15 2022-07-29 19:01:36,598 - INFO - [TRAIN] epoch=10/604, iter=1100/70000, loss=4.445996, lr=0.000125 | ETA 06:10:41 2022-07-29 19:01:54,154 - INFO - [TRAIN] epoch=10/604, iter=1150/70000, loss=4.638632, lr=0.000125 | ETA 06:41:06 2022-07-29 19:02:13,532 - INFO - [TRAIN] epoch=11/604, iter=1200/70000, loss=4.608636, lr=0.000125 | ETA 06:06:50 2022-07-29 19:02:30,903 - INFO - [TRAIN] epoch=11/604, iter=1250/70000, loss=4.438330, lr=0.000125 | ETA 06:08:01 2022-07-29 19:02:49,310 - INFO - [TRAIN] epoch=12/604, iter=1300/70000, loss=4.345051, lr=0.000125 | ETA 06:13:40 2022-07-29 19:03:07,368 - INFO - [TRAIN] epoch=12/604, iter=1350/70000, loss=4.431686, lr=0.000125 | ETA 06:18:58 2022-07-29 19:03:26,475 - INFO - [TRAIN] epoch=13/604, iter=1400/70000, loss=4.323106, lr=0.000125 | ETA 06:10:23 2022-07-29 19:03:44,578 - INFO - [TRAIN] epoch=13/604, iter=1450/70000, loss=4.349026, lr=0.000125 | ETA 05:56:47 2022-07-29 19:04:01,855 - INFO - [TRAIN] epoch=13/604, iter=1500/70000, loss=4.275749, lr=0.000125 | ETA 06:36:42 2022-07-29 19:04:20,998 - INFO - [TRAIN] epoch=14/604, iter=1550/70000, loss=3.898527, lr=0.000125 | ETA 08:09:36 2022-07-29 19:04:38,068 - INFO - [TRAIN] epoch=14/604, iter=1600/70000, loss=3.849521, lr=0.000125 | ETA 06:08:37 2022-07-29 19:04:57,803 - INFO - [TRAIN] epoch=15/604, iter=1650/70000, loss=3.865040, lr=0.000125 | ETA 07:23:31 2022-07-29 19:05:14,379 - INFO - [TRAIN] epoch=15/604, iter=1700/70000, loss=3.546768, lr=0.000125 | ETA 05:58:05 2022-07-29 19:05:33,343 - INFO - [TRAIN] epoch=16/604, iter=1750/70000, loss=3.503688, lr=0.000125 | ETA 09:28:29 2022-07-29 19:05:51,169 - INFO - [TRAIN] epoch=16/604, iter=1800/70000, loss=3.693413, lr=0.000125 | ETA 05:56:00 2022-07-29 19:06:09,151 - INFO - [TRAIN] epoch=16/604, iter=1850/70000, loss=3.556298, lr=0.000125 | ETA 07:05:57 2022-07-29 19:06:27,944 - INFO - [TRAIN] epoch=17/604, iter=1900/70000, loss=3.405636, lr=0.000125 | ETA 06:00:34 2022-07-29 19:06:45,974 - INFO - [TRAIN] epoch=17/604, iter=1950/70000, loss=3.694042, lr=0.000125 | ETA 08:10:08 2022-07-29 19:07:04,755 - INFO - [TRAIN] epoch=18/604, iter=2000/70000, loss=3.203901, lr=0.000125 | ETA 05:59:18 2022-07-29 19:07:21,907 - INFO - [TRAIN] epoch=18/604, iter=2050/70000, loss=3.267063, lr=0.000125 | ETA 06:00:01 2022-07-29 19:07:41,117 - INFO - [TRAIN] epoch=19/604, iter=2100/70000, loss=2.853395, lr=0.000125 | ETA 06:04:11 2022-07-29 19:07:59,702 - INFO - [TRAIN] epoch=19/604, iter=2150/70000, loss=3.067313, lr=0.000125 | ETA 05:59:50 2022-07-29 19:08:17,603 - INFO - [TRAIN] epoch=19/604, iter=2200/70000, loss=3.225055, lr=0.000125 | ETA 06:19:45 2022-07-29 19:08:37,491 - INFO - [TRAIN] epoch=20/604, iter=2250/70000, loss=2.956875, lr=0.000125 | ETA 06:00:12 2022-07-29 19:08:55,868 - INFO - [TRAIN] epoch=20/604, iter=2300/70000, loss=3.198838, lr=0.000125 | ETA 09:23:33 2022-07-29 19:09:15,381 - INFO - [TRAIN] epoch=21/604, iter=2350/70000, loss=3.334005, lr=0.000125 | ETA 05:55:05 2022-07-29 19:09:32,090 - INFO - [TRAIN] epoch=21/604, iter=2400/70000, loss=2.772740, lr=0.000125 | ETA 06:02:31 2022-07-29 19:09:52,027 - INFO - [TRAIN] epoch=22/604, iter=2450/70000, loss=2.769393, lr=0.000125 | ETA 06:04:25 2022-07-29 19:10:09,959 - INFO - [TRAIN] epoch=22/604, iter=2500/70000, loss=2.920102, lr=0.000125 | ETA 05:54:43 2022-07-29 19:10:26,686 - INFO - [TRAIN] epoch=22/604, iter=2550/70000, loss=2.656928, lr=0.000125 | ETA 05:49:12 2022-07-29 19:10:44,998 - INFO - [TRAIN] epoch=23/604, iter=2600/70000, loss=2.984165, lr=0.000125 | ETA 06:13:42 2022-07-29 19:11:03,014 - INFO - [TRAIN] epoch=23/604, iter=2650/70000, loss=2.654561, lr=0.000125 | ETA 06:07:40 2022-07-29 19:11:22,217 - INFO - [TRAIN] epoch=24/604, iter=2700/70000, loss=2.778376, lr=0.000125 | ETA 05:51:36 2022-07-29 19:11:39,140 - INFO - [TRAIN] epoch=24/604, iter=2750/70000, loss=2.720380, lr=0.000125 | ETA 05:57:02 2022-07-29 19:11:58,191 - INFO - [TRAIN] epoch=25/604, iter=2800/70000, loss=2.540605, lr=0.000125 | ETA 13:02:12 2022-07-29 19:12:15,955 - INFO - [TRAIN] epoch=25/604, iter=2850/70000, loss=2.560147, lr=0.000125 | ETA 05:58:11 2022-07-29 19:12:33,063 - INFO - [TRAIN] epoch=25/604, iter=2900/70000, loss=2.628005, lr=0.000125 | ETA 05:49:50 2022-07-29 19:12:52,770 - INFO - [TRAIN] epoch=26/604, iter=2950/70000, loss=2.541313, lr=0.000125 | ETA 06:06:51 2022-07-29 19:13:10,645 - INFO - [TRAIN] epoch=26/604, iter=3000/70000, loss=2.650129, lr=0.000125 | ETA 08:20:48 2022-07-29 19:13:29,096 - INFO - [TRAIN] epoch=27/604, iter=3050/70000, loss=2.508905, lr=0.000125 | ETA 05:49:13 2022-07-29 19:13:47,347 - INFO - [TRAIN] epoch=27/604, iter=3100/70000, loss=2.438685, lr=0.000125 | ETA 06:06:17 2022-07-29 19:14:07,259 - INFO - [TRAIN] epoch=28/604, iter=3150/70000, loss=2.520891, lr=0.000125 | ETA 11:15:33 2022-07-29 19:14:25,592 - INFO - [TRAIN] epoch=28/604, iter=3200/70000, loss=2.569669, lr=0.000125 | ETA 05:58:16 2022-07-29 19:14:44,353 - INFO - [TRAIN] epoch=29/604, iter=3250/70000, loss=2.236212, lr=0.000125 | ETA 13:46:54 2022-07-29 19:15:02,446 - INFO - [TRAIN] epoch=29/604, iter=3300/70000, loss=2.301049, lr=0.000125 | ETA 05:56:46 2022-07-29 19:15:19,607 - INFO - [TRAIN] epoch=29/604, iter=3350/70000, loss=2.104672, lr=0.000125 | ETA 06:04:44 2022-07-29 19:15:39,120 - INFO - [TRAIN] epoch=30/604, iter=3400/70000, loss=2.300979, lr=0.000125 | ETA 05:57:10 2022-07-29 19:15:57,025 - INFO - [TRAIN] epoch=30/604, iter=3450/70000, loss=2.269632, lr=0.000125 | ETA 05:55:31 2022-07-29 19:16:15,731 - INFO - [TRAIN] epoch=31/604, iter=3500/70000, loss=2.262026, lr=0.000125 | ETA 05:47:23 2022-07-29 19:16:33,376 - INFO - [TRAIN] epoch=31/604, iter=3550/70000, loss=2.370275, lr=0.000125 | ETA 06:00:25 2022-07-29 19:16:52,586 - INFO - [TRAIN] epoch=32/604, iter=3600/70000, loss=2.566372, lr=0.000125 | ETA 07:47:02 2022-07-29 19:17:09,688 - INFO - [TRAIN] epoch=32/604, iter=3650/70000, loss=2.199873, lr=0.000125 | ETA 05:51:35 2022-07-29 19:17:26,516 - INFO - [TRAIN] epoch=32/604, iter=3700/70000, loss=2.423089, lr=0.000125 | ETA 05:47:34 2022-07-29 19:17:44,812 - INFO - [TRAIN] epoch=33/604, iter=3750/70000, loss=2.261867, lr=0.000125 | ETA 05:49:14 2022-07-29 19:18:02,064 - INFO - [TRAIN] epoch=33/604, iter=3800/70000, loss=2.262235, lr=0.000125 | ETA 05:47:16 2022-07-29 19:18:21,167 - INFO - [TRAIN] epoch=34/604, iter=3850/70000, loss=2.188092, lr=0.000125 | ETA 06:12:32 2022-07-29 19:18:38,926 - INFO - [TRAIN] epoch=34/604, iter=3900/70000, loss=2.241339, lr=0.000125 | ETA 05:43:58 2022-07-29 19:18:58,326 - INFO - [TRAIN] epoch=35/604, iter=3950/70000, loss=2.330655, lr=0.000125 | ETA 06:22:12 2022-07-29 19:19:15,656 - INFO - [TRAIN] epoch=35/604, iter=4000/70000, loss=2.088782, lr=0.000125 | ETA 05:57:35 2022-07-29 19:19:32,941 - INFO - [TRAIN] epoch=35/604, iter=4050/70000, loss=2.093826, lr=0.000125 | ETA 10:42:29 2022-07-29 19:19:52,398 - INFO - [TRAIN] epoch=36/604, iter=4100/70000, loss=2.282511, lr=0.000125 | ETA 05:57:39 2022-07-29 19:20:10,606 - INFO - [TRAIN] epoch=36/604, iter=4150/70000, loss=2.077838, lr=0.000125 | ETA 08:25:38 2022-07-29 19:20:29,016 - INFO - [TRAIN] epoch=37/604, iter=4200/70000, loss=2.144048, lr=0.000125 | ETA 06:06:19 2022-07-29 19:20:48,099 - INFO - [TRAIN] epoch=37/604, iter=4250/70000, loss=2.317974, lr=0.000125 | ETA 07:21:54 2022-07-29 19:21:06,325 - INFO - [TRAIN] epoch=38/604, iter=4300/70000, loss=2.331056, lr=0.000125 | ETA 06:17:45 2022-07-29 19:21:24,787 - INFO - [TRAIN] epoch=38/604, iter=4350/70000, loss=2.089850, lr=0.000125 | ETA 07:39:32 2022-07-29 19:21:42,103 - INFO - [TRAIN] epoch=38/604, iter=4400/70000, loss=2.272411, lr=0.000125 | ETA 05:51:03 2022-07-29 19:22:00,388 - INFO - [TRAIN] epoch=39/604, iter=4450/70000, loss=2.037518, lr=0.000125 | ETA 06:08:35 2022-07-29 19:22:16,560 - INFO - [TRAIN] epoch=39/604, iter=4500/70000, loss=2.082662, lr=0.000125 | ETA 05:40:48 2022-07-29 19:22:35,618 - INFO - [TRAIN] epoch=40/604, iter=4550/70000, loss=2.050129, lr=0.000125 | ETA 05:53:06 2022-07-29 19:22:52,850 - INFO - [TRAIN] epoch=40/604, iter=4600/70000, loss=1.961292, lr=0.000125 | ETA 09:00:51 2022-07-29 19:23:11,966 - INFO - [TRAIN] epoch=41/604, iter=4650/70000, loss=2.242675, lr=0.000125 | ETA 05:49:40 2022-07-29 19:23:29,177 - INFO - [TRAIN] epoch=41/604, iter=4700/70000, loss=1.781408, lr=0.000125 | ETA 06:41:11 2022-07-29 19:23:46,760 - INFO - [TRAIN] epoch=41/604, iter=4750/70000, loss=1.967303, lr=0.000125 | ETA 05:56:26 2022-07-29 19:24:05,556 - INFO - [TRAIN] epoch=42/604, iter=4800/70000, loss=1.810064, lr=0.000125 | ETA 07:29:23 2022-07-29 19:24:22,823 - INFO - [TRAIN] epoch=42/604, iter=4850/70000, loss=2.106923, lr=0.000125 | ETA 05:44:22 2022-07-29 19:24:41,671 - INFO - [TRAIN] epoch=43/604, iter=4900/70000, loss=2.057648, lr=0.000125 | ETA 06:22:41 2022-07-29 19:24:59,719 - INFO - [TRAIN] epoch=43/604, iter=4950/70000, loss=1.835628, lr=0.000125 | ETA 06:03:45 2022-07-29 19:25:18,451 - INFO - [TRAIN] epoch=44/604, iter=5000/70000, loss=2.016412, lr=0.000125 | ETA 06:15:26 2022-07-29 19:25:19,631 - INFO - Push model to checkpoint ./output/iter_5000 2022-07-29 19:25:36,496 - INFO - [TRAIN] epoch=44/604, iter=5050/70000, loss=2.101997, lr=0.000125 | ETA 06:05:10 2022-07-29 19:25:53,143 - INFO - [TRAIN] epoch=44/604, iter=5100/70000, loss=1.879987, lr=0.000125 | ETA 05:47:28 2022-07-29 19:26:12,977 - INFO - [TRAIN] epoch=45/604, iter=5150/70000, loss=1.866087, lr=0.000125 | ETA 05:44:49 2022-07-29 19:26:30,552 - INFO - [TRAIN] epoch=45/604, iter=5200/70000, loss=1.930963, lr=0.000125 | ETA 05:55:42 2022-07-29 19:26:49,694 - INFO - [TRAIN] epoch=46/604, iter=5250/70000, loss=1.818094, lr=0.000125 | ETA 08:30:13 2022-07-29 19:27:07,170 - INFO - [TRAIN] epoch=46/604, iter=5300/70000, loss=1.846057, lr=0.000125 | ETA 05:55:32 2022-07-29 19:27:26,359 - INFO - [TRAIN] epoch=47/604, iter=5350/70000, loss=1.897522, lr=0.000125 | ETA 09:53:36 2022-07-29 19:27:43,552 - INFO - [TRAIN] epoch=47/604, iter=5400/70000, loss=1.850604, lr=0.000125 | ETA 05:40:20 2022-07-29 19:28:00,962 - INFO - [TRAIN] epoch=47/604, iter=5450/70000, loss=1.832733, lr=0.000125 | ETA 05:40:05 2022-07-29 19:28:20,223 - INFO - [TRAIN] epoch=48/604, iter=5500/70000, loss=1.644368, lr=0.000125 | ETA 05:38:23 2022-07-29 19:28:36,485 - INFO - [TRAIN] epoch=48/604, iter=5550/70000, loss=2.054608, lr=0.000125 | ETA 05:48:33 2022-07-29 19:28:55,539 - INFO - [TRAIN] epoch=49/604, iter=5600/70000, loss=1.802275, lr=0.000125 | ETA 06:24:36 2022-07-29 19:29:12,985 - INFO - [TRAIN] epoch=49/604, iter=5650/70000, loss=1.742356, lr=0.000125 | ETA 05:36:20 2022-07-29 19:29:31,382 - INFO - [TRAIN] epoch=50/604, iter=5700/70000, loss=1.791363, lr=0.000125 | ETA 06:35:29 2022-07-29 19:29:50,011 - INFO - [TRAIN] epoch=50/604, iter=5750/70000, loss=1.873395, lr=0.000125 | ETA 05:44:37 2022-07-29 19:30:06,336 - INFO - [TRAIN] epoch=50/604, iter=5800/70000, loss=1.663178, lr=0.000125 | ETA 05:43:58 2022-07-29 19:30:26,525 - INFO - [TRAIN] epoch=51/604, iter=5850/70000, loss=1.859119, lr=0.000125 | ETA 05:38:14 2022-07-29 19:30:44,105 - INFO - [TRAIN] epoch=51/604, iter=5900/70000, loss=1.855509, lr=0.000125 | ETA 06:58:50 2022-07-29 19:31:03,068 - INFO - [TRAIN] epoch=52/604, iter=5950/70000, loss=1.738470, lr=0.000125 | ETA 05:53:38 2022-07-29 19:31:21,101 - INFO - [TRAIN] epoch=52/604, iter=6000/70000, loss=1.739128, lr=0.000125 | ETA 05:34:11 2022-07-29 19:31:41,069 - INFO - [TRAIN] epoch=53/604, iter=6050/70000, loss=1.701924, lr=0.000125 | ETA 11:45:56 2022-07-29 19:31:58,140 - INFO - [TRAIN] epoch=53/604, iter=6100/70000, loss=1.773649, lr=0.000125 | ETA 05:46:23 2022-07-29 19:32:17,186 - INFO - [TRAIN] epoch=54/604, iter=6150/70000, loss=1.712424, lr=0.000125 | ETA 12:30:02 2022-07-29 19:32:34,620 - INFO - [TRAIN] epoch=54/604, iter=6200/70000, loss=1.679812, lr=0.000125 | ETA 05:35:09 2022-07-29 19:32:52,150 - INFO - [TRAIN] epoch=54/604, iter=6250/70000, loss=1.585357, lr=0.000125 | ETA 05:39:34 2022-07-29 19:33:12,157 - INFO - [TRAIN] epoch=55/604, iter=6300/70000, loss=1.646304, lr=0.000125 | ETA 05:34:32 2022-07-29 19:33:29,876 - INFO - [TRAIN] epoch=55/604, iter=6350/70000, loss=1.677436, lr=0.000125 | ETA 05:56:30 2022-07-29 19:33:49,464 - INFO - [TRAIN] epoch=56/604, iter=6400/70000, loss=1.480289, lr=0.000125 | ETA 06:03:52 2022-07-29 19:34:06,633 - INFO - [TRAIN] epoch=56/604, iter=6450/70000, loss=1.702550, lr=0.000125 | ETA 05:35:45 2022-07-29 19:34:25,148 - INFO - [TRAIN] epoch=57/604, iter=6500/70000, loss=1.805968, lr=0.000125 | ETA 07:32:21 2022-07-29 19:34:42,942 - INFO - [TRAIN] epoch=57/604, iter=6550/70000, loss=1.643995, lr=0.000125 | ETA 05:40:10 2022-07-29 19:35:00,661 - INFO - [TRAIN] epoch=57/604, iter=6600/70000, loss=1.676724, lr=0.000125 | ETA 05:44:25 2022-07-29 19:35:20,604 - INFO - [TRAIN] epoch=58/604, iter=6650/70000, loss=1.614583, lr=0.000125 | ETA 05:39:22 2022-07-29 19:35:37,569 - INFO - [TRAIN] epoch=58/604, iter=6700/70000, loss=1.586419, lr=0.000125 | ETA 05:37:41 2022-07-29 19:35:55,691 - INFO - [TRAIN] epoch=59/604, iter=6750/70000, loss=1.533400, lr=0.000125 | ETA 05:31:12 2022-07-29 19:36:13,857 - INFO - [TRAIN] epoch=59/604, iter=6800/70000, loss=1.697540, lr=0.000125 | ETA 10:08:40 2022-07-29 19:36:32,208 - INFO - [TRAIN] epoch=60/604, iter=6850/70000, loss=1.551163, lr=0.000125 | ETA 06:34:00 2022-07-29 19:36:51,115 - INFO - [TRAIN] epoch=60/604, iter=6900/70000, loss=1.515068, lr=0.000125 | ETA 06:03:35 2022-07-29 19:37:08,195 - INFO - [TRAIN] epoch=60/604, iter=6950/70000, loss=1.477339, lr=0.000125 | ETA 05:28:19 2022-07-29 19:37:26,666 - INFO - [TRAIN] epoch=61/604, iter=7000/70000, loss=1.708248, lr=0.000125 | ETA 05:29:37 2022-07-29 19:37:44,152 - INFO - [TRAIN] epoch=61/604, iter=7050/70000, loss=1.637165, lr=0.000125 | ETA 05:26:57 2022-07-29 19:38:03,627 - INFO - [TRAIN] epoch=62/604, iter=7100/70000, loss=1.643182, lr=0.000125 | ETA 09:15:25 2022-07-29 19:38:20,206 - INFO - [TRAIN] epoch=62/604, iter=7150/70000, loss=1.586201, lr=0.000125 | ETA 05:29:53 2022-07-29 19:38:38,029 - INFO - [TRAIN] epoch=63/604, iter=7200/70000, loss=1.620600, lr=0.000125 | ETA 05:51:43 2022-07-29 19:38:55,658 - INFO - [TRAIN] epoch=63/604, iter=7250/70000, loss=1.681756, lr=0.000125 | ETA 06:56:05 2022-07-29 19:39:12,881 - INFO - [TRAIN] epoch=63/604, iter=7300/70000, loss=1.680524, lr=0.000125 | ETA 05:25:48 2022-07-29 19:39:32,460 - INFO - [TRAIN] epoch=64/604, iter=7350/70000, loss=1.513169, lr=0.000125 | ETA 05:52:10 2022-07-29 19:39:49,593 - INFO - [TRAIN] epoch=64/604, iter=7400/70000, loss=1.552893, lr=0.000125 | ETA 05:26:49 2022-07-29 19:40:09,299 - INFO - [TRAIN] epoch=65/604, iter=7450/70000, loss=1.485591, lr=0.000125 | ETA 05:36:42 2022-07-29 19:40:27,195 - INFO - [TRAIN] epoch=65/604, iter=7500/70000, loss=1.454846, lr=0.000125 | ETA 05:32:36 2022-07-29 19:40:46,677 - INFO - [TRAIN] epoch=66/604, iter=7550/70000, loss=1.580914, lr=0.000125 | ETA 05:32:16 2022-07-29 19:41:03,996 - INFO - [TRAIN] epoch=66/604, iter=7600/70000, loss=1.438081, lr=0.000125 | ETA 05:28:42 2022-07-29 19:41:21,179 - INFO - [TRAIN] epoch=66/604, iter=7650/70000, loss=1.544823, lr=0.000125 | ETA 05:38:14 2022-07-29 19:41:40,219 - INFO - [TRAIN] epoch=67/604, iter=7700/70000, loss=1.470202, lr=0.000125 | ETA 05:33:57 2022-07-29 19:41:57,669 - INFO - [TRAIN] epoch=67/604, iter=7750/70000, loss=1.549081, lr=0.000125 | ETA 05:27:38 2022-07-29 19:42:16,923 - INFO - [TRAIN] epoch=68/604, iter=7800/70000, loss=1.422284, lr=0.000125 | ETA 05:57:28 2022-07-29 19:42:34,497 - INFO - [TRAIN] epoch=68/604, iter=7850/70000, loss=1.514155, lr=0.000125 | ETA 07:07:57 2022-07-29 19:42:54,380 - INFO - [TRAIN] epoch=69/604, iter=7900/70000, loss=1.455909, lr=0.000125 | ETA 05:32:32 2022-07-29 19:43:12,150 - INFO - [TRAIN] epoch=69/604, iter=7950/70000, loss=1.454929, lr=0.000125 | ETA 05:43:18 2022-07-29 19:43:29,869 - INFO - [TRAIN] epoch=69/604, iter=8000/70000, loss=1.406434, lr=0.000125 | ETA 05:22:16 2022-07-29 19:43:49,589 - INFO - [TRAIN] epoch=70/604, iter=8050/70000, loss=1.335294, lr=0.000125 | ETA 05:37:10 2022-07-29 19:44:06,851 - INFO - [TRAIN] epoch=70/604, iter=8100/70000, loss=1.479784, lr=0.000125 | ETA 05:25:02 2022-07-29 19:44:25,375 - INFO - [TRAIN] epoch=71/604, iter=8150/70000, loss=1.437849, lr=0.000125 | ETA 05:34:19 2022-07-29 19:44:43,230 - INFO - [TRAIN] epoch=71/604, iter=8200/70000, loss=1.513938, lr=0.000125 | ETA 08:08:14 2022-07-29 19:45:01,714 - INFO - [TRAIN] epoch=72/604, iter=8250/70000, loss=1.382511, lr=0.000125 | ETA 06:20:13 2022-07-29 19:45:19,821 - INFO - [TRAIN] epoch=72/604, iter=8300/70000, loss=1.476405, lr=0.000125 | ETA 05:40:47 2022-07-29 19:45:37,246 - INFO - [TRAIN] epoch=72/604, iter=8350/70000, loss=1.379877, lr=0.000125 | ETA 05:23:25 2022-07-29 19:45:56,974 - INFO - [TRAIN] epoch=73/604, iter=8400/70000, loss=1.420356, lr=0.000125 | ETA 05:32:13 2022-07-29 19:46:15,203 - INFO - [TRAIN] epoch=73/604, iter=8450/70000, loss=1.383840, lr=0.000125 | ETA 05:48:05 2022-07-29 19:46:34,404 - INFO - [TRAIN] epoch=74/604, iter=8500/70000, loss=1.377455, lr=0.000125 | ETA 05:43:28 2022-07-29 19:46:51,659 - INFO - [TRAIN] epoch=74/604, iter=8550/70000, loss=1.470865, lr=0.000125 | ETA 05:43:50 2022-07-29 19:47:09,562 - INFO - [TRAIN] epoch=75/604, iter=8600/70000, loss=1.425120, lr=0.000125 | ETA 05:28:02 2022-07-29 19:47:25,485 - INFO - [TRAIN] epoch=75/604, iter=8650/70000, loss=1.420878, lr=0.000125 | ETA 05:29:02 2022-07-29 19:47:41,572 - INFO - [TRAIN] epoch=75/604, iter=8700/70000, loss=1.351847, lr=0.000125 | ETA 05:29:19 2022-07-29 19:47:59,153 - INFO - [TRAIN] epoch=76/604, iter=8750/70000, loss=1.477936, lr=0.000125 | ETA 05:23:51 2022-07-29 19:48:15,199 - INFO - [TRAIN] epoch=76/604, iter=8800/70000, loss=1.403371, lr=0.000125 | ETA 05:23:06 2022-07-29 19:48:32,805 - INFO - [TRAIN] epoch=77/604, iter=8850/70000, loss=1.388853, lr=0.000125 | ETA 05:21:14 2022-07-29 19:48:48,941 - INFO - [TRAIN] epoch=77/604, iter=8900/70000, loss=1.416793, lr=0.000125 | ETA 05:26:40 2022-07-29 19:49:06,430 - INFO - [TRAIN] epoch=78/604, iter=8950/70000, loss=1.378012, lr=0.000125 | ETA 05:22:13 2022-07-29 19:49:22,400 - INFO - [TRAIN] epoch=78/604, iter=9000/70000, loss=1.314865, lr=0.000125 | ETA 05:27:01 2022-07-29 19:49:39,831 - INFO - [TRAIN] epoch=79/604, iter=9050/70000, loss=1.353521, lr=0.000125 | ETA 11:48:10 2022-07-29 19:49:55,797 - INFO - [TRAIN] epoch=79/604, iter=9100/70000, loss=1.373853, lr=0.000125 | ETA 05:16:46 2022-07-29 19:50:11,710 - INFO - [TRAIN] epoch=79/604, iter=9150/70000, loss=1.311927, lr=0.000125 | ETA 05:22:09 2022-07-29 19:50:29,394 - INFO - [TRAIN] epoch=80/604, iter=9200/70000, loss=1.347051, lr=0.000125 | ETA 05:37:14 2022-07-29 19:50:45,318 - INFO - [TRAIN] epoch=80/604, iter=9250/70000, loss=1.367125, lr=0.000125 | ETA 05:18:29 2022-07-29 19:51:02,914 - INFO - [TRAIN] epoch=81/604, iter=9300/70000, loss=1.611084, lr=0.000125 | ETA 05:24:39 2022-07-29 19:51:18,979 - INFO - [TRAIN] epoch=81/604, iter=9350/70000, loss=1.406483, lr=0.000125 | ETA 05:17:46 2022-07-29 19:51:37,030 - INFO - [TRAIN] epoch=82/604, iter=9400/70000, loss=1.179050, lr=0.000125 | ETA 07:04:34 2022-07-29 19:51:54,317 - INFO - [TRAIN] epoch=82/604, iter=9450/70000, loss=1.284567, lr=0.000125 | ETA 08:16:46 2022-07-29 19:52:11,871 - INFO - [TRAIN] epoch=82/604, iter=9500/70000, loss=1.426657, lr=0.000125 | ETA 05:22:19 2022-07-29 19:52:30,241 - INFO - [TRAIN] epoch=83/604, iter=9550/70000, loss=1.312981, lr=0.000125 | ETA 05:46:51 2022-07-29 19:52:47,419 - INFO - [TRAIN] epoch=83/604, iter=9600/70000, loss=1.357746, lr=0.000125 | ETA 05:15:53 2022-07-29 19:53:06,678 - INFO - [TRAIN] epoch=84/604, iter=9650/70000, loss=1.234739, lr=0.000125 | ETA 06:10:54 2022-07-29 19:53:23,764 - INFO - [TRAIN] epoch=84/604, iter=9700/70000, loss=1.214578, lr=0.000125 | ETA 05:17:40 2022-07-29 19:53:42,397 - INFO - [TRAIN] epoch=85/604, iter=9750/70000, loss=1.298652, lr=0.000125 | ETA 05:50:52 2022-07-29 19:53:59,322 - INFO - [TRAIN] epoch=85/604, iter=9800/70000, loss=1.246286, lr=0.000125 | ETA 05:32:24 2022-07-29 19:54:16,728 - INFO - [TRAIN] epoch=85/604, iter=9850/70000, loss=1.313460, lr=0.000125 | ETA 05:19:55 2022-07-29 19:54:34,881 - INFO - [TRAIN] epoch=86/604, iter=9900/70000, loss=1.260060, lr=0.000125 | ETA 05:18:21 2022-07-29 19:54:52,206 - INFO - [TRAIN] epoch=86/604, iter=9950/70000, loss=1.273915, lr=0.000125 | ETA 05:21:57 2022-07-29 19:55:10,792 - INFO - [TRAIN] epoch=87/604, iter=10000/70000, loss=1.250475, lr=0.000125 | ETA 05:27:04 2022-07-29 19:55:11,975 - INFO - Push model to checkpoint ./output/iter_10000 2022-07-29 19:55:28,690 - INFO - [TRAIN] epoch=87/604, iter=10050/70000, loss=1.176920, lr=0.000125 | ETA 05:19:06 2022-07-29 19:55:46,886 - INFO - [TRAIN] epoch=88/604, iter=10100/70000, loss=1.209995, lr=0.000125 | ETA 05:58:30 2022-07-29 19:56:04,393 - INFO - [TRAIN] epoch=88/604, iter=10150/70000, loss=1.208888, lr=0.000125 | ETA 07:22:35 2022-07-29 19:56:21,422 - INFO - [TRAIN] epoch=88/604, iter=10200/70000, loss=1.191250, lr=0.000125 | ETA 05:23:37 2022-07-29 19:56:40,388 - INFO - [TRAIN] epoch=89/604, iter=10250/70000, loss=1.172823, lr=0.000125 | ETA 05:51:37 2022-07-29 19:56:57,321 - INFO - [TRAIN] epoch=89/604, iter=10300/70000, loss=1.233271, lr=0.000125 | ETA 05:15:39 2022-07-29 19:57:16,065 - INFO - [TRAIN] epoch=90/604, iter=10350/70000, loss=1.285634, lr=0.000125 | ETA 06:23:35 2022-07-29 19:57:33,264 - INFO - [TRAIN] epoch=90/604, iter=10400/70000, loss=1.143099, lr=0.000125 | ETA 05:30:43 2022-07-29 19:57:51,893 - INFO - [TRAIN] epoch=91/604, iter=10450/70000, loss=1.274569, lr=0.000125 | ETA 05:16:10 2022-07-29 19:58:09,037 - INFO - [TRAIN] epoch=91/604, iter=10500/70000, loss=1.198897, lr=0.000125 | ETA 05:32:24 2022-07-29 19:58:27,653 - INFO - [TRAIN] epoch=91/604, iter=10550/70000, loss=1.207938, lr=0.000125 | ETA 06:25:50 2022-07-29 19:58:46,200 - INFO - [TRAIN] epoch=92/604, iter=10600/70000, loss=1.232341, lr=0.000125 | ETA 05:32:00 2022-07-29 19:59:03,313 - INFO - [TRAIN] epoch=92/604, iter=10650/70000, loss=1.143459, lr=0.000125 | ETA 05:11:07 2022-07-29 19:59:21,900 - INFO - [TRAIN] epoch=93/604, iter=10700/70000, loss=1.346149, lr=0.000125 | ETA 05:33:14 2022-07-29 19:59:38,057 - INFO - [TRAIN] epoch=93/604, iter=10750/70000, loss=1.210821, lr=0.000125 | ETA 06:30:30 2022-07-29 19:59:58,299 - INFO - [TRAIN] epoch=94/604, iter=10800/70000, loss=1.236875, lr=0.000125 | ETA 09:05:14 2022-07-29 20:00:16,456 - INFO - [TRAIN] epoch=94/604, iter=10850/70000, loss=1.150189, lr=0.000125 | ETA 05:11:04 2022-07-29 20:00:33,315 - INFO - [TRAIN] epoch=94/604, iter=10900/70000, loss=1.191390, lr=0.000125 | ETA 05:08:16 2022-07-29 20:00:52,336 - INFO - [TRAIN] epoch=95/604, iter=10950/70000, loss=1.100470, lr=0.000125 | ETA 05:09:09 2022-07-29 20:01:09,614 - INFO - [TRAIN] epoch=95/604, iter=11000/70000, loss=1.246401, lr=0.000125 | ETA 05:22:05 2022-07-29 20:01:27,944 - INFO - [TRAIN] epoch=96/604, iter=11050/70000, loss=1.104624, lr=0.000125 | ETA 05:13:13 2022-07-29 20:01:46,096 - INFO - [TRAIN] epoch=96/604, iter=11100/70000, loss=1.151698, lr=0.000125 | ETA 05:48:59 2022-07-29 20:02:05,026 - INFO - [TRAIN] epoch=97/604, iter=11150/70000, loss=1.163100, lr=0.000125 | ETA 05:12:22 2022-07-29 20:02:22,461 - INFO - [TRAIN] epoch=97/604, iter=11200/70000, loss=1.122288, lr=0.000125 | ETA 05:08:15 2022-07-29 20:02:39,568 - INFO - [TRAIN] epoch=97/604, iter=11250/70000, loss=1.085306, lr=0.000125 | ETA 05:50:38 2022-07-29 20:02:58,281 - INFO - [TRAIN] epoch=98/604, iter=11300/70000, loss=1.193102, lr=0.000125 | ETA 05:19:34 2022-07-29 20:03:16,197 - INFO - [TRAIN] epoch=98/604, iter=11350/70000, loss=1.085378, lr=0.000125 | ETA 06:58:29 2022-07-29 20:03:35,089 - INFO - [TRAIN] epoch=99/604, iter=11400/70000, loss=1.058115, lr=0.000125 | ETA 05:05:13 2022-07-29 20:03:52,449 - INFO - [TRAIN] epoch=99/604, iter=11450/70000, loss=1.030073, lr=0.000125 | ETA 05:14:39 2022-07-29 20:04:10,942 - INFO - [TRAIN] epoch=100/604, iter=11500/70000, loss=1.220027, lr=0.000125 | ETA 06:07:17 2022-07-29 20:04:28,230 - INFO - [TRAIN] epoch=100/604, iter=11550/70000, loss=1.144444, lr=0.000125 | ETA 05:10:22 2022-07-29 20:04:45,740 - INFO - [TRAIN] epoch=100/604, iter=11600/70000, loss=1.077955, lr=0.000125 | ETA 05:03:55 2022-07-29 20:05:04,985 - INFO - [TRAIN] epoch=101/604, iter=11650/70000, loss=1.114217, lr=0.000125 | ETA 05:05:45 2022-07-29 20:05:22,343 - INFO - [TRAIN] epoch=101/604, iter=11700/70000, loss=1.078032, lr=0.000125 | ETA 05:06:37 2022-07-29 20:05:41,369 - INFO - [TRAIN] epoch=102/604, iter=11750/70000, loss=1.009618, lr=0.000125 | ETA 05:37:19 2022-07-29 20:05:59,259 - INFO - [TRAIN] epoch=102/604, iter=11800/70000, loss=1.132366, lr=0.000125 | ETA 05:08:24 2022-07-29 20:06:18,493 - INFO - [TRAIN] epoch=103/604, iter=11850/70000, loss=1.148075, lr=0.000125 | ETA 05:40:45 2022-07-29 20:06:35,281 - INFO - [TRAIN] epoch=103/604, iter=11900/70000, loss=1.050375, lr=0.000125 | ETA 07:31:52 2022-07-29 20:06:53,770 - INFO - [TRAIN] epoch=104/604, iter=11950/70000, loss=1.082876, lr=0.000125 | ETA 11:24:39 2022-07-29 20:07:12,295 - INFO - [TRAIN] epoch=104/604, iter=12000/70000, loss=1.140535, lr=0.000125 | ETA 07:08:41 2022-07-29 20:07:29,656 - INFO - [TRAIN] epoch=104/604, iter=12050/70000, loss=1.102522, lr=0.000125 | ETA 08:33:30 2022-07-29 20:07:48,560 - INFO - [TRAIN] epoch=105/604, iter=12100/70000, loss=1.027037, lr=0.000125 | ETA 05:06:39 2022-07-29 20:08:05,723 - INFO - [TRAIN] epoch=105/604, iter=12150/70000, loss=1.035299, lr=0.000125 | ETA 08:07:55 2022-07-29 20:08:23,627 - INFO - [TRAIN] epoch=106/604, iter=12200/70000, loss=1.030475, lr=0.000125 | ETA 05:13:01 2022-07-29 20:08:41,255 - INFO - [TRAIN] epoch=106/604, iter=12250/70000, loss=1.021100, lr=0.000125 | ETA 05:02:20 2022-07-29 20:08:59,664 - INFO - [TRAIN] epoch=107/604, iter=12300/70000, loss=1.029254, lr=0.000125 | ETA 06:44:18 2022-07-29 20:09:18,096 - INFO - [TRAIN] epoch=107/604, iter=12350/70000, loss=1.101615, lr=0.000125 | ETA 08:30:34 2022-07-29 20:09:36,414 - INFO - [TRAIN] epoch=107/604, iter=12400/70000, loss=1.022607, lr=0.000125 | ETA 05:46:01 2022-07-29 20:09:55,565 - INFO - [TRAIN] epoch=108/604, iter=12450/70000, loss=1.158201, lr=0.000125 | ETA 05:00:32 2022-07-29 20:10:13,977 - INFO - [TRAIN] epoch=108/604, iter=12500/70000, loss=1.087421, lr=0.000125 | ETA 05:37:40 2022-07-29 20:10:32,633 - INFO - [TRAIN] epoch=109/604, iter=12550/70000, loss=1.132679, lr=0.000125 | ETA 05:26:38 2022-07-29 20:10:51,267 - INFO - [TRAIN] epoch=109/604, iter=12600/70000, loss=1.088629, lr=0.000125 | ETA 05:16:53 2022-07-29 20:11:10,312 - INFO - [TRAIN] epoch=110/604, iter=12650/70000, loss=1.081608, lr=0.000125 | ETA 05:33:36 2022-07-29 20:11:28,286 - INFO - [TRAIN] epoch=110/604, iter=12700/70000, loss=1.105344, lr=0.000125 | ETA 05:07:58 2022-07-29 20:11:45,250 - INFO - [TRAIN] epoch=110/604, iter=12750/70000, loss=1.032773, lr=0.000125 | ETA 05:06:45 2022-07-29 20:12:04,232 - INFO - [TRAIN] epoch=111/604, iter=12800/70000, loss=1.185317, lr=0.000125 | ETA 05:25:01 2022-07-29 20:12:20,900 - INFO - [TRAIN] epoch=111/604, iter=12850/70000, loss=1.009657, lr=0.000125 | ETA 04:58:20 2022-07-29 20:12:40,078 - INFO - [TRAIN] epoch=112/604, iter=12900/70000, loss=1.091695, lr=0.000125 | ETA 05:42:45 2022-07-29 20:12:57,180 - INFO - [TRAIN] epoch=112/604, iter=12950/70000, loss=1.173912, lr=0.000125 | ETA 04:58:05 2022-07-29 20:13:14,943 - INFO - [TRAIN] epoch=113/604, iter=13000/70000, loss=0.983391, lr=0.000125 | ETA 05:28:09 2022-07-29 20:13:34,079 - INFO - [TRAIN] epoch=113/604, iter=13050/70000, loss=1.109996, lr=0.000125 | ETA 05:04:30 2022-07-29 20:13:51,533 - INFO - [TRAIN] epoch=113/604, iter=13100/70000, loss=1.093069, lr=0.000125 | ETA 05:08:09 2022-07-29 20:14:11,091 - INFO - [TRAIN] epoch=114/604, iter=13150/70000, loss=1.079588, lr=0.000125 | ETA 06:23:16 2022-07-29 20:14:28,499 - INFO - [TRAIN] epoch=114/604, iter=13200/70000, loss=1.056028, lr=0.000125 | ETA 05:00:43 2022-07-29 20:14:46,809 - INFO - [TRAIN] epoch=115/604, iter=13250/70000, loss=1.052881, lr=0.000125 | ETA 05:00:29 2022-07-29 20:15:03,703 - INFO - [TRAIN] epoch=115/604, iter=13300/70000, loss=0.956948, lr=0.000125 | ETA 05:08:43 2022-07-29 20:15:22,210 - INFO - [TRAIN] epoch=116/604, iter=13350/70000, loss=1.020905, lr=0.000125 | ETA 05:02:46 2022-07-29 20:15:40,213 - INFO - [TRAIN] epoch=116/604, iter=13400/70000, loss=0.984507, lr=0.000125 | ETA 05:13:45 2022-07-29 20:15:57,570 - INFO - [TRAIN] epoch=116/604, iter=13450/70000, loss=1.021959, lr=0.000125 | ETA 05:38:22 2022-07-29 20:16:17,376 - INFO - [TRAIN] epoch=117/604, iter=13500/70000, loss=0.964080, lr=0.000125 | ETA 05:01:54 2022-07-29 20:16:34,574 - INFO - [TRAIN] epoch=117/604, iter=13550/70000, loss=0.939473, lr=0.000125 | ETA 04:57:11 2022-07-29 20:16:54,481 - INFO - [TRAIN] epoch=118/604, iter=13600/70000, loss=0.978792, lr=0.000125 | ETA 05:05:06 2022-07-29 20:17:11,246 - INFO - [TRAIN] epoch=118/604, iter=13650/70000, loss=1.046263, lr=0.000125 | ETA 05:01:13 2022-07-29 20:17:31,101 - INFO - [TRAIN] epoch=119/604, iter=13700/70000, loss=0.957513, lr=0.000125 | ETA 13:28:54 2022-07-29 20:17:48,338 - INFO - [TRAIN] epoch=119/604, iter=13750/70000, loss=0.993644, lr=0.000125 | ETA 05:01:44 2022-07-29 20:18:06,225 - INFO - [TRAIN] epoch=119/604, iter=13800/70000, loss=1.002627, lr=0.000125 | ETA 04:56:10 2022-07-29 20:18:25,184 - INFO - [TRAIN] epoch=120/604, iter=13850/70000, loss=0.995841, lr=0.000125 | ETA 05:04:40 2022-07-29 20:18:42,089 - INFO - [TRAIN] epoch=120/604, iter=13900/70000, loss=1.023465, lr=0.000125 | ETA 05:44:16 2022-07-29 20:19:00,674 - INFO - [TRAIN] epoch=121/604, iter=13950/70000, loss=0.917492, lr=0.000125 | ETA 05:03:22 2022-07-29 20:19:18,432 - INFO - [TRAIN] epoch=121/604, iter=14000/70000, loss=1.063566, lr=0.000125 | ETA 04:57:47 2022-07-29 20:19:38,968 - INFO - [TRAIN] epoch=122/604, iter=14050/70000, loss=0.951652, lr=0.000125 | ETA 07:11:21 2022-07-29 20:19:57,075 - INFO - [TRAIN] epoch=122/604, iter=14100/70000, loss=0.977704, lr=0.000125 | ETA 04:55:08 2022-07-29 20:20:14,317 - INFO - [TRAIN] epoch=122/604, iter=14150/70000, loss=1.004363, lr=0.000125 | ETA 04:54:38 2022-07-29 20:20:34,104 - INFO - [TRAIN] epoch=123/604, iter=14200/70000, loss=0.950308, lr=0.000125 | ETA 07:54:47 2022-07-29 20:20:51,104 - INFO - [TRAIN] epoch=123/604, iter=14250/70000, loss=1.039467, lr=0.000125 | ETA 04:57:14 2022-07-29 20:21:10,774 - INFO - [TRAIN] epoch=124/604, iter=14300/70000, loss=0.967168, lr=0.000125 | ETA 04:53:37 2022-07-29 20:21:29,322 - INFO - [TRAIN] epoch=124/604, iter=14350/70000, loss=0.930714, lr=0.000125 | ETA 04:51:32 2022-07-29 20:21:48,899 - INFO - [TRAIN] epoch=125/604, iter=14400/70000, loss=0.941907, lr=0.000125 | ETA 05:38:40 2022-07-29 20:22:05,790 - INFO - [TRAIN] epoch=125/604, iter=14450/70000, loss=1.037637, lr=0.000125 | ETA 04:53:41 2022-07-29 20:22:23,762 - INFO - [TRAIN] epoch=125/604, iter=14500/70000, loss=0.947876, lr=0.000125 | ETA 04:49:46 2022-07-29 20:22:42,371 - INFO - [TRAIN] epoch=126/604, iter=14550/70000, loss=0.991238, lr=0.000125 | ETA 04:52:17 2022-07-29 20:22:59,996 - INFO - [TRAIN] epoch=126/604, iter=14600/70000, loss=1.029029, lr=0.000125 | ETA 05:02:57 2022-07-29 20:23:18,580 - INFO - [TRAIN] epoch=127/604, iter=14650/70000, loss=0.951712, lr=0.000125 | ETA 04:57:09 2022-07-29 20:23:36,324 - INFO - [TRAIN] epoch=127/604, iter=14700/70000, loss=0.920379, lr=0.000125 | ETA 06:17:07 2022-07-29 20:23:55,729 - INFO - [TRAIN] epoch=128/604, iter=14750/70000, loss=0.954685, lr=0.000125 | ETA 06:12:28 2022-07-29 20:24:12,971 - INFO - [TRAIN] epoch=128/604, iter=14800/70000, loss=1.032970, lr=0.000125 | ETA 04:51:13 2022-07-29 20:24:32,273 - INFO - [TRAIN] epoch=129/604, iter=14850/70000, loss=1.010359, lr=0.000125 | ETA 10:46:31 2022-07-29 20:24:50,116 - INFO - [TRAIN] epoch=129/604, iter=14900/70000, loss=0.921711, lr=0.000125 | ETA 07:31:13 2022-07-29 20:25:07,847 - INFO - [TRAIN] epoch=129/604, iter=14950/70000, loss=0.901221, lr=0.000125 | ETA 04:59:22 2022-07-29 20:25:27,813 - INFO - [TRAIN] epoch=130/604, iter=15000/70000, loss=0.980739, lr=0.000125 | ETA 04:57:43 2022-07-29 20:25:29,014 - INFO - Push model to checkpoint ./output/iter_15000 2022-07-29 20:25:45,738 - INFO - [TRAIN] epoch=130/604, iter=15050/70000, loss=0.918730, lr=0.000125 | ETA 04:53:53 2022-07-29 20:26:05,444 - INFO - [TRAIN] epoch=131/604, iter=15100/70000, loss=0.939401, lr=0.000125 | ETA 13:49:55 2022-07-29 20:26:22,238 - INFO - [TRAIN] epoch=131/604, iter=15150/70000, loss=0.861909, lr=0.000125 | ETA 04:46:39 2022-07-29 20:26:40,872 - INFO - [TRAIN] epoch=132/604, iter=15200/70000, loss=0.956663, lr=0.000125 | ETA 06:24:14 2022-07-29 20:26:58,474 - INFO - [TRAIN] epoch=132/604, iter=15250/70000, loss=0.980184, lr=0.000125 | ETA 04:48:32 2022-07-29 20:27:15,328 - INFO - [TRAIN] epoch=132/604, iter=15300/70000, loss=0.942982, lr=0.000125 | ETA 04:45:48 2022-07-29 20:27:34,262 - INFO - [TRAIN] epoch=133/604, iter=15350/70000, loss=0.888023, lr=0.000125 | ETA 04:47:40 2022-07-29 20:27:51,065 - INFO - [TRAIN] epoch=133/604, iter=15400/70000, loss=0.922819, lr=0.000125 | ETA 04:52:19 2022-07-29 20:28:10,071 - INFO - [TRAIN] epoch=134/604, iter=15450/70000, loss=0.911693, lr=0.000125 | ETA 05:23:44 2022-07-29 20:28:28,029 - INFO - [TRAIN] epoch=134/604, iter=15500/70000, loss=0.927874, lr=0.000125 | ETA 04:54:07 2022-07-29 20:28:47,408 - INFO - [TRAIN] epoch=135/604, iter=15550/70000, loss=0.981312, lr=0.000125 | ETA 05:10:56 2022-07-29 20:29:05,962 - INFO - [TRAIN] epoch=135/604, iter=15600/70000, loss=0.917505, lr=0.000125 | ETA 04:56:14 2022-07-29 20:29:23,629 - INFO - [TRAIN] epoch=135/604, iter=15650/70000, loss=0.912253, lr=0.000125 | ETA 04:44:52 2022-07-29 20:29:43,195 - INFO - [TRAIN] epoch=136/604, iter=15700/70000, loss=0.917170, lr=0.000125 | ETA 06:29:00 2022-07-29 20:30:00,872 - INFO - [TRAIN] epoch=136/604, iter=15750/70000, loss=0.885766, lr=0.000125 | ETA 04:57:11 2022-07-29 20:30:19,520 - INFO - [TRAIN] epoch=137/604, iter=15800/70000, loss=0.883247, lr=0.000125 | ETA 04:56:29 2022-07-29 20:30:37,854 - INFO - [TRAIN] epoch=137/604, iter=15850/70000, loss=0.978009, lr=0.000125 | ETA 04:56:55 2022-07-29 20:30:57,898 - INFO - [TRAIN] epoch=138/604, iter=15900/70000, loss=0.898167, lr=0.000125 | ETA 04:53:44 2022-07-29 20:31:15,373 - INFO - [TRAIN] epoch=138/604, iter=15950/70000, loss=0.928189, lr=0.000125 | ETA 04:46:00 2022-07-29 20:31:33,187 - INFO - [TRAIN] epoch=138/604, iter=16000/70000, loss=0.998280, lr=0.000125 | ETA 08:52:27 2022-07-29 20:31:53,794 - INFO - [TRAIN] epoch=139/604, iter=16050/70000, loss=0.894295, lr=0.000125 | ETA 08:59:08 2022-07-29 20:32:11,326 - INFO - [TRAIN] epoch=139/604, iter=16100/70000, loss=0.877073, lr=0.000125 | ETA 05:43:52 2022-07-29 20:32:32,296 - INFO - [TRAIN] epoch=140/604, iter=16150/70000, loss=0.911754, lr=0.000125 | ETA 07:02:37 2022-07-29 20:32:50,294 - INFO - [TRAIN] epoch=140/604, iter=16200/70000, loss=0.824380, lr=0.000125 | ETA 05:04:30 2022-07-29 20:33:10,068 - INFO - [TRAIN] epoch=141/604, iter=16250/70000, loss=0.930106, lr=0.000125 | ETA 04:47:09 2022-07-29 20:33:27,864 - INFO - [TRAIN] epoch=141/604, iter=16300/70000, loss=0.914985, lr=0.000125 | ETA 04:50:51 2022-07-29 20:33:45,954 - INFO - [TRAIN] epoch=141/604, iter=16350/70000, loss=0.868137, lr=0.000125 | ETA 04:45:24 2022-07-29 20:34:05,584 - INFO - [TRAIN] epoch=142/604, iter=16400/70000, loss=0.873172, lr=0.000125 | ETA 04:52:43 2022-07-29 20:34:23,820 - INFO - [TRAIN] epoch=142/604, iter=16450/70000, loss=0.894617, lr=0.000125 | ETA 05:55:38 2022-07-29 20:34:44,093 - INFO - [TRAIN] epoch=143/604, iter=16500/70000, loss=0.877792, lr=0.000125 | ETA 07:42:28 2022-07-29 20:35:02,015 - INFO - [TRAIN] epoch=143/604, iter=16550/70000, loss=0.917149, lr=0.000125 | ETA 04:44:23 2022-07-29 20:35:20,817 - INFO - [TRAIN] epoch=144/604, iter=16600/70000, loss=0.892948, lr=0.000125 | ETA 04:42:00 2022-07-29 20:35:38,883 - INFO - [TRAIN] epoch=144/604, iter=16650/70000, loss=0.857829, lr=0.000125 | ETA 04:52:30 2022-07-29 20:35:57,802 - INFO - [TRAIN] epoch=144/604, iter=16700/70000, loss=0.873674, lr=0.000125 | ETA 05:09:29 2022-07-29 20:36:16,822 - INFO - [TRAIN] epoch=145/604, iter=16750/70000, loss=0.892522, lr=0.000125 | ETA 05:50:21 2022-07-29 20:36:33,529 - INFO - [TRAIN] epoch=145/604, iter=16800/70000, loss=0.847180, lr=0.000125 | ETA 04:51:39 2022-07-29 20:36:51,697 - INFO - [TRAIN] epoch=146/604, iter=16850/70000, loss=0.843902, lr=0.000125 | ETA 04:46:46 2022-07-29 20:37:09,162 - INFO - [TRAIN] epoch=146/604, iter=16900/70000, loss=0.922420, lr=0.000125 | ETA 04:47:40 2022-07-29 20:37:28,743 - INFO - [TRAIN] epoch=147/604, iter=16950/70000, loss=0.892194, lr=0.000125 | ETA 05:47:34 2022-07-29 20:37:45,843 - INFO - [TRAIN] epoch=147/604, iter=17000/70000, loss=0.911434, lr=0.000125 | ETA 05:05:23 2022-07-29 20:38:03,414 - INFO - [TRAIN] epoch=147/604, iter=17050/70000, loss=0.854886, lr=0.000125 | ETA 05:00:50 2022-07-29 20:38:24,120 - INFO - [TRAIN] epoch=148/604, iter=17100/70000, loss=0.863887, lr=0.000125 | ETA 04:42:15 2022-07-29 20:38:41,680 - INFO - [TRAIN] epoch=148/604, iter=17150/70000, loss=0.864988, lr=0.000125 | ETA 05:41:48 2022-07-29 20:39:01,253 - INFO - [TRAIN] epoch=149/604, iter=17200/70000, loss=0.954418, lr=0.000125 | ETA 05:00:25 2022-07-29 20:39:19,027 - INFO - [TRAIN] epoch=149/604, iter=17250/70000, loss=0.966925, lr=0.000125 | ETA 04:34:11 2022-07-29 20:39:38,453 - INFO - [TRAIN] epoch=150/604, iter=17300/70000, loss=0.940324, lr=0.000125 | ETA 04:45:12 2022-07-29 20:39:56,517 - INFO - [TRAIN] epoch=150/604, iter=17350/70000, loss=0.818480, lr=0.000125 | ETA 04:59:12 2022-07-29 20:40:14,939 - INFO - [TRAIN] epoch=150/604, iter=17400/70000, loss=0.843782, lr=0.000125 | ETA 04:31:51 2022-07-29 20:40:33,286 - INFO - [TRAIN] epoch=151/604, iter=17450/70000, loss=0.924874, lr=0.000125 | ETA 04:41:57 2022-07-29 20:40:51,398 - INFO - [TRAIN] epoch=151/604, iter=17500/70000, loss=0.889972, lr=0.000125 | ETA 07:25:48 2022-07-29 20:41:09,795 - INFO - [TRAIN] epoch=152/604, iter=17550/70000, loss=0.925264, lr=0.000125 | ETA 04:45:53 2022-07-29 20:41:28,830 - INFO - [TRAIN] epoch=152/604, iter=17600/70000, loss=0.853518, lr=0.000125 | ETA 10:00:45 2022-07-29 20:41:48,337 - INFO - [TRAIN] epoch=153/604, iter=17650/70000, loss=0.960315, lr=0.000125 | ETA 12:36:26 2022-07-29 20:42:06,892 - INFO - [TRAIN] epoch=153/604, iter=17700/70000, loss=0.842231, lr=0.000125 | ETA 08:04:37 2022-07-29 20:42:26,185 - INFO - [TRAIN] epoch=154/604, iter=17750/70000, loss=0.802987, lr=0.000125 | ETA 10:37:29 2022-07-29 20:42:43,962 - INFO - [TRAIN] epoch=154/604, iter=17800/70000, loss=0.854597, lr=0.000125 | ETA 05:41:12 2022-07-29 20:43:00,957 - INFO - [TRAIN] epoch=154/604, iter=17850/70000, loss=0.845768, lr=0.000125 | ETA 04:37:38 2022-07-29 20:43:20,054 - INFO - [TRAIN] epoch=155/604, iter=17900/70000, loss=0.815183, lr=0.000125 | ETA 04:45:55 2022-07-29 20:43:39,139 - INFO - [TRAIN] epoch=155/604, iter=17950/70000, loss=0.914943, lr=0.000125 | ETA 06:23:18 2022-07-29 20:43:59,144 - INFO - [TRAIN] epoch=156/604, iter=18000/70000, loss=0.819058, lr=0.000125 | ETA 04:46:46 2022-07-29 20:44:16,656 - INFO - [TRAIN] epoch=156/604, iter=18050/70000, loss=0.805301, lr=0.000125 | ETA 04:40:16 2022-07-29 20:44:35,404 - INFO - [TRAIN] epoch=157/604, iter=18100/70000, loss=0.882032, lr=0.000125 | ETA 06:26:26 2022-07-29 20:44:53,373 - INFO - [TRAIN] epoch=157/604, iter=18150/70000, loss=0.853320, lr=0.000125 | ETA 04:32:25 2022-07-29 20:45:10,542 - INFO - [TRAIN] epoch=157/604, iter=18200/70000, loss=0.914417, lr=0.000125 | ETA 04:33:22 2022-07-29 20:45:29,911 - INFO - [TRAIN] epoch=158/604, iter=18250/70000, loss=0.880662, lr=0.000125 | ETA 04:37:56 2022-07-29 20:45:46,906 - INFO - [TRAIN] epoch=158/604, iter=18300/70000, loss=0.830811, lr=0.000125 | ETA 04:30:46 2022-07-29 20:46:06,603 - INFO - [TRAIN] epoch=159/604, iter=18350/70000, loss=0.856389, lr=0.000125 | ETA 04:31:03 2022-07-29 20:46:25,774 - INFO - [TRAIN] epoch=159/604, iter=18400/70000, loss=0.847889, lr=0.000125 | ETA 05:47:19 2022-07-29 20:46:46,079 - INFO - [TRAIN] epoch=160/604, iter=18450/70000, loss=0.803650, lr=0.000125 | ETA 04:54:01 2022-07-29 20:47:03,258 - INFO - [TRAIN] epoch=160/604, iter=18500/70000, loss=0.888558, lr=0.000125 | ETA 04:31:32 2022-07-29 20:47:21,346 - INFO - [TRAIN] epoch=160/604, iter=18550/70000, loss=0.869083, lr=0.000125 | ETA 05:08:11 2022-07-29 20:47:40,511 - INFO - [TRAIN] epoch=161/604, iter=18600/70000, loss=0.793540, lr=0.000125 | ETA 04:29:17 2022-07-29 20:47:58,489 - INFO - [TRAIN] epoch=161/604, iter=18650/70000, loss=0.812595, lr=0.000125 | ETA 04:34:54 2022-07-29 20:48:16,772 - INFO - [TRAIN] epoch=162/604, iter=18700/70000, loss=0.734495, lr=0.000125 | ETA 04:49:11 2022-07-29 20:48:34,803 - INFO - [TRAIN] epoch=162/604, iter=18750/70000, loss=0.877197, lr=0.000125 | ETA 04:29:32 2022-07-29 20:48:53,684 - INFO - [TRAIN] epoch=163/604, iter=18800/70000, loss=0.915257, lr=0.000125 | ETA 04:38:14 2022-07-29 20:49:12,378 - INFO - [TRAIN] epoch=163/604, iter=18850/70000, loss=0.842352, lr=0.000125 | ETA 04:52:43 2022-07-29 20:49:30,076 - INFO - [TRAIN] epoch=163/604, iter=18900/70000, loss=0.854837, lr=0.000125 | ETA 04:37:35 2022-07-29 20:49:49,283 - INFO - [TRAIN] epoch=164/604, iter=18950/70000, loss=0.826688, lr=0.000125 | ETA 04:26:06 2022-07-29 20:50:07,806 - INFO - [TRAIN] epoch=164/604, iter=19000/70000, loss=0.784832, lr=0.000125 | ETA 04:39:12 2022-07-29 20:50:27,989 - INFO - [TRAIN] epoch=165/604, iter=19050/70000, loss=0.755963, lr=0.000125 | ETA 05:32:53 2022-07-29 20:50:46,259 - INFO - [TRAIN] epoch=165/604, iter=19100/70000, loss=0.824101, lr=0.000125 | ETA 04:32:02 2022-07-29 20:51:04,577 - INFO - [TRAIN] epoch=166/604, iter=19150/70000, loss=0.838055, lr=0.000125 | ETA 04:35:16 2022-07-29 20:51:22,707 - INFO - [TRAIN] epoch=166/604, iter=19200/70000, loss=0.790184, lr=0.000125 | ETA 05:02:34 2022-07-29 20:51:39,993 - INFO - [TRAIN] epoch=166/604, iter=19250/70000, loss=0.815399, lr=0.000125 | ETA 04:40:15 2022-07-29 20:51:59,928 - INFO - [TRAIN] epoch=167/604, iter=19300/70000, loss=0.838549, lr=0.000125 | ETA 07:42:19 2022-07-29 20:52:17,028 - INFO - [TRAIN] epoch=167/604, iter=19350/70000, loss=0.751371, lr=0.000125 | ETA 04:26:47 2022-07-29 20:52:37,446 - INFO - [TRAIN] epoch=168/604, iter=19400/70000, loss=0.763987, lr=0.000125 | ETA 04:55:17 2022-07-29 20:52:54,701 - INFO - [TRAIN] epoch=168/604, iter=19450/70000, loss=0.866449, lr=0.000125 | ETA 04:40:34 2022-07-29 20:53:15,463 - INFO - [TRAIN] epoch=169/604, iter=19500/70000, loss=0.810993, lr=0.000125 | ETA 05:08:52 2022-07-29 20:53:33,271 - INFO - [TRAIN] epoch=169/604, iter=19550/70000, loss=0.743156, lr=0.000125 | ETA 04:28:40 2022-07-29 20:53:51,210 - INFO - [TRAIN] epoch=169/604, iter=19600/70000, loss=0.807605, lr=0.000125 | ETA 04:27:24 2022-07-29 20:54:11,188 - INFO - [TRAIN] epoch=170/604, iter=19650/70000, loss=0.802747, lr=0.000125 | ETA 05:10:39 2022-07-29 20:54:29,799 - INFO - [TRAIN] epoch=170/604, iter=19700/70000, loss=0.812826, lr=0.000125 | ETA 04:34:22 2022-07-29 20:54:49,311 - INFO - [TRAIN] epoch=171/604, iter=19750/70000, loss=0.762347, lr=0.000125 | ETA 06:37:05 2022-07-29 20:55:06,465 - INFO - [TRAIN] epoch=171/604, iter=19800/70000, loss=0.814996, lr=0.000125 | ETA 04:41:29 2022-07-29 20:55:25,876 - INFO - [TRAIN] epoch=172/604, iter=19850/70000, loss=0.872846, lr=0.000125 | ETA 04:32:53 2022-07-29 20:55:43,830 - INFO - [TRAIN] epoch=172/604, iter=19900/70000, loss=0.781788, lr=0.000125 | ETA 05:01:46 2022-07-29 20:56:02,204 - INFO - [TRAIN] epoch=172/604, iter=19950/70000, loss=0.779254, lr=0.000125 | ETA 04:19:40 2022-07-29 20:56:22,012 - INFO - [TRAIN] epoch=173/604, iter=20000/70000, loss=0.830419, lr=0.000125 | ETA 04:22:47 2022-07-29 20:56:23,244 - INFO - Push model to checkpoint ./output/iter_20000 2022-07-29 20:56:39,754 - INFO - [TRAIN] epoch=173/604, iter=20050/70000, loss=0.894469, lr=0.000125 | ETA 04:25:07 2022-07-29 20:56:58,462 - INFO - [TRAIN] epoch=174/604, iter=20100/70000, loss=0.808434, lr=0.000125 | ETA 04:22:28 2022-07-29 20:57:16,573 - INFO - [TRAIN] epoch=174/604, iter=20150/70000, loss=0.823431, lr=0.000125 | ETA 04:23:26 2022-07-29 20:57:37,494 - INFO - [TRAIN] epoch=175/604, iter=20200/70000, loss=0.861655, lr=0.000125 | ETA 10:35:40 2022-07-29 20:57:54,728 - INFO - [TRAIN] epoch=175/604, iter=20250/70000, loss=0.793086, lr=0.000125 | ETA 04:40:43 2022-07-29 20:58:12,626 - INFO - [TRAIN] epoch=175/604, iter=20300/70000, loss=0.845696, lr=0.000125 | ETA 04:19:33 2022-07-29 20:58:31,533 - INFO - [TRAIN] epoch=176/604, iter=20350/70000, loss=0.785115, lr=0.000125 | ETA 04:36:30 2022-07-29 20:58:48,824 - INFO - [TRAIN] epoch=176/604, iter=20400/70000, loss=0.786066, lr=0.000125 | ETA 04:33:14 2022-07-29 20:59:09,046 - INFO - [TRAIN] epoch=177/604, iter=20450/70000, loss=0.814229, lr=0.000125 | ETA 04:37:33 2022-07-29 20:59:27,490 - INFO - [TRAIN] epoch=177/604, iter=20500/70000, loss=0.855413, lr=0.000125 | ETA 04:26:24 2022-07-29 20:59:47,349 - INFO - [TRAIN] epoch=178/604, iter=20550/70000, loss=0.815697, lr=0.000125 | ETA 08:22:29 2022-07-29 21:00:04,889 - INFO - [TRAIN] epoch=178/604, iter=20600/70000, loss=0.829928, lr=0.000125 | ETA 04:22:23 2022-07-29 21:00:25,042 - INFO - [TRAIN] epoch=179/604, iter=20650/70000, loss=0.743405, lr=0.000125 | ETA 10:05:53 2022-07-29 21:00:43,814 - INFO - [TRAIN] epoch=179/604, iter=20700/70000, loss=0.718537, lr=0.000125 | ETA 05:43:57 2022-07-29 21:01:02,331 - INFO - [TRAIN] epoch=179/604, iter=20750/70000, loss=0.785040, lr=0.000125 | ETA 04:33:39 2022-07-29 21:01:23,246 - INFO - [TRAIN] epoch=180/604, iter=20800/70000, loss=0.809681, lr=0.000125 | ETA 08:37:49 2022-07-29 21:01:41,318 - INFO - [TRAIN] epoch=180/604, iter=20850/70000, loss=0.758840, lr=0.000125 | ETA 04:44:36 2022-07-29 21:02:01,068 - INFO - [TRAIN] epoch=181/604, iter=20900/70000, loss=0.775522, lr=0.000125 | ETA 04:59:25 2022-07-29 21:02:19,767 - INFO - [TRAIN] epoch=181/604, iter=20950/70000, loss=0.785715, lr=0.000125 | ETA 04:31:16 2022-07-29 21:02:38,843 - INFO - [TRAIN] epoch=182/604, iter=21000/70000, loss=0.771118, lr=0.000125 | ETA 05:45:02 2022-07-29 21:02:57,587 - INFO - [TRAIN] epoch=182/604, iter=21050/70000, loss=0.742506, lr=0.000125 | ETA 04:56:14 2022-07-29 21:03:15,010 - INFO - [TRAIN] epoch=182/604, iter=21100/70000, loss=0.743564, lr=0.000125 | ETA 04:17:21 2022-07-29 21:03:35,119 - INFO - [TRAIN] epoch=183/604, iter=21150/70000, loss=0.727749, lr=0.000125 | ETA 07:08:56 2022-07-29 21:03:53,042 - INFO - [TRAIN] epoch=183/604, iter=21200/70000, loss=0.767873, lr=0.000125 | ETA 04:15:12 2022-07-29 21:04:12,508 - INFO - [TRAIN] epoch=184/604, iter=21250/70000, loss=0.839368, lr=0.000125 | ETA 04:24:22 2022-07-29 21:04:30,074 - INFO - [TRAIN] epoch=184/604, iter=21300/70000, loss=0.763852, lr=0.000125 | ETA 04:27:41 2022-07-29 21:04:50,321 - INFO - [TRAIN] epoch=185/604, iter=21350/70000, loss=0.781056, lr=0.000125 | ETA 05:05:57 2022-07-29 21:05:08,823 - INFO - [TRAIN] epoch=185/604, iter=21400/70000, loss=0.756880, lr=0.000125 | ETA 04:12:48 2022-07-29 21:05:27,706 - INFO - [TRAIN] epoch=185/604, iter=21450/70000, loss=0.782483, lr=0.000125 | ETA 04:15:26 2022-07-29 21:05:48,325 - INFO - [TRAIN] epoch=186/604, iter=21500/70000, loss=0.766731, lr=0.000125 | ETA 05:36:33 2022-07-29 21:06:05,635 - INFO - [TRAIN] epoch=186/604, iter=21550/70000, loss=0.793586, lr=0.000125 | ETA 04:13:52 2022-07-29 21:06:25,137 - INFO - [TRAIN] epoch=187/604, iter=21600/70000, loss=0.772710, lr=0.000125 | ETA 04:13:34 2022-07-29 21:06:42,706 - INFO - [TRAIN] epoch=187/604, iter=21650/70000, loss=0.764957, lr=0.000125 | ETA 04:11:52 2022-07-29 21:07:02,912 - INFO - [TRAIN] epoch=188/604, iter=21700/70000, loss=0.773010, lr=0.000125 | ETA 05:06:27 2022-07-29 21:07:21,824 - INFO - [TRAIN] epoch=188/604, iter=21750/70000, loss=0.792493, lr=0.000125 | ETA 06:46:23 2022-07-29 21:07:41,936 - INFO - [TRAIN] epoch=188/604, iter=21800/70000, loss=0.814400, lr=0.000125 | ETA 04:33:31 2022-07-29 21:08:01,543 - INFO - [TRAIN] epoch=189/604, iter=21850/70000, loss=0.784322, lr=0.000125 | ETA 04:24:52 2022-07-29 21:08:18,817 - INFO - [TRAIN] epoch=189/604, iter=21900/70000, loss=0.716297, lr=0.000125 | ETA 04:16:50 2022-07-29 21:08:38,189 - INFO - [TRAIN] epoch=190/604, iter=21950/70000, loss=0.831617, lr=0.000125 | ETA 04:50:01 2022-07-29 21:08:56,100 - INFO - [TRAIN] epoch=190/604, iter=22000/70000, loss=0.759397, lr=0.000125 | ETA 04:10:42 2022-07-29 21:09:14,703 - INFO - [TRAIN] epoch=191/604, iter=22050/70000, loss=0.798049, lr=0.000125 | ETA 04:19:23 2022-07-29 21:09:35,343 - INFO - [TRAIN] epoch=191/604, iter=22100/70000, loss=0.795391, lr=0.000125 | ETA 04:25:03 2022-07-29 21:09:52,659 - INFO - [TRAIN] epoch=191/604, iter=22150/70000, loss=0.790443, lr=0.000125 | ETA 04:20:00 2022-07-29 21:10:11,937 - INFO - [TRAIN] epoch=192/604, iter=22200/70000, loss=0.803101, lr=0.000125 | ETA 04:15:14 2022-07-29 21:10:30,358 - INFO - [TRAIN] epoch=192/604, iter=22250/70000, loss=0.749495, lr=0.000125 | ETA 04:20:19 2022-07-29 21:10:49,234 - INFO - [TRAIN] epoch=193/604, iter=22300/70000, loss=0.738142, lr=0.000125 | ETA 04:07:52 2022-07-29 21:11:07,960 - INFO - [TRAIN] epoch=193/604, iter=22350/70000, loss=0.725816, lr=0.000125 | ETA 04:09:15 2022-07-29 21:11:26,997 - INFO - [TRAIN] epoch=194/604, iter=22400/70000, loss=0.762250, lr=0.000125 | ETA 04:23:10 2022-07-29 21:11:45,316 - INFO - [TRAIN] epoch=194/604, iter=22450/70000, loss=0.785053, lr=0.000125 | ETA 04:08:24 2022-07-29 21:12:04,037 - INFO - [TRAIN] epoch=194/604, iter=22500/70000, loss=0.768911, lr=0.000125 | ETA 05:02:17 2022-07-29 21:12:22,947 - INFO - [TRAIN] epoch=195/604, iter=22550/70000, loss=0.732982, lr=0.000125 | ETA 04:10:38 2022-07-29 21:12:40,984 - INFO - [TRAIN] epoch=195/604, iter=22600/70000, loss=0.723152, lr=0.000125 | ETA 04:13:09 2022-07-29 21:13:00,321 - INFO - [TRAIN] epoch=196/604, iter=22650/70000, loss=0.655124, lr=0.000125 | ETA 04:25:44 2022-07-29 21:13:18,444 - INFO - [TRAIN] epoch=196/604, iter=22700/70000, loss=0.786784, lr=0.000125 | ETA 04:28:19 2022-07-29 21:13:38,193 - INFO - [TRAIN] epoch=197/604, iter=22750/70000, loss=0.783128, lr=0.000125 | ETA 04:15:26 2022-07-29 21:13:56,996 - INFO - [TRAIN] epoch=197/604, iter=22800/70000, loss=0.680536, lr=0.000125 | ETA 04:27:21 2022-07-29 21:14:14,958 - INFO - [TRAIN] epoch=197/604, iter=22850/70000, loss=0.784711, lr=0.000125 | ETA 04:15:47 2022-07-29 21:14:34,506 - INFO - [TRAIN] epoch=198/604, iter=22900/70000, loss=0.831906, lr=0.000125 | ETA 04:13:51 2022-07-29 21:14:52,262 - INFO - [TRAIN] epoch=198/604, iter=22950/70000, loss=0.889683, lr=0.000125 | ETA 04:06:06 2022-07-29 21:15:12,079 - INFO - [TRAIN] epoch=199/604, iter=23000/70000, loss=1.816318, lr=0.000125 | ETA 04:13:00 2022-07-29 21:15:30,259 - INFO - [TRAIN] epoch=199/604, iter=23050/70000, loss=1.179513, lr=0.000125 | ETA 04:17:17 2022-07-29 21:15:49,711 - INFO - [TRAIN] epoch=200/604, iter=23100/70000, loss=0.950035, lr=0.000125 | ETA 09:55:34 2022-07-29 21:16:08,078 - INFO - [TRAIN] epoch=200/604, iter=23150/70000, loss=0.891023, lr=0.000125 | ETA 04:25:04 2022-07-29 21:16:26,027 - INFO - [TRAIN] epoch=200/604, iter=23200/70000, loss=0.766622, lr=0.000125 | ETA 04:04:26 2022-07-29 21:16:45,296 - INFO - [TRAIN] epoch=201/604, iter=23250/70000, loss=0.786258, lr=0.000125 | ETA 04:12:45 2022-07-29 21:17:04,301 - INFO - [TRAIN] epoch=201/604, iter=23300/70000, loss=0.830361, lr=0.000125 | ETA 04:13:41 2022-07-29 21:17:25,096 - INFO - [TRAIN] epoch=202/604, iter=23350/70000, loss=0.797353, lr=0.000125 | ETA 04:05:31 2022-07-29 21:17:44,310 - INFO - [TRAIN] epoch=202/604, iter=23400/70000, loss=0.780461, lr=0.000125 | ETA 04:26:23 2022-07-29 21:18:02,690 - INFO - [TRAIN] epoch=203/604, iter=23450/70000, loss=0.781912, lr=0.000125 | ETA 04:11:32 2022-07-29 21:18:20,538 - INFO - [TRAIN] epoch=203/604, iter=23500/70000, loss=0.748065, lr=0.000125 | ETA 04:09:51 2022-07-29 21:18:38,744 - INFO - [TRAIN] epoch=204/604, iter=23550/70000, loss=0.727325, lr=0.000125 | ETA 08:57:14 2022-07-29 21:18:56,808 - INFO - [TRAIN] epoch=204/604, iter=23600/70000, loss=0.703817, lr=0.000125 | ETA 04:03:38 2022-07-29 21:19:14,943 - INFO - [TRAIN] epoch=204/604, iter=23650/70000, loss=0.712273, lr=0.000125 | ETA 04:17:05 2022-07-29 21:19:36,326 - INFO - [TRAIN] epoch=205/604, iter=23700/70000, loss=0.661561, lr=0.000125 | ETA 04:33:54 2022-07-29 21:19:53,959 - INFO - [TRAIN] epoch=205/604, iter=23750/70000, loss=0.724615, lr=0.000125 | ETA 04:00:20 2022-07-29 21:20:13,899 - INFO - [TRAIN] epoch=206/604, iter=23800/70000, loss=0.706434, lr=0.000125 | ETA 04:47:56 2022-07-29 21:20:32,213 - INFO - [TRAIN] epoch=206/604, iter=23850/70000, loss=0.724337, lr=0.000125 | ETA 04:14:32 2022-07-29 21:20:52,026 - INFO - [TRAIN] epoch=207/604, iter=23900/70000, loss=0.744422, lr=0.000125 | ETA 05:23:44 2022-07-29 21:21:10,920 - INFO - [TRAIN] epoch=207/604, iter=23950/70000, loss=0.742426, lr=0.000125 | ETA 04:01:30 2022-07-29 21:21:29,292 - INFO - [TRAIN] epoch=207/604, iter=24000/70000, loss=0.756617, lr=0.000125 | ETA 12:54:08 2022-07-29 21:21:49,348 - INFO - [TRAIN] epoch=208/604, iter=24050/70000, loss=0.794104, lr=0.000125 | ETA 04:02:13 2022-07-29 21:22:07,718 - INFO - [TRAIN] epoch=208/604, iter=24100/70000, loss=0.755344, lr=0.000125 | ETA 04:06:40 2022-07-29 21:22:26,836 - INFO - [TRAIN] epoch=209/604, iter=24150/70000, loss=0.782266, lr=0.000125 | ETA 04:03:52 2022-07-29 21:22:44,550 - INFO - [TRAIN] epoch=209/604, iter=24200/70000, loss=0.682858, lr=0.000125 | ETA 05:36:43 2022-07-29 21:23:04,575 - INFO - [TRAIN] epoch=210/604, iter=24250/70000, loss=0.692483, lr=0.000125 | ETA 04:24:43 2022-07-29 21:23:23,433 - INFO - [TRAIN] epoch=210/604, iter=24300/70000, loss=0.718012, lr=0.000125 | ETA 04:07:10 2022-07-29 21:23:41,786 - INFO - [TRAIN] epoch=210/604, iter=24350/70000, loss=0.673961, lr=0.000125 | ETA 04:02:19 2022-07-29 21:24:02,063 - INFO - [TRAIN] epoch=211/604, iter=24400/70000, loss=0.729881, lr=0.000125 | ETA 04:13:28 2022-07-29 21:24:20,213 - INFO - [TRAIN] epoch=211/604, iter=24450/70000, loss=0.705987, lr=0.000125 | ETA 04:13:52 2022-07-29 21:24:40,192 - INFO - [TRAIN] epoch=212/604, iter=24500/70000, loss=0.770437, lr=0.000125 | ETA 04:16:54 2022-07-29 21:24:58,789 - INFO - [TRAIN] epoch=212/604, iter=24550/70000, loss=0.747420, lr=0.000125 | ETA 07:18:18 2022-07-29 21:25:17,966 - INFO - [TRAIN] epoch=213/604, iter=24600/70000, loss=0.725444, lr=0.000125 | ETA 04:15:34 2022-07-29 21:25:36,264 - INFO - [TRAIN] epoch=213/604, iter=24650/70000, loss=0.759331, lr=0.000125 | ETA 04:17:43 2022-07-29 21:25:55,061 - INFO - [TRAIN] epoch=213/604, iter=24700/70000, loss=0.649108, lr=0.000125 | ETA 04:01:15 2022-07-29 21:26:14,631 - INFO - [TRAIN] epoch=214/604, iter=24750/70000, loss=0.774045, lr=0.000125 | ETA 06:42:35 2022-07-29 21:26:32,834 - INFO - [TRAIN] epoch=214/604, iter=24800/70000, loss=0.664526, lr=0.000125 | ETA 05:46:32 2022-07-29 21:26:51,975 - INFO - [TRAIN] epoch=215/604, iter=24850/70000, loss=0.634305, lr=0.000125 | ETA 04:00:11 2022-07-29 21:27:10,968 - INFO - [TRAIN] epoch=215/604, iter=24900/70000, loss=0.751767, lr=0.000125 | ETA 04:01:23 2022-07-29 21:27:30,109 - INFO - [TRAIN] epoch=216/604, iter=24950/70000, loss=0.687939, lr=0.000125 | ETA 03:56:04 2022-07-29 21:27:47,470 - INFO - [TRAIN] epoch=216/604, iter=25000/70000, loss=0.682674, lr=0.000125 | ETA 03:56:56 2022-07-29 21:27:48,824 - INFO - Push model to checkpoint ./output/iter_25000 2022-07-29 21:28:05,682 - INFO - [TRAIN] epoch=216/604, iter=25050/70000, loss=0.715110, lr=0.000125 | ETA 04:10:28 2022-07-29 21:28:26,169 - INFO - [TRAIN] epoch=217/604, iter=25100/70000, loss=0.690673, lr=0.000125 | ETA 04:13:35 2022-07-29 21:28:44,530 - INFO - [TRAIN] epoch=217/604, iter=25150/70000, loss=0.669658, lr=0.000125 | ETA 03:57:14 2022-07-29 21:29:03,103 - INFO - [TRAIN] epoch=218/604, iter=25200/70000, loss=0.695426, lr=0.000125 | ETA 04:10:48 2022-07-29 21:29:21,989 - INFO - [TRAIN] epoch=218/604, iter=25250/70000, loss=0.733162, lr=0.000125 | ETA 04:16:09 2022-07-29 21:29:40,433 - INFO - [TRAIN] epoch=219/604, iter=25300/70000, loss=0.702654, lr=0.000125 | ETA 03:55:30 2022-07-29 21:29:58,166 - INFO - [TRAIN] epoch=219/604, iter=25350/70000, loss=0.674212, lr=0.000125 | ETA 03:54:52 2022-07-29 21:30:16,279 - INFO - [TRAIN] epoch=219/604, iter=25400/70000, loss=0.728084, lr=0.000125 | ETA 04:09:21 2022-07-29 21:30:36,364 - INFO - [TRAIN] epoch=220/604, iter=25450/70000, loss=0.744128, lr=0.000125 | ETA 03:58:53 2022-07-29 21:30:53,954 - INFO - [TRAIN] epoch=220/604, iter=25500/70000, loss=0.642432, lr=0.000125 | ETA 03:51:57 2022-07-29 21:31:13,445 - INFO - [TRAIN] epoch=221/604, iter=25550/70000, loss=0.714510, lr=0.000125 | ETA 03:58:23 2022-07-29 21:31:30,948 - INFO - [TRAIN] epoch=221/604, iter=25600/70000, loss=0.676689, lr=0.000125 | ETA 03:58:36 2022-07-29 21:31:49,538 - INFO - [TRAIN] epoch=222/604, iter=25650/70000, loss=0.725256, lr=0.000125 | ETA 04:34:38 2022-07-29 21:32:07,351 - INFO - [TRAIN] epoch=222/604, iter=25700/70000, loss=0.767162, lr=0.000125 | ETA 03:55:29 2022-07-29 21:32:24,953 - INFO - [TRAIN] epoch=222/604, iter=25750/70000, loss=0.654670, lr=0.000125 | ETA 03:56:21 2022-07-29 21:32:44,705 - INFO - [TRAIN] epoch=223/604, iter=25800/70000, loss=0.701282, lr=0.000125 | ETA 05:17:06 2022-07-29 21:33:01,791 - INFO - [TRAIN] epoch=223/604, iter=25850/70000, loss=0.673683, lr=0.000125 | ETA 04:05:30 2022-07-29 21:33:21,782 - INFO - [TRAIN] epoch=224/604, iter=25900/70000, loss=0.753922, lr=0.000125 | ETA 04:56:24 2022-07-29 21:33:40,918 - INFO - [TRAIN] epoch=224/604, iter=25950/70000, loss=0.686218, lr=0.000125 | ETA 03:52:36 2022-07-29 21:33:59,807 - INFO - [TRAIN] epoch=225/604, iter=26000/70000, loss=0.687948, lr=0.000125 | ETA 03:50:35 2022-07-29 21:34:18,691 - INFO - [TRAIN] epoch=225/604, iter=26050/70000, loss=0.731369, lr=0.000125 | ETA 03:53:42 2022-07-29 21:34:36,446 - INFO - [TRAIN] epoch=225/604, iter=26100/70000, loss=0.678062, lr=0.000125 | ETA 03:47:58 2022-07-29 21:34:56,809 - INFO - [TRAIN] epoch=226/604, iter=26150/70000, loss=0.748489, lr=0.000125 | ETA 03:56:37 2022-07-29 21:35:14,401 - INFO - [TRAIN] epoch=226/604, iter=26200/70000, loss=0.688632, lr=0.000125 | ETA 04:02:45 2022-07-29 21:35:34,162 - INFO - [TRAIN] epoch=227/604, iter=26250/70000, loss=0.700735, lr=0.000125 | ETA 05:05:40 2022-07-29 21:35:51,606 - INFO - [TRAIN] epoch=227/604, iter=26300/70000, loss=0.689422, lr=0.000125 | ETA 03:56:46 2022-07-29 21:36:11,391 - INFO - [TRAIN] epoch=228/604, iter=26350/70000, loss=0.719330, lr=0.000125 | ETA 05:15:21 2022-07-29 21:36:28,977 - INFO - [TRAIN] epoch=228/604, iter=26400/70000, loss=0.695758, lr=0.000125 | ETA 03:48:54 2022-07-29 21:36:48,117 - INFO - [TRAIN] epoch=229/604, iter=26450/70000, loss=0.641005, lr=0.000125 | ETA 08:25:02 2022-07-29 21:37:05,731 - INFO - [TRAIN] epoch=229/604, iter=26500/70000, loss=0.658365, lr=0.000125 | ETA 03:46:05 2022-07-29 21:37:23,751 - INFO - [TRAIN] epoch=229/604, iter=26550/70000, loss=0.649731, lr=0.000125 | ETA 04:08:19 2022-07-29 21:37:43,281 - INFO - [TRAIN] epoch=230/604, iter=26600/70000, loss=0.696432, lr=0.000125 | ETA 03:59:35 2022-07-29 21:38:02,144 - INFO - [TRAIN] epoch=230/604, iter=26650/70000, loss=0.700195, lr=0.000125 | ETA 04:00:59 2022-07-29 21:38:20,602 - INFO - [TRAIN] epoch=231/604, iter=26700/70000, loss=0.759586, lr=0.000125 | ETA 03:52:24 2022-07-29 21:38:38,669 - INFO - [TRAIN] epoch=231/604, iter=26750/70000, loss=0.665184, lr=0.000125 | ETA 04:17:53 2022-07-29 21:38:58,085 - INFO - [TRAIN] epoch=232/604, iter=26800/70000, loss=0.727534, lr=0.000125 | ETA 05:12:30 2022-07-29 21:39:16,630 - INFO - [TRAIN] epoch=232/604, iter=26850/70000, loss=0.707926, lr=0.000125 | ETA 04:02:07 2022-07-29 21:39:35,487 - INFO - [TRAIN] epoch=232/604, iter=26900/70000, loss=0.735849, lr=0.000125 | ETA 04:05:46 2022-07-29 21:39:55,173 - INFO - [TRAIN] epoch=233/604, iter=26950/70000, loss=0.659435, lr=0.000125 | ETA 03:55:18 2022-07-29 21:40:13,226 - INFO - [TRAIN] epoch=233/604, iter=27000/70000, loss=0.671102, lr=0.000125 | ETA 04:13:10 2022-07-29 21:40:32,203 - INFO - [TRAIN] epoch=234/604, iter=27050/70000, loss=0.667875, lr=0.000125 | ETA 03:59:26 2022-07-29 21:40:50,058 - INFO - [TRAIN] epoch=234/604, iter=27100/70000, loss=0.678935, lr=0.000125 | ETA 03:56:37 2022-07-29 21:41:09,663 - INFO - [TRAIN] epoch=235/604, iter=27150/70000, loss=0.684777, lr=0.000125 | ETA 04:03:10 2022-07-29 21:41:27,737 - INFO - [TRAIN] epoch=235/604, iter=27200/70000, loss=0.598501, lr=0.000125 | ETA 03:46:47 2022-07-29 21:41:46,148 - INFO - [TRAIN] epoch=235/604, iter=27250/70000, loss=0.635058, lr=0.000125 | ETA 04:50:58 2022-07-29 21:42:05,427 - INFO - [TRAIN] epoch=236/604, iter=27300/70000, loss=0.671924, lr=0.000125 | ETA 03:58:18 2022-07-29 21:42:23,740 - INFO - [TRAIN] epoch=236/604, iter=27350/70000, loss=0.660373, lr=0.000125 | ETA 04:00:42 2022-07-29 21:42:42,940 - INFO - [TRAIN] epoch=237/604, iter=27400/70000, loss=0.737943, lr=0.000125 | ETA 03:53:57 2022-07-29 21:43:01,530 - INFO - [TRAIN] epoch=237/604, iter=27450/70000, loss=0.706806, lr=0.000125 | ETA 05:00:15 2022-07-29 21:43:21,699 - INFO - [TRAIN] epoch=238/604, iter=27500/70000, loss=0.702836, lr=0.000125 | ETA 06:58:00 2022-07-29 21:43:39,954 - INFO - [TRAIN] epoch=238/604, iter=27550/70000, loss=0.681602, lr=0.000125 | ETA 03:41:51 2022-07-29 21:43:59,984 - INFO - [TRAIN] epoch=238/604, iter=27600/70000, loss=0.702251, lr=0.000125 | ETA 09:00:51 2022-07-29 21:44:20,343 - INFO - [TRAIN] epoch=239/604, iter=27650/70000, loss=0.678915, lr=0.000125 | ETA 03:45:46 2022-07-29 21:44:38,597 - INFO - [TRAIN] epoch=239/604, iter=27700/70000, loss=0.683134, lr=0.000125 | ETA 05:35:16 2022-07-29 21:44:58,043 - INFO - [TRAIN] epoch=240/604, iter=27750/70000, loss=0.661361, lr=0.000125 | ETA 03:57:18 2022-07-29 21:45:16,273 - INFO - [TRAIN] epoch=240/604, iter=27800/70000, loss=0.659141, lr=0.000125 | ETA 03:43:11 2022-07-29 21:45:35,398 - INFO - [TRAIN] epoch=241/604, iter=27850/70000, loss=0.693105, lr=0.000125 | ETA 03:42:43 2022-07-29 21:45:53,852 - INFO - [TRAIN] epoch=241/604, iter=27900/70000, loss=0.699700, lr=0.000125 | ETA 05:59:00 2022-07-29 21:46:12,751 - INFO - [TRAIN] epoch=241/604, iter=27950/70000, loss=0.689675, lr=0.000125 | ETA 03:46:58 2022-07-29 21:46:33,526 - INFO - [TRAIN] epoch=242/604, iter=28000/70000, loss=0.750307, lr=0.000125 | ETA 03:41:36 2022-07-29 21:46:51,758 - INFO - [TRAIN] epoch=242/604, iter=28050/70000, loss=0.732028, lr=0.000125 | ETA 03:46:06 2022-07-29 21:47:11,323 - INFO - [TRAIN] epoch=243/604, iter=28100/70000, loss=0.710726, lr=0.000125 | ETA 03:46:20 2022-07-29 21:47:27,893 - INFO - [TRAIN] epoch=243/604, iter=28150/70000, loss=0.655298, lr=0.000125 | ETA 03:53:56 2022-07-29 21:47:45,569 - INFO - [TRAIN] epoch=244/604, iter=28200/70000, loss=0.662585, lr=0.000125 | ETA 03:52:37 2022-07-29 21:48:01,632 - INFO - [TRAIN] epoch=244/604, iter=28250/70000, loss=0.740501, lr=0.000125 | ETA 03:42:41 2022-07-29 21:48:17,562 - INFO - [TRAIN] epoch=244/604, iter=28300/70000, loss=0.664490, lr=0.000125 | ETA 03:37:05 2022-07-29 21:48:35,035 - INFO - [TRAIN] epoch=245/604, iter=28350/70000, loss=0.710751, lr=0.000125 | ETA 03:43:27 2022-07-29 21:48:51,002 - INFO - [TRAIN] epoch=245/604, iter=28400/70000, loss=0.656828, lr=0.000125 | ETA 03:49:59 2022-07-29 21:49:08,419 - INFO - [TRAIN] epoch=246/604, iter=28450/70000, loss=0.673504, lr=0.000125 | ETA 03:42:02 2022-07-29 21:49:24,503 - INFO - [TRAIN] epoch=246/604, iter=28500/70000, loss=0.668427, lr=0.000125 | ETA 03:36:29 2022-07-29 21:49:42,082 - INFO - [TRAIN] epoch=247/604, iter=28550/70000, loss=0.672699, lr=0.000125 | ETA 03:40:55 2022-07-29 21:49:58,065 - INFO - [TRAIN] epoch=247/604, iter=28600/70000, loss=0.641022, lr=0.000125 | ETA 03:39:08 2022-07-29 21:50:13,971 - INFO - [TRAIN] epoch=247/604, iter=28650/70000, loss=0.685032, lr=0.000125 | ETA 03:36:56 2022-07-29 21:50:31,331 - INFO - [TRAIN] epoch=248/604, iter=28700/70000, loss=0.670666, lr=0.000125 | ETA 03:37:24 2022-07-29 21:50:47,263 - INFO - [TRAIN] epoch=248/604, iter=28750/70000, loss=0.689284, lr=0.000125 | ETA 03:35:28 2022-07-29 21:51:04,797 - INFO - [TRAIN] epoch=249/604, iter=28800/70000, loss=0.691216, lr=0.000125 | ETA 03:44:09 2022-07-29 21:51:20,909 - INFO - [TRAIN] epoch=249/604, iter=28850/70000, loss=0.613046, lr=0.000125 | ETA 03:44:32 2022-07-29 21:51:38,376 - INFO - [TRAIN] epoch=250/604, iter=28900/70000, loss=0.661902, lr=0.000125 | ETA 03:49:06 2022-07-29 21:51:54,462 - INFO - [TRAIN] epoch=250/604, iter=28950/70000, loss=0.713144, lr=0.000125 | ETA 03:37:42 2022-07-29 21:52:10,725 - INFO - [TRAIN] epoch=250/604, iter=29000/70000, loss=0.677684, lr=0.000125 | ETA 03:34:04 2022-07-29 21:52:28,830 - INFO - [TRAIN] epoch=251/604, iter=29050/70000, loss=0.624557, lr=0.000125 | ETA 03:34:49 2022-07-29 21:52:45,374 - INFO - [TRAIN] epoch=251/604, iter=29100/70000, loss=0.641985, lr=0.000125 | ETA 03:48:40 2022-07-29 21:53:03,531 - INFO - [TRAIN] epoch=252/604, iter=29150/70000, loss=0.635187, lr=0.000125 | ETA 03:37:38 2022-07-29 21:53:19,903 - INFO - [TRAIN] epoch=252/604, iter=29200/70000, loss=0.675505, lr=0.000125 | ETA 03:48:29 2022-07-29 21:53:38,041 - INFO - [TRAIN] epoch=253/604, iter=29250/70000, loss=0.645829, lr=0.000125 | ETA 03:32:32 2022-07-29 21:53:54,831 - INFO - [TRAIN] epoch=253/604, iter=29300/70000, loss=0.640391, lr=0.000125 | ETA 03:32:47 2022-07-29 21:54:13,501 - INFO - [TRAIN] epoch=254/604, iter=29350/70000, loss=0.738545, lr=0.000125 | ETA 08:03:54 2022-07-29 21:54:30,544 - INFO - [TRAIN] epoch=254/604, iter=29400/70000, loss=0.710121, lr=0.000125 | ETA 03:40:14 2022-07-29 21:54:47,959 - INFO - [TRAIN] epoch=254/604, iter=29450/70000, loss=0.653146, lr=0.000125 | ETA 03:55:56 2022-07-29 21:55:06,540 - INFO - [TRAIN] epoch=255/604, iter=29500/70000, loss=0.727917, lr=0.000125 | ETA 03:49:41 2022-07-29 21:55:23,412 - INFO - [TRAIN] epoch=255/604, iter=29550/70000, loss=0.634460, lr=0.000125 | ETA 03:58:37 2022-07-29 21:55:41,377 - INFO - [TRAIN] epoch=256/604, iter=29600/70000, loss=0.637970, lr=0.000125 | ETA 04:04:24 2022-07-29 21:55:58,357 - INFO - [TRAIN] epoch=256/604, iter=29650/70000, loss=0.593476, lr=0.000125 | ETA 03:41:22 2022-07-29 21:56:16,655 - INFO - [TRAIN] epoch=257/604, iter=29700/70000, loss=0.643949, lr=0.000125 | ETA 04:47:08 2022-07-29 21:56:33,372 - INFO - [TRAIN] epoch=257/604, iter=29750/70000, loss=0.643215, lr=0.000125 | ETA 03:31:03 2022-07-29 21:56:50,340 - INFO - [TRAIN] epoch=257/604, iter=29800/70000, loss=0.707383, lr=0.000125 | ETA 03:44:42 2022-07-29 21:57:08,171 - INFO - [TRAIN] epoch=258/604, iter=29850/70000, loss=0.631654, lr=0.000125 | ETA 03:30:47 2022-07-29 21:57:24,686 - INFO - [TRAIN] epoch=258/604, iter=29900/70000, loss=0.669177, lr=0.000125 | ETA 03:29:51 2022-07-29 21:57:43,769 - INFO - [TRAIN] epoch=259/604, iter=29950/70000, loss=0.694006, lr=0.000125 | ETA 03:32:00 2022-07-29 21:58:01,381 - INFO - [TRAIN] epoch=259/604, iter=30000/70000, loss=0.627059, lr=0.000125 | ETA 03:34:12 2022-07-29 21:58:01,393 - INFO - Pop model from ./output/iter_5000 2022-07-29 21:58:02,461 - INFO - Push model to checkpoint ./output/iter_30000 2022-07-29 21:58:20,807 - INFO - [TRAIN] epoch=260/604, iter=30050/70000, loss=0.632971, lr=0.000125 | ETA 03:53:59 2022-07-29 21:58:37,623 - INFO - [TRAIN] epoch=260/604, iter=30100/70000, loss=0.686020, lr=0.000125 | ETA 03:43:06 2022-07-29 21:58:54,169 - INFO - [TRAIN] epoch=260/604, iter=30150/70000, loss=0.654036, lr=0.000125 | ETA 04:36:06 2022-07-29 21:59:12,159 - INFO - [TRAIN] epoch=261/604, iter=30200/70000, loss=0.641532, lr=0.000125 | ETA 03:42:39 2022-07-29 21:59:29,038 - INFO - [TRAIN] epoch=261/604, iter=30250/70000, loss=0.639749, lr=0.000125 | ETA 04:08:35 2022-07-29 21:59:48,215 - INFO - [TRAIN] epoch=262/604, iter=30300/70000, loss=0.684688, lr=0.000125 | ETA 03:30:21 2022-07-29 22:00:04,856 - INFO - [TRAIN] epoch=262/604, iter=30350/70000, loss=0.613212, lr=0.000125 | ETA 03:45:53 2022-07-29 22:00:23,309 - INFO - [TRAIN] epoch=263/604, iter=30400/70000, loss=0.655179, lr=0.000125 | ETA 03:34:19 2022-07-29 22:00:40,921 - INFO - [TRAIN] epoch=263/604, iter=30450/70000, loss=0.611166, lr=0.000125 | ETA 03:35:07 2022-07-29 22:00:57,681 - INFO - [TRAIN] epoch=263/604, iter=30500/70000, loss=0.668437, lr=0.000125 | ETA 03:36:32 2022-07-29 22:01:16,758 - INFO - [TRAIN] epoch=264/604, iter=30550/70000, loss=0.672825, lr=0.000125 | ETA 03:31:58 2022-07-29 22:01:33,615 - INFO - [TRAIN] epoch=264/604, iter=30600/70000, loss=0.669389, lr=0.000125 | ETA 03:28:54 2022-07-29 22:01:52,849 - INFO - [TRAIN] epoch=265/604, iter=30650/70000, loss=0.614531, lr=0.000125 | ETA 03:28:12 2022-07-29 22:02:10,018 - INFO - [TRAIN] epoch=265/604, iter=30700/70000, loss=0.605660, lr=0.000125 | ETA 04:32:59 2022-07-29 22:02:28,032 - INFO - [TRAIN] epoch=266/604, iter=30750/70000, loss=0.637636, lr=0.000125 | ETA 03:29:58 2022-07-29 22:02:45,827 - INFO - [TRAIN] epoch=266/604, iter=30800/70000, loss=0.645966, lr=0.000125 | ETA 04:25:50 2022-07-29 22:03:02,383 - INFO - [TRAIN] epoch=266/604, iter=30850/70000, loss=0.676252, lr=0.000125 | ETA 03:25:37 2022-07-29 22:03:21,447 - INFO - [TRAIN] epoch=267/604, iter=30900/70000, loss=0.662071, lr=0.000125 | ETA 03:35:41 2022-07-29 22:03:38,654 - INFO - [TRAIN] epoch=267/604, iter=30950/70000, loss=0.612388, lr=0.000125 | ETA 03:24:25 2022-07-29 22:03:57,026 - INFO - [TRAIN] epoch=268/604, iter=31000/70000, loss=0.639451, lr=0.000125 | ETA 03:24:26 2022-07-29 22:04:13,998 - INFO - [TRAIN] epoch=268/604, iter=31050/70000, loss=0.629760, lr=0.000125 | ETA 03:33:42 2022-07-29 22:04:33,281 - INFO - [TRAIN] epoch=269/604, iter=31100/70000, loss=0.660501, lr=0.000125 | ETA 03:23:15 2022-07-29 22:04:50,715 - INFO - [TRAIN] epoch=269/604, iter=31150/70000, loss=0.575916, lr=0.000125 | ETA 03:27:23 2022-07-29 22:05:07,942 - INFO - [TRAIN] epoch=269/604, iter=31200/70000, loss=0.646019, lr=0.000125 | ETA 03:23:32 2022-07-29 22:05:26,206 - INFO - [TRAIN] epoch=270/604, iter=31250/70000, loss=0.580429, lr=0.000125 | ETA 03:38:16 2022-07-29 22:05:43,369 - INFO - [TRAIN] epoch=270/604, iter=31300/70000, loss=0.702326, lr=0.000125 | ETA 03:40:53 2022-07-29 22:06:01,348 - INFO - [TRAIN] epoch=271/604, iter=31350/70000, loss=0.601857, lr=0.000125 | ETA 03:23:15 2022-07-29 22:06:18,841 - INFO - [TRAIN] epoch=271/604, iter=31400/70000, loss=0.590295, lr=0.000125 | ETA 04:30:07 2022-07-29 22:06:36,788 - INFO - [TRAIN] epoch=272/604, iter=31450/70000, loss=0.620887, lr=0.000125 | ETA 03:55:08 2022-07-29 22:06:53,117 - INFO - [TRAIN] epoch=272/604, iter=31500/70000, loss=0.626381, lr=0.000125 | ETA 03:27:27 2022-07-29 22:07:09,181 - INFO - [TRAIN] epoch=272/604, iter=31550/70000, loss=0.709360, lr=0.000125 | ETA 03:24:26 2022-07-29 22:07:27,452 - INFO - [TRAIN] epoch=273/604, iter=31600/70000, loss=0.682510, lr=0.000125 | ETA 03:34:29 2022-07-29 22:07:43,759 - INFO - [TRAIN] epoch=273/604, iter=31650/70000, loss=0.560144, lr=0.000125 | ETA 03:21:40 2022-07-29 22:08:02,889 - INFO - [TRAIN] epoch=274/604, iter=31700/70000, loss=0.604537, lr=0.000125 | ETA 03:32:40 2022-07-29 22:08:20,270 - INFO - [TRAIN] epoch=274/604, iter=31750/70000, loss=0.666855, lr=0.000125 | ETA 03:26:16 2022-07-29 22:08:39,134 - INFO - [TRAIN] epoch=275/604, iter=31800/70000, loss=0.678751, lr=0.000125 | ETA 04:17:53 2022-07-29 22:08:56,129 - INFO - [TRAIN] epoch=275/604, iter=31850/70000, loss=0.603802, lr=0.000125 | ETA 03:19:37 2022-07-29 22:09:12,806 - INFO - [TRAIN] epoch=275/604, iter=31900/70000, loss=0.594711, lr=0.000125 | ETA 03:18:29 2022-07-29 22:09:31,824 - INFO - [TRAIN] epoch=276/604, iter=31950/70000, loss=0.595028, lr=0.000125 | ETA 03:19:01 2022-07-29 22:09:50,319 - INFO - [TRAIN] epoch=276/604, iter=32000/70000, loss=0.680932, lr=0.000125 | ETA 03:33:11 2022-07-29 22:10:09,674 - INFO - [TRAIN] epoch=277/604, iter=32050/70000, loss=0.600889, lr=0.000125 | ETA 03:24:36 2022-07-29 22:10:26,789 - INFO - [TRAIN] epoch=277/604, iter=32100/70000, loss=0.595361, lr=0.000125 | ETA 03:22:18 2022-07-29 22:10:45,043 - INFO - [TRAIN] epoch=278/604, iter=32150/70000, loss=0.688480, lr=0.000125 | ETA 03:27:09 2022-07-29 22:11:02,138 - INFO - [TRAIN] epoch=278/604, iter=32200/70000, loss=0.592260, lr=0.000125 | ETA 03:17:40 2022-07-29 22:11:21,017 - INFO - [TRAIN] epoch=279/604, iter=32250/70000, loss=0.627886, lr=0.000125 | ETA 07:42:19 2022-07-29 22:11:39,212 - INFO - [TRAIN] epoch=279/604, iter=32300/70000, loss=0.625588, lr=0.000125 | ETA 03:23:22 2022-07-29 22:11:56,505 - INFO - [TRAIN] epoch=279/604, iter=32350/70000, loss=0.645825, lr=0.000125 | ETA 03:46:38 2022-07-29 22:12:14,980 - INFO - [TRAIN] epoch=280/604, iter=32400/70000, loss=0.612762, lr=0.000125 | ETA 03:21:36 2022-07-29 22:12:32,367 - INFO - [TRAIN] epoch=280/604, iter=32450/70000, loss=0.561098, lr=0.000125 | ETA 03:15:45 2022-07-29 22:12:50,747 - INFO - [TRAIN] epoch=281/604, iter=32500/70000, loss=0.592408, lr=0.000125 | ETA 03:34:19 2022-07-29 22:13:08,361 - INFO - [TRAIN] epoch=281/604, iter=32550/70000, loss=0.623407, lr=0.000125 | ETA 03:21:59 2022-07-29 22:13:27,057 - INFO - [TRAIN] epoch=282/604, iter=32600/70000, loss=0.631808, lr=0.000125 | ETA 04:25:41 2022-07-29 22:13:44,002 - INFO - [TRAIN] epoch=282/604, iter=32650/70000, loss=0.650387, lr=0.000125 | ETA 03:14:58 2022-07-29 22:14:01,989 - INFO - [TRAIN] epoch=282/604, iter=32700/70000, loss=0.539783, lr=0.000125 | ETA 03:42:26 2022-07-29 22:14:20,803 - INFO - [TRAIN] epoch=283/604, iter=32750/70000, loss=0.686600, lr=0.000125 | ETA 07:36:18 2022-07-29 22:14:37,212 - INFO - [TRAIN] epoch=283/604, iter=32800/70000, loss=0.655952, lr=0.000125 | ETA 03:12:42 2022-07-29 22:14:55,016 - INFO - [TRAIN] epoch=284/604, iter=32850/70000, loss=0.672665, lr=0.000125 | ETA 03:14:20 2022-07-29 22:15:12,095 - INFO - [TRAIN] epoch=284/604, iter=32900/70000, loss=0.716905, lr=0.000125 | ETA 03:20:07 2022-07-29 22:15:29,971 - INFO - [TRAIN] epoch=285/604, iter=32950/70000, loss=0.631643, lr=0.000125 | ETA 03:35:06 2022-07-29 22:15:47,852 - INFO - [TRAIN] epoch=285/604, iter=33000/70000, loss=0.709041, lr=0.000125 | ETA 03:26:33 2022-07-29 22:16:05,096 - INFO - [TRAIN] epoch=285/604, iter=33050/70000, loss=0.608312, lr=0.000125 | ETA 05:21:30 2022-07-29 22:16:23,871 - INFO - [TRAIN] epoch=286/604, iter=33100/70000, loss=0.588813, lr=0.000125 | ETA 03:15:53 2022-07-29 22:16:40,483 - INFO - [TRAIN] epoch=286/604, iter=33150/70000, loss=0.649496, lr=0.000125 | ETA 04:06:13 2022-07-29 22:16:58,729 - INFO - [TRAIN] epoch=287/604, iter=33200/70000, loss=0.609204, lr=0.000125 | ETA 03:15:57 2022-07-29 22:17:15,703 - INFO - [TRAIN] epoch=287/604, iter=33250/70000, loss=0.595947, lr=0.000125 | ETA 03:31:02 2022-07-29 22:17:33,789 - INFO - [TRAIN] epoch=288/604, iter=33300/70000, loss=0.644489, lr=0.000125 | ETA 03:20:15 2022-07-29 22:17:51,068 - INFO - [TRAIN] epoch=288/604, iter=33350/70000, loss=0.591960, lr=0.000125 | ETA 03:12:06 2022-07-29 22:18:07,741 - INFO - [TRAIN] epoch=288/604, iter=33400/70000, loss=0.630090, lr=0.000125 | ETA 03:14:51 2022-07-29 22:18:27,543 - INFO - [TRAIN] epoch=289/604, iter=33450/70000, loss=0.642068, lr=0.000125 | ETA 03:09:45 2022-07-29 22:18:46,034 - INFO - [TRAIN] epoch=289/604, iter=33500/70000, loss=0.602110, lr=0.000125 | ETA 04:26:50 2022-07-29 22:19:04,664 - INFO - [TRAIN] epoch=290/604, iter=33550/70000, loss=0.642006, lr=0.000125 | ETA 03:20:18 2022-07-29 22:19:22,763 - INFO - [TRAIN] epoch=290/604, iter=33600/70000, loss=0.628793, lr=0.000125 | ETA 03:16:27 2022-07-29 22:19:41,218 - INFO - [TRAIN] epoch=291/604, iter=33650/70000, loss=0.567665, lr=0.000125 | ETA 03:45:49 2022-07-29 22:19:59,318 - INFO - [TRAIN] epoch=291/604, iter=33700/70000, loss=0.654469, lr=0.000125 | ETA 03:14:16 2022-07-29 22:20:15,981 - INFO - [TRAIN] epoch=291/604, iter=33750/70000, loss=0.597520, lr=0.000125 | ETA 03:12:11 2022-07-29 22:20:35,164 - INFO - [TRAIN] epoch=292/604, iter=33800/70000, loss=0.581737, lr=0.000125 | ETA 03:14:10 2022-07-29 22:20:52,304 - INFO - [TRAIN] epoch=292/604, iter=33850/70000, loss=0.593761, lr=0.000125 | ETA 04:15:10 2022-07-29 22:21:11,000 - INFO - [TRAIN] epoch=293/604, iter=33900/70000, loss=0.592504, lr=0.000125 | ETA 03:22:06 2022-07-29 22:21:28,875 - INFO - [TRAIN] epoch=293/604, iter=33950/70000, loss=0.619492, lr=0.000125 | ETA 03:19:03 2022-07-29 22:21:47,482 - INFO - [TRAIN] epoch=294/604, iter=34000/70000, loss=0.608888, lr=0.000125 | ETA 03:10:29 2022-07-29 22:22:06,329 - INFO - [TRAIN] epoch=294/604, iter=34050/70000, loss=0.583710, lr=0.000125 | ETA 05:18:01 2022-07-29 22:22:23,260 - INFO - [TRAIN] epoch=294/604, iter=34100/70000, loss=0.621633, lr=0.000125 | ETA 03:17:23 2022-07-29 22:22:43,060 - INFO - [TRAIN] epoch=295/604, iter=34150/70000, loss=0.635781, lr=0.000125 | ETA 03:16:49 2022-07-29 22:23:00,636 - INFO - [TRAIN] epoch=295/604, iter=34200/70000, loss=0.648806, lr=0.000125 | ETA 04:14:00 2022-07-29 22:23:19,674 - INFO - [TRAIN] epoch=296/604, iter=34250/70000, loss=0.594064, lr=0.000125 | ETA 03:12:31 2022-07-29 22:23:37,053 - INFO - [TRAIN] epoch=296/604, iter=34300/70000, loss=0.618976, lr=0.000125 | ETA 03:12:04 2022-07-29 22:23:55,580 - INFO - [TRAIN] epoch=297/604, iter=34350/70000, loss=0.611470, lr=0.000125 | ETA 03:10:32 2022-07-29 22:24:13,522 - INFO - [TRAIN] epoch=297/604, iter=34400/70000, loss=0.653486, lr=0.000125 | ETA 03:12:47 2022-07-29 22:24:30,411 - INFO - [TRAIN] epoch=297/604, iter=34450/70000, loss=0.557706, lr=0.000125 | ETA 03:08:42 2022-07-29 22:24:49,228 - INFO - [TRAIN] epoch=298/604, iter=34500/70000, loss=0.557214, lr=0.000125 | ETA 03:11:15 2022-07-29 22:25:06,285 - INFO - [TRAIN] epoch=298/604, iter=34550/70000, loss=0.577632, lr=0.000125 | ETA 03:11:47 2022-07-29 22:25:26,026 - INFO - [TRAIN] epoch=299/604, iter=34600/70000, loss=0.626729, lr=0.000125 | ETA 03:08:27 2022-07-29 22:25:44,097 - INFO - [TRAIN] epoch=299/604, iter=34650/70000, loss=0.600698, lr=0.000125 | ETA 03:05:49 2022-07-29 22:26:02,288 - INFO - [TRAIN] epoch=300/604, iter=34700/70000, loss=0.598541, lr=0.000125 | ETA 03:11:23 2022-07-29 22:26:19,717 - INFO - [TRAIN] epoch=300/604, iter=34750/70000, loss=0.590702, lr=0.000125 | ETA 03:43:20 2022-07-29 22:26:36,640 - INFO - [TRAIN] epoch=300/604, iter=34800/70000, loss=0.570829, lr=0.000125 | ETA 03:04:46 2022-07-29 22:26:55,572 - INFO - [TRAIN] epoch=301/604, iter=34850/70000, loss=0.658197, lr=0.000125 | ETA 04:50:39 2022-07-29 22:27:12,455 - INFO - [TRAIN] epoch=301/604, iter=34900/70000, loss=0.590724, lr=0.000125 | ETA 03:04:13 2022-07-29 22:27:31,071 - INFO - [TRAIN] epoch=302/604, iter=34950/70000, loss=0.571401, lr=0.000125 | ETA 03:08:50 2022-07-29 22:27:49,712 - INFO - [TRAIN] epoch=302/604, iter=35000/70000, loss=0.571270, lr=0.000125 | ETA 05:51:05 2022-07-29 22:27:49,726 - INFO - Pop model from ./output/iter_10000 2022-07-29 22:27:51,032 - INFO - Push model to checkpoint ./output/iter_35000 2022-07-29 22:28:10,571 - INFO - [TRAIN] epoch=303/604, iter=35050/70000, loss=0.571215, lr=0.000125 | ETA 03:51:06 2022-07-29 22:28:28,248 - INFO - [TRAIN] epoch=303/604, iter=35100/70000, loss=0.557640, lr=0.000125 | ETA 03:11:38 2022-07-29 22:28:47,144 - INFO - [TRAIN] epoch=304/604, iter=35150/70000, loss=0.615996, lr=0.000125 | ETA 06:54:31 2022-07-29 22:29:04,704 - INFO - [TRAIN] epoch=304/604, iter=35200/70000, loss=0.619779, lr=0.000125 | ETA 03:02:07 2022-07-29 22:29:21,645 - INFO - [TRAIN] epoch=304/604, iter=35250/70000, loss=0.574040, lr=0.000125 | ETA 03:01:14 2022-07-29 22:29:40,267 - INFO - [TRAIN] epoch=305/604, iter=35300/70000, loss=0.603569, lr=0.000125 | ETA 03:07:44 2022-07-29 22:29:58,299 - INFO - [TRAIN] epoch=305/604, iter=35350/70000, loss=0.579895, lr=0.000125 | ETA 03:13:04 2022-07-29 22:30:17,379 - INFO - [TRAIN] epoch=306/604, iter=35400/70000, loss=0.565913, lr=0.000125 | ETA 03:07:57 2022-07-29 22:30:35,102 - INFO - [TRAIN] epoch=306/604, iter=35450/70000, loss=0.573673, lr=0.000125 | ETA 03:23:19 2022-07-29 22:30:53,909 - INFO - [TRAIN] epoch=307/604, iter=35500/70000, loss=0.567743, lr=0.000125 | ETA 04:01:41 2022-07-29 22:31:12,722 - INFO - [TRAIN] epoch=307/604, iter=35550/70000, loss=0.582716, lr=0.000125 | ETA 03:13:12 2022-07-29 22:31:28,693 - INFO - [TRAIN] epoch=307/604, iter=35600/70000, loss=0.572571, lr=0.000125 | ETA 03:02:16 2022-07-29 22:31:48,373 - INFO - [TRAIN] epoch=308/604, iter=35650/70000, loss=0.590748, lr=0.000125 | ETA 03:59:50 2022-07-29 22:32:05,441 - INFO - [TRAIN] epoch=308/604, iter=35700/70000, loss=0.557847, lr=0.000125 | ETA 03:21:50 2022-07-29 22:32:24,803 - INFO - [TRAIN] epoch=309/604, iter=35750/70000, loss=0.638232, lr=0.000125 | ETA 03:02:57 2022-07-29 22:32:42,448 - INFO - [TRAIN] epoch=309/604, iter=35800/70000, loss=0.673122, lr=0.000125 | ETA 02:58:09 2022-07-29 22:33:00,827 - INFO - [TRAIN] epoch=310/604, iter=35850/70000, loss=0.582204, lr=0.000125 | ETA 03:21:11 2022-07-29 22:33:18,835 - INFO - [TRAIN] epoch=310/604, iter=35900/70000, loss=0.627627, lr=0.000125 | ETA 02:58:07 2022-07-29 22:33:36,528 - INFO - [TRAIN] epoch=310/604, iter=35950/70000, loss=0.598335, lr=0.000125 | ETA 03:03:37 2022-07-29 22:33:54,963 - INFO - [TRAIN] epoch=311/604, iter=36000/70000, loss=0.621827, lr=0.000013 | ETA 03:03:01 2022-07-29 22:34:12,720 - INFO - [TRAIN] epoch=311/604, iter=36050/70000, loss=0.487072, lr=0.000013 | ETA 03:01:10 2022-07-29 22:34:32,160 - INFO - [TRAIN] epoch=312/604, iter=36100/70000, loss=0.522092, lr=0.000013 | ETA 03:04:47 2022-07-29 22:34:49,582 - INFO - [TRAIN] epoch=312/604, iter=36150/70000, loss=0.437186, lr=0.000013 | ETA 03:05:59 2022-07-29 22:35:07,851 - INFO - [TRAIN] epoch=313/604, iter=36200/70000, loss=0.504683, lr=0.000013 | ETA 03:07:08 2022-07-29 22:35:25,233 - INFO - [TRAIN] epoch=313/604, iter=36250/70000, loss=0.475775, lr=0.000013 | ETA 03:14:26 2022-07-29 22:35:42,785 - INFO - [TRAIN] epoch=313/604, iter=36300/70000, loss=0.449803, lr=0.000013 | ETA 03:02:32 2022-07-29 22:36:01,305 - INFO - [TRAIN] epoch=314/604, iter=36350/70000, loss=0.417356, lr=0.000013 | ETA 02:58:19 2022-07-29 22:36:20,136 - INFO - [TRAIN] epoch=314/604, iter=36400/70000, loss=0.466106, lr=0.000013 | ETA 03:16:28 2022-07-29 22:36:38,980 - INFO - [TRAIN] epoch=315/604, iter=36450/70000, loss=0.460537, lr=0.000013 | ETA 03:25:37 2022-07-29 22:36:55,618 - INFO - [TRAIN] epoch=315/604, iter=36500/70000, loss=0.433898, lr=0.000013 | ETA 02:55:46 2022-07-29 22:37:15,149 - INFO - [TRAIN] epoch=316/604, iter=36550/70000, loss=0.447303, lr=0.000013 | ETA 03:02:24 2022-07-29 22:37:32,455 - INFO - [TRAIN] epoch=316/604, iter=36600/70000, loss=0.443283, lr=0.000013 | ETA 02:54:24 2022-07-29 22:37:49,856 - INFO - [TRAIN] epoch=316/604, iter=36650/70000, loss=0.413713, lr=0.000013 | ETA 03:38:41 2022-07-29 22:38:10,025 - INFO - [TRAIN] epoch=317/604, iter=36700/70000, loss=0.384333, lr=0.000013 | ETA 04:22:55 2022-07-29 22:38:26,115 - INFO - [TRAIN] epoch=317/604, iter=36750/70000, loss=0.410290, lr=0.000013 | ETA 02:57:57 2022-07-29 22:38:45,362 - INFO - [TRAIN] epoch=318/604, iter=36800/70000, loss=0.408707, lr=0.000013 | ETA 03:01:35 2022-07-29 22:39:03,169 - INFO - [TRAIN] epoch=318/604, iter=36850/70000, loss=0.412943, lr=0.000013 | ETA 02:56:00 2022-07-29 22:39:22,291 - INFO - [TRAIN] epoch=319/604, iter=36900/70000, loss=0.433471, lr=0.000013 | ETA 04:05:01 2022-07-29 22:39:40,318 - INFO - [TRAIN] epoch=319/604, iter=36950/70000, loss=0.377575, lr=0.000013 | ETA 03:01:28 2022-07-29 22:39:58,484 - INFO - [TRAIN] epoch=319/604, iter=37000/70000, loss=0.424142, lr=0.000013 | ETA 03:31:07 2022-07-29 22:40:17,141 - INFO - [TRAIN] epoch=320/604, iter=37050/70000, loss=0.414358, lr=0.000013 | ETA 02:55:59 2022-07-29 22:40:34,849 - INFO - [TRAIN] epoch=320/604, iter=37100/70000, loss=0.400548, lr=0.000013 | ETA 03:25:22 2022-07-29 22:40:53,505 - INFO - [TRAIN] epoch=321/604, iter=37150/70000, loss=0.389643, lr=0.000013 | ETA 02:53:51 2022-07-29 22:41:11,761 - INFO - [TRAIN] epoch=321/604, iter=37200/70000, loss=0.399215, lr=0.000013 | ETA 05:05:41 2022-07-29 22:41:31,680 - INFO - [TRAIN] epoch=322/604, iter=37250/70000, loss=0.395443, lr=0.000013 | ETA 05:22:57 2022-07-29 22:41:49,332 - INFO - [TRAIN] epoch=322/604, iter=37300/70000, loss=0.388171, lr=0.000013 | ETA 03:14:09 2022-07-29 22:42:05,928 - INFO - [TRAIN] epoch=322/604, iter=37350/70000, loss=0.408518, lr=0.000013 | ETA 02:56:00 2022-07-29 22:42:26,253 - INFO - [TRAIN] epoch=323/604, iter=37400/70000, loss=0.385718, lr=0.000013 | ETA 02:58:05 2022-07-29 22:42:43,386 - INFO - [TRAIN] epoch=323/604, iter=37450/70000, loss=0.360693, lr=0.000013 | ETA 02:50:40 2022-07-29 22:43:02,728 - INFO - [TRAIN] epoch=324/604, iter=37500/70000, loss=0.361302, lr=0.000013 | ETA 02:56:50 2022-07-29 22:43:20,440 - INFO - [TRAIN] epoch=324/604, iter=37550/70000, loss=0.377395, lr=0.000013 | ETA 03:05:49 2022-07-29 22:43:39,305 - INFO - [TRAIN] epoch=325/604, iter=37600/70000, loss=0.417695, lr=0.000013 | ETA 02:56:52 2022-07-29 22:43:57,015 - INFO - [TRAIN] epoch=325/604, iter=37650/70000, loss=0.363408, lr=0.000013 | ETA 03:13:27 2022-07-29 22:44:13,806 - INFO - [TRAIN] epoch=325/604, iter=37700/70000, loss=0.390300, lr=0.000013 | ETA 02:58:28 2022-07-29 22:44:33,030 - INFO - [TRAIN] epoch=326/604, iter=37750/70000, loss=0.349948, lr=0.000013 | ETA 02:51:37 2022-07-29 22:44:50,280 - INFO - [TRAIN] epoch=326/604, iter=37800/70000, loss=0.375034, lr=0.000013 | ETA 02:48:00 2022-07-29 22:45:09,241 - INFO - [TRAIN] epoch=327/604, iter=37850/70000, loss=0.341101, lr=0.000013 | ETA 03:01:36 2022-07-29 22:45:27,448 - INFO - [TRAIN] epoch=327/604, iter=37900/70000, loss=0.356861, lr=0.000013 | ETA 03:12:31 2022-07-29 22:45:46,178 - INFO - [TRAIN] epoch=328/604, iter=37950/70000, loss=0.375093, lr=0.000013 | ETA 04:27:51 2022-07-29 22:46:04,228 - INFO - [TRAIN] epoch=328/604, iter=38000/70000, loss=0.340733, lr=0.000013 | ETA 03:12:51 2022-07-29 22:46:22,867 - INFO - [TRAIN] epoch=329/604, iter=38050/70000, loss=0.352794, lr=0.000013 | ETA 06:13:53 2022-07-29 22:46:40,613 - INFO - [TRAIN] epoch=329/604, iter=38100/70000, loss=0.356276, lr=0.000013 | ETA 03:35:46 2022-07-29 22:46:56,976 - INFO - [TRAIN] epoch=329/604, iter=38150/70000, loss=0.375562, lr=0.000013 | ETA 02:54:43 2022-07-29 22:47:16,375 - INFO - [TRAIN] epoch=330/604, iter=38200/70000, loss=0.375063, lr=0.000013 | ETA 03:41:19 2022-07-29 22:47:34,438 - INFO - [TRAIN] epoch=330/604, iter=38250/70000, loss=0.369713, lr=0.000013 | ETA 02:46:11 2022-07-29 22:47:52,778 - INFO - [TRAIN] epoch=331/604, iter=38300/70000, loss=0.396406, lr=0.000013 | ETA 02:52:40 2022-07-29 22:48:09,535 - INFO - [TRAIN] epoch=331/604, iter=38350/70000, loss=0.349613, lr=0.000013 | ETA 02:58:42 2022-07-29 22:48:28,145 - INFO - [TRAIN] epoch=332/604, iter=38400/70000, loss=0.336692, lr=0.000013 | ETA 03:50:23 2022-07-29 22:48:45,636 - INFO - [TRAIN] epoch=332/604, iter=38450/70000, loss=0.388818, lr=0.000013 | ETA 02:49:18 2022-07-29 22:49:02,686 - INFO - [TRAIN] epoch=332/604, iter=38500/70000, loss=0.366242, lr=0.000013 | ETA 02:54:25 2022-07-29 22:49:21,773 - INFO - [TRAIN] epoch=333/604, iter=38550/70000, loss=0.353199, lr=0.000013 | ETA 02:48:32 2022-07-29 22:49:39,816 - INFO - [TRAIN] epoch=333/604, iter=38600/70000, loss=0.338902, lr=0.000013 | ETA 06:13:23 2022-07-29 22:49:58,391 - INFO - [TRAIN] epoch=334/604, iter=38650/70000, loss=0.354468, lr=0.000013 | ETA 02:46:18 2022-07-29 22:50:15,678 - INFO - [TRAIN] epoch=334/604, iter=38700/70000, loss=0.396045, lr=0.000013 | ETA 02:46:23 2022-07-29 22:50:34,298 - INFO - [TRAIN] epoch=335/604, iter=38750/70000, loss=0.300875, lr=0.000013 | ETA 03:03:06 2022-07-29 22:50:52,349 - INFO - [TRAIN] epoch=335/604, iter=38800/70000, loss=0.329902, lr=0.000013 | ETA 03:33:28 2022-07-29 22:51:09,704 - INFO - [TRAIN] epoch=335/604, iter=38850/70000, loss=0.406688, lr=0.000013 | ETA 04:24:02 2022-07-29 22:51:29,518 - INFO - [TRAIN] epoch=336/604, iter=38900/70000, loss=0.370413, lr=0.000013 | ETA 03:39:36 2022-07-29 22:51:47,087 - INFO - [TRAIN] epoch=336/604, iter=38950/70000, loss=0.374552, lr=0.000013 | ETA 02:43:06 2022-07-29 22:52:06,099 - INFO - [TRAIN] epoch=337/604, iter=39000/70000, loss=0.321416, lr=0.000013 | ETA 02:54:24 2022-07-29 22:52:23,488 - INFO - [TRAIN] epoch=337/604, iter=39050/70000, loss=0.370490, lr=0.000013 | ETA 02:48:33 2022-07-29 22:52:42,351 - INFO - [TRAIN] epoch=338/604, iter=39100/70000, loss=0.361438, lr=0.000013 | ETA 02:52:35 2022-07-29 22:53:00,638 - INFO - [TRAIN] epoch=338/604, iter=39150/70000, loss=0.351808, lr=0.000013 | ETA 03:27:34 2022-07-29 22:53:18,166 - INFO - [TRAIN] epoch=338/604, iter=39200/70000, loss=0.383928, lr=0.000013 | ETA 02:40:54 2022-07-29 22:53:36,739 - INFO - [TRAIN] epoch=339/604, iter=39250/70000, loss=0.365222, lr=0.000013 | ETA 02:57:15 2022-07-29 22:53:53,559 - INFO - [TRAIN] epoch=339/604, iter=39300/70000, loss=0.344042, lr=0.000013 | ETA 02:44:02 2022-07-29 22:54:13,237 - INFO - [TRAIN] epoch=340/604, iter=39350/70000, loss=0.375041, lr=0.000013 | ETA 03:07:56 2022-07-29 22:54:31,038 - INFO - [TRAIN] epoch=340/604, iter=39400/70000, loss=0.354652, lr=0.000013 | ETA 02:41:55 2022-07-29 22:54:49,547 - INFO - [TRAIN] epoch=341/604, iter=39450/70000, loss=0.316390, lr=0.000013 | ETA 02:48:48 2022-07-29 22:55:07,305 - INFO - [TRAIN] epoch=341/604, iter=39500/70000, loss=0.302183, lr=0.000013 | ETA 02:42:38 2022-07-29 22:55:24,322 - INFO - [TRAIN] epoch=341/604, iter=39550/70000, loss=0.372215, lr=0.000013 | ETA 02:53:12 2022-07-29 22:55:43,786 - INFO - [TRAIN] epoch=342/604, iter=39600/70000, loss=0.308309, lr=0.000013 | ETA 04:03:40 2022-07-29 22:56:01,917 - INFO - [TRAIN] epoch=342/604, iter=39650/70000, loss=0.348575, lr=0.000013 | ETA 03:46:18 2022-07-29 22:56:20,511 - INFO - [TRAIN] epoch=343/604, iter=39700/70000, loss=0.290229, lr=0.000013 | ETA 02:47:39 2022-07-29 22:56:38,789 - INFO - [TRAIN] epoch=343/604, iter=39750/70000, loss=0.340722, lr=0.000013 | ETA 02:38:48 2022-07-29 22:56:57,289 - INFO - [TRAIN] epoch=344/604, iter=39800/70000, loss=0.353117, lr=0.000013 | ETA 02:37:41 2022-07-29 22:57:14,976 - INFO - [TRAIN] epoch=344/604, iter=39850/70000, loss=0.317162, lr=0.000013 | ETA 02:41:15 2022-07-29 22:57:32,470 - INFO - [TRAIN] epoch=344/604, iter=39900/70000, loss=0.343110, lr=0.000013 | ETA 02:51:52 2022-07-29 22:57:52,190 - INFO - [TRAIN] epoch=345/604, iter=39950/70000, loss=0.364710, lr=0.000013 | ETA 04:46:42 2022-07-29 22:58:09,644 - INFO - [TRAIN] epoch=345/604, iter=40000/70000, loss=0.385380, lr=0.000013 | ETA 02:43:16 2022-07-29 22:58:09,656 - INFO - Pop model from ./output/iter_15000 2022-07-29 22:58:10,914 - INFO - Push model to checkpoint ./output/iter_40000 2022-07-29 22:58:28,874 - INFO - [TRAIN] epoch=346/604, iter=40050/70000, loss=0.339196, lr=0.000013 | ETA 02:39:06 2022-07-29 22:58:46,561 - INFO - [TRAIN] epoch=346/604, iter=40100/70000, loss=0.334267, lr=0.000013 | ETA 02:48:20 2022-07-29 22:59:04,773 - INFO - [TRAIN] epoch=347/604, iter=40150/70000, loss=0.326211, lr=0.000013 | ETA 02:36:10 2022-07-29 22:59:22,482 - INFO - [TRAIN] epoch=347/604, iter=40200/70000, loss=0.351071, lr=0.000013 | ETA 02:35:28 2022-07-29 22:59:41,300 - INFO - [TRAIN] epoch=347/604, iter=40250/70000, loss=0.331479, lr=0.000013 | ETA 02:34:36 2022-07-29 23:00:00,512 - INFO - [TRAIN] epoch=348/604, iter=40300/70000, loss=0.375054, lr=0.000013 | ETA 02:38:09 2022-07-29 23:00:18,683 - INFO - [TRAIN] epoch=348/604, iter=40350/70000, loss=0.336219, lr=0.000013 | ETA 02:56:32 2022-07-29 23:00:37,403 - INFO - [TRAIN] epoch=349/604, iter=40400/70000, loss=0.358623, lr=0.000013 | ETA 02:36:24 2022-07-29 23:00:55,504 - INFO - [TRAIN] epoch=349/604, iter=40450/70000, loss=0.293824, lr=0.000013 | ETA 02:34:44 2022-07-29 23:01:15,879 - INFO - [TRAIN] epoch=350/604, iter=40500/70000, loss=0.326793, lr=0.000013 | ETA 03:41:14 2022-07-29 23:01:33,913 - INFO - [TRAIN] epoch=350/604, iter=40550/70000, loss=0.350379, lr=0.000013 | ETA 02:35:49 2022-07-29 23:01:51,926 - INFO - [TRAIN] epoch=350/604, iter=40600/70000, loss=0.359879, lr=0.000013 | ETA 02:33:45 2022-07-29 23:02:10,909 - INFO - [TRAIN] epoch=351/604, iter=40650/70000, loss=0.336326, lr=0.000013 | ETA 02:39:31 2022-07-29 23:02:28,487 - INFO - [TRAIN] epoch=351/604, iter=40700/70000, loss=0.339117, lr=0.000013 | ETA 02:35:39 2022-07-29 23:02:47,786 - INFO - [TRAIN] epoch=352/604, iter=40750/70000, loss=0.365069, lr=0.000013 | ETA 02:45:48 2022-07-29 23:03:04,247 - INFO - [TRAIN] epoch=352/604, iter=40800/70000, loss=0.334070, lr=0.000013 | ETA 02:41:44 2022-07-29 23:03:24,076 - INFO - [TRAIN] epoch=353/604, iter=40850/70000, loss=0.365259, lr=0.000013 | ETA 03:06:52 2022-07-29 23:03:42,611 - INFO - [TRAIN] epoch=353/604, iter=40900/70000, loss=0.339937, lr=0.000013 | ETA 02:35:01 2022-07-29 23:04:02,260 - INFO - [TRAIN] epoch=354/604, iter=40950/70000, loss=0.329608, lr=0.000013 | ETA 05:42:06 2022-07-29 23:04:20,301 - INFO - [TRAIN] epoch=354/604, iter=41000/70000, loss=0.330409, lr=0.000013 | ETA 02:41:56 2022-07-29 23:04:38,818 - INFO - [TRAIN] epoch=354/604, iter=41050/70000, loss=0.306091, lr=0.000013 | ETA 03:20:53 2022-07-29 23:04:58,528 - INFO - [TRAIN] epoch=355/604, iter=41100/70000, loss=0.325060, lr=0.000013 | ETA 02:43:46 2022-07-29 23:05:16,097 - INFO - [TRAIN] epoch=355/604, iter=41150/70000, loss=0.337137, lr=0.000013 | ETA 02:32:13 2022-07-29 23:05:35,171 - INFO - [TRAIN] epoch=356/604, iter=41200/70000, loss=0.369425, lr=0.000013 | ETA 02:31:07 2022-07-29 23:05:53,903 - INFO - [TRAIN] epoch=356/604, iter=41250/70000, loss=0.347998, lr=0.000013 | ETA 02:39:41 2022-07-29 23:06:12,250 - INFO - [TRAIN] epoch=357/604, iter=41300/70000, loss=0.317070, lr=0.000013 | ETA 03:28:08 2022-07-29 23:06:29,437 - INFO - [TRAIN] epoch=357/604, iter=41350/70000, loss=0.334757, lr=0.000013 | ETA 02:39:45 2022-07-29 23:06:46,078 - INFO - [TRAIN] epoch=357/604, iter=41400/70000, loss=0.363706, lr=0.000013 | ETA 02:29:28 2022-07-29 23:07:05,930 - INFO - [TRAIN] epoch=358/604, iter=41450/70000, loss=0.289867, lr=0.000013 | ETA 04:48:52 2022-07-29 23:07:22,206 - INFO - [TRAIN] epoch=358/604, iter=41500/70000, loss=0.314844, lr=0.000013 | ETA 02:28:42 2022-07-29 23:07:40,916 - INFO - [TRAIN] epoch=359/604, iter=41550/70000, loss=0.330418, lr=0.000013 | ETA 03:38:51 2022-07-29 23:07:57,854 - INFO - [TRAIN] epoch=359/604, iter=41600/70000, loss=0.342692, lr=0.000013 | ETA 02:33:37 2022-07-29 23:08:17,284 - INFO - [TRAIN] epoch=360/604, iter=41650/70000, loss=0.289725, lr=0.000013 | ETA 02:52:14 2022-07-29 23:08:34,930 - INFO - [TRAIN] epoch=360/604, iter=41700/70000, loss=0.299782, lr=0.000013 | ETA 02:35:40 2022-07-29 23:08:51,770 - INFO - [TRAIN] epoch=360/604, iter=41750/70000, loss=0.345455, lr=0.000013 | ETA 02:33:19 2022-07-29 23:09:11,731 - INFO - [TRAIN] epoch=361/604, iter=41800/70000, loss=0.316625, lr=0.000013 | ETA 04:35:22 2022-07-29 23:09:28,807 - INFO - [TRAIN] epoch=361/604, iter=41850/70000, loss=0.335920, lr=0.000013 | ETA 02:33:38 2022-07-29 23:09:48,343 - INFO - [TRAIN] epoch=362/604, iter=41900/70000, loss=0.332708, lr=0.000013 | ETA 02:37:58 2022-07-29 23:10:06,218 - INFO - [TRAIN] epoch=362/604, iter=41950/70000, loss=0.340095, lr=0.000013 | ETA 02:31:21 2022-07-29 23:10:24,638 - INFO - [TRAIN] epoch=363/604, iter=42000/70000, loss=0.311032, lr=0.000013 | ETA 02:32:54 2022-07-29 23:10:43,579 - INFO - [TRAIN] epoch=363/604, iter=42050/70000, loss=0.354379, lr=0.000013 | ETA 02:26:07 2022-07-29 23:11:01,557 - INFO - [TRAIN] epoch=363/604, iter=42100/70000, loss=0.356677, lr=0.000013 | ETA 02:28:59 2022-07-29 23:11:21,278 - INFO - [TRAIN] epoch=364/604, iter=42150/70000, loss=0.355833, lr=0.000013 | ETA 02:29:48 2022-07-29 23:11:39,472 - INFO - [TRAIN] epoch=364/604, iter=42200/70000, loss=0.338294, lr=0.000013 | ETA 02:25:48 2022-07-29 23:11:58,853 - INFO - [TRAIN] epoch=365/604, iter=42250/70000, loss=0.327020, lr=0.000013 | ETA 03:46:43 2022-07-29 23:12:15,196 - INFO - [TRAIN] epoch=365/604, iter=42300/70000, loss=0.305020, lr=0.000013 | ETA 02:29:10 2022-07-29 23:12:34,046 - INFO - [TRAIN] epoch=366/604, iter=42350/70000, loss=0.339357, lr=0.000013 | ETA 02:27:10 2022-07-29 23:12:51,763 - INFO - [TRAIN] epoch=366/604, iter=42400/70000, loss=0.339473, lr=0.000013 | ETA 02:25:40 2022-07-29 23:13:08,709 - INFO - [TRAIN] epoch=366/604, iter=42450/70000, loss=0.333454, lr=0.000013 | ETA 02:26:37 2022-07-29 23:13:27,868 - INFO - [TRAIN] epoch=367/604, iter=42500/70000, loss=0.325257, lr=0.000013 | ETA 02:33:24 2022-07-29 23:13:45,710 - INFO - [TRAIN] epoch=367/604, iter=42550/70000, loss=0.316692, lr=0.000013 | ETA 02:26:55 2022-07-29 23:14:05,255 - INFO - [TRAIN] epoch=368/604, iter=42600/70000, loss=0.322903, lr=0.000013 | ETA 04:27:13 2022-07-29 23:14:23,657 - INFO - [TRAIN] epoch=368/604, iter=42650/70000, loss=0.327301, lr=0.000013 | ETA 02:25:00 2022-07-29 23:14:42,468 - INFO - [TRAIN] epoch=369/604, iter=42700/70000, loss=0.298490, lr=0.000013 | ETA 03:12:09 2022-07-29 23:15:00,050 - INFO - [TRAIN] epoch=369/604, iter=42750/70000, loss=0.320258, lr=0.000013 | ETA 02:33:38 2022-07-29 23:15:17,679 - INFO - [TRAIN] epoch=369/604, iter=42800/70000, loss=0.330070, lr=0.000013 | ETA 02:23:17 2022-07-29 23:15:37,763 - INFO - [TRAIN] epoch=370/604, iter=42850/70000, loss=0.335296, lr=0.000013 | ETA 02:30:55 2022-07-29 23:15:56,344 - INFO - [TRAIN] epoch=370/604, iter=42900/70000, loss=0.335349, lr=0.000013 | ETA 02:26:33 2022-07-29 23:16:15,868 - INFO - [TRAIN] epoch=371/604, iter=42950/70000, loss=0.306830, lr=0.000013 | ETA 02:25:03 2022-07-29 23:16:33,383 - INFO - [TRAIN] epoch=371/604, iter=43000/70000, loss=0.307079, lr=0.000013 | ETA 02:27:58 2022-07-29 23:16:51,673 - INFO - [TRAIN] epoch=372/604, iter=43050/70000, loss=0.347939, lr=0.000013 | ETA 02:38:46 2022-07-29 23:17:10,014 - INFO - [TRAIN] epoch=372/604, iter=43100/70000, loss=0.355880, lr=0.000013 | ETA 02:22:29 2022-07-29 23:17:27,410 - INFO - [TRAIN] epoch=372/604, iter=43150/70000, loss=0.323127, lr=0.000013 | ETA 02:22:52 2022-07-29 23:17:46,213 - INFO - [TRAIN] epoch=373/604, iter=43200/70000, loss=0.368702, lr=0.000013 | ETA 02:25:56 2022-07-29 23:18:05,831 - INFO - [TRAIN] epoch=373/604, iter=43250/70000, loss=0.334131, lr=0.000013 | ETA 02:25:29 2022-07-29 23:18:25,030 - INFO - [TRAIN] epoch=374/604, iter=43300/70000, loss=0.315208, lr=0.000013 | ETA 02:30:17 2022-07-29 23:18:42,949 - INFO - [TRAIN] epoch=374/604, iter=43350/70000, loss=0.306786, lr=0.000013 | ETA 02:52:28 2022-07-29 23:19:01,191 - INFO - [TRAIN] epoch=375/604, iter=43400/70000, loss=0.297828, lr=0.000013 | ETA 02:27:44 2022-07-29 23:19:18,938 - INFO - [TRAIN] epoch=375/604, iter=43450/70000, loss=0.290288, lr=0.000013 | ETA 02:19:01 2022-07-29 23:19:35,425 - INFO - [TRAIN] epoch=375/604, iter=43500/70000, loss=0.343572, lr=0.000013 | ETA 02:19:34 2022-07-29 23:19:55,193 - INFO - [TRAIN] epoch=376/604, iter=43550/70000, loss=0.349120, lr=0.000013 | ETA 02:18:43 2022-07-29 23:20:12,732 - INFO - [TRAIN] epoch=376/604, iter=43600/70000, loss=0.334446, lr=0.000013 | ETA 03:06:01 2022-07-29 23:20:32,247 - INFO - [TRAIN] epoch=377/604, iter=43650/70000, loss=0.362963, lr=0.000013 | ETA 02:26:54 2022-07-29 23:20:49,941 - INFO - [TRAIN] epoch=377/604, iter=43700/70000, loss=0.318496, lr=0.000013 | ETA 02:17:56 2022-07-29 23:21:09,878 - INFO - [TRAIN] epoch=378/604, iter=43750/70000, loss=0.363718, lr=0.000013 | ETA 02:21:06 2022-07-29 23:21:27,313 - INFO - [TRAIN] epoch=378/604, iter=43800/70000, loss=0.366841, lr=0.000013 | ETA 02:49:02 2022-07-29 23:21:46,199 - INFO - [TRAIN] epoch=379/604, iter=43850/70000, loss=0.339565, lr=0.000013 | ETA 05:15:50 2022-07-29 23:22:04,675 - INFO - [TRAIN] epoch=379/604, iter=43900/70000, loss=0.310209, lr=0.000013 | ETA 02:24:17 2022-07-29 23:22:22,323 - INFO - [TRAIN] epoch=379/604, iter=43950/70000, loss=0.310246, lr=0.000013 | ETA 02:17:42 2022-07-29 23:22:41,696 - INFO - [TRAIN] epoch=380/604, iter=44000/70000, loss=0.325739, lr=0.000013 | ETA 02:21:03 2022-07-29 23:23:00,060 - INFO - [TRAIN] epoch=380/604, iter=44050/70000, loss=0.339599, lr=0.000013 | ETA 02:15:26 2022-07-29 23:23:20,721 - INFO - [TRAIN] epoch=381/604, iter=44100/70000, loss=0.309470, lr=0.000013 | ETA 02:18:47 2022-07-29 23:23:39,395 - INFO - [TRAIN] epoch=381/604, iter=44150/70000, loss=0.351644, lr=0.000013 | ETA 02:20:57 2022-07-29 23:23:59,305 - INFO - [TRAIN] epoch=382/604, iter=44200/70000, loss=0.320844, lr=0.000013 | ETA 02:58:52 2022-07-29 23:24:16,720 - INFO - [TRAIN] epoch=382/604, iter=44250/70000, loss=0.282768, lr=0.000013 | ETA 02:17:34 2022-07-29 23:24:33,608 - INFO - [TRAIN] epoch=382/604, iter=44300/70000, loss=0.319061, lr=0.000013 | ETA 02:13:25 2022-07-29 23:24:53,174 - INFO - [TRAIN] epoch=383/604, iter=44350/70000, loss=0.325364, lr=0.000013 | ETA 05:21:06 2022-07-29 23:25:10,449 - INFO - [TRAIN] epoch=383/604, iter=44400/70000, loss=0.330620, lr=0.000013 | ETA 02:45:54 2022-07-29 23:25:29,169 - INFO - [TRAIN] epoch=384/604, iter=44450/70000, loss=0.313960, lr=0.000013 | ETA 02:17:00 2022-07-29 23:25:47,360 - INFO - [TRAIN] epoch=384/604, iter=44500/70000, loss=0.316901, lr=0.000013 | ETA 02:18:02 2022-07-29 23:26:06,677 - INFO - [TRAIN] epoch=385/604, iter=44550/70000, loss=0.299896, lr=0.000013 | ETA 02:27:04 2022-07-29 23:26:25,621 - INFO - [TRAIN] epoch=385/604, iter=44600/70000, loss=0.329909, lr=0.000013 | ETA 02:26:54 2022-07-29 23:26:42,507 - INFO - [TRAIN] epoch=385/604, iter=44650/70000, loss=0.297822, lr=0.000013 | ETA 02:21:42 2022-07-29 23:27:01,550 - INFO - [TRAIN] epoch=386/604, iter=44700/70000, loss=0.334718, lr=0.000013 | ETA 02:15:26 2022-07-29 23:27:19,510 - INFO - [TRAIN] epoch=386/604, iter=44750/70000, loss=0.300642, lr=0.000013 | ETA 02:58:12 2022-07-29 23:27:38,598 - INFO - [TRAIN] epoch=387/604, iter=44800/70000, loss=0.307764, lr=0.000013 | ETA 02:23:13 2022-07-29 23:27:57,391 - INFO - [TRAIN] epoch=387/604, iter=44850/70000, loss=0.379530, lr=0.000013 | ETA 03:47:28 2022-07-29 23:28:16,637 - INFO - [TRAIN] epoch=388/604, iter=44900/70000, loss=0.307005, lr=0.000013 | ETA 02:16:24 2022-07-29 23:28:32,948 - INFO - [TRAIN] epoch=388/604, iter=44950/70000, loss=0.337323, lr=0.000013 | ETA 02:13:10 2022-07-29 23:28:50,379 - INFO - [TRAIN] epoch=388/604, iter=45000/70000, loss=0.342734, lr=0.000013 | ETA 02:12:35 2022-07-29 23:28:50,391 - INFO - Pop model from ./output/iter_20000 2022-07-29 23:28:51,563 - INFO - Push model to checkpoint ./output/iter_45000 2022-07-29 23:29:10,603 - INFO - [TRAIN] epoch=389/604, iter=45050/70000, loss=0.284614, lr=0.000013 | ETA 02:10:59 2022-07-29 23:29:29,186 - INFO - [TRAIN] epoch=389/604, iter=45100/70000, loss=0.363623, lr=0.000013 | ETA 02:13:07 2022-07-29 23:29:48,980 - INFO - [TRAIN] epoch=390/604, iter=45150/70000, loss=0.301108, lr=0.000013 | ETA 02:51:00 2022-07-29 23:30:06,927 - INFO - [TRAIN] epoch=390/604, iter=45200/70000, loss=0.302521, lr=0.000013 | ETA 02:14:54 2022-07-29 23:30:25,470 - INFO - [TRAIN] epoch=391/604, iter=45250/70000, loss=0.322857, lr=0.000013 | ETA 02:12:58 2022-07-29 23:30:42,598 - INFO - [TRAIN] epoch=391/604, iter=45300/70000, loss=0.305058, lr=0.000013 | ETA 02:09:48 2022-07-29 23:30:59,742 - INFO - [TRAIN] epoch=391/604, iter=45350/70000, loss=0.319936, lr=0.000013 | ETA 03:17:43 2022-07-29 23:31:18,453 - INFO - [TRAIN] epoch=392/604, iter=45400/70000, loss=0.319309, lr=0.000013 | ETA 02:10:15 2022-07-29 23:31:37,864 - INFO - [TRAIN] epoch=392/604, iter=45450/70000, loss=0.342592, lr=0.000013 | ETA 02:20:17 2022-07-29 23:31:57,943 - INFO - [TRAIN] epoch=393/604, iter=45500/70000, loss=0.321145, lr=0.000013 | ETA 02:58:13 2022-07-29 23:32:15,966 - INFO - [TRAIN] epoch=393/604, iter=45550/70000, loss=0.372249, lr=0.000013 | ETA 02:14:21 2022-07-29 23:32:35,377 - INFO - [TRAIN] epoch=394/604, iter=45600/70000, loss=0.312230, lr=0.000013 | ETA 02:26:23 2022-07-29 23:32:53,310 - INFO - [TRAIN] epoch=394/604, iter=45650/70000, loss=0.284469, lr=0.000013 | ETA 02:11:10 2022-07-29 23:33:11,184 - INFO - [TRAIN] epoch=394/604, iter=45700/70000, loss=0.329552, lr=0.000013 | ETA 02:07:07 2022-07-29 23:33:30,310 - INFO - [TRAIN] epoch=395/604, iter=45750/70000, loss=0.278570, lr=0.000013 | ETA 06:04:52 2022-07-29 23:33:46,947 - INFO - [TRAIN] epoch=395/604, iter=45800/70000, loss=0.265293, lr=0.000013 | ETA 02:15:44 2022-07-29 23:34:06,716 - INFO - [TRAIN] epoch=396/604, iter=45850/70000, loss=0.347128, lr=0.000013 | ETA 02:09:42 2022-07-29 23:34:23,403 - INFO - [TRAIN] epoch=396/604, iter=45900/70000, loss=0.299538, lr=0.000013 | ETA 02:08:34 2022-07-29 23:34:42,905 - INFO - [TRAIN] epoch=397/604, iter=45950/70000, loss=0.309636, lr=0.000013 | ETA 02:08:44 2022-07-29 23:35:02,206 - INFO - [TRAIN] epoch=397/604, iter=46000/70000, loss=0.324715, lr=0.000013 | ETA 02:16:40 2022-07-29 23:35:19,385 - INFO - [TRAIN] epoch=397/604, iter=46050/70000, loss=0.327951, lr=0.000013 | ETA 02:06:07 2022-07-29 23:35:38,592 - INFO - [TRAIN] epoch=398/604, iter=46100/70000, loss=0.325542, lr=0.000013 | ETA 02:06:26 2022-07-29 23:35:55,906 - INFO - [TRAIN] epoch=398/604, iter=46150/70000, loss=0.324236, lr=0.000013 | ETA 02:09:24 2022-07-29 23:36:14,726 - INFO - [TRAIN] epoch=399/604, iter=46200/70000, loss=0.327422, lr=0.000013 | ETA 02:04:48 2022-07-29 23:36:33,020 - INFO - [TRAIN] epoch=399/604, iter=46250/70000, loss=0.334050, lr=0.000013 | ETA 03:16:52 2022-07-29 23:36:51,565 - INFO - [TRAIN] epoch=400/604, iter=46300/70000, loss=0.346064, lr=0.000013 | ETA 02:43:00 2022-07-29 23:37:09,828 - INFO - [TRAIN] epoch=400/604, iter=46350/70000, loss=0.297862, lr=0.000013 | ETA 02:42:23 2022-07-29 23:37:27,578 - INFO - [TRAIN] epoch=400/604, iter=46400/70000, loss=0.305837, lr=0.000013 | ETA 02:03:38 2022-07-29 23:37:47,128 - INFO - [TRAIN] epoch=401/604, iter=46450/70000, loss=0.293104, lr=0.000013 | ETA 02:07:31 2022-07-29 23:38:04,476 - INFO - [TRAIN] epoch=401/604, iter=46500/70000, loss=0.348918, lr=0.000013 | ETA 02:04:44 2022-07-29 23:38:23,333 - INFO - [TRAIN] epoch=402/604, iter=46550/70000, loss=0.316521, lr=0.000013 | ETA 02:04:39 2022-07-29 23:38:40,356 - INFO - [TRAIN] epoch=402/604, iter=46600/70000, loss=0.319141, lr=0.000013 | ETA 02:17:44 2022-07-29 23:38:59,071 - INFO - [TRAIN] epoch=403/604, iter=46650/70000, loss=0.289286, lr=0.000013 | ETA 02:06:41 2022-07-29 23:39:18,172 - INFO - [TRAIN] epoch=403/604, iter=46700/70000, loss=0.313560, lr=0.000013 | ETA 02:11:25 2022-07-29 23:39:37,988 - INFO - [TRAIN] epoch=404/604, iter=46750/70000, loss=0.310816, lr=0.000013 | ETA 04:35:08 2022-07-29 23:39:56,138 - INFO - [TRAIN] epoch=404/604, iter=46800/70000, loss=0.336633, lr=0.000013 | ETA 02:08:22 2022-07-29 23:40:13,607 - INFO - [TRAIN] epoch=404/604, iter=46850/70000, loss=0.323374, lr=0.000013 | ETA 02:07:04 2022-07-29 23:40:33,045 - INFO - [TRAIN] epoch=405/604, iter=46900/70000, loss=0.303670, lr=0.000013 | ETA 02:00:56 2022-07-29 23:40:50,259 - INFO - [TRAIN] epoch=405/604, iter=46950/70000, loss=0.344402, lr=0.000013 | ETA 01:59:40 2022-07-29 23:41:08,906 - INFO - [TRAIN] epoch=406/604, iter=47000/70000, loss=0.292579, lr=0.000013 | ETA 02:04:20 2022-07-29 23:41:26,852 - INFO - [TRAIN] epoch=406/604, iter=47050/70000, loss=0.361946, lr=0.000013 | ETA 02:02:17 2022-07-29 23:41:45,622 - INFO - [TRAIN] epoch=407/604, iter=47100/70000, loss=0.309548, lr=0.000013 | ETA 02:53:40 2022-07-29 23:42:03,942 - INFO - [TRAIN] epoch=407/604, iter=47150/70000, loss=0.287149, lr=0.000013 | ETA 02:18:36 2022-07-29 23:42:20,842 - INFO - [TRAIN] epoch=407/604, iter=47200/70000, loss=0.290781, lr=0.000013 | ETA 02:02:25 2022-07-29 23:42:40,299 - INFO - [TRAIN] epoch=408/604, iter=47250/70000, loss=0.350473, lr=0.000013 | ETA 04:15:18 2022-07-29 23:42:57,219 - INFO - [TRAIN] epoch=408/604, iter=47300/70000, loss=0.329204, lr=0.000013 | ETA 02:04:50 2022-07-29 23:43:16,176 - INFO - [TRAIN] epoch=409/604, iter=47350/70000, loss=0.306253, lr=0.000013 | ETA 01:59:53 2022-07-29 23:43:35,508 - INFO - [TRAIN] epoch=409/604, iter=47400/70000, loss=0.333139, lr=0.000013 | ETA 01:58:06 2022-07-29 23:43:54,806 - INFO - [TRAIN] epoch=410/604, iter=47450/70000, loss=0.337435, lr=0.000013 | ETA 02:07:53 2022-07-29 23:44:13,357 - INFO - [TRAIN] epoch=410/604, iter=47500/70000, loss=0.280771, lr=0.000013 | ETA 02:04:01 2022-07-29 23:44:31,777 - INFO - [TRAIN] epoch=410/604, iter=47550/70000, loss=0.322273, lr=0.000013 | ETA 02:29:35 2022-07-29 23:44:50,918 - INFO - [TRAIN] epoch=411/604, iter=47600/70000, loss=0.348842, lr=0.000013 | ETA 02:15:27 2022-07-29 23:45:08,620 - INFO - [TRAIN] epoch=411/604, iter=47650/70000, loss=0.325289, lr=0.000013 | ETA 01:58:21 2022-07-29 23:45:27,617 - INFO - [TRAIN] epoch=412/604, iter=47700/70000, loss=0.330236, lr=0.000013 | ETA 02:00:51 2022-07-29 23:45:45,557 - INFO - [TRAIN] epoch=412/604, iter=47750/70000, loss=0.304610, lr=0.000013 | ETA 01:56:33 2022-07-29 23:46:04,752 - INFO - [TRAIN] epoch=413/604, iter=47800/70000, loss=0.314460, lr=0.000013 | ETA 01:58:18 2022-07-29 23:46:22,537 - INFO - [TRAIN] epoch=413/604, iter=47850/70000, loss=0.266591, lr=0.000013 | ETA 01:59:11 2022-07-29 23:46:38,578 - INFO - [TRAIN] epoch=413/604, iter=47900/70000, loss=0.353527, lr=0.000013 | ETA 01:57:11 2022-07-29 23:46:56,128 - INFO - [TRAIN] epoch=414/604, iter=47950/70000, loss=0.295538, lr=0.000013 | ETA 01:55:05 2022-07-29 23:47:12,094 - INFO - [TRAIN] epoch=414/604, iter=48000/70000, loss=0.313279, lr=0.000013 | ETA 01:56:32 2022-07-29 23:47:29,584 - INFO - [TRAIN] epoch=415/604, iter=48050/70000, loss=0.328519, lr=0.000013 | ETA 01:56:52 2022-07-29 23:47:45,745 - INFO - [TRAIN] epoch=415/604, iter=48100/70000, loss=0.278722, lr=0.000013 | ETA 01:55:33 2022-07-29 23:48:03,483 - INFO - [TRAIN] epoch=416/604, iter=48150/70000, loss=0.336846, lr=0.000013 | ETA 02:02:41 2022-07-29 23:48:19,480 - INFO - [TRAIN] epoch=416/604, iter=48200/70000, loss=0.345870, lr=0.000013 | ETA 01:57:58 2022-07-29 23:48:35,436 - INFO - [TRAIN] epoch=416/604, iter=48250/70000, loss=0.303298, lr=0.000013 | ETA 01:54:28 2022-07-29 23:48:52,925 - INFO - [TRAIN] epoch=417/604, iter=48300/70000, loss=0.299968, lr=0.000013 | ETA 01:58:15 2022-07-29 23:49:08,893 - INFO - [TRAIN] epoch=417/604, iter=48350/70000, loss=0.296008, lr=0.000013 | ETA 01:55:02 2022-07-29 23:49:26,256 - INFO - [TRAIN] epoch=418/604, iter=48400/70000, loss=0.287580, lr=0.000013 | ETA 01:52:23 2022-07-29 23:49:42,205 - INFO - [TRAIN] epoch=418/604, iter=48450/70000, loss=0.294602, lr=0.000013 | ETA 01:52:37 2022-07-29 23:49:59,878 - INFO - [TRAIN] epoch=419/604, iter=48500/70000, loss=0.316539, lr=0.000013 | ETA 01:57:11 2022-07-29 23:50:16,007 - INFO - [TRAIN] epoch=419/604, iter=48550/70000, loss=0.277620, lr=0.000013 | ETA 01:58:38 2022-07-29 23:50:31,923 - INFO - [TRAIN] epoch=419/604, iter=48600/70000, loss=0.309795, lr=0.000013 | ETA 01:53:47 2022-07-29 23:50:49,573 - INFO - [TRAIN] epoch=420/604, iter=48650/70000, loss=0.331963, lr=0.000013 | ETA 01:51:18 2022-07-29 23:51:06,393 - INFO - [TRAIN] epoch=420/604, iter=48700/70000, loss=0.304967, lr=0.000013 | ETA 01:59:04 2022-07-29 23:51:24,350 - INFO - [TRAIN] epoch=421/604, iter=48750/70000, loss=0.319137, lr=0.000013 | ETA 01:52:10 2022-07-29 23:51:40,865 - INFO - [TRAIN] epoch=421/604, iter=48800/70000, loss=0.326106, lr=0.000013 | ETA 02:25:40 2022-07-29 23:51:58,474 - INFO - [TRAIN] epoch=422/604, iter=48850/70000, loss=0.302431, lr=0.000013 | ETA 01:50:19 2022-07-29 23:52:15,831 - INFO - [TRAIN] epoch=422/604, iter=48900/70000, loss=0.329183, lr=0.000013 | ETA 01:51:01 2022-07-29 23:52:32,364 - INFO - [TRAIN] epoch=422/604, iter=48950/70000, loss=0.320803, lr=0.000013 | ETA 01:54:42 2022-07-29 23:52:50,761 - INFO - [TRAIN] epoch=423/604, iter=49000/70000, loss=0.321196, lr=0.000013 | ETA 01:51:21 2022-07-29 23:53:07,424 - INFO - [TRAIN] epoch=423/604, iter=49050/70000, loss=0.283364, lr=0.000013 | ETA 01:50:27 2022-07-29 23:53:25,120 - INFO - [TRAIN] epoch=424/604, iter=49100/70000, loss=0.315217, lr=0.000013 | ETA 01:49:54 2022-07-29 23:53:41,863 - INFO - [TRAIN] epoch=424/604, iter=49150/70000, loss=0.308120, lr=0.000013 | ETA 01:52:34 2022-07-29 23:54:00,051 - INFO - [TRAIN] epoch=425/604, iter=49200/70000, loss=0.343185, lr=0.000013 | ETA 01:54:13 2022-07-29 23:54:17,301 - INFO - [TRAIN] epoch=425/604, iter=49250/70000, loss=0.291024, lr=0.000013 | ETA 01:51:45 2022-07-29 23:54:33,548 - INFO - [TRAIN] epoch=425/604, iter=49300/70000, loss=0.282528, lr=0.000013 | ETA 01:50:54 2022-07-29 23:54:51,563 - INFO - [TRAIN] epoch=426/604, iter=49350/70000, loss=0.336956, lr=0.000013 | ETA 01:50:13 2022-07-29 23:55:08,266 - INFO - [TRAIN] epoch=426/604, iter=49400/70000, loss=0.301414, lr=0.000013 | ETA 01:54:04 2022-07-29 23:55:26,861 - INFO - [TRAIN] epoch=427/604, iter=49450/70000, loss=0.340203, lr=0.000013 | ETA 01:50:14 2022-07-29 23:55:43,932 - INFO - [TRAIN] epoch=427/604, iter=49500/70000, loss=0.279898, lr=0.000013 | ETA 01:48:03 2022-07-29 23:56:01,803 - INFO - [TRAIN] epoch=428/604, iter=49550/70000, loss=0.336024, lr=0.000013 | ETA 01:47:47 2022-07-29 23:56:18,132 - INFO - [TRAIN] epoch=428/604, iter=49600/70000, loss=0.343137, lr=0.000013 | ETA 01:54:16 2022-07-29 23:56:36,297 - INFO - [TRAIN] epoch=429/604, iter=49650/70000, loss=0.297623, lr=0.000013 | ETA 04:10:16 2022-07-29 23:56:54,075 - INFO - [TRAIN] epoch=429/604, iter=49700/70000, loss=0.312941, lr=0.000013 | ETA 02:39:36 2022-07-29 23:57:10,983 - INFO - [TRAIN] epoch=429/604, iter=49750/70000, loss=0.351258, lr=0.000013 | ETA 01:54:29 2022-07-29 23:57:29,495 - INFO - [TRAIN] epoch=430/604, iter=49800/70000, loss=0.297468, lr=0.000013 | ETA 01:53:33 2022-07-29 23:57:47,255 - INFO - [TRAIN] epoch=430/604, iter=49850/70000, loss=0.298426, lr=0.000013 | ETA 03:44:11 2022-07-29 23:58:06,162 - INFO - [TRAIN] epoch=431/604, iter=49900/70000, loss=0.332856, lr=0.000013 | ETA 03:43:55 2022-07-29 23:58:23,509 - INFO - [TRAIN] epoch=431/604, iter=49950/70000, loss=0.292142, lr=0.000013 | ETA 01:46:55 2022-07-29 23:58:42,666 - INFO - [TRAIN] epoch=432/604, iter=50000/70000, loss=0.331120, lr=0.000013 | ETA 02:27:47 2022-07-29 23:58:42,684 - INFO - Pop model from ./output/iter_25000 2022-07-29 23:58:43,879 - INFO - Push model to checkpoint ./output/iter_50000 2022-07-29 23:59:01,139 - INFO - [TRAIN] epoch=432/604, iter=50050/70000, loss=0.290205, lr=0.000013 | ETA 02:43:04 2022-07-29 23:59:17,699 - INFO - [TRAIN] epoch=432/604, iter=50100/70000, loss=0.302522, lr=0.000013 | ETA 01:45:52 2022-07-29 23:59:36,987 - INFO - [TRAIN] epoch=433/604, iter=50150/70000, loss=0.299524, lr=0.000013 | ETA 01:48:40 2022-07-29 23:59:53,807 - INFO - [TRAIN] epoch=433/604, iter=50200/70000, loss=0.288452, lr=0.000013 | ETA 01:43:49 2022-07-30 00:00:12,853 - INFO - [TRAIN] epoch=434/604, iter=50250/70000, loss=0.325719, lr=0.000013 | ETA 01:45:55 2022-07-30 00:00:30,555 - INFO - [TRAIN] epoch=434/604, iter=50300/70000, loss=0.296369, lr=0.000013 | ETA 03:51:13 2022-07-30 00:00:49,391 - INFO - [TRAIN] epoch=435/604, iter=50350/70000, loss=0.262825, lr=0.000013 | ETA 01:54:21 2022-07-30 00:01:06,114 - INFO - [TRAIN] epoch=435/604, iter=50400/70000, loss=0.331685, lr=0.000013 | ETA 01:49:00 2022-07-30 00:01:23,177 - INFO - [TRAIN] epoch=435/604, iter=50450/70000, loss=0.328840, lr=0.000013 | ETA 02:06:36 2022-07-30 00:01:41,338 - INFO - [TRAIN] epoch=436/604, iter=50500/70000, loss=0.302342, lr=0.000013 | ETA 01:45:11 2022-07-30 00:01:58,794 - INFO - [TRAIN] epoch=436/604, iter=50550/70000, loss=0.280878, lr=0.000013 | ETA 01:44:36 2022-07-30 00:02:16,963 - INFO - [TRAIN] epoch=437/604, iter=50600/70000, loss=0.277866, lr=0.000013 | ETA 01:43:31 2022-07-30 00:02:34,440 - INFO - [TRAIN] epoch=437/604, iter=50650/70000, loss=0.301734, lr=0.000013 | ETA 01:44:42 2022-07-30 00:02:52,908 - INFO - [TRAIN] epoch=438/604, iter=50700/70000, loss=0.349744, lr=0.000013 | ETA 01:42:44 2022-07-30 00:03:09,860 - INFO - [TRAIN] epoch=438/604, iter=50750/70000, loss=0.312703, lr=0.000013 | ETA 01:40:40 2022-07-30 00:03:26,970 - INFO - [TRAIN] epoch=438/604, iter=50800/70000, loss=0.309271, lr=0.000013 | ETA 01:41:44 2022-07-30 00:03:45,417 - INFO - [TRAIN] epoch=439/604, iter=50850/70000, loss=0.297314, lr=0.000013 | ETA 01:44:18 2022-07-30 00:04:02,697 - INFO - [TRAIN] epoch=439/604, iter=50900/70000, loss=0.312494, lr=0.000013 | ETA 01:45:56 2022-07-30 00:04:22,053 - INFO - [TRAIN] epoch=440/604, iter=50950/70000, loss=0.294902, lr=0.000013 | ETA 01:45:49 2022-07-30 00:04:39,522 - INFO - [TRAIN] epoch=440/604, iter=51000/70000, loss=0.311087, lr=0.000013 | ETA 01:44:39 2022-07-30 00:04:58,432 - INFO - [TRAIN] epoch=441/604, iter=51050/70000, loss=0.313099, lr=0.000013 | ETA 02:39:11 2022-07-30 00:05:16,458 - INFO - [TRAIN] epoch=441/604, iter=51100/70000, loss=0.310211, lr=0.000013 | ETA 01:41:56 2022-07-30 00:05:33,583 - INFO - [TRAIN] epoch=441/604, iter=51150/70000, loss=0.298397, lr=0.000013 | ETA 01:47:17 2022-07-30 00:05:53,507 - INFO - [TRAIN] epoch=442/604, iter=51200/70000, loss=0.322688, lr=0.000013 | ETA 02:13:15 2022-07-30 00:06:09,931 - INFO - [TRAIN] epoch=442/604, iter=51250/70000, loss=0.301507, lr=0.000013 | ETA 01:39:10 2022-07-30 00:06:28,909 - INFO - [TRAIN] epoch=443/604, iter=51300/70000, loss=0.303099, lr=0.000013 | ETA 01:44:48 2022-07-30 00:06:46,418 - INFO - [TRAIN] epoch=443/604, iter=51350/70000, loss=0.302038, lr=0.000013 | ETA 01:38:05 2022-07-30 00:07:04,218 - INFO - [TRAIN] epoch=444/604, iter=51400/70000, loss=0.280081, lr=0.000013 | ETA 01:38:32 2022-07-30 00:07:21,872 - INFO - [TRAIN] epoch=444/604, iter=51450/70000, loss=0.322820, lr=0.000013 | ETA 01:40:05 2022-07-30 00:07:40,374 - INFO - [TRAIN] epoch=444/604, iter=51500/70000, loss=0.309792, lr=0.000013 | ETA 02:07:30 2022-07-30 00:07:59,398 - INFO - [TRAIN] epoch=445/604, iter=51550/70000, loss=0.327560, lr=0.000013 | ETA 01:52:27 2022-07-30 00:08:16,156 - INFO - [TRAIN] epoch=445/604, iter=51600/70000, loss=0.283498, lr=0.000013 | ETA 01:40:35 2022-07-30 00:08:34,481 - INFO - [TRAIN] epoch=446/604, iter=51650/70000, loss=0.290459, lr=0.000013 | ETA 02:00:15 2022-07-30 00:08:51,321 - INFO - [TRAIN] epoch=446/604, iter=51700/70000, loss=0.315618, lr=0.000013 | ETA 01:35:45 2022-07-30 00:09:10,588 - INFO - [TRAIN] epoch=447/604, iter=51750/70000, loss=0.320017, lr=0.000013 | ETA 02:34:18 2022-07-30 00:09:27,553 - INFO - [TRAIN] epoch=447/604, iter=51800/70000, loss=0.321807, lr=0.000013 | ETA 01:39:15 2022-07-30 00:09:45,029 - INFO - [TRAIN] epoch=447/604, iter=51850/70000, loss=0.310219, lr=0.000013 | ETA 01:35:26 2022-07-30 00:10:03,435 - INFO - [TRAIN] epoch=448/604, iter=51900/70000, loss=0.297436, lr=0.000013 | ETA 01:36:36 2022-07-30 00:10:21,165 - INFO - [TRAIN] epoch=448/604, iter=51950/70000, loss=0.302102, lr=0.000013 | ETA 01:38:07 2022-07-30 00:10:40,139 - INFO - [TRAIN] epoch=449/604, iter=52000/70000, loss=0.321280, lr=0.000013 | ETA 01:41:22 2022-07-30 00:10:57,680 - INFO - [TRAIN] epoch=449/604, iter=52050/70000, loss=0.287172, lr=0.000013 | ETA 01:33:33 2022-07-30 00:11:16,991 - INFO - [TRAIN] epoch=450/604, iter=52100/70000, loss=0.325667, lr=0.000013 | ETA 02:11:32 2022-07-30 00:11:35,283 - INFO - [TRAIN] epoch=450/604, iter=52150/70000, loss=0.299236, lr=0.000013 | ETA 01:34:19 2022-07-30 00:11:52,569 - INFO - [TRAIN] epoch=450/604, iter=52200/70000, loss=0.327247, lr=0.000013 | ETA 01:33:12 2022-07-30 00:12:11,540 - INFO - [TRAIN] epoch=451/604, iter=52250/70000, loss=0.296294, lr=0.000013 | ETA 01:39:21 2022-07-30 00:12:29,345 - INFO - [TRAIN] epoch=451/604, iter=52300/70000, loss=0.307720, lr=0.000013 | ETA 01:41:03 2022-07-30 00:12:48,233 - INFO - [TRAIN] epoch=452/604, iter=52350/70000, loss=0.358677, lr=0.000013 | ETA 01:32:26 2022-07-30 00:13:05,813 - INFO - [TRAIN] epoch=452/604, iter=52400/70000, loss=0.285892, lr=0.000013 | ETA 01:43:30 2022-07-30 00:13:24,507 - INFO - [TRAIN] epoch=453/604, iter=52450/70000, loss=0.331974, lr=0.000013 | ETA 01:31:42 2022-07-30 00:13:42,121 - INFO - [TRAIN] epoch=453/604, iter=52500/70000, loss=0.289796, lr=0.000013 | ETA 01:41:11 2022-07-30 00:14:00,232 - INFO - [TRAIN] epoch=454/604, iter=52550/70000, loss=0.303141, lr=0.000013 | ETA 03:33:26 2022-07-30 00:14:17,134 - INFO - [TRAIN] epoch=454/604, iter=52600/70000, loss=0.297236, lr=0.000013 | ETA 01:44:41 2022-07-30 00:14:35,263 - INFO - [TRAIN] epoch=454/604, iter=52650/70000, loss=0.329517, lr=0.000013 | ETA 01:37:51 2022-07-30 00:14:54,513 - INFO - [TRAIN] epoch=455/604, iter=52700/70000, loss=0.313442, lr=0.000013 | ETA 01:36:56 2022-07-30 00:15:11,626 - INFO - [TRAIN] epoch=455/604, iter=52750/70000, loss=0.292272, lr=0.000013 | ETA 01:34:50 2022-07-30 00:15:30,157 - INFO - [TRAIN] epoch=456/604, iter=52800/70000, loss=0.328065, lr=0.000013 | ETA 01:35:08 2022-07-30 00:15:47,468 - INFO - [TRAIN] epoch=456/604, iter=52850/70000, loss=0.308912, lr=0.000013 | ETA 02:32:57 2022-07-30 00:16:06,088 - INFO - [TRAIN] epoch=457/604, iter=52900/70000, loss=0.294152, lr=0.000013 | ETA 01:59:05 2022-07-30 00:16:23,444 - INFO - [TRAIN] epoch=457/604, iter=52950/70000, loss=0.274821, lr=0.000013 | ETA 01:29:24 2022-07-30 00:16:41,234 - INFO - [TRAIN] epoch=457/604, iter=53000/70000, loss=0.298471, lr=0.000013 | ETA 01:31:44 2022-07-30 00:17:00,158 - INFO - [TRAIN] epoch=458/604, iter=53050/70000, loss=0.284537, lr=0.000013 | ETA 01:28:54 2022-07-30 00:17:17,439 - INFO - [TRAIN] epoch=458/604, iter=53100/70000, loss=0.310965, lr=0.000013 | ETA 02:15:15 2022-07-30 00:17:35,535 - INFO - [TRAIN] epoch=459/604, iter=53150/70000, loss=0.331181, lr=0.000013 | ETA 01:32:52 2022-07-30 00:17:53,350 - INFO - [TRAIN] epoch=459/604, iter=53200/70000, loss=0.305146, lr=0.000013 | ETA 02:49:08 2022-07-30 00:18:12,392 - INFO - [TRAIN] epoch=460/604, iter=53250/70000, loss=0.293009, lr=0.000013 | ETA 01:35:01 2022-07-30 00:18:30,518 - INFO - [TRAIN] epoch=460/604, iter=53300/70000, loss=0.311355, lr=0.000013 | ETA 02:16:49 2022-07-30 00:18:47,128 - INFO - [TRAIN] epoch=460/604, iter=53350/70000, loss=0.304338, lr=0.000013 | ETA 01:32:34 2022-07-30 00:19:06,531 - INFO - [TRAIN] epoch=461/604, iter=53400/70000, loss=0.276322, lr=0.000013 | ETA 01:57:32 2022-07-30 00:19:23,233 - INFO - [TRAIN] epoch=461/604, iter=53450/70000, loss=0.282230, lr=0.000013 | ETA 01:27:14 2022-07-30 00:19:41,690 - INFO - [TRAIN] epoch=462/604, iter=53500/70000, loss=0.316923, lr=0.000013 | ETA 01:53:13 2022-07-30 00:19:59,026 - INFO - [TRAIN] epoch=462/604, iter=53550/70000, loss=0.299187, lr=0.000013 | ETA 01:53:51 2022-07-30 00:20:16,689 - INFO - [TRAIN] epoch=463/604, iter=53600/70000, loss=0.336384, lr=0.000013 | ETA 01:30:04 2022-07-30 00:20:35,606 - INFO - [TRAIN] epoch=463/604, iter=53650/70000, loss=0.324141, lr=0.000013 | ETA 01:28:00 2022-07-30 00:20:52,355 - INFO - [TRAIN] epoch=463/604, iter=53700/70000, loss=0.325068, lr=0.000013 | ETA 01:25:45 2022-07-30 00:21:11,398 - INFO - [TRAIN] epoch=464/604, iter=53750/70000, loss=0.281756, lr=0.000013 | ETA 01:25:05 2022-07-30 00:21:29,672 - INFO - [TRAIN] epoch=464/604, iter=53800/70000, loss=0.309137, lr=0.000013 | ETA 01:29:29 2022-07-30 00:21:48,055 - INFO - [TRAIN] epoch=465/604, iter=53850/70000, loss=0.280138, lr=0.000013 | ETA 01:24:11 2022-07-30 00:22:05,281 - INFO - [TRAIN] epoch=465/604, iter=53900/70000, loss=0.339493, lr=0.000013 | ETA 01:30:17 2022-07-30 00:22:23,708 - INFO - [TRAIN] epoch=466/604, iter=53950/70000, loss=0.322769, lr=0.000013 | ETA 01:24:02 2022-07-30 00:22:42,109 - INFO - [TRAIN] epoch=466/604, iter=54000/70000, loss=0.270779, lr=0.000013 | ETA 01:25:16 2022-07-30 00:22:59,714 - INFO - [TRAIN] epoch=466/604, iter=54050/70000, loss=0.271614, lr=0.000013 | ETA 01:54:44 2022-07-30 00:23:18,862 - INFO - [TRAIN] epoch=467/604, iter=54100/70000, loss=0.283891, lr=0.000013 | ETA 01:42:01 2022-07-30 00:23:35,995 - INFO - [TRAIN] epoch=467/604, iter=54150/70000, loss=0.279725, lr=0.000013 | ETA 01:30:21 2022-07-30 00:23:55,823 - INFO - [TRAIN] epoch=468/604, iter=54200/70000, loss=0.264847, lr=0.000013 | ETA 01:34:56 2022-07-30 00:24:13,107 - INFO - [TRAIN] epoch=468/604, iter=54250/70000, loss=0.279319, lr=0.000013 | ETA 01:22:49 2022-07-30 00:24:32,203 - INFO - [TRAIN] epoch=469/604, iter=54300/70000, loss=0.273387, lr=0.000013 | ETA 01:25:11 2022-07-30 00:24:49,525 - INFO - [TRAIN] epoch=469/604, iter=54350/70000, loss=0.285803, lr=0.000013 | ETA 01:21:38 2022-07-30 00:25:07,507 - INFO - [TRAIN] epoch=469/604, iter=54400/70000, loss=0.273184, lr=0.000013 | ETA 01:23:56 2022-07-30 00:25:26,005 - INFO - [TRAIN] epoch=470/604, iter=54450/70000, loss=0.264842, lr=0.000013 | ETA 01:21:21 2022-07-30 00:25:43,650 - INFO - [TRAIN] epoch=470/604, iter=54500/70000, loss=0.315099, lr=0.000013 | ETA 01:32:47 2022-07-30 00:26:02,490 - INFO - [TRAIN] epoch=471/604, iter=54550/70000, loss=0.295067, lr=0.000013 | ETA 01:37:55 2022-07-30 00:26:19,778 - INFO - [TRAIN] epoch=471/604, iter=54600/70000, loss=0.270054, lr=0.000013 | ETA 01:24:41 2022-07-30 00:26:38,053 - INFO - [TRAIN] epoch=472/604, iter=54650/70000, loss=0.289583, lr=0.000013 | ETA 01:40:04 2022-07-30 00:26:55,619 - INFO - [TRAIN] epoch=472/604, iter=54700/70000, loss=0.315698, lr=0.000013 | ETA 01:24:33 2022-07-30 00:27:12,686 - INFO - [TRAIN] epoch=472/604, iter=54750/70000, loss=0.307317, lr=0.000013 | ETA 01:21:04 2022-07-30 00:27:31,665 - INFO - [TRAIN] epoch=473/604, iter=54800/70000, loss=0.300928, lr=0.000013 | ETA 01:22:16 2022-07-30 00:27:48,963 - INFO - [TRAIN] epoch=473/604, iter=54850/70000, loss=0.369434, lr=0.000013 | ETA 01:23:10 2022-07-30 00:28:07,839 - INFO - [TRAIN] epoch=474/604, iter=54900/70000, loss=0.273248, lr=0.000013 | ETA 01:18:58 2022-07-30 00:28:26,408 - INFO - [TRAIN] epoch=474/604, iter=54950/70000, loss=0.307876, lr=0.000013 | ETA 01:21:43 2022-07-30 00:28:46,020 - INFO - [TRAIN] epoch=475/604, iter=55000/70000, loss=0.325775, lr=0.000001 | ETA 02:07:41 2022-07-30 00:28:46,034 - INFO - Pop model from ./output/iter_30000 2022-07-30 00:28:47,271 - INFO - Push model to checkpoint ./output/iter_55000 2022-07-30 00:29:04,884 - INFO - [TRAIN] epoch=475/604, iter=55050/70000, loss=0.300161, lr=0.000001 | ETA 01:18:33 2022-07-30 00:29:23,036 - INFO - [TRAIN] epoch=475/604, iter=55100/70000, loss=0.257745, lr=0.000001 | ETA 01:19:14 2022-07-30 00:29:42,315 - INFO - [TRAIN] epoch=476/604, iter=55150/70000, loss=0.290512, lr=0.000001 | ETA 01:20:46 2022-07-30 00:29:59,412 - INFO - [TRAIN] epoch=476/604, iter=55200/70000, loss=0.265538, lr=0.000001 | ETA 01:19:05 2022-07-30 00:30:18,264 - INFO - [TRAIN] epoch=477/604, iter=55250/70000, loss=0.293075, lr=0.000001 | ETA 01:17:25 2022-07-30 00:30:35,553 - INFO - [TRAIN] epoch=477/604, iter=55300/70000, loss=0.273278, lr=0.000001 | ETA 01:17:42 2022-07-30 00:30:54,626 - INFO - [TRAIN] epoch=478/604, iter=55350/70000, loss=0.290989, lr=0.000001 | ETA 01:17:21 2022-07-30 00:31:12,246 - INFO - [TRAIN] epoch=478/604, iter=55400/70000, loss=0.291795, lr=0.000001 | ETA 01:23:30 2022-07-30 00:31:30,926 - INFO - [TRAIN] epoch=479/604, iter=55450/70000, loss=0.308894, lr=0.000001 | ETA 02:51:52 2022-07-30 00:31:48,170 - INFO - [TRAIN] epoch=479/604, iter=55500/70000, loss=0.320025, lr=0.000001 | ETA 01:16:29 2022-07-30 00:32:05,440 - INFO - [TRAIN] epoch=479/604, iter=55550/70000, loss=0.259738, lr=0.000001 | ETA 01:27:07 2022-07-30 00:32:24,649 - INFO - [TRAIN] epoch=480/604, iter=55600/70000, loss=0.302208, lr=0.000001 | ETA 01:16:31 2022-07-30 00:32:41,674 - INFO - [TRAIN] epoch=480/604, iter=55650/70000, loss=0.261480, lr=0.000001 | ETA 01:21:58 2022-07-30 00:33:00,054 - INFO - [TRAIN] epoch=481/604, iter=55700/70000, loss=0.292692, lr=0.000001 | ETA 02:02:49 2022-07-30 00:33:18,586 - INFO - [TRAIN] epoch=481/604, iter=55750/70000, loss=0.259535, lr=0.000001 | ETA 01:33:08 2022-07-30 00:33:37,635 - INFO - [TRAIN] epoch=482/604, iter=55800/70000, loss=0.272146, lr=0.000001 | ETA 01:38:30 2022-07-30 00:33:54,724 - INFO - [TRAIN] epoch=482/604, iter=55850/70000, loss=0.326584, lr=0.000001 | ETA 01:13:37 2022-07-30 00:34:12,194 - INFO - [TRAIN] epoch=482/604, iter=55900/70000, loss=0.291660, lr=0.000001 | ETA 01:20:27 2022-07-30 00:34:31,787 - INFO - [TRAIN] epoch=483/604, iter=55950/70000, loss=0.262594, lr=0.000001 | ETA 01:14:32 2022-07-30 00:34:50,015 - INFO - [TRAIN] epoch=483/604, iter=56000/70000, loss=0.272136, lr=0.000001 | ETA 01:18:01 2022-07-30 00:35:09,233 - INFO - [TRAIN] epoch=484/604, iter=56050/70000, loss=0.305121, lr=0.000001 | ETA 01:38:56 2022-07-30 00:35:26,009 - INFO - [TRAIN] epoch=484/604, iter=56100/70000, loss=0.278766, lr=0.000001 | ETA 01:14:45 2022-07-30 00:35:45,252 - INFO - [TRAIN] epoch=485/604, iter=56150/70000, loss=0.262031, lr=0.000001 | ETA 01:20:32 2022-07-30 00:36:03,578 - INFO - [TRAIN] epoch=485/604, iter=56200/70000, loss=0.285255, lr=0.000001 | ETA 01:49:11 2022-07-30 00:36:21,576 - INFO - [TRAIN] epoch=485/604, iter=56250/70000, loss=0.305915, lr=0.000001 | ETA 01:17:45 2022-07-30 00:36:40,350 - INFO - [TRAIN] epoch=486/604, iter=56300/70000, loss=0.275135, lr=0.000001 | ETA 01:13:19 2022-07-30 00:36:58,286 - INFO - [TRAIN] epoch=486/604, iter=56350/70000, loss=0.255969, lr=0.000001 | ETA 01:11:57 2022-07-30 00:37:16,972 - INFO - [TRAIN] epoch=487/604, iter=56400/70000, loss=0.262949, lr=0.000001 | ETA 01:20:31 2022-07-30 00:37:34,912 - INFO - [TRAIN] epoch=487/604, iter=56450/70000, loss=0.267865, lr=0.000001 | ETA 01:13:09 2022-07-30 00:37:52,674 - INFO - [TRAIN] epoch=488/604, iter=56500/70000, loss=0.288198, lr=0.000001 | ETA 01:13:57 2022-07-30 00:38:10,683 - INFO - [TRAIN] epoch=488/604, iter=56550/70000, loss=0.325827, lr=0.000001 | ETA 01:10:50 2022-07-30 00:38:28,044 - INFO - [TRAIN] epoch=488/604, iter=56600/70000, loss=0.278256, lr=0.000001 | ETA 01:27:58 2022-07-30 00:38:47,289 - INFO - [TRAIN] epoch=489/604, iter=56650/70000, loss=0.294875, lr=0.000001 | ETA 01:17:08 2022-07-30 00:39:05,269 - INFO - [TRAIN] epoch=489/604, iter=56700/70000, loss=0.344598, lr=0.000001 | ETA 01:12:53 2022-07-30 00:39:24,242 - INFO - [TRAIN] epoch=490/604, iter=56750/70000, loss=0.239695, lr=0.000001 | ETA 01:23:32 2022-07-30 00:39:41,661 - INFO - [TRAIN] epoch=490/604, iter=56800/70000, loss=0.306585, lr=0.000001 | ETA 01:30:33 2022-07-30 00:40:01,284 - INFO - [TRAIN] epoch=491/604, iter=56850/70000, loss=0.291649, lr=0.000001 | ETA 01:16:04 2022-07-30 00:40:18,430 - INFO - [TRAIN] epoch=491/604, iter=56900/70000, loss=0.268963, lr=0.000001 | ETA 01:14:20 2022-07-30 00:40:35,378 - INFO - [TRAIN] epoch=491/604, iter=56950/70000, loss=0.300897, lr=0.000001 | ETA 01:11:50 2022-07-30 00:40:55,225 - INFO - [TRAIN] epoch=492/604, iter=57000/70000, loss=0.277544, lr=0.000001 | ETA 01:39:36 2022-07-30 00:41:13,131 - INFO - [TRAIN] epoch=492/604, iter=57050/70000, loss=0.259735, lr=0.000001 | ETA 02:01:05 2022-07-30 00:41:31,460 - INFO - [TRAIN] epoch=493/604, iter=57100/70000, loss=0.297206, lr=0.000001 | ETA 01:10:43 2022-07-30 00:41:48,472 - INFO - [TRAIN] epoch=493/604, iter=57150/70000, loss=0.238926, lr=0.000001 | ETA 01:09:20 2022-07-30 00:42:07,359 - INFO - [TRAIN] epoch=494/604, iter=57200/70000, loss=0.267927, lr=0.000001 | ETA 01:21:45 2022-07-30 00:42:25,122 - INFO - [TRAIN] epoch=494/604, iter=57250/70000, loss=0.306256, lr=0.000001 | ETA 01:10:32 2022-07-30 00:42:41,631 - INFO - [TRAIN] epoch=494/604, iter=57300/70000, loss=0.321867, lr=0.000001 | ETA 01:06:25 2022-07-30 00:43:00,940 - INFO - [TRAIN] epoch=495/604, iter=57350/70000, loss=0.284623, lr=0.000001 | ETA 01:14:15 2022-07-30 00:43:18,464 - INFO - [TRAIN] epoch=495/604, iter=57400/70000, loss=0.282789, lr=0.000001 | ETA 01:43:08 2022-07-30 00:43:38,482 - INFO - [TRAIN] epoch=496/604, iter=57450/70000, loss=0.262110, lr=0.000001 | ETA 01:36:17 2022-07-30 00:43:55,829 - INFO - [TRAIN] epoch=496/604, iter=57500/70000, loss=0.257835, lr=0.000001 | ETA 01:16:32 2022-07-30 00:44:14,847 - INFO - [TRAIN] epoch=497/604, iter=57550/70000, loss=0.268024, lr=0.000001 | ETA 01:12:09 2022-07-30 00:44:32,690 - INFO - [TRAIN] epoch=497/604, iter=57600/70000, loss=0.252762, lr=0.000001 | ETA 01:08:39 2022-07-30 00:44:50,352 - INFO - [TRAIN] epoch=497/604, iter=57650/70000, loss=0.279692, lr=0.000001 | ETA 01:04:39 2022-07-30 00:45:10,059 - INFO - [TRAIN] epoch=498/604, iter=57700/70000, loss=0.260439, lr=0.000001 | ETA 01:15:09 2022-07-30 00:45:27,600 - INFO - [TRAIN] epoch=498/604, iter=57750/70000, loss=0.291285, lr=0.000001 | ETA 01:05:33 2022-07-30 00:45:46,841 - INFO - [TRAIN] epoch=499/604, iter=57800/70000, loss=0.308588, lr=0.000001 | ETA 01:06:19 2022-07-30 00:46:04,940 - INFO - [TRAIN] epoch=499/604, iter=57850/70000, loss=0.278743, lr=0.000001 | ETA 01:05:12 2022-07-30 00:46:25,084 - INFO - [TRAIN] epoch=500/604, iter=57900/70000, loss=0.292328, lr=0.000001 | ETA 02:22:20 2022-07-30 00:46:42,548 - INFO - [TRAIN] epoch=500/604, iter=57950/70000, loss=0.260525, lr=0.000001 | ETA 01:05:27 2022-07-30 00:47:00,417 - INFO - [TRAIN] epoch=500/604, iter=58000/70000, loss=0.274544, lr=0.000001 | ETA 01:02:21 2022-07-30 00:47:20,161 - INFO - [TRAIN] epoch=501/604, iter=58050/70000, loss=0.301522, lr=0.000001 | ETA 01:10:07 2022-07-30 00:47:38,869 - INFO - [TRAIN] epoch=501/604, iter=58100/70000, loss=0.268869, lr=0.000001 | ETA 01:05:08 2022-07-30 00:47:58,017 - INFO - [TRAIN] epoch=502/604, iter=58150/70000, loss=0.284589, lr=0.000001 | ETA 01:05:38 2022-07-30 00:48:15,536 - INFO - [TRAIN] epoch=502/604, iter=58200/70000, loss=0.274759, lr=0.000001 | ETA 01:01:50 2022-07-30 00:48:33,787 - INFO - [TRAIN] epoch=503/604, iter=58250/70000, loss=0.273827, lr=0.000001 | ETA 01:03:23 2022-07-30 00:48:51,499 - INFO - [TRAIN] epoch=503/604, iter=58300/70000, loss=0.287850, lr=0.000001 | ETA 01:04:45 2022-07-30 00:49:10,154 - INFO - [TRAIN] epoch=504/604, iter=58350/70000, loss=0.278859, lr=0.000001 | ETA 02:21:45 2022-07-30 00:49:29,152 - INFO - [TRAIN] epoch=504/604, iter=58400/70000, loss=0.220326, lr=0.000001 | ETA 01:10:37 2022-07-30 00:49:46,626 - INFO - [TRAIN] epoch=504/604, iter=58450/70000, loss=0.271158, lr=0.000001 | ETA 01:03:15 2022-07-30 00:50:05,814 - INFO - [TRAIN] epoch=505/604, iter=58500/70000, loss=0.285245, lr=0.000001 | ETA 01:03:55 2022-07-30 00:50:24,497 - INFO - [TRAIN] epoch=505/604, iter=58550/70000, loss=0.314933, lr=0.000001 | ETA 01:03:08 2022-07-30 00:50:43,929 - INFO - [TRAIN] epoch=506/604, iter=58600/70000, loss=0.240579, lr=0.000001 | ETA 01:00:25 2022-07-30 00:51:01,692 - INFO - [TRAIN] epoch=506/604, iter=58650/70000, loss=0.275742, lr=0.000001 | ETA 01:00:52 2022-07-30 00:51:21,301 - INFO - [TRAIN] epoch=507/604, iter=58700/70000, loss=0.277254, lr=0.000001 | ETA 01:18:42 2022-07-30 00:51:39,248 - INFO - [TRAIN] epoch=507/604, iter=58750/70000, loss=0.253118, lr=0.000001 | ETA 01:11:08 2022-07-30 00:51:57,473 - INFO - [TRAIN] epoch=507/604, iter=58800/70000, loss=0.264612, lr=0.000001 | ETA 01:03:32 2022-07-30 00:52:16,825 - INFO - [TRAIN] epoch=508/604, iter=58850/70000, loss=0.283411, lr=0.000001 | ETA 00:58:45 2022-07-30 00:52:34,518 - INFO - [TRAIN] epoch=508/604, iter=58900/70000, loss=0.266205, lr=0.000001 | ETA 00:59:04 2022-07-30 00:52:54,120 - INFO - [TRAIN] epoch=509/604, iter=58950/70000, loss=0.273212, lr=0.000001 | ETA 01:27:00 2022-07-30 00:53:11,839 - INFO - [TRAIN] epoch=509/604, iter=59000/70000, loss=0.252229, lr=0.000001 | ETA 00:59:36 2022-07-30 00:53:30,435 - INFO - [TRAIN] epoch=510/604, iter=59050/70000, loss=0.264224, lr=0.000001 | ETA 01:03:18 2022-07-30 00:53:48,597 - INFO - [TRAIN] epoch=510/604, iter=59100/70000, loss=0.234767, lr=0.000001 | ETA 01:09:07 2022-07-30 00:54:07,164 - INFO - [TRAIN] epoch=510/604, iter=59150/70000, loss=0.275583, lr=0.000001 | ETA 01:38:46 2022-07-30 00:54:25,828 - INFO - [TRAIN] epoch=511/604, iter=59200/70000, loss=0.303959, lr=0.000001 | ETA 00:56:22 2022-07-30 00:54:44,138 - INFO - [TRAIN] epoch=511/604, iter=59250/70000, loss=0.239017, lr=0.000001 | ETA 01:05:33 2022-07-30 00:55:04,359 - INFO - [TRAIN] epoch=512/604, iter=59300/70000, loss=0.312519, lr=0.000001 | ETA 01:34:24 2022-07-30 00:55:22,667 - INFO - [TRAIN] epoch=512/604, iter=59350/70000, loss=0.261583, lr=0.000001 | ETA 00:57:28 2022-07-30 00:55:41,830 - INFO - [TRAIN] epoch=513/604, iter=59400/70000, loss=0.299941, lr=0.000001 | ETA 00:58:37 2022-07-30 00:55:59,207 - INFO - [TRAIN] epoch=513/604, iter=59450/70000, loss=0.254010, lr=0.000001 | ETA 00:56:30 2022-07-30 00:56:17,230 - INFO - [TRAIN] epoch=513/604, iter=59500/70000, loss=0.236472, lr=0.000001 | ETA 00:57:17 2022-07-30 00:56:35,834 - INFO - [TRAIN] epoch=514/604, iter=59550/70000, loss=0.253555, lr=0.000001 | ETA 00:54:57 2022-07-30 00:56:54,107 - INFO - [TRAIN] epoch=514/604, iter=59600/70000, loss=0.285767, lr=0.000001 | ETA 00:54:21 2022-07-30 00:57:14,922 - INFO - [TRAIN] epoch=515/604, iter=59650/70000, loss=0.253962, lr=0.000001 | ETA 01:16:11 2022-07-30 00:57:33,258 - INFO - [TRAIN] epoch=515/604, iter=59700/70000, loss=0.277928, lr=0.000001 | ETA 00:54:02 2022-07-30 00:57:52,735 - INFO - [TRAIN] epoch=516/604, iter=59750/70000, loss=0.281208, lr=0.000001 | ETA 00:54:59 2022-07-30 00:58:11,474 - INFO - [TRAIN] epoch=516/604, iter=59800/70000, loss=0.241205, lr=0.000001 | ETA 01:00:11 2022-07-30 00:58:28,842 - INFO - [TRAIN] epoch=516/604, iter=59850/70000, loss=0.283378, lr=0.000001 | ETA 01:11:13 2022-07-30 00:58:48,534 - INFO - [TRAIN] epoch=517/604, iter=59900/70000, loss=0.261497, lr=0.000001 | ETA 00:53:52 2022-07-30 00:59:04,892 - INFO - [TRAIN] epoch=517/604, iter=59950/70000, loss=0.279885, lr=0.000001 | ETA 00:53:28 2022-07-30 00:59:24,331 - INFO - [TRAIN] epoch=518/604, iter=60000/70000, loss=0.275065, lr=0.000001 | ETA 01:07:15 2022-07-30 00:59:24,344 - INFO - Pop model from ./output/iter_35000 2022-07-30 00:59:25,503 - INFO - Push model to checkpoint ./output/iter_60000 2022-07-30 00:59:43,480 - INFO - [TRAIN] epoch=518/604, iter=60050/70000, loss=0.229390, lr=0.000001 | ETA 00:58:18 2022-07-30 01:00:01,098 - INFO - [TRAIN] epoch=519/604, iter=60100/70000, loss=0.223595, lr=0.000001 | ETA 00:51:59 2022-07-30 01:00:19,454 - INFO - [TRAIN] epoch=519/604, iter=60150/70000, loss=0.304203, lr=0.000001 | ETA 00:52:06 2022-07-30 01:00:37,105 - INFO - [TRAIN] epoch=519/604, iter=60200/70000, loss=0.262131, lr=0.000001 | ETA 00:52:07 2022-07-30 01:00:55,710 - INFO - [TRAIN] epoch=520/604, iter=60250/70000, loss=0.255420, lr=0.000001 | ETA 00:50:41 2022-07-30 01:01:13,694 - INFO - [TRAIN] epoch=520/604, iter=60300/70000, loss=0.257655, lr=0.000001 | ETA 01:23:45 2022-07-30 01:01:32,764 - INFO - [TRAIN] epoch=521/604, iter=60350/70000, loss=0.244378, lr=0.000001 | ETA 01:11:40 2022-07-30 01:01:51,066 - INFO - [TRAIN] epoch=521/604, iter=60400/70000, loss=0.254176, lr=0.000001 | ETA 00:50:07 2022-07-30 01:02:09,287 - INFO - [TRAIN] epoch=522/604, iter=60450/70000, loss=0.266143, lr=0.000001 | ETA 00:57:41 2022-07-30 01:02:27,077 - INFO - [TRAIN] epoch=522/604, iter=60500/70000, loss=0.284124, lr=0.000001 | ETA 00:52:38 2022-07-30 01:02:45,007 - INFO - [TRAIN] epoch=522/604, iter=60550/70000, loss=0.278715, lr=0.000001 | ETA 00:49:46 2022-07-30 01:03:04,977 - INFO - [TRAIN] epoch=523/604, iter=60600/70000, loss=0.287815, lr=0.000001 | ETA 00:52:52 2022-07-30 01:03:22,916 - INFO - [TRAIN] epoch=523/604, iter=60650/70000, loss=0.258539, lr=0.000001 | ETA 00:48:47 2022-07-30 01:03:41,955 - INFO - [TRAIN] epoch=524/604, iter=60700/70000, loss=0.258598, lr=0.000001 | ETA 00:48:46 2022-07-30 01:04:01,761 - INFO - [TRAIN] epoch=524/604, iter=60750/70000, loss=0.264562, lr=0.000001 | ETA 00:50:04 2022-07-30 01:04:20,574 - INFO - [TRAIN] epoch=525/604, iter=60800/70000, loss=0.269049, lr=0.000001 | ETA 00:50:06 2022-07-30 01:04:39,066 - INFO - [TRAIN] epoch=525/604, iter=60850/70000, loss=0.289135, lr=0.000001 | ETA 00:58:28 2022-07-30 01:04:56,325 - INFO - [TRAIN] epoch=525/604, iter=60900/70000, loss=0.284202, lr=0.000001 | ETA 00:48:35 2022-07-30 01:05:15,893 - INFO - [TRAIN] epoch=526/604, iter=60950/70000, loss=0.295039, lr=0.000001 | ETA 00:55:06 2022-07-30 01:05:33,253 - INFO - [TRAIN] epoch=526/604, iter=61000/70000, loss=0.264193, lr=0.000001 | ETA 00:58:46 2022-07-30 01:05:53,029 - INFO - [TRAIN] epoch=527/604, iter=61050/70000, loss=0.272587, lr=0.000001 | ETA 00:47:39 2022-07-30 01:06:10,823 - INFO - [TRAIN] epoch=527/604, iter=61100/70000, loss=0.255287, lr=0.000001 | ETA 01:41:15 2022-07-30 01:06:30,782 - INFO - [TRAIN] epoch=528/604, iter=61150/70000, loss=0.230812, lr=0.000001 | ETA 00:59:16 2022-07-30 01:06:48,255 - INFO - [TRAIN] epoch=528/604, iter=61200/70000, loss=0.259967, lr=0.000001 | ETA 00:47:36 2022-07-30 01:07:06,582 - INFO - [TRAIN] epoch=529/604, iter=61250/70000, loss=0.294594, lr=0.000001 | ETA 01:44:55 2022-07-30 01:07:24,168 - INFO - [TRAIN] epoch=529/604, iter=61300/70000, loss=0.290755, lr=0.000001 | ETA 00:46:54 2022-07-30 01:07:42,837 - INFO - [TRAIN] epoch=529/604, iter=61350/70000, loss=0.239962, lr=0.000001 | ETA 00:46:46 2022-07-30 01:08:01,458 - INFO - [TRAIN] epoch=530/604, iter=61400/70000, loss=0.257351, lr=0.000001 | ETA 01:15:16 2022-07-30 01:08:19,716 - INFO - [TRAIN] epoch=530/604, iter=61450/70000, loss=0.261789, lr=0.000001 | ETA 01:13:07 2022-07-30 01:08:38,976 - INFO - [TRAIN] epoch=531/604, iter=61500/70000, loss=0.264298, lr=0.000001 | ETA 00:50:41 2022-07-30 01:08:56,929 - INFO - [TRAIN] epoch=531/604, iter=61550/70000, loss=0.290632, lr=0.000001 | ETA 00:53:56 2022-07-30 01:09:15,615 - INFO - [TRAIN] epoch=532/604, iter=61600/70000, loss=0.257282, lr=0.000001 | ETA 00:57:56 2022-07-30 01:09:33,283 - INFO - [TRAIN] epoch=532/604, iter=61650/70000, loss=0.234151, lr=0.000001 | ETA 00:43:27 2022-07-30 01:09:50,330 - INFO - [TRAIN] epoch=532/604, iter=61700/70000, loss=0.238956, lr=0.000001 | ETA 00:44:43 2022-07-30 01:10:09,624 - INFO - [TRAIN] epoch=533/604, iter=61750/70000, loss=0.294313, lr=0.000001 | ETA 00:48:40 2022-07-30 01:10:26,261 - INFO - [TRAIN] epoch=533/604, iter=61800/70000, loss=0.297214, lr=0.000001 | ETA 00:45:33 2022-07-30 01:10:46,021 - INFO - [TRAIN] epoch=534/604, iter=61850/70000, loss=0.299555, lr=0.000001 | ETA 00:55:46 2022-07-30 01:11:04,823 - INFO - [TRAIN] epoch=534/604, iter=61900/70000, loss=0.264892, lr=0.000001 | ETA 01:23:22 2022-07-30 01:11:24,954 - INFO - [TRAIN] epoch=535/604, iter=61950/70000, loss=0.257437, lr=0.000001 | ETA 00:50:47 2022-07-30 01:11:43,482 - INFO - [TRAIN] epoch=535/604, iter=62000/70000, loss=0.267416, lr=0.000001 | ETA 01:03:07 2022-07-30 01:11:59,867 - INFO - [TRAIN] epoch=535/604, iter=62050/70000, loss=0.274746, lr=0.000001 | ETA 00:41:41 2022-07-30 01:12:19,241 - INFO - [TRAIN] epoch=536/604, iter=62100/70000, loss=0.318486, lr=0.000001 | ETA 00:41:58 2022-07-30 01:12:36,005 - INFO - [TRAIN] epoch=536/604, iter=62150/70000, loss=0.288164, lr=0.000001 | ETA 00:58:09 2022-07-30 01:12:55,552 - INFO - [TRAIN] epoch=537/604, iter=62200/70000, loss=0.276322, lr=0.000001 | ETA 00:41:39 2022-07-30 01:13:14,227 - INFO - [TRAIN] epoch=537/604, iter=62250/70000, loss=0.278628, lr=0.000001 | ETA 00:44:01 2022-07-30 01:13:32,582 - INFO - [TRAIN] epoch=538/604, iter=62300/70000, loss=0.314443, lr=0.000001 | ETA 00:41:50 2022-07-30 01:13:52,095 - INFO - [TRAIN] epoch=538/604, iter=62350/70000, loss=0.279303, lr=0.000001 | ETA 00:59:59 2022-07-30 01:14:09,612 - INFO - [TRAIN] epoch=538/604, iter=62400/70000, loss=0.252738, lr=0.000001 | ETA 01:21:06 2022-07-30 01:14:29,004 - INFO - [TRAIN] epoch=539/604, iter=62450/70000, loss=0.230897, lr=0.000001 | ETA 00:41:15 2022-07-30 01:14:46,027 - INFO - [TRAIN] epoch=539/604, iter=62500/70000, loss=0.289754, lr=0.000001 | ETA 00:41:51 2022-07-30 01:15:05,764 - INFO - [TRAIN] epoch=540/604, iter=62550/70000, loss=0.279371, lr=0.000001 | ETA 00:42:28 2022-07-30 01:15:24,252 - INFO - [TRAIN] epoch=540/604, iter=62600/70000, loss=0.296761, lr=0.000001 | ETA 00:38:57 2022-07-30 01:15:42,855 - INFO - [TRAIN] epoch=541/604, iter=62650/70000, loss=0.244687, lr=0.000001 | ETA 00:39:57 2022-07-30 01:16:00,927 - INFO - [TRAIN] epoch=541/604, iter=62700/70000, loss=0.240604, lr=0.000001 | ETA 00:38:17 2022-07-30 01:16:19,540 - INFO - [TRAIN] epoch=541/604, iter=62750/70000, loss=0.262374, lr=0.000001 | ETA 00:37:50 2022-07-30 01:16:38,340 - INFO - [TRAIN] epoch=542/604, iter=62800/70000, loss=0.277523, lr=0.000001 | ETA 00:40:35 2022-07-30 01:16:55,673 - INFO - [TRAIN] epoch=542/604, iter=62850/70000, loss=0.273755, lr=0.000001 | ETA 00:38:30 2022-07-30 01:17:15,240 - INFO - [TRAIN] epoch=543/604, iter=62900/70000, loss=0.266465, lr=0.000001 | ETA 00:46:28 2022-07-30 01:17:33,351 - INFO - [TRAIN] epoch=543/604, iter=62950/70000, loss=0.271581, lr=0.000001 | ETA 00:37:07 2022-07-30 01:17:52,745 - INFO - [TRAIN] epoch=544/604, iter=63000/70000, loss=0.252945, lr=0.000001 | ETA 00:36:46 2022-07-30 01:18:11,561 - INFO - [TRAIN] epoch=544/604, iter=63050/70000, loss=0.301726, lr=0.000001 | ETA 00:38:49 2022-07-30 01:18:30,519 - INFO - [TRAIN] epoch=544/604, iter=63100/70000, loss=0.288633, lr=0.000001 | ETA 00:38:03 2022-07-30 01:18:50,775 - INFO - [TRAIN] epoch=545/604, iter=63150/70000, loss=0.253381, lr=0.000001 | ETA 00:35:58 2022-07-30 01:19:08,038 - INFO - [TRAIN] epoch=545/604, iter=63200/70000, loss=0.289109, lr=0.000001 | ETA 00:39:01 2022-07-30 01:19:28,818 - INFO - [TRAIN] epoch=546/604, iter=63250/70000, loss=0.267228, lr=0.000001 | ETA 00:41:02 2022-07-30 01:19:47,185 - INFO - [TRAIN] epoch=546/604, iter=63300/70000, loss=0.297180, lr=0.000001 | ETA 00:35:19 2022-07-30 01:20:06,127 - INFO - [TRAIN] epoch=547/604, iter=63350/70000, loss=0.263806, lr=0.000001 | ETA 00:53:23 2022-07-30 01:20:25,203 - INFO - [TRAIN] epoch=547/604, iter=63400/70000, loss=0.309639, lr=0.000001 | ETA 00:34:53 2022-07-30 01:20:44,232 - INFO - [TRAIN] epoch=547/604, iter=63450/70000, loss=0.252512, lr=0.000001 | ETA 00:34:18 2022-07-30 01:21:02,776 - INFO - [TRAIN] epoch=548/604, iter=63500/70000, loss=0.280844, lr=0.000001 | ETA 00:33:45 2022-07-30 01:21:21,009 - INFO - [TRAIN] epoch=548/604, iter=63550/70000, loss=0.269818, lr=0.000001 | ETA 00:34:08 2022-07-30 01:21:39,750 - INFO - [TRAIN] epoch=549/604, iter=63600/70000, loss=0.294108, lr=0.000001 | ETA 00:33:38 2022-07-30 01:21:57,558 - INFO - [TRAIN] epoch=549/604, iter=63650/70000, loss=0.248046, lr=0.000001 | ETA 00:33:39 2022-07-30 01:22:16,718 - INFO - [TRAIN] epoch=550/604, iter=63700/70000, loss=0.284187, lr=0.000001 | ETA 00:41:02 2022-07-30 01:22:34,251 - INFO - [TRAIN] epoch=550/604, iter=63750/70000, loss=0.227673, lr=0.000001 | ETA 00:33:22 2022-07-30 01:22:52,361 - INFO - [TRAIN] epoch=550/604, iter=63800/70000, loss=0.250520, lr=0.000001 | ETA 00:32:21 2022-07-30 01:23:13,112 - INFO - [TRAIN] epoch=551/604, iter=63850/70000, loss=0.255721, lr=0.000001 | ETA 00:37:23 2022-07-30 01:23:30,918 - INFO - [TRAIN] epoch=551/604, iter=63900/70000, loss=0.314406, lr=0.000001 | ETA 00:33:46 2022-07-30 01:23:50,173 - INFO - [TRAIN] epoch=552/604, iter=63950/70000, loss=0.322346, lr=0.000001 | ETA 00:31:22 2022-07-30 01:24:08,190 - INFO - [TRAIN] epoch=552/604, iter=64000/70000, loss=0.261934, lr=0.000001 | ETA 00:31:41 2022-07-30 01:24:27,717 - INFO - [TRAIN] epoch=553/604, iter=64050/70000, loss=0.279335, lr=0.000001 | ETA 00:32:26 2022-07-30 01:24:44,486 - INFO - [TRAIN] epoch=553/604, iter=64100/70000, loss=0.279384, lr=0.000001 | ETA 00:31:57 2022-07-30 01:25:04,669 - INFO - [TRAIN] epoch=554/604, iter=64150/70000, loss=0.262983, lr=0.000001 | ETA 01:09:07 2022-07-30 01:25:24,448 - INFO - [TRAIN] epoch=554/604, iter=64200/70000, loss=0.267241, lr=0.000001 | ETA 01:09:23 2022-07-30 01:25:42,584 - INFO - [TRAIN] epoch=554/604, iter=64250/70000, loss=0.262080, lr=0.000001 | ETA 00:33:17 2022-07-30 01:26:03,101 - INFO - [TRAIN] epoch=555/604, iter=64300/70000, loss=0.223782, lr=0.000001 | ETA 00:30:27 2022-07-30 01:26:21,719 - INFO - [TRAIN] epoch=555/604, iter=64350/70000, loss=0.267001, lr=0.000001 | ETA 00:40:01 2022-07-30 01:26:40,741 - INFO - [TRAIN] epoch=556/604, iter=64400/70000, loss=0.234849, lr=0.000001 | ETA 00:30:24 2022-07-30 01:26:59,014 - INFO - [TRAIN] epoch=556/604, iter=64450/70000, loss=0.289759, lr=0.000001 | ETA 00:33:36 2022-07-30 01:27:17,352 - INFO - [TRAIN] epoch=557/604, iter=64500/70000, loss=0.290443, lr=0.000001 | ETA 00:38:06 2022-07-30 01:27:36,445 - INFO - [TRAIN] epoch=557/604, iter=64550/70000, loss=0.262954, lr=0.000001 | ETA 00:35:40 2022-07-30 01:27:53,935 - INFO - [TRAIN] epoch=557/604, iter=64600/70000, loss=0.240169, lr=0.000001 | ETA 00:28:11 2022-07-30 01:28:13,106 - INFO - [TRAIN] epoch=558/604, iter=64650/70000, loss=0.265867, lr=0.000001 | ETA 00:28:59 2022-07-30 01:28:30,652 - INFO - [TRAIN] epoch=558/604, iter=64700/70000, loss=0.303782, lr=0.000001 | ETA 00:29:17 2022-07-30 01:28:51,388 - INFO - [TRAIN] epoch=559/604, iter=64750/70000, loss=0.316028, lr=0.000001 | ETA 00:27:56 2022-07-30 01:29:09,268 - INFO - [TRAIN] epoch=559/604, iter=64800/70000, loss=0.245682, lr=0.000001 | ETA 00:27:18 2022-07-30 01:29:28,881 - INFO - [TRAIN] epoch=560/604, iter=64850/70000, loss=0.285787, lr=0.000001 | ETA 00:30:21 2022-07-30 01:29:47,106 - INFO - [TRAIN] epoch=560/604, iter=64900/70000, loss=0.268195, lr=0.000001 | ETA 00:29:22 2022-07-30 01:30:05,166 - INFO - [TRAIN] epoch=560/604, iter=64950/70000, loss=0.253381, lr=0.000001 | ETA 00:30:03 2022-07-30 01:30:23,940 - INFO - [TRAIN] epoch=561/604, iter=65000/70000, loss=0.275832, lr=0.000001 | ETA 00:26:02 2022-07-30 01:30:23,959 - INFO - Pop model from ./output/iter_40000 2022-07-30 01:30:25,125 - INFO - Push model to checkpoint ./output/iter_65000 2022-07-30 01:30:42,099 - INFO - [TRAIN] epoch=561/604, iter=65050/70000, loss=0.291394, lr=0.000001 | ETA 00:26:06 2022-07-30 01:31:02,254 - INFO - [TRAIN] epoch=562/604, iter=65100/70000, loss=0.274956, lr=0.000001 | ETA 00:27:26 2022-07-30 01:31:19,366 - INFO - [TRAIN] epoch=562/604, iter=65150/70000, loss=0.233545, lr=0.000001 | ETA 00:26:37 2022-07-30 01:31:37,972 - INFO - [TRAIN] epoch=563/604, iter=65200/70000, loss=0.287895, lr=0.000001 | ETA 00:27:59 2022-07-30 01:31:57,675 - INFO - [TRAIN] epoch=563/604, iter=65250/70000, loss=0.252423, lr=0.000001 | ETA 00:35:35 2022-07-30 01:32:15,088 - INFO - [TRAIN] epoch=563/604, iter=65300/70000, loss=0.241376, lr=0.000001 | ETA 00:24:52 2022-07-30 01:32:34,632 - INFO - [TRAIN] epoch=564/604, iter=65350/70000, loss=0.269084, lr=0.000001 | ETA 00:27:08 2022-07-30 01:32:51,508 - INFO - [TRAIN] epoch=564/604, iter=65400/70000, loss=0.305498, lr=0.000001 | ETA 00:25:22 2022-07-30 01:33:10,803 - INFO - [TRAIN] epoch=565/604, iter=65450/70000, loss=0.239917, lr=0.000001 | ETA 00:25:57 2022-07-30 01:33:29,650 - INFO - [TRAIN] epoch=565/604, iter=65500/70000, loss=0.267226, lr=0.000001 | ETA 00:39:29 2022-07-30 01:33:47,928 - INFO - [TRAIN] epoch=566/604, iter=65550/70000, loss=0.241825, lr=0.000001 | ETA 00:23:28 2022-07-30 01:34:06,396 - INFO - [TRAIN] epoch=566/604, iter=65600/70000, loss=0.286775, lr=0.000001 | ETA 00:23:21 2022-07-30 01:34:23,492 - INFO - [TRAIN] epoch=566/604, iter=65650/70000, loss=0.229533, lr=0.000001 | ETA 00:22:53 2022-07-30 01:34:43,360 - INFO - [TRAIN] epoch=567/604, iter=65700/70000, loss=0.273868, lr=0.000001 | ETA 00:22:26 2022-07-30 01:35:01,280 - INFO - [TRAIN] epoch=567/604, iter=65750/70000, loss=0.265534, lr=0.000001 | ETA 00:22:52 2022-07-30 01:35:19,916 - INFO - [TRAIN] epoch=568/604, iter=65800/70000, loss=0.290096, lr=0.000001 | ETA 00:28:07 2022-07-30 01:35:38,747 - INFO - [TRAIN] epoch=568/604, iter=65850/70000, loss=0.248779, lr=0.000001 | ETA 00:22:19 2022-07-30 01:35:57,646 - INFO - [TRAIN] epoch=569/604, iter=65900/70000, loss=0.244442, lr=0.000001 | ETA 00:23:38 2022-07-30 01:36:15,723 - INFO - [TRAIN] epoch=569/604, iter=65950/70000, loss=0.244412, lr=0.000001 | ETA 00:21:06 2022-07-30 01:36:33,866 - INFO - [TRAIN] epoch=569/604, iter=66000/70000, loss=0.285541, lr=0.000001 | ETA 00:22:01 2022-07-30 01:36:52,766 - INFO - [TRAIN] epoch=570/604, iter=66050/70000, loss=0.276777, lr=0.000001 | ETA 00:20:55 2022-07-30 01:37:10,664 - INFO - [TRAIN] epoch=570/604, iter=66100/70000, loss=0.252948, lr=0.000001 | ETA 00:20:38 2022-07-30 01:37:29,460 - INFO - [TRAIN] epoch=571/604, iter=66150/70000, loss=0.279895, lr=0.000001 | ETA 00:21:08 2022-07-30 01:37:48,101 - INFO - [TRAIN] epoch=571/604, iter=66200/70000, loss=0.288587, lr=0.000001 | ETA 00:22:48 2022-07-30 01:38:06,547 - INFO - [TRAIN] epoch=572/604, iter=66250/70000, loss=0.252361, lr=0.000001 | ETA 00:20:45 2022-07-30 01:38:24,563 - INFO - [TRAIN] epoch=572/604, iter=66300/70000, loss=0.282118, lr=0.000001 | ETA 00:20:17 2022-07-30 01:38:42,107 - INFO - [TRAIN] epoch=572/604, iter=66350/70000, loss=0.266783, lr=0.000001 | ETA 00:19:01 2022-07-30 01:39:02,240 - INFO - [TRAIN] epoch=573/604, iter=66400/70000, loss=0.274117, lr=0.000001 | ETA 00:28:22 2022-07-30 01:39:19,187 - INFO - [TRAIN] epoch=573/604, iter=66450/70000, loss=0.265370, lr=0.000001 | ETA 00:18:55 2022-07-30 01:39:40,516 - INFO - [TRAIN] epoch=574/604, iter=66500/70000, loss=0.254669, lr=0.000001 | ETA 00:38:27 2022-07-30 01:39:59,193 - INFO - [TRAIN] epoch=574/604, iter=66550/70000, loss=0.238776, lr=0.000001 | ETA 00:18:52 2022-07-30 01:40:19,110 - INFO - [TRAIN] epoch=575/604, iter=66600/70000, loss=0.315677, lr=0.000001 | ETA 00:19:19 2022-07-30 01:40:37,101 - INFO - [TRAIN] epoch=575/604, iter=66650/70000, loss=0.285680, lr=0.000001 | ETA 00:17:28 2022-07-30 01:40:54,899 - INFO - [TRAIN] epoch=575/604, iter=66700/70000, loss=0.288991, lr=0.000001 | ETA 00:17:25 2022-07-30 01:41:14,263 - INFO - [TRAIN] epoch=576/604, iter=66750/70000, loss=0.305429, lr=0.000001 | ETA 00:18:38 2022-07-30 01:41:32,827 - INFO - [TRAIN] epoch=576/604, iter=66800/70000, loss=0.336866, lr=0.000001 | ETA 00:16:40 2022-07-30 01:41:53,221 - INFO - [TRAIN] epoch=577/604, iter=66850/70000, loss=0.256047, lr=0.000001 | ETA 00:19:16 2022-07-30 01:42:11,363 - INFO - [TRAIN] epoch=577/604, iter=66900/70000, loss=0.263803, lr=0.000001 | ETA 00:16:42 2022-07-30 01:42:30,531 - INFO - [TRAIN] epoch=578/604, iter=66950/70000, loss=0.253316, lr=0.000001 | ETA 00:17:12 2022-07-30 01:42:49,956 - INFO - [TRAIN] epoch=578/604, iter=67000/70000, loss=0.294770, lr=0.000001 | ETA 00:16:41 2022-07-30 01:43:08,706 - INFO - [TRAIN] epoch=579/604, iter=67050/70000, loss=0.280612, lr=0.000001 | ETA 00:35:25 2022-07-30 01:43:26,836 - INFO - [TRAIN] epoch=579/604, iter=67100/70000, loss=0.272218, lr=0.000001 | ETA 00:16:03 2022-07-30 01:43:44,693 - INFO - [TRAIN] epoch=579/604, iter=67150/70000, loss=0.268630, lr=0.000001 | ETA 00:15:17 2022-07-30 01:44:04,899 - INFO - [TRAIN] epoch=580/604, iter=67200/70000, loss=0.278034, lr=0.000001 | ETA 00:15:16 2022-07-30 01:44:23,137 - INFO - [TRAIN] epoch=580/604, iter=67250/70000, loss=0.259729, lr=0.000001 | ETA 00:17:22 2022-07-30 01:44:41,939 - INFO - [TRAIN] epoch=581/604, iter=67300/70000, loss=0.280850, lr=0.000001 | ETA 00:14:33 2022-07-30 01:44:59,120 - INFO - [TRAIN] epoch=581/604, iter=67350/70000, loss=0.282743, lr=0.000001 | ETA 00:14:02 2022-07-30 01:45:17,678 - INFO - [TRAIN] epoch=582/604, iter=67400/70000, loss=0.282098, lr=0.000001 | ETA 00:18:23 2022-07-30 01:45:34,963 - INFO - [TRAIN] epoch=582/604, iter=67450/70000, loss=0.248486, lr=0.000001 | ETA 00:15:35 2022-07-30 01:45:51,632 - INFO - [TRAIN] epoch=582/604, iter=67500/70000, loss=0.249826, lr=0.000001 | ETA 00:13:18 2022-07-30 01:46:09,296 - INFO - [TRAIN] epoch=583/604, iter=67550/70000, loss=0.285160, lr=0.000001 | ETA 00:12:51 2022-07-30 01:46:25,204 - INFO - [TRAIN] epoch=583/604, iter=67600/70000, loss=0.256433, lr=0.000001 | ETA 00:12:32 2022-07-30 01:46:42,849 - INFO - [TRAIN] epoch=584/604, iter=67650/70000, loss=0.273447, lr=0.000001 | ETA 00:12:17 2022-07-30 01:46:58,823 - INFO - [TRAIN] epoch=584/604, iter=67700/70000, loss=0.276508, lr=0.000001 | ETA 00:12:20 2022-07-30 01:47:16,247 - INFO - [TRAIN] epoch=585/604, iter=67750/70000, loss=0.313140, lr=0.000001 | ETA 00:13:25 2022-07-30 01:47:32,173 - INFO - [TRAIN] epoch=585/604, iter=67800/70000, loss=0.244233, lr=0.000001 | ETA 00:11:43 2022-07-30 01:47:48,160 - INFO - [TRAIN] epoch=585/604, iter=67850/70000, loss=0.271723, lr=0.000001 | ETA 00:11:11 2022-07-30 01:48:05,693 - INFO - [TRAIN] epoch=586/604, iter=67900/70000, loss=0.238127, lr=0.000001 | ETA 00:11:52 2022-07-30 01:48:21,781 - INFO - [TRAIN] epoch=586/604, iter=67950/70000, loss=0.253043, lr=0.000001 | ETA 00:10:46 2022-07-30 01:48:39,127 - INFO - [TRAIN] epoch=587/604, iter=68000/70000, loss=0.300810, lr=0.000001 | ETA 00:10:36 2022-07-30 01:48:55,049 - INFO - [TRAIN] epoch=587/604, iter=68050/70000, loss=0.279983, lr=0.000001 | ETA 00:10:23 2022-07-30 01:49:12,456 - INFO - [TRAIN] epoch=588/604, iter=68100/70000, loss=0.242906, lr=0.000001 | ETA 00:10:10 2022-07-30 01:49:28,415 - INFO - [TRAIN] epoch=588/604, iter=68150/70000, loss=0.254220, lr=0.000001 | ETA 00:09:54 2022-07-30 01:49:44,450 - INFO - [TRAIN] epoch=588/604, iter=68200/70000, loss=0.264959, lr=0.000001 | ETA 00:09:33 2022-07-30 01:50:02,062 - INFO - [TRAIN] epoch=589/604, iter=68250/70000, loss=0.232904, lr=0.000001 | ETA 00:09:06 2022-07-30 01:50:18,234 - INFO - [TRAIN] epoch=589/604, iter=68300/70000, loss=0.283709, lr=0.000001 | ETA 00:08:58 2022-07-30 01:50:35,744 - INFO - [TRAIN] epoch=590/604, iter=68350/70000, loss=0.250054, lr=0.000001 | ETA 00:08:40 2022-07-30 01:50:52,619 - INFO - [TRAIN] epoch=590/604, iter=68400/70000, loss=0.256971, lr=0.000001 | ETA 00:08:29 2022-07-30 01:51:10,677 - INFO - [TRAIN] epoch=591/604, iter=68450/70000, loss=0.271040, lr=0.000001 | ETA 00:08:19 2022-07-30 01:51:27,022 - INFO - [TRAIN] epoch=591/604, iter=68500/70000, loss=0.234068, lr=0.000001 | ETA 00:07:57 2022-07-30 01:51:44,195 - INFO - [TRAIN] epoch=591/604, iter=68550/70000, loss=0.254789, lr=0.000001 | ETA 00:07:58 2022-07-30 01:52:03,037 - INFO - [TRAIN] epoch=592/604, iter=68600/70000, loss=0.253463, lr=0.000001 | ETA 00:07:32 2022-07-30 01:52:19,843 - INFO - [TRAIN] epoch=592/604, iter=68650/70000, loss=0.260171, lr=0.000001 | ETA 00:07:18 2022-07-30 01:52:38,139 - INFO - [TRAIN] epoch=593/604, iter=68700/70000, loss=0.259426, lr=0.000001 | ETA 00:08:21 2022-07-30 01:52:55,295 - INFO - [TRAIN] epoch=593/604, iter=68750/70000, loss=0.253756, lr=0.000001 | ETA 00:06:49 2022-07-30 01:53:14,008 - INFO - [TRAIN] epoch=594/604, iter=68800/70000, loss=0.228138, lr=0.000001 | ETA 00:06:38 2022-07-30 01:53:30,799 - INFO - [TRAIN] epoch=594/604, iter=68850/70000, loss=0.261776, lr=0.000001 | ETA 00:06:16 2022-07-30 01:53:47,785 - INFO - [TRAIN] epoch=594/604, iter=68900/70000, loss=0.279316, lr=0.000001 | ETA 00:05:58 2022-07-30 01:54:05,934 - INFO - [TRAIN] epoch=595/604, iter=68950/70000, loss=0.256913, lr=0.000001 | ETA 00:05:32 2022-07-30 01:54:22,663 - INFO - [TRAIN] epoch=595/604, iter=69000/70000, loss=0.261895, lr=0.000001 | ETA 00:05:28 2022-07-30 01:54:41,406 - INFO - [TRAIN] epoch=596/604, iter=69050/70000, loss=0.281365, lr=0.000001 | ETA 00:05:06 2022-07-30 01:54:57,478 - INFO - [TRAIN] epoch=596/604, iter=69100/70000, loss=0.309885, lr=0.000001 | ETA 00:04:48 2022-07-30 01:55:15,556 - INFO - [TRAIN] epoch=597/604, iter=69150/70000, loss=0.280833, lr=0.000001 | ETA 00:04:30 2022-07-30 01:55:32,224 - INFO - [TRAIN] epoch=597/604, iter=69200/70000, loss=0.280774, lr=0.000001 | ETA 00:04:41 2022-07-30 01:55:48,929 - INFO - [TRAIN] epoch=597/604, iter=69250/70000, loss=0.269050, lr=0.000001 | ETA 00:03:54 2022-07-30 01:56:07,296 - INFO - [TRAIN] epoch=598/604, iter=69300/70000, loss=0.251719, lr=0.000001 | ETA 00:03:51 2022-07-30 01:56:24,036 - INFO - [TRAIN] epoch=598/604, iter=69350/70000, loss=0.255260, lr=0.000001 | ETA 00:03:22 2022-07-30 01:56:42,370 - INFO - [TRAIN] epoch=599/604, iter=69400/70000, loss=0.244364, lr=0.000001 | ETA 00:03:09 2022-07-30 01:56:59,234 - INFO - [TRAIN] epoch=599/604, iter=69450/70000, loss=0.273271, lr=0.000001 | ETA 00:02:54 2022-07-30 01:57:17,542 - INFO - [TRAIN] epoch=600/604, iter=69500/70000, loss=0.262411, lr=0.000001 | ETA 00:03:22 2022-07-30 01:57:34,435 - INFO - [TRAIN] epoch=600/604, iter=69550/70000, loss=0.247869, lr=0.000001 | ETA 00:02:21 2022-07-30 01:57:51,503 - INFO - [TRAIN] epoch=600/604, iter=69600/70000, loss=0.276703, lr=0.000001 | ETA 00:02:06 2022-07-30 01:58:10,358 - INFO - [TRAIN] epoch=601/604, iter=69650/70000, loss=0.293850, lr=0.000001 | ETA 00:03:58 2022-07-30 01:58:27,198 - INFO - [TRAIN] epoch=601/604, iter=69700/70000, loss=0.291772, lr=0.000001 | ETA 00:01:38 2022-07-30 01:58:45,983 - INFO - [TRAIN] epoch=602/604, iter=69750/70000, loss=0.252726, lr=0.000001 | ETA 00:01:18 2022-07-30 01:59:02,743 - INFO - [TRAIN] epoch=602/604, iter=69800/70000, loss=0.255851, lr=0.000001 | ETA 00:01:03 2022-07-30 01:59:21,045 - INFO - [TRAIN] epoch=603/604, iter=69850/70000, loss=0.274803, lr=0.000001 | ETA 00:01:37 2022-07-30 01:59:38,528 - INFO - [TRAIN] epoch=603/604, iter=69900/70000, loss=0.233389, lr=0.000001 | ETA 00:00:31 2022-07-30 01:59:57,302 - INFO - [TRAIN] epoch=604/604, iter=69950/70000, loss=0.267711, lr=0.000001 | ETA 00:00:36 2022-07-30 02:00:14,416 - INFO - [TRAIN] epoch=604/604, iter=70000/70000, loss=0.285836, lr=0.000001 | ETA 00:00:00 2022-07-30 02:00:14,429 - INFO - Pop model from ./output/iter_45000 2022-07-30 02:00:15,574 - INFO - Push model to checkpoint ./output/iter_70000 2022-07-30 02:00:15,580 - INFO - Training is complete. LAUNCH INFO 2022-07-30 02:00:23,271 Pod completed LAUNCH INFO 2022-07-30 02:00:23,272 Exit code 0