LAUNCH INFO 2022-07-28 19:12:15,639 -----------  Configuration  ----------------------
LAUNCH INFO 2022-07-28 19:12:15,640 devices: None
LAUNCH INFO 2022-07-28 19:12:15,640 elastic_level: -1
LAUNCH INFO 2022-07-28 19:12:15,640 elastic_timeout: 30
LAUNCH INFO 2022-07-28 19:12:15,640 gloo_port: 6767
LAUNCH INFO 2022-07-28 19:12:15,640 host: None
LAUNCH INFO 2022-07-28 19:12:15,640 job_id: default
LAUNCH INFO 2022-07-28 19:12:15,640 legacy: False
LAUNCH INFO 2022-07-28 19:12:15,640 log_dir: log
LAUNCH INFO 2022-07-28 19:12:15,640 log_level: INFO
LAUNCH INFO 2022-07-28 19:12:15,640 master: None
LAUNCH INFO 2022-07-28 19:12:15,640 max_restart: 3
LAUNCH INFO 2022-07-28 19:12:15,640 nnodes: 1
LAUNCH INFO 2022-07-28 19:12:15,640 nproc_per_node: None
LAUNCH INFO 2022-07-28 19:12:15,640 rank: -1
LAUNCH INFO 2022-07-28 19:12:15,640 run_mode: collective
LAUNCH INFO 2022-07-28 19:12:15,640 server_num: None
LAUNCH INFO 2022-07-28 19:12:15,640 servers: 
LAUNCH INFO 2022-07-28 19:12:15,640 trainer_num: None
LAUNCH INFO 2022-07-28 19:12:15,640 trainers: 
LAUNCH INFO 2022-07-28 19:12:15,640 training_script: tools/train.py
LAUNCH INFO 2022-07-28 19:12:15,640 training_script_args: ['--config', 'configs/smoke/smoke_hrnet18_no_dcn_iter70000.yml', '--num_workers', '3', '--log_interval', '50', '--save_interval', '5000']
LAUNCH INFO 2022-07-28 19:12:15,640 with_gloo: 0
LAUNCH INFO 2022-07-28 19:12:15,640 --------------------------------------------------
LAUNCH INFO 2022-07-28 19:12:15,656 Job: default, mode collective, replicas 1[1:1], elastic False
LAUNCH INFO 2022-07-28 19:12:15,657 Run Pod: gvhsde, replicas 4, status ready
LAUNCH INFO 2022-07-28 19:12:15,711 Watching Pod: gvhsde, replicas 4, status running
2022-07-28 19:12:20,836 -     INFO - 
------------Environment Information-------------
platform:
    Linux-3.10.0-1062.18.1.el7.x86_64-x86_64-with-centos-7.7.1908-Core
    gcc (GCC) 9.1.0
    Python - 3.6.13 |Anaconda, Inc.| (default, Jun  4 2021, 14:25:59)  [GCC 7.5.0]

Science Toolkits:
    cv2 - 4.6.0
    numpy - 1.19.5
    numba - 0.53.1
    pandas - 1.1.5
    pillow - 8.4.0
    skimage - 0.17.2

PaddlePaddle:
    paddle(gpu) - 2.3.0
    paddle3d - 0.5.0
    paddleseg - 2.5.0
    FLAGS_cudnn_deterministic - Not set.
    FLAGS_cudnn_exhaustive_search - Not set.

CUDA:
    cudnn - 7605
    nvcc - Cuda compilation tools, release 10.2, V10.2.89

GPUs:
    GPU 0: Tesla V100-SXM2-32GB
    GPU 1: Tesla V100-SXM2-32GB
    GPU 2: Tesla V100-SXM2-32GB
    GPU 3: Tesla V100-SXM2-32GB
------------------------------------------------
2022-07-28 19:12:20,842 -     INFO - 
---------------Config Information---------------
batch_size: 8
iters: 70000
lr_scheduler:
  learning_rate: 0.000125
  milestones:
  - 36000
  - 55000
  type: MultiStepDecay
model:
  backbone:
    pretrained: https://bj.bcebos.com/paddleseg/dygraph/hrnet_w18_ssld.tar.gz
    type: $paddleseg.HRNet_W18
  depth_ref:
  - 28.01
  - 16.32
  dim_ref:
  - - 3.88
    - 1.63
    - 1.53
  - - 1.78
    - 1.7
    - 0.58
  - - 0.88
    - 1.73
    - 0.67
  head:
    in_channels: 270
    norm_type: gn
    num_chanels: 256
    num_classes: 3
    reg_channels:
    - 1
    - 2
    - 3
    - 2
    - 2
    type: SMOKEPredictor
  max_detection: 50
  pred_2d: true
  type: SMOKE
optimizer:
  type: Adam
train_dataset:
  dataset_root: datasets/KITTI
  mode: train
  transforms:
  - reader: pillow
    to_chw: false
    type: LoadImage
  - input_size:
    - 1280
    - 384
    mode: train
    num_classes: 3
    type: Gt2SmokeTarget
  - mean:
    - 0.485
    - 0.456
    - 0.406
    std:
    - 0.229
    - 0.224
    - 0.225
    type: Normalize
  type: KittiMonoDataset
val_dataset:
  dataset_root: datasets/KITTI
  mode: val
  transforms:
  - reader: pillow
    to_chw: false
    type: LoadImage
  - input_size:
    - 1280
    - 384
    mode: val
    num_classes: 3
    type: Gt2SmokeTarget
  - mean:
    - 0.485
    - 0.456
    - 0.406
    std:
    - 0.229
    - 0.224
    - 0.225
    type: Normalize
  type: KittiMonoDataset
------------------------------------------------
W0728 19:12:20.844254 57129 gpu_context.cc:278] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 10.2, Runtime API Version: 10.2
W0728 19:12:20.844295 57129 gpu_context.cc:306] device: 0, cuDNN Version: 7.6.
2022-07-28 19:12:32 [INFO]	Loading pretrained model from https://bj.bcebos.com/paddleseg/dygraph/hrnet_w18_ssld.tar.gz
2022-07-28 19:12:34 [INFO]	There are 1525/1525 variables loaded into HRNet.
server not ready, wait 3 sec to retry...
not ready endpoints:['10.21.226.153:57096', '10.21.226.153:55269']
I0728 19:12:37.624509 57129 nccl_context.cc:83] init nccl context nranks: 4 local rank: 0 gpu id: 0 ring id: 0
/home/wuzewu/miniconda3/envs/paddle3d/lib/python3.6/site-packages/paddle/fluid/dygraph/math_op_patch.py:278: UserWarning: The dtype of left and right variables are not the same, left dtype is paddle.float32, but right dtype is paddle.int32, the right dtype will convert to paddle.float32
  format(lhs_dtype, rhs_dtype, lhs_dtype))
2022-07-28 19:13:22,227 -     INFO - [TRAIN] epoch=1/604, iter=50/70000, loss=29.569356, lr=0.000125 | ETA 14:43:02
2022-07-28 19:14:00,715 -     INFO - [TRAIN] epoch=1/604, iter=100/70000, loss=9.172167, lr=0.000125 | ETA 15:00:46
2022-07-28 19:14:41,191 -     INFO - [TRAIN] epoch=2/604, iter=150/70000, loss=8.519751, lr=0.000125 | ETA 14:49:19
2022-07-28 19:15:20,001 -     INFO - [TRAIN] epoch=2/604, iter=200/70000, loss=7.014320, lr=0.000125 | ETA 15:12:25
2022-07-28 19:16:00,306 -     INFO - [TRAIN] epoch=3/604, iter=250/70000, loss=6.988534, lr=0.000125 | ETA 15:32:36
2022-07-28 19:16:39,641 -     INFO - [TRAIN] epoch=3/604, iter=300/70000, loss=6.488680, lr=0.000125 | ETA 14:56:01
2022-07-28 19:17:19,923 -     INFO - [TRAIN] epoch=4/604, iter=350/70000, loss=6.248091, lr=0.000125 | ETA 22:47:38
2022-07-28 19:17:58,868 -     INFO - [TRAIN] epoch=4/604, iter=400/70000, loss=5.568610, lr=0.000125 | ETA 14:46:52
2022-07-28 19:18:38,184 -     INFO - [TRAIN] epoch=4/604, iter=450/70000, loss=5.453793, lr=0.000125 | ETA 15:26:46
2022-07-28 19:19:19,267 -     INFO - [TRAIN] epoch=5/604, iter=500/70000, loss=5.237693, lr=0.000125 | ETA 15:02:11
2022-07-28 19:19:58,208 -     INFO - [TRAIN] epoch=5/604, iter=550/70000, loss=5.341011, lr=0.000125 | ETA 15:03:29
2022-07-28 19:20:38,893 -     INFO - [TRAIN] epoch=6/604, iter=600/70000, loss=4.865967, lr=0.000125 | ETA 14:51:36
2022-07-28 19:21:17,910 -     INFO - [TRAIN] epoch=6/604, iter=650/70000, loss=5.263319, lr=0.000125 | ETA 14:47:09
2022-07-28 19:21:58,089 -     INFO - [TRAIN] epoch=7/604, iter=700/70000, loss=5.007252, lr=0.000125 | ETA 16:42:03
2022-07-28 19:22:37,164 -     INFO - [TRAIN] epoch=7/604, iter=750/70000, loss=4.592325, lr=0.000125 | ETA 15:03:29
2022-07-28 19:23:15,879 -     INFO - [TRAIN] epoch=7/604, iter=800/70000, loss=4.566995, lr=0.000125 | ETA 15:09:49
2022-07-28 19:23:56,194 -     INFO - [TRAIN] epoch=8/604, iter=850/70000, loss=4.292913, lr=0.000125 | ETA 15:10:44
2022-07-28 19:24:35,821 -     INFO - [TRAIN] epoch=8/604, iter=900/70000, loss=3.705908, lr=0.000125 | ETA 15:25:58
2022-07-28 19:25:17,104 -     INFO - [TRAIN] epoch=9/604, iter=950/70000, loss=4.094101, lr=0.000125 | ETA 15:04:27
2022-07-28 19:25:56,876 -     INFO - [TRAIN] epoch=9/604, iter=1000/70000, loss=3.682915, lr=0.000125 | ETA 15:01:18
2022-07-28 19:26:38,234 -     INFO - [TRAIN] epoch=10/604, iter=1050/70000, loss=3.915973, lr=0.000125 | ETA 15:20:33
2022-07-28 19:27:17,120 -     INFO - [TRAIN] epoch=10/604, iter=1100/70000, loss=3.572447, lr=0.000125 | ETA 14:50:03
2022-07-28 19:27:55,847 -     INFO - [TRAIN] epoch=10/604, iter=1150/70000, loss=3.619666, lr=0.000125 | ETA 15:02:41
2022-07-28 19:28:36,504 -     INFO - [TRAIN] epoch=11/604, iter=1200/70000, loss=3.596467, lr=0.000125 | ETA 15:10:55
2022-07-28 19:29:15,319 -     INFO - [TRAIN] epoch=11/604, iter=1250/70000, loss=3.569915, lr=0.000125 | ETA 14:40:40
2022-07-28 19:29:55,461 -     INFO - [TRAIN] epoch=12/604, iter=1300/70000, loss=3.369740, lr=0.000125 | ETA 14:58:40
2022-07-28 19:30:34,181 -     INFO - [TRAIN] epoch=12/604, iter=1350/70000, loss=3.317513, lr=0.000125 | ETA 14:50:40
2022-07-28 19:31:15,158 -     INFO - [TRAIN] epoch=13/604, iter=1400/70000, loss=2.963429, lr=0.000125 | ETA 14:57:47
2022-07-28 19:31:53,922 -     INFO - [TRAIN] epoch=13/604, iter=1450/70000, loss=2.996324, lr=0.000125 | ETA 14:48:58
2022-07-28 19:32:32,803 -     INFO - [TRAIN] epoch=13/604, iter=1500/70000, loss=3.111676, lr=0.000125 | ETA 14:40:49
2022-07-28 19:33:13,336 -     INFO - [TRAIN] epoch=14/604, iter=1550/70000, loss=3.120995, lr=0.000125 | ETA 14:47:25
2022-07-28 19:33:52,281 -     INFO - [TRAIN] epoch=14/604, iter=1600/70000, loss=3.142760, lr=0.000125 | ETA 14:43:37
2022-07-28 19:34:33,459 -     INFO - [TRAIN] epoch=15/604, iter=1650/70000, loss=3.150092, lr=0.000125 | ETA 15:12:19
2022-07-28 19:35:12,432 -     INFO - [TRAIN] epoch=15/604, iter=1700/70000, loss=2.909386, lr=0.000125 | ETA 14:48:14
2022-07-28 19:35:52,673 -     INFO - [TRAIN] epoch=16/604, iter=1750/70000, loss=2.686850, lr=0.000125 | ETA 14:50:57
2022-07-28 19:36:32,689 -     INFO - [TRAIN] epoch=16/604, iter=1800/70000, loss=2.857958, lr=0.000125 | ETA 14:59:56
2022-07-28 19:37:12,684 -     INFO - [TRAIN] epoch=16/604, iter=1850/70000, loss=2.781331, lr=0.000125 | ETA 14:43:51
2022-07-28 19:37:52,930 -     INFO - [TRAIN] epoch=17/604, iter=1900/70000, loss=2.847682, lr=0.000125 | ETA 14:27:12
2022-07-28 19:38:31,531 -     INFO - [TRAIN] epoch=17/604, iter=1950/70000, loss=2.823695, lr=0.000125 | ETA 14:23:12
2022-07-28 19:39:12,032 -     INFO - [TRAIN] epoch=18/604, iter=2000/70000, loss=2.554484, lr=0.000125 | ETA 14:28:14
2022-07-28 19:39:50,729 -     INFO - [TRAIN] epoch=18/604, iter=2050/70000, loss=2.572089, lr=0.000125 | ETA 14:21:38
2022-07-28 19:40:31,281 -     INFO - [TRAIN] epoch=19/604, iter=2100/70000, loss=2.378567, lr=0.000125 | ETA 14:36:31
2022-07-28 19:41:10,123 -     INFO - [TRAIN] epoch=19/604, iter=2150/70000, loss=2.556400, lr=0.000125 | ETA 14:59:05
2022-07-28 19:41:49,002 -     INFO - [TRAIN] epoch=19/604, iter=2200/70000, loss=2.752879, lr=0.000125 | ETA 14:17:47
2022-07-28 19:42:29,600 -     INFO - [TRAIN] epoch=20/604, iter=2250/70000, loss=2.497516, lr=0.000125 | ETA 14:26:56
2022-07-28 19:43:08,606 -     INFO - [TRAIN] epoch=20/604, iter=2300/70000, loss=2.536241, lr=0.000125 | ETA 14:51:22
2022-07-28 19:43:49,508 -     INFO - [TRAIN] epoch=21/604, iter=2350/70000, loss=2.577360, lr=0.000125 | ETA 14:47:07
2022-07-28 19:44:28,339 -     INFO - [TRAIN] epoch=21/604, iter=2400/70000, loss=2.436629, lr=0.000125 | ETA 14:36:54
2022-07-28 19:45:08,663 -     INFO - [TRAIN] epoch=22/604, iter=2450/70000, loss=2.203816, lr=0.000125 | ETA 14:42:40
2022-07-28 19:45:47,459 -     INFO - [TRAIN] epoch=22/604, iter=2500/70000, loss=2.559406, lr=0.000125 | ETA 14:24:31
2022-07-28 19:46:26,332 -     INFO - [TRAIN] epoch=22/604, iter=2550/70000, loss=2.264664, lr=0.000125 | ETA 14:22:34
2022-07-28 19:47:06,845 -     INFO - [TRAIN] epoch=23/604, iter=2600/70000, loss=2.467072, lr=0.000125 | ETA 14:20:15
2022-07-28 19:47:46,161 -     INFO - [TRAIN] epoch=23/604, iter=2650/70000, loss=2.088249, lr=0.000125 | ETA 14:49:57
2022-07-28 19:48:26,789 -     INFO - [TRAIN] epoch=24/604, iter=2700/70000, loss=2.302872, lr=0.000125 | ETA 14:21:49
2022-07-28 19:49:06,570 -     INFO - [TRAIN] epoch=24/604, iter=2750/70000, loss=2.259560, lr=0.000125 | ETA 14:49:04
2022-07-28 19:49:48,276 -     INFO - [TRAIN] epoch=25/604, iter=2800/70000, loss=2.145419, lr=0.000125 | ETA 14:48:08
2022-07-28 19:50:27,152 -     INFO - [TRAIN] epoch=25/604, iter=2850/70000, loss=2.142180, lr=0.000125 | ETA 14:38:13
2022-07-28 19:51:06,601 -     INFO - [TRAIN] epoch=25/604, iter=2900/70000, loss=2.307938, lr=0.000125 | ETA 14:44:23
2022-07-28 19:51:48,175 -     INFO - [TRAIN] epoch=26/604, iter=2950/70000, loss=2.190021, lr=0.000125 | ETA 14:35:47
2022-07-28 19:52:28,091 -     INFO - [TRAIN] epoch=26/604, iter=3000/70000, loss=2.120543, lr=0.000125 | ETA 14:29:13
2022-07-28 19:53:08,772 -     INFO - [TRAIN] epoch=27/604, iter=3050/70000, loss=2.131144, lr=0.000125 | ETA 14:31:22
2022-07-28 19:53:47,859 -     INFO - [TRAIN] epoch=27/604, iter=3100/70000, loss=2.059775, lr=0.000125 | ETA 14:29:37
2022-07-28 19:54:28,327 -     INFO - [TRAIN] epoch=28/604, iter=3150/70000, loss=2.292354, lr=0.000125 | ETA 14:14:36
2022-07-28 19:55:07,095 -     INFO - [TRAIN] epoch=28/604, iter=3200/70000, loss=2.079325, lr=0.000125 | ETA 14:42:45
2022-07-28 19:55:47,493 -     INFO - [TRAIN] epoch=29/604, iter=3250/70000, loss=1.941279, lr=0.000125 | ETA 22:20:54
2022-07-28 19:56:26,056 -     INFO - [TRAIN] epoch=29/604, iter=3300/70000, loss=2.040657, lr=0.000125 | ETA 14:09:31
2022-07-28 19:57:04,632 -     INFO - [TRAIN] epoch=29/604, iter=3350/70000, loss=1.860450, lr=0.000125 | ETA 14:06:49
2022-07-28 19:57:45,015 -     INFO - [TRAIN] epoch=30/604, iter=3400/70000, loss=2.046053, lr=0.000125 | ETA 14:37:46
2022-07-28 19:58:24,014 -     INFO - [TRAIN] epoch=30/604, iter=3450/70000, loss=2.093495, lr=0.000125 | ETA 14:12:26
2022-07-28 19:59:04,500 -     INFO - [TRAIN] epoch=31/604, iter=3500/70000, loss=1.821450, lr=0.000125 | ETA 14:09:17
2022-07-28 19:59:43,744 -     INFO - [TRAIN] epoch=31/604, iter=3550/70000, loss=1.998256, lr=0.000125 | ETA 14:48:00
2022-07-28 20:00:24,618 -     INFO - [TRAIN] epoch=32/604, iter=3600/70000, loss=1.969081, lr=0.000125 | ETA 16:44:09
2022-07-28 20:01:04,400 -     INFO - [TRAIN] epoch=32/604, iter=3650/70000, loss=1.817239, lr=0.000125 | ETA 14:22:21
2022-07-28 20:01:43,147 -     INFO - [TRAIN] epoch=32/604, iter=3700/70000, loss=1.947206, lr=0.000125 | ETA 14:20:12
2022-07-28 20:02:24,445 -     INFO - [TRAIN] epoch=33/604, iter=3750/70000, loss=1.922046, lr=0.000125 | ETA 14:36:08
2022-07-28 20:03:04,101 -     INFO - [TRAIN] epoch=33/604, iter=3800/70000, loss=2.010849, lr=0.000125 | ETA 14:19:18
2022-07-28 20:03:45,275 -     INFO - [TRAIN] epoch=34/604, iter=3850/70000, loss=1.863584, lr=0.000125 | ETA 14:29:24
2022-07-28 20:04:25,227 -     INFO - [TRAIN] epoch=34/604, iter=3900/70000, loss=1.783629, lr=0.000125 | ETA 14:34:20
2022-07-28 20:05:06,678 -     INFO - [TRAIN] epoch=35/604, iter=3950/70000, loss=1.960289, lr=0.000125 | ETA 16:07:17
2022-07-28 20:05:47,017 -     INFO - [TRAIN] epoch=35/604, iter=4000/70000, loss=1.844544, lr=0.000125 | ETA 14:19:13
2022-07-28 20:06:26,862 -     INFO - [TRAIN] epoch=35/604, iter=4050/70000, loss=1.904098, lr=0.000125 | ETA 14:36:36
2022-07-28 20:07:08,170 -     INFO - [TRAIN] epoch=36/604, iter=4100/70000, loss=2.125352, lr=0.000125 | ETA 14:21:41
2022-07-28 20:07:47,643 -     INFO - [TRAIN] epoch=36/604, iter=4150/70000, loss=1.869399, lr=0.000125 | ETA 14:14:05
2022-07-28 20:08:29,166 -     INFO - [TRAIN] epoch=37/604, iter=4200/70000, loss=1.906155, lr=0.000125 | ETA 14:35:27
2022-07-28 20:09:09,776 -     INFO - [TRAIN] epoch=37/604, iter=4250/70000, loss=1.798516, lr=0.000125 | ETA 14:14:08
2022-07-28 20:09:51,940 -     INFO - [TRAIN] epoch=38/604, iter=4300/70000, loss=2.065342, lr=0.000125 | ETA 15:45:49
2022-07-28 20:10:33,039 -     INFO - [TRAIN] epoch=38/604, iter=4350/70000, loss=1.761970, lr=0.000125 | ETA 15:23:36
2022-07-28 20:11:13,445 -     INFO - [TRAIN] epoch=38/604, iter=4400/70000, loss=1.926878, lr=0.000125 | ETA 15:11:17
2022-07-28 20:11:55,587 -     INFO - [TRAIN] epoch=39/604, iter=4450/70000, loss=1.796247, lr=0.000125 | ETA 14:20:52
2022-07-28 20:12:37,337 -     INFO - [TRAIN] epoch=39/604, iter=4500/70000, loss=1.734532, lr=0.000125 | ETA 14:27:14
2022-07-28 20:13:19,289 -     INFO - [TRAIN] epoch=40/604, iter=4550/70000, loss=1.725379, lr=0.000125 | ETA 14:09:06
2022-07-28 20:14:00,006 -     INFO - [TRAIN] epoch=40/604, iter=4600/70000, loss=1.665894, lr=0.000125 | ETA 14:34:07
2022-07-28 20:14:42,895 -     INFO - [TRAIN] epoch=41/604, iter=4650/70000, loss=1.847432, lr=0.000125 | ETA 14:40:42
2022-07-28 20:15:23,390 -     INFO - [TRAIN] epoch=41/604, iter=4700/70000, loss=1.566436, lr=0.000125 | ETA 14:24:05
2022-07-28 20:16:03,821 -     INFO - [TRAIN] epoch=41/604, iter=4750/70000, loss=1.837183, lr=0.000125 | ETA 14:34:30
2022-07-28 20:16:46,588 -     INFO - [TRAIN] epoch=42/604, iter=4800/70000, loss=1.589155, lr=0.000125 | ETA 15:29:49
2022-07-28 20:17:27,781 -     INFO - [TRAIN] epoch=42/604, iter=4850/70000, loss=1.821338, lr=0.000125 | ETA 15:10:47
2022-07-28 20:18:10,940 -     INFO - [TRAIN] epoch=43/604, iter=4900/70000, loss=1.729595, lr=0.000125 | ETA 15:17:39
2022-07-28 20:18:51,910 -     INFO - [TRAIN] epoch=43/604, iter=4950/70000, loss=1.529425, lr=0.000125 | ETA 14:35:20
2022-07-28 20:19:35,089 -     INFO - [TRAIN] epoch=44/604, iter=5000/70000, loss=1.571196, lr=0.000125 | ETA 16:46:38
2022-07-28 20:19:36,140 -     INFO - Push model to checkpoint ./output/iter_5000
2022-07-28 20:20:17,229 -     INFO - [TRAIN] epoch=44/604, iter=5050/70000, loss=2.109184, lr=0.000125 | ETA 14:04:54
2022-07-28 20:20:58,073 -     INFO - [TRAIN] epoch=44/604, iter=5100/70000, loss=1.791398, lr=0.000125 | ETA 14:27:54
2022-07-28 20:21:41,366 -     INFO - [TRAIN] epoch=45/604, iter=5150/70000, loss=1.710760, lr=0.000125 | ETA 14:28:45
2022-07-28 20:22:23,520 -     INFO - [TRAIN] epoch=45/604, iter=5200/70000, loss=1.592057, lr=0.000125 | ETA 14:49:02
2022-07-28 20:23:06,779 -     INFO - [TRAIN] epoch=46/604, iter=5250/70000, loss=1.581434, lr=0.000125 | ETA 15:44:35
2022-07-28 20:23:47,763 -     INFO - [TRAIN] epoch=46/604, iter=5300/70000, loss=1.576128, lr=0.000125 | ETA 15:16:55
2022-07-28 20:24:30,074 -     INFO - [TRAIN] epoch=47/604, iter=5350/70000, loss=1.719603, lr=0.000125 | ETA 15:12:55
2022-07-28 20:25:11,612 -     INFO - [TRAIN] epoch=47/604, iter=5400/70000, loss=1.564866, lr=0.000125 | ETA 15:55:16
2022-07-28 20:25:53,103 -     INFO - [TRAIN] epoch=47/604, iter=5450/70000, loss=1.588238, lr=0.000125 | ETA 13:47:29
2022-07-28 20:26:35,540 -     INFO - [TRAIN] epoch=48/604, iter=5500/70000, loss=1.434050, lr=0.000125 | ETA 14:07:34
2022-07-28 20:27:17,214 -     INFO - [TRAIN] epoch=48/604, iter=5550/70000, loss=1.731218, lr=0.000125 | ETA 14:10:43
2022-07-28 20:28:00,886 -     INFO - [TRAIN] epoch=49/604, iter=5600/70000, loss=1.497462, lr=0.000125 | ETA 15:40:51
2022-07-28 20:28:42,062 -     INFO - [TRAIN] epoch=49/604, iter=5650/70000, loss=1.443073, lr=0.000125 | ETA 13:51:45
2022-07-28 20:29:24,707 -     INFO - [TRAIN] epoch=50/604, iter=5700/70000, loss=1.567569, lr=0.000125 | ETA 15:08:19
2022-07-28 20:30:05,021 -     INFO - [TRAIN] epoch=50/604, iter=5750/70000, loss=1.606782, lr=0.000125 | ETA 14:01:34
2022-07-28 20:30:45,848 -     INFO - [TRAIN] epoch=50/604, iter=5800/70000, loss=1.332104, lr=0.000125 | ETA 13:41:22
2022-07-28 20:31:29,621 -     INFO - [TRAIN] epoch=51/604, iter=5850/70000, loss=1.531975, lr=0.000125 | ETA 16:44:24
2022-07-28 20:32:12,010 -     INFO - [TRAIN] epoch=51/604, iter=5900/70000, loss=1.432055, lr=0.000125 | ETA 16:06:29
2022-07-28 20:32:55,166 -     INFO - [TRAIN] epoch=52/604, iter=5950/70000, loss=1.437537, lr=0.000125 | ETA 18:11:48
2022-07-28 20:33:36,290 -     INFO - [TRAIN] epoch=52/604, iter=6000/70000, loss=1.531217, lr=0.000125 | ETA 14:08:44
2022-07-28 20:34:19,692 -     INFO - [TRAIN] epoch=53/604, iter=6050/70000, loss=1.455887, lr=0.000125 | ETA 18:03:37
2022-07-28 20:35:02,195 -     INFO - [TRAIN] epoch=53/604, iter=6100/70000, loss=1.566358, lr=0.000125 | ETA 13:48:28
2022-07-28 20:35:44,941 -     INFO - [TRAIN] epoch=54/604, iter=6150/70000, loss=1.380217, lr=0.000125 | ETA 22:02:25
2022-07-28 20:36:26,684 -     INFO - [TRAIN] epoch=54/604, iter=6200/70000, loss=1.396390, lr=0.000125 | ETA 13:52:13
2022-07-28 20:37:08,233 -     INFO - [TRAIN] epoch=54/604, iter=6250/70000, loss=1.356744, lr=0.000125 | ETA 14:46:49
2022-07-28 20:37:51,995 -     INFO - [TRAIN] epoch=55/604, iter=6300/70000, loss=1.408759, lr=0.000125 | ETA 15:55:39
2022-07-28 20:38:34,352 -     INFO - [TRAIN] epoch=55/604, iter=6350/70000, loss=1.422529, lr=0.000125 | ETA 17:04:03
2022-07-28 20:39:17,305 -     INFO - [TRAIN] epoch=56/604, iter=6400/70000, loss=1.346506, lr=0.000125 | ETA 14:37:12
2022-07-28 20:39:58,652 -     INFO - [TRAIN] epoch=56/604, iter=6450/70000, loss=1.529631, lr=0.000125 | ETA 18:11:31
2022-07-28 20:40:41,100 -     INFO - [TRAIN] epoch=57/604, iter=6500/70000, loss=1.437675, lr=0.000125 | ETA 16:03:39
2022-07-28 20:41:23,323 -     INFO - [TRAIN] epoch=57/604, iter=6550/70000, loss=1.487311, lr=0.000125 | ETA 13:39:54
2022-07-28 20:42:05,014 -     INFO - [TRAIN] epoch=57/604, iter=6600/70000, loss=1.406337, lr=0.000125 | ETA 13:47:14
2022-07-28 20:42:48,106 -     INFO - [TRAIN] epoch=58/604, iter=6650/70000, loss=1.377470, lr=0.000125 | ETA 13:57:14
2022-07-28 20:43:30,954 -     INFO - [TRAIN] epoch=58/604, iter=6700/70000, loss=1.369408, lr=0.000125 | ETA 14:13:41
2022-07-28 20:44:14,933 -     INFO - [TRAIN] epoch=59/604, iter=6750/70000, loss=1.238117, lr=0.000125 | ETA 14:22:39
2022-07-28 20:44:57,181 -     INFO - [TRAIN] epoch=59/604, iter=6800/70000, loss=1.471097, lr=0.000125 | ETA 14:41:23
2022-07-28 20:45:39,771 -     INFO - [TRAIN] epoch=60/604, iter=6850/70000, loss=1.318835, lr=0.000125 | ETA 15:28:43
2022-07-28 20:46:21,105 -     INFO - [TRAIN] epoch=60/604, iter=6900/70000, loss=1.251798, lr=0.000125 | ETA 14:16:18
2022-07-28 20:47:02,604 -     INFO - [TRAIN] epoch=60/604, iter=6950/70000, loss=1.209221, lr=0.000125 | ETA 14:00:14
2022-07-28 20:47:45,629 -     INFO - [TRAIN] epoch=61/604, iter=7000/70000, loss=1.375493, lr=0.000125 | ETA 13:35:01
2022-07-28 20:48:26,183 -     INFO - [TRAIN] epoch=61/604, iter=7050/70000, loss=1.362801, lr=0.000125 | ETA 14:52:05
2022-07-28 20:49:09,540 -     INFO - [TRAIN] epoch=62/604, iter=7100/70000, loss=1.301373, lr=0.000125 | ETA 14:22:56
2022-07-28 20:49:52,571 -     INFO - [TRAIN] epoch=62/604, iter=7150/70000, loss=1.303963, lr=0.000125 | ETA 14:13:42
2022-07-28 20:50:35,689 -     INFO - [TRAIN] epoch=63/604, iter=7200/70000, loss=1.284128, lr=0.000125 | ETA 14:39:58
2022-07-28 20:51:16,841 -     INFO - [TRAIN] epoch=63/604, iter=7250/70000, loss=1.421196, lr=0.000125 | ETA 14:46:13
2022-07-28 20:51:57,879 -     INFO - [TRAIN] epoch=63/604, iter=7300/70000, loss=1.441461, lr=0.000125 | ETA 13:49:13
2022-07-28 20:52:41,508 -     INFO - [TRAIN] epoch=64/604, iter=7350/70000, loss=1.230724, lr=0.000125 | ETA 16:58:55
2022-07-28 20:53:22,751 -     INFO - [TRAIN] epoch=64/604, iter=7400/70000, loss=1.311334, lr=0.000125 | ETA 14:33:43
2022-07-28 20:54:06,359 -     INFO - [TRAIN] epoch=65/604, iter=7450/70000, loss=1.278392, lr=0.000125 | ETA 13:49:27
2022-07-28 20:54:47,990 -     INFO - [TRAIN] epoch=65/604, iter=7500/70000, loss=1.226051, lr=0.000125 | ETA 14:11:28
2022-07-28 20:55:30,947 -     INFO - [TRAIN] epoch=66/604, iter=7550/70000, loss=1.342020, lr=0.000125 | ETA 13:46:54
2022-07-28 20:56:12,609 -     INFO - [TRAIN] epoch=66/604, iter=7600/70000, loss=1.206631, lr=0.000125 | ETA 16:03:32
2022-07-28 20:56:53,943 -     INFO - [TRAIN] epoch=66/604, iter=7650/70000, loss=1.267441, lr=0.000125 | ETA 13:48:37
2022-07-28 20:57:37,107 -     INFO - [TRAIN] epoch=67/604, iter=7700/70000, loss=1.237896, lr=0.000125 | ETA 13:57:37
2022-07-28 20:58:18,575 -     INFO - [TRAIN] epoch=67/604, iter=7750/70000, loss=1.306515, lr=0.000125 | ETA 14:02:31
2022-07-28 20:59:02,200 -     INFO - [TRAIN] epoch=68/604, iter=7800/70000, loss=1.256024, lr=0.000125 | ETA 18:36:51
2022-07-28 20:59:43,421 -     INFO - [TRAIN] epoch=68/604, iter=7850/70000, loss=1.214990, lr=0.000125 | ETA 15:45:19
2022-07-28 21:00:26,275 -     INFO - [TRAIN] epoch=69/604, iter=7900/70000, loss=1.211498, lr=0.000125 | ETA 13:54:12
2022-07-28 21:01:08,010 -     INFO - [TRAIN] epoch=69/604, iter=7950/70000, loss=1.244405, lr=0.000125 | ETA 14:14:30
2022-07-28 21:01:48,566 -     INFO - [TRAIN] epoch=69/604, iter=8000/70000, loss=1.208071, lr=0.000125 | ETA 13:35:26
2022-07-28 21:02:32,285 -     INFO - [TRAIN] epoch=70/604, iter=8050/70000, loss=1.132973, lr=0.000125 | ETA 15:30:33
2022-07-28 21:03:13,507 -     INFO - [TRAIN] epoch=70/604, iter=8100/70000, loss=1.184798, lr=0.000125 | ETA 13:20:49
2022-07-28 21:03:56,710 -     INFO - [TRAIN] epoch=71/604, iter=8150/70000, loss=1.193161, lr=0.000125 | ETA 15:15:24
2022-07-28 21:04:37,965 -     INFO - [TRAIN] epoch=71/604, iter=8200/70000, loss=1.190105, lr=0.000125 | ETA 13:24:00
2022-07-28 21:05:21,161 -     INFO - [TRAIN] epoch=72/604, iter=8250/70000, loss=1.279973, lr=0.000125 | ETA 13:29:48
2022-07-28 21:06:02,866 -     INFO - [TRAIN] epoch=72/604, iter=8300/70000, loss=1.253047, lr=0.000125 | ETA 14:33:43
2022-07-28 21:06:44,526 -     INFO - [TRAIN] epoch=72/604, iter=8350/70000, loss=1.137502, lr=0.000125 | ETA 13:38:47
2022-07-28 21:07:28,019 -     INFO - [TRAIN] epoch=73/604, iter=8400/70000, loss=1.141306, lr=0.000125 | ETA 15:52:32
2022-07-28 21:08:09,245 -     INFO - [TRAIN] epoch=73/604, iter=8450/70000, loss=1.191826, lr=0.000125 | ETA 13:59:24
2022-07-28 21:08:52,503 -     INFO - [TRAIN] epoch=74/604, iter=8500/70000, loss=1.140224, lr=0.000125 | ETA 14:11:47
2022-07-28 21:09:34,809 -     INFO - [TRAIN] epoch=74/604, iter=8550/70000, loss=1.238264, lr=0.000125 | ETA 14:00:22
2022-07-28 21:10:17,223 -     INFO - [TRAIN] epoch=75/604, iter=8600/70000, loss=1.204592, lr=0.000125 | ETA 13:38:44
2022-07-28 21:10:59,121 -     INFO - [TRAIN] epoch=75/604, iter=8650/70000, loss=1.158238, lr=0.000125 | ETA 13:33:44
2022-07-28 21:11:40,225 -     INFO - [TRAIN] epoch=75/604, iter=8700/70000, loss=1.076705, lr=0.000125 | ETA 13:04:57
2022-07-28 21:12:24,185 -     INFO - [TRAIN] epoch=76/604, iter=8750/70000, loss=1.230747, lr=0.000125 | ETA 13:31:41
2022-07-28 21:13:05,218 -     INFO - [TRAIN] epoch=76/604, iter=8800/70000, loss=1.131504, lr=0.000125 | ETA 13:52:47
2022-07-28 21:13:48,313 -     INFO - [TRAIN] epoch=77/604, iter=8850/70000, loss=1.159226, lr=0.000125 | ETA 13:41:03
2022-07-28 21:14:30,260 -     INFO - [TRAIN] epoch=77/604, iter=8900/70000, loss=1.084713, lr=0.000125 | ETA 15:04:44
2022-07-28 21:15:13,267 -     INFO - [TRAIN] epoch=78/604, iter=8950/70000, loss=1.192129, lr=0.000125 | ETA 13:40:28
2022-07-28 21:15:55,476 -     INFO - [TRAIN] epoch=78/604, iter=9000/70000, loss=1.125049, lr=0.000125 | ETA 13:53:34
2022-07-28 21:16:39,165 -     INFO - [TRAIN] epoch=79/604, iter=9050/70000, loss=1.064772, lr=0.000125 | ETA 21:03:41
2022-07-28 21:17:20,313 -     INFO - [TRAIN] epoch=79/604, iter=9100/70000, loss=1.165547, lr=0.000125 | ETA 13:29:36
2022-07-28 21:18:01,774 -     INFO - [TRAIN] epoch=79/604, iter=9150/70000, loss=1.039857, lr=0.000125 | ETA 13:38:49
2022-07-28 21:18:45,315 -     INFO - [TRAIN] epoch=80/604, iter=9200/70000, loss=1.140483, lr=0.000125 | ETA 13:19:36
2022-07-28 21:19:26,342 -     INFO - [TRAIN] epoch=80/604, iter=9250/70000, loss=1.058024, lr=0.000125 | ETA 13:29:20
2022-07-28 21:20:09,797 -     INFO - [TRAIN] epoch=81/604, iter=9300/70000, loss=1.127095, lr=0.000125 | ETA 17:33:10
2022-07-28 21:20:50,623 -     INFO - [TRAIN] epoch=81/604, iter=9350/70000, loss=1.117979, lr=0.000125 | ETA 14:01:51
2022-07-28 21:21:32,843 -     INFO - [TRAIN] epoch=82/604, iter=9400/70000, loss=1.063727, lr=0.000125 | ETA 15:24:25
2022-07-28 21:22:15,616 -     INFO - [TRAIN] epoch=82/604, iter=9450/70000, loss=1.037530, lr=0.000125 | ETA 13:39:05
2022-07-28 21:22:57,510 -     INFO - [TRAIN] epoch=82/604, iter=9500/70000, loss=1.174424, lr=0.000125 | ETA 14:10:18
2022-07-28 21:23:40,488 -     INFO - [TRAIN] epoch=83/604, iter=9550/70000, loss=1.075915, lr=0.000125 | ETA 15:24:48
2022-07-28 21:24:21,735 -     INFO - [TRAIN] epoch=83/604, iter=9600/70000, loss=1.031068, lr=0.000125 | ETA 13:35:12
2022-07-28 21:25:04,557 -     INFO - [TRAIN] epoch=84/604, iter=9650/70000, loss=0.990268, lr=0.000125 | ETA 13:20:13
2022-07-28 21:25:45,785 -     INFO - [TRAIN] epoch=84/604, iter=9700/70000, loss=1.098864, lr=0.000125 | ETA 13:18:41
2022-07-28 21:26:28,471 -     INFO - [TRAIN] epoch=85/604, iter=9750/70000, loss=1.131036, lr=0.000125 | ETA 13:16:42
2022-07-28 21:27:10,027 -     INFO - [TRAIN] epoch=85/604, iter=9800/70000, loss=1.074549, lr=0.000125 | ETA 15:05:27
2022-07-28 21:27:51,353 -     INFO - [TRAIN] epoch=85/604, iter=9850/70000, loss=1.067233, lr=0.000125 | ETA 13:45:50
2022-07-28 21:28:34,455 -     INFO - [TRAIN] epoch=86/604, iter=9900/70000, loss=0.980036, lr=0.000125 | ETA 13:34:17
2022-07-28 21:29:15,836 -     INFO - [TRAIN] epoch=86/604, iter=9950/70000, loss=1.082063, lr=0.000125 | ETA 13:07:48
2022-07-28 21:29:59,522 -     INFO - [TRAIN] epoch=87/604, iter=10000/70000, loss=1.000701, lr=0.000125 | ETA 13:58:34
2022-07-28 21:30:00,421 -     INFO - Push model to checkpoint ./output/iter_10000
2022-07-28 21:30:42,170 -     INFO - [TRAIN] epoch=87/604, iter=10050/70000, loss=1.005841, lr=0.000125 | ETA 12:57:23
2022-07-28 21:31:25,374 -     INFO - [TRAIN] epoch=88/604, iter=10100/70000, loss=0.933013, lr=0.000125 | ETA 14:21:01
2022-07-28 21:32:07,222 -     INFO - [TRAIN] epoch=88/604, iter=10150/70000, loss=0.985456, lr=0.000125 | ETA 13:53:33
2022-07-28 21:32:48,446 -     INFO - [TRAIN] epoch=88/604, iter=10200/70000, loss=0.997000, lr=0.000125 | ETA 13:37:25
2022-07-28 21:33:32,244 -     INFO - [TRAIN] epoch=89/604, iter=10250/70000, loss=0.978377, lr=0.000125 | ETA 15:00:08
2022-07-28 21:34:13,896 -     INFO - [TRAIN] epoch=89/604, iter=10300/70000, loss=1.052365, lr=0.000125 | ETA 13:31:16
2022-07-28 21:34:57,060 -     INFO - [TRAIN] epoch=90/604, iter=10350/70000, loss=1.103797, lr=0.000125 | ETA 12:53:00
2022-07-28 21:35:40,739 -     INFO - [TRAIN] epoch=90/604, iter=10400/70000, loss=0.946282, lr=0.000125 | ETA 14:25:33
2022-07-28 21:36:24,234 -     INFO - [TRAIN] epoch=91/604, iter=10450/70000, loss=1.017220, lr=0.000125 | ETA 13:06:10
2022-07-28 21:37:05,876 -     INFO - [TRAIN] epoch=91/604, iter=10500/70000, loss=1.070397, lr=0.000125 | ETA 14:16:49
2022-07-28 21:37:47,664 -     INFO - [TRAIN] epoch=91/604, iter=10550/70000, loss=1.019626, lr=0.000125 | ETA 12:57:21
2022-07-28 21:38:30,296 -     INFO - [TRAIN] epoch=92/604, iter=10600/70000, loss=1.091909, lr=0.000125 | ETA 12:41:52
2022-07-28 21:39:12,353 -     INFO - [TRAIN] epoch=92/604, iter=10650/70000, loss=0.992481, lr=0.000125 | ETA 14:05:06
2022-07-28 21:39:54,981 -     INFO - [TRAIN] epoch=93/604, iter=10700/70000, loss=1.103975, lr=0.000125 | ETA 13:47:57
2022-07-28 21:40:36,829 -     INFO - [TRAIN] epoch=93/604, iter=10750/70000, loss=0.989546, lr=0.000125 | ETA 13:34:03
2022-07-28 21:41:19,591 -     INFO - [TRAIN] epoch=94/604, iter=10800/70000, loss=1.094486, lr=0.000125 | ETA 13:24:10
2022-07-28 21:42:01,514 -     INFO - [TRAIN] epoch=94/604, iter=10850/70000, loss=1.058362, lr=0.000125 | ETA 13:15:41
2022-07-28 21:42:42,463 -     INFO - [TRAIN] epoch=94/604, iter=10900/70000, loss=0.951325, lr=0.000125 | ETA 12:48:33
2022-07-28 21:43:26,820 -     INFO - [TRAIN] epoch=95/604, iter=10950/70000, loss=0.889021, lr=0.000125 | ETA 13:17:17
2022-07-28 21:44:08,516 -     INFO - [TRAIN] epoch=95/604, iter=11000/70000, loss=1.075166, lr=0.000125 | ETA 13:12:32
2022-07-28 21:44:51,219 -     INFO - [TRAIN] epoch=96/604, iter=11050/70000, loss=0.920361, lr=0.000125 | ETA 13:08:14
2022-07-28 21:45:32,821 -     INFO - [TRAIN] epoch=96/604, iter=11100/70000, loss=1.017136, lr=0.000125 | ETA 12:46:30
2022-07-28 21:46:16,260 -     INFO - [TRAIN] epoch=97/604, iter=11150/70000, loss=0.926371, lr=0.000125 | ETA 14:35:39
2022-07-28 21:46:58,237 -     INFO - [TRAIN] epoch=97/604, iter=11200/70000, loss=0.940697, lr=0.000125 | ETA 12:42:24
2022-07-28 21:47:39,275 -     INFO - [TRAIN] epoch=97/604, iter=11250/70000, loss=0.882264, lr=0.000125 | ETA 12:44:58
2022-07-28 21:48:23,541 -     INFO - [TRAIN] epoch=98/604, iter=11300/70000, loss=1.042544, lr=0.000125 | ETA 13:27:14
2022-07-28 21:49:04,220 -     INFO - [TRAIN] epoch=98/604, iter=11350/70000, loss=0.932097, lr=0.000125 | ETA 13:51:44
2022-07-28 21:49:47,158 -     INFO - [TRAIN] epoch=99/604, iter=11400/70000, loss=0.922337, lr=0.000125 | ETA 13:22:43
2022-07-28 21:50:26,589 -     INFO - [TRAIN] epoch=99/604, iter=11450/70000, loss=0.930347, lr=0.000125 | ETA 13:01:56
2022-07-28 21:51:07,983 -     INFO - [TRAIN] epoch=100/604, iter=11500/70000, loss=1.010200, lr=0.000125 | ETA 13:28:07
2022-07-28 21:51:47,248 -     INFO - [TRAIN] epoch=100/604, iter=11550/70000, loss=0.986939, lr=0.000125 | ETA 12:24:53
2022-07-28 21:52:26,607 -     INFO - [TRAIN] epoch=100/604, iter=11600/70000, loss=0.915703, lr=0.000125 | ETA 12:53:36
2022-07-28 21:53:08,545 -     INFO - [TRAIN] epoch=101/604, iter=11650/70000, loss=1.053495, lr=0.000125 | ETA 13:09:19
2022-07-28 21:53:48,992 -     INFO - [TRAIN] epoch=101/604, iter=11700/70000, loss=1.069952, lr=0.000125 | ETA 12:38:35
2022-07-28 21:54:32,741 -     INFO - [TRAIN] epoch=102/604, iter=11750/70000, loss=0.869651, lr=0.000125 | ETA 13:01:28
2022-07-28 21:55:12,775 -     INFO - [TRAIN] epoch=102/604, iter=11800/70000, loss=1.030961, lr=0.000125 | ETA 12:41:45
2022-07-28 21:55:55,994 -     INFO - [TRAIN] epoch=103/604, iter=11850/70000, loss=0.962061, lr=0.000125 | ETA 14:33:45
2022-07-28 21:56:36,409 -     INFO - [TRAIN] epoch=103/604, iter=11900/70000, loss=1.002656, lr=0.000125 | ETA 12:29:49
2022-07-28 21:57:18,738 -     INFO - [TRAIN] epoch=104/604, iter=11950/70000, loss=0.986862, lr=0.000125 | ETA 19:57:03
2022-07-28 21:58:00,407 -     INFO - [TRAIN] epoch=104/604, iter=12000/70000, loss=0.987408, lr=0.000125 | ETA 12:43:50
2022-07-28 21:58:42,478 -     INFO - [TRAIN] epoch=104/604, iter=12050/70000, loss=0.922086, lr=0.000125 | ETA 12:57:56
2022-07-28 21:59:25,690 -     INFO - [TRAIN] epoch=105/604, iter=12100/70000, loss=0.907541, lr=0.000125 | ETA 12:59:19
2022-07-28 22:00:07,312 -     INFO - [TRAIN] epoch=105/604, iter=12150/70000, loss=0.943445, lr=0.000125 | ETA 12:39:49
2022-07-28 22:00:50,558 -     INFO - [TRAIN] epoch=106/604, iter=12200/70000, loss=0.866529, lr=0.000125 | ETA 13:51:48
2022-07-28 22:01:31,397 -     INFO - [TRAIN] epoch=106/604, iter=12250/70000, loss=0.848949, lr=0.000125 | ETA 12:38:43
2022-07-28 22:02:13,958 -     INFO - [TRAIN] epoch=107/604, iter=12300/70000, loss=0.855352, lr=0.000125 | ETA 14:53:23
2022-07-28 22:02:54,920 -     INFO - [TRAIN] epoch=107/604, iter=12350/70000, loss=0.976823, lr=0.000125 | ETA 12:32:00
2022-07-28 22:03:36,171 -     INFO - [TRAIN] epoch=107/604, iter=12400/70000, loss=0.827475, lr=0.000125 | ETA 14:33:35
2022-07-28 22:04:18,592 -     INFO - [TRAIN] epoch=108/604, iter=12450/70000, loss=0.919387, lr=0.000125 | ETA 14:24:47
2022-07-28 22:05:00,411 -     INFO - [TRAIN] epoch=108/604, iter=12500/70000, loss=0.889168, lr=0.000125 | ETA 14:27:46
2022-07-28 22:05:44,626 -     INFO - [TRAIN] epoch=109/604, iter=12550/70000, loss=0.945292, lr=0.000125 | ETA 13:46:41
2022-07-28 22:06:25,977 -     INFO - [TRAIN] epoch=109/604, iter=12600/70000, loss=0.936879, lr=0.000125 | ETA 12:35:56
2022-07-28 22:07:09,530 -     INFO - [TRAIN] epoch=110/604, iter=12650/70000, loss=0.903963, lr=0.000125 | ETA 15:19:31
2022-07-28 22:07:50,495 -     INFO - [TRAIN] epoch=110/604, iter=12700/70000, loss=0.936668, lr=0.000125 | ETA 12:37:04
2022-07-28 22:08:32,991 -     INFO - [TRAIN] epoch=110/604, iter=12750/70000, loss=0.874056, lr=0.000125 | ETA 12:54:34
2022-07-28 22:09:16,445 -     INFO - [TRAIN] epoch=111/604, iter=12800/70000, loss=0.946842, lr=0.000125 | ETA 12:30:38
2022-07-28 22:09:58,630 -     INFO - [TRAIN] epoch=111/604, iter=12850/70000, loss=0.813634, lr=0.000125 | ETA 12:22:29
2022-07-28 22:10:42,525 -     INFO - [TRAIN] epoch=112/604, iter=12900/70000, loss=0.858458, lr=0.000125 | ETA 12:41:45
2022-07-28 22:11:24,524 -     INFO - [TRAIN] epoch=112/604, iter=12950/70000, loss=0.892165, lr=0.000125 | ETA 15:41:42
2022-07-28 22:12:08,559 -     INFO - [TRAIN] epoch=113/604, iter=13000/70000, loss=0.858790, lr=0.000125 | ETA 14:20:44
2022-07-28 22:12:51,390 -     INFO - [TRAIN] epoch=113/604, iter=13050/70000, loss=0.908257, lr=0.000125 | ETA 12:39:16
2022-07-28 22:13:33,753 -     INFO - [TRAIN] epoch=113/604, iter=13100/70000, loss=0.818603, lr=0.000125 | ETA 14:29:40
2022-07-28 22:14:18,377 -     INFO - [TRAIN] epoch=114/604, iter=13150/70000, loss=0.899398, lr=0.000125 | ETA 14:45:27
2022-07-28 22:14:59,582 -     INFO - [TRAIN] epoch=114/604, iter=13200/70000, loss=0.950903, lr=0.000125 | ETA 14:08:05
2022-07-28 22:15:43,930 -     INFO - [TRAIN] epoch=115/604, iter=13250/70000, loss=0.922746, lr=0.000125 | ETA 13:44:17
2022-07-28 22:16:26,260 -     INFO - [TRAIN] epoch=115/604, iter=13300/70000, loss=0.810549, lr=0.000125 | ETA 12:46:03
2022-07-28 22:17:10,101 -     INFO - [TRAIN] epoch=116/604, iter=13350/70000, loss=0.855373, lr=0.000125 | ETA 12:13:44
2022-07-28 22:17:52,084 -     INFO - [TRAIN] epoch=116/604, iter=13400/70000, loss=0.828566, lr=0.000125 | ETA 12:21:37
2022-07-28 22:18:34,145 -     INFO - [TRAIN] epoch=116/604, iter=13450/70000, loss=0.849198, lr=0.000125 | ETA 13:16:55
2022-07-28 22:19:17,461 -     INFO - [TRAIN] epoch=117/604, iter=13500/70000, loss=0.835965, lr=0.000125 | ETA 14:26:50
2022-07-28 22:20:00,238 -     INFO - [TRAIN] epoch=117/604, iter=13550/70000, loss=0.786154, lr=0.000125 | ETA 12:38:55
2022-07-28 22:20:44,898 -     INFO - [TRAIN] epoch=118/604, iter=13600/70000, loss=0.856887, lr=0.000125 | ETA 16:25:09
2022-07-28 22:21:28,130 -     INFO - [TRAIN] epoch=118/604, iter=13650/70000, loss=0.876504, lr=0.000125 | ETA 12:55:29
2022-07-28 22:22:12,133 -     INFO - [TRAIN] epoch=119/604, iter=13700/70000, loss=0.826466, lr=0.000125 | ETA 12:59:31
2022-07-28 22:22:53,987 -     INFO - [TRAIN] epoch=119/604, iter=13750/70000, loss=0.841194, lr=0.000125 | ETA 13:07:58
2022-07-28 22:23:35,444 -     INFO - [TRAIN] epoch=119/604, iter=13800/70000, loss=0.802683, lr=0.000125 | ETA 12:15:31
2022-07-28 22:24:19,845 -     INFO - [TRAIN] epoch=120/604, iter=13850/70000, loss=0.851753, lr=0.000125 | ETA 14:03:20
2022-07-28 22:25:01,580 -     INFO - [TRAIN] epoch=120/604, iter=13900/70000, loss=0.868722, lr=0.000125 | ETA 12:41:51
2022-07-28 22:25:45,757 -     INFO - [TRAIN] epoch=121/604, iter=13950/70000, loss=0.826137, lr=0.000125 | ETA 13:53:02
2022-07-28 22:26:28,078 -     INFO - [TRAIN] epoch=121/604, iter=14000/70000, loss=0.872075, lr=0.000125 | ETA 12:17:50
2022-07-28 22:27:12,118 -     INFO - [TRAIN] epoch=122/604, iter=14050/70000, loss=0.880568, lr=0.000125 | ETA 14:27:57
2022-07-28 22:27:54,347 -     INFO - [TRAIN] epoch=122/604, iter=14100/70000, loss=0.885037, lr=0.000125 | ETA 13:15:44
2022-07-28 22:28:35,498 -     INFO - [TRAIN] epoch=122/604, iter=14150/70000, loss=0.782125, lr=0.000125 | ETA 12:06:41
2022-07-28 22:29:20,501 -     INFO - [TRAIN] epoch=123/604, iter=14200/70000, loss=0.815750, lr=0.000125 | ETA 13:53:07
2022-07-28 22:30:02,459 -     INFO - [TRAIN] epoch=123/604, iter=14250/70000, loss=0.908904, lr=0.000125 | ETA 13:16:28
2022-07-28 22:30:46,519 -     INFO - [TRAIN] epoch=124/604, iter=14300/70000, loss=0.828011, lr=0.000125 | ETA 12:54:28
2022-07-28 22:31:28,428 -     INFO - [TRAIN] epoch=124/604, iter=14350/70000, loss=0.762063, lr=0.000125 | ETA 12:29:42
2022-07-28 22:32:12,075 -     INFO - [TRAIN] epoch=125/604, iter=14400/70000, loss=0.839983, lr=0.000125 | ETA 13:35:04
2022-07-28 22:32:54,544 -     INFO - [TRAIN] epoch=125/604, iter=14450/70000, loss=0.844293, lr=0.000125 | ETA 12:21:52
2022-07-28 22:33:36,827 -     INFO - [TRAIN] epoch=125/604, iter=14500/70000, loss=0.791352, lr=0.000125 | ETA 12:09:23
2022-07-28 22:34:20,894 -     INFO - [TRAIN] epoch=126/604, iter=14550/70000, loss=0.819335, lr=0.000125 | ETA 12:31:59
2022-07-28 22:35:03,893 -     INFO - [TRAIN] epoch=126/604, iter=14600/70000, loss=0.806079, lr=0.000125 | ETA 12:04:27
2022-07-28 22:35:48,651 -     INFO - [TRAIN] epoch=127/604, iter=14650/70000, loss=0.777888, lr=0.000125 | ETA 12:46:07
2022-07-28 22:36:30,933 -     INFO - [TRAIN] epoch=127/604, iter=14700/70000, loss=0.806434, lr=0.000125 | ETA 12:15:33
2022-07-28 22:37:14,009 -     INFO - [TRAIN] epoch=128/604, iter=14750/70000, loss=0.794339, lr=0.000125 | ETA 12:12:38
2022-07-28 22:37:55,938 -     INFO - [TRAIN] epoch=128/604, iter=14800/70000, loss=0.907006, lr=0.000125 | ETA 12:48:56
2022-07-28 22:38:39,142 -     INFO - [TRAIN] epoch=129/604, iter=14850/70000, loss=0.867123, lr=0.000125 | ETA 18:32:49
2022-07-28 22:39:21,897 -     INFO - [TRAIN] epoch=129/604, iter=14900/70000, loss=0.758714, lr=0.000125 | ETA 13:58:49
2022-07-28 22:40:03,832 -     INFO - [TRAIN] epoch=129/604, iter=14950/70000, loss=0.789267, lr=0.000125 | ETA 12:10:33
2022-07-28 22:40:46,795 -     INFO - [TRAIN] epoch=130/604, iter=15000/70000, loss=0.819636, lr=0.000125 | ETA 12:37:46
2022-07-28 22:40:47,767 -     INFO - Push model to checkpoint ./output/iter_15000
2022-07-28 22:41:29,393 -     INFO - [TRAIN] epoch=130/604, iter=15050/70000, loss=0.771979, lr=0.000125 | ETA 13:41:02
2022-07-28 22:42:13,286 -     INFO - [TRAIN] epoch=131/604, iter=15100/70000, loss=0.882957, lr=0.000125 | ETA 12:17:45
2022-07-28 22:42:55,998 -     INFO - [TRAIN] epoch=131/604, iter=15150/70000, loss=0.727803, lr=0.000125 | ETA 12:55:10
2022-07-28 22:43:40,175 -     INFO - [TRAIN] epoch=132/604, iter=15200/70000, loss=0.799391, lr=0.000125 | ETA 16:20:03
2022-07-28 22:44:23,083 -     INFO - [TRAIN] epoch=132/604, iter=15250/70000, loss=0.833547, lr=0.000125 | ETA 13:41:33
2022-07-28 22:45:05,743 -     INFO - [TRAIN] epoch=132/604, iter=15300/70000, loss=0.803422, lr=0.000125 | ETA 15:59:33
2022-07-28 22:45:50,449 -     INFO - [TRAIN] epoch=133/604, iter=15350/70000, loss=0.828263, lr=0.000125 | ETA 12:22:35
2022-07-28 22:46:32,436 -     INFO - [TRAIN] epoch=133/604, iter=15400/70000, loss=0.819201, lr=0.000125 | ETA 12:32:27
2022-07-28 22:47:15,349 -     INFO - [TRAIN] epoch=134/604, iter=15450/70000, loss=0.775616, lr=0.000125 | ETA 12:10:27
2022-07-28 22:47:57,741 -     INFO - [TRAIN] epoch=134/604, iter=15500/70000, loss=0.829361, lr=0.000125 | ETA 13:58:15
2022-07-28 22:48:40,963 -     INFO - [TRAIN] epoch=135/604, iter=15550/70000, loss=0.816770, lr=0.000125 | ETA 13:37:37
2022-07-28 22:49:23,372 -     INFO - [TRAIN] epoch=135/604, iter=15600/70000, loss=0.761665, lr=0.000125 | ETA 14:25:50
2022-07-28 22:50:05,818 -     INFO - [TRAIN] epoch=135/604, iter=15650/70000, loss=0.793909, lr=0.000125 | ETA 13:51:39
2022-07-28 22:50:49,393 -     INFO - [TRAIN] epoch=136/604, iter=15700/70000, loss=0.785873, lr=0.000125 | ETA 12:23:11
2022-07-28 22:51:31,370 -     INFO - [TRAIN] epoch=136/604, iter=15750/70000, loss=0.734012, lr=0.000125 | ETA 12:15:06
2022-07-28 22:52:15,356 -     INFO - [TRAIN] epoch=137/604, iter=15800/70000, loss=0.713663, lr=0.000125 | ETA 11:48:26
2022-07-28 22:52:58,377 -     INFO - [TRAIN] epoch=137/604, iter=15850/70000, loss=0.882065, lr=0.000125 | ETA 12:19:29
2022-07-28 22:53:42,822 -     INFO - [TRAIN] epoch=138/604, iter=15900/70000, loss=0.740203, lr=0.000125 | ETA 12:59:39
2022-07-28 22:54:25,422 -     INFO - [TRAIN] epoch=138/604, iter=15950/70000, loss=0.797708, lr=0.000125 | ETA 12:54:17
2022-07-28 22:55:07,133 -     INFO - [TRAIN] epoch=138/604, iter=16000/70000, loss=0.835018, lr=0.000125 | ETA 12:23:02
2022-07-28 22:55:51,863 -     INFO - [TRAIN] epoch=139/604, iter=16050/70000, loss=0.788093, lr=0.000125 | ETA 12:31:10
2022-07-28 22:56:34,331 -     INFO - [TRAIN] epoch=139/604, iter=16100/70000, loss=0.748445, lr=0.000125 | ETA 12:13:34
2022-07-28 22:57:18,544 -     INFO - [TRAIN] epoch=140/604, iter=16150/70000, loss=0.747661, lr=0.000125 | ETA 12:10:42
2022-07-28 22:58:01,663 -     INFO - [TRAIN] epoch=140/604, iter=16200/70000, loss=0.697298, lr=0.000125 | ETA 14:40:40
2022-07-28 22:58:45,437 -     INFO - [TRAIN] epoch=141/604, iter=16250/70000, loss=0.777004, lr=0.000125 | ETA 12:08:52
2022-07-28 22:59:29,117 -     INFO - [TRAIN] epoch=141/604, iter=16300/70000, loss=0.777773, lr=0.000125 | ETA 13:07:19
2022-07-28 23:00:12,329 -     INFO - [TRAIN] epoch=141/604, iter=16350/70000, loss=0.731121, lr=0.000125 | ETA 14:31:41
2022-07-28 23:00:55,766 -     INFO - [TRAIN] epoch=142/604, iter=16400/70000, loss=0.736898, lr=0.000125 | ETA 13:01:15
2022-07-28 23:01:37,910 -     INFO - [TRAIN] epoch=142/604, iter=16450/70000, loss=0.803595, lr=0.000125 | ETA 13:50:10
2022-07-28 23:02:22,248 -     INFO - [TRAIN] epoch=143/604, iter=16500/70000, loss=0.773481, lr=0.000125 | ETA 12:22:42
2022-07-28 23:03:04,337 -     INFO - [TRAIN] epoch=143/604, iter=16550/70000, loss=0.809999, lr=0.000125 | ETA 13:04:49
2022-07-28 23:03:48,300 -     INFO - [TRAIN] epoch=144/604, iter=16600/70000, loss=0.768910, lr=0.000125 | ETA 12:28:42
2022-07-28 23:04:31,855 -     INFO - [TRAIN] epoch=144/604, iter=16650/70000, loss=0.741337, lr=0.000125 | ETA 11:54:36
2022-07-28 23:05:14,252 -     INFO - [TRAIN] epoch=144/604, iter=16700/70000, loss=0.734342, lr=0.000125 | ETA 11:35:12
2022-07-28 23:05:58,726 -     INFO - [TRAIN] epoch=145/604, iter=16750/70000, loss=0.741294, lr=0.000125 | ETA 11:37:04
2022-07-28 23:06:39,777 -     INFO - [TRAIN] epoch=145/604, iter=16800/70000, loss=0.786714, lr=0.000125 | ETA 11:32:27
2022-07-28 23:07:23,104 -     INFO - [TRAIN] epoch=146/604, iter=16850/70000, loss=0.785133, lr=0.000125 | ETA 11:56:25
2022-07-28 23:08:05,564 -     INFO - [TRAIN] epoch=146/604, iter=16900/70000, loss=0.780546, lr=0.000125 | ETA 12:03:16
2022-07-28 23:08:49,262 -     INFO - [TRAIN] epoch=147/604, iter=16950/70000, loss=0.813480, lr=0.000125 | ETA 13:01:55
2022-07-28 23:09:31,889 -     INFO - [TRAIN] epoch=147/604, iter=17000/70000, loss=0.815679, lr=0.000125 | ETA 11:50:36
2022-07-28 23:10:13,855 -     INFO - [TRAIN] epoch=147/604, iter=17050/70000, loss=0.757869, lr=0.000125 | ETA 12:05:27
2022-07-28 23:10:59,148 -     INFO - [TRAIN] epoch=148/604, iter=17100/70000, loss=0.725435, lr=0.000125 | ETA 12:30:29
2022-07-28 23:11:41,007 -     INFO - [TRAIN] epoch=148/604, iter=17150/70000, loss=0.736074, lr=0.000125 | ETA 11:29:59
2022-07-28 23:12:24,860 -     INFO - [TRAIN] epoch=149/604, iter=17200/70000, loss=0.793974, lr=0.000125 | ETA 11:48:59
2022-07-28 23:13:07,467 -     INFO - [TRAIN] epoch=149/604, iter=17250/70000, loss=0.736926, lr=0.000125 | ETA 11:34:43
2022-07-28 23:13:50,546 -     INFO - [TRAIN] epoch=150/604, iter=17300/70000, loss=0.728060, lr=0.000125 | ETA 11:23:29
2022-07-28 23:14:32,903 -     INFO - [TRAIN] epoch=150/604, iter=17350/70000, loss=0.725826, lr=0.000125 | ETA 11:55:12
2022-07-28 23:15:14,288 -     INFO - [TRAIN] epoch=150/604, iter=17400/70000, loss=0.683678, lr=0.000125 | ETA 11:29:04
2022-07-28 23:15:58,796 -     INFO - [TRAIN] epoch=151/604, iter=17450/70000, loss=0.742099, lr=0.000125 | ETA 13:39:29
2022-07-28 23:16:41,669 -     INFO - [TRAIN] epoch=151/604, iter=17500/70000, loss=0.698566, lr=0.000125 | ETA 12:46:44
2022-07-28 23:17:25,898 -     INFO - [TRAIN] epoch=152/604, iter=17550/70000, loss=0.735730, lr=0.000125 | ETA 12:02:18
2022-07-28 23:18:08,352 -     INFO - [TRAIN] epoch=152/604, iter=17600/70000, loss=0.777086, lr=0.000125 | ETA 12:25:28
2022-07-28 23:18:52,749 -     INFO - [TRAIN] epoch=153/604, iter=17650/70000, loss=0.730934, lr=0.000125 | ETA 12:34:24
2022-07-28 23:19:34,795 -     INFO - [TRAIN] epoch=153/604, iter=17700/70000, loss=0.716544, lr=0.000125 | ETA 11:14:39
2022-07-28 23:20:17,899 -     INFO - [TRAIN] epoch=154/604, iter=17750/70000, loss=0.682877, lr=0.000125 | ETA 18:31:50
2022-07-28 23:21:00,611 -     INFO - [TRAIN] epoch=154/604, iter=17800/70000, loss=0.763804, lr=0.000125 | ETA 13:06:36
2022-07-28 23:21:43,015 -     INFO - [TRAIN] epoch=154/604, iter=17850/70000, loss=0.739786, lr=0.000125 | ETA 13:30:02
2022-07-28 23:22:26,962 -     INFO - [TRAIN] epoch=155/604, iter=17900/70000, loss=0.790095, lr=0.000125 | ETA 14:13:43
2022-07-28 23:23:08,997 -     INFO - [TRAIN] epoch=155/604, iter=17950/70000, loss=0.812582, lr=0.000125 | ETA 12:36:56
2022-07-28 23:23:52,380 -     INFO - [TRAIN] epoch=156/604, iter=18000/70000, loss=0.771684, lr=0.000125 | ETA 12:06:32
2022-07-28 23:24:34,935 -     INFO - [TRAIN] epoch=156/604, iter=18050/70000, loss=0.710415, lr=0.000125 | ETA 11:59:27
2022-07-28 23:25:18,131 -     INFO - [TRAIN] epoch=157/604, iter=18100/70000, loss=0.770689, lr=0.000125 | ETA 13:08:32
2022-07-28 23:26:02,053 -     INFO - [TRAIN] epoch=157/604, iter=18150/70000, loss=0.751137, lr=0.000125 | ETA 11:47:36
2022-07-28 23:26:44,311 -     INFO - [TRAIN] epoch=157/604, iter=18200/70000, loss=0.743075, lr=0.000125 | ETA 11:13:51
2022-07-28 23:27:29,153 -     INFO - [TRAIN] epoch=158/604, iter=18250/70000, loss=0.795314, lr=0.000125 | ETA 12:09:51
2022-07-28 23:28:11,091 -     INFO - [TRAIN] epoch=158/604, iter=18300/70000, loss=0.751376, lr=0.000125 | ETA 12:26:26
2022-07-28 23:28:55,111 -     INFO - [TRAIN] epoch=159/604, iter=18350/70000, loss=0.747599, lr=0.000125 | ETA 11:12:01
2022-07-28 23:29:37,710 -     INFO - [TRAIN] epoch=159/604, iter=18400/70000, loss=0.795225, lr=0.000125 | ETA 14:05:13
2022-07-28 23:30:21,819 -     INFO - [TRAIN] epoch=160/604, iter=18450/70000, loss=0.715047, lr=0.000125 | ETA 12:11:53
2022-07-28 23:31:05,391 -     INFO - [TRAIN] epoch=160/604, iter=18500/70000, loss=0.748510, lr=0.000125 | ETA 14:00:26
2022-07-28 23:31:46,940 -     INFO - [TRAIN] epoch=160/604, iter=18550/70000, loss=0.752881, lr=0.000125 | ETA 11:31:13
2022-07-28 23:32:30,951 -     INFO - [TRAIN] epoch=161/604, iter=18600/70000, loss=0.690476, lr=0.000125 | ETA 11:14:11
2022-07-28 23:33:13,971 -     INFO - [TRAIN] epoch=161/604, iter=18650/70000, loss=0.658091, lr=0.000125 | ETA 11:35:25
2022-07-28 23:33:57,709 -     INFO - [TRAIN] epoch=162/604, iter=18700/70000, loss=0.674849, lr=0.000125 | ETA 11:14:08
2022-07-28 23:34:40,850 -     INFO - [TRAIN] epoch=162/604, iter=18750/70000, loss=0.740145, lr=0.000125 | ETA 11:42:28
2022-07-28 23:35:25,726 -     INFO - [TRAIN] epoch=163/604, iter=18800/70000, loss=0.728593, lr=0.000125 | ETA 11:35:29
2022-07-28 23:36:07,380 -     INFO - [TRAIN] epoch=163/604, iter=18850/70000, loss=0.653929, lr=0.000125 | ETA 12:14:43
2022-07-28 23:36:50,159 -     INFO - [TRAIN] epoch=163/604, iter=18900/70000, loss=0.705434, lr=0.000125 | ETA 11:23:16
2022-07-28 23:37:33,773 -     INFO - [TRAIN] epoch=164/604, iter=18950/70000, loss=0.706653, lr=0.000125 | ETA 14:53:42
2022-07-28 23:38:16,655 -     INFO - [TRAIN] epoch=164/604, iter=19000/70000, loss=0.697365, lr=0.000125 | ETA 12:35:47
2022-07-28 23:38:58,858 -     INFO - [TRAIN] epoch=165/604, iter=19050/70000, loss=0.654431, lr=0.000125 | ETA 11:13:34
2022-07-28 23:39:38,533 -     INFO - [TRAIN] epoch=165/604, iter=19100/70000, loss=0.714211, lr=0.000125 | ETA 11:45:05
2022-07-28 23:40:20,300 -     INFO - [TRAIN] epoch=166/604, iter=19150/70000, loss=0.716450, lr=0.000125 | ETA 11:28:53
2022-07-28 23:41:00,123 -     INFO - [TRAIN] epoch=166/604, iter=19200/70000, loss=0.663732, lr=0.000125 | ETA 11:06:31
2022-07-28 23:41:39,328 -     INFO - [TRAIN] epoch=166/604, iter=19250/70000, loss=0.676418, lr=0.000125 | ETA 11:21:20
2022-07-28 23:42:20,592 -     INFO - [TRAIN] epoch=167/604, iter=19300/70000, loss=0.717973, lr=0.000125 | ETA 11:16:03
2022-07-28 23:43:00,712 -     INFO - [TRAIN] epoch=167/604, iter=19350/70000, loss=0.682235, lr=0.000125 | ETA 11:33:26
2022-07-28 23:43:42,623 -     INFO - [TRAIN] epoch=168/604, iter=19400/70000, loss=0.661294, lr=0.000125 | ETA 11:02:30
2022-07-28 23:44:22,588 -     INFO - [TRAIN] epoch=168/604, iter=19450/70000, loss=0.715578, lr=0.000125 | ETA 11:11:16
2022-07-28 23:45:04,581 -     INFO - [TRAIN] epoch=169/604, iter=19500/70000, loss=0.752551, lr=0.000125 | ETA 10:59:35
2022-07-28 23:45:45,363 -     INFO - [TRAIN] epoch=169/604, iter=19550/70000, loss=0.610312, lr=0.000125 | ETA 11:41:33
2022-07-28 23:46:26,396 -     INFO - [TRAIN] epoch=169/604, iter=19600/70000, loss=0.668978, lr=0.000125 | ETA 11:17:18
2022-07-28 23:47:09,264 -     INFO - [TRAIN] epoch=170/604, iter=19650/70000, loss=0.741929, lr=0.000125 | ETA 11:06:47
2022-07-28 23:47:51,105 -     INFO - [TRAIN] epoch=170/604, iter=19700/70000, loss=0.722165, lr=0.000125 | ETA 11:17:18
2022-07-28 23:48:33,554 -     INFO - [TRAIN] epoch=171/604, iter=19750/70000, loss=0.681652, lr=0.000125 | ETA 11:03:37
2022-07-28 23:49:15,601 -     INFO - [TRAIN] epoch=171/604, iter=19800/70000, loss=0.698019, lr=0.000125 | ETA 11:30:37
2022-07-28 23:49:59,461 -     INFO - [TRAIN] epoch=172/604, iter=19850/70000, loss=0.722691, lr=0.000125 | ETA 10:57:57
2022-07-28 23:50:40,257 -     INFO - [TRAIN] epoch=172/604, iter=19900/70000, loss=0.693653, lr=0.000125 | ETA 11:29:08
2022-07-28 23:51:21,230 -     INFO - [TRAIN] epoch=172/604, iter=19950/70000, loss=0.662506, lr=0.000125 | ETA 10:55:41
2022-07-28 23:52:06,390 -     INFO - [TRAIN] epoch=173/604, iter=20000/70000, loss=0.689593, lr=0.000125 | ETA 14:16:36
2022-07-28 23:52:07,340 -     INFO - Push model to checkpoint ./output/iter_20000
2022-07-28 23:52:50,262 -     INFO - [TRAIN] epoch=173/604, iter=20050/70000, loss=0.745679, lr=0.000125 | ETA 12:59:09
2022-07-28 23:53:33,746 -     INFO - [TRAIN] epoch=174/604, iter=20100/70000, loss=0.733232, lr=0.000125 | ETA 11:02:45
2022-07-28 23:54:16,456 -     INFO - [TRAIN] epoch=174/604, iter=20150/70000, loss=0.711176, lr=0.000125 | ETA 12:01:09
2022-07-28 23:54:59,920 -     INFO - [TRAIN] epoch=175/604, iter=20200/70000, loss=0.724105, lr=0.000125 | ETA 11:54:38
2022-07-28 23:55:41,467 -     INFO - [TRAIN] epoch=175/604, iter=20250/70000, loss=0.728266, lr=0.000125 | ETA 11:17:55
2022-07-28 23:56:22,659 -     INFO - [TRAIN] epoch=175/604, iter=20300/70000, loss=0.667829, lr=0.000125 | ETA 10:44:51
2022-07-28 23:57:05,868 -     INFO - [TRAIN] epoch=176/604, iter=20350/70000, loss=0.611505, lr=0.000125 | ETA 11:07:16
2022-07-28 23:57:47,117 -     INFO - [TRAIN] epoch=176/604, iter=20400/70000, loss=0.703511, lr=0.000125 | ETA 10:43:23
2022-07-28 23:58:31,120 -     INFO - [TRAIN] epoch=177/604, iter=20450/70000, loss=0.688911, lr=0.000125 | ETA 10:45:30
2022-07-28 23:59:12,036 -     INFO - [TRAIN] epoch=177/604, iter=20500/70000, loss=0.705324, lr=0.000125 | ETA 10:54:46
2022-07-28 23:59:56,156 -     INFO - [TRAIN] epoch=178/604, iter=20550/70000, loss=0.669938, lr=0.000125 | ETA 11:39:17
2022-07-29 00:00:39,151 -     INFO - [TRAIN] epoch=178/604, iter=20600/70000, loss=0.658893, lr=0.000125 | ETA 10:51:33
2022-07-29 00:01:22,484 -     INFO - [TRAIN] epoch=179/604, iter=20650/70000, loss=0.693269, lr=0.000125 | ETA 16:55:07
2022-07-29 00:02:04,721 -     INFO - [TRAIN] epoch=179/604, iter=20700/70000, loss=0.594141, lr=0.000125 | ETA 10:48:09
2022-07-29 00:02:47,278 -     INFO - [TRAIN] epoch=179/604, iter=20750/70000, loss=0.685239, lr=0.000125 | ETA 14:19:51
2022-07-29 00:03:31,032 -     INFO - [TRAIN] epoch=180/604, iter=20800/70000, loss=0.727320, lr=0.000125 | ETA 11:22:21
2022-07-29 00:04:13,458 -     INFO - [TRAIN] epoch=180/604, iter=20850/70000, loss=0.652608, lr=0.000125 | ETA 10:50:11
2022-07-29 00:04:57,401 -     INFO - [TRAIN] epoch=181/604, iter=20900/70000, loss=0.617559, lr=0.000125 | ETA 11:58:03
2022-07-29 00:05:38,840 -     INFO - [TRAIN] epoch=181/604, iter=20950/70000, loss=0.680914, lr=0.000125 | ETA 11:14:08
2022-07-29 00:06:22,856 -     INFO - [TRAIN] epoch=182/604, iter=21000/70000, loss=0.690077, lr=0.000125 | ETA 12:38:44
2022-07-29 00:07:05,593 -     INFO - [TRAIN] epoch=182/604, iter=21050/70000, loss=0.713903, lr=0.000125 | ETA 14:30:43
2022-07-29 00:07:47,832 -     INFO - [TRAIN] epoch=182/604, iter=21100/70000, loss=0.631335, lr=0.000125 | ETA 12:03:48
2022-07-29 00:08:31,875 -     INFO - [TRAIN] epoch=183/604, iter=21150/70000, loss=0.662938, lr=0.000125 | ETA 10:40:36
2022-07-29 00:09:14,510 -     INFO - [TRAIN] epoch=183/604, iter=21200/70000, loss=0.655766, lr=0.000125 | ETA 12:37:38
2022-07-29 00:09:57,650 -     INFO - [TRAIN] epoch=184/604, iter=21250/70000, loss=0.701453, lr=0.000125 | ETA 10:56:30
2022-07-29 00:10:38,611 -     INFO - [TRAIN] epoch=184/604, iter=21300/70000, loss=0.611224, lr=0.000125 | ETA 10:49:32
2022-07-29 00:11:22,042 -     INFO - [TRAIN] epoch=185/604, iter=21350/70000, loss=0.665345, lr=0.000125 | ETA 12:36:13
2022-07-29 00:12:03,729 -     INFO - [TRAIN] epoch=185/604, iter=21400/70000, loss=0.620825, lr=0.000125 | ETA 10:57:31
2022-07-29 00:12:46,365 -     INFO - [TRAIN] epoch=185/604, iter=21450/70000, loss=0.626469, lr=0.000125 | ETA 10:54:05
2022-07-29 00:13:30,894 -     INFO - [TRAIN] epoch=186/604, iter=21500/70000, loss=0.694504, lr=0.000125 | ETA 11:48:35
2022-07-29 00:14:12,667 -     INFO - [TRAIN] epoch=186/604, iter=21550/70000, loss=0.725830, lr=0.000125 | ETA 11:01:38
2022-07-29 00:14:56,864 -     INFO - [TRAIN] epoch=187/604, iter=21600/70000, loss=0.677090, lr=0.000125 | ETA 13:25:25
2022-07-29 00:15:38,978 -     INFO - [TRAIN] epoch=187/604, iter=21650/70000, loss=0.640112, lr=0.000125 | ETA 11:48:05
2022-07-29 00:16:22,356 -     INFO - [TRAIN] epoch=188/604, iter=21700/70000, loss=0.644252, lr=0.000125 | ETA 11:42:25
2022-07-29 00:17:04,770 -     INFO - [TRAIN] epoch=188/604, iter=21750/70000, loss=0.731813, lr=0.000125 | ETA 12:21:25
2022-07-29 00:17:46,676 -     INFO - [TRAIN] epoch=188/604, iter=21800/70000, loss=0.668840, lr=0.000125 | ETA 10:41:59
2022-07-29 00:18:30,254 -     INFO - [TRAIN] epoch=189/604, iter=21850/70000, loss=0.642196, lr=0.000125 | ETA 10:45:30
2022-07-29 00:19:13,337 -     INFO - [TRAIN] epoch=189/604, iter=21900/70000, loss=0.638259, lr=0.000125 | ETA 10:42:12
2022-07-29 00:19:56,447 -     INFO - [TRAIN] epoch=190/604, iter=21950/70000, loss=0.648491, lr=0.000125 | ETA 10:20:20
2022-07-29 00:20:38,480 -     INFO - [TRAIN] epoch=190/604, iter=22000/70000, loss=0.770738, lr=0.000125 | ETA 11:38:07
2022-07-29 00:21:22,699 -     INFO - [TRAIN] epoch=191/604, iter=22050/70000, loss=0.645569, lr=0.000125 | ETA 12:26:46
2022-07-29 00:22:04,132 -     INFO - [TRAIN] epoch=191/604, iter=22100/70000, loss=0.611290, lr=0.000125 | ETA 11:26:46
2022-07-29 00:22:46,053 -     INFO - [TRAIN] epoch=191/604, iter=22150/70000, loss=0.670410, lr=0.000125 | ETA 10:32:00
2022-07-29 00:23:30,482 -     INFO - [TRAIN] epoch=192/604, iter=22200/70000, loss=0.660043, lr=0.000125 | ETA 11:16:16
2022-07-29 00:24:12,345 -     INFO - [TRAIN] epoch=192/604, iter=22250/70000, loss=0.637556, lr=0.000125 | ETA 11:22:19
2022-07-29 00:24:56,971 -     INFO - [TRAIN] epoch=193/604, iter=22300/70000, loss=0.634850, lr=0.000125 | ETA 10:59:09
2022-07-29 00:25:38,932 -     INFO - [TRAIN] epoch=193/604, iter=22350/70000, loss=0.621620, lr=0.000125 | ETA 10:54:39
2022-07-29 00:26:22,421 -     INFO - [TRAIN] epoch=194/604, iter=22400/70000, loss=0.666928, lr=0.000125 | ETA 10:53:59
2022-07-29 00:27:04,628 -     INFO - [TRAIN] epoch=194/604, iter=22450/70000, loss=0.673338, lr=0.000125 | ETA 10:42:14
2022-07-29 00:27:45,951 -     INFO - [TRAIN] epoch=194/604, iter=22500/70000, loss=0.632360, lr=0.000125 | ETA 10:19:49
2022-07-29 00:28:30,450 -     INFO - [TRAIN] epoch=195/604, iter=22550/70000, loss=0.644220, lr=0.000125 | ETA 12:06:28
2022-07-29 00:29:12,835 -     INFO - [TRAIN] epoch=195/604, iter=22600/70000, loss=0.652470, lr=0.000125 | ETA 10:28:11
2022-07-29 00:29:56,030 -     INFO - [TRAIN] epoch=196/604, iter=22650/70000, loss=0.601254, lr=0.000125 | ETA 10:42:14
2022-07-29 00:30:38,373 -     INFO - [TRAIN] epoch=196/604, iter=22700/70000, loss=0.635631, lr=0.000125 | ETA 10:30:39
2022-07-29 00:31:21,765 -     INFO - [TRAIN] epoch=197/604, iter=22750/70000, loss=0.638566, lr=0.000125 | ETA 10:51:42
2022-07-29 00:32:03,088 -     INFO - [TRAIN] epoch=197/604, iter=22800/70000, loss=0.623931, lr=0.000125 | ETA 10:55:08
2022-07-29 00:32:44,681 -     INFO - [TRAIN] epoch=197/604, iter=22850/70000, loss=0.658636, lr=0.000125 | ETA 10:15:22
2022-07-29 00:33:28,956 -     INFO - [TRAIN] epoch=198/604, iter=22900/70000, loss=0.680698, lr=0.000125 | ETA 11:51:09
2022-07-29 00:34:11,399 -     INFO - [TRAIN] epoch=198/604, iter=22950/70000, loss=0.580865, lr=0.000125 | ETA 10:18:16
2022-07-29 00:34:54,374 -     INFO - [TRAIN] epoch=199/604, iter=23000/70000, loss=0.614564, lr=0.000125 | ETA 10:27:49
2022-07-29 00:35:35,962 -     INFO - [TRAIN] epoch=199/604, iter=23050/70000, loss=0.684047, lr=0.000125 | ETA 10:23:18
2022-07-29 00:36:19,613 -     INFO - [TRAIN] epoch=200/604, iter=23100/70000, loss=0.670641, lr=0.000125 | ETA 10:31:54
2022-07-29 00:37:02,290 -     INFO - [TRAIN] epoch=200/604, iter=23150/70000, loss=0.703074, lr=0.000125 | ETA 11:38:59
2022-07-29 00:37:44,415 -     INFO - [TRAIN] epoch=200/604, iter=23200/70000, loss=0.637614, lr=0.000125 | ETA 10:08:09
2022-07-29 00:38:29,052 -     INFO - [TRAIN] epoch=201/604, iter=23250/70000, loss=0.648152, lr=0.000125 | ETA 10:37:32
2022-07-29 00:39:11,494 -     INFO - [TRAIN] epoch=201/604, iter=23300/70000, loss=0.672435, lr=0.000125 | ETA 13:14:59
2022-07-29 00:39:55,511 -     INFO - [TRAIN] epoch=202/604, iter=23350/70000, loss=0.592584, lr=0.000125 | ETA 10:10:05
2022-07-29 00:40:37,694 -     INFO - [TRAIN] epoch=202/604, iter=23400/70000, loss=0.616770, lr=0.000125 | ETA 11:12:56
2022-07-29 00:41:21,192 -     INFO - [TRAIN] epoch=203/604, iter=23450/70000, loss=0.649940, lr=0.000125 | ETA 11:02:35
2022-07-29 00:42:03,622 -     INFO - [TRAIN] epoch=203/604, iter=23500/70000, loss=0.607236, lr=0.000125 | ETA 10:48:46
2022-07-29 00:42:47,347 -     INFO - [TRAIN] epoch=204/604, iter=23550/70000, loss=0.579338, lr=0.000125 | ETA 15:53:28
2022-07-29 00:43:30,335 -     INFO - [TRAIN] epoch=204/604, iter=23600/70000, loss=0.643363, lr=0.000125 | ETA 10:34:36
2022-07-29 00:44:12,679 -     INFO - [TRAIN] epoch=204/604, iter=23650/70000, loss=0.596615, lr=0.000125 | ETA 10:26:42
2022-07-29 00:44:57,808 -     INFO - [TRAIN] epoch=205/604, iter=23700/70000, loss=0.599391, lr=0.000125 | ETA 11:14:18
2022-07-29 00:45:40,502 -     INFO - [TRAIN] epoch=205/604, iter=23750/70000, loss=0.617891, lr=0.000125 | ETA 11:17:38
2022-07-29 00:46:24,785 -     INFO - [TRAIN] epoch=206/604, iter=23800/70000, loss=0.595589, lr=0.000125 | ETA 11:07:06
2022-07-29 00:47:05,869 -     INFO - [TRAIN] epoch=206/604, iter=23850/70000, loss=0.630348, lr=0.000125 | ETA 10:09:04
2022-07-29 00:47:49,395 -     INFO - [TRAIN] epoch=207/604, iter=23900/70000, loss=0.627368, lr=0.000125 | ETA 12:25:48
2022-07-29 00:48:33,271 -     INFO - [TRAIN] epoch=207/604, iter=23950/70000, loss=0.604198, lr=0.000125 | ETA 10:14:59
2022-07-29 00:49:15,725 -     INFO - [TRAIN] epoch=207/604, iter=24000/70000, loss=0.637700, lr=0.000125 | ETA 11:04:45
2022-07-29 00:49:59,773 -     INFO - [TRAIN] epoch=208/604, iter=24050/70000, loss=0.680990, lr=0.000125 | ETA 10:01:06
2022-07-29 00:50:42,279 -     INFO - [TRAIN] epoch=208/604, iter=24100/70000, loss=0.598581, lr=0.000125 | ETA 11:52:15
2022-07-29 00:51:26,204 -     INFO - [TRAIN] epoch=209/604, iter=24150/70000, loss=0.666969, lr=0.000125 | ETA 12:32:19
2022-07-29 00:52:07,658 -     INFO - [TRAIN] epoch=209/604, iter=24200/70000, loss=0.643755, lr=0.000125 | ETA 10:03:21
2022-07-29 00:52:53,007 -     INFO - [TRAIN] epoch=210/604, iter=24250/70000, loss=0.603430, lr=0.000125 | ETA 12:03:12
2022-07-29 00:53:34,689 -     INFO - [TRAIN] epoch=210/604, iter=24300/70000, loss=0.639863, lr=0.000125 | ETA 10:09:50
2022-07-29 00:54:15,886 -     INFO - [TRAIN] epoch=210/604, iter=24350/70000, loss=0.602531, lr=0.000125 | ETA 10:40:42
2022-07-29 00:55:00,601 -     INFO - [TRAIN] epoch=211/604, iter=24400/70000, loss=0.634872, lr=0.000125 | ETA 09:56:59
2022-07-29 00:55:43,598 -     INFO - [TRAIN] epoch=211/604, iter=24450/70000, loss=0.591186, lr=0.000125 | ETA 11:52:30
2022-07-29 00:56:27,560 -     INFO - [TRAIN] epoch=212/604, iter=24500/70000, loss=0.635616, lr=0.000125 | ETA 10:22:42
2022-07-29 00:57:10,020 -     INFO - [TRAIN] epoch=212/604, iter=24550/70000, loss=0.608485, lr=0.000125 | ETA 12:02:29
2022-07-29 00:57:53,727 -     INFO - [TRAIN] epoch=213/604, iter=24600/70000, loss=0.635512, lr=0.000125 | ETA 11:31:22
2022-07-29 00:58:36,354 -     INFO - [TRAIN] epoch=213/604, iter=24650/70000, loss=0.628794, lr=0.000125 | ETA 10:19:14
2022-07-29 00:59:19,775 -     INFO - [TRAIN] epoch=213/604, iter=24700/70000, loss=0.572820, lr=0.000125 | ETA 11:44:39
2022-07-29 01:00:02,981 -     INFO - [TRAIN] epoch=214/604, iter=24750/70000, loss=0.625778, lr=0.000125 | ETA 10:52:09
2022-07-29 01:00:44,895 -     INFO - [TRAIN] epoch=214/604, iter=24800/70000, loss=0.582222, lr=0.000125 | ETA 11:23:45
2022-07-29 01:01:29,480 -     INFO - [TRAIN] epoch=215/604, iter=24850/70000, loss=0.558567, lr=0.000125 | ETA 11:26:16
2022-07-29 01:02:11,385 -     INFO - [TRAIN] epoch=215/604, iter=24900/70000, loss=0.617101, lr=0.000125 | ETA 10:50:08
2022-07-29 01:02:55,571 -     INFO - [TRAIN] epoch=216/604, iter=24950/70000, loss=0.614055, lr=0.000125 | ETA 10:56:45
2022-07-29 01:03:37,702 -     INFO - [TRAIN] epoch=216/604, iter=25000/70000, loss=0.654134, lr=0.000125 | ETA 11:25:15
2022-07-29 01:03:38,671 -     INFO - Push model to checkpoint ./output/iter_25000
2022-07-29 01:04:21,929 -     INFO - [TRAIN] epoch=216/604, iter=25050/70000, loss=0.588732, lr=0.000125 | ETA 10:21:10
2022-07-29 01:05:05,445 -     INFO - [TRAIN] epoch=217/604, iter=25100/70000, loss=0.621950, lr=0.000125 | ETA 09:54:30
2022-07-29 01:05:47,968 -     INFO - [TRAIN] epoch=217/604, iter=25150/70000, loss=0.574144, lr=0.000125 | ETA 11:24:51
2022-07-29 01:06:32,160 -     INFO - [TRAIN] epoch=218/604, iter=25200/70000, loss=0.578256, lr=0.000125 | ETA 09:51:28
2022-07-29 01:07:14,963 -     INFO - [TRAIN] epoch=218/604, iter=25250/70000, loss=0.621472, lr=0.000125 | ETA 10:23:35
2022-07-29 01:07:58,195 -     INFO - [TRAIN] epoch=219/604, iter=25300/70000, loss=0.578646, lr=0.000125 | ETA 11:15:55
2022-07-29 01:08:40,855 -     INFO - [TRAIN] epoch=219/604, iter=25350/70000, loss=0.583564, lr=0.000125 | ETA 10:47:50
2022-07-29 01:09:22,664 -     INFO - [TRAIN] epoch=219/604, iter=25400/70000, loss=0.612131, lr=0.000125 | ETA 09:34:05
2022-07-29 01:10:07,296 -     INFO - [TRAIN] epoch=220/604, iter=25450/70000, loss=0.612367, lr=0.000125 | ETA 11:09:31
2022-07-29 01:10:49,019 -     INFO - [TRAIN] epoch=220/604, iter=25500/70000, loss=0.588243, lr=0.000125 | ETA 10:37:36
2022-07-29 01:11:33,528 -     INFO - [TRAIN] epoch=221/604, iter=25550/70000, loss=0.621581, lr=0.000125 | ETA 10:27:39
2022-07-29 01:12:15,980 -     INFO - [TRAIN] epoch=221/604, iter=25600/70000, loss=0.586490, lr=0.000125 | ETA 09:58:07
2022-07-29 01:13:00,307 -     INFO - [TRAIN] epoch=222/604, iter=25650/70000, loss=0.590398, lr=0.000125 | ETA 10:31:20
2022-07-29 01:13:41,754 -     INFO - [TRAIN] epoch=222/604, iter=25700/70000, loss=0.670322, lr=0.000125 | ETA 10:05:07
2022-07-29 01:14:23,171 -     INFO - [TRAIN] epoch=222/604, iter=25750/70000, loss=0.570201, lr=0.000125 | ETA 09:27:45
2022-07-29 01:15:08,183 -     INFO - [TRAIN] epoch=223/604, iter=25800/70000, loss=0.514508, lr=0.000125 | ETA 09:50:38
2022-07-29 01:15:50,159 -     INFO - [TRAIN] epoch=223/604, iter=25850/70000, loss=0.569166, lr=0.000125 | ETA 09:38:33
2022-07-29 01:16:35,956 -     INFO - [TRAIN] epoch=224/604, iter=25900/70000, loss=0.639552, lr=0.000125 | ETA 10:53:54
2022-07-29 01:17:18,158 -     INFO - [TRAIN] epoch=224/604, iter=25950/70000, loss=0.661880, lr=0.000125 | ETA 09:31:21
2022-07-29 01:18:02,454 -     INFO - [TRAIN] epoch=225/604, iter=26000/70000, loss=0.553602, lr=0.000125 | ETA 11:40:23
2022-07-29 01:18:45,522 -     INFO - [TRAIN] epoch=225/604, iter=26050/70000, loss=0.626093, lr=0.000125 | ETA 10:04:05
2022-07-29 01:19:28,446 -     INFO - [TRAIN] epoch=225/604, iter=26100/70000, loss=0.591420, lr=0.000125 | ETA 09:40:33
2022-07-29 01:20:13,382 -     INFO - [TRAIN] epoch=226/604, iter=26150/70000, loss=0.630690, lr=0.000125 | ETA 10:22:16
2022-07-29 01:20:54,865 -     INFO - [TRAIN] epoch=226/604, iter=26200/70000, loss=0.594182, lr=0.000125 | ETA 09:35:04
2022-07-29 01:21:39,269 -     INFO - [TRAIN] epoch=227/604, iter=26250/70000, loss=0.632834, lr=0.000125 | ETA 09:59:53
2022-07-29 01:22:22,246 -     INFO - [TRAIN] epoch=227/604, iter=26300/70000, loss=0.583751, lr=0.000125 | ETA 10:24:30
2022-07-29 01:23:07,094 -     INFO - [TRAIN] epoch=228/604, iter=26350/70000, loss=0.580634, lr=0.000125 | ETA 11:17:33
2022-07-29 01:23:50,964 -     INFO - [TRAIN] epoch=228/604, iter=26400/70000, loss=0.594628, lr=0.000125 | ETA 09:28:49
2022-07-29 01:24:34,675 -     INFO - [TRAIN] epoch=229/604, iter=26450/70000, loss=0.525397, lr=0.000125 | ETA 14:46:40
2022-07-29 01:25:16,440 -     INFO - [TRAIN] epoch=229/604, iter=26500/70000, loss=0.541635, lr=0.000125 | ETA 10:34:56
2022-07-29 01:25:59,079 -     INFO - [TRAIN] epoch=229/604, iter=26550/70000, loss=0.555782, lr=0.000125 | ETA 09:44:24
2022-07-29 01:26:44,586 -     INFO - [TRAIN] epoch=230/604, iter=26600/70000, loss=0.562388, lr=0.000125 | ETA 11:33:13
2022-07-29 01:27:27,366 -     INFO - [TRAIN] epoch=230/604, iter=26650/70000, loss=0.568372, lr=0.000125 | ETA 09:45:58
2022-07-29 01:28:11,150 -     INFO - [TRAIN] epoch=231/604, iter=26700/70000, loss=0.638742, lr=0.000125 | ETA 09:45:47
2022-07-29 01:28:50,636 -     INFO - [TRAIN] epoch=231/604, iter=26750/70000, loss=0.566465, lr=0.000125 | ETA 09:35:22
2022-07-29 01:29:31,742 -     INFO - [TRAIN] epoch=232/604, iter=26800/70000, loss=0.580772, lr=0.000125 | ETA 10:54:39
2022-07-29 01:30:11,663 -     INFO - [TRAIN] epoch=232/604, iter=26850/70000, loss=0.606731, lr=0.000125 | ETA 09:26:17
2022-07-29 01:30:51,026 -     INFO - [TRAIN] epoch=232/604, iter=26900/70000, loss=0.658168, lr=0.000125 | ETA 09:10:34
2022-07-29 01:31:31,558 -     INFO - [TRAIN] epoch=233/604, iter=26950/70000, loss=0.572601, lr=0.000125 | ETA 09:09:35
2022-07-29 01:32:11,710 -     INFO - [TRAIN] epoch=233/604, iter=27000/70000, loss=0.606184, lr=0.000125 | ETA 10:16:23
2022-07-29 01:32:54,175 -     INFO - [TRAIN] epoch=234/604, iter=27050/70000, loss=0.581245, lr=0.000125 | ETA 10:14:35
2022-07-29 01:33:34,990 -     INFO - [TRAIN] epoch=234/604, iter=27100/70000, loss=0.550866, lr=0.000125 | ETA 09:32:02
2022-07-29 01:34:18,550 -     INFO - [TRAIN] epoch=235/604, iter=27150/70000, loss=0.555999, lr=0.000125 | ETA 09:54:46
2022-07-29 01:34:59,836 -     INFO - [TRAIN] epoch=235/604, iter=27200/70000, loss=0.583380, lr=0.000125 | ETA 09:33:12
2022-07-29 01:35:40,637 -     INFO - [TRAIN] epoch=235/604, iter=27250/70000, loss=0.550250, lr=0.000125 | ETA 09:16:31
2022-07-29 01:36:24,021 -     INFO - [TRAIN] epoch=236/604, iter=27300/70000, loss=0.600791, lr=0.000125 | ETA 09:18:31
2022-07-29 01:37:05,350 -     INFO - [TRAIN] epoch=236/604, iter=27350/70000, loss=0.549090, lr=0.000125 | ETA 10:19:33
2022-07-29 01:37:47,992 -     INFO - [TRAIN] epoch=237/604, iter=27400/70000, loss=0.582288, lr=0.000125 | ETA 09:21:28
2022-07-29 01:38:29,154 -     INFO - [TRAIN] epoch=237/604, iter=27450/70000, loss=0.619155, lr=0.000125 | ETA 10:30:08
2022-07-29 01:39:12,669 -     INFO - [TRAIN] epoch=238/604, iter=27500/70000, loss=0.609233, lr=0.000125 | ETA 12:27:22
2022-07-29 01:39:54,567 -     INFO - [TRAIN] epoch=238/604, iter=27550/70000, loss=0.579895, lr=0.000125 | ETA 09:56:48
2022-07-29 01:40:36,714 -     INFO - [TRAIN] epoch=238/604, iter=27600/70000, loss=0.597143, lr=0.000125 | ETA 10:15:46
2022-07-29 01:41:20,106 -     INFO - [TRAIN] epoch=239/604, iter=27650/70000, loss=0.586648, lr=0.000125 | ETA 09:14:44
2022-07-29 01:42:02,507 -     INFO - [TRAIN] epoch=239/604, iter=27700/70000, loss=0.594463, lr=0.000125 | ETA 10:11:17
2022-07-29 01:42:46,292 -     INFO - [TRAIN] epoch=240/604, iter=27750/70000, loss=0.568939, lr=0.000125 | ETA 09:08:52
2022-07-29 01:43:28,373 -     INFO - [TRAIN] epoch=240/604, iter=27800/70000, loss=0.544650, lr=0.000125 | ETA 09:33:03
2022-07-29 01:44:12,055 -     INFO - [TRAIN] epoch=241/604, iter=27850/70000, loss=0.570590, lr=0.000125 | ETA 09:51:31
2022-07-29 01:44:53,471 -     INFO - [TRAIN] epoch=241/604, iter=27900/70000, loss=0.588615, lr=0.000125 | ETA 09:21:01
2022-07-29 01:45:35,673 -     INFO - [TRAIN] epoch=241/604, iter=27950/70000, loss=0.603876, lr=0.000125 | ETA 10:04:10
2022-07-29 01:46:19,643 -     INFO - [TRAIN] epoch=242/604, iter=28000/70000, loss=0.660089, lr=0.000125 | ETA 09:40:54
2022-07-29 01:47:02,737 -     INFO - [TRAIN] epoch=242/604, iter=28050/70000, loss=0.592971, lr=0.000125 | ETA 10:26:07
2022-07-29 01:47:47,004 -     INFO - [TRAIN] epoch=243/604, iter=28100/70000, loss=0.560808, lr=0.000125 | ETA 10:24:41
2022-07-29 01:48:30,609 -     INFO - [TRAIN] epoch=243/604, iter=28150/70000, loss=0.562882, lr=0.000125 | ETA 09:45:25
2022-07-29 01:49:14,409 -     INFO - [TRAIN] epoch=244/604, iter=28200/70000, loss=0.590030, lr=0.000125 | ETA 09:56:14
2022-07-29 01:49:56,775 -     INFO - [TRAIN] epoch=244/604, iter=28250/70000, loss=0.606214, lr=0.000125 | ETA 09:56:34
2022-07-29 01:50:38,436 -     INFO - [TRAIN] epoch=244/604, iter=28300/70000, loss=0.550282, lr=0.000125 | ETA 09:04:26
2022-07-29 01:51:21,488 -     INFO - [TRAIN] epoch=245/604, iter=28350/70000, loss=0.558779, lr=0.000125 | ETA 09:57:31
2022-07-29 01:52:04,072 -     INFO - [TRAIN] epoch=245/604, iter=28400/70000, loss=0.581029, lr=0.000125 | ETA 09:45:56
2022-07-29 01:52:47,831 -     INFO - [TRAIN] epoch=246/604, iter=28450/70000, loss=0.548997, lr=0.000125 | ETA 09:33:44
2022-07-29 01:53:30,317 -     INFO - [TRAIN] epoch=246/604, iter=28500/70000, loss=0.569658, lr=0.000125 | ETA 10:39:20
2022-07-29 01:54:13,729 -     INFO - [TRAIN] epoch=247/604, iter=28550/70000, loss=0.571500, lr=0.000125 | ETA 11:36:01
2022-07-29 01:54:58,663 -     INFO - [TRAIN] epoch=247/604, iter=28600/70000, loss=0.562708, lr=0.000125 | ETA 10:22:34
2022-07-29 01:55:38,917 -     INFO - [TRAIN] epoch=247/604, iter=28650/70000, loss=0.572870, lr=0.000125 | ETA 09:06:59
2022-07-29 01:56:22,755 -     INFO - [TRAIN] epoch=248/604, iter=28700/70000, loss=0.594463, lr=0.000125 | ETA 09:50:03
2022-07-29 01:57:04,736 -     INFO - [TRAIN] epoch=248/604, iter=28750/70000, loss=0.556657, lr=0.000125 | ETA 09:16:33
2022-07-29 01:57:48,215 -     INFO - [TRAIN] epoch=249/604, iter=28800/70000, loss=0.572942, lr=0.000125 | ETA 09:31:50
2022-07-29 01:58:31,140 -     INFO - [TRAIN] epoch=249/604, iter=28850/70000, loss=0.495544, lr=0.000125 | ETA 08:57:44
2022-07-29 01:59:14,608 -     INFO - [TRAIN] epoch=250/604, iter=28900/70000, loss=0.581509, lr=0.000125 | ETA 09:16:09
2022-07-29 01:59:56,430 -     INFO - [TRAIN] epoch=250/604, iter=28950/70000, loss=0.538279, lr=0.000125 | ETA 09:15:54
2022-07-29 02:00:38,590 -     INFO - [TRAIN] epoch=250/604, iter=29000/70000, loss=0.577074, lr=0.000125 | ETA 08:57:43
2022-07-29 02:01:23,242 -     INFO - [TRAIN] epoch=251/604, iter=29050/70000, loss=0.532345, lr=0.000125 | ETA 09:36:55
2022-07-29 02:02:04,677 -     INFO - [TRAIN] epoch=251/604, iter=29100/70000, loss=0.557183, lr=0.000125 | ETA 09:25:23
2022-07-29 02:02:48,639 -     INFO - [TRAIN] epoch=252/604, iter=29150/70000, loss=0.575828, lr=0.000125 | ETA 10:25:03
2022-07-29 02:03:31,337 -     INFO - [TRAIN] epoch=252/604, iter=29200/70000, loss=0.551132, lr=0.000125 | ETA 08:54:56
2022-07-29 02:04:15,344 -     INFO - [TRAIN] epoch=253/604, iter=29250/70000, loss=0.589215, lr=0.000125 | ETA 10:08:59
2022-07-29 02:04:59,442 -     INFO - [TRAIN] epoch=253/604, iter=29300/70000, loss=0.527751, lr=0.000125 | ETA 10:25:23
2022-07-29 02:05:43,717 -     INFO - [TRAIN] epoch=254/604, iter=29350/70000, loss=0.664222, lr=0.000125 | ETA 13:58:58
2022-07-29 02:06:27,276 -     INFO - [TRAIN] epoch=254/604, iter=29400/70000, loss=0.585903, lr=0.000125 | ETA 08:46:56
2022-07-29 02:07:08,949 -     INFO - [TRAIN] epoch=254/604, iter=29450/70000, loss=0.498069, lr=0.000125 | ETA 11:06:45
2022-07-29 02:07:54,009 -     INFO - [TRAIN] epoch=255/604, iter=29500/70000, loss=0.568897, lr=0.000125 | ETA 09:02:24
2022-07-29 02:08:36,150 -     INFO - [TRAIN] epoch=255/604, iter=29550/70000, loss=0.578399, lr=0.000125 | ETA 08:53:42
2022-07-29 02:09:21,035 -     INFO - [TRAIN] epoch=256/604, iter=29600/70000, loss=0.507853, lr=0.000125 | ETA 10:06:14
2022-07-29 02:10:03,777 -     INFO - [TRAIN] epoch=256/604, iter=29650/70000, loss=0.531577, lr=0.000125 | ETA 11:53:00
2022-07-29 02:10:46,856 -     INFO - [TRAIN] epoch=257/604, iter=29700/70000, loss=0.556371, lr=0.000125 | ETA 10:32:18
2022-07-29 02:11:29,288 -     INFO - [TRAIN] epoch=257/604, iter=29750/70000, loss=0.546333, lr=0.000125 | ETA 08:57:20
2022-07-29 02:12:10,025 -     INFO - [TRAIN] epoch=257/604, iter=29800/70000, loss=0.543362, lr=0.000125 | ETA 08:43:50
2022-07-29 02:12:53,699 -     INFO - [TRAIN] epoch=258/604, iter=29850/70000, loss=0.568279, lr=0.000125 | ETA 08:53:39
2022-07-29 02:13:36,916 -     INFO - [TRAIN] epoch=258/604, iter=29900/70000, loss=0.543215, lr=0.000125 | ETA 09:18:43
2022-07-29 02:14:20,446 -     INFO - [TRAIN] epoch=259/604, iter=29950/70000, loss=0.524412, lr=0.000125 | ETA 09:04:17
2022-07-29 02:15:05,279 -     INFO - [TRAIN] epoch=259/604, iter=30000/70000, loss=0.562381, lr=0.000125 | ETA 09:26:52
2022-07-29 02:15:05,488 -     INFO - Pop model from ./output/iter_5000
2022-07-29 02:15:06,241 -     INFO - Push model to checkpoint ./output/iter_30000
2022-07-29 02:15:50,275 -     INFO - [TRAIN] epoch=260/604, iter=30050/70000, loss=0.539020, lr=0.000125 | ETA 09:28:24
2022-07-29 02:16:32,677 -     INFO - [TRAIN] epoch=260/604, iter=30100/70000, loss=0.592887, lr=0.000125 | ETA 09:10:28
2022-07-29 02:17:15,492 -     INFO - [TRAIN] epoch=260/604, iter=30150/70000, loss=0.544994, lr=0.000125 | ETA 08:59:07
2022-07-29 02:18:00,291 -     INFO - [TRAIN] epoch=261/604, iter=30200/70000, loss=0.555801, lr=0.000125 | ETA 09:50:34
2022-07-29 02:18:44,011 -     INFO - [TRAIN] epoch=261/604, iter=30250/70000, loss=0.531350, lr=0.000125 | ETA 08:51:24
2022-07-29 02:19:27,874 -     INFO - [TRAIN] epoch=262/604, iter=30300/70000, loss=0.551540, lr=0.000125 | ETA 09:24:34
2022-07-29 02:20:09,985 -     INFO - [TRAIN] epoch=262/604, iter=30350/70000, loss=0.560665, lr=0.000125 | ETA 10:20:17
2022-07-29 02:20:54,999 -     INFO - [TRAIN] epoch=263/604, iter=30400/70000, loss=0.553711, lr=0.000125 | ETA 08:50:22
2022-07-29 02:21:37,437 -     INFO - [TRAIN] epoch=263/604, iter=30450/70000, loss=0.578872, lr=0.000125 | ETA 08:50:20
2022-07-29 02:22:21,511 -     INFO - [TRAIN] epoch=263/604, iter=30500/70000, loss=0.576490, lr=0.000125 | ETA 11:36:37
2022-07-29 02:23:05,181 -     INFO - [TRAIN] epoch=264/604, iter=30550/70000, loss=0.609568, lr=0.000125 | ETA 08:49:19
2022-07-29 02:23:49,384 -     INFO - [TRAIN] epoch=264/604, iter=30600/70000, loss=0.574866, lr=0.000125 | ETA 10:54:29
2022-07-29 02:24:33,888 -     INFO - [TRAIN] epoch=265/604, iter=30650/70000, loss=0.496704, lr=0.000125 | ETA 08:38:39
2022-07-29 02:25:16,000 -     INFO - [TRAIN] epoch=265/604, iter=30700/70000, loss=0.556552, lr=0.000125 | ETA 08:46:56
2022-07-29 02:26:00,345 -     INFO - [TRAIN] epoch=266/604, iter=30750/70000, loss=0.537965, lr=0.000125 | ETA 08:47:20
2022-07-29 02:26:42,295 -     INFO - [TRAIN] epoch=266/604, iter=30800/70000, loss=0.560432, lr=0.000125 | ETA 10:52:25
2022-07-29 02:27:26,133 -     INFO - [TRAIN] epoch=266/604, iter=30850/70000, loss=0.544704, lr=0.000125 | ETA 10:54:02
2022-07-29 02:28:09,980 -     INFO - [TRAIN] epoch=267/604, iter=30900/70000, loss=0.555024, lr=0.000125 | ETA 10:20:13
2022-07-29 02:28:52,571 -     INFO - [TRAIN] epoch=267/604, iter=30950/70000, loss=0.524044, lr=0.000125 | ETA 11:02:28
2022-07-29 02:29:37,536 -     INFO - [TRAIN] epoch=268/604, iter=31000/70000, loss=0.522543, lr=0.000125 | ETA 10:48:23
2022-07-29 02:30:20,380 -     INFO - [TRAIN] epoch=268/604, iter=31050/70000, loss=0.557769, lr=0.000125 | ETA 09:00:55
2022-07-29 02:31:04,387 -     INFO - [TRAIN] epoch=269/604, iter=31100/70000, loss=0.588081, lr=0.000125 | ETA 09:58:16
2022-07-29 02:31:47,571 -     INFO - [TRAIN] epoch=269/604, iter=31150/70000, loss=0.543281, lr=0.000125 | ETA 08:50:08
2022-07-29 02:32:30,077 -     INFO - [TRAIN] epoch=269/604, iter=31200/70000, loss=0.552721, lr=0.000125 | ETA 08:40:47
2022-07-29 02:33:14,820 -     INFO - [TRAIN] epoch=270/604, iter=31250/70000, loss=0.465793, lr=0.000125 | ETA 10:00:22
2022-07-29 02:33:57,741 -     INFO - [TRAIN] epoch=270/604, iter=31300/70000, loss=0.521231, lr=0.000125 | ETA 08:38:55
2022-07-29 02:34:41,921 -     INFO - [TRAIN] epoch=271/604, iter=31350/70000, loss=0.547248, lr=0.000125 | ETA 08:29:52
2022-07-29 02:35:24,623 -     INFO - [TRAIN] epoch=271/604, iter=31400/70000, loss=0.550074, lr=0.000125 | ETA 10:20:09
2022-07-29 02:36:08,325 -     INFO - [TRAIN] epoch=272/604, iter=31450/70000, loss=0.511532, lr=0.000125 | ETA 08:35:58
2022-07-29 02:36:50,695 -     INFO - [TRAIN] epoch=272/604, iter=31500/70000, loss=0.539330, lr=0.000125 | ETA 09:19:08
2022-07-29 02:37:33,743 -     INFO - [TRAIN] epoch=272/604, iter=31550/70000, loss=0.564949, lr=0.000125 | ETA 08:34:14
2022-07-29 02:38:18,701 -     INFO - [TRAIN] epoch=273/604, iter=31600/70000, loss=0.542930, lr=0.000125 | ETA 09:37:23
2022-07-29 02:39:01,482 -     INFO - [TRAIN] epoch=273/604, iter=31650/70000, loss=0.492917, lr=0.000125 | ETA 09:24:05
2022-07-29 02:39:46,496 -     INFO - [TRAIN] epoch=274/604, iter=31700/70000, loss=0.556388, lr=0.000125 | ETA 09:01:36
2022-07-29 02:40:29,096 -     INFO - [TRAIN] epoch=274/604, iter=31750/70000, loss=0.574906, lr=0.000125 | ETA 08:55:07
2022-07-29 02:41:13,775 -     INFO - [TRAIN] epoch=275/604, iter=31800/70000, loss=0.519398, lr=0.000125 | ETA 09:46:32
2022-07-29 02:41:55,611 -     INFO - [TRAIN] epoch=275/604, iter=31850/70000, loss=0.521354, lr=0.000125 | ETA 08:28:57
2022-07-29 02:42:37,290 -     INFO - [TRAIN] epoch=275/604, iter=31900/70000, loss=0.475124, lr=0.000125 | ETA 08:23:10
2022-07-29 02:43:23,268 -     INFO - [TRAIN] epoch=276/604, iter=31950/70000, loss=0.507178, lr=0.000125 | ETA 11:48:42
2022-07-29 02:44:05,285 -     INFO - [TRAIN] epoch=276/604, iter=32000/70000, loss=0.577840, lr=0.000125 | ETA 08:30:20
2022-07-29 02:44:50,083 -     INFO - [TRAIN] epoch=277/604, iter=32050/70000, loss=0.531746, lr=0.000125 | ETA 09:50:36
2022-07-29 02:45:32,648 -     INFO - [TRAIN] epoch=277/604, iter=32100/70000, loss=0.510734, lr=0.000125 | ETA 08:13:33
2022-07-29 02:46:17,099 -     INFO - [TRAIN] epoch=278/604, iter=32150/70000, loss=0.529222, lr=0.000125 | ETA 08:44:35
2022-07-29 02:47:00,937 -     INFO - [TRAIN] epoch=278/604, iter=32200/70000, loss=0.516750, lr=0.000125 | ETA 08:20:05
2022-07-29 02:47:44,475 -     INFO - [TRAIN] epoch=279/604, iter=32250/70000, loss=0.518493, lr=0.000125 | ETA 12:45:40
2022-07-29 02:48:27,967 -     INFO - [TRAIN] epoch=279/604, iter=32300/70000, loss=0.497530, lr=0.000125 | ETA 08:34:33
2022-07-29 02:49:10,907 -     INFO - [TRAIN] epoch=279/604, iter=32350/70000, loss=0.532233, lr=0.000125 | ETA 09:47:11
2022-07-29 02:49:54,360 -     INFO - [TRAIN] epoch=280/604, iter=32400/70000, loss=0.527239, lr=0.000125 | ETA 08:20:27
2022-07-29 02:50:37,179 -     INFO - [TRAIN] epoch=280/604, iter=32450/70000, loss=0.527419, lr=0.000125 | ETA 08:25:33
2022-07-29 02:51:22,201 -     INFO - [TRAIN] epoch=281/604, iter=32500/70000, loss=0.507121, lr=0.000125 | ETA 08:15:15
2022-07-29 02:52:05,855 -     INFO - [TRAIN] epoch=281/604, iter=32550/70000, loss=0.525804, lr=0.000125 | ETA 08:03:26
2022-07-29 02:52:49,591 -     INFO - [TRAIN] epoch=282/604, iter=32600/70000, loss=0.554223, lr=0.000125 | ETA 09:28:40
2022-07-29 02:53:32,298 -     INFO - [TRAIN] epoch=282/604, iter=32650/70000, loss=0.557061, lr=0.000125 | ETA 07:57:47
2022-07-29 02:54:15,266 -     INFO - [TRAIN] epoch=282/604, iter=32700/70000, loss=0.481634, lr=0.000125 | ETA 08:39:21
2022-07-29 02:55:00,719 -     INFO - [TRAIN] epoch=283/604, iter=32750/70000, loss=0.576182, lr=0.000125 | ETA 08:31:35
2022-07-29 02:55:45,399 -     INFO - [TRAIN] epoch=283/604, iter=32800/70000, loss=0.530807, lr=0.000125 | ETA 10:04:16
2022-07-29 02:56:30,858 -     INFO - [TRAIN] epoch=284/604, iter=32850/70000, loss=0.502056, lr=0.000125 | ETA 10:58:12
2022-07-29 02:57:14,245 -     INFO - [TRAIN] epoch=284/604, iter=32900/70000, loss=0.541743, lr=0.000125 | ETA 11:28:13
2022-07-29 02:57:57,780 -     INFO - [TRAIN] epoch=285/604, iter=32950/70000, loss=0.503844, lr=0.000125 | ETA 08:43:04
2022-07-29 02:58:40,225 -     INFO - [TRAIN] epoch=285/604, iter=33000/70000, loss=0.578289, lr=0.000125 | ETA 09:25:45
2022-07-29 02:59:23,339 -     INFO - [TRAIN] epoch=285/604, iter=33050/70000, loss=0.486942, lr=0.000125 | ETA 08:31:08
2022-07-29 03:00:06,450 -     INFO - [TRAIN] epoch=286/604, iter=33100/70000, loss=0.473158, lr=0.000125 | ETA 08:20:46
2022-07-29 03:00:49,332 -     INFO - [TRAIN] epoch=286/604, iter=33150/70000, loss=0.577865, lr=0.000125 | ETA 07:55:24
2022-07-29 03:01:32,621 -     INFO - [TRAIN] epoch=287/604, iter=33200/70000, loss=0.475221, lr=0.000125 | ETA 08:42:01
2022-07-29 03:02:15,885 -     INFO - [TRAIN] epoch=287/604, iter=33250/70000, loss=0.525761, lr=0.000125 | ETA 07:57:07
2022-07-29 03:03:00,633 -     INFO - [TRAIN] epoch=288/604, iter=33300/70000, loss=0.493618, lr=0.000125 | ETA 08:49:16
2022-07-29 03:03:43,920 -     INFO - [TRAIN] epoch=288/604, iter=33350/70000, loss=0.505616, lr=0.000125 | ETA 08:18:55
2022-07-29 03:04:26,609 -     INFO - [TRAIN] epoch=288/604, iter=33400/70000, loss=0.519592, lr=0.000125 | ETA 08:08:35
2022-07-29 03:05:09,924 -     INFO - [TRAIN] epoch=289/604, iter=33450/70000, loss=0.522897, lr=0.000125 | ETA 10:21:34
2022-07-29 03:05:53,141 -     INFO - [TRAIN] epoch=289/604, iter=33500/70000, loss=0.518376, lr=0.000125 | ETA 08:48:49
2022-07-29 03:06:37,501 -     INFO - [TRAIN] epoch=290/604, iter=33550/70000, loss=0.549605, lr=0.000125 | ETA 08:05:26
2022-07-29 03:07:19,460 -     INFO - [TRAIN] epoch=290/604, iter=33600/70000, loss=0.514445, lr=0.000125 | ETA 08:42:24
2022-07-29 03:08:04,234 -     INFO - [TRAIN] epoch=291/604, iter=33650/70000, loss=0.550409, lr=0.000125 | ETA 08:06:32
2022-07-29 03:08:45,773 -     INFO - [TRAIN] epoch=291/604, iter=33700/70000, loss=0.546309, lr=0.000125 | ETA 08:05:53
2022-07-29 03:09:28,001 -     INFO - [TRAIN] epoch=291/604, iter=33750/70000, loss=0.521791, lr=0.000125 | ETA 08:07:34
2022-07-29 03:10:12,312 -     INFO - [TRAIN] epoch=292/604, iter=33800/70000, loss=0.505330, lr=0.000125 | ETA 09:19:46
2022-07-29 03:10:54,758 -     INFO - [TRAIN] epoch=292/604, iter=33850/70000, loss=0.527235, lr=0.000125 | ETA 08:06:08
2022-07-29 03:11:39,290 -     INFO - [TRAIN] epoch=293/604, iter=33900/70000, loss=0.488884, lr=0.000125 | ETA 09:02:39
2022-07-29 03:12:21,262 -     INFO - [TRAIN] epoch=293/604, iter=33950/70000, loss=0.513187, lr=0.000125 | ETA 08:18:32
2022-07-29 03:13:04,825 -     INFO - [TRAIN] epoch=294/604, iter=34000/70000, loss=0.499847, lr=0.000125 | ETA 08:08:19
2022-07-29 03:13:46,843 -     INFO - [TRAIN] epoch=294/604, iter=34050/70000, loss=0.494162, lr=0.000125 | ETA 08:08:37
2022-07-29 03:14:29,890 -     INFO - [TRAIN] epoch=294/604, iter=34100/70000, loss=0.472782, lr=0.000125 | ETA 07:55:01
2022-07-29 03:15:14,637 -     INFO - [TRAIN] epoch=295/604, iter=34150/70000, loss=0.565922, lr=0.000125 | ETA 08:32:20
2022-07-29 03:15:56,291 -     INFO - [TRAIN] epoch=295/604, iter=34200/70000, loss=0.571904, lr=0.000125 | ETA 08:16:59
2022-07-29 03:16:39,274 -     INFO - [TRAIN] epoch=296/604, iter=34250/70000, loss=0.517293, lr=0.000125 | ETA 07:44:50
2022-07-29 03:17:21,889 -     INFO - [TRAIN] epoch=296/604, iter=34300/70000, loss=0.506227, lr=0.000125 | ETA 07:48:23
2022-07-29 03:18:05,733 -     INFO - [TRAIN] epoch=297/604, iter=34350/70000, loss=0.503501, lr=0.000125 | ETA 08:25:02
2022-07-29 03:18:48,872 -     INFO - [TRAIN] epoch=297/604, iter=34400/70000, loss=0.534117, lr=0.000125 | ETA 08:00:51
2022-07-29 03:19:28,346 -     INFO - [TRAIN] epoch=297/604, iter=34450/70000, loss=0.494230, lr=0.000125 | ETA 07:49:14
2022-07-29 03:20:09,310 -     INFO - [TRAIN] epoch=298/604, iter=34500/70000, loss=0.478891, lr=0.000125 | ETA 08:12:31
2022-07-29 03:20:49,327 -     INFO - [TRAIN] epoch=298/604, iter=34550/70000, loss=0.523498, lr=0.000125 | ETA 07:55:49
2022-07-29 03:21:30,754 -     INFO - [TRAIN] epoch=299/604, iter=34600/70000, loss=0.492362, lr=0.000125 | ETA 07:35:46
2022-07-29 03:22:09,969 -     INFO - [TRAIN] epoch=299/604, iter=34650/70000, loss=0.497601, lr=0.000125 | ETA 07:55:48
2022-07-29 03:22:51,178 -     INFO - [TRAIN] epoch=300/604, iter=34700/70000, loss=0.528820, lr=0.000125 | ETA 08:06:29
2022-07-29 03:23:30,442 -     INFO - [TRAIN] epoch=300/604, iter=34750/70000, loss=0.499630, lr=0.000125 | ETA 07:55:45
2022-07-29 03:24:11,008 -     INFO - [TRAIN] epoch=300/604, iter=34800/70000, loss=0.496291, lr=0.000125 | ETA 08:22:05
2022-07-29 03:24:53,378 -     INFO - [TRAIN] epoch=301/604, iter=34850/70000, loss=0.569822, lr=0.000125 | ETA 07:34:02
2022-07-29 03:25:34,052 -     INFO - [TRAIN] epoch=301/604, iter=34900/70000, loss=0.526144, lr=0.000125 | ETA 07:44:26
2022-07-29 03:26:17,741 -     INFO - [TRAIN] epoch=302/604, iter=34950/70000, loss=0.501104, lr=0.000125 | ETA 07:46:08
2022-07-29 03:26:59,702 -     INFO - [TRAIN] epoch=302/604, iter=35000/70000, loss=0.505381, lr=0.000125 | ETA 07:56:41
2022-07-29 03:26:59,905 -     INFO - Pop model from ./output/iter_10000
2022-07-29 03:27:00,632 -     INFO - Push model to checkpoint ./output/iter_35000
2022-07-29 03:27:43,779 -     INFO - [TRAIN] epoch=303/604, iter=35050/70000, loss=0.504997, lr=0.000125 | ETA 07:39:04
2022-07-29 03:28:25,262 -     INFO - [TRAIN] epoch=303/604, iter=35100/70000, loss=0.504150, lr=0.000125 | ETA 07:42:46
2022-07-29 03:29:08,485 -     INFO - [TRAIN] epoch=304/604, iter=35150/70000, loss=0.462112, lr=0.000125 | ETA 11:54:57
2022-07-29 03:29:50,744 -     INFO - [TRAIN] epoch=304/604, iter=35200/70000, loss=0.505201, lr=0.000125 | ETA 07:44:53
2022-07-29 03:30:32,195 -     INFO - [TRAIN] epoch=304/604, iter=35250/70000, loss=0.490427, lr=0.000125 | ETA 07:39:23
2022-07-29 03:31:15,460 -     INFO - [TRAIN] epoch=305/604, iter=35300/70000, loss=0.543514, lr=0.000125 | ETA 08:24:47
2022-07-29 03:31:58,618 -     INFO - [TRAIN] epoch=305/604, iter=35350/70000, loss=0.527306, lr=0.000125 | ETA 08:52:39
2022-07-29 03:32:42,291 -     INFO - [TRAIN] epoch=306/604, iter=35400/70000, loss=0.508748, lr=0.000125 | ETA 08:01:42
2022-07-29 03:33:24,320 -     INFO - [TRAIN] epoch=306/604, iter=35450/70000, loss=0.501621, lr=0.000125 | ETA 07:51:08
2022-07-29 03:34:09,605 -     INFO - [TRAIN] epoch=307/604, iter=35500/70000, loss=0.536367, lr=0.000125 | ETA 10:44:01
2022-07-29 03:34:52,088 -     INFO - [TRAIN] epoch=307/604, iter=35550/70000, loss=0.508106, lr=0.000125 | ETA 08:12:47
2022-07-29 03:35:34,432 -     INFO - [TRAIN] epoch=307/604, iter=35600/70000, loss=0.550471, lr=0.000125 | ETA 08:55:40
2022-07-29 03:36:17,813 -     INFO - [TRAIN] epoch=308/604, iter=35650/70000, loss=0.497873, lr=0.000125 | ETA 07:29:09
2022-07-29 03:36:59,521 -     INFO - [TRAIN] epoch=308/604, iter=35700/70000, loss=0.495610, lr=0.000125 | ETA 08:15:16
2022-07-29 03:37:42,942 -     INFO - [TRAIN] epoch=309/604, iter=35750/70000, loss=0.487498, lr=0.000125 | ETA 07:27:40
2022-07-29 03:38:25,001 -     INFO - [TRAIN] epoch=309/604, iter=35800/70000, loss=0.518329, lr=0.000125 | ETA 08:04:28
2022-07-29 03:39:12,593 -     INFO - [TRAIN] epoch=310/604, iter=35850/70000, loss=0.487428, lr=0.000125 | ETA 07:55:25
2022-07-29 03:39:54,130 -     INFO - [TRAIN] epoch=310/604, iter=35900/70000, loss=0.481469, lr=0.000125 | ETA 08:14:18
2022-07-29 03:40:36,924 -     INFO - [TRAIN] epoch=310/604, iter=35950/70000, loss=0.502882, lr=0.000125 | ETA 09:21:18
2022-07-29 03:41:19,782 -     INFO - [TRAIN] epoch=311/604, iter=36000/70000, loss=0.534364, lr=0.000013 | ETA 07:42:19
2022-07-29 03:42:02,583 -     INFO - [TRAIN] epoch=311/604, iter=36050/70000, loss=0.433555, lr=0.000013 | ETA 07:46:34
2022-07-29 03:42:44,508 -     INFO - [TRAIN] epoch=312/604, iter=36100/70000, loss=0.443050, lr=0.000013 | ETA 07:33:28
2022-07-29 03:43:25,100 -     INFO - [TRAIN] epoch=312/604, iter=36150/70000, loss=0.421026, lr=0.000013 | ETA 07:36:05
2022-07-29 03:44:06,747 -     INFO - [TRAIN] epoch=313/604, iter=36200/70000, loss=0.408486, lr=0.000013 | ETA 07:15:11
2022-07-29 03:44:47,384 -     INFO - [TRAIN] epoch=313/604, iter=36250/70000, loss=0.410193, lr=0.000013 | ETA 07:22:25
2022-07-29 03:45:26,383 -     INFO - [TRAIN] epoch=313/604, iter=36300/70000, loss=0.393392, lr=0.000013 | ETA 07:18:03
2022-07-29 03:46:08,359 -     INFO - [TRAIN] epoch=314/604, iter=36350/70000, loss=0.385468, lr=0.000013 | ETA 08:13:37
2022-07-29 03:46:48,624 -     INFO - [TRAIN] epoch=314/604, iter=36400/70000, loss=0.377680, lr=0.000013 | ETA 07:21:45
2022-07-29 03:47:31,089 -     INFO - [TRAIN] epoch=315/604, iter=36450/70000, loss=0.412999, lr=0.000013 | ETA 07:13:30
2022-07-29 03:48:10,629 -     INFO - [TRAIN] epoch=315/604, iter=36500/70000, loss=0.392542, lr=0.000013 | ETA 07:10:52
2022-07-29 03:48:52,134 -     INFO - [TRAIN] epoch=316/604, iter=36550/70000, loss=0.364544, lr=0.000013 | ETA 07:19:12
2022-07-29 03:49:32,055 -     INFO - [TRAIN] epoch=316/604, iter=36600/70000, loss=0.343263, lr=0.000013 | ETA 07:16:17
2022-07-29 03:50:12,658 -     INFO - [TRAIN] epoch=316/604, iter=36650/70000, loss=0.398123, lr=0.000013 | ETA 07:12:56
2022-07-29 03:50:54,626 -     INFO - [TRAIN] epoch=317/604, iter=36700/70000, loss=0.330536, lr=0.000013 | ETA 07:16:09
2022-07-29 03:51:34,237 -     INFO - [TRAIN] epoch=317/604, iter=36750/70000, loss=0.335615, lr=0.000013 | ETA 07:16:27
2022-07-29 03:52:16,351 -     INFO - [TRAIN] epoch=318/604, iter=36800/70000, loss=0.353763, lr=0.000013 | ETA 07:24:34
2022-07-29 03:52:57,258 -     INFO - [TRAIN] epoch=318/604, iter=36850/70000, loss=0.351785, lr=0.000013 | ETA 07:20:15
2022-07-29 03:53:41,181 -     INFO - [TRAIN] epoch=319/604, iter=36900/70000, loss=0.358764, lr=0.000013 | ETA 07:17:24
2022-07-29 03:54:25,472 -     INFO - [TRAIN] epoch=319/604, iter=36950/70000, loss=0.385465, lr=0.000013 | ETA 08:14:17
2022-07-29 03:55:08,471 -     INFO - [TRAIN] epoch=319/604, iter=37000/70000, loss=0.377318, lr=0.000013 | ETA 07:03:42
2022-07-29 03:55:52,354 -     INFO - [TRAIN] epoch=320/604, iter=37050/70000, loss=0.371456, lr=0.000013 | ETA 07:30:20
2022-07-29 03:56:35,944 -     INFO - [TRAIN] epoch=320/604, iter=37100/70000, loss=0.350945, lr=0.000013 | ETA 07:13:47
2022-07-29 03:57:20,528 -     INFO - [TRAIN] epoch=321/604, iter=37150/70000, loss=0.345470, lr=0.000013 | ETA 08:55:26
2022-07-29 03:58:04,019 -     INFO - [TRAIN] epoch=321/604, iter=37200/70000, loss=0.325686, lr=0.000013 | ETA 07:45:41
2022-07-29 03:58:48,953 -     INFO - [TRAIN] epoch=322/604, iter=37250/70000, loss=0.365516, lr=0.000013 | ETA 07:58:37
2022-07-29 03:59:31,127 -     INFO - [TRAIN] epoch=322/604, iter=37300/70000, loss=0.332741, lr=0.000013 | ETA 07:52:43
2022-07-29 04:00:13,998 -     INFO - [TRAIN] epoch=322/604, iter=37350/70000, loss=0.360367, lr=0.000013 | ETA 07:05:24
2022-07-29 04:00:57,695 -     INFO - [TRAIN] epoch=323/604, iter=37400/70000, loss=0.363919, lr=0.000013 | ETA 07:46:01
2022-07-29 04:01:39,542 -     INFO - [TRAIN] epoch=323/604, iter=37450/70000, loss=0.371516, lr=0.000013 | ETA 07:00:48
2022-07-29 04:02:24,108 -     INFO - [TRAIN] epoch=324/604, iter=37500/70000, loss=0.347898, lr=0.000013 | ETA 07:27:16
2022-07-29 04:03:05,606 -     INFO - [TRAIN] epoch=324/604, iter=37550/70000, loss=0.314692, lr=0.000013 | ETA 07:01:19
2022-07-29 04:03:49,479 -     INFO - [TRAIN] epoch=325/604, iter=37600/70000, loss=0.330930, lr=0.000013 | ETA 07:38:58
2022-07-29 04:04:31,296 -     INFO - [TRAIN] epoch=325/604, iter=37650/70000, loss=0.332664, lr=0.000013 | ETA 07:26:40
2022-07-29 04:05:12,970 -     INFO - [TRAIN] epoch=325/604, iter=37700/70000, loss=0.348666, lr=0.000013 | ETA 06:54:31
2022-07-29 04:05:56,426 -     INFO - [TRAIN] epoch=326/604, iter=37750/70000, loss=0.295141, lr=0.000013 | ETA 07:55:17
2022-07-29 04:06:38,011 -     INFO - [TRAIN] epoch=326/604, iter=37800/70000, loss=0.341686, lr=0.000013 | ETA 07:00:01
2022-07-29 04:07:23,052 -     INFO - [TRAIN] epoch=327/604, iter=37850/70000, loss=0.327618, lr=0.000013 | ETA 08:11:09
2022-07-29 04:08:05,120 -     INFO - [TRAIN] epoch=327/604, iter=37900/70000, loss=0.315935, lr=0.000013 | ETA 07:09:42
2022-07-29 04:08:49,087 -     INFO - [TRAIN] epoch=328/604, iter=37950/70000, loss=0.329029, lr=0.000013 | ETA 08:06:31
2022-07-29 04:09:31,731 -     INFO - [TRAIN] epoch=328/604, iter=38000/70000, loss=0.313694, lr=0.000013 | ETA 06:55:09
2022-07-29 04:10:15,075 -     INFO - [TRAIN] epoch=329/604, iter=38050/70000, loss=0.301399, lr=0.000013 | ETA 10:52:58
2022-07-29 04:10:59,027 -     INFO - [TRAIN] epoch=329/604, iter=38100/70000, loss=0.347327, lr=0.000013 | ETA 07:39:47
2022-07-29 04:11:41,300 -     INFO - [TRAIN] epoch=329/604, iter=38150/70000, loss=0.347151, lr=0.000013 | ETA 06:53:56
2022-07-29 04:12:24,738 -     INFO - [TRAIN] epoch=330/604, iter=38200/70000, loss=0.327091, lr=0.000013 | ETA 07:05:33
2022-07-29 04:13:07,822 -     INFO - [TRAIN] epoch=330/604, iter=38250/70000, loss=0.336358, lr=0.000013 | ETA 06:57:29
2022-07-29 04:13:51,643 -     INFO - [TRAIN] epoch=331/604, iter=38300/70000, loss=0.333234, lr=0.000013 | ETA 07:33:39
2022-07-29 04:14:35,326 -     INFO - [TRAIN] epoch=331/604, iter=38350/70000, loss=0.294275, lr=0.000013 | ETA 07:35:37
2022-07-29 04:15:20,147 -     INFO - [TRAIN] epoch=332/604, iter=38400/70000, loss=0.351793, lr=0.000013 | ETA 07:54:31
2022-07-29 04:16:02,075 -     INFO - [TRAIN] epoch=332/604, iter=38450/70000, loss=0.297785, lr=0.000013 | ETA 06:48:16
2022-07-29 04:16:45,218 -     INFO - [TRAIN] epoch=332/604, iter=38500/70000, loss=0.331605, lr=0.000013 | ETA 07:06:46
2022-07-29 04:17:28,655 -     INFO - [TRAIN] epoch=333/604, iter=38550/70000, loss=0.318842, lr=0.000013 | ETA 06:52:07
2022-07-29 04:18:11,095 -     INFO - [TRAIN] epoch=333/604, iter=38600/70000, loss=0.323630, lr=0.000013 | ETA 08:02:11
2022-07-29 04:18:55,879 -     INFO - [TRAIN] epoch=334/604, iter=38650/70000, loss=0.305919, lr=0.000013 | ETA 07:05:23
2022-07-29 04:19:37,978 -     INFO - [TRAIN] epoch=334/604, iter=38700/70000, loss=0.307656, lr=0.000013 | ETA 06:51:38
2022-07-29 04:20:21,263 -     INFO - [TRAIN] epoch=335/604, iter=38750/70000, loss=0.292388, lr=0.000013 | ETA 07:41:30
2022-07-29 04:21:03,408 -     INFO - [TRAIN] epoch=335/604, iter=38800/70000, loss=0.300586, lr=0.000013 | ETA 07:38:27
2022-07-29 04:21:45,852 -     INFO - [TRAIN] epoch=335/604, iter=38850/70000, loss=0.351387, lr=0.000013 | ETA 07:23:19
2022-07-29 04:22:30,276 -     INFO - [TRAIN] epoch=336/604, iter=38900/70000, loss=0.289894, lr=0.000013 | ETA 06:44:27
2022-07-29 04:23:13,592 -     INFO - [TRAIN] epoch=336/604, iter=38950/70000, loss=0.356781, lr=0.000013 | ETA 06:56:02
2022-07-29 04:23:59,102 -     INFO - [TRAIN] epoch=337/604, iter=39000/70000, loss=0.315207, lr=0.000013 | ETA 07:01:38
2022-07-29 04:24:41,363 -     INFO - [TRAIN] epoch=337/604, iter=39050/70000, loss=0.315551, lr=0.000013 | ETA 07:29:17
2022-07-29 04:25:25,133 -     INFO - [TRAIN] epoch=338/604, iter=39100/70000, loss=0.298028, lr=0.000013 | ETA 08:25:45
2022-07-29 04:26:08,701 -     INFO - [TRAIN] epoch=338/604, iter=39150/70000, loss=0.297897, lr=0.000013 | ETA 07:14:50
2022-07-29 04:26:50,262 -     INFO - [TRAIN] epoch=338/604, iter=39200/70000, loss=0.331525, lr=0.000013 | ETA 06:57:49
2022-07-29 04:27:35,007 -     INFO - [TRAIN] epoch=339/604, iter=39250/70000, loss=0.334301, lr=0.000013 | ETA 07:17:28
2022-07-29 04:28:18,066 -     INFO - [TRAIN] epoch=339/604, iter=39300/70000, loss=0.314849, lr=0.000013 | ETA 08:21:26
2022-07-29 04:29:01,705 -     INFO - [TRAIN] epoch=340/604, iter=39350/70000, loss=0.294792, lr=0.000013 | ETA 06:53:21
2022-07-29 04:29:44,051 -     INFO - [TRAIN] epoch=340/604, iter=39400/70000, loss=0.306862, lr=0.000013 | ETA 07:55:33
2022-07-29 04:30:27,251 -     INFO - [TRAIN] epoch=341/604, iter=39450/70000, loss=0.294976, lr=0.000013 | ETA 06:57:21
2022-07-29 04:31:09,973 -     INFO - [TRAIN] epoch=341/604, iter=39500/70000, loss=0.334398, lr=0.000013 | ETA 06:51:06
2022-07-29 04:31:53,037 -     INFO - [TRAIN] epoch=341/604, iter=39550/70000, loss=0.302240, lr=0.000013 | ETA 07:32:35
2022-07-29 04:32:36,653 -     INFO - [TRAIN] epoch=342/604, iter=39600/70000, loss=0.267986, lr=0.000013 | ETA 07:17:41
2022-07-29 04:33:18,967 -     INFO - [TRAIN] epoch=342/604, iter=39650/70000, loss=0.316937, lr=0.000013 | ETA 09:19:30
2022-07-29 04:34:03,787 -     INFO - [TRAIN] epoch=343/604, iter=39700/70000, loss=0.291358, lr=0.000013 | ETA 06:53:04
2022-07-29 04:34:45,906 -     INFO - [TRAIN] epoch=343/604, iter=39750/70000, loss=0.316542, lr=0.000013 | ETA 07:14:02
2022-07-29 04:35:30,520 -     INFO - [TRAIN] epoch=344/604, iter=39800/70000, loss=0.310361, lr=0.000013 | ETA 07:27:30
2022-07-29 04:36:13,002 -     INFO - [TRAIN] epoch=344/604, iter=39850/70000, loss=0.290078, lr=0.000013 | ETA 06:33:48
2022-07-29 04:36:55,598 -     INFO - [TRAIN] epoch=344/604, iter=39900/70000, loss=0.262546, lr=0.000013 | ETA 06:49:00
2022-07-29 04:37:41,340 -     INFO - [TRAIN] epoch=345/604, iter=39950/70000, loss=0.279026, lr=0.000013 | ETA 06:37:55
2022-07-29 04:38:23,737 -     INFO - [TRAIN] epoch=345/604, iter=40000/70000, loss=0.278677, lr=0.000013 | ETA 06:34:47
2022-07-29 04:38:23,936 -     INFO - Pop model from ./output/iter_15000
2022-07-29 04:38:24,648 -     INFO - Push model to checkpoint ./output/iter_40000
2022-07-29 04:39:09,118 -     INFO - [TRAIN] epoch=346/604, iter=40050/70000, loss=0.287074, lr=0.000013 | ETA 07:14:58
2022-07-29 04:39:51,427 -     INFO - [TRAIN] epoch=346/604, iter=40100/70000, loss=0.311201, lr=0.000013 | ETA 06:56:00
2022-07-29 04:40:35,292 -     INFO - [TRAIN] epoch=347/604, iter=40150/70000, loss=0.308334, lr=0.000013 | ETA 06:50:06
2022-07-29 04:41:17,891 -     INFO - [TRAIN] epoch=347/604, iter=40200/70000, loss=0.279277, lr=0.000013 | ETA 07:19:54
2022-07-29 04:42:00,257 -     INFO - [TRAIN] epoch=347/604, iter=40250/70000, loss=0.289838, lr=0.000013 | ETA 06:25:28
2022-07-29 04:42:44,206 -     INFO - [TRAIN] epoch=348/604, iter=40300/70000, loss=0.305664, lr=0.000013 | ETA 06:32:02
2022-07-29 04:43:26,674 -     INFO - [TRAIN] epoch=348/604, iter=40350/70000, loss=0.294373, lr=0.000013 | ETA 06:33:08
2022-07-29 04:44:11,829 -     INFO - [TRAIN] epoch=349/604, iter=40400/70000, loss=0.343377, lr=0.000013 | ETA 06:57:13
2022-07-29 04:44:54,742 -     INFO - [TRAIN] epoch=349/604, iter=40450/70000, loss=0.274098, lr=0.000013 | ETA 06:26:07
2022-07-29 04:45:38,517 -     INFO - [TRAIN] epoch=350/604, iter=40500/70000, loss=0.302139, lr=0.000013 | ETA 06:55:41
2022-07-29 04:46:20,774 -     INFO - [TRAIN] epoch=350/604, iter=40550/70000, loss=0.310649, lr=0.000013 | ETA 06:23:32
2022-07-29 04:47:03,018 -     INFO - [TRAIN] epoch=350/604, iter=40600/70000, loss=0.330405, lr=0.000013 | ETA 06:19:17
2022-07-29 04:47:48,451 -     INFO - [TRAIN] epoch=351/604, iter=40650/70000, loss=0.310718, lr=0.000013 | ETA 06:52:30
2022-07-29 04:48:31,678 -     INFO - [TRAIN] epoch=351/604, iter=40700/70000, loss=0.307123, lr=0.000013 | ETA 06:23:17
2022-07-29 04:49:17,160 -     INFO - [TRAIN] epoch=352/604, iter=40750/70000, loss=0.292779, lr=0.000013 | ETA 08:29:23
2022-07-29 04:49:58,549 -     INFO - [TRAIN] epoch=352/604, iter=40800/70000, loss=0.322876, lr=0.000013 | ETA 06:32:41
2022-07-29 04:50:43,011 -     INFO - [TRAIN] epoch=353/604, iter=40850/70000, loss=0.314013, lr=0.000013 | ETA 07:13:02
2022-07-29 04:51:26,702 -     INFO - [TRAIN] epoch=353/604, iter=40900/70000, loss=0.299187, lr=0.000013 | ETA 06:30:16
2022-07-29 04:52:10,772 -     INFO - [TRAIN] epoch=354/604, iter=40950/70000, loss=0.319395, lr=0.000013 | ETA 09:43:23
2022-07-29 04:52:53,485 -     INFO - [TRAIN] epoch=354/604, iter=41000/70000, loss=0.266921, lr=0.000013 | ETA 06:14:21
2022-07-29 04:53:36,100 -     INFO - [TRAIN] epoch=354/604, iter=41050/70000, loss=0.280855, lr=0.000013 | ETA 07:11:17
2022-07-29 04:54:20,302 -     INFO - [TRAIN] epoch=355/604, iter=41100/70000, loss=0.306687, lr=0.000013 | ETA 06:56:19
2022-07-29 04:55:02,998 -     INFO - [TRAIN] epoch=355/604, iter=41150/70000, loss=0.285267, lr=0.000013 | ETA 07:02:09
2022-07-29 04:55:46,585 -     INFO - [TRAIN] epoch=356/604, iter=41200/70000, loss=0.274200, lr=0.000013 | ETA 06:26:09
2022-07-29 04:56:30,236 -     INFO - [TRAIN] epoch=356/604, iter=41250/70000, loss=0.295164, lr=0.000013 | ETA 08:48:31
2022-07-29 04:57:14,760 -     INFO - [TRAIN] epoch=357/604, iter=41300/70000, loss=0.269373, lr=0.000013 | ETA 08:05:09
2022-07-29 04:57:59,112 -     INFO - [TRAIN] epoch=357/604, iter=41350/70000, loss=0.300939, lr=0.000013 | ETA 06:51:19
2022-07-29 04:58:42,667 -     INFO - [TRAIN] epoch=357/604, iter=41400/70000, loss=0.290375, lr=0.000013 | ETA 06:43:21
2022-07-29 04:59:27,638 -     INFO - [TRAIN] epoch=358/604, iter=41450/70000, loss=0.281096, lr=0.000013 | ETA 07:52:36
2022-07-29 05:00:09,393 -     INFO - [TRAIN] epoch=358/604, iter=41500/70000, loss=0.252966, lr=0.000013 | ETA 06:11:36
2022-07-29 05:00:53,371 -     INFO - [TRAIN] epoch=359/604, iter=41550/70000, loss=0.278430, lr=0.000013 | ETA 06:25:41
2022-07-29 05:01:36,858 -     INFO - [TRAIN] epoch=359/604, iter=41600/70000, loss=0.258641, lr=0.000013 | ETA 06:56:56
2022-07-29 05:02:21,196 -     INFO - [TRAIN] epoch=360/604, iter=41650/70000, loss=0.304410, lr=0.000013 | ETA 08:47:27
2022-07-29 05:03:04,924 -     INFO - [TRAIN] epoch=360/604, iter=41700/70000, loss=0.245257, lr=0.000013 | ETA 06:22:09
2022-07-29 05:03:46,782 -     INFO - [TRAIN] epoch=360/604, iter=41750/70000, loss=0.307617, lr=0.000013 | ETA 07:28:00
2022-07-29 05:04:31,086 -     INFO - [TRAIN] epoch=361/604, iter=41800/70000, loss=0.286052, lr=0.000013 | ETA 06:25:50
2022-07-29 05:05:14,939 -     INFO - [TRAIN] epoch=361/604, iter=41850/70000, loss=0.276988, lr=0.000013 | ETA 07:20:17
2022-07-29 05:05:58,998 -     INFO - [TRAIN] epoch=362/604, iter=41900/70000, loss=0.306298, lr=0.000013 | ETA 06:05:27
2022-07-29 05:06:40,422 -     INFO - [TRAIN] epoch=362/604, iter=41950/70000, loss=0.288476, lr=0.000013 | ETA 06:18:44
2022-07-29 05:07:24,343 -     INFO - [TRAIN] epoch=363/604, iter=42000/70000, loss=0.264968, lr=0.000013 | ETA 06:37:55
2022-07-29 05:08:07,114 -     INFO - [TRAIN] epoch=363/604, iter=42050/70000, loss=0.329259, lr=0.000013 | ETA 06:09:37
2022-07-29 05:08:49,697 -     INFO - [TRAIN] epoch=363/604, iter=42100/70000, loss=0.289286, lr=0.000013 | ETA 06:19:52
2022-07-29 05:09:32,728 -     INFO - [TRAIN] epoch=364/604, iter=42150/70000, loss=0.307729, lr=0.000013 | ETA 06:03:20
2022-07-29 05:10:12,222 -     INFO - [TRAIN] epoch=364/604, iter=42200/70000, loss=0.306170, lr=0.000013 | ETA 06:08:54
2022-07-29 05:10:53,391 -     INFO - [TRAIN] epoch=365/604, iter=42250/70000, loss=0.279858, lr=0.000013 | ETA 05:58:46
2022-07-29 05:11:32,298 -     INFO - [TRAIN] epoch=365/604, iter=42300/70000, loss=0.269463, lr=0.000013 | ETA 05:59:46
2022-07-29 05:12:13,289 -     INFO - [TRAIN] epoch=366/604, iter=42350/70000, loss=0.277656, lr=0.000013 | ETA 06:05:10
2022-07-29 05:12:52,471 -     INFO - [TRAIN] epoch=366/604, iter=42400/70000, loss=0.282058, lr=0.000013 | ETA 05:52:59
2022-07-29 05:13:31,907 -     INFO - [TRAIN] epoch=366/604, iter=42450/70000, loss=0.295396, lr=0.000013 | ETA 05:54:04
2022-07-29 05:14:13,640 -     INFO - [TRAIN] epoch=367/604, iter=42500/70000, loss=0.300290, lr=0.000013 | ETA 06:18:34
2022-07-29 05:14:55,300 -     INFO - [TRAIN] epoch=367/604, iter=42550/70000, loss=0.259061, lr=0.000013 | ETA 06:01:01
2022-07-29 05:15:38,679 -     INFO - [TRAIN] epoch=368/604, iter=42600/70000, loss=0.297912, lr=0.000013 | ETA 05:58:52
2022-07-29 05:16:19,723 -     INFO - [TRAIN] epoch=368/604, iter=42650/70000, loss=0.340611, lr=0.000013 | ETA 06:20:08
2022-07-29 05:17:03,277 -     INFO - [TRAIN] epoch=369/604, iter=42700/70000, loss=0.251979, lr=0.000013 | ETA 06:33:26
2022-07-29 05:17:45,093 -     INFO - [TRAIN] epoch=369/604, iter=42750/70000, loss=0.300308, lr=0.000013 | ETA 06:23:54
2022-07-29 05:18:26,608 -     INFO - [TRAIN] epoch=369/604, iter=42800/70000, loss=0.293559, lr=0.000013 | ETA 05:56:24
2022-07-29 05:19:09,984 -     INFO - [TRAIN] epoch=370/604, iter=42850/70000, loss=0.285634, lr=0.000013 | ETA 06:01:11
2022-07-29 05:19:51,391 -     INFO - [TRAIN] epoch=370/604, iter=42900/70000, loss=0.282383, lr=0.000013 | ETA 05:57:10
2022-07-29 05:20:34,450 -     INFO - [TRAIN] epoch=371/604, iter=42950/70000, loss=0.272356, lr=0.000013 | ETA 06:01:07
2022-07-29 05:21:15,823 -     INFO - [TRAIN] epoch=371/604, iter=43000/70000, loss=0.279521, lr=0.000013 | ETA 06:02:46
2022-07-29 05:21:59,524 -     INFO - [TRAIN] epoch=372/604, iter=43050/70000, loss=0.291874, lr=0.000013 | ETA 06:12:07
2022-07-29 05:22:42,412 -     INFO - [TRAIN] epoch=372/604, iter=43100/70000, loss=0.264249, lr=0.000013 | ETA 06:40:17
2022-07-29 05:23:24,753 -     INFO - [TRAIN] epoch=372/604, iter=43150/70000, loss=0.249243, lr=0.000013 | ETA 05:55:12
2022-07-29 05:24:09,377 -     INFO - [TRAIN] epoch=373/604, iter=43200/70000, loss=0.303805, lr=0.000013 | ETA 08:18:46
2022-07-29 05:24:50,926 -     INFO - [TRAIN] epoch=373/604, iter=43250/70000, loss=0.302010, lr=0.000013 | ETA 06:34:00
2022-07-29 05:25:33,895 -     INFO - [TRAIN] epoch=374/604, iter=43300/70000, loss=0.284005, lr=0.000013 | ETA 06:35:50
2022-07-29 05:26:17,770 -     INFO - [TRAIN] epoch=374/604, iter=43350/70000, loss=0.287659, lr=0.000013 | ETA 06:01:31
2022-07-29 05:27:02,928 -     INFO - [TRAIN] epoch=375/604, iter=43400/70000, loss=0.276272, lr=0.000013 | ETA 05:54:48
2022-07-29 05:27:44,588 -     INFO - [TRAIN] epoch=375/604, iter=43450/70000, loss=0.288608, lr=0.000013 | ETA 06:12:26
2022-07-29 05:28:27,254 -     INFO - [TRAIN] epoch=375/604, iter=43500/70000, loss=0.259188, lr=0.000013 | ETA 05:45:33
2022-07-29 05:29:12,636 -     INFO - [TRAIN] epoch=376/604, iter=43550/70000, loss=0.242275, lr=0.000013 | ETA 06:16:17
2022-07-29 05:29:54,415 -     INFO - [TRAIN] epoch=376/604, iter=43600/70000, loss=0.276805, lr=0.000013 | ETA 05:39:19
2022-07-29 05:30:37,506 -     INFO - [TRAIN] epoch=377/604, iter=43650/70000, loss=0.287803, lr=0.000013 | ETA 05:55:32
2022-07-29 05:31:19,615 -     INFO - [TRAIN] epoch=377/604, iter=43700/70000, loss=0.262028, lr=0.000013 | ETA 05:38:14
2022-07-29 05:32:03,498 -     INFO - [TRAIN] epoch=378/604, iter=43750/70000, loss=0.272255, lr=0.000013 | ETA 06:47:54
2022-07-29 05:32:45,946 -     INFO - [TRAIN] epoch=378/604, iter=43800/70000, loss=0.305182, lr=0.000013 | ETA 06:08:53
2022-07-29 05:33:29,003 -     INFO - [TRAIN] epoch=379/604, iter=43850/70000, loss=0.264985, lr=0.000013 | ETA 09:13:39
2022-07-29 05:34:12,275 -     INFO - [TRAIN] epoch=379/604, iter=43900/70000, loss=0.285686, lr=0.000013 | ETA 06:14:36
2022-07-29 05:34:55,645 -     INFO - [TRAIN] epoch=379/604, iter=43950/70000, loss=0.270335, lr=0.000013 | ETA 07:35:15
2022-07-29 05:35:38,958 -     INFO - [TRAIN] epoch=380/604, iter=44000/70000, loss=0.268820, lr=0.000013 | ETA 05:59:32
2022-07-29 05:36:20,615 -     INFO - [TRAIN] epoch=380/604, iter=44050/70000, loss=0.302319, lr=0.000013 | ETA 05:49:31
2022-07-29 05:37:04,095 -     INFO - [TRAIN] epoch=381/604, iter=44100/70000, loss=0.303034, lr=0.000013 | ETA 06:11:11
2022-07-29 05:37:46,794 -     INFO - [TRAIN] epoch=381/604, iter=44150/70000, loss=0.292824, lr=0.000013 | ETA 05:38:19
2022-07-29 05:38:30,648 -     INFO - [TRAIN] epoch=382/604, iter=44200/70000, loss=0.263213, lr=0.000013 | ETA 07:44:21
2022-07-29 05:39:12,225 -     INFO - [TRAIN] epoch=382/604, iter=44250/70000, loss=0.259718, lr=0.000013 | ETA 05:42:42
2022-07-29 05:39:54,121 -     INFO - [TRAIN] epoch=382/604, iter=44300/70000, loss=0.267075, lr=0.000013 | ETA 05:56:47
2022-07-29 05:40:38,263 -     INFO - [TRAIN] epoch=383/604, iter=44350/70000, loss=0.282661, lr=0.000013 | ETA 05:37:07
2022-07-29 05:41:20,794 -     INFO - [TRAIN] epoch=383/604, iter=44400/70000, loss=0.306898, lr=0.000013 | ETA 05:42:41
2022-07-29 05:42:06,469 -     INFO - [TRAIN] epoch=384/604, iter=44450/70000, loss=0.266256, lr=0.000013 | ETA 06:21:36
2022-07-29 05:42:49,732 -     INFO - [TRAIN] epoch=384/604, iter=44500/70000, loss=0.264349, lr=0.000013 | ETA 05:31:54
2022-07-29 05:43:34,330 -     INFO - [TRAIN] epoch=385/604, iter=44550/70000, loss=0.263704, lr=0.000013 | ETA 05:45:51
2022-07-29 05:44:17,182 -     INFO - [TRAIN] epoch=385/604, iter=44600/70000, loss=0.271627, lr=0.000013 | ETA 05:49:49
2022-07-29 05:44:59,801 -     INFO - [TRAIN] epoch=385/604, iter=44650/70000, loss=0.243063, lr=0.000013 | ETA 05:40:33
2022-07-29 05:45:44,714 -     INFO - [TRAIN] epoch=386/604, iter=44700/70000, loss=0.277936, lr=0.000013 | ETA 05:35:38
2022-07-29 05:46:27,064 -     INFO - [TRAIN] epoch=386/604, iter=44750/70000, loss=0.299692, lr=0.000013 | ETA 05:25:50
2022-07-29 05:47:12,181 -     INFO - [TRAIN] epoch=387/604, iter=44800/70000, loss=0.288552, lr=0.000013 | ETA 08:08:02
2022-07-29 05:47:55,350 -     INFO - [TRAIN] epoch=387/604, iter=44850/70000, loss=0.301234, lr=0.000013 | ETA 07:01:37
2022-07-29 05:48:38,683 -     INFO - [TRAIN] epoch=388/604, iter=44900/70000, loss=0.250507, lr=0.000013 | ETA 06:04:55
2022-07-29 05:49:20,968 -     INFO - [TRAIN] epoch=388/604, iter=44950/70000, loss=0.270699, lr=0.000013 | ETA 05:38:23
2022-07-29 05:50:03,933 -     INFO - [TRAIN] epoch=388/604, iter=45000/70000, loss=0.281173, lr=0.000013 | ETA 05:40:27
2022-07-29 05:50:04,141 -     INFO - Pop model from ./output/iter_20000
2022-07-29 05:50:04,850 -     INFO - Push model to checkpoint ./output/iter_45000
2022-07-29 05:50:49,731 -     INFO - [TRAIN] epoch=389/604, iter=45050/70000, loss=0.281057, lr=0.000013 | ETA 05:53:59
2022-07-29 05:51:32,060 -     INFO - [TRAIN] epoch=389/604, iter=45100/70000, loss=0.301255, lr=0.000013 | ETA 05:45:25
2022-07-29 05:52:16,026 -     INFO - [TRAIN] epoch=390/604, iter=45150/70000, loss=0.260186, lr=0.000013 | ETA 05:34:34
2022-07-29 05:52:59,471 -     INFO - [TRAIN] epoch=390/604, iter=45200/70000, loss=0.263180, lr=0.000013 | ETA 05:47:14
2022-07-29 05:53:44,524 -     INFO - [TRAIN] epoch=391/604, iter=45250/70000, loss=0.281901, lr=0.000013 | ETA 05:50:33
2022-07-29 05:54:27,660 -     INFO - [TRAIN] epoch=391/604, iter=45300/70000, loss=0.272162, lr=0.000013 | ETA 05:32:28
2022-07-29 05:55:11,439 -     INFO - [TRAIN] epoch=391/604, iter=45350/70000, loss=0.286513, lr=0.000013 | ETA 05:33:37
2022-07-29 05:55:56,980 -     INFO - [TRAIN] epoch=392/604, iter=45400/70000, loss=0.287924, lr=0.000013 | ETA 05:42:13
2022-07-29 05:56:39,487 -     INFO - [TRAIN] epoch=392/604, iter=45450/70000, loss=0.276640, lr=0.000013 | ETA 05:54:55
2022-07-29 05:57:24,113 -     INFO - [TRAIN] epoch=393/604, iter=45500/70000, loss=0.300179, lr=0.000013 | ETA 06:38:55
2022-07-29 05:58:07,213 -     INFO - [TRAIN] epoch=393/604, iter=45550/70000, loss=0.262107, lr=0.000013 | ETA 05:23:32
2022-07-29 05:58:51,929 -     INFO - [TRAIN] epoch=394/604, iter=45600/70000, loss=0.301633, lr=0.000013 | ETA 05:54:45
2022-07-29 05:59:34,964 -     INFO - [TRAIN] epoch=394/604, iter=45650/70000, loss=0.236655, lr=0.000013 | ETA 07:24:50
2022-07-29 06:00:15,962 -     INFO - [TRAIN] epoch=394/604, iter=45700/70000, loss=0.258658, lr=0.000013 | ETA 05:12:14
2022-07-29 06:01:00,751 -     INFO - [TRAIN] epoch=395/604, iter=45750/70000, loss=0.286103, lr=0.000013 | ETA 05:32:59
2022-07-29 06:01:44,142 -     INFO - [TRAIN] epoch=395/604, iter=45800/70000, loss=0.306146, lr=0.000013 | ETA 05:33:02
2022-07-29 06:02:29,590 -     INFO - [TRAIN] epoch=396/604, iter=45850/70000, loss=0.245965, lr=0.000013 | ETA 07:20:43
2022-07-29 06:03:12,625 -     INFO - [TRAIN] epoch=396/604, iter=45900/70000, loss=0.270702, lr=0.000013 | ETA 05:46:20
2022-07-29 06:03:56,599 -     INFO - [TRAIN] epoch=397/604, iter=45950/70000, loss=0.244056, lr=0.000013 | ETA 05:23:54
2022-07-29 06:04:39,885 -     INFO - [TRAIN] epoch=397/604, iter=46000/70000, loss=0.283327, lr=0.000013 | ETA 05:47:49
2022-07-29 06:05:23,203 -     INFO - [TRAIN] epoch=397/604, iter=46050/70000, loss=0.306640, lr=0.000013 | ETA 05:35:32
2022-07-29 06:06:08,608 -     INFO - [TRAIN] epoch=398/604, iter=46100/70000, loss=0.236282, lr=0.000013 | ETA 05:51:52
2022-07-29 06:06:51,408 -     INFO - [TRAIN] epoch=398/604, iter=46150/70000, loss=0.294242, lr=0.000013 | ETA 05:32:03
2022-07-29 06:07:35,474 -     INFO - [TRAIN] epoch=399/604, iter=46200/70000, loss=0.274378, lr=0.000013 | ETA 05:55:42
2022-07-29 06:08:19,699 -     INFO - [TRAIN] epoch=399/604, iter=46250/70000, loss=0.265012, lr=0.000013 | ETA 05:24:19
2022-07-29 06:09:03,682 -     INFO - [TRAIN] epoch=400/604, iter=46300/70000, loss=0.292173, lr=0.000013 | ETA 05:12:27
2022-07-29 06:09:48,264 -     INFO - [TRAIN] epoch=400/604, iter=46350/70000, loss=0.272337, lr=0.000013 | ETA 05:56:33
2022-07-29 06:10:30,088 -     INFO - [TRAIN] epoch=400/604, iter=46400/70000, loss=0.301839, lr=0.000013 | ETA 05:04:29
2022-07-29 06:11:14,511 -     INFO - [TRAIN] epoch=401/604, iter=46450/70000, loss=0.257714, lr=0.000013 | ETA 05:50:39
2022-07-29 06:11:57,893 -     INFO - [TRAIN] epoch=401/604, iter=46500/70000, loss=0.262044, lr=0.000013 | ETA 06:09:44
2022-07-29 06:12:42,740 -     INFO - [TRAIN] epoch=402/604, iter=46550/70000, loss=0.260354, lr=0.000013 | ETA 05:16:34
2022-07-29 06:13:26,471 -     INFO - [TRAIN] epoch=402/604, iter=46600/70000, loss=0.304412, lr=0.000013 | ETA 05:09:53
2022-07-29 06:14:10,282 -     INFO - [TRAIN] epoch=403/604, iter=46650/70000, loss=0.249588, lr=0.000013 | ETA 06:09:14
2022-07-29 06:14:52,773 -     INFO - [TRAIN] epoch=403/604, iter=46700/70000, loss=0.261172, lr=0.000013 | ETA 05:05:06
2022-07-29 06:15:36,516 -     INFO - [TRAIN] epoch=404/604, iter=46750/70000, loss=0.279321, lr=0.000013 | ETA 08:23:31
2022-07-29 06:16:21,043 -     INFO - [TRAIN] epoch=404/604, iter=46800/70000, loss=0.233734, lr=0.000013 | ETA 05:14:02
2022-07-29 06:17:04,695 -     INFO - [TRAIN] epoch=404/604, iter=46850/70000, loss=0.281841, lr=0.000013 | ETA 05:21:12
2022-07-29 06:17:49,784 -     INFO - [TRAIN] epoch=405/604, iter=46900/70000, loss=0.246825, lr=0.000013 | ETA 05:19:15
2022-07-29 06:18:31,744 -     INFO - [TRAIN] epoch=405/604, iter=46950/70000, loss=0.249386, lr=0.000013 | ETA 05:08:04
2022-07-29 06:19:15,397 -     INFO - [TRAIN] epoch=406/604, iter=47000/70000, loss=0.285383, lr=0.000013 | ETA 05:19:01
2022-07-29 06:19:59,351 -     INFO - [TRAIN] epoch=406/604, iter=47050/70000, loss=0.250550, lr=0.000013 | ETA 05:03:02
2022-07-29 06:20:44,720 -     INFO - [TRAIN] epoch=407/604, iter=47100/70000, loss=0.304316, lr=0.000013 | ETA 06:07:38
2022-07-29 06:21:28,522 -     INFO - [TRAIN] epoch=407/604, iter=47150/70000, loss=0.286496, lr=0.000013 | ETA 05:16:31
2022-07-29 06:22:10,746 -     INFO - [TRAIN] epoch=407/604, iter=47200/70000, loss=0.252296, lr=0.000013 | ETA 05:19:08
2022-07-29 06:22:54,669 -     INFO - [TRAIN] epoch=408/604, iter=47250/70000, loss=0.258230, lr=0.000013 | ETA 05:11:53
2022-07-29 06:23:38,091 -     INFO - [TRAIN] epoch=408/604, iter=47300/70000, loss=0.236134, lr=0.000013 | ETA 05:31:37
2022-07-29 06:24:22,085 -     INFO - [TRAIN] epoch=409/604, iter=47350/70000, loss=0.245537, lr=0.000013 | ETA 05:31:01
2022-07-29 06:25:05,200 -     INFO - [TRAIN] epoch=409/604, iter=47400/70000, loss=0.297068, lr=0.000013 | ETA 06:18:27
2022-07-29 06:25:49,906 -     INFO - [TRAIN] epoch=410/604, iter=47450/70000, loss=0.262716, lr=0.000013 | ETA 05:19:15
2022-07-29 06:26:32,497 -     INFO - [TRAIN] epoch=410/604, iter=47500/70000, loss=0.246704, lr=0.000013 | ETA 04:57:29
2022-07-29 06:27:16,002 -     INFO - [TRAIN] epoch=410/604, iter=47550/70000, loss=0.268871, lr=0.000013 | ETA 05:41:44
2022-07-29 06:28:01,479 -     INFO - [TRAIN] epoch=411/604, iter=47600/70000, loss=0.270665, lr=0.000013 | ETA 04:48:14
2022-07-29 06:28:45,126 -     INFO - [TRAIN] epoch=411/604, iter=47650/70000, loss=0.248020, lr=0.000013 | ETA 04:59:09
2022-07-29 06:29:29,046 -     INFO - [TRAIN] epoch=412/604, iter=47700/70000, loss=0.245532, lr=0.000013 | ETA 04:59:05
2022-07-29 06:30:12,056 -     INFO - [TRAIN] epoch=412/604, iter=47750/70000, loss=0.274590, lr=0.000013 | ETA 05:11:51
2022-07-29 06:30:56,045 -     INFO - [TRAIN] epoch=413/604, iter=47800/70000, loss=0.275809, lr=0.000013 | ETA 05:04:03
2022-07-29 06:31:39,335 -     INFO - [TRAIN] epoch=413/604, iter=47850/70000, loss=0.223471, lr=0.000013 | ETA 05:41:24
2022-07-29 06:32:21,339 -     INFO - [TRAIN] epoch=413/604, iter=47900/70000, loss=0.314956, lr=0.000013 | ETA 05:18:14
2022-07-29 06:33:05,836 -     INFO - [TRAIN] epoch=414/604, iter=47950/70000, loss=0.265904, lr=0.000013 | ETA 05:15:17
2022-07-29 06:33:48,322 -     INFO - [TRAIN] epoch=414/604, iter=48000/70000, loss=0.279543, lr=0.000013 | ETA 05:56:47
2022-07-29 06:34:32,812 -     INFO - [TRAIN] epoch=415/604, iter=48050/70000, loss=0.316310, lr=0.000013 | ETA 05:35:44
2022-07-29 06:35:15,168 -     INFO - [TRAIN] epoch=415/604, iter=48100/70000, loss=0.245647, lr=0.000013 | ETA 05:11:07
2022-07-29 06:35:59,058 -     INFO - [TRAIN] epoch=416/604, iter=48150/70000, loss=0.267334, lr=0.000013 | ETA 05:11:14
2022-07-29 06:36:42,709 -     INFO - [TRAIN] epoch=416/604, iter=48200/70000, loss=0.295403, lr=0.000013 | ETA 05:08:48
2022-07-29 06:37:26,042 -     INFO - [TRAIN] epoch=416/604, iter=48250/70000, loss=0.223549, lr=0.000013 | ETA 06:04:22
2022-07-29 06:38:11,234 -     INFO - [TRAIN] epoch=417/604, iter=48300/70000, loss=0.249312, lr=0.000013 | ETA 05:29:28
2022-07-29 06:38:54,207 -     INFO - [TRAIN] epoch=417/604, iter=48350/70000, loss=0.272948, lr=0.000013 | ETA 04:44:17
2022-07-29 06:39:38,470 -     INFO - [TRAIN] epoch=418/604, iter=48400/70000, loss=0.254366, lr=0.000013 | ETA 04:51:14
2022-07-29 06:40:22,898 -     INFO - [TRAIN] epoch=418/604, iter=48450/70000, loss=0.278482, lr=0.000013 | ETA 05:07:36
2022-07-29 06:41:06,778 -     INFO - [TRAIN] epoch=419/604, iter=48500/70000, loss=0.271723, lr=0.000013 | ETA 04:44:30
2022-07-29 06:41:49,638 -     INFO - [TRAIN] epoch=419/604, iter=48550/70000, loss=0.250366, lr=0.000013 | ETA 05:39:12
2022-07-29 06:42:31,648 -     INFO - [TRAIN] epoch=419/604, iter=48600/70000, loss=0.285888, lr=0.000013 | ETA 04:53:32
2022-07-29 06:43:15,230 -     INFO - [TRAIN] epoch=420/604, iter=48650/70000, loss=0.267330, lr=0.000013 | ETA 04:50:44
2022-07-29 06:43:56,927 -     INFO - [TRAIN] epoch=420/604, iter=48700/70000, loss=0.269748, lr=0.000013 | ETA 05:09:03
2022-07-29 06:44:42,865 -     INFO - [TRAIN] epoch=421/604, iter=48750/70000, loss=0.300814, lr=0.000013 | ETA 05:04:55
2022-07-29 06:45:25,353 -     INFO - [TRAIN] epoch=421/604, iter=48800/70000, loss=0.284051, lr=0.000013 | ETA 04:37:22
2022-07-29 06:46:09,991 -     INFO - [TRAIN] epoch=422/604, iter=48850/70000, loss=0.259157, lr=0.000013 | ETA 06:23:11
2022-07-29 06:46:52,236 -     INFO - [TRAIN] epoch=422/604, iter=48900/70000, loss=0.289930, lr=0.000013 | ETA 04:52:25
2022-07-29 06:47:35,300 -     INFO - [TRAIN] epoch=422/604, iter=48950/70000, loss=0.270553, lr=0.000013 | ETA 04:35:05
2022-07-29 06:48:19,235 -     INFO - [TRAIN] epoch=423/604, iter=49000/70000, loss=0.272787, lr=0.000013 | ETA 05:34:43
2022-07-29 06:49:01,363 -     INFO - [TRAIN] epoch=423/604, iter=49050/70000, loss=0.265914, lr=0.000013 | ETA 04:34:23
2022-07-29 06:49:47,906 -     INFO - [TRAIN] epoch=424/604, iter=49100/70000, loss=0.303770, lr=0.000013 | ETA 05:42:09
2022-07-29 06:50:31,469 -     INFO - [TRAIN] epoch=424/604, iter=49150/70000, loss=0.230780, lr=0.000013 | ETA 05:15:41
2022-07-29 06:51:17,496 -     INFO - [TRAIN] epoch=425/604, iter=49200/70000, loss=0.279324, lr=0.000013 | ETA 05:30:53
2022-07-29 06:52:00,061 -     INFO - [TRAIN] epoch=425/604, iter=49250/70000, loss=0.248799, lr=0.000013 | ETA 04:49:33
2022-07-29 06:52:42,666 -     INFO - [TRAIN] epoch=425/604, iter=49300/70000, loss=0.255120, lr=0.000013 | ETA 04:26:58
2022-07-29 06:53:26,823 -     INFO - [TRAIN] epoch=426/604, iter=49350/70000, loss=0.286783, lr=0.000013 | ETA 04:43:35
2022-07-29 06:54:10,307 -     INFO - [TRAIN] epoch=426/604, iter=49400/70000, loss=0.284262, lr=0.000013 | ETA 04:34:13
2022-07-29 06:54:54,176 -     INFO - [TRAIN] epoch=427/604, iter=49450/70000, loss=0.266623, lr=0.000013 | ETA 04:33:36
2022-07-29 06:55:36,601 -     INFO - [TRAIN] epoch=427/604, iter=49500/70000, loss=0.275222, lr=0.000013 | ETA 04:25:55
2022-07-29 06:56:20,641 -     INFO - [TRAIN] epoch=428/604, iter=49550/70000, loss=0.262787, lr=0.000013 | ETA 05:31:00
2022-07-29 06:57:04,456 -     INFO - [TRAIN] epoch=428/604, iter=49600/70000, loss=0.278568, lr=0.000013 | ETA 05:12:00
2022-07-29 06:57:48,884 -     INFO - [TRAIN] epoch=429/604, iter=49650/70000, loss=0.258374, lr=0.000013 | ETA 06:57:35
2022-07-29 06:58:33,157 -     INFO - [TRAIN] epoch=429/604, iter=49700/70000, loss=0.268100, lr=0.000013 | ETA 04:26:20
2022-07-29 06:59:16,868 -     INFO - [TRAIN] epoch=429/604, iter=49750/70000, loss=0.259492, lr=0.000013 | ETA 05:41:46
2022-07-29 07:00:01,013 -     INFO - [TRAIN] epoch=430/604, iter=49800/70000, loss=0.250652, lr=0.000013 | ETA 04:54:02
2022-07-29 07:00:43,813 -     INFO - [TRAIN] epoch=430/604, iter=49850/70000, loss=0.245269, lr=0.000013 | ETA 04:31:44
2022-07-29 07:01:24,494 -     INFO - [TRAIN] epoch=431/604, iter=49900/70000, loss=0.258676, lr=0.000013 | ETA 04:22:57
2022-07-29 07:02:04,186 -     INFO - [TRAIN] epoch=431/604, iter=49950/70000, loss=0.250494, lr=0.000013 | ETA 04:31:16
2022-07-29 07:02:45,400 -     INFO - [TRAIN] epoch=432/604, iter=50000/70000, loss=0.277737, lr=0.000013 | ETA 04:56:17
2022-07-29 07:02:45,605 -     INFO - Pop model from ./output/iter_25000
2022-07-29 07:02:46,333 -     INFO - Push model to checkpoint ./output/iter_50000
2022-07-29 07:03:25,548 -     INFO - [TRAIN] epoch=432/604, iter=50050/70000, loss=0.291756, lr=0.000013 | ETA 04:23:35
2022-07-29 07:04:05,636 -     INFO - [TRAIN] epoch=432/604, iter=50100/70000, loss=0.246919, lr=0.000013 | ETA 04:26:05
2022-07-29 07:04:47,281 -     INFO - [TRAIN] epoch=433/604, iter=50150/70000, loss=0.259416, lr=0.000013 | ETA 04:19:31
2022-07-29 07:05:27,364 -     INFO - [TRAIN] epoch=433/604, iter=50200/70000, loss=0.269143, lr=0.000013 | ETA 04:45:01
2022-07-29 07:06:09,632 -     INFO - [TRAIN] epoch=434/604, iter=50250/70000, loss=0.287565, lr=0.000013 | ETA 04:25:30
2022-07-29 07:06:52,270 -     INFO - [TRAIN] epoch=434/604, iter=50300/70000, loss=0.293088, lr=0.000013 | ETA 06:04:18
2022-07-29 07:07:34,573 -     INFO - [TRAIN] epoch=435/604, iter=50350/70000, loss=0.275431, lr=0.000013 | ETA 05:09:30
2022-07-29 07:08:17,917 -     INFO - [TRAIN] epoch=435/604, iter=50400/70000, loss=0.275564, lr=0.000013 | ETA 04:33:56
2022-07-29 07:09:00,428 -     INFO - [TRAIN] epoch=435/604, iter=50450/70000, loss=0.265446, lr=0.000013 | ETA 04:26:53
2022-07-29 07:09:42,500 -     INFO - [TRAIN] epoch=436/604, iter=50500/70000, loss=0.283403, lr=0.000013 | ETA 04:21:35
2022-07-29 07:10:25,642 -     INFO - [TRAIN] epoch=436/604, iter=50550/70000, loss=0.236064, lr=0.000013 | ETA 04:45:50
2022-07-29 07:11:09,123 -     INFO - [TRAIN] epoch=437/604, iter=50600/70000, loss=0.267312, lr=0.000013 | ETA 04:37:47
2022-07-29 07:11:51,059 -     INFO - [TRAIN] epoch=437/604, iter=50650/70000, loss=0.299019, lr=0.000013 | ETA 04:19:09
2022-07-29 07:12:33,924 -     INFO - [TRAIN] epoch=438/604, iter=50700/70000, loss=0.256124, lr=0.000013 | ETA 04:18:50
2022-07-29 07:13:15,713 -     INFO - [TRAIN] epoch=438/604, iter=50750/70000, loss=0.286042, lr=0.000013 | ETA 04:53:51
2022-07-29 07:13:57,122 -     INFO - [TRAIN] epoch=438/604, iter=50800/70000, loss=0.271011, lr=0.000013 | ETA 04:32:53
2022-07-29 07:14:40,230 -     INFO - [TRAIN] epoch=439/604, iter=50850/70000, loss=0.234380, lr=0.000013 | ETA 05:01:45
2022-07-29 07:15:22,098 -     INFO - [TRAIN] epoch=439/604, iter=50900/70000, loss=0.257052, lr=0.000013 | ETA 04:11:45
2022-07-29 07:16:05,123 -     INFO - [TRAIN] epoch=440/604, iter=50950/70000, loss=0.268341, lr=0.000013 | ETA 04:14:34
2022-07-29 07:16:48,511 -     INFO - [TRAIN] epoch=440/604, iter=51000/70000, loss=0.276133, lr=0.000013 | ETA 04:19:53
2022-07-29 07:17:32,799 -     INFO - [TRAIN] epoch=441/604, iter=51050/70000, loss=0.276332, lr=0.000013 | ETA 04:10:13
2022-07-29 07:18:14,422 -     INFO - [TRAIN] epoch=441/604, iter=51100/70000, loss=0.233699, lr=0.000013 | ETA 04:13:46
2022-07-29 07:18:57,110 -     INFO - [TRAIN] epoch=441/604, iter=51150/70000, loss=0.270065, lr=0.000013 | ETA 04:06:49
2022-07-29 07:19:40,954 -     INFO - [TRAIN] epoch=442/604, iter=51200/70000, loss=0.264489, lr=0.000013 | ETA 04:50:39
2022-07-29 07:20:24,081 -     INFO - [TRAIN] epoch=442/604, iter=51250/70000, loss=0.232719, lr=0.000013 | ETA 04:54:22
2022-07-29 07:21:07,671 -     INFO - [TRAIN] epoch=443/604, iter=51300/70000, loss=0.291992, lr=0.000013 | ETA 04:14:22
2022-07-29 07:21:48,994 -     INFO - [TRAIN] epoch=443/604, iter=51350/70000, loss=0.271967, lr=0.000013 | ETA 04:20:33
2022-07-29 07:22:32,939 -     INFO - [TRAIN] epoch=444/604, iter=51400/70000, loss=0.250337, lr=0.000013 | ETA 04:19:30
2022-07-29 07:23:15,816 -     INFO - [TRAIN] epoch=444/604, iter=51450/70000, loss=0.302910, lr=0.000013 | ETA 04:31:21
2022-07-29 07:23:57,296 -     INFO - [TRAIN] epoch=444/604, iter=51500/70000, loss=0.266283, lr=0.000013 | ETA 04:03:05
2022-07-29 07:24:41,298 -     INFO - [TRAIN] epoch=445/604, iter=51550/70000, loss=0.280336, lr=0.000013 | ETA 04:01:32
2022-07-29 07:25:24,659 -     INFO - [TRAIN] epoch=445/604, iter=51600/70000, loss=0.241718, lr=0.000013 | ETA 04:27:14
2022-07-29 07:26:09,177 -     INFO - [TRAIN] epoch=446/604, iter=51650/70000, loss=0.267274, lr=0.000013 | ETA 04:18:59
2022-07-29 07:26:51,964 -     INFO - [TRAIN] epoch=446/604, iter=51700/70000, loss=0.254792, lr=0.000013 | ETA 04:24:07
2022-07-29 07:27:36,202 -     INFO - [TRAIN] epoch=447/604, iter=51750/70000, loss=0.292202, lr=0.000013 | ETA 04:06:42
2022-07-29 07:28:19,531 -     INFO - [TRAIN] epoch=447/604, iter=51800/70000, loss=0.246773, lr=0.000013 | ETA 04:15:33
2022-07-29 07:29:01,269 -     INFO - [TRAIN] epoch=447/604, iter=51850/70000, loss=0.267228, lr=0.000013 | ETA 03:55:19
2022-07-29 07:29:46,154 -     INFO - [TRAIN] epoch=448/604, iter=51900/70000, loss=0.264677, lr=0.000013 | ETA 04:32:24
2022-07-29 07:30:27,941 -     INFO - [TRAIN] epoch=448/604, iter=51950/70000, loss=0.236620, lr=0.000013 | ETA 04:00:45
2022-07-29 07:31:12,889 -     INFO - [TRAIN] epoch=449/604, iter=52000/70000, loss=0.229261, lr=0.000013 | ETA 04:33:22
2022-07-29 07:31:56,011 -     INFO - [TRAIN] epoch=449/604, iter=52050/70000, loss=0.285165, lr=0.000013 | ETA 04:06:29
2022-07-29 07:32:40,878 -     INFO - [TRAIN] epoch=450/604, iter=52100/70000, loss=0.260131, lr=0.000013 | ETA 04:40:10
2022-07-29 07:33:23,395 -     INFO - [TRAIN] epoch=450/604, iter=52150/70000, loss=0.245181, lr=0.000013 | ETA 03:59:27
2022-07-29 07:34:06,106 -     INFO - [TRAIN] epoch=450/604, iter=52200/70000, loss=0.280624, lr=0.000013 | ETA 03:58:09
2022-07-29 07:34:49,865 -     INFO - [TRAIN] epoch=451/604, iter=52250/70000, loss=0.261383, lr=0.000013 | ETA 03:59:21
2022-07-29 07:35:32,248 -     INFO - [TRAIN] epoch=451/604, iter=52300/70000, loss=0.261989, lr=0.000013 | ETA 03:53:31
2022-07-29 07:36:17,895 -     INFO - [TRAIN] epoch=452/604, iter=52350/70000, loss=0.256046, lr=0.000013 | ETA 03:55:54
2022-07-29 07:37:00,116 -     INFO - [TRAIN] epoch=452/604, iter=52400/70000, loss=0.256443, lr=0.000013 | ETA 03:54:05
2022-07-29 07:37:45,104 -     INFO - [TRAIN] epoch=453/604, iter=52450/70000, loss=0.328015, lr=0.000013 | ETA 04:30:07
2022-07-29 07:38:27,229 -     INFO - [TRAIN] epoch=453/604, iter=52500/70000, loss=0.246934, lr=0.000013 | ETA 03:48:56
2022-07-29 07:39:10,077 -     INFO - [TRAIN] epoch=454/604, iter=52550/70000, loss=0.251332, lr=0.000013 | ETA 05:54:35
2022-07-29 07:39:54,144 -     INFO - [TRAIN] epoch=454/604, iter=52600/70000, loss=0.270648, lr=0.000013 | ETA 03:50:08
2022-07-29 07:40:38,624 -     INFO - [TRAIN] epoch=454/604, iter=52650/70000, loss=0.260232, lr=0.000013 | ETA 04:43:28
2022-07-29 07:41:22,901 -     INFO - [TRAIN] epoch=455/604, iter=52700/70000, loss=0.235289, lr=0.000013 | ETA 03:53:40
2022-07-29 07:42:04,637 -     INFO - [TRAIN] epoch=455/604, iter=52750/70000, loss=0.267989, lr=0.000013 | ETA 03:46:04
2022-07-29 07:42:49,551 -     INFO - [TRAIN] epoch=456/604, iter=52800/70000, loss=0.280913, lr=0.000013 | ETA 04:02:43
2022-07-29 07:43:34,489 -     INFO - [TRAIN] epoch=456/604, iter=52850/70000, loss=0.287837, lr=0.000013 | ETA 05:02:04
2022-07-29 07:44:19,855 -     INFO - [TRAIN] epoch=457/604, iter=52900/70000, loss=0.260544, lr=0.000013 | ETA 04:20:22
2022-07-29 07:45:02,696 -     INFO - [TRAIN] epoch=457/604, iter=52950/70000, loss=0.261453, lr=0.000013 | ETA 03:53:10
2022-07-29 07:45:45,284 -     INFO - [TRAIN] epoch=457/604, iter=53000/70000, loss=0.263746, lr=0.000013 | ETA 03:46:11
2022-07-29 07:46:29,599 -     INFO - [TRAIN] epoch=458/604, iter=53050/70000, loss=0.265322, lr=0.000013 | ETA 03:50:12
2022-07-29 07:47:13,294 -     INFO - [TRAIN] epoch=458/604, iter=53100/70000, loss=0.237184, lr=0.000013 | ETA 04:18:01
2022-07-29 07:47:57,320 -     INFO - [TRAIN] epoch=459/604, iter=53150/70000, loss=0.260307, lr=0.000013 | ETA 03:51:20
2022-07-29 07:48:40,898 -     INFO - [TRAIN] epoch=459/604, iter=53200/70000, loss=0.319257, lr=0.000013 | ETA 04:38:37
2022-07-29 07:49:26,214 -     INFO - [TRAIN] epoch=460/604, iter=53250/70000, loss=0.253174, lr=0.000013 | ETA 04:17:17
2022-07-29 07:50:09,061 -     INFO - [TRAIN] epoch=460/604, iter=53300/70000, loss=0.256614, lr=0.000013 | ETA 03:45:31
2022-07-29 07:50:51,906 -     INFO - [TRAIN] epoch=460/604, iter=53350/70000, loss=0.299152, lr=0.000013 | ETA 03:41:57
2022-07-29 07:51:36,469 -     INFO - [TRAIN] epoch=461/604, iter=53400/70000, loss=0.290201, lr=0.000013 | ETA 03:51:22
2022-07-29 07:52:21,273 -     INFO - [TRAIN] epoch=461/604, iter=53450/70000, loss=0.251285, lr=0.000013 | ETA 04:33:15
2022-07-29 07:53:06,720 -     INFO - [TRAIN] epoch=462/604, iter=53500/70000, loss=0.253093, lr=0.000013 | ETA 03:40:15
2022-07-29 07:53:50,075 -     INFO - [TRAIN] epoch=462/604, iter=53550/70000, loss=0.305771, lr=0.000013 | ETA 04:17:00
2022-07-29 07:54:34,284 -     INFO - [TRAIN] epoch=463/604, iter=53600/70000, loss=0.258225, lr=0.000013 | ETA 03:52:33
2022-07-29 07:55:16,412 -     INFO - [TRAIN] epoch=463/604, iter=53650/70000, loss=0.302593, lr=0.000013 | ETA 04:00:53
2022-07-29 07:55:58,858 -     INFO - [TRAIN] epoch=463/604, iter=53700/70000, loss=0.259434, lr=0.000013 | ETA 03:46:26
2022-07-29 07:56:42,825 -     INFO - [TRAIN] epoch=464/604, iter=53750/70000, loss=0.264892, lr=0.000013 | ETA 03:31:15
2022-07-29 07:57:25,470 -     INFO - [TRAIN] epoch=464/604, iter=53800/70000, loss=0.255888, lr=0.000013 | ETA 03:40:36
2022-07-29 07:58:08,951 -     INFO - [TRAIN] epoch=465/604, iter=53850/70000, loss=0.253589, lr=0.000013 | ETA 03:29:11
2022-07-29 07:58:52,606 -     INFO - [TRAIN] epoch=465/604, iter=53900/70000, loss=0.271778, lr=0.000013 | ETA 04:48:45
2022-07-29 07:59:37,183 -     INFO - [TRAIN] epoch=466/604, iter=53950/70000, loss=0.285425, lr=0.000013 | ETA 03:34:54
2022-07-29 08:00:20,618 -     INFO - [TRAIN] epoch=466/604, iter=54000/70000, loss=0.274257, lr=0.000013 | ETA 03:47:40
2022-07-29 08:01:04,145 -     INFO - [TRAIN] epoch=466/604, iter=54050/70000, loss=0.252013, lr=0.000013 | ETA 03:40:42
2022-07-29 08:01:48,246 -     INFO - [TRAIN] epoch=467/604, iter=54100/70000, loss=0.243846, lr=0.000013 | ETA 03:40:17
2022-07-29 08:02:30,870 -     INFO - [TRAIN] epoch=467/604, iter=54150/70000, loss=0.268139, lr=0.000013 | ETA 03:39:42
2022-07-29 08:03:16,001 -     INFO - [TRAIN] epoch=468/604, iter=54200/70000, loss=0.247345, lr=0.000013 | ETA 04:55:03
2022-07-29 08:03:58,835 -     INFO - [TRAIN] epoch=468/604, iter=54250/70000, loss=0.291866, lr=0.000013 | ETA 03:55:33
2022-07-29 08:04:42,535 -     INFO - [TRAIN] epoch=469/604, iter=54300/70000, loss=0.256405, lr=0.000013 | ETA 03:25:11
2022-07-29 08:05:25,969 -     INFO - [TRAIN] epoch=469/604, iter=54350/70000, loss=0.259816, lr=0.000013 | ETA 03:59:14
2022-07-29 08:06:08,663 -     INFO - [TRAIN] epoch=469/604, iter=54400/70000, loss=0.242121, lr=0.000013 | ETA 03:22:45
2022-07-29 08:06:52,906 -     INFO - [TRAIN] epoch=470/604, iter=54450/70000, loss=0.252676, lr=0.000013 | ETA 04:01:29
2022-07-29 08:07:36,383 -     INFO - [TRAIN] epoch=470/604, iter=54500/70000, loss=0.313619, lr=0.000013 | ETA 03:33:48
2022-07-29 08:08:20,276 -     INFO - [TRAIN] epoch=471/604, iter=54550/70000, loss=0.242683, lr=0.000013 | ETA 03:47:42
2022-07-29 08:09:02,933 -     INFO - [TRAIN] epoch=471/604, iter=54600/70000, loss=0.272036, lr=0.000013 | ETA 04:10:17
2022-07-29 08:09:47,640 -     INFO - [TRAIN] epoch=472/604, iter=54650/70000, loss=0.249292, lr=0.000013 | ETA 03:24:55
2022-07-29 08:10:30,411 -     INFO - [TRAIN] epoch=472/604, iter=54700/70000, loss=0.251475, lr=0.000013 | ETA 03:35:28
2022-07-29 08:11:14,071 -     INFO - [TRAIN] epoch=472/604, iter=54750/70000, loss=0.261679, lr=0.000013 | ETA 03:19:35
2022-07-29 08:11:59,567 -     INFO - [TRAIN] epoch=473/604, iter=54800/70000, loss=0.253744, lr=0.000013 | ETA 03:55:02
2022-07-29 08:12:42,765 -     INFO - [TRAIN] epoch=473/604, iter=54850/70000, loss=0.239376, lr=0.000013 | ETA 03:34:29
2022-07-29 08:13:28,002 -     INFO - [TRAIN] epoch=474/604, iter=54900/70000, loss=0.245422, lr=0.000013 | ETA 03:29:15
2022-07-29 08:14:11,286 -     INFO - [TRAIN] epoch=474/604, iter=54950/70000, loss=0.299059, lr=0.000013 | ETA 03:23:28
2022-07-29 08:14:55,595 -     INFO - [TRAIN] epoch=475/604, iter=55000/70000, loss=0.292768, lr=0.000001 | ETA 03:19:25
2022-07-29 08:14:55,807 -     INFO - Pop model from ./output/iter_30000
2022-07-29 08:14:56,539 -     INFO - Push model to checkpoint ./output/iter_55000
2022-07-29 08:15:38,490 -     INFO - [TRAIN] epoch=475/604, iter=55050/70000, loss=0.307953, lr=0.000001 | ETA 03:18:42
2022-07-29 08:16:21,764 -     INFO - [TRAIN] epoch=475/604, iter=55100/70000, loss=0.246685, lr=0.000001 | ETA 03:27:08
2022-07-29 08:17:07,190 -     INFO - [TRAIN] epoch=476/604, iter=55150/70000, loss=0.291394, lr=0.000001 | ETA 04:22:07
2022-07-29 08:17:50,421 -     INFO - [TRAIN] epoch=476/604, iter=55200/70000, loss=0.250163, lr=0.000001 | ETA 03:15:14
2022-07-29 08:18:36,050 -     INFO - [TRAIN] epoch=477/604, iter=55250/70000, loss=0.252635, lr=0.000001 | ETA 03:16:26
2022-07-29 08:19:20,140 -     INFO - [TRAIN] epoch=477/604, iter=55300/70000, loss=0.273061, lr=0.000001 | ETA 03:12:29
2022-07-29 08:20:05,398 -     INFO - [TRAIN] epoch=478/604, iter=55350/70000, loss=0.252212, lr=0.000001 | ETA 03:14:51
2022-07-29 08:20:47,710 -     INFO - [TRAIN] epoch=478/604, iter=55400/70000, loss=0.250979, lr=0.000001 | ETA 03:26:16
2022-07-29 08:21:31,311 -     INFO - [TRAIN] epoch=479/604, iter=55450/70000, loss=0.263881, lr=0.000001 | ETA 04:57:48
2022-07-29 08:22:15,741 -     INFO - [TRAIN] epoch=479/604, iter=55500/70000, loss=0.244713, lr=0.000001 | ETA 03:18:16
2022-07-29 08:22:58,427 -     INFO - [TRAIN] epoch=479/604, iter=55550/70000, loss=0.262874, lr=0.000001 | ETA 03:23:24
2022-07-29 08:23:44,554 -     INFO - [TRAIN] epoch=480/604, iter=55600/70000, loss=0.284291, lr=0.000001 | ETA 03:11:04
2022-07-29 08:24:28,160 -     INFO - [TRAIN] epoch=480/604, iter=55650/70000, loss=0.274749, lr=0.000001 | ETA 03:10:26
2022-07-29 08:25:12,961 -     INFO - [TRAIN] epoch=481/604, iter=55700/70000, loss=0.254730, lr=0.000001 | ETA 03:46:36
2022-07-29 08:25:56,853 -     INFO - [TRAIN] epoch=481/604, iter=55750/70000, loss=0.264538, lr=0.000001 | ETA 03:15:58
2022-07-29 08:26:41,891 -     INFO - [TRAIN] epoch=482/604, iter=55800/70000, loss=0.250786, lr=0.000001 | ETA 03:33:28
2022-07-29 08:27:25,357 -     INFO - [TRAIN] epoch=482/604, iter=55850/70000, loss=0.269765, lr=0.000001 | ETA 04:01:16
2022-07-29 08:28:09,025 -     INFO - [TRAIN] epoch=482/604, iter=55900/70000, loss=0.254237, lr=0.000001 | ETA 03:40:20
2022-07-29 08:28:53,710 -     INFO - [TRAIN] epoch=483/604, iter=55950/70000, loss=0.261951, lr=0.000001 | ETA 03:13:45
2022-07-29 08:29:37,627 -     INFO - [TRAIN] epoch=483/604, iter=56000/70000, loss=0.226739, lr=0.000001 | ETA 03:13:37
2022-07-29 08:30:23,247 -     INFO - [TRAIN] epoch=484/604, iter=56050/70000, loss=0.244809, lr=0.000001 | ETA 03:16:05
2022-07-29 08:31:06,583 -     INFO - [TRAIN] epoch=484/604, iter=56100/70000, loss=0.278252, lr=0.000001 | ETA 03:04:26
2022-07-29 08:31:51,702 -     INFO - [TRAIN] epoch=485/604, iter=56150/70000, loss=0.267825, lr=0.000001 | ETA 03:20:09
2022-07-29 08:32:35,718 -     INFO - [TRAIN] epoch=485/604, iter=56200/70000, loss=0.265269, lr=0.000001 | ETA 03:07:39
2022-07-29 08:33:19,702 -     INFO - [TRAIN] epoch=485/604, iter=56250/70000, loss=0.276300, lr=0.000001 | ETA 03:15:32
2022-07-29 08:34:04,697 -     INFO - [TRAIN] epoch=486/604, iter=56300/70000, loss=0.223037, lr=0.000001 | ETA 02:57:05
2022-07-29 08:34:48,642 -     INFO - [TRAIN] epoch=486/604, iter=56350/70000, loss=0.253961, lr=0.000001 | ETA 03:07:29
2022-07-29 08:35:34,496 -     INFO - [TRAIN] epoch=487/604, iter=56400/70000, loss=0.265958, lr=0.000001 | ETA 03:40:28
2022-07-29 08:36:17,772 -     INFO - [TRAIN] epoch=487/604, iter=56450/70000, loss=0.242797, lr=0.000001 | ETA 03:30:03
2022-07-29 08:37:03,378 -     INFO - [TRAIN] epoch=488/604, iter=56500/70000, loss=0.237114, lr=0.000001 | ETA 03:07:26
2022-07-29 08:37:47,174 -     INFO - [TRAIN] epoch=488/604, iter=56550/70000, loss=0.270666, lr=0.000001 | ETA 03:16:21
2022-07-29 08:38:31,121 -     INFO - [TRAIN] epoch=488/604, iter=56600/70000, loss=0.229453, lr=0.000001 | ETA 03:30:41
2022-07-29 08:39:16,272 -     INFO - [TRAIN] epoch=489/604, iter=56650/70000, loss=0.269717, lr=0.000001 | ETA 04:06:03
2022-07-29 08:39:59,705 -     INFO - [TRAIN] epoch=489/604, iter=56700/70000, loss=0.273491, lr=0.000001 | ETA 03:13:38
2022-07-29 08:40:45,401 -     INFO - [TRAIN] epoch=490/604, iter=56750/70000, loss=0.228320, lr=0.000001 | ETA 03:01:36
2022-07-29 08:41:28,201 -     INFO - [TRAIN] epoch=490/604, iter=56800/70000, loss=0.265351, lr=0.000001 | ETA 02:56:40
2022-07-29 08:42:12,561 -     INFO - [TRAIN] epoch=491/604, iter=56850/70000, loss=0.256542, lr=0.000001 | ETA 03:41:00
2022-07-29 08:42:55,487 -     INFO - [TRAIN] epoch=491/604, iter=56900/70000, loss=0.250394, lr=0.000001 | ETA 03:15:15
2022-07-29 08:43:37,072 -     INFO - [TRAIN] epoch=491/604, iter=56950/70000, loss=0.284129, lr=0.000001 | ETA 02:47:51
2022-07-29 08:44:22,082 -     INFO - [TRAIN] epoch=492/604, iter=57000/70000, loss=0.262405, lr=0.000001 | ETA 03:16:15
2022-07-29 08:45:05,106 -     INFO - [TRAIN] epoch=492/604, iter=57050/70000, loss=0.283182, lr=0.000001 | ETA 02:57:37
2022-07-29 08:45:49,123 -     INFO - [TRAIN] epoch=493/604, iter=57100/70000, loss=0.241343, lr=0.000001 | ETA 02:47:29
2022-07-29 08:46:32,229 -     INFO - [TRAIN] epoch=493/604, iter=57150/70000, loss=0.186567, lr=0.000001 | ETA 03:15:27
2022-07-29 08:47:15,877 -     INFO - [TRAIN] epoch=494/604, iter=57200/70000, loss=0.238394, lr=0.000001 | ETA 03:40:00
2022-07-29 08:47:58,784 -     INFO - [TRAIN] epoch=494/604, iter=57250/70000, loss=0.211978, lr=0.000001 | ETA 02:57:46
2022-07-29 08:48:41,678 -     INFO - [TRAIN] epoch=494/604, iter=57300/70000, loss=0.274443, lr=0.000001 | ETA 02:48:08
2022-07-29 08:49:27,154 -     INFO - [TRAIN] epoch=495/604, iter=57350/70000, loss=0.256229, lr=0.000001 | ETA 03:26:21
2022-07-29 08:50:09,493 -     INFO - [TRAIN] epoch=495/604, iter=57400/70000, loss=0.285296, lr=0.000001 | ETA 02:46:51
2022-07-29 08:50:54,185 -     INFO - [TRAIN] epoch=496/604, iter=57450/70000, loss=0.248472, lr=0.000001 | ETA 02:51:07
2022-07-29 08:51:37,436 -     INFO - [TRAIN] epoch=496/604, iter=57500/70000, loss=0.255807, lr=0.000001 | ETA 02:50:58
2022-07-29 08:52:20,071 -     INFO - [TRAIN] epoch=497/604, iter=57550/70000, loss=0.216356, lr=0.000001 | ETA 02:45:23
2022-07-29 08:52:59,273 -     INFO - [TRAIN] epoch=497/604, iter=57600/70000, loss=0.265564, lr=0.000001 | ETA 02:42:40
2022-07-29 08:53:38,332 -     INFO - [TRAIN] epoch=497/604, iter=57650/70000, loss=0.243180, lr=0.000001 | ETA 02:38:32
2022-07-29 08:54:20,034 -     INFO - [TRAIN] epoch=498/604, iter=57700/70000, loss=0.248211, lr=0.000001 | ETA 02:41:09
2022-07-29 08:54:59,229 -     INFO - [TRAIN] epoch=498/604, iter=57750/70000, loss=0.245068, lr=0.000001 | ETA 02:42:06
2022-07-29 08:55:39,991 -     INFO - [TRAIN] epoch=499/604, iter=57800/70000, loss=0.263198, lr=0.000001 | ETA 02:38:38
2022-07-29 08:56:19,260 -     INFO - [TRAIN] epoch=499/604, iter=57850/70000, loss=0.271798, lr=0.000001 | ETA 02:39:48
2022-07-29 08:57:01,803 -     INFO - [TRAIN] epoch=500/604, iter=57900/70000, loss=0.278516, lr=0.000001 | ETA 02:37:37
2022-07-29 08:57:43,202 -     INFO - [TRAIN] epoch=500/604, iter=57950/70000, loss=0.228989, lr=0.000001 | ETA 02:38:51
2022-07-29 08:58:24,377 -     INFO - [TRAIN] epoch=500/604, iter=58000/70000, loss=0.209981, lr=0.000001 | ETA 02:34:55
2022-07-29 08:59:07,956 -     INFO - [TRAIN] epoch=501/604, iter=58050/70000, loss=0.225051, lr=0.000001 | ETA 02:47:27
2022-07-29 08:59:49,916 -     INFO - [TRAIN] epoch=501/604, iter=58100/70000, loss=0.225946, lr=0.000001 | ETA 03:14:30
2022-07-29 09:00:34,020 -     INFO - [TRAIN] epoch=502/604, iter=58150/70000, loss=0.252066, lr=0.000001 | ETA 02:40:08
2022-07-29 09:01:14,770 -     INFO - [TRAIN] epoch=502/604, iter=58200/70000, loss=0.239278, lr=0.000001 | ETA 02:36:27
2022-07-29 09:01:58,021 -     INFO - [TRAIN] epoch=503/604, iter=58250/70000, loss=0.276321, lr=0.000001 | ETA 02:48:36
2022-07-29 09:02:39,917 -     INFO - [TRAIN] epoch=503/604, iter=58300/70000, loss=0.255786, lr=0.000001 | ETA 02:55:55
2022-07-29 09:03:22,832 -     INFO - [TRAIN] epoch=504/604, iter=58350/70000, loss=0.220416, lr=0.000001 | ETA 04:21:06
2022-07-29 09:04:05,218 -     INFO - [TRAIN] epoch=504/604, iter=58400/70000, loss=0.259693, lr=0.000001 | ETA 02:32:03
2022-07-29 09:04:47,648 -     INFO - [TRAIN] epoch=504/604, iter=58450/70000, loss=0.234280, lr=0.000001 | ETA 02:57:22
2022-07-29 09:05:31,783 -     INFO - [TRAIN] epoch=505/604, iter=58500/70000, loss=0.254604, lr=0.000001 | ETA 02:34:05
2022-07-29 09:06:14,908 -     INFO - [TRAIN] epoch=505/604, iter=58550/70000, loss=0.238939, lr=0.000001 | ETA 02:36:27
2022-07-29 09:06:59,115 -     INFO - [TRAIN] epoch=506/604, iter=58600/70000, loss=0.203585, lr=0.000001 | ETA 03:08:06
2022-07-29 09:07:41,692 -     INFO - [TRAIN] epoch=506/604, iter=58650/70000, loss=0.235810, lr=0.000001 | ETA 02:28:56
2022-07-29 09:08:25,394 -     INFO - [TRAIN] epoch=507/604, iter=58700/70000, loss=0.253108, lr=0.000001 | ETA 02:55:04
2022-07-29 09:09:07,898 -     INFO - [TRAIN] epoch=507/604, iter=58750/70000, loss=0.261383, lr=0.000001 | ETA 02:29:24
2022-07-29 09:09:49,974 -     INFO - [TRAIN] epoch=507/604, iter=58800/70000, loss=0.261435, lr=0.000001 | ETA 02:45:17
2022-07-29 09:10:34,132 -     INFO - [TRAIN] epoch=508/604, iter=58850/70000, loss=0.260151, lr=0.000001 | ETA 02:36:34
2022-07-29 09:11:16,362 -     INFO - [TRAIN] epoch=508/604, iter=58900/70000, loss=0.250572, lr=0.000001 | ETA 02:23:05
2022-07-29 09:12:00,352 -     INFO - [TRAIN] epoch=509/604, iter=58950/70000, loss=0.265083, lr=0.000001 | ETA 02:26:20
2022-07-29 09:12:42,248 -     INFO - [TRAIN] epoch=509/604, iter=59000/70000, loss=0.224064, lr=0.000001 | ETA 03:04:37
2022-07-29 09:13:25,996 -     INFO - [TRAIN] epoch=510/604, iter=59050/70000, loss=0.216779, lr=0.000001 | ETA 02:43:06
2022-07-29 09:14:07,997 -     INFO - [TRAIN] epoch=510/604, iter=59100/70000, loss=0.258280, lr=0.000001 | ETA 02:33:02
2022-07-29 09:14:50,760 -     INFO - [TRAIN] epoch=510/604, iter=59150/70000, loss=0.229843, lr=0.000001 | ETA 02:37:50
2022-07-29 09:15:35,057 -     INFO - [TRAIN] epoch=511/604, iter=59200/70000, loss=0.222251, lr=0.000001 | ETA 02:27:07
2022-07-29 09:16:17,379 -     INFO - [TRAIN] epoch=511/604, iter=59250/70000, loss=0.233227, lr=0.000001 | ETA 02:35:02
2022-07-29 09:17:01,615 -     INFO - [TRAIN] epoch=512/604, iter=59300/70000, loss=0.251726, lr=0.000001 | ETA 02:21:37
2022-07-29 09:17:44,868 -     INFO - [TRAIN] epoch=512/604, iter=59350/70000, loss=0.227384, lr=0.000001 | ETA 02:30:49
2022-07-29 09:18:29,805 -     INFO - [TRAIN] epoch=513/604, iter=59400/70000, loss=0.251877, lr=0.000001 | ETA 02:40:31
2022-07-29 09:19:11,851 -     INFO - [TRAIN] epoch=513/604, iter=59450/70000, loss=0.220367, lr=0.000001 | ETA 02:18:46
2022-07-29 09:19:55,580 -     INFO - [TRAIN] epoch=513/604, iter=59500/70000, loss=0.217562, lr=0.000001 | ETA 02:30:16
2022-07-29 09:20:40,998 -     INFO - [TRAIN] epoch=514/604, iter=59550/70000, loss=0.218621, lr=0.000001 | ETA 02:20:48
2022-07-29 09:21:24,274 -     INFO - [TRAIN] epoch=514/604, iter=59600/70000, loss=0.234073, lr=0.000001 | ETA 02:31:22
2022-07-29 09:22:09,013 -     INFO - [TRAIN] epoch=515/604, iter=59650/70000, loss=0.219181, lr=0.000001 | ETA 02:21:29
2022-07-29 09:22:52,242 -     INFO - [TRAIN] epoch=515/604, iter=59700/70000, loss=0.271483, lr=0.000001 | ETA 02:46:47
2022-07-29 09:23:37,096 -     INFO - [TRAIN] epoch=516/604, iter=59750/70000, loss=0.249407, lr=0.000001 | ETA 02:17:53
2022-07-29 09:24:20,381 -     INFO - [TRAIN] epoch=516/604, iter=59800/70000, loss=0.252845, lr=0.000001 | ETA 02:32:47
2022-07-29 09:25:04,246 -     INFO - [TRAIN] epoch=516/604, iter=59850/70000, loss=0.262200, lr=0.000001 | ETA 02:42:33
2022-07-29 09:25:48,639 -     INFO - [TRAIN] epoch=517/604, iter=59900/70000, loss=0.248356, lr=0.000001 | ETA 02:17:28
2022-07-29 09:26:32,084 -     INFO - [TRAIN] epoch=517/604, iter=59950/70000, loss=0.272711, lr=0.000001 | ETA 02:09:57
2022-07-29 09:27:17,781 -     INFO - [TRAIN] epoch=518/604, iter=60000/70000, loss=0.222606, lr=0.000001 | ETA 02:39:19
2022-07-29 09:27:17,986 -     INFO - Pop model from ./output/iter_35000
2022-07-29 09:27:18,748 -     INFO - Push model to checkpoint ./output/iter_60000
2022-07-29 09:28:02,665 -     INFO - [TRAIN] epoch=518/604, iter=60050/70000, loss=0.245431, lr=0.000001 | ETA 02:31:41
2022-07-29 09:28:49,049 -     INFO - [TRAIN] epoch=519/604, iter=60100/70000, loss=0.222538, lr=0.000001 | ETA 02:20:33
2022-07-29 09:29:32,473 -     INFO - [TRAIN] epoch=519/604, iter=60150/70000, loss=0.260507, lr=0.000001 | ETA 02:31:09
2022-07-29 09:30:16,530 -     INFO - [TRAIN] epoch=519/604, iter=60200/70000, loss=0.236429, lr=0.000001 | ETA 02:11:51
2022-07-29 09:31:02,074 -     INFO - [TRAIN] epoch=520/604, iter=60250/70000, loss=0.268108, lr=0.000001 | ETA 02:16:50
2022-07-29 09:31:45,374 -     INFO - [TRAIN] epoch=520/604, iter=60300/70000, loss=0.236597, lr=0.000001 | ETA 02:08:58
2022-07-29 09:32:29,393 -     INFO - [TRAIN] epoch=521/604, iter=60350/70000, loss=0.241212, lr=0.000001 | ETA 02:29:02
2022-07-29 09:33:12,400 -     INFO - [TRAIN] epoch=521/604, iter=60400/70000, loss=0.235205, lr=0.000001 | ETA 02:08:24
2022-07-29 09:33:57,996 -     INFO - [TRAIN] epoch=522/604, iter=60450/70000, loss=0.252058, lr=0.000001 | ETA 02:28:05
2022-07-29 09:34:40,648 -     INFO - [TRAIN] epoch=522/604, iter=60500/70000, loss=0.251350, lr=0.000001 | ETA 02:04:39
2022-07-29 09:35:24,960 -     INFO - [TRAIN] epoch=522/604, iter=60550/70000, loss=0.259886, lr=0.000001 | ETA 02:04:00
2022-07-29 09:36:10,639 -     INFO - [TRAIN] epoch=523/604, iter=60600/70000, loss=0.249556, lr=0.000001 | ETA 02:07:20
2022-07-29 09:36:54,073 -     INFO - [TRAIN] epoch=523/604, iter=60650/70000, loss=0.240758, lr=0.000001 | ETA 02:09:03
2022-07-29 09:37:39,557 -     INFO - [TRAIN] epoch=524/604, iter=60700/70000, loss=0.214973, lr=0.000001 | ETA 02:11:17
2022-07-29 09:38:21,897 -     INFO - [TRAIN] epoch=524/604, iter=60750/70000, loss=0.255265, lr=0.000001 | ETA 02:16:24
2022-07-29 09:39:08,253 -     INFO - [TRAIN] epoch=525/604, iter=60800/70000, loss=0.186310, lr=0.000001 | ETA 02:28:47
2022-07-29 09:39:51,865 -     INFO - [TRAIN] epoch=525/604, iter=60850/70000, loss=0.267246, lr=0.000001 | ETA 02:10:56
2022-07-29 09:40:36,426 -     INFO - [TRAIN] epoch=525/604, iter=60900/70000, loss=0.220822, lr=0.000001 | ETA 02:00:55
2022-07-29 09:41:22,280 -     INFO - [TRAIN] epoch=526/604, iter=60950/70000, loss=0.245402, lr=0.000001 | ETA 02:01:07
2022-07-29 09:42:07,757 -     INFO - [TRAIN] epoch=526/604, iter=61000/70000, loss=0.247454, lr=0.000001 | ETA 01:57:37
2022-07-29 09:42:52,398 -     INFO - [TRAIN] epoch=527/604, iter=61050/70000, loss=0.217706, lr=0.000001 | ETA 02:26:19
2022-07-29 09:43:35,020 -     INFO - [TRAIN] epoch=527/604, iter=61100/70000, loss=0.260890, lr=0.000001 | ETA 01:58:45
2022-07-29 09:44:23,101 -     INFO - [TRAIN] epoch=528/604, iter=61150/70000, loss=0.237593, lr=0.000001 | ETA 02:30:58
2022-07-29 09:45:06,753 -     INFO - [TRAIN] epoch=528/604, iter=61200/70000, loss=0.251623, lr=0.000001 | ETA 02:48:00
2022-07-29 09:45:51,286 -     INFO - [TRAIN] epoch=529/604, iter=61250/70000, loss=0.269474, lr=0.000001 | ETA 03:01:29
2022-07-29 09:46:35,999 -     INFO - [TRAIN] epoch=529/604, iter=61300/70000, loss=0.268734, lr=0.000001 | ETA 02:00:19
2022-07-29 09:47:19,184 -     INFO - [TRAIN] epoch=529/604, iter=61350/70000, loss=0.215522, lr=0.000001 | ETA 01:52:06
2022-07-29 09:48:05,988 -     INFO - [TRAIN] epoch=530/604, iter=61400/70000, loss=0.229672, lr=0.000001 | ETA 02:34:53
2022-07-29 09:48:50,137 -     INFO - [TRAIN] epoch=530/604, iter=61450/70000, loss=0.289795, lr=0.000001 | ETA 02:12:44
2022-07-29 09:49:35,873 -     INFO - [TRAIN] epoch=531/604, iter=61500/70000, loss=0.257352, lr=0.000001 | ETA 01:57:02
2022-07-29 09:50:18,480 -     INFO - [TRAIN] epoch=531/604, iter=61550/70000, loss=0.268754, lr=0.000001 | ETA 01:54:23
2022-07-29 09:51:03,969 -     INFO - [TRAIN] epoch=532/604, iter=61600/70000, loss=0.209402, lr=0.000001 | ETA 02:12:26
2022-07-29 09:51:49,120 -     INFO - [TRAIN] epoch=532/604, iter=61650/70000, loss=0.210634, lr=0.000001 | ETA 01:49:50
2022-07-29 09:52:33,013 -     INFO - [TRAIN] epoch=532/604, iter=61700/70000, loss=0.231878, lr=0.000001 | ETA 02:05:34
2022-07-29 09:53:18,111 -     INFO - [TRAIN] epoch=533/604, iter=61750/70000, loss=0.236797, lr=0.000001 | ETA 01:46:37
2022-07-29 09:54:00,686 -     INFO - [TRAIN] epoch=533/604, iter=61800/70000, loss=0.252791, lr=0.000001 | ETA 01:52:43
2022-07-29 09:54:45,421 -     INFO - [TRAIN] epoch=534/604, iter=61850/70000, loss=0.257042, lr=0.000001 | ETA 02:02:51
2022-07-29 09:55:29,494 -     INFO - [TRAIN] epoch=534/604, iter=61900/70000, loss=0.233947, lr=0.000001 | ETA 02:29:55
2022-07-29 09:56:14,991 -     INFO - [TRAIN] epoch=535/604, iter=61950/70000, loss=0.230440, lr=0.000001 | ETA 02:10:32
2022-07-29 09:56:59,171 -     INFO - [TRAIN] epoch=535/604, iter=62000/70000, loss=0.235658, lr=0.000001 | ETA 01:52:59
2022-07-29 09:57:43,362 -     INFO - [TRAIN] epoch=535/604, iter=62050/70000, loss=0.238758, lr=0.000001 | ETA 02:18:32
2022-07-29 09:58:28,126 -     INFO - [TRAIN] epoch=536/604, iter=62100/70000, loss=0.239752, lr=0.000001 | ETA 02:27:40
2022-07-29 09:59:12,073 -     INFO - [TRAIN] epoch=536/604, iter=62150/70000, loss=0.239294, lr=0.000001 | ETA 02:03:42
2022-07-29 09:59:57,894 -     INFO - [TRAIN] epoch=537/604, iter=62200/70000, loss=0.275037, lr=0.000001 | ETA 01:45:44
2022-07-29 10:00:41,266 -     INFO - [TRAIN] epoch=537/604, iter=62250/70000, loss=0.236022, lr=0.000001 | ETA 02:02:43
2022-07-29 10:01:26,194 -     INFO - [TRAIN] epoch=538/604, iter=62300/70000, loss=0.252020, lr=0.000001 | ETA 02:00:55
2022-07-29 10:02:08,910 -     INFO - [TRAIN] epoch=538/604, iter=62350/70000, loss=0.240667, lr=0.000001 | ETA 01:53:59
2022-07-29 10:02:52,975 -     INFO - [TRAIN] epoch=538/604, iter=62400/70000, loss=0.268599, lr=0.000001 | ETA 01:56:50
2022-07-29 10:03:36,433 -     INFO - [TRAIN] epoch=539/604, iter=62450/70000, loss=0.225066, lr=0.000001 | ETA 01:37:57
2022-07-29 10:04:20,286 -     INFO - [TRAIN] epoch=539/604, iter=62500/70000, loss=0.230257, lr=0.000001 | ETA 01:50:47
2022-07-29 10:05:04,587 -     INFO - [TRAIN] epoch=540/604, iter=62550/70000, loss=0.257976, lr=0.000001 | ETA 01:37:47
2022-07-29 10:05:47,917 -     INFO - [TRAIN] epoch=540/604, iter=62600/70000, loss=0.250142, lr=0.000001 | ETA 01:44:25
2022-07-29 10:06:32,697 -     INFO - [TRAIN] epoch=541/604, iter=62650/70000, loss=0.251568, lr=0.000001 | ETA 01:41:40
2022-07-29 10:07:15,698 -     INFO - [TRAIN] epoch=541/604, iter=62700/70000, loss=0.235346, lr=0.000001 | ETA 01:41:10
2022-07-29 10:07:58,471 -     INFO - [TRAIN] epoch=541/604, iter=62750/70000, loss=0.269081, lr=0.000001 | ETA 01:36:46
2022-07-29 10:08:42,574 -     INFO - [TRAIN] epoch=542/604, iter=62800/70000, loss=0.262711, lr=0.000001 | ETA 01:46:56
2022-07-29 10:09:25,866 -     INFO - [TRAIN] epoch=542/604, iter=62850/70000, loss=0.206855, lr=0.000001 | ETA 01:45:01
2022-07-29 10:10:11,096 -     INFO - [TRAIN] epoch=543/604, iter=62900/70000, loss=0.258281, lr=0.000001 | ETA 01:40:33
2022-07-29 10:10:55,994 -     INFO - [TRAIN] epoch=543/604, iter=62950/70000, loss=0.252371, lr=0.000001 | ETA 02:15:16
2022-07-29 10:11:40,848 -     INFO - [TRAIN] epoch=544/604, iter=63000/70000, loss=0.237794, lr=0.000001 | ETA 01:51:09
2022-07-29 10:12:25,060 -     INFO - [TRAIN] epoch=544/604, iter=63050/70000, loss=0.235327, lr=0.000001 | ETA 01:37:18
2022-07-29 10:13:09,781 -     INFO - [TRAIN] epoch=544/604, iter=63100/70000, loss=0.247902, lr=0.000001 | ETA 01:37:26
2022-07-29 10:13:55,720 -     INFO - [TRAIN] epoch=545/604, iter=63150/70000, loss=0.262977, lr=0.000001 | ETA 01:34:41
2022-07-29 10:14:40,663 -     INFO - [TRAIN] epoch=545/604, iter=63200/70000, loss=0.270128, lr=0.000001 | ETA 01:55:52
2022-07-29 10:15:25,353 -     INFO - [TRAIN] epoch=546/604, iter=63250/70000, loss=0.264186, lr=0.000001 | ETA 01:40:12
2022-07-29 10:16:08,474 -     INFO - [TRAIN] epoch=546/604, iter=63300/70000, loss=0.233301, lr=0.000001 | ETA 01:34:47
2022-07-29 10:16:52,500 -     INFO - [TRAIN] epoch=547/604, iter=63350/70000, loss=0.234685, lr=0.000001 | ETA 01:33:14
2022-07-29 10:17:36,647 -     INFO - [TRAIN] epoch=547/604, iter=63400/70000, loss=0.215155, lr=0.000001 | ETA 01:33:21
2022-07-29 10:18:21,505 -     INFO - [TRAIN] epoch=547/604, iter=63450/70000, loss=0.243441, lr=0.000001 | ETA 01:25:06
2022-07-29 10:19:07,155 -     INFO - [TRAIN] epoch=548/604, iter=63500/70000, loss=0.220767, lr=0.000001 | ETA 01:36:47
2022-07-29 10:19:51,976 -     INFO - [TRAIN] epoch=548/604, iter=63550/70000, loss=0.243976, lr=0.000001 | ETA 01:57:17
2022-07-29 10:20:37,095 -     INFO - [TRAIN] epoch=549/604, iter=63600/70000, loss=0.238602, lr=0.000001 | ETA 01:39:06
2022-07-29 10:21:21,767 -     INFO - [TRAIN] epoch=549/604, iter=63650/70000, loss=0.205729, lr=0.000001 | ETA 01:25:01
2022-07-29 10:22:06,430 -     INFO - [TRAIN] epoch=550/604, iter=63700/70000, loss=0.235969, lr=0.000001 | ETA 01:39:19
2022-07-29 10:22:50,136 -     INFO - [TRAIN] epoch=550/604, iter=63750/70000, loss=0.234146, lr=0.000001 | ETA 01:25:30
2022-07-29 10:23:33,399 -     INFO - [TRAIN] epoch=550/604, iter=63800/70000, loss=0.265804, lr=0.000001 | ETA 01:20:36
2022-07-29 10:24:20,299 -     INFO - [TRAIN] epoch=551/604, iter=63850/70000, loss=0.213662, lr=0.000001 | ETA 01:26:12
2022-07-29 10:25:03,905 -     INFO - [TRAIN] epoch=551/604, iter=63900/70000, loss=0.250062, lr=0.000001 | ETA 01:19:10
2022-07-29 10:25:48,620 -     INFO - [TRAIN] epoch=552/604, iter=63950/70000, loss=0.234736, lr=0.000001 | ETA 01:30:27
2022-07-29 10:26:33,529 -     INFO - [TRAIN] epoch=552/604, iter=64000/70000, loss=0.211900, lr=0.000001 | ETA 01:21:31
2022-07-29 10:27:19,760 -     INFO - [TRAIN] epoch=553/604, iter=64050/70000, loss=0.228108, lr=0.000001 | ETA 01:28:57
2022-07-29 10:28:03,351 -     INFO - [TRAIN] epoch=553/604, iter=64100/70000, loss=0.240901, lr=0.000001 | ETA 01:18:50
2022-07-29 10:28:49,754 -     INFO - [TRAIN] epoch=554/604, iter=64150/70000, loss=0.246791, lr=0.000001 | ETA 01:59:20
2022-07-29 10:29:32,751 -     INFO - [TRAIN] epoch=554/604, iter=64200/70000, loss=0.230714, lr=0.000001 | ETA 01:17:41
2022-07-29 10:30:16,052 -     INFO - [TRAIN] epoch=554/604, iter=64250/70000, loss=0.219492, lr=0.000001 | ETA 01:37:54
2022-07-29 10:31:01,480 -     INFO - [TRAIN] epoch=555/604, iter=64300/70000, loss=0.238119, lr=0.000001 | ETA 01:15:07
2022-07-29 10:31:45,409 -     INFO - [TRAIN] epoch=555/604, iter=64350/70000, loss=0.203824, lr=0.000001 | ETA 01:40:57
2022-07-29 10:32:30,341 -     INFO - [TRAIN] epoch=556/604, iter=64400/70000, loss=0.247152, lr=0.000001 | ETA 01:26:59
2022-07-29 10:33:13,671 -     INFO - [TRAIN] epoch=556/604, iter=64450/70000, loss=0.252982, lr=0.000001 | ETA 01:16:25
2022-07-29 10:33:58,274 -     INFO - [TRAIN] epoch=557/604, iter=64500/70000, loss=0.217383, lr=0.000001 | ETA 01:22:37
2022-07-29 10:34:42,861 -     INFO - [TRAIN] epoch=557/604, iter=64550/70000, loss=0.229536, lr=0.000001 | ETA 01:27:40
2022-07-29 10:35:26,501 -     INFO - [TRAIN] epoch=557/604, iter=64600/70000, loss=0.224787, lr=0.000001 | ETA 01:19:08
2022-07-29 10:36:11,122 -     INFO - [TRAIN] epoch=558/604, iter=64650/70000, loss=0.244566, lr=0.000001 | ETA 01:10:41
2022-07-29 10:36:54,684 -     INFO - [TRAIN] epoch=558/604, iter=64700/70000, loss=0.256231, lr=0.000001 | ETA 01:13:34
2022-07-29 10:37:41,183 -     INFO - [TRAIN] epoch=559/604, iter=64750/70000, loss=0.265464, lr=0.000001 | ETA 01:19:49
2022-07-29 10:38:25,178 -     INFO - [TRAIN] epoch=559/604, iter=64800/70000, loss=0.206200, lr=0.000001 | ETA 01:11:56
2022-07-29 10:39:11,404 -     INFO - [TRAIN] epoch=560/604, iter=64850/70000, loss=0.227718, lr=0.000001 | ETA 01:16:41
2022-07-29 10:39:57,173 -     INFO - [TRAIN] epoch=560/604, iter=64900/70000, loss=0.262054, lr=0.000001 | ETA 01:22:22
2022-07-29 10:40:42,162 -     INFO - [TRAIN] epoch=560/604, iter=64950/70000, loss=0.243408, lr=0.000001 | ETA 01:08:50
2022-07-29 10:41:27,128 -     INFO - [TRAIN] epoch=561/604, iter=65000/70000, loss=0.230353, lr=0.000001 | ETA 01:26:22
2022-07-29 10:41:27,331 -     INFO - Pop model from ./output/iter_40000
2022-07-29 10:41:28,039 -     INFO - Push model to checkpoint ./output/iter_65000
2022-07-29 10:42:10,881 -     INFO - [TRAIN] epoch=561/604, iter=65050/70000, loss=0.237890, lr=0.000001 | ETA 01:19:51
2022-07-29 10:42:56,061 -     INFO - [TRAIN] epoch=562/604, iter=65100/70000, loss=0.231182, lr=0.000001 | ETA 01:04:41
2022-07-29 10:43:40,101 -     INFO - [TRAIN] epoch=562/604, iter=65150/70000, loss=0.252542, lr=0.000001 | ETA 01:08:22
2022-07-29 10:44:24,584 -     INFO - [TRAIN] epoch=563/604, iter=65200/70000, loss=0.226089, lr=0.000001 | ETA 01:06:02
2022-07-29 10:45:06,490 -     INFO - [TRAIN] epoch=563/604, iter=65250/70000, loss=0.258406, lr=0.000001 | ETA 01:03:14
2022-07-29 10:45:46,652 -     INFO - [TRAIN] epoch=563/604, iter=65300/70000, loss=0.224276, lr=0.000001 | ETA 01:02:56
2022-07-29 10:46:27,858 -     INFO - [TRAIN] epoch=564/604, iter=65350/70000, loss=0.231975, lr=0.000001 | ETA 01:01:59
2022-07-29 10:47:07,002 -     INFO - [TRAIN] epoch=564/604, iter=65400/70000, loss=0.240892, lr=0.000001 | ETA 01:00:08
2022-07-29 10:47:47,654 -     INFO - [TRAIN] epoch=565/604, iter=65450/70000, loss=0.227090, lr=0.000001 | ETA 00:58:09
2022-07-29 10:48:27,638 -     INFO - [TRAIN] epoch=565/604, iter=65500/70000, loss=0.309435, lr=0.000001 | ETA 00:59:58
2022-07-29 10:49:09,776 -     INFO - [TRAIN] epoch=566/604, iter=65550/70000, loss=0.228560, lr=0.000001 | ETA 01:00:34
2022-07-29 10:49:51,216 -     INFO - [TRAIN] epoch=566/604, iter=65600/70000, loss=0.264987, lr=0.000001 | ETA 01:06:15
2022-07-29 10:50:32,870 -     INFO - [TRAIN] epoch=566/604, iter=65650/70000, loss=0.224371, lr=0.000001 | ETA 00:56:21
2022-07-29 10:51:15,108 -     INFO - [TRAIN] epoch=567/604, iter=65700/70000, loss=0.272717, lr=0.000001 | ETA 00:55:11
2022-07-29 10:51:57,237 -     INFO - [TRAIN] epoch=567/604, iter=65750/70000, loss=0.245495, lr=0.000001 | ETA 00:59:03
2022-07-29 10:52:42,817 -     INFO - [TRAIN] epoch=568/604, iter=65800/70000, loss=0.270453, lr=0.000001 | ETA 01:17:46
2022-07-29 10:53:26,120 -     INFO - [TRAIN] epoch=568/604, iter=65850/70000, loss=0.254893, lr=0.000001 | ETA 00:59:39
2022-07-29 10:54:08,663 -     INFO - [TRAIN] epoch=569/604, iter=65900/70000, loss=0.281831, lr=0.000001 | ETA 00:54:02
2022-07-29 10:54:50,729 -     INFO - [TRAIN] epoch=569/604, iter=65950/70000, loss=0.248663, lr=0.000001 | ETA 01:01:51
2022-07-29 10:55:32,703 -     INFO - [TRAIN] epoch=569/604, iter=66000/70000, loss=0.282582, lr=0.000001 | ETA 00:55:37
2022-07-29 10:56:17,462 -     INFO - [TRAIN] epoch=570/604, iter=66050/70000, loss=0.236450, lr=0.000001 | ETA 01:06:10
2022-07-29 10:56:59,617 -     INFO - [TRAIN] epoch=570/604, iter=66100/70000, loss=0.245694, lr=0.000001 | ETA 00:53:58
2022-07-29 10:57:43,701 -     INFO - [TRAIN] epoch=571/604, iter=66150/70000, loss=0.239121, lr=0.000001 | ETA 00:50:37
2022-07-29 10:58:26,229 -     INFO - [TRAIN] epoch=571/604, iter=66200/70000, loss=0.222355, lr=0.000001 | ETA 00:51:14
2022-07-29 10:59:09,176 -     INFO - [TRAIN] epoch=572/604, iter=66250/70000, loss=0.198214, lr=0.000001 | ETA 00:55:11
2022-07-29 10:59:51,975 -     INFO - [TRAIN] epoch=572/604, iter=66300/70000, loss=0.259386, lr=0.000001 | ETA 00:53:27
2022-07-29 11:00:34,117 -     INFO - [TRAIN] epoch=572/604, iter=66350/70000, loss=0.224340, lr=0.000001 | ETA 00:47:58
2022-07-29 11:01:18,163 -     INFO - [TRAIN] epoch=573/604, iter=66400/70000, loss=0.278644, lr=0.000001 | ETA 00:59:42
2022-07-29 11:02:01,468 -     INFO - [TRAIN] epoch=573/604, iter=66450/70000, loss=0.244261, lr=0.000001 | ETA 00:52:42
2022-07-29 11:02:46,382 -     INFO - [TRAIN] epoch=574/604, iter=66500/70000, loss=0.233134, lr=0.000001 | ETA 00:48:22
2022-07-29 11:03:27,705 -     INFO - [TRAIN] epoch=574/604, iter=66550/70000, loss=0.285497, lr=0.000001 | ETA 00:44:48
2022-07-29 11:04:11,976 -     INFO - [TRAIN] epoch=575/604, iter=66600/70000, loss=0.260670, lr=0.000001 | ETA 00:51:52
2022-07-29 11:04:54,975 -     INFO - [TRAIN] epoch=575/604, iter=66650/70000, loss=0.316973, lr=0.000001 | ETA 00:46:40
2022-07-29 11:05:38,062 -     INFO - [TRAIN] epoch=575/604, iter=66700/70000, loss=0.225087, lr=0.000001 | ETA 00:43:01
2022-07-29 11:06:21,938 -     INFO - [TRAIN] epoch=576/604, iter=66750/70000, loss=0.240203, lr=0.000001 | ETA 00:43:15
2022-07-29 11:07:05,074 -     INFO - [TRAIN] epoch=576/604, iter=66800/70000, loss=0.247390, lr=0.000001 | ETA 00:42:57
2022-07-29 11:07:50,183 -     INFO - [TRAIN] epoch=577/604, iter=66850/70000, loss=0.262370, lr=0.000001 | ETA 00:44:40
2022-07-29 11:08:32,294 -     INFO - [TRAIN] epoch=577/604, iter=66900/70000, loss=0.281178, lr=0.000001 | ETA 00:40:43
2022-07-29 11:09:15,930 -     INFO - [TRAIN] epoch=578/604, iter=66950/70000, loss=0.224286, lr=0.000001 | ETA 00:42:20
2022-07-29 11:09:58,141 -     INFO - [TRAIN] epoch=578/604, iter=67000/70000, loss=0.272721, lr=0.000001 | ETA 00:40:31
2022-07-29 11:10:42,541 -     INFO - [TRAIN] epoch=579/604, iter=67050/70000, loss=0.208204, lr=0.000001 | ETA 01:01:07
2022-07-29 11:11:26,091 -     INFO - [TRAIN] epoch=579/604, iter=67100/70000, loss=0.265240, lr=0.000001 | ETA 00:39:02
2022-07-29 11:12:09,673 -     INFO - [TRAIN] epoch=579/604, iter=67150/70000, loss=0.253175, lr=0.000001 | ETA 00:47:44
2022-07-29 11:12:53,827 -     INFO - [TRAIN] epoch=580/604, iter=67200/70000, loss=0.232974, lr=0.000001 | ETA 00:36:26
2022-07-29 11:13:37,626 -     INFO - [TRAIN] epoch=580/604, iter=67250/70000, loss=0.223723, lr=0.000001 | ETA 00:37:18
2022-07-29 11:14:21,583 -     INFO - [TRAIN] epoch=581/604, iter=67300/70000, loss=0.232597, lr=0.000001 | ETA 00:35:47
2022-07-29 11:15:03,514 -     INFO - [TRAIN] epoch=581/604, iter=67350/70000, loss=0.246812, lr=0.000001 | ETA 00:34:28
2022-07-29 11:15:47,849 -     INFO - [TRAIN] epoch=582/604, iter=67400/70000, loss=0.240609, lr=0.000001 | ETA 00:41:34
2022-07-29 11:16:31,741 -     INFO - [TRAIN] epoch=582/604, iter=67450/70000, loss=0.247495, lr=0.000001 | ETA 00:33:48
2022-07-29 11:17:15,745 -     INFO - [TRAIN] epoch=582/604, iter=67500/70000, loss=0.280311, lr=0.000001 | ETA 00:36:30
2022-07-29 11:18:01,108 -     INFO - [TRAIN] epoch=583/604, iter=67550/70000, loss=0.249004, lr=0.000001 | ETA 00:33:48
2022-07-29 11:18:42,858 -     INFO - [TRAIN] epoch=583/604, iter=67600/70000, loss=0.259354, lr=0.000001 | ETA 00:30:59
2022-07-29 11:19:27,604 -     INFO - [TRAIN] epoch=584/604, iter=67650/70000, loss=0.229241, lr=0.000001 | ETA 00:37:32
2022-07-29 11:20:11,164 -     INFO - [TRAIN] epoch=584/604, iter=67700/70000, loss=0.204621, lr=0.000001 | ETA 00:31:44
2022-07-29 11:20:55,930 -     INFO - [TRAIN] epoch=585/604, iter=67750/70000, loss=0.261623, lr=0.000001 | ETA 00:32:07
2022-07-29 11:21:39,594 -     INFO - [TRAIN] epoch=585/604, iter=67800/70000, loss=0.221468, lr=0.000001 | ETA 00:33:11
2022-07-29 11:22:22,039 -     INFO - [TRAIN] epoch=585/604, iter=67850/70000, loss=0.253097, lr=0.000001 | ETA 00:28:53
2022-07-29 11:23:07,917 -     INFO - [TRAIN] epoch=586/604, iter=67900/70000, loss=0.247091, lr=0.000001 | ETA 00:30:18
2022-07-29 11:23:52,344 -     INFO - [TRAIN] epoch=586/604, iter=67950/70000, loss=0.209155, lr=0.000001 | ETA 00:28:29
2022-07-29 11:24:36,284 -     INFO - [TRAIN] epoch=587/604, iter=68000/70000, loss=0.248605, lr=0.000001 | ETA 00:27:10
2022-07-29 11:25:20,937 -     INFO - [TRAIN] epoch=587/604, iter=68050/70000, loss=0.248041, lr=0.000001 | ETA 00:26:43
2022-07-29 11:26:06,446 -     INFO - [TRAIN] epoch=588/604, iter=68100/70000, loss=0.256676, lr=0.000001 | ETA 00:30:24
2022-07-29 11:26:50,517 -     INFO - [TRAIN] epoch=588/604, iter=68150/70000, loss=0.235537, lr=0.000001 | ETA 00:25:53
2022-07-29 11:27:32,896 -     INFO - [TRAIN] epoch=588/604, iter=68200/70000, loss=0.205408, lr=0.000001 | ETA 00:32:14
2022-07-29 11:28:19,310 -     INFO - [TRAIN] epoch=589/604, iter=68250/70000, loss=0.251976, lr=0.000001 | ETA 00:30:55
2022-07-29 11:29:03,087 -     INFO - [TRAIN] epoch=589/604, iter=68300/70000, loss=0.264124, lr=0.000001 | ETA 00:22:11
2022-07-29 11:29:48,585 -     INFO - [TRAIN] epoch=590/604, iter=68350/70000, loss=0.235369, lr=0.000001 | ETA 00:22:05
2022-07-29 11:30:32,438 -     INFO - [TRAIN] epoch=590/604, iter=68400/70000, loss=0.248160, lr=0.000001 | ETA 00:22:46
2022-07-29 11:31:17,414 -     INFO - [TRAIN] epoch=591/604, iter=68450/70000, loss=0.267604, lr=0.000001 | ETA 00:27:33
2022-07-29 11:31:59,599 -     INFO - [TRAIN] epoch=591/604, iter=68500/70000, loss=0.219313, lr=0.000001 | ETA 00:21:16
2022-07-29 11:32:42,309 -     INFO - [TRAIN] epoch=591/604, iter=68550/70000, loss=0.232931, lr=0.000001 | ETA 00:22:25
2022-07-29 11:33:27,011 -     INFO - [TRAIN] epoch=592/604, iter=68600/70000, loss=0.249953, lr=0.000001 | ETA 00:24:07
2022-07-29 11:34:10,765 -     INFO - [TRAIN] epoch=592/604, iter=68650/70000, loss=0.245541, lr=0.000001 | ETA 00:20:04
2022-07-29 11:34:55,843 -     INFO - [TRAIN] epoch=593/604, iter=68700/70000, loss=0.244538, lr=0.000001 | ETA 00:17:35
2022-07-29 11:35:39,060 -     INFO - [TRAIN] epoch=593/604, iter=68750/70000, loss=0.260468, lr=0.000001 | ETA 00:17:11
2022-07-29 11:36:23,082 -     INFO - [TRAIN] epoch=594/604, iter=68800/70000, loss=0.229354, lr=0.000001 | ETA 00:15:52
2022-07-29 11:37:06,954 -     INFO - [TRAIN] epoch=594/604, iter=68850/70000, loss=0.247950, lr=0.000001 | ETA 00:17:24
2022-07-29 11:37:48,322 -     INFO - [TRAIN] epoch=594/604, iter=68900/70000, loss=0.249719, lr=0.000001 | ETA 00:14:40
2022-07-29 11:38:33,303 -     INFO - [TRAIN] epoch=595/604, iter=68950/70000, loss=0.243329, lr=0.000001 | ETA 00:15:05
2022-07-29 11:39:15,357 -     INFO - [TRAIN] epoch=595/604, iter=69000/70000, loss=0.217331, lr=0.000001 | ETA 00:13:50
2022-07-29 11:39:59,875 -     INFO - [TRAIN] epoch=596/604, iter=69050/70000, loss=0.260233, lr=0.000001 | ETA 00:16:38
2022-07-29 11:40:42,767 -     INFO - [TRAIN] epoch=596/604, iter=69100/70000, loss=0.232035, lr=0.000001 | ETA 00:12:03
2022-07-29 11:41:27,545 -     INFO - [TRAIN] epoch=597/604, iter=69150/70000, loss=0.238583, lr=0.000001 | ETA 00:11:35
2022-07-29 11:42:11,497 -     INFO - [TRAIN] epoch=597/604, iter=69200/70000, loss=0.225168, lr=0.000001 | ETA 00:12:49
2022-07-29 11:42:56,338 -     INFO - [TRAIN] epoch=597/604, iter=69250/70000, loss=0.238929, lr=0.000001 | ETA 00:10:02
2022-07-29 11:43:43,691 -     INFO - [TRAIN] epoch=598/604, iter=69300/70000, loss=0.248977, lr=0.000001 | ETA 00:10:37
2022-07-29 11:44:27,132 -     INFO - [TRAIN] epoch=598/604, iter=69350/70000, loss=0.243820, lr=0.000001 | ETA 00:09:09
2022-07-29 11:45:10,538 -     INFO - [TRAIN] epoch=599/604, iter=69400/70000, loss=0.263715, lr=0.000001 | ETA 00:08:22
2022-07-29 11:45:54,184 -     INFO - [TRAIN] epoch=599/604, iter=69450/70000, loss=0.216062, lr=0.000001 | ETA 00:07:19
2022-07-29 11:46:40,683 -     INFO - [TRAIN] epoch=600/604, iter=69500/70000, loss=0.248890, lr=0.000001 | ETA 00:06:55
2022-07-29 11:47:24,787 -     INFO - [TRAIN] epoch=600/604, iter=69550/70000, loss=0.269212, lr=0.000001 | ETA 00:07:48
2022-07-29 11:48:08,465 -     INFO - [TRAIN] epoch=600/604, iter=69600/70000, loss=0.211266, lr=0.000001 | ETA 00:05:10
2022-07-29 11:48:53,786 -     INFO - [TRAIN] epoch=601/604, iter=69650/70000, loss=0.248480, lr=0.000001 | ETA 00:05:52
2022-07-29 11:49:38,167 -     INFO - [TRAIN] epoch=601/604, iter=69700/70000, loss=0.221263, lr=0.000001 | ETA 00:05:07
2022-07-29 11:50:22,853 -     INFO - [TRAIN] epoch=602/604, iter=69750/70000, loss=0.270884, lr=0.000001 | ETA 00:03:46
2022-07-29 11:51:07,425 -     INFO - [TRAIN] epoch=602/604, iter=69800/70000, loss=0.204046, lr=0.000001 | ETA 00:02:48
2022-07-29 11:51:52,142 -     INFO - [TRAIN] epoch=603/604, iter=69850/70000, loss=0.254135, lr=0.000001 | ETA 00:02:14
2022-07-29 11:52:35,221 -     INFO - [TRAIN] epoch=603/604, iter=69900/70000, loss=0.235803, lr=0.000001 | ETA 00:01:24
2022-07-29 11:53:21,092 -     INFO - [TRAIN] epoch=604/604, iter=69950/70000, loss=0.217118, lr=0.000001 | ETA 00:01:02
2022-07-29 11:54:06,073 -     INFO - [TRAIN] epoch=604/604, iter=70000/70000, loss=0.260172, lr=0.000001 | ETA 00:00:00
2022-07-29 11:54:06,278 -     INFO - Pop model from ./output/iter_45000
2022-07-29 11:54:07,013 -     INFO - Push model to checkpoint ./output/iter_70000
2022-07-29 11:54:07,028 -     INFO - Training is complete.
LAUNCH INFO 2022-07-29 11:54:15,455 Pod completed
LAUNCH INFO 2022-07-29 11:54:15,455 Exit code 0