とりあえずsystemd のntp 止めて，
$ sudo timedatectl set-ntp false
$ sudo timedatectl set-ntp false
$ sudo systemctl stop systemd-timesyncd
$ sudo systemctl disable systemd-timesyncd

ntpdate 入れて，
$ sudo apt install ntpdate

いっぱい時間ずれてる……
$ /usr/sbin/ntpdate -q ntp.nict.jp
server 133.243.238.163, stratum 1, offset 89.572923, delay 0.07468
server 133.243.238.243, stratum 1, offset 89.572751, delay 0.07541
server 133.243.238.164, stratum 1, offset 89.572830, delay 0.07477
server 133.243.238.244, stratum 1, offset 89.573035, delay 0.07434
31 May 01:19:57 ntpdate[30108]: step time server 133.243.238.244 offset 89.573035 sec

とりあえず時間合わせる
$ sudo /usr/sbin/ntpdate ntp.nict.jp
31 May 01:22:11 ntpdate[30881]: step time server 133.243.238.164 offset 89.572992 sec

totp はうまく行った