Looks like I ran out of memory or SWAP?
ATOP - ip-172-31-92-154 2021/06/09 12:00:04 ---a---------- 10m0s elapsed
PRC | sys 9m56s | user 4.16s | | #proc 100 | #trun 4 | #tslpi 120 | | #tslpu 17 | #zombie 0 | clones 1 | | #exit 0 |
CPU | sys 99% | user 0% | irq 1% | idle 0% | wait 0% | steal 0% | guest 0% | | ipc notavail | cycl unknown | curf 2.40GHz | curscal ?% |
CPL | avg1 18.91 | avg5 18.65 | | avg15 11.81 | | | csw 1321637 | intr 702191 | | | numcpu 1 | |
MEM | tot 978.6M | free 63.5M | cache 19.7M | dirty 0.0M | buff 0.3M | slab 81.3M | slrec 26.4M | shmem 13.0M | shrss 0.0M | vmbal 0.0M | hptot 0.0M | hpuse 0.0M |
SWP | tot 1.0G | free 0.0M | | | | | | | | | vmcom 2.9G | vmlim 1.5G |
PAG | scan 15052e4 | steal 1007e4 | stall 0 | | | | | | | | swin 286 | swout 501 |
PSI | cs 23/22/20 | ms 99/99/94 | | mf 10/9/9 | is 99/99/95 | | if 99/99/95 | | | | | |
DSK | xvda | busy 100% | read 558634 | | write 515 | KiB/r 67 | KiB/w 4 | MBr/s 61.0 | MBw/s 0.0 | | avq 13.66 | avio 1.07 ms |
DSK | xvda1 | busy 100% | read 558634 | | write 515 | KiB/r 67 | KiB/w 4 | MBr/s 61.0 | MBw/s 0.0 | | avq 13.66 | avio 1.07 ms |
NET | transport | tcpi 97 | tcpo 74 | udpi 0 | udpo 0 | tcpao 0 | tcppo 23 | tcprs 7 | tcpie 0 | tcpor 0 | udpnp 0 | udpie 0 |
NET | network | ipi 97 | ipo 76 | | ipfrw 0 | deliv 97 | | | | | icmpi 0 | icmpo 0 |
NET | eth0 ---- | pcki 114 | pcko 93 | sp 0 Mbps | si 0 Kbps | so 0 Kbps | coll 0 | mlti 0 | erri 0 | erro 0 | drpi 0 | drpo 0 |
All processes/threads will be shown/accumulated...
PID SYSCPU USRCPU VGROW RGROW RDDSK WRDSK RUID EUID ST EXC THR S CPUNR CPU CMD 1/3
82 8m26s 0.00s 0K 0K 0K 0K root root -- - 1 R 0 84% kswapd0
463 55.57s 0.00s 0K 84K 531.8M 0K root root -- - 9 S 0 9% snapd
900 11.17s 0.00s 0K -164K 13.0G 0K mysql mysql -- - 42 S 0 2% mysqld
1 4.12s 1.01s 0K -592K 4.0G 0K root root -- - 1 D 0 1% systemd
370 3.66s 0.00s 0K 0K 1.7G 0K root root -- - 1 D 0 1% loop9
890 2.82s 0.36s 0K 20K 4.0G 0K redis redis -- - 4 D 0 1% redis-server
442 2.44s 0.57s 0K 384K 3.4G 0K messageb messageb -- - 1 D 0 1% dbus-daemon
874 2.05s 0.85s 0K 800K 3.5G 12K www-data www-data -- - 1 D 0 0% nginx
10 1.71s 0.00s 0K 0K 0K 0K root root -- - 1 S 0 0% ksoftirqd/0
902 0.70s 0.28s 0K -668K 808.1M 0K www-data www-data -- - 1 D 0 0% php-fpm7.4
935 0.62s 0.33s 0K 364K 710.7M 0K www-data www-data -- - 1 D 0 0% php-fpm7.4
13040 0.82s 0.12s 0K 136K 729.5M 0K www-data www-data -- - 1 D 0 0% php-fpm7.4
13041 0.78s 0.16s 0K 788K 712.9M 0K www-data www-data -- - 1 D 0 0% php-fpm7.4
2836 0.61s 0.33s 0K 812K 717.2M 0K www-data www-data -- - 1 D 0 0% php-fpm7.4
437 0.72s 0.00s 0K -452K 848.4M 0K root root -- - 3 S 0 0% accounts-daemo
852 0.53s 0.00s 0K 1324K 573.4M 0K root root -- - 1 S 0 0% php-fpm7.4
849 0.43s 0.05s 0K -464K 482.4M 0K root root -- - 1 S 0 0% php-fpm7.0
10995 0.44s 0.00s 0K 0K 0K 0K root root -- - 1 I 0 0% kworker/0:2-ev
850 0.37s 0.01s 0K -232K 391.9M 0K root root -- - 1 D 0 0% php-fpm7.1
94 0.20s 0.00s 0K 0K 0K 0K root root -- - 1 I 0 0% kworker/0:1H-k
464 0.16s 0.03s 0K 396K 224.8M 0K root root -- - 1 S 0 0% systemd-logind
11 0.16s 0.00s 0K 0K 0K 0K root root -- - 1 I 0 0% rcu_sched
8256 0.05s 0.04s 0K 0K 144K 12K root root -- - 1 R 0 0% atop
846 0.06s 0.00s 0K 572K 28828K 0K root root -- - 1 S 0 0% cron
654 0.02s 0.00s 0K -568K 19484K 0K systemd- systemd- -- - 1 S 0 0% systemd-resolv
369 0.01s 0.00s 0K 688K 7060K 0K root root -- - 1 S 0 0% systemd-udevd
399 0.00s 0.01s 0K -304K 18312K 0K systemd- systemd- -- - 2 S 0 0% systemd-timesy
12 0.01s 0.00s 0K 0K 0K 0K root root -- - 1 S 0 0% migration/0
25 0.00s 0.01s 0K 0K 0K 0K root root -- - 1 S 0 0% khugepaged
454 0.00s 0.00s 0K 0K 0K 0K root root -- - 1 S 0 0% networkd-dispa
313 0.00s 0.00s 0K 4K 0K 0K root root -- - 1 S 0 0% systemd-journa
877 0.00s 0.00s 0K 0K 0K 0K root root -- - 1 S 0 0% sshd
452 0.00s 0.00s 0K 0K 0K 0K root root -- - 11 S 0 0% lxcfs
623 0.00s 0.00s 0K 0K 0K 0K root root -- - 4 S 0 0% dhclient
859 0.00s 0.00s 0K 4K 0K 0K root daemon -- - 1 S 0 0% atd
867 0.00s 0.00s 0K 0K 0K 0K root root -- - 1 S 0 0% agetty
440 0.00s 0.00s 0K 0K 0K 0K root root -- - 1 S 0 0% atopacctd
869 0.00s 0.00s 0K 0K 0K 0K root root -- - 1 S 0 0% agetty
459 0.00s 0.00s 0K -500K 2788K 0K syslog syslog -- - 4 S 0 0% rsyslogd
896 0.00s 0.00s 0K 0K 0K 0K www-data www-data -- - 1 S 0 0% php-fpm7.0
897 0.00s 0.00s 0K 0K 0K 0K www-data www-data -- - 1 S 0 0% php-fpm7.0
906 0.00s 0.00s 0K 0K 0K 0K www-data www-data -- - 1 S 0 0% php-fpm7.1
907 0.00s 0.00s 0K 0K 0K 0K www-data www-data -- - 1 S 0 0% php-fpm7.1
13044 0.00s 0.00s 7036K 792K 6240K 0K root root N- - 1 D 0 0% cron
511 0.00s 0.00s 0K 0K 0K 0K root root -- - 3 S 0 0% polkitd
2 0.00s 0.00s 0K 0K 0K 0K root root -- - 1 S 0 0% kthreadd
Is my only fix to give my server more memory?
This was an interesting outage. Usually I just restart the aws instance and I am good to go. For some reason this time after restarting it still wouldn't run. I saw NGNX was not running so I tried to start that and it failed... Noticed apache somehow started and was using port 80 so I had to kill apache and then restart NGINX. Was a fun morning :-)
