software optimization (php mysql apache)

kayoo123

Good morning, everyone,
After putting our website on line: https://deepin-fr.org

We notice some delays (100% CPU load) when many users are present.
Many error in error.log :

[Mon Dec 18 22:07:31.880999 2017] [mpm_event:error] [pid 22117:tid 140132269020352] AH: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
[Mon Dec 18 22:07:32.882159 2017] [mpm_event:error] [pid 22117:tid 140132269020352] AH: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
[Mon Dec 18 22:07:33.883321 2017] [mpm_event:error] [pid 22117:tid 140132269020352] AH: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
[Mon Dec 18 22:07:34.884473 2017] [mpm_event:error] [pid 22117:tid 140132269020352] AH: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
...

All our configurations are by default and host by the same server...
We would like to optimize all this:

OS: Debian GNU/Linux 9.3 (stretch)
RAM: 4Gb
CPU: Intel (R) Atom (TM) CPU C2338 @ 1.74GHz CPU
Apache: 2.4.25
PHP: 7.0.19 FastCGI / MPM EVENT
MariaDB: 10.1.26
Flarum: 0.1.0-beta. 7

Here are our configuration files:

module apache

grep -v ^# /etc/apache2/mods-available/mpm_event.conf

<IfModule mpm_event_module>
        StartServers                     2
        MinSpareThreads          25
        MaxSpareThreads          75
        ThreadLimit                      64
        ThreadsPerChild          25
        MaxRequestWorkers         150
        MaxConnectionsPerChild   0
</IfModule>

PHP

grep -v ^\; /etc/php/7.0/fpm/pool. d/www. conf |uniq

[www]
user = www-data
group = wwww-data
listen = /run/php/php7.0-fpm. sock
listen. owner = www-data
listen. group = www-data
pm = dynamic
pm.max_children = 5
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3

MariaDB

grep -v ^# /etc/mysql/mariadb.conf.d/50-server.cnf |uniq

[server]
[mysqld]
user            = mysql
pid-file        = /var/run/mysqld/mysqld.pid
socket          = /var/run/mysqld/mysqld.sock
port            = 3306
basedir         = /usr
datadir         = /var/lib/mysql
tmpdir          = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
bind-address            = 127.0.0.1
key_buffer_size         = 16M
max_allowed_packet      = 16M
thread_stack            = 192K
thread_cache_size       = 8
myisam_recover_options  = BACKUP
query_cache_limit       = 1M
query_cache_size        = 16M
log_error = /var/log/mysql/error.log
expire_logs_days        = 10
max_binlog_size   = 100M
character-set-server  = utf8mb4
collation-server      = utf8mb4_general_ci
[embedded]
[mariadb]
[mariadb-10.1]

Any help would be more than appreciated ?

kayoo123

And my php.ini :

grep -v ^\; /etc/php/7.0/fpm/php.ini |grep \

engine = On
short_open_tag = Off
precision = 14
output_buffering = 4096
zlib.output_compression = Off
implicit_flush = Off
unserialize_callback_func =
serialize_precision = 17
disable_functions = pcntl_alarm,pcntl_fork,pcntl_waitpid,pcntl_wait,pcntl_wifexited,pcntl_wifstopped,pcntl_wifsignaled,pcntl_wifcontinued,pcntl_wexitstatus,pcntl_wtermsig,pcntl_wstopsig,pcntl_signal,pcntl_signal_dispatch,pcntl_get_last_error,pcntl_strerror,pcntl_sigprocmask,pcntl_sigwaitinfo,pcntl_sigtimedwait,pcntl_exec,pcntl_getpriority,pcntl_setpriority,
disable_classes =
zend.enable_gc = On
expose_php = Off
max_execution_time = 30
max_input_time = 60
memory_limit = 256M
error_reporting = E_ALL & ~E_DEPRECATED & ~E_STRICT
display_errors = Off
display_startup_errors = Off
log_errors = On
log_errors_max_len = 1024
ignore_repeated_errors = Off
ignore_repeated_source = Off
report_memleaks = On
track_errors = Off
html_errors = On
variables_order = "GPCS"
request_order = "GP"
register_argc_argv = Off
auto_globals_jit = On
post_max_size = 8M
auto_prepend_file =
auto_append_file =
default_mimetype = "text/html"
default_charset = "UTF-8"
doc_root =
user_dir =
enable_dl = Off
file_uploads = On
upload_max_filesize = 2M
max_file_uploads = 20
allow_url_fopen = On
allow_url_include = Off
default_socket_timeout = 60
[CLI Server]
cli_server.color = On
pdo_mysql.cache_size = 2000
[mail function]
SMTP = localhost
smtp_port = 25
mail.add_x_header = On
sql.safe_mode = Off
odbc.allow_persistent = On
odbc.check_persistent = On
odbc.max_persistent = -1
odbc.max_links = -1
odbc.defaultlrl = 4096
odbc.defaultbinmode = 1
ibase.allow_persistent = 1
ibase.max_persistent = -1
ibase.max_links = -1
ibase.timestampformat = "%Y-%m-%d %H:%M:%S"
ibase.dateformat = "%Y-%m-%d"
ibase.timeformat = "%H:%M:%S"
mysqli.max_persistent = -1
mysqli.allow_persistent = On
mysqli.max_links = -1
mysqli.cache_size = 2000
mysqli.default_port = 3306
mysqli.default_socket =
mysqli.default_host =
mysqli.default_user =
mysqli.default_pw =
mysqli.reconnect = Off
mysqlnd.collect_statistics = On
mysqlnd.collect_memory_statistics = Off
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0
bcmath.scale = 0
session.save_handler = files
session.use_strict_mode = 0
session.use_cookies = 1
session.use_only_cookies = 1
session.name = PHPSESSID
session.auto_start = 0
session.cookie_lifetime = 0
session.cookie_path = /
session.cookie_domain =
session.cookie_httponly =
session.serialize_handler = php
session.gc_probability = 0
session.gc_divisor = 1000
session.gc_maxlifetime = 1440
session.referer_check =
session.cache_limiter = nocache
session.cache_expire = 180
session.use_trans_sid = 0
session.hash_function = 0
session.hash_bits_per_character = 5
url_rewriter.tags = "a=href,area=href,frame=src,input=src,form=fakeentry"
zend.assertions = -1
tidy.clean_output = Off
soap.wsdl_cache_limit = 5
ldap.max_links = -1

Sanguine

Is Mariadb or PHP the bottleneck? Check with top.

Configure php-fpm and mariadb’s slow log and set them to 2s. This will output slow queries and a php stacktrace of the function that likely took a long time.

kayoo123

I've put module apache config file :

module apache

grep -v ^# /etc/apache2/mods-available/mpm_event.conf

<IfModule mpm_event_module>
        StartServers                     2
        MinSpareThreads          25
        MaxSpareThreads          75
        ThreadLimit                      64
        ThreadsPerChild          25
        MaxRequestWorkers         150
        MaxConnectionsPerChild   0
</IfModule>

Sanguine Is Mariadb or PHP the bottleneck? Check with top.

It's apache which use 99% CPU when this happen

Sanguine Configure php-fpm and mariadb’s slow log and set them to 2s.

I will try to do this

JoshyPHP You're gonna want to increase key_buffer_size to 256 MB - 1 GB depending on the size of your indexes.

up to 256 for now ^^

JoshyPHP

You're gonna want to increase key_buffer_size to 256 MB - 1 GB depending on the size of your indexes.

Sanguine

kayoo123 It's apache which use 99% CPU when this happen

Are you sure you are not using mod_php (instead of php-fpm) ? If Apache is maxing out, then it is no use to optimize PHP-FPM or MariaDB.

You can monitor what Apache is doing by enabling the /server-status block in the config.

CPU: Intel (R) Atom

They are not very good for server applications. But again, if Apache is the bottleneck, your problem is possibly not related to a capacity shortage.

Sanguine

We should turn this thread into proper documentation right away! Looking into it now..

kayoo123

Yes, I'm using FastCGI

I will enable /server-status to see what's going on

kayoo123

This is my htop

Sanguine

kayoo123 Anything in the apache error.log? Otherwise, we need the server-status (screenshot?) or a strace dump of one busy apache process to say anything useful (strace -fp <pid of apache> -o apache.log).

luceos

Do you have debug disabled in your config.php? And what extensions do you have installed, output of php flarum info would help.

kayoo123

Shame on me, I have trouble to access to my server-status ?
Will post it when succeded...

I've made some change of my MPM_EVENT modul and every seems stabilize...
I have to wait a day to see if everything is OK.

I follow this guide to understand parameters : https://www.tectut.com/2016/04/workaround-for-scoreboard-is-full-not-at-maxrequestworkers/

The true Life Saver was to set a bigger ServerLimit.
https://httpd.apache.org/docs/2.4/en/mod/mpm_common.html#serverlimit

<IfModule mpm_event_module>
        StartServers             3
        ServerLimit              64
        MinSpareThreads          25
        MaxSpareThreads          75
        ThreadLimit              64
        ThreadsPerChild          25
        MaxRequestWorkers        400
        MaxConnectionsPerChild   0
        GracefulShutdownTimeout 5
</IfModule>
MaxKeepAliveRequests 20

If not, maybe I shoud use FPM with WORKER-mode instead of EVENT.
Or directly with MOD_PHP, I read that on small website with simple access without pool, it's faster because it's already loaded on every request...

I will keep you updated :-)

luceos Do you have debug disabled in your config.php? And what extensions do you have installed, output of php flarum info would help.

Yes it's disabled and this is my extension list :
php flarum info

Flarum core 0.1.0-beta.7
PHP 7.0.19-1
Loaded extensions: Core, date, libxml, openssl, pcre, zlib, filter, hash, pcntl, Reflection, SPL, session, standard, mysqlnd, PDO, xml, bz2, calendar, ctype, curl, dom, mbstring, fileinfo, ftp, gd, geoip, gettext, iconv, json, exif, mcrypt, mysqli, pdo_mysql, Phar, posix, readline, shmop, SimpleXML, sockets, sysvmsg, sysvsem, sysvshm, tokenizer, wddx, xmlreader, xmlwriter, xsl, zip, Zend OPcache
EXT flarum-approval v0.1.0-beta.7
EXT flarum-bbcode v0.1.0-beta.5
EXT flarum-emoji v0.1.0-beta.6
EXT clarkwinkelmann-emojionearea 0.1.2
EXT flarum-english v0.1.0-beta.7
EXT flarum-auth-facebook v0.1.0-beta.6
EXT flagrow-upload 0.6.0
EXT flarum-flags v0.1.0-beta.7
EXT milescellar-french v1.1.0
EXT flarum-auth-github v0.1.0-beta.6
EXT johnhearfield-oauth-google v0.1.0-beta.8.1
EXT sijad-recaptcha 0.0.2
EXT flarum-likes v0.1.0-beta.6
EXT sijad-links 0.1.0-beta.6
EXT flarum-lock v0.1.0-beta.7
EXT flarum-markdown v0.1.0-beta.5
EXT xengine-markdown-editor 1.3.1
EXT s9e-mediaembed 0.3.2
EXT flarum-mentions v0.1.0-beta.7
EXT sijad-pages 0.1.0-beta.3
EXT flarum-sticky v0.1.0-beta.7
EXT flarum-subscriptions v0.1.0-beta.6
EXT flarum-suspend v0.1.0-beta.7
EXT amaurycarrade-syndication v0.1.5
EXT flarum-tags v0.1.0-beta.8
EXT avatar4eg-users-list 0.1.1
Base URL: https://deepin-fr.org
Installation path: /DATA/www/deepin-fr.org/public_html