Errors in discussions list

sylv38

Hi all !
I have a strange problem : when I scroll discussions, it seems that the xml can't be loaded. The error returned is this one :

GET https://www.xxxxxx.fr/api/discussions

InvalidArgumentException: Cannot load XML: xmlParseCharRef: invalid xmlChar value 0
 in /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/s9e/text-formatter/src/Renderer.php:46
Stack trace:
#0 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/s9e/text-formatter/src/Renderers/PHP.php(303): s9e\TextFormatter\Renderer->loadXML('<r><p>Ce soir, ...')
#1 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/s9e/text-formatter/src/Renderer.php(66): s9e\TextFormatter\Renderers\PHP->renderRichText('<r><p>Ce soir, ...')
#2 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/flarum/core/src/Formatter/Formatter.php(113): s9e\TextFormatter\Renderer->render('<r><p>Ce soir, ...')
#3 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/flarum/core/src/Post/CommentPost.php(172): Flarum\Formatter\Formatter->render('<r><p>Ce soir, ...', Object(Flarum\Post\CommentPost), Object(Laminas\Diactoros\ServerRequest))
#4 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/flarum/core/src/Api/Serializer/BasicPostSerializer.php(44): Flarum\Post\CommentPost->formatContent(Object(Laminas\Diactoros\ServerRequest))
#5 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/flarum/core/src/Api/Serializer/AbstractSerializer.php(88): Flarum\Api\Serializer\BasicPostSerializer->getDefaultAttributes(Object(Flarum\Post\CommentPost))
#6 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/tobscure/json-api/src/Resource.php(169): Flarum\Api\Serializer\AbstractSerializer->getAttributes(Object(Flarum\Post\CommentPost), NULL)
#7 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/tobscure/json-api/src/Resource.php(75): Tobscure\JsonApi\Resource->getAttributes()
#8 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/tobscure/json-api/src/Document.php(187): Tobscure\JsonApi\Resource->toArray()
#9 [internal function]: Tobscure\JsonApi\Document->Tobscure\JsonApi\{closure}(Object(Tobscure\JsonApi\Resource))
#10 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/tobscure/json-api/src/Document.php(186): array_map(Object(Closure), Array)
#11 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/tobscure/json-api/src/Document.php(224): Tobscure\JsonApi\Document->toArray()
#12 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/flarum/core/src/Api/JsonApiResponse.php(27): Tobscure\JsonApi\Document->jsonSerialize()
#13 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/flarum/core/src/Api/Controller/AbstractSerializeController.php(129): Flarum\Api\JsonApiResponse->__construct(Object(Tobscure\JsonApi\Document))
#14 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/flarum/core/src/Http/RouteHandlerFactory.php(41): Flarum\Api\Controller\AbstractSerializeController->handle(Object(Laminas\Diactoros\ServerRequest))
#15 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/flarum/core/src/Http/Middleware/ExecuteRoute.php(27): Flarum\Http\RouteHandlerFactory->Flarum\Http\{closure}(Object(Laminas\Diactoros\ServerRequest), Array)
#16 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/laminas/laminas-stratigility/src/Next.php(61): Flarum\Http\Middleware\ExecuteRoute->process(Object(Laminas\Diactoros\ServerRequest), Object(Closure))
#17 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/flarum/core/src/Api/Middleware/ThrottleApi.php(33): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#18 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/laminas/laminas-stratigility/src/Next.php(61): Flarum\Api\Middleware\ThrottleApi->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#19 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/flarum/core/src/Http/Middleware/CheckCsrfToken.php(36): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#20 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/laminas/laminas-stratigility/src/Next.php(61): Flarum\Http\Middleware\CheckCsrfToken->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#21 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/flarum/core/src/Http/Middleware/ResolveRoute.php(67): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#22 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/laminas/laminas-stratigility/src/Next.php(61): Flarum\Http\Middleware\ResolveRoute->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#23 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/flarum/core/src/Http/Middleware/SetLocale.php(51): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#24 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/laminas/laminas-stratigility/src/Next.php(61): Flarum\Http\Middleware\SetLocale->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#25 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/flarum/core/src/Http/Middleware/AuthenticateWithHeader.php(56): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#26 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/laminas/laminas-stratigility/src/Next.php(61): Flarum\Http\Middleware\AuthenticateWithHeader->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#27 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/flarum/core/src/Http/Middleware/AuthenticateWithSession.php(31): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#28 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/laminas/laminas-stratigility/src/Next.php(61): Flarum\Http\Middleware\AuthenticateWithSession->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#29 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/flarum/core/src/Http/Middleware/RememberFromCookie.php(52): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#30 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/laminas/laminas-stratigility/src/Next.php(61): Flarum\Http\Middleware\RememberFromCookie->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#31 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/flarum/core/src/Http/Middleware/StartSession.php(61): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#32 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/laminas/laminas-stratigility/src/Next.php(61): Flarum\Http\Middleware\StartSession->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#33 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/flarum/core/src/Api/Middleware/FakeHttpMethods.php(29): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#34 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/laminas/laminas-stratigility/src/Next.php(61): Flarum\Api\Middleware\FakeHttpMethods->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#35 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/flarum/core/src/Http/Middleware/ParseJsonBody.php(28): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#36 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/laminas/laminas-stratigility/src/Next.php(61): Flarum\Http\Middleware\ParseJsonBody->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#37 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/flarum/core/src/Http/Middleware/HandleErrors.php(57): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#38 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/laminas/laminas-stratigility/src/Next.php(61): Flarum\Http\Middleware\HandleErrors->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#39 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/flarum/core/src/Http/Middleware/InjectActorReference.php(25): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#40 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/laminas/laminas-stratigility/src/Next.php(61): Flarum\Http\Middleware\InjectActorReference->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#41 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/laminas/laminas-stratigility/src/MiddlewarePipe.php(84): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#42 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/middlewares/request-handler/src/RequestHandler.php(84): Laminas\Stratigility\MiddlewarePipe->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#43 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/laminas/laminas-stratigility/src/Next.php(61): Middlewares\RequestHandler->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#44 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/middlewares/base-path-router/src/BasePathRouter.php(101): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#45 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/laminas/laminas-stratigility/src/Next.php(61): Middlewares\BasePathRouter->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#46 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/laminas/laminas-stratigility/src/Middleware/OriginalMessages.php(42): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#47 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/laminas/laminas-stratigility/src/Next.php(61): Laminas\Stratigility\Middleware\OriginalMessages->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#48 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/middlewares/base-path/src/BasePath.php(73): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#49 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/laminas/laminas-stratigility/src/Next.php(61): Middlewares\BasePath->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#50 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/flarum/core/src/Http/Middleware/ProcessIp.php(24): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#51 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/laminas/laminas-stratigility/src/Next.php(61): Flarum\Http\Middleware\ProcessIp->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#52 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/laminas/laminas-stratigility/src/MiddlewarePipe.php(84): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#53 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/laminas/laminas-stratigility/src/MiddlewarePipe.php(73): Laminas\Stratigility\MiddlewarePipe->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\EmptyPipelineHandler))
#54 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/laminas/laminas-httphandlerrunner/src/RequestHandlerRunner.php(96): Laminas\Stratigility\MiddlewarePipe->handle(Object(Laminas\Diactoros\ServerRequest))
#55 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/vendor/flarum/core/src/Http/Server.php(44): Laminas\HttpHandlerRunner\RequestHandlerRunner->run()
#56 /home/clients/50d3e640d21eed4e2418f41d6fd6963e/beta/public/index.php(26): Flarum\Http\Server->listen()
#57 {main}

The invalid xml char can be different than 0, it depends of the discussion that generates the crash. My installation is this one :

Flarum core 1.0.3
PHP version: 7.4.19
Loaded extensions: Core, date, libxml, openssl, pcre, sqlite3, zlib, bcmath, bz2                                   , calendar, ctype, curl, dom, hash, fileinfo, filter, ftp, gd, gettext, gmp, SPL                                   , iconv, session, intl, json, ldap, mbstring, standard, pcntl, PDO, mysqlnd, pdo                                   _sqlite, pgsql, Phar, posix, Reflection, imap, SimpleXML, soap, sockets, sodium,                                    pdo_mysql, exif, tidy, tokenizer, xml, xmlreader, xmlwriter, xsl, mysqli, geos,                                    imagick, mailparse, mcrypt, mongodb, OAuth, PDFlib, propro, raphf, ssh2, timezo                                   nedb, uploadprogress, xmlrpc, yaz, zip, http, ionCube Loader
+-------------------------------+--------------+--------+
| Flarum Extensions             |              |        |
+-------------------------------+--------------+--------+
| ID                            | Version      | Commit |
+-------------------------------+--------------+--------+
| flarum-tags                   | v1.0.3       |        |
| flarum-flags                  | v1.0.0       |        |
| flarum-suspend                | v1.0.0       |        |
| fof-spamblock                 | 1.0.0        |        |
| flarum-markdown               | v1.0.1       |        |
| fof-formatting                | 1.0.0        |        |
| justoverclock-guestengagement | 1.0.0        |        |
| fof-split                     | 1.0.0        |        |
| fof-upload                    | 1.0.1        |        |
| fof-pages                     | 1.0.1        |        |
| fof-byobu                     | 1.0.0-beta.7 |        |
| fof-share-social              | 1.0.0        |        |
| fof-stopforumspam             | 1.0.0        |        |
| flarum-likes                  | v1.0.0       |        |
| v17development-seo            | v1.8.0       |        |
| rooaaar-lang-french-extended  | dev-master   |        |
| fof-user-directory            | 1.0.1        |        |
| flarum-emoji                  | v1.0.0       |        |
| flarum-pusher                 | v1.0.0       |        |
| qiaeru-lang-french            | v2.0.0       |        |
| flarum-subscriptions          | v1.0.0       |        |
| flarum-sticky                 | v1.0.0       |        |
| flarum-mentions               | v1.0.0       |        |
| flarum-lock                   | v1.0.0       |        |
| flarum-bbcode                 | v1.0.0       |        |
| clarkwinkelmann-emojionearea  | 1.0.0        |        |
| fof-linguist                  | 1.0.0        |        |
| fof-user-bio                  | 1.0.0        |        |
| fof-terms                     | 1.0.0        |        |
| fof-subscribed                | 1.0.0        |        |
| fof-sitemap                   | 1.0.0        |        |
| fof-nightmode                 | 1.0.0        |        |
| fof-links                     | 1.0.0        |        |
| fof-disposable-emails         | 1.0.0        |        |
| fof-merge-discussions         | 1.0.0        |        |
| clarkwinkelmann-circle-groups | 1.0.0        |        |
| fof-polls                     | 1.0.2        |        |
| fof-profile-image-crop        | 1.0.0        |        |
| askvortsov-rich-text          | v2.0.3       |        |
| fof-pretty-mail               | 1.0.1        |        |
+-------------------------------+--------------+--------+

I don't know when the problem first started, if it is linked to 1.0 or not.

Do you know where I can search to fix this big problem ? I tried to change the content of the post where it seems to crash looking at debug info, rename the concerned discussion, but even with an empty content, it crashes the same way, I don't know where there is a bad character.

Thank you all !

clarkwinkelmann

sylv38 are you able to share the full content value (from the database) of the post that starts with <r><p>Ce soir ?

Likely some invalid XML ended up in that field and Flarum can't handle this. This can happen if an extension misbehaves, or maybe there was some file corruption, or maybe there is something that we don't escape properly.

EDIT: if you need to fix this right away, you should be able to set the value to <t></t> and it should render an empty post without any error. A completely empty string isn't a valid TextFormatter content either so that would also crash the rendering.

sylv38

You're a god 😉 I tried to replace with <t></t> and then the discussion could load (I was becoming crazy because I tried with an empty message or just a string without any tag). Some posts are wrong, I have to try to detect the bad character for each one, I think the problem comes from the MyBB migration, the posts are old ones.

In this message there was <E></E> tag, I don't know what it is useful for. But it messed with a &#XXX and a smiley 😉 . Luckily, there are not a lot of posts with it, a simple replace command in database should do the work !

THANK YOU THANK YOU THANK YOU 😃

sylv38

It's me again !

There are a lot of errors in my DB because of MyBB posts... It's always with the & character :

some smileys with &#XXXXXX; syntax used in parenthesis became : &#XXXXXX<E>;)</E>. I tried to correct them all => looks like it's ok.
some & are used alone, I replaced with & => ok
urls like youtube contain the & sign and not & => it crashes the discussion each time there is a post like this... but this time I don't know how to replace them in my DB ???? Is there a magic way, with a regex ???

I don't know why S9E Textformatter crashes now => when I first installed Flarum one year ago, everything worked. It looks like the plugin is more restrictive than before ? Or errors were bypassed ?

sylv38

MyBB old posts are very very messy with Mycode intertwined with flarum tags.
Finally I've done a big regex replace with php as I don't have mysql 8 to do it, searching all urls in posts with & sign and replacing it if necessary. It seems to work now, I parsed a lot of discussions without any crash and looking good. I'm exhausted, I hope there won't be another case I haven't found yet !

luceos

sylv38 welcome to trial and error 🙈

sylv38

luceos thank you 😅

Justoverclock

what if u use the native regex?
https://dev.mysql.com/doc/refman/8.0/en/regexp.html#function_regexp-replace

mysql> SELECT REGEXP_REPLACE('a b c', 'b', 'X');
+-----------------------------------+
| REGEXP_REPLACE('a b c', 'b', 'X') |
+-----------------------------------+
| a X c                             |
+-----------------------------------+
mysql> SELECT REGEXP_REPLACE('abc def ghi', '[a-z]+', 'X', 1, 3);
+----------------------------------------------------+
| REGEXP_REPLACE('abc def ghi', '[a-z]+', 'X', 1, 3) |
+----------------------------------------------------+
| abc def X                                          |
+----------------------------------------------------+

sylv38

Justoverclock Simply because my host use MySQL 5.7 and REGEX_REPLACE only works since MySQLl 8 or MariaDB. Thus, I had to connect to the database via PHP and use PHP functions to transform posts.

Thanks for your suggestion anyway 😉

Justoverclock

sylv38 i'm not expert, so it just a question, is it possible to migrate your db locally to a mysql 8.0, clean and then put again into 5.7?

sylv38

Justoverclock Yes, I could also do that, but my database is big, I'm always afraid when I have to load it to the host server because of timeouts. Working directly on the production server is not a good idea neither... but I made a backup, crossing my fingers not to have to use it ! Anyway, it's not so hard with php : I just had to initialize a mysqli connection and to make my calls in a php file, with only around twenty lines of code and that's all ! I could also write on my screen the results of the transformations using echo before saving it into the database, convenient !