-

Sun, 01 May 2022 09:15:16 +0200

adminsforum@hubzilla.org

RockyIII wrote the following post Sun, 01 May 2022 09:15:16 +0200

queued Messages

Got serious problems on my Raspi homeserver / HZ hub.
Messages i sent 3 day ago are still queued and don´t get delivered, even private messages to just on recipient don´t get out of the queue.

Fore some strange reason all non ZOT contact seam to get  the messages and marked as „accepted for delivery“ in the delivery report… So is there a problem with the ZOT network also?

If I look as admin in to   /admin/  i see over 4000 Message queued

This happened before but his time things seam to be staged

Is there a way to kill this process for my hub?
____

This is a wall post so I hope it will not be queued by my hub.

show all 26 comments

Sun, 01 May 2022 10:30:28 +0200

RockyIII

rocky3@im.allmendenetz.de

@Hans could you please visit my site / wall and check if all the messages you see there reaches you hub also... this would be interesting to know

Sun, 01 May 2022 10:30:30 +0200

Aksel Kies Linc @Hz

aksel@social.aksel.at

@RockyIII
which php version do you use? and which web server?

Sun, 01 May 2022 10:36:02 +0200

RockyIII

rocky3@im.allmendenetz.de

@Hans could you please visit my site / wall and check if all the messages you see there reaches you hub also... this would be interesting to know

@Aksel Kies Linc @Hz

HZ 6.0
PHP-fpm 7.3
Nginx
Mariadb

__

this is a wall post

Sun, 01 May 2022 11:28:05 +0200

RockyIII

rocky3@im.allmendenetz.de

THANKS @Hans

so you mean

max_input_time
and
max_execution_time

in /etc/php/7.3/fpm/php.ini ??

Sun, 01 May 2022 11:43:41 +0200

Aksel Kies Linc @Hz

aksel@social.aksel.at

Maybe check also max children:
https://chrismoore.ca/2018/10/finding-the-correct-pm-max-children-settings-for-php-fpm/

Sun, 01 May 2022 11:47:25 +0200

Aksel Kies Linc @Hz

aksel@social.aksel.at

Plus: For php error logging you can edit the .htconfig.php:

// Uncomment the following 4 lines to turn on PHP error logging.
//error_reporting(E_ERROR | E_WARNING | E_PARSE ); 
//ini_set('error_log','php.out'); 
//ini_set('log_errors','1'); 
//ini_set('display_errors', '0');

and watch php.out in hubzilla root dir.

Mon, 02 May 2022 10:19:38 +0200

RockyIII

rocky3@im.allmendenetz.de

Messages are still stuck in the queue

I did edit php.ini and also max children but sill get warning link this:

WARNING: [pool hubzilla] child 1596 said into stderr: "

What else could i try ?

Thanks

Mon, 02 May 2022 10:33:56 +0200

RockyIII

rocky3@im.allmendenetz.de

before i did edit php.ini i found things like this in

/var/log/nginx/

b'\t/usr/share/ssowat/access.lua:23: in function </usr/share/ssowat/access.lua:1>, client: 17.121.xxx.xxx, server: hub.domian.de, request: "GET /view/js/main.js?v=6.0 HTTP/1.1", host: "hub.domian.de", referrer: "https://hub.domain.de/gallery/user"'
b"2022/04/30 22:45:30 [error] 1660#1660: *18054 lua entry thread aborted: runtime error: /usr/share/ssowat/config.lua:28: bad argument #1 to 'sub' (string expected, got table)"
b'stack traceback:'
b'coroutine 0:'
b"\t[C]: in function 'sub'"
b"\t/usr/share/ssowat/config.lua:28: in function 'update_language'"
b"\t/usr/share/ssowat/config.lua:39: in function 'get_config'"
b"\t/usr/share/ssowat/helpers.lua:23: in function 'refresh_config'"

this Warings are gone

Mon, 02 May 2022 10:56:06 +0200

RockyIII

rocky3@im.allmendenetz.de

@Hans OK

but on my yunohost this lines were uncomment by default

// PHP error logging setup
// Before doing this ensure that the webserver has permission
// to create and write to php.out in the top level Red directory,
// or change the name (below) to a file/path where this is allowed.

// Uncomment the following 4 lines to turn on PHP error logging.
error_reporting(E_ERROR | E_WARNING | E_PARSE ); 
ini_set('error_log','php.log'); 
ini_set('log_errors','1'); 
ini_set('display_errors', '0');
App::$config['system']['addon'] = 'ldapauth';

Mon, 02 May 2022 11:40:58 +0200

RockyIII

rocky3@im.allmendenetz.de

@Hans

the php.log file besides .htconfig.php inside the HZ folder is over 700.000 KB
can´t open it
just created a new php.log file and replaced it with the old one... let´s see

Mon, 02 May 2022 11:50:11 +0200

Aksel Kies Linc @Hz

aksel@social.aksel.at

@RockyIII
withtail -50 php.out you can see the last 50 lines of the file

Mon, 02 May 2022 12:22:59 +0200

RockyIII

rocky3@im.allmendenetz.de

php.out does not exist - where could i find it?

sudo tail -50 /var/www/hubzilla/php.out

tail: cannot open '/var/www/hubzilla/php.out' for reading: No such file or directory

-
sudo tail -50 /var/www/hubzilla/php.log

gives me nothing special just thinks like:

[02-May-2022 00:00:08 Europe/Berlin] PHP Warning: Invalid argument supplied for foreach() in /var/www/hubzilla/include
/text.php on line 3471

Mon, 02 May 2022 13:05:43 +0200

Aksel Kies Linc @Hz

aksel@social.aksel.at

@RockyIII
Ah, yes! yunohost seems to change the log file name from php.out to php.log.

Mon, 02 May 2022 13:59:36 +0200

RockyIII

rocky3@im.allmendenetz.de

THANKS @Aksel Kies Linc @Hz

What could i try next?

Mon, 02 May 2022 14:59:02 +0200

Aksel Kies Linc @Hz

aksel@social.aksel.at

@RockyIII Unfortunally my knowlegde ends here.

Maybe Mario can help here or at framagit.

Mon, 02 May 2022 18:55:50 +0200

RockyIII

rocky3@im.allmendenetz.de

i increased pm.max_children even more...

if that does not help ...

if i set up the contact rols for all contacts that way for a while that no new posts will be excepted would that be a chance to clear up the queue ? @Mario Vavti

Mon, 02 May 2022 23:21:33 +0200

oRx-Qx pirateradio

michal@node9.org

I have also 9000 queue messages. All zot servers. Msgs not delivered between channels on the same site, although 2 days before they were arriving. No change, not even restart.

Tue, 03 May 2022 08:17:05 +0200

RockyIII

rocky3@im.allmendenetz.de

@oRx-Qx pirateradio

All zot servers

same on my server. what could that mean?

Which directory_server do you use?

Tue, 03 May 2022 09:01:40 +0200

Mario Vavti

mario@hub.somaton.com

Are you using the queueworker addon?
What is your Queue Threshold setting?

Tue, 03 May 2022 09:23:14 +0200

RockyIII

rocky3@im.allmendenetz.de

@Mario Vavti
using "queueworker - 0.8.1"
but
where do i set "Queue Threshold" ?

Tue, 03 May 2022 09:42:29 +0200

RockyIII

rocky3@im.allmendenetz.de

/admin/site/

Queue Threshold = 3000

Tue, 03 May 2022 09:55:43 +0200

Mario Vavti

mario@hub.somaton.com

Well, if your queue is larger than your threshold, then new messages will be queued automaticallly (without trying to deliver at all). You could try to raise your threshold...

Tue, 03 May 2022 10:14:50 +0200

RockyIII

rocky3@im.allmendenetz.de

ok right now i still have 2900 messages in the queue and messages don´t get delivered to zot sites - even messages older than 3 days.
What could i try to get this messages out?

Tue, 03 May 2022 11:36:40 +0200

Mario Vavti

mario@hub.somaton.com

Your messages do arrive here @Hans

Tue, 03 May 2022 12:16:34 +0200

oRx-Qx pirateradio

michal@node9.org

My comment above was sent from regular timeline and arrived here.

Once I debugged queue delivery and I wandered how it can even deliver something. My outcome is when there is unsuccessful direct delivery and things go to queue everything goes bad. My impression is that one offline hub with high traffic means a lot of queue items. They are selected in delivery attempts in random order, so there is high probability it will be item for offline site again. With unsuccessful delivery again all items are shifted to be delivered one hour later and so on. So with queue above certain size is not delivering and new messages start being queued and not delivered directly so even to live sites the delivery stays queued. Regardless of queue settings or queworker plugin, the queue delivery is very slow and takes many days to recover. Of course the dead site with many items has to be cleared via expert mode.

Wed, 04 May 2022 08:09:03 +0200

RockyIII

rocky3@im.allmendenetz.de

@oRx-Qx pirateradio

Thanks for this link /admin/queue?expert=1

I deleted some items and now thinks seam to work better again on my hub

I still don´t understand why messages only forZOT servers where marked "queued" in the delivery reports