Proposal: Move handling of incoming activities to QueueRunner
One of the Lemmy developers (I think) contacted us over at the
Hubzilla Matrix room with an issue where they never got a response to an activity they sent us. Due to the way Lemmy works, this broke federation between the instances.
This case was a "like" activity for an object on a third instance that was offline, and we traced it to Hubzilla hanging indefinitely when trying to retrieve the object from the non-responsive third server.
This was again caused by a logic bug in Hubzilla where the default timeouts for fetching the remote object was not honoured, so that the fetch would hang forever (or until killed by the php runner.)
I've submitted a
merge request addressing this, but also wanted to discuss some possible further enhancements to this scenario.
Even with the default timeouts properly set, this scenario would hold the connection for more than 30 seconds (potentially for a full minute or more if the server third connects, but don't send a response.)
That's a very long time!
What more is, even though the process will be idle while waiting for the unresponsive server, it occupies a process, whatever memory is allocated to the process, a database connection, and any other resources allocated by the process up until the request to the remote host.
All that is resources that could be better used to respond to other incoming requests instead.
My proposal is to move as much of the processing of incoming activities as we can to a separate process.
After performing a minimal verification that the activity is legitimate and valid, we can queue it up for further processing, return a 202 (Accepted) status to the sender, and free up the process and resources for further incoming requests.
We already have the QueueWorker, that I think can be used for this.
I don't know how much work it will be to make these changes, but I wanted to bring it up for discussion. Does anybody have any objections, or see and potential pitfalls we need to be aware of before trying this?
