-

2025-04-05 00:54:31

adminsforum@hubzilla.org

Marshall Sutherland wrote the following post Sat, 05 Apr 2025 00:55:25 +0200

@!Hubzilla Support Forum
It seems like repeats are ignore filters. I added a particular tag to my filters recently, yet I saw a repeat containing that tag just now.

show all 7 comments

2025-04-05 12:39:33

Der Pepe (Hubzilla) ⁂ ⚝

pepecyb@hub.hubzilla.hu

¯\_(ツ)_/¯ I hadn't noticed it yet because my filter lists are pretty empty.

2025-04-05 16:16:13

Marshall Sutherland

dwatney@hub.farthinghalearms.com

This is the first entry I have added that wasn't a language filter.

2025-04-05 17:54:44

Der Pepe (Hubzilla) ⁂ ⚝

pepecyb@hub.hubzilla.hu

@Marshall Sutherland

Did you enter that in the channel filter or in individual contact filters?

Is it the only filtering, or are there different filters in there line by line?

Maybe it's because there are pure text filter expressions and RE or special filters mixed in there. There are problems there. I think as soon as there is an RE, the word filters are ignored... or something like that.

@Jupiter Rowland has already explained this several times and I keep forgetting. Maybe he would like to explain it again here 🙏 (and then I'll remember it so I never have to ask again 😉😂).

2025-04-05 17:58:23

Mario Vavti

mario@hub.somaton.com

Which filter is that?

2025-04-05 18:54:43

Marshall Sutherland

dwatney@hub.farthinghalearms.com

It is in the channel filter. I have probably a dozen "lang=XX" lines and the new "#sometag" line.

2025-04-05 19:01:23

Der Pepe (Hubzilla) ⁂ ⚝

pepecyb@hub.hubzilla.hu

@Marshall Sutherland

I hope @Jupiter Rowland will comment on when filters work together and when they don't.

Otherwise, I'll try to find the last post on the subject.

2025-04-05 20:19:23

Jupiter Rowland

jupiter_rowland@hub.netzgemeinde.eu

@Der Pepe (Hubzilla) ⁂ ⚝ I don't know for a fact how Hubzilla's filters work (and malfunction). I only have theories. They came to exist when I tried to build an advanced filter whitelist of a kind that had never been considered, much less tried outside of very specific and basic use-cases under lab conditions.

The filter probably doesn't use both keyword lines and filter syntax lines together

The idea was to only let posts through when they contain at least one out of a choice of keywords, but to let comments and private messages through regardless.

I think I was actually told that a filter cannot contain both keyword lines like so...

Foo
Bar
Baz

...and filter syntax lines like so...

?body ~= Qux
?body ~= Quux
?item_private == 0
?item_thread_top == 1

This means that the filter parser would only uses either keyword lines or filter syntax lines.

According to what I believe having observed, the filter parser uses whatever comes last. If the last line is a keyword line, the filter parser uses only keyword lines. If the last line is a filter syntax line, the filter parser uses only filter syntax lines.

This, in turn, means:

Either the filter parser uses all lines of whatever comes last and ignores everything else.

~~Keyword~~
~~Keyword~~
Filter Syntax
Filter Syntax
~~Keyword~~
~~Keyword~~
Filter Syntax <- this line defines that filter syntax lines are used
Filter Syntax

Or whenever the filter parser comes across the other kind of filter line, it not only switches to that kind of filter line, but it forgets all lines above.

~~Keyword~~
~~Keyword~~
~~Filter Syntax~~ <- filter forgot all previous lines
~~Filter Syntax~~
~~Keyword~~ <- filter forgot all previous lines
~~Keyword~~
Filter Syntax <- filter forgot all previous lines
Filter Syntax

In other words: If you have

?verb == Announce

in your blacklist to get rid of repeats, but the same blacklist ends with a keyword line, the anti-repeat filter syntax line is not parsed.

Filter lines are connected with OR except for whitelists in filter syntax mode which are connected with AND

To make matters worse, filter lines are not always connected with OR.

AFAIK, the blacklist uses OR. But the whitelist only uses OR when keyword lines are parsed. When filter syntax lines are parsed in the whitelist, they're connected with AND. This would be another reason, at least in the case of the whitelist, why keyword lines and filter syntax lines aren't parsed together.

Regex don't work in filter syntax lines

This is something I know for a fact because I was told about it:

Filter syntax lines don't understand regular expressions.

/\b([#]?masto[don]?)\b/ works.

?body ~= /\b([#]?masto[don]?)\b/ doesn't.

It probably doesn't matter in your case. I just wanted to mention it.

I've partially explained this in Hubzilla's bug tracker on Framagit in a comment on an issue by jayrope.

Also, I myself have submitted a feature request for being able to choose whether the lines in a particular filter list shall be connected with OR or AND. This request includes filter lists always parsing both keyword lines and filter syntax lines together.