data and database size question

Thu, 20 Apr 2023 03:51:23 +0200

adminsforum@hubzilla.org

amal wrote the following post Thu, 20 Apr 2023 03:51:22 +0200

data and database size question

so, basically when I am connecting with some-one on mastodon or diaspora or another hubzilla hub, all the text messages of my "connections/friends/whatever" are basically added to my database, and I am not seeing at those sites with my site being a client? Is this right?
Thus the database site will be HUGE soon if there are moderate number of people having small to moderate number of connections and they are actively exchanging texts? I understand that older messages/texts can be deleted from my server (HOW??) but then users will miss those data in their stream, without any prior notice?
And what about image files? Are they also downloaded to my server (the server owned by others I am using or the server I "own" (webhosting account) ?

show all 7 comments

Thu, 20 Apr 2023 11:52:12 +0200

amal

amal@hort.fan

@od thanks for your input. So
- data from another hubzilla instance/mastodon/diaspora is added to my/our server
- what happens to image files, they are seen from here to "their" server or downloaded to my/our server (and in which folder?)
- how I can limit the database size
- what warnings or messages or user instructions would be appropriate?

Thu, 20 Apr 2023 11:53:15 +0200

amal

amal@hort.fan

@od - how did you solve the problem of VPS crashing?

Thu, 20 Apr 2023 22:15:45 +0200

Mario Vavti

mario@hub.somaton.com

Right, posts you receive from your connections are stored in the DB. Also profile meta data and images are cached. Other images are just linked. Allthough you can install the photocache addon which will cache photos at your hub which is good for your privacy but will of course use more memory. You can limit your database size by setting up content expiration.

Fri, 21 Apr 2023 00:37:09 +0200

amal

amal@hort.fan

@Mario Vavti  @Maria Karlsen @od

Let us consider that I am the owner of an instance. So, user-uploaded photos from other instances are linked to my instance or stored in the files/folders of my instance? - what is the exact location of this folder/file? How can the size of this file/folder be limited and/or content expiration be set? Any detailed help file please?

Regarding content expiration, just similar to we have privacy settings when an user posts texts (or other stuffs) how do we let him know that his content can expire in this instance and/or other instances (in contrast to comment system of fixed cms es like WP, Drupal or even the regular social networks where comments etc never expire)?

As of now content expiration expires posts only or comments also or both?

How to limit (where is the administrative interface or what line to add to config / htconfig file)  the quota of data - text or photo only or text+photo of an user so that user cannot upload photo further unless he backups (and deletes) some of his photo ? Do we show him explicit and friendly warning message in such cases ?

How can all user uploaded photos be basically stored in a paid service by the instance owner in some image hosts / webhosts other than those owned by GAFAM - what are these and any plugin that makes this task a breeze? There was one such service (or two) but I am forgetting the names and necessary help instructions - so any one who can throw light on this or contribute code (merge request?) will be welcome!

PS : so it appears the main or one of the main  obstacle of a censorship-free decentralized web is the problem of data storage. Individual instances unless rich in money and resources has have to do content expiration and this is a "time-censorhip" (after a certain time his content is "censored" aka deleted) - how could there be distributed data storage model so that ALL instances together stored ALL data together rather than individual instance/s storing ALL data and under pressure either crashes or has to do content expiration or just run out of service?  Any work going on this "federation"? For example, ajax or similar page loading that will draw data (say text data) from various instances in my stream - some part of this page will load fast where instances are good and some part may be slow if instances are poor quality but that is ??acceptable to the data storage problem? Thoughts? Definitive plugin solutions? Core changes?

Fri, 21 Apr 2023 20:50:48 +0200

Mario Vavti

mario@hub.somaton.com

Files are stored in /store. Only content from remote profiles where nobody from the local hub interacted with is being expired. Expiration can be set in admin/site or per channel in the channel settings. For upload quota aka service classes see here: /help/admin/administrator_guide#Service_Classes

Sat, 22 Apr 2023 02:36:14 +0200

amal

amal@hort.fan

Thanks a lot @Mario Vavti
Thanks so much @Grouchy
The further questions would be basically user-to-user question, with no intension of taking up Dev time.

Can someone explain more/explain with details what "Only content from remote profiles where nobody from the local hub interacted with" means?

- Content from remote profiles would be in first place in the local hub because some sort of interaction was there, or not? For example, "connections" were set with a remote profile and his text posts are seen in my stream, or he (from remote) commented to my post here but I did not re-comment ... whatever is this, some better examples would be welcome
- How to make sure from time to time expiration is working as intended indeed?

@Grouchy - so you mean if a remote user who is my "friend" here has uploaded an image in his server, that same image will be again added to my server in /store - why not simply see from their server? Same image will be stored in both servers? Is this not a huge waste of space where images usage are heavy?

@Grouchy  @od @Maria Karlsen @Jupiter Rowland

help/en/admin/administrator_guide#Service_Classes is very much appreciated and many thanks for mentioning it - however it or some portions of it are somewhat not clear to the uninitiated - though it is written very explicitly in clear details.
- what are "standard" "premium" and "firstclass"  (these three classes has been mentioned there)
- does a site member come to know that the site (the account) belongs to a specified class or there can be mixture of classes for example in rare cases, a paid member belongs to "premium" class and non-paid members belong to "standard" class?
- OR all members in the single account (instance) belongs to same class?
- if photo_upload_limit'=>20000000000 can be made in standard class, why do we need premium class at all - I mean are these classes not arbitrary? Or standard class will not accept big values such as 20000000000?
- where or how is the setting of the text of warning message that an user sees when his quota is over? Shouldn't  he see his total quota from the beginning, like email or file storage shows us this option from very beginning?
- total_items' =>0, // number of top level posts a channel can create - what figure (or no figure?) will make this unlimited for text posts?
- what happens when  configurations are changed / re-changed during the life-course of an instance?
Sorry for the questions, but if any user/member here can clarify with patience it will be great.

@Grouchy - "can use other methods to map it to a remote service. (fusefs, s3fs-fuse, etc)" - any plugin (3rd party) that provides simple UI/web-interface to do this? Any tutorial or help-page some where? Thanks for the help.

Sat, 22 Apr 2023 07:08:46 +0200

Grouchy

grouchy@hub.grouchysysadmin.com

@amal

- so you mean if a remote user who is my "friend" here has uploaded an image in his server, that same image will be again added to my server in /store - why not simply see from their server? Same image will be stored in both servers? Is this not a huge waste of space where images usage are heavy?

No, that image would not go in your store folder. You'd pull the image remotely when needed. Anything you upload on your instance will go in your store folder.

"can use other methods to map it to a remote service. (fusefs, s3fs-fuse, etc)" - any plugin (3rd party) that provides simple UI/web-interface to do this? Any tutorial or help-page some where? Thanks for the help.

This is really more of a server setup item than a Hubzilla task. There's lots of tutorials on how to mount remote storage, but they all require changes at the system level. I really don't recommend doing this, but it is an option if you're tight on local disk space. A better option would be to upgrade your local space.