I made this tool to help self-hosters, new admins, or smaller instances have more global and updated content on their instances.
This is the similar to Lemmy Community Seeder but is designed to be run periodically to capture new communities, and include EVERYTHING by default.
EDIT: As noted in the comments, this is an admin tool. Please do not run it as a user if you don’t know what you are doing. If you want a better “All,” ask your admin first! That said, lemmony in no way constitutes abuse! You can cause a DOS with curl, but that’s not what curl was written for. This tool is to legitimately use an API to enhance our experience. Admins that desire to accommodate high volume on a public service will not know this tool is running against, or on their instances. If it causes performance issues, that is unfortunate. They are free to throttle, ban or block API access to their instance in a multitude of ways.
EDIT 2: Donate to your instance/admin if you like Lemmy!
aa
Quite a bit of space could be saved with database compression. The database side of things has lower hanging fruit right now though.
aa
Images are not federated, they only live on the hosting instance.
Thumbnails might copied though, I’m not sure.
aa
There is some discussion. https://github.com/LemmyNet/lemmy/issues/2947
I am still fairly confident that it shouldn’t be storing images, but I’ll admit my pict-rs directory is growing quite fast compared to the database. Have to keep a close eye on this.
aa
I’m not convinced either one of us knows what the software is SUPPOSED to do, and I am pretty sure nobody knows what it’s actually doing. Here’s another thread: https://github.com/LemmyNet/lemmy/issues/3163
Does the image get stores on the poster’s instance or the instance hosting the community they’re posting to?
The poster’s.
EVERYTHING by default. Also working on “discover only” for searching without the subscribe-to-everything. That said: It’s far less than 3GB per day for EVERYTHING I can see, plus: you don’t HAVE to keep it forever. Were you doing something that got other than text?
Do you have a link to a documentation concerning retention/cleanup for instances?
I don’t. I haven’t looked yet either because I haven’t crossed that bridge. I think there were some admins on matrix chatting about it though. It will become an issue for large instances like near term, so I suspect someone will tackle it very soon, if they haven’t already.
aa
They’re not supposed to, and don’t call me friend, buddy.
aa
I ran this and it causes a lot of load. Only issue is any user can run it so basically lemmy servers are pretty much open to be ddos attacked by a user subscribing to everything it seems. I have a pretty good server and it consumed 30% of the cpu continuously just adding all of lemmy.world. Unclear how much disk space I just commited myself to also. Small instances would be decimated by this. Wouldnt be hard for someone to load this docker up 10 times and pull from the biggest 10 lemmy servers all at once to max out a server and cause a lot of other issues.
Other than this, good job. Seems to work well. Maybe too well.
Added a lot of features if you want to try again.
I’d recommend anyone using this to really consider how much data this’ll use on their system.
Just for context, subscribing to about ~800 communities, of which have about more than 50 users/month, my metrics have indicated a disk space usage of about 2GiB/day, 20% of a single CPU core, and about 8~10GiB/traffic a day.
I like this idea, I’ve been thinking about running my own private instance but decided against it as I like the main feed with many different communities that larger instances offer.
This is a neat idea. I’m guessing it’s something the admin of the instance has to run?
If you are on a smaller instance you should probably ask the admin(s) if they are okay with something like this, it would put a lot of extra strain on the server and might overload small instances.
Technically no. Any user could. I wouldn’t recommend it though, as it will subscribe you to every community.
Hosting an instance myself, I’m not amused, because if forces my instance to literally sync all content there is on the lemmyverse, drastically increasing traffic, storage use etc.
Please don’t force resource consumption beyond any rational usage!
Frankly speaking, what did you expect from running this tool? That’s exactly what it’s meant to do.
The weird rage people have about this. I’m not sure where it comes from. If there are 100 communities, only the top 1-5 will contribute 90% of the content. If you have even one user subscribed to the top 20 or 50 communities, you are already likely getting 90%+ of this traffic. After subscribing to literally every community in the lemmyverse, I promise your instance will not see any meaningful increase. I’m willing to be proven wrong, but not one of the ragers has offered a credible reason other than fears based on misunderstanding. No offense.
As someone who has created a similar tool, I can understand where the concerns from other people come from, but your arguments have merits too, since as you’ve said, it only takes a user subscribing to the top communities to replicate what tools like these can do in a hundredth of the time.
Much of the issues that can arise from the usage of this tool, or an overly enthusiastic user subscribing to all the communities, is simply the centralization of top communities in top instances. In an ideal federated universe, instances like lemmy.ml, lemmy.world or even burggit.moe wouldn’t be the top instances that everyone knows, but rather the communities they host, would have be properly and evenly spread out through the fediverse, greatly lowering any load that any one server might get. Unfortunately, that is not the case, and the top servers have to contend with growing traffic, lowering performance, and of course, being the target of subscriptions from the other smaller Lemmy instances, which would only grow worse with usage of tools like this, and my own.
Is this socially acceptable? Is writing my own tool socially acceptable? Frankly, at this point, after reading all these comments, I’m not too sure myself. My only mitigating factor is that the tool I wrote for my personal Lemmy instance serves around ~50 people, thus I hope that, at the very least, those people would be able to benefit from having the communities pre-seeded for their viewing experience. No one really likes a barren front page after all.
I see that you’ve added a configurable option to only subscribe to the top N instances in each instance, which seems a bit more harsh than my own tool (which subscribes to any instance above a threshold), but given the views in this thread, and your other cross-posted threads, perhaps I’ll be looking to implement that into my own tool as well, before I get witch hunted myself.
That said, for anyone out there curious about numbers, subscribing to about ~800 communities, of which have about more than 50 users/month, my metrics have indicated a disk space usage of about 2GiB/day, 20% of a single CPU core, and about 8~10GiB/traffic a day.
