In the last weeks Lemmy has seen a lot of growth, with thousands of new users. To welcome them we are holding this AMA to answer questions from the community. You can ask about the beginnings of Lemmy, how we see the future of Lemmy, our long-term goals, what makes Lemmy different from Reddit, about internet and social media in general, as well as personal questions.
We’d also like to hear your overall feedback on Lemmy: What are its greatest strengths and weaknesses? How would you improve it? What’s something you wish it had? What can our community do to ensure that we keep pulling users away from US tech companies, and into the fediverse?
Lemmy and Reddit may look similar at first glance, but there is a major difference. While Reddit is a corporation with thousands of employees and billionaire investors, Lemmy is nothing but an open source project run by volunteers. It was started in 2019 by @dessalines and @nutomic, turning into a fulltime job since 2020. For our income we are dependent on your donations, so please contribute if you can. We’d like to be able to add more full-time contributors to our co-op.
We will start answering questions from tomorrow (Wednesday). Besides @dessalines and @nutomic, other Lemmy contributors may also chime in to answer questions:
Here are our previous AMAs for those interested.
Do you plan to introduce some kind of post tags into Lemmy, preferably something that will behave like Hashtags on Mastodon and other activitypub platforms? I know that Lemmy has been embedding community name as a hashtag for a while now, though having tags that can be populated by users would help discovery greatly.
What are your thoughts on blocking AI scraper access? Any attempts to improve that on the side of Lemmy? Basic things like allowing to customize the robots.txt easily would already help.
I also recently tried this new AI block tool called Anubis with Lemmy, but for some reason it fails with Lemmy-ui. Might be interesting to investigate further.
Anyone that wants to scrape Lemmy would have an easier time setting up their own server, federating with everyone, and reading straight from their DB. No web scraping required. Though, web scraping defenses would be useful against general web scrapers/crawlers.
That would require the authors of these AI scrapers to actually give a f*ck. The problem is that they don’t, and just scrape what ever they can find repeatatly almost like a ddos attack on the open web.
Yup, same as they could clone git repos in one shot, but they instead crawl every single page.
I’ve previously worked in anti-scraping. There is a negative 0% chance the Lemmy devs have the resources to effectively do this without tanking the server for everyone else.
I just set up Anubis today. Specifically I’m only testing it for Lemmy-ui, and it seems to work fine.
It looks like the distributed waves that keep bringing the service down hit exclusively our lemmy-ui subdomain, so maybe non-SSR photon is also a good defense, heh.
Hmm, that is odd. I guess I need to double check my Nginx config for lemmy-ui then. You have your setup documented somewhere?
Edit: ah, you run Photon as the main UI and lemmy-ui somewhere else? I think specifically the split between frontend and backend on the root domain somehow makes Anubis fail to set the correct cookie.
I don’t think it should be a problem, but I’m not that sure either. Lemmy.fedi.zutto.fi also runs it and that’s just a normal lemmy-ui installation. I think Zutto simply forwarded all traffic to Anubis and then fixed federation. There was some discussion and config shared in sopuli’s finnish matrix room.
You can load a different robots.txt in your nginx config, something like this:
location /robotx.txt { index /path/to/my/robots.txt; }Additionally 1.0 will change the “private instance” to work with federation enabled (see https://github.com/LemmyNet/lemmy/pull/5530). Then only logged-in users will see content, while AI scrapers wont see anything except the login page.
Reddit has far more niche communities. There’s the saying that “there’s a subreddit for everything.”
What do you think the trajectory/timeline looks like for lemmy to develop a more robust array of niche communities (aka niche subreddits)?
- I have no idea how and which server I joined, is there any manial I can read better yet visually see how servers are connected that are federated? Thx. And when we search something does it search across all servers? Thanks.
What was the last post that made you really laugh?
This one by @lka1988@sh.itjust.works
Hahaha thanks! That’s great!
On the server perspective, I have a question, what are your thoughts for horizontal scaling on the database? This seems to be the biggest limitation and requiring higher spec hardware to scale especially for the bigger instances.
My tiny instance for example I give over 20GB of RAM just to postgres to make it perform efficient enough.
20 GB RAM for a single user instance sounds like a lot. Did you use pgtune? It may also help to run a reindex or full vacuum.
Yeah I used pgtune as a base and found more memory needed to be assigned to certain spots especially to keep federation with bigger instances, otherwise timeouts would occur resulting in my instance being constantly behind.
That said I read postgres 17 is much more memory efficient, though I have yet to move my lemmy database to it yet since its the largest haha.
Maybe your disk is too slow, or latency between Lemmy and Postgres is too high?
It is a k8s cluster and using ceph for all of my storage so the latency from that I bet is the largest reason and upping the memory offsets the disk writes. i also have another postgres DB syncing as a fallback for high availability. Fortunately after tuning the database and giving it enough RAM my instance has been running pretty stable for over a year without any changes.
I am also using less powerful computers for the entire infrastructure (not server grade) which brings to the point of having horizontal scaling on database I imagine will be a growing need with growing instances, communities, and users since it can be cheaper to run multiple smaller spec servers rather than a single with the added benefit of high availability.
Postgres supports sharding which should work without any changes in Lemmy. But so far not even lemmy.world needs that. There are also read replicas which would require support directly in Lemmy afaik. Such a feature will surely be added as instances grow bigger over time and need more resources.
I didn’t think of using read only replicas, that would probably be a very good way to go since its probably 80%+ of actions are reads. Thanks for answering, I am excited to see the how lemmy grows and thanks for all the devs hard work!
The way to solve the database problems isn’t to keep throwing more and more money at powerful servers and scaling. Its to fix it at the root: lemmy’s unoptimized database.
@dullbananas has done invaluable work in making our DB better (and all of these will be in
1.0), but I’m convinced that if we had even 1-2 more Postgresql experts do a pass over the DB, and ideally one full-time expert, all of these problems could be solved.Does the project maintain a list of known slow queries? This is my favorite type of work
The post list query is by far the worst offender. It needs to filter, sort, cursor paginate, and join to many tables, and indexes are hard to follow and keep up with.
What’s more is that the problems only surface with lots of historical data, meaning we can only really test the query plans with a fully populated DB.
All this requires running lemmy locally, and inspecting the postgres query durations. We really need proper test suites (lemmy DB perf is one example) that can stress-test production data also.
Here is one historical issue:
I’d very much appreciate any help.
I should have some time tonight to start looking at this. Thanks for the info!
Thank you in advance!
Good evening Dessalines, I have started looking at the posts query.
The lowest hanging fruit I think would be if we could replace some of the joins with
WHERE EXISTSwhich can have a huge impact on the query time. It seems this is supported in Diesel: https://stackoverflow.com/a/74300447This is my first time looking at the codebase so I can’t tell yet which joins are purely for filtering (in which case they can be replaced by
WHERE EXISTS) and which joins need to be left in because some of their columns end up in the finalSELECTI can’t tell for sure yet but it also looks like this might also be using
LIMIT...OFFSETpagination? That can be a real drag on performance but isn’t as easy to fix.EDIT:
Looking some more, and reading some linked github discussion - I think to really get this out of the performance pits will require some denormalization like a materialized view or manual cache tables populated by triggers. I really like the ranking algorithm but so far I’m finding it difficult to optimize from a query perspective
Sounds promising
This is helpful. Could you make a github issue and copy-paste this there? Thx.
Nice number
I 100% agree with this and there have been great strides since I started using Lemmy ~v0.17! That said at some point optimization will have lower returns and have a higher effort to put into and once a community grows extensively it likely might not be enough, so I was curious to what you guys were thinking at that point, something like Ctius for sharding postgres?
I’m sure we’re nowhere near that level yet. We haven’t come close to postgres’s limits, and most of our bottlenecks are unoptimized queries.
That’s a very interesting point, have you tried asking for support on !lemmy@lemmy.ml or other general communities? There are probably a few Postgres experts on the platform
We’ve asked for help various times, but don’t usually get much help. Despite the seemingly large number of “experts” out there, only a tiny number of them contribute to open source. I’d still consider it mostly a wasteland, with a few people doing the work that should be done by 100x their number.
Some companies use Reddit as their main forum or an established way to communicate with customers. Are there any companies that have explored Lemmy and have their community yet?
There’s a KDE one, if I’m not mistaken
Not that I know of, only some open source projects as mentioned by others.
Do you guys have plans to add a spoiler tag? I post a lot of memes about tv shows that I watch, but the users complain that the post isn’t blurred.
I know I can use the NSFW tag, but this gives the wrong idea and limits the post visibility (since people can hide nsfw posts).
You can put spoilers in the body
spoiler
Helloooo
Doesn’t work on boost client apparently.
It may still be using reddit style markdown
Yeah a lot of former Reddit apps that switched to Lemmy did a really lazy job of it and haven’t implemented all of the Lemmy text parsing syntax properly. Spoilers are one of the most common issues, but so are subscript (including ~multiple word subscript~) and superscript (and ^multiple words of it^).
If your app doesn’t parse text correctly 2 years later, it may be time to consider switching.
Doesn’t work on the android app Eternity
Open up an issue on their repo.
Eternity hasn’t been updated in a while. You might want to try !summit@lemmy.world or !thunder_app@lemmy.world
but then the user might not realize that there’s an image in the post, which will also limit it’s reach.
So a spoiler tag for post links? This could potentially be added later as an addition to the post tags feature.
Even that isn’t too necessary, since you can already put images in spoiler markdown blocks.
I see. Would the spoiler tag also blur the thumbnail?
The only thing that concerns me about handling spoilers is how the third party apps handle them. Do you think it would be a good idea to also blur the entire image (not only the thumbnail) and remove the blur only when the user clicks the image?
Not sure, we would have to see whenever we get around to implementing that.
You can hide images behind spoiler tags also:
check it out
How are you?
Chilling in the morning before I start my day job.
Not bad, the swiss chard and spinach I planted recently are sprouting, so that’s got me excited.
Spinach is a finicky bastard in my experience, take good care of it
A bit tired because my whole family is half sick. Luckily the kids are still okay to go to school.
Otherwise Im excited for this AMA, because I rarely have such direct conversations with users about Lemmy. The discussions on Github are usually quite technical.
Hi, I think that Lemmy is great thank you for your hard work
I actually think that given the ads and other distorsions, and thanks to federation, Lemmy is overall actually better than reddit!
Some features I miss are:
- tags
- direct messages outside Lemmy (even if not encrypted)
- better rendering of posts on mastodon (something beyond the title only). Not sure what side is responsible for this, tho!
Keep up the good work guys!
- Tags are work in progress
- Not exactly sure what you mean by “direct messages outside Lemmy”, but in version 1.0 they will be compatible with Mastodon and other platforms
- Its a known problem with Mastodon because it only renders
Noteobjects properly, which are meant for short texts less than a paragraph. Lemmy usesPagewhich is meant for longer text. Some platforms like Wordpress (iirc) have an option to federate even long posts asNoteso that it gets rendered fully in Mastodon, but that seems like a bad idea to me. In the end its up to Mastodon how to render different types of federated content on their frontend, so it needs to be fixed by them. Here is an entire discussion about this by developers of different Fediverse platforms (including a Mastodon dev).
Thank you, great to hear all my points are being addressed! The thing about post rendering, well, I just hope a common solution is found 😊
How is it some can mod 15+ comms, like this awful character PugJesus , ban anyone for no reason and then comment stuff like this without consequence:
Be less of a dick.
Be less of a moron.Are there any plans to deal with the most common annoyances regarding Lemmy? In my opinion these are all based on federation:
- Some completely automated way for users to join Lemmy. Yeah, it’s not hard to select a server and it’s a “good thing to do”, but it’s still better to give people the option to go for convenience instead of the “proper” path. Maybe some kind of system where instances sign up for this general, convenience way of signing up, and the registered users just get automatically distributed evenly across those instances.
- Duplicate post aggregation. The nature of federation will always make it make sense to have duplicate communities, but this will also make posts with the same links, same images, same videos, etc show up in people’s “all” feeds multiple times. It is technically possible to algorithmically detect these duplicates and offer users a UI option (not actual backend merge) to merge them all visually into one post.
- A way to backup your whole user data and completely restore it on any instance you want. If an instance goes under, it should be possible to keep all subscriptions, all your posts, all your comments, and migrate them to a new instance.
it’s still better to give people the option to go for convenience instead of the “proper” path.
https://phtn.app/signup gives a prepopulated list
show up in people’s “all” feeds multiple times.
Which interface do you use? Crossposts only show up once on the default UI
A way to backup your whole user data and completely restore it on any instance you want. If an instance goes under, it should be possible to keep all subscriptions, all your posts, all your comments, and migrate them to a new instance.
You can already export and import your subscriptions between instances (account settings - import/export)
Posts and comments can’t be migrated, but Mastodon doesn’t allow it either.
Mastodon currently does not support importing posts or media due to technical limitations, but your archive can be viewed by any software that understands how to parse Activity Streams 2.0 documents.
gives a prepopulated list
The official one also does that. I’m talking about choosing a username, password, and email maybe, and then clicking register, and being done. No thinking involved.
Crossposts only show up once on the default UI
False, you get links to the other posts, of which you posted a screenshot, but each post is handled as being completely separate. If you are in the subscribed, local or all feeds, you would see all of these posts separately. Have you really never noticed scrolling by “the same” post multiple times? You have to go to each post manually to get all the comments to the “same” thing.
but Mastodon doesn’t allow it either […] due to technical limitations
Yes, I know that. But I’m also a programmer and I know that “technical limitations” is mostly a term for “that’s how we started it and it would be too costly to solve now, so we’ll just dismiss it” and not for actual limitations (i.e. not technically possible). It’d maybe require breaking changes of some kind or some kind of annoying backwards compatibility workaround, but that is why I’m asking. I’m not completely familiar with activity pub, but there’s likely some key used to verify posts/messages are made by a certain user, and there’s currently no way to transfer or change that key to a new account. But it seems very technically possible to me, and also possible without massive security issues. So that was my question, is there any plans to do this or no?
The official one also does that. I’m talking about choosing a username, password, and email maybe, and then clicking register, and being done. No thinking involved.
This should probably be handled more by people when they talk about Lemmy. Instead of explaining what Federation is, just point people to https://vger.app/settings/install so that they can install an app.
Voyager by default suggests Lemm.ee as the instance to register, so no thinking indeed: https://vger.app/profile
That’s the recommended approach nowadays on !fedibridge@lemmy.dbzer0.com
False, you get links to the other posts, of which you posted a screenshot, but each post is handled as being completely separate. If you are in the subscribed, local or all feeds, you would see all of these posts separately.
I only see them once in my subscribed feed. You may indeed see them multiple times if you are not subscribed to all the communities.
You have to go to each post manually to get all the comments to the “same” thing.
This one I agree, and it’s why I generally suggest to consolidate similar communities to solve that issue. !fedigrow@lemm.ee has quite a few examples of successful consolidations.
I’m not completely familiar with activity pub, but there’s likely some key used to verify posts/messages are made by a certain user, and there’s currently no way to transfer or change that key to a new account.
You’re looking for https://activitypods.org/ . I haven’t been following their recent progress, not sure how mature their solution is, and how many Fediverse platforms support them.
False, you get links to the other posts, of which you posted a screenshot, but each post is handled as being completely separate. If you are in the subscribed, local or all feeds, you would see all of these posts separately.
I understand your frustration, however these can be multiple posts but to different communities with varying focuses and moderation styles.
Simply consolidating all the comments in one introduces its own problems.
That’s why no one suggested “simply consolidating”. I didn’t suggest any solution at all. I’m just posing a question of if this actually pretty big problem is attempted to be handled.
I guess it’s just been mentioned too much in the past that it still comes to mind when I hear this. Sorry.
- A way to backup your whole user data and completely restore it on any instance you want. If an instance goes under, it should be possible to keep all subscriptions, all your posts, all your comments, and migrate them to a new instance.
This would be great… also I feel offering this might even be a requirement for a server to be fully GDPR compliant (though I could be wrong on that, IANAL).
Multispam is one of the things that is genuinely a threat to Lemmy’s usability. If you follow certain topics, you start seeing 2-5 copies of every post. It’s a genuine spam problem and “just block” or “just scroll” is as much of a non solution as it is with other spam.
Some users use it as an ideological war on instances they don’t like, which is extremely funny.
Personally when I want to share something to multiple communities, I deliberately space out the posts in time. It has the slight downside of potentially showing up for a person repeatedly for days, but I think this is outweighed by the upside of them not seeing the same post multiple times within a few minutes, which is really annoying. And it has the added advantage of being more likely to be seen by people who weren’t online at the right time to see it the first time.
Sometimes. But other times the instance/moderation vibes of each post will be different. Sometimes I enjoy seeing how different groups respond.
Yeah, it’s not hard to select a server and it’s a “good thing to do”, but it’s still better to give people the option to go for convenience instead of the “proper” path.
We could add a “fast join” button to the signup dialog on join-lemmy.org , where it takes you to a random instance’s signup page.
Overall though, we should ignore the “advice” from reddit that tells us that people are too stupid to sign up for anything now. People did this for every forum and every other site all until ~2005 when US tech gobbled up most services, and ppl continue to show us that yes, they do know how to type in a username, password, and email to sign up for something.
Duplicate post aggregation. The nature of federation will always make it make sense to have duplicate communities, but this will also make posts with the same links, same images, same videos, etc show up in people’s “all” feeds multiple times.
In lemmy-ui we have a post-deduplicator for feeds, but unfortunately not a lot of other apps (including jerboa, that’s my bad) have added something similar.
A way to backup your whole user data and completely restore it on any instance you want. If an instance goes under, it should be possible to keep all subscriptions, all your posts, all your comments, and migrate them to a new instance.
Settings export already exists. Copying historical content and rewriting history isn’t possible in a federated system, but we do have an open issue for data export.
we should ignore the “advice” from reddit that tells us that people are too stupid to sign up for anything
Definitely agree. The problem is just when someone in the past said “you should join <forum x>!”, you were always able to just immediately go to forum x’s signup page and sign up. But if someone hears of Lemmy, and goes to join-lemmy.org, there is no way to go to a signup page directly. They have to first learn about the multiple servers, and choose one. I think a “fast join” button like you say should be fine, and immediately next to it something to catch all the advanced actually curious users with something like a “advanced sign-up”
In lemmy-ui we have a post-deduplicator for feeds
That’s weird, because that’s exactly from where I’m coming from, I’m always using the lemm.ee website directly on all my devices, and I constantly see duplicate posts.
Copying historical content and rewriting history isn’t possible in a federated system
I have less knowledge of this topic so I’ll defer to you, but I have the feeling this may not be true. You might of course not be able to ensure consistency between all instances, ensure that it’s been changed everywhere, but I really can’t see why this is any different than “editing” a comment’s content or a post title, which is already possible. Why wouldn’t it be possible to “edit” the comment/post author in exactly the same way?
Thanks for your response and all you’re doing!
That’s weird, because that’s exactly from where I’m coming from, I’m always using the lemm.ee website directly on all my devices, and I constantly see duplicate posts.
I was thinking about what you were saying yesterday and I had another look.
- on the subscribed feed, using “New Comments”, crossposts seem to indeed aggregate
- on the All feed, using “Top of the Day”, crossposts would indeed appear multiple times, depending on their respective scores
I like to use “Top of X” a lot 🤔
The problem is just when someone in the past said “you should join <forum x>!”, you were always able to just immediately go to forum x’s signup page and sign up. But if someone hears of Lemmy, and goes to join-lemmy.org, there is no way to go to a signup page directly.
People should probably stop saying “join Lemmy”, and instead link directly to the site/instance they suggest
- From a code architecture perspective, how close is Lemmy/ActivityPub to reaching its maximum capacity for posts/comments per second? Are there any ways to 10x the load ActivityPub can handle?
- With Nicole in everyone’s DMs, what does the future of spam filtering look like on Lemmy?
- There is no specific maximum capacity, in theory it can scale indefinitely with horizontal scaling.
- 0.19.10 already includes a fix to remove private messages when a user gets banned which should help a lot. There is an issue about disabling private messages by default, but Im not sure if that will be necessary. Also 1.0 will include a plugin system, so other devs and instance admins can write their own checks. That way spam waves can be fought in a more flexible way, without having to get a change merged into Lemmy and then waiting for a new release.
I can answer the first point.
We’ve already tackled part of that problem with the Parallel Sending feature that can be enabled on instances with a tremendous amount of traffic. Currently the only instance that makes sense to enable that is LemmyWorld and the only reason is so servers in geographical far away can get more than 3-4 activities/second.With that feature, servers that eventually house and generate the biggest amounts of traffic will be able to successfully communicate all of those activities to everyone else who needs them.
I predict a 10x increase is well in our grasp of easily accessible by all of our current systems. 1000x? That’s a different story which I don’t have the answers too.
Currently the only instance that makes sense to enable that is LemmyWorld and the only reason is so servers in geographical far away can get more than 3-4 activities/second.
For the context, it’s still not enabled at this moment: https://lemm.ee/post/59055817
I like Lemmy a lot, but when you share a URL it’s just an ID number. Compare that to Reddit, where you can get a lot more information on what you’re about to look at just from the URL alone.
https://lemmy.ml/post/27659153 vs
https://old.reddit.com/r/AsahiLinux/comments/1jg3vlk/progress_report_linux_614/Are there any plans to make Lemmy URLs more meaningful?
Another issue is that post links are instance-specific, since the post ID isn’t the same across instances.
ex: https://lemmy.ml/post/27659153 is https://lemmy.ca/post/41237641 on Lemmy.ca
There are external tools like https://lemmyverse.link/ and some browser addons to alleviate those issues, but it’d be nice if this could be addressed at the source if doable.
And I dream of a
lemmy:\\protocol handler one day.Protocol handlers are the way to go IMO. I’ve opened an issue based on @phiresky’s comment.
Potentially, using some sort of predictable hashing to get the same id across instances might also help in the detection of duplicate links so that they can be aggregated in a single place (sort of what was suggested at point 2 here).
I fear this could be too much of a breaking change though.
We won’t add UUIDs or any “universal” sort of identifier, but universal links are still possible without them.
Seems like a good time to introduce a breaking change, jumping from 0.19 to 1.0.
It will be a feature for Lemmy 1.0: https://feddit.org/post/5390705
Until then https://lemmyverse.link/feddit.org/post/5390705 ;)
I even discovered there was a new one today because the creator wasn’t aware of Lemmyverse link x)
There’s some discussion of this here. It’s not something I care too much about, but anyone is free to add more detailed URLs to either lemmy-ui or other front-ends.
Its been years but no one has found it important enough to them to work on.
It will be a feature for Lemmy 1.0: https://feddit.org/post/5390705
I’m just a user and don’t follow the project super closely so please forgive me if this has already been addressed somewhere.
Is there any sort of JavaScript-less interface that would work properly on the Tor network with strict settings enabled? And could you set up instances only with a .onion domain? That way you don’t have to pay for a domain and you’re not at risk of having your domain yanked by ICANN, etc.
If I remember correctly, there was a mastodon instance that was using like a Pakistan domain or something like that and they yanked it.
Also, Federation between Onion and Standard Domains that way tor users would not be isolated.
My main reason for asking is that in my worldview, governments want to break encryption and break freedom of speech, if at all possible, and so the dark web is going to be more and more necessary as time goes on.
Federation between Onion and Standard Domains that way tor users would not be isolated
This is the hardest part as you would need to be both have an onion and have a standard domain, or be a tor-only Federation.
You can easily create a server and allow tor users to use it, which unless a Lemmy server actively blocks tor, you’d be welcome to join via it. But federation from a clearnet to onion cannot happen. It’s the same reason behind why email hasn’t taken off in onionland. The only way email happens is when the providers actively re-map a cleanet domain to an onion domain.
This is what Lemmy would need to do. But then you would have people who could signup continuously over tor and reek havok on the fediverse with no real stopping them. You would then have onion users creating content that would be federated out to other instances. & User generated content from tor users also is … Not portrayed in the best light.
I’m sure someone will eventually create an onion Lemmy instance, but it has it’s own problems to deal with.
This is especially true for lack of moderation tools, automated processes, and spammers who already are getting through the cracks.@ciferecaNinjo@fedia.io time to shine
There is documentation for running a Lemmy instance over Tor, and one of the many frontends probably works without JS. If not someone could implement it with the API. Anyway there doesnt seem to be much interest in practice, because the clearnet works good enough for now.
Okay, that shows how to make the interfaces available over the network. But is there planned functionality for making Federation work? Even if you don’t federate to clearnet domains, you could have several onion instances that could federate with each other and have a network of onion instances.
Federation uses standard HTTP so it shouldnt require any specific support. If you want to send outgoing federation over Tor you can set a proxy via environment. Youre welcome to test these things and update the documentation with your results.
Oh, okay, good to know it should work. I’m just a standard user, but it is something that interests me.
