I had been having trouble getting meaningful results from the fediverse on Google, and after seeing this post, it seems I’m not the only one. So, I created a site that helps search the fediverse in your search engine of choice (it currently supports Google, Bing, Yahoo, DuckDuckGo, and Dogpile).
Due to query limitations with most search engines, it currently only searches the top 15 lemmy/kbin instances, but I’ve tested it and it seems to provide access to a good chunk of fediverse content. The exception is Google, which should be far more reliable overall as well as providing the ability to search Mastodon and PeerTube.
If you have contributions or ideas for improvement, feel free to check out the project here or shoot me a message. Hope this helps people! :)
Edit: Update in progress including improved search queries and support for Mastodon/PeerTube (Google only, unfortunately)
Edit 2: Update is live, along with a dedicated domain name. If the website doesn’t look any different for you, try Ctrl+F5 or clearing site data - it seems some browsers are caching the old page.
Searching Mastodon is a bit of a… contentious issue. A lot of smaller Mastodon-based sites are full of traumatized vulnerable people who really just want to do their own thing, and they’ll rattle cages if they find out someone’s indexing their sites or posts. If anyone’s making third party search tools, it’s best to be careful to respect discoverability and indexing flags.
I find this to be incredibly fair, but also makes it much harder to dive into the fediverse. Where is the middle ground do you think?
Mastodon has flags for opting in to discoverability features (being featured in the profile directory, and having posts be searchable via Mastodon’s search bar) and for search engine indexing (for Google, bing, etc.).
Just don’t return posts from users that have opted out of those, and things should be mostly ok.
This is the main problem I see. User settings are part of the mastodon API. If you’re building a general-purpose search engine, you use a crawler to index pages and your crawler has no idea those flags even exist.