Lemmy.ca
  • Communities
  • Create Post
  • Create Community
  • heart
    Support Lemmy
  • search
    Search
  • Login
  • Sign Up
haxor@derp.fooMB to Hacker News@derp.fooEnglish · 1 year ago

Indexing a Billion Pages

blog.mwmbl.org

external-link
message-square
0
link
fedilink
  • cross-posted to:
  • [email protected]
  • [email protected]
7
external-link

Indexing a Billion Pages

blog.mwmbl.org

haxor@derp.fooMB to Hacker News@derp.fooEnglish · 1 year ago
message-square
0
link
fedilink
  • cross-posted to:
  • [email protected]
  • [email protected]
Indexing a billion pages
blog.mwmbl.org
external-link
It’s two years since we launched Mwmbl, the open source, non-profit search engine, on Boxing Day 2021. A good time to take stock of where we are and where we’re going. We’ve indexed over 100 million pages Thanks to our volunteers, who crawl the web using the Firefox extension and command line script, we’re crawling up to a million pages a day, as you can see on our stats page. There are around 50-60 users crawling on an average day.

There is a discussion on Hacker News, but feel free to comment here as well.

alert-triangle
You must log in or register to comment.

Hacker News@derp.foo

hackernews@derp.foo

Subscribe from Remote Instance

You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: [email protected]
lock
Community locked: only moderators can create posts. You can still comment on posts.

This community serves to share top posts on Hacker News with the wider fediverse.

Rules
  1. Keep it legal
  2. Keep it civil and SFW
  3. Keep it safe for members of marginalised groups
Visibility: Public
globe

This community can be federated to other instances and be posted/commented in by their users.

  • 1 user / day
  • 1 user / week
  • 1 user / month
  • 1 user / 6 months
  • 90 local subscribers
  • 4.12K subscribers
  • 18K Posts
  • 10.7K Comments
  • Modlog
  • mods:
  • haxor@derp.foo
  • UI: 0.19.11
  • BE: 0.19.11-n.1
  • Modlog
  • Legal
  • Instances
  • Docs
  • Code
  • join-lemmy.org