Marginalia Goes Open Source

https://memex.marginalia.nu/log/58-marginalia-open-source.gmi

by georgehill • 4 years ago

358 72 4 years ago

memex.marginalia.nu

marginalia_nu13• 4 years ago

Actual announcement: https://memex.marginalia.nu/log/58-marginalia-open-source.gm...

My gitea instance is on a poor Raspberry Pi, probably won't survive HN front page :P

tejtm• 4 years ago

I am not surprised you seem compelled to do the right thing

pabs32• 4 years ago

Thanks for choosing the GNU AGPLv3 license! That is the best license for services like search engines, especially if they intend to cement the license by having a diverse set of copyright holders that ensure the license doesn't change, rather than going the open core route and requiring copyright assignment via a CLA and selling proprietary versions.

What is your plan for sustaining the project in terms of contributions, funding and hardware?

marginalia_nu13• 4 years ago

Yeah, that was my analysis as well.

What is your plan for sustaining the project in terms of contributions, funding and hardware?

Still much to figure out. The project is still fairly immature in general.

As far as hardware and funding goes, it's sort of sustainable now through not demanding very much in either, but more of both would no doubt be necessary to grow much farther. It's a bit claustrophobic not really having a proper CI machine or test instance for example.

dang• 4 years ago

We changed the URL to that from https://git.marginalia.nu/marginalia/marginalia.nu. Thanks!

pronoiac• 4 years ago

A brief "what is marginalia.nu" section would be awesome!

em-bee3• 4 years ago

i found these pages:

about the search engine: https://memex.marginalia.nu/projects/edge/about.gmi

about the encyclopedia: https://encyclopedia.marginalia.nu/wiki-clean.html

em-bee3• 4 years ago

If you feel the need to complain about how something doesn't align with your personal philosophical convictions and fails to satisfy your criteria for ideological purity, please write a really long and angry essay about this topic, and send it to <kontakt@marginalia.nu>.

Don't forget to press caps lock as you begin typing to save your pinky fingers, I wouldn't want to be responsible for nasty RSI.

i like your attitude. i think you are off to a good start.

that data file, could you host that on github (or gitlab maybe)?

jwilk2• 4 years ago

Reminds me of this classic:

http://bash.org/?835030

mseri• 4 years ago

Seems to be up and running fine. Gitea seems to be quite robust and reliable! I hope you have regular backups, I had a number of unfortunate IO corruptions with the PIs.

Thanks for the open sourcing

slowbdotro• 4 years ago

If you want, come mirror it on my gitea instance: gitea.slowb.ro. I'd be happy to take the load ^-^

opan• 4 years ago

AGPLv3+, nice!

formerkrogemp• 4 years ago

Haven't there been estimates that running a globally indexed search engine product costs a billion dollars per year? As always, thank you for marginalia!

marginalia_nu13• 4 years ago

This only does a fraction of what the big search engines do, though. Still intresting to find out what is possible given those constraints.

Aeolun• 4 years ago

I think the helpful thing is that if you don’t intend to keep all blogspam, you are in a relatively good position.

thrtythreeforty• 4 years ago

Open source has a wonderful property: it only ever gets better as time goes on.

Thank you for permanently improving the FOSS search state of the art.

mikewarot• 4 years ago

TLDR; There are lots of interesting tangents here, even if you don't want to run a web search engine.

I'm in search of kindred spirits seeking to flush out the full power of the memex. I found the notes it quite interesting[1]

I've gone through quite a bit of the related content, up to watching a bit about the Gemini protocol, which is interesting.

1 - https://memex.marginalia.nu/log/08-whatever-happened-to-the-...

jwilk2• 4 years ago

The search engine discussed on HN:

https://news.ycombinator.com/item?id=28550764 (September 2021, 725 comments)

ternaryoperator• 4 years ago

I feel GitHub has taken an incredibly toxic turn with its emphasis on social features

This the first time I've seen this view of GitHub, and I'm struggling to figure out what specifically he's referring to.

rhn_mk1• 4 years ago

Seconded. GitHub's social features are undeniable (why distribute a VCS if not to make it more social?), but I'm curious about the toxicity.

mssdvd• 4 years ago

I think things like the personalized timeline and comments reactions may be considered toxic by some people.

daptaq2• 4 years ago

why distribute a VCS if not to make it more social?

Because it is easier to set up?

t-3• 4 years ago

Distributed version control is about making code and patches easier to share, if anything the point is to reduce the socialization. It's a more convenient and development-centric form of mailing lists, not a more social version.

metadat2• 4 years ago

The GitHub UI forces their view of the world and their development flow on you.

It doesn't bother me personally, but perhaps I've already been indoctrinated over the past ten years.

cookiengineer• 4 years ago

I would guess that the chan board members and other asshats raided their projects in the past, because they regularly attack / harass these kind of projects when they're not fully open source.

And let's put it this way: the moderation tools at GitHub are a bad joke. Can't even disallow abusive language, so they're 100% ineffective.

charcircuit• 4 years ago

The author of the article was saying that Github was toxic and I personally do not see an issue tracker as a social feature.

baisq2• 4 years ago

"disallow abusive language"? Do you think chan board users are too stupid to know how to bypass an "abusive language" filter?

pelagicAustral• 4 years ago

Sneedacity debacle.

baisq2• 4 years ago

Funny you'd mention this. I installed Audacity a few hours ago and much to my dismay the application informed me upon starting it for the first time that update checks were enabled by default - in other words, Audacity had already called home and I couldn't have done anything to stop it from doing so.

So yes, they deserved the abuse.

Xeoncross3• 4 years ago

This is a great gift to the community, but I really wish it would be hosted where the community is active. It seems like a lot of people will miss out because they won't be able to discover this.

I find a lot of projects because I follow people on github that star or fork really neat projects. I download projects from bitbucket, gitlab, and other self-hosted instances.

I don't know that I've ever contributed a PR back as it's just a more friction-prone process. However, I've opened hundreds of PR's and Issues on github because it's already a part of my daily process.

Thanks again

robertlagrant3• 4 years ago

Thinking that a community lives in a git provider is a bad idea. It shouldn't matter who hosts a git instance, and I would argue it's better to fight for diversity here.

throwaway832422• 4 years ago

Thanks for making this open source. What about the index? Are you open sourcing that also?

marginalia_nu13• 4 years ago

If I can solve the logistics of publishing that data, then sure. In its most compressed form it's still of order 100 Gb.

The intermediate goal is to have some standardized testing dataset of a couple of hundred megabytes to a gigabyte or so.

Jarwain• 4 years ago

Like another commenter suggested, torrents might be a good solution once it's seeded

throwaway832422• 4 years ago

Cool. Looking forward to see the intermediate dataset.

I think you should post a ToDo list on the git repo. People can then contribute their skills.

sdoering• 4 years ago

I'm currently looking for hosting for a large term frequency data file that is necessary for several of the search engine's core functions. I really don't have the bandwidth to serve it myself. It's only a couple of hundred megabytes so it'll probably be solvable somehow.

I have a bit of (rented) space available. Most of it on Uberspace.

What are the expected traffic requirements?

Contact me if you like. Will update info in profile immediately.

closedloop129• 4 years ago

Wouldn't this be a good match for BitTorrent distribution? The data will be updated regularly and then, all participants will be online and able to share their bandwidth while they download the new frequencies.

sva_• 4 years ago

I'm not sure if it is appropriate for this use case, but wouldn't it be possible to host it on Backblaze, distribute through Cloudflare for free traffic? Or even just a torrent.

remram• 4 years ago

GitHub Releases can host files up to 2 GB, for free.

trenchgun• 4 years ago

Alright, this seems great. Going to test it in daily use.

I am ready to start paying money for search engine. So fed up with google SEO spam.

marginalia_nu13• 4 years ago

Probably not going to replace Google, it's at best aiming to pick up the slack where it's struggling. But dunno, maybe down the line it will have grown into a real open source alternative.

lowwave• 4 years ago

Thank god a search engine that try to sell you stuff. Glad it is open sourced. Are they from Scandinavia?

marginalia_nu13• 4 years ago

I'm based in Sweden, yeah.

SemanticStrengh• 4 years ago

That's one small step for man, but one giant leap for mankind.

nonrandomstring• 4 years ago

Well done and thank you. Not just for the work, but for the attitude and courage of your convictions that come across clearly in your comments here on HN.

Search is presently a neighbourhood of technology that's fallen into a downtrodden ghetto. Maybe you've sowed some seeds that can grow between the cracks. But I think there are gangs and pushers roaming the streets that would be happy for it remain derelict.

Thinking about the dynamics of open source, with respect to the many different interpretations of search that create tension within such a project: It occurs to me that whenever a somewhat mature project enters the ecosystem a robust strategy is to immediately clone-fork multiple identical copies of a project. This is like a "backup strategy". From there, form several loosely coordinated project groups that can spread risk, share threat intelligence, but each work with different models of crawling, filtering, funding, presentation and so on. Keep the original as an upstream parent. One of the children may survive.

josefcullhed• 4 years ago

Great job with Marginalia. Do you plan to open source your data as well or only the code?

marginalia_nu13• 4 years ago

If I can solve the logistics, maybe. I don't have the needed bandwidth and off-site storage at this point.

SahAssar2• 4 years ago

What sort of sizes are we talking about? I'm thinking if it would be possible to "crowdfund" the storage-costs for a requester-pays s3 bucket for it.

metadat2• 4 years ago

Idea: Host datasets as one or more torrents. Thoughts?

hestefisk2• 4 years ago

Very cool! And brave! Good on you.

hestefisk2• 4 years ago

Do you have any high level architecture docs? Would love to hack on it.

marginalia_nu13• 4 years ago

I'm working on it. Still in the process of setting up the project, so it will be a while before any sort of useful level of documentation is in place.

firedev• 4 years ago

What is Marginalia? Feels like a links database.

marginalia_nu13• 4 years ago

It's a grab-bag of projects geared toward internet discovery and information access.

It's got an internet search engine, a lo-fi wikipedia mirror, this thing <https://search.marginalia.nu/explore/random>, and more.

Marginalia Goes Open Source

Related Stories