marknadal

Hey, I'm Mark Nadal and I was accepted into Mozilla's "Fix the Internet" program - it has been fantastic so far & huge respect to everyone at Mozilla for putting this on.

Note: I doubt anyone will see this comment as HackerNews has shadow banned my account because my Open Source project competes with some of YC's investments (it undermines & destroys the need/market for some of their crypto-coin scams).

I interviewed with YC a few years ago and their entire process was terribly unprofessional - I've gone through 3 accelerator programs now, and interviewed at many more. YC itself seems pretty awesome, but beware of the politics - if a single person (HN mod in my case) feels like you are a threat to their status, that person has enough marketing power to hurt you. Find the GOOD people in YC and work with them, don't leave it to chance or you'll fall out of good graces.

In contrast, look what Mozilla is doing! They're helping move the internet (the world, the community) forward, and are a non-profit foundation. I highly recommend applying to their program. They are GOOD people, doing good work. Please please do everything you can to help them make this a success - the internet depends on it.

I think that is a good hill to die on, tho I would rather prioritize UX (browser not freezing) and server responsiveness. Ideally we'd have no CPU chunking & good UX, but if we have to choose one, which would you sacrifice?

If string[index-1] === `\\` Then skipAgain

Holy Cow! 2.5GB/s that is amazing.

Meanwhile I can barely get Chrome/NodeJS to parse 20MB in less than 100ms :(.

How useful (or useless) would Simdjson as a Native Addon to V8 be? I assume transferring the object into JS land would kill all the speed gains?

I wrote my own JSON parser just last week, to see if I could improve the NodeJS situation. Discovered some really interesting factoids:

(A) JSON parse is CPU-blocking, so if you get a large object, your server cannot handle any other web request until it finishes parsing, this sucks.

(B) At first I fixed this by using setImmediate/shim, but discovered to annoying issues:

(1) Scheduling too many setImmediates will cause the event loop to block at the "check" cycle, you actually have to load balance across turns in the event loop like so (https://twitter.com/marknadal/status/1242476619752591360)

(2) Doing the above will cause your code to be way slow, so a trick instead, is to actually skip setImmediate and invoke your code 3333 (some divider of NodeJS's ~11K stack depth limit) times or for 1ms before doing a real setImmediate.

(C) Now that we can parse without blocking, our parser's while loop (https://github.com/amark/gun/blob/master/lib/yson.js) marches X byte increments at a time (I found 32KB to be a sweet spot, not sure why).

(D) I'm seeing this pure JS parser be ~2.5X slower than native for big complex JSON objects (20MB).

(E) Interestingly enough, I'm seeing 10X~20X faster than native, for parsing JSON records that have large values (ex, embedded image, etc.).

(F) Why? This happened when I switched my parser to skip per-byte checks when encountering `"` to next indexOf. So it would seem V8's built in JSON parser is still checking every character for a token which slows it down?

(G) I hate switch statements, but woah, I got a minor but noticeable speed boost going from if/else token checks to a switch statement.

Happy to answer any other Qs!

But compared to OP's 2.5GB/s parsing?! Ha, mine is a joke.

I get some hate my Open Source project & actively abuse downvotes/flagging to censor me.

But for every 1 hater there are 100s of hackers that have found, starred, used, or told me (in our chat channel) they were thankful they discovered GUN via HackerNews.

& HN guidelines encourage on-topic submissions & comments (https://news.ycombinator.com/newsguidelines.html) "Anything that good hackers would find interesting ... anything that gratifies one's intellectual curiosity" even you have to admit that GUN discussions have sparked a lot of intellectual & algorithm chats.

Finally, you state I'm not spamming then say I am spamming. Spam is indiscriminate posting of something, yet your very own comment says "you always post GUN in discussions about CRDTs & decentralization." That is indeed on-topic according to HN policy.

Not sure why my post was flagged.

Just to be clear, Martin (site owner) added GUN to the list, not me. Proof:

https://twitter.com/martinkl/status/1244672269395406849

Thanks, was on mobile which doesn't show it.

Edit: (reply is now showing)

Hey, my Open Source database is the #1 CRDT rated system on GitHub, and is used in-production by 10M+ people a month, at non-profits like the Internet Archive, and others.

https://github.com/topics/crdt

Mind adding it to the site?

Gorgeous work! Such a great blessing & story, thanks for sharing.

Wow, even better / awesome! Thank you!

(no affiliation) https://www.headliner.app/ will do this for free < videos/month.

And how do they expect to legally & constitutionally enforce this?

Not that SF isn't already a ghost town tho.

Does this mean return flights too?

Or just travel originating from Europe and going inbound to USA?

Thanks for the reply.

If you ever have time, it'd be great to snag whatever code or data you had, to create a replicable test case for us to make a fix against. Sorry about the wasted time you experienced, I hate that.

Thanks for the comment.

Could you expand further what didn't work? We have some pretty heavy load in-production systems running. Definitely some rough edges still, so I'd like to hear what problems you had, so I can focus on fixing them in the future.

Thanks again.

Links?

IPFS, a YC company, is trying to censor Open Source competitors? Interesting. What are you scared of?

https://web.stanford.edu/~nadal/A-Decentralized-Data-Synchro...

https://era.eco/

http://axe.eco/

I think parent was saying the projects using IPFS are relatively unknown, so where are the millions come from?

I have the same question.

In contrast, GUN, has 20M+ downloads/month from known sources: Internet Archive, HackerNoon, etc. (https://github.com/amark/gun see jsdelivr download stats).

Correct, this is what we solved for.

I gave a presentation at Berkeley a few years back, right before Bram Cohen (BitTorrent) presented his Chia proof, Proofs of Space and Time (before IPFS ripped the name from Bram).

https://www.facebook.com/BlockchainatBerkeley/videos/2006069...

Not any that I have to pay for yet! :)

Huh, 350K for $3 is good?

I'm handling Millions for $0 :/ with my https://github.com/amark/gun .

This was a phenomenal interview, learned many rich nuggets.

I'm curious to go back to the "old days of graphics" with limited RAM, loading from slow CDs, etc. & program like that myself, where can I start?

We're doing this!

Just hit 19M+ monthly users.

In production with Internet Archive, HackerNoon, etc.

( https://github.com/amark/gun )

Scaling systems that have to run out of the browser by default is tough, but we're doing this on $0 in server costs, in javascript, with millions of users.

Hmm, how I handle UNKNOWN in our distributed system (10M+ monthly users running on $0 cost infrastructure, https://github.com/amark/gun ) is to say that originator is responsible for retries until satisfactorily ACKed.

This means all other nodes in the network (routers, relays, security, storage, etc.) can safely error or not recurse or not retry, without any loss in redundancy guarantees.

Either the operation did succeed silently, and it is OK for originator to try again (idempotency keeps this safe), or things never finished processing, and it is OK to try again despite being out of order (CRDTs keep this safe).

The _Future_ of the Internet, a post-scarce good, is not going to be a Profit Model Powered by The Extraction Economy.

It needs a fundamentally different economic model, that measures the gain in value added to the network, not the transaction (rebalance, distribution, etc.) of it: http://free.eco .

YaCy has been around for a long time, no?

What are other projects in this space?

I know Martti Malmi (Satoshi's 1st Bitcoin contributor) is doing some stuff (no tokens) in this space.

Anybody else/know/have links/talks/articles on other (no blockchain) decentralized search?

Eh, this article was disappointing.

It teaches good business practice of tackling 1 thing at a time & not over-engineering.

But simply using read replicas, caches, CDNs, etc. does not mean things will scale.

Actually, often this will break your app unless you write concurrency-safe code. Learning and writing concurrent code is how you scale, the "adding more boxes" is just the after effect.

For instance, we have about 10M+ monthly users running on GUN (https://github.com/amark/gun), this is because it makes it easy/fun to work/play with concurrent data structures, so you get high scalability for free without having to re-architect.

But learning something new is never an excuse for shipping stuff today. Ship stuff today, you can always learn when you need it.

Nice work!

I implemented an entire database in VanillaJS:

It runs in-production with 10M+ monthly users!

https://github.com/amark/gun ~13KB

I get a lot of hate for not-using-2ton-framework-of-the-month.

But I think prioritizing performance is worth the flack.

Seeing repos/articles like this deeply warm my heart, maybe us VanillaJS devs are not alone after all!

Does anybody have other favorite VanillaJS projects, that they could share?