HN user

zenogais

615 karma

Software developer, armchair philosopher, and sometimes writer.

Posts7
Comments250
View on HN

Like the author said this is completely unoptimized. The natural next step in optimization might be to profile and then SIMD optimize the slow bits in compression and decompression. This would likely produce a significant speedup and may even bridge the gap with lz4.

On UML 5 years ago

Agree with this and I think it's the crucial point. Where I've had the most success with UML-like tools and diagrams is early on in a project since its much faster to rip up and rebuild a few diagrams. The diagrams also help with getting consensus among stakeholders that this direction makes sense and doesn't have any glaring holes.

Later on you can drop the diagrams entirely and perhaps keep them as some onboarding documentation if they're still useful.

Mission Protocol 6 years ago

But it links to the Coinbase blog post in the "Inspirations" section on the front page?

It should work fine in the examples, but it won't work if you also attempt to use the -1 in the next slot to tell you which items are non-free.

This way you don't need to worry about keeping track of size and capacity either.

The only reason you don't have to worry about this is because std::vector handles it for you, at least in the code examples provided by the author. If you choose to go with a pure C implementation (which is what I'm trying out) then you will have to keep track of these.

The reason I suggested -1 is because when we iterate through bd->items, we need a way to know if it's a valid value or just "holed".

Yep, I was able to get an example using -1 as a sentinel working and passing fuzz testing.

Update: I was indeed missing something.

I think I've figured out roughly what the author intended, code below [0].

First, it looks like he's relying implicitly on data stored in std::vector. Namely vectors have both a capacity and a size. The capacity is total number of allocated elements. The size is the total number of elements stored actually stored.

Second, vector::resize won't reallocate until it runs out of capacity, but it will give you access to extra elements if you need them. So this is used to lazily re allocate while bumping up the size of the vector.

Both of these effectively make it "do the right thing" by leaning on the vector storing both size and capacity.

If you hand manage those values yourself you can get a pretty compact C implementation without a lot of code.

One last thing: Using a union here for the item_t is pretty much guaranteed to get you a segfault. The whole thing should really be a struct. This also allows for setting next to sentinel value if necessary.

[0]: C code for bulk_data_t example: https://pastebin.com/Tfcdt39h

I wanted to like this article because I'm been thinking about this a lot in the context of game development, noticed a few things. One thing I'll say from briefly playing with this - the code leaves lots out a looks ostensibly simpler than it really is. Would very much appreciate tips / pointers on this or a more fleshed out and working implementation of the code.

For the bulk data with holes code:

First, there's an initialization step that has to happen the first time you allocate your bulk_data_t. Namely, you need to iterate through every item in the list and set its next_free item to the item following it, looping the last item back around to zero. You also need to do this for all the items between the new size and old size every time you resize your item list.

Second, safe iteration over all of the bulk data doesn't seem possible without adding some sort of flag to indicate whether or not an item is free.

Am I missing something here?

Another one of these?

Bad estimation techniques - eg. guesses without a basis in historical data (most developers don't collect historical data) - are usually wrong. No one would argue this point.

If you do iterative development with intelligent planning and collect some historical data to use in estimates the game changes. First, most of the examples of "unexpected complexity" you gave can now be identified early and factored into your estimates. That leaves the 1% of truly unexpected stuff that you can then deal with as it comes up and factor into your future thinking in a post-mortem. Second, since you now identify risks earlier and can factor them into estimates, most of your work becomes estimable, and with historical data your estimates will improve.

There's even multiple process products around this idea already, but they're mostly used in the "safety critical" industry, see [1] [2].

[1] http://resources.sei.cmu.edu/library/asset-view.cfm?assetid=... [2]: http://www.sei.cmu.edu/tsp/

With all these competing, possibly contradictory moralities, how do you see this reliably making the world a better place?

PS. Understandably, we might not all even agree on what a better world would look like.

So the interesting thing here is the "majority assent" phrase. All the polls I saw included < 15% of Ethereum stakeholders / hashing power. Some included < 5%, but were taking by many (who already had a chosen outcome in mind) as definitive proof of community preference.

Really this shows that a small group of people plus a core team already leaning in one direction are probably enough to fork the network.

I feel like those fields with highest facility needs / costs would come last if at all. There are many fields that require pretty small amounts of resources for example: Computer Science (I did most of my research on a personal laptop, with other equipment costs <$5,000), Mathematics, Philosophy, Economics, Psychology.

All of these seem very possible to crowdfund with the ultimate goal of unhooking them from perverse incentive systems of typical universities.

Oooo sarcasm. You're probably right though, the old system seems to be working out pretty well. Besides, science is all about never questioning existing institutions right?

Perhaps offering some sort of crowdsourced funding mechanism and a reputation system would go a long way toward correcting some of these incentives?

For example, giving authors / organizations a Bitcoin address where they can receive funds from individuals / organizations who want to support their research.

Also, awarding reputation to authors based on the level of peer review their research has successfully undergone (number of peers, level of rigor, etc.), and conversely awarding reputation and funding to those who perform peer reviews. Allowing users to contribute to a peer review fund for individual articles or in general.

All that to say this is very exciting and opens up a lot of new possibilities.

You're also talking about systems 20,000x larger than what The DAO contract was [1]. The total DAO contract is roughly 2,000 LOC by my estimate [2].

IMO it probably won't be overly onerous to develop reliable contracts of this size using many of these techniques. It's just so few people have the discipline to delay implementation for design and planning at the level required when it needs to be bug free. Most software developers have no training in how to even begin designing and developing using these techniques.

[1]: Shuttle's primary flight software contains 400,000 LOC http://www.nasa.gov/mission_pages/shuttle/flyout/flyfeature_...

[2] https://github.com/slockit/DAO