dagheti

Dear Esther was funded by the Indie Fund. One of the founders of the Indie Fund is Jonathan Blow.

http://indie-fund.com/about/

Duplicate values that should be unique is such a common problem if you don't use constraints.

This is a good example of why ACID and declarative constraints are a very good idea for data management.You can do a bad delete from query if you don't have the proper safeguards (though querying ability of SQL lets you more easily preview your changes), but the initial corruption of the data is an easily avoidable problem.

ACID isn't a banking issue, it's an issue in any database where data integrity is important. Having written administration software for life insurance companies, you can be sure that we designed our software to rely on ACID semantics.

Without transactions, correctly writing the software would have been far more difficult, especially in situations where you need to rollback a transaction that is partially completed because of system errors.

Caring about branding things NoSQL isn't just not useful, it's harmful because it replaces really understanding things (like how a particular method works, and what it's good for) with a useless shorthand. Reminds me of a quote:

“You can know the name of a bird in all the languages of the world, but when you’re finished, you’ll know absolutely nothing whatever about the bird… So let’s look at the bird and see what it’s doing — that’s what counts. I learned very early the difference between knowing the name of something and knowing something.”

R. Feynman - "The pleasure of finding things out" 1981

I believe they are using SQL server, which keeps statistics on the most longest total runing, and longest per-run queries. It's a good idea to monitor this list and take a very close look at every execution plan that comes up. Many times you end up doing RID lookups due to non-clustered indexes that don't cover the requested data.

The name relational model is quite unfortunate, as it leads to confusion about what the model is (not much to do with 'relationships').

The key concepts are using sets to store data (making you think about your data in a way where duplication of tuples and orderings don't mean anything), and always using values instead of pointers to store your data and references.

I think that these concepts are very useful when it comes to managing data, but they seem to get lost in the all the talk about "relationships" and "sql" and "structured tables".

At least most of the time people stick to the Turing incomplete subset of the language.

One of the joys of working in F# is how the stronger type system combined with the type hover-overs allows for a sort of debugging-as-you-type that allows you to think with the help of the type system. It really helps unwind difficult problems and when there are errors you can easily see inline how to fix them. This is an impressive use of that facility, though it's even more useful when trying to solve a problem.

Oh I agree they are both very bad. Putting NULL in a non-NULL field is totally wrong.

However I really do think that inserting an unexpected default value is worse than inserting NULL into a NON-NULL field. The NULLs will cause problems, but they are problems you can see and resolve.

The default values are silent errors that will corrupt your data and be very difficult to recover from in the future. You can only guess which data was wrongly inserted.

Inserting a default value is even worse than inserting NULL. Now you can't tell what data was wrongly inserted if you tried to clean up the mess later.

I find that F#'s (close to OCaml) type system helps me think through problems and lets me get correct programs faster, that somehow just seem to work correctly the first time.

Usually when writing a program in F# I usually write out the types of the functions I need first, before worrying about how the functions work. Because the type system is so descriptive I can validate that a plan to solve a problem is going to work before I start writing code.

Because the type system is powerful, when I need to make a change, it is simple to determine where my assumptions are broken and make the correct fixes.

This is true only if we accept that AI trading is actually making the market more efficient. The 'wisdom' of crowds is often foolishness, and this could be even more true for a crowd of computers.

It is very possible that by using AI that looks mostly at numbers for trading is just destabilizing by relying too heavily on specific variables (the ones the algorithms can actually use as inputs) , and reduces efficient allocation of capital.

The pipe operator in F# |> is used in this way. When combined with currying it is especially effective:

  let double_then_sum a_list =
      a_list
      |> List.map  ((*) 2)
      |> List.fold (+) 0

I recently was tutoring a high school student who was instructed in their data-management course to create several graphs from the same set of data, comparing rainfall in different Canadian cities.

Unfortunately this data didn't make any sense to be plotted as a line-graph, a bar-graph, a histogram and a pie chart, as the comparisons didn't make sense for any but the bar-graph. It was if the students were actively being taught how to use graphs incorrectly. Creating pie charts when the entire pie doesn't mean the whole of anything I think was the worst culprit.

The key question is "Why" that needs to be asked. A customer may think they want to sort, select, and aggregate the data, but fundamentally "get the data" is very rarely the real reason for doing something.

That data is used to make a decision, and it's important that developers seek to understand exactly what that decision needs. Your users will often walk their way into a bad solution to their real problem because they will use their (incomplete) understanding of what is available.

So what happens when you create the follower but not followed index then realize you need it. Is there a way to create that new ColumnFamily without writing a object-by-object program that must traverse all the data? I suppose the concept of data locality doesn't apply, but what is the best way to do that sort of operation?

In an interview between Miyamoto and Iwata the genius of this first level is discussed

http://us.wii.com/iwata_asks/nsmb/vol1_page4.jsp

"But if you avoid the first Goomba and then jump and hit a block above you, a mushroom will spring out and you'll get a shock. But then you'll see that it's going to the right so you'll think: "I'm safe! Something strange appeared but I'm okay!" But of course when it goes against a pipe up ahead, the mushroom will come back! (laughs)" ... "At that point, even if you panic and try to jump out of the way, you'll hit the block above you. Then just at the instant where you accept that you're done for, Mario will suddenly shake and grow bigger!"

It turns out they panned it so you cannot avoid picking up that first mushroom that makes you big. They expected the first players to see it as an enemy, so they wanted to trap you into picking it up and then showing you that it made you more powerful.

Barely anyone notices that this was forced, but now everyone takes it for granted that the the mushrooms are good!

What is the source of this? If you look at http://en.wikipedia.org/wiki/Subprime_crisis_impact_timeline and search for subprime you'll see many cases of Freddie and Fannie buying or guaranteeing subprime activity.

- In 2000 Fannie buys $600 million, and Freddie buys $18.6 billion, and guarantees 7.7 billion more. - 2002-2006 the GSE's buy 38-90 billion a year in subprime mortagages

As far back as 1999 you can find stories stating that Fannie was being pressured into subprime:

http://www.nytimes.com/1999/09/30/business/fannie-mae-eases-...

Their activity appears smaller than the market at large, but it's not non-existant. It doesn't seem on face value that they were not allowed to invest. Is there some rule I am missing?

If found working in a ML language like Haskell or F# puts a really tight straight-jacket on code style, and is wonderful at forcing you to adopt a new paradigm.

When you go back to coding in other languages, you approach the same problems in different ways because you were forced to solve them in a strongly typed functional manner.

The question that this story raises is "Can we solve these hard problems of aging, thinking, and feeling if we had intelligence far greater than we do today?"

Are we meant to read the narrator's idea that they can even solve the problems of aging and diseases and voluntary ovulation through science and intelligence as naive or inevitable?

Just because we can imagine a virus or a team of super smart people solving these problems, doesn't mean in reality it actually is possible to do so given limitations of physics and human nature.

I guess it doesn't really matter if RMS meant this story to be a satire of a technocrat's fantasy or a example of how increased intelligence would solve some of our biggest problems but I think how people read the story will depend a lot on their ideas about what is ultimately achievable and what is not through intelligence.

Your criticism of this article is off the mark (though it may apply to other articles). The entire point of this article is that he is trying to replicate the exact same problem that Digg was reporting using a different RDBMS and discussing what that means.

Maybe his numbers are off and his technique is wrong, but this article isn't making a one-sized-fits-all mistake.

A great point, and it's important to remember that this effect also maps to public choice theory in government or excessive compensation of management in the non-profit sector (though statistically I believe it is lower than in private entities), where the incentives in some ways can make it even harder to prevent abuse.

It is possible that more interesting articles are posted in the afternoon due to the work habits of the people who actually create the content that is being linked, and posts linking these articles happening in a short time span.

It may be possible to correct for this by only including links that are for content that was already "old" at the time of posting?

The fact that a system requiring ad-hoc attributes wouldn't work so "hot" in a relation isn't a problem with the relational model. This difficulty comes from the application of the relational model bringing to light that such a situation is necessarily more complex than one without ad-hoc attributes, and you cannot treat such data the same way as you can treat data in a relation. The important lesson is that you cannot abstract this added complexity away without losing something else of value (data integrity and simplicity of reasoning).

This gives you more power when your situation isn't ad-hoc attributes (by providing more guarantees) and lets you model the ad-hoc situation with a EAV model if you must. SQL maybe should provide some more sugar for accessing that EAV, but it's not a fundamental relational model issue.

Anti-pattern one was critiquing using values in table definitions, the second was using attributes in table definitions. These are not the same thing. There isn't a similarity between values and attributes in the relational model, which is the whole point of avoiding key-value strategies which are confusing an attribute with a value.

A big part of why the modelling of data that you don't know the shape of in advance in a relational database is hard is because reasoning about data you don't know in advance is hard, and the model exposes that difficulty up front.

(Also, if like has its wildcard at the end of a string not the beginning, indexes can help you as they can look into the sorted trees.)

I think the article is (as titled) about the end of _a_ era, not _the_ era. I don't think this article is saying the RDBMS is dead (the author works for Vertica, which I believe is a column-store RDBMS). He is saying that the era of the "row based physical model for everyone" era is over.

Ideally a RDBMS should have logical-physical seperation, and be able to support both row and column based stores (or any physical strategy at all) in the same DB. I think the author is just saying that the next stage on this path will have various competing RDBMS physical implementation strategies as they have such big performance impacts.

I don't think that these different storage method DBMS's are the end point, but a way point to RDBMS technology that can mix physical strategies while keeping the logical relational model intact.

Isn't the argument for taxing height not that it better captures the effects of someones advantages, but that because it is taxing something you cannot control, you are removing the disincentives created by taxing something that society may want more of (income).

So while height taxing a more blunt instrument than income taxing, it doesn't distort people's choices in the same ways and avoids some of the deadweight losses. Unfortunately it probably will end up distorting people's choices in new and creative ways that we'd probably only find out about later.

I think part of the backlash is coming from developers on web-applications that have business models that require scaling to very large loads before they make sense. This puts a premium on keeping things very cheap (no heavy commercial SQLDBMS), and inherently scalable (hierarchical).

Web developers on these types of projects cannot either a: buy a "real" SQLDBMS, or b: charge their customers so their business model makes sense without a huge number of transactions. This means they are willing to suffer it out with solutions that don't provide flexibility or data correctness guarantees of a RDBMS.

This is just a guess, but it's quite likely that it is the rise of hyper-scale-intensive, speed-intensive, revenue-light business models that drives devs towards deciding SQL isn't worth it.

The issue with key-value store isn't that you can't create an application, its that the tools to control data integrity and access your data in unexpected ways are missing or will be alot of work to create. A simple example is that two way key-value pairs for the UID and username in this article. Now any update to one value (or is it key?) must update the other.

If your main concern is scalability, then Redis and the like can look very attractive, but we should be clear on what is being given up: data coherence and query flexibility. That may be fine on your app, or even many apps, but its important to point out what is missing.

Isn't it important though how good CPP is at being Haskell, and how good IO is at being C?

This seems to come back to that original Scala-is-not-functional article, which seems to argue that it isn't features or purity that makes a language "functional" or not, it's how good the language is at working with that style.