codeslinger

Disclaimer: ex-AWS here.

This article ends up making a compelling case for DynamoDB. It has the properties he describes wanting. Many, many systems inside of Amazon are built with DDB as the primary datastore. I don't know of any OSS commensurate to DDB, but it would be quite interesting for one to appear.

"Every transaction can only be in one shard" only works for simple business logics.

You'd be quite surprised at what you can get out of this model. Check out the later chapters of the DynamoDB Book [1] for some neat examples.

[1] https://dynamodbbook.com/

Kamailio is what is called a Session Border Controller. Its primary purpose is to provide protection and some lightweight filtering for the media servers/PBXs behind it. Once you want "advanced" features like voicemail, parking, hunt groups, three-way calling, etc, you will need to use a PBX like FreeSWITCH (recommended) or Asterisk (not) behind it anyway. If you're only running a single machine, an SBC isn't really worth the trouble.

I always thought "intelligence" was defined as the capacity to learn. Websters' first definition of "intelligence":

capacity for learning, reasoning, understanding, and similar forms of mental activity; aptitude in grasping truths, relationships, facts, meanings, etc.

seems to partially bear that out.

Its not clear what definition Microsoft is using, but I don't see any real learning going on here. These models are painstakingly hand-trained by lots of humans over a significant period of time. The model itself is not "learning" anything, the humans are.

Write a love song from Luke to Leia.

...It proceeds to produce a poem, not a song, and without any references to Star Wars that would indicate that it "knows" who either of them are...

This post is very long on hype and very short on fact or supporting evidence. Many things are attributed to GPT models' capabilities that are not in evidence, so its hard to take the content here seriously.

Link is busted: http://mlbase.org

You may have a problem in your math. A mean of 4.3us with a stdev of 10.9us should mean that your 99% should be over 26us, unless it's super smooth with a few huge outliers.

Also, are you sure you didn't mean ms and not us? Seems more likely for an HTTP service.

Anthony Eden has a project for this from a while back: http://activewarehouse.rubyforge.org/ Not sure how well its maintained at this point but it looks like a pretty good mapping of the concepts to me.

Noir is not bad. I've been using it for a side project. Its no Rails, but its the best option I've found for Clojure at this point.

http://webnoir.com/

IANAL, but: It would appear to me that the author doesn't understand the GPL very well. If my code is BSD licensed and yours is GPL, I can submit pieces of my code to your project but if I take pieces of your code into my project, my project is now subject to the virality properties of the GPL and itself becomes essentially licensed under GPL. This is where the author seems confused: if I subsume GPL code into a BSD-licensed project, my project becomes licensed under GPL. Given that I chose the BSD license for my project I'm assuming it would now be obvious to the OP why this might be onerous.

The article says they are using Office Communicator, part of the Office suite, so I'd imagine they are going full-bore on that. Wonder if they are using SharePoint for the wiki, too.

"DSP" is a term for an service that connects to multiple ad inventory sources (ad exchanges, ad networks, etc) and allows an advertiser to upload their creative media and targeting criteria once and then be able to buy on all the different inventory sources that the DSP integrates at once. The "demand-side" refers to the fact that DSPs (Invite Media, Turn, DataXu, etc) are used by advertisers to buy inventory, as opposed to the "supply-side", which are the publishers (web sites) or their representatives (ad networks, Rubicon, Pubmatic, etc) that have the inventory slots which the DSP is to fill. In display advertising, "supply" is code for people (they call it "audience") and "demand" is code for money.

All of this works via the magic of RTB (real-time bidding) APIs which certain ad exchanges and networks implement that allows a DSP to see and bid on each individual impression right before it is rendered in real-time. It works as follows:

* An end user loads a web page in his browser with an RTB-enabled ad tag on it

* Ad tag loads and calls back to its origin (e.g. Google AdX) for a creative to show

* The origin initiates a real-time auction with all of its partners (DSPs or Appnexus(+)) to find a creative to show

* The origin will send each partner a bid request containing such information as the origin IP address, geographical information, site on which the tag is placed, etc

* The partner searches its inventory for a matching creative(s)

* The partner picks one to bid on and chooses a bid price to offer

* It sends that back to the origin

* The origin chooses the winner of the auction and then:

* Sends the winning creative tag back to the publisher's ad tag to be shown

* Debits the winning partner's account by the amount bid (or something less, depending on the auction model)

* The ad loads in the browser and the end user is pissed off that he has yet to enable AdBlock

As for who offers RTB APIs, those are typically ad exchanges, SSPs ("supply-side platforms", also known as "yield optimizers", e.g. Rubicon, Pubmatic, AdMeld) and some ad networks. The largest ad exchange, Yahoo! RightMedia, has yet to offer an RTB API yet, although they are purported to be working on it.

DSPs will typically use the information provided in the bid request to help aid the bidding decision but also they rely heavily on third-party data sources for extra information about the user and the publisher domain/URL in order to maximize the likelihood of offering a relevant ad for the cheapest price. Proximic, BlueKai, Exelate, Bizo, TARGUSInfo and many more offer specialized data streams for this purpose.

DSPs serve as aggregators of RTB-enabled inventory sources for advertisers so that they can buy across all of them without having to manage N sets of creatives, N sets of bids, N sets of targeting criteria, N relationships, etc. This allows advertisers to concentrate on the audience they want to reach and not where to find them. Having said that, the larger brand advertisers will typically use an ad agency and not deal with DSPs directly; the agency will use a DSP on their behalf and charge a markup for the service. This is typically called "execution" in the same vein as "execution of trades". DSPs are the algo traders of the online ad world.

(+) I mention Appnexus separately because they are not technically a DSP. While they do connect to the RTB APIs of many of the inventory sources and can serve as a DSP, they are a primarily a platform on which to create a DSP or some other real-time-enabled ad processing engine. Many of the DSPs out there are just whitelabeled Appnexus, in that they use Appnexus' bidder and UI for their "DSP". Appnexus also has exchange-like properties, e.g. they have inventory exclusive to themselves (i.e. what used to be called Microsoft AdECN) that is only available by integrating with them/using their platform.

(++) Full disclosure: I was Invite Media's first Chief Architect and I continue to work in the online ad business today.

Become a lawyer. We don't need any more unskilled labor running around.

Why have SSDs killed this question? Seems like you could tweak the amount of data on disk and/or the size of the values and/or the requests/second requirement and keep using it. Also, there are no 1TB SSDs available at this point, so you'd still have to assume spinning platters for this, no?

Anybody have free access to this?

There are plenty:

* Monetate * Infuse Media * Venmo * Vuzit * MyYearbook * Relay Network * PackLate * ClickEquations * Portico

And these are just the ones I could think of before I got bored with this question ;-)

Also, First Round doesn't really invest in a lot of Philly startups at this point. They do know most of the people in the area, though.

I have all the books currently out now on Scala and I think "Programming Scala" by Wampler and Payne is the best one for learning the language: http://www.amazon.com/Programming-Scala-Scalability-Function...

Steer clear of the Artima book: its huge, out of date and not written particularly well.

You might be interested to know that this method has been employed by Cloudmark for many years in their collaborative spam filtering network: http://portal.acm.org/citation.cfm?id=1105677

However, it works so well for them because the consensus on what is "ham" and what is "spam" is nearly universal enough that it can be relied upon to be agreed with by the majority of agents (people) the vast majority of the time. For news links, this will unlikely be the case unless you are actively attempting to create a news site where only a very few have power over what is shown and what is not.

Also, the Cloudmark technique chooses very few people to reward and this is on purpose. They are sparing with the reputation points and quick to downgrade on controversy in order to prevent a high-reputation individual gaming the system. Your news reputation system would likely have to incorporate a similar technique in order to be similarly robust. However, in doing so, I would imagine that most people would not see their reputation ever rise and would likely stop using the system. (due to lack of the "incentives" of which you speak) Cloudmark gets around this because people are already marking emails as spam or not in their email clients already, so there's no marginal cost associated with training Cloudmark's network or building their own reputation.

Its an interesting direction you propose, but I believe you'd be exposing your user base to scale-free network effects in which only a small few would gain sufficient reputation to "move the needle" and the rest would wallow in relative obscurity. This would then essentially replicate the Slashdot of 2000 AD but with the marketing message saying that it was "fairer" because of all the "incentives" people have.

You might want to think of how to structure a social news site as a kind of online auction, choosing a structure in which truth-telling is a dominant strategy, instead, such as http://en.wikipedia.org/wiki/Vickrey%E2%80%93Clarke%E2%80%93...

Except for email, that sounds a lot like Friendfeed ;-)

You know, I think they might have been thinking along the lines of a "distributed Facebook"-type thing when this came out. Anyone looking to replace Facebook should probably learn the lessons of Wave's failure before embarking on their own project.

I was thinking along the same lines, actually, in terms of a "distributed Facebook"'s similarity to Lotus Notes. However, one of the big niceties of SaaS services is that you don't bear the cost of storing and transforming all the photos, videos, etc that you upload. Replication of this data would also get quite expensive as you gained followers. And then there's the problem of bandwidth in the US. Seems to me there are some infrastructural problems with a distributed Facebook model that require some more thinking than just "lets run a server on people's home machines" (e.g. Diaspora)

The big thing in replacing Facebook isn't "distributed" or "central", its "what could make people switch?" Facebook came to prominence because of the elitist factor associated with top schools, the "poke" concept and its lack of "cesspool-edness" compared with MySpace.

At this point, I personally don't see how you could compete with Facebook on functionality, so what is your 10x improvement that would make people switch from Facebook to your new thing? People don't seem to be outraged enough about the privacy debacles as of yet to be making enough noise that a competitor could capture a large share just from that issue alone. (e.g. this is likely why Diaspora will fail, at least initially)

Knocking Facebook out of its top spot is much more of a economics (i.e. incentives, etc) and marketing issue than a technical one, IMO. People won't switch to a me-too; there has to be some compelling reason to move.

I couldn't agree more. XML sounded like such a great idea for this back in '99. Attempted implementation will disabuse you of that feeling very quickly ;-)

Wave is probably too complicated for the average Netizen. I've personally only found it useful as a sort of real-time chat. Haven't checked my inbox in months at this point.

If you could trust a central service to be good, we wouldn't be talking about replacing Facebook, IMO.

I don't want to sound like an ass or ungrateful or anything, but the FSF needs to wise up. People won't switch until the alternative is a lot better than what they currently have. Why would a normal person care about the file format of an attachment? Hell, how many people even know the format of the files they are working with? (I mean the extensions, not the file layout or anything) They just care about being able to open it. They have a raft of better shit to be doing and that's how it should be.

tl;dr: Make OpenOffice 10x better than Office and you won't have to cajole people to switch. Don't do that and we'll ignore you like we've been doing.

Some interesting points from the whitepaper regarding the above:

"VoltDB provides data access via stored procedures in order to minimize round trips between the client and the server. Stored procedures are written in Java - each procedure is a single Java class. Within the stored procedure definition, users interact with a VoltDB database via an OLTP-focused subset of standard SQL, including many-way joins, group-bys, ordering, aggregates, common arithmetic expressions, limits, etc."

"Distributed execution planner assigns work to engines, which operate exclusively on their assigned partitions"

I'm assuming this means they've reduced join functionality in order to get a much better scale-out profile. I wonder what this means for people writing apps on top of it.

Just as I mentioned in today's thread on FathomDB, I'd be interested in what kind of workload scales linearly to 50 cores. I'm guessing its not very join heavy. H-Store's main advance, IIRC, was that it pushed many points of RDBMS engine operation into RAM. While this would indeed make the engine much faster and eliminate some historical cruft, I don't remember any fundamental breakthrough that would make distributed joins scale out very well. Anyone have any insights on how they are handling joins?

I'm quite interested in how you're handling joins in a manner that is scalable. What tradeoffs did you have to make? Do you support full join semantics, or are some types of joins not supported? Its not that much of a stretch to scale out an RDBMS not using any joins, but I can't bring myself to really take your claims seriously until you elaborate on how you handle joins (if at all).

I would argue with the assertion that he doesn't look at other products when evaluating Facebook. Its clear they've been copying from Twitter and FriendFeed for years, so I don't know where the Boz is getting that from.

The attack isn't using brute force. The defense is against rainbow tables, not brute force.