Amazon Kinesis Available For All Customers

http://aws.amazon.com/kinesis/

by seaghost • 13 years ago

72 19 13 years ago

aws.amazon.com

This looks interesting at first glance, but to be honest, I don't fully get it from the description. Is it some kind of Storm/CEP/Graphite combination? Where does the actual "processing" happen? Do you have rent additional EC2 resources for your workers? It looks like a potentially powerful and interesting service, but the documentation could be a little more clear and maybe include an example of a DAG layout for a typical application.

Kequc2• 13 years ago

When I think of streaming my first thought is that this is a cloud service that processes audio and video in real-time. But none of the example use cases mention that is a possibility. I'd love the ability to stream and convert audio then forward converted files to S3 in a simple way.

Currently it involves running an EC2 instance which only converts after the audio is received in full. Then when it's done, transferring it to S3 is... slow.

n3thin• 13 years ago

you might need to look into http://aws.amazon.com/elastictranscoder/ if it can satisfy your use-case

Kequc2• 13 years ago

The audio file must exist somewhere, it offers no live audio support. So the bottleneck would be at the beginning rather than the end.

warptera1• 13 years ago

Take a look at the Zencoder (YC W10) API: http://zencoder.com/en/

It's since been acquired by Brightcove. [Disclaimer: I work there.]

bgruber• 13 years ago

amazon says real-time, but i'm 99% sure they mean (comparatively) low latency.

dzhiurgis• 13 years ago

How many visitors you'll need to have to make some sense out of their behaviour?

nnx• 13 years ago

It sounds exciting but I don't fully get what this is good for. It sounds like a stream (of "records") on this would be the equivalent of a queue and messages in Amazon SQS? only Kinesis is lower latency and billing is both capacity-based and quantity-based rather than only quantity-based in SQS?

socialist_coder• 13 years ago

I can't wait to rip out my "client => server => SQS => queue processor => S3 => redshift" flow and turn it into a "client => kinesis => redshift" flow.

Web scale analytics capturing just got way easier.

itsmeduncan• 13 years ago

We just wrote something very similar to this using Kafka which replaced a very similar system you have. Do it. We would have used Kinesis if it was around.

metabrew• 13 years ago

As I understand it, this is very similar to Linkedin's Kafka.

To understand how you would use it, read some of the kafa docs: https://kafka.apache.org/documentation.html#introduction

dkersten• 13 years ago

From my reading of it (and my use of Kafka in the past, as an input to Storm), Kafka is a message queuing system, where Kinesis is a message processing system. That is, Kafka is more similar to SQS or RabbitMQ and Kinesis is more similar to Storm or S4.

Of course, I haven't used kinesis and have only glanced at the docs, so I could be wrong - it could very well be that Kinesis is the glue between data input and data processor rather than the data processor itself, in which case, yeah, it looks a lot like Kafka.

lhaussknecht• 13 years ago

Looks similar to my logstash -> elastic search -> kibana workflow.

6cxs2hd6• 13 years ago

I wish Amazon hadn't given up even trying to do RESTful APIs. Nowadays they just shrug and bang it out RPC style (payload is JSON instead of XML, but the song remains the same).

e.g. How do you delete a stream?

Is it a DELETE request for, say, /streams/<stream-id> ?

No, it's a POST to "/" with an entity:

  {
      "StreamName": "string"
  }

Sigh.

dalore• 13 years ago

Not all proxies support DELETE and PUT. But you're right, it should support both.

mfenniak• 13 years ago

Amazon success is a strong indicator: RESTful APIs don't really matter. You can build a very successful business without focusing on the format of your APIs, as long as you build something valuable.

atwebb• 13 years ago

It helps if you're already one of the biggest players. A smaller company may be able to make a more attractive offering with REST.

chollida1• 13 years ago

Does anyone have any experience using this technology? I'd assume it works best for post processing or pseudo real time data feeds( ie within a couple of seconds as opposed to sub 20 millisecond reactions).

Where do people usually store the processed data from this system?

I can see something like this being useful for things like post trade analytics or post day analyzing the efficiency of an given algo.

We use an inhouse CEP to do this sort of work currently but I'd rather rent than buy servers for this offline processing.

By the way, if anyone is in the Toronto area and interested in working for a hedge fund my contact info is in my profile

Amazon Kinesis Available For All Customers

Related Stories