Lambdoku – AWS Lambda with Heroku-like Experience

https://github.com/kubek2k/lambdoku

by kubek2k • 10 years ago

128 41 10 years ago

github.com

oldopsguy• 10 years ago

Another nice project that helps to ease lambda workflows is https://github.com/garnaat/kappa

mullen2• 10 years ago

There is similar project called Zappa.

pip install zappa

Mizza• 10 years ago

Zappa actually uses some of Kappa under the hood, but with many many more features. Learn more here: https://github.com/Miserlou/Zappa

stcredzero7• 10 years ago

I'm wondering if it would be possible to create something like Lambda, but optimized for multiplayer servers with game loops? What if there was a platform that let you write pure functions to create player accounts, spawn game instances, and handle the transformation of player and game state from tick to tick? (You'd have to choose a particular architecture for synchronization with the client, of course. Also, before someone makes the mistake of assuming I'm as stupid as they are -- I'm not proposing that a function be transmitted, compiled, then evaluated at each tick over the network! Functions would be installed into virtualized servers.)

Clojure, Javascript, and Golang would be good languages for this. The environment possibly would let you inject and replace functions live, lending to a Light Table like experience, but while developing a multiplayer game.

keithwhor2• 10 years ago

You could, but the latency issues (network hops) would make the solution untenable. Function as a Service platforms are ideal for solutions where scalability is a huge concern, latency not so much. These functions are all running inside of containers that have been optimized to be "warmed up" extremely quickly, but that warm-up takes time, and the more robust the solution, the longer the warm-up.

You'd probably want architecture extremely specialized to a game loop, and probably your game loop specifically, if you wanted low-latency, high-availability FaaS for this purpose.

stcredzero7• 10 years ago

You could, but the latency issues (network hops) would make the solution untenable.

...ok... Did you read my comment?

These functions are all running inside of containers that have been optimized to be "warmed up" extremely quickly, but that warm-up takes time, and the more robust the solution, the longer the warm-up.

Yeah, of course that's not going to work.

I've basically built a specialized "container" system, but instead of using kernel virtualization, it's actually a specific Go "object" residing in an already running server process. These "containers" (really instances) start running in an environment where we already have a virtual server with a process with a server/game loop up and running. I have a cluster of processes that does this, and idempotently spawns an instance of star system in a multiplayer game consisting of 2^87 star systems.

www.emergencevector.com

Such a system can still be built on Docker style containers, it's just that those Docker style containers represent an entire sub-cluster of many server processes that can each host many "instance" containers which are where the functions that transform from tick n to tick n+1 reside.

You'd probably want architecture extremely specialized to a game loop

That is what I said in my comment.

and probably your game loop specifically

I'm saying that the game loop could be customized/built up by a developer by injecting functions into such a system. (Or, possibly, by injecting such functions into a dev version running locally, then exporting them to a server in the cloud.)

wolfgang422• 10 years ago

Your basic description sounds like LambdaMOO (written in 1990, the name is a coincidence).

The LambdaMOO server has a main event loop which handles user events, then schedules tasks which execute functions ('verbs' in LambdaMOO parlance) to modify the game state. Verbs are not pure per se but can be thought of as a transaction which executes in an atomic, consistent, and isolated manner. Verbs are attached to objects in the game, and can be modified in-game on the fly by players with their programmer bit set. (Programmers can only modify verbs on objects they own.)

The LambdaMOO language is a lot like JavaScript with a Luaesque syntax. It has prototypical inheritance, which meshes well with its concept of objects as physical things. For example, to get a new instance of a duck one would create a Generic Duck named "My Duck" ('Generic' being the conventional prefix for an object specifically intended to be used as a prototype.)

There was a time in the late 1990s when it regularly supported 300 active users at once (with a certan amount of lag); on modern hardware it could probably handle significantly more than that. If you want a modern scalable system then LambdaMOO itself is probably not for you; the server is single-threaded (in fact the whole thing is remarkably similar to NodeJS with async/await support) and state is saved in-memory and checkpointed periodically (hence the Durability missing from the above description of verbs).

However, the basic design (prototypal inheritance, in-database verbs, ACI[D] tasks, etc) is extremely well proven, and would lend itself well to being copied into a modern distributed system like AWS Lambda. As long as you had an ACID datastore to back it up, you could scale the task execution as far as you wanted, and with a bit of clever work you could probably come up with a partitioning scheme to make the database scale as well.

In fact, I've partially implemented this already; what's stopped me has primarily been that I'm more interested in the technical aspects and have no idea what I would actually use it for once I built it. If you have a use case for this sort of thing I'd love to hear about it.

stcredzero7• 10 years ago

There was a time in the late 1990s when it regularly supported 300 active users at once (with a certan amount of lag)

I had a pseudo-roguelike server first implemented in Clojure, then in Go that updated 12 frames a second. It could support 250 simultaneous users. It even had Conway's life as an area attack. I had it posted to Show HN.

All that's needed is pure functions, soft realtime, and good tooling and APIs. ACID takes some doing, in terms of implementation and will also make scaling complicated, as you mention above. Inheritance? Nice but not necessary. Just let developers inject functions into a game loop, then tell them their load and when their simulation tick rate starts to drop.

If you have a use case for this sort of thing I'd love to hear about it.

Yes. Something like that could be packaged into MMOGAAS -- Massively Multiplayer Online As A Service.

buttershakes• 10 years ago

I didn't log in for a while and my account was deleted circa 1999, I was pissed, since I'd had it since about 93, although there was little I could do about it. What a great mud though. :)

inopinatus• 10 years ago

If I was building a MUD or MMO from scratch today I'd try with Elixir (the language built on Erlang). Functional programming with hot code replacement, implementing the actor model for distributed, fault-tolerant, soft-real-time massively concurrent applications, does that sound similar to what you're looking for?

Growing that into a FaaS product would be hard though; Erlang processes all share a trust domain, from the infosec point of view, so containerizing each function/process for invocation lambda-style on otherwise shared infrastructure may prove difficult.

stcredzero7• 10 years ago

implementing the actor model

Each of my server processes in Go looks a little like an Erlang actor. There's a switch statement where a queue of input messages is processed, then each instance is run in turn. Instances also have a similar structure and maintain a "space" using an R-Tree. When it comes to efficiency, game loops are still a great way to do concurrency. Game loops per-instance allow you to sidestep many pitfalls of multi processor concurrency, much as per-process GC does with Erlang.

mifreewil• 10 years ago

Related: Apex https://www.github.com/apex/apex

coenhyde• 10 years ago

This is great. I was going to create something similar to manage lambda functions across environments. Awesome to see you're already using the newly released environment variables for lambdas.

kubek2k7• 10 years ago

thanks! - please give it a try and let me know in the issues if something doesn't work as expected.

keithwhor2• 10 years ago

Hey! Great project. We actually built stdlib [1] to capture an experience similar to Heroku's for building microservices, and we'll be presenting at AWS Re:invent on Tuesday [2]. Our workflows with our public API are accessible via our open source library, would love contributors such as yourself (and anybody else that finds it interesting!). We have a diverse team working on the project - Xooglers, support from early Heroku team members, and even some YC alumni lending a hand. :)

[1] https://github.com/poly/stdlib

[2] https://www.portal.reinvent.awsevents.com/connect/sessionDet...

josegonzalez• 10 years ago

For a second there, I thought someone implemented a push-to-deploy experience for Heroku (with some of the other Heroku niceties). I actually wrote a blog post about that this week: http://josediazgonzalez.com/2016/11/21/push-to-deploy-lambda...

kubek2k7• 10 years ago

really good read. thx! :bow:

I didn't plan to do the push-to-deploy thing, mostly because that would totally complicate things. The rationale was more that we would use the CI that would automatically build and deploy things to DEV env that would be then promoted around.

charlesetc• 10 years ago

You could easily prefix your special environment variable with 'lambdoku_' so you don't have to fear that people could use it..

kubek2k7• 10 years ago

:bow:

https://github.com/kubek2k/lambdoku/issues/8

michaelbuckbee• 10 years ago

Very cool project, but it has me thinking what/if it would be possible to build out more of the infrastructure management that Heroku has to make it even easier (domain setup, addon store, etc.)

pepicon• 10 years ago

Lambdoku = lick from ass in portuguese

fiatjaf2• 10 years ago

ahahah, I thought things wouldn't get worse than "Heroku" and "Dokku".

fiatjaf2• 10 years ago

The guide on the README doesn't tell me how am I supposed to push and update a function.

Also, does it help with dependencies or something like that? Does it manage packaging files?

kubek2k7• 10 years ago

the initial idea was that CI deploys lambda, and you promote it from stage to stage. I might consider upload sometime.

From what you write I think apex would be better suited for you

ohstopitu• 10 years ago

How is this any different from Serverless? (not trolling, I could not find any differences in the README)

kubek2k7• 10 years ago

I don't mind being trolled :)

Serverless is way much more advanced machinery than this tool. My point was to create a tool, not a whole ecosystem around AWS Lambda.

When it comes to serverless - my main concern (as it was with Lambda without env variables), is with storing configuration data along the code in the repo. Additionally the reason for creating was mostly pipelines.

brilliantcode2• 10 years ago

what I'd really like is ability to take a swagger spec and click a button to deploy it on AWS Lambda.

This is great for internal facing REST Api. I don't know about scaling but hypothetically serverless means it should handle public requests fine.

kubek2k7• 10 years ago

I guess you mean the API-Gates + Lambda. Not sure - I think thats something that should be handled by some separate tool (APIGatesgger? ;) - lets hope its not another dirty word)

brilliantcode2• 10 years ago

that's right swagger -> api gateway triggering lambda functions.

I was using Chalice but seems like Zappas is another one a HNer pointed out. I couldn't get Chalice to work and sort of gave up on the serverless-api-gatway-lambda idea.

tjholowaychuk• 10 years ago

Looks cool! I like the release message stuff, I wanted to add similar to apex(1) at some point, not having a clue what the releases actually are in Lambda makes rollbacks a little tricky.

kubek2k7• 10 years ago

thats a wonderful improvement of apex, thats for sure :)

aghillo• 10 years ago

Has anyone characterised server less architectures? I'm interested in testing / exploring the limits with some predictive modelling work and implementing (mathematical) functions as FAAS? No doubt, it 'depends' (e.g. nature of data and its locality or otherwise) but interested to hear if anyone is looking at very fine-granularity server less systems?

stcredzero7• 10 years ago

You acted like no one offers a PaaS for MMOs (and gave it a ridiculous name).

I was talking something specifically like Lambda, but where pure functions could be injected into an already running game loop. The name was intentionally ridiculous. Also, you suggested that my proposal would suffer from the latency of lambda requests going over the network, which shows you didn't read my original comment.

with your cloud functions being called on triggers

Yeah, I pretty much covered that this was not what I was proposing, when I wrote "(You'd have to choose a particular architecture for synchronization with the client, of course. Also, before someone makes the mistake of assuming I'm as stupid as they are -- I'm not proposing that a function be transmitted, compiled, then evaluated at each tick over the network! Functions would be installed into virtualized servers.)" As far as I knew (which might be out of date) Photon was developing something that would be suitable for a persistent world MMORPG, but not quite ready with it. What I'm proposing would be suitable for an MMORPG, not just MMO game sessions.

dang• 10 years ago

We detached this subthread from https://news.ycombinator.com/item?id=13062880 and marked it off-topic.

Lambdoku – AWS Lambda with Heroku-like Experience

Related Stories