A RESTful Micro-Framework in Go

http://dougblack.io/words/a-restful-micro-framework-in-go.html

by Nogwater • 12 years ago

234 63 12 years ago

mjallday• 12 years ago

Languages live or die on their libraries.

This is great. I sat down to write a quick api in go about three weeks ago but couldn't find a framework that I wanted to use. This looks like it has a decent abstraction and libraries like this really help language adoption.

Much thanks to the author of the library.

skj• 12 years ago

Wonderfully small, easy to grok, and put into practice.

dark_ph0enix• 12 years ago

I'm yet to test this, but any idea how it compares against Martini?

jjsz• 12 years ago

I don't understand why, when questions like these are asked, they are never answered...

macu• 12 years ago

It doesn't have the same aim as Martini, which provides more of a full middleware manager.

Sleepy is a simple wrapper for RESTful resources that takes a few steps out of the handler work, e.g. internally handles the work of encoding a response to JSON and then sending it.

Nice to see new developers delving in, albeit a little clumsy at first :P

_ak• 12 years ago

Nice, easy to use, really straightforward code. The only thing I would have done differently is make sleepy.Api implement http.Handler so that it would easily integrate with other libraries that build upon and play nicely with net/http.

swah2• 12 years ago

Kinda off-topic, but would you write a real-time web application in Go today?

tptacek2• 12 years ago

We did. Worked well.

swah2• 12 years ago

I'm glad to read that. May I ask what technology did you guys use? SockJS' Python implementation works well (well, I haven't deployed yet...), but the Go port isn't that mature IIRC.

aninteger• 12 years ago

Worked or did you mean it works? If you meant worked can you further expand on why it no longer works well today?

jchrisa• 12 years ago

The Go runtime is still being optimized. For instance we found that the upgrade to Go 1.2 made a big difference for some of our performance tests. So it's nice to get surprises like that but also shows that the ecosystem is still young.

tptacek2• 12 years ago

This looks like great work, so probably this comment is a little off topic, but here goes anyways:

Every web framework I've seen for Golang seems to do a competent job of tackling the easiest part of designing a web framework, which is routing requests to handlers.

Actually, the built-in net/http library already does a serviceable job of handling requests, and if you understand closures, filtering them and maintaining request state as well.

The thing no framework I have seen so far addresses is persistence. SQL persistence in Golang is a bit of a nightmare. It's raw SQL, and the database libraries are fussy.

The thing that makes Rails so easy to write in, especially for newcomers, is ActiveRecord. As far as I can tell, nothing like it exists (in a stable form) for Golang.

chimeracoder3• 12 years ago

I'm going to chime in, since I've been using Go as my primary language for over a year, and I'm currently working on a major (production) Go project that others might see as well-suited for Rails (in fact, the early prototype was written in Rails[0])

The thing no framework I have seen so far addresses is persistence. SQL persistence in Golang is a bit of a nightmare. It's raw SQL, and the database libraries are fussy.
The thing that makes Rails so easy to write in, especially for newcomers, is ActiveRecord. As far as I can tell, nothing like it exists (in a stable form) for Golang.

I wouldn't recommend newcomers to Go use SQL right off the bat. Document-based storage (like CouchDB or MongoDB), or even key/value (Redis, etc.) is much more aligned with Go's concept of structs, and all three of those databases have strong library support in Go. If you're just looking to dabble in Go and just happen to need a database (and it doesn't matter which), I'd recommend playing around with those first. Even Neo4j works very well with Go, though as of January 2013 (when I last used it), the only Go library for Neo4j was third-party, and not very well-documented.

That said, once you know the Go language, using SQL (either with or without an ORM) is straightforward, if not as convenient as ActiveRecord. jmoiron has a good blog post[1] comparing the good, bad, and ugly of Go ORMs, so I won't try to top that here - rather, I'll share my experience.

(As pointed out elsewhere, ORMs are not universally liked as a general concept, so I'm writing this comparison assuming that one actually does want the features an ORM provides - naturally, if you prefer prepared SQL, Go already provides perfect support for that.)

Gorp is definitely not as "magical" as ActiveRecord, but it certainly does the job. A number of the ways in which it differs from ActiveRecord have more to do with the (strongly) static vs. dynamic difference in type systems, though some are specific to Go's architecture.

There are a few odd features here and there that are still missing (for example, I'm actually not sure if there's a wrapper function for "INSERT IF NOT EXISTS" yet, though it's not a huge bother).

I do end up spending more time thinking about the queries that I'm using, compared to Django or Rails. There is some additional verbosity - I have to write some helper functions that I wouldn't otherwise have to. On the other hand, I know exactly what queries are being executed at all times, so it's easy to identify potential bottlenecks.

In short - I agree that ActiveRecord is more convenient than anything that currently exists in Go. On the other hand, it's reasonably advanced enough that relative newcomers to the language[2] shouldn't have a problem with it. So far, the biggest bottleneck in development of this application has not been with the SQL bindings, even though it would be nice to spend a bit less time on that front.

It hasn't had as much time to develop and mature as ActiveRecord have, but those of us using it in production have accepted those tradeoffs.

[0] If you're wondering why we didn't continue it in Rails, we had to scrap the original codebase for a number of other reasons, so given that we weren't keeping any of the original code, there was no logical reason we needed to use the same language.

[1] http://jmoiron.net/blog/golang-orms/

[2] As opposed to newcomers to programming in general

icholy2• 12 years ago

IMO having some ORM automagically construct queries for you is a pretty scary idea. Obviously it's not a good idea to have SQL scattered all over your codebase (we've all seen it). But including SQL in your code isn't hard if you keep it in one place and abstract it properly.

mbreese• 12 years ago

I find the lack of ORM a feature in a web framework. Personally, I find them frustrating and an extra layer of abstraction that just gets in my way. Then again, I also don't mind writing queries. I usually end up using my own lightweight set of helper functions to manage queries, regardless of the language.

I still have Hibernate flashbacks from back in the day.

masklinn• 12 years ago

I find the lack of ORM a feature in a web framework.

There's quite a lot of stuff between "write raw SQL queries" and "full-blown ORM". http://docs.sqlalchemy.org/en/latest/core/tutorial.html for instance.

carbocation• 12 years ago

I agree with you re: the lack of ORMs in golang. I actually enjoy writing and profiling my own SQL, but if I want to prototype, ORMs can speed up the process.

I will say that some of the parts of SQL in golang that I find most annoying, such as populating structs, are handled very nicely by jmoiron's sqlx library. [1] This is not an ORM, but it's an extension to sql that removes some of the most annoying boilerplate nevertheless.

1 = https://github.com/jmoiron/sqlx

gtaylor• 12 years ago

I hadn't seen sqlx before, but this looks super convenient. Thanks for the tip!

jamra3• 12 years ago

This looks like an awesome library. I like how he does this:

type Person struct { FirstName string `db:"first_name"` LastName string `db:"last_name"` Email string }

to map columns to structs.

jamra3• 12 years ago

What's wrong with using raw sql classes? A web framework should be database agnostic.

I agree with the usefulness of a better SQL library though.

Edit: Wow that was worded poorly. What I meant was, what do you find bad about the SQL library?

shadowmint• 12 years ago

How about having two statements for running queries, Exec, and Query, but unlike other database engines where you might expect to be able to execute a PQ query that returns values like:

    INSERT INTO PERSON (NAME) VALUES ('JACK') RETURNING ID_PERSON, CREATION_TIME

...no, actually, you cannot perform a query for an INSERT statement, even if it returns values.

This isn't a database driver issue even, apparently. It's just the way the database API was defined. Query is not for inserts.

Or the fact that if you run a query (eg. CREATE TABLE blah...) you get a result set that you have to call Next() on to flush the query to the database, even if you close the connection.

...because obviously, running next on a create statement is what you would do.

Or how you have to import a singleton to load the database driver you want.

    import (
	_ "github.com/lib/pq"
	"database/sql" <----- Previous import has side effects.
    )

How is that idiomatic go? -__- It's totally a hack.

It's nothing major, but the database/sql library api just doesn't feel right.

It's messy.

bliti2• 12 years ago

Yes, SQL is still spotty in Go. I wrote a medium sized program with it and the learning curve was fairly high. Documentation for the database libraries is mostly nill. Though I'm not sure if it would benefit from an ORM. This is a systems language. I'd rather have it use interfaces to access data from a database than have it deal with it itself. That way I keep it tidy and fast. Which is Go's aim.

icholy2• 12 years ago

I find the docs very comprehensive. http://golang.org/pkg/database/sql/

sdegutis• 12 years ago

ActiveRecord may be a stepping stone for those just starting out, but for serious, long-living, evolving applications I firmly believe it does more harm than good.

Back when I did Rails contracting, cleaning up code which conflated database access with business logic was quite possibly what was keeping us in business.

JonnieCache• 12 years ago

If anyone is planning to write a golang ORM, you should have a look at sequel: http://sequel.jeremyevans.net/

I'm not aware of a better ORM anywhere. The simplicity, and the way it separates the model classes from everything else, is sublime. It does pretty much everything activerecord does and lots more, without all the layers of abstraction.

cronos• 12 years ago

Why exactly do Resource's methods take variadic argument? In what scenario can there be more than one?

Also, small remarks about code: You define type HandleFunc in your package which is identical to http.HandlerFunc. Looks like it can be removed. Some error logging (rw.Write, json.Marshal) would be useful as well.

dougblack• 12 years ago

Author.

Great points. They don't need to be variadic. I misunderstood the type of url.Values. And I didn't see http.HandlerFunc!

I eschewed error logging in the interest of simplicity.

Thanks for the corrections! Updating the article now.

mitchellh• 12 years ago

This looks like a great resource! A bit of a rant (not directed at the person who wrote this, because the link at hand is great):

Writing APIs in Go is not the hard part. There are a ton of options for routing basic URLs to handlers. There are a few areas that are a nightmare, total total nightmare:

* SQL - The raw database/sql package in Go is very low-level. It is a terrible experience to use in a project that makes anything more than a handful of simple queries. I'm not asking for an ORM, but it would really benefit the community if there was a SQLAlchemy style (the non-ORM part of it) library for Go. It should be MUCH easier to make basic queries to SQL and get out results.

* Templating - The built in "html/template" package is just not good enough for a web app when compared to something like Jinja (Python). "html/template" gets points for being simple, and actually its not a complaint against "html/template": I think there just needs to be a more feature-packed templating package like Jinja for Go. "html/template" doesn't support things like inheritance, didn't even support comparison operators until Go 1.2, etc.

These two problems have made it so I've pretty much given up web apps in Go for now. Services in Go are _amazing_. But if you're trying to get a full blown web app (again, not the point of the linked article above), then you're gonna have a bad time.

chimeracoder3• 12 years ago

"html/template" doesn't support things like inheritance

I don't want to get into a debate about what "inheritance" means, but html/template does support template composition and nested templates, which seem to address the same use case.

These two problems have made it so I've pretty much given up web apps in Go for now. Services in Go are _amazing_. But if you're trying to get a full blown web app (again, not the point of the linked article above), then you're gonna have a bad time.

I'm sorry to hear that, especially since writing a "full-blown web app" is really not difficult in Go at all, once you know what you're doing.

I've actually been working on Jetpack[0], a sample web application in Go to demonstrate how straightforward this is, once one understands how it actually works. (Disclaimer: it's still a work in progress, so it's not complete/ready yet).

[0] https://github.com/ChimeraCoder/go-angular-jetpack

BarkMore• 12 years ago

The html/template package seems to address the use case, but it comes up short in several ways. One example is defining a site template with sections that are optionally specified in a page specific template.

memset• 12 years ago

Many of your comments on this thread state that writing webapps in Go (SQL, etc) is not so difficult "once you know what you're doing." Could you elaborate on this? What should one know before attempting to use Go for such things?

zemo• 12 years ago

the problem with html/template is that it's made for Go programmers. A good template language is one that's made for front end devs and designers. html/template tends to be painful in practice because it's too difficult for a non-gopher to use.

aktau• 12 years ago

I'm usually pretty anti-ORM and actually like low-level SQL interfaces (I guess because I did a short stint as an Oracle developer once, once you see what can be done with some custom SQL...). But for a recent sideproject of mine, I really didn't want to create a laborious interface to the database. I wanted something quick and dirty. I got something quick, and it doesn't feel dirty at all. It's called gorp: http://github.com/coopernurse/gorp.

By the way, my sideproject is about getting financial data from sources like Yahoo Finance, Bloomberg, et cetera. You can find it here: https://github.com/aktau/gofinance

jamra3• 12 years ago

I agree with the templating problem. Additionally, it is difficult to find proper documentation on how to do templating and not just basic examples.

lann4• 12 years ago

SQL

I'm working on an arel-esque fluent SQL generator now: https://github.com/lann/squirrel

I'll release it once I'm a little more confident in the interface. Currently you can do things like this:

  rows, err :=
        Select("col").
        From("tab").
        Where("x = ?", 1).
        RunWith(db).Query()

rckclmbr• 12 years ago

Honest question... how is this different from, say:

rows, err := db.execute("SELECT col FROM tab WHERE x = ?", 1);

wc-• 12 years ago

I'll release it once I'm a little more confident in the interface.

Release it now! I've said the same thing in the past about some code I was working on. Add a little readme, explicitly say it is a work in progress, and get it out there. At the very least the rush of publicizing something will push your drive to work on this codebase, and you might even gain some contributors. I'd rather use a repo that is obviously a work in progress but has active contributions from the creator over some more 'complete' but stagnant repo any day.

hierro3• 12 years ago

We're about to release our Go web framework which, I think, implements all the features you're mentioning and a few more. We're aiming for a release in Q1, but getting everything cleaned up and writing good documentation is taking a bit longer than expected. Right now, our internal framework (codenamed Gondola, but subject to change), includes:

* A full ORM which generates tables from structs and allows relations between models, as well as a simple querying system based on Go functions which joins the tables automatically for you e.g.

   memeId := base36.Decode(param)
   var meme *Meme
   var template *Template
   err := ctx.Orm().One(orm.Eq("Meme|Id", memeId), &meme, &template)

Meme declares a FK to Template, so the ORM automatically joins the tables and populates both pointers. Note that the ORM does not require the backends to be relational, so even when ATM we only have database/sql based backends you could write a backend for mongodb or rethinkdb.

* Inheritance based template system, with a declarative assets pipeline supporting compilation (e.g. it compiles coffescript to JS and less to CSS on demand - also pluggable). This is the base template from one of our apps:

   {{/*
      jquery|if=~ie-gte-9: 2.0.3
      jquery|if=ie: 1.10.2
      bootstrap|fontawesome=4.0.3: 3.0.0
      styles|bundle: css/style.css, lightbox/css/lightbox.css
      scripts|top,bundle: js/modernizr.js, js/detectizr.js
      scripts|bundle: js/responsiveslides.js, lightbox/js/lightbox-2.6.min.js, js/site.js
      analytics|nodebug: {{ $Config.Analytics }}
   */}}
    <!DOCTYPE html>
    ...
    {{ template "main" }}...

This server query 1.10 and 2.0 from Google's CDN, bootstrap and fontawesome from bootstrapcdn.com and bundles all our styles and scripts a CSS and a JS file. Then other templates can extend this one and override its blocks e.g.

   {{/*
     extends: base.html
   */}} 
   {{ define "main" }}...

(note that base.html could also define its own "main" block which would act as a default if you loaded base.html and it would be replaced by main.html's "main" block if you loaded "main").

* Easy to use signed and encrypted cookies.

* Regular expression based URL routing.

* Functions for easily parsing input parameters.

* A development server which automatically rebuilds the source as it changes and reports compilation errors in the browser itself. While developing, runtime errors are also reported in the browser with complete backtraces and printing the values passed to each function. While running in production, those reports are sent by email to the project administrator(s).

* App-wide and low-level caching system with pluggable backends, currently supporting Redis and Memcache as well as a dummy backend for development.

* Form generator and validator from Go structs, with renderers for both Bootstrap and Foundation (renderers are also pluggable, so you could also write your own. We also support generating forms from multiple objects so you e.g. write an object which generates a captcha and include it in any form with a single line in code (in fact, we provide just that, but a simple numeric captcha a a full-blown recaptcha object). It also does automatic CSRF protection.

* A complete i18n system, which extracts strings from Go code and templates, generates .pot files which can be translated using any editor (or a specialized .po editor if you feel like it) and them compiles .po files back to Go code again. This makes packages with their translations "go-get-able".

* Periodic and background tasks which can either be scheduled or just started from any web request.

* A configuration package which takes a struct and fills it by parsing an optional configuration file as well as command line flags.

* Support for administrative commands, which can be executed from the command line or started remotely.

* Support for pluggable apps. e.g. we do have a "users" app which provides user registration, authentication and social integration. This can be included in any other app with a couple of lines of code (one line for the import, another line to tell our main app to include it). Included apps can do a lot of stuff, like inserting code in every template rendered by the parent app, including external assets like images, styles or scripts and even rendering their own templates contained inside your parent's app base template.

* Last but not least, a minimal overhead (in the range of 100ns per request) profiling framework which lets you know how much time a request spent in the cache, the orm, the template, e.g... as well as seeing how much each operation took (e.g. you can check how much time a SELECT took and also perform an EXPLAIN on it). Users can also define their own events and profile them by instrumenting their code (it usually requires just a line of code per resource, to indicate its name). You can also profile a live server by using a command line app included in the framework.

We also have helper and utility functions for the more mundane stuff and basically everything you could expect from a web framework. We took inspiration from Django, Tornado and Rails and tried to take the best of them while building this framework (I used to contribute to Tornado back in the day). We've been developing this internally for almost 2 years and it's now around 40KLOC of Go and a couple of K of coffescript, less and html.

chrissnell• 12 years ago

That sounds incredible. Where can I follow news about your release?

sandGorgon• 12 years ago

Support for pluggable apps. e.g. we do have a "users" app which provides user registration, authentication and social integration.

Is this something like Rack (in Ruby on Rails)?

Also your templating system looks very similar to Handlebar/Mustache - is that what the general principles are going to be ?

jksmith2• 12 years ago

Interested in learning more about the plugins. Maybe use for filters to support interceptor pattern?

johne20• 12 years ago

You mentioned "Services in Go are _amazing_", but how do you avoid the first issue you mentioned? I have found so far getting data in and out to be somewhat cumbersome. Also how do you handle data validations and logging?

latch• 12 years ago

FWIW, I started to work on an erb-inspired templating package:

https://github.com/karlseguin/gerb

I didn't get to finish it before leaving on a trip, but I'll be back on it Feb 1st and can hopefully wrap it up in the first week.

I did finish a liquid-inspired template: https://github.com/karlseguin/liquid

but I wasn't particularly happy with it (neither the implementation nor liquid itself).

elithrar• 12 years ago

* SQL - The raw database/sql package in Go is very low-level. It is a terrible experience to use in a project that makes anything more than a handful of simple queries. I'm not asking for an ORM, but it would really benefit the community if there was a SQLAlchemy style (the non-ORM part of it) library for Go. It should be MUCH easier to make basic queries to SQL and get out results.

I can highly recommend https://github.com/jmoiron/sqlx which allows you to easily unpack results into structs/maps, or pass in a struct/map and insert it (using struct tags + named queries). If you're writing a CRUD application this covers a lot of the basics. There's also https://github.com/eaigner/jet which is a 'micro ORM' (and very light on the ORM part).

"html/template" doesn't support things like inheritance, didn't even support comparison operators until Go 1.2, etc.

I grew to like html/template over time, but the lack of comparison operators pre Go 1.2 was a huge sore point and I had I been working on this little project a year ago I'd have been extremely frustrated. The lack of inheritance was a bit of a pain for me too—I ended up splitting my templates into many (!) smaller pieces so I could at least re-use the portions I needed to. Hopefully something "Jinja-esque" comes in time.

There's an (unmaintained) Mustache port and a Mustache-alike in Mandira (https://github.com/jmoiron/mandira) but I've not seen anything like Jinja yet. If there's one thing html/template has going for it is that I can trust it (as much as one can) to be 'safe'.

I wouldn't say web apps in Go should be avoided at all costs, but I would say that you should consider the scope/scale and do a bit of research first. Sometimes Django/Rails or Flask/Sinatra might make it easier on you.

natural219• 12 years ago

This post really highlights what I like about Go so far vs other languages -- the evolving community. Like Rails, there is an emphasis on abstraction and productivity. Unlike Rails, the blog posts I've seen so far seem to be centered around explaining how to implement the functionality instead of just wrapping the functionality and providing an easy API. Now that I've read this post, I could download his library, implement my own, or extend the parts I want to customize very easily. I feel more powerful.

jevinskie• 12 years ago

I appreciated the development narrative rather than a post-project summary. Thanks!

polymathist• 12 years ago

I would have taken a different approach for handling unsupported resource methods. I'd love to hear some feedback on whether or not this is considered idiomatic. Instead of this:

  type HelloResource struct {
     sleepy.PostNotSupported
     sleepy.PutNotSupported
     sleepy.DeleteNotSupported
  }

Something like this:

  type HelloResource struct {
     sleepy.DefaultResponder
  }

DefaultResponder is simply a struct which has all the methods of sleepy.PostNotSupported, sleepy.PutNotSupported, etc, and is defined:

  type DefaultResponder struct{}
  func (DefaultResponder) Get(values url.Values) (int, interface{}) {
     return 405, map[string]string{"error": "Not implemented"}
  }
  func (DefaultResponder) Post(values url.Values) (int, interface{}) {
      return 405, map[string]string{"error": "Not implemented"}
  }
  func (DefaultResponder) Put(values url.Values) (int, interface{}) {
      return 405, map[string]string{"error": "Not implemented"}
  }
  func (DefaultResponder) Delete(values url.Values) (int, interface{}) {
      return 405, map[string]string{"error": "Not implemented"}
  }

Now, if you want to implement resource methods, you define them on the top-level struct, like so:

  type HelloResource struct {
     sleepy.DefaultResponder
  }
  def (HelloResource) Get(values url.Values) (int, interface{}) {
     data := map[string]string{"hello": "world"}
     return 200, data
  }

So now, if you call Get on a HelloResource type, it uses the method defined on HelloResource, not the one defined on it's embedded DefaultResponder type. But if you call HelloResource.Post, it uses the method defined on DefaultResponder, i.e., it returns a 405 error. This is effectively like "overriding" in Java or other object oriented languages and saves a few lines when people are using your library.

What do you think? Again, I'm really just looking for feedback here as I too am new to go. Is this something that would be considered idiomatic, or is the original solution better?

(Apologies in advance for any typos or syntax errors. Didn't check this with a compiler, but hopefully you get the idea.)

optymizer1• 12 years ago

You could even go further and assume all user resources are logical extensions of sleepy.Resource:

  type HelloResource struct {
    sleepy.Resource
  }

This makes it clear that you're inheriting everything from sleepy.Resource, so you can provide reasonable implementations for all methods of resources, not just HTTP methods handlers, e.g. getBodyAsJSON(), redirect(resource), etc.

Also, depending on how your mux works, you could store app-level data in sleepy.Resource, to be accessed by all resources:

  sleepy.Resource.DB
  sleepy.Resource.Session
  sleepy.Resource.Root

and other things I can't think of right now.

jrockway• 12 years ago

Drive-by code review:

I don't see any reason for the Api type to exist. Just make them normal functions?

It would also be helpful to print the portstring rather than just "Hi.".

dclara• 12 years ago

I haven't seen the full picture of Go yet, partially by some examples. I found it makes calling multi-threading functions a lot easier.

I consider Ruby, Python, Go and Julia are the new generation of programming languages aiming to be quickly picked up and made hands-on working.

Can anybody tell me if Ruby, Python, Go supports a full spectrum of functionalities for various network protocols, multi-threading, messaging (sync and async) and concurrent accessing etc. Those kind of functionalities don't only require additional libraries, but also frameworks and algorithms.

Maybe by using Jython, JRuby or something like that, they are able to achieve the complete functionalities like Java.

According to some research, Python evolved from Lisp, and Ruby is closed to Perl. So why aren't those script-based language more popular than Java/C++?

A RESTful Micro-Framework in Go

Related Stories