Ask HN: How do you monitor API fraffic/performance?

Hey HN,

At our startup we've just released a new version of our API (REST, web-based), and folks are finally signing up to use it. Unfortunately, we've neglected to set up any kind of traffic or performance monitoring of our API specifically. Here are some of the problems I'd like to solve:

* An new API client emails me, saying that a particular API call isn't working. I'd like to be able to go into this magical traffic logger and see precisely what calls this client has made, to what endpoints, at what times, and what we responded with. * We release a new endpoint. We've tested the endpoint and feel confident in it, but still, we'd like to ensure that it's working the way we thought it was going to work in production. We should be able to go into this traffic logger, filter by the endpoint that we want, and look through the logs manually, and ensure that everything looks reasonable. * I'd like to be able to track the performance of endpoints over time. I'd like to see statistics about error rates and response times, broken down by endpoint.

Here are the services that I've considered:

- 3scale/Mashery: Though it seems to satisfy my requirements, these services offer a whole suite of things that we don't need and wouldn't want to pay for, like monetization and authentication. Also, I'm slightly sketched out by the API proxy part of Mashery -- I don't want our API to be dependent on Mashery being online. - Loggly/Splunk: Maybe this is the time to jump to a big boy logging solution? Right now all our logs just live on our app servers. Perhaps I could just log some information about requests, and set up fancy filters and such inside the app itself. - New Relic: I believe that New Relic can satisfy these requirements if configured properly. One downside is it seems to be crazy expensive.

Anyways, I'd love some feedback. Have any of you tackled this particular solution, and what have you done to solve it?