Super Meat Boy leaves database wide open

http://forums.somethingawful.com/showthread.php?noseen=0&threadid=2803713&pagenumber=258#post398884189

by llambda • 15 years ago

204 96 15 years ago

forums.somethingawful.com

nicksergeant3• 15 years ago

From what I read, the database isn't "wide open". It's read-only and there's nothing juicy in it.

Also, a bit of NSFW stuff going on here if you dig too far.

Edit: It's not read-only, INSERT/UPDATE works.

jnorthrop• 15 years ago

It's not read-only though. He mentions that INSERT and UPDATE work.

http://forums.somethingawful.com/showthread.php?noseen=0&...

This is a really embarrassing mistake on their part.

nicksergeant3• 15 years ago

Ah, missed that.

citricsquid2• 15 years ago

It's not read only, it's read, insert and update.

TheEskimo• 15 years ago

Sorry, if you read the page linked you'd see that that is not the case. It is open to SELECT, UPDATE, and INSERT. if it were SELECT only that would be read-only.

As it is, that's just stupid as any user can wantonly edit anything. I could trivially edit every level's author to be myself or do intensive operations which result in a DOS.

The only smart way to give clients access to a database is through some sort of frontend entirely under your control which prevents them from having the user/pass and sanitizes the queries.

Edit: Whoops, while I typed this multiple other people did the same. Sorry for the redundancy.

nicksergeant3• 15 years ago

Yes, I replied to my comment and even updated the original comment indicating that I was incorrect.

soult2• 15 years ago

> The only smart way to give clients access to a database is through some sort of frontend entirely under your control which prevents them from having the user/pass and sanitizes the queries.

MySQL maybe, but enterprise DBs (think Oracle, DB2, Postgres) support a very fine-grained access model.

dustingetz• 15 years ago

via smosher above: http://img820.imageshack.us/img820/1641/itsfinetrustme.png

kevingadd2• 15 years ago

The idea of doing an UPDATE on the custom levels table every time someone dies in order to increment the deaths counter terrifies the database administrator in me. That'll scale REAL well. Yikes.

citricsquid2• 15 years ago

Asking the database administrator in you, what's the most efficient way to do it? Queue and push updates every 10 minutes?

benologist3• 15 years ago

We do millions of operations like that a minute by queuing, aggregating and then committing. SQL Server's MERGE is particularly useful for it, although on our MongoDB stuff (and ironically, for player created levels) we do $incs.

kevingadd2• 15 years ago

If I had to use the database, I'd probably aggressively shard the data so that it's spread across multiple databases/tables. That way the load of counter updates could be spread across machines. I'd probably also store the counters in a separate table from the rest of the levels, for the same reason, but also to ensure that the rest of the information about a level isn't locked during every increment.

Of course, this is given that IIRC, MySQL uses row locking and used to even use table locks at times during writes. The last thing you want is for your spew of counter updates to lock out other, more important operations (like deleting a level). MySQL locking behavior might be better now.

The real 'right' solution to a high-traffic counter like a death count is to store the realtime counter(s) in something like memcached: High throughput, low overhead, and built in support for operations like increment.

As counter traffic increases, you could partition the counter itself (and approximate the actual value by taking partition_value * num_partitions) or only record a percentage of the time (say, +4 the counter on 25% of deaths). Both solutions reduce load and could produce 'good enough' numbers as long as you don't have wildly different behavior on each partition.

biot• 15 years ago

Off the top of my head, one way to do this without rearchitecting everything would be to insert a new record into a very simple in-memory temp table. Every minute, update the appropriate record in the real table by count(*) of the temp table and clear the temp table. As it's being used for a non-critical counter, data loss in the event of a MySQL restart isn't a concern.

That should be sufficient for something of this scale and only requires a minor update to the client SQL, ensuring the temp table is created when MySQL starts, and executing one stored procedure on a schedule.

Retric• 15 years ago

The easiest way to have a counter that is both correct and fast is to split things up into multiple separate counters and then count them. So for a simple example: User ID mod 50 = 7 so lock row, 7 increment row, 7 unlock row 7. Want the total? sum (all rows).

You can also get fancy with lock row 1, lock total, update total = total + value of row 1, unlock total, set row 1 to 0, unlock row 1. Repeat row 2...50. The advantage being even lower latency access to a number that is withing X seconds of being correct.

PS: Both of the above can be split across multiple cores / machines. But, for a less accurate but still reasonable have the client a = (rand (0,100)), b = rand (0, 100 + a), if b == 7 update death count by (a + 100). This only increments the counter 1 time in ~150 times while avoiding counters that are always multiples of some number. Just be careful of off by one errors.

jrockway• 15 years ago

One is: don't use the database for this. Lock/read/write/unlock is very slow when you need a round-trip over the Internet for each phase. (Remember, with optimistic locking, many transactions are in the Lock/read state as the same time. When someone writes the new death count, all transactions in the lock/read state are rolled back and started from the beginning, as the data read is now invalid. And that's assuming MySQL is not in read uncommitted mode; then you just throw away data randomly.)

The correct way to count something in a relational database is to add a relation (row) for each event, and then COUNT them all. How do you count log message in log files; by having syslogd update a count file? Nope. wc -l.

(And no, I'm not a fan of auto-incrementing integer primary keys, because read/calculate/write is slow. Databases have ACID so that read/calculate/rewrite is safe. But that doesn't mean that type of workload is going to run fast. Of course, autoincrementing primary keys is specially optimized by most databases, so you never notice in this particular case.)

patrickgzill• 15 years ago

How many deaths per second does it take (INSERTs or UPDATEs per second) to bring your database to its knees? Maybe you are using the wrong database?

smosher• 15 years ago

I found this image succinct and telling: http://img820.imageshack.us/img820/1641/itsfinetrustme.png

ojbyrne2• 15 years ago

So it could be that they create a mysql "database" for each user, and give them all the privileges needed there, and no privileges anywhere else.

And conceivably you could have some kind of proxy that looked like mysql but actually sanitized/logged/whatever any queries, before passing them on to the real server.

pferde• 15 years ago

Since that would take more effort and time than simply doing it right, I call Occam's razor.

ojbyrne2• 15 years ago

I wasn't trying to argue that it would be the correct approach, just a possible explanation.

z926• 15 years ago

So what would be the solution? He used a password protected database connection and put the password compiled in binary. If I was doing it I would have probably done the same. How else can it be done? Use web service? That would still look "open" to someone digging inside the compiled binary and getting the keys.

JoshTriplett• 15 years ago

Use a web service, and make sure that service only allows precisely the requests you want to allow, rather than arbitrary SQL.

jarrett2• 15 years ago

Fair question. A web service would indeed be a better solution. With a web service, you have a server-side application layer, and all database reading and writing is done by that layer. Sure, you might be able to authenticate and send bogus info to the web service. Even that can be made very difficult, e.g. by cryptographically signing requests or encrypting the data on the wire.

So if you do that, worst case scenario is someone reverse-engineers the protocol, including whatever cryptography you're using. And that person then sends phony data of the sort the game would really send, e.g. messages saying "I completed this level" or "I scored X points." Which, in the big scheme of things, isn't a huge security breach.

The difference between that and having an open MySQL server is that the open MySQL server allows you to read and write anything stored in the database. That's much worse than being able to spoof normal data the game would be sending anyway. In this situation, the worst case scenario would be total corruption of the database, as opposed to the player falsely accruing points or something like that.

z926• 15 years ago

There is a misunderstanding. Someone connecting to a "open" MySQL server will only be able to run those type of queries (select,update or delete) that he is explicitly permitted to run and only on those database and tables where the admin has granted him access.

Not too different from a web service.

Also no amount of encryption will secure the system because it's not about man in the middle attack. It's about decompiling the binary. The attacker has full access to all your encryption functions, routines and everything you put there. See it as like he has all your client side source code.

The only benefit I see in using a web service is restricting one user from accessing other users data in the same table. It's about only 20% improvement in security as I see it. And that's only because MySQL doesn't support row level authorization.

Other issues that are not any more secure through a web service:

1. Crash the MySQL server by running rouge queries. On the other side an attacker can DOS a web service too.

2. One can run Insert/Update/Delete queries on MySQL. Well web services has to run insert/update/delete queries too based on user input --- where the attacker modifies the source code that generates the input.

3. One can validate input using web service. One can validate input using MySQL trigger/SP too.

Just trying to show that I don't see much improvement in security using a web service when the attacker is controlling your client side source code. Basically MySQL is also a service that you can control, but it doesn't run on port 80.

dangero2• 15 years ago

I think your question is very valid and I think the answer is that the "solution" depends on what your goals are. The author may even consider it to be perfectly fine if he has to reset these statistics regularly, or even move to a more protected system in a later version if it gets hacked.

His main goal was probably something like "put a system in to send me some analytic information, but spend as little time as possible on it and make no concern for security or performance."

Especially considering the developer may have had decent knowledge in some things like mysql c connectors, but not in php or something else he could use for the web service, his approach may have been the best approach to take. There's no way we can judge. I'm pretty sure that was your point. It seems like the mysql approach could have even come from an Agile methodology where the "story" makes no concern for security or performance.

brown9-2• 15 years ago

There's no real reason to have the end users connecting to the database directly. Any advantages of this approach are far outweighed by the disadvantages. Having a server process that handles input from the game and then write the results to the DB is far simpler and more secure in the end.

toyg5• 15 years ago

a "story" that doesn't consider security over the internet is doomed to get your rooted sooner or later.

funkah2• 15 years ago

I can't believe how many people here think this is okay. YES, a web service! Please never apply to a company that I am working for!

Natsu• 15 years ago

Maybe the new FizzBuzz should be to show prospective devs some DailyWTF articles and see whether they laugh, cry, or think it's a great idea....

__david__• 15 years ago

Well they say when all you have is a hammer... Actually, a web service doesn't seem like a bad idea to me--it's easy to implement and would allow finer grained permissions than direct database access. It's also easy to move to SSL. What would you do instead?

z926• 15 years ago

The less one knows about a system the more layers of firewall he will put up to protect it.

People get astonished when I insert a virus infected USB disk into my Windows machine and use Explorer to safely copy files from it. And when they ask what anti-virus I use, I say "None, never used any anti-virus in my life. I reversed engineered a lot of viruses and I know how they work."

> Please never apply to a company that I am working for!

Sure. Thanks.

badclient• 15 years ago

With a web service, majority of the potential abuses would be related to being able to manipulate your own scores and records in the db. That isn't that big of a deal in this use case since we are dealing with games.

Most simply the web service can require a 20 char unique key every time it is taking a request from a client. You can find yours using a sniffer. But it'll be hard to guess others'. This is the most basic implementation.

jiggy20112• 15 years ago

Perhaps generate individual keys using some secure random process. Then you can check your logs now and again and if a few particular keys are submitting obviously bogus data you can just remove them from your records and block them in future.

As has been mentioned elsewhere there are other risks by the fact that your allowing people to enter data into your system using an actual language (SQL) as opposed to a few POST vars (assuming your webservice sanitizes input properly).

There are just many more possible attack vectors with mysql , one of which would be sending massive crossjoins or similar to the DB to crash it.

RKearney• 15 years ago

    SELECT * FROM smb_editor_author CROSS JOIN smb_editor_leveldata

This query is taking forever to finish...

samlev• 15 years ago

You should try locking the tables first. Maybe other people connecting is slowing it down, so a full table lock would be a good idea.

Remember to keep the lock alive after the query finishes, just in case you need to query it again...

haileys• 15 years ago

I should probably note that at no point did I disclose the database credentials - other users discovered those and publicly posted them. I contacted Super Meat Boy several times before going public with this but they totally disregarded my warnings.

dangero2• 15 years ago

I'm going to dangle myself on a limb here.

If you consider the constraint that the indie programmer may have had incomplete skills to set up a proper service, saw this method as faster for any number of other reasons, or didn't want to maintain a proper web service, I think this could very well have been the right approach at the time. Judging from how much money this game has made, I think that's hard to argue with since time to market should have been the number 1 concern at the time and early on he needed a way to make sure people were able to complete all the levels and measure the difficulty curve.

When the developer responded that it didn't matter, that may have been because they don't need that data any more and don't even look at it.

barrybe• 15 years ago

Maybe you could argue that it was the right approach at the time. But deciding that the data doesn't matter anymore is not cool, you have to take care of your paying customers or else you lose credibility.

zyb09• 15 years ago

In good tradition the most critical comment is the top voted on HN. Yada Yada indie life is hard, but no one with a right mind would think of connecting the client directly to the remote master database. This is just so WTF on so many levels, you can't help but wonder how they even managed to produce such a good game after all.

socksy2• 15 years ago

Yes, it seems obvious to web developers and developers that are used to working with databases and CRUD apps, but that's one set of skills over a particular domain. Game development tends to emphasise a whole different set of skills — those of efficient graphic rendering, for example.

A points table is no doubt a last minute add on in a field that certainly wouldn't be the expertise of a small indie developer (no-one goes into game development for their love of CRUD apps, after all). They needed a database so they used the most popular, in a way that probably seemed the most appropriate for their application.

funkah2• 15 years ago

Exactly. There's no need to equivocate, just call this what it is -- lazy and incompetent.

mey• 15 years ago

If you read Edmund McMillen's formspring page, you'll see that there are several disconnects from a development perspective about super meat boy. Such as them not being able to reproduce builds from source.

http://www.formspring.me/EdmundM

Conversely, I love me some Binding of Issac

rmc3• 15 years ago

This is just so WTF on so many levels, you can't help but wonder how they even managed to produce such a good game after all.

When I started working in Industry™/The Real World™ I learned a lot of important lessons in How Things Are Actually Done™. One of them was how much software was just held together with duct-tape and string.

So yes, you can produce a good game and think that this sort of DB access is a good idea.

drewblaisdell2• 15 years ago

The thing is, we would never have heard about this if the developer wasn't a pompous ass and tried to claim that the way he is doing it is "fine".

username4• 15 years ago

Maybe he built a way to detect mischievous updates or he's baiting to sue.

sohamsankaran• 15 years ago

supermeatboy.com seems to be down

richardeid2• 15 years ago

Hi, I registered because I had read this story earlier today and decided to contact Team Meat's developer and ask him a question. His response prompted me to post this here. My question is of a political nature, but you can ignore that part. The part that struck me as odd, and the part that matters for this conversation, was where he claimed nothing happened.

https://lh5.googleusercontent.com/-kc0f6ZQZebY/TvWDhFSHYEI/A...

Note that I sent him the tweet at exactly midnight my time. A short time later, a thread on the Steam forums was updated with:

https://lh5.googleusercontent.com/-itY306KgvLk/TvWGR_cQkiI/A...

To give that a little context, once the details of this vulnerability were in the wild the shenanigans everyone ran on their database had affected everybody's ability to play the game. It would crash upon launch.

Anyway, he responded ~two hours later, just after that post on the Steam forums noted that the game was playable again.

Does "nothing happened" seem accurate here? And for anyone that has the skills to access that database, is it still accessible and...modifyable?

dandelany• 15 years ago

"My question is of a political nature, but you can ignore that part."

No, your question was of a rude nature, which is pretty hard to ignore. It's not surprising that he responded curtly.

richardeid2• 15 years ago

I was trying not to focus on that aspect of the conversation, but this is what my comment was based on:

https://twitter.com/#!/SuperMeatBoy/status/14761253025520435...

In my opinion, this isn't the sort of behavior that should be advocated. If people pirate the software produced by these companies that support SOPA, then that just validates SOPA. Additionally, he doesn't exactly give his games away for free so it's a strange stance to take in my eyes.

Anyway, my thoughts and feelings on copyright infringement weren't relevant discussion for this thread. I instead chose to focus on him acting as if nothing had happened, when the rest of the Internet proved otherwise.

the_mitsuhiko3• 15 years ago

The game ships a libmysql library. Unless they got a commercial license for it, that's a GPL violation in the wild.

drewblaisdell2• 15 years ago

It isn't a GPL violation because the game is just interfacing with a master MySQL database, not using one to store local content.

the_mitsuhiko3• 15 years ago

libmysql is GPL licensed as well.

shearn89• 15 years ago

Well, this seems to be fixed - the user posted in the image now only has SELECT privileges to the database. Unless my mysql is reeeally rusty and I misunderstood the output...

seagaia• 15 years ago

http://www.formspring.me/EdmundM/q/274876035885957785

I find it disturbing that people would go through and even attempt deleting levels and altering things en masse. It's okay to point out the bug, and maybe change one thing to display this fact...but the people changing everything? It's a selfish thing to do this to thousands of creative works (the user-created levels), in the motivation of just showing one developer something he did wrong. Thankfully, there were backups.

Parts of this thread, the somewhat arrogant reaction of the Team Meat programmer, and the motives and subsequent actions of some of the people modifying the database, made me a little sick. If anything it's a showing (on both sides) of the lack of maturity of some programmers.

Look, I know that this could be justified in saying "well, someone would have done it sooner or later." In that case, I suppose whether or not this was okay is left up to whether Tommy (the programmer - Edmund did design) would have patched it. And with his defensive response, I'm sure he knew and was going to do so, but probably didn't want to admit it (from interviews he seems to have bit of a bitter ego)

MBCook2• 15 years ago

Unfortunately this doesn't surprise me too much. I'm a Mac user and eagerly awaited the Mac release, only to find it had terrible performance and game breakage bugs. I emailed the developers and received two replies.

Tommy, the developer quoted here, started his email implying my complaint was faked, then saying they couldn't reproduce the issue of the game crashing every time you entered a warp zone, although they said they were working on it. His response made me quite a bit madder.

Edmund, the other developer, replied separately. He was apologetic and mentioned that the person they had originally paid to port the game fell through so it had to be done very fast. This was the kind of email I was expecting. It wasn't confrontational at all.

They later released a patch that fixed the crashing issues, but performance is still a major problem. The Windows system requirements are listed as a 1.4 GHz P4 with 768 MB of RAM. On the official blog, they list a dual core 2.6 GHz machine with 2 GB of RAM as the minimum requirements for the Mac. The game is officially supposed to support the 360 gamepad, but a note was left on the Steam forum that despite the promise it's not possible to support it correctly on the Mac, so the problem won't be fixed.

I thought the binding of Isaac looked interesting, and it has been getting good reviews. But after my experience I'm not going to play it. Super Meat Boy was a lot of fun (even though it didn't perform well on my Mac), but after my experience seeing the response to this issue doesn't surprise me.

starwed• 15 years ago

Hmm, Super Meat Boy seems to be totally fine on my mac. (2010 Air 11").

However, Binding of Isaac is pretty lagtastic. (But, quite possibly the most fun I've had with a game this year.)

MBCook2• 15 years ago

I considered buying it on XBLA a few times, but I wanted to support a developer for supporting the Mac, and I didn't want to have to boot my XBox to play it.

The individual levels are generally quite fluid, unless you die an awful lot of times. But the map screens and some of the bosses are slower (ESPECIALLY the one in World 4/Hell). I wonder if it's the extra visual effects (like the fire in Hell).

I wish I had bought it on XBLA. Microsoft's quality control meant it would have been a much better experience. I was actually disappointed with Steam, which is the first time that happened. I guess I assumed they ran strong QC, which they must not.

PS: MacBook Pro, early 2010, 2.53 GHz, 8 GB. Graphics card (integrated vs nVidia) never seemed to make a difference.

ido• 15 years ago

BoI is Edmund's game, Tommy wasn't on it.

jcapote• 15 years ago

They don't really care, could've even been intentional: http://img820.imageshack.us/img820/1641/itsfinetrustme.png

Super Meat Boy leaves database wide open

Related Stories