Ask HN: What do you use to test Frontend code?

https://news.ycombinator.com/item?id=2957310

by outside1234 • 15 years ago

56 46 15 years ago

news.ycombinator.com

At our startup, we are trying to sort out the best way to handle testing against our supported Browser matrix. We want something that is repeatable and automated. We've looked into QUnit and Selenium but have this nagging feeling that we are doing it wrong.

Our application is primarily one page and uses a lot of jQuery to both animate things on screen and to work with the backend via AJAX.

Any hard won tips out there?

simplegeek• 15 years ago

I guess Selenium is fine, out of curiosity, why do you think using Selenium isn't the right choice for you?

outside1234• 15 years ago

I'm not sure that it isn't. I'm worried about the maintenance costs of keeping it up to date - but maybe that is the price you pay.

otaku_coder• 15 years ago

If you're worried about keeping it up to date, why not outsource the test runner to something like saucelabs?

hiwaylon• 15 years ago

Take a look at JsTestDriver, for hooking in to your CI system.

mfolnovic• 15 years ago

You could try with Jasmine, although it's not for this specific purpose...

And I agree with simplegeek, selenium should be fine in your case. Although can't imagine how you could test animations... :)

rchiniquy• 15 years ago

Jasmine isn't what you're asking for, but you should use it in addition to what you're asking for.

dbalatero2• 15 years ago

If you're in the Ruby world, Capybara is a great addon to Cucumber and RSpec. It solves the animation problem by adding a timeout to wait for elements in the DOM. A step like this:

When I click the link that does a 4000ms animation I should see "#mydiv"

Capybara will wait for #mydiv to appear for some timeout period (30 seconds). If it appears before then, it keeps going.

This way you can have complex animation and still assert final state of the page. Capybara allows Selenium, headless WebKit, and other JS drivers. It's one line to swap.

If you aren't in the Ruby world, you might at least steal some ideas from that lib to make your Selenium tests better.

Also, you might drop the folks at Pivotal Tracker an email. They have a fat one-page JS-heavy product, and I bet you anything they automatically test it. They might have cool stuff to share.

schrieaj• 15 years ago

Capybara works pretty well for any web based front end. I'm using it to do some basic automation of websites (scheduling tasks and the like). You write the tests in ruby but honestly it is really simple: http://pastie.org/2476405 is my script for logging in to google voice and sending a text message. If I recall it was simply gem install capybara, a gem install selenium-webdriver and that was it.

dbalatero2• 15 years ago

Good point, I forgot that you could just use it standalone!

pacmon• 15 years ago

Could try watir (http://watir.com/)

tombot• 15 years ago

Watir is what Facebook use I think, it has plugin support for most of the major browsers and tests are fairly easy to write.

timruffles• 15 years ago

We use jsTestDriver at Picklive to test our game, alongside Sinon.js for a mock implementation of our game server. It's very easy to test simultaneously on multiple browsers, run tests on the command line, and we run it headless for continuous integration.

In comparison with Selenium, I've found unit and integration tests using this setup fast and resilient.

juanbyrge• 15 years ago

zombiejs?

mpd• 15 years ago

Having dealt with the Selenium and its frustrations in the past, I tend to outsource this sort of testing if at all possible. Some things can only be truly tested with eyeballs.

One tip I do have is to disable jQ animations during your test runs so you aren't having to jump through hoops in your test suite just to wait through animations -

jQuery.fx.off = true will do that.

arbrandes• 15 years ago

This! Also, try to mock any setTimeouts and setIntervals so that they execute callbacks synchronously (i.e, timeout = 0).

riffraff2• 15 years ago

on a side note, do you people test graphics? I mean, it's kind of easy to say "I get a div #foo". But how do you check if it has the right shade of $color and a rounded corner and the text content is not overflowing the containing box?

I live in constant worry of changing css and totally screwing up a completely unrelated bit of the web app, and the usual thing I get told is "well, it's very fast to check if it still works", which reminds me of me not understanding why unit testing is useful.

socratic• 15 years ago

Why is unit testing useful, to you?

I have often heard two justifications for testing: (a) correctness and (b) developer speed.

Phrasing (a) differently, is X functioning as intended? Phrasing (b) differently, how quickly can I make a change to Y, and if I change Y, how long will it take to manually check Y (with a browser, etc.), how likely am I to introduce a bug (leading to extra development time) and/or will I have to rewrite a test?

If your primary justification for testing is developer speed rather than correctness, does that impact which features (models, controllers, views with div ids, actual view graphics) you should write tests for?

jtheory4• 15 years ago

I'm putting extra work into unit and UI testing to build a safety net so I can convince my CEO & other developers to let me carry out serious refactorings on our code, like deleting large chunks that aren't used anymore, and changing objects that are used pervasively.

We also have an increasingly broad application; as we add new features, I (as a developer) find that I don't visit older parts of the app, though of course the users still need them to work. Particularly when refactoring basic back end stuff that's called from lots of places, it's too tedious to manually track down every part of the UI that would need to be retested due to a given change (and then test it manually... ugh).

MartinCron4• 15 years ago

I live in constant worry of changing css and totally screwing up a completely unrelated bit of the web app

I understand this worry, but it makes me think that it may be a design problem with the CSS. Unrelated bits shouldn't be using the same CSS, related bits should probably change together.

In a more practical note, I have created selenium tests which capture screenshots and then saved those screenshots as artifacts in the CI (TeamCity) system. Having a human look at a bunch of static images makes for a faster and less error-prone sanity test to see if things are totally broken.

riffraff2• 15 years ago

I agree it may be a design problem with the CSS, I agree.

But my point is exactly that: I can't just go in and _improve the code_ (extracting classes, rewriting it in less/sass or whatever) because to verify the changes do not mess up the UI requires clicking around many pages in half a dozen browsers in three OSes.

And thank you very much for the practical note, it's an interesting idea!

hopeless2• 15 years ago

Asking a computer to check aesthetics is start of your descent into madness. Use Selenium to check that links do what you expect, elements are loaded etc. Don't use it to check if a button graphic has the right icon loaded.

Computers are great at functional testing but you still a Mk1 eyeball to check aesthetics. Luckily, functionality is what your customers will really care about. No one will shout at you if an element is misaligned for a while before you catch it with manual testing

jtheory4• 15 years ago

Side note -- I definitely agree about not using Selenium to check for alignment, images loading, etc..

But... it's pretty cool to have a single Selenium test that does nothing but zip through your site and show all of the unique views at rapid fire speed, so as soon as the page is loaded it has already filled in any relevant form and is loading the next page.

Make sure someone with good concentration sits and watches that test run once in a while; anything seriously wrong will pop out, and they can watch it zip along while taking a coffee break.

gudmundur• 15 years ago

Check out Ghostbuster (https://github.com/joshbuddy/ghostbuster).

qixxiq• 15 years ago

I use a custom selenium test runner that runs them in parallel (php). I plan on neatening it up and releasing it at some point if theres interest.

It also has a 'live mode' which is useful for writing up new tests. It basically follows a file and whenever there is a change it begins running the commands from that line onwards again. If it hits an exception it'll pause there and wait for you to fix the broken test/site before resuming.

troels• 15 years ago

I'd be interested in seeing that. Oh, and I'd rather see a mess soon, than wait for the neated-up version.

jtheory4• 15 years ago

I'd be interested as well.

I wrote a custom test runner as well, though not addressing the same issues; mine is specifically for non automated testing. It's a Java GUI that lets you choose from all our existing tests, tinker with parameters, and run the test live in your own browser against our test server with a few extras (like stepping through the test, go/pause, run all tests in sequence, etc.).

It's partly to let our CEO really interact with our testing -- i.e, see what's being tested, and rapidly explore the site as a whole -- but also is pretty useful as a robot; i.e., when I want to register three new users to try something out, I don't have to do it manually.

ByteMuse• 15 years ago

Some things require manual testing. Animations and a lot of visual stuff needs to be analyzed holistically; there are sometimes visual quirks that are near impossible to write tests for but can easily be detected from someone quickly viewing the site.

Have different environments set up. After every big push, run through a simple user flow to make sure what the user will actually see is what you intended.

MartinCron4• 15 years ago

After every big push...

One way that I minimize my front-end testing liability is to do as few "big" pushes as I can. The safest change to make is the smallest change possible.

barnaby• 15 years ago

I use qUnit all the time because I wrote lots of jQuery plugins for everything that we do at my small startup. Then we have a QA team in India that are always clicking through the site and finding new bugs (they also have extensive selenium scripts that go through the whole site every night, but oddly enough none that run the unit tests... grr).

ltamake• 15 years ago

Selenium works fine for me.

sgrove• 15 years ago

I always feel frustration at this question, I can't believe that sauce labs hasn't taken off. Use capybara as the driver layer, and a proper test framework that separates the different layers, and sauce labs to hit any browser matrix you want. Tie it into a CI system and never worry about this again - it's a solved problem, and it was solved by Sauce Labs.

disclaimer: I used to work there, and put my blood, sweat, and tears into building it out just for situations like this.

tdavis• 15 years ago

For Javascript testing, I really like JSTestDriver for automation and Sinon for mocking/stubbing.

I found bulky solutions like Selenium to be somewhat hit-or-miss and overkill for most tasks. It's very rare that with proper unit testing I really need a whole additional layer just to click around the site and make sure stuff is working as expected. Maybe some basic sanity checking, but it's really not my preferred way to think about testing--which of course is just a personal preference.

pmorel2• 15 years ago

Same, I feel that if there is a good coverage from unit testing, then you should only need sanity checking. Furthermore, if you are using jquery (which itself is well testing on all the browsers), then you should have more confidence that your javascript will work (providing you're not using non-cross-browser javascript on top of jquery)

pmorel2• 15 years ago

For reference, the list of tools and framework mentioned on this page can be found at http://www.romku.com/list/343858/web-testing-framework. Feel free to rate them and add any missing information.

BerislavLopac• 15 years ago

Admittedly, Selenium's standard interface leaves a lot to be desired; however, I've had great time using it in combination with a Python wrapper, where you write Selenium test cases as Python unit tests.

zzleeper• 15 years ago

This. Just write the Selenium base code in the Firefox addon, and then click, export as python :D

long• 15 years ago

you might want to check out SIKULI (http://sikuli.org/) - it's like python on top of a computer vision stack, so you can write code like click([part of a screenshot]).

jph• 15 years ago

We use SauceLabs with Selenium, Jellyfish (very fast), Capybara (very flexible), and Jasmine. Yes, Selenium may feel hard to work with at times yet the payoff is enormous when you aim to support many browser for many years.

Edit: I have a great outsourcing team that builds Selenium for me, and runs in SauceLabs. PM me if you want details.

JesseAldridge• 15 years ago

I'm not using full on unit tests for my current project. I just break the code into a few independently functioning layers, and make a single html file to test the code in each layer. I run the corresponding html file manually as I develop code in a given layer. It may be heresy to say this, but I've found that automated unit tests are more trouble than they're worth in a lot of cases.

The second thing I do is use my tool every day. I use the unstable branch on the unstable server whenever possible. Problems usually become obvious very quickly. I don't push to production until I've used the unstable server for 24 hours without incident. I find this combined with the testing practices described above to be the best compromise between reliability and getting-stuff-done-ability.

wlievens• 15 years ago

For algorithms or application logic, even rather simple cases, unit tests are an absolute must.

For user interfaces, it's a lot more effort for a lot less return. YMMV

douglasmiranda• 15 years ago

Hmmm, I'm using Splinter (http://splinter.cobrateam.info/) to do tests. It lets you automate browser actions and much more, like execute javascript, find elements returned via AJAX, interact with forms, links and more... it has drivers for chrome, firefox... You shold to test it :)

Ask HN: What do you use to test Frontend code?

Related Stories