Show HN: Percy.io, a continuous visual integration tool

by fotinakis • 11 years ago

249 52 11 years ago

I've used gatling on a few redesigns with success

https://github.com/gabrielrotbart/gatling

I have some tweaks i was going to merge in that gave the diff's as %'s (so you could ignore small differences like fonts being different) as well as animate between the two versions. On my TODO was grabbing the actual element style attributes and then actually just diffing that, so the differences would be more or less spelled out.

Still, cool stuff!

planetjones• 11 years ago

This looks promising. Our project has a plethora of selenium tests, but they're focused on functionality only, not appearance. We can tell there is a table, button, link, form, etc. and click it. On that side we're reliable.

But we really let ourselves down sometimes with regressions to the look and feel of the application.

Are there any competitor tools for this visual testing ?

mholmes680• 11 years ago

check eggplant / testplant (not free though)

werkshy• 11 years ago

We (Harry's) have released a tool called Kontrast: https://github.com/harrystech/kontrast, http://engineering.harrys.com/2014/09/24/kontrast.html

It's written in Ruby but is not really dependent on Rails or anything to run. We use it to compare a list of URLs between the version we are currently testing in CircleCI and our production site. Any differences are highlighted in a nice gallery and we get a message in our Hipchat room.

lindig• 11 years ago

You can try https://app.webmate.io. It's a hosted service for layout, regression, and cross-browser testing. Basic features are free. The diff-ing works on the DOM, is visualised in screenshots and identifies the XPath of elements that are found to be changed, removed, or added. Tests are run on browsers/vm that are part of the service, giving you access to 200+ os/browser combinations.

johns• 11 years ago

ghostinspector.com (one of the products from my company)

jbnicolai• 11 years ago

There's VisualReview[0], but only self-hosted.

[0] https://github.com/xebia/VisualReview

siegecraft• 11 years ago

So.. is this just a lot of wrappers around wraith? Or something else?

fotinakis7• 11 years ago

Actually it's all home grown, right now it renders the page and screenshots in Firefox 38 ESR and then the actual diff image is produced by imagemagick's compare function, with some special things around it to handle edge cases with width/height differences and some others.

arenaninja• 11 years ago

This is neat. Recently started doing Selenium tests with Scrutinizer CI, but we've had issues where the UI is broken and we find out a day later... I'll definitely keep an eye out for you guys

fotinakis7• 11 years ago

Woohoo! Glad to hear it. :) We have an annoucement-only mailing list if anyone doesn't want to sign up with GitHub: https://percy.io/#updates

ahmetmsft• 11 years ago

Are you an one man army? Congrats! :) This is great. How do you plan to react to potential changes in rendering engines? Would those be shown as test failures as well?

todd3834• 11 years ago

If you are interested in how something like this was made, there was a good talk about building something like this at RailsConf this year: http://youtu.be/Q4ttqkIEM7g

fotinakis7• 11 years ago

It's an awesome talk! Jessica Dillion has actually helped me with Percy a bit. :) She's great, you can see her slides here: http://www.slideshare.net/jessicardillon/implementing-a-visu...

balls1873• 11 years ago

I wasn't a fan of giving you github permissions just to sign up for an invite.

Sir_Cmpwn4• 11 years ago

I would have agreed with this, but the permissions they ask for are pretty benign and clearly relevant to their mission.

andybak• 11 years ago

I'm a member of organisations that may not be comfortable with me giving permission by proxy.

I couldn't find a way to sign up and exclude private repos or organizations.

nerdy• 11 years ago

This seems like a really convenient way to watch for front-end changes, but how do things like margins work? Will my whole page be red because everything has been shifted around or squished?

Responsive capabilities will really improve usefulness, and the absence is somewhat of a deal-breaker. On responsive sites developers will have to eyeball each change even when using Percy, so checking the desktop resolution wouldn't be much additional effort relative to checking all other resolutions for each page.

On the biz-dev side, if it's going to be a paid product it might be a good idea to start charging ASAP!

dubcanada2• 11 years ago

Based on the screenshots, yes if you changed margin on say body, or something high up it would cause everything to shift and be "red".

rich90usa• 11 years ago

A coworker has has a small tool for accomplishing this: https://github.com/LesterTheTester/dfrnt

It has been integrated with Travis CI and GitHub pull requests so that non-passing diffs are appended as a comment to the pull request by a bot account.

If there's an intended diff, a pull request can also automatically be submitted against the original pull request which, on merge, updates the "gold master" images and allows the original pull request to pass.

anarchitect• 11 years ago

This looks really great. The integration with Github and CircleCI makes it a very attractive option for us as it would slot right in to our existing workflows.

sangnoir• 11 years ago

Percy.io seems somewhat similar to BBC's wraith[1], albeit easier to integrate.

1.http://bbc-news.github.io/wraith/index.html

cloudforcer2• 11 years ago

Has anyone used visualdiff.com?

cloudforcer2• 11 years ago

Has anyone used visualdiff.com? The services seem pretty similar.

fotinakis7• 11 years ago

I think they might only do live sites, the thing about Percy is that it can be integrated earlier into your feature/browser tests for all pages, even those behind logins.

Kmandr• 11 years ago

I've been working on VisualReview (https://github.com/xebia/VisualReview), an open source project for visually reviewing a (web-)application as part of a pull-request, very similar to percy.io.

We're releasing our first version somewhere next week which includes a basic comparison GUI and testing api's. PR workflow planned in the release after that. Interesting to see we're not the only ones with this idea.

mark2hayes• 11 years ago

There's another tool out there that does something similar called Screenbeacon (https://www.screenbeacon.com/).

idibidiart• 11 years ago

I wonder, does it handle responsive UIs?

kordless2• 11 years ago

I signed up. Clicked on the survey, clicked back, got a blank page. Did it again, got a live page. Did it one final time, got a blank page. Tried it again just now, blank page (but there is HTML in the page source).

joshdance• 11 years ago

Really interested how this will work with responsive. Does it test a set size each time? Multiple sizes?

pimlottc• 11 years ago

Devil's advocate - won't this tend to bolster the outdated attitude that the front end must be pixel-perfect?

olefoo• 11 years ago

It depends on the project. But when you're dealing with mobile devices you don't get to test even a fraction of the models available unless you've got enough scale to have a dedicated testing lab.

proexploit• 11 years ago

I'm not sure I consider that to be true. Why is pixel perfection out-dated?

vladharbuz• 11 years ago

Perhaps because rigid constraints are useless with the number of different devices out today, and layouts should instead be highly adaptive?

ecesena2• 11 years ago

There's no mention about pricing... any rough idea/estimation?

fotinakis7• 11 years ago

Hey! Good question, we're still working out the details on pricing but I want it to be very reasonable, simple and transparent for people. Just need some time to work through with a few initial beta testers. Stay tuned!

ecesena2• 11 years ago

Great, thanks!

amitaibu• 11 years ago

Congrats @fotinakis

Check out also http://shoov.io/ - there are a few blog posts, that explain the concepts and point to examples. Apart of the visual regression testing it also allows monitoring the live sites with your preferred functional testing frameworks (e.g. casperJS, Behat, etc').

Open source of course :)

jmprobert• 11 years ago

Interesting read! Bummer to hear the news though.

leoi11• 11 years ago

would pixel by pixel diffs be an overkill?

alkonaut• 11 years ago

Doesn't say what it does. I can deduce from the title and landing page that it has something to do with UIs and CI. From the FAQ it seems it is web only, as it mentions "the page" (assume that is an html page).

Is this an appropriate summary?

"It can help you visually review changes made to a web front end as part of CI/QA"

Would be neat if you could plug any kind of front end screenshot rendering and not just web.

On• 11 years ago

My favorite in this space is Applitools eyes (https://applitools.com/).

Eyes has really advances image processing capabilities, which meant two huge advantages over everything else I looked at:

1. It's not prone to rendering differences. I can set a baseline in one browser/OS combo, run it on other combinations and set it to only consider real regressions as a failure.

2. It can validate the layout of the page even with dynamic content.

Show HN: Percy.io, a continuous visual integration tool

Related Stories