Show HN: SEO for JavaScript, HTML5 and Single Page Applications

https://snapsearch.io/

by CMCDragonkai • 12 years ago

50 56 12 years ago

snapsearch.io

CMCDragonkai17• 12 years ago

Hey HN, Roger here, happy to answer any questions.

ilaksh2• 12 years ago

This seems exactly like prerender.io but with more code to insert (harder to use) and more expensive..

CMCDragonkai17• 12 years ago

I calculated the average price per usage versus Prerender.io for average sites. And it comes out to be cheaper or equivalent to the standard price of Prerender.io.

Basically take the amount of usages you use of Prerender.io per month, and compare it yourself.

What do you mean by more code to insert? I do know of Prerender.io but more code depends on which framework/language you're using. If it's node. It's pretty much one line.

The extra code is just options that allow you to customise the way the interceptor works, these options include things like blacklist, whitelist, regex, extensions ignoring, client side caching.. etc. But the basic usage is the just key + email, and you're good to go.

Also note that unlike Prerender, our cache storage is free, so we don't track how many pages you have.

andrenotgiant• 12 years ago

When you scrape the HTML from a single page app (let's say the default app homepage) do you re-write the javascript links to be HTML links with unique URLs?

If not, how does Google find anything more than the app homepage?

albemuth• 12 years ago

Sitemap works

CMCDragonkai17• 12 years ago

Yep a sitemap can also work. But it's not needed, since Google will follow your anchored links anyway. SnapSearch in the future can also create sitemaps dynamically.

CMCDragonkai17• 12 years ago

Your links on your homepage, should already be "<a></a>" anchor links. When google hits your page, and it's served up by SnapSearch, then it will automatically find those anchor tags and navigate to the subpages. If you're talking about buttons, perhaps Google will click on those buttons. But any links should be represented by anchor links.

This works for SnapSeach.io. Check on Google, search https://snapsearch.io and all the subpages of SnapSearch are indexed. Also works for http://dreamitapp.com

So to answer your question, there's no need to rewrite your links.

programminggeek• 12 years ago

Am I crazy for thinking that SEO is a huge reason to NOT do a Single Page App for a public facing website that you want Google to index?

It seems like the wrong business choice from a marketing perspective that we are now looking for a tech solution to solve a problem we created ourselves. Single Page Apps feel like building Flash web apps 5 years ago. It's a valid choice, but with clear drawbacks.

CMCDragonkai17• 12 years ago

Well SnapSearch allows you to make dynamic web applications without worrying about SEO problems. It's not so much a technical solution to a technical problem that we created ourselves, it's the fact that search engines and social network robots have not caught up to next generation websites (which I believe will be a fully dynamic user experience).

We're just pushing the web ahead.

integraton• 12 years ago

You are not crazy. The single page app trend is solidly in the "peak of inflated expectations" phase of the hype cycle ( http://en.wikipedia.org/wiki/Hype_cycle ), which is why there exists a subculture of developers who are trying to use the approach for everything, even where it isn't appropriate.

CMCDragonkai17• 12 years ago

I think the single page app trend is coinciding with the growth of javascript, HTML5. It shows that the websites of the future will be more and more like web applications, and they need dynamic UI. The thing that's stopping us the SEO problem, which I hope is no longer problem anymore with snapsearch.

jbeja• 12 years ago

Agree, to build website maybe not the best solution but i think it has it place for internal and no exposed apps like (CMS, andministrators, dashboards, etc) that normaly the public viewer wouldn't ever see.

mercer3• 12 years ago

I suppose React.js and similar approaches would make it easier, but I also don't immediately see the use case for building single-page apps that need indexation.

fidlefodl• 12 years ago

I feel like applications and tooling to pre-render your HTML (eg: Reactjs) are more interesting. It's definitely more work currently (when done right), but it seems like the most natural flow. Always render html, let clientside link it up, everybody wins.

timdorr• 12 years ago

I think the logo could use a little work. The lightning bolt doesn't come off as an S, so it looks more like "napSearch".

CMCDragonkai17• 12 years ago

Thanks for the feedback :). I'll look into it. Did you know the bolt is also a tiger stripe? That's why I got the tiger theme.

mattlenz• 12 years ago

Will Google eventually begin javascript-enabled scraping themselves?

CMCDragonkai17• 12 years ago

They may one day. But it's a big computational overhead from straight curling to running a JS VM for everything. Right now SPAs are still a minority on the web. But I guess the question is will you wait for Google to support your site and lose out on the awesomeness of single page applications?

Encosia2• 12 years ago

I don't know the specifics, but they already have been for years (at least to some extent). Several of my clients have found content deep inside public-facing client-side apps indexed in situations where we didn't care if it was indexed, but assumed it wouldn't/couldn't be.

CMCDragonkai17• 12 years ago

Do you have any examples? Because I've been building single page apps for 2 years, and none of it is indexed. Also indexing for search is not the only thing, sometimes clients need google adverts, which require the google media bot check your site's content. One of my clients could not get any data out, and the site could not get approved by Google for the adverts. But with SnapSearch, the google media bot was able to extract the necessary information.

Check your google webmaster tools, and you can force a scrape from google bots, and you can see that SPA sites are not discoverable.

puppetmaster33• 12 years ago

Dear N00bs, SPA works great w/ SEO.

Hashtags! (ex: https://github.com/puppetMaster3/ModuleBU/blob/master/latest... in AppBu section, github has examples).

Just because twitter or Facebook programers don't know how to do it, but then they do hire 0 experience out of school.

Cheers.

puppetmaster33• 12 years ago

Down votes?

It upsets you that I show you how to SEO w/ SPA?

Says a bit about you I think.

puppetmaster33• 12 years ago

Dear downvoter: http://theunboundedspirit.com/wp-content/uploads/2014/05/tru...

ant_sz• 12 years ago

I think this should be more a problem for google than for web developer. web developer have the right to choose new technique they like, it's google's responsibility to scratch your content well.

But now things are reversed, It's kind of monopoly, isn't it?

CMCDragonkai17• 12 years ago

But it's not just google now. It's anybody who does any kind of crawling on the web, which includes social media robots (that acquire images from the link), personal web crawlers, private internal search engine, data mining, even advertisement bots that check content. Would they all have to embed and maintain/update a JS VM?

erichurkman• 12 years ago

Developers have always been free to choose techniques. Search engines owe you, the developer, nothing. If Google et al can't scrape your content, they'll scrape a competitor who better serves parse-able content. If you want to play ball with a particular search engine, you have to serve content that they can parse at the time.

nnq• 12 years ago

Am I crazy thinking that we as web developers should NOT HAVE TO have to care about this? It's Google's and Bing's and Yahoo's problem to go build decent spiders that run multiple versions of the most recent web browsers inside them, and they do have the computing power that's needed.

All spiders/crawlers should see webpages just as human users see them that's the point! They should do whatever it takes to get this working, regardless the effort, because this is why they get paid (indirectly, though adds, but still...). And this is still not enough: it's any respectable search engine's duty to try and develop near-human-level AIs to be able to also extract meaningful information from visual structure, from pictures, and from hard to understand prose. And longer term, it will be their duty to develop above-human-level AIs to be even better at this than humans so they can provide even better search results.

Since when are we, web developers, expected to do the search engines' work for them?!

I expect this is just a transition phase and things will get on the right track soon.

(Note: and no, I'm not against helping search engines. Providing semantic markup, microformats and all this is great. But there is a difference between helping them and doing the work for them, on our dime!)

EDIT+: @OP: I'm not critical, what you and prerender.io are doing is awesome and immensely helpful for web developers! But you do realize that you're doing what search engines should be doing themselves, right? And when they will wake up and realize they should get into the game, they will have basically put you out of business... or buy your business, which I think is what you expect :) Anyway, good luck with it!

CMCDragonkai17• 12 years ago

Hey I understand where you're coming from. I had the same impression before I began on SnapSearch. But right now SPAs are still a minority of the websites in the world, but eventually everybody will start making web applications.

But do note, as I said in the other comments, it's not just search engines. There'll be a day when curlling sites will require a JS VM. There's a big difference between a standard curl vs a full blown chrome/firefox level VM. Considering the amount of sites google crawls per second, their computing costs would explode compared to what they're doing now.

And this doesn't even address all the social network bots, and pretty much all the other bots doing stuff other than search indexing.

So I think we're still got quite some time before we're redundant.

At any case, the question for you is, are you going to wait for Google to implement this before you can start using single page application technologies which can give you an edge when it comes to customer user experience?

tracker13• 12 years ago

Why would curl need a full VM? Just curl the API/Service endpoints for the data you want.. if you need more that's what PhantomJS is for.

SilkRoadie• 12 years ago

We as web developers need to create websites that make money. Part of that is to create websites which are visible and indexable by search engines..

In the UK people have been naming their businesses AAA whatever to appear first in the yellow pages for like 20 years. I don't see how this is much different from what we are doing now.

tracker13• 12 years ago

The search engines (google and bing) both run js content. Hell bing's IE browsers show up in google analytics. I worked on a site where we cleaned up our url structure with proper redirects etc... and bing went crazy with it, totally threw off our analytics (tons of unique clients, no retention, high drop off).

I also know that google loads content in a headless browser as well. If you use push-state or shebang urls that load properly, and have proper output, google and bing get it fine.

over 35k results for photogallery.classiccars.com in google. (I didn't design it, just implemented it a couple years ago)

If you want an SPA interface, be prepared to do the work for SEO. If you can't handle that, then you should deal with it... I don't expect actual applications to be SEO friendly.

Isofarro6• 12 years ago

Since when are we, web developers, expected to do the search engines' work for them?!

The Web is a trifecta of HTTP, URLs and hyperlinks. That is the fundamental baseline of what it is to be Web. That's what makes it accessible to anyone with a basic internet connection. With that universality comes an overall improvement in education, which improves the quality of life of people on this planet. Not just your little home-town, but the planet.

The Web isn't an artificial intelligence. Not even in the dark days of RDF, respite with logical reasoning engines and ontologies. That isn't going to be your saviour.

So far, there's nothing in the HTTP spec document that stipulates that it requires a JavaScript capable execution. The Web is built to be extensible, which is why techniques like progressive enhancement are just as fundamental. Assume the trifecta of HTTP, URLs and hyperlinks are present and build with that, then enhance with the litany of technologies and shiny toys built on top of the Web stack. Don't just assume those technical building blocks are always going to be there.

So what you are doing, by requiring JavaScript at the HTTP level, isn't Web Development. It's probably more akin to the death-throes of the last Flash developers struggling to justify their existence.

If you need your site to be indexable and searchable, universal, a JavaScript-dependent solution is sub-standard.

These "post-implementation" Javascript static site scrapers, they emulate progressive enhancement badly, because they are backwards. They are about as qualitatively as useful as building a separate accessible website to your main website, you know for them blind people with their talking browsers. It's a band-aid at the point you realised you did not spend the requisite amount of time understanding the environment in which your "website" runs.

And, please, read the full specification of the Google Crawlable Ajax spec before you throw it at me. I have. Including the part that says you really should have built it properly in the first place.

philbo2• 12 years ago

If you're building JavaScript-heavy/single-page applications, you should be applying the principle of progressive enhancement anyway, there is no reason not to do so.

Progressive enhancement brings a number of benefits, of which friendliness to search engines is just one. User has JS turned off? Page still works. Request to CDN fails? Page still works. Silly lint error slips through to production, breaking all your JS? Page still works. Weird cross-browser issue breaks your JS in some clients? Page still works.

The most frequent arguments against progressive enhancement, which has been prevalent amongst a number of back-end and product people I've worked with, is that it is a waste of time and that it adds too much extra effort. I disagree with both. It is not always necessary to reach full feature parity between your JS and non-JS implementations, for instance an auto-suggest input in your JS code can happily be a vanilla input in the non-JS. But if you do just enough to ensure that your application is functional without JavaScript enabled, my (subjective, biased) experience has been that it doesn't take too long.

Kiro• 12 years ago

Easier said than done if you're using something like Angular or Ember.

philbo2• 12 years ago

If angular or ember make it difficult to follow best practices, are they the best tool for the job?

Less trollishly, although I have no experience with angular or ember, I have built a progressively enhanced application with knockout and I thought it went quite well. As with any progressively enhanced application, it requires that your back-end is set up to respond with both markup and JSON responses, but the data- attributes responsible for binding behaviour are just ignored by the browser when script is not running.

Perhaps my experience is too simplistic, though. Can you give me an example of the kind of things that angular and ember make difficult to progressively enhance?

Isofarro6• 12 years ago

Yehuda Katz, the lead on Ember.js was categorical about the use of ember.js when you require progressive enhancement: Don't use ember.js.

So this is just a case of using the wrong tools for the job, buying the hype without validating the requirements.

mercer3• 12 years ago

Don't you generally use those for web apps where SEO and search engine indexation is less important? I'm not very experienced in this area, so I find it hard to think of an example where you need indexation/SEO-optimization for the kinds of things you do with Angular or Ember.

I'm very curious to hear of some examples.

schmrz• 12 years ago

A web site can be Javascript-heavy[0] without being a web application[1]. There's a big difference.

If you just need auto-completion, modal dialogs and similar features then you shouldn't build a web application, and you definitely should make your site work without Javascript.

On the other hand, if you are building Google Maps, then you should build a web application. Google Maps doesn't work when you disable Javascript.

Supporting clients without Javascript in web applications often doesn't make sense. If it does, you should consider building a standard web site. Just imagine building Google Maps without Javascript. Or better yet, imagine using a version of Google Maps without Javascript. I'm sure that there are people that could pull that off and still end up with something good but it's not the case for most teams (not enough time, resources or skill) building web applications.

[0]: What's considered Javascript-heavy anyways? I think that including support for standard stuff like auto-completion, modal dialogs or even just including bootstrap's javascript these days makes your site Javascript-heavy.

[1]: By this I mean a single page web application that uses Javascript for state management/routing, rendering, communication with the backend, etc. You should be able to deploy an application like this on S3 and it should work. That's not a requirement but I'm just trying to point out that it's an application built completely with Javascript, with no backend support (except for the api that it communicates with).

Isofarro6• 12 years ago

On the other hand, if you are building Google Maps, then you should build a web application. Google Maps doesn't work when you disable Javascript.

The "web application" label you use here means what? That it's okay to ignore web development best practices like progressive enhancement, because "web application"?

That Google Maps ignored progressive enhancement is not a good reason for labelling it as a web application.

Is Fix My Street (http://fixmystreet.com/) a web application? It uses a Google Maps type interface so you can report broken street lights, fly tipping:

https://www.mysociety.org/2011/07/08/technical-fixmystreet-m...

it's an application built completely with Javascript, with no backend support (except for the api that it communicates with).

It's a circuitous definition. Surely an API requires a backend? And surely that API predominantly uses HTTP?

There isn't much difference between returning structured data, and returning an HTML view of structured data. That's just a twiddling of views from a controller (assuming a well-structured codebase). Both deal with an HTTP request and emit an HTTP response, one just does an extra step of rendering HTML.

kilemensi• 12 years ago

My problem with this approach is most people don't/can't pull it off properly with the end result being mediocre/sub-standard in either case. It's like when iPhone competitors tried to compete by producing duo keyboard/touch screen phones. They just ended up loosing both ways.

Pick one and do the best design/implementation you can pull off. One design/implementation is hard enough and unless you don't plan to do any real hustling (like getting out of the office and do some face-to-face marketing and lead generation), SEO shouldn't be that high on your list anyway. Good if SEO works for you but it's not the end of the world if it doesn't.

CMCDragonkai17• 12 years ago

I agree, the goal of Snapsearch is to allow the possibility of going full steam ahead with heavy javascript, and not having to worry about trying to make it all compatible with non-js clients like search engines. It's all about reducing the workload in making single page apps.

Isofarro6• 12 years ago

I'm finding a handy benefit of progressive enhancement is that it forces two distinct server side web-controllers for each piece of functionality - a server-side representation, and an Ajax representation.

That's useful in separating the the HTML generation from the business logic in a way that's obviously evident. So it helps me structure the code in a good clean way.

Show HN: SEO for JavaScript, HTML5 and Single Page Applications

Related Stories