Show HN: A Search Engine for Developers, Marketers, Web Designers, and Hackers

https://news.ycombinator.com/item?id=6102405

by nerdydata • 13 years ago

63 38 13 years ago

news.ycombinator.com

Watup Hackernews,

We have created a new type of search engine that searches on the full source code of webpages, not just the on-page text. It's called NerdyData (nerdydata.com) and we're launching today!

Our search engine is different from search engines you've used before. Traditional search engines are geared towards providing answers, whereas our goal is to give you the best list of results for a query. Our crawler has visited over 140 million homepages and collected terabytes of HTML, Javascript, and CSS code. We've also designed several search interfaces that allow anybody to query against the source code of webpages, or download a list of sites containing a specific term.

We even offer a search interface specifically for SEO's and marketers that allow you to search within specific HTML tags like meta descriptions and meta keywords.

About us: We're two 23 year olds who recently graduated from Stony Brook University in New York, and we are ridiculously passionate about making search better for our fellow nerds out there.

If you'd like to know more, you can email me at dave@nerdydata.com

Thank you so much for you time!

nerdydata7• 13 years ago

Clickable link: http://nerdydata.com

aram• 13 years ago

Very interesting service. At first it didn't occur to me why would I want to search the source code, but your examples were good.

What is the logic behind the search criteria? I entered `<div id="main">`[1] and the first result is LinkedIn with `<div id="main-wrapper">` highlighted, which shouldn't match my query directly as I entered a closed tag. Some of the other results also didn't contain exactly what I was searching for - e.g. `<div id="main" class="main">`, `<div id="main" class="clear">` etc. A technical explanation on this would be great.

Also, one minor note - David's photo in the front page footer is broken [1]

[1] https://search.nerdydata.com/search/#!/searchTerm=<div id="main">/searchPage=1/sort=pop

[2] http://www.arc.losrios.edu/Images/Images-arc/Foreign_Lang_De...

jayzalowitz• 13 years ago

First off Stony Brook! Second, open crawl? Third, what's your stack?

nerdydata7• 13 years ago

1. Go Seawolves!

2. We built our own distributed crawler with a couple of dope servers.

3. Not ready to share our stack setup yet (work in progress) but we will have a write up shortly!

ErikAugust• 13 years ago

Do results sort based on mozRank, Alexa, PageRank? Or something proprietary?

ScottWhigham2• 13 years ago

Love it - great idea and great presentation. I only tried a few cursory searches but it was definitely bookmark-worthy. Kudos :)

I'd love to hear you guys talk about the risks of searching for "dicey"/risky characters/terms and how you protect your data/server from malicious searches.

nerdydata7• 13 years ago

When the launch craziness winds down, we plan to start a blog and nerd the fuck out. We had so much fun making this and want to help others get past the insane obstacles we faced while building a search engine.

013• 13 years ago

Is there any content filtering? When I went on to Image Locator, then "See an example" it will search for imgur.com. The first page of results shows 'thebiglistofporn.com', 'entensity.com'(Not sure if it's NSFW, but it looks it from the thumbnail.)

I know it's not too much of a big deal, but porn thumbnails wasn't something I was expecting to see on a websites example page.

nerdydata7• 13 years ago

We do not filter content, but thanks for letting us know about that thumbnail on the example page.

michaelmior• 13 years ago

Looks pretty cool. I really dig the design. I was a little annoyed that I seem to have 10 credits without noticing and used them up just messing around. (I can't really tell because clicking on the credits doesn't do anything). It would be nice to make this more obvious up front. Anyway, I can see this being a pretty useful tool in some scenarios.

Also note that "Locater" is spelled "Locator" (sidebar on the left)

ilikehunny• 13 years ago

your site has a really bad user rating? http://www.mywot.com/en/scorecard/nerdydata.com?utm_source=a...

suhastech• 13 years ago

This might give a hint

http://web.archive.org/web/*/nerdydata.com

sadkingbilly• 13 years ago

I've never trusted wot. They are abused by groups of "wot cops" that organize in the forums and target web sites to bash (deserved or not).

As far as the site goes, it looks like they used to be a more generic seo search engine, as recently as a few months ago.

skinnymuch• 13 years ago

That site is awful. Only thing worse is using BBB for credibility.

kevrone• 13 years ago

This is awesome. Especially for finding use cases and implementation examples of front-end libraries.

nirvanatikku• 13 years ago

That was my first thought, too. I checked out where my various projects are being used on the web, something I've been wanting to see for some time, and was pleased to learn about various properties using it.

tehaaron• 13 years ago

Exactly what I would use it for but then I realized it wasn't free and to me its not worth the money.

snake_plissken• 13 years ago

Good god, my attention span just reached terminal velocity. I like. Very cool and original.

jwarkentin3• 13 years ago

I thought it would be really cool until I realized the whole credit thing. It often takes many permutations of a search to find the results you're looking for. If each attempt costs money and the whole thing might be a waste in the end then it's hardly worthwhile.

I've always been annoyed that Google won't search for symbols. As a programmer that makes searching for certain things anywhere from difficult to impossible. I was hopeful that this would be a search engine I could use to search for code snippets or queries that involve symbols.

Let's say, for example, that I was browsing through some PHP code and saw a double dollar sign ($$) and wanted to know what it was used for (yes, I already know, it's just an example). I might try searching for something like 'PHP $$'. This search engine is useless for that. That's the real problem I need a search engine to solve.

brodney• 13 years ago

Check out http://www.symbolhound.com/ for a programmer friendly search engine.

jwarkentin3• 13 years ago

You just made my day :)

gtani• 13 years ago

You can search stackoverflow, but not github for "symbolic operators"

http://stackoverflow.com/search?q=[php]+%22%24%24%22

also, languages that need it have their own type signature based search (haskell, scala)

http://scalex.org/

http://www.haskell.org/hoogle/

ultimatedelman• 13 years ago

agree about the credits. i know you're trying to monetize, but forcing me to pay for searches (and view more results) is a bit on the ridiculous side. there are better ways to monetize than that.

from a UX perspective, any time you _prevent_ a user from doing something, ie, you present them with the option to view more things and when they click on it, it says you need to pay to view the things, that is BAD UX. better UX would be NOT presenting the user with the option to view more, but where they would look for this feature, have a link that says something like, "want to view more results? upgrade to our premium plan!" Much better UX because as a user I'm not tricked into thinking I'm getting something and not having it be delivered.

but yeah, unless you're more explicit about the credits thing, it's going to be your downfall. i like the design and the results that i got from my one search, though :)

jwarkentin3• 13 years ago

I think charging for search in general is going to be their downfall. There are so many search engines these days that without bringing something truly revolutionary to the table, you'll be hard pressed to get much notice even offering it for free. Charging for it? Not a chance. There are other ways to monetize search.

ollerac• 13 years ago

I don't get how to use the seo search. How do I search for all websites that have a certain name in their title for example?

    <title>name</title>

Just returns the exact match.

And how do I search meta descriptions and meta keywords?

nerdydata7• 13 years ago

SEO Search is a premium feature which lets you search inside of pre-parsed html tags. You can learn more about it here https://search.nerdydata.com/documentation/#seoSearch

kyle_martin1• 13 years ago

This would be a very good tool to find XSS vulnerable websites.

cwings• 13 years ago

Or find all sites with a vulnerable installation of wordpress.

brbcoding• 13 years ago

Whoa... Searches must be based on IP. Shared with a co-worker and they can see all of my searches, and I can see theirs.

ScottWhigham2• 13 years ago

Anyone tried comparing to http://searchcode.com/ ? I haven't - just ran across it right now doing an unrelated search.

jwcrux• 13 years ago

Just a small request - it would be way cool to have a "site:" filter like in Google searches. A lot of the results found are irrelevant to me as it is.

Cool site - great work!

nerdydata7• 13 years ago

We have a Refine Tool that allows for AND,OR,NOT filters. It's only available for paid users, but really allows you to push the limits and get the exact query you want. Thanks for the feedback!

bapi2• 13 years ago

Search terms like consumer_secret ou api_secret and you'll find lot of things... :( https://search.nerdydata.com/search/#!/searchTerm=consumer_s...

With great power, comes great responsibility.

nerdydata7• 13 years ago

Did we mention our "Basic" account is free and gives you 200 credits to use? https://search.nerdydata.com/pricing?hackerNews

bapi2• 13 years ago

No , because yesterday it was 49$ or 99$ Idon't remember exactly :). Good move I think to let people like me use it and find value in it as the promise is great.

adamnemecek• 13 years ago

Honestly, I'm not a fan of the name.

sachin0235• 13 years ago

while it is good, i am wondering what it can do which google can not? For example i searched for android accountmanager and results i got were far less productive than what google returns me back. If you want to build it, build it the way it perceives user perspective well

dgunn• 13 years ago

This could be great. You need to monetize in a better way. Make searching free for everyone (cause no one is going to pay for it). Then, package up some pre-built searches (backlink search, competitor stuff, etc), build a special interface, and sell that to SEO's as a tool. This is just my first thought. You have data from over 140 MM pages. You can almost certainly use that data to make a product worth money to someone.

I just can't imagine selling searches will work. The big players in the space have made search into a human right. I know you have to make money somehow but charging for searches almost has to be the wrong answer.

Show HN: A Search Engine for Developers, Marketers, Web Designers, and Hackers

Related Stories