zeratul

Ask HN: Could you suggest a fast library for converting documents into a sparse matrix representation (e.g., COO or CSR) in any programming language? I'm guessing C beats most of the implementation? But there is also the issue of efficient n-gram hashing/indexing.

99.3% accuracy is not that great. Currently the best performer on this data set has 99.77% accuracy:

http://yann.lecun.com/exdb/mnist/

Many points raised here are valid concerns and should be addressed by the data scientists when designing the system.

Let's just not forget about the positives here: there is a company that openly uses term "A.I." and raised a lot of money. Am I the only one here waiting for the AI Winter to be over [1]? Maybe it's not the best A.I., yet, but they got the funding. Personally, I think that's great and I hope for others to come out of the closet.

[1] http://en.wikipedia.org/wiki/AI_winter

Brendangregg, do you know if anyone tried to use this platform to predict failures or DDOS attacks ahead of time? Or it is not feasible with this API? Thanks.

To Ryan Kirkman and Thomas Davis: could you make some of the data freely available to the Natural Language Processing community?

If this is something that interests people, then it will be worth the time to automate this process. I suspect that there are blog entries, twitts, facebook updates that already answer these questions. Now we have to mine them. "Gold standard" set of Q/A would make it easier.

EDIT: But I hope there will be a way of getting Q/A in a bulk, right? Thanks

NOTE TO SELF: This could be a data mining or text mining web app. Check if the interest grows in time.

I think no one is like Ed. There maybe be 25% of programmers that have a long standing experience, even with the new tools, but they don't share wisdom like Ed does.

Ed why can't you have your own company? Why can't you be the one who is hiring or at least teach others how to hire. I bet you can smell a talent from 1000 miles away - even if someone did not finish a good school; the world of software engineering would make so much more sense.

[EDIT:] I imagine a job interview; an interviewee says: "I didn't program this but I did program something more difficult"; interviewer makes a frowny face; Ed comes round the corner, puts his hand on the interviewer shoulder and says: "He did not call you stupid; he just said he is ready for the challenge - he has the experience".

S.A. is talking about general-purpose AI (position 2 in the RFS). This means processing natural language. There is a lot of progress but it's just slow so it's almost invisible.

Also it's a very difficult field of science. Now you need to be proficient in AI, machine learning, computational linguistics, linguistic corpora research, cognitive sciences, statistics, and sometimes physics if the text changes over time. Of course, you also need to be a good programmer. This combination of skills is very rare. Thus, very slow progress.

I suggest to start with well defined practical problems. For example, no one seems to do much with user generated reviews. There is some sentiment analysis but that is just a binary text categorization problem - not even close to general purpose AI.

It would be much more interesting to show a seller a time ordered stream of clustered reviews that depict only the most representative review for each cluster. This way a seller can see how his/her fixes/changes impact user reviews. Also it would be a great source for features and bug fixes requests. This is an ideal testing bed for clustering, novelty detection, categorization and mild inference. The inference is required because of sparseness of data.

This would create a good data set for a more general purpose AI. We would have reviews and text documenting changes and improvements of a new version of a product. Now the computer could start learning the dialog between users and product developers. Then, we are just one more step from statistical inference based question-answering system. Not a brute force system like "Watson" or a hand crafted rule base system like "Siri".

[EDIT:] I was thinking more about a decision support system that can recommend product changes. But in a way that maximizes customer satisfaction and minimizes the cost of implementation. The dialogue between past changes and customer reaction would give us the surface that needs to be optimized. This would generalize well to other domains where there is a text for request and a text for response - just to name one: clinical text in healthcare (position 5 in the RFS).

Maybe it's also good idea to revisit the reason why these systems are not widely in use. They all operate under so called "closed-world" assumptions. It means that the knowledge about the world is very limited.

Once the limitation is lifted the system has to deal with non-monotonic reasoning (http://en.wikipedia.org/wiki/Non-monotonic_logic) and that leads to multiple inheritance problem (http://en.wikipedia.org/wiki/Multiple_inheritance). Unfortunately, the multiple inheritance problem has, so far, only NP-hard solutions (http://ijcai.org/Past%20Proceedings/IJCAI-89-VOL-2/PDF/047.p...). [EDIT: think of a command "eggs are in a fridge"; to find eggs in a fridge you need to know that they are inside a container; you need to know its shape and how to open it; if your software knows that and inherits this knowledge each time when asked for eggs, then it will brake when the eggs are not in the box.]

We are in need of software that deals with large networks of human knowledge. Then we can take SHRDLU to the next level. Otherwise we are stuck with Will Smith movie (http://www.imdb.com/title/tt0343818/).

[EDIT2: there are many theoretical "suggestions" but very little software or practical internet applications]

Entangld, this is a great app. I hope you will find some time for healing.

Stress burns the mind in the same way as physical activity burns the muscle. At some point you need to give your mind a rest. I suggest this book:

http://www.amazon.com/Full-Catastrophe-Living-Wisdom-Illness...

or this one:

http://www.amazon.com/Wherever-You-Go-There-Are/dp/140130778...

This should help you keep your strengths intact regardless of your current situations ... who is to say that next endeavor will be better than the previous one.

I think posts like that should be accompanied with some tools that help to deal with grief or suicidal thinking.

Here is an online grief support group:

http://www.onlinegriefsupport.com/

Here is a phone number to a grief counseling group for those in US:

1-800-260-0094

Here is online suicide support group:

http://www.suicidepreventionlifeline.org/GetHelp/LifelineCha...

Here is a suicide prevention lifeline for those in US:

1-800-273-8255

[EDIT: suicide is a very complex disease that is far from being understood. I'm sure that prof. Yoshiki Sasai had to deal with much, much more than just two retracted articles.]

I also saw ~7000 visits first day but that was in 2012. Here are some conclusions of my web traffic analysis:

https://github.com/entaroadun/hnpickup/wiki/Hacker-News-Pick...

Here are multiple screen shots of the google web traffic analytics interface:

http://hnpickup.appspot.com/hnpickup_web_app_statistics_snap...

I would say that this is true for unstructured data that does not have availability of experts who participated in the creation of that data.

I noticed that by talking with the domain experts, for example radiologists that read head MRIs, you can greatly simplify the problem. It usually comes at the cost of experts annotating the data, which might be too expensive for some businesses. But then, I think, the data is analyzed just by scholars until they built-in the expert knowledge into their algorithms.

That's at least the pattern in healthcare data. The first one to notice a paper or source code like that dominates the market - this is just my conjecture.

I agree. What the internet needs is semantics not regular expressions. If there is a need for regular expression, it's only because semantics is not working, for example, the search engine does not have all the synonyms so I might write "do|does|done". We should be focusing more on textual query expansion (if the query hides some background knowledge), auto-tagging of web pages (if the web pages assume some background knowledge) and clever disambiguation of queries (if we don't know which background knowledge tap into).

You might be onto something. Could this be easily converted into an annotation tool for machine learning purposes? It looks like you could use it to annotate images and text. You need an interface to create annotation schema. Then, you need to have two or more annotators per text/image. And finally, you calculate inter-annotator agreement and export the data.

I know that there are couple startups that got seed money for "annotating the web". For example "Rap Genius".

Please let us know. I think they are using their own internal lat/lon mapping to encode images. I was able to get some images using their API:

https://ddnhehdam2vn0.cloudfront.net/malaysiaair2014_02/18/2...

https://ddnhehdam2vn0.cloudfront.net/malaysiaair2014_02/12/3...

but most of the numbers return 403. Above links roughly match upper left and lower right corner.

Justin, I like your idea behind "circl.es": people already are generating a lot of data, there is no need to ask for more.

If you need help with your data just email me.

Note to self: data mining web app, it seems to use simple a similarity measure, same idea can be used for recommendation engine [Twitter + Facebook + Pinterest, etc ... to get good vector representation for each person, then clustering (probably non-convex variable density or not), and assigning recommended items from a cluster to all people in that cluster]) - product recommendation is not much different than dating

It means that our ranking algorithms have good recall but very poor precision. We value web page connectivity more than its content. We don't know how to teach machines to evaluate web page for its merit so we hope that a large number of Twitting, Liking and Plusing non-experts will approximate single expert. Millionshort shows that this model isn't good enough.

The course teacher's book costs $828 + $4 shipping:

http://www.amazon.com/gp/offer-listing/1600490069/

Is this an error?

You say in your blog that you sat on the "Heritage Health Prize" advisory board. Have you looked at these data sets?

https://www.i2b2.org/NLP/DataSets/Main.php

De-identification of medical charts is a bottleneck in clinical research. It's impractical to ask for thousands consent forms, however, smaller sample sizes are inconclusive, so much, that most of medicine is driven by inconclusive research findings. Moreover, full anonymization does not allow to follow patient records over time. This will kill any big patient outcome study, at least financially. What are your thoughts?

You are welcome. I've studied what makes a story be picked up. It's mostly time of the day and the name of the submitter. Here are my partial findings:

http://hnpickup.appspot.com/hnpickup_ratio_visual_analysis.p...

More to come. Probably, this week I will get around publishing something on the "karma growth effect".

BTW, I've sent an email to "help@leanpub.com" about latex math support. So far I haven't heard back.

Note to self: Leanpub.com disrupts book publishing industry, it took them just ~ 11 months (EDIT: 2 yrs - look below) to get traction. Surprisingly, their original post got just TWO karma points:

http://news.ycombinator.com/item?id=2329397

Getting Mr Braithwaite to publish at Leanpub.com is also a sign of success.

Congrats to Andrei! He has one more, similar, product:

http://www.yasiv.com/amazon#/Search?q=data%20mining&cate...

Both youtube and amazon visualization show very nice clustering. In case of technical books they cluster into, e.g., beginners, intermediate, advanced book groupings. It's impossible to notice that without the graph visualization.

The original post is here: http://www.linkedin.com/groups/R-web-apps-77616.S.91223876

Maybe instead of "and I currently make ___" you should have "and I have ___ years experience in ____". Or at least have both.

We are getting the news too late. Did you know that there are 7 bills that might change how startups operate?

    H.R.3427 - Startup Technical Assistance for Reemployment 
              Training and Unemployment Prevention Act
    H.R.3571 - Entrepreneur Startup Growth Act of 2011
    S.1965 - Startup Act of 2011
    S.1826 - Startup Technical Assistance for Reemployment 
             Training and Unemployment Prevention Act
    H.R.2941 - Startup Expansion and Investment Act
    H.R.1114 - StartUp Visa Act of 2011
    S.565 - StartUp Visa Act of 2011

http://ycombinator.com/people.html needs updating. There are at least 3 people missing.

I would like to know if YC has data mining/analytics specialist that reviews YC applications and helps people.

> We are going to change healthcare from the ground-up

This is so impossible!

You have a great product for 1-10 general doctor practices. You should focus on that and just that. If you feel like you have a lot of free time on your hands try knocking on 1-5 pain management doctors clinic. See how you do there. If you manage to customize their workflows and menus for pain management, move to 1-5 MDs obesity clinics, and so on. I suspect that after single pain management clinic you get burned out.

Have you ever looked at UMLS http://www.nlm.nih.gov/research/umls/ ? It has 2,612,024 mln medical concepts and counting. Many diseases are still poorly covered. Any time you talk with a specialist in a hospital he/she will need those terms in their documentation. Every disease has different workflow and different vocabulary. If a hospital thinks about adopting a new EMR, they will send ~100 their own IT people to EPIC/GE HQ to have them trained how to configure their system. Then, those ~100 IT people will work with ~1000 MD to have progress notes, flowsheets, and what have you configured to everyones liking. This is SO MUCH work and huge investment that has to have 0.0001% risk for the hospital and for the patients. Keep in mind that if patient dies and there are lacks in documentations someone will have to pay ... a lot.

I'm sure that some day you will have ~1k engineers and ~2k support team, each encumbered with basic medical knowledge and patient workflows. But then you will notice that healthcare can be changed only top-down not ground-up.

What about rules for an educational material? Let's say I create an open source project for others to learn from. It cannot be too complicated but it should do something useful. Here is what I think:

    Is the project actively developed? 

    Are there more people developing?

    Are there releases?

    Are there docs?

    Methods

PG or Aaron: Could you de-identify YC applications data and make it publicly available for others to analyze?

[EDIT:] Or maybe you could make a prediction challenge using the YC application data? You could host the challenge on http://www.kaggle.com/ or http://tunedit.org

Note to self: There is actually a very sparse body of literature that talks about data driven approach to VC, e.g., http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=4621..., this one seems interesting (predicting exits): http://www.springerlink.com/content/w024ku34221u3258/

Yes, I like it and I use it. Thank you for the app.

I wanted to ask, though, did you know that "upvote" and "save a story for later" means same thing on HN? What was the reason for adding "star" button in your app? Couldn't you use this instead http://news.ycombinator.com/saved?id=marketer and have just the "upvote" button?