The IIT Bombay English-Hindi Parallel Corpus [pdf]

by aq3cn • 9 years ago

103 13 9 years ago

mongodude• 9 years ago

Since so long, I have been waiting for Indian universities especially IITs to invest and publish in building such corpora. Being a founder of AI/ML startup, I am surprised at the appalling lack of datasets available to work on Indian problems. Contrast this with Chinese universities where they have built some world class datasets to build NLP solutions in Mandarin. Our sentiment analysis works in 8 different languages but none of it is in Indian languages despite we being in India!

ravivooda• 9 years ago

I guess it’s not the most demanding requirement at the moment. But happy to see progress.

Btw, where do you work? Can we see your work?

woodson• 9 years ago

The data set is released as CC-BY-NC and, thus, cannot be used in commercial applications.

rspeer3• 9 years ago

And also cannot be used in open-source projects.

CC-BY-NC amounts to saying: you can play around with this in demos and academic projects that no lawyer would ever go after anyway, but you can't use it for real.

muktabh• 9 years ago

A lot of universities are open to give a separate commercial license when contacted. They charge for their efforts, which is fair. Whether public universities should charge given we already pay them from our taxes is a different issue. source: I am also cofounder at an AI startup, we often buy licenses to use academic datasets for commercial usage.

gumby2• 9 years ago

Being it's IIT Bombay I hope for Marathi some day soon.

I know different people have different priorities :-)

praneshp2• 9 years ago

Does it matter that it is in Bombay? I thought the prod/student body would be diverse due to to how the intake works.

Nothing against Marathi, I feel like Telugu, Tamil etc would also have equal chances of happening there.

gumby2• 9 years ago

Oh not really, I do agree that languages like Tamil, Telugu, Bengali and others are sort of “more deserving” in that they have more speakers than Marathi. I just don’t speak them and probably never will :-( .

The “Bombayness” was only that Mumbai is in Maharashtra. There really are great folks from all over at that institution. My comment was NOT intended to make any implication about any community over another! I see someone downvoted my comment and might have understandably thought that that might have been my intention.

The IIT Bombay English-Hindi Parallel Corpus [pdf]

Related Stories