Node Summary – Naive summarization algorithm for Node.js

http://jbrooksuk.github.io/node-summary/

by charlieirish • 12 years ago

32 14 12 years ago

jbrooksuk.github.io

jbrooksuk6• 12 years ago

Hey, developer of node-summary here! Thanks for posting!

I was wondering why it's being starred again :)

runj__• 12 years ago

It would be nice to see some example output.

jbrooksuk6• 12 years ago

I just added the example output, https://github.com/jbrooksuk/node-summary#example

Thanks!

quarterto2• 12 years ago

Literally nothing about this is node-specific. It uses node-style function(err, result) callbacks despite doing absolutely nothing asynchronous. Hooray cargoculting.

I notice it was originally written in synchronous style, but rewritten using callbacks. Callbacks don't magically make something CPU-bound asynchronous. They just make it full of callbacks.

jbrooksuk6• 12 years ago

Yep. It's easily adjusted to work in the browser, in fact I've been meaning to make it interoperable for a while.

The reason it's "node-summary" is because that's what I was doing at the time, Node.js and never thought it would be as popular as it's become, so it's not neat or well coded.

Feel free to make changes though :)

malandrew• 12 years ago

Yup. Wrapping synchronous code in a callback does not magically make all that wrapped code async. Instead it just makes it a really really large blocking function. I've seen this before in other libraries and tools in the node.js ecosystem when I want to use something on the server instead of at the command line. I've ended up having to completely rewrite some libs/tools to make them full async for server use because their use of synchronous code internally was everywhere in the code base.

I'd love it if there was a quick way to analyze a code base to tell you what level of asychronicity you've achieved. I don't know how such a tool would be written, but I reckon you'd need to start with a list of node.js core API functions sorted into those that are sync and those that are async. Then the level of sync to async is roughly a function of average function size and the ratio of sync to async functions. Many small async functions that consume async core functions would be ideal.

That being said, still kudos to the author of this lib and other others for contributing code to the commons.

morganherlocker• 12 years ago

I would point out that there are cases where sync callbacks do make sense (granted, this does not look like one of them).

1. You want to add optional results. See underscore's each function that returns an object for the current iteration, then an optional iterator value in addition, which you sometimes need.

2. There are many functional patterns that utilize callbacks where you want to pass functions in as parameters (which could do anything, including accessing the file system or waiting on an external call). Underscore also has loads of examples where this is the case, despite being a sync library.

3. This is probably the least valid reason, but sometimes you do not know going in whether or not a function you are writing will be async. If you create a sync interface, it will need to be changed if the function needs to be async in the future, or if it needs to pass additional results. In my experience the latter is not terribly uncommon. The former is pretty unlikely in library code, but I could see it happening in application code regularly. Breaking apis is not fun for anyone.

quarterto2• 12 years ago

Callbacks ≠ passing functions as arguments. Callbacks (in the node sense) is passing a single function as the final argument which will be called exactly once with an error or result.

The function passed to _.each is used to access the elements of an array. It is called once for each element. It is used as a mechanism to pass a block of code to _.each, not to get a result from _.each.

PLenz• 12 years ago

I've used this in a few production projects where I work - it does the job it needs to do.

jbrooksuk6• 12 years ago

Amazing! I'm glad it's worked for you :)

ptwobrussell• 12 years ago

Nice work. This looks similar to an algorithm developed in the 1950s at IBM by H.P. Luhn. Luhn's algorithm basically finds the most "important" words (mostly via frequency) and then ranks the "importance" of sentences according to the co-occurrences of these "important" words. The summary of the document then becomes the "top n" sentences in chronological order.

An implementation of Luhn's algorithm (which accepts a URL or HTML input and also handles relatively clean text extraction with boilerpipe) is available here: http://nbviewer.ipython.org/github/ptwobrussell/Mining-the-S...

I haven't kept up much with "state of the art" summarization techniques, but this approach is pretty intuitive and works about as well as anything I can imagine (without getting into machine-driven language generation, which is a whole different topic.)

jbrooksuk6• 12 years ago

Thanks!

shlomib• 12 years ago

One thing, If you take a look at line 53 in my original code: https://gist.github.com/shlomibabluki/5473521#file-summary_t...

You can change the code to run only -> for j in range(0, i)

* (instead of range(0, n))

and then store the same score both in values[i][j] and values[j[i]

jbrooksuk6• 12 years ago

Thanks for your original work! It was perfect to learn from, so thanks again!

I'll look at implementing your changes shortly, perhaps after a rewrite to circumvent the blocking behaviour experienced with the callbacks.

Node Summary – Naive summarization algorithm for Node.js

Related Stories