Show HN: Out-of-the-box text classification models

https://www.ycombinator.com/launches/KfT-taylor-tag-your-text-data-at-scale

by brianjkim21 • 2 years ago

20 10 2 years ago

I like this approach it treats the problem like a reranker it would be interesting to byo classifiers you could do the embedding to match the hierarchy approach which is a good idea it seems. Also the code for Raptor came out recently and they are showing gains through a tree/hierarchy too.

https://arxiv.org/html/2401.18059v1

brianjkim213• 2 years ago

thanks. yup, it's on our roadmap :)

Terretta• 2 years ago

TL;DR: API needs user-defined taxonomies and better data confidentiality. Intent classification is a hit.

---

This is fantastic, but as you acknowledge with the 'reach out to us' on your launch page, people are going to need custom topic taxonomies. We use several custom ones, maintained as YAML that non-technical users can edit.

I'm guessing from having been looking for a project like yours for a decade now, that it's that custom taxonomy problem that means most OOTB don't work for people, so they make their own which they don't open source because they ended up ... tailoring ... a topic text classifier for themselves.

The only thing I've found close to this "OOTB" is:

https://cloud.google.com/natural-language/docs/classifying-t...

https://cloud.google.com/natural-language/docs/categories#ca...

And, to be frank, I can't see why I'd send my confidential information to you when I can send it to Google. (Ahem!)

But the problem with theirs and yours is the OOTB categories are for a global topic set, something like Yahoo directory, rather than for a given discipline. And what's generally needed is a set of disciplines, or several topic trees. (Think Amazon.com instead of Yahoo.)

I've found the general lists, like LCM[^1] (what you really want is LCSH[^2] subject headings, not LCM), too broad for my business or personal content, while something like ACM[^3] is more what's needed for, say, computing related content.

For a firmwide knowledge base at a {field}-tech firm, you have a mix of the firm's focus field, and computing, and a broad scope fallback like you're starting with. Even libraries have their own topic hierarchy! [^4]. Plenty fields have controlled vocabularies[^6], and if you can't find one for a field, you can usually generate one by finding someone who is already classifying that field, and looking at their TOC. All of which is to say, to be generally useful, you have to let people BYOT (bring your own topics) for this.

For instance, we built our topic list based on combining a reference taxonomy for our field, a reference taxonomy for computing, a reference taxonomy for business books, and the Google NLP tool mentioned above.

There are occasional tools that try to match arbitrary documents to arbitrary hierarchies such as clerk [^5] but they are challenging for various reasons.

You have a note to contact you for different topics, but raising this here since so far (6 hours) you had no feedback, and I'm a big fan of what you're doing and the niche is underserved.

A couple other thoughts:

Aside from topics taxonomy or hierarchy, we've recently found that something like properties based classification proves needed when we're 10K+ to 100K+ short and long form content documents in the knowledge base. For instance, https://en.wikipedia.org/wiki/Colon_classification, that adds "facets" like time dimension. This is incredibly helpful for relevance while still being able to drill in and just browse a topics/branch/leaf.

I really like your "intent" classification, far more interesting than sentiment, since it could help separate blog posts from new articles, self-guided tutorials from reviews, and so on: Problem Solving, News, Informational, maybe?. Sifting these to focus a robust KB can be tremendously valuable.

Your privacy policy is by-and-large useless, since the information being classified is unlikely personal (PII) class, and more likely confidential or non-public (NPI) class.

You are, effectively, saying "let us have a copy of all info you're classifying", yet nowhere on your main site nor docs site do you explain how you actively prevent yourselves from seeing an API user's information.

Ideally your "architecture" would explain how you built it to be able to do the work without you being able to see the content, not just a "pinky swear we won't look" sort of promise. Many businesses have their own confidentiality and privacy policies. Those require looping in subprocessors, which is you, and right now you can't be used.

Your API is on the right track with the types of classification already, especially its intent classification—it's a feature that users will find useful. But don't overlook the customization of topic classifications; it's what many are seeking and not finding elsewhere. The real concern, though, is confidentiality. You are asking firms to trust you with their content. Without that trust, even the best features can't win firms over because they don't have a choice but to control their data. Make sure firms understand exactly how you're keeping their data secure. Get this right, and you'll have a product that stands out for all the right reasons.

---

[^1]: https://en.wikipedia.org/wiki/Library_of_Congress_Classifica...

[^2]: https://id.loc.gov/authorities/subjects.html

[^3]: https://en.wikipedia.org/wiki/ACM_Computing_Classification_S...

[^4]: https://www.ala.org/tools/topics/atoz

[^5]: https://github.com/blankenshipz/clerk/tree/main

[^6]: https://pitt.libguides.com/metadatadiscovery/controlledvocab...

jsdeveloper• 2 years ago

Learned so many new things from your comments itself. Thank you

throwaway81523• 2 years ago

Any info about how it works? I'm more interested in the the technology than in using the product. Is there a query limit for the free API though? I do have a couple of ideas.

brianjkim213• 2 years ago

yup, we designed an ensemble that considers both lexical & semantic similarity and trained on large datasets of labeled text. also building data pipelines to prevent models going stale.

max 20 docs per request for free api. no official max doc length but recently had issues with large (think book length) docs.

ramses0• 2 years ago

You're doing yourself a disservice by making your "Class/Subclass/Sub-Subclass" outcome being a fading disappearing text gif.

It takes ~30 seconds for your animation to loop, of which ~8 seconds of it lets you see the outcome.

Once you've started to see/parse the outcome, then your (my) instinct is to go back to the initial (small, tough to read) "Support Ticket" text and see if it matches, then I'm back in the "Spend 22 more seconds to get 8 seconds of validation" loop. Times 4. Because you have 4 of these examples listed.

Give up and use tables: https://twitter.com/tectonic/status/552241947604054016

brianjkim213• 2 years ago

appreciate the feedback. will update, thanks.

yantrams• 2 years ago

Congrats on the launch. This is something I'd spent some time on few years ago. I hacked together something similar for my usecase by reverse engineering. No ML model though - Using Nearest neighbours and Tversky similarity measures in Julia with the same taxonomy that you are using.

Tested with one of the comments from this thread.

        requests.post(
            "https://x2vud9xfq0.execute-api.ap-south-1.amazonaws.com/api/text/classify",
            json={
                "text": """
                And, to be frank, I can't see why I'd send my confidential information to you when I can send it to Google. (Ahem!)
                But the problem with theirs and yours is the OOTB categories are for a global topic set, something like Yahoo directory, rather than for a given discipline. And what's generally needed is a set of disciplines, or several topic trees. (Think Amazon.com instead of Yahoo.)
                I've found the general lists, like LCM[^1] (what you really want is LCSH[^2] subject headings, not LCM), too broad for my business or personal content, while something like ACM[^3] is more what's needed for, say, computing related content.
                For a firmwide knowledge base at a {field}-tech firm, you have a mix of the firm's focus field, and computing, and a broad scope fallback like you're starting with. Even libraries have their own topic hierarchy! [^4]. Plenty fields have controlled vocabularies[^6], and if you can't find one for a field, you can usually generate one by finding someone who is already classifying that field, and looking at their TOC. All of which is to say, to be generally useful, you have to let people BYOT (bring your own topics) for this.
                For instance, we built our topic list based on combining a reference taxonomy for our field, a reference taxonomy for computing, a reference taxonomy for business books, and the Google NLP tool mentioned above.
                There are occasional tools that try to match arbitrary documents to arbitrary hierarchies such as clerk [^5] but they are challenging for various reasons.
                You have a note to contact you for different topics, but raising this here since so far (6 hours) you had no feedback, and I'm a big fan of what you're doing and the niche is underserved.
                A couple other thoughts:
                """,
                'key': 'HACKERNEWS'
            }
        ).json()
        
        
        {
            'genres': {'Technology': 24, 'Finance': 16, 'Education': 11},
            'tags': {'/Business & Industrial/Small Business/MLM & Business Opportunities': 5.094265117745211,
            '/Internet & Telecom/Web Services': 5.51434499612552,
            '/Finance/Investing': 5.72584536853734,
            '/Business & Industrial/Business Operations': 5.888633926463297,
            '/Jobs & Education/Education/Standardized & Admissions Tests': 6.0132143106028435,
            '/Business & Industrial/Business Services': 6.100261915913882,
            '/Jobs & Education/Jobs': 6.126547614437338,
            '/Science/Earth Sciences/Atmospheric Science': 6.1553064528175545,
            '/Finance': 6.249046550441405,
            '/Business & Industrial': 6.333431648078183},
            'id': '65f891a111ec14ddd4b56bda'
        }

Your result

        {
            "result": [
                [
                "/Arts & Entertainment/Books & Literature/Reference",
                0.138976
                ],
                [
                "/Jobs & Education/Job Listings",
                0.138976
                ],
                [
                "/Computers & Technology/Networking/Distributed & Cloud Computing",
                0.069488
                ],
                [
                "/Jobs & Education/Online Learning",
                0.069488
                ],
                [
                "/Arts & Entertainment/Music & Audio/Music Reference",
                0.046325
                ]
            ]
        }

aleksiy123• 2 years ago

I feel like it would be better to display the speed in ms instead of seconds.

30-50ms seems easier to read than 0.03s - 0.05s. especially since you say milliseconds in one of the taglines.

Also curious how no rate limiting works. Feels like a promise waiting to be broken.

Otherwise, really like that you have a nice hobby plan. Will need to find a place to try it out.

Custom topics is the killer feature though I feel.

Show HN: Out-of-the-box text classification models

Related Stories