HN user
github.com/faustomorales
I think the “interests” that might “conflict” are “business” vs. “public policy.”
For example, a criticism an opponent of Al Gore might make is something like, “Would you still be an advocate for climate policy X if you didn’t stand to gain $Y due to its implementation?”
Of course, I’m not saying the criticism necessarily holds up to careful scrutiny. And, from what I can tell, neither does the author of this article.
Right you are, yes, it was that same message. I suppose what I meant to convey is, "despite the thoughtful safeguards built into Slack, I still managed to screw this up." :) I really appreciate Slack and it is my preferred workplace communication platform, so I hope no one reads my anecdote as criticism of the product. I can't think of any way Slack itself could have done more to prevent my mistake.
As more context, I do remember thinking, "that X number seems higher than I would expect, but maybe we had a lot of folks who signed up for the event that didn't show." I worked at a small company so the size of the company was on the same order of magnitude as the number of invitees to the event (~100). I explained away the Y time zones because I knew some people traveled internationally. I was also operating on very little sleep, so that probably didn't help.
Seems like an innocent enough mistake. Remind me of when I once helped organize a volunteer hackathon with people from different companies. We created a Slack organization just for the occasion. At the end of the event, I was supposed to ask all attendees to delete all the data we had given them before they went home. The message I posted to #general (the channel everyone in a Slack organization is required to be in) with @everyone tagged was something to the effect of:
"Thank you everyone for contributing to our shared mission. When you are done with your work today, please delete all the data from your machines. Hope we see each other again soon!"
Slack gave me a dire warning that my message would send notifications to so many people across so many time zones. This didn't surprise me because attendees came from various countries for the event. So I dismissed it.
But I had accidentally sent it to my company's Slack organization instead of the hackathon-specific one. I didn't realize it until a co-worker sent me a private message asking why I had just tried to fire everyone at our company.
Hi HN! I frequently find myself needing to label small batches of images with class labels as well as bounding boxes and masks. Last year, I developed a Python library to do that sort of thing relatively quickly using a command line application that launched a web interface. But I often want to do it right inside the Jupyter instance that I'm already in. So I recently developed the functionality into a Jupyter Widget. To make it easy to try out, I've made a short Colab notebook that shows what it can do.
Hi HN! I created remstat (short for: remote status) because I found myself wanting a simple way to monitor long-running tasks like training a deep learning model, downloading and processing a bunch of images, etc. I was tired of running over to my laptop to check for a pulse and see how things were going. A lot of great services out there would have met that need, but they also seemed to involve setting up a database or installing service-specific libraries or integrating APIs designed for use cases that were more sophisticated than what I really wanted or needed for these one-off tasks. I just wanted to record some numbers and see them on a plot on my phone, ideally using nothing other than the basic Python libraries I already know and love. I'm sharing it in its current, very basic state because I have a sneaking suspicion that what I was looking for already exists and that I just didn't do a good job finding it. Feedback welcome, especially of the form, "XYZ already does exactly what you want, and it has the option to do much more."
You do have to set up an account and sign in to use the service, which I know is not appreciated in a Show HN. But the main page has a pretty clear demo showing how it works to help you decide if it's worth a look, the privacy policy is straightforward, we only collect your email address, there are no trackers at all, and you can delete your account along with all your data with just one click.
I developed QSL because I found myself appreciating existing labeling tools (e.g., VGG Image Annotator, labelImg, VoTT) while also wanting additional features (e.g., remote file labeling, batching, keyboard shortcuts) that didn't all fit neatly into any one of them. Would greatly appreciate feedback from this community, even if it's just to say that I've needlessly re-re-invented the wheel.
If you are (a) willing to take the word bounding boxes and convert them to paragraphs yourself, and (b) okay with a deep learning approach, you may want to give keras-ocr [0] a try.
Full disclosure: I'm the primary package developer. Shameless plug. :)
Tesseract is a great solution for scanning books -- but agree that it doesn't work very well for most other use cases, especially when compared to cloud providers. FWIW, I have started trying to compare keras-ocr against the cloud options. [1]
[1] https://github.com/faustomorales/keras-ocr#comparing-keras-o...
Hi HN! I made this because I wanted a toolkit for training custom OCR models that included both text detection and recognition along with the necessary tools to create synthetic data. Existing synthetic data generators had more dependencies and set-up than I felt was absolutely necessary so I took a different tack that limited dependencies to PIL only.
Some use cases for this package:
- You can use the pretrained (trained by others!) models for OCR (see the README for an example) on English text. [0]
- You can fine-tune a version of the detection and recognition models on a different alphabet / language (see the tutorial [1]).
- You can just use the data generator with backgrounds and fonts (I provide a packaged set of both) to create images with character-level annotations for some other model [2].
I'd really like to continue improving the image generator to render more realistic images while retaining the existing mix of simplicity / flexibility. Ideas welcome!
[0] https://keras-ocr.readthedocs.io/en/latest/examples/using_pr...
[1] https://keras-ocr.readthedocs.io/en/latest/examples/end_to_e...
[2] https://keras-ocr.readthedocs.io/en/latest/examples/end_to_e...
Interesting that you mention ExxonMobil. ExxonMobil does, in fact, cite "technological advances in energy storage that make wind and solar more competitive for power generation or increased consumer demand for alternative fueled or electric vehicles" [1] as a risk factor on their annual form 10-K.
In a bit of a twist, worth noting that Exxon Research & Engineering was involved in the invention of the rechargeable lithium ion battery [2]. Quoting from [3], "Thus Stanley Whittingham invented the first rechargeable LIB, patented in 1977 and assigned to Exxon."
[1] https://ir.exxonmobil.com/static-files/37b7e1b7-620b-4248-a3...
[2] https://patents.justia.com/patent/4009052
[3] https://www.currentscience.ac.in/Volumes/117/09/1416.pdf
Disclosure: I worked for ExxonMobil from 2012-2017 but don't have any remaining ties.
Perceptual hashes in this family will probably not be the right fit for that use case. There is work out there to build hashes that are intended to find find semantically similar content, typically using CNNs. Imagededup [1] seems to have one of these though I have never used it myself.
Hi there! Our example on deduplication [1] takes the case of deduplicating of the Caltech256 [2] dataset for the purpose I think you're describing, which I interpreted as avoiding duplicates in a dataset so that you don't have images that end up in both your training and test sets. Even though the dataset contains >30K images, you can still do this in memory (and find a handful of duplicates!) because each category is relatively small and you probably don't need to deduplicate images of trains with images of dogs.
On the same page, we mention two tools (specifically, FAISS [3] and Annoy [4]) to help with doing approximate search for scales where computing the distance matrix is impractical.
[1] https://perception.thorn.engineering/en/latest/examples/dedu...
[2] https://authors.library.caltech.edu/7694/
Great question — organizations decide how to handle reviews internally. So the answer to your question on “review all” versus “automatically submit” is a, perhaps unsatisfying, but honest: it depends. We provide a guide [1] to help organizations formulate their own policies. And we're currently working on a content moderation tool that focuses on helping organizations operationally handle problematic content and considers the wellness and resiliency of reviewers.
[1] https://www.thorn.org/sound-practices-guide-stopping-child-a...
Thank you so much for asking! Naturally, part of why we wanted to share this with the broader community was to make it so interested people can jump in and help out in the open. And I would be remiss if I didn’t mention the fact that we’re hiring! [1]
We can always use help raising awareness. Advocating for more survivor resources is a great place to start. Help spread the word through your social networks by connecting with us on Facebook and Twitter. Learn about even more ways to get involved and subscribe to our newsletter for general updates [2].
Thanks, old friend! Hope you decide to give the package a peek and maybe lend us some of your Python chops. :)
Great question -- all of the hashes we included in the package have been public for years (except for PDQ, which was open-sourced this year). So this package doesn't reveal anything new with respect to the algorithms themselves. What we add is an easy path to using at least one of them for the CSAM hashing / matching use case. Non-public perceptual hashes for this use case exist and, naturally, are not available in the package.
Cool (re: writing a PHP version)! Generally speaking, hashes from different hash algorithms cannot be used with each other.
By the way, would be glad to connect and chat, especially if you have any thoughts on pain points associated with perceptual hashing (email in profile).
Great question! PhotoDNA is the most well-known and supported hash function for this use case. And we do support hashing and matching with PhotoDNA in the Safer product. However, the PhotoDNA hash function is non-public so we cannot include it in an open source package. We support pHash as an open source alternative so that companies without PhotoDNA licenses can get started with hashing.
To build a classifier, yes, you are correct. But this isn’t a classifier to identify new content that has never been seen. This uses perceptual hashes to help organizations detect if known CSAM is being shared on their platform.
Thanks for asking these important questions!
* What are the challenges surrounding verification that your system functions properly, given that the test material is illicit?
You're right that storing child sexual abuse material (CSAM) is illegal, unless you are the National Center for Missing and Exploited Children (NCMEC) or law enforcement. What is legal is to maintain a hash of known CSAM. NCMEC, Law Enforcement, and large tech companies maintain their own data sets of known CSAM hashes and, where appropriate, share them. The Technology Coalition [1] has more information on this. All that said, we can and do simulate the system to verify that it works properly using bench testing with non-illegal content [2].
* Can you speak to the reliability of the system in a sensitivity/specificity kind of way? In other words, what are the false positive and false negative rates?
The false positive rate in practice is very low. We set our thresholds based on bench tests with an expected false positive rate of less than 1/1000 (the thresholds vary based on which hash function was used). Different hash functions are more resilient to some transformations than others (e.g., cropping, watermarks, etc.).
For the false negative rate, it depends entirely on the kind of modification made to the image. For many common operations, it is close to zero.
* Are you aware of any large organizations leveraging your solution?
Thorn builds technology to defend children from sexual abuse, one of the products we build for this purpose is Safer [3]. Perception provides an easy way to get started using the Safer matching service. Safer provides a more robust and complete solution including handling a queue of content and reporting tools. Some organizations using Safer include Imgur, Flickr, and Slack.
But this technology (perceptual hashing) is used by many companies who don't use our tools. Our goal is just to make it easier for more people to get started.
* Do you feel that the availability of these tools obligates service providers to use them, either morally or legally?
Not being a lawyer or a public policy expert, what I can say is that the law, as I understand it, requires companies to report CSAM once they are aware of it. Working in this field I’ve learned two things pertinent to this question: (1) Most people don’t know how pervasive of an issue this is, and (2) There aren’t a lot of easy ways to start protecting your platform from this abuse. No one wants the cool new products and platforms they make to be used to abuse children. Privacy is important too, which is why solutions that preserve privacy and avoid leaking private information to third parties are critical, and perceptual hashing allows us to do both.
[1] https://www.technologycoalition.org/
[2] https://perception.thorn.engineering/en/latest/examples/benc...
EDIT: Line breaks
Hi HN! I'm a data scientist at Thorn, a non-profit dedicated to defending children from sexual abuse. We're excited to open source some elements of our perceptual hashing tooling as a Python package. We've tried to make it very flexible both for ourselves and hopefully also for others. Our aim with is to provide tools that (1) help more people eliminate child sexual abuse material from the internet and (2) assist with common tasks where perceptual hashing can be helpful (e.g., media deduplication). We hope you'll take a look, get some use out of the package (check out the example use cases), and even contribute feedback and/or code to make it better.
For more information on the issue, I urge you to check out our CEO's TED talk here: https://www.thorn.org/blog/time-is-now-eliminate-csam/
Documentation for the package here: https://perception.thorn.engineering/