HN user
Leading up to ChatGPT, virtually no model innovation was needed. Just some minor tweaks to activation functions and the order in which operations were completed in the feedfoward layers. So they were essentially trying to trademark the innovation published openly in Attention is All You Need?
Whereas vanilla scaled dot-product attention forces outputs to fall in the convex hull of V, your proposal allows them to drift into the convex hull of V+{0}. It's pretty unlikely that the origin is already contained in the convex hull of V especially for long inputs, so your proposal genuinely changes the quality of outputs.
I was only matching queries against parts of the YouTube auto-generated captions. In theory, this should work for queries that are guests' names because Lex states the guest name in the beginning of the episode. But, in practice it doesn't because of transcription errors and how results are aggregated/sorted/filtered. Now, I'm matching queries against text that accompanies each episode as well as the transcripts and separating results into "Episodes" and "Clips" sections. This seems to enable querying by guest. Thanks for the feedback!
Sure! https://github.com/joepatmckenna/podsearch. Thanks!
For some videos, embedding on third-party sites has been disabled. Previously filtered, but revised now. Thanks!
As a side project, I created a search engine for the Lex Fridman podcast. It uses https://huggingface.co/sentence-transformers/all-MiniLM-L6-v... to encode the YouTube auto-generated captions. Feedback is greatly appreciated!