npip99

Bradley-Terry and Elo scores are equivalent mathematical models! The fundamental presumption is the same Thurstone model - that an individual's skill in a particular game is a normally distributed random variable around their fundamental skill.

We did experiment with a Bradley-Terry loss function (https://hackmd.io/eOwlF7O_Q1K4hj7WZcYFiw), but we found that even better was to calculate Elo scores, do cross-query bias adjustment, and then MSE loss to predict the Elo score itself.

Yeah absolutely. In your link, it iterates on _ = ^{_}, until it finds the fixed point.

In our training pipeline, we had to convert the fixed point iteration to be on _ directly for numerical stability. I have a post on that here!: https://hackmd.io/x3_EkXGKRdeq-rNHo_RpZA

Bradley-Terry also very cleanly turns into a loss function that you can do gradient descent on, which will cause your model to efficiently learn Elo scores! Our calculations are at: https://hackmd.io/eOwlF7O_Q1K4hj7WZcYFiw

In our case training and inferencing the models takes days, calculating all of the ELOs take 1min haha. So we didn't need to optimize the calculation.

But, we did need to work on numeric stability!

I have our calculations here: - https://hackmd.io/@-Gjw1zWMSH6lMPRlziQFEw/B15B4Rsleg

tldr; wikipedia iterates on <e^elo>, but that can go to zero or infinity. Iterating on <elo> stays between -4 and 4 in all of our observed pairwise matrices, so it's very well-bounded.

We tried a bradley-terry loss function, as calculated with https://hackmd.io/@-Gjw1zWMSH6lMPRlziQFEw/SJ8sRl1Zge

We found that MSE after elo-adjustment worked equally well. And, MSE lets you shuffle (q, d) across the dataset which has good statistical properties (Versus contrastive, which makes you sample the same query many times within a single minibatch)

In this case "InfoNCE" isn't applicable because the reranker's output is a scalar, not a vector. So that's why we checked both bradley-terry and MSE.

Yes our pairwise method is based entirely on 2AFC comparisons, for both intra-query and inter-query ELO calculations.

It's definitely the best if not only way to get extremely high signal, and a score assignment that actually converges the more you sample.

In terms of the "F" in 2AFC, we actually have this amusing snippet from our prompt:

Do NOT output a score of 0.0, ensure to focus on which document is superior, and provide a negative or positive float between -1.0 and 1.0.

Yeah that's exactly what we observed. Our goal was to create an absolute score that's completely independent from the Corpus, which is difficult because naturally all ELO distributions are inherently tied to the corpus itself!

When we were exploring the mathematical foundations, we considered ELO scoring against a "Universal Corpus" based on the natural entropy of human language (Obviously that's intractable, but sometimes this term cancels out like in the DPO proof).

But eventually we figured out a method using cross-query comparisons to assign an "ELO bias" to all document ELOs within a given query's candidate list. This normalizes it correctly such that when a candidate list is all bad, the ELOs shift low. And when the candidate list is all good, the ELOs shift high. Even when the relative ELOs are all the same.

I often see it rendered as "Elo" but I've always found it more natural to capitalize as "ELO", but perhaps I should swap to "Elo" given this. Pronouncing "ee-low" is certainly the way it's done in chess/esports though!

Hey! We actually did a lot of research into ELO consistency, i.e. to check whether or not the NxN pairwise matrix followed the ELO model. It was a long road that's probably grounds for an entirely separate blog post, but the TLDR is that we observe that:

For each document, there is a secret hidden score "s" which is the "fundamental relevance according to the LLM". Then, when we sample (q, d1, d2) from the LLM, the LLM follows the statistical property that:

- The "fundamental hidden preference" is `pref = s_{d1} - s_{d2}`, usually ranging between -4 and 4.

- The LLM will sample a normal distribution around the `pref` with stddev ~0.2, which is some "inner noise" that the LLM experiences before coming to a judgement.

- The preference will pass through the sigmoid to get a sampled_score \in [0, 1].

- There is an additional 2% noise. i.e., 0.98 * sampled_score + 0.02 * random.random()

When we use Maximum Likelihood Estimation to find the most likely predicted "hidden scores" \hat{s} associated with each document, then we go ahead and sample pairwise matrices according to `0.98 * sigmoid( \hat{s}_1 - \hat{s}_2 + N(0, 0.02) ) + Uniform(0.02)`, then we get a pairwise matrix with virtually identical statistical properties to the observed pairwise matrices.

Yes, a step where you do a structured extraction into a database column would be a potential solution. But, it requires a preprocessing step.

It all depends on the use-case, sometimes you get a query that you couldn't have predicted the filter beforehand. In those cases, usually what you have to do is open up a spreadsheet and then manually categorize every document by hand. LLMs and modern AI are great ways to automate this.

A really good solution might be to have a system that computes these filters on-the-fly, but also caches them for later reuse if a query asks for that filter again.

Good question!

For long documents we have a rolling window strategy. So, we cut the document into 5,000 token groupings for use in inference. There's also a 400 token overlap, and we prefer the earlier chunk for overlap tokens.

For example, if Group #0 overlaps with Group #1 at index 5,200, then we use the logprob from Group #0, because it had more context. Group #1 gets the benefit of context for indices 5,000-5,400, even though we toss out the logprobs for that range.

Details about the groupings are here: https://github.com/ZeroEntropy-AI/llama-chunk/blob/0cac10a34...

--

No need to keep track of chunks that it's already made, we just want heatmap values and then we use those heatmaps to split at the hottest character that's around our target chunk length (Or use a threshold value and binary search the threshold for our target # Chunks or average chunk size).

Yep benchmarks are available at https://github.com/ZeroEntropy-AI/llama-chunk?tab=readme-ov-... , we used this dataset https://github.com/ZeroEntropy-AI/legalbenchrag which is a retrieval-focused version of LegalBench.

It scored better than LlamaIndex's recursive character text splitter and that was including some custom regex work to improve it. If you put enough effort into the regex you could probably get there, but the whole point of the agentic chunking is for it to be automatic and contextual.

Hey HN!

I've written a lot of RAG pipelines over the last year, and one consistent pain-point is writing regex to chunk the documents correctly.

Right now, the most common chunking algorithms are:

- Split every 1000 characters

- Split on whitespace

- Recursively split on: (many newlines, then one newline, then periods, then spaces)

The best is recursive character text splitter, but regex is super brittle and when it fails to match it ends up creating huge chunks. Worse, this solution also has the overhead of needing to maintain regexes for every single filetype.

Here we propose LlamaChunk, an inference-efficient method of LLM-powered chunking. Using this method, it only requires a single LLM inference over your document in order to provide the most optimal recursive character text splitting, without needing to hope that a bunch of hard-coded rules work on your unstructured data.

We're hoping to build a community for state-of-the-art RAG research @ https://discord.com/invite/yv2hQQytne , come join!

"Who wouldn’t prefer to stay home if they had the choice? I could spend my days traveling"

It's not travelling... If you're staying at home...

That's because the $50M were owned by a large percentage of the users of Ethereum, not a single person. It represented 15% of the total ETH in circulation, back when the currency was in a very nascent state and the flagship product that ETH provided that BTC didn't, was the DAO. When the DAO break happened ETH lost 60% of its value, so not only did 15% of people have their money stolen, but everyone across the board lost 60% of their ETH value. There was simply immense demand for people to get their money back, so people much preferred the fork that kept their money than the fork where the robber stole their money.

Simple as that, it's decentralized, that's the whole point, you fundamentally can't tell people which fork to believe, people use whatever fork they want. And the people mostly wanted the fork with their money in the DAO preserved. People who wanted the unaltered chain stayed there, no big deal.

Wait a second, am I missing out right now? I have an opportunity at Jane Street, but I haven't been exposed to any of the other small name firms. Where do I look to find the higher paying jobs?

This seems like a waste of time. It's not illegal to register a domain name, no matter how similar it is to your domain name. If you want those domain names, then buy them. My family runs a business and we bought the domain names that we thought were similar. We respect the fact that if someone buys our domain but with ".io" at the end instead, then that's their right. I would be very scared of a world where people can sue over names being "too similar to this large company with lots of money and power". These domain name companies serve one purpose: selling available domain names. It's not complicated. Let's not make it complicated.

I mean, I can be more lenient. Europe had good researchers. They were the Athens, between the Germans ones of the 20-50's, and Turing, there was a lot. The Russians got a lot done too. But again, all research. Nothing profitable. It was never self propagating. Rome was self propagating, they conquered. Athens, only sustained intellectualism until it bled away.

(I'm not counting the mathematicians of the 1700s that did amazing work, I'm referring to modern Europe and the US, 20th century and beyond)

It's kind of sad that a huge market for businessmen is taking an American business, and just doing the same thing in Europe or Latin America. It's, it's sad. There's uh, absolutely no foreign competition in most situations that we have to meet. The ratio is clearly wrong, and we compete with more companies than we can found so Europe should statistically be represented in the other direction. It's not even that profitable, American consumers are a gold mine they're so price insensitive. And the modern world is mostly what was invented in the roaring 20s in the US, and computing technology that was developed almost entirely in the US.

(The Japanese were trailing along in the 80s I guess, but they were mostly our China, they were just a factory line that we abandoned when China caught up and their GDP has stagnated for 20 years. They had a few great companies, and they still do. South Korea is the only actual one that goes toe-to-toe but they're too small to profit off 300 million people like we can, per capita I do think they are more entrepreneurial and inventive than us though)

There's not even VC money in Europe it's almost impossible to get funded, you need American or Chinese money which means they own most of the business anyway. Like in the US, take the money away, you can get started again. But in Europe? You better buy a plane ticket. No one will fund you. The non-tech companies are mostly incumbents. Or brands like WeWork that are much more expensive than other co-working spaces but they have the brand to hold them up.

Eh, I don't think so. There's too much interdependence, you can have anxiety over how many things your dependent on, especially in Tech. It's actually terrifying. If Azure closed down tomorrow I'd be fucked so hard, though any business will eventually try to be as independent as they can when they're big enough (Other than Netflix and AWS, I'm sure there's some under the table money making that happen as 50% margins apparently disappear from Netflix's wallet even though Netflix could just run it themselves). Too many companies and systems supply the whole world at a near monopoly, if epackets disappeared everyone would be fucked beyond belief. We all know that there's a lot of people involved, and we all do respect the workers too. Go to any area with businessmen, Grand Central Station or an international airport or something. And look at the section with books that are clearly targeted towards businessmen. At least 1/3rd of the selection is emotional support in response to having to fire people, and the emotional support of handling an economic downturn. People jumped out of buildings because of 1987 all over Downtown Manhattan, it was horrible. They know so many things need to be carefully in balance for them to continue.

You always have to be humble unless you're a billionaire because there's always someone vastly more powerful and rich than you are, by orders of magnitude. Your network's wealth and power as a normal distribution always ends up trailing behind you as you try to meet new people above you, but there's always the right tail end where you know a few people who could spend your net worth in a night because they want to.

https://www.youtube.com/watch?v=vrl5PFB35Ec

Look at how humble the CEO of the Maverick's has to be as he meets Mark Cuban. It's somewhere in there, I'm not gonna find the time but it's in there.

I don't know if you've ever seen the Black Mirror dystopia about social media and kissing up to people. There's a lot of that. It sucks. That's why most business events involve drinking until tispy, it loosens everything up.

I'm not sure what you mean by "Quality of life is just a sham invented by politicians to keep the plebs from demanding... quality of life?". I don't think politicians invented anything like that? I remain confused.

This is kind of what I touch on at the end, with the getting the kids into business, there is a moral concern there. The first executives are intelligent, and caring. They're the engineers. Once you have MBAs flooding in, which HR doesn't know how else to hire people, it just becomes horrible. Feynmann famously ripped NASA a new one after asking the engineers "What's the probability of the Shuttle exploding?". They say: 1/100, 2/100. He asked the executives at NASA, they say 1/100000, 1/1000000. How, how is that even possible? You need the executives to be the engineer, or they fuck everything up. An engineer warned them about the o-ring, the executives ignored it because they were stupid. Same with the max:

https://www.nbcnews.com/news/us-news/former-boeing-manager-s...

Notably: "For the first time in my life, I’m sorry to say that I’m hesitant about putting my family on a Boeing airplane," Ed Pierson wrote to a company executive before the first tragedy.

This would never happen if the executives were engineers. Every time we, humanity, eventually gets stuck in a rut. The Romans invent the most unbelievably complex and well structured government of their time, while the gauls hunt like animals, and they conquer the world, they develop all of the technology (Well, the greeks were even better with their near idealistic Athens, but like the PhDs it never gets implemented or spreads). So the Romans do this, they win. Then...., nothing happens. It goes to shit. The Patricians murder the Plebians who protest, they become corrupt, the people have less and less control, it becomes awful. Then, dictatorship. Then, emperor. Who were the people who drafted the first setup of the roman government? They're 600 years dead, and those who replaced them made it shit. Took 1000 years to recover. The founding fathers. Compare George Washington, and his morality, to our modern politicians. No one had any reason to do anything other than create the perfect government, everyone helped, it was collaborative. Now they're cutthroat. It always goes to shit. Corporations are the new target. The founding fathers tried so hard to put every single failsafe they could, because they knew. They knew they knew. They knew those who would come after them would make it shit. While the founding fathers intentionally put failsafes - the entire article is a big ass fail safe you just read it (It just says exactly what the restrictions are - if people acted just like the founded fathers did they would never need restrictions in the first place), our modern politicians research every possible exploit to the system as if they're in infosec. Gerrymandering, oops! Jefferson forgot about that one, it took 300 years to discover the buffer overflow, but if it's discovered it's abused. Every metric becomes a target. The target of "leading the revolution" gave people who were intelligent and passionate about saving their country. The target of "becoming a politician" in 2020 is https://en.wikipedia.org/wiki/Roger_Stone I highly suggest you read just the paragraph about high school. It shows what you need to be, what the selection criterion is. The only selection criterion that works is: Coming up with something new, a new business, a new idea, a new government. After that, they hunt. They get their MBAs to overthrow you with credentials so that they can squeeze themselves in, even though they didn't found Boeing. The patricians bribe who they had to to squeeze even, even though they never came up with the original government. The politicians cheat what they must to squeeze into what the founding fathers tried to prevent. Someone has their eyes on the prize.

Mind you, the French are just as intelligent as the Italians, they're nearly identical at birth, only culture separates them. A french man who grew up in italy is an italian man. It's the system that is what separated the gauls from the romans, and it's survival of the fittest, you're the first to have an opposable thumb then you win. Whoever figures out how to create a self propagating government wins, a couple people did, got a few thousand followers, and it propagated. Business is the new self propagating system, until stability is reached. Only the new people have what it takes. It's not really too genetic either, it is in some way have 4 kids one of them will work, what of the other three? Kahn couldn't stop it, Alexander the great couldn't stop it, Romans worked for 300 years because of adoption, but it was still bleeding. Go through the history. Each half century involved more failures by the plebians and more success from the corrupt patricians, it was slow but sped up, more and more restrictions were put on the people by the incumbents. You always need fresh people. Carnegie? Vanderbilt? Compare to the modern CEO of BP...., of Boeing, stability kills. In hackernews, for people in IT, no one knows shitty bosses and executives more than they do.

I mean if you have $100k in wealth you absolutely have travel freedom. Backpacking in Europe is a common experience so it's pretty cheap and the rest of the countries have crazy USD conversion rates. $100k could be a lifetime's worth of money in a non-western country. You can still find lodging in Ukraine for like $4/day.

The article says "Emits more pollution than all of Europe's Cars", which is a bit of a garbage claim, because obviously all of the CO2 is coming from the cars.

Why is giving a "Delivery Unlimited" option considered to by "Taking on Target Shipt, Amazon Prime, etc"? It's just a reorganization of what you pay, since you're going to pay for those shipping costs one way or another. And it's going to be through increased prices of the items in their store. They didn't actually do anything at all. How do people fall for this?

As much as this technology is powerful, you can't make it "illegal" to take photos of a public park, and then bring them home and do whatever you want with them at home. That would just be absurd. Banning the software online only isn't really an option, because the knowledge on how the software works is already public. Anyone could rewrite the code on their own (Esp because 300 line neural nets, no libraries used, are still pretty dam good at facial recognition). The real privacy should absolutely be on the data side. There should not be public databases of profiles. If you have a big insta profile or you are a big youtuber, then yeah, you'll be public even through you're not a celebrity. That was choice. Random people should be private. No mugshots should be freely available, and photos of social media profiles should be obscured by default. Or, alternatively, we accept that this is the way things are.

None of the top comments make any sense to me. I simply don't understand them. Anyway, here is my understanding of the subject:

(1) Yes, Uber should have trained the driver better to look at the road. By trained, I mean there should have been a sticky note on the wheel saying "PAY ATTENTION OR YOU WILL KILL SOMEONE"

(2) Yes, The driver absolutely should have to pay some penalty for this, if Uber told him that he should have been paying attention (Which they most certainly did). Watching a video while in a self-driving car is IDENTICAL to watching a video in a normal car. Modern self-driving cars are NOT fully autonomous, and they SHOULD be viewed as IDENTICAL to cruise control for all legal considerations, and thought experiments. Most of the top comments, which are blindly attacking self-driving cars, are not making this analysis.

However, (1) and (2) do not justify the lack of logic displayed by most of the top comments here. There seems to be violations of

(a) There is no logical difference between a person accidentally killing someone, and the self-driving car accidentally killing someone. Actually, because the car is already known to not be fully autonomous, this already is a case of the person accidentally killing someone. However, even if the car is fully autonomous, we MUST be considering the ODDS of an accident. None of the other comments are doing this. There is always an odds of an accident, so a specific accident means absolute bullshit. Literally nothing. This post doesn't even mean anything. What SHOULD be posted is "Self-driving cars with humans at the wheel kill X people per road-hour. Human-only cars kill Y people per road-hour". If X > Y, then yes we have a fking problem. But without that information, we literally have nothing to even think about, or process.

(b) Disabling the safety feature of the car is not a fk'ing concern, at all. Almost no cars have these features - only expensive ones do. Turning an expensive car into a normal car is NOT something you can be sued for, or is even something anyone should care about. Who is at blame for the situation is simply not dependent on this fact. I don't understand why people are discussing this aspect.

(c) Do you see the image? Maybe it's not showing it all. But as far as I can see there isn't a light there. Wtf is she doing crossing the road if there's no light there, without waiting for the cars? As a citydweller that is constantly found in the middle of streets trying to cross parts of the road that don't have stop lights, I just can't fathom ever being in her position. Maybe my city has less considerate drivers than her city, but if I tried to cross roads without a red light blocking cars, and didn't consciously give right of way to the traffic, I would die sometime this week.

(d) Many people seem to quote "self-driving system classified the pedestrian as an unknown object, then as a vehicle, and then as a bicycle" as "Oh this self-driving system is complete sh*t it's all Uber's fault". Makes no damn sense. As a side note, if you've ever worked with a neural net, especially with video as opposed to still photos, you already understand that the given sentence means nothing. That's just how they work, and there will always be milliseconds in-between frames where it reassigns the object's identification. But, anyway, this is not relevant. The self-driving part could have been completely off, or disabled. The car is SUPPOSED to be driven by the driver, and any deviation from that is the fault of either the driver, or Uber's training of the driver. Whether it's the former, or the latter, is exactly where and how suing should be directed and handled. Nothing else matters, despite most comments putting much emphasis on many other aspects of the situation.