utkuumur

I think one problem is that the models that hallucinate often, a few times out of 8 or 16 so that they get good results on benchmarks, most of which measures success out of top k. From benchmark perspective, you don't really care whether 15 of yours 16 generations failed, as long as one succeeded, but as a user you mostly care that 1 out of 16 you get is actually the successful one. I think this effects is more easy to see on Gemini Flash, it hallucinates like crazy but looks like its by design to boost benchmarks.

I don't see why many people complaining on this issue. Not everyone mastered English unfortunately. I am especially very weak at writing a paper, and to be honest, find it taxing. I love research but after having results, turning it into a paper is not fun. I edit almost everything important I write like emails and papers with LLMs because even though the content is nice my writing feels very bland and lacks lots of transition. I believe many people do this and actually, this helps you learn over time. However, what you learn is to write like LLMs since basically we are supervised by the LLM.

I couldn't agree more! What am I supposed to do with the related work section? Especially after reading many similar works in the field, it is very hard not to be influenced by what you read but you have to make sure not to say the same thing.

I believe it helps but not the sole reason. Because there are also autoregressive models that perform slightly worse. Unsupervised learning + Diffusion + Neural Search is the way to go in my opinion. However, currently, the literature lacks efficient Neural search space exploration. The diffusion process is a good starting point for neural search space exploration, especially when it is used not just to create a solution from scratch but also as a local search method. Still, there is no clear exploration and exploration control in current papers. We need to incorporate more ideas from heuristic search paradigms to neural network CO pipelines to take it to the next step.

There are recent papers based on diffusion that perform quite well. Here's an example of a recent paper https://arxiv.org/pdf/2406.01661. I am also working on ML-based CO. My approach has a close 1% gap on hard instances with 800-1200 nodes and less than 0.1% for 200-300 nodes on Maximum Cut, Minimum Independent Set, and Maximum Clique problems. I think these are very promising times for neural network-based discrete optimization.