The Silent (R)evolution of SAT

https://cacm.acm.org/magazines/2023/6/273222-the-silent-revolution-of-sat/fulltext

by panic • 3 years ago

245 103 3 years ago

cacm.acm.org

degoodm• 3 years ago

"Is progress in SAT solving the sole result of hardware advancement? Time Leap Challenge compared 20-year-old SAT solvers on new computer hardware and modern SAT solvers on 20-year-old computer hardware. Although hardware improvements make old solvers faster, algorithmic progress dominates and drives today's SAT solving."*

Pretty cool given computing progress over the last 20 years:

  1. CPUs sped up 40-60x
  2. GPU FLOPS/$ increased ~10,000x

Sources:

  1. https://www.cpubenchmark.net/year-on-year.html
  2. https://en.wikipedia.org/wiki/FLOPS#Hardware_costs

*Quote shortened for brevity

sanxiyn5• 3 years ago

You may want to read "A Time Leap Challenge for SAT Solving". It's just a really fun paper: https://arxiv.org/abs/2008.02215

bmc750514• 3 years ago

And yet, despite tremendous progress accelerating continuous optimization workloads, there are still no competitive GPU-based SAT solvers. I wonder why?

sanxiyn5• 3 years ago

It's not just GPU. There are not even good multicore SAT solvers. State of the art SAT algorithms are serial and they don't parallelize.

porcoda• 3 years ago

Super short answer: SAT doesn’t exhibit the nice properties other optimization problems often have where you can tell when you’re near a correct answer. So it doesn’t resemble most continuous optimization problems. Regarding parallelism, there has been research into parallel SAT, but it’s hard in part due to the problem of sharing information between threads/tasks in conflict clause learning algorithms. I don’t remember specific papers but a quick search on google scholar or looking at past SAT competitions would turn up material if you’re interested in the topic.

cwzwarich• 3 years ago

Unit propagation (a major component of CDCL-based SAT solvers) is P-complete, and thus has no effective parallel algorithm.

GregarianChild3• 3 years ago

Because all known SAT algorithms rely heavily on data-dependent branching based on non-local data. That makes the existing algorithms slow on GPUs. An algorithmic breakthrough is needed to change this.

zero_k5• 3 years ago

There is a solver that can make effective use of GPUs (shameless self-promotion): https://github.com/nicolasprevot/GpuShareSat

It is a VERY fun work. Code entirely written by Nicolas Prevot, a magician of CUDA. Paper link here: https://comp.nus.edu.sg/~meel/Papers/sat21-psm.pdf

thesz3• 3 years ago

GPU's are not friends with pointer chasing, which is the basis of contemporary CDCL solvers, because internal lazily-updateable structures are multi-linked lists.

They are not friends with bottlenecks like clause queues used in contemporary stochastic SAT solvers.

They can be good at something like survey and/or belief propagation [1].

[1] https://simons.berkeley.edu/talks/belief-survey-propagation

The problem is that survey/belief propagatin is effective in the area of random CNF problems and it is not a complete solution technique. It may compute a solution but it cannot provide you with the evidence that there is no soolution.

xavxav5• 3 years ago

A fun exercise is to write a series of sat solvers: brute force, DPLL and CDCL you really get an immediate feel for the strength of the algorithms and none of these implementations need to go much beyond 50-100 lines in a modern language.

You can then of course spend months tuning and optimizing the hell out of your code to squeeze a further 10x performance.

If you're really crazy like Sarek, you can also formally verify these implementations: https://sarsko.github.io/_pages/SarekSkot%C3%A5m_thesis.pdf

ykonstant• 3 years ago

What a lovely and well-written thesis; I am reading it with great interest.

philzook• 3 years ago

Excellent link. I would love to see a CDCL implementation in a 100 (understandable) lines. From what I see in the repo, creusat is probably more like 1000?

GregarianChild3• 3 years ago

It's worth mentioning Mathias Fleury work who first verified a CDCL implementation [1]. He has written follow-up stuff.

[1] https://fmv.jku.at/fleury/papers/Fleury-thesis.pdf

geysersam2• 3 years ago

Parallelizing SAT solvers seems like a strikingly important avenue for future research.

hgsgm• 3 years ago

Similar to faster-than-light travel.

doetoe2• 3 years ago

I'm not an expert in any of these, but in the past few years, in addition to the success of LLM's for natural language processing, I have repeatedly read about the impressive advances in SAT solvers and in proof assistants. Given how impressive LLM's are in spite of their inability to reliably perform reasoning tasks or follow instructions with logical strictness, I wonder how much more impressive it could get if such systems got integrated

sanxiyn5• 3 years ago

You will enjoy "Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations". It prompts LLM to generate tree of explanations and run MAX-SAT solver over it: https://arxiv.org/abs/2205.11822

doetoe2• 3 years ago

Thanks, I'll check it out!

ComplexSystems2• 3 years ago

I don't really get why people think P≠NP anymore. I mean, isn't everyone expecting some kind of superintelligent AGI that can recursively improve itself to turn the planet into goo? Don't people think that this kind of AGI will get so good at nonlinear optimization that it will turn all humans into paperclips as a way to maximize the output of a paperclip factory? Why do we think that this insane next-level intelligence will somehow be able to do things like that, but not be able to figure out what pattern of bits gets a bunch of AND and OR gates to output a "1"?

10-15 years ago, the basic idea was "P≠NP because otherwise computers could do crazy shit." Well, looks like they can do crazy shit!

adwn• 3 years ago

but not be able to figure out what pattern of bits gets a bunch of AND and OR gates to output a "1"? 10-15 years ago, the basic idea was "P≠NP because otherwise computers could do crazy shit." Well, looks like they can do crazy shit!

I think you've fundamentally misunderstood the meaning of the P!=NP problem. In very simplified terms, it's not about what a computer can do, but about how long it takes to do something in the worst case.

sirwhinesalot• 3 years ago

There already exist local-search based algorithms that can find solutions way faster than a SAT solver can... Or they get completely stuck unable to make any progress.

All an LLM does is guess the next token based on the previous tokens and its training weights. For it to give you a solution to a large SAT problem it'll have to spit out a million character long binary string.

The likelyhood most of that string will be entirely hallucinated is very high. LLMs are not magic.

Deep Learning is already used internally in some SAT solvers to heuristically pick in which direction the search should go.

ftxbro2• 3 years ago

"10-15 years ago, the basic idea was "P≠NP because otherwise computers could do crazy shit."

Everyone is downvoting you because that's not the mathematical explanation, but it's absolutely the informal explanation that the 'pop science' ones were saying.

bmacho2• 3 years ago

I don't think it was the informal explanation ever. People think that P≠NP, because they tried it hard, and did not find any NP-hard problem in P.

ComplexSystems2• 3 years ago

I don't mind the downvotes as I expected this would be somewhat provocative, going against the general grain that people think P≠NP. I agree that, in general, this is not a mathematical explanation for why P≠NP, but a rough heuristic to explain why people think it is true, predominantly in pop sci circles. I would also emphasize that there is no mathematical explanation for P≠NP - or else the problem would be solved - and at the end of the day, the reasons that computer scientists suspect this to be true are equally heuristic in nature, not amounting to much more than "we tried to solve these problems and couldn't do it."

abetusk5• 3 years ago

P != NP is more likely akin to a fundamental mathematical law, like Noethers theorem [0]. No amount of self improving AI will alter the fundamental laws of physics. More provincially, in our solar system, there's a cap on energy so that any system that uses it will eventually hit a ceiling (like a self improving AI that eats up energy resources at an exponential pace).

As motivation for why P != NP, one can think of it as a finite restatement of the Halting Problem [1] in the form of asking whether a (polynomially sized) Turing machine has an input that will halt in K steps.

[0] https://en.wikipedia.org/wiki/Noether%27s_theorem

[1] https://en.wikipedia.org/wiki/Halting_problem

js89• 3 years ago

As motivation for why P != NP, one can think of it as a finite restatement of the Halting Problem

That's a really poor motivation. Analogically, you can think of the NP question as solving a polynomial equation over Z_2. What you're arguing is that because for Z the problem is unsolvable (due to DPRM theorem), then for Z_2, we have to try all the solutions.

That argument fundamentally ignores that the source of unsolvability is the infinite nature of Z.

For the record, I believe that P=NP (see https://news.ycombinator.com/item?id=35729626). I wish people spend more time looking for polynomial algorithm using logic approach, as I am doing (trying to extend the 2SAT and XORSAT proof to 2XSAT). This approach could give new algorithms that actually compose and can be paralelized (DIMACS is a really awkward representation).

ufo2• 3 years ago

The catch here is that P vs NP talks about worst-case behavior. Many SAT solver inputs can be solved quickly with good heuristics, but there are some that are still difficult no matter what you do. One way to see this is via the phase transition behavior of SAT.

https://www.researchgate.net/profile/Lavallee-Ivan/publicati...

If you have too few constraints it's easy to find a solution to the problem. If there are too many constraints it's also easy to show there are no solutions. But in the middle there's a sweet spot in the ratio of constraints to variables that's when the problem becomes fiendishly difficult.

bigbillheck• 3 years ago

isn't everyone expecting some kind of superintelligent AGI that can recursively improve itself to turn the planet into goo?

I most certainly am not.

Hirrolot• 3 years ago

I wonder what is the target audience of articles like this. I think the article is well-written, however 90-95% of the terminology make little or no sense to me who is not proficient in this field. Would the article be useful for someone who _is_ proficient in the field?

bjarneh2• 3 years ago

Would the article be useful for someone who _is_ proficient in the field?

Yes, it is a very well written summary of what has been going on with SAT solvers in the last decades. I personally think most of this (r)evolution can be attributed to miniSAT. A relatively easy to read (at least compared to other theorem provers), free software implementation of a SAT solver, with instantiations if I remember correctly. Since variables only have two possible valuations (true/false) in boolean logic, they can be instantiated and the problem can be split up, and sometimes greatly simplified in doing so. I.e. we can replace each occurrence of variable 'a' with 'true' to make a new smaller problem, and then replace each occurrence variable 'a' with 'false' to make another smaller problem. I.e. split 1 problem into 2 simpler problems, and solve them in parallel etc.

sanxiyn5• 3 years ago

As I remember, Glucose was a larger advance than MiniSat. MiniSat was well-engineered but not exceptional. Glucose started as a hack, but the heuristic it introduced was so effective that you basically couldn't compete without copying it.

Jaxan• 3 years ago

The target audience is members of the acm. So that mostly includes computer science researchers, I guess.

YeGoblynQueenne3• 3 years ago

It's not just you. I found the technical sections of the article impenetrable. And yet my background is on resolution theorem-proving (I did my PhD research on Inductive Logic Programming) and I read the article wishing to understand how CDCL is related to Resolution, but I couldn't figure that out.

I don't have a background in SAT solving but the "clause learning" in CDCL looks a lot like what we call a Resolution-derivation in Resolution theorem-proving. In short, given a conjunction of two (propositional or first order) clauses (¬φ ∨ ψ) ∧ (φ ∨ χ) Resolution eliminates the contradiction in the "complementary pair" (¬φ, φ) and derives the new clause ψ ∨ χ. If that's what CDCL folk call learning then that's really interesting to me because most of Inductive Logic Programming is an effort to use Resolution for inductive inference i.e. machine learning, but in logic. I guess I should check the article's bibliography. They reference s a textbook that might be useful ("Handbook of Satisfiability". Though one of the authors is an author of the article so that may not bode well for the textbook).

xavxav5• 3 years ago

Conflict explanation in CDCL is entirely based around resolution, the idea being to take a conflicting set of clauses and apply resolution a bunch of times to “simplify” the conflict into its most general form.

js89• 3 years ago

The resolution is actually also what 2SAT algorithm is doing.

vivegi• 3 years ago

SAT is tantalizingly simple to describe and is an intense rabbit-hole if you are intellectually curious. I liked the article.

The SAT Competition [1] is a good place to find state of the art.

[1] http://www.satcompetition.org/

xavxav5• 3 years ago

I wonder if they'll write a followup on the SMT revolution that happened after the CDCL breakthroughs for SAT.

The lazy approach to SMT was a huge step forward, but itself spawned a whole lineage of refinements, alternatives and changed the state of automated reasoning entirely.

A lot of work today seems to be going into theory combinations so that you can efficiently answer problems for example involve arrays and integers in a manner that has dependencies between the two.

'Solving' (it's an undecidable problem) SMT would open up a bunch of new possibilities..

maweki• 3 years ago

it's an undecidable problem

In what sense? People often use SMT (Satisfiability modulo theories) but don't state _which_ theories. I often SMT-solve with decidable theories, like Presburger Arithmetic.

xavxav5• 3 years ago

I almost exclusively use it with quantifiers which makes it equivalent to FOL. But even without quantifiers, theory combinations can introduce undecidability.

There’s the auxiliary problem that decidable but slow can often be equivalent to undecidable.

zero_k5• 3 years ago

For anyone who understands easier through code, I suggest:

https://github.com/msoos/minisat-v1.14

It's an early version of MiniSat by Niklas Eén and Niklas Sörensson. You can get the original ZIP from minisat.se, but it's easier to read from GitHub. Enjoy!

andrewla• 3 years ago

Increasingly it seems that P=NP is not really all that interesting of a problem from a practical perspective.

The progress made in various NP-Complete problems, SAT especially, shows that the vast majority of problem instances tend to be solvable, with the worst-case complexity problems being relatively rare but having much greater complexity cost.

kqr• 3 years ago

I keep asking this question and not getting satisfying answers so I'll try again: I'm sure I encounter problems every month that reduce to SAT (or SMT) and I could save a lot of time and energy (for both me and my computer) if I would just ask a SAT solver to find the answer instead of writing custom code each time.

However, that thought never strikes me because I haven't gotten used to thinking about problems as SAT reducible. But since the thought never strikes me, I also don't get many opportunities to get used to that!

How does one get started with incorporating SAT (or SMT) solving as a natural part of going about one's day?

ufo2• 3 years ago

Keep your eyes open for problems where you have to do some form of brute-force search. Those that doesn't have a clear solution other than trying all the possibilities.

Another thing is that the best kind of problem for SAT solvers are problems that ask a yes/no question. Is there a way to satisfy all the constraints, and if so what are the variable assignments that do so? It's also possible to use SAT solvers for other kinds of problems (e.g. find the solution with the best score) but it's more advanced.

JonChesterfield• 3 years ago

Try CSP instead. It works as as higher level abstraction over SAT that is easier to model with.

petters• 3 years ago

Right, SAT can sometimes be thought of as the assembly language of discrete optimisation.

milemi• 3 years ago

What is CSP?

Edit: looked it up, Constraint Satisfaction Problem.

The Silent (R)evolution of SAT

Related Stories