emeryberger

Since we are talking about Python type-checkers: we've built a (non-AI based) type assistant for Python called RightTyper (https://github.com/RightTyper/RightTyper). Below is a brief description; a technical paper describing RightTyper is here: https://arxiv.org/abs/2507.16051, "Getting Python Types Right with RightTyper"

RightTyper is a Python tool that automatically generates type annotations for your code. It monitors your program as it runs and records the types of function arguments, return values, local variables, and class fields — with only about 25% runtime overhead. This makes it easy to integrate into your existing tests and development workflow, and lets a type checker like mypy catch type mismatches in your code.

RightTyper is much better in addition to running orders of magnitude faster.

https://github.com/RightTyper/RightTyper

(full disclosure, I am one of its authors).

You can adjust the code on the page easily (it’s open source javascript) to determine the question you are after, which is a valid one: if an adversary starts today and adds x nodes to the existing network, what is their success rate?

BUT the author asked a different (but valid) question: assuming the adversary controls x out of N existing nodes, what is the success rate? I am unclear: is the assertion that everyone’s relay is honest today? From a privacy standpoint, that’s not a great assumption.

  The Tor Project web site makes a bold claim to its users:

  "Tor Browser prevents someone watching your connection from knowing what websites you visit. All anyone monitoring your browsing habits can see is that you're using Tor."

  Don't misinterpret this claim. It's not true that Tor protects you against "anyone monitoring your browsing".

  On this web page, I provide you with the critical information missing from the Tor Project's website: if you estimate your adversary's resources in dollars, I'll estimated the probability that Tor will fail to protect you.

Yes. It lets the LLM immediately obtain precise information rather than having to reason across the entire source code of the code base (which ChatDBG also enables). For example (from the paper, Section 4.6):

  The second command, `definition`, prints the location and source
  code for the definition corresponding to the first occurrence of a symbol on a
  given line of code. For example, `definition polymorph.c:118` target prints the
  location and source for the declaration of target corresponding to its use on
  that line. The definition implementation
  leverages the `clangd` language server, which supports source code queries via
  JSON-RPC and Microsoft’s Language Server Protocol.

You should check out ChatDBG project - which AFAICT goes much further than this work, though in a different direction, and which, among other things, lets the LLM drive the debugging process - has been out since early 2023. We initially did a WinDBG integration but have since focused on lldb/gdb and pdb (the Python debugger), especially for Python notebooks. In particular, for native code, it integrates a language server to let the LLM easily find declarations and references to variables, for example. We spent considerable time developing an API that enabled the LLM to make the best use of the debugger’s capabilities. (It also is not limited to post mortem debugging). ChatDBG’s been out since 2023, though it has of course evolved since that time. Code is here [1] with some videos; it’s been downloaded north of 80K times to date. Our technical paper [2] will be presented at FSE (top software engineering conference) in June. Our evaluation shows that ChatDBG is on its own able to resolve many issues, and that with some slight nudging from humans, it is even more effective.

[1] https://github.com/plasma-umass/ChatDBG (north of 75K downloads to date) [2] https://arxiv.org/abs/2403.16354

Nice UI. We started on a project that does this about 2 years ago. ChatDBG (https://github.com/plasma-umass/ChatDBG), downloaded about 70K times to date. It integrates into debuggers like `lldb`, `gdb`, and `pdb` (the Python debugger). For C/C++, it also leverages a language server, which makes a huge difference. You can also chat with it. We wrote a paper about it, should be published shortly in a major conference near you (https://arxiv.org/abs/2403.16354). One of the coolest things we found is that the LLM can leverage real-world knowledge to diagnose errors; for example, it successfully debugged a problem where the number of bootstrap samples was too low.

Correction: it’s been a feature for at least two years. To our knowledge, Scalene was the first profiler to incorporate LLMs.

  One major limitation the LLM has is that it can't run a profiler on the code, 
  but we can. (This would be a fun thing to do in the future - feed the output
  of perf to the LLM and say 'now optimize').

Debuggers are great, and even better with LLMs! https://github.com/plasma-umass/ChatDBG https://arxiv.org/abs/2403.16354

However, the resulting type annotations should work for Python 3.9+.

It relies on some features introduced in Python 3.12, specifically the `sys.monitoring` API.

Can’t honestly remember the details! But it’s very much an AWS-owned org.

In addition to being a faculty member at UMass Amherst, I am an Amazon Scholar, working at Amazon Web Services, where this work was conducted.

I agree that writing type hints can be painful, especially if you are starting with a large code base that is mostly untyped. You might consider using RightTyper (https://github.com/RightTyper/RightTyper) - basically run your Python 3.12+ program with it, and it will add type hints to your code. It’s fast, basically automatic, and by design RightTyper avoids overfitting to your types, letting a type checker like MyPy surface edge cases. In effect, the type checker becomes an anomaly detector (Full disclosure, I am one of the authors of RightTyper.)

From the GitHub page:

RightTyper is a Python tool that generates types for your function arguments and return values. RightTyper lets your code run at nearly full speed with almost no memory overhead. As a result, you won't experience slow downs in your code or large memory consumption while using it, allowing you to integrate it with your standard tests and development process. By virtue of its design, and in a significant departure from previous approaches, RightTyper only captures the most commonly used types, letting a type checker like mypy detect possibly incorrect type mismatches in your code.

Now posted: https://www.youtube.com/watch?v=_VF3pISRYRc&t=4862s

In his talk at REBASE, Fil said the overhead is now 1.5x to 5x, and he is just getting started optimizing.

From Fil Pizlo (http://www.filpizlo.com/), major contributor to WebKit's JavaScript engine, among other things:

Fil-C is already powerful enough to run a memory-safe curl and a memory-safe OpenSSH (both client and server) on top of a memory-safe OpenSSL, memory-safe zlib, memory-safe pcre (which required no changes), memory-safe CPython (which required some changes and even found a bug), memory-safe SQLite, memory-safe libcxx and libcxxabi, and memory-safe musl (Fil-C's current libc).

Scalene author here - please file an issue!

(Scalene author here) Nope, but WSL2 (Windows Subsystem for Linux) is, and Scalene works great with it.

Those papers say "language" over and over again, in the titles, in the body of the text. That work confounds languages and their implementations, and make it sound like there is a one-to-one connection between the two (of course, there is not necessarily such a correspondence).

With respect to Ruby vs. JRuby: my student just checked and verified that some but not all of the benchmarks are implemented differently (k-nucleotide, mandelbrot, pidigits, spectral-norm).

We only pin to one core for one experiment described in Section 4.3. All the remaining experiments are run with full access to all cores.

Hi, co-author here! We're happy to answer questions. See some discussion here:

* Mastodon: https://mastodon.social/@ltratt/113282264909342842 * Lobsters: https://lobste.rs/s/y12hdo/it_s_not_easy_being_green_on_ener...

Would love to hear a technical comparison between this and King et al.'s classic paper on Time-Traveling VMs from USENIX ATC 2006: "Debugging operating systems with time-traveling virtual machines" (https://www.usenix.org/legacy/events/usenix05/tech/general/k..., 505 citations).

Here's the relevant quote that speaks to your statement:

  Mossad thus defies the conventional wisdom that defeating plagiarism detection
  is difficult or requires significant programming ability. The techniques that
  underlie Mossad could be implemented manually, relying on only the most basic
  understanding of programming language principles, letting them evade detection
  by both plagiarism detectors and some degree of manual inspection...

If you read the paper, you'll see that the attack is entirely feasible to implement by hand (we did this ourselves but do not report on it in the paper). It's a pretty mechanical process. A bit of trial and error will get the job done; it's a hell of a lot easier than most assignments.

FWIW the attack we describe in the paper works against MOSS, too (that was the original inspiration for the name, “Mossad”).

JPlag, like similar plagiarism detectors, is vulnerable to attack. We outline the attack in this paper and show its effectiveness against JPlag and another widely used plagiarism detector, Moss. Note that this was written in 2020, in the pre “CheatGPT” era!

https://arxiv.org/abs/2010.01700

Mossad: Defeating Software Plagiarism Detection

Breanna Devore-McDonald, Emery D. Berger

Automatic software plagiarism detection tools are widely used in educational settings to ensure that submitted work was not copied. These tools have grown in use together with the rise in enrollments in computer science programs and the widespread availability of code on-line. Educators rely on the robustness of plagiarism detection tools; the working assumption is that the effort required to evade detection is as high as that required to actually do the assigned work.

This paper shows this is not the case. It presents an entirely automatic program transformation approach, Mossad, that defeats popular software plagiarism detection tools. Mossad comprises a framework that couples techniques inspired by genetic programming with domain-specific knowledge to effectively undermine plagiarism detectors. Mossad is effective at defeating four plagiarism detectors, including Moss and JPlag. Mossad is both fast and effective: it can, in minutes, generate modified versions of programs that are likely to escape detection. More insidiously, because of its non-deterministic approach, Mossad can, from a single program, generate dozens of variants, which are classified as no more suspicious than legitimate assignments. A detailed study of Mossad across a corpus of real student assignments demonstrates its efficacy at evading detection. A user study shows that graduate student assistants consistently rate Mossad-generated code as just as readable as authentic student code. This work motivates the need for both research on more robust plagiarism detection tools and greater integration of naturally plagiarism-resistant methodologies like code review into computer science education.

Mesh's fragmentation protection is probabilistic (but highly effective) and as fragmentation grows, the likelihood of it being able to perform compaction also grows. Intuitively, if you have lots of sparsely allocated pages, Mesh is almost certainly going to be able to find many pairs of pages that it can "mesh" together into a single physical page. If the pages are too dense, then Mesh becomes ineffective, but that's exactly the point where there's little opportunity to squeeze out memory savings and the pages aren't really fragmented.

In the steady state, I would not expect this to happen. Mesh prefers to allocate from nearly full virtual pages (see Section 3.1). This gives emptier pages time to get emptier; once they become completely empty, their virtual addresses can be reclaimed. In short, Mesh maintains roughly the same TLB footprint (note: we should otherwise have observed a performance decrease, which we did not), while significantly reducing physical memory consumption. It's also worth noting that meshing is only triggered when memory is fragmented and the heap is sparsely occupied, so it really lowers the peaks of physical memory consumption exactly at the moment when meshing helps the most, and at the same time, has no impact on the TLB (except of course updating entries).