robertelder

This is my favorite observation.

This is my favourite comment.

This is my favourite comment.

That's a very impressive story! I was hoping this post would get a bit more attention so I could read of more tales like this, but I guess this will have to do.

I assume that a robot capable of handling 400lb must have a pretty beefy power supply. You can probably get an intuitive sense of how much damage a robot can do to you by imagining what would happen if all of the power output was directed directly at your body (in the form of heat/electrical/kinetic). A tiny 5v motor might on a breadboard might be enough warm your hand up or fling a small pebble toward you, but a motor that requires one of those bigger 400v 3-phase 60A supplies can probably pull enough energy to melt iron or generate the same kinetic energy as some military ordnances.

The title of this post stood out to me because it's a fairly niche topic that I've been meaning to write a 'rant' post about myself. I think in a very 'pure' parsing theory sense, there is no real advantage to separating the lexer and the parser. I claim that any counter-argument is due to the limitations of individual software tools, and their inability to let up specify the grammar that we actually want.

If I recall correctly, the original historical reason for separating them was due to memory limitations on early mainframe computers. Actual source code would be fed into the 'lexer' and a stream of 'tokens' would come out the other end and then be saved to a file, which would then be fed into the next stage.

Having said this, you can ask "In practice, is there currently an advantage to separating the lexer and the parser with the tools we use now?" The answer is 'yes', but I claim that this is just a limitation of the tools we have available to us today. Tokens are usually described using a simple regular expression, whereas the parsing rules are 'context free', so worst-case complexity of parsing the two is not the same. If you pull tokens into the parser and just parse them as naive 'single-character parser items', then you end up doing more work than you would have otherwise since your parser is going to try all kinds of unlikely and impossible chopped up token combinations. The other big issue is memory savings. Turning every token into a (small) parse tree is going to increase memory usage 10-20x (depending on the length of your tokens).

Personally, I think there is a substantial need for more advanced ways of describing language grammars. The ideal model would be to have various kinds of annotations or modifiers that you could attach to grammar rules, and then the parser generator would do all sorts optimizations based on the constraints that these annotations imply. Technically, we already do this. It's called putting the lexer in one file, and the parser in another file. There is no good reason why we can't specify both using a more unified grammar model and let the parser generator figure out what to do about each rule to be as efficient as possible.

The computer science theory behind grammars seems to have peaked in the 1980s, and there doesn't seem to have been many new innovations that have made it into daily programming life since then.

I can't wait to watch the FitMC video on this.

I spent a fair amount of time working on building my own from-scratch C compiler that I no longer work on anymore:

https://recc.robertelder.org/

One of the reasons that I stopped working on it was because of how slow it became, so I might be able to contribute to answering your question.

Initially, when the compiler was simpler, it was actually much faster. I was able to do some meaningful proof of concept demos with it like compiling a small microkernel, and compiling most of its own source code. Of course, the natural thing to do is to make it so that could cross-compile itself and run in the browser, and that's where it became terribly slow, which required more code to optimize, and the new code that was added to make it faster in the long term made it much slower in the short term.

To start with, if you think of a simple piece of code like this:

    if ( 1 ) { putc('a'); }

So, now we already have a memory overhead that is more than 5 times as big as the program, but we also have to build the parse tree, control flow graphs, linker objects etc. and you also have to pull in a mess of header files, bloated libraries etc. If you're wasteful with memory in the compiler, you can easily run out of memory from compiling a few megabytes of source code. Being more intelligent with memory management requires copying memory around a lot, which also adds to the latency.

So, now you need to think about optimizing your memory use, and do 'smarter' things that trade memory usage for CPU. Plus, you're likely to also start needing free/delete a lot from heap memory which is a system call and therefore slower than a call within your program. By the time you implement all this 'optimization', you compiler has become an incredibly complicated and bloated system that requires even more code to optimize all the opportunities for improvement.

Since we're on the topic of snap updates:

A couple weeks ago I was working away in the terminal when all of a sudden, my USB camera turned on and its light started flashing at me indicating something had just started interacting with my webcam. I immediately assumed "Oh, that's probably just some hackers watching me through my web-cam.", so I looked through /var/log a bit and noticed that it had just re-detected all USB devices and two new users had just been added to my system:

    snapd-range-12345-root:x:12345:12345::/nonexistent:/usr/bin/false

    snap_daemon:x:12345:12345::/nonexistent:/usr/bin/false

Interesting, thank you for that analysis. From what I understand, the RCE exploit really needs two things to work: 1) The interpretation of the JNDI reference by log4j, and 2) The 'auto-execute loaded classes' (which I don't quite understand).

Is there any kind of low-level flag you can pass to Java or your environment to completely disable JNDI? I recall that there is a flag you can pass to log4j, but I can't see any reason why I would ever use JNDI anywhere in Java.

Also, do you have any additional insights on how exactly the mechanism for 2) works? From what I understand, this is a feature of Java itself?

I've been thinking about this since I saw it here on HN yesterday, and I can't help but entertain the idea that this might end up being 'the worst software security flaw ever'.

Neat. I wrote a tool that shows interactive examples of how you can actually do the calculations for multiplications with Fourier transforms:

https://blog.robertelder.org/fast-multiplication-using-fouri...

The trick is really just about realizing that you can re-write a number like 1234 as a polynomial like 1 * 10^3 + 2 * 10^2 + 3 * 10^1 + 4 * 10^0.

That's interesting that it was exactly 7kg. Maybe they read this document first:

(§ 40.22 Small quantities of source material.) https://www.nrc.gov/reading-rm/doc-collections/cfr/part040/p...

"(a) A general license is hereby issued authorizing commercial and industrial firms; research, educational, and medical institutions; and Federal, State, and local government agencies to receive, possess, use, and transfer uranium and thorium, in their natural isotopic concentrations and in the form of depleted uranium, for research, development, educational, commercial, or operational purposes in the following forms and quantities:

... (2) No more than a total of 7 kg (15.4 lb) of uranium and thorium at any one time...

"

+1 For Brother laser printers. I've had mine for a few years now, and only gone through 1 cartridge of toner. It works with my Linux setup too.

It may very well be the case that users more often click a link to unsubscribe directly on the site more than they click the 'unsubscribe' feature in the email client (I don't have any stats on that).

The original question remains though: How does the stated service here work when there is only a List-Unsubscribe: header and no web form?

I just checked a few newsletter emails in my inbox, and they all offer List-Unsubscribe: mailto: based unsubscription. From what I see, it is quite universal and more common than link-based unsubscription. Newsletter providers prefer not to use the url based unsubscription because some email providers will crawl the URLs in emails and automatically unsubscribe people without them knowing.

It's also possible that you're talking about a 'link' (a href) in the email body that goes to the newsletter provider's site and ask them a bunch of questions (why are you unsubscribing etc.). That's a different concept entirely.

From what I've seen, the email-based way is the primary unsubscription method in use today. Even when you use the gmail feature "Mark this as spam", depending on what you click, it will just send an email on your behalf to the to whatever is specified in the List-Unsubscribe: mailto: header.

How does this work from a technical perspective? If I forward mail to this service instead of clicking the 'unsubscribe' link, then how do they 'unsubscribe' me on my behalf? In order to complete the unsubscription process, the newsletter owner needs to receive an email (usually generated automatically by the mail client when clicking the 'unsubscribe link') that comes from the person being unsubscribed. If I outsource this process, this means that 'please-unsubscribe.com' will need to spoof emails from me on my behalf and send them the email specified in the List-Unsubscribe email header. If spoofed emails are not used and they come from 'please-unsubscribe.com' this won't be very actionable to the newsletter owner. From their perspective, they're now going to get 'unsubscribe' notifications that all come from an email at 'please-unsubscribe.com' which never signed up to the newsletter in the first place.

This all assumes that the newsletter unsubscription process is using the email based method instead of the link based method (which is a good assumption email-based is more common).

hah, because you care so much, I'll go ahead and make that change. It's probably better for SEO anyway. I made the change on my local copy. It will get pushed next time I re-build the site.

As the author of this blog post, I find this statement interesting. Can you expand upon the sentiments expressed in your statement? Also, by 'read this', do you mean just the blog post, or contents of the works by Jim Roskind?

Agreed on the point that there are better mental models out there. The difficulty is to make content that is simultaneously correct, but also not over-complicated. I finished off the post with a reference to Russ Cox's work so any sufficiently motivated individual will find the right models there.

You found a bug. The bug was: When I was computing the animation frames for the visual, after a backtracking event, I forgot to restore the 'progress' values for the PROGRESS nodes to what they were before the stack push took place. I think I fixed this issue, and I just pushed an update which should appear any second when the cache invalidates.

The goal with this article was never to show the best or most efficient way to match regular expressions, but rather to convey the correct interpretation of what a given regular expression will do when you try to use it. Even discussions of slow and simple backtracking regular expression matchers are a pretty niche topic and likely too much for casual users of regular expressions.

Yes, I was thinking about also allowing it to optionally cross-compile to the more asymptotically efficient matching algorithm, also documented by Russ Cox. Everything takes time though, and I figured it would make to put this out there before investing too much time in something that possibly no one would use.

It's worth pointing out that the resource

https://swtch.com/~rsc/regexp/regexp1.html

linked above is also by the same author, Russ Cox,

as the reference I included at the end of the blog post:

https://swtch.com/~rsc/regexp/regexp2.html

All of the insights that I presented through this visualization tool are based upon the knowledge found in Russ Cox's articles. Also, the link above on RE2:

https://github.com/google/re2

is a project that was was also (started by|heavily contributed to by) Russ Cox. His writings on the topic of regular expressions are absolutely world-class and I have never found any better resource for understanding the deep low-level theory on how they work in practice.

The (?:stuff) part is for a non-capturing group which I didn't add support for. Maybe in the near future. That would be one of the easiest features to add since the tool also doesn't support backreferences yet.

Edit: Actually, since the non-capturing group part doesn't add much here, you can just remove the '?:'s and this works:

https://blog.robertelder.org/regular-expression-visualizer/?...

I tried to follow a middle-of-the-road works mostly everywhere made up flavour. Here is a grammar for it:

https://blog.robertelder.org/regular-expression-parser-gramm...

I may add to it in the future if there is enough interest. At the moment it doesn't support a few common things like word breaks and backreferences, but I don't think it'd be too hard to add.

Hi, I'm the author of this page. This tool has a lot of complexity in it and a huge number of corner-cases. If you discover any bugs or XSS vulnerabilities, feel free to let me know.

I'll provide another voice to support the idea that d3 is a really great library. For anyone who has tried to learn it in the past and found it a bit too complicated, I'd suggest you approach it with the following perspective: Learning d3 is more about learning SVG than it is about learning d3 (unless of course you already know SVG).

I did something similar recently using my custom terminal diff tool. Here is a more colourful terminal-based view of the differences between MN908947.3 and sars NC_004718.3:

https://twitter.com/RobertElderSoft/status/12226906294923796...

To produce that image I used this tool:

https://github.com/RobertElderSoftware/roberteldersoftwaredi... With these two genomes:

https://www.ncbi.nlm.nih.gov/nuccore/MN908947

https://www.ncbi.nlm.nih.gov/nuccore/NC_004718.3

with the leading numbers, spaces and newlines removed from each genome file.

My tool (as well as unix diff with --minimal) uses the myers diff algorithm which just compute a mathematically minimal diff. More advanced algorithms exist for computing phylogenetic trees that take into account the biological likelihood of certain sequence change (deletions vs. additions vs. translations etc.).

"Do not rely on security.txt as authorization to test a site!", I was actually thinking of putting up a security.txt recently to do exactly this and explicitly define the terms upon which it would be ok to have people probe my site without retribution. Arguably, this might actually be the most valuable thing to include in a security.txt. I can't really see any down side (can anyone think of any?) as this would help attract attention from the most noble white hats who are likely to help you and are eager to sharpen their skills. Black hats won't care what you put in it. You'd probably want a lawyer to help you flesh out decent terms, but it would be a great playground to have a list of sites that invite people to pen test them under clear terms for mutual benefit.

Over the years I've found that I usually fall into a pattern of starting with low-fidelity automation in languages like shell and slowly re-writing it over time into more higher-level languages, usually python first, then Java. This way, unimportant tasks can be automated in less than 5 min with one of these shell commands. If it breaks or has errors, no big deal. Python works well for figuring out the structure of the solution as an actual program, and then finally a language with static type checking when it really needs to run without errors.

'expect' is definitely for edge-case automation, but in the few cases where it is useful, it is extremely useful. Despite the fact that it's quite 'old fashioned', it is a dependency of DejaGnu which, itself, is a dependency of the test cases for a huge number of GNU projects. This leads me to believe that it'll still be around for a very long time and will continue to work the same as it does now.