yorhel

And I'm saying you can't have fast rescans in all cases - it very much depends on the filesystem and directory structure.

I'm not trying to downplay nnn - I meant it when I said it's a cool project! I'm saying each project has its strengths and weaknesses, but your marketing doesn't reflect that (or I missed it).

ncdu's memory usage is definitely its weak point - that's not news to me - but it's because I chose the other side of the trade-off: No rescans. If you're curious where that 60MB goes to, it's 400K of 'struct dir's: https://g.blicky.net/ncdu.git/tree/src/global.h?id=d95c65b0#...

I appreciate your work, but you're not being very honest with your claims.

nnn is not keeping information about 400K files in memory in that benchmark. As a result, the rescan is necessary when changing directory. The rescan may be fast in many cases and in some cases it may even be what you'd want, but I can also name many cases where you certainly won't want it (large NFS mounts being one example).

Sorry for the pedantry. I spent a fair amount of time optimizing ncdu's memory usage, so I tend to have an opinion on this topic. :)

This looks incredibly cool, I love the speed and minimalism. It doesn't seem as configurable and feature-packed as vifm[1], so I'll probably stick with that, but it certainly fills a good niche.

nnn vs. ncdu memory usage in disk usage analyzer mode (400K files on disk):

I assume that's because nnn isn't keeping the entire directory structure in memory, which means that browsing to a subdirectory involves rescanning the entire directory. That's a fair trade-off, but an unfair comparison.

1. https://vifm.info/ - I'm surprised this hasn't been mentioned yet in this thread.

There was a similar talk[1] at FOSDEM, where the speaker describes how, as an experiment, he replaces a full ELK stack plus other monitoring tools with PostgreSQL. He even goes as far as implementing a minimal logstash equivalent (i.e. log parsing) into the database itself.

It wasn't an "we do this at scale" talk, but I'd love to see more experiments like it.

For the impatient: Skip to 17 minutes into the video, where he describes the previous architecture and what parts are replaced with Postgres.

1. https://fosdem.org/2017/schedule/event/postgresql_infrastruc...

Shameless plug: I also maintain a few small single-file C libs at https://dev.yorhel.nl/ylib

And, separately, a small-single-file-but-correct XML parser: https://dev.yorhel.nl/yxml

I'm surprised that even the fastest implementation needs 20+ms to search a 1000-element array. I would expect even a linear search to finish within 1 or 2ms with such a small data set. How large are the array elements? How were the times measured?

EDIT: Oh the time measured is the total of 1,000,000 random lookups? Nevermind my confusion then, that would certainly explain it.

Location: Netherlands

Remote: No

Willing to relocate: Within the Netherlands

Technologies: C, Perl, PostgreSQL, Apache, lots of Linux stuff

Résumé/CV: Mail me for full CV, check my OSS projects in the mean time: http://dev.yorhel.nl/

Email: contact@yorhel.nl

Recently finished my Master's degree in Embedded Systems, and still looking into the exact direction I'm heading for. Experienced (non-professionally) as (embedded) software dev, back-end web dev, and Linux sysadmin (websites/email). I have a broad interest in technology and am a quick learner, so I'm open to any offer that is relevant to the above listed technologies. :)

Note that quvi support has been removed on mpv git recently. They seem to be considering a switch to youtube-dl instead. https://github.com/mpv-player/mpv/commit/9b45b48c466765b6491...

SEEKING WORK - Remote or near Enschede, Netherlands

Available for any C (not C++, sorry) development where performance and efficiency matter. Love working with low level stuff, embedded systems, network protocols, security, and making it all work together.

My OSS projects and contact info can be found at http://dev.yorhel.nl/

Hmm? The stack buffer in yxml will never cause parsing to block, and there's no added dependencies on... anything? I don't think that buffer causes any problems even on a size-restricted microcontroller that doesn't have malloc(). As long as you can find ~512 bytes or so of free memory you can parse a lot of files.

The only situation in which that buffer would cause an error is when the application used a too small buffer, or when the document is far too deeply nested or has extremely long element/property names. Both the maximum nesting level name lengths should, IMO, be limited in the parser in order to protect against malicious documents. Most parsers have separate settings for that, yxml simplifies that by letting the application control the size of a buffer.

The stack buffer in yxml is also used to make the API a bit easier to use. With the buffer I can pass element/property names as a single zero-terminated C string to the application, without it I would have to use the same mechanism as used for attribute values and element contents, and that mechanism isn't all that easy to use. (This is the one case where I chose convenience over simplicity, but I kinda wanted the validation anyway so that wasn't really a problem)

It's new, yes. I wrote it about two months ago because I needed a parser and found the existing solutions too bloated.

It doesn't allocate or buffer, so when it says that it verifies proper nesting, I assume that doesn't mean verifying that start tags and end tags have matching names. I think that would be impossible.

It does match start tags and end tags, and in order to do so it does, indeed, need a small buffer. However, the only thing that needs to be stored in this buffer is a stack of element names, and that rarely needs much space - a simple fixed-size buffer is sufficient. It can be safely allocated on the stack if the application wants to. See the documentation for the yxml_init() for more details. The "mostly-bufferless" part of the homepage refers to the lack of a complicated input buffer and the avoidance of complex buffer management for output where possible.

It looks like you pass the parse function a character at a time. To be fast, you'd want to somehow encourage that to get inlined.

The function is quite large. You can probably get it inlined when put inside a header file and when it's only called in a single place, and you may indeed see a performance boost. I haven't measured how significant such an improvement may be.

However, if performance is the primary goal (it wasn't for yxml), a completely different design would probably be better. Yxml needs to manage an explicit state object for each input character. For each new character it has to look at the state to know how it's handled, handle it, and then update the state for the next character. I was honestly surprised to see that yxml performed quite well in my benchmarks.

A more efficient design is probably the goto-based state machine approach that, e.g. ragel[1] can output. The reason I didn't go for that approach is that it complicates buffering. You either need to have the full file mapped into memory (not possible when reading from a socket or from a compressed stream), or you need to implement your own buffering and provide hooks so that the application can fill the buffer when it is depleted. The latter approach is used by most other XML parsers.

[1] http://www.complang.org/ragel/

Agree with the article; Linda is great, both the name and the project. It was one of the more interesting topics of a Distributed Systems course that I followed, but I never ended up using it (or its concepts) in practice due to the performance overhead.

As a nitpick: Perl is originally not an acronym, the language used to be called "Pearl". https://en.wikipedia.org/wiki/Perl#Name

Start with the Autotools Mythbuster: http://www.flameeyes.eu/autotools-mythbuster/ The autoconf & automake documentation is generally quite good as well.

Please do use autotools! You only need two files: configure.ac and Makefile.am, both at the top-level of your project. The autogenerated stuff can be ignored, you don't have to learn M4 to use autoconf. And (if you are a bit careful, but that's not too hard) you'll have many nice features such as out-of-source builds, proper feature checking, amazing portability, 'make distcheck', and acceptable cross-compiling (still hard to get right, but the alternatives tend to be even harder). Don't switch to another build system purely based on the idea that it's "more elegant".

> Firstly you need a way to incrementally verify a file

Following the proposal of the article, you don't really need that. If you don't have the hash in your local cache, you should assume that the link provided by the site is correct and resume as you normally would without the hash present. When it's downloaded, you verify the downloaded contents with the hash, and add the file to the cache if the verification succeeds.

If you already had the hash in the local cache, then you already have the entire file and there's no need for incremental verification.

As I understand it, the point of the hash is purely for caching purposes, not strictly for validating that the downloaded file matches it.

Odd-sounding request: Is it possible to have an option to not use gpg, or any encryption, at all? I only store my passwords on a LUKS loopback file, which I mount/umount as I need to access some information on it. The double-encryption does not really add any better security, but does add another password to type. I'd love to make use of the nice looking pass interface: A password generator, copy-to-clipboard, and simple management operations.

Minor downside: You have to ensure that the contents of a {code ..} block have all their open-brackets matched with close-brackets. This can be quite tricky with larger code examples. The bbcode [/code] equivalent is slightly less problematic.

Other than that it looks like a good alternative.

On a related note: Be wary of the dlclose() function, it may not necessarily free any resources[1] and not every libc implements it correctly[2].

1. http://www.openwall.com/lists/musl/2012/08/20/1 2. http://www.openwall.com/lists/musl/2012/08/23/20

If the /robots.txt does not mention a Crawl-delay, one page per 3 seconds is often a safe value. Of course this rather heavily depends on the site. In any case, if you have any specific need, always contact the people responsible for the site. I occasionaly run custom queries against the database on request, for example.

> I managed to crawl [..] more than 300k movies from IMDB in just a few hours

I suppose IMDB already has a pretty good architecture to handle that load, but please, if you're crawling from a single site, be careful. I host a similar database myself, and the CPU/load graphs of my server can tell me exactly when someone has a crawler active again. That's not fun if your goal is to keep a site responsive while keeping the hosting at low cost.

That's what I thought as well, but "yorhel" has been taken on Twitter for quite a while.

Oops, just actually tried to compile from git and it just worked. I was under the impression that the qt-psql library wasn't included by default, so I went looking for a package and only found something outdated on AUR[1]. At which point I gave up, expecting that to be too old to still work. Sorry for the FUD, I feel stupid now. Keep up the good work!

1. https://aur.archlinux.org/packages.php?ID=18709

Just tried to compile from git, but the qt-psql requirement wasn't very easy to satisfy on Arch. Your Ubuntu binary worked fine.

Looks quite nice, and there certainly is a need for more PostgreSQL GUIs. Some feedback from the minute I used it:

- You don't handle bytea columns very nicely. I'd expect those to be displayed in hex or so.

- Browsing a table is very slow on tables with large columns. I was testing on a table with a ~10KiB text column.

- I personally prefer a more compact interface, having an option for smaller buttons would be a good start. :-)

(Edit: formatting)

Ah, I wasn't aware of that. Thanks!

Somewhat related question: It's fairly common for sites to have static files (images/css) served on a different (sub)domain. What are you supposed to do when the html content is being served on HTTPS? Should the static files be on HTTPS as well? If so, wouldn't it need a different certificate? Certificates are only valid for a single domain, after all.

I've had very positive experiences with statically linking against musl[1], As long as your code and the required libraries don't use dlopen() tricks (e.g. as glib GIO does), you can quite reliably create a single statically linked binary for your program. Admittedly, this isn't as easy to setup as Go is.

[1] http://www.etalabs.net/musl/

Just nitpicking: DC++ is one implementation. The network is usually referred to as "direct connect". DC++ wasn't even the first DC client, it just happened to be the most popular one at the time the network itself was still quite popular.

Also an interesting graph from the two polls: http://attractivechaos.wordpress.com/2012/03/24/two-interest...

It's hard to pinpoint exactly why I was having so many problems with deadlocks. Most of the time they could easily be solved by having channels with an unbounded buffer, thus guaranteeing asynchronous behaviour. But since the Go authors intentionally left that out, I must be doing something wrong at a more fundamental level.

You're probably right about the IRC clients using the latter style of callbacks. But then it turns pretty much into the following style:

  irc_connect();
  register_event_x(callback_x);
  register_event_y(callback_y);
  while(read())
    dispatch_event();

On the scaling on multiple cores: Sure it's a nice advantage when you're already used to concurrent programming, but for most things I've written so far it doesn't really matter. In the few cases that I had a component that required heavy disk I/O or computational resources, it was only a small and easily-identifyable component that could be implemented in a separate OS thread. I'm not familiar with Node, but both glib2 and libev allow spawning a separate event loop in each thread and provide mechanisms to communicate between each other (idle functions in glib, ev_async in libev). Those are for C, however, and I have to admit that callback-style programming isn't too convenient there due to the lack of closures and automatic garbage collection.

Rather flamey indeed, but let me take the bait. I've been using callback-style programming in many applications, ranging from IRC bots to application servers and whatnot. I've also been playing around with Go for several months and tried very hard to implement something that is inherently asynchronous. I gave up after about two months, I just couldn't get used to the concurrency model. I always end up with either data races, deadlocks and have otherwise no good overview of the program logic. This is exactly what the article describes, and seems to apply to me as well.

Incidentally, while figuring out how to implement something inherently asynchronous in Go I looked at some IRC client implementations, and noticed how all of those used callback-style programming one way or another. That doesn't inspire much confidence. :(

All in all, I am not sure yet whether this is purely due to my inexperience with concurrent programming or whether I am just inherently incompatible with it. I'd love to be able to play with alternatives to callbacks in the future, but for now it's just way over my head for the things I want to do.