michalc

Reminds me of another “slow client”-related bug in gunicorn: https://github.com/benoitc/gunicorn/issues/3334

The GOV.UK Design System summary list component is a description list https://design-system.service.gov.uk/components/summary-list...

And... it also uses the wrapper div for styling

the rest will soon follow

If you’re looking for requests ;-), I would love an ECS (and specifically Fargate) emulator that actually ran Docker containers locally as though they were in ECS

Submitted to Unicode yesterday (so please be gentle!)

I think I can understand why this wasn’t addressed for so long: in the vast majority of cases if your db is exposed on a network level to untrusted sources, then you probably have far bigger problems?

So my definition of big data was data so big it cannot be processed on a single machine in a reasonable amount of time.

I guess they’re using a different definition?

Hmmm... depends on the project / phase of the project?

I am particularly not a fan of doing unnecessary work/over engineering, e.g. see https://charemza.name/blog/posts/agile/over-engineering/not-..., but even I think that sometimes things _are_ worth it

Short answer is no, not as far as I am aware/can reason about it

In more detail: so by my understanding there are two techniques in making zip bombs…

Firstly nested ZIPs that leverage the fact that some unZIP programs recursively extract member files. stream-unzip doesn’t do this (Although you could probably use stream-unzip as a component in a vulnerable recursive ZIP parser if you really wanted to… but that I would argue is not the responsibility of stream-unzip)

The second technique is overlapping member files, but this depends on them overlapping as defined by the central directory at the end of the ZIP, which stream-unzip does not use

But if you are accepting files from an untrusted source, then you should validate the size of the uncompressed data as you unZIP (which you can do as you validate along with any other properties of the data)

Beyond that, I've grown fond of 'sticking to the defaults' over the years.

This resonates with me! Both in terms of things I use and things I make - I want them to "just work"

without regard for the maintenance burden 1, 2, 5, 10 years down the road.

To me software craftsmanship isn't just about the code, it's about engineering use of time.

In general shouldn't knowingly make choices that would result in pain in the future, but if you're increasing the chance of the project not making it to the future, then is that really the better option? Finding out enough information to make the judgement call between long term/far future pain and short term benefits is all part of the craftsmanship.

I don't blame agile. But I do kind of blame Agile™

(Loving the phrasing here! I think I'm right on board, especially if we're talking Scrum/Scum-ish)

why not remind people of the purpose?

To answer this, I suspect that trying to change what certain words/phrases mean to people en-masse is extremely difficult, to the point of impossibility in most cases. However, we each have the power to be clearer in the words we use so they are understood by the people we're communicating with.

engineering quality matters

But also, this to me suggests that there is some sort of absolute definition of quality, but it's much more nuanced. Nothing is inherently "bad quality", but instead has certain consequences, which may or may not happen or may or may not be acceptable in certain circumstances, and you might not even know what these are until the future. This I think is the point I'm trying to make - there is no absolute definition of engineering quality, and I suspect the term "technical debt" all too often suggests there is.

Have to admit the lazy thing threw me, but I can see how the “doing less” I’m arguing for could be taken that way. The “less” is not about avoiding handling edge cases that are possible now, but about avoiding putting in layers of code to handle cases possible only in some future versions of the code (with some limited exceptions that I mention at the bottom of the post)

In fact, it’s crossing my mind that people might not want to be accused of being lazy, and that is a motivation to over-engineer solutions.

You’re very welcome!

Have to admit I am curious: what’s the context / how has it helped you more specifically?

Very much agree.

It was a few years ago, and very AngularJS focused, but I posted something along these lines: https://charemza.name/blog/posts/angularjs/e2e/consider-not-...

In summary: having thing look cleaner at a glance is not helping if you’re (almost) always going to need to do more than glancing

real mastery often involves learning when to do less, not more

Really love and agree with this, and (shameless plug?) I think really aligns with a way of working I (and some colleagues) have been working on: https://delivervaluedaily.dev/

I've now added a section to https://delivervaluedaily.dev/ that is at least a starting point of trying to make sure it doesn't make bad environments worse.

This has been helpful - thank you. @RayFrankenstein if you would like a small acknowledgement at the bottom of https://delivervaluedaily.dev/, let me know

Although a follow up… could “Deliver value daily” actually help push _away_ from toxic behaviours like micromanagement?

As long as there is some sort of visible/tangible progress most days, and in a justifiable direction, there could be less of a desire for tight control for example?

I was actually thinking this is all from the developer point of view (or at the very least mine: a developer). I try to deliver some value every working day, and I have to say I pretty much love it. This principle really pushes me to make sure I understand the problems and engineer solutions to them appropriately. And from feedback I get, I am, apparently, very productive doing this. And I do encourage others to work this way (as I was myself encouraged), but I do emphasise the point made repeatedly in the manifesto: the increments can _really_ be extremely small. And to also emphasise another point in the manifesto here - it's not a concern if the aim isn't fulfilled every day. If most developers currently aim for value every 2 weeks say, and that's what they're used to, then of course every day won't always happen, certainly in the beginning of working this way. Or yes, there may be unexpected things along the way. And this is all completely fine.

(This last point was a late addition to the manifesto: but I _think_ made before your comment)

Can you give a bit more detail on the destructive aspects though? If you mean if an organisation follows this, then it ends up as a stick to beat developers with if they don't deliver value every day? I guess my counter to that is, if management is toxic, they will always find a stick to beat developers with, no matter what.

What the manifesto does, or at least tries to do, is encourage developers to think about how larger pieces of work are tackled - so they end up really understanding what the problems are, and so engineer solutions to them appropriately. Wherever possible getting feedback, and if possible on a cadence of at least daily, to me seems such a good way of doing this - but do you have a better suggestion?

Edit: I see https://github.com/rayfrankenstein/AITOW and have started to give it a read… and… wow. I certainly understand where you’re coming from, and I would hate to see “Deliver value daily” make all these situations worse.

I think I’ve been really lucky - only maybe experienced light versions of the worlds described, and in the past few years have worked in a very supportive environment and had a lot of autonomy. In this environment I/my team decide on a lot, including ways of working like how to tackle bigger pieces of work. And in such an environment, “Deliver value daily” I think can really work, and essentially has.

But I think I am sticking with my point - I suspect toxic management will find any excuse, and the three words “Deliver value daily”, will do little to change that.

Although - I am tempted to make it clearer up front that autonomy and a supportive environment (and maybe other things?) are crucial, to try to at least limit how it could be “misused”

For a while now some colleagues and I have been thinking about how to improve ways of working, and specifically around the limitations of Agile - especially how it's practiced. As has been suggested by others before us, Agile is all too often a cargo cult: surface level features - often ceremonies taken from Scrum - are prioritised over delivery and getting feedback. Our solution to this is a simplified manifesto with exactly one principle: "Deliver value daily", which we're hoping will reshape thinking, at least a little bit.

Full disclosure: I posted an earlier version here about a month ago, but have since refined it a bit, so hopefully it's appropriate to post it again.

I made something along these lines a while back too: https://projections.charemza.name/

Ah I _think_ I ran this on a MacBook that as far as I know has 8 real cores…

But I since found that 4 are “efficiency” cores, so that could be the reason for the poor scaling(?)

Ah thanks for the suggestion! Will see if I can add examples or otherwise make things clearer

Ah I did! https://news.ycombinator.com/item?id=39254487

(Feels a bit soon to repost somehow…)

I have posted this before, but I’ve been working on a wrapper for OpenTTD to turn it from a game to a (slightly!) more serious system for research/experimentation, especially using its AI system:

https://github.com/michalc/OpenTTDLab

If you use this, I owe you nothing.

Sure the legalese states I owe nothing, but if I’ve shared the code, written some documentation, and encouraged others to use it as I have on a few projects…

… I feel as though I do owe something. I feel like I’ve made a sort of social contract to provide a bit of support on what I’ve, er, “supplied”.

Not really sure why.

I was surprised to see that Python 3.6 is still the most popular version, even though it's been over 2 years since the end of security releases for it.

I didn't know about this - thanks!!

Ah the name - you're not the first to mention it! Do you (or anyone lurking...) have any suggestions as to what it might better be called?

On what it does/why it exists, we've kept the README quite light to avoid duplication, with the main bits of the docs at https://pg-bulk-ingest.docs.trade.gov.uk/

But to try to answer the question here:

A set of insert statements - there are lots of cases where this would be fine, so pg-bulk-ingest (/its future name ;-) would be unnecessary, and so you might as well use insert statements.

But there are lots of things that pg-bulk-ingest does that a set of insert statements don't:

- It uses COPY, which in many cases is (much?) faster than INSERT

- It creates the table if it doesn't exist

- It migrates an existing table to match the definition if it does exist (pg-bulk-ingest can't handle all possible changes, but enough to make it useful to us)

- And it migrates without holding a long ACCESS EXCLUSIVE lock on the table

- It offers a fairly structured way to do high-watermarked ingest - it stores and retrieves a high watermark so you can fetch the data "since" some previous ingest, with the exact meaning of "since" custom to each source you're ingesting from.

- And it (now) combines COPYs with multiple tables, without loading all incoming data into memory at once, and keeping the ingest within a single transaction, so other clients see all the data, or none of it.

(The high watermarking I think is maybe the least clear of these in the docs, while also one of the most important things. Would really like an example that is somehow standalone that also makes it clear what it does...)

I thought it was time I got off my... and starting making an AI (https://github.com/michalc/SupplyChainLabAI - but it does nothing but log a debug message repeatedly at this point, I feel I have a long journey...)

But it makes me realise there is another use case for OpenTTDLab that I don't think I considered: regression tests for OpenTTD AIs?

Edit: And it has been mentioned to me separately that maybe it could be used as part of regression tests of OpenTTD itself https://github.com/OpenTTD/OpenTTD/discussions/11863#discuss...

Short answer: yes, exactly

Long answer: trying to make it much easier to do simulation-based research using OpenTTD, and for any results found to be as reproducible as possible. There have been a few papers/dissertations that use OpenTTD, but it's tricky to do exactly what the authors did, or even in a few cases I suspect it was hard for them to repeat their own experiments or extract results from them. OpenTTDLab seeks to make these easier.

And, such research seems to often (/always?) use AIs, so that's the focus.

I also would like to be able to train an AI in some automated(/formal?) way, and maybe use OpenTTD to investigate something a bit more... supply-chain-y, but that's a work in progress (I've not even written an AI yet!)

But also a bit of a less formal "just run an AI/some AIs over some seeds and see" is also very much a use case. Am trying to strike that balance between something quite... exploratory, but still reproducible.