UglyToad

Yes this is generally the fallback approach if finding the objects via the index (xref) fails. It is slightly slower but it's a one time cost, though I imagine it was a lot slower back when PDFs were first used on the machines of the time.

If you don't have a known set of PDF producers this is really the only way to safely consume PDF content. Type 3 fonts alone make pulling text content out unreliable or impossible, before even getting to PDFs containing images of scans.

I expect the current LLMs significantly improve upon the previous ways of doing this, e.g. Tesseract, when given an image input? Is there any test you're aware of for model capabilities when it comes to ingesting PDFs?

You're right, this was a fairly common failure state seen in the sample set. The previous reference or one in the reference chain would point to offset of 0 or outside the bounds of the file, or just be plain wrong.

What prompted this post was trying to rewrite the initial parse logic for my project PdfPig[0]. I had originally ported the Java PDFBox code but felt like it should be 'simple' to rewrite more performantly. The new logic falls back to a brute-force scan of the entire file if a single xref table or stream is missed and just relies on those offsets in the recovery path.

However it is considerably slower than the code before it and it's hard to have confidence in the changes. I'm currently running through a 10,000 file test-set trying to identify edge-cases.

[0] https://github.com/UglyToad/PdfPig/pull/1102

Yes, you're right there are Linearized PDFs which are organized to enable parsing and display of the first page(s) without having to download the full file. I skipped those from the summary for now because they have a whole chunk of an appendix to themselves.

FWIW they are acting, these things just take a while, current phase of gathering comments ends December 2nd https://www.fdic.gov/news/press-releases/2024/fdic-proposes-...

The point, which seems to be routinely massively downvoted on here, is that both things can be true at once:

- these drugs are good and a paradigm shift in the treatment of obesity (and have other benefits)

- we must not lose sight of the need to address a thoroughly sick food industry that necessitate so many people needing to use these. Junk food advertising, lack of subsidies for fresh vegetables, HFCS, food deserts, etc.

Chile is experimenting with banning junk food ads to children and is seeing some early behaviour changes.

The point which people seem to be wilfully missing is that we can have both these drugs and advocate for cracking down on a food system that deliberately poisons everyone in society. Having everyone be on this drug because we shrug and say "free market innit" while big corps continue to feed us crap is not a solution, obviously.

The idea with the named delegate would be if you need some way to:

    delegate Task<string> GetUserEmail(int userId);

    Func<int, Task<string>> getUserEmail

Edit: I completely forgot the main reason which is if using a DI container it can inject the named delegate for you correctly in the constructor. Versus only being able to register a single func shape per container.

My aim is to use dependency injection to inject the minimal dependency and nothing more. Versus the grab bag every interface in a medium-complexity C# project eventually devolves into.

I've had this on my blogpost-to-write backlog for a year at this point but in every project I've worked on an interface eventually becomes a holding zone for related but disparate concepts. And so injecting the whole interface it becomes unclear what the dependency actually is.

E.g. you have some service that does data access for users, then someone adds some Salesforce stuff, or a notification call or whatever. Now any class consuming that service could be doing a bunch of different things.

The idea is basically single method interfaces without the overhead of writing the interface. Just being able to pass around free functions but with the superior DevX most C# tools offer.

I guess I want a more functional C# without having to learn F# which I've tried a few times and bounced off.

I've been experimenting with this, it makes testing trivial and removes the coupling that inevitably occurs with multi method interfaces.

However I think there's one missing enhancement that would turn it from esoteric and difficult to reason about to actually usable that the language will never get.

This is being able to indicate a method implements a delegate so that compilation errors and finding references work much more easily.

E.g. suppose you have:

    delegate Task<string> GetEntityName(int id)

    public async Task<string> MyEntityNameImpl(int id)

    public async Task<string> MyEntityNameImpl(int id) : GetEntityName

Having built recurring stuff in the past (date based with no time component, luckily for me) I think you gain a lot of usability gains for generating a row for each occurrence of the event.

Inevitably the user will come back and say "oh, I want it monthly except this specific instance" or if it's a time based event "this specific one should be half an hour later". You could just store the exceptions to the rule as their own data-structure but then you need to correlate the exception to the scheduler 'tick' and if they can edit the schedule, well, you're S.O.O.L either way but I think having concrete occurrences is potentially easier to recover from.

But the problem is the accounting jargon is counter (contra?) to the layman's gut understanding.

If I get credited or I use a credit card money came from nowhere, woohoo. If I have a debit well that sounds like debt and my money decreased, boo.

I get that actually there's a good reason for the names but a field that doggedly sticks to non intuitive jargon that runs counter to every usage yet encountered for outsiders could do with some different non-overloaded terms.

Sounds sort of like the Citroen Ami? Slightly below your ideal range and quite expensive too but the same concept.

For sure as others have mentioned review it yourself in the web interface of whichever PR tool you use. (Generally after having a break first)

This is something of a superpower for catching things before review and in my experience makes the actual third party review pointless 95% of the time.

As the sibling comment says (max reply depth reached) the sub-postmasters in general were just about getting by. Part of why this scandal is so egregious is that these people were often just-about-managing under terms of an incredibly unfair contract with the PO.

The amounts of money involved may be small to business owners in other domains but for SPMs many were almost bankrupted by trying to replace the sums out of their own earnings as you say. This wouldn't have been a rounding error to their lifestyle, it should have been provable to any halfway decent investigator. And if not? Then they get away with it and it's the price we pay not to live in tyranny.

I think any (financial crime) case built solely on computer evidence is too weak to be prosecuted, even if that means you end up accepting some non-zero amount of financial crime. "it is better a hundred guilty persons should escape than one innocent person should suffer" as someone once said.

In the case of the sub-postmasters the Post Office, as far as I'm aware, never proved where these stolen sums supposedly went. The computer evidence was thought terminating and was the only thing (except false confessions under duress) used to secure these convictions, rather than proper investigative work.

The other complication was that a change in the law was made to assume computer systems were correct by default due to the problems with the existing law around breathalysers and speed cameras:

"In 1997 the Law Commission published a paper which went into some detail about the use of mechanical and computer evidence in court. It seemed a little too fixated with the effective workings of speedometers, traffic lights and breathalysing devices called ‘Intoximeters.’ It concluded that the present law is ‘unsatisfactory’ because of the necessity for prosecutors to ‘prove that the computer is reliable.’"[0]

The amended law changed the burden of proof from the prosecution proving the system functioned correctly to the defence proving it didn't, without access to the systems being used to prosecute them.

[0]: The Great Post Office Scandal, Nick Wallis

Can wholeheartedly add my recommendation for the book. In fact I think it should be required reading for anyone in the business of software. While I felt it was a bit over-long it is incredibly thorough and well told and covers multiple different aspects of the scandal while making the human cost obvious in a way that aggregate facts and figures don't.

Agree with this, if one is thinking in terms of typing letters out you're working at entirely the wrong level of abstraction and with the wrong primitives.

I often use the example of ".toString" being meaningful whereas ".tosling" is nonsense (unless you have defined it elsewhere). I'm not even sure if that's the correct capitalization of the to string method in JS, but I shouldn't have to care. If my tools can't tell me which one is valid to use in the current context, then the tools are bad and I should look for better tools. (This may not be universally applicable and maybe Lisp is the promised land etc etc. But raw text input speed is a silly metric to prioritize and leads to a local maxima in programming)

Presumably that's only when using the Windows platform identifiers? This is news to me, thanks for the correction.

It amazes me that these mega wealthy corporations don't understand uh... going abroad.

Like I know it's not so common on the US but here if you pop across to France for a week you get ads for everything in French. Despite my best* efforts I still speak abysmal French. You're not going to sell me shit in French (or indeed anything anyway).

I literally just crossed a border, I didn't become French, please.

This despite me being pretty lazy and generally letting my data get slurped left, right and center.

Perfect market efficiency strikes again...

*Citation needed

I can try but will doubtless miss something.

.NET Framework was the .NET we all knew and tolerated. It was on version 4.6/7 or so. Windows specific, requires a full install of the framework on the machine to run. Support for Winforms, WPF, WCF, Webforms, ASP.NET MVC/API etc.

Microsoft wants to make a play for the cloud. Windows Server is expensive and no one likes it. Luckily for Azure OS is irrelevant, whichever server Microsoft still gets paid.

Step forward .NET Core. This takes the .NET runtime and makes it cross-platform (and open-source). Only really supports ASP.NET in the beginning. There are some libraries/APIs that are available in both .NET Framework and .NET Core runtimes. This common interface is .NET Standard, you can basically ignore this now I think but it means libraries can be used from both.

.NET Core goes through multiple versions, 1, 2 and 3. More things are ported to .NET Core but Winforms, WPF and Webforms are left behind. Meanwhile Core becomes a better and better platform for web servers.

.NET Core drops the 'Core' naming, becomes .NET starting at .NET 5. .NET Framework is legacy but gets a bump to .NET 4.8 or so. So basically unless you're doing Winforms/WPF/Webforms you can be on .NET Core/.NET 5+ with minimal issues. Hopefully that clears something up.

I agree with you.

This is why I think the idea of technocracy or "evidence based politics" is ultimately a mirage. Sure you can maybe assess some policy but the metrics you're choosing to measure or optimise for are political by their very nature. One's evidence based policy isn't the same as mine.

Health-outcomes-wise it would be better to force everyone to eat salad or whatever but that's only one dimension to optimise on at the expense of freedom and life enjoyment.

Tying it back to tech maybe going down market improves your conversion and lowers your CAC but maybe you've just acquired a bunch of customers with low value, high churn and high costs.

Maybe the sales of Amazon Prime are showing gangbusters returns with the dark patterns but now people loathe your brand and are hoping to see you hit with an FTC banhammer.

There's no silver bullet to this stuff, sure you should probably measure it but ultimately you have to make a decision and be guided by gut instinct and beliefs.

I think while the name things correctly principle is nice and good to aspire to like anything at the intersection of people doing the right thing and natural language there's a bunch of places it doesn't work.

I'm usually reading code for 2 purposes. First, to understand the code and the domain. If this is the case I want to see how the code handles paid weekly versus monthly. Is it possible it's not accounting for fortnightly (biweekly) payments? If we were to account for that how would it slot in. How does `is_end_of_month` or week account for federal holidays, etc?

Secondly, to fix it. Something has gone wrong or some assumption has become outdated. The call is coming from inside the house. Nothing is to be trusted. This is most of my work to be honest, spelunking into code and trying to solve the mystery in which case I'm not going to take the current name-as-written to be factual and need to include the function call in my logical model of the code under investigation.

That's not to say I disagree entirely. Like all things in software it's a balance and you don't want to define getting a lighter color everywhere you need to use it, eventually you want to have a function that just wraps it. But over-eager splitting out of functions also incurs a working memory cost and it's striking the balance between the stack depth and memory overhead of the single function. Since the article's working memory model incurs no cost per stack frame I don't think it's well designed to encourage careful thought on this trade-off.

It feels like their calculation missed 2 crucial confounders, stack depth and type tracking.

1) Stack depth. In their refactored example if I truly want to understand the code I need to now store everything I was thinking about in the current function and jump to their new `is_paid_today` function, then into each of the 3 dependent functions, I'd say as finger in the air type estimates each stack frame should add 2 * depth and then some multiplier for breadth or something. This is why I prefer big functions like the initial state read from top to bottom with no indirection, abstraction or surprises where reasonable. (curly braces wouldn't go amiss to give visual indicators of scope...) This is completely counter to 'clean code' dogma but clean code dogma is... bad.

2) Type tracking. To reignite this holy war if I have to track the type information mentally then I'd say every parameter and variable has an additional +1 working memory overhead. What is employee, what is employee_database, what is `is_end_of_month` and where on earth is it coming from? Try as I might I just cannot understand people who onboard to big codebases without type hints at least, y'all are a different breed, you should work as like super-rememberers.

The real 4D chess move is fake negative reviews of your own company that are so unreasonable in content they cast doubt on other negative reviews.

This seems like an unsolvable problem with online reviews in general. However I generally find Google Maps reviews for restaurants fairly reliable. You'd think with the amount of data they slurp up they could cut out fake reviews based on location data alone. Maybe it's just confirmation bias on my end though.

There's probably newer AI powered tools but Tabula is the main library I know of https://github.com/tabulapdf/tabula-java

There is some approach you can use sort of like this within PNGs themselves I think. It's been a while so I might be misrembering but effectively each "row" of data that is compressed can be encoded as the difference from the preceding row using 4 or 5 different operations.

You can achieve better compression by brute forcing the possible operations used to encode the rows to find the "most compressible" output. Not quite what you meant but sort of similar in that you try multiple approaches and pick the best.

I gave up before implementing it but in the stub I left this comment to myself " A heuristic approach is to use adaptive filtering as follows: independently for each row, apply all five filters and select the filter that produces the smallest sum of absolute values per row.".

In addition more similar to your approach PDFs support many compression filters for objects internally like RLE and ZIP so you can choose the best algorithm per object but generally it's quicker just to ZIP everything.

I think I have realised after ruining a previous codebase with overeager abstraction. Most code should read like fiction, not a reference book.

That is they should read top to bottom with minimal jumping around and information hiding.

(Of course not in all cases, YMMV, etc)

Yeah every iteration of this question in the GP thread is bad and has the same ambiguity that means the gotcha of "haha your interpretation of this poorly worded riddle doesn't match the answer I had in my head, checkmate" seems weak.

Possible answers:

- 19 chairs, least likely answer given the original

- 18 chairs, 10 of original 19 removed and 9 added

- 28 chairs, 19 chairs in the room with 10 in active use and 9 new chairs added

Just seems like a question a human would fail too. Probably opens a can of worms on interview questions and personal bias etc too in a more general sense. Not bias in the political sense, just how interview questions encode a lot of the asker's assumptions.

Edit: the only unambiguous phrasing I can think of is "19 chairs are in a room, 9 of the chairs are green, how many chairs are in the room?".

But do you not find the problem is you now end up with tests that assume the existence of #some_method with the tests tightly coupled to the internal logic and flow of that method?

If you decide to refactor, every part of the system now has one or more tests that break because they mimic or clone each method 1:1 rather than testing input and output.