annowiki

How do you get around 403/401's from WSJ/Reuters/Axios? Because I've tried user agent manipulation and it seems like I'd have to use selenium and headless to deal with them.

Another area is with software we’ve had to build (instead of buy). When we started out, we strongly preferred buying software over building it because a team of only a few engineers can’t afford the time cost of building everything. That was the right choice at the time even though the “buy” option generally gives you tools that don’t work. In cases where vendors can’t be convinced to fix showstopping bugs that are critical blockers for us, it does make sense to build more of our own tools and maintain in-house expertise in more areas, in contradiction to the standard advice that a company should only choose to “build” in its core competency. Much of that complexity is complexity that we don’t want to take on, but in some product categories, even after fairly extensive research we haven’t found any vendor that seems likely to provide a product that works for us. To be fair to our vendors, the problem they’d need to solve to deliver a working solution to us is much more complex than the problem we need to solve since our vendors are taking on the complexity of solving a problem for every customer, whereas we only need to solve the problem for one customer, ourselves.

This is more and more my philosophy. I've been working on a data science project with headline scraping (I want to do topic modeling on headlines during the course of the election) and kept preferring roll your own solutions to off the shelf ones.

For instance, instead of using flask (as I did in a previous iteration of this project a few years ago) I went with Jinja2 and rolled my own static site generator. For scraping I used scrapy on my last project, on this one I wrote my own queue and scraper class. It works fantastically.

Actually it seems pretty accurate. Novelty-seeking is a well known phenomenon in curious individuals. https://en.wikipedia.org/wiki/Novelty_seeking

Literally getting dopamine rewards for seeing something new is what keeps people glued to tik tok feeds and twitter.

I tend to get bored halfway through a book if it is predictable.

This is not really "sieve-ing" per the article, but what prevents me from running another process that periodically queries the data in a cache? Like just running a celery queue in Python that continually checks the cache for out of date information constantly updating it? Is there a word for this? Is this a common technique?

There's a piece of graffiti from the May 1968 Paris riots that goes "I have something to say but I don't know what." Reminds me of this. I can't find the original French but I think it's something like "J'ai quelque chose a dire mais je ne sais quoi."

Found it: https://www.dicocitations.com/citations/citation-67489.php

They use the pas

I started as a python programmer and was very used to package managers. I believed in them, I championed them. When I switched to C++ for work I was very disheartened that there wasn't a standard.

Conan obviously has promise, I haven't spent much time with it, most of my experience with C++ package managers is with nuget and vcpkg. However, my attitude toward package managers is changing.

I increasingly like _not_ using package managers because it makes me (and my company) way way way less likely to bloat our software with unnecessary third party dependencies.

I wrote this in another thread: I never believed you should write something yourself if you can find a package for it. My boss told me I should write it all myself, I could probably write it to be faster. I encountered a case where I needed to compare version numbers in python. For the heck of it I wrote the simplest, quickest, most naive solution I could come up with and then timed it against the most recommended version comparison package in python. I blew it away by 20x throughput.

I don't believe in package managers anymore. Obviously I'll keep using pip and sqlalchemy in Python, but I'll happily spend the 20-30 minutes it takes adding something like nlohmann-json or md4c to my project over worrying about maintaining a package manager for c++ these days. Precisely because it makes me think twice about adding another dependency.

A Programmer's Introduction to Mathematics https://pimbook.org/

It introduces math from a mathematician's point of view (complete with proofs, etc.) rather than rote memorization and exercises, but it does so from the perspective of a programmer.

You have me thinking about kind of a cool board idea. 150 person twitter boards. Cap it at 150. People in that group can all vote on their own moderation, they can't interact with groups in other boards through quote tweeting or voting, though obviously they can copy paste.

You might get racist boards, but then its easy to get rid of all of them at once.

150 being https://en.wikipedia.org/wiki/Dunbar%27s_number of course.

I have no way to distribute anything. I tried to do my own annotations board on literature but no one joined. I just think it sounds cool to be in a personable board like that.

When I run ls I don't want to see all the configuration files. Just my files. I think that's the point of hidden files.

One more example. Imagine if you have a project and want to edit an .env file. But as dotfiles are hidden in Linux you don't see this file and cannot open it.

How likely is it that someone is going to want to edit a .env file and not know how to view hidden files?

When I started playing with C/C++ the first thing I complained about was how difficult package management was. Conan is not as simple as cargo or pip. It's often simpler to find a header only library and plop it into your repo, but you still have to modify build configurations to include it.

This is the first time I thought to myself "Maybe that's actually a good thing."

For our purposes, we never have strings that are other than the form XX.XX.XX.XXXX so there was no reason to generalize. Which improves the speed, makes it easier to read, and provides all the more reason not to use a library.

Its usually not so much "a range of wheels optimized for specific roles" and more "one size fits all."

You want a wheel that fits your cart. You have a lathe. Why make do with a wheel someone else made, standardized, and sells to fit a wide range of carts when you can make the perfect wheel for _your_ cart?

Most of the functionality in libraries use standard algorithms anyway. I doubt anyone thinks its a good idea to write your own cryptography or markdown processor, but why do you need a library to left pad a string with zeroes?

if statements and flags take time to process. You only have one way you need to do something. Do you want to take that much more compute just checking flags that will never change so you can get the library function to do what you always want it to do?

Its a numpy style docstring. PyCharm can be configured to auto generate them. I guess I just add the types to be thorough.

Ideally I want to generate api documentation via a tool. A long time ago I wrote a script to parse the AST and generate markdown from the docstrings and function information, but it was mediocre and I haven't wanted to use sphinx because its too heavy and doesn't seem to produce ideal markdown output.

numpy seems to sometimes use type hints in the docstring, other times not: https://numpydoc.readthedocs.io/en/latest/format.html#parame...

I come from a Python background so libraries were always just a pip install away.

In the past year, however, I've been working on a large C++ codebase (couple million lines) and the result has been a considerably greater amount of "roll your own."

This has filtered back into my Python. If it's not in the standard library, I don't install it unless I really need it.

My boss always tells me that you can probably write it faster than a library, something I never used to believe. Until I tried it. I needed to check which of two version strings, of XX.XX.XX.XXXX format, were bigger. I tried the most recommended version number library, then I tried writing my own solution, the simplest solution I could think of:

    def version_compare(v1: str, v2: str) -> int:

        """
        Compares two versions.

        Parameters
        ----------
        v1: str
        v2: str

        Returns
        -------
        int
            1 if v1 is greater, -1 if v2 is greater, 0 if equal.
        """
        for el1, el2 in zip(v1.split('.'), v2.split('.')):
            if el1 != el2:
                return 1 if int(el1) > int(el2) else -1
        return 0

I believe he means "the use of asterisks to mark required fields" was invented for the web. Otherwise, you'd have to be quite young and quite sheltered to not have seen asterisks in places older than the web.

https://en.wikipedia.org/wiki/Free_Speech_Movement

Mario's "Operation of the Machine" speech is pretty good.

Also worth noting, often these groups were not quite as egalitarian as they're thought to be. SDS for instance had quite a bit of sexism in its operations: https://en.wikipedia.org/wiki/Students_for_a_Democratic_Soci...

I have no position on this debate, I just thought you'd want a little context.

I can strongly recommend Fussel's (the author of this essay) book "The Boys' Crusade", which provides a very eye opening account to WWII. It is most likely a longer version of this essay (I haven't read this essay).

I can also recommend his book Poetic Meter & Poetic Form, but for different reasons.

Other books in the same vein:

- The Warriors: Reflections on Men in Battle by J Glenn Gray (Gray was a philosophy PhD and a 2nd Lieutenant in the war)

- War is a Force that Gives Us Meaning by Chris Hedges (Hedges was a war correspondent during the balkans and numerous other conflicts, this is much more interested in the psychological build up to war in common society, and its effects on society)

These books make me very pessimistic about human nature, but Nicholas Wade's Before the Dawn: Recovering the Lost History of Our Ancestors provides a nice antidote: the total number of people killed by warfare in the 20th Century, if we followed similar patterns to our prehistoric ancestors, would have dwarfed what actually happened. We are becoming less warlike, even if its not cured.

Small typos everywhere. Init functions without the dunders, move_ahead in an abstract class then moveahead later.

I just linked your blog post in a separate comment because I was reading it over the weekend. I was looking for articles about writing static site generators and didn't realize most people don't roll their own templating engines. I was disappointed to see not much on templating in your post but it definitely gave me the confidence to keep going. Managed to get aspects I'd been struggling to work for months working yesterday.

This is mindblowing synchronicity. I thought this article was one I had read this weekend for a second but checked the post date and saw it wasn't available yesterday.

I'm working on trying to write a static site generator for my own personal site/blog in C++, been tinkering for a couple months. It started as a passion project in tribute to my dog that passed. GatsbyJS wasn't working (again) which is what my current site is built with, so I just said screw it, I'll write my own. Chose C to begin with and quickly gave up. Decided to switch to C++ because its what I'm supposed to be learning for work.

I named it bluesky, after my dog that passed, Sky Blue. https://github.com/mas-4/bluesky

Templating is a lot harder than I had initially thought. I wanted the templating system to be as bare bones as possible, similar to another static site generator I rather like, sergey[0]. Most of these articles I found ([1], [2]) about writing your own SSG don't go into templating much, they just use an off the shelf library. Inja[3] is available for C++ but, like I said, I want something really bare bones, like if you were designing html now, you'd obviously include html-includes and html-templates. I finally got it working for includes and templates, now I have to add the markdown support, and then I plan on migrating my personal site to using it.

[0] https://sergey.cool/ [1]: https://www.smashingmagazine.com/2020/09/stack-custom-made-s... [2]: https://blog.hamaluik.ca/posts/build-your-own-static-site-ge... [3]: https://github.com/pantor/inja

edit: a link

I've been doing this but without the jinja. It's trivial to store a latex file or markdown file with something like %PLACEHOLDER% and then just run

    with open('template.tex', 'rt') as f_in:
        template = f_in.read()
    template = template.replace('%PLACEHOLDER%', data)
    fname = f'rendered-{dt.now().isoformat()}'
    with open(fname, 'wt') as f_out:
        f_out.write(template)
    fileout = 'outdir'
    subprocess.call(['pdflatex',  '--output-directory', fileout, fname])

I consider myself quite adept at cli, I can sed, grep, find, vim, and git quite well.

However, what I never found convenient in cli was debugging. I was a print debugger for years until my job convinced me to try PyCharm.

I've always thought the reason I never found a suitable debugger in cli was just because I was either too dumb to figure out integrating something like vimspector, or too lazy to get used to gdb/pdb. Ctrl-f on this article has no mention of the word "debug." I think I'm not the only one that never used a debugger before I tried an IDE. I wonder how many people still feel superior in their CLI-Vim/Emacs-only workflow and are just print debugging.

IdeaVim works fine. I can't use vim sandwich and that's about my only complaint.

I was firmly resistant to IDEs like OP and quite in love with neovim for years. Tried PyCharm because of work, never looking back.

I could never find just a simple debugger for vim that was easy and intuitive to set up. Always had trouble.

And now, the in-IDE Jupyter Notebooks that allow me to use vim keybindings (albeit with an annoying bug that at least has a workaround https://youtrack.jetbrains.com/issue/VIM-2504) has completely changed the way I work.

Are there occasionally show stopper bugs? Yes. Is it worth it for amazing debugger integration? Absolutely.

Scikit-Learn's api is masterfully consistent. We've taken to using it as a source of inspiration for our own code.

Why wouldn't they just use the older version of C++? If you don't want non-c compatible c++ you could just stick to C++20 instead of upgrading to "C++breaking_changes".

I ask because its not like support will ever be phased out like Python 2.

The debugger.

I made your same argument for years to a buddy of mine that always used WebStorm (I'm a Python dev and he kept pushing me to try PyCharm). A coworker pushed me to try PyCharm since we're on Windows and I haven't really looked back. I even bought CLion.

If you can point me to an easy debugger to set up in neovim I'd probably be right back, but I can't. Couldn't figure out how to set up vimspector. And in PyCharm, at this point, I find it has better tooling than coc (I haven't wanted to spend the time setting up LSP/treesitter).

Furthermore, doing development in wsl isn't that great in my opinion and native windows neovim isn't either, even when using Windows Terminal. Its slow when handling a massive repository. FZF with ripgrep still works massively faster than telescope for my main work repository. I mostly dip into wsl when I want to do complex grep/sed/find operations and to make edits to my hledger timeclock sheet.

But one thing I've found to be a killer app for Pycharm Pro: vim keybindings in jupyter notebooks. Works way better than the plugins for Jupyter Lab.

IdeaVim is sometimes kind of disappointing, I wish JetBrains would just add a neovim plugin (I want to use vim sandwich instead of vim surround but alas). Still, it's good enough.

I did a double take because I've been using pictures from unsplash found by searching "tokyo night"[0] and "night city"[1] for wallpapers and they're nearly identical in color, saturation, and mood. I just rechecked because I thought maybe someone was stealing his work. Nope, they're originals, but amazingly similar.

[0] https://unsplash.com/s/photos/tokyo-night [1]: https://unsplash.com/s/photos/night-city

Duncan Lewis's look especially similar, especially this one: https://unsplash.com/photos/_iS8zogaLcg

It's nearly identical to the left image from the fourth set from the top.

Fascinating. What did you do for scraping? I built something similar but to do sentiment analysis on news (it is shamefully slow https://maudlin.standingwater.io/ ). I used scrapy to get the articles and post them to a postgres database that flask reads.

I think I'm going to take some design queues from you and simplify my system. Word clouds are cool but too intensive on the $5 server I pay for.

Is it basically a PID controller?

This is amazing. I just started working on my own todo list system[1] the other week after trying tons of todo list apps. I focused on a command line flat file system first because I've started using hledger and I have a markdown wiki in Obsidian but I want to develop a graphical interface eventually. Yours is lovely. Great ui! I hope you don't mind me taking inspiration!

The biggest thing I've found lacking in most todo list apps is recurring checklists. The closest I've come to having the feature set I wanted was Microsoft's Todo[2] app, which was based on Wunderlist[3]. Microsoft bought and killed it before I had the chance to use it but it sounded pretty excellent.

I still haven't set up my checklist system but I started to take inspiration from scrum at work to just design a hierarchical todo list system with separate files for each long term "epic" project, and then a rolling todo list that should have copies of the lines from the epic files (all of them have unique ids) so that I can check them off easily. The workflow is still evolving.

Thanks for sharing! I'm gonna be watching your repo!

1: https://github.com/mas-4/scrummy 2: https://todo.microsoft.com/tasks/ 3: https://en.wikipedia.org/wiki/Wunderlist