tw4l

They could! They'd have to also change how faculty review and tenure work, to incentivize faculty to publish in alternative open access journals instead of the top (paywalled) journals in their field. Otherwise, faculty will continue publishing with the for-profit publishers even if there's alternatives available.

Former academic librarian and archivist here. It's true that university libraries are changing their shape, as you say. More study space, fewer physical collections and book purchases (and, in some areas, universities moving toward sharing their physical collections in remote offsite storage and only calling back materials when requested), and much more investment in electronic resources like journal and database access.

It's a complicated issue, but in general library budgets have been consistently shrinking for years/decades, even as the subscription costs for journal access from big for-profit like Elsevier skyrockets. It's even gotten to the point where universities with endowments larger than many nations' GDPs are struggling to afford publisher prices (see https://www.theguardian.com/science/2012/apr/24/harvard-univ...).

Decreasing library budgets plays into this, but at its heart I think that there's a crisis of purpose as well. By not investing in open access publishing infrastructure themselves and not modifying the incentives for faculty review and tenure, universities have put themselves at the mercy of these for-profit publishers and are suffering for it. It's hard to blame faculty who are vying for one of the ever-diminishing tenured spots for publishing their articles in the best journals they can, regardless of whether they'll be behind a paywall, especially when the bar for tenure keeps getting higher and higher.

So we find ourselves in a situation where universities and public funding agencies are paying academics to conduct research and do peer review, then faculty are giving the results to a for-profit publisher, who then sells access to the work right back to them at an astronomical fee.

I'm hopeful if not optimistic that at some point, universities might choose to break out of the cat-and-mouse game they're in by investing in open access publishing and incentivizing their faculty to use it rather than continuing to feed for-profit publishers' wild profits. I've seen some (too limited) action on the first part of that but pretty much none on incentivization part. Certain funding agencies are starting to require open access publication as a condition of some grants, but the change has been slow.

Unfortunately not so easy because archiveweb.page relies heavily on CDP. But contributions are welcome if you want to have a go at it!

We experimented moving to Playwright but Playwright doesn't handle long-running browser sessions well, as the devs have (maybe rightly) prioritized its use for testing over archival use cases and want you to spin up a browser each time. For archival purposes, that doesn't work as well because we're not able to save the browser profile to retain cookies such as login credentials, so we've moved back to using Puppeteer for now.

(Disclaimer: I work at Webrecorder)

Our automated crawler browsertrix-crawler (https://github.com/webrecorder/browsertrix-crawler) uses Puppeteer to run browsers that we archive in by loading pages, running behaviors such as auto-scroll, and then recording the request/response traffic in the WARC format (by default in Webrecorder tools, then packaged into a portable WACZ file: https://specs.webrecorder.net/wacz/1.1.1/). We have custom behaviors for some social media and video sites to make sure that content is appropriately captured. It is a bit of a cat-and-mouse game as we have to continue to update these behaviors as sites change, but for the most part it works pretty well. The crawler also has some job queuing functionality, supports multiple workers/browsers, and is highly configurable to set timeouts, page limits, etc.

The trickier part is in replaying the archived websites, as a certain amount of re-writing has to happen in order to make sure the HTML and JS are working with archived assets rather than the live web. One implementation of this is replayweb.page (https://github.com/webrecorder/replayweb.page), which does all of the rewriting client-side in the browser. This sets you interact with archived websites in WARC or WACZ format as if interacting with the original site. replayweb.page can run locally in your browser without needing to send any data to a server or can be hosted, including in an embedded mode.

(edit: fixed typos)

I appreciate you taking the time to share your perspective. I think where we disagree is in characterizing making inclusive language changes as "political" and keeping the status quo as "non-political", when the status quo is demonstrably not working for a lot of people. Changing "master" to "main" is admittedly likely near the end of the list of what the tech industry needs to do to in that regard, but if we can't even get low-effort token gestures like that to happen without a lot of outcry I'm not optimistic about any kind of substantive change.

The idea of places like HN being meritocratic tech utopias where race and gender are irrelevant and everyone's just a username with some neat ideas is really compelling. I can only offer that in my experience it's common knowledge among woman developers that if you want to have a good day, it's best not to read the comments here. It seems I've forgotten that myself!

I do very clearly see how it increases discord and contention in tech communities (see: this thread, previous threads on the topic, Code of Conduct threads, etc).

And I think that increased discord and contention in communities is not worth increased inclusivity (however you measure that).

One measure of inclusivity might be hiring, retention, and promotion within the tech industry (and the developer subset of that) relative to the general population. Looking at the Stack Overflow 2020 survey results as one (admittedly biased) indicator, it's pretty clear that the tech industry does abysmally. As two quick examples: <5% of developers surveyed were Black, and <12% were women and non-binary. Source: https://insights.stackoverflow.com/survey/2020#developer-pro...

Speaking from personal experience as one of those minorities, I don't think people who are outside of these underrepresented groups understand just how many small reminders a day we get that the "default" is for us not to be welcome.

Something like changing language that is rooted in ownership of people is not a huge change, admittedly, but the fact that so many people within the industry complain and fight it definitely serves as a reminder to me that I'm tolerated more than welcomed here.

Edit: Added some clarifying language

That's rather... convenient advice for employees of a company that is alleged to be involved in human rights violations.

Just yesterday, by means of example: https://www.forbes.com/sites/chantaldasilva/2020/09/28/palan...

As David Rosenthal (formerly of Sun, NVIDIA, and Stanford) explains, the actual Arctic Code Vault is a PR stunt, and has almost no chance of helping anyone in any kind of realistic disaster scenario: https://blog.dshr.org/2019/11/seeds-or-code.html

That said, the rest of the project, which focuses on preserving several independent copies of repositories hosted on GitHub with a handful of partner organizations, is quite useful. From the same post: "They are using a range of technologies, making feeds available over the Internet, and partnering with the Internet Archive, the Software Heritage Foundation and the Bodleian Library. These are mostly things which will get used in the foreseeable future, and should be applauded for that reason."