HN user
"Which shows that there are 39 movies in Amazon Prime US from Ebert's "Great Movies," not 21 like this guy's spreadsheet says."
I could be wrong, I am not a Prime Video user, but the result I got was that there are 217 movies in Prime Video from Ebert's great movies.
links -dump 1.html|grep -c Prime.Video
No agent at all is required. I got past the anti-robot response using no user-agent header and a simple delay.
Instead of using Python, here is a solution that only requires sh, curl, sed, sort, uniq and grep.
This solution uses a generous 87s delay to retrieve the Amazon pages. There are 328 films listed as "great movies" on rogerebert.com. As such, the script, named "1.sh", needs 8h to complete, e.g., the time while you are at work or sleeping. No cookies, no state, no problems.
Usage: sh -c 1.sh > 1.html
#!/bin/sh
curl -HUser-Agent: -H'Accept: application/json' --compressed 'https://www.rogerebert.com/great-movies/page/[1-16]?utf8=%E2%9C%93&filters%5Btitle%5D=&sort%5Border%5D=newest&filters%5Byears%5D%5B%5D=1914&filters%5Byears%5D%5B%5D=2020&filters%5Bstar_rating%5D%5B%5D=0.0&filters%5Bstar_rating%5D%5B%5D=4.0&filters%5Bno_stars%5D=1'|grep -o "/reviews/great-movie-[^\\]*"|sed 's/.reviews.great-movie-//'|sort|uniq|while read x;do y=$(echo $x|sed 's/-/+/g');echo $x;curl -s --compressed -HUser-Agent: https://www.amazon.com/s/?k=$y 2>/dev/null|grep -m1 -C4 a-link-normal.a-text-bold;sleep 87;done|sed '/^[^< ]/s/.*/@&,/;1s|.*|<base href=https://www.amazon.com />&|;s/ *//;/^$/d;/^[@<]/!s|$|</a>|;1s/@//;s/@/<br>/'If Pike and his crew represent users, then the tech industry are the Talosians?
Absolute favourite episode hands down, The Cage. According to Shatner's autobiography, NBC called the pilot "too cerebral" and "too intellectual".
"The Commercial Web (of Marketing)
There has always been a place for commerce and marketing on the web."
Not really true as I remember it. The web opened up to the public in 1993. There was no commerce and marketing in the beginning. Even by 1996 while commerce and marketing may have existed, e.g., Amazon founded in 1995, its place was in the background. As I rememember the early web, the foreground, the "starting point" or "portal", was something like Yahoo! You had to pick a topic (direction) that you wanted to go in. For example, if you were after music, you might end up browsing the Internet Underground Music Archive. The "front page" of the portal was predominantly non-commercial, mostly generic headings for topics. If you wanted to search out something commercial, no doubt you could but the initial starting point was intellectual curiosity. This is IMO what has been lost over time with regard to web use: intellectual curiosity and the ability to actually satisfy it. (A fun tangent here is the collections of inane queries that people type into Google. These are simultaneously hilarious and disturbing.)
As an experiment have a look at the Yahoo! page today. It is full of low quality mainstream "news". There is zero attention to intellectual curiosity. Nothing to see here, folks, but here is the latest news. For part 2 of the experiment, run a Google search for the term "music". The results are dominated by YouTube. Every result is directly or indirectly commercial (either selling something or conducting surveillance and serving ads), except one: Wikipedia. The chances of someone new to the web not following a link to YouTube or some other Google-controlled domain would seem almost nil.
The "onboarding" process for new web users is very different today than it was in the early 1990's. Perhaps it is still possible to approach the web with a sense of awe and wonder, pondering "What is out there?" However a new web user is scant likely to end up on a non-commercial website besides Wikipedia. What is out there? Surveillance, ads and an endless supply of soon-to-be-obsolete Javascript du jour.
Why dump billions of dollars then? Nowhere else to spend it? Effective marketing?[1] Is no one asking this question?
"... and critically: there's no one to hold responsible for getting it wrong."
Could this be part of "AI"'s appeal? A dream of absolving businesses and individuals from accountability.[2]
1. "What's more, artificial research teams lack an awareness of the specific business processes and tasks that could be automated in the first place. Researchers would need to develop an intuition of the business processes involved. We haven't seen this happen in too many areas."
2. Including the ones who designed the "AI" system.
"DNA is not code, it's just a molecule that reacts with stuff, as are all other molecules."
This sentence exemplifies the perspective to which I referred.
Improved
#/bin/sh
test -s max-PMID||echo 32449615 > max-PMID;read x < max-PMID;h=pubmed.ncbi.nlm.nih.gov;
test ${#x} -eq 8||rm max-PMID;sed -i "s/[0-9]\{8\}/$x/" $0;
case $1 in update) mkfifo 1.fifo 2>/dev/null;test -p 1.fifo||exec echo need 1.fifo;
(grep "<title>PMID .* is not available" < 1.fifo|sed 1q|sed -n 's/<title>PMID //;s/ *//;s/ .*//;wmax-PMID')&
y=$((x+10000));seq $x $y|sed '$!s|.*|GET /&/ HTTP/1.1\r\nHost: '"$h"'\r\nConnection: keep-alive\r\n\r\n|;
$s|.*|GET /&/ HTTP/1.1\r\nHost: '"$h"'\r\nConnection: close\r\n\r\n|'|socat - ssl:$h:443 2>/dev/null|grep -o '<title>[^<]*' >1.fifo;
;;"")awk -v min=1 -v max=$((x-1)) 'BEGIN{srand();printf "https://'$h'/" int(min+rand()*(max-min+1)) "/\n"}';esac"It's not super productive to try and understand this stuff through analogy to computers and algorithms."
You mean like this: https://ds9a.nl/amazing-dna
Having worked in both industries I prefer working with wet science people. For some reason they generally have a much healthier perspective on life. Their work is humbling because it is, and will forever be, full of unsolved mysteries, not simply because it is challenging. The other folks, whether they call themselves "scientists" or "engineers" or "developers" or "coders" or whatever, are working with something that as far as I can see has no inherent connection to the natural world, other than being a production of the human mind. Perhap that affects the perspective many of them have on life. For example, how common among them is this belief that all things, not simply computers, can be thoroughly understood and mastered. Note this is pure opinion, not fact, and I am generalising; there are exceptions to every generalisation.
What if PubMed had something like Google's "I'm feeling lucky"? What if we could explore PubMed by selecting a random PubMed URL instead of searching? This script generates a random PubMed URL. To do this we need to know the maximum PMID number in the PubMed database. The current max is included in the script and will be saved in a 9-byte file named "max-PMID" when the script is run. If run with the argument "update" it will search for a newer max PMID. If a newer max PMID is found, the script updates the number in the max-PMID file and in the script itself. An alternative is to use the ftp server[1] to find the max PMID; I noticed the latest ftp update was missing new PMID's caught by this script. If run without any arguments it selects a random PMID between 1 and the max and outputs a URL. uses socat, GNU sed and requires a fifo named "1.fifo" 1. ftp://ftp.ncbi.nlm.nih.gov/pubmed/updatefiles/
#!/bin/sh
test -s max-PMID||echo 32446294 > max-PMID;read x < max-PMID;x=$((x-1));h=pubmed.ncbi.nlm.nih.gov;
test ${#x} -eq 8||exec echo weird max-PMID;sed -i "/test/s/echo [0-9]\{8\} /echo $x /" $0;
case $1 in update) mkfifo 1.fifo 2>/dev/null;test -p 1.fifo||exec echo need 1.fifo;
(grep "<title>PMID .* is not available" < 1.fifo|sed 1q|sed 's/<title>PMID //;s/ *//;s/ .*//;' >max-PMID)&
y=$((x+10000));seq $x $y|sed '$!s|.*|GET /&/ HTTP/1.1\r\nHost: '"$h"'\r\nConnection: keep-alive\r\n\r\n|;
$s|.*|GET /&/ HTTP/1.1\r\nHost: '"$h"'\r\nConnection: close\r\n\r\n|'|socat - ssl:$h:443 >1.fifo 2>/dev/null;
;;"")awk -v min=1 -v max=$x 'BEGIN{srand();printf "https://'$h'/" int(min+rand()*(max-min+1)) "/\n"}';esacFor those not using Javascript (SSRN uses third party scripts), link to PDF: https://poseidon01.ssrn.com/delivery.php?ID=5550640311191050...
The long line of sed is out-of-date and thus "broken". For something simpler that works, try this:
nyt tr |sed 's/ *//;/</!d'|uniq > travel.html
An interesting point of discussion might be the amount of third party cruft on the template article page versus the more dynamic front page. When Javascript is disabled, on each article page all images display and there are no ads. Downloading any video in the page is as simple as
curl -O `grep -o https://[^\"]*mp4 article.html`"It's text, just send the damn text."
They only send what the user requests.
Using a software program that makes automatic requests that you are not easily in control of, e.g., a popular web browser, might give the impression that they control what is sent.
They do not control what is sent. The user does.^1
The user makes a request and they send a response.
One of the requests a fully-automatic web browser makes to NYT is to static01.nyt.com
Personally, as a user who prefers text-only, this is the only request I need to make. As such I don't really need a heavily marketed, fully-automatic, graphical, ad-blocking web browser to make a single request for some text.^2
#! /bin/sh
case $1 in
world |w*) x=world # shortcut: w
;;us |u*) x=us # shortcut: u
;;politics |p*) x=politics # shortcut: p
;;nyregion |n*) x=nyregion # shortcut: n
;;business |bu*) x=business # shortcut: bu
;;opinion |o*) x=opinion # shortcut: o
;;technology |te*) x=technology # shortcut: te
;;science |sc*) x=science # shortcut: sc
;;health |h*) x=health # shortcut: h
;;sports |sp*) x=sports # shortcut: sp
;;arts |a*) x=arts # shortcut: a
;;books |bo*) x=books # shortcut: bo
;;style |st*) x=style # shortcut: st
;;food |f*) x=food # shortcut: f
;;travel |tr*) x=travel # shortcut: tr
;;magazine |m*) x=magazine # shortcut: m
;;t-magazine |t-*) x=t-magazine # shortcut: t-
;;realestate |r*) x=realestate # shortcut: r
;;*)
echo usage: $0 section
exec sed -n '/x=/!d;s/.*x=//;/sed/!p' $0
esac
curl -s https://static01.nyt.com/services/json/sectionfronts/$x/index.jsonp
Example: Make simple page of titles, article urls and captions, where above script is named "nyt".
nyt tr | sed '/\"headline\": \"/{s//<p>/;s/\".*/<\/p>/;p};/\"full\": \"/{s//<p>/;s/..$/<\/p>/;p};/\"link\": \"/{s///;s/ *//;s/\".*//;s|.*|<a href=&>&</a>|;p}' > travel.html
firefox ./travel.html
The truth is that they are just sending the damn text. However you are voluntarily choosing to use a software program that is automatically making requests for things other than the text of the article, i.e., "cruft".
1. The Google-sponsored HTTP/[23] protocol is seeking to change this dynamic, so if websites sending stuff to you without you requesting it first bothers you, you might want to think about how online advertisers and the companies that enable them might use these new protocols.
2. However I might use one for for viewing images, watching video, reading PDFs, etc., offline. Web browsers are useful programs for consuming media. It is in the simple task of making HTTP requests that their utility has diminished over time. The user is not really in control.
"Which brings us to the question - what is the point of all this? These platforms are all losing money. Just think of all the meetings and lines of code and phone calls to make all of these nefarious things happen which just continue to bleed money. Why go through all this trouble?
How did we get to a place where billions of dollars are exchanged in millions of business transactions but there are no winners? My co-host Can and my restaurant friend both defaulted to the notion "delivery is a shitty margin business" when discussing this post.
You have insanely large pools of capital creating an incredibly inefficient money-losing business model.
It's used to subsidize an untenable customer expectation.
Third-party delivery platforms, as they've been built, just seem like the wrong model, but instead of testing, failing, and evolving, they've been subsidized into market dominance.
The more I learn about food delivery platforms, as they exist today, I wonder if we've managed to watch an entire industry evolve artificially and incorrectly."
A contrary view, from 4 days ago, arguing third party food delivery market is not created by VC, the startups are not over-funded and that they are delivering splendid returns to investors.
There is a link to the PDF on the page. No ACM membership required.
"Also known as rump kernels a name inspired by the infamous purge of royalists from Parliament following the English Civil War this process involves creating a fork of an existing kernel codebase and manually purging it of the components deemed unnecessary to the target unikernel."
That appears to come from the FAQ on Github for "rumprun" dated 2015.
Rump kernels were introduced in NetBSD in 2009. It was an described then as an acronym for runnable userspace meta programs.
https://blog.netbsd.org/tnf/entry/runnable_userspace_meta_pr...
I have not used rumprun for Linux but I still use the rump utilities included with NetBSD.
"They are redirecting those customers to their platform where they can purchase from a competitor or they can charge us a fee to provide food to that same customer. Whereas the customer could have just gone to our website and ordered directly from our online ordering store and we would not be charged a commission to GrubHub or DoorDash."
Here he refers to Grubhub and DoorDash as "competitors" with the restaurant.
Yet at the conclusion of the story he refers to them as "partners" with the restaurants who pay them.
"DoorDash and GrubHub have a long history of screwing their partners."
These companies are "middlemen". That description in fact fits a large number of "tech" companies. It is a common theme when we look at the "tech" industry.
A few days ago food delivery, including Grubhub, was discussed on HN.
https://news.ycombinator.com/item?id=23169597
One commenter argued that companies like Grubhub exist in "fake markets" that would not exist were it not for VC funding.
Another argued that these markets are not fake because these companies have managed to be profitable and deliver returns to investors.
Not sure what you all have experienced, in my experience with restaurants and delivery, the owners always seem to prefer customers who order directly from the restaurant, not via a middleman. However, I can sense that they feel compelled to sign up with the middlemen for fear of losing business.
HN commenters seem to enjoy comparing first page search results on Google versus alternatives such as DDG. Try this one out. Search for your favourite restaurant. On Google, I see more middlemen than I see on DDG, including a result for "[restaurant-name].business.site" which I believe belongs to Google. I don't see that result on DDG.
Imagine if the telephone book Yellow Pages listed multiple numbers for each business, some of them not belonging to the business but to middlemen who would "make calling easier". Even more, imagine it changed the order of the listings according to a "secret" methodology instead of presenting them in alphabetical order.
One can use "port knocking" for more than remotely opening ports. It is a crude form of messaging in its own right, that can be based on some pre-determined code, like Morse code. The "secret knock" need not open any ports. It can be simply a message to the person (or program) reading the logs, to be translated according to the pre-determined code. Actions might be taken in response to the message, or not.
It is possible to do similar things with haproxy. Configured to listen for TLS connections on a large number of ports, look for a secret combination of custom headers and values, then, if found and matching, forward to a localhost ucspi tcpserver on the backend. The tcpserver may then execute some program, for example sshd or pfctl.
"How do you reconcile this narrative with the fact that Grubhub turned a net profit every available year until 2019 per their SEC filings."
Easy. Take away the VC funding and see if anyone wants to try or can manage to start a similar company.
In most cases, these companies are just middlemen. Many folks do not see middlemen as "legitimate" businesses, even though the middlemen make money.
Hosts and parasites can each thrive. However only one can survive on its own. Ideally hosts would prefer to live free of parasites.
It should be expected that some consumers or producers will want to cut out the middlemen. We cannot reasonably expect everyone to appreciate those middlemen for the success they may have in taking a cut (and collecting data on consumer behaviour).
"This whole industry is yet another example of "fake markets" that were created by just massively subsidizing something with VC cash."
To what extent has America (and its followers) been led to believe these types of companies, who always seem to have a non-sensical name, and the narratives they construct ("fake markets") are the future of the American economy. While these middlemen may be a part of the future and play a role, are they receiving a disproportionate amount of attention relative to their true importance.
The article is about Canada not America.
Protocol mandates use of nothing less than TLS v1.2 so one could skip/remove all support for lesser versions.
known servers: https://portal.mozz.us/gemini/gemini.circumlunar.space/serve...
spec: https://portal.mozz.us/gemini/gemini.circumlunar.space/docs/...
docs: https://portal.mozz.us/gemini/gemini.circumlunar.space/docs/
software: https://portal.mozz.us/gemini/gemini.circumlunar.space/softw...
test ur gemini client: https://portal.mozz.us/gemini/gemini.conman.org/test/torture...
using the protocol ...
known servers
printf 'gemini://gemini.circumlunar.space/servers/\r\n' |openssl s_client -connect gemini.circumlunar.space:1965 -ign_eof
printf 'gemini.circumlunar.space/spec/spec-spec.txt\r\n'|socat - ssl:gemini.circumlunar.space:1965
echo -e '/docs\r\n' |socat - ssl:gemini.circumlunar.space:1965
echo -e '/software\r\n'|socat - ssl:gemini.circumlunar.space:1965
printf '/software/\r\n' |openssl s_client -connect 168.235.111.58:1965 -servername gemini.circumlunar.space -ign_eof
printf 'gemini://gemini.circumlunar.space/software/\r\n' |openssl s_client -connect 168.235.111.58:1965 -ign_eof"I'm really sick of everything being made "secure", when in fact the "security" is for someone other than the legitimate user of the thing."
There must have been some groundswell movement amongst users all demanding that the boot process be made more "secure". There must have been well-publicised cases where "bad guys" were hijacking the boot process.
Perhaps different people have different definitions of "secure". If some third party, including the seller, has control over access to the computer or what I can run or disable on it after I purchase it, then I do not consider that computer to be more "secure". I just consider it to be less useful and less trustworthy to use with any personal data.
Are farmers considered to be "working from home"?
"An arbitrary gTLD is not something you can get from free, until we have a better solution for DNS that the current appraoch."
This implies that there is a better approach. I agree. There are many possibilities. I have been running own root zone at home for several decades. I routinely experiment with different ideas. I am aware of many possibilities.
I do generate a CA pem file using openssl in order to view own traffic. However even with all these hoops we jump through to do such basic things, "Big Tech" seems to actively try to discourage users even more, .e.g., from monitoring their own traffic. For example, subjecting the user to a keylogger in Chrome and forcing them to type some undocumented, opinionated shibboleth such as "badidea" or "thisisunsafe" to proceed.
https://stackoverflow.com/questions/35274659/does-using-badi...
The subject of the OP is what was originally supposed to be a "non-commercial" TLD, which since abandoned that designation and began charging fees, and, more importantly, the current and future lack of any restrictions on raising those fees. If one is using a TLD in ways that demand transparency, e.g., to represent an entity that is taking money from the public, then involving a third party that accepts personally identifiable information, e.g., a registrar, makes some sense. However, interactions with the public, e.g., commercial uses, are not the only ways the internet can be used and such uses are not the only way domains can be used.
That's a good point. I mainly use it for browsing websites that change daily as well as ones with many successive pages, e.g., 1, 2, 3, etc. that IA does automatically not crawl. Thus, not many existing copies if any.
HAproxy changes the Host header and modifies the URL. I can either use the text-only browser's http-proxy option or I can direct the request to the web.archive.org backend by adding a custom HTTP header to the request. If I am not mistaken, the so-called "modern" browsers do not have built-in capability to add headers.
If the point of the domain name for you is marketing, then get one you think is "memorable" and use it as a CNAME.
The point for me would be non-marketing uses, e.g., to get an SSL cert so I do not have to use a self-signed one when monitoring encrypted egress traffic from the local network. That name does not need to be memorable. I can get a free cert from Let's Encrypt but I cannot get free domain name. Best I can get for free is a subdomain of someone else's domain name.
As you know, not every domain name is used for the purposes of email addresses or a website. To use a common example, AWS Cloudfront subdomains are not memorable but they are useful to AWS. Like subdomains, domain names can still be useful without being memorable.
What if no one gets "x3blah" and we each get some random string, e.g., an ed25519 public key.
Handshake looks like it wants to create another "gold rush" for "vanity" names. We have already had that. We have already seen how it plays out. It has been a while since I read through the Handshake docs. Are there limits on how many names a single applicant/organisation could register.