gwu78

http://www.internetsociety.org/deploy360/blog/2017/05/google...

I was using the first example with a char in the first column.

   A 4
   B 5
   B 8
   C 9
   A 6

Regarding the 1gram file at https://storage.googleapis.com/books/ngrams/books/googlebook...

This is the result I got

   3| 1742563279

   q)\ts d:(!/)(" II";"\t")0:`:1gram
   q)\ts 1#desc sum each group d
   1897 134218176
   371 238872864

   k)\ts d:(!/)(" II";"\t")0:`:1gram
   k)\ts desc:{$[99h=@x;(!x)[i]!r i:>r:. x;0h>@x;'`rank;x@>x]}
   k)\ts 1#desc (sum'=:d)
   1897 134218176
   0 3152
   372 238872864

"The task is to sum the values for each key and print the key with the largest sum."

What is the smart way to do this in kdb+?

This is my naive, sloppy 15min approach.

Warning: Noob. May offend experienced k programmers.

   k)`t insert+:`k`v!("CI";"\t")0:`:tsvfile
   k)f:{select (*:k),(sum v) from t where k=x}
   k)a:f["A"]
   k)b:f["B"]
   k)c:f["C"]
   k)select k from a,b,c where v=(max v)

Long before amp, Google began prefixing search result urls with "google.tld?url=" and adding Google parameters as suffixes such as "sa=", "ved=", etc.

Unless I am mistaken this parasitic cruft only serves Google, not end users.

Below is quick and dirty program to filter out the above. Replace .com with .cctld as needed.

Requirements: cc, lex

Usage:

   curl -o 1.htm https://www.google.com/search?q=xyz
   yyg < 1.htm > 2.htm
   your-ad-supported-web-browser 2.htm

   flex -Crfa -8 -i g.l;
   cc -Wall -pipe lex.yy.c -static -o yyg;

   %%
   [^\12\40-\176]
   \/url[?]q= 
   "http://www.google.com/gwt\/x?hl=en&u=" 
   "&"[^\"]* 
   %%
   main(){yylex();}
   yywrap(){}

amphtml does look great in a text-only browser that does not load iframes automatically.

Favorite part is how he mixes the register allocator with a chain of seds.

I did something similar with a youtube downloader I wrote, using a long chain of seds. It is not as beautiful as Python but it is smaller and faster.

When I shared it with HN the youtube-dl author called it "unmaintainable". By who? I have had no problems maintaining it. :)

This is my "Hacker News Reader". It converts HN to csv. (Only selected fields of interest to me.) From there it can easily be imported into kdb+. I have more reusable generalized lex techniques for other websites but HN is so simple it can be done via a braindead one-off as below.

Requirements: lex, cc

Usage:

   fetch -4o yc.htm https://news.ycombinator.com
   yc < yc.htm 

    flex -Crfa -8 -i yc.l;
    cc -Wall -pipe lex.yy.c -static -o yc;

    #define jmp BEGIN
    #define p printf
    #define x yytext
   %s aa bb cc dd ee ff gg hh
   %s ii jj kk ll mm nn oo 
   aa "span class=\"rank\""
   bb "a href=\""
   cc score_........
   dd \>
    /* #include <time.h> */
    /* #include <util.h> */
   %%
   [^\12\40-\176]
   , p("%2c");
    /* rank (dont care) */
   {aa} jmp aa;
    /* <aa>[1-9][^<\.]* p("\n%s,",x);jmp bb; */
   <aa>[1-9][^<\.]* p("\n");jmp bb; 
    /* url */
   <bb>{bb} jmp cc;
   <cc>http[^"]* p("%s,",x);jmp dd; 
    /* title */
   <dd>{dd} jmp ee;
    /* <ee>[^><]* p("%s,",x);jmp ff; */
   <ee>[^><]* p("%s",x);jmp ff;
    /* host (omit) */
    /* points (dont care) */
   <ff>{cc} jmp gg;
   <gg>{dd} jmp hh;
    /* <hh>[1-9][^<> p]* p("%s,",x);jmp ii; */
   <hh>[1-9][^<> p]* p(",");jmp ii; 
    /* user */
   <ii>{bb} jmp jj;
   <jj>http[^"]* p("%s,",x);jmp kk;
    /* time (dont care) */
   <kk>{bb} jmp ll;
    /* <ll>http[^"]* ; */
   <ll>http[^"]* jmp mm;
    /* unix time (dont care) */
    /* <ll>[1-9][^<]* { 
    time_t t0; time_t t1; time_t t2;
    t1=time(&t0);
    t2=parsedate(x,&t1,0);
    p("%d,",t2); 
    jmp mm;
    } */
    /* item */
   <mm>{bb} jmp nn;
    /* <nn>http[^"]* p("%s,",x);jmp oo; */
   <nn>http[^"]* p("%s",x);jmp oo;
    /* comments (dont care) */
   <oo>{dd} jmp oo;
    /* <oo>[1-9d][^ <]* p("%s",x);jmp 0; */
   <oo>[1-9d][^ <]* jmp 0;
   .
   \n
   %%
   main(){ yylex();}
   yywrap(){ p("\n");}

"I suggest compiler writers stop viewing the x86 stack as a twisted version of a traditional 8-register set."

What if compiler writers will not fix the problem?

Write own register allocator: http://cr.yp.to/qhasm.html

Bonus: allow use of a "portable assembly language" to generate assembly, being respectful of the fact that users might not all be using computers with the same CPU architecture.

This could be like assembler with C-like operators and structure.

In addtion to facilitating portability it might also make writing in assembler a little easier.

Reminds me of Bell Labs' LIL: http://www.ultimate.com/phil/lil/tut.html

Yes, this sounds right. It has been a while since I looked at it. Is it just one name? I have a faint recollection it tried more than one.

Anyway, how is the difference significant?

A localhost cache can point at a custom root.zone. The user can make her own authoritative nameserver assignments for any given zone or domain. Zone files can contain wildcards.

Responses can also be rewritten on the fly.

The end user can exercise full control over what is and is not a "valid" domain name. She can prevent her applications from ever receiving an "NXDOMAIN" response.

Maybe I am missing something but this "test" seems brittle; it only tests ICANN DNS.

Notable that he calls the "kill-switch" a "mistake". For example, Chrome does the same thing. When it starts it checks for some presumably non-existant domain name.

Is this a "JSON Feed" from NYTimes?

Example below filters out all URLs for a specific section of the paper.

   test $# = 1 ||exec echo usage: $0 section

   curl -o 1.json https://static01.nyt.com/services/json/sectionfronts/$1/index.jsonp
   exec sed '/\"guid\" :/!d;s/\",//;s/.*\"//' 1.json

Personally, I think a protocol like netstrings/bencode is better than JSON because it better respects the memory resources of the user's computer.

Every proposed protocol will have tradeoffs.

To me, RAM is sacred. I can "parse" netstrings in one pass but I have been unable to do this with a state machine for JSON. I have to arbitrarily limit the number of states or risk a crash. As easy as it is to exhaust a user's available RAM with Javascript so too can this be done with JSON. Indeed they go well together.

"Lessons learnt by ransomware developers..."

If you are suggesting that developers, regardless whether they develop mobile apps or ransomware, will start relying less on DNS, I respectfully disagree.

Someone else in this thread commented how reliance on DNS makes systems "fragile". With that I strongly agree.

The same old assumptions will continue to be made, such as the one that DNS, specifcally, ICANN DNS, is always going to be used.

How to break unwanted software? Do not follow the assumptions.

For example, to break a very large quantity of shellcode change the name or location of the shell to something other than "/bin/sh".[1]

Will shellcoders switch to a "robust statistical model" instead of hard coding "/bin/sh"?

Someone once said that programmers are lazy. Was he joking?

1. Yes, I know it may also break wanted third party software. When I first edited init.c, renamed and moved sh I was seeking to learn about dependencies. I expected things to break. That was the point: an experiment. I wanted to see what would break and what would not.

Here is a spitbol/snobol4 solution. Assumes the number of items in the set is not greater than the alphabet.

   * routine stolen from gimpel
   * algorithm from peck and schrack 

    define('p(s)t,n,c,k','p_init') :(p_end)
   p_init n = size(s)
    r = array('2:' n, 0)
    &alphabet len(n) . y
    x = array('2:' n, y)
    k = n + 1
   p_0 k = k - 1
    x[k] len(1) . s1 tab(k) . s2 = s2 s1  :s(p_0) 
    define('p(s)i,k') 
    p = s :(return)
   p k = size(s)
   p_1 s = replace(x[k],y,s) :f(p_2)
    r[k] = r[k] + 1
    k = eq(remdr(r[k], k),0) k - 1 :s(p_1)
    p = s :(return)
   p_2 define('p(s)t,n,s1,s2','p_init') :(freturn)
   p_end

   * example: all permutations of string abcdefgh
    s = 'abcdefgh'
   abc output = p(s) :s(abc)f(end)
   end

    define('r(s,ors)c,f,s1,a,d,os') 
    :(r_end)
   r ors rtab(1) len(1) . c :f(freturn)
    s (span(c) | null) . f =
    s arb . s1 len(1) . d c = :f(r_1)
    r = s1 f c d s :(return)
   r_1 ors break(c) . os
    r = r(s,os) f :s(return)f(freturn)
   r_end

    s = 'abcdefgh' 
    output = s
   x01 output = r(output,s) :s(x01)
   end

"... is silly, there are better ways."

I see this usage often where it seems like

  grep pattern1 | grep -v pattern2 

  sed -n '/pattern1/d;/pattern2/p'

  sed '/pattern1/!d' | sed '/pattern2/d'

  sed -n 's/pattern1pattern2//g;/pattern2/p'

For example look at the "grep -v" usage here:

https://github.com/thomwiggers/qhasm/raw/master/qhasm-arm

Is there something wrong with using

   sed '/^op:livefloat80:/d'

  sed 's/\$/#/g'

  tr '$' '#'

Example line from the Openshift reload-haproxy script:

  old_pids=$(ps -A -opid,args | grep haproxy | egrep -v -e 'grep|reload-haproxy' | awk '{print $1}' | tr '\n' ' ')

  old_pids=$(exec ps -A -opid,args |sed -n '/sed/d;/reload-haproxy/d;/haproxy/{s/ .*//;p};'|tr '\n' ' ')

Is it true these packets are HTTP requests full of XML, i.e., SOAP? Do they use HTTPS on ports 16994 and 16995?

To avoid a crash, users can mount potentially malicious filesystems in userspace, i.e. users can run kernel drivers like ffs outside of the kernel. This feature comes from a non-Linux kernel. I have read this may be able to work on Linux too but I have never tried it.

First time I learned about SETL at NYU was when I was learning about Spitbol.

Here is a fizzbuzz in Spitbol. Probably not the best way to do it.

      var a = 0;
      start
      +a = a + 1;
      break
      +lt(a,101) :f(end);
      x01
      +x = (remdr(a,3) + remdr(a,5)); eq(x,0) :s(x06);
      x02
      +y = remdr(a,3); eq(y,0) :s(x04);
      x03
      +z = remdr(a,5); eq(z,0) :s(x05)f(x07);
      x04
      +output = 'fizz' :(start);
      x05
      +output = 'buzz' :(start);
      x06
      +output = 'fizzbuzz' :(start);
      x07
      +output = a :(start);
      end
      

He also has pages on APL and k.

The knee-jerk answer upon seeing the title was: "Too many".

So many of these instructions I never see anyone use, ever.

No doubt there are some folks using them, but as mere mortals, do the rest of us really need all these features to control our small amount of commodity hardware? As a user, I have modest goals. Is it not true that Torvalds wrote his kernel with a similarly modest goal in mind: control over his own commodity computer?

The situation resembles that of an overcomplex software program where a majority of the features are unused by an even larger majority of its users. In other words the depth of features benefit only the very few people who use them.

Given the choice between several alternatives with differing levels of features I tend to opt for software that is less featureful and hence more simple. Call me simple-minded if you wish. The same goes for processors, although when it comes to hardware how much choice to we really have as end users? (Hobbyist boards excluded.)

For a taste of some non-x86 assmbler, I enjoyed experimenting with a MIPS simulator still found at spimsimulator.sourceforge.net. I can report that the non-GUI portion at least still compiles relatively cleanly on BSD. This simulator has been mentioned on HN several times.

I have a no problem with using a processor with fewer instructions even if I have to sacrafice something by making that choice -- I leave it to the experts to detail those sacrafices and why I would be a fool to make them. NB: I am already a fool so it may not be worth the effort.

How many HN readers have tried RISC-V? A poll for those who have not: will RISC-V inspire you to purchase a new computer?

https://mobile.twitter.com/hashbreaker/status/85322416941220...

The strange "counterargument" I commonly see on HN to any suggestion that Microsoft closed source software could potentially be unsafe for use on an internet-connected computer is that the company has "improved" since some earlier 1990's/2000's time period.

Are these commenters suggesting that other, open source operating system choices have not also improved since that time period? Should one consider how much did each respective system need to improve?

(By "other, open source operating system choices", I mean the ones that were able to connect to the internet for years before Gates decided the www was something his company should be interested in and to copy the TCP/IP stack from an open source kernel into the Windows kernel).

Are there convincing arguments why Microsoft deserves special treatment compared to the open source alternatives, i.e., why their users should not be permitted to freely evaluate the Windows kernel or Office source code via the public web? Are there compelling reasons why MS users should not be allowed i.e. given the option to edit/remove source code they are uncomfortable with and recompile? Consider the effects of limiting the number of people who can find and fix defects in a product.

Does closed source status of Windows make Microsoft's software superior to the longstanding open source operating system alternatives?

https://www.symantec.com/connect/tr/blogs/symantec-ca-propos...

Works for me without Javascript.

"UUCP can be setup rather easily with few configuration files and few lines in each of them. But you have to add some encryption and authentication overlay for securing you data transmission."

Any projects that attempted to do this sort of overlay for UUCP?

Incidentally, I like the way the author presents this project. Usage examples, thoughtful use cases, comparisons, protocol description and packet formats.

No hype.

https://github.com/skarnet/execline/raw/master/src/execline/...

https://github.com/skarnet/execline/raw/master/src/libexecli...

Simple.

http://www.in-ulm.de/~mascheck/various/argmax/

   execlineb -c 'elglob a /*/*/*/* ls $a'

If I am not mistken, ARG_MAX will be the limit.

Straightforward.

Can an "empty" variable have a length greater than zero?

In Bourne once scripts I have been testing

   test ${#var} -gt 0 and

   test ${#var} -ge 1

   test x"" = x$var

Netty was first released in 2016?

HAproxy was first released around 2001.

Probably irrelevant to the question of speed, but there is a comment in another thread about hype driven development on HN first page right now where a commenter states they prefer Netty to the alternatives apparently because the alternatives are older or more cumbersome to use, although I may have misread.

Edit: This was a hasty, dumb comment. Please accept my apologies. Netty is not new and I should have known better. For whatever irrational reason, I have a bias against Java and deliberately avoid it. I do know it helps professional programmers get things done easier and faster. I'm an HAproxy user and have probably developed an HAproxy bias.

Text-only browser shows the IDN, not the phished domain.

</sarcasm>I guess I need to "upgrade to a modern browser" for websites to work correctly?</sarcasm>

As an aside, I still do not understand how "modern" browsers evolved to hiding portions of the URL or using a phony address bar i.e. "omnibox" to the right of the real address bar.

In the first case, it seems to offer no benefit other than to hide important details.

In the second case, it seems so overtly deceptive for newcomers to the www that I am surprised they could pull it off.

Maybe these things have changed recently as these monster programs are constantly changing. If so, pardon my ignorance.

Is it not true that users who do not understand the basics of www usage e.g., what is a domain, a URL, etc. are always going to be at risk of manipulation?

"Thinking like a computer" would not involve abstraction. That would be thinking like a language author or a young programmer.

"Thinking like a computer" means thinking without abstraction. (Aside from symbols and maths.)

By learning abstraction the student is only learning how a language author thinks, not a computer. There is no need to teach abstraction. It comes naturally. Humans do not operate like computers. Humans naturally gravtitate to using symbols.

Why not teach how to convert from binary to hex to decimal and how to do binary addition? Every child has to memorize a decimal multiplication table at some stage of their education. They should be memorizing binary/hex tables too.

The first of their three nameservers is non-responsive for at least some users. There is no round-robin.

ns-a.sdf.org 205.166.94.20

This will obviously slow down lookups for those users. If this is intentional, I would be curious why they are doing this.

I do not have "block ads". I simply do not request them.

Users do not intentionally make requests for ads or pixels from tracking servers. Browsers do. Automatically.

People writing web pages that aim to cash in on advertising budgets depend on this "feature". However it is optional. I read hundreds of web pages and never see any ads. Because for eading the news I do not use a so-called "modern" browser.

It seems the entire web ad industry requires browsers to operate a certain way. If browsers do not follow these assumptions, then the user sees no ads.

Despite strange notions like the one in the top-voted comment in this thread, there is nothing that requires any user to use browsers written by people whose salaries are paid directly or indirectly from ad sales revenue.

Assuming certain companies were as all-powerful as the commenter suggests, then why not require users to access pages using software written by companies who profit from such web traffic? And make the software proprietary?

Surely no one would complain. Thank you sir, may I have another?

Let us not forget some of these "multi-billion dollar entities" are just websites. If the traffic dies down, the business of selling ads is no longer feasible. And the company disappears along with the website. It has happened before.

98% of revenue from web traffic/ad sales.

Castles made of sand.

FYI the browser I use does not download resources automatically. This way I just get the page with the content: the text of the article. That is all I want.

To read articles, which are just text, I access the web in VGA textmode with a text-only browser. I never see any ads.

There is no such thing as "load time" with this "UX". All pages load fast.

I do DNS lookups in bulk and store IPs before doing any HTTP requests, so I take time consumed by DNS out of the equation. I do not store IPs of ad servers, Facebook domains serving hidden beacon images and other nonsense. I never access these resources.

If I want to look at images or watch videos, I download them and use other programs running on other computers with graphics to view them. Those computers have no gateway to the internet.

For someone who likes to read, it all works very well. But if I was forced to use a "stock" browser of some organization's choosing (Google, MS, Apple, Mozilla, etc.), then perhaps I would have a very different "user experience" - heaps of ads. I suspect this is because those organizations place advertisers before users. If the ad sales business were to dry up it would be a disaster for some of those organizations.

Fortunately, I do not need those browsers to read the news.

"This is a screenshot from an article on NPR discussing the rising use of ad blockers. It is 12 megabytes in size..."

Hmmm. When I save this page from my browser it's 116k.

"... in a stock web browser."

Ah-ha. Perhaps the web browser I am using is not "stock".

If so, I can still read the article without a "stock" browser, so not sure what is behind this "stock" idea.

I am assuming reading the article is the goal?

Could it be the "stock" browser, whatever that is, is the root cause/enabler of the "web page obesity crisis"?

"Unfortunately complexity has become a bit of a bragging point."

Does that include "stock" web browsers?

OT.

Symantec: Why require Javascript? Why not make it optional?

For users who do not use Javascript to read news: https://www.symantec.com/connect/tr/blogs/longhorn-tools-use...