Removing duplicate lines from files keeping the original order with Awk

https://iridakos.com/how-to/2019/05/16/remove-duplicate-lines-preserving-order-linux.html

by laz_arus • 7 years ago

375 154 7 years ago

iridakos.com

founderling2• 7 years ago

Awk ' visited[$0]++' or how badly HNs automatic title formatter messes up awk commands :)

Hint: You can edit it after you posted.

laz_arus4• 7 years ago

Done, thanks for the hint :)

founderling2• 7 years ago

You are !done yet.

aidos• 7 years ago

This is a nice little run through of a real life example.

Once you realise that awk's model is 'match pattern { do actions; }' everything makes a whole lot more sense.

sohkamyung• 7 years ago

Awk also supports BEGIN and END actions that take place at the start and at the end of execution.

BEGIN might be used to initialise Awk variables or print initial messages, while END can be used to print a summary of actions at the end.

See [1] https://www.grymoire.com/Unix/Awk.html#uh-1

RBerenguel5• 7 years ago

You can also use either BEGIN or END as the only entrypoints, essentially using AWK as you'd use any other programming language. Yes, sometimes this defeats the point of what AWK excels at, but it's good to know.

mehrdadn6• 7 years ago

Where I run into trouble with awk is gawk incompatibilities with the implementation on Mac. The gawk manual really sucks at telling you what exactly is an extension to the language, and I haven't been able to find a good source -- you just have to either guess and check, or cross-check against other ones' manuals (like BSD). Otherwise it's an amazing tool...

asicsp8• 7 years ago

I thought the gawk book/documentation [1] did a good job of mentioning differences between various implementations, do you have an example?

You might find this [2] helpful (oops, seems like it got deleted, see [3] - thanks @bionoid)

[1] https://www.gnu.org/software/gawk/manual/gawk.html

[2] https://www.reddit.com/r/awk/comments/4omosp/differences_bet...

[3] https://archive.is/btGky

bionoid• 7 years ago

[2] is available here:

https://www.removeddit.com/r/awk/comments/4omosp/differences...

Archive for posterity: http://archive.is/btGky

mehrdadn6• 7 years ago

do you have an example?

Sure, try this:

  echo 1 2 | awk '{ print gensub(/1/, "3", "g", $1); }'

The logical thing for them to do would be to mention in bold and/or big and/or red font under gensub's documentation that it's an extension (e.g. try nawk), whereas looking through it I don't see any mention at all: https://www.gnu.org/software/gawk/manual/html_node/String-Fu...

If I may rant about this for a bit, GNU software manuals are generally rather awful (though they're neither alone in this nor is it impossible to find exceptions). They frequently make absolutely zero effort to display important information more prominently and unimportant information less so (if you're even lucky enough that they tell you the important information in the first place). Like if passing --food will accidentally blow up a nuke in your hometown, you can expect that if they documented it at all, they just casually buried it in the middle of some random paragraph. Their operating assumption seems to be that if you can't be bothered to spend the next 4 hours reading a novel before writing your one-liner then it's just obviously your fault for sucking so much.

avar2• 7 years ago

You should be looking at the POSIX specification, and assuming anything GNU awk documents on top of that is an extension: http://pubs.opengroup.org/onlinepubs/009695399/utilities/awk...

mehrdadn6• 7 years ago

Honestly that page isn't great at showing up in search results, but in any case -- POSIX can be too restrictive. I don't specifically recall an example for awk, but when every implementation supports something then it's a de-facto standard. If you follow POSIX then you become over-restricted compared to any shell you'll actually encounter. It'd be really nice if there was a page that showed the actual common features between various implementations of POSIX tools, not just the POSIX official features...

rwbhn• 7 years ago

brew install gawk

mehrdadn6• 7 years ago

Yes, but I don't control every computer my code runs on...

akx• 7 years ago

I'd suggest just installing `gawk` from Homebrew and using it instead of having to guess which sort of crippled `awk` you're lucky to be using.

The same thing stands for GNU coreutils (`brew install coreutils`); here's macOS `cut` vs GNU `cut` as a quick example.

    ~ $ cut --help
    cut: illegal option -- -
    usage: cut -b list [-n] [file ...]
           cut -c list [file ...]
           cut -f list [-s] [-d delim] [file ...]    

    ~ $ gcut --help
    Usage: gcut OPTION... [FILE]...
    Print selected parts of lines from each FILE to standard output.    

    With no FILE, or when FILE is -, read standard input.    

    Mandatory arguments to long options are mandatory for short options too.
      -b, --bytes=LIST        select only these bytes
      -c, --characters=LIST   select only these characters
      -d, --delimiter=DELIM   use DELIM instead of TAB for field delimiter
      -f, --fields=LIST       select only these fields...
      -n                      (ignored)
          --complement        complement the set of selected bytes, characters or fields
      -s, --only-delimited    do not print lines not containing delimiters
          --output-delimiter=STRING  use STRING as the output delimiter
                                the default is to use the input delimiter
      -z, --zero-terminated    line delimiter is NUL, not newline
          --help     display this help and exit
          --version  output version information and exit

fanf22• 7 years ago

Looking at the Mac OS X source code https://opensource.apple.com/source/awk/awk-24/src/ its version of awk is the “one true awk” maintained by Brian Kernighan. For some reason Apple have deleted the README but you can find a copy at https://svnweb.freebsd.org/base/head/contrib/one-true-awk/

RBerenguel5• 7 years ago

I was there as well, and eventually decided to just always `brew install gawk` and alias `awk` to `gawk`, because more often than not I want to rely on gawk extensions (gawk has includes, for instance!).

To be fair though, every time I have read the manual for a gawk function it clearly says "this is a gawk extension" for non-standard implementations (case in point, delete[1], which is now POSIX, although Mac's awk is too old to have that implemented).

[1] https://www.gnu.org/software/gawk/manual/html_node/Delete.ht...

asicsp8• 7 years ago

I have a collection of such one-liners, for duplicates including how to form key for multiple fields, see [1]

[1] https://github.com/learnbyexample/Command-line-text-processi...

laz_arus4• 7 years ago

This repo is awesome, great work.

seriousaccount• 7 years ago

Wow! Thanks for this! This is exactly what I was looking for :)

asauce• 7 years ago

Wow this is great. I want to become more competent with text processing in the command line and this looks like a great place to start.

Thanks for linking this repo!

rusk3• 7 years ago

Got to love awk. My weapon of choice for ad hoc arbitrary text processing and data analysis. I’ve tried to replace it with more modern tools time and again but nothing else really comes close in that domain.

RBerenguel5• 7 years ago

Completely agree. Even Python, which has a very low barrier to entry to "read file, possibly csv, do something" has a barrier to entry. Column projections are one-liners in AWK, and aggregates and/or some stats can be a couple of lines in an AWK script proper.

I've been replacing some ad-hoc bash scripts (nothing fancy, just a few if conditions and some formatting of outputs for a deployment) with some AWK, and it's so much handier to write (after 10 years I still can't remember if syntax) and read (it's a proper programming language) than bash

edit: wrong markdown style

vidarh• 7 years ago

Interesting Ruby (MRI anyway) has command line options to make it act pretty similar to awk:

-n adds an implicit "while gets ... end" loop. "-p" does the same but prints the contents of $_ at the end. "-e" lets you put an expression on the command line. "-F" specified the field separator like for awk. "-a" turns on auto-split mode when you use it with -n or -p, which basically adds an implicit "$F = $_.split to the while gets .. end loops.

So "ruby -[p or n]a -F[some separator] -e ' [expression gets run once every loop]'" is good for tasks that are suitable for "awk-like" processing but where you may need access to other functionality than what awk provides..

rusk3• 7 years ago

Thanks, I actually looked at trying to make ruby do my awk work a few years ago. I'll take a look again.

asicsp8• 7 years ago

I'd say it is more similar to perl than awk for options like -F -l -a -n -e -0 etc. And perl borrowed stuff from sed, awk, etc

I have a collection for ruby one-liners too [1]

[1] https://github.com/learnbyexample/Command-line-text-processi...

vram222• 7 years ago

You probably know it, but in case not, and for others who might not know: Ruby was influenced by Perl, and Perl was influenced by awk. (Both Ruby and Perl were influenced by other languages too.) And (relevant to this thread) Perl was influenced by C, sed, and Unix shell too.

https://en.wikipedia.org/wiki/Ruby_(programming_language)

https://en.wikipedia.org/wiki/Perl

See "Influenced by" sections at both above pages.

wernsey• 7 years ago

Seconded.

Just the other day I helped some colleagues clean up a text file using an awk one-liner. It seemed like magic to them.

(Even though the one-liner turned out to be a bit more difficult to write than I thought at first due to '\r' characters in the input file)

vram222• 7 years ago

(Even though the one-liner turned out to be a bit more difficult to write than I thought at first due to '\r' characters in the input file)

Although you solved it, another way is that one can always pipe the input through a filter like dos2unix first. Very easy to write and versions can be found/written for/in many languages. Essentially, you just have to read each character from stdin and write it to stdout, unless it is a '\r', a.k.a. Carriage Return a.k.a. ASCII character 13, in which case you don't write it.

I've often found that beginners these days don't know what carriage return, line feed, etc. are, and their ASCII codes. Basic but important stuff for text processing.

https://www.google.com/search?q=ascii+table

https://en.wikipedia.org/wiki/Carriage_return

omh2• 7 years ago

Awk is wonderful. It's an odd way to write programs, but for quick one-off processing tasks it almost can't be beaten.

Somewhat related blog post which I like to refer people to: "Command-line Tools can be 235x Faster than your Hadoop Cluster" https://adamdrake.com/command-line-tools-can-be-235x-faster-...

chasil8• 7 years ago

Why do you find it odd? I find it to be the very best introduction to C-style control structures.

Chapter 2 of The AWK Programming Language has incredible benefits for a novice.

https://archive.org/download/pdfy-MgN0H1joIoDVoIC7/The_AWK_P...

jolmg2• 7 years ago

They probably refer to how it has the top-level as

  <line-condition> { <code> }

That's unusual enough among programming languages to call it odd. Being able to do stuff like

  if (/some-pattern/) { ...

and have the regex be evaluated like a condition where it matches with the current line implicitly is also pretty unique.

sansnomme• 7 years ago

Are there any pipeline tools for command line stream processing? Because when you have several terabytes of data you can't exactly afford to restart due to a stray comma in your CSV file.

creatornator2• 7 years ago

If you have a stray comma in your multi-TB CSV file, you probably don't _want_ it to keep going. You risk misinterpreting the mistake and having a grossly malformed output... There's no way to reliably and elegantly recover from something like that. Validation should preferably happen before processing

sdegutis• 7 years ago

Article's like OPs about removing duplicate lines make me want to learn awk.

But it feels like it would be a net loss based on how seldom I currently need to write one-off scripts.

Based on experience, I'd probably have a perfect use-case for it every 2-3 years.

sametmax2• 7 years ago

Even if you do, it's a better investment to learn a general high level language with strong scripting capabilities, but that is also good at many things else.

Sure, the days you'll need awk, you'll take 15 minutes instead of 2 writing your script. So what ?

But the rest of the year, you'll have a more versatile toolbox at your disposal for automatic things, testing, prototype network processes, make quick web sites or API, and explore data sets.

That being said, I can see the point of learning awk because, well, it's fun.

tehlike• 7 years ago

Or if you know you can achieve some tasks with one of these tools, you might suddenly realize more tasks than you imagined might be solved with them.

dancek• 7 years ago

Do you consider learning a tool you won't use a waste of time?

IMHO knowing what kinds of tools exist and how they are used for different tasks is enormously useful. Most software projects require me to create a set of tools to solve problems in a certain space efficiently. In any long-living non-trivial project there will be feature requests you couldn't have anticipated in the beginning. They tend to be painful if your program is just a bunch of features hacked together. But if you take a tools-first approach, the unexpected features can often be solved with what you have.

Of course, time is limited and you can't learn everything. But learning one of every different kind of tool is a very good use of time.

EDIT: Note that I'm not claiming you'll build a web app with awk. I'm saying you might write code that can be used similarly to awk in some abstract sense, and that might be a core part of a web app.

augustk2• 7 years ago

And here is the ungolfed version:

awk '{ if (! visited[$0]) { print $0; visited[$0] = 1 } }'

zufallsheld• 7 years ago

Much more readable and understandable.

RBerenguel5• 7 years ago

You can always write AWK in a file and read the script with -f, making it fully readable (and AWK is quite a pleasantly readable and surprisingly versatile language to write at that point)

augustk2• 7 years ago

Or maybe better:

awk '! visited[$0] { print $0; visited[$0] = 1 }'

asicsp8• 7 years ago

yet another version:

    awk '!($0 in seen); {seen[$0]}'

mtzet• 7 years ago

Yes. Examples like the one linked make it appear that awk is write-only language, which it really isn't.

'awk' is really a very beautiful little language. It's concise enough to solve many tasks in a single line, making it easy to use interactively while still being able to grow to moderately-sized scripts. It's not supposed to replace a full-blown scripting language like Python, but for processing files line-by-line it's superb.

hk__23• 7 years ago

The tradeoff of this solution is it stores all (unique) lines in memory. If you have a large file with a lot of unique lines you might prefer using `sort -u`, although it doesn’t keep the order.

omh2• 7 years ago

Doesn't sort require keeping lines in memory as well? In fact, doesn't sort keep all lines in memory, whereas this awk solution just keeps the unique lines.

ot2• 7 years ago

No, most implementations of sort use a bounded memory buffer and an external-memory algorithm, spilling to disk.

ot2• 7 years ago

This is mentioned in the article, together with a method to keep the order by decorating with the line number before sorting.

hk__23• 7 years ago

The sort/uniq tradeoff is mentioned but not the awk one.

triangleman2• 7 years ago

Are you sure this is the case? Perhaps it's only storing the hash of the lines? If not then how do you do that?

IME this one-liner can churn through 100MB of log lines in a second. Other solutions like powershell's "select-object -unique" totally choke on the file.

hk__23• 7 years ago

It can’t just store the hash of the lines otherwise it would drop lines in case of hash collision.

chasil8• 7 years ago

The AWK program:

awk '{a[$0]++}; END{for(b in a) print b, a[b]}'

...will print every unique line in a file with the count. Obviously, that could not be done if the array index was a hash - the array index is the entire line, and the array value is the count.

The original program moves the maintenance of the array into the implicit conditional "pattern," and only prints when the array entry does not yet exist.

kazinator5• 7 years ago

Even if sort overcomes the memory limitation by using disk space, it's O(n log n), whereas de-duplication through a hash is O(n).

You might be better off switching to another scripting language that has some database API for storing key/value pairs on disk.

Or: use a 64 bit machine for a bigger address space, and add temporary swap files so you have more virtual memory.

0815test2• 7 years ago

Or: use a 64 bit machine for a bigger address space

There are people who are still on 32-bit hardware for serious work?

(Even if I was still on that, I'd probably just fire up a RV64 virtual machine (with swap space added within the VM, of course) simply to access the convenience of a larger address space when needed.)

creatornator2• 7 years ago

Sorting is O(n log(n)) but you still have to make a second pass at the end to remove duplicates, making it O(n), isn't it?

lifeisstillgood3• 7 years ago

There was a twitter thread ages ago where someone had written a collection of (php?) utilities - and the twitterer posted a laughing slap down saying why write a utility when this one liner and that one liner will do?

There was a lot of push back - and this article is a good example of why

If I wanted to remove duplicate lines from a file I would almost certainly not use awk

I have never spent the time to get good enough with the whole new and different languge of awk, and am unlikely to need to (my large scale file processing needs seem small, and if I do it's almost always in context of other processing chains - so a normal languge like python would be the natural choice

I could whip up something like this in python in a less time than it would take to google the answer, read up why the syntax works that way and verify I have not mistyped anything on a few test files.

Basically using awk takes me out of my comfort zone - for a one off task it loses me time, for a production like repeat task I am going to reach for a slew of other solutions.

I mean the title of this page loses the exclamation mark - and it took me two goes to spot it.

asicsp8• 7 years ago

I don't get your point, it seems like you do not often use cli text processing tools

Just like Python, there are users who use cli and are comfortable using grep/sed/awk/sort/etc

nerdponx5• 7 years ago

The point is that the people who do use such tools tend to have a derisive attitude towards those who don't, and that the derisiveness is completely unwarranted.

ashelmire• 7 years ago

I don’t think you need to learn every cli tool in depth. I don’t know awk, but I recognize its power. However, I’ve got a set of tools I find easier to use, a few that I’ve written, and Ruby or Python at the command line. I’m sure awk could replace many cli tools, but at the cost of learning a new language (and cognitive load with each use) it hasn’t been worth it.

lifeisstillgood3• 7 years ago

Thank you for putting on three sentences my multi-paragraphs :-)

oftenwrong4• 7 years ago

For many years my AWK knowledge was limited to basic '{print $1}' style usage. I never bothered to learn more. I tended to use perl when I needed a custom text-processing operation. Later, as perl became less a part of my working life, I began using ruby instead - they are pretty similar in spirit.

One day, nearly 20 years after it was published, I picked up a used copy of The AWK Programming Language by Aho, Kernighan, and Weinberger. Yes, they are credited in that order on the cover... I suspect intentionally. I only read the first N chapters, but it was enough. I used AWK many times within the following month, and I continue to use AWK on a daily basis. When the task is complicated, I will still use ruby, but often enough AWK is easier.

The point: you think "Why would I learn X when I can use Y?", but you won't really know the answer until you learn X. If I had never learned perl, python, ruby, AWK, shell script, vi macros, then I would probably be editing files by hand (!) like I sometimes catch developers actually doing (!!!). For a person who doesn't know these tools, that might actually be the path of least resistance. Investing some time here and there to learn new tools pays off in the future in ways that are unpredictable.

jacobolus2• 7 years ago

The basic imperative Python version is much easier to remember and read though, even for not-that-experienced Python programmers. I would expect laypeople to be able to more-or-less figure out what it is supposed to do.

  seen = set()
  with open(filename, "r") as file:
    for line in file:
      if line not in seen:
        print(line)
        seen.add(line)

Often (at least in my experience) this kind of operation is either (a) part of some larger automated data processing pipeline for which it’s really nice to have version control, tests, ... or (b) part of some interactive data exploration by a programmer sitting at a repl somewhere, not just a one-off action we want to apply to one file from the command line.

In those contexts, the Python (or Ruby or Clojure or whatever general-purpose programming language) version is easy to type out more-or-less bug-free from memory, debug when it fails, slot into the rest of the project, modify as part of a team with varied experience, etc. etc.

reacweb2• 7 years ago

I have taken as rule to use awk only for trivial tasks and to switch to perl as soon as the syntax is slightly beyond my usual use cases. In perl, I would do:

    perl -nle 'print unless exists $h{$_};$h{$_}++' < your_file

asicsp8• 7 years ago

perl borrowed stuff from awk, so you could also do

    perl -ne 'print if !$seen{$_}++'

rlonstein• 7 years ago

Play a round of perl golf?

    perl -pne '$_=$#$_++?$_:""'

I'm rusty at this but shaved off six chars, five if you count the 'p' added to switches.

chasil8• 7 years ago

There are many AWK implementations that are obscenely portable. A case in point is busybox AWK. Another might be http://unxutils.sourceforge.net

A complex AWK script is very, very easy to move to a platform that lacks any AWK parsers. It can easily be done on Windows, without administrative rights, by the placement of a single .EXE - Perl can do many things, but that is not one of them (to the best of my knowledge).

triangleman2• 7 years ago

So then, what is the one liner to preserve the filename rather than get a new deduped.txt?

Also how do you apply that command to the next file using shell history?

gcmeplz• 7 years ago

Use sponge! https://linux.die.net/man/1/sponge

  awk '!n[$0]++' fileName | sponge fileName

Vogtinator• 7 years ago

Or using gawk, gawk -i inplace 'foo {bar}' file

dredmorbius3• 7 years ago

    dedupe.awk <file >file.tmp && mv file.tmp file

Multifile versions vary, I'd prefer listing them out, alternatively you could read from a command output (ls, find, etc.) with a 'while read; do ... done' loop:

    for f in file1 file2 file3
    do
        dedupe.awk <$f > ${f}.tmp && mv ${f}.tmp $f
    done

If you want to apply to specific files on an ad hoc basis, you could wrap the whole thing in a shell function with filename or list as a parameter.

Or 'gawk -i' as suggested.

Properly using tempfile would also be an improvement.

julienfr1122• 7 years ago

I was wondering how awk work internally. Does it compile the script then run it ? Is it bytecode or lower level ? A finite state machine ?

CodeArtisan• 7 years ago

With GNU awk, scripts are compiled to bytecode then interpreted with a big switch/case loop.

edit: https://git.savannah.gnu.org/cgit/gawk.git/tree/interpret.h

chasil8• 7 years ago

The mawk version of the language will output C, if I remember correctly. It is the fastest AWK, and supports some of the GNU extensions.

"The One True AWK" from Brian Kernighan (that is still the system AWK in OpenBSD) switched from a yacc implementation to a custom parser sometime within the last decade (fairly recently).

Busybox also has an awk; I'm not sure what they do.

GNU awk is elsewhere reported to be interpreted bytecode.

snaky• 7 years ago

Dennis Ritchie about yacc history details

In some ways the interesting thing is that the parser (probably for B, couldn't have been C based on radiocarbon dating evidence) was tiny and dead simple using recursive descent for most parts, a precedence table for expressions. But out of the intellectual culture-meets-culture encounter, an enduring tool was created.

https://yarchive.net/comp/handwritten_parse_tables.html

fanf22• 7 years ago

Looks like OpenBSD awk still uses yacc, like other copies of bwk‘s one true awk https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/awk/#d...

benhoyt• 7 years ago

They parse the script to a parse tree (abstract syntax tree) and then either interpret that directly (tree-walking interpreter) or compile to bytecode and then execute that. The original awk ("one true awk") uses a simple tree-walking interpreter, as does my own GoAWK implementation. gawk and mawk are slightly faster and compile to bytecode first.

If you're interested, you can read more about how GoAWK works and performs here: https://benhoyt.com/writings/goawk/

srean• 7 years ago

Did gawk change to bytecode interpreter recently ? As far as I can recall it used to be an AST walker. Wonder if I am misremembering things. Mawk used tobe more than 'slightly' faster than gawk. Maybe the recent change to bytecode have brought their perf characteristics closer.

julienfr1122• 7 years ago

Nice work !

bigato2• 7 years ago

This is very memory intensive. Which may not matter if the data volume is small enough. But it is also a bit hard to understand, at least not so obvious at first sight. For most use cases sort -u would be ideal and way simpler to understand, if you don't mind having an ordered file at output.

beefsack• 7 years ago

`sort` would also be memory intensive would it not?

chasil8• 7 years ago

No, sort will use intermediate temporary files instead of exhausting your ram.

mitnk2• 7 years ago

This is very memory intensive.

Only for ones not familiar with awk.

It would make a lot of sense after you understand how awk works (as the article explains).

anc84• 7 years ago

That does not make any sense. If it is memory intensive depends on awk, not on the person being familiar with it.

Sois it memory intensive or not?

Scriptor• 7 years ago

I think they're talking about your machine's memory, not human memory.

zimpenfish2• 7 years ago

if you don't mind having an ordered file at output.

I needed to remove duplicates from a sequenced CSV file yesterday but couldn't figure out the flags for "remove duplicates, output sorted by field 1 asciily, 2 numerically, 3 numerically".

The AWK version worked perfectly.

bigato2• 7 years ago

The awk version does not order it. If your output was ordered the way you wanted, it was because the input already was.

mitnk2• 7 years ago

The man page of (n)awk [0][1] is surprisingly short and readable.

[0] `man awk` on mac

[1] online version https://www.mankier.com/1/nawk

[2] gawk's man page works great as a reference https://www.mankier.com/1/gawk

hjk05• 7 years ago

On a previous post people were complaining that math wasn’t as clear as code. I’d argue that this this is exactly the kind of code-like clarity math notation provides you. I makes perfect sense, but only after 2 full pages describing what’s going on in one line.

oftenwrong4• 7 years ago

See also, 'nauniq' (non-adjacent uniq), an implementation of this text-processing task as a full utility, with some convenient options for reducing memory usage:

https://metacpan.org/pod/distribution/App-nauniq/script/naun...

kazinator5• 7 years ago

You need "gawk -M" for this for bignum support, so visited[$0]++ doesn't wrap back to zero, otherwise it is not correct for huge files with huge numbers of duplicates.

The portable one-liner that doesn't suffer from integer wraparound is actually

   awk '!($0 in seen) { seen[$0]; print }'

which can be golfed a bit:

   awk '!($0 in s); s[$0]'

$0 in s tests whether the line exists in the s[] assoc array. We negate that, so we print if it doesn't exist.

Then we unconditionally execute s[$0]. This has an undefined value that behaves like Boolean false. In awk if we mention an array location, it materializes, so this has the effect that "$0 in s" is now true, though s[$0] continues to have an undefined value.

zimpenfish2• 7 years ago

huge files with huge numbers of duplicates

At least on the stock MacOS awk, you can get up to 2^53 before arithmetic breaks (doesn't wrap, just doesn't go up any more which means the one-liner still works.)

    > echo '2^53-1' | bc
    9007199254740991
    > seq 1 10 | awk 'BEGIN{a[123]=9007199254740991;b=a[123]}{a[123]++}END{print a[123],b,a[123]-b}'
    9007199254740992 9007199254740991 1

Even with one character per line, you'd need an 18PB file before you got to this limit, afaict.

jancsika4• 7 years ago

Me: Wow, that associative array looks very powerful. Is there a way to leverage it to do something useful like convert curl-obtained JSON array of file patches from Github's API to the mbox format that `git am` expects?

Unix: No, that JSON data is too structured. But if you have a more error-prone format like CSV I can show you a neat trick to filter your bowlers by number of spares.

james_s_tayler4• 7 years ago

can't u just process JSON data with jq?

https://stedolan.github.io/jq/

jancsika4• 7 years ago

I'd still need to find a way to massage the data into the mbox format because that is the ancient format that git understands.

I'm not saying that there isn't a way to do that. Only that it can only be done poorly with a big ugly (and probably buggy) spaghetti script that looks nothing like what the expressive demo suggests it should look like.

0815test2• 7 years ago

Luckily for us, Unix is easy to enhance by adding new "skills" to it - a quick search brought me to https://ilya-sher.org/2018/04/10/list-of-json-tools-for-comm...

PyroLagus• 7 years ago

Well, there's jq, which is perfect for that job, but unfortunately it's not a standard utility, and I'm not sure it ever will be, because we have to stick only with stuff invented in the 90's.

lazyjones• 7 years ago

https://github.com/dominictarr/JSON.sh

https://github.com/step-/JSON.awk

JdeBP2• 7 years ago

You have conflated the awk tool with an entire operating system. The operating system of course has many tools, and there are many more tools that one can further add to it, dealing with a range of file formats.

ori_b• 7 years ago

Gron converts json into something processable.

pvaldes• 7 years ago

open the file with emacs

menu edit -> select all

press 'escape' and 'x' keys together and write: delete-duplicate-lines

done

aabbcc1241• 7 years ago

If I know this, I wouldn't make https://github.com/beenotung/uniqcp

Removing duplicate lines from files keeping the original order with Awk

Related Stories