HN user

m0shen

235 karma

http://moshen.net

Posts1
Comments141
View on HN
medium.com 6y ago

Are We Heading to Digital Disenfranchisement of Indigenous Peoples?

m0shen
1pts0
ht: Headless Terminal 2 years ago

I see what you're saying. When I was writing scripts in `expect`, I didn't really ever try to automate tui programs. So, this could absolutely be a better way to script the the terminal as a ui as you said.

I'll certainly tuck it into my toolbox. Thanks :)

0ThreadHN
Magika: AI powered fast and efficient file type identification 2 years ago

I've updated this script with some single-file cli numbers, which are (as expected) not good. Mostly just comparing python startup time for that.

    make
    sqlite3 < analyze.sql
    file_avg              python_avg         python_x_times_slower_single_cli
    --------------------  -----------------  --------------------------------
    0.000874874856301821  0.179884610224334  205.611818568799
    file_avg            python_avg     python_x_times_slower_bulk_cli
    ------------------  -------------  ------------------------------
    0.0231715865881818  0.69613745142  30.0427184289163
0ThreadHN
Magika: AI powered fast and efficient file type identification 2 years ago

Absolutely, and honestly in a non-interactive ingestion workflow you're probably doing multiple checks anyway. I've worked with systems that call multiple libraries and hand-coded validation for each incoming file.

Maybe it's my general malaise, or disillusionment with the software industry, but when I wrote that I was really just expecting more.

0ThreadHN
Magika: AI powered fast and efficient file type identification 2 years ago

Good to know! Thank you. I'll definitely be trying it out. Though, I might download and hardcode the model ;)

I also appreciate the use of ONNX here, as I'm already thinking about using another version of the runtime.

Do you think you'll open source your F1 benchmark?

0ThreadHN
Magika: AI powered fast and efficient file type identification 2 years ago

My little script is trying to identify in bulk, at least by passing 165 file paths to `magika`, and `file`.

Though, I absolutely agree with you. I think realistically it's better to do this kind of thing in a library rather than shell out to it at all. I was just trying to get an idea on how it generally compares.

Another note, I was trying to be generous to `magicka` here because when it's single file identification, it's about 160-180ms on my machine vs <1ms for `file`. I realize that's going to be quite a bit of python startup in that number, which is why I didn't go with it when pushing that benchmark up earlier. I'll probably push an update to that gist to include the single file benchmark as well.

0ThreadHN
Magika: AI powered fast and efficient file type identification 2 years ago

Thank you for the release! I understand you're just getting it out the door. I just hope to see it delivered as a native library or something more reusable.

I did try the python cli, but it seems to be about 30x slower than `file` for the random bag of files I checked.

I'll probably take some time this weekend to make a couple of issues around misidentified files.

I'll definitely be adding this to my toolset!

0ThreadHN
Magika: AI powered fast and efficient file type identification 2 years ago

At multiple points in my career I've been responsible for apis that accept PDFs. Many non-tech savvy people seeing this, will just change the extension of the file they're uploading to `.pdf`.

To make matters worse, there is some business software out there that will actually bastardize the PDF format and put garbage before the PDF file header. So for some things you end up writing custom validation and cleanup logic anyway.

0ThreadHN
Magika: AI powered fast and efficient file type identification 2 years ago

This is false in every sense for https://www.darwinsys.com/file/ (probably the most used file version). It depends on the magic for a specific file, but it can check any part of your file. Many Linux distros are years out of date, you might be using a very old version.

FILE_45:

    ./src/file -m magic/magic.mgc ../../OpenCalc.v2.3.1.apk
    ../../OpenCalc.v2.3.1.apk: Android package (APK), with zipflinger virtual entry, with APK Signing Block
0ThreadHN
Magika: AI powered fast and efficient file type identification 2 years ago

Yeah, there is this line:

    By open-sourcing Magika, we aim to help other software improve their file identification accuracy and offer researchers a reliable method for identifying file types at scale.
Which implies a production-ready release for general usage, as well as usage by security researchers.
0ThreadHN
Magika: AI powered fast and efficient file type identification 2 years ago

Made a small test to try it out: https://gist.github.com/moshen/784ee4a38439f00b17855233617e9...

    hyperfine ./magika.bash ./file.bash
    Benchmark 1: ./magika.bash
      Time (mean ± σ):     706.2 ms ±  21.1 ms    [User: 10520.3 ms, System: 1604.6 ms]
      Range (min … max):   684.0 ms … 738.9 ms    10 runs
    
    Benchmark 2: ./file.bash
      Time (mean ± σ):      23.6 ms ±   1.1 ms    [User: 15.7 ms, System: 7.9 ms]
      Range (min … max):    22.4 ms …  29.0 ms    111 runs
    
    Summary
      './file.bash' ran
       29.88 ± 1.65 times faster than './magika.bash'
0ThreadHN
Magika: AI powered fast and efficient file type identification 2 years ago

As someone that has worked in a space that has to deal with uploaded files for the last few years, and someone who maintains a WASM libmagic Node package ( https://github.com/moshen/wasmagic ) , I have to say I really love seeing new entries into the file type detection space.

Though I have to say when looking at the Node module, I don't understand why they released it.

Their docs say it's slow:

https://github.com/google/magika/blob/120205323e260dad4e5877...

It loads the model an runtime:

https://github.com/google/magika/blob/120205323e260dad4e5877...

They mark it as Experimental in the documentation, but it seems like it was just made for the web demo.

Also as others have mentioned. The model appears to only detect 116 file types:

https://github.com/google/magika/blob/120205323e260dad4e5877...

Where libmagic detects... a lot. Over 1600 last time I checked:

https://github.com/file/file/tree/4cbd5c8f0851201d203755b76c...

I guess I'm confused by this release. Sure it detected most of my list of sample files, but in a sample set of 4 zip files, it misidentified one.

0ThreadHN
Ask HN: Who is hiring? (May 2022) 4 years ago

Xero | Multiple Roles | REMOTE (ON/AB/BC Canada), HYBRID Toronto | Full-Time | https://www.xero.com/

Xero is one of the fastest growing software-as a-service companies globally. We lead the New Zealand, Australian, and United Kingdom cloud accounting markets, employing a world-class team of 4,000+ people. The future of our growth is in the Americas and as the Product Development Hub for Xero North America we're continuing to rapidly grow and expand and have roles available, in both a hybrid and fully remote capacity, across almost every department. In terms of tech, we build in .NET and Node on the backend, React on the front-end and run everything in AWS. We're looking for engineers and engineering leads who are enthusiastic about solving problems at scale to come build the future of Xero in North America.

Selected open Remote roles:

  - Software Engineer
  - Machine Learning Engineer
  - Senior Android Engineer
  - Senior iOS Engineer
More information here:

Remote Roles: https://jobs.lever.co/xero?lever-via=XwiHpkoGYx&location=Rem...

Hybrid, Ontario Roles: https://jobs.lever.co/xero?lever-via=XwiHpkoGYx&location=Tor...

Please apply directly through Lever above or feel free to reach out directly at colin [dot] kennedy [at] xero [dot] com

0ThreadHN
DaVinci Resolve 18 4 years ago

I don't know about the preview issues, but I have also had some bizarre issues during encoding. Specifically h264 using the mp4 container will crash encoding almost 100% of the time. This started a couple of major versions ago, and continues to today. When encoding h264 in a mov container, no crashes, any other codec, no crashes. I would say simply try a different container and then use ffmpeg to copy / transcode into a format you want to deliver.

0ThreadHN