sheetjs

Knuth is also still reviewing TAOCP-related email and issuing reward checks. A teammate received a reward check for 1 hex dollar last month after finding an error in Seminumerical Algorithms. Along with the check, there was a printout of the original email with hand-written annotations.

SheetJS | https://sheetjs.com/ | Software Developer | Full time, Remote (US) | $165K - $240K

We're a bootstrapped company building open source solutions for spreadsheets and structured data. With over 1.8M unique monthly visitors, companies across the business world turn to us for challenging data processing problems. Over the last 10 years, we have pushed the boundaries of JavaScript and the web.

In this role, you will master new and established technologies while working on high-impact projects used by millions of people across the world. Balancing research and engineering, you will design and implement creative solutions that draw from your academic and professional experience.

https://sheetjs.com/careers/ more details

Last year, Bing and Edge erroneously flagged our website https://sheetjs.com/ as "dangerous": https://i.imgur.com/BvA3zrk.png

At the time, there was no "Safety Report" to indicate why Bing thought it was dangerous. The report page linked to https://www.bing.com/toolbox/bing-site-safety?url=https%3a%2... and it said "That web page doesn't exist"

To fix it, we had to register with "Bing Webmaster Tools" (https://www.bing.com/webmasters/about) and raise a support ticket.

Within a few days, the issue "resolved itself". It's possible that raising a ticket forced some automatic refresh of the indexed data for the domain.

Under the hood, modern Numbers stores values in Decimal128 (16 bytes)

SheetJS | https://sheetjs.com/ | Software Developer | Full time, Remote (US) | $165K - $240K

We're a bootstrapped company building open source solutions for spreadsheets and structured data. With over 1.5M unique monthly visitors, companies across the business world turn to us for challenging data processing problems. Over the last 10 years, we have pushed the boundaries of JavaScript and the web.

In this role, you will master new and established technologies while working on high-impact projects used by millions of people across the world. Balancing research and engineering, you will design and implement creative solutions that draw from your academic and professional experience.

https://sheetjs.com/careers/ more details

Under the hood, numpy and friends punt to native code. FFI / native module performance becomes a bottleneck, and Bun is correctly focusing in this area. Shumai [1] is a cool library for Bun that also punts to native code

[1] https://github.com/facebookresearch/shumai

JavaScript has a massive and unrivaled ecosystem outside of core data analysis. The main hurdle to adoption is ecosystem and community. A generation of applied mathematicians and data scientists were trained on Python. Python overcame older ecosystems like Matlab and Fortran for its ease of use for other general computing tasks (Matlab is a decent language for large array manipulation but soft tasks like string processing are kludgy compared to Python).

In the same way that Python developers are interested in extending their ecosystem to other spaces, JavaScript developers are interested in growing into the data science space. But that doesn't happen unless there's a clear benefit for switching.

What advantages do a native JS solution have over Python? Performance: There are arguably many more developers and companies focused on pure JS performance than python. Community: A JS solution opens up a much larger pool of talent to build innovative solutions. Cost: Running heavy computations client-side, in the web browser, reduces costs for service providers.

PS: We're hiring for this and other related problems https://sheetjs.com/careers

The hardest unsolved problem is document compatibility. It's not enough to have a cool tool -- you need to be able to interchange data with existing users of Excel and other workflows. This was an important piece during Excel's journey to overtake Lotus 1-2-3.

We (https://sheetjs.com/) have been looking into document compatibility for 10 years (celebrating our 10 year birthday this week!), and our eponymous open source project https://github.com/sheetjs/sheetjs is used by companies large and small. It's not a particularly glamorous subject and doesn't tend to electrify people in the same way as build tools or frameworks.

.

There are ways to improve upon the space, but the problem is that the world has changed. Excel was designed to be the "center of the universe", a creative substrate that bypassed org security policies and enabled extremely flexible line of business tools. Excel was designed to be used by one user at a time, with fundamental inconsistencies blocked at the UI level (for example, try entering a bad custom number format). This made sense 20 years ago, but it doesn't make as much sense now.

The current crop of SaaS companies effectively monetize access to the data. They aren't incentivized to make it easy to export metadata back to Excel -- they want to keep you using the software. This runs at odds with the data portability and freedom that is needed for a successful replacement.

Whatever will replace Excel won't be a facsimile of the current tooling (what Google and Apple are trying to do), nor will it be a siloed experience (what the myriad of SaaS apps are trying to do).

Unfortunately the source spreadsheet was not shared, but the root cause was a misreported codepage (which you can actually fix with a hex editor!)

"Â ", where the second character is a unicode non-breaking space characters (U+00A0), is the CP1252 encoding of the byte sequence [0xC2, 0xA0] which is the UTF-8 encoding of "\u00A0". So all that really needs to happen is to get Excel to think that the file is UTF-8 encoded.

The fix? For XLS, there is a special Codepage record that indicates how strings are to be interpreted. The magic value to force UTF-8 interpretation is 65001, and it is possible to use a hex editor to find exactly where that record is located and change it.

A Seattle Times article from 1989:

Inside Microsoft – A ‘Velvet Sweatshop’ or a High-Tech Heaven?

https://www.seattletimes.com/business/archive-inside-microso...

This password specifically refers to "Password to Modify" and Excel 2019 / 365 clearly warn in the reenter password popup:

Caution: Password to modify is not a security feature. ... Malicious users can edit the file and remove the password.

This type of "protection" is also present in the VBA blobs (where you can change a few bytes and work around the VBA protection)

Saving a file with a password to open actually employs encryption. The exact setting can be tuned with registry settings, but is typically AES-128-CBC.

Saving a file as "read-only" encrypts the file with the standard password "VelvetSweatshop"

Hi, random developer that somehow ended up in the top 500 but didn't have 2FA turned on (https://www.npmjs.com/package/xlsx)! npm inc invalidated all of our authentication tokens in mid April and we have been unable to sign in via the web interface since then. Assumably the same fate befalls other random devs.

Software Arts published an official technical specification of the DIF file format in 1983. [1] (to save a file in the DIF format, /S#S in VC)

It is certainly an interesting format, and Excel completely mangles the data (whether it was intentional is up for debate). For example, the value

    1,0
    "0.3"

    1,0
    "=""0.3"""

 [1] https://atariwiki.org/wiki/attach/VisiCalc/DIF_Technical_Specification.pdf

Maybe a story from a maintainer would help. To contextualize, the main SheetJS open source project https://github.com/SheetJS/sheetjs has over 28K stars.

tl;dr: the project involves "crowdsourced research" which benefits from popularity.

The main social goal with the project is data preservation and integrity. Large-scale economic and political decisions are made from data and analyses in spreadsheets. For example, last year in the UK, COVID cases were underreported thanks to Excel minutiae https://www.bbc.com/news/technology-54423988

Due to various corporate stratagems, the older data representations were intentionally obfuscated. To support Excel, many developers poked around at Excel files and guessed at the structures.

In this environment, the biggest challenge is finding worksheets with random corner cases. These types of files are not easy to create and fuzzing has limited effectiveness. This is where open source and popularity come into play. The open source and JS nature of the project helps reduce testing friction (https://oss.sheetjs.com/ runs in the web browser, no need to install anything) and encourage bug reports with test cases.

There will always be "entitled users" and "low quality bug reports" but that comes with the territory. There are also meaningful issues and code contributions. Efforts at trying to prevent the low quality contributions also discourage higher quality contributions.

Pre-Unicode issues still haunt us today, kept alive by various file formats that rely on system encoding.

Under the Apple "Mac-Roman" encoding [1], the standard MacOS encoding before OSX switched to Unicode, byte 0xBD currently is capital omega (U+03A9 Ω). However, in the original 1994 release of the character set, they erroneously mapped to the ohm sign (U+2126 Ω) Apple eventually fixed this in 1997, as noted in the changelog:

    #       n04  1997-Dec-01    Update to match internal utom<n3>, ufrm<n22>:
    #                           Change standard mapping for 0xBD from U+2126
    #                           to its canonical decomposition, U+03A9.

This discrepancy leads to a huge number of strange issues with various versions of Excel for Mac (BOM-less CSV, SYLK and other plaintext formats default to system encoding) and other software that use Microsoft's interpretation of Apple's Mac-Roman encoding rather than Apple's official character set mapping.

[1] http://www.unicode.org/Public/MAPPINGS/VENDORS/APPLE/ROMAN.T...

[2] http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/MAC/RO...

Our story is very similar. I wrote a small library for converting XLSX and XLS files to CSV. Over the years, that grew into one of the most popular open source libraries on npm/github: https://github.com/SheetJS/sheetjs

Back in 2015, 'patio11 reached out to us. In addition to a structured license purchase, he gave great insights and actually wrote a blog post about the experience https://www.kalzumeus.com/2015/01/28/design-and-implementati...

Today, we offer paid software builds to solve related problems and it allows us to work on SheetJS full-time!

Spreadsheet evolution has been slow though.

Excel has to contend with nearly 40 years of backwards compatibility (MultiPlan, the predecessor to Excel, was released in 1982) and a deep userbase that literally has decades of experience and muscle memory with the software. The Symbolic Link "SYLK" file format introduced in MultiPlan is still supported in recent versions of Excel, leading to the infamous CSV "ID" issue.

Many of our users still run very old versions of Excel and Windows (e.g. Excel 5.0 on Windows 95) because a change in a future version of Excel caused problems or gave different results.

Newer versions of Excel have native JavaScript support https://docs.microsoft.com/en-us/office/dev/add-ins/referenc...

When saving as CSV, Excel will use the regional "List separator" setting. You can override this in Windows 7 with Region and Language > Additional Settings > List separator.

If you are trying to generate a file that plays nice with Excel, there is a way to force a specific delimiter with the "sep" pragma:

    sep=|
    a|b|c
    1|2|3

We have a few customers in reinsurance, and for the most part the goal is to do the opposite of what the python solutions try to do. Instead of integrating foreign stuff into existing workbooks, the goal is to retain the existing worksheets as source of truth and build modern tools around the files. The most common use case is building out a web interface to replicate the Excel formula engine.

In the python space, there are libraries like openpyxl and xlrd, but the real hurdle is introducing python into an ecosystem which otherwise has no natural knowledge. JavaScript is the language of choice for modern Excel addins as Excel provides an actual API for it https://docs.microsoft.com/en-us/office/dev/add-ins/referenc...

For that type of pattern, you can also use the new LET function:

    LET(myval, A2+B2, IF(myval>10, myval, 10))

SheetJS was created because of a Microsoft licensing issue with a library (https://github.com/stephen-hardy/xlsx.js/issues/8). The other project had a nonstandard license with a clause that only applied to browsers run on Microsoft Windows, which is really bizarre for a JS library that can run on any browser. Apparently the original developer was working for Microsoft at the time, and Microsoft mandated the license clause.

In a funny twist of fate, Microsoft now uses SheetJS open source to power some Excel exports in Office 365! https://tasks.office.com/License.html is the license disclosure, and you can actually see it in action in the exported files.

We were blown away when we found out. It's the ultimate endorsement! Oftentimes we just wonder what would have happened if Microsoft just let the original project adopt a standard open source license.

Article doesn't explain the issue, but OFFSET lets you access cells that aren't referenced in the formula.

For example, OFFSET(B2,1,1,1,1) is a live reference to cell C3, which means you can use functions like COLUMN to investigate the range. C3 shows up nowhere in the formula, so there's no non-volatile way to implement it.

The first argument to INDEX is a "sqref" (cell, range, or set of ranges) and INDEX will error if you try to reference a cell outside of the sqref, so use of INDEX doesn't break the obvious dependency structure

Excel predates RFC4180 by nearly 20 years (RFC4180 is October 2005, Excel 1.0 was September 1985) and this behavior was already cemented when the RFC was written.

As for the actual RFC, it's worth taking a read. Any sort of value interpretation is left up to the implementation, to the extent that Excel's behavior in interpreting formulae is 100% in compliance with the spec.

The correct way to generate a CSV cell with a leading 0 is

    ="01"

    01,"01",="01"

It was literally cited by Britain's finance minister George Osborne as justification for UK government austerity:

As Ken Rogoff himself puts it, "there's no question that the most significant vulnerability as we emerge from recession is the soaring government debt. It's very likely that will trigger the next crisis as governments have been stretched so wide."

https://web.archive.org/web/20100414205630/http://www.conser...

If you spend some time and effort removing whatever obstacles you have in place that are keeping you from being able to do that

This is literally impossible for many JS libraries. Chromium / NodeJS / other JS environments are themselves constantly changing. Irrespective of the evolving timezone info, the core MomentJS can only be "done" for a particular set of browser versions. Each bug pertaining to dates, like https://bugs.chromium.org/p/v8/issues/detail?id=7863 , is a potential browser/engine version for which Moment needs a fix.

this world does in fact exist

It only exists for certain proprietary software and SaaS developers because of the hard work of open source developers to keep up with the changing landscape. If everyone adopted your attitude, you would be forced to contend with the true nature of the ecosystem directly.

CSV is a very bad example. Yes, it is easy to throw together a simple regex to parse simple RFC4180 CSV strings, but Excel is its own black box with a huge number of hacks.

For example, en-US excel will automagically parse TRUE and "TRUE" to be the logical value TRUE. The way to get Excel to see a literal string TRUE is to make a formula ="TRUE". Many CSV writers implement this hack specifically assuming files will be read back in Excel. So now your parser, if you're trying to process data like Excel, has to do the same.

So then you discover that this is actually localized! If you set your UI language to French (France), Excel will treat VRAI and FAUX as booleans while TRUE and FALSE are treated as literal strings.

What you thought was a simple CSV parser now has to handle localization as well. So that CSV parser library can roll its own dodgy localization support, use a tried and true solution, or just choose not to support the feature. Each choice has its own drawbacks

2020 feels like a new age because of the browser ecosystem. The actual endpoint of the "second age" is probably the combined forces of:

1) Windows 7 Support officially ended

2) Edge now switched over to Chromium

With the exception of highly niche applications and markets, sites meant to be consumed on computers can aim for Chromium support and developers can rest assured they are covering the overwhelming majority of the market. Many slower-moving enterprises are standardizing on Chromium-powered browsers. Even NetFront (browsers used by video game consoles) is switching over to Chromium

The "Third Age of JavaScript" isn't marked by tooling or frameworks or code structure, but rather by developers able to focus on one browser instead of multiple. This new age will be marked by innovative use of APIs that people disregarded in the past because they only worked in Chrome.

It's even worse than the author is suggesting. For most people, "RFC4180" is meaningless, all that matters is what Excel does. And that means you need to handle a bunch of cases if you are reading AND if you are writing files. A few cases not discussed in the blog post:

- if your file starts with \x49\x44 ("ID"), Excel will interpret the file as their symbolic link .SLK format. So if you're writing files, the ID should be wrapped in double quotes even if it isn't necessary according to RFC4180

- Excel will proactively try to "evaluate" fields that start with \x3d ("="). You can see this in action with the sample file

    1,2,3
    =SUM(A1:C1)

CSV parsing / writing certainly isn't going to be a value driver for most companies (if you're supporting user imports, you really care about XLSX/XLSB/XLS files and Google Sheets import), but it's not a trivial problem.