TinyBase: A JavaScript library for structured state

by davidbarker • 5 years ago

248 75 5 years ago

xcambar• 5 years ago

I am just discovering Tinybase but by reading the docs and API, if both are true to their word and hold their promises: wow!!!

Everything clicked quickly for me and the API covers the basic use cases and more.

Of course, reading docs doesn't say anything about maintainability or performance, but at least, it got me curious enough to assess them myself :)

Congrats to the authors!

jamesgpearce14• 5 years ago

I'm pleased with the performance but I stopped short of quoting numbers, because hardware. If you compile and `npm run testPerf` on your own machine you will see a bunch of benchmarks that clock most operations in low µs.

(That also spits out ascii charts to visually make sure there are no complexity explosions. See https://tinybase.org/guides/how-tinybase-is-built/testing/ for example.)

As for maintainability... I have no data yet, since it's a couple of days old ;)

fbn79• 5 years ago

Pro and Contro vs existing similar projects? example https://github.com/google/lovefield

NetOpWibby• 5 years ago

An immediate pro for me is that it’s not Google.

doteka• 5 years ago

Lovefield is essentially unmaintained. Something something Google projects.

pygy_3• 5 years ago

TinyBase lets you set up event handlers with arbitrary data granularity, making the data store reactive.

samwillis3• 5 years ago

This looks interesting.

I’m guessing it’s in memory only, and so has load the whole dataset?

Quite similar to LokiJS:

https://github.com/techfort/LokiJS

I like how’s its modular so you only have to include the bits you want and not a massive library.

jamesgpearce14• 5 years ago

You can persist the Store to local or session storage (in a browser), to a file (where available) or to a remote server. More on the concept of 'Persisters' is in https://tinybase.org/guides/schemas-and-persistence/persisti...

You can see this in action in the TinyDraw demo: https://tinybase.org/demos/tinydraw - make some changes, refresh the page, even open it up in two windows of the same browser at the same time.

pizzathehut1• 5 years ago

The codebase is quite something.

I thought the author had lost his mind, declaring dozens of new functions to simply wrap inbuilt functions and religiously not using classes.

Then I realized it probably cuts the bundle size by 2/3rds to remove every "this.longBuiltinFnName()" and replace it with a minified function call.

dotancohen3• 5 years ago

Laravel has a similar convention, wrapping everything in static classes. I hated it at first - and still hate the convention outside Laravel - but in context it works well. Not for minification, but rather for reducing cognitive load while maintaining code while preserving good separation of concerns.

hubeh• 5 years ago

This is done mainly due to the PHP autoloading limitations - classes are able to be autoloaded but functions aren't.

onion2k• 5 years ago

It'd be interesting to see if the brotli/gzip size is smaller using this sort of strategy. I suspect the benefit is lost after compression

Kiro2• 5 years ago

Where can I see it in the code?

vorticalbox• 5 years ago

https://github.com/tinyplex/tinybase

apatheticonion• 5 years ago

Classes can be effectively minified also

class Foo { foo() {} #bar() {} }

Becomes:

class f { foo() {} b() {} }

Combined with any type of compression the negligible difference is eliminated. Though it is true that static functions might be more optimisable in some cases, I'd say that on balance, the effect on maintainability is too significant for the returns. I'll take DI and testability over maybe a few kb difference

jamesgpearce14• 5 years ago

I thought the author had lost his mind

Quite possible. This was a journey! https://tripleodeon.com/2022/01/software-without-compromise/

When I first started thinking about this project, classes still meant a bunch of transpilation overhead (which dates the origin story!). I persuaded myself that plain old objects with TypeScript interfaces would strike a better balance between performance, size, and developer experience.

So I ended up with a pattern like:

  const longMethod1 = () => {...};
  const longMethod2 = () => {...};
  return {longMethod1, longMethod2};

These 'methods' can still call each other by name, and it minifies like a champ.

ahmed_ds• 5 years ago

Now this is something great. Many projects use full fledged frameworks just for using the state management aspects of it. The store module is 2.6 kb which is fantastic!

xialvjun• 5 years ago

No TypeSafe, No Transaction. This thing seems not helpful.

dotancohen3• 5 years ago

"Developers" (I use the term loosely) coming from e.g. React are getting used to storing data in a global state. The Redux library lets one change the value in the global state, and all places that use the value are updated "automatically" (e.g. by tons of behind-the-scenes code that the "developer" on his brand new mac doesn't notice, but users on five year old Windows machines with a dozen open tabs do notice). Vue has a similar implementation, Vuex.

When not using React or Vue, people used to these conventions will not know how to properly pass state between objects. This library is for them. It really offers little more than `window.pets.dog.species = "poodle"` would offer but because it is a library it feels "less hacky".

nvegater2• 5 years ago

Better explanation of when Tinybase makes sense than the one in the website.

jillesvangurp• 5 years ago

Global variables and random stuff self initializing and writing to those global variables makes me cry on the inside every time I see it. You basically create a hard to (unit) test mess because by doing that. Because of this, a lot of frontend developers never get into the habit of unit testing because they don't even realize that the reason it is hard is self inflicted. Because of the use of global variables combined with a failure to separate the business of creating stuff (plumbing/glue code) from using that stuff. I.e. business logic that should be tested.

So, you create a thing that creates more things that live in global variables that is impossible to untangle from all the other bits of code that create yet more global variables and end up depending on each other's global variables having particular names that are hard-coded left, right, and center. Basically, any attempt to test anything ends up firing most or all of the code at which point it's not a unit test but an integration test. If you need an entire browser running to test a thing, it's because it breaks this rule of separating glue code from business logic. Don't do that.

If you fix this, you can create stuff in a slightly different way in your tests (e.g. use stubs or mocks) and test them in isolation. And the same properties actually also make it easier to reuse things.

I've been using kotlin-js for the last year with a reactive web framework called fritz2 that is loosely inspired by React. One of the first things I did there was bring in koin, which is a dependency injection framework that many Android developers would know (similar to Dagger, which is a popular alternative). At startup time, koin ensures anything that has dependencies gets their dependencies and that those dependencies get created. Koin doesn't really do anything fancy or expensive. It just forces you to separate your plumbing and glue code from your business logic and makes that easier than doing it manually would be (which isn't all that hard, actually).

Implementing something like koin in javascript would not be hard and there are probably multiple npms that do that. And you can do this manually (it's called DIY dependency injection). But you need to know to do this and be a bit disciplined about using it. You can do this in almost any language actually and it's almost never a bad idea and almost always a mistake not to separate your glue and business logic. Most bigger projects get organized on this front at some point; or they just fail. That happens a lot with Javascript code written by developers that don't understand how to properly structure their code.

lagrange77• 5 years ago

I like it a lot. If you're a 'react' 'developer', also have a look at Zustand. https://github.com/pmndrs/zustand

jamesgpearce14• 5 years ago

Just for the record, I developed TinyBase on a 12" 2016 1.1GHz MacBook! Constraints are good :) - and it keeps me honest with respect to performance, size, and toolchain.

janjones• 5 years ago

I guess that instead of transactions there is undo/redo stack, a.k.a. checkpoints [1]

[1] https://tinybase.org/guides/relationships-and-checkpoints/us...

pygy_3• 5 years ago

No wireless. Less space than a nomad. Lame.

This is a stepping stone. A reactive, relational store is very useful, something I've been thinking about for a long time.

jamesgpearce14• 5 years ago

There are transactions! https://tinybase.org/api/store/interfaces/store/store/method...

This allows you to wrap a batch of mutations together and then the fully reconciled change events are all fired together at the end. Apparently I could do a better job of marketing this secret feature :)

As another commenter has mentioned, there are also 'checkpoints' that let you create a stack of points in history to undo/redo back and forwards through. https://tinybase.org/guides/relationships-and-checkpoints/us...

pygy_3• 5 years ago

Awesome! Do you support locks/rollback or conflict detection? If not is it planned?

nvegater2• 5 years ago

I've been using react with typescript since 2019 and I haven't encounter any problems using Objects to manage data. Recoiljs is type safe and very efficient.

The website does a great job explaining how to use the api, but it would be nice to see some functional examples as well. Its difficult to imagine for me, in which use cases Tinybase is shining.

lolive• 5 years ago

I really wonder, as a Redux user, how to deal with a state that is a deeply connected graph. You cannot exactly rely upon the … keyword to simplify the definition of the new state.

nyellin2• 5 years ago

I really like the concept. We use Supabase as the backend for a Vue application.

We end up pulling in very large amounts of data to the frontend (details on every alert and event in kubernetes clusters) and then filter it multiple ways and graph it on a timeline.

So we end up implementing lots of db like logic in typescript. I've long wished for a simpler and more reusable way to do this that still let's me process the data clientside.

Here's an example of a simple timeline graph (built with vegalite) which needs the ability to filter by several columns and do various groupings

https://ucarecdn.com/0c76c5ef-1eb4-4320-8e46-abb9c3956658/-/...

vaughan4• 5 years ago

I think most dbs, paas, frameworks skip this use case.

For a snappy data-centric app, you pretty much need your entire app logic to run client-side.

Everyone always ends up with this terrible denormalized client-side cache with duplicated Server side logic.

What you really want is the exact same db running server-side and client-side.

I thought for a while this would be an sql db but now I think sql and the relational model is the wrong approach for web app data as they are inefficient for watching queries and syncing.

nyellin2• 5 years ago

Hmm, interesting.

I'm using supabase which essentially autogenerates the entire server-side part. So all my logic is client side. What's missing is tighter integration between swrv/vue-swrv and the supabase client so that when you run a query it can be smart enough to know if you apply it locally or remotely.

crate_barre• 5 years ago

It’s good for not introducing convoluted ideas that other state management libraries have been guilty of. You store some shit, and retrieve some shit. Sounds about right. The extra stuff is nice, but it doesn’t seem like the library is making you use schemas, and so on (especially when all most people want to do is store some values).

Now, if we can get some autocomplete going on with those schemas, that would would be great when inevitably the global store becomes a giant blob.

jamesgpearce14• 5 years ago

Noted! Need to think about this one.

talolard• 5 years ago

I used to use redux-orm which offered a similar promise. This library looks seems more mature and thought out, kudos!

My app is still alive and kicking and I ended up implementing many of the ideas here myself, in a less reusable fashion. A primary motivator was that as the use cases evolve, a storage model optimized for application specific access patterns becomes compelling.

The convenience of having properly implemented relational semantics and performance enhancements of Indices is huge, but the risk of hitting assumptions that don't fit your use case is also non-trivial

jamesgpearce14• 5 years ago

Thanks for the feedback! I'm going to push back on your assertion that it's mature since it's about two days old and I only revved it to v1.0.3 because of some .d.ts resolution issues :)

I agree with your very final point. The same might be true of the undo stack implementation too - at some point the usage patterns might invalidate some of my performance assumptions.

danr4• 5 years ago

Looks very interesting. Does it have TypeScript definitions so the schema are type safe?

jamesgpearce14• 5 years ago

The definition of a Schema itself is typed of course (https://tinybase.org/api/store/type-aliases/schema/schema/) but I haven't thought of a way to then magically create developer-time definitions of a Schema on a Store. There is also a way to 'programmatically' enforce a Schema (https://tinybase.org/guides/schemas-and-persistence/mutating...) so you could (in theory) implement all sorts of wild and wonderful constraints.

I think for now you'll need to create your own local type definition that models the data you know is in the Store. But let me think a bit more about this.

injidup2• 5 years ago

I wrote a similar thing for svelte stores. The live example is here.

https://svelte.dev/repl/4434d8fcd12242d79887343fd95e429c?ver...

You can wrap pretty much any POD/array data type as a store then index into it to generate new stores with view on to sub elements.

Note that a "store" is a reactive data store. You can set a value on it and also get notified of changes.

This means you can have a single store for your main data and then pass views on that data to sub components.

Views are created using selector expressions.

    let data = {
       name: 'thomas',
       age: 25
    }

    let root = writable(data)

    let rootUndo = undoStore(root)

    let nameStore = subStore(rootUndo, v=>v.name)

    let ageStore = subStore(rootUndo, v=>v.age)

    let undoDisabled = derived(rootUndo.canUndo, u=>!u)
    let redoDisabled = derived(rootUndo.canRedo, u=>!u)

   ----

    <input type="text" bind:value={$nameStore}/>
    <input type="number" bind:value={$ageStore}/>
    <button on:click={undo} disabled=   {$undoDisabled}>undo</button>
    <button on:click={redo} disabled=   {$redoDisabled}>redo</button>

Library is here https://github.com/bradphelan/immer.loves.svelte

hamilton• 5 years ago

love this. I similarly use immer for just about every svelte project I build & maintain.

injidup2• 5 years ago

When I built this I used immer under the hood. But actually I think it is not required. I'd have to have another look. I'm not really a front end dev but I just felt the whole "store" concept in svelte was incomplete and I could never understand the react/redux model. I had implemented something in .net years ago to simulate functional lenses and when I looked into immer I saw this JS proxy thing and realised I could easily build fully reactive lenses for svelte. In the end mostly an academic exercise as it is used in one internal office website and my day job is writing algorithms for CNC machines.

ngcc_hk• 5 years ago

Interesting and I need a library for browser first and later move to node.

Looking through the comment seems good to go. Thanks.

lloydatkinson• 5 years ago

This looks very nice and easier to model relationships with than other flux style state managers. I'd like to see a demo using Vue.

I know that redux has a nice guide here on how to make relationships but this library seems to make it an upfront concept. https://redux.js.org/usage/structuring-reducers/normalizing-...

As long as it has good Typescript support this could become very popular.

acemarke• 5 years ago

Hey, I wrote that docs page back in 2016 :) I'd been using an early version of Redux-ORM on a project, there were some Redux repo discussion threads about this problem space, and it seemed like something that was worth documenting.

We did eventually add a `createEntityAdapter` API to Redux Toolkit [0] [1], which handles the process of storing items in normalized form and provides "CRUD"-style reducers for typical operations on that data, but it doesn't provide any support for managing relations specifically.

That topic _has_ come up a few times recently, so I've been contemplating that as a thing we might consider trying in a future version of RTK.

If folks have thoughts on what that API's requirements might look like, I'd be happy to start discussing that over in the RTK repo!

[0] https://redux-toolkit.js.org/api/createEntityAdapter

[1] https://redux.js.org/tutorials/essentials/part-6-performance...

newsbinator• 5 years ago

This looks well developed and I'd love to see a Vue version.

I've been using Vuex-Pathify, which makes state management in Vue so, so much cleaner and friendlier than plain Vuex.

https://davestewart.github.io/vuex-pathify/#/?id=home

jamesgpearce14• 5 years ago

Yes! Though there is an optional ui-react module, this is not a React library per se. There's nothing to stop a Vue equivalent except... my lack of current knowledge about it. Noted!

vaughan4• 5 years ago

Web apps are so over-complicated because of piecemeal client side data stores and the need to sync with the latest fashion in api protocol.

Imagine how simple apps would be if your data model and query interface was exactly the same on the server and client. With automatic caching and fetching logic.

There is such an incredibly complex chain of code to do the simplest of things.

What we lack is a good client-side relational+graph database that has an identical server-side implementation.

HatchedLake721• 5 years ago

Orbit.js does some of these things and coordinates client with a variety of data sources through a standard set of interfaces and using normalized data structures.

I believe Orbit.js was inspired by Ember Data, as I know Dan Gebhardt is involved with Ember.js and https://jsonapi.org

okal• 5 years ago

PouchDB has offered something like that for close to a decade now, if I'm not mistaken. Might be worth looking into. It is document-based, though, rather than relational+graph, though I suppose you could build graph utilities on top of it.

hunterb123• 5 years ago

RxDB does this. Recently PouchDB integration is more abstracted to swap it out but it's still the best client side database. RxDB syncs with GraphQL and Pouch/CouchDBs

There's WatermelonDB which uses IndexedDB on web and SQLite on native, it's nice for syncing to custom backends.

There's GUN and Orbit for distributed graph databases.

Ontopic: TinyBase looks really really nice, fills the gap in-between hefty client side databases and state system solutions like Redux + Persist. I'd like to see Redux middleware integration for time travel debugging, event lots, and snapshotting if possible. The analytics and rollback APIs are a nice touch. Size is enticing.

samwillis3• 5 years ago

PouchDB is incredible, so much respect for the devs that built it.

The trouble is though, it is now quite an ageing codebase with relatively little maintenance. The original developers have moved on and it’s a little neglected. A few community members have picked up the mantle in the last 6 months but I would be careful picking it for something new.

Last time I looked the bug tracker auto closed tickets after a month if there was no activity. That makes it look like there are few bugs being tracked. Problem is there is loads, they are just all closed.

bcherny2• 5 years ago

The requirements on clients are fundamentally different than on the server. On clients, you want at least:

1. Batching, to avoid network waterfalls (a consequence of network use being more expensive on clients than on the server).

3. Data dependencies, so the framework knows how to stitch queries together when batching them.

3. Consistency, so all UIs update when an update comes in (a consequence of SPAs being long-lived, as opposed to the typical stateless server request).

These all take some code/framework overhead. It’s valuable to pay that cost on clients, but is unnecessarily verbose for the server.

People have been trying to unify local and remote function calls for years, and it’s a similar problem: the two are fundamentally different.

wrnr• 5 years ago

I is just me or does anyone else have to keep endless models and endpoints and commands and clients in sync in different languages?

vaughan4• 5 years ago

"code/framework overhead" is fine as long as its hidden in a library. Too many people re-implement a data layer client-side that is always a poor mapping to the server data model. My frontend should act as if its running off a local database and sync with the server. It could batch together all requests made to this database that it cannot fulfil or that are stale.

zonotope• 5 years ago

There's Datascript[0] and Datomic[1]. While not "identical", they are definitely complimentary. There's a (now defunct) library[2] for keeping them in sync too.

1: https://github.com/tonsky/datascript 2: https://www.datomic.com/ 3: https://github.com/metasoarous/datsync

samwillis3• 5 years ago

SQLite on WASM[0] is absolutely what you are looking for. There is also “Absurd SQL”[1] which extends it to use indexedDB as a VFS for storage allowing proper atomic transactions and not loading the whole thing into memory.

Combine it with the various JavaScript ORMs and you have a nice developer UX.

I’m waiting for someone with more time than myself to build a syncing feature on top of SQLites Sessions[2] so that changes locally are synced back to the server.

(Feel like I’m a cheer leader posting this comment every week)

0: https://sql.js.org/

1: https://github.com/jlongster/absurd-sql

2: https://www.sqlite.org/sessionintro.html

steve_adams_86• 5 years ago

Would this suffer from the overhead of using WASM, both to make the calls and to transfer the data back to the UI thread?

I'm not sure but I expect it would depend on your use cases. High frequency i/o of small data might be slower compared to low frequency i/o of very large data.

Mind you I'm working off of a cursory understanding of WASM performance issues from 2 years ago. Maybe this has changed a lot, or my understanding was incorrect in the first place. Do you know much about this?

vaughan4• 5 years ago

Absurd is really cool. I've always thought it would be great to have an SQL db implemented in native JS though.

With the database running in the same process as your app, you could store native JS Objects of your table rows in the cache, which you could bind events to. So if you modify a row in a table in the db, events could be instantly propagated to the UI. Haven't thought it through fully yet.

SQL is also a bit cumbersome for things like deeply nested data, hierarchies, trees, graphs which a lot of client-side application state ends up being.

Sessions looks cool!

dotancohen3• 5 years ago

  > (Feel like I’m a cheer leader posting this comment every week)

I haven't seen it yet, so thank you! And in the name of others like me who haven't seen it yet, keep cheering please.

DrFell• 5 years ago

The entire point of an API is decoupling. A decoupled API makes double sense for a SPA. It's the client-server architecture on a silver platter.

This idea got muddled when people started making websites as SPAs, which necessitated SSR for SEO and first-page load. Now you blurred the lines between client and server, otherwise known as isomorphism, which is a good sounding word for a bad idea.

PaulBGD_• 5 years ago

Funny enough, a lot of this is describing blitz.js.

fohara• 5 years ago

The reactive data store bits seem very similar to baobab.js [0]. Baobab's api seems a bit more terse though, which I prefer.

[0] https://github.com/Yomguithereal/baobab

bionhoward• 5 years ago

This is beautiful and I just installed it for testing in my new app, but I have a sneaking suspicion tiny databases are the sort of thing we’re all better off making from scratch so we can tinker with implementation and learn (even if that’s more work and less “optimal”)

jamesgpearce14• 5 years ago

Haha. That's kind of how this started in the first place :)

jamesgpearce14• 5 years ago

Author here, but I just woke up. AMA!

mariogintili• 5 years ago

I've seen so many of these why does javascript always javascript?

z3t4• 5 years ago

Upvoted for using perf tests

jamesgpearce14• 5 years ago

Thank you! I added a fun little thing where it charts the results to check for complexity problems. eg:

  Grow store, different table to index with multiple row count, µs per row
  First: 16.3 µs per row
   Last: 3.61 µs per row
    Min: 2.76 µs per row
    Max: 29.89 µs per row
    Avg: 7.33 µs per row

        90.00 ┼───────────────────────────────────────
        84.00 ┤
        78.00 ┤
        72.00 ┤
        66.00 ┤
        60.00 ┤
        54.00 ┤
        48.00 ┤
        42.00 ┤
        36.00 ┤
        30.00 ┤╭╮
        24.00 ┤││     ╭╮
        18.00 ┼╯│     ││         ╭╮
        12.00 ┤ ╰╮   ╭╯│    ╭╮   ││          ╭╮
         6.00 ┼──╰───╯─╰────╯╰───╯╰──────────╯╰╮──╭───
         0.00 ┤                                ╰──╯

dmaevsky• 5 years ago

Nice. I wrote a similar thing a couple years ago, even "tinier" :) https://github.com/dmaevsky/tinyx Immutable state, undo/redo, immer-like patch recording etc in less then 200 lines of code

TinyBase: A JavaScript library for structured state

Related Stories