HN user
with the amount of traction this has gotten... coming with a clear set of experiments even on arxiv paper would be of great help to showcase your improvements. And if they're easily reproducible, they could get integrated in the mainstream inference engines as well, as the main point here is compression with little degradation.
Actively doing this. It indeed forces me to think things through, organize thoughts and speak them out. I open paint/miro to draw. It's good practice.
and with little data (i.e. <10Mb), this matters much less than accessibility and easy understanding of the data using a simple text editor or jq in the terminal + some filters.
it's skills first and then money and hardware for scale
A more skilled person that understands all the underlying steps will always be more efficient in scaling up due to knowing where to allocate more.
basically... you always need the skills and the money is the fine tuning.
Can you append new columns to a file stored on disk without reading it all in mempey? Somehoe this is beyond parquet capabilities.
Yeah, but if you can do topologies based on latencies you may get some decent tradeoffs. For example with N=1M nodes each doing batch updates in a tree manner, i.e the all reduce is actually layered by latency between nodes.
lossy encycopledia that can also do some short-term memory (RAG) things.
I've played years of KZ and HNS after years of playing competitive CS on local communities (old PGL in romania!). I got over 6k hours in steam CS1.6 + many more on "non-steam". That game shaped me. I even learned the basics of programming while modding a KZ plugin: https://forums.alliedmods.net/showthread.php?t=130417
Nowadays I code for a living, but for sure this is the game that started the spark for me.
It was a great time and I feel that I can always run this game and get back to that childhood feeling.
Tbh, the web won the application platform mostly because it's a standard. Everybody knows html, css and a little JS.
On the other hand, for mobile apps, there is still a device-specific mentality.
Imagine web apps being built with a different flavor for all the major browsers...
I hope that the same level of standardization comes to mobile apps too with the option to use more device-specific features on top of the generic UI.
Mine is much more barebone:
- one single machine - nginx proxy - many services on the same machine; some are internal, some are supposed to be public, are all accessible via the web! - internal ones have a humongous large password for HTTP basic auth that I store in an external password manager (firefox built in one) - public ones are either public or have google oauth
I coded all of them from scratch as that's the point of what I'm doing with homelabbing. You want images? browsers can read them. Videos? Browsers can play them.
The hard part is the backend for me. The frontend is very much "90s html".
the last point got me.
How can you idiomatically do a read only request with complex filters? For me both PUT and POST are "writable" operations, while "GET" are assumed to be read only. However, if you need to encode the state of the UI (filters or whatnot), it's preferred to use JSON rather than query params (which have length limitations).
So ... how does one do it?
vscode with the markdown plugin works really good.
text on the left, render on the right pane
example: https://imgur.com/9rjoMa2.png
At work we have a pretty big Python monorepo. The way we scale it is by having many standalone CLI mini apps ( about 80) atm with most of them outputting json/parquet in GCS or bigquery tables. Inputs are the same.
I insisted a lot on this unix (ish as it's not pipes) philosophy. It paid off so far.
We can test each cli app as well as make broader integration tests.
I meant that they should be separate tools that can be piped together. For example: you have 1 directory of many files (1Gb in total)
`zip out.zip dir/`
This results in a single out.zip file that is, let's say 500Mb (1:2 compression)
If you want to shard it, you have a separate tool, let's call it `shard` that works on any type of byte streams:
`shard -I out.zip -O out_shards/ --shard_size 100Mb`
This results in `out_shards/1.shard, ..., out_shards/5.shard`, each of 100Mb each.
And then you have the opposite: `unshard` (back into 1 zip file) and `unzip`.
No need for 'sharding' to exist as a feature in the zip utility.
And... if you want only the shard from the get go without the original 1 file archive, you can do something like:
`zip dir/ | shard -O out_shards/`
Now, these can be copied to the floppy disks (as discussed above) or sent via the network etc. The main thing here is that the sharding tool works on bytes only (doesn't know if it's an mp4 file, a zip file, a txt file etc.) and does no compression and the zip tool does no sharding but optimizes compression.
couldn't agree more!
We need to separate and design modules as unitary as possible:
- zip should ARCHIVE/COMPRESS, i.e. reduce the file size and create a single file from the file system point of view. The complexity should go in the compression algorithm.
- Sharding/sending multiple coherent pieces of the same file (zip or not) is a different module and should be handled by specialized and agnostic protocols that do this like the ones you mentioned.
People are always doing tools that handle 2 or more use cases instead of following the UNIX principle to create generic and good single respectability tools that can be combined together (thus allowing a 'whitelist' of combinations which is safe). Quite frankly it's annoying and very often leads to issues such as this that weren't even thought in the original design because of the exponential problem of combining tools together.
I think Open Weights is a better name for AI models that don't share the reproducible training scripts and data.
one answer is due to the fact that humans also do this with just 2 pretty bad cameras and a lot of offloading to the cortex.
It also simplifies the stack a lot to have a single set of sensors, so the software becomes mostly: getting good training data (iterative loops from failing production cases) and an efficient training algorithm.
This scales to more than just AD and also can leverage new breakthroughs from academia
what about this pattern? https://www.inngest.com/blog/python-errors-as-values
I tried it once in an sqlite DB connector with some business logic and simply checking stuff like
res: DBException | Result = db_handler.some_business_logic()
if isinstance(res, DBException):
return res # you can also log or even raise if this function isn't returning exceptions as values
# guaranteed to be Result type here
- https://gitlab.com/meehai/drpciv-flask/-/blob/main/be/db_han...
- https://gitlab.com/meehai/drpciv-flask/-/blob/main/be/app.py...
how does this compare to streamlit ?
I'm in the 'visualising the words as text' category, and I've always thought I have a bad memory compared to other friends, though I can recall every little detail of large software projects that I've been in.
During exams, where I had to cram lots of theory in a small amount of time, I recall trying to 'access' the slides via how they looked like in my mind and trying to somehow read the stored image, because that's for me easier to remember than the actual text when I'm not doing any deep understanding, but just memorization :/ I hated these kinds of exams, what's the point of repeating 500+ slides out of 14 weeks of courses word by word ?
Open book exams + internet + tricky questions were the best.
Linear is perfect.
you had me in the first half, not gonna lie.
isn't this what llamaindex is doing though?
can confirm that it is a nice thing to work with parquet files. Before this, we've worked for ~1 year with CSVs (I know the horror) and we made an effort to port all the 'legacy' code to Parquet files
We interface with BigQuery (via Airflow) mostly, and except one very annoying situation it's a big improvement in terms of speed (parsing floats after querying the DB is NEVER a good option).
---
In case anyone's wondering, it's basically storing and loading native numpy arrays in BigQuery via the python client(s).
You have a bunch of options (assume you have one or more cols with float32 numpy arrays):
- dataframe -> to_parquet -> upload to GCS -> GCSToBigQueryOperator (https://airflow.apache.org/docs/apache-airflow-providers-goo...)
-> instead of storing as a `FLOAT, REPEATED` it will be stored as a STRUCT with a structure of `list>item` OR `list>element` (pyarrow==11 OR pyarrow==13).This requires a manual parsing from this 'json structure' that you get when querying the DB back to np.array -> slow and basically you are using CSVs again.
-> Read more: https://stackoverflow.com/questions/68303327/unnecessary-list-item-nesting-in-bigquery-schemas-from-pyarrow-upload-dataframe
-> set the schema before uploading? Nope, all values will uploaded as null in BQ.
- very slow, you need to batch your data (imagine 24h vs 5 minutes kind of slow as dataframe sizes increase + necessity to keep all data in memory or batch it so extra save/load of each batch before uploading)
- arrays are stored properly
- `df.to_parquet(..., use_compliant_nested_type=True)` (in pyarrow==14 it's True by default, but airflow needs pyarrow==11, where it's False by default)
- use `enable_list_inference=True` (link: https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-parquet#list_logical_type)
- when both of this are true (i.e. save parquet files [to GCS] using that flag and load parquet files [from GCS to BQ] using the other flag arrays can be stored as (FLOAT, REPEATED) and queried as numpy arrays out of the box without any manual management.
I think this latest profit-driven crap made me finally switch to HN as a 'main' platform for news. I'll miss /r/machinelearning, but posts here are like a combination of programming + ml so, I guess that's fine.