ramity

:)

I run a business that makes 3d printed braille molds that are used to repeatably emboss paper. I haven't considered the molds being offline storage, but I suppose they are. I mostly operate with the assumption of a shelf life of 10 years for the PETG molds, but ink free, embossed paper has excellent lifespan if stored correctly.

I guess you could consider it an "offline datastore as a service." It would be a pretty good offline storage of keys with a way to request a paper copy. Certainly issues of trust and physical security but wrapping it with encryption would be easy. Also benefit from your government's legal protections for mail. There might actually be a usecase here.

Couple fast facts:

- Current 26 * 32 = 832 cells * 6 dot braille = 4992 bits/mold/page

- Possible 28 * 34 = 952 cells * 6 dot braille = 5712 bits/mold/page

- Maybe some more headroom, but that's what is possible with current spacings

Let me first start off by saying I and many others have stepped in this pitfall. This is not an attack, but a good faith attempt to share painfully acquired knowledge. I'm actively using AI tooling, and this comment isn't a slight on the tooling but rather how we're all seemingly putting the circle in the square hole and it fits.

Querying an LLM to output its confidence in its output is a misguided pattern despite being commonly applied by many. LLMs are not good at classification tasks as the author states. They can "do" it, yes. Perhaps better than random sampling can, but random sampling can "do" it as well. Don't get too tied to that example. The idea here is that if you are okay with something getting the answer wrong every so often, LLMs might be your solve, but this is a post about conforming non-deterministic AI into classical systems. Are you okay if your agentic agent picks the red tool instead of the blue tool 1%, 10%, etc of the time? If so, you're never not going to be wrangling, and that's the reality often left unspoken when integrating these tools.

While tangential to this article, I believe its worth stating that when interacting with an LLM in any capacity, remember your own cognitive biases. You often want the response to work, and while generated responses may look good and fit your mental model, it requires increasingly obscene levels of critical evaluation to see through the fluff.

For some, there will be inevitable dissonance reading this, but consider that these experiments are local examples. Its lack of robustness will become apparent with large scale testing. The data spaces these models have been trained on are unfathomably large in both quantity and depth, but under/over sampling bias will be ever present (just to name one).

Consider the the following thought experiment: You are an applicant for a job submitting your resume with knowledge it will be fed into an LLM. Let's confine your goal into something very simple. Make it say something. Let's oversimplify for the sake of the example and say complete words are tokens. Consider "collocations". [Bated] breath, [batten] down, [diametrically] opposed, [inclement] weather, [hermetically] sealed. Extend this to contexts. [Oligarchy] government, [Chromosome] biology, [Paradigm] technology, [Decimate] to kill. With this in mind, consider how each word of your resume "steers" the model's subsequent response, and consider how the data each model is trained on can subtly influence its response.

Now let's bring it home and tie the thought experiment into confidence scoring in responses. Let's say its reasonable to assume that the results of low accuracy/low confidence models are less commonly found on the internet than higher performing ones. If that can be entertained, extend the argument to confidence responses. Maybe the term "JSON" or any other term used in the model input is associated with high confidences.

Alright, wrapping it up. The end point here is that the model output provided confidence value is not the likelihood of the answer provided in the response but rather the most likely value following the stream of tokens in the combined input and output. The real sampled confidence values exist closer to code, but they are limited to each token. Not series of tokens.

I see I was mistaken, but I'm tempted to continue poking holes. Trying a different angle, though it may be a stretch, but could a caching layer within the VPN provider cause these sort of "too fast" RTTs?

Let's say you're a global VPN provider and you want to reduce as much traffic as possible. A user accesses the entry point of your service to access a website that's blocked in their country. For the benefit of this thought experiment, let's say the content is static/easily cacheable or because the user is testing multiple times, that dynamic content becomes cached. Could this play into the results presented in this article? Again, I know I'm moving goalposts here, but I'm just trying to be critical of how the author arrived at their conclusion.

Thanks for your informative reply. I see now I was approaching this incorrectly. I was considering drawing conclusions from a high RTT rather than a RTT so small it would be impossible to have gone the distance.

Contrasting take: RTT and a service providing black box knowledge is not equivalent to knowledge of the backbone. To assume traffic is always efficiently routed seems dubious when considering a global scale. The supporting infrastructure of telecom is likely shaped by volume/size of traffic and not shortest paths. I'll confess my evaluation here might be overlooking some details. I'm curious on others' thoughts on this.

35m ago edit: Apple uses many predictive systems for typing. My sentiment in pointing out just slide to type might be misguided as it does not exist in a vacuum. I'd love to see these tests redone with slide to type disabled. I'm leaving the original comment below for reference.

Slide to type. This "issue" is at most 6 years old for iOS users.

Turn off slide to type if you do not use it. Slide to type does key resizing logic. This is the direct cause of this issue. Please upvote this comment for visibility.

Please reply if you think I'm wrong. I see this get posted frequently enough I'm actually losing it.

Please refer to https://youtu.be/hksVvXONrIo?si=XD7AKa8gTl85_rJ6&t=72 (timestamp 1:12) to see that slide to type is enabled.

I didn't see any reference to a sender or actively blasting RF from the same access point. I think the approach relies on other signal sources creating reflections to a passively monitoring access point and attempting to make sense of that.

5GHz WiFi has a wavelength of ~6cm and 2.4GHz ~12.5cm. Anything achieving smaller is a result of interferometry or a non WiFi signal. Mentioning this might not add much substance to the conversation, but it felt worth adding.

I'm interested but am also incredibly dubious. Not because it seems impossible but the opposite. On one hand, an open source repo like this making an approach for hackable extension should be praised, but the "Why Built WiFi-3D-Fusion" section[0] gives me very, very bad vibes. Here's some excerpts I especially take issue with:

"Why? Because there are places where cameras fail, dark rooms, burning buildings, collapsed tunnels, deep underground. And in those places, a system like this could mean the difference between life and death."

"I refuse to accept 'impossible.'"

WiFi sensing is an established research domain that has long struggled with line of sight requirements, signal reflection, interference, etc. This repo has the guise of research, but it seems to omit the work of the field it resides in. It's one thing to detect motion or approximately track a connected device through space, but "burning buildings, collapsed tunnels, deep underground" are exactly the kind of non-standardized environments where WiFi sensing performs especially poorly.

I hate to judge so quickly based on a readme, but I'm not personally interested in digging deeper or spinning up an environment. Consider this before aligning with my sentiment.

[0] https://github.com/MaliosDark/wifi-3d-fusion/blob/main/READM...

I really want to love rust, and I understand the niches it fills. My temporary allegiance with it comes down to performance, but I'm drawn by the crate ecosystem and support provided by cargo.

What's so damning to me is how debilitatingly unopinionated it is during situations like error handling. I've used it enough to at least approximate its advantages, but strongly hinting towards including a crate (though not required) to help with error processing seems to mirror the inconvenience of having to include an exception type in another language. I don't think it would be the end of the world if it came with some creature comforts here and there.

I'll provide a contrasting, pessimistic take.

How do you write programs when a bug can kill their user?

You accept that you will have a hand in killing users, and you fight like hell to prove yourself wrong. Every code change, PR approval, process update, unit test, hell, even meetings all weigh heavier. You move slower, leaving no stone unturned. To touch on the pacemakers example, even buggy code that kills X% of users will keep Y% alive/improve QoL. Does the good outweigh the bad? Even small amounts of complexity can bubble up and lead to unintended behavior. In a corrected vibrator example, what if frequency becomes so large it overflows and leads to burning the user? Youch.

The best insight I have to offer is that time is often overlooked and taken for granted. I'm talking Y2K data type, time drift, time skew, special relativity, precision, and more. Some of the most interesting and disturbing bugs I've come across all occurred because of time. "This program works perfectly fine, but after 24 hours it starts infinitely logging." If time is an input, do not underestimate time.

How do we get to a point to `trust` it?

You traverse the entire input space to validate the output space. This is not always possible. In these cases, audit compliance can take the form of traversing a subset of the input space deemed "typical/expected" and moving forward with the knowledge that edge cases can exist. Even with a fully audited software, oddities like a cosmic bit flip can occur. What then? At some point, in this beautifully imperfect world, one must settle for good enough over perfection.

The astute reading above might be furiously pounding their keyboards mentioning the halting problem. We can't even verifiably prove a particular input will provide an output - moreover an entire space.

I am convinced that open code, specs and (processes) must be requirement going forward.

I completely agree, but I don't believe this will outright prevent user deaths. Having open code, specs, etc aids towards accountability, transparency, and external verification. I must express I feel there are pressures against this, as there is monumental power in being the only party able to ascertain the facts.

elzbardico is pointing out how the author is having the confidence value generated in the output of the response rather than it being the confidence of the output.

I too once fell into the trap of having an LLM generate a confidence value in a response. This is a very genuine concern to raise.

I think it's fair to say AI generated code isn't visibly making a meaningful impact in open source. Absence of evidence is not evidence of absence, but that shouldn't be interpreted as a defense to orgs or the fanciful predictions made by tech CEOs. In its current forms, AI feels comparable to piracy where the real impact is fuzzy and companies claim a number is higher or lower depending on the weather.

Yes, open source projects would be the main place where these claims could be publicly verifiable, but established open source projects aren't just code--they're usually complex, organic, and ever shifting organizations of people. I'd argue the metric of interacting with a large group of people whom have cultivated their own working process and internal communication patterns is closer to AGI than coding assistant, so maybe the goal posts we're using for AI PRs are too grand. I think it's expected to hear claims from within walled gardens, where processes and teams can be upended at will, that AI is making an unverifiable splash, because they're precisely the environments where AI could be the most disruptive.

Additionally, I think we're willfully looking in the wrong places when trying to measure AI impact by looking for AI PRs. Programmers don't flag PRs when they use IntelliJ or confer with X flavor of LLM(tm), and expecting mature open source projects to have AI PRs seems as dubious as expecting then to use blockchain or any other technology that could be construed as disruptive. It just may not be compatible or reasonable with their current process. Calculated change is often incremental and boring, where real progress is only felt by looking away.

I made a really simple project that automatically forwards browser console logs to a central server, programmatically pull the file(s) from the trace, and had an LLM consume a templated prompt + error + file. It'd make a PR with what it thought was the correct fix. Sometimes it was helpful. The problem was it needed to do more than code, because the utility of a one shot prompt to PR is low.

I've been off socials and on forums for 8+ years now for the same reason. I share similar sentiment as Bizzy's sibling reply. I say these things because lately I've been thinking about lot about dead internet theory and how strongly some believe it.

One of the most profound realizations I've had lately is that the perception of the medium of communication itself is a well that can be poisoned with artificial interactions. Major empahsis on perception. The meer presence of artifical can immediately taint real interactions; you don't need a majority to poison the well.

How many spam calls does it take for you to presume spam? How many linkedin autoreply ai comments does it take to presume all comments are ai? How many emails before you immediately presume phishing? How many rage baiting social posts do you need to see before you believe the entire site is composed of synthetic engagement? How many tinder bots do you need to interact with before you feel the entire app is dead? How many autodeny job application responses until you assume the next one is a ghost job posting? How many interactions with greedy people does it take to presume that it's human nature?

Thanks for your reply. The target market is anyone who has interest, as I've open sourced the tooling I used for translating text to braille and generating the geometry of the molds. I'm hoping to create a follow up post announcing a web ui that will make the functionality more accessible to a broader audience.

It's sad to hear the poor performance of embossers. In terms of lifetime, I only have about 150 or so pages created with my prototype molds, but I haven't been able to detect a degradation in the embossing quality. I think at that number of copies created and a set of molds only costing $1.50, it makes me feel it's gotten some value out of it. I think someone can buy the equipment I'm using for around $800 in total. Obviously there's a bit of a tradeoff and at some point the number of molds would exceed the cost of an embosser, but I argue this approach is pretty quick for a manual one (for now).

The pain of the approach is the printing times for sure. One bit of confusion I can clear up is that, while yes, the single 3D printer I use can only print 4 unique sets of page molds a day, using the molds is fast! My time to beat is 15 seconds to manually position a sheet of paper between the mold bodies and roll it though a cheap roller press. 4 pages a minute is pretty quick for an embosser, but this is still just manual. Now imagine it automated :) Expecting a human to do 15 seconds/press for a full work day makes me sad, but that'd be a lot of birthday cards.

I really appreciate the link to the BrailleRap project. It's cool to see innovation in the space, and I subscribe to the RepRap philosophy. I'm currently prototyping designs to automate the pressing process, but until that component is automated, this approach lies somewhere between manual and automated embossing in terms of utility. That being said, still much more accessible and cost-effective.

You absolutely can. Deterministic inference is achievable, but it isn't as performant. The reason why sadly boils down to floating point math.

I could have worded my reply better, but the simplified explanation stands :b

The way I understand it, an LLM response is a chain of tokens where each is the most probable token. Maybe there exists more complicated candidate and selection approaches than that, but biggest number works for me. For the sake of simplicity, let's just say tokens are words. You'd have access to the probability of each word in the ordering of the sentence, but I'm not sure how that would then be used to evaluate to the probability of the sentence itself or its truthiness.

#4 was a profound and very validating read for me understanding how the brain handles traumatic events. Thanks for sharing.

While somewhat tangential to this, I'd like comment on a different problem. "What's the fewest number of plates you need to go from 45lb (bar) -> 240lb at a resolution of 5lb?" Look no further than the 185lb set[1]. Composed of pairs of 2.5lb, 5lb, 2x10lb, 25lb, 45lb plates, one can do just this. Need to lift 245lb? Buy another pair of 45lb plates and your range expands to 45lb -> 330lb. Another pair of 45lb plates yields 45lb -> 420lb. You get the idea. This approach comes with the added bonus of being the most cost efficient method of buying plates, as higher pound plates yield slightly better $/lb.

1: https://github.com/ramity/athena/blob/master/notebooks/plate...

Side note: I've yet to do the calculations for kg sets, but I'm certain something similar to this exists.

Edit: I made a laughably simple mistake interpreting my results and corrected the above. Many thanks to those who replied and brought this to my attention (credited in the commit https://github.com/ramity/athena/commit/4a17a3d16058f850d09e...).

I don't really subscribe to the idea of "finishing" or "completing" a project; I'm sure my personal github can attest to that. I think "real" software is never done. The only things in software that are completed are the fractions of software we abstractly define (tasks, features, sprints, deliverables, etc). Much like us, real software lives until it doesn't. It changes through time sometimes regressing and expanding. Software whose goalposts remain static becomes deadware.

Outside of commercial projects, I program for the joy of creation and commonly, and paradoxically, automating for the sake of "not automating." I jump from project to project, sure, but I've found the largest source of not wanting to go back to a project is the difficulty of doing so. Having to pick things back up to juggle and going through the motions of learning what my software did and what needs to be done was always a pain.

My real breakthrough was "optimizing being able to leave." Comments like I'd be picking up the software months/years later, READMEs detailing build steps and rationale and planned features, automating dev environment setups with docker, break features/work into pieces so it isn't overwhelming, etc. These are just some of the many ways to make it easier.

Sometimes you don't want to go back because all you can think of is the known (or unknown) work that lies ahead of you. The fewer the reasons to not go back, the easier it is, and if it's easy to pick back up, you'll find yourself picking things back up when the time is right. Sometimes inspiration hits while working on other stuff, and I say that's fine. Embrace that.

Commercial software is a bit more narrow in the selection of how one can start and stop on work (I call this "task shopping"), but being in tune with yourself and vocalizing that during standups/meetings/whatever can help. Can't seem to finish a task? Maybe the task was too big to begin with, scope/feature creep set in, or whatever. Create tasks for what you've gotten done and what needs to be done. Lay out some groundwork to explain how someone might pick up the new tasks. Do that and you'll find yourself "optimizing being able to leave."

"You must become comfortable with the grind-it-out nature of the last 10% of a project." I really don't align with this statement. Software can and should be a joy to do. Sure, there are aspects that can make it feel like a grind, but this is a question of framing. After all software development is technically data entry (don't think about this too much).

I appreciate your input on bfloat. I've always been under the impression that precision matters a lot when attempting to avoid local min/maxima if the landscape of the error function is jagged, but I suppose there's a good argument to be made that any floating point format can be used if the data, learning rate, network structure, etc are molded to match. Perhaps it's my perspective or maybe there actually isn't enough discourse on FP format being equally or more important factor to consider than just its affect on compute and memory requirements.

The use of FP64 could aid against vanishing gradients and just general information loss in deep networks, but that's probably comparable to using an atomic bomb to power a wind turbine. It certainly works, but is it the best way to go about it?

I personally think the use of mixed precision in deep networks will become more common as time goes on. I'm doubtful that all of a network really benefits from having large amounts of precision.

The precision used should match the requirements of the dataset, the training process, and the available compute. There are practical uses to 16-Bit FP training.

"Our findings demonstrate that pure 16-bit floating-point neural networks can achieve similar or even better performance than their mixed-precision and 32-bit counterparts." This is a very deceptive statement. Take 100 initialization states and train a FP16 vs a FP32 network, and you'll find FP32 will have an accuracy advantage. It's certainly possible to conclude this if a small sample of networks are trained. This paper goes on to state, "Lowering the precision of real numbers used for the neural network’s weights to fixed-point, as shown in [11], leads to a significant decrease in accuracy.", while later concluding, "we have shown that pure 16-bit networks can perform on par with, if not better than, mixed-precision and 32-bit networks in various image classification tasks." The results certainly do, but that doesn't really give an accurate evaluation of what's really going on here. A FP64 network can fall into a local minima and be outperformed by a PF16 network, but is it correct to say the FP16 network is better. I'm getting a lot of mixed signals.

I feel like, "significant implications" is quite a stretch.

A few concerns: Besides figure 3, other results do not provide side-by-side test vs validation accuracy to attempt demonstrate the network is not overfit, and the only mention of normalization was the custom batch normalization operation.

This may more be a rant about the current state of ML, but in a perfect world, we wouldn't use GPUs/would enforce deterministic calculations, results would be replicable, we'd train hundreds if not thousands of networks to draw conclusions from, we'd better understand how to visualize network accuracies and overfitting, and all datasets would be free of bias and accurately generalize the problem attempting to be modelled. We can dream.

I didn't feel like the parent comment was worth downvoting, and I share most of your perspectives. I agree that AI safety is ripe for contributions, but I'd personally argue AI safety is a very wide and encompassing topic that requires a good foundational understanding of AI. In the context of ML, AI safety is applicable to the entire process; from dataset creation, training, utilization, etc.

In regards to your comment on AI safety being "underutilized", my thoughts are that it's just simply difficult to do. Let's put aside all the difficulties of training, verification, etc and just look at the data problem.

If you wish to make certain that your system meets some given AI safety standard, then you must somehow prove two things: the data the model ingests when deployed will always return the correct response and that the dataset composes/generalizes the data the model will ingest when deployed. For simple problems, this may be doable. For complex, multidimensional problems wherein the dataset must only hope to generalize the complex input it will encounter during deployment, this may be next to infeasible.

I'm definitely getting off topic here, but bias of all kinds exists even human operated systems eg car. I can't say I've ever seen a firetruck stopped on the highway before, but perhaps I'd know what it is and how to avoid it. If a dataset does not contain that event, how can we be certain an AI system would understand? I'm not sure if it's possible to create a dataset that will be without bias in the case of complex problems, but I'm certain we can create one that's performant at driving than I. So the questions of "how safe is enough?", then proving/demonstrating that safety, and more are particularly open topics. I enjoy making the point that there is the lack of rigorous standards for humans, as we hold computers to far higher standards, but ML models probabilistically navigate decisions similar to us.

I'm sure this reply could extend further, but this and more are my defense of why I believe AI safety is a wide topic. None of the above should dissuade beginners from exploring the subtopic, but it's certainly not something you'd be able to learn first without strong, foundational context.