HN user
79966b
Would this work have the potential to speed up encoding/decoding of the PAR2 format[0]? This format is widely used to protect datasets against bitrot and small losses, but is held back because of the significant compute overhead when dealing with large datasets.
Our scenario is trying to produce a diagram during CICD. We have ongoing changes that happen to our SQL codebase, and it's a constant problem to align our documentation and diagrams to that. If we could see the current ER diagram of 'main' branch, and ideally also any given historic commit, it would be huge for our product owners, as they could consume the diagram and understand the current schema without disrupting the engineers. The CICD pipeline would ideally spit out a static page of the ER diagram during build, either as SVG or HTML.
Dbeaver and PGAdmin's diagramming functions aren't anywhere near as advanced as what you've built here, and fall apart for large diagrams. Your team has a lot of smarts!
This tool is amazing for us. There's so many pieces to this - the capability is a big step forward for architecting databases.
Rudimentary compared to what you've done, but is it possible to take an existing database schema, developed either in the Supabase migrations style or another tool like Flyway, and draw the diagram? That alone is huge for us in an enterprise setting, and would allow us to communicate the structure of the database to business folks (especially large databases). How does the tool build that from migrations currently?
Also forgot to say, love the project and love the objectives! Allowing Postgres to do it all means so much from a server management perspective - imagine not having to manage any redundancy/performance/analysis outside of the health of the Postgres box - fantastic :)
I was interested in how you do authentication, but currently the 'Omni_web' link & readme is missing. Suggest you could use the pgjwt[0] approach for this for simple logins in the short term, but supporting OpenID Connect would be a larger engineering effort.
This is all I could find: https://www.mouser.com/ProductDetail/Kingston/KSM56E46BD8KM-....
RE contributing to neonmodem, I was thinking about it! But baseline NNTP, as it sits today, is a fetid pit of spam, and I don't think it would add value. In fact, spam is, I believe, a far bigger problem with these networks than the technical distribution of messages over P2P. I took a sample of a random Usenet group today: [0] - but I really struggled to find one to post an image of here because even a lower-spam group that I found (e.g. alt.politics.uk) was full of profanities in the subject lines of the posts.
I think superhighway84 remains no/low-spam because of the technical hurdle of connecting. I don't think you've got any inbuilt spam protection? Plebbit [1], full of spam. The innovation of Reddit is arguably that people love the power-trip that comes with moderating a reddit group, and will do it for free - there's been no shortage of moderators to replace the protesters in the latest rebellion [2]. Hacker News, where I get to talk to smarter people than myself - very well/heavily moderated [3]. The SomethingAwful forums just resorted to charging everyone $10 for an account when they started having a spam problem, and that happily paid for the hosting costs and a life for the main admin for years.
To deal with the spam, you need some kind of filter where users can't just create thousands of accounts, especially in the age of LLMs. Logging in with a social account is the obvious one - Github/Facebook/Google have expensive processes in place to reduce the deluge of spam accounts, but some obviously creep through. Do you then run on an algorithmic chain of trust, promoting posts based on the quality/ratings of the individual's contributions elsewhere? If you do this, you're creating a system to be gamed. Running on invites only is another potential solution, but then it's difficult to start the gravy train of quality posts - who wants to apply effort to talk to nobody? Do you instead run a pyramid scheme - charge $10 upfront, but give a share of the site's ongoing revenue to those who get their posts upvoted, Twitch/Youtube/Instagram style? This to me seems like the one solution that could potentially displace Reddit, but I lack the personal belief/gusto to make it a reality.
Even if you managed to register and motivate a thousand decent posters, I don't have a clear view of how you keep topics on track within a group without a human moderator, but some research has been done in using LLMs to pre-rate posts based on the history of the group. But if the LLM is agendaless, you obviously get a groupthink echochamber. Give it an agenda, and you start dealing with bias - not every post of value is war and peace, and sometimes you just want to thumbs up a funny cat.
Please forgive the above musings if they're low value. I feel like I have no answers, only problems and questions, and I believe I'll be posting on Hackernews for tech, Instagram for comedy, and Facebook groups for special interests (e.g. car repair) for some time to come.
[0] https://raw.githubusercontent.com/dvasdekis/images/master/20... [1] https://news.ycombinator.com/item?id=36203610 [2] https://www.forbes.com/sites/barrycollins/2023/07/21/reddit-... [3] https://news.ycombinator.com/item?id=34920400
Thank you very much for the thoughtful reply.
RE the libraries, while it is true that I can't find anything made specifically for Usenet in Go, Usenet itself is just an extension of the Telnet protocol[0], and there are Telnet clients in Go[1] and Node[2]. It probably isn't simple, but I'm sure working with OrbitDB wasn't easy either!
RE the resilience of content on Usenet, the vast majority of binaries are heavily encrypted, and doesn't make sense to anyone without the key, despite being conveyed en-masse between the world's 10-ish full-scale Usenet backbones [3]. I'm proposing that the backend of a service that makes use of Usenet could be similar, with a single 'background server' on one trusted machine enough to continuously push the history to Usenet. A regular user client could then search for the latest version of this history and quickly refresh their side from Usenet, regardless of the status of IPFS at the time.
RE democratic access to technology, at least with Superhighway84 it was very expensive for me to actually run the software, as I have a small allocation of bandwidth from my ISP and not much I can do about that in my area, and I ultimately had to delete it due to ongoing transfers of 3GB/day running the IPFS node. Quiet itself notes a limit of 30-100 individuals with its application - I'm proposing that using the one remaining federated multicast technology with some modern encryption might help with issues around blasting data everywhere from a bandwidth-constrained environment. I know that definitely in Africa, there are ongoing issues with bandwidth and networks that we forget about in the West. Usenet, with extremely lean network overheads, could be part of the answer.
I do agree with your vision of a future of truly peer-to-peer technologies, but for those of us who are bandwidth-constrained or otherwise limited in our access to those technologies, having a technology-agnostic application that just 'does magic' to do whatever it needs to do with your content is what's going to make a majority of users happy.
[0] https://www.itprotoday.com/windows-78/how-can-i-use-telnet-a... [1] https://github.com/reiver/go-telnet [2] https://www.npmjs.com/package/telnet-client [3] https://svgshare.com/i/oti.svg
Would love to see some support for this with MiSTeR setups for FPGA emulation
Could I ask, with these p2p/federated projects (and thank you mrusme for helping me to get superhighway84 running! dope personal homepage, big fan of yours) why doesn't anyone use Usenet itself as resilient backup storage?
The Usenet network itself is always online and highly resilient - most providers offer ~5000 days of binary retention, and endless retention on text - and great bandwidth to boot. If a user doesn't have Usenet, or the Usenet isn't at the 'current' timestamp, that's where the Tor/P2P layer could kick in. You would only need a single server (with a private key, trusting the public key in the main executable) that continuously archives new posts to Usenet to make it work.
It isn't like farmed salmon is a crucial staple food without substitutes - nobody will starve without it. There are plenty of other fish breeds that are less damaging across their production lifecycle.
Farmed salmon is a luxury good and we shouldn't be harming the environment for that.
Unrelated blast from the past, it looks like Fragile Allegiance is getting a spiritual successor: https://www.fragilecontinuum.com/
How did you get so many great company logos onto your website? Did each one have to go through a legal approval? Was a struggle for us at my old startup.
Wonderful, thank you!
Would love to have a tutorial on how to install and run this locally with a nice model, for those of us who are behind the 8-ball with torch, transformers, diffusers, llama2 etc.
I have this. But every time I go on holiday (like, proper leave the city holiday) it goes away immediately.
I believe it's stress related for me. Meditation and sleep hygiene works, but nothing else so far. YMMV, good luck.
Have a think then about modern encryption and consensus on top of Usenet, which is the SMTP-compatible message queue of your dreams (in terms of available capacity)
Apologies - you're correct, but it's a very early version on Github. The author hasn't updated the public open source repo in yonks.
Suggest checking out the sqlparse library for a way to do the different flavours without needing to address each case directly: https://github.com/andialbrecht/sqlparse
The worst part about this is that blind Reddit users are being left completely without a solution, and won't be able to read Reddit at all. Multiple attempts at engagement with Reddit itself have been ignored. See the discussion here: https://www.reddit.com/r/apolloapp/comments/144f6xm/apollo_w...
So sad he never went Open Source, it would have been so instructive for us mortals
I personally wonder if some heart attacks have been reclassified as cardiac arrests or other outcomes as our understanding of heart health has advanced?
It's just a thought, I don't have any data to support this. Not sure if anyone has any insight.
A new CDC (Change Data Capture) approach open-sourced by Google? Quickly I, a subsequently disappointed data engineer, started reading
Logging on to Discord now - I think I can help out with this. Thanks for the welcome! :)
Hi guys, love the idea and the code examples look simple and understandable. Nicely done!
One of the reasons we use proprietary pipelines is the automatic 'service principal' login benefits that exist on e.g. Azure Devops, where the pipeline doesn't need to authenticate via secrets or tokens, and instead the running machine has the privileges to interact directly with Azure. (See https://learn.microsoft.com/en-us/azure/devops/pipelines/tas... particularly "addSpnToEnvironment" parameter). I'm sure other clouds have something similar.
Running the same pipeline locally, there are ways to synthetically inject this, but there's no ready support in your framework yet for this (as ideally you'd have an 'authentication' parameter that you can set the details for). Is something like this planned?
Migrants often come to the country with the physical skills, but not the certifications to do work like construction. I used the word 'allowed' in this context to mean that they, generally, lacked the ability to negotiate the complex modern construction standards of the Swedish state.
Mankind has been building homes for itself for 1.8 million years.[1] It's the norms of what constitutes a home (electricity, indoor plumbing, double glazing, certification) that have shifted, and I do not believe the migrants stood any chance of building homes compliant with modern swedish codes.
To answer your question, I believe you could build a home better than any caveman on your days off. But I think you're saying you don't have the money or time to build a modern, compliant home with the features I mention above, or to purchase the land for it.
There is plenty of rural land on the outskirts of Malmo. Sweden allowing a ghetto of poorly-developed, noncompliant houses would have been better than the tent-city I saw.
When I lived there, I was living in a noncompliant shipping-container construction that the government had erected hastily for overseas students with special permission, and it was more than enough. I believe the migrants had sufficient skill to build shipping-container homes, but there wasn't the political will to allow this bypass of swedish building compliance and due process.
[1] https://fivethirtyeight.com/features/who-built-the-first-hou...
Lived in Sweden (Malmo) during the Syrian refugee crisis. Watched in horror as my Swedish friends swung from refugee-welcoming to authoritarian "Nej!" in just 6 months.
Why? The rent doubled, the buses were full, taxes increased and crime sky-rocketed as immigrants who didn't have enough to eat crowded on to Sweden's generous social benefits scheme. The swedes, with a proud tradition of construction unions, heritage planning and social consensus didn't have the ability to reach an agreement on how to feed and house so many immigrants so quickly. This led to the migrants literally building a shanty town around the Malmo benefits office, instead of being allowed to build homes for themselves, as the EU refused to support the lone member state that did the right thing.
Proportionally, this influx would be like 5 million people arriving and applying for asylum in the US in a single year. [1]
As I witnessed these kind and tolerant people transform into the racist bigots this lady encountered, I think things would have very been different if the planet had treated the 2015 refugee crisis as a common problem, and not dumped it all on Sweden.
No, there is no excuse for racism, or for the lack of action in this case. But there are forces that provoke it, and every country that didn't take in a proportionate number of refugees during this crisis shares the blame for the social mess that Sweden is in today.
[1] https://www.brookings.edu/research/the-rise-of-sweden-democr...
I have 2x 4K monitors, one vertical, so Windows turns most 4K images into a blurry mess as it tries to fit them widthwise and lengthwise.
This, on the other hand, still looks stunning.