jborak

I already had 2x 5070's that I had purchased a year or more ago, so getting an additional two to fill up the PCIe slots on the motherboard seemed reasonable.

I did some math/shopping as well. To get 48GB of VRAM you can get 2x 3090s but that is $3k. A single 5090 is $4k but has 32GB, great for running models like Qwen 27B but maybe nothing else depending on your model settings. Already having 2x 5070, where each card is around $600, it made sense for me to get two more which was $1200 and the memory speeds aligned.

The best value option if you're building from scratch is go with 5060 ti (16GB VRAM). Each of those cards are $570/each on Amazon, cheaper than 4x 5070's. Only downside is memory speed is slightly slower, but you wind up with 64GB of VRAM and you can run big models and small models alongside each other comfortably.

In my setup I ran Qwen3.5 9B for fast inference on simple things and Qwen3.6 27B Q6 for coding work. But I ran into stability issues, so I use llama-swap to dynamically swap models. But with 64GB of VRAM, you wouldn't have that issue. There is overhead to loading LLMs into VRAM that isn't clear, so having extra VRAM is a helpful buffer.

I'm using 4x RTX 5070's and first-gen AMD threadripper (1950X) to run Qwen3.6 27B (MTP) Q6_K with llama.cpp and it works great as a daily driver with Pi. Around 50-60 toks/sec. I also connect a few other applications to it such as OpenWeb UI and recently set up Bifrost, an LLM gateway, to be the primary access point for the models I serve.

I've tried other models such as Qwen3.6 35B A3B and I've found that 27B works better for me when it comes to coding. It's slower being a dense model but the quality seems much better. Inference on my system for Qwen3.6 35B A3B is around 130-140 toks/sec, non-MTP, which is insanely fast!

You don't need 4x 5070's to run Qwen3.6 27B, three or maybe even two will work. However, I use MTP (multi-token prediction) to speed up 27B and that eats up more memory because the draft model requires its own context.

Another thing to keep in mind is that the tools you're using have their system prompts that are loaded into the model for each conversation. When I fire up Pi, working with the model is very snappy at start. When I interact with the LLM via Hermes CLI, it's much slower. That's because each prompt with Hermes is loading so much stuff (skills, tools, etc.) into the context and then it's there forever until the conversation ends.

I like running models at home for privacy, but I also like how there are no quotas, usage isn't a worry. If the future is "loop engineering" then you will be burning through tokens and $$$ using a cloud models.

My system idles around 200W and is around 350-450W when inference load is high. Decoding (token generation) isn't all that efficient, and your GPUs sit idle more than you think during inference. Advancements like diffusion may 1) speed up decoding and 2) let you utilize more of your idle GPU.

I run Packetriot: https://packetriot.com, which is an alternative to ngrok. It's been in operation for 6-ish years and reached $500/month in year two. It doesn't replace my salary (yet) but that's what I'm working toward. Thankful to have thousands of users and some big companies using our services or self-hosting our server.

This is a fun space to work in but there's lot of small competitors and opensource alternatives. In the first few years it was sometimes demotivating, but at some point I began seeing competitors bow out and Packetriot kept getting better and better each year.

This year I published a lot of updates across the platform. Our UX is better with a new web-based UI for our client and all of the features of the platform can be managed in the client UI. No more going back forth between the client and user portal.

I'm planning on releasing a community edition of our server in 2026 so that anyone can use our network server for free, for personal use or to evaluate for commercial use.

Our services were used for C2 as well. I investigated it a bit but eventually decided to just drop TCP forwarding from our free-tier and that reduced our abuse/malware reports for C2 over TCP to zero essentially.

One path I looked at was to use the VirusTotal API to help identify C2's that other security organizations were identifying and leverage that to automatically take down malicious TCP endpoints. I wrote some POCs but did not deploy them. It's something I plan on taking up again at some point next year.

Thanks for sharing this. I run packetriot.com, another tunneling service and I ended up writing my own scanner for endpoints using keyword lists I gathered from various infosec resources.

I had done some account filtering for origins coming out of Tor, VPN networks, data centers, etc. but I recently dropped those and added an portal page for free accounts, similar to what ngrok does.

It was very effective at preventing abuse. I also added mechanism for reporting abuse on the safety page that's presented.

I built a similar service as well called Packetriot: https://packetriot.com

Building these types of tunneling systems are great projects. You learn a lot and can master skills in many different areas.

Packetriot has been operating for five years and the first few years was all spent on performance and stability of the core networking services. As the software and network matured, I spent more time on the operations and maintenance, and automating as much of that as possible.

Recently I've begun building tools to detect phishing and potential malicious behaviors. This is a common problem that operators of these tunnel networks have to deal with. It's an interesting and fun technical area and helps make the Internet a safer place :)

I build a secure tunneling service called Packetriot ($2k/mo) https://packetriot.com.

Similar to ngrok with our own differences and approach. I also publish another product called Spokes Gateway which builds on the tunneling server and includes support for service meshes, high-availability, clusters and some other features.

I'm building a separate website for Spokes and its related software, hoping to publish it soon. It's eventual home will be https://spokes.network.

I use ProtonMail and overall it works pretty good and the value is great. The product is focused on security first so I think user experience suffers a bit. It's works fine but if you previously used a mail client like Spark on your phone you'll definitely miss it.

The mobile app is okay, it's good enough for reading and sending short replies. The web interface is alright. I like to use the bridge application from ProtonMail on my desktop and use a mail client like Thunderbird or Apple Mail.

https://packetriot.com

Provides HTTP/S and TCP tunneling and is a comprehensive alternative to ngrok. I started it in Jan 2019. $4.3k ARR in 2020.

It's great to have options and alternatives. I feel like calling your application a copy-cat is being harsh. What you've discovered is that there is a now a market of providers for the category of your application.

If there was only one instance of any sort of application or tool the world would be boring and we'd be stuck with only stale options.

A lot of people have seen the Clint Eastwood "Man with no name" westerns but if you haven't watched "Once Upon a Time in the West" I would definitely check it out. One of the best movies I've seen and if you're not fan of westerns (which I wasn't) you will appreciate the genre some more after seeing this movie.

I wrote a tunneling service called Packetriot that is similar. It's probably slightly older than inlets. However, it's not an opensource project. The tunnels are assigned static hostnames and the servers you connect to are static as well, so setting up CNAME and A records for custom sites are simple. You can host any number of websites or services behind a single tunnel.

Let's Encrypt is built-in and can manage all of your certs automatically. The client can serve static assets and upstream to app servers, so you can eliminate the need for an extra web server running in your environment.

There are packages for Deb, RPM, containers, mac and windows, and for almost all architectures (x86/amd64/arm32/arm64).

I designed the service for 24/7 operation so uptimes for tunnels are weeks at a time. They auto-reconnect if there's a connection drop. You can find it here: https://packetriot.com

It's built exactly for that. The tunnel is established in reverse over a websocket, so outgoing traffic from your PC is TCP:443 which looks like normal TLS traffic.

I've been using this for almost 6 months and haven't seen any throttling. Your Netflix, YouTube or any other streaming traffic will dwarf what bandwidth you consume, unless we're talking terabytes...

Hey HN, this is the author, thanks for checking out my project and any feedback. As I built this I found alternatives, here are some differentiators:

* access logs & metrics - these types of systems obscure this info (src IPs, etc...)

* system health checks - probes if the destination (upstream service) is available, if not, we'll email you

* self-hosted - want to host the edge server software on your own, you can do that

* specification - the plan is to publish the protocol and spec so others can write clients or integrations

Glad to help with questions, cheers.

DO has bandwidth included in the price of the droplet and if you exceed then you pay a fair price for the additional bandwidth. For the cheapest droplet ($5/mo) you are provided 1TB of bandwidth. What they also do is take the all the bandwidth across all droplets you purchase and make that your pool. So if you have 4 droplets, then 4 TB, if one of the droplets experience the most traffic like 3.5 TB, and the rest are zero, you are under your total bandwidth quota for the billing period.

I have 4 edge servers that are small VMs at the moment. For the number of users I have more than enough capacity. I also have a separate server that has more resources for the main site and APIs.

I use DO, altogether hosting costs are $40/mo for now.

Packetriot (https://packetriot.com) - around 4 months old and I haven't begun seriously marketing, but it generates $60/month and makes a tiny profit.

With it you can create a secure public endpoints (HTTP/TCP) from any network. The client can host static sites, reverse-proxy to other hosts, terminate TLS. It manages Lets-Encrypt for you. The higher-end tiers provides access logs, metrics, firewall rules, and service-health checks.

With a public endpoint you can pretty much do anything. I initially built it to reduce my hosting costs to something minuscule (which it did). I found alternatives as I developed it, but I was so interested in the problem I continued and created some features that aren't available in others. I still have more ideas in the pipeline.

I've been putting together video tutorials to showcase what you can do with it and how to use it. I'm focusing on people that self-host since most developers/hackers understand the concept. A lot of my registrations and customers have discovered Packetriot through my YouTube channel.