ownCloud Infinite Scale: Go instead of PHP, microservices instead of LAMP

https://www.heise.de//news/ownCloud-Infinite-Scale-Go-statt-PHP-Microservices-statt-LAMP-5029244.html

by veddox • 6 years ago

104 119 6 years ago

veddox4• 6 years ago

Here's a more detailed (German) discussion of what the rewrite entails: https://www.heise.de//news/ownCloud-Infinite-Scale-Go-statt-...

I wonder what this will mean for the competition with Nextcloud? So far, NC seemed to be more active and slightly better than OC, but this move may change the playing board quite drastically.

uselpa• 6 years ago

Why would that change the competition? I don't think that at the technical underpinnings have been a problem to rollout one or the other. Honestly curious.

8fingerlouie8• 6 years ago

Other than possibly gaining performance from Go vs PHP, as well as less memory consumption. I have no idea how large scale Next/Own-cloud implementations behave, but i'd assume you just throw more hardware after it.

What will change though is the relatively well tested and hardened code that is Owncloud today. Switching to Go doesn't just magically make your webapp secure.

piaste3• 6 years ago

I run Nextcloud on a Pi 4 and it's a struggle to get acceptable performance out of it. The ecosystem is nice, but I would absolutely trade a few less plugin for a smoother file management experience.

I will definitely be checking out OCIS, assuming I find no reason for concern from a license / privacy / security scale.

Golang isn't an automatic guarantee of performance, and "microservices" has me pause in the context of a personal server, but it happens that most of the Golang-based applications I host tend to perform extremely well. This coming from someone who strongly dislikes the "simple is better than correct" design of golang.

KingOfCoders3• 6 years ago

I have so many problems with core syncing with my NextCloud here - I wish I had used something else. But I guess "competition" is over who has more features.

pestaa3• 6 years ago

What issues have you encountered?

I can't imagine using anything else, it works so flawlessly for me on multiple devices across different platforms.

choeger3• 6 years ago

It is obviously a good idea to use a safe language that compiles to machine-code for something performance-sensitive. But I do not quite understand why someone should choose go over rust for any large-scale project.

switch007• 6 years ago

Why Rust over Go? Imagine I'm the CTO of an ownCloud-like company who is showing a strong preference for Go.

fxtentacle• 6 years ago

I'd also go with Go, just because of the amazing standard library. The ability to reuse part of Google's production code for HTTP / JSON is a huge win in my book.

Also, Go has been around for long enough that you'll find libraries and tutorials even for rare issues. When I evaluated Rust, I was missing that. Their ecosystem, while highly motivated, isn't mature enough yet for infrastructure level code.

choeger3• 6 years ago

You should choose rust for the long term. The type system will pay off any initial investment. Trust me, I am maintaining a large python code base. The number of problems that appear "organically" and would have been caught with a good type system is staggering.

tmikaeld2• 6 years ago

It has more to do with maturity and reliability.

Go has mature and fast standard libraries included in the language for web-related things like http, gateway/api, proxies and above all encryption.

For rust, you'll need quite a lot of crates with a lot of unknown variables to consider.

Why do large businesses choose Webpack when there's a lot of faster, more feature-full alternatives? Because it's reliable and maintained.

hnlmorg3• 6 years ago

Because Go is more mature than Rust for web development.

I get Rust is the current trendy language and people on HN love to argue that $TRENDY should be used instead of $BORING but this is exactly the kind of domain which Go was built for and it actually does a pretty fine job of too.

choeger3• 6 years ago

Rust is not just trendy. It is inherently more powerful when it comes to express and check invariants. I had thought that after decades of buggy dynamic code, people have come around to value that. Comparing go and rust, I see two statically compiled languages with many different and interesting features. But one has a strong type system and the other has a trivial one. I'd choose the strong one for any new project that I expect to maintain for years to come.

sondr3• 6 years ago

Another thing to consider is that Go is much easier for beginners to contribute to, I've contributed to a couple of Go projects without having written a single program myself. Rust is a different story, and often larger projects tend to abstract things behind type level "magic", which for a beginner/drive-by contributor is a big ask to understand.

bilekas• 6 years ago

I would argue that go has a more strict policy for language proposal changes too.. This can be good or bad depending on situation.

I would like some more article in OP about the process of 'rewritting' it would be more interesting than all the buzzwork bizspeak.

paxys• 6 years ago

If your primary use case is setting up a web server then there is no reason to choose Rust over Go.

diveanon• 6 years ago

Have you ever done web development with Rust?

I have. Rust is a great language and I really enjoy working with it, but it is far from mature as a web development language.

Go is the correct choice here if those are your only two options.

tscherno2• 6 years ago

All major components communicate via grpc so rust is not out of the picture. GC-less languages might make sense for storage-drivers.

yyyk• 6 years ago

Lets reverse that - why write in a language without a GC when you can write in a language with a GC? Rust's advantages shine when it's time to go low-level, but being without a GC is a disadvantage otherwise.

This application isn't merely 'performance-sensitive', it's a very particular form of performance - performance over network (I assume that's the main bottleneck here). Better write in a ecosystem all-but-officially optimized for networked applications rather than lose developer productivity dealing with low-level stuff.

8fingerlouie8• 6 years ago

I just shut down my Nextcloud instance yesterday. Not that there was anything wrong with it, but all i really need is a file synchronization "platform", and there are others that don't open quite as large an attack surface as a complete nextcloud instance does.

Not saying Nextcloud is insecure, and i've never had any problems with it.

Currently trying to decide if Seafile is the way for me, though i dislike it's on disk fileformat (IIRC it's some adaptation of Git).

For now i use Resilio Sync.

dmacvicar• 6 years ago

I am curious if you have tried https://syncthing.net/ ?

pqs• 6 years ago

I made the transition from Resilio to Syncthing and I am very happy.

8fingerlouie8• 6 years ago

I have, but syncthing doesn't have iOS clients, or at least it didn't last time i checked.

There was a client on the appstore called "f-sync", but that is gone now.

Also, Resilio (with paid license) supports features that Syncthing didn't support like selective sync, and encrypted folders, which allows you to share a folder with someone for redundancy and have things stored encrypted on disk.

andreasley2• 6 years ago

We've been using Seafile for two years at my workplace. So far it's been very reliable and much faster than Nextcloud for file transfers. The current drive clients work well, but aren't as polished as Google File Drive Stream, for example. On macOS, I hope they will transition from macfuse.fs to the new FileProvider framework soon.

Seafile's file storage format has the advantage that it's easy to revert a file or folder to some earlier revision, for example after accidentally deleting files.

RL_Quine3• 6 years ago

How do you justify the risk model of the chinese developer?

teekert2• 6 years ago

Yeah, NextCloud is overkill for that. I see it as the back-end of the phones of my family though, calendar/contacts sync, auto picture uploading (which you can view on a map even), sharing files within the family, of-site backups for everyone. I'm just waiting for NextCloud Talk to support federation so I can talk between servers (I say family but it means different families, in my language there is a word for just parents and kids and when you also include more ("gezin" vs "familie"), not sure what to use, but we have multiple domains and servers (all in my basement though ;))

dsr_• 6 years ago

US usage: parents and kids = "nuclear family"

Everyone who lives together = "household"

Family not living together = "extended family"

ktpsns2• 6 years ago

Interesting – for basically the same reason we made the inverse switch from Resilio to Nextcloud with a small startup team with <10 devs.

Nextcloud provides us custom shares, user groups, public shares, URLs, better local clients, compared to the Resilio performance. Resilio was especially bad on Linux (but worked remarkably well on my android with a large SD card). With Nextcloud I can even choose to use WebDAV only if I don't want to mess with clients.

8fingerlouie8• 6 years ago

I guess it depends on your workload. My Nextcloud was only for myself and my family, and we only used it for "files on the go".

Calendar/contacts is handled by iCloud (Apple household, it's a Danish thing...)

Notes are handled by whatever each person finds the easiest. My wife defaults to the iOS notes app, i switch between various clear text editors.

File synchronization on desktops/laptops is handled by Synology Drive, which syncs beautifully whenever the machine is connected to our LAN, either directly or through VPN.

The only problem i needed to solve was ad-hoc access to files on mobile devices, preferably without opening ports, and since VPN doesn't always work from other private networks (ip scope clash usually), i chose not to use Synology tools for this. Besides, Synology Drive doesn't support selective sync, and while documents probably wouldn't be a problem, synchronizing gigabytes of books to my phone isn't really an option :)

Resilio on Linux does have a nasty habit of doing disk IO all the time, a habit that syncthing doesn't have. When i look at running processes, Resilio on linux is constantly using 2-5% CPU.

lillecarl• 6 years ago

If you're using resilio I would like to recommend Syncthing. Developed out of neutral Sweden, files stored as-is.

m463• 6 years ago

I run nextcloud internally in a container.

Thing I don't like is that it expects to be hooked to the internet, and it nags you about apps you should install but you have to install them from their cloud/app store.

Now there is a way to install the app store on your server, but it would be nicer to get started without having to buy into all that. So I just run the default apps.

piaste3• 5 years ago

Not ideal, but you could give it internet access, install all the apps you want, and then isolate it again (remove it from the docker non-internal network or whatever).

As long as the apps you install don't need to make web requests on their own, they'll work fine.

8fingerlouie8• 5 years ago

I briefly considered putting up separate VMs for front and backend containers, but ultimately decided against it. Instead I have a “web” docker network where my nginx reverse proxy runs, and a “services” docker network where I run the stuff I proxy.

Databases and other containers needed for the backend services run on another network as well, so if you make it inside nginx you will (of course) be able to access my already exposed backend services, but no direct access to databases and other services.

Before shutting down Nextcloud I used to have a resilio container running, and I would mount the data from that inside Nextcloud, so no direct contact between the two.

But then again, I just need access to files through a browser, and don’t need any of the advanced features of Nextcloud, so I’m still trying to find a better match. Looking at seafile if I ever find the time.

jojo2000• 6 years ago

good and bold choice. but count at least a year to get visible results.

encom• 6 years ago

This is terrible news. I hope they keep the PHP version up to date, or someone forks it. I've used Owncloud over Nextcloud, because it seems more focused on just file syncing, which is all I want. Maybe it's time to migrate.

Aissen• 6 years ago

This reminds me of cozy cloud's rewrite from node.js to go (https://github.com/cozy/cozy-stack ); this seems to be a rewrite from php to go, as well as re-architecturing it:

https://github.com/owncloud/ocis

Pydio also rewrote itself from php to go with pydio cells: https://github.com/pydio/cells

aneutron• 6 years ago

Anyone has a link for the actual technical architecture ?

This talks about "three-tier architecture" and "no database needed", which is obviously pure marketing spiel.

Are they re-inventing the wheel by implementing a distributed database directly into ownCloud ? I don't get it.

tmikaeld2• 6 years ago

Seems it has a modular storage back-end[0], the default uses a built-in storage engine local files with oCIS storage driver [1].

[0] https://owncloud.github.io/ocis/storage-backends/eos/

[1] "The default oCIS storage driver deconstructs a filesystem to be able to efficiently look up files by fileid as well as path. It stores all folders and files by a uuid and persists share and other metadata using extended attributes. This allows using the linux VFS cache using stat syscalls instead of a database or key/value store. The driver implements trash, versions and sharing. It not only serves as the current default storage driver, but also as a blueprint for future storage driver implementations."

tscherno2• 6 years ago

With this design we aim for the storage to become the single-point of truth and scale-out as the metadata is spatially close to the files.

ThePhysicist5• 6 years ago

That sounds scary. Going from a monolith to a microservices architecture doesn't seem like a great move for a software like ownCloud. I think rewriting this as a Go application with a modern JS-based frontend might make sense, I just don't get why you need to throw microservices into the mix. But let's see if they can pull this off, historically such rewrites don't have a great track record though.

jeandejean• 6 years ago

I'm curious about past examples of failures, can you provide some? Thanks!

bayindirh• 6 years ago

The thing with Owncloud/Nextcloud is the complexity of the system. When many users are working with the files at the same time, there's a lot of messaging going on in the backbone. If people are working in shared folders, that traffic goes visibly up.

Messaging between systems is hard. When things are async, it's harder. When there's async concurrency, it becomes spicy.

I'm managing a Nextcloud instance at the office. Even with this relatively mature state, it can bork itself pretty decently. Also, it has separation of responsibilities and a system admin is not an admin the traditional sense so, a lot of things are hidden between logs and other layers.

Complicating this further by adding microservices is a brave idea at least. We're going to see what's going to happen.

I hope they succeed but it's not a piece of cake.

fundatus• 6 years ago

Netscape for example: https://www.joelonsoftware.com/2000/04/06/things-you-should-...

ThePhysicist5• 6 years ago

There are many examples where complete rewrites have caused problems and killed or hampered companies, Netscape and Firefox come to mind.

Regarding microservices I don't know any OS projects that have failed because of that (because there aren't many OS products that use a microservices architecture), but personally I witnessed many corporate software projects burn and fail when trying to rewrite an "ugly" but functional monolith as a collection of microservices.

The thing is that the energy which you invest into rewriting your entire software in a new stack could also be invested into improving your existing stack. LAMP is not dead and actually evolving quite well, so there are definitely ways to improve the stability and speed of a LAMP application without rewriting it in Golang (even Facebook still uses PHP, albeit with some modifications). Having written extensive applications in both scripting languages and Golang I can say that the latter has its advantages but is also slower to develop in and in many respects not as agile as a scripting language like PHP or Python.

Doing a rewrite of a popular project with thousands of deploys in the wild will also force you to split your attention between keeping the old system running and building the new one. Since most of your users will want to migrate from one system to the other without losing their existing data and configuration you'll have to ensure that there is a clear migration path that works, which can be quite hard. Also, all external systems that are interacting with existing installations (e.g. via APIs) will also need to be supported by the new system.

In the end it might be possible to make it, but in my humble experience I think a complete rewrite is almost never a good idea and the energy you put into it is often better invested in making gradual improvements to your existing codebase. But that's just my 2c of course.

dosethree2• 6 years ago

Incremental rewrites are just always better. The issue with full rewrites is that you have to rewrite everything - which I suppose sounds good to some engineers, but I dread being forced to spent weeks on parts of the product that are not an issue. Usually you do the rewrite to make things faster, more maintainable, and easier to add features. The issue is, the larger the rewrite, the worse you make all those things in the short-term.

cientifico• 6 years ago

The problem is not refactor vs rewrite. What most people oversees is the domain knowledge.

If the domain knowledge is on the code, you can only refactor. If the domain knowledge is on the team, then rewrite tends to be faster.

Most rewrites fails because the team rewriting is not the team that did the initial development.

twsted• 6 years ago

Really well said, this is also my own experience in many projects.

LukasReschke2• 6 years ago

Most rewrites fails because the team rewriting is not the team that did the initial development.

Looking at https://github.com/owncloud/core/graphs/contributors, most of the initial core contributors contribute now to Nextcloud instead.

I am interested in seeing how the rewrite affects end-users. Currently, it seems to be mainly focused on File Sync and I don't see things such as calendar or contacts management. (they totally could appear in the future though)

I'd assume that limiting the scope of components, makes a rewrite also easier. Nextcloud for example has a ton of hooks that allow you to write apps to customize the behaviour (want users to sign the ToS before downloading a share? Should be doable etc.). When you leave these out, implementing new things just got a whole lot easier :) (so I guess I am envious on the fact that they got rid of a ton of backwards compatible APIs to maintain :-) )

Disclaimer: Contributed to ownCloud for a few years, then to Nextcloud.

simias• 6 years ago

That's very insightful. It's also interesting that the tendency is to do the opposite: if it's your code you may not want to trash it completely and instead work on an incremental refactor, while if the codebase is foreign it's tempting to just get rid of it and do it your way.

Anecdotally I've been recoding from scratch one of my applications at work (converting from python to rust) and while it's a lot of work and it put new features on hold in the meantime I'm fairly optimistic that it's going to be a big win in the long term. Not only will the Rust code perform better and be more maintainable, the rewrite lets me address architectural flaws with the previous solution, some caused by lack of foresight and others by tacked-on features that weren't expected when the code was originally designed.

hedora2• 5 years ago

You’re describing the CADT (Cascade of Attention Deficit Teenagers) development model.

The longer I work in this industry, the more I’m convinced it is the dominant software development paradigm.

Search for “jwz cadt” to read more (can’t link it here).

ampdepolymerase• 6 years ago

There is nothing stopping them from offering a fat binary. Not everything has to run on Kubernetes.

cosmotic• 6 years ago

You suggest a 'modern' JS frontend... Microservices were a 'modern' architecture a few years ago. Whichever reasons lie behind choosing a modern JS frontend would similarly support (or not support) the microservice backend.

danpalmer• 6 years ago

ownCloud Infinite Scale neither requires Apache nor any PHP infrastructure

I don't know much about production web services in Go, but is it true that they wouldn't need Apache?

I work on a site using Python/Gunicorn and Node behind Nginx, and Nginx is really doing a lot of critical stuff for us, normalising requests, preventing certain classes of security issues, DoS attacks, etc. I wouldn't expect that sort of protection to be built into Go's standard HTTP server?

ThePhysicist5• 6 years ago

Go has good support for TLS and HTTP serving, so you can directly expose a Golang service to the Internet without a web server in front.

Many people still choose to put nginx or caddy in front as a load balancer and TLS termination though.

vbsteven• 6 years ago

As with most builtin language HTTP servers it is advised to run Nginx/Apache/Caddy in front of it.

I think what they mean with PHP infrastructure is the specific configuration required to make Apache work with PHP, in the old days this was with an Apache plugin, or more recently the php-fpm project.

kissgyorgy2• 6 years ago

Forget ownCloud, Nextcloud is orders of magnitude better either as a company and as software: https://nextcloud.com/

It is my favorite self-hosted software of all time. It's really a dream to use!

The AGPL license makes it impossible to disappear or becoming closed source, which is a huge plus!

teekert2• 6 years ago

I also like and run NextCloud, it is the "cloud" back-end for my entire family. That said, it seems to me that this means that OwnCloud en NextCloud are now truly diverging and OwnCloud is not just a less feature rich version of NextCloud. I think it is very interesting to see what happens to OwnCloud now. I mean, imagine OpenOffice being rewritten with different technologies from LibreOffice... I'd be interested once again!

alpaca1282• 6 years ago

No, thanks. I decided to stop using it after Nextcloud already got confused by a few of my Git repos, overwriting my local updates with the older version from the server. Not to mention the "don't sync hidden files" default setting that's simply not a good idea, to put it mildly. This new ownCloud product promises to handle that better("with millions of small files or thousands with hundreds of GB each, ownCloud Infinite Scale runs smoothly"), but I'll believe it when I see it.

So far I haven't found a sync tool that I feel like I can trust. I don't care about user group management or online office suites, but I absolutely want ~100% reliable file sync that doesn't randomly do nothing for 15 minutes without me knowing why(I care about my files and if I don't know what your software does with them I won't use it), or like NextCloud, falsely detect merge conflicts when 100% of changes came from the same client machine.

diarrhea2• 6 years ago

You have naked git repos in synced cloud folders, without using git remote functionalities?

That would be careless and you are actively begging for things to break that way, so I'm not surprised. In fact I'd bet any other service, Dropbox/GDrive/..., would also suffer from such issues.

kissgyorgy2• 6 years ago

What does Git have to do with Nextcloud at all?

veddox4• 6 years ago

I very much like Nextcloud too, and have set up/administrated three instances in various contexts (family, friends, work).

It has a lot of great features and good multi-platform support. I also like that the company is based in Germany, and I had good experiences contributing to their code.

However, I do have to say that the software is somewhat clunky and slow, and the clients can be a little rough around the edges. (The desktop client has a particularly nasty bug that likes to clobber git repos.)

So if ownCloud comes out of the rewrite as sleek and reliable as they claim, they would be worth taking another look at. Until then, I'll stick with my Nextcloud...

jbverschoor• 6 years ago

Good move. I'd never consider owncloud on LAMP.

mangecoeur• 6 years ago

They mention the involvement of research institutions - the Swiss universities use a common OwnCloud system (Switch Drive) for researchers to store and share files within and between institutions; which generally works very well. So seems likely they are involved since they must be one of the biggest OwnCloud users in the world.

joshspankit• 6 years ago

Huge respect for this direction: I’d put ownCloud on the “too old to seriously consider” pile, but they’ve potentially not just started fresh, but brought forward everything they’ve learned in all the time since they started.

Done right, this means they are a serious, safe, and secure option.

veddox4• 6 years ago

I'm pretty sure this is an attempt to regain some of the market share they lost against Nextcloud. IIRC, the latter's popularity has been growing continuously, while OC has been stagnating/shrinking.

Let's see if they can pull it off.

ThinkBeat• 6 years ago

There is a lot of hate for PHP, and monoliths. The great thing about WordPress, mediaWiki, Drupal, nextCloud, and many other open source web solutions is that you can get up a running for $2 -> with little computer knowledge.

A lot of cheap consumer hosting services offer managed PHP/MySql stack very cheap. A lot of them have variations of one click installs of various kinds.

You pay the monthly fee and do not have to know about Linux, PHP or MySql.

If you are familiar with PHP, MySql finding a place to host it is still really cheap and available.

A common argument would be that you can get a VPS for $5 maybe even less. Then you have to know about what a database is, what version of PHP, when to do upgrades of all of them. Managed VPS solutions are usually fairly pricy.

With the new version of ownCloud, based on Go and micrnosercivices the barrier of entry is much higher.

The best bet will be to buy managed nextCloud hosting from someone.

While PHP is often scoffed at, the ecosystem that is ready and waiting to help host it for each is amazing. It enables a lot more people to be able to get involved and get inspired and maybe learn more and get start contributing.

If you are writing software that needs hosting but want as many people as possible to be able to use it then PHP is still the best bet, even if not the sexy bet

dosethree2• 6 years ago

IMO there is not a lot of hate for monoliths. Service oriented architectures and their successor microservices have experienced a lot of push back from teams that have tried it and it doesn't work out for all applications.

PHP - I mean, it gets the job done. I would never start a new company and choose it, but if you're already on it, its good enough for most things. Go can be far superior depending on your needs.

hedora2• 5 years ago

The big issue I have with deploying PHP is the sprawling, historically low-quality ecosystem.

Maybe it’s gotten better, but last I checked, running RATS (which might not even exist any more) against random, popular PHP code bases would find piles of zero day vulnerabilities.

It’s been over a decade, so I might be being unfair.

LargoLasskhyfv• 5 years ago

Hannisch wat verpasst, oder war dat jezz allet op deutsch?

ownCloud Infinite Scale: Go instead of PHP, microservices instead of LAMP

Related Stories