hemancuso

It does not stream in content, but I would guess the API might support that some day.

Hey all! This is something we've been working on for a while now. It's the future of native cloud storage on macOS. I run ExpanDrive (www.expandrive.com). We use a derivative of macFUSE (https://osxfuse.github.io/) on macOS.

Third party kernel extensions on macOS are on the way out. On M1 machines any third party kernel extension requires a reboot into recovery mode, lowering the security settings, and then finishing the install.

This is a terrible and unfriendly model for users (by design). This is the primary reason these Google/Box/etc aren’t shipping M1 versions of their software.

The replacement for apps like Google & Box Drive (and everyone else) is their new File Provider framework (https://developer.apple.com/documentation/fileprovider), which they quietly launched but didn’t publicize at all. It’s basically a technology that enables something like Dropbox Smart Sync (project infinite) built directly into macOS fully managed by APFS.

It’s a pretty big departure from the macFUSE style interfaces we've been using for years now. File Provider apps are userspace extensions that work with with placeholder files and on-demand hydration of remote files by the system via APFS. It’s a pretty cool framework, but pretty new (and a little raw).

Box/Google/Dropbox/Microsoft will all eventually be using it, but the rollout has been slow.

The whole macOS File Provider project is an off-shoot of the iCloud drive and iOS file provider projects, but with one key component they’ve added called the replicated extension that has an apple-blessed extension to APFS called “FPFS” (presumably file provider FS) that can intercept read/writes/etc to these on-disk placeholder files and download content (and thumbnails!) on demand. Provides data directly to spotlight indexes, etc.

Whereas FUSE is a really low level API exported into a safer user-space process, this is a little higher level. You give the file provider framework lists of directories, help it monitor changes, provide item info and then it will in turn issue callbacks into your extension to initiate downloads to a temporary location. Benefits

1. All on demand, driven by placeholders. “regular” sync downloads contents ahead of time, this is not the case with file providers. It’s more like network filesystems in this regard

2. Doesn’t eat up free space. Providers can mark their content as “evictable” which lets APFS know it can toss out the data if space gets low. But what I think is extra interesting is that when you mark the content as evictable it doesn’t even register as being used against free space. You could bring down a 10GB file from the cloud and your free space remains the same

3. Integrated with all the higher level APIs so that applications that open a file with swift/objective C don’t beachball while waiting for a download (for an open) to complete. They appropriate waits and expectations are in there now

4. All sorts of other stuff, happy to keep going

Hey all! This is something we've been working on for a while now. It's the future of native cloud storage on macOS. I run ExpanDrive (www.expandrive.com). We use a derivative of macFUSE (https://osxfuse.github.io/) on macOS.

Third party kernel extensions on macOS are on the way out. On M1 machines any third party kernel extension requires a reboot into recovery mode, lowering the security settings, and then finishing the install.

This is a terrible and unfriendly model for users (by design). This is the primary reason these Google/Box/etc aren’t shipping M1 versions of their software.

But that’s okay, they’ve got something better they’re working on. Just keeping it a little quiet right now since it’s not 100% done.

The replacement for apps like Google & Box Drive (and everyone else) is their new File Provider framework (https://developer.apple.com/documentation/fileprovider), which they quietly launched but didn’t publicize at all. It’s basically a technology that enables something like Dropbox Smart Sync (project infinite) built directly into macOS fully managed by APFS.

It’s a pretty big departure from the macFUSE style interfaces we've been using for years now. File Provider apps are userspace extensions that work with with placeholder files and on-demand hydration of remote files by the system via APFS. It’s a pretty cool framework, but pretty new (and a little raw).

Box/Google/Dropbox/Microsoft will all eventually be using it, but the rollout has been slow.

The whole macOS File Provider project is an off-shoot of the iCloud drive and iOS file provider projects, but with one key component they’ve added called the replicated extension that has an apple-blessed extension to APFS called “FPFS” (presumably file provider FS) that can intercept read/writes/etc to these on-disk placeholder files and download content (and thumbnails!) on demand. Provides data directly to spotlight indexes, etc.

Whereas FUSE is a really low level API exported into a safer user-space process, this is a little higher level. You give the file provider framework lists of directories, help it monitor changes, provide item info and then it will in turn issue callbacks into your extension to initiate downloads to a temporary location.

Benefits

1. All on demand, driven by placeholders. “regular” sync downloads contents ahead of time, this is not the case with file providers. It’s more like network filesystems in this regard

2. Doesn’t eat up free space. Providers can mark their content as “evictable” which lets APFS know it can toss out the data if space gets low. But what I think is extra interesting is that when you mark the content as evictable it doesn’t even register as being used against free space. You could bring down a 10GB file from the cloud and your free space remains the same

3. Integrated with all the higher level APIs so that applications that open a file with swift/objective C don’t beachball while waiting for a download (for an open) to complete. They appropriate waits and expectations are in there now

4. All sorts of other stuff, happy to keep going

Hey all! This is something we've been working on for a while now. It's the future of native cloud storage on macOS.

I run ExpanDrive (www.expandrive.com). We use a derivative of macFUSE (https://osxfuse.github.io/) on macOS.

Third party kernel extensions on macOS are on the way out.

On M1 machines any third party kernel extension requires a reboot into recovery mode, lowering the security settings, and then finishing the install.

This is a terrible and unfriendly model for users (by design). This is the primary reason these Google/Box/etc aren’t shipping M1 versions of their software.

But that’s okay, they’ve got something better they’re working on. Just keeping it a little quiet right now since it’s not 100% done.

The replacement for apps like Google & Box Drive (and everyone else) is their new File Provider framework (https://developer.apple.com/documentation/fileprovider), which they quietly launched but didn’t publicize at all. It’s basically a technology that enables something like Dropbox Smart Sync (project infinite) built directly into macOS fully managed by APFS.

It’s a pretty big departure from the macFUSE style interfaces we've been using for years now. File Provider apps are userspace extensions that work with with placeholder files and on-demand hydration of remote files by the system via APFS. It’s a pretty cool framework, but pretty new (and a little raw).

Box/Google/Dropbox/Microsoft will all eventually be using it, but the rollout has been slow.

The whole macOS File Provider project is an off-shoot of the iCloud drive and iOS file provider projects, but with one key component they’ve added called the replicated extension that has an apple-blessed extension to APFS called “FPFS” (presumably file provider FS) that can intercept read/writes/etc to these on-disk placeholder files and download content (and thumbnails!) on demand. Provides data directly to spotlight indexes, etc.

Whereas FUSE is a really low level API exported into a safer user-space process, this is a little higher level. You give the file provider framework lists of directories, help it monitor changes, provide item info and then it will in turn issue callbacks into your extension to initiate downloads to a temporary location.

Benefits

1. All on demand, driven by placeholders. “regular” sync downloads contents ahead of time, this is not the case with file providers. It’s more like network filesystems in this regard

2. Doesn’t eat up free space. Providers can mark their content as “evictable” which lets APFS know it can toss out the data if space gets low. But what I think is extra interesting is that when you mark the content as evictable it doesn’t even register as being used against free space. You could bring down a 10GB file from the cloud and your free space remains the same

3. Integrated with all the higher level APIs so that applications that open a file with swift/objective C don’t beachball while waiting for a download (for an open) to complete. They appropriate waits and expectations are in there now

4. All sorts of other stuff, happy to keep going

It remains the case even if you’re only a few ms away.

As a developer that supports B2 (I write ExpanDrive) I think it’s great that they are moving on from an API that doesn’t expose any extra value.

That being said, I wish B2 performance was better. Throughput is dramatically slower than S3.

I think the APFS snapshot integration is easily the coolest part of Arq 6. Arq now has access to a special Apple entitlement to take full-desk point-in-time snapshots of an APFS container and backup from that. It's like what time machine would've/should've been, for the cloud.

For the most part, I'd just be using macOS WindowServer. Which isn't very helpful. That's part of what is driving the question about whether one big monitor is ultimately better.

I’ve been playing with the beta of Infra, which serves a similar purpose. Also with checking out

https://www.infra.app/

They have access to a kernel module and entitlement that only they have access to, com.apple.fileutil - and Apple has said they aren't giving out any more entitlements for it.

Fine how?

Curious if anyone knows what Dropbox will do about SmartSync. KAuth extensions are likely out as of 10.16, and the Endpoint Security extensions don’t let you block a reply for more than 60 seconds, so you can’t dynamically page in large files anymore.

That’s not accurate.

They currently offer no replacement for VFS. But I bet they will offer something fuse-like if they offer anything at all. Also: the file provider api they use for iCloud Drive that was supposed to ship in Catalina but got yanked is still likely to happen.

I have no inside information but assuming they continue to expose the VFS layer they will very likely build a usermode extension framework that is quite like FUSE, but supported by the OS and maintained by Apple.

I think this is fairly overblown, there are a fair number of FUSE for macOS forks out there with signing certificates.

I have kext signing certificate for ExpanDrive, Google has one for Google Filestream, I suspect many others have one as well. Rightfully, Apple doesn't hand them out as easily as they do with regular developer certificates, but if you want one and do a reasonable job representing that you're not going to panic end-user systems, you can get one too.

FUSE for macOS remains open source, fork it if you want. Benjamin merely decided not to work on it for free anymore and essentially providing bug fixes etc for those who pay for it.

Lastly - FUSE of macOS is not going to be around in the current form much longer. Apple has made it abundantly clear that Kernel Extensions are on the way out, and that macOS 10.15 will be the last release to fully support kexts without compromises. Check this slide from WWDC

https://imgur.com/a/EAzT6Ch

ExpanDrive [which I maintain], has a certificate to sign our fork of FUSE

Fairly sure this is not true, unless you have a source. Drew did all the client side work in python/librsync to get going.

We only email a couple times a year. And we definitely don’t silently update to a new major version. Ping support and link this thread and we can upgrade you free. Sorry for the trouble.

Projects like this seem like a great idea, but deliverability seems like a big concern that is hard to measure unless you have a reasonable amount of experience.

What are best practices for using/selecting an ESP if you were to use a project like this and want to ensure reasonable deliverability?

But it only connects to their environments, correct? With their special mouses

Having mouse support to enable a high quality remote desktop client could be a real game changer. I'd love to use an iPad, but can't develop on it. While I don't need it to be a primary development environment by any means, the ability to RDP into a machine and get work done seems great.

It's a network filesystem that can connect to a huge array of cloud storage services

Every time my wife goes to target for wrapping paper or something it costs $200.

Nearly any SFTP client would assume a rename would be a near-instant operation on the server, and would probably fall-over if it took an hour.

What if you open an SFTP handle, and then write 5 bytes halfway through a 20 GB file and close the handle? How do you translate that?

I'd be curious how this handles all the posix cases not well suited to object storage.

Renaming a folder than has a million files/folders inside is a single operation in SFTP, but 2 million operations on S3.

Does it handle writing at arbitrary offsets within a file? Does it download the file first then let you start writing?

What about just writing a few bytes at the beginning of a large existing file and then closing your SFTP handle?

How about 2 users accessing same file via SFTP at the same time?

What is the replication factor on these objects? 0.1% objects unavailable is a worrisome number as it suggests a meaningful number of objects are under-replicated and the cluster is poorly configured. I hope DO speaks to the durability of Spaces in a meaningful fashion after this is sorted out. Multiple days of having a large amount of data unavailable should shake the confidence of anyone considering Spaces for object storage. Are objects stored with 3 replicas? Are they on heavily striped volumes? What is going on that could cause this failure mode and what's stopping a much larger failure? Lastly, nowhere in their status updates are they saying "don't worry, no data-loss, just unavailable for a bit" - which I read as "we hope to not lose any data, but no promises"

Technically the lie really only extends to the "soon" part, since it is coming. And software regularly has a list of future features that often don't ship. Would love to chat if you were ever interested!

A bit rude. I like that phrasing.