Show HN: Realm (YC S11), a mobile database

by bjchrist • 12 years ago

188 96 12 years ago

ovi256• 12 years ago

This is looking very good and it could lower the pain of developping mobile apps, especially for independent studios. Looking forward to learn more about the details of data syncing.

astigsen7• 12 years ago

Data syncing is an obvious need and Realm has actually been designed for that from day one. The entire underlying core is build to seamlessly record changes and coordinate them with a backend.

It is not part of the initial launch as we want to nail down the local api's with the community first, but look out for announcements soon.

jrullmann• 12 years ago

What kind of consistency do you expect to provide with this future syncing feature? I assume it will be eventually consistent. Is that right? How will conflicts be resolved?

orandolabs• 12 years ago

EnduroSync (https://orandolabs.com) does exactly the same thing, and it already has syncing. It's available for iOS and Android now, with more platform support coming.

bradhe• 12 years ago

Awesome, I'm really happy to see this kind of technology. This stuff needs to exist, and is really the future of the cloud IMHO.

meeps• 12 years ago

This is awesome. Really interested to see how this can affect wearable device development and data.

itsdrewmiller• 12 years ago

Repost:

https://news.ycombinator.com/item?id=8037029

pharkmillups• 12 years ago

Very cool. What's the thinking behind shipping with Swift support despite it still being in beta?

jpsim2• 12 years ago

JP from Realm here, I worked on our (experimental) Swift support. Swift as a language is pretty unstable right now, but it's clear that Apple sees it as the future of iOS and OSX development. So we wanted to start experimenting with how to make Realm feel right at home in Swift.

edanuff• 12 years ago

Looks good. Would really like to see Swift support. Also, the example showing integration with REST services in interesting, seems like you could do more with that.

jpsim2• 12 years ago

Realm currently supports Swift, though we're actively working to improve our Swift APIs[0]. You can find Swift sample code throughout our docs[1], and in our Swift examples on GitHub[2].

We learnt a lot about Swift introspection and interacting with Swift objects from the Objective-C runtime. We'll share our findings in some upcoming blog posts, so stay tuned!

[0] https://github.com/realm/realm-cocoa/pull/549 [1]: http://realm.io/docs [2]: https://github.com/realm/realm-cocoa/tree/master/examples/sw...

zznate• 12 years ago

The API approach vs. SQLite, etc. is interesting - particularly that it does not look to have very many moving parts. I'm curious about transactional semantics though. Are they 'always on' or is there API control (still haven't gone through docs completely yet, so I may answer my own question later).

astigsen7• 12 years ago

Alexander from Realm here, all interactions with the database is through full ACID transactions, but it is a bit untraditional in that the interaction model is build specifically to mobile apps.

The transactions interact with the runloop, so that you are always in an up-to-date read transaction (which always give you a consistent view and never blocks, even when other threads modify the same data). To write, you have to do explicit write transactions.

So your data is always live, consistent and up-to-date without you having to do any explicit coordination (apart from the write transaction on changes) and notifications allow you to always keep your UI up-to-date with the latest changes.

erichocean• 12 years ago

For people using SQLite, it's trivial to do the same with WAL mode and a read-only transaction on the main thread (which is what you should be doing). All writes happen off the main thread, so performance is great.

magsafe3• 12 years ago

What does this provide that CoreData doesn't? Whatever it is, it needs to be clearly explained on the landing page. Otherwise it's difficult to understand what problem this solves compared to CoreData.

SmileyKeith• 12 years ago

To me the more important part is the interface and thread safety.

timanglade13• 12 years ago

The big thing is really ease of use. Our blog post has a few more details on that point [0] but basically what we heard over and over and saw ourselves, is that Core Data quickly gets in your way if your app grows [1][2][3]. In essence, we tried with this initial release to offer a rich feature set with a simple API, less side-effects (especially for multi-threaded apps), and a performance that’s even better than using raw SQLite.

In early testing we got a lot of higher-level questions about what Realm was exactly, so we decided to go for a descriptive tagline instead, but you’re right that we should add more details about how we compare to Core Data!

  [0]http://realm.io/news/introducing-realm/
  [1]http://wbyoung.tumblr.com/post/27851725562/core-data-growing-pains
  [2]http://alastairs-place.net/blog/2013/04/17/why-core-data-is-a-bad-idea/
  [3]http://inessential.com/2014/06/11/core_data_thoughts_june_2014

(EDITS: formatting)

magsafe3• 12 years ago

Core Data has been improved and stabilized by Apple for over a decade, is used internally by many first party apps in iOS, is well documented and fully integrated into Xcode, and heavily "marketed" by Apple at WWDC. Any perceived performance issues can be eliminated or reduced by using faults, indexes and improving queries through NSPredicates. Even one of the blog posts you quote above [3] states that if he was starting a new app today, he'd go with Core Data. So I'm just not seeing a need for yet another ORM solution. I'm not saying you shouldn't continue building it, but you guys need to do a better job explaining the differentiation against Core Data. One area could be sync, where Core Data still needs work, and is limited to iCloud's backend. Perhaps that could be your differentiator? But you're not talking about sync yet. So there's work to do in crafting your message. Fwiw, I'm a senior iOS developer who evaluates these things for a living.

tillk2• 12 years ago

How do you backup this database? Does it integrate with iCloud or whatever is available on the device?

astigsen7• 12 years ago

The database is stored in a single file that is easily backed up. It does not have any explicit integration with iCloud, but you can easily use it to back up a copy of the db, or you can stream the copy back home yourself via ftp or http.

tillk2• 12 years ago

I'm somewhat of a newbie in regards to developing on mobile devices. Got any documentation how this is done from the phone?

mamcx• 12 years ago

A big plus for sqlite is that you can use it everywhere. For example, for a REST service instead of return a JSON I just return the sqlite database. Also, I can see the data in the desktop, use it on python, etc.

This could work elsewhere too? Be usable in python?

timanglade13• 12 years ago

You can do the same thing with Realm. We actually have a couple of key benefits over SQLite there, which are that our file size is much smaller (usually about 50%), and we don’t require deserialization to get objects out of the file (Realm data can be streamed directly into memory as native objects in your language).

On the minus side, We don’t work from everywhere yet, but we have internal bindings for Python, Ruby, C#, Java, PHP and a few other languages. We focused on the iOS launch so far, but we do plan to wrap up those additional implementations and share them on GitHub.

lylepstein• 12 years ago

Does this mean Realm would be usable for Xamarin-based apps in the near future?

melvinmt• 12 years ago

Looks interesting! The Swift examples on the homepage seem to be showing Obj-C code: http://cl.ly/image/2r250h3a0A1y

timanglade13• 12 years ago

Sorry about that. We’ve been seeing some issues with our CloudFront caches all morning, but if you give it a hard reload or two in your browser, you should eventually see our Swift samples — we’ll keep working to fix the issue.

jonahx2• 12 years ago

The foursquare map video shows how the experience with realm is better (and does that well), but it didn't give me any sense of how it's achieving those results. As a developer, I'd love to see an accompanying explanation with details on how the original version works, and how the realm version works.

timanglade13• 12 years ago

Absolutely. We didn’t get a chance to clean up the codebase for this app before the launch (it’s actually using an old version of our API that would be confusingly different than what our docs detail) but we’ll try to set up a blogpost about that since I may not be able to do the topic justice in a comment here :)

The short version is that most apps tend to do a query on click, fetch from an API and then cache the results in memory. The Realm version fetches from the API asynchronously in the back, pre-fetching results and writing them form the DB on one thread and reading & displaying them on the map from another. This has a few benefits (& drawbacks). On one hand, you do hit your servers a bit more often at first, and you may actually get so much data in the map that the UI component will start to lag. On the plus side, your data points are cached so you can do a lot less API queries over time (especially for something like Foursquare where users tend to stay in the same area and the dataset of venues doesn’t change that often). You also get better control as a designer or developer, and you can trim down & display relevant datasets on the fly.

To sum up, I’d say the key difference is the availability of a local, concurrent data structure that can easily be updated & read from multiple places (i.e. in a few lines of code if you look at the samples on http://realm.io)

jonahx2• 12 years ago

Thank you! So in theory one could do the same thing writing your own threads and using vanilla local storage? But realm is removing all the boilerplate custom code you'd have to write yourself, and making the local side more performant than custom code would be? Is that it in a nutshell?

mbesto• 12 years ago

I'm looking for exactly this. Thanks for building.

troydo42• 12 years ago

Well, we have a local database now. Now we need a terminal for the phone and a text editor.

jc4p2• 12 years ago

Cool idea. I had the same problem with the intro video that others have mentioned. I'm gonna read more when the Android version is out, since this might actually be able to fix a common pain point I have with my apps with data persistence (I'd like to be able to read recently touched questions in my Stack Exchange app even when I don't have internet, which requires a damn complex caching layer).

Just a side note: The SO link you have in the footer seems to not have any questions about your actual product in the tag, but about Java realms instead.

timanglade13• 12 years ago

Yup, offline mode is definitely a big area of focus for us. Regarding the tag, I’m not sure what the protocol is for these things, but since it wasn’t used for a clear product/project/topic, we figured it was as good a destination as any, since we’d prefer to let everybody benefit from answers we may give to support questions, via SO.

I’m drafting something to dive deeper into the video now; anything in particular you’d like us to dive into?

jc4p2• 12 years ago

I'd love deeper info about "we don’t require deserialization to get objects out of the file", but that might be somewhere in the documentation. I'm assuming you're faster than SQLite, so a demo showing those side by side might be a good selling point too.

bravura• 12 years ago

If you could make this automatically sync to a backend database, that would be amazing. It would be great to have the hard bits solved for us.

rjsamson• 12 years ago

Wow - this looks really great! As someone who has struggled with Core Data in the past, the API looks great and things seem, on the whole, a lot cleaner. Looking forward to giving it a go.

EDIT: I just came across the REST API / JSON import section in the documentation - really cool stuff!!

chrishough• 12 years ago

looks cool, can't wait to try this out!

rraway• 12 years ago

I could do with some clarification on what problem the product solves, and actually, what the product even is. To me this looks like a new DB, and an ORM style interface to interact with it?

I like that it is multi-platform, but SQLLite is a pretty mature DB store available on all the major platforms anyway. I get that it is faster... but presumably there are there trade-offs for this speed gain. I would like a better understanding of what they are. Transaction support?

(PS: I would also much prefer if your objects can be typed in a way so they are plain, rather than forced inheritance from RLMObject)

EDIT: The blog post at http://realm.io/news/introducing-realm/ answers some of these questions.

timanglade13• 12 years ago

Absolutely. There’s a lot more details in our launch blogpost [0], but you got the right impression: it’s a custom C++ storage engine underneath (not SQLite), that is tightly integrated with the object layer. Not an ORM per se, because the engine isn’t relational, and because there’s not copying/mapping/translation going on — the data on disk is the same data you manipulate in your language.

The big problems we try to solve are really ease of use & performance. We heard from many developers dissatisfied with the current options on mobile, and the relative complexity to accomplish tasks like accessing data across threads, JSON handling, performance on graph queries, etc. — which the reactions on Twitter seem to confirm. We’ll write a lot more about the tradeoffs in coming weeks, but we’ve really grabbed a lot of perf cycles just by taking advantage of modern techniques such as zero-copy architectures, bitpacking and vectorization. Of course the big implicit tradeoff is that you give up a bit of the relational paradigm in exchange for that speed, although we do maintain ACID transactions, immediate disk persistence by default and schemas.

Completely hear you on wanting plain objects — I personally feel that way too! Unfortunately, there’s no way to achieve what we do without making you inherit from RLMObject, although we do try to make those behave as close as possible to NSObject.

[0] http://realm.io/news/introducing-realm/

webstersx2• 12 years ago

Although I don't know exactly what the implications are just yet, I've created a category on one of my RLMObject subclasses which accepts an NSArray or NSDictionary property and in the getter/setter converts to and from the NSData backing property. The only thing is after [Venue creatInDefaultRealmWithObject:] I have to explicity set the value for it to propogate to the backing NSData property.

It's extra work, but it lets me manipulate Realm into doing what I want it to do to store data which doesn't conform to an explicit schema. (e.g. a Venue which has an array of address strings, where I don't want the headache of iterating through venue.address[i].string values:

(note: this doesn't look pretty, its doing some funky formatting!)

"venue" : { "name":"Test Venue", "address" : [ "address line 1", "address line 2", "address line 3 for rare instances" ] }

@interface Venue : RLMObject @property NSString name; @property NSData addressData; @end

@interface Venue (AddressToData) @property NSArray address; @end

+ (NSDictionary )defaultPropertyValues { return @{ @"addressData":[NSJSONSerialization dataWithJSONObject:@[] options:0 error:nil] }; }

- (NSArray) address { NSArray address = nil;

    if (self.addressData) {
        address = [NSJSONSerialization JSONObjectWithData:self.addressData options:0 error:nil];
    } else {
        address = @[];
    }
        
    return address;

}

- (void) setAddress:(NSArray )address { if (!address) { address = @[]; }

    NSData *data = [NSJSONSerialization dataWithJSONObject:address options:0 error:nil];
    
    self.addressData = data;
}

Reply

wuliwong2• 12 years ago

I would love to share the swift-specific portion of the getting started docs on www.sososwift.com but http://realm.io/docs/ios/0.80.0/#swift doesn't navigate directly to that. If there's a link to Swift instructions for realm, I'll post them and you will find them here http://www.sososwift.com/?a=realm.io Good luck, looks cool!

Reply

timanglade13• 12 years ago

CloudFront is giving us a lot of issues at the moment, but I implemented + deployed that feature for you! Hopefully it will be live very soon. (btw, thanks for creating & running SoSoSwift!)

Reply

wuliwong2• 12 years ago

Hey great, it worked! Sorry for the delay, I had to work at my actual job. :/ BTW, we just deployed a new app last week, had a bunch of problems with CloudFront. Good luck!

Reply

lnanek2• 12 years ago

I was going to say Android already has SQLite built-in, but they do a good job explaining advantages vs that (more object like syntax, more document like vs. table and schema like, better performance, etc.). Good job.

Reply

ernestocp• 12 years ago

Has any succeeded building this in rubymotion?

Reply

timanglade13• 12 years ago

I don’t think anyone at Realm has ever tried, but I doubt it would work at this point. We do a lot of things specific to each language in the implementation, so we’d have to support RubyMotion explicitly for it work.

Reply

Gertig• 12 years ago

RubyMotion is Objective-C under some thin covers so it seems like it would be relatively straightforward to have it work with Realm without a lot of extra trouble.

Reply

fluffyjack• 12 years ago

I'll be following up with this a lot because I would like to cover it on MotionInMotion

Reply

alooPotato3• 12 years ago

LOVE this. I've dealt with the pain of multiple CoreData contexts in different threads and ugggh. This was easily the longest part of our Streak app development, as we needed it to be fast, offline, and fully synced. This is definitely a way better API that accomplishes the same thing as CoreData. If its actually faster too, wow, just amazing.

One question - CoreData works really well with built in UIViewControllers like TableViewControllers. Any chance you'll be releasing some code that makes wiring up changes in the models to the controls that are displaying that data?

Reply

alooPotato3• 12 years ago

Sorry one more question - Android?

Reply

timanglade13• 12 years ago

Android is coming! We have an internal build that’s almost at parity with iOS but still needs a bit of tuning and is missing a few key features like Migrations. We’ll work to release & open-source it soon. You can register to be notified of that [0].

[0] http://realm.us5.list-manage1.com/subscribe?u=2aab5198c2f56b...

Reply

brmunk6• 12 years ago

Android is in the works! You can signup to be notified of progress : http://realm.us5.list-manage1.com/subscribe?u=2aab5198c2f56b...!

Reply

timanglade13• 12 years ago

Thanks for the kind words! Regarding wiring this to a TableViewController, we have some code samples already for Swift[0] and Objective-C[1]. Right now you get notified of changes and refresh, but we’re very interested in exposing a reactive-style binding mechanism as well. The storage engine underneath supports that very well actually, so stay tuned…

  [0] https://github.com/realm/realm-cocoa/tree/master/examples/swift/RealmSwiftTableViewExample
  [1] https://github.com/realm/realm-cocoa/tree/master/examples/objc/RealmTableViewExample

Reply

Maarten88• 12 years ago

What about support for Windows Store and Windows Phone? Those platforms need something like this too.

Reply

barumrho• 12 years ago

The main benefit of using CoreData is seamless faulting behavior. It's not obvious from the example or the site that Realm supports something comparable. How does Realm deal with a lot of objects?

Reply

grayprog• 12 years ago

Is there a Mac OS X support planned? Or perhaps the iOS version works as is?

Reply

timanglade13• 12 years ago

The framework in our release zip only works for iOS but if you build from source[0] it will actually generate an OSX framework as well!

[0] https://github.com/realm/realm-cocoa#building-realm

Reply

brmunk6• 12 years ago

The RealmBrowser (which can be found under /tools/ at the above source link) is already using Realm internally to show/edit any Realm file dynamically.

Reply

tvitesse• 12 years ago

Salmon pink? Their swag is going to rock.

Reply

neobaba12892• 12 years ago

I have to give it a try, but how well does it work with different BaaS providers like Parse, Kinvey, Appacitive etc?

Reply

brmunk6• 12 years ago

Right now there is no specific integration into any BaaS.

A Realm database is just a (compact) file that you can send as you see fit. As most BaaS solutions has their own local storage and API it may not always make sense to integrate, except that you could easily store a small Realm file as a blob in any BaaS and transfer it remotely.

It will also really soon be possible to export some or all objects as json which you then again can send to remote services. That does however take up much more space/bandwidth compared to Realms compact data format, but could be needed if you want to persist remotely in another database format.

Reply

neobaba12892• 12 years ago

Not all of them generally have their own local storage. But as you do with the foursquare APIs, it can definitely be done with other API providers. But thanks.

Reply

darkslave• 12 years ago

Well I think everyone agrees it looks like something pretty interesting but I couldn't find anywhere what's the easiest way of migrating from CoreData w/sqlite to Realm? Not sure if the tradeoff of rebuilding everything would be worth it for the increase in performance.

Reply

brmunk6• 12 years ago

We will surely add some guides to address that need (and many others). In the coming time we would love to expand on the documentation and clarification so you can learn Realm as easily as possible. Feel free to ping us with anything needing specific attention in the support forum: https://groups.google.com/forum/#!forum/realm-users

As for the tradeoff, I guess that heavily depends on the specifics of your app. You are the only one who can judge that depending on your current app pain points, maturity and future needs.

Reply

hamidpalo• 12 years ago

I can't tell -- does it support upserts?

Reply

brmunk6• 12 years ago

Currently not as a specific one liner. It's super simple to do it yourself, but we will likely add that convenience method shortly.

Reply

edalvsan• 12 years ago

Looks great. Really looking forward to the Android version too!

Reply

golergka• 12 years ago

There's a quote from Unity CEO on the frontpage — should we expect Realm appear in Unity-land as well?

Reply

nbody• 12 years ago

Doubt it. It's pretty common to ask "testimonials" from people with authority for publicity etc.

Reply

ergofreak• 12 years ago

That Four Square demo is quite amazing, I'm looking forward to give a try in near future.

Reply

mcgraw• 12 years ago

The speed, simplicity, and lack of headaches (threads, contexts, etc) really make this an interesting solution. Hope these guys/gals crush this!

Reply

Groxx4• 12 years ago

Sounds interesting. I'll definitely watch for the Android version :)

That said, the "insert" benchmark source for SQLite is a bit disingenuous. It makes a new compiled statement every time it inserts - if you're doing 150,000 inserts, you should definitely be reusing that thing. And on Android, greenDAO (probably others) does this automatically for every insert/update. It's not some edge-casey optimization trick, it's normal SQLite use.

Reply

astigsen7• 12 years ago

Thanks for the feedback, benchmarks are notoriously difficult to get accurate (and fair), which is exactly why we published the benchmark code. We will incorporate your feedback to improve them.

Reply

Groxx4• 12 years ago

Great! I don't know enough iOS-SQLite to confidently tweak it myself, and my small experiment (in an emulator, so ginormous caveats apply) gave mostly-unbelievable results so I somewhat doubt I did it right.

I'll definitely watch with interest! I'd love a safe, faster, easier data store than SQLite. I'm certain such a thing is possible, and it would address a massive range of uses. Good luck though - SQLite is pretty fantastic, it's a tough competitor to have :)

Reply

Groxx4• 12 years ago

Should I submit an Issue on github to help push this, or is it relatively certain that it'll get done? If I get time + a device provisioning profile, I could also try to do it myself, but that's not likely to be any time soon :)

Reply

seepel2• 12 years ago

I'm a little late to the party, but I ran the benchmark inserts with cached statements on both my Macbook Pro and my iPhone 5S. sqlite definitely fairs better under this scenario.

That being said I still find the speed of Realm pretty impressive. And it's hard to gauge real world performance from this test. In my experience the real bottle necks around insert are upsets and establishing maintaining relationships.

For reference here are the benchmarks without reusing sqlite statements.

  Macbook Pro - Realm: 0.780414 sqlite: 0.464804 FMDB: 0.654659 
  iPhone 5S   - Realm: 2.235381 sqlite: 2.786381 FMDB: 3.336679

And with statement reuse
Macbook Pro - Realm: 0.766406 sqlite: 0.189747 FMDB: 0.683823 iPhone 5S - Realm: 2.214847 sqlite: 0.991864 FMDB: 2.832800
Interesting side note: FMDB seems to be spending just as much time finding the cached statement as sqlite takes to prepare a new one. I didn't expect that. I haven't dug into it at all, but I wouldn't be surprised if this due to the fact that the insert statement is relatively simple.
EDIT: Just fixed up formatting for the results

Reply

Verlet• 12 years ago

In my experience object persistence rarely needs SQL or other "relational additional stuff" underneath. Typically I need to store objects and their mutual relations persisted as if they were just kept in memory. Have often thought about how weird it was that you could not just persist objects "as they are" without additional mappings etc. I saw a demo of Realm and had a talk with one the developers a couple of weeks ago. As the demo and our talk progressed I found myself nodding more and more excited. As I understand Realm it seems to be based on a very natural and straightforward principle. It can't NOT be efficient... I can't wait to try this out in a real project!

Reply

alexdaukrainian• 12 years ago

Realm peeps, can you explain this: "The only restriction is that objects can be used only on the thread on which they were created"? Is this same as Core Data limitation or is there more to it?

Reply

Show HN: Realm (YC S11), a mobile database

Related Stories