Stream processing, Event sourcing, Reactive, CEP… and making sense of it all

http://blog.confluent.io/2015/01/29/making-sense-of-stream-processing/

by martinkl • 11 years ago

31 6 11 years ago

blog.confluent.io

turtle4• 11 years ago

Thanks for posting. There's one piece of this that hasn't quite clicked yet for me, which is UI refresh coupled with the event stream approach. In the article's example, for instance, you have the user events going into the stream, which refreshes the cache, db, data warehouse, etc.

In a normal UI, the user makes some action, which would update the db, clear the cache for the affected key(s), and display the updated page to the user. In the event stream processing way, if the front end events are what is being logged and the cache refresh for example is running via Samza or some other processor, how does the UI get refreshed appropriately since the processing of the stream is decoupled? Or does the main app cache remain coupled to the form submission and there are just 'other' uncoupled caches, etc?

martinkl• 11 years ago

This talk was mostly about backend systems, but it hints at Functional Reactive Programming at the end. That's a field in which people are working out how to update the UI based on event streams. Some cool projects to look at in this area are Elm, React and Meteor, for example.

Once you've got a UI that can be dynamically updated based on event streams, you can hook the backend streams into the UI using a WebSocket or similar.

ghuntley• 11 years ago

Imagine a simple iPhone application which has the following components:

        // a text box in which the user types what they are searching for.
	// a submit button on which the user can optionally press to initiate the search.
	// a List<T> of search results
	// a service that takes the search query and executes the search.

This is how you can implement the above in a FRP manner:

        public class SearchViewModel : ISearchViewModel
	{
	    public ReactiveList<SearchResults> SearchResults { get; set; }

	    private string searchQuery;
	    public string SearchQuery {
	        get { return searchQuery; }
	        set { this.RaiseAndSetIfChanged(ref searchQuery, value); }
	 
	    }
	 
	    public ReactiveCommand<List<SearchResults>> Search { get; set; }
	 
	    public ISearchService SearchService { get; set; }
	}


	public SearchViewModel(ISearchService searchService = null) : ReactiveObject, IRoutableViewHost
	{
	    SearchService = searchService ?? Locator.Current.GetService<ISearchService>();
	 
	    // Here we're describing here, in a *declarative way*, the conditions in
	    // which the Search command is enabled.  Now our Command IsEnabled is
	    // perfectly efficient, because we're only updating the UI in the scenario
	    // when it should change.
	    var canSearch = this.WhenAny(x => x.SearchQuery, x => !String.IsNullOrWhiteSpace(x.Value));
	 
	    // ReactiveCommand has built-in support for background operations and
	    // guarantees that this block will only run exactly once at a time, and
	    // that the CanExecute will auto-disable and that property IsExecuting will
	    // be set according whilst it is running.
	    Search = ReactiveCommand.CreateAsyncTask(canSearch, async _ => {
	        return await searchService.Search(this.SearchQuery);
	    });
	 
	    // ReactiveCommands are themselves IObservables, whose value are the results
	    // from the async method, guaranteed to arrive on the UI thread. We're going
	    // to take the list of search results that the background operation loaded, 
	    // and them into our SearchResults.
	    Search.Subscribe(results => {
	        SearchResults.Clear();
	        SearchResults.AddRange(results);
	    });
	 
	    // ThrownExceptions is any exception thrown from the CreateAsyncTask piped
	    // to this Observable. Subscribing to this allows you to handle errors on
	    // the UI thread. 
	    Search.ThrownExceptions
	        .Subscribe(ex => {
	            UserError.Throw("Potential Network Connectivity Error", ex);
	        });
	 
	    // Whenever the Search query changes, we're going to wait for one second
	    // of "dead airtime", then automatically invoke the subscribe command.
	    this.WhenAnyValue(x => x.SearchQuery)
	        .Throttle(TimeSpan.FromSeconds(1), RxApp.MainThreadScheduler)
	        .InvokeCommand(this, x => x.Search);
	}

If the above sounds compelling then check out http://reactiveui.net/ which uses the Reactive Extensions for .NET to create elegant, testable User Interfaces that run on any mobile or desktop platform. Supports Xamarin.iOS, Xamarin.Android, Xamarin.Mac, WPF, Windows Forms, Windows Phone 8, Windows Store apps.

If you want to learn more about FRP then I suggest the academic paper "Out of the Tar Pit" by Ben Moseley & Peter Marks @ https://raw.githubusercontent.com/papers-we-love/papers-we-l... then make your way back to the origin of FRP - http://elm-lang.org/learn/What-is-FRP.elm

As for a good example of a backend designed around Event Streams/Reactive Extensions then check out https://github.com/AdaptiveConsulting/ReactiveTrader

rationalthug• 11 years ago

I think this is the best article I've read about stream processing yet. Love how all of the related concepts are tied together and how concepts in one form of stream processing naturally fall out of the structure preferred in another form of stream processing.

Great explanations and insights.

eoincathal• 11 years ago

Seconded. I was glued to the article from start to finish.

nell• 11 years ago

If you've heard of the terms Event Sourcing, Storm, Kafka but don't really know what it means this write up explains the concept really well. Take the time to read it.

Stream processing, Event sourcing, Reactive, CEP… and making sense of it all

Related Stories