bhaisaab

ShapeBlue | Remote (Europe/Asia/Flexible timezones) | Dev and QA engineers | Full time | https://shapeblue.com

Hi all, ShapeBlue is a remote-only 100% employee-owned international business ( more on this on https://www.shapeblue.com/shapeblue-has-become-an-employee-o... ).

We are hiring devs and QA engineers to work on opensource Apache Cloudstack ( see https://cloudstack.apache.org https://github.com/apache/cloudstack ).

Read more on https://www.shapeblue.com/careers/

Tech stack is (mainly): Java, Python, Vue.js, (and Go for sub-projects). Expertise in IaaS, hypervisors (such as KVM, Vmware, XenServer/XCP-ng), storage and networking is a plus.

To apply, you can email me on rohit.yadav AT shapeblue.com

Never heard of the person

http://cloudstack.apache.org/users.html

Has anybody seen/used the new HP spectre x360 gem-cut? Any feedback on spectre x360 vs XPS13 for developers?

Another reason to be away from MS cloud. Good work man.

Thanks a lot, as a backend distributed systems person I have hard time understanding UI in general. I'm half way through your videos now, and for the first time I could grok what React is all about.

100% REMOTE

We work on opensource Apache CloudStack, with our team spread across Europe, Asia, Africa and America. We're hiring for positions of software engineer, test engineer and consultant engineer.

Our work involves deep knowledge of hypervisors, storage, and/or networking. We are a polyglot environment – developing Apache CloudStack mostly in Java and Python. Our team values collaboration, continuous improvement, and the Apache Way.

For more details see http://www.shapeblue.com/careers or email jobs@shapeblue.com

'the only serious opensource contender around', have you heard of Apache CloudStack? It has seen some serious usage: http://cloudstack.apache.org/users.html

Not sure if people know about Apache CloudStack or not, it has all those IaaS feature and it just works with various basic to advance networking models.

Checked four major ISPs, Github is accessible from India. Can anyone confirm which ISP is (still) blocking it?

It is subjective, we've papers on this area but the industry is yet to find something concrete. Most of us measure code quality by different apparatus, such as: - code coverage - average number of bugs per kloc - no. of bugs/faults reports per day/hour/per kloc etc.

Actually, the newer VWO is much simpler and intuitive: https://vwo.com/early-access

Note: I'm one of the engineers who works on VWO

collyw, if someone is new to Linux I would also suggest them to use Ubuntu or Mint. But, as you start using it at some point of time you may want to tweak your system like you wanted(given that you've bandwidth and motivation to do so). It's about that time, most novice to advance sysadmins/users would want to use something like Fedora or Arch.

Fedora Linux for Desktop - Great documentation, community and support. You get the greatest, latest and the most robust distro in terms of drivers and stability (I've tried Ubuntu, Debian, Arch, Manjaro) that requires very less sysadmin work (unlike Arch etc.).

Debian Linux for server (stable, tested).

Rant: Few years ago the state of yum based distros was very bad and at that time Ubuntu came and was instantly favoured. Right now rpm based distros are in much better shape and deserves a shot. About pkg-management -- I find apt clumsy for example you need to apt-cache search/list, but to install do apt-get install; I like yum (search/install etc. just one tool) and rpm a lot.

While it may depend on a company's requirements; speaking for the startup I work for -- we use bare-metal servers that give us more bang for the buck and they are cheaper than aws and cloud solutions in general that we've compared (note: we were on Linode in the beginning but are now on SoftLayer)

Great post by an awesome humble programmer :)

(NB: he's a friend who also happens to be a colleague at work)

For my company's server infra automation I evaluated Chef, Puppet, Salt, Ansible, Fabric. The deciding factors for me were a stable software with a good ecosystem. I found that Puppet is much better than the rest when it comes to having an active community, but I like the scriptability, hackability and simplicity of Fabric. Chef was nice too but found it cumbersome to setup and write recipes compared to Puppet's module.

I wanted a masterless setup where there would be no central master so I use Puppet as a git repo with Fabric, where Fabric takes care of parallel ssh keys management with Gitlab (git hosting software, used internally) and deployments. I've added custom python modules to manage continuous deployment of code as well, I really like the basic command and programmable interfaces it offers, roles and just programming in Python. The whole setup thus uses a masterless Puppet (git) + Fabric, works on an exclusive private network across datacenters.

Who cares? I think it's a matter of acceptance, people have come up with different meaning of words which get adopted around the world before the formally land into a widely accepted dictionary etc. Hacker before the culture started, meant a person who hacks (chops wood?), but now has different meaning.

We could create some 'xyz' word and if everyone starts using it, it becomes acceptable; that does not mean you flame early adopters, there are many words people would just throw around (such as fck :P), I guess they use they creative/poetic license and get away :P

Sure, I agree and I understand things go wrong, so we've monitoring tools (munin, pingdom, pagerduty etc.) and we check them often or they post notifications often. There are at least two folks on-call 24x7. Based on present workloads we've calculations on how much time it would take to exhaust resources and we plan our servers accordingly, this buys us time to react.

Yes. Among several queuing systems we played with RabbitMQ just worked out of the box with features we wanted such as reliability, confirms and the queueing patterns (routing, topologies, fan-out etc.), it was easy to use and deploy. 0MQ gave us no message persistence or broker implementation (having a broker decouples our producers and consumers), if we were to use 0MQ many features we wanted would have to written which RabbitMQ provided out of the box. I think 0MQ is more like a framework than a queueing system/platform like RabbitMQ which just works.

So you make sure that you've multiple RabbitMQ servers or a RabbitMQ cluster so there is no single point of failure :)

We love opensource, AMQP was created by wall-street giants and RabbitMQ fits our case and Tibco RV does not solve our particular case in our particular environment.

Yes, you're correct there are corner cases where every component could fail. For example, a DDOS on our servers could slow us down or a kill -9 of RabbitMQ (even with persistence), OpenResty or agentredrabbit and the consumers could result in loss of messages or unexpected results. We've done such tests and seen loss of messages. Our servers have enough disk space and RAM to hold some 100s of billions messages and the multiple servers make sure we don't run out of resources. Queue settings are very important, for our case we've persistent messages with durable queues with few other tweaks and settings. Lastly, yes we explored our options and at that time used the best of what we could get and hack the best we could come up with in limited time and resources. If you're just starting with RabbitMQ I would recommend the tutorial on RabbitMQ's website and the book "RabbitMQ in Action" by old_sound et al

Great, thanks for sharing.

Thanks for your comment. I'm the backend engineer at Wingify who wrote this post. This post talks mostly about data acquisition, I'll suggest my team to post more technical details of our dynamic CDN which serves the dynamic content.

Hi, I'm the backend engineer at Wingify who wrote this post.

whisk3rs: For our use case, message had to be reliable, we required publisher confirms and you're sort of right, RabbitMQ in our use case did not satisfy our latency requirements. I've tried shovel and it does not solve our problem like we wanted. I don't have component based benchmarks between Redis and RabbitMQ but I've already shared the loader.io results of our two pipelines. Some details below.

noelwelsh: We're using Redis as an intermediate storage sink for RabbitMQ and instead of just passing messages from Redis to RabbitMQ, we move them in chunks. I'll explain below why we could not move away from Redis. "mbell" is correct about why do it that way.

NOTE: this is going to be a long reply;

This blog post talks mostly about data acquisition, let me start by giving some background on our backend services. You may read more about VWO on visualwebsiteoptimizer.com, so I'm skipping that. We've multiple servers across globe which helps us do data acquisition (capturing data for analytics) and servers which serve javascript snippets which are applied on one's website. Our users install VWO code on their website and depending on the test etc. the code from our servers is served dynamically and applied (like "karolisd" commented, we do it as fast as possible and we're still tuning our systems, and it is dynamic).

We don't use any CDN (such as akamai, cloudfront etc.) for our dynamic content as they fail at providing us tweaking mechanisms while serving dynamic content on same url and such a design would break user experience and we don't want our users to keep changing the installed code on their websites -- for them it should just work. Many such services require you to install some code that would brings some js from a url, each time you modify something they may either ask you to install new code with the new url or if they're using CDN they might send you all the changes by the same url (for ex. increase of payload size due to unnecessary data).

So, we have two most important requirements; one -- to do data acquisition reliably, as fast as possible with minimum payload; two -- to serve content dynamically with minimum payload and as fast as possible because it is most important for us to not slow down website of our users.

To do that we use a custom-compiled OpenResty (nginx mod) with luajit and our (Lua) code runs inside OpenResty offering us minimum latencies and processing speeds (no reverse proxies). At the time we started solving our scaling problem, there was no lua-resty library that does publishing from Lua/OpenResty to RabbitMQ. Writing a production grade lua-resty AMQP library would require a lot of time (you may search our discussions on openresty-en mailing list). So I started by writing an opensource stomp based library to use RabbitMQ's STOMP adapter and STOMP was much light weight and easy to implement compared to AMQP (there are multiple versions as well). So, in our Lua/OpenResty code we had two options either to publish messages directly to RabbitMQ or to Redis. Publishing to RabbitMQ was slow due to latencies, the AMQP overhead and the small payload size (less than 1kB), it was a deal breaker for us. But running Redis locally on unix sockets to which our Lua/OpenResty code would write was much better in terms of latencies, and transferring data in chunks from Redis to RabbitMQ improved throughputs.

Coming back to the earlier comment on this thread, Redis does not add any failure case, instead it provides us a failover: So far I've seen that there may be more cases or chances of the whole datacenter (network) going down instead of a server, in that case data sits in Redis. Our cron jobs along with monitoring tools make sure local services on each servers are up and running. In case our server goes down, our Anycast DNS (with low TTL) would switch traffic to other available servers automatically. In case RabbitMQ servers/datacenter goes down, data would be pushed into it next time the network/server goes down; using reliable messaging ensures messages/chunks are written to disk (fsync) so they persist if not consumed, publisher confirms give us reliability. In case the consumers die, data sits in RabbitMQ. In case of timeouts and latencies of moving data to RabbitMQ, agentredrabbit handles that.

Comments, questions welcome.

Our talented intern tried to explain the technology and what it took for him to create that page. The post was never intended to be a tutorial, but feel free to comment - what you want to know :)

Heya, this was done by one of our talented interns Kushagra A. (https://twitter.com/kushsolitary). We did not use any parallax page making tools, it was written using HTML5, CSS3 and some js library.

No formal benchmarking I know of but OpenStack has better traction, is written in Python (I love Python) but has no real world production deployments I know of which is most likely to change. We don't have to have only one winner, but winners, so all stacks can have their share of market and everyone can win as everyone bring their pluses and minuses.

w00t CloudStack!

Some quick facts:

- CloudStack is the most active Apache project now by no. of commits/day and code/development activity: https://www.ohloh.net/orgs/apache

- Stable, mature code, used in production. Works for Xen, KVM and VMWare.

- Real world deployments, largest known deployment consists of some 20k hosts (source Collaboration12, I don't know exactly which talk/video, someone can comment with a link)

(Note, I'm a CloudStack developer and committer and I'm loving it :)