thinkersilver

It helps to instrument the journey of your work from jira all the way to build, deployment, run and monitoring (observation).

From that you can get measurements on how long each stage takes and the duration of each transition.

From there you can compare team A and B. The transition times is where the human time cost usually sits.

Just getting the time when a Jira or feature is raised, to the time it is picked up to the time of the first commit to the time of the first test and final build does already give you valuable insight.

The points you raised towards the end can be answered if observability of your CI/CD pipeline is actually in place or at least a place to start a line of inquiry.

Naturally the blockers will be aggregated into some of the values but as you work through the journey, they will start clustering at certain stages and maybe highlight a significant problem that needs to be addressed.

There's a wealth of data being left on the table that can help inform management decisions.

This feels all to familiar to a Java developer and similar to what they do in k8s with the genericapiserver component but not to the full extent described in the article. It's interesting to see how the service container pattern appears in other language ecosystems. The nice thing about go's module system is that it feels like a monorepo without the tooling feeling separate to the language. So splitting up a repo into different repos doesn't come with the versioning hell when working with maven

It's been a while since I've looked at Fabric8 but it had good java -> k8s integration and was great for writing k8s tools.

It appears though that Fabric8 is useful for solo java projects without complex dependencies on non-java projects or small java shop. It overlaps with where jenkins-x is going, which has made major strides in the last 24 months.

The original team that worked on Fabric8 lead by James Strachan all moved on from Redhat and many of them are working on Jenkins-x.

Kubernetes is becoming the lingua-franca of building infrastructure. Through CRDs and the kube api spec I can

- start an single application

- deploy a set of interconnected apps

- define network topologies

- define traffic flows

- define vertical and horizontal scaling of resources

And now I can define AWS resources.

This creates an interesting scenario where infrastructure can be defined by the k8s API resources and not necessarily have k8s build it. For example podman starting containers off a K8S deployment spec. It's an API first approach and its great for interoperability. The only downside is managing the yaml and keeping it consistent across the interdependencies.

I've been a big fan of brain storming tools for a long time so appreciate new tools with different takes on the problem.

I do have one observation and it maybe just me. I have no way of clustering a group of connected thoughts, or a way to 'chunk' the concepts. I see some attempt by using coloured lines but there should be a way to make these ideas hierchical and from that be able to zoom up and down various levels of conceptualisation. It would make managing a large network of ideas a lot easier, possibly a bit more ergonomic. Similar to the way the brain works.

It may just be me but was wondering if there's a way to achieve that with the current tooling or if you think there are workarounds?

Great article. I think you might be taking a subtle point in the article for granted. Namely, the ability to assemble a fairly complex distributed system from open source components.

Democratisation of complex machinery like the raft and consensus algorithms, schedulers, append logs, query optimisers and so on is a superb thing. It is something that has only been possible in the last 3/4 years and would have been devilishly difficult before then without significant upfront work.

Fetishising complexity might be a bit strong here. Seeing more posts like this is a good thing.

It would be nice if there was a readme for this repository. Like these AI and speaker recognition examples [1],[2].

[1] https://github.com/wq2012/awesome-diarization

[2] https://github.com/terryum/awesome-deep-learning-papers

If AWS gets you going definitely go for it but if you're looking for existing tools type 'python speaker diarization' in github.'

Most of what you'll find would require downsampling your audio to 16khz and you'll find a combination of NN based diarizer and hmm based models pre-ML.

One thing to note a lot of the systems will work well for interviews, broadcast media and footage taken from a camera because the audio will tend to be clean.

Film and movies will be a challenge because of the background music being identified as a separate voice. It tends to to confuse it.

Haven't used the cloud based systems with audio the background sounds you tend to find in film movies.

iPAAS' are essentially ESBs remarketed for the cloud for organatisations without the in-house expertise to coordinate building their own integration hub with a combination of tools like Kafka and API management gateways. Organizations lacking the * Architectural leadership

* Coordination capabilities

* Engineering resources

* strong comms between teams

tend to go for these solutions to paper over their existing problems. Particularly those around engineering capabilities.

These solutions work great for a while until the edge cases around data arrival and consumption usages resurface.

It's important to note that these solutions shine when it comes to the API discoverability across teams and light data format translations but as soon as that has happened get your data out quick!

Don't be tempted by their in-house solutions that try to solve caching, scaling and the ETL languages provided because they will tend to be half-baked and not as battle tested and will lock you in. The commitment for a large org using these systems is a 5-10 horizon.

By all means check them out and be ready for alot upfront design work to find the sweet spot and alot of engineering effort and frustration (alot).

Yes I was. I've posted the comment to the correct story now. I don't how that happened.

The poster is holding a line of bash to the standard of code and is illustrating that readability should be the goal and a way of bringing bash commands to a standard of readability for something like a PR. Readability is really there to show _intent_ I would say though that if you are bringing this to the code standards of today then this should really be wrapped up in some kind of unit test (https://github.com/sstephenson/bats )for it to pass the PR. That would make the code a bit more maintainable and can be integrated as a stage in your CI/CD pipeline.

If we do that then the intent would be clarified by the input and the expected output of the test. Then then the code would at least be maintainable and the readability problem becomes less of an issue when it comes to technical debt.

I've done this plenty of times with my teams and its certainly helped.

A software architect that can't code or refuses to has no business architecting the system. A software architect who doesn't show up to stand ups really shouldn't be dictating a solution to the team who have to build it. Ideally the software architect should be as hands on as possible with the team leads as the system is built up to the point that the architecture is proven or at least to see if the assumptions around the architecture actually hold up. Really strictly on teams leads to accurately communicate the architecture before it has been proven to hold up is risky.

The poster is holding a line of bash to the standard of code and is illustrating that readability should be the goal and a way of bringing bash commands to a standard of readability for something like a PR. Readability is really there to show _intent_

I would say though that if you are bringing this to the code standards of today then this should really be wrapped up in some kind of unit test (https://github.com/sstephenson/bats )for it to pass the PR. That would make the code a bit more maintainable and can be integrated as a stage in your CI/CD pipeline.

If we do that then the intent would be clarified by the input and the expected output of the test. Then then the code would at least be maintainable and the readability problem becomes less of an issue when it comes to technical debt.

I've done this plenty of times with my teams and its certainly helped.

The snippets feature looks quite powerful and particularly like the enumeration conversion example. Are there any plans of making these kinda of transformations extensible?

I've recently gone into speech assisted video editing and came across Kald about a month ago, was interested specifically in its diarization feature ... it's quite daunting. How did you get up to speed on it ... I've talked to a few people who've worked with it but it took them months to feel comfortable with it.

Well they did qualify with that it was a complex topic. And am not sure he hinted that it was compulsive.

I tend to work in a banking infrastructure environment and it's common practice to take notes,

The number of conversations I have during the day often has me saying, I'm not going to remember everything you've said, could you put that in a email so I can get back to you properly.

It does depend on the environment,

This may not be useful but have you tried to ask why you want to be heard. It's worth exploring your motives. Is it to make yourself feel more valued in the team or is it to provide value to the team. Figuring this out may allow you to find a way to change how your team members view you ( if it's needing validation from the team) or if it's the second, which I suspect it is, you may find a way to influence decisions _without_ having to talk over other people.

Team discussions are a competition of views and attention for space to express them and can be chaotic. Exploring

1. Verbal strategies - some have pointed out couple in the thread

2. Preparation of what you want to say - and framing the negatives strongly and the positives of your argument confidently

3. Seeding - Doing the groundwork and preparation before the discussion itself. Talking to your team members about your ideas and gauging they're reactions - in a way reading the room, and planting your idea so it's not out of the blue at the discussion

There's more but it's down to you. It's probably what you don't want to hear. You'll have to improve the way you communicate within discussions or learn how to influence the leaders and decision makes of the group outside the chaotic conversations.

I'm like you in a way but you've got to choose your battles.

Am not sure how much this discussion is happening now but do remember having these conversations more often a few years ago when tools like Prometheus were coming up.

This is my take and shouldn't be taken as gospel but I've observed that metrics are perfect for isolating and identifying an issue and logs attempt to explain the behaviour seen. Having a layered dashboard set up top to bottom showing the throughput, error rates and latencies expressing the health of each layer from the exposed service or api all the way down to the subsystems and hardware supporting them is a good place to start. I'd write more but don't want to make this post overly long. There are several useful articles online on the different methodologies and approaches to achieve this.

There isn't much conversation though about the knowledge an organisation has around incident analysis workflows, how past incident resolutions are captured and integrated with dashboards in monitoring systems and how they are shared and reused as SREs and engineers work with log analytics. I've seen the same lessons having to be relearned when new engineers join a team. Checklists are a good example( there are more ways)for directing incident analysis in large complex systems but how many times do you see this being supported by the monitoring system? The focus is always on displaying pretty charts in a grid when more value could be gleaned from the same dashboard content presented as a rich living document with charts, integrated checklists in something resembling a guide to resolving an issue.

There certainly is enough data to achieve this.

Bear in mind, I'm from the U.K so I don't have an American perspective on this.

This is a bit closer to the truth when you read or listen to what analysis they were doing.

I think what they were doing and it's association with Trump conflated how bad it was.

I'd seen in some places that mining of Facebook data in the pre-trump era, was quite prolific by both parties.

I did this about 4 years ago as well and haven't looked back either. The habits I'd formed around news reading just were not helpful or useful. If you think about it for it a second there is very little informational value given by news articles. The who, where, what, why can be done in a paragraph, the implications and future projects can be done in an opinion piece, because any forward future looking projection is 'opinion' and has error, bias embedded in it. All of this is cheekily now lumped into one. Well that's my view.

I've seen my colleagues go through the most anguishing emotional roller coaster over the the last 3 months with the brexit madness. I've seen emotions run high, I've seen the highs and lows. Everyone is stressed and I'm blissfully content.

I'm yet to experience a life altering impact from skipping the news.

What you're learning will fall into different categories, physical, mental or a combination of both. They will employ their on strategies to help them stick. I put language learning in the third category. Anki + srs for memorising vocab, When recalling vocab and phrases, mnemonics etc there's tonnes information online nowadays.

Something you won't hear often is this tip: Try and action/gesture what you're trying to say before you say it. I have no explanation why this works so well when learning but it's a technique actors use when embodying a character.

Then the obvious tip is to practice with native speakers,

If you have something purely physical, you have * Mental rehearsal of the action (anki) * Execution of the action in practice. For this to be helpful you need feedback, you could use a coach or record yourself * performance - like in a sport.

Had to rewrite my response because I went to heavy on the language learning tips

This is what I've used to date

There's so much to comment on in this article. First off gravity looks like an interesting product, it's shame a blog article on that hadn't made it onto the front page.

I hate to be nit-picky but I do feel that articles like this do more harm than good by oversimplifying a fairly advanced architectural pattern by downplaying the testing, resiliency and deployment approaches.

* easier to test - this is flat out wrong. Your application or system logic is spread out across multiple process boundaries - the only way to test it, is to deploy the dependent services on your dev machine and test the set or design your services so that all the application logic can run in a single process. Think spark and spark workers that move from a thread to a distributed model through configuration. Application logic can be tested with this approach, but not necessarily system behaviour (which can be simulated)

* rapid and flexible deployment models - in a large microservices fleet where code is managed by multiple teams and sits in several repositories the dependencies are not explicit (I can not do a "find usages" on a API call and see all microservices that are using it) - so deployments can be decoupled but there tends to be lots of breakages unless you have sophisticated well-thought out testing (see first point).

* resiliency - I'm not going to expand on that because 1st and 2nd point allude to a brittle system and hence reduced resiliency. There's also data and transnational boundaries across services which need to be addressed to. To be fair, there is a hint to solving this problem through something like kakfa but it isn't called out.

Microservices can be simpler with good tooling. Kubernetes is only a very small part of it. Maybe a follow up article may be good to clarify on what tooling can be provided for 1) better testing 2) reliable deployments 3) greater resiliency.

I liked the idea but couldn't quite get the anki workflow running smoothly. To be fair I tried this because of the anki integration but I don't think that's what it's meant for

This looks great, we're planning to test with the intention of moving our CI/CD orchestration onto this.

We use Calico internally, is there a plan in the future to allow other SDNs?

I remember writing this for my GCSE computer project in C. It didn't take me a weekend though back then. This is a Wolf3D style game. It would be interesting to illustrate the Doom engine that followed with the diagonal walls and lighting effect.

I looked at this after doing GCSe project, finding it to hard and skipping to a real 3D engine like quake because it was conceptually easier.

It was this paper and John Wilkes' talk at QCon Lon 2015 on Borg/Omega that helped me understand the architecture and motivation behind K8S, Mesos, Spark and so on.

It's a good paper and definitely worth the read.

Echoing what others I've already said, it's your supervisor and topic should be your main criterion.

Secondly, it depends on what you want to do afterwards whether is a career in academia or industry. If academia see the first point I made.

If it's industry then I'd stress on the networks the institutions afford you. OxBridge in the UK and the usual suspects in the US including Stanford. Remember in industry, the value of the PhD is more in the process rather than the achievement of the title,

Finally look at the time and monetary cost. It takes less time in the UK (3/4 yrs Vs 5/6) and you obtain the same utility from the title as you do from the US equivalent if and only if your optimising for industry placement.

Personally I think the right mix for industry is Undergrad in the US, PhD in the UK; and for academia UK for undergrad and US PhD ( all things being equal).

I hope this helps.

This is such a great idea!

I would use this as an annotation tool in meetings, particularly architectural diagrams and other complex information-rich image artefacts. I'm not sure if it has been mentioned yet but loading images would greatly add to its utility in meetings.

Have you given any thought to adding image loading?

Hi Ahmed,

I imagine you looked at other solutions before starting this. A distributed log is a fairly simple idea to understand (hard to implement) but what pain point is being solved?

Seeing that it is written in C/C++ - would it be that logdevice is optimised purely for speed and responsiveness?

The use cases overlap neatly with Kafka's. Everything from it's usage of zookeeper, time-and-storage-based retention tuning are similar

The announcement does not clarify the reason they use this over kafka. Is it because Kafka doesn't scale to millions of logs on a single cluster or is it because kafka is not sympathetic to heterogeneous disk arrays containing SSD and HDD. I strongly suspect it may be latency of writes at scale but this is pure speculation.

I don't know. If I understand why anyone might use this I'd contribute to building language bindings for the APIs.

I'm in the same boat mate. Kudos for getting this far this quickly