Microservices, Monitoring, Infrastructure as Code & Test Automation

How best should an organization transition its monolith architecture into a set of microservices?  Apparently, it might not have to. Matt Klein, principal software engineer at car-sharing service Lyft, told The New Stack at PagerDuty Summit 2017 that a startup can develop its own monolith more easily than it can develop complex microservices. But with an underlying service mesh architecture, such as Lyft’s Envoy, that monolith can still be providing service-oriented functions to customers in the same way, and probably without service degradation.

Envoy Proxy is a high performance, small footprint edge and service proxy, designed for modern cloud-native architectures. Built by the engineering team at Lyft.

Google, IBM, and Lyft released Istio an open source project that provides a uniform way to connect, secure, manage and monitor microservices.

The current release is targeted at the Kubernetes environment but support for other environments including virtual machines and cloud foundry is promised in the coming months.

Jennifer Riggins from TheNewStack talks through Chaos Engineering and how it can be used to bring stability to large distributed container style systems.

I really like this articular in particle the Chaos Monkey toolkit they wrote for Docker called Pumba

Pumba is a chaos testing and network emulation tool for Docker.

with Pumba you can

Stop running Docker containers.Kill the send termination signal. Remove containers.Stop a random container once every ten minutes.Kill a MySQL container every 15 minutes.Kill random containers every 5 minutes.Pause the queue for 15 seconds every 3 minutes.
========
In chaos engineering, as you try to achieve stability at scale, you experiment following these four steps:

Define that ideal state of the system’s normal behavior.
Create a control group and an experimental group.
Introduce real-world wrenches, like changing servers.
Try to find the difference or weakness between the control and what is crashing.
========

The future of microservices monitoring depends on what kind of solutions become standardized in the industry and what new features will we see in the future that will make your applications much better.

In this article Peter Marton takes a look at trends for 2018 and some interesting aspects of the article include

Vendor Neutral Agents
Distributed Tracing
Extracting metrics from OpenTracing API

Project Flogo™ lets developers build applications that run on edge devices and integrate them with IoT gateways. With the Project Flogo framework, you can extend the reach of core applications and infrastructure to edge devices to interconnect everything anywhere.

http://www.flogo.io/

Zappa makes it super easy to build and deploy server-less, event-driven Python applications (including, but not limited to, WSGI web apps) on AWS Lambda + API Gateway. Think of it as "serverless" web hosting for your Python apps. That means infinite scaling, zero downtime, zero maintenance - and at a fraction of the cost of your current deployments!

Nice article I came across this evening here by Joey Clover, Technical Co-founder @ TabbDrink

Tools covered for microservice management are
Containers (Docker)
Orchestration (Kubernetes)
Management (Forge)
Api Gateway / Canary (Ambassador)
Edge Proxy (Envoy)
Monitoring (Prometheus)
Local Testing (Telepresence)

I love this diagram

figo.io is the first Banking-as-a-Service in Europe. It enables access to every financial services provider through one API.

With the figo Banking API, we can query a bank account, the history of transactions, and the submission of payments.

Authentication is available via OAuth2.

Formats in JSON and REST allow to request and receive responses to retrieve bank accounts, modify bank accounts, and remove a PIN number from the server.

Below is a library of microservices written in GO that will allow you interact with the FIGO API.  Before you begin you will need to obtain a client ID / token ID from FIGO. There is a development version available.

1. Install pre-reqs
go get golang.org/x/oauth2
go get github.com/denizs/go-figo/fig


....more to post

Very nice and concise article with respect to secret management and what you need to consider when building microservices to meet with security requirements

Microservices and Secrets management - How to comply with security must-dos

The article highlights the following must-dos when considering a solution for secret management as part of your overall microservices deployment architecture

Secure storage of various type of secrets (API Token, Keys, Certificates, username & passwords)Reliable API based access to secretsDynamic secret distribution for automated encryption and authentication of keysFull Audit of access to secrets.Multi-level role based access to secretsCentralized revocation of secrets and redistribution

The OWASP Zed Attack Proxy (ZAP) is one of the world’s most popular free security tools and is actively maintained by hundreds of international volunteers*. It can help you automatically find security vulnerabilities in your web applications while you are developing and testing your applications. Its also a great tool for experienced pentesters to use for manual security testing

My first attempt at a basic terraform module written in GO to interface with VictorOps thorugh the VOPS API

resource_alerts.go
This is rough, just to connect the terraform module to generate an alert in VictorOps
Need to clean up the passing of parameters from the test.tf instead of hard coding in GO
But this is easy enough

package main

import (
"github.com/hashicorp/terraform/helper/schema"
  "bytes"
  "encoding/json"
  "net/http"
  "io/ioutil"
)

type VictoropsAlerts struct {
Code int `json:"code"`
Message string `json:"message"`
Data struct {
MessageType       string `json:"message_type"`
EntityID          string `json:"entity_id"`
EntityDisplayName string `json:"entity_display_name"`
StateMessage      string `json:"state_message"`
} `json:"data"`

}

package main

import (
"github.com/hashicorp/terraform/helper/schema"
  "bytes"
  "encoding/json"
 …

Terraform, as of v0.9, offers locking remote state management. To get it up and running in AWS create a terraform s3 backend, an s3 bucket and a dynamDB table.

For full details check out this blog post

Next - Terraform + CloudFormation = ELK Cluster

resource "aws_security_group" "rabbitmq" {
  name   = "rabbitmq"
  vpc_id = "${aws_vpc.vpc.id}"

  ingress {
    protocol    = "tcp"
    from_port   = 5439
    to_port     = 5439
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    protocol    = "icmp"
    from_port   = -1
    to_port     = -1
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    protocol    = -1
    from_port   = 0
    to_port     = 0
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "template_file" "rabbitmq_cloudformation" {
  template = "${file("rabbitmq.cloudformation")}"

  vars {
    rabbitmq_public_subnet_id  = "${element(split(",",

terraform_remote_state.shared.output.public_subnet_ids), 0)}"
    rabbitmq_security_group_id = "${aws_security_group.rabbitmq.id}"
  }
}

resource "aws_cloudformation_stack" "heavy_rabbitmq" {
  name          = &qu…