Wednesday, 15 May 2019

jax devops 2019 Day 1



Engineering discipline

Engineering discipline

Craft -> Mass Production -> Lean production

Huge variation in how we work.

Order of magnitude change when optimising process -> bottom line

  • Iterative -> approach desired goal
  • Feedback
  • Incremental -> modular
  • Experimental
  • Empirical

Lean - reduce batch size

Scientific approach

-> Theory -> Prediction -> Experimental -> Observation ->

Design Develop Test Release

Idea -> in hands of user

Engineering amplifies ; Skill, Creativity, Apprentiship

Rigorous disciplined process to solve problems.



cd java-simple-stream-benchmark
mvn install
java -jar target/benchmarks.jar


Jussi Nummelin @JNummelin

Release often : Go faster ; Lean fast ; Adapt.

Root vs non-root : 86% of images in Docker Hub use root ; Userns remap not available in K8S (probably not any time soon).

capsh --print | grep -i current

=> not safe by default "simple CSR auto approver operator to help bootstrapping kubelet serving certificates easily"

RBAC - - configure who can do what.

ClusterRole -> ClusterRoleBinding

RBAC can be a sea of YAML for fine grained control ; easy to make mistakes

RBAC Manager ; kubectl-who-can => who can do what

  runAsNonRoot: true

kubectl get events | grep runAsNonRoot

Pod Security Policy ; Enforcer

apiVersion: policy/v1beta1
kind: PodSecurityPolicy

PSP - Pains ; Easy to break your cluster

Resource Quota ; Limits on CPU, memory, Storage, object counts

Namespaces, ensure : Resource Quotas, PSP setup right, Network Policies, LimitRanges

Secrets Vault by HashiCorp Managing Secrets in Kubernetes with Vault by HashiCorp

Neuroscience and devops


Helen Beal

CALMS - Culture ; Automation ; Lean ; Measurement ; Sharing

Squirrel's brains grow in the Autumn

Working memory - what can you memory. Frontal lobe can only cope with about 3 bits of information at a time.

Practice changes the way the brain works.

The Improvement Kata

=> Automacity.

We're asking people to change the way they're working.

Fear & Blame. Automonous cells. Not doing things in big project ways, change to small iterations.

Devops => Batch size = Micro, Autonomous, decentralised, "Like breathing", Actionable, High trust / fail early, metric = flow (value and time), DoD = Value outcome realised.

We are biological wired to resist change. Change can be felt as a threat.

Big Bang, Big J (Transformation) vs Incremental, Little Js (Evolution)

Kaizen Approach to devops

Encourage failure (i.e. experimentation & learning which implies failure along the way), however We want to avoid catastrophic failure in production.

Chaos Monkey

David Rock's SCARF model -

Mirror Neurons

Evolution : Discover what engages your colleagues ; Create psychological safety to improve learning ; Model the behaviour you want to see - you are the mirror.

The Rational for Continuous Delivery

Dave Farley - Continuous Delivery Ltd

AWS containers roadmap

AWS Fargate - Run containers without managing servers or clusters

Deploying Code Changes with Confidence using Consumer-driven contracts

PACT - contract testing tool

Serverless Operations

Soam Vasani -

Free-when-idle billing

Function as entry point, OS etc taken care of by platform.

Same principles as other devops, just different tools.

Inner loop - Optimised for fast feedback, usually focused on a single problem Outer loop - designed to be robust

Declaritive App Specifications - specify the end goal (not the details of how it should do it) => Uniform deployment, Easy upgrades, Validation before deploy, live-reload testing

  • aws-sam - AWS Serverless Application Model
  • framework
  • k8s declaritive by default - kubectl apply *.yaml
  • fission- Open source, Kubernetes-native Serverless Framework - fission spec apply

Monitoring : log aggregation ; metrics ; tracking ; alerts

Prometheus - From metrics to insight. Power your metrics and alerting with a leading open-source monitoring solution - Grafana

Tracing : opentracing , jaeger, zipkin

Incremental deployment

Automated Canaries - collect metrics, decide if AOK, if AOK send it more traffic.

Automated Canaries in Fission

Automated Canaries in Lambda Natively supported by AWS Lambda using function aliases.

fission canary-config create --name canary-1        \
       --funcN func-v2 --funcN-1 func-v1            \
       --httptrigger route-canary                   \
       --increment-step 10 --increment-interval 30s \
       --failure-threshold 10

Cost management

Lambda optimised for interactive workloads, spotty usage elasticicity of usage. Not great for batch processing / CPU bound processes.

Cold-starts - keep-warm hacks don't really work. How Slow Are Cold Starts?

You can set rate limits - "reserved concurrency".

Open source tool may be better for batch processing. Give you more controls. Deploy k8s on EC2 instance.

Deploying with Security in Mind

Steve Poole

Tools for the ignorant - there aren't any. You need to reduce your ignorance first.

We download stuff to get the job done.

e.g. AlwaysValidTrustManager, curl --insecure

How one developer just broke Node, Babel and thousands of projects in 11 lines of JavaScript

Malware actors - organised, methodical and highly skilled.

Vulnerabilities in application / container / server/ cluster / network. New vulnerabilities emerge.
Defense in depth and detection.


API discovery. Swagger publications etc

curl --head <>/api/vi

90% of time they are looking for initial chance to run arbitrary code.

Using HSTS?

Known vulnerabilities - POODLE / BREACH / CRIME / BEAST

mitmproxy - man in the middle https proxy ; reverse engineer ; malware / xss in browser.

Always assume there is malovent robot at other end of connection.

Privilege escalation ; find non-secure routes

Overlapping authority not concentric. Having an all powerful admin user is a risk.

Compartmentalisation is key ; no one set of keys gets you master access.

shodan - search engine for Internet-connected devices.

Fuzzing - firing in data to break / discover behaviour.


Less guessable paths, different port, special headers

Reduce opportunity for drive by discovery, use UUIDs not sequential numbers, simple IDs.

Review error messages - don't give unnecessary insight.

Log unexpected behaviour.

Build Vulnerability remediation into your pipeline


Data dog - monitoring