Craft -> Mass Production -> Lean production
Huge variation in how we work.
Order of magnitude change when optimising process -> bottom line
- Iterative -> approach desired goal
- Incremental -> modular
Lean - reduce batch size
-> Theory -> Prediction -> Experimental -> Observation ->
Design Develop Test Release
Idea -> in hands of user
Engineering amplifies ; Skill, Creativity, Apprentiship
Rigorous disciplined process to solve problems.
cd java-simple-stream-benchmark mvn install java -jar target/benchmarks.jar
Jussi Nummelin @JNummelin
Release often : Go faster ; Lean fast ; Adapt.
Root vs non-root : 86% of images in Docker Hub use root ; Userns remap not available in K8S (probably not any time soon).
capsh --print | grep -i current
=> not safe by default
https://github.com/kontena/kubelet-rubber-stamp "simple CSR auto approver operator to help bootstrapping kubelet serving certificates easily"
RBAC - https://kubernetes.io/docs/reference/access-authn-authz/rbac/ - configure who can do what.
ClusterRole -> ClusterRoleBinding
RBAC can be a sea of YAML for fine grained control ; easy to make mistakes
RBAC Manager ; kubectl-who-can => who can do what
securityContext: runAsNonRoot: true
kubectl get events | grep runAsNonRoot
Pod Security Policy ; Enforcer
apiVersion: policy/v1beta1 kind: PodSecurityPolicy
PSP - Pains ; Easy to break your cluster
Resource Quota ; Limits on CPU, memory, Storage, object counts
Namespaces, ensure : Resource Quotas, PSP setup right, Network Policies, LimitRanges
Neuroscience and devops
CALMS - Culture ; Automation ; Lean ; Measurement ; Sharing
Squirrel's brains grow in the Autumn
Working memory - what can you memory. Frontal lobe can only cope with about 3 bits of information at a time.
Practice changes the way the brain works.
We're asking people to change the way they're working.
Fear & Blame. Automonous cells. Not doing things in big project ways, change to small iterations.
Devops => Batch size = Micro, Autonomous, decentralised, "Like breathing", Actionable, High trust / fail early, metric = flow (value and time), DoD = Value outcome realised.
We are biological wired to resist change. Change can be felt as a threat.
Big Bang, Big J (Transformation) vs Incremental, Little Js (Evolution)
Encourage failure (i.e. experimentation & learning which implies failure along the way), however We want to avoid catastrophic failure in production.
Evolution : Discover what engages your colleagues ; Create psychological safety to improve learning ; Model the behaviour you want to see - you are the mirror.
The Rational for Continuous Delivery
Dave Farley - Continuous Delivery Ltd
AWS Fargate - Run containers without managing servers or clusters
Deploying Code Changes with Confidence using Consumer-driven contracts
PACT - contract testing tool
Function as entry point, OS etc taken care of by platform.
Same principles as other devops, just different tools.
Inner loop - Optimised for fast feedback, usually focused on a single problem Outer loop - designed to be robust
Declaritive App Specifications - specify the end goal (not the details of how it should do it) => Uniform deployment, Easy upgrades, Validation before deploy, live-reload testing
- aws-sam - AWS Serverless Application Model
- serverless.com framework
- k8s declaritive by default -
kubectl apply *.yaml
- fission- Open source, Kubernetes-native Serverless Framework -
fission spec apply
Monitoring : log aggregation ; metrics ; tracking ; alerts
Automated Canaries - collect metrics, decide if AOK, if AOK send it more traffic.
Automated Canaries in Lambda Natively supported by AWS Lambda using function aliases.
fission canary-config create --name canary-1 \ --funcN func-v2 --funcN-1 func-v1 \ --httptrigger route-canary \ --increment-step 10 --increment-interval 30s \ --failure-threshold 10
Lambda optimised for interactive workloads, spotty usage elasticicity of usage. Not great for batch processing / CPU bound processes.
Cold-starts - keep-warm hacks don't really work. How Slow Are Cold Starts?
You can set rate limits - "reserved concurrency".
Open source tool may be better for batch processing. Give you more controls. Deploy k8s on EC2 instance.
Deploying with Security in Mind
Tools for the ignorant - there aren't any. You need to reduce your ignorance first.
We download stuff to get the job done.
Malware actors - organised, methodical and highly skilled.
Vulnerabilities in application / container / server/ cluster / network. New vulnerabilities emerge.
Defense in depth and detection.
API discovery. Swagger publications etc
curl --head <your.web.site>/api/vi
90% of time they are looking for initial chance to run arbitrary code.
Known vulnerabilities - POODLE / BREACH / CRIME / BEAST
mitmproxy - man in the middle https proxy ; reverse engineer ; malware / xss in browser.
Always assume there is malovent robot at other end of connection.
Privilege escalation ; find non-secure routes
Overlapping authority not concentric. Having an all powerful admin user is a risk.
Compartmentalisation is key ; no one set of keys gets you master access.
shodan - search engine for Internet-connected devices.
Fuzzing - firing in data to break / discover behaviour.
Less guessable paths, different port, special headers
Reduce opportunity for drive by discovery, use UUIDs not sequential numbers, simple IDs.
Review error messages - don't give unnecessary insight.
Log unexpected behaviour.
Build Vulnerability remediation into your pipeline
Data dog - monitoring