Is DevOps in banking possible?

by Piotr Stapp

Piotr Stapp

Blog: http://stapp.space
Twitter: @ptrstpp950
Work at

Is DevOps in banking possible?

DevOps was originally about WebOps at Internet companies working in the Cloud and a handful of Lean Startups in Silicon Valley. It started at these companies because they had to move quickly, so they found new, simple, and collaborative ways of working that allowed them to innovate much faster and to scale much more effectively than organizations had done before.
DevOps for Finance - Jim Bird

Why not?

Because it is complicated
Because of law regulations
When Facebook is down, people tweet about it, but when the bank is down people take money somewhere else.

Let me tell you a history about "what can happen when the bank is down". Imagine that you invite parents of your bride-to-be to the official dinner in the very expensive restaurant. To show off, how good you are. After the dinner, you would like to pay, but your bank is down. How do you feel?. This history is true and was published on social media. It happen not from our fault, but it happened. We lost our client. Such situation would happen on facebook or eBay because banking is much more serious stuff

Now you will say no way. Facebook is different. On January a lot of services published information about
"Facebook made its Android app crash to test your loyalty".
The conclusion was amaizing: "People never stopped coming back"

But what am I doing here if the answer is nooooooo? next slide

But ...

ideas - can be adopted
practices - can be reused
tools - are ready

IT vs Sport

Tennis, swimming, runners, ... - we aren't alone
Football - no many goals in the match
Handball - who is "goalkeeper" in IT?

Basketball

Basketball is a sport [...]. The objective is to shoot a ball through a hoop 18 inches (46 cm) in diameter and 10 feet (3.048 m) high mounted to a backboard at each end.

Main rule

We are wathing Top 10 Long Distance Shots of 2011-2012!
----------
Why? Because DevOps is like a Basketball.
We want to take a ball and put into the basket.
The objective is to install software on production :)
There is many ways to achive that. For example throwing from the long distance. Like guys above. Just throw, cross your fingers and you are ready :)

Quater 1: Statistics

skip quater

In every professional sport, we can browse stats. To view the history of a player, team or even a coach.

To ask questions like:

Do he or she perform well?
Why is the team A better?
Which players have problems? And much more.
Even maybe why Athletico win with Barcelona in Champions League?

In IT system situation is similar. But instead of statistics we have logs.

Why do I need log aggregator?

Easy to browse log - there are in one place
Easy to correlate - there are in one place
Easy to browse history - it is in one place
Make alerts - if platform supports

Logs are a critical part of any system, they give you insight into what a system is doing as well what happened
Virtually every process running on a system generates logs in some form or another.
Usually, these logs are written to files on local disks.
When your system grows to multiple hosts, managing the logs and accessing them can get complicated.
Searching for a particular error across hundreds of log files on hundreds of servers is difficult without good tools.
A common approach to this problem is to setup a centralized logging solution so that multiple logs can be aggregated in a central location.

Splunk VS ELK

Most popular logs aggregators in onsite model are:

ELK - Elastic Search with Kibana
Splunk
Systems based on HDFS and Kafka

Splunk - It's a very powerful, mainly on-premise solution. More or less very expensive :)
If you only want a "supergrep" command it is probably a wrong choice

ELK - License fees not incurred, but you'll end up spending $ on building various features because ELK doesn't have them. It is a "supergrep". Rest you will have to do by your own

About the rest - I don't know. I never used them.

Logging SaaS

Top 10 from
http://stackshare.io/log-management

Papertail - 10 stars, 221 Stacks
Logstash - 13 stars, 240 stacks
Logentries - 9 stars, 157 stacks
Loggy - 3 stars, 114 stacks
Graylog - 26 stars, 23 stacks
Sumo logic - 10 stars, 30 stacks
Splunk Cloud - 2 stars, 10 stacks
AWS CloudTrail - 0 stars, 19 stacks
Log DNA - 17 stars, 5 stacks
Fluentd- 3 stars, 7 stacks
Scalyr- 2 stars, 5 stacks
Logsene- 0 stars, 2 stacks

If you could use it, you will save probably time and money. There are a lot of them.
Check your needs and decide to use one. Because it is a SaaS it is easy to change to better one :)

Log aggregators

Demo time

Something more?

Marketing queries
Alerts
Data analytics

Business queries - for example, I created dashboard with information how many PB users started new application in time period
Alerts - I set up an everyday query about interesting events. They can be real-time
Data analytics - how your system performs in comparison to last year/month/week

Quater 2: Server&Release automation

Lots of services + frequent deployment = a global headache

skip quater

Do I need a deployment tool?

"It depends" is the best answer

Example simple tools:

Azure Web Sites - automatically deploy from git
XCOPY - ugly but enough in simple cases
Setup project - great for Windows app
"Store" app - separate process

Config hell

How to manage configuration for 3+ different environments:

Manually create config
Use App.Release.config (with SlowCheetah)
Put in source control MyApp.ENV.config
No config differences - changes in hosts file

The first symptom of complicated deployment is Config hell

Let's stop with the number "3". Why is it 3+? Because simplest architecture assumes 3: dev, test, prem.
Simple next step is "prem-copy". Then we add "performance env", "uat env", "feature branch env". And we end with 10, 20 or even 100

Manual steps are bad. Always bad. After 100 reapeats you will kill someone:D
Next try - config transormation are the easy way to manage simple config definitions. For different files than Web.Config we can use SlowCheetah plugin.
Files like MyApp.Enviroment.config in git are so difficult to manage. When we do any change we have to remember about all files. It is too easy to make an error. But this is a very popular solution. One very big disadvantage for above solutions: every developer can access and change all config files.
That is a really cool approach. Imagine that you hardcode all hosts names like "myAppApplicationServer" and uses a certificate for connection called "mySuperSecretCertificate". On diffrent enviroments, you just switch contents of DNS or cerificate store or anything you need.
The only problems are with external secret keys like facebook api secret key. But you can solve it :) I want to try this approach one day. It sounds really cool.

Tokens

One more approach


<add name="ContractX" path="Something/ABC" server="${AppsrvName}" 
     port="${Core_port}" binding="TCP" 
     serverThumbPrint="${AppSrvServerThumbPrint}" 
     clientThumbPrint="${AppSrvClientThumbPrint}"/>

+


<AppsrvName>XYZ.mbank.pl</AppsrvName>
<Core_port>8888</Core_port>
<AppSrvServerThumbPrint>29[...]76</AppSrvServerThumbPrint>
<AppSrvClientThumbPrint>98[...]84</AppSrvClientThumbPrint>

=


<add name="ContractX" path="Something/ABC" server="XYZ.mbank.pl" 
     port="8888" binding="TCP" 
     serverThumbPrint="29[...]76" 
     clientThumbPrint="98[...]84"/>

My favorite approach. In my company, we are using it almost everywhere. Instead of having multiple config files, we have one. But with "gaps" called tokes. They have specific notation. In our case, it is the $ sign and mustache brackets.
This approach is used in mamy popular DevOps tools. For example Octopus Deploy and Chef. Of course, notaion is different.
Every instalation just fill tokens and configs are almost same on every enviroment.
---
Now let's talk about the funny error from migration into .NET 4.6.1. We had a token with SQL server intance name. On every enviroments we had a named server like "SqlServerHost", but on production, we had IP adress. Fortunately, Jacek (the administaror) checked fast what is wrong on StackOverflow and fix the token. That is a great example of DevOps culture and responsible on both sides.

Lesson learned

Config managment can be easy. Just try!

Quater 3: Version control

skip quater

Why should I use version control?

Lost code or had a backup that was too old?

Had to maintain multiple versions of a product?

Wanted to see the diff between versions?

Wanted to submit a change to someone else's code?

Wanted to share your code?

[...]

DB with VCS?

In most cases all questions from previous stuff apply to databases.
But a lot of SQL code is not (full) in VCS.

DB with VCS - tools

A few examples:

VS database project
Redgate stuff
Fluent or EF migrations
DbUp - simple great #;OSS stuff
Home-made tool - e.g:
Dacpac Data Migrations
by Maciej Warszawski https://github.com/maciejw/dacpac-data-migrations

The tools that belong to the same class retain the same principles and ideas. If you are familiar with one of such tool, you will find it pretty easy to learn how to work with another one. Nevertheless, the functionality behind them might differ a lot, so it’s important to carefully choose one that fulfills your project’s needs the most.
----
A small funny history from my life. In mBank one of the databases is an audit one. It is backuped every day. I want to change backup procedure to make another copy. It is completly not important why here so I will ommit this part.
----
The orginal procedure wast commited as simple SQL file in the repository with all DB structure. If something had changed in the structure we send all SQL files. We can just drop and create a new DB because it was audit stuff, backuped somewhere.
----
I decided to change step number 3 out of 5. But to be sure I send the question to DBA with changes. Now I would like You to do a small survey:

DBA survey

Question: how DBA answer to changes in the third step:

Sure it looks ok
Sure here are a small fix - this way it will work better
You are stupid, get out (in polite words)
The third step is commented out on production

Lesson learned

No matter how:

good tools you have
you deploy stuff
release changes

remember: DevOps is about people and trust

Quater 4: Deploy

skip quater

How to deploy? - tools

Custom stuff (e.g. Jenkis+PowerShell)
Chef / Puppet / Ansible
OctopusDeploy
Other

Why custom stuff?

You already have it :)
You have something strange/differenent, so you will have to code it anyway
You like to create your own:)

Why do I put Chef/Puppet/Ansible in one line and Octopus in other>

Octopus and Chef/Puppet/Ansible are different. In fact, I don't think there's any reason to choose.

Puppet and Chef are configuration management tools. They work by describing the state that a system should be in, and they take steps to ensure systems are in that state.

For example, imagine you were managing 30 web servers. Rather than using remote desktop to configure them all, installing Windows components, enabling features, and so on, you could use Puppet or Chef to specify:

All web servers should have IIS configured with the Windows Authentication module enabled

MSMQ should be installed and running

A folder at C:\Logs should exist and be writable by a given user account

https://octopus.com/blog/octopus-vs-puppet-chef

More tools

Versus

Lesson learned

Solve existing problems with the best tools.

Overtime: Service discovery

skip overtime

The problem

The Client-Side Discovery Pattern

The Server-Side Discovery Pattern

The server-side discovery pattern has several benefits and drawbacks. One great benefit of this pattern is that details of discovery are abstracted away from the client. Clients simply make requests to the load balancer. This eliminates the need to implement discovery logic for each programming language and framework used by your service clients. Also, as mentioned above, some deployment environments provide this functionality for free. This pattern also has some drawbacks, however. Unless the load balancer is provided by the deployment environment, it is yet another highly available system component that you need to set up and manage.
https://www.nginx.com/blog/service-discovery-in-a-microservices-architecture/

The Service Registry

Files++
Simple database
Netflix Eureka
etcd
Apache Zookeeper
Consul

Files - good as first step, but nothing more. Tokens+automatic deployment helps but it doesn't solve everything. It is a quite popular solution. For example included in RabbitMq, Cassandra as service discovery start. Also my company use something like in retail banking.
Variation of files. But provides single point of failure. Used in other part of my company. In the corporate banking.
, Netflix Eureka is a good example of a service registry. It provides a REST API for registering and querying service instances. A service instance registers its network location using a POST request. Every 30 seconds it must refresh its registration using a PUT request. A registration is removed by either using an HTTP DELETE request or by the instance registration timing out. As you might expect, a client can retrieve the registered service instances by using an HTTP GET request.
etcd – A highly available, distributed, consistent, key-value store that is used for shared configuration and service discovery. Two notable projects that use etcd are Kubernetes and Cloud Foundry.
Apache Zookeeper – A widely used, high-performance coordination service for distributed applications. Apache Zookeeper was originally a subproject of Hadoop but is now a top-level project.
consul – A tool for discovering and configuring services. It provides an API that allows clients to register and discover services. Consul can perform health checks to determine service availability.

Consul.io

Service Discovery - REST & DNS
Failure Detection
Key/Value Storage
Multi-Datacenter / Multi-agent

Lesson learned

Invite me in the future :)

Players: References

Service Discovery in a Microservices Architecture https://www.nginx.com/blog/service-discovery-in-a-microservices-architecture/
DevOps for Finance - Jim Bird https://www.oreilly.com/ideas/devops-for-finance
Wikpedia https://en.wikipedia.org/wiki/Basketball
Why should I use version control? http://stackoverflow.com/questions/1408450/why-should-i-use-version-control
Octopus Deploy vs. Puppet/Chef https://octopus.com/blog/octopus-vs-puppet-chef
Facebook made its Android app crash to test your loyaltyhttp://www.theverge.com/2016/1/4/10708590/facebook-google-android-app-crash-tests

Questions?

Few links:

Blog: http://stapp.space
Twitter: @ptrstpp950
Facebook: https://www.facebook.com/stapp.space
Mail piotr.stapp AT gmail.com
Slides: http://stapp.space/speaking