Cloud-Powered Continuous Integration and Deployment

The speaker comes from Amazon and wants to give us advice on Continuous Integration (CI) and Continuous Delivery (CD) - in the cloud. In the end, I only participated in the first half of his presentation that contained CI basic stuff. When it would have been interesting, the speaker started to be really Amazon EC2 centric... not moch to retain from this session, maybe except the Poka yo-ke technique (see below).

Presentation slides here.

At amazon, the Customer is the center of their universe

Old way was to flow from Requirements to Development, Check-in to Testing and QA to Release. Most of the learning from customers is at the end at the Release phase and a little bit on Requirement phase but pretty much none the middle phases so to learn faster, the goal is to make the inner steps smaller and apply CI, CD and Continuous Optimizations

CI

• Goal: to have a working state of the code at any point in time
• Benefit: fix bug earlier when they are cheaper to fix
• Metric: a new guy can checkout and compile on his first day on the job

Poka yo-ke technique

See here: A poka-yoke is any mechanism in a lean manufacturing process that helps an equipment operator avoid (yokeru) mistakes (poka). Its purpose is to eliminate product defects by preventing, correcting, or drawing attention to human errors as they occur.

Ex: ATM machines with card swipers designed so you keep you card at hand instead of those that "eat" your card offering you a way to possibly forget to take it back when you leave the ATM.

Basic lessons

1. Keep absolutely everything in version control (scripts, etc.)
2. Commit early, commit often
3. Always checkin to trunk, avoid branching
4. take responsibility for check ins breaking the build
5. automate the build, test, deploy process
6. be prepared to stop the mainline when breaking occurs
7. only one way to deploy and everybody uses the same way
8. be prepared to revert to previous versions

Cloud

VCS systems in the cloud, CI servers in the cloud, distributed build in the cloud

They use Jenkins, Team city, cruise control, bamboo

Uptime in High Volume Systems - Lessons Learned

Excellent presentation from Urban Airship (UA) - mainly, what struck me is the way they seem to be Agile and Lean in the real sense of the terms, and at all levels of their company, from Engineering to Operations, etc. The guy had way too many slides for an hour however... here's the overview (more details at first, then, less and less as the speaker accelerated... I will put the slides below for those interested in more details!):

Presentation slides here.

About Urban Airship

• Hosting for mobile services
• Unified API for services across platforms
• Content delivery at all scale
• SLAs for throughput, latency
• Apple, Android, RIM

UA is a Lean Company

• Specifically Lean startup
• From wikipedia: Use of FOSS and employment of Agile Techniques; is a "ferocious customer-centric rapid iteration company"
• Attention to Continuous Improvement
• Value the elimination of waste
• Transparent, open processes
• Does not apply to just Engineering but also to Operations

UA by numbers

• more than 20K developers
• 300 millions active applications installs use our APIs across
• more than 170 millions unique devices
• 10s of billions of API requests per month
• 10 millions direct socket connections to our servers
• more than 50TB worth of analytic data
• 30 software engineers, 5 operations engineers

Obligatory Architecture slide

See slides that will be posted soon

3-tier architecture (using Apache Cassandra, PostgreSQL, Java, Python, HDFS, Hbase big user (like Facebook))

Architecture - General principles

• Keep everyone moving in the same direction
• Help discrete teams understand how they interact
• Think in terms of small discrete services
• Continuous capacity planning based on real data
• Avoid local optimization decision making

Architecture - Services

Trending towards a service based architecture

Critical traits of a services

• Minimal exposed functionality (Smallest reasonable surface area to the API and Operate on one type of data and do it well)
• Simple to operate
• Over exposure of metrics and stats
• Discoverable via ZooKeeper (future)
• Zero visibility into inner workings of other services
• No shared storage mechanisms across services (services are completely fronting their datasets) - Motivation for this: security, performance, scalability
• Minimize shared state - use ZooKeeper if absolutely necessary
• Consistent logging and configuration properties
• Consistent implementation idioms
• Consistent message passing
• Convention for on-disk layout and structure (directory structure is standard on all their nodes)

Architecture waste reduction

• All back-end services are in Java and Python
• All Java services are made to use a single set of operational scripts
• Always looking for new ways to eliminate waste
• Architectural waste comes in many forms (lots of data storage engines - postgreSQL, MongoDB, Cassandra, Hbase, using a complex, unfamiliar queuing system was wasteful, large diversity in approaches for managing services, worker processes, process management, etc.
• Developer silos - avoid the bus factor - they always have at least 2-3 persons per service (and they currently have 35-40 services)

Architecture - fault domains

essentially, they worked hard to ensure that when 2 resources are completely unrelated, they should are isolated fault domains if at all possible.

Engineering at UA

• 46% of the time they develop new features
• 28% spent sustaining internal support
• 21% production support
• 2% - social stuff (beer , ping-pong)

Engineering for iteration

• Team of about 30 engineers
• Small teams organized around functional area (DB guys, etc.)
• Shortest iterations possible - Lean MVP concept (Minimum Viable Product)
• No formal QA team (!) - and they seem to be happy with giving this responsibility to all their people.
• Frequently pairing, but not mandatory - this is the choice of the developers themselves
• They always leave code better than how they found it
• All bug fixed requires a code review in a review board
• Large new developments require a sit-down code/design review (a little more formal, but not that much)

Engineering for automation

3 level of testing (Unit, Functional (with mocks)and Integration). Commits are done to a single main git branch

Engineering for simplicity

They simplified metrics and stats capture and so they can do it everywhere!

They capture latency for Service critical operations and External service invocation

They capture counters for Service critical operations and Services faults

Engineering for Operations

• Tests and Deployment scripts are done within the development teams (their definition of done essentially includes the deployment automation).
• Services deployments done via automation tools and
• Automation scripts always pull from prod git branch after passing auto and manual tests
• They apply the "Put the mechanics on the helicopter" principles

Engineering for Responsiveness

• Low latency, high throughput message paths using an in-house developed RPC system based on Netty and Google PBs.
• They support sync and async clients, journaling of messages
• Latency tolerant message paths using Kafka for pub-sub messaging (they generally favor pub-sub model)

Engineering for Availability

• They use dark launch (like Facebook) which essentially is a roll out of a new functionality to a subset of customers
• Take new service in or out of prod with no customers impact (Double writes, single reads, migration , cutover, Load balanced http with blended traffic to new and old service
• Their service abstraction helps immensely
• Requires extra discipline for co-existing versions or services

Engineering for Continuous Improvement

They use the 5 whys approach

Operations at UA

A team of 5 operations engineers handling more than 100 servers (Mostly bare metal, using EC2 for surge capacity)

Operations for Transparency

They measure absolutely everything, and monitor only the important things

See the slides here: TBD

Retrospectives, Who Needs Them, Anyway?

Book references:

1. Project Retrospectives: A Handbook for Teams Reviews)
2. Agile Retrospectives (Making Good Teams great)

Presentation slides here.


What is a retro?

Essentially it is a chance to reflect and learn (historically called postmortem made to learn from failures)

Norman Keirth (Project Retrospectives: A Handbook for Teams Reviews author) Prime Directive

"regardless of what we discover, we must understand and truly believe that everyone did their best job he or she could, given what was known at the time, his/her skills and abilities, the resources available, and the situation at hand."

What happens during retros - the 5 steps

1. Set the stage - getting ready (ex: create safety, I'm too busy)
2. Gather data - the past (ex: artifacts contest, Timeline)
3. Generate insigths - now (ex: fish-bone, 5 whys, patterns and shifts)
4. Decide what to do - the future (ex: Making the magic happen, SMART goals)
5. Closing the retro - retro sumary (ex: what helped, what hindered, delta)

Stories from the trenches

• We should never impose retros on people not wanting retros... it never works!
• There is a difference between retros and discussions at the cafeteria: for retros, you have (or need!) shared commitment to find solutions.
• Serves to celebrate good things as well.
• Serves as points to evaluate where we are going

Esther Derby's (Agile Retrospectives author) findings

retros can be :

• boring
• painful (ex: finger-pointing) to the point people want to stop them

those 2 issues come from a lack of :

• focus
• participation (thus the need to set the stage properly)
• genuine insight (you can't only do start/stop/cont because they only show symptoms... what is the root cause behind the symptoms)
• buy-in (takes a genuine facilitator - never the team leader - has to be neutral to the team, ideally from another team)
• follow-through (followup is key otherwise, this is a lost of time)

More stories

• 30 minutes is the shortest retros possible (for a team that knows exactly what to do) - typically, minimum of an hour.
• Finger pointing is a retrospective killer... a key to get out of this pattern is to change subject
• Ground rules are really useful to put constraints on how retros are conducted (no naming no blaming rule for example - interpersonal issues shall not be addressed in retros but by management in parallel)

No naming, no blaming!

people, like kids, respond in much better ways to positive comments than negative comments. Negative comments leads to bad team work and eventually bad results! See the Prime directive again and how it actually helps with this essential ground rules.

Identify changes

It is not enough to reflect, you need to take actions to change things. Use SMART goal and name a responsible to follow up on each goal:

Specific

Measurable

Attainable

Relevant

Timely

(http://en.wikipedia.org/wiki/SMART_criteria)

You start the next retros by asking how we are doing with regards to previously set SMART goals.

Don't "sell" retros, instead

sell a way of learning how to:

• avoid repeated mistakes
• identify and share success

Take away:

If you leave out a step in retrospectives, you will make problems emerge that you do not understand fully nor find solutions to.

And It All Went Horribly Wrong: Debugging Production Systems

Intro: Maurice Wilkes quote:

"As soon as we started programming, we found out to our surprise that it wasn't as easy to get programs right as we had thought. Debugging had to be discovered. I can remember the exact instant when I realized that a large part of my life from then on was going to be spent in finding mistakes in my own programs."

Presentation slides here.

Debugging through the ages :

Production systems are more complicated (abstraction, componeurization, etc.) and less debuggable! When something goes wrong, it is all opaque and more and more difficult to fix...

How have we made it this far?

• we architect ourselves to survive component failure
• we forced ourselves to stateless tiers
• when there are states, we considered semantics (ACID, BASE) to increase availability
• redundant systems
• clouds (especially unreliable ones, like Amazon) have expended the architectural imperative to survive data-center failure

Do we still need to care about failure?

single component failures still has significant costs (both economic and run-time)... but most dangerously, a single component failure puts the global system in a more vulnerable mode where further failures is more likely to happen... This is a cascading failure - and this is what induces failures in mature and reliable systems.

Cascaded failure example 1:

An example of a bridge that collapsed in Tampa Bay because of a boat with ballasts full (which was not supposed to happen) hit the bridge (which was not supposed to happen either) that had been built by a crooked contractor playing with sand/cement ratio in concrete to save money (which was not supposed to happen either - after all, this is Florida, not Quebec!). In the end, it took all those "unlikely" events to all occur to cause the bridge to fall.

Wait, it gets worse

• this assumes that the failure is fail stop
• if failure is transient a single component failure can alone induce system failure
• monitoring attempts to get at this by establishing liveness criteria for the system - and allowing operator to turn transient failure into fatal failure...
• ... but if monitoring becomes too sophisticated or invasive, it risks becoming so complicated as to compound failure.

Cascaded failure example 2:

An image of 737 rudder PCU schematic (details here). Another example of a cascaded failure that led to B737 landing issues.

Debugging in the modern era

- Failure - even or a single component - erodes oeverall reliability system

- When a single component fails, we need to understand why and fix it

Debugging fatal component failure

• when a component fails fatally, its state is static and invalid
• by saving the state, to stable storage, in DRAM for example, the component can be debugged postmortem
• one starts with the invalid states and proceeds backward to find the transition from a valid state to an invalid one
• this technique is old: core dumps

Postmortem advantages

• no run-time systems overhead
• debugging can occur anytime, in parallel of the production system
• tooling can be very rich since the overhead caused to a run-time system is not a problem

Cascaded failure example 3:

Flight Data Recorder from Air France crash (747) found 1.5 years after the crash

This recovery definitely permitted postmortem analysis

Postmortem challenges

• need the mechanism for saving state on failure (ex: core dumps)
• must record sufficient state (program text + program data)
• need sufficient state present in DRAM to allow for debugging
• must manage state such that storage is not overrun by a repeatedly pathological system

Conslusion:

these challenges are real but surmountable - as in some open source systems presented below (MDB, node.js, DTrace, etc.)

Postmortem debugging: MDB

• a debugger in illumos OS (solaris derivative)
• extensible with custom debugger module
• well advanced for native code but much less for dynamic envirnoments such as Java, Python, Ruby, JS, Erlang...
• if components going into infrastructures are developed using those languages it is critical that they support postmortem debugging.

Postmortem debugging: node.js

• not really interesting for non JAVA...
• debugging a dynamic environment requires a high degree of VM specificity in the debugger...
• see all details on dtrace.org/.../nodejs-v8-postmortem-debugging

Debugging transient component failure

• fatal failures, despite its violence, can be root-caused from a single failure
• non fatal failure, it is more difficult to compensate for and debug

Reliability Engineering Matters, Except When It Doesn't

A presentation on Reliability Engineering (RE) and how it can be used for software systems. Quite a confusing presentation doing a full circle from

1. RE is good for many domains including software to
2. RE is very complex for software systems to
3. But it is still a tool that you can use, not just an absolute one...

Presentation content:

This book written by the presenter was introduced by the host as an excellent book on the subject: "Release It!" by Michael T. Nygard

Presentation slides here.

Context and example

Reliability Engineering (RE) is a lot about maths so the speaker tried to present concrete examples. What if the hotel bar lighting rack falls on happy drinkers (ex: Pascal)? We could analyse the supporting chain and everything using static & mechanics principles. This is fine, but this abstracts other possibilities affecting the reliability such as earthquakes, beam wearing out, drunk people hanging on it occasionally, etc. or we can analyse this from another perspective: The rack held properly yesterday, today is a lot like yesterday, so the rack will be OK today again... but the point is that it will for sure eventually break and this is what RE is about.

RE Maths

The presenter went through many mathematical models with hazards equations, fault density, etc. Those came from many disciplines from which RE software inspires itself, and in which the probability of failures augments with time; but software do not really wear out... so we cannot simply apply blindly those models.

By taking a single server example and then a multiple servers example, he presented the Reliability Graphs (http://en.wikipedia.org/wiki/Reliability_block_diagram) where there is a start node and an end node and everything in between are successful reliability paths. A single path is subject to a global system failure if any subsystem fails.

3 Types of failures

1. Independent failure: Failure of one unit does not make another unit more likely to fail (excellent! but is that true for software systems? not likely)
2. Correlated failure: Failure of one unit makes another unit more likely to fail.
3. Common mode failure: Something else, external to the measured system, makes 2 redundant units likely to fail (ex: redundant LEDs in an Apollo capsule that were both subject to overheating)

Important Note for RE when applied to software systems:

Lots of software reliability analysis make the error of assuming a perfect independence between duplicated resources. The 2 last types of failures are way more common.

Blabla you should probably skip unless you know Andrey Markov (the mathematician)...

The speaker then went through more real-life examples, mainly to lead us to the various pitfalls of formal analysis with regards to software reliability. He showed that because of Load Balacing algorithms for example, redundant systems were not independent and that a failure on one was certainly augmenting the probability of failure on other ones (receiving the load of the failed ones in this example). Also, if you run a system with 9 servers, it is certainly because you need most of them to be alive for the system to work (otherwise, you overbuilt your system...); he introduced the concept of what is the minimum number of systems required to be alive out of the total number of systems deployed. Interestingly, one of the pitfall in increasing reliability seems to be that the fail over mechanisms tend to bring with them their own failure paths... In the end, when he used quite a small system for an example (maybe 5 machines) and put in the "invisible" systems from the logical system block diagram (ex: switches, routers, hard drives, etc.) in the probability of failure equation, the result was no less than spectacular (I can't write this equation down!). But it certainly proved again that true independence is not existing most often in software systems. Also, unlike the famous constructions analogies, a failed software system can come back up. To handle that aspect, he introduced the work of this mathematician, Andrey Markov and its Markov model (essentially state diagrams with probabilities of change states on each arcs). Markov systems are awfully complex for the simplest systems...

Limitations of RE with regards to software systems:

1. Intractable math (only few systems have closed forms distributions like gaussian - more often they have exponential (ex: a single server), log normal (ex: multiplicative failures) or weibull (ex: hardware) distributions). There is also the "repair" distributions that could be modeled with a Poisson distribution. But in the end, which distributions do we apply to software systems? None: Software fails based on load, not on time (as opposed to typical engineering disciplines)
2. Curse of Dimensionnality (you should have seen the Markov model for the simple system)
3. Mean Time Between Failure (MTBF) is Bullshit! Note: See Google analysis a few years back on hard drive failures. Would be good to get this data for other devices (servers)

other big killers:

1. human error (50-65% of all outages!)
2. interiority
3. distributed failure modes
4. Lack of independence between nodes and layers

So in the end, should we abandon Reliability Engineering for software system?

NO! Even if RE cannot tell you when your system is OK, it can tell you whenit is not. Modeling reduces uncertainty. Use RE like other model and apply it when your system is at risk.

Reference back to the ambivalent presentation title "Reliability Engineering Matters ... Except When It Doesn't".

Thursday's Plan

Here is today's plan following the keynote. Don't hesitate to give your feedback or start a discussion in the comments.

Myself:

• Software Quality - You Know it When You See It - link
• Lean Startup: Why It Rocks Far More Than Agile Development - link
• Experiences with Architecture Governance of Large Industrial Software - link
• The DevOps Approach to Performance - link
• Dealing With Performanc Challenges: Optimized Serialization Techniques - link

Steve 2.0:

• Reliability Engineering Matters, Except When It Doesn't - link
• And It All Went Horribly Wrong: Debugging Production Systems - link
• Retrospectives, Who Needs Them, Anyway? - link
• Uptime in High Volume Systems - Lessons Learned - link
• Cloud-Powered Continuous Integration and Deployment - link

Impressions of the conference so far...

Ok, day 1 of the conference is done. Now, what can be said about the experience up until now? Well, for one, I think QCon SF is awesome. The organization of such an event can be difficult as there are many people, many speakers, lots of coordination and a bunch of areas where slip-ups can occur but Trifork and InfoQ have got everything under control. The venue layout minimizes walking distance between presentations and the hotel staff is very efficient and courteous, unlike the cable car operators. So, organization-wise, I have no complaints; We are in very good hands.

On the (slightly) down side, as with any such event, no matter how well you pre-plan on which topics you will cover you, will eventually end-up in a presentation that just sucks. Well… maybe the word is a little strong but, lets instead say that some of the presentation content was not in line with the expected return value inferred from the provided abstracts :) This is when having a pre-determined backup presentation plan becomes very handy.

On the brighter side, the organizers have conjured a very innovative way to pick-up feedback after each presentation. They post people at each exit holding one or two iPods with a simple and very intuitive user interface to provide feedback. The screen is divided into three zones with a smily :( , :|  or :) associated to red, yellow and green; I'll let you guess which is which. Simply brilliant! Like it? Just touch the happy green smily. Hate it? Touch the red frowny one.

Presentation Rating System

Now, what are two things I take away from this first day at the QCon SF conference?

1. To my surprise, I think I have created a blogging monster. Let call him "Steve 2.0". Coming here, we had a choice of documentation mechanisms to bring back information to the mothership: produce a standard (i.e. boring, static, non-interactive) conference report or use a blogging platform that would allow us to take notes and publish content as the day went on. Option B was taken. I'm only hoping that "Steve 2.0" is not a transient state and that he will transpose his newly learned skills to another platform (you know which one…) after all this is over.
2. Mike Lee is an awesome presenter. Going through a somewhat tiresome day of active listening, idea exchanging and intensive note taking was well worth the trouble for this guy's keynote. If you do not know who Mike Lee is, here is a little background information. His keynote on Product Engineering was done wearing a mariachi suit which fitted this guy's personality and on-stage presence very well. I wish I could show you the video of it but as a consolation prize [Update: Here's a video on InfoQ of the same presentation given at the Strange Loop conference], I will link you to his presentation slides. 

Mike Lee on Product Engineering

So, one down and two more to go. We will be posting the expected attendance schedule for day 2 very soon, so keep reading and don't forget to comment on our posts. We'll definitely take the time to answer any questions you may have.

Stratos - an Open Source Cloud Platform

Presentation given by WSO2, the producer of Stratos cloud-reated products. Here's some pointers from their presentation. I think this should interest all our people working on SOA implementation.

Presentation slides here.

Slide 1

• Presentation of Platform as a Service (PaaS)
• online data doubles online every 15 months and the number of apps kind of follow the same trend
• what the Cloud is really depends on who you are (ex: online music for teenagers, emails for your mom, prospects for a sales guy, etc.)

Slide 2:

• PaaS (Platform as a Service) is what is positioned between IaaS (Infrastructure as a Service, ex: Amazon) and SaaS (Software as a Service, ex: Google Apps)
• Stratos tries to insert itself in between the hardware and the software platforms.

Slide 3:

What does Cloud native means:

• Distributed/Dynamically wired (i.e. it works properly in the cloud)
• Elastic (i.e. it uses the cloud efficiently)
• Multi-tenant (i.e. it only costs when you use it)
• Self-service (i.e. you put it in the hands of users)
• Granularity billed (i.e. you pay for just what you use)
• Incrementally deployed and tested (i.e. it support seamless live upgrades - Continuous update, SxS, in-place testing and incremental deployment)

Slide 4:

Apply those concepts to an Enterprise architecture… Cloud is no only about Web apps, it is also about Portal, Queues and topics, DB, Registry/Repositories, Rules/CEP queries, Integrations, etc.

Slide 5:

What are the various dimensions to evaluate a PaaS:

• Which languages and APIs does it support? (Are you locked in?)
• Can it run on a private cloud? (Are you locked in?)
• Which services does it offer? (Are you locked in?)
• Is it open source? (Are you locked in?)

Slide 6:

What are the cloud players:

1- Those without private PaaS

• Force.com / Heroku
• Google App Engine
• Amazon Elastic Beanstalk

2- Those with provate PaaS

• Tibco
• Microsoft
• Cloudbees

3- Those with both

• Stratos (obviously...)
• others I missed.

Slide 6:

What is Stratos

• A product: Open source, based on OSGi
• Services (based on the product) called Stratos Live

Then we got a live demo of Stratus Live

For more details, see http://wso2.com/cloud/stratos/

This maybe interesting for App server, ESB, Repositories, etc. seems to be a fairly complete, consistent, integrated portfolio of middleware services.

Here's a list of what they claim their offer in Stratos Live:

1. Application Server as a service
2. Data as a service
3. identity as a service
4. Governance as a service (this one is for Pascal!)
5. Business Activity Monitoring as a service
6. Business Processes as a service
7. Business Rules as a service
8. Enterprise Service Bus (ESB) as a service
9. Message Broker as a service
10. etc.

JVM performance optimizations at Twitter's scale

Complete presentation about application optimization based on twitter experiments; it was Java-centric, but I tried to generalize the presented principles.

and for those interested... I still do not like Java!




Takeaway: Twitter went from 60 second of Garbage Collection (GC) per hour to 5 seconds per 3.5 days, mainly because of the internal slab allocator.

Presentation slides here.

Twitter's biggest enemy: LatencyWhat are the main latency contributors (from worst to less):

1. Garbage collector (GC)
2. in process locking and thread scheduling
3. I/O
4. application inefficient algorithms

but only the first one can be addressed by JVM tuning (which happens to be the expertise of the speaker)

Areas of performance tuning:

• Memory tuning (footprint, allocation rate, garbage collector rate)
• Lock contention

Memory footprint tuning

• Maybe you just use too much data!
• Maybe your data representation is fat! (bad data model)
• Maybe you have a memory leak (not covered in remaining presentation)

Global advice for performance tuning:

Profile your applications, especially when 3rd parties are involved abd not well known (ex: primitive SCALA array types eating lots of memory; another ex: Guava's concurrent map eating a lot of resource - especially if you do not need concurrency).

Even more important advice:

Writing a new memory manager is not a good idea unless you already optimized your data and your JVM... if you start doing this, it is a strong smell that you are doing something wrong!

Fighting latency

Performance tradeoff: Memory vs Time (convenient and valid, but oversimplified view); the performance triangle (press memory usage down, push throughput up and keep latency low) is a better model, but still too simple for Twitter optimization team. In the end they use a (optimization increase) = C x T x R where C=compactness (good data model), T= throughput and R=responsiveness.

The rest of the presentation was about tuning and optimizing Java JVM Garbage Collector (GC) - if you are interested, let me know (I will not document all of this in this post - lucky you!).

Side note:

At Twitter, they seem to also have a big gap between developer's "done" and "production ready" done.

"DevOps" - Change is NOT the Enemy

From Wikipedia (link):

"DevOps" is an emerging set of principles, methods and practices for communication, collaboration and integration between software development (application/software engineering) and IT operations (systems administration/infrastructure) professionals"

Sounds familiar? It should.

If you've ever been part of one of these groups at some point in your career, you probably said to yourself or heard things such as: "Those developers just want to rapidly push on us all this new technology but we've yet to handle issues with the currently deployed tech." or maybe "Operations are dinosaurs; they reject any opportunity to use our new software and make us stick to old outdated tech. without realizing the benefits of the new!". In essence, agile development creates change and operations want less change.

As organizations make the move to new technologies (e.g. cloud, SaaS, PaaS, WCF, .NET 4.0, etc...), the role of development versus operations teams is evolving and this change must be properly managed but, change isn't the enemy, the lack of alignment is. Businesses fail to see this and think pushing change down the pipe will magically break their reliance on monolithic applications and help them move towards more modern distributed service based applications. The fact to the matter is, distributed and loosely coupled applications are more complex to manage and also fail in a distributed way!

This type of change has to be monitored, measure and managed so both parties can establish a better communication and collaboration relationship. Can you think of any examples where this has happened to you? (post them in the comments)

Obviously, this post is merely skims the topic of "DevOps" but, I'm hoping it got you thinking a little and wanting to read-up on this more. I will be attending a presentation on Thursday more specifically on "DevOps" applied to addressing performance issues so hopefully I will have some detailed examples. I'll post a follow-up on this subject then.