DevOps rant: Pussy developers

Series Overview

I moved to a DevOps team about a year ago and although we’re not really doing DevOps, it’s a good team and we try really hard sometimes! While trying hard I have come across all sorts of funny stuff and recently I have decided to blog about it, maybe someone reading this won’t allow folks to do the same mistakes when presented with the same funny stuff.

Overview

When a development team, or a group of teams, collectively act like a bunch of pussies, they get into trouble easily. Often, this trouble they get themselves into will span across into other folks and teams, and when they aren’t also collectively a bunch of pussies, this will upset them.

The Pussy Developer

A pussy developer is typically a guy that will always agree and do whatever they throw at him. The most intense example would be when you ask the technical lead of your off-shore team in India: “Hey, can you guys build a button on the system that every time I press it, popcorn will come out at my desk?” and you get the obvious answer “Sure, that’s not a problem“.

I’m of the opinion that today successful software is very different then what it was 10 years ago. With the agile mind-set the best thing you can do as a developer is to write lean code, the leaner, the better you will be able to cope with change and the more Agile you’ll be. So things like abstractions don’t really fit today’s lean codebases, you want to deal with the now and ignore the “What if in the future we (…)” and instead align the codebase to deal with those “What if’s” in a very quick way when they actually happen.

I keep seeing folks that haven’t written good code in the last 5 years, tell developers how to write code, as if 5 years later you’d expect things in the fastest changing engineering space to be done the same. Worse, I keep on seeing folks being pushed by others into technology hubs they aren’t comfortable with. And while all this is going on, not a single fuck off.

The genuine knowledge driven nature of developers

I love listening to genuine sports folks talking about their art. Living in Ireland, while being for Portugal has inevitably let to me following two genuine sports figures: Guy Martin and Conor McGregor. I was listening to the radio a few weeks ago and they had on a radio show: Conor McGregor and a high reputation sports comentator. They start the show and McGregor starts disagreeing with the commentator on the subject of enduring pain and commiting to a certain sport and after disagreeing with her twice he says “You sit your fat ass in a fucking sofa all day long and you’re talking to me about pain? What do you know about pain?” and then he carries on until he completly destroys the show. This is expectable when you mix the doer (and he’s not a pussy) with the thinker and they start to disagree.

If McGregor was a developer he’d probably say something in the lines of: “You haven’t written code in 5 years and you’re fucking telling me how to write code?“.

Developers are technical folks. “Technical” comes from technique and it means these folks are more interested in technique then in application, or in other words they care about how a system works instead of what a system does, unlike business users for example. So they pride themselves in how the system works and if it doesn’t work properly, despite doing everything it’s supposed to do, there is no pride, no joy and no fun. This is one of the reasons why turnover is so high in developers and still today a lot of IT management doesn’t get this.

So if you pride yourself on building stuff that works well, why would you ever let someone that no longer knows how to build stuff push you around?

Avoid being a pussy

Just tell people to fuck off, in a blunt fashion if the environment allows you to or in a politer and diplomatic fashion if not: I’ll take what you just said into consideration and evaluate before doing the task.

If you’re being pressured into writing code in a specific way that you don’t agree with, ask the person where can you see any of the commits he(she) has done so that you can evaluate if he(she) is a peer or not, because if the person isn’t your peer you shouldn’t be wasting your time being taught how to code by someone who doesn’t code.

If you’re being pressured into the Java stack as a .Net developer, just say “Hey, just get rid of me and find a Java guy, I have no hard feelings guys, it’s business as usual”.

Because if you don’t do these things, the outcome won’t be anything that will ever give you or anyone else that actually built it any pride, joy or even a small spark of fun.

 

Moving XAML CD builds to vNext

I’m currently in the process of porting a series of Continuous Deployment XAML builds to the new vNext builds. I want to share the ins and outs of doing this and some of the current constraints you can face today while doing this.

Continuous Deployment build passes

I like to break down CD builds into two distinct passes: the build and deploy pass and the test pass.

While building and deploying I prefer to target specific projects, because a lot of times you will see multiple deployment targets in the same solution and these will sometimes have different MSBuild /target’s, so applying a single MSBuild target to a solution will sometimes imply you either route targets in the project files (this is a terrible idea) or you build the same solution multiple times, which can lead to confusion.

While for the test pass it’s preferable to build solutions, otherwise you need to constantly maintain the build definitions as people add more test projects. If folks are following good agile practices they will have 1 test assembly per assembly under test and as folks add assemblies, test assemblies will spawn. If it’s an over-engineered system you will notice assemblies will get created like popcorn popping out in the pot, so if you go down the path of targeting projects in the test pass the amount of maintnance required will be high.

Deployment tasks in vNext

One of the main reasons for us to move to vNext was the fact that now if you wire things properly, you get direct feedback from PowerShell tasks as the new engine is PowerShell driven so it will bubble onto the build logs and summary warnings and errors that PowerShell tasks have. Unlike the previous XAML workflow that would log issues in the text log but wouldn’t bubble out anything.

With this in mind, one of the cool out-of-the-box features of vNext is the amount of deployment tasks that the VSO folks have written for us, so right away you can decommission some of the custom deployment scripts and start using the tasks. If these aren’t really ideal for you, you can get their source from github, tweak them and then publish them again into your collection or project as tweaked versions of the original tasks.

So for example, a given Cloud Service build and deploy pass for us looks like this:

Some of the deployment tasks don’t suit our needs, for example we often deploy Azure Web Sites as Web Jobs shells, so what we really care about is the Web Jobs publish process and especially the scheduling pass. The out-of-the-box deployment tasks for Web Sites won’t schedule web jobs.

Build and deployment pass

The first 3 are build + deployment tasks for Cloud Services and the last 3 and pure webdeploy build passes for web deployment targets. One of them is actually a pure web deploy into a Web Server IaaS VM pass.

So for web deployments we are actually calling MSBuild with a build /target and then setting the web deployment properties to trigger the packaging and deployment and giving it a publish profile.

The Publish Build artefacts task is a nice addition to the build engine, because we do not allow developers to access any build rigs. This will actually publish the selected artefacts into TFS and they will stay there for as long as the build stays according to what retention policy we have applied to that build definition. So for this specific case, where we are storing Cloud Service packages, the developer can actually download them, unpack and check if a certain file that should be there is actually there.

Security wise, because the build artefacts can only be viewed by someone with view permissions on the build, you get to control this. For example for Production builds, you wouldn’t have your developers viewing the builds anyways, so they won’t see the artefacts nor any configuration contained in them.

Test pass

Ideally we would be calling a VS Build task for each solution. However, in our current TFS on-premises version (2015 RTM) we have found out that the VS Build task keeps adding a bunch of stuff to the path environment variable each time it runs, so after a while you reach the maximum size for an environment variable and the task will error right away breaking the build. For us this is calling the VS Build task 8 times, if we try the 9th we will get the path error right away.

So we have a single pass that picks up solution on a matching pattern that we, by convention, have set for solutions that contain tests: *.B.sln. Then we run all test passes individually for whatever test passes a given project has. These test passes use Test Traits to pick and filter the tests that should run on them.

We do not usually run Unit Tests, because these are in all Continuous Integration builds, so from a quality point of view the CI pass should have cleared out any issue related to failing Unit Tests by now.

Creating builds that are easily cloned

The first thing you need to do before you fully automate the creation of Release Pipelines is to make sure that creating vNext builds from code doesn’t need to deal with a lot of moving parts.

So ideally, if you can create a CD build for a new environment just by cloning a build and changing its configuration, you’re in a very good space for automation. One of the tricks of achieving this is making sure that everything is aligned, from code to infrastructure, with solution configurations.

So in our case, we align all PaaS components when we create them with in a fully automated way with the configuration names for a project, so if there is a solution configuration called AT, then the Cloud Service for the AT environment will be called sysname-component-AT. Ideally you want a single set of configurations across all branches, so that you have a leaner amount of configurations you need to deal with and so that you prevent in a smooth automated way branches to deploy to environments they shouldn’t be deploying to. However, sometimes folks will resist this idea.

With this in mind, try to use $(BuildConfiguration) as much as possible, for example:

 

DevOps rant: The maintenance test

Series Overview

I moved to a DevOps team about a year ago and although we’re not really doing DevOps, it’s a good team and we try really hard sometimes! While trying hard I have come across all sorts of funny stuff and recently I have decided to blog about it, maybe someone reading this won’t allow folks to do the same mistakes when presented with the same funny stuff.

Overview

By now you’re probably wondering What the fuck is a maintenance test?

Well, it’s definitely not a test, but instead it’s an automated runbook that a developer, because he probably lacks operations/infrastructure knowledge, decided to write as a test and wire it in a test run in an automated release pipeline.

This specific one is worth mentioning, because the reasons that cause it to be written are the same old mistakes people were doing 10 years ago, and sadly keep repeating them today in hopes of different outcomes.

Context

There is a set of performance tests that create a lot of documents in Sharepoint (in Office365). After a while, the container of these documents has more than 5.000 of these, so Sharepoint, with the default list size applied, will start showing you nothing but an error page saying you have more than 5.000 documents in that list.

This means the test needs to clean up. Tests that require cleaning up after them will always do it “after” and never before, because you never want to leave a given environment dirty until to you at it again, it’s a bad principle. However, this set of performance tests decided to “try” to clean up, before the test run, leaving the environment unusable during performance test runs.

This is like you only cleaning up your house before a party, so that it’s always clean on parties but the rest of the time while you’re leaving there you get to enjoy all the dirt and the mess of the previous party.

Moral of the Story

About 10 years ago, all stacks had examples of frameworks or tools that were designed with the goal of anyone can build apps in mind. In the generic sense, without taking into account specific niches, they all failed. In the .Net space the biggest crash was webforms, which was designed around the notion of anyone can drag a few boxes around in the editor, populate a few properties and build any kind of app. The resulting programming model was awful and developers usually tried to stay away from it as much as they could!

The only platforms that truly succeeded in this space were the ones that were built on top of very strong programming frameworks and always allowed for developers to go in and customize/tweak things their way. A good example is Unity3D where the level designer can do a lot in the graphic designer by dragging boxes around, but Mono and C# are at the disposal of developers to build the boxes the other guy drags around.

So, you might think, with all these failures in the history of software, have we all learned that you always need developers around to actually build a piece of code? Obviously not, there are lots of folks out there that actually jump through hundreds of hoops trying to reach the utopia of software without developers.

So sadly we keep on witnessing people using testers to “build” automated UI tests, testers to “build” automated performance tests, etc. This specific example is one of these, where a tester built a performance suite. Because he’s a tester, he has a hard time coming up with a way to properly clean up Sharepoint after his test suite runs.

Because the developer doesn’t want anything to do with a bunch of generated code from the performance test recorder, he wants to stay away from the tester built performance suite, where, ideally, the clean-up code should be written.

My previous contract had a tester building an automated UI test suite for about 6 months, only to realize it wasn’t maintainable. So instead what they decided to do was get a full team of testers to build a new one …

DevOps rant: TFS merge discard strategy

Series Overview

I moved to a DevOps team about a year ago and although we’re not really doing DevOps, it’s a good team and we try really hard sometimes! While trying hard I have come across all sorts of funny stuff and recently I have decided to blog about it, maybe someone reading this won’t allow folks to do the same mistakes when presented with the same funny stuff.

Overview

Today, I’m a solid believer that most TFS projects should be on Git, not TFS SVC. Yes Git does have a learning curve over the massively supported by Visual Studio UI TFS SVC, but once that learning curve is climbed, the rewards are greater.

This is especially true on projects that are using PaaS components and are built by folks that love to over-engineer, so instead of a few components, you end up with tens of components and instead of a few config files you should avoid merging, you end up with tens or even hundreds of these. If you are in a Git repo you just combine clever use of Git Attributes with Git-Filter-Branch, however if you are on a TFS repo, your options are a lot more limited.

Real Life Example

I’m currently working with two projects, one should definitely be using Git as the repo as the level of over-engineering is high, and the other fits nicely in TFS.

The super engineered project never knew how to deal with merges, basically for a very long time what they did was do a “blind merge” then manually undo the changes they thought shouldn’t go in. While this was done by a single person, it actually worked, their problems started when other folks started to merge and they didn’t really know what not to merge.

So their solution was simple: let’s create a project configuration per environment per branch. Let’s not argue about the fact that this is a lot harder to maintain, because honestly if it’s over-engineered, going down the path of arguing about maintainability indexes is purely a waste of time for everyone. But instead focus on what this prevents my DevOps team from doing in the scope of this project.

Let’s imagine DevOps is now given the time resources to build a magic button, that when you press it you get a new branch, a new set of environments and a new release pipeline (after we have built the magic buttons that bring expressos and popcorn!). Currently we aren’t very far from this, the only real automation we are missing is the release pipeline, but that’s not that hard.

When you add the fact you now need new configurations and all sorts of crap related to that, like new config transforms, new service configuration files, etc. you immediately drop the idea of automating.

I have been blabbling about the notion of controlling the merge process through scripting a set of tf merge /discard commands for a while now, but every time I mention it I get that feeling I’m talking Portuguese to a bunch of Indian folks and although they always nod saying “Yes” they are actually thinking “I have no idea what this crazy guy is babbling about“.

So the other project, that’s more on the Lean side of things had this same problem recently. But due to its simplicity I decided to step in and instead of babbling anything just write the script for the project and kick-off the merge workflow instead of giving them the chance to wonder into the realms of creating 10 more solution configurations.

Later I sent the script to the first set of guys so that they could understand what I have been babbling about all this time, but the feedback I indirectly got was that it was “technically advanced” …

The tf merge /discard PowerShell script

	function ApplyMergeDiscard
	{
	[cmdletbinding(SupportsShouldProcess=$true)]
	param
	(
	[Parameter(Mandatory=$true)]
	[string] $LocalPath,
	[Parameter(Mandatory=$true)]
	[ValidateSet("MainIntoDev", "DevIntoMain")]
	[string] $Direction,
	[Parameter(Mandatory=$false)]
	[string] $BaseDevBranch = "$/YOUR PROJECT/BRANCH1/",
	[Parameter(Mandatory=$false)]
	[string] $BaseMainBranch = "$/YOUR PROJECT/BRANCH2/"
	)
	$env:Path = $env:Path + ";C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE"
	$discards = @( `
	# Some stuff you shouldn't merge
	"Stuff1.publish.proj", `
	"Stuff2.publish.proj", `
	# Some more stuff you shouldn't merge
	"Some.Project/AConfiguration.Debug.config", `
	"Some.Project/AConfiguration.Release.config" `
	)
	Set-Location $LocalPath
	$discards | ForEach-Object {
	if($Direction -eq "MainIntoDev") {
	$sourcePath = $BaseMainBranch + $_
	$targetPath = $BaseDevBranch + $_
	}
	else {
	$sourcePath = $BaseDevBranch + $_
	$targetPath = $BaseMainBranch + $_
	}
	if($WhatIfPreference -eq $false) {
	Write-Verbose "Discarding $sourcePath into $targetPath"
	& tf merge /discard $sourcePath $targetPath
	}
	else {
	Write-Host "WhatIf: Discarding $sourcePath into $targetPath"
	}
	}
	}

view raw

MergeDiscard.ps1

hosted with ❤ by GitHub

This scrip supports both -Verbose and -WhatIf commandlet bindings and it’s written in a way that the only thing you actually need to maintain is the array of strings of the sub paths of stuff you don’t want to merge.

So, unlike the feedback I got, this is definitely not rocket science to maintain and it’s a good starting foundation to deal with merges.

You run the script before you actually do the merge, if you didn’t have it right you can simply undo pending changes, tweak the script, and check again. When you’re happy with the discards you perform the merge and then check in.

Applying configuration transforms outside Visual Studio

Recently I was putting a NuGet package together and one of the things the package needs to do is change the configuration file when added to a project.

NuGet 2.7 and forward supports the use of XDT transformation files in the form of .install.xdt and .uninstall.xdt that run during package install and uninstall respectively.

However, all the files that are part of a NuGet package composition are usually outside projects in Visual Studio, so I needed a way to test that these transformations actually worked on a live config file.

To achieve this I wrote a PowerShell script that references the Microsoft.Web.XmlTransform.dll assembly that executes the transformation, applies the transformation and writes to an out.xml file in the same location as the script. The path to the XmlTransform dll is hardcoded into the script, but it is the path on a default installation of Visual Studio 2015.

	param
	(
	[parameter(Mandatory=$true)]
	[string]
	$Xml,
	[parameter(Mandatory=$true)]
	[string]
	$Xdt
	)
	if (!(Test-Path -path $Xml -PathType Leaf))
	{
	throw "XML File not found. $Xml"
	}
	if (!(Test-Path -path $Xdt -PathType Leaf))
	{
	throw "XDT File not found. $Xdt"
	}
	Add-Type -LiteralPath "C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\Extensions\spzpqgng.yjx\Microsoft.Web.XmlTransform.dll"
	$xmldoc = New-Object Microsoft.Web.XmlTransform.XmlTransformableDocument
	$xmldoc.PreserveWhitespace = $true
	$xmldoc.Load($Xml)
	$transf = New-Object Microsoft.Web.XmlTransform.XmlTransformation($xdt)
	if ($transf.Apply($xmldoc) -eq $false)
	{
	throw "Transformation failed."
	}
	$xmldoc.Save("$PSScriptRoot\out.xml")

view raw

Apply-XdtTransform.ps1

hosted with ❤ by GitHub

New open source project – DevOpsFlex

It’s been a while since I last posted. On November last year I decided to take on a project in pure Waterfall, and from a developer point of view, the problem of Waterfall is because of the nature of the cycles, you never really build something cool or good, you’re always trying to deliver instead of building. So you get stuck into this delivery cycle and you’re not really accomplishing anything good and worth writing for.

That is behind now and I’m now back to Agile working in a DevOps team doing automation for a .Net programme. The development work I will be doing will be fully open sourced.
So far I have been working on a single TFS build workflow activity that scales Azure VMs up and down depending on what you want to do with it. For us, we want to scale down development environments during the night and during the weekend, but not completely shut them down so that we can still do continuous deployments during nightly builds. Reducing the VMs down to A1’s, or even A0’s will save a lot of money as environments ramp up during the development cycle.

The home for these TFS build activities is:
https://github.com/sfa-gov-uk/devopsflex

And they are already published to NuGet:
https://www.nuget.org/packages/DevOpsFlex.Activities/

I have a couple more things I want to do with this activity:

• Add a nice WPF designer to the activity.
• Add the ability to shutdown and start VMs instead of up scaling and down scaling.
• Add the ability to wait for the VMs to be back up before you exit the activity execution cycle. This allows developers to track the TFS build for when the environment is back up fully functional and if they are tracking TFS builds they will get notifications for it.