Monday, 7 May 2012

Stuff that works - Performance Driven Development


If you have read any of my other posts I will admit they have been a "tad negative", but this is with good reason. I feel the software industry needs a good kick up the backside if it's ever to become a real profession.

To this aim, here is a positive post, its a description of a development philosophy that I feel works for nearly all enterprise level applications. There are no frameworks, no toolkits, no fancy technologies, no documents, its simply an idea called "Performance Driven Development", implement it as you wish.

So what is Performance Driven Development all about?
Well its the missing link between what you think is going on and what is actually going on.
Things like BDD, TDD and continuous integration have their place but they dont address how the system really is performing during development and production.

You can have 100% code coverage and all the continuous integration you like but that is no guarantee that the system works as expected. In fact, I feel mystical green builds give developers an unwarranted feeling of confidence in the system they are developing.

"The build's green... it must work!" - Failed project #83571

This is where Performance Driven Development comes in, its goal is to help developers show everyone the heart and soul of what they are actually creating. It lets developers show off the fancy internal systems they have built, and as a handy side effect it also forces them to have a good hard look in the mirror at their code and themselves. When other people can easily see and judge how your creation is really working it forces the developer to produce better quality. Pride is a big motivator.

Key points:
- PDD is a common sense iterative development approach
- Developers have to create Key performance Indicator dashboards. (KPIs)
- These KPIs are created and updated continually throughout development. Not as an afterthought.
- These KPI dashboards are always available in realtime
- Its about delivery of real requirements with better quality and certainty.
- The same Performance information is available in both development and production systems.
- It helps both testers and developers build and support the system
- Its NOT about micro optimisation
- Its NOT about performance counters or WMI objects.


A real world example of using this technique is when building backend services.
These services could be hosted in IIS or be windows services serving requests and performing batch processing tasks. Pretty much all enterprise systems use them in some form or other.

The problem we have is these services tend to be black boxes with all internals hidden from view.
So how do we look into the soul of a service?

How do we ask questions like:
- How is the service running at the moment?
- What's the last 10 things it did? How long did they take?
- Are its caches working as expected?
- Is it running slow? What's the slowest operation?
- Are the inputs as expected?
- Whats are the details of the items in the cache? Are they as expected?
- Is cache size as expected? What was the cache size over the last 2hrs?
- Is database access speed as expected?
- Who are the top ten users?
- What are the top 10 widgets accessed today?

In the past many developers would add perfmons and WMI queries as an attempt to show information about whats going on inside a service. However in my experience these values are generally useless during development and not much better in production. They are used more for production alerts than general running of the service and these values are hidden away from everyone except system admin types.
In short WMI and perfmons are crap for development and limited in production.

What we need to do is create a window into the soul of this black box. A window that is accessible by developers, testers, support and system admins straight from their own machines (even in production). A simple way to achieve this is to create a performance KPI dashboard as a web page inside your service. This is simple with an IIS service, however for a windows service you need to create your own HTTP handler inside your existing service. This takes about a 100 lines of code or use a simple webserver something like nancyfx (nancyfx.org) or kayak (kayakhttp.com).

So now we have all the power of html to display performance data, using tables, links, css etc,  you can have drill down functionality and even show charts of KPIs using things such as google charts.(http://developers.google.com/chart/). There are lots of possibilities.

The goal is to allow access to these dashboards in realtime either in development or production via a simple http addresses. All the data is readonly and easily viewed with any browser.

i.e
http://23.23.23.23/kpi/widgetcache
http://23.23.23.23/kpi/processedcache.



Ok, so what's the big deal? A simple web dashboard is not exactly ground breaking.

This is where the Performance Driven Development KPI idea comes in. As you build your service you need to think about what would be useful to see on the dashboard to help developers and testers during development as well as support and sysadmins in production. You then build into the system ways to measure these values and add them to the dashboard as you go. The key point is you don't add them at the end of development as an after thought, instead you continually improve the KPI dashboards as you build the application.

The goal is too heavily use the dashboard as a debugging tool during development. You can have multiple dashboards broken up as you see fit. i.e Each cache could have its own dashboard.

For example say you have a cache that stores widgets that are loaded from a database. When the widget is loaded from the database and stored in the cache you simply wrap that widget in an object that not only stores the widget but in continually updates KPIs values such as:
- Time it took to load the widget from the database (milliseconds)
- Create time
- last accessed time
- total access count
- etc

The WidgetCache itself could have KPIs such as creation time, total widgets, last cache drop, total cache items added, deleted, etc. The cache manager also exposes readonly properties to things such as all cached widgets, top 10 accessed widgets etc. You create these as you see fit over time, continually improving them as you go along.

Then you simply give the widget dashboard access to these KPIs and then display the basic cache values as you see fit. You could have a grid displaying the top 10 accessed widgets showing columns with name, create time, access count , load time etc, a chart showing cache size over time. It's up to you to figure out what is useful.  To view the KPI dashboard you simply refreshes the web page to see latest values.

There will be a slight overhead calculating these values but once you start using this technique it lets you catch bugs well before you get to production. Things like low cache hits, empty caches and long cache load times for certain widgets will stick out like a sore thumb. This can point you in directions that help you find issues you never would have seen without the dashboard. For more detail you can add drill down links on a widget that display a new dashboard with full details of a particular widget in the cache.
The possibilites are large but keep it as simple as you can. These dashboards are only for internal use, they should not be accessed by real users.


PDD Dashboard example created for a financial modelling service  

The above example dashboard shows the soul of a very complex modelling cache running on a large enterprise system. You can drill down into each instrument to show a detailed google chart showing the modelled values in realtime. The great thing is both testers and developers are using this screen during development and production.



Creating these simple PDD dashboards doesn't really add to development time, in fact it reduces overall development time because it quickly exposes errors during development, reducing code rewrites and hard to find production bugs. It also simplifies handing over code to other developers, as it gives you a visual display of the inner workings of complex code and structures.

Simply put Performance Drive Development produces better quality code in a shorter timeframe by empowering developers to show off their skills for all to see.

Give it a go!


GCS Wiz




Sunday, 9 October 2011

Cargo Cult Over Engineering - Pointless Generic Design

Try to open your mind...

Look back to a time long ago...

A time where there are no computers, no phones, no modern technologies....
Now look back ever further, beyond ice ages, beyond the Earth itself....
Back to a time even before our Sun existed, keep going all the way back, further and further to a point before even time itself existed, a point where our Universe was NULL.
There is nothing not even darkness.

All is still...

The omni gods are tasked to come up with a design to recreate our entire universe. However they don't stop there, they work away and come up with a brilliant generic design so they can create any type of universe.

After a little while one omni god asks the question "Do we really need to allow for all these other universes? Surely we just want to recreate the one we had?"

The other omni gods bare down upon him. "How dare you question the order of generic design!" they say, he is ridiculed, excluded from further discussions, labelled as a fool and banished.
A murmur begins to ring around the room.
The murmur slowly turns into mantra.

"Universe<T>..."
"Universe<T>..."
"Universe<T>..."


The omni gods think long and hard about all the things all possible universes have in common.
The ultimate glorious generic solution awaits...They can almost touch it... excitement is building.

Universe<OursWithUsAndTheEarthInIt>

Universe<AllGreenSunsPlusInfiniteMuggleWumps>
Universe<SlightlyToTheLeftWithHammers>
Universe<BendingLightSometimes>
Universe<BlackHoleHideout>
Universe<NULL>
Universe<FireIsBlackAndNoMarshmellows>
...

The list goes on and on and the High Priests have to bend the system to handle more and more possibilities.The mantra continues:

"Universe<T>..."
"Universe<T>..."
"Universe<T>..."


Universe<RockPillowsAndSundials>
Universe<NoGravityOrJugglers>
Universe<PlanetsAreSquareSometimes>



"Universe<T>..."
"Universe<T>..."
"Universe<T>..."


Universe<AntiUniverse>
Universe<DarkMatterAndPigsOnlyOnTuesdays>
Universe<AnyoneCanCode>



Time passes as the omni gods try to create a generic solution to handle all the Universes they can imagine. There is a lot to think about and they have barely started their brillant solution when...

KAPOW!

From nowhere our Universe explodes into existence!
Once again our universe lives and breathes just as we remember it.

The omni gods are in shock. What is this blasphemy? Our perfect solution is not ready.

To their horror the banished god has gone off and made exactly what was required.
He simply recreated our universe.

The omni gods shout scorn at him "What about our brilliant generic solution?"
This banished voice doesn't answer, he has finished his work for the day and has gone to the pub.

He will never be a member of the cargo cult that is "Pointless Generic Design".


GCS Wiz




Tuesday, 9 August 2011

Cargo Cult Epic Fail - WPF

WPF,
some say it was the chosen one, the answer to the prays of all who dreamt of a new world. A world where designers and coders are living in harmony, both reading from the same magical text, a text called XAML.

Quickly a silent mantra began to consume us all "XAML...XAML...XAML..". As one we began a holy pilgrimage to the promised land. In this land magical dreamscape applications beckoned, it all seemed so simple, all we needed to do was climb to the WPF summit for enlightenment and we would be free.

The mantra continued:
"All praise the XAML, all praise the the WPF"
"All praise the XAML, all praise the the WPF"


The high priests prayed that C++ Applications would finally be banished, Winforms retired and Java destroyed.

Years passed in the blink of an eye, new XAML tomes came and went, new features, new versions.
The high priests noticed the mantra was fading,  a new mantra was needed or all is lost.

"MVVM...MVVM...MVVM" began to ring out over the confused masses.
"MVVM will solve your frustrations, your confusions, your pain. WPF will live forever through its son MVVM"

Once again we climbed to the summit for enlightenment, surely this time we would be free. Surely?

Alas as much as we tried we could never make those magical dreamscape applications. They always lacked something, they seemed second rate. They never 'felt' as good as they should, but we persevered.

"KAPOW!!!"

From nowhere a crack in the sky suddenly appeared. The blinding light made us turn our heads away, but as we looked back we stared in astonishment.

"IOS apps, what are these? They are like dreamscapes, they are fluid and fast. They are fun and simple"

We asked our priests "How can this be? How can these beautiful IOS things be created without using XAML, without using MVVM, without using WPF??. This is surely a trick!"

The priests have no answers. They have fled, and left us to fend for ourselves.
This is no less than we deserve because we have become card carrying members of  the cargo cult that is WPF.


GCS Wiz




Thursday, 20 January 2011

Cargo Cult Complexity -> Magical Framework Madness

An observation I have made from my long and ultimately successful journey through the development industry is that very few people make quality software. In fact hardly anyone does.

The main reason is the Complexity of the task at hand. Complexity is the double edged sword that makes the development industry what it is, and if you can tame this wild beast great rewards may come your way. If a developer can reorder seemingly infinite chaos into straight forward patterns you can reach the upper echelons of this mystical and misunderstood art. However if you are not a master of this art then it seems you must scour google to find a high preist who has created a miracle framework to solve your problems before all hope is lost.

Unfortunately even if a developers pattern matching skills are high there is no guarantee that these skills will translate into business value. Many developers just look for a magical pattern they have read about, add a couple frameworks and a tool or two from 'Top of the pops' and start churning code, the actual requirements are secondary.

This is not to say the majority of developers are not clever well educated people, in fact most of them are.
The problem is many of these same developers live in a parallel world of frameworks, acronyms and methodologies. They have lost touch of the real requirements of business and spend most of their time trying to shoehorn fancy generic frameworks, tools and methodologies into their ideal solution. They care little for what is really needed, instead they concentrate on building something that will allow them to stand up amongst their peers and say we used X,Y, and Z. Aren't we clever.

This is cargo cult software development in full flight.

The amount of applications that are simply not fit for purpose because  "God has deemed we must use this  framework otherwise we will be shunned from society" is astounding. Developers naively believe the framework will solve any issue no matter how complex or bizarre. They also cast non believers into the cold, labelling them as 'out of touch' thus banishing them from all discussions.

When the juggernaut of "Slow performance and high resource use" arrives the stunned masses once again search the framework tomes to find hidden answers. They must not think for themselves; the framework knows all.

As you read these words, you are probably thinking to yourself:

"I know what he's talking about ... but I'm not one of these developers".

Its interesting how developers never think that they themselves are the culprits. I put it to you reader that you are guilty as charged.

However this is your chance to break away from the crowd and think for yourself.
A real opportunity to overcome the flawed belief that the framework somehow has a magical unlimited power. Freedom is at hand, a new horizon awaits.

You don't need to jump straight into a crazy 'not invented here' policy, but you need to know why you are actually using a particular tool or framework. Is it to help you succeed? Or is it simply because you are a card carrying member of the "cargo cult that is software development".

Think about it...


GCS Wiz




Friday, 19 November 2010

Cargo Cult Control -> The Microsoft Marketing Machine

As I have been blessed with a microsoft tagged comment to my previous post (http://cargocultsoftwaredevelopment.blogspot.com/2010/10/cargo-cult-mvvm-death-knell-of-wpf.html) I feel I should respond to Pete Browns carefully worded critique. I welcome opinions other than mine and would like to encourage anyone who feels moved enough to leave a comment on these pages to do so. (Just make sure you read the bi-line of my blog first so you know what to expect)

Pete Brown - Microsoft Community PM WPF/Silverlight wrote...
"The title of this is a bit misleading. As a reader, I thought this would be about MVVM with perhaps some good alternatives presented, but really your issue is XAML and WPF itself."


This seems to be a common theme throughout the industry. It is expected that if you write about development topics you must present it akin to a research paper. You need examples, facts, figures and a clearly presented alternatives. 


It irks me that this is the case and is simply another example of cargo cult behaviour. Readers eager to jump to a new 'improved' tech religion at the press of a key. All along hoping they can achieve a higher level of enlightenment and wow their peers and perhaps one day reach to the skies themselves.


The alternatives I present are not in the words I write, but instead in the freedom they aspire too. The industry needs to open its mind to the possibility that maybe they are all caught up in an endless spiral of hype and company positioning. 


Ask yourself this question:


"What is the point of software development?"      
(before you answer - Try and word it as if you are talking to an isolated tribe in the south pacific) 


Pete continues...
"FTR, I really do like the XAML model. That's why I joined Microsoft.
In WPF and Silverlight development, you should use *some* architectural pattern. I don't personally care what it is, but IMHO it should leverage binding. Many of the features in the platforms are easier to use with binding."


Pete obviously likes his XAML, and points out I seem more unhappy with WPF and XAML than MVVM. The simple reason for this is MVVM is the bastard son of XAML, without XAML it wouldn't even exist. So I am just going to the source of the problem.

As for the statement "you should use *some* architectural pattern". 
There is an alternative anti pattern I use in all my apps:

 "The Common Sense Anti Pattern" or TCSAP as it is commonly known.

Pete tells it like it is...
"MVVM happens to be a leading pattern. However, many people throw a lot of extra complexity into the definition of MVVM, burdening it with things that really are optional and confusing the community in the process.
...
Pete Brown
Microsoft Community PM WPF/Silverlight"

So why is MVVM a leading pattern if it confuses the community? Its not because it just happened to be in the bar one day when the talent scouts popped in for a beer and decided to give MVVM it's big break. 

Its all down to the WPF marketing machine needing a hook to feed the fanboy culture. This hook then drives the cargo cult in the direction Microsoft  requires. A purely invented reason to get these fan boys excited and start talking about WPF again. 

Any negative press that is hurting the brand (such as this blog) must be "managed" or the sky will fall in.
In the end Its a business, so winning is simply about getting more people to use this set of tools over another set, the goal of delivering better software is a minor concern. 
Market share is king.


Despite all this, they all know good developers will always deliver good software no matter what tools, patterns or methodologies try to get in the way. This is because good developers use anti patterns like TCSAP extensively.


Finally for those who are waiting for the answer to my question above:

"It's to make money, plain and simple."



GCS Wiz




Tuesday, 26 October 2010

Cargo Cult MVVM => The death knell of WPF

The MVVM Cargo Cult is certainly a strong and powerful one.  It seems to be worshipped by every two bit WPF/Silverlight developer on the planet. Developer job sites list this as one of the multitude of pointless acronyms you must know, and if you are a non believer then you are a philistine who is not worthy of their time.

Millions of bloggers give thanks to their software gods by offering tips on how to use this magical beast, hoping that these same gods will look kindly upon them and deliver the software they crave. The less clever among them heap scorn on the gods and wait for a mystical MVVM tome that will explain their every confusion and answer all questions on this mystical pattern.

So lets get back to reality...
Over the last 17 years I have developed a truckload of GUI applications using a wide range of languages from assembler and C++ through to Winforms and WPF. I have also created several complicated multi threaded silverlight and WPF Apps using MVVM, I was pragmatic and used it only when it made sense and when it is worthy of my time. I too was tempted by its claims, but not any more.

It is my considered opinion that if MVVM could be erased from space and time we would all be producing better applications and in general be a lot happier.

The MVVM pattern was designed to try and accommodate the flawed XAML concept which was naively put forward by Microsoft all those years ago. They had big plans to replace HTML with this stuff, big plans which are now a distant memory.

WPF is dead!
Its been hanging on for grim death for years but MVVM has ripped open its chest and let the world see the true horror that is hidden within. It has laid bare the evil black magic that must be carved into XAML to make MVVM work. This pattern is the final dagger in WPF's barely beating heart.

MVVM asks you to create untyped XML files that contain free text view model names, property links, black magic bindings, mystical triggers, obscure commands, baffling templates and untraceable style overrides.
Then in the 'real' code you create notification properties with untyped string names, cross your fingers observable collections, even more magical routed events and commands and then pray to the gods you can make it all link together.

As I write this blasphemy I can hear the impassioned cries of  'Testability!' and 'Separation of concerns!'. Wondrous tails of a soon to be future where designers and developers live in a perfect world of magical collaboration and mutual respect. Its within our grasp they cry!

Sorry guys, you have all been led down the garden path on this one.
MVVM and WPF basically produces unmaintainable code as soon as the application gets even slightly complex. XAML is the problem, all these binding features MVVM forces you to use are at best an Alpha in WPF, and because WPF is such a sprawling mess of legacy 'features' it wont be fixed ever.

Trying to fix bugs in someone else's MVVM screen is nigh on impossible because its so hard to decipher the flow of control. When you make changes most of the time you only get runtime errors instead of compile time. Debugging these runtime errors is painful as they are generally hidden in the black magic of bindings or triggers or some other crazy abstraction in XAML.

MVVM testability claims are dubious at best. If you breakdown your frontend code correctly you can test it pretty well anyway, you dont need MVVM for that.

Also the mantra that  XAML lets designers design and developers code is flawed. It sounds great but in practice any slightly complicated screen basically needs a developer to make such complicated XAML that its sticky taped together just to work, and as soon as a designer tries to load it into blend or edit manually. Kaboom!  

Finally, MVVM produces terrible performance. It is nearly impossible to see where the bottle necks are as all rendering is done via black magic bindings and triggers. Trying to get any WPF application running smoothly is a struggle but using MVVM  makes it even worse. As have I often said during WPF development, 'My old C64 scrolled smoother than this and it only had 38k of available memory'

If you insist on still using this pattern remember this.
MVVM is a sharp tool, be very careful or you will cut yourself.


GCS Wiz




Monday, 18 October 2010

JobTitle<T> - Software development is not a real profession

Lets jump straight to it. Software development is a not a real profession. I should know, I have worked in it for nearly 17 years. I have seen technology continually change over those years but no where near as much as my job title. Lets see how this professional identity crisis unfolded:

I'm a new dude..
- Graduate programmer
- Junior programmer

programmer is so 1980 lets get a new word..

- Developer
- Senior Developer

hangon where now for developers??? We build stuff sort of...

- Software engineer
- Senior Software Engineer

hmm software engineer sounds a bit like an engineer...thats not what i do. but i'm better than the other guys

- team leader
- team manager
- senior consultant?

but im still developing, i come up with designs, I'm not a pen pusher??

- Architect
- senior Architect
- CTO
- IT manager

Hmm I may need to wear a suit...

- Just call me God

Lets clear the air, there is a small percentage of people in the software industry that are great at what they do. I would say I am one of them, and all I do really is play a game of fancy pattern matching, the more challenging the better, and on top of this I also get paid loads of money for the privilege.

I have completed many projects working with small to medium teams using too many technologies to name. From a junior greenhorn right through to the dude calling all the shots. I have been part of startups that have been bought, I have worked on big failures and big successes and now I am a highly paid contractor working in the city. All the while being part of this world we call the 'software industry'.

After 17 years I can sum up the feeling of the whole industry in two words:  'Identity Crisis'

The above made up hierarchy of names is a simple attempt by 'real' business people to understand developers and to try and make the development process work like a factory. A fair enough goal.

Unfortunately I've got some news for them, it ain't going to work and the sooner we all realise this the better.

I've also got some news for the developer guys as well, just because you stick 'Architect' at the bottom of your email doesn't make you a great developer. In fact in my experience most Architects just use that title so they can over engineer everything and feel clever.  More often than not they are missing the point on why we are all doing this stuff in the first place. Its all about making money and paying wages.

Take this example for the factory idea:
Imagine a universe where the 1980's C64 computer was where hardware development stopped. We all are still using C64 hardware in 2010.

Some weird virus infected all the computer engineers and they left the industry and became accountants.
No new hardware has been made, nothing, they just keep making C64 by the millions in pre existing factories

Software development techniques would move forward and I'm certain that by now (25 years later) we would have a factory type methodology for writing software on this hardware.

Clever people would no longer be needed for anything other than coming up with the idea in the first place. You just get a whole bunch of grunts to write the code following a tried and tested series of steps. Grunt factory workers in an assembly line pressing out code templates simply by pressing a big red button. Any project can be designed, estimated and built as long as it passes the 'laws of C64' and these laws are well known, just like the laws of physics when building a bridge. What is and is not possible is what you teach at college. All of a sudden its easy to make money using cheap labour. Business is happy.

Unfortunately in the universe i work in the 'laws of C64' where quickly discarded and a new set of laws invoked. Hmm they thought,  we need a new factory for this, and some new work methodologies quick, shit we better get the clever people back in too sort it out.

Wind forward 25 years and this new 'laws of the universe' cycle continues to happen every 6 months, new tools, new frameworks, new hardware, new bright ideas, new techniques. This has the effect of the factory being burnt to the ground every time and barely a new brick relaid before it is demolished again.

The only constant in all this is the top people can always make it work because they don't need a factory.

Until the computers can write the software themselves forget the factory idea. Just let it go.
Forget the bullshit job titles that we give ourselves and have a good hard look in the mirror.
Clever people make it all happen and the rest just get in the way. I don't class people like Scott Gu and co as the clever people, they are just marketing men. They all need to have a look in the mirror and ask 'What is the point of software development?'

When I look in the mirror I think to myself, "My degree says I'm a computer scientist".
I think i will go back to using that...


GCS Wiz