KPI’s, Complexity and Resilience

Purpose

KPI’s, in theory, should monitor the organisations ‘critical success factors’ which should be linked to the organisations overall strategy. 

KPI’s may be used to:-

1.       Monitor internal performance between business units

2.       Monitor the effect of a continuous improvement programme

3.       Benchmark the organisation against other similar organisations

4.       Allow external organisations, such as customers, to benchmark the organisation against similar organisations.

5.       KPI’s are often used as part of a ‘motivational’ (either reward or punishment) system with a view to the improvement of personal performance.

Background

Origins in Accounting

KPI’s in their simplest sense are not new.  KPI’s in various forms have been used in accounting and include long established indicators like stock, debtors, creditors, profit etc.

Key or Not

The prefix ‘key’ is supposed to infer the significance or importance of the KPI to the organisation as a whole.  They should link to the organisations ‘critical success factors’.   This suggests there should be a limited number of indicators otherwise the truly key ones will be lost in the ‘noise’.

Balanced Scorecard

Single dimensional KPI’s, e.g. all financial ones, drive an organisation forward in one dimensional manner and ignore other parameters essential to the organisation such as long term sustainability, reliability, safety, ethics, morals etc.   To deal with this issue the balanced scorecard concept was developed to build a bounded structure within which the organisation should operate.

The flaw with balanced scorecards is one of the reaction time and predictability of KPI’s.  KPI’s related to operating costs for example are to some degree predictable and their reaction can be immediate.   Other indicators are more ‘emergent’ such as accidents and incidents are not predictable and operate in a longer time frame.  Thus a balanced scorecard reviewed annually may show reductions in operating costs without any change to the underlying safety of the organisation as the operation structures removed may only increase the chance of an incident.  That incident may happen outside the current year leaving an impression that the scorecard is still ‘balanced’.  This is called ‘asymmetric feedback’.

Leading and Lagging Indicators

Some indicators are indicators of an outcome ie are lagging and some are leading.  Leading indicators give an indication that a parameter is changing before a serious outcome results.

Issues

Reluctance to Simplify

A cornerstone of ‘high reliability organisations’ (HRO’s) is the ‘reluctance to simplify’.  High reliability organisations encourages problems and issues to be considered in the language relevant to the discipline in which the issue exists rather than for it to be ‘simplified’ into management speak, bullet points and key indicators.  One often quoted example of this relates to the space shuttle incidents.  It is believed that the use of bullet points in slides simplified the problem to such an extent that key decisions were made without the full facts being understood.  

KPI’s can have the effect of simplifying operating issues and drawing attention to derivative issues instead of the core one and may attract uninformed involvement in finding solutions.  

Comparison across dissimilar organisations in a vertically integrated corporation

One issue with the use of KPI’s in large diverse organisations is that in an attempt to extend the span of control in a vertically integrated organisation key understanding is lost.  Key performance indicators may be applied across a range of diverse organisations that are not appropriate to all of those organisations.  When aggregated and reviewed brad brush decisions may be made on where to apply investment, where to cut costs and to apply inappropriate continuous improvement. 

A KPI for example that compares deferred maintenance across shipping, refining and exploring business for example hides the complexity of the underlying issues.

Timescale and short termism

KPI’s are generally reviewed on a short term monthly, quarterly, annual basis.  This tends to drive a short term view of the business by those who may influence the KPI’s especially as targets are made more ‘challenging’. 

Cause and effect

The process of measuring and monitoring a KPI can change the relationship that KPI has with the underlying ‘reality’.   A KPI may have been created based on a historic correlation between that KPI and the operation of the organisation. The underlying correlation may be robust but ‘servicing’ the KPI may become the ‘end’ rather than ‘a means to an end’. 

For example a correlation was identified between the time to close safety actions and good safety performance.   The creation of a KPI that tracks action closure will improve the action closure rates but may not improve safety to any great extent as the closure of action becomes the objective not the good safety performance. The result of this is the KPI delinks from the performance improvement required.  The action closure performance was an outcome of a good safety culture not a driver.

Motivation

KPI’s are often linked to bonus type system.  When so linked the effect of the KPI can become more ‘active’ on the organisation.  Cases of reward driven cost cutting or profit/bonus driven criminal behaviour are widely known.  To a lesser extent punishment can lead to distorted or even criminal behaviour (e.g. suppressing information etc).  The figure below shows the various phases that resulting from increasing reward and the increasing threat of punishment.

the effects of reward and punishment

Moral Hazard

The purpose of KPI’s is to measure performance.  The achievement of that performance will in general have some impact on the performance assessment, reward or punishment of key managers in the organisation.  In some cases this can result in those managers taking action that will ‘improve’ the KPI while at the same time increasing risk.  These actions may include:-

Misreporting

Where the data is within the control of the manager the manager may take the opportunity to adjust the figures.  This could include for example ‘case management ‘of accidents to causes them to be reduced to a lower category.

Short term fixes

Taking physical actions that cause the KPI’s to change , for example over a quarter or year end.  This is common where budgets and stocks are involved.  

Inappropriate operational changes

The increased focus on KPI’s along with motivational (bonus/punishment) links can create a new reality in which managers take inappropriate action, against their underlying professional training or morals and ethics, which increase the risk to the organisation.   Examples of this may be cutting back on essential maintenance to meet an operating cost KPI in the short term

Knowing when to Stop

Few KPI’s are enduring and the point at which a KPI should be ‘retired’.   A KPI has generally reached the limits of its usefulness when the effort and resource required to achieve a change becomes excessive.   I.e. when the cost/benefit curve becomes non linear or the law of diminishing returns takes over.   The KPI can be retained to ensure no ‘backsliding’ but should not be actively monitored and controlled.

Validation

KPI’s linked to reward and punishment are prone to distortion and some form of validation is required

Good KPI’s

There are many good and valuable KPI’s in the shipping industry.  These can include such things as availability, utilisation, operating costs, freight rates etc.  For these to be valuable most or all of the following requirements should be met.

Direct links

The KPI should be directly linked to the ‘critical success factor’ in a manner that ensures that a positive or negative change in the KPI is directly reflected in the critical success factor and flows to the organisations overall performance.   An example would be utilisation of a vessel where the percentage utilisation links directly to the number of earning days and hence its profitability.

Objective measurement

The KPI needs to be measured objectively and without distortion based on recognisable and established information.   Utilisation, for example, should be based on on-hire and off-hire  times as per the official log book.

Cost/Effort to produce

The effort required to produce and track the KPI must be consistent with the benefit produced by it.   If considerable then the delay in producing the KPI will demotivate

Linear

Changes in the KPI should follow a linear profile so that an instinctive feel for the data exists

Predictable and linear cost benefit

There needs to be a clear understanding of the incremental cost of improvement of the KPI.  In particular it needs to be clear when the cost becomes ‘non linear’.  Ie the law of diminishing returns comes into effect.   For example an ‘availability’ KPI is unlikely to reach 100% without significant and impractical investment.   The reaction of the KPI to changes needs to be predictable allowing sensible investment decisions to be made.

Drives Right Behaviours

A good KPI drives the correct behaviours.  It challenges people to look afresh at the workings of the organisation and to take action that will improve its performance without creating moral hazard and unrecognised risks.

Bad KPI’s

Many KPI’s have a negative overall effect while apparently generating improvement in the measured index.  These KPI’s have some or all of the following characteristics:-

Abstract

A KPI which is abstracted from base data or which has an uncertain correlation will result in actions directed to correct the KPI.   Many abstract indicators are leading indicators.   An example may be ‘training days’.  Training days may be seen as a leading indicator where the training is specific and targeted.  Where it is a general measure it will become abstract

Doubtful measurement

Where the measurement of a KPI lacks integrity and is capable of manipulation the KPI can affect performance.

Effort to produce

Where the effort required to produce the data exceeds any value that may be generated the tracking and management of the KPI can reduce performance.

Delayed response

Where the response of the KPI is delayed longer than the normal performance monitoring period the value of the KPI will be limited and conclusions drawn from the behaviour of the KPI will be flawed.

Drives wrong behaviours

A KPI which drives the wrong behaviours such as rampant cost cutting will affect business performance.  In the extreme the behaviours generated can be amoral or illegal.  An example being the use of enemy body count as an indicator during the Vietnam war is believed to have led to civilian deaths.

KPI’s drive complexity

KPI’s impact ship operations both positively and negatively and directly and indirectly.

‘Good KPI’s’ provide clarity and focus on the organisations goals and by reinforcing a mental model of what is important to the organisation.

They facilitate improvement in performance and when properly balanced helps the organisation improve without creating unnecessary risk.

Bad KPI’s create complexity as follows:-

Directly

By creating a data load and effort to manage the KPI that does not directly benefit the organisation.

Distorting Goals

Bad KPI’s can take the focus off what is important to the organisation and focus people on short term and abstract goals.

Creating workload

Bad KPI’s can result in workload that does not improve the organisation as a whole but which may result in risk to the organisation

KPI’s and resilience

Good KPI’s aid downward resilience by reinforcing the goals of the organisation.   This means that in an abnormal or emergency operating mode goal conflicts are avoided.  People need to be aware that the KPI’s do not cove everything and they are still expected to make ‘professional ‘decisions.

Bad KPI’s damage resilience by distorting goals, changing priorities, increasing workload and by creating a belief that by maintaining the KPI’s alone the operation will be flawless.