KPI’s, Complexity and Resilience
Purpose
KPI’s, in theory, should monitor the organisations ‘critical
success factors’ which should be linked to the organisations overall
strategy.
KPI’s may be used to:-
1.
Monitor internal performance between business
units
2.
Monitor the effect of a continuous improvement
programme
3.
Benchmark the organisation against other similar
organisations
4.
Allow external organisations, such as customers,
to benchmark the organisation against similar organisations.
5.
KPI’s are often used as part of a ‘motivational’
(either reward or punishment) system with a view to the improvement of personal
performance.
Background
Origins in Accounting
KPI’s in their simplest sense are not new. KPI’s in various forms have been used in
accounting and include long established indicators like stock, debtors,
creditors, profit etc.
Key or Not
The prefix ‘key’ is supposed to infer the significance or importance of the KPI to the organisation as a whole. They should link to the organisations ‘critical success factors’. This suggests there should be a limited number of indicators otherwise the truly key ones will be lost in the ‘noise’.
Balanced Scorecard
Single dimensional KPI’s, e.g. all financial ones, drive an organisation
forward in one dimensional manner and ignore other parameters essential to the
organisation such as long term sustainability, reliability, safety, ethics,
morals etc. To deal with this issue the
balanced scorecard concept was developed to build a bounded structure within
which the organisation should operate.
The flaw with balanced scorecards is one of the reaction
time and predictability of KPI’s. KPI’s
related to operating costs for example are to some degree predictable and their
reaction can be immediate. Other
indicators are more ‘emergent’ such as accidents and incidents are not
predictable and operate in a longer time frame.
Thus a balanced scorecard reviewed annually may show reductions in
operating costs without any change to the underlying safety of the organisation
as the operation structures removed may only increase the chance of an
incident. That incident may happen
outside the current year leaving an impression that the scorecard is still
‘balanced’. This is called ‘asymmetric
feedback’.
Leading and Lagging Indicators
Some indicators are indicators of an outcome ie are lagging
and some are leading. Leading indicators
give an indication that a parameter is changing before a serious outcome
results.
Issues
Reluctance to Simplify
A cornerstone of ‘high reliability organisations’ (HRO’s) is
the ‘reluctance to simplify’. High
reliability organisations encourages problems and issues to be considered in
the language relevant to the discipline in which the issue exists rather than
for it to be ‘simplified’ into management speak, bullet points and key
indicators. One often quoted example of
this relates to the space shuttle incidents.
It is believed that the use of bullet points in slides simplified the
problem to such an extent that key decisions were made without the full facts
being understood.
KPI’s can have the effect of simplifying operating issues
and drawing attention to derivative issues instead of the core one and may
attract uninformed involvement in finding solutions.
Comparison across dissimilar organisations in a vertically integrated
corporation
One issue with the use of KPI’s in large diverse
organisations is that in an attempt to extend the span of control in a
vertically integrated organisation key understanding is lost. Key performance indicators may be applied
across a range of diverse organisations that are not appropriate to all of
those organisations. When aggregated and
reviewed brad brush decisions may be made on where to apply investment, where
to cut costs and to apply inappropriate continuous improvement.
A KPI for example that compares deferred maintenance across
shipping, refining and exploring business for example hides the complexity of
the underlying issues.
Timescale and short termism
KPI’s are generally reviewed on a short term monthly,
quarterly, annual basis. This tends to
drive a short term view of the business by those who may influence the KPI’s
especially as targets are made more ‘challenging’.
Cause and effect
The process of measuring and monitoring a KPI can change the
relationship that KPI has with the underlying ‘reality’. A KPI may have been created based on a
historic correlation between that KPI and the operation of the organisation.
The underlying correlation may be robust but ‘servicing’ the KPI may become the
‘end’ rather than ‘a means to an end’.
For example a correlation was identified between the time to
close safety actions and good safety performance. The creation of a KPI that tracks action
closure will improve the action closure rates but may not improve safety to any
great extent as the closure of action becomes the objective not the good safety
performance. The result of this is the KPI delinks from the performance
improvement required. The action closure
performance was an outcome of a good safety culture not a driver.
Motivation
KPI’s are often linked to bonus type system. When so linked the effect of the KPI can
become more ‘active’ on the organisation.
Cases of reward driven cost cutting or profit/bonus driven criminal
behaviour are widely known. To a lesser
extent punishment can lead to distorted or even criminal behaviour (e.g.
suppressing information etc). The figure
below shows the various phases that resulting from increasing reward and the
increasing threat of punishment.
the effects of reward and
punishment
Moral Hazard
The purpose of KPI’s is to measure performance. The achievement of that performance will in
general have some impact on the performance assessment, reward or punishment of
key managers in the organisation. In
some cases this can result in those managers taking action that will ‘improve’
the KPI while at the same time increasing risk.
These actions may include:-
Misreporting
Where the data is within the control of the manager the
manager may take the opportunity to adjust the figures. This could include for example ‘case
management ‘of accidents to causes them to be reduced to a lower category.
Short term fixes
Taking physical actions that cause the KPI’s to change , for
example over a quarter or year end. This
is common where budgets and stocks are involved.
Inappropriate operational changes
The increased focus on KPI’s along with motivational
(bonus/punishment) links can create a new reality in which managers take
inappropriate action, against their underlying professional training or morals
and ethics, which increase the risk to the organisation. Examples of this may be cutting back on
essential maintenance to meet an operating cost KPI in the short term
Knowing when to Stop
Few KPI’s are enduring and the point at which a KPI should
be ‘retired’. A KPI has generally
reached the limits of its usefulness when the effort and resource required to
achieve a change becomes excessive.
I.e. when the cost/benefit curve becomes non linear or the law of
diminishing returns takes over. The KPI
can be retained to ensure no ‘backsliding’ but should not be actively monitored
and controlled.
Validation
KPI’s linked to reward and punishment are prone to
distortion and some form of validation is required
Good KPI’s
There are many good and valuable KPI’s in the shipping
industry. These can include such things
as availability, utilisation, operating costs, freight rates etc. For these to be valuable most or all of the
following requirements should be met.
Direct links
The KPI should be directly linked to the ‘critical success
factor’ in a manner that ensures that a positive or negative change in the KPI
is directly reflected in the critical success factor and flows to the
organisations overall performance. An
example would be utilisation of a vessel where the percentage utilisation links
directly to the number of earning days and hence its profitability.
Objective measurement
The KPI needs to be measured objectively and without
distortion based on recognisable and established information. Utilisation, for example, should be based on
on-hire and off-hire times as per the
official log book.
Cost/Effort to produce
The effort required to produce and track the KPI must be
consistent with the benefit produced by it.
If considerable then the delay in producing the KPI will demotivate
Linear
Changes in the KPI should follow a linear profile so that an
instinctive feel for the data exists
Predictable and linear cost benefit
There needs to be a clear understanding of the incremental
cost of improvement of the KPI. In
particular it needs to be clear when the cost becomes ‘non linear’. Ie the law of diminishing returns comes into
effect. For example an ‘availability’
KPI is unlikely to reach 100% without significant and impractical investment. The reaction of the KPI to changes needs to
be predictable allowing sensible investment decisions to be made.
Drives Right Behaviours
A good KPI drives the correct behaviours. It challenges people to look afresh at the
workings of the organisation and to take action that will improve its
performance without creating moral hazard and unrecognised risks.
Bad KPI’s
Many KPI’s have a negative overall effect while apparently
generating improvement in the measured index.
These KPI’s have some or all of the following characteristics:-
Abstract
A KPI which is abstracted from base data or which has an
uncertain correlation will result in actions directed to correct the KPI. Many abstract indicators are leading
indicators. An example may be ‘training
days’. Training days may be seen as a
leading indicator where the training is specific and targeted. Where it is a general measure it will become
abstract
Doubtful measurement
Where the measurement of a KPI lacks integrity and is
capable of manipulation the KPI can affect performance.
Effort to produce
Where the effort required to produce the data exceeds any
value that may be generated the tracking and management of the KPI can reduce
performance.
Delayed response
Where the response of the KPI is delayed longer than the
normal performance monitoring period the value of the KPI will be limited and
conclusions drawn from the behaviour of the KPI will be flawed.
Drives wrong behaviours
A KPI which drives the wrong behaviours such as rampant cost
cutting will affect business performance.
In the extreme the behaviours generated can be amoral or illegal. An example being the use of enemy body count
as an indicator during the Vietnam war is believed to have led to civilian
deaths.
KPI’s drive complexity
KPI’s impact ship operations both positively and negatively
and directly and indirectly.
‘Good KPI’s’ provide clarity and focus on the organisations
goals and by reinforcing a mental model of what is important to the
organisation.
They facilitate improvement in performance and when properly
balanced helps the organisation improve without creating unnecessary risk.
Bad KPI’s create complexity as follows:-
Directly
By creating a data load and effort to manage the KPI that
does not directly benefit the organisation.
Distorting Goals
Bad KPI’s can take the focus off what is important to the
organisation and focus people on short term and abstract goals.
Creating workload
Bad KPI’s can result in workload that does not improve the
organisation as a whole but which may result in risk to the organisation
KPI’s and resilience
Good KPI’s aid downward resilience by reinforcing the goals
of the organisation. This means that in
an abnormal or emergency operating mode goal conflicts are avoided. People need to be aware that the KPI’s do not
cove everything and they are still expected to make ‘professional ‘decisions.
Bad KPI’s damage resilience by distorting goals, changing
priorities, increasing workload and by creating a belief that by maintaining
the KPI’s alone the operation will be flawless.