PMV: A Cautionary Tale of Targets
IN 2017, Queensland police department successfully lowered crime rates by asking victims to stop reporting crimes.
The shocking discovery was made by the Queensland Audit Office, which discovered that 22 percent of all Police Service reports from 2011 to 2016 were incomplete, inaccurate or both.
Queensland police were fudging their reports in response to mounting pressure from the government to get crime rates down. They adopted various tactics: crime data was not logged properly; ‘unsolved’ cases were recorded as ‘unfounded’; others were labelled as ‘solved – bar to prosecution’ and offices asked victims to withdraw accusations.
This is not the first nor last example of measures, designed to improve outcomes, making matters worse. Somewhat closer to home, in a speech given by Jeremy Hunt, then Secretary of State for Health, in August 2014, the Department for Health acknowledged that their focus on waiting time targets for patients had had unintended consequences:
When the NHS started measuring performance against the [18 week waiting time target] in 2007, something perverse happened. If faced with a choice between treating a patient who had missed the [waiting time] target or someone who had not yet reached it, the incentive was to treat the person who had not yet missed the target rather than someone who had – because that would help the performance statistics, whereas dealing with the long waiter would not.
So a target intended to do the right thing ended up incentivising precisely the wrong thing.
Almost 40 years prior, social scientist, Donald T Campbell had foretold the dangers of applying metrics and the response they can invoke:
The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.[1]
At a similar time, former Chief Economist of the Bank of England, Charles Goodhart established a maxim about this conundrum – put simply:
When a measure becomes a target it ceases to be a good measure. – Goodhart’s Law
In recent years, the construction industry has sought to develop a more consistent approach to measuring performance. As the prevalence of modern methods of construction (MMC) has begun to grow, so with it has the term ‘Pre-Manufactured Value’ (PMV). Initially referenced within the Farmer Report, PMV is the proportion of offsite construction as a percentage of the total measured works on a project.
Image courtesy of HTA Design
This measure has been embraced by the government as a indicator of the extent to which public sector has delivered against the ‘presumption in favour of offsite’. The 2018 Industrial Strategy Construction Sector Deal explicitly commits to
It represents a strong proxy for a shift in industry behaviours and the realisation of improved asset delivery – however it remains a proxy nonetheless. Designers of policy and interventions should be cautious to recognise the dangers of defining targets, without situational context or consideration of other criteria[2]. Driving PMV as a primary focus could have unintended consequences, not least the potential to compromise design quality. For example, the introduction of passive ventilation as part of a net zero carbon solution may, by virtue of reducing mechanical ventilation and shifting relative cost proportions, result in a decrease in PMV. The view through a telescope has its limits.
Donald Campbell promoted that we should protect ourselves against such unintended consequences by:-
-
Using multiple measures (quantitative and qualitative) to assure that numerical data does become the only lens through which we judge a project’s effectiveness or impact
-
Identifying ways to uncover data distortion and/or misuse and institutionalise them.
While most want to hear a single number to summarise an analysis, policymakers should heed this sage advice and remember that a suite of interventions, reporting multiple measures, is usually required to achieve sustainable change.
PMV, if measured during design, can represent a leading indicator to enhanced productivity – in turn rooted in delivering better quality of life. For it to be truly effective however it must remain part of a balanced scored, rather than a primary or exclusive measure. The greater the consequence we attach to evaluations, the more we need to diversify the indicators.
Goodhart’s Law reminds us that setting targets can change the way in which people work towards the goals these targets are intended to help us reach. Undoubtedly the pressure to realise higher percentages of PMV will invoke some parties to focus as closely on the definition of pre-manufacture, as they will the principles of design standardisation to deliver better quality. With business case viability influenced by minimum thresholds of PMV, the temptation to blur interpretation will exist. A clear system of classification will be required to ensure that creative license is not applied as the least path of resistance to raising percentages.
PMV was developed as part of a holistic review of the UK construction labour model. It should rightfully remain as one of several key measure that informs a broad perspective towards delivering better social, environmental and economic outcomes
Targeted in isolation however, our intentions to do the right thing may otherwise predictably incentivise the wrong thing.
[1] Donald T Campbell ‘Assessing the impact of planned social change’. Evolution and Programme Planning, 1979
[2] Oliver, Lorenc, Tinkler and Bonnel ‘Understanding the unintended consequences of public health policies: the views of policymakers and evaluators’, 2019
