Tutorial: Using Analytic Hierarchy Process to define score weights

Defining weights through expert judgment

• How expert can come up with values for the weight?
• How to reach agreement among expert on the suggested values?
• Is there an alternative to lenghty consensus building?

Often teams of expert suffer from non-aligned goals, power politics, group dynamics and lack of mutual understanding.

How to build consensus on complex decisions in order to raise sufficient confidence in decision outcomes?

Use case with the Scoring of vulnerabilities

Vulnerability is not observed, it’s a latent variable, an intellectual construction based on multiple criteria.

The challenge is:

• Define the criteria
• Provide a measurement for each criteria
• Define relative weights to be used for each criteria

The Analytic Hierarchy Process (AHP)

• Structured technique for organizing and analysing complex decisions, based on mathematics and psychology
• Comprehensive and rational framework to organise feeling, intuition & logic for structuring group decision making. Rather than prescribing a “correct” decision, the AHP helps decision makers find one that best suits their goal and their understanding of the problem
• Comparison is made to define priorities between 2 alternatives based on decision-maker’s feeling of priority dues to importance, preference and likelihood of influence. This breaks down complex decision into small judgement
• Developed in the 70’s by Thomas Saaty in Wharton Business School. Decision Making for Leaders: The Analytic Hierarchy Process for Decisions in a Complex World

Limitation of AHP

• Assumes a minimum level of judgement consistency among experts

• Limited number of criteria to minimise the numer of pairwise comparison estimation -

• Data should be available for all criteria

• Assumes that when new alternatives are added to a decision problem, the ranking of the old alternatives is not changing

How to? A step by step approach

1. Define vulnerability criteria
2. Build the expert consultation form
3. Generate the AHP file from the collected data
4. Run AHP algorythm
5. Review results
6. Apply the formula

1. Define vulnerability criteria

• List potential criteria that would contribute to vulnerability within the current context. Note that criteria can be grouped togehter using hierarchy.

• Establish treehold for criteria so that they can be formulated as simple binary questions

Criteria should be saved in a configuration file using the format here data/criteria.csv.

Criteria without hierarchy

In this case N*(N-1)/2 pairs to review: 5 criteria -> 10 comparisons

PS: 6 criteria -> 15 comparison, 7 criteria -> 24 comparisons, 8 criteria -> 28 comparisons, 9 criteria -> 36 comparisons

Criteria with hierarchy

In this case 6 pairs to review…

2. Build the expert consultation form

A form will allow to collect from expert priorities between criteria by making a series of judgments based on pairwise comparisons:

• Collect judgment for all pairwise comparison and each expert

• Ranking Scales for Criteria

Use the script 1-Build-xlsform.R to build a xlsform file based on criteria defined above.

Form

The form can be used within UNHCR Kobo server. Experts can be humanitarian case workers that are used to assess vulnerability. See an example here

3. Generate the AHP file from the collected data

Once the selected experts (aka. decision-makers) have filled the online form , data can be exported from UNHCR Kobo server in csv format.

Use the script 2-build-hierarchy.R to build a xlsform file to create the AHP file.

This create the file that format correctly the pairwise preferences of each decision-makers to run the next step.

4. Run AHP algorithm

• Calculation done using R statistical language.
• Synthesize these judgments to yield a set of overall priorities
• Check judgments consistency (consistency ratio)
• Compute weights for each expert
• Mathematical calculations to convert these judgments to priorities for each of the four criteria

Note: Some proprietary software options are also available but would require a Data Protection Impact Assessment before using them.

5. Review results

Knit the 3-final-report.Rmd to get the report. You can see an example here.

An interactive interface is available to interact with results.

• Lack of consistency is often observed

• If consistency ratio is above 0.1, then judgement are untrustworthy because they are too close to randomness -> exercise needs to be repeated or abandonned.

6. Apply the formula

• Calculate mean relative weight
• Apply the vulnerability formula
• Collect data on each criteria for all household
• Apply weight to each record for the different criteria to get the vulnerability level of each household

Feedback / Comments

Please fill in an issue .

Thanks to Christoph Glur for developping the original AHP package and answering questions.