Changes in the Main Analysis Method 

SquidMat's CAS³ predecessor, DECMAT¹, employs two different types of matrices, each using its own method of analysis. The table below shows each method's advantages and disadvantages as I see them.

Note that there are other, more technical criticisms of each method. But in a training environments like ours, I believe that we can safely ignore those and use decision matrices effectively to teach basic decision-support methodology. Also note that there are decent fixes for every one of the disadvantages expressed above—even the Relative Values Matrix's thoroughly flawed mandate to convert all measures to relative values whenever you have at least one relative-values measure in your problem. 

Of course, my solution was to write a program that does it the way I would do it. Hence, SquidMat. SquidMat uses weighted sums, which eliminates the Multiplication Matrix and my concerns with weighted products. 

Overcoming the disadvantages of the Relative Values Matrix

So that just leaves the Relative Values Matrix and its disadvantages. There are two.

First Disadvantage (mathematical): The weighted sums method cannot fairly mix apples & oranges, i.e., different units of measurement. Measures with the largest scale dominate the decision. That dominance should rightly come from a measure's importance (as an evaluation criterion, EC) as reflected by its weight.

The Fix. SquidMat converts all values to the same metric (Z scores) before applying weights.

Second Disadvantage (procedural): Converting all measures to rankings (relative values) unnecessarily degrades (or destroys) the interval information in all your measures.

The Fix: Use rankings when you have to, just don't change ALL measures to relative values. Then convert those measures to Z scores.

Using Z scores

This, I feel, has three advantages.

1. It avoids what is essentially our only mathematical concern with weighted sums (that it requires everything to be in the same metric). 

It's a basic tactic we use in psychometrics (psychological, educational, and employment testing) to standardize and compare scores from different tests. And since the decision matrix keys in on the intervals between COA values in any given criterion measure (EC column), we can rescale each column to the same metric. This gives us our ability to compare apples and oranges.

2. It's easy for students to understand and explain.

Anyone who has taken a semester of statistics knows what a Z score is. For each EC column, SquidMat computes a mean (M) and standard deviation (S). Using those values, the program converts each raw score ( X ) from that column into a Z using this formula:

Z = ( X - M ) / S.

SquidMat then multiplies each Z by its column weight and adds them across each COA row. 

3. It makes it very straightforward to convert an EC's values when it's LIB to MIB. Since Z scores are centered on zero, all you have to do to convert LIB values for an MIB analysis is change that column's Z scores' signs. That is, you simply multiply the Zs by -1. The inversion employed by DECMAT (1/X) gets very messy, resulting in values of 10-to-the-huge-power. (You know what I'm talking about. You've seen that note at the bottom right-hand corner of DECMAT.)

Using rankings²

SquidMat allows you to mix rankings (and other relative values) with "real" values. 

Real values have two critical properties: order and interval. The order property allows you to say that one numerical value is greater than another (as opposed to simply labeling things, like 1=Democrat, 2=Republican, 3=Trekkie, etc.). The interval property allows for differences in the size of the gap between one value and the next. 

For example, if A crosses the finish line 10 seconds ahead of B, but B crosses the finish line 1 second ahead of C, ranking them as A = 1, B = 2, and C = 3 preserves the order information but not the interval information. Reducing the data to ranks forces equal intervals between the values; there is no way to know that the gap between A and B was huge while that between B and C was relatively small.

As we know from doing DECMAT problems, there are some situations where we don't know the interval information, so we're forced to use rankings. My thinking, though, is that it's bad enough when we have to resort to rankings. Don't change all your measures to rankings to satisfy the Relative Values Matrix need for the same metric (which was the practice at CAS³ when I got there). The better solution is to use rankings when you have to and limit the damage caused by this loss of interval information by allowing all those ECs whose values are "real" to retain their superior interval properties. Don't make it worse by making them all rankings, for crying out loud.

² I wander back and forth between the terms rankings and relative values, because, when I arrived at CAS³, the practice was to feed the Relative Values Matrix exclusively rankings. That is, the two terms were being used synonymously. It depends on who you're asking, but some definitions of relative values include measures containing interval information. The "better" solution I offer next is an example.

An even better solution is to try and inject some interval information into the rankings. 

For example, I have a situation where I don't know three people's weights, but I know that A is heavier than B and B is heavier than C. Instead of just ranking them from 1 to 3, I could try and rate them on a scale of 1 to 10. I could make A a 10 and C a 1 and try to place B between 1 and 10. If B is very, very close to C while being far away from A, I'd rate B as a 2 or a 3. This technique restores some of the lost interval information. It's still a flawed measure compared to actually weighing them, but it's what we in the biz call an improvement.

Note that the actual scale values don't really matter in SquidMat. You can use 1 to 1000, 3 to 17 ½, -449 to 1 billion, etc., if you like. The whole thing will get rescaled when the values are converted to Z values. Just choose a scale that you feel will give you reasonably accurate interval differences for the measure.

Speaking of which (and this does come up occasionally)…

For those of you who know about, and insist on, a third property of good (ratio) data values—that the data have a real zero, as opposed to an arbitrary one like the zero in temperature—it doesn't matter. The conversion to Z scores produces real zeroes. So that one's automatically covered.

Changes in the Weighting  

The next major change in the way SquidMat does business is to get rid of the pairwise comparison method for determining weights. I understand why DECMAT uses this approach, but I don't think it's really necessary. In fact, I believe it does more harm than good, particularly in a basic learning environment where most of the students aren't of the quant-monkey persuasion. (That is, they tend not to think about or even care about the conditions and assumptions and such behind the analysis they're told to run. I think they should.)

Briefly, without getting too technical, the pairwise comparison is part of a general decision-support methodology known as (the) Analytic Hierarchy Process (AHP, Godfather Thomas Saaty). Among other things, AHP is useful when you have potential relationships like this: Oklahoma beats Texas, Texas beats Texas Tech, Texas Tech beats Oklahoma. Who's the better team? 

AHP uses pairwise value judgments (and the accompanying eigenanalysis) to divine the best team. DECMAT uses AHP to determine weights. 

Reservations with using AHP in this way. While I appreciate, even admire, the idea of using AHP to lend more objectivity to the problem of subjectively determining weights, I have three reservations.

1. I believe that if it's important enough to grill the students on how the weights are determined, then they should actually be taught how it's done. It does no one any good to give them canned replies (which we always did: "the weights are eigenvalues derived from the pairwise comparison matrix") that neither instructor nor student understands.

2. There is no reason why we shouldn't have perfect consistency for our weight priorities. It seems to me that when I say that one EC is more important than another, then that's that. It is more important. And if EC 1 is twice as important as EC 2, and EC 2 is equal to EC 3, then EC 1 will be twice as important as EC 3 as well. I don't see why that shouldn't be true for what we're doing. Using AHP allows us to be less certain about this.

3. Pairwise comparison makes perfect consistency virtually impossible, even when  we're absolutely certain about the order. In DECMAT's pairwise comparison, we get 4 values to choose from when rating how much more important one EC is over another. (Saaty recommends a range of 1 to 9; DECMAT uses 1 to 4. Both ranges are too small to avoid inconsistency in most situations, even when we want perfect consistency.) Theoretically, you'd need a very large range to achieve perfect consistency.

I'd rather see pairwise comparison as a stand-alone tool, reserved for exceptional situations where it's warranted and not a mandatory step in the process.

The range of importance values. A related concern that I have with the AHP scheme is that a rating of "this EC is slightly more important than the next EC" uses a value of 2. (Both Saaty and DECMAT use a value of 2.) If the scheme were perfectly consistent, you'd be saying that an EC that is slightly more important than the next is actually twice as important. In other words, under perfect consistency, AHP makes the weight of a slightly more important EC twice that of the other one. I personally don't feel comfortable with that assessment. Twice as important doesn't mean slightly more in my way of thinking.

The cure. Here's what I did to fix all that. The pairwise comparison, its eigenanalysis, and the consistency ratio are all gone. The user shuffles the ECs into the order he or she wants and then tells the program how much more important an EC is than the next EC. So each EC except the last one gets a single rating, i.e., how much more important it is than the next EC. The last EC automatically gets a weight of 1. There is no need to assess consistency, since I dogmatically enforce perfect consistency (i.e., all consistency ratios would be 1.00).

Here are the possible Importance values:

• 1.00: same as next EC

• 1.25: slightly more important than the next EC

• 1.50: somewhat more important than the next EC

• 1.75: much more important than the next EC

• 2.00: considerably more important than the next EC

Note that these values are based entirely on my opinion that an EC should be no more than twice as important as the one it's standing next to. I say this because I've seen how easy it is for students to get carried away with them. For example, if I have four ECs, and I think that each one is considerably more important than the next, I get weights of 8, 4, 2, and 1. I think that's a decent spread. If I allow the max to be 4, I get weights of 64, 16, 4, and 1. I think that's a bit much.

Room for other opinions. Because weight determination is an essentially subjective undertaking, I don't want to be entirely dogmatic about it. I've built in an override so that the user can bypass my scale and enter his or her own Importance values. For programming convenience, I limit the override value to 9.99. That is, the most any EC can be over the next EC is not quite 10 times as important. 

Changes in the Sensitivity Analysis

I don't do the "go up and down 3 and see if the answer changes" method used by DECMAT, but the idea is the same. With each EC, I set the weight to 1 and see if the answer changes. I then find each weight value where the answer would change and report what that weight is and what COA wins at that weight. This effectively gives the user all possibilities for the entire range of 1 to infinity.

(Specifically, I use a formula that finds all the weight values at which two or more courses of action tie. I then add 0.0001 to the weight and report it.)