Do the numbers truly represent what they're supposed to?

Last week I had a post about holdings-based attribution, where I laid the groundwork for future commentary on analysis I've been doing on this subject, where I look at the results using both holdings and transaction-based methods. Well, it resulted in comments from my friend and colleague Andre Mirabelli, who questioned the fundamental model's ability to properly evaluate the attributes that produces the excess return. While not wishing to debate him on this topic, per se, it does bring up a broader question that is worthy of consideration.

When portfolio managers, prospects, and clients look at the numbers on a performance report, they draw various conclusions; and the folks who produced the reports no doubt hope that these conclusions are consistent with what they hope would be drawn. However, is everything working as it is intended? Just because a computer has run a particular model, which causes numbers to be produced, which then are assembled in a nice format on a piece of paper or a computer screen, does it mean that everything is correct?

The following graphic will be the basis for what we'll discuss today:

The center set of figures  represent the process that we typically employ: we gather data from a variety of sources; this data is fed into a model, or a series of formulas, and out comes the information, which is presented in reports or on computer screens, iPads, smartphones, etc.

The data issues are decades old: the acronym GIGO still lives on (garbage in, garbage out). One must take strides to ensure that the data is accurate and, of course, appropriate.

The real issue which Andre addressed is the model appropriateness. This is a fundamental issue which doesn't get enough attention. Although I've become less and less a fan of Warren Buffet, I will nevertheless quote him here: "beware of geeks bearing models." And yes, one must be cautious about what models they employ. Do we understand how they work? Do we understand what assumptions they make? What are the results intended to convey?

We recently completed our attribution survey, where we address a variety of issues on this important topic. It has amazed me how over the years, we've seen a shift from folks using the Brinson-Hood-Beebower model to the Brinson-Fachler model. BHB was published a year after BF and in fact is preferred by Gary Brinson. I believe we deserve much of the credit for identifying and communicating the huge difference between the models (through our training classes and various articles, not to mention our books). The late Damien Laker challenged me on this, writing articles which he posted on the Internet, that said that there was no difference; but there is a HUGE difference. If you're not already familiar with it, I'll briefly state that it's how the allocation effect is derived.

Well, folks for years used the BHB model with complete satisfaction; but did they really understand how it worked? Did they understand that there was an alternative, which they might prefer? In most cases the answers are "no." Years ago, when I was first composing my attribution book (which is long overdue for a rewrite), I did a fair amount of research and discovered that many folks simply said "we use the Brinson model." "THE" Brinson model. "The" means that there's only one, as in "THE" president of the United States. It should have been "A" Brinson model, as there are two. But, many developers weren't even aware of this. While the BHB model was published in the Financial Analysts Journal, which meant is was available to tens of thousands of individuals, the BF was published in The Journal of Portfolio Management, which has a much smaller subscriber base, and consequently, less opportunity to be read by the masses.

But even the employment of these models should call into question their appropriateness, given the basic rule that models should align with the investment process. Should a quantitative manager use a Brinson model, that only looks at allocation and selection effects (and, for the more enlightened, the interaction of these effects)? Most likely, no; instead, a multifactor model that looks at the factors they employ in their investment process would make more sense.

I (and many of my esteemed colleagues, such as Stefan Illmer and Steve Campisi) have been on our respective soap boxes for the past few years championing the merits of money-weighting; again, a hugely fundamental issue that is too rarely considered in model development and report production. I often challenge individuals who want me to review reports as to what they're trying to convey. What questions are they trying to answer? If you use the wrong formula, you're producing the wrong result, which can be misleading and not meet your reporting objectives.

This topic isn't a simple one, and covering it briefly in a blog post is impossible (as is obvious from today's attempt, which is only just scratching the surface). Perhaps I'll address it further in this month's newsletter.

By the way, the BF and BHB articles can both be found in Classics in Investment Performance Measurement.