Monday, March 5, 2012

Do the numbers truly represent what they're supposed to?

Last week I had a post about holdings-based attribution, where I laid the groundwork for future commentary on analysis I've been doing on this subject, where I look at the results using both holdings and transaction-based methods. Well, it resulted in comments from my friend and colleague Andre Mirabelli, who questioned the fundamental model's ability to properly evaluate the attributes that produces the excess return. While not wishing to debate him on this topic, per se, it does bring up a broader question that is worthy of consideration.

When portfolio managers, prospects, and clients look at the numbers on a performance report, they draw various conclusions; and the folks who produced the reports no doubt hope that these conclusions are consistent with what they hope would be drawn. However, is everything working as it is intended? Just because a computer has run a particular model, which causes numbers to be produced, which then are assembled in a nice format on a piece of paper or a computer screen, does it mean that everything is correct?

The following graphic will be the basis for what we'll discuss today:















The center set of figures  represent the process that we typically employ: we gather data from a variety of sources; this data is fed into a model, or a series of formulas, and out comes the information, which is presented in reports or on computer screens, iPads, smartphones, etc.

The data issues are decades old: the acronym GIGO still lives on (garbage in, garbage out). One must take strides to ensure that the data is accurate and, of course, appropriate.

The real issue which Andre addressed is the model appropriateness. This is a fundamental issue which doesn't get enough attention. Although I've become less and less a fan of Warren Buffet, I will nevertheless quote him here: "beware of geeks bearing models." And yes, one must be cautious about what models they employ. Do we understand how they work? Do we understand what assumptions they make? What are the results intended to convey?

We recently completed our attribution survey, where we address a variety of issues on this important topic. It has amazed me how over the years, we've seen a shift from folks using the Brinson-Hood-Beebower model to the Brinson-Fachler model. BHB was published a year after BF and in fact is preferred by Gary Brinson. I believe we deserve much of the credit for identifying and communicating the huge difference between the models (through our training classes and various articles, not to mention our books). The late Damien Laker challenged me on this, writing articles which he posted on the Internet, that said that there was no difference; but there is a HUGE difference. If you're not already familiar with it, I'll briefly state that it's how the allocation effect is derived.

Well, folks for years used the BHB model with complete satisfaction; but did they really understand how it worked? Did they understand that there was an alternative, which they might prefer? In most cases the answers are "no." Years ago, when I was first composing my attribution book (which is long overdue for a rewrite), I did a fair amount of research and discovered that many folks simply said "we use the Brinson model." "THE" Brinson model. "The" means that there's only one, as in "THE" president of the United States. It should have been "A" Brinson model, as there are two. But, many developers weren't even aware of this. While the BHB model was published in the Financial Analysts Journal, which meant is was available to tens of thousands of individuals, the BF was published in The Journal of Portfolio Management, which has a much smaller subscriber base, and consequently, less opportunity to be read by the masses.

But even the employment of these models should call into question their appropriateness, given the basic rule that models should align with the investment process. Should a quantitative manager use a Brinson model, that only looks at allocation and selection effects (and, for the more enlightened, the interaction of these effects)? Most likely, no; instead, a multifactor model that looks at the factors they employ in their investment process would make more sense.

I (and many of my esteemed colleagues, such as Stefan Illmer and Steve Campisi) have been on our respective soap boxes for the past few years championing the merits of money-weighting; again, a hugely fundamental issue that is too rarely considered in model development and report production. I often challenge individuals who want me to review reports as to what they're trying to convey. What questions are they trying to answer? If you use the wrong formula, you're producing the wrong result, which can be misleading and not meet your reporting objectives.

This topic isn't a simple one, and covering it briefly in a blog post is impossible (as is obvious from today's attempt, which is only just scratching the surface). Perhaps I'll address it further in this month's newsletter.

By the way, the BF and BHB articles can both be found in Classics in Investment Performance Measurement.

4 comments:

  1. Stephen Campisi, Intuitive Performance SolutionsMarch 5, 2012 at 7:36 AM

    To everyone interested I say: "Get ready for PMAR X in Philadelphia this May." We will take on "The Myth of BHB" along with a few other juicy and relevant topics that bring much of the unnecessary confusion that's present in the world of performance analysis.

    To Dave I say: You've been much too kind to the BHB model. It's not an "alternative" method of attribution; it's closer to a pile of dinosaur poop. Like in the first Jurrasic Park movie, unless you dropped your cell phone in it, I see no reason why anyone would want to go near it.

    Stay tuned. And get ready for PMAR X.

    ReplyDelete
  2. Stephen, first, thank you "for the plug." Second, my kindness knows no bounds. Can't wait to hear you in Philly!

    ReplyDelete
  3. David,
    To give the reductionist explanation that I believe nevertheless points the correct way forward:

    If one has two sectors and one’s decision process is to first distribute one’s fund among the sectors and then separately within each sector, one has only made three decisions.
    The first decision, sector allocation, that distributes between the two sectors, is only a single decision. It is not two decisions of, say, first deciding to put 70% in sector A and then to put 30% in sector B. Putting 30% in sector B is not a separate decision.
    Thus, the amount of active return attributed to the first decision (the one and only sector decision) is

    Sector Allocation equals the sum of the bet of sector one times the BM return of sector one plus the bet of sector two times the BM return of sector two (BHB).

    But this is exactly equivalent, numerically and otherwise, to having

    Sector Allocation equal the sum of the bet of sector one times the BM return of sector one minus the total BM return, plus the bet of sector two times the BM return of sector two minus the total BM return (BF).

    So I believe that Damien was correct in this instance. The fact that you can here decompose Sector Allocation into two (and many more) separate terms (in many different ways) does not mean that they separately have any relevant meaning. They do not. Only the total sector allocation has any meaning in a two-step decision process. So the difference between BHB and BF is meaningless.

    David, as you know, I also take objection to the interaction effect (since it corresponds to no actual investment decision) and to the use in attribution of money-weighted returns (due to their trouble in defining meaningful weights).

    I also look forward to further discussing all of this again at PMAR X. See you there.

    ReplyDelete
  4. Andre, thanks for your note. If you cannot see that there is a difference in the allocation methods, I am at a loss to convince you otherwise. I look forward to seeing you at PMAR, and to hear your presentation.

    ReplyDelete

Note: Only a member of this blog may post a comment.