Is performance attribution incomplete?

On a recent drive to and from a GIPS(R) (Global Investment Performance Standards) verification client, I began to listen to a book on Einstein and some of his friends, which included Kurt Gödel. I don't recall hearing of Gödel before, but do recollect his "incompleteness theorems."

According to Wikipedia, "The theorems, proven by Kurt Gödel in 1931, are important both in mathematical logic and in the philosophy of mathematics." It further reports that "The first incompleteness theorem states that no consistent system of axioms whose theorems can be listed by an 'effective procedure' (e.g., a computer program, but it could be any sort of algorithm) is capable of proving all truths about the relations of the natural numbers (arithmetic)."

Does this not hold for performance attribution? Recall that I recently touched on the question as to whether or not attribution answers the questions we wish it to. Perhaps even the best model will leave something out. Might Gödel's theorem hold here, too?

You know the saying, "a little knowledge is dangerous," and it definitely applies here, as I've only lightly scratched the surface of this topic, and would need to devote several hours to have any real understanding of it. But the brief statement above seems to hold some truth.

Perhaps this might be a good topic for Jose Menchero, PhD to address, given that his PhD is in physics, and he is no doubt familiar with Gödel. Interesting subject, I think.

By the way, Gödel is an interesting subject, himself.

What were they thinking?

Those who were around "at the creation" recall the debates regarding whether composite returns should be equal- or asset-weighted. Two groups in particular, the ICAA (Investment Council Association of America; now the IAA) and IMCA (Investment Management  Consultants' Association), lobbied AIMR (Association for Investment Management and Research; what is now the CFA Institute) for equal-weighting. I'll confess that at the time, I didn't pay this a whole lot of attention, and didn't formulate an opinion.

AIMR wanted the composite return to represent the experience of a "single account." That is, what the return would be if the composite was an account itself. IMCA and the ICAA felt that asset-weighting might influence some managers to favor larger accounts, knowing  that their returns would skew the results. And, I suspect that they also thought that equal-weighting made more sense as it shows the average return of actual accounts. But AIMR was steadfast ("resolute," in "W" speak) in their position, and refused to budge. IMCA was so determined that they created their own standard, which went into effect the same time the AIMR-PPS(R) did: it never caught on, however.

The AIMR-PPS did, of course, catch on, and motivated other countries to develop standards, which led to the creation of the Global Investment Performance Standards (GIPS(R)). And as with the AIMR-PPS, asset-weighting because the required way to derive composite returns.

But why? What is the benefit of the composite looking like an account, when it isn't one? The composite is comprised of one or more real accounts, that were managed individually; no one "managed" the composite. Would it not be better to see the average experience of real accounts?

When I conduct GIPS verifications I occassionally run across cases that SHOUT OUT to me that this is all wrong. Here's one recent example:

Because of the huge size difference, account A's return IS the composite's: account B doesn't even have to show up. What's the point of worrying about B? It has zero influence on the return. And yet, the manager's ACTUAL performance in this discipline lies between these two accounts: actually RIGHT IN THE MIDDLE of them (what mathematicians and statisticians call, the average)!

Okay, so the Standards recommend that firms show the equal-weighted composite return. Great! How many firms do? The number is approximately zero. And why not? Perhaps it's because they would prefer not to hand out their presentations on legal size (i.e., 8 1/2" x 14") paper, or resort to a 9 or 10 point font size to fit everything that's required on the page.

I know that this commentary is about as welcome to some as ants at a picnic. But seriously, what were they thinking when they advocated asset-weighting? NO ONE MANAGES COMPOSITES! Firms don't get paid TO MANAGE A COMPOSITE! Would it really be so bad to say, "okay, maybe equal-weighting makes more sense, so effective 1 January 2015, equal-weighting will be mandatory, asset-weighting is optional, and the change goes into effect on this date, but firms are encouraged to restate history"? And what's the likelihood of this occurring? Again, approximately zero. Oh, well.

p.s., Yes, the figures in the table come from a client, though they've been altered slightly, out of respect for our client's confidentiality.

Did the WSJ jinx the DJIA?

In today's WSJ, on page C1 there's an article titled "You Hear That? It's Quiet...Too Quiet," that mentions that it's been 45 trading days without a 100-point decline in the Dow, which is apparently the longest stretch since 2006.

And what happens?  Well, as of this post the market is down 170 points.

And so, we know who to blame!

Overlays and GIPS

Many firms avail themselves of currency overlays, as well as other overlay strategies, which often involve forwards or other derivatives, where there actually are no assets technically "under management." Does this mean that these firms cannot claim compliance with the (Global Investment Performance Standards (GIPS(R))?  Well, let's consider this for a moment.

If you look on pages 17-18 of my comment letter to the GIPS Exposure Draft you'll find the following:

Overlays: The standards don’t speak to overlays. Overlays deal with exposure, not real market values. Many overlay managers believe they can’t comply with GIPS because of this difference. I would like to see the standards speak specifically to overlays and have exposure take the place of market value.

The GIPS Executive Committee did not take me up on my suggestion, but this is probably understandable, given that this would mean introducing brand new material into the Standards, which arguably would have warranted yet another round of comments. Hopefully GIPS 2015 will reference this topic. But, that leaves open the question, "what to do in the meantime?"

It is my position that for overlay managers, the "exposure" is equivalent to "market value," and that these firms therefore should be able to claim compliance. There is only one Q&A on the GIPS website that speaks to the topic of "overlays," and it does not rule out the ability for a firm to include overlays, and so I believe this gives credibility to my position. 

I recommend that overlay managers who wish to comply use their "exposure" in an equivalent way to "assets under management," because this is essentially it is. They should include appropriate disclosures explaining what the information means. I think this is reasonable and in the spirit of the Standards. Have a different view? Let us know.

Do the numbers truly represent what they're supposed to?

Last week I had a post about holdings-based attribution, where I laid the groundwork for future commentary on analysis I've been doing on this subject, where I look at the results using both holdings and transaction-based methods. Well, it resulted in comments from my friend and colleague Andre Mirabelli, who questioned the fundamental model's ability to properly evaluate the attributes that produces the excess return. While not wishing to debate him on this topic, per se, it does bring up a broader question that is worthy of consideration.

When portfolio managers, prospects, and clients look at the numbers on a performance report, they draw various conclusions; and the folks who produced the reports no doubt hope that these conclusions are consistent with what they hope would be drawn. However, is everything working as it is intended? Just because a computer has run a particular model, which causes numbers to be produced, which then are assembled in a nice format on a piece of paper or a computer screen, does it mean that everything is correct?

The following graphic will be the basis for what we'll discuss today:

The center set of figures  represent the process that we typically employ: we gather data from a variety of sources; this data is fed into a model, or a series of formulas, and out comes the information, which is presented in reports or on computer screens, iPads, smartphones, etc.

The data issues are decades old: the acronym GIGO still lives on (garbage in, garbage out). One must take strides to ensure that the data is accurate and, of course, appropriate.

The real issue which Andre addressed is the model appropriateness. This is a fundamental issue which doesn't get enough attention. Although I've become less and less a fan of Warren Buffet, I will nevertheless quote him here: "beware of geeks bearing models." And yes, one must be cautious about what models they employ. Do we understand how they work? Do we understand what assumptions they make? What are the results intended to convey?

We recently completed our attribution survey, where we address a variety of issues on this important topic. It has amazed me how over the years, we've seen a shift from folks using the Brinson-Hood-Beebower model to the Brinson-Fachler model. BHB was published a year after BF and in fact is preferred by Gary Brinson. I believe we deserve much of the credit for identifying and communicating the huge difference between the models (through our training classes and various articles, not to mention our books). The late Damien Laker challenged me on this, writing articles which he posted on the Internet, that said that there was no difference; but there is a HUGE difference. If you're not already familiar with it, I'll briefly state that it's how the allocation effect is derived.

Well, folks for years used the BHB model with complete satisfaction; but did they really understand how it worked? Did they understand that there was an alternative, which they might prefer? In most cases the answers are "no." Years ago, when I was first composing my attribution book (which is long overdue for a rewrite), I did a fair amount of research and discovered that many folks simply said "we use the Brinson model." "THE" Brinson model. "The" means that there's only one, as in "THE" president of the United States. It should have been "A" Brinson model, as there are two. But, many developers weren't even aware of this. While the BHB model was published in the Financial Analysts Journal, which meant is was available to tens of thousands of individuals, the BF was published in The Journal of Portfolio Management, which has a much smaller subscriber base, and consequently, less opportunity to be read by the masses.

But even the employment of these models should call into question their appropriateness, given the basic rule that models should align with the investment process. Should a quantitative manager use a Brinson model, that only looks at allocation and selection effects (and, for the more enlightened, the interaction of these effects)? Most likely, no; instead, a multifactor model that looks at the factors they employ in their investment process would make more sense.

I (and many of my esteemed colleagues, such as Stefan Illmer and Steve Campisi) have been on our respective soap boxes for the past few years championing the merits of money-weighting; again, a hugely fundamental issue that is too rarely considered in model development and report production. I often challenge individuals who want me to review reports as to what they're trying to convey. What questions are they trying to answer? If you use the wrong formula, you're producing the wrong result, which can be misleading and not meet your reporting objectives.

This topic isn't a simple one, and covering it briefly in a blog post is impossible (as is obvious from today's attempt, which is only just scratching the surface). Perhaps I'll address it further in this month's newsletter.

By the way, the BF and BHB articles can both be found in Classics in Investment Performance Measurement.

BREAKING NEWS!!! GIPS help has gotten easier!

The Spaulding Group, Inc. has just announced the creation of a new website service, that provides answers to GIPS(R) (Global Investment Performance Standards) related questions.

GIPSHelp.com.

"We are often contacted by clients and colleagues with questions dealing with the Global Investment Performance Standards," said Christopher Spaulding, a Senior Vice President at The Spaulding Group. "We felt GIPSHelp.com would be a tremendous resource for compliant firms and firms looking to become compliant. In addition to serving as a valuable time saving compliance resource for our industry, there will also be a private, members-only section dedicated to The Spaulding Group's verification clients."

With the tremendous growth in our verification practice (both GIPS and non-GIPS), we see many situations on a regular basis that require interpretation. And, we regularly receive questions from clients and colleagues. It just seemed to us that this would be a beneficial service for the industry.

And, it's free! All you need to do is register to use it.

We look forward to your feedback.

What's wrong with holdings-based attribution?

The Spaulding Group recently completed its most recent survey, which deals with performance attribution. Jed Schneider, CIPM, FRM discussed many of the results at a luncheon held in NYC, where we learned that, as with the prior three editions of this research, most folks prefer transaction-based attribution, though roughly half use holdings-based. The probable reasons as to why the contradiction are interesting, but not part of this post.

For several years, I have attempted to encourage a proper and unbiased evaluation of the two methods, to determine the true differences, and whether there is a point when one cannot justify using the holdings-based approach; a point where the use of transaction-based attribution is a "must." But, no one chose to do the research, so I did. And I discussed some of my preliminary findings at last year's PMAR (Performance Measurement, Attribution & Risk) conferences, and will discuss further findings at this year's events.

Today, I merely want to discuss the three problems with the holdings-based approach. And, in "David Letterman" style, I will do it in reverse order of significance.

Number 3: we will usually have a residual. For many readers, this will seem odd to have been placed third, because surely the residual is the main problem with holdings-based, but I'd say it is not. A "residual" is a non-zero amount which reflects the inability to fully reconcile to the excess return. Recall that with relative attribution, our goal is to completely account for the excess return. However, with holdings-based models we often (actually, more like usually) can't do this, but have a "residual," meaning we don't fully account for the excess return. In reality, it's worse than that.

Number 2: getting the proportions wrong. Here I mean that the amount that is assigned to the different effects may, in realty, be incorrect. We may have too much or too little assigned to allocation, for example. Thus, it's contribution to the excess return is over or understated. This goes beyond merely not reconciling to the excess return; rather, the numbers that are produced can be allocated in a manner which doesn't properly align with reality; their proportions are incorrect.

Number 1: having the wrong signs. To me, this is the major problem. What do I mean? Well, with holdings based attribution, we may be showing a positive selection effect, but in reality, it's negative! And so, instead of saying "great job!" we should be saying "you've got to do better!" And the problem is, you won't know this. You'll see a positive selection effect and conclude that those decisions were good ones that contributed to the excess return, when in reality the decisions hurt your performance!

This third problem (#1, actually) means that the results will be misleading,  spurious, invalid. What's the risk of this happening? I think you'll be surprised.

An article will be forthcoming which will provide additional details on my research. Suffice it to say, the results are fairly startling, and should encourage most users of holdings-based models to seriously consider switching.

p.s., for more details on the attribution survey, contact Patrick Fowler.

p.p.s., our next survey deals with the GIPS(R) Standards (Global Investment Performance Standards). Please join in! It will begin this summer.