Tuesday, October 13, 2009

Beacon Economics Rips Sacramento State Study on Regulatory Costs

Recently, Dean Sanjay Varshney of Sacramento State has made a big splash with a new bi-annual economic forecast for Sacramento and some high profile consulting reports on economic policies. I like competition and have refrained from commenting on his new Sac State forecast or any of his consulting reports... until now.

On Monday, the Sacramento Bee ran an op-ed by Chris Thornburg and Jon Haveman from Beacon Economics that ripped Varshney's new report on the cost of regulations on small business in California. Varshney claims regulations cost the California economy $500 billion per year. Beacon says,
Disturbingly, this result is one of the worse examples of schlock science we've ever seen. And chillingly, it is being held up by a number of state leaders... This is truly bad analysis, and it matters because it once again points to the urgent need to have policy research vetted through third parties for methodological soundness and intellectual honesty... The state must make it a priority to base policy decisions on rigorous research rather than pseudoanalysis that twists or flat-out ignores the truth.

Ouch. That's about as harsh as it gets without resorting to personal attacks. However, I actually sent Thornburg and Haveman an email with the title, "You guys are too kind." I wasn't commenting on their manners, but the fact that their op-ed only explained half of the problem with the Sac State study.

The Sac State study has 2 parts: 1. a regression estimating Gross State Product as a function of indicators from Forbes magazine and 2. enters the loss in GSP estimated from this regression into an input-output model (IMPLAN) to get multiplier effects. The Beacon op-ed accurately explains the problem with the first part, but not the second (probably due to word limits or an attempt to keep it simple). The error in the 2nd part may be even worse. This is one of the worst abuses of IMPLAN I have ever seen.

IMPLAN is a very useful input-output economic model. I use it all the time. However, it is also the most misused model in economic research. Because it is relatively cheap and easy to use (as far as economic modeling software goes), there are a lot of people out there doing bad consulting with it. IMPLAN studies often double count impacts and fail to make proper adjustments when the modeling or data assumptions don't hold. Many credible academics won't even use input-output models at all because of these notorious pitfalls. In this case, it is simply inappropriate to use IMPLAN at all since the first step of Varshney and Associates regression is econometrically (I use the term loosely here) estimating a macro output variable (GSP) directly. Even if you trust the first part of their study (and you shouldn't), the 2nd stage is simply double and triple counting the effects of the first stage.

I think if we had a Middle Ages style plague that killed 20% of the state's population, the impact would probably be less than Varshney's $500 billion estimate of regulatory effects. Yes, regulations are costly, but exagerrated studies like these are very destructive to real policy debates. The KQED blog gives a little of the back story on the study here.

This is the 2nd time in a few months that Varshney and Associates have issued a report with exagerrated costs of regulations. A few months ago, they released this report commisioned by a small business group stating that AB 32 (greenhouse gas reduction policy) will cost the California economy over 1.1 million jobs (more than have been lost in the current recession, except the AB 32 effect is permanent). I agree with them that AB 32 regulation will have costs on the CA economy, but Varshney's job loss and cost numbers are unbelievably large and come from poor research methods (I'll spare you the details here).

I have nothing personal against Dean Varshney, but I hope his public policy consulting business fails. Like a good businessman, he knows what his clients want to hear and delivers, but these reports are very misleading. I recommend his clients give me a call. I am confident our Center can produce a better study for a lower price. You may not get the number you want, but it will stand up much better to review.

Postscript:
In an earlier post, I scored Beacon Economics the winner in a debate with the Milkin Institute over manufacturing jobs.

Update (Oct. 15):
In a complete coincidence, 12 hours after I posted this, Dean Varshney walked into my office at Pacific and introduced himself. He was on campus for another meeting, and just wanted to take a moment to meet me and suggest that we might collaborate on a project in the future. He was very nice and obviously had no idea that I just posted this criticism. I was surprised and felt a little guilty inside, so I briefly said nice to meet you too and exchanged small talk. In hindsight, I probably should have politely told him that I think he is working on interesting and important topics, but his research has serious problems.

I like him, and I have heard from others that he is a good business school Dean. Like many successful executives, he appears to be very good at building relationships and creating opportunities. That said, his economic research products need a lot of help. On the other hand, I'm pretty good at the research, but need to work on public relations. Maybe we would make good business partners :)

11 comments:

  1. Is there a good critique of their AB32 study anywhere online? (Seeing as how the Bee is still writing about it...)

    Also, is it S.O.P. to disclose how much one was paid to do a study like this?

    And third - is there any hit to a business school faculty member's reputation, if it transpires that he saw a business niche (for an "opinion laundry") and exploited it?

    ReplyDelete
  2. Also - this is somewhat pedantic, but isn't the name "Varshney and Associates" a Fictitious Business Name?

    I don't see it registered, either in Sacto county or in El Dorado county. Perhaps I'm not searching for it the right way?

    I tried from
    http://main.co.el-dorado.ca.us/recorder.html
    and
    http://www.efbn.saccounty.net/

    ReplyDelete
  3. Anna,
    Matt Kahn's (UCLA) review of the AB32 study is here http://www.arb.ca.gov/cc/scopingplan/economics-sp/matthew_kahn.pdf

    Disclosing funding sources on reports is standard, amounts are not usually printed on the reports but are typically not secrets. The Bee article on the AB32 report says the contract was $54,000, and other reports on the Regulation Costs report say it was $85,000 from the state.

    Yes, the quality of research impacts one's academic reputation, it doesn't matter whether that research is for consulting reports or academic journals.

    Jeff

    ReplyDelete
  4. Anna,

    Easy on the business name sleuthing. The name is right on the reports and it is clear who is doing the work and who is paying.

    The issue is that the reports are poorly done. There are some overly rosy studies by environmental groups on the green jobs issue too. However, this report from Varshney really stands out from the rest.

    ReplyDelete
  5. > The name is right on the reports and it is clear who is doing the work and who is paying.

    Agreed; but I do think business school faculty members should be familiar with business fundamentals, including what the requirements are for filing a fictitious business name statement.

    ReplyDelete
  6. FYI, my humble blog post: Did "fatally flawed" Calif AB32 analysis authors, CSUS Business School dean Varshney & prof. Tootelian, neglect to register their own business names?

    Feedback and corrections much appreciated.

    (pretty much the only new news in it, beyond what's already in your comments right here Jeff, is the stonewalling.)

    ReplyDelete
  7. Anna,

    I guess I'll stick up for Varshney and Tootelian on this point and say I may have hung up on you too. I am not sure that "and Associates" legally means additional owners, just implies additional employees. My wife works for an "and Associates" consulting firm with 1 owner/principal, and 3 part-time employees. I think the term is used a little deceptively sometimes by people trying to make their business sound bigger than it is, but that hardly seems illegal.

    Jeff

    ReplyDelete
  8. > "I am not sure that "and Associates" legally means..."

    The "and associates" does mean that, legally speaking, the name is a Fictitious Business Name.

    From Business and professions code section 17900(c) (?):
    "(c) A name that suggests the existence of additional owners... is one that includes such words as "Company," "& Company," "& Son," "& Sons," "& Associates," "Brothers," and the like...

    And legally, FBNs must be registered.

    You do realize this, don't you? Or am I missing something?

    (or is this not what they teach, in business school?)

    ReplyDelete
  9. You are right Anna, I don't know about fictitious business names. I have an excuse, I'm an economist and have never taken or taught business law, despite being a prof. in a business school.

    Here is what I do know. I have 3 degrees in Economics and have completed a specialization in environmental economics and natural resource policy, and have published in peer-reviewed journals about global warming issues. Thus, I am qualified to conduct and review studies on the costs of AB32 and similar policies. Varshney and Tootelian's study is the worst I have seen on the subject and seems to reflect little training or experience in the area. I looked at their bio's and saw they have degrees in Finance and Marketing, not Economics (yes, there is a big difference and it shows here).

    I don't know if their business is "fictious" or not and I really don't care. I do care that they are doing research outside their field of expertise that is confusing people, and muddying very important policy debates.

    ReplyDelete
  10. > "...they are doing research outside their field of expertise that is confusing people, and muddying very important policy debates."

    What I find particularly disturbing is that McClatchy - the ones with a reputation for accuracy, who got it right in the runup to the war on Iraq - would be an active party in confusing the public on this - they took Jim Downing's Oct 19 Sac Bee article ( originally titled Fiscal impact of state climate law disputed; it addressed the criticisms of Varshney&Tootelian's study, albeit gently) and for McClatchyDC they retitled it "California State dean predicts major fiscal impact from climate law"

    Earth to McClatchy - please leave your readers knowing *more* than when they came to you, not less.

    ReplyDelete
  11. Interesting - my previous comment is somehow completely screwed up on the post page, but it's fine on the "submit a comment" page.

    ReplyDelete