BOTEC Estimates of Washington Cannabis Market are UNDER-Estimates

Executive Summary:  (for the TL;DR crowd)

BOTEC’s recent estimates of the size of Washington’s Cannabis market are terrible, horrible, and rather gnarly.  One might think that BOTEC would be embarrassed by the work they sold to the Washington State Liquor and Cannabis Board.  Of more immediate importance, these BOTEC-generated magical estimates are, simply,  way too small.  Systematically way too small.  This is important in that these under-estimates have already contributed to the too-small allocation of only 222 new retail access points to support the medical cannabis market.  If you agree with my assessment of BOTEC’s work, I would suggest that you write to let both your state legislators and the WSLCB know that you do not believe they should be using (or repeating) the errors that BOTEC has supplied as finished work product.  That finished work product is published on the WSLCB site  here, in case you wish to follow some of the specific page references below.

This is my first HI-Blog post in the last 4 weeks.  I’ve been busy transitioning HI-Blog so that it is now  hosted under my new commercial website www. straightlineanalytics.biz.  Attempts to go to the old HI-Blog site should re-direct to the blog page on SLA.  The SLA site is basically complete, but note that I have turned off the  “Shop” until later this week.  This is  where documents, service packages and tools will be made available for purchase.  I just have to do some banking things in order to set that up, and I did not want to hold off any longer on blogging …. my lack of a productive outlet for my words here has resulted in me getting way too verbose on Danielle’s forum.

I expect to be posting at least 2-3 articles per week on HI-Blog  going forward.  With luck the pressure of production will lead to less verbosity (on average) going forward.

On to BOTEC – or how to make a living producing meaningless numbers.

The following is the distillation of the top 9 things I found horribly wrong with BOTEC’s work – coupled with a bit of related commentary.

My thanks to the colleagues that provided input on my initial drafts of this work.

.

Points of Critique on BOTEC’s Estimates of the Size of the Medical Market in Washington State

Released by the Washington State Liquor and Cannabis Board Dec 2015

Point #1) – Lack of Accuracy and Quality Control in BOTEC’s work

  • The tables summarizing the primary results of BOTEC’s study are full of errors (tables on pages 2 and 30) … to the point that 2/3 of the rows do not even add up to the expected total (note … this is not a rounding problem, it is a blatant error and makes the results of the study virtually impossible to understand).

Point #2) – The use of DOLLARS to define the size of each market component is problematic

  • Given the different price-points (per gram) in the three markets, it is difficult to see how money corresponds to either the “consumer count” or “product volume” by which markets are traditionally sized.  BOTEC does appear to have pulled in external estimates relating to what the price (per gram) should be in the medical and illicit markets.  Setting the average price of Medical Cannabis in Washington at $7.11 per gram does, however, seem to be low (by a factor of approximately 60%).  Setting the I-502 Retail price per gram at between $5.26 and $7.36 per gram is, simply, ridiculous (Fig 11, page 21) (and compounds their systematic under-estimation)

Point #3) – Multiplying the level of recreational sales that occurred in October 2015 by 12 is not a reasonable way to estimate the expected annual size of a market that is still in it’s infancy

  • aside from the fact that Oct’15 saw diminished sales in the face of Oregon beginning to offer tax-free Cannabis on Oct 1, our recreational market is far from saturation  in it’s 19th month of existence.

Point #4) – The use of CURRENTLY OPEN DISPENSARIES as the fundamental proxy by which to “size” the medical Cannabis market is inappropriate in an environment where dispensaries have been the focus of increased enforcement and closure over the past year.

  • This almost seems designed to low-ball the estimated size of the medical market.  I believe the use of this proxy is indefensible.  Taken to an extreme, had the state and local authorities been effective in closing ALL operating dispensaries by last October, BOTEC’s methodology would have declared the size of the medical market to be ZERO (dollars).  Should BOTEC wish to defend this aspect of their methodology, might I suggest a public debate on the topic.  I volunteer myself to be the “con” party in that debate.  This is, IMHO, one of the key logical flaws to how BOTEC chose to approach the initial question.  It will be interesting to see BOTEC’s defense of this methodology – even if not in the form of a debate.

Point #5) – Having grossly underestimated the sizes of both the recreational and medical components of Washington’s Cannabis marketplace, BOTEC then estimated the size of the Illicit market.  Given the apparently systematic underestimates of the other two market components (recreational and medical), the method by which BOTEC has chosen to estimate the size (in dollars) of the illicit market is largely irrelevant.

  • I’m being a bit hard on them here.  Illicit market estimates are tough at the best of times.  Even if done perfectly by BOTEC, their estimates regarding the illicit market are rendered less useful given the flaws with their estimates concerning the two more visible components.  If the tables mentioned in Point #1 above actually totaled to 100%, (or the Low/Best/High overall market size dollar amounts) there might be more value in what they produced for their illicit market estimates.  As it is, I am not quite sure what the final estimates are for the sizes of this market (Low, Best and/or High).  Is anyone?  Are they?

 Point #6) – BOTEC seems to assume that “no relevant data exist” upon which to make certain key estimates (such as the price per gram at Retail for the recreational market) and thereby justifies less-than-optimal proxies with which to fill in the blanks.

  • This methodology is acceptable if there are no relevant data.  BOTEC appears to be unaware of  the richness of data contained in the LCB Traceability system (on, for example, retail pricing, market growth, transactional volume, product turnover rates, Etcetera).  It is a little hard to see how this omission got by WSLCB Staff while reviewing earlier drafts of this work, but I guess they don’t spend as much time with their data as do I.

Point #7) – BOTEC appears to use biased language in describing the “significance” of certain key intermediate results.

  • Calling correlations of .47 and .44 “Strong Relationships” in relatively low-sample-size analyses seems odd (see near the top of page 15).  It is highly likely that these two correlations were not statistically significant.  BOTEC does not follow the standard of reporting on the p-value associated with these statistics.  They should be required to do so.  Should they have called statistically non-significant correlations “Strong Relationships”, their work would be verging on fraudulent reporting.  At a minimum, it would be misleading.

Point #8 –BOTEC appears to use Monte Carlo simulation in both appropriate and inappropriate ways   They may believe that Monte Carlo will shed light that cuts through the uncertainties inherent in their methodology.  If so, they would be wrong.

  • This is a technical point, but it basically reduces into a “garbage in / garbage out” argument.  At least a portion of how they appear to have used Monte Carlo appears to be garbage.  It is, however, difficult to tell with certainty, given how few specific model details are supplied.  While I appreciate the transparency afforded by  Figure 10 on page 18, that transparency is tarnished by the apparent error in the reported parameters for the  “uniform distribution” reported in the last row of the table.  Seeing the full model that they used would be helpful in fully diagnosing just how bad (or good) their Monte Carlo work actually is.  Regardless, using Monte Carlo simulations to illustrate the uncertainty implicit in the underlying model is OK, but it is circular and adds no incremental value to their analysis.  But it’s “fancy” and “complicated” and does “10s of thousands of iterations”, so it must be OK — and worth lots of money in fees.

Point #9) –How can the work of an entity that apparently has no shame be trusted?

  • This work appears to have been designed with little consideration of the original mandate to provide “fair and balanced” (my words) estimates of the size of the medical Cannabis market in Washington.  I was not aware that the standards for work provided to Public Sector clients had become so low as to enable work of this quality to be put forward as finished work-product.  That BOTEC felt the need to place a disclaimer in their work that it should, effectively, not be used for anything may help them sleep better at night.  It does not lessen the disservice that they have done to their Client and to the entity and people that it represents.
  • That it was subsequently, published by a Government Agency is unfortunate.
  • That it has, apparently, informed the under-allocation of only 222 additional retail access points to service the Patient Protection Act is potentially tragic.

I found BOTEC’s work disturbing.  I thought they were one of the “experts” in this field.  I expected more of them.

I would imagine that a reasonable person might well expect them to be ashamed of their work and the negative impact it is poised to  have on public health and safety in Washington..

If allowed to stand, the allocation of only 222 additional retail access points for Cannabis (be it recreational or medical) will cause harm to public health and safety and it will do so BEFORE July 1 of this year.  Ironically, the emergency rule that proposed this allocation is accelerating down a path that will, predictably, in itself create a serious emergency that will have negative outcomes on human health.

I’d like to offer one specific and crucial piece of advice regarding any efforts that might arise to mitigate and/or avert this fully predictable impending (and real) emergency.  That is that the way in which the Board chose to allocate the new access points in December REINFORCES the lack of access in areas suffering the impact of bans or moratoria.  A good rule – a good process would, hopefully, mitigate or (at least) remain neutral regarding the impact of local obstructionism on the implementation of the Patient Protection Act.

Note to the WSLCB:  Medicine is not beer is not wine is not whiskey is not tobacco.

It is my belief that it is neither the role nor the intention of the WSLCB to double-down on local obstructions to access.  Particularly for stores with medical endorsements. That the allocation method used for the Dec 16 announcement did, effectively, double-down on obstructionism, is unfortunate.  It may well have been a well-intentioned attempt to ensure that the much-needed new access points would at least exist SOMEWHERE in the state.  Ban/Moratoria areas, after all DO have stores allocated — just not open or licensed, for the most part.

If the LCB reinforces local obstructionism today, that will contribute to ongoing compromise to medical access for the populace of affected areas.  I hope they reconsider and allow for additional store allocations in affected areas.  The amount that the local people need —- not the amount that the local leaders desire.

At a minimum, how about ensuring that the number of stores allocated is AT LEAST half (rounded up) of the number of retail pharmacies that exist in the area?  That is an easy and quick empirically-based  allocation that does not cost the people of Washington any consultant fees and reflects local pharmaceutical demand.  It’s a way of letting the “market” tell you what it needs.

Regardless, the LCB and other Governmental agencies should stop using BOTEC’s recent analysis to inform their rule-making and policy-making efforts relating to Washington’s Cannabis marketplaces.

This, of course, suggests that the WSLCB should be trying to get some good estimates of the size of Washington’s Cannabis market(s).

Me stating this may come across as self-serving, as I have produced such estimates, and know how it SHOULD be done.

CASP might be able to help if the WSLCB finds they need help with this.   They can certainly do a better job than BOTEC has done on this question.

Perhaps the WSLCB might think of dusting off the work that a group of us (CASP, myself & the CCSE) supplied to them (free of charge) last year which estimated how many incremental access points were needed to support the forced migration of Washington’s medical Cannabis infrastructure to the regulated and controlled world.

I would like to think, given how horribly bad BOTEC’s recent work is, that the WSLCB might assign some critical eyes to look over the prior work BOTEC has submitted to the State of Washington.  Work, remember, that the State has used to design and implement the I-502 market and much of their legislative and regulatory decision-making regarding State-Legal Cannabis.

To summarize, it is my strong belief that the results of the BOTEC study are, essentially, meaningless.

They systematically under-estimate values commonly known to be higher.

Their results should not be used to inform or guide policy or business decisions.

Let’s try not to hurt people in 2016 if we can avoid it.  Using BOTEC’s results (and even assuming them to be useable) will hurt people in 2016.  Doing so will also hurt businesses — lots of businesses.

Underestimating the size of the medical market guides policy that causes patients harm.

Underestimating the size of the recreational market guides policy that harms the development of the recreational market.

Underestimating the size of the illicit market is something I expect will generally happen (for lots of reasons).  Regardless, state policy should be to diminish the illicit market.  I think most agree on that.  To measure progress on illicit-market-reduction efforts, good estimates of the true (not dollar-based) sizes of the three market components is key – as is an understanding of the flows of patient (and product) between the three.

It is unfortunate that, in attempting to fulfill whatever terms were placed on their contract to estimate the sizes of this State’s Cannabis markets, BOTEC has failed to deliver meaningful information on any of these things.

One wonders how many of our tax-payer dollars they received for this work.  Dollars that could have gone to help pay for education (or to get a better traceability solution).

Back Of The Envelope Calculations, indeed!

Last I saw, computers were better than envelopes for doing quantitative analytic work.

 

 

9 comments

  1. Great work Dr.jim! Botec’s incompetence should be obvious to the LCB by now but they continue to ignore the obvious. With your help hopefully we can walk them closer to the light. Keep it up!!

  2. This is scary. We the People put trust in our government and their agencies to do their due dillagence. Albeit a misguided trust, this blatant disregard for accurate numbers in horrifying.

    Have you spoken with BOTEC to see what their response is?

    1. I, also, believe this is scary. (Somewhat like the lab assessments … wherein the initial observations regarding problematic results were shared with the LCB in the FIRST HALF OF 2015 by the Cannabis Transparency Project Team and – yet – when I testified last month, the Board was informed that “Staff is aware of this lab work”. Personally, I’d prefer that they be both aware and willing to do something to address the “issue” at hand.).

      I contacted BOTEC early last month … and have had no response whatsoever from them.

      I’ve attached the text of the e-mail that I sent to BOTEC regarding my early observations regarding their so-called work below.
      After you read it, ask yourself the following: would YOU have replied to the following e-mail? BOTEC did not. Probably buried in envelopes.

      *****************************************************************************************************************

      From: JAMES [mailto:jim@straightlineanalytics.biz]
      Sent: Thursday, January 7, 2016 3:30 PM
      To: ‘jeremy.ziskind@botecanalysis.com’ ; ‘nate.mladenovic@botecanalysis.com’
      Subject: Concerns re: BOTEC Report “Estimating the Size of the Medical Cannabis Market in Washington State”

      Jeremy & Nate: I am sending this to you both, as yours are the only e-mail addresses I could find relating to BOTEC (and I noticed you are both authors of the report in question).

      Consider it a courtesy, in that I have looked closely at this report and find much about it disturbing. Deeply disturbing.
      Having previously examined your previous work informing the LCB in their formation of the I-502 market, I was expecting more/better when I read the report published Dec 15.

      In my opinion, BOTEC has harmed the people of Washington with this work. BOTEC has, also in my opinion, done a disservice to the WSLCB in providing work of this quality. Decisions based on this report are, at best, uninformed. I would take it as far as to suggest that such decisions will, actually, be mis-informed. (Note that I, also, read the disclaimer contained in your report. Good to see you engage counsel while communicating your analytic results.)

      In producing this report, I suspect that BOTEC has also hurt their future status in this industry.

      I have shared, in general terms, my overall concern about the utility of this report in informing the store allocation decision before the WSLCB with their Director (and very briefly in my public comment to the Board yesterday morning).

      I expect to be following up with more detailed commentary directly to the Board (and, quite likely, the media) over the next few weeks.

      Thought you should know.

      You may want to check your work in the future before releasing it.

      Harming vulnerable patients for profit does not sit well with me, and I do not believe that it would be something that sits well with most participants in the emerging state-legal Cannabis industry.

      Primum non nocere has served physicians well for centuries. Perhaps you should read Hippocrates before attempting to influence Medical markets in the future.

      Dr. James MacRae
      Straight Line Analytics

  3. Hi Jim,

    I’m assuming you saw this article published a couple of days ago.

    Seattle Dispensaries Sue WA Oversight Board
    http://mjbizdaily.com/seattle-dispensaries-sue-wa-oversight-board/?utm_source=feedly&utm_medium=rss&utm_campaign=seattle-dispensaries-sue-wa-oversight-board

    They are upset over the allocation of licenses. This paragraph is interesting in light of your above article.

    “The board also relied on a flawed report by BOTEC Analysis, an independent contractor that was hired to determine how many more licenses the state should issue, in order to satisfy MMJ demand, the suit alleges. That led to an uneven distribution of licenses among municipalities, and is especially egregious because the board knew the BOTEC report was not reliable, the suit says.”

    Who knew what when and what connections they have, both politically and financially, would be extremely interesting to know. And I’m not just talking about BOTEC being paid for this “report”.

    1. Hey Ed, thanks for the link. Glad to see that BOTEC is getting the sort of exposure they’ve earned.

      You might find my comment at the site you linked to amusing, and hopefully illustrative of what is so flawed with BOTEC’s analytical process.

  4. Wow, that report is embarrassingly sloppy. I’m convinced that the fact that the individual contributions add up to the sum on the page 2 table in the Best Estimate row under Market Share is pure coincidence, as only the I502 column is within rounding error’s reach of classical math results. Even the ones that look right at a glance are wrong: 480/1330 is 36.1, not 37; and 390/1330 is 29.3, not 28. The rest of the chart is so obviously and incomprehensibly errant that it’s hard to imagine how it wasn’t noticed before publication. I’ve heard it said that pot’s most devastating cognitive effects fall hardest on those who don’t use it, and this seems like one more data point in support of that idea.

    But I can’t say I’m surprised. I’ve been following the meanderings of one Dr. Mark A. R. Kleiman (chairman of BOTEC and lead author of this report) for quite a few years now. I find his writings boorish, his analysis sloppy and heavily biased, and his attitude arrogant and condescending.

    You can be thankful that you’ve yet to receive a response from this bunch. Those who dare to critique the great Mark Always Right Kleiman and crew have been known to be smeared as aliar with a simple mind full of hatred and a half-witted man who can’t manage to be a decent human being. Classy group, this BOTEC cartel.

    But it’s no surprise they haven’t responded. Their analytical prowess and mad cipherin’ skilz are matched only by their intellectual cowardice. Over at their blog, the “Reality-Based” Community, registration is required to comment, comments are heavily moderated, and most critical responses do not seem to get past moderation. Those willing to tolerate such treatment with whom Mark tends to disagree regularly might find themselves treated as a less than equal member of the “community”, like this poor fellow Brett Bellmore near the bottom of the page:

    Finally: General note: Mark has banned me from making more than one comment per post, (Says he appreciates the diversity I bring to the site, but apparently not enough to let me comment on the same basis as anybody else.) so don’t expect a reply if you ask me something, in this or any of the threads.

    I doubt you’ll be surprised to learn that our distinguished gentleman from BOTEC thinks medical marijuana is a fraudulent sham. You got it right, Dr. MacRae; this dangerous group puts its greed and prejudices well before the needs of vulnerable patients, the rights of their fellow citizens, and their own responsibility to deliver honest and accurate analysis to the public that pays way too much for it.

  5. Jim, once again your analysis is absolutely correct – and in fact this was the reason a lot of medical outfits would not answer BOTEC’s survey (I refused) – the wrong questions were being asked with the wrong assumptions. NO other state will be following BOTEC’s advice – hard to fathom why we are. I really love reading your thoughts Jim! Muraco

    1. Thank-you, Dusty.
      I do not believe BOTEC to be full of idiots. They may be headed up by one, but the jury is still out on that.
      Indeed, BOTEC does appear to have done work of decent-to-good quality in the past.

      I do, however, stand firm in my assertion that the estimate that they produced in December of the size of the medical market for our Liquor and Cannabis Board was awfully horribly bad. It was horribly executed (seriously … check out the summary table of their results and see how many rows and/or columns add up to what they should add up to). More importantly, it introduced a “method” to deal with the complexity of a dynamic market in a state of rapid flux that is bullshit-wrong and has been replicated by the University of Washington Cannabis Law & Policy Project in it’s inaugural “research” publication recently made public by the LCB.

      Bullshit method, bullshit results.

      As I said in an e-mail to BOTEC back in January:

      In my opinion, BOTEC has harmed the people of Washington with this work. BOTEC has, also in my opinion, done a disservice to the WSLCB in providing work of this quality. Decisions based on this report are, at best, uninformed. I would take it as far as to suggest that such decisions will, actually, be mis-informed. (Note that I, also, read the disclaimer contained in your report. Good to see you engage counsel while communicating your analytic results.)

      In producing this report, I suspect that BOTEC has also hurt their future status in this industry.

      I then went on to suggest that this would hurt patients and that that was neither nice nor appropriate nor likely to be appreciated by the Cannabis community or by reasonable policy-makers and regulators around the planet.

      Personally, my respect for BOTEC (and more recently for the thing that leads them) has diminished.

Leave a Reply to Danielle Cancel reply

Your email address will not be published. Required fields are marked *