Executive Summary: (for the TL;DR crowd)
BOTEC’s recent estimates of the size of Washington’s Cannabis market are terrible, horrible, and rather gnarly. One might think that BOTEC would be embarrassed by the work they sold to the Washington State Liquor and Cannabis Board. Of more immediate importance, these BOTEC-generated magical estimates are, simply, way too small. Systematically way too small. This is important in that these under-estimates have already contributed to the too-small allocation of only 222 new retail access points to support the medical cannabis market. If you agree with my assessment of BOTEC’s work, I would suggest that you write to let both your state legislators and the WSLCB know that you do not believe they should be using (or repeating) the errors that BOTEC has supplied as finished work product. That finished work product is published on the WSLCB site here, in case you wish to follow some of the specific page references below.
This is my first HI-Blog post in the last 4 weeks. I’ve been busy transitioning HI-Blog so that it is now hosted under my new commercial website www. straightlineanalytics.biz. Attempts to go to the old HI-Blog site should re-direct to the blog page on SLA. The SLA site is basically complete, but note that I have turned off the “Shop” until later this week. This is where documents, service packages and tools will be made available for purchase. I just have to do some banking things in order to set that up, and I did not want to hold off any longer on blogging …. my lack of a productive outlet for my words here has resulted in me getting way too verbose on Danielle’s forum.
I expect to be posting at least 2-3 articles per week on HI-Blog going forward. With luck the pressure of production will lead to less verbosity (on average) going forward.
On to BOTEC – or how to make a living producing meaningless numbers.
The following is the distillation of the top 9 things I found horribly wrong with BOTEC’s work – coupled with a bit of related commentary.
My thanks to the colleagues that provided input on my initial drafts of this work.
Points of Critique on BOTEC’s Estimates of the Size of the Medical Market in Washington State
Released by the Washington State Liquor and Cannabis Board Dec 2015
Point #1) – Lack of Accuracy and Quality Control in BOTEC’s work
Point #2) – The use of DOLLARS to define the size of each market component is problematic
Point #3) – Multiplying the level of recreational sales that occurred in October 2015 by 12 is not a reasonable way to estimate the expected annual size of a market that is still in it’s infancy
Point #4) – The use of CURRENTLY OPEN DISPENSARIES as the fundamental proxy by which to “size” the medical Cannabis market is inappropriate in an environment where dispensaries have been the focus of increased enforcement and closure over the past year.
Point #5) – Having grossly underestimated the sizes of both the recreational and medical components of Washington’s Cannabis marketplace, BOTEC then estimated the size of the Illicit market. Given the apparently systematic underestimates of the other two market components (recreational and medical), the method by which BOTEC has chosen to estimate the size (in dollars) of the illicit market is largely irrelevant.
Point #6) – BOTEC seems to assume that “no relevant data exist” upon which to make certain key estimates (such as the price per gram at Retail for the recreational market) and thereby justifies less-than-optimal proxies with which to fill in the blanks.
Point #7) – BOTEC appears to use biased language in describing the “significance” of certain key intermediate results.
Point #8 –BOTEC appears to use Monte Carlo simulation in both appropriate and inappropriate ways They may believe that Monte Carlo will shed light that cuts through the uncertainties inherent in their methodology. If so, they would be wrong.
Point #9) –How can the work of an entity that apparently has no shame be trusted?
I found BOTEC’s work disturbing. I thought they were one of the “experts” in this field. I expected more of them.
I would imagine that a reasonable person might well expect them to be ashamed of their work and the negative impact it is poised to have on public health and safety in Washington..
If allowed to stand, the allocation of only 222 additional retail access points for Cannabis (be it recreational or medical) will cause harm to public health and safety and it will do so BEFORE July 1 of this year. Ironically, the emergency rule that proposed this allocation is accelerating down a path that will, predictably, in itself create a serious emergency that will have negative outcomes on human health.
I’d like to offer one specific and crucial piece of advice regarding any efforts that might arise to mitigate and/or avert this fully predictable impending (and real) emergency. That is that the way in which the Board chose to allocate the new access points in December REINFORCES the lack of access in areas suffering the impact of bans or moratoria. A good rule – a good process would, hopefully, mitigate or (at least) remain neutral regarding the impact of local obstructionism on the implementation of the Patient Protection Act.
Note to the WSLCB: Medicine is not beer is not wine is not whiskey is not tobacco.
It is my belief that it is neither the role nor the intention of the WSLCB to double-down on local obstructions to access. Particularly for stores with medical endorsements. That the allocation method used for the Dec 16 announcement did, effectively, double-down on obstructionism, is unfortunate. It may well have been a well-intentioned attempt to ensure that the much-needed new access points would at least exist SOMEWHERE in the state. Ban/Moratoria areas, after all DO have stores allocated — just not open or licensed, for the most part.
If the LCB reinforces local obstructionism today, that will contribute to ongoing compromise to medical access for the populace of affected areas. I hope they reconsider and allow for additional store allocations in affected areas. The amount that the local people need —- not the amount that the local leaders desire.
At a minimum, how about ensuring that the number of stores allocated is AT LEAST half (rounded up) of the number of retail pharmacies that exist in the area? That is an easy and quick empirically-based allocation that does not cost the people of Washington any consultant fees and reflects local pharmaceutical demand. It’s a way of letting the “market” tell you what it needs.
Regardless, the LCB and other Governmental agencies should stop using BOTEC’s recent analysis to inform their rule-making and policy-making efforts relating to Washington’s Cannabis marketplaces.
This, of course, suggests that the WSLCB should be trying to get some good estimates of the size of Washington’s Cannabis market(s).
Me stating this may come across as self-serving, as I have produced such estimates, and know how it SHOULD be done.
CASP might be able to help if the WSLCB finds they need help with this. They can certainly do a better job than BOTEC has done on this question.
Perhaps the WSLCB might think of dusting off the work that a group of us (CASP, myself & the CCSE) supplied to them (free of charge) last year which estimated how many incremental access points were needed to support the forced migration of Washington’s medical Cannabis infrastructure to the regulated and controlled world.
I would like to think, given how horribly bad BOTEC’s recent work is, that the WSLCB might assign some critical eyes to look over the prior work BOTEC has submitted to the State of Washington. Work, remember, that the State has used to design and implement the I-502 market and much of their legislative and regulatory decision-making regarding State-Legal Cannabis.
To summarize, it is my strong belief that the results of the BOTEC study are, essentially, meaningless.
They systematically under-estimate values commonly known to be higher.
Their results should not be used to inform or guide policy or business decisions.
Let’s try not to hurt people in 2016 if we can avoid it. Using BOTEC’s results (and even assuming them to be useable) will hurt people in 2016. Doing so will also hurt businesses — lots of businesses.
Underestimating the size of the medical market guides policy that causes patients harm.
Underestimating the size of the recreational market guides policy that harms the development of the recreational market.
Underestimating the size of the illicit market is something I expect will generally happen (for lots of reasons). Regardless, state policy should be to diminish the illicit market. I think most agree on that. To measure progress on illicit-market-reduction efforts, good estimates of the true (not dollar-based) sizes of the three market components is key – as is an understanding of the flows of patient (and product) between the three.
It is unfortunate that, in attempting to fulfill whatever terms were placed on their contract to estimate the sizes of this State’s Cannabis markets, BOTEC has failed to deliver meaningful information on any of these things.
One wonders how many of our tax-payer dollars they received for this work. Dollars that could have gone to help pay for education (or to get a better traceability solution).
Back Of The Envelope Calculations, indeed!
Last I saw, computers were better than envelopes for doing quantitative analytic work.