Facts and Alt-Facts: How 2 Labs Report on Potency

MayDay Anarchists Funding our Government
May 2, 2017
Potency – as it should be?
May 10, 2017
Show all

Facts and Alt-Facts: How 2 Labs Report on Potency

In this post, I am going to spare you my interpretation.
I’m just going to show you two graphs (and a morphed combination of the two).

My expectation is that some of the readers of this post who take a minute or two to go through the homework I’m about to suggest will then take the time to share how the actual data in the graphs supports and/or opposes the image they formed in their heads by conducting the following mind experiment:

Please relax. If not contra-indicated by any of your medical conditions, take a deep breath or two and think about the following question:

If you took all of the flower samples tested for potency in the state and looked at the monthly average levels of potency across time, what do you think that curve would look like (and why)?

Would it be a flat line?
Would it be a line that tended to go up over time?
Would it be a line that tended to go down over time?
Would that little sucker bounce around all over the place?
Would it be a random walk, a parabola, a double-helix, or something that looks like a cistern?

What do you think it would do? What would it look like?

Take a minute … close your eyes (if you are not driving or doing other incompatible things) … think about it.

What would potency tend to do over time?

Consider the context in which these numbers were reported:
Growers dialing in their processes under the new constraints of a regulated grow.
Processors dialing in their trimming capabilities and building a relationship with their lab(s).
Labs dialing in their potency-assessment capabilities.
RJ Lee keeping the labs certified-and-kind-of-proficient-like.
Biotrash attempting to report upon the reported results correctly and completely.
Jim trying to clean and summarize the resulting data and make graphs from it.

What would potency tend to do over time?
Hint — Jim does not believe that Jim is the wildcard in this equation.

OK …. have you finished your homework?

If not … then go back up a few paragraphs and repeat.
If so … then please look at the following graphs and use the comment “thread” on this post to share your thoughts regarding how they jibe with the mental image(s) you formed doing your homework.

These two graphs represent the monthly levels of TOTAL CANNABINOIDS reported by the top-volume FRIENDLY lab and by the top-volume PRAGUE lab over the first 34-some months that they were engaged in testing flower samples.

The third graph superimposes the other two so that you can see their distinct character without having to flip up and down and up and down and up and down between multiple charts. Whiplash hurts, after all.

Let me know what you think.

I WILL NOT be responding to your comments until the weekend. Please feel free to respond to each other’s comments in the meanwhile. Remember to check the boxes so that you get updated when subsequent comments occur and/or new posts from me occur (I think you only have this option if you leave a comment).

If you really don’t like leaving your e-mail and the other stuff it asks you, make something up that is clearly a BS e-mail (e.g., FU4Asking@NoneofYourBusiness would be good). I don’t want self-identification and transparency to be a barrier to your response. I think I’m the only one that sees that stuff and I generally prefer to know who is commenting, but I’ll let that slide on this post.

I truly would love to hear what people think about these two very different curves.

Have fun. (If you click on the graphs, they get bigger).

————————————————————————————————————————————————————————————————

————————————————————————————————————————————————————————————————

Again … I’d love to hear what you think.

18 Comments

  1. Dana Luce says:

    Not a surprise.

    Also can you resend me the Email we discussed at the meeting? Thanks.

    • Jim MacRae says:

      Re-sent just a few minutes ago, Dana.

      It was good speaking with you and a number of other folk involved in PRAGUE Lab-testing after the LCB hearing yesterday on the Lab QA rules.

  2. Jedidiah Haney says:

    Very interesting stuff Jim. My take is that Confidence had some pretty high averages out the gate and I remember the industry scratching their heads when that was going on. Analytical 360, on the other hand, has displayed what my mind originally guessed when I did your little mental exercise above. Thanks for publishing this information. I appreciate your diligence in creating a transparent testing program.

    • Nick Mosely says:

      You’re absolutely right, Jedidiah. There was definitely confusion about cannabinoids as the industry took off. There probably always will be. Cannabis is complicated stuff! Pretty cool commodity we work with. That’s cool in a science way 😉

      While I can’t comment on the nature of cannabis testing prior to legalization — before standards and accreditation — what I can say with certainty is that Confidence Analytics and Analytical 360 have always tracked similar results since legalization. When you compare the two labs apples-to-apples you see we have very similar results. See our blog for more info on that. Confidence was actually lower than A360 in the first 5 months.

      What’s confusing about the graphs depicted here is that they don’t compare the two labs apples-to-apples. These are graphs of a field in traceability called “Total”, which used to be a point of contention among the labs. Mike Steenhout, formerly of the LCB, originally told us it was meant to mean “Total Cannabinoid” as in “all the measurable cannabinoids”. Analytical 360 interpreted it to mean “maximum potential activated THC”, or “THCmax” as Jim likes to call it (THCmax is a good name for it, I’m with Jim on that one). Other labs interpreted it even differently.

      In spring of 2015 the labs all met to discuss establishing a consistent meaning to biotrack’s ambiguous “Total”. We agreed on “THCmax” and we communicated that to the board. The LCB made a rule announcement that by July 1, 2015 all labs would report “Total” as “THCmax”. That’s what the graphs are depicting, a definition’s implementation. Regardless of how you define “Total”, Confidence and A360 have always measured THC and THCA with parity.

      Useable marijuana is required to be labeled with THCmax and CBDmax. That has not changed. Total Cannabinoid has always been optional, and is a meaningful measure.

      I highly recommend visiting CASP’s article on the subject. Dominic has some great ideas.

    • Jim MacRae says:

      Thank-you, Jed.

      While the industry was scratching it’s head, I was doing the same trying to figure out what the folks in Redmond were reporting.

      Those were good old days. Smaller samples, lower prices, higher numbers, very few QA fails …. what’s not to like about that?

      Remember … those were the days when dispensaries and patients still, for the most part, thrived.


  3. Thank-you, CONfident Scientist, for bringing a smile to my morning.

    Your accusatory comment evoked in me the image of one of your more vocal PR staff sitting in a nice hotel lobby somewhere forming the words that you share below while looking at some pretty pictures in a magazine.

    But … that was a different little Nikki.

    Readers: I am not big on selective censorship, so I’m letting CONfident’s comment through unedited. As with all “thought experiments” that readers share before the end of THIS WEEKEND (ending midnight Sunday May 7extended to Monday evening, May 8), I will put off responding to your comments until Monday evening.

    PLEASE COMMENT on this post (or on it’s comments). A degree of engagement on the part of my readers will show me that my efforts on this topic have some value and meaning to you. Share how your thought experiment was or was not different than what my allegedly massaged data showed you. There are no wrong answers when opinions are what are being solicited.

    If you want to get all analytic and scientific and technical …. that’s great. If you want to make a detailed chart that shows apples, when the charts that are in my article show oranges, that’s fine as well. Just try to be honest in making and supporting any argument you make. If you want to express your thoughts in a poem or through interpretive dance, that would be wonderful. If you do either of the latter two well, I might even buy you a drink.

    FULL DISCLOSURE from Dr. Jim:
    The only massage I made to CONfident’s comment below is that I removed the link to his/her corporate website and replaced it with a link to a nice commentary (and proposed solution) that Dr. Dominic Corva recently shared on the CASP website blog. I removed CONfident’s original link as I do not want to route any traffic her/his way and I believe that your chances of catching worms may be higher with such a link than with the one I subbed in.

    I include the text and graphic that were referred to in CONfident’s original link below their comment, so that you can see the materials that support their assertion without risking a visit to their website.

    Support CASP with a donation if you can. Dr. Corva does good work and I am quite happy to endorse CASP and what it does.

    *************************************************************************************************
    ORIGINAL COMMENT from CONfident Scientist:

    Alt-Facts, indeed.
    Jim,

    It seems to me like you’ve been massaging the data before you post it. Here’s a link to our actual flower results (generated using the same dataset you supposedly have). As you can see, our running average for flower has never topped 20%. You’re using a publically available dataset to do your analyses, so you should expect people to call you out when you present such alt-facts as these.

    CASP SILVER BULLET LINK

    Nice try.

    ************************************************************************
    Materials referenced in the original link CONfident was kind enough to attempt to share:

    Flower Potency Over time
    2017-05-04

    This chart depicts the THC Total values for all flower samples submitted to Confidence Analytics and reported to the WSLCB’s seed-to-sale traceability system to date. As you can see, it’s pretty steady at 18% average, but there’s always a slight dip in the late fall season. Typically, the central 50% of the distribution is between 15 and 20%.

    CONfident Scientist’s Pretty Graph

    • Jim, I appreciate that you linked to the CASP article. Our CSO is the first commenter. Dominic’s idea is solid.

      As I read more into your post here, I see what’s going on now. This isn’t a 1 to 1 comparison you’re making. For the first 12 months you’re graphing Total Cannabinoids at Confidence against Total THC at A360. That’s not a direct comparison.

      Here’s a pro tip: the traceability dataset you have includes columns for THC and THCA. If you combine those columns with the following formula, then you can make a direct comparison between labs. If you do, you’ll find these two leading labs are very comparable to one another.

      THC + (0.877 * THCA)

      Check out our blog to see what a direct comparison between the labs looks like.

  4. Steve W says:

    Well, Jim given the graph provided by confidence ( and assuming it to be correct) I’d say you owe those guys an apology. Their distribution seems to be quite reasonable and is pretty close to other graphs of large numbers of samples I have seen that also manifest a bell curve distribution like theirs would if the element of time was removed.
    Nothing nefarious from the look of the data they provided. It speaks well that you provided a forum for them to respond and, hopefully, they appreciate you doing so.
    It seems to me that to draw any conclusions one needs more data points as Con Scien provided.
    I am a bit surprised that there is not more of a trend in THC levels going up during outdoor harvest testing in the fall and early winter with a slight decline into summer as I would have guessed the outdoor tests would trend higher than indoor. We are in the early stages of all this and some trends may be masked by the random addition of new grows, new strains and improved techniques as you mentioned as modified by invasions of russet mites, marauding goats or unknown unknowns!

    • Nick Mosely says:

      Thanks for the positive comment, Steve. It’s easy to stand outside and throw rocks, harder to be inside and actually deal with the issues. Yelling about things gets attention, working on them gets solutions. That’s why Dominic Corva’s approach to the whole thing is admirable. There’s a guy who sees a problem and brainstorms a solution. Kind regards.

    • Jim MacRae says:

      Thank-you, Steve.

      First … I just removed a handful of subsequent comments that came from CONfident Scientist and from Nick of CONfidence …. as I don’t appreciate the way they are trying to re-frame things. I’ll release those at the end of the weekend, but I will not allow the intentional misuse of my site to spread what is, effectively, intellectual dishonesty.

      The graph from CONfidence is not “correct”. The two graphs in my post report TOTAL, which is one of the original 4 (now 5) mandated Cannabinoid numbers that are reported when flower is tested (currently, THC, THCA, CBD, CBDA and TOTAL are reported). In the old days, CBDA was not broken out in the traceability data.

      What CONfident Scientist posted was a graph of THCmax (totally decarbed THC …. the one that has the .877 in the formula). THCmax is not reported directly in the traceability system … one has to calculate it.

      In doing so, CONfident appears to be attempting to re-frame the discussion away from the focus of my original post. I reported on apples (TOTAL) and they responded with a graph of oranges (THCmax).

      You may all feel free to speculate as to why they might have chosen to do this. After speculating, you may want to take a look at the video I posted last year that shows lab-by-lab inflation of the reporting of TOTAL that was going on until the summer of 2015 (when the LCB put their foot down). That is available here: INFLATION VIDEO.

      In that post, CONfidence Analytics is lab N …. pay particular attention to how their reporting differs from that of other labs … and how, when most of the labs go to zero inflation later in the video, Lab N continues to inflate for about 5-6 weeks. The new “rule” that the LCB put down applied to everyone … it just seems to have applied to CONfidence Analytics later than it did for the other labs.

      The most obvious intellectually dishonest thing that CONfident chose to do in her/his graph is to limit the upper end to 40%, thereby hiding all of the numbers that they have reported that would be over 40% THCmax (I believe that you are aware that such numbers exist Steve). That is “massaging” the data … or, more specifically, massaging the graph to obfuscate an inconvenient fact.

      The other thing they did that is not unambiguously bad, but is a bit odd, is to report their curated subset of the data as a moving 1000-point average. I have no problem with moving averages per se (I use them frequently), but using a moving average with a fixed number of data points per averaged point tends to make the time-window being reflected in each point quite different (early in her/his curve, each point represents a few months of data …. further to the right, each point represents a smaller window of time). Not a bad thing, but potentially dangerous when interpreting the moving average.

      The thing I don’t like about her/his use of a moving average in this case is that it tends to obfuscate (smooth over) changes that occur over time. It is EXACTLY the changes that have occurred in CONfidence Analytics reported TOTAL numbers over time (relative to those displayed by a “good” lab such as Analytical 360) that are the main point of my original post. One of the more interesting things (for example) that is, effectively, hidden by her/his moving average is the pattern one can observe in the very left-most portion of the graph … where CONfidence initially reported TOTAL numbers that were much higher than what they subsequently reported and quite a bit higher than what their initial competitor (Analytical 360) reported during the first month of the market..

      Should I get around to writing a post on UNFAIR BUSINESS PRACTICES (or should I get around to filing an unfair business practices complaint with the appropriate agency), I’ll cover what happened to CONfidence’s share of the testing market in the month immediately following that initial “bump” in reported TOTAL that CONfidence chose to report.

      I’ll give you a hint — in the first month, Analytical 360 had a higher share than CONfidence. In the second month, CONfidence had achieved a dominant share of the testing market. This was, of course, following the little bump in higher numbers they were able to report in month 1 (which may have been inflated by the loosey-goosey way in which CONfidence was playing with the TOTAL number for the first year or so of the market). I don’t know if it is cause-and-effect, but the market has clearly responded well to the reporting of higher numbers — particularly to the reporting of numbers that are higher than the “truth” would suggest.

      It’s rather like the “butterfly effect”. Little changes initially can have large ramifications downstream.

      Butterflies, like unicorns and rainbows, are nice things. It is a shame to see them so misused and abused.

  5. Adah says:

    Jim,

    I remember months ago explaining to you the facts – That “total” in the state traceability system was open for interpretation when the industry was in its infancy, and indeed WAS interpreted in several different ways by several different labs. By using the “total” field YOU are comparing apples to oranges (and possibly bananas and mangoes too, because as I said, there were several schools of thought as to what “total” means).

    Nick, thank you for providing a graph that compares apples to apples.

  6. Sam Smith says:

    Something about “playing by the same rules differently”?

  7. Amy Wells says:

    I have heard that the high result labs were refusing low volume, (hence lower revenue) customers. Thank you for airing the data, Jim!!

    • Jim MacRae says:

      Interesting, Amy. Thank-you for sharing.

      I have, in some of my weaker moments, speculated that the cannabinoid numbers and (numbers of fails) being reported by labs might, in some way, be “correlated” with the number of tests a given wholesaler does with a given lab. Lots of putative reasons for this (if it actually existed beyond my speculation) including “bigger farms are better”, “we are more Friendly to our larger and/or more influential customers” and “random chance, you data-massaging ingrate”.

      Thanks again, Amy.

  8. Eugene Flynn says:

    My answer to your initial question was that I’d expect a steady incline during these early years, with a leveling off maybe another few years off.

    Leaving imperfections or skullduggery in lab testing aside, there are at least a few factors that might drive up overall potency levels. First, the market has shown a strong preference for high THC potencies, in large measure to the exclusion of other attributes, e.g., terpenes and other cannabinoids. Right or wrong, high potency is perhaps the number-two consumer consideration, right after price point. This has led producers to grow more of the most potent strains, while marginalizing or even eliminating lower potency strains. So a pronounced trend in strain selection based on potency would naturally be reflected in overall test results.

    It is possible that some unscrupulous processors might resort to dusting their flower with keif or another additive, legal or illegal, to boost potency. If competitive pressures compelled an increasing number of growers to game the test results in a similar way, this could push an upward trend in potency levels.

    If what is being measured is not just, say, THC and CBD, but includes all cannabinoids, then an increase in the overall potency test results could result simply from more labs testing for more types of cannabinoids over time.

    I’m not asserting that these are all real market pressures, I’m just suggesting they may be possible. And in any event, I’m not surprised that the test results of the “objective lab” do in fact show a rising average potency level as I had predicted.

    As to the conclusions to be drawn from the varied results between these two labs, that is above my pay grade and I will defer.

    Cheers,

    Gene

    • Jim MacRae says:

      Thank-you, Gene.
      I appreciate your thoughtful, well-articulated response.

      In discussions I have had with growers, most have agreed that a gradual increase in average cannabinoid levels would be expected over time, with a leveling-off at some point. I’ve had some discussions where the opinion was that the levels would be expected to be flat (e.g., no good reason for them to be going up or down). I have not yet heard anyone advocate that reality would be reflected in average cannabinoid levels going down over time.

      2 things on your comment –

      First, I appreciate your discussion on how the existing market pressures may be leading to strain-selection with a bias toward higher-yielding (re: cannabinoids) plants. Probably worth looking into, but what you suggest certainly makes sense. I think that growers dialing in their processes (particularly since they are now growing under the new constraints of the regulated system) would tend to also contribute to an expected up-trend over time.

      Second … you are correct about how the higher numbers shown by the Friendly lab (vs the PRAGUE Lab) are due in part to the fact that they chose to add additional cannabinoids into how they reported “Total” in the traceability system. Until May 11, 2015 there was apparently some ambiguity in the rule that stated what should be reported in that field. Although THC, THCA and “CBD” were required, the TOTAL field was not explicitly defined as being a total derived from these three values.

      Many of the Labs (possibly being sensitive to differences in how HPLC vs Flame-driven Mass Spectrometry express THC and THCA) chose to utilize THCmax (fully-decarbed “THC” = THC + (.877 x THCA)). They did so, presumably, because that number probably best corresponds to the “% Ethanol” labelling that folks are familiar with in alcohol potency reporting. Seems consumer-friendly to me (but it yields the lowest number of any measure that the labs seem to have been reporting as TOTAL prior to May 11 2015). In the previous sentence, I meant “friendly” in the more traditional sense of the word than I have been using in relation to the labs on HI-Blog.

      Some labs appeared to have been adding THC and THCA without discounting the THCA —it is hard to see why they would do that, but it yields a higher number than does THCmax.

      Some labs were using one of the above methods, but throwing the CBD number into the mix, as well (one can’t tell if they had done the proper .877 thing to get CBDmax or not).

      Finally, at least one lab (the Friendly one in this post) chose to add in different cannabinoids above and beyond those mandated for testing. From the shape of their curves between July ’14 and Dec ’15, the data suggest that their “Total” was further enhanced by the addition of more and more cannabinoids over time. I can’t confirm that …. perhaps CONfident Scientist, Nick, or Adah would be kind enough to shed some light on whether or not that was the case. If it was the case, why would they change the metric they were reporting from month-to-month? One has to be quite confident as a scientist to do something like that and still call oneself a Scientist.

      Thanks again, Gene.

      I’m now going to “approve” Nick’s two additional posts, CONfident’s additional one, and Adah’s.

      I held these in queue, as I did not think they added anything to the original question of “share what you think about the differences/similarities of these two graphs and what you’d expect to see over time”. They seemed more like an attempt at re-framing or deflection (or bamboozlement). I’ll let my readers be the judge of that. In any case, I will not be directly responding to these 4 comments.

      I will, however, remind folks that Adah’s reference to his/her earlier efforts to explain the facts to me were embedded in her/his comments on my “If a Picture is Worth a Thousand Words” HI-Blog post showing the video of how “TOTAL” was being inflated (vs. THCmax) over time by some of the labs.

      I’ll be following up today with a brief HI-Blog post highlighting what I find most interesting about how these two curves differ over time. Hint: It has a good deal to do with external events and how one curve seems to pivot during key periods of change (like media attention, regulatory attention, and/or mis-guided blog posts).

  9. […] follow-up to my recent post “Facts and Alt-Facts: How 2 labs Report on Potency” serves primarily as a response to those readers and interested parties that took the time to […]

Leave a Reply

Your email address will not be published. Required fields are marked *