Here’s something important I’ve observed in my experience in working with data, and changing organizations with ideas: Great Analysts are always skeptical. Deeply so.
This was always true, of course. But, it has become mission critical over the last few years as the depth, breadth, quantity and every other dimension you could apply to data has simply exploded. There is too much data. There are too many tables/charts/”insights” being rammed down your throat. There has been an explosion of “experts.”
If you are not skeptical, you are going to die (from a professional perspective).
And, yet… You can’t be paralyzed by skepticism. At some point, you have to jump. Or, you are dead (again, professionally).
Let’s do this post in two pieces.
First, a plea to be skeptical, of everything and everybody, illustrated using an example from one of the most respected sources of data out there. Followed, by advice on getting to a decision rather than what happens to poor analysts: paralysis.
Second, as we are on the topic of great analysts, I want to share how to recognize that you might be one, from a macro perspective, and, if you are, or are not, what’s your value to your company.
Surely, you are intrigued!
#1A: Skepticism is your BFF.
I saw these two numbers presented the other day: 42% of online shoppers use video for pre-purchase research. 64% use YouTube to find products.
As soon as I heard them, I knew they were horse-manure.
The source of skepticism was simple, neither number is true for me – and I’m in a place, with people, who are the most connected people on the planet with more devices to do this type of research if it was true. I stood up. Did two things. I asked the 100 or so people in the room if either of these two numbers was even close to reflecting their reality, one person raised their hand. Then, I asked for the source of data. A 2014 AOL report and an online survey with n=600.
It was horse-manure.
Yet, they were being presented as facts on a tablet handed to Moses.
You might not yet have the experience to know if a number is true or not, perhaps you are evolving. But, if you actively invest in your education, awareness, being hungry to always want to dig just a little deeper, you’ll get there in no time.
For example, you might read NADA say this: “85% of customers made up their mind to purchase a vehicle before they left their house.” Your skepticism radar should go beep, Beep, BEep, BEEp, BEEP, and you should stop and listen to it. It does not matter how big NADA is and how many analysts they have – because accepting imprecise information will cause you to make career-limiting recommendations.
Here’s a great example of hopping on the skepticism train right away.
[Update: As Mauro Avello points out in his comment, this could be an April Fool’s joke by the team at The Economist. I did notice the date when I wrote this post, and read the comments and follow ups, and did not see anything that indicated to me that it was a joke. Hence it was used below. The lessons you’ll learn still stand, but please do be open to the possibility that this was just a joke.]
The ever wonderful data viz team at The Economist had an irresistible link: Ice Cream and IQ.
Hard not to click on that, right?
It is a short article containing a line chart plotting ice cream consumption on the x-axis and the mean score on PISA reading scale…
THE DATA TEAM (that’s who the article is credited to) go on very seriously to share that more ice cream eating might be the solution to poor student score. They dutifully compare the Aussies and the Finns, commend the Canadians and crap on the Peruvians.
So. You are the smart Analyst.
Your first skepticism flag should be: The title of the article says IQ, do PISA scores measure IQ? Quick Google search. They do not.
Your second skepticism flag should be: Look for things in the data-set that disprove the summary statement. Notice Hong Kong, Singapore and it’s neighbors have very high PISA scores, yet very low ice cream consumption.
Your third skepticism flag (for a smart Analyst, usually this is the first one) should be the perennial favorite: Correlation does not imply causation!
You pour over the article for signs that this simple rule is not broken. Is there anything that shows they looked into causation? No.
Giant red flag.
And at this point, a tiny part of you also died because you do so love THE DATA TEAM at the Economist.
To normal people (non-Analysts), this graph and article looks legit. After all this is a reputable site and it is a reputable team. Oh, and look there is a red line, what looks like a believable distribution, and a R-squared! Most normal people will take this as truth (and at least 67 of them will proceed to comment on the article and have fun).
You should not.
The thing that should go through your head is… Causation. What could cause this?
Here’s one hypothesis: People who really care about their kids educational accomplishments come from families that tend to have parents who are a little bit well off – middle class -, they can focus on the kids. These families usually reward educational accomplishments. The reward of choice tends to be ice cream!
Remember, it’s an hypothesis. We can go look for data. If it turns out to be true… It is not ice cream consumption that is the reason for the performance scores, it is the fact that families tend to have a certain income. Or, that they tend to have structured work time, which gives parents free time to focus on how their kids are doing vis-à-vis education.
There could be a number of other things. Weather. Number of women in the country. Longitude. Number of child workers. Crime. Anything honestly.
Look for causation. No causation means… data crime against humanity.
Let’s bring this baby home, one more example, this one a bit more fun.
There is an extremely tight correlation between the amount of US spending on science and suicides by hanging… r-squared of 0.997…
If you are with me thus far, you are screaming that there is no causal connection between the two!
And, you would be right. Spending more money on science (please let’s spend more!) will not result in more suicides. Though the two are as tightly correlated as any two things can be.
[The above graph is from Tyler Vigen. His website – and book – Spurious Correlations is wonderful. You can checkout many more correlations, and laugh and cry and laugh and cry. Start with the one about Nicolas Cage movies causing people to drown!]
Let’s look at another example, because it is fresh off the presses.
All of us travel and it is of immense interest to us as to which airline has high “quality ratings” when it comes to performance. Here are ratings that came out today…
Most press reports you’ll read about these performance ratings will talk breathlessly about the ranking and the movement up or down of a particular airline. What none of them do is talk about how this data is calculated. But, you the smart Analyst, will have your skepticism radar up and you’ll dig!
Your starting point can be the press release from the source. There is more drill down data there.
Then you’ll poke around to figure out how on time arrivals and departures are actually calculated, who sets the standards/formulas, what kind of control airlines have when it comes to setting their schedules, who decides what data gets reported and who audits it, and more lines of inquiry before you buy any of this.
When you dig in, you’ll earn that there is actually no standard for what the word delay means. Airlines are in full control of setting the duration of a flight. For example, my flights to JFK on United routinely arrive “on time,” even though they depart 20 mins late, because United adds a half hour to the “scheduled flight time.” That padding then controls what is reported (crap basically). Across airlines there is no standardization of how duration between two destinations should be set – so basically the ranking above is comparing apples and monkeys and asteroids. The airlines, like Frontier, that set the tightest durations, most accuracy, are being penalized in a way for good behavior.
And, this is just the start of the problem.
Very quickly you’ll realize, the rankings and the data above is basically garbage. If, at the end of the day the purpose of this data is to help you make a smarter decision, it is miserably failing to do that because of the above issues.
Skepticism. A good thing in an Analyst because you know what to use as a source of decisions, and what not to.
Most data you see in the real world won’t be as obviously wrong as you’ll see in the gems shared by Bad Fox Graphics. The example’s you’ll see will be more subtle, they will look like they make sense, they will come from sources you trust, from tools you use and even implemented yourself, etc. That’s when you need to be most vigilant of all to be a great Analyst.
Here are some techniques you can use:
1. Look to see if the conclusion (“insight”) expressed has anything to do with the data in front of you. This, honestly will only take you a couple minutes.
2. Here’s a great question: Where did the data come from? Tools, countries, people, devices, etc. Known gaps of what’s unknown (particularly relevant in digital data).
3. Another one that you’ll love: What types of bias might exist in the data? Sample bias? Sampling bias? What could cause it to be incomplete?
4. What principles you’ve learned already that might be broken by the analysis presented? Correlation/causation is the one we covered above.
5. Always, always, always ask this question: What assumptions were made in doing this analysis?
6. Your experience. You have a ton of it. Don’t let it go to waste.
7. (Added via Richard Hren’s comment below) Who gains or loses from this analysis? As with many things, follow the money, power, politics.
I’m sure there are others. Would you please help me expand this list by adding techniques you’ve learned to help bring your healthy skepticism forward by adding a comment below?
When you see a piece of data, from inside your company or from the outside, be skeptical in general. It is a good trait to have as an Analyst.
#1B: Skepticism should not paralyze you.
You are going to feel I’m going to run all of the above under a bus now. Please stick with me.
The real world is not perfect, and you are paid to help your company (non-profit or for-profit) make smarter decisions every day (hopefully). One important thing at play here? A decision has to be made.
Novice Analysts get so caught up in the skepticism that they become paralyzed because if you even lift the covers under digital analytics a tiny bit, the way data is collected with scare the bejesus out of you. Oh, and offline analytics? A million times worse. And, tiny data samples to boot!
Great analysts get good at one of the most critical elements of our jobs: Timeliness. The ability to deliver an insight, a specific recommendation, in a duration that it will have an impact on the business.
An educated mistake is better than no action at all.
Our job is to be skeptical, to dig and understand and poke and prod and to reject the outrageously wrong and if it is not outrageously wrong then to figure out how right it might be so that you can make an educated recommendation.
This post is from 2006: Data Quality Sucks, Let’s Just Get Over It You’ll learn the six step process you can use to overcome the paralysis.
Here’s a simple way to think about using your skepticism, but still making a decision.
If you were 100% certain about the data, you would immediately recommend to your company that they should start making plans to build a colony on the moon.
If you were 80% certain about the data, you could recommend that they shift the strategy with the International Space Station to start sending short visits to the moon.
If you were 40% certain about the data, you could still recommend a tripling of the investment in entities on earth that would study how to live on the moon.
If you were 20% certain about the data, you would go back to your team and figure out what strategies you all should put in place to get to at least 40% certainty.
That’s what I mean by being skeptical – your quest is to get a more concrete feel for where that certainty lies. By not being paralyzed by perfection, I mean making a decision that reflects that certainty because the business needs timely decision making.
We are on the topic of being great Analysts. So. Here’s a detour, and yet on theme extension of that idea.
#2: The Difference Between Knowledge-Insight-Wisdom.
As some of you know, I’m writing a twice-a-week short newsletter called The Marketing – Analytics Intersect. You can (should!) sign-up for it.
I often find that most people who have the title Analyst are essentially data collectors and data sharers with most of the value being added by them in the process is a tablefied or chartfied summary.
My newsletter on 29th March, shared a fantastic cartoon that exquisitely captured the difference between data – information – knowledge – insight – wisdom. It also added a layer, dare I say, wisdom by outlining the value of the job, salary and how quickly you can be replaced in the job.
Here’s that TMAI, in it’s entirety, I’ll pick the story up again on the other side…
Of all the cartoons related to data, and analysis, this one is my all time favorite…
Isn’t it incredible, it captures so much about the work we do in so little.
I love this cartoon because there are so many insights, :), to draw from it. Let me focus on one, how valuable you are to your company.
Information: You are a report creator, you fix code in emergencies. Value: Low. Salary: Lowish. Replacement: Easy.
Knowledge: You run a team of data pukers, you help meet divisional data needs, your team merges data sources. Value: Medium. Salary: Medium. Replacement: Takes two months.
Insight: You hold the Analyst title, most of the time you avoid being see as a data provider, you get invited to director-level business meetings. Value: High. Salary: High. Replacement: Hard, six to nine months.
Wisdom: You are an Analyst, but sit in a business team, d3js.org is your second home, you meet with the CMO every other week. Value: Priceless. Salary: High times 5. Replacement: Impossible’ish.
So, what job are you doing at your company? Information? Insight?
Is there anyone in your company in the analyst team, or the marketing team, whose explicit job it is to deliver wisdom?
Yes, you want to be in the Wisdom business. But, realize how hard it is to do. You have to be on a constant quest for self-improvement, and the most powerful skills you’ll bring to bear are your business savvy and not data-crunching prowess. Ironic, no?
Great Analysts solve for Wisdom. And, above and beyond what you read in the newsletter, you can see how both being skeptical and not being paralyzed helps you get to Wisdom faster.
One last bonus before we close… If you would like to get a sense for specific salaries, four key choices you have to make to have a fabulous analytics career AND how to get there… Here’s a post you’ll find to be of value: Web Analytics Career Guide: From Zero To Hero In Five Steps!
As always, it is your turn now.
Do you agree we are not skeptical enough about data floating around in our companies or the interwebs? What are strategies you use to fuel your skepticism? How do you torture the data you see/get? Is there something that works for you particularly well when it comes to solving for timeliness? Are you solving for Wisdom in your current job? Insights? What caused your career to leap from Data to Information to Knowledge faster?
Please share your wisdom, :), experiences, tips, tricks and lessons from the front lines via comments below.
PS: Topics covered in my last few newsletters: Should you analyze individuals? NO!, The Best Ecommerce Experience In The World, Formulate Your Life, The Very Best Metric: Email Marketing. You should sign-up!