Is research bad because recruiting is hard, question

Let's lead off with some definitions because, yeah, that's how you kick off an exciting email!

Bad research: Research that does not meaningfully improve a specific class of decisions.

Recruiting: Getting people to participate in your research.

Y'all excited yet?

When it comes to research, it's a good idea to ask two questions and look at the relationship between the answers:

Q1: What makes research expensive and/or difficult?

Q2: What makes research useful?

Everything that makes research expensive/difficult but doesn't make it useful is an opportunity for leverage! Taking advantage of this leverage leads to what I call small-scale research (SSR).

Lots of what can make research expensive or difficult isn't actually needed to make it useful in our context -- that context being the indie developer, consultant, or small firm that wants to reduce the uncertainty of a consequential question.

Unfortunately for us, recruiting is not one of those un-needed things. Yes, there are some specific research designs that don't require recruiting. For an example, see RedMonk's Programming Language Rankings research. Search the aforelinked for the word "extract" to see a quick description of their no-recruiting-needed methodology, which relies on data scraping via the public APIs of GitHub and Stack Overflow. But most small-scale research designs do require that you to recruit participants. Those participants are real people with busy lives who can easily say "no thanks" in a hundred different ways (unless they are college students LOL).

Here's what a typical recruiting message looks like:

This is not a particularly strong or a particularly weak recruiting message. It's fine and normal. I responded to their invitation largely out of curiosity, honestly answered what are known as "screener questions", and got screened out because I didn't fit their criteria. Also fine and not unusual.

I don't think ProductReportCard.com is a household name, but if you're recruiting for research, being a household name helps. I don't have numbers to prove that claim, but try this thought experiment: think through how you would get 100 honest, thoughtful responses to a set of non-trivial questions you have (you might deliver them via a survey, interviews, or some other configuration). What would it really take to gather that much data?

Some of you will let this question haunt you a bit, and if you do I think you'll start wondering the same thing I led this email off with: Maybe much small-scale research is bad (again, bad == doesn't meaningfully improve a specific class of decisions) because it's hard to get the right data from the right people?

Let's extend the thought experiment. Let's say you must publish some research. Doesn't matter why, but there will be real bad consequences if you don't. But also, the enforcer of these bad consequences doesn't care what kind of research you ship, and whether it can meaningfully improve a specific class of decisions or not. Doesn't matter, just ship something that's an honest effort to answer your research question in a defensible way.

Two questions follow:

1. What research question would you try to answer?

2. What recruiting approach would you use to gather data to help answer your question?

I propose to you that if you're a normal, not-masochistic person, you would try to avoid those real bad consequences I mentioned above by making it easier to just ship some damn research, and -- you'll just have to trust me here for now -- since for any given question you usually can't make the recruiting part easier, you'll seek to make the research question easier to answer. And how do you do that?

You design a research question that is not totally trivial, is superficially interesting, but does not have the power to meaningfully improve a specific class of consequential business decisions. In other words, you would intentionally design bad research.

Indeed, there is no enforcer of real bad consequences that is forcing you to ship research and therefore incentivizing you to ship bad research. But there is a cultural worship of data and of those who clothe themselves in it. (This is not to dismiss the real power that skillfully and honestly-used data has to make things better!) And so there is an incentive to conceive and ship some research. It differentiates your firm! It helps you build authority! It can help you earn new reach and visibility!

Don't misread my wording here as mockery - it’s not. These are all legitimately good things that are worth pursuing. But you can see, I hope, how these common incentives line up to push me, you, and everyone who tries to execute small-scale research towards doing bad research, right?

I'll try to bring this home with an example. Which of the following 2 research questions would you rather recruit 100 participants for?

1. How do those who have purchased over $1MM in custom software projects find, vet, and negotiate with the vendor(s) they selected? Furthermore, how do those who have been involved in more than 5 such projects differ in their behavior from those who have been involved in less than 5?

2. What do custom software developers think about the future of $1MM custom software projects? Specifically, do they think that more, fewer, or the same number of $1MM custom software projects will be sold in the coming year?

This is an exaggerated example, but do you see the point I'm making? Can you kind of feel in your bones why much more research that looks like question #2 gets published than research that looks like question #1?

The only incentive that would push you away from question #2 and towards question #1 is if you really, REALLY wanted an answer to question #1!

A stylized but true 1-sentence takeaway from this: the research question you pursue dictates the recruiting approach you use, and the best lever we have to make recruiting easier is to dumb down the research question we pursue.

This is a decent place to wrap this. What question(s) did I leave you with?

I would really like a few of you to approach me about getting my guidance on a small-scale research project, or introduce me to any people you know who might have that need/interest. At least one person is publicly excited about one of my too-many writing projects -- The Small-Scale Research Guide -- but until that book is done, doing better research than example question #2 above is something you'll have to figure out for yourself or get my guidance on. I legit don't know anyone else who has the kind of interest, POV, and understanding of this subject that I do, but if you do, please intro me to them, I’d love to talk shop.

Spikily, -P