The Problem of Offline Data



image

We love data. But when you run an online/offline startup gathering data can be a challenge.

Grouper, for the uninitiated, sets up drinks between two groups of friends: 3 guys and 3 girls. The experience starts online, but the real magic happens offline.

Some of our most important metrics, therefore, are difficult to measure directly. What happens after the groups meet each other? Do they exchange phone numbers? Do they become friends? Do they hook up? Date? Join each other’s kickball teams?

What you say vs. What you do

Actions speak louder than words. This is especially true in analytics. I don’t care if a user says they like a feature; I care if they actually use it (as evidenced by actions: clicks, log-ins, etc.). For purely online startups this is easy to do. For us, it’s harder.

Questionnaires

To figure out what actually happens offline we really have two options:

Ask (but what users say is never as good as watching what they do) or Observe what they do indirectly (because short of using video cameras… not something we’re planning on, by the way… we can’t directly know what they’re doing outside of our site)

We do ask. The morning after your Grouper we send you a short questionnaire. It includes questions like, “rate your chemistry with the other Grouper on a scale from 1 to 5.” The feedback is useful to a certain extent: it helps us refine our matching process, weed out bad bars, etc., but it has its limits.

If a certain match was a “4” out of 5, what does that actually mean? What we really want to know are the second order effects: are they going to see each other again? Do they actually see each other again?

We’ve experimented with some of these questions more directly, but what we found is that the more future-oriented they are (e.g. “will you… see them again?”, etc.) and/or the more personal they are (e.g. “do you think you’ll… date him?”) the less reliable the answers. For instance, we used to ask “did you find a potential date?”, but we realized that most of the people who actually ended up dating (and e-mailed us randomly to tell us so) actually answered “no” to this question after their Groupers. Go figure.

We’re going to try checking in with people later (e.g. 4 weeks later ask “so, did you end up seeing any of them again?”), but we suspect the response rates will be fairly low and could potentially suffer from the same issues with getting people to answer more personal questions.

Measuring “actions”

On the other end of the spectrum from self-reported feedback are actions that are highly telling but very difficult to track. For instance, one day we received this email:

Hey guys,

I don’t know how else to say this to you, but I had to kick out two of your customers last night because I discovered them having intercourse in one of our bathrooms.

I’m not exactly sure what type of business you’re running, but…

We never hear it directly from our members. It’s always the managers who drop us a note.

In terms of saying vs. ummm doing, this is a pretty strong signal, but it’s virtually impossible to measure directly. We also can’t really ask people. And even if we did (#awkward) they probably wouldn’t tell us.

The same goes for related questions like do people see each other again, be best friends, date, etc. We have some ideas at getting at this data that we’re going to experiment with, but in short, these things are extremely hard to track directly.

Indirect Actions (e.g. do they become Facebook friends?)

The most reliable way to measure offline data we’ve found is looking at indirect actions (i.e. things people do (instead of say) that shed light on a more important action that we’re curious about).

A great example of this is looking at whether or not people who meet on a Grouper become Facebook friends afterwards. We suspect that this is an okay proxy for whether or not people will see each other again.

On the plus side, this is something we can measure and it’s even quantifiable. On the down side, it doesn’t necessarily mean anything (e.g. you’re not guaranteed to see other again) and, also, Facebook friending is becoming less cool by the minute. We’ve found, anecdotally, that people are something like 5X more likely to exchange phone numbers than Facebook friend requests.

So this happens more often than questionable activity in restrooms and is more conclusive than a questionnaire answer, but it happens less frequently than people fill out questionnaires and means less than bathroom activities (to most people).

Future experiments

We’re constantly experimenting with new ways to gather offline data and make Grouper better. Any ideas? In the mean time, you should go on a Grouper. I don’t care if you think it sounds cool; I care if you go. After all, actions speak louder than words.

- Michael, CEO of Grouper

Reserve 3 on 3 drinks with Grouper here: www.joingrouper.com

blog comments powered by Disqus