You will optimize whatever metrics you choose to measure, which is why if you evaluate programmers on lines of code, you’ll end up with gigantic comment blocks, and if you evaluate them on bugs closed, you’ll be seeing a lot more initially-buggy code.A
So how do you measure something fuzzy like “built a generation of webmakers”?
Until we’ve installed mind-reading microchips in everyone’s brain (patent pending), we’re probably going to have to settle for indirect measurement: measuring side-effects, close approximations and proxies.
Here’s some examples of things we might be able to measure. Thanks to Ben Simon for brainstorming this stuff with me.
Total authorship “levels” on the web
Imagine that you had some reasonable way to figure out who authored a document on the web, and some reasonable statistic about the number of personas an average person has on the web. Now all you need is to run those numbers across every document, and you can collect a reasonable figure for the number of authors online.
There’s an extra step to make this figure even more compelling. Divide the web into several content “types”:
- Commented on someone else’s blog
- Created a wordpress blog and posted something
- Created a wordpress blog and manually changed the template
- Created an HTML page from scratch
- etc.
Post-event sampling
After attending one of “our” events, could we do a focus-group-style sampling of the people who leave the event and what they go on to do thereafter? I put “our” in quotation marks, because I don’t just mean Mozilla events, but actually any webmaking style event that is affiliated with us, belongs to our community, etc.
Measuring deltas of public participants
Some of our participants (like journalists) are going to have very public artifacts (ie: their articles) before and after their participation in a program like Open News. We could measure the difference in their output, or the complexity of elements that they use to see whether or not they learned anything and are putting those skills to use.
Drop-off in our tools
This might be the easiest thing to measure, though we’d need to couch the tool usage in some sort of opt-in measurement thingything, but when our tools are being used to teach these skills, are the “lessons” completed? If not, where does drop-off happen? etc.
Measure bottlenecks to making
Are you not making webpages because you don’t know how? Because you’re scared of breaking shit? Because you can’t see how it has anything to do with your life? Because you tried and failed? Unless we know where the bottlenecks are, it’s going to be hard for us to improve those bottlenecks. Ben suggested that we could do old-school phone surveys to get a general population wide metric. As an engineer who hates the telephone, this made me cry a little on the inside, but it’s not a bad idea, especially since if we want a “before” metric, we should be jumping on this train now.
Measure what we actually want to measure
This is an interesting point: why do we want to build a generation of webmakers? If we go up a little and tackle that “why” (forthcoming blog entry, I suspect…), then maybe that‘s what we should be measuring. For example, if we believe that by making things on the web, we’ll convince people that the open web should be protected, then let’s actually measure that.
So much food for thought!! I’m gonna end up with a brain tummyache at this rate…
“You will optimize whatever metrics you choose to measure, which is why if you evaluate programmers on lines of code, you’ll end up with gigantic comment blocks, and if you evaluate them on bugs closed, you’ll be seeing a lot more initially-buggy code.
So how do you measure something fuzzy like “built a generation of webmakers”?”
=> The examples you gave are quantitative measures. Maybe a something fuzzy had better being measured with “fuzzy metrics” (qualitative).
The idea of measuring the amount of content created by person is an interesting idea (regardless of the technical difficulties to find accurate enough numbers), but that would be in no way a measure of Mozilla’s success in its mission.
I can already say that at every second, the planet is becoming more web literate/web aker. People create more blogs, create more video channels, etc… The question is how much Mozilla is an influence on this progression and in what direction.
In this video (http://www.youtube.com/watch?v=ehDAP1OQ9Zw), a kid shows that he made iphone apps. Based on what is being described, no one can deny that this kid is a web maker. However, is it the kind of web making that Mozilla wants to promote?
“Post-event sampling”
=> The hardest part will certainly be to list and discriminate what are “our” events and what is not. Is FOSDEM one of our event?
I love “Measure bottlenecks to making” as well as the idea to do it
“why do we want to build a generation of webmakers?”
=> Probably the first time I hear/read this question since the “web maker” movement started at Mozilla. I’ll push the question even further: “is it even possible?”
A couple of centuries ago, few people knew how to write. At this time, there were few novel writers for that reason. Nowadays, almost everyone can write (in the developed-enough countries). But a generation of novel writers hasn’t popped up out of everyone being able to write.
Maybe building a generation of web maker is just as pointless as trying to build a generation of novel writers. Just a thought