RSSTwitter vote-monitoring effort mainly sound and fury
By Angela Gunn | Published November 12, 2008, 3:41 PM
The ad hoc volunteer election-monitoring effort undertaken on Twitter last week had problems with imprecise methodology and random trolls, so much so that one researcher calls the compiled data "a mess."
A three-page paper from Bob Conrad, a Nevada-based PR person, blogger and doctoral student, looks at last Tuesday's Twitter Vote Report -- a volunteer effort to give voters a place to describe what was happening at the polls on election day. The project allowed observers to send reports in via Twitter, text or phone, with special apps available for the iPhone and Android.
The Twitter Vote Report coordination effort was pretty impressive, especially for a project that started about a month before the election itself, and the theory that crowd-sourcing works was powerfully proven Tuesday night by poll analysis site <1external href="http://www.fivethirtyeight.com/">FiveThirtyEight.com. But a lot depends on how you work that crowd, and Conrad's report (PDF available here) indicates that the gatekeepers in charge of deciding which reports were to be publicly posted were acting, for want of a better word, whimsically.
The gatekeepers approved which Tweets were posted to the Vote Report site. If necessary, they were allowed to add a hashtag that clarified the intent of the post -- for instance, #wait or #registration. They could also "dismiss" posts from showing up in the live stream of comments, especially if they were off-topic -- though deciding what constituted an off-topic post was mainly a gatekeeper judgment call. (Which explains how posts like "@DwayneH dude I love egg salad sandwiches, but liquor store is scary. downtown scarier, even. best of luck. #votereport" made it to the service.) "Dismissing" of data is perhaps useful in a public free-for-all system such as the project was conceived to be, but the thought does tend to make grown statisticians weepy.
Conrad's evaluation of the project found that not only was participation far too low (about 11,000 voters out of a potential 110 million-plus) to provide robust data, but that the ratio of relevant and irrelevant approved tweets was so bad (36.1% were relevant as per the stated rules for the project) that the only conclusion Conrad could draw is that "the data, and their criteria for acceptance, are essentially a mess."
And the users themselves, when they weren't improperly tagging their Tweets, were as whimsical as the gatekeepers. Conrad found that in his home state of Nevada, which had 72 participants, just three users accounted for over two-thirds of the published tweets.
Some participants weren't playing well at all; Twitter user Nico laughed that "Haha, @DwayneH got caught on trolling the #votereport." Add to that the variation in reportage -- the 0-to-100 rating system for "quality of experience" comes to mind -- and the stage was set for something much different than the precise, sabrmetrics-inspired breakdowns of fivethirtyeight.com.
"Had the Twitter Voter Report followed clearly specified criteria for posting reports and worked with a stronger chain of command in its organizational structure, it is likely many of these problems would not have occurred," Conrad said in an e-mailed statement.
It's unclear whether the project (or, for that matter, Twitter itself) will be a going concern by the time the 2010 and 2012 elections roll around. But on the whole, political junkies have to be pleased that the election itself went off more smoothly than did the Twitter-based effort to monitor it.
Now there's a surprise!
NOT! LOL!
"sound and fury"??? - how about just "noise".
But gee, it was on the Internet, it MUST be true!
Score: 0
Perhaps, though I'd dismiss the "on the Internet" snark with "Dude, did you SEE fivethirtyeight? Spectacular success, and only possible with Net technology."
I just finished another stats-related piece, concerning Google's new flu-tracking project. The paper they put together for Nature is a beautiful example of how online data can be used well in the hands of folk who understand and love statistical analysis. Ditto fivethirtyeight, as far as that goes.
But Twitter Vote Report, not so much. I think Twitter Vote Report was a useful idea for quick-fire reporting of trouble at the polls, had such trouble arisen, but *not* a data-collection device for deriving any great statistical truths. Nothing wrong with using on-the-spot tech to report trouble or even just conditions, but it's all anecdotal. What you *do* with that anecdotal information might be of interest, but this business of gatekeepers and "dismissing" data points is just silly. Either compile the data -- understanding that it's still a tiny, tiny pool of data and thus not likely to lead us to Big Truths -- or don't. Volunteer arbiters of what matters just muddy the water; they obviously don't kneecap the trolls effectively and there's no evidence that their efforts could have accomplished anything that wouldn't have been accomplished by improving usability.
Score: 0
Wait, you mean there is a distinction between anecdotal comments from an anything but representative pool segment versus a well formed analysis of a statistcally random and well-controlled segment? ;-)
Monitoring Twitter, MySpace, or any of the so-called social networking sites is not going to provide any sort of statistically accurate data representative of any complete pool.
Answers follow from the questions asked, and the problem with using Twitter didn't result due to an anomaly at step 39, 53, or 167.
The problem with an ill-formed fundamentally flawed question begins with the assumptions inherent in the question itself. And what we have here is a perfect example of such.
Score: 0
I was the victim of the votereport "data sweep" just because of this innocent tweet:
"Remember, Nov. 4 is official "hashtag your favorite mom joke day" Use any of the following: #votereport, #machine, #disenfranchise, #ballot"
Democracy? Ha.
Score: 0
At least they are using valid scientific criteria for their sample selection! LOL!
;-) ;-)
Score: 0