User talk:ErrantX/Essays/RFA Study

< BACK to study page

This is excellent stuff :o) Pesky (talk) 05:38, 30 March 2011 (UTC)

Pesky's Thoughts

Who/when to ask: I would say one for immediately after, one during (particularly if it seems to have got heated / unpleasant), and one after a cool-off period.

Real Life Effects:

• can you give a parallel to something that has actually affected you in real life?
• Can you give examples of the effects? (maybe this should be a 'tick all boxes that apply' question, with the option to add 'other' (insomnia, loss of appetite, depression, anxiety, poorer judgement than usual, irritability, relationship problems, nightmares, etc. etc.)

On-Wiki Effects: We can add more examples in here, too. Check boxes again?

About Your RfA: Further questions:

• What / who did you feel was most unfair about the process? (checkboxes)

• False accusations (and did it generate a free-for-all pile-on?
• Insignificant stuff being blown out of all proportion / taken out of context?
• Ancient sins being dredged up and treated as current?
• Your 'personal style' being attacked, rather than your abilities?
• (other)

• How do you feel your RfA could have been improved?

Since RfA: More checkboxes ...

• Have you stopped interacting with people you worked with before?
• Have other people treated you differently from before?
• Do you feel your RfA has made people view you differently?
• Have you felt tempted to resign altogether?
• Or maybe just edit as an IP to be less visible?
• (other) - name it/them

Pesky (talk) 05:38, 30 March 2011 (UTC)

Comment from Kudpung

This an excellent idea and will greatly support the assumptions we're working on with our goals for reform to be proposed by User:Kudpung/RfA reform|this RfA reform task-force. Similar polls have been tried in the past but perhaps this time it will get response. It needs to be done quickly though. --Kudpung กุดผึ้ง (talk) 08:03, 30 March 2011 (UTC)

Thoughts

It's a great idea. I do think it should be given to the candidate very soon after and preferable with a spiel that they are still a valued member of the community, adminship isn't for everyone, you can always try again etc.

I wonder if any of the following questions would be a good idea

• Do you agree with the communities concerns?
• Was the feedback constructive?
• Do you think you will run for adminship again?
• Do you have a clear idea how to carry on as a member of this community?
• What do you plan to do now?

The great thing about the survey is that it confirms we still value the editor's opinion, and that should be reinforced with a welcome blurb. Worm^TT · (talk) 08:14, 30 March 2011 (UTC)

I think the poll should be targeted at every RfA candidate over the last 12 months - the feedback would be invaluable to the project we're working on at here. Kudpung กุดผึ้ง (talk) 08:19, 30 March 2011 (UTC)

Worm; my thinking was that initially such a survey could be used as Kudpung suggests - to inform the task force he has going by surveying a wider array of people. Then in the future, yes, we could have some sort of ongoing process to get feedback. --Errant ^(chat!) 08:23, 30 March 2011 (UTC)

Sounds sensible to me - I'll adjust my perceptions accordingly ;) Worm^TT · (talk) 08:27, 30 March 2011 (UTC)

Trial questionnaire run

I think the poll should be carried out anonymously on a check-box and answer field form, we have developers for that. A bot could send the form to all admin candidates over the past 12 months, whether successful, failed, or withdrawn, but not to NOTNOW closures. I don't mind my answers here being public on this trial run. My RfA was very recent. The answers are true. I hope this feedback helps.

Trial run

Proposed survey questions Real Life Effects Did your RFA cause real life stress? Yes. If Yes, how much: (a little/not much/some/a lot/severe) a lot. If Yes, did it cause more stress than you have previously experienced due to Wikipedia? (Yes/Same/No) Yes. If you are happy to do so please give some details of how real: Having to keep cool in the face of some of the most contemptuous commenting I have ever been involved with on Wikipedia. On-Wiki Effects During the RFA process did your on-wiki editing activities change? (Yes/No) Yes If Yes, did they: increase/decrease: stayed the same but obviously with a shift of focus to the RfA If Yes, did you do more, less or the same amount of your usual activities (i.e. article work, anti-vandalism)? Yes/No: No 'Yes/No' answer possible to this question. Continued doing some less stressful activities such as NPP. Positive or Negative process? Overall, a negative experience About the RFA Was your RFA successful? Yes How many previous RFA's did you have? None What length of time elapsed between your previous two RFA's? N/A (to answer the following questions you could link to the RFA itself and let us fill in the detail) Were you nominated or self-nominated? Nominated and co-nominated. Both noms were admins. How many edits did you have at the time? around 28,000 In article space? 35% : about 9,000 How many good or featured articles did you have at the time? 5 GA Since RFA Has your editing changed since RFA? Yes In a positive or negative way? Positive If negative explain why: Real Life Effects: can you give a parallel to something that has actually affected you in real life? Yes. Nasty things about me put on social networking sites by people I have had to fire. (I have never subscribed to a social networking site. I think they are evil.) Can you give examples of the effects? (maybe this should be a 'tick all boxes that apply' question, with the option to add 'other' (insomnia, loss of appetite, depression, anxiety, poorer judgement than usual, irritability, relationship problems, nightmares, etc. etc.): Insomnia, because I live in the opposite time zone (east Asia). Gritting my teeth in the face of adversary, pacing round the house, with my wife not understanding what was going on. Overwhelmed with joy when editors came to my defence and embracing my wife with her not understanding what was going on. Generally displaying rare emotions at home that are not to be expected from admins - but RfA is a very personal thing. On-Wiki Effects: We can add more examples in here, too. Check boxes again? About Your RfA: Further questions: What / who did you feel was most unfair about the process? (checkboxes) False accusations (and did it generate a free-for-all pile-on)?: Yes, and Yes. Insignificant stuff being blown out of all proportion / taken out of context? Yes, and Yes. Ancient sins being dredged up and treated as current?: Yes Your 'personal style' being attacked, rather than your abilities?: Yes (other) How do you feel your RfA could have been improved? All the reasons in my project at User:Kudpung/RfA reform Since RfA: More checkboxes ... Have you stopped interacting with people you worked with before? No. But I have reduced my work in some Wikipedia areas that I had a great interest for. Have other people treated you differently from before? Not that I have noticed Do you feel your RfA has made people view you differently? Not that I have noticed Have you felt tempted to resign altogether? While the RfA was going very badly, I had considered retiring (without a speech) if it failed, but in hindsight I would probably not have done. Or maybe just edit as an IP to be less visible? No (other) - name it/them Do you agree with the communities concern: (Question too vague) Was the feedback constructive? No, not really. Nothing I didn't know already. Do you think you will run for adminship again? N/A Do you have a clear idea how to carry on as a member of this community? Yes What do you plan to do now? I have shifted my focus slightly towards more involvement in policy and process improvements Additional possible questions: How long had you been a regular editor to Wikipedia before your RfA?: 5 years How many times had you voted on RfA before your own RfA? around 100 How many times have you voted on RfA since your own RfA? around 12 (RfA was 4 weeks ago) Did you take part in discussions at WT:RfA before your RfA? Yes Did you read any RfA advice/instruction pages such as Wikipedia:Requests for adminship before your RfA? Yes Did you read any RfA essays before your RfA? Yes Did you read any RfA user essays before your RfA? Yes Did you read any users' RfA criteria essays before your RfA? Yes What was the total number of support+oppose+neutral votes on the final tally? 137 Do you think enough people turned up to vote? Yes How many additional questions were posed on your RfA in the question section (count compound questions as separate questions): 16 What do you think would have been a reasonable maximum number? 10 Would you have preferred for the additional questions to be posed, and answered by you before the voting starts? Did you find the questions posed by voters reasonable? Most of them Did you find any question(s) totally irresponsible, impertinent, or completely irrelevant to the RfA process? Yes Did you consult your nominator or another editor off-Wiki for help in answering the questions? No Do you think response to additional questions should be optional? Yes Would it have been helpful to you if the bureaucrat had provided a closing summary/rationale? No particulary, the RfA closed with a clear majority for support If your RfA had failed, would you have considered running again?: probably not If your RfA failed, would you consider running again?: N/A Additional questons I missed or that were added later Did any of the following occur: False accusations or assumption of bad faith: Yes Insignificant (in your opinion) behaviour/action being given undue prominence: Yes Historical (> 6 Months) indiscretions (blocks etc.) being brought up as current: No (I have never been blocked or warned for anything). Personal attacks (i.e. on your character): Yes, quite rudely. I would have placed a civility warning on such a user's talk page without hesitation.

Kudpung กุดผึ้ง (talk) 11:02, 30 March 2011 (UTC)

Polling !voters

This would be more difficult. Of the 207 or so RfA over the last 12 months, this could run into possible 1,000s. Apart from the 50 or so 'regular' voters, many are one-time voters. However, We're working on getting a statistical voter profile made for the same period. --Kudpung กุดผึ้ง (talk) 11:25, 30 March 2011 (UTC)

What do voters check?

I like the idea of surveying the voters, but would suggest a few more questions:

1. Which of the following do you check before deciding to !Vote?
   1. The candidates contributions
   2. The candidate's deleted contributions
   3. The candidate's block log
   4. The candidate's talkpage and its history
   5. The candidate's userpages
   6. The Nomination and acceptance section in the RFA
   7. The question section in the RFA
   8. The Support section in the RFA
   9. The Oppose section in the RFA
   10. The Neutral section in the RFA
   11. The Talkpage of the RFA
2. Before you !vote in an RFA, how long do you spend reviewing the candidate and the discussion in that RFA.
   1. Less than ten minutes
   2. Ten minutes to half an hour
   3. Half an hour to an hour
   4. More than an hour
3. When you !vote in an RFA do you watchlist it and follow the subsequent debate?
4. Have you ever changed your !vote as a result of other things emerging in the RFA?
5. Have you ever changed your !vote as a result of responses to your voting rationale.

ϢereSpielChequers 12:51, 30 March 2011 (UTC)

Also, how about a more general - "what one thing do you look for most in a candiate?" I like this idea, but I'm not sure it covers everything without letting the voter have a general place to talk. Worm^TT · (talk) 13:21, 30 March 2011 (UTC)

Good stuff people! :) --Errant ^(chat!) 13:24, 30 March 2011 (UTC)

Good stuff WSC, but how many would answer it honestly, even if it were 100% anonymous? An endemic problem at RfA is the high number of rare, or one-time !voters. They run into the 1,000s. I've asked SW if he can run up a stats table to be able to extrapolate a voter profile. Kudpung กุดผึ้ง (talk) 13:41, 30 March 2011 (UTC)

Speaking of stats - wasn't there a tool which you could use to see how someone had voted at RfA? Is that still about? Worm^TT · (talk) 13:45, 30 March 2011 (UTC)

It's still about but for some reason access to it through the X-tools home interface has been disabled. You can reach it through here, you just need to change the user name in your browser bar once it's loaded, and then reload. Kudpung กุดผึ้ง (talk) 14:12, 30 March 2011 (UTC)

Re honesty of response. Actually I don't want an honest response to the amount of scrutiny that most voters give candidates, if I wanted an honest response I would include options for under a minute and 1-5 minutes. I'd rather use the survey to gently chide a number of our voters into thinking that perhaps they should put a bit more effort into their own research before they voted. As for those who've only voted once, I see no harm in excluding them from this study, if we ever survey them it should be with a tailored survey. ϢereSpielChequers 14:37, 30 March 2011 (UTC)

<outdent> Might be an idea to ask them why they only voted once? (In case it was something along the lines of 'so disgusted / put off by whole thing that I never wanted to go back there') Pesky (talk) 02:53, 31 March 2011 (UTC)

Possible checkbox answers:

Support voters:

• The candidate gave me a barnstar
• The candidate is in my class at school
• The candidate emailed me
• The candidate sent me a text message
• The candidate is my big sister's boy friend
• The candidate is my big brother
• The candidate gave me $10 to vote
• The candidate gave me $100 to vote
• The candidate promised me a new skateboard if I voted
• I thought it would be cool to vote at RfA

Oppose voters:

• The candidate redirected my article about my elementary school
• The candidate CSD'd/PRODed/AfD'd my article
• The candidate reported me to ANI for calling him an 'obtuse jerk'
• The candidate is in my class at school and s/he's the teacher's pet
• The candidate is an atheist
• The candidate has barnstars on their talk page
• No special reason.
• I thought it would be cool to vote at RfA

Kudpung กุดผึ้ง (talk) 17:50, 1 April 2011 (UTC)

Preview questionnaire

I've created a "preview" questionnaire for candidates based on what we have so far, on Survey Monkey. It's not intended to be ready for the wild or anything (if you fill it in the results go nowhere :D) but it was mostly to test the abilities of the tool and get a feel for the format/style of questions. http://tool.surveypirate.com/SurveyPage.aspx?surveyid=27689

FWIW I'm unimpressed with the overall result of SM.. so will look at other tools too.

I noticed above that someone talked about doing it on-wiki with dev support; given the timeframe I don't think this is possible. But if it is - what options are available? --Errant ^(chat!) 13:24, 30 March 2011 (UTC)

Yes, I mentioned it. Polls have made before and it's quite easy to do. There's no harm in asking dev support if they can do it quickly enough. They could do it in less than the time you spent on SurveyPirate, especially if they had that as a model, and it would be hosted on the WMF servers and be quicker and more secure. --Kudpung กุดผึ้ง (talk) 14:06, 30 March 2011 (UTC)

Well there is Special:SecurePoll, which gets used in elections. I dug through the code for it (as there is no documentation) and it could be usable *except* it is designed for elections so it's more an approval voting system at the moment than a survey. In addition it only has support for radio check boxes - so no text-box entries etc. However, it could be useful, I will try to ping someone about its use. Certainly for a quick "end of RFA feedback" that we use day-to-day it would be *perfect* --Errant ^(chat!) 14:37, 30 March 2011 (UTC)

There's also Special:SimpleSurvey under special pages. Always been curious as to what that was... Worm^TT · (talk) 14:41, 30 March 2011 (UTC)

Well, would you look at that. I knew I should go digging more often.. Worm^TT · (talk) 14:44, 30 March 2011 (UTC)

Oh cool, that would work. Assuming the syntax for creating them is as it is on the SVN then that should be pretty simple --Errant ^(chat!) 14:51, 30 March 2011 (UTC)

<outdent> Hey, that questionnaire was fun! Pesky (talk) 16:52, 30 March 2011 (UTC) @Worm - that looks like just the biscuit :o) Pesky (talk) 16:53, 30 March 2011 (UTC)

Wording

Query:

• Was your RFA...

1. A Self-Nomination
2. A Nomination by another editor

My first run was a self nom, my second I was nominated. How should I answer this question, and can the question be reworded? "Was your most recent RFA" would be clear, but adding a third answer

3, "I've run as a self nomination and also when nominated by another editor." would be better.

• At the time of your MOST RECENT RFA, how many ARTICLE space edits did you have?

It's never a good idea to ask people a survey question which they won't know the answer to. You risk them guessing or wandering off and looking this up. If you must ask it put in a link to the answer - in this case it will be the talkpage of their most recent RFA, better still you could retrieve that info if they tell you their username. However that raises a secondary issue with this sort of question, Privacy. The exact number of article edits at the time of your last RFA is on its own a unique identifier for most people who have had an RFA, combine that with another variable as a tie breaker and you could uniquely identify anyone completing the survey. Much better to band it or ask "to the nearest 1,000".

• At the time of your MOST RECENT RFA, how many Good and/or Featured Articles (combined) did you have?

I would suggest "had you contributed" or "had you been a significant author of" instead of "did you have".

ϢereSpielChequers 11:13, 31 March 2011 (UTC)

All good, sound sense. Pesky (talk) 12:32, 31 March 2011 (UTC)