Wikipedia:WikiProject User warnings/Testing/28bot

This space is for documentation on an A/B test with 28bytes' 28bot.

Hypothesis

Based on what we've seen through qualitative coding of new user contributions, the majority of first edits to Wikipedia are simple tests – tinkering with formatting, adding or removing random text, inserting [[File:Example]] into articles, etc. Some of these users probably don't even know their edits are visible to everyone until they receive a template warning, like the ones issued by 28bot. We'd like to test whether a few specific changes to the content of the message might increase retention of future good editors and decrease the likelihood that malicious test editors become persistent vandals. Specifically, we want to test for the effects of:

personalization - a message written in the active rather than passive voice, with more friendly, welcoming, and encouraging language. Also, personalization in the sense that we will do different messages for registered and anonymous users.
directives - a more detailed explanation of why their edit was reverted, where they can edit safely, where they can ask for help, and where/why to register an account (for anonymous editors)
length - a much shorter message that only includes the most important links

Templates

Analysis

Outcomes we will want to measure include...

Retention: how many editors edit after receiving the warning? 30 days after? 60 days after?
Contact: how many editors (IP or registered) contact the bot or warning user? Is this contact constructive or not?
Improvement: how many editors receive further warnings (from any user or bot) or are blocked?
Registrations: how many IP editors go on to register accounts after being warned vs. those not given the warning?

Additionally, we may compare the users warned during this test period with the test editors who are not warned but have their edits reverted by RscprinterBot, to see if getting no warning is better/worse for retention/improvement. Log of the bot's test editor reverts can be found here.

Other notes and comments

Should/can we alter the bot to send different templates to IPs and registered editors?