Metrics

Revision as of 16:15, 4 January 2011

Status Update

Metrics for November are posted below --Dawnfoster 18:53, 21 December 2010 (UTC)

Monthly Metrics

More details than you could possibly want to know about the activity in the MeeGo community.

• November 2010 (PDF) or read the blog post summary
• October 2010 (PDF) or read the blog post summary
• September 2010 (PDF) or read the blog post summary
• August 2010 (PDF) or read the blog post summary
• July 2010 (PDF) or read the blog post summary
• June 2010 (PDF) or read the blog post summary
• May 2010 (PDF) or read the blog post summary
• April 2010 (PDF) or read the blog post summary

Community Health Metrics

My approach to metrics is to focus on measuring the things that help indicate whether the community is healthy and growing and identify any issues early. I have an internal deliverable to provide a monthly report in the first week of every month to my management team, and my plan is to deliver a similar report for the community. I also try to focus on picking the things that have the most impact rather than trying to measure everything. -- Dawn Foster

The metrics that are currently being gathered are organized into four categories:

• Awareness: Finding the community and visiting it to learn more (potential community members)
• Membership: Taking the time to sign up and join the community (passive and active community members)
• Engagement: Interacting with other community members (active community members)
• Development: Engaging with the MeeGo project at a code level (developers)

Note: Dawn Foster is currently gathering the metrics listed in this section.

Awareness Metrics

• Website Visits (source Google Analytics)
• Unique Visitors (source Google Analytics)
• Page Views (source Google Analytics)
• Social Media Posts (source Radian6)

Membership Metrics

• MeeGo.com Total Members (source Drupal)
• MeeGo-Dev ML Subscribers (source mailing lists)
• MeeGo-Community ML Subscribers (source mailing lists)
• MeeGo-SDK ML Subscribers (source mailing lists)
• MeeGo L10N ML subscribers (source mailing lists)

Engagement Metrics

• MeeGo-Dev ML Messages (source mailing lists)
• MeeGo-Community ML Messages (source mailing lists)
• MeeGo-SDK ML Messages (source mailing lists)
• MeeGo-iL10n ML messages (source mailing lists)
• MeeGo-Dev People with 2+ messages (source mailing lists / mlstats)
• MeeGo-Community People with 2+ messages (source mailing lists / mlstats)
• Forum Threads (source vBulletin)
• Blog Comments (source Drupal)
• Wiki Content Pages (source MediaWiki)
• Wiki Page Edits (source MediaWiki)
• Overall MeeGo Translation in the 16 POR languages (source Margie)
• Community-sourced translations (beyond the POR 16) that are at 50% complete or above (source Margie)
• Active language teams (source Margie)
• Overall MeeGo Translation in the 37 languages (source Margie)
• IRC top contributors and common words used (source irssistats from Stskeeps)

Code Metrics

• Downloads (source Anas)
• Commits (source Gitorius)
• New bugs (source Bugzilla)
• Closed bugs (source Bugzilla)
• Total bugs (source Bugzilla)

Content Analysis

• Most popular mailing list posts across each list (source mailing lists / mlstats)
• Most popular forum threads (source vBulletin)

Metrics Dashboard Automation Project

• Lead: Dave Neary
• Help from: Dawn Foster
• Dave to add details.

Biggest Gaps We Need to Fill Now

• Add Community OBS Metrics - work with lbt.
• Add Transifex (localization) metrics - work with Dmitry
• Add Download numbers (including mirrors) - work with Anas / Adam

Future Improvement Ideas

This is a great way to help out on the MeeGo project - anyone can contribute to the metrics!

If you are starting a task, please add your name next to it to claim your task to avoid duplication (we don't want multiple people working on the same task without coordinating).

Dashboard - high priority

• An automated dashboard that people can access at any time gathering some of these metrics.
• We can start with some of the easy ones to gather and add more over time.
• Would be great to cut this by month for easy report generation.
• Note: see the points to consider section below for some additional ideas

Other metrics that people may want to capture

• Software packaging
  • New packages added to repository
  • Updates to existing packages
  • Number of developers/maintainers uploading packages
• Server performance and maintenance metrics
  • Server use levels
  • Network bandwidth
  • Downtime and error reports
  • OS and firmware patching
• bugzilla reports
  • Fixed bug reports
  • Unfixed closed bugreports (dups, not a bug, etc)
  • Upstreamed bug reports (MeeGo bug matched with upstream bug report)
• various open source aspects
  • Activity of Intel & Nokia developers vs activity of "NIN" (Not Intel/Nokia) developers
  • Number of NIN committers, number of MeeGo specific modules with NIN committers.
  • Not sure what you might want to quantitatively measure here - qualitative is hard to automate. See Siobhan O'Mahony's reports for Eclipse & various Eclipse Foundation reports for inspiration Dneary 10:00, 10 March 2010 (UTC)
• membership / user accounts
  • member satisfaction
  • member recognition
  • Not sure what you might want to quantitatively measure here - qualitative is hard to automate. Dneary 10:00, 10 March 2010 (UTC)
• Planet MeeGo
  • Blog posts
  • Voting activity
  • Reactions to posts from outside meego.com

Alerts

• Planned events
  • Infrastructure service outages
• Unplanned events
  • Capacity shortages (monitor storage and throughput; report on threshold breaches)
  • Attacks on infrastructure

Points to consider

• Reporting solution
  • A comparison of open source solutions - compares Jasper Reports, Pentaho, BIRT and more
  • Cricket looks promising
  • Direct graphing with graphviz might be simplest
  • Drupal solution example
• Report sharing methods
  • Dashboard(s) for ongoing data reports (server health, member stats, bugs, app stats, etc)
  • Voluntary subscriptions (ie, RSS) for periodic public reports
• Public vs Private data

References

• Eclipse Foundation dashboards
• The Art of Community - Chapter 7: Measuring community
• gitdm - Git data miner
• Metrics for healthy communities - Tara Hunt
• Collection of metrics
• Tools for analysing CVS/SVN, mailing lists, social networks and more. Community metrics tools mecca.

Contributors

• Coordinator: Dawn Foster/dawnfoster - Intel's MeeGo community manager.

• Dave Neary/dneary - Maemo docmaster, I help grow free software communities.

• Henri Bergius/bergie - helping to automate the metrics.

Additional Helpful Details on Specific Metrics

mlstats queries:

• Count posts of most popular threads:

  select subject,year(first_date) as y, monthname(first_date),count(*) as c from messages group by subject, month(first_date) order by y, month(first_date), c;

• Count the number of posts for each person

  select p.email_address,year(m.first_date) as y, monthname(m.first_date),count(*) as c from messages as m,messages_people as p where m.message_id=p.message_ID group by p.email_address, month(m.first_date) order by y, month(m.first_date), c;

• Restricting queries to a date range / month

  The query above gives month-by-month totals, you could add an order by year(first_date) to get the year too

  The easiest way is to add where month(first_date)=3 and year(first_date)=2010 for March 2010. For the current month, month(m.first_date)=month(NOW()) and year(first_date)=year(NOW()) works.

Bugzilla:

• A Bugzilla report that shows new bugs, closed bugs, and total bugs (or instructions for creating this report)

  Standard Bugzilla search queries & reports can give counts for this. Using relative dates in the date fields you can get all changes from one month ago (set date to -1m) or one week ago (-1w).

  • New bugs this month, by product, and their current states (shortened link)
  • Resolved & closed bugs in the past month (Status changed in the past month, and current status is one of resolved, released, verified or closed - not sure if this is what you want)
  • There are lots of variations - play with bugzilla tabular reports and relative dates for more.

  This might be easiest to automate as a Bugzilla extension querying the BZ database directly. Some potential resources:

  • Bugzilla database schema by version
  • Demo site for standard Bugzilla reports
  • The Bugzilla SQL cookbook
  • Bugzilla Metrics - Third party add-on to Bugzilla that produces lots of metrics
  • browse.cgi and weekly-summary.html extensions of GNOME Bugzilla: Code, examples in practice: browse.cgi, weekly-summary.cgi

MediaWiki

• A MediaWiki extension exists to provide "user scores" for MediaWiki users, ordered by number of edits and number of pages changed. The guts of the query is:

SELECT COUNT(wr.rev_id) as value,
       COUNT(DISTINCT wr.rev_page) as page_value,
       wu.user_name as name,
       wu.user_real_name as real_name
FROM   $user wu,
       $revision wr,
       $page wp
WHERE  wu.user_id = wr.rev_user
   and wp.page_id = wr.rev_page
   and wp.page_namespace = 0
GROUP BY wu.user_name
ORDER BY value desc;

where $user, $revision and $page are the names of the respective MediaWiki tables (MediaWiki tables have a prefix associated with them for a given instance, specified by $wgDBprefix in LocalSettings.php).

IRC

Stskeeps has started running monthly IRC reports:

• February
• March
• April