Metrics/Gathering data

setup.py install --prefix=/install/path
/path/to/mlstats --db-user=<username> --db-password=<password> http://lists.meego.com/pipermail/meego-announce/

--no-report
for list in meego-adaptation-intel-automotive meego-announce meego-architecture \
            meego-commits meego-community meego-dev meego-distribution-tools \
            meego-events meego-handset meego-il10n meego-inputmethods meego-it \
            meego-ivi meego-kernel meego-packaging meego-pm meego-porting \
            meego-python meego-qa meego-releases meego-sdk meego-security \
            meego-security-discussion meego-touch-dev meego-tv;
do
  /path/to/mlstats --no-report --db-user=<username> --db-password=<password> http://lists.meego.com/pipermail/${list} >> /tmp/output.txt
done;

SELECT
     `messages`.`mailing_list_url` AS list,
     year(first_date) AS y,
     monthname(first_date) AS mon,
     month(first_date) AS m,
     date_format(first_date, '%M %Y') as monthstr,
     date_format(first_date,'%Y%m') as monthnum, 
     count(*) AS c
FROM
     `messages`
WHERE
     year(first_date)		 > 1979 and 
     mailing_list_url not like '%meego-commits%' and first_date<'2011-05-01'
GROUP BY
     `messages`.`mailing_list_url`,
     y,m
ORDER BY
     monthnum ASC,
     list ASC,
     c ASC

for y in 2010 2011; do
 for m in `seq -f '%02g' 1 12`; do
  for d in `seq -f '%02g' 1 31`; do
   wget http://mg.pov.lt/meego-irclog/%23meego.$y-$m-$d.log;
  done;
 done;
done;

   logdate= `date +%Y-%m-%d -d yesterday`
   wget -P $logdir http://mg.pov.lt/meego-irclog/%23meego.${logdate}.log;

#
#    This file contains all of sss' settings along with their defaults.
#    All values must be placed between double quotes, even empty ones.
#
#################################  Required  ###################################

channel = "#meego"		# Name of the IRC channel.
timezone = "UTC"	# Timezone the logs are in. Used for time offset
			# calculations and conversions.
			# See http://php.net/manual/en/timezones.php
db_host = "host"		# IP address or FQDN of the MySQL server.
db_port = "3306"	# Port the MySQL server is listening on.
db_user = "db_user"	# MySQL user.
db_pass = "db_password"		# MySQL password.
db_name = "db_name"		# Name of the MySQL database used for sss.
parser = "parser_supybot"	# The parser to use depending on logfile format.
			# e.g. "parser_irssi" or "parser_eggdrop"

# This string contains the format of the date within a logfile filename.
# Examples:
#   filename: #chatroom.20030131	dateformat: *.Ymd
#   filename: #chatroom.20030131	dateformat: \#c\h\atroo\m.Ymd
#   filename: chatroom.log-31012003	dateformat: *.*-dmY
#   filename: chatroom.log-31012003.gz	dateformat: *.*-dmY.\g\z
#   filename: chatroom.log-31012003.gz	dateformat: *.*-dmY.*
# See http://php.net/date_create_from_format for more specific syntax options.
logfile_dateformat = "*.Y-m-d.\lo\g"

php sss.php -c meego.conf -i ${logdir} -o report.html

   logdate=`date +%Y-%m-%d -d yesterday`
   logdir=/path/to/logfiles
   sssdir=/path/to/sss-4.0

   /usr/bin/wget -P $logdir http://mg.pov.lt/meego-irclog/%23meego.${logdate}.log;
   /usr/bin/php ${sssdir}/sss.php -c meego.conf -i ${logdir}/\#meego.${logdate}.log

select date_format(`date`, '%Y-%m') as `date`, sum(`l_total`) as `l_total`, sum(`l_night`) as `l_night`, sum(`l_morning`) as `l_morning`, sum(`l_afternoon`) as `l_afternoon`, sum(`l_evening`) as `l_evening` from `channel` group by year(`date`), month(`date`);

select `date`, `l_total`, `l_night`, `l_morning`, `l_afternoon`, `l_evening` from `channel` where `date` > DATE_SUB(CURDATE(),INTERVAL 30 DAY);

Views

Metrics/Gathering data

Revision as of 16:07, 4 July 2011

Contents

Mailing lists - MLStats

Mailing list graph over time

IRC - SuperSeriousStats

Getting the IRC logs

Parsing the logs

Queries

Wiki Navigation

Search

Toolbox

Personal tools

@@ Line 89: / Line 89: @@
 === Parsing the logs ===
-To run sss, you will need to modify a number of fields in sss.conf. Notably, I had to change the filename format to match the log file names, and set log format to supybot. I recommend creating a separate config file for each channel you want to parse.
+To run sss, you will need to modify a number of fields in sss.conf. Notably, I had to change the filename format to "*.Y-m-d.\lo\g" to match the log file names, and set log format to supybot. I recommend creating a separate config file for each channel you want to parse. Here is the appropriate sss.conf for #meego logs:
+<pre>
+#
+#    This file contains all of sss' settings along with their defaults.
+#    All values must be placed between double quotes, even empty ones.
+#
+#################################  Required  ###################################
-sss has a mechanism to prevent it from re-importing log files for the same channel, but you can store summary data from several channels in one database.
+channel = "#meego"		# Name of the IRC channel.
+timezone = "UTC"	# Timezone the logs are in. Used for time offset
+			# calculations and conversions.
+			# See http://php.net/manual/en/timezones.php
+db_host = "host"		# IP address or FQDN of the MySQL server.
+db_port = "3306"	# Port the MySQL server is listening on.
+db_user = "db_user"	# MySQL user.
+db_pass = "db_password"		# MySQL password.
+db_name = "db_name"		# Name of the MySQL database used for sss.
+parser = "parser_supybot"	# The parser to use depending on logfile format.
+			# e.g. "parser_irssi" or "parser_eggdrop"
+# This string contains the format of the date within a logfile filename.
+# Examples:
+#   filename: #chatroom.20030131	dateformat: *.Ymd
+#   filename: #chatroom.20030131	dateformat: \#c\h\atroo\m.Ymd
+#   filename: chatroom.log-31012003	dateformat: *.*-dmY
+#   filename: chatroom.log-31012003.gz	dateformat: *.*-dmY.\g\z
+#   filename: chatroom.log-31012003.gz	dateformat: *.*-dmY.*
+# See http://php.net/date_create_from_format for more specific syntax options.
+logfile_dateformat = "*.Y-m-d.\lo\g"
+</pre>
+sss has a mechanism to prevent it from re-importing log files for the same channel, by storing dates for which it has already parsed files in the table "parse_history", but you can store summary data from several channels in one database.
 To do the initial import, just run
-  php sss.php -i /path/to/logs/
+  php sss.php -c meego.conf -i ${logdir} -o report.html
-And to do the nightly import, add this after the wget line above:
+And to do the nightly import, put the following in a shell script, and call it from cron:
- php sss.php -c channel.conf -i ${logdir}/%23meego.${logdate}.log
+<pre>
+   logdate=`date +%Y-%m-%d -d yesterday`
+   logdir=/path/to/logfiles
+   sssdir=/path/to/sss-4.0
+   /usr/bin/wget -P $logdir http://mg.pov.lt/meego-irclog/%23meego.${logdate}.log;
+   /usr/bin/php ${sssdir}/sss.php -c meego.conf -i ${logdir}/\#meego.${logdate}.log
+</pre>
 === Queries ===
-A number of useful sss queries are already listed in [[Metrics/Dashboard#IRC | the Dashboard page]]
+The sss report is built from a number of queries. A number of other useful sss queries are already listed in [[Metrics/Dashboard#IRC | the Dashboard page]]
+'''Activity by month'''
+<pre>
+select date_format(`date`, '%Y-%m') as `date`, sum(`l_total`) as `l_total`, sum(`l_night`) as `l_night`, sum(`l_morning`) as `l_morning`, sum(`l_afternoon`) as `l_afternoon`, sum(`l_evening`) as `l_evening` from `channel` group by year(`date`), month(`date`);
+</pre>
+'''Activity by day over last 30 days'''
+<pre>
+select `date`, `l_total`, `l_night`, `l_morning`, `l_afternoon`, `l_evening` from `channel` where `date` > DATE_SUB(CURDATE(),INTERVAL 30 DAY);
+</pre>