Subject: Email Problems
Date: Thu, 25 Nov 2004 21:38:37 +1100 (EST)
From: Mick Houlahan
To: undisclosed-recipients: ;
Over the last week we have had a number of problems with the student email
service.
Last Thursday, a large number of students were invited by email to
participate in a survey. Unfortunately the list mechanism that was used to
issue this invitation was incorrectly configured. The end result was that
each reply to the survey instead of being directed to the survey author
was copied to each and every other member of the list. With some 20,000
students invited to participate, one can imagine the flow on effect of
each reply generating a further 20,000 emails.
Although the UWS IT Group became aware of this on Friday morning and took
immediate steps to adjust the list processing configuration, much of the
damage had already been done. Mail queues on the student email server were
saturated with close to half a million messages in the queue at its
peak. The server was still severely stressed on Monday of this week having
been spent the weekend trying to handle the mail load. As well as the
sheer number of emails, the file systems on the server filled up and
exacerbated the problem.
A disk failure on Tuesday no doubt brought on by the severe processing
load didn't help things at all!
The end result for most students has been an unreliable email service for
much of the last week with a large number of students receiving numerous
messages responding to the survey.
The University does apologise for any inconvenience that these disruptions
have caused. The survey department has been briefed on the appropriate
procedures to be followed for any future surveys in order to avoid a
recurrence in the future.
Mick Houlahan
Cheer,
Steve