-
2:30 pm, December 3, 2003
- The changes to ITS DNS config to switch over to roc/osprey have been made! Yes indeed, we are now moved over to new servers.
- As noted below, the changes may take a few days to propagate. Use the ip addresses to connect until then.
- Some people will notice their ITS forwards from Swatmail are no
longer functioning. This is a glitch on the ITS end; you will need
to setup email forwarding again from the ITS mail account management
page. Your email is sitting in Swathmail. Do not despair.
- Yes, now is the time to inundate us with problems. Staff is going
on caffeine IVs so that we can fix any new issues that appear. Or
at least, I am.
-
2:03 pm, December 3, 2003
- Within minutes, we will be moving DNS over to Roc and completely removing Merlin from the picture
- The change will take time to propagate across the Net, so use IP addresses to connect if you can't connect directly
- Login: sccs.swarthmore.edu, osprey.sccs.swarthmore.edu,
130.58.218.6; Secure (webmail): secure.sccs.swarthmore.edu,
130.58.218.4; Web: www.sccs.swarthmore.edu, 130.58.218.5
-
3:01 am, December 3, 2003
- Rejoice! Osprey is back up, now faster than ever (yes, that's not saying much, but we swears on the precious, it's fast)
- Webmail should now work, too
- For a moment, it looked like Osprey's ethernet card had killed itself
- However, the problem was a mismatch between our switch (set at full duplex) and our ethernet card (set at half duplex)
- The misconfiguration has gone the way of the dinosaur, so you should have nice and snappy access from now on
- We are going to sleep. Do not poke us. We are like caged polar bears. Very dangerous when provoked.
-
2:18 am, December 3, 2003
- Osprey is down for the moment
- We think we found one of the sources for the massive slowdown: mail spools were being doubly mounted on Osprey from Roc
- We are headed to the server room now to diagnose our network switch, in case that was causing speed problems
- For the time being, Osprey is down
-
5:14 pm, December 2, 2003
- Whoops! If you got your password changed but couldn't log in, it's
because we were only allowing staff logins. How selfish of us!
- Anyone can login now...
-
4:48 pm, December 2, 2003
- If you are having trouble logging in and you've updated your
password, make sure that your are SSHing into osprey.sccs.swarthmore.edu
- If osprey doesn't work either, connect directly by IP address:
osprey (login) is 130.58.218.6, and roc (web) is 130.58.218.5, and roc
(secure web) is 130.58.218.4
- This change is temporary, and will go away once DNS issues have been resolved. For now, do not use sccs.swarthmore.edu.
-
4:40 pm, December 2, 2003
- The password change is underway!
- If you have come to see us and gotten your password changed, you
can log in. If you haven't come to see us, visit, because we are so
very, very alone. If you can't come see us, we need to make
arrangements over the phone. See How to change your password - Alumni below.
-
4:40 pm, December 2, 2003
- It looks like some combination of problems with the SCCS and problems with ITS has broken forwards from SwatMail
- If you had SwatMail set up to forward your mail from ITS to SCCS, you will need to set it up again to unbreak it.
- Note on Webmail: if you have an exceedingly large Inbox, you may
not be able to open it in the webmail. We upped Squirrelmail's memory,
but if you're still having problems, we'll help you lose that flab,
guaranteed or your money back!
-
12:39 am, December 2, 2003
- All ssh public keys have been removed; if you had a public key, it's gone now
- This is to prevent you from logging into the systems before you reset your password
- Once your password has been reset, put new keys on our systems
-
10:44 pm, December 1, 2003
- Webmail is back up! Regular secure imap to osprey (130.58.218.6)
should also work from a graphical mail client like Eudora, Outlook,
Mail, Firebird, etc.
-
10:10 pm, December 1, 2003
- Mailman lists have been migrated to the new server
- Mailman administration powers will not be available during the migration
- You will still be able to send/receive mail to/from lists
- Big list admins: your passwords have been changed! You will be
contacted by staff with your new list password, which has been randomly
generated. You can change this password later.
- Big list admins: your list config has been changed back to some defaults. You can modify these settings later.
- Small list users: your lists are now managed by mailman, meaning it
is a big list. You will be contacted by staff, and you need to select
list admin(s). In the meantime, we can administrate your list.
-
9:33 pm, December 1, 2003
- There may be some burps in webmail service as we point it to the new server
- Webmail will be up by the end of the night
-
9:06 pm, December 1, 2003
- We are implementing the migration TONIGHT!!!
- Login has been turned off on Merlin
- Users on Merlin will be kicked off within the next hour or so
- You will not be able to login to ANY of our machines until you get your password reset
- There may be DNS problems for the next few days as the changes propagate across the net
- You will not be able to use mailman until the migration is complete
- You will be able to check your mail through our web interface, SquirrelMail
-
8:54 pm, Thursday, November 13, 2003
- It looks like the restart last night fixed everything
- Sometimes computers are smarter than we think they are
- Webmail had actually been available for a while: See Checking your Mail for more info
-
1:13 am, Thursday, November 13, 2003
- Merlin went down at some point tonight after we finished work
- The NFS connection between Roc and Merlin died for some reason
- A restart on Merlin fixed the problem, for now
- There is the possibility that Merlin will die tonight for the same reason, since we don't know the cause
-
8:33 pm, Wednesday, November 12, 2003
- A minor problem with DNS prevented Merlin from mounting /home and /Web after a restart
- The problem is fixed, and Merlin is back up
- Our apologies go out to you mailing list admins. Apparently mailman
quiet mode isn't as quiet as we thought. We still need to set up
mailing lists on Roc, but we should be able to prevent your inboxes
from getting clogged with "new user" notifications the next time.
-
8:05 pm, Thursday, October 23, 2003
- At last, we have conquered the demons of bind! Hopefully, this DNS foolishness should be over for good.
-
12:05 am, Thursday, October 23, 2003
- While the servers were down, DNS lost entries for the SCCS domains
- Use 130.58.218.7 for Merlin, and 130.58.218.5 for web access
-
11:40 pm, Wednesday, October 22, 2003
- Tonight's downtime was due to major security flaws in PHP
- We've had to disable some functions of PHP-- this may be an
inconvenience to some users and organizations, but there is no way
around it. These features will NOT be re-enabled
- The affected PHP functions are: exec, passthru, proc_close,
proc_get_status, proc_nice, proc_open, proc_terminate, shell_exec,
system, popen, pclose, set_ini
- If you use any of these functions in PHP scripts for your web page, you will have to find other ways to do what you want to do
-
12:25 pm, Sunday, October 19, 2003
- The DNS problems from a few days ago may still be propagating through DNS servers
- For our Web hosting, use Roc
- For secure webmail, use Roc (Secure)
- For logging in, ssh into Merlin: 130.58.218.7
-
9:55 pm, Saturday, October 18, 2003
- Webmail is now available for accessing your accounts on Merlin!
- Go to Secure webmail
- Use your current username and password to login
- Sample how mail access will work during the password reset
-
8:55 pm, Saturday, October 18, 2003
- Login to merlin has been reenabled
- We are setting up webmail for use with Merlin, so you can get used to it before the password transition period
-
7:25 pm, Saturday, October 18, 2003
- All inboxes which were deleted during the upgrade should now be
restored. If you are still missing mail, let us know; we have your
mail, it is not lost
- Oops. Some users may find they have...more... mail than they used to.
We're sorry for the inconvenience caused to users whose mail spools
have been doubled. Our bad
- Logins on merlin are still disabled, but you should be able to
check email via IMAPS or POP3 at this time
- Continue to watch this space for updates; the new servers are just
about ready to go live...
-
5:00 pm, Saturday, October 18, 2003
- Some inboxes were deleted during the upgrade
- We're restoring them from tape now. Once that's done, we'll concatenate them with all mail received since the upgrade
- If all goes well, merlin will be up by 10pm
- We're worried that other files may also have been affected. If
files in your home directory are missing or modified, email staff and
we'll restore them
-
8:00 pm, Friday, October 17, 2003
- We are blocking login to merlin and stopping postfix (part of
our mail delivery system) until we figure out what's going on with
people's mail spools
-
6:15 pm, Friday, October 17, 2003
- We know there is a problem with some of your inboxes
- We are working on the problem, so please don't inundate our inboxes :o)
4:05 pm, Friday, October 17, 2003
- Merlin is staggering slowly out of the ashes, but we believe that all file permissions have been repaired
- If you have problems with files belonging to you that shouldn't, or vice versa, please contact staff@sccs
- All files are now being hosted on Roc, and shared over the local network to Merlin
- All logins will still be to Merlin for the time being
- Directory access could be very, very slow; Merlin has a gimpy network card
- Please be patient when using cd, ls, etc.
- Please do not stay logged in if you don't have to, and don't leave programs (e.g. pine) running if you're not using them
- The
slowdown will persist until we switch login to Osprey, which will
happen as soon as we get mail delivery set up on Roc and Osprey
- Some
mail may have bounced during last night's and today's downtime, but it
does not appear that any mail bounced, so all mail should get delivered
eventually
12:55 pm, Friday, October 17, 2003
- We are moving home directories from Merlin to Roc
- Roc has more modern versions of certain scripting languages, which should allow us to repair all permissions problems
- Once the transfer and repair is complete, Merlin will read Home off of Roc just like web-docs
- Expect Merlin to be back up late this afternoon or evening
3:10 am, Friday, October 17, 2003
- Where by "back up", we meant "still down"
- We are keeping Merlin offline tonight until we either repair all permissions issues or restore from a backup
- Be
aware that if we are forced to restore from backup, files created or
modified since around 6 pm Thursday, including received e-mail, will
disappear
1:00 am, Friday, October 17, 2003
- Merlin is back up! Users should be able to log in normally. Web docs will be back up shortly...
-
10:15 pm, Thursday, October 16th, 2003
- Merlin is still down. We ran into an unforseen quirk, but never fear; merlin shall return soon!
-
8:15 pm, Thursday, October 16th, 2003
- Merlin will be going down any minute now
-
7:55 pm, Thursday, October 16th, 2003
- Merlin will be going down in about 30 minutes
- We do not anticipate the maintenance to take very long, but the outage could last up to two hours if there are problems
-
7:50 pm, Thursday, October 16th, 2003
- Active mailing lists have been migrated to mailman on Roc
- New random list passwords will be sent to the list administrators shortly
- List administrators: you will have to change list settings as you see fit
- All list settings have been reset to SCCS-wide defaults, to ensure that our configuration is clean
- Inactive lists have been DELETED. You will have to request a new list if you want to recreate one of these lists
- For the time being, mailing lists are still being run unchanged on Merlin
-
3:30 pm, Thursday, October 16th, 2003
- We are going to eat lunch
- After lunch, we will be bringing Merlin down for (hopefully) a short time
- We are remapping user and group IDs to ease the move to Roc
- All files in home directories will be updated, followed by files in Web (hosted on Roc)
- Once we are back up, you will not have access to web-docs
-
1:35 pm, Thursday, October 16th, 2003
- The DNS problems have been resolved
- We are remapping group and user IDs in preparation for moving account information from Merlin to Roc
- We are setting up NIS to push password information to the clients in the media lounge from Roc
-
7:35 pm, Wednesday, October 15th, 2003
- Some kind of awful is going on with DNS. We think it's not our fault.
- Use 130.58.218.7 to get to merlin (to log in, check mail, transfer files, etc), and 130.58.218.5 for HTTP.
- Waiting on ITS to open up that port for HTTPS... (Yay, campus firewall!)
-
6:05 pm, Wednesday, October 15th, 2003
- Osprey now has a happy new kernel
- We are making a restore tape for Roc so that we can go back to a clean slate if we get |-|@x}{0r3d in the future
-
5:30 pm, Wednesday, October 15th, 2003
- We are going to delete all mailing lists that have never been
used, or have not seen significant traffic since the semester begin
- List admins will be alerted before deletion
-
4:50 pm, Wednesday, October 15th, 2003
- The secure website is not accessible off-campus because of the campus firewall
- We will talk to ITS about this problem ASAP
- "sccs.swarthmore.edu" may fail to resolve due to some sort of DNS problem between SCCS and ITS machines
- Use 130.58.218.7 to connect to merlin directly, and 130.58.218.5 for SCCS-hosted websites
-
3:20 pm, Wednesday, October 15th, 2003
- Mailman lists and archives have been copied over to Roc
- All lists are still running on merlin
-
1:35 pm, Wednesday, October 15th, 2003
- https://secure.sccs.swarthmore.edu/ is up
- There is nothing there... yet
- It may take time for DNS servers to refresh, but https://130.58.218.4/ will definitely work for now
-
1:20 pm, Wednesday, October 15th, 2003
- The web server is running again
- We now have a working SSL certificate
-
1:00 pm, Wednesday, October 15th, 2003
- We are doing some web server reconfiguration
- This may temporarily break SCCS-hosted web sites
- This will not affect your access to web-docs on merlin
-
12:35 pm, Wednesday, October 15th, 2003
- A website just called us on the phone to confirm that we ordered an SSL certificate
- We are reconfiguring the webserver now
-
12:10 pm, Wednesday, October 15th, 2003
- We are generating an SSL certificate to allow secure login to the webmail system
-
12:40 am, Wednesday, October 15th, 2003
- Almost finished tweaking Roc's config... for the night, anyway.
-
7:45 pm, Tuesday, October 14th, 2003
- All SCCS web services are available!! Yay!!
- Last entry for today... we hope
-
7:40 pm, Tuesday, October 14th, 2003
- All SCCS-hosted web sites are now available
- Expect web-docs to be available on Merlin again soon
-
7:30 pm, Tuesday, October 14th, 2003
- Roc has been configured, and the kernel is recompiling
- Expect web to be available again soon
-
4:30 pm, Tuesday, October 14th, 2003
- Roc has been reinstalled
- We are configuring Roc now
-
2:45 pm, Tuesday, October 14th, 2003
- All web hosting is down while we reinstall Roc. No SCCS hosted websites will be available during the reinstall.
-
2:30 pm, Tuesday, October 14th, 2003
- We are preparing boot disks for both Roc and Osprey
- We are shuffling hard drives between machines
- Merlin will remain up during these changes
-
2:00 pm, Tuesday, October 14th, 2003
- Web-docs backup is complete
- We are beginning the Roc reinstall
- Web-docs will remain unavailable until Roc is back up and running
-
1:00 pm, Tuesday, October 14th, 2003
- We are backing up web-docs (currently on Roc) before reinstalling that system.
-
12:00 pm, Tuesday, October 14th, 2003
- Access to web-docs has been temporarily disabled.