UESPWiki:Status

Current Status

MediaWiki Upgrade to 1.29 -- 26 July 2022 -- Finished

• The UESP wiki will be upgrading to MediaWiki 1.29 on the morning of 26 July. There should be no service interruption and a very minimal interruption to search updates (< 5 minutes). -- Daveh ^(talk) 18:19, 25 July 2022 (UTC)
• The upgrade has finished. -- Daveh ^(talk) 16:09, 26 July 2022 (UTC)

Monitoring Status

There are a variety of methods that can be used to monitor the status of the servers making up the UESP.

Anonymous Access

1. External Status Page on Uptime.com.
2. monitor.uesp.net is the web interface for the Zabbix monitoring software shows the basic status of all servers.
3. content1 Apache status, content2 Apache status, and content3 Apache status show the Apache status for the content servers.
4. files1 Lighttpd status show the Lighttpd status for the content servers.
5. Special:PageSpeed page rendering speed stats.

E-mail Access

The Zabbix monitoring can e-mail warnings and notifications about possible issues with any of the UESP servers. If you would like your e-mail to be included in the notification list please contact Daveh. Text access can be also be supplied if your cell phone provider as a free "e-mail to text" service.

In Case of Site Issues

In case of site issues including page display, lag and service outages

• Minor issues can be brought up on the Administrator Noticeboard or directly with a Wiki Admin if appropriate.
• Ping @Dave_UESP on Discord.
• Daveh (dave@uesp.net) is the main contact for most site hardware and software issues.
• Other Wiki Admins can also be contacted if Daveh is unavailable but have limited access to fix anything.

Status History

File/Image Server Upgrade -- 13 April 2022 -- Complete

• The primary file/image server for UESP is being moved to a new one with higher capacity. There is no service interruption expected. -- Daveh ^(talk) 17:37, 13 April 2022 (UTC)
• All services should now be operating from the new server and there are no known issues. -- Daveh ^(talk) 16:38, 15 April 2022 (UTC)

MediaWiki Upgrade -- 9 February 2022 -- Completed

• We're upgrading the MediaWiki software for the wiki. No service interruption is expected other than search being broken for a short period. -- Daveh ^(talk) 14:23, 9 February 2022 (UTC)
• Upgrade has been completed! If there are any issues in the next few days/weeks let me know here. -- Daveh ^(talk) 16:55, 9 February 2022 (UTC)

Firewall Upgrade -- 12 July 2021 (Evening) -- Completed

• Between 09:30 PM and 11:30 PM EDT/UTC-4, we will activate a more recent firmware on the site's firewall. This will cause all UESP services to be unavailable for 5-15 minutes. -- Daveh ^(talk) 20:32, 13 July 2021 (UTC)

DB2 SSD Upgrade -- 15-24 June 2021 -- Completed

• DB2 is being upgraded from HDDs to SSDs. It has been removed from all APIs/wiki and db1 has been set as the main read database for now. -- Daveh ^(talk) 16:49, 15 June 2021 (UTC)
• DB2 is currently offline and being upgraded. -- Daveh ^(talk) 16:23, 16 June 2021 (UTC)
• DB2 has finished upgrading and is back in service. -- Daveh ^(talk) 18:19, 24 June 2021 (UTC)

Intermittent Performance Issues -- 1-14 June 2021 -- Completed

• There were some performance issues causing loading page loading times with no obvious cause. It was eventually traced down to random TCP timeouts on content1 which was causing memcached connections to fail. Around 1 in 100 connections would take 1000-3000ms instead of the usual 0.1-0.2ms.
• No root cause was found and the issue went away on its own. Memcached was moved to backup1 to prevent a similar issue from causing performance issues in the future. -- Daveh ^(talk) 16:48, 15 June 2021 (UTC)

Squid1 Hard Drive Replacement -- 20-25 September 2020 -- Completed

• A failing hard drive on squid1 is being replaced. All cache services on the server have been redirected to content1 via DNS so no service interruption is expected. -- Daveh ^(talk) 00:28, 21 September 2020 (UTC)
• Maintenance has been completed. -- Daveh ^(talk) 13:16, 25 September 2020 (UTC)

Server Upgrades -- March 2020 -- All servers are being upgraded this month. No downtime is expected however there may be some short periods where uploads/reads are disabled on the wiki and other services. See Upgrade_Announcements for more details.

• Files1 Hard Disk Maintenance -- 8-12 December 2013 -- Done -- The files1 server has a failing hard drive that needs to be replaced and is currently causing minor random lag issues on the wiki. To do this the Wiki uploads must be temporarily disabled during this time. No other service interruption is expected. This maintenance has been successfully completed. Note that there were some intermittent issues during the maintenance period for logged in users. This maintenance has been successfully completed (both hard drives in the server were replaced).
• Files1 Hard Disk Maintenance -- 7-8 November 2013 -- Done -- The files1 server has a failing hard drive that needs to be replaced. To do this the Wiki uploads must be temporarily disabled during this time. No other service interruption is expected. This maintenance has been successfully completed. Note that there were some intermittent issues during the maintenance period for logged in users.
• 4 March 2012 -- Database Hard Drive Issue -- Done -- Another hard drive issue on db1 will result in a short period of read-only (<5 min) for the wiki and forums sometime today while we switch the primary database to db2. No service interruption is expected.
• Database Hardware Replacement -- 1 February 2013 -- Done -- A hard drive is failing on our master database server so the wiki/forums will be locked for a few minutes sometime today while we switch servers. No downtime or service interruption is expected. The hard drive has been replaced and there will be another short period of lock time when we switch back to db1 in the next day or two.
• Database Issue -- 7:00-8:30 EST 9 December 2012 -- Resolved -- More intermittent load issues due to a misbehaving backup process on db1.
• Database Issue -- 7:30-8:30 EST 2 December 2012 -- Resolved -- The drive on the primary database filled up after the last full backup this morning which lead to various site outages. Free space on the drive has been restored which resolved the issue.

• Wiki Upgrade -- 7-10:00 EST 3 September 2012 -- Done -- The Wiki will be set to read-only (no edits permitted) during most of this time in order to perform a long overdue upgrade of the MediaWiki from 1.14 to 1.19. No service interruption is expected. The upgrade was completed successfully and barring a few minor display issues everything is back to normal.

• Wiki Database Restore -- 18-20:00 21 August 2012 -- The wiki database had to be restored which resulted in 1-2 hours of downtime for logged in users, some intermittent errors, and a lost half-day of edits.
• 20:00 EST 31 January 2012 -- Upcoming Planned Maintenance (Completed) -- One Drive on db2 needs replacing. Currently waiting for db1 to be ready for live use before switching back over to db1 for writes. There will be a brief period of read-only access to the forums and wiki when the switch is made.
• 20:00 EST 13 December 2011 -- Planned Maintenance Completed -- All apps will be switched to using db2 as a read/write database in order to perform repairs on db1. There will be a minimal period (~5 minutes) of things being in read-only mode. Everything should be now running completely off of db2.
• 17:00 EST 9 December 2011, Forums Resolved -- Forums were partially inaccessible due to a corrupt database table. Table has been repaired and forums should be mostly operable again.
• 16:05 EST 2 December 2011, db2 Resolved -- 5 minute downtime due to db2 becoming unavailable. Issue temporarily bypassed and being looked into at the moment. Motherboard on server failed and was replaced. Content servers switched back to using db2 for wiki reads.
• Db1: 20 November 2011 -- One drive on the RAID-1 array failed and is awaiting replacement.
• Firewall: 11-18 November 2011 -- The firewall was hitting its connection limit of 10,000 due to the increased traffic from Skyrim's release. Fixed by both upgrading the firewall and using a temporary squid2 server outside of the cluster.
• Files1: 5 November 2011 -- Drive failed in a RAID array which took the server down. Drive replaced and server back up.
• Content3: 8 November 2010 -- content3 taken off-line to fix a random data corruption issue. Only affect will be no UESP mail and the minor dave.uesp.net website. This does not affect registration e-mails sent by the UESP Wiki/Forums.
• All Sites: 5-8 November 2010 -- Register.com had issues which caused intermittent DNS requests for the UESP domains to fail. The effect was small and overall traffic was interrupted by less than 10% over the weekend.
• All Sites: 3 November 2010 -- iWeb had a major power failure that took out several servers at midnight for around an hour. All servers except content3 came back up when power was restored.
• Squid1: 27 Feb 2010, 21:30 EST -- Hard drive replaced and service restored. Waiting for DNS entries to update.