Mysterious power failure takes down Wikipedia, Wikinews

Wednesday, February 23, 2005

On Monday, at 10.14 pm UTC, the Wikimedia server cluster experienced a total power failure, taking down Wikipedia, Wikinews, and all other Wikimedia websites.

The servers are housed in a colocation facility (colo) in Tampa, Florida, USA. They occupy two racks, with each rack receiving electricity from two independent supplies. However, both supplies have circuit-breakers in them, and both opened at the same time, leading to a total power failure. All computers immediately went down. It’s normal for fire safety regulations to prohibit uninterruptible power supplies in colos, with the colo providing its own UPS and generator instead. The circuit breakers were on the computer side of this emergency power system, so none of the computers continued to receive power to survive the breaker trip or shut down safely.

The actual reason why the circuit breakers tripped is currently unknown.

When power was restored, it was discovered that most of the MySQL databases that store the data which makes up Wikipedia et al had been corrupted. The main database and the four slaves had all damaged the data on their hard disk drives beyond the ability of the auto-correction to repair. Only one copy survived safely, on a machine that is used for report generation and maintenance tasks, which remained 31 hours backlogged while catching up after an unusually heavy update load during the previous week.

Volunteer Wikimedia engineers worked through the night rebuilding the databases from the sole good copy onto the other servers. The Wikipedia database is over 180Gb in size, making the copying process last 1.5 – 2 hours for every server it was performed upon.

Regular back-ups of the database of Wikipedia projects are maintained – the encyclopedia in its entirety was not at risk. The last database download was made on February 9; all edits since then could only have been laboriously rebuilt from logs and recovered from the damaged database requiring much more time and effort.

Limited read-only service was established late Tuesday afternoon, with editing becoming possible 24 hours after the power failure. Final repairs continue now, as well as upgrades to prevent similar issues in the future. Server-intensive features, such as categories and ‘watchlists’ that display recent changes to selected articles to registered users, remain disabled to ease the load on the recovering systems.

The process which led to the damage originated with the operating system, disk controllers, or hard drives failing to flush the data correctly.

If the power to a database server is cut mid-write, the database may be corrupted and unreadable, however the operating system, hardware, and software are designed to make this very unlikely. In a previous incident in 2004 power was also lost to a server but the database was undamaged.

To avoid such damage, each database server saves a copy of an edit to be applied to the database on a separate storage system before making the actual update to the database itself. This so-called ‘write-ahead logging’ should ensure that in the event of a system crash, the database can be rebuilt from a ‘last-good’ state by replaying the edits saved in the log.

Earlier this year popular blogging site LiveJournal suffered a similar power failure when another customer at their colocation facility pressed an Emergency Power Off button, intended for use only by firefighters. The company suffered database corruption similar to that seen at Wikimedia.

LiveJournal are now fitting UPS to their servers to ensure that they have time to shut down safely in the event of a power failure. Wikimedia was said to be investigating the possibility of fitting similar equipment at the time of this failure.

Several pundits have suggested that the use of another database, such as the proprietary database Oracle or the free PostgreSQL, would have avoided the database corruption seen at the server cluster. A post-mortem of the incident show the failure was in the operating system, or the hardware, or some combination of the two. LiveJournal, which also uses MySQL, reported similar database corruption after their power cut.

The Wikimedia foundation only allows the use of free software on its systems, and future versions of the Mediawiki software will support the PostgreSQL database.

Users are reminded that during times of system failure or excessive demand, they can still search Wikipedia using Google. The articles may be viewed using Google’s cache.

Monday, November 28, 2005

SpaceX called off the much-delayed inaugural launch of their new Falcon 1 rocket on Saturday from Kwajalein’s Omelek Island launch site. The intent was to launch the U.S. Air Force Academy’s FalconSat 2 satellite, which will monitor plasma interactions with the Earth’s upper atmosphere and magnetosphere.

The launch was delayed, then finally cancelled after an oxygen boil-off vent had accidentally been left open. The oxygen was unable to cool the helium pressurant, which then proceeded to evaporate faster than it could be replenished. A main computer issue, probably serious enough to cause a scrub on its own, was also discovered.

This long-anticipated flight was originally expected to be launched in January 2005, however a series of setbacks forced a series of delays, with the flight most recently scheduled to be in early 2006. It was intended to be launched from the Kwajalein atoll in the middle of the Pacific Ocean.

The maiden voyage was originally intended to launch from Vandenberg Air Force Base in California with a Naval Research Laboratory satellite and a Space Services Incorporated space burial payload.

Monday, September 24, 2007

Marion Schaffer is running for the Green Party of Ontario in the Ontario provincial election, in the Oakville riding. Wikinews’ Nick Moreau interviewed her regarding her values, her experience, and her campaign.

Stay tuned for further interviews; every candidate from every party is eligible, and will be contacted. Expect interviews from Liberals, Progressive Conservatives, New Democratic Party members, Ontario Greens, as well as members from the Family Coalition, Freedom, Communist, Libertarian, and Confederation of Regions parties, as well as independents.

Friday, January 1, 2021

After an active year of original content published on the English-language Wikinews, we take a look back at some of the two dozen-plus original reports from our contributors during 2020.

Friday, March 9, 2007

IBM and Cisco have announced the development of a new platform, based on Eclipse and OSGi (Open Services Gateway initiative), which should unite all communication and collaboration software developers under a single platform.

Before the UC2 (Unified Communication and Collaboration) the unified communication has suffered because of a lack of a platform to be used by all software developers, rather than a series of them and, moreover, provided by different vendors. According to Adam Gartenberg from IBM Lotus Software Group this was the main cause for this partial stagnation.

However, the UC2 is meant to ease the work of software developers. Gartenberg stated that this platform is very flexible and will certainly draw much attention form programmers. They will be able to create different plug-ins and small applications to remotely control other, major application.

The companies have also unveiled a series of their other joint projects. These project will be based on a set of application programming interface from Lotus (SameTime) and Cisco.

The add-ons planned are integration of a series of additional functions and features into the SameTime software from Lotus. Thus, in a few months, the SameTime users will benefit from such functions as click-to-call and voicemail. This will enable easier instant messaging inside the Cisco’s Unified IP phones’ system.

The fact that the platform will be based on Eclipse will make the numerous developers create many communications services, mainly for remote users to benefit from a series of new functions of ERP and CRM applications.

Nortel Networks – a partner of Microsoft since last year and Cisco’s rival in the area – didn’t have much success in the communication services field.

As for Microsoft, the experts are waiting for the reaction of the Bill Gates’ company on the ideas and offers made by IBM and Cisco for enterprise communications and collaboration solutions.

Wednesday, April 8, 2009

A Boeing 757 passenger jet, operated by Yakutia airlines, successfully made an emergency landing at the Vnukovo airport in the Russian capital of Moscow on Tuesday. The plane requested the landing after technical problems were reported on board. None of the 174 passengers and nine crew members aboard were injured.

“The airplane took off from Vnukovo airport, en route to Yakutsk. After some time, some technical difficulties started, due to which the crew made a decision to return to the airport,” ^{((Translated from Russian))Russian language: ???????? ??????? ?? “???????” ? ??????, ?????? ????? ????????? ????? ????????? ?????-?? ??????????? ?????????, ??-?? ???? ?????? ?????? ??????? ????????? ??????? ?? “???????”} said a spokeswoman for Vnukovo airport. The plane was originally scheduled to land at 20:20 local time, but circled around the airport in a holding pattern for three and a half hours to burn off excess fuel.

Yakutia Flight 478 departed from Vnukovo at 17:26 local time. Twenty minutes after takeoff, at 17:46, the captain of the airplane reported technical troubles. He then decided to return to Vnukovo and land the plane. Reports suggest that the cabin depressurized while the jet was climbing to cruising altitude.

This incident occurred only a few days after Sky Express Flight 229, a Boeing 737, had to perform a similar landing shortly after takeoff, though no reports have indicated the two incidents are related.

Saturday, April 9, 2005

The Toronto Transit Commission is expected to strike Monday. The TTC workers union, the Amalgamated Transit Union Local 113, rejected a deal offered by TTC management. They last went on strike in 1999. The average daily ridership exceeds 1.3 million passengers.

The city of Toronto has prepared to help deal with the forecasted traffic congestion by opening up free parking for car-pools at 14 city-owned community centres, banning parking on some city streets, and opening up some bus-only lanes to car-pools.

The initial deal, which was rejected, offered a 2% wage increase over five years. With a huge turnout the union-members voted against accepting the offer with a 99% majority according to a press release by the union. The union rejected the latest offer Friday reported to be a three-year contract with raises of 2.75 percent in the first year, three percent in the second and 3.25 percent in the third.