And all out of a sudden the blog was down. Well first not entirely down but it took several minutes to reach the page or open the admin panel. In other word the blog got a cold and I myself wasn’t feeling well since thursday either: coincidence?
Step 1: Don’t panick
As I was feeling pretty sick and was not able to stare at a monitor for a long time I first suspected a problem with the server itself which would only last a few minutes or hours and decided to better stay in bed. And well I had a pretty fresh backup from the blog. What could go wrong?
On saturday evening, at least 24h after I recognised that something was wrong, Svij contacted me and informed me that the blog still couldn’t be reached that well. But I couldn’t do anything about it before today. I was still feeling sick but not getting dizzy while looking at the screen. But what I saw kinda made me feel a bit sicker. Because the style of my blog had slightly changed.
At that point and also by viewing the server logs, I could sort out my hosting provider Host Europe, yes I’m not hosting the blog on Uberspace…yet, pretty quick and something must have been gone wrong with WordPress itself.
Step 2: First Suspect
So what to do next? I suspected that at least one of my plugins was causing all the mess but which one? First I suspected Updraft Plus which I’m using to make backups to dropbox, ownCloud isn’t supported. As I had recently changed my Dropbox password and the Updraft log file was getting bigger and bigger, but was no real hint for a problem, I suspected that a backup job wasn’t running correctly and slowed down everything. Of course this wouldn’t explain why the style had changed. But maybe one plugin, Updraft Plus, was causing strange effects for other plugins like Jetpack which holds my custom css. And if WordPress can’t communicate through Jetpack with my blog because it is not reachable, well maybe there we have it.
Of course I can hear you now: ‘Use static pages, get rid of all the WordPress crap and all the plugins making everything more complicated than it should’. And yes to some extend you are right. I just can say that I’m kinda lazy at this part and I like what I can easily achieve, at least most of the time, with WordPress.
That being said I now had to find a way to disable the plugins, or at least the one I was suspecting to cause all the trouble, manually without having to delete every plugin folder. The solution here is pretty simple: Access your blogs database e.g. with phpmyadmin, look for the option ‘active_plugins’ and change the entry. It goes like this:
SELECT * from wp_options WHERE option_name LIKE 'active_plugins';
As a result you’ll get something like this:
If you have installed only one plugin it is pretty simple, just click on the red X to delete the entry. But if you have several plugins installed and only want to disable one or a few you have to change the entry by editing the option_value by clicking on the edit-icon (the pencil). As Option Value you’ll see something like this (example for 5 plugins):
The line starts with a: and a number in this case 5 which means there are 5 active plugins. To disable plugins you have to reset the counter and delete the corresponding entry inside the brackets. If you’d like to disable the custom-sidebars the value should look like this:
Just save the entry and you should be good to go. Well at least that’s what I thought.
Step 3: Cleaning up
Sadly my initial assumption, that Updraft Plus was guilty, was not correct and I had to deactivate every plugin. This did work and I could access my blog again. Eureka! Well not yet. Because the design was still messed up and I also saw that my database was overrun by ‘transient’ entries. So something might have been wrong with the Transients API or better said with something accessing the API. This is a quite common problem and some plugins indeed do cause a whole lot of these entries. But they are supposed to be cleared after they have expired.
So I did the job myself and did a bit of cleaning. The following sql statement will should give you all the expired ones:
SELECT option_name FROM wp_options WHERE option_name LIKE '_transient_timeout%' AND option_value < now();
From what I’ve read it should be no problem just to delete every single entry because the plugins which are using them will just create new ones without any problems.
After cleaning up the mess I started to re-activate my plugins.
Step 4: Who is the troublemaker?
Successively I activated one plugin after the other, always checking if the blog was still running as it was supposed to, despite still looking like crap. To cut things short: I couldn’t find a single plugin that was causing issues. It was always like ‘activate plugin 1 -> fine -> plugin 2 -> fine…-> plugin 5 -> problems -> disable plugin 5 -> still problems -> disable all and start again leaving out plugin 5 -> plugin 7 causing problems…but running the apparent troublemakers on theier own –> everything is fine -> adding other plugins after startung with an apparent troublemaker -> still everything fine until something else caused problems’.
Nothing stood really out, nothing seemed to be the source of the issues. I couldn’t really find out what was causing the trouble and so I just wanted to restore the last backup.
Step 5: Hacked?
The process should be easy. Just tell Updraft Plus to restore the last backup and everything should be fine…I thought. I wanted to to this step by step. First the themes, then the plugins, then the database, and if nothing elese helps even all the uploads. After restoring the themes and expecting to see the old layout I saw…nothing. Just an empty, blank page. What???
So I started looking through the PHP-Code to see if I could at least find a reason there and I found several index.php files just containing the following lines:
// Silence is golden.
This seemed pretty strange and I asked ‘Google’ about that. And the first results pointed to discussions in the official WordPress-Forum about pages being hacked with these exact empty index files. I didn’t see any evidence of a hack but well could I be sure about that? First things first: don’t panic, change all passwords and scan Laptop and PC for intruders.
But the most important thing about these ‘silence is golden’ files is not that they directly point to a hack. No they are standard WordPress files! So if you encounter any of these, don’t freak out! The index files are just there to prevent directory listings. Again this shows: don’t trust everything you read on the internet and don’t trust every result of the searchengine of your choice. Even if it looks like this:
Read the posts carefully and think.
That being said I think that my blog wasn’t hacked. I had the usual Spam Bots trying to flood everything but nothing out of the ordinary.
In case you suspect a hack you might want to scan your WordPress database for suspical code:
Step 6: Backup and recreation
Nevertheless my problem still wasn’t solved and even got worse by restoring my old themes folder. I was getting tired and sick(er) of this and choose to do it the hard way. Which just meant spending a lot of time for uploading things. First I deleted the whole wordpress installation, than I installed a new clean wordpress environment. After this was done I again thought I could recreate the page by restoring the backups. But now Updraft Plus told me that all of the bigger files – plugins, galleries, uploads, had the wrong size and the backup couldn’t be restored. Great!
A well Updraft does nothing really special, they just zip several folders and transfer them to a cloud drive or your local drive for a manual backup. So I uploaded the files by sftp and unzipped them to the place they belonged. Afterwards I just had to reinstall some plugins, because of the clean-up in step one and two, and I had to recreate my custom sidebars.
All in all this took me a while and I can imagine nicer things to do on a sunday despite being sick in the first place. Could I have solved this faster/easier? Yes maybe. But with a sore thraot, a mild headache and a nearly constant running nose nothing is that easy. Hopefully everything is fixed now. And if something goes wrong with your (WordPress)-Blog: don’t panic and always create backups. They might not initially work as you thought but the always can save you a lot of trouble.