Help my Joomla web site has been hacked!!

..or how do you keep your Joomla web site secure?

I've sat on this blog post for a few weeks as I wanted to separate any connection to the specific client for who this relates to.

So I'm sat in a hotel bedroom, idling away the time before bed, browsing a news sites when a skype window pops up on my Mac.

"Please help!!!! I've got 12 sites all on different servers and they keep getting defaced."

Now obviously I then ran through my usual set of questions:

1. "Are the sites running the latest version of Joomla?"
2. "Are the file and directory permissions set sensibly?"
3. "Do you monitor the extensions and make sure they are all up to date?"

And the answer was yes to all of the above.

Now you're probably thinking that sounds just like yourself, at least I hope it does, so how were the sites being hacked.

Firstly lets look at what the hack was.

On all 12 sites the index.php file of the template was being edited and hidden links to various sites were being inserted. The site owner was fixing this as soon as he saw it but it just kept on happening.

He suspected that there must be a security whole in Joomla or one of his extensions that was allowing a hacker to automate this "link insertion" but he couldn't see where and had heard nothing on the grapevine about any unpatched vulnerability.

The investigation

One of the first things I learnt is that "users lie" so I double checked the permissions and the versions of joomla and extensions installed to make sure they were the most recent. But the client was correct they were all up to date.

So how were they doing it?

Whenever you are doing forensic analysis it is important to preserve the data and if possible to carry out your investigations on a copy and not the live data.

So after taking an ftp backup of one of the sites I set about looking for anything unusual.

Check the files

My first thoughts were that a hacker had been able to modify one of the core joomla files to provide a gateway into the site so I ran a "diff" tool against the backup and a core joomla installation. But there was nothing immediately obvious as unusual there.

Now the site had many third party extensions installed and I really didn't want to have to download and install them all in order to do another diff. So I thought I would try a shortcut on the off chance that I would be lucky.

Experience with other hacked web sites had shown me that hackers often "encode" their files with base64 and gzinflate so I decided to "grep" the backup for those strings.

After removing several hits that I knew were legitimate as they were in core joomla files, and I'd already verified them, I was left with a few other hits that I couldn't be certain of.

They were in files that appeared to be part of an extension but as they were "encoded" and I wasn't familiar with the extension. They could be legitimate or they could be a "problem".

The sensible next step was to download the extensions directly from the developer web sites and do a "diff". (At least I was only going to have to check 2 extensions and not 20.)

Bingo!! The "encoded" files were not part of the extension.

So what were these encrypted files?

I do not recommend that you do this yourself!!!

If you have the required skills and tools then it is possible to "decrypt" the extra files to see what they do. In many cases you would need to do this as they require a password to run but sometimes this isn't necessary as they will just run.

It's actually not necessary to run these "extra" files as it's enough to know that they shouldn't be there and you can delete them.

Typically this file will be a "c99" script that is an incredibly powerful hackers toolkit that could allow a hacker to edit files, modify the database and various other naughty things. Below are some screen shots of a typical c99 script.

Pretty scary stuff!!!

How did they do it?

So if the site was always kept up to date, with the latest version of joomla and extensions, how did this file get there. It's fairly simple but to explain how lets take an example of a fictitious joomla extension called com_subway (yes I am sitting in a subway sandwich shop as I write this blog).

1. Site owner installs com_subway on their site.
2. An exploit is discovered in com_subway and a new version is released.
3. Site owner uninstalls com_subway and re-installs the new secured version of com_subway.
4. Hacker comes along and inserts links into the website template.

But wait you told me that the site owner made sure that they had the latest secured version so how did this happen. The answer is fairly obvious.

1. Hacker discovers "vulnerability" in com_subway
2. Using various "dork" techniques they discover web sites using com_subway
3. Hacker uses the vulnerability to upload the c99 script into an unrelated directory on the server
4. com_subway announce the availability of a security release
5. Site owner uninstalls current version of com_subway and removes all com_subway files
6. Site owner install secured version of com_subway
7. Hacker returns and accesses the c99 script they left behind and inserts links into the website template.

Did you spot the problem?

The site owner assumed that their site had not been hacked and that upgrading the vulnerable extension would suffice.

When you assume you make an ass out of you and me

The problem is that uninstalling an extension will only remove the files belonging to that extension. It will obviously not remove any other files. In this case the c99 script placed elsewhere on the server.

The same is true of a Joomla upgrade. This will only add or change files it will not remove any that should not be there.

So whilst ensuring that you are always using the most up to date versions of the software will prevent you from being hacked today it will not prevent you from being hacked yesterday.

Steps you can take

There are many different ways that you can limit your exposure to hacks like this but fundamentaly they are all take the same approach.

Take a full backup of your site as soon as it is configured and store it offline

Regularly compare the backup with the current site

Any new files will be quickly identified and you can then check them to make sure that they are not "naughty files

Obviously some directories, especially those used for string images or documents, will regularly have new files added to them and you need to be especially vigilant checking all new file additions here.

For this I would recommend using a "grep" tool, especialy if you have a lot of new files, to search through the files to ensure that they do not contain strings like gzinflate and base64.

More advanced users might write a script that traverses your web server's directories and produces a list of all the files and calculates an md5sum for each file. This script can then be run on a regular basis by a cron job and compared against a master list with any changes or additions emailed to you.

Remember just because you keep your server secure and your software up to date there is still the possibility that you were exploited yesterday, ready to be hacked tomorrow.