Category Archives: Security

Operation Prism

I’d started writing a piece about CALEA earlier this week, other things came up, and I thought it would just be another nonstarter. The US’s Communications Assistance for Law Enforcement Act (CALEA) is designed to make it easier for law enforcement to spy on criminals, which of course no law abiding person would want to impede, it’s been used successfully to assist in the conviction of countless numbers of criminals since it was came into force in 1995. Allowing law enforcement to tap into telephone conversations, forcing telecommunication companies to both allow the wiretapping and hide the fact from the user that their conversations are being monitored. Putting the burden on the supplier to provide the required monitoring equipment to the law enforcement agencies, which meant the installation of new hardware and software across the industry.

prismAll modern telephone switches (PBX) now have these features built into them. But today, more and more voice communication is carried out using diverse methods of internet telephony, there are various moves to expand CALEA it IP telephony (VoIP) which weren’t previously covered. A major controversy is that by expansion is that by the very nature of building back doors into secure systems, backdoors that will quickly become targets of criminals that are already abusing our systems.

Is it even possible to build a backdoor into a product such as Skype, as I understand it Skype in its standard mode of person to person VoIP connection is an encrypted session between the 2 users, with no ability to listen in except at either end point. Whole products would need to change, if you forced the session to travel via a 3rd party so monitoring by authorities were possible, this would increase traffic and latency a great deal.

Now yesterday there was a major news story on the BBC which first came to my attention US ‘orders Verizon to disclose millions of phone records’ (this headline has now changed).  Which exposes what later in the day proved to be the tip of the iceberg on what many would say was immense privacy abuse by the US National Security Agency (NSA) in collecting data on call session information from 10’s of millions of subscribers, not the content of the calls, but who’s been talking to who when and for how long, which was quite amusing as I’d just been reading ISACA’s take on “Big Data” just the day before. I can only think we’re talking about hundreds of millions of records a month. I don’t use Verizon, don’t know much about them, but I didn’t give it much thought, till later in the day it was exposed that this has been going on for years, and quite a number of telecoms and ISP have been supplying similar vast amounts of data to the NSA, as well as Google and Microsoft having passed on all manner of data including search queries, Microsoft apparently were the first to offer up the data in what we now know is called Operation Prism.

Personally I doubt there is anything the FBI or NSA would find interesting that I might have to say but if they’re going to start putting security holes in our systems we traditionally thought of as secure, I’m thinking that I may have to start looking using systems that are beyond their control, it’s not like there aren’t plenty of choices such as Cellcrypt and I expect we will see a lot more being developed, necessity being the mother of invention. It’s not like backdoors are the only issue, following the military leaks by Bradley Manning we are reminded of 2 things, the majority of confidentiality breaches are by insiders and all security can be bypassed.

Are the real threats, both criminals and terrorist that dumb to use insecure communications, the vast majority of the relevant court cases I have read about where IT and forensic evidence were involved were public threats or where cached information was found in browser history?

I am sure there’s a lot more to be said over the coming day, looks like interesting times ahead.

What’s a good backup plan?

Well the answer is pretty simple really; it’s one that’s been tested. It’s not the first time someone has come to me asking to restore a system claiming to have backups that I have been unable to help.

In this case they nearly did everything right, an online business with a very attractive and popular site in a niche market, who’d started retailing nearly a decade back on EBay, had expanded quite a bit needing as well as its own website, connectivity to a Sage accounts package in their office, Sage managed (past tense) everything for them, stock control, fulfillment, invoicing, supplier orders, credit control, contact management etc. so much so, the website became really only a front end.

The owners did take the initiative to implement a backup strategy; firstly they pay the websites hosting company to back up the website each night, which is what they understood it as, I did check this and in fact it’s a lot more, they backup the whole hosted virtual server instance, configuration and all. This comes with a suitable SLA that transfers the risk to the Hosting company. But really all the valuable business data is held on Sage.

Now here’s where the problem starts, although Sage has its own built-in backup application, which basically is a one click, choose a destination and click save type solution. Scheduling regular backups has never been simple enough for accounts staff, and as they knew best practice said it needed to go to either another media or even better off site, they opted to sign up for one of the many services where you install some software on the Server or Workstation and you can just forget about it.

So for the last couple of years, every night the server has been backing up to this off site service, they have even recovered a couple of word files and spread sheets that got corrupted or overwritten over that time, which has been very comforting for them. It’s so easy, browse the backup server file list, and drag and drop the file to be recovered. That’s exactly what they said to me “surely it’s that easy to just drag and drop like we have always done”. I guess no-one noticed when the first hard drive died some time back, as they told me that 2 of the 3 drives in the raid 5 died at the same time last 2 weeks ago, I don’t know if that’s even possible, definitely highly unlikely, but it’s safe to say the drives are beyond recovery. So they spoke to HP who rapidly sent them 2 replacement drives to rebuild their raid. They then spoke to the offsite backup service who’d told them they would need to rebuild the operating system, and re-install the applications preferably to its original state. It sounds like this was quite a shock to the company who thought they had a one click solution; they didn’t even have the original installation media handy, or know the server license numbers. It took them a couple of days to gather the prerequisites and re-provision the server, remaking user accounts with the help of a local “IT guy”.

This is where I came in, I have worked with the “IT Guy” in the past, and he suspected the worst but either wasn’t so confident in his own understanding or better, was unsure of how to explain the dire situation, so asked me to speak to the client. I hope one day the client can revisit this page to try to help him understand what went wrong and why I was unable to help him, he dismissed my negativity and last I heard is looking elsewhere “for someone that knows their job and what they are talking about”.

The reason all the data files bar one were recovered is quite simple, that one is the single file which is the entire database; the Sage database could never be backed up without specialist intervention either using the built in application or some other “open file” solution. The offsite backup solution clearly warns you that open files may not back up correctly. I haven’t seen the historical logs, but I am sure you got regular warnings similar to “one or more files were not successfully backed up” which you ignored, after all what’s one file amongst the 20,000+ that are saved? Your whole business revolved around the availability of that one file.

The solution would have been simple, if Sage had been scheduled to automatically make a local backup of the database locally, that backup would have been copied off site every night. Testing recovery of the backed up file is very simple in Sage as you can open up the archived copy side by side with the live instance, this should have been done occasionally and you would have noticed the massive hole in your contingency plan. I’m sorry I just cannot see a solution to this disaster, just a lot of “if only…”.

Spammers, its nothing personal

Over the last week I’ve been helping out a couple of bloggers on WordPress issues.

Both have similar problems, despite having captchas enabled on registration. Both have ridiculous numbers of fake subscribers (one, 5000+ a week), an equally high number of spam, although Akismet catches the vast majority and both these popular blogs that have been around for well over 5 years, being on fairly good hosting plans, with no exotic plugins enabled, are seeing frequent and random outages.


Off course this is distressing for them; the hosts in one of the cases where it’s a managed service deny any liability and offer no advice.  Both blogs have any number of competitors that would happily see them offline and knowing how easy and cheap it is for anyone with a PayPal account to launch a denial of service attack, but that wouldn’t make sense due to randomness.

So having installed a simple Event Viewer, I quickly see a second problem, huge numbers of attempts at guessing the password for “admin” the default account for wordpress, which in both these blogs cases has been removed which is simple best practice, strangely enough I also see attempts at the “administrator” which is strange as that’s only really used on windows operating systems.

While I wouldn’t normally worry too much about password guessing on the “admin” account if none exists, the frequency could be to blame here. The fact so many large numbers of spammer accounts are being so quickly being created is resolved by increasing the captcha complexity and finding the pain threshold of potential sites commentators, and in one case deleting an immense number of obviously fake accounts, upon agreement with the client, we were aware that quite a large number of valid accounts were also deleted.

When I logged back in to one of them, the spam has died back quite a bit, and in the last 24 hours I can’t see one new spammer account created, but I notice a number valid accounts have been recreated, so deem that a success. But on both of them I’ve found the password guessing to have ramped up quite a bit, one of them quite a concerted effort employed, multiple IP addresses a second or less between each attempt. Now I have not had a great experience on WordPress with the firewall widgets available, finding they can put a greater load on the installations than the actual abuse, but I installed what appears to be a reasonably lightweight one (won’t name till it’s proved itself), with the pleasant addition of a username blacklist which I quickly added “admin” and “administrator” to.

Again I am asked if Disqus could be an option, well I’ve tested it myself on demo installations with default templates and no other plugins, and it’s worked without an issue. But when I’ve backed up, and restored a live site with a few years (50,000+) of comments, it failed miserably and due to client not wanting to potentially pour more money down the drain, we gave up. I think it I best quantify my objection to Disqus, firstly reviews are not as great as I would like to see on a widget that can’t simply be switched off and reversing a larger installation could potentially take days of work and disruption. Secondly they seem to have full control over what really is your content which might be ok today (I really don’t know), but what if they suddenly stop providing the service either generally or for a reason related to your site specifically, what happens to your content, or they start adding inappropriate advertising.

All seems well with these blogs now, time will tell, the spammers will no doubt find another way to sell their fake designer goods and Viagra. It’s not malice, it’s just their job.