Last night I had a Windows 2003 web server become compromised. It appears the attacker deleted the boot.ini / NTLDR files to prevent the system from starting up. The problem was a little tricky to troubleshoot, but with the right tools I was able to resolve the issue relatively quickly. Incase anyone runs into the same problems below are a good set of steps to troubleshoot.


[boot loader]
timeout=30
default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS

[operating systems]
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows Server 2003, Standard" /fastdetect


I believe I dodged a bullet here. The web server was still had all the site files, and all the data. The individual beyond this could have easy do more damage then they did.

One of the biggest parts of my job is managing website log files. Our business is very “marketing-centric�, so missing log file data isn’t expectable. There have been a few situations were log files have been split or broken. Below are three nice little commands which have helped tremendously.

Removes all log files with no data
find /home/httpd/logs -size 0 -type f -print0 | xargs -0 rm –f
This is a simple little command which helps keep things neat and organized. Once files rotate I’ve found a bunch of empty files on the folder. This really confuses some of the marketing people.

Converts log files extention into logical month.
for i in `dir /home/httpd/logs/*-access_log.1|sed s/\.1$//`; do mv $i.1 $i.feb; done
A little more complicated. Again, the overall goal here is to make the log files more friendly for marketing people connecting into the FTP server. I am converting the *.1 extension file created by the log rotate into *.month. I use this single line to change the extension on 300+ files. Works fantastic!

Combines log files with extentions .1.1 -> .1 and creates a new logical month.
for i in `dir /home/httpd/logs/*-access_log.1|sed s/\.1$//`; do cat $i.1.1 $i.1 >$i.mar; done
Here is the big daddy. One month because of a mistake the log files rotated mid-month. This would have been a nightmare to manually concatenate each file. This script above takes the files and combines them. Try to do this in windows!

For the most part I’ve learned to hate access logs. They are a big, always changes, difficult to backup, and from a technical perspective somewhat useless. These little few commands above however make life a little easier.

Enf’said.

Over the weekend it appears my “play” FreeBSD / CPanel server decided it no longer wanted to “play”. Although it can still serve up webpage’s, it appears to have lost most of its shared libraries (breaking MySQL / CPanel). Rather then try and restore the machine I am just moving everything over to our core web server. It makes sense to keep everything in one place, right?

This experience as taught me a very important lesson; although I know how to use FreeBSD it doesn’t makes sense to use it in a production environment until I really know it. With Linux I would have been able to get things running again much faster. In this odd situation I decided to call my CPanel vendor to see if they would be able to provide any insight. They were extremely helpful, but had limited experience with FreeBSD. Again a lot of pain could have been avoided had I just used Linux.

I have most of the websites and databases moved over to another server, but right now I am trying to resolve to e-mail issue. CPanel handles e-mail differently then our current mail solution. If I want to completely cut out the CPanel server I’ll need contact people. I’ll probably post my experience making that switch over later. Talking to people sucks.

CONCLUSION

I’ve been playing with control panels for a about a year now, the idea sounds great, but implementing them effectively is another story all together. By giving end-users access to system resources you open up a bunch of stability issues. After careful consideration, the benefits of control panels simply don’t out weight the security / stability of keeping a tight lockdown.

Then again, maybe I’m just bitter because mine crashed and burned.

Approximately one week ago I was given an assignment to create a new dedicated server for a client. This would not be worth discussing in any normal situation, but as usual around here, this was a far from a normal situation. The client, who will remain nameless, wanted to host an advertisement splash page for an e-mail marketing campaign. Starting yesterday, they are planning to send out 60 million e-mails. To compound the matter I had no budget and three days to develop a robust, reliable solution. Needless to say I was concerned.

THE SOLUTION
OpenBSD has long been a pillar of security and reliability in the IT community. Until now I have never needed to use it. I have had experience with FreeBSD, and of course extensive experience with Linux. OpenBSD has always been on my list of exciting projects, but I have never had the opportunity to install it. With no hardware firewall, pissing off 60 million people requires a fairly secure operating system. Also, with little time to secure the server, I wanted a solution which was relatively safe out of the box. Below is the specific equipment

Dell OmniPlex GXPro
Intel Pentium Pro 200Mhz
256MB RAM
4GB Hard Drive

Yes that’s right, 60 million people are going to hit my old Pentium Pro. This might seem a little odd with the abundance of cheap equipment available. Frankly, I wouldn’t trust this project with anything else. This old workstation (around the office referred to as Old-Deller) is solid as a rock. Beside, it’s only going to be handing out two pages of static content. What could possibly go wrong?

I’ll admit the initial installation was a bit intimidating. Any time the install software recommends a calculator, you should be concerned. After some research I found a very helpful setup guide. I did not follow this exactly, but it helped me get past some of the initial sticky points, and gave me some helpful tips. If you’re going to install OpenBSD for the first time I recommend you read this guide.

Although the jury is still out, for now I can say this operating system truly rocks. Its fairly light weight so nothing get installed which you don’t need. If you like FreeBSD / Linux I recommend giving OpenBSD a try. It makes a prefect secure web server, firewall, or just about anything other mindless server job.

Below is a fairly interesting review of the latest release of OpenBSD.
http://os.newsforge.com/article.pl?sid=05/05/20/1426216&tid=8