I believe these failures can we reduced though better utilization of existing technology, specifically though the magic of open source. Open source software is developed with the intention of being sharing with the general public. This results in code which has been peer reviewed resulting in an arguably more secure final product.

A problem I frequently encounter are managers who want to “reinvent the wheel”. The result is less time is spent on quality control, detailed planning, and true development. The Internet is teeming with thousands of open source projects covering all disciplines, which most developers already know. These projects can be used to fill in gaps, and cut out large chucks of development time. As an example, there are at least a half dozen legitimate content management system which are completely open source. When commissioned to build a website, does it really make sense to build a custom CMS? Doubtful.

So why does the wheel get redesigned every day? Generic managers. Typically managers do not fully understand how to utilize open source and by their nature like to keep everything under their control.

The solution? Train developers to be managers. The same way a factory worker trains to become a line manager — IT developers must establish a place in management. As technology continues to grow in sophistication, the way we develop needs to evolve and mature. Eliminate managers who cannot develop and you will significantly reduce IT failure rates. I guarantee it.

Message to Dell – Quality matter, just because you CAN build something cheap doesn’t mean you should. Laptops shouldn’t burn your lap and definately don’t need tacky glowing LED lights (producing more heat and wasting power).

Apple has always been known for outstanding quality, however they’re use has been limited to a select group of dedicated “Mac People”. The landscape is changing… PC’s are no longer limited to beige box towers. The introduction of the iPhone shows how computers are jumping into different arenas. The Mac Mini is an illustration of how our perception of a computer is all wrong and we need to re-evaluate our assumptions and expectations. Computers don’t have to be noisy and don’t have to waste space and power. The Mac Mini makes a prefect media center machine and can be used as a server as well. It’s no surprise that there are companies selling Mac Mini’s within data centers (Mac Mini Hosting) as a replacement for expensive rack mount servers. These little machines have a genius design! They use very little power and have no annoying loud fans. Apple is really on top of their game when it comes to envisioning where the market is headed.

The solution can be found at the following URL:

http://www.dougknox.com/xp/file_assoc.htm

This site also has registry fixes for a bunch of system file associations. In my case running the EXE and LCK repairs worked prefect.

When it rains it pours! It’s hard enough to keep websites running without hackers trying to break in. An even worse situation is when hackers simply want you off-line. A few days ago I experienced just that… my first DDOS (distributed denial of service) attack against a high-profile website we host.

The attacker original exploited some legacy code to gain access to the web sites administration tool. After thwarting their attack, our web server immediately started to hit “max client connections”. Even after restarting Apache the max client connections were reached within seconds.

We quickly created bad routes for IP addresses associated with the hack attempts; however this did not resolve the problem. Below is the syntax we used to add bad routes.

Route add –host xxx.xxx.xxx.xxx reject

The Apache log files left no clues as to who was connecting so we had to look elsewhere. After some quick searching we found a terrific PHP script which shows which IP’s are connecting to your web server too much. The script was just want we needed. After about a ½ hour of blocking listed IP addresses the problem went away. In all we blocked a little over 100 IP addresses.

If you’re ever in a situation where Apache is clearly being flooded with connections. Run this script! It will save you a ton of time identifying the attacker(s).

– BEGIN SCRIPT –

<?php

## Functions ##

function getIP($line) {
        ereg("[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}",$line,$regMatch);
        $ip = $regMatch[0];
        if($ip) return $ip; else return "false";
}

function processString($string, $size = 18) {
        $string = "[ ".$string;
        $length = strlen($string);
        $toAdd = $size - $length;

        for($x = 0; $x < $toAdd; $x++) {
                $string = $string." ";
        }
        $string = $string."]";
        return $string;
}

## Code ##

while (true) {
        $cmd = "netstat -n | awk '{ print $5 }'";
        exec($cmd, $netstatArray);
        $ipArray = array();

        foreach($netstatArray as $line) {
                $ip = getIP($line);
                if($ip != "false" && ip != "127.0.0.1") {
                        if(array_key_exists($ip, $ipArray))
                                                 {
                                                                        $ipArray[$ip]+=1;
                                                 }
                                                 else // if not, count=1
                                                 {
                                                                        $ipArray[$ip] = 1;
                                                 }
                }
        }

        asort($ipArray);

        system("clear");
        foreach($ipArray as $ip => $count) {
                if ($count < 15)
                        continue;
                echo processString($ip);
                echo "\t" .processString(gethostbyaddr($ip), 55);
                echo "\tTimes Accessed: " .$count ."\n";
        }

        echo str_repeat("-", 50) ."\n";
        exec("top -n 1", $top_str);
        preg_match("#load average:(.+)#i", $top_str[0], $match);
        echo "Load Average: " .$match[1] ."\n";
        echo str_repeat("-", 50) ."\n";
        echo 'Showing $count >= 15: (Escape with ctrl+c)' ."\n";

        sleep(10);
}
?>

I was also successful in merging two images together. gddrescue allows you to define the start position and attempts to rebuild the whole image. If you start at block 10,000,000 the image will leave “white” space for the first 9,999,999 blocks. This allows you to combine multiple chunks together. In my situation I had an image created with dd_rescue which needs applied to a more complete gddrescue image. All I had to do was point to image and define the start / end position.

About a week ago my wife’s laptop hard drive failed sudden. She had not been making regular backups of her data so now the goal was to restore as much data as possible as quickly as possible.

My first attempts at recovering the data were weak. My gut instinct was to reach for my trusty hirum boot CD which has a variety of recovery tools. Sadly, most of these tools either didn’t work or locked up when trying to read from the disk. It was time to get more serious…

After doing some additional research I found dd_rescue, a bulk copy tool similar to UNIX / Linux “dd”, but is designed to recover gracefully from disk errors. I was thrilled to find such a simple application specifically designed for this problem. I left the disk connected to a machine to recover data over night.

The next morning I found the entire process failed around 5GB. I tried to cancel and restart the process, but apparently dd_rescue doesn’t handle interruption very well. After some investigation I found a similar application called gddrescue. This package comes directly from the GNU foundation and has the added feature of defining a “log” so you can easily pickup where you left off. Clearly gddrescue has more functionality and is superior to dd_rescue.

I started the recovery around the point of failure using gddrescue now. This time when the process was interupted starting back up was easy. After about 7 days of work the drive is almost finished, however its missing the first 10,000,000 blocks from my dd_rescue image. It should be possible to merge the devices together, I’ll discuss that in part 2…

Here is a quick summary of how to use gddrescue /w Knoppix boot CD.

• Burn a copy of the latest Knoppix boot CD.
• Install your damaged disk in another computer, make sure the BIOS sees both disks.
• Boot using Knoppix and mount your good hard drive. Make sure you have more space then the entire size of the disk your trying to recover.
• Here is where it gets tricky, Knoppix doesn’t come with gddrescue, only dd_rescue. You can install gddrescue using a *.deb package.

• Execute the following command ' ddrescue /dev/hdb1 (damaged disk) /media/disk1/recovery.img (good disk) /media/disk1/log_file '

• Watch and wait. If you have to cancel simply re-execute the command.

A few warnings about data recovery…

• Whatever you do do not write any more data to a damaged disk.
• Work quickly to extract an image of the disk.  Run recovery tools against the image, not the damaged disk.
• Make backups so you don’t have to go through this.

Part 2 will cover the following…

• The use of file “carving” software to extract data from a raw partition.
• Merging two images together using gddrescue.
• The outcome of my week long recovery effort.

Dell PowerEdge 2850
Dual Xeon 2.8GHz CPU
2GB ECC DDR-SDRAM
RAID 5 LSI RAID Controller
6 x 73GB Maxtor SCSI Disks

Below is the complete saga with Dell broken down.

On 4/27/2006 around 10AM we attempted to log into our clients production web server which hosts their critical business application. Our logins were successful, but we were immediately being bumped back to the login screen. We checked the servers drive state and noticed it had rejected a hard drive from the array. To resolve the problem we attempted to restart the server, however it was unable to fully boot into Windows. FAIL at this point the server is completely unresponsive and will not bring up a login screen.

After some research we found the RAID control was corrupting the data which was being written to the disk. This had been going on for some time, and appears to have corrupted the winlogon.exe.

Within a short amount of time we were able to bring their corporate website back online on the backup server. However their business application took significantly longer to bring back up because of the frequently changing data. To retrieve the live data off the server we booted using a restore tool, and copies the files / database onto a spare hard drive. This was to ensure we had the 100% most recent version of data from any morning transaction. We were able to bring everything back online by 3:30PM on the backup server.

We immediately called Dell who recommended we upgrade the firmware server RAID controller. Dell pointed out that this specific machine had shipped with a firmware which had known problems. First thing in the morning on 4/28/2006 we upgraded the firmware to the recommend version. We also upgraded the motherboard BIOS firmware as recommended by Dell. After letting the server run we scheduled a turn-up for Tuesday May 2nd. This was intended to give the server time to burn-in and ensure the firmware fixed the problem. We also had to completely reinstall Windows 2003 Server + MS SQL server 2000 to bring the server back online.

On 05/02/2006 at 8PM we attempted to bring all the data back over to the production machine. Immediately we checked the servers drive state and noticed it had rejected another hard drive from the array. This is a sign that the problem was not fixed from our firmware update.

The next morning I called Dell back and they shipped overnight a new RAID “Key” chip, controller card memory, and a new backplane for the drives to mount into. We replaced all of this equipment and let the server “burn-in” to ensure this fix would work. We scheduled another launch date for 05/09/2006 after letting the server run over the weekend. At 7PM we meet at the office and moved the site files and database back over. At approximately 9PM, after a final reboot we noticed the server rejected another hard drive. To be safe we immediately moved the site back to the backup server.

On 05/16/2006 after heavy lobbying Dell shipped a new server which seems to have resolved the problem. After further inspection I noticed they changed SCSI hard disk vendors. It’s my theory there is something wrong between MAXTOR + LSI RAID, but at this point I cannot prove anything. The replacement Seagate’s seem to resolve the problem.

This is intended to be a heads up for anyone dealing with the same issue. Level 1 Dell server support seemed to have failed us here, however once the problem was escalated they took action quickly to ship a new server.

My first handheld computer was the Compaq iPaq 3850. I was happy with the device, but it didn’t offer much connectivity options back to home base. Also the device was a tad bulky to carry around in a pocket. It always seemed silly to carry around a cell phone and iPaq. The Treo 700w solves that problem by combined an amazing cell phone with all the Windows Mobile features you need. Although there is less screen real estate, the device seems much faster then my iPaq. They have definitely made major improvements in mobile computing. This palm device uses Microsoft Windows Mobile (scary). I’d prefer the traditional palm interface, but the ability to synchronize with Exchange without any special configuration too nice to pass up. I am willing to deal with some interface short-comings for this extra functionality.

Some of the most important features are:

• ActiveSync with Exchange server
• Slimmer profile then Treo 650. More like a 90’s cell phone then a handheld computer
• Clear bright screen
• Improved layout on desktop page

After using the Treo 700w for just a few days I would highly recommend it to anyone who wants to access all of their change contact, calander, and e-mail on the road. This is not only the best solution, but also one of the only solutions which works directly with Exchange. I am sure more smart phones will be released using this new technology, but for now this is by far the best handheld and phone I’ve ever owned.

As if it wasn’t hard enough to understand their support staff. Although this article points out this is just a manufacturing center. It will take Dell one step closer to finally outsourcing all of their technical / support staff. There is no doubt this will have an impact on their quality of service. I believe a company as big as Dell should have support safe in all different regions of the country. This would help do a number of things:

For more information about Dell checkout my article – Dell Laptops Suck