Below is a simple solution for operating AWstats within a clustered environment in CentOS 5. This solution should work for other operating systems as well. This explanation assumes you’ll be using two application nodes, with front-end directors. I have not tried this solution using three nodes.

• Install AWstats & NFS from yum repository.
• Create profile for your domain within /etc/awstats. (i.e. awstats.www.domain.com.conf)
• Configure NFS mounts on each node. This can be accomplished by modifying your /etc/fstab file as illustrated below.
• Adjust your awstats configuration file to point at both log sources now that the NFS mount has been created.
• On second node, mount /var/lib/awstats
• Configure nightly cron on first application node.
• Configure calling either awstats.pl should display the data processed data stored on node 1, regardless of which node the user is accessing.

Notes / Exceptions

Be sure to backup your Apache log files, and your AWstats data. In this solution the processed information will only be stored on the primary node

Full Setup Process

1. SSH APP1
2. yum install awstats nfs -y
3. nano /etc/exports
4. Add following lines:
/var/log/httpd APP2(rw,sync)
/var/lib/awstats APP2(rw,sync)
/etc/awstats APP2(rw,sync)
5. nano /etc/fstab
6a. Add lines, APP2:/var/log/httpd /var/log/httpd-APP2 nfs rsize=16384,wsize=16384 0 0
6b. mount /var/log/httpd-APP2
7. ln -s /var/log/httpd /var/log/httpd-APP1
8. service nfs start
9. chkconfig nfs on
10. cp /etc/awstats/awstats.model.conf /etc/awstats/awstats.www.domain.com.conf
11. nano /etc/awstats/awstats.www.domain.com.conf
12. Configure normally, exception add
LogFile="/usr/local/awstats/tools/logresolvemerge.pl /var/log/httpd-APP1/domain.com-access_log \
    /var/log/httpd-APP2/domain.com-access_log |"
13. SSH APP2
14. yum install awstats nfs -y
15. nano /etc/exports
16. Add following lines:
/var/log/httpd APP1(rw,sync)
/var/lib/awstats APP1(rw,sync)
/etc/awstats APP1(rw,sync)
17. nano /etc/fstab
18a. Add following lines,
APP1:/var/log/httpd /var/log/httpd-APP1 nfs rsize=16384,wsize=16384 0 0
APP1:/var/lib/awstats /var/lib/awstats nfs rsize=16384,wsize=16384 0 0
18b. mount /var/log/httpd-APP1
18c. mount /var/lib/awstats
19. ln -s /var/log/httpd /var/log/httpd-APP2
20. service nfs start
21. cp /etc/awstats/awstats.model.conf /etc/awstats/awstats.www.domain.com.conf
22. nano /etc/awstats/awstats.www.domain.com.conf
23. Configure normally, exception add
LogFile="/usr/local/awstats/tools/logresolvemerge.pl /var/log/httpd-APP1/domain.com-access_log \
    /var/log/httpd-APP2/domain.com-access_log |"
24. SSH APP1
25. crontab -e add line, perl /usr/local/awstats/tools/awstats_updateall.pl now
26. Configure Apache to expose awstats.pl, refer to awstats setup for additional information.
27. Confirm setup (df -h, perl awstats.pl -update -config=www.domain.com)

This should allow users accessing either node to display the same stats information compiled from both nodes access logs. Be sure to do backups!
Enjoy,
Dustin

-> Build two CentOS 4.4 Linux routers. Minimal installation plus development tools. In my situation I used a dual 350Mhz PII and tunnel router #2 was an old Pentium Pro 200Mhz. The dual 350Mhz is located at the main data center and the older Linux box is located at my house.

-> Download and configure vtund from www.vtund.info. There are a number of example configurations available on their website. Below is what I have used.


# IP Tunnel Server Configuration
cobra {
passwd XXXXXXX; # Password
type tun; # IP tunnel
proto udp; # UDP protocol
compress lzo:9; # LZO compression level 9
encrypt no; # Encryption
keepalive yes; # Keep connection alive

up {
# Connection is Up
# XXX.XXX.XXX.XXX – local, XXX.XXX.XXX.XXX – remote
ifconfig “%% XXX.XXX.XXX.XXX pointopoint XXX.XXX.XXX.XXX mtu 1450″;
};
down {
# Connection is Down
# Shutdown interface
ifconfig “%% down”;
};
}

# IP Tunnel Client Configuration
cobra {
passwd XXXXXXX; # Password
device tun0; # Device tun0
persist yes; # Persist mode
up {
# Connection is Up

# Assign IP addresses.
ifconfig “%% XXX.XXX.XXX.XXX pointopoint XXX.XXX.XXX.XXX mtu 1450″;
};
}
The configuration above is very simlar to the example configurations provided by vtund. There is more you can do here to make this connection more robust but for the purpose of this blog entry I’ll show it works in more detail. To make the connection between to the two servers you run the following command on the client machine.

vtund cobra

This should create a new network interface called tun0 on both servers. This will allow you to connect across the tunnel. To test this i recommend using the following commands.


On the client
tcpdump -i tun0
On the server
ping


You should see packets come across the interface and reply back. Once you have established the tunnel connection there is a bit more required to route public IP address both in and out of the tunnel interface. For the sake of this example we’ll use 10.10.10.0/24 as the network block your routing. You’ll want to be using a public IP block for this to work correctly not the interal block from the example!

-> Assuming 10.10.10.0/24 is already routed to the tunnel server you’ll now need to route to packets from the tunnel server to the client. To do this you simply type:

ip route add 10.10.10.0/24 dev tun0


This will push packets from the server -> client. Once the packets are on the client end of the tunnel however they do not have a correct route back to the sender. In my situation the tunnel client is located on a cable modem with a default gateway on a different network. My block of 10.10.10.0/24 cannot be routed across someone elses network. The solution is to flag packets as they enter into the tunnel client router. Once the packets get flagged we can apply an alternate route table to them. The alternate route cable is identical to the standard table except the default gateway is tun0 rather then the cable modem interface. Below is a simple script to generate a duplicate route table and add the make the default gateway the tunnel server:


ip route show table main | grep -Ev ^default | while read ROUTE ; do
ip route add table 200 $ROUTE
done
ip route add default via XXX.XXX.XXX.XXX table 200


Now that the alternate table has been created you can verify it works by typing:

ip route list table 200


This should be a simlar table to your main route table except the default gateway should be the IP address of the server-side of the tunnel interface. Now that this table is in place we need to start flagging packets. In order to do this we need to make some special rules in ip tables. Below are the commands to create the flagging rules:

iptables -t mangle -A PREROUTING -i eth2 -j MARK --set-mark=200
iptables -t mangle -A PREROUTING -s 10.10.10.0/24 -j MARK --set-mark=200
iptables -t mangle -A OUTPUT -s 10.10.10.0/24 -j MARK --set-mark=200

Once the iptable rules are in place you can check them by issuing the ‘iptables -nvL -t mangle’ command. In the output you should see packets hitting your new iptable rules. Once you have active hits you’ll need to add a rule that all packets with a flag need to use the alternate table. To create this rule you need the following

ip rule add fwmark 200 table 200

To verify the rule was added you can issue the ‘ip rule list’ command.
Once you have does this your ready to route packets in and out of the tunnel. In my situation I added a third network interface card running NAT for my home computers.

Once I found the source of my frustration I contacted Webtrends for a solution to this problem. They referred me to their professional development department to pay to have an application written to remove the pages from my log files. It sounds like this has been a problem with web trends in the past. This custom programming from webtrends was going to be very expensive. Why not development this small “frame stripper� application and integrate it into Webtrends!

Why should I spend thousands more dollars, and tons more man hours when webtrends has already written this application before. Why not simply include their completed code with the final product? I seriously doubt it would be hard to add-on to webtrends. My opinion is they are attempting to create another revenue stream. Customers either need to upgrade to WAY more page view licenses then they need to, OR pay for custom programming to fix the application. It’s a lose – lose situation.

I completely understand their desire to make money. I also believe if you have a giant website you should pay more for analytics. I feel as a customer I’ve fallen through the cracks and been overlooked. The situation which I am in is not that out of the ordinary. The same way a website needs to be accessible using different browsers, analytics programs need to be functional on different style websites, including frames.

Annual Page Views (and why they suck)
For our client we’ve ordered WebTrends 7 Small Business with an add-on of 1,000,000 annual page views to get them started. This should be plenty of page views per year right? WRONG. According to WebTrends you need to have licenses for all the page views you’re going to generate in a given year, which means, if I process logs from 2003 in 2005 those work against my 2,000,000 cap. Their solution is to call their sales staff and have them give me unlimited licenses while we import all the old data. Sure this might resolve the problem, but imagine how many people are out there ripping their hair out trying to import old data? This is the old application I know of which handles licensing is such an odd way. I believe WebTrends is more concerned about making money on their larger clients, then making their software user-friendly.

Interface Reviewed
Once I got the software working correctly, things started to look better. They have definitely created a very user friendly analytics solution. The user management seems to be top notice. I can tell they’ve pulled some ideas from their competition however. I see some similarities to both NetTracker & Urchin. The reporting layout is easy to follow, and makes better sense then many of the applications on the market. You can definitely tell WebTrends has been doing this for awhile. I can give them no fault on the application itself. They have definitely got their act together since WebTrends 6.0 log analyzer (which I still use for some websites).

Conclusion
Although the Interface is nice, the licensing is very poorly done. With that said, I cannot recommend this software solution. There are a number of other applications on the market which are more capable for significantly less investment. If you’re looking to buying WebTrends 7 Small Business I recommend you try NetTracker. If you still want to buy WebTrends be ready for a fight over licensing.

SPECS
Compaq Desktop
300MHz 64MB RAM
10GB Hard Drive
2 x Netgear FA311 NIC’s

INSTALL
Before installing NoCat we setup a DHCP server on the router. It turns out the installation for the DHCP server was more complicated then the hotspot! The process was very smooth and refined. These guys make it easy to quickly bring up a hotspot. The configuration was created automatically during installation. Because we run a cluster of access points we decided to just run two network cards, and do basically routing on the machine, if you only want one access point you can configure it with a wireless network card. We then connected three access point to a switch network behind the new “hotspot router� and everything worked prefect. As always we created some basic scripts to automatically start the services after a reboot and manage the servers log files since this was going to be an unmanaged solution.

TESTING
It worked exactly as expected. We configured the software to store users MAC addresses for 24 hours. Afterward the user would be forced to reread our splash page. The solution has been running for a full year now, and we’ve had no complaints.

CONCLUSION
We’d recommend this solution to anyone who is looking to launch a hotspot. The process is simple from beginning to end. All you need to do is install a vanilla version of your favorite Linux distribution, setup DHCP, setup NoCat, and your up and running. From the initial installation to launch the setup only took a few hours.

INSTALL OS
The installation process is the same as it has always been. Things went very smooth, and it automatically dedicated my 3ware RAID controller. I am very happy with 3ware controllers because of their extensive support in Linux. For the most part we’ve standardized our office on 3ware for our RAID controller needs.

INSTALL CPANEL
Now things are starting to get interesting. According to the CPanel’s website they completely support 64-bit CentOS 4.1. Right…

FAIL
Support and work-out-of-the-box are two completely different things. It turns out there are a number of issues between dependences which cause specific pages from working in 64-bit. The only solution at this point is to remove the RPM’s which conflict and move on without them. The simple fix worked and we’re back in business. The RPM which conflicts was something which we didn’t need anyway… Stupid eh?

TESTING
Now that we have it up and running everything seems extremely quick! We’ve added a group of test domains to the server to get an idea how it will hold up on a mild load. Everything seems to work perfectly. We’ve used only small personal sites to test, but for the most part everything has been smooth sailing once we fixed the 64-bit RPM issue.

SUMMARY
It’s my opinion that Linux + 64-bit is here. Obviously some software has some catching up to do, but for a most the simple tasks a Linux server does 64-bit is a viable solution. With the prices on AMD64 chips fairly low it makes a great value server. I would recommend one of these chips to anyone looking for a high performance server, but unable to afford the traditional Xeon-class chips.

Webmin
This terrific application was a great tool to get me started managing Linux boxes. It’s not something which I use every day, but there are a few tools in webmin which I couldn’t live without. Specifically I’ve found the send mail module to be very effective way to manage the queue, read users mail, and troubleshoot problem. Over the years I’ve used webmin less and less, but for anyone new to managing a Linux server this application is incredibly useful.

IPCheck Server Monitor
This is an application which I’ve just started to use. Mainly out of disappointment with freeware monitoring software. This application does cost money, but it will well worth it. The biggest benefit I’ve found is the ability to “test� the connection before sending an alarm. With other application every time my cable modem at home would blink the alarm would go off. Now I can run a final check of internet connectivity before reporting the problem. IPCheck also gives you a very easy to use web or desktop based management solution. It’s easy enough to a novice computer user can add / remove domains from being monitored. All of these features have definitely helped me sleep better at night.

MySQL Tools
I am not a SQL expert by any stretch; this application has given me exactly what I needed, when I needed it. Although not prefect it does offer a desktop equivalent to Enterprise Manager for Microsoft SQL. This allows our Microsoft developers to make any easy transition from MS-SQL > MySQL. It also give me management tools to add / remove users permissions with ease.

Some of the requirements we established:
1.) Quick processing, mail cannot queue longer then a few seconds. Remove load from current mail server.
2.) Detailed accounting, we need to know exactly why messages get blocked.
3.) Management, we need to be able to quickly add additional domains without complicated configuration. Release messages from queue which need delivered.

MailScanner
The solution we found which works best is Mail Scanner. Mail Scanner is an application which gives us the ability to integrate a number of application (SpamAssassin / ClamAV, etc) to give each message a complete score. This allows for use to run each message through multiple scans before eventually passing the message off to the primary mail server. This application gave us part of the solution, but didn’t offer the detailed accounting which we were looking for.

This is where Mail Watch comes into play.

Mail Watch is probably the most exciting piece of the configuration. It’s given us detailed accounting information, and lets us generate reports to get an idea on who is abusing our mail system. I would highly recommend this application package to anyone who is using Mail Scanner but wants to add an easy to use interface for quick accounting.

Some of the additional tweaks we’ve made include:

1.) Leaving the incoming queue in memory rather then writing it to the drive. This seems to speed up the scanning process tremendously.
2.) Purging the MySQL accounting data every quarter to keep database queries quick.

As we continue to grow we will likely add two MX servers balancing via DNS. We currently process around 20,000 messages per day on our Dual Xeon 1.8Ghz server with 2GB RAM, running a RAID 1 Mirror. The extra MX server should add some additional reliability as well.

So far the MX server has worked prefectly. We could not be happier. Special thanks to everyone who helped make this project come together.