Archive for category Software
AWstats in a clustered environment
Posted by dwright in Software, Technology on February 17th, 2009
Below is a simple solution for operating AWstats within a clustered environment in CentOS 5. This solution should work for other operating systems as well. This explanation assumes you’ll be using two application nodes, with front-end directors. I have not tried this solution using three nodes.
- Install AWstats & NFS from yum repository.
- Create profile for your domain within /etc/awstats. (i.e. awstats.www.domain.com.conf)
- Configure NFS mounts on each node. This can be accomplished by modifying your /etc/fstab file as illustrated below.
- Adjust your awstats configuration file to point at both log sources now that the NFS mount has been created.
- On second node, mount /var/lib/awstats
- Configure nightly cron on first application node.
- Configure calling either awstats.pl should display the data processed data stored on node 1, regardless of which node the user is accessing.
Notes / Exceptions
Be sure to backup your Apache log files, and your AWstats data. In this solution the processed information will only be stored on the primary node
Full Setup Process
1. SSH APP1
2. yum install awstats nfs -y
3. nano /etc/exports
4. Add following lines:
/var/log/httpd APP2(rw,sync)
/var/lib/awstats APP2(rw,sync)
/etc/awstats APP2(rw,sync)
5. nano /etc/fstab
6a. Add lines, APP2:/var/log/httpd /var/log/httpd-APP2 nfs rsize=16384,wsize=16384 0 0
6b. mount /var/log/httpd-APP2
7. ln -s /var/log/httpd /var/log/httpd-APP1
8. service nfs start
9. chkconfig nfs on
10. cp /etc/awstats/awstats.model.conf /etc/awstats/awstats.www.domain.com.conf
11. nano /etc/awstats/awstats.www.domain.com.conf
12. Configure normally, exception add
LogFile="/usr/local/awstats/tools/logresolvemerge.pl /var/log/httpd-APP1/domain.com-access_log \
/var/log/httpd-APP2/domain.com-access_log |"
13. SSH APP2
14. yum install awstats nfs -y
15. nano /etc/exports
16. Add following lines:
/var/log/httpd APP1(rw,sync)
/var/lib/awstats APP1(rw,sync)
/etc/awstats APP1(rw,sync)
17. nano /etc/fstab
18a. Add following lines,
APP1:/var/log/httpd /var/log/httpd-APP1 nfs rsize=16384,wsize=16384 0 0
APP1:/var/lib/awstats /var/lib/awstats nfs rsize=16384,wsize=16384 0 0
18b. mount /var/log/httpd-APP1
18c. mount /var/lib/awstats
19. ln -s /var/log/httpd /var/log/httpd-APP2
20. service nfs start
21. cp /etc/awstats/awstats.model.conf /etc/awstats/awstats.www.domain.com.conf
22. nano /etc/awstats/awstats.www.domain.com.conf
23. Configure normally, exception add
LogFile="/usr/local/awstats/tools/logresolvemerge.pl /var/log/httpd-APP1/domain.com-access_log \
/var/log/httpd-APP2/domain.com-access_log |"
24. SSH APP1
25. crontab -e add line, perl /usr/local/awstats/tools/awstats_updateall.pl now
26. Configure Apache to expose awstats.pl, refer to awstats setup for additional information.
27. Confirm setup (df -h, perl awstats.pl -update -config=www.domain.com)
This should allow users accessing either node to display the same stats information compiled from both nodes access logs. Be sure to do backups!
Enjoy,
Dustin
Routing public IP addresses across via virtual tunnel
Posted by dwright in Communications, Software, Technology on May 2nd, 2007
A recent situation came up which required that I move a group of servers off-site without changing their public IP address. The off-site location is behind my cable modem at my house. To solve the problem I enlisted the help of an open source application called vtun or Virtual Tunnel and IP tables on a standard CentOS 4.4 installation. Below is a outline of the steps I took to solve the problem.
-> Build two CentOS 4.4 Linux routers. Minimal installation plus development tools. In my situation I used a dual 350Mhz PII and tunnel router #2 was an old Pentium Pro 200Mhz. The dual 350Mhz is located at the main data center and the older Linux box is located at my house.
-> Download and configure vtund from www.vtund.info. There are a number of example configurations available on their website. Below is what I have used.
# IP Tunnel Server Configuration
cobra {
passwd XXXXXXX; # Password
type tun; # IP tunnel
proto udp; # UDP protocol
compress lzo:9; # LZO compression level 9
encrypt no; # Encryption
keepalive yes; # Keep connection alive
up {
# Connection is Up
# XXX.XXX.XXX.XXX – local, XXX.XXX.XXX.XXX – remote
ifconfig “%% XXX.XXX.XXX.XXX pointopoint XXX.XXX.XXX.XXX mtu 1450″;
};
down {
# Connection is Down
# Shutdown interface
ifconfig “%% down”;
};
}
# IP Tunnel Client Configuration
cobra {
passwd XXXXXXX; # Password
device tun0; # Device tun0
persist yes; # Persist mode
up {
# Connection is Up
# Assign IP addresses.
ifconfig “%% XXX.XXX.XXX.XXX pointopoint XXX.XXX.XXX.XXX mtu 1450″;
};
}
The configuration above is very simlar to the example configurations provided by vtund. There is more you can do here to make this connection more robust but for the purpose of this blog entry I’ll show it works in more detail. To make the connection between to the two servers you run the following command on the client machine.
vtund cobra
This should create a new network interface called tun0 on both servers. This will allow you to connect across the tunnel. To test this i recommend using the following commands.
On the client
tcpdump -i tun0
On the server
ping
You should see packets come across the interface and reply back. Once you have established the tunnel connection there is a bit more required to route public IP address both in and out of the tunnel interface. For the sake of this example we’ll use 10.10.10.0/24 as the network block your routing. You’ll want to be using a public IP block for this to work correctly not the interal block from the example!
-> Assuming 10.10.10.0/24 is already routed to the tunnel server you’ll now need to route to packets from the tunnel server to the client. To do this you simply type:
ip route add 10.10.10.0/24 dev tun0
This will push packets from the server -> client. Once the packets are on the client end of the tunnel however they do not have a correct route back to the sender. In my situation the tunnel client is located on a cable modem with a default gateway on a different network. My block of 10.10.10.0/24 cannot be routed across someone elses network. The solution is to flag packets as they enter into the tunnel client router. Once the packets get flagged we can apply an alternate route table to them. The alternate route cable is identical to the standard table except the default gateway is tun0 rather then the cable modem interface. Below is a simple script to generate a duplicate route table and add the make the default gateway the tunnel server:
ip route show table main | grep -Ev ^default | while read ROUTE ; do
ip route add table 200 $ROUTE
done
ip route add default via XXX.XXX.XXX.XXX table 200
Now that the alternate table has been created you can verify it works by typing:
ip route list table 200
This should be a simlar table to your main route table except the default gateway should be the IP address of the server-side of the tunnel interface. Now that this table is in place we need to start flagging packets. In order to do this we need to make some special rules in ip tables. Below are the commands to create the flagging rules:
iptables -t mangle -A PREROUTING -i eth2 -j MARK --set-mark=200
iptables -t mangle -A PREROUTING -s 10.10.10.0/24 -j MARK --set-mark=200
iptables -t mangle -A OUTPUT -s 10.10.10.0/24 -j MARK --set-mark=200
Once the iptable rules are in place you can check them by issuing the ‘iptables -nvL -t mangle’ command. In the output you should see packets hitting your new iptable rules. Once you have active hits you’ll need to add a rule that all packets with a flag need to use the alternate table. To create this rule you need the following
ip rule add fwmark 200 table 200
To verify the rule was added you can issue the ‘ip rule list’ command.
Once you have does this your ready to route packets in and out of the tunnel. In my situation I added a third network interface card running NAT for my home computers.
OpenTTD – My Addiction
Posted by dwright in Software, Technology on March 23rd, 2006
You may be wondering why I haven’t posted anything to this blog recently. Although I am busy getting married and taking care of a puppy; I must admit something. I am completely addicted to OpenTTD. OpenTTD is based on the original computer game Transportation Tycoon Deluxe. It’s a complete rewrite and has major improvements over the original. Although not widely played in the United States, there is a large following in Europe. There are always servers available to play on. For anyone who played the original, this is something you should definitely check out. I’ve wasted a massive amount of my time on this game, so I thought it’s high time I pass this addition on to others.
Webtrends Small Business Reviewed (cont.)
Posted by Administrator in Software, Technology on January 9th, 2006
Sometimes you just can’t get enough of a bad thing. Here is another rant about page view licenses, specifically those implemented in Web Trends 7.5b. After continually fighting with their support staff I came to the realization framed pages are counted as individual pages. The site we’re attempting to process contains 9 pages inside of a framed page. This means for every single page view, we’re getting 9 page views against our licensing. The site should be well under our 2,400,000 million yearly licenses, but with this calculation we’re using over a million per month!
Once I found the source of my frustration I contacted Webtrends for a solution to this problem. They referred me to their professional development department to pay to have an application written to remove the pages from my log files. It sounds like this has been a problem with web trends in the past. This custom programming from webtrends was going to be very expensive. Why not development this small “frame stripper� application and integrate it into Webtrends!
Why should I spend thousands more dollars, and tons more man hours when webtrends has already written this application before. Why not simply include their completed code with the final product? I seriously doubt it would be hard to add-on to webtrends. My opinion is they are attempting to create another revenue stream. Customers either need to upgrade to WAY more page view licenses then they need to, OR pay for custom programming to fix the application. It’s a lose – lose situation.
I completely understand their desire to make money. I also believe if you have a giant website you should pay more for analytics. I feel as a customer I’ve fallen through the cracks and been overlooked. The situation which I am in is not that out of the ordinary. The same way a website needs to be accessible using different browsers, analytics programs need to be functional on different style websites, including frames.
Webtrends Small Business Reviewed
Posted by Administrator in Software, Technology on December 19th, 2005
Project Overview
For one of our larger clients we’ve decided to move their website / database / analytics to a dedicated group of servers. While this project was fairly straight forward on the surface it became a real pain when it came time to setup their analytics software. For this project the poison of choice was Webtrends 7.5b. Working for a marketing driven company I’ve had the chance to use nearly all analytics at one point or another, however this was my first experiencing using the newest version of Webtrends. It didn’t take long before I learned to hate this application, just like its parent programs. There is one word which all computer people have learned to hate. Licensing. Nothing can drive a person insane more then having all the tools to fix something, but not having the “permissions� to use them. The worst part is we’ve spent over $1000 to be back to square one. The problem is how Webtrends handles licensing.
Annual Page Views (and why they suck)
For our client we’ve ordered WebTrends 7 Small Business with an add-on of 1,000,000 annual page views to get them started. This should be plenty of page views per year right? WRONG. According to WebTrends you need to have licenses for all the page views you’re going to generate in a given year, which means, if I process logs from 2003 in 2005 those work against my 2,000,000 cap. Their solution is to call their sales staff and have them give me unlimited licenses while we import all the old data. Sure this might resolve the problem, but imagine how many people are out there ripping their hair out trying to import old data? This is the old application I know of which handles licensing is such an odd way. I believe WebTrends is more concerned about making money on their larger clients, then making their software user-friendly.
Interface Reviewed
Once I got the software working correctly, things started to look better. They have definitely created a very user friendly analytics solution. The user management seems to be top notice. I can tell they’ve pulled some ideas from their competition however. I see some similarities to both NetTracker & Urchin. The reporting layout is easy to follow, and makes better sense then many of the applications on the market. You can definitely tell WebTrends has been doing this for awhile. I can give them no fault on the application itself. They have definitely got their act together since WebTrends 6.0 log analyzer (which I still use for some websites).
Conclusion
Although the Interface is nice, the licensing is very poorly done. With that said, I cannot recommend this software solution. There are a number of other applications on the market which are more capable for significantly less investment. If you’re looking to buying WebTrends 7 Small Business I recommend you try NetTracker. If you still want to buy WebTrends be ready for a fight over licensing.
Making Life Easy
Posted by Administrator in Software, Technology on September 26th, 2005
There are a few applications which have made my life significantly easier. I thought I would take a moment to express a special thank you to the following programs.
Webmin
This terrific application was a great tool to get me started managing Linux boxes. It’s not something which I use every day, but there are a few tools in webmin which I couldn’t live without. Specifically I’ve found the send mail module to be very effective way to manage the queue, read users mail, and troubleshoot problem. Over the years I’ve used webmin less and less, but for anyone new to managing a Linux server this application is incredibly useful.
IPCheck Server Monitor
This is an application which I’ve just started to use. Mainly out of disappointment with freeware monitoring software. This application does cost money, but it will well worth it. The biggest benefit I’ve found is the ability to “test� the connection before sending an alarm. With other application every time my cable modem at home would blink the alarm would go off. Now I can run a final check of internet connectivity before reporting the problem. IPCheck also gives you a very easy to use web or desktop based management solution. It’s easy enough to a novice computer user can add / remove domains from being monitored. All of these features have definitely helped me sleep better at night.
MySQL Tools
I am not a SQL expert by any stretch; this application has given me exactly what I needed, when I needed it. Although not prefect it does offer a desktop equivalent to Enterprise Manager for Microsoft SQL. This allows our Microsoft developers to make any easy transition from MS-SQL > MySQL. It also give me management tools to add / remove users permissions with ease.
MX Server
Posted by dwright in Software, Technology on August 8th, 2005
With the additional load from virus / spam filtering effecting the performance of our core mail servers, we decided to create an MX server to place in front of our current mail system. This server is designed to receive all incoming mail for our domains, and pass them off to our primary mail server. The server is also designed to accept outgoing messages to prevent users from distributing viruses.
Some of the requirements we established:
1.) Quick processing, mail cannot queue longer then a few seconds. Remove load from current mail server.
2.) Detailed accounting, we need to know exactly why messages get blocked.
3.) Management, we need to be able to quickly add additional domains without complicated configuration. Release messages from queue which need delivered.
MailScanner
The solution we found which works best is Mail Scanner. Mail Scanner is an application which gives us the ability to integrate a number of application (SpamAssassin / ClamAV, etc) to give each message a complete score. This allows for use to run each message through multiple scans before eventually passing the message off to the primary mail server. This application gave us part of the solution, but didn’t offer the detailed accounting which we were looking for.
This is where Mail Watch comes into play.
Mail Watch is probably the most exciting piece of the configuration. It’s given us detailed accounting information, and lets us generate reports to get an idea on who is abusing our mail system. I would highly recommend this application package to anyone who is using Mail Scanner but wants to add an easy to use interface for quick accounting.
Some of the additional tweaks we’ve made include:
1.) Leaving the incoming queue in memory rather then writing it to the drive. This seems to speed up the scanning process tremendously.
2.) Purging the MySQL accounting data every quarter to keep database queries quick.
As we continue to grow we will likely add two MX servers balancing via DNS. We currently process around 20,000 messages per day on our Dual Xeon 1.8Ghz server with 2GB RAM, running a RAID 1 Mirror. The extra MX server should add some additional reliability as well.
So far the MX server has worked prefectly. We could not be happier. Special thanks to everyone who helped make this project come together.
