SquidFire 1.4.0 is shaping up to have more changes than all the previous versions combined. This is why its taking longer than normal to get it out the door. The ability to browse Squid Log files by IP or domain will be in this release as well as a massive code re-organization. The code cleanup will allow features to be easily added in the future with much less work.
I don't think I'll have it ready to go until next weekend, but I'll try to post some screen shots before then of the current development.
SquidFire 1.3.0 is the first version I announced on Freshmeat. As a result I learned that something is horribly wrong with my sourceforge.net email address. I've had trouble with it from the beginning and I thought they were resolved but apparently not. As a result, I am switching my primary contact address for this project to one of my personal email addresses, grok@mozillanews.org.
As a result of this problem, if you have emailed me and not received a response, I am truely sorry. Please send it to me again and I promise I will respond as quickly as possible. If you haven't emailed me, what are you waiting for? Even if you just want to let me know your using SquidFire, let me know.
Finally after an entire two weeks of waiting, the new version of SquidFire is ready to go. Be sure to take a look at the new screenshots and the changelog.
Grep is no longer used to filter the log files. Instead, all filtering is done in PHP which results in much faster searching especially on compressed logs. However, because SquidFire now uses PHP to read all log files, PHP must be compiled with the --wigh-Zlib option.
There is now a lot more information displayed in the results header including how many rows were searched, how many rows total are in the log file, how long the search took to complete and how many results were returned. Unlike the previous version, all these stats always show at the top of the results table and not at the bottom.
When the "Only show web pages" option is not checked, all web pages show up in bold text for easy identification. All secondary requests such as images, javascript, etc. are indented.
The columns displayed are not completely user configurable via a new option called "columns". The default columns now included request size. Request size is now also a search option so you can find out who is downloading those 200mb files.
In addition to all the above major changes, there are many small changes and bug fixes.
I knew I couldn't keep up the weekly release pace for very long and sure enough, no new release this week. I should have a release out by next week, so don't forget to check back on a regular basis so you can get it hot off the text editor.
As for whats planned for the next release, I've done some testing and I've decided to drop grep in favor of PHP's file handling functions. From my tests with a 1.5 million line gzip compressed log file, PHP should be an order of magnatude faster than grep. PHP can read and parse the entire log file in 96 seconds. With a more reasonable number of lines, say 100K, it only takes 15 seconds. I was having performance problems with 10K lines using grep. This plus the functionality differences between some BSDs has lead to the decision to drop grep.
I have also got a pretty good "deep processing" library coded. This will allow for some pretty interesting data mining. For instance, the question "What are the 10 most popular sites after 5pm" or "What domain has been visited every day between 5pm and 6pm during the last week". The questions are by no means limited to these two examples and are only limited to your or my imagination. For instance, am tracking the following about each unique IP I find in the system:
- Total number of hits
- How many hits for each domain
- How many hits for each day of the year
- How many hits for each hour of the day
- How many seconds spent surfing
- How many bytes transfered
Most of the above is tracked per domain, date and hour too, so the data can be sliced in hundreds of ways depending on what your looking for. All this comes with a price though. Using the 1.5 million line log file test it takes about 5 minutes to generate the staticts and the resulting stats are about 12mb. Once I have a final list of what I'm going to track I'm sure I can reduce this by at least half.
Another week, another version of SquidFire. As always, we put a lot of work into making this version even easier to get up and running with the addition of a setup wizard. The wizard runs automatically on new installs and does its best not to ask any questions that can be determined by scanning the system. For instance, it only takes one step to setup a standard RedHat server.
We didn't forget about all our existing users who don't care about new setup wizards. SquidFire can now parse Common Log formated Squid and Apache logs so you can search any log format squid outputs plus do quick and dirty searches through your Apache logs. Also added is the ability to restrict search access with the new "$password" option, which will prompt for a password before allowing access to SquidFire. Lastly, for those of you running grep 2.5 or better, expect to see as much as a 10x speed improvement on simple searches on large log files.
Be sure to checkout the new screenshots.