I recently decided to support there the “linux community” by hosting 1 mirror. I did it few years ago for Sabayon on a small server in France (Kimsufi) as I had some spare space and bandwidth, but had to turn it down after a year.
As I’m leaving in Vietnam now and the connectivity here is not great for the least to say, having a mirror here in Vietnam makes a lot of sense and with a real needs for Vietnam and neighborhood countries. Some bandwidth and few hundreds GB after, the mirror is live at mirror.freedif.org on selected projects with no or limited presence in Vietnam/Asia.
So the next step was to find a way to monitor my bandwidth and understand where all that data goes, by project with a daily records and with some stats on which files is the most downloaded, etc…
There are several ways to monitor your bandwidth (MRTG, munin, etc..) but none of them satisfy me (either because they can’t do what I need, or I don’t know how to configure it to do it). On top of it, I wanted something fresh with a fancy interface to check the stats.
Well and here goes GoAccess, it’s basically a log reader in command line but can also display the results in a nice HTML page. Here is a live demo and here are the stats of my mirror: http://mirror.freedif.org/Stats/.
Want to do the same? Here it’s how.
Good news, GoAccess is available in several repo. But to have the latest stable version, you may want to add GoAccess official repository to your source.list.
Assuming you are using Debian/Ubuntu style of distribution,
1) Add GoAccess repo
echo "deb http://deb.goaccess.io/ $(lsb_release -cs) main" | sudo tee -a /etc/apt/sources.list.d/goaccess.list wget -O - https://deb.goaccess.io/gnugpg.key | sudo apt-key add - sudo apt-get update
2) Install GoAccess
sudo apt-get install goaccess
3) Install optional dependencies
You might be interested in installing couples of dependencies to further improve your statistics, like GeoIP and Ncurses
To do so, simply run:
sudo apt-get install libncursesw5-dev libgeoip-dev
The GeoIP package will significantly help to determine from which country/city your traffic come from.
Basically, we will need to do few configuration in GoAccess then update our virtualhost and logs (To split them) and run a cronjob to get a regular refresh of the stats.
1) GoAccess conf
Edit the GoAccess configuration file
In root or with sudo, type:
and uncomment the following lines:
time-format %H:%M:%S date-format %d/%b/%Y log-format %h %^[%d:%t %^] "%r" %s %b "%R" "%u" #log-format COMBINED
This is basically to use the standard Apache (with virtualhosts) formats for time, date and log.
2) Customized your virtualhosts
You need to have 1 log file for each website or folder in order to split the stats.
Assuming you have 1 virtualhost for 1 website that you want to monitor, in each of those virtualhost, add something like this:
CustomLog /var/log/apache2/mirrorlog/fullmirror.log combined
As you can see, I have created a dedicated folder to store all my logs. Ensure that you have a unique log file for each of them.
- If like me, you are having multiple folders to monitor, but using only 1 virtualhost, (I got a mirror virtualhost, but several projects inside, I want to monitor each of them), the above command won’t suit you, as you need to split the log
In that case, you need to add something like this instead:
SetEnvIf Request_URI "^/Antergos/.*$" Antergos CustomLog /var/log/apache2/mirrorlog/Antergos.log combined env=Antergos
On the first line, “/Antergos/” is the folder name and Antergos the environment name (ID) to be reused below.
The second line is basically there to create the log file for the environment Antergos.
Redo this step as often as you need for each of your folder you want to monitor. (6 in my case)
3) Generate the report
What we want to do is to generate the HTML report for each of the folder or virtualhost and copy them somewhere on the folder that can be accessed to render them. (In my case, they are accessible under mirror.freedif.org/Stats (a simple folder containing all the html reports).
To generate the HTML report, you basically need to run the following:
goaccess -f /var/log/apache2/mirrorlog/fullmirror.log -o /var/www/mirror/Stats/Combined.html
The first path indicates which log file to read and the option -o with the path, indicates where to write it. (On my /Stats folder accessible online)
You need to do it for each log file you have.
But what you will prefer to is to have a Cron task to do the update when you want (every hour for example). To do it, open your cron (in root or with sudo)
0 * * * * goaccess -f /var/log/apache2/mirrorlog/fullmirror.log -o /var/www/Stats/Combined.html
for a hourly update.
Simply do the same for all your log files you want to monitor.
That’s it ! You know have a hourly updated bandwidth statistics for all your virtualhosts or folders.