The SAR suite of utilities originated in Solaris. It became popular and now runs on most flavors of UNIX, including AIX, HP-UX, and Linux. Sysstat package is installed by default in standard Red Hat installation. For Suse it is not installed by default and you need to install sysstat package manually (package is provided by Novell).
The reason for sar creation was that gathering system activity data from vmstat and iostat is pretty time-consuming. If you try to automate the gathering of system activity data, and creation of periodic repots you naturally come to creation of a tool like sar. To avoid reinventing the bicycle again and again, Sun engineers wrote sar sar (System Activity Reporter) and included it in standard Solaris distribution. The rest is history.
Linux reimplementation is a part of sysstat package and like in Solaris sar pulls its data out of the virtual /proc filesystem (pioneered by Solaris).
The sysstat package contains the sar, sadf, iostat, mpstat, and pidstat commands for Linux. The sar command collects and reports system activity information. The statistics reported by sar concern I/O transfer rates, paging activity, process-related activites, interrupts, network activity, memory and swap space utilization, CPU utilization, kernel activities, and TTY statistics, among others. The sadf command may be used to display data collected by sar in various formats. The iostat command reports CPU statistics and I/O statistics for tty devices and disks. The pidstat command reports statistics for Linux processes. The mpstat command reports global and per-processor statistics.
In addition to sar, Linux package provides several other useful utilities:
sadf(1) -- similar to sar but can write its data in different formats (CSV, XML, etc.). This is useful to load performance data into a database, or import them in a spreadsheet to make graphs.
iostat(1) reports CPU statistics and input/output statistics for devices, partitions and network filesystems.
mpstat(1) reports individual or combined processor related statistics.
pidstat(1) reports statistics for Linux tasks (processes) : I/O, CPU, memory, etc.
nfsiostat(1) reports input/output statistics for network filesystems (NFS).
cifsiostat(1) reports CIFS statistics.
As in Solaris there are two main binaries and two shell scripts that constitute sar package:
Binaries
/usr/lib64/sa/sadc -- System activity data collector binary, a backend to the sar command. Writes binary log of kernel data to the /var/log/sa/sadd file, where the dd parameter indicates the current day
/usr/bin/sar -- reporting utility
Scripts
/usr/lib64/sa/sa1
/usr/lib64/sa/sa2
System activity data collector binary can monitor a a large number of different metrics, selectable by options:
Input / Output and transfer rate statistics (global, per device, per partition, per network filesystem and per Linux task / PID).
CPU statistics (global, per CPU and per Linux task / PID), including support for virtualization architectures.
Memory and swap space utilization statistics.
Virtual memory, paging and fault statistics.
Per-task (per-PID) memory and page fault statistics.
Global CPU and page fault statistics for tasks and all their children.
Process creation activity.
Interrupt statistics (global, per CPU and per interrupt, including potential APIC interrupt sources, hardware and software interrupts).
Extensive network statistics: network interface activity (number of packets and kB received and transmitted per second, etc.) including failures from network devices; network traffic statistics for IP, TCP, ICMP and UDP protocols based on SNMPv2 standards; support for IPv6-related protocols.
NFS server and client activity.
Socket statistics.
Run queue and system load statistics.
Kernel internal tables utilization statistics.
System and per Linux task switching activity.
Swapping statistics.
TTY device activity.
Power management statistics (CPU clock frequency, fans speed, devices temperature, voltage inputs).
Script sa1 calls sadc to write stats to the file
#!/bin/sh
# /usr/lib64/sa/sa1.sh
# (C) 1999-2006 Sebastien Godard (sysstat wanadoo.fr)
#
umask 0022
ENDIR=/usr/lib64/sa
cd ${ENDIR}
if [ $# = 0 ]
then
# Note: Stats are written at the end of previous file *and* at the
# beginning of the new one (when there is a file rotation) only if
# outfile has been specified as '-' on the command line...
exec ${ENDIR}/sadc -F -L 1 1 -
else
exec ${ENDIR}/sadc -F -L $* -
fi
Script sa2 produces reports from the binary file
#!/bin/sh
# /usr/lib64/sa/sa2.sh
# (C) 1999-2006 Sebastien Godard (sysstat wanadoo.fr)
#
# Changes:
# - 2004-01-22 Nils Philippsen
# make history configurable
#
HISTORY=7
[ -r /etc/sysconfig/sysstat ] && . /etc/sysconfig/sysstat
[ ${HISTORY} -gt 25 ] && HISTORY=25
S_TIME_FORMAT=ISO ; export S_TIME_FORMAT
umask 0022
DATE=`date +%d`
RPT=/var/log/sa/sar${DATE}
ENDIR=/usr/bin
DFILE=/var/log/sa/sa${DATE}
[ -f "$DFILE" ] || exit 0
cd ${ENDIR}
${ENDIR}/sar $* -f ${DFILE} > ${RPT}
find /var/log/sa \( -name 'sar??' -o -name 'sa??' \) -mtime +"${HISTORY}" -exec rm -f {} \;
Here are files installed:
/etc/init.d/sysstat
/etc/sysstat
/etc/sysstat/sysstat
/etc/sysstat/sysstat.cron
/etc/sysstat/sysstat.ioconf
/usr/bin/iostat
/usr/bin/mpstat
/usr/bin/pidstat
/usr/bin/sadf
/usr/bin/sar
/usr/lib64/sa
/usr/lib64/sa/sa1
/usr/lib64/sa/sa2
/usr/lib64/sa/sadc -- System activity data collector.
/usr/sbin/rcsysstat
/usr/share/doc/packages/sysstat
/usr/share/doc/packages/sysstat/CHANGES
/usr/share/doc/packages/sysstat/COPYING
/usr/share/doc/packages/sysstat/CREDITS
/usr/share/doc/packages/sysstat/FAQ
/usr/share/doc/packages/sysstat/README
/usr/share/doc/packages/sysstat/TODO
/usr/share/doc/packages/sysstat/sysstat-8.0.4.lsm
/usr/share/locale/af/LC_MESSAGES/sysstat.mo
/usr/share/locale/da/LC_MESSAGES/sysstat.mo
/usr/share/locale/de/LC_MESSAGES/sysstat.mo
/usr/share/locale/es/LC_MESSAGES/sysstat.mo
/usr/share/locale/fr/LC_MESSAGES/sysstat.mo
/usr/share/locale/it/LC_MESSAGES/sysstat.mo
/usr/share/locale/ja/LC_MESSAGES/sysstat.mo
/usr/share/locale/ky
/usr/share/locale/ky/LC_MESSAGES
/usr/share/locale/ky/LC_MESSAGES/sysstat.mo
/usr/share/locale/nb/LC_MESSAGES/sysstat.mo
/usr/share/locale/nl/LC_MESSAGES/sysstat.mo
/usr/share/locale/nn/LC_MESSAGES/sysstat.mo
/usr/share/locale/pl/LC_MESSAGES/sysstat.mo
/usr/share/locale/pt/LC_MESSAGES/sysstat.mo
/usr/share/locale/pt_BR/LC_MESSAGES/sysstat.mo
/usr/share/locale/ro/LC_MESSAGES/sysstat.mo
/usr/share/locale/ru/LC_MESSAGES/sysstat.mo
/usr/share/locale/sk/LC_MESSAGES/sysstat.mo
/usr/share/locale/sv/LC_MESSAGES/sysstat.mo
/usr/share/locale/vi/LC_MESSAGES/sysstat.mo
/usr/share/man/man1/iostat.1.gz
/usr/share/man/man1/mpstat.1.gz
/usr/share/man/man1/pidstat.1.gz
/usr/share/man/man1/sadf.1.gz
/usr/share/man/man1/sar.1.gz
/usr/share/man/man8/sa1.8.gz
/usr/share/man/man8/sa2.8.gz
/usr/share/man/man8/sadc.8.gz
/var/log/sa
To activate sar you need to either to create a link in /etc/cron.d to /etc/sysstat/sysstat.cron (approach taken by Suse) or copy the file (simpler approach used by Red Hat).
On Suse /etc/init.d/sysstat creates and deleted symbolic link to cron.d on start and stop commands :
lrwxrwxrwx 1 root root 25 Sep 12 10:32 sysstat -> /etc/sysstat/sysstat.cron
here is the fragment of the script that does that
case "$1" in
start)
echo "Running sadc"
/usr/lib64/sa/sa1 1>/dev/null 2>&1 \
&& ln -fs /etc/sysstat/sysstat.cron /etc/cron.d/sysstat \
|| rc_failed 1
rc_status -v
;;
stop)
echo "Removing sysstat's crontab"
rm -f /etc/cron.d/sysstat
rc_status -v
;;
This is a neat trick which permits running sadc only on certain runlevels as well as the ability to enable/disable data collection anytime.
On Red Hat /etc/init.d/sysstat the script inserts message LINUX RESTART and tell sar that the kernel counters have been reinitialized (/usr/lib64/sa/sadc -F -L -).
case "$1" in
start)
echo -n "Calling the system activity data collector (sadc): "
/usr/lib64/sa/sadc -F -L - && touch /tmp/sysstat.run
# Try to guess if sadc was successfully launched. The difficulty
# here is that the exit code is lost when the above command is
# run via "su foo -c ..."
if [ ! -f /tmp/sysstat.run ]; then
RETVAL=1
else
rm -f /tmp/sysstat.run
fi
echo
;;
stop|status|restart|reload)
;;
*)
echo "Usage: sysstat {start|stop|status|restart|reload}"
exit 1
esac
exit ${RETVAL}
The sysstat bootscript only needs to run at system startup, therefore only one symlink is required. Create this symlink in /etc/rc.d/rcsysinit.d using the following command
If sar package is activated the crontab for root should contain something like this (this is actual Red Hat file /etc/cron.d/sysstat ):
# run system activity accounting tool every 10 minutes
*/10 * * * * root /usr/lib64/sa/sa1 1 1
# generate a daily summary of process accounting at 23:53
53 23 * * * root /usr/lib64/sa/sa2 -A;
or for 64-bit Suse the content of the file /etc/sysstat/sysstat.cron looks like
root@usrklxbck01:/etc/init.d # cat /etc/sysstat/sysstat.cron
#crontab for sysstat
# activity reports every 10 minutes everyday
-*/10 * * * * root /usr/lib64/sa/sa1 -d 1 1
# update reports every 6 hours
0 */6 * * * root /usr/lib64/sa/sa2 -A
# Generate a daily summary of process accounting at 23:53
53 23 * * * root /usr/lib64/sa/sa2 -A
As you can see there are two shell scripts sa1 and sa2. The first script invokes the utility sard for collecting the data, the second produces reports.
System Activity Recorder can monitor several system functions related to overall system performance, for example:
cpu utilization (it is pretty effective tool for spotting CPU bottlenecks)
hard disk utilization
terminal IO
number of files open
processes running
SAR as many options and provides queuing, paging, CPU and many other metrics. The system maintains a series of system activity counters that record various activities and provide the data that sar reports. The command merely extracts the data in the counters and saves them based on the sampling rate and the number of samples specified to sar. It costs of two programs sarc and sar:
The sadc command is a data collecting part of the package. It writes in binary format to the specified output file. To run sar in real time type:
sar -u 2 5
In this case the sar command calls sadc to access system data.
To report on previously captured data – type
sar -u -f filename > file
Two shell scripts /usr/lib/sa/sa1 and usr/ib/sa/sa2 can be run by the cron daemon and provide daily statistics and reports.
Side effects
Any tool that collects performance data has some impact on system performance, but with sar, it seems to be minimal. Even one minute sampling usually does not cause any serious issues. That may not hold true on a system that is very busy.
Alternatives
There is also older and not currently maintained atsar project:
The atsar command can be used to detect performance bottlenecks on Linux systems. It is similar to the sar command on other UNIX platforms. Atsar has the ability to show what is happening on the system at a given moment. It also keeps track of the past system load by maintaining history files from which information can be extracted. Statistics about the utilization of CPUs, disks and disk partitions, memory and swap, tty's, TCP/IP (v4/v6), NFS, and FTP/HTTP traffic are gathered. Most of the functionality of atsar has been incorporated in the atop project.
http://www.softpanorama.org/Admin/Monitoring/Sar/linux_implementation_of_sar.shtml
