Problem This recipe demonstrates how to create a snapshot of one or more subject systems using ftimes, ssh, and sudo such that all output will be directed to the ssh connection. Using ssh from a central location simplifies the process of aggregating data harvested from multiple systems, and it also reduces the amount of perturbation to each system in evidence collection scenarios. To achieve fine-grained control over the snapshots, the config file will be sent through the ssh connection on STDIN. This allows for centralized management of configuration files. Motivation Taking system snapshots on a regular basis can dramatically reduce the amount of time required to answer the following questions: - What files and directories have been modified (changed)? - What files and directories have been added (new)? - What files and directories have been removed (missing)? - What files are known and can be attributed to a trusted source? - What files are unknown and must be reviewed? - How do the production file systems compare to the original file systems (i.e., as deployed or designed)? - How much much drift is there between systems of the same type? - When were patches, software upgrades, or packages installed? - What files and directories need to be restored to repair damage done by an intruder? By regularly harvesting and analyzing snapshot data, you'll be able to answer these questions and more. Requirements This recipe assumes that 1) all relative commands are in your PATH and that they will be executed from a Bourne shell; 2) all systems are supported flavors of UNIX; 3) the master system can establish SSH connections to each subject; and 4) each subject system has ftimes and sudo installed and properly configured. Solution The solution for the master is to create a snapshots directory structure, ssh keys, and a set of ftimes config files. Once the basic framework is setup, cron jobs can be established to periodically reach out and take snapshots. The solution for each subject is to create a locked ftimes account, install a customized authorized keys file, and establish limited sudo access that allows a particular instance of ftimes to be run as root. 1. Create a locked user account called 'ftimes' on each subject system. This account must be locked to prevent unnecessary logins and password authentication. All authentication for this recipe will be done using ssh keys. Modify the environment variables (listed below) to suit your needs. Then, set them in your shell. This will allow you to cut and paste the various commands that follow. # FTIMES_ID="55" # FTIMES_USER="ftimes" # FTIMES_GROUP="ftimes" # FTIMES_HOME="/usr/home/${FTIMES_USER}" FreeBSD: # pw groupadd ${FTIMES_GROUP} -g ${FTIMES_ID} # pw useradd ${FTIMES_USER} -u ${FTIMES_ID} -g ${FTIMES_GROUP} -d ${FTIMES_HOME} -s /bin/sh -c "FTimes User" Linux and Solaris: # groupadd -g ${FTIMES_ID} ${FTIMES_GROUP} # useradd -u ${FTIMES_ID} -g ${FTIMES_GROUP} -d ${FTIMES_HOME} -s /bin/sh -c "FTimes User" ${FTIMES_USER} Check /etc/shadow on Solaris systems to see if the password field is set to '*LK*'. If it is, change it to 'NP'. Create a root-owned, home directory for the ftimes user. This directory must be root-owned so that the ftimes user will not have sufficient privileges to create or modify its home directory or any dot files that may reside there. # mkdir -p ${FTIMES_HOME} # chown -R 0:${FTIMES_GROUP} ${FTIMES_HOME} # chmod -R 750 ${FTIMES_HOME} 2. Create the snapshots directory structure depicted below on the master. This is where the configuration files, ssh keys, and harvested snapshots will reside. Create sub-directories as needed for each system and profile that will be monitored. Here, a profile is defined to be a set of files and directories that you wish to monitor independently. For example, you may want to create one profile for monitoring the entire system and another profile that is dedicated to system or mission critical files. snapshots | + | | | + | + | + ... | + + ... Thus, if you had two hosts (host1 and host2) that you want to monitor and each of those has three profiles (all, system, and test), then your snapshots directory would look like this: snapshots | + host1 | | | + all | + system | + test | + host2 | + all + system + test The following commands will create the example directory structure. Modify them to suit your environment. # FTIMES_TREE=snapshots # FTIMES_SUBJECTS="host1 host2" # FTIMES_PROFILES="all system test" # for SUBJECT in ${FTIMES_SUBJECTS}; do for PROFILE in ${FTIMES_PROFILES} ; do mkdir -p ${FTIMES_TREE}/${SUBJECT}/${PROFILE} ; done ; done 3. Create an ssh key pair for the ftimes user on the master, and store the resulting files in the snapshots tree as master-ssh.key and master-ssh.key.pub, respectively. These keys will be used to access subject systems. # KEY_NAME="master-ssh.key" # KEY_TYPE="rsa" # ssh-keygen -q -t ${KEY_TYPE} -C bimvs -f ${FTIMES_TREE}/${KEY_NAME} -N "" 4. Create an ftimes config file for each profile you defined in step 2. Use the following naming convention: master-map-.cfg Appendices 1, 2, and 3 contain sample config files for the "all", "system", and "test" profiles. Copy those files to the snapshots tree and edit them as needed. At this point, your snapshots tree should look like this: snapshots | + host1 | | | + all | + system | + test | + host2 | | | + all | + system | + test | - master-ssh.key - master-ssh.key.pub - master-map-all.cfg - master-map-system.cfg - master-map-test.cfg If a particular profile requires a custom configuration, simply create a file called map.cfg in the appropriate profile directory and edit it as needed. For example, suppose that the system profile for host2 needs to include /opt/{bin,etc,lib,sbin} in addition to the directories already included. In this scenario, you could do the following: # cp snapshots/master-map-system.cfg snapshots/host2/system/map.cfg # cat >> snapshots/host2/system/map.cfg Include=/opt/bin Include=/opt/etc Include=/opt/etc Include=/opt/lib Include=/opt/sbin ^D The resulting file would look like this: --- map.cfg --- Basename=- FieldMask=all-magic Include=/bin Include=/etc Include=/sbin Include=/usr/bin Include=/usr/local/bin Include=/usr/local/etc Include=/usr/local/lib Include=/usr/local/sbin Include=/usr/lib Include=/usr/sbin Include=/opt/bin Include=/opt/etc Include=/opt/lib Include=/opt/sbin --- map.cfg --- Likewise, if a particular profile requires a custom ssh key, simply copy the desired key to the appropriate profile directory and name it ssh.key. 5. Create an authorized keys file, called master-ssh.authorized_keys, on the master, and propagate it to each subject. Note that the key you created in step 3 was not protected with a passphrase. Therefore, you should restrict its use by applying various ssh key options. In particular, I recommend that you use the "command" and "from" options. Details about the various key options can be found in the "AUTHORIZED_KEYS FILE FORMAT" section of the sshd(8) man page. Edit ALLOWED_HOSTS, ALLOWED_COMMAND, and KEY_OPTIONS as necessary before running the command that creates master-ssh.authorized_keys. # ALLOWED_HOSTS="10.0.0.1,10.0.0.2" # ALLOWED_COMMAND="/usr/bin/sudo /usr/local/integrity/bin/ftimes --maplean -" # KEY_OPTIONS="from=\"${ALLOWED_HOSTS}\",command=\"${ALLOWED_COMMAND}\",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty" # echo ${KEY_OPTIONS} `cat ${FTIMES_TREE}/${KEY_NAME}.pub` > ${FTIMES_TREE}/master-ssh.authorized_keys As you propagate the authorized keys file to each subject, you'll need to lock its permissions down such that the ftimes user will be unable to modify or delete it. This can be done as follows: # chown -R 0:${FTIMES_GROUP} ${FTIMES_HOME}/.ssh # chmod 750 ${FTIMES_HOME}/.ssh # chmod 640 ${FTIMES_HOME}/.ssh/authorized_keys 6. Modify the sudoers file on each subject such that the ftimes user is able to run ftimes in --maplean mode as root. The corresponding sudoers entry should look like this: ftimes ALL= NOPASSWD: /usr/local/integrity/bin/ftimes --maplean - 7. Pick one of the newly configured subjects and run the following commands (from the master) to test your setup: # FTIMES_TREE=snapshots # FTIMES_USER=ftimes # FTIMES_SUBJECT=host1 # FTIMES_KEY=${FTIMES_TREE}/master-ssh.key # ssh -i ${FTIMES_KEY} -o BatchMode=yes -o StrictHostKeyChecking=no ${FTIMES_USER}@${FTIMES_SUBJECT} \ /usr/bin/sudo /usr/local/integrity/bin/ftimes --maplean - < ${FTIMES_TREE}/master-map-test.cfg > test.map 2> test.log Review the results, fix any problems, and repeat this step until you're satisfied that everything is working -- be prepared to roll your sleeves up and dig in ;) One problem to watch out for is OpenSSH's host key checking. If the subject's key isn't known to the master, OpenSSH's default action will be to ask, and that could cause the job to hang. Basically, there are two ways to avoid this situation: (1) make sure the client's known_hosts file contains the appropriate host keys or (2) disable StrictHostKeyChecking. 8. Extract ftimes-bimvs.sh from this recipe (Appendix 4), edit to suit your environment, and install it on the master in /usr/local/integrity/bin (mode 0755). This script has the following usage: ftimes-bimvs.sh [-b] [-d] [-H bimvs-home] [-f ftimes] [-s sudo] [-u user] -p profile host [host ...] where '-b' forces jobs to be run in the background; '-d' enables debugging; '-H' is a path that points to the master's snapshots tree; '-f' and '-s' are paths that point to the ftimes and sudo binaries, respectively, on the subjects; '-u' is the name of the ftimes user account; and '-p' is the name of the profile to harvest. 9. Establish cron jobs on the master for each profile group that you want to harvest/monitor on a regular basis. Below are two example crontab entries. The first harvests the "system" profile every 6 hours, and the second harvests the "all" profile once a day. 0 0,6,12,18 * * * /usr/local/integrity/bin/ftimes-bimvs.sh -p system host1 host2 > /dev/null 2>&1 0 * * * * /usr/local/integrity/bin/ftimes-bimvs.sh -p all host1 host2 > /dev/null 2>&1 Closing Remarks This recipe is best suited to environments where remote and privileged administrative tasks are routinely handled through a combination of ssh and sudo. Because sudo is the mechanism used to elevate privileges, there is no requirement to allow remote root logins via ssh (refer to the PermitRootLogin option in sshd_config). Because ssh keys are used to authenticate the ftimes user, there is no requirement to allow password authentication (refer to the PasswordAuthentication option in sshd_config). When harvesting snapshots via cron, it is important to choose a job interval that is at least double the amount of time it takes to process all systems included in that job. For example, if it can take up to 30 minutes to map all your systems in a single job, then the overall snapshot interval should be at least one hour. The logic in ftimes-binvs.sh is not sufficient to keep snapshots in time order -- especially if multiple jobs for the same profile run in parallel. Consequently, various output files (e.g., *.cmp, *.map, *.log) could get biffed. Refer to the previous comment for one suggestion on how to help reduce this risk. There's lots of room for improvements to ftimes-binvs.sh, so get to it ;) Credits This recipe was brought to you by Klayton Monroe. Appendix 1 The following command may be used to extract this Appendix: $ sed -e '1,/^--- master-map-all.cfg ---$/d; /^--- master-map-all.cfg ---$/,$d' ftimes-bimvs.txt > master-map-all.cfg --- master-map-all.cfg --- Basename=- FieldMask=all-magic --- master-map-all.cfg --- Appendix 2 The following command may be used to extract this Appendix: $ sed -e '1,/^--- master-map-system.cfg ---$/d; /^--- master-map-system.cfg ---$/,$d' ftimes-bimvs.txt > master-map-system.cfg --- master-map-system.cfg --- Basename=- FieldMask=all-magic Include=/bin Include=/etc Include=/sbin Include=/usr/bin Include=/usr/local/bin Include=/usr/local/etc Include=/usr/local/lib Include=/usr/local/sbin Include=/usr/lib Include=/usr/sbin --- master-map-system.cfg --- Appendix 3 The following command may be used to extract this Appendix: $ sed -e '1,/^--- master-map-test.cfg ---$/d; /^--- master-map-test.cfg ---$/,$d' ftimes-bimvs.txt > master-map-test.cfg --- master-map-test.cfg --- Basename=- FieldMask=all-magic Include=/etc/passwd --- master-map-test.cfg --- Appendix 4 The following command may be used to extract this Appendix: $ sed -e '1,/^--- ftimes-bimvs.sh ---$/d; /^--- ftimes-bimvs.sh ---$/,$d' ftimes-bimvs.txt > ftimes-bimvs.sh --- ftimes-bimvs.sh --- #!/bin/sh ###################################################################### # # Copyright 2004-2004 The FTimes Project, All Rights Reserved. # ###################################################################### # # Purpose: Basic Integrity Monitoring Via SSH (BIMVS) # ###################################################################### IFS=' ' PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/local/integrity/bin PROGRAM=`basename $0` ProcessSubject() { MY_HOME=$1 MY_PROFILE=$2 MY_USER=$3 MY_HOST=$4 MY_SUDO=$5 MY_FTIMES=$6 MY_DATE=`date +%Y-%m-%d` MY_TIME=`date +%H.%M.%S` if [ -z "${MY_HOME}" -o -z "${MY_PROFILE}" -o -z "${MY_USER}" -o \ -z "${MY_HOST}" -o -z "${MY_SUDO}" -o -z "${MY_FTIMES}" -o \ -z "${MY_DATE}" -o -z "${MY_TIME}" ] then echo "${PROGRAM}: Hey, I need some input!" 1>&2 exit 2 fi MY_CFG=${MY_HOME}/${MY_HOST}/${MY_PROFILE}/map.cfg if [ ! -f ${MY_CFG} ] ; then MY_CFG=${MY_HOME}/master-map-${MY_PROFILE}.cfg if [ ! -f ${MY_CFG} ] ; then echo "${PROGRAM}: Error='Unable to locate config file for the specified host/profile (${MY_HOST}/${MY_PROFILE}).'" 1>&2 exit 2 fi fi MY_KEY=${MY_HOME}/${MY_HOST}/${MY_PROFILE}/ssh.key if [ ! -f ${MY_KEY} ] ; then MY_KEY=${MY_HOME}/master-ssh-${MY_PROFILE}.key if [ ! -f ${MY_KEY} ] ; then MY_KEY=${MY_HOME}/master-ssh.key if [ ! -f ${MY_KEY} ] ; then echo "${PROGRAM}: Error='Unable to locate ssh key for the specified host/profile (${MY_HOST}/${MY_PROFILE}).'" 1>&2 exit 2 fi fi fi MY_BASEDIR=${MY_HOME}/${MY_HOST}/${MY_PROFILE} MY_FIRST=${MY_BASEDIR}/baseline.1st MY_BASELINE=${MY_BASEDIR}/baseline.map MY_SNAPSHOT=${MY_BASEDIR}/snapshot.map MY_ANALYZED=${MY_BASEDIR}/analyzed.cmp if [ ! -d ${MY_BASEDIR} ] ; then mkdir -p ${MY_BASEDIR} else if [ -f ${MY_BASELINE} ] ; then if [ -f ${MY_FIRST} ] ; then rm -f ${MY_BASELINE} else mv ${MY_BASELINE} ${MY_FIRST} fi fi if [ -f ${MY_SNAPSHOT} ] ; then mv ${MY_SNAPSHOT} ${MY_BASELINE} fi fi MY_DATEDIR=${MY_HOME}/${MY_HOST}/${MY_PROFILE}/${MY_DATE} MY_MAP=${MY_DATEDIR}/${MY_TIME}.map MY_LOG=${MY_DATEDIR}/${MY_TIME}.log MY_CMP=${MY_DATEDIR}/${MY_TIME}.cmp if [ ! -d ${MY_DATEDIR} ] ; then mkdir -p ${MY_DATEDIR} fi ssh -i ${MY_KEY} -o BatchMode=yes -o StrictHostKeyChecking=no ${MY_USER}@${MY_HOST} \ ${MY_SUDO} ${MY_FTIMES} --maplean - < ${MY_CFG} > ${MY_MAP} 2> ${MY_LOG} if [ ${MY_STATUS=$?} -ne 0 ] ; then echo "${PROGRAM}: Error='The snapshot process failed for the specified host/profile (${MY_HOST}/${MY_PROFILE}).'" 1>&2 exit ${MY_STATUS} fi cp -p ${MY_MAP} ${MY_SNAPSHOT} if [ -s ${MY_BASELINE} -a -s ${MY_MAP} ] ; then ftimes --compare none+md5 ${MY_BASELINE} ${MY_MAP} -l 6 > ${MY_CMP} if [ ${MY_STATUS=$?} -ne 0 ] ; then echo "${PROGRAM}: Error='The analysis process failed for the specified host/profile (${MY_HOST}/${MY_PROFILE}).'" 1>&2 exit ${MY_STATUS} fi cp -p ${MY_CMP} ${MY_ANALYZED} fi } Usage() { echo 1>&2 echo "Usage: ${PROGRAM} [-b] [-d] [-H bimvs-home] [-f ftimes] [-s sudo] [-u user] -p profile host [host ...]" 1>&2 echo 1>&2 exit 1 } BIMVS_BACKGROUND=0 BIMVS_DEBUG=0 while getopts "bdf:H:p:s:u:" OPTION ; do case "${OPTION}" in b) BIMVS_BACKGROUND=1 ;; d) BIMVS_DEBUG=1 ;; f) BIMVS_FTIMES="${OPTARG}" ;; H) BIMVS_HOME="${OPTARG}" ;; p) BIMVS_PROFILE="${OPTARG}" ;; s) BIMVS_SUDO="${OPTARG}" ;; u) BIMVS_USER="${OPTARG}" ;; *) Usage ;; esac done if [ ${OPTIND} -gt $# ] ; then Usage fi shift `expr ${OPTIND} - 1` BIMVS_HOME=${BIMVS_HOME-/var/snapshots} BIMVS_FTIMES=${BIMVS_FTIMES-/usr/local/integrity/bin/ftimes} BIMVS_SUDO=${BIMVS_SUDO-/usr/bin/sudo} BIMVS_USER=${BIMVS_USER-ftimes} if [ -z "${BIMVS_PROFILE}" ] ; then Usage fi for BIMVS_HOST in $@ ; do if [ ${BIMVS_DEBUG} -eq 1 ] ; then echo "===> ${BIMVS_HOST}/${BIMVS_PROFILE} (host/profile)" fi if [ ! -d ${BIMVS_HOME}/${BIMVS_HOST} ] ; then echo "${PROGRAM}: Error='The dropzone (${BIMVS_HOME}/${BIMVS_HOST}) does not exist or is not a directory.'" 1>&2 continue fi if [ ${BIMVS_BACKGROUND} -eq 1 ] ; then ( ProcessSubject "${BIMVS_HOME}" "${BIMVS_PROFILE}" "${BIMVS_USER}" "${BIMVS_HOST}" "${BIMVS_SUDO}" "${BIMVS_FTIMES}" ) & else ( ProcessSubject "${BIMVS_HOME}" "${BIMVS_PROFILE}" "${BIMVS_USER}" "${BIMVS_HOST}" "${BIMVS_SUDO}" "${BIMVS_FTIMES}" ) fi done --- ftimes-bimvs.sh ---