rover/doc/tex/tracing/RoverTracing.tex

\input{header}
\definecolor{mygray}{gray}{.95}

\begin{document}
\title{\rule{\textwidth}{1pt}\textbf{ Rover Documentation} \\ Tracing with \textbf{Perf}, Conversion to \textbf{CTF}, and analysis with \textbf{TraceCompass} \rule{\textwidth}{1pt}\vspace{-20pt}}
\date{}
\maketitle 
\setlength{\headsep}{20pt}
\vspace{-20pt}
\hspace{-12pt}
\begin{tabularx}{\textwidth}{Xlll}
	& \textit{Version} & \textit{\today }&\\
	&\textit{Implementation} &\textit{Mustafa Özcelikörs} & \textit{mozcelikors@gmail.com}\\
	&\textit{Supervision \& revision} &\textit{Robert Höttger} &\textit{robert.hoettger@fh-dortmund.de}\\ \\
	&&\multicolumn{2}{l}{University of Applied Sciences and Arts Dortmund}\\ 
	&&\multicolumn{2}{l}{IDiAL Institute, Project AMALTHEA4public}\\
	&&\multicolumn{2}{l}{BMBF  	Fund.Nb. 01|S14029K} 
\end{tabularx} 
\vspace{15pt}\\
\begin{tabularx}{\textwidth}{cXc}
		\includegraphics[width=0.4\textwidth]{../images/rover.png}&&
		\includegraphics[width=0.42\textwidth]{../images/tracecompass.png}
\end{tabularx}

\section{Scope}
This documentation describes (a) the generation of \textbf{\underline{\texttt{Perf}}} traces \cite{perf} on a Linux system, (b) an approach to convert \texttt{perf} traces to the \textbf{CTF} (Common Tracing Format) format \cite{babeltrace}, and (c) the \textbf{CTF} import to an analysis tool e.g. TraceCompass \cite{TraceCompass}. %How the trace conversion is performed and preliminary steps required to perform the trace conversion are also explained.
\section{Installing BabelTrace}
The perf module is already included with the Linux kernel but requires a LibBabelTrace specific build for the conversion support to CTF. Hence, BabelTrace has to be installed first.
%The current distributions of perf do not allow data conversion functionality. Therefore, perf should be compiled from the Linux kernel with libbabeltrace support in order to be able to transform perf traces into CTF traces. 
Therefore, the repository list of the Raspberry Pi has to be up to date:
\begin{lstlisting}
	sudo apt-get update
\end{lstlisting}
The following command installs the LibBabelTrace module:
\begin{lstlisting}
	sudo apt-get install libbabeltrace-ctf-dev libbabeltrace-ctf1 libbabeltrace1 libbabeltrace-dev python3-babeltrace
\end{lstlisting}
In addition, the module's dependencies must be installed as well via:
%To install the LibBabelTrace from the source, which is required to compile perf with data conversion support, the following dependencies have to be installed:
\begin{lstlisting}
	sudo apt-get install dh-autoreconf bison libdw-dev libelf-dev flex uuid-dev libpopt-dev 
\end{lstlisting}
Afterwards, the following commands should be used in order to clone, configure, build, and install LibBabelTrace:
\begin{lstlisting}
	cd /home/pi
	sudo git clone git://git.efficios.com/babeltrace.git
	cd babeltrace
	sudo ./bootstrap
	sudo ./configure --prefix=/opt/libbabeltrace LDFLAGS=-L/usr/local/lib
	sudo make -j4 prefix=/opt/libbabeltrace
	sudo make install prefix=/opt/libbabeltrace
\end{lstlisting}
\section{Building Perf with BabelTrace support}
After LibBabelTrace is installed, perf should be rebuilt from the Linux kernel with the babeltrace support. By using the following command, the dependencies and features regarding the new perf build can be installed:
\begin{lstlisting}
	sudo apt-get install libnewt-dev binutils-arm-none-eabi libcrypto++-dev binutils-multiarch-dev libunwind-dev systemtap-sdt-dev libssl-dev libperl-dev libiberty-dev
\end{lstlisting}
Having all the dependencies, one should clone, build, and install the following linux repository with a LibBabelTrace adapted perf that supports data conversion via:
\begin{lstlisting}
	cd /home/pi
	sudo git clone git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git
	cd linux/tools/perf/
	sudo git checkout perf/core
	sudo LIBBABELTRACE=1 LIBBABELTRACE_DIR=/opt/libbabeltrace/ make
	sudo LIBBABELTRACE=1 LIBBABELTRACE_DIR=/opt/libbabeltrace/ make install
\end{lstlisting}
This will create a perf executable located at the \texttt{/home/pi/linux/tools/perf/} directory.
\section{Tracing processes and threads}
\subsection{Option 1: Tracing Processes and Threads Manually}
In order to record a system trace for e.g. 15 seconds, the following command can be used while the applications are running:
\begin{lstlisting}
	sudo ./home/pi/linux/tools/perf/perf sched record -- sleep 15
\end{lstlisting}
This command creates a perf trace file namely \texttt{perf.data}. Since the Eclipse TraceCompass tool does not support the perf format directly, the perf trace can be converted to the CTF format via:
\begin{lstlisting}
	sudo LD_LIBRARY_PATH=/opt/libbabeltrace/lib ./home/pi/linux/tools/perf/perf data convert --to-ctf=./ctf
\end{lstlisting}
The command above should be called within the path where \texttt{perf.data} is located. In order to avoid having to do this, the input argument can also be used to specify the location of the perf trace e.g.:
\begin{lstlisting}
	sudo LD_LIBRARY_PATH=/opt/libbabeltrace/lib ./home/pi/linux/tools/perf/perf data convert -i=/path/to/perf.data --to-ctf=./ctf
\end{lstlisting}
In order to use the trace in Eclipse TraceCompass on a Windows operating system, the CTF trace can be archived by using the following command:
\begin{lstlisting}
	sudo tar -czvf trace.tar.gz ctf/
\end{lstlisting}
\subsection{Option 2: Tracing processes and threads using a script}
In order to trace and manually convert the trace, the following shell script \texttt{TraceLinuxProcesses.sh} (located at src/tracing) can be used. The script should have the following content:
\begin{lstlisting}
	#!/bin/bash
	args=("$@")
	trace_name=${args[0]}
	seconds=${args[1]}
	perf_directory=${args[2]}
	
	if [ "$#" -ne 3 ]; then
		echo "Entered arguments seem to be incorrect"
		echo "Right usage: sudo TraceLinuxProcesses.sh <trace_name> <period> <path_to_perf>"
		echo "e.g. sudo TraceLinuxProcesses.sh APP4MC_Trace 15 /home/pi/linux/tools/perf"
	else
		echo "### Creating directory.."
		sudo mkdir out_$trace_name/
		echo "### Writing out process names.."
		ps -aux >> out_$trace_name/Processes_List.txt
		echo "### Tracing with perf for $seconds seconds.."
		sudo $perf_directory/./perf sched record -o out_$trace_name/perf.data -- sleep $seconds
		echo "### Converting to data to CTF (Common Tracing Format).."
		sudo LD_LIBRARY_PATH=/opt/libbabeltrace/lib $perf_directory/./perf data convert -i out_$trace_name/perf.data --to-ctf=./ctf
		sudo tar -czvf out_$trace_name/trace.tar.gz ctf/
		sudo rm -rf ctf/
		
		echo "### Process IDs are written to out_$trace_name/Processes_List.txt"
		echo "### Trace in Perf format is written to out_$trace_name/perf.data"
		echo "### Trace in CTF format is written to out_$trace_name/trace.tar.gz"
		echo "### Exiting.."
	fi
\end{lstlisting}
To run the scrip with root privileges, execute:
\begin{lstlisting}
	sudo chmod 777 TraceLinuxProcesses.sh
\end{lstlisting}
The provided script takes three arguments: trace name, amount of time to trace, installed perf directory. An example way to call this script is shown below which traces the system for 15 seconds and creates a folder out\texttt{\_}rover\texttt{\_}trace1 using the perf object located at \texttt{/home/pi/linux/tools/perf/} directory:
\begin{lstlisting}
	sudo TraceLinuxProcesses.sh out_rover_trace1 15 /home/pi/linux/tools/perf
\end{lstlisting}
The script produces an output directory which contains perf format data, ctf format and the process id list to interpret the output. 
\section{Visualization and Interpretation of the Trace}
By using Eclipse TraceCompass, which can be downloaded from \textit{http://www.tracecompass.org}, traces in CTF format can be interpreted. By choosing the trace file by selecting \textit{File} $\rightarrow$ \textit{Import} menu items, the trace can be imported into Eclipse TraceCompass seen in Figure \ref{fig:tracecompass}.

TraceCompass enables users to analyze a system regarding call graphs, threads, context switches, cpu usage, critical path, I/O, control flow, and resources which can be seen as (1) in the Figure \ref{fig:tracecompass}. The control flow window (2) shows each process state with respect to time including the transitions along all the processes. System-wide CPU usage and individual processes' CPU usages are shown in the CPU Usage window which is shown as (3) in the figure. Therefore, if the system has 4 cores, the CPU usage of up to 400 percent could be observed. The resources window (shown as (4)) depicts how processes are distributed amongst the existing cores with respect to time. Therefore, the load balancing could be roughly observed from this view by simply looking at each of the cores. Moreover, using the resources window, one can measure and estimate the timing properties of the scheduling done by the kernel. Finally, the trace event list (shown as (5)) can be used to see the exact events that occurred in a specific time frame selected in another view.
\begin{figure}[!ht]
	\centering
	\includegraphics[scale=0.44]{../images/tracecompass.png}
	\caption{Eclipse TraceCompass}
	\label{fig:tracecompass}
\end{figure}

\section{Making Thread Names Visible to TraceCompass and Monitoring Core Utilization of Threads}
In order to monitor core utilization of each thread using Linux shell, first the thread names of each thread should be registered in user application. That is, using the \texttt{pthread{\_}setname{\_}np()} function after each thread is created, which is already implemented in the Rover project. Once this function is used in the implementation, the Linux command name of the thread will be overwritten. This helps Linux tools and TraceCompass recognize the thread name and be able to show the thread name when tracing.

Implemented scripts help to monitor threads of a process easily. The scripts are located in the rover project repository and are briefly explained below.

The following script can be used in order to monitor the core utilization of every registered thread \\ (\texttt{MonitorThreads.sh}):
\begin{lstlisting}
	args=("$@")
	process_name=${args[0]}
	
	if [ "$#" -ne 1 ]; then
		echo "Entered arguments seem to be incorrect"
		echo "Right usage: MonitorThreads.sh <process_name>"
		echo "e.g. MonitorThreads.sh <process_name>"
	else
		pid=$(pgrep -f $process_name -o)
		top H -p $pid
	fi
\end{lstlisting} 
Similarly, by using the \texttt{ListThreads.sh} given below, one can list thread names and thread ID's of all the threads of a process given its process name.
\newpage
\begin{lstlisting}
	args=("$@")
	process_name=${args[0]}
	if [ "$#" -ne 1 ]; then
		echo "Entered arguments seem to be incorrect"
		echo "Right usage: ListThreads.sh <process_name>"
		echo "e.g. ListThreads.sh <process_name>"
	else
		pid=$(pgrep -f $process_name -o)
		ps H -p $pid -o 'pid tid cmd comm'
	fi
\end{lstlisting}
The Figure \ref{fig:MonitorThreads} depicts the output from the \texttt{MonitorThreads.sh}.

\begin{figure}[!ht]
	\centering
	\includegraphics[width=\textwidth]{../images/MonitorThreads.png}
	\caption{MonitorThreads.sh output}
	\label{fig:MonitorThreads}
\end{figure}


\bibliography{bib}{}
\bibliographystyle{plain}
\end{document}