doc/org.eclipse.tracecompass.incubator.virtual.machine.analysis.doc.user/doc/User-Guide.mediawiki - tracecompass.incubator/org.eclipse.tracecompass.incubator - Git at Google


 = Table of Contents =

 __TOC__

 = Virtual Machine Analysis User Documentation =

 Virtual environments are usually composed of host machines, who each run an hypervisor program on which one or many guests can be run. Tracing a guest machine alone can often yield some strange results as from its point of view, it has full use of the resources, but in reality, most resources are shared with the host and other guests.

 To better understand what is happening in such an environment, it is necessary to trace all the machines involved, guests and hosts, and correlate this information in an experiment that will display a complete view of the virtualized environment.

 == Virtual Machine Experiment ==

 A trace has to be taken for each machine, guest and host, in the virtualized environment. The host trace is the most important to have, as missing guests will only give an incomplete view of the system, but missing hosts usually won't allow to identify the hypervisor, nor determine when a guest is preempted from the host CPUs. The virtual machine analysis only makes sense if the host trace is available.

 Once all the traces are imported in Trace Compass, they can be [http://archive.eclipse.org/tracecompass/doc/stable/org.eclipse.tracecompass.doc.user/Trace-Compass-Main-Features.html#Creating_an_Experiment added to an experiment]. The type of the experiment should by set to '''Virtual Machine Experiment''' by clicking on the right mouse button over the experiment name, then selecting '''Select Experiment Type...'''.

 [[Image:images/VM_experiment.png | Virtual Machine Experiment]]

 Depending on the hypervisor used, traces might need to be [http://archive.eclipse.org/tracecompass/doc/stable/org.eclipse.tracecompass.doc.user/Trace-synchronization.html#Trace_synchronization synchronized] so that they have the same time reference and their events can be correctly correlated.

 == Virtual CPU View ==

 The Virtual CPU view shows the status of CPUs and threads on guests augmented with the preemption and hypervisor data we get from the host.

 In the image below, we see for the virtual CPU status that it has a few more states than the CPUs in the [http://archive.eclipse.org/tracecompass/doc/stable/org.eclipse.tracecompass.doc.user/LTTng-Kernel-Analysis.html#Resources_View Resources View]: in red and purple respectively, when the virtual CPU is running hypervisor code and when the CPU is preempted on the host.

 The entries for each thread of the machine corresponds to the one from the [http://archive.eclipse.org/tracecompass/doc/stable/org.eclipse.tracecompass.doc.user/LTTng-Kernel-Analysis.html#Control_flow  Control Flow View], augmented with the data from the Virtual CPU, so that we see that even though it is running from the guest's point of view, it is actually not running when the Virtual CPU it runs on is in preempted or hypervisor mode.

 [[Image:images/VM_CPU_view.png | Virtual CPU view]]

 ==== Using the keyboard ====
 *'''Ctrl + F''': Search in the view. (see [http://archive.eclipse.org/tracecompass/doc/stable/org.eclipse.tracecompass.doc.user/Trace-Compass-Main-Features.html#Searching_in_Time_Graph_Views Searching in Time Graph Views])

 == Hypervisor-specific Tracing ==

 In order to be able to correlate data from the guests and hosts traces, each hypervisor supported by Trace Compass requires some specific events, that are sometimes not available in the default installation of the tracer.

 The following sections describe how to obtain traces for each hypervisor.

 === Qemu/KVM ===

 The Qemu/KVM hypervisor require extra tracepoints not yet shipped in LTTng for both guests and hosts, as well as compilation with the full kernel source tree on the host, to have access to kvm_entry/kvm_exit events on x86.

 Obtain the source code with extra tracepoints, along with lttng-modules

     # git clone https://github.com/tahini/lttng-modules
     # cd lttng-modules

 Checkout the addons branch, compile and install lttng-modules as per the lttng-modules documentation.

     # git checkout addons_vm
     # make
     # sudo make modules_install
     # sudo depmod -a

 On the host, to have complete kvm tracepoints support, the make command has to include the full kernel tree. So first, you'll need to obtain the kernel source tree. See your distribution's documentation on how to get it. This will compile extra modules, including lttng-probe-kvm-x86, which we need.

     # make KERNELDIR=/path/to/kernel/dir

 The lttng addons modules must be inserted manually for the virtual machine extra tracepoints to be available:

     # sudo modprobe lttng-addons
     # sudo modprobe lttng-vmsync-host # on the host
     # sudo modprobe lttng-vmsync-guest # on the guest

 The following tracepoints will be available

     # sudo lttng list -k
     Kernel events:
     -------------
       ...
       kvm_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
       kvm_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
       vmsync_gh_guest (loglevel: TRACE_EMERG (0)) (type: tracepoint) # on the guest
       vmsync_hg_guest (loglevel: TRACE_EMERG (0)) (type: tracepoint) # on the guest
       vmsync_gh_host (loglevel: TRACE_EMERG (0)) (type: tracepoint) # on the host
       vmsync_hg_host (loglevel: TRACE_EMERG (0)) (type: tracepoint) # on the host
       ...

 Host and guests can now be traced together and their traces added to an experiment. Because each guest has a different clock than the host, it is necessary to synchronize the traces together. Unfortunately, automatic synchronization with the virtual machine events is not completely implemented yet, so another kind of synchronization needs to be done, with TCP packets for instance. See section on [http://archive.eclipse.org/tracecompass/doc/stable/org.eclipse.tracecompass.doc.user/Trace-synchronization.html#Trace_synchronization trace synchronization] for information on how to obtain synchronizable traces.

 = Virtual Machine Analysis Developer Documentation =

 Virtualized environment are becoming more popular and understanding them can be challenging as machines share resources (CPU, disks, memory, etc), but from their point of view, they are running on bare metal. Tracing all the machines (guests and hosts) in a virtualized environment allows information to be correlated between all the nodes to better understand the system. See the User documentation for more info on this analysis.

 The virtual machine analysis has been implemented in the following plugins:

 * '''org.eclipse.tracecompass.lttng2.kernel.core''' contains the virtual machine analysis itself, the model of the virtualized environment, as well as its implementation for different hypervisors.
 * '''org.eclipse.tracecompass.lttng2.kernel.ui''' contains the views for the analysis.

 == Adding support for an hypervisor ==

 Supporting a new hypervisor in Trace Compass requires implementing the model for this new hypervisor. The following sections will describe for each part of the model what has to be considered, what information we need to have, etc. Note that each hypervisor will require some work and investigation. The information might already be available as a single tracepoint for some, while other may require many tracepoints. It is also possible that some will require to add tracepoints, either to the kernel, or the hypervisor code itself, in which case a userspace trace (LTTng UST) might be necessary to get all the information.

 === Virtual CPU analysis ===

 This analysis tracks the state of the virtual CPUs in conjunction with the physical CPU it is running on. For this, we need the following information:

 * A way to link a virtual CPU on a guest with a process on the host, such that it is possible to determine when the virtual CPU is preempted on the host. If trace data does not provide this information, some hypervisors have a command line option to dump that information. Manually feeding that information to the analysis is not supported now though.
 * A way to differentiate between hypervisor mode and normal mode for the virtual CPU. A virtual CPU usually runs within a process on the host, but sometimes that process may need to run hypervisor-specific code. That is called '''hypervisor mode'''. During that time, no code from the guest itself is run. Typically, the process is running on the host (not preempted), but from the guest's point of view, the virtual CPU should be preempted.

 A model implementation for a new hypervisor will need to implement class '''IVirtualMachineModel''', that can be found in package '''org.eclipse.tracecompass.internal.lttng2.kernel.core.analysis.vm.model'''. See the javadoc in the class itself for more information on what each method does.

	= Table of Contents =

	__TOC__

	= Virtual Machine Analysis User Documentation =

	Virtual environments are usually composed of host machines, who each run an hypervisor program on which one or many guests can be run. Tracing a guest machine alone can often yield some strange results as from its point of view, it has full use of the resources, but in reality, most resources are shared with the host and other guests.

	To better understand what is happening in such an environment, it is necessary to trace all the machines involved, guests and hosts, and correlate this information in an experiment that will display a complete view of the virtualized environment.

	== Virtual Machine Experiment ==

	A trace has to be taken for each machine, guest and host, in the virtualized environment. The host trace is the most important to have, as missing guests will only give an incomplete view of the system, but missing hosts usually won't allow to identify the hypervisor, nor determine when a guest is preempted from the host CPUs. The virtual machine analysis only makes sense if the host trace is available.

	Once all the traces are imported in Trace Compass, they can be [http://archive.eclipse.org/tracecompass/doc/stable/org.eclipse.tracecompass.doc.user/Trace-Compass-Main-Features.html#Creating_an_Experiment added to an experiment]. The type of the experiment should by set to '''Virtual Machine Experiment''' by clicking on the right mouse button over the experiment name, then selecting '''Select Experiment Type...'''.

	[[Image:images/VM_experiment.png \| Virtual Machine Experiment]]

	Depending on the hypervisor used, traces might need to be [http://archive.eclipse.org/tracecompass/doc/stable/org.eclipse.tracecompass.doc.user/Trace-synchronization.html#Trace_synchronization synchronized] so that they have the same time reference and their events can be correctly correlated.

	== Virtual CPU View ==

	The Virtual CPU view shows the status of CPUs and threads on guests augmented with the preemption and hypervisor data we get from the host.

	In the image below, we see for the virtual CPU status that it has a few more states than the CPUs in the [http://archive.eclipse.org/tracecompass/doc/stable/org.eclipse.tracecompass.doc.user/LTTng-Kernel-Analysis.html#Resources_View Resources View]: in red and purple respectively, when the virtual CPU is running hypervisor code and when the CPU is preempted on the host.

	The entries for each thread of the machine corresponds to the one from the [http://archive.eclipse.org/tracecompass/doc/stable/org.eclipse.tracecompass.doc.user/LTTng-Kernel-Analysis.html#Control_flow Control Flow View], augmented with the data from the Virtual CPU, so that we see that even though it is running from the guest's point of view, it is actually not running when the Virtual CPU it runs on is in preempted or hypervisor mode.

	[[Image:images/VM_CPU_view.png \| Virtual CPU view]]

	==== Using the keyboard ====
	*'''Ctrl + F''': Search in the view. (see [http://archive.eclipse.org/tracecompass/doc/stable/org.eclipse.tracecompass.doc.user/Trace-Compass-Main-Features.html#Searching_in_Time_Graph_Views Searching in Time Graph Views])

	== Hypervisor-specific Tracing ==

	In order to be able to correlate data from the guests and hosts traces, each hypervisor supported by Trace Compass requires some specific events, that are sometimes not available in the default installation of the tracer.

	The following sections describe how to obtain traces for each hypervisor.

	=== Qemu/KVM ===

	The Qemu/KVM hypervisor require extra tracepoints not yet shipped in LTTng for both guests and hosts, as well as compilation with the full kernel source tree on the host, to have access to kvm_entry/kvm_exit events on x86.

	Obtain the source code with extra tracepoints, along with lttng-modules

	# git clone https://github.com/tahini/lttng-modules
	# cd lttng-modules

	Checkout the addons branch, compile and install lttng-modules as per the lttng-modules documentation.

	# git checkout addons_vm
	# make
	# sudo make modules_install
	# sudo depmod -a

	On the host, to have complete kvm tracepoints support, the make command has to include the full kernel tree. So first, you'll need to obtain the kernel source tree. See your distribution's documentation on how to get it. This will compile extra modules, including lttng-probe-kvm-x86, which we need.

	# make KERNELDIR=/path/to/kernel/dir

	The lttng addons modules must be inserted manually for the virtual machine extra tracepoints to be available:

	# sudo modprobe lttng-addons
	# sudo modprobe lttng-vmsync-host # on the host
	# sudo modprobe lttng-vmsync-guest # on the guest

	The following tracepoints will be available

	# sudo lttng list -k
	Kernel events:
	-------------
	...
	kvm_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
	kvm_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
	vmsync_gh_guest (loglevel: TRACE_EMERG (0)) (type: tracepoint) # on the guest
	vmsync_hg_guest (loglevel: TRACE_EMERG (0)) (type: tracepoint) # on the guest
	vmsync_gh_host (loglevel: TRACE_EMERG (0)) (type: tracepoint) # on the host
	vmsync_hg_host (loglevel: TRACE_EMERG (0)) (type: tracepoint) # on the host
	...

	Host and guests can now be traced together and their traces added to an experiment. Because each guest has a different clock than the host, it is necessary to synchronize the traces together. Unfortunately, automatic synchronization with the virtual machine events is not completely implemented yet, so another kind of synchronization needs to be done, with TCP packets for instance. See section on [http://archive.eclipse.org/tracecompass/doc/stable/org.eclipse.tracecompass.doc.user/Trace-synchronization.html#Trace_synchronization trace synchronization] for information on how to obtain synchronizable traces.

	= Virtual Machine Analysis Developer Documentation =

	Virtualized environment are becoming more popular and understanding them can be challenging as machines share resources (CPU, disks, memory, etc), but from their point of view, they are running on bare metal. Tracing all the machines (guests and hosts) in a virtualized environment allows information to be correlated between all the nodes to better understand the system. See the User documentation for more info on this analysis.

	The virtual machine analysis has been implemented in the following plugins:

	* '''org.eclipse.tracecompass.lttng2.kernel.core''' contains the virtual machine analysis itself, the model of the virtualized environment, as well as its implementation for different hypervisors.
	* '''org.eclipse.tracecompass.lttng2.kernel.ui''' contains the views for the analysis.

	== Adding support for an hypervisor ==

	Supporting a new hypervisor in Trace Compass requires implementing the model for this new hypervisor. The following sections will describe for each part of the model what has to be considered, what information we need to have, etc. Note that each hypervisor will require some work and investigation. The information might already be available as a single tracepoint for some, while other may require many tracepoints. It is also possible that some will require to add tracepoints, either to the kernel, or the hypervisor code itself, in which case a userspace trace (LTTng UST) might be necessary to get all the information.

	=== Virtual CPU analysis ===

	This analysis tracks the state of the virtual CPUs in conjunction with the physical CPU it is running on. For this, we need the following information:

	* A way to link a virtual CPU on a guest with a process on the host, such that it is possible to determine when the virtual CPU is preempted on the host. If trace data does not provide this information, some hypervisors have a command line option to dump that information. Manually feeding that information to the analysis is not supported now though.
	* A way to differentiate between hypervisor mode and normal mode for the virtual CPU. A virtual CPU usually runs within a process on the host, but sometimes that process may need to run hypervisor-specific code. That is called '''hypervisor mode'''. During that time, no code from the guest itself is run. Typically, the process is running on the host (not preempted), but from the guest's point of view, the virtual CPU should be preempted.

	A model implementation for a new hypervisor will need to implement class '''IVirtualMachineModel''', that can be found in package '''org.eclipse.tracecompass.internal.lttng2.kernel.core.analysis.vm.model'''. See the javadoc in the class itself for more information on what each method does.