7. IWATCH Performance report

FMS contains a WEB based performance report that monitors your application in real time. When you run your application, FMS generates and updates the pages listed below. This list contains links to examles of the information provided.

• Main
  This page lists the network node name of the machine.
• CPU's
  This page lists the number and type of CPU installed in the system. Note that for modern processors, the Clock Speed listed may change with utilization.
• GPU's
  This page lists information about the GPU's used by FMS. It does not include GPU's used for graphics with insufficient compute capability.

  Nvidia uses two numbering systems: one by the Run Time Library (RTL) and another by the nvidia-smi utility (NVML). These may not be the same and can change on reboot. Correlation between these numbering systems is established by the PCI bus location of the device. This page lists the devices in the RTL order and shows the correcponding NVML number for reference. If you use the nvidia-smi utility to change the properties of a device, you should use the NVLM number.

  By default, the nvidia-smi utility is installed in the directory /usr/bin. If this utility is in a different directory, then the FMS Parameter CUPATH must be included in the license file to provide the directory where nvidia-smi is located.

• Memory
  This page lists the CPU and GPU memory allocations. The table lists each region allocated from the FMS Memory Management utilities.
• Disks
  This page lists the disks used by FMS. The first table shows the individual mount points used by the FMS File Striping System. The second table lists the performance by file type.
• Files
  This page lists the open FMS files, their stripe configuration and current access information.
• Software
  This page lists version numbers of the operating system and supporting libraries.
• Calls
  This page lists the subroutines which have been called. The order in each table is determined by the order of first call. The following tables are included:
  • Application Subroutines
    Subroutines that you have instrumented with calls to FMSPSH and FMSPOP.
  • FMS Main Subroutines
    FMS subroutines called directly by your application.
  • FMS Callback Subroutines
    Subroutines you provided to define matrix data or special processing of diagonal pivots.
  • FMS Internal Subroutines
    Subroutines called internally by FMS to measure the performance of specific utilities.
  • FMS GPU Subroutines
    Kernel subroutines run on the GPU's.
  Routines highlighted in red are currently active on the stack.
• Parameters
  This page lists the current values of FMS parameters which have been changed from default values. In some cases, changing one parameter will cause others to be changed also. Some of the parameters are output only and are computed by FMS.
• Performance
  This is the main page for monitoring performance. It first lists the FMS routine currently executing with a link to it's manual page.

  The first status bar shows the percent completion for the current routine, based on the total number of operations required and the number completed. The second bar shows the percent contribution of major tasks, based on the amount of wall time used.

  The Performance table lists the average Gigaflops achieved by each type computational element (CPU's, GPU's) and task (Assembly, Matrix Multiply, Triangle Solve, Diagonal Factor). A snapshot of the most recent task as well as the overall performance of the routine is also included.

  Where appropriate, a picture of the matrix and vector(s) are shown as they are blocked for processing. The status of each block changes as the computation proceeds.

  The Times and Problem table lists the time the job was started, the time the current routine was started, the current time and the estimated time of completion for the current routine. It also lists some useful information about the size and properties of the problem being solved.

• Usage
  This page lists the overall usage of system resources since the job started.
• Links
  This page lists links to Multipath's home page and the FMS manual.
• Help This help page.

To enable report generation, set the FMS Parameter IWATCH to a nonzero value in the license file.

Initally all pages are generted when FMS starts. A page becomes a candidate for updating whenever one of the following occurs:

• A subroutine is called or returned;
• Memory is allocated or released;
• A file is opened, closed, read or written;
• A FMS Parameter is changed;
• A subtask (matrix multiply, etc.) is completed.

In some cases the page is updated immediately. For less critical pages the updates are accumulated and performed after NSUPD seconds.

Once the pages are written, they need to be reloaded in your browser. You may do this in one of the following ways:

• Typing a control-r
• Setting the FMS Parameter NSREF to automatically refresh the page at a specified interval. For the Performance page, FMS automatically sets the refresh rate based on the time required for the previous task.

By default, FMS creates the pages in the current working directory. You may change the location where the pages are created by specifying the environment variable FMSHTML. It may be useful to create this directory on a cross mounted disk on a WEB server or other host with greater access than the machine currently used for running FMS. At the top of each page are listed the number of times it has been updated, the automatic refresh rate if specified and the time it was last updated.