3.2. Using FMS Parameters

• Hardware configuration including memory and processors,

• NonUniform Memory Access (NUMA) memory options,

• The FMS Database file system including file striping,

• Print output and format,

• Matrix file initialization and blocking,

• Matrix assembly,

• Matrix factoring and pivoting,

• Vector solution,

• Vector and matrix multiply,

• Error handling,

When you start FMS by calling FMSINI, FMS initializes all Parameters do default values. In most cases, the default values will allow you to get FMS running in your application. However, the FMS Parameters do provide some features that are useful during software development. Therefore you should become familiar with the FMS Parameters to know what options are available.

Setting FMS Parameters

1. CALL FMSIST(NAME, IVALUE)
CALL FMSRST(NAME, RVALUE)
CALL FMSCST(NAME, CVALUE)


   These subroutines provide a way to change INTEGER, REAL*8 and CHARACTER parameters respectively. You supply the NAME of the parameter as a character string and the new value as an INTEGER, 8-byte real or character string. FMS checks your value and if appropriate changes the value of the FMS Parameter.

   This method of setting FMS Parameters is recommended for modifications that become a permanent part of your application. For example, if know your matrix should always be positive definite, you should set the FMS Parameter MPOSDF as follows:

   CALL FMSIST ('MPOSDF', 1)

2. CALL FMSSET


   Inserting this call in your application will cause FMS to solicit input interactively for viewing or changing FMS Parameters as follows:

   FMSSET>name

   where

   FMSSET> = the prompt displayed by FMS and

   name = the name of the FMS Parameter whose value you want to display.

   You may change the value of a FMS Parameter as follows:

   FMSSET>name=value

   where

   value = the value you want to assign to the parameter.

   When you have finished changing all the required parameters, you enter the characters RETURN at the FMSSET> prompt,

   FMSSET>RETURN

   and FMS continues on with your application.

   This method of changing FMS Parameters is especially useful during development and tuning. It allows you to print out intermediate results or try out Parameters without them becoming a permanent part of your application.

3. License File
   You may also change the initial (default) value of FMS Parameters in the License File by adding the following lines before the END record:

   FMSSET
name=value
...
name=value
RETURN


   When FMS encounters the FMSSET record in the License File, it switches input to FMSSET for further processing. FMSSET then processes each modification name=value as though it was entered directly from a terminal. When FMSSET encounters the RETURN record, it transfers processing back to FMS.

   You should use this method to set FMS Parameters that are not part of the application, especially parameters that describe your hardware configuration. You may also use this method to try different parameters during development in a similar way to calling FMSSET.

Calling FMSSET, either from within your application or through the License File allows you to change FMS parameters without recompiling and relinking your application.

Reading FMS Parameters

1. CALL FMSIGT(NAME, IVALUE)
CALL FMSRGT(NAME, RVALUE)
CALL FMSCGT(NAME, CVALUE)


   These subroutines provide a way to obtain the value of INTEGER, REAL*8 and CHARACTER parameters respectively. You supply the NAME of the parameter as a character string and FMS returns the INTEGER, 8-byte real or character string value.

2. CALL FMSCST('SHOW', 'PARAMETERS')
CALL FMSCST('SHOW', 'MODIFIED')


   The first call will print a report listing all the FMS Parameters and their current value. The second call lists only those parameters that have been modified after the License File.

3. FMSSET>SHOW=PARAMETERS
FMSSET>SHOW=MODIFIED

   You may also direct FMS to display all the Parameters or the modified Parameters from FMSSET. Using the SHOW=PARAMETERS from FMSSET is useful to display a list of all the parameters that you can change.

With this background, let's take a quick tour of the FMS Parameters. If you want more information on any Parameter, just click on it's name.

Hardware Parameters

These parameters provide a way for FMS to know your hardware configuration. Some operating systems provide hooks to determine the number of installed processors and amount of memory. If available, FMS bases it's default values on this information. In other cases this information is not available. You should review what FMS uses for the default values of MAXCPU and MAXMD and modify the value in the License File, if required.

The MYNODE Parameter allows you to obtain the thread number that is currently being used to execute your subroutine. The threads are numbered from 0 (parent) to (MAXCPU-1). Typically you would use this value in subroutines you are running in parallel using the FMS parallel processing routine FMSPAR.

The MDUSED Parameter allows you to obtain the amount of memory that has been used from the FMS memory pool. You may use this if you are calling FMSROW or FMSCOL to populate multiple matrix files simultaneously, or are using the FMS Memory Management System.

The MEMPTR returns the starting address of the pool of memory allocated by the FMS Memory Manager. It is used in FORTRAN programs to set the POINTER value in the arrays used to access FMS memory. In C applications, it provides a pointer to the start of the FMS memory.

The MDHOW Parameter determines which system services FMS uses to allocated memory. The options include mmap, malloc, shm and a fixed array. Not all options are available on all machines. You should use the default value selected by FMS.

The MDINIT Parameter provides options for initializing the memory in the FMS memory pool. Normally this is not required. However, if you are performing benchmarking and want to set the system page table to a known state, you may do so with this parameter. When NUMA options are used, the default value of this Parameter is automatically changed to explicitly place the memory.

The MALIGN Parameter determines the alignment of FMS memory. Typically this is set to the page size LPAGE.

On most systems, FMS uses the system page file for backing up the memory allocated by FMS. The NOPAGE Parameter provides the option of creating a special page file for FMS. However you should use this option with caution. Some systems will attempt to continuously keep this private page file consistent with the FMS memory pool, resulting in degraded performance.

The ISYNC Parameter is used to control the type of parallel synchronization. The default value of semaphores should almost always be used instead of the optional spin-lock method.

The ISCOPE Parameter controlls the priority and scheduling of the threads. FMS automatically sets this Parameter for best performance.

The remaining hardware Parameters MDPADD, LCACHE, LSTACK, ITPCPU and LENVRG control hardware architecture features and should be left at the default values.

NUMA Parameters

NAME	Function
NUMAFL	Master flag. Must be set to 1 for any of the following parameters to take effect.
NUMAPR	Print output level
MYCPU1	Starting processor to use. 1 is first processor.
NPNODE	Number of processors per node
NUMNOD	Number of nodes (read only)
NUMAFX	Thread binding to processor options.
MDWHEN	When memory is allocated
MAXLMD	Stride between nodes for placing memory (bytes)
NUMATP	Topology (SGI only)
MMGLUE	Combine blocks during matrix multiply

NUMAFL is the main flag that turns all the FMS NUMA features on or off. By default, NUMA features are turned off, providing an environment best suited for small jobs in a multi-user environment. When NUMAFL is set, threads and memory become bound to processors, providing explicit control of the machine environment. This feature is best used for large jobs on a dedicated machine, or part of a machine.

The NUMAPR Parameter controlls the amount of printed output that occurs when the NUMA environment is being established. This includes information about thread and memory placement.

When NUMA effects are on, MYCPU1 specifies the starting CPU number for the job. This Parameter must be set explicitly if multiple jobs are being run concurrently.

Within FMS, a "NODE" is defined as a part of the NUMA machine having uniform memory access. For current NUMA machines, this is between two and four processors. NPNODE specifies the number of processors per node.

The NUMAFX Parameter determines how the threads are bound to the processors.

The MDWHEN Parameter determines when memory for the FMS memory pool is allocated from the system. For proper NUMA placement, memory is physically allocated from the system and properly placed as required by FMS or your application.

MAXLMD specifies the stride by which memory is allocated among the nodes in a round-robin fashon. If you are allocating memory from the FMS memory pool by the parent in your application, the memory will be allocated among the nodes using this stride. A zero value will override this feature and allocate all the memory on the node running the parent thread. When a memory request is made by a child thread, all memory is automatically allocated on the node running the child thread. This feature provides the control required for you to maximize local memory references in the non-FMS part of your application.

The reamining NUMA Parameters, NUMATP and MMGLUE, should be left at the default values selected by FMS.

File Parameters

There are several parameters which provide options for configuring the FMS Database.

You may direct FMS to place files in memory with the INCORE Parameter. This can provide improved performance for small problems.

By default, FMS treats files in the Database as scratch files that are deleted at the end of the job. You may set the IFKEEP Parameter before the file is closed to retain a file. Later, when you want to reopen the file, you can set the IEXIST Parameter to use an existing file, rather than to create a new one.

You may also direct FMS to write a restart file during matrix factoring, or to restart a job in the middle of matrix factoring using a previously written restart file. The FMS Parameter IREST controlls this feature.

Normally FMS opens a file for access by the parent. If you plan to read or write a FMS file from threads you created by calling FMSPAR, you should set the IOKIDS Parameter.

The FMS File Striping System is controlled by the NUMIOQ, NUMSTP, NUMVOL and MBYTES Parameters. These Parameters specify the number of queues, stripes per queue, volumes per stripe and capacity of each volume. The FMS File Striping System can significantly improve I/O performance. You should review this option before completing your installation. Because the disk configuration on most systems may change, you should set these parameters in the License File.

On some Unix systems, FMS can bypass the buffer cache and perform I/O directly to the disks using raw or direct I/O. The FMS Parameter IOTYPE discusses these options.

By default, FMS performs transfers to the Database asynchronously. You may turn this feature off and perform transfers synchronously with the IOSYNC Parameter.

The LSECTR specifies the minimum number of 8-byte words which may be transfered to a physical disk dirve. Normally the default value selected by FMS will be correct. However if your file system contains multiple drives striped into a single mount point, the defalut value may be too low. FMS automatically detects this condition and provides instructions on how to change this Parameter to the correct value.

The NOOPEN Parameter skips the physical opening of files. This feature is used to estimate the disk space that will be required when the files are actually opened.

If you are using submatrices to define the matrix data, FMS will buffer the data as it is written by your application. The LBUFSI, LBUFSR and LBUFSV parameters control the size of the buffers used for the integer, real and vector files.

Data Formatting Input/Output Parameters

FMS provides complete control over printed output with the print Parameters. You may select the level of printing at any stage of your application with the IPRI, IPRA, IPRF, IPRS, IPRMV, IPRVV and IPRDV Parameters. If you want to set all these parameters to the same value, you may use PRINT. The Print Code Summary is a useful list of the options that are available with each parameter.

The default print level includes status and timings for each subroutine, which is appropriate for most production applications. However, during installation you may wish to modify these parameters to check intermediate results. The FMSSET utility is especially useful for setting these parameters during installation.

Normally FMS accumulates timing information for subroutines you call. The LOGTIM Parameter provides additional options for timing internal routines. This information may be useful for final tuning. Because the additional low level timings will increase run time, this Parameter should be left at the default value for production.

The amount of data in a matrix or vector may be very large. FMS provides the NEQPR1 and NEQPR2 parameters to control the window of data that is printed.

The NROWPG and NCOLPG control the size of your display. The default values of 24 by 80 is appropriate for most terminals.

The IFMAT, RFMAT and CFMAT control the format for INTEGER, REAL*8 and COMPLEX*16 data. You may adjust these Parameters to control the amount of information printed by each matrix and vector term.

The SHOW Parameter is useful for printing reports on FMS Parameters, subroutine calls, memory usage and the status of the FMS Database.

You may use the RESET Parameter to reset all parameter values to the default values, including any changes you made in the License File.

FMS prints output to file LUPR. FMSSET reads input from file LUINPT.

Matrix File Initialization Parameters

When you open a matrix file, FMS stores the data in one of three formats: PROFILE, BLOCK or SLAB. You may use the MFMAT Parameter to override the default selection. SLAB format, which provides for full column partial pivoting, must always be selected by setting MFMAT. Because the best format depends only on your matrix, you should call FMSIST in your application to change this value.

The LENSDF and NEQBIO Parameters control the record size that is used for the matrix file. The default values of these parameters should provide the best performance.

The storage within each record is controlled by the NEQBLK, IJSTEP and INSIDE Parameters. The default values of these parameters are carefully selected to match optimized kernels within FMS. FMS selects the value of NEQBLK to be between NEQBLO and NEQBHI. You may experiment with these during final tuning, but the default values should provide the best performance.

Matrix Assembly Parameters

The FMS Parameters also provide options for building the matrix. LOWASM may be used during nonlinear analysis to reuse the part of the matrix factors which have not changed. LUOK will convert a nonsymmetric matrix to symmetric. The MDATAU Parameter directs FMS to call subroutines you provide to define or modify the matrix data.

Matrix Factoring Parameters

There are several options provided for handling pivots during the factoring process. MPOSDF instructs FMS to halt if a negative pivot is found. NBITSP controls warning messages for pivots "close" to zero. NBITSZ sets the tolerance for zero. PIVMIN also sets a limit on the minimum pivot value.

When FMS detects a bad pivot, it halts by default. You may use the MZERO Parameter to override this action. The options include setting the pivot reciprocal to RPIVOT and continuing or calling a subroutine you provide.

The NEQSUB Parameter offers a useful modification to the factoring and solving process for many applications.

For some applications, the algorithm used for matrix multiply and/or complex arithmetic may be altered to improve performance without loss of accuracy. The IALGOR, MINDIM and SAVOPS Parameters provide these options.

After the factoring process, you may call FMSIGT to obtain information about the matrix factors. NUMSCG provides the number of sign changes on the diagonal and NPIVOT provides the number of equations pivoted for SLAB matrices.

You may also call FMSRGT to find You may find out information about how the matrix filled in during the factoring process from ATERMS, which lists the total number of terms in the matrix, AZEROS, which lists the number zero terms in the unfactored matrix and FZEROS which lists the number of zero terms in the factored matrix.

During factoring the parallel processes must synchronize as they compute terms around the diagonal. The IPOLLD parameter selects between event flags and spin-locks for this synchronization. It is strongly recommended that you leave this parameter at the default value.

Solving Parameters

As an option, you may use the IPBACK Parameter to direct FMS to perform back substitution with inner products. For most cases, the default value is preferred.

The LUTRAN Parameter may be used to solve using the transpose of the matrix [A] for nonsymmetric problems. This feature has applications in evaluating derivatives of potential functions.

Multiply Parameters

When using FMSMM to compute matrix or vector multiply operations, the resulting product is normally stored in the upper-left corner of the output matrix. The FMS Parameters MMROW, MMCOL and MMKA provide a way of shifting where the product is stored.

Error Handling Parameters

FMS contains an error monitoring system that provides useful information whenever an error occurs. This information is printed on unit LUERR. The amount of information is controlled by IPRERR. The FMS error number is provided by NERROR. If the error was due to a system service, the system error is available in IOSTAT.

The FMS error reporting system is designed to provide the information you need without modification. If, however, you want to process your own error conditions, you may set the IERROR Parameter and check the status in NERROR after each FMS call.

This completes the quick tour of FMS Parameters. You may find more information about any parameter through the FMS Parameters available in the Table of Contents.