HPC System Administrators Guide (SAG)
Casper

This page is meant to serve as a general System Administrators Guide (SAG) for the HPC's Casper system.

LoadLeveler

AIX Large Page support added to LoadLeveler

In AIX 5.1D and subsequent releases, a feature called Technical Large Page Support is available. Technical Large Page Support involves the selective use of large virtual and physical memory pages to back private data segments of a process. When specified, the user process heap, the main program BSS, and the main program data areas are backed by large pages. LoadLeveler users can take advantage of this AIX feature and enable Large Page support for their jobs.

Note for administrators:
Although large pages are supported when running jobs under LoadLeveler, running LoadLeveler daemons with large pages enabled is not supported. If you attempt to start up LoadLeveler on a node using the llctl start or llctl recycle commands and the AIX environment variable LDR_CNTRL=LARGE_PAGE_DATA=Y or LDR_CNTRL=LARGE_PAGE_DATA=M is set then LoadLeveler will not start and the LoadL_master daemon will report an error.

For more information about LDR_CNTRL, please refer to the AIX documentation.

If some of the machines in your LoadLeveler cluster are configured to exploit the Large Page feature, and if you want LoadLeveler to provide support for large pages, then the information contained in the items listed below is needed for effective use of this feature:

  1. VM_IMAGE_ALGORITHM keyword setting

    The LoadLeveler configuration keyword VM_IMAGE_ALGORITHM must be set to the value FREE_PAGING_SPACE_PLUS_FREE_REAL_MEMORY if jobs are submitted with large_page = M. This keyword specifies which algorithm the Central Manager uses to decide whether a machine has enough virtual memory to meet the requirement of the image_size keyword of a job step.

    Jobs which specify large_page = M will remain in the idle state if the default setting of FREE_PAGING_SPACE is used for VM_IMAGE_ALGORITHM keyword. Using the FREE_PAGING_SPACE_PLUS_FREE_REAL_MEMORY algorithm allows LoadLeveler to consider both the free "regular" memory and the free Large Page memory when deciding if a machine in the cluster has enough virtual memory to run a job step.

     

  2. New LoadLeveler job command file keyword: large_page

    The job command file keyword large_page is used to inform LoadLeveler that a job step requires Large Page support from AIX. The syntax of this keyword is:

    # @ large_page = <value>
    Where <value> can be:
    Y : Use Large Page memory if available, otherwise use regular memory
    M : Use of Large Page memory is mandatory
    N : Do not use Large Page memory (default value)

     

  3. llq and llsummary enhancements

    The "llq -l" and "llsummary -l" commands have been enhanced to display information associated with the large_page keyword for LoadLeveler jobs. The output listings of these commands contain lines similar to the following:

    System Priority: -37
    Notifications: Complete
    Virtual Image Size: 500.000 mb
    Large Page: Y
    Checkpointable: no

     

  4. llstatus enhancement

    The "llstatus -l" command has been enhanced to show information for total memory and free memory for both Large Page memory and regular memory. Below is a fragment of a representative "llstatus -l" command output:

    Max_Starters = 50
    Total Memory = 3583 mb
    Memory = 1536 mb
    FreeRealMemory = 1113 mb
    LargePageSize = 16.000 mb
    LargePageMemory = 2.000 gb
    FreeLargePageMemory = 2.000 gb

    In the above listing, Total Memory refers to the sum of regular and Large Page memory. Memory and FreeRealMemory refer to regular and free regular memory.

     

  5. New requirements and preferences variables: TotalMemory and LargePageMemory

    Two new LoadLeveler variables, TotalMemory and LargePageMemory, have been added. These variables are supported by the requirements and preferences expressions.

    In the following sample job command file, the person submitting this job to LoadLeveler has requested that the job be run on a machine that has at least 1500 MB of Large Page memory configured and that machines having total memory (regular and Large Page) greater than 2800 MB are preferred.

    #!/bin/ksh
    # @ step_name = test_batch_job
    # @ class = small
    # @ restart = no
    # @ arguments = arg_01 arg_02 arg_3
    # @ environment = env_001=0001; LDR_CNTRL=LARGE_PAGE_DATA=Y; env_002=0002
    # @ requirements = (LargePageMemory > 1500)
    # @ preferences = (TotalMemory > 2800)
    # @ executable = test_program
    # @ cpu_limit = 120
    # @ large_page = Y
    # @ image_size = 512000
    # @ input = /dev/null
    # @ output = job1.$(Host).$(Cluster).$(Process).out
    # @ error = job1.$(Host).$(Cluster).$(Process).err
    # @ queue

    Notes:

    1. The units for LargePageMemory and TotalMemory are megabytes and these variables are 64-bit integers.
    2. The environment variable LDR_CNTRL is an AIX variable. LDR_CNTRL=LARGE_PAGE_DATA has the values, Y (Large Page use optional), M (Large Page use mandatory), and undefined (do not use Large Page) as the default value.

      For more information about LDR_CNTRL, please refer to the AIX documentation.

    3. The person submitting this job has requested that Large Page memory should be used to run the test_program if it is available.

     

  6. ll_get_data() enhancements

    The function ll_get_data() of the LoadLeveler API has been enhanced so that Large Page information of machines can be accessed by the specifications:

    The large_page information associated with job steps can be accessed by the specification LL_StepLargePage.

     

  7. Workload Manager

    The AIX Workload Manager (WLM) program does not support Large Page memory. The information associated with memory statistics in the outputs of commands such as "llq -w <job_id>" is not meaningful and should be ignored.

     

Compiler License Server

  • Setup the licenses for the compilers:
  • Fix License Server

    Kerberos

  • Could not dsh to new peg nodes. Error message in the system log (/syslog/syslog.all):
  • "Decrypt integrity check failed"
  • The destroy of old tickets and generating new tickets was the magic.

    NIM Install

    Monotord

    Quota

    STATA 9