Login | Register
Login | Register

My pages Projects SunSource.net openCollabNet

2.2. Hedeby system administration

The Hedeby distribution contains the command line utility sdmadm. All administrative tasks can be executed with sdmadm.

2.2.1. Managing bootstrap configuration

2.2.1.1. Overview

The bootstrap configuration of Hedeby gives the sdmadm command the basic information about the available Hedeby systems.

Hedeby knows two different types of bootstrap configurations.

SYSTEM

The SYSTEM boostrap configuration is the default location for installing a Hedeby system. On unix based system the SYSTEM bootstrap configurations is stored in the directory /etc/sdm/bootstrap. Normally only user root has write access to directory.

USER

On unix based system the USER boostrap configurations are stored in the home directory in the sub directory .sdm. The user boostrap configurations can be used to define the bootstrap configuration for a Hedeby system on a host which is not part of Hedeby system. This is useful if an administrator want invoke sdmadm from his/her preferred workstation and not from a managed host (see also Section 2.3.4.1, “Add Bootstrap Config”). If the home directory is on a shared file system the user bootstrap is also shared.

Note

Windows™ is currently not supported by Hedeby, however it is on the TODO list. On Windows the boostrap configuration will be stored in the Registry.

The following information is stored in the boostrap configuration:

2.2.1.1.1. Target Hedeby System

There exists three different ways howto define the target system.

  1. Specifying the system name with the global -system (shortcut -s) parameter when starting sdmadm.

    % sdmadm -s test_system ...
                        

  2. Setting the environment variable SDM_SYSTEM. The following example uses bourne shell syntax:

    % SDM_SYSTEM=test_system
    % export SDM_SYSTEM
    % sdmadm ...
                        

  3. Defining the a default system in the bootstrap configuration. This is done with the sdmadm set_default_bootstrap_config command (shortcut is sedbc).

    % sdmadm -s test_system set_default_bootstrap_config                                
                        
    The sdmadm set_default_bootstrap_config command command stores the default system in the home directory of the user in the .sdm directory. This information is kept alive even if the command terminal is closed.

    The sdmadm show_bootstrap_config command prints in the properties column the information what system is the default system. The following command shows that the system with name test_system is the default system.

    % sdmadm show_bootstrap_config
    system        type   host  port  properties version
    ---------------------------------------------------
    test_system   SYSTEM foo   31016 default    1.0
    prod_system   SYSTEM foo1  31118            1.0
                        
    If the user does not want a default system it can be reset with the sdmadm unset_default_bootstrap_config

The sdmadm -system overwrites the SDM_SYSTEM enviroment variable. SDM_SYSTEM overwrites the default system from the bootstrap config.

2.2.1.1.2. CS Contact Information

The central communication contact point of a Hedeby system is the host and port the JVM running the configuration service. Both values are stored in the bootstrap information.

2.2.1.1.3. Local Spool Directory

sdmadm and Hedeby JVMs has to known the path to the local spool directory. This directory contains e.g. trusted certificates and credentials for establishing a SSL connections. The local spool directory can be displayed with sdmadm show_bootstrap_config -all:

% sdmadm -s test_system show_bootstrap_config -all
system      type host  port  properties version
-----------------------------------------------
test_system SYSTEM foo   31016 default  1.0
    spool=/var/spool/sdm/test_system
...    
                    
The local spool directory of a single Hedeby system can have different path on different hosts.

2.2.1.1.4. Distribution Directory

The Hedeby bootstrap mechanism can start different versions of Hedeby with the same sdmadm command as long as systems have the same version the bootstrap configuration. To enable this feature the path to the distribution directory must be stored in the bootstrap configuration. sdmadm reads first the distribution directory from the bootstrap config and loads the libraries of the target system before invoking further actions.

The path to the distribution directory can be displayed with:

% sdmadm -s test_system show_bootstrap_config -all
system      type host  port  properties version
-----------------------------------------------
test_system SYSTEM foo   31016 default  1.0
    dist=/opt/sge
...    
              

2.2.1.1.5. SSL Encryption

Per default the communication between the Hedeby component is encrypted with SSL. Turning of SSL encryption is dangerous and not recommended, however it is possible. In the bootstrap configuration the Hedeby system property no_ssl can be set to true. This is done with sdmadm set_bootstrap_config_property.

%sdmadm -s test_sytem -p user set_bootstrap_config_property \
        -p no_ssl -v true
System property "no_ssl" set to "true"
              
The sdmadm sbc print in the properties column no_ssl for systems where SSL is disabled.
% sdmadm sbc
system        type   host  port  properties     version
-------------------------------------------------------
test_sytem    USER   foo   31016 no_ssl,default 1.0    
                  

2.2.1.1.6. Boot Time Startup

Hedeby has boot time startup support. If the bootstrap configuration of a system has set the auto_start property it is started during boot time. The property can also set with sdmadm set_bootstrap_config_property.

%smadm -s ts31016user -p user set_bootstrap_config_property -p auto_start -v true
System property "auto_start" set to "true"
                  
The sdmadm sbc prints in the properties column auto_start for for system where auto_start property is set.
% sdmadm sbc
system        type   host  port  properties     version
-------------------------------------------------------
test_sytem    USER   foo   31016 auto_start,default 1.0    
                  

For more information about boot time startup please have a look at Section 2.2.6, “Autostart Feature in Hedeby”

2.2.1.2. Adding a System to the Bootstrap Configuration

The sdmadm add_bootstrap_config adds a Hedeby system to the boostrap configuration. The global option -s defines the system name, the -p switch defines used bootstrap configuration (SYSTEM or USER).

  

% sdmadm -s test_system -p user add_bootstrap_config -cs foo:31016
Using default local spool directory: /var/spool/sdm/test_system
Using default distribution directory: /opt/sdm
A configuration for system, "test_system", has been added

            

For more details please reference the cli command reference of sdmadm add_bootstrap_config (see Section 2.3.4.1, “Add Bootstrap Config”);

Once the system is defined in the boostrap configuration sdmadm can be used to invoke commands on the system. The add_bootstrap_config does not store any cretedentials for password less authentication to the local spool directory, sdmadm will report a "permission denied" error when contacting a Hedeby component.

% sdmadm sc
Error: permission denied
             

To enable username/password authentication the global option -ppw can be specified.


% sdmadm -ppw sc -t Other
username [sdm_admin] > 
password > 
host  jvm         component type  state       
----------------------------------------------
foo   executor_vm ca        Other STARTED     
      rp_vm       reporter  Other STARTED

                

For password less authentication the credentials of the user must be stored in form of a keystore in the local spool directory. This keystore can be obtained with the sdmadm update_keystore. The ca component must be active for the update_keystore command.


% sdmadm -ppw -s test_system update_keystore -t user -n sdm_admin
username [sdm_admin] > 
password > 
The specified keystore has been exported to the file 
/var/spool/sdm/test_system/security/users/sdm_admin.keystore

             
Any further call of sdmadm will use this keystore for authenticating the user.

2.2.1.3. Removing a system from the boostrap configuration

If the bootstrap configuration of a system is not longer necessary it can be removed with sdmadm remove_bootstrap_config. The system can only be removed if no JVM of the system is running on the host.


 sdmadm -s test_system -p user remove_bootstrap_config
The configuration for system "test_system" has been removed

                

The sdmadm command does not delete the local spool directory.

2.2.2. JVMs and Components

Hedeby is a component based system. The known component types are Configuration Service, Resource Provider, Reporter, Service, CA, Executor. All these components runs inside of different java processes. These processes are called Java Virtual Machines (JVMs).

2.2.2.1. Managing JVMs

The default installation of Hedeby knows three different JVMs:

cs_vm

Runs the Configuration service. The cs_vm is the only process which must bind a static port. Only one instance of the cs_vm exists and this is running on the master host.

executor_vm

The executor_vm normally exists on any managed host (only with executor component) and also on the master host (runs the executor and the CA component). The executor_vm must run as privileged user (on unix systems the user root) otherwise the executor component can not execute privileged action like starting command as different user or changing the owner ship of a file.

rp_vm

The rp_vm is the multi purpose JVM. It runs as non privileged user. After a default installation the rp_vm on the master host runs Resource Provider, Reporter and Spare Pool. GE Service component should run also inside the rp_vm on the qmaster of the Grid Engine installation.

2.2.2.1.1. Starting JVMs

Starting a Hedeby system means starting the JVMs of the system. Hedeby provides mechanism to start JVMs remotely. The administrator has to start sdmadm startup_jvm directly on the host.

If sdmadm startup_jvm without parameter is invoked it will start any not running JVM on the local host. If no component for a JVM is configured it is not started. To startup a JVM the CS component must be active (exception the cs_vm). sdmadm startup_jvm knows this dependy, if it is started on the master host it will first startup the cs_vm before starting any other JVM.

The following example shows a typical manual startup of a Hedeby system:


% rlogin <master_host>
% su -
password:
# sdmadm -s test_system startup_jvm
jvm         host            result  message                       
--------------------------------------------------------
cs_vm       <master_host>   STARTED                               
executor_vm <master_host>   STARTED                               
rp_vm       <master_host>   STARTED              
#

            

In the next steps the administrator has to execute the sdmadm startup_jvm on each managed host.


% rlogin <managed_host>
% su -
# sdmadm -s test_system startup_jvm
jvm         host           result  message                       
--------------------------------------------------------
executor_vm <managed_host> STARTED                               

              

If the startup of a JVM failed the error is printed to the message column. So times the real error is written into the log file of the JVM. The log file is stored at <local_spool_dir>/log/<jvm_name>-0.log.

With the sdmadm show_jvm the state of the JVMs can be monitored.


% sdmadm -s test_system -p system set_default_bootstrap_config
Default system set to 'test_system'
% sdmadm sj
name        host            state      used_mem  max_mem   message                                 
-----------------------------------------------------------------------------------------
cs_vm       <master_host>   STARTED           7M      113M                                         
executor_vm <master_host>   STARTED           5M       28M
            <managed_host>  STARTED           5M       28M
...
rp_vm       <master_host>   STARTED           5M      113M                                         

            
2.2.2.1.2. Stopping JVMs

The sdmadm shutdown_jvm stops JVMs. This command can perform the follwoing actions:

  • Stop a single JVM
  • Stop all JVMs with the same name on all hosts
  • Stop all JVMs on a single host
  • Stop all JVMs of a Hedeby system

For more details please have a look at Section 2.3.5.2, “Shutdown Jvm”

2.2.2.2. Managing Components

At JVM startup normally all components of are started automatically. sdmadm startup_component and sdmadm shutdown_component manages the lifecylce of a component. sdmadm show_component display the state of components. The following example shows the usage of these commands. For more information please have a look at Section 2.3.5, “Component Cli Commands”.


% sdmadm show_component
host  jvm         component         type             state       
-----------------------------------------------------------------
foo   executor_vm ca                Other            STARTED     
                  executor          Executor         STARTED     
      rp_vm       reporter          Other            STARTED     
                  resource_provider ResourceProvider STARTED     
                  spare_pool        Service          STARTED     
% sdmadm shutdown_component -c ca -h foo
comp host  message           
-----------------------------
ca   foo shutdown triggered
% sdmadm show_component -c ca
host  jvm         component type  state       
----------------------------------------------
foo executor_vm ca        Other STOPPED
% sdmadm suc -c ca
comp host  message          
----------------------------
ca   foo   startup triggered

        
2.2.2.2.1. Change the Configuration of a Component

The configuration of the components is stored in the configuration service. The sdmadm mod_component opens the configuration in an prefered text editor (EDITOR environment variable). The user can make the modifications. After saving the changes and closing the editor the modified configuration is validated and stored in the configuration service.


% sdmadm mod_component -c ca
-----------------------------
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<common:componentConfig xsi:type="security:CAComponentConfig" 
                        caHost="foo" 
                        adminUser="sdm_admin" 
    <security:sgeCaScript>/opt/sge62/util/sgeCA/sge_ca</security:sgeCaScript>
</common:componentConfig>
-----------------------------
:x
Component configuration updated

           

The new configuration is now stored in the configuration service. However all instances of the component still uses the old configuration. The sdmadm update_component command triggers the component to reload the configuration from the configuration service.


% sdmadm update_component -c ca
comp host  message         
---------------------------
ca   foo   reload triggered

           

2.2.3. Managing Services

If a service should be managed by Hedeby it is necessary to add the service to the system and start the component with service. Once a service is started it will be autodiscovered by Resource Provider and then the system starts to manage the service.

In order to be able to add a service to the system, the JVM with CS has to be run. If the Resource Provider is not running during the start up of service, this service is not managed by system. Resource Provider is the key component for managing services.

2.2.3.1. Add a Spare Pool

The default installation has already defined a Spare Pool. Normally it is not necessary to add new spare pools. However with sdmadm add_spare_pool_service command it is possible. The admininstrator has to specify the name of the Spare Pool, the host where it should live and the name of the JVM. If the JVM is running the add_spare_pool_service will start the spare spool component if the -start option is specified.

                
% sdmadm add_spare_pool_service -n spare_pool1 -h foo -j rp_vm -start
service_name hostname jvm_name message
--------------------------------------
spare_pool1  foo      rp_vm    ADDED  

            
The sdmadm show_service displays the new Spare Pool.
                
grawp% sdmadm show_service
host        service     cstate  sstate 
---------------------------------------
master_host spare_pool  STARTED RUNNING
foo         spare_pool1 STARTED RUNNING
...

        

2.2.3.2. Add a Grid Engine Service

Before adding a Grid Engine Service to a Hedeby system the following requirements must be fulfilled

  • The version of the Grid Engine instance must be 6.2 or higher.

  • The Java Management Extension (JMX, new feature of Grid Engine 6.2) must be enabled in qmaster.

    For more information about enabling the JMX feature please have a look into the Grid Engine 6.2 Installation Guide.

  • The Certitifcate Authority of the Grid Engine must be initialized and a java keystore for an Grid Engine admin user must exist.

    The Grid Engine Service will use this keystore for password less authentication against Grid Engine. For more information please have a look at the man page of Grid Engines sge_ca tool (man sge_ca).

  • The host where the Grid Engine Adapter should run must have access on the Grid Engine Root directory (SGE_ROOT). This host must be a managed host of the Hedeby system. Normally the Grid Engine Adapter should run on the same host as Grid Engines qmaster process.

The Grid Engine service can be added with the sdmadm add_ge_service. For the correct configuration the following parameters are required:

  • Host and JVM where the GE service component should run

  • Path to SGE_ROOT

  • Name of the Grid Engine cell

  • Name of the Grid Engine admin user

  • Path to the keystore of the Grid Engine admin user (/var/sgeCA/port$SGE_QMASTER_PORT/$SGE_CELL/userkeys/<user_name>/keystore)

  • Password of the keystore if it is encrypted

  • Port of the qmaster JMX server (defined in $SGE_ROOT/$SGE_CELL/common/jmx/management.properties

  • Qmaster port

  • Execd port

  • Clustername of Grid Engine (defined in $SGE_ROOT/$SGE_CELL/common/cluster_name)

The sdmadm add_ge_service opens a template for a Grid Engine Service configuration in the editor. The Administrator has to editor the parameters. After saving the changes and closing the editor the GE Service component is created. For more details about the configuration of the Grid Engine Service please have a look at Section 2.4.3.3.1, “GEAdapter Configuration”.

If the JVM of the Grid Engine Service is already started the service component can be started by sdmadm add_ge_service (add the -start option). Otherwise the JVM has to be started after adding the service (sdmadm startup_jvm).


% sdmadm add_ge_service -s ge -h <qmaster_host> -j rp_vm -start
--- output of the editor ------------------------------------------------
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<common:componentConfig xsi:type="ge_adapter:GEServiceConfig" 
                        mapping="default">
    ...
    <ge_adapter:connection     
      keystore="/var/sgeCA/port31004/default/userkeys/sge_admin/keystore" 
      password="changeit" 
      username="sge_admin" jmxPort="31016" 
      execdPort="31005" masterPort="31004" 
      cell="default" 
      root="/opt/sge62" 
      clusterName="sge62"/>
     ...                                  
</common:componentConfig>
-------------------------------------------------------------------------
% sdmadm show_serivce
host         service     cstate  sstate 
-----------------------------------------
master_host  spare_pool  STARTED RUNNING
foo          spare_pool1 STARTED RUNNING
qmaster_host ge          STARTED RUNNING
...
        
        

The component state of the ge service (column cstate) should be STARTED. The service state (column sstate) will be RUNNING of all configuration parameter where correct and qmaster could be contacted. If the service state is not RUNNING please check the configuration parameters and reconfiguration the GE service component (sdmadm mod_component -c ge).

2.2.3.3. Starting and Stopping a Service

A service component has two states. The component state reflects the state of the component, while the service state reflects the state of the service. The service will only produce resource requests if it the service state is RUNNING. Assigning and Unassigning is also only possible if the service state is RUNNING. The service state will automatically set to UNKNOWN if the service adapter detects that the communication to the service is interrupted (e.g in the case of a Grid Engine service if qmaster is not reachable).

To change the service state of a service manually the command sdmadm startup_service/shutdown_service are used.


% sdmadm show_service
host  service    cstate  sstate 
--------------------------------
foo spare_pool STARTED RUNNING
% sdmadm show_resource_request
type    service     slo     urgency
-----------------------------------
pending spare_pool  default 10     

% sdmadm shutdown_service -s spare_pool
service    result  message
--------------------------
spare_pool STOPPED
% sdmadm show_service
host  service    cstate  sstate 
--------------------------------
foo spare_pool   STARTED STOPPED
% sdmadm srr
type    service     slo                 urgency
-----------------------------------------------
% sdmadm startup_service -s spare_pool
service    result  message
--------------------------
spare_pool STARTED

% sdmadm show_resource_request
type    service     slo     urgency
-----------------------------------
pending spare_pool  default 10

        

2.2.3.4. Removing a Service

A service can be removed with sdmadm remove_service. This command can be used to remove any kind of service (Spare Pool and ge services). If the service has assigned resources they are also removed from the Hedeby system. A component can only be removed if it is not running.

                    
% sdmadm show_serivce
host         service     cstate  sstate 
-----------------------------------------
master_host  spare_pool  STARTED RUNNING
foo          spare_pool1 STARTED RUNNING
qmaster_host ge          STARTED RUNNING
% sdmadm shutdown_component -c spare_pool1 -h foo
comp        host  message           
------------------------------------
spare_pool1 foo   shutdown triggered
% sdmadm remove_service -s spare_pool1
service     host  result 
-------------------------
spare_pool1 foo   REMOVED
% sdmadm ss
host  service    cstate  sstate 
--------------------------------
grawp spare_pool STARTED RUNNING


                

2.2.4. Managing Resources

2.2.4.1. Resource Definition Overview

2.2.4.1.1. Resource definition for the first use case

In order to manage resources in the Hedeby system it is necessary to define what a resource is. The current implementation only supports host resources. Resources are represented by an aggregation of properties. Each property is a name/value pair which represents some required attribute of the resource. When a resource is requested these properties are used for to find matching resources, a matching resource is one that fulfills each of the requested resource properties values.

All resources have some common resource properties:

  • Resource ID: This is a unique resource identifier for resources of type "host". The resource id for a host resource is based on the host's fully qualified name (FQN).

  • Type: Represents the type of a resource, such as "host" or "license" (currently only the type "host" is supported).

  • State: Represents the current state of the resource, with respect to assignment. Valid states are "assigned" (to a service), "unassigned" (not assigned to a service), "assigning" (to a service), "unassigning" (from a service), "inprocess" (resource has been just assigned to RP and there is no info about resource state) and "error" (general resource problem).

  • Annotation: The annotation is a human readable segment of text, assigned by the Resource Provider, which provides additional information about the resources state for the administrator, such as text describing why the resource is in an error state.

  • Properties: Each host resource has a free definable set of name/value pairs defined in the properties list. Depending on the type of the resource some properties can be mandatory. The following example prints show the predefined properties of a host resource:

    
    % sdmadm show_resource_types
    name property                  flags type    
    ---------------------------------------------
    host ambiguous                 M     Boolean 
         annotation                      String  
         hardwareCpuArchitecture         String  
         hardwareCpuCount                Integer 
         hardwareCpuFrequency            String  
         operatingSystemName             String  
         operatingSystemPatchlevel       String  
         operatingSystemRelease          String  
         operatingSystemVendor           String  
         resourceHostname          M     Hostname
         resourceIPAddress               String  
         static                    M     Boolean 
         usage                     M     Usage   
    
                                

    All resource properties with the M flag are mandatory. All other properties are optional. The resource may have additional properties which are not defined in the type of the resource

    Hedeby system is using the following property values which have to be defined for a host resources:

    Table 2.1. Used host resource property names

    PropertyDescription
    resourceHostnameResolvable name of the host.
    resourceIPAddressIP address of the host.
    hardwareCpuArchitectureArchitecture type, e.g. "sparc64", "x86".
    hardwareCpuCountNumber of CPUs
    operatingSystemNameOperating system name, e.g. "solaris", "windows".
    operatingSystemReleaseOperating system version number.

    The following properties might be used in future releases:

    Table 2.2. Reserved host resource property names

    PropertyDescription
    hardwareCpuFrequencyCPU clock speed in MHz.
    operatingSystemPatchlevelOperating system patch level.
    operatingSystemVendorOperating system vendor, e.g. "Sun Microsystems".

2.2.4.1.2. What can I do with a resource?

The current implementation can add host resources to the Spare Pool. When the hosts resource is added the Resource Provider itselfs will assign the hosts to the services among reported needs from the services and the specified Resource Provider policies. The administrator can assign and remove the host resources from a service or to a service manually. For a detailed information about resource operations see Section 1.1.6.2, “Managing Resources” and Section 1.1.1, “Resource”.

2.2.4.2. Adding a Resource to the System

sdmadm add_resource|ar -pf <resource_properties> This command will add a resource to the system. Its up to Resource Provider to which service it will be added. Resource will go through the decision process and will end up in a service if needed or will be rejected if there is no service that wants that resource. The -pf switch is required. It specify the file name for a resource properties file.

                
% sdmadm -s test ar -pf resource1.properties    
    
            
Other optional switches: i - Id of the resource. It could be obtain from resource(generated). t - Type of the resource. You can specify type of the resource if not defined in resource properties file. s - Service id. Specify the service that you want resource to be assign to.

For more information about the add_resource command please have a look at Section 2.3.6.1, “Add resource to the system”

2.2.4.3. Removing a Resource from the System

sdmadm remove_resource|rr <resource_id> ... This command will remove a resource from the system. It is possible to specify just one resource or more at one time to remove. Specifying at least one is necessary.

                
% sdmadm -s test rr foo3.czech foo4.czech    
    
            

For more information about the remove_resource command please have a look at Section 2.3.6.4, “Remove resource to the service”.

2.2.4.4. Moving an existing resource to a Service

Normally Resource Provider does automatic resource assignments according to the configured SLOs. With the sdmadm move_resource resources can be moved manually. The resource id and the target service must be specified.

                
% sdmadm -s test mvr -r foo3.czech -s spare_pool2    
    
        

There is no guarantee that the resource is really moved the targeted service. It highly depends on the configured SLOs. It can also happen that after moving the resource to the target service Resource Provider moves this resource back to the orignal service.

For more information about the remove_resource command please have a look at Section 2.3.6.2, “Move resource to the service”.

2.2.5. How to Monitor the System

The current Hedeby system has the possibility to show the current available resources and services via command line interface. The components itself are logging information into log files in the Hedeby local spool directory on the local host.

2.2.5.1. Show Registered Services

The sdmadm show_services command is used to show all services which are managed by the Resource Provider component:

                
    %  sdmadm show_services
    host  service    cstate  sstate
    --------------------------------
    host1 spare_pool STARTED RUNNING
    host2 ge         STARTED RUNNING

    
            

The listing shows that the Hedeby system is managing two services. Service spare_spool runns on host host1, service ge runs host host host2. The components representing representing the services are both running. The state of both services is RUNNING.

2.2.5.2. Show available Resources and their States

The sdmadm show_resource|sr command is used to show all resources and their states in Hedeby.

                
    % sdmadm -s test sr
    service    id     state    type flags usage annotation            
    ------------------------------------------------------------------
    p31112     hostA  ASSIGNED host S     50    Got execd update event
               hostB  ASSIGNED host       50    Got execd update event
    p31114     hostC  ASSIGNED host       50    Got execd update event
               hostD  ASSIGNED host S     50    Got execd update event
    p31116     hostE  ASSIGNED host S     50    Got execd update event
               hostF  ASSIGNED host       50    Got execd update event
    spare_pool hostA  ASSIGNED host       1                           
               hostH  ASSIGNED host       1 
    
            

The sdmadm show_resource|sr -s <service_id> command is used to show all resources and their states just for given service id.

                
    % sdmadm -s test sr -s spare_pool
    service    id     state    type flags usage annotation
    ------------------------------------------------------
    spare_pool hostA  ASSIGNED host       1               
               hostH  ASSIGNED host       1
    
            

The sdmadm show_resource|sr -r <resource_id> command is used to show resource state and service that its already assigned to for given resource id.

                
    % sdmadm -s test sr -r hostA
    service id     state    type flags usage annotation            
    ---------------------------------------------------------------
    p31112  hostA  ASSIGNED host S     50    Got execd update event
    
            

The sdmadm show_resource|sr -r <resource_id> -all command is used to show resource state and service that its already assigned to a given Resource id. By specifying -all switch we will get full information about given Resource like, type, annotation and all its properties.

                
    % sdmadm -s test sr -r hostF -all
    service id    state    type flags usage annotation            
    --------------------------------------------------------------
    ge      hostF ASSIGNED host       50    Got execd update event
        hardwareCpuArchitecture=x86
        hardwareCpuCount=1
        operatingSystemName=Solaris
        resourceHostname=hostF
    
            
type - indicates the type of the resource(i.e. host). annotation - indicates the reason why are resource has reached this state. properties - indicates all properties that describe the resource. More information about the resource and it properties you can find here Section 2.2.4.1.1, “Resource definition for the first use case”

2.2.5.3. JVM Log Files

Each JVM of a SDM system writes log message into the subdirectory log in the local spool directory. The following files are stored there

  • <jvm name>-<sequence-number>.log - SDM uses the Java logging module. All log messages of a jvm are written into these files. The jvm will perform a log file rotation.
  • <jvm name>.stdout - All messages written to stdout are redirected into this file. This file should have the size 0.
  • <jvm name>.stderr - All messages written to stderr are redirected into this file. This file should have the size 0.

The configuration of the Java logging module is stored in the local spool directory in the file logging.properties. Changes in these file will become active after the next restart of the jvm. For more information about the Java logging module please have a look at http://java.sun.com/j2se/1.5.0/docs/guide/logging/index.html

2.2.5.3.1. Control log file rotation

With the default configuration the maximum size of a log file is 5MB, the maximum number of logging files is 4. This values can be adjusted by changing the following lines in logging.properties:

# approximate maximum amount to write (in bytes) to any one file. If this is zero,
# then there is no limit. (Defaults to no limit).
# set the maximum amount per file to 5 MBs
#   (1 MB = 2^20 bytes = 1048576 bytes )
java.util.logging.FileHandler.limit= 5242880

# how many output files to cycle through (defaults to 1).
java.util.logging.FileHandler.count = 4
        
2.2.5.3.2. Control log levels

The Java logging module defines the following log levels:

  • SEVERE (highest value)
  • WARNING
  • INFO
  • CONFIG
  • FINE
  • FINER
  • FINEST (lowest value)

The global log level is defined in logging.properties. With the default installation the globel log level is set to INFO.

.level=INFO
    

The Java logging module provides different loggers for writing log messages. For each logger the logging level can be defined in logging.properties. SDM provides for each java class a separate logger. The name of the logger is the full qualified Java class of the class. It is possible to define the individial log levels on java package name level or java class level.

All java class of the SDM system are in a sub package of com.sun.grid.grm. The following configuration sets the log level of all SDM logger to FINE.

# set log level for the java package com.sun.grid.grm to FINE
com.sun.grid.grm.level=FINE
    

For debugging the resource provider and service it is useful to set the log level if the package com.sun.grid.grm.service to FINE.

# set log level for the java package com.sun.grid.grm.service to FINE
com.sun.grid.grm.service.level=FINE
    

2.2.5.4. Reporter Component

For more information about reporter component see Section 2.4.3.2, “Reporter” and Section 1.1.4, “Reporter”. Reporter component stores its files in "<local_spool>/spool/reporter" directory at startup. The listing below shows the files for the Reporter component while running:

                
    % localSpool/spool/reporter ls   
    report-0.log 
    report-0.log.lck
    ...
    
            
The file contains data that are in the Arco database format. They aren't much readable for the user and they look like this:
                
1200415399278:NOTIFICATION:1:CS_OBJECT_ADDED:10:cs:Configuration object [resource_provider@ge6] was added under [active_component] context to the CS.
1200415399278:CONFIGURATION:1:resource_provider:active_component:Other
1200415399293:NOTIFICATION:2:SERVICE_ADDED:1:spare_pool:Service [spare_pool] has been added.
1200415399633:NOTIFICATION:3:CS_OBJECT_ADDED:11:cs:Configuration object [reporter@ge6] was added under [active_component] context to the CS.
1200415399633:CONFIGURATION:3:reporter:active_component:Other
1200415425377:NOTIFICATION:4:CS_OBJECT_REMOVED:12:cs:Configuration object [spare_pool@ge6] was removed from [active_component] context in the CS.
1200415425377:CONFIGURATION:4:spare_pool:active_component:com.sun.grid.grm.service.Service
1200415425382:NOTIFICATION:5:CS_OBJECT_REMOVED:13:cs:Configuration object [resource_provider@ge6] was removed from [active_component] context in the CS.
1200415425382:CONFIGURATION:5:resource_provider:active_component:Other
....
    
            
so the better way to obtain the data is to use CLI commands.

The command to print out all the data from reporter is sdmadm show_history | shist. This command will print out all the data that are stored in the reporter files. The output will look like this:

                
time_stamp              type                        service_name      resource_id           description                                                                                       
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
16/01/2008 11:28:07.888 SERVICE_ADDED               spare_pool                              Service [spare_pool] has been added.                                                              
16/01/2008 11:28:08.073 CS_OBJECT_ADDED             cs                                      Configuration object [resource_provider@ge6] was added under [active_component] context to the CS.
16/01/2008 11:28:08.303 CS_OBJECT_ADDED             cs                                      Configuration object [reporter@ge6] was added under [active_component] context to the CS.         
16/01/2008 11:28:34.855 RESOURCE_ADD                resource_provider ge6@resource_provider Resource is going to be added to temporary storage                                                
16/01/2008 11:28:34.869 RESOURCE_ADDED              resource_provider ge6@resource_provider Resource has been added to temporary storage                                                      
16/01/2008 11:28:34.878 RESOURCE_ADD                resource_provider ge6@spare_pool        Resource is going to be added to temporary storage                                                
16/01/2008 11:28:34.887 RESOURCE_ADDED              resource_provider ge6@spare_pool        Resource has been added to temporary storage                                                      
16/01/2008 11:28:35.092 RESOURCE_ADD                spare_pool        ge6                   Resource [ge6] is going to be added to spare pool service [spare_pool].                           
16/01/2008 11:28:35.122 RESOURCE_ADDED              spare_pool        ge6                   Resource [ge6] has been added to spare pool service [spare_pool].                                 
16/01/2008 11:28:35.129 RESOURCE_REQUEST            spare_pool                              Service [spare_pool] sent resource request that was triggered by [PermanentRequestSLO] SLO.       
16/01/2008 11:28:35.135 RESOURCE_REMOVE             resource_provider ge6@resource_provider Resource is going to be removed from temporary storage                                            
16/01/2008 11:28:35.136 RESOURCE_REMOVED            resource_provider ge6@resource_provider Resource has been removed from temporary storage                                                  
16/01/2008 11:28:35.172 RESOURCE_PROPERTIES_CHANGED resource_provider ge6@spare_pool        Resource in temporary storage has been updated                                                    
16/01/2008 11:28:35.173 RESOURCE_REMOVE             resource_provider ge6@spare_pool        Resource is going to be removed from temporary storage                                            
                        RESOURCE_REMOVED            resource_provider ge6@spare_pool        Resource has been removed from temporary storage                                                  
16/01/2008 11:28:35.174 REQUEST_QUEUED              spare_pool                              Request queued in resource provider.                                                              
                        REQUEST_PROCESS             spare_pool                              Request is being processed in resource provider.                                                  
16/01/2008 11:28:35.175 REQUEST_PENDING             spare_pool                              Request is pending in resource provider to be processed.                                          
                        REQUEST_PROCESSED           spare_pool                              Request was processed by resource provider.                                                       
16/01/2008 11:28:35.199 RESOURCE_PROPERTIES_CHANGED spare_pool        ge6                   Resource [ge6] has been modified in spare pool service [spare_pool].                              
16/01/2008 11:28:35.201 REQUEST_QUEUED              spare_pool                              Request queued in resource provider.                                                              
16/01/2008 11:28:35.202 REQUEST_PROCESS             spare_pool                              Request is being processed in resource provider.                                                  
16/01/2008 11:28:35.203 REQUEST_PENDING             spare_pool                              Request is pending in resource provider to be processed.                                          
16/01/2008 11:28:35.204 REQUEST_PROCESSED           spare_pool                              Request was processed by resource provider.                                                       
16/01/2008 11:28:35.225 REQUEST_QUEUED              spare_pool                              Request queued in resource provider.                                                              
16/01/2008 11:28:35.226 REQUEST_PROCESS             spare_pool                              Request is being processed in resource provider.                                                  
16/01/2008 11:28:35.227 REQUEST_PENDING             spare_pool                              Request is pending in resource provider to be processed.                                          
                        REQUEST_PROCESSED           spare_pool                              Request was processed by resource provider.                                                       

....
    
            
column time_stamp - show us the time_stamp when event occur in the system. column type - this is the type of the notification. column service_name - the service or component on which event occur column resource_id - the resource id of resource that was involved in the event. column description - comprehensive description of what acctually happend in the system.

You can always use "grep" command to parse an output of the shist command, but there are some more convient ways to obtain needed data. By using shist command with its option you can filter the output as you want. More information about shist command you can find here: Section 2.3.3.6, “Show History”

2.2.6. Autostart Feature in Hedeby

Currently, the Hedeby support, for autostart during the bootstrap of the system, is done in two possible ways: using startup scripts or by leveraging from Solaris SMF (Service Management Facility) support. Autostart is supported only for systems that are installed using SYSTEM preferences. The autostart is host specific feature. This means that one host in the system can have autostart feature and the others do not need to have this feature installed. During the installation of the system (for master or managed host) flag -autostart needs to be specified to install autostart feature. If the current host supports SMF, the SMF support will be installed. Otherwise the autostart will be done using bootstrab scripts.

2.2.6.1. Support with Bootstrap Scripts

Currently, the scipts installed during installation are for all Hedeby systems on the host. This means that once the scripts are installed all Hedeby systems with flag autostart will be started during boot time of machine. Shutdown will also be triggered for those system when the machine is going down. The scripts support will be uninstalled once the last Hedeby system on current host is being uninstalled using sdmadm uninstalled command. sdmadm uninstall_hosts is described in

2.2.6.2. SMF Support

SMF support is installed when the autostart flag was set in installation command and current host support SMF. Once such installation is completed the services are automatically enabled. This means that after installation the system on current host is started.

The services are installed per one JVM. This means that each JVM. in the system, has one SMF service. To check the status of services the command: svcs should be used. After the installation the output of svcs command should be the same as in example below:

                
# svcs "*sdm*" STATE          STIME    FMRI
online         13:56:29 svc:/application/management/sdm/smfsystem/cs_vm:default
online         13:56:38 svc:/application/management/sdm/smfsystem/jvm:rp_vm
online         13:56:38 svc:/application/management/sdm/smfsystem/jvm:executor_vm

                
        

We can distinguish two types of Hedeby services, first one is service for CS component. There is only one such service in the whole system and it represents the JVM with Configuration Service (CS) component. The other services that reside on master host have dependency on CS_VM service. The SMF services, that are running on managed host, do not have dependecy on CS_VM as it is not possible to check whether the service on remote host is running.

Note: If the CS_VM service onmaster host is not online ( cs_vm in system is not running), the start of SMF services on managed hosts will fail and the service will go into maintenance state. Once SMF service is in maintenence state it can be brought back online by using command svcadm

                
# svcadm enable svc:/application/management/sdm/smfsystem/cs_vm:default
# svcadm disable svc:/application/management/sdm/smfsystem/cs_vm:default
# svcadm clear svc:/application/management/sdm/smfsystem/cs_vm:default
                
        

In the example above there are three basic operations that can be done on SMF service using svcadm.

Table 2.3. Supported system properties

CommandDescription
svcadm enable svc:/application/management/sdm/smfsystem/cs_vm:defaultThis command will try to start the SMF service. In this particular example it will try to start cs_vm. If the SMF service was already in "online" state, nothing will be done. If the service was in "disabled" state the cs_vm will be started. If the start operation will fail, the SMF service will go into maintenance state.
svcadm disable svc:/application/management/sdm/smfsystem/cs_vm:defaultThis command will try to stop the SMF service. In this particular example it will try to stop cs_vm. If the SMF service was already in "disabled" state, nothing will be done. If the service was in "online" state the cs_vm will be stopped. If the stop operation will fail, the SMF service will go into maintenance state. If the service was in the "maintenance" state, the state will be changed to "disabled"
svcadm clear svc:/application/management/sdm/smfsystem/cs_vm:defaultThis command will try to restart the SMF service. In this particular example it will try to restart cs_vm. This command takes affect only if service is in "maintenance" state. If the "clear" operation will finish successfully the SMF service will become "online".

Note: The SMF service can also be in temporary states (online/disabled) that means that this states are valid till the reboot of machine. This means if service is temaporary in "disabled" state, it will try to become online after the reboot of machine.

The usage of sdmadm can influence on SMF services. This means that if SMF service for rp_vm is online, and the administrator will shutdown the rp_vm using sdmadm command, the SMF will become temporary disabled (it will become available after the reboot of host). The example can be found in the following example.

                
# svcs "*sdm*"
STATE          STIME    FMRI
disabled       14:12:33 svc:/application/management/sdm/smfsystem/jvm:executor_vm
online         13:56:38 svc:/application/management/sdm/smfsystem/jvm:rp_vm
online         14:38:04 svc:/application/management/sdm/smfsystem/cs_vm:default
# sdmadm -s smfsystem sdj -j rp_vm
Shutdown has been triggered for following JVMs: [rp_vm]
# svcs "*sdm*"
STATE          STIME    FMRI
disabled       14:12:33 svc:/application/management/sdm/smfsystem/jvm:executor_vm
disabled       14:39:01 svc:/application/management/sdm/smfsystem/jvm:rp_vm
online         14:38:04 svc:/application/management/sdm/smfsystem/cs_vm:default
# svcs -lp svc:/application/management/sdm/smfsystem/jvm:rp_vm
fmri         svc:/application/management/sdm/smfsystem/jvm:rp_vm
name         Service Domain Manager - SDM
enabled      false (temporary)
state        disabled
next_state   none
state_time   Tue Jan 15 14:39:01 2008
logfile      /var/svc/log/application-management-sdm-smfsystem-jvm:rp_vm.log
restarter    svc:/system/svc/restarter:default
contract_id
dependency   require_all/none svc:/system/filesystem/local (online)
dependency   optional_all/none svc:/system/filesystem/autofs (online)
dependency   require_all/none svc:/milestone/network (online)
dependency   optional_all/none svc:/network/nfs/client (online)
dependency   require_all/none svc:/application/management/sdm/smfsystem/cs_vm:default (online)
                
        
2.2.6.2.1. Naming explanation

The following table explains the naming structure for SMF service.

Table 2.4. Supported system properties

Prefix that should be the same for all Hedeby SMF services. It describes the type of service.The name of application - Service Domain ManagerName of the Hedeby system.Type of JVM - for CS component JVM it is "CS_VM", otherwise it is "JVM"Name of the instance - for CS component JVM it is "default", otherwise it is the name of JVM
svc:/application/management/sdm/ smfsystem/ cs_vm: default
svc:/application/management/sdm/ smfsystem/ JVM: rp_vm

2.2.6.2.2. Command for showing smf support

To see the installed SMF services for given system the sdmadm ssmf should be used. It prints the table with JVM_Names and corresponding SMF service names. If SMF is not supported on the host the proper message is printed that SMF is not supported.

   
                
jvm name    service name
---------------------------------------------------------------------
cs_vm       svc:/application/management/sdm/smfsystem/cs_vm:default
executor_vm svc:/application/management/sdm/smfsystem/jvm:executor_vm
rp_vm       svc:/application/management/sdm/smfsystem/jvm:rp_vm

                
            

2.2.7. How to manage security

TODO, needs cleanup