Hedeby defines resource as an aggregation of properties (each property is key/value pair which represents required attribute of the resource - for more see Section 1.1.1, “Resource”). Currently only one type of resources (as defined by Hedeby) is available to be used with GE that is managed by Hedeby - it is a resource which type is "HOST".

"HOST" type of resource means that resource properties are describing a host computer (machine that communicates via a network). See Section 2.2.4.1, “Resource Definition Overview” for more details about current host resource implementation.

From the GE perspective host resource could be a computer that is running an instance of GE Queue Master, an instance of GE Execution Daemon or simply any computer (node) that is part of GE cluster. Thanks to Hedeby functionality host resource for a GE managed by Hedeby can be almost any computer - if there is need (expressed by GE cluster instance) and possibility (evaluated by Agent/Executor running on a host) Hedeby can add that host to GE cluster.

The way how resources are managed (added/removed to GE cluster, daemons are started/stopped) is affected by service level objectives (SLO). Hedeby supports several SLOs (see Section 1.2.1.5, “SLO/SLA”).

Sun GE cluster resources

The component responsible for communication between Hedeby and managed GE is GE service adapter that acts like a bridge for communication between Resource provider and the managed GE cluster. In general, GE service adapter is kind of special driver responsible for translating "generic" management functionality needed by Hedeby to "native" functionality of a GE - for example it is able to translate "remove resource" message originating from Resource Provider to a set of GE commands, that results into :

GE service adapter is also responsible for providing information that is needed by Resource Provider . It reports cpu load values, memory consumption, number of pending jobs etc. - hence everything that helps Resource Provider to make qualified decisions to satisfy needs of managed services.

Communication between service adapter and GE

Current implementation of GE service adapter is using custom Java interface to obtain vital information from underlying GE (this API is known as JGDI).

There is no 1:1 or similar relation between GE service adapter and GE components (queue master, execution daemon, shadow daemon etc.), in fact GE service adapter is using a combination of all GE components that all provide information and functionality needed for management of a GE cluster.

GE service adapter responsibilities currently involves:

Service adapter concepts are described in greater detail at Section 1.1.2, “Service”. More information about configuration is at Section 2.4.3.3, “Service Adapters, Grid Engine Adapter ”.

Although Resource Provider (RP in the next paragraphs) can be considered as a component that is independent of other components, its functionality has direct impact on managed GE clusters. Regarding the GE managed by Hedeby RP is responsible for following actions:

GE adapter is using executor for the following set of actions:

The executor commands that handle functionality related to GE (installation/uninstallation of execd, etc.) are currently implemented as shell commands or shell scripts (GE has high quality CLI interface which allows to do that).

Service Adapter and executor on a host resource

Once the executor daemon is installed on the host resource grid engine service adapter will use GE Java api to retrieve information from (or pass to) the resource.

Service Adapter and executor on a managed host resource

See Section 1.1.5, “Executor” for general concept of executor and Section 2.4.3.4, “Executors” for configuration details.

According to Hedeby project specification, SLO defines the acceptance state of a given service attribute (with appropriate metrics). Very simple example of SLO for a GE service is that "the number of pending jobs should be less than 100" in a GE cluster. More about SLO can be found at Section 1.1.2.4, “Service Level Objectives ( SLO) ”.

The original plan was to support following SLOs for managed grids :

While priority has been placed on other core Hedeby features, only a subset of the above mentioned SLOs has been implemented so far. Currently supported SLOs are summarized in the next table (with short comments).

Each of currently implemented SLOs has several attributes that can be customized. See samples with short explanation in the following tables.

Due fact that only host resource are currently supported core Hedeby functionality related to GE cluster management can be described by relatively short set of use cases. These use cases are compiled in the following chapters.

1.2.2.1.1.  GE cluster without Hedeby

The only instance of GE cluster (regardless of total nodes in the cluster) that is not managed by Hedeby is the simplest example of Hedeby functionality. As the GE cluster instance is not managed by Hedeby, there is not much that Hedeby can do - in fact it does nothing. All resources (the only supported resource type is "HOST" therefore resources equals nodes) are staticaly assigned to GE cluster.

Sun GE cluster managed by administrator

Only the GE administrator can add/remove nodes to cluster or start/stop GE daemons on nodes. To increase a performance of a GE cluster, administrator would have to perform following steps.

Procedure 1.1.  Increasing the performance of a GE cluster

1. Add the node configuration to the GE cluster.

   Administrator would use GE administrator interface (qconf or qmon).

2. Start the grid daemon (execution daemon) on the node.

   Administrator would use shell script to start the daemon (sge_execd).

A resource added by GE administrator - sequence diagram

Similar procedure is needed when administrator has to remove a node from a GE cluster (to use it in different GE cluster, for example)

Procedure 1.2.  Decreasing the performance of a GE cluster

1. Stop the grid daemon (execution daemon) on the node.

   Administrator would use shell command to kill the daemon ( pkill sge_execd or qconf -ke).

2. Remove the node configuration from the GE cluster.

   Administrator would use GE administrator interface (qconf or qmon).

A resource removed by GE administrator - sequence diagram

1.2.2.1.2.  Two GE clusters without Hedeby

Follow up to previous use case is the situation of two (3,4, many) instances of GE cluster (again, regardless of total nodes in the cluster) that none of them is managed by Hedeby. And again, as none of the instances is managed by Hedeby, Hedeby does nothing in this use case. All resources (the only supported resource type is "HOST" therefore resources are GE nodes) are statically assigned to a corresponding GE cluster.

Two Sun GE clusters managed by administrator

Only the GE administrator can add/remove nodes to cluster or start/stop GE daemons on nodes. To increase a performance of a GE cluster #2 using a node taken from a GE cluster #1 (assuming that it has spare capacity), administrator would have to perform following steps.

Procedure 1.3.  Increasing the performance of a GE cluster #2 using a spare resource from a GE cluster #1

1. Stop the grid daemon (execution daemon) on a node that is part of cluster #1.

   Administrator would use shell command to kill the daemon ( pkill sge_execd or qconf -ke).

2. Remove the node configuration from the GE cluster #1.

   Administrator would use GE administrator interface (qconf or qmon).

3. Add the node configuration to the GE cluster #2.

   Administrator would use GE administrator interface (qconf or qmon).

4. Start the grid daemon (execution daemon) on the node that is now part of the GE cluster #2.

   Administrator would use shell script to start the daemon (sge_execd).

A resource shuffled between two grids by GE administrator - sequence diagram

1.2.2.1.3.  Two GE clusters with Hedeby

Logical "upgrade" to previous use case is the situation where thoe instances of GE cluster are managed by Hedeby - from the Hedeby's view, those GE clusters are managed services. Each resource (the only supported resource type is "HOST" therefore resource equals node) are dynamically assigned to GE cluster which expresses need for additional resource (with the only exception of static/fixed hosts). Hedeby is therefore responsible for automatical adding/removing of nodes to relevant cluster or starting/stopping of GE daemons on related nodes.

Two Sun GE clusters managed by Hedeby

If a managed GE cluster #2 expresses need for a resource and if there is another managed GE cluster #1, Hedeby could take following set of actions.

Procedure 1.4.  Hedeby satisfies need of a managed GE cluster #2

1. SLO of the GE cluster #2 is not met

   Service Adapter for the GE cluster #2 gathers information and finds that at least one of the configured SLO for the GE cluster #2 is not met.

2. Resource provider receives non-satisfied SLO event from GE cluster #2

   Resource provider looks for any resource that can be taken and assigned to the GE cluster #2. Resource provider considers any resource that:

   • is assigned to a service (GE cluster) with lower priority than the GE cluster #2

   • is not marked as fixed resource

   • is owned by a Spare Pool

   Regarding on the details of the need, some of the resources are be left out (if they do not match needed host architecture, operating system version etc.).

3. Resource provider finds an appropriate resource.

   (Assuming that the only available resource matching expressed need for a resource is assigned to a GE cluster #1) Resource provider asks the related service container to release the resource.

4. The GE cluster #1 removes the node.

   Service container dispatch to its service adapter message that the resource should be released form the managed grid engine cluster #1. GE service adapter contacts the executor running on the node with order to stop the grid daemon (execution daemon). Service Adapter then removes the node configuration from the GE cluster #1.

5. Resource provider assigns the resource to GE cluster #2.

   Once the resource (node) is released from GE cluster #1, it is temporarily marked as "RESERVED" which makes it reserved fo assignment to GE cluster #2. Resource provider asks service container related to GE cluster #2 to add the resource. Service container delegates the message to its service adapter which first adds a configuration for the node to the GE cluster #2. Service Adapter then again contacts executor running on the node with order to start the GE daemon (execution daemon) - this time the node will be part of the GE cluster #2.

Although the above procedure looks kind of complicated, it is worth to notice that it does not involve any single interaction of a human administrator.

A resource shuffled between two grids by Hedeby - sequence diagram

Juggling resource back and forth between managed GEs is not always enough to satisfy their needs - sometimes it is more convenient to add additional (new) resources to managed grids (typical example is buying a new hardware). With Hedeby there is no need first to manually add such resource to one of the managed GE clusters - it is possible to add such resource to Spare Pool and let Hedeby decide what to do with it.

Procedure 1.5.  Adding a resource to Hedeby Spare Pool

1. Administrator starts the executor on the resource

   Running executor is a MUST HAVE requirement to manage any of the hosts. Executor can be deployed on the resource using several ways (see Section 2.1, “Install Hedeby”).

2. Administrator adds the resource to the Spare Pool

   Administrator would use Hedeby administrator's interface - typically it would be "sdmadm add_resource" shell command. Once the command execution finishes, the resource is available to be assigned to any of managed GE clusters.

Adding a host resource to Spare Pool by GE administrator - sequence diagram

Hedeby is supposed to manage 2 and more GE clusters (although in combination with Spare Pool it is possible to use it to manage only one GE cluster) and therefore it has to provide best scalability and performance. Test runs with 3 managed GE clusters and cca 3000 host resources showed very small memory and cpu consumption of Hedeby components, while overall response time of Hedeby system was very fast.

Next table shows list of basic actions and time that needs Hedeby to perform it.

The data introduced in this section were captured using following configuration:

The very first real-life application of Hedeby will be using it as an automatic resource manager for several Sun GE clusters. This premise has driven the implementation effort and caused that priority has been placed on certain of its features (while others are not implemented yet).

Here is the brief resume of all aspects that apply to usablity of Hedeby (in relation to Sun GE management) with short comments.

Hedeby as a Java application is virtually able to run on any platform that complies with Section 1.2.3.2, “Java”.

Hedeby does not rely on any special HW platform (although it contains few lines of a native code). In general it is aimed to support all platforms that are supported by GE 6.2.

Hedeby makes strong use of Java features that were introduced in JDK 1.5 (generics, concurrency utilities etc.) which makes any pre 1.5 version of Java unusable for it. During the Hedeby's implementation phase JDK 6.0 was released and Hedeby is working fine with Java 6, too.

Hedeby is using to manage GE set of GE features that were introduced only in the latest updates to Sun GE and some features that are part of the development maintrunc of the Sun GE (not available for public use yet). The main restriction applies to JGDI (Java wrapper to GDI) which as a private Java GE API provides vital functionality for Hedeby and is not part of any Sun GE distribution.

Despite the restrictions stated in the above paragraph, support of the most of Sun GE releases is planned with the release of Hedeby reloaded.