
% Template for a simple journal or conference style article

\documentstyle[12pt,fullpage]{article}
\begin{document}
\pagestyle{plain}
\title{\bf Implementation of Farming Under PVM}
\author {Sameer S. Shende\\ Department of Computer and Information Science \\
University of Oregon}

\maketitle
\vspace*{.3in}

\begin{abstract} 
Dynamic load balancing can improve the performance of a parallel application 
significantly. This is needed in the PVM environment because different processors in the network have different processing capacities, networks have different
data carrying capacities which directly affects the latency of communication,
in a timesharing environment other user processes can slow down the processing 
capacity of a particular node in a network for the tasks in a given parallel 
application and a large number of data parallel algorithms exist which require unequal
processing times for each quantum of the data given to the processor. 

This paper explains the design and implementation of one such load balancing 
scheme under PVM, which is fast becoming a popular parallel programming environment both on a network of workstations, multiprocessors and other parallel and
vector platforms. 
\end{abstract}


\section{Introduction}

This paper focuses on implementation of process farms under the parallel virtual machine environment. PVM uses a cluster of workstations to form a heterogenous network based MIMD computing environment. The problems associated with 
such an environment are that, to form a parallel application, typically point 
to point communication constructs are provided to distribute data in a data parallel approach. Most parallel applications need some form of load balancing 
technique to equitably distribute work packets and keep all processors busy.
This is usually a static off line process in which a particular application
is evaluated for a particular problem and a log is examined to determine the
placement of 
tasks on the nodes. The application has to explicitly distribute the data so as to have equal load on all processors.

\section {Load Balancing}
\subsection{Need for Dynamic Load Balancing under PVM}

In PVM, the user tasks which comprise a parallel application are distributed
over the nodes in a network. The nodes could be parallel/vector machines or 
workstations. Thus, a heterogenous computing environment is used to solve the
problem. To extract maximum performance from the parallel application, it is
important to keep the idle time of tasks to a barest minimum. To ensure that
each process gets an optimum load to be processed, a dynamic load balancing 
scheme is desirable. The relevant issues that the application developer has to
take into consideration are :
\begin{enumerate}
\item All nodes in the network do not have identical compuation capacities.
Even among workstation class machines, different processor architecture,
network interfaces, and I/O subsystems can be encountered. Some machines 
can have processing rates which are an order of magnitude higher than some 
other machines in the network.
\item A parallel application running over a wide area network will encompass
a range of networks which have different transfer rates and latencies. For
e.g., Ethernet LANs have 10 MB/s rates, WANs have 64 Kbps (X.25), Token ring
LANs have 16 MB/s capacities. This results in different message transfer rates
between tasks in the parallel application.

\item There are background jobs in most time sharing multiuser systems (typical
unix workstations) so the the CPU time that a task under PVM can get may 
vary from one run to another depending on the usage of a workstation.

\item A large number of algorithms are not structured nicely and on breaking
the data to be processed into work packets, each work packet may require a 
different processing time (even if we keep the above factors constant). This
processing time requirement cannot be always determined apriori and can  be 
difficult to account for while designing the application.
\end{enumerate}

\subsection{Dynamic Load Balancing}

Dynamic load balancing in an application can address these concerns. Dynamic load balancing can
be implemented in a heterogenous environment either at the operating system 
level or at a process lavel. 
\begin{enumerate}
\item At the {\em operating system level}, the emphasis is to
keep all the {\em processors} busy all the time. This is accomplished by process
migration from a heavily loaded node in the network to a lighlty loaded one.
This raises issues about doing this effectively at runtime on a heterogenous
collection of nodes. While it may be easy to incorporate such a scheme for a
``state-less'' server process by killing one instance and starting another 
instance on the other node (different binaries being used for dissimilar 
processors and operating systems), it can be difficult to migrate a process
at run-time when the the state of a process needs to be maintained. The process
migration approch can be seen in systems such as Condor.

\item At the {\em process level}, it is important to keep all the {\em user tasks}
busy all the time. If the data to be processed is divided into independent work packets, then the load balancing scheme at the process level can be implemented
 by giving more work packets to process to 
the tasks on faster processors and less work packets to the slower ones. This way, the
data can be distributed on the basis of the processing capacity of a particular
task. This approach takes into account the differences in processing times for
different data packets. The aim of the paper is to implement a dynamic load
balancing scheme under PVM using this approach by constructing {\em process farms}. 
\end{enumerate}

\section {Farming}
\subsection{Farming - Definition of terms}
Under farming, the parallel application is divided into one or more process farms. A process farm comprises one farmer task and one or more worker classes. A 
worker class has one or more worker tasks which are grouped together because
they perform similar functions. A worker (task) can be associated with one and
only one farmer task (and in turn one worker class). In a {\em hierarchical farm}, a worker task can in turn be a farmer to some other set of worker classes and worker tasks. Again, the worker can have multiple farms to which it is a farmer. This could be useful in subdivision of a work packet into sub parts for
processing by another worker class hierarchy. Also, the farm implementation should be such that a farm should be free to leave a process farm (dynamic process
migration to another farm) and join any other farm if needed. 

To implement an algorithm using the farming paradigm, the data parallel application would need to create one or more farms. The farmer task would need to take
the data to be processed and split it up into a number of work packets which 
would be  independent of each other and could be processed by a worker task. It
would then give out the work packets to a worker class without knowing which
particular worker task would process it. The operating environment would decide
the appropriate worker to forward the work packet to. After processing the work
packet, the worker would create a reply packet and address it to the owner of
the process farm to which it would be attached. The farmer task would collect 
all the work packets from a particular worker class and reassemble these 
appropriately. More than one farm and more than one worker class could be active
at a particular instant creating, sending or receiving packets. The farmer could
terminate a farm after it is no longer needed at which stage the worker tasks 
would automatically be expelled from the farm and informed accordingly. However,
before this happens, a worker should also be able to leave the farm and stop
receiving work packets from the worker class.   

\subsection {Farming - Classification}
To focus our attention on the operating environment (or a farm daemon task), it should be able to manage
the creation of the farms and operations such as termination of a farm, leaving
a farm (by a worker task), and routing of work packets to the worker. There are
ways in which the packet distribution to a worker can occur. Farms are classifed into these categories.
\begin{enumerate}
\item {\em Static Farm}

In this case, the farm daemon could distribute all the work packets, that it 
has at any instant, equally to all the worker tasks. The advantage of this 
scheme is that the worker task 
would not need to request the farm daemon for a work packet but would retrieve
it locally from its queue locally. The diadvantage of this scheme is that at
run-time if another worker task joins a worker class after all work packets 
have been didtributed, then a redistribution of work packets would not take
place and the new worker task could be idle. Also, this approach does not 
take into account the different processing requirements for each work packet.
So, in this approach the farm is static in the sense that all worker tasks can
process an equal number of packets only and thus does not provide for dynamic
load balancing.

\item {\em Dynamic Farm}

In this case, the farm daemon would distribute the work packets to workers
only when the worker would request for a work packet. Thus, different workers
could receive different number of work packets depending upon their respective
processing capacities. Also, this scheme is suitable to a farm which allows 
migration of a worker task from one farm to another as the new worker task could receive a work packet as these are all contained in the central farm daemon and
it could be integrated into the farm easily. The advantage of the dynamic farm
is that it can support worker task migration (between farms) efficiently, it can
distribute work packets depending upon the processing capacities of the worker
tasks and keep all the workers busy most of the times and so it implements a 
dynamic load balancing scheme. The disadvantage of a dynamic farm is that it 
requires the work packets for a farm to be stored at a central location which
requires a large storage to be allocated at the farm daemon. Also, the worker
tasks need to request for each work packet and so there is a greater latency 
in getting a work packet as compared to the static farm case where all
the work packets are locally stored and can be retrieved without any additional
messages to be sent to the central server (farm daemon). 

Different distributed data management schemes with respect to locks are 
discussed in \cite{th:tom} . These schemes are analogous to managing the work
packets in the case of process farms.
\begin{figure}
\input{f1.pic}
\caption{Representation of Farming}
\end{figure}
\end{enumerate} 

\subsection{Goals of the farming implementation}

The aim of this paper was to implement a hierarchical farm which supported
worker task migration from one farm to another and implemented a dynamic load
balancing scheme by using a dynamic farm model described above. It needed to 
support multiple farms at each level with multiple worker classes at each farm
level. Furthermore, it was necessary to formulate an Application Programmers
Interface (API) for the farm which was independent of the nature of the farm
(static or dynamic farm implementation) and would work with the PVM environment.
It was important to design it in such a way that there would be {\em no polling}
at any stage in the components. Any polling for an event to happen would only
degrade the system performance.
Also, it was a goal to use the farming constructs as a layer above the PVM
software and to use the interface of PVM to build the farming model. This was
necessary to support the farm implementation across different versions of 
PVM (starting from 3.2.0) so that it would give backward source code compatibilty (as long as PVM maintained backward compatibility).

\subsection{Previous work in this area - Comparison}
Work related to implementation of process farms has been done on transputer
based environments which are typically a distributed memory MIMD message passing
environments. The work done at EPCC \cite{pul:tf} on process farming runs on top
of their CHIMP environment which also works on a heterogenous computing platform
in the form of workstation clusters. Their implementation of farming however 
consists of a source process and a sink process (instead of a single farmer 
task). 
Implementations of static farms with a worker task specified queue lengths (of
the local worker task queue of work packets) have been done at C-DAC, India 
\cite{corec} and TUL, India \cite{sss} but have been restricted to a homogeneous
environment. The advantage of \cite{sss} over this papers implementation is
that the farm related information is maintained in a distributed manner and 
so when multiple farms are implemented it would have no bottlenecks at the 
central server site. This may happen in our case if all farms simultaneously
dispatch  work packets to the workers. 

The author is not aware of any process farm implementation under the PVM 
heterogenous computing environment and this work would aim at bridging the
gap of providing a dynamic load balancing construct to expand the PVM API.

\section{Implementation and Design Issues}
The implementation of process farms consisted of
\begin{itemize}
\item Defining the API for farming.
\item Designing the software components viz., the farm daemon (pvmfarmd - 
the server task) and the library of the API (libfarm.a - the client) which 
would be used by both the farmer and the worker tasks.
\item Defining the protocol for interprocess communication between the
farmer task and the farm daemon task, and between the worker task and the
farm daemon task so as to avoid any polling by any of the tree components.
\end{itemize}
\subsection{API definition}
{\em At the farmer end:}
\begin{enumerate}
\item
{\bf int 	pvm\_farm\_init (char *farm\_name); }

The routine pvm\_farm\_init() creates a process farm with a name specified by the 
argument {\em farm\_name}. In an application, the farm name is unique (i.e., the 
user should not invoke this routine twice with the same name). Multiple farms 
can be used with a separate farm names. This name is used by the worker task to enroll in a farm.
	Returns OK or ERROR.
\begin{verbatim}
	e.g.,
	pvm_farm_init("Farmer1");
\end{verbatim}
\item
{\bf int pvm\_get\_worker\_class\_id (char *farm\_name, char *worker\_class\_name); }

The routine pvm\_get\_worker\_class\_id() returns the identifier that 
represents a worker class named {\em worker\_class\_name} associated 
with the farmer named {\em farm\_name}.  This is used by the farmer to uniquely
identify a particular worker class, to send work packets and receive reply 
packets. Returns ERROR on error or a positive integer {\em id}. This identifier
is used by subsequent farmer calls to send data packets to this worker class or
 to receive replies from the worker class.
\begin{samepage}
\begin{verbatim}
	e.g.,
	id = pvm_get_worker_class_id("Farmer1", "WC1");
	if (id == ERROR)
	{
		printf("Error in getting worker class id\n");
		pvm_exit();
		exit(1);
	}
\end{verbatim}
\end{samepage}
\item
{\bf int	pvm\_send\_work\_packet(int id); 	 }

The routine pvm\_send\_work\_packet() sends the message to one worker task associated with the 
worker class whose identifier is specified by id. The data should be packed in
a buffer (the active send buffer) created by pvm\_initsend() or pvm\_mkbuf();
PvmDataDefault is used to create the buffer so that XDR encoding can be done
for communication between dissimilar hosts (little endian and big endian hosts).
	Returns OK or ERROR.
\begin{verbatim}
	e.g.,
	int id, data, bufid; 
	id = pvm_get_worker_class_id("Farmer1","WC1");
	bufid = pvm_mkbuf(PvmDataDefault); /* pvm call creates buffer */
	pvm_setsbuf(bufid); /* sets this as the active send buffer
		This is also the default action taken */
	data = 500; /* gets data ready for packing in the buffer */
	pvm_pkint(&data, 1, 1); /* packs one int data item, stride = 1 */
	/* calls to pack the data */
	pvm_send_work_packet(id, 34);
	pvm_freebuf(bufid); /* frees the buffer */
\end{verbatim}

\item 
{\bf int	bufid = pvm\_recv\_reply\_packet(id); }

This blocking receive reply packet routine will wait until has arrived from the
worker class specified by the identifier {\em id}.  This call 
places the message in a new active receive buffer that is created. The previous
active receive buffer is cleared unless it has been saved with a pvm\_setrbuf() call.
\begin{verbatim}
	e.g.,	
	pvm_recv_reply_packet(id);
\end{verbatim}

\item 
{\bf int	pvm\_farm\_terminate(char *farm\_name); }
	
This routine shuts down a farm and frees the worker classes associated with the farm whose name is specified by {\em farm\_name}.
The tasks waiting on a receive are sent a message that the farm has terminated.
Returns OK or ERROR.
\begin{verbatim}
	e.g.,
	pvm_farm_terminate("Farmer1");
\end{verbatim}

\item 
{\bf int	pvm\_start\_farmd(void); }

This routine starts the pvmfarmd daemon which is responsible for farming 
related calls. This daemon can be started manually from the pvm shell or by
using this call. This should be called only once in the entire application.
Returns OK or ERROR.
\begin{verbatim}
	e.g.,
	pvm_start_farmd(); /* starts the daemon.*/
\end{verbatim}
\item 
{\bf  int	pvm\_stop\_farmd(void); }

This routine sends a message to the pvmfarmd to terminate itself. This should
also be called by only one task in the application. This should be done by the
farmer task after it has terminated the farm.
Returns OK or ERROR.
\begin{verbatim}
	e.g.,
	pvm_stops_farmd(); /* stops the daemon.*/
\end{verbatim}

	
{\em At the worker end:}
\item 
{\bf int	pvm\_init\_worker\_class(char *farm\_name, char *worker\_class\_name); }

The pvm\_init\_worker\_class() routine creates (if it doesn't exist) a worker class
named {\em worker\_class\_name} and enrolls the calling task in the worker class.
This worker class is associated with the farmer task specified by {\em farm\_name}. This call blocks till a farm is created (in case the farmer has not created the farm).
	Returns OK or ERROR.
\begin{verbatim}
	e.g.,
	pvm_init_worker_class("Farmer1", "WC1");
\end{verbatim}
\item
{\bf int	bufid = pvm\_recv\_work\_packet(void);  }

This blocking routine will wait until a message arrivs from the farmer task.A 
worker task can be associated with at most one farmer (and a worker class)  at
a time. The previous active receive buffer is cleared unless it has been 
saved with a pvm\_setrbuf() call. This call returns bufid or PvmFarmTerminated
if the farmer has executed a farm\_terminate() call in which case the worker is
free to do any other activity not associated with the current farm.
\begin{verbatim}
	e.g.,
	buffer_id = pvm_recv_reply_packet();
	/* All unpacking routines */
\end{verbatim}

\item 
{\bf int	pvm\_send\_reply\_packet(void); }

	This routine sends a reply packet which has been packed into the 
active send buffer, to the farmer task.
	Returns OK or ERROR.
\begin{verbatim}
	e.g.,
	/* initialise the buffer for sending */
	int bufid,data;
	bufid = pvm_mkbuf(PvmDataDefault); /* makes a new buffer */
	pvm_setsbuf(bufid); /* done by default - sets this 
		as the active send buffer used by pvm_send_reply_packet() */
	data = 5; /* some reply packet */
	pvm_pkint(&data, 1, 1); /* packs the integer into the buffer */
	pvm_send_reply_packet(); /* sends the buffer to the farmer */
	pvm_freebuf(bufid);
\end{verbatim}
\item 
{\bf int	pvm\_leave\_farm(void); }

	This routine is called by the worker task to disassociate itself from
the farm in which it is currently enrolled. This call removes the task related
information of this task from the worker class information repository maintained
by the central pvmfarmd daemon (farm daemon). After this call is executed, the
calling worker task no longer receives work packets from the farmer. The worker
can enroll in any farm subsequently, by calling the pvm\_init\_worker\_class()
routine. This allows for the worker process to migrate from one farm to another
dynamically at runtime.  
	Returns OK or ERROR.
\begin{verbatim}
	e.g.,
	pvm_leave_farm(); /* leaves the farm */
\end{verbatim}
	
\end{enumerate}

\subsection{Design issues}
The process farms were implemented following the design of the farm daemon
and the definition of the protocol for interprocess communication as described 
in Appendix A of the paper. 
\begin{figure}
\input{f2.pic}
\caption{pvmfarmd - Internals}
\end{figure}
The farm daemon (pvmfarmd) was implemented using lists. The entire code was
written in C++ making list as a class with member functions like append, delete,
pop, search by name, search by value, etc. The farm daemon had a list of all
farms (each farm was identified by a unique name and farmer task id). Each item
in the farm list had a list of worker classes that were associated with it. 
Each item in the list of worker classes had a unique worker class identifier,
the list of work packets awaiting distribution and a list of worker tasks 
enrolled in the worker class. Apart from these lists, there was a global list
of all pending requests (e.g., the farmer requested for a worker class id and
the worker class was not created, so it had to wait for a worker class to be
created, etc.). When a work packet arrived at the farm daemon, it had to be 
forwarded to a single task. To search for the worker class to which the packet
had to be addressed, one approach was to search each item in the farmer list
for a matching farm name, then to get to the worker class list and then to 
search for the matching worker class id item. This was inefficient
when a large number of work packets were sent to the farm daemon. Thus, to 
improve performance, a cache of worker class ids was maintained. The entries
in this cache array pointed to the worker class item in the list of worker 
classes. So, in addition to the lists, the array of worker class items (indexed
by the unique worker class id) was maintained. This structure is illustrated in figure 2.


\begin{thebibliography}{99}
\bibitem{th:tom} Anderson T. E., Lazowska E. D., and Levy H. M., ``The Performance
Implications of Thread Management Alternatives for Shared Memory Multiprocessors
'', Department of Computer Science, University of Washington, Technical Report
88-09-04, Sep. 1988.
\bibitem{pul:tf} ``PUL-TF Prototype User Guide'', {\em EPCC User Guide}, 1992.
\bibitem{sss} Shende, S., and B. Ananda Krishna , ``Simulation of Concurrent
Runtime Environment of PARAM on UNIX'', {\em High Performance Computing, 
Proceedings of the Southern Regional Convention, Computer Society of India}, July 1994.
\bibitem{corec} ``CORE Concurrent Runtime Environment for C Users Manual'', C-DAC 
, India, 1991.
\bibitem{paras} ``PARAS Overview Manual'', C-DAC, India, 1991.
\bibitem{3lc} ``3LC Users Guide'', Inmos U.K., 1990.
\end{thebibliography}

\end{document}

% ------------ end of document ----------------------

% here are some templates of different environments you can cut and paste:

\begin{figure}
\vspace{3in}
\caption{}
\end{figure}

\begin{itemize}
\item
...
\end{itemize}

\begin{enumerate}
\item
...
\end{enumerate}

