techniques  specifically  for  the  detection  of  moving future research. objects  that  are to be grasped  by  a robotic   system.

They   use   a   complex   motion   model   for  tracking

objects moving within a fixed plane. The robot control system used in their research is very similar to ours, utilizing many processing devices (operating at different rates) through the use of predictive filtering. However, this system requires additional modifica- tions before it can be directly applied to situations with multiple moving objects. In addition, it assumes the  use  of a static camera.

Considering the visual tracking portion of our framework, we rely on the sum-of-squared differ- ences algorithm discussed by Anandan'5 as a means for calculating displacement vectors. This algorithm has previously been used by Papanikolopoulos' in his

implementation of controlled active vision, and it has

also  been  used  by  Tomasi  and  Kanade l6  to measure

the suitability of feature windows for  tracking.  A very   interesting   approach   to  the  computer vision

aspect of the visual tracking problem has been proposed by Nayar ei of. 17

The use of tracking information as feedback to our robot  control  scheme  is based  on  a MIMO adaptive

controller    of    Feddema    and    Mitchell l8     Similar

adaptive   schemes   have   previously   been   used by

Koivo  and   Houshangi,  9  Weiss   et   at  20   and Nelson

and Khosla" for the vision-based control of manipulators.    Moreover,    adaptive   schemes   have

been  used  by  Brow   22  and  Dickmaiins  and  Zap    23

for the control of various other mechanical systems (e.g. robotic heads, satellites and cars). This   research

has also been influenced by the work of Ghosh and Loucks 2     who   have   proposed    the   use   of   the

“perspective theory” (an elaborate adaptive scheme) for the computation of motion and structure in machine vision.

2. DETECTION OF OBJECTS OF INTEREST

2. I . Theoretical basis of the detection problem

The basis of the detection framework is that every image consists of pixels belonging to one of two categories: figure or ground. Figure pixels are those which are believed to be part of the projection of an object of interest, while ground pixels correspond to that object’s surroundings. If we allow the possibility of processing multiple objects, then a pixel is figure if it is part of any of the objects’ projections. We formalize the detection problem as the identification and the analysis of figure pixels in each frame of a temporal sequence of images, i.e. the computation of fax, y, /)

1 if f, (x,y, i)   0 otherwise