Orientation With No Frame Of Reference

Team

The OWNFOR project is funded by NSF SGER #0512167 to C.B. Paris and is involving:

Claire B. Paris, PI: OWNFOR concept and design
Cédric Guigand: OWNFOR design and technology
Jean-Olivier Irisson: OWNFOR software and documentation
Rebecca Fisher: Scientific advice
Charles Cousin: Engineering consultant - See Bellamare company

Publications

Irisson J-O, Guigand CM, Paris CB (2009) Detection and quantification of marine larvae orientation in the pelagic environment, L&O Methods 7:664-672.[pdf]
Paris CB, Guigand CM, Irisson J-O, Fisher R, D'Alessandro E (2008) Orientation with no frame of reference: A novel system to observe and quantify orientation in reef fish larvae. In: Caribbean Connectivity: Implications for marine protected area management. Grober-Dunsmore R and Keller B (eds), U.S. Department of Commerce, NOAA, National Marine Sanctuary Program, NMSP-08-07, Silver Spring, MD, Marine Sanctuaries Conservation Series NMSP-08-07. 54-64 .[pdf]

Press Releases

Turn Signals in Conservation magazine
Press release in RSMAS.

Introduction

Biological question

Most coral reef fish have a dispersive larval stage that is thought to distance them from reef predators. This dispersive stage ends when the well-developed larva leaves the pelagic environment and actively recruits back onto coral reef habitat. The supply rate of larvae is critical for the structure of reef populations and is recognized as an important factor in understanding the connectivity of marine populations. There has been considerable investment in high-resolution numerical simulations of hydrodynamic features around coral reefs to be able to model larval transport. Yet, a major question in marine ecology is how the larval stages of benthic marine organisms behave while in the pelagic environment long prior to settlement. Recent advances in the study of the sensory system of reef fish larvae as well as on their behavior have demonstrated that larval reef fishes have considerable capabilities that enable them to actively control their position in the ocean at relatively large scales. However, to have a significant impact on their dispersal and survival, coral reef fish larvae must be capable of orienting in the open ocean with respect to the location of nearest coral reefs and prevailing currents. They should be able to detect the direction of currents and/or use a variety of different cues for orientation. Whether reef fish larvae have any orientation behavior in the open ocean where there is no apparent frame of reference (i.e. no visual cue of the benthic habitat or the ocean bottom) is so far unknown.

If and how coral reef fish larvae orient in the open ocean are questions that need to be urgently tackled. Once we are able to assess the modalities in larval orientation coming close to an undisturbed in situ environment, it will be possible to make significant advances in modeling larval transport and address the larger issue of population structure and connectivity. So far, there is strong evidence of the use of cues for orientation and navigation at small spatial scales, yet no information is currently available for orientation at spatial scales that are relevant to navigate over long distances (i.e., 10-100's km). This experimental study develops new methodology to investigate the ability of both newly hatched and late stage reef fish larvae to behave in relation to the direction of current flow in the open ocean.

Goals

The main focus of this experimental research is to test whether coral reef fish larvae can orient in the open ocean, as they move within surrounding water masses. Subsequently, in order to understand some of the mechanisms underlying orientation behavior, we plan to test a combination of the most probable cues that could be used in the open ocean by reef fish larvae.

Overview of approach

A novel device is developed to observe and quantify orientation behavior in coral reef fish larvae within the pelagic environment where there is no apparent frame of reference. The device is composed of drifter-like structure equipped with environmental sensors and a semi-enclosed circular chamber in which a larva is placed and filmed while the autonomous apparatus is deployed at sea. The effectiveness of the device depends of the success in recording true orientation behavior. This novel device, which has a simple design and is relatively inexpensive to build, represents a useful tool for larval ecologists to identify when and where early life history stages of coral reef fish can perceive orientation cues and measure their undisturbed swimming speeds. The picture to the right shows the device underwater.

This page documents a video analysis program for analyzes the video produced in the device, captures the movement of the larvae and extract information statistically.

A typical frame of the video looks like this:

There is a simplified compass in the upper-right corner. The white dot on the right is a reference. The circular area in the middle is the arena. The larva is the small black mark near the top of the image, in the arena. The objective is to extract the trajectory of the larva in the arena.

The program is a bash script (called bbscript) which wraps some pieces of open source software together:

MPlayer: which is used to enhance and convert the original video
ImageJ: which is used to process the images and provide some user interface for the tracking of the larvae. Two pdf documents provide additional information on using ImageJ and writing plugins for ImageJ.
R: which is used to perform various corrections and statistical analyses on the recorded tracks

The code of the program itself and of all modules specifically written for it is licensed under the GNU General Public License. It can be downloaded, used and modified freely.

Download bbscript-1.0

The history of the source code and proof of concept data is kept in a Version Control System which front end is accessible at the address: http://cbetm.univ-perp.fr/irisson/svn/ownfor/bbscript/trunk/

Brief overview of a bbscript session

In this section a typical bbscript session will be briefly described. For more insights on what is actually done, please refer to next section.

The easiest way to get help and guess what to do along the process is to issue the command ./bb -h on the command line. It outputs a help message:

The options can be set directly on the command line according to this help message or their equivalent can be written in a configuration file named bbscript.conf and located in the directory where the program bbscript is. It is advised to set the general options (such as id) in the configuration file in order to avoid having to write them again on each command line. In the following, I am assuming that you did this.

Video processing

./bb -video -vfps ? -vdenoise ? -vcontrast ? \
         -vbright ? video_file.???

The work directory is created and the video is copied in it. The video is sub-sampled according to the vfps parameter (in frame per second). It is denoised according to vdenoise value (min: 0, no max), contrasted according to vcontrast (min: 0, max: 1) and brightness is increased according to vbright (min: 0, max:1).

Tracking

./bb -tcalib -tfix -tcompas -tlarva -tcorrect

ImageJ is called to proceed to the tracking. Basic indications are given to the user on the Terminal, at each step of the tracking. The steps are: acquisition of calibration data (tcalib), automatic thresholding and tracking of the fixed point (tfix) and of the compass (tcompas), manual tracking of the larva(e) (tlarva). The correction step (tcorrect) is performed automatically with no user input.

Statistics

./bb -stats -nb-pie ??

The statistical analysis is performed (stats). Plots of the trajectory of the larva, of the displacement vectors, rose diagrams of positions and directions and well as a speed histogram are produced. The number of subdivisions of the rose diagrams is determined by nb-pie. Everything is automatically saved in the work directory.

Detail of the processes along a bbscript session

In this section we will present a typical bbscript session in depth. At each step of the analysis (video processing, tracking and statistical analysis) this document will explain:

what needs to be done
which command to issue in order to do it
how the the associated code works
when relevant, how to interpret the data

Basic usage of an unix shell/command line is considered to be known.

Main script

The global process is described by flowchart above. It is run by a bash script called bbscript (or bb for short). The general code is designed to set a working directory per video file, which contains:

all data files and results
a Test directory where the same data is stored for tests, see below
a Logs directory where errors or settings are logged
a Temp directory where all the work is done; at each step all necessary information is copied to the Temp directory, all results are generated in this directory and only at the end of the step, when the user confirms it, all this information is committed to the work directory.

The role of the main script is to:

parse default options, configuration file and command line arguments using a big case loop
create work directories if they do not exist
export all needed environment variables (command line options, workspace paths...) to the subprocesses of the shell
decide which steps/processes are performed depending on the options

The options to the main script (bb) can be set directly on the command line or their equivalent can be written in a configuration file named bb.conf and located in the directory where the program bbscript is run. The options of the bash script are read in order from the defaults (hard coded into the script), the configuration file (bb.conf) and the command line. Meaning that a command-line option will override a configuration file setting which itself overrides a hard coded option.

In addition to the options related to each step there are a few general ones:

-id defines the identification tag of the video. This tag is used as the name of the work directory and is prepended to the names of the files resulting from the analysis.
-t is a test switch. This makes bbscript use a subsample of the data and perform the analysis on this subsample only, which speeds the program up. It is useful for trial and error purpose at different steps of the analysis.

The three steps of the analysis (video processing, tracking and statistical analysis) have to be performed one after the other because each step depends on the results of the previous one. As for the regular steps, the test steps have to be performed one after the other.

1. Processing the video

A manageable number of frames need to be extracted from the video clip which is recorded at 30 frames per second. So we subsample at a given frequency. The images can be corrected to improve their quality.

Minimal command to issue:

./bb -video [-t]

Options to set:

-vfps -vdenoise -vconstrast -vbright

In addition, the first time the command is run it has to include an argument

./bb -video some_video_file.???

The command issued the first time (with a video file as argument) will create the work directory using the video-id and copy the video file there.

The command line options are passed to a shell script (video_process.sh). It sets parameters for MPlayer which is used to subsample, denoise, contrast and correct brightness of the video. All these functionalities are built into MPlayer and are documented by its man page.

An important parameter is the number of frames extracted per second, set by the -vfps flag. It allows the resampling of the original video because analyzing all the frames would be too time consuming. Values of 0.5, 1 or 2 are decent (they mean respectively: one image every two seconds, one image per second and two image per second).

The video filters applied can modify the contrast (-vconstrast, no correction: 0, max contrast: 1) and the brightness (-vbright, no correction: 0, max brightness: 1) of the images. Depending on the video, denoising (with the option -vdenoise, almost no denoising: 0, no absolute maximum) could also be useful. The effect of the denoising filter is not visible on the first frame: it takes effect through time so one has to check frames appearing later in the video sequence to set this filter value. The aim of all these corrections is to highlight the larvae, the white dot and the north of the compass. A tip is to issue the command with the -t flag and different settings until the correct combination is found. Then issue the command with the correct settings and without -t.

Video settings are saved in Logs/video_settings.txt each time the video step is executed. To make them easier to find, the last settings used are on the first line, those of the previous video call are on the second line and so on.

Then these images have to be converted to a format that ImageJ can open easily. MPlayer is used to convert the video to PGM images. PGM is an uncompressed grayscale image format and therefore it can be opened very quickly (there is no need to decompress it as there would be with JPEG or PNG for example). This allows us to open a great number of files very quickly in the following part of the program but has a cost in term of disc space as the PGM images are big (>300 kB each). Then an ImageJ macro (ij.macro.open_process_stack.txt) is used to open the previously produced PGM images. We open them as a stack, meaning that all the PGM images will be converted to one big image file (TIFF format) with layers. This allows to browse the images easily as if we moved inside the movie, frame by frame. In addition, the whole stack is normalized: the darkest point of the whole stack is made black and the lightest point is made white. The rest of the stack is corrected according to this reference. This provides a more consistent grayscale scheme between the images, removing some of the light irregularities generated by changes in the environment and the delay in the auto-setting of the video camera.

2. Tracking the larvae

This is the more time consuming part of the analysis. The larva has to be tracked manually but the arena moves with respect to the video camera, and the whole device rotates. Hence we have to correct the trajectory of the larva for the movement of a fixed point on the arena (suppresses the movement of the arena) and for the rotation of a compass (suppresses the rotation of the device). These two elements are tracked first and automatically. In addition, we need some calibration data in order to make the corrections on the larvae tracks.

The processes are driven by a tracking.sh which decides which tracking has to be done depending on command line options and which prints basic instructions at each step of the tracking, in the Terminal window.

2.1. Manual tracking of the larva

Larvae are too small and too transparent to be tracked automatically. Therefore we record the successive positions of one larvae by clicking on it on each frame of the stack where it can be found.

Minimal command to issue:

./bb -tlarva [-t]

We use Plugins > Manual Tracking to follow the larvae. You just need to click "Add Track"

and click on the larva. At each click the next frame is brought up, waiting for a new click and so on. The trajectory can be drawn along the process, this can be toggled on or off in the Manual Tracking window by selecting / unselecting the "Show Path" box. If you are not satisfied with a point you can delete it with the "Delete last point button". Each track is ended when the last frame is reached or when the "End track" button is clicked. Incorrect tracks can be deleted by ending them, selecting their number in the central menu and using the "Delete Track" button. In the end, the track(s) have to be saved to a text file named track_larva.txt in the Temp directory. This file has already been created, you just need to overwrite it.

We use a simplified version (Manual_Tracking.java) of the official Manual Tracking plugin of ImageJ. It does not take advantage of some new features of the plugin (directionality, center correction, 3D...) because they are useless here. These parts are therefore commented in plugin source. The plugin records the track number (useful if there are several larvae simultaneously in the arena), the frame number and the associated position of the clicked point. The coordinates are in pixels, measured from the upper left corner. All this is saved in ASCII form (i.e. to a text file).

2.2. Automatic tracking of the fixed point

The arena moves with respect to the camera and this leads to a somewhat "shaking" image in the video. Therefore, when the larva is tracked, it induces errors relatively to a fixed reference (the first image). We need to correct these errors by recording the position of a point fixed to the arena and subtracting the movement of this point to the coordinates of the larva.

Minimal command to issue:

./bb -tfix [-t]

First the fixed point needs to be isolated from the rest of the image. We use the oval selection tool oval tool

to select the fixed point oval selection

Then the image is thresholded because the automatic tracking plugins work only on 8 bits (black and white) images. On the selection we apply the Threshold Selection plugin (Plugins > Threshold Selection). This plugin isolates the selection, turns the lightest parts in black and the rest in white. A "watershed" is applied at the end. It allows, in certain proportions, to separate two particles overlapping each other. It can be useful if there are parasites on the video. Then all black artifacts remaining on the first frame only, except the fixed point, have to be erased. This is done with the freehand tool

by selecting the artifact artifact erase

and hitting DELETE.
Eventually, the tracking is done automatically by applying the Automatic Tracking plugin (Plugins > Auto Tracking). The tolerance in particle size is asked for. The default should be ok. You can check the results table quickly and if you find that the plugin misses the particle to often you can increase the tolerance. Some warning dialogs may also appear: if there is less or more than one particle on at least one frame. Usually the automatic tracking deals with it but you should check manually the results in case there is a problem. The results are saved in a text file named track_fix.txt in the Temp directory.

Automatic tracking makes a large use of the Particle Tracking functionality of ImageJ. On each frame, particles are detected and their trajectory through the stack is recorded. Tracking the fixed point is a matter of detecting the "particles" (thresholded to black) present on each frame, choosing which is probably the point we are interested in and grabbing the coordinates of its centroid.

Our Automatic Tracking plugin is a modification from two existing tracking plugins. As mentioned, it uses the Particle Analysis tool of ImageJ. On the first frame, the area of the point to follow is detected (this is the reason why all other particles have to be erased from the first frame). Then a particle analysis is made on each frame, looking for particles of similar area, given a tolerance. Eventually, if there are several particles on a frame, the closest to previous position is chosen; if there is no particle on a frame the coordinates are kept from previous frame, considering that the particle did not move. This outputs the coordinates of the fixed point on each frame which are saved in an ASCII file.

2.3. Automatic tracking of the compass

The arena also rotates which prevents us to study the trajectories in a fixed cardinal reference. This is mandatory to detect orientation. Therefore we need to suppress this rotation. We record the trajectory of the north of a simplified compass and will subtract this rotation to the larva trajectory

Minimal command to issue:

./bb -tcompas [-t]

The north of the compass is tracked though the whole stack the same way the fixed point is. The steps are very similar to the fixed point tracking except that the selection of the region of interest is different: we want to select a "ring" in the image, containing the north of the compass and excluding the central area of the compass, to avoid parasites. This is done with the oval selection tool. A first selection covers the whole compass window, then a smaller selection is made inside, while pressing the ALT key, in order to eliminate the central part. The result is showed on the right. oval selection compass

Then the image is thresholded by using Plugins > Threshold Selection and, as before, the remaining artifacts are removed on the first frame only. Finally the Plugins > Auto Tracking plugin is applied and the results saved in a text file named track_compas.txt in the Temp directory.

2.4 Measuring calibration data

The aim of this step is to measure the coordinates of the compass and the arena and to have some size calibration in order to compute speeds in standard units (cm/s and not pixels/framestep).

Minimal command to issue:

./bb -tcalib [-t]

The center of the compass needs to be detected in order to allow to compute its rotation. This is done with the point selection tool point tool

by clicking on the image point compass

and selecting Plugins > Macros > Measure. The results are saved in a text file named coord_compas.txt in the Temp directory.

To calibrate the movement, we need to know the coordinates of the arena. We use the oval selection tool to fit an oval on the arena oval arena which is measured using Plugins > Macros > Measure. The results are saved in a text file named coord_arena.txt in the Temp directory.

The "Measure" macro is defined as a Startup Macro of Image J. It measures the coordinates and the value of the point when a point is clicked and measure the coordinates of the centroid of an oval selection as well as its perimeter. This way we have the coordinates of the center of the compass, the coordinates of the center of the arena and the perimeter of the arena (which is used to compute its radius) all expressed in pixels.

2.5. Correction of the tracks

We need to suppress the movement of the fixed point to the coordinates of the positions of the larva to suppress the shaking of the image and suppress the rotation of the compass to correct for the rotation of the arena. In addition it is interesting to compute the swimming speed of the larva between each frame.

Minimal command to issue:

./bb -track -tcorrect [-t]

The tracks recorded in parts 2.1 to 2.3 are corrected with the help of data gathered in part 2.4. Therefore theses steps have to be all done before invoking tracks correction. Error messages are issued when one step is not done and the program exits. Correction is done automatically and corrected tracks are saved in ASCII form to tracks.csv.

The correction of the tracks is done by tracking.R. The code should be self explanatory but let us precise the different steps:

First we read the text file save from the Manual Tracking plugin of ImageJ and we split it into matrices according to the Track Number field: this separates the different larvae tracks. If there is only one track this step has no effect. The matrices are saved in a list.
We read the text files saved from the Automatic Tracking plugin and save the trajectories of the fixed point and of the compass in the same list.
Some frames could have been skipped while tracking the larva if it is not visible for example. When this is done, the Manual Tracking plugin just skips the entry. We need to fill these entries so that the indexes in the reference tracks and the larvae tracks point to the same video frame. Therefore, the blanks are filled with NAs.
After this treatment, if all tracks have been followed to the end, the reference tracks and the larvae tracks have the same number of lines. Nevertheless, we have to account for strange events such as your cat stepping on your keyboard while you are tracking, stopping everything, saving the tracks and erasing the original video (well, not your cat maybe...). Considering that a larva coordinate without reference is useless and that a reference point that is never reached by any track is useless too, we crop to the limiting size and discard additional useless information.
Then we have two types of corrections to do on the recorded tracks:
1. we start by correcting the larvae tracks and the compass track for the movement of the arena in respect to the camera. This is done by subtracting the movement of the fixed point between two slices to the coordinates of the tracks. This brings the fixed point at the same position on each slice: the position it had on the first slice.
2. then we correct the larvae tracks for the rotation. To perform the correction we need to convert the coordinates of the north of the compass and of the larvae tracks to polar coordinates. We need the coordinates of the centers of the arena and of the compass, recorded during the calibration. We subtract the angle of the compass to the angle of the larvae position, on each slice. Therefore we bring the north to 0 radians on each slice. Eventually we rotate everything by pi/2 so that the north actually points upwards.
Finally we convert everything to human significant measures. We compute the displacement between two frames from these corrected coordinates and convert it to speed in cm/s knowing the video sampling rate (vfps) and the radius of the arena (from the calibration step).
Eventually we save some variables in tracks.csv.

3. Statistical analysis

For each larva, we end up with positions (on all frames where it has been clicked), swimming directions and speeds (when they are defined: when two successive positions are available). We want to know wether the positions and the swimming directions are random or concentrated in around a particular bearing.

Minimal command to issue:

./bb -stats [-t]

Options to set:

-nb-pie

The tracks are analyzed. Everything is done automatically and the process outputs a CSV file (video-id-stats.csv) summarizing the statistical information and a PDF file (video-id-graphics.pdf) containing plots. The only user configurable part is the refinement of the rose diagram plots.

The statistical analysis is performed using and R script stat.R and a function library lib_circular_stats.R.

There are two kinds of statistics: the statistics on directions, which is an independent variable and the statistics on positions, which are not independent. As the directions are independent we can conduct the analysis on the whole dataset. Because the positions are not independent we use a partial bootstrapping technique: resample randomly 3% of the data, perform the statistics on this independent subset and repeat this 1000 times.

Being enclosed in a circular area, the trajectories deserve a specific statistical description. For each dataset, we use Rayleigh statistic, compute a mean angle, an angular deviation around the mean and an angular dispersion. A very important concept in circular statistics is the sum vector. Each observation is viewed as a vector which starts from the center of the arena and points in some direction. If our data is the position of the larva in the arena, the vectors will be defined by (0,0) (=the center) and (X,Y) (=the position). But we do not care if the larva is near the center or close to the border. Therefore the vectors are all normalized. For direction data we just normalize displacement vectors. Then, to appreciate the orientation of the larva we sum all the vectors. If the positions are evenly distributed in the arena, each vector in a particular direction will be countered by an other vector in the opposite direction. The norm of the sum vector will therefore be small (or even zero). In contrast, if the larva is more present in one portion of the arena, the sum vector will be long and pointing in this direction. Therefore, the sum vectors captures the direction and the intensity of the orientation behavior. We can derive statistics from this quantity:

Mean Angle: The mean angle is simply angle of the sum vector we previously calculated.
Angular deviation: The angular deviation quantifies the extend to which the observations are dispersed around the Mean Angle we calculated. It is a "port" of the standard deviation concept to circular statistics and is measured in radians. As it can be expected it uses the sum vector:
where r_bar is: where V is the sum vector and N the number of observations.
Angular dispersion: The angular dispersion also quantifies the dispersion around the Mean Angle but is useful for comparison purposes only. It also uses the sum vector:
where r is:
Rayleigh's statistic: The idea behind Rayleigh's test is to compute the probability to obtain the observed distribution of the data randomly given the number of observations. Performing the test is in fact comparing the sum vector to a sum vector generated from the same number of normalized vectors (N) picked at random. The resulting p value of this test is the probability to observe the observation by chance, on a random sample.

Such statistical descriptors are all computed and saved in the CSV file. They have to be interpreted together with the graphic:

	a plot of the trajectories in the arena
	a rose diagram of the bearings of the significant mean vectors in the bootstrap-like analysis. A larger "pie-part" means more vectors in this particular direction
	a plot of swimming vectors in the arena; each vector of the trajectory is plotted originating from the center
	a rose diagram of the swimming bearings
	a speed histogram. The red dotted line separates slow speeds (< to 4 times the measure error) from fast ones (which are used in the analysis)

4. Cleaning and packaging

Many big files are generated along the process (the PGM images, the stack etc.). We need to remove them cleanly and to be able to store only useful data.

Minimal command to issue:

./bb -clean

./bb -package

clean removes the pgm images, the test data directory, the temporary directory, some logs and asks for the removal of the stack. package compresses the data and log directories, excluding the stack, the test directory and the pgm images, in a zip archive, named after the video-id.

Page by Claire B. Paris & Jean-Olivier Irisson