Introduction

This free code is an ImageJ plugin that allows the simultaneous registration of two images based on elastic deformations represented by B-splines. The invertibility of the deformations is forced through a consistency restriction. bUnwarpJ is also integrated in Fiji. Contact author: Ignacio Arganda-Carreras.

For a quick start and introduction, you can watch a video tutorial (v2.0) here (awarded at the Second ImageJ User & Developer Conference). 

General description

bUnwarpJ is an algorithm for elastic and consistent image registration developed as an ImageJ plugin. It performs a simultaneous registration of two images, A and B. Image A is elastically deformed in order to look as similar as possible to image B, and, at the same time, the "inverse" transformation (from B to A) is also calculated so a pseudo-invertibility of the final deformation could be guaranteed. Two images are given as a result: the deformed versions of A and B images.

bUnwarpJ scheme
Figure 1: bUnwarpJ scheme: bidirectional Unwarping in Java.

Downloads

The latest stable distribution of bUnwarpJ can always be found included in the Fiji toolkit or at the git repository. If you detect any bug, please feel free to contact the author. Any feedback will be very appreciated. The class files can be downloaded with the following JAR file:

Older versions of bUnwarpJ and their related documentation can be found here. You can also browse the source since October 6, 2008 in the source repository.

For Fiji users, just upgrade your program to the last Fiji version.

API documentation

The API documentation can be reached online here or downloaded from here:

Installation

You must simply download bUnwarpJ_.jar to the Plugins folder of ImageJ, restart ImageJ and there will be a new "Registration > bUnwarpJ" command in the Plugins menu.

To execute bUnwarpJ as a macro or from the command line, see the description in the User Manual.

Related work

The algorithm implemented on bUnwarpJ and its technical explanations are detailed on the paper:

download paper Ignacio Arganda-Carreras, Carlos O. S. Sorzano, Roberto Marabini, Jose M. Carazo, Carlos Ortiz de Solorzano, and Jan Kybic, “Consistent and Elastic Registration of Histological Sections using Vector-Spline Regularization”, Lecture Notes in Computer Science, Springer Berlin / Heidelberg, volume 4241/2006, CVAMIA: Computer Vision Approaches to Medical Image Analysis, pages 85-95, 2006.

The previous work (UnwarpJ) is reachable here, and its related paper is:

"Elastic Registration of Biological Images Using Vector-Spline Regularization", C.Ó. Sánchez Sorzano, P. Thévenaz, M. Unser, IEEE Transactions on Biomedical Engineering, vol. 52, no. 4, pp. 652-663, April 2005. 

Technical Explanations

This image registration algorithm is based on the minimization of an energy functional that includes the dissimilarity between the source and target images -in both directions- Eimg, an optional landmark constraint Eµ, a regularization term (EdivErot), and an energy term Econs that accounts for the geometrical consistency between the elastic deformation in both directions. Namely, the energy function is given by

E = wiEimg + wµEµ + (wdEdiv + wrErot) + wcEcons

Where the weights of every term are set by the user in the main window of the plugin. The optimization process is a Levenberg-Marquardt minimization enhanced by a Broyden-Fletcher-Goldfarb-Shanno (BFGS) estimate of the local Hessian of the goal function, and both, images and deformations are represented by cubic B-splines.

User manual

Generalities

First of all, the plugin must be installed as described on Installation (for further information go to ImageJ). Once it is properly installed, the plugin should appear in the main menu under Plugins > Registration > bUnwarpJ. Two images (8, 16, 32-bit grayscale or RGB Color) need to be opened in order to be able to use the plugin. If so, the following window will open:

Main window
Figure 2: Main plugin window (v2.6)


Both selected images will work simultaneously as source and target, their tags are there only for the sake of clarification. The registration mode can be "Accurate", "Fast" and "Mono". The registration mode "Mono" (included since version 2.5) makes the program to perform only unidirectional registration, i.e. from source to target. The two registration modes "Accurate" and "Fast" involve performing bidirectional registration and affect the stopping criteria internally used by the program.  More internal options can be modified in the "Advanced Options" panel. This panel gives you access to most of the internal parameters of the algorithm. The "Initial" and "Final" deformation lists allow you to select the coarsest and finest scale of the spline deformation field. "Very coarse" corresponds to 4 splines (one in each corner of the image). As you increase the deformation level, the number of splines is doubled in each direction (horizontal and vertical).

Since bUnwarpJ 2.5 there is a new parameter on the main window to allow subsampling the input images. The registration will be then calculated using the subsampled versions of the images but the results will be applied to the original ones. The image subsampling parameter can be chosen between 0 and 7, i.e. the image dimensions can be reduced by a factor of 20 = 1 to 27 = 128. This is very useful when registering large images.

The different weights of the goal function control the relative weight of each one of the terms. These weights are not restricted to be between 0 and 1, and they may take any value as long as it is non-negative=2E. You can see a description of the different function weights in the presentation "bUnwarpJ: Consistent and Elastic Registration in ImageJ. Methods and Applications.", given at the Second ImageJ User & Developer Conference (2008) or have a look at the FAQ page.

The stop threshold is used by the algorithm to stop the optimization process at each multiresolution level when the error relative change is not larger than this threshold.

RGB Color images will be converted to grayscale during the registration process but the resulting transformations will be applied to the original color images.

If you want to exit the plugin, press "Cancel". When you want the plugin to perform the registration press "OK". After running the plugin (on "Accurate" or "Fast" mode), the results are two stacks with the following three images:

1. One image (warping image) registered as to fit the other image (fixed image);
2. The fixed image;
3. The warping mask with the same deformation as the warping image. 

The final registration values appear in a separate ("Results") window.

Figure 3 shows one of the resulting stacks from registering a source Lena image to a target warped version of the same image:

Resulting registered target imageOriginal target imageResulting source maskFigure 3: Result stack (basic view)


The verbose mode produces more information:

1. The elastic deformation as a vector field. Each point in the fixed image must be deformed according to this field to fit into the warping image;
2. The grid obtained after deforming the fixed image with the vector field described above;
3. The step values of the optimization process in a separate ("Results") window.

Resulting target deformation vectorsResulting target deformation grid
Figure 4: Additional result images (verbose mode), deformation field and grid

Results window
Figure 5: Results window with values of the goal function f during the optimization process and final optimal values


Since both, source and target images work as warping and fixed images, there are two sets (stacks) of results: from source to target and from target to source.

The "Mono" mode produces only results from the source to the target image.

During the registration process, the current difference images and a mapping of the grid from the fixed images onto the moving images are shown.

Lena difference imageLena processing grid image
Figure 6: Examples of output images during the registration process (current target-source difference image and original source image with current grid)

During the registration process the toolbar will be changed to

Processing toolbar
Figure 7. Appearance of the window while registration is in progress


Click on the stop button to stop the process. The output at the current state of the optimization will be returned in the normal way.

Landmarks

When the plugin is called and before pressing "OK" in the main window, the toolbar changes its appearance and it is possible to manually add landmarks to the selected images:

Toolbar add landmarks
Figure 8: Adding landmarks activated


The depressed button indicates that you may ADD A LANDMARK now. Landmarks are added in either image. The landmark will be automatically placed in the same position on both images. The new landmark becomes the "current landmark" (indicated by a thicker [+] sign in the current image and a [×] sign in the other image, while all the rest are represented by [+] signs). To move any landmark, press on the MOVE LANDMARK button.

Toolbar move landmarks
Figure 9: Moving landmarks activated


Click and drag on any landmark to make it correspond to the same position in both images. Here goes an example of the two Lena images with corresponding landmarks:

Lena manula landmarksWarped Lena manual landmarks
Figure 10: Corresponding landmarks


Landmarks can be removed through the REMOVE LANDMARK (crosses) button. 

Toolbar remove landmarks
Figure 11: Removing landmarks activated


This is the way of manually adding landmarks to the registration process. However, since bUnwarpJ v2.0 there is the option as well of using automatic landmarks as explained in the section SIFT and MOPS support, or manually adding point selections in both images before calling the plugin. If the number of point selections is the same in both images, they will be transformed into landmarks.

When exiting bUnwarpJ all the images are restored to their previous state, i.e. the original regions of interest and point selections are restored.

Masks

This program allows you using masks in two mutually exclusive ways. In the first way, masks are introduced together with the input images. In this mode, input images must be a stack of images (first slice: the image itself, second slice: the mask). In this way, the mask can have any shape. In the second way, the input images must not be stacks and simple polygonal masks can be used. These masks are defined using the two buttons (INNER MASK and OUTER MASK) shown below:

Toolbar inner mask
Figure 12: Inner mask activated

Toolbar outer mask
Figure 13: Outer mask activated

The inner mask keeps the information in the interior of the polygon, while the outer mask keeps the information in the exterior of the polygon. The thrown-out information is grayed. Here goes an example of an inner mask:

Lena inner mask
Figure 14: Inner mask example

Masks can be used for one of the images, both, or none. You can put a mask in one of the images and not in the other, you can put a mask (with different shapes) in both images, or you may not use masks at all. After calling the plugin, the masks are erased and the initial images are restored.

Input/Output options

When using the "I/O Menu" from the toolbar, we have the possibility of processing different input and output files that will affect the registration.

toolbar
Figure 15: ImageJ toolbar when using bUnwarpJ plugin. I/O menu option is selected.

Compare Transformations
Figure 16: I/O Menu options.

In the last release, the plugin presents the following Input/Output options:

These new options (since version 1.1) allow the user comparing the results of our program with any other registration method. 

You can also test bUnwarpJ with SplineDeformationGenerator, an ImageJ plugin that allows the user to generate five different image deformations: elastic, fisheye, perspective, barrel/pincushion and smile effect. SplineDeformationGenerator uses the same raw transformation file format as bUnwarpJ, so they are completely compatible.

Macro call

Since bUnwarpJ v2.0 the plugin allows using ImageJ macro language to be called. Therefore, the user can launch the plugin from a macro setting all the parameters of the plugin main window, for instance:

run("bUnwarpJ", "source_image=A target_image=B registration=Accurate image_subsample_factor=0 initial_deformation=[Very Coarse] final_deformation=Fine divergence_weight=0 curl_weight=0 landmark_weight=0 image_weight=1 consistency_weight=10 stop_threshold=0.01");

When calling the plugin like this, if the "save_transformations" parameter is set to true, the transformations files will be saved in the following predefined format (not save dialog is shown):source image name + "_direct_transf.txt" and target image name + "_inverse_transf.txt".

Only the main registration method can be called like this. To use the main Input/Output options from a macro, the call function must be used. There is a corresponding static method defined in the plugin for any of this options: elasticTransformImageMacro, rawTransformImageMacro, composeRawElasticTransformationsMacro, composeRawTransformationsMacro, composeElasticTransformationsMacro, compareRawElasticTransformationsMacro, compareElasticTransformationsMacro, compareRawTransformationsMacro, convertToRawTransformationMacro, adaptCoefficientsMacro.

Notice here that the input and output file names must include the path, since they are not taken from the list of images. For instance, if we want to apply an elastic deformation stored in the file A_direct_transf.txt to the source image A.jpg with target image B.jpg and save the result in output.tif, we call:

call("bunwarpj.bUnwarpJ_.elasticTransformImageMacro", "My_path/A.jpg", "My_path/B.jpg", "My_path/A_direct_transf.txt", "My_path/output.tif");

Command line call

bUnwarpJ might be called as well as an ImageJ from the command line. In the command line, the program offers the following options:

  • -help: shows the syntax of the program
  • -align: launches the registration of two input images
  • -elastic_transform: transforms the source image with a given elastic deformation (previously calculated)
  • -raw_transform: transforms the source image with a given raw deformation (previously calculated)
  • -compare_elastic: compares two previously calculated opposite elastic deformations through the warping index
  • -compare_elastic_raw: compares an elastic deformation with a raw deformation (both direct transformations) through the warping index
  • -compare_raw: compares two previously calculated and direct raw deformations through the warping index
  • -convert_to_raw: converts an elastic transformation into raw format
  • -compose_elastic: composes two elastic deformations, the result will be in raw format
  • -compose_raw: composes two raw deformations, the result will be too in raw format
  • -compose_raw_elastic: composes a raw deformation and an elastic one. Result in raw format
  • -adapt_transform: adapts an specific elastic transformation given a resolution image factor

For instance, to see the program help we can call the program from the command line (where $IJDIR is the directory where ImageJ is installed) like this:

java -Xmx512m -cp $IJDIR/ij.jar:$IJDIR/plugins/bUnwarpJ_.jar bUnwarpJ_ -help


For the rest of the options, follow the help instructions.

Consistency weight

The main reason to create bUnwarpJ was the idea of enforcing deformations' consistency through the consistency weight. This number forces the algorithm to move into solutions that ensure the invertibility of the resulting deformations. Therefore, higher is this number, more strictly is one deformation the inverse of the other. Due to the different units, there is no rule for selecting the right parameters. For instance, to perform the registration of the tumor images above, we recommend consistency weight values around 20.0.

One important advantage of bUnwarpJ over the previous method lies in the fact that many registration problems can be solved without using the landmarks and regularization terms of the energy function (that means setting their corresponding parameters to 0.0). Therefore, no user interaction is needed and the computational complexity of the algorithm is reduced.

SIFT and MOPS plugin support

The last release of bUnwarpJ has compatibility with the Stephan Saalfeld's plugin for automatic feature extraction: SIFT and MOPS algorithms. You only need to download the following files to your ImageJ plugins folder:

An explanation of the parameters is here. This plugin is also integrated in Fiji.

After applying SIFT or MOPS methods, you will get two sets of corresponding points in both images. If you call then bUnwarpJ, then the corresponding points will appear as source and target landmarks.

Lena MOPSWorped Lena MOPSLena bUnwarpJ LandmarksWarped Lena bUnwarpJ Landmarks
Figure 17: SIFT/MOPS - bUnwarpJ compatibility. Top row: MOPS results. Bottom row: landmarks after calling bUnwarpJ.

Conditions of use

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation (http://www.gnu.org/licenses/gpl.txt).

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more details.  

We just expect you to include a citation or acknowledgment whenever you present or publish results that are based on it. Enjoy it!

Acknowledgments

bUnwarpJ has been developed during several years already and many people need to be acknowledged for:

It started as an extension of UnwarpJ from Carlos O. S. Sorzano and thanks to him and Jan Kybic at the Center for Machine Perception in Prague, bUnwarpJ was born in the summer 2005 and published online in 2006.

Many of the plugin updates and improvements would have never been possible without the hackathons that took place in Janelia Farm Research Campus (Virginia, summer 2008) and the Institute of Neuroinformatics (Zürich, winter 2008).

Albert Cardona organized the hackathons and is responsible for the code parallelization and javascripting support.

Stephan Saalfeld proposed and helped to make the SIFT/MOPS support possible.

Marta Rivera-Alba helped with the macro support and code debugging.

Johannes Schindelin is responsible for the Fiji integration and he is simply there whenever a Java problem comes up!