Optical Flow Active Contours with Primitive Shape Priors for Echocardiography
© A. K. Hamou and M. R. El-Sakka. 2010
Received: 2 May 2009
Accepted: 5 August 2009
Published: 24 September 2009
Accurate delineation of object borders is highly desirable in echocardiography, especially at the left ventricle. Among other model-based techniques, active contours (or snakes) provide a unique and powerful approach to image analysis. In this work, we propose the use of a new external energy for a gradient vector flow (GVF) snake, being the optical flow of a moving sequence (modeling the mechanical movement of the heart). This external energy can provide additional information to the active contour model garnering adequate results for moving sequences. An automatic iterative primitive shape prior was also applied in order to further improve the results of a GVF snake, when dealing with especially noisy echocardiographic images. Results were compared with expert-defined segmentations yielding acceptable sensitivity, precision rate and overlap ratio performance.
The assessment of cardiac function has been a major area of interest in the medical field. Normal heart function consists of pumping chambers (known as ventricles) which regulate the systemic and pulmonary circulation systems by delivering blood to the proper areas. Detection of nonnormal heart function in the left ventricle (LV), for instance, can cause systolic dysfunction, being the reduction in the ability to contract, or diastolic dysfunction, being the inability to fill efficiently. Along with mechanical functions, various heart structures may also fail causing cardiomyopathies, endangering the life of the host individual. Fortunately, many myopathies are treatable (with medication, implanted pacemakers, defibrillators, or ventricular assist devices) given early detection. Echocardiography, imaging the heart using ultrasound waves, facilitates the ability to do so.
The advent of real time ultrasonography provides the ability to image an entire LV and surrounding anatomy within one cardiac cycle (approximately one second). However, depending on the patient's "photogenicity" (impacting factors include and surrounding fatty tissues, calcifications, gender), these images are most likely marred by speckle artifacts. Many computer vision techniques attempt to reduce such speckle noise by means of filtering  or incorporating the speckle effect directly into their algorithms . Boundary detection techniques are employed in order to segment the wanted regions for analyses on the heart structures, such as endocardial borders , stress and strain of the septum wall , and wall motility  to name a few.
Various computer vision techniques have been introduced to accomplish boundary detection. Kass el al.  first proposed the original active contour model (commonly known as a snake or a deformable model). In their formulation, image segmentation was posed as an energy minimization problem.
Active contours treat the surface of an object as an elastic sheet that stretches and deforms when external and internal forces are applied to it. These models are physically-based, since their behavior is designed to mimic the physical laws that govern real-world objects . Since this approach relied on variational calculus to find a solution, time complexity was one of the main drawbacks of this original model. Amini et al.  proposed an algorithm for using dynamic programming, in order to incorporate soft and hard constraints into the formulation, improving time complexity and results. Further improvements to time complexity were proposed by Williams et al. , by using a greedy algorithm while incorporating a simple curvature approximation. Issues with large capture ranges and concavities are solved by other advances, which include inflation forces , probabilistic models , oriented particles , and gradient vector flows . For the purposes of this study, focus will be placed on those advances best suited for echocardiographic images.
Since the LV represents one of the most important heart functions, many semiautomatic techniques attempt to segment this region from surrounding tissues [3, 14, 15]. Yet, no universally accepted standard exists for segmenting echocardiographic images.
Papademetris et al.  took advantage of a b-splines parameterized deformable model for segmenting cardiac regions. The external energy consisted of the standard intensity term and a markov random field (MRF) texture-based term. The MRF is based on a combination of gradient, regional, and curvature data computed from the original image. Initial contours are manually placed for each 2D plane and are passed to a shape tracking algorithm. Displacements are probabilistically computed using a confidence measurement for the entire set. Final displacements are fed into an anisotropic linear elastic model which is computed vis-à-vis a Bayesian estimation framework. The manual placement of the contours makes this technique quite labour intensive.
Felix-Gonzalez and Valedes-Cristerna  proposed a segmentation technique for echocardiographic images using an active surface model (ASM). The ASM is made up of cubic splines and is based on a gradient descent procedure. When using gradient descent, the empirical setting of parameters is required based on the quality and types of images used. This makes this proposed technique extremely sensitive to its input. Furthermore, Felix-Gonzalez et al.'s work was only tested on two limited datasets.
Leung et al.  proposed the use of an active appearance model (AAM) and intensity, based registration for segmenting multiple 2D image slices. An AAM uses all the information in an image region covered by the target object, rather than just that near modeled edges. An AAM involves the principal component analysis of the various shapes and textures from several manually segmented 2D slices for training. The AAM makes use of the training set to converge the initial set mesh to the best textures on the image. However, this trained set required several manual segmentations of the 2D image slices to tune it to the medium being used.
The gradient vector flow (GVF)  snake was introduced as a modification to the original snake model in order to overcome the capture range and curve concavity issues. However using the GVF snake on echocardiograms directly will not provide an adequate solution due to the complication of speckle noise and the existence of valves within the heart cavity, inhibiting a proper segmentation.
Zhou et al.  proposed the segmentation of MRI cardiac sequences using a generalized fuzzy gradient vector flow (FGVF) map along with a relative optical flow field. Optical flow measurements are computed on the cardiac sequence being considered, and amaximum a posteriori probability (MAP) was used as a window for the movement of the curve. The use of optical flow with GVF provides promising results; however this technique is used exclusively on clear MRI data, and hence the presence of speckle noise on echocardiographic images would require modifications of this technique. Both GVF and optical flow measurements will be used in the proposed technique.
In this paper, we propose to utilize optical flow measurements as an external energy of a GVF snake. We will show that there is ample information in the movement of tissues within the heart cavity that will provide the necessary knowledge to segment out the region of interest. We will further improve the results by providing means of incorporating an iterative prior knowledge process into the proposed solution which takes into account the primitive shape of an object during the active contour evolution cycle. The rest of the paper is organized as follows. Section 2 will outline each module of the proposed model. Section 3 will explain the experimental setup and results, and Section 4 will contain concluding comments.
2. Description of Model
2.1. Parametric Active Contour Review
A snake is an energy minimization problem. Its energy is represented by two forces (internal energy, , and external energy, ) which work against each other. The total energy should converge to a local minimum—in the perfect case—at the desired boundary. The snake is defined as , where s belongs to the interval . Hence, the total energy to be minimized, , to give the best fit between a snake and a desired object shape is
The internal energy decreases as the curve becomes smooth (by incorporating both elasticity and stiffness), whereas the external energy decreases as approaching the features of interest, such as image structures or edges.
The internal energy of the active contour formulation is further defined as
where and are weighting factors of elasticity and stiffness terms, respectively. The first-order term makes the snake's surface act like a membrane. The weight controls the tension along the spine (stretching a balloon or elastic band). The second-order term makes the snake act like a thin plate. The weight controls the rigidity of the spine (bending a thin plate or wire).
where ∇ denotes the gradient operator. In the case of a noisier image the edges are further smoothed:
where is a two-dimensional Gaussian function with standard deviation , and * denotes a convolution operator. Since the contour may get trapped by the noisy areas of the image, must be large enough to compensate for the image noise that would interfere with the active contour's capture range. The standard snake algorithm also suffers from poor capture range because initialization and the inability to capture concavities. These problems are largely solved by the advent of the GVF snake .
The concavity problem exists due to the gradient vectors in an image generally have large magnitude only in the immediate vicinity of the boundary and are nearly zero at points away from the boundary. As such the capture range of the snake will be quite small. In order to resolve this, the gradient map is extended to points away from boundaries using a computational diffusion process. The GVF field is used as an external energy in the active contour and is characterized by the vector field that minimizes the energy functional :
where is an edge map derived from the image, μ is the degree of smoothness of the field, and characterize the direction and strength of the field. Hence, when is small, the energy will be dominated by the partial derivates, yielding a slow field. Alternatively, when is large, the latter term dominates and the function is minimized by setting .
The external energy for the proposed scheme will be generated using a virtual electric field (VEF)  of f over the traditional GVF technique. Traditional GVF field generation is performed by optimizing the cost function represented in (5), which is quite a time consuming process. The VEF is defined by considering each edge pixel as a point charge within an electric field. This can be accomplished by convolving the edge map with the following two masks:
2.2. Optical Flow Review
Optical flow approximates the apparent motion of an object over a series of images (or time). The relationship between the optical flow in the image plane and the velocities of objects in the three-dimensional world is not necessarily obvious . For the sake of convenience, most optical flow techniques consider a particularly simple world where the apparent velocity of brightness patterns can be directly identified with the movement of surfaces in the scene. This implies that objects maintaining structure but changing intensity would break this assumption.
Equation (11) is known as the optical flow constraint equation, where at a particular pixel location , is how fast its intensity at this location is changing with respect to time: and are the spatial rates of change for any given pixel (i.e., how fast an intensity is moving across an image). However, effectively estimating the component of the flow (along with intensity values) cannot directly be solved in this form since it will yield one equation per pixel for every two unknowns, and . In order to do so, additional constraints must be applied to this equation.
When dealing with noisy echocardiograms, a global regularization parameter will deal with the speckle better than the static window. This is due to the speckle noise remaining relatively static, lacking fluidity, throughout an image. Hence, the speckle will be "filtered", since the optical flow calculations will fail to realize it within the frames.
Optical flow magnitudes will be combined with the image's edge maps (Section 2.4) in order to generate the external energy in (1) of the GVF snake. This would help to reduce artifacts due to static speckle noise, while also providing more information for the contour points to track (i.e., the tissue mass movement).
2.3. Primitive Shape Prior Knowledge
Since we are dealing with structures that have known shapes and sizes, and many real world models have been already measured, prior knowledge information can be directly used to increase the performance of a segmentation algorithm. Priors based on shape statistical models require modifications to the standard active contour model. An iterative solution can be incorporated directly into any optimization model by using the proposed framework which we first outlined in Hamou et al. .
Since it is desirable to incorporate shape priors without directly involving the user for training, automatic shape detection takes place on the set of discrete snake points . This is achieved by first generating the least squares fit polynomial(s) of the current points. For our left ventricle application, snake points are divided into an upper region and a lower region, which will be approximated by two separate third-order hyperbolas. These hyperbolas will better suit the shape of the left ventricle during both systole and diastole, though any shape prior can be represented by means of simple primitives. Least squares fitting technique is utilized to estimate the two hyperbolas coefficients. The axis separating the two regions is computed by taking the two-thirds upper and one-third lower bounds on all snake points. This separator can be tuned by shifting it upwards or downwards (either manually by the user or automatically by the system) in order to minimize the distance between the fitted hyperbolas and snake points. Priors are then generated by joining the fitted primitives to form one solid shape. Primitives are bounded by the furthest easterly and westerly points by the snake points, in order to prevent the possibility of a nonconnecting shape.
The fitting of a primitive shape (or a series of primitives as needed for the left ventricle) to the snake points, , will help compensate for the noise that inhibits the snake from migrating past a certain point. It will also help retract the snake towards the primitive prior when an occluded border exists, common in many echocardiographic images. The user can control the number of cycles between any two consecutive prior calculation cycles (prior step parameter). This allows for the increase or decrease in the inherent effect of the prior knowledge to the snake's convergence cycle.
This primitive prior module is useful in the medical arena where the specialist or clinician has a clear understanding of the underlying structure being detected, such as a liver, an artery, or a heart. They can choose their desired primitive shape (or series of shapes) before curve evolution takes place.
2.4. Complete System
3. Experimental Results
For this study, a series of B-mode echocardiogram cross sectional videos of the heart have been used to investigate the proposed snake algorithm. These videos were acquired using an SONOS 5500 by Philips Medical System. The transducer frequency was set at 2.5 Mhz in order to insure adequate penetration of tissue, while maintaining good image quality. Longitudinal views of the heart, which nicely visualize the left ventricle, were acquired. The generated videos were treated as sears of individual images in sequence.
The mask size for generating the GVF field was set to 65 by 65 in order to calculate the point charges and is then normalized for active contour use. For left ventricle segmentation, the initial contour was set to be a small oval (of radius greater than 30 pixels along its minor axis and 60 pixels along its major), which was placed by the user within the left ventricle of the heart on the first image cine. Following image in the sequence would grab the previous segmented left ventricle contour and use it as its initial contour. Hence each dataset (or set of image sequences) was only initialized once.
Snake parameters and were set to 4.0 and 0, respectively. was set to 0 due to the influence of the prior knowledge component which offsets the curvature, whereas w as set to 4.0 in order to make sure that the external and internal energies were neatly balanced. The prior step parameter was set to 5. By decreasing the value of this step parameter, greater shape regularity would occur, but it would take much longer for the snake to reach equilibrium. Empirically, we found that a prior step parameter of 5 was balanced enough to maintain the shape regularity of the snake, without greatly affecting the time complexity.
During our experimentations, we assumed that equilibrium is achieved when there is less than 10% movement of contour points throughout the snake between two consecutive snake evolutions.
Since the proposed model is iterative, other adjustments can be made during the prior knowledge phase, such as increasing internal energy weights dependent on the distance of fitted priors to snake curve and other various optimizations without upsetting the actual active contour model, though none were done for the purposes of this experiment.
Experiments were run on eight complete cardiac cycles from different patients. The performance of the proposed system was measured by comparing 130 indexed segmented image cines from the eight cardiac cycles to the manually delineated segmentations by an expert radiologist, representing the gold standard used.
Since the images at hand were mainly small segmented foregrounds (left ventricular surface area) against vast backgrounds, the system performance would best be measured by means of its sensitivity, precision rate, and overlap ratio. Let us consider the following metrics: a true positive pixel is a pixel that is considered part of the left ventricle by both of the proposed method and the gold standard. A false positive pixel is a pixel that is considered part of the left ventricle by the proposed method but it is not considered as such on the gold standard. A false negative pixel is a pixel that is not considered as part of the left ventricle by the proposed method; yet it is considered to be part of the ventricle according to the gold standard. The sensitivity is the percentage of the number of true positive pixels divided by the sum of the number of true positive pixels, and false negative pixels. In other words, it classifies how well a binary classification test correctly identifies a condition. Precision rate is the percentage of the number of true positive pixels divided by the sum of the number of true positive pixels and false positive pixels. In other words, it classifies how accurate the results of the test when the results are positive. Overlap ratio is the percentage of the number of true positive pixels divided by the sum of the number of true positive pixels, false positive pixels and false negative pixels.
Sensitivity, precision rate, and overlap ratio of the proposed segmentation system.
Standard GVF snake
GVF snake with optical flow only
GVF snake with optical flow and priors
4. Concluding Remarks
In this paper, we have shown that optical flow can be utilized as an external energy within the GVF active contour framework. We have demonstrated that an active contour method is able to make use of the knowledge derived from the apparent motion of tissue via optical flow measurements. This strengthens the principle that tissue movement should be considered within segmentation techniques, where the data facilitates it.
Furthermore, results were improved by using primitive shapes, which helped overcome the inherent difficulties in segmenting echocardiographic images, by maintaining shape regularity. The proposed prior technique does not require training samples that are expert delineated; rather they are built from the current active contour control points. Furthermore, different primitive priors can be used depending on the feature of interest to be segmented. Such segmentations would improve the calculation of various clinical measures by reducing the inconsistencies and variability between clinicians while simultaneously reducing the time for clinician interaction.
This research is partially funded by the Natural Sciences and Engineering Research Council of Canada (NSERC). This support is greatly appreciated.
- Mazumdar B, Mediratta A, Bhattacharyya J, Banerjee S: A real time speckle noise cleaning filter for ultrasound images. Proceedings of the IEEE Symposium on Computer-Based Medical Systems (CBMS '06), 2006 341-346.View ArticleGoogle Scholar
- Tauber C, Batatia H, Ayache A: Robust B-spline snakes for ultrasound image segmentation. Journal of Signal Processing Systems 2009, 54(1–3):159-169.View ArticleGoogle Scholar
- Choy M, Jin J: Morphological image analysis of left-ventricular endocardial borders in 2D echocardiograms. Medical Imaging 1996: Image Processing, February 1996, Newport Beach, Calif, USA, Proceedings of SPIE 2710: 852-863.Google Scholar
- Montagnat J, Delingette H: Space and time shape constrained deformable surfaces for 4D medical image segmentation. Proceedings of the Medical Image Computing and Computer-Assisted Intervention (MICCAI '00), 2000, Lecture Notes in Computer Science 196-205.View ArticleGoogle Scholar
- Amini A, Radeva P, Elayyadi M, Li D: Measurement of 3D motion of myocardial material points from explicit B-surface reconstruction of tagged MRI data. Proceedings of the International Conference on Medical Image Computing and Computer Assisted Intervention, 1998, Lecture Notes in Computer Science 1496: 110-118.Google Scholar
- Kass M, Witkin A, Terzopoulos D: Snakes: active contour models. International Journal of Computer Vision 1988, 1(4):321-331. 10.1007/BF00133570View ArticleMATHGoogle Scholar
- Cohen L: On active contour models and balloons. Computer Vision, Graphics, and Image Processing: Image Understanding 1991, 53(2):211-218.MATHGoogle Scholar
- Amini A, Weymouth T, Jain R: Using dynamic programming for solving variational problems in vision. IEEE Transactions on Pattern Analysis and Machine Intelligence 1990, 12(9):855-867. 10.1109/34.57681View ArticleGoogle Scholar
- Williams DJ, Shah M: A fast algorithm for active contours and curvature estimation. Computer Vision Graphics and Image Processing: Image Understanding 1992, 55(1):14-26.MATHGoogle Scholar
- Cohen L, Cohen I: Finite-element methods for active contour models and balloons for 2-D and 3-D images. IEEE Transactions on Pattern Analysis and Machine Intelligence 1993, 15(11):1131-1147. 10.1109/34.244675View ArticleGoogle Scholar
- Mallouche H, de Guise J, Goussard Y: Probabilistic model of multiple dynamic curve matching for a semitransparent scene. Vision Geometry IV, July 1995, San Diego, Calif, USA, Proceedings of SPIE 2573: 148-157.View ArticleGoogle Scholar
- Szeliski R, Tonnesen D: Surface modeling with oriented particle systems. Computer Graphics 1992, 26(2):185-194. 10.1145/142920.134037View ArticleGoogle Scholar
- Xu C, Prince J: Snakes, shapes, and gradient vector flow. IEEE Transactions on Image Processing 1998, 7(3):359-369. 10.1109/83.661186MathSciNetView ArticleMATHGoogle Scholar
- Felix-Gonzalez N, Valdes-Cristerna R: 3D echocardiographic segmentation using the mean-shift algorithm and an active surface model. Medical Imaging 2006: Image Processing, February 2006, San Diego, Calif, USA, Proceedings of SPIE 6144: 147-151.Google Scholar
- Papademetris X, Sinusas A, Dione D, Duncan J: 3D cardiac deformation from ultrasound images. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI '00), 1999, Lecture Notes in Computer Science 1679: 420-429.View ArticleGoogle Scholar
- Leung K, van Stralen M, van Burken G, et al.: Sparse appearance model based registration of 3D ultrasound images. Medical Imaging and Augmented Reality, 2006, Lecture Notes in Computer Science 4091: 236-243.View ArticleGoogle Scholar
- Zhou S, Liang B, Chen W: A new method for robust contour tracking in cardiac image sequences. Proceedings of the 2nd IEEE International Symposium on Biomedical Imaging: Macro to Nano, 2004 1: 181-184.Google Scholar
- Park H, Chung M: A new external force for active contour model: virtual electric field. Proceedings of the International Conference on Visualization, Imaging and Image Processing (VIIP '02), 2002 103-106.Google Scholar
- Barron J, Fleet D, Beauchemin S: Performance of optical flow techniques. International Journal of Computer Vision 1994, 12(1):43-77. 10.1007/BF01420984View ArticleGoogle Scholar
- Lucas B, Kanade T: An iterative image registration technique with an application to stereo vision. Proceedings of Imaging Understanding Workshop, 1981 121-130.Google Scholar
- Horn B, Schunck B: Determining optical flow. Artificial Intelligence Journal 1981, 185-203.Google Scholar
- Russell S, Norvig P: AI, A Modern Approach. Prentice Hall, Englewood Cliffs, NJ, USA; 1995.MATHGoogle Scholar
- Hamou AK, Osman S, El-Sakka MR: Carotid ultrasound segmentation using DP active contours. Proceedings of the International Conference on Image Analysis and Recognition (ICIAR '07), 2007, Lecture Notes in Computer Science 4633: 961-971.Google Scholar
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.