A pilot study: 3D stereo photogrammetric image superimposition on to 3D CT scan images – the future of orthognathic surgery.

Àâòîðû: Balvinder Khambaya, Jean-Christophe Nebelb, Janet Bowmanb, Ashraf Ayouba, Frasier Walkerc and Donald Hadleyd
Èñòî÷íèê: http://staffnet.kingston.ac.uk/~ku33185/Papers/PDFformat/Portsmouth02.pdf

Abstract

Over the last three decades orthognathic surgery has become a routine procedure for the correction of facial deformity. The most commonly used method of planning is to cut up profile photographs magnified to the same size as the standardized lateral skull radiograph. These are then superimposed over the cephalographs. The various portions of soft tissue and underlying bone are moved around to produce the most acceptable result. Unfortunately, radiographic and photographic registration and superimposition are inaccurate. The prediction of the postoperative appearance is limited to two dimensions. To address these problems a truly 3D modality of planning is required. Using a stereo photogrammetry machine, we are able to capture the 3D geometry of the soft tissue air boundary – that is a three dimensional shape of the face itself. The acquisition of 3D images of the underlying skeletal hard tissue is made using a CT scanner. We present the process we have developed which allows the registration of these 2 sets of 3D geometry. The registration itself is performed semi automatically using a ICP based software.

Introduction

Over the last three decades orthognathic surgery has become a routine procedure for the correction of facial deformity. The most commonly used method of planning is to cut up profile photographs magnified to the same size as the standardized lateral skull radiograph. These are then superimposed over the cephalographs. The various portions of soft tissue and underlying bone are moved around to produce the most acceptable result. Unfortunately, radiographic and photographic registration and superimposition are inaccurate. The prediction of the postoperative appearance is limited to two dimensions. To address these problems a truly 3D modality of planning is required. The key to archiving a full three-dimensional simulation of soft tissue displacement hinges on being able to capture the 3D geometry of the soft tissue air boundary – that is a three dimensional shape of the face itself. Obtaining a 3D image of the underlying skeletal hard tissue is routinely carried out using computerised tomography (CT) or magnetic resonance imaging (MRI). Many techniques for 3D soft tissue capture are available including, biostereometrics [1], morphanalysis [2], laser scanning [3], direct digitisation [4], Moire scanning, sterolithography, ultrasonography [5] and stereo video techniques [6]. The most promising method of soft tissue capture is stereo photogrammetry. This involves the use of a pair of stereo video cameras to capture a stereo image pair of each side of the face, software than allows the construction of a photo-realistic 3D facial model. The model can be rotated, translated and dilated on the computer screen. In this paper we present the process we have developed which allows the registration of the 3D geometry of the soft tissue air boundary (acquired by photogrammetry) with a 3D image of the underlying skeletal hard tissue generated by a CT scanner. In particular we detail the ICP based algorithm that is used to perform the registration semi automatically.

Data acquisition

A Perspex head in which an intact human skull was embedded, was obtained from the Radiology Department of the Glasgow Dental School, Figure 1. A mask was then constructed on top of the Perspex head to reproduce some anatomical features of a human face, i.e. eyes, nose, lips and surface colouring etc, Figure 1. A stereo photogrammetric image of the face and a 3D spiral CT image of the head were captured, Figures 2 and 3.

Figure 1: Perspex head
Figure 1: Perspex head

Three-dimensional stereo photogrammetry image acquisition
The three-dimensional images were obtained using a stereo photogrammetry machine (C3D) [6]. The technique is based on the use of two pairs of stereo videocameras which are connected to a personal computer. The model head was placed at a fixed distance of 60 cm from the camera. Following calibration, the computer began a sequence of flash and acquisition of video images from both sides of the face. The process took 50 milliseconds. Upon completion of the image capture, the resultant six images (two black and white and one colour image for each side of the face) were saved. Then the data, determined from calibration, were attached to the images of the head. The final image was built using the C3D software and stored as a VRML file.

3D spiral CT image acquisition
The three-dimensional images were obtained using a four-slice spiral CT scanner (Marconi MX8000). The rotation time was set at 0.75 seconds with a pitch of 0.625, this gave an effective slice thickness of 1.3mm. A total of 400 slices were captured and stored as DICOM images on a CD-ROM. The 400 DICOM slices were then imported into AmiraTM and two 3D CT models built. The first image representing the air / Perspex boundary layer and the second image the skull / Perspex boundary layer. This was achieved by adjusting the threshold during model building in AmiraTM. These two images were then exported as VRML files with a common co-ordinate system.

Registration

In order to register data generated by CT scanners and data generated by the C3D image-based capture system, the two sets of data have to be converted in a common 3D file format, which is able to handle 3D models with or without associated texture file. VRML (Virtual Reality Modelling Language) was chosen for several reasons. First, VRML is the open standard for 3D multimedia and shared virtual worlds on the Internet, which can be read by any Internet browser; therefore VRML data can be visualised without the need of investing in any specialised 3D software. Secondly VRML is the format of the models generated by C3D, Figure 2. Finally the software used for processing the CT scanned data, AmiraTM, has the ability to export files in that format, Figure 2.

Figure 2: 3D data generated by the 3D imagers (with and without photographic texture)
Figure 2: 3D data generated by the 3D imagers (with and without photographic texture)
Figure 3: VRML models of the segmented CT scanned data (skull, skin and both)
Figure 3: VRML models of the segmented CT scanned data (skull, skin and both)

Since the models of the skin and the skull extracted from the CT scanned data are in the same coordinate system, the registration can be performed only between the C3D and the skin data to get the three registered models. The registration process is done in two steps. The first step normalises the position of the C3D and skin data using a Procrustes registration. The second one refines the registration using a modified version of the Iterative Closest Point algorithm (ICP) [7].

Since the Procrustes registration is based on the a priori knowledge of 3D point correspondences, a specially built graphic interface software has been developed [8] to set manually the corresponding 3D landmarks on the 2 models. These landmarks are used to solve a rigid body transformation (translation, scaling and rotation) mapping one model on the other. The relative translation is evaluated by measuring the distance between the centroids of the 2 sets of data. The scale factor is calculated by comparing their sizes: they are estimated by summing the distances between each landmark and the centroid. Finally the relative rotation can be efficiently determined by an established non-iterative method called the Singular Value Decomposition (SVD) method [9]. If a large number of corresponding landmarks could be set accurately on the 2 models, the Procrustes registration would be sufficient to provide an accurate registration between the two 3D models. Previous researches [10] showed that landmarks could be set on textured 3D models with an accuracy of 0.5mm. However, since the skin data generated by CT scanners is not associated with a texture, landmarks can only be extracted from geometrical features, what limits the number of available landmarks and the accuracy of their selection.

The second step of the registration, which is ICP based, establishes correspondences between data sets by matching points in one data set to the closest points in the other data set. It is an iterative process going through the following steps:

  1. For each point of mesh A, compute the closest point of mesh B.
  2. Solving a minimization problem, compute the registration vector.
  3. Apply the registration and update the position of the points of mesh B.
  4. Compute the mean square error of the previous iteration and the current iteration.
  5. Terminate the iteration if the change in mean square error is less than a preset threshold.

ICP is a very powerful algorithm; in particular it can handle a reasonable amount of noise. However since it is an iteration of minimisation problem, the algorithm may converge towards a local minimum. Generally it is overcome by starting the iteration loop from an approximate registration. Secondly, the mesh A has to be a proper subset of mesh M, otherwise some of the closest points used in the registration vector calculation are meaningless. The modified version of ICP we used was developed by Mao et al [8] and is called HICP because it incorporates a weight function firstly presented by Haralick et al [11]. The main difference from the original ICP algorithm [7] is that HICP considers the outlier problem and instead of using all the closest point pairs obtained in each iteration, each pair is weighted with a weight depending on the distance between them [11]. HICP also optimises the computation of the registration vector: the complexity of the algorithm is reduced by using a SVD algorithm [9] instead of using quaternions.

Experiments

ÒUsing the set of data previously described, we started the registration process by setting landmarks on the C3D and the skin meshes. Since there were very few visual cues on the CT scanned data, we only set 5 points, Figure 4a. Then the first registration was done between the 2 models using the Procrustes registration. As expected a rough registration was generated with a misalignment of the order of 1 cm, Figure 4b.

Figure 4: (a) Location of the 5 landmarks, (b) Procrustes registration based on these landmarks
Figure 4: (a) Location of the 5 landmarks, (b) Procrustes registration based on these landmarks

Then the areas used for the HICP registrations were defined on both models. We chose an area around the nose because that is where the features are the most marked on the human face, Figure 5a. In a couple of seconds the registration is automatically processed, Figure 5b. In order to assess the accuracy of the registration measurements were made around the nose area (where visually the misalignment is maximum): most vertices lie within a distance of 1 mm from the other mesh, however some odd ones were found at a distance of 1.7 mm.

Figure 5: (a) Area defined for the HICP registration, (b) registered meshes and detail of the nose area
Figure 5: (a) Area defined for the HICP registration, (b) registered meshes and detail of the nose area

We believe that the remaining misalignment comes mainly from the difference of 3D shapes between the 2 sets of data and not from the registration method itself: the accuracy of data generated by the C3D imager or the CT scanner is for both within 1 mm. Finally it is now possible to visualise the skull and C3D data together, Figure 6.

Figures 6: Registered skull with the C3D mesh
Figures 6: Registered skull with the C3D mesh

Conclusions

In this paper we have attempted to superimpose two 3-dimensional images obtained by two different modalities, stereo photogrammetry and a 3D spiral CT scan. The aim of the superimposition is to produce a 3D spiral CT scan of a subjects’ hard tissue i.e. skull, and over lay this with the subjects soft tissue drape in colour. The soft tissue needs to be positioned accurately over the underlying hard tissue. The space between the two would represent the soft tissue thickness. This study goes someway to addressing these objectives. Future studies will need to address the accuracy and validity of this superimposition technique and develop a more automated approach to image superimposition. A registration accuracy of 1-2mm at this very early stage is promising. The effect of CT scan slice thickness and the number of slices on the accuracy of superimposition needs to be calculated since the ionising radiation levels need to be kept as low as possible. This research is an exciting step forward for both the clinician and patient, from the point of procedure planning on a virtual patient to the proposed surgical outcomes in 3-dimensions for the patient.

References

  1. S. Berkowitz, J. Cuzzi. “Biostereometric analysis of surgically corrected abnormal faces”. Am. J. Orthod. 1977: 72: 526- 38.
  2. G. P. Rabey. “Current principles of morphanalysis and their implications in oral surgical procedure”. Br. J. Oral Surg. 1977: 15: 97-109.
  3. AM McCance, JP Moss, WR Wright, AD Linney, DR James. “A three dimensional soft tissue analysis of 16 skeletal class III patients following bimaxillary surgery”. Br J Oral Maxillofac Surg 1992;30:221-32.
  4. H. H. S Ip, L.Yin. “Constructing a 3D individualized head model from two orthogonal views”. Vis. Comput. 1996: 12: 254-67
  5. B. Hell. “3D sonography”. Int. L. Oral Maxillofac. Surg. 1995: 24: 84-89
  6. Ayoub, A. F., Siebert, J. P., Wray, D. and Moos, K. F., “A vision-based three dimensional capture system for maxillofacial assessment and surgical planning”, Brit. J. Oral Maxillofacial Surg. 36, p.p. 353-357, 1998.
  7. P. J. Besl and N. D. McKay, “A Method of Registration of 3-D Shapes”, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 14, no. 2, pp. 239 - 255, Feb. 1992.
  8. Z. Mao, P. Siebert and A. F. Ayoub, “Development of 3D measuring techniques for the analysis of facial soft tissue change”, Proc. Medical Image Computing and Computer-Assisted Intervention 2000 (MICCAI 2000), Pittsburgh, USA, 2000.
  9. K.S.Arun, T.S.Huang, and S.D.Blostein. “Least-square fitting of two 3d point sets”. IEEE Trans. Pattern Anal. Machine Intell., 9(5):698--700, September 1987.
  10. M. Y Hajeer, D. T. Millett, A Ayoub, W. J. S. Kerr and M. Bock, “3 dimension soft-tissue changes following orthognathic surgery - a preliminary report”, British Orthodontic Conference, Harrogate, 30th Sep. - 3rd Oct., 2001
  11. R. M. Haralick, H. Joo, C. Lee, X. Zhuang, V. Vaidya, and M. Kim. “Pose estimation from corresponding point data”, IEEE Transactions on Systems, Man and Cybernetics, 19:1426-1446, 1989.