Seamless Heterogeneous 3D Tessellation via DWT Domain Smoothing and Mosaicking

The contemporary geobrowsers—like Google Earth, NASA's World Wind, or Microsoft's Virtual Earth—give little importance to individual client characteristics, and with the low-end clients, the performance is far from being satisfactory. In addition, area coverage by these browsers is slack, if not judicious, in the case of the underdeveloped and developing parts of the world as opposed to their developed counterparts. There is a need to reform all these browsers to cater for the diversity of clients through some scalable data structuring. One obvious choice is to create multiple levels of detail (LOD) by representing the shape at different levels of approximation. This is essential for the terrain tessellation adjustment as a function of the view parameters [1]. For LOD, one can rely on the multiresolution nature of the discrete wavelet transform (DWT). It is better to employ some standard state-of-the-art DWT, like the now widely accepted JPEG2000 standard (The ISO/IEC 15444-1 standard.). In fact JPEG2000 is better than the popular pyramidal resolution representation with JPEG or other format closer to normal graphic cards that are of high storage cost: there is a need to store different resolutions. Additional advantages of the JPEG2000, in this context, include capabilities like better compression and progressive data transfer.

Traditionally, tile-based approaches have been favored to render large terrains. The problem, however, arises when the tiles have either been rendered at different resolutions or not photographed in identical luminance or radiometric conditions. To elaborate this we rely on the two snapshot examples of Google Earth, shown in Figure 1. In the first example (Figure 1(a)), where the observer is changing his view to the right, it can be seen that the resolution is not uniform. Usually there are two reasons for this heterogeneity. First is the download latency and second is the nonavailability of proper resolution due to a variety of reasons like area security, data inaccessibility, and the like. The second snapshot (Figure 1(b)), which pertains to the area surrounding the Pakistani capital of Islamabad, suffers from both heterogeneous resolution and aerial photography in nonidentical light conditions. Obviously the photographs have been taken at different times and then joined together to what we see in the snapshot.

In essence, one can safely deduce that even for today's geobrowsers the tessellations are far from being smooth due to varying resolutions or heterogeneous luminance conditions or even both when the tiles have been acquired. Whilst, the traditional focus of the mosaicking community has been the seamless rendering of tiles that may vary in luminance, little attention has been paid to the heterogeneity due to resolution. That is why traditional mosaicking may correct the light difference phenomenon but the underlying tiles may still be hardly seamless if they vary in resolution. In this paper we focus on the resolution part, mainly, as we try to smoothen the tile interfaces to avoid the artifacts like those in Figure 1, for example. Our methods are aimed at rendering texture and DEM tessellations, from JPEG2000-coded tiles, in a scenario where only visible tiles are rendered and the focused ones, out of these, have the highest quality depending on the viewer's distance. As the viewer moves in, the tiles corresponding to the new view must be rendered and gradually refined in quality as the focus is coming near to it. This refinement is additive, as more and more subbands are added to it, at runtime. The main challenge to this strategy is that the tile must be seamlessly stitched for both the texture and the DEM and the boundaries must be diluted enough to prevent popping artifacts. In this paper, although we are more inclined to the texture, the DEM has also been treated as its integral part. We, therefore, present two different methods to look for a seamless tessellation that is composed of heterogeneous resolution tiles. In fact this work is a continuation of our earlier efforts about the synchronous unification of the DEM/texture pair through data hiding [2] for scalable 3D visualization. That work is our inspiration for the LOD aspect, since its scalable character can result in various tile qualities for both the DEM and its texture.

The first method is local in nature and relies on context-based smoothing of the tile boundaries. The smoothing is carried out at the subband level in the DWT domain by employing three different smoothing masks. We assume that, at worst, there can be three possible tile resolutions, namely, high, middle, and low. The lowest-resolution tiles of the panorama are not treated while the highest-resolution tile are smoothed by a radial smoothing function. The rest of the tiles are rendered at mid-level resolution. Of these the left/right neighbors of center are smoothed horizontally, while the above/below neighbors are smoothed vertically. The treated tiles can be joined, after each has been subjected to an inverse DWT, to get a smoother tessellation.

In the second method, we take the change in viewpoint focus, analogous to a window-sliding approach. The worst we expect is that the window may come up to be composed of three different tile qualities with the resultant artifacts at tile interfaces. To dilute these artifacts, we treat the tiles at the subband level, in the DWT domain, by employing operations involving suitable subband-sized composite masks, conceived with smoothing and mosaicking in perspective. The resultant composite subbands are subjected to a global inverse DWT to get the final seamless tessellation.

The rest of the paper is arranged as follows. Section 2 gives a brief literature survey on the subject. The presentation of the proposed methods is the theme of Section 3. Application of our methods to a practical worst case example is being demonstrated in Section 4. Section 5 concludes the paper.

The process of mosaicking involves the connection of two or more images to get a new wide-area image with no visible seams. In the case of aerial or satellite photographs, a wide screen-view is synthesized by the proportional combination of a set of images. The mosaic is hence commonly used to increase the view field with pasting successively overlapped images onto the same plane or other structures, like a cylinder or a sphere. In order to be efficient, a mosaicking method should be robust to light change, moving objects, image noise and to some extent rotation and zooming [3]. An integral part of the process is image registration which pertains to the overlaying two or more images of the same scene taken at different times, from different viewpoints or by different sensors. The final information is gained from variety of sources like image fusion, change detection and multichannel image restoration. In this context Zitova and Flusser [4] classify the various approaches according to their nature (area-based and feature-based) and to four basic steps of image registration procedure, namely, feature detection, feature matching, mapping function design and image transformation and resampling. The thrust of the mosaicking methods, in general, is either on the interest point matching or on theoretical corner models. All these, hardly consider the aspect where a resolution image is to be stitched to a considerably lower-resolution image.

Terrain LOD algorithms can be classified [1], on the basis of the hierarchical structure, into four groups, namely, the triangulated irregular networks (TINs), bin-tree hierarchies, bin-tree regions and tiled blocks. The tessellation, in the last category, is carried out with the square tiles of different resolutions as has been done by Wagner [5]. But the process requires seamless stitching at the boundaries. According to Deb et al. [6], Wagner's technique is not suited for the terrains with large height variations and that is why they tweak the technique by having the projection on real-time average height of terrain for a client/server terrain streaming to handle heterogeneous clients. The blending factors are calculated on a per-tile basis because of the use of a regular tile structure, thus, reducing the amount of computation. Special techniques based on mipmaps and clipmaps [7] can be found in the literature. Losasso and Hoppe [1] break the terrains into geometric clipmaps of varying metric sizes which can be used as LODs through a view-centered hierarchy resulting in a simplified spatial and temporal interlevel continuity. A thresholding scheme based on calculated ground sample distances is proposed by Tsai and Chiu [8] with a nested LOD system for efficient seamless visualization of large datasets. In order to remove the T-junctions they have put forward a mesh refinement algorithm. Ueng and Chuang [9] propose a dike structure between two adjacent blocks,for a smooth blending between two meshes of different LOD. This approach is combined by Li et al. [10] with heuristics to dynamically stitch the tile meshes together seamlessly. For the texture part they use high-resolution aerial photos subjected to the similar LOD mechanism as described for the meshes. Larsen and Christensen [11] try to avoid popping for low-end users by exploiting the low-level hardware programmability in order to maintain interactive frame-rates. They claim their work as pioneering as far as a smooth LOD implementation in commodity hardware is concerned.

Little attention has been paid to the texture images which require their own LOD structure. There have been proposals, like texture clipmaps [7], but texture tiling is usually preferred. The multiresolution technique, described by Döllner et al. [12], presents static LOD terrain models, allowing for combination of multiple large-scale textures of different size. The view-dependent texture management technique of Okamoto et al. [13] manages multiresolution textures in multiple caching levels—database, main memory and graphics card memory—which the authors claim to be suitable for hardware with limited resources. The approach of Buchholz and Döllner [14] includes dynamic texture loading for memory management and, combine several textures into atlases for avoiding too much texture switches [15]. Their computationally complex preprocessing part helps them to significantly reduce runtime rendering overhead. Frueh et al. [16] described an approach to create a specialized texture atlas for building facades and supports efficient rendering for virtual flythroughs. The created atlas is static, and different texture resolutions are not considered. Based on the minimum error boundary cut idea of the image quilting algorithm as well as Efros and Freeman [17], Somol and Haindl [18] propose their minimum error path search algorithm. To avoid visible seams,for combining incompatible pieces of texture, they propose to find (possibly irregular) boundaries between the image pieces to minimize the visual error. Based on eccentric image overlapping, the source image is cropped at the prospective interface along the minimum error path and placed over the target background image. The problem is, however, if no good path exists in the error map, visible artifacts may be inevitable. Integer wavelet transform has been applied to DEM and a simple coding algorithm with high efficiency was introduced by Chen and Li [19]. Many references can be found on the subject in the works of Danovaro et al. [20] and Buchholz and Döllner [14]. A survey about 3D interactive rendering by Pajarola and Gobbetti [21] is worth reading.

In this section we explain the two methods for seamlessly stitching heterogeneous resolution tiles. This heterogeneity is traced back to the concept of the level of detail (LOD). For the LOD we are relying on the multiresolution nature of DWT from the JPEG2000 codec. The reasons for choosing JPEG2000 are its scalability, its wide acceptance as a standard, its support to various wavelet forms for multiresolution, progressive transfer capability and last but not the least is its approximation paradigm for LOD. The various approximations do not differ in size but only in quality which is additive in nature. If each tile is DWTed in the JPEG2000 codec at level then a level- approximate image is the one that is constructed with percent of the total coefficients. These coefficients correspond to the available lowest subbands and for the coefficients corresponding to the rest of the subbands, zeros are stuffed. Subsequent application of -level inverse DWT yields what is known as the -approximate image. For example, level- approximate image is constructed from all the coefficients and level- approximate image is constructed from of the count of the initial coefficients.

3.1. Scenario

We are assuming a simplistic scenario of a panorama of tiles given in Figure 2. This use case should represent almost all the possibilities we will have in our application. An individual tile is large enough to circumscribe the field of view if the observer is close enough to require highest visual resolution. If the observer is far off, his view is limited to at least tessellation with all the tiles at their lowest visual resolutions but as soon as he draws closer, further details may be added to the tile of focus and, to some extent, its -neighbors to improve their visual resolution. This leads to variations in the tile qualities of the view tessellation. The tessellation is thus heterogeneous in terms of resolution and a given tile can have one of the three qualities, namely, high, middle and low. The high-quality resolution (H) corresponds to the center which is the viewpoint focus for a given distance to the observer in a flyby process. We are rendering the -neighbors of the center with the middle-resolution (M) tiles and the rest, that is, the corners, with low resolution (L). The terms high, low and middle are not static and their extent depends on the distance of observer from the view as well as the time passed after the viewer changed his position. It also depends on the capacities of the media employed, as we may have M and L or just L for poorer conditions. There may be a tessellation having concentric square rings with each successive ring having a different quality [7] but we believe that at any instant one would hardly be dealing with a rendering involving more than three resolutions.

3.2. Shape of the Smoothing Function

Figure 3 depicts the shape of an example smoothing function in the simplest of forms. Note that we are affecting the quality of only a small part on the periphery, of the higher resolution tiles in the view, during a given smoothing process. For any given subband, the smoothing function creates a scalar multiplication mask (over ) of the size of the subband. This depends on the position of the tile, to which the subband belongs, in the view. for example, for the subbands of a tile having low resolution neighbors on left/right, we may consider the central column of coefficients as the origin in the figure and may then be the offset from this central column on right or left with being half the width of the subband. In a similar fashion, for a tile with up/down neighbors in the view, the origin may be the center row with row offset represented by . Finally for a tile in the -neighbor environment, the center of its subband may be the origin and may be the offset of the polygonal ring of coefficients. The function has two parameters, and . the parameter decides where to start the smoothing process in the subband whereas denotes the index where it should fall to zero. To have a full smoothing in a subband we must have and , but to dilute the seam, it is preferable to have and to not deteriorate too much the subband it is preferable to have . To not have a strong smoothing between and and avoiding the full deterioration of the subband, one can locate at one quarter of the subband, that is, , where, as already stated, is roughly half the subband dimension (width, length or radius).

In Figure 3 the given function is of the form,

(1)

Of special importance is the shape of the fall from to which may be streamlined to emulate a more sophisticated function, for example, tail of a Gaussian distribution. One such function could be:

(2)

Note that beyond we shall be dealing with consecutive integers as ; therefore . One can borrow from sinusoidal functions too and a smoothing function that we tested is given below:

(3)

There can be many more functions to emulate but due to space limitation we would restrict to the three given in (1)–(3) only.

Let the high, the middle, and the low resolution of our scenario correspond to , , and approximate images, respectively. The quality gap would essentially bring with it the popping artifacts at tile boundaries. To undo that, we are proposing two different DWT domain smoothing methods in the next two sections.

3.3. Method I: Smoothing the Tiles before Stitching

In this section we present a wavelet-domain context-dependent smoothing strategy illustrated in Figure 4. At the corners (labeled L) are the -approximate images obtained from lowest subbands with the rest of subbands being replaced by s. For the rest of the tiles one can partition their subbands into the following three sets.

1. (i)

   The lowest subbands will remain untouched since they represent the lowest rendered quality.

    
2. (ii)

   The highest subbands of center tile (H) and subbands of its -neighbors (labeled M) are stuffed with s.

    
3. (iii)

   Last are the remaining and subbands of central tile and its -neighbors, respectively.

    

It is the third set (or its subset thereof) which will be subjected to the DWT domain treatment with three different smoothing functions on each subband individually. These smoothing functions are the following.

1. (1)

   The medium-resolution left/right neighbors of center require the horizontal smoothing function.

    
2. (2)

   The medium-resolution above/below neighbors of center need the vertical smoothing function.

    
3. (3)

   The high-resolution center is treated through the radial smoothing function.

    

The horizontal smoothing mask would have a central band of rows of 's. The thickness of this band depends on the maximum diameter, , of the view focus. If the subband belongs to the resolution level, then the thickness should be around . Beyond the band of 's, the rows gradually attenuate on both sides approaching at the periphery, complying anyone of the (1)–(3). An example smoothing function, for the horizontal case and loosely based on (1), is described in the appendix in the form of Algorithm 1. Along similar lines we can get a vertical smoothing mask by transposing the horizontal mask for a given subband. An example illustration of horizontal smoothing between two tiles (a high-resolution tile H and a medium-resolution tile M on right) is given in Figure 5.

Algorithm 1: Example horizontal smoothing function.

  Input: dimensions ( ) of the DWT domain subband and smoothing parameters ( )

  Output: Horizontal smoothing mask

( ) begin

( ) set  

( ) set  the coefficients of the first and last rows of to

( ) set  the coefficients of the to rows of to

(5) for     do

(6)         

(7)   for     do

(8)      

(9)      

(10)  end for

(11) end for

(12) end

The radial smoothing mask is a bit tricky. We would have a center core of 's having a diameter of about for a the subband of the resolution level. Ideally, this core should be circular with being its radius, but due to the discrete character of the subband, the shape is at best polygonal. For the sake of simplicity of Algorithm 2 presented in the appendix, we are considering a square core of 's with dimensions . Each ring of coefficients around the core would gradually attenuate approaching at the corners (or the ring to be precise) and around in the up/down and left/right directions.

Algorithm 2: Example radial smoothing function.

  Input: dimensions ( ) of the DWT domain subband and smoothing parameters ( )

  Output: Radial smoothing square mask of dimensions ( )

( ) begin

( ) set  

( ) declare an array with all elements initialized to

( ) for     do

(5)    

(6)    

(7) end for

(8) declare  a array

(9) for     do

(10)  for     do

(11)   if     then

(12)    

(13)    

(14)   else

(15)    

(16)    

(17)   end if

(18)  end for

(19) end for

(20) end

After the smoothing treatment of a subset or all of the third group of subbands and inverse DWT, one can have a seamless visualization of the type given in Figure 4. In our application, we are using the same strategy for the DEM to avoid popping effect on elevation data.

3.4. Method II: Sliding Window Analogy to View Field

For method II we introduce a sliding window analogy, as shown in Figure 6, where the red square/window represent our initial point of focus which had been rendered with high-quality resolution, H. Let the focus changes and the window slides to the position represented by the blue square. The new window may have, at the worst, three different resolutions. This situation is illustrated by the panorama in the right part of Figure 6. Rendering three different resolutions, side by side, will obviously give rise to nonsmooth transitions at the interface of heterogeneous tiles.

For a 1D viewpoint displacement from H to M or M to L, whether horizontal or vertical, a smooth transition can be realized by following the scheme given in Figure 7. This figure illustrates the DWT domain, H to M seamless smoothing, in the horizontal direction. For H we have all the subbands available but for the M part we have just level-2 subbands available, and six highest-frequency subbands are missing which will be replaced with zero coefficients. This gives rise to two types of combinations. First, the direct concatenation of nonzero subbands with their counterparts from H. Second, the concatenation of each of the zero subbands with the pretreated corresponding subband of H. The pretreatment involves scalar multiplication with a subband-sized mask generated by one of the functions, , outlined Section 3.2, with being the column index of the subband coefficient. Note that for a vertical transition, column index will have to be replaced with the row index in the function . The resultant concatenated H/M subbands, upon inverse DWT, results the required seamlessly smoothed image.

Note that we are affecting the quality of only a small part on the periphery of the tile during a given smoothing process. For any given subband, the smoothing function creates a scalar multiplication mask (over ) of the size of the subband. For example, the H to M horizontal smoothing mask would have a band of columns of 's on the left. The thickness of this band depends on the maximum diameter of the view focus. Beyond the band of 's the columns gradually attenuate toward right, approaching at the periphery. Along similar lines we can get a vertical smoothing mask for a given subband.

In two dimensions, the diagonal transition from H to L is also to be taken into account. One can think of the 2D version as an operation involving either two horizontal 1D smoothings followed by a vertical 1D smoothing, as shown in Figure 8, or two vertical 1D smoothings followed by a horizontal smoothing. Keeping this in view, these operations can be easily carried out in parallel to avoid time delay, thanks to the involvement of subbands and the independence of blocks as envisioned in the JPEG2000 standard. Taking the argument further, since we have three different resolution tiles, for concatenation we may face three subband combinations as follows.

1. (1)

   The corresponding subbands of all of the H, M, and L are nonzero implying direct concatenation according to their localization in space.

    
2. (2)

   With zero subbands of L corresponding to nonzero subbands of M and H, we would follow the smoothing strategy described in Figure 8.

    
3. (3)

   For nonzero subbands of H, corresponding to zero subbands of L and M, we would concatenate the former in the pretreated form with the zero subbands keeping in view their localization in space.

    

But we believe that if one think at the outset about the diagonal transition by starting the smoothing process from those subbands of M and H whose counterparts are zero in L, the overall smoothing may still be commendable. In addition the closest part of H to L can be programmed to be degraded more than the rest of its interfaces during smoothing.

The results, presented above, have been interesting in the face of the fact that we presented a worst case example tessellation in terms of quality difference as well as DEM size. Whilst the two methods were at par in the case of texture stitching, method II had the edge in the case of DEM. The problem of synchronization may arise if the DEM is different enough to warrant its own LOD. One probable answer to this is the fact that since the LOD is wavelet driven, one can always synchronously unify the DEM texture pairs as explained by Hayat et al. [2] or even adapt the synchronization [22] to exclude some subbands from embedding. One can still argue that the seam elimination has been realized at the expense of blurring the better resolution tile. The counter argument to this is the fact that the resultant blur is temporary and as soon as further resolution information for a given tile arrives, one can employ complement of the mask to restore the lost information.

In our method we are using a regular grid. To increase the performance, we would like to employ an irregular grid depending on the DEM resolution. The well-known T-cracks problem will be avoided by the application of the junction technique of Pajarola and Gobbetti [21]. In the continuation of our work we would like to extensively analyze and optimize the parameters of the functions employed for smoothing. Alongside, a better function needs to be discovered to further improve the stitching. The overhead in terms of the DWT and its inverse has to be quantified and optimization must be proposed for practical applications. The ultimate is its extension to real-time environment, with the emphasis on scalability.