Seven decades of image super-resolution: achievements, challenges, and opportunities

Abstract

Super-resolution imaging has, for more than seventy years, gradually evolved to produce advanced methods for enhancing the resolution of images beyond the diffraction limits. Notwithstanding its foreseeable practical capabilities, we noted that this technology has received undeserved attention. The present work provides an extensive review of super-resolution imaging since its first conception in 1952, contextualizing studies into four research directions: reviews, advances, applications, and hardware implementations. We have discussed achievements, challenges, and potential opportunities of super-resolution imaging to equip researchers, especially those in their early careers, with knowledge to further advance the technology. Our work may open interesting research avenues that may accelerate realization of the technology into commercial imaging devices.

1 Introduction

The long-standing idiom

“A picture is worth a thousand words”

reflects varied interpretations depending on the context and discipline [1,2,3]. For our case, the idiom may be interpreted as pictures (images), unlike words, provide a quicker content-rich visual communication. A single image may contain multiple messages with a story that could otherwise be told through many words. Given this advantage, there has been massive efforts to generate images with higher visual qualities for easier interpretation and analysis.

Perceptually attractive images embed sharper, clearer, and detailed features—hence the term resolution that defines the information density in the image. Common types of image resolution include spatial resolution [4] (number of pixels that an image contains), angular resolution [5] (minimum angular distance that an optical instrument can discern two distant objects), radiometric resolution [6] (number of bits per pixel that distinguishes different gray-scale values), temporal resolution [7] (time needed to revisit the same location to acquire an image), and spectral resolution [8] (distinguishable wavelength bands detected by the imaging sensor from the electromagnetic spectrum). This work focuses on the spatial resolution that defines the quality of an image based on its information density. From now onwards, resolution, unless otherwise stated, means spatial resolution.

The scientific inquiry and human desire for quality scenes have necessitated the development of approaches (methods, algorithms, and techniques) to improve the resolution of images. Typical approaches include hardware modification [9] and image processing [10]. The former approach may be achieved by increasing the number of pixels on the surface of the imaging sensor. This process necessitates reduction of the pixel size and integration of complex analog and digital circuits on the sensor chip [9, 11]. However, the amount of incident light on the sensor surface decreases with the pixel size. Consequently, the imaging sensor tends to generate shot noise that degrades the quality of an image [11]. In addition, resolution enhancement through hardware modification increases cost and bulkiness of the imaging device. Challenged by these limitations, researchers have proposed software approaches, including super-resolution [12,13,14], to increase the resolution of images without modifying the hardware.

In 1952, the concept of super-resolution was conceived for the first time by Giuliano Toraldo di Francia [15]. The author’s original idea was to improve the angular resolution of an optical system beyond its diffraction limit,¹ governed by uncertainty principle stating that [16] “a wave cannot be localized much tighter than half of its vacuum wavelength.” All developments in (optical) super-resolution imaging centers on addressing this limitation, and advanced techniques attempt to lower the pre-defined maximum threshold of the wavelength.

Since conception of the idea, there has been some developments in super-resolution imaging across different science and engineering fields. Notwithstanding the developments, there has been inadequate comprehensive review works tracking the origin of this technology to date. The current work explores the evolution of this important technology over the last 70 years. We discuss achievements, challenges, and opportunities of the super-resolution methods to guide researchers on the possible research avenues to advance the technology.

2 Super-resolution imaging

2.1 Fundamental concepts

The field of super-resolution imaging has been evolving over time (Fig. 1), capturing a broad range of science and engineering applications. However, compared with most other image processing fields, the rate of publications in this field seems unsatisfactory. Despite being an old field, we noted a skewed publication landscape of super-resolution imaging. Using the VOSviewer tool,² we analyzed 2,504 publications on super-resolution imaging extracted from the Scopus³ and PubMed⁴ databases. Ranging between 1988 and 2022, these publications—extracted using the search rule “super-resolution imaging” OR “optical super-resolution” OR “geometrical super-resolution”—describe different aspects of the field (reviews, advances, applications, and hardware implementations). Our observation from the analysis shows that China and United States are the leading countries in super-resolution imaging, but links for the collaboration network between these developed countries and the developing ones is relatively weak (Fig. 2). Given several advantages of super-resolution imaging, including hardware cost reduction, strengthening the collaboration between developed and developing countries may be important. Generally, advancing the super-resolution imaging field globally requires coordinated plans and support from funding organizations. The target should be to implement super-resolution imaging algorithms into electronics devices, hence reducing their costs for accessibility by the developing world. In this respect, hardware and software developers may need to focus on four factors when designing super-resolution techniques: hardware resolution requirements, computational load (speed), algorithmic complexity, and overall product price.

Fig. 1

Trend of publications on super-resolution imaging

Fig. 2

Collaborative network visualization of publications on super-resolution imaging

Studies on super-resolution imaging take four directions (Table 1): reviews, advances, applications, and hardware implementations. Understanding these directions may help early-career researchers to focus their research in a comprehensible way. Considering studies that advance the field, super-resolution methods can broadly be grouped into three categories, namely optics-based, geometry-based, and hybrid [17] (Fig. 3). The general objective of these methods is to improve the quality of images generated by imaging systems. While the optics-based category deals with improvement of the angular resolution of an optical instrument, the geometry-based category focuses directly on the spatial resolution enhancement of an image. Hybrid super-resolution approaches, which have received wide attention in recent years, combine both optical and geometrical super-resolution techniques to generated more detailed images [18,19,20].

The current work covers geometrical super-resolution methods, specifically those exploiting the number of input images to reconstruct a high-resolution image. Subsequently, we have single-frame and multiframe methods. To generate an image with a higher perceptual quality, the former method uses a single degraded and noisy image while the later method explores additional information from multiple low resolution images of the same scene. Over the past few decades, multiframe super-resolution (MSR) methods have gained wider attention for their ability to add extra details into the final reconstructed image. Inspired by its potential benefits, this work covers MSR methods.

In 1964, Harris [21], extending the work of 1955 by Toraldo di Francia [22], established fundamental theories and concepts to address the diffraction problem in optical systems. Ten years later, Gerchberg [23] showed that reducing energy error can significantly improve the resolution of an image beyond the constraints posed by diffraction. The author attempted to recover some high frequency components from a single degraded image through an iterative phase retrieval technique. However, involving a single image in the reconstruction process fails to incorporate additional information into the final solution. In 1984, Tsai [24] proposed the first MSR method based on the frequency domain to improve the resolution of LandSat Thematic Mapper images [25]. This method allows quicker implementation and offers lower computational load. Since the work by Tsai, several advanced MSR methods have been proposed and applied in various fields [26,27,28,29,30,31].

Table 1 Directions of super-resolution imaging publications

Fig. 3

Classification of image super-resolution methods. Colored arrows show the selected path taken by this research

The MSR problem can be described using the observation model that demonstrates how an ideal high resolution image undergoes multiple degradations to generate a sequence of degraded (low resolution) images (Fig. 4). Let the (unknown) high resolution image, u, captured by the imaging device be warped, blurred, and decimated by operators $W_k$, $B_k$, and $D_k$, respectively, to generate a sequence of low resolution images, $\{y_k\}$, with k indexing the generated image in the sequence. If $y_k$ gets corrupted by an additive noise, $\eta _k$, (independent and identically distributed) then the observation model can be represented mathematically as

$$\begin{aligned} y_k=W_kB_kD_ku+\eta _k. \end{aligned}$$

(2.1)

From (2.1), the observation model reduces to estimating u given the limited number of $\{y_k\}$ and unknown degradation operators ($W_k, B_k, D_k,$ and $\eta _k$). Equation (2.1) denotes an inverse problem [111], often addressed using iterative algorithms and optimization methods.

Fig. 4

Observation model for a multiframe super-resolution problem

Let M low resolution images be generated by the imaging device represented by the observation model. Then, rearranging (2.1) and introducing the widely used $\ell _2$-norm error minimization strategy yields the MSR optimization problem

$$\begin{aligned} 0={{\,\mathrm{\arg \!\min }\,}}_u\left\{ \frac{1}{2M}\sum _{k=1}^{M}\Vert W_kB_kD_ku-y_k\Vert ^2_2\right\} . \end{aligned}$$

(2.2)

Other strategies for error minimization, including $\ell _0$ and $\ell _1$, may be applied as well to derive the optimization problem from which u can be estimated.

Inverse problems, including image super-resolution in (2.2), are inherently ill-posed and can, therefore, generate unstable and undesirable solutions. Regularization techniques are usually applied to address the ill-posed nature of the inverse problems [112, 113]. Guided by these techniques, a regularization term should be incorporated into (2.2), giving a formulation

$$\begin{aligned} 0={{\,\mathrm{\arg \!\min }\,}}_u\left\{ \frac{1}{2M}\sum _{k=1}^{M}\Vert W_kB_kD_ku-y_k\Vert ^2_2+\frac{\lambda }{2}\Vert \textrm{Ru}\Vert _2^2\right\} , \end{aligned}$$

(2.3)

where $\lambda >0$ and R denote the regularization parameter and stabilization matrix, respectively. Solving (2.3) and subjecting the resulting formulation into the continuous dynamical system gives an estimate of u as

$$\begin{aligned} \frac{\partial u}{\partial t}=\frac{1}{M}\sum _{k=1}^{M}W_k^TB_k^TD_k^T(W_kB_kD_ku-y_k)+\lambda (\textrm{Ru}). \end{aligned}$$

(2.4)

The solution space in (2.4) can be discretized in the computer using well-established numerical schemes [114,115,116,117]. Advances in MSR mostly focus on estimating degradation operators ($W_k$, $B_k$, and $D_k$), designing R, and formulating MRS optimization problems (Table 1).

2.2 Achievements and challenges

The quest for quality images has attracted researchers, professionals, and practitioners to invest massively in super-resolution imaging. The potential benefits, applications, and capabilities of this promising technology make it attractive, interesting, and increasingly useful. Google, for instance, initiated a project to apply artificial intelligence in upscaling the resolution of images [118, 119]. Google researchers applied two approaches to achieve outstanding super-resolution results: iterative refinement [118] and cascaded diffusion models [119].

The achievements in image super-resolution can be discussed broadly along four directions. Firstly, development of methods, algorithms, and techniques for resolution enhancement. Secondly, establishment of frameworks to address the super-resolution problem. Thirdly, practical applications and use cases of the super-resolution technology. Fourthly, development of quality assessment metrics for images generated through the super-resolution process. Researchers and practitioners should be guided by these directions to advance the super-resolution technology.

In the first direction of achievement, researchers have proposed several approaches (methods, algorithms, and techniques) to restore the quality of degraded images (Fig. 3). Of the approaches, those based on machine learning have, in recent years, gained a considerable attention of researchers [13, 120,121,122,123]. There seem to be a promising future of the super-resolution technology under machine learning approaches, especially when combined with MSR. In their recent article, Ooi and Ibrahim [77] highlighted the challenges of these approaches for researchers to capitalize and advance the technology.

The second direction of achievement calls for efforts to establish efficient and robust frameworks for image super-resolution. Studies may be conducted to investigate strengths and weaknesses of the available frameworks [124], then devise practical solutions to address potential weaknesses and limitations. The frameworks may form the basis for researchers to build methods for image super-resolution. In our survey, we could not locate sufficient information that systematically guides researchers on the development of super-resolution methods based on standard frameworks. In machine learning (e.g., deep neural networks) approaches, for instance, most frameworks observed in the literature lack information on why they work and how they are systematically and logically designed.

The third achievement of image super-resolution can be observed in domestic and industrial products, where the technology has been applied to generate detailed and sharper images [34, 56, 125]. Despite the current achievements, this direction is still in the early stage with a number potential research opportunities. Most super-resolution algorithms in the literature have not been tested and implemented in practical imaging devices, including mobile phones, microscopes, scanners, and cameras. This challenge emanates from limitations and high cost of hardware implementation. In essence, we have not fully exploited the capabilities offered by the super-resolution technology and its applications in different scientific and engineering products, including imaging devices for super-resolving text images [126, 127], astronomical objects [128], face [50, 129], and underwater creatures [130, 131].

Fourthly, the super-resolution technology has resulted into the establishment of metrics for image quality assessment (IQA). Considering image super-resolution, the primary goal of the IQA metric is to measure the information richness of the super-resolved image. In objective quality assessment, the IQA metric quantifies the degree of image resolution enhancement after the super-resolution process. Despite the efforts and achievements in the development of IQA metrics, we observed lack of recommendations on how researchers should select such metrics, based on specific criteria, for different application domains. Given the image processing task (e.g., resolution enhancement, noise removal, or inpainting), conclusions from results may be misleading if an incorrect IQA metric is applied to assess the quality of the generated images. This knowledge gap calls for researchers to establish benchmarks for selecting IQA metrics.

Considering the degradation model (Fig. 4) that generates (2.3) and (2.4), MSR suffers from additional challenges requiring scholarly attention. Firstly, there has been no standard guidelines on the choice of M, number of low-resolution frames. Practical applications require a proper value of M for generating optimum results. Secondly, estimation of degradation operators has mostly been done under simulation experimental settings. In practice, these operators occur naturally within the imaging system. Therefore, researchers should develop more advanced approaches to accurately estimate values of the degradation operators, an attempt that may facilitate realization of MSR in practical devices. Thirdly, the degradation model (Fig. 4) deals with additive noise that cannot completely represent the natural imaging environment. Images encounter different noise types (e.g., multiplicative and mixed) uncaptured by the degradation model. This limitation calls for a need to revise the model and make it adaptive. In practical settings, the super-resolution method should adaptively and simultaneously perform resolution enhancement and noise removal, reversing all degradations that corrupt the original image. Fourthly, more effective regularization functionals should be established to address the ill-posed nature of the MSR model. Equation (2.4) incorporates a typical variation of a regularization functional, obtained after $\ell _2$ minimization of the corresponding energy functional from (2.3). Other types of norms should be explored and, more importantly, evaluation metrics to gauge their performance should be researched. In addition, more work is needed to determine superior regularization functionals that can effectively address the missing-data super-resolution problem. In this respect, compelling results may be achieved by adaptively adjusting the regularization parameter, $\lambda$, and fidelity term relative to the local image features.

3 Practical applications of image super-resolution

The super-resolution technology has revolutionized the imaging industry, providing some real-world applications to domestic and commercial devices. For instance, the technology has provided methods and techniques to manufacture inexpensive and portable imaging devices. The today’s generation has witnessed smartphones and miniature cameras that apply image processing techniques to capture high-quality images. Given its wide practical applications, the super-resolution imaging has remained an interesting research topic to date.

3.1 Face image super-resolution

In several practical applications, we desire high-quality face images with well-preserved and clear features [132, 133]. For example, surveillance images should display clear human faces of criminals to assist police officers in law enforcement. Another practical example can be observed in access control systems that use a face image for human recognition. Given these applications, there has been intensive research to ensure that imaging systems generate quality face images that meet the intended demands. One direction of research is face image super-resolution (also called face hallucination) that deals with increasing the resolution of a face image [50, 54, 129, 134]. Currently, machine learning approaches have demonstrated promising results in face image super-resolution [62, 66, 121].

3.2 Medical imaging

Medical images provide a cost-effective solution for doctors to make diagnosis on the disease and conditions of patients. The fundamental premise for drawing appropriate diagnostic decisions depends on the quality of a medical image. Therefore, the imaging modalities (e.g., X-ray, ultrasound, magnetic resonance imaging or MRI, computerized tomography or CT, and positron emission tomography or PET) should generate high-resolution images with distinctive medical features. To this end, image super-resolution has played a key role to improve the resolution of medical images [35, 36, 40, 57, 58, 61, 70, 78]. Specific applications of medical image super-resolution can be found in X-ray imaging [82, 86, 89, 92, 95], ultrasound imaging [36, 57, 98], MRI [102,103,104,105, 135], CT imaging [136,137,138,139,140], and PET imaging [70, 141,142,143,144].

3.3 Multispectral and hyperspectral imaging

The ordinary camera can capture images within the visible electromagnetic spectrum. Some applications, however, require utilization of other electromagnetic spectrum bands to reveal important features of objects. This demand compelled researchers to introduce multispectral and hyperspectral imaging methods that explore a broader range of the electromagnetic spectrum. Hyperspectral imagery generates images with higher spectral resolution compared with those generated by multispectral imagery. Nevertheless, these modes generate images with poor spatial resolution. Challenged by the limitation, efforts have been put to apply super-resolution techniques to improve the spatial resolution of multispectral and hyperspectral images [106, 107, 110, 145,146,147].

3.4 Synthetic-aperture radar imaging

Synthetic-aperture radar (SAR) [148, 149], an emerging technology in remote sensing, uses an imaging sensor for active data collection from the earth. During operation, the SAR sensor generates energy and transmits it to the earth. Afterwards, the sensor receives and records the reflected energy after interaction with the earth. SAR imaging, despite its wide applications [148,149,150,151], requires an expensive sensor to generate images with higher spatial resolution. Responding to the challenge, scholars have proposed super-resolution techniques that facilitate resolution enhancement without sensor modification—an approach that significantly lowers the overall cost of the imaging system [152, 153].

3.5 Astronomical imaging

Despite the considerable achievements in astrophotography (imaging of space objects) [154], diffraction limits and other technical challenges cause space telescopes to generate low-resolution images of astronomical objects and celestial events. Typical degradations in space images include noise, warping, and decimation. These degradations, if not addressed, may hinder the advancement of scientific exploration in astronomy. Therefore, super-resolution imaging has been considered as an optimal solution to simultaneously increase angular and spatial resolutions of astronomical images [128, 155,156,157].

3.6 Microscopy imaging

Microscopy allows scientists and researchers to observe microscopic objects (e.g., cell structures) and study complex biological processes using microscopes [158]. Because of technological challenges, super-resolution techniques have been proposed to increase the angular resolution of microscopic objects beyond the diffraction limit [33, 49, 108, 159,160,161,162,163]. Techniques for super-resolution microscopy include stimulated emission depletion (STED) microscopy [65, 69, 109], structured illumination microscopy (SIM) [73, 81, 164], stochastic optical reconstruction microscopy (STORM)/photoactivation localization microscopy (PALM) [165,166,167,168], Fourier ptychographic microscopy (FPM) [169, 170], and super-oscillation microscopy (SOM) [171, 172]. These techniques use different mechanisms to overcome optical limitations (scattering, reflection, diffraction, attenuation, and absorption), hence advancing the scientific inquiry of biological processes. For example, the advancement of super-resolution microscopy has greatly improved our understanding on animal and plant cells.

3.7 Multimedia industry and video enhancement

In recent years, there has been increasing demands for high-quality scenes in the multimedia industry. People desire to watch high definition videos (e.g., movies), animations, and visual effects for entertainment or other multimedia applications. Therefore, motivated by the sophistication in computing, researchers have proposed different super-resolution methods to improve the resolution of images and videos [173, 174]. Such methods may be embedded into computing devices, such as smartphones and tablets, to give users the deserved experience.

3.8 Biometrics

Super-resolution imaging may be applied to enhance the resolution of biometrics features [17], such as fingerprint [175,176,177], iris [178, 179], and palm veins [180]. This image enhancement procedure, usually implemented as a pre-processing component, may help to improve the accuracy of biometric identification system. For example, super-resolution algorithms may be embedded in a smartphone to enhance the quality of compact fingerprint signatures captured by the sensor.

3.9 Electronics manufacturing industries

Fabrication of printed circuit boards (PCBs) using vision-driven systems involves several steps, including detection of defects (e.g., broken electrical circuits or contacts) from PCB surfaces [181,182,183]. This step becomes rather challenging for tiny defects, calling for a need of high-quality PCB images. Advanced cameras may address this challenge at the expense of increased hardware cost. Subsequently, super-resolution imaging techniques may be used to improve the resolution of PCB images, especially in defective regions of the boards. There has been some little attempts to apply super-resolution algorithms to improve the fabrication processes of PCBs [184, 185].

4 Evaluation of super-resolution methods

4.1 Image quality metrics

Performance of the super-resolution method can be determined by gauging the quality of the images that such a method generates. Traditionally, subjective and objective metrics have been used for performance evaluation [186]. In recent years, scholars have attempted to apply machine learning approaches in the quality assessment of the super-resolved images [187, 188].

Subjective image quality assessment involves visual inspection to investigate features of the image [189]. Results from this method depends on the perceptual abilities of the human, driven by several physiological and psychological factors. For the same image, people can provide different perceptions on its quality. Therefore, a clear methodology should be devised before using the subjective IQA. One approach could be to develop an instrument, such as a questionnaire or an interview guide, and visit groups of people to provide their opinions on the perceptual qualities of the images. Then, the responses can be analyzed to provide statistical values that will form the basis for drawing conclusions regarding the visual appeal of the images.

Three approaches of subjective IQA may be applied to evaluate the super-resolution methods [189]: firstly, categorical rating where an observer judges the quality of a single image (single stimulus) or a pair of images (double stimulus) based on a fixed five-point scale; secondly, forced-choice that requires an observer to perform pairwise comparison of images, then order them from highest to lowest quality; and thirdly, similarity judgement that, in addition to the observer selecting an image with the highest quality, estimates the image quality difference on a continuous scale.

The subjective IQA approach, if carefully performed, may provide promising results consistent with the human visual system. Our investigation from the literature reveals that authors tend to apply their personal experiences to subjectively evaluate the quality of the images, and this tendency provides biased conclusions that disregard opinions from a wider population.

In objective IQA, the quality of an image is quantified numerically. This evaluation metric provides a universal standard in assessing the quality of a super-resolved image. There exists three common types of objective IQA methods: full-reference, no-reference, and reduced reference. Each IQA method gives a number that shows the degree of deviation between the images under comparison.

In full-reference IQA, the restored (super-resolved) image is compared with the given reference (ideal) image. The limitation of this metric is that it requires a reference image that may not always be available. Examples of full-reference IQA include the following [190]: mean absolute error [191], mean squared error [192], peak signal-to-noise ratio [193,194,195], structural similarity [186, 192, 196], visual information fidelity [197], most apparent distortion [198], feature similarity [199], gradient magnitude similarity deviation [200], visual saliency induced [201], normalized Laplacian pyramid distance [202], learned perceptual image patch similarity [203], and deep image structure and texture similarity [204].

The no-reference IQA metric does not require a reference image to quantify its quality [205]. This metric may be suitable in situations where only information of the restored (or degraded) image is available, such as in single-frame super-resolution imaging. Because the metric requires only a single test image, robust methods are usually needed to estimate the statistical information, such as noise and probability density functions, in the image. Examples of the no-reference IQA metrics include the following [205]: blind image integrity notator using discrete cosine transform statistics [206], blind multiple pseudo reference image [207], blind/referenceless image spatial quality evaluator [208], curvelet quality assessment [209], distortion identification-based image verity and integrity evaluation [210], entropy-based no-reference IQA [211], blind IQA [212], novel-blind IQA [213], spatial-spectral entropy-based quality index [209], no-reference perception-based image quality evaluator [214], no-reference IQA [215], and oriented gradients IQA index [216].

The reduced-reference IQA metric evaluates the perceptual quality of an image with respect to the partial information of a reference image [217]. Examples of this metric include wavelet marginal index [218], divisive normalization transform marginal index [219], reduced-reference structural similarity [220], wavelet reduced-reference IQA index [221], and feature-based reduced-reference IQA index  [222].

4.2 Datasets and implementation codes

The best practice when developing a super-resolution method, or any other image and video processing method, is to share the datasets and implementation codes of the developed method to the public repository. This practice, supported by the open science (movement that promotes accessibility of scientific research) [223], allows researchers to reproduce results from authors’ works. Studies show that publications linked to open datasets and implementation codes receive more citations⁵ [224], an observation that translates to a significant research impact across the community.

Supporting the open science, the current work includes links and publications with open datasets and implementation codes of the super-resolution methods (Appendices A and B). We believe that this information may be useful to researchers, especially those in their early career of publication, to quickly benchmark their methods. Our belief, founded by the literature on open science, is that the research community should strive to advance science through dissemination of results and associated datasets.

5 Latest developments of super-resolution imaging

Super-resolution imaging has continued to be advanced for its immense domestic and industrial applications. Scientists envisage to stimulate their understanding on microscopic objects. This quest for new knowledge may be critical in the development of science and technology.

In optical microscopy, there has been struggles by scientists to further improve label-free super-resolution (LFSR) imaging [225, 226], which employs principles of light scattering in nanoscale materials for spatial resolution enhancement. LFSR has demonstrated remarkable achievements in microbiology to study cellular, molecular, and genetic processes from plants and animals. An interesting article by Astratov et al. [225] provides a roadmap of LFSR imaging, exposing current and future developments of this promising field in biomedical imaging. Furthermore, scholars have been investigating the impact of integrating optical microscopy and image post-processing techniques to address diffraction limits in optical systems [20, 31, 227].

Deep (and machine) learning gives us a promising future of super-resolution imaging. Scholars have established different learning models and techniques with high performance to extend resolution of images and videos [228,229,230,231]. It may be important for scholars to investigate the impact of combining deep learning techniques and optical microscopy.

Perhaps an area that still needs intensive scientific inquiry is the implementation of super-resolution imaging algorithms in practical electronics devices. This research direction has received little attention because of several hardware and software limitations [232], including computational issues and compression artifacts. Electronics manufacturing (and semiconductor) industries may develop dedicated hardware for real-time processing of complex super-resolution algorithms. On $28^{\textrm{th}}$ February 2023, Nvidia responded to the challenge by releasing a RTX Video Super-resolution driver for their GeForce RTX 40 and 30 Series Graphics Processing Units.⁶ The innovation has allowed for streaming of high-quality (super-resolved) videos content in Google Chrome and Microsoft Edge browsers.

6 Conclusion

This work has tracked the developments and achievements of the image super-resolution technology over the last seventy years. Challenges and potential opportunities have been provided for researchers to further advance this technology. One notable observation from our work is that the super-resolution technology, despite being in existence for over 70 years, has unsatisfactorily made its way to practical devices. Several super-resolution methods have been developed but not directly applied in the real-world environment, partly due to complexity and memory demands of such methods. Therefore, the super-resolution problem seems to remain an active research area for several years ahead.

Appendices

Appendix A: Datasets for testing super-resolution methods

• 1. University of Southern Califonia: https://sipi.usc.edu/database/

• 2. DIV2K Dataset: https://data.vision.ee.ethz.ch/cvl/DIV2K/

• 3. Peyman Milanfar: www.soe.ucsc.edu/~milanfar/DataSets/

• 4. Flickr1024: https://yingqianwang.github.io/Flickr1024/

• 5. SISR: https://cvnote.ddlee.cc/2019/09/22/image-super-resolution-datasets

• 6. RELLISUR: https://openreview.net/forum?id=aqCD8RINP54

• 7. SelfExSR: https://github.com/jbhuang0604/SelfExSR

• 8. PROBA-V: https://kelvins.esa.int/proba-v-super-resolution/

Appendix B: Implementation codes for super-resolution methods

• 1. https://ccia.ugr.es/pi/superresolution/software.html

• 2. https://www.ece.lsu.edu/ipl/Software.html

• 3. https://faculty.idc.ac.il/toky/old_courses/videoProc-07/projects/SuperRes/srproject.html

• 4. http://www.ok.sc.e.titech.ac.jp/res/CSR/MTSR/index.html

• 5. https://www.mathworks.com/matlabcentral/fileexchange/30488-superresolution-demo

• 6. https://www.mathworks.com/matlabcentral/fileexchange/49538-superresolutiondemo

• 7. http://staff.utia.cas.cz/sroubekf/research/bsr_gui.html

• 8. http://soellerlab.ex.ac.uk/pages/PYME.html

• 9. https://github.com/sairajk/Image-Super-Resolution-Application

• 10. https://yapengtian.org/

• 11. http://zoi.utia.cas.cz/mobilesr

• 12. http://people.rennes.inria.fr/Aline.Roumy/results/SR_BMVC12.html

• 13. http://mmlab.ie.cuhk.edu.hk/projects/FSRCNN.html

• 14. https://www.mathworks.com/matlabcentral/fileexchange/33839-image-super-resolution-iterative-back-projection-algorithm

• 15. https://github.com/topics/super-resolution?l=matlab

• 16. https://github.com/topics/single-image-super-resolution

• 17. https://www.robots.ox.ac.uk/~vgg/software/SR/

• 18. http://mmlab.ie.cuhk.edu.hk/projects/SRCNN.html

• 19. https://github.com/jspan/PHYSICS_SR

• 20. https://matlab1.com/shop/matlab-code/matlab-code-high-resolution-image-set-low-resolution-images/

• 21. https://elad.cs.technion.ac.il/software/

• 22. http://freesourcecode.net/matlabprojects/59355/image-super-resolution---iterative-back-projection-algorithm-in-matlab#.Ym5JVdNBzIU

• 23. https://jiaya.me/research/

• 24. https://www.vision.uji.es/srtoolbox/

• 25. https://compphotolab.northwestern.edu/project/spatial-spectral-representation-for-x-ray-fluorescence-image-super-resolution/

• 26. https://xinli.faculty.wvu.edu/reproducible-research/reproducible-research-in-image-processing