Skip to main content

Design of Experiments for Performance Evaluation and Parameter Tuning of a Road Image Processing Chain


Tuning a complete image processing chain (IPC) is not a straightforward task. The first problem to overcome is the evaluation of the whole process. Until now researchers have focused on the evaluation of single algorithms based on a small number of test images and ad hoc tuning independent of input data. In this paper, we explain how the design of experiments applied on a large image database enables statistical modeling for IPC significant parameter identification. The second problem is then considered: how can we find the relevant tuning and continuously adapt image processing to input data? After the tuning of the IPC on a typical subset of the image database using numerical optimization, we develop an adaptive IPC based on a neural network working on input image descriptors. By testing this approach on an IPC dedicated-to-road obstacle detection, we demonstrate that this experimental methodology and software architecture can ensure continuous efficiency. The reason is simple: the IPC is globally optimized, from a large number of real images and with adaptive processing of input data.


  1. 1.

    Haralick RM: Performance characterization protocol in computer vision. Proceedings of the ARPA Image Understanding Workshop, November 1994, Monterey, Calif, USA I: 667–673.

    Google Scholar 

  2. 2.

    Courtney P, Thacker N, Clark A: Algorithmic modeling for performance evaluation. Proceedings of the ECCV Workshop on Performance Characteristics of Vision Algorithms, April 1996, Cambridge, UK 13.

    Google Scholar 

  3. 3.

    Forstner W: 10 pros and cons against performance characterization of vision algorithms. Proceedings of the ECCV Workshop on Performance Characteristics of Vision Algorithms, April 1996, Cambridge, UK

    Google Scholar 

  4. 4.

    Bowyer KW, Phillips PJ: Empirical Evaluation Techniques in Computer Vision. Wiley-IEEE Computer Society Press, Los Alamitos, Calif, USA; 1998.

    Google Scholar 

  5. 5.

    Meer P, Matei B, Cho K: Input guided performance evaluation. Theoretical Foundations of Computer Vision (TFCV '98), March 1998, Dagstuhl, Germany 115–124.

    Google Scholar 

  6. 6.

    Phillips IT, Chhabra AK: Empirical performance evaluation of graphics recognition systems. IEEE Transactions on Pattern Analysis and Machine Intelligence 1999, 21(9):849–870. 10.1109/34.790427

    Article  Google Scholar 

  7. 7.

    Blanc-Talon J, Ropert V: Evaluation des chaînes de traitement d'images. Revue Scientifique et Technique de la Défense 2000, (46):29–38.

  8. 8.

    Philipp-Foliguet S: Evaluation de la segmentation. ETIS, Cergy-Pontoise, France; 2001.

    Google Scholar 

  9. 9.

    Sebe N, Tian Q, Loupias E, Lew MS, Huang TS: Evaluation of salient point techniques. Proceedings of International Conference on Image and Video Retrieval (CIVR '02), July 2002, London, UK 2383: 367–377.

    Article  Google Scholar 

  10. 10.

    Rosin PL, Ioannidis E: Evaluation of global image thresholding for change detection. Pattern Recognition Letters 2003, 24(14):2345–2356. 10.1016/S0167-8655(03)00060-6

    Article  Google Scholar 

  11. 11.

    Yitzhaky Y, Peli E: A method for objective edge detection evaluation and detector parameter selection. IEEE Transactions on Pattern Analysis and Machine Intelligence 2003, 25(8):1027–1033. 10.1109/TPAMI.2003.1217608

    Article  Google Scholar 

  12. 12.

    Ropert V: Proposition d'une architecture de contrôle pour un système de vision, Thèse de l'Université René Descartes (Paris 6). , Paris, France; Décembre 2001.

    Google Scholar 

  13. 13.

    Levner I, Bulitko V: Machine learning for adaptive image interpretation. Proceedings of the 16th Innovative Applications of Artificial Intelligence Conference (IAAI '04), July 2004, San Jose, Calif, USA 870–876.

    Google Scholar 

  14. 14.

    Schimmerling P, Sisson J-C, Zaïdi A: Pratique des Plans d'Expériences. Lavoisier Tec & Doc, Paris, France; 1998.

    Google Scholar 

  15. 15.

    Treuillet S: Analyse de l'influence des paramètres d'une chaîne de traitements d'images par un plan d'expériences. 19e colloque GRETSI sur le traitement du signal et des images (GRETSI '03), September 2003, Paris, France

    Google Scholar 

  16. 16.

    Treuillet S, Driouchi D, Ribereau P:Ajustement des paramètres d'une chaîne de traitement d'images par un plan d'expériences fractionnaire. Traitement du Signal 2004, 21(2):141–155.

    Google Scholar 

  17. 17.

    Wright MH: The nelder-mead simplex method: recent theory and practice. Proceedings of the 16th International Symposium on Mathematical Programming (ISMP '97), August 1997, Lausanne, Switzerland

    Google Scholar 

  18. 18.

    Domingues A, Lucas Y, Baudrier D, Marché P: Détection et suivi d'objets en temps réel par un système embarqué multi capteurs. Proceedings of the 18th Symposium GRETSI on Signal and Image Processing (GRETSI '01), Septembre 2001, Toulouse, France

  19. 19.

    Domingues A: Système embarqué multicapteurs pour la détection d'obstacles routiers—Développement du prototype et réglage automatique de la chaîne de traitement d'images, Thèse de l'Université d'Orléans. , Orléans, France; Juillet 2004.

    Google Scholar 

  20. 20.

    Lucas Y, Domingues A, Boubal M, Marché P: Système de vision embarqué pour la détection d'obstacles routiers. Techniques de l'Ingénieur—Recherche & Innovation 2005, 9. IN-24

    Google Scholar 

  21. 21.

    Lamaty P: Opérateurs de niveau intermédiaire pour le traitement temps réel des images, Thèse de Doctorat. Thèse de l'Université de Cergy-Pontoise, Cergy-Pontoise, France; 2000.

    Google Scholar 

  22. 22.

    Lucas Y, Domingues A, Driouchi D, Marché P: Modeling, evaluation and control of a road image processing chain. Proceedings of the 14th Scandinavian Conference on Image Analysis (SCIA '05), June 2005, Joensuu, Finland 3540: 1076–1085.

    Google Scholar 

  23. 23.

    Fries A, Hunter WG:Minimum aberration designs. Technometrics 1980, 22(4):601–608. 10.2307/1268198

    MathSciNet  MATH  Google Scholar 

  24. 24.

    Rechtschaffner RL: Saturated fractions of 2n and 3n factorial designs. Technometrics 1967, 9: 569–575. 10.2307/1266195

    MathSciNet  Google Scholar 

  25. 25.

    Cheng H-D, Sun Y: A hierarchical approach to color image segmentation using homogeneity. IEEE Transactions on Image Processing 2000, 9(12):2071–2082. 10.1109/83.887975

    Article  Google Scholar 

Download references

Author information



Corresponding author

Correspondence to Yves Lucas.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Lucas, Y., Domingues, A., Driouchi, D. et al. Design of Experiments for Performance Evaluation and Parameter Tuning of a Road Image Processing Chain. EURASIP J. Adv. Signal Process. 2006, 048012 (2006).

Download citation


  • Input Image
  • Parameter Tuning
  • Image Database
  • Software Architecture
  • Numerical Optimization