Pavement cracks detection and classification using deep convolutional networks

Document Type : Research Article

Authors

1 Department of Civil & Environmental Engineering, Amirkabir University of Technology

2 RESEARCHER /AUT

Abstract

Pavement inspection is one of the most important steps in the implementation of the pavement management system and extend efforts have been conducted to increase the efficiency of this system by using new technologies. In recent years, transportation agencies focus on creating automatic and more efficient systems for pavement inspection and a large number of researches have been done for this aim. According to the progress of computer science, data mining and machine learning as computer-based methods are used more in various areas (such as engineering, medical and economy), and significant results have been achieved. In the pavement management area, several researches have been performed to apply the machine learning, especially in pavement distresses evaluation. In this paper, the theoretical concepts have been explained, and several models have been created based on deep convolutional networks using transfer learning to detect and classify pavement cracks as the most prevalent pavement distress, and the performance of these models has been evaluated considering learning and test speed, and accuracy as the most important performance parameters. The results of this research indicate that the speed of models almost depends on the characteristics of pre-trained models that applied in the transfer learning process. Also, the accuracy of models based on various metrics (Sensitivity, F-score, etc.) is in range of 0.94 to 0.99 and indicates that deep learning method can be used to create expert systems for detection, classification, and quantification of pavement distresses such as cracking.

Keywords

Main Subjects

References

[1]Y. Shi, S. Guo, P. Sun, The role of infrastructure in China’s regional economic growth, Journal of Asian Economics, 49 (2017) 26-41.
[2]E. Ivanova, J. Masarova, Importance of road infrastructure in the economic development and competitiveness, Economics Management, 18(2) (2013) 263-274.
[3]R. Engström, The Roads’ Role in the Freight Transport System, Transportation Research Procedia, 14 (2016) 1443-1452.
[4]C.Y. Chan, B. Huang, X. Yan, S. Richards, Investigating effects of asphalt pavement conditions on traffic accidents in Tennessee based on the pavement management system )PMS(, Journal of advanced transportation, 44(3) (2010) 150-161.
[5]ASCE, 2017 infrastructure report card, Roads, ASCE Reston, VA, 2017.
[6]M.Y. Shahin, Pavement management for airports, roads, and parking lots, 1994.
[7]H. Zakeri, F.M. Nejad, A. Fahimifar, Image Based Techniques for Crack Detection, Classification and Quantification in Asphalt Pavement: A Review, Archives of Computational Methods in Engineering, 24(4) (2017) 935-977.
[8]H. Zakeri, F.M. Nejad, A. Fahimifar, Rahbin: A quadcopter unmanned aerial vehicle based on a systematic image processing approach toward an automated asphalt pavement inspection, Automation in Construction, 72(Part 2) (20160 211-235.
[9]C. Koch, K. Georgieva, V. Kasireddy, B. Akinci, P. Fieguth, A review on computer vision based defect detection and condition assessment of concrete and asphalt civil infrastructure, Advanced Engineering Informatics, 29(2) (2015) 196-210.
[10]T.B. Coenen, A. Golroo, A review on automated pavement distress detection methods, Cogent Engineering, 4(1) (2017) 1374822.
[11]N.-D. Hoang, An Artificial Intelligence Method for Asphalt Pavement Pothole Detection Using Least Squares Support Vector Machine and Neural Network with Steerable Filter-Based Feature Extraction, Advances in Civil Engineering, 2018 (2018) 12.
[12]P. Wang, Y. Hu, Y. Dai, M. Tian, Asphalt Pavement Pothole Detection and Segmentation Based on Wavelet Energy Field, Mathematical Problems in Engineering, 2017 (2017) 13.
[13]B. Mataei, F. Moghadas Nejad, M. Zahedi, H. Zakeri, Evaluation of pavement surface drainage using an automated image acquisition and processing system, Automation in Construction, 86 (2018) 240-255.
[14]F.M. Nejad, N. Karimi, H. Zakeri, Automatic image acquisition with knowledge-based approach for multidirectional determination of skid resistance of pavements, Automation in Construction, 71(Part 2) (2016) 414-429.
[15]K. Kamal, S. Mathavan, T. Zafar, I. Moazzam, A. Ali, S.U. Ahmad, M. Rahman, Performance assessment of Kinect as a sensor for pothole imaging and metrology, International Journal of Pavement Engineering, 19(7) (2018) 565-576.
[16]Y.-C. Tsai, A. Chatterjee, Pothole Detection and Classification Using 3D Technology and Watershed Method, Journal of Computing in Civil Engineering, 32(2) (2017) 04017078.
[17]J.Y.-C. Tsai, Z.-H. Wang, F. Li, Assessment of rut depth measurement accuracy of point-based rut bar systems using emerging 3d line laser imaging technology, Journal of Marine Science and Technology, 23)3( )2015( 322-330.
[18]J.Y.-C. Tsai, F. Li, Y.-C. Wu, A new rutting measurement method using emerging 3D line-laser-imaging system, International Journal of Pavement Research and Technology, 6(5) (2013) 667-672.
[19]S. Dai, K. Hoegh, 3D step frequency GPR Asphalt pavement stripping detection: Case study evaluating filtering approaches, in:  Advanced Ground Penetrating Radar (IWAGPR), 2017 9th International Workshop on, IEEE, 2017, pp. 1-7.
[20]S. Li, C. Yuan, D. Liu, H. Cai, Integrated processing of image and GPR data for automated pothole detection, Journal of computing in civil engineering, 30(6) (2016) 04016015.
[21]M. Solla, S. Lagüela, H. González-Jorge, P. Arias, Approach to identify cracking in asphalt pavement using GPR and infrared thermographic methods: Preliminary findings, NDT & E International, 62 (2014) 55-65.
[22]H. Song, K. Baek, Y. Byun, Pothole detection using machine learning, Advanced science and technology letters, 150 (2018) 151-155.
[23]M.R. Carlos, M.E. Aragón, L.C. González, H.J. Escalante, F. Martínez, Evaluation of Detection Approaches for
Road Anomalies Based on Accelerometer Readings-Addressing Who's Who, IEEE Transactions on Intelligent Transportation Systems,  (20180.
[24]A. Fox, B.V. Kumar, J. Chen, F. Bai, Multi-lane pothole detection from crowdsourced undersampled vehicle sensor data, IEEE Transactions on Mobile Computing, 16(12) (2017) 3417-3430.
[25]S. Nakashima, S. Aramaki, Y. Kitazono, S. Mu, K. Tanaka, S. Serikawa, Application of ultrasonic sensors in road surface condition distinction methods, Sensors, 16(10) (2016) 1678.
[26]A. Bystrov, E. Hoare, T.-Y. Tran, N. Clarke, M. Gashinova, M. Cherniakov, Road surface classification using automotive ultrasonic sensor, Procedia Engineering, 168 (2016) 19-22.
[27]R. Madli, S. Hebbar, P. Pattar, V. Golla, Automatic detection and notification of potholes and humps on roads to aid drivers, IEEE Sensors Journal, 15(8) (2015) 4313-4318.
[28]J. Mehta, V. Mathur, D. Agarwal, A. Sharma, K. Prakasha, Pothole Detection and Analysis System (Pol) AS( for Real Time Data Using Sensor Networks, Journal of Engineering and Applied Sciences, 12(12) (2017) 3090-3097.
[29]J. Huang, W. Liu, X. Sun, A pavement crack detection method combining 2D with 3D information based on Dempster‐Shafer theory, Computer‐Aided Civil Infrastructure Engineering, 29(4) (2014) 299-313.
[30]G. Zhou, L. Wang, D. Wang, S. Reichle, Integration of GIS and data mining technology to enhance the pavement management decision making, Journal of Transportation Engineering, 136(4) (2009) 332-341.
[31]K. Gopalakrishnan, Deep Learning in Data-Driven Pavement Image Analysis and Automated Distress Detection: A Review, Data, 3(3) (20180 28.
[32]Y. LeCun, Y. Bengio, G. Hinton, Deep learning, Nature, 521 (2015) 436.
[33]C. Robert, Machine Learning, a Probabilistic Perspective, CHANCE, 27(2) (2014) 62-63.
[34]N.K. Warrier, K. Sathish, Object Detection on Roads using Deep Learning and Neural Networks, Journal of Network Communications Emerging Technologies 8(4) (2018).
[35]M. Kantardzic, Data mining: concepts, models, methods, and algorithms, John Wiley & Sons, 2011.
[36]K. Gopalakrishnan, S.K. Khaitan, A. Choudhary, A. Agrawal, Deep Convolutional Neural Networks with transfer learning for computer vision-based data-driven pavement distress detection, Construction and Building Materials, 157 (2017) 322-330.
[37]L. Deng, D. Yu, Deep learning: methods and applications, Foundations Trends® in Signal Processing, 7(3–4) (2014) 197-387.
[38]Y. LeCun, Y. Bengio, G.J.n. Hinton, Deep learning, 521(7553) (2015) 436.
[39]S. Dorafshan, R.J. Thomas, M. Maguire, Comparison of deep convolutional neural networks and edge detectors for image-based crack detection in concrete, Construction and Building Materials, 186 (2018) 1031-1045.
[40]Z. Tong, J. Gao, Z. Han, Z. Wang, Recognition of asphalt pavement crack length using deep convolutional neural networks, Road Materials and Pavement Design, 19(6) (2018) 1334-1349.
[41]Q. Zhu, Pavement crack detection algorithm Based on image processing analysis, in:  Proceedings - 2016 8th International Conference on Intelligent Human-Machine Systems and Cybernetics, IHMSC 2016, 2016, pp. 15-18.
[42]Y.O. Ouma, M. Hahn, Wavelet-morphology based detection of incipient linear cracks in asphalt pavements from RGB camera imagery and classification using circular Radon transform, Advanced Engineering Informatics, 30(3) (2016) 481-499.
[43]V. Ananth, P. Ananthi, V. Elakkiya, J. Priyadharshini, R. Shiyamili, Automatic Pavement Crack Detection Algorithm, International Innovative Research Journal of Engineering and Technology, 2 (2017).
[44]I.H. Witten, E. Frank, M.A. Hall, C.J. Pal, Data Mining: Practical machine learning tools and techniques, Morgan Kaufmann, 2016.
[45]J. Han, J. Pei, M. Kamber, Data mining: concepts and techniques, Elsevier, 2011.
[46]F. Gorunescu, Data Mining: Concepts, models and techniques, Springer Science & Business Media, 2011.
[47]E. Alpaydin, Introduction to machine learning, MIT press, 2009.
[48]A.T. Azar, S. Vaidyanathan, Computational intelligence applications in modeling and control, Springer, 2015.
[49]S.B. Kotsiantis, I. Zaharakis, P. Pintelas, Supervised machine learning: A review of classification techniques, 160 (2007) 3-24.
[50]I. Goodfellow, Y. Bengio, A. Courville, Y. Bengio, Deep learning, MIT press Cambridge, 2016.
[51]M.M. Najafabadi, F. Villanustre, T.M. Khoshgoftaar, N. Seliya, R. Wald, E. Muharemagic, Deep learning applications and challenges in big data analytics, Journal of Big Data, 2(10 (2015) 1.
[52]R. Vidal, J. Bruna, R. Giryes, S. Soatto, Mathematics of deep learning, arXiv preprint arXiv:.04741, (2017).
[53]T. Wiatowski, H. Bölcskei, A mathematical theory of deep convolutional neural networks for feature extraction, IEEE Transactions on Information Theory, 64(3) (2018) .6681-5481
[54]A. Bhandare, M. Bhide, P. Gokhale, R. Chandavarkar, Applications of Convolutional Neural Networks, International Journal of Computer Science Information Technologies, (2016) 2206-2215.
[55]S. Albelwi, A. Mahmood, A framework for designing the architectures of deep convolutional neural networks, Entropy, 19(6) (2017) 242.
[56]Y. LeCun, Y. Bengio, Convolutional networks for images, speech, and time series, The handbook of brain theory neural networks, 3361(10) (1995) 1995.
[57]K. Zhang, H. Cheng, B. Zhang, Unified Approach to Pavement Crack and Sealed Crack Detection Using Preclassification Based on Transfer Learning, Journal of Computing in Civil Engineering, 32(2) (2018) 04018001.
[58]T. Wang, K. Gopalakrishnan, O. Smadi, A.K.J.T. Somani, Automated shape-based pavement crack detection approach, 33(3) (2018) 598-608.
[59]D. Seichter, M. Eisenbach, R. Stricker, G. H-M, How to Improve Deep Learning based Pavement Distress Detection while Minimizing Human Effort, in:  Proc. Int. Conf. on, 2018, pp. 63-70.
[60]H. Maeda, Y. Sekimoto, T. Seto, T. Kashiyama, H. Omata, Road Damage Detection Using Deep Neural Networks with Images Captured Through a Smartphone, arXiv preprint arXiv:1801.09454,  (2018).
[61]B. Li, K.C. Wang, A. Zhang, E. Yang, G. Wang, Automatic classification of pavement crack using deep convolutional neural network, International Journal of Pavement Engineering,  (2018) 1-7.
[62]G. Ciaparrone, A. Serra, V. Covito, P. Finelli, C.A. Scarpato, R. Tagliaferri, A Deep Learning Approach for Road Damage Classification, in:  Advanced Multimedia and Ubiquitous Engineering, Springer, 2018, pp. 655-661.
[63]Z. Tong, J. Gao, H. Zhang, Recognition, location, measurement, and 3D reconstruction of concealed cracks using convolutional neural networks, Construction and Building Materials, 146 (2017) 775-787.
[64]Y. Liu, J. Yao, X. Lu, R. Xie, L. Li, DeepCrack: A Deep Hierarchical Feature Learning Architecture for Crack Segmentation, Neurocomputing,  (2019).
[65]J. Singh, S. Shekhar, Road Damage Detection And Classification In Smartphone Captured Images Using Mask R-CNN, arXiv,  (2018).
[66]K. Gopalakrishnan, H. Gholami, A. Vidyadharan, A. Choudhary, A. Agrawal, Crack damage detection in unmanned aerial vehicle images of civil infrastructure using pre-trained deep learning model, International Journal for Traffic Transport Engineering, 8 (2018) 1.
[67]Z. Tong, J. Gao, H. Zhang, Innovation for evaluating aggregate angularity based upon 3D convolutional neural network, Construction and Building Materials, 155 (2017) 919-929.
[68]M.A. Nielsen, Neural networks and deep learning, Determination press USA, 2015.
[69]H. Eom, H. Choi, Alpha-Pooling for Convolutional Neural Networks, arXiv preprint arXiv:.03436,  (2018).
[70]D.C. Ciresan, U. Meier, J. Masci, L. Maria Gambardella, J. Schmidhuber, Flexible, high performance convolutional neural networks for image classification, in:  IJCAI Proceedings-International Joint Conference on Artificial Intelligence, Barcelona, Spain, 2011, pp. 1237.
[71]L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, A. Talwalkar, Hyperband: A novel bandit-based approach to hyperparameter optimization, The Journal of Machine Learning Research, 18(1) (2017) 6765-6816.
[72]S.R. Young, D.C. Rose, T.P. Karnowski, S.-H. Lim, R.M. Patton, Optimizing deep learning hyper-parameters through an evolutionary algorithm, in:  Proceedings of the Workshop on Machine Learning in High-Performance Computing Environments, ACM, 2015, pp. 4.
[73]T. Domhan, J.T. Springenberg, F. Hutter, Speeding Up Automatic Hyperparameter Optimization of Deep Neural Networks by Extrapolation of Learning Curves, in:  IJCAI, 2015, pp. 3460-3468.
[74]J. Snoek, H. Larochelle, R.P. Adams, Practical bayesian optimization of machine learning algorithms, in: Advances in neural information processing systems, 2012, pp. 2951-2959.
[75]Y. Gao, K.M. Mosalam, Deep Transfer Learning for Image-Based Structural Damage Recognition, 33(9) (2018) 748-768.
[76]K. Zhang, H. Cheng, A Novel Pavement Crack Detection Approach Using Pre-selection Based on Transfer Learning, in: Y. Zhao, X. Kong, D. Taubman (Eds.) Image and Graphics, Springer International Publishing, Cham, 2017, pp. 273-283.
[77]S.J. Pan, Q. Yang, A survey on transfer learning, IEEE Transactions on knowledge data engineering, 22(10) (2010) 1345-1359.
[78]O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, Imagenet large scale visual recognition challenge, International Journal of Computer Vision, 115(3) (2015) 211-252.
[79]A. Krizhevsky, I. Sutskever, G.E. Hinton, Imagenet classification with deep convolutional neural networks, in:  Advances in neural information processing systems, 2012, pp. 1097-1105.
[80]C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A. Rabinovich, Going deeper with convolutions, in:  Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 1-9.
[81]K. He, X. Zhang, S. Ren, J. Sun, Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, in:  Proceedings of the IEEE international conference on computer vision, 2015, pp. 1026-1034.
[82]F.N. Iandola, S. Han, M.W. Moskewicz, K. Ashraf, W.J. Dally, K.J.a.p.a. Keutzer, Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size,  (2016).
[83]K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in:  Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770-778.
[84]G. Huang, Z. Liu, L. Van Der Maaten, K.Q. Weinberger, Densely connected convolutional networks, in:  CVPR, 2017, pp. 3.
[85]C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, Z. Wojna, Rethinking the inception architecture for computer vision, in:  Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2818-2826.
[86]F.M. Nejad, H. Zakeri, An optimum feature extraction method based on Wavelet–Radon Transform and Dynamic Neural Network for pavement distress classification, Expert Systems with Applications, 38(8) (2011) 9442-9460.
[87]F. Moghadas Nejad, H. Zakeri, An expert system based on wavelet transform and radon neural network for pavement distress classification, Expert Systems with Applications, 38(6) (2011) 7088-7101.
[88]R.C. Gonzalez, R.E. Woods, Digital image processing, 2 (2007).
Amirkabir Journal of Civil Engineering
Volume 52, Issue 9
December 2020
Pages 2255-2278

Files

Share

How to cite

Statistics