N. N. Soboleva

ANALYSIS OF METAL MICROSTRUCTURE BY NEURAL NETWORKS, EXEMPLIFIED BY SEGMENTATION OF CARBIDES IN COMPOSITE COATINGS

The use of artificial neural networks in metal science to solve image analysis problems, in particular segmentation or classification of metal microstructures, includes 6 main stages: problem definition, dataset collection, model selection, model training, model evaluation, and integration with an existing workflow. The paper discusses these stages in detail, provides an example of their implementation for semantic segmentation of microstructures of composite coatings containing coarse primary carbides. The separation of carbides by a neural network makes it possible to automate the process of determining their volume fraction in the coating structure.

Acknowledgment: The work was performed under the state assignment for the IES UB RAS, theme No. 124020600045-0, and the IMP UB RAS, theme No. 121102900049-1 “Additivity”. The equip-ment of the Plastometriya shared research facilities center of the IES UB RAS was used to obtain the microscopic images.

References:

1. Saakov, D.V. Application of machine learning methods to optimize production processes in the metallurgical industry. Innovatsii i Investitsii, 2023, 5, 308–311. (In Russian).
2. Klyuev, A.V., Stolbov, V.Yu., Koptseva, N.V., and Efimova, Yu.Yu. Assessment of the strength characteristics of steels after thermomechanical treatment based on a neural network analysis of microstructures digital photographs. Chernye Metally, 2020, 6, 50–56. (In Russian).
3. Holm, E.A., Cohn, R., Gao, N., Kitahara, A.R., Matson, T.P., Lei B., and Yarasi S.R. Overview: computer vision and machine learning for microstructural characterization and analysis. Metallurgical and Materials Transactions A: Physical Metallurgy and Materials Science, 2020, 51 (12), 5985–5999. DOI: 10.1007/s11661-020-06008-4.
4. Gupta, S., Banerjee, A., Sarkar, J., Kundu, M., Sinha, S.K., Bandyopadhyay, N.R., and Ganguly, S. Modelling the steel microstructure knowledge for in-silico recognition of phases using machine learning. Materials Chemistry and Physics, 2020, 252, 123286. DOI: 10.1016/j.matchemphys.2020.123286.
5. Jang, J., Van, D., Jang, H., Baik, D.H., Yoo, S.D., Park, J., Mhin, S., Mazumder, J., and Lee, S.H. Residual neural network-based fully convolutional network for microstructure segmentation. Science and Technology of Welding and Joining, 2020, 25 (4), 282–289. DOI: 10.1080/13621718.2019.1687635.
6. Mulewicz, B., Korpala, G., Kusiak, J., and Prahl, U. Autonomous interpretation of the microstructure of steels and special alloys. Materials Science Forum, 2019, 949, 24–31. DOI: 10.4028/www.scientific.net/MSF.949.24.
7. DeCost, B.L., Francis, T., and Holm, E.A. Exploring the microstructure manifold: Image texture representations applied to ultrahigh carbon steel microstructures. Acta Materialia, 2017, 133, 30–40. DOI: 10.1016/j.actamat.2017.05.014.
8. De Albuquerque, V.H.C., Cortez, P.C., De Alexandria, A.R., and Tavares, J.M.R.S. A new solution for automatic microstructures analysis from images based on a backpropagation artificial neural network. Nondestructive Testing and Evaluation, 2008, 23 (4), 273–283. DOI: 10.1080/10589750802258986.
9. Kondo, R., Yamakawa, S., Masuoka, Y., Tajima, S., and Asahi, R. Microstructure recognition using convolutional neural networks for prediction of ionic conductivity in ceramics. Acta Materialia, 2017, 141, 29–38. DOI: 10.1016/j.actamat.2017.09.004.
10. Baskaran, A., Kautz, E.J., Chowdhary, A., Ma, W., Yener, B., and Lewis, D.J. Machine learning for microstructure characterization and materials design: a perspective. JOM, 2021, 73 (11), 3639−3657. DOI: 10.1007/s11837-021-04805-9.
11. Kovun, V.A. and Kashirina, I.L. Usage of U-Net and W-Net neural network architecture for steel sample metallographic analysis. Vestnik VGU, Seriya Sistemnyi Analiz i Informatsionnye Tekhnologii, 2022, 1, 101–110. (In Russian). DOI: 10.17308/sait.2022.1/9205.
12. Wang, X.-d., Li, N., Su, H., and Meng, H.-m. Prior austenite grain boundary recognition in martensite microstructure based on deep learning. Journal of Iron and Steel Research International, 2023, 30, 1050–1056. DOI: 10.1007/s42243-023-00947-z.
13. Sharybin, S.I. Avtomatizatsiya kontrolya kachestva funktsionalnykh materialov na osnove sredstv intellektualnoy podderzhki prinyatiya resheniy [Automation of Quality Control of Functional Materials Based on Intellectual Decision Support Tools: Cand. Thesis]. Perm, 2015, 162 p. (In Russian).
14. Xie, L., Li, W., Fan, L., and Zhou, M. Automatic identification of the multiphase microstructures of steels based on ASPP-FCN. Steel Research International, 2023, 94, 2200204. DOI: 10.1002/srin.202200204.
15. Rose, D., Forth, J., Henein, H., Wolfe, T., and Qureshi, A.J. Automated semantic segmentation of NiCrBSi-WC optical microscopy images using convolutional neural networks. Computational Materials Science, 2022, 210, 111391. DOI: 10.1016/j.commatsci.2022.111391.
16. Sudyin, V.V. Issledovanie osobennostey razrusheniya nizkolegirovannykh staley i ikh svarnykh soedineniy v intervale vyazko-khrupkogo perekhoda [Investigation of the Features of Destruction of low-Alloy Steels and Their Welded Joints in the Interval of Ductile-Brittle Transition: Cand. Thesis]. Moscow, 2021, 189 p. (In Russian).
17. Krizhevsky, A., Sutskever, I., and Hinton, G.E. ImageNet classification with deep convolutional neural networks. Communications of the ACM, 2017, 60 (6), 84−90. DOI: 10.1145/3065386.
18. Lukashik, D.V. Analysis of modern image segmentation methods. Ekonomika i Kachestvo Sistem Svyazi, 2022, 2 (24), 57−65. (In Russian).
19. Chen, Q., Zheng, C., Cui, Y., Lin, Y.-R., and Zinkle, S.J. A deep learning model for automatic analysis of cavities in irradiated materials. Computational Materials Science, 2023, 221, 112073. DOI: 10.1016/j.commatsci.2023.112073.
20. Hurtado, J.V. and Valada, A. Semantic scene segmentation for robotics. Ch. 12. In: Deep Learning for Robot Perception and Cognition, Academic Press, 2022, 279–311. DOI: 10.1016/B978-0-32-385787-1.00017-8.
21. Xu, H. and Huang, H. Plasma remelting and injection method for fabricating metal matrix composite coatings reinforced with tungsten carbide. Ceramics International, 2022, 48, 2645–2659. DOI: 10.1016/j.ceramint.2021.10.048.
22. ASTM E 562–02. Standard test method for determining volume fraction by systematic manual point count.
23. https://siams.com/
24. Kazakov, A.A. and Kiselev, D. Industrial application of thixomet image analyzer for quantitative description of steel and alloy’s microstructure. Metallography, Microstructure, and Analysis, 2016, 5, 294–301. DOI: 10.1007/s13632-016-0289-6.
25. Schneider, C.A., Rasband, W.S., and Eliceiri, K.W. NIH Image to ImageJ: 25 years of image analysis. Nature Methods, 2012, 9, 671–675. DOI: 10.1038/nmeth.2089.
26. https://jmicrovision.github.io/
27. Jacobs, R. Deep learning object detection in materials science: current state and future directions. Computational Materials Science, 2022, 211, 111527. DOI: 10.1016/j.commatsci.2022.111527.
28. Benjdira, B., Bazi, Y., Koubaa, A, and Ouni, K. Unsupervised domain adaptation using generative adversarial networks for semantic segmentation of aerial images. Remote Sensing, 2019, 11, 1369. DOI: 10.3390/rs11111369.
29. Wang, M., Wu, F., and Zhao, J. A comprehensive research and strategy of transfer learning for image segmentation. In: Lecture Notes on Data Engineering and Communications Technologies book series, Springer, Cham, 2021, 88, 1394–1406. DOI: 10.1007/978-3-030-70665-4_152.
30. Aristov, G.V. and Klyuev, A.V. Recognition and classification of the microstructure of metals and alloys using deep neural networks. In: Trudy XXVII Mezhdunarodnoj konferentsii po kompyuternoy grafike i mashinnomu zreniyu [Proceedings of the 27th International Conference on Computer Graphics and Machine Vision GraphiCon, Perm, September 24–28, 2017]. PGNIU Publ., Perm, 2017, pp. 180−183. (In Russian).
31. Poletaeva, N.G. Classification of systems machine learning. Vestnik Baltijskogo Federalnogo Universiteta im. I. Kanta. Seriya: Fiziko-Matematicheskie i Tekhnicheskie Nauki, 2020, 1, 5−22. (In Russian).
32. https://habr.com/ru/companies/smartengines/articles/264677
33. Lozhkin, I.A., Dunaev, M.E., Zaitsev K.S., and Garmash, A.A. Augmentation of image sets for training neural networks in solving semantic segmentation problems. International Journal of Open Information Technologies, 2023, 11 (1), 109−117. (In Russian).
34. Soboleva, N. and Mushnikov, A. Improving the accuracy of semantic segmentation of carbides in the microstructure of composite coatings by the neural network. Materials Today Communications, 2024, 38, 108276. DOI: 10.1016/j.mtcomm.2024.108276.
35. Sikorsky, O.S. Overview of convolutional neural networks for image classification problem. Novye Informatsionnye Tekhnologii v Avtomatizirovannykh Sistemakh, 2017, 20, 37−42. (In Russian).
36. Ronneberger, O., Fischer, P., and Brox, T. U-Net: convolutional networks for biomedical image segmentation. In: N. Navab, J. Hornegger, W. Wells, A. Frangi, eds. Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, Lecture Notes in Computer Science Series, vol. 9351, Springer, Cham, 2015. DOI: 10.1007/978-3-319-24574-4_28
37. Badrinarayanan, V., Kendall, A., and Cipolla, R. SegNet: a deep convolutional encoder-decoder architecture for image segmentation. In: IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39, 2481–2495. DOI: 10.1109/tpami.2016.2644615.
38. Long, J., Shelhamer, E., and Darrell, T. Fully convolutional networks for semantic segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), MA, Boston, USA, 2015, IEEE, 2015, 3431–3440. DOI: 10.1109/cvpr.2015.7298965.
39. Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J. Pyramid scene parsing network. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), HI, Honolulu, USA, 2017, IEEE, 2017, 6230–6239. DOI: 10.1109/cvpr.2017.660.
40. Lin, G., Milan, A., Shen, C., and Reid, I. RefineNet: multi-path refinement networks for high-resolution semantic segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), HI, Honolulu, USA, 2017, IEEE, 2017, 5168–5177. DOI: 10.1109/cvpr.2017.549.
41. Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., and Yuille, A.L. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40 (4), 834–848. DOI: 10.1109/TPAMI.2017.2699184.
42. arXiv:1706.05587 [cs.CV]. DOI: 10.48550/arXiv.1706.05587.
43. Soboleva, N.N. and Mushnikov, A.N. Determination of the volume fraction of primary carbides in the micro-structure of composite coatings using semantic segmentation. Frontier Materials & Technologies, 2023, 3, 95–102. DOI: 10.18323/2782-4039-2023-3-65-9.
44. Yamashita, R., Nishio, M., Do, R.K.G., and Togashi, K. Convolutional neural networks: an overview and application in radiology. Insights into Imaging, 2018, 9, 611–629. DOI: 10.1007/s13244-018-0639-9.
45. Vik, K.V., Druki, A.A., Grigoriev, D.S., and Spitsyn, V.G. Application of deep learning neural networks for solving the problem of forest fire segmentation on satellite images. Vestnik Tomskogo Gosudarstvennogo Universiteta. Upravlenie, Vychislitelnaja Tekhnika i Informatika, 2021, 55, 18–25. (In Russian). DOI: 10.17223/19988605/55/3.
46. arXiv:1609.04836 [cs.LG]. DOI: 10.48550/arXiv.1609.04836.
47. Xu H., He, H., Zhang, Y., Ma, L., and Li, J. A comparative study of loss functions for road segmentation in remotely sensed road datasets. International Journal of Applied Earth Observation and Geoinformation, 2023, 116, 103159. DOI: 10.1016/j.jag.2022.103159.
48. Emelianov, V., Zhilenkov, A., Chernyi, S., Zinchenko, A., and Zinchenko, E. Application of artificial intelligence technologies in metallographic analysis for quality assessment in the shipbuilding industry. Heliyon, 2022, 8 (8), e10002. DOI: 10.1016/j.heliyon.2022.e10002.
49. Raketov, K.V., Israilev, N.Yu., Kazachkov, A.O., Zablotskaya, E.Yu., Rod, I.A., Ryabkov, M.S., Isachenko, A.V., and Shamiryan, D.G. Automated defect detection system in MEMS production. Nanoindustriya, 2018, 11, 7–8 (86), 542–548. (In Russian). DOI: 10.22184/1993-8578.2018.11.7-8.542.548.
50. https://axalit.ru
51. Matveev, A.V., Nartova, A.V., Sankova, N.N., and Okunev, A.G. DLgram cloud service for deep-learning analysis of microscopy images. Microscopy Research and Technique, 2024, vol. 87 (5), pp. 991–998. DOI: 10.1002/jemt.24480.
52. https://t.me/nanoparticles_nsk
53. Soboleva, N.N. RF Software Registration Certificate No. 2023660789, 2023.

Article reference