Intelligent bridge management system based on the image data from robotic devices

Bridge Maintenance, Safety, Management, Health Monitoring and Informatics Koh & Frangopol (eds) 2008 Taylor & Francis Group, London, ISBN 978-0-415-46844-2 Intelligent bridge management system based on the image data from robotic devices Sungkon Kim Department of Civil Engineering, Seoul National University of Technology, Seoul, Korea Jung Seok Lee Korea Infrastructure Safety and Technology Corp., Ilsan, Korea Youngjin Choi Department of Electronic, Electrical, Control and Instrumentation Engineering, Hanyang University, Ansan, Korea Young Shik Moon Department of Conputer Science and Engineering, Hanyang University, Ansan, Korea ABSTRACT: This paper addresses features of a Bridge Management System which is a computerized total information system relating to inventory data management, bridge inspection, condition assessment, repair, scheduling, and budgeting. A BMS consists of software and hardware system. Software system covers database software for managing all bride maintenance related items, and application software. The database system manages overall physical description, structural and inspection reports, repair and strengthening history, assessment results. The application program consists of software engineered overall bridge maintenance work program, input and output program for database, assessment and analysis program to convert stored data to usable information, and reporting program. The heart of the program is the assessment and analysis program that covert individual items stored in the database to useful information for the bridge managers. The core part of BMS operation will be on the data structure in which all the information from the field inspection is well archived, so that the system could perform the bridge assessment tasks. Conventional systems have been relied on the hand writing data for the damage aspects by field engineers. In this paper, an advanced technique is introduced to acquire damage aspects of the bridge members from the image data which are taken by the robotic systems. 1 INTRODUCTION The Bridge Management System aims intelligent managing and analyzing all the necessary data such as bridge information, construction, maintenance, rehabilitation, traffic, and accident report which are produced during service life of a specific bridge. Manipulating these data, computerized bridge management system is able to produce proper inspection scheme, decision making on repair and further works. The bridge management system supplies bridge manager with objective data for long-term damage prediction, safety evaluation and effective decision making for optimize repair and rehabilitation. In order to make this multi purpose information system successfully implemented, various types of programming schemes, skills, and tools are necessarily involved. Typical requirements for BMS are itemized as follows (Chen & Kim 1995): Well organized knowledge base on bridge engineering domain: physical descriptions which contains geometrical and topological descriptions of the bridge itself, behavioral descriptions which describes how structural components responds to stimulus, and kn owledge for diagnosis which describes domain knowledge for evaluating the current b ridge condition in light of observed symptoms. AI (Artificial Intelligent) based representation of identified knowledge to generate beh avior hypotheses and reason about the bridge s observed and expected function based on this computer-based representation. 1215

Hybrid programming paradigm which is amalgamating several different programming methodologies such as objected-oriented (Booch 1991) and logic programming (Bratko 1990): Object-oriented programming is particularly useful for problems in which data objects can be categorized hierarchically. While logic programming s declarative style provides a natural way to represent rule-based knowledge such as condition assessme nt. Well integrated multi-disciplinary system encompassing high-tech computer hard ware, communication networks, various advanced software tools, and GUI tools.. A proposed overall architecture of the BMS has been depicted in the Figure 1. Operating Modules D/B Scheduling Reporting Inspection Assessment I/O D/B Manual Inventory Damage Inspection Specification s Monitering Inspection Assessmen Signal Drawings Report Expertise H/W, S/W TOOLS, GRAPHICS TOOL, NETWORKS Figure 1. Architecture of a BMS. The core part of BMS operation will be on the data structure in which all the information from the field inspection is well archived, so that the system could perform the bridge assessment tasks. Conventional systems have been relied on the hand writing data for the damage aspects by field engineers. In this paper, an advanced technique is introduced to acquire damage aspects of the bridge members from the image data which are taken by the robotic systems. 2 OPERATION OF BRIDGE INSPECTION IN FIELD In order to operate a BMS work flow from field inspection to assessment is as shown in Figure 2. Contents of the principal tasks are briefly described. 2.1 Inspection Schedule Inspection schedule is automatically generated from the system based on the attributes of the inspection units which are saved in the inspection data base. Manager can down load inspection items on the computer mounted on the inspection vehicle based on the schedule for a certain inspection period. This screen on the computer mounted on the various types of inspection devices 1216

always has the same format and ready for field inspection. There are number of inspection units listed and their features are also supplied in various formats such as drawings, images, or photos. With these information inspector can recognize his/her schedule and number of inspection units, even the way how to inspect if necessary. In the BIRDI Project (LEE et al. 2007) four types of the devices for bridge inspection are under developing; a smart inspection vehicle, railmounted robots, aerial device and crawling robots. FTP- Server DB Server Main Server Inspection DB Client Inspec- Conversion Damage (Image or Vector) Images Field Inspection Figure 2. Inspection flow. 2.2 Field Inspection Field inspection begins according to the order listed on the computer as shown in Figure 3. When inspector clicks the first item (inspection unit), computer screen prompts number of queries along with necessary help tools such as images or drawings. Inspector then should follow the instruction for making input and taking some drawing or photos if asked. In this project, a number of image data from the cameras mounted on the robots are transferred to the computers in real time. 1217

Figure 3. Inspection schedule in field. 2.3 Condition Assessment When the field inspection is completed, all the data collected during inspection are uploading to the host computer and main BMS module. Each database allocates its own data in proper address in the inventory, inspection, and assessment DB for further analysis. Several types of report from inspection results are supplied to human monitor. Inspection status report is produced daily base to manager to figure out the overall inspection schedule and status. Whenever damages are found, the system analyzes its possible cause, contents, and its severity then finally the assessment report is produced when a certain inspection period is completed. Details for assessment algorithm are addressed in the reference (KHC 2000). 3 DAMAGE EVALUATION SCHEME BASED ON DIGITAL IAMAGES 3.1 Conventional Tool for Damage Depiction Damage information during inspection is usually recorded in hand writing format or image picture, and then obtained information is registered into a BMS for further assessment and data management. In conventional system damage information such as cracks are depicted on the paper-based log-book as shown in Figure 4 in the field and this paper-based log-book is used for damage evaluation at the office as well. Figure 4. Damage depiction in field. 1218

This type of conventional method by depicting damage type and location in hand writing has been widely used, however it has a number of difficulties to incorporate a computerized BMS. First of all, 3.2 Image Based Damage Management In order to manage the various structural damages, such as cracks, deformation, or delamination, etc. founded during inspection in a computerized format in field it is necessary to provide scientific and systemized measures. And also it is inevitable to introduce and operate a reasonable maintenance/management system that can integrate the developing computer systems, measuring devices, structural stability assessment techniques, and repair/reinforcement techniques in a single process. This study has been performed to replace the conventional system, on which bridge inspectors ride on inspection vehicles to perform visual inspections, into a robotic system equipped with an autonomous robot with the small size cameras. This advanced system would greatly reduce the effort required to perform inspection by engineers in field, and also enhance the safety for inspectors. Precisely controlled robot system provides more accurate and efficient data that are objective and quantitative. Since the size of the robotic system is relatively small, the vehicle for inspection could be manufactured for minimized size which leads the inspection and maintenance works on the bridge would require less traffic restriction (Yang & Nam 2007). Machine vision system including camera and image processing tool which are under developing in BIRDI project is introduced in this section. 3.2.1 Robot Based Bridge Inspection System The integrated system for the robot-based bridge inspection consists of a specially designed inspection truck, guide rails and the inspection robot as shown in Figure 5. The guide rail is located on the end of the folded shafts. Also, the inspection robot system is mounted on the guide rail. It was designed to be able to move longitudinally on the guide rail to cover more precisely wide area of the bottom of the bridge. The bridge inspection robot platform consists of three parts: the base platform, up and down platform, and the camera mounting mechanism, which is designed to accommodate to obtain clear images of the structural damage on various types of bridge members. Figure 5. Robot based bridge inspection system. 3.2.2 Camera Mounting Devices Two types of mechanism for mounting camera: the sliding and scissors types. The scissors type as shown in Figure 6 is able to have the larger workspace than the sliding type. Also, the motion of the former is smoother than the latter, however it has the weak point as the mechanism of 1219

scissors type is more complicated. On the other hand, the sliding type in Figure 7 is more stable and substantial than the scissors type in operational concerns. Figure 6. Scissors type. Figure 7. Sliding type. 3.2.3 Machine Vision Mechanism This system is composed of CCD cameras, a DVR board and a computer. The specifications for the vision system should be determined, considering weight, electric power, communication scheme and cable width. Conforming to the system specifications, the hardware system is designed. Related algorithms for processing the images captured from cameras have been designed and implemented. 3.2.4 Damage Identification Conventional systems for crack detection simply display the found damages as describing in the previous section. However, for more effective and scientific bridge inspection, we also need some information about the damage contents. Cracks in concrete member, for instance, information such as length, width and growth rate are necessary for evaluating the cracks. In order to acquiring crack information, however, there are many difficulties such as irregularities in crack shape and size, various scratches and painted surfaces, and irregularly illuminated conditions. These may cause serious problems in automatic crack identification. In order to solve these problems, we propose the following method for automatic crack detection. Our method consists of two steps: crack detection and crack tracing. For the crack detection, we perform three steps of pre-processing and extract the candidate cracks. Firstly, we subtract the smoothed image from the original image. The smoothed image is obtained by using a median filter (Fujita 2006). Smoothing with the median filter is used to remove thin line structures such as cracks. Therefore, in the subtracted image, cracks are prevented without variations. Secondly, we remove some artifacts using a filter for removing isolated points. Thirdly, we apply some morphological processing such as dilation and thinning to guarantee the connection between cracks, where the number of iterations is determined by the distribution of candidate cracks. After this process, we obtain the real cracks from the original image. Figure 8. Result of crack detection. 1220

For the crack tracing, we divide the image with detected cracks into several regions and select a seed point in each region. For each seed point, we examine the intensities of 8-neighbor pixels to determine the next pixel with the minimal intensity. To avoid local minima, the range of direction is restricted. We measure the width and the length of each crack. Figure 8 shows the result of crack detection. 3.2.5 3.2.5 BMS Operation The results of identification processing should be stored into database in a BMS for further condition assessment of the bridges. The raw images information of the damages are converted into an interchangeable file so that the results can be utilized for current condition assessment and data archiving in the Bridge Management System (BMS). The interchangeable file is stored in dxf format that is compatible with any computer graphic tools. Moreover, we should also be able to depict the whole image of the damages in the wide bottom area of the bridge. The structure of the dxf file format is carefully investigated to parse the syntax of each component, in order to write the information of detected cracks into a dxf file. Figure 9 shows the result of created dxf file. The detected cracks and the result of image stitching are shown in Figure 9. Figure 9. Depicted Cracks in dxf format. Digitized damage information are used two ways for condition assessment in the BMS. As displayed in Figure 10a, engineers can register damage on the computerized drawings using the predefined damage icons based on the transferred dxf files shown in Figure 9. Figure 10. (a) Damage Registration by Icon (b) Image Overlay. Another method is a kind of automatic image overlay on the digital drawings in the graphic tool as shown in Figure 10b. 1221

In both methods, damage information including images is archived in layered data structures so that any information can be retrieved for any members or structure levels in any time. 4 CONCLUSION An advanced technique to manage damage information in a BMS is introduced to acquire damage aspects of the bridge members from the image data which are taken by the robotic systems. The core part of this task will be on the data structure in which all the information from the field inspection is well archived, so that the system could perform the bridge assessment tasks. Conventional systems have been relied on the hand writing data for the damage aspects by field engineers. Damage information during inspection is usually recorded in hand writing format or image picture, and then obtained information is registered into a BMS for further assessment and data management. In conventional system damage information such as cracks are depicted on the paper-based log-book. On the other hand, the proposed system in this study is that bridge inspectors ride on inspection vehicles to perform visual inspections, into a robotic system equipped with an autonomous robot with the small size cameras. Eventually acquired damage information in digitized format can be used in the various manners in BMS for the condition assessment of the bridges. ACKNOWLEDGMENT This work was supported by Ministry of Construction and Transportation (MOCT) and Korea Institute of Construction & Transportation technology Evaluation and Planning (KICTEP), of Korea. REFERENCES Stuart S. Chen and Sungkon Kim. 1995. Information Architecture Considerations for a Smart Structural System, Civil Engineering Systems 12: 1-19. G. Booch. 1991. Object-Oriented Design: with Applications, The Benjamin/Cummings Pub., California. I. Bratko.1990. Prolog Programming for Artificial Intelligence, Second edition, Addison-Wesley. J. S. Lee, I. Hwang, J. H. Park, J. H. Lee. 2007. Robotic Systems for Automated Bridge Inspection, Proceedings of the SHM-III, Vancouver, 2007 Korea Highway Corporation. 2000. Development of Seo-Hae Bridge Management System, Technical Report. K. T. Yang & S. Nam. 2007. Development of Robotic System over Bridge Superstructures, Proceedings of the SHM-III, Vancouver, 2007. Y. Fujita. 2006. A method for crack detection on a concrete structure, International Conference on Pattern Recognition 2006 : 901-904. 1222