FACULDADE DE ENGENHARIA DA UNIVERSIDADE DO PORTO

Transcription

1 FACULDADE DE ENGENHARIA DA UNIVERSIDADE DO PORTO Deep learning methods for multimodal hepatic lesion segmentation: Fusing functional and structural medical images Thesis Plan Jessica Condesso Delmoral, UP Doctoral Program in Biomedical Engineering Supervisors: Prof. Dr. João Manuel R. S. Tavares (FEUP) Co-supervisor: Prof. Durval C. Costa (HPP/Champalimaud Foundation) 2016/2017

2

3 Summary: Liver cancer is one of the most lethal cancers worldwide, occurring in both primary liver cancer and as secondary cancer form, derived from metastizations of other primary tumours. Physicians recur to Computed Tomography (CT) for the tasks of visualization of anomalies in shape and texture of liver lesions towards disease diagnosis, resective surgery planning and progression evaluation. Novel developments in functional image have also validated the exploration of liver function with Positron Emission Tomography (PET). Surgical tumour resection is considered as gold-standard for liver tumour treatment, requiring surgical planning based on the image data, retrieved by the former imaging techniques. Medical image analysis advances everyday towards the development of segmentation algorithms of relevant structures in images. Emerging developments on CT image, have contributed for the appropriate analysis of liver function in liver cancer clinical settings. In turn, developments in functional image computational analysis of liver are very limited. The combination of both imaging modalities for the extraction of accurate liver anatomy representations, confers a higher sensitivity and specificity in the localization of cancer lesions. The aim of the project proposed, is to combine the anatomical and functional lesion information towards a better visualization of the total lesion extent and anatomic location. Moreover, for resective surgery planning the additional extraction of the liver vasculature and segmental anatomy are necessary. State-of-the-art liver Computer Vision and image analysis techniques, include methods that require the usage of manual or semi-manual feature engineering and fully-automatic methods. Emerging developments in deep learning methods present great potential for the development of automatic liver lesion analysis based on medical image. Moreover, for computer aided resection surgery planning, a subjectspecific deep learning-based liver, liver lesion and vasculature segmentation tool is proposed combined with a final liver segments segmentation.

4

5 Content Introduction Motivation Research Goals Structure of the document... 4 Anatomy of Liver Cancer CT Functional imaging Challenges State of the art Results Liver and tumor segmentation in structural image Liver and tumor segmentation in functional image Liver vessel segmentation Methods Work package 1 CT liver segmentation Work package 2 PET-CT tumour segmentation Work package 3 Liver segments map segmentation Work package 4 Integration and Visualization Overview Timetable Final Remarks 29 References i

6 ii

7 List of Figures Figure 1.1. Estimated age-standardized incidence rates of liver cancer, in both sexes, worldwide in 2012 [2] Figure 1.2. Cancer mortality worldwide, according to the World Health Organization and Globocan 2012 reports [2] Figure 2.1. Axial abdominal cavity represented in a CT image (a), corresponding structure representation(b), abdominal cavity location (c) and segmental division (d). 1. Right lobe of liver, 2. External oblique muscle, 3. Inferior vena cava, 4. Caudate lobe of liver, 5. Left lobe of liver, 6. Linea alba, 7. Coeliac trunk, 8. Left gastric artery, 9. Rectus abdominis muscle, 10. Stomach, 11. Left colic flexure, 12. Diaphragm, 13. Vertebral arch, 14. Spinalis muscle, 15. Spinous process, 16. Thoracolumbar fascia, 17. Vertebral canal and spinal cord, 18. Thoracic vertebra, 19. Longissimus thoracis muscle, 20. Iliocostalis muscle, thoracic part, 21. Spleen, 22. Posterior gastric artery, 23. Latissimus dorsi muscle, 24. Left lung, 25. Costodiaphragmatic recess, 26. Right subphrenic recess, 27. Bare area of liver, 28. Intercostal lymph nodes, 28. Ostium cardiacum, 30. Superior diaphragmatic lymph nodes, 31. Posterior mediastinum, 32. Cardiac notch, 33. Inferior diaphragmatic lymph nodes, 34. Left paracolic gutter [92]... 5 Figure 2.2. The Couinaud's classification system [93]... 6 Figure 2.3. Hepatocellular carcinoma example, seen in a CT image [94]... 8 Figure 2.4. Examples of difficulties of liver segmentation: sample CT slices with ambiguous boundary between liver and heart (a), stomach (b) and tumor must be included in segmentation (c) Figure 2.5. A 65-year-old patient with multiple liver metastases due to colon cancer. The CT image (a) clearly shows one large confluent lesion (asterisk) and the fused PET/CT image (b) shows two smaller hypodense lesions (arrows) that demonstrate abnormal FDG uptake [10] Figure 2.6. Examples why liver segmentation is a challenging task. In the first two images, liver tissue has to be separated from adjacent organs stomach (a) and heart (b). The gray-values in all structures are highly similar, which makes boundary detection difficult without a-priori information about the expected shape in these regions. In the third image (c), the tumor should be segmented as part of the liver. However, there is a considerable intensity difference between both structures, which often leads to misclassification of the tumor as non-liver tissue Figure 4.1. Examples architectures of input feature maps integration from multimodal images [95] iii

8 iv

9 List of Tables Table I. Average values of performance measures and scores of different method results evaluated on the MICCAI liver segmentation grand challenge. Measures reported as mean and standard deviation over all test images. Maximum overlap error, average volume different, average distance, root mean squared distance, max volume distance, and the final average score. Table II. Average values of performance measures and scores of different method results evaluated on the MICCAI 2008 liver lesion challenge. Measures reported as mean and standard deviation over all test images. Maximum overlap error, average volume different, average distance, root mean squared distance, max volume distance, and the final average score. Table III. Average values of performance measures and scores of different method results evaluated on the image dataset, using CNN-based methods. Measures reported with different segmentation accuracies. v

10 vi

11 Abbreviations and Symbols 2D 3D AAM ASM CAD CNN CT DBN DICOM DL DSC FOV GC GM HCC HU IVC ML MRI RBM ROI PET SAE SPECT SSM SVM WHO Two-dimensional Three-dimensional Active Appearance Model Active Shape Model Conputer Aided Diagnosis Convolutional Neural Network Computed Tomography Deep Belief Networks Digital Imaging and Communications in Medicine Deep Learning Dice Similarity Coefficient Field of View Graph cut Genetic Algorithm Hepatocellular carcinoma Hounsfield Units Inferior Vena Cava Machine Learning Magnetic Resonance Imaging Restricted Boltzmann Machine Region of Interest Positron Emission Tomography Sparse Autoencoder Single Positron Emission Tomography Statistical Shape Model Support Vector Machine World Health Organization vii

12 viii

13 Introduction The liver is essential for human survival and is a vital organ in the execution of key functions such as human metabolism, detoxification, protein synthesis, glycogen storage, hormone production, and production of biochemicals necessary for digestion. It is the largest organ in the human body weighting approximately 1.5 Kg, accounting for nearly 2% to 3% of average body weight. The functions that this organ performs are necessary for the proper homeostasis of the body, whereas functional alterations suffered upon disease of the liver tissues can become life threatening. Alterations of liver cells can be caused by inherited/genetic disorders (such as hemochromatosis or Wilson s disease); immune system -related pathologies (such as autoimmune hepatitis, primary biliary cirrhosis, primary sclerosing cholangitis); infections (such as viral infections e.g. hepatitis A, B, or C); cancer (such as hepatocellular carcinoma or cholangiocarcinoma); chronic alcohol abuse, obesity (causes non-alcoholic fatty liver disease) or intoxication (e.g. with toxic drugs). Pathological alterations in the liver, evolves firstly into tissue scarring (cirrhosis), which can further progress to liver failure and become a life-threatening illness. Among the abovementioned diseases, liver cancer presents an alarming prevalence in a global scale and is one of the most lethal cancers worldwide [1]. Liver cancer is characterized by the development of abnormal cell accumulations, that will appear represented differently in structural images, such as Computed Tomography (CT), and in turn, appear with differentiated functional behaviour in functional images, such as Positron Emission Tomography (PET). Physicians recur to different medical imaging techniques for the tasks of visualization of anomalies in shape and texture of liver towards disease diagnosis, resective surgery planning and progression evaluation. One of the most characteristic properties of the liver is its exceptional regenerative capability, derived from its modular/segmental structure which makes it possible to consider the organ as a set of functionally independent units. The anatomy of the liver has been standardized by the known Couinaud system, which divides the liver into eight functional lobes, which are independently supplied by its artery, veins, and bile duct. To preserve the adequate liver function, the resection of lesion masses is made with respect to the segments that contain the liver lesion. Moreover, the liver s high regenerative power, can be useful and improve the success of specified treatment courses. Hence, in the presence of liver cancer, physicians need to eliminate the malignant mass or masses present in the tissues, based on an informed decision of the liver, liver lesions, vasculature and segments locations. Software tools that extract accurate anatomic representations in three-dimensions (3D), for different patients and in a convenient amount of time, are the solution that bioengineering, computer vision and medical image analysis can solve. Computerized medical image analysis, can assist in tasks of disease detection, visualization, diagnosis, staging, monitoring and therapy planning. State-of-the-art liver Computer Vision techniques, are continuously developing, presenting novel techniques for the proper analysis of liver lesions in medical images, for tumour quantification. For this purpose, as in many other medical conditions, the need for software tools that would aid the clinical analysis of the information present in medical images, is of high importance. In the current study, we intend to present an overview of the state of the art techniques of visualization, segmentation and data analysis targeting the liver, cancer diagnosis and surgery planning. This chapter describes the clinical motivations of the presented work, the research goals that are expected from liver medical image analysis, followed by a summarized list of the objectives aimed with this project, and finally, an outline of the contents organization in this document, by chapter. 1

14 1.1. Motivation Cancer is a major clinical issue in healthcare worldwide, being a current open subject tackled by researchers, from different areas. Liver cancer occurs as both primary liver cancer, whose genesis occurred within the liver, and as secondary cancer form, derived from metastizations of other organ s primary tumours. Hepatocellular carcinoma (HCC) is the most common type of primary liver cancer. It usually develops after either a viral Hepatitis B/C infection or a liver cirrhosis. Cirrhosis is a liver condition characterized by replacement of healthy liver tissue with fibrosis, scar tissue, and nodules. Approximately 50% of the deaths caused by HCC occurred in China. In Western countries, the most frequent liver tumors are metastasis. The disease s world distribution in 2012 is presented in Figure 1.1. Figure 1.1. Estimated age-standardized incidence rates of liver cancer, in both sexes, worldwide in 2012 [2]. The problem resides in the increasing incidence of the disease. According to the World Health Organization (WHO), liver cancer is the fifth most common, and the second most lethal type of cancer, accounting for 7.5 million deaths in 2012 (Figure 1.2). Figure 1.2. Cancer mortality worldwide, according to the World Health Organization and Globocan 2012 reports [2]. Understanding the anatomy of a target organ is fundamental in medical diagnosis. The task of lesion characterization and quantification with respect to type and size of cancer tissue, and localization with respect to segments, vessel structures, or surrounding organs, are the primary factors to investigate. The lack of symptoms in the early stage of the disease makes it very difficult to diagnose liver cancer before 2

15 it is advanced. Hence, for the majority of patients, surgical tumour resection is the gold-standard treatment as the reoccurrence rate is the lowest and the long-term survival rate is the highest for this type of procedure. The following task of treatment planning depends highly on 1) disease staging and 2) lesion location relatively to vessels and critical structures, meaning that it is highly dependant on the previous task. Course of treatment may include radiotherapy, where gross target tumor delineation is required, chemotherapy, specially used when tumor is divided in several outbreaks, which requires knowledge of all the tumor locations and size, and surgical resection, where precise location of the tumor with respect to the whole liver and individual segments and vessels is required. The development of efficient and highly accurate computational tools that can execute the abovementioned tasks would constitute a major contribution in the clinical context of the disease. In computational science, an emerging class of algorithms, called deep learning, that have already revolutionized a wide number of data analysis problems, has been in the past decade introduced in image analysis and later in medical image analysis. Deep learning methods derive from the broader field of machine learning, which is based the concept of giving systems the ability to acquire their own knowledge based on example, by extracting patterns from raw data, instead of relying on hard-coded knowledge and rules engineered by humans. These algorithms could extract statistical discrimination rules from data representations of the given problems, and so forth researchers developed a wide set of algorithms that solve higher and higher complex problems. Deep learning methods, derived from the need of more complex computational tools when the degree of complexity of the data increased. They are inspired on the structural arrangement of human brains, and have presented an unparalleled performance, relatively to previous algorithms. They have moreover, in several cases even rival and exceeded the accuracies achieved by humans. For these reasons, the usage of these algorithms to solve the problems proposed in this project represent a very promising step Research Goals The aim of this PhD project is to develop an image-based, patient-specific, automatic liver and lesion segmentation tool to assist the physician in successfully visualize the livers entire anatomy and in performing an informed decision of the anatomic distribution of liver tissue, hepatic lesions, vasculature tree and liver segments. A proper Computer Aided Diagnosis (CAD) and surgery planning system requires the collection of all this information. The aim of this project consists in the development of a segmentation framework of liver lesions from CT and functional abdominal images, using deep learning methodologies, namely Convolutional Neural Networks. The goal is to combine the anatomical and biological image data towards a better visualization and volume estimation of both the liver and total lesion extent, to be included in a threedimensional visualization tool, for resective surgery planning. Specifically, the following objectives will be set: An algorithm for the segmentation of the whole liver in CT images; An algorithm for the segmentation of the liver lesions in CT images; An algorithm for the joint segmentation of liver lesions using both CT and PET images; An algorithm for the segmentation of the liver vessels; Building an anatomic model of the 8 segments that constitute the liver; An algorithm to segment the extracted liver anatomies into the eight segments that constitute it; A computational framework for 3D visualization of all the previously segmented liver structures. The present document intends gather a comprehensive understanding in the state of the art techniques used for computational analysis of liver cancer in medical images. Firstly, many research sources have been tuned into accurate segmentation of the liver in CT images, being extensively validated in the literature using a vast range of medical image processing algorithms. Secondly, the usage of functional images for liver cancer computational analysis is quite limited. This, it firstly due to the lack of access to functional imaging in the clinical setting, also hampering the access to viable image datasets, and secondly, because only in the past decade the usage of functional imaging in oncological diagnosis has validated the development of computational tools for these purposes. Hence, computational tools for the segmentation of liver and liver lesions, using both functional and structural images has not been reported previously in the literature, to the best of our knowledge. 3

16 Moreover, the integration of liver, liver lesion, and segment partition into one patient specific analysis tool is the main contribution of the works proposed in this document. Finally, the usage the most state of the art deep learning algorithms in the medical field is very promising. Namely, Convolutional Neural Networks have presented the potential to enable the development of the complex computational system proposed Structure of the document This document is organized into five chapters. Chapter 2 presents an overview regarding the main considerations of liver anatomy. Chapter 3 provides a comprehensive list of the most cited methods focused on liver segmentation, liver lesion segmentation, CAD systems and liver vasculature extraction, in the area of computer vision, namely in classification and recognition of objects represented in images. Chapter 4 features a detailed explanation about the methods that we intend to develop during the PhD project. Finally, the fifth chapter presents the final remarks of this work, indicating the expected results and advantages of this novel framework. 4

17 Anatomy of Liver Cancer The liver is located in the right upper quadrant of the abdominal cavity, between the right hemidiaphragm, protected by the thoracic cage, in right and anterior direction, and in close vicinity with the right kidney (distally), stomach, adrenal gland (spleen), the small intestine and colon (left), and pancreas (left). The anatomic positioning of the liver and organ arrangement within the abdominal cavity is presented in Figure 2.1. The anatomic position of the liver in the circulatory system is optimal for two of its main functions: 1) gathering, processing, and storage of metabolites and 2) neutralization and elimination of toxic substances. ) ) Figure 2.1. Axial abdominal cavity represented in a CT image (a), corresponding structure representation(b), abdominal cavity location (c) and segmental division (d). 1. Right lobe of liver, 2. External oblique muscle, 3. Inferior vena cava, 4. Caudate lobe of liver, 5. Left lobe of liver, 6. Linea alba, 7. Coeliac trunk, 8. Left gastric artery, 9. Rectus abdominis muscle, 10. Stomach, 11. Left colic flexure, 12. Diaphragm, 13. Vertebral arch, 14. Spinalis muscle, 15. Spinous process, 16. Thoracolumbar fascia, 17. Vertebral canal and spinal cord, 18. Thoracic vertebra, 19. Longissimus thoracis muscle, 20. Iliocostalis muscle, thoracic part, 21. Spleen, 22. Posterior gastric artery, 23. Latissimus dorsi muscle, 24. Left lung, 25. Costodiaphragmatic recess, 26. Right subphrenic recess, 27. Bare area of liver, 28. Intercostal lymph nodes, 28. Ostium cardiacum, 30. Superior diaphragmatic lymph nodes, 31. Posterior mediastinum, 32. Cardiac notch, 33. Inferior diaphragmatic lymph nodes, 34. Left paracolic gutter [92]. The healthy liver tissue is called parenchyma which is constituted by hepatocytes, the liver cells, specialized in blood filtering to remove toxins. The remaining connective tissue is called the stroma and serves as support, maintaining the liver fixated the abdominal cavity through ligamentous extensions. 5

18 The liver is the most vascularized organ in the human body receiving approximately a total of 25% of the total cardiac arterial output. It is supplied by four vessel structures: the hepatic artery and subbranches supply 25% to 30% of the liver blood input with oxygen; the portal vein, supplies the remaining 70% to 75% of blood input, carries nutrient-rich blood from the intestines and the colon; the processed blood leaves the organ through the hepatic vein; and the bile produced by liver cells drains through the biliary tree into the gallbladder [2]. The special arrangement of vessels allows division of the liver into eight segments. The livers segmental structure is able to regenerate after a serious disease that affects only a part of the liver. The Couinaud s segment distribution is represented in Figure 2.2. Since segments are functionally independent, the one(s) affected by the pathology can be completely removed. In such cases the other segments can grow significantly, which can compensate the loss of liver volume due to the operation or intervention. This way, the disabled segments are not regenerated rather their function is restored by the other parts. Figure 2.2. The Couinaud's classification system [93] The liver s job is vital, unlike for instance that of the spleen, and cannot be substituted by an artificial organ or medical device, as is the case of the heart or the kidneys. Hence, alterations to the normal function of liver cells and tissues can potentially become life threatening. Cancer is characterized of abnormal cells that start to rapidly divide (forming lumps), aggregate inside an organ (primary tumour), and can subsequently spread to other parts of the body, by the circulatory tract, to start growing in other organs (metastasis). When these alterations occur to the liver, a wide variety of clinical scenarios can occur in the anatomical and functional sense. There are several different pathologies that can occur in the liver, whereas, the oncological medical scenarios can be derived from benign or malignant alterations of the normal tissues of the liver. Alterations to the liver morphology can occur as benign lesions such as hepatocellular adenomas, hemangiomas, focal nodular hyperplasias, or nodular regenerative hyperplasia, or as malignant lesions such as hepatocellular carcinoma or secondary liver cancer. Benign Tumors Hepatocellular Adenoma: hepatic adenomas are more frequently observed in young women, whose pathogenesis is suspected to have a hormonal influence. The condition is associated in several studies with exposure to estrogens (i.e. oral contraceptives, OCPs), anabolic androgens, and genetic disorders. These adenomas are cell aggregations, present normally in the right lobe, which can grow into large dimensions 5 cm in diameter, and be multiple. These two characteristics make these adenomas highly 6

19 susceptible to becoming malignant, and susceptible to vascular invasion and subsequently sudden haemorrhage [3]. Focal Nodular Hyperplasia: is the second most common benign tumor and are also more frequent in women. These tumors are more solid masses with fibrous core and stellate projections containing atypical hepatocytes, biliary epithelium, Kupffer cells, and inflammatory cells. These lesions have decreased portal vein supply and are associated with the development of portal hypertension. Detection can be made via helical CT, angiography and Magnetic Resonance Imaging (MRI) but difficultly with ultrasound. Hemangioma: similarly to the previous two tumors, it is more frequent in women and the most common benign liver tumor among the three. Hemangiomas are asymptomatic and therefore, are usually detected by accident. Result of an abnormal development of blood vessel endothelial cells. The risk of malignant evolution, from these structure is null. Nodular Regenerative Hyperplasia: Nodular Regenerative Hyperplasia consists of multiple hepatic nodules formed as a result of periportal hepatocyte regeneration accompanied with atrophy of surrounding tissue. Its pathogenesis is unknown but studies show that it may be related with abnormalities in the hepatic blood, being formed as the response to atrophy of the liver, caused by vascular obstruction of small portal veins or hepatic arteries. The most common symptom in non-cirrhotic liver is portal hypertension and, in some cases, anomalous liver function with progressive symptoms are observed [4]. Malignant Tumors Hepatocellular Carcinoma (primary tumor): in the context of liver cancer, previous studies refer in the vast majority to HCC given that it is the most common malignant condition of the liver. Epidemiology shows that HCC is more common in men and most frequently arises in a cirrhotic liver. However, cirrhosis may be an important contributor to HCC although not being an essential precondition. In turn, the mortality due to the disease is very high in countries where there is high prevalence of viral infection by hepatitis B and C. Other causes may be hereditary hemochromatosis and food contamination. Hence, it arises from hepatocytes in a liver which is almost always chronically diseased, and often cirrhotic. Geographically, in countries where alcohol abuse is very high, and where this abuse subsequently becomes the most common cause of cirrhosis, most likely are countries with high incidence of hepatocellular carcinomas. This is the case of the United States, where data shows that 80% of hepatocellular carcinomas are derived from cirrhotic livers, with alcohol as the most common underlying cause, followed by hepatitis B infection. Moreover, an early diagnosis becomes more conditioned due to the similarity of symptoms caused by HCC which may be interpreted as a progression of the cirrhosis disease. The disease evolution presents a very characteristic behaviour. Upon the tumor hepatocellular carcinoma mass growth, the irrigation supply becomes more demanding and mechanisms of angiogenesis are promoted locally. Hence, new branches from the hepatic artery are formed invading many times the liver masses, in order to provide more nutrients and oxygen to the cancerous tissue. Screening is primarily done via ultrasound imaging since it is less expensive and is able to identify tumors greater than 3 cm. When masses are identified in more sophisticated imaging modalities such as CT (Figure 2.3) or MRI, and malignancy is suspected, a percutaneous liver biopsy of a part of the region detected can confirm the diagnosis. Moreover, liver cancers can be classified according to tumor severity and corresponding life expectancy by two standards: the CLIP (Cancer of the Liver Italian Program) and Okuda staging systems [5]. It is nonetheless difficult to diagnose liver tumor at a stage where resectability is still viable, and it may be carefully studied though patient imaging, taking into account the liver segments and the vascularized tumor mass, which add risk to this type of surgery. In some particular cases liver transplantation is also a course of treatment, whereas survival after transplantation for patients with a single lesion of no more than 5 cm, or 3 or less lesions with maximum of 3 cm, is viable [4]. Metastic liver tumor (secondary tumor): travelling of cancerigenous cells into the liver and metastization is very common, deriving very commonly from lung, stomach of colon cancer. This may be due to high intake of blood supply. Depending on the stage of detection, the strategies of resection have evolved and increased patients life expectancy [6]. 7

20 Figure 2.3. Hepatocellular carcinoma example, seen in a CT image [94] 2.1. CT Clinical diagnosis of liver disease is firstly accessed through CT imaging. The clinical protocols of identification of liver disease include several variations upon the stage of clinical intervention. Diagnosis is performed via biopsy but previous lesion location is done via morphological imaging. Computed tomography imaging works based on similar principles of traditional X-Rays, consisting in the computerized image reconstruction of the radiation image function into tomographic image sections. A beam of X-rays is wheeled around the target body section, from which the transmitted beam intensity is reconstructed according to the acquisition angle. The result is a stack of multiple 2D grayscale image slices, representing the screened result of a given body thickness, usually between 1 to 4 cm, hence representing a 3D volumetric representation of the soft tissues. Image resolution varies from 256x256 to 512x512 pixels. The key properties to retain from CT imaging of abdominal soft tissues is the reliability in the representations extracted, due to knowledge that the intensity values represent X-ray absorption coefficients. This factor, makes CT imaging the golden-standard modality of liver imaging, as opposed to MRI. There are some studies that still try to validate MRI imaging of liver. However, for structural analysis of soft tissues intensity values in MRI cannot be associated, with reliability, with any material or tissue type, due to the physics of MR image acquisition. This implies that the liver present intensity inhomogeneities in MRI, as opposed to CT, where the liver parenchyma is nearly homogeneous. Finally, given that the liver is a soft organ, and the abdominal cavity is highly populated, its shape is dependent on adjacent organs within the abdomen. Hence, the imaging conditions are never ideal, since a ) b c Figure 2.4. Examples of difficulties of liver segmentation: sample CT slices with ambiguous boundary between liver and heart (a), stomach (b) and tumor must be included in segmentation (c). 8

21 the appearance and shape of the liver does not entirely present very clearly defined edges, which may in times be not visible on many sides of the liver. In figure 2.3 this is observable as well as the close intensity differences between the liver and the heart, spleen or stomach. This may be aggravated by other pathologies that can affect the appearance of the liver Functional imaging Functional or molecular image consists in Positron Emission Tomography (PET) and Single Positron Emission Tomography (SPECT). PET, such as all molecular imaging methods, is considered a diagnostic complementary method, minimally invasive. This imaging modality requires the intravenous administration of a radiopharm. Specifically, it measures the dynamic metabolic uptake of specific proteins in target human tissues. This imaging methodology is carried out by the usage of relevant high affinity molecules that are known to bind to specific receptors and tissues in the human body. Thus, pertinent metabolic studies use key functional molecules metabolized by the target tissue to be analysed, which are labelled with a radio-isotope, that will start to decay. The decay radiation, that takes place upon intra-venous injection of the radiolabelled molecules will allow us to map the binding of these molecules into the target tissues. Functional studies based on this imaging process are spread over thirty years of history. Functional imaging plays an important role in cancer treatment. By revealing chemical activity of specially designed radiotracers. Functional imaging is capable of detecting a wide range of abnormal organ events in the human body, including tumorous activity. Normally, radiological imaging techniques depend on structural and morphological alterations of organs, in order to detect the presence of a pathological alteration. However, molecular image provides functional information about the metabolic processes carried out by tissues, and subsequently, of alterations these may suffer. Cancer is a disease that is primordially characterized by the alteration of tissue cells, which may have suffered some metabolic alterations, causing them to abnormally replicate, becoming malignant, accumulating into build-ups of abnormal cells. Thus, the advantage resides in the earlier detection of abnormal metabolic behaviour of tissues, that precede morphological alterations of those tissues. In this setting, functional imaging plays an important role in earlier detection of disease, as well as in proper spatial delineation of abnormal tissue. Figure 2.5. A 65-year-old patient with multiple liver metastases due to colon cancer. The CT image (a) clearly shows one large confluent lesion (asterisk) and the fused PET/CT image (b) shows two smaller hypodense lesions (arrows) that demonstrate abnormal FDG uptake [10]. It has been extensively validated, for more than 50 years now, that most tumors have a higher utilization and metabolism rate of glucose. Moreover, PET imaging has been widely applied to cancer clinics as a tool for tumor detection and diagnosis. However, the produced images have low spatial resolution in its nature, making it very difficult for precise localization of tumor. Precise localization is important for radiation therapy, a mainstream tumor treatment method, for treatment efficiency and avoiding side-effect. Therefore, PET needs to be combined with CT. Improvements performed to functional imaging equipments allowed for the scientific validation of the resulting images, and wide commercialization for clinical usage. In the last decade, specially designed scanners combining PET and CT were developed, and SPECT-CT has become a mainstream imaging modality ever since. 9

22 For the specific diagnosis of liver cancer, the literature provides studies over the last five years that validate the added value of using functional imaging, whereas an example of the added information that can be obtained from the combination of both modalities is presented in Figure Quantitative evaluation of radiotracer uptake in molecular image A qualitative assessment of the functional image done by experts is often sufficient for tumor assessment and detection. However, accurate tumor diagnosis and assessment require quantitative evaluation of the PET scanner records, since such records vary with time and dose of radiotracer. Popular semi-quantitative and quantitative methods are standard uptake value (SUV), tumor-to background ratio (TBR), nonlinear regression techniques, total lesion evaluation (TLE), and the Patlak-derived methods. Among them, SUV is the most popular technique, quantifying the physiological activity of cellular metabolism at a pixel level or at a voxel level. SUV represents the tissue concentration of the radiotracer at a given time divided by a few normalization factors. These factors include injected dose, patient size in terms of weight (BW, and a decay factor related to the radiotracer. It is defined as: SUV = C img(t) ID/BW where C img (t) stands for total radiotracer concentration in time t, ID represents injected dose (in Bq units) and BW is body weight (in g or kg). SUV is used in localization for normalization purposes. The SUV measurement provides an index of tracer uptake in a ROI or voxel of interest, normalised to the injected dose and to a normalisation factor (NF) based on the subject s anthropometric characteristics. The simplicity and versatility of the SUV make it suitable for clinical routine, as it can be used with a variety of PET tracers and does not require any arterial cannulation. [ 18 F]-FDG is the most widely used tracer applied in oncology for diagnosis and tumour staging. For details of the available different quantitative evaluation methods, the reader can refer to the review work [7] Challenges Many surgeons have developed strategies to improve resectability techniques applied to large tumors combining it with other complementary treatments such as chemotherapy, which require a close monitoring of the treatment effect and disease evolution. The literature states that despite the increase pf successful surgical resection procedures, statistics regarding long-term survival remained somewhat similar, ranging between 25 % and 40 %. These studies showed the high number of patients developing recurrent intra- or extrahepatic tumor manifestations mainly due to the presence of undetected tumor sites at the time of surgery [8]. Hence, the increase in accuracy of the extensive preoperative study of the patient s anatomy before liver resection is highly necessary, to accurately define the hepatic lesions, intra- and extrahepatically. Liver segmentation is the basis for computer-based surgery planning of interventions as tumor resection, living donor transplantations, or minimal invasive surgery [9]. a b c Figure 2.6. Examples why liver segmentation is a challenging task. In the first two images, liver tissue has to be separated from adjacent organs stomach (a) and heart (b). The gray-values in all structures are highly similar, which makes boundary detection difficult without a-priori information about the expected shape in these regions. In the third image (c), the tumor should be segmented as part of the liver. However, there is a considerable intensity difference between both structures, which often leads to misclassification of the tumor as non-liver tissue. 10

23 The liver shares a similar intensity distributions with its surrounding organs (e.g., the heart, the right kidney and the spleen). This may difficult the analysis of computational tools, especially for automatic liver detection. Furthermore, other structures may appear with intensity values, and/or adjacent to the liver walls, making the process of discarding of these structures, by computational algorithms, highly challenging. In turn, the shape and appearance of the liver vary largely across subjects. Finally, the presence of tumors or other abnormalities may result in serious intensity inhomogeneity. The added value of FDG PET to conventional anatomical imaging such as computed tomography (CT) for assessment of patients with liver metastases has been well documented [8], [10]. However, previous record of computational algorithms for multi-modal liver cancer analysis are sparse, and the presence of hybrid PET/CT image acquisitions equipment has been made widely available in clinical centres very recently. 11

24 12

25 State of the art In this section, we report the criteria used to identify the relevant works for this review. An overview of the main topics addressed by the selected articles is presented. A search was performed on PubMed, Scopus and ISI Web of Sciences databases between 10 of April of 2017 and 10 of June of Keywords used: liver, segmentation, vessel, vasculature, lesion, hepatocellular carcinoma. As a result, a total of 721 articles were obtained and, posteriorly evaluated. In all 721 articles, we analyse the title and the abstract. In this step, the follow criteria were established, namely: 1) the study must be written in English; 2) patents were excluded; 3) studies with main focus different of the proposed study were excluded. Tables I, II and III presents an overview of the included works. As such, a total of 78 articles were selected for the current review Results Liver and liver tumor computational analysis is a theme that has deserved the attention of researchers for several years. The challenge however, became highly popularized in the past decade, motivating the organization of three conference challenges within this period. The area has evolved from 2D analyses to 3D advanced computational simulations, 3D models of varied nature from the fields of Computer Vision, image analysis and machine learning. The nature of different algorithms allows the classification of these algorithms based on the amount, and type of data they use. The primary segmentation algorithms were based on image analysis, human engineered rules, and solution-specific methods, which we will henceforth call in this document as non-learning methods. Thresholding is the simplest of these algorithms, to separate image regions based on intensity limits. More advanced methods were included in this group such as deformable, graph-based, and level-set based models. Deformable models have proven to be very effective tools for complex anatomic motion reconstructions, or in segmentation of broadly varying shape objects. In previous work, the usage of simple intensity based segmentation algorithms were used to iteratively segment and adapt a model into the structures boundaries, called active contours. Other methods derived by different approaches were extensively validated in the literature such as level-set representations whereas a contour iteratively evolves dependant on a specific mathematical problem formulation and in turn, graph representations of images have also been very successful at partitioning dissimilar regions. Novel segmentation algorithms are continuously emerging, since the accurate modelling of liver structures still requires further development. Machine learning algorithm came to revolutionize the field of image analysis. They are in their majority based in advanced statistical analysis of the image data, via the extraction of different, and simpler representations of the original image data. Tumor segmentation using both medical images under study can be subdivided into two parts: initial tumor recognition and its sequential delineation. Recognition determines the tumor location from other similar regions in the image. In the CT image, this consists in the identification of the liver by manual or computational methods. In functional images, this consists in the discrimination from other normal tissues, of regions with high uptake. In many modern algorithms, a rough area of tumor region defined by clinicians is still needed, hence working as semi-automatic methods. A comprehensive outline of the literature studies relevant in the field under analysis will be presented in the following sub-sections. 13

26 3.2. Liver and tumor segmentation in structural image Liver detection and segmentation in medical images has been reported using CT, MRI and Ultrasound. In a liver medical diagnosis system, fully automatic methods are desirable, however, is a challenging task suffering several stages of development. Methods in the literature have focused on tumor segmentation in abdominal Dynamic Contrast-Enhanced MRI [11]. However, the most widely used method for liver diagnosis is CT, and hence, it is also the imaging modality with most prevalence the literature. As an overview, the study of CT-based liver segmentation has been validated until the last decade using mainly statistical and atlas-based segmentation models, and were in the past decade overcome by the higher performance presented by machine learning methods. Moreover, in the past decade, three liver and liver lesion segmentation workshop challenges were organized, boosting the interest of the scientific community towards this task. Non-learning and Image analysis methods In 2007 a CT liver segmentation challenge was organized for MICCAI 2007 conference. Until the past decade, the greatest contributions made in the field of whole liver segmentation were reported upon the MICCAI Sliver 2007 challenge, making available a dataset of 40 contrast-enhanced CT images. It should be noted that the image dataset was composed of livers containing liver lesions, testing the robustness of the segmentation algorithms to identify liver lesions correctly as belonging to the object of interest. In this setting, the top ten methods proposed in the automatic segmentation category, included deformable models [12] [16], Statistical shape models (SSMs) [17] [19], level-set methods [20], [21], atlas-based methods [22]. The semi-automatic methods proposed by the top six authors in this category included graph-cuts segmentation [23], flood-filling segmentation [24], a level-set based deformable segmentation [20], [25], radial basis function guided level-sets segmentation [26] and atlas-based segmentation [27], further refined by manual interaction. As could be expected, in the vast majority, the semi-automatic segmentation methods proposed at the time surpassed the performance of the automatic methods. The challenge evaluated the results by comparison to expert-generated references and using in a combined scoring method which evaluated a set deviation metrics to ground truths. The performance results of the MICCAI Sliver challenge are presented in Table I. It should be noted that the top three best performing automatic segmentation methods in the challenge were all based on Statistical shape models. SSMs, primarily presented by Cootes et al., consisted in statically model prior shape data as a parametric set of equations that focus rather on the boundary of the region to be segmented instead of its internal voxels [28]. These types of algorithms became very popular in medical image segmentation, despite presenting lack of flexibility. To overcome this, Kainmuller et al. presented a combined SSM-constrained segmentation followed by a deformable mesh guided by a heuristic tissue classifier. The method included a thresholding initialization step and allies the free-form intensity dependant flexibility conferred by the deformable step, being the best automatic method presented in the challenge [18]. Semi-automatic methods based on user interactive refined graph-cut (GC) segmentation won the competition [23]. Graph partitioning methods consider the image to be segmented as a graph, composed of nodes representing image voxels, and edges connecting the neighbouring nodes, weighted by a given dissimilarity rule. This graph representation can be partitioned into connected components according to criteria describing the properties of the expected segments resulting in the segmentation of the image. The dissimilarity measure can incorporate gradient, intensity, texture or any other images features. GC methods use a minimum cost function between all possible cuts of a graph representation of the images, requiring background and object manual initialization. In subsequent years, other methods have continuously been proposed in the literature and tested on the same MICCAI grand challenge dataset, allowing the direct performance comparison. Later, automatic segmentation methods were submitted to the challenge outperforming the abovementioned studies. SSM methods were continuously proposed with different modifications, such as combinations with free-form constraining [29], level-set [30], graph-cut [31] methods, among others. Graph-based methods such as GC was proposed by many authors in semi-automatic segmentation methods due to its ability to interactively edit the segmentation [32]. 14

27 Table I. Average values of performance measures and scores of different method results evaluated on the MICCAI liver segmentation grand challenge. Measures reported as mean and standard deviation over all test images. Maximum overlap error, average volume different, average distance, root mean squared distance, max volume distance, and the final average score. Method Kainmuller et al [18] Heimann et al. [17] Saddi et al. [15] Schmidt et al. [13] Chi et al. [12] Ruskó et al. [14] Seghers et al. [19] Furukawa et al. [21] Rikxoort et al. [22] Susomboom et al. [16] Beichel et al. [23] Beck and Aurich [24] Dawant et al. [25] Lee et al. [20] Wimmer et al. [26] Slagmolen et al. [27] Chen et al. [31] Moghbel et al. [33] Platero and Tobar [34] Hu et al. [35] Lu et al. [36] Dou et al. [37] Runtime (min) Overlap error (%) Volume difference (%) Avg. distance (mm) RMS (mm) MICCAI grand challenge automatic methods Max distance (mm) Final Score ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± MICCAI grand challenge semi-automatic methods ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± Other methods validated in the MICCAI grand challenge dataset ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± , The level-set algorithm is another mathematical formulation of an iteratively evolving surface or contour. According to this technique the contour is represented using a signed function, i.e. the level-set function, where the zeros valued locations corresponds to the actual contour. The level-set function requires 15

28 a contour initialization which then is computed, incorporating the contour propagation speed that is defined at each voxel. The advantage of this approach is that it can handle topological changes of the contour, but these methods can be time-consuming and it is difficult to handle over-segmentation. Level-set have been successfully applied to whole liver segmentation. Wimmer et al. (2009) latter proposed a level-set based ASM, that used a Parzen density estimation, to boost classifiers to analyse appearance information [30]. Image-based methods, besides level-set algorithms were proposed in semi-automatic methods. Ruskó et al. proposed an improved histogram-guided Region Growing (RG) method, obtaining competitive results in multiphase CT images [38]. More sophisticated SSMs such as Active Appearance Models (AAMs) have also been proposed in the literature. Transversal to SSMs and image and contour-based algorithms, initialization always stood as the biggest issue in the performance of these methods. Chen et al. proposed a novel combined AAM with a free-form segmentation based on Livewire and GC algorithms, for abdominal multi-organ pose estimation, initialization and segmentation [31]. The method was validated and compared with the dataset and results of the MICCAI Sliver challenge presenting similar performance and demonstrating less computational cost. Tomoshige et al. proposed a relaxed conditional SSM with conditional features error model, with a subsequent free deformation step demonstrating very positive results for automatic liver segmentation, on a private dataset of 144 non-contrast CT images, expressing performance as a Jaccard index valued 0.86 [39]. The MICCAI Sliver dataset was also used for the development of liver lesions segmentation. Another pertinent graph-based algorithm that became popular in medical image analysis is the Random Walker (RW) algorithm for segmentation, proposed by Grady (2006) [40]. Moghbel et al. tested the performance RW for liver lesion segmentation, developed on an image dataset composed of healthy and unhealthy livers, combined with a rib-cage removal using B-spline contouring, achieving a promising average performance expressed as a Dice similarity index (DSC) valued 0.94, and 0.91 for the MICCAI Sliver challenge [33]. In turn, another popular type of segmentation algorithms were atlas-based methods. Probabilistic atlases (PA) use shape priors and spatial relationship information. Atlas-based methods were also presented in the MICCAI Sliver challenge, however their initial formulation did not outperform the previous modeland image- based methods, not being able to handle properly the very high variation that characterized the liver shape among different patients. Nonetheless, Okada et al. proposed a novel method combining the properties of both probabilistic atlases for initialization followed by an optimized SSMs fitting to the image intensities [41]. Lastly, Xu et al. proposed a registration based on dense 3D-scale invariant feature transform (SIFT) features to find correspondences between source images and target atlas. The labels of the source image are later used to segment the target image, and the method was validated on the MICCAI Sliver challenge, with performances of 96% dice overlap [42]. In turn, Platero and Tobar developed a method combining the spatial normalization with the segmentation method based on standard CRF models to guide 19 atlases for liver segmentation, presenting high DSC valued 95% 97% for the MICCAI 2007 Sliver dataset [34]. In the following year, MICCAI 2008 conference launched the MICCAI 2008 Workshop on 3D Liver Tumor Segmentation Challenge, making available CT data from 30 liver tumors. At this time, research in medical image started to incorporate machine learning methods. Hence the proposed methods are composed by a mixed variety of algorithms. Similarly to the previous challenge, semi-automatic methods outperformed automatic ones. In this competition, the top ranked method presented, consisted in a semi-automatic method combining graph-cuts with a watershed low-level segmentation [43]. Moreover, region-growing algorithms [44], thresholding only [45] and combined with filtering techniques [46], intensity-based analysis [47], [48], deformable models [49], [50]. Machine Learning based methods Apart from the mentioned methods, novel segmentation techniques incorporating machine learning algorithms have gained attention by researchers. Machine learning, as in many other fields of image analysis, has grown preponderantly, presenting superior results in a varied number of computational tasks. More recently, a sub-field of machine learning algorithms, denominated as deep learning, is the current state of the art set of algorithms that are being validated in the literature for signal and image processing tasks. Medical image analysis research field has also followed the deep learning trend 16

29 and is currently applying these algorithms to a wide number of segmentation tasks. An overview of the key first machine learning incorporations with standard methods, pure machine learning algorithms and more advanced deep learning methods applied to liver organ segmentation follow. Among these, data-driven methods for pixel labelling were widely used for target object segmentation. In the context of the MICCAI 2008 workshop challenge, learning-based algorithms were proposed based on Adaboost models [51], Bayesian probabilistic methods and Support Vector Machines (SVM) voxel classification. SVM belongs to the supervised learning methods that combine linear algorithms with linear or non-linear kernel functions SVM by finding the best generalizing hyperplane with maximal margin separating the two classes. The second best method proposed in the MICCAI 2008 competition was a semiautomatic method using supervised SVM voxel classification strategy in 2D slices, further propagated adjacent slices for tumor segmentation [52]. Other Bayesian learning algorithms [44] were also proposed. The results of the segmentation algorithms presented in the competition for liver tumor segmentation are outlined in Table II. In subsequent years also, many authors used the challenge dataset for method validation. Zhang et al. proposed a watershed liver initialization followed by SVM-based semi-automatic regional tumor classification algorithm [53]. The authors validate their method on the MICCAI liver tumor challenge dataset, and the method outperforms the results distance to ground truths, than the previous methods. Table II. Average values of performance measures and scores of different method results evaluated on the MICCAI 2008 liver lesion challenge. Measures reported as mean and standard deviation over all test images. Maximum overlap error, average volume different, average distance, root mean squared distance, max volume distance, and the final average score. Method Overlap error (%) Volume difference (%) Avg. distance (mm) RMS (mm) MICCAI grand challenge automatic methods Max distance (mm) Final Score Stawiaski [43] 29,49 23,87 1,50 2,07 8,29 73 Zhou et al. [52] Nugroho et al. [48] Choudhary et al. [46] Wong et al. [44] Moltz et al. [45] Smeets et al. [54] Shimizu et al. [51] Ben-Dan et al. [55] Schmidt et al. [49] Hame et al. [50] Kubota et al. [47] Other methods validated in the MICCAI 2008 grand challenge dataset Zhang et al. [53] Machine learning algorithms were also applied successfully to whole segmentation. Selver et al. developed a fully automated liver segmentations method that employs pre-processing for exclusion of neighbouring structures, k-means clustering, and multilayer perceptron (MLP) for feature based boundary recognition, and post-processing for removing miss-segmented objects and smoothing liver contours [56]. 17

30 The method is validated in CT images with and without contrast media. Goryawala et al. propose a 3D whole liver volume segmentation method, with semi-automatic k-means clustering initialization followed by 3D RG algorithm, ensuring a parallel computational processing framework [57]. Wu et al. propose a supervoxels analysis, generated by the 2D simple linear iterative clustering, of images segmented with graph-cut algorithm [58]. Supervoxels are generated from a set of clustering algorithm types that divide the images according to texture distributions. Another feature-based segmentation using SVM classification of textural features with combined morphological operations was proposed by Luo et al. [59]. He et al. proposed a three-level AdaBoost-guided ASM to segment the liver in CT images, including an Adaboost voxel labelling initialization, a profile classifier, refined by an ASM mesh model [60]. Zheng et al. propose a feature-learning-based random walk method for liver segmentation using CT images. A learning step consisting on Haar, histogram of oriented gradient (HOG), local binary pattern (LBP), and gray level coocurrence matrix (GLCM) features are learned by an Adaboost guided Support Vector Machines (SVM) model, which generates automatic seed points on the original test image, to carry the automatic segmentation step via Random Walks algorithm [61]. Markov Random Fields (MRF) theory is another ML algorithm widely used in the literature. This algorithm considers that hidden node representing a label (e.g. object of interest, background, etc.) is assigned to each observation node (e.g. set of features extracted form voxel or a connected set of voxels). The method computes the hidden node configuration with the highest probability given the observation nodes and the built-in model. Alomari et al, proposed an MRF contour initialization followed by an active contour segmentation refinement [62]. Finally, a particular interesting method was proposed by Vijayalakshmil and Subiah [63]. The authors method performs a classification with local binary pattern images for categorising normal and abnormal liver images. They use texture feature extraction with LBP and Legendre moments, and validate the classification method with a dataset of 197 CT (77 normal, 125 HCC), achieving a very positive accuracy of 96.17%. Deep learning based methods From the machine learning field, emerged the sub-field of deep learning algorithms, which have presented a previously unseen performance by computational methods, in a variety of tasks. The field of deep learning methods has evolved from the first applications of these methods to signal processing and later to image analysis. Hence, deep architectures consist in specific kinds of neural networks, and range from Deep Belief Neural Networks (DBNs), Sparse Autoencoders (SAEs) and finally Convolutional Neural Networks (CNNs). From these basic types of architectures, based on the known Neural Networks architectures, each author s method is characterised by different neural network design. These design details range from the number of hidden layers, type of layer, layer distribution, type of activation among others. Hence, the methods technical characteristics are described briefly for each approach in the literature. The first applications of DL approaches to liver segmentation were proposed using SAEs. The first method using deep neural networks for liver segmentation was proposed by Shin et al. in a multi-organ segmentation pipeline applied to Dynamic Contrast Enhanced Magnetic Resonance Imaging (DCE-MRI) [61]. The authors base their knowledge in previous applications of deep architectures to object recognition in natural images and propose an SAE neural network to separately learn 256 visual and temporal features in an unsupervised and automatic manner, and detect multiple organs in a time series. The authors seek to propose the first deep architecture procedure for medical image segmentation tasks, tackling the substantial shape changes that characterize the abdominal organs such as kidneys, liver, heart among others, hence using a data sets from two studies of liver metastases and one study of kidney metastases. The SAE was characterized by sparsity Kullback-Leibler -based constraint application to logistic sigmoid activations, interleaved with max-pooling operations after each layer. A first step of tissue type labelling was performed, followed by separation of each organ recurring to a parameter optimization using to context-specific feature manipulation, to identify each organ in a supervised manner, obtaining a classification accuracy of approximately 64.8% for the liver. Lu et al. proposed a segmentation method combining a 3D CNN to obtain liver probability maps which would be further incorporated in the image information term of the energy functional of a GC algorithm. The authors validated the method on forty CT volumes, 20 taken from the MICCAI Sliver07 challenge and 20 from the 3Dircadb datasets. The authors use stacked convolutional and average pooling layers, trained by gradient-based backpropagation. The authors take advantage of the capability of GC 18

31 method of handling loose boundaries between tissues with similar intensity distributions. The proposed method is the first 3D CNN application to liver segmentation, and obtained a score of 77,8 by the Sliver challenge evaluation method [36]. Hu et al. proposed an automatic 3D CNN, trained to output liver probability maps complemented with a globally optimized surface segmentation. The authors used 42 CT images from MICCAI Sliver challenge train dataset and a local database, and validated the method on the challenge s test dataset obtaining an overall score of 80.3, surpassing any other methods previously proposed [35]. Christ et al. proposed a cascaded CNN in 2D with a 3D dense conditional random field (CRF) approach as a post-processing step, to achieve higher segmentation accuracy while preserving low computational cost and memory consumption [64]. Dou et al. presented a novel 3D CNN equipped with fully convolutional architecture and a 3D deep supervision mechanism to comprehensively address the challenges of volumetric medical image segmentation [37]. The deep supervision method, to which the authors name DSN, uses an objective function that guided the training of the upper and lower layer, propagating more efficiently the representation of the features, as well as speeding up the training process, by patch-based methods and volume-to-volume learning and inference. The authors CNN consists in 6 convolutional layers, 2 maxpooling layers, and one softmax output layer layers finalised with 2 deconvolutional layers, trained via stochastic gradient descent (SGD), in a fully convolutional architecture with 3D kernels, outputting a spatially arranged classification across the whole input pixels. The resulting predictions are rougher and obtained in low-dimensional layers, which is thus solved by the deconvolutional final layers. Furthermore, the authors emphasise the gradient instability that has been reported in previous works, where propagated fading, exploding or vanishing of gradient magnitudes occurs. A DSN mechanism is proposed in the network architecture by softmax extraction of outputs predictions from intermediate layers convolutional results, connecting them directly to the final output layer, and whose weighting is incorporated in the backpropagation objective function. Finally, the authors propose the incorporation of a fully connected CRF for segmentation refinement, which has advantage in capturing complicated shaped object such as those with holes or thin structures. The results were evaluated on the MICCAI Sliver challenge dataset presenting superior Volume Overlap Error and computational cost. From this point, the ISBI 2017 conference, released a challenge focusing on liver lesion segmentation, this time making available a total of 200 abdominal contrast-enhanced CT images of liver tumors. The dataset was obtained from from several different clinical sites with different scanners and protocols, thus having largely varying spatial resolution and fields-of-view. The in-plane resolution ranges from 0.60 mm to 0.98 mm, and the slice spacing from 0.45 mm to 5.0 mm. The axial slices of all scans have an identical size of , but the number of slices in each scan differs greatly and varies between 42 and The following three works presented, were developed under this scope of this challenge and will be reviewed in detail. Chlebus et al. proposed a two-step segmentation method, initiated by a liver volume segmentation followed by a liver lesion candidate detection and classification [65]. In the first step, a coarse segmentation was obtained from an ensemble of three CNNs trained with each of the three orthogonal interpolated volumes, followed by a 3D CNN liver refinement. The axial, sagittal and coronal volumes trained three 2D U-net models which whose softmax outputs were used to train a final 3D U-net that originated a full mask volume of original resolution. Based on the liver mask ROI extraction the second step consists in a twostep tumor segmentation using a 2D CNN for tumor segmentation, tumor candidate detection via 3D connected component analysis of the mask volumes, followed by a Random Forest tumor candidate filtering. Similar to the first step, a 2D U-net was used for the tumor segmentation and were trained with class balanced boundary patches of the axial dataset, however, the authors hypothesize that the patching step used for training penalized the specificity of these segmentations, requiring thus, the incorporation of a 46 feature extraction and classification step of these candidate tumor mask. The method scored a Dice coefficient of 0.65, and second place in the challenge, including a tumor candidate classification accuracy with the random forest approach of 90%. Bi et al. propose a cascaded deep residual networks (ResNet) approach to segment the liver lesions [66]. As pre-processing these authors apply data argumentation strategies including random scaling, crops and flips. ResNet uses shortcut connections to avoid training degradation though deeper layers, whereas the optimal results are calculated by an averaging output of the different networks. The key feature of the architecture proposed is the cascading of parameter learning, made from learning the training data and from the previous iteration result outputs. The testing segmentations are further incorporated with a multi-scale 19

32 rescaling strategies whose outputs averaging produce the final predictions. Given the architecture depth of 20 hidden layers training the entire volumes would become computationally non-viable, so the authors randomly pick a balanced dataset of axial slices, of a total of 8802 images to train the neural network via stochastic gradient descent. The network is pretrained firstly on the ImageNet dataset for parameter finetuning, which is then further fine-tuned with the liver dataset. Hoogi et al. focus their works on level-set parameter optimization via a CNN. The rough liver lesion probability maps outputted from a CNN are used to compute the contour initialization and level-set curve parameters [67]. Finally, Ben-Cohen et al. proposed a fully convolutional CNN, presenting a detailed architecture evaluation. The authors obtain a maximum 0.89 DSC for the whole liver segmentation validated on the Sliver dataset, and a true positive rate of 0.86 for liver lesion segmentation. Table III. Average values of performance measures and scores of different method results evaluated on the image dataset, using CNN-based methods. Measures reported with different segmentation accuracies. Author Method Dataset Accuracy Lu et al. [36] Liver 3D CNN, combined with GC refinement Sliver challenge: 30 CT VOE: 5.70 Hu et al. [35] Liver 3D CNN combined with globally optimized surface Sliver challenge: 30 CT VOE: 5.35; DSC: Dou et al. [37] Fully convolutional CNN with deep supervision Sliver challenge: 30 CT VOE: 5.42 Christ et al. [64] Han X. [68] Chlebus et al. [65] Bi et al. [66] Ben-Cohen et al. [69] Hoogi et al. [67] Liver and tumor CT CNN U-net, cascaded fcnn and dense 3D CRF Deep Convolutional Neural Network CNN followed by RF-based lesion candidate filtering ResNet based CNN with cascaded optimization CT: 5 healthy, 15 tumor LiTs challenge: 70 healthy, 130 tumor CT LiTs challenge: 70 healthy, 130 tumor CT LiTs challenge: 70 healthy, 130 tumor CT 0.94 DSC 0.67 DSC 0.65 DSC 0.5 DSC Liver and tumor 2D CNN 22 CT tumor 0.89 DSC Lesion CT CNN 2D CNN obtained probabilities are used to drive active contour model 112 CT, 164 MRI tumor 0.71 DSC 3.3. Liver and tumor segmentation in functional image Liver and/or liver tumor segmentation methods are very sparse in the literature is considered a more challenging task as a wide range of tumor tissue appearances makes for a difficult segmentation. Moreover, this segmentation is usually done on a noisy background as the low contrast of liver tumors mandates the use of contrast enhanced CT scans, increasing the amount of noise in images. Segmentation of hepatic tumors has only been explored in mainly, the literature records, to the best of our knowledge. Due to the specific intensity distribution these images have, thresholding methods intuitively present fairly good results. Magnander et al. presents a method to extract liver foci from SPECT images acquired with 111 In-octreotide. The authors propose segmentation with a thresholding method based on connected component analysis [70]. Hsu et al. propose an ASM method initialized by a Genetic Algorithm (GM) -generated initial contour to segment the whole liver in PET images [71]. Finally, a robust watershed method was also experimented for the task of liver segmentation. Shonket et al. experiment the performance of 2D and 3D Watershed (WS) method [72]. According to the watershed approach the gradient magnitude of a grayscale image is considered as a topographic surface. Voxels located on segment boundaries, where the gradient magnitude has local 20

33 maximum, correspond to watershed lines. The method virtually places water drops to each voxel position. From each voxel the water flows downhill to a local minimum. Voxels draining into the same local minimum form one basin and represent one segment of in the image. The simplest implementation of this approach uses priority queue to encounter all voxels of the gradient magnitude image starting from those located in a local maximum. Watershed segmentation usually divides images into large number of partitions. There are various strategies to merge smaller basins into larger ones based on different similarity criteria. Shonket et al. conclude that 2D WS presented higher volumetric accuracy than 3D WS, in the phantom studies performed. Phantoms model a range of tumor shapes, and structures, whose known dimensions allow the accuracy measurement of imaging equipments, and subsequently of segmentation algorithms Liver vessel segmentation Liver vessels are structures of interest when it comes to liver cancer surgery planning. Specifically, vessels are used to approximate liver segments. Region growing methods are one of the most common in the literature, because of their simplicity. Region growing segmentation of liver vessels has been reported as a pre-processing step in the literature [73]. Oliveira et al., for example, calculated a Gaussian mixture model of three Gaussians for liver vessels, nodules and parenchyma. The resulting mixture model was then used to derive thresholds for a region growing algorithm. The authors further propose a method that classifies branches to belong to the hepatic vein if they were longer than 15% of the liver height. Therefore, the biggest component in the first slice containing vessels is searched and merged with the biggest overlapping component on the next slice until no overlapping vessels are found anymore. This is repeated until three main branches are found. The portal vein is identified as the biggest connected component when the previously found hepatic veins are subtracted from the vessel segmentation [74]. Level set methods have also been proposed to segment liver vessels in their original formulation [75] and in a modified energy minimization problem, where the level set function is set to the output of a quadrature vessel enhancement filter. The resulting energy formula is solved using the calculus of variation and Euler-Lagrange method [76]. Methods using graph-cuts also showed potential in this task [74]. Pamulapati et al. registered contrast enhanced and non-contrast enhanced CT data to define a regional term derived from three energy functions. A data energy computes penalties on a 4D histogram of object and background voxels. Another energy penalizes voxels that do not rapidly enhance between both phases, and the third energy is derived from a vesselness function [74]. Drechsler et al. proposed a novel multiscale Hessian-based analysis for vessel enhancement and a and the proposed thresholded region growing method [73]. Only recently, machine learning methods were applied to this segmentation. Zeng et al. proposed the extreme learning machine method for liver vessel segmentation [77]. At first, the image noise is removed by anisotropic filter and then vessel filters like sato, frangi, offset medialness filter and strain energy filter are used to extract vessel structure features. Then extreme learning machine (ELM) is applied to segment liver vessels from the background. Shi et al. proposed an automatic liver segmentation method based on multilevel local region-based sparse shape composition with initialization based on the shape of blood vessels. The idea is that the liver vessels can be used to acquire patient specific prior information such as shape and size of the liver, providing a more accurate initialization [78]. Finally, very recently, Kitrungrotsakul et al. proposed a CNN method, composed of three stacked 2D CNN, in a deeply convolutional architecture, combining liver vasculature segmentations of sagittal, coronal and axial views. The authors validate their method on 12 CT volumes, obtaining an overall DSC performance valued as 0.83 [79]. Very few of the proposed methods in the literature make a performance analysis, aside from visual inspection. This is due to the time-consuming task that would represent manual annotation of liver vessels. 21

34 22

35 Methods The PhD project consists of four main work packages (WPs) executed in two stages. In the first stage (WP1) an extensive study of segmentation strategies with different Convolutional Neural Networks architectures will be evaluated to segment abdominal CT images. Next the further integration of the architectures developed will be adapted to a multi-modal segmentation, from PET and CT images, in WP2. In a WP3 stage the incorporation of key aspects of the liver for the target purpose of Liver resective surgery planning will be studied, integrating a vasculature segmentation and liver lobule map extraction. A final WP4 of integration the former three data sources, of liver, lesion, and lobule map division in an interactive visualization tool will be carried Work package 1 CT liver segmentation The study of liver and liver lesions has been tackled very recently in ISBI challenge, the Lits liver lesions challenge. The top scoring algorithms consisted in three different CNN architectures, obtaining a maximal accuracy of 0.65 DSC. There is still much, to be improved regarding CNN methods used for the liver and liver lesion segmentation based solely on CT images. The dataset of the Lits challenge is composed of 200 contrast -enhanced abdominal CT volumes, obtained from different scanner equipments (retrieved from different clinical imaging centres), and of images with different number of slices and resolution. The development of segmentation algorithms capable of extracting volumetric segmentations of these structures, accurately, regardless of the equipment and conditions of acquisition is very relevant in the clinical treatment planning of the patients. Accurate liver segmentation, isolating it from the surrounding anatomical structure is a critical component of a cancer CAD system. Data available: ISBI 2017 Lits challenge for liver lesion segmentation 200 contrast-enhanced CT. MICCAI 2007 Sliver challenge for liver segmentation 35 CT. Partner medical institutions providing 30 CT, at the moment. This number of studies can possibly increasing over time. Task Pre-processing: Image noise is present in CT volume images, which is one the fundamental issues with this type of images. One of the most successful algorithms is non-local means denoising, which has proven, in previous works, to be effective in boundary preservation and enhancement. Hounsfield unit conversion of the images has been reported as an advantageous pre-processing of the images, eliminating non-liver tissues and facilitating the segmentation task. Task Convolutional Neural Networks building: CNNs are models (networks) composed of many layers that transform input data (images) to outputs (object location probability maps) while learning increasingly higher-level features. A CNN will be used as model, and trained end-to-end for parameter optimization of the hidden convolutional operations towards liver and lesion segmentation. Different architectures can be evaluated, including post-processing methods for result refinement: 3D multi-scale convolutional neural networks (CNN) [80]. Deep convolutional neural networks (DCNN) present key representation robustness in their learning process to object locations, which is useful for object recognition tasks. Ben-Cohen et al. validated this approach in a reduced dataset [69]. 23

36 Research has shown that combining DCNNs and Conditional Random Fields (CRFs) can significantly improve scene parsing accuracy. For learning-based methods including CNNs, the learned predictions are usually refined by traditional methods for further accuracy improvement. Further fine tuning of the derived contours. Level-sets are a class of active contours that present very robust segmentations, by mathematical modelling of a signed distance function of an initial contour (zero level-set). The robustness of level-sets can thus be added to the final refinement of the segmentations derived from the probability maps generated from CNNs [81]. Implementations are expected to be developed using Google Tensorflow, Keras, Theano and Caffe python tools and adequate GPU processors. Task Validation of the segmentation approach: The methods performance will be evaluated with expert-generated references, considered the golden standard, and rated accordingly to detected deviations. Commonly used metrics based on volumetric overlap and surface distances, such as Volumetric Overlap Error, Dice similarity coefficient, Relative Volume Difference, Average Symmetric Surface Distance and Maximum Symmetric Surface Distance. The segmentations also will be reviewed by an expert collaborator. Manual segmentations, whereas metrics of accuracy between the segmented point clouds will be computed Work package 2 PET-CT tumour segmentation Very simple methods have attempted the present task of liver lesion segmentation using multimodal imaging. Specifically, only two research records were found. The first, consists in a semiautomatic model-guided segmentation, using a primary Statistical Region Merging algorithm followed by a model-to-image registration. The method was tested on both CT-only and PET/CT image volumes [82]. Secondly, a learning-based segmentation was performed using a binary tree quantization algorithm for clustering of image structures [83]. Both methods were applied to the fused PET/CT image. Task Tumor segmentation: The goal of using FDG-PET for tumor volume delineation is complementing the morphological information provided by CT, with the biological tumour data. Many segmentation algorithms have been applied in PET-only segmentations such as thresholding, region growing, classifiers, clustering, edge detection, Markov random field models, deformable models among others. It is intended to take advantage of the natural co-registration of the PET/CT images to integrate the previous ROI volume segmentation of the whole liver. Data available: Partner medical institutions providing 30 PET/CT, at the moment. The number of studies can possibly increasing over time. Multi-modal segmentation using CNN methods, based on both structural and functional image has been reported by several authors in segmentation tasks targeting other body structures. For instance, very recent works are being published targeting the field of brain structures and tumors segmentation, using multimodal images. The usage of the multimodal image data can be performed by different approaches. In classical CNN approaches, image data information is used, retrieved as high-level patch features, to which the subsequent layers hierarchically apply further transformations, generating in a final layer, final segmentation maps as outputs. These architectures can integrate multimodality. It is not of great value and does not allow the proper evaluation of the hierarchical feature learning from each individual modality, to use the originally fused PET/CT image data as input in CNN architectures. Multimodal image data, as has been used in previous studies, can be integrated by two essential strategies. Feature maps extracted from the input images can be fused into one feature map that is inputted into the CNN. Other architectures include an initial layered architecture of individually trained CNNs that act as feature extractors from each image type, which are in more deep layers shared and used for the final probability maps output. Illustrations of both examples are represented in Figure

37 Figure 4.1. Examples architectures of input feature maps integration from multimodal images [95]. Based on the literature, the CNN methods can be proposed using different approaches such as: Fused feature maps from different MRI contrast modalities have been used for brain tissue segmentations, by convolutional feature extraction, and subsequent deep convolutional layers, that enforce competitions between features at the same spatial location across the different feature maps [84]. Deep CNN architectures, can be composed of different feature extractor constituting individual CNNs, that can in deeper layers be joint by local response normalization layers [85]. Lesion candidates pre-selection in each modality, using strategies of PET thresholding, and CT pre-cnn liver mask and lesion detection, can be combined in a final CNN lesion candidate refinement [86]. The choice of an adequate architecture for this task will evaluated. The inclusion of multi-modality in this segmentation task is the main contribution of the proposed work plan, and has not been reported in the literature using CNN methods. Task 4.2.2: Experimental validation The developed automatic segmentation method requires the careful validation by an expert collaborator. Manual segmentation will be considered the gold standard, whereas metrics of accuracy between the segmented point clouds will be computed Work package 3 Liver segments map segmentation Liver resections are performed by physicians based on CT volume data, which mentally visualize the liver segments, lesions and vasculature positioning to plan the segments to resect. Analysis of the 3D vasculature structure of the liver is important to modulate and improve surgical and resection planning, as well as radiotherapy puncture location decisions. Moreover, it delimits the liver segments, which determine the surgery strategy adopted and aids in predicting postoperative residual liver capacity. Liver segments are divided by the Couinaud s segmentation standard into nine regions. Task Liver vasculature segmentation - Recent advances in machine learning methods, deformable models are becoming very popular for liver vessel segmentation. Usually, an initial segmentation is achieved using machine learning algorithms such as pixel clustering methods and then refined by deformable models such as level sets. Validating vessel segmentation results are very difficult since these validations are generally obtained using corrosion casts that require expensive equipment and 25