Healthcare sector is totally different from other industry. It is on high priority sector and people expect highest level of care and services regardless of cost. It did not achieve social expectation even though it consumes huge percentage of budget. Mostly the interpretations of medical data are being done by medical expert. In terms of image interpretation by human expert, it is quite limited due to its subjectivity, complexity of the image, extensive variations exist across different interpreters, and fatigue. After the success of deep learning in other real-world application, it is also providing exciting solutions with good accuracy for medical imaging and is seen as a key method for future applications in health sector. we discussed state of the art deep learning architecture and its optimization used for medical image segmentation and classification. In the last section, we have discussed the challenges deep learning based methods for medical imaging and open research issue.