The aim of this study was to examine challenges faced by university students with special needs (SwSNs) and the implications the challenges have on such students’ retention. The study adopted a cross sectional survey design employing both quantitative and qualitative approaches. The study participants (N=34) were SwSNs who were purposively selected from three universities (comprising: two public, and one private university). Quantitative responses were analysed using simple descriptive data analysis techniques, to generate frequency counts and the respective percentage scores. Thematic content data analysis was used to analyse qualitative data. The researchers further applied the Tesch’s (1990) procedure of open coding method of data analysis, which involves a systematic process of examining, selecting, categorising, comparing, synthesising and interpreting data to unpack the major research question of the study. Findings reveal that SwSNs in some Ugandan universities experience challenges such as: lack of suitable facilities, difficulty to access learning materials, lack of understanding and respect from both fellow students with special needs and the ordinary students (those without special needs). One of the findings is that some of the key challenges faced by SwSNs were lack of coordinated efforts across the universities. It is further established that SwSNs feel they had to work harder than other students, because they had to manage a double workload. Generally, these challenges negatively affected SwSNs’ retention. The study recommends that for SwSNs to be assisted to overcome the existing challenges, the universities should provide necessary and appropriate facilities as far as is possible with the resources available and without compromising academic standards.