This study investigates the mechanisms underlying the processing of second-language (L2) speech by native and non-native listeners. Two issues regarding Interlanguage Speech Intelligibility Benefit (ISIB) were explored: Whether non-native listeners have an advantage over native listeners in understanding accented speech (i.e., ISIB-Listener), and whether non-native listeners find accented speech more intelligible than native speech (i.e., ISIB-Talker). Words where accents are realised at different linguistic levels (i.e., phonological, lexical, prosodic) were examined using a word transcription task. In Cantonese-accented English, ISIB-Listener was found when a more frequent word (e.g., thin) was mispronounced as a less frequent one (e.g., fin). In Cantonese-accented Mandarin, ISIB-Listener was found for Cantonese listeners for words that were mispronounced by Cantonese speakers due to negative homophonic transfer from Cantonese, while ISIB-Talker was observed for Mandarin tone-mispronounced words by Cantonese listeners with low Mandarin proficiency. The results indicate that L2 proficiency and choice of stimulus items are important factors mediating whether ISIB can be found. A three-route L2 Mandarin word production and recognition model was proposed to account for the ISIB observed in Mandarin. In this model, a concept route links the concept and the L2 Mandarin phonological representation directly, while a lexical route links the concept and the L2 Mandarin phonological representation through the L1 Cantonese phonological representation. A sublexical route activates the concept and/or the L2 Mandarin phonological representation through the mediation of L1 Cantonese phonological and sublexical representation using Cantonese-Mandarin pronunciation correspondences at all sublexical levels (i.e., onset, rime, and tone). Beginning learners of Cantonese mainly use the sublexical route in L2 Mandarin word production and recognition. Advanced learners gradually shift from the sublexical to lexical/concept route in producing Mandarin words while both lexical/conceptual and sublexical routes are still in active use to generate possible word candidates in L2 Mandarin word recognition. Further support for this model was obtained from a character-sound matching task and a Mandarin pinyin transcription task. The computational processes that might occur when a Cantonese speaker starts to learn Mandarin words were also presented.