First language (L1) phonological knowledge may influence second language (L2) processing at different levels. The current study disentangles L2 low-level perception and a higher level of phonological encoding in the lexicon. L1-Mandarin L2-English bilinguals listened to English low vowel + nasal (loVN) contrasts in which the vowels pattern as allophones in Mandarin. The AX discrimination task revealed that sequential Mandarin-English bilinguals accurately perceived the sequences. A spoken word recognition task in a Visual World Paradigm, however, showed that in response to words with loVN sequences Mandarin-English bilinguals experienced more lexical competition than English L1 listeners in a relatively later time window. Taken together, this suggests that the influence of Mandarin phonology affects phono-lexical encoding to a greater extent than lower-level phonetic encoding. Moreover, the symmetric L2 competition between the L1-licit and L1-illicit loVN contexts suggests that L2 listeners are able to repurpose L1 allophones to phonemes in the L2. In addition, across language backgrounds, a larger receptive vocabulary size facilitates spoken word recognition. The phonological knowledge generalized across larger lexicons resolves competition more quickly for both bilingual and monolingual listeners. Overall, the study suggests that allophones, rather than phonemes, are mappable units in phono-lexical encoding by sequential bilinguals.