Originally a compound of 根(ne, “root”, more specifically signifying somewhere hidden away) + 住み(sumi, the 連用形(ren'yōkei, “continuative or stem form”) of verb 住むsumu, “to live in a place, to reside”), meaning roughly “one who lives in hidden places”. The sumi changes to zumi as an instance of rendaku (連濁).
Japanese does not generally distinguish between mice and rats, and both are commonly called nezumi. If a distinction is needed, speakers may use the adjectives 大きい(ōkii, “large”) or 小さい(chīsai, “small”), or may use the word マウス(mausu, “mouse”, borrowed from either German or English).
As with many terms that name organisms, this term is often spelled in katakana, especially in biological contexts, as ネズミ.
Note that, although the Japanese term 子年 (nezumidoshi, nedoshi) for The Year of the Rat in the Chinese zodiac does derive from this same nezumi term, it is not written with this 鼠kanji.