In Arabic morphology (علم الصرف), the مصدر (verbal noun / masdar) is the noun form that expresses the action of the verb without reference to time or subject.
The verb استمع (istama'a) means "to listen." It follows the pattern استفعل (ifta'ala), which is the eighth verb form (الباب الثامن). The standard masdar pattern for استفعل verbs is استفعال (ifti'al). Applying this pattern to the root س-م-ع:
استمع → استماع (istima', meaning "listening")
The other options represent different morphological forms: سماع is the masdar of the base form سمع (to hear), not of استمع. سمّاع is an intensive adjective form (صيغة مبالغة) meaning "one who listens a great deal." مستمع is the active participle (اسم الفاعل) meaning "a listener," not a verbal noun.