To mute the current speaker, so that you cannot hear their audio click on the icon of the speaker, shown to the left of "Current Speaker"

If the speaker icon looks the same as above, you will be able to hear audio from whoever is the Current Speaker.

If the speaker icon has a red X by it (shown above). You will not hear the audio coming from the Current Speaker. It can sometimes take a few seconds for the audio to stop, after muting the Current Speaker.

These settings stay that way until you change them manually. So remember that the current speaker may change from time to time and you may want to to un-mute the new Current Speaker.