A man is defiled by what come out of him, not by what goes into him. Jesus is concerned about our hearts. He makes this point throughout the Gospels. The Beatitudes are an example where he equates anger to murder and lust to adultery. Whether or not you actually physically commit murder or adultery, is not any more important to God than if you desire these sins in your heart.