An issue exists in Mattermost Server prior to 5.8.0. It does not always generate a robots.txt file.
mattermost mattermost server