From their POV if he's getting viewers they shouldn't really care.
I won't be watching Raw if he's not on and I'm sure many are the same
I do see your point from a creative standpoint though, it's harder to make someone an attraction if they're on every show