I'll be honest. I was really enjoying S4, with the exception of Connor doing Cordy. There were some great episodes, such as the first episode, Faith trilogy, the episode in which the Beast summons the rain of fire, The Beast destroying W&H, etc....
That being said, after Angel gets his soul back, it went downhill, IMO. The story was all over the place.
The finale was great, though.
S5 is very polarizing: People either think it's the best thing Whedon ever worked on, or they think it was one of the weaker seasons.
Definitely looking to S5 either way. I always enjoyed the Wolfram & Hart guys. A fascinating set of villains. It really is a shame Lilah isn't coming back, since her character's really grown on me this season.