Homeworld was the better game as far as I am concerned. Homeworld did everything so much better, the story (both the backstory and the campaign) was one of the best I have ever seen, the gameplay was great, and the music was the icing on the cake.
I have only played the demo for Homeworld 2, and I wasn't that impressed, I didn't like the voice acting, and the gameplay was okay, strike craft squadrons and the improved interface were the only things I really liked. Unfortunately, good luck finding anything Homeworld nowadays. I have never even heard of a 3D RTS that was as good as Homeworld, not even its sequels.
GoaFan77 is right as far as I know, I have never heard of a game that fulfills those requirements.