While I absolutely agree Trek works best when it has an outsider character to provide an alternate perspective and voice, I think the issue with VOY and ENT was just bad writing or what I call the Braga Influence.... LOL.
I mean VOY, along with the Doctor (who I also agree was the best character of the show) had a Vulcan and Half-Klingon, which they almost completely ignored. And Neelix and Kess.... why were they even there? Great potential alien characters that were for the most part left under developed for dull human centric stories.
ENT had Dr. Phlox, who provided some interesting insights but for the most part was relegated to humour. T'Pol did provide an alternate voice, often proving the Captain wrong on how to interact with other species and she brought new elements to how Vulcans were portrayed (though some would argue this has more to do with the actress's acting ability or lack of....? But I enjoyed her performance). In fact, if Braga did one thing right with Ent, it was his new, if controversial, take on Vulcan.
But I think both ENT and VOY suffered more from "Been-There, Done-That" Syndrome than anything.
DS9 clearly proved, you can take established Trek alien races, and breath new life into them and make them fresh and exciting again.
So I don't think you
must have a totally unique character, ( I speaking in terms of using established alien races), you just need to be bold and do new unique things with them.
When a franchise has been around as long as Star Trek, with countless tie-ins like novels, fans begin to think they "know" everything about a certain fictional character or race or planet. When actually, due to the restraints of tv storytelling, we only ever really see the smallest sliver of a character's world or background onscreen, but we extrapolate from that in our imaginations and think we know more than we do.
I still find it funny at the fan outrage about Sybok being Spock's half brother. I never had an issue with this. It seemed totally in keeping with Spock's character and the extreme privacy he seemed have about himself and culture... not to mention, there were only maybe 3 episodes in it's entire 79 episode run that dealt directly with Spock's history or Vulcan... that leaves a lot of unexplored territory. Plus, narratively, I thought it was brilliant. Spock, who is half human struggles his whole life to be the perfect unemotional Vulcan, while his full Vulcan brother casts it all aside to embrace his emotions. It's classic.
But like anything, in the end, it really just comes down to new ideas and good writing. And Star Trek, under the reign of Rick Berman, was completely exhausted and mined dry by the time of Voyager and Enterprise. But as long as it was still turning a profit, Paramount/CBS didn't care about protecting and shepherding the franchise towards a long future. It's a shame really. But at the same time, maybe it was appropriate that new Trek should vanish from out view screens for more than a decade, for even audiences can take something for granted and the only way to realize one misses and wants or needs something is to not have it....lol