A discussion on Review Ratings

[Wraith]

...and to contrast....I had an eleaborate review by someone who hated the source material, and still hated the edit...roll on poor all round scores...not even the technical were spared...

But I do concur... a detailed review is what lights up the room for me...good or bad...as long as it is honest and heartfelt...
We all make mistakes, and I for one like to clear them up..

Back to the "guide"lines then...

 

[Wraith]

There is also the question of FILE SIZE...

I often am asked for smaller files, so that viewers can watch on their 4 inch phones...

I long stopped doing that, but there is still an issue that a 4GB file on a 55 inch display will NEVER look as good as a BD Compliant M2TS file that is passthru rendered at 35 to 40 mbps...BUT is 40 GB in size...

My bandwidth gets crushed, not all viewers have the set up to watch it in the best manner...and so now only a handfull of my edits are available in MAXED out UBER size...there will always be a trade off in how edits are delivered.

How does one calibrate for DISTRIBUTION STANDARDS let alone viewing set ups...?

And thats before we get on to cheaper systems and setup NOt hvaing the codecs to view correctly.

 

[tremault]

Glad to hear we are on the same page! Though please dont take this personally but if someone says he have it an 8/10 based on his own personal standards and that upset you that sounds a bit fragile. I mean if he had given it a 4/10, sure. But an 8/10 is a respectable score. If I get an 8/10 for anything I'm ecstatic lol.

You cant preach for more freedom then expect 10/10 reviews because it suits your narrative.

Well I have had lots of 8s and as low as a 6 and I sometime spit out my drink but then I laugh it off. I'm not fragile, don't worry about that, it was more the fact they said that, when the guidelines of the site seem pretty clear that the rating is meant to compare against the original.

 

[tremault]

...and to contrast....I had an eleaborate review by someone who hated the source material, and still hated the edit...roll on poor all round scores...not even the technical were spared...

But I do concur... a detailed review is what lights up the room for me...good or bad...as long as it is honest and heartfelt...
We all make mistakes, and I for one like to clear them up..

I concur, I do feel a sting if the numbers are low, but I totally appreciate the writing that goes along with it. I drink that up and enjoy the interaction with that person. I also learn a lot from those kinds of reviews. Very rewarding.

 

[Wraith]

Personally, I am nearing the point where unless a REQUESTOR, has at least XX interactions on the site/forum and posted at least YY reviews already....I may not facilitate viewing to the DRIVE BY BRIGADE....
They are wearing me out...

 

[Wraith]

I concur, I do feel a sting if the numbers are low, but I totally appreciate the writing that goes along with it. I drink that up and enjoy the interaction with that person. I also learn a lot from those kinds of reviews. Very rewarding.

I had a review for my Ep 8 of TTT, which was well reasoned and low scored.

I engaged with the reviewer...we had a productive discussion...they tweaked their score (not perfect) and then argued back to ME what would make the edit perfect.

Their reasoning was not only sound, but was a mirror to my own bias that had held me back from not only including some of tjose tweak, but my own ego was risking ignoring the sound observation.

I went in and made the 3 suggested tweak, plus two additional tweaks that were consequential to those changes....and sure, they then cam back with a 10. BUT, There was one point I held firm on, and did not act on. however, what started as a heated debate taught us BOTH where the other party was coming from, and the end result was...

A HAPPIER VIEWER
A HAPPEIR EDITOR
A BETTER EDIT for ALL

Is that not what we should all strive for?

 

[Wraith]

On a seperate note, the below scale has issues when editors produce ambitious and narratively challenging edits...
10 = Flawless
9 = Almost Undetectable
8 = Smooth, but slightly noticeable
7 = Noticeable, but not jarring
6 = Slightly jarring
5 = Jarring/distracting

I've watched edits where changes a JARRING or NOTICEABLE, but are 100% technically fine and only noticeable coz I know the source material so well..

These would need some adjustment for Narrative etc to work...

In many cases a so called JARRING cut/juxtaposition is chose for dramatic or narrative progression...

Over to the BRAINSTRUST me thinks...(my ears and eyes are bleeding just thinking about this all)...

 

[DonkeyKonga]

Heres my take after reading all this.

Dont change the current system. These are user reviews, as such it's to be expected that these users (partly) hold their own standards. Appreciate the effort of people writing reviews, as they will always have good intentions, considering most people dont even bother to leave a review.

However, you could add maybe a 'Highly recommended' button or something or a 'Must see' button to seasoned reviewers with x amount of reviews under their belt. Like a *top reviewer* status, reward those who leave reviews often and give them for example a 'highly recommendes' badge only they can give to an edit. Also their reviews show on top when you look at a fanedit on IFDB.

That way you can still have the popularity of superhero/star wars movies, but also give the more obscure edits a bit of love ánd reward people who review regularly! Just an idea.

 

[Wraith]

Dont change the current system.

However, you could add maybe a 'Highly recommended' button or something or a 'Must see' button

I LIKE!!!

 

[DonkeyKonga]

I LIKE!!!

Caps lock and everything! Haha yes in the end i think the solution lies in rewarding reviewers and not restricting reviewers. I think that's the only option all parties will agree on.

 

[Milesman1971]

I’ve said this in previous discussions, but as a non-editor, I find the numerical scores to be completely useless. Worse yet, I think those scores often lead to reviewers not feeling the need to elaborate on those technical areas in the narrative reviews. As a result you end up with reviews that read like this:

“This was great edit. I didn’t notice any editing flaws and the cuts were mostly good. It replaces my theatrical version. Great work!”

That tells me very little and honestly is how most reviews end up reading more or less. If you scrap the number ratings altogether and replace them with separate narrative windows perhaps you’ll get more detailed reviews and a more useful guide for viewers.

I'm with you (Roger Ebert used to say something similar about his reviews - read the review, forget the arbitrary rating), but given the lack of detail in so many of the reviews, we often have no choice but to look at the cumulative score.

My biggest issue is there is little room for nuance in the scoring, so a slightly lower score can have a disproportionate effect on the cumulative rating. I am relatively new here, so forgive me if this has already been litigated, but why not allow reviewers more options for scoring? If I can only drop the Audio score from a ten to a nine, I am reluctant to do so because I know the review will harm that edit's score more than I'd like it to, particularly when an edit is new and has fewer reviews. If I can drop the Audio score to a 9.7 or a 9.4, for example, that allows me to deduct points appropriately without hitting it too hard.

If everyone were scoring edits correctly, the current system would be fine, but given the reality of how reviewers, myself included, have been utilizing it, more options would allow us to stop giving perfect scores without adversely impacting the ratings of edits we think are excellent if imperfect.

I do like the idea of narrative windows, perhaps even with minimum character requirements. Not sure if that would discourage reviews, though, and you certainly don't want to do that.

 

[DigModiFicaTion]

Based on friendship? People cant be friends and review? That's a slippery slope right there. I think realistically FotM is already partly a popularity contest. Which makes sense considering the rules. You need to own a movie to watch an edit. The more popular a movie the more likely it is to win FotM.

Same goes for popular people. That's just the way humans work.

What you're searching for is a seperate review system where only a select group of reviewers review an edit. 'Official' reviews for a lack of better words. It's a bit much for a fanedit community tbh.

You're making more of what I said than what I actually said. Some people have expressed that they don't want to give an honest score because they don't want to offend someone. That is putting an emotional connection before the honest critique.

No we aren't looking for a separate review system. We're asking people not to make one by devaluing the actual rating scale.

The prequels are a result of people not wanting to offend a popular person. We want to avoid that here.

 

[DigModiFicaTion]

If everyone were scoring edits correctly, the current system would be fine, but given the reality of how reviewers, myself included, have been utilizing it, more options would allow us to stop giving perfect scores without adversely impacting the ratings of edits we think are excellent if imperfect.

The ratings were originally expanded to 10 from 5 to allow more choice. This thread is to help calibrate what each score actually is so we don't have a bunch of false positive or negative ratings and so we can ensure that we are all collectively upholding the core of IFDb, the highest quality edits.

 

[Wraith]

We do also need to bear in mind, edit quality is generally improving with the tech, increased knowledge and availability of advice and tools and so much more.

So scores will creep up… I watched a first time submission recently which frankly blew my socks off…

So , and we discussed this before, it’s gonna get near straight 10s from me… but how does one differentiate an extended movie, which may be perfect, but has no where near the technical difficulty and creativity …
ANSWER:
The written word.

Complexity is just not a factor that is captured in all honesty…astute reviewers pick up on it and comment… and then there are those who miss it completely… which ironically, is what we actually want… that no one sees the joins…

Kobayashi Maru me thinks.

 

[DigModiFicaTion]

Personally, I am nearing the point where unless a REQUESTOR, has at least XX interactions on the site/forum and posted at least YY reviews already....I may not facilitate viewing to the DRIVE BY BRIGADE....
They are wearing me out...

I'm honestly struggling to understand what you mean by this post.

On a seperate note, the below scale has issues when editors produce ambitious and narratively challenging edits...
10 = Flawless
9 = Almost Undetectable
8 = Smooth, but slightly noticeable
7 = Noticeable, but not jarring
6 = Slightly jarring
5 = Jarring/distracting

I've watched edits where changes a JARRING or NOTICEABLE, but are 100% technically fine and only noticeable coz I know the source material so well..

These would need some adjustment for Narrative etc to work...

In many cases a so called JARRING cut/juxtaposition is chose for dramatic or narrative progression...

Over to the BRAINSTRUST me thinks...(my ears and eyes are bleeding just thinking about this all)...

We intentionally want it to be free for interpretation at that level. We don't want to police ratings, but we do want people to give accurate and honest ratings.

War of the Stars is a great example for us to look at in this category. It's DRASTICALLY departed from the source. BUT, one should know that going in. The jarringness of the cuts can still be objectively ascertained even if the flow is not in line with the original. For me, War of the Stars maintains the quality of its source and is most likely a 9 or a 10 for A/V quality. The audio work in it is pretty great. I'd probably give it an 8 or 9. The visual editing is good, but there are some noticeable mask lines. I'd probably give an 8 knowing that it came out before some of the more technical software was available. The narrative flows incredibly well for such a departure. It's probably an 8 or 9 for me in that department. My enjoyment is a 7 or 8 due to the fact that I just don't care for grindhouse gore, but I love that it challenged my thoughts on what is possible narratively and editing wise.

We do also need to bear in mind, edit quality is generally improving with the tech, increased knowledge and availability of advice and tools and so much more.

So scores will creep up… I watched a first time submission recently which frankly blew my socks off…

So , and we discussed this before, it’s gonna get near straight 10s from me… but how does one differentiate an extended movie, which may be perfect, but has no where near the technical difficulty and creativity …
ANSWER:
The written word.

It's not about technical difficulty. It's about technical proficiency. One might have an issue with this if one is comparing edits and ordinal ranking. That rank itself is based on an average that attempts to accommodate ratings based on the integrity of a score rather than the quantity of a score.

Complexity is just not a factor that is captured in all honesty…astute reviewers pick up on it and comment… and then there are those who miss it completely… which ironically, is what we actually want… that no one sees the joins…

Is that an issue? We generally don't allow such as a first time submission as it doesn't show if an editor can sustain quality, but why should we punish someone for doing a few excellent edits in an extended edit because we think it has less technical worth than another. If someone nails it in an extended, they nailed it.

Kobayashi Maru me thinks.

Possibly if the expectation is how it effects the ordinal ranking.

 

[DonkeyKonga]

You're making more of what I said than what I actually said. Some people have expressed that they don't want to give an honest score because they don't want to offend someone. That is putting an emotional connection before the honest critique.

No we aren't looking for a separate review system. We're asking people not to make one by devaluing the actual rating scale.

The prequels are a result of people not wanting to offend a popular person. We want to avoid that here.

And I'm saying its inevitable that user ratings will have *slight* deviations in standards, with examples that most seem to agree on. Which is that written reviews are valued more than the ratings attached to them. That's the general consensus.

So with that in mind the written part of the review should be encouraged. Not necessarily the proper use of a numeral rating system. Because at the end of the day I can understand where the other viewpoint comes from, but I also think I've made a few valid arguments.

However the written down part we all agree on: That's the part of the review with the most merit. And at the end of the day it's about the difference between and 8/10 and a 10/10 which is honestly just sad when you think about it. These strangers take time and effort to write an entire review and we argue about wether they used the right accompanying number to what, stroke our ego enough? And then we go a step further and have to check wether someone feels pressured into giving a 10 for whatever reason? Like that's just over complicating things instead of making the reviewing proces more fun. Which you can do by rewarding reviewers as I mentioned earlier.

Let's keep reviews fun, and remind ourselves that anything that's an 8 or higher is a solid rating whatever way you look at it. There is no outcome where the ratings match everyone's opinion. So why not deviate from this and find a solution in a different, actually fun way?

 

[DigModiFicaTion]

And I'm saying its inevitable that user ratings will have *slight* deviations in standards, with examples that most seem to agree on. Which is that written reviews are valued more than the ratings attached to them. That's the general consensus.

And that's ok. We just want to make sure that we aren't running into a situation where we're collectively 2+points off. That makes the rating system useless.

So with that in mind the written part of the review should be encouraged. Not necessarily the proper use of a numeral rating system. Because at the end of the day I can understand where the other viewpoint comes from, but I also think I've made a few valid arguments.

As per the pre existing guidelines, it has and continues to be strongly encouraged.

However the written down part we all agree on: That's the part of the review with the most merit. And at the end of the day it's about the difference between and 8/10 and a 10/10 which is honestly just sad when you think about it. These strangers take time and effort to write an entire review and we argue about wether they used the right accompanying number to what, stroke our ego enough? And then we go a step further and have to check wether someone feels pressured into giving a 10 for whatever reason? Like that's just over complicating things instead of making the reviewing proces more fun. Which you can do by rewarding reviewers as I mentioned earlier.

In all honesty this statement is frustrating to me as I feel almost the opposite is occuring. Those who have historicially complained about ratings have done so about those who rate low, not high. As a result, people don't rate accurately as much anymore as it is almost frowned upon to not gush over someone's edit.

Let's keep reviews fun, and remind ourselves that anything that's an 8 or higher is a solid rating whatever way you look at it. There is no outcome where the ratings match everyone's opinion. So why not deviate from this and find a solution in a different, actually fun way?

Reviews can be fun and accurate. I would agree anything 8 or higher is solid. A 7 is good. A 6 is something that most likely didn't receive enough vetting. Anything lower and it's probably not ready to be listed.

 

[DonkeyKonga]

And that's ok. We just want to make sure that we aren't running into a situation where we're collectively 2+points off. That makes the rating system useless.

As per the pre existing guidelines, it has and continues to be strongly encouraged.

In all honesty this statement is frustrating to me as I feel almost the opposite is occuring. Those who have historicially complained about ratings have done so about those who rate low, not high. As a result, people don't rate accurately as much anymore as it is almost frowned upon to not gush over someone's edit.


Reviews can be fun and accurate. I would agree anything 8 or higher is solid. A 7 is good. A 6 is something that most likely didn't receive enough vetting. Anything lower and it's probably not ready to be listed.

And if you feel like we've already reached that point where we feel we have to gush over an edit then honestly we probably have already and there is no turning back. Because what's the alternative? Removing all previous reviews? Having a review police department? Because lets face it the avarage reviewer wont read this thread.

If I have to choose between reviews that are too stern or reviews that are too lenient I'll choose too lenient. YOU GET A 10/10 YOU GET A 10/10 EVERYONE GETS A 10/10! *insert Oprah gif here*

An example.of skewed reviews can be found in my Mario edit. I did the voice over and people love it. But if I am 100% honest it's not a 10/10 audio quality voice over. But i understand that people are excited, and dont want to give it a 9/10 because of the amount of work it is. Same happens with edits the Hobbit. I mean we can talk about the technical element of a review system, but we cant dent that its 100% human to reward something that took a painstaking amount of work, even if the end result isnt *quite* perfect.

 

[DigModiFicaTion]

And if you feel like we've already reached that point where we feel we have to gush over an edit then honestly we probably have already and there is no turning back. Because what's the alternative? Removing all previous reviews? Having a review police department? Because lets face it the avarage reviewer wont read this thread.

If I have to choose between reviews that are too stern or reviews that are too lenient I'll choose too lenient. YOU GET A 10/10 YOU GET A 10/10 EVERYONE GETS A 10/10! *insert Oprah gif here*

I'd respectively ask you to refrain then, but you can choose what you want. There is never a too far moment unless people themselves have given up. Also, you're talking in absolutes and implying things that we've never said were going to happen. This calibration is something for going forward. If people wish to retroactively update their reviews that's fine. It certainly won't be required, but anyone who chooses to do so shouldn't feel any fear for doing so.

 

[tremault]

However the written down part we all agree on: That's the part of the review with the most merit. And at the end of the day it's about the difference between and 8/10 and a 10/10 which is honestly just sad when you think about it. These strangers take time and effort to write an entire review and we argue about wether they used the right accompanying number to what, stroke our ego enough?

Personally, I think it's more a case of whether our work is likely to get seen. I see many new releases going by with very little attention and it's sad. I want those editors to feel the encouragement I have. I'd watch some of them myself but I have a constant battle with the clock XD