Ratings of Mormon Movies on IMDB

After writing a silly post a few weeks ago with suggestions for movies the Church should make, I got to wondering how actual Mormon movies are rated on IMDB. I’m thinking here not just of Church-made movies, but of movies made about Mormons by non-Church studios, stuff like God’s Army or Saturday’s Warrior. I didn’t have any particular questions I wanted to answer with the data or any hypotheses to test. I just thought it might be interesting to look descriptively at what the ratings look like.

I got ratings from IMDB for a list of 75 Mormon movies. I made an initial list working from the following sources:

• Movies I have personally seen or am aware of
• Movies at least 30 minutes long on the Hard-to-Find Mormon Videos YouTube channel (which is for Church-produced movies only)
• Movies appearing on the Wikipedia page on Mormon cinema
• Movies Google suggested when I searched for “Mormon movies”
• Movies appearing on some user-created lists of Mormon movies on IMDB

I dropped from the initial lists any movies that met any of these criteria:

• Have fewer than 10 ratings on IMDB
• Don’t prominently feature Mormons or Mormonism, unless the movie is made by the Church (e.g., Johnny Lingo is included even though it doesn’t mention church because it was made by BYU, but Napoleon Dynamite, which showed up on some of the searches and lists, is excluded)
• Don’t take a positive view of Mormonism (This is an easy call for movies like The God Makers, but I also made some judgment calls based on descriptions and reviews I read on IMDB. For example, I used this rule to exclude the 1950 movie The Wagon Master, which has the protagonists guiding a group of Mormon pioneers, but it sounds like isn’t really a movie about Mormons, and the Mormons are more just neutral background.)

I grouped the 75 movies into four categories:

• Church produced, scripture or Church history topic – 7 movies
• Church produced, other topic – 6 movies
• Not Church produced, scripture or Church history topic – 16 movies
• Not Church produced, other topic – 46 movies

Ratings by Movie Category

One thought I had when looking at these ratings is that perhaps raters of Mormon movies, who I would expect to be largely Mormon themselves, might give these movies high ratings for reasons other than (or in addition to) thinking they were good. For example, giving Mormon movies high ratings might be seen as a kind of missionary work, because if you can inflate their ratings, maybe non-Mormons will be more likely to take notice and watch them and discover how great the Church is. I also thought these extra reasons might be more of a factor in rating movies that are closer to the core mission of the Church, so they would have the greatest effect for movies that are produced by the Church, or are about a scripture or Church history topic, or both.

This graph shows the average rating by category. Note that this is the average of movie averages, not the average of the individual ratings. I chose this to avoid having one movie count for more than another just because it received more ratings. I also wanted to use IMDB’s weighted average ratings, where they adjust the rating “in order to eliminate and reduce attempts at vote stuffing by people more interested in changing the current rating of a movie than giving their true opinion of it,” and these ratings are only available for movies as a whole, not for individual raters.

It looks like the primary difference here is just between Church-made movies and non-Church-made movies. Of course, there’s no evidence that the Church-made movies aren’t actually better by whatever standard, but I do think it’s interesting that, in line with my guess, they are generally rated higher than the non-Church-made movies.

An aside: I apologize if you’re looking for me to do statistical significance tests. The sample sizes are pretty small so I didn’t even bother with any, figuring that they would mostly be non-significant, and even if I did find significance somewhere, it would likely just be a result of my having done lots of tests and chanced upon a large difference. So feel free to ignore all differences if you don’t like unsupported speculation.

Ratings by Rater Gender

IMDB also provides rating breakdowns by gender of the rater, although it’s important to note that not all raters report their gender, as the women’s counts and men’s counts add up to less than the total count for virtually every movie I’ve looked at. This next graph shows the breakdown by rater gender and movie category.

It looks like women’s pattern of ratings follows exactly what I was expecting, being highest for Church-made movies about scripture and Church history topics, and lower as movies are about other topic, or are not Church made, or both. Men’s pattern, on the other hand, is like the overall pattern, with the difference appearing mostly just between Church-made and non-Church-made movies. That the men’s pattern and the overall pattern are similar is not a coincidence: Most IMDB raters who report their gender are male.

But of course the larger question is why women rate these movies higher than men do. I skimmed through a bunch of random movies on IMDB and didn’t see a consistent gender difference. (To be complete, I should note that for the highest-rated movies on IMDB, I did see a gender difference, but in the opposite direction. The highest-rated movies are generally rated higher by men than by women. This isn’t interesting, though, because the large majority of IMDB raters who report their gender are men, so the list of the highest-rated movies on the site is pretty much just a list of the highest-rated movies by men.)

Another thought I had is that perhaps women give higher ratings to Mormon movies because women on the whole are more religious than men on the whole (at least in the US; one of my first posts on ZD dealt with this issue and links to some survey data on gender differences in religiosity.) The comparison group this hypothesis suggests, then, is non-Mormon religious movies. I decided to look at Christian movies, and using similar searches as I did for the Mormon movies (Google searches, IMDB and Wikipedia lists), I recorded men’s and women’s ratings for 50 Christian movies that had at least 1,000 ratings. Also, again to parallel what I did with the Mormon movies, I put them into three categories:

• Scripture based (e.g., Jesus of Nazareth)
• Scripture adjacent (e.g., Left Behind: The Movie)
• Inspirational story (e.g., Overcomer)

This graph shows the average ratings by women and men and by the three categories.

Following the same reasoning as with Mormon movies, I expected that the scripture based would be rated the highest, followed by the scripture adjacent, followed by the inspirational stories. The scripture based are rated the highest, but the inspirational stories beat out the scripture-adjacent stories.

More importantly, though, for the reason I originally decided to look at Christian movies, there’s a clear gender difference that looks a lot like the gender difference for Mormon movies. Women rate Christian movies in my sample higher than men do, across movie category. This is consistent with my guess that women rate Mormon movies higher because women are more religious in general.

Rating Distribution

Another interesting pattern I noticed in the ratings is that there seem to be a lot of raters using the highest and lowest possible ratings. Here’s an example, for the 2006 movie Church Ball.

This is a strange distribution. Almost a quarter of raters give the movie a 1. By itself, that isn’t unusual. What’s odd is that for a typical movie that gets a lot of ratings of 1, there would also be a lot of ratings of 2 and 3. Here though, 2 and 3 are used much less. Instead there’s another peak of rating where a relatively large number of raters give the movie a 5, followed by another decline, and then an abrupt jump again at 10, which ties for the third-most-used rating.

It looks like what’s going on is that there are (at least) three groups of people rating this movie. One group wants the movie to rate as low as possible. Perhaps people in this group hate movies about Mormons and want to be sure the world knows how awful Mormons are. The second group wants the movie to rate as high as possible, probably for the opposite reason. (This is the type of rating manipulation I mentioned at the beginning of the post.) The third group is actually rating the movie based on how much they liked it. It looks like the average is about a 4 or 5, with fewer ratings as you move away from those middle scores. This group probably even includes a few people who legitimately rated the movie 1 or 10. By “legitimately,” I mean they’re not just trying to game the system to give the movie a particularly high or low rating.

Incidentally, this gaming the system is, I suspect, just the type of thing that IMDB is trying to subtract out when they calculate their weighted average ratings I mentioned in the paragraph above the first graph above. Of course, they’re deliberately vague about exactly what they’re doing, because they don’t want their process to be too easy to defeat. But if I were them, I would definitely look at things like whether there’s a disproportionate number of 1’s or 10’s in a movie’s ratings. Plus probably I would also do things like discount the ratings of individual raters more if they were prone to use mostly 1’s and 10’s across all the movies they rated.

Anyway, for comparison, here’s a rating distribution for the movie Saints and Soldiers.

There’s maybe a slight excess of ratings of 1 and 10, given that both are used more than their immediately adjacent category. But unlike with Church Ball, it looks like here the ratings are dominated by the group who are just expressing how much they like the movie, rather than trying to game the ratings.

Going the other direction, here’s a rating distribution that’s completely dominated by the 1’s and 10’s: Meet the Mormons.

The movie is pretty much rated on a two-point scale: loved it or hated it. It makes sense, I think, that a movie that’s made by the Church itself explicitly to be a proselytizing tool, is kind of hard to have an opinion on other than “I love it! The Church is true!” or “Stop trying to get me to convert, you weirdos!”

Here’s another one that looks similar but skews more in the “hated it” direction. The Book of Mormon Movie, Volume 1: The Journey. (Perhaps the abundance of low ratings explains why I didn’t see any subsequent volumes on IMDB.)

This type of gaming the system doesn’t just happen for Mormon movies, of course. It’s often more pronounced for them because they have fewer ratings overall, so a small group of ratings gamers can have more of a noticeable effect. But you can see it for all kinds of movies. I’ve looked at ratings for a lot of highly-rated movies, and the gaming I see most often there is that there’s an excess of 1’s compared to 2’s and 3’s. For example, here’s the rating distribution for The Godfather, which is the second-highest-rated movie on IMDB.

The ratings are dominated by high scores, but down at the bottom, there are more 1’s than there are 2’s through 5’s combined. I would guess that what’s going on here is that if people don’t like a movie that is so generally acclaimed, they might decide to push their rating down to the lowest possible just to counteract all the ratings they see as unjustifiably high. If they had seen the same movie and rated it with no background about how positively many people viewed the movie, they might have rated it 2 or 3 or even 4, but given that they feel they have to push back against a deluded majority, they go to the extreme.

This type of push also happens at the high ratings end. Here’s the rating distribution for the final Harry Potter movie.

The 10’s aren’t way out of line, but it does look suspicious that they markedly outnumber the 9’s, especially given that the clear favorite rating is 8. I would guess that this type of gaming the system might be done by big fans of Harry Potter, who want the movies to rate as high as possible, and so adjust their ratings up from what they might have given if they had watched a similar movie about a Harriet Pitter, for example.

For another comparison, here’s the rating distribution for 12 Monkeys, which seems pretty free of raters trying to push its rating either up or down. Perhaps this movie is saved from ratings gaming by not provoking strong feelings outside the actual perceived quality of the movie (like proselytizing with Mormon movies or being a discussion item in broader culture like The Godfather and Harry Potter).

Highest and lowest rated Mormon movies

Okay, getting back to Mormon movies, I want to show just a few lists of top and bottom rated movies from my sample, and then I’ll be done.

Four of the top five overall are Church made, with the exception being Praise to the Man, which was made by Living Scriptures. I think it’s also maybe interesting that three of the top five are about scriptures or Church history, and the other two are Christmas movies.

I don’t see really any interesting patterns in the women’s most and least favorites versus the men’s, other than that you can see again how much more positive women’s ratings are. Women’s top five go down to an average rating of 8.4, which is the highest point for men’s top five. Women’s bottom five go down to 4.1, while men’s bottom five only go up to 3.6 at the highest.

This last set of tables shows movies most often rated by women and men (among raters who report their gender), two tables that try to capture movies where there has been most gaming of the ratings (percentage of ratings that are either 1 or 10, and largest differences between the calculated average rating and IMDB’s weighted average weighting), and finally a list of the movies that have received the most ratings.

Note that the Pride and Prejudice referred to in a couple of the tables is the 2003 Mormon version.

That’s all I’ve got. Please feel free to weigh in on any of the stuff I’ve shown, or on Mormon movies you particularly like or dislike.