STEAM GROUP
IndieGems IndieGems
STEAM GROUP
IndieGems IndieGems
162
IN-GAME
695
ONLINE
Founded
5 May, 2018
Language
English
Location
United States 
Shalandir 2 Aug, 2018 @ 4:55am
Avoiding Perceived Bias in Reviewing
This is just a statistical note: only 17 of your 260 rated reviews are **NOT RECOMMENDED** (with the other 243 being positive). I find it hard to believe that only 6.5% of indie titles are not worth buying even if I'm a hardcore gaming addict with a very low bar for entertainment, but we all understand critiques are skewed. The problem is, the more skewed they are the more the audience has to adjust expectations to accommodate the reviewer's bias.

I'm not going to dive into the exact reasons for the one-sided analyses because I know we all have limited time to play games much less review them. I just want to say it may be a disservice to the nearly 8000 followers to quietly ignore bad indie games instead of giving them a bad review (perhaps because you think saying nothing is "nicer") - instead you skew perceptions which invites assumptions or accusations or we become jaded with a **RECOMMEND** from IndieGems because almost every game (93.5%) has been given that review.

These assumptions range from benign - like me making excuses for the overwhelmingly positive reviews by guessing the bad games are being "politely" swept under the rug - to more sinister accusations the devs are effectively bribing you with free copies of games for a positive critical reception. Again, zero insinuations but the more balanced and fair something appears, the more positive the reception and I want IndieGems to continue to grow.

Much love, and I really do appreciate all 260 reviews!
Last edited by Shalandir; 2 Aug, 2018 @ 4:56am
< >
Showing 1-4 of 4 comments
YQMaoski 2 Aug, 2018 @ 6:21am 
Basically same reply here:

I think the issue mostly comes from us having a lot of bad games, and none of us wanting to spend time playing them. In the end, we play the games we feel deserve our time and then since they are the games that drew us to play them, there’s a more favorable outlook.

Sometimes I start a game I feel like looks good, but I just can’t get into it. It looks like someone else could enjoy it, and I just spend half hour, or sometimes even less time, before I want to go do something else. I don’t want to give it a negative review just because I can’t get into the game, for personal preference. If I were to write a negative review, it would have to be something I spent some time with, but ultimately decide it’s not worth my time. In the end, time is worth more than the cost of the games themselves, since most of us have way more games than we know what to do with. Thus I think the reviews are for games that we ourselves gravitate to.

Looking back at some of the reviews written, perhaps some of the ones that we gave positive reviews for, is that we often tend to be a bit more generous because we get a lot of background stories from the developers, how big their team is, how much time they have worked on the project, their own personal struggles, motivations for the game. We are not being bribed at all, and you might be surprised at how many games we decide not to review from the curator page, they are there for a month from the time they are offered, and if we don’t want them, they expire. Unfortunately we don’t review everything we get, there are times we sit there in our own discussion laughing because none of us want the additional filler/shovelware in our libraries. I agree, the numbers are skewed. I will be honest, I have a list of games that I am struggling to write reviews for, just because I don’t think the games themselves are very good. In the end, when I sit down to write something, I am drawn away instead to go play something I like, instead of completing the review. I think I need to make a better effort to complete these. But time management has never been my strong suit.

I focus on the actual content of my reviews more than anything else. I prefer to be open about my experience playing the game, what I liked about it, the things that stood out, and my struggles with the game. A lot of my positive reviews outline some issues I had while playing the game, but not enough to deter me from playing. A lot of these developers are quite active on the community forums as well, and they make a pretty good effort to try to improve something if need be, especially if the problems are causing some games to be broken and unplayable.

I don’t much about the data distribution of game scoring, and there are games I am surprised only have 1-2000 reviews when the leaderboards go out past the million mark. I am always surprised that if there are that many people playing, and the review system is set up that anyone can give a thumbs up or down with little to no text, that so many people would play and not mention it ever. I saw a negative review for a game that had a “☐” in the text, and I sat there thinking, well, I guess someone didn’t like the game, but I have no idea why. Even a sentence or two about a bad experience would be helpful.
yoshirules 2 Aug, 2018 @ 6:54am 
Your math isn't right. The sample size is too small
You said: "I find it hard to believe that only 6.5% of indie titles are not worth buying". It's impossible to know if we didn't review all games at Steam.
YQMaoski 2 Aug, 2018 @ 7:04am 
Originally posted by Shalandir:
I find it hard to believe that only 6.5% of indie titles are not worth buying

Let me just answer to this specifically, your math from our review numbers is correct, but once again, we dropped a lot of proffered games that nobody wanted to try, and as a result, the numbers are skewed. But extrapolation toward all indie games is a dangerous assumption. I don’t think we want to imply anything about the way too many games on Steam at the moment. I think to project our current small sample and numbers to the whole is incorrect, however, because we mainly focus on games that we liked up to this point.
dk 2 Aug, 2018 @ 9:59am 
YQMaoski has addressed many issues already.
Your assumptions are reasonable given that quite a few curators have been found to be untrustworthy. This is something we have discussed several times internally.

At the end of the day writing reviews takes time. It is not our full-time job. Often even a short review takes hours to write. It should also be reasonable to understand that since we are not being paid for this, we would rather play and review games that we enjoy. I personally don't care about free games (and I can vouch for yoshirules and YQMaoski). All of us have very healthy backlogs of games that we have purchased with our own money but don't have enough time to play.

Funny enough I just finished writing a negative review on the game that is coming out on Aug 3. It will be posted tomorrow. YQMaoski is currently also working on a review that will be negative.

Not to throw developers under the bus, but here is an example of our current curator offerings (a big portion of the games we receive are similar to these):
https://store.steampowered.com/app/828540/_____Game_of_the_forgotten_Gods_Wake_up/
https://store.steampowered.com/app/861840/Bathroom_Chef/

Does the community need detailed reviews and extensive coverage of games like this? If it does, please do let us know and we will put an extra effort reviewing this kind of gems.
< >
Showing 1-4 of 4 comments
Per page: 1530 50