Fans judge their team after every game. That is part of the sport, every sport. The question is if it is fair to call for his head after five games. The honest answer is probably no. To turn around a program takes time.
However, there are both emotional variables and real warning signs all over the place. Some, including me, were uneasy about this hire from the beginning. We were lectured by others that this was a great hire and raising questions meant we were bad fans, had blind loyalty to the last coach, or didn't know enough about the game.
Now, five games in, we are not a fun team to watch, but the losses haven't even told the whole story. The defense is amongst the worst in the game. They don't run the ball. They don't play disciplined and smart. The team has gone from disappointing to a complete counter-culture of everything that defined Nebraska football. It is easy to look at what has been done at Michigan, in the same amount of time, and be envious.
So to answer, no, it is not too early. Everything affects everything. Losses to bottom tier teams impact more than the current win-loss column. How do you think recruiting will be impacted in a conference that has good options closer to the home for many targets? In the social media era, everything is magnified. It will be a long year, but will next year be something to look forward to with the current staff.