Pederson and Callahan were simply incompetent. Every program has had to deal with that and it's not a big deal, it just makes you lose a few games. Guys like Phillips and Collins are entirely different. They make the program look guilty by association to a lot of people. That kind of association is very hard to overcome. It takes a whole bunch of good people a very long time.
So, with all due respect to the horrible jobs done by SP and BC, the evil guys are worse for the program. Of those 2, Collins is just a gangster. An inept gangster apparently, but just a gangster none the less. Phillips, on the other hand, is a mean ugly-spirited excuse for a human being. He has repeatedly brought negative publicity on the program, not through any kind of real criminal activity and not through incompetence, but simply by being a bad person. He has had by far the most negative impact on the football program. It's not even close.
I think if you ask most Husker haters over the age of 20 to name the worst thing about the Huskers, Lawrence Phillips will be the knee-jerk answer.