Having lived in San Diego/San Clemente for a few years back when I was in the Marines, I can tell you that the firsherman's restaurant is good, but it's also about an hour drive (In southern Cali traffic) from San Diego, and not sure if I'd say it's worth it or not.
When you're in San Diego, the Gaslamp is a blast, it's akin to the haymarket in Lincoln, but bigger, better, and with more bars, specifically, I loved going to a bar called the yardhouse, which has something like 100+ beers on tap, as well as a place called 'dicks last resort' where the wait staff isn't polite or curteous...they're dicks to everyone, lol.
Old town is a good time, it's a bit more touristy with a lot of little shops and such
There's a huge pier/wharf shopping/eating area right off of downtown on the water, I don't remember what it's called exactly, but there's some nice upscale restaurants and such there, as well as stuff for little kids to do during the day.
If you're planning a family trip, Legoland is about a half hour drive north, and is a lot of fun, if you feel like making the 2 hour drive to LA, Univeral Studios and DisneyLand are always good ones to talk up.