Definitely, if the size and complexity of the project is anything more than "simple", I always use Zend Framework, but only for the admin UI. For things where speed matters, , e.g., tracking a million clicks a day through an IM management system, you definitely don't want the overhead of a full MVC framework.
One of my problems has been my reluctance to learn a framework.. I'm always building from scratch, so I know that probably is a bit of time waster.
CodeIgniter is fucking amazing. Layered Links is built on it, and I was able to do all of that coding in about 5 weeks because of CodeIgniter. It's MVC based, but not a picky bitch like Rails so you can hack at it a bit if you need to.
Frameworks are a smart decision when coding. They're generally created by coders who are way fucking smarter than me so I prefer to follow in someone else's footsteps rather than pursue some janky coding path I dreamed up.
I have built multiple large systems using CI, things powering dozens of sites and handling hundreds of users, and it is great for rapid development. It's also easy to get started with, so I recommend it to anyone who wants to move forward a bit with PHP.
Me 3. I have sort of a rough framework that I use for this, basically a set of folders called "blank site" (/css, /images, /js and a blank index.html). In CSS I have the 960 framework stuff and a global reset loosely based on the Eric Myers one. In JS I have minified jQuery, which sometimes I end up yanking out. Whenever I make a new site, I just copy this structure and rename the folder to whatever the project is. Saves me some time, but not a ton. Curious to see what you're doing though.
Zend isn't going to make any more difference than gzip output buffering and properly optimized SQL queries... you are mistaken that developing an application (like you described) around a framework is making any difference whatsoever for performance/security/whatever