Can't handle the traffic - HELP

Engyj

Engyj

New member
Sep 4, 2011
68
5
0
London
Currently running an image blog site that spikes in traffic at certain times in the evening. The moment it hits about 400-500 active users on the site, the server cant handle it. Ive moved from server to server, currently running on Intel core 2 duo 2.16 with 2 gigs of ram (godaddy hosting).

Its almost certainly an issue of CPU, as thats what overloads everytime the traffic spikes.

Needless to say, this is impeding on the sites growth. Is there anything out there that can solve this problem?
 


Have you tried using a CDN? If not this would probably help solve your problems immediately. Either Amazon or MaxCDN would be the first places I would try.

What CMS are you running?
 
The CMS currently running is MySQL.

And no havent tried CDN. On a dedicated server (if that makes a difference)
 
Are you sure CPU is the problem?
Do you have any self-management knowledge? If so, check your server logs and determine what's the problem. Use commands like top to identify the process that's killing your server.

Are you using a custom made CMS or a popular one like WordPress?
A poorly written SQL query can take your server down at the minimum increase of traffic.
 
Top shows that mySQL is the process taking the majority of CPU during traffic spikes.

I dont use a CMS its a custom made one. Its heavily database dependant. Login/rating/commenting. All use SQL queries. I don't believe that they are poorly written but in the instance that they were do you think that it would reduce the numbers of users that can access the website down to 400 based on the system that i am running?
 
Regarding to your question, sincerely I don't know cause it could be a lot of things. I'm not an expert at all but I have some experience. Probably your queries are ok but your server configuration is not optimized. Here you have a nice resource where you can learn to adjust the settings to improve performance:

Troubleshooting Memory and Networking Issues
 
1) Try logging slow queries in MySQL with slow query log option. Fix all slow queries.

2) See if you can use nginx as your front end server. If your app is written in PHP, I highly recommend php-fpm -> nginx setup. Nginx servers static files directly, and passes dynamic requests to php-fpm. On top of that I'd install APC caching for PHP as well, to cache compiled code. Give it at least 64MB.

3) Definitely rewrite your code to use CDN. You can easily do this with "origin pull" setup. Simply prefix all your images with "cdn.yourdomain.com" and configure that to point to CDN. Very easy to do. Should give you a quick fix that you need at least temporarily.

If you are still struggling, and none of the above helped, then you need to refactor your code and do some heavy optimization. To give you some ideas, read on how Reddit manages their infrastructure.
 
Engyj said:
Currently running an image blog site that spikes in traffic at certain times in the evening. The moment it hits about 400-500 active users on the site, the server cant handle it. Ive moved from server to server, currently running on Intel core 2 duo 2.16 with 2 gigs of ram (godaddy hosting).

Its almost certainly an issue of CPU, as thats what overloads everytime the traffic spikes.

Needless to say, this is impeding on the sites growth. Is there anything out there that can solve this problem?
Click to expand...

no, it's actually quite unlikely to be the cpu. here's what happens:

1. when things are running fine, your http server (most likely apache) processes a request quickly and the pool of processes or threads (depending on your chosen processing model) stays relatively stable.
2. then you get a spike in traffic and bottleneck on, say, mysql or the disk.
3. now each apache thread/process takes too long to process each request, so when new requests come in, it forks another process/thread. this is slow and can be expensive, hence the cpu load.

there's a whole lot of other things that can be happening, mind you, with an arbitrary nesting level of brainfuck. for example, extra ram for these new apache processes could be causing mysql indices and even whole tables to now be read from disk rather than from the page cache, which would slow mysql queries down catastrophically, causing even more apache thrashing etc.

it's impossible to diagnose this without knowing a lot of things about both the site and the workload.

here's a list of things you can throw at the problem without diagnosing it (although i would suggest diagnosing it first in the strongest possible terms) in no particular order (assuming php+mysql+wp):

hardware:
* add more ram
* faster hdds with more spindles

* enable php bytecode caching (eaccelerator, apc etc)
* enable output caching (eg total-cache, wp-cache etc)
* stick a caching proxy in front of your httpd (squid, varnish (with pagemem locking) or even better zeus if you've got 15k ;) )
* optimize mysql
* mysql on a separate spindle-set from images
* images in a nested dir structure (if on ext3) or on reiserfs
* serve images with nginx
* move the whole thing to nginx

the list is endless really and my breakfast's getting cold ;)

pm me with your im handles and i'll be happy to take a quick look

-p
 
Engyj said:
Top shows that mySQL is the process taking the majority of CPU during traffic spikes.

I dont use a CMS its a custom made one. Its heavily database dependant. Login/rating/commenting. All use SQL queries. I don't believe that they are poorly written but in the instance that they were do you think that it would reduce the numbers of users that can access the website down to 400 based on the system that i am running?
Click to expand...

sorry about the above post, shouldn't have assumed wordpress and should've read the entire thread ;)

there are three options:

1. any one or more of table design, index design or queries could be bad in the absolute sense.
2. they might just not be properly designed to scale. the query cache could be getting flushed all the time at high concurrency because of writes related to view counters or ratings, for example.
3. constant accesses to images might be displacing mysql indices and tables in the page cache.

and yes, any one of those things could very easily hang everything, nevermind slow it down.

-p
 
rish3 said:
Varnish can be a quick solution while you figure out what's wrong with your mysql setup (probably missing indexes) and investigate a CDN.

http://varnish-cache.org/
Click to expand...

assuming he's outputting correct last-modified headers etc. beyond that, unless you really know what you're doing varnish has some problems under ram contention scenarios. in his case, he'd at the very least need to make sure it wasn't caching images.

-p
 
moltar said:
1) Try logging slow queries in MySQL with slow query log option. Fix all slow queries.
Click to expand...

this won't help by itself. you don't know *why* they're slow - under overload conditions a vast majority of his queries are going to end up getting logged as slow.

mytop + query analysis under normal *and* peak usage would get him a much better list of issues to work on. i've done performance+scalability for a number of the alexa top 5k sites, trust me when i say that slow query logging is next to useless unless you're already running a well-tuned system where a slow query is an outlier

-p
 
Be very careful you're not doing a lot of outer join queries in your code.

We ran into a similar issue with a crm where outer joins ran vs the db were causing MASSIVE load spikes, and in most circumstances you can rewrite somehow as an inner join instead, OR parse the table in a loop and avoid the load on the MYSQL server process.
Problem reared its ugly head when we approached queries that were having to outer join 50,000 plus records then run time compares inside them.
 
You must log in or register to reply here.
Top Bottom